Compare commits
18 Commits
35f9446ed8
...
3013ee044d
| Author | SHA1 | Date | |
|---|---|---|---|
| 3013ee044d | |||
| adfd3bd052 | |||
| 4023c66508 | |||
| adfe3825ee | |||
| 814fcdc6ba | |||
| 66de1a5520 | |||
| 52e6b31408 | |||
| 142ba36695 | |||
| b949e9bbc0 | |||
| ce3a07512f | |||
| 1c84d19e5f | |||
| fc1b57bde2 | |||
| 075888c97f | |||
| 40709e4ad8 | |||
| 15c369c509 | |||
| a81b9f1e1f | |||
| 0327659355 | |||
| c99bad19ae |
@@ -1,6 +1,6 @@
|
|||||||
# ADR-0002: Linear pipes-and-filters pipeline, not a general DAG
|
# ADR-0002: Linear pipes-and-filters pipeline, not a general DAG
|
||||||
|
|
||||||
**Status:** Proposed
|
**Status:** Accepted
|
||||||
**Date:** 2026-07-13
|
**Date:** 2026-07-13
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# ADR-0003: Strongly typed stage interfaces with a two-zone data model
|
# ADR-0003: Strongly typed stage interfaces with a two-zone data model
|
||||||
|
|
||||||
**Status:** Proposed
|
**Status:** Accepted
|
||||||
**Date:** 2026-07-13
|
**Date:** 2026-07-13
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# ADR-0004: Package modules by domain, not by stage
|
# ADR-0004: Package modules by domain, not by stage
|
||||||
|
|
||||||
**Status:** Proposed
|
**Status:** Accepted
|
||||||
**Date:** 2026-07-13
|
**Date:** 2026-07-13
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|||||||
@@ -41,6 +41,7 @@ rejected; execution profiles now come from Scriptorium.
|
|||||||
Built-in defaults:
|
Built-in defaults:
|
||||||
|
|
||||||
- `concurrency.total_llm`: `1`
|
- `concurrency.total_llm`: `1`
|
||||||
|
- `concurrency.stage_workers.extract`: effective `concurrency.total_llm`
|
||||||
- `diagnostics.work_dir`: `/tmp/notarius`
|
- `diagnostics.work_dir`: `/tmp/notarius`
|
||||||
- `diagnostics.retention`: `auto`
|
- `diagnostics.retention`: `auto`
|
||||||
- `workspace.directory`: unset
|
- `workspace.directory`: unset
|
||||||
@@ -88,6 +89,7 @@ These environment variables are applied after the config file:
|
|||||||
|
|
||||||
- `NOTARIUS_CONFIG`: config discovery path.
|
- `NOTARIUS_CONFIG`: config discovery path.
|
||||||
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
|
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
|
||||||
|
- `NOTARIUS_STAGE_WORKERS_EXTRACT`: integer extract worker limit.
|
||||||
- `NOTARIUS_WORKSPACE_DIR`: workspace directory.
|
- `NOTARIUS_WORKSPACE_DIR`: workspace directory.
|
||||||
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`: boolean diagnostics enablement.
|
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`: boolean diagnostics enablement.
|
||||||
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`: workspace diagnostics retention
|
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`: workspace diagnostics retention
|
||||||
@@ -107,6 +109,25 @@ The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider
|
|||||||
endpoint, model, and credential environment variable names through Scriptorium
|
endpoint, model, and credential environment variable names through Scriptorium
|
||||||
profiles.
|
profiles.
|
||||||
|
|
||||||
|
## Concurrency
|
||||||
|
|
||||||
|
`concurrency` fields:
|
||||||
|
|
||||||
|
- `total_llm`: positive integer ceiling on concurrent provider calls.
|
||||||
|
- `stage_workers`: optional map of framework worker limits. The only supported
|
||||||
|
key is `extract`.
|
||||||
|
|
||||||
|
`stage_workers.extract` defaults to the effective `total_llm` value after file
|
||||||
|
and environment precedence. It must be between `1` and `total_llm`, inclusive.
|
||||||
|
Unknown or empty stage-worker keys are rejected. The environment override
|
||||||
|
`NOTARIUS_STAGE_WORKERS_EXTRACT` takes precedence over the file value, as does
|
||||||
|
`NOTARIUS_TOTAL_LLM_CONCURRENCY` for the global ceiling.
|
||||||
|
|
||||||
|
The worker value is present in effective and redacted configuration. It bounds
|
||||||
|
the fixed run-wide extract pool and its bounded dispatch queue. Extract jobs are
|
||||||
|
submitted by source chunk and then resolved lane; `total_llm` independently
|
||||||
|
bounds actual provider calls made by extracts, retries, and validators.
|
||||||
|
|
||||||
## Pipelines
|
## Pipelines
|
||||||
|
|
||||||
A pipeline selects implementations for the fixed workflow defined by
|
A pipeline selects implementations for the fixed workflow defined by
|
||||||
@@ -236,6 +257,10 @@ Binding fields:
|
|||||||
chain; set a non-empty list to use exactly those validators in configured
|
chain; set a non-empty list to use exactly those validators in configured
|
||||||
order.
|
order.
|
||||||
|
|
||||||
|
During resolution, each selected module's registered option validator runs.
|
||||||
|
Production input, chunk, and output bindings reject unknown or invalid options
|
||||||
|
with the affected binding context.
|
||||||
|
|
||||||
Validator bindings use the same shorthand or object module-binding form, but
|
Validator bindings use the same shorthand or object module-binding form, but
|
||||||
only these fields are supported:
|
only these fields are supported:
|
||||||
|
|
||||||
@@ -258,9 +283,9 @@ production validators do not call the LLM and must not set `llm_profile`.
|
|||||||
| input | `seriatim` | Reads Seriatim transcript JSON. |
|
| input | `seriatim` | Reads Seriatim transcript JSON. |
|
||||||
| chunk | `generic` | Splits source units into ordered chunks. |
|
| chunk | `generic` | Splits source units into ordered chunks. |
|
||||||
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
|
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
|
||||||
| extract | `dnd/spells` | Extracts D&D spell raw outputs. |
|
| extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. |
|
||||||
| merge | `appendorder` | Merges JSON raw extract outputs in chunk order. |
|
| merge | `appendorder` | Combines typed artifacts in chunk order. |
|
||||||
| normalize | `noop` | Passes merged raw outputs through unchanged. |
|
| normalize | `noop` | Passes merged typed artifacts through unchanged. |
|
||||||
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
|
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
|
||||||
|
|
||||||
## Implemented Production Validators
|
## Implemented Production Validators
|
||||||
@@ -271,7 +296,7 @@ production validators do not call the LLM and must not set `llm_profile`.
|
|||||||
| `generic/always_reject` | deterministic | Rejects returned module output with reason `always_reject`. |
|
| `generic/always_reject` | deterministic | Rejects returned module output with reason `always_reject`. |
|
||||||
| `generic/valid_json` | deterministic | Rejects payloads that are not syntactically valid JSON. |
|
| `generic/valid_json` | deterministic | Rejects payloads that are not syntactically valid JSON. |
|
||||||
| `generic/valid_json_schema` | deterministic | Rejects invalid JSON or JSON that does not conform to the module response schema. |
|
| `generic/valid_json_schema` | deterministic | Rejects invalid JSON or JSON that does not conform to the module response schema. |
|
||||||
| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-cast JSON payloads. |
|
| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-list artifacts. |
|
||||||
| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
|
| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
|
||||||
| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
|
| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
|
||||||
|
|
||||||
@@ -362,6 +387,8 @@ Configuration validation checks:
|
|||||||
- mutually exclusive `scriptorium.profile_dir` and `scriptorium.profile_file`;
|
- mutually exclusive `scriptorium.profile_dir` and `scriptorium.profile_file`;
|
||||||
- non-empty, non-duplicated IDs after trimming;
|
- non-empty, non-duplicated IDs after trimming;
|
||||||
- positive global LLM concurrency;
|
- positive global LLM concurrency;
|
||||||
|
- supported stage-worker keys and an effective extract worker count in the
|
||||||
|
inclusive range `1..concurrency.total_llm`;
|
||||||
- supported diagnostics retention and non-empty work directory;
|
- supported diagnostics retention and non-empty work directory;
|
||||||
- stale removed fields such as `llm_profiles`.
|
- stale removed fields such as `llm_profiles`.
|
||||||
|
|
||||||
|
|||||||
@@ -1,11 +1,12 @@
|
|||||||
# D&D Spell Raw Output
|
# D&D Spell Artifact
|
||||||
|
|
||||||
This document is the durable raw output contract for the production D&D spell
|
This document is the durable serialized artifact contract for the production
|
||||||
extractor. Selectable extractor keys are cataloged in
|
D&D spell extractor. Selectable extractor keys are cataloged in
|
||||||
[Configuration](../config.md#implemented-production-modules).
|
[Configuration](../config.md#implemented-production-modules).
|
||||||
|
|
||||||
## Identity
|
## Identity
|
||||||
|
|
||||||
|
- Artifact kind: `dnd/spell-list`
|
||||||
- Prompt ID: `dnd.spells`
|
- Prompt ID: `dnd.spells`
|
||||||
- Response schema key: `dnd_spells`
|
- Response schema key: `dnd_spells`
|
||||||
- Response schema ID: `notarius.dnd.spells`
|
- Response schema ID: `notarius.dnd.spells`
|
||||||
@@ -13,6 +14,12 @@ extractor. Selectable extractor keys are cataloged in
|
|||||||
- Response schema version: `v1`
|
- Response schema version: `v1`
|
||||||
- Media type: `application/json`
|
- Media type: `application/json`
|
||||||
|
|
||||||
|
The durable JSON Schema is owned by the D&D spell artifact codec. The
|
||||||
|
extractor's private LLM response schema is a separate transport contract: its
|
||||||
|
source-reference objects omit `source_id`, which the extractor assigns while
|
||||||
|
mapping the response to the canonical artifact. The LLM DTO and transport
|
||||||
|
schema are not part of this durable contract.
|
||||||
|
|
||||||
The output contains canonical spell casts derived from transcript evidence.
|
The output contains canonical spell casts derived from transcript evidence.
|
||||||
Source IDs are assigned from the input identity; source-unit ranges identify
|
Source IDs are assigned from the input identity; source-unit ranges identify
|
||||||
the evidence location.
|
the evidence location.
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ The encoder writes:
|
|||||||
|
|
||||||
- `index.json`
|
- `index.json`
|
||||||
- `manifest.json`
|
- `manifest.json`
|
||||||
- `lanes/<lane-id>.json`, one file per normalized raw lane output
|
- `lanes/<lane-id>.json`, one file per normalized serialized artifact
|
||||||
- `rejected.json`
|
- `rejected.json`
|
||||||
- `warnings.json`
|
- `warnings.json`
|
||||||
|
|
||||||
@@ -112,8 +112,8 @@ Reference `stage` is `chunk`, `extract`, `merge`, or `normalize`. `lane_id` is
|
|||||||
omitted for chunk references and present for extract, merge, and normalize
|
omitted for chunk references and present for extract, merge, and normalize
|
||||||
references.
|
references.
|
||||||
|
|
||||||
`validation_status` is `approved` when no raw outputs were rejected and
|
`validation_status` is `approved` when no outputs were rejected and `rejected`
|
||||||
`rejected` when one or more raw outputs were rejected.
|
when one or more outputs were rejected.
|
||||||
|
|
||||||
`validator_chains` records the resolved validator chain for each validation
|
`validator_chains` records the resolved validator chain for each validation
|
||||||
point. Entries include stage, lane ID when applicable, module key, and validators
|
point. Entries include stage, lane ID when applicable, module key, and validators
|
||||||
@@ -131,12 +131,13 @@ message, attempt count, and optional diagnostic artifact path.
|
|||||||
|
|
||||||
## Output Payload Files
|
## Output Payload Files
|
||||||
|
|
||||||
Each normalized raw output is written to `lanes/<sanitized-lane-id>.json`.
|
Each normalized serialized artifact is written to
|
||||||
The JSON output encoder accepts only `application/json` normalized outputs. The
|
`lanes/<sanitized-lane-id>.json`. The JSON output encoder is domain-neutral and
|
||||||
file contains the raw JSON payload pretty-printed.
|
accepts only artifacts whose codec media type is `application/json`. The file
|
||||||
|
contains the codec-owned JSON bytes pretty-printed.
|
||||||
|
|
||||||
The schema of each lane payload is owned by that artifact contract. For the
|
The schema of each lane payload is owned by that artifact contract. For the
|
||||||
current D&D spell lane, see [D&D Spell Raw Output](dnd-spell-artifacts.md).
|
current D&D spell lane, see [D&D Spell Artifact](dnd-spell-artifacts.md).
|
||||||
|
|
||||||
## `rejected.json`
|
## `rejected.json`
|
||||||
|
|
||||||
@@ -148,7 +149,7 @@ Shape:
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
When raw output validation rejects an output, each entry contains `stage` and
|
When output validation rejects an output, each entry contains `stage` and
|
||||||
`message`. It includes `lane_id`, `module_key`, `chunk_id`, `chunk_index`,
|
`message`. It includes `lane_id`, `module_key`, `chunk_id`, `chunk_index`,
|
||||||
`validator_name`, `reason_code`, `attempt_count`, and
|
`validator_name`, `reason_code`, `attempt_count`, and
|
||||||
`diagnostic_artifact_path` when applicable.
|
`diagnostic_artifact_path` when applicable.
|
||||||
|
|||||||
@@ -52,9 +52,14 @@ Notarius identifies the parsed source in this order:
|
|||||||
2. `metadata.source_id`, when it is a non-empty string after trimming;
|
2. `metadata.source_id`, when it is a non-empty string after trimming;
|
||||||
3. `seriatim:<first-16-hex-chars-of-raw-sha256>`.
|
3. `seriatim:<first-16-hex-chars-of-raw-sha256>`.
|
||||||
|
|
||||||
The source digest recorded in output provenance is `sha256:<hex>` of the exact
|
The exact raw input SHA-256 remains the basis of the fallback source ID. The
|
||||||
raw input bytes. Segment IDs become the unit IDs used by artifact source
|
source digest recorded in output provenance is instead the SHA-256 of the
|
||||||
references.
|
canonical generic source document, excluding the digest field itself. It covers
|
||||||
|
the derived source identity, document kind and format, ordered units and their
|
||||||
|
self-references, and accepted metadata. Segment IDs become the unit IDs used by
|
||||||
|
artifact source references; each produced unit carries a self-reference whose
|
||||||
|
source ID is the derived document ID and whose start and end IDs both equal the
|
||||||
|
segment ID.
|
||||||
|
|
||||||
## Compatibility Limit
|
## Compatibility Limit
|
||||||
|
|
||||||
|
|||||||
@@ -28,12 +28,20 @@ without exposing Scriptorium types through stage contracts.
|
|||||||
|
|
||||||
`internal/cli` constructs the production runtime by:
|
`internal/cli` constructs the production runtime by:
|
||||||
|
|
||||||
1. collecting embedded prompt and response-schema assets from production module
|
1. allocating the asset registry populated by the generic, Seriatim, and D&D
|
||||||
packages;
|
package-family registrars;
|
||||||
2. creating a `ScriptoriumClient` from the effective profile source;
|
2. creating a `ScriptoriumClient` from the effective profile source;
|
||||||
3. attaching an `LLMProfileRecorder`;
|
3. attaching an `LLMProfileRecorder`;
|
||||||
4. creating a scheduler from the effective concurrency limit;
|
4. creating a scheduler from the effective concurrency limit;
|
||||||
5. returning a `ScheduledClient` wrapper.
|
5. returning a `ScheduledClient` wrapper;
|
||||||
|
6. decorating that shared client before preparation when debug recording is
|
||||||
|
enabled; and
|
||||||
|
7. injecting that one shared client into complete pipeline preparation before
|
||||||
|
the source file is read or the runner is invoked.
|
||||||
|
|
||||||
|
The D&D scene chunker and spell extractor retain this injected client and use
|
||||||
|
it for every structured completion. Operation requests do not carry an LLM
|
||||||
|
client.
|
||||||
|
|
||||||
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
|
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
|
||||||
and validator bindings. It prepares a small internal check prompt for each ID so
|
and validator bindings. It prepares a small internal check prompt for each ID so
|
||||||
@@ -75,12 +83,20 @@ cancelled waiter that has already received a permit releases it.
|
|||||||
defers release on every result path. The effective limit and default are
|
defers release on every result path. The effective limit and default are
|
||||||
configuration facts in [Configuration](../config.md#defaults).
|
configuration facts in [Configuration](../config.md#defaults).
|
||||||
|
|
||||||
|
This provider-call ceiling is independent of the pipeline's extract worker
|
||||||
|
limit. Concurrent lanes, retries, and validators all use the same scheduled
|
||||||
|
client, so increasing framework workers cannot exceed `total_llm`. Pipeline
|
||||||
|
dispatch and cancellation mechanics are documented in
|
||||||
|
[Pipeline Internals](pipeline.md#execution-flow).
|
||||||
|
|
||||||
## Prompt And Schema Assets
|
## Prompt And Schema Assets
|
||||||
|
|
||||||
`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes
|
`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes
|
||||||
and rejects invalid or conflicting registrations. Production module packages
|
and rejects invalid or conflicting registrations. Production module packages
|
||||||
register their own prompt and schema assets; generic framework code contains no
|
register their own prompt and schema assets; generic framework code contains no
|
||||||
D&D prompt content.
|
D&D prompt content. `internal/framework/promptfs` provides the domain-neutral
|
||||||
|
filesystem composition helper used to combine module-owned files with shared
|
||||||
|
domain prompt fragments.
|
||||||
|
|
||||||
Schema helpers load embedded JSON Schema with identity and digest metadata,
|
Schema helpers load embedded JSON Schema with identity and digest metadata,
|
||||||
return defensive copies, and expose a diagnostics map that omits schema bytes.
|
return defensive copies, and expose a diagnostics map that omits schema bytes.
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
# Module And Validator Internals
|
# Module And Validator Internals
|
||||||
|
|
||||||
Production stage implementations live under `internal/modules`; production
|
Production module and validator implementations live under their domain-first
|
||||||
validators live under `internal/validators`. The selectable keys, configuration
|
trees in `internal/modules`.
|
||||||
options, reference slots, and default validator chain are canonical in the
|
The selectable keys, configuration options, reference slots, and default
|
||||||
|
validator chain are canonical in the
|
||||||
[module](../config.md#implemented-production-modules) and
|
[module](../config.md#implemented-production-modules) and
|
||||||
[validator](../config.md#implemented-production-validators) catalogs in
|
[validator](../config.md#implemented-production-validators) catalogs in
|
||||||
Configuration.
|
Configuration.
|
||||||
@@ -12,10 +13,27 @@ Configuration.
|
|||||||
A stage module package provides a stable key, constructor, contract
|
A stage module package provides a stable key, constructor, contract
|
||||||
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
|
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
|
||||||
tests. A validator package follows the same pattern with `ValidatorSpec` and the
|
tests. A validator package follows the same pattern with `ValidatorSpec` and the
|
||||||
validator registry.
|
validator registry. Package-family registrars compose those leaf registrations
|
||||||
|
into the production catalog and own family-level policy such as default
|
||||||
|
validator chains and prompt asset collection.
|
||||||
|
|
||||||
|
Production input, chunk, output, and D&D spell-extract packages register strict
|
||||||
|
option decoders and run-local builders. Preparation decodes their options into
|
||||||
|
implementation-owned values and injects dependencies. The spell extractor is
|
||||||
|
typed over the canonical D&D model. D&D validators, merge, and normalize use
|
||||||
|
typed variants; JSON representation validators use serialized requests; and
|
||||||
|
unconditional validators expose separate chunk and typed variants. The D&D
|
||||||
|
production registrar registers only the canonical typed spell implementations.
|
||||||
|
|
||||||
|
Prepared extractors, extract validators, and codecs may be reused concurrently
|
||||||
|
by the run-wide extract pool. Production implementations are immutable after
|
||||||
|
construction: they retain only typed options, immutable assets, or the shared
|
||||||
|
concurrency-safe LLM client. Implementations that introduce mutable state must
|
||||||
|
synchronize that state without creating a separate provider scheduler.
|
||||||
|
|
||||||
Specs expose capability and execution metadata without constructing an
|
Specs expose capability and execution metadata without constructing an
|
||||||
implementation. Chunk, extract, merge, and normalize modules that accept
|
implementation. Registry entries separately expose option validation and
|
||||||
|
run-local construction. Chunk, extract, merge, and normalize modules that accept
|
||||||
auxiliary material declare identical reference slots from both
|
auxiliary material declare identical reference slots from both
|
||||||
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
|
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
|
||||||
that agreement. Runtime delivery uses the corresponding stage request's
|
that agreement. Runtime delivery uses the corresponding stage request's
|
||||||
@@ -23,10 +41,10 @@ that agreement. Runtime delivery uses the corresponding stage request's
|
|||||||
|
|
||||||
LLM-backed extensions own their prompt definitions and response schemas under
|
LLM-backed extensions own their prompt definitions and response schemas under
|
||||||
package-local embedded assets. Shared filesystem composition belongs in
|
package-local embedded assets. Shared filesystem composition belongs in
|
||||||
`internal/modules/sharedassets`; reusable D&D prompt fragments, reference
|
`internal/framework/promptfs`; reusable D&D prompt fragments, reference
|
||||||
declarations, prompt-input assembly, and source-unit helpers belong in
|
declarations, prompt-input assembly, and source-unit helpers belong in
|
||||||
`internal/modules/sharedassets/dnd`. Stage contracts expose only Notarius
|
`internal/modules/dnd/shared`. Stage contracts expose only Notarius structured-
|
||||||
structured-completion types, not Scriptorium public types.
|
completion types, not Scriptorium public types.
|
||||||
|
|
||||||
Reference material may inform a module or prompt but must not become source
|
Reference material may inform a module or prompt but must not become source
|
||||||
evidence. The resolver and materializer behavior is described in
|
evidence. The resolver and materializer behavior is described in
|
||||||
@@ -34,12 +52,13 @@ evidence. The resolver and materializer behavior is described in
|
|||||||
|
|
||||||
## Input Adapter
|
## Input Adapter
|
||||||
|
|
||||||
### `internal/modules/input/seriatim`
|
### `internal/modules/seriatim/input/transcript`
|
||||||
|
|
||||||
The adapter decodes the supported transcript JSON, selects the source identity,
|
The adapter decodes the supported transcript JSON, selects the source identity,
|
||||||
computes the raw-input digest, validates segments, and maps each segment into a
|
computes canonical source provenance, validates segments, and maps each segment
|
||||||
generic source unit with speaker and timestamp metadata. Its spec advertises the
|
into a generic source unit with a self-reference plus speaker and timestamp
|
||||||
transcript capabilities consumed by D&D modules.
|
metadata. It accepts no module options. Its spec advertises the transcript
|
||||||
|
capabilities consumed by D&D modules.
|
||||||
|
|
||||||
Parsing is strict about required values and duplicate unit IDs but deliberately
|
Parsing is strict about required values and duplicate unit IDs but deliberately
|
||||||
ignores unrelated Seriatim fields. The external format and derived-identity
|
ignores unrelated Seriatim fields. The external format and derived-identity
|
||||||
@@ -48,23 +67,29 @@ rules are defined in the
|
|||||||
|
|
||||||
## Chunkers
|
## Chunkers
|
||||||
|
|
||||||
### `internal/modules/chunk/generic`
|
### `internal/modules/generic/chunk/units`
|
||||||
|
|
||||||
The generic chunker validates the source document, walks units in configured
|
The generic chunker validates the source document, walks units in configured
|
||||||
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
|
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
|
||||||
Overlap changes the next window start but never reorders units. It records the
|
Overlap changes the next window start but never reorders units. It records the
|
||||||
first and last unit and unit count in chunk metadata.
|
first and last unit and unit count in chunk metadata, and derives the chunk's
|
||||||
|
canonical source reference from those unit references.
|
||||||
|
|
||||||
The accepted options and defaults are defined in
|
The accepted options and defaults are defined in
|
||||||
[Configuration](../config.md#implemented-production-modules). Generic
|
[Configuration](../config.md#implemented-production-modules). Generic
|
||||||
framework validation canonicalizes the returned unit slices before extraction.
|
framework validation canonicalizes the returned unit slices before extraction.
|
||||||
|
The chunker decodes its options during construction and retains only the typed
|
||||||
|
window settings used by `Chunk`.
|
||||||
|
|
||||||
### `internal/modules/chunk/dnd/scenes`
|
### `internal/modules/dnd/chunk/scenes`
|
||||||
|
|
||||||
The scene chunker prepares a structured Scriptorium request from the full
|
The scene chunker prepares a structured Scriptorium request from the full
|
||||||
transcript, session, and optional D&D reference inputs. It validates the model's
|
transcript, session, and optional D&D reference inputs. It validates the model's
|
||||||
scene boundaries against source-unit IDs and converts them into deterministic
|
scene boundaries against source-unit IDs and converts them into deterministic
|
||||||
chunks.
|
chunks with canonical source references spanning each scene's units.
|
||||||
|
Preparation injects the shared structured LLM client into the chunker; `Chunk`
|
||||||
|
supplies only the run-specific profile, session, source, references, and
|
||||||
|
metadata.
|
||||||
|
|
||||||
Scene validation requires sequential, contiguous, non-overlapping coverage from
|
Scene validation requires sequential, contiguous, non-overlapping coverage from
|
||||||
the first source unit through the last. Each chunk contains JSON scene content
|
the first source unit through the last. Each chunk contains JSON scene content
|
||||||
@@ -79,40 +104,42 @@ file types remain canonical in [Configuration](../config.md).
|
|||||||
|
|
||||||
## Extractor
|
## Extractor
|
||||||
|
|
||||||
### `internal/modules/extract/dnd/spells`
|
### `internal/modules/dnd/extract/spells`
|
||||||
|
|
||||||
The spell extractor prepares a structured request from one chunk, the
|
The spell extractor prepares a structured request from one chunk, the
|
||||||
chunk-scoped source input, the session, and optional D&D reference inputs. It
|
chunk-scoped source input, the session, and optional D&D reference inputs. It
|
||||||
decodes the model response, assigns the generic source identity to every source
|
decodes the model response, assigns the generic source identity to every source
|
||||||
reference, canonicalizes duplicate references, orders spell casts by their
|
reference, canonicalizes duplicate references, orders spell casts by their
|
||||||
earliest cited unit, and returns raw JSON plus response-schema provenance.
|
earliest cited unit, and returns `dnd.SpellList`.
|
||||||
|
|
||||||
The package owns its embedded prompt, response schemas, and prompt/schema
|
The extractor owns its private model-response DTO, embedded prompt, LLM response
|
||||||
manifest metadata. Shared D&D helpers keep prompt input names and source-unit
|
schema, strict option decoder, injected shared LLM client, and prompt/schema
|
||||||
reference conversion consistent with the scene chunker. The extractor produces
|
manifest metadata. The separate `internal/modules/dnd/codec/spells` package
|
||||||
raw output; production validators own approval policy.
|
owns the durable schema and stable JSON representation for artifact kind
|
||||||
|
`dnd/spell-list`. The runner keeps the result typed through validators and later
|
||||||
|
stages, using the codec only for checkpoint, debug, and output boundaries.
|
||||||
|
Shared D&D helpers keep prompt input
|
||||||
|
names and source-unit reference conversion consistent with the scene chunker.
|
||||||
|
|
||||||
The durable payload and manifest metadata shapes are defined in the
|
The durable payload and manifest metadata shapes are defined in the
|
||||||
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
|
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
|
||||||
|
|
||||||
## Merger And Normalizer
|
## Merger And Normalizer
|
||||||
|
|
||||||
### `internal/modules/merge/appendorder`
|
### `internal/modules/generic/merge/appendorder`
|
||||||
|
|
||||||
The merger preserves extract-result order. It passes through one JSON result,
|
The merger passes typed values to an injected combine function in framework
|
||||||
concatenates a common top-level array field across multiple JSON objects, and
|
source-chunk order. The D&D registrar specializes it with a spell-list append
|
||||||
otherwise emits an array of the decoded values. It rejects invalid JSON and
|
function.
|
||||||
non-JSON media types, and it preserves compatible schema provenance.
|
|
||||||
|
|
||||||
### `internal/modules/normalize/noop`
|
### `internal/modules/generic/normalize/noop`
|
||||||
|
|
||||||
The normalizer defensively clones the accepted merge result, including payload
|
The normalizer returns the merged domain value unchanged and is reusable for
|
||||||
bytes, metadata, warnings, and schema provenance, without changing its logical
|
any registered artifact type.
|
||||||
content.
|
|
||||||
|
|
||||||
## Output Encoder
|
## Output Encoder
|
||||||
|
|
||||||
### `internal/modules/output/json`
|
### `internal/modules/generic/output/json`
|
||||||
|
|
||||||
The JSON encoder sorts normalized results by lane, derives collision-checked
|
The JSON encoder sorts normalized results by lane, derives collision-checked
|
||||||
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
|
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
|
||||||
@@ -125,26 +152,26 @@ paths and schemas.
|
|||||||
|
|
||||||
## Generic Validators
|
## Generic Validators
|
||||||
|
|
||||||
The unconditional accept and reject validators provide deterministic production
|
The generic validator implementations live under
|
||||||
registrations used primarily for controlled composition and tests.
|
`internal/modules/generic/validate`.
|
||||||
|
|
||||||
The JSON syntax validator uses `encoding/json` to reject malformed payloads. The
|
The unconditional accept and reject validators provide explicit chunk and
|
||||||
JSON Schema validator requires schema bytes on the validation request, parses
|
typed-artifact variants used primarily for controlled composition and tests.
|
||||||
the instance and schema with `jsonschema`, and distinguishes payload rejection
|
|
||||||
from schema loading or compilation errors. Neither validator calls the LLM.
|
The serialized JSON syntax validator uses `encoding/json` to reject malformed
|
||||||
|
representation bytes. The serialized JSON Schema validator requires schema
|
||||||
|
bytes, parses the instance and schema with `jsonschema`, and distinguishes
|
||||||
|
payload rejection from schema loading or compilation errors. The framework
|
||||||
|
serialized-validation request carries either canonical chunk bytes or artifact
|
||||||
|
codec bytes according to its target context. Neither validator calls the LLM.
|
||||||
|
|
||||||
## D&D Spell Validators
|
## D&D Spell Validators
|
||||||
|
|
||||||
`internal/validators/extract/dnd/spells/spellpayload` provides strict decoding,
|
All three validators receive `dnd.SpellList` directly. The shape validator
|
||||||
shape checks, source-reference candidates, and cited-text lookup shared by the
|
rejects missing or empty spell fields and empty reference lists. The
|
||||||
three validators.
|
source-reference validator applies generic source-reference validation to every
|
||||||
|
cited range. The relatedness validator warns when a case-insensitive spell name
|
||||||
The shape validator rejects malformed JSON, unknown fields, missing or empty
|
is absent from all cited source text.
|
||||||
spell fields, and empty reference lists. The source-reference validator applies
|
|
||||||
generic source-reference validation to every cited range. The relatedness
|
|
||||||
validator approves structurally valid payloads but warns when a case-insensitive
|
|
||||||
spell name is absent from all cited source text. It leaves malformed payloads to
|
|
||||||
the earlier validators in the configured chain.
|
|
||||||
|
|
||||||
These validators are deterministic. Their selectable keys and production order
|
These validators are deterministic. Their selectable keys and production order
|
||||||
are defined in
|
are defined in
|
||||||
@@ -154,10 +181,13 @@ payload rules are defined in the
|
|||||||
|
|
||||||
## Production Registration
|
## Production Registration
|
||||||
|
|
||||||
`internal/cli/catalog.go` builds the production registries, registers module and
|
The CLI allocates one complete framework registry set and one LLM asset
|
||||||
validator constructors, installs default validator-chain mappings, and exposes
|
registry. It invokes `internal/modules/generic/register`,
|
||||||
the matching catalog for resolution. It also collects prompt assets from
|
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
|
||||||
LLM-backed packages before constructing the production client.
|
that order, then exposes the matching catalog for resolution. The generic and
|
||||||
|
Seriatim registrars own their production leaf registrations. The D&D registrar
|
||||||
|
owns D&D leaf registrations, the spell default-validator chain, and D&D
|
||||||
|
prompt/schema asset collection.
|
||||||
|
|
||||||
Framework packages must not import production extensions. Tests may compose
|
Framework packages must not import production extensions. Tests may compose
|
||||||
registries and catalogs directly with fakes.
|
registries and catalogs directly with fakes.
|
||||||
@@ -170,8 +200,8 @@ When adding a production module or validator:
|
|||||||
2. expose and test its spec, constructor, and registration function;
|
2. expose and test its spec, constructor, and registration function;
|
||||||
3. keep format or domain parsing inside the concrete package;
|
3. keep format or domain parsing inside the concrete package;
|
||||||
4. add package-owned prompt/schema assets when the extension is LLM-backed;
|
4. add package-owned prompt/schema assets when the extension is LLM-backed;
|
||||||
5. register it in `internal/cli/catalog.go` and add a default chain only when
|
5. register it through its package-family registrar and add a default chain
|
||||||
production policy requires one;
|
there only when production policy requires one;
|
||||||
6. add resolution and composition coverage for capabilities, options,
|
6. add resolution and composition coverage for capabilities, options,
|
||||||
references, and validation behavior;
|
references, and validation behavior;
|
||||||
7. update the selectable-key catalog in [Configuration](../config.md), the
|
7. update the selectable-key catalog in [Configuration](../config.md), the
|
||||||
@@ -184,11 +214,13 @@ does not inventory implementations.
|
|||||||
## Tests To Inspect
|
## Tests To Inspect
|
||||||
|
|
||||||
- Package-local `*_test.go` files under the module or validator being changed.
|
- Package-local `*_test.go` files under the module or validator being changed.
|
||||||
- `internal/framework/pipeline/registry_integration_test.go`: registry and spec
|
- `internal/framework/pipeline/typed_resolution_test.go`: typed registry, spec,
|
||||||
composition.
|
and heterogeneous artifact composition.
|
||||||
- `internal/framework/pipeline/default_modules_test.go`: framework binding
|
- `internal/framework/pipeline/default_modules_test.go`: framework binding
|
||||||
defaults.
|
defaults.
|
||||||
- `internal/cli/run_test.go`: production catalog, config resolution, and
|
- `internal/cli/run_test.go`: production catalog, config resolution, and
|
||||||
end-to-end CLI composition.
|
end-to-end CLI composition.
|
||||||
- `internal/modules/sharedassets/**/*_test.go`: shared prompt and reference
|
- `internal/framework/promptfs/*_test.go` and
|
||||||
assembly.
|
`internal/modules/dnd/shared/*_test.go`: shared prompt and reference assembly.
|
||||||
|
- `internal/modules/integration/*_test.go`: black-box composition across
|
||||||
|
production extension domains.
|
||||||
|
|||||||
@@ -14,15 +14,18 @@ collaborators, invokes `internal/framework/pipeline`, and places the logical
|
|||||||
output files returned by the runner. Diagnostics, checkpoints, and debug
|
output files returned by the runner. Diagnostics, checkpoints, and debug
|
||||||
recorders are optional side-channel collaborators supplied at this boundary.
|
recorders are optional side-channel collaborators supplied at this boundary.
|
||||||
|
|
||||||
Pipeline execution is serial. Resolution produces a fixed ordered workflow and
|
Resolution produces a fixed ordered workflow and a sorted set of artifact
|
||||||
a sorted set of artifact lanes before the runner constructs any stage module.
|
lanes. Preparation constructs the complete module and validator set before the
|
||||||
|
runner receives source bytes. Source parsing and chunking are serial; extraction
|
||||||
|
uses a bounded run-wide worker pool, followed by serial per-lane merge and
|
||||||
|
normalize continuations that may overlap across lanes.
|
||||||
|
|
||||||
## Application Boundary
|
## Application Boundary
|
||||||
|
|
||||||
| Package | Implemented responsibility |
|
| Package | Implemented responsibility |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `cmd/notarius` | Executable entry point and process exit delegation. |
|
| `cmd/notarius` | Executable entry point and process exit delegation. |
|
||||||
| `internal/cli` | Command parsing, config discovery, production registration, prompt asset collection, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
|
| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
|
||||||
|
|
||||||
## Core Packages
|
## Core Packages
|
||||||
|
|
||||||
@@ -31,23 +34,42 @@ a sorted set of artifact lanes before the runner constructs any stage module.
|
|||||||
| `internal/core/artifacts` | Run-manifest and provenance models. |
|
| `internal/core/artifacts` | Run-manifest and provenance models. |
|
||||||
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
||||||
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
|
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
|
||||||
| `internal/core/source` | Generic source documents, units, references, lookup, and validation. |
|
| `internal/core/source` | Generic source documents, units, chunks, canonical references, lookup, validation, and deterministic source digests. |
|
||||||
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
|
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
|
||||||
|
|
||||||
## Framework Packages
|
## Framework Packages
|
||||||
|
|
||||||
| Package | Implemented responsibility |
|
| Package | Implemented responsibility |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `internal/framework/contracts` | Stage, validator, reference, output, and structured-completion interfaces and data types. |
|
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
|
||||||
| `internal/framework/pipeline` | Registries, profile resolution, capability checks, reference materialization, validator-chain resolution, retries, orchestration, warnings, and manifest population. |
|
| `internal/framework/pipeline` | Module and artifact-codec registries, option validation, profile resolution, capability checks, reference materialization, complete pipeline preparation, retries, orchestration, warnings, and manifest population. |
|
||||||
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
|
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
|
||||||
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
|
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
|
||||||
|
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
|
||||||
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
|
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
|
||||||
| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
|
| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
|
||||||
|
|
||||||
Framework contracts carry raw stage results between implementations. The
|
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
|
||||||
runner owns handoff provenance, validation sequencing, rejection handling,
|
serialized-validator, and
|
||||||
checkpoint and debug boundaries, and final manifest assembly.
|
typed-validator interfaces. The runner owns handoff provenance, validation
|
||||||
|
sequencing, rejection handling, checkpoint and debug boundaries, and final
|
||||||
|
manifest assembly.
|
||||||
|
|
||||||
|
Artifact registries support heterogeneous typed extraction entries and
|
||||||
|
kind-specific merger, normalizer, and validator variants. Resolution derives a
|
||||||
|
lane's kind from its extractor, requires the matching codec, verifies exact Go
|
||||||
|
type equality across the lane, and records schema identity in the resolved lane
|
||||||
|
and pipeline digest. Registry entries carry separate option-validation and
|
||||||
|
run-local construction closures. Preparation injects shared dependencies and
|
||||||
|
constructs input, chunk, validators, ordered lanes, and output before source
|
||||||
|
parsing. Production modules use strict construction-time option decoding, and
|
||||||
|
LLM-backed modules retain the injected shared client. The D&D family registers
|
||||||
|
the canonical `dnd/spell-list` codec, typed spell extractor and validators, and
|
||||||
|
kind-specific generic merge and normalize strategies; generic JSON validators
|
||||||
|
use the serialized-validation contract. The runner executes lanes through
|
||||||
|
private exact-type-checked closures, coordinates extract results independently
|
||||||
|
of completion timing, and serializes artifacts only through their codec at
|
||||||
|
checkpoint, debug, and output boundaries.
|
||||||
|
|
||||||
## Production Extensions
|
## Production Extensions
|
||||||
|
|
||||||
@@ -58,24 +80,38 @@ Configuration. The implemented module packages are:
|
|||||||
|
|
||||||
| Package | Implemented responsibility |
|
| Package | Implemented responsibility |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `internal/modules/input/seriatim` | Parses the supported Seriatim transcript format into the generic source model. |
|
| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. |
|
||||||
| `internal/modules/chunk/generic` | Splits ordered source units by unit count and overlap. |
|
| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. |
|
||||||
| `internal/modules/chunk/dnd/scenes` | Produces contiguous D&D scene chunks from structured model output. |
|
| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. |
|
||||||
| `internal/modules/extract/dnd/spells` | Produces source-grounded D&D spell-cast raw output. |
|
| `internal/modules/dnd` | Owns the canonical D&D spell-list and spell-cast artifact types. |
|
||||||
| `internal/modules/merge/appendorder` | Combines accepted extraction results in chunk order. |
|
| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. |
|
||||||
| `internal/modules/normalize/noop` | Preserves accepted merged output. |
|
| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. |
|
||||||
| `internal/modules/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
|
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
|
||||||
|
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
|
||||||
|
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
|
||||||
|
|
||||||
`internal/modules/sharedassets` composes shared prompt filesystems.
|
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
|
||||||
`internal/modules/sharedassets/dnd` owns reusable D&D prompt fragments,
|
|
||||||
reference declarations, prompt input assembly, and source-unit reference
|
reference declarations, prompt input assembly, and source-unit reference
|
||||||
helpers.
|
helpers. Domain-neutral prompt filesystem composition lives in
|
||||||
|
`internal/framework/promptfs`.
|
||||||
|
|
||||||
Concrete validators live under `internal/validators`. Generic packages provide
|
Generic validators under `internal/modules/generic/validate` provide
|
||||||
unconditional test decisions, JSON syntax validation, and JSON Schema
|
unconditional test decisions, JSON syntax validation, and JSON Schema
|
||||||
validation. D&D spell packages provide shape, source-reference, and
|
validation. D&D spell validators under `internal/modules/dnd/validate/spells`
|
||||||
source-relatedness decisions, with `spellpayload` holding their shared parser
|
consume the canonical spell-list type directly to provide shape,
|
||||||
and lookup helpers. Production chain composition is owned by `internal/cli`.
|
source-reference, and source-relatedness decisions.
|
||||||
|
|
||||||
|
Production composition is grouped behind package-family registrars, and every
|
||||||
|
implemented production extension uses its domain-first tree:
|
||||||
|
|
||||||
|
| Package | Implemented responsibility |
|
||||||
|
| --- | --- |
|
||||||
|
| `internal/modules/generic/register` | Registers domain-neutral chunk, merge, normalize, output, and validator implementations. |
|
||||||
|
| `internal/modules/seriatim/register` | Registers the Seriatim input adapter. |
|
||||||
|
| `internal/modules/dnd/register` | Registers D&D modules, validators, default validator policy, and prompt/schema assets. |
|
||||||
|
|
||||||
|
The CLI allocates the framework registries and asset registry, then invokes
|
||||||
|
these registrars in generic, Seriatim, and D&D order.
|
||||||
|
|
||||||
Implementation details for all production extensions are in
|
Implementation details for all production extensions are in
|
||||||
[Module Internals](modules.md).
|
[Module Internals](modules.md).
|
||||||
|
|||||||
@@ -6,8 +6,12 @@ Their fixed workflow and ownership boundaries are defined by
|
|||||||
defaults, and selectable keys are defined in
|
defaults, and selectable keys are defined in
|
||||||
[Configuration](../config.md#pipelines).
|
[Configuration](../config.md#pipelines).
|
||||||
|
|
||||||
Pipeline execution is serial. Resolution fixes the selected lanes and all
|
Resolution fixes the selected lanes and all stage bindings; preparation
|
||||||
stage bindings before the runner constructs stage implementations.
|
constructs every selected implementation before the runner begins source work.
|
||||||
|
After serial input parsing and chunking, the runner dispatches extract work to
|
||||||
|
one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
|
||||||
|
merge and normalize operations remain serial and may overlap other lanes once
|
||||||
|
all extracts for that lane are terminal.
|
||||||
|
|
||||||
## Resolution
|
## Resolution
|
||||||
|
|
||||||
@@ -20,9 +24,14 @@ calls `pipeline.ResolvePipeline`.
|
|||||||
1. selects and sorts artifact lanes;
|
1. selects and sorts artifact lanes;
|
||||||
2. completes omitted bindings using the documented configuration defaults;
|
2. completes omitted bindings using the documented configuration defaults;
|
||||||
3. looks up each module and validator spec without constructing it;
|
3. looks up each module and validator spec without constructing it;
|
||||||
4. checks required and provided capabilities in workflow order;
|
4. for a typed extractor, derives its artifact kind, requires the codec, and
|
||||||
5. resolves target-aware reference bindings and validator chains;
|
selects exact-type merger, normalizer, and validator variants;
|
||||||
6. calculates a digest over the resolved structure.
|
5. checks required and provided capabilities in workflow order;
|
||||||
|
6. resolves target-aware reference bindings and validator chains;
|
||||||
|
7. validates each selected module and validator option set through its registry
|
||||||
|
entry; and
|
||||||
|
8. calculates a digest over the resolved structure, including typed artifact
|
||||||
|
kind and schema identity.
|
||||||
|
|
||||||
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
|
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
|
||||||
bindings, validator chains, reference targets, and the digest. It does not read
|
bindings, validator chains, reference targets, and the digest. It does not read
|
||||||
@@ -50,31 +59,76 @@ runtime sensitive-data handling belongs in [Operations](../operations.md).
|
|||||||
|
|
||||||
## Registries And Specs
|
## Registries And Specs
|
||||||
|
|
||||||
`pipeline.Registries` holds constructors used during execution.
|
`pipeline.Registries` holds option validators and run-local builders used during
|
||||||
|
resolution and preparation.
|
||||||
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
|
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
|
||||||
resolution. Separate registries exist for every stage and for validators;
|
resolution. Separate registries exist for every stage and for validators;
|
||||||
`ValidatorChainRegistry` stores production default-chain mappings.
|
`ValidatorChainRegistry` stores production default-chain mappings. Both
|
||||||
|
containers also carry an `ArtifactCodecRegistry`. Generic registration records
|
||||||
|
one codec per stable artifact kind, validates its schema metadata and JSON
|
||||||
|
Schema, retains the exact schema digest and Go type, and safely encodes or
|
||||||
|
decodes framework-erased values with typed errors on incompatibility.
|
||||||
|
|
||||||
|
Typed extractor entries are keyed by module key and declare one artifact kind.
|
||||||
|
Merger, normalizer, and typed-validator variants are keyed by module or
|
||||||
|
validator key plus artifact kind. Chunk and serialized validators occupy
|
||||||
|
separate target namespaces; serialized registrations declare whether they
|
||||||
|
support chunks, artifacts, or both. Duplicate variants and exact Go-type
|
||||||
|
mismatches are rejected deterministically.
|
||||||
|
|
||||||
|
Production composition registers the D&D spell-list codec and typed extractor,
|
||||||
|
matching typed merge, normalize, and semantic-validator variants, and
|
||||||
|
serialized JSON validators. Every artifact lane resolves through the typed
|
||||||
|
registries and a matching codec.
|
||||||
|
|
||||||
A `ModuleSpec` declares its stage plus required and provided capabilities.
|
A `ModuleSpec` declares its stage plus required and provided capabilities.
|
||||||
Chunk, extract, merge, and normalize specs may also declare reference slots.
|
Chunk, extract, merge, and normalize specs may also declare reference slots.
|
||||||
Registry implementations defensively copy spec metadata, reject duplicate keys,
|
Registry implementations defensively copy spec metadata, reject duplicate keys,
|
||||||
and verify that a constructed implementation reports the registered key.
|
and verify that a constructed implementation reports the registered key.
|
||||||
|
Builder registrations accept `ModuleDependencies` and cloned configuration
|
||||||
|
options through one `BuildRequest`. Builders decode those options and retain
|
||||||
|
typed values or injected dependencies in the constructed implementation.
|
||||||
|
Extractors declare their artifact kind, and merger, normalizer, and validator
|
||||||
|
resolution selects the matching typed variant.
|
||||||
|
|
||||||
A `ValidatorSpec` declares a validator key and execution class. Resolution uses
|
A `ValidatorSpec` declares a validator key and execution class. Resolution uses
|
||||||
the execution class to reject incompatible profile bindings before execution.
|
the execution class to reject incompatible profile bindings before execution.
|
||||||
The current production catalog and default chain are listed only in
|
The current production catalog and default chain are listed only in
|
||||||
[Configuration](../config.md#implemented-production-validators).
|
[Configuration](../config.md#implemented-production-validators).
|
||||||
|
|
||||||
## Runner Boundary
|
## Preparation And Runner Boundary
|
||||||
|
|
||||||
`pipeline.RunInput` carries the resolved pipeline, raw source input, structured
|
`pipeline.Prepare` receives a resolved pipeline, the registries, and shared
|
||||||
LLM client, run identity and timing, optional session and profile metadata, and
|
module dependencies. It constructs input; chunk and its validators; each lane's
|
||||||
checkpoint/debug collaborators. The runner parses source bytes through the
|
extract, merge, and normalize modules and validator chains in resolved order;
|
||||||
selected input adapter. Later stage requests receive the generic source model;
|
then output. It stops at the first error with pipeline, stage, lane, module, and
|
||||||
extract requests receive chunk-scoped input material, while chunk, merge, and
|
validator context as applicable. It never invokes an operation method.
|
||||||
normalize requests retain access to the original source material.
|
|
||||||
|
|
||||||
`pipeline.RunOutput` carries the run manifest, accepted normalized results,
|
`PreparedPipeline` keeps private constructed executors and exposes cloned
|
||||||
|
resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
|
||||||
|
that prepared pipeline, raw source input, run identity and timing, optional
|
||||||
|
session and profile metadata, and checkpoint/debug collaborators. The runner
|
||||||
|
parses source bytes through the already constructed input adapter. Later stage
|
||||||
|
requests receive the generic source model; extract requests receive
|
||||||
|
chunk-scoped input material, while chunk, merge, and normalize requests retain
|
||||||
|
access to the original source material. Input, chunk, and output operation
|
||||||
|
requests do not carry raw module options. The chunk request also does not carry
|
||||||
|
an LLM client; an LLM-backed chunker receives the shared client during
|
||||||
|
preparation. Their operation requests retain run-specific source, reference,
|
||||||
|
profile, session, and metadata context as applicable.
|
||||||
|
|
||||||
|
Prepared lanes retain exact-type-checked erased operation closures. The runner
|
||||||
|
uses those closures to keep each value typed through extraction, validation,
|
||||||
|
merge, and normalization.
|
||||||
|
|
||||||
|
Source validation requires every unit to carry a canonical self-reference to
|
||||||
|
its containing document and its own unit ID. Explicit clone, checkpoint, and
|
||||||
|
debug boundaries retain that reference, and the canonical source digest covers
|
||||||
|
it deterministically. Chunks use the same source model and carry one canonical
|
||||||
|
reference spanning the first selected unit through the last.
|
||||||
|
|
||||||
|
`pipeline.RunOutput` carries the run manifest, accepted normalized serialized
|
||||||
|
artifacts with lane and normalizer provenance,
|
||||||
rejected results, warnings, checkpoint events, and logical files returned by the
|
rejected results, warnings, checkpoint events, and logical files returned by the
|
||||||
output encoder. The CLI owns diagnostics and durable filesystem writes after the
|
output encoder. The CLI owns diagnostics and durable filesystem writes after the
|
||||||
runner returns.
|
runner returns.
|
||||||
@@ -83,21 +137,22 @@ runner returns.
|
|||||||
|
|
||||||
The runner:
|
The runner:
|
||||||
|
|
||||||
1. validates its input and registries;
|
1. validates its prepared input;
|
||||||
2. builds the input adapter, parses the raw input, and validates the generic
|
2. parses the raw input with the prepared adapter and validates the generic
|
||||||
source document;
|
source document;
|
||||||
3. obtains or executes the chunk result;
|
3. obtains or executes the chunk result;
|
||||||
4. validates and canonicalizes chunks;
|
4. validates and canonicalizes chunks;
|
||||||
5. executes each resolved artifact lane in order;
|
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
|
||||||
6. builds the output encoder and validates its logical file results;
|
bounded lane continuation when all extracts for that lane are terminal;
|
||||||
|
6. invokes the prepared output encoder and validates its logical file results;
|
||||||
7. returns the assembled manifest, outcomes, warnings, and files.
|
7. returns the assembled manifest, outcomes, warnings, and files.
|
||||||
|
|
||||||
Within each artifact lane, it builds the extractor, merger, and normalizer,
|
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
|
||||||
then performs these transitions:
|
and validators while performing these transitions:
|
||||||
|
|
||||||
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
|
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
|
||||||
provenance;
|
provenance;
|
||||||
2. validate each raw extract result and omit rejected results from merge input;
|
2. validate each extract result and omit rejected results from merge input;
|
||||||
3. skip the rest of the lane when no extract result is accepted;
|
3. skip the rest of the lane when no extract result is accepted;
|
||||||
4. merge accepted extract results in their existing order;
|
4. merge accepted extract results in their existing order;
|
||||||
5. validate the merge result and skip normalization on rejection;
|
5. validate the merge result and skip normalization on rejection;
|
||||||
@@ -107,25 +162,35 @@ then performs these transitions:
|
|||||||
Module-provided warnings and payload warnings are promoted only from attempts
|
Module-provided warnings and payload warnings are promoted only from attempts
|
||||||
whose results are accepted and used.
|
whose results are accepted and used.
|
||||||
|
|
||||||
|
The extract job channel has the same capacity as the effective extract worker
|
||||||
|
count, so dispatch applies backpressure. A fixed continuation executor prevents
|
||||||
|
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
|
||||||
|
continuations publish lane-local results; the coordinator is the only writer of
|
||||||
|
aggregate output and merges those results in resolved lane and source-chunk
|
||||||
|
order.
|
||||||
|
|
||||||
## Chunk Canonicalization
|
## Chunk Canonicalization
|
||||||
|
|
||||||
Before lane execution, generic validation requires unique chunk IDs, matching
|
Before lane execution, generic validation requires unique chunk IDs, matching
|
||||||
source identity, indexes matching returned order, valid ordered boundaries,
|
source identity, indexes matching returned order, a valid canonical reference,
|
||||||
non-empty content and media type, and at least one valid source unit per chunk.
|
non-empty content and media type, and at least one valid source unit per chunk.
|
||||||
Units may not repeat inside a chunk and must preserve source-document order.
|
Units may not repeat inside a chunk and must form a contiguous range in
|
||||||
|
source-document order. The chunk reference must exactly match the source and
|
||||||
|
the first and last unit references.
|
||||||
|
|
||||||
The runner then rebuilds each chunk's unit slice from the source document by
|
The runner then rebuilds each chunk's unit slice from the source document by
|
||||||
unit ID. It preserves the module-owned boundaries, content, media type, and
|
unit ID. It preserves the canonical reference, content, media type, and cloned
|
||||||
cloned metadata. The framework permits gaps and overlap between separate
|
metadata. The framework permits gaps and overlap between separate
|
||||||
chunks; stricter coverage policy belongs to the chunk implementation.
|
chunks; stricter coverage policy belongs to the chunk implementation.
|
||||||
|
|
||||||
## Validation And Retries
|
## Validation And Retries
|
||||||
|
|
||||||
Chunk, extract, merge, and normalize results pass through the resolved validator
|
Chunk, extract, merge, and normalize results pass through the resolved validator
|
||||||
chain for their stage and module. Each validator receives the raw payload plus
|
chain for their stage and module. Chunk validators receive canonical chunks;
|
||||||
the relevant source, chunk, prior-stage, schema, reference, session, LLM, option,
|
typed validators receive the domain value; and serialized validators receive
|
||||||
and run context. Validators execute in resolved order and stop at the first
|
canonical chunk JSON or artifact codec bytes. Validators execute in resolved
|
||||||
error or rejection. An empty chain approves the result.
|
order and stop at the first error or rejection. An empty chain approves the
|
||||||
|
result.
|
||||||
|
|
||||||
`runWithRetry` applies the effective retry policy around module execution and
|
`runWithRetry` applies the effective retry policy around module execution and
|
||||||
its complete validation chain. A module or validator error becomes a framework
|
its complete validation chain. A module or validator error becomes a framework
|
||||||
@@ -143,13 +208,20 @@ The runner depends on recorder and loader interfaces, using no-op
|
|||||||
implementations when collaborators are absent. Each checkpointed workflow
|
implementations when collaborators are absent. Each checkpointed workflow
|
||||||
boundary records a running, succeeded, or failed transition. Reuse decisions
|
boundary records a running, succeeded, or failed transition. Reuse decisions
|
||||||
are consulted in workflow order and accepted payloads are cloned before
|
are consulted in workflow order and accepted payloads are cloned before
|
||||||
entering the normal handoff path. Dependency fingerprints connect later
|
entering the normal handoff path. Typed extract, merge, and normalize
|
||||||
checkpoints to the exact accepted results on which they depend.
|
checkpoints store codec bytes with artifact kind, schema ID and version, exact
|
||||||
|
schema digest, and media type. Reuse compares that identity with the prepared
|
||||||
|
codec and decodes through the codec; missing identity, mismatches, corrupt
|
||||||
|
bytes, and decode failures become explicit reuse misses and execute the step
|
||||||
|
normally. Dependency fingerprints and debug content digests use the same stable
|
||||||
|
codec bytes that cross those boundaries.
|
||||||
|
|
||||||
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
|
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
|
||||||
boundaries. Context scopes associate nested LLM calls with the module or
|
boundaries. Context scopes associate nested LLM calls with the module or
|
||||||
validator attempt that made them. Debug-write failures are framework errors;
|
validator attempt that made them. Debug-write failures are framework errors;
|
||||||
debug data is never used as a checkpoint source.
|
debug data is never used as a checkpoint source. Typed artifact debug envelopes
|
||||||
|
are domain-neutral, redact sensitive metadata and bytes through the common
|
||||||
|
debug policy, and record codec identity plus schema and content digests.
|
||||||
|
|
||||||
Checkpoint identity, physical layout, reuse behavior, and debug artifact
|
Checkpoint identity, physical layout, reuse behavior, and debug artifact
|
||||||
handling are operator contracts in [Operations](../operations.md). Serialization
|
handling are operator contracts in [Operations](../operations.md). Serialization
|
||||||
@@ -161,25 +233,42 @@ and recorder implementation are inventoried in
|
|||||||
The runner owns manifest assembly and handoff summaries but not the durable JSON
|
The runner owns manifest assembly and handoff summaries but not the durable JSON
|
||||||
schema. It records resolved module and lane provenance, validator chains,
|
schema. It records resolved module and lane provenance, validator chains,
|
||||||
source/reference identities, selected LLM profiles, normalized and rejected
|
source/reference identities, selected LLM profiles, normalized and rejected
|
||||||
summaries, status, and timing. Raw payload bytes remain outside the manifest.
|
summaries, status, and timing. Serialized artifact content remains outside the manifest.
|
||||||
Module metadata providers may add non-secret singleton or lane-scoped metadata.
|
Module metadata providers may add non-secret singleton or lane-scoped metadata.
|
||||||
|
|
||||||
Execution errors include stage, module, lane, or validator context. Once a
|
Execution errors include stage, module, lane, or validator context. Once a
|
||||||
manifest exists, a failing run returns it with failed status and completion
|
manifest exists, a failing run returns it with failed status and completion
|
||||||
time. Successful status reflects whether any raw result was rejected. The
|
time. Successful status reflects whether any result was rejected. The
|
||||||
durable manifest and logical file schemas are defined in the
|
durable manifest and logical file schemas are defined in the
|
||||||
[JSON output contract](../integrations/json-output.md).
|
[JSON output contract](../integrations/json-output.md).
|
||||||
|
|
||||||
|
On a framework failure, the runner cancels its derived context, stops submitting
|
||||||
|
new extract work, drains started tasks, and skips the output encoder. Parent
|
||||||
|
cancellation takes precedence. Otherwise context-cancellation fallout is
|
||||||
|
discarded when a substantive error exists, and the primary error is selected by
|
||||||
|
stage, resolved lane, and source chunk rather than completion time.
|
||||||
|
|
||||||
## Tests To Inspect
|
## Tests To Inspect
|
||||||
|
|
||||||
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
|
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
|
||||||
- `internal/framework/pipeline/profile_test.go`: selection, defaults,
|
- `internal/framework/pipeline/profile_test.go`: selection, defaults,
|
||||||
capabilities, validator chains, and digest behavior.
|
capabilities, validator chains, and digest behavior.
|
||||||
|
- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec
|
||||||
|
metadata, registration, erasure safety, strict decoding, and cloning.
|
||||||
|
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
|
||||||
|
lane resolution and preparation, target-specific validators,
|
||||||
|
incompatibilities, ordering, and schema-sensitive pipeline identity.
|
||||||
|
- `internal/framework/pipeline/runner_concurrency_test.go`: bounded dispatch and
|
||||||
|
continuations, reverse completion, stable errors, rejection, cancellation,
|
||||||
|
retries, and independent provider-call limits.
|
||||||
|
- `internal/framework/pipeline/preparation_test.go`: option validation,
|
||||||
|
construction order, dependency failures, and the before-source-work boundary.
|
||||||
- `internal/framework/pipeline/references_test.go`: target resolution and
|
- `internal/framework/pipeline/references_test.go`: target resolution and
|
||||||
materialization.
|
materialization.
|
||||||
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,
|
- `internal/cli/run_test.go`: production stage transitions, retries, rejections,
|
||||||
rejections, warnings, checkpoints, debug hooks, and manifests.
|
warnings, debug hooks, manifests, and end-to-end composition.
|
||||||
- `internal/framework/pipeline/walking_skeleton_test.go`: fake-backed complete
|
- `internal/modules/integration/*_test.go` and
|
||||||
workflow composition.
|
`internal/modules/seriatim/input/transcript/runner_test.go`: typed runner
|
||||||
|
composition across concrete module families.
|
||||||
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
|
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
|
||||||
collaborators.
|
collaborators.
|
||||||
|
|||||||
@@ -91,6 +91,20 @@ digests match the current invocation. Changes to input bytes, the resolved
|
|||||||
pipeline, selected lanes, the runtime LLM profile override, or bound reference
|
pipeline, selected lanes, the runtime LLM profile override, or bound reference
|
||||||
content invalidate reuse.
|
content invalidate reuse.
|
||||||
|
|
||||||
|
Typed artifact checkpoints additionally record codec-owned bytes, artifact
|
||||||
|
kind, schema ID and version, exact schema digest, and media type. A missing or
|
||||||
|
mismatched codec identity, or bytes the current codec cannot decode, is reported
|
||||||
|
as a checkpoint reuse miss. The affected operation executes normally and, when
|
||||||
|
checkpoint writing is enabled, replaces the incompatible checkpoint.
|
||||||
|
|
||||||
|
Current checkpoint manifests use workspace schema `notarius.workspace.v2`.
|
||||||
|
Manifests written with `notarius.workspace.v1` are incompatible because their
|
||||||
|
chunk provenance has an older shape. On the first explicit resume after an
|
||||||
|
upgrade, each affected checkpoint is treated as a reuse miss and its workflow
|
||||||
|
step executes normally. The compatibility check does not migrate or delete the
|
||||||
|
v1 files; when checkpoint writing is enabled, normal execution refreshes the
|
||||||
|
affected checkpoint files in the current schema.
|
||||||
|
|
||||||
Runs do not reuse checkpoints unless explicitly requested. Without reuse, the
|
Runs do not reuse checkpoints unless explicitly requested. Without reuse, the
|
||||||
workflow executes normally and refreshes checkpoint files when checkpointing is
|
workflow executes normally and refreshes checkpoint files when checkpointing is
|
||||||
enabled.
|
enabled.
|
||||||
@@ -118,7 +132,9 @@ the attempt `llm_calls` array. Prompt content is written inline in the prompt
|
|||||||
artifact. The response metadata and body use the paired files described above;
|
artifact. The response metadata and body use the paired files described above;
|
||||||
the body is pretty-printed JSON when possible and raw text otherwise. Debug
|
the body is pretty-printed JSON when possible and raw text otherwise. Debug
|
||||||
artifacts may contain source material, reference material, prompt inputs, model
|
artifacts may contain source material, reference material, prompt inputs, model
|
||||||
outputs, and other sensitive data. API keys are not written, and obvious
|
outputs, and other sensitive data. Typed artifact envelopes include
|
||||||
|
domain-neutral codec identity, redacted metadata and content, and digests of
|
||||||
|
the stable codec bytes. API keys are not written, and obvious
|
||||||
credential-shaped values and sensitive map keys are redacted, but debug
|
credential-shaped values and sensitive map keys are redacted, but debug
|
||||||
directories should still be protected as sensitive local state.
|
directories should still be protected as sensitive local state.
|
||||||
|
|
||||||
@@ -182,5 +198,11 @@ selected execution profile. Pipeline module retry settings are defined in
|
|||||||
[Configuration](config.md#module-bindings). There is no separate CLI retry
|
[Configuration](config.md#module-bindings). There is no separate CLI retry
|
||||||
command.
|
command.
|
||||||
|
|
||||||
|
Extract worker concurrency and actual provider-call concurrency are separate
|
||||||
|
limits. Their configuration, defaults, and validation are defined in
|
||||||
|
[Configuration](config.md#concurrency). Cancellation stops undispatched extract
|
||||||
|
work; already started work is allowed to finish or observe cancellation before
|
||||||
|
the run reports failure.
|
||||||
|
|
||||||
Notarius writes local files only. Remote storage and archive management are not
|
Notarius writes local files only. Remote storage and archive management are not
|
||||||
part of the implemented CLI.
|
part of the implemented CLI.
|
||||||
|
|||||||
@@ -66,12 +66,19 @@ Framework stages operate on source documents, source units, and source
|
|||||||
references rather than format-specific structures. A source reference identifies
|
references rather than format-specific structures. A source reference identifies
|
||||||
an ordered range of generic source units. Framework code preserves those ranges
|
an ordered range of generic source units. Framework code preserves those ranges
|
||||||
and does not merge or rewrite them unless a stage module explicitly owns that
|
and does not merge or rewrite them unless a stage module explicitly owns that
|
||||||
behavior.
|
behavior. Every source unit carries a validated self-reference to its containing
|
||||||
|
document and its own unit ID.
|
||||||
|
|
||||||
Extract modules own artifact semantics, prompt use, response schemas, and
|
Extract modules own artifact semantics, prompt use, response schemas, and
|
||||||
domain interpretation. Domain-specific concepts remain in the relevant module,
|
domain interpretation. Domain-specific concepts remain in the relevant module,
|
||||||
validator, shared domain helper, and artifact contract.
|
validator, shared domain helper, and artifact contract.
|
||||||
|
|
||||||
|
Typed artifact registrations declare one stable artifact kind and exact Go
|
||||||
|
type from extraction through merge, normalization, and semantic validation.
|
||||||
|
Pipeline resolution requires a compatible codec and matching kind-specific
|
||||||
|
variants before a typed lane can be accepted. Framework-owned erasure remains
|
||||||
|
private and must report type incompatibility as an error rather than a panic.
|
||||||
|
|
||||||
Auxiliary references provide context or disambiguation. They are not source
|
Auxiliary references provide context or disambiguation. They are not source
|
||||||
evidence and must not be converted into source references.
|
evidence and must not be converted into source references.
|
||||||
|
|
||||||
@@ -83,6 +90,12 @@ and verifies module availability and capabilities before execution. Structural
|
|||||||
pipeline choices must not be scattered through conditionals or hidden behind
|
pipeline choices must not be scattered through conditionals or hidden behind
|
||||||
ad hoc command flags.
|
ad hoc command flags.
|
||||||
|
|
||||||
|
Resolution validates every selected module and validator option set. A separate
|
||||||
|
preparation boundary then constructs the complete input, chunk, lane,
|
||||||
|
validation, and output implementation set in pipeline order. The runner accepts
|
||||||
|
only that prepared set, so construction and dependency failures occur before
|
||||||
|
source parsing or any other module operation.
|
||||||
|
|
||||||
Stage ownership is explicit:
|
Stage ownership is explicit:
|
||||||
|
|
||||||
- input modules convert external material into the generic source model;
|
- input modules convert external material into the generic source model;
|
||||||
@@ -96,13 +109,32 @@ The framework owns orchestration and handoff provenance. Modules return logical
|
|||||||
results and warnings; they do not own CLI reporting, workspace paths, durable
|
results and warnings; they do not own CLI reporting, workspace paths, durable
|
||||||
file placement, checkpoints, or diagnostics.
|
file placement, checkpoints, or diagnostics.
|
||||||
|
|
||||||
|
After pipeline-wide chunking, extraction uses bounded framework concurrency.
|
||||||
|
One run-wide worker pool receives chunk-scoped lane jobs in deterministic
|
||||||
|
chunk-first, lane-second order. A lane may begin its merge and normalize
|
||||||
|
continuation only after all of its extract jobs are terminal; that continuation
|
||||||
|
remains serial within the lane, while bounded continuations for different lanes
|
||||||
|
may overlap. The framework must not create unbounded goroutines per lane or
|
||||||
|
chunk.
|
||||||
|
|
||||||
|
Completion timing does not choose public ordering or errors. The coordinator
|
||||||
|
orders accepted artifacts, warnings, rejections, checkpoint events, and
|
||||||
|
framework errors by stable pipeline scope. Rejections do not cancel unrelated
|
||||||
|
work. A framework error cancels derived work, prevents undispatched work from
|
||||||
|
starting, waits for started work, and prevents output encoding.
|
||||||
|
|
||||||
## Validation
|
## Validation
|
||||||
|
|
||||||
Validation is a framework-managed boundary around raw outputs from chunk,
|
Validation is a framework-managed boundary around outputs from chunk, extract,
|
||||||
extract, merge, and normalize stages. Validators receive immutable stage output
|
merge, and normalize stages. Validators receive immutable stage output
|
||||||
and make an explicit whole-output decision: approve, approve with warnings, or
|
and make an explicit whole-output decision: approve, approve with warnings, or
|
||||||
reject.
|
reject.
|
||||||
|
|
||||||
|
Typed artifact validators receive the domain value directly. Chunk validators
|
||||||
|
receive source-zone chunks, while serialized validators receive immutable
|
||||||
|
representation bytes and declared schema metadata. A validator registered for
|
||||||
|
one target or artifact kind cannot satisfy an incompatible selection.
|
||||||
|
|
||||||
Rejection is a recorded pipeline outcome, not a framework execution error.
|
Rejection is a recorded pipeline outcome, not a framework execution error.
|
||||||
Validator execution failures are framework errors. Rejected output does not
|
Validator execution failures are framework errors. Rejected output does not
|
||||||
advance to the next stage.
|
advance to the next stage.
|
||||||
@@ -126,6 +158,11 @@ LLM calls and other external operations accept cancellation and respect
|
|||||||
timeouts. Concurrency control belongs in shared runtime plumbing rather than in
|
timeouts. Concurrency control belongs in shared runtime plumbing rather than in
|
||||||
individual modules.
|
individual modules.
|
||||||
|
|
||||||
|
The application-wide LLM scheduler bounds actual provider calls independently
|
||||||
|
of framework worker limits. Every LLM-backed module, retry, and validator uses
|
||||||
|
the single injected scheduled client, including work performed by overlapping
|
||||||
|
lanes.
|
||||||
|
|
||||||
## Configuration And Provenance
|
## Configuration And Provenance
|
||||||
|
|
||||||
Configuration loading, precedence, defaults, environment overrides, redaction,
|
Configuration loading, precedence, defaults, environment overrides, redaction,
|
||||||
|
|||||||
@@ -2,16 +2,16 @@
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
Decision-complete; implementation pending. This roadmap defines the desired end
|
Implemented on 2026-07-17. This roadmap records the design delivered for
|
||||||
state for [ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
|
[ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
|
||||||
[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and
|
[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and
|
||||||
[ADR-0004](../adr/0004-package-modules-by-domain.md). The work needed to reach
|
[ADR-0004](../adr/0004-package-modules-by-domain.md). Its implementation history
|
||||||
that state is owned by the
|
is summarized in the [completion record](implementation.md).
|
||||||
[domain pipeline implementation plan](implementation.md).
|
|
||||||
|
|
||||||
Until that plan is complete, current behavior remains defined by the
|
This file is historical design context, not a current-behavior reference.
|
||||||
architecture, configuration, integration, operations, and internal
|
Implemented contracts and mechanics are defined by the architecture,
|
||||||
documentation outside `docs/roadmap/`.
|
configuration, integration, operations, and internal documentation outside
|
||||||
|
`docs/roadmap/`.
|
||||||
|
|
||||||
## User Intent
|
## User Intent
|
||||||
|
|
||||||
|
|||||||
@@ -4,12 +4,6 @@ Current Notarius behavior is documented in the canonical README, CLI,
|
|||||||
configuration, operations, internal, and integration docs. This roadmap records
|
configuration, operations, internal, and integration docs. This roadmap records
|
||||||
future work only.
|
future work only.
|
||||||
|
|
||||||
## Focused Roadmaps
|
|
||||||
|
|
||||||
- [Domain-Typed Pipeline Implementation](domain.md): proposed migration to
|
|
||||||
domain-owned typed artifact lanes, domain-first packages, explicit
|
|
||||||
serialization boundaries, and bounded deterministic extract execution.
|
|
||||||
|
|
||||||
## Candidate Product Work
|
## Candidate Product Work
|
||||||
|
|
||||||
- Additional input adapters, such as Markdown or note-export formats.
|
- Additional input adapters, such as Markdown or note-export formats.
|
||||||
|
|||||||
@@ -1,836 +1,48 @@
|
|||||||
# Domain-Typed Pipeline Implementation Plan
|
# Domain-Typed Pipeline Completion Record
|
||||||
|
|
||||||
## Purpose
|
## Status
|
||||||
|
|
||||||
This document is the executable implementation plan for the target state in the
|
Implemented on 2026-07-17.
|
||||||
[domain-typed pipeline feature roadmap](domain.md). It assumes the decisions in
|
|
||||||
|
This file is a concise historical record. Current behavior is owned by
|
||||||
|
[Architecture](../policy/architecture.md),
|
||||||
|
[Configuration](../config.md), [Operations](../operations.md),
|
||||||
|
[Pipeline Internals](../internal/pipeline.md),
|
||||||
|
[Module Internals](../internal/modules.md),
|
||||||
|
[LLM Runtime Internals](../internal/llm.md), and the durable
|
||||||
|
[integration contracts](../integrations/). The accepted decisions remain in
|
||||||
[ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
|
[ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
|
||||||
[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and
|
[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and
|
||||||
[ADR-0004](../adr/0004-package-modules-by-domain.md).
|
[ADR-0004](../adr/0004-package-modules-by-domain.md).
|
||||||
|
|
||||||
The intended operator is an LLM coding agent working through one stage per
|
## Delivered
|
||||||
implementation prompt. Complete the stages in order. Each stage must leave the
|
|
||||||
repository buildable and tested; do not defer a broken intermediate state to a
|
- Production extensions use domain-first packages and package-family
|
||||||
later stage.
|
registrars without changing selectable module or validator keys.
|
||||||
|
- Source units and chunks carry canonical engine-owned provenance using
|
||||||
## Implementation Rules
|
`source.SourceRef`; workspace checkpoints use schema v2.
|
||||||
|
- Artifact lanes keep one domain-owned Go type through extraction, merge,
|
||||||
For every stage:
|
normalization, and typed validation.
|
||||||
|
- Artifact codecs own stable schema-aware checkpoint, debug, and output
|
||||||
1. Read `docs/development.md` and follow its task-specific reading guide. Read
|
serialization; generic representation validators consume serialized views.
|
||||||
the current implementation and focused tests for every touched subsystem.
|
- Resolution verifies artifact-kind, codec, typed variant, capability,
|
||||||
2. Treat the feature roadmap as the canonical owner of desired behavior and
|
reference, validator, and option compatibility before source work.
|
||||||
this document as the canonical owner of task sequencing. Do not restate
|
- Preparation constructs the full run-local implementation set and injects one
|
||||||
future behavior in current-behavior documentation before it exists.
|
shared scheduled LLM client.
|
||||||
3. Preserve unrelated user changes. Use mechanical moves where possible so file
|
- Extraction uses bounded chunk-first, lane-second dispatch. Lane continuations
|
||||||
history and test intent remain legible.
|
are also bounded, public results are deterministic, and framework failures
|
||||||
4. Add focused tests with the change. Run those tests while iterating, then run
|
cancel undispatched work without treating rejections as errors.
|
||||||
`go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` before ending
|
- Maintained D&D payloads, logical output paths, prompt and schema identities,
|
||||||
the stage.
|
default validators, warnings, rejections, and manifest provenance remain
|
||||||
5. Run `go test -race ./...` in stages that introduce or change concurrency and
|
covered by compatibility tests.
|
||||||
in the final stage.
|
|
||||||
6. Update the canonical current-behavior documents in the same stage in which
|
## Completion Evidence
|
||||||
behavior changes. At minimum, reconsider `docs/policy/architecture.md`,
|
|
||||||
`docs/internal/overview.md`, `docs/internal/pipeline.md`,
|
Focused tests cover durable compatibility, typed resolution and preparation,
|
||||||
`docs/internal/modules.md`, `docs/internal/llm.md`, `docs/config.md`,
|
codec boundaries, checkpoint invalidation, debug recording, bounded scheduling,
|
||||||
`docs/operations.md`, and `docs/integrations/` according to the documentation
|
reverse completion, cancellation, rejection, retries, and independent extract
|
||||||
policy; edit only the documents whose owned facts changed.
|
worker and provider-call limits. Maintained example configurations and input are
|
||||||
7. Do not change user-visible module keys, validator keys, output paths, durable
|
validated and exercised by the CLI test suite.
|
||||||
D&D JSON, prompt/schema identities, default chains, or rejection semantics
|
|
||||||
unless this plan explicitly requires it.
|
Repository validation is defined in [Development](../development.md). The Go
|
||||||
8. Stop after a stage if an exit criterion cannot be met. Record the concrete
|
race suite requires a CGO-capable toolchain in the execution environment.
|
||||||
blocker rather than implementing a second architecture alongside this one.
|
|
||||||
|
|
||||||
## Fixed Technical Decisions
|
|
||||||
|
|
||||||
The following choices are inputs to implementation, not questions to reopen in
|
|
||||||
individual stages.
|
|
||||||
|
|
||||||
### Source types
|
|
||||||
|
|
||||||
Keep the existing names `source.SourceDocument`, `source.SourceUnit`, and
|
|
||||||
`source.SourceRef`. Add `Ref source.SourceRef` to `SourceUnit`. Move
|
|
||||||
`contracts.SourceChunk` to `internal/core/source` as `source.Chunk`, with this
|
|
||||||
logical shape:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type Chunk struct {
|
|
||||||
ID string
|
|
||||||
SourceID string
|
|
||||||
Index int
|
|
||||||
Ref SourceRef
|
|
||||||
Content []byte
|
|
||||||
MediaType string
|
|
||||||
Units []SourceUnit
|
|
||||||
Metadata map[string]any
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Remove `StartUnitID` and `EndUnitID`; `Ref` is the only chunk-boundary
|
|
||||||
representation. A Seriatim unit's self-reference is
|
|
||||||
`{SourceID: document ID, StartUnitID: unit ID, EndUnitID: unit ID}`. A chunk
|
|
||||||
reference spans its first and last included units. Advance persisted workspace
|
|
||||||
state from `notarius.workspace.v1` to `notarius.workspace.v2`; v1 state is
|
|
||||||
incompatible and must be recomputed, but never deleted automatically.
|
|
||||||
|
|
||||||
### Artifact contracts
|
|
||||||
|
|
||||||
Place engine-owned artifact primitives with the other universal contracts under
|
|
||||||
`internal/framework/contracts`:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type ArtifactKind string
|
|
||||||
|
|
||||||
type ArtifactSchema struct {
|
|
||||||
ID string
|
|
||||||
Name string
|
|
||||||
Version string
|
|
||||||
JSONSchema []byte
|
|
||||||
}
|
|
||||||
|
|
||||||
type SerializedArtifact struct {
|
|
||||||
Kind ArtifactKind
|
|
||||||
Schema ArtifactSchema
|
|
||||||
MediaType string
|
|
||||||
Content []byte
|
|
||||||
Metadata map[string]any
|
|
||||||
}
|
|
||||||
|
|
||||||
type ArtifactCodec[T any] interface {
|
|
||||||
Kind() ArtifactKind
|
|
||||||
Schema() ArtifactSchema
|
|
||||||
MediaType() string
|
|
||||||
Encode(T) ([]byte, error)
|
|
||||||
Decode([]byte) (T, error)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Use strict JSON decoding for JSON codecs: reject unknown fields and trailing
|
|
||||||
tokens. Encoding must be deterministic for equal canonical values. Clone byte
|
|
||||||
slices and maps at framework ownership boundaries. Validate non-empty artifact
|
|
||||||
kind, schema ID, schema name, schema version, media type, and JSON Schema during
|
|
||||||
registration. Compute and retain a SHA-256 digest of the JSON Schema bytes.
|
|
||||||
|
|
||||||
Use generic `Extractor[T]`, `Merger[T]`, `Normalizer[T]`, and
|
|
||||||
`TypedValidator[T]` contracts. The exact request/result structs may retain
|
|
||||||
existing names where that reduces churn, but they must satisfy these rules:
|
|
||||||
|
|
||||||
- the extractor returns `T`, warnings, and framework-owned chunk provenance;
|
|
||||||
- merge receives accepted per-chunk typed values carrying lane ID, source ID,
|
|
||||||
chunk ID, chunk index, and chunk reference, already sorted by chunk index;
|
|
||||||
- normalize receives and returns `T`;
|
|
||||||
- typed validators receive `T` plus the relevant universal source, chunk,
|
|
||||||
reference, lane, stage, profile, and metadata context;
|
|
||||||
- the LLM client and decoded module options are held by constructed
|
|
||||||
implementations, not passed in operation requests; and
|
|
||||||
- no module-facing Zone-B request or result contains `RawPayload`, `any`, or
|
|
||||||
serialized JSON as its artifact value.
|
|
||||||
|
|
||||||
Retain separate framework wrappers around `T` for extract, merge, and normalize
|
|
||||||
provenance. Do not put lane IDs, module keys, or framework warnings into the D&D
|
|
||||||
domain value itself.
|
|
||||||
|
|
||||||
Support a second `SerializedValidator` contract for representation-level
|
|
||||||
generic validators. Its request contains immutable bytes, media type, and
|
|
||||||
optional schema metadata. For a Zone-B value, the framework produces that
|
|
||||||
request with the lane codec. Keep a separate non-generic `ChunkValidator`
|
|
||||||
contract for semantic validation of immutable `[]source.Chunk`; when a
|
|
||||||
representation validator is selected at chunk, the framework instead supplies
|
|
||||||
its canonical JSON chunk encoding. `valid_json` and `valid_json_schema` use the
|
|
||||||
serialized path, domain validators use `TypedValidator[T]`, and generic
|
|
||||||
approve/reject validators register explicit chunk and typed-artifact variants.
|
|
||||||
|
|
||||||
### Typed registry model
|
|
||||||
|
|
||||||
Because Go methods cannot introduce type parameters, expose free generic
|
|
||||||
registration functions in `internal/framework/pipeline`, backed by private
|
|
||||||
non-generic registry entries. Use `reflect.TypeFor[T]()` only inside registration
|
|
||||||
and framework assembly to prove exact type equality.
|
|
||||||
|
|
||||||
- Codec registry key: artifact kind. Exactly one codec may be registered per
|
|
||||||
kind.
|
|
||||||
- Extractor registry key: existing module key. Each entry declares one artifact
|
|
||||||
kind and exact Go type.
|
|
||||||
- Merger and normalizer registry key: `(existing module key, artifact kind)`.
|
|
||||||
- Typed validator registry key: `(existing validator key, artifact kind)`.
|
|
||||||
- Chunk-validator registry key: existing validator key in the distinct chunk
|
|
||||||
target namespace.
|
|
||||||
- Serialized validators retain their existing validator key and declare whether
|
|
||||||
they support chunk values, artifact values, or both.
|
|
||||||
- Duplicate keys/variants, missing codecs, Go-type mismatches, and incompatible
|
|
||||||
selected variants are errors.
|
|
||||||
|
|
||||||
The extractor selected for a lane establishes the lane artifact kind. During
|
|
||||||
resolution, look up merger, normalizer, and validators against that kind.
|
|
||||||
Record artifact kind, schema ID, schema version, and schema digest on the
|
|
||||||
resolved lane and in the resolved-pipeline digest. A resolved pipeline with an
|
|
||||||
incompatible lane must fail before preparation or source execution.
|
|
||||||
|
|
||||||
The private erased lane entry owns closures for construction and execution of
|
|
||||||
one concrete `T`. It may store a value as `any` internally, but it must verify
|
|
||||||
the exact registered `reflect.Type` at every erased boundary and return a
|
|
||||||
descriptive framework error rather than panic.
|
|
||||||
|
|
||||||
### Construction and preparation
|
|
||||||
|
|
||||||
Use one uniform construction context:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type ModuleDependencies struct {
|
|
||||||
LLM contracts.StructuredLLMClient
|
|
||||||
}
|
|
||||||
|
|
||||||
type BuildRequest struct {
|
|
||||||
Dependencies ModuleDependencies
|
|
||||||
Options map[string]any
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Each registry entry stores both an option-validation closure and a constructor.
|
|
||||||
Implementations own concrete option structs and one decoder used by both
|
|
||||||
closures. Resolution/configuration validation calls the decoder and discards
|
|
||||||
the value; preparation calls it once and supplies the decoded value to the
|
|
||||||
constructor. Reject unknown option fields. Empty options produce the
|
|
||||||
implementation's explicit defaults.
|
|
||||||
|
|
||||||
Add:
|
|
||||||
|
|
||||||
```go
|
|
||||||
func Prepare(
|
|
||||||
resolved ResolvedPipeline,
|
|
||||||
registries Registries,
|
|
||||||
deps ModuleDependencies,
|
|
||||||
) (*PreparedPipeline, error)
|
|
||||||
```
|
|
||||||
|
|
||||||
`PreparedPipeline` retains explicit resolved input, chunk, artifact-lane, and
|
|
||||||
output fields and all constructed validators. Construct in stable pipeline
|
|
||||||
order: input; chunk and its validators; each resolved lane in order with extract,
|
|
||||||
merge, normalize, and their validator chains in stage order; then output. On the
|
|
||||||
first failure, return an error identifying stage, lane if any, module or
|
|
||||||
validator key, and cause. No operation method may have run.
|
|
||||||
|
|
||||||
Create the one scheduled production LLM client first, inject it into
|
|
||||||
preparation, and then run only the prepared pipeline. Test-only deterministic
|
|
||||||
modules may accept a nil LLM dependency; any implementation that declares or
|
|
||||||
uses LLM-backed execution must reject a nil client at preparation. Remove LLM
|
|
||||||
clients and raw option maps from operation requests after every production
|
|
||||||
implementation has migrated.
|
|
||||||
|
|
||||||
Constructed modules and validators are reused for a run. Anything callable from
|
|
||||||
parallel extract workers must be concurrency-safe; production implementations
|
|
||||||
should be immutable after construction.
|
|
||||||
|
|
||||||
### Package registration
|
|
||||||
|
|
||||||
Each package-family registrar exposes:
|
|
||||||
|
|
||||||
```go
|
|
||||||
func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error
|
|
||||||
```
|
|
||||||
|
|
||||||
The generic and Seriatim registrars ignore the asset argument until they need
|
|
||||||
it. Registrars validate the registry pointers they use and return contextual
|
|
||||||
errors. The CLI creates one complete registry set and one asset registry, then
|
|
||||||
calls registrars in this order: generic, Seriatim, D&D. The D&D registrar owns
|
|
||||||
D&D codecs, implementations, typed generic specializations, prompt/schema
|
|
||||||
assets, and default validator chains.
|
|
||||||
|
|
||||||
`internal/modules/dnd` owns D&D shared types. Its `register` sibling may import
|
|
||||||
children and generic strategies; the root package must not import its children.
|
|
||||||
`internal/modules/generic` never imports D&D. `internal/modules/seriatim` does not
|
|
||||||
import D&D. Move domain-neutral embedded prompt-filesystem helpers to
|
|
||||||
`internal/framework/promptfs`.
|
|
||||||
|
|
||||||
### D&D typed model
|
|
||||||
|
|
||||||
Define `dnd.SpellList`, `dnd.SpellCast`, and evidence/source-reference fields at
|
|
||||||
the D&D package root. Use `source.SourceRef`; do not create another D&D unit-ref
|
|
||||||
type for artifact provenance. The stable artifact kind is
|
|
||||||
`dnd/spell-list`.
|
|
||||||
|
|
||||||
The spell extractor owns a private LLM DTO and the existing
|
|
||||||
`dnd_spells_llm.v1.json` response schema. It canonicalizes and maps that DTO to
|
|
||||||
`dnd.SpellList`. The codec package owns `dnd_spells.v1.json` and the durable
|
|
||||||
encoding. Keep those schemas separate. Remove the validator-only
|
|
||||||
`spellpayload` model after all three D&D validators consume `dnd.SpellList`.
|
|
||||||
|
|
||||||
Make `appendorder` a generic strategy that accepts a typed combine function at
|
|
||||||
registration/construction. The D&D registrar supplies a function that appends
|
|
||||||
spell casts in already-sorted source-chunk order. Make `noop` a generic typed
|
|
||||||
strategy. Neither generic package imports D&D.
|
|
||||||
|
|
||||||
### Serialized boundaries
|
|
||||||
|
|
||||||
After normalize, encode `T` once to `SerializedArtifact` for final output.
|
|
||||||
Output remains domain-neutral and receives serialized artifacts. Preserve the
|
|
||||||
existing output bundle and index contract.
|
|
||||||
|
|
||||||
Extract, merge, and normalize checkpoints encode and decode through the same
|
|
||||||
lane codec. Checkpoint identity includes artifact kind, schema ID, schema
|
|
||||||
version, and schema digest. A missing codec or any mismatch invalidates reuse
|
|
||||||
and recomputes the stage; it is not a fatal run error by itself. Codec decode
|
|
||||||
failure also invalidates that checkpoint and records the reason. Never pass
|
|
||||||
serialized checkpoint content directly to the next typed stage.
|
|
||||||
|
|
||||||
Debug recording uses the codec for typed artifact values and preserves existing
|
|
||||||
opt-in/sensitive-content rules. Artifact/checkpoint digests use the stable codec
|
|
||||||
bytes. Aggregate manifests, checkpoint indexes, warning slices, and rejection
|
|
||||||
slices have one coordinator writer.
|
|
||||||
|
|
||||||
### Concurrency
|
|
||||||
|
|
||||||
Version-2 configuration gains:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
concurrency:
|
|
||||||
total_llm: 4
|
|
||||||
stage_workers:
|
|
||||||
extract: 4
|
|
||||||
```
|
|
||||||
|
|
||||||
Represent effective stage worker limits as `map[string]int`. Initially accept
|
|
||||||
only `extract`; reject unknown keys. Missing extract defaults to `total_llm` and
|
|
||||||
its valid range is `1..total_llm`. Add the environment override
|
|
||||||
`NOTARIUS_STAGE_WORKERS_EXTRACT`. Preserve precedence rules and keep the file
|
|
||||||
configuration version at 2.
|
|
||||||
|
|
||||||
Use one run-wide fixed extract worker pool. Dispatch jobs in round-robin order
|
|
||||||
with source chunk as the outer loop and resolved lane as the inner loop. Use one
|
|
||||||
job channel whose capacity equals the effective extract worker count, so the
|
|
||||||
dispatcher applies bounded backpressure. Do not create one goroutine per job. A
|
|
||||||
job includes extract retries and extract-stage validators. Store results by
|
|
||||||
lane index and chunk index.
|
|
||||||
|
|
||||||
When all extract jobs for a lane are terminal, run that lane's merge and then
|
|
||||||
normalize serially. Lane continuations may overlap. All LLM calls at all stages,
|
|
||||||
including retries and validators, use the single injected scheduled client, so
|
|
||||||
`total_llm` remains the authoritative process-wide provider-call ceiling.
|
|
||||||
|
|
||||||
Rejections are terminal results and do not cancel other work. A framework error
|
|
||||||
cancels a derived run context, stops dispatching jobs not yet started, and waits
|
|
||||||
for started tasks to finish or observe cancellation. Choose the returned error
|
|
||||||
as follows:
|
|
||||||
|
|
||||||
1. if the parent context is canceled, return its error;
|
|
||||||
2. otherwise discard internal `context.Canceled`/`DeadlineExceeded` errors when
|
|
||||||
at least one non-context framework error exists; and
|
|
||||||
3. choose the earliest remaining error by stage order (`extract`, `merge`,
|
|
||||||
`normalize`), resolved lane index, chunk index for chunk-scoped work, and
|
|
||||||
configured validator/operation index.
|
|
||||||
|
|
||||||
Use a sentinel chunk index after all real chunks for lane-scoped merge and
|
|
||||||
normalize errors. Retain other started-task errors only in opt-in diagnostics.
|
|
||||||
Sort accepted artifacts, warnings, and rejections by resolved lane index, source
|
|
||||||
chunk index where applicable, stage order, validator order, and original
|
|
||||||
within-result order. Completion timing must not affect public output. Run output
|
|
||||||
only if every lane has a successful or rejection-only terminal outcome and no
|
|
||||||
framework error occurred.
|
|
||||||
|
|
||||||
## Staged Implementation
|
|
||||||
|
|
||||||
### Stage 1: Compatibility Baselines and ADR Acceptance
|
|
||||||
|
|
||||||
Goal: lock down behavior that subsequent internal migrations must preserve and
|
|
||||||
record the architectural decisions as accepted.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Add semantic or golden compatibility tests for the maintained Seriatim-to-D&D
|
|
||||||
path: durable output files and JSON, output index, manifest provenance,
|
|
||||||
warnings, rejections, and stable lane/chunk ordering.
|
|
||||||
- Snapshot production module keys, validator keys, default validator chains,
|
|
||||||
prompt/schema identities, and maintained example/profile resolution in tests.
|
|
||||||
- Strengthen runner tests for fixed topology, validator rejection as a nonfatal
|
|
||||||
outcome, framework-error abort, retries, parent cancellation, checkpoint reuse
|
|
||||||
and invalidation, diagnostics, and opt-in debug behavior.
|
|
||||||
- Add an instrumented scheduled-client test showing that all existing production
|
|
||||||
LLM callers share `concurrency.total_llm`. It need not demonstrate parallel
|
|
||||||
lanes yet.
|
|
||||||
- Review the three ADRs against this decision-complete plan, set their status to
|
|
||||||
`Accepted`, and update their dates only if ADR policy requires an acceptance
|
|
||||||
date. Do not rewrite accepted decision text after this stage.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- compatibility tests fail on an unintended durable-output, key, chain,
|
|
||||||
provenance, or outcome-semantics change; and
|
|
||||||
- ADR-0002, ADR-0003, and ADR-0004 are accepted.
|
|
||||||
|
|
||||||
### Stage 2: Registrar Composition Without Package Moves
|
|
||||||
|
|
||||||
Goal: replace CLI leaf-by-leaf registration with package-family composition
|
|
||||||
before changing imports.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Add `internal/modules/generic/register`, `internal/modules/seriatim/register`,
|
|
||||||
and `internal/modules/dnd/register` using the fixed registrar signature.
|
|
||||||
- Initially let those registrars import the existing stage-oriented packages.
|
|
||||||
Move ownership of production validators, default chains, and prompt assets out
|
|
||||||
of `internal/cli/catalog.go` and into the appropriate registrar.
|
|
||||||
- Have the CLI allocate complete registries and the asset registry once, invoke
|
|
||||||
generic, Seriatim, then D&D registration, and retain its existing test
|
|
||||||
injection paths.
|
|
||||||
- Test nil registry handling, duplicate registration errors, stable registered
|
|
||||||
keys, default chains, and asset identities.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- the CLI composition root names only the three registrar packages, framework
|
|
||||||
registry types, and asset registry; and
|
|
||||||
- no production key, chain, prompt, schema, or runtime behavior changes.
|
|
||||||
|
|
||||||
### Stage 3: Mechanical Generic and Seriatim Package Moves
|
|
||||||
|
|
||||||
Goal: establish the domain-first generic and source-format trees without
|
|
||||||
changing contracts.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Move the Seriatim adapter and tests to
|
|
||||||
`internal/modules/seriatim/input/transcript`.
|
|
||||||
- Move the generic chunker to `internal/modules/generic/chunk/units`, retaining
|
|
||||||
the configured key `generic`.
|
|
||||||
- Move append-order merge, no-op normalize, JSON output, and all generic
|
|
||||||
validators to their target paths under `internal/modules/generic`.
|
|
||||||
- Update only registrar imports and affected black-box tests. Preserve package
|
|
||||||
behavior and all public registry keys.
|
|
||||||
- Remove the emptied old directories.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- generic and Seriatim production implementations exist only under their target
|
|
||||||
trees; and
|
|
||||||
- compatibility baselines remain green.
|
|
||||||
|
|
||||||
### Stage 4: Mechanical D&D Package Move and Import Guard
|
|
||||||
|
|
||||||
Goal: establish the D&D tree and enforce ADR-0004 dependency direction while
|
|
||||||
legacy contracts are still intact.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Move the D&D scenes chunker, spell extractor, validators, schemas, prompt
|
|
||||||
assets, and D&D shared helpers into the target D&D tree. Do not create the
|
|
||||||
typed root model or codec yet.
|
|
||||||
- Move domain-neutral prompt filesystem helpers from
|
|
||||||
`internal/modules/sharedassets` to `internal/framework/promptfs`.
|
|
||||||
- Move tests with their owning implementation. Relocate tests that intentionally
|
|
||||||
compose domains to a black-box integration-test package rather than creating
|
|
||||||
peer-domain production imports.
|
|
||||||
- Add a Go-parser-based import-boundary test. It must reject concrete
|
|
||||||
D&D-to-Seriatim and Seriatim-to-D&D imports, all generic-to-D&D imports, and
|
|
||||||
root-domain imports of child implementations. Allow domain registrars, the CLI
|
|
||||||
composition root, and designated external integration tests to compose
|
|
||||||
packages.
|
|
||||||
- Remove old empty stage-oriented and validator directories.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- all production extensions use the target domain-first package layout except
|
|
||||||
the not-yet-created typed codec/model pieces;
|
|
||||||
- the import guard detects a deliberate fixture violation; and
|
|
||||||
- behavior and keys remain unchanged.
|
|
||||||
|
|
||||||
### Stage 5: Source-Unit Provenance
|
|
||||||
|
|
||||||
Goal: add canonical provenance to engine-owned source units without yet changing
|
|
||||||
the chunk type.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Add `Ref source.SourceRef` to `source.SourceUnit`, including clone and debug
|
|
||||||
representations.
|
|
||||||
- Make the Seriatim adapter assign the fixed self-reference for every unit.
|
|
||||||
- Extend `source.ValidateDocument` to require the unit reference's source ID to
|
|
||||||
match the document, require start and end IDs to equal the unit ID, reject
|
|
||||||
missing/invalid/reversed references, and preserve the existing unit-order and
|
|
||||||
uniqueness checks.
|
|
||||||
- Make source digests and source checkpoint serialization include the new
|
|
||||||
reference deterministically.
|
|
||||||
- Add focused tests for valid refs and missing, foreign, non-self, and reversed
|
|
||||||
refs, plus source checkpoint/debug round trips.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- every produced source unit has a validated self-reference; and
|
|
||||||
- source state preserves it through clone, debug, digest, and checkpoint paths.
|
|
||||||
|
|
||||||
### Stage 6: Engine-Owned Chunks and Workspace v2
|
|
||||||
|
|
||||||
Goal: finish the Zone-A source model and make its persisted compatibility break
|
|
||||||
explicit.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Add `source.Chunk` with the fixed shape and update universal chunk contracts,
|
|
||||||
modules, validators, runner code, checkpoints, debug envelopes, and tests to
|
|
||||||
use it.
|
|
||||||
- Derive `Chunk.Ref` from the first and last included unit references. Validate
|
|
||||||
source identity, non-empty ordered units, contiguous boundary agreement, and
|
|
||||||
exact correspondence between the chunk ref and first/last unit refs.
|
|
||||||
- Remove `contracts.SourceChunk`, `StartUnitID`, and `EndUnitID` after all
|
|
||||||
consumers migrate. Do not keep aliases.
|
|
||||||
- Advance `workspace.WorkspaceSchemaVersion` to `notarius.workspace.v2`. Make
|
|
||||||
loader behavior explicitly classify v1 as incompatible and recompute while
|
|
||||||
leaving files untouched.
|
|
||||||
- Update checkpoint identity/digest tests and operations documentation for the
|
|
||||||
one-time v1 resume miss.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- no production code imports a framework-owned chunk type;
|
|
||||||
- chunk provenance round-trips exactly; and
|
|
||||||
- v1 workspaces are safely ignored while v2 workspaces reuse successfully.
|
|
||||||
|
|
||||||
### Stage 7: Artifact and Codec Foundation
|
|
||||||
|
|
||||||
Goal: add the typed primitives and prove strict serialization independently of
|
|
||||||
production lanes.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Add the fixed artifact types, codec interface, schema digest helper, and clone
|
|
||||||
helpers under `internal/framework/contracts`.
|
|
||||||
- Add `ArtifactCodecRegistry` to `pipeline.Registries` and `ModuleCatalog`.
|
|
||||||
- Implement generic codec registration and private erasure/type tracking.
|
|
||||||
- Validate registration metadata and duplicates. Ensure erased encode/decode
|
|
||||||
returns typed errors, never reflection panics.
|
|
||||||
- Use two small test artifact types to cover registration, exact type identity,
|
|
||||||
deterministic encoding, strict decoding, cloning, duplicate kind rejection,
|
|
||||||
and schema metadata/digest behavior.
|
|
||||||
- Wire the new empty registry through CLI/test registry constructors without
|
|
||||||
changing production lane execution.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- codecs for heterogeneous test types can coexist and safely round-trip through
|
|
||||||
erased framework storage; and
|
|
||||||
- current production behavior remains on the legacy raw path and unchanged.
|
|
||||||
|
|
||||||
### Stage 8: Typed Contracts, Variants, and Resolution
|
|
||||||
|
|
||||||
Goal: resolve a complete type-compatible lane before executing it.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Add the typed stage, typed validator, chunk validator, serialized validator,
|
|
||||||
and provenance wrapper contracts from the fixed decisions.
|
|
||||||
- Extend extractor specs with artifact kind/type. Convert merger, normalizer,
|
|
||||||
and typed validator registries to artifact-kind variants while retaining
|
|
||||||
serialized-validator registration by key.
|
|
||||||
- Implement free generic registration helpers and private erased entries.
|
|
||||||
- Extend lane resolution to derive kind from extractor, require its codec,
|
|
||||||
select exact merger/normalizer/validator variants, and include artifact/schema
|
|
||||||
identity in resolved lanes and pipeline digest.
|
|
||||||
- Keep legacy registration helpers only as explicitly named transitional APIs;
|
|
||||||
do not let a raw registration satisfy a typed lane.
|
|
||||||
- Add composition tests with two artifact types and heterogeneous lanes. Cover
|
|
||||||
missing codec, missing variant, Go-type mismatch, duplicate variant, wrong
|
|
||||||
validator kind, stable resolution order, and digest changes on schema identity
|
|
||||||
or schema digest changes.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- heterogeneous typed test lanes resolve without module-facing erasure;
|
|
||||||
- every incompatible selection fails before execution; and
|
|
||||||
- existing raw production lanes continue to resolve only through their visible
|
|
||||||
transitional path.
|
|
||||||
|
|
||||||
### Stage 9: Preparation and Construction Foundation
|
|
||||||
|
|
||||||
Goal: construct and validate an entire resolved pipeline before source work.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Add `ModuleDependencies`, `BuildRequest`, `PreparedPipeline`, and `Prepare` as
|
|
||||||
specified.
|
|
||||||
- Extend registry entries/specs with option validation and construction
|
|
||||||
functions. Supply adapters for legacy zero-argument constructors during the
|
|
||||||
migration.
|
|
||||||
- Call option validation for every selected module and validator during
|
|
||||||
resolution/config validation. Reject unknown fields and contextualize errors.
|
|
||||||
- Have preparation construct all selected components in fixed order and retain
|
|
||||||
immutable prepared lane executors.
|
|
||||||
- Update the runner API so `Run` receives a prepared pipeline. At the CLI, create
|
|
||||||
the shared scheduled LLM client, prepare, and only then invoke the runner.
|
|
||||||
- Prove with fakes that malformed options, missing required LLM dependencies,
|
|
||||||
and late-component construction failures occur before the input adapter's
|
|
||||||
`Parse` method.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- all components are constructed before source work;
|
|
||||||
- preparation errors identify exact scope and perform no operations; and
|
|
||||||
- legacy production modules still run through temporary construction adapters.
|
|
||||||
|
|
||||||
### Stage 10: Migrate Universal Modules to Construction
|
|
||||||
|
|
||||||
Goal: remove legacy option/dependency handling from input, chunk, and output.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Give Seriatim input, generic units chunking, D&D scenes chunking, generic JSON
|
|
||||||
output, and all applicable chunk validators implementation-owned option
|
|
||||||
structs and strict decoders.
|
|
||||||
- Build those implementations with decoded options and injected dependencies.
|
|
||||||
Require the LLM client for D&D scenes; keep deterministic implementations
|
|
||||||
independent of it.
|
|
||||||
- Remove `Options` and `LLMClient` from the corresponding operation requests.
|
|
||||||
Retain per-run source, reference, profile, session, and metadata fields.
|
|
||||||
- Update registrars and focused tests. Verify options are decoded once during
|
|
||||||
preparation and operation methods do not inspect raw maps.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- no universal production module parses raw options during execution; and
|
|
||||||
- every universal LLM call uses the injected shared client.
|
|
||||||
|
|
||||||
### Stage 11: Canonical D&D Model, Codec, and Typed Extractor
|
|
||||||
|
|
||||||
Goal: establish the first production `T` and its extraction boundary.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Add canonical spell types at the D&D root using engine-owned source refs.
|
|
||||||
- Add `internal/modules/dnd/codec/spells`, register kind `dnd/spell-list`, and
|
|
||||||
make it own the existing durable `dnd_spells.v1.json` schema and strict stable
|
|
||||||
encoding.
|
|
||||||
- Keep the private spell-extraction LLM DTO and
|
|
||||||
`dnd_spells_llm.v1.json` in the extractor package. Map canonicalized DTO values
|
|
||||||
to `dnd.SpellList` and do not expose the DTO to validators or the codec.
|
|
||||||
- Convert the extractor to `Extractor[dnd.SpellList]`, construction-time options
|
|
||||||
and dependency injection. Preserve prompt assets, retries, warnings, evidence,
|
|
||||||
and LLM response validation.
|
|
||||||
- Register the codec and typed extractor from the D&D registrar.
|
|
||||||
- Add codec compatibility tests against existing durable fixtures and tests
|
|
||||||
proving LLM schema ownership is separate from durable schema ownership.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- the typed extractor returns the canonical domain model;
|
|
||||||
- codec output is semantically identical to the maintained durable spell JSON;
|
|
||||||
and
|
|
||||||
- no downstream production consumer is switched until the next stages.
|
|
||||||
|
|
||||||
### Stage 12: Typed Validators and Generic Typed Strategies
|
|
||||||
|
|
||||||
Goal: complete all typed components required by the D&D lane.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Convert D&D spell shape, source-reference, and source-relatedness validators
|
|
||||||
to `TypedValidator[dnd.SpellList]` with construction-time options/dependencies.
|
|
||||||
- Remove JSON reparsing from those validators. Remove the duplicate
|
|
||||||
`spellpayload` package after its final consumer migrates.
|
|
||||||
- Convert `valid_json` and `valid_json_schema` to serialized validators. Register
|
|
||||||
them so the framework uses the D&D codec when they occur in the spell chain.
|
|
||||||
- Implement generic typed append-order merge and no-op normalize. In the D&D
|
|
||||||
registrar, register D&D variants using a spell-list append function and
|
|
||||||
`noop[dnd.SpellList]`.
|
|
||||||
- Convert always-accept/reject into explicit chunk and typed variants and
|
|
||||||
register the D&D variants without changing their keys. Verify serialized
|
|
||||||
validators operate on the framework encoding at chunk and the codec encoding
|
|
||||||
at artifact stages.
|
|
||||||
- Test typed validator requests, source refs, relatedness LLM injection, generic
|
|
||||||
strategy reuse with a second test type, default chain order, and rejection
|
|
||||||
behavior.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- every selected spell-lane component has a compatible D&D typed variant;
|
|
||||||
- generic packages import no D&D code; and
|
|
||||||
- the duplicate validator payload model and inter-validator JSON parsing are
|
|
||||||
gone.
|
|
||||||
|
|
||||||
### Stage 13: Typed D&D Runner Vertical Slice
|
|
||||||
|
|
||||||
Goal: execute one complete production lane as `dnd.SpellList` while retaining
|
|
||||||
temporary raw output/checkpoint adapters.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Implement the erased typed lane executor and runner path for extract, stage
|
|
||||||
retries, typed/serialized validation, merge, and normalize.
|
|
||||||
- Keep accepted extract values indexed by chunk and pass them to merge in source
|
|
||||||
order. Preserve warnings and rejections in stable scope order.
|
|
||||||
- Add narrow transitional adapters from typed stage outputs to the existing raw
|
|
||||||
checkpoint/debug/output envelopes. These adapters must use the registered
|
|
||||||
codec and be named/commented as migration-only.
|
|
||||||
- Route the production D&D lane through the typed path; leave no production raw
|
|
||||||
D&D stage module registered in parallel.
|
|
||||||
- Add end-to-end and checkpoint-disabled tests proving the maintained D&D output
|
|
||||||
and outcome semantics are unchanged. Add incompatible-lane tests proving
|
|
||||||
failure occurs before input.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- D&D values remain typed from extractor through normalize and typed validation;
|
|
||||||
- the runner's only D&D erasure is its private lane adapter and explicit codec
|
|
||||||
boundary; and
|
|
||||||
- durable output remains unchanged through the transitional adapter.
|
|
||||||
|
|
||||||
### Stage 14: Typed Checkpoints, Debugging, and Output
|
|
||||||
|
|
||||||
Goal: move every serialization side effect and the final Zone-C boundary to the
|
|
||||||
codec model.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Change runner/output contracts so final normalized results are
|
|
||||||
`SerializedArtifact` values. Update generic JSON output without importing D&D.
|
|
||||||
- Preserve logical file names, index fields, media types, manifest contents, and
|
|
||||||
durable spell JSON.
|
|
||||||
- Change extract, merge, and normalize checkpoints to store codec bytes plus
|
|
||||||
artifact kind, schema ID, version, and schema digest. Decode reused values
|
|
||||||
back to `T` before the next stage.
|
|
||||||
- Implement safe invalidation for missing/mismatched codecs and decode failure,
|
|
||||||
with explicit checkpoint-event reasons.
|
|
||||||
- Change typed debug envelopes to serialize through the codec, preserving opt-in
|
|
||||||
and redaction behavior. Use stable codec bytes for artifact digests.
|
|
||||||
- Remove the transitional raw output/checkpoint/debug adapters introduced in
|
|
||||||
Stage 13 after all tests use the typed boundaries.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- typed values round-trip at every stage checkpoint;
|
|
||||||
- incompatible checkpoint artifacts recompute safely;
|
|
||||||
- output and debug code are domain-neutral; and
|
|
||||||
- no D&D typed lane depends on a raw-boundary adapter.
|
|
||||||
|
|
||||||
### Stage 15: Finish Construction Migration and Remove Raw Contracts
|
|
||||||
|
|
||||||
Goal: leave one production extension system rather than parallel raw and typed
|
|
||||||
models.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Migrate any remaining merge, normalize, extract, and validator implementations
|
|
||||||
to construction-time option decoding and dependency injection.
|
|
||||||
- Remove LLM clients and raw option maps from all remaining operation requests.
|
|
||||||
- Remove legacy raw extractor/merger/normalizer/validator contracts,
|
|
||||||
constructors, registry entries, `RawPayload`, `ResponseSchema` if superseded,
|
|
||||||
raw clone helpers, raw checkpoint envelopes, and migration-only adapters.
|
|
||||||
- Remove dead duplicate models and compatibility helpers. Search for production
|
|
||||||
references to old stage-oriented paths and raw Zone-B types.
|
|
||||||
- Keep serialized artifacts only at codec, checkpoint/debug, and output
|
|
||||||
boundaries.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- no production Zone-B handoff uses JSON bytes, `RawPayload`, or `any`;
|
|
||||||
- all production configuration options are validated before execution;
|
|
||||||
- all production LLM users receive the one injected client; and
|
|
||||||
- there is no legacy production registration path.
|
|
||||||
|
|
||||||
### Stage 16: Stage-Worker Configuration
|
|
||||||
|
|
||||||
Goal: add the decided scheduling control without changing runner execution yet.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Add `StageWorkers map[string]int` to effective concurrency configuration and
|
|
||||||
`stage_workers` to the version-2 YAML shape. Deep-clone the map.
|
|
||||||
- Accept only `extract`, reject unknown or empty keys, default missing extract to
|
|
||||||
effective `total_llm`, and validate the inclusive range
|
|
||||||
`1..total_llm` after file/environment precedence resolves.
|
|
||||||
- Add `NOTARIUS_STAGE_WORKERS_EXTRACT` with the existing environment precedence
|
|
||||||
and integer error style.
|
|
||||||
- Preserve redaction/effective-config diagnostics and version 2.
|
|
||||||
- Update `docs/config.md` and maintained examples that intentionally demonstrate
|
|
||||||
concurrency. Do not add the field to every example when the default conveys
|
|
||||||
the intended behavior.
|
|
||||||
- Add file, default, merge/precedence, environment, unknown-key, boundary, and
|
|
||||||
redacted-effective-config tests.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- every run has a validated effective extract worker count;
|
|
||||||
- omitted configuration preserves current effective behavior at the default
|
|
||||||
`total_llm: 1`; and
|
|
||||||
- current configuration documentation owns the implemented contract.
|
|
||||||
|
|
||||||
### Stage 17: Concurrent Lane and Extract Scheduling
|
|
||||||
|
|
||||||
Goal: implement bounded concurrent lanes while preserving deterministic public
|
|
||||||
behavior and the global LLM invariant.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Add the fixed-size run-wide worker pool and central round-robin dispatcher.
|
|
||||||
Bound queued work so dispatch applies backpressure; do not enqueue the whole
|
|
||||||
run into unbounded memory.
|
|
||||||
- Treat extract, retries, and extract validators as one job. Publish immutable
|
|
||||||
results to a coordinator indexed by lane and chunk.
|
|
||||||
- Start each lane's serial merge/normalize continuation only after all its
|
|
||||||
extract jobs are terminal. Permit different lane continuations to overlap.
|
|
||||||
- Make the coordinator the sole writer of aggregate output, manifest,
|
|
||||||
checkpoint-event collection, warnings, and rejections. Use attempt-specific
|
|
||||||
debug paths and synchronize any recorder state that remains shared.
|
|
||||||
- Implement rejection, cancellation, stable sorting, deterministic primary
|
|
||||||
error selection, and output gating exactly as specified under Concurrency.
|
|
||||||
- Audit every concurrently reused extractor, validator, codec, LLM/debug wrapper,
|
|
||||||
checkpoint loader/recorder, and manifest metadata provider. Make production
|
|
||||||
implementations immutable or add narrowly scoped synchronization.
|
|
||||||
- Add deterministic barrier-controlled tests that force reverse completion
|
|
||||||
order, simultaneous failures, parent cancellation, rejections mixed with
|
|
||||||
successes, lane continuation overlap, and undispatched-job cancellation.
|
|
||||||
- Add instrumented integration tests issuing calls from multiple lanes, retries,
|
|
||||||
and LLM-backed validators. Assert provider calls never exceed
|
|
||||||
`total_llm`, extract jobs never exceed the effective extract worker count, and
|
|
||||||
both limits are exercised independently.
|
|
||||||
- Run `go test -race ./...` and eliminate races rather than weakening tests.
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- lanes and extract jobs actually overlap when configured above one;
|
|
||||||
- job and provider-call limits are independently enforced;
|
|
||||||
- public results and primary errors are identical across forced completion
|
|
||||||
orders; and
|
|
||||||
- full race testing passes.
|
|
||||||
|
|
||||||
### Stage 18: Documentation, Cleanup, and Final Verification
|
|
||||||
|
|
||||||
Goal: make the implemented repository and its canonical documentation agree,
|
|
||||||
then close the focused roadmap.
|
|
||||||
|
|
||||||
Tasks:
|
|
||||||
|
|
||||||
- Review every current-behavior document routed by `docs/development.md` and
|
|
||||||
update only its owned facts: architecture and dependency direction, package
|
|
||||||
inventory, pipeline resolution/preparation/execution, typed module contracts,
|
|
||||||
LLM scheduling, configuration, checkpoint compatibility, diagnostics,
|
|
||||||
operations, and durable integration contracts.
|
|
||||||
- Verify maintained examples and copyable files against the implementation.
|
|
||||||
- Remove stale old package paths, raw-contract terminology, and superseded
|
|
||||||
future-work entries. Validate all changed documentation links.
|
|
||||||
- Update ADR consequences only in ways permitted for accepted ADR metadata; do
|
|
||||||
not edit accepted decision text. Record a new superseding ADR if final code
|
|
||||||
required an architectural change.
|
|
||||||
- Mark the feature roadmap implemented and reduce this implementation plan to a
|
|
||||||
concise completion record, or move it to the repository's established
|
|
||||||
completed-roadmap location if one exists. Do not let this file become a second
|
|
||||||
current-behavior reference.
|
|
||||||
- Run final checks:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./...
|
|
||||||
go test -race ./...
|
|
||||||
go vet ./...
|
|
||||||
go build ./cmd/notarius
|
|
||||||
```
|
|
||||||
|
|
||||||
Exit criteria:
|
|
||||||
|
|
||||||
- all feature-roadmap completion criteria are met;
|
|
||||||
- no stale production paths or legacy typed/raw bridge remain;
|
|
||||||
- current documentation and examples describe only implemented behavior; and
|
|
||||||
- all final validation commands pass.
|
|
||||||
|
|
||||||
## Open Questions
|
|
||||||
|
|
||||||
None. The feature-policy decisions and the implementation choices required to
|
|
||||||
begin each stage are resolved above. If implementation evidence contradicts one
|
|
||||||
of the accepted architectural decisions, stop and handle that as an ADR change
|
|
||||||
or supersession rather than treating it as an implicit implementation choice.
|
|
||||||
|
|||||||
@@ -1,6 +1,8 @@
|
|||||||
version: 2
|
version: 2
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 1
|
total_llm: 1
|
||||||
|
stage_workers:
|
||||||
|
extract: 1
|
||||||
workspace:
|
workspace:
|
||||||
directory: /var/lib/notarius
|
directory: /var/lib/notarius
|
||||||
diagnostics:
|
diagnostics:
|
||||||
|
|||||||
@@ -9,26 +9,21 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/dnd/scenes"
|
dndregister "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/register"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
|
genericregister "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/register"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
|
seriatimregister "gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/register"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
|
|
||||||
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
|
|
||||||
spellshape "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/shape"
|
|
||||||
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_refs"
|
|
||||||
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_relatedness"
|
|
||||||
alwaysaccept "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_accept"
|
|
||||||
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_reject"
|
|
||||||
validjson "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json"
|
|
||||||
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json_schema"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func productionRegistries() (pipeline.Registries, error) {
|
type productionComponents struct {
|
||||||
|
registries pipeline.Registries
|
||||||
|
assets *llm.AssetRegistry
|
||||||
|
}
|
||||||
|
|
||||||
|
func newProductionComponents() (productionComponents, error) {
|
||||||
registries := pipeline.Registries{
|
registries := pipeline.Registries{
|
||||||
Inputs: pipeline.NewInputAdapterRegistry(),
|
Inputs: pipeline.NewInputAdapterRegistry(),
|
||||||
Chunkers: pipeline.NewChunkerRegistry(),
|
Chunkers: pipeline.NewChunkerRegistry(),
|
||||||
|
ArtifactCodecs: pipeline.NewArtifactCodecRegistry(),
|
||||||
Extractors: pipeline.NewExtractorRegistry(),
|
Extractors: pipeline.NewExtractorRegistry(),
|
||||||
Mergers: pipeline.NewMergerRegistry(),
|
Mergers: pipeline.NewMergerRegistry(),
|
||||||
Normalizers: pipeline.NewNormalizerRegistry(),
|
Normalizers: pipeline.NewNormalizerRegistry(),
|
||||||
@@ -36,72 +31,26 @@ func productionRegistries() (pipeline.Registries, error) {
|
|||||||
ValidatorChains: pipeline.NewValidatorChainRegistry(),
|
ValidatorChains: pipeline.NewValidatorChainRegistry(),
|
||||||
Outputs: pipeline.NewOutputEncoderRegistry(),
|
Outputs: pipeline.NewOutputEncoderRegistry(),
|
||||||
}
|
}
|
||||||
if err := seriatim.Register(registries.Inputs); err != nil {
|
assets := llm.NewAssetRegistry()
|
||||||
return pipeline.Registries{}, fmt.Errorf("register seriatim input: %w", err)
|
registrars := []struct {
|
||||||
}
|
|
||||||
if err := generic.Register(registries.Chunkers); err != nil {
|
|
||||||
return pipeline.Registries{}, fmt.Errorf("register generic chunker: %w", err)
|
|
||||||
}
|
|
||||||
if err := scenes.Register(registries.Chunkers); err != nil {
|
|
||||||
return pipeline.Registries{}, fmt.Errorf("register dnd scenes chunker: %w", err)
|
|
||||||
}
|
|
||||||
if err := spells.Register(registries.Extractors); err != nil {
|
|
||||||
return pipeline.Registries{}, fmt.Errorf("register dnd spells extractor: %w", err)
|
|
||||||
}
|
|
||||||
if err := appendorder.Register(registries.Mergers); err != nil {
|
|
||||||
return pipeline.Registries{}, fmt.Errorf("register appendorder merger: %w", err)
|
|
||||||
}
|
|
||||||
if err := noop.Register(registries.Normalizers); err != nil {
|
|
||||||
return pipeline.Registries{}, fmt.Errorf("register noop normalizer: %w", err)
|
|
||||||
}
|
|
||||||
if err := registerProductionValidators(registries.Validators); err != nil {
|
|
||||||
return pipeline.Registries{}, err
|
|
||||||
}
|
|
||||||
if err := registerProductionValidatorChains(registries.ValidatorChains); err != nil {
|
|
||||||
return pipeline.Registries{}, err
|
|
||||||
}
|
|
||||||
if err := jsonoutput.Register(registries.Outputs); err != nil {
|
|
||||||
return pipeline.Registries{}, fmt.Errorf("register json output encoder: %w", err)
|
|
||||||
}
|
|
||||||
return registries, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func registerProductionValidators(registry *pipeline.ValidatorRegistry) error {
|
|
||||||
registrations := []struct {
|
|
||||||
name string
|
name string
|
||||||
register func(*pipeline.ValidatorRegistry) error
|
register func(pipeline.Registries, *llm.AssetRegistry) error
|
||||||
}{
|
}{
|
||||||
{name: "generic always accept validator", register: alwaysaccept.Register},
|
{name: "generic", register: genericregister.Register},
|
||||||
{name: "generic always reject validator", register: alwaysreject.Register},
|
{name: "seriatim", register: seriatimregister.Register},
|
||||||
{name: "generic valid json validator", register: validjson.Register},
|
{name: "dnd", register: dndregister.Register},
|
||||||
{name: "generic valid json schema validator", register: validjsonschema.Register},
|
|
||||||
{name: "dnd spell shape validator", register: spellshape.Register},
|
|
||||||
{name: "dnd spell source references validator", register: spellsourcerefs.Register},
|
|
||||||
{name: "dnd spell source relatedness validator", register: spellrelatedness.Register},
|
|
||||||
}
|
}
|
||||||
for _, registration := range registrations {
|
for _, registrar := range registrars {
|
||||||
if err := registration.register(registry); err != nil {
|
if err := registrar.register(registries, assets); err != nil {
|
||||||
return fmt.Errorf("register %s: %w", registration.name, err)
|
return productionComponents{}, fmt.Errorf("register %s module family: %w", registrar.name, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return nil
|
return productionComponents{registries: registries, assets: assets}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func registerProductionValidatorChains(registry *pipeline.ValidatorChainRegistry) error {
|
func productionRegistries() (pipeline.Registries, error) {
|
||||||
if err := registry.Register(pipeline.ValidatorChainMapping{
|
components, err := newProductionComponents()
|
||||||
Stage: pipeline.StageExtract,
|
return components.registries, err
|
||||||
Module: spells.Key,
|
|
||||||
Validators: []pipeline.ModuleBinding{
|
|
||||||
pipeline.Binding(validjson.Key),
|
|
||||||
pipeline.Binding(validjsonschema.Key),
|
|
||||||
pipeline.Binding(spellshape.Key),
|
|
||||||
pipeline.Binding(spellsourcerefs.Key),
|
|
||||||
pipeline.Binding(spellrelatedness.Key),
|
|
||||||
},
|
|
||||||
}); err != nil {
|
|
||||||
return fmt.Errorf("register dnd spells validator chain: %w", err)
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func productionCatalog() (pipeline.ModuleCatalog, error) {
|
func productionCatalog() (pipeline.ModuleCatalog, error) {
|
||||||
@@ -113,14 +62,8 @@ func productionCatalog() (pipeline.ModuleCatalog, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func productionPromptAssets() (*llm.AssetRegistry, error) {
|
func productionPromptAssets() (*llm.AssetRegistry, error) {
|
||||||
registry := llm.NewAssetRegistry()
|
components, err := newProductionComponents()
|
||||||
if err := scenes.RegisterPromptAssets(registry); err != nil {
|
return components.assets, err
|
||||||
return nil, fmt.Errorf("register dnd scenes prompt assets: %w", err)
|
|
||||||
}
|
|
||||||
if err := spells.RegisterPromptAssets(registry); err != nil {
|
|
||||||
return nil, fmt.Errorf("register dnd spells prompt assets: %w", err)
|
|
||||||
}
|
|
||||||
return registry, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func effectiveCatalog(opts Options) (pipeline.ModuleCatalog, error) {
|
func effectiveCatalog(opts Options) (pipeline.ModuleCatalog, error) {
|
||||||
@@ -147,6 +90,7 @@ func catalogFromRegistries(registries pipeline.Registries) pipeline.ModuleCatalo
|
|||||||
return pipeline.ModuleCatalog{
|
return pipeline.ModuleCatalog{
|
||||||
Inputs: registries.Inputs,
|
Inputs: registries.Inputs,
|
||||||
Chunkers: registries.Chunkers,
|
Chunkers: registries.Chunkers,
|
||||||
|
ArtifactCodecs: registries.ArtifactCodecs,
|
||||||
Extractors: registries.Extractors,
|
Extractors: registries.Extractors,
|
||||||
Mergers: registries.Mergers,
|
Mergers: registries.Mergers,
|
||||||
Normalizers: registries.Normalizers,
|
Normalizers: registries.Normalizers,
|
||||||
@@ -160,6 +104,7 @@ func registriesFromCatalog(catalog pipeline.ModuleCatalog) pipeline.Registries {
|
|||||||
return pipeline.Registries{
|
return pipeline.Registries{
|
||||||
Inputs: catalog.Inputs,
|
Inputs: catalog.Inputs,
|
||||||
Chunkers: catalog.Chunkers,
|
Chunkers: catalog.Chunkers,
|
||||||
|
ArtifactCodecs: catalog.ArtifactCodecs,
|
||||||
Extractors: catalog.Extractors,
|
Extractors: catalog.Extractors,
|
||||||
Mergers: catalog.Mergers,
|
Mergers: catalog.Mergers,
|
||||||
Normalizers: catalog.Normalizers,
|
Normalizers: catalog.Normalizers,
|
||||||
@@ -172,6 +117,7 @@ func registriesFromCatalog(catalog pipeline.ModuleCatalog) pipeline.Registries {
|
|||||||
func isEmptyCatalog(catalog pipeline.ModuleCatalog) bool {
|
func isEmptyCatalog(catalog pipeline.ModuleCatalog) bool {
|
||||||
return catalog.Inputs == nil &&
|
return catalog.Inputs == nil &&
|
||||||
catalog.Chunkers == nil &&
|
catalog.Chunkers == nil &&
|
||||||
|
catalog.ArtifactCodecs == nil &&
|
||||||
catalog.Extractors == nil &&
|
catalog.Extractors == nil &&
|
||||||
catalog.Mergers == nil &&
|
catalog.Mergers == nil &&
|
||||||
catalog.Normalizers == nil &&
|
catalog.Normalizers == nil &&
|
||||||
@@ -183,6 +129,7 @@ func isEmptyCatalog(catalog pipeline.ModuleCatalog) bool {
|
|||||||
func isEmptyRegistries(registries pipeline.Registries) bool {
|
func isEmptyRegistries(registries pipeline.Registries) bool {
|
||||||
return registries.Inputs == nil &&
|
return registries.Inputs == nil &&
|
||||||
registries.Chunkers == nil &&
|
registries.Chunkers == nil &&
|
||||||
|
registries.ArtifactCodecs == nil &&
|
||||||
registries.Extractors == nil &&
|
registries.Extractors == nil &&
|
||||||
registries.Mergers == nil &&
|
registries.Mergers == nil &&
|
||||||
registries.Normalizers == nil &&
|
registries.Normalizers == nil &&
|
||||||
@@ -199,6 +146,22 @@ func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileI
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, err
|
return nil, nil, err
|
||||||
}
|
}
|
||||||
|
return buildProductionLLMClient(ctx, cfg, profileID, assets)
|
||||||
|
}
|
||||||
|
|
||||||
|
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
|
||||||
|
return func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
|
return buildProductionLLMClient(ctx, cfg, profileID, assets)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
if assets == nil {
|
||||||
|
return nil, nil, fmt.Errorf("production asset registry must not be nil")
|
||||||
|
}
|
||||||
recorder := llm.NewLLMProfileRecorder()
|
recorder := llm.NewLLMProfileRecorder()
|
||||||
client, err := llm.NewScriptoriumClient(llm.ScriptoriumClientConfig{
|
client, err := llm.NewScriptoriumClient(llm.ScriptoriumClientConfig{
|
||||||
ProfileDir: cfg.Scriptorium.ProfileDir,
|
ProfileDir: cfg.Scriptorium.ProfileDir,
|
||||||
|
|||||||
600
internal/cli/compatibility_test.go
Normal file
600
internal/cli/compatibility_test.go
Normal file
@@ -0,0 +1,600 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"io/fs"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
||||||
|
spellshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/shape"
|
||||||
|
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_refs"
|
||||||
|
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_relatedness"
|
||||||
|
validjson "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/valid_json"
|
||||||
|
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/valid_json_schema"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestProductionCompatibilitySnapshot(t *testing.T) {
|
||||||
|
normalizedOptions, err := normalizeOptions(Options{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("normalizeOptions() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if normalizedOptions.Catalog.Inputs != normalizedOptions.Registries.Inputs ||
|
||||||
|
normalizedOptions.Catalog.Chunkers != normalizedOptions.Registries.Chunkers ||
|
||||||
|
normalizedOptions.Catalog.Extractors != normalizedOptions.Registries.Extractors ||
|
||||||
|
normalizedOptions.Catalog.Mergers != normalizedOptions.Registries.Mergers ||
|
||||||
|
normalizedOptions.Catalog.Normalizers != normalizedOptions.Registries.Normalizers ||
|
||||||
|
normalizedOptions.Catalog.Validators != normalizedOptions.Registries.Validators ||
|
||||||
|
normalizedOptions.Catalog.ValidatorChains != normalizedOptions.Registries.ValidatorChains ||
|
||||||
|
normalizedOptions.Catalog.Outputs != normalizedOptions.Registries.Outputs {
|
||||||
|
t.Fatal("production catalog and execution registries do not share one composition")
|
||||||
|
}
|
||||||
|
if normalizedOptions.LLMClientFactory == nil {
|
||||||
|
t.Fatal("production LLM client factory is nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
registries, err := productionRegistries()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("productionRegistries() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
keySnapshots := []struct {
|
||||||
|
name string
|
||||||
|
got []string
|
||||||
|
want []string
|
||||||
|
}{
|
||||||
|
{name: "inputs", got: registries.Inputs.RegisteredKeys(), want: []string{"seriatim"}},
|
||||||
|
{name: "chunkers", got: registries.Chunkers.RegisteredKeys(), want: []string{"dnd/scenes", "generic"}},
|
||||||
|
{name: "extractors", got: registries.Extractors.RegisteredKeys(), want: []string{"dnd/spells"}},
|
||||||
|
{name: "mergers", got: registries.Mergers.RegisteredKeys(), want: []string{"appendorder"}},
|
||||||
|
{name: "normalizers", got: registries.Normalizers.RegisteredKeys(), want: []string{"noop"}},
|
||||||
|
{name: "validators", got: registries.Validators.RegisteredKeys(), want: []string{
|
||||||
|
"extract/dnd/spells/shape", "extract/dnd/spells/source_refs", "extract/dnd/spells/source_relatedness",
|
||||||
|
"generic/always_accept", "generic/always_reject", "generic/valid_json", "generic/valid_json_schema",
|
||||||
|
}},
|
||||||
|
{name: "outputs", got: registries.Outputs.RegisteredKeys(), want: []string{"json"}},
|
||||||
|
}
|
||||||
|
for _, snapshot := range keySnapshots {
|
||||||
|
t.Run(snapshot.name, func(t *testing.T) {
|
||||||
|
if !reflect.DeepEqual(snapshot.got, snapshot.want) {
|
||||||
|
t.Fatalf("registered keys = %#v, want compatibility snapshot %#v", snapshot.got, snapshot.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
wantChain := []pipeline.ModuleBinding{
|
||||||
|
pipeline.Binding(validjson.Key),
|
||||||
|
pipeline.Binding(validjsonschema.Key),
|
||||||
|
pipeline.Binding(spellshape.Key),
|
||||||
|
pipeline.Binding(spellsourcerefs.Key),
|
||||||
|
pipeline.Binding(spellrelatedness.Key),
|
||||||
|
}
|
||||||
|
if got := registries.ValidatorChains.Validators(pipeline.StageExtract, spells.Key); !reflect.DeepEqual(got, wantChain) {
|
||||||
|
t.Fatalf("spell validator chain = %#v, want compatibility snapshot %#v", got, wantChain)
|
||||||
|
}
|
||||||
|
|
||||||
|
assets, err := productionPromptAssets()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("productionPromptAssets() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
assertAssetNames(t, assets.PromptFS, []string{
|
||||||
|
"dnd.scenes/dnd.scenes.yaml",
|
||||||
|
"dnd.scenes/instructions.md",
|
||||||
|
"dnd.scenes/sharedassets/common-dnd-references.md",
|
||||||
|
"dnd.scenes/sharedassets/common-dnd-system.md",
|
||||||
|
"dnd.scenes/sharedassets/common-dnd-transcript.md",
|
||||||
|
"dnd.scenes/task.md",
|
||||||
|
"dnd.spells/dnd.spells.yaml",
|
||||||
|
"dnd.spells/instructions.md",
|
||||||
|
"dnd.spells/sharedassets/common-dnd-references.md",
|
||||||
|
"dnd.spells/sharedassets/common-dnd-system.md",
|
||||||
|
"dnd.spells/sharedassets/common-dnd-transcript.md",
|
||||||
|
"dnd.spells/task.md",
|
||||||
|
})
|
||||||
|
assertAssetNames(t, assets.SchemaFS, []string{
|
||||||
|
"dnd_scenes.v1.json",
|
||||||
|
"dnd_spells_llm.v1.json",
|
||||||
|
})
|
||||||
|
|
||||||
|
identitySnapshot := map[string]map[string]any{
|
||||||
|
"scenes": sceneManifestMetadata(t),
|
||||||
|
"spells": spellManifestMetadata(t),
|
||||||
|
}
|
||||||
|
for name, metadata := range identitySnapshot {
|
||||||
|
for _, key := range []string{"prompt_id", "prompt_version", "prompt_sha256", "response_schema_key", "response_schema_id", "response_schema_name", "response_schema_version", "response_schema_sha256"} {
|
||||||
|
if value, ok := metadata[key].(string); !ok || value == "" {
|
||||||
|
t.Fatalf("%s metadata[%q] = %#v, want non-empty identity", name, key, metadata[key])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if got := []any{
|
||||||
|
identitySnapshot["scenes"]["prompt_id"], identitySnapshot["scenes"]["prompt_version"],
|
||||||
|
identitySnapshot["scenes"]["response_schema_key"], identitySnapshot["scenes"]["response_schema_id"], identitySnapshot["scenes"]["response_schema_name"], identitySnapshot["scenes"]["response_schema_version"],
|
||||||
|
}; !reflect.DeepEqual(got, []any{"dnd.scenes", "v1", "dnd_scenes", "notarius.dnd.scenes", "notarius_dnd_scenes_v1", "v1"}) {
|
||||||
|
t.Fatalf("scene identities = %#v, want compatibility snapshot", got)
|
||||||
|
}
|
||||||
|
if got := []any{
|
||||||
|
identitySnapshot["spells"]["prompt_id"], identitySnapshot["spells"]["prompt_version"],
|
||||||
|
identitySnapshot["spells"]["response_schema_key"], identitySnapshot["spells"]["response_schema_id"], identitySnapshot["spells"]["response_schema_name"], identitySnapshot["spells"]["response_schema_version"],
|
||||||
|
}; !reflect.DeepEqual(got, []any{"dnd.spells", "v1", "dnd_spells", "notarius.dnd.spells", "notarius_dnd_spells_v1", "v1"}) {
|
||||||
|
t.Fatalf("spell identities = %#v, want compatibility snapshot", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
fileConfig, err := config.LoadFileConfig(fixturePath(t, "examples/dnd-spells.config.yml"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LoadFileConfig() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
cfg := config.Default()
|
||||||
|
if err := cfg.ApplyFileConfig(fileConfig); err != nil {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalogFromRegistries(registries)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
resolved := effective.ResolvedPipeline
|
||||||
|
if resolved.Input.Module != "seriatim" || resolved.Chunk.Module != "generic" || resolved.Output.Module != "json" || len(resolved.ArtifactLanes) != 1 {
|
||||||
|
t.Fatalf("resolved example = %#v, want maintained production topology", resolved)
|
||||||
|
}
|
||||||
|
lane := resolved.ArtifactLanes[0]
|
||||||
|
if lane.ID != "spells" || lane.Extract.Module != "dnd/spells" || lane.Merge.Module != "appendorder" || lane.Normalize.Module != "noop" {
|
||||||
|
t.Fatalf("resolved lane = %#v, want maintained spell lane", lane)
|
||||||
|
}
|
||||||
|
if got := resolvedValidatorKeys(resolved.ValidatorChains, pipeline.StageExtract, "spells", spells.Key); !reflect.DeepEqual(got, []string{
|
||||||
|
"generic/valid_json", "generic/valid_json_schema", "extract/dnd/spells/shape", "extract/dnd/spells/source_refs", "extract/dnd/spells/source_relatedness",
|
||||||
|
}) {
|
||||||
|
t.Fatalf("resolved validator keys = %#v, want compatibility snapshot", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
productionFileConfig, err := config.LoadFileConfig(fixturePath(t, "examples/dnd-spells-production.config.yml"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LoadFileConfig(production example) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
productionConfig := config.Default()
|
||||||
|
if err := productionConfig.ApplyFileConfig(productionFileConfig); err != nil {
|
||||||
|
t.Fatalf("ApplyFileConfig(production example) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
productionEffective, err := productionConfig.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalogFromRegistries(registries)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve(production example) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
productionResolved := productionEffective.ResolvedPipeline
|
||||||
|
if productionConfig.Concurrency.TotalLLM != 1 || productionConfig.Concurrency.StageWorkers["extract"] != 1 || !reflect.DeepEqual(productionResolved.Chunk.Options, map[string]any{"max_units": 50}) {
|
||||||
|
t.Fatalf("production example concurrency/options = %#v/%#v, want compatibility snapshot", productionConfig.Concurrency, productionResolved.Chunk.Options)
|
||||||
|
}
|
||||||
|
bindings := productionResolved.ArtifactLanes[0].ExtractReferences.Bindings
|
||||||
|
if len(bindings) != 2 || bindings[0].SlotName != "glossary" || bindings[0].Source != "./dnd-spells-glossary.txt" || bindings[1].SlotName != "party" || bindings[1].Source != "./dnd-spells-roster.txt" {
|
||||||
|
t.Fatalf("production example reference bindings = %#v, want maintained glossary and party bindings", bindings)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMaintainedSeriatimToDNDCompatibilityBundle(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
client contracts.StructuredLLMClient
|
||||||
|
wantStatus string
|
||||||
|
wantLaneFile bool
|
||||||
|
wantRejectedCount int
|
||||||
|
}{
|
||||||
|
{name: "approved", client: newFakeRunLLMClient(false), wantStatus: "approved", wantLaneFile: true},
|
||||||
|
{name: "validator rejection is nonfatal", client: newFakeRunLLMClient(true), wantStatus: "rejected", wantRejectedCount: 1},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
outputDir := t.TempDir()
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := RunWithOptions([]string{
|
||||||
|
"run", "dnd-session",
|
||||||
|
"--config", fixturePath(t, "examples/dnd-spells.config.yml"),
|
||||||
|
"--input", fixturePath(t, "examples/seriatim-minimal-transcript.json"),
|
||||||
|
"--output-dir", outputDir,
|
||||||
|
"--diagnostics-dir", t.TempDir(),
|
||||||
|
}, &stdout, &stderr, Options{LLMClientFactory: fakeLLMFactory(test.client, nil)})
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("RunWithOptions() code = %d, stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
runDir := onlyChildDir(t, outputDir)
|
||||||
|
wantFiles := []string{"index.json", "manifest.json", "rejected.json", "warnings.json"}
|
||||||
|
if test.wantLaneFile {
|
||||||
|
wantFiles = append(wantFiles, "lanes/spells.json")
|
||||||
|
}
|
||||||
|
sort.Strings(wantFiles)
|
||||||
|
if got := relativeFileNames(t, runDir); !reflect.DeepEqual(got, wantFiles) {
|
||||||
|
t.Fatalf("durable files = %#v, want compatibility snapshot %#v", got, wantFiles)
|
||||||
|
}
|
||||||
|
|
||||||
|
var manifest artifacts.RunManifest
|
||||||
|
readJSONFile(t, filepath.Join(runDir, "manifest.json"), &manifest)
|
||||||
|
if manifest.PipelineID != "dnd-session" || manifest.InputModule != "seriatim" || manifest.Chunker != "generic" || manifest.OutputEncoder != "json" {
|
||||||
|
t.Fatalf("manifest module provenance = %#v, want maintained production modules", manifest)
|
||||||
|
}
|
||||||
|
if len(manifest.ArtifactLanes) != 1 || manifest.ArtifactLanes[0].ID != "spells" {
|
||||||
|
t.Fatalf("artifact lanes = %#v, want one spells lane", manifest.ArtifactLanes)
|
||||||
|
}
|
||||||
|
laneManifest := manifest.ArtifactLanes[0]
|
||||||
|
if laneManifest.Extractor != "dnd/spells" || laneManifest.Merger != "appendorder" || laneManifest.Normalizer != "noop" {
|
||||||
|
t.Fatalf("manifest lane module provenance = %#v, want maintained production modules", laneManifest)
|
||||||
|
}
|
||||||
|
if len(manifest.Extractors) != 0 || manifest.Merger != "" || manifest.Normalizer != "" {
|
||||||
|
t.Fatalf("legacy top-level lane summaries = %#v/%q/%q, want empty compatibility snapshot", manifest.Extractors, manifest.Merger, manifest.Normalizer)
|
||||||
|
}
|
||||||
|
if manifest.ValidationStatus != test.wantStatus || len(manifest.RejectedOutputs) != test.wantRejectedCount {
|
||||||
|
t.Fatalf("manifest outcome = status %q rejected %#v, want %q/%d", manifest.ValidationStatus, manifest.RejectedOutputs, test.wantStatus, test.wantRejectedCount)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(manifest.SourceDigests, []string{"sha256:1c98d94ae632fb10a2b56f684cd4fb1019cedb1a629e57dc0977cf4a54135be0"}) {
|
||||||
|
t.Fatalf("source digests = %#v, want maintained fixture provenance", manifest.SourceDigests)
|
||||||
|
}
|
||||||
|
if got := manifestValidatorKeys(manifestValidatorChain(t, manifest, pipeline.StageExtract, "spells", spells.Key)); !reflect.DeepEqual(got, []string{
|
||||||
|
"generic/valid_json", "generic/valid_json_schema", "extract/dnd/spells/shape", "extract/dnd/spells/source_refs", "extract/dnd/spells/source_relatedness",
|
||||||
|
}) {
|
||||||
|
t.Fatalf("manifest validator chain = %#v, want compatibility snapshot", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
var index struct {
|
||||||
|
ManifestFile string `json:"manifest_file"`
|
||||||
|
OutputFiles []struct {
|
||||||
|
LaneID string `json:"lane_id"`
|
||||||
|
MediaType string `json:"media_type"`
|
||||||
|
File string `json:"file"`
|
||||||
|
ModuleKey string `json:"module_key"`
|
||||||
|
SchemaID string `json:"schema_id"`
|
||||||
|
SchemaName string `json:"schema_name"`
|
||||||
|
SchemaVersion string `json:"schema_version"`
|
||||||
|
} `json:"output_files"`
|
||||||
|
RejectedFile string `json:"rejected_file"`
|
||||||
|
WarningsFile string `json:"warnings_file"`
|
||||||
|
}
|
||||||
|
readJSONFile(t, filepath.Join(runDir, "index.json"), &index)
|
||||||
|
if index.ManifestFile != "manifest.json" || index.RejectedFile != "rejected.json" || index.WarningsFile != "warnings.json" {
|
||||||
|
t.Fatalf("output index fixed files = %#v, want compatibility snapshot", index)
|
||||||
|
}
|
||||||
|
if test.wantLaneFile {
|
||||||
|
if len(index.OutputFiles) != 1 {
|
||||||
|
t.Fatalf("output index entries = %#v, want one", index.OutputFiles)
|
||||||
|
}
|
||||||
|
wantOutput := struct {
|
||||||
|
LaneID, MediaType, File, ModuleKey, SchemaID, SchemaName, SchemaVersion string
|
||||||
|
}{"spells", "application/json", "lanes/spells.json", "noop", "notarius.dnd.spells", "notarius_dnd_spells_v1", "v1"}
|
||||||
|
gotOutput := index.OutputFiles[0]
|
||||||
|
got := struct {
|
||||||
|
LaneID, MediaType, File, ModuleKey, SchemaID, SchemaName, SchemaVersion string
|
||||||
|
}{gotOutput.LaneID, gotOutput.MediaType, gotOutput.File, gotOutput.ModuleKey, gotOutput.SchemaID, gotOutput.SchemaName, gotOutput.SchemaVersion}
|
||||||
|
if got != wantOutput {
|
||||||
|
t.Fatalf("output index entries = %#v, want compatibility snapshot %#v", index.OutputFiles, wantOutput)
|
||||||
|
}
|
||||||
|
assertJSONEqual(t, readFile(t, filepath.Join(runDir, "lanes/spells.json")), []byte(`{
|
||||||
|
"spell_casts": [{
|
||||||
|
"caster": "Aria",
|
||||||
|
"spell": "Cure Wounds",
|
||||||
|
"effect": "Heals a wounded ally.",
|
||||||
|
"narrative_description": "Aria casts Cure Wounds.",
|
||||||
|
"source_refs": [{"source_id": "session-alpha", "start_unit_id": 1, "end_unit_id": 1}]
|
||||||
|
}]
|
||||||
|
}`))
|
||||||
|
} else if len(index.OutputFiles) != 0 {
|
||||||
|
t.Fatalf("output index entries = %#v, want none for rejected lane", index.OutputFiles)
|
||||||
|
}
|
||||||
|
|
||||||
|
var warnings struct {
|
||||||
|
Warnings []contracts.Warning `json:"warnings"`
|
||||||
|
}
|
||||||
|
readJSONFile(t, filepath.Join(runDir, "warnings.json"), &warnings)
|
||||||
|
if len(warnings.Warnings) != 0 {
|
||||||
|
t.Fatalf("warnings = %#v, want empty compatibility snapshot", warnings.Warnings)
|
||||||
|
}
|
||||||
|
var rejected struct {
|
||||||
|
Rejected []contracts.RejectedOutput `json:"rejected"`
|
||||||
|
}
|
||||||
|
readJSONFile(t, filepath.Join(runDir, "rejected.json"), &rejected)
|
||||||
|
if len(rejected.Rejected) != test.wantRejectedCount {
|
||||||
|
t.Fatalf("rejected outputs = %#v, want %d", rejected.Rejected, test.wantRejectedCount)
|
||||||
|
}
|
||||||
|
if test.wantRejectedCount == 1 {
|
||||||
|
got := rejected.Rejected[0]
|
||||||
|
if got.Stage != "extract" || got.LaneID != "spells" || got.ModuleKey != "dnd/spells" || got.ChunkID != "chunk-000001" || got.ChunkIndex != 0 || got.ValidatorName != "extract/dnd/spells/source_refs" || got.ReasonCode != "invalid_source_refs" || got.AttemptCount != 1 {
|
||||||
|
t.Fatalf("rejection = %#v, want maintained nonfatal validator outcome", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProductionLLMCallersShareScheduledClient(t *testing.T) {
|
||||||
|
underlying := newBlockingProductionLLMClient()
|
||||||
|
scheduler, err := frameworkllm.NewScheduler(1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewScheduler() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
client := frameworkllm.NewScheduledClient(underlying, scheduler)
|
||||||
|
doc := &source.SourceDocument{
|
||||||
|
ID: "session-alpha", Kind: "transcript", Format: "application/json", Digest: "sha256:source",
|
||||||
|
Units: []source.SourceUnit{
|
||||||
|
{ID: 1, Kind: "segment", Text: "Aria casts Cure Wounds.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 1}},
|
||||||
|
{ID: 2, Kind: "segment", Text: "The spell takes effect.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
chunk := source.Chunk{
|
||||||
|
ID: "session-alpha:chunk:0", SourceID: doc.ID, Index: 0, Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 2},
|
||||||
|
Content: []byte(`{"scene":"Aria casts Cure Wounds."}`), MediaType: "application/json", Units: append([]source.SourceUnit(nil), doc.Units...),
|
||||||
|
}
|
||||||
|
|
||||||
|
var started sync.WaitGroup
|
||||||
|
started.Add(2)
|
||||||
|
errs := make(chan error, 2)
|
||||||
|
go func() {
|
||||||
|
started.Done()
|
||||||
|
chunker, err := scenes.New(client, scenes.Options{})
|
||||||
|
if err == nil {
|
||||||
|
_, err = chunker.Chunk(context.Background(), contracts.ChunkRequest{Source: doc})
|
||||||
|
}
|
||||||
|
errs <- err
|
||||||
|
}()
|
||||||
|
go func() {
|
||||||
|
started.Done()
|
||||||
|
extractor, err := spells.New(client, spells.Options{})
|
||||||
|
if err == nil {
|
||||||
|
_, err = extractor.Extract(context.Background(), contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk})
|
||||||
|
}
|
||||||
|
errs <- err
|
||||||
|
}()
|
||||||
|
started.Wait()
|
||||||
|
|
||||||
|
for i := 0; i < 2; i++ {
|
||||||
|
<-underlying.entered
|
||||||
|
underlying.release <- struct{}{}
|
||||||
|
}
|
||||||
|
for i := 0; i < 2; i++ {
|
||||||
|
if err := <-errs; err != nil {
|
||||||
|
t.Fatalf("production LLM caller error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if underlying.maxActive != 1 {
|
||||||
|
t.Fatalf("maximum concurrent provider calls = %d, want total_llm limit 1", underlying.maxActive)
|
||||||
|
}
|
||||||
|
sort.Strings(underlying.stageNames)
|
||||||
|
if !reflect.DeepEqual(underlying.stageNames, []string{"dnd/scenes", "dnd/spells"}) {
|
||||||
|
t.Fatalf("scheduled stage names = %#v, want both production LLM callers", underlying.stageNames)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProductionBundlePreservesLaneAndChunkOrder(t *testing.T) {
|
||||||
|
configPath := writeTestConfig(t, `version: 2
|
||||||
|
pipelines:
|
||||||
|
dnd-session:
|
||||||
|
input: seriatim
|
||||||
|
chunk:
|
||||||
|
module: generic
|
||||||
|
options:
|
||||||
|
max_units: 1
|
||||||
|
artifacts:
|
||||||
|
zeta:
|
||||||
|
extract: dnd/spells
|
||||||
|
alpha:
|
||||||
|
extract: dnd/spells
|
||||||
|
`)
|
||||||
|
outputDir := t.TempDir()
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := RunWithOptions([]string{
|
||||||
|
"run", "dnd-session",
|
||||||
|
"--config", configPath,
|
||||||
|
"--input", fixturePath(t, "examples/seriatim-minimal-transcript.json"),
|
||||||
|
"--output-dir", outputDir,
|
||||||
|
"--diagnostics-dir", t.TempDir(),
|
||||||
|
}, &stdout, &stderr, Options{LLMClientFactory: fakeLLMFactory(orderingProductionLLMClient{}, nil)})
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("RunWithOptions() code = %d, stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
runDir := onlyChildDir(t, outputDir)
|
||||||
|
var index struct {
|
||||||
|
OutputFiles []struct {
|
||||||
|
LaneID string `json:"lane_id"`
|
||||||
|
} `json:"output_files"`
|
||||||
|
}
|
||||||
|
readJSONFile(t, filepath.Join(runDir, "index.json"), &index)
|
||||||
|
if len(index.OutputFiles) != 2 {
|
||||||
|
t.Fatalf("output index entries = %#v, want two lanes", index.OutputFiles)
|
||||||
|
}
|
||||||
|
if got := []string{index.OutputFiles[0].LaneID, index.OutputFiles[1].LaneID}; !reflect.DeepEqual(got, []string{"alpha", "zeta"}) {
|
||||||
|
t.Fatalf("output lane order = %#v, want resolved lane order", got)
|
||||||
|
}
|
||||||
|
for _, laneID := range []string{"alpha", "zeta"} {
|
||||||
|
var payload struct {
|
||||||
|
SpellCasts []struct {
|
||||||
|
Spell string `json:"spell"`
|
||||||
|
SourceRefs []source.SourceRef `json:"source_refs"`
|
||||||
|
} `json:"spell_casts"`
|
||||||
|
}
|
||||||
|
readJSONFile(t, filepath.Join(runDir, "lanes", laneID+".json"), &payload)
|
||||||
|
if len(payload.SpellCasts) != 2 {
|
||||||
|
t.Fatalf("lane %q spell casts = %#v, want one per source chunk", laneID, payload.SpellCasts)
|
||||||
|
}
|
||||||
|
got := []any{
|
||||||
|
payload.SpellCasts[0].Spell, payload.SpellCasts[0].SourceRefs[0].StartUnitID,
|
||||||
|
payload.SpellCasts[1].Spell, payload.SpellCasts[1].SourceRefs[0].StartUnitID,
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got, []any{"Cure Wounds", 1, "Shield", 2}) {
|
||||||
|
t.Fatalf("lane %q chunk handoff order = %#v, want source chunk order", laneID, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type blockingProductionLLMClient struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
active int
|
||||||
|
maxActive int
|
||||||
|
stageNames []string
|
||||||
|
entered chan struct{}
|
||||||
|
release chan struct{}
|
||||||
|
}
|
||||||
|
|
||||||
|
type orderingProductionLLMClient struct{}
|
||||||
|
|
||||||
|
func (orderingProductionLLMClient) CompleteStructured(_ context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
|
material := req.Inputs["transcript"]
|
||||||
|
unitID := 1
|
||||||
|
spellName := "Cure Wounds"
|
||||||
|
if strings.Contains(string(material.Content), "Shield") {
|
||||||
|
unitID = 2
|
||||||
|
spellName = "Shield"
|
||||||
|
}
|
||||||
|
payload, err := json.Marshal(map[string]any{
|
||||||
|
"spell_casts": []map[string]any{{
|
||||||
|
"caster": "Aria",
|
||||||
|
"spell": spellName,
|
||||||
|
"effect": "Fixture effect.",
|
||||||
|
"narrative_description": "Fixture spell cast.",
|
||||||
|
"source_refs": []map[string]any{{
|
||||||
|
"start_unit_id": unitID,
|
||||||
|
"end_unit_id": unitID,
|
||||||
|
}},
|
||||||
|
}},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return contracts.StructuredCompletionResponse{}, err
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(payload, out); err != nil {
|
||||||
|
return contracts.StructuredCompletionResponse{}, err
|
||||||
|
}
|
||||||
|
return contracts.StructuredCompletionResponse{Content: payload}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newBlockingProductionLLMClient() *blockingProductionLLMClient {
|
||||||
|
return &blockingProductionLLMClient{entered: make(chan struct{}, 2), release: make(chan struct{}, 2)}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (client *blockingProductionLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
|
client.mu.Lock()
|
||||||
|
client.active++
|
||||||
|
if client.active > client.maxActive {
|
||||||
|
client.maxActive = client.active
|
||||||
|
}
|
||||||
|
client.stageNames = append(client.stageNames, req.StageName)
|
||||||
|
client.mu.Unlock()
|
||||||
|
client.entered <- struct{}{}
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return contracts.StructuredCompletionResponse{}, ctx.Err()
|
||||||
|
case <-client.release:
|
||||||
|
}
|
||||||
|
|
||||||
|
client.mu.Lock()
|
||||||
|
client.active--
|
||||||
|
client.mu.Unlock()
|
||||||
|
|
||||||
|
var payload []byte
|
||||||
|
switch req.StageName {
|
||||||
|
case scenes.Key:
|
||||||
|
payload = []byte(`{"scenes":[{"start_unit_id":1,"end_unit_id":2,"short_title":"Spell","primary_mode":"Narrative","main_participants":["Aria"],"summary":"Aria casts a spell.","boundary_note":"Complete source.","boundary_confidence":"High"}],"boundary_caveats":[]}`)
|
||||||
|
case spells.Key:
|
||||||
|
payload = []byte(`{"spell_casts":[{"caster":"Aria","spell":"Cure Wounds","effect":"Healing","narrative_description":"Aria casts Cure Wounds.","source_refs":[{"start_unit_id":1,"end_unit_id":1}]}]}`)
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(payload, out); err != nil {
|
||||||
|
return contracts.StructuredCompletionResponse{}, err
|
||||||
|
}
|
||||||
|
return contracts.StructuredCompletionResponse{Content: payload}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func sceneManifestMetadata(t *testing.T) map[string]any {
|
||||||
|
t.Helper()
|
||||||
|
chunker, err := scenes.New(orderingProductionLLMClient{}, scenes.Options{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("construct scene chunker: %v", err)
|
||||||
|
}
|
||||||
|
return chunker.ManifestMetadata()
|
||||||
|
}
|
||||||
|
|
||||||
|
func spellManifestMetadata(t *testing.T) map[string]any {
|
||||||
|
t.Helper()
|
||||||
|
extractor, err := spells.New(orderingProductionLLMClient{}, spells.Options{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("construct spell extractor: %v", err)
|
||||||
|
}
|
||||||
|
return extractor.ManifestMetadata()
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertAssetNames(t *testing.T, getFS func() (fs.FS, error), want []string) {
|
||||||
|
t.Helper()
|
||||||
|
fSys, err := getFS()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("asset filesystem error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
var got []string
|
||||||
|
if err := fs.WalkDir(fSys, ".", func(path string, entry fs.DirEntry, err error) error {
|
||||||
|
if err == nil && !entry.IsDir() {
|
||||||
|
got = append(got, path)
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("walk assets: %v", err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("asset names = %#v, want compatibility snapshot %#v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolvedValidatorKeys(chains []pipeline.ResolvedValidatorChain, stage pipeline.ModuleStage, laneID, moduleKey string) []string {
|
||||||
|
for _, chain := range chains {
|
||||||
|
if chain.Stage == stage && chain.LaneID == laneID && chain.ModuleKey == moduleKey {
|
||||||
|
keys := make([]string, 0, len(chain.Validators))
|
||||||
|
for _, validator := range chain.Validators {
|
||||||
|
keys = append(keys, validator.Binding.Module)
|
||||||
|
}
|
||||||
|
return keys
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func relativeFileNames(t *testing.T, root string) []string {
|
||||||
|
t.Helper()
|
||||||
|
var names []string
|
||||||
|
if err := filepath.WalkDir(root, func(path string, entry fs.DirEntry, err error) error {
|
||||||
|
if err != nil || entry.IsDir() {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
rel, err := filepath.Rel(root, path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
names = append(names, filepath.ToSlash(rel))
|
||||||
|
return nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("walk durable output: %v", err)
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
return names
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertJSONEqual(t *testing.T, got, want []byte) {
|
||||||
|
t.Helper()
|
||||||
|
var gotValue any
|
||||||
|
var wantValue any
|
||||||
|
if err := json.Unmarshal(got, &gotValue); err != nil {
|
||||||
|
t.Fatalf("unmarshal actual JSON: %v", err)
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(want, &wantValue); err != nil {
|
||||||
|
t.Fatalf("unmarshal expected JSON: %v", err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(gotValue, wantValue) {
|
||||||
|
t.Fatalf("JSON = %#v, want compatibility snapshot %#v", gotValue, wantValue)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -51,7 +51,12 @@ func Run(args []string, stdout, stderr io.Writer) int {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func RunWithOptions(args []string, stdout, stderr io.Writer, opts Options) int {
|
func RunWithOptions(args []string, stdout, stderr io.Writer, opts Options) int {
|
||||||
opts = normalizeOptions(opts)
|
var err error
|
||||||
|
opts, err = normalizeOptions(opts)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
if len(args) == 0 {
|
if len(args) == 0 {
|
||||||
writeUsage(stdout)
|
writeUsage(stdout)
|
||||||
return 0
|
return 0
|
||||||
@@ -78,17 +83,28 @@ func writeUsage(w io.Writer) {
|
|||||||
fmt.Fprint(w, usage)
|
fmt.Fprint(w, usage)
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeOptions(opts Options) Options {
|
func normalizeOptions(opts Options) (Options, error) {
|
||||||
if opts.LookupEnv == nil {
|
if opts.LookupEnv == nil {
|
||||||
opts.LookupEnv = os.LookupEnv
|
opts.LookupEnv = os.LookupEnv
|
||||||
}
|
}
|
||||||
if opts.Now == nil {
|
if opts.Now == nil {
|
||||||
opts.Now = time.Now
|
opts.Now = time.Now
|
||||||
}
|
}
|
||||||
|
if isEmptyCatalog(opts.Catalog) && isEmptyRegistries(opts.Registries) {
|
||||||
|
components, err := newProductionComponents()
|
||||||
|
if err != nil {
|
||||||
|
return Options{}, err
|
||||||
|
}
|
||||||
|
opts.Registries = components.registries
|
||||||
|
opts.Catalog = catalogFromRegistries(components.registries)
|
||||||
|
if opts.LLMClientFactory == nil {
|
||||||
|
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(components.assets)
|
||||||
|
}
|
||||||
|
}
|
||||||
if opts.LLMClientFactory == nil {
|
if opts.LLMClientFactory == nil {
|
||||||
opts.LLMClientFactory = productionLLMClientFactory
|
opts.LLMClientFactory = productionLLMClientFactory
|
||||||
}
|
}
|
||||||
return opts
|
return opts, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) int {
|
func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) int {
|
||||||
@@ -195,6 +211,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("create debug recorder: %w", err))
|
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("create debug recorder: %w", err))
|
||||||
}
|
}
|
||||||
|
debugRecorder = pipeline.SynchronizedDebugRecorder(debugRecorder)
|
||||||
|
|
||||||
catalog, err := effectiveCatalog(opts)
|
catalog, err := effectiveCatalog(opts)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -247,11 +264,6 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics resolved references: %w", err))
|
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics resolved references: %w", err))
|
||||||
}
|
}
|
||||||
|
|
||||||
rawInput, err := os.ReadFile(strings.TrimSpace(*inputPath))
|
|
||||||
if err != nil {
|
|
||||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("read input %q: %w", strings.TrimSpace(*inputPath), err))
|
|
||||||
}
|
|
||||||
|
|
||||||
registries, err := effectiveRegistries(opts)
|
registries, err := effectiveRegistries(opts)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
|
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
|
||||||
@@ -266,25 +278,34 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
|
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
|
||||||
}
|
}
|
||||||
|
llmClient = pipeline.WithDebugLLMRecording(llmClient, debugRecorder)
|
||||||
|
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, registries, pipeline.ModuleDependencies{LLM: llmClient})
|
||||||
|
if err != nil {
|
||||||
|
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("prepare pipeline %q: %w", pipelineID, err))
|
||||||
|
}
|
||||||
|
rawInput, err := os.ReadFile(strings.TrimSpace(*inputPath))
|
||||||
|
if err != nil {
|
||||||
|
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("read input %q: %w", strings.TrimSpace(*inputPath), err))
|
||||||
|
}
|
||||||
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(workspaceSettings, effective.ResolvedPipeline, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), *resume)
|
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(workspaceSettings, effective.ResolvedPipeline, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), *resume)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
|
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
output, err := pipeline.New(registries).Run(ctx, pipeline.RunInput{
|
output, err := pipeline.New().Run(ctx, pipeline.RunInput{
|
||||||
Pipeline: effective.ResolvedPipeline,
|
Prepared: prepared,
|
||||||
Path: strings.TrimSpace(*inputPath),
|
Path: strings.TrimSpace(*inputPath),
|
||||||
RawInput: rawInput,
|
RawInput: rawInput,
|
||||||
LLMClient: llmClient,
|
SessionID: strings.TrimSpace(sessionID.value),
|
||||||
SessionID: strings.TrimSpace(sessionID.value),
|
RunID: runID,
|
||||||
RunID: runID,
|
StartedAt: startedAt,
|
||||||
StartedAt: startedAt,
|
LLMProfiles: llmProfiles,
|
||||||
LLMProfiles: llmProfiles,
|
Metadata: runMetadata(*outputDir, *diagnosticsDir),
|
||||||
Metadata: runMetadata(*outputDir, *diagnosticsDir),
|
Warnings: referenceWarnings,
|
||||||
Warnings: referenceWarnings,
|
Checkpoints: checkpointRecorder,
|
||||||
Checkpoints: checkpointRecorder,
|
Checkpoint: checkpointLoader,
|
||||||
Checkpoint: checkpointLoader,
|
Debug: debugRecorder,
|
||||||
Debug: debugRecorder,
|
ExtractWorkers: cfg.Concurrency.StageWorkers["extract"],
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if output.Manifest.PipelineID != "" && runDir != nil {
|
if output.Manifest.PipelineID != "" && runDir != nil {
|
||||||
|
|||||||
@@ -22,20 +22,20 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/dnd/scenes"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
|
spellshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/shape"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
|
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_refs"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
|
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_relatedness"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/chunk/units"
|
||||||
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/merge/appendorder"
|
||||||
spellshape "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/shape"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
||||||
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_refs"
|
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/output/json"
|
||||||
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_relatedness"
|
alwaysaccept "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/always_accept"
|
||||||
alwaysaccept "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_accept"
|
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/always_reject"
|
||||||
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_reject"
|
validjson "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/valid_json"
|
||||||
validjson "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json"
|
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/valid_json_schema"
|
||||||
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json_schema"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/input/transcript"
|
||||||
"gitea.maximumdirect.net/eric/scriptorium"
|
"gitea.maximumdirect.net/eric/scriptorium"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -135,6 +135,12 @@ func TestProductionCatalogIncludesProductionModulesValidatorsAndDefaults(t *test
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("productionCatalog() error = %v, want nil", err)
|
t.Fatalf("productionCatalog() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
if catalog.ArtifactCodecs == nil {
|
||||||
|
t.Fatal("production artifact codec registry = nil, want initialized empty registry")
|
||||||
|
}
|
||||||
|
if got := catalog.ArtifactCodecs.RegisteredKinds(); !reflect.DeepEqual(got, []contracts.ArtifactKind{"dnd/spell-list"}) {
|
||||||
|
t.Fatalf("production artifact codec kinds = %#v, want dnd/spell-list", got)
|
||||||
|
}
|
||||||
|
|
||||||
moduleTests := []struct {
|
moduleTests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -143,13 +149,13 @@ func TestProductionCatalogIncludesProductionModulesValidatorsAndDefaults(t *test
|
|||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "seriatim input",
|
name: "seriatim input",
|
||||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Inputs.Spec(seriatim.Key) },
|
got: func() (pipeline.ModuleSpec, bool) { return catalog.Inputs.Spec(transcript.Key) },
|
||||||
want: seriatim.ModuleSpec(),
|
want: transcript.ModuleSpec(),
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "generic chunker",
|
name: "generic chunker",
|
||||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Chunkers.Spec(generic.Key) },
|
got: func() (pipeline.ModuleSpec, bool) { return catalog.Chunkers.Spec(units.Key) },
|
||||||
want: generic.ModuleSpec(),
|
want: units.ModuleSpec(),
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "dnd scenes chunker",
|
name: "dnd scenes chunker",
|
||||||
@@ -164,12 +170,12 @@ func TestProductionCatalogIncludesProductionModulesValidatorsAndDefaults(t *test
|
|||||||
{
|
{
|
||||||
name: "appendorder merger",
|
name: "appendorder merger",
|
||||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Mergers.Spec(appendorder.Key) },
|
got: func() (pipeline.ModuleSpec, bool) { return catalog.Mergers.Spec(appendorder.Key) },
|
||||||
want: appendorder.ModuleSpec(),
|
want: appendorder.TypedModuleSpec(spells.ModuleSpec().ArtifactKind),
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "noop normalizer",
|
name: "noop normalizer",
|
||||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Normalizers.Spec(noop.Key) },
|
got: func() (pipeline.ModuleSpec, bool) { return catalog.Normalizers.Spec(noop.Key) },
|
||||||
want: noop.ModuleSpec(),
|
want: noop.TypedModuleSpec(spells.ModuleSpec().ArtifactKind),
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "json output",
|
name: "json output",
|
||||||
@@ -222,11 +228,28 @@ func TestProductionCatalogIncludesProductionModulesValidatorsAndDefaults(t *test
|
|||||||
if !reflect.DeepEqual(gotChain, wantChain) {
|
if !reflect.DeepEqual(gotChain, wantChain) {
|
||||||
t.Fatalf("dnd spell default validator chain = %#v, want %#v", gotChain, wantChain)
|
t.Fatalf("dnd spell default validator chain = %#v, want %#v", gotChain, wantChain)
|
||||||
}
|
}
|
||||||
if got := catalog.ValidatorChains.Validators(pipeline.StageChunk, generic.Key); len(got) != 0 {
|
if got := catalog.ValidatorChains.Validators(pipeline.StageChunk, units.Key); len(got) != 0 {
|
||||||
t.Fatalf("generic chunker default validator chain = %#v, want empty", got)
|
t.Fatalf("generic chunker default validator chain = %#v, want empty", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestCatalogConversionsPreserveArtifactCodecRegistry(t *testing.T) {
|
||||||
|
codecs := pipeline.NewArtifactCodecRegistry()
|
||||||
|
registries := pipeline.Registries{ArtifactCodecs: codecs}
|
||||||
|
if isEmptyRegistries(registries) {
|
||||||
|
t.Fatal("registries with artifact codecs reported empty")
|
||||||
|
}
|
||||||
|
|
||||||
|
catalog := catalogFromRegistries(registries)
|
||||||
|
if catalog.ArtifactCodecs != codecs || isEmptyCatalog(catalog) {
|
||||||
|
t.Fatalf("catalog artifact codecs = %p empty=%t, want %p and non-empty", catalog.ArtifactCodecs, isEmptyCatalog(catalog), codecs)
|
||||||
|
}
|
||||||
|
converted := registriesFromCatalog(catalog)
|
||||||
|
if converted.ArtifactCodecs != codecs {
|
||||||
|
t.Fatalf("converted artifact codecs = %p, want %p", converted.ArtifactCodecs, codecs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestProductionPromptAssetsRegisterAndPrepareDndPrompts(t *testing.T) {
|
func TestProductionPromptAssetsRegisterAndPrepareDndPrompts(t *testing.T) {
|
||||||
registry, err := productionPromptAssets()
|
registry, err := productionPromptAssets()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -3533,6 +3556,10 @@ func fakeExecutionRegistries(t *testing.T) pipeline.Registries {
|
|||||||
mergers := pipeline.NewMergerRegistry()
|
mergers := pipeline.NewMergerRegistry()
|
||||||
normalizers := pipeline.NewNormalizerRegistry()
|
normalizers := pipeline.NewNormalizerRegistry()
|
||||||
outputs := pipeline.NewOutputEncoderRegistry()
|
outputs := pipeline.NewOutputEncoderRegistry()
|
||||||
|
codecs := pipeline.NewArtifactCodecRegistry()
|
||||||
|
if err := pipeline.RegisterArtifactCodec(codecs, fakeRunCodec{}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
if err := inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "fake/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
|
if err := inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "fake/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
|
||||||
return fakeRunInputAdapter{}, nil
|
return fakeRunInputAdapter{}, nil
|
||||||
@@ -3544,25 +3571,25 @@ func fakeExecutionRegistries(t *testing.T) pipeline.Registries {
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register fake chunker: %v", err)
|
t.Fatalf("register fake chunker: %v", err)
|
||||||
}
|
}
|
||||||
if err := extractors.RegisterWithSpec(pipeline.ModuleSpec{
|
if err := pipeline.RegisterExtractor(extractors, pipeline.ModuleSpec{
|
||||||
Key: "fake/extract",
|
Key: "fake/extract",
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
Requires: []string{"chunks"},
|
Requires: []string{"chunks"},
|
||||||
Provides: []string{"artifact"},
|
Provides: []string{"artifact"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster"},
|
{Name: "roster"},
|
||||||
},
|
}, ArtifactKind: fakeRunArtifactKind,
|
||||||
}, func() (contracts.Extractor, error) {
|
}, func() (contracts.Extractor[fakeRunArtifact], error) {
|
||||||
return fakeRunExtractor{}, nil
|
return fakeRunExtractor{}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register fake extractor: %v", err)
|
t.Fatalf("register fake extractor: %v", err)
|
||||||
}
|
}
|
||||||
if err := mergers.RegisterWithSpec(pipeline.ModuleSpec{Key: "appendorder", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}}, func() (contracts.Merger, error) {
|
if err := pipeline.RegisterMerger(mergers, pipeline.ModuleSpec{Key: "appendorder", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: fakeRunArtifactKind}, func() (contracts.Merger[fakeRunArtifact], error) {
|
||||||
return fakeRunMerger{}, nil
|
return fakeRunMerger{}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register fake merger: %v", err)
|
t.Fatalf("register fake merger: %v", err)
|
||||||
}
|
}
|
||||||
if err := normalizers.RegisterWithSpec(pipeline.ModuleSpec{Key: "noop", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer, error) {
|
if err := pipeline.RegisterNormalizer(normalizers, pipeline.ModuleSpec{Key: "noop", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: fakeRunArtifactKind}, func() (contracts.Normalizer[fakeRunArtifact], error) {
|
||||||
return fakeRunNormalizer{}, nil
|
return fakeRunNormalizer{}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register fake normalizer: %v", err)
|
t.Fatalf("register fake normalizer: %v", err)
|
||||||
@@ -3572,12 +3599,13 @@ func fakeExecutionRegistries(t *testing.T) pipeline.Registries {
|
|||||||
}
|
}
|
||||||
|
|
||||||
return pipeline.Registries{
|
return pipeline.Registries{
|
||||||
Inputs: inputs,
|
Inputs: inputs,
|
||||||
Chunkers: chunkers,
|
Chunkers: chunkers,
|
||||||
Extractors: extractors,
|
ArtifactCodecs: codecs,
|
||||||
Mergers: mergers,
|
Extractors: extractors,
|
||||||
Normalizers: normalizers,
|
Mergers: mergers,
|
||||||
Outputs: outputs,
|
Normalizers: normalizers,
|
||||||
|
Outputs: outputs,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -3594,7 +3622,7 @@ func (fakeRunInputAdapter) Parse(ctx context.Context, req contracts.ParseRequest
|
|||||||
Format: "test",
|
Format: "test",
|
||||||
Digest: "sha256:source",
|
Digest: "sha256:source",
|
||||||
Units: []source.SourceUnit{
|
Units: []source.SourceUnit{
|
||||||
{ID: 1, Kind: "text", Text: string(req.Raw)},
|
{ID: 1, Kind: "text", Text: string(req.Raw), Ref: source.SourceRef{SourceID: "source", StartUnitID: 1, EndUnitID: 1}},
|
||||||
},
|
},
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
@@ -3611,14 +3639,33 @@ func (fakeRunChunker) ReferenceSlots() []contracts.ReferenceSlot {
|
|||||||
|
|
||||||
func (fakeRunChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
func (fakeRunChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
||||||
return contracts.ChunkResult{
|
return contracts.ChunkResult{
|
||||||
Chunks: []contracts.SourceChunk{
|
Chunks: []source.Chunk{
|
||||||
{ID: "chunk-1", SourceID: req.Source.ID, Index: 0, StartUnitID: 1, EndUnitID: 1, Content: []byte(`{"units":[1]}`), MediaType: "application/json", Units: req.Source.Units},
|
{ID: "chunk-1", SourceID: req.Source.ID, Index: 0, Ref: source.SourceRef{SourceID: req.Source.ID, StartUnitID: 1, EndUnitID: 1}, Content: []byte(`{"units":[1]}`), MediaType: "application/json", Units: req.Source.Units},
|
||||||
},
|
},
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type fakeRunExtractor struct{}
|
type fakeRunExtractor struct{}
|
||||||
|
|
||||||
|
const fakeRunArtifactKind contracts.ArtifactKind = "test/fake"
|
||||||
|
|
||||||
|
type fakeRunArtifact struct {
|
||||||
|
Value bool `json:"value"`
|
||||||
|
}
|
||||||
|
type fakeRunCodec struct{}
|
||||||
|
|
||||||
|
func (fakeRunCodec) Kind() contracts.ArtifactKind { return fakeRunArtifactKind }
|
||||||
|
func (fakeRunCodec) Schema() contracts.ArtifactSchema {
|
||||||
|
return contracts.ArtifactSchema{ID: "fake.artifact", Name: "fake_artifact", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
|
||||||
|
}
|
||||||
|
func (fakeRunCodec) MediaType() string { return "application/json" }
|
||||||
|
func (fakeRunCodec) Encode(v fakeRunArtifact) ([]byte, error) { return json.Marshal(v) }
|
||||||
|
func (fakeRunCodec) Decode(b []byte) (fakeRunArtifact, error) {
|
||||||
|
var v fakeRunArtifact
|
||||||
|
err := json.Unmarshal(b, &v)
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
|
||||||
func (fakeRunExtractor) Key() string {
|
func (fakeRunExtractor) Key() string {
|
||||||
return "fake/extract"
|
return "fake/extract"
|
||||||
}
|
}
|
||||||
@@ -3627,16 +3674,8 @@ func (fakeRunExtractor) ReferenceSlots() []contracts.ReferenceSlot {
|
|||||||
return []contracts.ReferenceSlot{{Name: "roster"}}
|
return []contracts.ReferenceSlot{{Name: "roster"}}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (fakeRunExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
func (fakeRunExtractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[fakeRunArtifact], error) {
|
||||||
return contracts.ExtractionResult{
|
return contracts.TypedExtractionResult[fakeRunArtifact]{Value: fakeRunArtifact{Value: true}}, nil
|
||||||
Output: contracts.ExtractOutput{
|
|
||||||
Schema: contracts.ResponseSchema{ID: "fake.artifact", Name: "fake_artifact", Version: "v1"},
|
|
||||||
Payload: contracts.RawPayload{
|
|
||||||
Content: []byte(`{"value":true}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type fakeRunMerger struct{}
|
type fakeRunMerger struct{}
|
||||||
@@ -3645,21 +3684,11 @@ func (fakeRunMerger) Key() string {
|
|||||||
return "appendorder"
|
return "appendorder"
|
||||||
}
|
}
|
||||||
|
|
||||||
func (fakeRunMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
|
func (fakeRunMerger) Merge(ctx context.Context, req contracts.TypedMergeRequest[fakeRunArtifact]) (contracts.TypedMergeResult[fakeRunArtifact], error) {
|
||||||
output := contracts.MergeOutput{
|
|
||||||
LaneID: req.LaneID,
|
|
||||||
SourceID: req.Source.ID,
|
|
||||||
Schema: contracts.ResponseSchema{ID: "fake.artifact", Name: "fake_artifact", Version: "v1"},
|
|
||||||
Payload: contracts.RawPayload{
|
|
||||||
Content: []byte(`{"merged":true}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
if len(req.ExtractOutputs) > 0 {
|
if len(req.ExtractOutputs) > 0 {
|
||||||
output.Schema = req.ExtractOutputs[0].Schema
|
return contracts.TypedMergeResult[fakeRunArtifact]{Value: req.ExtractOutputs[0].Value}, nil
|
||||||
output.Payload = req.ExtractOutputs[0].Payload
|
|
||||||
}
|
}
|
||||||
return contracts.MergeResult{Output: output}, nil
|
return contracts.TypedMergeResult[fakeRunArtifact]{}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type fakeRunNormalizer struct{}
|
type fakeRunNormalizer struct{}
|
||||||
@@ -3672,15 +3701,8 @@ func (fakeRunNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (fakeRunNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
|
func (fakeRunNormalizer) Normalize(ctx context.Context, req contracts.TypedNormalizeRequest[fakeRunArtifact]) (contracts.TypedNormalizeResult[fakeRunArtifact], error) {
|
||||||
return contracts.NormalizeResult{
|
return contracts.TypedNormalizeResult[fakeRunArtifact]{Value: req.MergeOutput.Value}, nil
|
||||||
Output: contracts.NormalizeOutput{
|
|
||||||
LaneID: req.LaneID,
|
|
||||||
SourceID: req.MergeOutput.SourceID,
|
|
||||||
Schema: req.MergeOutput.Schema,
|
|
||||||
Payload: req.MergeOutput.Payload,
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func onlyChildDir(t *testing.T, root string) string {
|
func onlyChildDir(t *testing.T, root string) string {
|
||||||
@@ -3942,9 +3964,18 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
|
|||||||
for _, override := range overrides {
|
for _, override := range overrides {
|
||||||
specs[override.Key] = override
|
specs[override.Key] = override
|
||||||
}
|
}
|
||||||
|
for _, key := range []string{"fake/extract", "appendorder", "noop"} {
|
||||||
|
spec := specs[key]
|
||||||
|
spec.ArtifactKind = fakeRunArtifactKind
|
||||||
|
specs[key] = spec
|
||||||
|
}
|
||||||
|
|
||||||
mustRegisterInput(t, inputs, specs["fake/input"])
|
mustRegisterInput(t, inputs, specs["fake/input"])
|
||||||
mustRegisterChunker(t, chunkers, specs["generic"])
|
mustRegisterChunker(t, chunkers, specs["generic"])
|
||||||
|
codecs := pipeline.NewArtifactCodecRegistry()
|
||||||
|
if err := pipeline.RegisterArtifactCodec(codecs, fakeRunCodec{}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
mustRegisterExtractor(t, extractors, specs["fake/extract"])
|
mustRegisterExtractor(t, extractors, specs["fake/extract"])
|
||||||
mustRegisterMerger(t, mergers, specs["appendorder"])
|
mustRegisterMerger(t, mergers, specs["appendorder"])
|
||||||
mustRegisterNormalizer(t, normalizers, specs["noop"])
|
mustRegisterNormalizer(t, normalizers, specs["noop"])
|
||||||
@@ -3953,6 +3984,7 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
|
|||||||
return pipeline.ModuleCatalog{
|
return pipeline.ModuleCatalog{
|
||||||
Inputs: inputs,
|
Inputs: inputs,
|
||||||
Chunkers: chunkers,
|
Chunkers: chunkers,
|
||||||
|
ArtifactCodecs: codecs,
|
||||||
Extractors: extractors,
|
Extractors: extractors,
|
||||||
Mergers: mergers,
|
Mergers: mergers,
|
||||||
Normalizers: normalizers,
|
Normalizers: normalizers,
|
||||||
@@ -3964,53 +3996,58 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
|
|||||||
|
|
||||||
func mustRegisterInput(t *testing.T, registry *pipeline.InputAdapterRegistry, spec pipeline.ModuleSpec) {
|
func mustRegisterInput(t *testing.T, registry *pipeline.InputAdapterRegistry, spec pipeline.ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := registry.RegisterWithSpec(spec, func() (contracts.InputAdapter, error) { return nil, nil }); err != nil {
|
if err := registry.RegisterWithSpec(spec, func() (contracts.InputAdapter, error) { return fakeRunInputAdapter{}, nil }); err != nil {
|
||||||
t.Fatalf("register input: %v", err)
|
t.Fatalf("register input: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mustRegisterChunker(t *testing.T, registry *pipeline.ChunkerRegistry, spec pipeline.ModuleSpec) {
|
func mustRegisterChunker(t *testing.T, registry *pipeline.ChunkerRegistry, spec pipeline.ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Chunker, error) { return nil, nil }); err != nil {
|
if err := registry.RegisterWithSpec(spec, func() (contracts.Chunker, error) { return fakeRunChunker{}, nil }); err != nil {
|
||||||
t.Fatalf("register chunker: %v", err)
|
t.Fatalf("register chunker: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mustRegisterExtractor(t *testing.T, registry *pipeline.ExtractorRegistry, spec pipeline.ModuleSpec) {
|
func mustRegisterExtractor(t *testing.T, registry *pipeline.ExtractorRegistry, spec pipeline.ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Extractor, error) { return nil, nil }); err != nil {
|
if err := pipeline.RegisterExtractor(registry, spec, func() (contracts.Extractor[fakeRunArtifact], error) { return fakeRunExtractor{}, nil }); err != nil {
|
||||||
t.Fatalf("register extractor: %v", err)
|
t.Fatalf("register extractor: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pipeline.ModuleSpec) {
|
func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pipeline.ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Merger, error) { return nil, nil }); err != nil {
|
if err := pipeline.RegisterMerger(registry, spec, func() (contracts.Merger[fakeRunArtifact], error) { return fakeRunMerger{}, nil }); err != nil {
|
||||||
t.Fatalf("register merger: %v", err)
|
t.Fatalf("register merger: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mustRegisterNormalizer(t *testing.T, registry *pipeline.NormalizerRegistry, spec pipeline.ModuleSpec) {
|
func mustRegisterNormalizer(t *testing.T, registry *pipeline.NormalizerRegistry, spec pipeline.ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Normalizer, error) { return nil, nil }); err != nil {
|
if err := pipeline.RegisterNormalizer(registry, spec, func() (contracts.Normalizer[fakeRunArtifact], error) { return fakeRunNormalizer{}, nil }); err != nil {
|
||||||
t.Fatalf("register normalizer: %v", err)
|
t.Fatalf("register normalizer: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mustRegisterOutput(t *testing.T, registry *pipeline.OutputEncoderRegistry, spec pipeline.ModuleSpec) {
|
func mustRegisterOutput(t *testing.T, registry *pipeline.OutputEncoderRegistry, spec pipeline.ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := registry.RegisterWithSpec(spec, func() (contracts.OutputEncoder, error) { return nil, nil }); err != nil {
|
if err := registry.RegisterWithSpec(spec, func() (contracts.OutputEncoder, error) { return jsonoutput.New(), nil }); err != nil {
|
||||||
t.Fatalf("register output: %v", err)
|
t.Fatalf("register output: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mustRegisterValidator(t *testing.T, registry *pipeline.ValidatorRegistry, spec pipeline.ValidatorSpec) {
|
func mustRegisterValidator(t *testing.T, registry *pipeline.ValidatorRegistry, spec pipeline.ValidatorSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Validator, error) {
|
if err := pipeline.RegisterChunkValidator(registry, spec, func() (contracts.ChunkValidator, error) {
|
||||||
return fakeConfigValidator{name: spec.Key, executionClass: spec.ExecutionClass}, nil
|
return fakeConfigValidator{name: spec.Key, executionClass: spec.ExecutionClass}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register validator: %v", err)
|
t.Fatalf("register validator: %v", err)
|
||||||
}
|
}
|
||||||
|
if err := pipeline.RegisterTypedValidator[fakeRunArtifact](registry, fakeRunArtifactKind, spec, func() (contracts.TypedValidator[fakeRunArtifact], error) {
|
||||||
|
return fakeConfigTypedValidator{fakeConfigValidator{name: spec.Key, executionClass: spec.ExecutionClass}}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register typed validator: %v", err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
type fakeConfigValidator struct {
|
type fakeConfigValidator struct {
|
||||||
@@ -4026,7 +4063,13 @@ func (validator fakeConfigValidator) ExecutionClass() contracts.ExecutionClass {
|
|||||||
return validator.executionClass
|
return validator.executionClass
|
||||||
}
|
}
|
||||||
|
|
||||||
func (validator fakeConfigValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
|
func (validator fakeConfigValidator) Validate(ctx context.Context, req contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
|
||||||
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeConfigTypedValidator struct{ fakeConfigValidator }
|
||||||
|
|
||||||
|
func (validator fakeConfigTypedValidator) Validate(ctx context.Context, req contracts.TypedValidationRequest[fakeRunArtifact]) (contracts.ValidationResult, error) {
|
||||||
return contracts.ValidationResult{Approved: true}, nil
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -24,7 +24,11 @@ type ScriptoriumConfig struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type ConcurrencyConfig struct {
|
type ConcurrencyConfig struct {
|
||||||
TotalLLM int `json:"total_llm"`
|
TotalLLM int `json:"total_llm"`
|
||||||
|
StageWorkers map[string]int `json:"stage_workers"`
|
||||||
|
|
||||||
|
extractWorkersConfigured bool
|
||||||
|
defaultedExtractWorkers int
|
||||||
}
|
}
|
||||||
|
|
||||||
type DiagnosticsConfig struct {
|
type DiagnosticsConfig struct {
|
||||||
@@ -58,7 +62,9 @@ func Default() Config {
|
|||||||
return Config{
|
return Config{
|
||||||
Pipelines: map[string]pipeline.PipelineProfile{},
|
Pipelines: map[string]pipeline.PipelineProfile{},
|
||||||
Concurrency: ConcurrencyConfig{
|
Concurrency: ConcurrencyConfig{
|
||||||
TotalLLM: 1,
|
TotalLLM: 1,
|
||||||
|
StageWorkers: map[string]int{"extract": 1},
|
||||||
|
defaultedExtractWorkers: 1,
|
||||||
},
|
},
|
||||||
Diagnostics: DiagnosticsConfig{
|
Diagnostics: DiagnosticsConfig{
|
||||||
WorkDir: "/tmp/notarius",
|
WorkDir: "/tmp/notarius",
|
||||||
@@ -101,6 +107,7 @@ func (c Config) workspaceDirectory() string {
|
|||||||
|
|
||||||
func cloneConfig(in Config) Config {
|
func cloneConfig(in Config) Config {
|
||||||
out := in
|
out := in
|
||||||
|
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
|
||||||
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
|
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
|
||||||
for key, profile := range in.Pipelines {
|
for key, profile := range in.Pipelines {
|
||||||
out.Pipelines[key] = clonePipelineProfile(profile)
|
out.Pipelines[key] = clonePipelineProfile(profile)
|
||||||
@@ -108,6 +115,34 @@ func cloneConfig(in Config) Config {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func cloneIntMap(in map[string]int) map[string]int {
|
||||||
|
if len(in) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make(map[string]int, len(in))
|
||||||
|
for key, value := range in {
|
||||||
|
out[key] = value
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *ConcurrencyConfig) recomputeStageWorkerDefaults() {
|
||||||
|
if c == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if c.StageWorkers == nil {
|
||||||
|
c.StageWorkers = make(map[string]int)
|
||||||
|
}
|
||||||
|
if !c.extractWorkersConfigured {
|
||||||
|
if value, ok := c.StageWorkers["extract"]; ok && (c.defaultedExtractWorkers == 0 || value != c.defaultedExtractWorkers) {
|
||||||
|
c.extractWorkersConfigured = true
|
||||||
|
return
|
||||||
|
}
|
||||||
|
c.StageWorkers["extract"] = c.TotalLLM
|
||||||
|
c.defaultedExtractWorkers = c.TotalLLM
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func clonePipelineProfile(in pipeline.PipelineProfile) pipeline.PipelineProfile {
|
func clonePipelineProfile(in pipeline.PipelineProfile) pipeline.PipelineProfile {
|
||||||
out := in
|
out := in
|
||||||
out.Input = cloneModuleBinding(in.Input)
|
out.Input = cloneModuleBinding(in.Input)
|
||||||
|
|||||||
@@ -18,6 +18,9 @@ func TestDefaultValues(t *testing.T) {
|
|||||||
if cfg.Concurrency.TotalLLM != 1 {
|
if cfg.Concurrency.TotalLLM != 1 {
|
||||||
t.Fatalf("unexpected total LLM concurrency: %d", cfg.Concurrency.TotalLLM)
|
t.Fatalf("unexpected total LLM concurrency: %d", cfg.Concurrency.TotalLLM)
|
||||||
}
|
}
|
||||||
|
if got := cfg.Concurrency.StageWorkers["extract"]; got != 1 {
|
||||||
|
t.Fatalf("unexpected extract workers: %d", got)
|
||||||
|
}
|
||||||
if cfg.Diagnostics.WorkDir != "/tmp/notarius" {
|
if cfg.Diagnostics.WorkDir != "/tmp/notarius" {
|
||||||
t.Fatalf("unexpected diagnostics work dir: %q", cfg.Diagnostics.WorkDir)
|
t.Fatalf("unexpected diagnostics work dir: %q", cfg.Diagnostics.WorkDir)
|
||||||
}
|
}
|
||||||
@@ -68,6 +71,9 @@ pipelines:
|
|||||||
if cfg.Concurrency.TotalLLM != 1 {
|
if cfg.Concurrency.TotalLLM != 1 {
|
||||||
t.Fatalf("expected default concurrency preserved, got %d", cfg.Concurrency.TotalLLM)
|
t.Fatalf("expected default concurrency preserved, got %d", cfg.Concurrency.TotalLLM)
|
||||||
}
|
}
|
||||||
|
if got := cfg.Concurrency.StageWorkers["extract"]; got != 1 {
|
||||||
|
t.Fatalf("expected default extract workers preserved, got %d", got)
|
||||||
|
}
|
||||||
if cfg.Diagnostics.Retention != diagnostics.RetentionAuto {
|
if cfg.Diagnostics.Retention != diagnostics.RetentionAuto {
|
||||||
t.Fatalf("expected default diagnostics retention preserved, got %q", cfg.Diagnostics.Retention)
|
t.Fatalf("expected default diagnostics retention preserved, got %q", cfg.Diagnostics.Retention)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -26,6 +26,7 @@ type EffectiveConfig struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
||||||
|
c.Concurrency.recomputeStageWorkerDefaults()
|
||||||
if err := c.Validate(); err != nil {
|
if err := c.Validate(); err != nil {
|
||||||
return EffectiveConfig{}, err
|
return EffectiveConfig{}, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -27,6 +27,19 @@ func TestResolveRejectsEmptyAndUnknownPipelineID(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestResolveMaterializesDefaultExtractWorkersFromEffectiveTotal(t *testing.T) {
|
||||||
|
cfg := validConfig()
|
||||||
|
cfg.Concurrency.TotalLLM = 4
|
||||||
|
|
||||||
|
effective, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: fakeCatalog(t)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if got := effective.Config.Concurrency.StageWorkers["extract"]; got != 4 {
|
||||||
|
t.Fatalf("effective extract workers = %d, want total concurrency 4", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResolveLaneFilteringSuccessAndFailure(t *testing.T) {
|
func TestResolveLaneFilteringSuccessAndFailure(t *testing.T) {
|
||||||
effective, err := validConfig().Resolve(ResolveInput{
|
effective, err := validConfig().Resolve(ResolveInput{
|
||||||
PipelineID: " example ",
|
PipelineID: " example ",
|
||||||
|
|||||||
@@ -36,6 +36,18 @@ func (c *Config) applyEnvOverridesWithLookup(lookup func(string) (string, bool))
|
|||||||
}
|
}
|
||||||
c.Concurrency.TotalLLM = value
|
c.Concurrency.TotalLLM = value
|
||||||
}
|
}
|
||||||
|
if raw, ok := lookup("NOTARIUS_STAGE_WORKERS_EXTRACT"); ok {
|
||||||
|
value, err := parseIntEnv("NOTARIUS_STAGE_WORKERS_EXTRACT", raw)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if c.Concurrency.StageWorkers == nil {
|
||||||
|
c.Concurrency.StageWorkers = make(map[string]int)
|
||||||
|
}
|
||||||
|
c.Concurrency.StageWorkers["extract"] = value
|
||||||
|
c.Concurrency.extractWorkersConfigured = true
|
||||||
|
}
|
||||||
|
c.Concurrency.recomputeStageWorkerDefaults()
|
||||||
if raw, ok := lookup("NOTARIUS_WORK_DIR"); ok {
|
if raw, ok := lookup("NOTARIUS_WORK_DIR"); ok {
|
||||||
c.Diagnostics.WorkDir = strings.TrimSpace(raw)
|
c.Diagnostics.WorkDir = strings.TrimSpace(raw)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ func TestApplyEnvOverridesOperationalValues(t *testing.T) {
|
|||||||
|
|
||||||
err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{
|
err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{
|
||||||
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "3",
|
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "3",
|
||||||
|
"NOTARIUS_STAGE_WORKERS_EXTRACT": "2",
|
||||||
"NOTARIUS_WORK_DIR": "/tmp/notarius-env",
|
"NOTARIUS_WORK_DIR": "/tmp/notarius-env",
|
||||||
"NOTARIUS_DIAGNOSTICS_RETENTION": "never",
|
"NOTARIUS_DIAGNOSTICS_RETENTION": "never",
|
||||||
"NOTARIUS_WORKSPACE_DIR": "/var/lib/notarius-env",
|
"NOTARIUS_WORKSPACE_DIR": "/var/lib/notarius-env",
|
||||||
@@ -33,6 +34,9 @@ func TestApplyEnvOverridesOperationalValues(t *testing.T) {
|
|||||||
if cfg.Concurrency.TotalLLM != 3 {
|
if cfg.Concurrency.TotalLLM != 3 {
|
||||||
t.Fatalf("unexpected total concurrency: %d", cfg.Concurrency.TotalLLM)
|
t.Fatalf("unexpected total concurrency: %d", cfg.Concurrency.TotalLLM)
|
||||||
}
|
}
|
||||||
|
if got := cfg.Concurrency.StageWorkers["extract"]; got != 2 {
|
||||||
|
t.Fatalf("extract workers = %d, want 2", got)
|
||||||
|
}
|
||||||
if cfg.Workspace.Directory != "/var/lib/notarius-env" {
|
if cfg.Workspace.Directory != "/var/lib/notarius-env" {
|
||||||
t.Fatalf("unexpected workspace directory: %q", cfg.Workspace.Directory)
|
t.Fatalf("unexpected workspace directory: %q", cfg.Workspace.Directory)
|
||||||
}
|
}
|
||||||
@@ -54,12 +58,74 @@ func TestApplyEnvOverridesOperationalValues(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestApplyEnvOverridesRejectsInvalidIntegers(t *testing.T) {
|
func TestApplyEnvOverridesRejectsInvalidIntegers(t *testing.T) {
|
||||||
|
for _, name := range []string{"NOTARIUS_TOTAL_LLM_CONCURRENCY", "NOTARIUS_STAGE_WORKERS_EXTRACT"} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
cfg := Default()
|
||||||
|
err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{name: "many"}))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), name) {
|
||||||
|
t.Fatalf("expected named integer error, got %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStageWorkerEnvironmentPrecedenceAndDefaulting(t *testing.T) {
|
||||||
|
fileCfg, err := ParseFileConfigYAML([]byte(`
|
||||||
|
version: 2
|
||||||
|
concurrency:
|
||||||
|
total_llm: 4
|
||||||
|
stage_workers:
|
||||||
|
extract: 2
|
||||||
|
`))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseFileConfigYAML() error = %v", err)
|
||||||
|
}
|
||||||
cfg := Default()
|
cfg := Default()
|
||||||
err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{
|
if err := cfg.applyFileConfigWithLookup(fileCfg, emptyLookup); err != nil {
|
||||||
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "many",
|
t.Fatalf("ApplyFileConfig() error = %v", err)
|
||||||
}))
|
}
|
||||||
if err == nil || !strings.Contains(err.Error(), "NOTARIUS_TOTAL_LLM_CONCURRENCY") {
|
if err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{
|
||||||
t.Fatalf("expected named integer error, got %v", err)
|
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "5",
|
||||||
|
"NOTARIUS_STAGE_WORKERS_EXTRACT": "3",
|
||||||
|
})); err != nil {
|
||||||
|
t.Fatalf("ApplyEnvOverrides() error = %v", err)
|
||||||
|
}
|
||||||
|
if cfg.Concurrency.TotalLLM != 5 || cfg.Concurrency.StageWorkers["extract"] != 3 {
|
||||||
|
t.Fatalf("effective concurrency = %#v, want total 5 and extract 3", cfg.Concurrency)
|
||||||
|
}
|
||||||
|
if err := cfg.Validate(); err != nil {
|
||||||
|
t.Fatalf("Validate(overridden) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defaulted := Default()
|
||||||
|
if err := defaulted.applyEnvOverridesWithLookup(mapLookup(map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"})); err != nil {
|
||||||
|
t.Fatalf("ApplyEnvOverrides(defaulted) error = %v", err)
|
||||||
|
}
|
||||||
|
if got := defaulted.Concurrency.StageWorkers["extract"]; got != 6 {
|
||||||
|
t.Fatalf("defaulted extract workers = %d, want effective total 6", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStageWorkerRangeValidationUsesFinalEnvironmentTotal(t *testing.T) {
|
||||||
|
fileCfg, err := ParseFileConfigYAML([]byte(`
|
||||||
|
version: 2
|
||||||
|
concurrency:
|
||||||
|
total_llm: 4
|
||||||
|
stage_workers:
|
||||||
|
extract: 5
|
||||||
|
`))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseFileConfigYAML() error = %v", err)
|
||||||
|
}
|
||||||
|
cfg := Default()
|
||||||
|
if err := cfg.applyFileConfigWithLookup(fileCfg, emptyLookup); err != nil {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"})); err != nil {
|
||||||
|
t.Fatalf("ApplyEnvOverrides() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := cfg.Validate(); err != nil {
|
||||||
|
t.Fatalf("Validate() error = %v, want final total to make extract workers valid", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -111,4 +177,7 @@ func TestLoadFromEnvUsesDefaultConfig(t *testing.T) {
|
|||||||
if cfg.Concurrency.TotalLLM != 2 {
|
if cfg.Concurrency.TotalLLM != 2 {
|
||||||
t.Fatalf("expected env concurrency override, got %+v", cfg.Concurrency)
|
t.Fatalf("expected env concurrency override, got %+v", cfg.Concurrency)
|
||||||
}
|
}
|
||||||
|
if got := cfg.Concurrency.StageWorkers["extract"]; got != 2 {
|
||||||
|
t.Fatalf("expected extract workers to default to total, got %d", got)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -43,7 +43,8 @@ type FileArtifactLaneProfile struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type FileConcurrencyConfig struct {
|
type FileConcurrencyConfig struct {
|
||||||
TotalLLM *int `yaml:"total_llm,omitempty"`
|
TotalLLM *int `yaml:"total_llm,omitempty"`
|
||||||
|
StageWorkers map[string]int `yaml:"stage_workers,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type FileDiagnosticsConfig struct {
|
type FileDiagnosticsConfig struct {
|
||||||
@@ -316,6 +317,15 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
|||||||
if fileCfg.Concurrency != nil && fileCfg.Concurrency.TotalLLM != nil {
|
if fileCfg.Concurrency != nil && fileCfg.Concurrency.TotalLLM != nil {
|
||||||
c.Concurrency.TotalLLM = *fileCfg.Concurrency.TotalLLM
|
c.Concurrency.TotalLLM = *fileCfg.Concurrency.TotalLLM
|
||||||
}
|
}
|
||||||
|
if fileCfg.Concurrency != nil && fileCfg.Concurrency.StageWorkers != nil {
|
||||||
|
workers, configured, err := normalizeStageWorkers(fileCfg.Concurrency.StageWorkers)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
c.Concurrency.StageWorkers = workers
|
||||||
|
c.Concurrency.extractWorkersConfigured = configured
|
||||||
|
}
|
||||||
|
c.Concurrency.recomputeStageWorkerDefaults()
|
||||||
if fileCfg.Diagnostics != nil {
|
if fileCfg.Diagnostics != nil {
|
||||||
if fileCfg.Diagnostics.WorkDir != nil {
|
if fileCfg.Diagnostics.WorkDir != nil {
|
||||||
c.Diagnostics.WorkDir = strings.TrimSpace(*fileCfg.Diagnostics.WorkDir)
|
c.Diagnostics.WorkDir = strings.TrimSpace(*fileCfg.Diagnostics.WorkDir)
|
||||||
@@ -350,6 +360,26 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func normalizeStageWorkers(values map[string]int) (map[string]int, bool, error) {
|
||||||
|
workers := make(map[string]int, len(values))
|
||||||
|
configured := false
|
||||||
|
for rawKey, value := range values {
|
||||||
|
key := strings.TrimSpace(rawKey)
|
||||||
|
if key == "" {
|
||||||
|
return nil, false, fmt.Errorf("concurrency.stage_workers key must not be empty")
|
||||||
|
}
|
||||||
|
if key != "extract" {
|
||||||
|
return nil, false, fmt.Errorf("concurrency.stage_workers key %q is not supported", rawKey)
|
||||||
|
}
|
||||||
|
if _, exists := workers[key]; exists {
|
||||||
|
return nil, false, fmt.Errorf("concurrency.stage_workers key %q is duplicated after trimming", key)
|
||||||
|
}
|
||||||
|
workers[key] = value
|
||||||
|
configured = true
|
||||||
|
}
|
||||||
|
return workers, configured, nil
|
||||||
|
}
|
||||||
|
|
||||||
func normalizedMapKeys[T any](values map[string]T, keyName string) ([]string, map[string]string, error) {
|
func normalizedMapKeys[T any](values map[string]T, keyName string) ([]string, map[string]string, error) {
|
||||||
keys := make([]string, 0, len(values))
|
keys := make([]string, 0, len(values))
|
||||||
rawByNormalized := make(map[string]string, len(values))
|
rawByNormalized := make(map[string]string, len(values))
|
||||||
|
|||||||
@@ -250,7 +250,7 @@ pipelines:
|
|||||||
}
|
}
|
||||||
lane := profile.Artifacts["events"]
|
lane := profile.Artifacts["events"]
|
||||||
if !reflect.DeepEqual(lane.References, map[string]string{"lore": "./lore.md", "roster": "./legacy-roster.yml"}) {
|
if !reflect.DeepEqual(lane.References, map[string]string{"lore": "./lore.md", "roster": "./legacy-roster.yml"}) {
|
||||||
t.Fatalf("legacy lane references = %#v, want trimmed map", lane.References)
|
t.Fatalf("lane references = %#v, want trimmed map", lane.References)
|
||||||
}
|
}
|
||||||
wantExtract := map[string]string{
|
wantExtract := map[string]string{
|
||||||
"glossary": "./glossary.md",
|
"glossary": "./glossary.md",
|
||||||
@@ -546,6 +546,9 @@ diagnostics:
|
|||||||
if cfg.Concurrency.TotalLLM != 4 {
|
if cfg.Concurrency.TotalLLM != 4 {
|
||||||
t.Fatalf("unexpected total concurrency: %d", cfg.Concurrency.TotalLLM)
|
t.Fatalf("unexpected total concurrency: %d", cfg.Concurrency.TotalLLM)
|
||||||
}
|
}
|
||||||
|
if got := cfg.Concurrency.StageWorkers["extract"]; got != 4 {
|
||||||
|
t.Fatalf("default extract workers = %d, want total concurrency", got)
|
||||||
|
}
|
||||||
if cfg.Diagnostics.WorkDir != "/tmp/notarius-test" {
|
if cfg.Diagnostics.WorkDir != "/tmp/notarius-test" {
|
||||||
t.Fatalf("unexpected work dir: %q", cfg.Diagnostics.WorkDir)
|
t.Fatalf("unexpected work dir: %q", cfg.Diagnostics.WorkDir)
|
||||||
}
|
}
|
||||||
@@ -554,6 +557,58 @@ diagnostics:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestApplyFileConfigStageWorkers(t *testing.T) {
|
||||||
|
cfg := parseAndApplyConfig(t, `
|
||||||
|
version: 2
|
||||||
|
concurrency:
|
||||||
|
total_llm: 4
|
||||||
|
stage_workers:
|
||||||
|
extract: 3
|
||||||
|
`)
|
||||||
|
|
||||||
|
if got := cfg.Concurrency.StageWorkers["extract"]; got != 3 {
|
||||||
|
t.Fatalf("extract workers = %d, want 3", got)
|
||||||
|
}
|
||||||
|
if err := cfg.Validate(); err != nil {
|
||||||
|
t.Fatalf("Validate() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestApplyFileConfigEmptyStageWorkersDefaultsExtractToTotal(t *testing.T) {
|
||||||
|
cfg := parseAndApplyConfig(t, `
|
||||||
|
version: 2
|
||||||
|
concurrency:
|
||||||
|
total_llm: 4
|
||||||
|
stage_workers: {}
|
||||||
|
`)
|
||||||
|
if got := cfg.Concurrency.StageWorkers["extract"]; got != 4 {
|
||||||
|
t.Fatalf("extract workers = %d, want total concurrency 4", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestApplyFileConfigRejectsUnsupportedStageWorkerKeys(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
key string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "empty", key: "' '", want: "must not be empty"},
|
||||||
|
{name: "unknown", key: "merge", want: "not supported"},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
fileCfg, err := ParseFileConfigYAML([]byte("version: 2\nconcurrency:\n stage_workers:\n " + test.key + ": 1\n"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseFileConfigYAML() error = %v", err)
|
||||||
|
}
|
||||||
|
cfg := Default()
|
||||||
|
err = cfg.applyFileConfigWithLookup(fileCfg, emptyLookup)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestApplyFileConfigWorkspaceSection(t *testing.T) {
|
func TestApplyFileConfigWorkspaceSection(t *testing.T) {
|
||||||
cfg := parseAndApplyConfig(t, `
|
cfg := parseAndApplyConfig(t, `
|
||||||
version: 2
|
version: 2
|
||||||
|
|||||||
@@ -50,6 +50,8 @@ func cloneResolvedValidatorChain(in pipeline.ResolvedValidatorChain) pipeline.Re
|
|||||||
out.Validators[i] = pipeline.ResolvedValidator{
|
out.Validators[i] = pipeline.ResolvedValidator{
|
||||||
Binding: cloneModuleBinding(validator.Binding),
|
Binding: cloneModuleBinding(validator.Binding),
|
||||||
ExecutionClass: validator.ExecutionClass,
|
ExecutionClass: validator.ExecutionClass,
|
||||||
|
Target: validator.Target,
|
||||||
|
ArtifactKind: validator.ArtifactKind,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ func TestRedactedConfigCopiesScriptoriumConfig(t *testing.T) {
|
|||||||
cfg.Scriptorium.ProfileDir = "./profiles"
|
cfg.Scriptorium.ProfileDir = "./profiles"
|
||||||
cfg.Workspace.Directory = "/var/lib/notarius"
|
cfg.Workspace.Directory = "/var/lib/notarius"
|
||||||
cfg.Workspace.Resume.Enabled = true
|
cfg.Workspace.Resume.Enabled = true
|
||||||
|
cfg.Concurrency.StageWorkers["extract"] = 1
|
||||||
|
|
||||||
redacted := cfg.Redacted()
|
redacted := cfg.Redacted()
|
||||||
|
|
||||||
@@ -29,6 +30,10 @@ func TestRedactedConfigCopiesScriptoriumConfig(t *testing.T) {
|
|||||||
if cfg.Workspace.Directory != "/var/lib/notarius" {
|
if cfg.Workspace.Directory != "/var/lib/notarius" {
|
||||||
t.Fatalf("redaction mutated original workspace config")
|
t.Fatalf("redaction mutated original workspace config")
|
||||||
}
|
}
|
||||||
|
redacted.Concurrency.StageWorkers["extract"] = 9
|
||||||
|
if cfg.Concurrency.StageWorkers["extract"] != 1 {
|
||||||
|
t.Fatalf("redaction aliased stage worker map")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestConfigRedactedDiagnosticsPayloadCopiesConfig(t *testing.T) {
|
func TestConfigRedactedDiagnosticsPayloadCopiesConfig(t *testing.T) {
|
||||||
@@ -46,6 +51,8 @@ func TestConfigRedactedDiagnosticsPayloadCopiesConfig(t *testing.T) {
|
|||||||
|
|
||||||
func TestEffectiveConfigRedactedDiagnosticsPayloadCopies(t *testing.T) {
|
func TestEffectiveConfigRedactedDiagnosticsPayloadCopies(t *testing.T) {
|
||||||
cfg := validConfig()
|
cfg := validConfig()
|
||||||
|
cfg.Concurrency.TotalLLM = 4
|
||||||
|
cfg.Concurrency.StageWorkers["extract"] = 2
|
||||||
lane := cfg.Pipelines["example"].Artifacts["events"]
|
lane := cfg.Pipelines["example"].Artifacts["events"]
|
||||||
lane.Extract.Options = map[string]any{"temperature": 0.2}
|
lane.Extract.Options = map[string]any{"temperature": 0.2}
|
||||||
lane.References = map[string]string{"roster": "./roster.yml"}
|
lane.References = map[string]string{"roster": "./roster.yml"}
|
||||||
@@ -101,6 +108,10 @@ func TestEffectiveConfigRedactedDiagnosticsPayloadCopies(t *testing.T) {
|
|||||||
if payload.PipelineID != effective.PipelineID || payload.ResolvedPipeline.Digest != effective.ResolvedPipeline.Digest {
|
if payload.PipelineID != effective.PipelineID || payload.ResolvedPipeline.Digest != effective.ResolvedPipeline.Digest {
|
||||||
t.Fatalf("expected pipeline metadata preserved, got %+v", payload)
|
t.Fatalf("expected pipeline metadata preserved, got %+v", payload)
|
||||||
}
|
}
|
||||||
|
payload.Config.Concurrency.StageWorkers["extract"] = 4
|
||||||
|
if effective.Config.Concurrency.StageWorkers["extract"] != 2 {
|
||||||
|
t.Fatalf("expected effective stage worker map to be copied")
|
||||||
|
}
|
||||||
|
|
||||||
payload.Only[0] = "changed"
|
payload.Only[0] = "changed"
|
||||||
if effective.Only[0] != "events" {
|
if effective.Only[0] != "events" {
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package config
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
|
||||||
@@ -9,6 +10,7 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func (c Config) Validate() error {
|
func (c Config) Validate() error {
|
||||||
|
c.Concurrency.recomputeStageWorkerDefaults()
|
||||||
if err := validateScriptorium(c.Scriptorium); err != nil {
|
if err := validateScriptorium(c.Scriptorium); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -21,9 +23,36 @@ func (c Config) Validate() error {
|
|||||||
if c.Concurrency.TotalLLM <= 0 {
|
if c.Concurrency.TotalLLM <= 0 {
|
||||||
return fmt.Errorf("total LLM concurrency must be greater than zero")
|
return fmt.Errorf("total LLM concurrency must be greater than zero")
|
||||||
}
|
}
|
||||||
|
if err := validateStageWorkers(c.Concurrency); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
return validatePipelineProfiles(c.Pipelines)
|
return validatePipelineProfiles(c.Pipelines)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateStageWorkers(cfg ConcurrencyConfig) error {
|
||||||
|
keys := make([]string, 0, len(cfg.StageWorkers))
|
||||||
|
for key := range cfg.StageWorkers {
|
||||||
|
keys = append(keys, key)
|
||||||
|
}
|
||||||
|
sort.Strings(keys)
|
||||||
|
for _, key := range keys {
|
||||||
|
if strings.TrimSpace(key) == "" {
|
||||||
|
return fmt.Errorf("concurrency.stage_workers key must not be empty")
|
||||||
|
}
|
||||||
|
if key != "extract" {
|
||||||
|
return fmt.Errorf("concurrency.stage_workers key %q is not supported", key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
extractWorkers, ok := cfg.StageWorkers["extract"]
|
||||||
|
if !ok {
|
||||||
|
extractWorkers = cfg.TotalLLM
|
||||||
|
}
|
||||||
|
if extractWorkers < 1 || extractWorkers > cfg.TotalLLM {
|
||||||
|
return fmt.Errorf("concurrency.stage_workers.extract must be between 1 and concurrency.total_llm (%d)", cfg.TotalLLM)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func validateScriptorium(cfg ScriptoriumConfig) error {
|
func validateScriptorium(cfg ScriptoriumConfig) error {
|
||||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
||||||
return fmt.Errorf("scriptorium profile_dir and profile_file are mutually exclusive")
|
return fmt.Errorf("scriptorium profile_dir and profile_file are mutually exclusive")
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package config
|
package config
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
@@ -78,6 +79,41 @@ func TestValidateRejectsInvalidNumericFields(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestValidateStageWorkerBoundaries(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
workers int
|
||||||
|
wantErr bool
|
||||||
|
}{
|
||||||
|
{name: "below minimum", workers: 0, wantErr: true},
|
||||||
|
{name: "minimum", workers: 1},
|
||||||
|
{name: "maximum", workers: 4},
|
||||||
|
{name: "above maximum", workers: 5, wantErr: true},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
cfg := validConfig()
|
||||||
|
cfg.Concurrency.TotalLLM = 4
|
||||||
|
cfg.Concurrency.StageWorkers["extract"] = test.workers
|
||||||
|
err := cfg.Validate()
|
||||||
|
if test.wantErr && (err == nil || !strings.Contains(err.Error(), "stage_workers.extract")) {
|
||||||
|
t.Fatalf("Validate() error = %v, want extract worker range error", err)
|
||||||
|
}
|
||||||
|
if !test.wantErr && err != nil {
|
||||||
|
t.Fatalf("Validate() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateRejectsUnknownEffectiveStageWorkerKey(t *testing.T) {
|
||||||
|
cfg := validConfig()
|
||||||
|
cfg.Concurrency.StageWorkers["merge"] = 1
|
||||||
|
err := cfg.Validate()
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "stage_workers key") || !strings.Contains(err.Error(), "merge") {
|
||||||
|
t.Fatalf("Validate() error = %v, want unknown stage worker key", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestValidateRejectsMutuallyExclusiveScriptoriumProfileSources(t *testing.T) {
|
func TestValidateRejectsMutuallyExclusiveScriptoriumProfileSources(t *testing.T) {
|
||||||
cfg := validConfig()
|
cfg := validConfig()
|
||||||
cfg.Scriptorium.ProfileDir = "./profiles"
|
cfg.Scriptorium.ProfileDir = "./profiles"
|
||||||
@@ -497,6 +533,11 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
|
|||||||
for _, override := range overrides {
|
for _, override := range overrides {
|
||||||
specs[override.Key] = override
|
specs[override.Key] = override
|
||||||
}
|
}
|
||||||
|
for _, key := range []string{"fake/extract", "appendorder", "noop"} {
|
||||||
|
spec := specs[key]
|
||||||
|
spec.ArtifactKind = fakeArtifactKind
|
||||||
|
specs[key] = spec
|
||||||
|
}
|
||||||
|
|
||||||
inputs := pipeline.NewInputAdapterRegistry()
|
inputs := pipeline.NewInputAdapterRegistry()
|
||||||
chunkers := pipeline.NewChunkerRegistry()
|
chunkers := pipeline.NewChunkerRegistry()
|
||||||
@@ -515,9 +556,15 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
|
|||||||
mustRegisterValidator(t, validators, specs["fake/llm-validator"])
|
mustRegisterValidator(t, validators, specs["fake/llm-validator"])
|
||||||
mustRegisterOutput(t, outputs, specs["json"])
|
mustRegisterOutput(t, outputs, specs["json"])
|
||||||
|
|
||||||
|
codecs := pipeline.NewArtifactCodecRegistry()
|
||||||
|
if err := pipeline.RegisterArtifactCodec(codecs, fakeArtifactCodec{}); err != nil {
|
||||||
|
t.Fatalf("register artifact codec: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
return pipeline.ModuleCatalog{
|
return pipeline.ModuleCatalog{
|
||||||
Inputs: inputs,
|
Inputs: inputs,
|
||||||
Chunkers: chunkers,
|
Chunkers: chunkers,
|
||||||
|
ArtifactCodecs: codecs,
|
||||||
Extractors: extractors,
|
Extractors: extractors,
|
||||||
Mergers: mergers,
|
Mergers: mergers,
|
||||||
Normalizers: normalizers,
|
Normalizers: normalizers,
|
||||||
@@ -543,21 +590,32 @@ func mustRegisterChunker(t *testing.T, registry *pipeline.ChunkerRegistry, spec
|
|||||||
|
|
||||||
func mustRegisterExtractor(t *testing.T, registry *pipeline.ExtractorRegistry, spec pipeline.ModuleSpec) {
|
func mustRegisterExtractor(t *testing.T, registry *pipeline.ExtractorRegistry, spec pipeline.ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Extractor, error) { return nil, nil }); err != nil {
|
validateOptions := func(options map[string]any) error {
|
||||||
|
if err := pipeline.RejectUnknownOptions(options, "temperature"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if value, ok := options["temperature"]; ok {
|
||||||
|
if _, ok := value.(float64); !ok {
|
||||||
|
return fmt.Errorf("temperature must be a number")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err := pipeline.RegisterExtractorBuilder[fakeArtifact](registry, spec, validateOptions, func(pipeline.BuildRequest) (contracts.Extractor[fakeArtifact], error) { return nil, nil }); err != nil {
|
||||||
t.Fatalf("register extractor: %v", err)
|
t.Fatalf("register extractor: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pipeline.ModuleSpec) {
|
func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pipeline.ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Merger, error) { return nil, nil }); err != nil {
|
if err := pipeline.RegisterMerger[fakeArtifact](registry, spec, func() (contracts.Merger[fakeArtifact], error) { return nil, nil }); err != nil {
|
||||||
t.Fatalf("register merger: %v", err)
|
t.Fatalf("register merger: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mustRegisterNormalizer(t *testing.T, registry *pipeline.NormalizerRegistry, spec pipeline.ModuleSpec) {
|
func mustRegisterNormalizer(t *testing.T, registry *pipeline.NormalizerRegistry, spec pipeline.ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Normalizer, error) { return nil, nil }); err != nil {
|
if err := pipeline.RegisterNormalizer[fakeArtifact](registry, spec, func() (contracts.Normalizer[fakeArtifact], error) { return nil, nil }); err != nil {
|
||||||
t.Fatalf("register normalizer: %v", err)
|
t.Fatalf("register normalizer: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -569,11 +627,32 @@ func mustRegisterValidator(t *testing.T, registry *pipeline.ValidatorRegistry, s
|
|||||||
executionClass = contracts.ExecutionClassLLMBacked
|
executionClass = contracts.ExecutionClassLLMBacked
|
||||||
}
|
}
|
||||||
validatorSpec := pipeline.ValidatorSpec{Key: spec.Key, ExecutionClass: executionClass}
|
validatorSpec := pipeline.ValidatorSpec{Key: spec.Key, ExecutionClass: executionClass}
|
||||||
if err := registry.RegisterWithSpec(validatorSpec, func() (contracts.Validator, error) { return nil, nil }); err != nil {
|
if err := pipeline.RegisterTypedValidator[fakeArtifact](registry, fakeArtifactKind, validatorSpec, func() (contracts.TypedValidator[fakeArtifact], error) { return nil, nil }); err != nil {
|
||||||
t.Fatalf("register validator: %v", err)
|
t.Fatalf("register validator: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const fakeArtifactKind contracts.ArtifactKind = "test/artifact"
|
||||||
|
|
||||||
|
type fakeArtifact string
|
||||||
|
|
||||||
|
type fakeArtifactCodec struct{}
|
||||||
|
|
||||||
|
func (fakeArtifactCodec) Kind() contracts.ArtifactKind { return fakeArtifactKind }
|
||||||
|
func (fakeArtifactCodec) Schema() contracts.ArtifactSchema {
|
||||||
|
return contracts.ArtifactSchema{ID: "urn:notarius:test:artifact", Name: "Test artifact", Version: "1", JSONSchema: []byte(`{"type":"string"}`)}
|
||||||
|
}
|
||||||
|
func (fakeArtifactCodec) MediaType() string { return "application/json" }
|
||||||
|
func (fakeArtifactCodec) Encode(value fakeArtifact) ([]byte, error) {
|
||||||
|
return []byte(fmt.Sprintf("%q", value)), nil
|
||||||
|
}
|
||||||
|
func (fakeArtifactCodec) Decode(content []byte) (fakeArtifact, error) {
|
||||||
|
if len(content) < 2 {
|
||||||
|
return "", fmt.Errorf("invalid test artifact")
|
||||||
|
}
|
||||||
|
return fakeArtifact(content[1 : len(content)-1]), nil
|
||||||
|
}
|
||||||
|
|
||||||
func mustRegisterOutput(t *testing.T, registry *pipeline.OutputEncoderRegistry, spec pipeline.ModuleSpec) {
|
func mustRegisterOutput(t *testing.T, registry *pipeline.OutputEncoderRegistry, spec pipeline.ModuleSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := registry.RegisterWithSpec(spec, func() (contracts.OutputEncoder, error) { return nil, nil }); err != nil {
|
if err := registry.RegisterWithSpec(spec, func() (contracts.OutputEncoder, error) { return nil, nil }); err != nil {
|
||||||
|
|||||||
65
internal/core/source/digest.go
Normal file
65
internal/core/source/digest.go
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
package source
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
)
|
||||||
|
|
||||||
|
// DigestDocument returns a deterministic digest of the canonical source
|
||||||
|
// document content. The existing Digest field is excluded from its own digest.
|
||||||
|
func DigestDocument(doc *SourceDocument) (string, error) {
|
||||||
|
if doc == nil {
|
||||||
|
return "", fmt.Errorf("source document must not be nil")
|
||||||
|
}
|
||||||
|
payload := struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Kind string `json:"kind"`
|
||||||
|
Format string `json:"format"`
|
||||||
|
Units []SourceUnit `json:"units"`
|
||||||
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
|
}{
|
||||||
|
ID: doc.ID,
|
||||||
|
Kind: doc.Kind,
|
||||||
|
Format: doc.Format,
|
||||||
|
Units: doc.Units,
|
||||||
|
Metadata: doc.Metadata,
|
||||||
|
}
|
||||||
|
encoded, err := json.Marshal(payload)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("encode source document for digest: %w", err)
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256(encoded)
|
||||||
|
return "sha256:" + hex.EncodeToString(sum[:]), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// DigestChunk returns a deterministic digest of a chunk, including its source
|
||||||
|
// provenance, content, units, and metadata.
|
||||||
|
func DigestChunk(chunk Chunk) (string, error) {
|
||||||
|
payload := struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
SourceID string `json:"source_id"`
|
||||||
|
Index int `json:"index"`
|
||||||
|
Ref SourceRef `json:"ref"`
|
||||||
|
Content []byte `json:"content"`
|
||||||
|
MediaType string `json:"media_type"`
|
||||||
|
Units []SourceUnit `json:"units"`
|
||||||
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
|
}{
|
||||||
|
ID: chunk.ID,
|
||||||
|
SourceID: chunk.SourceID,
|
||||||
|
Index: chunk.Index,
|
||||||
|
Ref: chunk.Ref,
|
||||||
|
Content: chunk.Content,
|
||||||
|
MediaType: chunk.MediaType,
|
||||||
|
Units: chunk.Units,
|
||||||
|
Metadata: chunk.Metadata,
|
||||||
|
}
|
||||||
|
encoded, err := json.Marshal(payload)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("encode source chunk for digest: %w", err)
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256(encoded)
|
||||||
|
return "sha256:" + hex.EncodeToString(sum[:]), nil
|
||||||
|
}
|
||||||
@@ -13,6 +13,7 @@ type SourceUnit struct {
|
|||||||
ID int `json:"id"`
|
ID int `json:"id"`
|
||||||
Kind string `json:"kind"`
|
Kind string `json:"kind"`
|
||||||
Text string `json:"text"`
|
Text string `json:"text"`
|
||||||
|
Ref SourceRef `json:"ref"`
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -21,3 +22,14 @@ type SourceRef struct {
|
|||||||
StartUnitID int `json:"start_unit_id"`
|
StartUnitID int `json:"start_unit_id"`
|
||||||
EndUnitID int `json:"end_unit_id"`
|
EndUnitID int `json:"end_unit_id"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type Chunk struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
SourceID string `json:"source_id"`
|
||||||
|
Index int `json:"index"`
|
||||||
|
Ref SourceRef `json:"ref"`
|
||||||
|
Content []byte `json:"-"`
|
||||||
|
MediaType string `json:"media_type"`
|
||||||
|
Units []SourceUnit `json:"units"`
|
||||||
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
|
}
|
||||||
|
|||||||
@@ -142,6 +142,122 @@ func TestValidateDocumentDuplicateUnitIDs(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestValidateDocumentUnitReferences(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*SourceDocument)
|
||||||
|
wantErr string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "missing",
|
||||||
|
mutate: func(doc *SourceDocument) { doc.Units[0].Ref = SourceRef{} },
|
||||||
|
wantErr: "source unit[0].ref: source ref source_id must not be empty",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "foreign source",
|
||||||
|
mutate: func(doc *SourceDocument) { doc.Units[0].Ref.SourceID = "source-2" },
|
||||||
|
wantErr: "source unit[0].ref: source ref source_id \"source-2\" does not match document id \"source-1\"",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "non-self range",
|
||||||
|
mutate: func(doc *SourceDocument) {
|
||||||
|
doc.Units[0].Ref.StartUnitID = 2
|
||||||
|
doc.Units[0].Ref.EndUnitID = 2
|
||||||
|
},
|
||||||
|
wantErr: "source unit[0].ref must identify source unit id 1",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "reversed range",
|
||||||
|
mutate: func(doc *SourceDocument) {
|
||||||
|
doc.Units[0].Ref.StartUnitID = 2
|
||||||
|
doc.Units[0].Ref.EndUnitID = 1
|
||||||
|
},
|
||||||
|
wantErr: "source unit[0].ref: source ref start_unit_id 2 appears after end_unit_id 1",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
doc := validDocument()
|
||||||
|
tt.mutate(doc)
|
||||||
|
|
||||||
|
err := ValidateDocument(doc)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("ValidateDocument() error = nil, want unit reference error")
|
||||||
|
}
|
||||||
|
if err.Error() != tt.wantErr {
|
||||||
|
t.Fatalf("ValidateDocument() error = %q, want %q", err.Error(), tt.wantErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDigestDocumentIsDeterministicAndIncludesUnitReference(t *testing.T) {
|
||||||
|
doc := validDocument()
|
||||||
|
doc.Metadata = map[string]any{"second": "value", "first": true}
|
||||||
|
first, err := DigestDocument(doc)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DigestDocument() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
reordered := validDocument()
|
||||||
|
reordered.Metadata = map[string]any{"first": true, "second": "value"}
|
||||||
|
second, err := DigestDocument(reordered)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DigestDocument(reordered) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if first != second {
|
||||||
|
t.Fatalf("digests = %q and %q, want deterministic map ordering", first, second)
|
||||||
|
}
|
||||||
|
|
||||||
|
changed := validDocument()
|
||||||
|
changed.Metadata = map[string]any{"first": true, "second": "value"}
|
||||||
|
changed.Units[0].Ref.SourceID = "different-source"
|
||||||
|
changedDigest, err := DigestDocument(changed)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DigestDocument(changed) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if first == changedDigest {
|
||||||
|
t.Fatalf("digest = %q after reference change, want different digest", changedDigest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDigestChunkIsDeterministicAndIncludesReference(t *testing.T) {
|
||||||
|
doc := validDocument()
|
||||||
|
chunk := Chunk{
|
||||||
|
ID: "chunk-1",
|
||||||
|
SourceID: doc.ID,
|
||||||
|
Index: 0,
|
||||||
|
Ref: SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 2},
|
||||||
|
Content: []byte("chunk content"),
|
||||||
|
MediaType: "text/plain",
|
||||||
|
Units: doc.Units,
|
||||||
|
Metadata: map[string]any{"second": "value", "first": true},
|
||||||
|
}
|
||||||
|
first, err := DigestChunk(chunk)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DigestChunk() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
chunk.Metadata = map[string]any{"first": true, "second": "value"}
|
||||||
|
second, err := DigestChunk(chunk)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DigestChunk(reordered metadata) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if first != second {
|
||||||
|
t.Fatalf("digests = %q and %q, want deterministic map ordering", first, second)
|
||||||
|
}
|
||||||
|
|
||||||
|
chunk.Ref.EndUnitID = 1
|
||||||
|
changed, err := DigestChunk(chunk)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DigestChunk(changed ref) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if first == changed {
|
||||||
|
t.Fatalf("digest = %q after reference change, want different digest", changed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestValidateRefValid(t *testing.T) {
|
func TestValidateRefValid(t *testing.T) {
|
||||||
doc := validDocument()
|
doc := validDocument()
|
||||||
ref := SourceRef{
|
ref := SourceRef{
|
||||||
@@ -274,11 +390,13 @@ func validDocument() *SourceDocument {
|
|||||||
ID: 1,
|
ID: 1,
|
||||||
Kind: "paragraph",
|
Kind: "paragraph",
|
||||||
Text: "First unit.",
|
Text: "First unit.",
|
||||||
|
Ref: SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
ID: 2,
|
ID: 2,
|
||||||
Kind: "paragraph",
|
Kind: "paragraph",
|
||||||
Text: "Second unit.",
|
Text: "Second unit.",
|
||||||
|
Ref: SourceRef{SourceID: "source-1", StartUnitID: 2, EndUnitID: 2},
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -44,6 +44,14 @@ func ValidateDocument(doc *SourceDocument) error {
|
|||||||
}
|
}
|
||||||
seenUnitIDs[unit.ID] = struct{}{}
|
seenUnitIDs[unit.ID] = struct{}{}
|
||||||
}
|
}
|
||||||
|
for i, unit := range doc.Units {
|
||||||
|
if err := ValidateRef(doc, unit.Ref); err != nil {
|
||||||
|
return fmt.Errorf("source unit[%d].ref: %w", i, err)
|
||||||
|
}
|
||||||
|
if unit.Ref.StartUnitID != unit.ID || unit.Ref.EndUnitID != unit.ID {
|
||||||
|
return fmt.Errorf("source unit[%d].ref must identify source unit id %d", i, unit.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,7 +2,10 @@ package workspace
|
|||||||
|
|
||||||
import "time"
|
import "time"
|
||||||
|
|
||||||
const WorkspaceSchemaVersion = "notarius.workspace.v1"
|
const (
|
||||||
|
WorkspaceSchemaVersion = "notarius.workspace.v2"
|
||||||
|
WorkspaceSchemaVersionV1 = "notarius.workspace.v1"
|
||||||
|
)
|
||||||
|
|
||||||
type StageName string
|
type StageName string
|
||||||
|
|
||||||
|
|||||||
@@ -7,6 +7,12 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func TestStageManifestDefaults(t *testing.T) {
|
func TestStageManifestDefaults(t *testing.T) {
|
||||||
|
if WorkspaceSchemaVersion != "notarius.workspace.v2" {
|
||||||
|
t.Fatalf("current schema version = %q, want notarius.workspace.v2", WorkspaceSchemaVersion)
|
||||||
|
}
|
||||||
|
if WorkspaceSchemaVersionV1 != "notarius.workspace.v1" {
|
||||||
|
t.Fatalf("legacy schema version = %q, want notarius.workspace.v1", WorkspaceSchemaVersionV1)
|
||||||
|
}
|
||||||
manifest := NewStageManifest(StageExtract, StatusRunning)
|
manifest := NewStageManifest(StageExtract, StatusRunning)
|
||||||
|
|
||||||
if manifest.WorkspaceSchemaVersion != WorkspaceSchemaVersion {
|
if manifest.WorkspaceSchemaVersion != WorkspaceSchemaVersion {
|
||||||
|
|||||||
@@ -78,80 +78,98 @@ func (l *WorkspaceLoader) Chunk(moduleKey string, sourceDigest string) (pipeline
|
|||||||
if len(chunks) == 0 {
|
if len(chunks) == 0 {
|
||||||
return pipeline.ChunkCheckpoint{}, invalidDecision("chunk checkpoint payload has no chunks")
|
return pipeline.ChunkCheckpoint{}, invalidDecision("chunk checkpoint payload has no chunks")
|
||||||
}
|
}
|
||||||
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), chunkOutputDigests(chunks)) {
|
outputDigests, err := chunkOutputDigests(chunks)
|
||||||
|
if err != nil {
|
||||||
|
return pipeline.ChunkCheckpoint{}, invalidDecision("chunk checkpoint output cannot be digested: %v", err)
|
||||||
|
}
|
||||||
|
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), outputDigests) {
|
||||||
return pipeline.ChunkCheckpoint{}, invalidDecision("chunk checkpoint output digests do not match payload")
|
return pipeline.ChunkCheckpoint{}, invalidDecision("chunk checkpoint output digests do not match payload")
|
||||||
}
|
}
|
||||||
return pipeline.ChunkCheckpoint{Chunks: chunks, Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
return pipeline.ChunkCheckpoint{Chunks: chunks, Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||||
}
|
}
|
||||||
|
|
||||||
func (l *WorkspaceLoader) Extract(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.ExtractCheckpoint, pipeline.CheckpointDecision) {
|
func (l *WorkspaceLoader) Extract(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.ExtractCheckpoint, pipeline.CheckpointDecision) {
|
||||||
var manifest coreworkspace.ExtractLaneManifest
|
var manifest coreworkspace.ExtractLaneManifest
|
||||||
if decision := l.readJSON(laneManifestPath("extract", laneID), &manifest); !decision.Reused {
|
if d := l.readJSON(laneManifestPath("extract", laneID), &manifest); !d.Reused {
|
||||||
return pipeline.ExtractCheckpoint{}, decision
|
return pipeline.ExtractCheckpoint{}, d
|
||||||
}
|
}
|
||||||
if decision := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageExtract, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded, coreworkspace.StatusSucceededWithRejections); !decision.Reused {
|
if d := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageExtract, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded, coreworkspace.StatusSucceededWithRejections); !d.Reused {
|
||||||
return pipeline.ExtractCheckpoint{}, decision
|
return pipeline.ExtractCheckpoint{}, d
|
||||||
}
|
}
|
||||||
var payload extractOutputsEnvelope
|
var payload artifactExtractEnvelope
|
||||||
if decision := l.readJSON(lanePayloadPath("extract", laneID, "outputs.json"), &payload); !decision.Reused {
|
if d := l.readJSON(lanePayloadPath("extract", laneID, "outputs.json"), &payload); !d.Reused {
|
||||||
return pipeline.ExtractCheckpoint{}, decision
|
return pipeline.ExtractCheckpoint{}, d
|
||||||
}
|
}
|
||||||
outputs, err := extractOutputsFromEnvelope(payload.Outputs)
|
outputs, err := artifactCheckpointOutputs(payload.Outputs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return pipeline.ExtractCheckpoint{}, invalidDecision("extract checkpoint payload is invalid: %v", err)
|
return pipeline.ExtractCheckpoint{}, invalidDecision("extract artifact checkpoint payload is invalid: %v", err)
|
||||||
}
|
}
|
||||||
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), rawOutputDigests(extractPayloads(outputs))) {
|
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(outputs)) {
|
||||||
return pipeline.ExtractCheckpoint{}, invalidDecision("extract checkpoint output digests do not match payload")
|
return pipeline.ExtractCheckpoint{}, invalidDecision("extract artifact checkpoint output digests do not match payload")
|
||||||
}
|
}
|
||||||
return pipeline.ExtractCheckpoint{
|
return pipeline.ExtractCheckpoint{Outputs: outputs, Rejected: cloneRejectedOutputs(payload.Rejected), Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||||
Outputs: outputs,
|
|
||||||
Rejected: cloneRejectedOutputs(payload.Rejected),
|
|
||||||
Warnings: cloneWarnings(payload.Warnings),
|
|
||||||
}, reusedDecision()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (l *WorkspaceLoader) Merge(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
|
func (l *WorkspaceLoader) Merge(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
|
||||||
var manifest coreworkspace.MergeLaneManifest
|
var manifest coreworkspace.MergeLaneManifest
|
||||||
if decision := l.readJSON(laneManifestPath("merge", laneID), &manifest); !decision.Reused {
|
if d := l.readJSON(laneManifestPath("merge", laneID), &manifest); !d.Reused {
|
||||||
return pipeline.MergeCheckpoint{}, decision
|
return pipeline.MergeCheckpoint{}, d
|
||||||
}
|
}
|
||||||
if decision := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageMerge, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded); !decision.Reused {
|
if d := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageMerge, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded); !d.Reused {
|
||||||
return pipeline.MergeCheckpoint{}, decision
|
return pipeline.MergeCheckpoint{}, d
|
||||||
}
|
}
|
||||||
var payload mergeOutputEnvelope
|
var payload artifactSingleEnvelope
|
||||||
if decision := l.readJSON(lanePayloadPath("merge", laneID, "output.json"), &payload); !decision.Reused {
|
if d := l.readJSON(lanePayloadPath("merge", laneID, "output.json"), &payload); !d.Reused {
|
||||||
return pipeline.MergeCheckpoint{}, decision
|
return pipeline.MergeCheckpoint{}, d
|
||||||
}
|
}
|
||||||
output, err := mergeOutputFromEnvelope(payload.Output)
|
values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return pipeline.MergeCheckpoint{}, invalidDecision("merge checkpoint payload is invalid: %v", err)
|
return pipeline.MergeCheckpoint{}, invalidDecision("merge artifact checkpoint payload is invalid: %v", err)
|
||||||
}
|
}
|
||||||
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), rawOutputDigests([]contracts.RawPayload{output.Payload})) {
|
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
||||||
return pipeline.MergeCheckpoint{}, invalidDecision("merge checkpoint output digest does not match payload")
|
return pipeline.MergeCheckpoint{}, invalidDecision("merge artifact checkpoint output digest does not match payload")
|
||||||
}
|
}
|
||||||
return pipeline.MergeCheckpoint{Output: output, Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
return pipeline.MergeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||||
}
|
}
|
||||||
|
|
||||||
func (l *WorkspaceLoader) Normalize(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
|
func (l *WorkspaceLoader) Normalize(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
|
||||||
var manifest coreworkspace.NormalizeLaneManifest
|
var manifest coreworkspace.NormalizeLaneManifest
|
||||||
if decision := l.readJSON(laneManifestPath("normalize", laneID), &manifest); !decision.Reused {
|
if d := l.readJSON(laneManifestPath("normalize", laneID), &manifest); !d.Reused {
|
||||||
return pipeline.NormalizeCheckpoint{}, decision
|
return pipeline.NormalizeCheckpoint{}, d
|
||||||
}
|
}
|
||||||
if decision := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageNormalize, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded); !decision.Reused {
|
if d := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageNormalize, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded); !d.Reused {
|
||||||
return pipeline.NormalizeCheckpoint{}, decision
|
return pipeline.NormalizeCheckpoint{}, d
|
||||||
}
|
}
|
||||||
var payload normalizeOutputEnvelope
|
var payload artifactSingleEnvelope
|
||||||
if decision := l.readJSON(lanePayloadPath("normalize", laneID, "output.json"), &payload); !decision.Reused {
|
if d := l.readJSON(lanePayloadPath("normalize", laneID, "output.json"), &payload); !d.Reused {
|
||||||
return pipeline.NormalizeCheckpoint{}, decision
|
return pipeline.NormalizeCheckpoint{}, d
|
||||||
}
|
}
|
||||||
output, err := normalizeOutputFromEnvelope(payload.Output)
|
values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize checkpoint payload is invalid: %v", err)
|
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize artifact checkpoint payload is invalid: %v", err)
|
||||||
}
|
}
|
||||||
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), rawOutputDigests([]contracts.RawPayload{output.Payload})) {
|
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
||||||
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize checkpoint output digest does not match payload")
|
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize artifact checkpoint output digest does not match payload")
|
||||||
}
|
}
|
||||||
return pipeline.NormalizeCheckpoint{Output: output, Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
return pipeline.NormalizeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||||
|
}
|
||||||
|
|
||||||
|
func artifactCheckpointOutputs(values []artifactCheckpointEnvelope) ([]pipeline.CheckpointArtifact, error) {
|
||||||
|
if len(values) == 0 {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
out := make([]pipeline.CheckpointArtifact, 0, len(values))
|
||||||
|
for _, v := range values {
|
||||||
|
content, err := contentFromEnvelope(v.Content)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(string(v.Kind)) == "" || strings.TrimSpace(v.Schema.ID) == "" || strings.TrimSpace(v.Schema.Version) == "" || strings.TrimSpace(v.SchemaDigest) == "" {
|
||||||
|
return nil, fmt.Errorf("artifact codec identity is incomplete")
|
||||||
|
}
|
||||||
|
out = append(out, pipeline.CheckpointArtifact{LaneID: v.LaneID, ModuleKey: v.ModuleKey, SourceID: v.SourceID, ChunkID: v.ChunkID, ChunkIndex: v.ChunkIndex, ChunkRef: v.ChunkRef, SchemaDigest: v.SchemaDigest, Artifact: contracts.SerializedArtifact{Kind: v.Kind, Schema: v.Schema, MediaType: v.Content.MediaType, Content: content, Metadata: cloneMetadata(v.Content.Metadata)}})
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (l *WorkspaceLoader) readJSON(name string, out any) pipeline.CheckpointDecision {
|
func (l *WorkspaceLoader) readJSON(name string, out any) pipeline.CheckpointDecision {
|
||||||
@@ -180,6 +198,9 @@ func (l *WorkspaceLoader) validateManifest(manifest coreworkspace.StageManifest,
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (l *WorkspaceLoader) validateLaneManifest(manifest coreworkspace.StageManifest, stage coreworkspace.StageName, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, statuses ...coreworkspace.StageStatus) pipeline.CheckpointDecision {
|
func (l *WorkspaceLoader) validateLaneManifest(manifest coreworkspace.StageManifest, stage coreworkspace.StageName, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, statuses ...coreworkspace.StageStatus) pipeline.CheckpointDecision {
|
||||||
|
if manifest.WorkspaceSchemaVersion == coreworkspace.WorkspaceSchemaVersionV1 {
|
||||||
|
return invalidDecision("checkpoint workspace schema version %q is incompatible with %q and must be recomputed", manifest.WorkspaceSchemaVersion, coreworkspace.WorkspaceSchemaVersion)
|
||||||
|
}
|
||||||
if manifest.WorkspaceSchemaVersion != coreworkspace.WorkspaceSchemaVersion {
|
if manifest.WorkspaceSchemaVersion != coreworkspace.WorkspaceSchemaVersion {
|
||||||
return invalidDecision("checkpoint workspace schema version %q is not supported", manifest.WorkspaceSchemaVersion)
|
return invalidDecision("checkpoint workspace schema version %q is not supported", manifest.WorkspaceSchemaVersion)
|
||||||
}
|
}
|
||||||
@@ -211,95 +232,30 @@ func (l *WorkspaceLoader) validateLaneManifest(manifest coreworkspace.StageManif
|
|||||||
return reusedDecision()
|
return reusedDecision()
|
||||||
}
|
}
|
||||||
|
|
||||||
func sourceChunksFromEnvelope(values []chunkEnvelope) ([]contracts.SourceChunk, error) {
|
func sourceChunksFromEnvelope(values []chunkEnvelope) ([]source.Chunk, error) {
|
||||||
if len(values) == 0 {
|
if len(values) == 0 {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
out := make([]contracts.SourceChunk, 0, len(values))
|
out := make([]source.Chunk, 0, len(values))
|
||||||
for _, value := range values {
|
for _, value := range values {
|
||||||
content, err := contentFromEnvelope(value.Content)
|
content, err := contentFromEnvelope(value.Content)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
out = append(out, contracts.SourceChunk{
|
out = append(out, source.Chunk{
|
||||||
ID: value.ID,
|
ID: value.ID,
|
||||||
SourceID: value.SourceID,
|
SourceID: value.SourceID,
|
||||||
Index: value.Index,
|
Index: value.Index,
|
||||||
StartUnitID: value.StartUnitID,
|
Ref: value.Ref,
|
||||||
EndUnitID: value.EndUnitID,
|
Content: content,
|
||||||
Content: content,
|
MediaType: value.Content.MediaType,
|
||||||
MediaType: value.Content.MediaType,
|
Units: cloneSourceUnits(value.Units),
|
||||||
Units: cloneSourceUnits(value.Units),
|
Metadata: cloneMetadata(value.Metadata),
|
||||||
Metadata: cloneMetadata(value.Metadata),
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
return out, nil
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func extractOutputsFromEnvelope(values []extractOutputEnvelope) ([]contracts.ExtractOutput, error) {
|
|
||||||
if len(values) == 0 {
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
out := make([]contracts.ExtractOutput, 0, len(values))
|
|
||||||
for _, value := range values {
|
|
||||||
payload, err := rawPayloadFromEnvelope(value.Payload)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
out = append(out, contracts.ExtractOutput{
|
|
||||||
LaneID: value.LaneID,
|
|
||||||
ExtractorKey: value.ExtractorKey,
|
|
||||||
SourceID: value.SourceID,
|
|
||||||
ChunkID: value.ChunkID,
|
|
||||||
ChunkIndex: value.ChunkIndex,
|
|
||||||
Schema: value.Schema,
|
|
||||||
Payload: payload,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
return out, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func mergeOutputFromEnvelope(value mergeOutputPayload) (contracts.MergeOutput, error) {
|
|
||||||
payload, err := rawPayloadFromEnvelope(value.Payload)
|
|
||||||
if err != nil {
|
|
||||||
return contracts.MergeOutput{}, err
|
|
||||||
}
|
|
||||||
return contracts.MergeOutput{
|
|
||||||
LaneID: value.LaneID,
|
|
||||||
MergerKey: value.MergerKey,
|
|
||||||
SourceID: value.SourceID,
|
|
||||||
Schema: value.Schema,
|
|
||||||
Payload: payload,
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func normalizeOutputFromEnvelope(value normalizeOutputPayload) (contracts.NormalizeOutput, error) {
|
|
||||||
payload, err := rawPayloadFromEnvelope(value.Payload)
|
|
||||||
if err != nil {
|
|
||||||
return contracts.NormalizeOutput{}, err
|
|
||||||
}
|
|
||||||
return contracts.NormalizeOutput{
|
|
||||||
LaneID: value.LaneID,
|
|
||||||
NormalizerKey: value.NormalizerKey,
|
|
||||||
SourceID: value.SourceID,
|
|
||||||
Schema: value.Schema,
|
|
||||||
Payload: payload,
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func rawPayloadFromEnvelope(value binaryEnvelope) (contracts.RawPayload, error) {
|
|
||||||
content, err := contentFromEnvelope(value)
|
|
||||||
if err != nil {
|
|
||||||
return contracts.RawPayload{}, err
|
|
||||||
}
|
|
||||||
return contracts.RawPayload{
|
|
||||||
Content: content,
|
|
||||||
MediaType: value.MediaType,
|
|
||||||
Metadata: cloneMetadata(value.Metadata),
|
|
||||||
Warnings: cloneWarnings(value.Warnings),
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func contentFromEnvelope(value binaryEnvelope) ([]byte, error) {
|
func contentFromEnvelope(value binaryEnvelope) ([]byte, error) {
|
||||||
content, err := base64.StdEncoding.DecodeString(value.ContentBase64)
|
content, err := base64.StdEncoding.DecodeString(value.ContentBase64)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -73,7 +73,11 @@ func (r *WorkspaceRecorder) ChunkRunning(moduleKey string, sourceDigest string)
|
|||||||
return r.writeManifest("chunk/manifest.json", coreworkspace.ChunkManifest{StageManifest: manifest})
|
return r.writeManifest("chunk/manifest.json", coreworkspace.ChunkManifest{StageManifest: manifest})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *WorkspaceRecorder) ChunkSucceeded(moduleKey string, sourceDigest string, chunks []contracts.SourceChunk, warnings []contracts.Warning) error {
|
func (r *WorkspaceRecorder) ChunkSucceeded(moduleKey string, sourceDigest string, chunks []source.Chunk, warnings []contracts.Warning) error {
|
||||||
|
outputDigests, err := chunkOutputDigests(chunks)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("digest chunk checkpoint output: %w", err)
|
||||||
|
}
|
||||||
payload := chunksEnvelope{Chunks: chunkEnvelopes(chunks), Warnings: cloneWarnings(warnings)}
|
payload := chunksEnvelope{Chunks: chunkEnvelopes(chunks), Warnings: cloneWarnings(warnings)}
|
||||||
if err := r.writePayload("chunk/chunks.json", payload); err != nil {
|
if err := r.writePayload("chunk/chunks.json", payload); err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -81,7 +85,7 @@ func (r *WorkspaceRecorder) ChunkSucceeded(moduleKey string, sourceDigest string
|
|||||||
manifest := r.newStageManifest(coreworkspace.StageChunk, coreworkspace.StatusSucceeded)
|
manifest := r.newStageManifest(coreworkspace.StageChunk, coreworkspace.StatusSucceeded)
|
||||||
manifest.ModuleKey = moduleKey
|
manifest.ModuleKey = moduleKey
|
||||||
manifest.DependencyFingerprints = workspaceFingerprints(digestFingerprints("source_document", sourceDigest))
|
manifest.DependencyFingerprints = workspaceFingerprints(digestFingerprints("source_document", sourceDigest))
|
||||||
manifest.OutputDigests = workspaceFingerprints(chunkOutputDigests(chunks))
|
manifest.OutputDigests = workspaceFingerprints(outputDigests)
|
||||||
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
||||||
manifest.CompletedAt = timePtr(r.timestamp())
|
manifest.CompletedAt = timePtr(r.timestamp())
|
||||||
return r.writeManifest("chunk/manifest.json", coreworkspace.ChunkManifest{
|
return r.writeManifest("chunk/manifest.json", coreworkspace.ChunkManifest{
|
||||||
@@ -115,25 +119,17 @@ func (r *WorkspaceRecorder) ExtractRunning(laneID string, moduleKey string, depe
|
|||||||
return r.writeManifest(laneManifestPath("extract", laneID), coreworkspace.ExtractLaneManifest{StageManifest: manifest})
|
return r.writeManifest(laneManifestPath("extract", laneID), coreworkspace.ExtractLaneManifest{StageManifest: manifest})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *WorkspaceRecorder) ExtractSucceeded(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []contracts.ExtractOutput, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
func (r *WorkspaceRecorder) ExtractSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||||
payload := extractOutputsEnvelope{
|
payload := artifactExtractEnvelope{Outputs: artifactCheckpointEnvelopes(outputs), Rejected: cloneRejectedOutputs(rejected), Warnings: cloneWarnings(warnings)}
|
||||||
Outputs: extractOutputEnvelopes(outputs),
|
|
||||||
Rejected: cloneRejectedOutputs(rejected),
|
|
||||||
Warnings: cloneWarnings(warnings),
|
|
||||||
}
|
|
||||||
if err := r.writePayload(lanePayloadPath("extract", laneID, "outputs.json"), payload); err != nil {
|
if err := r.writePayload(lanePayloadPath("extract", laneID, "outputs.json"), payload); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
manifest := r.laneManifest(coreworkspace.StageExtract, statusForRejected(rejected), laneID, moduleKey, dependencies)
|
manifest := r.laneManifest(coreworkspace.StageExtract, statusForRejected(rejected), laneID, moduleKey, dependencies)
|
||||||
manifest.OutputDigests = workspaceFingerprints(rawOutputDigests(extractPayloads(outputs)))
|
manifest.OutputDigests = workspaceFingerprints(artifactOutputDigests(outputs))
|
||||||
manifest.ValidationStatus = validationStatusString(warnings, rejected)
|
manifest.ValidationStatus = validationStatusString(warnings, rejected)
|
||||||
manifest.Rejections = rejectionSummaries(rejected)
|
manifest.Rejections = rejectionSummaries(rejected)
|
||||||
manifest.CompletedAt = timePtr(r.timestamp())
|
manifest.CompletedAt = timePtr(r.timestamp())
|
||||||
return r.writeManifest(laneManifestPath("extract", laneID), coreworkspace.ExtractLaneManifest{
|
return r.writeManifest(laneManifestPath("extract", laneID), coreworkspace.ExtractLaneManifest{StageManifest: manifest, ChunkCount: len(outputs) + len(rejected), OutputCount: len(outputs)})
|
||||||
StageManifest: manifest,
|
|
||||||
ChunkCount: len(outputs) + len(rejected),
|
|
||||||
OutputCount: len(outputs),
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *WorkspaceRecorder) ExtractFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
|
func (r *WorkspaceRecorder) ExtractFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
|
||||||
@@ -149,19 +145,15 @@ func (r *WorkspaceRecorder) MergeRunning(laneID string, moduleKey string, depend
|
|||||||
return r.writeManifest(laneManifestPath("merge", laneID), coreworkspace.MergeLaneManifest{StageManifest: manifest})
|
return r.writeManifest(laneManifestPath("merge", laneID), coreworkspace.MergeLaneManifest{StageManifest: manifest})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *WorkspaceRecorder) MergeSucceeded(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output contracts.MergeOutput, warnings []contracts.Warning) error {
|
func (r *WorkspaceRecorder) MergeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
|
||||||
payload := mergeOutputEnvelope{Output: mergeOutputEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}
|
if err := r.writePayload(lanePayloadPath("merge", laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
|
||||||
if err := r.writePayload(lanePayloadPath("merge", laneID, "output.json"), payload); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
manifest := r.laneManifest(coreworkspace.StageMerge, coreworkspace.StatusSucceeded, laneID, moduleKey, dependencies)
|
manifest := r.laneManifest(coreworkspace.StageMerge, coreworkspace.StatusSucceeded, laneID, moduleKey, dependencies)
|
||||||
manifest.OutputDigests = workspaceFingerprints(rawOutputDigests([]contracts.RawPayload{output.Payload}))
|
manifest.OutputDigests = workspaceFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
|
||||||
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
||||||
manifest.CompletedAt = timePtr(r.timestamp())
|
manifest.CompletedAt = timePtr(r.timestamp())
|
||||||
return r.writeManifest(laneManifestPath("merge", laneID), coreworkspace.MergeLaneManifest{
|
return r.writeManifest(laneManifestPath("merge", laneID), coreworkspace.MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||||
StageManifest: manifest,
|
|
||||||
InputCount: len(dependencies),
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *WorkspaceRecorder) MergeRejected(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
func (r *WorkspaceRecorder) MergeRejected(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||||
@@ -185,13 +177,12 @@ func (r *WorkspaceRecorder) NormalizeRunning(laneID string, moduleKey string, de
|
|||||||
return r.writeManifest(laneManifestPath("normalize", laneID), coreworkspace.NormalizeLaneManifest{StageManifest: manifest})
|
return r.writeManifest(laneManifestPath("normalize", laneID), coreworkspace.NormalizeLaneManifest{StageManifest: manifest})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *WorkspaceRecorder) NormalizeSucceeded(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output contracts.NormalizeOutput, warnings []contracts.Warning) error {
|
func (r *WorkspaceRecorder) NormalizeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
|
||||||
payload := normalizeOutputEnvelope{Output: normalizeOutputEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}
|
if err := r.writePayload(lanePayloadPath("normalize", laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
|
||||||
if err := r.writePayload(lanePayloadPath("normalize", laneID, "output.json"), payload); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
manifest := r.laneManifest(coreworkspace.StageNormalize, coreworkspace.StatusSucceeded, laneID, moduleKey, dependencies)
|
manifest := r.laneManifest(coreworkspace.StageNormalize, coreworkspace.StatusSucceeded, laneID, moduleKey, dependencies)
|
||||||
manifest.OutputDigests = workspaceFingerprints(rawOutputDigests([]contracts.RawPayload{output.Payload}))
|
manifest.OutputDigests = workspaceFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
|
||||||
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
||||||
manifest.CompletedAt = timePtr(r.timestamp())
|
manifest.CompletedAt = timePtr(r.timestamp())
|
||||||
return r.writeManifest(laneManifestPath("normalize", laneID), coreworkspace.NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
return r.writeManifest(laneManifestPath("normalize", laneID), coreworkspace.NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||||
@@ -260,56 +251,13 @@ type chunksEnvelope struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type chunkEnvelope struct {
|
type chunkEnvelope struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
SourceID string `json:"source_id"`
|
SourceID string `json:"source_id"`
|
||||||
Index int `json:"index"`
|
Index int `json:"index"`
|
||||||
StartUnitID int `json:"start_unit_id"`
|
Ref source.SourceRef `json:"ref"`
|
||||||
EndUnitID int `json:"end_unit_id"`
|
Content binaryEnvelope `json:"content"`
|
||||||
Content binaryEnvelope `json:"content"`
|
Units []source.SourceUnit `json:"units,omitempty"`
|
||||||
Units []source.SourceUnit `json:"units,omitempty"`
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type extractOutputsEnvelope struct {
|
|
||||||
Outputs []extractOutputEnvelope `json:"outputs"`
|
|
||||||
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
|
|
||||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type extractOutputEnvelope struct {
|
|
||||||
LaneID string `json:"lane_id"`
|
|
||||||
ExtractorKey string `json:"extractor_key"`
|
|
||||||
SourceID string `json:"source_id"`
|
|
||||||
ChunkID string `json:"chunk_id"`
|
|
||||||
ChunkIndex int `json:"chunk_index"`
|
|
||||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
|
||||||
Payload binaryEnvelope `json:"payload"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type mergeOutputEnvelope struct {
|
|
||||||
Output mergeOutputPayload `json:"output"`
|
|
||||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type mergeOutputPayload struct {
|
|
||||||
LaneID string `json:"lane_id"`
|
|
||||||
MergerKey string `json:"merger_key"`
|
|
||||||
SourceID string `json:"source_id,omitempty"`
|
|
||||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
|
||||||
Payload binaryEnvelope `json:"payload"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type normalizeOutputEnvelope struct {
|
|
||||||
Output normalizeOutputPayload `json:"output"`
|
|
||||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type normalizeOutputPayload struct {
|
|
||||||
LaneID string `json:"lane_id"`
|
|
||||||
NormalizerKey string `json:"normalizer_key"`
|
|
||||||
SourceID string `json:"source_id,omitempty"`
|
|
||||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
|
||||||
Payload binaryEnvelope `json:"payload"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type binaryEnvelope struct {
|
type binaryEnvelope struct {
|
||||||
@@ -320,74 +268,70 @@ type binaryEnvelope struct {
|
|||||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func chunkEnvelopes(chunks []contracts.SourceChunk) []chunkEnvelope {
|
type artifactCheckpointEnvelope struct {
|
||||||
|
LaneID string `json:"lane_id"`
|
||||||
|
ModuleKey string `json:"module_key"`
|
||||||
|
SourceID string `json:"source_id,omitempty"`
|
||||||
|
ChunkID string `json:"chunk_id,omitempty"`
|
||||||
|
ChunkIndex int `json:"chunk_index,omitempty"`
|
||||||
|
ChunkRef source.SourceRef `json:"chunk_ref,omitempty"`
|
||||||
|
Kind contracts.ArtifactKind `json:"artifact_kind"`
|
||||||
|
Schema contracts.ArtifactSchema `json:"schema"`
|
||||||
|
SchemaDigest string `json:"schema_digest"`
|
||||||
|
Content binaryEnvelope `json:"content"`
|
||||||
|
}
|
||||||
|
type artifactExtractEnvelope struct {
|
||||||
|
Outputs []artifactCheckpointEnvelope `json:"outputs"`
|
||||||
|
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
|
||||||
|
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||||
|
}
|
||||||
|
type artifactSingleEnvelope struct {
|
||||||
|
Output artifactCheckpointEnvelope `json:"output"`
|
||||||
|
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func artifactCheckpointEnvelopeFromOutput(output pipeline.CheckpointArtifact) artifactCheckpointEnvelope {
|
||||||
|
schema := contracts.CloneArtifactSchema(output.Artifact.Schema)
|
||||||
|
schema.JSONSchema = nil
|
||||||
|
return artifactCheckpointEnvelope{LaneID: output.LaneID, ModuleKey: output.ModuleKey, SourceID: output.SourceID, ChunkID: output.ChunkID, ChunkIndex: output.ChunkIndex, ChunkRef: output.ChunkRef, Kind: output.Artifact.Kind, Schema: schema, SchemaDigest: output.SchemaDigest, Content: binaryEnvelopeFromContent(output.Artifact.Content, output.Artifact.MediaType, output.Artifact.Metadata, nil)}
|
||||||
|
}
|
||||||
|
func artifactCheckpointEnvelopes(outputs []pipeline.CheckpointArtifact) []artifactCheckpointEnvelope {
|
||||||
|
if len(outputs) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]artifactCheckpointEnvelope, 0, len(outputs))
|
||||||
|
for _, v := range outputs {
|
||||||
|
out = append(out, artifactCheckpointEnvelopeFromOutput(v))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
func artifactOutputDigests(outputs []pipeline.CheckpointArtifact) []pipeline.CheckpointFingerprint {
|
||||||
|
values := make([]pipeline.CheckpointFingerprint, 0, len(outputs))
|
||||||
|
for i, v := range outputs {
|
||||||
|
values = append(values, pipeline.CheckpointFingerprint{Name: fmt.Sprintf("artifact[%d]", i), Value: contentDigest(v.Artifact.Content)})
|
||||||
|
}
|
||||||
|
return normalizeFingerprints(values)
|
||||||
|
}
|
||||||
|
|
||||||
|
func chunkEnvelopes(chunks []source.Chunk) []chunkEnvelope {
|
||||||
if len(chunks) == 0 {
|
if len(chunks) == 0 {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
out := make([]chunkEnvelope, 0, len(chunks))
|
out := make([]chunkEnvelope, 0, len(chunks))
|
||||||
for _, chunk := range chunks {
|
for _, chunk := range chunks {
|
||||||
out = append(out, chunkEnvelope{
|
out = append(out, chunkEnvelope{
|
||||||
ID: chunk.ID,
|
ID: chunk.ID,
|
||||||
SourceID: chunk.SourceID,
|
SourceID: chunk.SourceID,
|
||||||
Index: chunk.Index,
|
Index: chunk.Index,
|
||||||
StartUnitID: chunk.StartUnitID,
|
Ref: chunk.Ref,
|
||||||
EndUnitID: chunk.EndUnitID,
|
Content: binaryEnvelopeFromContent(chunk.Content, chunk.MediaType, chunk.Metadata, nil),
|
||||||
Content: binaryEnvelopeFromContent(chunk.Content, chunk.MediaType, chunk.Metadata, nil),
|
Units: cloneSourceUnits(chunk.Units),
|
||||||
Units: cloneSourceUnits(chunk.Units),
|
Metadata: cloneMetadata(chunk.Metadata),
|
||||||
Metadata: cloneMetadata(chunk.Metadata),
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func extractOutputEnvelopes(outputs []contracts.ExtractOutput) []extractOutputEnvelope {
|
|
||||||
if len(outputs) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
out := make([]extractOutputEnvelope, 0, len(outputs))
|
|
||||||
for _, output := range outputs {
|
|
||||||
out = append(out, extractOutputEnvelope{
|
|
||||||
LaneID: output.LaneID,
|
|
||||||
ExtractorKey: output.ExtractorKey,
|
|
||||||
SourceID: output.SourceID,
|
|
||||||
ChunkID: output.ChunkID,
|
|
||||||
ChunkIndex: output.ChunkIndex,
|
|
||||||
Schema: schemaEnvelope(output.Schema),
|
|
||||||
Payload: binaryEnvelopeFromPayload(output.Payload),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
func mergeOutputEnvelopeFromOutput(output contracts.MergeOutput) mergeOutputPayload {
|
|
||||||
return mergeOutputPayload{
|
|
||||||
LaneID: output.LaneID,
|
|
||||||
MergerKey: output.MergerKey,
|
|
||||||
SourceID: output.SourceID,
|
|
||||||
Schema: schemaEnvelope(output.Schema),
|
|
||||||
Payload: binaryEnvelopeFromPayload(output.Payload),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func normalizeOutputEnvelopeFromOutput(output contracts.NormalizeOutput) normalizeOutputPayload {
|
|
||||||
return normalizeOutputPayload{
|
|
||||||
LaneID: output.LaneID,
|
|
||||||
NormalizerKey: output.NormalizerKey,
|
|
||||||
SourceID: output.SourceID,
|
|
||||||
Schema: schemaEnvelope(output.Schema),
|
|
||||||
Payload: binaryEnvelopeFromPayload(output.Payload),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func schemaEnvelope(schema contracts.ResponseSchema) contracts.ResponseSchema {
|
|
||||||
schema.JSONSchema = nil
|
|
||||||
return schema
|
|
||||||
}
|
|
||||||
|
|
||||||
func binaryEnvelopeFromPayload(payload contracts.RawPayload) binaryEnvelope {
|
|
||||||
return binaryEnvelopeFromContent(payload.Content, payload.MediaType, payload.Metadata, payload.Warnings)
|
|
||||||
}
|
|
||||||
|
|
||||||
func binaryEnvelopeFromContent(content []byte, mediaType string, metadata map[string]any, warnings []contracts.Warning) binaryEnvelope {
|
func binaryEnvelopeFromContent(content []byte, mediaType string, metadata map[string]any, warnings []contracts.Warning) binaryEnvelope {
|
||||||
return binaryEnvelope{
|
return binaryEnvelope{
|
||||||
ContentBase64: base64.StdEncoding.EncodeToString(content),
|
ContentBase64: base64.StdEncoding.EncodeToString(content),
|
||||||
@@ -414,6 +358,7 @@ func cloneSourceUnits(units []source.SourceUnit) []source.SourceUnit {
|
|||||||
ID: unit.ID,
|
ID: unit.ID,
|
||||||
Kind: unit.Kind,
|
Kind: unit.Kind,
|
||||||
Text: unit.Text,
|
Text: unit.Text,
|
||||||
|
Ref: unit.Ref,
|
||||||
Metadata: cloneMetadata(unit.Metadata),
|
Metadata: cloneMetadata(unit.Metadata),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -445,37 +390,19 @@ func cloneMetadata(metadata map[string]any) map[string]any {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func rawOutputDigests(payloads []contracts.RawPayload) []pipeline.CheckpointFingerprint {
|
func chunkOutputDigests(chunks []source.Chunk) ([]pipeline.CheckpointFingerprint, error) {
|
||||||
values := make([]pipeline.CheckpointFingerprint, 0, len(payloads))
|
|
||||||
for i, payload := range payloads {
|
|
||||||
values = append(values, pipeline.CheckpointFingerprint{
|
|
||||||
Name: fmt.Sprintf("payload[%d]", i),
|
|
||||||
Value: contentDigest(payload.Content),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
return normalizeFingerprints(values)
|
|
||||||
}
|
|
||||||
|
|
||||||
func extractPayloads(outputs []contracts.ExtractOutput) []contracts.RawPayload {
|
|
||||||
if len(outputs) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
payloads := make([]contracts.RawPayload, 0, len(outputs))
|
|
||||||
for _, output := range outputs {
|
|
||||||
payloads = append(payloads, output.Payload)
|
|
||||||
}
|
|
||||||
return payloads
|
|
||||||
}
|
|
||||||
|
|
||||||
func chunkOutputDigests(chunks []contracts.SourceChunk) []pipeline.CheckpointFingerprint {
|
|
||||||
values := make([]pipeline.CheckpointFingerprint, 0, len(chunks))
|
values := make([]pipeline.CheckpointFingerprint, 0, len(chunks))
|
||||||
for _, chunk := range chunks {
|
for _, chunk := range chunks {
|
||||||
|
digest, err := source.DigestChunk(chunk)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("chunk %q: %w", chunk.ID, err)
|
||||||
|
}
|
||||||
values = append(values, pipeline.CheckpointFingerprint{
|
values = append(values, pipeline.CheckpointFingerprint{
|
||||||
Name: chunk.ID,
|
Name: chunk.ID,
|
||||||
Value: contentDigest(chunk.Content),
|
Value: digest,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
return normalizeFingerprints(values)
|
return normalizeFingerprints(values), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func digestFingerprints(name string, digest string) []pipeline.CheckpointFingerprint {
|
func digestFingerprints(name string, digest string) []pipeline.CheckpointFingerprint {
|
||||||
|
|||||||
@@ -22,18 +22,17 @@ func TestWorkspaceRecorderWritesSuccessfulCheckpointFiles(t *testing.T) {
|
|||||||
Kind: "document",
|
Kind: "document",
|
||||||
Format: "text/plain",
|
Format: "text/plain",
|
||||||
Digest: "sha256:source",
|
Digest: "sha256:source",
|
||||||
Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "hello"}},
|
Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "hello", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1}}},
|
||||||
}
|
}
|
||||||
chunks := []contracts.SourceChunk{
|
chunks := []source.Chunk{
|
||||||
{
|
{
|
||||||
ID: "chunk-1",
|
ID: "chunk-1",
|
||||||
SourceID: "source-1",
|
SourceID: "source-1",
|
||||||
Index: 0,
|
Index: 0,
|
||||||
StartUnitID: 1,
|
Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1},
|
||||||
EndUnitID: 1,
|
Content: []byte("chunk content"),
|
||||||
Content: []byte("chunk content"),
|
MediaType: "text/plain",
|
||||||
MediaType: "text/plain",
|
Units: doc.Units,
|
||||||
Units: doc.Units,
|
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -80,96 +79,43 @@ func TestWorkspaceRecorderWritesSuccessfulCheckpointFiles(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestWorkspaceLoaderReusesSuccessfulCheckpointFiles(t *testing.T) {
|
func TestWorkspaceArtifactCheckpointsRoundTripCodecIdentityAndBytes(t *testing.T) {
|
||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
recorder := newTestRecorder(t, root)
|
recorder := newTestRecorder(t, root)
|
||||||
loader := &WorkspaceLoader{root: root}
|
loader := &WorkspaceLoader{root: root}
|
||||||
doc := &source.SourceDocument{
|
schema := contracts.ArtifactSchema{ID: "dnd.spell_response", Name: "spell response", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
|
||||||
ID: "source-1",
|
artifact := contracts.SerializedArtifact{Kind: "dnd.spells", Schema: schema, MediaType: "application/json", Content: []byte(`{"spell_casts":[]}`), Metadata: map[string]any{"spell_cast_count": float64(0)}}
|
||||||
Kind: "document",
|
stored := pipeline.CheckpointArtifact{LaneID: "spells", ModuleKey: "dnd/spells", SourceID: "source-1", ChunkID: "chunk-1", ChunkIndex: 2, ChunkRef: source.SourceRef{SourceID: "source-1", StartUnitID: 4, EndUnitID: 8}, Artifact: artifact, SchemaDigest: contracts.DigestArtifactSchema(schema)}
|
||||||
Format: "text/plain",
|
|
||||||
Digest: "sha256:source",
|
|
||||||
Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "hello"}},
|
|
||||||
}
|
|
||||||
chunks := []contracts.SourceChunk{
|
|
||||||
{
|
|
||||||
ID: "chunk-1",
|
|
||||||
SourceID: "source-1",
|
|
||||||
Index: 0,
|
|
||||||
StartUnitID: 1,
|
|
||||||
EndUnitID: 1,
|
|
||||||
Content: []byte("chunk content"),
|
|
||||||
MediaType: "text/plain",
|
|
||||||
Units: doc.Units,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
extractOutput := contracts.ExtractOutput{
|
|
||||||
LaneID: "spells",
|
|
||||||
ExtractorKey: "dnd/spells",
|
|
||||||
SourceID: doc.ID,
|
|
||||||
ChunkID: "chunk-1",
|
|
||||||
Payload: contracts.RawPayload{
|
|
||||||
Content: []byte(`{"spell":"cure wounds"}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
mergeOutput := contracts.MergeOutput{
|
|
||||||
LaneID: "spells",
|
|
||||||
MergerKey: "appendorder",
|
|
||||||
SourceID: doc.ID,
|
|
||||||
Payload: contracts.RawPayload{
|
|
||||||
Content: []byte(`{"merged":true}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
normalizeOutput := contracts.NormalizeOutput{
|
|
||||||
LaneID: "spells",
|
|
||||||
NormalizerKey: "noop",
|
|
||||||
SourceID: doc.ID,
|
|
||||||
Payload: contracts.RawPayload{
|
|
||||||
Content: []byte(`{"normalized":true}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := recorder.SourceSucceeded("seriatim", doc); err != nil {
|
|
||||||
t.Fatalf("SourceSucceeded: %v", err)
|
|
||||||
}
|
|
||||||
if err := recorder.ChunkSucceeded("generic", doc.Digest, chunks, nil); err != nil {
|
|
||||||
t.Fatalf("ChunkSucceeded: %v", err)
|
|
||||||
}
|
|
||||||
extractDeps := []pipeline.CheckpointFingerprint{{Name: "chunks", Value: "sha256:chunks"}}
|
extractDeps := []pipeline.CheckpointFingerprint{{Name: "chunks", Value: "sha256:chunks"}}
|
||||||
if err := recorder.ExtractSucceeded("spells", "dnd/spells", extractDeps, []contracts.ExtractOutput{extractOutput}, nil, nil); err != nil {
|
if err := recorder.ExtractSucceeded("spells", "dnd/spells", extractDeps, []pipeline.CheckpointArtifact{stored}, nil, nil); err != nil {
|
||||||
t.Fatalf("ExtractSucceeded: %v", err)
|
t.Fatalf("ExtractSucceeded: %v", err)
|
||||||
}
|
}
|
||||||
mergeDeps := rawOutputDigests([]contracts.RawPayload{extractOutput.Payload})
|
extracted, decision := loader.Extract("spells", "dnd/spells", extractDeps)
|
||||||
if err := recorder.MergeSucceeded("spells", "appendorder", mergeDeps, mergeOutput, nil); err != nil {
|
if !decision.Reused || len(extracted.Outputs) != 1 {
|
||||||
t.Fatalf("MergeSucceeded: %v", err)
|
t.Fatalf("extract decision=%#v checkpoint=%#v, want reused", decision, extracted)
|
||||||
}
|
}
|
||||||
normalizeDeps := rawOutputDigests([]contracts.RawPayload{mergeOutput.Payload})
|
got := extracted.Outputs[0]
|
||||||
if err := recorder.NormalizeSucceeded("spells", "noop", normalizeDeps, normalizeOutput, nil); err != nil {
|
if got.Artifact.Kind != artifact.Kind || got.Artifact.Schema.ID != schema.ID || got.Artifact.Schema.Version != schema.Version || got.SchemaDigest != stored.SchemaDigest || string(got.Artifact.Content) != string(artifact.Content) || got.ChunkRef != stored.ChunkRef {
|
||||||
t.Fatalf("NormalizeSucceeded: %v", err)
|
t.Fatalf("artifact checkpoint = %#v, want codec identity, bytes, and provenance", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
sourceCheckpoint, decision := loader.Source("seriatim")
|
mergeDeps := artifactOutputDigests([]pipeline.CheckpointArtifact{stored})
|
||||||
if !decision.Reused || sourceCheckpoint.Document.ID != "source-1" {
|
if err := recorder.MergeSucceeded("spells", "merge", mergeDeps, stored, nil); err != nil {
|
||||||
t.Fatalf("source decision = %#v checkpoint=%#v, want reused", decision, sourceCheckpoint)
|
t.Fatalf("MergeSucceeded: %v", err)
|
||||||
}
|
}
|
||||||
chunkCheckpoint, decision := loader.Chunk("generic", doc.Digest)
|
merged, decision := loader.Merge("spells", "merge", mergeDeps)
|
||||||
if !decision.Reused || len(chunkCheckpoint.Chunks) != 1 || string(chunkCheckpoint.Chunks[0].Content) != "chunk content" {
|
if !decision.Reused || string(merged.Output.Artifact.Content) != string(artifact.Content) {
|
||||||
t.Fatalf("chunk decision = %#v checkpoint=%#v, want reused", decision, chunkCheckpoint)
|
t.Fatalf("merge decision=%#v checkpoint=%#v, want reused", decision, merged)
|
||||||
}
|
}
|
||||||
extractCheckpoint, decision := loader.Extract("spells", "dnd/spells", extractDeps)
|
|
||||||
if !decision.Reused || len(extractCheckpoint.Outputs) != 1 || string(extractCheckpoint.Outputs[0].Payload.Content) != `{"spell":"cure wounds"}` {
|
normalizeDeps := artifactOutputDigests([]pipeline.CheckpointArtifact{stored})
|
||||||
t.Fatalf("extract decision = %#v checkpoint=%#v, want reused", decision, extractCheckpoint)
|
if err := recorder.NormalizeSucceeded("spells", "normalize", normalizeDeps, stored, nil); err != nil {
|
||||||
|
t.Fatalf("NormalizeSucceeded: %v", err)
|
||||||
}
|
}
|
||||||
mergeCheckpoint, decision := loader.Merge("spells", "appendorder", mergeDeps)
|
normalized, decision := loader.Normalize("spells", "normalize", normalizeDeps)
|
||||||
if !decision.Reused || string(mergeCheckpoint.Output.Payload.Content) != `{"merged":true}` {
|
if !decision.Reused || normalized.Output.SchemaDigest != stored.SchemaDigest {
|
||||||
t.Fatalf("merge decision = %#v checkpoint=%#v, want reused", decision, mergeCheckpoint)
|
t.Fatalf("normalize decision=%#v checkpoint=%#v, want reused", decision, normalized)
|
||||||
}
|
|
||||||
normalizeCheckpoint, decision := loader.Normalize("spells", "noop", normalizeDeps)
|
|
||||||
if !decision.Reused || string(normalizeCheckpoint.Output.Payload.Content) != `{"normalized":true}` {
|
|
||||||
t.Fatalf("normalize decision = %#v checkpoint=%#v, want reused", decision, normalizeCheckpoint)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -184,7 +130,7 @@ func TestWorkspaceLoaderInvalidatesMissingCorruptAndMismatchedCheckpoints(t *tes
|
|||||||
t.Run("dependency mismatch", func(t *testing.T) {
|
t.Run("dependency mismatch", func(t *testing.T) {
|
||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
recorder := newTestRecorder(t, root)
|
recorder := newTestRecorder(t, root)
|
||||||
chunks := []contracts.SourceChunk{{
|
chunks := []source.Chunk{{
|
||||||
ID: "chunk-1",
|
ID: "chunk-1",
|
||||||
SourceID: "source-1",
|
SourceID: "source-1",
|
||||||
Content: []byte("chunk content"),
|
Content: []byte("chunk content"),
|
||||||
@@ -199,10 +145,41 @@ func TestWorkspaceLoaderInvalidatesMissingCorruptAndMismatchedCheckpoints(t *tes
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
t.Run("incompatible workspace schema remains untouched", func(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
recorder := newTestRecorder(t, root)
|
||||||
|
doc := &source.SourceDocument{
|
||||||
|
ID: "source-1", Kind: "document", Format: "text/plain", Digest: "sha256:source",
|
||||||
|
Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "hello", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1}}},
|
||||||
|
}
|
||||||
|
if err := recorder.SourceSucceeded("seriatim", doc); err != nil {
|
||||||
|
t.Fatalf("SourceSucceeded: %v", err)
|
||||||
|
}
|
||||||
|
manifestPath := filepath.Join(root, "source", "manifest.json")
|
||||||
|
manifest := strings.Replace(string(readFile(t, manifestPath)), coreworkspace.WorkspaceSchemaVersion, coreworkspace.WorkspaceSchemaVersionV1, 1)
|
||||||
|
if err := os.WriteFile(manifestPath, []byte(manifest), 0o644); err != nil {
|
||||||
|
t.Fatalf("write legacy manifest: %v", err)
|
||||||
|
}
|
||||||
|
beforeManifest := readFile(t, manifestPath)
|
||||||
|
payloadPath := filepath.Join(root, "source", "source-document.json")
|
||||||
|
beforePayload := readFile(t, payloadPath)
|
||||||
|
|
||||||
|
loader := &WorkspaceLoader{root: root}
|
||||||
|
if _, decision := loader.Source("seriatim"); decision.Reused || !strings.Contains(decision.Reason, "incompatible") || !strings.Contains(decision.Reason, coreworkspace.WorkspaceSchemaVersionV1) {
|
||||||
|
t.Fatalf("decision = %#v, want incompatible legacy schema invalidation", decision)
|
||||||
|
}
|
||||||
|
if got := readFile(t, manifestPath); string(got) != string(beforeManifest) {
|
||||||
|
t.Fatal("legacy manifest changed during reuse decision")
|
||||||
|
}
|
||||||
|
if got := readFile(t, payloadPath); string(got) != string(beforePayload) {
|
||||||
|
t.Fatal("legacy payload changed during reuse decision")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run("corrupt payload", func(t *testing.T) {
|
t.Run("corrupt payload", func(t *testing.T) {
|
||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
recorder := newTestRecorder(t, root)
|
recorder := newTestRecorder(t, root)
|
||||||
chunks := []contracts.SourceChunk{{
|
chunks := []source.Chunk{{
|
||||||
ID: "chunk-1",
|
ID: "chunk-1",
|
||||||
SourceID: "source-1",
|
SourceID: "source-1",
|
||||||
Content: []byte("chunk content"),
|
Content: []byte("chunk content"),
|
||||||
@@ -286,23 +263,20 @@ func TestWorkspaceRecorderRecordsFailedStages(t *testing.T) {
|
|||||||
func TestWorkspaceRecorderRecordsWarningOnlyValidation(t *testing.T) {
|
func TestWorkspaceRecorderRecordsWarningOnlyValidation(t *testing.T) {
|
||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
recorder := newTestRecorder(t, root)
|
recorder := newTestRecorder(t, root)
|
||||||
output := contracts.NormalizeOutput{
|
schema := contracts.ArtifactSchema{ID: "test.artifact", Name: "test_artifact", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
|
||||||
LaneID: "spells",
|
output := pipeline.CheckpointArtifact{
|
||||||
NormalizerKey: "noop",
|
LaneID: "events", ModuleKey: "noop", SourceID: "source-1",
|
||||||
SourceID: "source-1",
|
Artifact: contracts.SerializedArtifact{Kind: "test/artifact", Schema: schema, MediaType: "application/json", Content: []byte(`{"ok":true}`)},
|
||||||
Payload: contracts.RawPayload{
|
SchemaDigest: contracts.DigestArtifactSchema(schema),
|
||||||
Content: []byte(`{"ok":true}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
},
|
|
||||||
}
|
}
|
||||||
warnings := []contracts.Warning{{ReasonCode: "note", Message: "warning"}}
|
warnings := []contracts.Warning{{ReasonCode: "note", Message: "warning"}}
|
||||||
|
|
||||||
if err := recorder.NormalizeSucceeded("spells", "noop", nil, output, warnings); err != nil {
|
if err := recorder.NormalizeSucceeded("events", "noop", nil, output, warnings); err != nil {
|
||||||
t.Fatalf("NormalizeSucceeded: %v", err)
|
t.Fatalf("NormalizeSucceeded: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
var manifest coreworkspace.NormalizeLaneManifest
|
var manifest coreworkspace.NormalizeLaneManifest
|
||||||
readJSON(t, filepath.Join(root, "normalize", "spells", "manifest.json"), &manifest)
|
readJSON(t, filepath.Join(root, "normalize", "events", "manifest.json"), &manifest)
|
||||||
if manifest.Status != coreworkspace.StatusSucceeded || manifest.ValidationStatus != "approved_with_warnings" {
|
if manifest.Status != coreworkspace.StatusSucceeded || manifest.ValidationStatus != "approved_with_warnings" {
|
||||||
t.Fatalf("normalize manifest status = %q validation=%q", manifest.Status, manifest.ValidationStatus)
|
t.Fatalf("normalize manifest status = %q validation=%q", manifest.Status, manifest.ValidationStatus)
|
||||||
}
|
}
|
||||||
|
|||||||
81
internal/framework/contracts/artifact.go
Normal file
81
internal/framework/contracts/artifact.go
Normal file
@@ -0,0 +1,81 @@
|
|||||||
|
package contracts
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ArtifactKind is the stable logical identity of a domain artifact.
|
||||||
|
type ArtifactKind string
|
||||||
|
|
||||||
|
// ArtifactSchema describes the durable representation owned by an artifact
|
||||||
|
// codec. JSONSchema is cloned whenever framework ownership changes.
|
||||||
|
type ArtifactSchema struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Name string `json:"name"`
|
||||||
|
Version string `json:"version"`
|
||||||
|
JSONSchema []byte `json:"-"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// SerializedArtifact is the domain-neutral representation of a typed
|
||||||
|
// artifact at an explicit serialization boundary.
|
||||||
|
type SerializedArtifact struct {
|
||||||
|
Kind ArtifactKind `json:"kind"`
|
||||||
|
Schema ArtifactSchema `json:"schema"`
|
||||||
|
MediaType string `json:"media_type"`
|
||||||
|
Content []byte `json:"-"`
|
||||||
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// SerializedOutput associates a domain-neutral artifact with the pipeline
|
||||||
|
// operation that produced it. Provenance remains outside codec-owned bytes.
|
||||||
|
type SerializedOutput struct {
|
||||||
|
LaneID string `json:"lane_id"`
|
||||||
|
NormalizerKey string `json:"normalizer_key"`
|
||||||
|
SourceID string `json:"source_id,omitempty"`
|
||||||
|
Artifact SerializedArtifact `json:"artifact"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ArtifactCodec owns the stable encoding for one concrete artifact type.
|
||||||
|
type ArtifactCodec[T any] interface {
|
||||||
|
Kind() ArtifactKind
|
||||||
|
Schema() ArtifactSchema
|
||||||
|
MediaType() string
|
||||||
|
Encode(T) ([]byte, error)
|
||||||
|
Decode([]byte) (T, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// DigestArtifactSchema returns the SHA-256 digest of the exact JSON Schema
|
||||||
|
// bytes. Schema formatting is therefore part of the registered identity.
|
||||||
|
func DigestArtifactSchema(schema ArtifactSchema) string {
|
||||||
|
sum := sha256.Sum256(schema.JSONSchema)
|
||||||
|
return "sha256:" + hex.EncodeToString(sum[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
func CloneArtifactSchema(schema ArtifactSchema) ArtifactSchema {
|
||||||
|
schema.JSONSchema = append([]byte(nil), schema.JSONSchema...)
|
||||||
|
return schema
|
||||||
|
}
|
||||||
|
|
||||||
|
func CloneSerializedArtifact(artifact SerializedArtifact) SerializedArtifact {
|
||||||
|
artifact.Schema = CloneArtifactSchema(artifact.Schema)
|
||||||
|
artifact.Content = append([]byte(nil), artifact.Content...)
|
||||||
|
artifact.Metadata = cloneArtifactMetadata(artifact.Metadata)
|
||||||
|
return artifact
|
||||||
|
}
|
||||||
|
|
||||||
|
func CloneSerializedOutput(output SerializedOutput) SerializedOutput {
|
||||||
|
output.Artifact = CloneSerializedArtifact(output.Artifact)
|
||||||
|
return output
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneArtifactMetadata(metadata map[string]any) map[string]any {
|
||||||
|
if len(metadata) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make(map[string]any, len(metadata))
|
||||||
|
for key, value := range metadata {
|
||||||
|
out[key] = value
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
45
internal/framework/contracts/artifact_test.go
Normal file
45
internal/framework/contracts/artifact_test.go
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
package contracts
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestDigestArtifactSchemaUsesExactBytes(t *testing.T) {
|
||||||
|
first := ArtifactSchema{JSONSchema: []byte(`{"type":"object"}`)}
|
||||||
|
second := ArtifactSchema{JSONSchema: []byte("{\n \"type\": \"object\"\n}")}
|
||||||
|
|
||||||
|
if got := DigestArtifactSchema(first); got != "sha256:a2c799262a3ce3c19ef5cdd983bf3d12b43ab3c426227091b909dcb7054738c0" {
|
||||||
|
t.Fatalf("DigestArtifactSchema() = %q, want stable SHA-256", got)
|
||||||
|
}
|
||||||
|
if DigestArtifactSchema(first) == DigestArtifactSchema(second) {
|
||||||
|
t.Fatal("schema digests match for different exact bytes")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestArtifactCloneHelpersOwnSlicesAndMaps(t *testing.T) {
|
||||||
|
schema := ArtifactSchema{ID: "notes.v1", Name: "notes", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
|
||||||
|
artifact := SerializedArtifact{
|
||||||
|
Kind: "test/notes",
|
||||||
|
Schema: schema,
|
||||||
|
MediaType: "application/json",
|
||||||
|
Content: []byte(`{"items":["one"]}`),
|
||||||
|
Metadata: map[string]any{"origin": "test"},
|
||||||
|
}
|
||||||
|
|
||||||
|
clonedSchema := CloneArtifactSchema(schema)
|
||||||
|
cloned := CloneSerializedArtifact(artifact)
|
||||||
|
schema.JSONSchema[0] = '['
|
||||||
|
artifact.Content[0] = '['
|
||||||
|
artifact.Metadata["origin"] = "changed"
|
||||||
|
|
||||||
|
if string(clonedSchema.JSONSchema) != `{"type":"object"}` {
|
||||||
|
t.Fatalf("cloned schema = %q, want original bytes", clonedSchema.JSONSchema)
|
||||||
|
}
|
||||||
|
if string(cloned.Schema.JSONSchema) != `{"type":"object"}` {
|
||||||
|
t.Fatalf("serialized artifact schema = %q, want original bytes", cloned.Schema.JSONSchema)
|
||||||
|
}
|
||||||
|
if string(cloned.Content) != `{"items":["one"]}` {
|
||||||
|
t.Fatalf("cloned content = %q, want original bytes", cloned.Content)
|
||||||
|
}
|
||||||
|
if cloned.Metadata["origin"] != "test" {
|
||||||
|
t.Fatalf("cloned metadata = %#v, want independent map", cloned.Metadata)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,298 +0,0 @@
|
|||||||
package contracts_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
|
||||||
)
|
|
||||||
|
|
||||||
var _ contracts.InputAdapter = compositionAdapter{}
|
|
||||||
var _ contracts.Chunker = compositionChunker{}
|
|
||||||
var _ contracts.Extractor = compositionExtractor{}
|
|
||||||
var _ contracts.Merger = compositionMerger{}
|
|
||||||
var _ contracts.Normalizer = compositionNormalizer{}
|
|
||||||
var _ contracts.Validator = compositionValidator{}
|
|
||||||
var _ contracts.StructuredLLMClient = compositionLLMClient{}
|
|
||||||
var _ contracts.OutputEncoder = compositionOutputEncoder{}
|
|
||||||
|
|
||||||
func TestContractsComposeAcrossPackages(t *testing.T) {
|
|
||||||
ctx := context.Background()
|
|
||||||
adapter := compositionAdapter{}
|
|
||||||
chunker := compositionChunker{}
|
|
||||||
extractor := compositionExtractor{}
|
|
||||||
merger := compositionMerger{}
|
|
||||||
normalizer := compositionNormalizer{}
|
|
||||||
encoder := compositionOutputEncoder{}
|
|
||||||
|
|
||||||
doc, err := adapter.Parse(ctx, contracts.ParseRequest{SourceID: "source-1"})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Parse() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if err := source.ValidateDocument(doc); err != nil {
|
|
||||||
t.Fatalf("ValidateDocument() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
chunking, err := chunker.Chunk(ctx, contracts.ChunkRequest{
|
|
||||||
Source: doc,
|
|
||||||
LLMClient: compositionLLMClient{},
|
|
||||||
Metadata: map[string]any{"max_units": 2},
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if len(chunking.Chunks) != 1 {
|
|
||||||
t.Fatalf("len(Chunks) = %d, want 1", len(chunking.Chunks))
|
|
||||||
}
|
|
||||||
|
|
||||||
extraction, err := extractor.Extract(ctx, contracts.ExtractionRequest{
|
|
||||||
Source: doc,
|
|
||||||
Chunk: &chunking.Chunks[0],
|
|
||||||
AmbientContext: map[string]any{"synopsis": "example synopsis"},
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Extract() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if extraction.Output.Payload.MediaType != "application/json" {
|
|
||||||
t.Fatalf("extract media type = %q, want application/json", extraction.Output.Payload.MediaType)
|
|
||||||
}
|
|
||||||
|
|
||||||
merge, err := merger.Merge(ctx, contracts.MergeRequest{
|
|
||||||
Source: doc,
|
|
||||||
LaneID: "generic-lane",
|
|
||||||
ExtractOutputs: []contracts.ExtractOutput{extraction.Output},
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Merge() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if string(merge.Output.Payload.Content) != `{"value":"example"}` {
|
|
||||||
t.Fatalf("merge output = %s, want extract payload", merge.Output.Payload.Content)
|
|
||||||
}
|
|
||||||
|
|
||||||
normalize, err := normalizer.Normalize(ctx, contracts.NormalizeRequest{
|
|
||||||
Source: doc,
|
|
||||||
LaneID: "generic-lane",
|
|
||||||
MergeOutput: merge.Output,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Normalize() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if string(normalize.Output.Payload.Content) != `{"value":"example"}` {
|
|
||||||
t.Fatalf("normalize output = %s, want merge payload", normalize.Output.Payload.Content)
|
|
||||||
}
|
|
||||||
|
|
||||||
output, err := encoder.Encode(ctx, contracts.OutputRequest{
|
|
||||||
Manifest: artifacts.RunManifest{RunID: "run-1"},
|
|
||||||
NormalizeOutputs: []contracts.NormalizeOutput{normalize.Output},
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Encode() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if len(output.Files) != 1 {
|
|
||||||
t.Fatalf("len(Files) = %d, want 1", len(output.Files))
|
|
||||||
}
|
|
||||||
if output.Files[0].ContentType != "application/json" {
|
|
||||||
t.Fatalf("ContentType = %q, want application/json", output.Files[0].ContentType)
|
|
||||||
}
|
|
||||||
if len(output.Files[0].Bytes) == 0 {
|
|
||||||
t.Fatal("len(Bytes) = 0, want encoded bytes")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
type compositionAdapter struct{}
|
|
||||||
|
|
||||||
func (adapter compositionAdapter) Key() string {
|
|
||||||
return "generic-input"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (adapter compositionAdapter) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
|
|
||||||
return &source.SourceDocument{
|
|
||||||
ID: req.SourceID,
|
|
||||||
Kind: "document",
|
|
||||||
Format: "text/plain",
|
|
||||||
Digest: "sha256:abc123",
|
|
||||||
Units: []source.SourceUnit{
|
|
||||||
{ID: 1, Kind: "unit", Text: "First source unit."},
|
|
||||||
{ID: 2, Kind: "unit", Text: "Second source unit."},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type compositionChunker struct{}
|
|
||||||
|
|
||||||
func (chunker compositionChunker) Key() string {
|
|
||||||
return "generic-chunker"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (chunker compositionChunker) ReferenceSlots() []contracts.ReferenceSlot {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (chunker compositionChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
|
||||||
if req.Source == nil {
|
|
||||||
return contracts.ChunkResult{}, errors.New("source document is required")
|
|
||||||
}
|
|
||||||
if req.LLMClient == nil {
|
|
||||||
return contracts.ChunkResult{}, errors.New("structured llm client is required")
|
|
||||||
}
|
|
||||||
|
|
||||||
return contracts.ChunkResult{
|
|
||||||
Chunks: []contracts.SourceChunk{
|
|
||||||
{
|
|
||||||
ID: req.Source.ID + ":chunk:0",
|
|
||||||
SourceID: req.Source.ID,
|
|
||||||
Index: 0,
|
|
||||||
StartUnitID: req.Source.Units[0].ID,
|
|
||||||
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
|
|
||||||
Content: []byte(`{"units":[{"id":1,"kind":"unit","text":"First source unit."},{"id":2,"kind":"unit","text":"Second source unit."}]}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
Units: append([]source.SourceUnit(nil), req.Source.Units...),
|
|
||||||
Metadata: map[string]any{"strategy": "whole-document"},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type compositionLLMClient struct{}
|
|
||||||
|
|
||||||
func (client compositionLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
|
||||||
return contracts.StructuredCompletionResponse{}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type compositionExtractor struct{}
|
|
||||||
|
|
||||||
func (extractor compositionExtractor) Key() string {
|
|
||||||
return "generic-extractor"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (extractor compositionExtractor) ReferenceSlots() []contracts.ReferenceSlot {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (extractor compositionExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
|
||||||
if req.Source == nil {
|
|
||||||
return contracts.ExtractionResult{}, errors.New("source document is required")
|
|
||||||
}
|
|
||||||
if req.AmbientContext["synopsis"] == "" {
|
|
||||||
return contracts.ExtractionResult{}, errors.New("ambient synopsis is required")
|
|
||||||
}
|
|
||||||
|
|
||||||
return contracts.ExtractionResult{
|
|
||||||
Output: contracts.ExtractOutput{
|
|
||||||
Schema: contracts.ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
|
|
||||||
Payload: contracts.RawPayload{
|
|
||||||
Content: []byte(`{"value":"example"}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type compositionMerger struct{}
|
|
||||||
|
|
||||||
func (merger compositionMerger) Key() string {
|
|
||||||
return "generic-merger"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (merger compositionMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
|
|
||||||
output := req.ExtractOutputs[0]
|
|
||||||
return contracts.MergeResult{Output: contracts.MergeOutput{
|
|
||||||
LaneID: req.LaneID,
|
|
||||||
MergerKey: merger.Key(),
|
|
||||||
SourceID: output.SourceID,
|
|
||||||
Schema: output.Schema,
|
|
||||||
Payload: cloneCompositionPayload(output.Payload),
|
|
||||||
}}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type compositionNormalizer struct{}
|
|
||||||
|
|
||||||
func (normalizer compositionNormalizer) Key() string {
|
|
||||||
return "generic-normalizer"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (normalizer compositionNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (normalizer compositionNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
|
|
||||||
return contracts.NormalizeResult{Output: contracts.NormalizeOutput{
|
|
||||||
LaneID: req.LaneID,
|
|
||||||
NormalizerKey: normalizer.Key(),
|
|
||||||
SourceID: req.MergeOutput.SourceID,
|
|
||||||
Schema: req.MergeOutput.Schema,
|
|
||||||
Payload: cloneCompositionPayload(req.MergeOutput.Payload),
|
|
||||||
}}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func cloneCompositionPayload(payload contracts.RawPayload) contracts.RawPayload {
|
|
||||||
return contracts.RawPayload{
|
|
||||||
Content: append([]byte(nil), payload.Content...),
|
|
||||||
MediaType: payload.MediaType,
|
|
||||||
Metadata: cloneCompositionMetadata(payload.Metadata),
|
|
||||||
Warnings: append([]contracts.Warning(nil), payload.Warnings...),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func cloneCompositionMetadata(metadata map[string]any) map[string]any {
|
|
||||||
if len(metadata) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
out := make(map[string]any, len(metadata))
|
|
||||||
for key, value := range metadata {
|
|
||||||
out[key] = value
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
type compositionValidator struct{}
|
|
||||||
|
|
||||||
func (validator compositionValidator) Name() string {
|
|
||||||
return "generic-validator"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (validator compositionValidator) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassDeterministic
|
|
||||||
}
|
|
||||||
|
|
||||||
func (validator compositionValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
|
|
||||||
return contracts.ValidationResult{
|
|
||||||
Approved: true,
|
|
||||||
ReasonCode: "accepted",
|
|
||||||
Message: "output accepted",
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type compositionOutputEncoder struct{}
|
|
||||||
|
|
||||||
func (encoder compositionOutputEncoder) Key() string {
|
|
||||||
return "generic-output"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (encoder compositionOutputEncoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
|
||||||
payload := struct {
|
|
||||||
RunID string `json:"run_id"`
|
|
||||||
OutputCount int `json:"output_count"`
|
|
||||||
}{
|
|
||||||
RunID: req.Manifest.RunID,
|
|
||||||
OutputCount: len(req.NormalizeOutputs),
|
|
||||||
}
|
|
||||||
encoded, err := json.Marshal(payload)
|
|
||||||
if err != nil {
|
|
||||||
return contracts.OutputResult{}, err
|
|
||||||
}
|
|
||||||
|
|
||||||
return contracts.OutputResult{
|
|
||||||
Files: []contracts.OutputFile{
|
|
||||||
{
|
|
||||||
Name: "artifacts/generic.json",
|
|
||||||
ContentType: "application/json",
|
|
||||||
Bytes: encoded,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
@@ -129,7 +129,6 @@ type ParseRequest struct {
|
|||||||
Path string `json:"path,omitempty"`
|
Path string `json:"path,omitempty"`
|
||||||
Raw []byte `json:"-"`
|
Raw []byte `json:"-"`
|
||||||
LLMProfile string `json:"llm_profile,omitempty"`
|
LLMProfile string `json:"llm_profile,omitempty"`
|
||||||
Options map[string]any `json:"options,omitempty"`
|
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -138,32 +137,18 @@ type InputAdapter interface {
|
|||||||
Parse(ctx context.Context, req ParseRequest) (*source.SourceDocument, error)
|
Parse(ctx context.Context, req ParseRequest) (*source.SourceDocument, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
type SourceChunk struct {
|
|
||||||
ID string `json:"id"`
|
|
||||||
SourceID string `json:"source_id"`
|
|
||||||
Index int `json:"index"`
|
|
||||||
StartUnitID int `json:"start_unit_id"`
|
|
||||||
EndUnitID int `json:"end_unit_id"`
|
|
||||||
Content []byte `json:"-"`
|
|
||||||
MediaType string `json:"media_type"`
|
|
||||||
Units []source.SourceUnit `json:"units"`
|
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type ChunkRequest struct {
|
type ChunkRequest struct {
|
||||||
Source *source.SourceDocument `json:"-"`
|
Source *source.SourceDocument `json:"-"`
|
||||||
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
|
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
|
||||||
SessionID string `json:"session_id,omitempty"`
|
SessionID string `json:"session_id,omitempty"`
|
||||||
References ReferenceSet `json:"references,omitempty"`
|
References ReferenceSet `json:"references,omitempty"`
|
||||||
LLMClient StructuredLLMClient `json:"-"`
|
|
||||||
LLMProfile string `json:"llm_profile,omitempty"`
|
LLMProfile string `json:"llm_profile,omitempty"`
|
||||||
Options map[string]any `json:"options,omitempty"`
|
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ChunkResult struct {
|
type ChunkResult struct {
|
||||||
Chunks []SourceChunk `json:"chunks"`
|
Chunks []source.Chunk `json:"chunks"`
|
||||||
Warnings []Warning `json:"warnings,omitempty"`
|
Warnings []Warning `json:"warnings,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type Chunker interface {
|
type Chunker interface {
|
||||||
@@ -222,37 +207,6 @@ type ReferenceSet struct {
|
|||||||
Slots map[string]ResolvedReferenceSlot `json:"slots,omitempty"`
|
Slots map[string]ResolvedReferenceSlot `json:"slots,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ExtractionRequest struct {
|
|
||||||
Source *source.SourceDocument `json:"-"`
|
|
||||||
Chunk *SourceChunk `json:"chunk,omitempty"`
|
|
||||||
AmbientContext map[string]any `json:"ambient_context,omitempty"`
|
|
||||||
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
|
|
||||||
SessionID string `json:"session_id,omitempty"`
|
|
||||||
References ReferenceSet `json:"references,omitempty"`
|
|
||||||
LLMClient StructuredLLMClient `json:"-"`
|
|
||||||
LLMProfile string `json:"llm_profile,omitempty"`
|
|
||||||
Options map[string]any `json:"options,omitempty"`
|
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type ExtractionResult struct {
|
|
||||||
Output ExtractOutput `json:"output"`
|
|
||||||
Warnings []Warning `json:"warnings,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type Extractor interface {
|
|
||||||
Key() string
|
|
||||||
ReferenceSlots() []ReferenceSlot
|
|
||||||
Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
type RawPayload struct {
|
|
||||||
Content []byte `json:"-"`
|
|
||||||
MediaType string `json:"media_type"`
|
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
|
||||||
Warnings []Warning `json:"warnings,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type ExecutionClass string
|
type ExecutionClass string
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -260,29 +214,6 @@ const (
|
|||||||
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
|
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
|
||||||
)
|
)
|
||||||
|
|
||||||
type ValidationRequest struct {
|
|
||||||
Stage string `json:"stage"`
|
|
||||||
LaneID string `json:"lane_id,omitempty"`
|
|
||||||
ModuleKey string `json:"module_key"`
|
|
||||||
Source *source.SourceDocument `json:"-"`
|
|
||||||
SourceID string `json:"source_id,omitempty"`
|
|
||||||
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
|
|
||||||
SessionID string `json:"session_id,omitempty"`
|
|
||||||
References ReferenceSet `json:"references,omitempty"`
|
|
||||||
LLMClient StructuredLLMClient `json:"-"`
|
|
||||||
LLMProfile string `json:"llm_profile,omitempty"`
|
|
||||||
Options map[string]any `json:"options,omitempty"`
|
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
|
||||||
Schema ResponseSchema `json:"schema,omitempty"`
|
|
||||||
Payload RawPayload `json:"payload"`
|
|
||||||
ChunkID string `json:"chunk_id,omitempty"`
|
|
||||||
ChunkIndex int `json:"chunk_index,omitempty"`
|
|
||||||
Chunk *SourceChunk `json:"chunk,omitempty"`
|
|
||||||
Chunks []SourceChunk `json:"chunks,omitempty"`
|
|
||||||
ExtractOutputs []ExtractOutput `json:"extract_outputs,omitempty"`
|
|
||||||
MergeOutput MergeOutput `json:"merge_output,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type ValidationResult struct {
|
type ValidationResult struct {
|
||||||
Approved bool `json:"approved"`
|
Approved bool `json:"approved"`
|
||||||
ReasonCode string `json:"reason_code,omitempty"`
|
ReasonCode string `json:"reason_code,omitempty"`
|
||||||
@@ -291,92 +222,6 @@ type ValidationResult struct {
|
|||||||
Warnings []Warning `json:"warnings,omitempty"`
|
Warnings []Warning `json:"warnings,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type Validator interface {
|
|
||||||
Name() string
|
|
||||||
ExecutionClass() ExecutionClass
|
|
||||||
Validate(ctx context.Context, req ValidationRequest) (ValidationResult, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
type ResponseSchema struct {
|
|
||||||
ID string `json:"id,omitempty"`
|
|
||||||
Name string `json:"name,omitempty"`
|
|
||||||
Version string `json:"version,omitempty"`
|
|
||||||
JSONSchema []byte `json:"-"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type ExtractOutput struct {
|
|
||||||
LaneID string `json:"lane_id"`
|
|
||||||
ExtractorKey string `json:"extractor_key"`
|
|
||||||
SourceID string `json:"source_id"`
|
|
||||||
ChunkID string `json:"chunk_id"`
|
|
||||||
ChunkIndex int `json:"chunk_index"`
|
|
||||||
Schema ResponseSchema `json:"schema,omitempty"`
|
|
||||||
Payload RawPayload `json:"payload"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type MergeRequest struct {
|
|
||||||
Source *source.SourceDocument `json:"-"`
|
|
||||||
LaneID string `json:"lane_id"`
|
|
||||||
ExtractOutputs []ExtractOutput `json:"extract_outputs"`
|
|
||||||
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
|
|
||||||
SessionID string `json:"session_id,omitempty"`
|
|
||||||
References ReferenceSet `json:"references,omitempty"`
|
|
||||||
LLMClient StructuredLLMClient `json:"-"`
|
|
||||||
LLMProfile string `json:"llm_profile,omitempty"`
|
|
||||||
Options map[string]any `json:"options,omitempty"`
|
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type MergeResult struct {
|
|
||||||
Output MergeOutput `json:"output"`
|
|
||||||
Warnings []Warning `json:"warnings,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type MergeOutput struct {
|
|
||||||
LaneID string `json:"lane_id"`
|
|
||||||
MergerKey string `json:"merger_key"`
|
|
||||||
SourceID string `json:"source_id,omitempty"`
|
|
||||||
Schema ResponseSchema `json:"schema,omitempty"`
|
|
||||||
Payload RawPayload `json:"payload"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type Merger interface {
|
|
||||||
Key() string
|
|
||||||
Merge(ctx context.Context, req MergeRequest) (MergeResult, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
type NormalizeRequest struct {
|
|
||||||
Source *source.SourceDocument `json:"-"`
|
|
||||||
LaneID string `json:"lane_id"`
|
|
||||||
MergeOutput MergeOutput `json:"merge_output"`
|
|
||||||
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
|
|
||||||
SessionID string `json:"session_id,omitempty"`
|
|
||||||
References ReferenceSet `json:"references,omitempty"`
|
|
||||||
LLMClient StructuredLLMClient `json:"-"`
|
|
||||||
LLMProfile string `json:"llm_profile,omitempty"`
|
|
||||||
Options map[string]any `json:"options,omitempty"`
|
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type NormalizeResult struct {
|
|
||||||
Output NormalizeOutput `json:"output"`
|
|
||||||
Warnings []Warning `json:"warnings,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type NormalizeOutput struct {
|
|
||||||
LaneID string `json:"lane_id"`
|
|
||||||
NormalizerKey string `json:"normalizer_key"`
|
|
||||||
SourceID string `json:"source_id,omitempty"`
|
|
||||||
Schema ResponseSchema `json:"schema,omitempty"`
|
|
||||||
Payload RawPayload `json:"payload"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type Normalizer interface {
|
|
||||||
Key() string
|
|
||||||
ReferenceSlots() []ReferenceSlot
|
|
||||||
Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
type Warning struct {
|
type Warning struct {
|
||||||
Scope string `json:"scope,omitempty"`
|
Scope string `json:"scope,omitempty"`
|
||||||
ReasonCode string `json:"reason_code"`
|
ReasonCode string `json:"reason_code"`
|
||||||
@@ -385,11 +230,10 @@ type Warning struct {
|
|||||||
|
|
||||||
type OutputRequest struct {
|
type OutputRequest struct {
|
||||||
Manifest artifacts.RunManifest `json:"manifest"`
|
Manifest artifacts.RunManifest `json:"manifest"`
|
||||||
NormalizeOutputs []NormalizeOutput `json:"normalize_outputs,omitempty"`
|
NormalizeOutputs []SerializedOutput `json:"normalize_outputs,omitempty"`
|
||||||
Rejected []RejectedOutput `json:"rejected,omitempty"`
|
Rejected []RejectedOutput `json:"rejected,omitempty"`
|
||||||
Warnings []Warning `json:"warnings,omitempty"`
|
Warnings []Warning `json:"warnings,omitempty"`
|
||||||
LLMProfile string `json:"llm_profile,omitempty"`
|
LLMProfile string `json:"llm_profile,omitempty"`
|
||||||
Options map[string]any `json:"options,omitempty"`
|
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -12,14 +12,13 @@ import (
|
|||||||
|
|
||||||
var _ InputAdapter = fakeAdapter{}
|
var _ InputAdapter = fakeAdapter{}
|
||||||
var _ Chunker = fakeChunker{}
|
var _ Chunker = fakeChunker{}
|
||||||
var _ Extractor = fakeExtractor{}
|
var _ Extractor[fakeArtifact] = fakeExtractor{}
|
||||||
var _ Merger = fakeMerger{}
|
var _ Merger[fakeArtifact] = fakeMerger{}
|
||||||
var _ Normalizer = fakeNormalizer{}
|
var _ Normalizer[fakeArtifact] = fakeNormalizer{}
|
||||||
var _ Validator = fakeValidator{}
|
|
||||||
var _ StructuredLLMClient = fakeLLMClient{}
|
var _ StructuredLLMClient = fakeLLMClient{}
|
||||||
var _ OutputEncoder = fakeOutputEncoder{}
|
var _ OutputEncoder = fakeOutputEncoder{}
|
||||||
|
|
||||||
func TestFakeExtractorReturnsRawOutput(t *testing.T) {
|
func TestFakeExtractorReturnsTypedOutput(t *testing.T) {
|
||||||
extractor := fakeExtractor{
|
extractor := fakeExtractor{
|
||||||
key: "generic-extractor",
|
key: "generic-extractor",
|
||||||
}
|
}
|
||||||
@@ -33,7 +32,7 @@ func TestFakeExtractorReturnsRawOutput(t *testing.T) {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
result, err := extractor.Extract(context.Background(), ExtractionRequest{Source: doc})
|
result, err := extractor.Extract(context.Background(), TypedExtractionRequest{Source: doc})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Extract() error = %v, want nil", err)
|
t.Fatalf("Extract() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
@@ -41,14 +40,8 @@ func TestFakeExtractorReturnsRawOutput(t *testing.T) {
|
|||||||
if extractor.Key() != "generic-extractor" {
|
if extractor.Key() != "generic-extractor" {
|
||||||
t.Fatalf("Key() = %q, want generic-extractor", extractor.Key())
|
t.Fatalf("Key() = %q, want generic-extractor", extractor.Key())
|
||||||
}
|
}
|
||||||
if result.Output.ExtractorKey != "" {
|
if result.Value.Value != "example" {
|
||||||
t.Fatalf("ExtractorKey = %q, want runner-owned empty value", result.Output.ExtractorKey)
|
t.Fatalf("Value = %q, want example", result.Value.Value)
|
||||||
}
|
|
||||||
if result.Output.Schema.Version != "v1" {
|
|
||||||
t.Fatalf("Schema.Version = %q, want v1", result.Output.Schema.Version)
|
|
||||||
}
|
|
||||||
if result.Output.Payload.MediaType != "application/json" || string(result.Output.Payload.Content) != `{"value":"example"}` {
|
|
||||||
t.Fatalf("payload = %q %s, want JSON raw output", result.Output.Payload.MediaType, result.Output.Payload.Content)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -78,26 +71,26 @@ func TestFakeChunkerReturnsSourceChunks(t *testing.T) {
|
|||||||
|
|
||||||
chunk := result.Chunks[0]
|
chunk := result.Chunks[0]
|
||||||
if chunk.ID != "source-1:chunk:0" {
|
if chunk.ID != "source-1:chunk:0" {
|
||||||
t.Fatalf("SourceChunk.ID = %q, want source-1:chunk:0", chunk.ID)
|
t.Fatalf("source.Chunk.ID = %q, want source-1:chunk:0", chunk.ID)
|
||||||
}
|
}
|
||||||
if chunk.SourceID != doc.ID {
|
if chunk.SourceID != doc.ID {
|
||||||
t.Fatalf("SourceChunk.SourceID = %q, want %q", chunk.SourceID, doc.ID)
|
t.Fatalf("source.Chunk.SourceID = %q, want %q", chunk.SourceID, doc.ID)
|
||||||
}
|
}
|
||||||
if chunk.Index != 0 {
|
if chunk.Index != 0 {
|
||||||
t.Fatalf("SourceChunk.Index = %d, want 0", chunk.Index)
|
t.Fatalf("source.Chunk.Index = %d, want 0", chunk.Index)
|
||||||
}
|
}
|
||||||
if chunk.StartUnitID != 1 || chunk.EndUnitID != 1 {
|
if chunk.Ref.StartUnitID != 1 || chunk.Ref.EndUnitID != 1 {
|
||||||
t.Fatalf("SourceChunk boundaries = %d-%d, want 1-1", chunk.StartUnitID, chunk.EndUnitID)
|
t.Fatalf("source.Chunk.Ref = %#v, want source-1:1-1", chunk.Ref)
|
||||||
}
|
}
|
||||||
if chunk.MediaType != "application/json" || string(chunk.Content) != `{"units":[{"id":1,"kind":"section","text":"Source text."}]}` {
|
if chunk.MediaType != "application/json" || string(chunk.Content) != `{"units":[{"id":1,"kind":"section","text":"Source text."}]}` {
|
||||||
t.Fatalf("SourceChunk payload = %q %s, want JSON units", chunk.MediaType, chunk.Content)
|
t.Fatalf("source.Chunk payload = %q %s, want JSON units", chunk.MediaType, chunk.Content)
|
||||||
}
|
}
|
||||||
if len(chunk.Units) != 1 {
|
if len(chunk.Units) != 1 {
|
||||||
t.Fatalf("len(SourceChunk.Units) = %d, want 1", len(chunk.Units))
|
t.Fatalf("len(source.Chunk.Units) = %d, want 1", len(chunk.Units))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFakeChunkerReceivesLLMClient(t *testing.T) {
|
func TestFakeChunkerReceivesPerRunContext(t *testing.T) {
|
||||||
doc := &source.SourceDocument{
|
doc := &source.SourceDocument{
|
||||||
ID: "source-1",
|
ID: "source-1",
|
||||||
Kind: "document",
|
Kind: "document",
|
||||||
@@ -107,14 +100,13 @@ func TestFakeChunkerReceivesLLMClient(t *testing.T) {
|
|||||||
{ID: 1, Kind: "section", Text: "Source text."},
|
{ID: 1, Kind: "section", Text: "Source text."},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
client := fakeLLMClient{}
|
|
||||||
chunker := &recordingChunker{key: "llm-chunker"}
|
chunker := &recordingChunker{key: "llm-chunker"}
|
||||||
|
|
||||||
if _, err := chunker.Chunk(context.Background(), ChunkRequest{Source: doc, LLMClient: client}); err != nil {
|
if _, err := chunker.Chunk(context.Background(), ChunkRequest{Source: doc, SessionID: "session", LLMProfile: "profile"}); err != nil {
|
||||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
if chunker.request.LLMClient == nil {
|
if chunker.request.SessionID != "session" || chunker.request.LLMProfile != "profile" {
|
||||||
t.Fatal("ChunkRequest.LLMClient = nil, want structured LLM client")
|
t.Fatalf("ChunkRequest = %#v, want per-run session and profile", chunker.request)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -130,18 +122,17 @@ func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
|
|||||||
{ID: 2, Kind: "section", Text: "Second source text."},
|
{ID: 2, Kind: "section", Text: "Second source text."},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
chunk := SourceChunk{
|
chunk := source.Chunk{
|
||||||
ID: "source-1:chunk:1",
|
ID: "source-1:chunk:1",
|
||||||
SourceID: doc.ID,
|
SourceID: doc.ID,
|
||||||
Index: 1,
|
Index: 1,
|
||||||
StartUnitID: 2,
|
Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 2, EndUnitID: 2},
|
||||||
EndUnitID: 2,
|
Content: []byte(`{"units":[{"id":2,"kind":"section","text":"Second source text."}]}`),
|
||||||
Content: []byte(`{"units":[{"id":2,"kind":"section","text":"Second source text."}]}`),
|
MediaType: "application/json",
|
||||||
MediaType: "application/json",
|
Units: []source.SourceUnit{doc.Units[1]},
|
||||||
Units: []source.SourceUnit{doc.Units[1]},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
result, err := extractor.Extract(context.Background(), ExtractionRequest{
|
result, err := extractor.Extract(context.Background(), TypedExtractionRequest{
|
||||||
Source: doc,
|
Source: doc,
|
||||||
Chunk: &chunk,
|
Chunk: &chunk,
|
||||||
AmbientContext: map[string]any{"mode": "chunked"},
|
AmbientContext: map[string]any{"mode": "chunked"},
|
||||||
@@ -149,11 +140,8 @@ func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Extract() error = %v, want nil", err)
|
t.Fatalf("Extract() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
if result.Output.ChunkID != "" || result.Output.ChunkIndex != 0 {
|
if result.Value.Value != "chunked" {
|
||||||
t.Fatalf("chunk provenance = %q/%d, want runner-owned zero values", result.Output.ChunkID, result.Output.ChunkIndex)
|
t.Fatalf("Value = %q, want chunked", result.Value.Value)
|
||||||
}
|
|
||||||
if string(result.Output.Payload.Content) != `{"value":"chunked"}` {
|
|
||||||
t.Fatalf("Payload.Content = %s, want chunked payload", result.Output.Payload.Content)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -322,8 +310,8 @@ func TestLLMInputSetCloneCopiesContent(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResponseSchemaJSONOmitRawSchemaContent(t *testing.T) {
|
func TestArtifactSchemaJSONOmitsSchemaContent(t *testing.T) {
|
||||||
schema := ResponseSchema{
|
schema := ArtifactSchema{
|
||||||
ID: "schema-id",
|
ID: "schema-id",
|
||||||
Name: "schema-name",
|
Name: "schema-name",
|
||||||
Version: "v1",
|
Version: "v1",
|
||||||
@@ -349,26 +337,14 @@ func TestResponseSchemaJSONOmitRawSchemaContent(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
|
func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
|
||||||
extractOutput := ExtractOutput{
|
extractOutput := ExtractArtifact[fakeArtifact]{LaneID: "generic-lane", ExtractorKey: "generic-extractor", SourceID: "source-1", ChunkID: "source-1:chunk:0", ChunkIndex: 0, Value: fakeArtifact{Value: "example"}}
|
||||||
LaneID: "generic-lane",
|
|
||||||
ExtractorKey: "generic-extractor",
|
|
||||||
SourceID: "source-1",
|
|
||||||
ChunkID: "source-1:chunk:0",
|
|
||||||
ChunkIndex: 0,
|
|
||||||
Schema: ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
|
|
||||||
Payload: RawPayload{
|
|
||||||
Content: []byte(`{"value":"example"}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
Metadata: map[string]any{"confidence": 0.75},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
merger := fakeMerger{key: "generic-merger"}
|
merger := fakeMerger{key: "generic-merger"}
|
||||||
normalizer := fakeNormalizer{key: "generic-normalizer"}
|
normalizer := fakeNormalizer{key: "generic-normalizer"}
|
||||||
encoder := fakeOutputEncoder{key: "generic-output"}
|
encoder := fakeOutputEncoder{key: "generic-output"}
|
||||||
|
|
||||||
merged, err := merger.Merge(context.Background(), MergeRequest{
|
merged, err := merger.Merge(context.Background(), TypedMergeRequest[fakeArtifact]{
|
||||||
LaneID: "generic-lane",
|
LaneID: "generic-lane",
|
||||||
ExtractOutputs: []ExtractOutput{extractOutput},
|
ExtractOutputs: []ExtractArtifact[fakeArtifact]{extractOutput},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Merge() error = %v, want nil", err)
|
t.Fatalf("Merge() error = %v, want nil", err)
|
||||||
@@ -376,13 +352,13 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
|
|||||||
if merger.Key() != "generic-merger" {
|
if merger.Key() != "generic-merger" {
|
||||||
t.Fatalf("Merger.Key() = %q, want generic-merger", merger.Key())
|
t.Fatalf("Merger.Key() = %q, want generic-merger", merger.Key())
|
||||||
}
|
}
|
||||||
if string(merged.Output.Payload.Content) != `{"value":"example"}` {
|
if merged.Value.Value != "example" {
|
||||||
t.Fatalf("merged content = %s, want raw extract content", merged.Output.Payload.Content)
|
t.Fatalf("merged value = %q, want example", merged.Value.Value)
|
||||||
}
|
}
|
||||||
|
|
||||||
normalized, err := normalizer.Normalize(context.Background(), NormalizeRequest{
|
normalized, err := normalizer.Normalize(context.Background(), TypedNormalizeRequest[fakeArtifact]{
|
||||||
LaneID: "generic-lane",
|
LaneID: "generic-lane",
|
||||||
MergeOutput: merged.Output,
|
MergeOutput: MergeArtifact[fakeArtifact]{LaneID: "generic-lane", MergerKey: merger.Key(), SourceID: "source-1", Value: merged.Value},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Normalize() error = %v, want nil", err)
|
t.Fatalf("Normalize() error = %v, want nil", err)
|
||||||
@@ -390,13 +366,13 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
|
|||||||
if normalizer.Key() != "generic-normalizer" {
|
if normalizer.Key() != "generic-normalizer" {
|
||||||
t.Fatalf("Normalizer.Key() = %q, want generic-normalizer", normalizer.Key())
|
t.Fatalf("Normalizer.Key() = %q, want generic-normalizer", normalizer.Key())
|
||||||
}
|
}
|
||||||
if string(normalized.Output.Payload.Content) != `{"value":"example"}` {
|
if normalized.Value.Value != "example" {
|
||||||
t.Fatalf("normalized content = %s, want raw merge content", normalized.Output.Payload.Content)
|
t.Fatalf("normalized value = %q, want example", normalized.Value.Value)
|
||||||
}
|
}
|
||||||
|
|
||||||
encoded, err := encoder.Encode(context.Background(), OutputRequest{
|
encoded, err := encoder.Encode(context.Background(), OutputRequest{
|
||||||
Manifest: artifacts.RunManifest{RunID: "run-1"},
|
Manifest: artifacts.RunManifest{RunID: "run-1"},
|
||||||
NormalizeOutputs: []NormalizeOutput{normalized.Output},
|
NormalizeOutputs: []SerializedOutput{{LaneID: "generic-lane", NormalizerKey: normalizer.Key(), SourceID: "source-1", Artifact: SerializedArtifact{Kind: "test/artifact", Schema: ArtifactSchema{ID: "schema-id", Name: "schema-name", Version: "v1"}, MediaType: "application/json", Content: []byte(`{"value":"example"}`)}}},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Encode() error = %v, want nil", err)
|
t.Fatalf("Encode() error = %v, want nil", err)
|
||||||
@@ -472,16 +448,19 @@ func (chunker fakeChunker) ReferenceSlots() []ReferenceSlot {
|
|||||||
|
|
||||||
func (chunker fakeChunker) Chunk(ctx context.Context, req ChunkRequest) (ChunkResult, error) {
|
func (chunker fakeChunker) Chunk(ctx context.Context, req ChunkRequest) (ChunkResult, error) {
|
||||||
return ChunkResult{
|
return ChunkResult{
|
||||||
Chunks: []SourceChunk{
|
Chunks: []source.Chunk{
|
||||||
{
|
{
|
||||||
ID: req.Source.ID + ":chunk:0",
|
ID: req.Source.ID + ":chunk:0",
|
||||||
SourceID: req.Source.ID,
|
SourceID: req.Source.ID,
|
||||||
Index: 0,
|
Index: 0,
|
||||||
StartUnitID: req.Source.Units[0].ID,
|
Ref: source.SourceRef{
|
||||||
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
|
SourceID: req.Source.ID,
|
||||||
Content: []byte(`{"units":[{"id":1,"kind":"section","text":"Source text."}]}`),
|
StartUnitID: req.Source.Units[0].ID,
|
||||||
MediaType: "application/json",
|
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
|
||||||
Units: append([]source.SourceUnit(nil), req.Source.Units...),
|
},
|
||||||
|
Content: []byte(`{"units":[{"id":1,"kind":"section","text":"Source text."}]}`),
|
||||||
|
MediaType: "application/json",
|
||||||
|
Units: append([]source.SourceUnit(nil), req.Source.Units...),
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
}, nil
|
}, nil
|
||||||
@@ -509,6 +488,8 @@ type fakeExtractor struct {
|
|||||||
key string
|
key string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type fakeArtifact struct{ Value string }
|
||||||
|
|
||||||
func (extractor fakeExtractor) Key() string {
|
func (extractor fakeExtractor) Key() string {
|
||||||
return extractor.key
|
return extractor.key
|
||||||
}
|
}
|
||||||
@@ -517,21 +498,12 @@ func (extractor fakeExtractor) ReferenceSlots() []ReferenceSlot {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (extractor fakeExtractor) Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error) {
|
func (extractor fakeExtractor) Extract(ctx context.Context, req TypedExtractionRequest) (TypedExtractionResult[fakeArtifact], error) {
|
||||||
payload := json.RawMessage(`{"value":"example"}`)
|
value := "example"
|
||||||
if req.AmbientContext["mode"] == "chunked" {
|
if req.AmbientContext["mode"] == "chunked" {
|
||||||
payload = json.RawMessage(`{"value":"chunked"}`)
|
value = "chunked"
|
||||||
}
|
}
|
||||||
|
return TypedExtractionResult[fakeArtifact]{Value: fakeArtifact{Value: value}}, nil
|
||||||
return ExtractionResult{
|
|
||||||
Output: ExtractOutput{
|
|
||||||
Schema: ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
|
|
||||||
Payload: RawPayload{
|
|
||||||
Content: append([]byte(nil), payload...),
|
|
||||||
MediaType: "application/json",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type fakeMerger struct {
|
type fakeMerger struct {
|
||||||
@@ -542,15 +514,8 @@ func (merger fakeMerger) Key() string {
|
|||||||
return merger.key
|
return merger.key
|
||||||
}
|
}
|
||||||
|
|
||||||
func (merger fakeMerger) Merge(ctx context.Context, req MergeRequest) (MergeResult, error) {
|
func (merger fakeMerger) Merge(ctx context.Context, req TypedMergeRequest[fakeArtifact]) (TypedMergeResult[fakeArtifact], error) {
|
||||||
output := req.ExtractOutputs[0]
|
return TypedMergeResult[fakeArtifact]{Value: req.ExtractOutputs[0].Value}, nil
|
||||||
return MergeResult{Output: MergeOutput{
|
|
||||||
LaneID: req.LaneID,
|
|
||||||
MergerKey: merger.key,
|
|
||||||
SourceID: output.SourceID,
|
|
||||||
Schema: output.Schema,
|
|
||||||
Payload: cloneTestRawPayload(output.Payload),
|
|
||||||
}}, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type fakeNormalizer struct {
|
type fakeNormalizer struct {
|
||||||
@@ -565,54 +530,8 @@ func (normalizer fakeNormalizer) ReferenceSlots() []ReferenceSlot {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (normalizer fakeNormalizer) Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error) {
|
func (normalizer fakeNormalizer) Normalize(ctx context.Context, req TypedNormalizeRequest[fakeArtifact]) (TypedNormalizeResult[fakeArtifact], error) {
|
||||||
return NormalizeResult{Output: NormalizeOutput{
|
return TypedNormalizeResult[fakeArtifact]{Value: req.MergeOutput.Value}, nil
|
||||||
LaneID: req.LaneID,
|
|
||||||
NormalizerKey: normalizer.key,
|
|
||||||
SourceID: req.MergeOutput.SourceID,
|
|
||||||
Schema: req.MergeOutput.Schema,
|
|
||||||
Payload: cloneTestRawPayload(req.MergeOutput.Payload),
|
|
||||||
}}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func cloneTestRawPayload(payload RawPayload) RawPayload {
|
|
||||||
return RawPayload{
|
|
||||||
Content: append([]byte(nil), payload.Content...),
|
|
||||||
MediaType: payload.MediaType,
|
|
||||||
Metadata: cloneTestMetadata(payload.Metadata),
|
|
||||||
Warnings: append([]Warning(nil), payload.Warnings...),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func cloneTestMetadata(metadata map[string]any) map[string]any {
|
|
||||||
if len(metadata) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
out := make(map[string]any, len(metadata))
|
|
||||||
for key, value := range metadata {
|
|
||||||
out[key] = value
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
type fakeValidator struct {
|
|
||||||
name string
|
|
||||||
}
|
|
||||||
|
|
||||||
func (validator fakeValidator) Name() string {
|
|
||||||
return validator.name
|
|
||||||
}
|
|
||||||
|
|
||||||
func (validator fakeValidator) ExecutionClass() ExecutionClass {
|
|
||||||
return ExecutionClassDeterministic
|
|
||||||
}
|
|
||||||
|
|
||||||
func (validator fakeValidator) Validate(ctx context.Context, req ValidationRequest) (ValidationResult, error) {
|
|
||||||
return ValidationResult{
|
|
||||||
Approved: true,
|
|
||||||
ReasonCode: "accepted",
|
|
||||||
Message: "output accepted",
|
|
||||||
}, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type fakeLLMClient struct{}
|
type fakeLLMClient struct{}
|
||||||
|
|||||||
164
internal/framework/contracts/typed_pipeline.go
Normal file
164
internal/framework/contracts/typed_pipeline.go
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
package contracts
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ExtractArtifact carries a typed per-chunk value with framework provenance.
|
||||||
|
type ExtractArtifact[T any] struct {
|
||||||
|
LaneID string
|
||||||
|
ExtractorKey string
|
||||||
|
SourceID string
|
||||||
|
ChunkID string
|
||||||
|
ChunkIndex int
|
||||||
|
ChunkRef source.SourceRef
|
||||||
|
Value T
|
||||||
|
}
|
||||||
|
|
||||||
|
// MergeArtifact carries a typed merged value with framework provenance.
|
||||||
|
type MergeArtifact[T any] struct {
|
||||||
|
LaneID string
|
||||||
|
MergerKey string
|
||||||
|
SourceID string
|
||||||
|
Value T
|
||||||
|
}
|
||||||
|
|
||||||
|
// NormalizeArtifact carries a typed normalized value with framework provenance.
|
||||||
|
type NormalizeArtifact[T any] struct {
|
||||||
|
LaneID string
|
||||||
|
NormalizerKey string
|
||||||
|
SourceID string
|
||||||
|
Value T
|
||||||
|
}
|
||||||
|
|
||||||
|
type TypedExtractionRequest struct {
|
||||||
|
Source *source.SourceDocument
|
||||||
|
Chunk *source.Chunk
|
||||||
|
AmbientContext map[string]any
|
||||||
|
SourceInput LLMInputMaterial
|
||||||
|
SessionID string
|
||||||
|
References ReferenceSet
|
||||||
|
LLMProfile string
|
||||||
|
Metadata map[string]any
|
||||||
|
}
|
||||||
|
|
||||||
|
type TypedExtractionResult[T any] struct {
|
||||||
|
Value T
|
||||||
|
Warnings []Warning
|
||||||
|
}
|
||||||
|
|
||||||
|
type Extractor[T any] interface {
|
||||||
|
Key() string
|
||||||
|
ReferenceSlots() []ReferenceSlot
|
||||||
|
Extract(context.Context, TypedExtractionRequest) (TypedExtractionResult[T], error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type TypedMergeRequest[T any] struct {
|
||||||
|
Source *source.SourceDocument
|
||||||
|
LaneID string
|
||||||
|
ExtractOutputs []ExtractArtifact[T]
|
||||||
|
SourceInput LLMInputMaterial
|
||||||
|
SessionID string
|
||||||
|
References ReferenceSet
|
||||||
|
LLMProfile string
|
||||||
|
Metadata map[string]any
|
||||||
|
}
|
||||||
|
|
||||||
|
type TypedMergeResult[T any] struct {
|
||||||
|
Value T
|
||||||
|
Warnings []Warning
|
||||||
|
}
|
||||||
|
|
||||||
|
type Merger[T any] interface {
|
||||||
|
Key() string
|
||||||
|
Merge(context.Context, TypedMergeRequest[T]) (TypedMergeResult[T], error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type TypedNormalizeRequest[T any] struct {
|
||||||
|
Source *source.SourceDocument
|
||||||
|
LaneID string
|
||||||
|
MergeOutput MergeArtifact[T]
|
||||||
|
SourceInput LLMInputMaterial
|
||||||
|
SessionID string
|
||||||
|
References ReferenceSet
|
||||||
|
LLMProfile string
|
||||||
|
Metadata map[string]any
|
||||||
|
}
|
||||||
|
|
||||||
|
type TypedNormalizeResult[T any] struct {
|
||||||
|
Value T
|
||||||
|
Warnings []Warning
|
||||||
|
}
|
||||||
|
|
||||||
|
type Normalizer[T any] interface {
|
||||||
|
Key() string
|
||||||
|
ReferenceSlots() []ReferenceSlot
|
||||||
|
Normalize(context.Context, TypedNormalizeRequest[T]) (TypedNormalizeResult[T], error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type TypedValidationRequest[T any] struct {
|
||||||
|
Stage string
|
||||||
|
LaneID string
|
||||||
|
ModuleKey string
|
||||||
|
Source *source.SourceDocument
|
||||||
|
SourceID string
|
||||||
|
SourceInput LLMInputMaterial
|
||||||
|
SessionID string
|
||||||
|
References ReferenceSet
|
||||||
|
LLMProfile string
|
||||||
|
Metadata map[string]any
|
||||||
|
Chunk *source.Chunk
|
||||||
|
Chunks []source.Chunk
|
||||||
|
Ref source.SourceRef
|
||||||
|
Value T
|
||||||
|
}
|
||||||
|
|
||||||
|
type TypedValidator[T any] interface {
|
||||||
|
Name() string
|
||||||
|
ExecutionClass() ExecutionClass
|
||||||
|
Validate(context.Context, TypedValidationRequest[T]) (ValidationResult, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type ChunkValidationRequest struct {
|
||||||
|
ModuleKey string
|
||||||
|
Source *source.SourceDocument
|
||||||
|
SourceID string
|
||||||
|
SourceInput LLMInputMaterial
|
||||||
|
SessionID string
|
||||||
|
References ReferenceSet
|
||||||
|
LLMProfile string
|
||||||
|
Metadata map[string]any
|
||||||
|
Chunks []source.Chunk
|
||||||
|
}
|
||||||
|
|
||||||
|
type ChunkValidator interface {
|
||||||
|
Name() string
|
||||||
|
ExecutionClass() ExecutionClass
|
||||||
|
Validate(context.Context, ChunkValidationRequest) (ValidationResult, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type SerializedValidationRequest struct {
|
||||||
|
Stage string
|
||||||
|
LaneID string
|
||||||
|
ModuleKey string
|
||||||
|
Source *source.SourceDocument
|
||||||
|
SourceID string
|
||||||
|
SourceInput LLMInputMaterial
|
||||||
|
SessionID string
|
||||||
|
References ReferenceSet
|
||||||
|
LLMProfile string
|
||||||
|
Metadata map[string]any
|
||||||
|
Chunk *source.Chunk
|
||||||
|
Chunks []source.Chunk
|
||||||
|
Schema ArtifactSchema
|
||||||
|
MediaType string
|
||||||
|
Content []byte
|
||||||
|
}
|
||||||
|
|
||||||
|
type SerializedValidator interface {
|
||||||
|
Name() string
|
||||||
|
ExecutionClass() ExecutionClass
|
||||||
|
Validate(context.Context, SerializedValidationRequest) (ValidationResult, error)
|
||||||
|
}
|
||||||
318
internal/framework/pipeline/artifact_codec_registry.go
Normal file
318
internal/framework/pipeline/artifact_codec_registry.go
Normal file
@@ -0,0 +1,318 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"reflect"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/santhosh-tekuri/jsonschema/v6"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
type ArtifactCodecSpec struct {
|
||||||
|
Kind contracts.ArtifactKind
|
||||||
|
Schema contracts.ArtifactSchema
|
||||||
|
SchemaDigest string
|
||||||
|
MediaType string
|
||||||
|
}
|
||||||
|
|
||||||
|
// ArtifactCodecTypeError reports a value that does not have the exact Go type
|
||||||
|
// registered for an artifact kind.
|
||||||
|
type ArtifactCodecTypeError struct {
|
||||||
|
Operation string
|
||||||
|
Kind contracts.ArtifactKind
|
||||||
|
ExpectedType string
|
||||||
|
ActualType string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *ArtifactCodecTypeError) Error() string {
|
||||||
|
return fmt.Sprintf("%s artifact %q: expected exact Go type %s, got %s", e.Operation, e.Kind, e.ExpectedType, e.ActualType)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ArtifactCodecCompatibilityError reports serialized metadata that does not
|
||||||
|
// identify the registered representation for an artifact kind.
|
||||||
|
type ArtifactCodecCompatibilityError struct {
|
||||||
|
Kind contracts.ArtifactKind
|
||||||
|
Reason string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *ArtifactCodecCompatibilityError) Error() string {
|
||||||
|
return fmt.Sprintf("decode artifact %q: %s", e.Kind, e.Reason)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ArtifactCodecOperationError preserves an encode or decode failure from the
|
||||||
|
// registered domain codec.
|
||||||
|
type ArtifactCodecOperationError struct {
|
||||||
|
Operation string
|
||||||
|
Kind contracts.ArtifactKind
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *ArtifactCodecOperationError) Error() string {
|
||||||
|
return fmt.Sprintf("%s artifact %q: %v", e.Operation, e.Kind, e.Err)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *ArtifactCodecOperationError) Unwrap() error { return e.Err }
|
||||||
|
|
||||||
|
type ArtifactCodecRegistry struct {
|
||||||
|
entries map[contracts.ArtifactKind]artifactCodecEntry
|
||||||
|
}
|
||||||
|
|
||||||
|
type artifactCodecEntry struct {
|
||||||
|
spec ArtifactCodecSpec
|
||||||
|
valueType reflect.Type
|
||||||
|
encode func(any) ([]byte, error)
|
||||||
|
encodeCandidate func(any) ([]byte, error)
|
||||||
|
metadata func(any) map[string]any
|
||||||
|
decode func([]byte) (any, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewArtifactCodecRegistry() *ArtifactCodecRegistry {
|
||||||
|
return &ArtifactCodecRegistry{entries: make(map[contracts.ArtifactKind]artifactCodecEntry)}
|
||||||
|
}
|
||||||
|
|
||||||
|
// RegisterArtifactCodec registers one codec for T. The concrete type is kept
|
||||||
|
// private and checked at every erased encode boundary.
|
||||||
|
func RegisterArtifactCodec[T any](registry *ArtifactCodecRegistry, codec contracts.ArtifactCodec[T]) error {
|
||||||
|
if registry == nil {
|
||||||
|
return fmt.Errorf("artifact codec registry must not be nil")
|
||||||
|
}
|
||||||
|
if nilInterface(codec) {
|
||||||
|
return fmt.Errorf("artifact codec must not be nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
spec, err := artifactCodecSpec(codec.Kind(), codec.Schema(), codec.MediaType())
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if _, ok := registry.entries[spec.Kind]; ok {
|
||||||
|
return fmt.Errorf("artifact codec %q is already registered", spec.Kind)
|
||||||
|
}
|
||||||
|
|
||||||
|
valueType := reflect.TypeFor[T]()
|
||||||
|
entry := artifactCodecEntry{
|
||||||
|
spec: cloneArtifactCodecSpec(spec),
|
||||||
|
valueType: valueType,
|
||||||
|
encode: func(value any) ([]byte, error) {
|
||||||
|
actualType := reflect.TypeOf(value)
|
||||||
|
if actualType != valueType {
|
||||||
|
return nil, newArtifactCodecTypeError("encode", spec.Kind, valueType, actualType)
|
||||||
|
}
|
||||||
|
typed, ok := value.(T)
|
||||||
|
if !ok {
|
||||||
|
return nil, newArtifactCodecTypeError("encode", spec.Kind, valueType, actualType)
|
||||||
|
}
|
||||||
|
encoded, err := codec.Encode(typed)
|
||||||
|
if err != nil {
|
||||||
|
return nil, &ArtifactCodecOperationError{Operation: "encode", Kind: spec.Kind, Err: err}
|
||||||
|
}
|
||||||
|
return append([]byte(nil), encoded...), nil
|
||||||
|
},
|
||||||
|
decode: func(content []byte) (any, error) {
|
||||||
|
decoded, err := codec.Decode(append([]byte(nil), content...))
|
||||||
|
if err != nil {
|
||||||
|
return nil, &ArtifactCodecOperationError{Operation: "decode", Kind: spec.Kind, Err: err}
|
||||||
|
}
|
||||||
|
return decoded, nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
entry.encodeCandidate = entry.encode
|
||||||
|
if candidate, ok := any(codec).(interface{ EncodeCandidate(T) ([]byte, error) }); ok {
|
||||||
|
entry.encodeCandidate = func(value any) ([]byte, error) {
|
||||||
|
typed, err := exactTypedValue[T]("encode candidate artifact", value)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
content, err := candidate.EncodeCandidate(typed)
|
||||||
|
if err != nil {
|
||||||
|
return nil, &ArtifactCodecOperationError{Operation: "encode", Kind: spec.Kind, Err: err}
|
||||||
|
}
|
||||||
|
return append([]byte(nil), content...), nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if provider, ok := any(codec).(interface{ Metadata(T) map[string]any }); ok {
|
||||||
|
entry.metadata = func(value any) map[string]any {
|
||||||
|
typed, err := exactTypedValue[T]("artifact metadata", value)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return cloneMetadata(provider.Metadata(typed))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if registry.entries == nil {
|
||||||
|
registry.entries = make(map[contracts.ArtifactKind]artifactCodecEntry)
|
||||||
|
}
|
||||||
|
registry.entries[spec.Kind] = entry
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ArtifactCodecRegistry) Spec(kind contracts.ArtifactKind) (ArtifactCodecSpec, bool) {
|
||||||
|
if r == nil {
|
||||||
|
return ArtifactCodecSpec{}, false
|
||||||
|
}
|
||||||
|
entry, ok := r.entries[normalizeArtifactKind(kind)]
|
||||||
|
if !ok {
|
||||||
|
return ArtifactCodecSpec{}, false
|
||||||
|
}
|
||||||
|
return cloneArtifactCodecSpec(entry.spec), true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ArtifactCodecRegistry) valueType(kind contracts.ArtifactKind) (reflect.Type, bool) {
|
||||||
|
if r == nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
entry, ok := r.entries[normalizeArtifactKind(kind)]
|
||||||
|
if !ok {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
return entry.valueType, true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ArtifactCodecRegistry) RegisteredKinds() []contracts.ArtifactKind {
|
||||||
|
if r == nil || len(r.entries) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
kinds := make([]contracts.ArtifactKind, 0, len(r.entries))
|
||||||
|
for kind := range r.entries {
|
||||||
|
kinds = append(kinds, kind)
|
||||||
|
}
|
||||||
|
sort.Slice(kinds, func(i, j int) bool { return kinds[i] < kinds[j] })
|
||||||
|
return kinds
|
||||||
|
}
|
||||||
|
|
||||||
|
// Encode serializes an erased framework value after proving its exact
|
||||||
|
// registered Go type.
|
||||||
|
func (r *ArtifactCodecRegistry) Encode(kind contracts.ArtifactKind, value any) (contracts.SerializedArtifact, error) {
|
||||||
|
entry, normalizedKind, err := r.entry(kind)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.SerializedArtifact{}, err
|
||||||
|
}
|
||||||
|
content, err := entry.encode(value)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.SerializedArtifact{}, err
|
||||||
|
}
|
||||||
|
return contracts.SerializedArtifact{
|
||||||
|
Kind: normalizedKind,
|
||||||
|
Schema: contracts.CloneArtifactSchema(entry.spec.Schema),
|
||||||
|
MediaType: entry.spec.MediaType,
|
||||||
|
Content: content,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Decode verifies serialized identity before invoking the registered codec.
|
||||||
|
func (r *ArtifactCodecRegistry) Decode(artifact contracts.SerializedArtifact) (any, error) {
|
||||||
|
entry, normalizedKind, err := r.entry(artifact.Kind)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if artifact.Schema.ID != entry.spec.Schema.ID || artifact.Schema.Name != entry.spec.Schema.Name || artifact.Schema.Version != entry.spec.Schema.Version {
|
||||||
|
return nil, &ArtifactCodecCompatibilityError{Kind: normalizedKind, Reason: "schema identity does not match registered codec"}
|
||||||
|
}
|
||||||
|
if contracts.DigestArtifactSchema(artifact.Schema) != entry.spec.SchemaDigest {
|
||||||
|
return nil, &ArtifactCodecCompatibilityError{Kind: normalizedKind, Reason: "schema digest does not match registered codec"}
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(artifact.MediaType) != entry.spec.MediaType {
|
||||||
|
return nil, &ArtifactCodecCompatibilityError{Kind: normalizedKind, Reason: "media type does not match registered codec"}
|
||||||
|
}
|
||||||
|
return entry.decode(artifact.Content)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ArtifactCodecRegistry) entry(kind contracts.ArtifactKind) (artifactCodecEntry, contracts.ArtifactKind, error) {
|
||||||
|
if r == nil {
|
||||||
|
return artifactCodecEntry{}, "", fmt.Errorf("artifact codec registry must not be nil")
|
||||||
|
}
|
||||||
|
normalizedKind := normalizeArtifactKind(kind)
|
||||||
|
if normalizedKind == "" {
|
||||||
|
return artifactCodecEntry{}, "", fmt.Errorf("artifact kind must not be empty")
|
||||||
|
}
|
||||||
|
entry, ok := r.entries[normalizedKind]
|
||||||
|
if !ok {
|
||||||
|
return artifactCodecEntry{}, normalizedKind, fmt.Errorf("artifact codec %q is not registered", normalizedKind)
|
||||||
|
}
|
||||||
|
return entry, normalizedKind, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func artifactCodecSpec(kind contracts.ArtifactKind, schema contracts.ArtifactSchema, mediaType string) (ArtifactCodecSpec, error) {
|
||||||
|
kind = normalizeArtifactKind(kind)
|
||||||
|
schema.ID = strings.TrimSpace(schema.ID)
|
||||||
|
schema.Name = strings.TrimSpace(schema.Name)
|
||||||
|
schema.Version = strings.TrimSpace(schema.Version)
|
||||||
|
mediaType = strings.TrimSpace(mediaType)
|
||||||
|
switch {
|
||||||
|
case kind == "":
|
||||||
|
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec kind must not be empty")
|
||||||
|
case schema.ID == "":
|
||||||
|
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q schema id must not be empty", kind)
|
||||||
|
case schema.Name == "":
|
||||||
|
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q schema name must not be empty", kind)
|
||||||
|
case schema.Version == "":
|
||||||
|
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q schema version must not be empty", kind)
|
||||||
|
case mediaType == "":
|
||||||
|
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q media type must not be empty", kind)
|
||||||
|
case len(bytes.TrimSpace(schema.JSONSchema)) == 0:
|
||||||
|
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema must not be empty", kind)
|
||||||
|
case !json.Valid(schema.JSONSchema):
|
||||||
|
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema must be valid JSON", kind)
|
||||||
|
}
|
||||||
|
schemaDocument, err := jsonschema.UnmarshalJSON(bytes.NewReader(schema.JSONSchema))
|
||||||
|
if err != nil {
|
||||||
|
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema is invalid: %w", kind, err)
|
||||||
|
}
|
||||||
|
compiler := jsonschema.NewCompiler()
|
||||||
|
if err := compiler.AddResource("artifact-schema.json", schemaDocument); err != nil {
|
||||||
|
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema is invalid: %w", kind, err)
|
||||||
|
}
|
||||||
|
if _, err := compiler.Compile("artifact-schema.json"); err != nil {
|
||||||
|
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema is invalid: %w", kind, err)
|
||||||
|
}
|
||||||
|
schema = contracts.CloneArtifactSchema(schema)
|
||||||
|
return ArtifactCodecSpec{
|
||||||
|
Kind: kind,
|
||||||
|
Schema: schema,
|
||||||
|
SchemaDigest: contracts.DigestArtifactSchema(schema),
|
||||||
|
MediaType: mediaType,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeArtifactKind(kind contracts.ArtifactKind) contracts.ArtifactKind {
|
||||||
|
return contracts.ArtifactKind(strings.TrimSpace(string(kind)))
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneArtifactCodecSpec(spec ArtifactCodecSpec) ArtifactCodecSpec {
|
||||||
|
spec.Schema = contracts.CloneArtifactSchema(spec.Schema)
|
||||||
|
return spec
|
||||||
|
}
|
||||||
|
|
||||||
|
func nilInterface(value any) bool {
|
||||||
|
if value == nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
reflected := reflect.ValueOf(value)
|
||||||
|
switch reflected.Kind() {
|
||||||
|
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
|
||||||
|
return reflected.IsNil()
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newArtifactCodecTypeError(operation string, kind contracts.ArtifactKind, expected reflect.Type, actual reflect.Type) error {
|
||||||
|
expectedName := "<nil>"
|
||||||
|
if expected != nil {
|
||||||
|
expectedName = expected.String()
|
||||||
|
}
|
||||||
|
actualName := "<nil>"
|
||||||
|
if actual != nil {
|
||||||
|
actualName = actual.String()
|
||||||
|
}
|
||||||
|
return &ArtifactCodecTypeError{
|
||||||
|
Operation: operation,
|
||||||
|
Kind: kind,
|
||||||
|
ExpectedType: expectedName,
|
||||||
|
ActualType: actualName,
|
||||||
|
}
|
||||||
|
}
|
||||||
307
internal/framework/pipeline/artifact_codec_registry_test.go
Normal file
307
internal/framework/pipeline/artifact_codec_registry_test.go
Normal file
@@ -0,0 +1,307 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
type codecNotes struct {
|
||||||
|
Items []string `json:"items"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type codecScore struct {
|
||||||
|
Value int `json:"value"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type codecNotesAlias codecNotes
|
||||||
|
|
||||||
|
type testArtifactCodec[T any] struct {
|
||||||
|
kind contracts.ArtifactKind
|
||||||
|
schema contracts.ArtifactSchema
|
||||||
|
mediaType string
|
||||||
|
encodeFunc func(T) ([]byte, error)
|
||||||
|
decodeFunc func([]byte) (T, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c testArtifactCodec[T]) Kind() contracts.ArtifactKind { return c.kind }
|
||||||
|
func (c testArtifactCodec[T]) Schema() contracts.ArtifactSchema { return c.schema }
|
||||||
|
func (c testArtifactCodec[T]) MediaType() string { return c.mediaType }
|
||||||
|
func (c testArtifactCodec[T]) Encode(value T) ([]byte, error) { return c.encodeFunc(value) }
|
||||||
|
func (c testArtifactCodec[T]) Decode(content []byte) (T, error) { return c.decodeFunc(content) }
|
||||||
|
|
||||||
|
func TestArtifactCodecRegistryStoresHeterogeneousExactTypes(t *testing.T) {
|
||||||
|
registry := NewArtifactCodecRegistry()
|
||||||
|
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
|
||||||
|
t.Fatalf("RegisterArtifactCodec(notes) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if err := RegisterArtifactCodec(registry, scoreCodec()); err != nil {
|
||||||
|
t.Fatalf("RegisterArtifactCodec(score) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got, want := registry.RegisteredKinds(), []contracts.ArtifactKind{"test/notes", "test/score"}; !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("RegisteredKinds() = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
notes := codecNotes{Items: []string{"second", "first"}}
|
||||||
|
first, err := registry.Encode("test/notes", notes)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Encode(notes) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
second, err := registry.Encode(" test/notes ", notes)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Encode(notes again) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(first.Content, second.Content) {
|
||||||
|
t.Fatalf("equal values encoded as %q and %q, want deterministic bytes", first.Content, second.Content)
|
||||||
|
}
|
||||||
|
decodedNotes, err := registry.Decode(first)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Decode(notes) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(decodedNotes, notes) {
|
||||||
|
t.Fatalf("Decode(notes) = %#v, want %#v", decodedNotes, notes)
|
||||||
|
}
|
||||||
|
|
||||||
|
score := codecScore{Value: 17}
|
||||||
|
serializedScore, err := registry.Encode("test/score", score)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Encode(score) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
decodedScore, err := registry.Decode(serializedScore)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Decode(score) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if decodedScore != score {
|
||||||
|
t.Fatalf("Decode(score) = %#v, want %#v", decodedScore, score)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = registry.Encode("test/notes", codecNotesAlias(notes))
|
||||||
|
var typeErr *ArtifactCodecTypeError
|
||||||
|
if !errors.As(err, &typeErr) {
|
||||||
|
t.Fatalf("Encode(alias) error = %T %v, want ArtifactCodecTypeError", err, err)
|
||||||
|
}
|
||||||
|
if typeErr.ExpectedType == typeErr.ActualType {
|
||||||
|
t.Fatalf("type error = %#v, want distinct exact types", typeErr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestArtifactCodecRegistryStoresValidatedSchemaMetadata(t *testing.T) {
|
||||||
|
registry := NewArtifactCodecRegistry()
|
||||||
|
codec := notesCodec()
|
||||||
|
codec.kind = " test/notes "
|
||||||
|
codec.schema.ID = " notes.v1 "
|
||||||
|
codec.schema.Name = " notes "
|
||||||
|
codec.schema.Version = " v1 "
|
||||||
|
codec.mediaType = " application/json "
|
||||||
|
if err := RegisterArtifactCodec(registry, codec); err != nil {
|
||||||
|
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
codec.schema.JSONSchema[0] = '['
|
||||||
|
|
||||||
|
spec, ok := registry.Spec("test/notes")
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("Spec() ok = false, want true")
|
||||||
|
}
|
||||||
|
if spec.Kind != "test/notes" || spec.Schema.ID != "notes.v1" || spec.Schema.Name != "notes" || spec.Schema.Version != "v1" || spec.MediaType != "application/json" {
|
||||||
|
t.Fatalf("Spec() = %#v, want normalized metadata", spec)
|
||||||
|
}
|
||||||
|
if spec.SchemaDigest != contracts.DigestArtifactSchema(spec.Schema) {
|
||||||
|
t.Fatalf("schema digest = %q, want %q", spec.SchemaDigest, contracts.DigestArtifactSchema(spec.Schema))
|
||||||
|
}
|
||||||
|
|
||||||
|
spec.Schema.JSONSchema[0] = '['
|
||||||
|
again, _ := registry.Spec("test/notes")
|
||||||
|
if string(again.Schema.JSONSchema) != `{"additionalProperties":false,"properties":{"items":{"items":{"type":"string"},"type":"array"}},"required":["items"],"type":"object"}` {
|
||||||
|
t.Fatalf("stored JSON Schema changed through Spec result: %q", again.Schema.JSONSchema)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestArtifactCodecRegistryRejectsInvalidRegistration(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*testArtifactCodec[codecNotes])
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "kind", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.kind = " " }, want: "kind"},
|
||||||
|
{name: "schema id", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.ID = "" }, want: "schema id"},
|
||||||
|
{name: "schema name", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.Name = "" }, want: "schema name"},
|
||||||
|
{name: "schema version", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.Version = "" }, want: "schema version"},
|
||||||
|
{name: "media type", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.mediaType = "" }, want: "media type"},
|
||||||
|
{name: "empty JSON Schema", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.JSONSchema = nil }, want: "JSON Schema"},
|
||||||
|
{name: "invalid JSON Schema", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.JSONSchema = []byte(`{"type":`) }, want: "valid JSON"},
|
||||||
|
{name: "non-schema JSON", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.JSONSchema = []byte(`[]`) }, want: "JSON Schema is invalid"},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
registry := NewArtifactCodecRegistry()
|
||||||
|
codec := notesCodec()
|
||||||
|
test.mutate(&codec)
|
||||||
|
if err := RegisterArtifactCodec(registry, codec); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("RegisterArtifactCodec() error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestArtifactCodecRegistryRejectsDuplicateKind(t *testing.T) {
|
||||||
|
registry := NewArtifactCodecRegistry()
|
||||||
|
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
|
||||||
|
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
duplicate := notesCodec()
|
||||||
|
duplicate.kind = " test/notes "
|
||||||
|
if err := RegisterArtifactCodec(registry, duplicate); err == nil || !strings.Contains(err.Error(), "already registered") {
|
||||||
|
t.Fatalf("duplicate registration error = %v, want duplicate kind error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestArtifactCodecRegistryRejectsNilRegistryAndCodec(t *testing.T) {
|
||||||
|
codec := notesCodec()
|
||||||
|
if err := RegisterArtifactCodec[codecNotes](nil, codec); err == nil || !strings.Contains(err.Error(), "registry") {
|
||||||
|
t.Fatalf("nil registry error = %v, want registry error", err)
|
||||||
|
}
|
||||||
|
var nilCodec *testArtifactCodec[codecNotes]
|
||||||
|
if err := RegisterArtifactCodec(NewArtifactCodecRegistry(), nilCodec); err == nil || !strings.Contains(err.Error(), "must not be nil") {
|
||||||
|
t.Fatalf("nil codec error = %v, want codec error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestArtifactCodecRegistryStrictDecodeAndTypedFailures(t *testing.T) {
|
||||||
|
registry := NewArtifactCodecRegistry()
|
||||||
|
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
|
||||||
|
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
valid, err := registry.Encode("test/notes", codecNotes{Items: []string{"one"}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Encode() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, content := range [][]byte{
|
||||||
|
[]byte(`{"items":["one"],"unknown":true}`),
|
||||||
|
[]byte(`{"items":["one"]} {}`),
|
||||||
|
} {
|
||||||
|
candidate := contracts.CloneSerializedArtifact(valid)
|
||||||
|
candidate.Content = content
|
||||||
|
_, err := registry.Decode(candidate)
|
||||||
|
var operationErr *ArtifactCodecOperationError
|
||||||
|
if !errors.As(err, &operationErr) || operationErr.Operation != "decode" {
|
||||||
|
t.Fatalf("Decode(%q) error = %T %v, want typed decode error", content, err, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
wrongSchema := contracts.CloneSerializedArtifact(valid)
|
||||||
|
wrongSchema.Schema.Version = "v2"
|
||||||
|
_, err = registry.Decode(wrongSchema)
|
||||||
|
var compatibilityErr *ArtifactCodecCompatibilityError
|
||||||
|
if !errors.As(err, &compatibilityErr) {
|
||||||
|
t.Fatalf("Decode(wrong schema) error = %T %v, want ArtifactCodecCompatibilityError", err, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestArtifactCodecRegistryWrapsEncodeFailure(t *testing.T) {
|
||||||
|
codec := notesCodec()
|
||||||
|
cause := errors.New("cannot encode notes")
|
||||||
|
codec.encodeFunc = func(codecNotes) ([]byte, error) { return nil, cause }
|
||||||
|
registry := NewArtifactCodecRegistry()
|
||||||
|
if err := RegisterArtifactCodec(registry, codec); err != nil {
|
||||||
|
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := registry.Encode("test/notes", codecNotes{})
|
||||||
|
var operationErr *ArtifactCodecOperationError
|
||||||
|
if !errors.As(err, &operationErr) || operationErr.Operation != "encode" || !errors.Is(err, cause) {
|
||||||
|
t.Fatalf("Encode() error = %T %v, want typed wrapping encode error", err, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestArtifactCodecRegistryClonesCodecBytes(t *testing.T) {
|
||||||
|
shared := []byte(`{"items":["one"]}`)
|
||||||
|
codec := notesCodec()
|
||||||
|
codec.encodeFunc = func(codecNotes) ([]byte, error) { return shared, nil }
|
||||||
|
codec.decodeFunc = func(content []byte) (codecNotes, error) {
|
||||||
|
content[0] = '['
|
||||||
|
return codecNotes{Items: []string{"one"}}, nil
|
||||||
|
}
|
||||||
|
registry := NewArtifactCodecRegistry()
|
||||||
|
if err := RegisterArtifactCodec(registry, codec); err != nil {
|
||||||
|
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
artifact, err := registry.Encode("test/notes", codecNotes{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Encode() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
shared[0] = '['
|
||||||
|
if string(artifact.Content) != `{"items":["one"]}` {
|
||||||
|
t.Fatalf("encoded content = %q after codec buffer mutation, want owned bytes", artifact.Content)
|
||||||
|
}
|
||||||
|
before := append([]byte(nil), artifact.Content...)
|
||||||
|
if _, err := registry.Decode(artifact); err != nil {
|
||||||
|
t.Fatalf("Decode() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(artifact.Content, before) {
|
||||||
|
t.Fatalf("serialized content changed during decode: %q", artifact.Content)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func notesCodec() testArtifactCodec[codecNotes] {
|
||||||
|
return testArtifactCodec[codecNotes]{
|
||||||
|
kind: "test/notes",
|
||||||
|
schema: contracts.ArtifactSchema{
|
||||||
|
ID: "notes.v1",
|
||||||
|
Name: "notes",
|
||||||
|
Version: "v1",
|
||||||
|
JSONSchema: []byte(`{"additionalProperties":false,"properties":{"items":{"items":{"type":"string"},"type":"array"}},"required":["items"],"type":"object"}`),
|
||||||
|
},
|
||||||
|
mediaType: "application/json",
|
||||||
|
encodeFunc: func(value codecNotes) ([]byte, error) {
|
||||||
|
return json.Marshal(value)
|
||||||
|
},
|
||||||
|
decodeFunc: func(content []byte) (codecNotes, error) {
|
||||||
|
var value codecNotes
|
||||||
|
return value, decodeStrictJSON(content, &value)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func scoreCodec() testArtifactCodec[codecScore] {
|
||||||
|
return testArtifactCodec[codecScore]{
|
||||||
|
kind: "test/score",
|
||||||
|
schema: contracts.ArtifactSchema{
|
||||||
|
ID: "score.v1",
|
||||||
|
Name: "score",
|
||||||
|
Version: "v1",
|
||||||
|
JSONSchema: []byte(`{"additionalProperties":false,"properties":{"value":{"type":"integer"}},"required":["value"],"type":"object"}`),
|
||||||
|
},
|
||||||
|
mediaType: "application/json",
|
||||||
|
encodeFunc: func(value codecScore) ([]byte, error) {
|
||||||
|
return json.Marshal(value)
|
||||||
|
},
|
||||||
|
decodeFunc: func(content []byte) (codecScore, error) {
|
||||||
|
var value codecScore
|
||||||
|
return value, decodeStrictJSON(content, &value)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeStrictJSON(content []byte, out any) error {
|
||||||
|
decoder := json.NewDecoder(bytes.NewReader(content))
|
||||||
|
decoder.DisallowUnknownFields()
|
||||||
|
if err := decoder.Decode(out); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := decoder.Decode(&struct{}{}); err != io.EOF {
|
||||||
|
if err == nil {
|
||||||
|
return fmt.Errorf("unexpected trailing JSON value")
|
||||||
|
}
|
||||||
|
return fmt.Errorf("decode trailing JSON: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -21,18 +21,18 @@ type CheckpointRecorder interface {
|
|||||||
SourceSucceeded(moduleKey string, doc *source.SourceDocument) error
|
SourceSucceeded(moduleKey string, doc *source.SourceDocument) error
|
||||||
SourceFailed(moduleKey string, err error) error
|
SourceFailed(moduleKey string, err error) error
|
||||||
ChunkRunning(moduleKey string, sourceDigest string) error
|
ChunkRunning(moduleKey string, sourceDigest string) error
|
||||||
ChunkSucceeded(moduleKey string, sourceDigest string, chunks []contracts.SourceChunk, warnings []contracts.Warning) error
|
ChunkSucceeded(moduleKey string, sourceDigest string, chunks []source.Chunk, warnings []contracts.Warning) error
|
||||||
ChunkRejected(moduleKey string, sourceDigest string, rejected contracts.RejectedOutput) error
|
ChunkRejected(moduleKey string, sourceDigest string, rejected contracts.RejectedOutput) error
|
||||||
ChunkFailed(moduleKey string, sourceDigest string, err error) error
|
ChunkFailed(moduleKey string, sourceDigest string, err error) error
|
||||||
ExtractRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
ExtractRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||||
ExtractSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, outputs []contracts.ExtractOutput, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error
|
ExtractSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error
|
||||||
ExtractFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
ExtractFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||||
MergeRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
MergeRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||||
MergeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output contracts.MergeOutput, warnings []contracts.Warning) error
|
MergeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
|
||||||
MergeRejected(laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
MergeRejected(laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
||||||
MergeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
MergeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||||
NormalizeRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
NormalizeRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||||
NormalizeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output contracts.NormalizeOutput, warnings []contracts.Warning) error
|
NormalizeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
|
||||||
NormalizeRejected(laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
NormalizeRejected(laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
||||||
NormalizeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
NormalizeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||||
}
|
}
|
||||||
@@ -55,23 +55,35 @@ type SourceCheckpoint struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type ChunkCheckpoint struct {
|
type ChunkCheckpoint struct {
|
||||||
Chunks []contracts.SourceChunk
|
Chunks []source.Chunk
|
||||||
Warnings []contracts.Warning
|
Warnings []contracts.Warning
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// CheckpointArtifact is the durable, domain-neutral value stored at a lane
|
||||||
|
// checkpoint boundary.
|
||||||
|
type CheckpointArtifact struct {
|
||||||
|
LaneID string
|
||||||
|
ModuleKey string
|
||||||
|
SourceID string
|
||||||
|
ChunkID string
|
||||||
|
ChunkIndex int
|
||||||
|
ChunkRef source.SourceRef
|
||||||
|
Artifact contracts.SerializedArtifact
|
||||||
|
SchemaDigest string
|
||||||
|
}
|
||||||
|
|
||||||
type ExtractCheckpoint struct {
|
type ExtractCheckpoint struct {
|
||||||
Outputs []contracts.ExtractOutput
|
Outputs []CheckpointArtifact
|
||||||
Rejected []contracts.RejectedOutput
|
Rejected []contracts.RejectedOutput
|
||||||
Warnings []contracts.Warning
|
Warnings []contracts.Warning
|
||||||
}
|
}
|
||||||
|
|
||||||
type MergeCheckpoint struct {
|
type MergeCheckpoint struct {
|
||||||
Output contracts.MergeOutput
|
Output CheckpointArtifact
|
||||||
Warnings []contracts.Warning
|
Warnings []contracts.Warning
|
||||||
}
|
}
|
||||||
|
|
||||||
type NormalizeCheckpoint struct {
|
type NormalizeCheckpoint struct {
|
||||||
Output contracts.NormalizeOutput
|
Output CheckpointArtifact
|
||||||
Warnings []contracts.Warning
|
Warnings []contracts.Warning
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -94,7 +106,7 @@ func (noopCheckpointRecorder) SourceRunning(string) error
|
|||||||
func (noopCheckpointRecorder) SourceSucceeded(string, *source.SourceDocument) error { return nil }
|
func (noopCheckpointRecorder) SourceSucceeded(string, *source.SourceDocument) error { return nil }
|
||||||
func (noopCheckpointRecorder) SourceFailed(string, error) error { return nil }
|
func (noopCheckpointRecorder) SourceFailed(string, error) error { return nil }
|
||||||
func (noopCheckpointRecorder) ChunkRunning(string, string) error { return nil }
|
func (noopCheckpointRecorder) ChunkRunning(string, string) error { return nil }
|
||||||
func (noopCheckpointRecorder) ChunkSucceeded(string, string, []contracts.SourceChunk, []contracts.Warning) error {
|
func (noopCheckpointRecorder) ChunkSucceeded(string, string, []source.Chunk, []contracts.Warning) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) ChunkRejected(string, string, contracts.RejectedOutput) error {
|
func (noopCheckpointRecorder) ChunkRejected(string, string, contracts.RejectedOutput) error {
|
||||||
@@ -104,14 +116,14 @@ func (noopCheckpointRecorder) ChunkFailed(string, string, error) error { return
|
|||||||
func (noopCheckpointRecorder) ExtractRunning(string, string, []CheckpointFingerprint) error {
|
func (noopCheckpointRecorder) ExtractRunning(string, string, []CheckpointFingerprint) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) ExtractSucceeded(string, string, []CheckpointFingerprint, []contracts.ExtractOutput, []contracts.RejectedOutput, []contracts.Warning) error {
|
func (noopCheckpointRecorder) ExtractSucceeded(string, string, []CheckpointFingerprint, []CheckpointArtifact, []contracts.RejectedOutput, []contracts.Warning) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) ExtractFailed(string, string, []CheckpointFingerprint, error) error {
|
func (noopCheckpointRecorder) ExtractFailed(string, string, []CheckpointFingerprint, error) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) MergeRunning(string, string, []CheckpointFingerprint) error { return nil }
|
func (noopCheckpointRecorder) MergeRunning(string, string, []CheckpointFingerprint) error { return nil }
|
||||||
func (noopCheckpointRecorder) MergeSucceeded(string, string, []CheckpointFingerprint, contracts.MergeOutput, []contracts.Warning) error {
|
func (noopCheckpointRecorder) MergeSucceeded(string, string, []CheckpointFingerprint, CheckpointArtifact, []contracts.Warning) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) MergeRejected(string, string, []CheckpointFingerprint, contracts.RejectedOutput) error {
|
func (noopCheckpointRecorder) MergeRejected(string, string, []CheckpointFingerprint, contracts.RejectedOutput) error {
|
||||||
@@ -123,7 +135,7 @@ func (noopCheckpointRecorder) MergeFailed(string, string, []CheckpointFingerprin
|
|||||||
func (noopCheckpointRecorder) NormalizeRunning(string, string, []CheckpointFingerprint) error {
|
func (noopCheckpointRecorder) NormalizeRunning(string, string, []CheckpointFingerprint) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) NormalizeSucceeded(string, string, []CheckpointFingerprint, contracts.NormalizeOutput, []contracts.Warning) error {
|
func (noopCheckpointRecorder) NormalizeSucceeded(string, string, []CheckpointFingerprint, CheckpointArtifact, []contracts.Warning) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) NormalizeRejected(string, string, []CheckpointFingerprint, contracts.RejectedOutput) error {
|
func (noopCheckpointRecorder) NormalizeRejected(string, string, []CheckpointFingerprint, contracts.RejectedOutput) error {
|
||||||
@@ -150,28 +162,6 @@ func (noopCheckpointLoader) Normalize(string, string, []CheckpointFingerprint) (
|
|||||||
return NormalizeCheckpoint{}, CheckpointDecision{Reason: "checkpoint loading disabled"}
|
return NormalizeCheckpoint{}, CheckpointDecision{Reason: "checkpoint loading disabled"}
|
||||||
}
|
}
|
||||||
|
|
||||||
func rawOutputDigests(payloads []contracts.RawPayload) []CheckpointFingerprint {
|
|
||||||
values := make([]CheckpointFingerprint, 0, len(payloads))
|
|
||||||
for i, payload := range payloads {
|
|
||||||
values = append(values, CheckpointFingerprint{
|
|
||||||
Name: fmt.Sprintf("payload[%d]", i),
|
|
||||||
Value: checkpointContentDigest(payload.Content),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
return normalizeCheckpointFingerprints(values)
|
|
||||||
}
|
|
||||||
|
|
||||||
func extractPayloads(outputs []contracts.ExtractOutput) []contracts.RawPayload {
|
|
||||||
if len(outputs) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
payloads := make([]contracts.RawPayload, 0, len(outputs))
|
|
||||||
for _, output := range outputs {
|
|
||||||
payloads = append(payloads, output.Payload)
|
|
||||||
}
|
|
||||||
return payloads
|
|
||||||
}
|
|
||||||
|
|
||||||
func digestFingerprints(name string, digest string) []CheckpointFingerprint {
|
func digestFingerprints(name string, digest string) []CheckpointFingerprint {
|
||||||
digest = strings.TrimSpace(digest)
|
digest = strings.TrimSpace(digest)
|
||||||
if digest == "" {
|
if digest == "" {
|
||||||
@@ -180,17 +170,21 @@ func digestFingerprints(name string, digest string) []CheckpointFingerprint {
|
|||||||
return []CheckpointFingerprint{{Name: name, Value: digest}}
|
return []CheckpointFingerprint{{Name: name, Value: digest}}
|
||||||
}
|
}
|
||||||
|
|
||||||
func joinedChunkDigest(chunks []contracts.SourceChunk) string {
|
func joinedChunkDigest(chunks []source.Chunk) (string, error) {
|
||||||
if len(chunks) == 0 {
|
if len(chunks) == 0 {
|
||||||
return ""
|
return "", nil
|
||||||
}
|
}
|
||||||
values := make([]string, 0, len(chunks))
|
values := make([]string, 0, len(chunks))
|
||||||
for _, chunk := range chunks {
|
for _, chunk := range chunks {
|
||||||
values = append(values, chunk.ID+"="+checkpointContentDigest(chunk.Content))
|
digest, err := source.DigestChunk(chunk)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("digest chunk %q: %w", chunk.ID, err)
|
||||||
|
}
|
||||||
|
values = append(values, chunk.ID+"="+digest)
|
||||||
}
|
}
|
||||||
sort.Strings(values)
|
sort.Strings(values)
|
||||||
sum := sha256.Sum256([]byte(strings.Join(values, "\n")))
|
sum := sha256.Sum256([]byte(strings.Join(values, "\n")))
|
||||||
return "sha256:" + hex.EncodeToString(sum[:])
|
return "sha256:" + hex.EncodeToString(sum[:]), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeCheckpointFingerprints(values []CheckpointFingerprint) []CheckpointFingerprint {
|
func normalizeCheckpointFingerprints(values []CheckpointFingerprint) []CheckpointFingerprint {
|
||||||
|
|||||||
@@ -5,10 +5,9 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []contracts.SourceChunk) ([]contracts.SourceChunk, error) {
|
func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []source.Chunk) ([]source.Chunk, error) {
|
||||||
if len(chunks) == 0 {
|
if len(chunks) == 0 {
|
||||||
return nil, fmt.Errorf("chunks must not be empty")
|
return nil, fmt.Errorf("chunks must not be empty")
|
||||||
}
|
}
|
||||||
@@ -20,7 +19,7 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
|
|||||||
sourceUnits[unit.ID] = unit
|
sourceUnits[unit.ID] = unit
|
||||||
}
|
}
|
||||||
|
|
||||||
canonicalChunks := make([]contracts.SourceChunk, 0, len(chunks))
|
canonicalChunks := make([]source.Chunk, 0, len(chunks))
|
||||||
seenChunkIDs := make(map[string]struct{}, len(chunks))
|
seenChunkIDs := make(map[string]struct{}, len(chunks))
|
||||||
for chunkIndex, chunk := range chunks {
|
for chunkIndex, chunk := range chunks {
|
||||||
if strings.TrimSpace(chunk.ID) == "" {
|
if strings.TrimSpace(chunk.ID) == "" {
|
||||||
@@ -37,17 +36,6 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
|
|||||||
if chunk.Index != chunkIndex {
|
if chunk.Index != chunkIndex {
|
||||||
return nil, fmt.Errorf("chunk %q index %d does not match returned order %d", chunk.ID, chunk.Index, chunkIndex)
|
return nil, fmt.Errorf("chunk %q index %d does not match returned order %d", chunk.ID, chunk.Index, chunkIndex)
|
||||||
}
|
}
|
||||||
startIndex, ok := sourceUnitIndexes[chunk.StartUnitID]
|
|
||||||
if !ok {
|
|
||||||
return nil, fmt.Errorf("chunk %q start_unit_id %d was not found in source document %q", chunk.ID, chunk.StartUnitID, doc.ID)
|
|
||||||
}
|
|
||||||
endIndex, ok := sourceUnitIndexes[chunk.EndUnitID]
|
|
||||||
if !ok {
|
|
||||||
return nil, fmt.Errorf("chunk %q end_unit_id %d was not found in source document %q", chunk.ID, chunk.EndUnitID, doc.ID)
|
|
||||||
}
|
|
||||||
if startIndex > endIndex {
|
|
||||||
return nil, fmt.Errorf("chunk %q start_unit_id %d appears after end_unit_id %d", chunk.ID, chunk.StartUnitID, chunk.EndUnitID)
|
|
||||||
}
|
|
||||||
if len(chunk.Units) == 0 {
|
if len(chunk.Units) == 0 {
|
||||||
return nil, fmt.Errorf("chunk %q units must not be empty", chunk.ID)
|
return nil, fmt.Errorf("chunk %q units must not be empty", chunk.ID)
|
||||||
}
|
}
|
||||||
@@ -57,6 +45,9 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
|
|||||||
if strings.TrimSpace(chunk.MediaType) == "" {
|
if strings.TrimSpace(chunk.MediaType) == "" {
|
||||||
return nil, fmt.Errorf("chunk %q media_type must not be empty", chunk.ID)
|
return nil, fmt.Errorf("chunk %q media_type must not be empty", chunk.ID)
|
||||||
}
|
}
|
||||||
|
if err := source.ValidateRef(doc, chunk.Ref); err != nil {
|
||||||
|
return nil, fmt.Errorf("chunk %q ref: %w", chunk.ID, err)
|
||||||
|
}
|
||||||
|
|
||||||
seenUnitIDs := make(map[int]struct{}, len(chunk.Units))
|
seenUnitIDs := make(map[int]struct{}, len(chunk.Units))
|
||||||
previousSourceIndex := -1
|
previousSourceIndex := -1
|
||||||
@@ -74,23 +65,33 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
|
|||||||
if !ok {
|
if !ok {
|
||||||
return nil, fmt.Errorf("chunk %q source unit %d was not found in source document %q", chunk.ID, unit.ID, doc.ID)
|
return nil, fmt.Errorf("chunk %q source unit %d was not found in source document %q", chunk.ID, unit.ID, doc.ID)
|
||||||
}
|
}
|
||||||
if sourceIndex <= previousSourceIndex {
|
if previousSourceIndex >= 0 && sourceIndex != previousSourceIndex+1 {
|
||||||
return nil, fmt.Errorf("chunk %q source units must appear in source document order", chunk.ID)
|
return nil, fmt.Errorf("chunk %q source units must form a contiguous range in source document order", chunk.ID)
|
||||||
|
}
|
||||||
|
if unit.Ref != sourceUnits[unit.ID].Ref {
|
||||||
|
return nil, fmt.Errorf("chunk %q source unit %d ref does not match source document", chunk.ID, unit.ID)
|
||||||
}
|
}
|
||||||
previousSourceIndex = sourceIndex
|
previousSourceIndex = sourceIndex
|
||||||
canonicalUnits = append(canonicalUnits, cloneSourceUnit(sourceUnits[unit.ID]))
|
canonicalUnits = append(canonicalUnits, cloneSourceUnit(sourceUnits[unit.ID]))
|
||||||
}
|
}
|
||||||
|
expectedRef := source.SourceRef{
|
||||||
|
SourceID: doc.ID,
|
||||||
|
StartUnitID: canonicalUnits[0].Ref.StartUnitID,
|
||||||
|
EndUnitID: canonicalUnits[len(canonicalUnits)-1].Ref.EndUnitID,
|
||||||
|
}
|
||||||
|
if chunk.Ref != expectedRef {
|
||||||
|
return nil, fmt.Errorf("chunk %q ref %#v does not match unit span %#v", chunk.ID, chunk.Ref, expectedRef)
|
||||||
|
}
|
||||||
|
|
||||||
canonicalChunks = append(canonicalChunks, contracts.SourceChunk{
|
canonicalChunks = append(canonicalChunks, source.Chunk{
|
||||||
ID: chunk.ID,
|
ID: chunk.ID,
|
||||||
SourceID: chunk.SourceID,
|
SourceID: chunk.SourceID,
|
||||||
Index: chunk.Index,
|
Index: chunk.Index,
|
||||||
StartUnitID: chunk.StartUnitID,
|
Ref: expectedRef,
|
||||||
EndUnitID: chunk.EndUnitID,
|
Content: append([]byte(nil), chunk.Content...),
|
||||||
Content: append([]byte(nil), chunk.Content...),
|
MediaType: chunk.MediaType,
|
||||||
MediaType: chunk.MediaType,
|
Units: canonicalUnits,
|
||||||
Units: canonicalUnits,
|
Metadata: cloneMetadata(chunk.Metadata),
|
||||||
Metadata: cloneMetadata(chunk.Metadata),
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -102,6 +103,7 @@ func cloneSourceUnit(unit source.SourceUnit) source.SourceUnit {
|
|||||||
ID: unit.ID,
|
ID: unit.ID,
|
||||||
Kind: unit.Kind,
|
Kind: unit.Kind,
|
||||||
Text: unit.Text,
|
Text: unit.Text,
|
||||||
|
Ref: unit.Ref,
|
||||||
Metadata: cloneMetadata(unit.Metadata),
|
Metadata: cloneMetadata(unit.Metadata),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -8,16 +8,19 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type ChunkerConstructor func() (contracts.Chunker, error)
|
type ChunkerConstructor func() (contracts.Chunker, error)
|
||||||
|
type ChunkerBuilder func(BuildRequest) (contracts.Chunker, error)
|
||||||
|
|
||||||
type ChunkerRegistry struct {
|
type ChunkerRegistry struct {
|
||||||
constructors map[string]ChunkerConstructor
|
builders map[string]ChunkerBuilder
|
||||||
specs map[string]ModuleSpec
|
optionValidators map[string]OptionValidator
|
||||||
|
specs map[string]ModuleSpec
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewChunkerRegistry() *ChunkerRegistry {
|
func NewChunkerRegistry() *ChunkerRegistry {
|
||||||
return &ChunkerRegistry{
|
return &ChunkerRegistry{
|
||||||
constructors: make(map[string]ChunkerConstructor),
|
builders: make(map[string]ChunkerBuilder),
|
||||||
specs: make(map[string]ModuleSpec),
|
optionValidators: make(map[string]OptionValidator),
|
||||||
|
specs: make(map[string]ModuleSpec),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -26,6 +29,15 @@ func (r *ChunkerRegistry) Register(key string, constructor ChunkerConstructor) e
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
|
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
|
||||||
|
if constructor == nil {
|
||||||
|
return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
}
|
||||||
|
return r.RegisterBuilderWithSpec(spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.Chunker, error) {
|
||||||
|
return constructor()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ChunkerRegistry) RegisterBuilderWithSpec(spec ModuleSpec, validateOptions OptionValidator, builder ChunkerBuilder) error {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return fmt.Errorf("chunker registry must not be nil")
|
return fmt.Errorf("chunker registry must not be nil")
|
||||||
}
|
}
|
||||||
@@ -34,25 +46,36 @@ func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerC
|
|||||||
if err := validateModuleSpec("chunker", StageChunk, normalizedSpec); err != nil {
|
if err := validateModuleSpec("chunker", StageChunk, normalizedSpec); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if constructor == nil {
|
if validateOptions == nil {
|
||||||
return fmt.Errorf("chunker constructor for %q must not be nil", normalizedSpec.Key)
|
return fmt.Errorf("chunker option validator for %q must not be nil", normalizedSpec.Key)
|
||||||
}
|
}
|
||||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
if builder == nil {
|
||||||
|
return fmt.Errorf("chunker builder for %q must not be nil", normalizedSpec.Key)
|
||||||
|
}
|
||||||
|
if _, ok := r.builders[normalizedSpec.Key]; ok {
|
||||||
return fmt.Errorf("chunker %q is already registered", normalizedSpec.Key)
|
return fmt.Errorf("chunker %q is already registered", normalizedSpec.Key)
|
||||||
}
|
}
|
||||||
|
|
||||||
if r.constructors == nil {
|
if r.builders == nil {
|
||||||
r.constructors = make(map[string]ChunkerConstructor)
|
r.builders = make(map[string]ChunkerBuilder)
|
||||||
|
}
|
||||||
|
if r.optionValidators == nil {
|
||||||
|
r.optionValidators = make(map[string]OptionValidator)
|
||||||
}
|
}
|
||||||
if r.specs == nil {
|
if r.specs == nil {
|
||||||
r.specs = make(map[string]ModuleSpec)
|
r.specs = make(map[string]ModuleSpec)
|
||||||
}
|
}
|
||||||
r.constructors[normalizedSpec.Key] = constructor
|
r.builders[normalizedSpec.Key] = builder
|
||||||
|
r.optionValidators[normalizedSpec.Key] = validateOptions
|
||||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ChunkerRegistry) Build(key string) (contracts.Chunker, error) {
|
func (r *ChunkerRegistry) Build(key string) (contracts.Chunker, error) {
|
||||||
|
return r.BuildWithRequest(key, BuildRequest{})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ChunkerRegistry) BuildWithRequest(key string, request BuildRequest) (contracts.Chunker, error) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return nil, fmt.Errorf("chunker registry must not be nil")
|
return nil, fmt.Errorf("chunker registry must not be nil")
|
||||||
}
|
}
|
||||||
@@ -62,12 +85,12 @@ func (r *ChunkerRegistry) Build(key string) (contracts.Chunker, error) {
|
|||||||
return nil, fmt.Errorf("chunker key must not be empty")
|
return nil, fmt.Errorf("chunker key must not be empty")
|
||||||
}
|
}
|
||||||
|
|
||||||
constructor, ok := r.constructors[normalizedKey]
|
builder, ok := r.builders[normalizedKey]
|
||||||
if !ok {
|
if !ok {
|
||||||
return nil, fmt.Errorf("chunker %q is not registered", normalizedKey)
|
return nil, fmt.Errorf("chunker %q is not registered", normalizedKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
chunker, err := constructor()
|
chunker, err := builder(cloneBuildRequest(request))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("build chunker %q: %w", normalizedKey, err)
|
return nil, fmt.Errorf("build chunker %q: %w", normalizedKey, err)
|
||||||
}
|
}
|
||||||
@@ -81,6 +104,18 @@ func (r *ChunkerRegistry) Build(key string) (contracts.Chunker, error) {
|
|||||||
return chunker, nil
|
return chunker, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (r *ChunkerRegistry) ValidateOptions(key string, options map[string]any) error {
|
||||||
|
if r == nil {
|
||||||
|
return fmt.Errorf("chunker registry must not be nil")
|
||||||
|
}
|
||||||
|
normalizedKey := strings.TrimSpace(key)
|
||||||
|
validator, ok := r.optionValidators[normalizedKey]
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("chunker %q is not registered", normalizedKey)
|
||||||
|
}
|
||||||
|
return validateRegisteredOptions(validator, options)
|
||||||
|
}
|
||||||
|
|
||||||
func (r *ChunkerRegistry) Spec(key string) (ModuleSpec, bool) {
|
func (r *ChunkerRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return ModuleSpec{}, false
|
return ModuleSpec{}, false
|
||||||
@@ -98,5 +133,5 @@ func (r *ChunkerRegistry) RegisteredKeys() []string {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return sortedRegistryKeys(r.constructors)
|
return sortedRegistryKeys(r.builders)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -338,34 +338,6 @@ func (chunker registryChunker) Chunk(ctx context.Context, req contracts.ChunkReq
|
|||||||
return contracts.ChunkResult{}, nil
|
return contracts.ChunkResult{}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type registryMerger struct {
|
|
||||||
key string
|
|
||||||
}
|
|
||||||
|
|
||||||
func (merger registryMerger) Key() string {
|
|
||||||
return merger.key
|
|
||||||
}
|
|
||||||
|
|
||||||
func (merger registryMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
|
|
||||||
return contracts.MergeResult{}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type registryNormalizer struct {
|
|
||||||
key string
|
|
||||||
}
|
|
||||||
|
|
||||||
func (normalizer registryNormalizer) Key() string {
|
|
||||||
return normalizer.key
|
|
||||||
}
|
|
||||||
|
|
||||||
func (normalizer registryNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (normalizer registryNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
|
|
||||||
return contracts.NormalizeResult{}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type registryOutputEncoder struct {
|
type registryOutputEncoder struct {
|
||||||
key string
|
key string
|
||||||
}
|
}
|
||||||
@@ -377,19 +349,3 @@ func (encoder registryOutputEncoder) Key() string {
|
|||||||
func (encoder registryOutputEncoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
func (encoder registryOutputEncoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||||
return contracts.OutputResult{}, nil
|
return contracts.OutputResult{}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type registryValidator struct {
|
|
||||||
name string
|
|
||||||
}
|
|
||||||
|
|
||||||
func (validator registryValidator) Name() string {
|
|
||||||
return validator.name
|
|
||||||
}
|
|
||||||
|
|
||||||
func (validator registryValidator) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return contracts.ExecutionClassDeterministic
|
|
||||||
}
|
|
||||||
|
|
||||||
func (validator registryValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
|
|
||||||
return contracts.ValidationResult{Approved: true}, nil
|
|
||||||
}
|
|
||||||
|
|||||||
69
internal/framework/pipeline/construction.go
Normal file
69
internal/framework/pipeline/construction.go
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ModuleDependencies contains run-scoped collaborators shared by constructed
|
||||||
|
// modules. Implementations retain only the dependencies they use.
|
||||||
|
type ModuleDependencies struct {
|
||||||
|
LLM contracts.StructuredLLMClient
|
||||||
|
}
|
||||||
|
|
||||||
|
// BuildRequest contains the stable dependencies and configured options used to
|
||||||
|
// construct one module or validator for a run.
|
||||||
|
type BuildRequest struct {
|
||||||
|
Dependencies ModuleDependencies
|
||||||
|
Options map[string]any
|
||||||
|
}
|
||||||
|
|
||||||
|
// OptionValidator validates one module binding without constructing it.
|
||||||
|
type OptionValidator func(map[string]any) error
|
||||||
|
|
||||||
|
func rejectUnconfiguredOptions(options map[string]any) error {
|
||||||
|
return RejectUnknownOptions(options)
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateRegisteredOptions(validator OptionValidator, options map[string]any) error {
|
||||||
|
if validator == nil {
|
||||||
|
return fmt.Errorf("option validator must not be nil")
|
||||||
|
}
|
||||||
|
return validator(cloneOptions(options))
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneBuildRequest(request BuildRequest) BuildRequest {
|
||||||
|
return BuildRequest{
|
||||||
|
Dependencies: request.Dependencies,
|
||||||
|
Options: cloneOptions(request.Options),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// RejectUnknownOptions provides the common strict-map check used by module-
|
||||||
|
// owned option decoders. Values remain the implementation's responsibility.
|
||||||
|
func RejectUnknownOptions(options map[string]any, allowed ...string) error {
|
||||||
|
known := make(map[string]struct{}, len(allowed))
|
||||||
|
for _, key := range allowed {
|
||||||
|
key = strings.TrimSpace(key)
|
||||||
|
if key != "" {
|
||||||
|
known[key] = struct{}{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
unknown := make([]string, 0)
|
||||||
|
for key := range options {
|
||||||
|
if _, ok := known[key]; !ok {
|
||||||
|
unknown = append(unknown, key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(unknown) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
sort.Strings(unknown)
|
||||||
|
if len(unknown) == 1 {
|
||||||
|
return fmt.Errorf("unknown option %q", unknown[0])
|
||||||
|
}
|
||||||
|
return fmt.Errorf("unknown options %q", unknown)
|
||||||
|
}
|
||||||
@@ -82,10 +82,6 @@ type debugBinaryEnvelope struct {
|
|||||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type debugRawPayload struct {
|
|
||||||
Content debugBinaryEnvelope `json:"content"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type debugSourceInput struct {
|
type debugSourceInput struct {
|
||||||
SourceID string `json:"source_id,omitempty"`
|
SourceID string `json:"source_id,omitempty"`
|
||||||
Path string `json:"path,omitempty"`
|
Path string `json:"path,omitempty"`
|
||||||
@@ -104,40 +100,23 @@ type debugSourceDocument struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type debugSourceChunk struct {
|
type debugSourceChunk struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
SourceID string `json:"source_id"`
|
SourceID string `json:"source_id"`
|
||||||
Index int `json:"index"`
|
Index int `json:"index"`
|
||||||
StartUnitID int `json:"start_unit_id"`
|
Ref source.SourceRef `json:"ref"`
|
||||||
EndUnitID int `json:"end_unit_id"`
|
Content debugBinaryEnvelope `json:"content"`
|
||||||
Content debugBinaryEnvelope `json:"content"`
|
Units []source.SourceUnit `json:"units,omitempty"`
|
||||||
Units []source.SourceUnit `json:"units,omitempty"`
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type debugExtractOutput struct {
|
type debugSerializedOutput struct {
|
||||||
LaneID string `json:"lane_id"`
|
|
||||||
ExtractorKey string `json:"extractor_key"`
|
|
||||||
SourceID string `json:"source_id"`
|
|
||||||
ChunkID string `json:"chunk_id"`
|
|
||||||
ChunkIndex int `json:"chunk_index"`
|
|
||||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
|
||||||
Payload debugBinaryEnvelope `json:"payload"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type debugMergeOutput struct {
|
|
||||||
LaneID string `json:"lane_id"`
|
|
||||||
MergerKey string `json:"merger_key"`
|
|
||||||
SourceID string `json:"source_id,omitempty"`
|
|
||||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
|
||||||
Payload debugBinaryEnvelope `json:"payload"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type debugNormalizeOutput struct {
|
|
||||||
LaneID string `json:"lane_id"`
|
LaneID string `json:"lane_id"`
|
||||||
NormalizerKey string `json:"normalizer_key"`
|
NormalizerKey string `json:"normalizer_key"`
|
||||||
SourceID string `json:"source_id,omitempty"`
|
SourceID string `json:"source_id,omitempty"`
|
||||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
Kind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||||
Payload debugBinaryEnvelope `json:"payload"`
|
Schema contracts.ArtifactSchema `json:"schema"`
|
||||||
|
SchemaDigest string `json:"schema_digest"`
|
||||||
|
Content debugBinaryEnvelope `json:"content"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type debugLLMInputMaterial struct {
|
type debugLLMInputMaterial struct {
|
||||||
@@ -193,28 +172,9 @@ type debugLLMCallReference struct {
|
|||||||
Error bool `json:"error,omitempty"`
|
Error bool `json:"error,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type debugValidationRequest struct {
|
|
||||||
Stage string `json:"stage"`
|
|
||||||
LaneID string `json:"lane_id,omitempty"`
|
|
||||||
ModuleKey string `json:"module_key"`
|
|
||||||
SourceID string `json:"source_id,omitempty"`
|
|
||||||
SessionID string `json:"session_id,omitempty"`
|
|
||||||
LLMProfile string `json:"llm_profile,omitempty"`
|
|
||||||
Options map[string]any `json:"options,omitempty"`
|
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
|
||||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
|
||||||
Payload *debugBinaryEnvelope `json:"payload,omitempty"`
|
|
||||||
ChunkID string `json:"chunk_id,omitempty"`
|
|
||||||
ChunkIndex int `json:"chunk_index,omitempty"`
|
|
||||||
Chunk *debugSourceChunk `json:"chunk,omitempty"`
|
|
||||||
Chunks []debugSourceChunk `json:"chunks,omitempty"`
|
|
||||||
ExtractOutputs []debugExtractOutput `json:"extract_outputs,omitempty"`
|
|
||||||
MergeOutput *debugMergeOutput `json:"merge_output,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type debugValidationCall struct {
|
type debugValidationCall struct {
|
||||||
ValidatorName string `json:"validator_name"`
|
ValidatorName string `json:"validator_name"`
|
||||||
Request debugValidationRequest `json:"request"`
|
Request any `json:"request"`
|
||||||
Result contracts.ValidationResult `json:"result,omitempty"`
|
Result contracts.ValidationResult `json:"result,omitempty"`
|
||||||
Error string `json:"error,omitempty"`
|
Error string `json:"error,omitempty"`
|
||||||
}
|
}
|
||||||
@@ -239,7 +199,16 @@ func wrapDebugLLMClient(client contracts.StructuredLLMClient, recorder DebugReco
|
|||||||
if client == nil || recorder == nil || !recorder.Enabled() {
|
if client == nil || recorder == nil || !recorder.Enabled() {
|
||||||
return client
|
return client
|
||||||
}
|
}
|
||||||
return &debugLLMClient{inner: client, recorder: recorder}
|
if _, ok := client.(*debugLLMClient); ok {
|
||||||
|
return client
|
||||||
|
}
|
||||||
|
return &debugLLMClient{inner: client, recorder: synchronizedDebugRecorder(recorder)}
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithDebugLLMRecording decorates a shared LLM client so calls made by
|
||||||
|
// construction-injected modules participate in the run's debug recording.
|
||||||
|
func WithDebugLLMRecording(client contracts.StructuredLLMClient, recorder DebugRecorder) contracts.StructuredLLMClient {
|
||||||
|
return wrapDebugLLMClient(client, recorder)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (client *debugLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
func (client *debugLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
@@ -430,10 +399,6 @@ func debugContentEnvelope(content []byte, mediaType string, metadata map[string]
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func debugPayloadEnvelope(payload contracts.RawPayload) debugBinaryEnvelope {
|
|
||||||
return debugContentEnvelope(payload.Content, payload.MediaType, payload.Metadata, payload.Warnings)
|
|
||||||
}
|
|
||||||
|
|
||||||
func debugSourceDocumentEnvelope(doc *source.SourceDocument) *debugSourceDocument {
|
func debugSourceDocumentEnvelope(doc *source.SourceDocument) *debugSourceDocument {
|
||||||
if doc == nil {
|
if doc == nil {
|
||||||
return nil
|
return nil
|
||||||
@@ -448,20 +413,19 @@ func debugSourceDocumentEnvelope(doc *source.SourceDocument) *debugSourceDocumen
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func debugSourceChunkEnvelope(chunk contracts.SourceChunk) debugSourceChunk {
|
func debugSourceChunkEnvelope(chunk source.Chunk) debugSourceChunk {
|
||||||
return debugSourceChunk{
|
return debugSourceChunk{
|
||||||
ID: chunk.ID,
|
ID: chunk.ID,
|
||||||
SourceID: chunk.SourceID,
|
SourceID: chunk.SourceID,
|
||||||
Index: chunk.Index,
|
Index: chunk.Index,
|
||||||
StartUnitID: chunk.StartUnitID,
|
Ref: chunk.Ref,
|
||||||
EndUnitID: chunk.EndUnitID,
|
Content: debugContentEnvelope(chunk.Content, chunk.MediaType, chunk.Metadata, nil),
|
||||||
Content: debugContentEnvelope(chunk.Content, chunk.MediaType, chunk.Metadata, nil),
|
Units: cloneSourceUnits(chunk.Units),
|
||||||
Units: cloneSourceUnits(chunk.Units),
|
Metadata: redactSensitiveMap(chunk.Metadata),
|
||||||
Metadata: redactSensitiveMap(chunk.Metadata),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func debugSourceChunkEnvelopes(chunks []contracts.SourceChunk) []debugSourceChunk {
|
func debugSourceChunkEnvelopes(chunks []source.Chunk) []debugSourceChunk {
|
||||||
if len(chunks) == 0 {
|
if len(chunks) == 0 {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -472,59 +436,26 @@ func debugSourceChunkEnvelopes(chunks []contracts.SourceChunk) []debugSourceChun
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func debugExtractOutputEnvelope(output contracts.ExtractOutput) debugExtractOutput {
|
func debugSerializedOutputEnvelope(output contracts.SerializedOutput) debugSerializedOutput {
|
||||||
output.Schema.JSONSchema = nil
|
schema := contracts.CloneArtifactSchema(output.Artifact.Schema)
|
||||||
return debugExtractOutput{
|
digest := contracts.DigestArtifactSchema(schema)
|
||||||
LaneID: output.LaneID,
|
schema.JSONSchema = nil
|
||||||
ExtractorKey: output.ExtractorKey,
|
content := debugContentEnvelope(output.Artifact.Content, output.Artifact.MediaType, output.Artifact.Metadata, nil)
|
||||||
SourceID: output.SourceID,
|
content.ContentDigest = debugContentDigest(output.Artifact.Content)
|
||||||
ChunkID: output.ChunkID,
|
return debugSerializedOutput{
|
||||||
ChunkIndex: output.ChunkIndex,
|
LaneID: output.LaneID, NormalizerKey: output.NormalizerKey, SourceID: output.SourceID,
|
||||||
Schema: output.Schema,
|
Kind: output.Artifact.Kind, Schema: schema, SchemaDigest: digest,
|
||||||
Payload: debugPayloadEnvelope(output.Payload),
|
Content: content,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func debugExtractOutputEnvelopes(outputs []contracts.ExtractOutput) []debugExtractOutput {
|
func debugSerializedOutputEnvelopes(outputs []contracts.SerializedOutput) []debugSerializedOutput {
|
||||||
if len(outputs) == 0 {
|
if len(outputs) == 0 {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
out := make([]debugExtractOutput, 0, len(outputs))
|
out := make([]debugSerializedOutput, 0, len(outputs))
|
||||||
for _, output := range outputs {
|
for _, output := range outputs {
|
||||||
out = append(out, debugExtractOutputEnvelope(output))
|
out = append(out, debugSerializedOutputEnvelope(output))
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
func debugMergeOutputEnvelope(output contracts.MergeOutput) debugMergeOutput {
|
|
||||||
output.Schema.JSONSchema = nil
|
|
||||||
return debugMergeOutput{
|
|
||||||
LaneID: output.LaneID,
|
|
||||||
MergerKey: output.MergerKey,
|
|
||||||
SourceID: output.SourceID,
|
|
||||||
Schema: output.Schema,
|
|
||||||
Payload: debugPayloadEnvelope(output.Payload),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func debugNormalizeOutputEnvelope(output contracts.NormalizeOutput) debugNormalizeOutput {
|
|
||||||
output.Schema.JSONSchema = nil
|
|
||||||
return debugNormalizeOutput{
|
|
||||||
LaneID: output.LaneID,
|
|
||||||
NormalizerKey: output.NormalizerKey,
|
|
||||||
SourceID: output.SourceID,
|
|
||||||
Schema: output.Schema,
|
|
||||||
Payload: debugPayloadEnvelope(output.Payload),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func debugNormalizeOutputEnvelopes(outputs []contracts.NormalizeOutput) []debugNormalizeOutput {
|
|
||||||
if len(outputs) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
out := make([]debugNormalizeOutput, 0, len(outputs))
|
|
||||||
for _, output := range outputs {
|
|
||||||
out = append(out, debugNormalizeOutputEnvelope(output))
|
|
||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
@@ -673,36 +604,6 @@ func debugResponseModel(response contracts.StructuredCompletionResponse) string
|
|||||||
return response.Debug.Response.ModelName
|
return response.Debug.Response.ModelName
|
||||||
}
|
}
|
||||||
|
|
||||||
func debugValidationRequestEnvelope(req contracts.ValidationRequest) debugValidationRequest {
|
|
||||||
req.Schema.JSONSchema = nil
|
|
||||||
out := debugValidationRequest{
|
|
||||||
Stage: req.Stage,
|
|
||||||
LaneID: req.LaneID,
|
|
||||||
ModuleKey: req.ModuleKey,
|
|
||||||
SourceID: req.SourceID,
|
|
||||||
SessionID: req.SessionID,
|
|
||||||
LLMProfile: req.LLMProfile,
|
|
||||||
Options: redactSensitiveMap(req.Options),
|
|
||||||
Metadata: redactSensitiveMap(req.Metadata),
|
|
||||||
Schema: req.Schema,
|
|
||||||
ChunkID: req.ChunkID,
|
|
||||||
ChunkIndex: req.ChunkIndex,
|
|
||||||
}
|
|
||||||
payload := debugPayloadEnvelope(req.Payload)
|
|
||||||
out.Payload = &payload
|
|
||||||
if req.Chunk != nil {
|
|
||||||
chunk := debugSourceChunkEnvelope(*req.Chunk)
|
|
||||||
out.Chunk = &chunk
|
|
||||||
}
|
|
||||||
out.Chunks = debugSourceChunkEnvelopes(req.Chunks)
|
|
||||||
out.ExtractOutputs = debugExtractOutputEnvelopes(req.ExtractOutputs)
|
|
||||||
if len(req.MergeOutput.Payload.Content) > 0 || req.MergeOutput.LaneID != "" {
|
|
||||||
merge := debugMergeOutputEnvelope(req.MergeOutput)
|
|
||||||
out.MergeOutput = &merge
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
func debugValidationResultEnvelope(result contracts.ValidationResult) contracts.ValidationResult {
|
func debugValidationResultEnvelope(result contracts.ValidationResult) contracts.ValidationResult {
|
||||||
result.Message = string(redactSecretBytes([]byte(result.Message)))
|
result.Message = string(redactSecretBytes([]byte(result.Message)))
|
||||||
result.DiagnosticArtifactPath = string(redactSecretBytes([]byte(result.DiagnosticArtifactPath)))
|
result.DiagnosticArtifactPath = string(redactSecretBytes([]byte(result.DiagnosticArtifactPath)))
|
||||||
@@ -712,6 +613,14 @@ func debugValidationResultEnvelope(result contracts.ValidationResult) contracts.
|
|||||||
return result
|
return result
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func debugWarningEnvelopes(warnings []contracts.Warning) []contracts.Warning {
|
||||||
|
out := cloneWarnings(warnings)
|
||||||
|
for i := range out {
|
||||||
|
out[i].Message = string(redactSecretBytes([]byte(out[i].Message)))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
func debugRejectedOutputEnvelope(rejected contracts.RejectedOutput) contracts.RejectedOutput {
|
func debugRejectedOutputEnvelope(rejected contracts.RejectedOutput) contracts.RejectedOutput {
|
||||||
rejected.Message = string(redactSecretBytes([]byte(rejected.Message)))
|
rejected.Message = string(redactSecretBytes([]byte(rejected.Message)))
|
||||||
rejected.DiagnosticArtifactPath = string(redactSecretBytes([]byte(rejected.DiagnosticArtifactPath)))
|
rejected.DiagnosticArtifactPath = string(redactSecretBytes([]byte(rejected.DiagnosticArtifactPath)))
|
||||||
|
|||||||
51
internal/framework/pipeline/debug_test.go
Normal file
51
internal/framework/pipeline/debug_test.go
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestDebugSourceDocumentPreservesUnitReferences(t *testing.T) {
|
||||||
|
doc := validSourceDocument()
|
||||||
|
envelope := debugSourceDocumentEnvelope(doc)
|
||||||
|
encoded, err := json.Marshal(envelope)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Marshal(debug source document) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var decoded debugSourceDocument
|
||||||
|
if err := json.Unmarshal(encoded, &decoded); err != nil {
|
||||||
|
t.Fatalf("Unmarshal(debug source document) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if got, want := decoded.Units[0].Ref, (source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1}); got != want {
|
||||||
|
t.Fatalf("debug source unit ref = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
envelope.Units[0].Ref.SourceID = "mutated"
|
||||||
|
if got := doc.Units[0].Ref.SourceID; got != "source-1" {
|
||||||
|
t.Fatalf("source document ref = %q after debug mutation, want source-1", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebugSourceChunkPreservesReference(t *testing.T) {
|
||||||
|
doc := validSourceDocument()
|
||||||
|
chunk := source.Chunk{
|
||||||
|
ID: "chunk-1", SourceID: doc.ID, Index: 0,
|
||||||
|
Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 1},
|
||||||
|
Content: []byte("chunk content"), MediaType: "text/plain", Units: doc.Units[:1],
|
||||||
|
}
|
||||||
|
envelope := debugSourceChunkEnvelope(chunk)
|
||||||
|
encoded, err := json.Marshal(envelope)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Marshal(debug source chunk) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
var decoded debugSourceChunk
|
||||||
|
if err := json.Unmarshal(encoded, &decoded); err != nil {
|
||||||
|
t.Fatalf("Unmarshal(debug source chunk) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if decoded.Ref != chunk.Ref {
|
||||||
|
t.Fatalf("debug chunk ref = %#v, want %#v", decoded.Ref, chunk.Ref)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -2,16 +2,17 @@ package pipeline_test
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"encoding/json"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/chunk/units"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/merge/appendorder"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
||||||
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
|
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/output/json"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestPipelineConfigResolvesWithProductionDefaultsRegistered(t *testing.T) {
|
func TestPipelineConfigResolvesWithProductionDefaultsRegistered(t *testing.T) {
|
||||||
@@ -34,8 +35,8 @@ func TestPipelineConfigResolvesWithProductionDefaultsRegistered(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
pipeline := resolved.ResolvedPipeline
|
pipeline := resolved.ResolvedPipeline
|
||||||
if pipeline.Chunk.Module != generic.Key {
|
if pipeline.Chunk.Module != units.Key {
|
||||||
t.Fatalf("Chunk.Module = %q, want %q", pipeline.Chunk.Module, generic.Key)
|
t.Fatalf("Chunk.Module = %q, want %q", pipeline.Chunk.Module, units.Key)
|
||||||
}
|
}
|
||||||
if pipeline.Output.Module != jsonoutput.Key {
|
if pipeline.Output.Module != jsonoutput.Key {
|
||||||
t.Fatalf("Output.Module = %q, want %q", pipeline.Output.Module, jsonoutput.Key)
|
t.Fatalf("Output.Module = %q, want %q", pipeline.Output.Module, jsonoutput.Key)
|
||||||
@@ -68,32 +69,43 @@ func defaultModuleCatalog(t *testing.T) pipeline.ModuleCatalog {
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register input: %v", err)
|
t.Fatalf("register input: %v", err)
|
||||||
}
|
}
|
||||||
if err := generic.Register(chunkers); err != nil {
|
if err := units.Register(chunkers); err != nil {
|
||||||
t.Fatalf("register generic chunker: %v", err)
|
t.Fatalf("register generic chunker: %v", err)
|
||||||
}
|
}
|
||||||
if err := extractors.RegisterWithSpec(pipeline.ModuleSpec{
|
if err := pipeline.RegisterExtractor[defaultArtifact](extractors, pipeline.ModuleSpec{
|
||||||
Key: "extract",
|
Key: "extract",
|
||||||
Stage: pipeline.StageExtract,
|
Stage: pipeline.StageExtract,
|
||||||
Requires: []string{"chunks"},
|
ArtifactKind: defaultArtifactKind,
|
||||||
Provides: []string{"records"},
|
Requires: []string{"chunks"},
|
||||||
}, func() (contracts.Extractor, error) {
|
Provides: []string{"records"},
|
||||||
|
}, func() (contracts.Extractor[defaultArtifact], error) {
|
||||||
return defaultExtractor{}, nil
|
return defaultExtractor{}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("register extractor: %v", err)
|
t.Fatalf("register extractor: %v", err)
|
||||||
}
|
}
|
||||||
if err := appendorder.Register(mergers); err != nil {
|
if err := appendorder.RegisterTyped(mergers, defaultArtifactKind, func(values []defaultArtifact) (defaultArtifact, error) {
|
||||||
|
if len(values) == 0 {
|
||||||
|
return defaultArtifact{}, nil
|
||||||
|
}
|
||||||
|
return values[0], nil
|
||||||
|
}); err != nil {
|
||||||
t.Fatalf("register appendorder merger: %v", err)
|
t.Fatalf("register appendorder merger: %v", err)
|
||||||
}
|
}
|
||||||
if err := noop.Register(normalizers); err != nil {
|
if err := noop.RegisterTyped[defaultArtifact](normalizers, defaultArtifactKind); err != nil {
|
||||||
t.Fatalf("register noop normalizer: %v", err)
|
t.Fatalf("register noop normalizer: %v", err)
|
||||||
}
|
}
|
||||||
if err := jsonoutput.Register(outputs); err != nil {
|
if err := jsonoutput.Register(outputs); err != nil {
|
||||||
t.Fatalf("register json output: %v", err)
|
t.Fatalf("register json output: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
codecs := pipeline.NewArtifactCodecRegistry()
|
||||||
|
if err := pipeline.RegisterArtifactCodec(codecs, defaultArtifactCodec{}); err != nil {
|
||||||
|
t.Fatalf("register artifact codec: %v", err)
|
||||||
|
}
|
||||||
return pipeline.ModuleCatalog{
|
return pipeline.ModuleCatalog{
|
||||||
Inputs: inputs,
|
Inputs: inputs,
|
||||||
Chunkers: chunkers,
|
Chunkers: chunkers,
|
||||||
|
ArtifactCodecs: codecs,
|
||||||
Extractors: extractors,
|
Extractors: extractors,
|
||||||
Mergers: mergers,
|
Mergers: mergers,
|
||||||
Normalizers: normalizers,
|
Normalizers: normalizers,
|
||||||
@@ -116,6 +128,25 @@ func (defaultExtractor) Key() string { return "extract" }
|
|||||||
|
|
||||||
func (defaultExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
func (defaultExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||||
|
|
||||||
func (defaultExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
func (defaultExtractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[defaultArtifact], error) {
|
||||||
return contracts.ExtractionResult{}, nil
|
return contracts.TypedExtractionResult[defaultArtifact]{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
const defaultArtifactKind contracts.ArtifactKind = "test/default"
|
||||||
|
|
||||||
|
type defaultArtifact struct {
|
||||||
|
Value string `json:"value"`
|
||||||
|
}
|
||||||
|
type defaultArtifactCodec struct{}
|
||||||
|
|
||||||
|
func (defaultArtifactCodec) Kind() contracts.ArtifactKind { return defaultArtifactKind }
|
||||||
|
func (defaultArtifactCodec) Schema() contracts.ArtifactSchema {
|
||||||
|
return contracts.ArtifactSchema{ID: "urn:notarius:test:default", Name: "default", Version: "1", JSONSchema: []byte(`{"type":"object"}`)}
|
||||||
|
}
|
||||||
|
func (defaultArtifactCodec) MediaType() string { return "application/json" }
|
||||||
|
func (defaultArtifactCodec) Encode(value defaultArtifact) ([]byte, error) { return json.Marshal(value) }
|
||||||
|
func (defaultArtifactCodec) Decode(content []byte) (defaultArtifact, error) {
|
||||||
|
var value defaultArtifact
|
||||||
|
err := json.Unmarshal(content, &value)
|
||||||
|
return value, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,102 +1,108 @@
|
|||||||
package pipeline
|
package pipeline
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
type ExtractorConstructor func() (contracts.Extractor, error)
|
|
||||||
|
|
||||||
type ExtractorRegistry struct {
|
type ExtractorRegistry struct {
|
||||||
constructors map[string]ExtractorConstructor
|
typedEntries map[string]typedExtractorEntry
|
||||||
specs map[string]ModuleSpec
|
specs map[string]ModuleSpec
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type typedExtractorEntry struct {
|
||||||
|
spec ModuleSpec
|
||||||
|
valueType reflect.Type
|
||||||
|
validateOptions OptionValidator
|
||||||
|
builder func(BuildRequest) (any, error)
|
||||||
|
extract typedExtractOperation
|
||||||
|
}
|
||||||
|
|
||||||
func NewExtractorRegistry() *ExtractorRegistry {
|
func NewExtractorRegistry() *ExtractorRegistry {
|
||||||
return &ExtractorRegistry{
|
return &ExtractorRegistry{typedEntries: map[string]typedExtractorEntry{}, specs: map[string]ModuleSpec{}}
|
||||||
constructors: make(map[string]ExtractorConstructor),
|
}
|
||||||
specs: make(map[string]ModuleSpec),
|
|
||||||
|
func RegisterExtractor[T any](registry *ExtractorRegistry, spec ModuleSpec, constructor func() (contracts.Extractor[T], error)) error {
|
||||||
|
if constructor == nil {
|
||||||
|
return fmt.Errorf("extractor constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
}
|
}
|
||||||
|
return RegisterExtractorBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.Extractor[T], error) { return constructor() })
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ExtractorRegistry) Register(key string, constructor ExtractorConstructor) error {
|
func RegisterExtractorBuilder[T any](registry *ExtractorRegistry, spec ModuleSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.Extractor[T], error)) error {
|
||||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageExtract), constructor)
|
if registry == nil {
|
||||||
}
|
|
||||||
|
|
||||||
func (r *ExtractorRegistry) RegisterWithSpec(spec ModuleSpec, constructor ExtractorConstructor) error {
|
|
||||||
if r == nil {
|
|
||||||
return fmt.Errorf("extractor registry must not be nil")
|
return fmt.Errorf("extractor registry must not be nil")
|
||||||
}
|
}
|
||||||
|
normalized := normalizeModuleSpec(spec)
|
||||||
normalizedSpec := normalizeModuleSpec(spec)
|
if err := validateModuleSpec("extractor", StageExtract, normalized); err != nil {
|
||||||
if err := validateModuleSpec("extractor", StageExtract, normalizedSpec); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if constructor == nil {
|
if normalized.ArtifactKind == "" {
|
||||||
return fmt.Errorf("extractor constructor for %q must not be nil", normalizedSpec.Key)
|
return fmt.Errorf("typed extractor %q artifact kind must not be empty", normalized.Key)
|
||||||
}
|
}
|
||||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
if validateOptions == nil {
|
||||||
return fmt.Errorf("extractor %q is already registered", normalizedSpec.Key)
|
return fmt.Errorf("extractor option validator for %q must not be nil", normalized.Key)
|
||||||
}
|
}
|
||||||
|
if builder == nil {
|
||||||
if r.constructors == nil {
|
return fmt.Errorf("extractor builder for %q must not be nil", normalized.Key)
|
||||||
r.constructors = make(map[string]ExtractorConstructor)
|
|
||||||
}
|
}
|
||||||
if r.specs == nil {
|
if _, ok := registry.specs[normalized.Key]; ok {
|
||||||
r.specs = make(map[string]ModuleSpec)
|
return fmt.Errorf("extractor %q is already registered", normalized.Key)
|
||||||
}
|
}
|
||||||
r.constructors[normalizedSpec.Key] = constructor
|
entry := typedExtractorEntry{spec: cloneModuleSpec(normalized), valueType: reflect.TypeFor[T](), validateOptions: validateOptions, builder: func(request BuildRequest) (any, error) { return builder(cloneBuildRequest(request)) }, extract: func(ctx context.Context, implementation any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
extractor, ok := implementation.(contracts.Extractor[T])
|
||||||
|
if !ok {
|
||||||
|
return erasedTypedResult{}, fmt.Errorf("extractor %q has incompatible implementation %T", normalized.Key, implementation)
|
||||||
|
}
|
||||||
|
result, err := extractor.Extract(ctx, request)
|
||||||
|
if err != nil {
|
||||||
|
return erasedTypedResult{}, err
|
||||||
|
}
|
||||||
|
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
|
||||||
|
}}
|
||||||
|
if registry.typedEntries == nil {
|
||||||
|
registry.typedEntries = map[string]typedExtractorEntry{}
|
||||||
|
}
|
||||||
|
if registry.specs == nil {
|
||||||
|
registry.specs = map[string]ModuleSpec{}
|
||||||
|
}
|
||||||
|
registry.typedEntries[normalized.Key] = entry
|
||||||
|
registry.specs[normalized.Key] = cloneModuleSpec(normalized)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ExtractorRegistry) Build(key string) (contracts.Extractor, error) {
|
func (r *ExtractorRegistry) validateOptions(key string, options map[string]any) error {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return nil, fmt.Errorf("extractor registry must not be nil")
|
return fmt.Errorf("extractor registry must not be nil")
|
||||||
}
|
}
|
||||||
|
normalized := strings.TrimSpace(key)
|
||||||
normalizedKey := strings.TrimSpace(key)
|
entry, ok := r.typedEntries[normalized]
|
||||||
if normalizedKey == "" {
|
|
||||||
return nil, fmt.Errorf("extractor key must not be empty")
|
|
||||||
}
|
|
||||||
|
|
||||||
constructor, ok := r.constructors[normalizedKey]
|
|
||||||
if !ok {
|
if !ok {
|
||||||
return nil, fmt.Errorf("extractor %q is not registered", normalizedKey)
|
return fmt.Errorf("extractor %q is not registered", normalized)
|
||||||
}
|
}
|
||||||
|
return validateRegisteredOptions(entry.validateOptions, options)
|
||||||
extractor, err := constructor()
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("build extractor %q: %w", normalizedKey, err)
|
|
||||||
}
|
|
||||||
if extractor == nil {
|
|
||||||
return nil, fmt.Errorf("extractor %q constructor returned nil", normalizedKey)
|
|
||||||
}
|
|
||||||
if extractor.Key() != normalizedKey {
|
|
||||||
return nil, fmt.Errorf("extractor %q returned key %q", normalizedKey, extractor.Key())
|
|
||||||
}
|
|
||||||
|
|
||||||
return extractor, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ExtractorRegistry) Spec(key string) (ModuleSpec, bool) {
|
func (r *ExtractorRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return ModuleSpec{}, false
|
return ModuleSpec{}, false
|
||||||
}
|
}
|
||||||
|
|
||||||
spec, ok := r.specs[strings.TrimSpace(key)]
|
spec, ok := r.specs[strings.TrimSpace(key)]
|
||||||
if !ok {
|
return cloneModuleSpec(spec), ok
|
||||||
return ModuleSpec{}, false
|
}
|
||||||
}
|
func (r *ExtractorRegistry) typedEntry(key string) (typedExtractorEntry, bool) {
|
||||||
return cloneModuleSpec(spec), true
|
if r == nil {
|
||||||
|
return typedExtractorEntry{}, false
|
||||||
|
}
|
||||||
|
entry, ok := r.typedEntries[strings.TrimSpace(key)]
|
||||||
|
return entry, ok
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ExtractorRegistry) RegisteredKeys() []string {
|
func (r *ExtractorRegistry) RegisteredKeys() []string {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
return sortedRegistryKeys(r.specs)
|
||||||
return sortedRegistryKeys(r.constructors)
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,376 +0,0 @@
|
|||||||
package pipeline
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"reflect"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestExtractorRegistryRegisterAndBuild(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
|
|
||||||
if err := registry.Register("generic-extractor", fakeExtractorConstructor("generic-extractor")); err != nil {
|
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
extractor, err := registry.Build("generic-extractor")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Build() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if extractor.Key() != "generic-extractor" {
|
|
||||||
t.Fatalf("extractor.Key() = %q, want generic-extractor", extractor.Key())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryRegisterAndBuildTrimKeys(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
|
|
||||||
if err := registry.Register(" generic-extractor ", fakeExtractorConstructor("generic-extractor")); err != nil {
|
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
extractor, err := registry.Build("\tgeneric-extractor\n")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Build() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if extractor.Key() != "generic-extractor" {
|
|
||||||
t.Fatalf("extractor.Key() = %q, want generic-extractor", extractor.Key())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
spec := ModuleSpec{
|
|
||||||
Key: " generic-extractor ",
|
|
||||||
Stage: StageExtract,
|
|
||||||
Provides: []string{" generic-artifact ", "source-citations", "generic-artifact", ""},
|
|
||||||
Requires: []string{" source-document ", "source-document", ""},
|
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
|
||||||
{
|
|
||||||
Name: " glossary ",
|
|
||||||
Description: " Supporting terms ",
|
|
||||||
AcceptedMediaTypes: []string{" text/plain ", "text/markdown", "text/plain", ""},
|
|
||||||
MaxBytes: 1024,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: " roster ",
|
|
||||||
Description: " Characters ",
|
|
||||||
Required: true,
|
|
||||||
Multiple: true,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := registry.RegisterWithSpec(spec, fakeExtractorConstructor("generic-extractor")); err != nil {
|
|
||||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
got, ok := registry.Spec("\tgeneric-extractor\n")
|
|
||||||
if !ok {
|
|
||||||
t.Fatal("Spec() ok = false, want true")
|
|
||||||
}
|
|
||||||
want := ModuleSpec{
|
|
||||||
Key: "generic-extractor",
|
|
||||||
Stage: StageExtract,
|
|
||||||
Provides: []string{"generic-artifact", "source-citations"},
|
|
||||||
Requires: []string{"source-document"},
|
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
|
||||||
{
|
|
||||||
Name: "glossary",
|
|
||||||
Description: "Supporting terms",
|
|
||||||
AcceptedMediaTypes: []string{"text/markdown", "text/plain"},
|
|
||||||
MaxBytes: 1024,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
Name: "roster",
|
|
||||||
Description: "Characters",
|
|
||||||
Required: true,
|
|
||||||
Multiple: true,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
if !reflect.DeepEqual(got, want) {
|
|
||||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
got.Provides[0] = "changed"
|
|
||||||
got.ReferenceSlots[0].Name = "changed"
|
|
||||||
got.ReferenceSlots[0].AcceptedMediaTypes[0] = "changed"
|
|
||||||
again, ok := registry.Spec("generic-extractor")
|
|
||||||
if !ok {
|
|
||||||
t.Fatal("Spec() after caller mutation ok = false, want true")
|
|
||||||
}
|
|
||||||
if !reflect.DeepEqual(again, want) {
|
|
||||||
t.Fatalf("Spec() after caller mutation = %#v, want %#v", again, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
|
|
||||||
if err := registry.Register(" generic-extractor ", fakeExtractorConstructor("generic-extractor")); err != nil {
|
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
got, ok := registry.Spec("generic-extractor")
|
|
||||||
if !ok {
|
|
||||||
t.Fatal("Spec() ok = false, want true")
|
|
||||||
}
|
|
||||||
want := ModuleSpec{Key: "generic-extractor", Stage: StageExtract}
|
|
||||||
if !reflect.DeepEqual(got, want) {
|
|
||||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
|
|
||||||
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-extractor", Stage: StageInput}, fakeExtractorConstructor("generic-extractor"))
|
|
||||||
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("RegisterWithSpec() error = nil, want error")
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), "stage") {
|
|
||||||
t.Fatalf("RegisterWithSpec() error = %q, want stage error", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryRejectsInvalidReferenceSlots(t *testing.T) {
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
slots []contracts.ReferenceSlot
|
|
||||||
want string
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
name: "empty name",
|
|
||||||
slots: []contracts.ReferenceSlot{{Name: " "}},
|
|
||||||
want: "name",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "duplicate name after trim",
|
|
||||||
slots: []contracts.ReferenceSlot{
|
|
||||||
{Name: "roster"},
|
|
||||||
{Name: " roster "},
|
|
||||||
},
|
|
||||||
want: "duplicated",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "negative max bytes",
|
|
||||||
slots: []contracts.ReferenceSlot{{Name: "roster", MaxBytes: -1}},
|
|
||||||
want: "max_bytes",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, test := range tests {
|
|
||||||
t.Run(test.name, func(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
err := registry.RegisterWithSpec(ModuleSpec{
|
|
||||||
Key: "generic-extractor",
|
|
||||||
Stage: StageExtract,
|
|
||||||
ReferenceSlots: test.slots,
|
|
||||||
}, fakeExtractorConstructor("generic-extractor"))
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("RegisterWithSpec() error = nil, want error")
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), test.want) {
|
|
||||||
t.Fatalf("RegisterWithSpec() error = %q, want %q", err.Error(), test.want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistrySpecRejectsUnknownKey(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
|
|
||||||
if _, ok := registry.Spec("missing-extractor"); ok {
|
|
||||||
t.Fatal("Spec() ok = true, want false")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
|
|
||||||
err := registry.Register(" \t", fakeExtractorConstructor("generic-extractor"))
|
|
||||||
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("Register() error = nil, want error")
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), "key must not be empty") {
|
|
||||||
t.Fatalf("Register() error = %q, want empty key error", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
if err := registry.Register("generic-extractor", fakeExtractorConstructor("generic-extractor")); err != nil {
|
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err := registry.Register(" generic-extractor ", fakeExtractorConstructor("generic-extractor"))
|
|
||||||
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("Register() error = nil, want error")
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), "already registered") {
|
|
||||||
t.Fatalf("Register() error = %q, want duplicate key error", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryRegisterRejectsNilConstructor(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
|
|
||||||
err := registry.Register("generic-extractor", nil)
|
|
||||||
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("Register() error = nil, want error")
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), "constructor") {
|
|
||||||
t.Fatalf("Register() error = %q, want constructor error", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryBuildRejectsUnknownKey(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
|
|
||||||
_, err := registry.Build("missing-extractor")
|
|
||||||
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("Build() error = nil, want error")
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), "not registered") {
|
|
||||||
t.Fatalf("Build() error = %q, want unknown key error", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryBuildWrapsConstructorError(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
constructorErr := errors.New("constructor failed")
|
|
||||||
if err := registry.Register("generic-extractor", func() (contracts.Extractor, error) {
|
|
||||||
return nil, constructorErr
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err := registry.Build("generic-extractor")
|
|
||||||
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("Build() error = nil, want error")
|
|
||||||
}
|
|
||||||
if !errors.Is(err, constructorErr) {
|
|
||||||
t.Fatalf("Build() error = %v, want wrapped constructor error", err)
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), "generic-extractor") {
|
|
||||||
t.Fatalf("Build() error = %q, want key context", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryBuildRejectsNilExtractor(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
if err := registry.Register("generic-extractor", func() (contracts.Extractor, error) {
|
|
||||||
return nil, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err := registry.Build("generic-extractor")
|
|
||||||
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("Build() error = nil, want error")
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), "returned nil") {
|
|
||||||
t.Fatalf("Build() error = %q, want nil extractor error", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryBuildRejectsExtractorKeyMismatch(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
if err := registry.Register("generic-extractor", fakeExtractorConstructor("other-extractor")); err != nil {
|
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err := registry.Build("generic-extractor")
|
|
||||||
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("Build() error = nil, want error")
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), "returned key") {
|
|
||||||
t.Fatalf("Build() error = %q, want key mismatch error", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
for _, key := range []string{"zeta", "alpha", "middle"} {
|
|
||||||
if err := registry.Register(key, fakeExtractorConstructor(key)); err != nil {
|
|
||||||
t.Fatalf("Register(%q) error = %v, want nil", key, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
keys := registry.RegisteredKeys()
|
|
||||||
|
|
||||||
want := []string{"alpha", "middle", "zeta"}
|
|
||||||
if !reflect.DeepEqual(keys, want) {
|
|
||||||
t.Fatalf("RegisteredKeys() = %#v, want %#v", keys, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
keys[0] = "changed"
|
|
||||||
if got := registry.RegisteredKeys(); !reflect.DeepEqual(got, want) {
|
|
||||||
t.Fatalf("RegisteredKeys() after caller mutation = %#v, want %#v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryNilRegistryBehavior(t *testing.T) {
|
|
||||||
var registry *ExtractorRegistry
|
|
||||||
|
|
||||||
if err := registry.Register("generic-extractor", fakeExtractorConstructor("generic-extractor")); err == nil {
|
|
||||||
t.Fatal("Register() error = nil, want error")
|
|
||||||
}
|
|
||||||
if _, err := registry.Build("generic-extractor"); err == nil {
|
|
||||||
t.Fatal("Build() error = nil, want error")
|
|
||||||
}
|
|
||||||
if _, ok := registry.Spec("generic-extractor"); ok {
|
|
||||||
t.Fatal("Spec() ok = true, want false")
|
|
||||||
}
|
|
||||||
if keys := registry.RegisteredKeys(); keys != nil {
|
|
||||||
t.Fatalf("RegisteredKeys() = %#v, want nil", keys)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExtractorRegistryBuildRejectsEmptyKey(t *testing.T) {
|
|
||||||
registry := NewExtractorRegistry()
|
|
||||||
|
|
||||||
_, err := registry.Build(" \n")
|
|
||||||
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("Build() error = nil, want error")
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), "key must not be empty") {
|
|
||||||
t.Fatalf("Build() error = %q, want empty key error", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
type registryFakeExtractor struct {
|
|
||||||
key string
|
|
||||||
}
|
|
||||||
|
|
||||||
func fakeExtractorConstructor(key string) ExtractorConstructor {
|
|
||||||
return func() (contracts.Extractor, error) {
|
|
||||||
return registryFakeExtractor{key: key}, nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (extractor registryFakeExtractor) Key() string {
|
|
||||||
return extractor.key
|
|
||||||
}
|
|
||||||
|
|
||||||
func (extractor registryFakeExtractor) ReferenceSlots() []contracts.ReferenceSlot {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (extractor registryFakeExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
|
||||||
return contracts.ExtractionResult{}, nil
|
|
||||||
}
|
|
||||||
@@ -8,16 +8,19 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type InputAdapterConstructor func() (contracts.InputAdapter, error)
|
type InputAdapterConstructor func() (contracts.InputAdapter, error)
|
||||||
|
type InputAdapterBuilder func(BuildRequest) (contracts.InputAdapter, error)
|
||||||
|
|
||||||
type InputAdapterRegistry struct {
|
type InputAdapterRegistry struct {
|
||||||
constructors map[string]InputAdapterConstructor
|
builders map[string]InputAdapterBuilder
|
||||||
specs map[string]ModuleSpec
|
optionValidators map[string]OptionValidator
|
||||||
|
specs map[string]ModuleSpec
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewInputAdapterRegistry() *InputAdapterRegistry {
|
func NewInputAdapterRegistry() *InputAdapterRegistry {
|
||||||
return &InputAdapterRegistry{
|
return &InputAdapterRegistry{
|
||||||
constructors: make(map[string]InputAdapterConstructor),
|
builders: make(map[string]InputAdapterBuilder),
|
||||||
specs: make(map[string]ModuleSpec),
|
optionValidators: make(map[string]OptionValidator),
|
||||||
|
specs: make(map[string]ModuleSpec),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -26,6 +29,15 @@ func (r *InputAdapterRegistry) Register(key string, constructor InputAdapterCons
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
|
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
|
||||||
|
if constructor == nil {
|
||||||
|
return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
}
|
||||||
|
return r.RegisterBuilderWithSpec(spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.InputAdapter, error) {
|
||||||
|
return constructor()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *InputAdapterRegistry) RegisterBuilderWithSpec(spec ModuleSpec, validateOptions OptionValidator, builder InputAdapterBuilder) error {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return fmt.Errorf("input adapter registry must not be nil")
|
return fmt.Errorf("input adapter registry must not be nil")
|
||||||
}
|
}
|
||||||
@@ -34,25 +46,36 @@ func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor Inp
|
|||||||
if err := validateModuleSpec("input adapter", StageInput, normalizedSpec); err != nil {
|
if err := validateModuleSpec("input adapter", StageInput, normalizedSpec); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if constructor == nil {
|
if validateOptions == nil {
|
||||||
return fmt.Errorf("input adapter constructor for %q must not be nil", normalizedSpec.Key)
|
return fmt.Errorf("input adapter option validator for %q must not be nil", normalizedSpec.Key)
|
||||||
}
|
}
|
||||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
if builder == nil {
|
||||||
|
return fmt.Errorf("input adapter builder for %q must not be nil", normalizedSpec.Key)
|
||||||
|
}
|
||||||
|
if _, ok := r.builders[normalizedSpec.Key]; ok {
|
||||||
return fmt.Errorf("input adapter %q is already registered", normalizedSpec.Key)
|
return fmt.Errorf("input adapter %q is already registered", normalizedSpec.Key)
|
||||||
}
|
}
|
||||||
|
|
||||||
if r.constructors == nil {
|
if r.builders == nil {
|
||||||
r.constructors = make(map[string]InputAdapterConstructor)
|
r.builders = make(map[string]InputAdapterBuilder)
|
||||||
|
}
|
||||||
|
if r.optionValidators == nil {
|
||||||
|
r.optionValidators = make(map[string]OptionValidator)
|
||||||
}
|
}
|
||||||
if r.specs == nil {
|
if r.specs == nil {
|
||||||
r.specs = make(map[string]ModuleSpec)
|
r.specs = make(map[string]ModuleSpec)
|
||||||
}
|
}
|
||||||
r.constructors[normalizedSpec.Key] = constructor
|
r.builders[normalizedSpec.Key] = builder
|
||||||
|
r.optionValidators[normalizedSpec.Key] = validateOptions
|
||||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *InputAdapterRegistry) Build(key string) (contracts.InputAdapter, error) {
|
func (r *InputAdapterRegistry) Build(key string) (contracts.InputAdapter, error) {
|
||||||
|
return r.BuildWithRequest(key, BuildRequest{})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *InputAdapterRegistry) BuildWithRequest(key string, request BuildRequest) (contracts.InputAdapter, error) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return nil, fmt.Errorf("input adapter registry must not be nil")
|
return nil, fmt.Errorf("input adapter registry must not be nil")
|
||||||
}
|
}
|
||||||
@@ -62,12 +85,12 @@ func (r *InputAdapterRegistry) Build(key string) (contracts.InputAdapter, error)
|
|||||||
return nil, fmt.Errorf("input adapter key must not be empty")
|
return nil, fmt.Errorf("input adapter key must not be empty")
|
||||||
}
|
}
|
||||||
|
|
||||||
constructor, ok := r.constructors[normalizedKey]
|
builder, ok := r.builders[normalizedKey]
|
||||||
if !ok {
|
if !ok {
|
||||||
return nil, fmt.Errorf("input adapter %q is not registered", normalizedKey)
|
return nil, fmt.Errorf("input adapter %q is not registered", normalizedKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
adapter, err := constructor()
|
adapter, err := builder(cloneBuildRequest(request))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("build input adapter %q: %w", normalizedKey, err)
|
return nil, fmt.Errorf("build input adapter %q: %w", normalizedKey, err)
|
||||||
}
|
}
|
||||||
@@ -81,6 +104,18 @@ func (r *InputAdapterRegistry) Build(key string) (contracts.InputAdapter, error)
|
|||||||
return adapter, nil
|
return adapter, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (r *InputAdapterRegistry) ValidateOptions(key string, options map[string]any) error {
|
||||||
|
if r == nil {
|
||||||
|
return fmt.Errorf("input adapter registry must not be nil")
|
||||||
|
}
|
||||||
|
normalizedKey := strings.TrimSpace(key)
|
||||||
|
validator, ok := r.optionValidators[normalizedKey]
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("input adapter %q is not registered", normalizedKey)
|
||||||
|
}
|
||||||
|
return validateRegisteredOptions(validator, options)
|
||||||
|
}
|
||||||
|
|
||||||
func (r *InputAdapterRegistry) Spec(key string) (ModuleSpec, bool) {
|
func (r *InputAdapterRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return ModuleSpec{}, false
|
return ModuleSpec{}, false
|
||||||
@@ -98,5 +133,5 @@ func (r *InputAdapterRegistry) RegisteredKeys() []string {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return sortedRegistryKeys(r.constructors)
|
return sortedRegistryKeys(r.builders)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -301,7 +301,7 @@ func (adapter fakeAdapter) Parse(ctx context.Context, req contracts.ParseRequest
|
|||||||
Format: "text/plain",
|
Format: "text/plain",
|
||||||
Digest: "sha256:abc123",
|
Digest: "sha256:abc123",
|
||||||
Units: []source.SourceUnit{
|
Units: []source.SourceUnit{
|
||||||
{ID: 1, Kind: "unit", Text: "Source unit."},
|
{ID: 1, Kind: "unit", Text: "Source unit.", Ref: source.SourceRef{SourceID: req.SourceID, StartUnitID: 1, EndUnitID: 1}},
|
||||||
},
|
},
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,102 +1,155 @@
|
|||||||
package pipeline
|
package pipeline
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
type MergerConstructor func() (contracts.Merger, error)
|
type artifactVariantKey struct {
|
||||||
|
module string
|
||||||
|
kind contracts.ArtifactKind
|
||||||
|
}
|
||||||
|
|
||||||
type MergerRegistry struct {
|
type MergerRegistry struct {
|
||||||
constructors map[string]MergerConstructor
|
typedEntries map[artifactVariantKey]typedMergerEntry
|
||||||
specs map[string]ModuleSpec
|
}
|
||||||
|
|
||||||
|
type typedMergerEntry struct {
|
||||||
|
spec ModuleSpec
|
||||||
|
valueType reflect.Type
|
||||||
|
validateOptions OptionValidator
|
||||||
|
builder func(BuildRequest) (any, error)
|
||||||
|
merge typedMergeOperation
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewMergerRegistry() *MergerRegistry {
|
func NewMergerRegistry() *MergerRegistry {
|
||||||
return &MergerRegistry{
|
return &MergerRegistry{
|
||||||
constructors: make(map[string]MergerConstructor),
|
typedEntries: make(map[artifactVariantKey]typedMergerEntry),
|
||||||
specs: make(map[string]ModuleSpec),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *MergerRegistry) Register(key string, constructor MergerConstructor) error {
|
func RegisterMerger[T any](registry *MergerRegistry, spec ModuleSpec, constructor func() (contracts.Merger[T], error)) error {
|
||||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageMerge), constructor)
|
if constructor == nil {
|
||||||
|
return fmt.Errorf("merger constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
}
|
||||||
|
return RegisterMergerBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.Merger[T], error) {
|
||||||
|
return constructor()
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *MergerRegistry) RegisterWithSpec(spec ModuleSpec, constructor MergerConstructor) error {
|
func RegisterMergerBuilder[T any](registry *MergerRegistry, spec ModuleSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.Merger[T], error)) error {
|
||||||
if r == nil {
|
if registry == nil {
|
||||||
return fmt.Errorf("merger registry must not be nil")
|
return fmt.Errorf("merger registry must not be nil")
|
||||||
}
|
}
|
||||||
|
|
||||||
normalizedSpec := normalizeModuleSpec(spec)
|
normalizedSpec := normalizeModuleSpec(spec)
|
||||||
if err := validateModuleSpec("merger", StageMerge, normalizedSpec); err != nil {
|
if err := validateModuleSpec("merger", StageMerge, normalizedSpec); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if constructor == nil {
|
if normalizedSpec.ArtifactKind == "" {
|
||||||
return fmt.Errorf("merger constructor for %q must not be nil", normalizedSpec.Key)
|
return fmt.Errorf("typed merger %q artifact kind must not be empty", normalizedSpec.Key)
|
||||||
}
|
}
|
||||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
if validateOptions == nil {
|
||||||
return fmt.Errorf("merger %q is already registered", normalizedSpec.Key)
|
return fmt.Errorf("merger option validator for %q must not be nil", normalizedSpec.Key)
|
||||||
}
|
}
|
||||||
|
if builder == nil {
|
||||||
if r.constructors == nil {
|
return fmt.Errorf("merger builder for %q must not be nil", normalizedSpec.Key)
|
||||||
r.constructors = make(map[string]MergerConstructor)
|
|
||||||
}
|
}
|
||||||
if r.specs == nil {
|
key := artifactVariantKey{module: normalizedSpec.Key, kind: normalizedSpec.ArtifactKind}
|
||||||
r.specs = make(map[string]ModuleSpec)
|
if _, ok := registry.typedEntries[key]; ok {
|
||||||
|
return fmt.Errorf("merger %q variant for artifact kind %q is already registered", key.module, key.kind)
|
||||||
|
}
|
||||||
|
if registry.typedEntries == nil {
|
||||||
|
registry.typedEntries = make(map[artifactVariantKey]typedMergerEntry)
|
||||||
|
}
|
||||||
|
registry.typedEntries[key] = typedMergerEntry{
|
||||||
|
spec: cloneModuleSpec(normalizedSpec),
|
||||||
|
valueType: reflect.TypeFor[T](),
|
||||||
|
validateOptions: validateOptions,
|
||||||
|
builder: func(request BuildRequest) (any, error) {
|
||||||
|
return builder(cloneBuildRequest(request))
|
||||||
|
},
|
||||||
|
merge: func(ctx context.Context, implementation any, request contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||||
|
merger, ok := implementation.(contracts.Merger[T])
|
||||||
|
if !ok {
|
||||||
|
return erasedTypedResult{}, fmt.Errorf("merger %q has incompatible implementation %T", normalizedSpec.Key, implementation)
|
||||||
|
}
|
||||||
|
outputs := make([]contracts.ExtractArtifact[T], len(request.ExtractOutputs))
|
||||||
|
for i, output := range request.ExtractOutputs {
|
||||||
|
value, err := exactTypedValue[T]("merge extract value", output.Value)
|
||||||
|
if err != nil {
|
||||||
|
return erasedTypedResult{}, err
|
||||||
|
}
|
||||||
|
outputs[i] = contracts.ExtractArtifact[T]{LaneID: output.LaneID, ExtractorKey: output.ExtractorKey, SourceID: output.SourceID, ChunkID: output.ChunkID, ChunkIndex: output.ChunkIndex, ChunkRef: output.ChunkRef, Value: value}
|
||||||
|
}
|
||||||
|
result, err := merger.Merge(ctx, contracts.TypedMergeRequest[T]{Source: request.Source, LaneID: request.LaneID, ExtractOutputs: outputs, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, Metadata: request.Metadata})
|
||||||
|
if err != nil {
|
||||||
|
return erasedTypedResult{}, err
|
||||||
|
}
|
||||||
|
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
|
||||||
|
},
|
||||||
}
|
}
|
||||||
r.constructors[normalizedSpec.Key] = constructor
|
|
||||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *MergerRegistry) Build(key string) (contracts.Merger, error) {
|
func (r *MergerRegistry) validateOptions(key string, kind contracts.ArtifactKind, options map[string]any) error {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return nil, fmt.Errorf("merger registry must not be nil")
|
return fmt.Errorf("merger registry must not be nil")
|
||||||
}
|
}
|
||||||
|
|
||||||
normalizedKey := strings.TrimSpace(key)
|
normalizedKey := strings.TrimSpace(key)
|
||||||
if normalizedKey == "" {
|
entry, ok := r.typedEntry(normalizedKey, kind)
|
||||||
return nil, fmt.Errorf("merger key must not be empty")
|
|
||||||
}
|
|
||||||
|
|
||||||
constructor, ok := r.constructors[normalizedKey]
|
|
||||||
if !ok {
|
if !ok {
|
||||||
return nil, fmt.Errorf("merger %q is not registered", normalizedKey)
|
return fmt.Errorf("merger %q variant for artifact kind %q is not registered", normalizedKey, kind)
|
||||||
}
|
}
|
||||||
|
return validateRegisteredOptions(entry.validateOptions, options)
|
||||||
merger, err := constructor()
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("build merger %q: %w", normalizedKey, err)
|
|
||||||
}
|
|
||||||
if merger == nil {
|
|
||||||
return nil, fmt.Errorf("merger %q constructor returned nil", normalizedKey)
|
|
||||||
}
|
|
||||||
if merger.Key() != normalizedKey {
|
|
||||||
return nil, fmt.Errorf("merger %q returned key %q", normalizedKey, merger.Key())
|
|
||||||
}
|
|
||||||
|
|
||||||
return merger, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *MergerRegistry) Spec(key string) (ModuleSpec, bool) {
|
func (r *MergerRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return ModuleSpec{}, false
|
return ModuleSpec{}, false
|
||||||
}
|
}
|
||||||
|
module := strings.TrimSpace(key)
|
||||||
spec, ok := r.specs[strings.TrimSpace(key)]
|
for variant, entry := range r.typedEntries {
|
||||||
if !ok {
|
if variant.module == module {
|
||||||
return ModuleSpec{}, false
|
return cloneModuleSpec(entry.spec), true
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return cloneModuleSpec(spec), true
|
return ModuleSpec{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *MergerRegistry) typedEntry(key string, kind contracts.ArtifactKind) (typedMergerEntry, bool) {
|
||||||
|
if r == nil {
|
||||||
|
return typedMergerEntry{}, false
|
||||||
|
}
|
||||||
|
entry, ok := r.typedEntries[artifactVariantKey{module: strings.TrimSpace(key), kind: normalizeArtifactKind(kind)}]
|
||||||
|
return entry, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *MergerRegistry) registeredKinds(key string) []contracts.ArtifactKind {
|
||||||
|
if r == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
module := strings.TrimSpace(key)
|
||||||
|
kinds := make([]contracts.ArtifactKind, 0)
|
||||||
|
for variant := range r.typedEntries {
|
||||||
|
if variant.module == module {
|
||||||
|
kinds = append(kinds, variant.kind)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sortArtifactKinds(kinds)
|
||||||
|
return kinds
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *MergerRegistry) RegisteredKeys() []string {
|
func (r *MergerRegistry) RegisteredKeys() []string {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
keys := make(map[string]struct{}, len(r.typedEntries))
|
||||||
return sortedRegistryKeys(r.constructors)
|
for key := range r.typedEntries {
|
||||||
|
keys[key.module] = struct{}{}
|
||||||
|
}
|
||||||
|
return sortedRegistryKeys(keys)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,58 +0,0 @@
|
|||||||
package pipeline
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestMergerRegistryBehavior(t *testing.T) {
|
|
||||||
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Merger]{
|
|
||||||
name: "MergerRegistry",
|
|
||||||
key: "generic-merger",
|
|
||||||
stage: StageMerge,
|
|
||||||
wrongStage: StageExtract,
|
|
||||||
newRegistry: func() any {
|
|
||||||
return NewMergerRegistry()
|
|
||||||
},
|
|
||||||
register: func(registry any, key string, constructor func() (contracts.Merger, error)) error {
|
|
||||||
return registry.(*MergerRegistry).Register(key, constructor)
|
|
||||||
},
|
|
||||||
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Merger, error)) error {
|
|
||||||
return registry.(*MergerRegistry).RegisterWithSpec(spec, constructor)
|
|
||||||
},
|
|
||||||
build: func(registry any, key string) (contracts.Merger, error) {
|
|
||||||
return registry.(*MergerRegistry).Build(key)
|
|
||||||
},
|
|
||||||
spec: func(registry any, key string) (ModuleSpec, bool) {
|
|
||||||
return registry.(*MergerRegistry).Spec(key)
|
|
||||||
},
|
|
||||||
registeredKeys: func(registry any) []string {
|
|
||||||
return registry.(*MergerRegistry).RegisteredKeys()
|
|
||||||
},
|
|
||||||
nilRegister: func(key string, constructor func() (contracts.Merger, error)) error {
|
|
||||||
var registry *MergerRegistry
|
|
||||||
return registry.Register(key, constructor)
|
|
||||||
},
|
|
||||||
nilBuild: func(key string) (contracts.Merger, error) {
|
|
||||||
var registry *MergerRegistry
|
|
||||||
return registry.Build(key)
|
|
||||||
},
|
|
||||||
nilSpec: func(key string) (ModuleSpec, bool) {
|
|
||||||
var registry *MergerRegistry
|
|
||||||
return registry.Spec(key)
|
|
||||||
},
|
|
||||||
nilRegisteredKey: func() []string {
|
|
||||||
var registry *MergerRegistry
|
|
||||||
return registry.RegisteredKeys()
|
|
||||||
},
|
|
||||||
constructor: func(key string) func() (contracts.Merger, error) {
|
|
||||||
return func() (contracts.Merger, error) {
|
|
||||||
return registryMerger{key: key}, nil
|
|
||||||
}
|
|
||||||
},
|
|
||||||
moduleKey: func(module contracts.Merger) string {
|
|
||||||
return module.Key()
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
@@ -23,6 +23,7 @@ const (
|
|||||||
type ModuleSpec struct {
|
type ModuleSpec struct {
|
||||||
Key string
|
Key string
|
||||||
Stage ModuleStage
|
Stage ModuleStage
|
||||||
|
ArtifactKind contracts.ArtifactKind
|
||||||
Provides []string
|
Provides []string
|
||||||
Requires []string
|
Requires []string
|
||||||
ReferenceSlots []contracts.ReferenceSlot
|
ReferenceSlots []contracts.ReferenceSlot
|
||||||
@@ -39,6 +40,7 @@ func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
|
|||||||
return ModuleSpec{
|
return ModuleSpec{
|
||||||
Key: strings.TrimSpace(spec.Key),
|
Key: strings.TrimSpace(spec.Key),
|
||||||
Stage: spec.Stage,
|
Stage: spec.Stage,
|
||||||
|
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
|
||||||
Provides: normalizeCapabilities(spec.Provides),
|
Provides: normalizeCapabilities(spec.Provides),
|
||||||
Requires: normalizeCapabilities(spec.Requires),
|
Requires: normalizeCapabilities(spec.Requires),
|
||||||
ReferenceSlots: normalizeReferenceSlots(spec.ReferenceSlots),
|
ReferenceSlots: normalizeReferenceSlots(spec.ReferenceSlots),
|
||||||
@@ -74,6 +76,7 @@ func cloneModuleSpec(spec ModuleSpec) ModuleSpec {
|
|||||||
return ModuleSpec{
|
return ModuleSpec{
|
||||||
Key: spec.Key,
|
Key: spec.Key,
|
||||||
Stage: spec.Stage,
|
Stage: spec.Stage,
|
||||||
|
ArtifactKind: spec.ArtifactKind,
|
||||||
Provides: append([]string(nil), spec.Provides...),
|
Provides: append([]string(nil), spec.Provides...),
|
||||||
Requires: append([]string(nil), spec.Requires...),
|
Requires: append([]string(nil), spec.Requires...),
|
||||||
ReferenceSlots: contracts.CloneReferenceSlots(spec.ReferenceSlots),
|
ReferenceSlots: contracts.CloneReferenceSlots(spec.ReferenceSlots),
|
||||||
@@ -87,6 +90,9 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
|
|||||||
if spec.Stage != expectedStage {
|
if spec.Stage != expectedStage {
|
||||||
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
|
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
|
||||||
}
|
}
|
||||||
|
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
|
||||||
|
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
|
||||||
|
}
|
||||||
if !referenceSlotStage(spec.Stage) && len(spec.ReferenceSlots) > 0 {
|
if !referenceSlotStage(spec.Stage) && len(spec.ReferenceSlots) > 0 {
|
||||||
return fmt.Errorf("%s %q must not declare reference slots", kind, spec.Key)
|
return fmt.Errorf("%s %q must not declare reference slots", kind, spec.Key)
|
||||||
}
|
}
|
||||||
@@ -113,6 +119,10 @@ func sortedRegistryKeys[C any](constructors map[string]C) []string {
|
|||||||
return keys
|
return keys
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func sortArtifactKinds(kinds []contracts.ArtifactKind) {
|
||||||
|
sort.Slice(kinds, func(i, j int) bool { return kinds[i] < kinds[j] })
|
||||||
|
}
|
||||||
|
|
||||||
func normalizeReferenceSlots(slots []contracts.ReferenceSlot) []contracts.ReferenceSlot {
|
func normalizeReferenceSlots(slots []contracts.ReferenceSlot) []contracts.ReferenceSlot {
|
||||||
if len(slots) == 0 {
|
if len(slots) == 0 {
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -1,102 +1,146 @@
|
|||||||
package pipeline
|
package pipeline
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
type NormalizerConstructor func() (contracts.Normalizer, error)
|
|
||||||
|
|
||||||
type NormalizerRegistry struct {
|
type NormalizerRegistry struct {
|
||||||
constructors map[string]NormalizerConstructor
|
typedEntries map[artifactVariantKey]typedNormalizerEntry
|
||||||
specs map[string]ModuleSpec
|
}
|
||||||
|
|
||||||
|
type typedNormalizerEntry struct {
|
||||||
|
spec ModuleSpec
|
||||||
|
valueType reflect.Type
|
||||||
|
validateOptions OptionValidator
|
||||||
|
builder func(BuildRequest) (any, error)
|
||||||
|
normalize typedNormalizeOperation
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewNormalizerRegistry() *NormalizerRegistry {
|
func NewNormalizerRegistry() *NormalizerRegistry {
|
||||||
return &NormalizerRegistry{
|
return &NormalizerRegistry{
|
||||||
constructors: make(map[string]NormalizerConstructor),
|
typedEntries: make(map[artifactVariantKey]typedNormalizerEntry),
|
||||||
specs: make(map[string]ModuleSpec),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *NormalizerRegistry) Register(key string, constructor NormalizerConstructor) error {
|
func RegisterNormalizer[T any](registry *NormalizerRegistry, spec ModuleSpec, constructor func() (contracts.Normalizer[T], error)) error {
|
||||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageNormalize), constructor)
|
if constructor == nil {
|
||||||
|
return fmt.Errorf("normalizer constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
}
|
||||||
|
return RegisterNormalizerBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.Normalizer[T], error) {
|
||||||
|
return constructor()
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *NormalizerRegistry) RegisterWithSpec(spec ModuleSpec, constructor NormalizerConstructor) error {
|
func RegisterNormalizerBuilder[T any](registry *NormalizerRegistry, spec ModuleSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.Normalizer[T], error)) error {
|
||||||
if r == nil {
|
if registry == nil {
|
||||||
return fmt.Errorf("normalizer registry must not be nil")
|
return fmt.Errorf("normalizer registry must not be nil")
|
||||||
}
|
}
|
||||||
|
|
||||||
normalizedSpec := normalizeModuleSpec(spec)
|
normalizedSpec := normalizeModuleSpec(spec)
|
||||||
if err := validateModuleSpec("normalizer", StageNormalize, normalizedSpec); err != nil {
|
if err := validateModuleSpec("normalizer", StageNormalize, normalizedSpec); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if constructor == nil {
|
if normalizedSpec.ArtifactKind == "" {
|
||||||
return fmt.Errorf("normalizer constructor for %q must not be nil", normalizedSpec.Key)
|
return fmt.Errorf("typed normalizer %q artifact kind must not be empty", normalizedSpec.Key)
|
||||||
}
|
}
|
||||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
if validateOptions == nil {
|
||||||
return fmt.Errorf("normalizer %q is already registered", normalizedSpec.Key)
|
return fmt.Errorf("normalizer option validator for %q must not be nil", normalizedSpec.Key)
|
||||||
}
|
}
|
||||||
|
if builder == nil {
|
||||||
if r.constructors == nil {
|
return fmt.Errorf("normalizer builder for %q must not be nil", normalizedSpec.Key)
|
||||||
r.constructors = make(map[string]NormalizerConstructor)
|
|
||||||
}
|
}
|
||||||
if r.specs == nil {
|
key := artifactVariantKey{module: normalizedSpec.Key, kind: normalizedSpec.ArtifactKind}
|
||||||
r.specs = make(map[string]ModuleSpec)
|
if _, ok := registry.typedEntries[key]; ok {
|
||||||
|
return fmt.Errorf("normalizer %q variant for artifact kind %q is already registered", key.module, key.kind)
|
||||||
|
}
|
||||||
|
if registry.typedEntries == nil {
|
||||||
|
registry.typedEntries = make(map[artifactVariantKey]typedNormalizerEntry)
|
||||||
|
}
|
||||||
|
registry.typedEntries[key] = typedNormalizerEntry{
|
||||||
|
spec: cloneModuleSpec(normalizedSpec),
|
||||||
|
valueType: reflect.TypeFor[T](),
|
||||||
|
validateOptions: validateOptions,
|
||||||
|
builder: func(request BuildRequest) (any, error) {
|
||||||
|
return builder(cloneBuildRequest(request))
|
||||||
|
},
|
||||||
|
normalize: func(ctx context.Context, implementation any, request contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||||
|
normalizer, ok := implementation.(contracts.Normalizer[T])
|
||||||
|
if !ok {
|
||||||
|
return erasedTypedResult{}, fmt.Errorf("normalizer %q has incompatible implementation %T", normalizedSpec.Key, implementation)
|
||||||
|
}
|
||||||
|
value, err := exactTypedValue[T]("normalize merge value", request.MergeOutput.Value)
|
||||||
|
if err != nil {
|
||||||
|
return erasedTypedResult{}, err
|
||||||
|
}
|
||||||
|
result, err := normalizer.Normalize(ctx, contracts.TypedNormalizeRequest[T]{Source: request.Source, LaneID: request.LaneID, MergeOutput: contracts.MergeArtifact[T]{LaneID: request.MergeOutput.LaneID, MergerKey: request.MergeOutput.MergerKey, SourceID: request.MergeOutput.SourceID, Value: value}, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, Metadata: request.Metadata})
|
||||||
|
if err != nil {
|
||||||
|
return erasedTypedResult{}, err
|
||||||
|
}
|
||||||
|
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
|
||||||
|
},
|
||||||
}
|
}
|
||||||
r.constructors[normalizedSpec.Key] = constructor
|
|
||||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *NormalizerRegistry) Build(key string) (contracts.Normalizer, error) {
|
func (r *NormalizerRegistry) validateOptions(key string, kind contracts.ArtifactKind, options map[string]any) error {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return nil, fmt.Errorf("normalizer registry must not be nil")
|
return fmt.Errorf("normalizer registry must not be nil")
|
||||||
}
|
}
|
||||||
|
|
||||||
normalizedKey := strings.TrimSpace(key)
|
normalizedKey := strings.TrimSpace(key)
|
||||||
if normalizedKey == "" {
|
entry, ok := r.typedEntry(normalizedKey, kind)
|
||||||
return nil, fmt.Errorf("normalizer key must not be empty")
|
|
||||||
}
|
|
||||||
|
|
||||||
constructor, ok := r.constructors[normalizedKey]
|
|
||||||
if !ok {
|
if !ok {
|
||||||
return nil, fmt.Errorf("normalizer %q is not registered", normalizedKey)
|
return fmt.Errorf("normalizer %q variant for artifact kind %q is not registered", normalizedKey, kind)
|
||||||
}
|
}
|
||||||
|
return validateRegisteredOptions(entry.validateOptions, options)
|
||||||
normalizer, err := constructor()
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("build normalizer %q: %w", normalizedKey, err)
|
|
||||||
}
|
|
||||||
if normalizer == nil {
|
|
||||||
return nil, fmt.Errorf("normalizer %q constructor returned nil", normalizedKey)
|
|
||||||
}
|
|
||||||
if normalizer.Key() != normalizedKey {
|
|
||||||
return nil, fmt.Errorf("normalizer %q returned key %q", normalizedKey, normalizer.Key())
|
|
||||||
}
|
|
||||||
|
|
||||||
return normalizer, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *NormalizerRegistry) Spec(key string) (ModuleSpec, bool) {
|
func (r *NormalizerRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return ModuleSpec{}, false
|
return ModuleSpec{}, false
|
||||||
}
|
}
|
||||||
|
module := strings.TrimSpace(key)
|
||||||
spec, ok := r.specs[strings.TrimSpace(key)]
|
for variant, entry := range r.typedEntries {
|
||||||
if !ok {
|
if variant.module == module {
|
||||||
return ModuleSpec{}, false
|
return cloneModuleSpec(entry.spec), true
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return cloneModuleSpec(spec), true
|
return ModuleSpec{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *NormalizerRegistry) typedEntry(key string, kind contracts.ArtifactKind) (typedNormalizerEntry, bool) {
|
||||||
|
if r == nil {
|
||||||
|
return typedNormalizerEntry{}, false
|
||||||
|
}
|
||||||
|
entry, ok := r.typedEntries[artifactVariantKey{module: strings.TrimSpace(key), kind: normalizeArtifactKind(kind)}]
|
||||||
|
return entry, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *NormalizerRegistry) registeredKinds(key string) []contracts.ArtifactKind {
|
||||||
|
if r == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
module := strings.TrimSpace(key)
|
||||||
|
kinds := make([]contracts.ArtifactKind, 0)
|
||||||
|
for variant := range r.typedEntries {
|
||||||
|
if variant.module == module {
|
||||||
|
kinds = append(kinds, variant.kind)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sortArtifactKinds(kinds)
|
||||||
|
return kinds
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *NormalizerRegistry) RegisteredKeys() []string {
|
func (r *NormalizerRegistry) RegisteredKeys() []string {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
keys := make(map[string]struct{}, len(r.typedEntries))
|
||||||
return sortedRegistryKeys(r.constructors)
|
for key := range r.typedEntries {
|
||||||
|
keys[key.module] = struct{}{}
|
||||||
|
}
|
||||||
|
return sortedRegistryKeys(keys)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,58 +0,0 @@
|
|||||||
package pipeline
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestNormalizerRegistryBehavior(t *testing.T) {
|
|
||||||
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Normalizer]{
|
|
||||||
name: "NormalizerRegistry",
|
|
||||||
key: "generic-normalizer",
|
|
||||||
stage: StageNormalize,
|
|
||||||
wrongStage: StageExtract,
|
|
||||||
newRegistry: func() any {
|
|
||||||
return NewNormalizerRegistry()
|
|
||||||
},
|
|
||||||
register: func(registry any, key string, constructor func() (contracts.Normalizer, error)) error {
|
|
||||||
return registry.(*NormalizerRegistry).Register(key, constructor)
|
|
||||||
},
|
|
||||||
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Normalizer, error)) error {
|
|
||||||
return registry.(*NormalizerRegistry).RegisterWithSpec(spec, constructor)
|
|
||||||
},
|
|
||||||
build: func(registry any, key string) (contracts.Normalizer, error) {
|
|
||||||
return registry.(*NormalizerRegistry).Build(key)
|
|
||||||
},
|
|
||||||
spec: func(registry any, key string) (ModuleSpec, bool) {
|
|
||||||
return registry.(*NormalizerRegistry).Spec(key)
|
|
||||||
},
|
|
||||||
registeredKeys: func(registry any) []string {
|
|
||||||
return registry.(*NormalizerRegistry).RegisteredKeys()
|
|
||||||
},
|
|
||||||
nilRegister: func(key string, constructor func() (contracts.Normalizer, error)) error {
|
|
||||||
var registry *NormalizerRegistry
|
|
||||||
return registry.Register(key, constructor)
|
|
||||||
},
|
|
||||||
nilBuild: func(key string) (contracts.Normalizer, error) {
|
|
||||||
var registry *NormalizerRegistry
|
|
||||||
return registry.Build(key)
|
|
||||||
},
|
|
||||||
nilSpec: func(key string) (ModuleSpec, bool) {
|
|
||||||
var registry *NormalizerRegistry
|
|
||||||
return registry.Spec(key)
|
|
||||||
},
|
|
||||||
nilRegisteredKey: func() []string {
|
|
||||||
var registry *NormalizerRegistry
|
|
||||||
return registry.RegisteredKeys()
|
|
||||||
},
|
|
||||||
constructor: func(key string) func() (contracts.Normalizer, error) {
|
|
||||||
return func() (contracts.Normalizer, error) {
|
|
||||||
return registryNormalizer{key: key}, nil
|
|
||||||
}
|
|
||||||
},
|
|
||||||
moduleKey: func(module contracts.Normalizer) string {
|
|
||||||
return module.Key()
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
65
internal/framework/pipeline/options.go
Normal file
65
internal/framework/pipeline/options.go
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import "fmt"
|
||||||
|
|
||||||
|
func validateResolvedOptions(resolved ResolvedPipeline, catalog ModuleCatalog) error {
|
||||||
|
if err := catalog.Inputs.ValidateOptions(resolved.Input.Module, resolved.Input.Options); err != nil {
|
||||||
|
return moduleOptionsError(resolved.ID, "", StageInput, resolved.Input.Module, err)
|
||||||
|
}
|
||||||
|
if err := catalog.Chunkers.ValidateOptions(resolved.Chunk.Module, resolved.Chunk.Options); err != nil {
|
||||||
|
return moduleOptionsError(resolved.ID, "", StageChunk, resolved.Chunk.Module, err)
|
||||||
|
}
|
||||||
|
if err := validateChainOptions(resolved, catalog, StageChunk, "", resolved.Chunk.Module); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, lane := range resolved.ArtifactLanes {
|
||||||
|
if err := catalog.Extractors.validateOptions(lane.Extract.Module, lane.Extract.Options); err != nil {
|
||||||
|
return moduleOptionsError(resolved.ID, lane.ID, StageExtract, lane.Extract.Module, err)
|
||||||
|
}
|
||||||
|
if err := validateChainOptions(resolved, catalog, StageExtract, lane.ID, lane.Extract.Module); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := catalog.Mergers.validateOptions(lane.Merge.Module, lane.ArtifactKind, lane.Merge.Options); err != nil {
|
||||||
|
return moduleOptionsError(resolved.ID, lane.ID, StageMerge, lane.Merge.Module, err)
|
||||||
|
}
|
||||||
|
if err := validateChainOptions(resolved, catalog, StageMerge, lane.ID, lane.Merge.Module); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := catalog.Normalizers.validateOptions(lane.Normalize.Module, lane.ArtifactKind, lane.Normalize.Options); err != nil {
|
||||||
|
return moduleOptionsError(resolved.ID, lane.ID, StageNormalize, lane.Normalize.Module, err)
|
||||||
|
}
|
||||||
|
if err := validateChainOptions(resolved, catalog, StageNormalize, lane.ID, lane.Normalize.Module); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := catalog.Outputs.ValidateOptions(resolved.Output.Module, resolved.Output.Options); err != nil {
|
||||||
|
return moduleOptionsError(resolved.ID, "", StageOutput, resolved.Output.Module, err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateChainOptions(resolved ResolvedPipeline, catalog ModuleCatalog, stage ModuleStage, laneID, moduleKey string) error {
|
||||||
|
chain := resolvedValidatorChain(stage, laneID, moduleKey, resolved.ValidatorChains)
|
||||||
|
for _, validator := range chain.Validators {
|
||||||
|
if err := catalog.Validators.validateOptions(validator); err != nil {
|
||||||
|
return validatorOptionsError(resolved.ID, laneID, stage, moduleKey, validator.Binding.Module, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func moduleOptionsError(pipelineID, laneID string, stage ModuleStage, moduleKey string, cause error) error {
|
||||||
|
if laneID == "" {
|
||||||
|
return fmt.Errorf("pipeline %q %s module %q options: %w", pipelineID, stage, moduleKey, cause)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("pipeline %q lane %q %s module %q options: %w", pipelineID, laneID, stage, moduleKey, cause)
|
||||||
|
}
|
||||||
|
|
||||||
|
func validatorOptionsError(pipelineID, laneID string, stage ModuleStage, moduleKey, validatorKey string, cause error) error {
|
||||||
|
if laneID == "" {
|
||||||
|
return fmt.Errorf("pipeline %q %s module %q validator %q options: %w", pipelineID, stage, moduleKey, validatorKey, cause)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("pipeline %q lane %q %s module %q validator %q options: %w", pipelineID, laneID, stage, moduleKey, validatorKey, cause)
|
||||||
|
}
|
||||||
@@ -8,16 +8,19 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type OutputEncoderConstructor func() (contracts.OutputEncoder, error)
|
type OutputEncoderConstructor func() (contracts.OutputEncoder, error)
|
||||||
|
type OutputEncoderBuilder func(BuildRequest) (contracts.OutputEncoder, error)
|
||||||
|
|
||||||
type OutputEncoderRegistry struct {
|
type OutputEncoderRegistry struct {
|
||||||
constructors map[string]OutputEncoderConstructor
|
builders map[string]OutputEncoderBuilder
|
||||||
specs map[string]ModuleSpec
|
optionValidators map[string]OptionValidator
|
||||||
|
specs map[string]ModuleSpec
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewOutputEncoderRegistry() *OutputEncoderRegistry {
|
func NewOutputEncoderRegistry() *OutputEncoderRegistry {
|
||||||
return &OutputEncoderRegistry{
|
return &OutputEncoderRegistry{
|
||||||
constructors: make(map[string]OutputEncoderConstructor),
|
builders: make(map[string]OutputEncoderBuilder),
|
||||||
specs: make(map[string]ModuleSpec),
|
optionValidators: make(map[string]OptionValidator),
|
||||||
|
specs: make(map[string]ModuleSpec),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -26,6 +29,15 @@ func (r *OutputEncoderRegistry) Register(key string, constructor OutputEncoderCo
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
|
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
|
||||||
|
if constructor == nil {
|
||||||
|
return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
}
|
||||||
|
return r.RegisterBuilderWithSpec(spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.OutputEncoder, error) {
|
||||||
|
return constructor()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *OutputEncoderRegistry) RegisterBuilderWithSpec(spec ModuleSpec, validateOptions OptionValidator, builder OutputEncoderBuilder) error {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return fmt.Errorf("output encoder registry must not be nil")
|
return fmt.Errorf("output encoder registry must not be nil")
|
||||||
}
|
}
|
||||||
@@ -34,25 +46,36 @@ func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor Ou
|
|||||||
if err := validateModuleSpec("output encoder", StageOutput, normalizedSpec); err != nil {
|
if err := validateModuleSpec("output encoder", StageOutput, normalizedSpec); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if constructor == nil {
|
if validateOptions == nil {
|
||||||
return fmt.Errorf("output encoder constructor for %q must not be nil", normalizedSpec.Key)
|
return fmt.Errorf("output encoder option validator for %q must not be nil", normalizedSpec.Key)
|
||||||
}
|
}
|
||||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
if builder == nil {
|
||||||
|
return fmt.Errorf("output encoder builder for %q must not be nil", normalizedSpec.Key)
|
||||||
|
}
|
||||||
|
if _, ok := r.builders[normalizedSpec.Key]; ok {
|
||||||
return fmt.Errorf("output encoder %q is already registered", normalizedSpec.Key)
|
return fmt.Errorf("output encoder %q is already registered", normalizedSpec.Key)
|
||||||
}
|
}
|
||||||
|
|
||||||
if r.constructors == nil {
|
if r.builders == nil {
|
||||||
r.constructors = make(map[string]OutputEncoderConstructor)
|
r.builders = make(map[string]OutputEncoderBuilder)
|
||||||
|
}
|
||||||
|
if r.optionValidators == nil {
|
||||||
|
r.optionValidators = make(map[string]OptionValidator)
|
||||||
}
|
}
|
||||||
if r.specs == nil {
|
if r.specs == nil {
|
||||||
r.specs = make(map[string]ModuleSpec)
|
r.specs = make(map[string]ModuleSpec)
|
||||||
}
|
}
|
||||||
r.constructors[normalizedSpec.Key] = constructor
|
r.builders[normalizedSpec.Key] = builder
|
||||||
|
r.optionValidators[normalizedSpec.Key] = validateOptions
|
||||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *OutputEncoderRegistry) Build(key string) (contracts.OutputEncoder, error) {
|
func (r *OutputEncoderRegistry) Build(key string) (contracts.OutputEncoder, error) {
|
||||||
|
return r.BuildWithRequest(key, BuildRequest{})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *OutputEncoderRegistry) BuildWithRequest(key string, request BuildRequest) (contracts.OutputEncoder, error) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return nil, fmt.Errorf("output encoder registry must not be nil")
|
return nil, fmt.Errorf("output encoder registry must not be nil")
|
||||||
}
|
}
|
||||||
@@ -62,12 +85,12 @@ func (r *OutputEncoderRegistry) Build(key string) (contracts.OutputEncoder, erro
|
|||||||
return nil, fmt.Errorf("output encoder key must not be empty")
|
return nil, fmt.Errorf("output encoder key must not be empty")
|
||||||
}
|
}
|
||||||
|
|
||||||
constructor, ok := r.constructors[normalizedKey]
|
builder, ok := r.builders[normalizedKey]
|
||||||
if !ok {
|
if !ok {
|
||||||
return nil, fmt.Errorf("output encoder %q is not registered", normalizedKey)
|
return nil, fmt.Errorf("output encoder %q is not registered", normalizedKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
encoder, err := constructor()
|
encoder, err := builder(cloneBuildRequest(request))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("build output encoder %q: %w", normalizedKey, err)
|
return nil, fmt.Errorf("build output encoder %q: %w", normalizedKey, err)
|
||||||
}
|
}
|
||||||
@@ -81,6 +104,18 @@ func (r *OutputEncoderRegistry) Build(key string) (contracts.OutputEncoder, erro
|
|||||||
return encoder, nil
|
return encoder, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (r *OutputEncoderRegistry) ValidateOptions(key string, options map[string]any) error {
|
||||||
|
if r == nil {
|
||||||
|
return fmt.Errorf("output encoder registry must not be nil")
|
||||||
|
}
|
||||||
|
normalizedKey := strings.TrimSpace(key)
|
||||||
|
validator, ok := r.optionValidators[normalizedKey]
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("output encoder %q is not registered", normalizedKey)
|
||||||
|
}
|
||||||
|
return validateRegisteredOptions(validator, options)
|
||||||
|
}
|
||||||
|
|
||||||
func (r *OutputEncoderRegistry) Spec(key string) (ModuleSpec, bool) {
|
func (r *OutputEncoderRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return ModuleSpec{}, false
|
return ModuleSpec{}, false
|
||||||
@@ -98,5 +133,5 @@ func (r *OutputEncoderRegistry) RegisteredKeys() []string {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return sortedRegistryKeys(r.constructors)
|
return sortedRegistryKeys(r.builders)
|
||||||
}
|
}
|
||||||
|
|||||||
240
internal/framework/pipeline/preparation_test.go
Normal file
240
internal/framework/pipeline/preparation_test.go
Normal file
@@ -0,0 +1,240 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestResolvePipelineValidatesModuleAndValidatorOptions(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*PipelineProfile)
|
||||||
|
want []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "module",
|
||||||
|
mutate: func(profile *PipelineProfile) {
|
||||||
|
profile.Input.Options = map[string]any{"surprise": true}
|
||||||
|
},
|
||||||
|
want: []string{`pipeline "construction" input module "input" options`, `unknown option "surprise"`},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "validator",
|
||||||
|
mutate: func(profile *PipelineProfile) {
|
||||||
|
profile.Chunk.Validators = ValidatorOverride{Set: true, Validators: []ModuleBinding{{Module: "configured", Options: map[string]any{"surprise": true}}}}
|
||||||
|
},
|
||||||
|
want: []string{`pipeline "construction" chunk module "chunk" validator "configured" options`, `unknown option "surprise"`},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
registries, _ := constructionRegistries(t, nil, nil)
|
||||||
|
profile := constructionProfile()
|
||||||
|
test.mutate(&profile)
|
||||||
|
_, err := ResolvePipeline(profile, ResolveOptions{}, registries.catalog())
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("ResolvePipeline() error = nil, want option validation error")
|
||||||
|
}
|
||||||
|
for _, want := range test.want {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %q, want substring %q", err, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPrepareConstructsEverythingInStableOrder(t *testing.T) {
|
||||||
|
var built []string
|
||||||
|
registries, _ := constructionRegistries(t, &built, nil)
|
||||||
|
resolved, err := ResolvePipeline(constructionProfile(), ResolveOptions{}, registries.catalog())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
prepared, err := Prepare(resolved, registries, ModuleDependencies{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
want := []string{"input", "chunk", "validator", "extract", "validator", "merge", "validator", "normalize", "validator", "output"}
|
||||||
|
if !reflect.DeepEqual(built, want) {
|
||||||
|
t.Fatalf("construction order = %#v, want %#v", built, want)
|
||||||
|
}
|
||||||
|
if prepared.Input.Module != "input" || prepared.Chunk.Module != "chunk" || prepared.Output.Module != "output" || len(prepared.ArtifactLanes) != 1 {
|
||||||
|
t.Fatalf("PreparedPipeline = %#v, want explicit resolved components", prepared)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPrepareFailuresOccurBeforeInputParse(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
deps ModuleDependencies
|
||||||
|
configure func(*constructionFailure)
|
||||||
|
want string
|
||||||
|
wantBuilt []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "missing required llm dependency",
|
||||||
|
configure: func(failure *constructionFailure) {
|
||||||
|
failure.requireExtractorLLM = true
|
||||||
|
},
|
||||||
|
want: `lane "artifact" extract module "extract"`,
|
||||||
|
wantBuilt: []string{"input", "chunk", "validator", "extract"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "late output construction",
|
||||||
|
configure: func(failure *constructionFailure) {
|
||||||
|
failure.output = errors.New("output unavailable")
|
||||||
|
},
|
||||||
|
want: `output module "output"`,
|
||||||
|
wantBuilt: []string{"input", "chunk", "validator", "extract", "validator", "merge", "validator", "normalize", "validator", "output"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
failure := &constructionFailure{}
|
||||||
|
test.configure(failure)
|
||||||
|
var built []string
|
||||||
|
registries, input := constructionRegistries(t, &built, failure)
|
||||||
|
resolved, err := ResolvePipeline(constructionProfile(), ResolveOptions{}, registries.catalog())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = Prepare(resolved, registries, test.deps)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("Prepare() error = %v, want substring %q", err, test.want)
|
||||||
|
}
|
||||||
|
if len(input.requests) != 0 {
|
||||||
|
t.Fatalf("input Parse calls = %d, want zero", len(input.requests))
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(built, test.wantBuilt) {
|
||||||
|
t.Fatalf("construction order = %#v, want %#v", built, test.wantBuilt)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type constructionFailure struct {
|
||||||
|
requireExtractorLLM bool
|
||||||
|
output error
|
||||||
|
}
|
||||||
|
|
||||||
|
func constructionProfile() PipelineProfile {
|
||||||
|
validators := ValidatorOverride{Set: true, Validators: []ModuleBinding{{Module: "configured"}}}
|
||||||
|
return PipelineProfile{
|
||||||
|
ID: "construction",
|
||||||
|
Input: Binding("input"),
|
||||||
|
Chunk: ModuleBinding{Module: "chunk", Validators: validators},
|
||||||
|
Artifacts: map[string]ArtifactLaneProfile{
|
||||||
|
"artifact": {
|
||||||
|
Extract: ModuleBinding{Module: "extract", Validators: validators},
|
||||||
|
Merge: ModuleBinding{Module: "merge", Validators: validators},
|
||||||
|
Normalize: ModuleBinding{Module: "normalize", Validators: validators},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
Output: Binding("output"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func constructionRegistries(t *testing.T, built *[]string, failure *constructionFailure) (Registries, *constructionInput) {
|
||||||
|
t.Helper()
|
||||||
|
if built == nil {
|
||||||
|
built = &[]string{}
|
||||||
|
}
|
||||||
|
if failure == nil {
|
||||||
|
failure = &constructionFailure{}
|
||||||
|
}
|
||||||
|
record := func(name string) { *built = append(*built, name) }
|
||||||
|
strict := func(options map[string]any) error { return RejectUnknownOptions(options, "known") }
|
||||||
|
input := &constructionInput{key: "input"}
|
||||||
|
registries := Registries{
|
||||||
|
Inputs: NewInputAdapterRegistry(), Chunkers: NewChunkerRegistry(), ArtifactCodecs: NewArtifactCodecRegistry(),
|
||||||
|
Extractors: NewExtractorRegistry(), Mergers: NewMergerRegistry(), Normalizers: NewNormalizerRegistry(),
|
||||||
|
Validators: NewValidatorRegistry(), ValidatorChains: NewValidatorChainRegistry(), Outputs: NewOutputEncoderRegistry(),
|
||||||
|
}
|
||||||
|
if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := registries.Inputs.RegisterBuilderWithSpec(defaultModuleSpec("input", StageInput), strict, func(BuildRequest) (contracts.InputAdapter, error) {
|
||||||
|
record("input")
|
||||||
|
return input, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := registries.Chunkers.RegisterBuilderWithSpec(defaultModuleSpec("chunk", StageChunk), strict, func(BuildRequest) (contracts.Chunker, error) {
|
||||||
|
record("chunk")
|
||||||
|
return &typedTestChunker{key: "chunk"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
extractSpec := defaultModuleSpec("extract", StageExtract)
|
||||||
|
extractSpec.ArtifactKind = "test/notes"
|
||||||
|
if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) {
|
||||||
|
record("extract")
|
||||||
|
if failure.requireExtractorLLM && request.Dependencies.LLM == nil {
|
||||||
|
return nil, errors.New("structured LLM client is required")
|
||||||
|
}
|
||||||
|
return typedTestExtractor[codecNotes]{key: "extract"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
mergeSpec := defaultModuleSpec("merge", StageMerge)
|
||||||
|
mergeSpec.ArtifactKind = "test/notes"
|
||||||
|
if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(BuildRequest) (contracts.Merger[codecNotes], error) {
|
||||||
|
record("merge")
|
||||||
|
return typedTestMerger[codecNotes]{key: "merge"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
normalizeSpec := defaultModuleSpec("normalize", StageNormalize)
|
||||||
|
normalizeSpec.ArtifactKind = "test/notes"
|
||||||
|
if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(BuildRequest) (contracts.Normalizer[codecNotes], error) {
|
||||||
|
record("normalize")
|
||||||
|
return typedTestNormalizer[codecNotes]{key: "normalize"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
validatorSpec := ValidatorSpec{Key: "configured", ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
|
if err := RegisterChunkValidatorBuilder(registries.Validators, validatorSpec, strict, func(BuildRequest) (contracts.ChunkValidator, error) {
|
||||||
|
record("validator")
|
||||||
|
return typedTestChunkValidator{key: "configured"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := RegisterTypedValidatorBuilder(registries.Validators, "test/notes", validatorSpec, strict, func(BuildRequest) (contracts.TypedValidator[codecNotes], error) {
|
||||||
|
record("validator")
|
||||||
|
return typedTestValidator[codecNotes]{key: "configured"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := registries.Outputs.RegisterBuilderWithSpec(defaultModuleSpec("output", StageOutput), strict, func(BuildRequest) (contracts.OutputEncoder, error) {
|
||||||
|
record("output")
|
||||||
|
if failure.output != nil {
|
||||||
|
return nil, failure.output
|
||||||
|
}
|
||||||
|
return &typedTestOutput{key: "output"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return registries, input
|
||||||
|
}
|
||||||
|
|
||||||
|
type constructionInput struct {
|
||||||
|
key string
|
||||||
|
requests []contracts.ParseRequest
|
||||||
|
}
|
||||||
|
|
||||||
|
func (input *constructionInput) Key() string { return input.key }
|
||||||
|
func (input *constructionInput) Parse(_ context.Context, request contracts.ParseRequest) (*source.SourceDocument, error) {
|
||||||
|
input.requests = append(input.requests, request)
|
||||||
|
return typedTestDocument(), nil
|
||||||
|
}
|
||||||
348
internal/framework/pipeline/prepare.go
Normal file
348
internal/framework/pipeline/prepare.go
Normal file
@@ -0,0 +1,348 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
// PreparedPipeline owns the constructed, run-local implementation set for one
|
||||||
|
// resolved pipeline. Its implementation values are private so execution cannot
|
||||||
|
// replace or reconfigure them after preparation.
|
||||||
|
type PreparedPipeline struct {
|
||||||
|
Input ModuleBinding
|
||||||
|
Chunk ModuleBinding
|
||||||
|
ArtifactLanes []PreparedArtifactLane
|
||||||
|
Output ModuleBinding
|
||||||
|
|
||||||
|
resolved ResolvedPipeline
|
||||||
|
dependencies ModuleDependencies
|
||||||
|
input contracts.InputAdapter
|
||||||
|
chunker contracts.Chunker
|
||||||
|
chunkValidators preparedValidatorChain
|
||||||
|
lanes []preparedLaneExecutor
|
||||||
|
output contracts.OutputEncoder
|
||||||
|
}
|
||||||
|
|
||||||
|
type PreparedArtifactLane struct {
|
||||||
|
Resolved ResolvedArtifactLane
|
||||||
|
}
|
||||||
|
|
||||||
|
type preparedLaneExecutor struct {
|
||||||
|
resolved ResolvedArtifactLane
|
||||||
|
typed *preparedTypedLane
|
||||||
|
extractValidators preparedValidatorChain
|
||||||
|
mergeValidators preparedValidatorChain
|
||||||
|
normalizeValidators preparedValidatorChain
|
||||||
|
}
|
||||||
|
|
||||||
|
type preparedTypedLane struct {
|
||||||
|
extractor any
|
||||||
|
merger any
|
||||||
|
normalizer any
|
||||||
|
extract typedExtractOperation
|
||||||
|
merge typedMergeOperation
|
||||||
|
normalize typedNormalizeOperation
|
||||||
|
codec artifactCodecEntry
|
||||||
|
}
|
||||||
|
|
||||||
|
type preparedValidatorChain struct {
|
||||||
|
resolved ResolvedValidatorChain
|
||||||
|
validators []preparedValidator
|
||||||
|
}
|
||||||
|
|
||||||
|
type preparedValidator struct {
|
||||||
|
resolved ResolvedValidator
|
||||||
|
typed any
|
||||||
|
typedValidate typedValidateOperation
|
||||||
|
chunk contracts.ChunkValidator
|
||||||
|
serialized contracts.SerializedValidator
|
||||||
|
}
|
||||||
|
|
||||||
|
// Prepare validates all configured options and constructs every selected
|
||||||
|
// module and validator before any operation method can run.
|
||||||
|
func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDependencies) (*PreparedPipeline, error) {
|
||||||
|
if err := validateResolvedPipeline(resolved); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := validateRegistrySet(resolved, registries); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
stable := cloneResolvedPipeline(resolved)
|
||||||
|
prepared := &PreparedPipeline{
|
||||||
|
Input: cloneModuleBinding(stable.Input),
|
||||||
|
Chunk: cloneModuleBinding(stable.Chunk),
|
||||||
|
Output: cloneModuleBinding(stable.Output),
|
||||||
|
resolved: stable,
|
||||||
|
dependencies: deps,
|
||||||
|
}
|
||||||
|
request := func(binding ModuleBinding) BuildRequest {
|
||||||
|
return BuildRequest{Dependencies: deps, Options: cloneOptions(binding.Options)}
|
||||||
|
}
|
||||||
|
|
||||||
|
input, err := registries.Inputs.BuildWithRequest(stable.Input.Module, request(stable.Input))
|
||||||
|
if err != nil {
|
||||||
|
return nil, constructionError(stable.ID, "", StageInput, stable.Input.Module, "", err)
|
||||||
|
}
|
||||||
|
prepared.input = input
|
||||||
|
|
||||||
|
chunker, err := registries.Chunkers.BuildWithRequest(stable.Chunk.Module, request(stable.Chunk))
|
||||||
|
if err != nil {
|
||||||
|
return nil, constructionError(stable.ID, "", StageChunk, stable.Chunk.Module, "", err)
|
||||||
|
}
|
||||||
|
prepared.chunker = chunker
|
||||||
|
prepared.chunkValidators, err = prepareValidatorChain(stable, registries, deps, StageChunk, "", stable.Chunk.Module)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
prepared.ArtifactLanes = make([]PreparedArtifactLane, 0, len(stable.ArtifactLanes))
|
||||||
|
prepared.lanes = make([]preparedLaneExecutor, 0, len(stable.ArtifactLanes))
|
||||||
|
for _, lane := range stable.ArtifactLanes {
|
||||||
|
executor, err := prepareLane(stable, lane, registries, deps)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
prepared.ArtifactLanes = append(prepared.ArtifactLanes, PreparedArtifactLane{Resolved: cloneResolvedArtifactLane(lane)})
|
||||||
|
prepared.lanes = append(prepared.lanes, executor)
|
||||||
|
}
|
||||||
|
|
||||||
|
output, err := registries.Outputs.BuildWithRequest(stable.Output.Module, request(stable.Output))
|
||||||
|
if err != nil {
|
||||||
|
return nil, constructionError(stable.ID, "", StageOutput, stable.Output.Module, "", err)
|
||||||
|
}
|
||||||
|
prepared.output = output
|
||||||
|
return prepared, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func prepareLane(pipeline ResolvedPipeline, lane ResolvedArtifactLane, registries Registries, deps ModuleDependencies) (preparedLaneExecutor, error) {
|
||||||
|
executor := preparedLaneExecutor{resolved: cloneResolvedArtifactLane(lane)}
|
||||||
|
request := func(binding ModuleBinding) BuildRequest {
|
||||||
|
return BuildRequest{Dependencies: deps, Options: cloneOptions(binding.Options)}
|
||||||
|
}
|
||||||
|
extractEntry, ok := registries.Extractors.typedEntry(lane.Extract.Module)
|
||||||
|
if !ok {
|
||||||
|
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageExtract, lane.Extract.Module, "", fmt.Errorf("typed construction entry is not registered"))
|
||||||
|
}
|
||||||
|
module, err := buildErasedModule(extractEntry.builder, request(lane.Extract), lane.Extract.Module, "extractor")
|
||||||
|
if err != nil {
|
||||||
|
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageExtract, lane.Extract.Module, "", err)
|
||||||
|
}
|
||||||
|
codec, _, codecErr := registries.ArtifactCodecs.entry(lane.ArtifactKind)
|
||||||
|
if codecErr != nil {
|
||||||
|
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageExtract, lane.Extract.Module, "", codecErr)
|
||||||
|
}
|
||||||
|
executor.typed = &preparedTypedLane{extractor: module, extract: extractEntry.extract, codec: codec}
|
||||||
|
|
||||||
|
executor.extractValidators, err = prepareValidatorChain(pipeline, registries, deps, StageExtract, lane.ID, lane.Extract.Module)
|
||||||
|
if err != nil {
|
||||||
|
return preparedLaneExecutor{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
mergeEntry, ok := registries.Mergers.typedEntry(lane.Merge.Module, lane.ArtifactKind)
|
||||||
|
if !ok {
|
||||||
|
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageMerge, lane.Merge.Module, "", fmt.Errorf("typed construction entry is not registered"))
|
||||||
|
}
|
||||||
|
module, err = buildErasedModule(mergeEntry.builder, request(lane.Merge), lane.Merge.Module, "merger")
|
||||||
|
if err != nil {
|
||||||
|
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageMerge, lane.Merge.Module, "", err)
|
||||||
|
}
|
||||||
|
executor.typed.merger = module
|
||||||
|
executor.typed.merge = mergeEntry.merge
|
||||||
|
executor.mergeValidators, err = prepareValidatorChain(pipeline, registries, deps, StageMerge, lane.ID, lane.Merge.Module)
|
||||||
|
if err != nil {
|
||||||
|
return preparedLaneExecutor{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
normalizeEntry, ok := registries.Normalizers.typedEntry(lane.Normalize.Module, lane.ArtifactKind)
|
||||||
|
if !ok {
|
||||||
|
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageNormalize, lane.Normalize.Module, "", fmt.Errorf("typed construction entry is not registered"))
|
||||||
|
}
|
||||||
|
module, err = buildErasedModule(normalizeEntry.builder, request(lane.Normalize), lane.Normalize.Module, "normalizer")
|
||||||
|
if err != nil {
|
||||||
|
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageNormalize, lane.Normalize.Module, "", err)
|
||||||
|
}
|
||||||
|
executor.typed.normalizer = module
|
||||||
|
executor.typed.normalize = normalizeEntry.normalize
|
||||||
|
executor.normalizeValidators, err = prepareValidatorChain(pipeline, registries, deps, StageNormalize, lane.ID, lane.Normalize.Module)
|
||||||
|
if err != nil {
|
||||||
|
return preparedLaneExecutor{}, err
|
||||||
|
}
|
||||||
|
return executor, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func prepareValidatorChain(pipeline ResolvedPipeline, registries Registries, deps ModuleDependencies, stage ModuleStage, laneID, moduleKey string) (preparedValidatorChain, error) {
|
||||||
|
resolved := resolvedValidatorChain(stage, laneID, moduleKey, pipeline.ValidatorChains)
|
||||||
|
prepared := preparedValidatorChain{resolved: resolved}
|
||||||
|
for _, validator := range resolved.Validators {
|
||||||
|
request := BuildRequest{Dependencies: deps, Options: cloneOptions(validator.Binding.Options)}
|
||||||
|
built, err := buildPreparedValidator(registries.Validators, validator, request)
|
||||||
|
if err != nil {
|
||||||
|
return preparedValidatorChain{}, constructionError(pipeline.ID, laneID, stage, moduleKey, validator.Binding.Module, err)
|
||||||
|
}
|
||||||
|
prepared.validators = append(prepared.validators, built)
|
||||||
|
}
|
||||||
|
return prepared, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildPreparedValidator(registry *ValidatorRegistry, resolved ResolvedValidator, request BuildRequest) (preparedValidator, error) {
|
||||||
|
prepared := preparedValidator{resolved: resolved}
|
||||||
|
key := resolved.Binding.Module
|
||||||
|
var implementation any
|
||||||
|
var err error
|
||||||
|
switch resolved.Target {
|
||||||
|
case ValidatorTargetTyped:
|
||||||
|
entry, ok := registry.typedEntry(key, resolved.ArtifactKind)
|
||||||
|
if !ok {
|
||||||
|
return preparedValidator{}, fmt.Errorf("typed construction entry is not registered")
|
||||||
|
}
|
||||||
|
implementation, err = entry.builder(cloneBuildRequest(request))
|
||||||
|
prepared.typed = implementation
|
||||||
|
prepared.typedValidate = entry.validate
|
||||||
|
case ValidatorTargetChunk:
|
||||||
|
entry, ok := registry.chunkEntry(key)
|
||||||
|
if !ok {
|
||||||
|
return preparedValidator{}, fmt.Errorf("chunk construction entry is not registered")
|
||||||
|
}
|
||||||
|
prepared.chunk, err = entry.builder(cloneBuildRequest(request))
|
||||||
|
implementation = prepared.chunk
|
||||||
|
case ValidatorTargetSerialized:
|
||||||
|
entry, ok := registry.serializedEntry(key)
|
||||||
|
if !ok {
|
||||||
|
return preparedValidator{}, fmt.Errorf("serialized construction entry is not registered")
|
||||||
|
}
|
||||||
|
prepared.serialized, err = entry.builder(cloneBuildRequest(request))
|
||||||
|
implementation = prepared.serialized
|
||||||
|
default:
|
||||||
|
return preparedValidator{}, fmt.Errorf("validator construction target %q is not supported", resolved.Target)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return preparedValidator{}, err
|
||||||
|
}
|
||||||
|
if isNilImplementation(implementation) {
|
||||||
|
return preparedValidator{}, fmt.Errorf("validator %q builder returned nil", key)
|
||||||
|
}
|
||||||
|
identity, ok := implementation.(interface {
|
||||||
|
Name() string
|
||||||
|
ExecutionClass() contracts.ExecutionClass
|
||||||
|
})
|
||||||
|
if !ok {
|
||||||
|
return preparedValidator{}, fmt.Errorf("validator %q builder returned incompatible implementation %T", key, implementation)
|
||||||
|
}
|
||||||
|
if identity.Name() != key {
|
||||||
|
return preparedValidator{}, fmt.Errorf("validator %q returned name %q", key, identity.Name())
|
||||||
|
}
|
||||||
|
if identity.ExecutionClass() != resolved.ExecutionClass {
|
||||||
|
return preparedValidator{}, fmt.Errorf("validator %q returned execution class %q, want %q", key, identity.ExecutionClass(), resolved.ExecutionClass)
|
||||||
|
}
|
||||||
|
return prepared, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildErasedModule(builder func(BuildRequest) (any, error), request BuildRequest, key, kind string) (any, error) {
|
||||||
|
implementation, err := builder(cloneBuildRequest(request))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if isNilImplementation(implementation) {
|
||||||
|
return nil, fmt.Errorf("%s %q builder returned nil", kind, key)
|
||||||
|
}
|
||||||
|
identity, ok := implementation.(interface{ Key() string })
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("%s %q builder returned incompatible implementation %T", kind, key, implementation)
|
||||||
|
}
|
||||||
|
if identity.Key() != key {
|
||||||
|
return nil, fmt.Errorf("%s %q returned key %q", kind, key, identity.Key())
|
||||||
|
}
|
||||||
|
return implementation, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isNilImplementation(value any) bool {
|
||||||
|
if value == nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
reflected := reflect.ValueOf(value)
|
||||||
|
switch reflected.Kind() {
|
||||||
|
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
|
||||||
|
return reflected.IsNil()
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func constructionError(pipelineID, laneID string, stage ModuleStage, moduleKey, validatorKey string, cause error) error {
|
||||||
|
scope := fmt.Sprintf("pipeline %q %s module %q", pipelineID, stage, moduleKey)
|
||||||
|
if laneID != "" {
|
||||||
|
scope = fmt.Sprintf("pipeline %q lane %q %s module %q", pipelineID, laneID, stage, moduleKey)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(validatorKey) != "" {
|
||||||
|
scope += fmt.Sprintf(" validator %q", validatorKey)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("prepare %s: %w", scope, cause)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (registries Registries) catalog() ModuleCatalog {
|
||||||
|
return ModuleCatalog{
|
||||||
|
Inputs: registries.Inputs, Chunkers: registries.Chunkers, ArtifactCodecs: registries.ArtifactCodecs,
|
||||||
|
Extractors: registries.Extractors, Mergers: registries.Mergers, Normalizers: registries.Normalizers,
|
||||||
|
Validators: registries.Validators, ValidatorChains: registries.ValidatorChains, Outputs: registries.Outputs,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateRegistrySet(resolved ResolvedPipeline, registries Registries) error {
|
||||||
|
if registries.Inputs == nil {
|
||||||
|
return fmt.Errorf("input registry must not be nil")
|
||||||
|
}
|
||||||
|
if registries.Chunkers == nil {
|
||||||
|
return fmt.Errorf("chunker registry must not be nil")
|
||||||
|
}
|
||||||
|
if registries.Extractors == nil {
|
||||||
|
return fmt.Errorf("extractor registry must not be nil")
|
||||||
|
}
|
||||||
|
if registries.Mergers == nil {
|
||||||
|
return fmt.Errorf("merger registry must not be nil")
|
||||||
|
}
|
||||||
|
if registries.Normalizers == nil {
|
||||||
|
return fmt.Errorf("normalizer registry must not be nil")
|
||||||
|
}
|
||||||
|
if registries.Validators == nil {
|
||||||
|
for _, chain := range resolved.ValidatorChains {
|
||||||
|
if len(chain.Validators) > 0 {
|
||||||
|
return fmt.Errorf("validator registry must not be nil")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if registries.Outputs == nil {
|
||||||
|
return fmt.Errorf("output encoder registry must not be nil")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneResolvedPipeline(in ResolvedPipeline) ResolvedPipeline {
|
||||||
|
out := in
|
||||||
|
out.Input = cloneModuleBinding(in.Input)
|
||||||
|
out.Chunk = cloneModuleBinding(in.Chunk)
|
||||||
|
out.Output = cloneModuleBinding(in.Output)
|
||||||
|
out.ChunkReferences = CloneReferenceTarget(in.ChunkReferences)
|
||||||
|
out.ValidatorChains = cloneResolvedValidatorChains(in.ValidatorChains)
|
||||||
|
if len(in.ArtifactLanes) > 0 {
|
||||||
|
out.ArtifactLanes = make([]ResolvedArtifactLane, len(in.ArtifactLanes))
|
||||||
|
for i, lane := range in.ArtifactLanes {
|
||||||
|
out.ArtifactLanes[i] = cloneResolvedArtifactLane(lane)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneResolvedArtifactLane(in ResolvedArtifactLane) ResolvedArtifactLane {
|
||||||
|
out := in
|
||||||
|
out.Extract = cloneModuleBinding(in.Extract)
|
||||||
|
out.Merge = cloneModuleBinding(in.Merge)
|
||||||
|
out.Normalize = cloneModuleBinding(in.Normalize)
|
||||||
|
out.Validators = cloneModuleBindings(in.Validators)
|
||||||
|
out.ExtractReferences = CloneReferenceTarget(in.ExtractReferences)
|
||||||
|
out.MergeReferences = CloneReferenceTarget(in.MergeReferences)
|
||||||
|
out.NormalizeReferences = CloneReferenceTarget(in.NormalizeReferences)
|
||||||
|
return out
|
||||||
|
}
|
||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"encoding/hex"
|
"encoding/hex"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"reflect"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
@@ -102,14 +103,19 @@ type ResolvedReferenceTarget struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type ResolvedArtifactLane struct {
|
type ResolvedArtifactLane struct {
|
||||||
ID string
|
ID string
|
||||||
Extract ModuleBinding
|
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||||
Merge ModuleBinding
|
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
|
||||||
Normalize ModuleBinding
|
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
|
||||||
Validators []ModuleBinding
|
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
|
||||||
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
|
||||||
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
Extract ModuleBinding
|
||||||
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
Merge ModuleBinding
|
||||||
|
Normalize ModuleBinding
|
||||||
|
Validators []ModuleBinding
|
||||||
|
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
||||||
|
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
||||||
|
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ResolvedValidatorChain struct {
|
type ResolvedValidatorChain struct {
|
||||||
@@ -122,6 +128,8 @@ type ResolvedValidatorChain struct {
|
|||||||
type ResolvedValidator struct {
|
type ResolvedValidator struct {
|
||||||
Binding ModuleBinding `json:"binding"`
|
Binding ModuleBinding `json:"binding"`
|
||||||
ExecutionClass contracts.ExecutionClass `json:"execution_class"`
|
ExecutionClass contracts.ExecutionClass `json:"execution_class"`
|
||||||
|
Target ValidatorTarget `json:"target,omitempty"`
|
||||||
|
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ResolvedPipeline struct {
|
type ResolvedPipeline struct {
|
||||||
@@ -138,6 +146,7 @@ type ResolvedPipeline struct {
|
|||||||
type ModuleCatalog struct {
|
type ModuleCatalog struct {
|
||||||
Inputs *InputAdapterRegistry
|
Inputs *InputAdapterRegistry
|
||||||
Chunkers *ChunkerRegistry
|
Chunkers *ChunkerRegistry
|
||||||
|
ArtifactCodecs *ArtifactCodecRegistry
|
||||||
Extractors *ExtractorRegistry
|
Extractors *ExtractorRegistry
|
||||||
Mergers *MergerRegistry
|
Mergers *MergerRegistry
|
||||||
Normalizers *NormalizerRegistry
|
Normalizers *NormalizerRegistry
|
||||||
@@ -215,7 +224,7 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
|||||||
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
||||||
Output: resolveBinding(profile.Output, DefaultOutputModule),
|
Output: resolveBinding(profile.Output, DefaultOutputModule),
|
||||||
}
|
}
|
||||||
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, catalog)
|
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, "", nil, catalog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedPipeline{}, err
|
return ResolvedPipeline{}, err
|
||||||
}
|
}
|
||||||
@@ -240,6 +249,9 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
|||||||
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
|
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
|
||||||
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
|
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
|
||||||
}
|
}
|
||||||
|
if err := validateResolvedOptions(resolved, catalog); err != nil {
|
||||||
|
return ResolvedPipeline{}, err
|
||||||
|
}
|
||||||
|
|
||||||
digest, err := resolvedPipelineDigest(resolved)
|
digest, err := resolvedPipelineDigest(resolved)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -278,6 +290,10 @@ func resolveArtifactLane(
|
|||||||
if missing, ok := capabilities.missing(extractSpec.Requires); ok {
|
if missing, ok := capabilities.missing(extractSpec.Requires); ok {
|
||||||
return ResolvedArtifactLane{}, nil, nil, capabilityError(pipelineID, laneID, StageExtract, lane.Extract.Module, missing)
|
return ResolvedArtifactLane{}, nil, nil, capabilityError(pipelineID, laneID, StageExtract, lane.Extract.Module, missing)
|
||||||
}
|
}
|
||||||
|
artifactType, err := resolveArtifactIdentity(pipelineID, laneID, &lane, extractSpec, catalog)
|
||||||
|
if err != nil {
|
||||||
|
return ResolvedArtifactLane{}, nil, nil, err
|
||||||
|
}
|
||||||
extractReferences := mergeReferenceMaps(profile.References, lane.Extract.References)
|
extractReferences := mergeReferenceMaps(profile.References, lane.Extract.References)
|
||||||
references, err := resolveReferenceTargetBindings(referenceResolutionTarget{
|
references, err := resolveReferenceTargetBindings(referenceResolutionTarget{
|
||||||
PipelineID: pipelineID,
|
PipelineID: pipelineID,
|
||||||
@@ -295,7 +311,7 @@ func resolveArtifactLane(
|
|||||||
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
|
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
|
||||||
capabilities.add(extractSpec.Provides...)
|
capabilities.add(extractSpec.Provides...)
|
||||||
|
|
||||||
mergeSpec, err := mergerSpec(catalog, lane.Merge.Module)
|
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedArtifactLane{}, nil, nil, moduleLookupError(pipelineID, laneID, StageMerge, lane.Merge.Module, err)
|
return ResolvedArtifactLane{}, nil, nil, moduleLookupError(pipelineID, laneID, StageMerge, lane.Merge.Module, err)
|
||||||
}
|
}
|
||||||
@@ -318,7 +334,7 @@ func resolveArtifactLane(
|
|||||||
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
|
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
|
||||||
capabilities.add(mergeSpec.Provides...)
|
capabilities.add(mergeSpec.Provides...)
|
||||||
|
|
||||||
normalizeSpec, err := normalizerSpec(catalog, lane.Normalize.Module)
|
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedArtifactLane{}, nil, nil, moduleLookupError(pipelineID, laneID, StageNormalize, lane.Normalize.Module, err)
|
return ResolvedArtifactLane{}, nil, nil, moduleLookupError(pipelineID, laneID, StageNormalize, lane.Normalize.Module, err)
|
||||||
}
|
}
|
||||||
@@ -345,15 +361,15 @@ func resolveArtifactLane(
|
|||||||
return ResolvedArtifactLane{}, nil, nil, configuredValidatorsError(pipelineID, laneID)
|
return ResolvedArtifactLane{}, nil, nil, configuredValidatorsError(pipelineID, laneID)
|
||||||
}
|
}
|
||||||
|
|
||||||
extractValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageExtract, lane.Extract.Module, lane.Extract.Validators, catalog)
|
extractValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageExtract, lane.Extract.Module, lane.Extract.Validators, lane.ArtifactKind, artifactType, catalog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedArtifactLane{}, nil, nil, err
|
return ResolvedArtifactLane{}, nil, nil, err
|
||||||
}
|
}
|
||||||
mergeValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageMerge, lane.Merge.Module, lane.Merge.Validators, catalog)
|
mergeValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageMerge, lane.Merge.Module, lane.Merge.Validators, lane.ArtifactKind, artifactType, catalog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedArtifactLane{}, nil, nil, err
|
return ResolvedArtifactLane{}, nil, nil, err
|
||||||
}
|
}
|
||||||
normalizeValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageNormalize, lane.Normalize.Module, lane.Normalize.Validators, catalog)
|
normalizeValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageNormalize, lane.Normalize.Module, lane.Normalize.Validators, lane.ArtifactKind, artifactType, catalog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedArtifactLane{}, nil, nil, err
|
return ResolvedArtifactLane{}, nil, nil, err
|
||||||
}
|
}
|
||||||
@@ -366,7 +382,131 @@ func configuredValidatorsError(pipelineID string, laneID string) error {
|
|||||||
return fmt.Errorf("pipeline %q lane %q validators are not supported at artifact lane level; use extract.validators, merge.validators, or normalize.validators", pipelineID, laneID)
|
return fmt.Errorf("pipeline %q lane %q validators are not supported at artifact lane level; use extract.validators, merge.validators, or normalize.validators", pipelineID, laneID)
|
||||||
}
|
}
|
||||||
|
|
||||||
func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage, module string, override ValidatorOverride, catalog ModuleCatalog) (ResolvedValidatorChain, error) {
|
func resolveArtifactIdentity(pipelineID, laneID string, lane *ResolvedArtifactLane, extractSpec ModuleSpec, catalog ModuleCatalog) (reflect.Type, error) {
|
||||||
|
if extractSpec.ArtifactKind == "" {
|
||||||
|
return nil, fmt.Errorf("pipeline %q lane %q extract module %q does not declare an artifact kind", pipelineID, laneID, lane.Extract.Module)
|
||||||
|
}
|
||||||
|
if catalog.Extractors == nil {
|
||||||
|
return nil, fmt.Errorf("pipeline %q lane %q extractor registry must not be nil", pipelineID, laneID)
|
||||||
|
}
|
||||||
|
extractor, ok := catalog.Extractors.typedEntry(lane.Extract.Module)
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("pipeline %q lane %q extract module %q declares artifact kind %q without a typed registration", pipelineID, laneID, lane.Extract.Module, extractSpec.ArtifactKind)
|
||||||
|
}
|
||||||
|
if catalog.ArtifactCodecs == nil {
|
||||||
|
return nil, fmt.Errorf("pipeline %q lane %q artifact codec registry must not be nil for kind %q", pipelineID, laneID, extractSpec.ArtifactKind)
|
||||||
|
}
|
||||||
|
codecSpec, ok := catalog.ArtifactCodecs.Spec(extractSpec.ArtifactKind)
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("pipeline %q lane %q artifact codec %q is not registered", pipelineID, laneID, extractSpec.ArtifactKind)
|
||||||
|
}
|
||||||
|
codecType, ok := catalog.ArtifactCodecs.valueType(extractSpec.ArtifactKind)
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("pipeline %q lane %q artifact codec %q has no Go type", pipelineID, laneID, extractSpec.ArtifactKind)
|
||||||
|
}
|
||||||
|
if extractor.valueType != codecType {
|
||||||
|
return nil, artifactTypeMismatchError(pipelineID, laneID, StageExtract, lane.Extract.Module, extractSpec.ArtifactKind, codecType, extractor.valueType)
|
||||||
|
}
|
||||||
|
lane.ArtifactKind = codecSpec.Kind
|
||||||
|
lane.ArtifactSchemaID = codecSpec.Schema.ID
|
||||||
|
lane.ArtifactSchemaName = codecSpec.Schema.Name
|
||||||
|
lane.ArtifactSchemaVersion = codecSpec.Schema.Version
|
||||||
|
lane.ArtifactSchemaDigest = codecSpec.SchemaDigest
|
||||||
|
return codecType, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func mergerSpecForArtifact(catalog ModuleCatalog, key string, kind contracts.ArtifactKind, expectedType reflect.Type) (ModuleSpec, error) {
|
||||||
|
if kind == "" {
|
||||||
|
return mergerSpec(catalog, key)
|
||||||
|
}
|
||||||
|
if catalog.Mergers == nil {
|
||||||
|
return ModuleSpec{}, fmt.Errorf("module %q is not registered", key)
|
||||||
|
}
|
||||||
|
entry, ok := catalog.Mergers.typedEntry(key, kind)
|
||||||
|
if !ok {
|
||||||
|
return ModuleSpec{}, missingArtifactVariantError("merger", key, kind, catalog.Mergers.registeredKinds(key))
|
||||||
|
}
|
||||||
|
if entry.valueType != expectedType {
|
||||||
|
return ModuleSpec{}, fmt.Errorf("artifact kind %q requires Go type %s, but merger %q variant uses %s", kind, typeName(expectedType), key, typeName(entry.valueType))
|
||||||
|
}
|
||||||
|
return cloneModuleSpec(entry.spec), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizerSpecForArtifact(catalog ModuleCatalog, key string, kind contracts.ArtifactKind, expectedType reflect.Type) (ModuleSpec, error) {
|
||||||
|
if kind == "" {
|
||||||
|
return normalizerSpec(catalog, key)
|
||||||
|
}
|
||||||
|
if catalog.Normalizers == nil {
|
||||||
|
return ModuleSpec{}, fmt.Errorf("module %q is not registered", key)
|
||||||
|
}
|
||||||
|
entry, ok := catalog.Normalizers.typedEntry(key, kind)
|
||||||
|
if !ok {
|
||||||
|
return ModuleSpec{}, missingArtifactVariantError("normalizer", key, kind, catalog.Normalizers.registeredKinds(key))
|
||||||
|
}
|
||||||
|
if entry.valueType != expectedType {
|
||||||
|
return ModuleSpec{}, fmt.Errorf("artifact kind %q requires Go type %s, but normalizer %q variant uses %s", kind, typeName(expectedType), key, typeName(entry.valueType))
|
||||||
|
}
|
||||||
|
return cloneModuleSpec(entry.spec), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validatorSpecForTarget(registry *ValidatorRegistry, stage ModuleStage, key string, kind contracts.ArtifactKind, expectedType reflect.Type) (ValidatorSpec, ValidatorTarget, error) {
|
||||||
|
key = strings.TrimSpace(key)
|
||||||
|
if stage == StageChunk {
|
||||||
|
if entry, ok := registry.chunkEntry(key); ok {
|
||||||
|
return entry.spec, ValidatorTargetChunk, nil
|
||||||
|
}
|
||||||
|
if entry, ok := registry.serializedEntry(key); ok && entry.spec.SupportsChunks {
|
||||||
|
return entry.spec.ValidatorSpec, ValidatorTargetSerialized, nil
|
||||||
|
}
|
||||||
|
if spec, ok := registry.Spec(key); ok {
|
||||||
|
return spec, "", nil
|
||||||
|
}
|
||||||
|
return ValidatorSpec{}, "", fmt.Errorf("references unknown validator %q for chunk target", key)
|
||||||
|
}
|
||||||
|
if kind == "" {
|
||||||
|
if spec, ok := registry.Spec(key); ok {
|
||||||
|
return spec, "", nil
|
||||||
|
}
|
||||||
|
return ValidatorSpec{}, "", fmt.Errorf("references unknown validator %q without an artifact kind", key)
|
||||||
|
}
|
||||||
|
if entry, ok := registry.typedEntry(key, kind); ok {
|
||||||
|
if entry.valueType != expectedType {
|
||||||
|
return ValidatorSpec{}, "", fmt.Errorf("artifact kind %q requires Go type %s, but validator %q variant uses %s", kind, typeName(expectedType), key, typeName(entry.valueType))
|
||||||
|
}
|
||||||
|
return entry.spec, ValidatorTargetTyped, nil
|
||||||
|
}
|
||||||
|
if entry, ok := registry.serializedEntry(key); ok && entry.spec.SupportsArtifacts {
|
||||||
|
return entry.spec.ValidatorSpec, ValidatorTargetSerialized, nil
|
||||||
|
}
|
||||||
|
if _, ok := registry.Spec(key); !ok {
|
||||||
|
return ValidatorSpec{}, "", fmt.Errorf("references unknown validator %q for artifact kind %q", key, kind)
|
||||||
|
}
|
||||||
|
return ValidatorSpec{}, "", missingArtifactVariantError("validator", key, kind, registry.registeredTypedKinds(key))
|
||||||
|
}
|
||||||
|
|
||||||
|
func missingArtifactVariantError(moduleType, key string, kind contracts.ArtifactKind, registered []contracts.ArtifactKind) error {
|
||||||
|
if len(registered) == 0 {
|
||||||
|
return fmt.Errorf("%s %q has no typed variant for artifact kind %q", moduleType, key, kind)
|
||||||
|
}
|
||||||
|
values := make([]string, len(registered))
|
||||||
|
for i, value := range registered {
|
||||||
|
values[i] = string(value)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("%s %q has no typed variant for artifact kind %q; registered kinds: %s", moduleType, key, kind, strings.Join(values, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
|
func artifactTypeMismatchError(pipelineID, laneID string, stage ModuleStage, module string, kind contracts.ArtifactKind, expected, actual reflect.Type) error {
|
||||||
|
return fmt.Errorf("pipeline %q lane %q %s module %q artifact kind %q requires Go type %s, got %s", pipelineID, laneID, stage, module, kind, typeName(expected), typeName(actual))
|
||||||
|
}
|
||||||
|
|
||||||
|
func typeName(value reflect.Type) string {
|
||||||
|
if value == nil {
|
||||||
|
return "<nil>"
|
||||||
|
}
|
||||||
|
return value.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage, module string, override ValidatorOverride, artifactKind contracts.ArtifactKind, artifactType reflect.Type, catalog ModuleCatalog) (ResolvedValidatorChain, error) {
|
||||||
chain := ResolvedValidatorChain{
|
chain := ResolvedValidatorChain{
|
||||||
Stage: stage,
|
Stage: stage,
|
||||||
LaneID: strings.TrimSpace(laneID),
|
LaneID: strings.TrimSpace(laneID),
|
||||||
@@ -395,9 +535,9 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
|
|||||||
}
|
}
|
||||||
chain.Validators = make([]ResolvedValidator, 0, len(bindings))
|
chain.Validators = make([]ResolvedValidator, 0, len(bindings))
|
||||||
for _, validator := range bindings {
|
for _, validator := range bindings {
|
||||||
spec, ok := catalog.Validators.Spec(validator.Module)
|
spec, target, err := validatorSpecForTarget(catalog.Validators, stage, validator.Module, artifactKind, artifactType)
|
||||||
if !ok {
|
if err != nil {
|
||||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q references unknown validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
|
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(validator.LLMProfile) != "" && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
if strings.TrimSpace(validator.LLMProfile) != "" && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
||||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q assigns llm_profile to deterministic validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
|
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q assigns llm_profile to deterministic validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
|
||||||
@@ -405,6 +545,8 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
|
|||||||
chain.Validators = append(chain.Validators, ResolvedValidator{
|
chain.Validators = append(chain.Validators, ResolvedValidator{
|
||||||
Binding: cloneModuleBinding(validator),
|
Binding: cloneModuleBinding(validator),
|
||||||
ExecutionClass: spec.ExecutionClass,
|
ExecutionClass: spec.ExecutionClass,
|
||||||
|
Target: target,
|
||||||
|
ArtifactKind: artifactKind,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
return chain, nil
|
return chain, nil
|
||||||
@@ -435,6 +577,8 @@ func cloneResolvedValidators(validators []ResolvedValidator) []ResolvedValidator
|
|||||||
out[i] = ResolvedValidator{
|
out[i] = ResolvedValidator{
|
||||||
Binding: cloneModuleBinding(validator.Binding),
|
Binding: cloneModuleBinding(validator.Binding),
|
||||||
ExecutionClass: validator.ExecutionClass,
|
ExecutionClass: validator.ExecutionClass,
|
||||||
|
Target: validator.Target,
|
||||||
|
ArtifactKind: validator.ArtifactKind,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
@@ -497,7 +641,14 @@ func validatePipelineReferenceDefaults(
|
|||||||
}
|
}
|
||||||
|
|
||||||
merge := resolveBinding(laneProfile.Merge, DefaultMergeModule)
|
merge := resolveBinding(laneProfile.Merge, DefaultMergeModule)
|
||||||
mergeSpec, err := mergerSpec(catalog, merge.Module)
|
var artifactType reflect.Type
|
||||||
|
artifactKind := extractSpec.ArtifactKind
|
||||||
|
if artifactKind != "" && catalog.Extractors != nil {
|
||||||
|
if entry, ok := catalog.Extractors.typedEntry(extract.Module); ok {
|
||||||
|
artifactType = entry.valueType
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mergeSpec, err := mergerSpecForArtifact(catalog, merge.Module, artifactKind, artifactType)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return moduleLookupError(pipelineID, laneID, StageMerge, merge.Module, err)
|
return moduleLookupError(pipelineID, laneID, StageMerge, merge.Module, err)
|
||||||
}
|
}
|
||||||
@@ -506,7 +657,7 @@ func validatePipelineReferenceDefaults(
|
|||||||
}
|
}
|
||||||
|
|
||||||
normalize := resolveBinding(laneProfile.Normalize, DefaultNormalizeModule)
|
normalize := resolveBinding(laneProfile.Normalize, DefaultNormalizeModule)
|
||||||
normalizeSpec, err := normalizerSpec(catalog, normalize.Module)
|
normalizeSpec, err := normalizerSpecForArtifact(catalog, normalize.Module, artifactKind, artifactType)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return moduleLookupError(pipelineID, laneID, StageNormalize, normalize.Module, err)
|
return moduleLookupError(pipelineID, laneID, StageNormalize, normalize.Module, err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -766,20 +766,22 @@ func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t
|
|||||||
profile := baselineProfile()
|
profile := baselineProfile()
|
||||||
profile.References = map[string]string{"roster": "./roster.yml"}
|
profile.References = map[string]string{"roster": "./roster.yml"}
|
||||||
catalog := emptyProfileCatalog()
|
catalog := emptyProfileCatalog()
|
||||||
|
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, notesCodec())
|
||||||
for _, spec := range defaultProfileSpecs() {
|
for _, spec := range defaultProfileSpecs() {
|
||||||
if spec.Key != "event-extractor" {
|
if spec.Key != "event-extractor" {
|
||||||
registerProfileSpecs(t, catalog, spec)
|
registerProfileSpecs(t, catalog, spec)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err := catalog.Extractors.RegisterWithSpec(ModuleSpec{
|
if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{
|
||||||
Key: "event-extractor",
|
Key: "event-extractor",
|
||||||
Stage: StageExtract,
|
Stage: StageExtract,
|
||||||
Requires: []string{"chunk"},
|
ArtifactKind: "test/notes",
|
||||||
Provides: []string{"candidate"},
|
Requires: []string{"chunk"},
|
||||||
|
Provides: []string{"candidate"},
|
||||||
ReferenceSlots: []contracts.ReferenceSlot{
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
{Name: "roster", Required: true},
|
{Name: "roster", Required: true},
|
||||||
},
|
},
|
||||||
}, func() (contracts.Extractor, error) {
|
}, func() (contracts.Extractor[codecNotes], error) {
|
||||||
return nil, errors.New("constructor should not run")
|
return nil, errors.New("constructor should not run")
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
||||||
@@ -1177,6 +1179,7 @@ func newProfileCatalog(t *testing.T) ModuleCatalog {
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
catalog := emptyProfileCatalog()
|
catalog := emptyProfileCatalog()
|
||||||
|
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, notesCodec())
|
||||||
registerProfileSpecs(t, catalog, defaultProfileSpecs()...)
|
registerProfileSpecs(t, catalog, defaultProfileSpecs()...)
|
||||||
return catalog
|
return catalog
|
||||||
}
|
}
|
||||||
@@ -1192,6 +1195,9 @@ func newProfileCatalogWithOverrides(t *testing.T, overrides ...ModuleSpec) Modul
|
|||||||
|
|
||||||
specs := defaultProfileSpecs()
|
specs := defaultProfileSpecs()
|
||||||
for _, override := range overrides {
|
for _, override := range overrides {
|
||||||
|
if override.ArtifactKind == "" && (override.Stage == StageExtract || override.Stage == StageMerge || override.Stage == StageNormalize) {
|
||||||
|
override.ArtifactKind = "test/notes"
|
||||||
|
}
|
||||||
replaced := false
|
replaced := false
|
||||||
for index, spec := range specs {
|
for index, spec := range specs {
|
||||||
if spec.Stage == override.Stage && spec.Key == override.Key {
|
if spec.Stage == override.Stage && spec.Key == override.Key {
|
||||||
@@ -1206,6 +1212,7 @@ func newProfileCatalogWithOverrides(t *testing.T, overrides ...ModuleSpec) Modul
|
|||||||
}
|
}
|
||||||
|
|
||||||
catalog := emptyProfileCatalog()
|
catalog := emptyProfileCatalog()
|
||||||
|
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, notesCodec())
|
||||||
registerProfileSpecs(t, catalog, specs...)
|
registerProfileSpecs(t, catalog, specs...)
|
||||||
return catalog
|
return catalog
|
||||||
}
|
}
|
||||||
@@ -1214,6 +1221,7 @@ func emptyProfileCatalog() ModuleCatalog {
|
|||||||
return ModuleCatalog{
|
return ModuleCatalog{
|
||||||
Inputs: NewInputAdapterRegistry(),
|
Inputs: NewInputAdapterRegistry(),
|
||||||
Chunkers: NewChunkerRegistry(),
|
Chunkers: NewChunkerRegistry(),
|
||||||
|
ArtifactCodecs: NewArtifactCodecRegistry(),
|
||||||
Extractors: NewExtractorRegistry(),
|
Extractors: NewExtractorRegistry(),
|
||||||
Mergers: NewMergerRegistry(),
|
Mergers: NewMergerRegistry(),
|
||||||
Normalizers: NewNormalizerRegistry(),
|
Normalizers: NewNormalizerRegistry(),
|
||||||
@@ -1227,10 +1235,10 @@ func defaultProfileSpecs() []ModuleSpec {
|
|||||||
return []ModuleSpec{
|
return []ModuleSpec{
|
||||||
ModuleSpec{Key: "text", Stage: StageInput, Provides: []string{"source"}},
|
ModuleSpec{Key: "text", Stage: StageInput, Provides: []string{"source"}},
|
||||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
ModuleSpec{Key: "appendorder", Stage: StageMerge, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||||
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
ModuleSpec{Key: "noop", Stage: StageNormalize, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||||
ModuleSpec{Key: "grounded", Stage: StageValidate, Requires: []string{"normalized"}, Provides: []string{"validated"}},
|
ModuleSpec{Key: "grounded", Stage: StageValidate, Requires: []string{"normalized"}, Provides: []string{"validated"}},
|
||||||
ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||||
}
|
}
|
||||||
@@ -1240,30 +1248,40 @@ func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSp
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
for _, spec := range specs {
|
for _, spec := range specs {
|
||||||
|
if spec.ArtifactKind == "" && (spec.Stage == StageExtract || spec.Stage == StageMerge || spec.Stage == StageNormalize) {
|
||||||
|
spec.ArtifactKind = "test/notes"
|
||||||
|
}
|
||||||
switch spec.Stage {
|
switch spec.Stage {
|
||||||
case StageInput:
|
case StageInput:
|
||||||
if err := catalog.Inputs.RegisterWithSpec(spec, profileInputConstructor(spec.Key)); err != nil {
|
if err := catalog.Inputs.RegisterWithSpec(spec, profileInputConstructor(spec.Key)); err != nil {
|
||||||
t.Fatalf("register input spec %#v: %v", spec, err)
|
t.Fatalf("register input spec %#v: %v", spec, err)
|
||||||
}
|
}
|
||||||
case StageChunk:
|
case StageChunk:
|
||||||
if err := catalog.Chunkers.RegisterWithSpec(spec, profileChunkerConstructor(spec.Key)); err != nil {
|
validateOptions := func(options map[string]any) error { return RejectUnknownOptions(options, "a", "b", "size") }
|
||||||
|
if err := catalog.Chunkers.RegisterBuilderWithSpec(spec, validateOptions, func(BuildRequest) (contracts.Chunker, error) { return &typedTestChunker{key: spec.Key}, nil }); err != nil {
|
||||||
t.Fatalf("register chunk spec %#v: %v", spec, err)
|
t.Fatalf("register chunk spec %#v: %v", spec, err)
|
||||||
}
|
}
|
||||||
case StageExtract:
|
case StageExtract:
|
||||||
if err := catalog.Extractors.RegisterWithSpec(spec, profileExtractorConstructor(spec.Key)); err != nil {
|
if err := RegisterExtractor(catalog.Extractors, spec, func() (contracts.Extractor[codecNotes], error) {
|
||||||
|
return typedTestExtractor[codecNotes]{key: spec.Key}, nil
|
||||||
|
}); err != nil {
|
||||||
t.Fatalf("register extractor spec %#v: %v", spec, err)
|
t.Fatalf("register extractor spec %#v: %v", spec, err)
|
||||||
}
|
}
|
||||||
case StageMerge:
|
case StageMerge:
|
||||||
if err := catalog.Mergers.RegisterWithSpec(spec, profileMergerConstructor(spec.Key)); err != nil {
|
if err := RegisterMerger(catalog.Mergers, spec, func() (contracts.Merger[codecNotes], error) { return typedTestMerger[codecNotes]{key: spec.Key}, nil }); err != nil {
|
||||||
t.Fatalf("register merger spec %#v: %v", spec, err)
|
t.Fatalf("register merger spec %#v: %v", spec, err)
|
||||||
}
|
}
|
||||||
case StageNormalize:
|
case StageNormalize:
|
||||||
if err := catalog.Normalizers.RegisterWithSpec(spec, profileNormalizerConstructor(spec.Key)); err != nil {
|
if err := RegisterNormalizer(catalog.Normalizers, spec, func() (contracts.Normalizer[codecNotes], error) {
|
||||||
|
return typedTestNormalizer[codecNotes]{key: spec.Key}, nil
|
||||||
|
}); err != nil {
|
||||||
t.Fatalf("register normalizer spec %#v: %v", spec, err)
|
t.Fatalf("register normalizer spec %#v: %v", spec, err)
|
||||||
}
|
}
|
||||||
case StageValidate:
|
case StageValidate:
|
||||||
validatorSpec := ValidatorSpec{Key: spec.Key, ExecutionClass: contracts.ExecutionClassDeterministic}
|
validatorSpec := ValidatorSpec{Key: spec.Key, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
if err := catalog.Validators.RegisterWithSpec(validatorSpec, profileValidatorConstructor(spec.Key)); err != nil {
|
if err := RegisterTypedValidator(catalog.Validators, "test/notes", validatorSpec, func() (contracts.TypedValidator[codecNotes], error) {
|
||||||
|
return typedTestValidator[codecNotes]{key: spec.Key}, nil
|
||||||
|
}); err != nil {
|
||||||
t.Fatalf("register validator spec %#v: %v", spec, err)
|
t.Fatalf("register validator spec %#v: %v", spec, err)
|
||||||
}
|
}
|
||||||
case StageOutput:
|
case StageOutput:
|
||||||
@@ -1278,7 +1296,9 @@ func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSp
|
|||||||
|
|
||||||
func registerProfileValidatorSpec(t *testing.T, catalog ModuleCatalog, spec ValidatorSpec) {
|
func registerProfileValidatorSpec(t *testing.T, catalog ModuleCatalog, spec ValidatorSpec) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
if err := catalog.Validators.RegisterWithSpec(spec, profileValidatorConstructor(spec.Key)); err != nil {
|
if err := RegisterTypedValidator(catalog.Validators, "test/notes", spec, func() (contracts.TypedValidator[codecNotes], error) {
|
||||||
|
return typedTestValidator[codecNotes]{key: spec.Key}, nil
|
||||||
|
}); err != nil {
|
||||||
t.Fatalf("register validator spec %#v: %v", spec, err)
|
t.Fatalf("register validator spec %#v: %v", spec, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1301,36 +1321,6 @@ func (adapter profileInputAdapter) Parse(ctx context.Context, req contracts.Pars
|
|||||||
return &source.SourceDocument{}, nil
|
return &source.SourceDocument{}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func profileChunkerConstructor(key string) ChunkerConstructor {
|
|
||||||
return func() (contracts.Chunker, error) {
|
|
||||||
return registryChunker{key: key}, nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func profileExtractorConstructor(key string) ExtractorConstructor {
|
|
||||||
return func() (contracts.Extractor, error) {
|
|
||||||
return registryFakeExtractor{key: key}, nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func profileMergerConstructor(key string) MergerConstructor {
|
|
||||||
return func() (contracts.Merger, error) {
|
|
||||||
return registryMerger{key: key}, nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func profileNormalizerConstructor(key string) NormalizerConstructor {
|
|
||||||
return func() (contracts.Normalizer, error) {
|
|
||||||
return registryNormalizer{key: key}, nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func profileValidatorConstructor(key string) ValidatorConstructor {
|
|
||||||
return func() (contracts.Validator, error) {
|
|
||||||
return registryValidator{name: key}, nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func profileOutputConstructor(key string) OutputEncoderConstructor {
|
func profileOutputConstructor(key string) OutputEncoderConstructor {
|
||||||
return func() (contracts.OutputEncoder, error) {
|
return func() (contracts.OutputEncoder, error) {
|
||||||
return registryOutputEncoder{key: key}, nil
|
return registryOutputEncoder{key: key}, nil
|
||||||
|
|||||||
14
internal/framework/pipeline/reference_test_helpers_test.go
Normal file
14
internal/framework/pipeline/reference_test_helpers_test.go
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import "gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
|
||||||
|
func validSourceDocument() *source.SourceDocument {
|
||||||
|
return &source.SourceDocument{
|
||||||
|
ID: "source-1", Kind: "document", Format: "text/plain", Digest: "sha256:source",
|
||||||
|
Units: []source.SourceUnit{
|
||||||
|
{ID: 1, Kind: "unit", Text: "Source unit.", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1}},
|
||||||
|
{ID: 2, Kind: "unit", Text: "Second source unit.", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 2, EndUnitID: 2}},
|
||||||
|
{ID: 3, Kind: "unit", Text: "Third source unit.", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 3, EndUnitID: 3}},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,261 +0,0 @@
|
|||||||
package pipeline
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"reflect"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestRunnerUsesRegistries(t *testing.T) {
|
|
||||||
var built []string
|
|
||||||
var executed []string
|
|
||||||
registries := integrationRegistries(t, &built, &executed)
|
|
||||||
|
|
||||||
output, err := New(registries).Run(context.Background(), RunInput{
|
|
||||||
Pipeline: integrationPipeline(),
|
|
||||||
SourceID: "source-1",
|
|
||||||
RawInput: []byte("source text"),
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Run() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantBuilt := []string{"input", "chunk", "extract-first", "merge", "normalize", "extract-second", "merge", "normalize", "output"}
|
|
||||||
if !reflect.DeepEqual(built, wantBuilt) {
|
|
||||||
t.Fatalf("built = %#v, want %#v", built, wantBuilt)
|
|
||||||
}
|
|
||||||
if !reflect.DeepEqual(executed, []string{"extract-first:chunk-0", "extract-second:chunk-0"}) {
|
|
||||||
t.Fatalf("executed = %#v, want extractor chunk execution", executed)
|
|
||||||
}
|
|
||||||
if got := normalizeOutputKeys(output.NormalizeOutputs); !reflect.DeepEqual(got, []string{"normalize", "normalize"}) {
|
|
||||||
t.Fatalf("normalize output keys = %#v, want one output from each lane", got)
|
|
||||||
}
|
|
||||||
if len(output.Rejected) != 0 {
|
|
||||||
t.Fatalf("len(Rejected) = %d, want none", len(output.Rejected))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func integrationRegistries(t *testing.T, built, executed *[]string) Registries {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
registries := Registries{
|
|
||||||
Inputs: NewInputAdapterRegistry(),
|
|
||||||
Chunkers: NewChunkerRegistry(),
|
|
||||||
Extractors: NewExtractorRegistry(),
|
|
||||||
Mergers: NewMergerRegistry(),
|
|
||||||
Normalizers: NewNormalizerRegistry(),
|
|
||||||
Outputs: NewOutputEncoderRegistry(),
|
|
||||||
}
|
|
||||||
if err := registries.Inputs.Register("input", func() (contracts.InputAdapter, error) {
|
|
||||||
*built = append(*built, "input")
|
|
||||||
return integrationInput{}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("register input: %v", err)
|
|
||||||
}
|
|
||||||
if err := registries.Chunkers.Register("chunk", func() (contracts.Chunker, error) {
|
|
||||||
*built = append(*built, "chunk")
|
|
||||||
return integrationChunker{}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("register chunker: %v", err)
|
|
||||||
}
|
|
||||||
registerIntegrationExtractor(t, registries.Extractors, "extract-first", built, executed)
|
|
||||||
registerIntegrationExtractor(t, registries.Extractors, "extract-second", built, executed)
|
|
||||||
if err := registries.Mergers.Register("merge", func() (contracts.Merger, error) {
|
|
||||||
*built = append(*built, "merge")
|
|
||||||
return integrationMerger{}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("register merger: %v", err)
|
|
||||||
}
|
|
||||||
if err := registries.Normalizers.Register("normalize", func() (contracts.Normalizer, error) {
|
|
||||||
*built = append(*built, "normalize")
|
|
||||||
return integrationNormalizer{}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("register normalizer: %v", err)
|
|
||||||
}
|
|
||||||
if err := registries.Outputs.Register("output", func() (contracts.OutputEncoder, error) {
|
|
||||||
*built = append(*built, "output")
|
|
||||||
return integrationOutput{}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("register output: %v", err)
|
|
||||||
}
|
|
||||||
return registries
|
|
||||||
}
|
|
||||||
|
|
||||||
func registerIntegrationExtractor(t *testing.T, registry *ExtractorRegistry, key string, built, executed *[]string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
if err := registry.Register(key, func() (contracts.Extractor, error) {
|
|
||||||
*built = append(*built, key)
|
|
||||||
return integrationExtractor{key: key, executed: executed}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("Register(%q) error = %v, want nil", key, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
type integrationInput struct{}
|
|
||||||
|
|
||||||
func (input integrationInput) Key() string {
|
|
||||||
return "input"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (input integrationInput) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
|
|
||||||
return integrationSourceDocument(), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type integrationChunker struct{}
|
|
||||||
|
|
||||||
func (chunker integrationChunker) Key() string {
|
|
||||||
return "chunk"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (chunker integrationChunker) ReferenceSlots() []contracts.ReferenceSlot {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (chunker integrationChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
|
||||||
return contracts.ChunkResult{
|
|
||||||
Chunks: []contracts.SourceChunk{
|
|
||||||
{
|
|
||||||
ID: "chunk-0",
|
|
||||||
SourceID: req.Source.ID,
|
|
||||||
Index: 0,
|
|
||||||
StartUnitID: req.Source.Units[0].ID,
|
|
||||||
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
|
|
||||||
Content: []byte(`{"units":[1]}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
Units: req.Source.Units,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type integrationExtractor struct {
|
|
||||||
key string
|
|
||||||
executed *[]string
|
|
||||||
}
|
|
||||||
|
|
||||||
func (extractor integrationExtractor) Key() string {
|
|
||||||
return extractor.key
|
|
||||||
}
|
|
||||||
|
|
||||||
func (extractor integrationExtractor) ReferenceSlots() []contracts.ReferenceSlot {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (extractor integrationExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
|
||||||
*extractor.executed = append(*extractor.executed, extractor.key+":"+req.Chunk.ID)
|
|
||||||
return contracts.ExtractionResult{
|
|
||||||
Output: contracts.ExtractOutput{
|
|
||||||
Schema: contracts.ResponseSchema{ID: "integration", Name: "integration", Version: "v1"},
|
|
||||||
Payload: contracts.RawPayload{
|
|
||||||
Content: []byte(`{"value":true}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type integrationNormalizer struct{}
|
|
||||||
|
|
||||||
type integrationMerger struct{}
|
|
||||||
|
|
||||||
func (merger integrationMerger) Key() string {
|
|
||||||
return "merge"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (merger integrationMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
|
|
||||||
output := contracts.MergeOutput{
|
|
||||||
LaneID: req.LaneID,
|
|
||||||
Schema: contracts.ResponseSchema{ID: "integration", Name: "integration", Version: "v1"},
|
|
||||||
Payload: contracts.RawPayload{
|
|
||||||
Content: []byte(`{"merged":true}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
if len(req.ExtractOutputs) > 0 {
|
|
||||||
output.SourceID = req.ExtractOutputs[0].SourceID
|
|
||||||
output.Schema = req.ExtractOutputs[0].Schema
|
|
||||||
output.Payload = req.ExtractOutputs[0].Payload
|
|
||||||
}
|
|
||||||
return contracts.MergeResult{Output: output}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (normalizer integrationNormalizer) Key() string {
|
|
||||||
return "normalize"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (normalizer integrationNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (normalizer integrationNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
|
|
||||||
return contracts.NormalizeResult{
|
|
||||||
Output: contracts.NormalizeOutput{
|
|
||||||
LaneID: req.LaneID,
|
|
||||||
SourceID: req.MergeOutput.SourceID,
|
|
||||||
Schema: req.MergeOutput.Schema,
|
|
||||||
Payload: req.MergeOutput.Payload,
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type integrationOutput struct{}
|
|
||||||
|
|
||||||
func (output integrationOutput) Key() string {
|
|
||||||
return "output"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (output integrationOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
|
||||||
return contracts.OutputResult{
|
|
||||||
Files: []contracts.OutputFile{
|
|
||||||
{Name: "output.json", ContentType: "application/json", Bytes: []byte(`{}`)},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func integrationPipeline() ResolvedPipeline {
|
|
||||||
return ResolvedPipeline{
|
|
||||||
ID: "pipeline-1",
|
|
||||||
Digest: "sha256:pipeline",
|
|
||||||
Input: Binding("input"),
|
|
||||||
Chunk: Binding("chunk"),
|
|
||||||
ArtifactLanes: []ResolvedArtifactLane{
|
|
||||||
{
|
|
||||||
ID: "first",
|
|
||||||
Extract: Binding("extract-first"),
|
|
||||||
Merge: Binding("merge"),
|
|
||||||
Normalize: Binding("normalize"),
|
|
||||||
},
|
|
||||||
{
|
|
||||||
ID: "second",
|
|
||||||
Extract: Binding("extract-second"),
|
|
||||||
Merge: Binding("merge"),
|
|
||||||
Normalize: Binding("normalize"),
|
|
||||||
},
|
|
||||||
},
|
|
||||||
Output: Binding("output"),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func integrationSourceDocument() *source.SourceDocument {
|
|
||||||
return &source.SourceDocument{
|
|
||||||
ID: "source-1",
|
|
||||||
Kind: "document",
|
|
||||||
Format: "text/plain",
|
|
||||||
Digest: "sha256:abc123",
|
|
||||||
Units: []source.SourceUnit{
|
|
||||||
{ID: 1, Kind: "unit", Text: "Source unit."},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func normalizeOutputKeys(outputs []contracts.NormalizeOutput) []string {
|
|
||||||
keys := make([]string, 0, len(outputs))
|
|
||||||
for _, output := range outputs {
|
|
||||||
keys = append(keys, output.NormalizerKey)
|
|
||||||
}
|
|
||||||
return keys
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
408
internal/framework/pipeline/runner_concurrency_test.go
Normal file
408
internal/framework/pipeline/runner_concurrency_test.go
Normal file
@@ -0,0 +1,408 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
)
|
||||||
|
|
||||||
|
type countingProvider struct {
|
||||||
|
active atomic.Int32
|
||||||
|
maximum atomic.Int32
|
||||||
|
calls atomic.Int32
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *countingProvider) CompleteStructured(ctx context.Context, _ contracts.StructuredCompletionRequest, _ any) (contracts.StructuredCompletionResponse, error) {
|
||||||
|
p.calls.Add(1)
|
||||||
|
current := p.active.Add(1)
|
||||||
|
defer p.active.Add(-1)
|
||||||
|
for {
|
||||||
|
seen := p.maximum.Load()
|
||||||
|
if current <= seen || p.maximum.CompareAndSwap(seen, current) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-time.After(5 * time.Millisecond):
|
||||||
|
return contracts.StructuredCompletionResponse{}, nil
|
||||||
|
case <-ctx.Done():
|
||||||
|
return contracts.StructuredCompletionResponse{}, ctx.Err()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func preparedConcurrentPipeline(t *testing.T, chunkCount int) *PreparedPipeline {
|
||||||
|
t.Helper()
|
||||||
|
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
|
||||||
|
resolved, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, catalog)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v", err)
|
||||||
|
}
|
||||||
|
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Prepare() error = %v", err)
|
||||||
|
}
|
||||||
|
doc := typedTestDocument()
|
||||||
|
chunks := make([]source.Chunk, chunkCount)
|
||||||
|
for i := range chunks {
|
||||||
|
chunks[i] = source.Chunk{ID: fmt.Sprintf("chunk-%d", i+1), SourceID: doc.ID, Index: i, Ref: doc.Units[0].Ref, Content: []byte(fmt.Sprintf(`{"chunk":%d}`, i+1)), MediaType: "application/json", Units: []source.SourceUnit{doc.Units[0]}}
|
||||||
|
}
|
||||||
|
prepared.chunker = &typedTestChunker{key: "typed/chunk", chunks: chunks}
|
||||||
|
return prepared
|
||||||
|
}
|
||||||
|
|
||||||
|
func installExtractOperation(prepared *PreparedPipeline, laneIndex int, operation func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error)) {
|
||||||
|
prepared.lanes[laneIndex].typed.extract = func(ctx context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
|
return operation(ctx, request)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func typedValueForLane(laneIndex int, chunkIndex int) any {
|
||||||
|
if laneIndex == 0 {
|
||||||
|
return codecNotes{Items: []string{fmt.Sprintf("chunk-%d", chunkIndex)}}
|
||||||
|
}
|
||||||
|
return codecScore{Value: chunkIndex}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerBoundsExtractJobsAndStabilizesReverseCompletion(t *testing.T) {
|
||||||
|
prepared := preparedConcurrentPipeline(t, 3)
|
||||||
|
started := make(chan int, 6)
|
||||||
|
release := make([]chan struct{}, 6)
|
||||||
|
for i := range release {
|
||||||
|
release[i] = make(chan struct{})
|
||||||
|
}
|
||||||
|
var active atomic.Int32
|
||||||
|
var maximum atomic.Int32
|
||||||
|
for laneIndex := range prepared.lanes {
|
||||||
|
lane := laneIndex
|
||||||
|
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
|
jobIndex := request.Chunk.Index*2 + lane
|
||||||
|
current := active.Add(1)
|
||||||
|
defer active.Add(-1)
|
||||||
|
for {
|
||||||
|
seen := maximum.Load()
|
||||||
|
if current <= seen || maximum.CompareAndSwap(seen, current) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
started <- jobIndex
|
||||||
|
select {
|
||||||
|
case <-release[jobIndex]:
|
||||||
|
case <-ctx.Done():
|
||||||
|
return erasedTypedResult{}, ctx.Err()
|
||||||
|
}
|
||||||
|
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index), Warnings: []contracts.Warning{{Scope: fmt.Sprintf("lane-%d/chunk-%d", lane, request.Chunk.Index), ReasonCode: "observed", Message: "ordered"}}}, nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
type outcome struct {
|
||||||
|
output RunOutput
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
done := make(chan outcome, 1)
|
||||||
|
go func() {
|
||||||
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||||
|
done <- outcome{output: output, err: err}
|
||||||
|
}()
|
||||||
|
for pair := 0; pair < 3; pair++ {
|
||||||
|
first, second := <-started, <-started
|
||||||
|
want := map[int]bool{pair * 2: true, pair*2 + 1: true}
|
||||||
|
if !want[first] || !want[second] || first == second {
|
||||||
|
t.Fatalf("started jobs = %d, %d; want round-robin pair %d", first, second, pair)
|
||||||
|
}
|
||||||
|
close(release[second])
|
||||||
|
close(release[first])
|
||||||
|
}
|
||||||
|
result := <-done
|
||||||
|
if result.err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", result.err)
|
||||||
|
}
|
||||||
|
if got := maximum.Load(); got != 2 {
|
||||||
|
t.Fatalf("maximum concurrent extract jobs = %d, want 2", got)
|
||||||
|
}
|
||||||
|
wantScopes := []string{"lane-0/chunk-0", "lane-0/chunk-1", "lane-0/chunk-2", "lane-1/chunk-0", "lane-1/chunk-1", "lane-1/chunk-2"}
|
||||||
|
gotScopes := make([]string, len(result.output.Warnings))
|
||||||
|
for i := range result.output.Warnings {
|
||||||
|
gotScopes[i] = result.output.Warnings[i].Scope
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(gotScopes, wantScopes) {
|
||||||
|
t.Fatalf("warning order = %#v, want %#v", gotScopes, wantScopes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerStartsLaneContinuationWhileOtherLaneExtractsRemain(t *testing.T) {
|
||||||
|
prepared := preparedConcurrentPipeline(t, 2)
|
||||||
|
releaseOtherLane := make(chan struct{})
|
||||||
|
mergeStarted := make(chan struct{}, 1)
|
||||||
|
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
|
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
|
||||||
|
})
|
||||||
|
installExtractOperation(prepared, 1, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
|
select {
|
||||||
|
case <-releaseOtherLane:
|
||||||
|
return erasedTypedResult{Value: typedValueForLane(1, request.Chunk.Index)}, nil
|
||||||
|
case <-ctx.Done():
|
||||||
|
return erasedTypedResult{}, ctx.Err()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
originalMerge := prepared.lanes[0].typed.merge
|
||||||
|
prepared.lanes[0].typed.merge = func(ctx context.Context, implementation any, request contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||||
|
select {
|
||||||
|
case mergeStarted <- struct{}{}:
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
return originalMerge(ctx, implementation, request)
|
||||||
|
}
|
||||||
|
done := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||||
|
done <- err
|
||||||
|
}()
|
||||||
|
select {
|
||||||
|
case <-mergeStarted:
|
||||||
|
close(releaseOtherLane)
|
||||||
|
case <-time.After(2 * time.Second):
|
||||||
|
t.Fatal("first lane merge did not start while the second lane remained in extract")
|
||||||
|
}
|
||||||
|
if err := <-done; err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerBoundsConcurrentLaneContinuations(t *testing.T) {
|
||||||
|
prepared := preparedConcurrentPipeline(t, 1)
|
||||||
|
base := prepared.lanes[0]
|
||||||
|
lanes := make([]preparedLaneExecutor, 8)
|
||||||
|
resolvedLanes := make([]ResolvedArtifactLane, len(lanes))
|
||||||
|
publicLanes := make([]PreparedArtifactLane, len(lanes))
|
||||||
|
var active atomic.Int32
|
||||||
|
var maximum atomic.Int32
|
||||||
|
for i := range lanes {
|
||||||
|
lane := base
|
||||||
|
lane.resolved.ID = fmt.Sprintf("lane-%02d", i)
|
||||||
|
lane.typed = &preparedTypedLane{
|
||||||
|
extractor: base.typed.extractor,
|
||||||
|
merger: base.typed.merger,
|
||||||
|
normalizer: base.typed.normalizer,
|
||||||
|
extract: func(_ context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
|
return erasedTypedResult{Value: codecNotes{Items: []string{request.Chunk.ID}}}, nil
|
||||||
|
},
|
||||||
|
merge: func(_ context.Context, _ any, _ contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||||
|
current := active.Add(1)
|
||||||
|
defer active.Add(-1)
|
||||||
|
for {
|
||||||
|
seen := maximum.Load()
|
||||||
|
if current <= seen || maximum.CompareAndSwap(seen, current) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
time.Sleep(5 * time.Millisecond)
|
||||||
|
return erasedTypedResult{Value: codecNotes{}}, nil
|
||||||
|
},
|
||||||
|
normalize: base.typed.normalize,
|
||||||
|
codec: base.typed.codec,
|
||||||
|
}
|
||||||
|
lanes[i] = lane
|
||||||
|
resolvedLanes[i] = lane.resolved
|
||||||
|
publicLanes[i] = PreparedArtifactLane{Resolved: lane.resolved}
|
||||||
|
}
|
||||||
|
prepared.lanes = lanes
|
||||||
|
prepared.resolved.ArtifactLanes = resolvedLanes
|
||||||
|
prepared.ArtifactLanes = publicLanes
|
||||||
|
|
||||||
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(output.NormalizeOutputs) != len(lanes) {
|
||||||
|
t.Fatalf("normalize outputs = %d, want %d", len(output.NormalizeOutputs), len(lanes))
|
||||||
|
}
|
||||||
|
if got := maximum.Load(); got != 2 {
|
||||||
|
t.Fatalf("maximum concurrent lane continuations = %d, want 2", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerSelectsFrameworkErrorByStableLaneOrder(t *testing.T) {
|
||||||
|
prepared := preparedConcurrentPipeline(t, 1)
|
||||||
|
ready := make(chan struct{}, 2)
|
||||||
|
release := make(chan struct{})
|
||||||
|
for laneIndex := range prepared.lanes {
|
||||||
|
lane := laneIndex
|
||||||
|
installExtractOperation(prepared, lane, func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
|
ready <- struct{}{}
|
||||||
|
<-release
|
||||||
|
return erasedTypedResult{}, fmt.Errorf("lane-%d failure", lane)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
done := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||||
|
done <- err
|
||||||
|
}()
|
||||||
|
<-ready
|
||||||
|
<-ready
|
||||||
|
close(release)
|
||||||
|
if err := <-done; err == nil || !strings.Contains(err.Error(), "lane-0 failure") {
|
||||||
|
t.Fatalf("Run() error = %v, want first resolved lane failure", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerSelectsMergeErrorBeforeEarlierLaneNormalizeError(t *testing.T) {
|
||||||
|
prepared := preparedConcurrentPipeline(t, 1)
|
||||||
|
for laneIndex := range prepared.lanes {
|
||||||
|
lane := laneIndex
|
||||||
|
installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
|
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
ready := make(chan struct{}, 2)
|
||||||
|
release := make(chan struct{})
|
||||||
|
prepared.lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||||
|
ready <- struct{}{}
|
||||||
|
<-release
|
||||||
|
return erasedTypedResult{}, errors.New("earlier lane normalize failure")
|
||||||
|
}
|
||||||
|
prepared.lanes[1].typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||||
|
ready <- struct{}{}
|
||||||
|
<-release
|
||||||
|
return erasedTypedResult{}, errors.New("later lane merge failure")
|
||||||
|
}
|
||||||
|
done := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||||
|
done <- err
|
||||||
|
}()
|
||||||
|
<-ready
|
||||||
|
<-ready
|
||||||
|
close(release)
|
||||||
|
if err := <-done; err == nil || !strings.Contains(err.Error(), "later lane merge failure") {
|
||||||
|
t.Fatalf("Run() error = %v, want merge failure before normalize failure", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerKeepsSuccessfulChunksWhenAnotherExtractIsRejected(t *testing.T) {
|
||||||
|
prepared := preparedConcurrentPipeline(t, 2)
|
||||||
|
for laneIndex := range prepared.lanes {
|
||||||
|
lane := laneIndex
|
||||||
|
installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
|
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
validator := &prepared.lanes[0].extractValidators.validators[0]
|
||||||
|
validator.typedValidate = func(_ context.Context, _ any, target typedValidationTarget) (contracts.ValidationResult, error) {
|
||||||
|
if target.chunk != nil && target.chunk.Index == 0 {
|
||||||
|
return contracts.ValidationResult{Approved: false, ReasonCode: "expected_rejection", Message: "rejected by test"}, nil
|
||||||
|
}
|
||||||
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
|
}
|
||||||
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(output.Rejected) != 1 || output.Rejected[0].LaneID != prepared.lanes[0].resolved.ID || output.Rejected[0].ChunkIndex != 0 {
|
||||||
|
t.Fatalf("rejections = %#v, want the first lane's first chunk", output.Rejected)
|
||||||
|
}
|
||||||
|
if len(output.NormalizeOutputs) != 2 {
|
||||||
|
t.Fatalf("normalize outputs = %d, want both lanes to continue", len(output.NormalizeOutputs))
|
||||||
|
}
|
||||||
|
if output.Manifest.ValidationStatus != "rejected" {
|
||||||
|
t.Fatalf("validation status = %q, want rejected", output.Manifest.ValidationStatus)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerSeparatelyBoundsExtractJobsAndSharedProviderCalls(t *testing.T) {
|
||||||
|
prepared := preparedConcurrentPipeline(t, 3)
|
||||||
|
provider := &countingProvider{}
|
||||||
|
scheduler, err := frameworkllm.NewScheduler(2)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewScheduler() error = %v", err)
|
||||||
|
}
|
||||||
|
client := frameworkllm.NewScheduledClient(provider, scheduler)
|
||||||
|
var jobActive atomic.Int32
|
||||||
|
var jobMaximum atomic.Int32
|
||||||
|
var attempts sync.Map
|
||||||
|
for laneIndex := range prepared.lanes {
|
||||||
|
lane := laneIndex
|
||||||
|
prepared.lanes[lane].resolved.Extract.Retries = 1
|
||||||
|
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
|
current := jobActive.Add(1)
|
||||||
|
defer jobActive.Add(-1)
|
||||||
|
for {
|
||||||
|
seen := jobMaximum.Load()
|
||||||
|
if current <= seen || jobMaximum.CompareAndSwap(seen, current) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, callErr := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{StageName: "extract"}, nil); callErr != nil {
|
||||||
|
return erasedTypedResult{}, callErr
|
||||||
|
}
|
||||||
|
key := fmt.Sprintf("%d/%d", lane, request.Chunk.Index)
|
||||||
|
countValue, _ := attempts.LoadOrStore(key, &atomic.Int32{})
|
||||||
|
count := countValue.(*atomic.Int32).Add(1)
|
||||||
|
if lane == 0 && request.Chunk.Index == 0 && count == 1 {
|
||||||
|
return erasedTypedResult{}, errors.New("retry extract")
|
||||||
|
}
|
||||||
|
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
|
||||||
|
})
|
||||||
|
validator := &prepared.lanes[lane].extractValidators.validators[0]
|
||||||
|
validator.typedValidate = func(ctx context.Context, _ any, _ typedValidationTarget) (contracts.ValidationResult, error) {
|
||||||
|
if _, callErr := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{StageName: "validate"}, nil); callErr != nil {
|
||||||
|
return contracts.ValidationResult{}, callErr
|
||||||
|
}
|
||||||
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 3})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(output.NormalizeOutputs) != 2 {
|
||||||
|
t.Fatalf("normalize outputs = %d, want 2", len(output.NormalizeOutputs))
|
||||||
|
}
|
||||||
|
if got := jobMaximum.Load(); got != 3 {
|
||||||
|
t.Fatalf("maximum extract jobs = %d, want 3", got)
|
||||||
|
}
|
||||||
|
if got := provider.maximum.Load(); got != 2 {
|
||||||
|
t.Fatalf("maximum provider calls = %d, want 2", got)
|
||||||
|
}
|
||||||
|
if got := provider.calls.Load(); got != 13 {
|
||||||
|
t.Fatalf("provider calls = %d, want 13 including retry and validators", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerReturnsParentCancellationAndStopsQueuedExtracts(t *testing.T) {
|
||||||
|
prepared := preparedConcurrentPipeline(t, 4)
|
||||||
|
started := make(chan struct{}, 8)
|
||||||
|
for laneIndex := range prepared.lanes {
|
||||||
|
lane := laneIndex
|
||||||
|
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
|
started <- struct{}{}
|
||||||
|
<-ctx.Done()
|
||||||
|
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, ctx.Err()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
done := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
_, err := New().Run(ctx, RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||||
|
done <- err
|
||||||
|
}()
|
||||||
|
<-started
|
||||||
|
<-started
|
||||||
|
cancel()
|
||||||
|
if err := <-done; !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatalf("Run() error = %v, want context.Canceled", err)
|
||||||
|
}
|
||||||
|
if got := len(started); got != 0 {
|
||||||
|
t.Fatalf("additional started jobs = %d, want none after cancellation", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
399
internal/framework/pipeline/runner_concurrent.go
Normal file
399
internal/framework/pipeline/runner_concurrent.go
Normal file
@@ -0,0 +1,399 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"path"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
type laneExtractState struct {
|
||||||
|
index int
|
||||||
|
prepared preparedLaneExecutor
|
||||||
|
deps []CheckpointFingerprint
|
||||||
|
decision CheckpointDecision
|
||||||
|
values []erasedExtractArtifact
|
||||||
|
serialized []CheckpointArtifact
|
||||||
|
warnings []contracts.Warning
|
||||||
|
rejected []contracts.RejectedOutput
|
||||||
|
results map[int]extractJobResult
|
||||||
|
remaining int
|
||||||
|
failed bool
|
||||||
|
}
|
||||||
|
|
||||||
|
type extractJob struct {
|
||||||
|
lane *laneExtractState
|
||||||
|
chunk source.Chunk
|
||||||
|
}
|
||||||
|
|
||||||
|
type extractJobResult struct {
|
||||||
|
laneIndex int
|
||||||
|
chunkIndex int
|
||||||
|
value erasedExtractArtifact
|
||||||
|
serialized CheckpointArtifact
|
||||||
|
warnings []contracts.Warning
|
||||||
|
rejected *contracts.RejectedOutput
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type laneCompletion struct {
|
||||||
|
index int
|
||||||
|
output RunOutput
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type orderedRunError struct {
|
||||||
|
stage int
|
||||||
|
lane int
|
||||||
|
chunk int
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type completedExtractLoader struct {
|
||||||
|
CheckpointLoader
|
||||||
|
laneID string
|
||||||
|
checkpoint ExtractCheckpoint
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l completedExtractLoader) Extract(laneID, _ string, _ []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||||
|
if laneID == l.laneID {
|
||||||
|
return l.checkpoint, CheckpointDecision{Reused: true, Reason: "coordinated extract result"}
|
||||||
|
}
|
||||||
|
return ExtractCheckpoint{}, CheckpointDecision{Reason: "extract result unavailable"}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk) (RunOutput, error) {
|
||||||
|
output := RunOutput{Manifest: manifestFromPipeline(input)}
|
||||||
|
states := make([]*laneExtractState, len(input.Prepared.lanes))
|
||||||
|
for i, prepared := range input.Prepared.lanes {
|
||||||
|
if prepared.typed == nil {
|
||||||
|
return output, fmt.Errorf("typed lane %q executor is not prepared", prepared.resolved.ID)
|
||||||
|
}
|
||||||
|
setTypedLaneManifestMetadata(&output, prepared.resolved.ID, prepared.typed.extractor, prepared.typed.merger, prepared.typed.normalizer)
|
||||||
|
state, err := prepareLaneExtract(input, loader, doc, chunks, i, prepared)
|
||||||
|
if err != nil {
|
||||||
|
return output, err
|
||||||
|
}
|
||||||
|
if !state.decision.Reused {
|
||||||
|
if err := checkpoints.ExtractRunning(prepared.resolved.ID, prepared.resolved.Extract.Module, state.deps); err != nil {
|
||||||
|
return output, fmt.Errorf("write extract checkpoint for lane %q: %w", prepared.resolved.ID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
states[i] = state
|
||||||
|
}
|
||||||
|
|
||||||
|
workerCount := input.ExtractWorkers
|
||||||
|
if workerCount < 1 {
|
||||||
|
workerCount = 1
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(parent)
|
||||||
|
defer cancel()
|
||||||
|
jobs := make(chan extractJob, workerCount)
|
||||||
|
results := make(chan extractJobResult, workerCount)
|
||||||
|
completions := make(chan laneCompletion, len(states))
|
||||||
|
continuations := make(chan *laneExtractState, workerCount)
|
||||||
|
|
||||||
|
var workers sync.WaitGroup
|
||||||
|
for i := 0; i < workerCount; i++ {
|
||||||
|
workers.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer workers.Done()
|
||||||
|
for job := range jobs {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
result := r.runExtractJob(ctx, input, doc, sourceInput, sessionID, job)
|
||||||
|
results <- result
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
go func() {
|
||||||
|
defer close(jobs)
|
||||||
|
for chunkIndex := range chunks {
|
||||||
|
for laneIndex := range states {
|
||||||
|
state := states[laneIndex]
|
||||||
|
if state.decision.Reused {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case jobs <- extractJob{lane: state, chunk: chunks[chunkIndex]}:
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
go func() { workers.Wait(); close(results) }()
|
||||||
|
var continuationWorkers sync.WaitGroup
|
||||||
|
for i := 0; i < workerCount; i++ {
|
||||||
|
continuationWorkers.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer continuationWorkers.Done()
|
||||||
|
for state := range continuations {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
completions <- laneCompletion{index: state.index, err: err}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
laneOutput, err := r.continueLane(ctx, input, checkpoints, loader, doc, sourceInput, sessionID, chunks, state)
|
||||||
|
completions <- laneCompletion{index: state.index, output: laneOutput, err: err}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
completedOutputs := make([]RunOutput, len(states))
|
||||||
|
var runErrors []orderedRunError
|
||||||
|
var pendingContinuations []*laneExtractState
|
||||||
|
launched, completed := 0, 0
|
||||||
|
for _, state := range states {
|
||||||
|
if state.decision.Reused {
|
||||||
|
pendingContinuations = append(pendingContinuations, state)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
resultChannel := results
|
||||||
|
for resultChannel != nil || len(pendingContinuations) > 0 || completed < launched {
|
||||||
|
var continuationChannel chan<- *laneExtractState
|
||||||
|
var nextContinuation *laneExtractState
|
||||||
|
if len(pendingContinuations) > 0 && ctx.Err() == nil {
|
||||||
|
continuationChannel = continuations
|
||||||
|
nextContinuation = pendingContinuations[0]
|
||||||
|
} else if ctx.Err() != nil {
|
||||||
|
pendingContinuations = nil
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case continuationChannel <- nextContinuation:
|
||||||
|
pendingContinuations = pendingContinuations[1:]
|
||||||
|
launched++
|
||||||
|
case result, ok := <-resultChannel:
|
||||||
|
if !ok {
|
||||||
|
resultChannel = nil
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
state := states[result.laneIndex]
|
||||||
|
state.remaining--
|
||||||
|
if result.err != nil {
|
||||||
|
state.failed = true
|
||||||
|
runErrors = append(runErrors, orderedRunError{stage: 0, lane: state.index, chunk: result.chunkIndex, err: result.err})
|
||||||
|
_ = checkpoints.ExtractFailed(state.prepared.resolved.ID, state.prepared.resolved.Extract.Module, state.deps, result.err)
|
||||||
|
cancel()
|
||||||
|
} else {
|
||||||
|
state.results[result.chunkIndex] = result
|
||||||
|
}
|
||||||
|
if state.remaining == 0 && !state.failed && ctx.Err() == nil {
|
||||||
|
if err := finalizeLaneExtract(checkpoints, state); err != nil {
|
||||||
|
state.failed = true
|
||||||
|
runErrors = append(runErrors, orderedRunError{stage: 0, lane: state.index, chunk: len(chunks), err: err})
|
||||||
|
cancel()
|
||||||
|
} else {
|
||||||
|
pendingContinuations = append(pendingContinuations, state)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case completion := <-completions:
|
||||||
|
completed++
|
||||||
|
completedOutputs[completion.index] = completion.output
|
||||||
|
if completion.err != nil {
|
||||||
|
runErrors = append(runErrors, classifyLaneError(completion.index, len(chunks), completion.err))
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
close(continuations)
|
||||||
|
continuationWorkers.Wait()
|
||||||
|
for i := range completedOutputs {
|
||||||
|
mergeLaneOutput(&output, completedOutputs[i])
|
||||||
|
}
|
||||||
|
if err := selectRunError(parent, runErrors); err != nil {
|
||||||
|
return output, err
|
||||||
|
}
|
||||||
|
return output, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func prepareLaneExtract(input RunInput, loader CheckpointLoader, doc *source.SourceDocument, chunks []source.Chunk, index int, prepared preparedLaneExecutor) (*laneExtractState, error) {
|
||||||
|
lane, typed := prepared.resolved, prepared.typed
|
||||||
|
digest, err := joinedChunkDigest(chunks)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("digest chunks for lane %q: %w", lane.ID, err)
|
||||||
|
}
|
||||||
|
state := &laneExtractState{index: index, prepared: prepared, deps: digestFingerprints("chunks", digest), remaining: len(chunks), results: make(map[int]extractJobResult, len(chunks))}
|
||||||
|
cp, decision := loadExtract(loader, lane.ID, lane.Extract.Module, state.deps)
|
||||||
|
if decision.Reused {
|
||||||
|
for _, stored := range cp.Outputs {
|
||||||
|
if _, decodeErr := decodeCheckpointArtifact(typed.codec, stored); decodeErr != nil {
|
||||||
|
decision = CheckpointDecision{Reason: "extract artifact checkpoint codec is incompatible: " + decodeErr.Error()}
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
state.decision = decision
|
||||||
|
if decision.Reused {
|
||||||
|
state.remaining = 0
|
||||||
|
for _, stored := range cp.Outputs {
|
||||||
|
value, decodeErr := decodeCheckpointArtifact(typed.codec, stored)
|
||||||
|
if decodeErr != nil {
|
||||||
|
return nil, fmt.Errorf("decode extract checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||||
|
}
|
||||||
|
stored = hydrateCheckpointArtifact(typed.codec, stored, value)
|
||||||
|
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: stored.ChunkID, ChunkIndex: stored.ChunkIndex, ChunkRef: stored.ChunkRef, Value: value}
|
||||||
|
if stored.ChunkIndex >= 0 && stored.ChunkIndex < len(chunks) && artifact.ChunkRef == (source.SourceRef{}) {
|
||||||
|
artifact.ChunkRef = chunks[stored.ChunkIndex].Ref
|
||||||
|
}
|
||||||
|
state.values = append(state.values, artifact)
|
||||||
|
state.serialized = append(state.serialized, cloneCheckpointArtifact(stored))
|
||||||
|
}
|
||||||
|
state.warnings, state.rejected = cloneWarnings(cp.Warnings), cloneRejectedOutputs(cp.Rejected)
|
||||||
|
}
|
||||||
|
return state, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, job extractJob) extractJobResult {
|
||||||
|
state, chunk := job.lane, job.chunk
|
||||||
|
lane, typed := state.prepared.resolved, state.prepared.typed
|
||||||
|
result := extractJobResult{laneIndex: state.index, chunkIndex: chunk.Index}
|
||||||
|
var accepted erasedExtractArtifact
|
||||||
|
var serialized CheckpointArtifact
|
||||||
|
var acceptedWarnings []contracts.Warning
|
||||||
|
ok, rejection, err := runWithRetry(ctx, lane.Extract.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
|
||||||
|
started := time.Now().UTC()
|
||||||
|
attemptPath := path.Join("extract", debugPathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), fmt.Sprintf("attempt-%02d", attempt))
|
||||||
|
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||||
|
extracted, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(lane.ExtractReferences.ReferenceSet), LLMProfile: lane.Extract.LLMProfile, Metadata: cloneMetadata(input.Metadata)})
|
||||||
|
if callErr != nil {
|
||||||
|
_ = writeDebugTimed(input.Debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started, Error: callErr.Error()}, llmScope))
|
||||||
|
return false, nil, fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, lane.Extract.Module, callErr)
|
||||||
|
}
|
||||||
|
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: chunk.ID, ChunkIndex: chunk.Index, ChunkRef: chunk.Ref, Value: extracted.Value}
|
||||||
|
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageExtract, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: lane.ExtractReferences.ReferenceSet, metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: extracted.Value}, state.prepared.extractValidators, attempt, input.Debug)
|
||||||
|
if validateErr != nil || rejected != nil {
|
||||||
|
return false, rejected, validateErr
|
||||||
|
}
|
||||||
|
stored, encodeErr := checkpointArtifact(typed.codec, artifact.LaneID, artifact.ExtractorKey, artifact.SourceID, artifact.Value)
|
||||||
|
if encodeErr != nil {
|
||||||
|
return false, nil, encodeErr
|
||||||
|
}
|
||||||
|
stored.ChunkID, stored.ChunkIndex, stored.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
|
||||||
|
accepted, serialized = artifact, stored
|
||||||
|
acceptedWarnings = append(cloneWarnings(extracted.Warnings), warnings...)
|
||||||
|
if debugErr := writeDebugTimed(input.Debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started, Payload: map[string]any{"output": debugCheckpointArtifact(stored), "warnings": debugWarningEnvelopes(acceptedWarnings)}}, llmScope)); debugErr != nil {
|
||||||
|
return false, nil, debugErr
|
||||||
|
}
|
||||||
|
return true, nil, nil
|
||||||
|
})
|
||||||
|
result.err = err
|
||||||
|
if err == nil && !ok {
|
||||||
|
result.rejected = rejection
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
result.value, result.serialized, result.warnings = accepted, serialized, acceptedWarnings
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
func finalizeLaneExtract(checkpoints CheckpointRecorder, state *laneExtractState) error {
|
||||||
|
lane := state.prepared.resolved
|
||||||
|
indexes := make([]int, 0, len(state.results))
|
||||||
|
for index := range state.results {
|
||||||
|
indexes = append(indexes, index)
|
||||||
|
}
|
||||||
|
sort.Ints(indexes)
|
||||||
|
for _, index := range indexes {
|
||||||
|
result := state.results[index]
|
||||||
|
if result.rejected != nil {
|
||||||
|
state.rejected = append(state.rejected, *result.rejected)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
state.values = append(state.values, result.value)
|
||||||
|
state.serialized = append(state.serialized, result.serialized)
|
||||||
|
state.warnings = append(state.warnings, result.warnings...)
|
||||||
|
}
|
||||||
|
sort.SliceStable(state.values, func(i, j int) bool { return state.values[i].ChunkIndex < state.values[j].ChunkIndex })
|
||||||
|
sort.SliceStable(state.serialized, func(i, j int) bool { return state.serialized[i].ChunkIndex < state.serialized[j].ChunkIndex })
|
||||||
|
sort.SliceStable(state.rejected, func(i, j int) bool { return state.rejected[i].ChunkIndex < state.rejected[j].ChunkIndex })
|
||||||
|
if !state.decision.Reused {
|
||||||
|
if err := recordExtract(checkpoints, lane.ID, lane.Extract.Module, state.deps, state.serialized, state.rejected, state.warnings); err != nil {
|
||||||
|
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Runner) continueLane(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk, state *laneExtractState) (RunOutput, error) {
|
||||||
|
lane := state.prepared.resolved
|
||||||
|
local := RunOutput{Manifest: manifestFromPipeline(input)}
|
||||||
|
checkpoint := ExtractCheckpoint{Outputs: state.serialized, Rejected: state.rejected, Warnings: state.warnings}
|
||||||
|
coordinatedLoader := completedExtractLoader{CheckpointLoader: loader, laneID: lane.ID, checkpoint: checkpoint}
|
||||||
|
input.extractDecision = &state.decision
|
||||||
|
err := r.runTypedLane(ctx, input, checkpoints, coordinatedLoader, doc, sourceInput, sessionID, chunks, state.prepared, &local)
|
||||||
|
return local, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func classifyLaneError(lane, sentinel int, err error) orderedRunError {
|
||||||
|
stage := 1
|
||||||
|
var laneErr *laneRunError
|
||||||
|
if errors.As(err, &laneErr) {
|
||||||
|
switch laneErr.stage {
|
||||||
|
case StageExtract:
|
||||||
|
stage = 0
|
||||||
|
case StageNormalize:
|
||||||
|
stage = 2
|
||||||
|
}
|
||||||
|
} else if strings.Contains(strings.ToLower(err.Error()), "normalize") {
|
||||||
|
stage = 2
|
||||||
|
}
|
||||||
|
return orderedRunError{stage: stage, lane: lane, chunk: sentinel, err: err}
|
||||||
|
}
|
||||||
|
|
||||||
|
func selectRunError(parent context.Context, values []orderedRunError) error {
|
||||||
|
if err := parent.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if len(values) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
hasReal := false
|
||||||
|
for _, value := range values {
|
||||||
|
if !errors.Is(value.err, context.Canceled) && !errors.Is(value.err, context.DeadlineExceeded) {
|
||||||
|
hasReal = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
filtered := values[:0]
|
||||||
|
for _, value := range values {
|
||||||
|
if hasReal && (errors.Is(value.err, context.Canceled) || errors.Is(value.err, context.DeadlineExceeded)) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
filtered = append(filtered, value)
|
||||||
|
}
|
||||||
|
sort.SliceStable(filtered, func(i, j int) bool {
|
||||||
|
if filtered[i].stage != filtered[j].stage {
|
||||||
|
return filtered[i].stage < filtered[j].stage
|
||||||
|
}
|
||||||
|
if filtered[i].lane != filtered[j].lane {
|
||||||
|
return filtered[i].lane < filtered[j].lane
|
||||||
|
}
|
||||||
|
return filtered[i].chunk < filtered[j].chunk
|
||||||
|
})
|
||||||
|
return filtered[0].err
|
||||||
|
}
|
||||||
|
|
||||||
|
func mergeLaneOutput(dst *RunOutput, src RunOutput) {
|
||||||
|
if dst == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
dst.NormalizeOutputs = append(dst.NormalizeOutputs, cloneSerializedOutputs(src.NormalizeOutputs)...)
|
||||||
|
dst.Rejected = append(dst.Rejected, cloneRejectedOutputs(src.Rejected)...)
|
||||||
|
dst.Warnings = append(dst.Warnings, cloneWarnings(src.Warnings)...)
|
||||||
|
dst.CheckpointEvents = append(dst.CheckpointEvents, src.CheckpointEvents...)
|
||||||
|
for i := range dst.Manifest.ArtifactLanes {
|
||||||
|
for j := range src.Manifest.ArtifactLanes {
|
||||||
|
if dst.Manifest.ArtifactLanes[i].ID == src.Manifest.ArtifactLanes[j].ID && src.Manifest.ArtifactLanes[j].Metadata != nil {
|
||||||
|
dst.Manifest.ArtifactLanes[i].Metadata = cloneMetadata(src.Manifest.ArtifactLanes[j].Metadata)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
463
internal/framework/pipeline/runner_typed.go
Normal file
463
internal/framework/pipeline/runner_typed.go
Normal file
@@ -0,0 +1,463 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"fmt"
|
||||||
|
"path"
|
||||||
|
"sort"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
func loadExtract(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||||
|
return loader.Extract(laneID, moduleKey, deps)
|
||||||
|
}
|
||||||
|
func loadMerge(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
|
||||||
|
return loader.Merge(laneID, moduleKey, deps)
|
||||||
|
}
|
||||||
|
func loadNormalize(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
|
||||||
|
return loader.Normalize(laneID, moduleKey, deps)
|
||||||
|
}
|
||||||
|
func recordExtract(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||||
|
return recorder.ExtractSucceeded(laneID, moduleKey, deps, outputs, rejected, warnings)
|
||||||
|
}
|
||||||
|
func recordMerge(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||||
|
return recorder.MergeSucceeded(laneID, moduleKey, deps, output, warnings)
|
||||||
|
}
|
||||||
|
func recordNormalize(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||||
|
return recorder.NormalizeSucceeded(laneID, moduleKey, deps, output, warnings)
|
||||||
|
}
|
||||||
|
func cloneCheckpointArtifact(output CheckpointArtifact) CheckpointArtifact {
|
||||||
|
output.Artifact = contracts.CloneSerializedArtifact(output.Artifact)
|
||||||
|
return output
|
||||||
|
}
|
||||||
|
|
||||||
|
func hydrateCheckpointArtifact(codec artifactCodecEntry, output CheckpointArtifact, value any) CheckpointArtifact {
|
||||||
|
output.Artifact.Schema = contracts.CloneArtifactSchema(codec.spec.Schema)
|
||||||
|
if codec.metadata != nil {
|
||||||
|
output.Artifact.Metadata = cloneMetadata(codec.metadata(value))
|
||||||
|
} else {
|
||||||
|
output.Artifact.Metadata = nil
|
||||||
|
}
|
||||||
|
return output
|
||||||
|
}
|
||||||
|
func artifactCheckpointDigests(outputs []CheckpointArtifact) []CheckpointFingerprint {
|
||||||
|
values := make([]CheckpointFingerprint, 0, len(outputs))
|
||||||
|
for i, output := range outputs {
|
||||||
|
sum := sha256.Sum256(output.Artifact.Content)
|
||||||
|
values = append(values, CheckpointFingerprint{Name: fmt.Sprintf("artifact[%d]", i), Value: "sha256:" + hex.EncodeToString(sum[:])})
|
||||||
|
}
|
||||||
|
return normalizeCheckpointFingerprints(values)
|
||||||
|
}
|
||||||
|
func debugCheckpointArtifact(output CheckpointArtifact) map[string]any {
|
||||||
|
artifact := output.Artifact
|
||||||
|
schema := contracts.CloneArtifactSchema(artifact.Schema)
|
||||||
|
digest := output.SchemaDigest
|
||||||
|
if digest == "" {
|
||||||
|
digest = contracts.DigestArtifactSchema(schema)
|
||||||
|
}
|
||||||
|
schema.JSONSchema = nil
|
||||||
|
content := debugContentEnvelope(artifact.Content, artifact.MediaType, artifact.Metadata, nil)
|
||||||
|
content.ContentDigest = debugContentDigest(artifact.Content)
|
||||||
|
return map[string]any{"lane_id": output.LaneID, "module_key": output.ModuleKey, "source_id": output.SourceID, "chunk_id": output.ChunkID, "chunk_index": output.ChunkIndex, "chunk_ref": output.ChunkRef, "artifact_kind": artifact.Kind, "schema": schema, "schema_digest": digest, "content": content}
|
||||||
|
}
|
||||||
|
func debugCheckpointArtifacts(outputs []CheckpointArtifact) []map[string]any {
|
||||||
|
if len(outputs) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]map[string]any, 0, len(outputs))
|
||||||
|
for _, output := range outputs {
|
||||||
|
out = append(out, debugCheckpointArtifact(output))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func serializeArtifact(codec artifactCodecEntry, value any, candidate bool) (contracts.SerializedArtifact, error) {
|
||||||
|
encode := codec.encode
|
||||||
|
if candidate {
|
||||||
|
encode = codec.encodeCandidate
|
||||||
|
}
|
||||||
|
content, err := encode(value)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.SerializedArtifact{}, err
|
||||||
|
}
|
||||||
|
schema := codec.spec.Schema
|
||||||
|
metadata := map[string]any(nil)
|
||||||
|
if codec.metadata != nil {
|
||||||
|
metadata = codec.metadata(value)
|
||||||
|
}
|
||||||
|
return contracts.SerializedArtifact{Kind: codec.spec.Kind, Schema: contracts.CloneArtifactSchema(schema), MediaType: codec.spec.MediaType, Content: append([]byte(nil), content...), Metadata: cloneMetadata(metadata)}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeCheckpointArtifact(codec artifactCodecEntry, artifact CheckpointArtifact) (any, error) {
|
||||||
|
expectedDigest := contracts.DigestArtifactSchema(codec.spec.Schema)
|
||||||
|
if artifact.Artifact.Kind != codec.spec.Kind {
|
||||||
|
return nil, fmt.Errorf("artifact kind %q does not match codec %q", artifact.Artifact.Kind, codec.spec.Kind)
|
||||||
|
}
|
||||||
|
if artifact.Artifact.Schema.ID != codec.spec.Schema.ID || artifact.Artifact.Schema.Version != codec.spec.Schema.Version {
|
||||||
|
return nil, fmt.Errorf("artifact schema %q version %q does not match codec schema %q version %q", artifact.Artifact.Schema.ID, artifact.Artifact.Schema.Version, codec.spec.Schema.ID, codec.spec.Schema.Version)
|
||||||
|
}
|
||||||
|
if artifact.SchemaDigest != expectedDigest {
|
||||||
|
return nil, fmt.Errorf("artifact schema digest %q does not match codec schema digest %q", artifact.SchemaDigest, expectedDigest)
|
||||||
|
}
|
||||||
|
if artifact.Artifact.MediaType != codec.spec.MediaType {
|
||||||
|
return nil, fmt.Errorf("artifact media type %q does not match codec media type %q", artifact.Artifact.MediaType, codec.spec.MediaType)
|
||||||
|
}
|
||||||
|
return codec.decode(append([]byte(nil), artifact.Artifact.Content...))
|
||||||
|
}
|
||||||
|
|
||||||
|
func checkpointArtifact(codec artifactCodecEntry, laneID, moduleKey, sourceID string, value any) (CheckpointArtifact, error) {
|
||||||
|
serialized, err := serializeArtifact(codec, value, false)
|
||||||
|
if err != nil {
|
||||||
|
return CheckpointArtifact{}, err
|
||||||
|
}
|
||||||
|
return CheckpointArtifact{LaneID: laneID, ModuleKey: moduleKey, SourceID: sourceID, Artifact: serialized, SchemaDigest: contracts.DigestArtifactSchema(serialized.Schema)}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type laneRunError struct {
|
||||||
|
stage ModuleStage
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *laneRunError) Error() string { return e.err.Error() }
|
||||||
|
func (e *laneRunError) Unwrap() error { return e.err }
|
||||||
|
|
||||||
|
func (r *Runner) runTypedLane(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk, prepared preparedLaneExecutor, output *RunOutput) (err error) {
|
||||||
|
activeStage := StageExtract
|
||||||
|
defer func() {
|
||||||
|
if err != nil {
|
||||||
|
err = &laneRunError{stage: activeStage, err: err}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
lane, typed := prepared.resolved, prepared.typed
|
||||||
|
if typed == nil {
|
||||||
|
return fmt.Errorf("typed lane %q executor is not prepared", lane.ID)
|
||||||
|
}
|
||||||
|
setTypedLaneManifestMetadata(output, lane.ID, typed.extractor, typed.merger, typed.normalizer)
|
||||||
|
|
||||||
|
values := make([]erasedExtractArtifact, 0, len(chunks))
|
||||||
|
serializedExtracts := make([]CheckpointArtifact, 0, len(chunks))
|
||||||
|
extractWarnings := []contracts.Warning{}
|
||||||
|
rejectedStart := len(output.Rejected)
|
||||||
|
chunksDigest, err := joinedChunkDigest(chunks)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("digest chunks for lane %q: %w", lane.ID, err)
|
||||||
|
}
|
||||||
|
extractDeps := digestFingerprints("chunks", chunksDigest)
|
||||||
|
cp, decision := loadExtract(loader, lane.ID, lane.Extract.Module, extractDeps)
|
||||||
|
if decision.Reused {
|
||||||
|
for _, stored := range cp.Outputs {
|
||||||
|
if _, decodeErr := decodeCheckpointArtifact(typed.codec, stored); decodeErr != nil {
|
||||||
|
decision = CheckpointDecision{Reason: "extract artifact checkpoint codec is incompatible: " + decodeErr.Error()}
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
reportedDecision := decision
|
||||||
|
if input.extractDecision != nil {
|
||||||
|
reportedDecision = *input.extractDecision
|
||||||
|
}
|
||||||
|
recordCheckpointEvent(output, loader, string(StageExtract), lane.ID, lane.Extract.Module, reportedDecision)
|
||||||
|
if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": reportedDecision.Reused, "decision": reportedDecision, "source": debugSourceDocumentEnvelope(doc), "chunks": debugSourceChunkEnvelopes(chunks), "options": redactSensitiveMap(lane.Extract.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if decision.Reused {
|
||||||
|
for _, stored := range cp.Outputs {
|
||||||
|
value, decodeErr := decodeCheckpointArtifact(typed.codec, stored)
|
||||||
|
if decodeErr != nil {
|
||||||
|
return fmt.Errorf("decode extract checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||||
|
}
|
||||||
|
stored = hydrateCheckpointArtifact(typed.codec, stored, value)
|
||||||
|
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: stored.ChunkID, ChunkIndex: stored.ChunkIndex, ChunkRef: stored.ChunkRef, Value: value}
|
||||||
|
if stored.ChunkIndex >= 0 && stored.ChunkIndex < len(chunks) && artifact.ChunkRef == (source.SourceRef{}) {
|
||||||
|
artifact.ChunkRef = chunks[stored.ChunkIndex].Ref
|
||||||
|
}
|
||||||
|
values = append(values, artifact)
|
||||||
|
serializedExtracts = append(serializedExtracts, cloneCheckpointArtifact(stored))
|
||||||
|
}
|
||||||
|
extractWarnings = cloneWarnings(cp.Warnings)
|
||||||
|
output.Warnings = append(output.Warnings, extractWarnings...)
|
||||||
|
output.Rejected = append(output.Rejected, cloneRejectedOutputs(cp.Rejected)...)
|
||||||
|
} else {
|
||||||
|
if err := checkpoints.ExtractRunning(lane.ID, lane.Extract.Module, extractDeps); err != nil {
|
||||||
|
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
|
||||||
|
}
|
||||||
|
for i := range chunks {
|
||||||
|
chunk := chunks[i]
|
||||||
|
var accepted erasedExtractArtifact
|
||||||
|
var serializedAccepted CheckpointArtifact
|
||||||
|
var acceptedWarnings []contracts.Warning
|
||||||
|
ok, rejection, runErr := runWithRetry(ctx, lane.Extract.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
|
||||||
|
started := time.Now().UTC()
|
||||||
|
attemptPath := path.Join("extract", debugPathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), fmt.Sprintf("attempt-%02d", attempt))
|
||||||
|
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||||
|
result, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(lane.ExtractReferences.ReferenceSet), LLMProfile: lane.Extract.LLMProfile, Metadata: cloneMetadata(input.Metadata)})
|
||||||
|
if callErr != nil {
|
||||||
|
_ = writeDebugTimed(input.Debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started, Error: callErr.Error()}, llmScope))
|
||||||
|
return false, nil, fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, lane.Extract.Module, callErr)
|
||||||
|
}
|
||||||
|
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: chunk.ID, ChunkIndex: chunk.Index, ChunkRef: chunk.Ref, Value: result.Value}
|
||||||
|
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageExtract, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: lane.ExtractReferences.ReferenceSet, metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: result.Value}, prepared.extractValidators, attempt, input.Debug)
|
||||||
|
if validateErr != nil || rejected != nil {
|
||||||
|
return false, rejected, validateErr
|
||||||
|
}
|
||||||
|
stored, encodeErr := checkpointArtifact(typed.codec, artifact.LaneID, artifact.ExtractorKey, artifact.SourceID, artifact.Value)
|
||||||
|
if encodeErr != nil {
|
||||||
|
return false, nil, encodeErr
|
||||||
|
}
|
||||||
|
stored.ChunkID, stored.ChunkIndex, stored.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
|
||||||
|
accepted, serializedAccepted = artifact, stored
|
||||||
|
acceptedWarnings = append(cloneWarnings(result.Warnings), warnings...)
|
||||||
|
if debugErr := writeDebugTimed(input.Debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started, Payload: map[string]any{"output": debugCheckpointArtifact(stored), "warnings": debugWarningEnvelopes(acceptedWarnings)}}, llmScope)); debugErr != nil {
|
||||||
|
return false, nil, debugErr
|
||||||
|
}
|
||||||
|
return true, nil, nil
|
||||||
|
})
|
||||||
|
if runErr != nil {
|
||||||
|
_ = checkpoints.ExtractFailed(lane.ID, lane.Extract.Module, extractDeps, runErr)
|
||||||
|
return runErr
|
||||||
|
}
|
||||||
|
if !ok {
|
||||||
|
output.Rejected = append(output.Rejected, *rejection)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
values = append(values, accepted)
|
||||||
|
serializedExtracts = append(serializedExtracts, serializedAccepted)
|
||||||
|
extractWarnings = append(extractWarnings, acceptedWarnings...)
|
||||||
|
output.Warnings = append(output.Warnings, acceptedWarnings...)
|
||||||
|
}
|
||||||
|
if err := recordExtract(checkpoints, lane.ID, lane.Extract.Module, extractDeps, serializedExtracts, cloneRejectedOutputs(output.Rejected[rejectedStart:]), extractWarnings); err != nil {
|
||||||
|
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.SliceStable(values, func(i, j int) bool { return values[i].ChunkIndex < values[j].ChunkIndex })
|
||||||
|
sort.SliceStable(serializedExtracts, func(i, j int) bool { return serializedExtracts[i].ChunkIndex < serializedExtracts[j].ChunkIndex })
|
||||||
|
if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": reportedDecision.Reused, "outputs": debugCheckpointArtifacts(serializedExtracts), "rejected": debugRejectedOutputEnvelopes(output.Rejected[rejectedStart:]), "warnings": debugWarningEnvelopes(extractWarnings)}}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if len(values) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
activeStage = StageMerge
|
||||||
|
|
||||||
|
mergeInputs := make([]contracts.ExtractArtifact[any], len(values))
|
||||||
|
for i, value := range values {
|
||||||
|
mergeInputs[i] = contracts.ExtractArtifact[any]{LaneID: value.LaneID, ExtractorKey: value.ExtractorKey, SourceID: value.SourceID, ChunkID: value.ChunkID, ChunkIndex: value.ChunkIndex, ChunkRef: value.ChunkRef, Value: value.Value}
|
||||||
|
}
|
||||||
|
mergeDeps := artifactCheckpointDigests(serializedExtracts)
|
||||||
|
mergeCP, mergeDecision := loadMerge(loader, lane.ID, lane.Merge.Module, mergeDeps)
|
||||||
|
if mergeDecision.Reused {
|
||||||
|
if _, decodeErr := decodeCheckpointArtifact(typed.codec, mergeCP.Output); decodeErr != nil {
|
||||||
|
mergeDecision = CheckpointDecision{Reason: "merge artifact checkpoint codec is incompatible: " + decodeErr.Error()}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
recordCheckpointEvent(output, loader, string(StageMerge), lane.ID, lane.Merge.Module, mergeDecision)
|
||||||
|
if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageMerge), LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "decision": mergeDecision, "source": debugSourceDocumentEnvelope(doc), "extract_outputs": debugCheckpointArtifacts(serializedExtracts), "options": redactSensitiveMap(lane.Merge.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
var merged erasedMergeArtifact
|
||||||
|
var serializedMerge CheckpointArtifact
|
||||||
|
var mergeWarnings []contracts.Warning
|
||||||
|
if mergeDecision.Reused {
|
||||||
|
value, decodeErr := decodeCheckpointArtifact(typed.codec, mergeCP.Output)
|
||||||
|
if decodeErr != nil {
|
||||||
|
return fmt.Errorf("decode merge checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||||
|
}
|
||||||
|
merged = erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: value}
|
||||||
|
serializedMerge = hydrateCheckpointArtifact(typed.codec, cloneCheckpointArtifact(mergeCP.Output), value)
|
||||||
|
mergeWarnings = cloneWarnings(mergeCP.Warnings)
|
||||||
|
output.Warnings = append(output.Warnings, mergeWarnings...)
|
||||||
|
} else {
|
||||||
|
if err := checkpoints.MergeRunning(lane.ID, lane.Merge.Module, mergeDeps); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
ok, rejection, runErr := runWithRetry(ctx, lane.Merge.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
|
||||||
|
result, callErr := typed.merge(ctx, typed.merger, contracts.TypedMergeRequest[any]{Source: doc, LaneID: lane.ID, ExtractOutputs: mergeInputs, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(lane.MergeReferences.ReferenceSet), LLMProfile: lane.Merge.LLMProfile, Metadata: cloneMetadata(input.Metadata)})
|
||||||
|
if callErr != nil {
|
||||||
|
return false, nil, fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, lane.Merge.Module, callErr)
|
||||||
|
}
|
||||||
|
candidate := erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: result.Value}
|
||||||
|
warnings, rejected, validateErr := r.validateTypedArtifact(ctx, typed.codec, typedValidationTarget{stage: StageMerge, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: lane.MergeReferences.ReferenceSet, metadata: input.Metadata, value: result.Value}, prepared.mergeValidators, attempt, input.Debug)
|
||||||
|
if validateErr != nil || rejected != nil {
|
||||||
|
return false, rejected, validateErr
|
||||||
|
}
|
||||||
|
stored, encodeErr := checkpointArtifact(typed.codec, candidate.LaneID, candidate.MergerKey, candidate.SourceID, candidate.Value)
|
||||||
|
if encodeErr != nil {
|
||||||
|
return false, nil, encodeErr
|
||||||
|
}
|
||||||
|
merged, serializedMerge = candidate, stored
|
||||||
|
mergeWarnings = append(cloneWarnings(result.Warnings), warnings...)
|
||||||
|
return true, nil, nil
|
||||||
|
})
|
||||||
|
if runErr != nil {
|
||||||
|
_ = checkpoints.MergeFailed(lane.ID, lane.Merge.Module, mergeDeps, runErr)
|
||||||
|
return runErr
|
||||||
|
}
|
||||||
|
if !ok {
|
||||||
|
output.Rejected = append(output.Rejected, *rejection)
|
||||||
|
if err := checkpoints.MergeRejected(lane.ID, lane.Merge.Module, mergeDeps, *rejection); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
output.Warnings = append(output.Warnings, mergeWarnings...)
|
||||||
|
if err := recordMerge(checkpoints, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge, mergeWarnings); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageMerge), LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedMerge), "warnings": debugWarningEnvelopes(mergeWarnings)}}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
activeStage = StageNormalize
|
||||||
|
normalizeDeps := artifactCheckpointDigests([]CheckpointArtifact{serializedMerge})
|
||||||
|
normalizeCP, normalizeDecision := loadNormalize(loader, lane.ID, lane.Normalize.Module, normalizeDeps)
|
||||||
|
if normalizeDecision.Reused {
|
||||||
|
if _, decodeErr := decodeCheckpointArtifact(typed.codec, normalizeCP.Output); decodeErr != nil {
|
||||||
|
normalizeDecision = CheckpointDecision{Reason: "normalize artifact checkpoint codec is incompatible: " + decodeErr.Error()}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
recordCheckpointEvent(output, loader, string(StageNormalize), lane.ID, lane.Normalize.Module, normalizeDecision)
|
||||||
|
if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageNormalize), LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "decision": normalizeDecision, "source": debugSourceDocumentEnvelope(doc), "merge_output": debugCheckpointArtifact(serializedMerge), "options": redactSensitiveMap(lane.Normalize.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
var serializedNormalize CheckpointArtifact
|
||||||
|
var normalizeWarnings []contracts.Warning
|
||||||
|
if normalizeDecision.Reused {
|
||||||
|
value, decodeErr := decodeCheckpointArtifact(typed.codec, normalizeCP.Output)
|
||||||
|
if decodeErr != nil {
|
||||||
|
return fmt.Errorf("decode normalize checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||||
|
}
|
||||||
|
serializedNormalize, normalizeWarnings = hydrateCheckpointArtifact(typed.codec, cloneCheckpointArtifact(normalizeCP.Output), value), cloneWarnings(normalizeCP.Warnings)
|
||||||
|
output.Warnings = append(output.Warnings, normalizeWarnings...)
|
||||||
|
} else {
|
||||||
|
if err := checkpoints.NormalizeRunning(lane.ID, lane.Normalize.Module, normalizeDeps); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
ok, rejection, runErr := runWithRetry(ctx, lane.Normalize.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
|
||||||
|
result, callErr := typed.normalize(ctx, typed.normalizer, contracts.TypedNormalizeRequest[any]{Source: doc, LaneID: lane.ID, MergeOutput: contracts.MergeArtifact[any]{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: merged.Value}, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(lane.NormalizeReferences.ReferenceSet), LLMProfile: lane.Normalize.LLMProfile, Metadata: cloneMetadata(input.Metadata)})
|
||||||
|
if callErr != nil {
|
||||||
|
return false, nil, fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, lane.Normalize.Module, callErr)
|
||||||
|
}
|
||||||
|
warnings, rejected, validateErr := r.validateTypedArtifact(ctx, typed.codec, typedValidationTarget{stage: StageNormalize, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: lane.NormalizeReferences.ReferenceSet, metadata: input.Metadata, value: result.Value}, prepared.normalizeValidators, attempt, input.Debug)
|
||||||
|
if validateErr != nil || rejected != nil {
|
||||||
|
return false, rejected, validateErr
|
||||||
|
}
|
||||||
|
stored, encodeErr := checkpointArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, result.Value)
|
||||||
|
if encodeErr != nil {
|
||||||
|
return false, nil, encodeErr
|
||||||
|
}
|
||||||
|
serializedNormalize = stored
|
||||||
|
normalizeWarnings = append(cloneWarnings(result.Warnings), warnings...)
|
||||||
|
return true, nil, nil
|
||||||
|
})
|
||||||
|
if runErr != nil {
|
||||||
|
_ = checkpoints.NormalizeFailed(lane.ID, lane.Normalize.Module, normalizeDeps, runErr)
|
||||||
|
return runErr
|
||||||
|
}
|
||||||
|
if !ok {
|
||||||
|
output.Rejected = append(output.Rejected, *rejection)
|
||||||
|
if err := checkpoints.NormalizeRejected(lane.ID, lane.Normalize.Module, normalizeDeps, *rejection); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
output.Warnings = append(output.Warnings, normalizeWarnings...)
|
||||||
|
if err := recordNormalize(checkpoints, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize, normalizeWarnings); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageNormalize), LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedNormalize), "warnings": debugWarningEnvelopes(normalizeWarnings)}}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
output.NormalizeOutputs = append(output.NormalizeOutputs, contracts.SerializedOutput{LaneID: lane.ID, NormalizerKey: lane.Normalize.Module, SourceID: doc.ID, Artifact: contracts.CloneSerializedArtifact(serializedNormalize.Artifact)})
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func setTypedLaneManifestMetadata(output *RunOutput, laneID string, extractor, merger, normalizer any) {
|
||||||
|
if output == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for i := range output.Manifest.ArtifactLanes {
|
||||||
|
if output.Manifest.ArtifactLanes[i].ID != laneID {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
metadata := make(map[string]any)
|
||||||
|
for _, item := range []struct {
|
||||||
|
name string
|
||||||
|
module any
|
||||||
|
}{{"extractor", extractor}, {"merger", merger}, {"normalizer", normalizer}} {
|
||||||
|
if value, ok := moduleManifestMetadata(item.module); ok {
|
||||||
|
metadata[item.name] = value
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(metadata) > 0 {
|
||||||
|
output.Manifest.ArtifactLanes[i].Metadata = metadata
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Runner) validateTypedArtifact(ctx context.Context, codec artifactCodecEntry, target typedValidationTarget, chain preparedValidatorChain, attempt int, debug DebugRecorder) ([]contracts.Warning, *contracts.RejectedOutput, error) {
|
||||||
|
var warnings []contracts.Warning
|
||||||
|
for index, item := range chain.validators {
|
||||||
|
binding := item.resolved.Binding
|
||||||
|
var result contracts.ValidationResult
|
||||||
|
var err error
|
||||||
|
started := time.Now().UTC()
|
||||||
|
attemptPath := path.Join("validate", debugPathComponent(string(target.stage)), debugPathComponent(target.laneID), debugPathComponent(target.moduleKey), fmt.Sprintf("%02d-%s-attempt-%02d", index+1, debugPathComponent(binding.Module), attempt))
|
||||||
|
validatorCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||||
|
switch item.resolved.Target {
|
||||||
|
case ValidatorTargetTyped:
|
||||||
|
target.llmProfile = binding.LLMProfile
|
||||||
|
result, err = item.typedValidate(validatorCtx, item.typed, target)
|
||||||
|
case ValidatorTargetSerialized:
|
||||||
|
artifact, encodeErr := serializeArtifact(codec, target.value, true)
|
||||||
|
if encodeErr != nil {
|
||||||
|
err = encodeErr
|
||||||
|
break
|
||||||
|
}
|
||||||
|
result, err = item.serialized.Validate(validatorCtx, contracts.SerializedValidationRequest{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Source: target.source, SourceID: target.sourceID, SourceInput: target.sourceInput.Clone(), SessionID: target.sessionID, References: CloneReferenceSet(target.references), LLMProfile: binding.LLMProfile, Metadata: cloneMetadata(target.metadata), Chunk: cloneSourceChunkPtr(target.chunk), Chunks: cloneSourceChunks(target.chunks), Schema: contracts.CloneArtifactSchema(artifact.Schema), MediaType: artifact.MediaType, Content: append([]byte(nil), artifact.Content...)})
|
||||||
|
default:
|
||||||
|
return nil, nil, fmt.Errorf("validator %q is incompatible with typed artifact validation", binding.Module)
|
||||||
|
}
|
||||||
|
artifact, _ := serializeArtifact(codec, target.value, true)
|
||||||
|
debugCall := debugValidationCall{ValidatorName: binding.Module, Request: map[string]any{"stage": string(target.stage), "lane_id": target.laneID, "module_key": target.moduleKey, "source_id": target.sourceID, "artifact": debugCheckpointArtifact(CheckpointArtifact{Artifact: artifact, SchemaDigest: contracts.DigestArtifactSchema(artifact.Schema)}), "metadata": redactSensitiveMap(target.metadata)}, Result: debugValidationResultEnvelope(result)}
|
||||||
|
if err != nil {
|
||||||
|
debugCall.Error = err.Error()
|
||||||
|
}
|
||||||
|
if debugErr := writeDebugTimed(debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall, Error: debugCall.Error}, llmScope)); debugErr != nil {
|
||||||
|
return nil, nil, fmt.Errorf("write validation debug artifact: %w", debugErr)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, fmt.Errorf("validate typed %s output with validator %q: %w", target.stage, binding.Module, err)
|
||||||
|
}
|
||||||
|
if !result.Approved {
|
||||||
|
reason := result.ReasonCode
|
||||||
|
if reason == "" {
|
||||||
|
reason = "artifact_rejected"
|
||||||
|
}
|
||||||
|
message := result.Message
|
||||||
|
if message == "" {
|
||||||
|
message = "artifact rejected"
|
||||||
|
}
|
||||||
|
return nil, &contracts.RejectedOutput{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, ChunkID: func() string {
|
||||||
|
if target.chunk != nil {
|
||||||
|
return target.chunk.ID
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}(), ChunkIndex: func() int {
|
||||||
|
if target.chunk != nil {
|
||||||
|
return target.chunk.Index
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}(), ValidatorName: binding.Module, ReasonCode: reason, Message: message, AttemptCount: attempt, DiagnosticArtifactPath: result.DiagnosticArtifactPath}, nil
|
||||||
|
}
|
||||||
|
warnings = append(warnings, result.Warnings...)
|
||||||
|
}
|
||||||
|
return warnings, nil, nil
|
||||||
|
}
|
||||||
45
internal/framework/pipeline/runner_typed_checkpoint_test.go
Normal file
45
internal/framework/pipeline/runner_typed_checkpoint_test.go
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestDecodeCheckpointArtifactRejectsIncompatibleCodecIdentityAndBytes(t *testing.T) {
|
||||||
|
registry := NewArtifactCodecRegistry()
|
||||||
|
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
|
||||||
|
t.Fatalf("RegisterArtifactCodec: %v", err)
|
||||||
|
}
|
||||||
|
codec, _, err := registry.entry("test/notes")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("entry: %v", err)
|
||||||
|
}
|
||||||
|
artifact, err := serializeArtifact(codec, codecNotes{Items: []string{"one"}}, false)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("serializeArtifact: %v", err)
|
||||||
|
}
|
||||||
|
base := CheckpointArtifact{Artifact: artifact, SchemaDigest: contracts.DigestArtifactSchema(artifact.Schema)}
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*CheckpointArtifact)
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "missing kind", mutate: func(v *CheckpointArtifact) { v.Artifact.Kind = "" }, want: "artifact kind"},
|
||||||
|
{name: "schema version", mutate: func(v *CheckpointArtifact) { v.Artifact.Schema.Version = "v999" }, want: "does not match codec schema"},
|
||||||
|
{name: "schema digest", mutate: func(v *CheckpointArtifact) { v.SchemaDigest = "sha256:different" }, want: "schema digest"},
|
||||||
|
{name: "media type", mutate: func(v *CheckpointArtifact) { v.Artifact.MediaType = "text/plain" }, want: "media type"},
|
||||||
|
{name: "decode failure", mutate: func(v *CheckpointArtifact) { v.Artifact.Content = []byte(`{"items":[`) }, want: "unexpected EOF"},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
stored := cloneCheckpointArtifact(base)
|
||||||
|
test.mutate(&stored)
|
||||||
|
if _, err := decodeCheckpointArtifact(codec, stored); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("decode error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
154
internal/framework/pipeline/synchronized_collaborators.go
Normal file
154
internal/framework/pipeline/synchronized_collaborators.go
Normal file
@@ -0,0 +1,154 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"sync"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
type lockedDebugRecorder struct {
|
||||||
|
inner DebugRecorder
|
||||||
|
mu sync.Mutex
|
||||||
|
}
|
||||||
|
|
||||||
|
func synchronizedDebugRecorder(inner DebugRecorder) DebugRecorder {
|
||||||
|
if _, ok := inner.(*lockedDebugRecorder); ok {
|
||||||
|
return inner
|
||||||
|
}
|
||||||
|
return &lockedDebugRecorder{inner: inner}
|
||||||
|
}
|
||||||
|
|
||||||
|
// SynchronizedDebugRecorder serializes access when a recorder is shared by
|
||||||
|
// pipeline operations and construction-injected LLM clients.
|
||||||
|
func SynchronizedDebugRecorder(inner DebugRecorder) DebugRecorder {
|
||||||
|
return synchronizedDebugRecorder(inner)
|
||||||
|
}
|
||||||
|
func (r *lockedDebugRecorder) Enabled() bool {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
return r.inner.Enabled()
|
||||||
|
}
|
||||||
|
func (r *lockedDebugRecorder) WriteJSON(name string, payload any) error {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
return r.inner.WriteJSON(name, payload)
|
||||||
|
}
|
||||||
|
func (r *lockedDebugRecorder) WriteBytes(name string, data []byte) error {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
return r.inner.WriteBytes(name, data)
|
||||||
|
}
|
||||||
|
|
||||||
|
type lockedCheckpointLoader struct {
|
||||||
|
inner CheckpointLoader
|
||||||
|
mu sync.Mutex
|
||||||
|
}
|
||||||
|
|
||||||
|
func synchronizedCheckpointLoader(inner CheckpointLoader) CheckpointLoader {
|
||||||
|
if _, ok := inner.(*lockedCheckpointLoader); ok {
|
||||||
|
return inner
|
||||||
|
}
|
||||||
|
return &lockedCheckpointLoader{inner: inner}
|
||||||
|
}
|
||||||
|
func (l *lockedCheckpointLoader) Enabled() bool {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
return l.inner.Enabled()
|
||||||
|
}
|
||||||
|
func (l *lockedCheckpointLoader) Source(key string) (SourceCheckpoint, CheckpointDecision) {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
return l.inner.Source(key)
|
||||||
|
}
|
||||||
|
func (l *lockedCheckpointLoader) Chunk(key, digest string) (ChunkCheckpoint, CheckpointDecision) {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
return l.inner.Chunk(key, digest)
|
||||||
|
}
|
||||||
|
func (l *lockedCheckpointLoader) Extract(lane, key string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
return l.inner.Extract(lane, key, deps)
|
||||||
|
}
|
||||||
|
func (l *lockedCheckpointLoader) Merge(lane, key string, deps []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
return l.inner.Merge(lane, key, deps)
|
||||||
|
}
|
||||||
|
func (l *lockedCheckpointLoader) Normalize(lane, key string, deps []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
return l.inner.Normalize(lane, key, deps)
|
||||||
|
}
|
||||||
|
|
||||||
|
type lockedCheckpointRecorder struct {
|
||||||
|
inner CheckpointRecorder
|
||||||
|
mu sync.Mutex
|
||||||
|
}
|
||||||
|
|
||||||
|
func synchronizedCheckpointRecorder(inner CheckpointRecorder) CheckpointRecorder {
|
||||||
|
if _, ok := inner.(*lockedCheckpointRecorder); ok {
|
||||||
|
return inner
|
||||||
|
}
|
||||||
|
return &lockedCheckpointRecorder{inner: inner}
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) call(fn func() error) error {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
return fn()
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) SourceRunning(key string) error {
|
||||||
|
return r.call(func() error { return r.inner.SourceRunning(key) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) SourceSucceeded(key string, doc *source.SourceDocument) error {
|
||||||
|
return r.call(func() error { return r.inner.SourceSucceeded(key, doc) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) SourceFailed(key string, err error) error {
|
||||||
|
return r.call(func() error { return r.inner.SourceFailed(key, err) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) ChunkRunning(key, digest string) error {
|
||||||
|
return r.call(func() error { return r.inner.ChunkRunning(key, digest) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) ChunkSucceeded(key, digest string, chunks []source.Chunk, warnings []contracts.Warning) error {
|
||||||
|
return r.call(func() error { return r.inner.ChunkSucceeded(key, digest, chunks, warnings) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) ChunkRejected(key, digest string, rejected contracts.RejectedOutput) error {
|
||||||
|
return r.call(func() error { return r.inner.ChunkRejected(key, digest, rejected) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) ChunkFailed(key, digest string, err error) error {
|
||||||
|
return r.call(func() error { return r.inner.ChunkFailed(key, digest, err) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) ExtractRunning(lane, key string, deps []CheckpointFingerprint) error {
|
||||||
|
return r.call(func() error { return r.inner.ExtractRunning(lane, key, deps) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) ExtractSucceeded(lane, key string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||||
|
return r.call(func() error { return r.inner.ExtractSucceeded(lane, key, deps, outputs, rejected, warnings) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) ExtractFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
|
||||||
|
return r.call(func() error { return r.inner.ExtractFailed(lane, key, deps, err) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) MergeRunning(lane, key string, deps []CheckpointFingerprint) error {
|
||||||
|
return r.call(func() error { return r.inner.MergeRunning(lane, key, deps) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) MergeSucceeded(lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||||
|
return r.call(func() error { return r.inner.MergeSucceeded(lane, key, deps, output, warnings) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) MergeRejected(lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||||
|
return r.call(func() error { return r.inner.MergeRejected(lane, key, deps, rejected) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) MergeFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
|
||||||
|
return r.call(func() error { return r.inner.MergeFailed(lane, key, deps, err) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) NormalizeRunning(lane, key string, deps []CheckpointFingerprint) error {
|
||||||
|
return r.call(func() error { return r.inner.NormalizeRunning(lane, key, deps) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) NormalizeSucceeded(lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||||
|
return r.call(func() error { return r.inner.NormalizeSucceeded(lane, key, deps, output, warnings) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) NormalizeRejected(lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||||
|
return r.call(func() error { return r.inner.NormalizeRejected(lane, key, deps, rejected) })
|
||||||
|
}
|
||||||
|
func (r *lockedCheckpointRecorder) NormalizeFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
|
||||||
|
return r.call(func() error { return r.inner.NormalizeFailed(lane, key, deps, err) })
|
||||||
|
}
|
||||||
63
internal/framework/pipeline/typed_execution.go
Normal file
63
internal/framework/pipeline/typed_execution.go
Normal file
@@ -0,0 +1,63 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"reflect"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
type erasedExtractArtifact struct {
|
||||||
|
LaneID, ExtractorKey, SourceID, ChunkID string
|
||||||
|
ChunkIndex int
|
||||||
|
ChunkRef source.SourceRef
|
||||||
|
Value any
|
||||||
|
}
|
||||||
|
|
||||||
|
type erasedMergeArtifact struct {
|
||||||
|
LaneID, MergerKey, SourceID string
|
||||||
|
Value any
|
||||||
|
}
|
||||||
|
|
||||||
|
type erasedTypedResult struct {
|
||||||
|
Value any
|
||||||
|
Warnings []contracts.Warning
|
||||||
|
}
|
||||||
|
|
||||||
|
type typedValidationTarget struct {
|
||||||
|
stage ModuleStage
|
||||||
|
laneID string
|
||||||
|
moduleKey string
|
||||||
|
source *source.SourceDocument
|
||||||
|
sourceID string
|
||||||
|
sourceInput contracts.LLMInputMaterial
|
||||||
|
sessionID string
|
||||||
|
references contracts.ReferenceSet
|
||||||
|
llmProfile string
|
||||||
|
metadata map[string]any
|
||||||
|
chunk *source.Chunk
|
||||||
|
chunks []source.Chunk
|
||||||
|
ref source.SourceRef
|
||||||
|
value any
|
||||||
|
}
|
||||||
|
|
||||||
|
func exactTypedValue[T any](operation string, value any) (T, error) {
|
||||||
|
want := reflect.TypeFor[T]()
|
||||||
|
if reflect.TypeOf(value) != want {
|
||||||
|
var zero T
|
||||||
|
return zero, fmt.Errorf("%s: expected exact Go type %s, got %T", operation, want, value)
|
||||||
|
}
|
||||||
|
typed, ok := value.(T)
|
||||||
|
if !ok {
|
||||||
|
var zero T
|
||||||
|
return zero, fmt.Errorf("%s: expected exact Go type %s, got %T", operation, want, value)
|
||||||
|
}
|
||||||
|
return typed, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type typedExtractOperation func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error)
|
||||||
|
type typedMergeOperation func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error)
|
||||||
|
type typedNormalizeOperation func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error)
|
||||||
|
type typedValidateOperation func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error)
|
||||||
477
internal/framework/pipeline/typed_resolution_test.go
Normal file
477
internal/framework/pipeline/typed_resolution_test.go
Normal file
@@ -0,0 +1,477 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
type typedTestExtractor[T any] struct{ key string }
|
||||||
|
|
||||||
|
func (e typedTestExtractor[T]) Key() string { return e.key }
|
||||||
|
func (typedTestExtractor[T]) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||||
|
func (typedTestExtractor[T]) Extract(context.Context, contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[T], error) {
|
||||||
|
return contracts.TypedExtractionResult[T]{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type typedTestMerger[T any] struct{ key string }
|
||||||
|
|
||||||
|
func (m typedTestMerger[T]) Key() string { return m.key }
|
||||||
|
func (typedTestMerger[T]) Merge(context.Context, contracts.TypedMergeRequest[T]) (contracts.TypedMergeResult[T], error) {
|
||||||
|
return contracts.TypedMergeResult[T]{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type typedTestNormalizer[T any] struct{ key string }
|
||||||
|
|
||||||
|
func (n typedTestNormalizer[T]) Key() string { return n.key }
|
||||||
|
func (typedTestNormalizer[T]) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||||
|
func (typedTestNormalizer[T]) Normalize(context.Context, contracts.TypedNormalizeRequest[T]) (contracts.TypedNormalizeResult[T], error) {
|
||||||
|
return contracts.TypedNormalizeResult[T]{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type typedTestValidator[T any] struct{ key string }
|
||||||
|
|
||||||
|
func (v typedTestValidator[T]) Name() string { return v.key }
|
||||||
|
func (typedTestValidator[T]) ExecutionClass() contracts.ExecutionClass {
|
||||||
|
return contracts.ExecutionClassDeterministic
|
||||||
|
}
|
||||||
|
func (typedTestValidator[T]) Validate(context.Context, contracts.TypedValidationRequest[T]) (contracts.ValidationResult, error) {
|
||||||
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type typedTestChunkValidator struct{ key string }
|
||||||
|
|
||||||
|
func (v typedTestChunkValidator) Name() string { return v.key }
|
||||||
|
func (typedTestChunkValidator) ExecutionClass() contracts.ExecutionClass {
|
||||||
|
return contracts.ExecutionClassDeterministic
|
||||||
|
}
|
||||||
|
func (typedTestChunkValidator) Validate(context.Context, contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
|
||||||
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type typedTestSerializedValidator struct{ key string }
|
||||||
|
|
||||||
|
type typedTestInput struct {
|
||||||
|
key string
|
||||||
|
doc *source.SourceDocument
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v *typedTestInput) Key() string { return v.key }
|
||||||
|
func (v *typedTestInput) Parse(context.Context, contracts.ParseRequest) (*source.SourceDocument, error) {
|
||||||
|
return v.doc, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type typedTestChunker struct {
|
||||||
|
key string
|
||||||
|
chunks []source.Chunk
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v *typedTestChunker) Key() string { return v.key }
|
||||||
|
func (v *typedTestChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||||
|
func (v *typedTestChunker) Chunk(context.Context, contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
||||||
|
return contracts.ChunkResult{Chunks: v.chunks}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type typedTestOutput struct{ key string }
|
||||||
|
|
||||||
|
func (v *typedTestOutput) Key() string { return v.key }
|
||||||
|
func (v *typedTestOutput) Encode(context.Context, contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||||
|
return contracts.OutputResult{}, nil
|
||||||
|
}
|
||||||
|
func typedTestDocument() *source.SourceDocument {
|
||||||
|
ref := source.SourceRef{SourceID: "source", StartUnitID: 1, EndUnitID: 1}
|
||||||
|
doc := &source.SourceDocument{ID: "source", Kind: "document", Format: "text/plain", Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "text", Ref: ref}}}
|
||||||
|
doc.Digest, _ = source.DigestDocument(doc)
|
||||||
|
return doc
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v typedTestSerializedValidator) Name() string { return v.key }
|
||||||
|
func (typedTestSerializedValidator) ExecutionClass() contracts.ExecutionClass {
|
||||||
|
return contracts.ExecutionClassDeterministic
|
||||||
|
}
|
||||||
|
func (typedTestSerializedValidator) Validate(context.Context, contracts.SerializedValidationRequest) (contracts.ValidationResult, error) {
|
||||||
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type typedCatalogOptions struct {
|
||||||
|
registerNotesCodec bool
|
||||||
|
registerScoreCodec bool
|
||||||
|
registerNotesMerger bool
|
||||||
|
registerScoreMerger bool
|
||||||
|
registerNotesNormalizer bool
|
||||||
|
registerScoreNormalizer bool
|
||||||
|
registerScoreValidator bool
|
||||||
|
scoreExtractorUsesNotes bool
|
||||||
|
notesCodec testArtifactCodec[codecNotes]
|
||||||
|
scoreCodec testArtifactCodec[codecScore]
|
||||||
|
}
|
||||||
|
|
||||||
|
func completeTypedCatalogOptions() typedCatalogOptions {
|
||||||
|
return typedCatalogOptions{
|
||||||
|
registerNotesCodec: true,
|
||||||
|
registerScoreCodec: true,
|
||||||
|
registerNotesMerger: true,
|
||||||
|
registerScoreMerger: true,
|
||||||
|
registerNotesNormalizer: true,
|
||||||
|
registerScoreNormalizer: true,
|
||||||
|
registerScoreValidator: true,
|
||||||
|
notesCodec: notesCodec(),
|
||||||
|
scoreCodec: scoreCodec(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveTypedHeterogeneousLanes(t *testing.T) {
|
||||||
|
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
|
||||||
|
resolved, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, catalog)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got, want := resolvedLaneIDs(resolved.ArtifactLanes), []string{"notes", "score"}; !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("lane order = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
assertResolvedArtifactIdentity(t, resolved.ArtifactLanes[0], "test/notes", "notes.v1")
|
||||||
|
assertResolvedArtifactIdentity(t, resolved.ArtifactLanes[1], "test/score", "score.v1")
|
||||||
|
|
||||||
|
chunkChain := resolved.ValidatorChains[0]
|
||||||
|
if got := resolvedValidatorTargets(chunkChain.Validators); !reflect.DeepEqual(got, []ValidatorTarget{ValidatorTargetChunk, ValidatorTargetSerialized}) {
|
||||||
|
t.Fatalf("chunk validator targets = %#v, want chunk then serialized", got)
|
||||||
|
}
|
||||||
|
notesExtract := resolved.ValidatorChains[1]
|
||||||
|
if got := resolvedValidatorTargets(notesExtract.Validators); !reflect.DeepEqual(got, []ValidatorTarget{ValidatorTargetTyped, ValidatorTargetSerialized}) {
|
||||||
|
t.Fatalf("notes extract validator targets = %#v, want typed then serialized", got)
|
||||||
|
}
|
||||||
|
if notesExtract.Validators[0].ArtifactKind != "test/notes" || resolved.ValidatorChains[4].Validators[0].ArtifactKind != "test/score" {
|
||||||
|
t.Fatalf("resolved validator kinds = %#v, want lane kinds", resolved.ValidatorChains)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPrepareConstructsHeterogeneousTypedLanes(t *testing.T) {
|
||||||
|
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
|
||||||
|
resolved, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, catalog)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if len(prepared.ArtifactLanes) != 2 || prepared.lanes[0].typed == nil || prepared.lanes[1].typed == nil {
|
||||||
|
t.Fatalf("PreparedPipeline lanes = %#v, want two typed executors", prepared.ArtifactLanes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunExecutesHeterogeneousTypedLanesWithCheckpointsDisabled(t *testing.T) {
|
||||||
|
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
|
||||||
|
resolved, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, catalog)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v", err)
|
||||||
|
}
|
||||||
|
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Prepare() error = %v", err)
|
||||||
|
}
|
||||||
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte(`{"source":true}`), RunID: "typed-run"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(output.NormalizeOutputs) != 2 {
|
||||||
|
t.Fatalf("normalize outputs = %#v, want two typed lane results", output.NormalizeOutputs)
|
||||||
|
}
|
||||||
|
if len(output.CheckpointEvents) != 0 {
|
||||||
|
t.Fatalf("checkpoint events = %#v, want none with checkpoint loading disabled", output.CheckpointEvents)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveTypedLaneRejectsIncompatibleComposition(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*typedCatalogOptions)
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "missing codec", mutate: func(options *typedCatalogOptions) { options.registerScoreCodec = false }, want: `artifact codec "test/score" is not registered`},
|
||||||
|
{name: "missing merger variant", mutate: func(options *typedCatalogOptions) { options.registerScoreMerger = false }, want: `merger "typed/merge" has no typed variant for artifact kind "test/score"`},
|
||||||
|
{name: "missing normalizer variant", mutate: func(options *typedCatalogOptions) { options.registerScoreNormalizer = false }, want: `normalizer "typed/normalize" has no typed variant for artifact kind "test/score"`},
|
||||||
|
{name: "extractor Go type mismatch", mutate: func(options *typedCatalogOptions) { options.scoreExtractorUsesNotes = true }, want: `artifact kind "test/score" requires Go type pipeline.codecScore, got pipeline.codecNotes`},
|
||||||
|
{name: "wrong validator kind", mutate: func(options *typedCatalogOptions) { options.registerScoreValidator = false }, want: `validator "typed/check" has no typed variant for artifact kind "test/score"; registered kinds: test/notes`},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
options := completeTypedCatalogOptions()
|
||||||
|
test.mutate(&options)
|
||||||
|
_, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, typedResolutionCatalog(t, options))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("ResolvePipeline() error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
|
||||||
|
registry := NewMergerRegistry()
|
||||||
|
spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}
|
||||||
|
constructor := func() (contracts.Merger[codecNotes], error) {
|
||||||
|
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
||||||
|
}
|
||||||
|
if err := RegisterMerger(registry, spec, constructor); err != nil {
|
||||||
|
t.Fatalf("RegisterMerger() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if err := RegisterMerger(registry, spec, constructor); err == nil || !strings.Contains(err.Error(), "already registered") {
|
||||||
|
t.Fatalf("duplicate RegisterMerger() error = %v, want duplicate variant error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
||||||
|
extractors := NewExtractorRegistry()
|
||||||
|
if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) {
|
||||||
|
return typedTestExtractor[codecNotes]{key: "typed/extract"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("RegisterExtractor() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := extractors.validateOptions("typed/extract", map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||||
|
t.Fatalf("extractor option validation error = %v, want unknown option", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
mergers := NewMergerRegistry()
|
||||||
|
if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) {
|
||||||
|
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("RegisterMerger() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := mergers.validateOptions("typed/merge", "test/notes", map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||||
|
t.Fatalf("merger option validation error = %v, want unknown option", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
normalizers := NewNormalizerRegistry()
|
||||||
|
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||||
|
return typedTestNormalizer[codecNotes]{key: "typed/normalize"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := normalizers.validateOptions("typed/normalize", "test/notes", map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||||
|
t.Fatalf("normalizer option validation error = %v, want unknown option", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
validators := NewValidatorRegistry()
|
||||||
|
if err := RegisterTypedValidator(validators, "test/notes", ValidatorSpec{Key: "typed/check", ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.TypedValidator[codecNotes], error) {
|
||||||
|
return typedTestValidator[codecNotes]{key: "typed/check"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("RegisterTypedValidator() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := validators.validateOptions(ResolvedValidator{Binding: ModuleBinding{Module: "typed/check", Options: map[string]any{"unexpected": true}}, Target: ValidatorTargetTyped, ArtifactKind: "test/notes"}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||||
|
t.Fatalf("validator option validation error = %v, want unknown option", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvedPipelineDigestIncludesArtifactSchemaIdentity(t *testing.T) {
|
||||||
|
baseOptions := completeTypedCatalogOptions()
|
||||||
|
base, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, typedResolutionCatalog(t, baseOptions))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline(base) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
identityOptions := completeTypedCatalogOptions()
|
||||||
|
identityOptions.notesCodec.schema.ID = "notes-renamed.v1"
|
||||||
|
identity, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, typedResolutionCatalog(t, identityOptions))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline(identity) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if base.Digest == identity.Digest {
|
||||||
|
t.Fatalf("pipeline digest = %q after schema identity change, want different digest", identity.Digest)
|
||||||
|
}
|
||||||
|
|
||||||
|
digestOptions := completeTypedCatalogOptions()
|
||||||
|
digestOptions.notesCodec.schema.JSONSchema = []byte(`{"additionalProperties":false,"description":"changed","properties":{"items":{"items":{"type":"string"},"type":"array"}},"required":["items"],"type":"object"}`)
|
||||||
|
changedSchema, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, typedResolutionCatalog(t, digestOptions))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePipeline(schema bytes) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if base.Digest == changedSchema.Digest {
|
||||||
|
t.Fatalf("pipeline digest = %q after schema digest change, want different digest", changedSchema.Digest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func typedResolutionCatalog(t *testing.T, options typedCatalogOptions) ModuleCatalog {
|
||||||
|
t.Helper()
|
||||||
|
catalog := ModuleCatalog{
|
||||||
|
Inputs: NewInputAdapterRegistry(),
|
||||||
|
Chunkers: NewChunkerRegistry(),
|
||||||
|
ArtifactCodecs: NewArtifactCodecRegistry(),
|
||||||
|
Extractors: NewExtractorRegistry(),
|
||||||
|
Mergers: NewMergerRegistry(),
|
||||||
|
Normalizers: NewNormalizerRegistry(),
|
||||||
|
Validators: NewValidatorRegistry(),
|
||||||
|
ValidatorChains: NewValidatorChainRegistry(),
|
||||||
|
Outputs: NewOutputEncoderRegistry(),
|
||||||
|
}
|
||||||
|
mustRegisterTypedTestBase(t, catalog)
|
||||||
|
if options.registerNotesCodec {
|
||||||
|
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, options.notesCodec)
|
||||||
|
}
|
||||||
|
if options.registerScoreCodec {
|
||||||
|
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, options.scoreCodec)
|
||||||
|
}
|
||||||
|
mustRegisterTypedExtractor(t, catalog.Extractors, "typed/extract-notes", "test/notes", typedTestExtractor[codecNotes]{key: "typed/extract-notes"})
|
||||||
|
if options.scoreExtractorUsesNotes {
|
||||||
|
mustRegisterTypedExtractor(t, catalog.Extractors, "typed/extract-score", "test/score", typedTestExtractor[codecNotes]{key: "typed/extract-score"})
|
||||||
|
} else {
|
||||||
|
mustRegisterTypedExtractor(t, catalog.Extractors, "typed/extract-score", "test/score", typedTestExtractor[codecScore]{key: "typed/extract-score"})
|
||||||
|
}
|
||||||
|
if options.registerNotesMerger {
|
||||||
|
mustRegisterTypedMerger(t, catalog.Mergers, "test/notes", typedTestMerger[codecNotes]{key: "typed/merge"})
|
||||||
|
}
|
||||||
|
if options.registerScoreMerger {
|
||||||
|
mustRegisterTypedMerger(t, catalog.Mergers, "test/score", typedTestMerger[codecScore]{key: "typed/merge"})
|
||||||
|
}
|
||||||
|
if options.registerNotesNormalizer {
|
||||||
|
mustRegisterTypedNormalizer(t, catalog.Normalizers, "test/notes", typedTestNormalizer[codecNotes]{key: "typed/normalize"})
|
||||||
|
}
|
||||||
|
if options.registerScoreNormalizer {
|
||||||
|
mustRegisterTypedNormalizer(t, catalog.Normalizers, "test/score", typedTestNormalizer[codecScore]{key: "typed/normalize"})
|
||||||
|
}
|
||||||
|
mustRegisterTypedValidator(t, catalog.Validators, "test/notes", typedTestValidator[codecNotes]{key: "typed/check"})
|
||||||
|
if options.registerScoreValidator {
|
||||||
|
mustRegisterTypedValidator(t, catalog.Validators, "test/score", typedTestValidator[codecScore]{key: "typed/check"})
|
||||||
|
}
|
||||||
|
if err := RegisterChunkValidator(catalog.Validators, ValidatorSpec{Key: "chunk/check", ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.ChunkValidator, error) {
|
||||||
|
return typedTestChunkValidator{key: "chunk/check"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("RegisterChunkValidator() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := RegisterSerializedValidator(catalog.Validators, SerializedValidatorSpec{
|
||||||
|
ValidatorSpec: ValidatorSpec{Key: "serialized/check", ExecutionClass: contracts.ExecutionClassDeterministic},
|
||||||
|
SupportsArtifacts: true,
|
||||||
|
}, func() (contracts.SerializedValidator, error) {
|
||||||
|
return typedTestSerializedValidator{key: "serialized/check"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("RegisterSerializedValidator() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := RegisterSerializedValidator(catalog.Validators, SerializedValidatorSpec{
|
||||||
|
ValidatorSpec: ValidatorSpec{Key: "serialized/chunks", ExecutionClass: contracts.ExecutionClassDeterministic},
|
||||||
|
SupportsChunks: true,
|
||||||
|
}, func() (contracts.SerializedValidator, error) {
|
||||||
|
return typedTestSerializedValidator{key: "serialized/chunks"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("RegisterSerializedValidator(chunks) error = %v", err)
|
||||||
|
}
|
||||||
|
return catalog
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustRegisterTypedTestBase(t *testing.T, catalog ModuleCatalog) {
|
||||||
|
t.Helper()
|
||||||
|
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput}, func() (contracts.InputAdapter, error) {
|
||||||
|
return &typedTestInput{key: "typed/input", doc: typedTestDocument()}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register input: %v", err)
|
||||||
|
}
|
||||||
|
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk}, func() (contracts.Chunker, error) {
|
||||||
|
doc := typedTestDocument()
|
||||||
|
return &typedTestChunker{key: "typed/chunk", chunks: []source.Chunk{{ID: "chunk-1", SourceID: doc.ID, Index: 0, Ref: doc.Units[0].Ref, Content: []byte(`{"chunk":1}`), MediaType: "application/json", Units: []source.SourceUnit{doc.Units[0]}}}}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register chunker: %v", err)
|
||||||
|
}
|
||||||
|
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput}, func() (contracts.OutputEncoder, error) {
|
||||||
|
return &typedTestOutput{key: "typed/output"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register output: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func registriesFromModuleCatalog(catalog ModuleCatalog) Registries {
|
||||||
|
return Registries{
|
||||||
|
Inputs: catalog.Inputs, Chunkers: catalog.Chunkers, ArtifactCodecs: catalog.ArtifactCodecs,
|
||||||
|
Extractors: catalog.Extractors, Mergers: catalog.Mergers, Normalizers: catalog.Normalizers,
|
||||||
|
Validators: catalog.Validators, ValidatorChains: catalog.ValidatorChains, Outputs: catalog.Outputs,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustRegisterArtifactCodec[T any](t *testing.T, registry *ArtifactCodecRegistry, codec contracts.ArtifactCodec[T]) {
|
||||||
|
t.Helper()
|
||||||
|
if err := RegisterArtifactCodec(registry, codec); err != nil {
|
||||||
|
t.Fatalf("RegisterArtifactCodec() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustRegisterTypedExtractor[T any](t *testing.T, registry *ExtractorRegistry, key string, kind contracts.ArtifactKind, extractor contracts.Extractor[T]) {
|
||||||
|
t.Helper()
|
||||||
|
if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil {
|
||||||
|
t.Fatalf("RegisterExtractor() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustRegisterTypedMerger[T any](t *testing.T, registry *MergerRegistry, kind contracts.ArtifactKind, merger contracts.Merger[T]) {
|
||||||
|
t.Helper()
|
||||||
|
if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil {
|
||||||
|
t.Fatalf("RegisterMerger() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustRegisterTypedNormalizer[T any](t *testing.T, registry *NormalizerRegistry, kind contracts.ArtifactKind, normalizer contracts.Normalizer[T]) {
|
||||||
|
t.Helper()
|
||||||
|
if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil {
|
||||||
|
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustRegisterTypedValidator[T any](t *testing.T, registry *ValidatorRegistry, kind contracts.ArtifactKind, validator contracts.TypedValidator[T]) {
|
||||||
|
t.Helper()
|
||||||
|
if err := RegisterTypedValidator(registry, kind, ValidatorSpec{Key: "typed/check", ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.TypedValidator[T], error) { return validator, nil }); err != nil {
|
||||||
|
t.Fatalf("RegisterTypedValidator() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func typedResolutionProfile() PipelineProfile {
|
||||||
|
validatorOverride := ValidatorOverride{Set: true, Validators: []ModuleBinding{Binding("typed/check"), Binding("serialized/check")}}
|
||||||
|
return PipelineProfile{
|
||||||
|
ID: "typed-pipeline",
|
||||||
|
Input: Binding("typed/input"),
|
||||||
|
Chunk: ModuleBinding{Module: "typed/chunk", Validators: ValidatorOverride{Set: true, Validators: []ModuleBinding{Binding("chunk/check"), Binding("serialized/chunks")}}},
|
||||||
|
Artifacts: map[string]ArtifactLaneProfile{
|
||||||
|
"score": typedLaneProfile("typed/extract-score", validatorOverride),
|
||||||
|
"notes": typedLaneProfile("typed/extract-notes", validatorOverride),
|
||||||
|
},
|
||||||
|
Output: Binding("typed/output"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func typedLaneProfile(extractor string, validators ValidatorOverride) ArtifactLaneProfile {
|
||||||
|
return ArtifactLaneProfile{
|
||||||
|
Extract: ModuleBinding{Module: extractor, Validators: validators},
|
||||||
|
Merge: Binding("typed/merge"),
|
||||||
|
Normalize: Binding("typed/normalize"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertResolvedArtifactIdentity(t *testing.T, lane ResolvedArtifactLane, kind contracts.ArtifactKind, schemaID string) {
|
||||||
|
t.Helper()
|
||||||
|
if lane.ArtifactKind != kind || lane.ArtifactSchemaID != schemaID || lane.ArtifactSchemaName == "" || lane.ArtifactSchemaVersion == "" || lane.ArtifactSchemaDigest == "" {
|
||||||
|
t.Fatalf("resolved lane identity = %#v, want kind %q schema %q with complete metadata", lane, kind, schemaID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolvedLaneIDs(lanes []ResolvedArtifactLane) []string {
|
||||||
|
ids := make([]string, len(lanes))
|
||||||
|
for i, lane := range lanes {
|
||||||
|
ids[i] = lane.ID
|
||||||
|
}
|
||||||
|
return ids
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolvedValidatorTargets(validators []ResolvedValidator) []ValidatorTarget {
|
||||||
|
targets := make([]ValidatorTarget, len(validators))
|
||||||
|
for i, validator := range validators {
|
||||||
|
targets[i] = validator.Target
|
||||||
|
}
|
||||||
|
return targets
|
||||||
|
}
|
||||||
|
|
||||||
|
var _ contracts.Extractor[codecNotes] = typedTestExtractor[codecNotes]{}
|
||||||
|
var _ contracts.Merger[codecNotes] = typedTestMerger[codecNotes]{}
|
||||||
|
var _ contracts.Normalizer[codecNotes] = typedTestNormalizer[codecNotes]{}
|
||||||
|
var _ contracts.TypedValidator[codecNotes] = typedTestValidator[codecNotes]{}
|
||||||
|
var _ contracts.ChunkValidator = typedTestChunkValidator{}
|
||||||
|
var _ contracts.SerializedValidator = typedTestSerializedValidator{}
|
||||||
@@ -1,125 +1,292 @@
|
|||||||
package pipeline
|
package pipeline
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"reflect"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
type ValidatorConstructor func() (contracts.Validator, error)
|
|
||||||
|
|
||||||
type ValidatorSpec struct {
|
type ValidatorSpec struct {
|
||||||
Key string `json:"key"`
|
Key string `json:"key"`
|
||||||
ExecutionClass contracts.ExecutionClass `json:"execution_class"`
|
ExecutionClass contracts.ExecutionClass `json:"execution_class"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type SerializedValidatorSpec struct {
|
||||||
|
ValidatorSpec
|
||||||
|
SupportsChunks bool `json:"supports_chunks,omitempty"`
|
||||||
|
SupportsArtifacts bool `json:"supports_artifacts,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ValidatorTarget string
|
||||||
|
|
||||||
|
const (
|
||||||
|
ValidatorTargetChunk ValidatorTarget = "chunk"
|
||||||
|
ValidatorTargetSerialized ValidatorTarget = "serialized"
|
||||||
|
ValidatorTargetTyped ValidatorTarget = "typed"
|
||||||
|
)
|
||||||
|
|
||||||
type ValidatorRegistry struct {
|
type ValidatorRegistry struct {
|
||||||
constructors map[string]ValidatorConstructor
|
typedEntries map[artifactVariantKey]typedValidatorEntry
|
||||||
specs map[string]ValidatorSpec
|
chunkEntries map[string]chunkValidatorEntry
|
||||||
|
serializedEntries map[string]serializedValidatorEntry
|
||||||
|
}
|
||||||
|
|
||||||
|
type typedValidatorEntry struct {
|
||||||
|
spec ValidatorSpec
|
||||||
|
kind contracts.ArtifactKind
|
||||||
|
valueType reflect.Type
|
||||||
|
validateOptions OptionValidator
|
||||||
|
builder func(BuildRequest) (any, error)
|
||||||
|
validate typedValidateOperation
|
||||||
|
}
|
||||||
|
|
||||||
|
type chunkValidatorEntry struct {
|
||||||
|
spec ValidatorSpec
|
||||||
|
validateOptions OptionValidator
|
||||||
|
builder func(BuildRequest) (contracts.ChunkValidator, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type serializedValidatorEntry struct {
|
||||||
|
spec SerializedValidatorSpec
|
||||||
|
validateOptions OptionValidator
|
||||||
|
builder func(BuildRequest) (contracts.SerializedValidator, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewValidatorRegistry() *ValidatorRegistry {
|
func NewValidatorRegistry() *ValidatorRegistry {
|
||||||
return &ValidatorRegistry{
|
return &ValidatorRegistry{
|
||||||
constructors: make(map[string]ValidatorConstructor),
|
typedEntries: make(map[artifactVariantKey]typedValidatorEntry),
|
||||||
specs: make(map[string]ValidatorSpec),
|
chunkEntries: make(map[string]chunkValidatorEntry),
|
||||||
|
serializedEntries: make(map[string]serializedValidatorEntry),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ValidatorRegistry) Register(key string, constructor ValidatorConstructor) error {
|
func RegisterTypedValidator[T any](registry *ValidatorRegistry, kind contracts.ArtifactKind, spec ValidatorSpec, constructor func() (contracts.TypedValidator[T], error)) error {
|
||||||
return r.RegisterWithSpec(ValidatorSpec{Key: key, ExecutionClass: contracts.ExecutionClassDeterministic}, constructor)
|
if constructor == nil {
|
||||||
|
return fmt.Errorf("validator constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
}
|
||||||
|
return RegisterTypedValidatorBuilder(registry, kind, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.TypedValidator[T], error) {
|
||||||
|
return constructor()
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ValidatorRegistry) RegisterWithSpec(spec ValidatorSpec, constructor ValidatorConstructor) error {
|
func RegisterTypedValidatorBuilder[T any](registry *ValidatorRegistry, kind contracts.ArtifactKind, spec ValidatorSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.TypedValidator[T], error)) error {
|
||||||
if r == nil {
|
if registry == nil {
|
||||||
return fmt.Errorf("validator registry must not be nil")
|
return fmt.Errorf("validator registry must not be nil")
|
||||||
}
|
}
|
||||||
|
|
||||||
normalizedSpec, err := normalizeValidatorSpec(spec)
|
normalizedSpec, err := normalizeValidatorSpec(spec)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if constructor == nil {
|
kind = normalizeArtifactKind(kind)
|
||||||
return fmt.Errorf("validator constructor for %q must not be nil", normalizedSpec.Key)
|
if kind == "" {
|
||||||
|
return fmt.Errorf("typed validator %q artifact kind must not be empty", normalizedSpec.Key)
|
||||||
}
|
}
|
||||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
if validateOptions == nil {
|
||||||
return fmt.Errorf("validator %q is already registered", normalizedSpec.Key)
|
return fmt.Errorf("validator option validator for %q must not be nil", normalizedSpec.Key)
|
||||||
}
|
}
|
||||||
|
if builder == nil {
|
||||||
if r.constructors == nil {
|
return fmt.Errorf("validator builder for %q must not be nil", normalizedSpec.Key)
|
||||||
r.constructors = make(map[string]ValidatorConstructor)
|
|
||||||
}
|
}
|
||||||
if r.specs == nil {
|
key := artifactVariantKey{module: normalizedSpec.Key, kind: kind}
|
||||||
r.specs = make(map[string]ValidatorSpec)
|
if _, ok := registry.typedEntries[key]; ok {
|
||||||
|
return fmt.Errorf("validator %q variant for artifact kind %q is already registered", key.module, key.kind)
|
||||||
|
}
|
||||||
|
if registry.typedEntries == nil {
|
||||||
|
registry.typedEntries = make(map[artifactVariantKey]typedValidatorEntry)
|
||||||
|
}
|
||||||
|
registry.typedEntries[key] = typedValidatorEntry{
|
||||||
|
spec: normalizedSpec,
|
||||||
|
kind: kind,
|
||||||
|
valueType: reflect.TypeFor[T](),
|
||||||
|
validateOptions: validateOptions,
|
||||||
|
builder: func(request BuildRequest) (any, error) {
|
||||||
|
return builder(cloneBuildRequest(request))
|
||||||
|
},
|
||||||
|
validate: func(ctx context.Context, implementation any, target typedValidationTarget) (contracts.ValidationResult, error) {
|
||||||
|
validator, ok := implementation.(contracts.TypedValidator[T])
|
||||||
|
if !ok {
|
||||||
|
return contracts.ValidationResult{}, fmt.Errorf("validator %q has incompatible implementation %T", normalizedSpec.Key, implementation)
|
||||||
|
}
|
||||||
|
value, err := exactTypedValue[T]("validate artifact value", target.value)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.ValidationResult{}, err
|
||||||
|
}
|
||||||
|
return validator.Validate(ctx, contracts.TypedValidationRequest[T]{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Source: target.source, SourceID: target.sourceID, SourceInput: target.sourceInput, SessionID: target.sessionID, References: target.references, LLMProfile: target.llmProfile, Metadata: target.metadata, Chunk: target.chunk, Chunks: target.chunks, Ref: target.ref, Value: value})
|
||||||
|
},
|
||||||
}
|
}
|
||||||
r.constructors[normalizedSpec.Key] = constructor
|
|
||||||
r.specs[normalizedSpec.Key] = normalizedSpec
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ValidatorRegistry) Build(key string) (contracts.Validator, error) {
|
func RegisterChunkValidator(registry *ValidatorRegistry, spec ValidatorSpec, constructor func() (contracts.ChunkValidator, error)) error {
|
||||||
if r == nil {
|
if constructor == nil {
|
||||||
return nil, fmt.Errorf("validator registry must not be nil")
|
return fmt.Errorf("validator constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
}
|
}
|
||||||
|
return RegisterChunkValidatorBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.ChunkValidator, error) {
|
||||||
|
return constructor()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
normalizedKey := strings.TrimSpace(key)
|
func RegisterChunkValidatorBuilder(registry *ValidatorRegistry, spec ValidatorSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.ChunkValidator, error)) error {
|
||||||
if normalizedKey == "" {
|
if registry == nil {
|
||||||
return nil, fmt.Errorf("validator key must not be empty")
|
return fmt.Errorf("validator registry must not be nil")
|
||||||
}
|
}
|
||||||
|
normalizedSpec, err := normalizeValidatorSpec(spec)
|
||||||
constructor, ok := r.constructors[normalizedKey]
|
|
||||||
if !ok {
|
|
||||||
return nil, fmt.Errorf("validator %q is not registered", normalizedKey)
|
|
||||||
}
|
|
||||||
|
|
||||||
validator, err := constructor()
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("build validator %q: %w", normalizedKey, err)
|
return err
|
||||||
|
}
|
||||||
|
if validateOptions == nil {
|
||||||
|
return fmt.Errorf("validator option validator for %q must not be nil", normalizedSpec.Key)
|
||||||
|
}
|
||||||
|
if builder == nil {
|
||||||
|
return fmt.Errorf("validator builder for %q must not be nil", normalizedSpec.Key)
|
||||||
|
}
|
||||||
|
if _, ok := registry.chunkEntries[normalizedSpec.Key]; ok {
|
||||||
|
return fmt.Errorf("chunk validator %q is already registered", normalizedSpec.Key)
|
||||||
|
}
|
||||||
|
if registry.chunkEntries == nil {
|
||||||
|
registry.chunkEntries = make(map[string]chunkValidatorEntry)
|
||||||
|
}
|
||||||
|
registry.chunkEntries[normalizedSpec.Key] = chunkValidatorEntry{spec: normalizedSpec, validateOptions: validateOptions, builder: builder}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func RegisterSerializedValidator(registry *ValidatorRegistry, spec SerializedValidatorSpec, constructor func() (contracts.SerializedValidator, error)) error {
|
||||||
|
if constructor == nil {
|
||||||
|
return fmt.Errorf("validator constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||||
|
}
|
||||||
|
return RegisterSerializedValidatorBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.SerializedValidator, error) {
|
||||||
|
return constructor()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func RegisterSerializedValidatorBuilder(registry *ValidatorRegistry, spec SerializedValidatorSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.SerializedValidator, error)) error {
|
||||||
|
if registry == nil {
|
||||||
|
return fmt.Errorf("validator registry must not be nil")
|
||||||
|
}
|
||||||
|
normalizedValidatorSpec, err := normalizeValidatorSpec(spec.ValidatorSpec)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
spec.ValidatorSpec = normalizedValidatorSpec
|
||||||
|
if !spec.SupportsChunks && !spec.SupportsArtifacts {
|
||||||
|
return fmt.Errorf("serialized validator %q must support chunks, artifacts, or both", spec.Key)
|
||||||
|
}
|
||||||
|
if validateOptions == nil {
|
||||||
|
return fmt.Errorf("validator option validator for %q must not be nil", spec.Key)
|
||||||
|
}
|
||||||
|
if builder == nil {
|
||||||
|
return fmt.Errorf("validator builder for %q must not be nil", spec.Key)
|
||||||
|
}
|
||||||
|
if _, ok := registry.serializedEntries[spec.Key]; ok {
|
||||||
|
return fmt.Errorf("serialized validator %q is already registered", spec.Key)
|
||||||
|
}
|
||||||
|
if registry.serializedEntries == nil {
|
||||||
|
registry.serializedEntries = make(map[string]serializedValidatorEntry)
|
||||||
|
}
|
||||||
|
registry.serializedEntries[spec.Key] = serializedValidatorEntry{spec: spec, validateOptions: validateOptions, builder: builder}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ValidatorRegistry) validateOptions(resolved ResolvedValidator) error {
|
||||||
|
if r == nil {
|
||||||
|
return fmt.Errorf("validator registry must not be nil")
|
||||||
|
}
|
||||||
|
key := strings.TrimSpace(resolved.Binding.Module)
|
||||||
|
var validator OptionValidator
|
||||||
|
switch resolved.Target {
|
||||||
|
case ValidatorTargetTyped:
|
||||||
|
entry, ok := r.typedEntry(key, resolved.ArtifactKind)
|
||||||
|
if ok {
|
||||||
|
validator = entry.validateOptions
|
||||||
|
}
|
||||||
|
case ValidatorTargetChunk:
|
||||||
|
entry, ok := r.chunkEntry(key)
|
||||||
|
if ok {
|
||||||
|
validator = entry.validateOptions
|
||||||
|
}
|
||||||
|
case ValidatorTargetSerialized:
|
||||||
|
entry, ok := r.serializedEntry(key)
|
||||||
|
if ok {
|
||||||
|
validator = entry.validateOptions
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if validator == nil {
|
if validator == nil {
|
||||||
return nil, fmt.Errorf("validator %q constructor returned nil", normalizedKey)
|
return fmt.Errorf("validator %q construction entry is not registered", key)
|
||||||
}
|
}
|
||||||
if validator.Name() != normalizedKey {
|
return validateRegisteredOptions(validator, resolved.Binding.Options)
|
||||||
return nil, fmt.Errorf("validator %q returned name %q", normalizedKey, validator.Name())
|
|
||||||
}
|
|
||||||
spec, ok := r.specs[normalizedKey]
|
|
||||||
if !ok {
|
|
||||||
return nil, fmt.Errorf("validator %q spec is not registered", normalizedKey)
|
|
||||||
}
|
|
||||||
if validator.ExecutionClass() != spec.ExecutionClass {
|
|
||||||
return nil, fmt.Errorf("validator %q returned execution class %q, want %q", normalizedKey, validator.ExecutionClass(), spec.ExecutionClass)
|
|
||||||
}
|
|
||||||
|
|
||||||
return validator, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ValidatorRegistry) Spec(key string) (ValidatorSpec, bool) {
|
func (r *ValidatorRegistry) Spec(key string) (ValidatorSpec, bool) {
|
||||||
if r == nil {
|
if r == nil {
|
||||||
return ValidatorSpec{}, false
|
return ValidatorSpec{}, false
|
||||||
}
|
}
|
||||||
|
normalized := strings.TrimSpace(key)
|
||||||
spec, ok := r.specs[strings.TrimSpace(key)]
|
if entry, found := r.chunkEntries[normalized]; found {
|
||||||
if !ok {
|
return entry.spec, true
|
||||||
return ValidatorSpec{}, false
|
|
||||||
}
|
}
|
||||||
return spec, true
|
if entry, found := r.serializedEntries[normalized]; found {
|
||||||
|
return entry.spec.ValidatorSpec, true
|
||||||
|
}
|
||||||
|
if kinds := r.registeredTypedKinds(normalized); len(kinds) > 0 {
|
||||||
|
entry, found := r.typedEntry(normalized, kinds[0])
|
||||||
|
return entry.spec, found
|
||||||
|
}
|
||||||
|
return ValidatorSpec{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ValidatorRegistry) typedEntry(key string, kind contracts.ArtifactKind) (typedValidatorEntry, bool) {
|
||||||
|
if r == nil {
|
||||||
|
return typedValidatorEntry{}, false
|
||||||
|
}
|
||||||
|
entry, ok := r.typedEntries[artifactVariantKey{module: strings.TrimSpace(key), kind: normalizeArtifactKind(kind)}]
|
||||||
|
return entry, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ValidatorRegistry) chunkEntry(key string) (chunkValidatorEntry, bool) {
|
||||||
|
if r == nil {
|
||||||
|
return chunkValidatorEntry{}, false
|
||||||
|
}
|
||||||
|
entry, ok := r.chunkEntries[strings.TrimSpace(key)]
|
||||||
|
return entry, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ValidatorRegistry) serializedEntry(key string) (serializedValidatorEntry, bool) {
|
||||||
|
if r == nil {
|
||||||
|
return serializedValidatorEntry{}, false
|
||||||
|
}
|
||||||
|
entry, ok := r.serializedEntries[strings.TrimSpace(key)]
|
||||||
|
return entry, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *ValidatorRegistry) registeredTypedKinds(key string) []contracts.ArtifactKind {
|
||||||
|
if r == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
module := strings.TrimSpace(key)
|
||||||
|
kinds := make([]contracts.ArtifactKind, 0)
|
||||||
|
for variant := range r.typedEntries {
|
||||||
|
if variant.module == module {
|
||||||
|
kinds = append(kinds, variant.kind)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sortArtifactKinds(kinds)
|
||||||
|
return kinds
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *ValidatorRegistry) RegisteredSpecs() []ValidatorSpec {
|
func (r *ValidatorRegistry) RegisteredSpecs() []ValidatorSpec {
|
||||||
if r == nil || len(r.specs) == 0 {
|
if r == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
keys := r.RegisteredKeys()
|
||||||
keys := make([]string, 0, len(r.specs))
|
|
||||||
for key := range r.specs {
|
|
||||||
keys = append(keys, key)
|
|
||||||
}
|
|
||||||
sort.Strings(keys)
|
|
||||||
|
|
||||||
specs := make([]ValidatorSpec, 0, len(keys))
|
specs := make([]ValidatorSpec, 0, len(keys))
|
||||||
for _, key := range keys {
|
for _, key := range keys {
|
||||||
specs = append(specs, r.specs[key])
|
if spec, ok := r.Spec(key); ok {
|
||||||
|
specs = append(specs, spec)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return specs
|
return specs
|
||||||
}
|
}
|
||||||
@@ -128,15 +295,21 @@ func (r *ValidatorRegistry) RegisteredKeys() []string {
|
|||||||
if r == nil {
|
if r == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
keys := make(map[string]struct{})
|
||||||
return sortedRegistryKeys(r.constructors)
|
for key := range r.typedEntries {
|
||||||
|
keys[key.module] = struct{}{}
|
||||||
|
}
|
||||||
|
for key := range r.chunkEntries {
|
||||||
|
keys[key] = struct{}{}
|
||||||
|
}
|
||||||
|
for key := range r.serializedEntries {
|
||||||
|
keys[key] = struct{}{}
|
||||||
|
}
|
||||||
|
return sortedRegistryKeys(keys)
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeValidatorSpec(spec ValidatorSpec) (ValidatorSpec, error) {
|
func normalizeValidatorSpec(spec ValidatorSpec) (ValidatorSpec, error) {
|
||||||
normalized := ValidatorSpec{
|
normalized := ValidatorSpec{Key: strings.TrimSpace(spec.Key), ExecutionClass: spec.ExecutionClass}
|
||||||
Key: strings.TrimSpace(spec.Key),
|
|
||||||
ExecutionClass: spec.ExecutionClass,
|
|
||||||
}
|
|
||||||
if normalized.Key == "" {
|
if normalized.Key == "" {
|
||||||
return ValidatorSpec{}, fmt.Errorf("validator key must not be empty")
|
return ValidatorSpec{}, fmt.Errorf("validator key must not be empty")
|
||||||
}
|
}
|
||||||
@@ -147,3 +320,7 @@ func normalizeValidatorSpec(spec ValidatorSpec) (ValidatorSpec, error) {
|
|||||||
}
|
}
|
||||||
return normalized, nil
|
return normalized, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func sortValidatorSpecs(specs []ValidatorSpec) {
|
||||||
|
sort.Slice(specs, func(i, j int) bool { return specs[i].Key < specs[j].Key })
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,117 +0,0 @@
|
|||||||
package pipeline
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"reflect"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestValidatorRegistryBehavior(t *testing.T) {
|
|
||||||
registry := NewValidatorRegistry()
|
|
||||||
if err := registry.Register(" generic-validator ", validatorConstructor("generic-validator", contracts.ExecutionClassDeterministic)); err != nil {
|
|
||||||
t.Fatalf("Register() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
validator, err := registry.Build("generic-validator")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Build() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if validator.Name() != "generic-validator" {
|
|
||||||
t.Fatalf("validator name = %q, want generic-validator", validator.Name())
|
|
||||||
}
|
|
||||||
|
|
||||||
spec, ok := registry.Spec(" generic-validator ")
|
|
||||||
if !ok {
|
|
||||||
t.Fatal("Spec() ok = false, want true")
|
|
||||||
}
|
|
||||||
want := ValidatorSpec{Key: "generic-validator", ExecutionClass: contracts.ExecutionClassDeterministic}
|
|
||||||
if !reflect.DeepEqual(spec, want) {
|
|
||||||
t.Fatalf("Spec() = %#v, want %#v", spec, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestValidatorRegistryRegistersSpecs(t *testing.T) {
|
|
||||||
registry := NewValidatorRegistry()
|
|
||||||
spec := ValidatorSpec{Key: " llm-validator ", ExecutionClass: contracts.ExecutionClassLLMBacked}
|
|
||||||
if err := registry.RegisterWithSpec(spec, validatorConstructor("llm-validator", contracts.ExecutionClassLLMBacked)); err != nil {
|
|
||||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
got, ok := registry.Spec("llm-validator")
|
|
||||||
if !ok {
|
|
||||||
t.Fatal("Spec() ok = false, want true")
|
|
||||||
}
|
|
||||||
want := ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked}
|
|
||||||
if !reflect.DeepEqual(got, want) {
|
|
||||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestValidatorRegistryRegisteredSpecsAreSorted(t *testing.T) {
|
|
||||||
registry := NewValidatorRegistry()
|
|
||||||
for _, key := range []string{"zeta", "alpha"} {
|
|
||||||
if err := registry.Register(key, validatorConstructor(key, contracts.ExecutionClassDeterministic)); err != nil {
|
|
||||||
t.Fatalf("Register(%q) error = %v", key, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
specs := registry.RegisteredSpecs()
|
|
||||||
if len(specs) != 2 || specs[0].Key != "alpha" || specs[1].Key != "zeta" {
|
|
||||||
t.Fatalf("RegisteredSpecs() = %#v, want sorted specs", specs)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestValidatorRegistryRejectsUnsupportedExecutionClass(t *testing.T) {
|
|
||||||
registry := NewValidatorRegistry()
|
|
||||||
err := registry.RegisterWithSpec(
|
|
||||||
ValidatorSpec{Key: "invalid-validator", ExecutionClass: contracts.ExecutionClass("unsupported")},
|
|
||||||
validatorConstructor("invalid-validator", contracts.ExecutionClass("unsupported")),
|
|
||||||
)
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("RegisterWithSpec() error = nil, want unsupported execution class error")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestValidatorRegistryRejectsConstructorExecutionClassMismatch(t *testing.T) {
|
|
||||||
registry := NewValidatorRegistry()
|
|
||||||
if err := registry.RegisterWithSpec(
|
|
||||||
ValidatorSpec{Key: "validator", ExecutionClass: contracts.ExecutionClassDeterministic},
|
|
||||||
validatorConstructor("validator", contracts.ExecutionClassLLMBacked),
|
|
||||||
); err != nil {
|
|
||||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err := registry.Build("validator")
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("Build() error = nil, want execution class mismatch")
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), "execution class") {
|
|
||||||
t.Fatalf("Build() error = %q, want execution class context", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
type testValidator struct {
|
|
||||||
name string
|
|
||||||
executionClass contracts.ExecutionClass
|
|
||||||
}
|
|
||||||
|
|
||||||
func validatorConstructor(name string, executionClass contracts.ExecutionClass) ValidatorConstructor {
|
|
||||||
return func() (contracts.Validator, error) {
|
|
||||||
return testValidator{name: name, executionClass: executionClass}, nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (validator testValidator) Name() string {
|
|
||||||
return validator.name
|
|
||||||
}
|
|
||||||
|
|
||||||
func (validator testValidator) ExecutionClass() contracts.ExecutionClass {
|
|
||||||
return validator.executionClass
|
|
||||||
}
|
|
||||||
|
|
||||||
func (validator testValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
|
|
||||||
return contracts.ValidationResult{Approved: true}, nil
|
|
||||||
}
|
|
||||||
@@ -1,462 +0,0 @@
|
|||||||
package pipeline
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"crypto/sha256"
|
|
||||||
"encoding/hex"
|
|
||||||
"encoding/json"
|
|
||||||
"fmt"
|
|
||||||
"os"
|
|
||||||
"reflect"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestWalkingSkeletonFixture(t *testing.T) {
|
|
||||||
inputBytes := readTestFixture(t, "testdata/walking_skeleton_input.json")
|
|
||||||
expectedBytes := readTestFixture(t, "testdata/walking_skeleton_output.json")
|
|
||||||
llmClient := &walkingSkeletonLLMClient{}
|
|
||||||
|
|
||||||
resolved, err := ResolvePipeline(walkingSkeletonProfile(), ResolveOptions{}, walkingSkeletonCatalog(t))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
output, err := New(walkingSkeletonRegistries(t)).Run(context.Background(), RunInput{
|
|
||||||
Pipeline: resolved,
|
|
||||||
SourceID: "fixture-source",
|
|
||||||
Path: "walking_skeleton_input.json",
|
|
||||||
RawInput: inputBytes,
|
|
||||||
LLMClient: llmClient,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Run() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
if len(output.OutputFiles) != 1 {
|
|
||||||
t.Fatalf("len(OutputFiles) = %d, want 1", len(output.OutputFiles))
|
|
||||||
}
|
|
||||||
if output.OutputFiles[0].ContentType != "application/json" {
|
|
||||||
t.Fatalf("ContentType = %q, want application/json", output.OutputFiles[0].ContentType)
|
|
||||||
}
|
|
||||||
assertStructuralJSONEqual(t, output.OutputFiles[0].Bytes, expectedBytes)
|
|
||||||
if llmClient.calls != 3 {
|
|
||||||
t.Fatalf("LLM calls = %d, want extractor calls plus normalizer call", llmClient.calls)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWalkingSkeletonResolutionRejectsMissingCapability(t *testing.T) {
|
|
||||||
catalog := walkingSkeletonCatalog(t)
|
|
||||||
catalog.Extractors = NewExtractorRegistry()
|
|
||||||
if err := catalog.Extractors.RegisterWithSpec(ModuleSpec{
|
|
||||||
Key: "fake/extract",
|
|
||||||
Stage: StageExtract,
|
|
||||||
Requires: []string{"missing"},
|
|
||||||
Provides: []string{"fake_artifacts"},
|
|
||||||
}, func() (contracts.Extractor, error) {
|
|
||||||
return walkingSkeletonExtractor{}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err := ResolvePipeline(walkingSkeletonProfile(), ResolveOptions{}, catalog)
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("ResolvePipeline() error = nil, want error")
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), "missing") {
|
|
||||||
t.Fatalf("ResolvePipeline() error = %q, want missing capability", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWalkingSkeletonResolutionRejectsUnknownOnlyLane(t *testing.T) {
|
|
||||||
_, err := ResolvePipeline(walkingSkeletonProfile(), ResolveOptions{Only: []string{"missing"}}, walkingSkeletonCatalog(t))
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("ResolvePipeline() error = nil, want error")
|
|
||||||
}
|
|
||||||
if !strings.Contains(err.Error(), "missing") || !strings.Contains(err.Error(), "not declared") {
|
|
||||||
t.Fatalf("ResolvePipeline() error = %q, want unknown lane error", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func walkingSkeletonProfile() PipelineProfile {
|
|
||||||
return PipelineProfile{
|
|
||||||
ID: "walking-skeleton",
|
|
||||||
Input: Binding("fake/input"),
|
|
||||||
Chunk: Binding("fake/chunk"),
|
|
||||||
Output: Binding("json"),
|
|
||||||
Artifacts: map[string]ArtifactLaneProfile{
|
|
||||||
"events": {Extract: Binding("fake/extract")},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func walkingSkeletonCatalog(t *testing.T) ModuleCatalog {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
catalog := ModuleCatalog{
|
|
||||||
Inputs: NewInputAdapterRegistry(),
|
|
||||||
Chunkers: NewChunkerRegistry(),
|
|
||||||
Extractors: NewExtractorRegistry(),
|
|
||||||
Mergers: NewMergerRegistry(),
|
|
||||||
Normalizers: NewNormalizerRegistry(),
|
|
||||||
ValidatorChains: NewValidatorChainRegistry(),
|
|
||||||
Outputs: NewOutputEncoderRegistry(),
|
|
||||||
}
|
|
||||||
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{
|
|
||||||
Key: "fake/input",
|
|
||||||
Stage: StageInput,
|
|
||||||
Provides: []string{"plain_text"},
|
|
||||||
}, func() (contracts.InputAdapter, error) {
|
|
||||||
return walkingSkeletonInput{}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("register fake input: %v", err)
|
|
||||||
}
|
|
||||||
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{
|
|
||||||
Key: "fake/chunk",
|
|
||||||
Stage: StageChunk,
|
|
||||||
Requires: []string{"plain_text"},
|
|
||||||
Provides: []string{"chunks"},
|
|
||||||
}, func() (contracts.Chunker, error) {
|
|
||||||
return walkingSkeletonChunker{}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("register fake chunker: %v", err)
|
|
||||||
}
|
|
||||||
if err := catalog.Extractors.RegisterWithSpec(ModuleSpec{
|
|
||||||
Key: "fake/extract",
|
|
||||||
Stage: StageExtract,
|
|
||||||
Requires: []string{"chunks"},
|
|
||||||
Provides: []string{"fake_artifacts"},
|
|
||||||
}, func() (contracts.Extractor, error) {
|
|
||||||
return walkingSkeletonExtractor{}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("register fake extractor: %v", err)
|
|
||||||
}
|
|
||||||
if err := catalog.Mergers.RegisterWithSpec(ModuleSpec{
|
|
||||||
Key: DefaultMergeModule,
|
|
||||||
Stage: StageMerge,
|
|
||||||
Requires: []string{"fake_artifacts"},
|
|
||||||
}, func() (contracts.Merger, error) {
|
|
||||||
return walkingSkeletonMerger{}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("register append-order merger: %v", err)
|
|
||||||
}
|
|
||||||
if err := catalog.Normalizers.RegisterWithSpec(ModuleSpec{
|
|
||||||
Key: DefaultNormalizeModule,
|
|
||||||
Stage: StageNormalize,
|
|
||||||
}, func() (contracts.Normalizer, error) {
|
|
||||||
return walkingSkeletonNormalizer{}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("register no-op normalizer: %v", err)
|
|
||||||
}
|
|
||||||
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{
|
|
||||||
Key: "json",
|
|
||||||
Stage: StageOutput,
|
|
||||||
}, func() (contracts.OutputEncoder, error) {
|
|
||||||
return walkingSkeletonOutput{}, nil
|
|
||||||
}); err != nil {
|
|
||||||
t.Fatalf("register fake output: %v", err)
|
|
||||||
}
|
|
||||||
return catalog
|
|
||||||
}
|
|
||||||
|
|
||||||
func walkingSkeletonRegistries(t *testing.T) Registries {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
catalog := walkingSkeletonCatalog(t)
|
|
||||||
return Registries{
|
|
||||||
Inputs: catalog.Inputs,
|
|
||||||
Chunkers: catalog.Chunkers,
|
|
||||||
Extractors: catalog.Extractors,
|
|
||||||
Mergers: catalog.Mergers,
|
|
||||||
Normalizers: catalog.Normalizers,
|
|
||||||
Outputs: catalog.Outputs,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
type walkingSkeletonInput struct{}
|
|
||||||
|
|
||||||
func (input walkingSkeletonInput) Key() string {
|
|
||||||
return "fake/input"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (input walkingSkeletonInput) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
|
|
||||||
var fixture struct {
|
|
||||||
ID string `json:"id"`
|
|
||||||
Units []struct {
|
|
||||||
ID int `json:"id"`
|
|
||||||
Text string `json:"text"`
|
|
||||||
} `json:"units"`
|
|
||||||
}
|
|
||||||
if err := json.Unmarshal(req.Raw, &fixture); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
units := make([]source.SourceUnit, 0, len(fixture.Units))
|
|
||||||
for _, unit := range fixture.Units {
|
|
||||||
units = append(units, source.SourceUnit{
|
|
||||||
ID: unit.ID,
|
|
||||||
Kind: "unit",
|
|
||||||
Text: unit.Text,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
return &source.SourceDocument{
|
|
||||||
ID: fixture.ID,
|
|
||||||
Kind: "fixture",
|
|
||||||
Format: "application/json",
|
|
||||||
Digest: rawDigest(req.Raw),
|
|
||||||
Units: units,
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type walkingSkeletonChunker struct{}
|
|
||||||
|
|
||||||
func (chunker walkingSkeletonChunker) Key() string {
|
|
||||||
return "fake/chunk"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (chunker walkingSkeletonChunker) ReferenceSlots() []contracts.ReferenceSlot {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (chunker walkingSkeletonChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
|
||||||
if len(req.Source.Units) < 3 {
|
|
||||||
return contracts.ChunkResult{}, fmt.Errorf("fixture source must contain at least three units")
|
|
||||||
}
|
|
||||||
return contracts.ChunkResult{
|
|
||||||
Chunks: []contracts.SourceChunk{
|
|
||||||
{
|
|
||||||
ID: req.Source.ID + ":chunk:0",
|
|
||||||
SourceID: req.Source.ID,
|
|
||||||
Index: 0,
|
|
||||||
StartUnitID: req.Source.Units[0].ID,
|
|
||||||
EndUnitID: req.Source.Units[1].ID,
|
|
||||||
Content: []byte(`{"units":[1,2]}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
Units: append([]source.SourceUnit(nil), req.Source.Units[:2]...),
|
|
||||||
},
|
|
||||||
{
|
|
||||||
ID: req.Source.ID + ":chunk:1",
|
|
||||||
SourceID: req.Source.ID,
|
|
||||||
Index: 1,
|
|
||||||
StartUnitID: req.Source.Units[2].ID,
|
|
||||||
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
|
|
||||||
Content: []byte(`{"units":[3]}`),
|
|
||||||
MediaType: "application/json",
|
|
||||||
Units: append([]source.SourceUnit(nil), req.Source.Units[2:]...),
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type walkingSkeletonExtractor struct{}
|
|
||||||
|
|
||||||
func (extractor walkingSkeletonExtractor) Key() string {
|
|
||||||
return "fake/extract"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (extractor walkingSkeletonExtractor) ReferenceSlots() []contracts.ReferenceSlot {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (extractor walkingSkeletonExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
|
||||||
var response struct {
|
|
||||||
Call int `json:"call"`
|
|
||||||
}
|
|
||||||
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
|
||||||
StageName: "fake/extract",
|
|
||||||
PromptID: "fake.event",
|
|
||||||
PromptVersion: "v1",
|
|
||||||
}, &response); err != nil {
|
|
||||||
return contracts.ExtractionResult{}, err
|
|
||||||
}
|
|
||||||
|
|
||||||
payload, err := json.Marshal(map[string]any{
|
|
||||||
"chunk_id": req.Chunk.ID,
|
|
||||||
"llm_call": response.Call,
|
|
||||||
"text": chunkText(req.Chunk.Units),
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return contracts.ExtractionResult{}, err
|
|
||||||
}
|
|
||||||
|
|
||||||
return contracts.ExtractionResult{
|
|
||||||
Output: contracts.ExtractOutput{
|
|
||||||
Schema: contracts.ResponseSchema{ID: "fake_event", Name: "fake_event", Version: "v1"},
|
|
||||||
Payload: contracts.RawPayload{
|
|
||||||
Content: payload,
|
|
||||||
MediaType: "application/json",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type walkingSkeletonLLMClient struct {
|
|
||||||
calls int
|
|
||||||
}
|
|
||||||
|
|
||||||
func (client *walkingSkeletonLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
|
||||||
client.calls++
|
|
||||||
if response, ok := out.(*struct {
|
|
||||||
Call int `json:"call"`
|
|
||||||
}); ok {
|
|
||||||
response.Call = client.calls
|
|
||||||
}
|
|
||||||
content, err := json.Marshal(map[string]any{"call": client.calls})
|
|
||||||
if err != nil {
|
|
||||||
return contracts.StructuredCompletionResponse{}, err
|
|
||||||
}
|
|
||||||
return contracts.StructuredCompletionResponse{
|
|
||||||
Content: content,
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type walkingSkeletonMerger struct{}
|
|
||||||
|
|
||||||
func (merger walkingSkeletonMerger) Key() string {
|
|
||||||
return DefaultMergeModule
|
|
||||||
}
|
|
||||||
|
|
||||||
func (merger walkingSkeletonMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
|
|
||||||
outputs := make([]json.RawMessage, 0, len(req.ExtractOutputs))
|
|
||||||
for _, output := range req.ExtractOutputs {
|
|
||||||
outputs = append(outputs, json.RawMessage(output.Payload.Content))
|
|
||||||
}
|
|
||||||
content, err := json.Marshal(map[string]any{"outputs": outputs})
|
|
||||||
if err != nil {
|
|
||||||
return contracts.MergeResult{}, err
|
|
||||||
}
|
|
||||||
return contracts.MergeResult{
|
|
||||||
Output: contracts.MergeOutput{
|
|
||||||
LaneID: req.LaneID,
|
|
||||||
SourceID: req.Source.ID,
|
|
||||||
Schema: contracts.ResponseSchema{ID: "fake_event", Name: "fake_event", Version: "v1"},
|
|
||||||
Payload: contracts.RawPayload{
|
|
||||||
Content: content,
|
|
||||||
MediaType: "application/json",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type walkingSkeletonNormalizer struct{}
|
|
||||||
|
|
||||||
func (normalizer walkingSkeletonNormalizer) Key() string {
|
|
||||||
return DefaultNormalizeModule
|
|
||||||
}
|
|
||||||
|
|
||||||
func (normalizer walkingSkeletonNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (normalizer walkingSkeletonNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
|
|
||||||
var response struct {
|
|
||||||
Call int `json:"call"`
|
|
||||||
}
|
|
||||||
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
|
||||||
StageName: "fake/normalize",
|
|
||||||
PromptID: "fake.normalize",
|
|
||||||
PromptVersion: "v1",
|
|
||||||
}, &response); err != nil {
|
|
||||||
return contracts.NormalizeResult{}, err
|
|
||||||
}
|
|
||||||
return contracts.NormalizeResult{
|
|
||||||
Output: contracts.NormalizeOutput{
|
|
||||||
LaneID: req.LaneID,
|
|
||||||
SourceID: req.MergeOutput.SourceID,
|
|
||||||
Schema: req.MergeOutput.Schema,
|
|
||||||
Payload: req.MergeOutput.Payload,
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type walkingSkeletonOutput struct{}
|
|
||||||
|
|
||||||
func (output walkingSkeletonOutput) Key() string {
|
|
||||||
return "json"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (output walkingSkeletonOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
|
||||||
type rawOutput struct {
|
|
||||||
LaneID string `json:"lane_id"`
|
|
||||||
NormalizerKey string `json:"normalizer_key"`
|
|
||||||
SourceID string `json:"source_id"`
|
|
||||||
Schema contracts.ResponseSchema `json:"schema"`
|
|
||||||
MediaType string `json:"media_type"`
|
|
||||||
Content json.RawMessage `json:"content"`
|
|
||||||
}
|
|
||||||
rawOutputs := make([]rawOutput, 0, len(req.NormalizeOutputs))
|
|
||||||
for _, output := range req.NormalizeOutputs {
|
|
||||||
rawOutputs = append(rawOutputs, rawOutput{
|
|
||||||
LaneID: output.LaneID,
|
|
||||||
NormalizerKey: output.NormalizerKey,
|
|
||||||
SourceID: output.SourceID,
|
|
||||||
Schema: output.Schema,
|
|
||||||
MediaType: output.Payload.MediaType,
|
|
||||||
Content: json.RawMessage(output.Payload.Content),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
encoded, err := json.Marshal(struct {
|
|
||||||
Manifest artifacts.RunManifest `json:"manifest"`
|
|
||||||
NormalizeOutputs []rawOutput `json:"normalize_outputs"`
|
|
||||||
}{
|
|
||||||
Manifest: artifacts.RunManifest{
|
|
||||||
PipelineID: req.Manifest.PipelineID,
|
|
||||||
PipelineDigest: req.Manifest.PipelineDigest,
|
|
||||||
ArtifactLanes: req.Manifest.ArtifactLanes,
|
|
||||||
ValidationStatus: req.Manifest.ValidationStatus,
|
|
||||||
},
|
|
||||||
NormalizeOutputs: rawOutputs,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return contracts.OutputResult{}, err
|
|
||||||
}
|
|
||||||
return contracts.OutputResult{
|
|
||||||
Files: []contracts.OutputFile{
|
|
||||||
{Name: "output.json", ContentType: "application/json", Bytes: encoded},
|
|
||||||
},
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func readTestFixture(t *testing.T, path string) []byte {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
bytes, err := os.ReadFile(path)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("read fixture %q: %v", path, err)
|
|
||||||
}
|
|
||||||
return bytes
|
|
||||||
}
|
|
||||||
|
|
||||||
func assertStructuralJSONEqual(t *testing.T, gotBytes, wantBytes []byte) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var got any
|
|
||||||
if err := json.Unmarshal(gotBytes, &got); err != nil {
|
|
||||||
t.Fatalf("unmarshal actual JSON: %v\n%s", err, gotBytes)
|
|
||||||
}
|
|
||||||
var want any
|
|
||||||
if err := json.Unmarshal(wantBytes, &want); err != nil {
|
|
||||||
t.Fatalf("unmarshal expected JSON: %v\n%s", err, wantBytes)
|
|
||||||
}
|
|
||||||
if !reflect.DeepEqual(got, want) {
|
|
||||||
gotFormatted, _ := json.MarshalIndent(got, "", " ")
|
|
||||||
wantFormatted, _ := json.MarshalIndent(want, "", " ")
|
|
||||||
t.Fatalf("actual JSON:\n%s\nwant:\n%s", gotFormatted, wantFormatted)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func chunkText(units []source.SourceUnit) string {
|
|
||||||
parts := make([]string, 0, len(units))
|
|
||||||
for _, unit := range units {
|
|
||||||
parts = append(parts, unit.Text)
|
|
||||||
}
|
|
||||||
return strings.Join(parts, " ")
|
|
||||||
}
|
|
||||||
|
|
||||||
func rawDigest(raw []byte) string {
|
|
||||||
sum := sha256.Sum256(raw)
|
|
||||||
return "sha256:" + hex.EncodeToString(sum[:])
|
|
||||||
}
|
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package sharedassets
|
package promptfs
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package sharedassets
|
package promptfs
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"io/fs"
|
"io/fs"
|
||||||
@@ -1,30 +0,0 @@
|
|||||||
package scenes
|
|
||||||
|
|
||||||
import "gitea.maximumdirect.net/eric/notarius/internal/modules/sharedassets/dnd"
|
|
||||||
|
|
||||||
type chunkResponse struct {
|
|
||||||
Scenes []sceneResponse `json:"scenes"`
|
|
||||||
BoundaryCaveats []string `json:"boundary_caveats"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type sceneResponse struct {
|
|
||||||
StartUnitID dnd.UnitRef `json:"start_unit_id"`
|
|
||||||
EndUnitID dnd.UnitRef `json:"end_unit_id"`
|
|
||||||
ShortTitle string `json:"short_title"`
|
|
||||||
PrimaryMode string `json:"primary_mode"`
|
|
||||||
MainParticipants []string `json:"main_participants"`
|
|
||||||
Summary string `json:"summary"`
|
|
||||||
BoundaryNote string `json:"boundary_note"`
|
|
||||||
BoundaryConfidence string `json:"boundary_confidence"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type normalizedScene struct {
|
|
||||||
StartUnitID int
|
|
||||||
EndUnitID int
|
|
||||||
ShortTitle string
|
|
||||||
PrimaryMode string
|
|
||||||
MainParticipants []string
|
|
||||||
Summary string
|
|
||||||
BoundaryNote string
|
|
||||||
BoundaryConfidence string
|
|
||||||
}
|
|
||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/sharedassets/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
const Key = "dnd/scenes"
|
const Key = "dnd/scenes"
|
||||||
@@ -23,7 +23,7 @@ var providedCapabilities = []string{
|
|||||||
"chunks.scenes",
|
"chunks.scenes",
|
||||||
}
|
}
|
||||||
|
|
||||||
var referenceSlotDescriptions = dnd.ReferenceSlotDescriptions{
|
var referenceSlotDescriptions = shared.ReferenceSlotDescriptions{
|
||||||
Glossary: "Optional campaign glossary reference material used only for scene disambiguation.",
|
Glossary: "Optional campaign glossary reference material used only for scene disambiguation.",
|
||||||
Party: "Optional party roster reference material used only for scene disambiguation.",
|
Party: "Optional party roster reference material used only for scene disambiguation.",
|
||||||
Players: "Optional player list reference material used only for scene disambiguation.",
|
Players: "Optional player list reference material used only for scene disambiguation.",
|
||||||
@@ -33,10 +33,17 @@ var referenceSlotDescriptions = dnd.ReferenceSlotDescriptions{
|
|||||||
var _ contracts.Chunker = (*Chunker)(nil)
|
var _ contracts.Chunker = (*Chunker)(nil)
|
||||||
var _ contracts.ManifestMetadataProvider = (*Chunker)(nil)
|
var _ contracts.ManifestMetadataProvider = (*Chunker)(nil)
|
||||||
|
|
||||||
type Chunker struct{}
|
type Options struct{}
|
||||||
|
|
||||||
func New() *Chunker {
|
type Chunker struct {
|
||||||
return &Chunker{}
|
llm contracts.StructuredLLMClient
|
||||||
|
}
|
||||||
|
|
||||||
|
func New(llmClient contracts.StructuredLLMClient, _ Options) (*Chunker, error) {
|
||||||
|
if llmClient == nil {
|
||||||
|
return nil, chunkerErrorf("LLM client must not be nil")
|
||||||
|
}
|
||||||
|
return &Chunker{llm: llmClient}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Chunker) Key() string {
|
func (c *Chunker) Key() string {
|
||||||
@@ -44,7 +51,7 @@ func (c *Chunker) Key() string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
|
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
|
||||||
return dnd.ReferenceSlots(referenceSlotDescriptions)
|
return shared.ReferenceSlots(referenceSlotDescriptions)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Chunker) ManifestMetadata() map[string]any {
|
func (c *Chunker) ManifestMetadata() map[string]any {
|
||||||
@@ -71,6 +78,9 @@ func (c *Chunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contra
|
|||||||
if c == nil {
|
if c == nil {
|
||||||
return contracts.ChunkResult{}, chunkerErrorf("chunker must not be nil")
|
return contracts.ChunkResult{}, chunkerErrorf("chunker must not be nil")
|
||||||
}
|
}
|
||||||
|
if c.llm == nil {
|
||||||
|
return contracts.ChunkResult{}, chunkerErrorf("LLM client must not be nil")
|
||||||
|
}
|
||||||
if ctx == nil {
|
if ctx == nil {
|
||||||
return contracts.ChunkResult{}, chunkerErrorf("context must not be nil")
|
return contracts.ChunkResult{}, chunkerErrorf("context must not be nil")
|
||||||
}
|
}
|
||||||
@@ -86,21 +96,14 @@ func (c *Chunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contra
|
|||||||
if err := source.ValidateDocument(req.Source); err != nil {
|
if err := source.ValidateDocument(req.Source); err != nil {
|
||||||
return contracts.ChunkResult{}, chunkerErrorf("validate source document: %w", err)
|
return contracts.ChunkResult{}, chunkerErrorf("validate source document: %w", err)
|
||||||
}
|
}
|
||||||
if req.LLMClient == nil {
|
|
||||||
return contracts.ChunkResult{}, chunkerErrorf("LLM client must not be nil")
|
|
||||||
}
|
|
||||||
if len(req.Options) > 0 {
|
|
||||||
return contracts.ChunkResult{}, chunkerErrorf("options are not supported")
|
|
||||||
}
|
|
||||||
|
|
||||||
var response chunkResponse
|
var response chunkResponse
|
||||||
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
if _, err := c.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||||
StageName: Key,
|
StageName: Key,
|
||||||
PromptID: PromptID,
|
PromptID: PromptID,
|
||||||
PromptVersion: ResponseSchemaVersion,
|
PromptVersion: ResponseSchemaVersion,
|
||||||
ProfileID: req.LLMProfile,
|
ProfileID: req.LLMProfile,
|
||||||
SessionID: req.SessionID,
|
SessionID: req.SessionID,
|
||||||
Inputs: dnd.PromptInputs(req.SourceInput, req.References),
|
Inputs: shared.PromptInputs(req.SourceInput, req.References),
|
||||||
}, &response); err != nil {
|
}, &response); err != nil {
|
||||||
return contracts.ChunkResult{}, chunkerErrorf("complete structured output: %w", err)
|
return contracts.ChunkResult{}, chunkerErrorf("complete structured output: %w", err)
|
||||||
}
|
}
|
||||||
@@ -125,17 +128,33 @@ func ModuleSpec() pipeline.ModuleSpec {
|
|||||||
Stage: pipeline.StageChunk,
|
Stage: pipeline.StageChunk,
|
||||||
Requires: append([]string(nil), requiredCapabilities...),
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
Provides: append([]string(nil), providedCapabilities...),
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
ReferenceSlots: dnd.ReferenceSlots(referenceSlotDescriptions),
|
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func Register(registry *pipeline.ChunkerRegistry) error {
|
func Register(registry *pipeline.ChunkerRegistry) error {
|
||||||
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.Chunker, error) {
|
return registry.RegisterBuilderWithSpec(ModuleSpec(), validateOptions, func(request pipeline.BuildRequest) (contracts.Chunker, error) {
|
||||||
return New(), nil
|
options, err := DecodeOptions(request.Options)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return New(request.Dependencies.LLM, options)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]contracts.SourceChunk, error) {
|
func validateOptions(options map[string]any) error {
|
||||||
|
_, err := DecodeOptions(options)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func DecodeOptions(options map[string]any) (Options, error) {
|
||||||
|
if err := pipeline.RejectUnknownOptions(options); err != nil {
|
||||||
|
return Options{}, chunkerErrorf("%w", err)
|
||||||
|
}
|
||||||
|
return Options{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]source.Chunk, error) {
|
||||||
if response.Scenes == nil {
|
if response.Scenes == nil {
|
||||||
return nil, fmt.Errorf("scenes must be present")
|
return nil, fmt.Errorf("scenes must be present")
|
||||||
}
|
}
|
||||||
@@ -148,7 +167,7 @@ func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]c
|
|||||||
unitIndexes[unit.ID] = i
|
unitIndexes[unit.ID] = i
|
||||||
}
|
}
|
||||||
|
|
||||||
chunks := make([]contracts.SourceChunk, 0, len(response.Scenes))
|
chunks := make([]source.Chunk, 0, len(response.Scenes))
|
||||||
previousEnd := -1
|
previousEnd := -1
|
||||||
for i, scene := range response.Scenes {
|
for i, scene := range response.Scenes {
|
||||||
normalized, err := normalizeScene(doc, i, scene)
|
normalized, err := normalizeScene(doc, i, scene)
|
||||||
@@ -186,15 +205,18 @@ func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]c
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
chunks = append(chunks, contracts.SourceChunk{
|
chunks = append(chunks, source.Chunk{
|
||||||
ID: fmt.Sprintf("scene-%06d", i+1),
|
ID: fmt.Sprintf("scene-%06d", i+1),
|
||||||
SourceID: doc.ID,
|
SourceID: doc.ID,
|
||||||
Index: i,
|
Index: i,
|
||||||
StartUnitID: units[0].ID,
|
Ref: source.SourceRef{
|
||||||
EndUnitID: units[len(units)-1].ID,
|
SourceID: doc.ID,
|
||||||
Content: content,
|
StartUnitID: units[0].Ref.StartUnitID,
|
||||||
MediaType: "application/json",
|
EndUnitID: units[len(units)-1].Ref.EndUnitID,
|
||||||
Units: units,
|
},
|
||||||
|
Content: content,
|
||||||
|
MediaType: "application/json",
|
||||||
|
Units: units,
|
||||||
Metadata: map[string]any{
|
Metadata: map[string]any{
|
||||||
"scene_title": normalized.ShortTitle,
|
"scene_title": normalized.ShortTitle,
|
||||||
"primary_mode": normalized.PrimaryMode,
|
"primary_mode": normalized.PrimaryMode,
|
||||||
@@ -228,11 +250,11 @@ func chunkContent(units []source.SourceUnit) ([]byte, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func normalizeScene(doc *source.SourceDocument, index int, scene sceneResponse) (normalizedScene, error) {
|
func normalizeScene(doc *source.SourceDocument, index int, scene sceneResponse) (normalizedScene, error) {
|
||||||
startUnitID, err := dnd.ResolveUnitID(doc, "start_unit_id", scene.StartUnitID)
|
startUnitID, err := shared.ResolveUnitID(doc, "start_unit_id", scene.StartUnitID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return normalizedScene{}, fmt.Errorf("scene[%d] %w", index, err)
|
return normalizedScene{}, fmt.Errorf("scene[%d] %w", index, err)
|
||||||
}
|
}
|
||||||
endUnitID, err := dnd.ResolveUnitID(doc, "end_unit_id", scene.EndUnitID)
|
endUnitID, err := shared.ResolveUnitID(doc, "end_unit_id", scene.EndUnitID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return normalizedScene{}, fmt.Errorf("scene[%d] %w", index, err)
|
return normalizedScene{}, fmt.Errorf("scene[%d] %w", index, err)
|
||||||
}
|
}
|
||||||
@@ -332,6 +354,7 @@ func cloneUnits(units []source.SourceUnit) []source.SourceUnit {
|
|||||||
ID: unit.ID,
|
ID: unit.ID,
|
||||||
Kind: unit.Kind,
|
Kind: unit.Kind,
|
||||||
Text: unit.Text,
|
Text: unit.Text,
|
||||||
|
Ref: unit.Ref,
|
||||||
Metadata: cloneMetadata(unit.Metadata),
|
Metadata: cloneMetadata(unit.Metadata),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -11,14 +11,12 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/sharedassets/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestNewModuleSpecAndRegister(t *testing.T) {
|
func TestNewModuleSpecAndRegister(t *testing.T) {
|
||||||
chunker := New()
|
client := &fakeScenesLLMClient{}
|
||||||
if chunker == nil {
|
chunker := newChunker(t, client)
|
||||||
t.Fatal("New() = nil, want chunker")
|
|
||||||
}
|
|
||||||
if chunker.Key() != Key {
|
if chunker.Key() != Key {
|
||||||
t.Fatalf("Key() = %q, want %q", chunker.Key(), Key)
|
t.Fatalf("Key() = %q, want %q", chunker.Key(), Key)
|
||||||
}
|
}
|
||||||
@@ -52,7 +50,7 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
|
|||||||
if !reflect.DeepEqual(registered, want) {
|
if !reflect.DeepEqual(registered, want) {
|
||||||
t.Fatalf("registered spec = %#v, want %#v", registered, want)
|
t.Fatalf("registered spec = %#v, want %#v", registered, want)
|
||||||
}
|
}
|
||||||
built, err := registry.Build(Key)
|
built, err := registry.BuildWithRequest(Key, pipeline.BuildRequest{Dependencies: pipeline.ModuleDependencies{LLM: client}})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Build(%q) error = %v, want nil", Key, err)
|
t.Fatalf("Build(%q) error = %v, want nil", Key, err)
|
||||||
}
|
}
|
||||||
@@ -62,6 +60,18 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
|
|||||||
if slots := built.ReferenceSlots(); !reflect.DeepEqual(slots, want.ReferenceSlots) {
|
if slots := built.ReferenceSlots(); !reflect.DeepEqual(slots, want.ReferenceSlots) {
|
||||||
t.Fatalf("ReferenceSlots() = %#v, want %#v", slots, want.ReferenceSlots)
|
t.Fatalf("ReferenceSlots() = %#v, want %#v", slots, want.ReferenceSlots)
|
||||||
}
|
}
|
||||||
|
if err := registry.ValidateOptions(Key, map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||||
|
t.Fatalf("ValidateOptions() error = %v, want unknown option error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConstructionRejectsMissingDependencyAndUnknownOptions(t *testing.T) {
|
||||||
|
if _, err := New(nil, Options{}); err == nil || !strings.Contains(err.Error(), "LLM client") {
|
||||||
|
t.Fatalf("New(nil) error = %v, want LLM client error", err)
|
||||||
|
}
|
||||||
|
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||||
|
t.Fatalf("DecodeOptions() error = %v, want unknown option error", err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRegisterNilRegistryReturnsError(t *testing.T) {
|
func TestRegisterNilRegistryReturnsError(t *testing.T) {
|
||||||
@@ -105,8 +115,8 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
|
|||||||
response: chunkResponse{
|
response: chunkResponse{
|
||||||
Scenes: []sceneResponse{
|
Scenes: []sceneResponse{
|
||||||
{
|
{
|
||||||
StartUnitID: dnd.UnitRefFromInt(1),
|
StartUnitID: shared.UnitRefFromInt(1),
|
||||||
EndUnitID: dnd.UnitRefFromInt(2),
|
EndUnitID: shared.UnitRefFromInt(2),
|
||||||
ShortTitle: " Goblin parley ",
|
ShortTitle: " Goblin parley ",
|
||||||
PrimaryMode: "Discussion",
|
PrimaryMode: "Discussion",
|
||||||
MainParticipants: []string{" Aria ", "Goblin scout"},
|
MainParticipants: []string{" Aria ", "Goblin scout"},
|
||||||
@@ -115,8 +125,8 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
|
|||||||
BoundaryConfidence: "High",
|
BoundaryConfidence: "High",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
StartUnitID: dnd.UnitRefFromInt(3),
|
StartUnitID: shared.UnitRefFromInt(3),
|
||||||
EndUnitID: dnd.UnitRefFromInt(4),
|
EndUnitID: shared.UnitRefFromInt(4),
|
||||||
ShortTitle: "Ambush at the gate",
|
ShortTitle: "Ambush at the gate",
|
||||||
PrimaryMode: "Combat",
|
PrimaryMode: "Combat",
|
||||||
MainParticipants: []string{"Aria", "Goblin ambushers"},
|
MainParticipants: []string{"Aria", "Goblin ambushers"},
|
||||||
@@ -129,7 +139,7 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
result, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
|
result, err := newChunker(t, client).Chunk(context.Background(), chunkRequest())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
@@ -179,8 +189,8 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
|
|||||||
if first.SourceID != "session-alpha" || first.Index != 0 {
|
if first.SourceID != "session-alpha" || first.Index != 0 {
|
||||||
t.Fatalf("first chunk = %#v, want source and index fields", first)
|
t.Fatalf("first chunk = %#v, want source and index fields", first)
|
||||||
}
|
}
|
||||||
if first.StartUnitID != 1 || first.EndUnitID != 2 {
|
if first.Ref != (source.SourceRef{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 2}) {
|
||||||
t.Fatalf("first boundaries = %d-%d, want 1-2", first.StartUnitID, first.EndUnitID)
|
t.Fatalf("first ref = %#v, want session-alpha:1-2", first.Ref)
|
||||||
}
|
}
|
||||||
if first.MediaType != "application/json" || len(first.Content) == 0 {
|
if first.MediaType != "application/json" || len(first.Content) == 0 {
|
||||||
t.Fatalf("first payload = media type %q length %d, want JSON content", first.MediaType, len(first.Content))
|
t.Fatalf("first payload = media type %q length %d, want JSON content", first.MediaType, len(first.Content))
|
||||||
@@ -210,8 +220,8 @@ func TestChunkPassesReferencesAsPromptInputs(t *testing.T) {
|
|||||||
client := &fakeScenesLLMClient{response: chunkResponse{
|
client := &fakeScenesLLMClient{response: chunkResponse{
|
||||||
Scenes: []sceneResponse{
|
Scenes: []sceneResponse{
|
||||||
{
|
{
|
||||||
StartUnitID: dnd.UnitRefFromInt(1),
|
StartUnitID: shared.UnitRefFromInt(1),
|
||||||
EndUnitID: dnd.UnitRefFromInt(4),
|
EndUnitID: shared.UnitRefFromInt(4),
|
||||||
ShortTitle: "Ambush",
|
ShortTitle: "Ambush",
|
||||||
PrimaryMode: "Combat",
|
PrimaryMode: "Combat",
|
||||||
MainParticipants: []string{"Aria"},
|
MainParticipants: []string{"Aria"},
|
||||||
@@ -221,7 +231,7 @@ func TestChunkPassesReferencesAsPromptInputs(t *testing.T) {
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
}}
|
}}
|
||||||
req := chunkRequestWithClient(client)
|
req := chunkRequest()
|
||||||
req.References = contracts.ReferenceSet{
|
req.References = contracts.ReferenceSet{
|
||||||
Slots: map[string]contracts.ResolvedReferenceSlot{
|
Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||||
"players": {
|
"players": {
|
||||||
@@ -245,7 +255,7 @@ func TestChunkPassesReferencesAsPromptInputs(t *testing.T) {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
if _, err := New().Chunk(context.Background(), req); err != nil {
|
if _, err := newChunker(t, client).Chunk(context.Background(), req); err != nil {
|
||||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
request := client.requests[0]
|
request := client.requests[0]
|
||||||
@@ -264,7 +274,7 @@ func TestChunkPassesReferencesAsPromptInputs(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestPromptInputsMapLegacyRosterReferenceToParty(t *testing.T) {
|
func TestPromptInputsMapLegacyRosterReferenceToParty(t *testing.T) {
|
||||||
inputs := dnd.PromptInputs(sceneSourceInput(), contracts.ReferenceSet{
|
inputs := shared.PromptInputs(sceneSourceInput(), contracts.ReferenceSet{
|
||||||
Slots: map[string]contracts.ResolvedReferenceSlot{
|
Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||||
"roster": {
|
"roster": {
|
||||||
Slot: contracts.ReferenceSlot{Name: "roster"},
|
Slot: contracts.ReferenceSlot{Name: "roster"},
|
||||||
@@ -293,7 +303,7 @@ func TestChunkRejectsWhitespaceOnlyBoundaryCaveats(t *testing.T) {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
|
_, err := newChunker(t, client).Chunk(context.Background(), chunkRequest())
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("Chunk() error = nil, want malformed structured output error")
|
t.Fatal("Chunk() error = nil, want malformed structured output error")
|
||||||
}
|
}
|
||||||
@@ -306,24 +316,27 @@ func TestChunkDefensivelyCopiesSourceUnitsAndMetadata(t *testing.T) {
|
|||||||
doc := sceneSourceDocument()
|
doc := sceneSourceDocument()
|
||||||
client := &fakeScenesLLMClient{response: validSceneResponse()}
|
client := &fakeScenesLLMClient{response: validSceneResponse()}
|
||||||
|
|
||||||
result, err := New().Chunk(context.Background(), contracts.ChunkRequest{
|
result, err := newChunker(t, client).Chunk(context.Background(), contracts.ChunkRequest{
|
||||||
Source: doc,
|
Source: doc,
|
||||||
SourceInput: sceneSourceInput(),
|
SourceInput: sceneSourceInput(),
|
||||||
SessionID: "session-123",
|
SessionID: "session-123",
|
||||||
LLMProfile: "profile-scenes",
|
LLMProfile: "profile-scenes",
|
||||||
LLMClient: client,
|
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
doc.Units[0].ID = 99
|
doc.Units[0].ID = 99
|
||||||
|
doc.Units[0].Ref.SourceID = "mutated"
|
||||||
doc.Units[0].Metadata["speaker"] = "mutated"
|
doc.Units[0].Metadata["speaker"] = "mutated"
|
||||||
client.response.Scenes[0].MainParticipants[0] = "mutated"
|
client.response.Scenes[0].MainParticipants[0] = "mutated"
|
||||||
|
|
||||||
if result.Chunks[0].Units[0].ID != 1 {
|
if result.Chunks[0].Units[0].ID != 1 {
|
||||||
t.Fatalf("chunk unit ID changed after source mutation: %#v", result.Chunks[0].Units[0])
|
t.Fatalf("chunk unit ID changed after source mutation: %#v", result.Chunks[0].Units[0])
|
||||||
}
|
}
|
||||||
|
if got := result.Chunks[0].Units[0].Ref.SourceID; got != "session-alpha" {
|
||||||
|
t.Fatalf("chunk unit ref changed after source mutation: %#v", result.Chunks[0].Units[0].Ref)
|
||||||
|
}
|
||||||
if result.Chunks[0].Units[0].Metadata["speaker"] != "Alice" {
|
if result.Chunks[0].Units[0].Metadata["speaker"] != "Alice" {
|
||||||
t.Fatalf("chunk unit metadata changed after source mutation: %#v", result.Chunks[0].Units[0].Metadata)
|
t.Fatalf("chunk unit metadata changed after source mutation: %#v", result.Chunks[0].Units[0].Metadata)
|
||||||
}
|
}
|
||||||
@@ -334,7 +347,7 @@ func TestChunkDefensivelyCopiesSourceUnitsAndMetadata(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestChunkerManifestMetadataIncludesPromptAndSchemaProvenance(t *testing.T) {
|
func TestChunkerManifestMetadataIncludesPromptAndSchemaProvenance(t *testing.T) {
|
||||||
metadata := New().ManifestMetadata()
|
metadata := newChunker(t, &fakeScenesLLMClient{}).ManifestMetadata()
|
||||||
|
|
||||||
tests := map[string]string{
|
tests := map[string]string{
|
||||||
"prompt_id": PromptID,
|
"prompt_id": PromptID,
|
||||||
@@ -364,7 +377,7 @@ func TestChunkerManifestMetadataIncludesPromptAndSchemaProvenance(t *testing.T)
|
|||||||
|
|
||||||
func TestChunkRejectsInvalidRequests(t *testing.T) {
|
func TestChunkRejectsInvalidRequests(t *testing.T) {
|
||||||
validClient := &fakeScenesLLMClient{response: validSceneResponse()}
|
validClient := &fakeScenesLLMClient{response: validSceneResponse()}
|
||||||
validReq := chunkRequestWithClient(validClient)
|
validReq := chunkRequest()
|
||||||
canceledCtx, cancel := context.WithCancel(context.Background())
|
canceledCtx, cancel := context.WithCancel(context.Background())
|
||||||
cancel()
|
cancel()
|
||||||
invalidDoc := sceneSourceDocument()
|
invalidDoc := sceneSourceDocument()
|
||||||
@@ -380,13 +393,11 @@ func TestChunkRejectsInvalidRequests(t *testing.T) {
|
|||||||
want string
|
want string
|
||||||
}{
|
}{
|
||||||
{name: "nil chunker", chunker: nil, ctx: context.Background(), req: validReq, want: "chunker"},
|
{name: "nil chunker", chunker: nil, ctx: context.Background(), req: validReq, want: "chunker"},
|
||||||
{name: "nil context", chunker: New(), ctx: nil, req: validReq, want: "context"},
|
{name: "nil context", chunker: newChunker(t, validClient), ctx: nil, req: validReq, want: "context"},
|
||||||
{name: "canceled context", chunker: New(), ctx: canceledCtx, req: validReq, want: "context"},
|
{name: "canceled context", chunker: newChunker(t, validClient), ctx: canceledCtx, req: validReq, want: "context"},
|
||||||
{name: "nil source", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{LLMClient: validClient}, want: "source"},
|
{name: "nil source", chunker: newChunker(t, validClient), ctx: context.Background(), req: contracts.ChunkRequest{}, want: "source"},
|
||||||
{name: "empty source units", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{Source: emptyDoc, LLMClient: validClient}, want: "units"},
|
{name: "empty source units", chunker: newChunker(t, validClient), ctx: context.Background(), req: contracts.ChunkRequest{Source: emptyDoc}, want: "units"},
|
||||||
{name: "invalid source", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{Source: invalidDoc, LLMClient: validClient}, want: "validate source document"},
|
{name: "invalid source", chunker: newChunker(t, validClient), ctx: context.Background(), req: contracts.ChunkRequest{Source: invalidDoc}, want: "validate source document"},
|
||||||
{name: "nil LLM client", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{Source: sceneSourceDocument()}, want: "LLM client"},
|
|
||||||
{name: "unsupported options", chunker: New(), ctx: context.Background(), req: requestWithOptions(validReq), want: "options"},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
@@ -451,8 +462,8 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
|
|||||||
name: "empty metadata field",
|
name: "empty metadata field",
|
||||||
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
||||||
{
|
{
|
||||||
StartUnitID: dnd.UnitRefFromInt(1),
|
StartUnitID: shared.UnitRefFromInt(1),
|
||||||
EndUnitID: dnd.UnitRefFromInt(4),
|
EndUnitID: shared.UnitRefFromInt(4),
|
||||||
ShortTitle: " ",
|
ShortTitle: " ",
|
||||||
PrimaryMode: "Narrative",
|
PrimaryMode: "Narrative",
|
||||||
MainParticipants: []string{"Aria"},
|
MainParticipants: []string{"Aria"},
|
||||||
@@ -467,8 +478,8 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
|
|||||||
name: "empty participant",
|
name: "empty participant",
|
||||||
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
||||||
{
|
{
|
||||||
StartUnitID: dnd.UnitRefFromInt(1),
|
StartUnitID: shared.UnitRefFromInt(1),
|
||||||
EndUnitID: dnd.UnitRefFromInt(4),
|
EndUnitID: shared.UnitRefFromInt(4),
|
||||||
ShortTitle: "Title",
|
ShortTitle: "Title",
|
||||||
PrimaryMode: "Narrative",
|
PrimaryMode: "Narrative",
|
||||||
MainParticipants: []string{"Aria", " "},
|
MainParticipants: []string{"Aria", " "},
|
||||||
@@ -484,7 +495,7 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
|
|||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
client := &fakeScenesLLMClient{response: tt.response}
|
client := &fakeScenesLLMClient{response: tt.response}
|
||||||
_, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
|
_, err := newChunker(t, client).Chunk(context.Background(), chunkRequest())
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("Chunk() error = nil, want error")
|
t.Fatal("Chunk() error = nil, want error")
|
||||||
}
|
}
|
||||||
@@ -498,7 +509,7 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
|
|||||||
func TestChunkWrapsLLMClientError(t *testing.T) {
|
func TestChunkWrapsLLMClientError(t *testing.T) {
|
||||||
client := &fakeScenesLLMClient{err: errors.New("provider unavailable")}
|
client := &fakeScenesLLMClient{err: errors.New("provider unavailable")}
|
||||||
|
|
||||||
_, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
|
_, err := newChunker(t, client).Chunk(context.Background(), chunkRequest())
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("Chunk() error = nil, want LLM error")
|
t.Fatal("Chunk() error = nil, want LLM error")
|
||||||
}
|
}
|
||||||
@@ -507,13 +518,12 @@ func TestChunkWrapsLLMClientError(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func chunkRequestWithClient(client contracts.StructuredLLMClient) contracts.ChunkRequest {
|
func chunkRequest() contracts.ChunkRequest {
|
||||||
return contracts.ChunkRequest{
|
return contracts.ChunkRequest{
|
||||||
Source: sceneSourceDocument(),
|
Source: sceneSourceDocument(),
|
||||||
SourceInput: sceneSourceInput(),
|
SourceInput: sceneSourceInput(),
|
||||||
SessionID: "session-123",
|
SessionID: "session-123",
|
||||||
LLMProfile: "profile-scenes",
|
LLMProfile: "profile-scenes",
|
||||||
LLMClient: client,
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -523,9 +533,13 @@ func sceneSourceInput() contracts.LLMInputMaterial {
|
|||||||
return contracts.NewLLMInputMaterial("source", "application/json", []byte(sceneTranscriptJSON), "sha256:transcript", "file:///session-alpha.json")
|
return contracts.NewLLMInputMaterial("source", "application/json", []byte(sceneTranscriptJSON), "sha256:transcript", "file:///session-alpha.json")
|
||||||
}
|
}
|
||||||
|
|
||||||
func requestWithOptions(req contracts.ChunkRequest) contracts.ChunkRequest {
|
func newChunker(t *testing.T, client contracts.StructuredLLMClient) *Chunker {
|
||||||
req.Options = map[string]any{"max_units": 2}
|
t.Helper()
|
||||||
return req
|
chunker, err := New(client, Options{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
return chunker
|
||||||
}
|
}
|
||||||
|
|
||||||
func sceneSourceDocument() *source.SourceDocument {
|
func sceneSourceDocument() *source.SourceDocument {
|
||||||
@@ -535,10 +549,10 @@ func sceneSourceDocument() *source.SourceDocument {
|
|||||||
Format: "application/vnd.seriatim.minimal+json",
|
Format: "application/vnd.seriatim.minimal+json",
|
||||||
Digest: "sha256:source",
|
Digest: "sha256:source",
|
||||||
Units: []source.SourceUnit{
|
Units: []source.SourceUnit{
|
||||||
{ID: 1, Kind: "transcript_segment", Text: "Aria asks whether the goblin will parley.", Metadata: map[string]any{"speaker": "Alice"}},
|
{ID: 1, Kind: "transcript_segment", Text: "Aria asks whether the goblin will parley.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 1}, Metadata: map[string]any{"speaker": "Alice"}},
|
||||||
{ID: 2, Kind: "transcript_segment", Text: "The goblin scout describes the gate guards."},
|
{ID: 2, Kind: "transcript_segment", Text: "The goblin scout describes the gate guards.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}},
|
||||||
{ID: 3, Kind: "transcript_segment", Text: "The guards rush out with blades drawn."},
|
{ID: 3, Kind: "transcript_segment", Text: "The guards rush out with blades drawn.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 3, EndUnitID: 3}},
|
||||||
{ID: 4, Kind: "transcript_segment", Text: "The party defeats the ambushers."},
|
{ID: 4, Kind: "transcript_segment", Text: "The party defeats the ambushers.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 4, EndUnitID: 4}},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -559,8 +573,8 @@ func replaceScenes(response chunkResponse, scenes []sceneResponse) chunkResponse
|
|||||||
|
|
||||||
func scene(startUnitID int, endUnitID int) sceneResponse {
|
func scene(startUnitID int, endUnitID int) sceneResponse {
|
||||||
return sceneResponse{
|
return sceneResponse{
|
||||||
StartUnitID: dnd.UnitRefFromInt(startUnitID),
|
StartUnitID: shared.UnitRefFromInt(startUnitID),
|
||||||
EndUnitID: dnd.UnitRefFromInt(endUnitID),
|
EndUnitID: shared.UnitRefFromInt(endUnitID),
|
||||||
ShortTitle: "Scene title",
|
ShortTitle: "Scene title",
|
||||||
PrimaryMode: "Narrative",
|
PrimaryMode: "Narrative",
|
||||||
MainParticipants: []string{"Aria"},
|
MainParticipants: []string{"Aria"},
|
||||||
@@ -570,7 +584,7 @@ func scene(startUnitID int, endUnitID int) sceneResponse {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func chunkIDs(chunks []contracts.SourceChunk) []string {
|
func chunkIDs(chunks []source.Chunk) []string {
|
||||||
ids := make([]string, 0, len(chunks))
|
ids := make([]string, 0, len(chunks))
|
||||||
for _, chunk := range chunks {
|
for _, chunk := range chunks {
|
||||||
ids = append(ids, chunk.ID)
|
ids = append(ids, chunk.ID)
|
||||||
30
internal/modules/dnd/chunk/scenes/model.go
Normal file
30
internal/modules/dnd/chunk/scenes/model.go
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
package scenes
|
||||||
|
|
||||||
|
import "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
|
|
||||||
|
type chunkResponse struct {
|
||||||
|
Scenes []sceneResponse `json:"scenes"`
|
||||||
|
BoundaryCaveats []string `json:"boundary_caveats"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type sceneResponse struct {
|
||||||
|
StartUnitID shared.UnitRef `json:"start_unit_id"`
|
||||||
|
EndUnitID shared.UnitRef `json:"end_unit_id"`
|
||||||
|
ShortTitle string `json:"short_title"`
|
||||||
|
PrimaryMode string `json:"primary_mode"`
|
||||||
|
MainParticipants []string `json:"main_participants"`
|
||||||
|
Summary string `json:"summary"`
|
||||||
|
BoundaryNote string `json:"boundary_note"`
|
||||||
|
BoundaryConfidence string `json:"boundary_confidence"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type normalizedScene struct {
|
||||||
|
StartUnitID int
|
||||||
|
EndUnitID int
|
||||||
|
ShortTitle string
|
||||||
|
PrimaryMode string
|
||||||
|
MainParticipants []string
|
||||||
|
Summary string
|
||||||
|
BoundaryNote string
|
||||||
|
BoundaryConfidence string
|
||||||
|
}
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user