19 Commits

Author SHA1 Message Date
e92bcfa74c Prepare the v0.4.0 release 2026-08-27 13:30:24 +00:00
e6b2ae88d2 Strengthen validation retries and checkpoint safety 2026-08-27 11:50:37 +00:00
b2e83bd6e7 Document feedback-aware validation retries 2026-08-27 01:35:23 +00:00
7f01c3e79e Add durable validation provenance 2026-08-27 01:27:25 +00:00
7b5f4ebd42 Apply validation retries to merge and normalize 2026-08-27 01:13:32 +00:00
4bb4582695 Integrate extraction validation retries 2026-08-27 01:06:17 +00:00
7d3434e5f6 Expose typed validation reports 2026-08-27 00:37:20 +00:00
29872b2e28 Integrate chunk validation retries 2026-08-27 00:35:37 +00:00
b487d93186 Add producer attempt state machine 2026-08-27 00:27:27 +00:00
8dd7a4324d Execute complete validator chains 2026-08-27 00:18:16 +00:00
04ba87e174 Enable correction-aware semantic reconciliation 2026-08-27 00:08:05 +00:00
a26d6ed042 Enable correction-aware downstream D&D extractors 2026-08-26 23:59:05 +00:00
759d32403f Enable correction-aware foundational D&D producers 2026-08-26 23:55:23 +00:00
1a7b20c766 Adapt corrections through PromptKit 2026-08-26 23:50:24 +00:00
85a5b52be7 Declare producer correction capability 2026-08-26 23:42:11 +00:00
9d0faabf61 Add validation policy configuration 2026-08-26 23:34:06 +00:00
1c3da3e869 Add feedback-aware correction contracts 2026-08-26 23:24:20 +00:00
9abd93502f Upgrade PromptKit to v0.9.0 2026-08-26 23:12:16 +00:00
c8a29a5fa2 Plan feedback-aware validation retries 2026-08-26 18:51:36 +00:00
176 changed files with 6073 additions and 2793 deletions

View File

@@ -83,9 +83,7 @@ entity identity, checkpoint contracts, or cross-request correlation. Changes
to shared protocol and policy assets must participate in the normal prompt,
schema, and checkpoint fingerprint mechanisms.
Acceptance of this decision does not imply that the shared mechanism or its
consumer migrations are implemented. The
[feature roadmap](../roadmap/semantic-reconciliation.md) owns target behavior
and status, and the
[implementation plan](../roadmap/implementation.md) owns delivery sequence
until the work is complete.
The shared mechanism and its initial D&D registry consumers are now
implemented. Current behavior is documented in
[Module Internals](../internal/modules.md#semantic-reconciliation) and
[D&D Module Internals](../internal/dnd.md#semantic-registry-reconciliation).

View File

@@ -0,0 +1,59 @@
# ADR-0014: Use feedback-aware validation retries
**Status:** Accepted
**Date:** 2026-08-26
## Context
Validation can identify a candidate defect after a producer has returned an
otherwise well-formed result. Retrying without the validator's deterministic,
bounded feedback wastes the useful diagnosis, while treating validator
execution failures as defects would ask a producer to repair conditions it
cannot control. The mechanism must preserve typed producer ownership,
checkpoint safety, and the repository's sensitive-data boundaries.
## Decision
The implementation will keep three independent budgets: the producer binding's
outer `retries` budget, PromptKit's structured-output repair budget, and each
validator's execution-retry budget. Validators will run sequentially in their
configured order and aggregate both rejections and execution failures before a
candidate disposition is selected.
A correction-capable producer will provide the exact single LLM response that
controlled its candidate using the `single_response_v1` protocol. A correction
attempt will reconstruct the ordinary request and append exactly two fresh
messages: that latest response as `assistant`, followed by one deterministic
aggregate correction request as `user`. Earlier turns will not accumulate.
Validator failures will not recurse into correction. Pipeline policy owns
terminal disposition, with field-by-field producer overrides over pipeline
defaults: structural failure and semantic rejection default to `fail_run`, and
validator execution failure defaults to `warn_continue`. Validators can report
facts and bounded corrective guidance, but never decide disposition.
Rejected and structurally invalid candidates will not advance. A candidate
allowed through after a validator execution failure will retain explicit
incomplete-validation provenance and will not be checkpointed. Exact response
and correction text remain attempt-local: they are excluded from ordinary
errors, warnings, manifests, receipts, caches, checkpoints, and default debug
summaries.
## Alternatives considered
- Retry every producer after any validation outcome. This conflates producer
defects with validator operational failures and wastes retry budget.
- Let validators decide whether to continue. This would distribute pipeline
disposition policy across validators and undermine consistent defaults.
- Reuse the full prior conversation. Accumulated turns introduce unbounded
prompt growth and make correction behavior depend on incidental history.
- Persist raw responses to simplify diagnosis. Raw model output and correction
guidance may be sensitive and do not belong in durable pipeline records.
## Consequences
The framework gains transport-neutral correction and candidate contracts,
producer capability checks, policy resolution, aggregated validation outcomes,
and conservative checkpoint handling. Prompt construction remains inside the
LLM adapter, while modules remain responsible for accurately exposing the
single response that directly controlled a candidate.

View File

@@ -132,7 +132,7 @@ model: example-model
Keep credentials out of the local-backend object. A PromptKit profile may name
its credential environment variable through `api_key_env`; set that variable
only in the run environment. PromptKit owns the
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/formats.md),
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md),
including `base_profile` inheritance. Notarius passes profiles through without
merging them. Filesystem profiles cannot express PromptKit's in-memory
`APIKeyRequired` setting; an unset `api_key_env` is optional and may reach the
@@ -223,6 +223,7 @@ pipelines:
| --- | --- | --- | --- |
| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. An explicitly present blank value is invalid. |
| **structured_output_repair_attempts** | integer | prompt-owned (1 in maintained production prompts) | Optional structural-repair limit from 0 through 3 for selected LLM-backed bindings and validators. Omission leaves the prompt's declared policy in control; explicit 0 disables structural repair at that scope. |
| **validation_policy** | object | see below | Optional terminal policy defaults for producer validation. Its fields inherit independently into chunk, extract, merge, and normalize bindings. |
| **input** | module binding | none | Required. |
| **chunk** | module binding | **generic** | Optional. |
| **output** | module binding | **json** | Optional. |
@@ -251,6 +252,33 @@ explicit `null` and non-integer values are invalid. An explicit value on a
deterministic binding or validator is invalid, while a pipeline value simply
does not apply to deterministic selections.
`validation_policy` controls terminal disposition for one complete producer
attempt and validator chain. It may appear on a pipeline or a **chunk**,
**extract**, **merge**, or **normalize** module binding; input, output, and
validator bindings reject it. Every field is optional and resolves in binding,
pipeline, then application-default order:
| Field | Values | Default |
| --- | --- | --- |
| **producer_structural_failure** | **fail_run**, **reject_output** | **fail_run** |
| **semantic_rejection** | **fail_run**, **reject_output** | **fail_run** |
| **validator_failure** | **warn_continue**, **fail_run** | **warn_continue** |
The policy object and its fields must be non-null, and unknown fields are
rejected. A deterministic producer may not explicitly set
**producer_structural_failure** on its binding, although a pipeline-level
default remains valid for pipelines that include LLM-backed producers.
After the producer binding's retry budget is exhausted, an invalid structured
response uses **producer_structural_failure**. One or more semantic validator
rejections use **semantic_rejection**; rejection takes precedence over an
exhausted validator failure or skip. With no rejection, an exhausted validator
failure or skip uses **validator_failure**. `reject_output` records the
terminal rejection without advancing that candidate. `warn_continue` is valid
only for validator execution failure: it advances a structurally valid,
otherwise unrejected result with incomplete-validation provenance and without
making it reusable checkpoint state.
A lane has these fields:
| Field | Type | Default | Rules |
@@ -289,7 +317,8 @@ extract:
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
| **structured_output_repair_attempts** | integer | pipeline or prompt-owned (1 in maintained production prompts) | Optional structural-repair limit from 0 through 3 for an LLM-backed binding. It overrides the pipeline value; explicit 0 disables structural repair. |
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
| **validation_policy** | object | pipeline or application defaults | Optional field-by-field terminal-policy override for a chunk, extract, merge, or normalize binding. |
| **retries** | integer | 0 | Non-negative additional complete producer attempts for chunk, extract, merge, and normalize bindings. This single budget covers operational errors, invalid structured output, module-requested normalization retry, and semantic correction. |
| **options** | object | none | Must satisfy the selected module. |
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
| **validators** | list | production chain | Valid only on chunk, extract, merge, and normalize bindings. |
@@ -297,11 +326,23 @@ extract:
Omitting **validators** uses the registered chain. **validators: []** selects
an empty chain; a non-empty list replaces the chain in the listed order.
Validator bindings accept only **module**, **llm_profile**,
**structured_output_repair_attempts**, and **options**. They reject
**references**, **retries**, and nested **validators**. Deterministic validators
reject explicit **llm_profile** and **structured_output_repair_attempts**.
**structured_output_repair_attempts**, **retries**, and **options**. Their
**retries** value is a non-negative additional validator-execution budget and
is valid only when the selected validator is LLM-backed. A validator retry
rechecks the same immutable candidate; it never regenerates the producer.
They reject
**validation_policy**, **references**, and nested **validators**. Deterministic
validators reject explicit **llm_profile** and
**structured_output_repair_attempts**.
Deterministic module bindings also reject those explicit fields.
An LLM-backed chunk, extract, merge, or normalize producer with both a
non-empty validator chain and positive **retries** must declare the supported
single-response correction capability. Preparation rejects a configuration
that could require semantic correction from a producer that cannot provide an
exact prior response. A deterministic producer, or an LLM attempt that did
not make a model call, cannot consume a semantic retry after rejection.
The **json** output module accepts optional **include_chunk_map** and
**evidence_context** settings:

View File

@@ -101,6 +101,12 @@ root. Do not scan the output root for its newest directory, guess a run ID, or
construct a bundle path. Resolve `index_file` beneath `output_directory` and
reject an absolute logical path or any result that escapes the bundle root.
The complete configuration uses the application validation defaults. A caller
that requires fully validated D&D artifacts must also require receipt
`validation_status: approved`; a successful `incomplete` result reflects the
configured validator-failure continuation policy and carries its bounded
validator provenance in `validation_summaries`.
Read `index.json` and locate each requested lane in `output_files` by its exact
`lane_id`. Do not guess a lane filename. Before decoding a payload:

View File

@@ -69,6 +69,13 @@ lanes. The caller decides which lane IDs are required for its own work and
which are optional; it should make that decision explicitly rather than infer
failure from the receipt counts alone.
When complete validation is required, also require receipt
`validation_status: approved` and inspect `validation_summaries`. A successful
run with `validation_status: incomplete` contains a structurally valid result
that advanced after validator execution could not complete under the configured
`warn_continue` policy. It is not reusable checkpoint state and should not be
silently treated as fully reviewed by the caller.
## Preserve Provenance And Handle Data Carefully
Keep the receipt with the published `manifest.json`, and retain

View File

@@ -92,7 +92,7 @@ group into the following externally observable summaries:
| Run identity and result | `run_id`, `pipeline_id`, `pipeline_digest`, `schema_version`, `validation_status`, `started_at`, `completed_at` |
| Resolved components | `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, `output_encoder`, `artifact_lanes`, `validator_chains`, `module_metadata` |
| Source and references | `source_digests`, `references` |
| Published result summaries | `normalized_outputs`, `rejected_outputs` |
| Published result summaries | `normalized_outputs`, `rejected_outputs`, `validation_summaries` |
| Execution summaries | `chunk_plan`, `checkpoint_decisions`, `llm_profiles`, `metadata` |
`references` records provenance such as the target, slot, origin, digest,
@@ -102,6 +102,16 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is
provenance for the plan used by this run; cache records, debug artifacts, and
other operational state are not published as bundle files.
Each `validation_summaries` entry is a bounded outcome for one producer result.
It has required `status`, `producer_attempt_count`, and `terminal_action`;
the stage and affected step, lane, module, or chunk identity are present when
applicable. `status` is `complete`, `rejected`, or `incomplete`.
`rejecting_validators`, `reason_codes`, and `incomplete_validators` preserve
configured validator order and omit later duplicates. Entries contain no raw
candidate response, correction guidance, validator diagnostic message, or
artifact payload. The same shape may appear as `validation` on an affected
rejection entry.
When present, `metadata.session_id` is the effective non-secret routing
correlation identifier used for the run. It can be visible to providers and is
not a substitute for a cache or checkpoint identity. Its generation and
@@ -127,7 +137,10 @@ distinct even when their profile, provider, and model are otherwise equal.
`rejected.json` is always an object with a `rejected` array. Each entry has
required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`,
`chunk_index`, `validator_name`, `reason_code`, `attempt_count`, and
`diagnostic_artifact_path` are present only when applicable.
`diagnostic_artifact_path` are present only when applicable. An entry may also
contain the bounded `validation` summary described above; the existing singular
validator and reason fields remain the first configured rejection for
compatibility.
`warnings.json` is always an object with a `warnings` array. Each warning has
`reason_code` and `message`; `scope` is optional. Both arrays are empty when

View File

@@ -1,11 +1,11 @@
# PromptKit Integration
Notarius pins
[`gitea.maximumdirect.net/eric/promptkit` v0.8.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0)
[`gitea.maximumdirect.net/eric/promptkit` v0.9.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0)
as its in-process prompt engine. The upstream
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/consumers/pkg-promptkit.md)
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/consumers/pkg-promptkit.md)
owns the public engine API, and the upstream
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/formats.md)
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md)
owns prompt, profile, and schema file contracts.
## Supported Boundary
@@ -15,7 +15,8 @@ Notarius relies on the root `promptkit` package to:
- construct an `Engine` with filesystem-backed prompt, schema, and optional
operator and application-fallback profile sources;
- prepare one frozen execution from a `RunRequest` with named inline artifacts,
variables, a direct session ID, prompt identity, and profile selection, then
variables, a direct session ID, prompt identity, profile selection, and
optional appended rendered messages, then
record credential-redacted details and run that exact execution;
- return rendered debug material, validated structured output, selected
profile, backend, effective model metadata, and token usage;
@@ -26,7 +27,7 @@ Notarius relies on the root `promptkit` package to:
admission exhaustion through `ErrCapacityExceeded`.
The pinned
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/backends.go)
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/backends.go)
owns the registration and backend-capacity contract.
For one completion, the adapter calls `PrepareExecution`, takes a
@@ -52,7 +53,7 @@ Notarius sends one stable effective session through PromptKit's direct session
field, which is authoritative for provider session behavior. It also retains
the same value as the `session_id` prompt variable for maintained prompt
compatibility. The generated identifier is 76 ASCII characters, within
PromptKit v0.8.0's 256-code-point session limit. Session IDs are non-secret
PromptKit v0.9.0's 256-code-point session limit. Session IDs are non-secret
correlation identifiers and may be exposed to providers and provider
observability. The CLI contract owns generation and override behavior.
@@ -85,12 +86,38 @@ configuration and deployment workflow are defined in
[Operations](../operations.md#promptkit-profile-deployment).
PromptKit owns `base_profile` resolution under its
[pinned format rules](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/formats.md).
[pinned format rules](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md).
Notarius records the selected leaf identity and resolved target without parsing
or merging inheritance. An unset filesystem `api_key_env` is optional and may
reach the provider without authorization, which can result in a 401 or 403.
Notarius supports this boundary against PromptKit v0.8.0. Its fallback source,
PromptKit v0.9.0 accepts only the `developer`, `system`, `user`, and
`assistant` text-chat roles after normalizing case and surrounding whitespace.
Maintained Notarius prompt definitions use only `system` and `user`.
For application-owned semantic correction, Notarius uses PromptKit v0.9.0's
`RunRequest.AppendedMessages` after the ordinary rendered prompt. It supplies
exactly two messages in order: the latest validated producer response with
role `assistant`, then deterministic validation guidance with role `user`.
It never exposes a general caller-selected role API, accumulates earlier
correction turns, or changes the ordinary prompt prefix. Ordinary requests
leave appended messages unset.
PromptKit preserves supplied content but does not own Notarius's correction
bounds. Notarius rejects invalid UTF-8, blank, or oversized assistant material
(at most 1 MiB), guidance (at most 64 KiB), and combined content (at most
1,114,112 bytes) before preparing the request. The transport-neutral
application contract owns defensive copying and these limits. Default request
and terminal summaries retain only safe counts, digests, identities, and usage;
complete appended messages remain limited to the explicitly requested detailed
debug trace.
PromptKit now obtains its maintained OpenRouter and Rakestrawhome backend and
profile catalogs from independently versioned transitive modules. Notarius
does not import or register either catalog; PromptKit retains catalog source,
identity, precedence, credential, and capacity ownership.
Notarius supports this boundary against PromptKit v0.9.0. Its fallback source,
prepared-execution, inspection, and typed capacity APIs are used as public
upstream contracts; other PromptKit APIs or file-format behavior are not
implicitly supported. A dependency upgrade requires reviewing the adapter,

View File

@@ -22,11 +22,15 @@ The current schema version is `notarius.run-result.v1`.
| `rejected_output_count` | Yes | Number of recorded rejected outputs. |
| `warning_count` | Yes | Number of final run warnings. |
| `validation_status` | Yes | The final run manifest validation status. |
| `validation_summaries` | No | Bounded per-producer validation outcomes; present when producer work ran. |
| `debug_directory` | No | Absolute path to the run-specific debug bundle when requested debug capture completed. |
For the production `json` output module, `index_file` is present only when the
completed run returned exactly one logical output file named `index.json`.
For another output module, its absence does not indicate a failed run.
`validation_status` is `approved`, `rejected`, or `incomplete`; `incomplete`
means one or more otherwise accepted results advanced under validator-failure
`warn_continue`.
```json
{
@@ -38,7 +42,17 @@ For another output module, its absence does not indicate a failed run.
"normalized_output_count": 6,
"rejected_output_count": 2,
"warning_count": 1,
"validation_status": "rejected"
"validation_status": "incomplete",
"validation_summaries": [
{
"stage": "extract",
"lane_id": "spells",
"status": "incomplete",
"incomplete_validators": ["dnd/spells/source_refs"],
"producer_attempt_count": 1,
"terminal_action": "warn_continue"
}
]
}
```
@@ -48,8 +62,11 @@ For another output module, its absence does not indicate a failed run.
paths. They identify the paths used by Notarius and do not resolve symlinks.
`output_directory` is the run-specific bundle, not the configured output root.
The receipt is a summary and discovery document. It does not contain lane
descriptors, payloads, manifest data, rejections, warnings, or file contents.
The receipt is a summary and discovery document. Its optional validation
summaries contain only stable status, identity, validator names, reason codes,
attempt counts, and terminal actions. It does not contain lane descriptors,
payloads, manifest payloads, rejection messages, warnings, raw model responses,
correction guidance, or file contents.
For the production JSON output, resolve `index_file` beneath
`output_directory`, reject path escapes, and use the
[Published JSON Output contract](json-output.md) to discover logical files and

View File

@@ -74,10 +74,19 @@ profile-free, and no second inheritance decision occurs during execution. The
public field definitions and precedence are owned by
[Configuration](../config.md#pipelines).
The resolver retains configured `validation_policy` overrides and derives one
detached concrete terminal policy for the chunk producer and every lane's
extract, merge, and normalize producers. That field-by-field inheritance is
complete before preparation, and the effective values contribute to pipeline
and checkpoint identity; execution does not interpret configuration defaults.
The framework resolver supplies defaults, selects lanes, resolves validator
chains, checks registered module and artifact compatibility, validates module
options, and returns the fixed ordered pipeline shape. The resulting
**EffectiveConfig** retains the selected ID, requested selection and reference
options, and returns the fixed ordered pipeline shape. Positive validator retry
budgets require an LLM-backed selected validator; deterministic validators are
rejected during resolution. Eligible LLM-backed producer specifications also
contribute their declared correction protocol to the resolved metadata. The
resulting **EffectiveConfig** retains the selected ID, requested selection and reference
changes, a clone of the input configuration, and the resolved pipeline.
Callers may therefore retain or modify their input slices and maps without
changing the resolved result, and later consumers cannot mutate the original
@@ -94,8 +103,8 @@ runtime error class described in the [CLI reference](../cli.md#output-streams-an
The framework assigns the resolved pipeline a deterministic SHA-256 digest
after defaults, lane selection, module bindings, reference bindings, validator
chains, effective LLM profiles, and artifact schema identity have been
resolved. The digest excludes
chains, selected correction protocols, effective LLM profiles, and artifact
schema identity have been resolved. The digest excludes
its own stored value. It identifies resolved composition rather than raw YAML
bytes, a debug payload, or all runtime state. The CLI records it as invocation
provenance before execution; cache and checkpoint identity have additional

View File

@@ -33,7 +33,9 @@ typed builder. Scene chunking, every extractor, and NPC, location, and item-regi
normalization are registered as `llm_backed`; the remaining current D&D mergers
and normalizers are `deterministic`. The metadata is available to catalog inspection and
resolved-pipeline debug data and determines which selected bindings inherit the
pipeline profile. Configuration remains the canonical owner of the exact keys,
pipeline profile. The registry normalizers use `single_response_v1`, forwarding
corrections to their reconciliation completion and retaining the accepted raw
proposal only as an owned model candidate. Configuration remains the canonical owner of the exact keys,
profile precedence, and validator order.
Private structured-LLM response schemas are deliberately minimal. They reject
@@ -42,6 +44,13 @@ unknown fields, while preserving semantic candidates for deterministic
validation. Do not promote a private response envelope into a durable schema;
the contracts above define durable data.
A D&D producer that declares `single_response_v1` forwards any supplied
semantic correction to its structured completion and returns an owned copy of
that completion's exact validated raw response as its model candidate. It does
not serialize normalized artifacts to create that candidate, so deterministic
identity, evidence, warning, and durable-schema behavior remains separate from
the model transport material.
## Prompt Construction
D&D LLM-facing content lives beneath `assets/dnd/`. Each module contributes a
@@ -125,6 +134,13 @@ relatedness validators report advisory evidence concerns. The configured order
is documented in
[Configuration](../config.md#production-validator-keys-and-default-chains).
Every D&D rejection describes the correction in transcript-grounded domain
terms, using contextual names, artifact fields, and source segment ranges when
useful. The guidance must not ask the model to reproduce durable entity IDs,
hashes, validator module keys, or reason codes. Those identifiers remain in
ordinary validation provenance; only the actionable semantic guidance is
eligible for the correction prompt.
Enemy-event extraction additionally rejects a second `engaged` observation for
the same comparison identity within one scene-scoped result. Normalization may
combine results from distinct scenes, so it intentionally does not apply that

View File

@@ -42,6 +42,22 @@ observability. The adapter returns PromptKits validated raw bytes rather than
re-encoding the decoded target. An empty optional material is represented as
one space so its named input is retained by PromptKit.
When a request includes semantic correction material, the adapter validates and
defensively copies it before preparation, then appends exactly two messages
after the ordinarily rendered prompt: the prior response as an assistant
message and the correction guidance as a user message. Requests without a
correction do not add messages or introduce caller roles. Ordinary request
summaries record correction byte counts and digests only; complete messages are
available solely in an explicitly requested debug trace.
The adapter leaves the ordinary rendered message prefix, named inputs,
variables, session, profile, execution overrides, prepared-execution path, and
PromptKit repair policy unchanged for a corrected request. It never imports a
PromptKit message type into a module or pipeline contract. PromptKit reports
actual structural repair count and cumulative token usage per completion; the
pipeline's safe terminal debug record projects those values without copying
message content.
Client construction may also receive a run-wide reasoning-effort override from
the CLI factory boundary. The adapter copies the caller-owned pointer and
creates a fresh PromptKit execution override for each request: a nil pointer
@@ -229,10 +245,11 @@ or corrective call remain provider-neutral operational errors with the same
redaction boundary.
Structural repair does not replace pipeline retry behavior: a binding's
configured retry count reruns its complete stage attempt after an error or
rejection. The pipeline owns attempt lifecycle, validation chains, and retry
diagnostics; see [Pipeline Internals](pipeline.md#validation-retries-and-output)
and the [binding reference](../config.md#module-bindings-and-validators).
configured retry count reruns its complete stage attempt after an operational
or structural error, module-requested retry, or actionable semantic rejection.
The pipeline owns attempt lifecycle, validation chains, and retry diagnostics;
see [Pipeline Internals](pipeline.md#validation-retries-and-output) and the
[binding reference](../config.md#module-bindings-and-validators).
## Timeout Ownership
@@ -262,6 +279,13 @@ surfaced; when the completion already failed, its call error remains the
result. Debug-bundle location, retention, and handling are operational concerns
documented in [Operations](../operations.md#debug-bundles).
The attempt-terminal summary is a separate safe trace record: it contains
attempt kinds, validator outcome counts and reason codes, effective policy,
terminal action, and repair/usage references. It excludes raw assistant
responses and correction text. Those values can appear only in the explicitly
requested detailed prompt and response artifacts, which require sensitive-data
handling.
Run manifests receive selected profile summaries, including optional effective
backend and reasoning provenance, and component identities—not prompt, schema,
source, reference, or response content. The published field semantics belong

View File

@@ -22,6 +22,15 @@ to bindings whose declared execution class is `llm_backed` and rejects a
binding-specific profile on a deterministic module. The user-facing precedence
contract belongs in [Configuration](../config.md#pipelines).
An eligible LLM-backed chunk, extract, merge, or normalize producer may also
declare correction protocol `single_response_v1`. That declaration is a
promise that the implementation accepts one attempt-local semantic correction
and returns an owned copy of the exact one model response that directly
controlled the candidate. It must forward correction only to its structured
completion request; it must not manufacture prior-response material by
serializing a normalized artifact or expose opaque application IDs. Input,
output, validator, and deterministic specs cannot declare the protocol.
Implementations that accept options must provide both an option validator and
a builder. The validator is used while resolving configuration; the builder
decodes the same options and constructs the implementation from the prepared
@@ -36,6 +45,15 @@ they need, register each leaf implementation, and add any family-owned assets
or default validator chains. They return contextual errors so production
composition fails at startup rather than at the first run.
A validator that returns a completed rejection must supply two separate
bounded values: a stable `ReasonCode` for provenance and actionable
`CorrectionGuidance` for the producer. Guidance identifies the semantic defect
and the constraints on one complete corrected replacement. It must not contain
validator keys, diagnostic paths, opaque application IDs, or other internal
identifiers. An operator-facing `Message` may explain the same event, but the
framework never copies it into a model request. Missing or invalid guidance is
a validator contract failure.
An artifact family can register an optional typed evidence projector alongside
its codec. The projector returns defensive copies of the artifact's direct
generic source references and must use the codec's exact Go type. It does not
@@ -91,6 +109,12 @@ combined-material bound preserves the deterministic result under the family's
fallback policy. Provider, transport, cancellation, and context-construction
failures remain execution errors.
When the engine actually makes a proposal call, its typed result carries the
owned exact proposal response under the same correction contract as other
eligible producers. Deterministic skip, limit, and fallback outcomes carry no
model candidate, so a later rejection applies terminal policy without spending
an ineffective semantic retry.
The core supplies a conservative generic prompt and the single private
response schema. A domain prompt may substitute its semantic instructions but
mounts the core-owned protocol and candidate/transcript presentation assets.
@@ -111,9 +135,13 @@ its domain prompt.
3. Implement strict option decoding, construction, and the typed stage
interface. Preserve caller ownership: do not retain mutable request data
and return defensive copies where an implementation exposes stored data.
If declaring correction capability, forward the request correction and
retain only the exact validated response that controlled the result.
4. Register the module through its typed registry helper and add it to the
owning family registrar. Add a default validator chain only when that
family owns the behavior; otherwise require an explicit compatible chain.
Every rejection path in a validator must provide actionable correction
guidance while retaining its stable internal reason code.
5. Update the selectable-key and chain reference in
[Configuration](../config.md#production-module-keys), the applicable
integration contract, and focused tests. Keep the configuration document

View File

@@ -32,8 +32,14 @@ Resolution turns a configured pipeline profile into a **ResolvedPipeline**.
It normalizes the pipeline and lane identities, applies stage defaults, selects
requested lanes where that is supported, resolves validator chains, checks
module capabilities and typed artifact compatibility, validates options, and
assigns a deterministic resolved-composition digest. The resolved pipeline
contains bindings and declared reference targets, not external reference bytes.
assigns a deterministic resolved-composition digest. A correction protocol is
selected from each eligible LLM-backed producer specification and becomes part
of that resolved identity; only `single_response_v1` is currently supported.
Preparation rejects an LLM-backed producer that combines a non-empty validator
chain with positive producer retries unless it declares that protocol. Producers
without validators or without retries remain valid without correction support.
The resolved pipeline contains bindings and declared reference targets, not
external reference bytes.
After selection, the resolver applies command, binding, and pipeline profile
precedence to LLM-backed bindings and validators only; prompt defaults remain
an empty resolved binding profile. It resolves structural output repair
@@ -59,9 +65,11 @@ remains declared but has no bytes until its producing step completes.
Preparation is the construction boundary. It validates the resolved shape and
registry set, clones the resolved data, then constructs the input adapter,
chunker, stage-local validators, every typed lane, and output encoder. Each
registered builder receives its own cloned build request immediately before its
module-owned code runs. Preparation also collects stable checkpoint
chunker, stage-local validators, every typed lane, and output encoder. The
prepared producer metadata preserves each selected correction protocol, and
the resolved digest carrying that metadata participates in checkpoint identity.
Each registered builder receives its own cloned build request immediately
before its module-owned code runs. Preparation also collects stable checkpoint
fingerprints. Missing registrations, incompatible typed entries, nil
implementations, and constructor failures are reported before source parsing
or any stage operation begins.
@@ -128,14 +136,45 @@ their target: chunks, codec-decoded typed candidates, or serialized codec
bytes. Each typed validator receives a newly decoded value from the one
candidate serialization for that attempt, while serialized validators receive
separately owned representation bytes and schema metadata. They may approve,
approve with warnings, reject, or fail. A rejection is an ordinary pipeline
result; a validator error is a framework error.
approve with warnings, reject, fail, or be skipped when a runtime prerequisite
is unavailable. The shared executor settles every configured validator in
order. A failed LLM-backed validator retries only itself against the same
immutable candidate; it does not regenerate the producer or alter the
validator request. Rejections stop that validator, while other configured
validators still run. The executor retains ordered results, bounded
deduplicated correction guidance from rejections, and only the final exhausted
failure outcome for each validator. The correction builder keeps first
occurrence order, omits internal reason codes, validator names, and operator
messages, and requests one complete replacement. Missing guidance or an
oversized aggregate is a framework contract error; guidance is never inferred
or truncated.
The runner applies the binding's retry policy around a stage operation and its
complete validation chain. It preserves warnings only from the final accepted
or rejected attempt. Cancellation stops retries. Normalizer-specific retry
directives consume this same budget and validate any final safe fallback through
the normalizer chain.
or rejected attempt, plus one fixed warning per validator whose execution
budget was exhausted under `warn_continue`. Cancellation stops retries.
Normalizer-specific retry directives consume this same budget and validate any
final safe fallback through the normalizer chain.
The artifact-neutral producer-attempt state machine owns that shared budget,
attempt provenance, semantic-correction material, and terminal-policy
selection. It accepts producer and complete-validation closures, so artifact
materialization, cache handling, checkpoints, and debug output stay at the
operation boundary. It distinguishes operational, structural, module-requested,
and semantic retries. A semantic retry is available only for a valid latest
`single_response_v1` candidate; a deterministic or no-model rejection instead
settles the semantic policy immediately. Structural-output errors alone use the
structural policy, and validation failure without rejection settles the
validator-failure policy without regenerating the producer.
Chunk planning uses this state machine for generated plans. A rejected or
validation-incomplete automatic cache hit is not model material and therefore
falls through to a fresh initial generation at producer attempt one; it neither
receives a correction, consumes retry budget, promotes cached-candidate
warnings, nor overwrites the stored record. An incomplete cache validation
under `fail_run` terminates instead. Only a newly generated, completely
validated plan is published to the chunk-plan store. Rejected plans never
advance, and validation-incomplete plans remain unpublishable.
After terminal lane work, the runner assembles manifest provenance, normalized
artifacts, rejections, warnings, and an optional accepted chunk map. When an
@@ -149,6 +188,15 @@ The CLI publishes those files only after the runner returns without a framework
error. Logical file names and schemas are defined by the [output integration
contracts](../integrations/).
For every completed producer disposition, the runner projects one bounded
validation summary to the manifest, the affected rejection when present, and
the CLI result receipt. The summary records status, configured-order rejecting
validators and reason codes, incomplete validators, producer-attempt count,
and terminal action. It contains no operator message, correction guidance, or
model response. `complete`, `rejected`, and `incomplete` describe the final
candidate disposition; a run-level `incomplete` status indicates at least one
current-run output advanced under `warn_continue`.
## Checkpoint And Debug Hooks
The runner receives checkpoint and debug interfaces rather than roots. It
@@ -158,6 +206,20 @@ handoff. Generated-reference dependencies participate in checkpoint decisions.
Selective recomputation can require a canonical accepted normalized predecessor
before a dependent lane starts.
The runner writes successful checkpoint artifacts only after complete accepted
validation. Chunk plans follow the same rule for publication. A rejection,
invalid structured response, or incomplete validation is never reusable state;
the current run may still hand off an otherwise valid `warn_continue` result
according to its terminal policy. The runner carries private reuse eligibility
through extract, merge, normalize, and generated-reference handoff. Any stage
derived from incomplete validation skips both checkpoint lookup and all
checkpoint publication even when that stage's own validation completes.
External references and fully validated generated references remain eligible.
Attempt debug records retain safe kind,
validator, repair-usage, policy, and terminal-decision provenance. Full
assistant and correction content remains confined to the requested detailed
LLM trace.
Debug recording is attempt-scoped and application-owned. A failure to persist
required debug data is a framework error. State roots, persistence, reason-code
meanings, resume, and cleanup are intentionally owned by

View File

@@ -109,6 +109,33 @@ On success, the command reports the output bundle path. A warning-bearing run
still succeeds and reports its warning count on standard error. Errors and
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).
## Validation Retries And Terminal Outcomes
Each producer binding has one outer **retries** budget. It covers complete
producer attempts for operational failures, invalid structured output,
normalizer fallback retry, and semantic correction. It is independent from
PromptKit's structural-repair calls inside one completion and from an
LLM-backed validator's own retry budget. A semantic correction rebuilds the
ordinary producer request and supplies only the latest rejected model response
plus aggregated validator guidance; it is not a conversation replay.
After the applicable budgets are exhausted, the resolved
[`validation_policy`](config.md#pipelines) determines the result. Structural
failure and semantic rejection normally fail the run; an explicit
`reject_output` records a rejection and allows unrelated work to finish. A
validator execution failure normally uses `warn_continue`, which keeps an
otherwise accepted result in the current run with `incomplete` validation
provenance. It emits one bounded warning for every validator whose execution
budget was exhausted. A corrected result that later passes validation does not
retain abandoned-attempt warnings.
Treat a successful process exit as a completed run, not as proof that every
candidate was fully validated. Inspect the receipt's `validation_status`,
`validation_summaries`, rejection count, and warning count when an orchestrator
requires complete validation. The durable fields and their meanings are owned
by the [run-result receipt](integrations/run-result.md) and
[published JSON output contract](integrations/json-output.md).
## Output Bundles
Each successful run receives a generated safe run identifier and writes beneath:
@@ -158,7 +185,9 @@ The configured cache mode controls one invocation:
A reused plan is still materialized and validated against the current source.
If a prior plan no longer gives acceptable results, use a refresh run rather
than editing cache files. Deleting a plan is recoverable but can repeat costly
chunking work.
chunking work. A plan accepted only under incomplete validation is not
published, and a rejected cache hit falls through to ordinary generation rather
than becoming a correction candidate.
## Checkpoint Recording, Resume, And Recompute
@@ -183,6 +212,12 @@ Reasoning-effort inheritance, replacement, and explicit clearing are distinct
runtime identities, so checkpoints created under one state are not reused by
either of the others.
Only accepted, completely validated chunk, extract, merge, and normalize
results are checkpointed for reuse. Rejected, structurally invalid, and
validation-incomplete producer results remain non-reusable, even when a
`warn_continue` result advanced during its original run. A resumed invocation
therefore reruns that producer rather than treating degraded state as accepted.
Checkpoint state is confined below an identity-specific path:
~~~
@@ -237,13 +272,16 @@ Only a [debug-enabled run](cli.md#run) creates a bundle:
~~~
The summary contains redacted invocation and resolution information plus run,
warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace
contains allowlisted application diagnostic records and can include source or
derived application data. Neither surface is a cache input. Do not treat a
debug bundle as safe to share merely because its configuration summary is
redacted. Invocation metadata omits reasoning effort when it is inherited,
records the replacement value when one is supplied, and records an empty value
when inherited reasoning was explicitly cleared.
warning, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt
terminal records contain bounded attempt kinds, validator outcomes, policy,
decision, PromptKit repair count, and usage; they do not contain assistant
responses or complete correction messages. The trace contains allowlisted
application diagnostic records and can include source, model, and correction
content. Neither surface is a cache input. Do not treat a debug bundle as safe
to share merely because its configuration summary is redacted. Invocation
metadata omits reasoning effort when it is inherited, records the replacement
value when one is supplied, and records an empty value when inherited reasoning
was explicitly cleared.
Notarius never creates debug state without an explicit request and never
automatically deletes a requested bundle. If allocation succeeds, the command

View File

@@ -167,18 +167,29 @@ starting, waits for started work, and prevents output encoding.
## Validation
Validation is a framework-managed boundary around outputs from chunk, extract,
merge, and normalize stages. Validators receive immutable stage output
and make an explicit whole-output decision: approve, approve with warnings, or
reject.
merge, and normalize stages. Validators receive immutable stage output and
make an explicit whole-output decision: approve, approve with warnings,
reject, fail, or skip when a runtime prerequisite is unavailable.
Typed artifact validators receive the domain value directly. Chunk validators
receive source-zone chunks, while serialized validators receive immutable
representation bytes and declared schema metadata. A validator registered for
one target or artifact kind cannot satisfy an incompatible selection.
Rejection is a recorded pipeline outcome, not a framework execution error.
Validator execution failures are framework errors. Rejected output does not
advance to the next stage.
The framework runs every applicable validator sequentially in configured order.
It aggregates rejections, exhausted validator failures, and skips before the
producer policy chooses a disposition. A completed rejection never advances.
With no rejection, an exhausted validator failure may fail the run or, under
the configured `warn_continue` policy, advance a structurally valid candidate
with explicit incomplete-validation provenance. Validators report findings;
they do not choose candidate disposition.
A completed rejection supplies a stable reason code for internal provenance
and bounded actionable correction guidance for the candidate producer. Reason
codes, validator keys, and operator-facing messages remain diagnostic data;
they are not model instructions. The framework constructs model-facing retry
text only from the semantic guidance and fails the contract rather than
inventing or truncating missing guidance.
Default validator chains are production composition policy and are registered
centrally by stage and module. Configuration may replace a stage-local default,
@@ -199,6 +210,15 @@ PromptKit owns bounded structural correction within one structured completion.
Notarius owns outer stage attempts, semantic validation, and acceptance policy;
the two budgets must remain separate.
An LLM-backed producer can participate in semantic correction only when it
declares `single_response_v1` and returns the exact one response that directly
controlled its candidate. On an actionable rejection, the framework rebuilds
the ordinary request and appends only the latest defective response as an
`assistant` message plus one aggregated `user` correction message. This is a
fresh replacement request, not a growing conversation. The retry budgets,
terminal policy, and sensitive-data rationale are recorded in
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md).
When a model selects an application entity, callers must supply a contextual
selection and deterministically attach the opaque application identity whenever
the selection resolves exactly. Models do not receive or reproduce opaque
@@ -233,7 +253,8 @@ invalid or incompatible.
Run manifests record enough resolved pipeline, module, source, reference, and
LLM provenance to make a run auditable after configuration changes. Manifests
record identities and summaries rather than secret or large payload content.
record identities and bounded validation summaries rather than secret, raw
model, correction, or large payload content.
## State, Output, And Safety
@@ -253,12 +274,23 @@ an invocation that explicitly requests resume. Debug is never a cache input and
is never created without an explicit request. Pipeline modules receive
collaborator interfaces and never physical roots.
Only accepted, completely validated producer output is reusable checkpoint or
chunk-plan state. Rejected, structurally invalid, and validation-incomplete
results cannot become cache or checkpoint inputs, even when a
`warn_continue` result is allowed to advance in the current run. This
ineligibility follows derived merge and normalize results and generated
references for the remainder of the run: current-run handoff remains allowed,
but no dependent cache or checkpoint may be loaded or published.
Writes are atomic where practical. Paths for writes, moves, overwrites, and
deletion must be narrow and explicit. Notarius never automatically deletes
output or requested debug bundles; cache cleanup is explicit and recoverable.
Secrets must not appear in errors, logs, output, cache, debug summaries,
traces, manifests, documentation, examples, or redacted configuration. Debug
traces, manifests, documentation, examples, or redacted configuration. Raw
assistant responses and complete correction messages are attempt-local and are
excluded from ordinary durable records and summaries; the requested detailed
debug trace is the sole diagnostic surface allowed to retain them. Debug
collection is allowlisted to application-owned payloads and must not capture
unrelated process environment values or filesystem content. Trace data may
contain application data and therefore inherits its sensitivity; operators own

83
docs/releases/v0.4.0.md Normal file
View File

@@ -0,0 +1,83 @@
# Notarius v0.4.0
This release strengthens LLM reliability and validation throughout the
configured pipeline, upgrades the PromptKit integration, and establishes the
source-release and downstream-consumer workflows needed for broader D&D
pipeline integration.
## Summary
Notarius now distinguishes PromptKit structural-output repair from
application-owned semantic validation retries. Producer candidates can run
through complete deterministic validator chains, receive bounded semantic
correction guidance, and retry under explicit stage policies. Final run
receipts and manifests preserve bounded validation provenance, while outputs
that advance with incomplete validation remain available to the current run
without entering reusable checkpoint state.
The release also adds a maintained complete D&D subprocess-consumer workflow,
diagnostic build versions, and the source-only release procedure used to
publish this version.
## Compatibility
- Configuration files must use schema version 4. Version 3 is not decoded or
rewritten; rename the top-level `scriptorium` section to `promptkit` when
migrating. See [Configuration](../config.md#migrating-version-3-configuration).
- PromptKit is pinned to v0.9.0. Operator profile files use PromptKit's v0.9.0
format and may use its profile-inheritance support. Notarius continues to
resolve operator profiles before embedded fallbacks.
- Structural-output repair and semantic stage retries are separate bounded
mechanisms. Maintained production prompts request one structural repair by
default; explicit configuration can override the supported repair count.
- Validation policy can now fail a run, reject an output, or permit an
otherwise valid candidate to advance with incomplete-validation provenance.
The application defaults are documented in
[Configuration](../config.md#pipelines).
- The `notarius.run-result.v1` receipt remains at schema version 1 and adds
optional validation summaries plus a required validation-status field.
Consumers of this pre-release contract should follow the current
[run-result receipt](../integrations/run-result.md).
- Existing D&D artifact schema identities remain unchanged. Validation and
producer-policy changes can nevertheless cause previously accepted weak
candidates to retry, reject, or fail instead.
## Upgrade
1. Migrate every Notarius configuration to version 4 and rename `scriptorium`
to `promptkit`.
2. Review deployed PromptKit profiles against the pinned v0.9.0 profile format
and ensure their credential environment variables are available at run
time.
3. Run `notarius config validate --config <path> --pipeline <id>` before the
first production invocation.
4. Review `structured_output_repair_attempts`, producer retry counts, and
`validation_policy` wherever the deployment needs behavior different from
the documented defaults.
5. Update subprocess consumers to inspect receipt `validation_status` and to
tolerate the optional bounded `validation_summaries` field. A consumer that
requires fully validated artifacts should require `approved`.
## Changes
- Upgraded PromptKit from v0.5.0 through v0.9.0 and adopted profile
inheritance, structured-output repair, typed error classification, and the
correction-aware completion protocol.
- Added pipeline and binding configuration for structural repair and terminal
validation policy, with strict startup validation and effective-setting
provenance.
- Added feedback-aware retries for chunking, extraction, merge, normalize, and
semantic reconciliation producers. Retry prompts contain the exact defective
response and actionable semantic correction guidance without exposing
internal reason codes or opaque entity identifiers.
- Added complete validator-chain execution, validator retry handling, bounded
warnings, terminal dispositions, and durable validation summaries.
- Prevented validation-incomplete artifacts and all derived lineage from
loading or publishing reusable checkpoints while preserving same-run
generated-reference handoff.
- Tightened cached chunk-plan validation so only completely validated plans are
reused or replace stored plans.
- Added a machine-readable subprocess receipt workflow and complete D&D
consumer documentation covering all maintained artifacts.
- Added source-release checks, immutable lightweight-tag guidance, Linux and
Darwin build verification, and diagnostic `notarius --version` output.

View File

@@ -1,224 +0,0 @@
# D&D Subprocess Consumer Documentation
## Status
Completed. The target guide is `docs/consumers/dnd-pipeline.md`.
## Purpose
Provide one task-oriented guide for applications that run Notarius as a
subprocess to execute the maintained complete D&D pipeline and consume its
published artifacts. The initial concrete consumer is Narratio, but the guide
must describe the public Notarius workflow rather than depend on Narratio
internals.
The guide should make the safe integration path obvious without duplicating
the CLI, input, receipt, output-bundle, or individual artifact contracts that
already have canonical documentation.
## Current State
The public integration surface is documented accurately but is distributed
across several documents:
- `docs/consumers/subprocess.md` defines the generic subprocess workflow;
- `docs/cli.md` owns commands, flags, stream behavior, and exit statuses;
- `docs/integrations/seriatim.md` owns the accepted transcript input shape;
- `docs/integrations/run-result.md` owns the machine-readable successful-run
receipt;
- `docs/integrations/json-output.md` owns bundle discovery and logical files;
- the D&D integration documents own the individual lane payload contracts;
- `examples/dnd-complete.config.yml` is the maintained complete pipeline.
A consumer can reconstruct the full workflow from those documents, but there
is no D&D-focused guide that connects the maintained example to its input,
invocation, complete artifact inventory, discovery procedure, and downstream
acceptance decisions.
## Target Documentation Set
### Create `docs/consumers/dnd-pipeline.md`
This document should own the end-to-end consumer workflow for the maintained
complete D&D configuration. It should be useful to Narratio and to another
subprocess orchestrator with the same needs.
The guide should contain the following sections.
#### Prerequisites And Deployment Configuration
- Link to `examples/dnd-complete.config.yml` rather than embedding a second
complete configuration.
- Explain that a deployment must provide the configured PromptKit profile and
campaign reference files.
- Recommend absolute paths for a service or orchestrator deployment.
- Call out the path-resolution distinction explicitly: YAML reference paths
are relative to the Notarius configuration file, while
`promptkit.profile_file` is relative to the Notarius process working
directory.
- Recommend validating the selected configuration and `dnd-session` pipeline
before processing sessions.
#### Transcript Input
- State that the complete pipeline consumes a Seriatim JSON document.
- Link to the canonical Seriatim contract for required fields and validation.
- Recommend the caller's final trimmed transcript when the caller maintains
transcript tiers. For Narratio, identify the implemented source as
`narratio.transcript.final_trimmed`, normally stored at
`transcripts/final.trimmed.json`.
- Explain that segment IDs must remain stable because D&D source references
cite those units.
- Explain that Notarius derives its default prompt session from the input
module and exact input bytes and that ordinary callers should not supply
`--session-id`.
#### Subprocess Invocation
- Show one concise invocation using `notarius run dnd-session`, explicit
absolute `--config`, `--input`, and `--output-dir` paths, and `--json`.
- Direct callers to capture stdout and stderr separately, propagate
cancellation, impose an operator-appropriate timeout, and wait for process
completion before parsing stdout.
- State that only exit status zero permits receipt decoding and link to the CLI
contract for the complete exit-status definition.
- Recommend retaining stderr and the invocation context for diagnosis without
logging secrets or transcript content.
#### Receipt And Bundle Discovery
- Require callers to accept only supported run-result schema versions while
tolerating unknown fields allowed by that version.
- Direct callers to obtain the exact run-specific bundle from the receipt's
absolute `output_directory`; they must not scan for the newest run directory
or construct a run ID.
- Require a confinement check when resolving `index_file` beneath the reported
bundle root.
- Direct callers to discover lane payloads by `lane_id` in `index.json`, then
verify descriptor media type and schema identity before decoding them.
- Explain that descriptor paths are untrusted relative paths and require the
same confinement discipline.
#### Complete D&D Artifact Inventory
Include a compact table for the ten lane IDs selected by the maintained
complete configuration:
- `item-registry`;
- `npc-registry`;
- `location-registry`;
- `scene-descriptions`;
- `item-occurrences`;
- `spells`;
- `combat-turns`;
- `npc-occurrences`;
- `location-occurrences`;
- `enemy-events`.
For each row, give a one-line purpose and link to the corresponding canonical
D&D artifact contract. Do not copy its fields or schema rules into the
consumer guide.
Document the four always-published bundle files—`index.json`, `manifest.json`,
`rejected.json`, and `warnings.json`—and the complete example's configured
`chunk-map.json` and `evidence-context.json` pipeline-wide artifacts. Link to
their canonical contracts and distinguish pipeline-wide artifacts from lane
outputs.
The inventory must say that a file is available only when its corresponding
artifact was accepted and published. It must not imply that process success
guarantees every configured lane.
#### Downstream Acceptance And Retention
- Explain that exit status zero can coexist with rejected outputs, warnings,
or absent lane descriptors.
- Require the consumer to define its required lane set explicitly. Recommend
treating all ten lanes as required when the caller claims to consume the
complete D&D workflow, while allowing another consumer to adopt a narrower
documented policy.
- Recommend retaining the receipt, the complete published bundle, and captured
diagnostic streams long enough to support provenance and failure analysis.
- Explain that `evidence-context.json` is a reading excerpt; authoritative
citations remain in lane payloads.
- Treat transcripts, lane artifacts, evidence context, manifests, and logs as
sensitive campaign data.
#### Compatibility Checklist
End with a concise checklist covering process exit, receipt schema, path
confinement, pipeline identity, index decoding, required descriptors,
descriptor schema/media compatibility, warnings and rejections, checksums or
retention, and secure handling. Compatibility should be based on published
receipt and artifact contracts rather than parsing a human version string.
### Update Existing Navigation
- Add a short link from `docs/consumers/subprocess.md` to the D&D-specific
workflow. Keep generic subprocess policy in the existing document.
- Add the guide to the documentation links in `README.md`.
- Extend the subprocess-consumer row in `docs/development.md` so maintainers
working on the D&D workflow are routed to the new guide and the canonical
contracts.
### Verify Canonical Contract Documents
Review the linked integration documents and the complete example while writing
the guide. Correct an integration document only if repository inspection finds
an actual stale contract. Do not move schema definitions, field tables, CLI
flags, or configuration semantics into the new guide.
## Narratio Alignment
The guide may name Narratio as the motivating consumer and identify its current
final-trimmed transcript source. It must not claim that Narratio already has a
Notarius adapter or extraction stage. Until that feature is implemented,
Narratio-specific architecture, configuration, stage behavior, manifest
records, and artifact source IDs belong in Narratio's roadmap.
Once Narratio implements the integration, its own integration documentation
should link to this guide and the durable Notarius contracts instead of
repeating them.
## Validation
Documentation implementation should include:
```sh
go run ./cmd/notarius config validate \
--config examples/dnd-complete.config.yml \
--pipeline dnd-session
go test ./...
```
Also verify all new and changed relative Markdown links, compare the artifact
inventory directly with the maintained complete configuration, and confirm
that commands and path semantics match the CLI and configuration references.
If the repository still has no automated link checker, record that fact and
perform a focused manual link review.
## Acceptance Criteria
- A subprocess integrator can follow one D&D-focused guide from a Seriatim
transcript through safe discovery of every artifact configured by the
complete example.
- The guide makes stdout, stderr, exit-status, receipt, and path-confinement
responsibilities unambiguous.
- The ten configured D&D lanes and both configured pipeline-wide artifacts are
listed and linked to their canonical contracts.
- The guide distinguishes process success from the caller's required-artifact
policy.
- The profile-path and reference-path resolution rules are clearly stated.
- Existing navigation makes the guide discoverable.
- No volatile contract is defined in two places, and no unimplemented Narratio
behavior is presented as current.
## Non-Goals
- Implementing or documenting Narratio's future adapter or stage as current
Notarius behavior.
- Adding a new Notarius command, receipt version, output format, or artifact
schema.
- Duplicating the complete configuration or individual D&D payload schemas in
prose.
- Defining a universal partial-result policy for every Notarius consumer.

View File

@@ -7,130 +7,13 @@ not as committed release dates.
## Near-Term Validation And LLM Reliability
The following work forms one related program but should be promoted into
separate feature roadmaps and implemented in dependency order. PromptKit owns
structural output repair within one completion. Notarius owns stage candidates,
validator chains, semantic rejection policy, and whether another stage attempt
is warranted.
PromptKit now owns structural output repair within one completion. Notarius
owns stage candidates, validator chains, semantic rejection policy, bounded
feedback-aware stage retries, validation provenance, and reusable-state
eligibility. The remaining near-term work applies those completed foundations
to domain review and operator-facing diagnostics.
### 1. Upgrade To PromptKit v0.8.0
This item has been promoted to the standalone
[PromptKit v0.8.0 Upgrade](promptkit-v0.8.md) roadmap. That document owns the
release-by-release compatibility review, adopted features, structured-repair
policy, target integration boundary, acceptance criteria, and settled design
decisions.
### 2. Feedback-Aware Stage Validation Retries
- Model Notarius's corrective stage-retry conversation explicitly after
PromptKit v0.8.0. The first attempt sends the ordinary complete initial
prompt. If application validation rejects the resulting LLM-produced
candidate and another stage attempt is available, reconstruct that complete
initial prompt byte-for-byte and append exactly two messages: an assistant
message containing the defective response and an application-owned user
message detailing every applicable semantic validation error and requesting
one corrected, complete replacement response. This is a freshly constructed
correction request, not continuation of an accumulating conversation.
- Use the configured stage `retries` value as the one outer retry budget for
this loop. `retries: N` continues to mean at most `N` additional complete
chunk, extract, merge, or normalize attempts after the initial attempt,
whether an attempt is needed because of a producer error or semantic
rejection. Do not add a second semantic-correction count. PromptKit's
prompt-level `repair_attempts` budget is independent and internal to each
individual LLM completion, and does not consume or replenish the Notarius
stage budget.
- Extend the framework-managed validation boundary for chunk, extract, merge,
and normalize stages so a rejected LLM-produced candidate and its exact raw
model response remain available to construct the next stage attempt.
Deterministic producers cannot improve by repeating the same inputs; a
rejection from a deterministic stage is therefore terminal under the
configured rejection policy rather than consuming retries mechanically.
- Preserve the original session ID, selected profile, structured-output
contract, prompt inputs, and reusable prompt prefix. Carry only the latest
candidate and latest aggregate feedback; do not build an unbounded retry
conversation. Keep model-facing corrective guidance separate from
operator-facing diagnostics, and apply explicit size, redaction, and debug
disclosure rules to both.
- Run every applicable validator in the configured chain before deciding
whether to retry. Do not short-circuit merely because an earlier validator
rejected the candidate. Aggregate all semantic rejection reason codes and
corrective guidance into the retry message so one retry can address the
whole candidate. A validator is applicable only when its declared target and
prerequisites can be satisfied; record a deterministic skipped diagnostic
rather than invoking a validator on an input it cannot interpret. Initially
execute the chain sequentially in configured order so results, diagnostics,
costs, and feedback ordering remain deterministic; consider validator
concurrency only in response to measured latency.
- Continue running independent applicable validators after one validator
execution failure so the attempt retains as much useful diagnostic
information as practical. Do not present validator operational failures as
defects in the producer candidate and do not include them in corrective
feedback.
- Distinguish three terminal conditions and make their policies configurable
at a coherent pipeline or binding scope:
- **producer structural failure:** PromptKit could not return a usable
structured candidate after its repair budget. Default to `fail_run`; an
allowed alternative may record a terminal stage or lane rejection where
execution can safely continue, but may not accept the invalid output;
- **semantic rejection:** one or more validators completed and rejected the
candidate. Default to `fail_run` after corrective stage retries are
exhausted; allow an explicit alternative that records the existing
rejected-output outcome without advancing that output;
- **validator execution failure:** a validator could not produce a valid
decision because of generation, structural-output, transport, or internal
failure. Default to a genuine warning and an explicitly recorded
`validation_incomplete` or equivalent degraded state while allowing the
candidate to continue; allow strict configuration to fail the run instead.
- An LLM-backed validator uses the same scheduled PromptKit boundary as every
other LLM-backed module. Its own response may use PromptKit's bounded
structural repair. Distinguish its possible output states:
- output rejected by PromptKit's structural contract should consume only the
validator prompt's configured PromptKit repair budget;
- output that is structurally valid but violates a deterministically
checkable validator-result invariant should be classified as a validator
execution failure;
- output that satisfies the complete validator-result contract is the
validator's decision, even though an LLM judgment may remain imperfect.
Automatically judging that judgment would require another semantic
validator and is outside this feature.
If the validator cannot return a contract-valid decision, do not recursively
create another Notarius semantic-validation loop around it. Apply the
configured validator-failure policy. The default warning must identify the
validator and affected stage without exposing sensitive content.
- Separate validator execution retry from producer correction. A transient
validator operational failure must not automatically discard and regenerate
an otherwise usable producer candidate. Any bounded retry of the validator
itself should reuse that same immutable candidate and remain subordinate to
PromptKit and provider retry behavior.
- Preserve attempt-level provenance, cumulative token usage, validator
outcomes, aggregated correction feedback, and terminal policy decisions in
the debug and manifest models without copying raw source material into
ordinary errors or durable summaries.
- Define terminal-outcome precedence. A semantic rejection dominates a
validator execution failure for the same candidate: use the completed
rejections to correct the producer while separately recording incomplete
validation. If a later candidate has no semantic rejection but one validator
still fails, apply the configured validator-failure policy to that candidate.
Never allow a known semantic rejection to become accepted through a
warn-and-continue setting, and never accept a structurally invalid producer
response. Permissive policy may preserve a rejected-output outcome or accept
a structurally valid candidate with explicitly incomplete validation; it may
not relabel known-invalid output as approved.
Before implementation, record the generic validation and retry state machine
in an ADR. The ADR should own the separation between PromptKit repair and
Notarius correction, use of the existing stage-retry budget, reconstruction of
correction conversations, all-applicable-validator aggregation, deterministic
validator ordering, non-recursive validator failure handling, outcome
precedence, default fail-open/fail-closed choices, configurable terminal
policies, and provenance and sensitive-data constraints. A dependency-upgrade
ADR is not needed for PromptKit v0.8.0 itself. Current behavior remains
authoritative until the validation ADR is implemented and the canonical
architecture, configuration, operations, and internal documentation are
updated.
### 3. D&D Combat Scene Semantic Validation
### D&D Combat Scene Semantic Validation
- Add an optional production LLM-backed D&D validator that determines whether
proposed scene boundaries and classifications represent substantive active
@@ -172,7 +55,7 @@ updated.
the chunker or otherwise changes stage ownership or the durable chunk-plan
contract.
### 4. Warning Signal And Presentation Reform
### Warning Signal And Presentation Reform
- Audit every warning producer and representative successful runs. Ordinary
success producing dozens of warnings is a failed operator experience: the

View File

@@ -1,782 +0,0 @@
# PromptKit v0.8.0 Upgrade Implementation Plan
## Purpose
Implement the target state defined by the
[PromptKit v0.8.0 Upgrade](promptkit-v0.8.md): adopt the useful PromptKit
v0.6.0, v0.7.0, and v0.8.0 changes; enable one bounded structural correction
by default; expose pipeline and binding overrides; preserve safe provider
diagnostics; and keep PromptKit behind Notarius's transport-neutral LLM
boundary.
This plan is ordered. Each numbered stage is one implementation prompt for a
gpt-5.6-terra coding agent. Complete and validate one stage before beginning
the next. Read `docs/development.md` and every policy under `docs/policy/` at
the start of each stage, inspect the current code and tests named by that
stage, preserve unrelated worktree changes, and update current-behavior
documentation in the same stage as the behavior it describes.
Do not retire this plan or `promptkit-v0.8.md` during implementation. Keep both
until the completed work has passed a separate review. Do not implement the
future Notarius semantic-validation retry loop, D&D combat-scene validator, or
warning redesign as part of this plan.
## Decisions Fixed For Implementation
- Pin `gitea.maximumdirect.net/eric/promptkit` v0.8.0 directly, with no
`replace`, workspace dependency, or vendored source.
- Every maintained eligible production prompt defaults to exactly one
PromptKit structural repair attempt.
- Add the exact configuration key
`structured_output_repair_attempts` at pipeline scope and on LLM-backed
module and validator bindings.
- Effective precedence is binding value, then pipeline value, then the prompt's
declared `repair_attempts` value. Omission inherits; explicit zero disables
structural repair at that scope.
- Accepted values are integers from zero through three. Explicit null and
non-integer values are invalid. An explicit binding value on a deterministic
module or validator is invalid. A pipeline value is applied only to selected
LLM-backed bindings and does not make deterministic bindings invalid.
- Keep file configuration version 4. This is an additive pre-release field and
does not require parallel versioned behavior.
- Use `StructuredOutputRepairAttempts *int` for presence-aware internal Go
fields. Clone pointers at every ownership boundary.
- A configured override never replaces schema identity, output format, or
validation mode. The PromptKit adapter calls `InspectPrompt`, copies the
complete normalized prompt-owned output contract, changes only
`RepairAttempts`, and supplies the complete replacement on `RunRequest`.
Do not add an inspection cache initially.
- PromptKit repair is internal to one `CompleteStructured` call and does not
consume or replenish a binding's existing `retries` budget.
- Add `RepairAttempts int` to Notarius's structured-completion response. It is
the actual corrective-call count reported by PromptKit; token usage remains
PromptKit's cumulative usage and must not be summed again.
- A valid repaired response is successful and produces no warning solely
because repair occurred. Exhausted structural validation maps to
`ErrInvalidStructuredOutput` with the final candidate and debug material
retained.
- Add an application-owned generation-error sentinel and typed status-bearing
error. PromptKit error types must not cross `internal/framework/llm`.
- HTTP status may appear in the application-owned generation error. Provider
code, type, and message are excluded from ordinary errors, warnings,
manifests, cache, and checkpoint identity; they may appear only in an
explicitly requested debug trace after Notarius redaction.
- Profile inheritance is owned entirely by PromptKit. Notarius passes sources
through, inspects and records the resolved target, and does not parse or merge
`base_profile` itself.
- PromptKit's built-in `rakestrawhome` backend and
`rakestrawhome-gemma-4-31b` profile are available generically. Notarius does
not register, shadow, or select them by default.
- Missing optional credential environment values are allowed to reach the
provider without `Authorization`; Notarius does not recreate v0.5.0's local
failure or add provider-specific authentication logic.
- No dependency-upgrade ADR is required. Update architecture only with the
durable ownership distinction between PromptKit structural repair and
Notarius stage/semantic validation policy.
## Stage 1: Upgrade The Dependency And Establish A Clean v0.8.0 Baseline ✅
### Goal
Move the repository to PromptKit v0.8.0, resolve source-compatibility issues,
and establish a passing baseline before adopting new behavior.
### Implementation
1. Re-read the upstream v0.6.0, v0.7.0, and v0.8.0 release guides and the
v0.8.0 package consumer and format documentation. Treat the pinned v0.8.0
tag, not the sibling checkout's moving branch, as authoritative.
2. Update `go.mod` and `go.sum` to PromptKit v0.8.0 and run `go mod tidy` with
`GOWORK=off`.
3. Compile before making compatibility edits. Correct only actual source or
behavior incompatibilities. In particular:
- convert any positional `promptkit.Profile` or
`promptkit.OpenAICompatibleProfileConfig` literals to keyed literals;
- confirm Notarius does not register the newly reserved `rakestrawhome`
backend ID; and
- preserve `PrepareExecution`/`Details`/`RunPrepared` snapshot ownership,
`Discard`, session forwarding, reasoning override, profile preflight,
and capacity adaptation.
4. Change `promptKitBuiltinProfileCatalogID` in
`internal/framework/llm/promptkit_profile_fingerprint.go` from the v0.5.0
catalog marker to an opaque v0.8.0 marker. Do not hash PromptKit internal
files or include catalog content in manifests.
5. Update `docs/integrations/pkg-promptkit.md` to pin and link v0.8.0 and to
state that this stage still leaves the production prompt-declared repair
budget at its current value. Do not document later configuration or default
behavior before it exists.
6. Update only those existing tests whose public PromptKit types or stable
v0.8.0 behavior genuinely changed. Do not rewrite tests merely to match
upstream diagnostic wording.
### Tests And Validation
```sh
GOWORK=off go mod tidy -diff
GOWORK=off go test ./internal/framework/llm ./internal/cli
GOWORK=off go test ./...
GOWORK=off go vet ./...
GOWORK=off go build ./cmd/notarius
git diff --check
```
### Acceptance Criteria
- `go list -m gitea.maximumdirect.net/eric/promptkit` reports v0.8.0.
- There is no PromptKit `replace`, active Go workspace dependency, or vendor
tree.
- The adapter still uses one frozen prepared execution and all existing LLM
tests pass.
- Checkpoint profile identity includes the v0.8.0 built-in catalog marker.
- Current integration documentation pins v0.8.0 without claiming that
later stages are already active.
- The full ordinary test suite, vet, and command build pass.
## Stage 2: Verify v0.6.0 Compatibility And Hardening ✅
### Goal
Audit Notarius's assets and boundary values against PromptKit v0.6.0's stricter
source, path, endpoint, JSON, and cancellation contracts, fixing only concrete
incompatibilities.
### Implementation
1. Inspect `internal/framework/llm/asset_registry.go`, prompt/profile source
composition, all registered asset roots, the conventional local backend,
and their focused tests.
2. Exercise every production asset registry through PromptKit engine
construction and the existing production composition tests. Confirm that:
- YAML IDs and versions, not filenames, select definitions;
- every `content_file` path is exact, relative, contained, and points to a
regular embedded file;
- every schema and JSON asset is one complete JSON value;
- every current output contract is valid under v0.8.0; and
- unrelated malformed definitions do not create a second Notarius identity
or fallback mechanism.
3. Review local endpoint parsing and validation. Retain a narrower Notarius
rule only if it has independent application value; otherwise rely on
PromptKit's absolute HTTP/HTTPS URL contract. Never accept a value that the
adapter will later reject.
4. Review conversion of Notarius variables, inputs, profile extras, and debug
values at the adapter boundary for PromptKit's bounded JSON-compatible-value
rules. Do not add a second generic JSON walker or duplicate upstream numeric
limits.
5. Verify cancellation and deadline identity through existing adapter tests.
Add or refine one focused regression only if Notarius currently destroys an
`errors.Is`-relevant context or transport error that the application owns.
6. Do not add a cross-operation schema cache, artifact cache, provider-body
reader, or duplicate JSON framing validation; v0.6.0 owns those mechanisms.
### Tests And Validation
Run the focused asset, profile-source, and adapter packages, then the ordinary
and race-enabled suites:
```sh
GOWORK=off go test ./internal/framework/llm ./internal/cli
GOWORK=off go test ./...
GOWORK=off go test -race ./...
git diff --check
```
Tests must remain offline and should validate Notarius's assembled boundary,
not reproduce PromptKit's internal path, JSON-depth, or response-size matrices.
### Acceptance Criteria
- Every maintained embedded prompt, schema, and fallback profile can be loaded
through the assembled v0.8.0 engine.
- Current local endpoint and JSON-compatible values either satisfy the stricter
upstream contract or fail during preparation with safe diagnostics.
- No duplicate PromptKit-owned cache, JSON, or response-bound mechanism is
introduced.
- Cancellation and deadline behavior remains discoverable at the Notarius
boundary.
- Ordinary and race-enabled tests pass.
## Stage 3: Adopt Profile Inheritance, Rakestrawhome, And Optional Credentials ✅
### Goal
Make the useful PromptKit v0.7.0 profile and backend behavior work through
Notarius's existing generic profile boundary without adding provider-specific
composition logic.
### Implementation
1. Inspect `promptkit_profiles.go`, `asset_registry.go`, profile fingerprinting,
CLI profile preflight, profile provenance recording, and their tests before
editing.
2. Add an offline integration test using a temporary operator profile source
whose leaf uses `base_profile`. Prove that:
- preflight reports the leaf ID;
- the effective backend, model, reasoning, and other inherited values match
the resolved PromptKit target;
- execution uses the same resolved target as inspection; and
- a missing parent or cycle fails before provider generation with a safe
profile-load diagnostic.
Do not duplicate PromptKit's entire field-by-field merge test matrix.
3. Add a checkpoint-safety test showing that changing a parent definition in
an operator profile directory changes Notarius's profile-source fingerprint
while profile content and paths remain absent from the fingerprint value.
Retain the v0.8.0 catalog marker as coverage for built-in-parent changes.
4. Verify `rakestrawhome-gemma-4-31b` through the ordinary profile inspector.
Assert its selected backend reaches Notarius's application-owned inspection
and provenance fields. Use a fake PromptKit client or transport if execution
coverage is needed; never contact the live service or require credentials.
5. Verify that Notarius registers no `rakestrawhome` override and that the
existing `local` registration remains independent.
6. Add one `httptest`-backed adapter integration test for a filesystem profile
with a missing optional `api_key_env`. The request must reach the test server
without an `Authorization` header. Add a focused in-memory PromptKit profile
test for `APIKeyRequired` only if needed to prove Notarius preserves upstream
preflight behavior; do not expose a new operator profile API.
7. Keep `assets/dnd/profiles/dnd-extraction.yaml` standalone and unchanged. No
matching v0.8.0 built-in profile owns its `openai/gpt-5.6-luna` target.
8. Update the current profile-source, deployment, and pinned-integration
sections in `docs/config.md`, `docs/operations.md`,
`docs/internal/llm.md`, and `docs/integrations/pkg-promptkit.md`. Link to the
pinned PromptKit format rules for inheritance. Explain that filesystem
profiles cannot express PromptKit's in-memory `APIKeyRequired` field and
that an optional missing credential may result in a provider 401/403.
### Tests And Validation
```sh
GOWORK=off go test ./internal/framework/llm ./internal/core/config ./internal/cli
GOWORK=off go test ./...
GOWORK=off go test -race ./internal/framework/llm ./internal/cli
git diff --check
```
### Acceptance Criteria
- Inherited operator profiles resolve identically during preflight and
execution, with the leaf ID and effective target kept distinct.
- Parent changes invalidate checkpoint reuse without leaking profile content or
paths.
- Rakestrawhome is available through generic PromptKit profile handling and is
not selected by default or registered by Notarius.
- Missing optional credentials omit authorization and reach the controlled
test provider; explicitly required credentials retain upstream behavior.
- Current documentation accurately describes the implemented profile and
credential behavior without duplicating PromptKit's merge algorithm.
## Stage 4: Adapt Structured Generation Errors Safely ✅
### Goal
Use PromptKit v0.7.0's structured generation errors for stable status
classification and debug-only provider diagnostics without leaking PromptKit
types or sensitive provider text.
### Implementation
1. In `internal/framework/contracts`, add:
- `ErrLLMGeneration` as the provider-neutral generation-failure sentinel;
- an application-owned `LLMGenerationError` with private status and safe
diagnostic fields, `Error`, `Unwrap`, and `StatusCode` methods; and
- a constructor that accepts a nonnegative status and an already-redacted
diagnostic. Status zero means no HTTP status was available.
Ordinary callers may inspect status with `errors.As` and category with
`errors.Is`, but cannot obtain provider code, type, or message from the
error.
2. Add an application-owned `LLMDebugProviderError` with `status_code`, `code`,
`type`, and `message` fields, referenced optionally from
`LLMDebugResponse`. This is debug material, not a manifest or durable public
artifact contract.
3. In `PromptKitClient.CompleteStructured`, preserve precedence in this order:
caller context cancellation/deadline, PromptKit capacity error, structured
PromptKit generation error, then other PromptKit generation failures.
Map every generation failure to `ErrLLMGeneration`; map
`*promptkit.GenerationError` to `LLMGenerationError` with its status.
Never wrap or return the PromptKit error value itself.
4. Keep the ordinary diagnostic limited to PromptKit's safe default error
formatting after bearer and known-credential redaction. Do not append
`ProviderCode`, `ProviderType`, or `ProviderMessage` to it.
5. For an explicitly requested debug path, preserve prepared prompt details and
attach the PromptKit provider code, type, and message after:
- reading only the selected prepared target's `APIKeyEnv`, if any, to obtain
the exact known credential solely for redaction;
- applying `RedactSecrets` and the existing bearer/key-pattern redaction;
- retaining PromptKit's already-normalized bounds; and
- discarding the credential value immediately rather than storing it.
Do not scan unrelated environment variables.
6. Return prompt/debug material alongside the error so the existing debug LLM
wrapper can persist it only when debug recording is enabled. Confirm that
provider fields do not appear in ordinary error text, warnings, manifests,
cache, checkpoint data, or a run without debug output.
7. Refactor error mapping into small helpers if needed to keep
`CompleteStructured` readable; do not create provider-specific policy in
modules or the pipeline runner.
8. Update the error and observability sections of `docs/internal/llm.md` and
`docs/integrations/pkg-promptkit.md`. Keep operator disclosure rules in
`docs/operations.md` concise and link to the internal boundary where useful.
### Tests And Validation
- Use `httptest.Server` to return representative structured 400 and 503
responses. Assert `errors.Is(ErrLLMGeneration)`, `errors.As` to the
application-owned type, and the exact status without asserting complete
human wording.
- Include a provider message containing the selected test credential and a
bearer-shaped value. Verify both are absent from the ordinary error and
debug artifact, while a non-sensitive marker appears only in the requested
debug trace.
- Retain existing capacity and context tests to prove their more specific
classifications still win.
```sh
GOWORK=off go test ./internal/framework/contracts ./internal/framework/llm ./internal/framework/pipeline ./internal/cli
GOWORK=off go test ./...
GOWORK=off go test -race ./internal/framework/llm ./internal/framework/pipeline
git diff --check
```
### Acceptance Criteria
- PromptKit generation errors never escape the adapter error chain.
- All generation failures match `ErrLLMGeneration`; structured non-success
responses expose only application-owned HTTP status to ordinary callers.
- Provider code, type, and message are available only in an explicitly
requested, redacted debug trace.
- Capacity and context classifications remain unchanged and more specific.
- Security tests prove selected credentials and bearer tokens are not leaked.
## Stage 5: Add Adapter-Level Structured Repair Support ✅
### Goal
Teach the transport-neutral completion boundary and PromptKit adapter to apply
an optional repair override and report actual repair behavior, without yet
exposing the setting in pipeline configuration.
### Implementation
1. Add `StructuredOutputRepairAttempts *int` to
`contracts.StructuredCompletionRequest`. Copy the pointed-to value wherever
requests are cloned or retained.
2. Add `RepairAttempts int` to `contracts.StructuredCompletionResponse`. It is
the actual number of corrective generation calls, not the configured budget
and not the number of total candidates.
3. Validate a non-nil request value as zero through three at the adapter
boundary so programmatic callers cannot bypass later file/config validation.
4. When the request value is nil, leave `promptkit.RunRequest.Validation` nil
so the prompt's complete contract remains authoritative.
5. When the value is non-nil:
- call `Engine.InspectPrompt(ctx, promptID, promptVersion)`;
- copy `PromptInspection.OutputContract` by value;
- replace only `RepairAttempts`;
- pass the complete copied contract as `RunRequest.Validation`; and
- prepare and execute exactly as before.
Do not infer or hard-code schema paths, validation modes, or formats. Do not
cache inspection in this stage.
6. Map `result.Validation.RepairAttempts` to the response and leave
`result.Usage` cumulative values unchanged. The existing debug validation
object and prepared output contract should show actual and configured values
respectively.
7. Preserve result semantics:
- valid initial and repaired candidates decode normally;
- repair exhaustion returns the final raw candidate/debug material with an
error matching `ErrInvalidStructuredOutput`;
- explicit empty or whitespace-only content follows PromptKit validation;
- missing/null/non-string content remains a generation/provider failure;
- corrective-call generation errors use Stage 4's application-owned mapping;
and
- context cancellation wins at every error boundary.
8. Keep `CompleteStructured` and its helpers provider neutral outside this
adapter package. Do not expose PromptKit validation or inspection types.
9. Update only the adapter-owned repair behavior in `docs/internal/llm.md` and
`docs/integrations/pkg-promptkit.md`. State that public pipeline configuration
and the production default are added by later stages of this plan.
### Tests And Validation
Add adapter-level behavioral tests using a deterministic fake PromptKit LLM:
- nil override uses the prompt declaration;
- explicit zero overrides a positive prompt declaration without dropping its
JSON Schema contract;
- explicit one turns an invalid first candidate followed by a valid candidate
into one successful response with the final raw bytes, actual repair count
one, and cumulative usage;
- repair exhaustion returns the final candidate and validation diagnostics as
`ErrInvalidStructuredOutput`;
- explicit empty content is eligible for repair;
- a corrective generation failure maps through Stage 4; and
- invalid direct values below zero or above three fail before provider work.
Do not assert PromptKit's exact assistant/user correction prose or copy its
full internal repair matrix.
```sh
GOWORK=off go test ./internal/framework/contracts ./internal/framework/llm ./internal/framework/pipeline
GOWORK=off go test -race ./internal/framework/llm ./internal/framework/pipeline
GOWORK=off go test ./...
git diff --check
```
### Acceptance Criteria
- The adapter changes only repair count when applying a request override.
- Nil and explicit zero remain distinct.
- Repaired success returns final raw output, cumulative usage, and actual count
without a warning.
- Exhaustion, empty content, corrective generation failure, and cancellation
match the target semantics.
- No PromptKit type crosses the LLM package boundary.
## Stage 6: Propagate Repair Policy Through Framework Requests ✅
### Goal
Carry an optional effective repair budget from each resolved stage or validator
binding to its module request without changing public file configuration yet.
### Implementation
1. Add `StructuredOutputRepairAttempts *int` alongside `LLMProfile` to every
stage request that can belong to an LLM-backed binding:
- `ParseRequest`;
- `ChunkRequest`;
- `TypedExtractionRequest`;
- `TypedMergeRequest`;
- `TypedNormalizeRequest`;
- `OutputRequest`;
- `TypedValidationRequest`;
- `ChunkValidationRequest`; and
- `SerializedValidationRequest`.
2. Add the same optional field to the erased/internal request carriers used by
registry builders, preparation, runner stage attempts, validator targets,
retry closures, and debug wrappers. Copy pointer values; never share a
mutable pointer owned by configuration.
3. At every runner stage invocation, obtain the value from the exact resolved
producer binding. At every validator invocation, obtain it from that exact
resolved validator binding. Do not use the producer's value for a validator
or vice versa.
4. Ensure all retry attempts for the same binding receive the same effective
structural-repair value. Do not decrement it in Notarius; PromptKit owns the
inner budget independently on each `CompleteStructured` call.
5. Extend `semanticreconcile.Request` with the optional field and carry it into
each generic reconciliation completion. A batched reconciliation may make
several completion calls; each call receives the same effective budget.
6. Update registry erasure/adaptation code for typed merge, normalize, and
validation requests so no field is lost. Preserve input/output support even
though current production input and output modules are deterministic.
7. Add focused framework tests for one chunk producer, one extraction
producer, one normalizer, and one LLM-backed validator. Verify exact pointer
value propagation and separation between producer and validator settings.
Do not add repetitive tests for every generic adapter.
### Tests And Validation
```sh
GOWORK=off go test ./internal/framework/contracts ./internal/framework/pipeline ./internal/framework/semanticreconcile
GOWORK=off go test -race ./internal/framework/pipeline ./internal/framework/semanticreconcile
GOWORK=off go test ./...
git diff --check
```
### Acceptance Criteria
- Every stage and validator request can carry a detached optional repair value.
- The runner sources the value from the exact resolved binding.
- Producer and validator values cannot overwrite one another.
- Stage retries reuse but do not mutate or consume the inner repair budget.
- Semantic reconciliation forwards the budget to every one of its completion
calls.
- Existing behavior remains unchanged while all values are nil.
## Stage 7: Forward Repair Policy From Every LLM-Backed Module ✅
### Goal
Complete the internal end-to-end path by having every production LLM-backed
module forward its stage request value to `CompleteStructured`.
### Implementation
1. Inventory every production `CompleteStructured` call with code search before
editing. The expected current owners include:
- `dnd/scenes` chunking;
- the combat-turn, enemy-event, item-occurrence, item-registry,
location-occurrence, location-registry, NPC-occurrence, NPC-registry,
scene-description, and spell extractors; and
- generic semantic reconciliation used by the item, location, and NPC
registry normalizers.
Reconcile this list with the actual repository; do not omit a newly added
production caller merely because it is not named here.
2. In each direct caller, set
`StructuredCompletionRequest.StructuredOutputRepairAttempts` from the
corresponding stage request. Clone the pointer or use a small shared helper
if that reduces repeated ownership mistakes without moving domain logic.
3. Ensure D&D registry normalizers pass their typed normalize request value into
`semanticreconcile.Request`, and that the generic engine forwards it as
established in Stage 6.
4. Update existing module prompt-mapping tests that already inspect a captured
structured-completion request to assert the new field. Do not create a new
one-test-per-module suite solely to memorialize field plumbing; rely on the
existing request-contract tests plus a final complete call-site audit.
5. Search again after editing for production `CompleteStructured` calls and
verify each either forwards the field or documents why it cannot receive a
pipeline binding. Test-only fakes need only preserve the field when their
contract test depends on it.
6. Do not set a module-specific fallback value. Nil must reach the adapter so
the prompt declaration remains authoritative.
### Tests And Validation
Run focused D&D and semantic-reconciliation packages, then the full suite:
```sh
GOWORK=off go test ./internal/modules/dnd/... ./internal/framework/semanticreconcile
GOWORK=off go test -race ./internal/modules/dnd/... ./internal/framework/semanticreconcile
GOWORK=off go test ./...
git diff --check
```
### Acceptance Criteria
- Every production LLM-backed completion receives the exact stage or validator
repair value.
- No module invents a default or imports PromptKit.
- Registry normalizers preserve the value through semantic reconciliation.
- Existing request-contract tests remain concise and pass.
- A final call-site audit finds no silent production omission.
## Stage 8: Add The Public Repair Configuration Contract ✅
### Goal
Add presence-aware pipeline and binding configuration for
`structured_output_repair_attempts` without yet changing runtime resolution.
### Implementation
1. Add `StructuredOutputRepairAttempts *int` to
`pipeline.PipelineProfile` and `pipeline.ModuleBinding`, using
`json:"structured_output_repair_attempts,omitempty"`.
2. Add presence-aware YAML support at pipeline and object-binding scope:
- exact key `structured_output_repair_attempts`;
- integer values zero through three;
- explicit null, non-integer, and out-of-range values rejected with scoped
diagnostics; and
- scalar shorthand bindings continue to omit the binding override.
Preserve file configuration version 4.
3. Update every configuration clone, conversion, redaction, summary, and JSON
round-trip carrier. Copy pointers by value into newly allocated storage so
parsed, configured, and redacted values do not alias.
4. Preserve omission versus explicit zero through YAML parsing, profile
inheritance, module-binding object form, and JSON round trips. Keep scalar
shorthand bindings equivalent to omission.
5. Do not add a top-level `promptkit.repair_attempts` setting or CLI override.
6. Add concise parser and ownership tests. Defer execution-class checks,
effective precedence, resolved digests, and runtime forwarding to Stage 9,
where module metadata is available.
### Tests And Validation
At the parser/config boundary, test omitted, explicit zero, positive bounds,
negative, above-three, null, non-integer, scalar shorthand, cloning, redaction,
and JSON round-trip behavior. Use relational boundary tests for the allowed
range and avoid duplicating the same cases at every layer.
```sh
GOWORK=off go test ./internal/core/config ./internal/cli
GOWORK=off go test -race ./internal/core/config
GOWORK=off go test ./...
go run ./cmd/notarius config validate \
--config examples/dnd-minimal.config.yml \
--pipeline dnd-session
go run ./cmd/notarius config validate \
--config examples/dnd-complete.config.yml \
--pipeline dnd-session
git diff --check
```
### Acceptance Criteria
- The exact public field parses at pipeline and object-binding scope with the
fixed range.
- Nil and explicit zero remain distinguishable through parsing, cloning,
inheritance, redaction, summaries, and round trips.
- Both maintained configurations remain valid without requiring the new field.
- No runtime or prompt default has changed prematurely.
## Stage 9: Resolve And Apply Repair Configuration ✅
### Goal
Resolve the public field against module execution classes, incorporate the
effective value into pipeline identity, and connect it to the request plumbing
completed in Stages 6 and 7.
### Implementation
1. During resolution, compute the effective value for every selected binding:
- explicit binding value wins;
- otherwise an explicit pipeline value applies to an LLM-backed binding;
- otherwise leave nil for prompt-owned policy.
Apply the pipeline value to LLM-backed validators as well as producers.
2. Reject an explicit binding value on a deterministic module or deterministic
validator using the same execution-class knowledge used for `llm_profile`.
Do not reject a pipeline-level value merely because a selected pipeline also
contains deterministic bindings; simply do not apply it to those bindings.
3. Clone every resolved pointer so the parsed pipeline, resolved profile,
redacted summaries, and runner requests have distinct ownership.
4. Include the effective field in resolved pipeline JSON and digest input. A
change between nil, zero, and a positive value must change the resolved
digest when it changes an LLM-backed selected binding. Unselected lanes must
retain the repository's existing digest and selection semantics.
5. Pass the resolved value into the Stage 6 request field for every selected
input, chunk, extract, merge, normalize, output, and validator binding.
6. Update `docs/config.md` as the canonical field, range, and precedence
contract; `docs/internal/pipeline.md` as the resolution owner; and
`docs/operations.md` for the distinction from binding `retries`. The
prompt-owned production default remains unchanged until Stage 10.
7. Add focused resolution and runner tests. Cover representative execution
classes rather than repeating the same assertion for every module type.
### Tests And Validation
Test:
- binding over pipeline over nil precedence;
- inheritance into each selected LLM-backed stage and validator;
- no inheritance into deterministic bindings;
- explicit deterministic-binding rejection;
- detached pointers;
- runner forwarding for representative producer and validator bindings; and
- digest changes for execution-relevant nil, zero, and positive changes.
```sh
GOWORK=off go test ./internal/framework/pipeline ./internal/cli
GOWORK=off go test -race ./internal/framework/pipeline
GOWORK=off go test ./...
go run ./cmd/notarius config validate \
--config examples/dnd-minimal.config.yml \
--pipeline dnd-session
go run ./cmd/notarius config validate \
--config examples/dnd-complete.config.yml \
--pipeline dnd-session
git diff --check
```
### Acceptance Criteria
- The exact public field has the fixed binding-over-pipeline-over-prompt
precedence for every selected LLM-backed producer and validator.
- Deterministic binding misuse fails during resolution before execution, while
a pipeline value coexists with deterministic bindings.
- Nil and explicit zero remain distinguishable through resolution, runtime,
summaries, and digests.
- A policy change invalidates checkpoint identity when it changes an effective
selected binding.
- Current configuration, pipeline, and operations documentation matches the
implemented behavior.
## Stage 10: Enable The Default, Finish Documentation, And Verify The Feature ✅
### Goal
Set the accepted production default of one repair, reconcile all canonical
documentation, and run the full repository verification pass.
### Implementation
1. Change `repair_attempts: 0` to `repair_attempts: 1` in every maintained
production prompt manifest that produces structured output, including the
generic semantic-reconciliation prompt and every D&D chunk, extraction, and
registry-normalization prompt. Do not mechanically change unrelated test
fixtures whose purpose is to exercise zero.
2. Inspect every production prompt output contract after the edit. Confirm that
each positive budget uses `basic`, `json`, or `json_schema`, remains no
greater than three, and retains its existing format and schema path.
3. Add or refine the smallest durable assembled-assets test that proves the
production engine can prepare the maintained prompts with the activated
contracts. Do not add a brittle test that asserts an exact prompt count,
file count, message prose, correction text, or asset length. The public
default may be tested at one canonical assembled boundary because its
literal value is an operational contract.
4. Confirm a successful repair does not create a warning and that exhausted
repair remains `ErrInvalidStructuredOutput`. Verify the debug prompt records
the configured contract, the debug response records actual repair count,
and cumulative usage is not double-counted.
5. Confirm scheduling behavior with one focused test or existing coverage: the
Notarius scheduled client admits one logical `CompleteStructured` operation
while PromptKit may make serial corrective provider calls inside it. Do not
attempt to reacquire a Notarius permit from inside PromptKit or add a second
scheduler.
6. Reconcile current-state documentation:
- `docs/integrations/pkg-promptkit.md` owns the pinned upstream boundary;
- `docs/config.md` owns field names, range, default, and precedence;
- `docs/operations.md` owns latency/cost, optional credentials, concurrency,
timeout, and the upper-bound formula;
- `docs/internal/llm.md` owns inspection-based contract replacement,
cumulative usage, actual repair count, generation errors, and debug data;
- `docs/internal/pipeline.md` owns effective policy propagation and the
separation from stage retries; and
- `docs/policy/architecture.md` adds only the durable rule that PromptKit
owns deterministic structural repair inside one completion while Notarius
owns stage attempts and semantic validation.
7. Remove current-behavior claims that PromptKit is v0.5.0, that every
production repair budget is zero, or that PromptKit is always single-pass.
Do not alter historical release notes or archived roadmaps.
8. Keep the maintained minimal and complete examples secret-free and valid.
They may omit the new field to demonstrate the default; do not add a
redundant complete profile or Rakestrawhome example merely to exercise an
upstream catalog entry.
9. Review `docs/roadmap/future.md` only for consistency. Leave the future
feedback-aware stage retry, combat-scene validator, and warning-reform work
unimplemented and clearly separate.
### Tests And Validation
Run focused tests first, then all repository checks:
```sh
GOWORK=off go test ./internal/framework/llm ./internal/framework/pipeline ./internal/framework/semanticreconcile ./internal/modules/dnd/...
GOWORK=off go test ./...
GOWORK=off go test -race ./...
GOWORK=off go vet ./...
GOWORK=off go build ./cmd/notarius
GOWORK=off go mod tidy -diff
go run ./cmd/notarius config validate \
--config examples/dnd-minimal.config.yml \
--pipeline dnd-session
go run ./cmd/notarius config validate \
--config examples/dnd-complete.config.yml \
--pipeline dnd-session
git diff --check
```
Also perform focused repository searches that exclude `docs/roadmap/archive/`
and historical release notes:
- no active v0.5.0 PromptKit pins or links remain;
- no maintained production prompt still declares `repair_attempts: 0`;
- every production `CompleteStructured` caller forwards the repair field; and
- no provider code, type, or message is added to ordinary errors, warnings,
manifests, cache, or checkpoint schemas.
If the repository's source-release checker is available and the ordinary
checks above pass, run `./scripts/check-release-source.sh v0.0.0` as the final
integrated validation. It must not create a tag, release note, or repository
artifact.
### Acceptance Criteria
- Every maintained structured prompt defaults to one corrective call and can
be overridden to zero through three at pipeline or binding scope.
- A real assembled Notarius completion follows the PromptKit v0.8.0 repair
contract without changing prompt schema identity or cacheable prefix.
- Actual repair count, cumulative usage, error classification, debug-only
provider diagnostics, scheduling, and checkpoint identity match the feature
roadmap.
- Profile inheritance, Rakestrawhome availability, optional credentials, and
v0.6.0 hardening remain covered and documented.
- All canonical documentation describes implemented v0.8.0 behavior in its
assigned home and leaves future semantic validation work in the roadmap.
- Maintained examples validate, all ordinary/race/vet/build/module checks pass,
and the worktree contains no generated or sensitive artifacts.

View File

@@ -1,520 +0,0 @@
# PromptKit v0.8.0 Upgrade
## Status
Proposed.
## Purpose
Upgrade Notarius from PromptKit v0.5.0 to v0.8.0 and deliberately adopt the
useful correctness, profile-composition, provider-diagnostic, backend, and
structured-output-repair capabilities introduced in PromptKit v0.6.0, v0.7.0,
and v0.8.0.
The upgrade should improve structured-output reliability without confusing
PromptKit's bounded deterministic repair with Notarius's existing stage retry
budget or the future feedback-aware semantic-validation loop. PromptKit types
and provider behavior must remain behind Notarius's transport-neutral LLM
boundary.
## Current State
Notarius currently pins PromptKit v0.5.0. Its production adapter prepares one
frozen execution, records credential-redacted details, and runs that same
prepared value. It maps PromptKit capacity failures to an application-owned
error, maps failed structured validation to `ErrInvalidStructuredOutput`, and
returns PromptKit's raw validated bytes and usage metadata.
Every maintained production prompt uses JSON Schema validation and currently
declares `repair_attempts: 0`. Notarius stage bindings separately expose
`retries`, which reruns a complete stage operation after an error or rejected
candidate. The two mechanisms have different ownership and must remain
independent.
Notarius also maintains:
- embedded prompt, schema, and fallback-profile filesystems;
- operator profile-file and profile-directory sources;
- one optional conventional `local` backend registration;
- explicit profile preflight through PromptKit inspection;
- one application-wide scheduled LLM client around the PromptKit adapter;
- PromptKit profile-source fingerprints for checkpoint safety; and
- redacted debug and manifest provenance at application-owned boundaries.
The upgrade must preserve those established responsibilities while revising
the pinned integration contract and any behavior affected by the three
intervening releases.
This roadmap is based on PromptKit's pinned release guides for
[v0.6.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/releases/v0.6.0.md),
[v0.7.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/releases/v0.7.0.md),
and
[v0.8.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/releases/v0.8.0.md),
plus the public API and format documentation at the v0.8.0 tag.
## Target End State
- `go.mod` and `go.sum` pin PromptKit v0.8.0 without a local replacement or
vendored copy.
- Every maintained PromptKit prompt and profile prepares successfully under
v0.8.0's stricter validation and source-loading rules.
- Eligible Notarius structured completions use one PromptKit corrective call by
default after a structurally invalid response. Operators can explicitly set
a value from zero through three for a configured pipeline, with a more local
LLM-backed binding override where needed.
- PromptKit repair remains an inner operation within one Notarius stage
attempt. It never consumes or replenishes the binding's `retries` budget.
- A successful repaired result exposes cumulative usage and the actual repair
count to Notarius's application-owned response and debug models. A repaired
success is not itself a warning.
- Exhausted PromptKit validation remains an invalid structured-output result,
preserving the final candidate and diagnostics for debug and for any
applicable outer Notarius stage policy. Invalid structured output is never
accepted merely because the repair budget was exhausted.
- Profile inheritance, the built-in Rakestrawhome backend/profile, optional
credential behavior, and structured generation errors work through the
existing Notarius PromptKit boundary and are accurately documented.
- Provider-specific PromptKit types do not escape `internal/framework/llm`.
- Checkpoint identity, effective configuration, redacted summaries, and debug
provenance reflect every execution-affecting repair or profile change.
- Current documentation pins and describes v0.8.0; future Notarius semantic
validation retries remain roadmap behavior rather than being conflated with
this dependency upgrade.
## Release-by-Release Adoption
### PromptKit v0.6.0: Correctness, Safety, And Efficiency
PromptKit v0.6.0 adds no public declarations, but intentionally rejects several
formerly permissive or ambiguous inputs. The upgrade must audit Notarius's
embedded and operator-facing integration against these rules:
- YAML `id` and `version` metadata, rather than filenames, define prompt and
profile identity.
- Prompt `content_file` paths are exact, relative, contained paths; built-in
file artifacts must resolve to regular files.
- execution controls, output contracts, and repair budgets must be finite and
within their documented ranges;
- provider endpoints must be absolute HTTP or HTTPS URLs with a host and no
user information, query, or fragment;
- JSON documents and successful provider responses contain exactly one value;
- successful provider responses are bounded to 16 MiB; and
- JSON-compatible values are bounded for depth and expansion.
Notarius should rely on PromptKit for these rules rather than duplicate its
parsers or internal limits. Existing Notarius validation may retain a narrower
application rule where it has independent value, but overlapping validation
must agree with PromptKit and must not accept a value PromptKit will reject
later.
The upgrade automatically receives operation-local schema-plan reuse,
artifact-text memoization, improved cancellation checks, and transport error
identity preservation. Notarius should verify these changes through its real
adapter boundary and avoid adding a second cache or response-body layer that
would duplicate PromptKit's ownership.
### PromptKit v0.7.0: Profiles, Backend Access, And Generation Errors
#### Profile Inheritance
Operator profiles may use `base_profile` to alias or selectively refine a
built-in, fallback, or higher-precedence operator profile. Notarius must pass
profile sources through unchanged and let PromptKit own parent lookup, merge
rules, source precedence, cycle detection, and fully resolved prepared targets.
Preflight inspection must resolve inherited profiles through the same source
and backend composition used at execution. The selected leaf profile ID remains
the public profile identity, while effective backend, endpoint, model, and
reasoning provenance reflect the resolved chain. Notarius must not implement a
second inheritance parser.
The existing complete `dnd-extraction` fallback remains a standalone profile:
PromptKit v0.8.0 does not provide a built-in `openai/gpt-5.6-luna` profile that
would be an appropriate parent. Documentation should nevertheless explain how
operators can use inheritance for environment-specific workload profiles and
should link to PromptKit's pinned format contract rather than duplicate its
field-by-field merge algorithm.
Checkpoint safety must cover inherited behavior. Operator file/directory
digests already cover changes to definitions in those sources, fallback asset
digests cover application parents, and the PromptKit built-in catalog marker
must change from its v0.5.0 identity to v0.8.0 so a changed built-in parent
cannot reuse an incompatible checkpoint.
#### Rakestrawhome Backend And Profile
PromptKit's reserved `rakestrawhome` backend and
`rakestrawhome-gemma-4-31b` profile become available without Notarius-specific
registration. Notarius must not register or shadow the reserved backend ID.
Profile preflight, backend-capacity reporting, scheduling, generation, and
provenance should work for it through the same generic paths used by OpenRouter
and `local`.
The D&D default remains `dnd-extraction`; this upgrade does not silently move a
production workload to Rakestrawhome. Operator documentation should identify
the built-in profile as an available selection and link to PromptKit for its
endpoint, credential environment, model, and capacity defaults.
#### Optional Credentials
An absent or blank optional `APIKeyEnv` now causes PromptKit to omit the
`Authorization` header and send the request. Notarius must not restore the old
failure behavior by pre-reading provider credential environment variables or
by adding provider-specific authentication logic.
Profile inspection may report an explicit `APIKeyRequired` policy without
reading the credential, and execution remains the boundary at which that
requirement is enforced. For optional profiles, an authentication-requiring
provider may instead return a structured 401 or 403 generation failure. The
configuration and operations documentation must explain this distinction.
Notarius does not currently expose PromptKit's in-memory profile-registration
API to operators, and PromptKit's filesystem profile format does not expose
`APIKeyRequired`; therefore Notarius must not promise that an operator profile
can force local credential preflight. Operators should provision the named
environment variable, while Notarius should preserve the provider's structured
authentication failure when it is absent.
Notarius must continue to document mechanisms and environment-variable names,
never secret values.
#### Structured Generation Errors
The adapter should recognize `*promptkit.GenerationError` with `errors.As` and
translate useful information into an immutable, provider-neutral Notarius
error classification. At minimum, retain the HTTP status code so callers and
future retry policy can distinguish transport success with provider rejection
from other generation failures.
PromptKit's provider code, type, and message accessors are bounded but remain
untrusted and potentially sensitive. They must never appear automatically in
ordinary CLI output, warnings, manifests, checkpoint identity, or cache data.
If retained for an explicitly requested debug trace, they must pass through
Notarius's known-secret and bearer redaction and remain clearly identified as
untrusted provider diagnostics. Default error formatting should continue to
use a bounded, redacted application-owned message.
Capacity and cancellation retain their current more specific classifications
and precedence. This upgrade does not add automatic provider-error retry
classification; it only preserves safe structured data needed for diagnosis
and later policy.
### PromptKit v0.8.0: Bounded Structured-Output Repair
#### Default Policy
Every maintained production prompt whose output is consumed as structured data
should declare one repair attempt. All current production prompts use eligible
JSON Schema validation, so no current prompt needs a zero default merely
because of its output mode.
One repair means at most one corrective generation after the initial
candidate. PromptKit reconstructs the immutable original conversation and
appends only the latest invalid assistant candidate and latest deterministic
validation diagnostics. It preserves the selected target, direct session ID,
provider-native structured-output contract, and backend capacity policy. This
shape preserves the original cacheable prompt prefix and avoids accumulating
unbounded failed history.
The default is deliberately small. A single repair captures the common case in
which a capable model can correct malformed JSON or a schema violation after
receiving an exact diagnostic, while bounding the extra latency and cost of a
single structured completion.
#### Configuration Contract
The public configuration is an optional, presence-aware
`structured_output_repair_attempts` integer at pipeline scope and at each
LLM-backed module or validator binding. Its effective precedence is:
1. the binding value, when present;
2. the pipeline value, when present; and
3. the selected prompt's declared `repair_attempts` value.
The value must be from zero through three. Explicit zero disables PromptKit
repair at that scope. A deterministic binding must reject the field because it
cannot perform structured LLM repair. Validator bindings may use it only when
the selected validator is LLM-backed. Shorthand module bindings continue to
inherit the pipeline or prompt default.
The long, provider-neutral name is intentional: it distinguishes PromptKit's
inner structural repair from the existing binding `retries` field, which owns
complete stage attempts, without exposing a dependency name in generic
pipeline contracts.
The effective value must survive file parsing, cloning, redacted summaries,
pipeline resolution, and pipeline digest construction without pointer aliasing
or loss of presence. It must affect checkpoint identity because it can change
the selected result, latency, token usage, and provider cost.
#### Adapter Contract
The transport-neutral structured-completion request should carry an optional
application-owned structural-repair budget. No `promptkit.OutputContract` or
other PromptKit type may cross the adapter boundary.
PromptKit v0.8.0 request validation replaces the complete prompt output
contract rather than merging one field. When Notarius has a configured
override, the adapter must therefore inspect the selected prompt, copy its
normalized declared format, validation mode, and schema path, change only the
repair count, and supply that complete contract on the prepared request. A nil
override continues to use the prompt declaration directly. Inspection and
preparation must use the same immutable engine sources; a small adapter-local
cache keyed by normalized prompt ID and version is acceptable but not required
without measured need.
This approach prevents configuration from accidentally dropping JSON Schema
validation, avoids duplicating schema paths in pipeline YAML, and keeps prompt
assets authoritative for every output-contract field other than the explicit
operator override.
The transport-neutral structured-completion response should report the actual
number of PromptKit repair calls. PromptKit's returned token usage is already
cumulative and must be passed through without re-summing it. Debug records
should distinguish the configured budget from the actual count. Ordinary run
manifests need not gain raw prompt or response data merely to report repairs;
any durable aggregate should be added only if it has a clear consumer contract.
#### Result And Failure Semantics
- A valid initial candidate returns normally with zero actual repairs.
- A valid corrected candidate returns normally with cumulative usage and its
positive actual repair count. It does not emit a warning solely because a
repair occurred.
- Exhausting the repair budget returns PromptKit's final candidate and failed
validation result. The adapter maps this to
`ErrInvalidStructuredOutput`, preserves the response and debug material, and
does not decode or accept the candidate.
- An explicitly empty or whitespace-only candidate participates in the
declared structural validation and repair flow. Missing, `null`, or
non-string provider content remains a malformed provider response.
- A generation failure during a corrective call is an operational generation
failure and uses the same safe structured-error adaptation as an initial
generation failure.
- Context cancellation remains authoritative throughout the initial and
corrective calls.
PromptKit repair happens inside one scheduled `CompleteStructured` operation.
The Notarius scheduler holds one permit for that logical operation while
PromptKit performs its initial and serial corrective calls; PromptKit
reacquires its own selected-backend capacity for each corrective generation.
Because corrective calls are serial, this cannot expand actual concurrent
provider work beyond the number of admitted Notarius operations, but
documentation must stop describing the Notarius permit as a separate admission
event for every internal repair call.
One `CompleteStructured` invocation with effective PromptKit repair budget `R`
may make at most `R + 1` provider calls. If one stage attempt makes `C`
structured-completion invocations, a binding with `retries: N` has an upper
bound of `(N + 1) * C * (R + 1)` provider calls; `C` may itself be a bounded,
data-dependent module property, as it is for batched semantic reconciliation.
LLM-backed validators have their own corresponding invocation counts, budgets,
and costs. These formulas are upper bounds, not promises that every failure is
retryable or that every attempt reaches the provider.
## Profile And Prompt Source Compatibility
The upgrade must preserve Notarius's source precedence: an operator source,
then registered application fallback profiles, then PromptKit built-ins. A
selected malformed definition remains authoritative and fails rather than
falling through. Parent resolution introduced by profile inheritance observes
that same precedence.
All embedded prompt manifests, shared content fragments, response schemas, and
fallback profiles must be prepared or inspected offline under v0.8.0. The
review should specifically catch:
- IDs inferred accidentally from filenames;
- stale or escaping `content_file` paths;
- missing or non-regular embedded artifacts;
- repair values outside zero through three or paired with ineligible
validation;
- schemas or examples that are not exact single JSON documents;
- unsupported endpoint forms; and
- JSON-compatible variables or profile extras that exceed upstream bounds.
No prompt prose, schema shape, durable D&D artifact contract, or default D&D
model should change merely to exercise the dependency. Prompt manifests should
change only as needed to enable the adopted repair default and satisfy v0.8.0
contracts.
## Provenance, Debugging, And Security
- Update the opaque PromptKit built-in profile-catalog identity from v0.5.0 to
v0.8.0. Do not hash or publish PromptKit's internal catalog bytes.
- Ensure a prompt's repair default remains covered by its existing prompt asset
fingerprint and a configured effective override remains covered by the
resolved pipeline digest.
- Preserve selected leaf profile identity while recording the inherited
effective target already exposed by PromptKit inspection and prepared
details.
- Add actual structural-repair count and, when useful, the configured budget to
application-owned debug material. Token totals remain PromptKit's cumulative
values.
- Do not generate a warning for a successful repair. Repair exhaustion is an
invalid-output failure, while provider rejection is a generation failure.
- Never expose raw provider diagnostic fields without explicit debug capture
and application redaction. Do not place them in normal errors or durable
summaries.
- Preserve context and transport error identity sufficiently for
`errors.Is`-based cancellation and deadline handling after adapting the
external error.
## Documentation And Examples
Implementation should update current-state documentation only when the new
behavior lands:
- `docs/integrations/pkg-promptkit.md` must pin v0.8.0 and define the revised
prepared-execution, repair, profile-inheritance, backend, credential, and
error-adaptation boundary.
- `docs/config.md` must own the repair configuration fields, precedence,
allowed range, explicit-zero behavior, profile inheritance availability, and
optional credential semantics.
- `docs/operations.md` must explain structural repair cost, timeout and
concurrency effects, credential failures, and its distinction from stage
retries.
- `docs/internal/llm.md` must describe adapter contract replacement, actual
repair metadata, error adaptation, source compatibility, and scheduling.
- `docs/internal/pipeline.md` must describe how effective repair configuration
is resolved and how inner repair differs from outer stage attempts.
- `docs/policy/architecture.md` should receive only the durable ownership rule:
PromptKit owns bounded deterministic structural repair within one completion,
while Notarius owns stage attempts and semantic validation policy. Detailed
fields and retry formulas belong in their canonical configuration and
operations documents.
Update maintained configuration examples only if the public Notarius
configuration contract changes. A short inheritance illustration may remain in
the configuration reference; do not create a complete example solely to copy
PromptKit's upstream profile catalog. All upstream links must point to the
v0.8.0 tag. Historical release or archived roadmap references should remain
historical.
No ADR is required solely to pin a newer dependency. The durable separation
between PromptKit structural repair and Notarius semantic stage retries should
be stated in architecture documentation now; the more extensive future
validation state machine still warrants the separate ADR already identified in
`future.md` when that work is promoted.
## Validation And Acceptance Criteria
The implementation is complete when:
- the repository builds and tests against PromptKit v0.8.0 with no replacement
directive, workspace dependency, or vendored source;
- every maintained prompt and profile prepares or inspects successfully under
the v0.8.0 source, path, endpoint, output-contract, and JSON-value rules;
- an invalid first JSON Schema candidate followed by a valid correction returns
the valid raw output, cumulative usage, and actual repair count through the
Notarius adapter;
- repair exhaustion returns the final raw candidate and debug material with an
error matching `ErrInvalidStructuredOutput`;
- a corrective generation failure retains safe generation classification and
provider status without leaking untrusted provider detail;
- explicit empty content follows structural validation rather than being
misclassified by Notarius;
- repair configuration is presence-aware, range checked, rejected on
deterministic bindings, resolved with documented precedence, and included in
effective pipeline identity;
- inherited profiles resolve consistently during preflight and execution, and
changes to any relevant operator, fallback, or built-in parent invalidate
checkpoint reuse;
- the Rakestrawhome built-in profile reaches generic preflight, scheduling, and
provenance paths without application-specific registration;
- optional missing credentials and explicitly required credentials behave as
documented without contacting real providers in tests;
- cancellation, timeout, backend capacity, prepared-execution snapshot,
session ID, raw-output, debug-redaction, and existing profile provenance
behavior remain intact;
- maintained examples validate successfully; and
- canonical documentation contains no active v0.5.0 pin or claim that PromptKit
is always single-pass.
Tests should follow `docs/policy/testing.md`: exercise observable Notarius
contracts with offline fake clients or `httptest` boundaries, and do not copy
PromptKit's entire internal repair test suite or assert its exact correction
message prose. The dependency's internal wording is not a Notarius contract.
## Non-Goals
- Implementing Notarius's future feedback-aware semantic stage-retry loop.
- Adding the D&D combat-scene semantic validator.
- Redesigning warning policy or treating successful structural repair as a
warning.
- Adding provider transport retries or deciding which HTTP statuses should
consume a stage retry.
- Exposing PromptKit request, response, profile, validation, capacity, or error
types outside the LLM adapter.
- Changing durable artifact schemas, D&D prompt semantics, the D&D default
model, or the fixed pipeline shape.
- Reimplementing PromptKit profile inheritance, schema validation, response
bounds, repair conversations, backend admission, or provider parsing inside
Notarius.
## Decisions
### 1. Default Structured-Output Repair Budget
**Decision: default to one repair attempt.** Set every maintained
eligible production prompt to `repair_attempts: 1`. One corrective call is a
strong fit for Notarius because every current production LLM response has a
strict JSON Schema contract, smaller cost-effective models are a deliberate
deployment target, and a precise structural diagnostic often makes one retry
materially more successful. The budget is paid only after a structurally
invalid candidate and remains tightly bounded.
**Alternative considered: retain zero by default.** This preserves single-pass
cost and latency and requires operators to opt in. It is preferable for an
environment where every additional request is expensive or where upstream
provider-native schema enforcement already produces negligible invalid output.
It is less suitable as the Notarius default because one malformed response can
otherwise discard substantial completed pipeline work.
**Alternative considered: default to two.** This may improve recovery for
weak models, but it doubles the worst-case corrective cost relative to the
selected default and compounds with outer stage retries. It should be an
operator choice supported by configuration, not the initial default, unless
observational evidence shows that the second correction has a worthwhile
marginal success rate.
### 2. Repair Override Scope
**Decision: support both pipeline and LLM-backed binding overrides.** Use
the presence-aware `structured_output_repair_attempts` field and precedence
defined above. A pipeline value provides the convenient one-line control the
operator requested, while a binding value permits an expensive normalizer or
future LLM-backed validator to use a deliberately different budget. This
mirrors Notarius's established pipeline/binding profile inheritance and scales
without editing embedded prompts.
**Alternative considered: support only a pipeline override.** This is smaller to
implement and document and still permits global enablement or disablement for
one pipeline. Its drawback is that one exceptional prompt cannot opt out or
request a larger budget without changing an embedded asset for every pipeline.
**Alternative considered: expose one global value under the top-level
`promptkit` configuration.** This makes client construction simple, but applies
the same budget to unrelated pipelines and leaks an execution policy into the
dependency configuration block. It is less compositional than pipeline-owned
policy and therefore not recommended.
### 3. Retention Of Provider-Supplied Generation Details
**Decision: retain status in the application-owned error contract and
retain redacted provider code, type, and message only in explicitly requested
debug traces.** Status is useful for diagnosis and future retry policy without
usually containing sensitive data. The other fields can materially explain a
400 response but may echo request or schema content, so they belong only in the
already-sensitive debug surface after Notarius redaction.
**Alternative considered: retain only HTTP status and discard all provider fields.**
This is the safest and smallest policy and still improves typed failure
handling. It sacrifices potentially decisive provider diagnostics, leaving an
operator with less information when a provider returns a terse status and the
problem cannot be reproduced easily.
**Alternative considered: include bounded provider code and type in normal
errors while keeping message debug-only.** Codes and types are often stable and
less sensitive than messages, but PromptKit explicitly classifies every
provider field as untrusted. Promoting them to ordinary output creates a
disclosure and compatibility burden that is not currently justified.

View File

@@ -1,282 +0,0 @@
# Source-Only Releases
## Status
Implemented. Creating the first release under this procedure remains a
separate maintainer operation.
## Purpose
Define a repeatable, guarded release process for Notarius without taking on a
binary-distribution system that its current operator audience does not need.
The process should make an exact source revision, its compatibility impact,
and its validation status easy to identify while keeping installation in the
hands of technically capable operators and deployment automation.
The model is adapted from Weatherreporter's release procedure, but its target
is deliberately narrower: an immutable source tag and checked-in release note
are the release. Notarius does not publish executable archives or support
Windows as part of this work.
## Release Model
Notarius releases come from commits on `main` and use stable semantic-version
tags in the form `vMAJOR.MINOR.PATCH`. Prerelease tags are not part of the
initial process.
Every release has one nonempty, version-matched note at
`docs/releases/<tag>.md`. The note and every affected current-state document
must be present in the tagged commit. The Git tag and checked-in note together
are the durable release record; no separately editable release page is
required.
Published tags are immutable. A maintainer must never move, reuse, or delete a
published tag. If a published candidate is defective, the correction is made
on `main` and released under a new patch version. An unpublished local tag may
be deleted when candidate inspection finds a problem before any remote push.
Before `v1.0.0`, a minor release may intentionally change a documented CLI,
configuration, durable artifact, integration, or operating contract when its
release note explains the impact and required operator action. A patch release
must not intentionally break those documented contracts within its minor
line.
The existing `v0.1.0`, `v0.2.0`, and `v0.3.0` tags remain unchanged. They
predate this procedure and do not need retrospective release notes. The first
release made under this process establishes the release-note series.
## Source-Only Distribution
Notarius does not publish release binaries, archives, installers, container
images, package-manager entries, checksum files, or signatures. A release tag
is suitable for Go-native installation and for an operator-controlled build
from an exact checkout.
The primary installation form is:
```sh
GOWORK=off go install \
gitea.maximumdirect.net/eric/notarius/cmd/notarius@vMAJOR.MINOR.PATCH
```
Operator documentation should also describe cloning the repository, checking
out the tag in detached-head state, and building `./cmd/notarius` with the Go
version declared by `go.mod`. Private-module authentication and `GOPRIVATE`
configuration belong to the operator environment and must be documented by
mechanism rather than with real credentials.
Consumers such as Narratio should pin the desired Notarius tag in provisioning
or deployment configuration. They must continue to decide runtime
compatibility from Notarius's published receipt and artifact schema contracts,
not merely from the executable's product version.
Packaged binaries may be reconsidered if distribution demand, installation
friction, or a broader user audience justifies their build, signing, retention,
and platform-support costs. They are not a prerequisite for a disciplined
release process.
## Platform Policy
Linux is the supported deployment platform. Release validation must run the
test suite and the release build on Linux and must confirm that the command
builds with `CGO_ENABLED=0` for Linux `amd64` and `arm64`.
macOS is a best-effort development and testing platform. Release validation
should confirm that the command cross-compiles with `CGO_ENABLED=0` for Darwin
`amd64` and `arm64`, but the project does not promise packaged artifacts or a
separate runtime test environment for those targets.
Windows is unsupported. The release process must not require Windows builds,
Windows-specific compatibility work, or Windows documentation. Platform-
specific implementation may intentionally use Unix facilities when they are
important to Notarius's filesystem safety and operational model. Any later
decision to support Windows requires its own feature scope and validation
policy.
## Version Reporting
Add a root `notarius --version` interface for deployment diagnostics. It
prints exactly one line:
```text
notarius vMAJOR.MINOR.PATCH
```
when the build has a valid release version, and:
```text
notarius development
```
when no release version is available.
The implementation must obtain the main-module version from Go build
information so `go install ...@vMAJOR.MINOR.PATCH` reports the selected tag. It
must also accept an optional link-time version override so controlled builds
and release CI can identify an exact tag from a checkout. The override must be
validated and must not silently turn arbitrary text into a release version.
Ordinary unversioned checkout builds remain `development`; the release process
must not modify a tracked source constant for each release.
Version reporting is an informational product interface. It does not replace
receipt, configuration, prompt, or artifact schema versioning, and it must not
be used as the sole downstream compatibility check.
## Release Notes
Each new `docs/releases/<tag>.md` document has this minimum structure:
```markdown
# Notarius vMAJOR.MINOR.PATCH
This release ...
## Summary
## Compatibility
## Upgrade
## Changes
```
The note should concisely explain the release's purpose, compatibility with the
preceding release, operator actions, and material user-visible, operational,
integration, and maintainer-visible changes. It should link to canonical
current-state documentation for exact contracts rather than duplicating those
contracts.
Release notes are durable historical summaries. They must not contain
credentials, private infrastructure detail, sensitive campaign material, or
claims that are not true of the tagged candidate. A release note does not
excuse stale current-state documentation; affected canonical documents are
updated in the same candidate.
## Candidate Validation
The release procedure must provide copyable POSIX-shell guards that validate
the release version, release-note filename and heading, required note sections,
repository state, and module hygiene. Validation must be run from the Notarius
repository root with Go workspace behavior disabled.
At minimum, a candidate must pass:
- no tracked `go.work` or `go.work.sum`, no vendored tree, and no `replace`
directive in `go.mod`;
- `GOWORK=off go test -count=1 ./...`;
- `GOWORK=off go test -race -count=1 ./...`;
- `GOWORK=off go vet ./...`;
- `GOWORK=off go build ./...`;
- `GOWORK=off go mod tidy -diff`;
- `gofmt` verification for every tracked Go file;
- `git diff --check` and `git diff --cached --check`;
- validation of both maintained D&D configuration examples with their selected
pipeline;
- Linux `amd64` and `arm64` static command builds;
- best-effort Darwin `amd64` and `arm64` static command builds; and
- a focused manual or automated check that every added or changed local
Markdown link resolves.
The candidate review also checks for generated binaries, test output,
credentials, temporary files, module replacements, vendored dependencies, and
other unintended source-control content. Tests remain offline and do not call
an LLM provider or require live credentials.
## Candidate Publication
The release procedure must guard the exact commit immediately before tagging.
It requires:
- the current branch is `main`;
- the worktree and index are clean;
- the candidate commit has been pushed and exactly matches `origin/main`;
- the matching release note exists in that commit;
- no local or remote tag already uses the selected version; and
- the substantive release checks have passed for that exact candidate.
The maintainer records the exact candidate commit, creates a lightweight tag
bound explicitly to that commit, verifies the local tag target, and pushes only
that tag ref. The procedure must not recommend `git push --tags`.
After publication, the maintainer verifies that the remote tag resolves to the
guarded commit and that the release note can be read from the tagged tree. A
fresh temporary checkout or `go install ...@<tag>` must build successfully, and
the resulting command must report the expected version through `--version`.
## Validation-Only Release Automation
Add a tag-triggered Woodpecker pipeline that validates source releases without
publishing artifacts. It should:
- accept only stable semantic-version tags;
- require the version-matched release note;
- run the same substantive module, test, race, vet, build, formatting, and
whitespace checks as the documented local procedure;
- validate the maintained configuration examples;
- perform the supported and best-effort cross-build checks; and
- verify a release-version build's `notarius --version` output on the CI host.
The pipeline must not upload binaries, create archives or checksums, create or
edit a Gitea release object, or require a release API token. Local guards remain
authoritative before tag publication because CI begins only after the tag is
already remote.
If tag validation fails, preserve the published tag, fix the cause on `main`,
select a new patch version, and repeat the full process. Do not weaken tag
immutability merely because the release contains source rather than binaries.
## Documentation Ownership
In the target state:
- `docs/release.md` owns the maintainer release procedure, commands, ordering,
publication checks, and failure recovery;
- `docs/releases/` owns one historical summary per release made under the new
process;
- `docs/cli.md` owns the `--version` contract;
- `README.md` owns the shortest source-installation example and links to the
release procedure where useful;
- `docs/development.md` routes release preparation, tagging, and verification
work to `docs/release.md`;
- `docs/policy/documentation.md` assigns canonical ownership to the release
procedure and release notes;
- `docs/policy/architecture.md` records Linux support, best-effort macOS
development, unsupported Windows, and source-only distribution only if those
are judged durable development invariants rather than release mechanics; and
- `docs/operations.md` describes only installation or deployment consequences
relevant to operators and links to canonical CLI and release contracts.
Current-state documentation must not describe the new release process,
`--version`, or automated validation until the corresponding behavior exists.
## Acceptance Criteria
- A maintainer can prepare, validate, tag, publish, and verify a source release
by following `docs/release.md` without relying on undocumented knowledge.
- Every new release has an immutable semantic-version tag and matching
checked-in release note in the tagged commit.
- The guarded candidate is clean, synchronized with `origin/main`, and passes
the documented substantive checks before tagging.
- Tag-triggered CI independently validates the published source and never
publishes binary artifacts.
- `go install` of a tagged version succeeds and `notarius --version` reports
that version; ordinary unversioned builds report `development`.
- Linux is the documented supported deployment platform, macOS has a
best-effort development build check, and Windows is explicitly unsupported.
- Downstream compatibility remains based on durable Notarius contracts rather
than the product version alone.
- Existing pre-procedure tags remain untouched and require no invented release
history.
## Non-Goals
- Publishing executable archives, installers, container images, checksums,
signatures, or package-manager entries.
- Supporting or cross-compiling for Windows.
- Creating or maintaining a mutable Gitea release page.
- Supporting prerelease tag syntax in the initial procedure.
- Automating version selection, release-note authorship, commits, or tag
creation.
- Retrospectively creating release notes for `v0.1.0` through `v0.3.0`.
- Treating a product version as a substitute for receipt, configuration,
prompt, or artifact schema compatibility.

7
go.mod
View File

@@ -3,9 +3,14 @@ module gitea.maximumdirect.net/eric/notarius
go 1.25.5
require (
gitea.maximumdirect.net/eric/promptkit v0.8.0
gitea.maximumdirect.net/eric/promptkit v0.9.0
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
gopkg.in/yaml.v3 v3.0.1
)
require golang.org/x/text v0.40.0
require (
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0 // indirect
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0 // indirect
)

8
go.sum
View File

@@ -1,5 +1,9 @@
gitea.maximumdirect.net/eric/promptkit v0.8.0 h1:NGd9hDLu0UMxKbvittMrqM5Ua94eFb+kOE7UIir8l08=
gitea.maximumdirect.net/eric/promptkit v0.8.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
gitea.maximumdirect.net/eric/promptkit v0.9.0 h1:IpvDRC8L6xRxQ9hpuyKOmMc5b6MeLTKYyx+h1YAjy08=
gitea.maximumdirect.net/eric/promptkit v0.9.0/go.mod h1:oMJ/WUJImUtwJ5e+6MAGECPYAErAkOaKel0G+3T/b4E=
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0 h1:lc062euk2qseO//D762i3JaFyulDNML3eQQX7DkYTho=
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0/go.mod h1:AIa7kAu2mfrRQgcspe4L+DW51WqgnALQT60lqkEywJI=
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0 h1:j9YY7wsTVjzke2kHH4YAzpU0oUpM+x+nXwl1IeS+2eg=
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0/go.mod h1:4RNS+LILDg4JbS4Ts9Lwy1C92wauXJIbeQaalps4Koo=
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=

View File

@@ -21,6 +21,10 @@ import (
const assembledSpellExtractorKey = "test/dnd/spell-casts"
const assembledCorrectingSpellExtractorKey = "test/dnd/correcting-spell-casts"
const assembledDirectSpellValidatorKey = "test/dnd/direct-spell-correction"
func TestAssembledSpellPipelineNormalizesMergedCasts(t *testing.T) {
registries, resolved, extractor := assembledSpellPipeline(t, assembledSpellPipelineOptions{})
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
@@ -101,6 +105,30 @@ func TestAssembledSpellPipelineNormalizesMergedCasts(t *testing.T) {
}
}
func TestAssembledSpellPipelineCorrectsRejectedDirectExtraction(t *testing.T) {
registries, resolved, extractor := assembledCorrectingSpellPipeline(t)
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
}
output, err := pipeline.New().Run(context.Background(), pipeline.RunInput{
Prepared: prepared,
RawInput: readRepositoryFile(t, "examples", "seriatim-minimal-transcript.json"),
ChunkCacheMode: pipeline.ChunkCacheBypass,
})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if len(output.Rejected) != 0 || output.Manifest.ValidationStatus != "approved" || len(output.NormalizeOutputs) != 1 {
t.Fatalf("run output = %#v, want corrected accepted spell output", output)
}
correction := extractor.correctionSnapshot()
if correction == nil || string(correction.AssistantResponse) != `{"spell":"Mysterious Burst"}` || !strings.Contains(correction.UserGuidance, "use a known spell name") || !strings.Contains(correction.UserGuidance, "complete corrected replacement") || strings.Contains(correction.UserGuidance, "unknown_spell") || strings.Contains(correction.UserGuidance, "spell is not in the catalog") {
t.Fatalf("extract correction = %#v, want exact rejected model response and semantic replacement guidance only", correction)
}
}
func TestAssembledSpellPipelineHonorsNormalizeValidatorOverride(t *testing.T) {
registries, resolved, _ := assembledSpellPipeline(t, assembledSpellPipelineOptions{normalizeValidatorOverride: true})
var normalizeChain *pipeline.ResolvedValidatorChain
@@ -136,6 +164,7 @@ func TestAssembledSpellPipelineHonorsNormalizeValidatorOverride(t *testing.T) {
func TestAssembledSpellPipelinePromotesTerminalUnknownSpellWarning(t *testing.T) {
registries, resolved, _ := assembledSpellPipeline(t, assembledSpellPipelineOptions{unknownSpell: true})
resolved.Steps[0].ArtifactLanes[0].NormalizeValidationPolicy.SemanticRejection = pipeline.SemanticRejectionRejectOutput
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
@@ -161,8 +190,8 @@ func TestAssembledSpellPipelinePromotesTerminalUnknownSpellWarning(t *testing.T)
if !reflect.DeepEqual(rejectedFile.Rejected, output.Rejected) {
t.Fatalf("rejected file = %#v, run rejections = %#v, want durable rejection diagnostic", rejectedFile.Rejected, output.Rejected)
}
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != spellnormalize.ReasonCodeSpellNameUnresolved || output.Warnings[0].Scope != "spell_casts[0]" {
t.Fatalf("warnings = %#v, want terminal normalize catalog warning", output.Warnings)
if len(output.Warnings) != 2 || output.Warnings[0].ReasonCode != spellnormalize.ReasonCodeSpellNameUnresolved || output.Warnings[0].Scope != "spell_casts[0]" || output.Warnings[1].ReasonCode != "spell_not_near_source" {
t.Fatalf("warnings = %#v, want complete terminal normalize validation warnings", output.Warnings)
}
}
@@ -206,6 +235,47 @@ type assembledSpellPipelineOptions struct {
unknownSpell bool
}
func assembledCorrectingSpellPipeline(t *testing.T) (pipeline.Registries, pipeline.ResolvedPipeline, *assembledCorrectingSpellExtractor) {
t.Helper()
components := productionTestComponents(t)
extractor := &assembledCorrectingSpellExtractor{}
if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{
Key: assembledCorrectingSpellExtractorKey,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
Requires: []string{"chunks", "source.transcript"},
Provides: []string{"dnd.spell_casts"},
ArtifactKind: dnd.SpellListKind,
}, func() (contracts.Extractor[dnd.SpellList], error) {
return extractor, nil
}); err != nil {
t.Fatalf("register correcting extractor: %v", err)
}
if err := pipeline.RegisterTypedValidator[dnd.SpellList](components.registries.Validators, dnd.SpellListKind, pipeline.ValidatorSpec{Key: assembledDirectSpellValidatorKey, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.TypedValidator[dnd.SpellList], error) {
return assembledDirectSpellValidator{}, nil
}); err != nil {
t.Fatalf("register direct spell validator: %v", err)
}
extract := pipeline.Binding(assembledCorrectingSpellExtractorKey)
extract.Retries = 1
extract.Validators = pipeline.ValidatorOverride{Set: true, Validators: []pipeline.ModuleBinding{{Module: assembledDirectSpellValidatorKey}}}
resolved, err := pipeline.ResolvePipeline(pipeline.PipelineProfile{
ID: "assembled-dnd-correcting-spells",
Input: pipeline.Binding("seriatim"),
Chunk: pipeline.ModuleBinding{Module: "generic", Options: map[string]any{"max_units": 1}},
Artifacts: map[string]pipeline.ArtifactLaneProfile{
"spells": {Extract: extract, Normalize: pipeline.Binding(spellnormalize.Key)},
},
Output: pipeline.Binding("json"),
}, pipeline.ResolveOptions{}, catalogFromRegistries(components.registries))
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
return components.registries, resolved, extractor
}
func assembledSpellPipeline(t *testing.T, options assembledSpellPipelineOptions) (pipeline.Registries, pipeline.ResolvedPipeline, *assembledSpellExtractor) {
t.Helper()
components := productionTestComponents(t)
@@ -251,6 +321,69 @@ type assembledSpellExtractor struct {
unknownSpell bool
}
type assembledCorrectingSpellExtractor struct {
mu sync.Mutex
correction *contracts.SemanticCorrection
}
func (*assembledCorrectingSpellExtractor) Key() string { return assembledCorrectingSpellExtractorKey }
func (*assembledCorrectingSpellExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (e *assembledCorrectingSpellExtractor) Extract(_ context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[dnd.SpellList], error) {
if req.Source == nil || req.Chunk == nil {
return contracts.TypedExtractionResult[dnd.SpellList]{}, fmt.Errorf("correcting assembled extractor requires source and chunk")
}
response := `{"spell":"accepted"}`
value := dnd.SpellList{SpellCasts: []dnd.SpellCast{}}
if req.Chunk.Index == 0 && req.Correction == nil {
response = `{"spell":"Mysterious Burst"}`
value.SpellCasts = []dnd.SpellCast{{Caster: "Aria", Spell: "Mysterious Burst", SourceRefs: []source.SourceRef{{SourceID: req.Source.ID, StartUnitID: 1, EndUnitID: 1}}}}
}
if req.Chunk.Index == 0 && req.Correction != nil {
correction, err := contracts.CloneSemanticCorrection(req.Correction)
if err != nil {
return contracts.TypedExtractionResult[dnd.SpellList]{}, err
}
e.mu.Lock()
e.correction = correction
e.mu.Unlock()
value.SpellCasts = []dnd.SpellCast{{Caster: "Aria", Spell: "Cure Wounds", SourceRefs: []source.SourceRef{{SourceID: req.Source.ID, StartUnitID: 1, EndUnitID: 1}}}}
}
candidate, err := contracts.NewModelCandidate([]byte(response), contracts.CorrectionProtocolSingleResponseV1)
if err != nil {
return contracts.TypedExtractionResult[dnd.SpellList]{}, err
}
return contracts.TypedExtractionResult[dnd.SpellList]{Value: value, ModelCandidate: candidate}, nil
}
func (e *assembledCorrectingSpellExtractor) correctionSnapshot() *contracts.SemanticCorrection {
e.mu.Lock()
defer e.mu.Unlock()
correction, err := contracts.CloneSemanticCorrection(e.correction)
if err != nil {
return nil
}
return correction
}
type assembledDirectSpellValidator struct{}
func (assembledDirectSpellValidator) Name() string { return assembledDirectSpellValidatorKey }
func (assembledDirectSpellValidator) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
func (assembledDirectSpellValidator) Validate(_ context.Context, req contracts.TypedValidationRequest[dnd.SpellList]) (contracts.ValidationResult, error) {
for _, cast := range req.Value.SpellCasts {
if cast.Spell == "Mysterious Burst" {
return contracts.ValidationResult{Approved: false, ReasonCode: "unknown_spell", Message: "spell is not in the catalog", CorrectionGuidance: "use a known spell name"}, nil
}
}
return contracts.ValidationResult{Approved: true}, nil
}
func (e *assembledSpellExtractor) Key() string { return assembledSpellExtractorKey }
func (*assembledSpellExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }

View File

@@ -396,8 +396,12 @@ func (client *enemyEventLLMClient) CompleteStructured(ctx context.Context, reque
if err := json.Unmarshal(content, output); err != nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate fake structured target: %w", err)
}
snapshot, err := contracts.CloneStructuredCompletionRequest(request)
if err != nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("clone fake request: %w", err)
}
client.mu.Lock()
client.requests = append(client.requests, request)
client.requests = append(client.requests, snapshot)
client.mu.Unlock()
return contracts.StructuredCompletionResponse{Content: content, Provider: "test", Model: "deterministic", ProfileID: request.ProfileID}, nil
}
@@ -408,7 +412,11 @@ func (client *enemyEventLLMClient) requestsFor(promptID string) []contracts.Stru
var requests []contracts.StructuredCompletionRequest
for _, request := range client.requests {
if request.PromptID == promptID {
requests = append(requests, request)
snapshot, err := contracts.CloneStructuredCompletionRequest(request)
if err != nil {
panic(err)
}
requests = append(requests, snapshot)
}
}
return requests

View File

@@ -343,6 +343,9 @@ func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
Inputs []struct {
Name string `yaml:"name"`
} `yaml:"inputs"`
Messages []struct {
Role string `yaml:"role"`
} `yaml:"messages"`
}
preparedPrompts := 0
if err := fs.WalkDir(promptFS, ".", func(path string, entry fs.DirEntry, walkErr error) error {
@@ -360,6 +363,11 @@ func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
if err := yaml.Unmarshal(data, &prompt); err != nil {
return err
}
for _, message := range prompt.Messages {
if message.Role != promptkit.RoleSystem && message.Role != promptkit.RoleUser {
return fmt.Errorf("production prompt %q uses role %q, want system or user", prompt.ID, message.Role)
}
}
inputs := make(map[string]promptkit.ArtifactRef, len(prompt.Inputs))
for _, input := range prompt.Inputs {
inputs[input.Name] = promptkit.Inline(`{}`)
@@ -1091,8 +1099,12 @@ func (client *productionFakeLLMClient) CompleteStructured(ctx context.Context, r
if err := json.Unmarshal(content, out); err != nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate fake structured target: %w", err)
}
snapshot, err := contracts.CloneStructuredCompletionRequest(req)
if err != nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("clone fake request: %w", err)
}
client.mu.Lock()
client.requests = append(client.requests, req)
client.requests = append(client.requests, snapshot)
client.mu.Unlock()
return contracts.StructuredCompletionResponse{Content: content, Provider: "test", Model: "deterministic", ProfileID: req.ProfileID}, nil
}
@@ -1103,7 +1115,11 @@ func (client *productionFakeLLMClient) requestsFor(promptID string) []contracts.
var requests []contracts.StructuredCompletionRequest
for _, req := range client.requests {
if req.PromptID == promptID {
requests = append(requests, req)
snapshot, err := contracts.CloneStructuredCompletionRequest(req)
if err != nil {
panic(err)
}
requests = append(requests, snapshot)
}
}
return requests

View File

@@ -7,22 +7,24 @@ import (
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const runResultSchemaVersion = "notarius.run-result.v1"
type runResult struct {
SchemaVersion string `json:"schema_version"`
RunID string `json:"run_id"`
PipelineID string `json:"pipeline_id"`
OutputDirectory string `json:"output_directory"`
IndexFile string `json:"index_file,omitempty"`
NormalizedOutputCount int `json:"normalized_output_count"`
RejectedOutputCount int `json:"rejected_output_count"`
WarningCount int `json:"warning_count"`
ValidationStatus string `json:"validation_status"`
DebugDirectory string `json:"debug_directory,omitempty"`
SchemaVersion string `json:"schema_version"`
RunID string `json:"run_id"`
PipelineID string `json:"pipeline_id"`
OutputDirectory string `json:"output_directory"`
IndexFile string `json:"index_file,omitempty"`
NormalizedOutputCount int `json:"normalized_output_count"`
RejectedOutputCount int `json:"rejected_output_count"`
WarningCount int `json:"warning_count"`
ValidationStatus string `json:"validation_status"`
ValidationSummaries []artifacts.ValidationSummary `json:"validation_summaries,omitempty"`
DebugDirectory string `json:"debug_directory,omitempty"`
}
func newRunResult(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput, outputDirectory, debugDirectory string) (runResult, error) {
@@ -59,6 +61,7 @@ func newRunResult(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput,
RejectedOutputCount: len(output.Rejected),
WarningCount: len(output.Warnings),
ValidationStatus: output.Manifest.ValidationStatus,
ValidationSummaries: cloneValidationSummaries(output.Manifest.ValidationSummaries),
}
if strings.TrimSpace(debugDirectory) != "" {
@@ -85,6 +88,17 @@ func newRunResult(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput,
return result, nil
}
func cloneValidationSummaries(summaries []artifacts.ValidationSummary) []artifacts.ValidationSummary {
if len(summaries) == 0 {
return nil
}
cloned := make([]artifacts.ValidationSummary, len(summaries))
for index, summary := range summaries {
cloned[index] = artifacts.CloneValidationSummary(summary)
}
return cloned
}
func encodeRunResult(result runResult) ([]byte, error) {
encoded, err := json.Marshal(result)
if err != nil {

View File

@@ -92,7 +92,7 @@ func TestRunResultReportsSuccessfulRejection(t *testing.T) {
if err != nil {
t.Fatal(err)
}
configBytes = []byte(replaceRequiredOnce(t, string(configBytes), " normalize: test/normalize\n", " normalize:\n module: test/normalize\n validators:\n - generic/always_reject\n"))
configBytes = []byte(replaceRequiredOnce(t, string(configBytes), " normalize: test/normalize\n", " normalize:\n module: test/normalize\n validators:\n - generic/always_reject\n validation_policy:\n semantic_rejection: reject_output\n"))
if err := os.WriteFile(roots.config, configBytes, 0o600); err != nil {
t.Fatal(err)
}

View File

@@ -55,6 +55,9 @@ func TestRunResultEncodesRequiredFieldsAndCounts(t *testing.T) {
if got := decoded["warning_count"]; got != float64(1) {
t.Fatalf("warning_count = %v", got)
}
if got := decoded["validation_summaries"]; got != nil {
t.Fatalf("validation_summaries = %#v, want omitted when empty", got)
}
if got := decoded["output_directory"]; got != filepath.Join(mustWorkingDirectory(t), "relative-output") {
t.Fatalf("output_directory = %q", got)
}
@@ -112,6 +115,26 @@ func TestRunResultOmitsIndexFileForOtherOutputModules(t *testing.T) {
}
}
func TestRunResultProjectsOwnedValidationSummaries(t *testing.T) {
output := testRunOutput()
output.Manifest.ValidationSummaries = []artifacts.ValidationSummary{{Status: "incomplete", IncompleteValidators: []string{"validator"}, ProducerAttemptCount: 1, TerminalAction: "warn_continue"}}
result, err := newRunResult(testResolvedPipeline(pipeline.DefaultOutputModule), output, "output", "")
if err != nil {
t.Fatal(err)
}
output.Manifest.ValidationSummaries[0].IncompleteValidators[0] = "caller mutation"
if got := result.ValidationSummaries[0].IncompleteValidators; len(got) != 1 || got[0] != "validator" {
t.Fatalf("result validation summaries = %#v", result.ValidationSummaries)
}
encoded, err := encodeRunResult(result)
if err != nil {
t.Fatal(err)
}
if !bytes.Contains(encoded, []byte(`"validation_summaries":[{"status":"incomplete","incomplete_validators":["validator"],"producer_attempt_count":1,"terminal_action":"warn_continue"}]`)) {
t.Fatalf("encoded result = %s", encoded)
}
}
func TestRunResultRequiresOneProductionIndexFile(t *testing.T) {
tests := []struct {
name string

View File

@@ -65,6 +65,7 @@ func TestProductionSpellCatalogValidationRetries(t *testing.T) {
t.Fatalf("materialize production references: %v", err)
}
materialized.Steps[0].ArtifactLanes[0].Extract.Retries = retries
materialized.Steps[0].ArtifactLanes[0].ExtractValidationPolicy.SemanticRejection = pipeline.SemanticRejectionRejectOutput
llmClient := &catalogRetryLLMClient{responses: tt.responses}
prepared, err := pipeline.Prepare(materialized, components.registries, pipeline.ModuleDependencies{LLM: llmClient})
@@ -91,8 +92,8 @@ func TestProductionSpellCatalogValidationRetries(t *testing.T) {
if rejection.ReasonCode != "unknown_spell" || rejection.AttemptCount != retries+1 {
t.Fatalf("rejection = %#v, want exhausted unknown-spell rejection", rejection)
}
if len(output.Warnings) != 0 {
t.Fatalf("warnings = %#v, want no emitted warnings from rejected attempts", output.Warnings)
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != "spell_not_near_source" {
t.Fatalf("warnings = %#v, want complete terminal validation warnings", output.Warnings)
}
return
}

View File

@@ -93,17 +93,44 @@ type NormalizedOutputManifest struct {
}
type RejectedOutputManifest struct {
Stage string `json:"stage"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"`
ChunkID string `json:"chunk_id,omitempty"`
ChunkIndex int `json:"chunk_index,omitempty"`
ValidatorName string `json:"validator_name,omitempty"`
ReasonCode string `json:"reason_code,omitempty"`
Message string `json:"message,omitempty"`
AttemptCount int `json:"attempt_count,omitempty"`
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
Stage string `json:"stage"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"`
ChunkID string `json:"chunk_id,omitempty"`
ChunkIndex int `json:"chunk_index,omitempty"`
ValidatorName string `json:"validator_name,omitempty"`
ReasonCode string `json:"reason_code,omitempty"`
Message string `json:"message,omitempty"`
AttemptCount int `json:"attempt_count,omitempty"`
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
Validation *ValidationSummary `json:"validation,omitempty"`
}
// ValidationSummary is the bounded, durable outcome of validating one
// producer result. It deliberately contains identities and stable codes, not
// model responses, corrective guidance, validator diagnostics, or payloads.
type ValidationSummary struct {
Stage string `json:"stage,omitempty"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"`
ChunkID string `json:"chunk_id,omitempty"`
ChunkIndex int `json:"chunk_index,omitempty"`
Status string `json:"status"`
RejectingValidators []string `json:"rejecting_validators,omitempty"`
ReasonCodes []string `json:"reason_codes,omitempty"`
IncompleteValidators []string `json:"incomplete_validators,omitempty"`
ProducerAttemptCount int `json:"producer_attempt_count"`
TerminalAction string `json:"terminal_action"`
}
// CloneValidationSummary returns an independently owned durable summary.
func CloneValidationSummary(summary ValidationSummary) ValidationSummary {
summary.RejectingValidators = append([]string(nil), summary.RejectingValidators...)
summary.ReasonCodes = append([]string(nil), summary.ReasonCodes...)
summary.IncompleteValidators = append([]string(nil), summary.IncompleteValidators...)
return summary
}
type CheckpointDecisionManifest struct {
@@ -163,6 +190,7 @@ type RunManifest struct {
References []ReferenceProvenance `json:"references,omitempty"`
NormalizedOutputs []NormalizedOutputManifest `json:"normalized_outputs,omitempty"`
RejectedOutputs []RejectedOutputManifest `json:"rejected_outputs,omitempty"`
ValidationSummaries []ValidationSummary `json:"validation_summaries,omitempty"`
CheckpointDecisions []CheckpointDecisionManifest `json:"checkpoint_decisions,omitempty"`
LLMProfiles []LLMProfileManifest `json:"llm_profiles,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`

View File

@@ -116,6 +116,7 @@ func (c *ConcurrencyConfig) recomputeStageWorkerDefaults() {
func clonePipelineProfile(in pipeline.PipelineProfile) pipeline.PipelineProfile {
out := in
out.ValidationPolicy = cloneValidationPolicyOverride(in.ValidationPolicy)
if in.StructuredOutputRepairAttempts != nil {
value := *in.StructuredOutputRepairAttempts
out.StructuredOutputRepairAttempts = &value
@@ -200,6 +201,7 @@ func cloneReferenceSource(in pipeline.ReferenceSource) pipeline.ReferenceSource
func cloneModuleBinding(in pipeline.ModuleBinding) pipeline.ModuleBinding {
out := in
out.ValidationPolicy = cloneValidationPolicyOverride(in.ValidationPolicy)
if in.StructuredOutputRepairAttempts != nil {
value := *in.StructuredOutputRepairAttempts
out.StructuredOutputRepairAttempts = &value
@@ -212,6 +214,26 @@ func cloneModuleBinding(in pipeline.ModuleBinding) pipeline.ModuleBinding {
return out
}
func cloneValidationPolicyOverride(in *pipeline.ValidationPolicyOverride) *pipeline.ValidationPolicyOverride {
if in == nil {
return nil
}
out := *in
if in.ProducerStructuralFailure != nil {
value := *in.ProducerStructuralFailure
out.ProducerStructuralFailure = &value
}
if in.SemanticRejection != nil {
value := *in.SemanticRejection
out.SemanticRejection = &value
}
if in.ValidatorFailure != nil {
value := *in.ValidatorFailure
out.ValidatorFailure = &value
}
return &out
}
func cloneValidatorOverride(in pipeline.ValidatorOverride) pipeline.ValidatorOverride {
out := pipeline.ValidatorOverride{Set: in.Set}
if len(in.Validators) > 0 {

View File

@@ -37,6 +37,7 @@ type FilePromptKitLocalBackendConfig struct {
type FilePipelineProfile struct {
LLMProfile *string `yaml:"llm_profile,omitempty"`
StructuredOutputRepairAttempts *int `yaml:"structured_output_repair_attempts,omitempty"`
ValidationPolicy *pipeline.ValidationPolicyOverride `yaml:"validation_policy,omitempty"`
Input fileModuleBinding `yaml:"input"`
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
@@ -55,12 +56,19 @@ func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
type plainFilePipelineProfile FilePipelineProfile
var decoded plainFilePipelineProfile
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
"llm_profile": {}, "structured_output_repair_attempts": {}, "input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
"llm_profile": {}, "structured_output_repair_attempts": {}, "validation_policy": {}, "input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
}, "pipeline profile")
if err != nil {
return err
}
*p = FilePipelineProfile(decoded)
if validationPolicyNode, ok := mappingValue(node, "validation_policy"); ok {
policy, err := parseValidationPolicy(validationPolicyNode, "pipeline profile")
if err != nil {
return err
}
p.ValidationPolicy = policy
}
_, p.artifactsSet = seen["artifacts"]
_, p.stepsSet = seen["steps"]
_, p.llmProfileSet = seen["llm_profile"]
@@ -154,6 +162,7 @@ type fileModuleBinding struct {
Module string
LLMProfile string
StructuredOutputRepairAttempts *int
ValidationPolicy *pipeline.ValidationPolicyOverride
Retries int
Options map[string]any
References map[string]fileReferenceSource
@@ -250,9 +259,14 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
b.Module = strings.TrimSpace(module)
return nil
case yaml.MappingNode:
seen := make(map[string]struct{}, len(node.Content)/2)
for i := 0; i < len(node.Content); i += 2 {
keyNode := node.Content[i]
valueNode := node.Content[i+1]
if _, exists := seen[keyNode.Value]; exists {
return fmt.Errorf("module binding field %q is duplicated", keyNode.Value)
}
seen[keyNode.Value] = struct{}{}
switch keyNode.Value {
case "module":
var module string
@@ -275,6 +289,12 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
return err
}
b.StructuredOutputRepairAttempts = attempts
case "validation_policy":
policy, err := parseValidationPolicy(valueNode, "module binding")
if err != nil {
return err
}
b.ValidationPolicy = policy
case "retries":
var retries int
if err := valueNode.Decode(&retries); err != nil {
@@ -318,6 +338,7 @@ func (b fileModuleBinding) toPipelineBinding() pipeline.ModuleBinding {
Module: strings.TrimSpace(b.Module),
LLMProfile: strings.TrimSpace(b.LLMProfile),
StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(b.StructuredOutputRepairAttempts),
ValidationPolicy: cloneValidationPolicyOverride(b.ValidationPolicy),
Retries: b.Retries,
Options: cloneOptions(b.Options),
References: fileReferenceSourcesToPipeline(b.References),
@@ -325,6 +346,51 @@ func (b fileModuleBinding) toPipelineBinding() pipeline.ModuleBinding {
}
}
func mappingValue(node *yaml.Node, key string) (*yaml.Node, bool) {
for i := 0; i < len(node.Content); i += 2 {
if node.Content[i].Value == key {
return node.Content[i+1], true
}
}
return nil, false
}
func parseValidationPolicy(node *yaml.Node, context string) (*pipeline.ValidationPolicyOverride, error) {
if node == nil || node.Tag == "!!null" || node.Kind != yaml.MappingNode {
return nil, fmt.Errorf("%s validation_policy must be an object", context)
}
policy := &pipeline.ValidationPolicyOverride{}
seen := make(map[string]struct{}, len(node.Content)/2)
for i := 0; i < len(node.Content); i += 2 {
key := node.Content[i].Value
value := node.Content[i+1]
if _, exists := seen[key]; exists {
return nil, fmt.Errorf("%s validation_policy field %q is duplicated", context, key)
}
seen[key] = struct{}{}
if value.Tag == "!!null" || value.Kind != yaml.ScalarNode || value.Tag != "!!str" {
return nil, fmt.Errorf("%s validation_policy.%s must be a string", context, key)
}
switch key {
case "producer_structural_failure":
value := pipeline.ProducerStructuralFailureAction(value.Value)
policy.ProducerStructuralFailure = &value
case "semantic_rejection":
value := pipeline.SemanticRejectionAction(value.Value)
policy.SemanticRejection = &value
case "validator_failure":
value := pipeline.ValidatorFailureAction(value.Value)
policy.ValidatorFailure = &value
default:
return nil, fmt.Errorf("field %s not found in %s validation_policy", key, context)
}
}
if err := policy.Validate(); err != nil {
return nil, fmt.Errorf("%s validation_policy: %w", context, err)
}
return policy, nil
}
func validateStructuredOutputRepairAttemptsNode(node *yaml.Node, context string) error {
if node.Kind != yaml.MappingNode {
return fmt.Errorf("%s must be an object", context)
@@ -573,6 +639,7 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
ID: pipelineID,
LLMProfile: llmProfile,
StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(filePipeline.StructuredOutputRepairAttempts),
ValidationPolicy: cloneValidationPolicyOverride(filePipeline.ValidationPolicy),
Input: filePipeline.Input.toPipelineBinding(),
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
References: fileReferenceSourcesToPipeline(filePipeline.References),

View File

@@ -32,6 +32,7 @@ func (e EffectiveConfig) RedactedResolvedPipelinePayload() pipeline.ResolvedPipe
func cloneResolvedPipeline(in pipeline.ResolvedPipeline) pipeline.ResolvedPipeline {
out := in
out.ConfiguredValidationPolicy = cloneValidationPolicyOverride(in.ConfiguredValidationPolicy)
out.Input = redactBinding(cloneModuleBinding(in.Input))
out.Chunk = redactBinding(cloneModuleBinding(in.Chunk))
out.ChunkReferences = pipeline.CloneReferenceTarget(in.ChunkReferences)

View File

@@ -225,6 +225,39 @@ func TestRedactedResolvedPipelinePayloadHandlesTypedOptionContainers(t *testing.
}
}
func TestRedactedEffectiveConfigPayloadOwnsValidationPolicies(t *testing.T) {
semantic := pipeline.SemanticRejectionRejectOutput
validator := pipeline.ValidatorFailureFailRun
configured := &pipeline.ValidationPolicyOverride{SemanticRejection: &semantic, ValidatorFailure: &validator}
effective := EffectiveConfig{
Config: Config{Pipelines: map[string]pipeline.PipelineProfile{
"main": {ValidationPolicy: configured},
}},
ResolvedPipeline: pipeline.ResolvedPipeline{
ConfiguredValidationPolicy: configured,
ChunkValidationPolicy: pipeline.ValidationPolicy{
ProducerStructuralFailure: pipeline.ProducerStructuralFailureFailRun,
SemanticRejection: pipeline.SemanticRejectionRejectOutput,
ValidatorFailure: pipeline.ValidatorFailureFailRun,
},
},
}
payload := effective.RedactedSummaryPayload().(EffectiveConfig)
encoded, err := json.Marshal(payload)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(encoded), `"configured_validation_policy":{"semantic_rejection":"reject_output","validator_failure":"fail_run"}`) || !strings.Contains(string(encoded), `"chunk_validation_policy":{"producer_structural_failure":"fail_run","semantic_rejection":"reject_output","validator_failure":"fail_run"}`) {
t.Fatalf("redacted payload omitted validation policy: %s", encoded)
}
*payload.Config.Pipelines["main"].ValidationPolicy.SemanticRejection = pipeline.SemanticRejectionFailRun
*payload.ResolvedPipeline.ConfiguredValidationPolicy.ValidatorFailure = pipeline.ValidatorFailureWarnContinue
if *effective.Config.Pipelines["main"].ValidationPolicy.SemanticRejection != pipeline.SemanticRejectionRejectOutput || *effective.ResolvedPipeline.ConfiguredValidationPolicy.ValidatorFailure != pipeline.ValidatorFailureFailRun {
t.Fatal("redacted payload aliases validation policy")
}
}
func redactionTestBinding(name string) pipeline.ModuleBinding {
return pipeline.ModuleBinding{
Module: "safe-" + name,

View File

@@ -119,6 +119,11 @@ func validatePipelineProfiles(profiles map[string]pipeline.PipelineProfile) erro
if err := validateStructuredOutputRepairAttempts(fmt.Sprintf("pipeline %q", id), profile.StructuredOutputRepairAttempts); err != nil {
return err
}
if profile.ValidationPolicy != nil {
if err := profile.ValidationPolicy.Validate(); err != nil {
return fmt.Errorf("pipeline %q validation_policy: %w", id, err)
}
}
if err := validateBinding(id, "", "input", profile.Input, false); err != nil {
return err
}
@@ -198,6 +203,16 @@ func validateBinding(
binding pipeline.ModuleBinding,
referencesAllowed bool,
) error {
if binding.ValidationPolicy != nil {
switch slot {
case "chunk", "extract", "merge", "normalize":
default:
return fmt.Errorf("%s validation_policy is not supported", referenceContext(pipelineID, laneID, slot))
}
if err := binding.ValidationPolicy.Validate(); err != nil {
return fmt.Errorf("%s validation_policy: %w", referenceContext(pipelineID, laneID, slot), err)
}
}
if err := validateStructuredOutputRepairAttempts(referenceContext(pipelineID, laneID, slot), binding.StructuredOutputRepairAttempts); err != nil {
return err
}
@@ -255,8 +270,11 @@ func validateValidatorOverride(pipelineID string, laneID string, slot string, ov
if validator.Validators.Set {
return fmt.Errorf("%s nested validators are not supported", context)
}
if validator.Retries != 0 {
return fmt.Errorf("%s retries are not supported", context)
if validator.ValidationPolicy != nil {
return fmt.Errorf("%s validation_policy is not supported", context)
}
if validator.Retries < 0 {
return fmt.Errorf("%s retries must be greater than or equal to zero", context)
}
if validator.LLMProfile != "" && strings.TrimSpace(validator.LLMProfile) == "" {
return fmt.Errorf("%s llm_profile must not be empty when set", context)

View File

@@ -374,17 +374,17 @@ func TestValidateValidatorBindingRules(t *testing.T) {
want: "chunk validators[0] module must not be empty",
},
{
name: "validator retries",
name: "negative validator retries",
setup: func(profile *pipeline.PipelineProfile) {
profile.Chunk.Validators = pipeline.ValidatorOverride{
Set: true,
Validators: []pipeline.ModuleBinding{{
Module: "validator",
Retries: 1,
Retries: -1,
}},
}
},
want: "chunk validators[0] retries are not supported",
want: "chunk validators[0] retries must be greater than or equal to zero",
},
{
name: "validator references",

View File

@@ -0,0 +1,97 @@
package config
import (
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestValidationPolicyFileConfigurationIsStrictAndPresenceAware(t *testing.T) {
const valid = `version: 4
pipelines:
main:
validation_policy:
producer_structural_failure: reject_output
semantic_rejection: fail_run
input: seriatim
chunk:
module: generic
validation_policy:
validator_failure: fail_run
artifacts:
lane:
extract:
module: extract
validation_policy:
semantic_rejection: reject_output
`
cfg := applyFileConfig(t, valid)
profile := cfg.Pipelines["main"]
if profile.ValidationPolicy == nil || profile.ValidationPolicy.ProducerStructuralFailure == nil || *profile.ValidationPolicy.ProducerStructuralFailure != pipeline.ProducerStructuralFailureRejectOutput || profile.ValidationPolicy.SemanticRejection == nil || *profile.ValidationPolicy.SemanticRejection != pipeline.SemanticRejectionFailRun || profile.ValidationPolicy.ValidatorFailure != nil {
t.Fatalf("pipeline validation policy = %#v", profile.ValidationPolicy)
}
if profile.Chunk.ValidationPolicy == nil || profile.Chunk.ValidationPolicy.ValidatorFailure == nil || *profile.Chunk.ValidationPolicy.ValidatorFailure != pipeline.ValidatorFailureFailRun {
t.Fatalf("chunk validation policy = %#v", profile.Chunk.ValidationPolicy)
}
lane := profile.Artifacts["lane"]
if lane.Extract.ValidationPolicy == nil || lane.Extract.ValidationPolicy.SemanticRejection == nil || *lane.Extract.ValidationPolicy.SemanticRejection != pipeline.SemanticRejectionRejectOutput {
t.Fatalf("extract validation policy = %#v", lane.Extract.ValidationPolicy)
}
for _, test := range []struct {
name string
yaml string
}{
{"null object", strings.Replace(valid, "validation_policy:\n producer_structural_failure: reject_output\n semantic_rejection: fail_run", "validation_policy: null", 1)},
{"null field", strings.Replace(valid, "semantic_rejection: fail_run", "semantic_rejection: null", 1)},
{"unknown field", strings.Replace(valid, "semantic_rejection: fail_run", "unknown: fail_run", 1)},
{"duplicate field", strings.Replace(valid, "semantic_rejection: fail_run", "semantic_rejection: fail_run\n semantic_rejection: reject_output", 1)},
{"invalid enum", strings.Replace(valid, "semantic_rejection: fail_run", "semantic_rejection: continue", 1)},
} {
t.Run(test.name, func(t *testing.T) {
if _, err := ParseFileConfigYAML([]byte(test.yaml)); err == nil {
t.Fatal("ParseFileConfigYAML() error = nil, want strict validation-policy rejection")
}
})
}
}
func TestValidationPolicyPlacementRules(t *testing.T) {
policy := &pipeline.ValidationPolicyOverride{}
semantic := pipeline.SemanticRejectionRejectOutput
policy.SemanticRejection = &semantic
base := pipeline.PipelineProfile{
ID: "main",
Input: pipeline.Binding("input"),
Artifacts: map[string]pipeline.ArtifactLaneProfile{
"lane": {Extract: pipeline.Binding("extract")},
},
}
for _, test := range []struct {
name string
mutate func(*pipeline.PipelineProfile)
}{
{"input", func(profile *pipeline.PipelineProfile) { profile.Input.ValidationPolicy = policy }},
{"output", func(profile *pipeline.PipelineProfile) {
profile.Output = pipeline.Binding("output")
profile.Output.ValidationPolicy = policy
}},
{"validator", func(profile *pipeline.PipelineProfile) {
lane := profile.Artifacts["lane"]
lane.Extract.Validators = pipeline.ValidatorOverride{Set: true, Validators: []pipeline.ModuleBinding{{Module: "validator", ValidationPolicy: policy}}}
profile.Artifacts["lane"] = lane
}},
} {
t.Run(test.name, func(t *testing.T) {
profile := base
profile.Artifacts = map[string]pipeline.ArtifactLaneProfile{"lane": base.Artifacts["lane"]}
test.mutate(&profile)
cfg := Default()
cfg.Pipelines = map[string]pipeline.PipelineProfile{"main": profile}
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "validation_policy") {
t.Fatalf("Config.Validate() error = %v, want placement rejection", err)
}
})
}
}

View File

@@ -10,6 +10,7 @@ import (
"strings"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/fileio"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
@@ -344,7 +345,15 @@ func cloneRejectedOutputs(rejected []contracts.RejectedOutput) []contracts.Rejec
if len(rejected) == 0 {
return nil
}
return append([]contracts.RejectedOutput(nil), rejected...)
cloned := make([]contracts.RejectedOutput, len(rejected))
for index, item := range rejected {
cloned[index] = item
if item.Validation != nil {
summary := artifacts.CloneValidationSummary(*item.Validation)
cloned[index].Validation = &summary
}
}
return cloned
}
func cloneMetadata(metadata map[string]any) map[string]any {

View File

@@ -0,0 +1,84 @@
package contracts
import (
"crypto/sha256"
"encoding/hex"
"fmt"
)
// DebugStructuredCompletionRequest is the content-safe representation of a
// structured completion request for ordinary diagnostics and summaries.
// Detailed prompt material remains available only through the explicitly
// requested LLM debug trace.
type DebugStructuredCompletionRequest struct {
StageName string `json:"stage_name,omitempty"`
PromptID string `json:"prompt_id,omitempty"`
PromptVersion string `json:"prompt_version,omitempty"`
ProfileID string `json:"profile_id,omitempty"`
SessionID string `json:"session_id,omitempty"`
InputCount int `json:"input_count"`
VariableCount int `json:"variable_count"`
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
Correction *DebugSemanticCorrection `json:"correction,omitempty"`
}
// DebugSemanticCorrection records only safe correction metadata. It never
// exposes the assistant response or user guidance text.
type DebugSemanticCorrection struct {
AssistantResponseBytes int `json:"assistant_response_bytes"`
AssistantResponseDigest string `json:"assistant_response_digest"`
UserGuidanceBytes int `json:"user_guidance_bytes"`
UserGuidanceDigest string `json:"user_guidance_digest"`
}
// DebugSummary returns a content-safe representation suitable for ordinary
// diagnostics. It does not validate or retain correction content.
func (request StructuredCompletionRequest) DebugSummary() DebugStructuredCompletionRequest {
summary := DebugStructuredCompletionRequest{
StageName: request.StageName,
PromptID: request.PromptID,
PromptVersion: request.PromptVersion,
ProfileID: request.ProfileID,
SessionID: request.SessionID,
InputCount: len(request.Inputs),
VariableCount: len(request.Vars),
}
if request.StructuredOutputRepairAttempts != nil {
attempts := *request.StructuredOutputRepairAttempts
summary.StructuredOutputRepairAttempts = &attempts
}
if request.Correction != nil {
summary.Correction = request.Correction.DebugSummary()
}
return summary
}
// DebugSummary returns content-safe correction metadata suitable for ordinary
// diagnostics.
func (correction *SemanticCorrection) DebugSummary() *DebugSemanticCorrection {
if correction == nil {
return nil
}
return &DebugSemanticCorrection{
AssistantResponseBytes: len(correction.AssistantResponse),
AssistantResponseDigest: debugContentDigest(correction.AssistantResponse),
UserGuidanceBytes: len(correction.UserGuidance),
UserGuidanceDigest: debugContentDigest([]byte(correction.UserGuidance)),
}
}
// String prevents ordinary request formatting from exposing correction
// content. Use the explicitly requested debug trace for complete messages.
func (request StructuredCompletionRequest) String() string {
return fmt.Sprintf("%+v", request.DebugSummary())
}
// GoString gives %#v formatting the same content-safe behavior as String.
func (request StructuredCompletionRequest) GoString() string {
return request.String()
}
func debugContentDigest(content []byte) string {
sum := sha256.Sum256(content)
return "sha256:" + hex.EncodeToString(sum[:])
}

View File

@@ -9,14 +9,15 @@ import (
)
type StructuredCompletionRequest struct {
StageName string `json:"stage_name"`
PromptID string `json:"prompt_id,omitempty"`
PromptVersion string `json:"prompt_version,omitempty"`
ProfileID string `json:"profile_id,omitempty"`
SessionID string `json:"session_id,omitempty"`
Inputs LLMInputSet `json:"inputs,omitempty"`
Vars map[string]any `json:"vars,omitempty"`
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
StageName string `json:"stage_name"`
PromptID string `json:"prompt_id,omitempty"`
PromptVersion string `json:"prompt_version,omitempty"`
ProfileID string `json:"profile_id,omitempty"`
SessionID string `json:"session_id,omitempty"`
Inputs LLMInputSet `json:"inputs,omitempty"`
Vars map[string]any `json:"vars,omitempty"`
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
Correction *SemanticCorrection `json:"-"`
}
type StructuredCompletionResponse struct {
@@ -156,12 +157,14 @@ type ChunkRequest struct {
References ReferenceSet `json:"references,omitempty"`
LLMProfile string `json:"llm_profile,omitempty"`
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
Correction *SemanticCorrection `json:"-"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type ChunkPlanResult struct {
Plan source.ChunkPlan `json:"plan"`
Warnings []Warning `json:"warnings,omitempty"`
Plan source.ChunkPlan `json:"plan"`
Warnings []Warning `json:"warnings,omitempty"`
ModelCandidate *ModelCandidate `json:"-"`
}
type Chunker interface {
@@ -283,6 +286,7 @@ type ValidationResult struct {
Approved bool `json:"approved"`
ReasonCode string `json:"reason_code,omitempty"`
Message string `json:"message,omitempty"`
CorrectionGuidance string `json:"-"`
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
Warnings []Warning `json:"warnings,omitempty"`
}
@@ -322,17 +326,18 @@ type OutputEncoder interface {
}
type RejectedOutput struct {
Stage string `json:"stage"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"`
ChunkID string `json:"chunk_id,omitempty"`
ChunkIndex int `json:"chunk_index,omitempty"`
ValidatorName string `json:"validator_name,omitempty"`
ReasonCode string `json:"reason_code,omitempty"`
Message string `json:"message"`
AttemptCount int `json:"attempt_count,omitempty"`
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
Stage string `json:"stage"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"`
ChunkID string `json:"chunk_id,omitempty"`
ChunkIndex int `json:"chunk_index,omitempty"`
ValidatorName string `json:"validator_name,omitempty"`
ReasonCode string `json:"reason_code,omitempty"`
Message string `json:"message"`
AttemptCount int `json:"attempt_count,omitempty"`
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
Validation *artifacts.ValidationSummary `json:"validation,omitempty"`
}
type ManifestMetadataProvider interface {

View File

@@ -0,0 +1,175 @@
package contracts
import (
"errors"
"fmt"
"strings"
"unicode/utf8"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
)
// CorrectionProtocol identifies how a producer can represent the model output
// that directly controlled a candidate.
type CorrectionProtocol string
const (
CorrectionProtocolSingleResponseV1 CorrectionProtocol = "single_response_v1"
)
const (
MaxValidationReasonCodeBytes = 128
MaxValidationCorrectionGuidanceBytes = 4 * 1024
MaxAssistantResponseBytes = 1 << 20
MaxCorrectionGuidanceBytes = 64 * 1024
MaxCorrectionContentBytes = MaxAssistantResponseBytes + MaxCorrectionGuidanceBytes
)
func (protocol CorrectionProtocol) Validate() error {
if protocol == CorrectionProtocolSingleResponseV1 {
return nil
}
return fmt.Errorf("unsupported correction protocol %q", protocol)
}
// SemanticCorrection carries the latest model response and application-owned
// guidance for one fresh corrected request. Its content is sensitive and is
// deliberately excluded from ordinary JSON serialization.
type SemanticCorrection struct {
AssistantResponse []byte `json:"-"`
UserGuidance string `json:"-"`
}
func NewSemanticCorrection(assistantResponse []byte, userGuidance string) (*SemanticCorrection, error) {
correction := &SemanticCorrection{
AssistantResponse: append([]byte(nil), assistantResponse...),
UserGuidance: userGuidance,
}
if err := correction.Validate(); err != nil {
return nil, err
}
return correction, nil
}
func CloneSemanticCorrection(correction *SemanticCorrection) (*SemanticCorrection, error) {
if correction == nil {
return nil, nil
}
return NewSemanticCorrection(correction.AssistantResponse, correction.UserGuidance)
}
// CloneStructuredCompletionRequest returns a request whose mutable values are
// owned by the caller. It is suitable for clients that retain requests after
// CompleteStructured returns.
func CloneStructuredCompletionRequest(request StructuredCompletionRequest) (StructuredCompletionRequest, error) {
correction, err := CloneSemanticCorrection(request.Correction)
if err != nil {
return StructuredCompletionRequest{}, fmt.Errorf("clone correction: %w", err)
}
vars, err := source.CloneMetadata(request.Vars)
if err != nil {
return StructuredCompletionRequest{}, fmt.Errorf("clone variables: %w", err)
}
request.Inputs = request.Inputs.Clone()
request.Vars = vars
request.Correction = correction
if request.StructuredOutputRepairAttempts != nil {
attempts := *request.StructuredOutputRepairAttempts
request.StructuredOutputRepairAttempts = &attempts
}
return request, nil
}
func (correction SemanticCorrection) Validate() error {
if err := validateAssistantResponse(correction.AssistantResponse); err != nil {
return fmt.Errorf("semantic correction assistant response: %w", err)
}
if err := validateBoundedText(correction.UserGuidance, MaxCorrectionGuidanceBytes, "semantic correction user guidance", false); err != nil {
return err
}
if len(correction.AssistantResponse)+len(correction.UserGuidance) > MaxCorrectionContentBytes {
return errors.New("semantic correction content exceeds maximum length")
}
return nil
}
// ModelCandidate preserves the exact single model response that directly
// controlled a producer result. It is attempt-local and never durable data.
type ModelCandidate struct {
Response []byte `json:"-"`
Protocol CorrectionProtocol `json:"-"`
}
func NewModelCandidate(response []byte, protocol CorrectionProtocol) (*ModelCandidate, error) {
candidate := &ModelCandidate{Response: append([]byte(nil), response...), Protocol: protocol}
if err := candidate.Validate(); err != nil {
return nil, err
}
return candidate, nil
}
func CloneModelCandidate(candidate *ModelCandidate) (*ModelCandidate, error) {
if candidate == nil {
return nil, nil
}
return NewModelCandidate(candidate.Response, candidate.Protocol)
}
func (candidate ModelCandidate) Validate() error {
if err := candidate.Protocol.Validate(); err != nil {
return err
}
if err := validateAssistantResponse(candidate.Response); err != nil {
return fmt.Errorf("model candidate response: %w", err)
}
return nil
}
func ValidateValidationResult(result ValidationResult) error {
if !result.Approved && result.ReasonCode == "" {
return errors.New("validation rejection reason code must not be empty")
}
if result.ReasonCode != "" {
if err := validateBoundedText(result.ReasonCode, MaxValidationReasonCodeBytes, "validation reason code", false); err != nil {
return err
}
}
if !result.Approved && result.CorrectionGuidance == "" {
return errors.New("validation rejection correction guidance must not be empty")
}
if result.CorrectionGuidance != "" {
if err := validateBoundedText(result.CorrectionGuidance, MaxValidationCorrectionGuidanceBytes, "validation correction guidance", false); err != nil {
return err
}
}
return nil
}
func validateAssistantResponse(response []byte) error {
if len(response) > MaxAssistantResponseBytes {
return errors.New("exceeds maximum length")
}
if !utf8.Valid(response) {
return errors.New("must be valid UTF-8")
}
if len(strings.TrimSpace(string(response))) == 0 {
return errors.New("must not be blank")
}
return nil
}
func validateBoundedText(value string, maximum int, name string, optional bool) error {
if value == "" && optional {
return nil
}
if !utf8.ValidString(value) {
return fmt.Errorf("%s must be valid UTF-8", name)
}
if strings.TrimSpace(value) == "" {
return fmt.Errorf("%s must not be blank", name)
}
if len(value) > maximum {
return fmt.Errorf("%s exceeds maximum length", name)
}
return nil
}

View File

@@ -0,0 +1,184 @@
package contracts
import (
"bytes"
"encoding/json"
"fmt"
"strings"
"testing"
)
func TestSemanticCorrectionOwnsValidatedContent(t *testing.T) {
assistant := []byte(`{"items":["original"]}`)
correction, err := NewSemanticCorrection(assistant, "Return one corrected replacement.")
if err != nil {
t.Fatalf("NewSemanticCorrection() error = %v", err)
}
assistant[0] = '['
if got := string(correction.AssistantResponse); got != `{"items":["original"]}` {
t.Fatalf("assistant response = %q, want owned original content", got)
}
clone, err := CloneSemanticCorrection(correction)
if err != nil {
t.Fatalf("CloneSemanticCorrection() error = %v", err)
}
clone.AssistantResponse[0] = '['
if got := string(correction.AssistantResponse); got != `{"items":["original"]}` {
t.Fatalf("source correction changed through clone = %q", got)
}
if nilClone, err := CloneSemanticCorrection(nil); err != nil || nilClone != nil {
t.Fatalf("CloneSemanticCorrection(nil) = %#v, %v; want nil, nil", nilClone, err)
}
encoded, err := json.Marshal(correction)
if err != nil {
t.Fatalf("marshal correction: %v", err)
}
if string(encoded) != "{}" {
t.Fatalf("correction JSON = %s, want no sensitive content", encoded)
}
}
func TestCorrectionContractsRejectInvalidContent(t *testing.T) {
tooLongAssistant := bytes.Repeat([]byte("a"), MaxAssistantResponseBytes+1)
tooLongGuidance := strings.Repeat("a", MaxCorrectionGuidanceBytes+1)
tooLongReason := strings.Repeat("a", MaxValidationReasonCodeBytes+1)
tooLongValidationGuidance := strings.Repeat("a", MaxValidationCorrectionGuidanceBytes+1)
for _, test := range []struct {
name string
call func() error
}{
{"blank assistant", func() error { _, err := NewSemanticCorrection([]byte(" \n"), "guidance"); return err }},
{"invalid assistant utf8", func() error { _, err := NewSemanticCorrection([]byte{0xff}, "guidance"); return err }},
{"oversized assistant", func() error { _, err := NewSemanticCorrection(tooLongAssistant, "guidance"); return err }},
{"blank guidance", func() error { _, err := NewSemanticCorrection([]byte("response"), " \t"); return err }},
{"invalid guidance utf8", func() error { _, err := NewSemanticCorrection([]byte("response"), string([]byte{0xff})); return err }},
{"oversized guidance", func() error { _, err := NewSemanticCorrection([]byte("response"), tooLongGuidance); return err }},
{"unsupported protocol", func() error { _, err := NewModelCandidate([]byte("response"), "multiple_responses"); return err }},
{"missing candidate protocol", func() error { _, err := NewModelCandidate([]byte("response"), ""); return err }},
{"blank candidate response", func() error { _, err := NewModelCandidate([]byte(" "), CorrectionProtocolSingleResponseV1); return err }},
{"missing rejection reason code", func() error {
return ValidateValidationResult(ValidationResult{CorrectionGuidance: "Correct the response."})
}},
{"missing rejection guidance", func() error { return ValidateValidationResult(ValidationResult{ReasonCode: "invalid"}) }},
{"oversized reason code", func() error {
return ValidateValidationResult(ValidationResult{ReasonCode: tooLongReason, CorrectionGuidance: "Correct the response."})
}},
{"blank reason code", func() error {
return ValidateValidationResult(ValidationResult{ReasonCode: " \t", CorrectionGuidance: "Correct the response."})
}},
{"invalid correction guidance utf8", func() error {
return ValidateValidationResult(ValidationResult{ReasonCode: "invalid", CorrectionGuidance: string([]byte{0xff})})
}},
{"oversized correction guidance", func() error {
return ValidateValidationResult(ValidationResult{ReasonCode: "invalid", CorrectionGuidance: tooLongValidationGuidance})
}},
} {
t.Run(test.name, func(t *testing.T) {
if err := test.call(); err == nil {
t.Fatal("validation error = nil, want error")
}
})
}
}
func TestModelCandidateOwnsValidatedResponse(t *testing.T) {
response := []byte(`{"items":["original"]}`)
candidate, err := NewModelCandidate(response, CorrectionProtocolSingleResponseV1)
if err != nil {
t.Fatalf("NewModelCandidate() error = %v", err)
}
response[0] = '['
if got := string(candidate.Response); got != `{"items":["original"]}` {
t.Fatalf("candidate response = %q, want owned original content", got)
}
clone, err := CloneModelCandidate(candidate)
if err != nil {
t.Fatalf("CloneModelCandidate() error = %v", err)
}
clone.Response[0] = '['
if got := string(candidate.Response); got != `{"items":["original"]}` {
t.Fatalf("source candidate changed through clone = %q", got)
}
if nilClone, err := CloneModelCandidate(nil); err != nil || nilClone != nil {
t.Fatalf("CloneModelCandidate(nil) = %#v, %v; want nil, nil", nilClone, err)
}
}
func TestValidationResultAllowsAbsentOptionalCorrectionFields(t *testing.T) {
if err := ValidateValidationResult(ValidationResult{Approved: true}); err != nil {
t.Fatalf("ValidateValidationResult() error = %v, want nil", err)
}
if err := ValidateValidationResult(ValidationResult{ReasonCode: "invalid-evidence", CorrectionGuidance: "Provide source-backed evidence."}); err != nil {
t.Fatalf("ValidateValidationResult() error = %v, want nil", err)
}
if err := CorrectionProtocol("").Validate(); err == nil {
t.Fatal("empty correction protocol validation error = nil, want error")
}
}
func TestCloneStructuredCompletionRequestOwnsCorrection(t *testing.T) {
correction, err := NewSemanticCorrection([]byte(`{"value":"original"}`), "Correct the value.")
if err != nil {
t.Fatalf("NewSemanticCorrection() error = %v", err)
}
attempts := 2
request := StructuredCompletionRequest{
Inputs: LLMInputSet{"source": NewLLMInputMaterial("source", "application/json", []byte(`{"source":true}`), "", "")},
Vars: map[string]any{"labels": []string{"original"}},
StructuredOutputRepairAttempts: &attempts,
Correction: correction,
}
clone, err := CloneStructuredCompletionRequest(request)
if err != nil {
t.Fatalf("CloneStructuredCompletionRequest() error = %v", err)
}
correction.AssistantResponse[0] = '['
request.Inputs["source"] = NewLLMInputMaterial("source", "application/json", []byte(`{"source":false}`), "", "")
*request.StructuredOutputRepairAttempts = 7
if got := string(clone.Correction.AssistantResponse); got != `{"value":"original"}` {
t.Fatalf("cloned correction response = %q, want owned original content", got)
}
if got := string(clone.Inputs["source"].Content); got != `{"source":true}` {
t.Fatalf("cloned input = %q, want owned original content", got)
}
if clone.StructuredOutputRepairAttempts == nil || *clone.StructuredOutputRepairAttempts != 2 {
t.Fatalf("cloned repair attempts = %v, want 2", clone.StructuredOutputRepairAttempts)
}
}
func TestStructuredCompletionRequestDebugSummaryOmitsCorrectionContent(t *testing.T) {
const assistantResponse = `{"secret":"assistant response"}`
const userGuidance = "secret user guidance"
correction, err := NewSemanticCorrection([]byte(assistantResponse), userGuidance)
if err != nil {
t.Fatalf("NewSemanticCorrection() error = %v", err)
}
request := StructuredCompletionRequest{
Inputs: LLMInputSet{"source": NewLLMInputMaterial("source", "application/json", []byte(`{"source":true}`), "", "")},
Vars: map[string]any{"custom": "value"},
Correction: correction,
}
summary := request.DebugSummary()
if summary.InputCount != 1 || summary.VariableCount != 1 || summary.Correction == nil {
t.Fatalf("debug summary = %#v, want input, variable, and correction metadata", summary)
}
if summary.Correction.AssistantResponseBytes != len(assistantResponse) || summary.Correction.UserGuidanceBytes != len(userGuidance) ||
summary.Correction.AssistantResponseDigest == "" || summary.Correction.UserGuidanceDigest == "" {
t.Fatalf("correction summary = %#v, want byte counts and digests", summary.Correction)
}
encoded, err := json.Marshal(summary)
if err != nil {
t.Fatalf("marshal debug summary: %v", err)
}
for _, rendered := range []string{string(encoded), fmt.Sprintf("%+v", request), fmt.Sprintf("%#v", request)} {
for _, secret := range []string{assistantResponse, userGuidance} {
if strings.Contains(rendered, secret) {
t.Fatalf("content-safe request rendering leaked %q: %s", secret, rendered)
}
}
}
}

View File

@@ -42,12 +42,14 @@ type TypedExtractionRequest struct {
References ReferenceSet
LLMProfile string
StructuredOutputRepairAttempts *int
Correction *SemanticCorrection
Metadata map[string]any
}
type TypedExtractionResult[T any] struct {
Value T
Warnings []Warning
Value T
Warnings []Warning
ModelCandidate *ModelCandidate
}
type Extractor[T any] interface {
@@ -65,12 +67,14 @@ type TypedMergeRequest[T any] struct {
References ReferenceSet
LLMProfile string
StructuredOutputRepairAttempts *int
Correction *SemanticCorrection
Metadata map[string]any
}
type TypedMergeResult[T any] struct {
Value T
Warnings []Warning
Value T
Warnings []Warning
ModelCandidate *ModelCandidate
}
type Merger[T any] interface {
@@ -87,13 +91,15 @@ type TypedNormalizeRequest[T any] struct {
References ReferenceSet
LLMProfile string
StructuredOutputRepairAttempts *int
Correction *SemanticCorrection
Metadata map[string]any
}
type TypedNormalizeResult[T any] struct {
Value T
Warnings []Warning
Retry *NormalizeRetry
Value T
Warnings []Warning
Retry *NormalizeRetry
ModelCandidate *ModelCandidate
}
// Normalize retry diagnostic limits bound module-provided values before the

View File

@@ -119,6 +119,10 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
if req.StructuredOutputRepairAttempts != nil && (*req.StructuredOutputRepairAttempts < 0 || *req.StructuredOutputRepairAttempts > 3) {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured output repair attempts must be between zero and three")
}
appendedMessages, err := promptKitCorrectionMessages(req.Correction)
if err != nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion correction: %w", err)
}
promptID := strings.TrimSpace(req.PromptID)
if promptID == "" {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty")
@@ -133,13 +137,14 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
}
runReq := promptkit.RunRequest{
PromptID: promptID,
PromptVersion: strings.TrimSpace(req.PromptVersion),
ProfileID: strings.TrimSpace(req.ProfileID),
SessionID: sessionID,
Inputs: promptKitInputs(req.Inputs),
Vars: promptKitVars(req, sessionID),
Execution: execution,
PromptID: promptID,
PromptVersion: strings.TrimSpace(req.PromptVersion),
ProfileID: strings.TrimSpace(req.ProfileID),
SessionID: sessionID,
Inputs: promptKitInputs(req.Inputs),
Vars: promptKitVars(req, sessionID),
Execution: execution,
AppendedMessages: appendedMessages,
}
if req.StructuredOutputRepairAttempts != nil {
inspection, err := c.engine.InspectPrompt(ctx, promptID, strings.TrimSpace(req.PromptVersion))
@@ -213,6 +218,20 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
return response, nil
}
func promptKitCorrectionMessages(correction *contracts.SemanticCorrection) ([]promptkit.RenderedMessage, error) {
owned, err := contracts.CloneSemanticCorrection(correction)
if err != nil {
return nil, err
}
if owned == nil {
return nil, nil
}
return []promptkit.RenderedMessage{
{Role: promptkit.RoleAssistant, Content: string(owned.AssistantResponse)},
{Role: promptkit.RoleUser, Content: owned.UserGuidance},
}, nil
}
func promptKitDebugGenerationError(prepared *promptkit.PreparedRun, generationErr *promptkit.GenerationError) *contracts.LLMDebugResponse {
if generationErr == nil {
return nil

View File

@@ -1,6 +1,7 @@
package llm
import (
"bytes"
"context"
"encoding/json"
"errors"
@@ -116,6 +117,74 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
}
}
func TestPromptKitClientAppendsSemanticCorrectionAfterRenderedPrompt(t *testing.T) {
fake := &fakePromptKitLLM{content: `{"ok":true}`}
client := newTestPromptKitClient(t, fake)
request := contracts.StructuredCompletionRequest{
PromptID: "adapter.direct-session",
ProfileID: "explicit-profile",
SessionID: "correction-session",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
Vars: map[string]any{"custom": "value"},
}
var ordinary map[string]any
if _, err := client.CompleteStructured(context.Background(), request, &ordinary); err != nil {
t.Fatalf("ordinary CompleteStructured() error = %v", err)
}
ordinaryMessages := append([]promptkit.RenderedMessage(nil), fake.lastRequest().Prompt.Messages...)
if len(ordinaryMessages) != 1 {
t.Fatalf("ordinary rendered messages = %#v, want only the declared prompt message", ordinaryMessages)
}
correction, err := contracts.NewSemanticCorrection([]byte(`{"previous":"response"}`), "Return the corrected JSON object.")
if err != nil {
t.Fatalf("NewSemanticCorrection() error = %v", err)
}
request.Correction = correction
var corrected map[string]any
if _, err := client.CompleteStructured(context.Background(), request, &corrected); err != nil {
t.Fatalf("corrected CompleteStructured() error = %v", err)
}
correctedMessages := fake.lastRequest().Prompt.Messages
if len(correctedMessages) != len(ordinaryMessages)+2 {
t.Fatalf("corrected message count = %d, want %d", len(correctedMessages), len(ordinaryMessages)+2)
}
if !reflect.DeepEqual(correctedMessages[:len(ordinaryMessages)], ordinaryMessages) {
t.Fatalf("ordinary rendered prefix changed: got %#v, want %#v", correctedMessages[:len(ordinaryMessages)], ordinaryMessages)
}
if got, want := correctedMessages[len(ordinaryMessages)], (promptkit.RenderedMessage{Role: promptkit.RoleAssistant, Content: `{"previous":"response"}`}); got != want {
t.Fatalf("assistant correction message = %#v, want %#v", got, want)
}
if got, want := correctedMessages[len(ordinaryMessages)+1], (promptkit.RenderedMessage{Role: promptkit.RoleUser, Content: "Return the corrected JSON object."}); got != want {
t.Fatalf("user correction message = %#v, want %#v", got, want)
}
}
func TestPromptKitClientRejectsInvalidCorrectionsBeforePromptPreparation(t *testing.T) {
const sensitiveResponse = "assistant-response-must-not-appear-in-errors"
for _, correction := range []*contracts.SemanticCorrection{
{AssistantResponse: []byte(sensitiveResponse), UserGuidance: " \t"},
{AssistantResponse: bytes.Repeat([]byte(sensitiveResponse), contracts.MaxAssistantResponseBytes/len(sensitiveResponse)+1), UserGuidance: "Use a smaller response."},
} {
fake := &fakePromptKitLLM{content: `{"ok":true}`}
client := newTestPromptKitClient(t, fake)
var out map[string]any
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{Correction: correction}, &out)
if err == nil || !strings.Contains(err.Error(), "correction") {
t.Fatalf("CompleteStructured() error = %v, want correction validation failure", err)
}
if strings.Contains(err.Error(), sensitiveResponse) {
t.Fatalf("correction validation error leaked response content: %v", err)
}
if got := atomic.LoadInt32(&fake.calls); got != 0 {
t.Fatalf("provider calls = %d, want no provider call after invalid correction", got)
}
}
}
func TestPromptKitClientUsesOnePreparedSnapshotForDebugAndGeneration(t *testing.T) {
const initialPrompt = `id: snapshot.test
version: "v1"
@@ -960,6 +1029,10 @@ func TestPromptKitClientRepairsStructuredOutputAndReportsCumulativeUsage(t *test
{Content: `{"ok":true}`, Usage: promptkit.TokenUsage{PromptTokens: 7, CompletionTokens: 11, TotalTokens: 18}},
}}
client := newTestPromptKitClient(t, fake)
correction, err := contracts.NewSemanticCorrection([]byte(`{"bad":true}`), "Return the required ok field.")
if err != nil {
t.Fatalf("NewSemanticCorrection() error = %v", err)
}
var out struct {
OK bool `json:"ok"`
@@ -968,6 +1041,7 @@ func TestPromptKitClientRepairsStructuredOutputAndReportsCumulativeUsage(t *test
PromptID: "adapter.test",
StructuredOutputRepairAttempts: &attempts,
SessionID: "repair-test",
Correction: correction,
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
@@ -978,6 +1052,17 @@ func TestPromptKitClientRepairsStructuredOutputAndReportsCumulativeUsage(t *test
if got := atomic.LoadInt32(&fake.calls); got != 2 {
t.Fatalf("provider calls = %d, want initial generation and one repair", got)
}
requests := fake.requestsSnapshot()
if len(requests) != 2 || len(requests[0].Prompt.Messages) < 3 {
t.Fatalf("repair requests = %#v, want correction messages on the initial prepared request", requests)
}
messages := requests[0].Prompt.Messages
if got, want := messages[len(messages)-2], (promptkit.RenderedMessage{Role: promptkit.RoleAssistant, Content: `{"bad":true}`}); got != want {
t.Fatalf("repair assistant correction = %#v, want %#v", got, want)
}
if got, want := messages[len(messages)-1], (promptkit.RenderedMessage{Role: promptkit.RoleUser, Content: "Return the required ok field."}); got != want {
t.Fatalf("repair user correction = %#v, want %#v", got, want)
}
if response.RepairAttempts != 1 || response.PromptTokens != 10 || response.CompletionTokens != 16 || response.TotalTokens != 26 {
t.Fatalf("response repair and usage = %#v, want one repair and PromptKit cumulative usage", response)
}

View File

@@ -17,7 +17,7 @@ const (
promptKitLocalBackendMarker = "notarius:promptkit-local-backend:v1"
// The built-in profile catalog is compiled into this pinned PromptKit
// release. Update this identity when the dependency is upgraded.
promptKitBuiltinProfileCatalogID = "promptkit:v0.8.0:builtin-profiles"
promptKitBuiltinProfileCatalogID = "promptkit:v0.9.0:builtin-profiles"
)
func promptKitProfileFingerprint(profileDir, profileFile, fallbackProfileDigest string) (CheckpointFingerprint, error) {

View File

@@ -89,6 +89,7 @@ const (
CheckpointReasonReused CheckpointReasonCode = "checkpoint_reused"
CheckpointReasonAcceptedArtifactReused CheckpointReasonCode = "accepted_artifact_reused"
CheckpointReasonRecomputeStep CheckpointReasonCode = "recompute_step"
CheckpointReasonValidationIncompleteLineage CheckpointReasonCode = "validation_incomplete_lineage"
)
type CheckpointDecision struct {
@@ -160,6 +161,8 @@ func checkpointDecisionDetail(reasonCode CheckpointReasonCode) string {
return "accepted normalized artifact is reusable"
case CheckpointReasonRecomputeStep:
return "selected step requires execution"
case CheckpointReasonValidationIncompleteLineage:
return "checkpoint reuse is disabled by validation-incomplete input lineage"
default:
return "checkpoint decision"
}

View File

@@ -42,6 +42,7 @@ type debugTimedEnvelope struct {
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"`
Attempt int `json:"attempt,omitempty"`
AttemptKind string `json:"attempt_kind,omitempty"`
StartedAt time.Time `json:"started_at"`
CompletedAt time.Time `json:"completed_at"`
DurationMS int64 `json:"duration_ms"`
@@ -110,24 +111,7 @@ type debugSerializedOutput struct {
Content debugBinaryEnvelope `json:"content"`
}
type debugLLMInputMaterial struct {
Name string `json:"name"`
MediaType string `json:"media_type,omitempty"`
Content string `json:"content_base64,omitempty"`
Digest string `json:"digest,omitempty"`
OriginURI string `json:"origin_uri,omitempty"`
SizeBytes int64 `json:"size_bytes,omitempty"`
}
type debugStructuredCompletionRequest struct {
StageName string `json:"stage_name"`
PromptID string `json:"prompt_id,omitempty"`
PromptVersion string `json:"prompt_version,omitempty"`
ProfileID string `json:"profile_id,omitempty"`
SessionID string `json:"session_id,omitempty"`
Inputs map[string]debugLLMInputMaterial `json:"inputs,omitempty"`
Vars map[string]any `json:"vars,omitempty"`
}
type debugStructuredCompletionRequest = contracts.DebugStructuredCompletionRequest
type debugStructuredCompletionResponse struct {
Content string `json:"content,omitempty"`
@@ -160,9 +144,37 @@ type debugLLMCallReference struct {
PromptID string `json:"prompt_id,omitempty"`
ProfileID string `json:"profile_id,omitempty"`
Model string `json:"model,omitempty"`
PromptTokens int `json:"prompt_tokens,omitempty"`
CompletionTokens int `json:"completion_tokens,omitempty"`
TotalTokens int `json:"total_tokens,omitempty"`
RepairAttempts int `json:"repair_attempts,omitempty"`
Error bool `json:"error,omitempty"`
}
type debugValidationOutcome struct {
ValidatorName string `json:"validator_name"`
Outcome string `json:"outcome"`
AttemptCount int `json:"attempt_count"`
ReasonCode string `json:"reason_code,omitempty"`
}
type debugProducerAttempt struct {
Number int `json:"number"`
Kind string `json:"kind"`
Outcome string `json:"outcome"`
Validation []debugValidationOutcome `json:"validation,omitempty"`
}
type debugProducerTerminal struct {
ProducerAttemptCount int `json:"producer_attempt_count"`
Attempts []debugProducerAttempt `json:"attempts,omitempty"`
AggregateReasonCodes []string `json:"aggregate_reason_codes,omitempty"`
ValidationComplete bool `json:"validation_complete"`
EffectivePolicy ValidationPolicy `json:"effective_policy"`
TerminalAction string `json:"terminal_action"`
Summary artifacts.ValidationSummary `json:"validation_summary"`
}
type debugValidationCall struct {
ValidatorName string `json:"validator_name"`
Request any `json:"request"`
@@ -267,6 +279,10 @@ func (client *debugLLMClient) CompleteStructured(ctx context.Context, req contra
PromptID: req.PromptID,
ProfileID: debugFirstNonEmptyString(response.ProfileID, req.ProfileID),
Model: debugFirstNonEmptyString(response.Model, debugResponseModel(response)),
PromptTokens: response.PromptTokens,
CompletionTokens: response.CompletionTokens,
TotalTokens: response.TotalTokens,
RepairAttempts: response.RepairAttempts,
Error: err != nil,
}
if scope := debugLLMScopeFromContext(ctx); scope != nil {
@@ -423,6 +439,50 @@ func (r attemptTerminalRecorder) record(payload any, terminalErr error) error {
return terminalErr
}
func debugValidationOutcomes(report validationReport) []debugValidationOutcome {
if len(report.records) == 0 {
return nil
}
outcomes := make([]debugValidationOutcome, 0, len(report.records))
for _, record := range report.records {
outcomes = append(outcomes, debugValidationOutcome{
ValidatorName: record.validatorName,
Outcome: string(record.outcome),
AttemptCount: record.attemptCount,
ReasonCode: record.reasonCode,
})
}
return outcomes
}
func writeProducerTerminalDebug(recorder DebugRecorder, name string, terminal producerAttemptTerminal, policy ValidationPolicy, summary artifacts.ValidationSummary) error {
attempts := make([]debugProducerAttempt, 0, len(terminal.Provenance))
for _, item := range terminal.Provenance {
attempts = append(attempts, debugProducerAttempt{
Number: item.Number,
Kind: string(item.Kind),
Outcome: string(item.Outcome),
Validation: debugValidationOutcomes(item.Validation),
})
}
return writeDebugTimed(recorder, name, debugTimedEnvelope{
Stage: summary.Stage,
StepID: summary.StepID,
LaneID: summary.LaneID,
ModuleKey: summary.ModuleKey,
StartedAt: time.Now().UTC(),
Payload: debugProducerTerminal{
ProducerAttemptCount: summary.ProducerAttemptCount,
Attempts: attempts,
AggregateReasonCodes: append([]string(nil), summary.ReasonCodes...),
ValidationComplete: summary.Status == "complete",
EffectivePolicy: policy,
TerminalAction: string(terminal.Action),
Summary: artifacts.CloneValidationSummary(summary),
},
})
}
func debugContentEnvelope(content []byte, mediaType string, metadata map[string]any, warnings []contracts.Warning) debugBinaryEnvelope {
content = redactSecretBytes(content)
return debugBinaryEnvelope{
@@ -567,29 +627,7 @@ func debugOutputFiles(files []contracts.OutputFile) []debugOutputFile {
}
func debugCompletionRequest(req contracts.StructuredCompletionRequest) debugStructuredCompletionRequest {
inputs := make(map[string]debugLLMInputMaterial, len(req.Inputs))
for key, material := range req.Inputs {
inputs[key] = debugLLMInputMaterial{
Name: material.Name,
MediaType: material.MediaType,
Content: base64.StdEncoding.EncodeToString(redactSecretBytes(material.Content)),
Digest: material.Digest,
OriginURI: material.OriginURI,
SizeBytes: material.SizeBytes,
}
}
if len(inputs) == 0 {
inputs = nil
}
return debugStructuredCompletionRequest{
StageName: req.StageName,
PromptID: req.PromptID,
PromptVersion: req.PromptVersion,
ProfileID: req.ProfileID,
SessionID: req.SessionID,
Inputs: inputs,
Vars: redactSensitiveMap(req.Vars),
}
return req.DebugSummary()
}
func debugCompletionResponse(response contracts.StructuredCompletionResponse) debugStructuredCompletionResponse {
@@ -691,6 +729,7 @@ func debugResponseModel(response contracts.StructuredCompletionResponse) string
func debugValidationResultEnvelope(result contracts.ValidationResult) contracts.ValidationResult {
result.Message = string(redactSecretBytes([]byte(result.Message)))
result.CorrectionGuidance = ""
result.DiagnosticArtifactPath = string(redactSecretBytes([]byte(result.DiagnosticArtifactPath)))
for i := range result.Warnings {
result.Warnings[i].Message = string(redactSecretBytes([]byte(result.Warnings[i].Message)))

View File

@@ -47,6 +47,32 @@ func TestDebugLLMPathsKeepDotIdentitiesDistinct(t *testing.T) {
}
}
func TestDebugCompletionRequestOmitsCorrectionContent(t *testing.T) {
const assistantResponse = `{"secret":"assistant response"}`
const userGuidance = "secret user guidance"
correction, err := contracts.NewSemanticCorrection([]byte(assistantResponse), userGuidance)
if err != nil {
t.Fatalf("NewSemanticCorrection() error = %v", err)
}
summary := debugCompletionRequest(contracts.StructuredCompletionRequest{
Inputs: contracts.LLMInputSet{"source": contracts.NewLLMInputMaterial("source", "application/json", []byte(`{"source":true}`), "", "")},
Vars: map[string]any{"custom": "value"},
Correction: correction,
})
encoded, err := json.Marshal(summary)
if err != nil {
t.Fatalf("marshal completion summary: %v", err)
}
for _, secret := range []string{assistantResponse, userGuidance} {
if strings.Contains(string(encoded), secret) {
t.Fatalf("debug completion summary leaked %q: %s", secret, encoded)
}
}
if summary.InputCount != 1 || summary.VariableCount != 1 || summary.Correction == nil {
t.Fatalf("debug completion summary = %#v, want counts and correction metadata", summary)
}
}
func TestDebugSourceDocumentPreservesUnitReferences(t *testing.T) {
doc := validSourceDocument()
envelope := debugSourceDocumentEnvelope(doc)

View File

@@ -114,9 +114,10 @@ func TestRunnerEvidenceContextOmitsAbsentAndRejectedLanes(t *testing.T) {
prepared.Steps[1].lanes[0].mergeValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("reject"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted"}, nil
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted", CorrectionGuidance: "return an acceptable candidate"}, nil
},
}}
prepared.Steps[1].lanes[0].resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
installEvidencePlan(prepared, 0, []string{"absent", "present", "rejected"}, func(notes codecNotes) ([]source.SourceRef, error) {
if len(notes.Items) > 0 && notes.Items[0] == "present" {
return []source.SourceRef{{SourceID: "source", StartUnitID: 1, EndUnitID: 1}}, nil

View File

@@ -58,11 +58,20 @@ func RegisterExtractorBuilder[T any](registry *ExtractorRegistry, spec ModuleSpe
if !ok {
return erasedTypedResult{}, fmt.Errorf("extractor %q has incompatible implementation %T", normalized.Key, implementation)
}
correction, err := contracts.CloneSemanticCorrection(request.Correction)
if err != nil {
return erasedTypedResult{}, fmt.Errorf("clone extraction correction: %w", err)
}
request.Correction = correction
result, err := extractor.Extract(ctx, request)
if err != nil {
return erasedTypedResult{}, err
}
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
candidate, err := contracts.CloneModelCandidate(result.ModelCandidate)
if err != nil {
return erasedTypedResult{}, fmt.Errorf("clone extraction model candidate: %w", err)
}
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), ModelCandidate: candidate}, nil
}}
if registry.typedEntries == nil {
registry.typedEntries = map[string]typedExtractorEntry{}

View File

@@ -41,6 +41,54 @@ func operationReferenceSet(input RunInput, target ResolvedReferenceTarget) contr
return CloneReferenceSet(target.ReferenceSet)
}
// referenceTargetReuseEligible reports whether every generated artifact in a
// stage's reference set descends exclusively from fully validated work. Static
// references and callers that do not supply lineage metadata are reusable.
func referenceTargetReuseEligible(input RunInput, target ResolvedReferenceTarget) bool {
if input.referenceReuseEligibility == nil {
return true
}
eligible, ok := input.referenceReuseEligibility[keyForReferenceTarget(target)]
return !ok || eligible
}
func laneReferencesReuseEligible(input RunInput, lane ResolvedArtifactLane) bool {
return referenceTargetReuseEligible(input, lane.ExtractReferences) &&
referenceTargetReuseEligible(input, lane.MergeReferences) &&
referenceTargetReuseEligible(input, lane.NormalizeReferences)
}
// buildStepReferenceReuseEligibility carries validation completeness alongside
// generated references without exposing the internal lineage flag in artifact
// payloads. A target becomes ineligible when any generated input is ineligible.
func buildStepReferenceReuseEligibility(step PreparedPipelineStep, outputs map[generatedOutputKey]bool) map[referenceTargetKey]bool {
eligibility := make(map[referenceTargetKey]bool)
for _, prepared := range step.lanes {
lane := prepared.resolved
for _, target := range []ResolvedReferenceTarget{lane.ExtractReferences, lane.MergeReferences, lane.NormalizeReferences} {
generated := false
eligible := true
for _, binding := range target.Bindings {
if binding.Artifact == nil {
continue
}
generated = true
producer := generatedOutputKeyFor(binding.Artifact.Step, binding.Artifact.Lane)
if reusable, ok := outputs[producer]; ok && !reusable {
eligible = false
}
}
if generated {
eligibility[keyForReferenceTarget(target)] = eligible
}
}
}
if len(eligibility) == 0 {
return nil
}
return eligibility
}
// buildStepReferenceSets resolves every generated binding for a step before
// any lane in that step is allowed to start. Each returned set is a fresh
// operation-time view; prepared reference sets are never modified.

View File

@@ -85,11 +85,19 @@ func RegisterMergerBuilder[T any](registry *MergerRegistry, spec ModuleSpec, val
}
outputs[i] = contracts.ExtractArtifact[T]{LaneID: output.LaneID, ExtractorKey: output.ExtractorKey, SourceID: output.SourceID, ChunkID: output.ChunkID, ChunkIndex: output.ChunkIndex, ChunkRef: output.ChunkRef, Value: value}
}
result, err := merger.Merge(ctx, contracts.TypedMergeRequest[T]{Source: request.Source, LaneID: request.LaneID, ExtractOutputs: outputs, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, StructuredOutputRepairAttempts: request.StructuredOutputRepairAttempts, Metadata: request.Metadata})
correction, err := contracts.CloneSemanticCorrection(request.Correction)
if err != nil {
return erasedTypedResult{}, fmt.Errorf("clone merge correction: %w", err)
}
result, err := merger.Merge(ctx, contracts.TypedMergeRequest[T]{Source: request.Source, LaneID: request.LaneID, ExtractOutputs: outputs, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, StructuredOutputRepairAttempts: request.StructuredOutputRepairAttempts, Correction: correction, Metadata: request.Metadata})
if err != nil {
return erasedTypedResult{}, err
}
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
candidate, err := contracts.CloneModelCandidate(result.ModelCandidate)
if err != nil {
return erasedTypedResult{}, fmt.Errorf("clone merge model candidate: %w", err)
}
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), ModelCandidate: candidate}, nil
},
}
return nil

View File

@@ -21,25 +21,28 @@ const (
)
type ModuleSpec struct {
Key string
Stage ModuleStage
ExecutionClass contracts.ExecutionClass
ArtifactKind contracts.ArtifactKind
Provides []string
Requires []string
ReferenceSlots []contracts.ReferenceSlot
Key string
Stage ModuleStage
ExecutionClass contracts.ExecutionClass
CorrectionProtocol contracts.CorrectionProtocol
ArtifactKind contracts.ArtifactKind
Provides []string
Requires []string
ReferenceSlots []contracts.ReferenceSlot
}
func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
executionClass := contracts.ExecutionClass(strings.TrimSpace(string(spec.ExecutionClass)))
correctionProtocol := contracts.CorrectionProtocol(strings.TrimSpace(string(spec.CorrectionProtocol)))
return ModuleSpec{
Key: strings.TrimSpace(spec.Key),
Stage: spec.Stage,
ExecutionClass: executionClass,
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
Provides: normalizeCapabilities(spec.Provides),
Requires: normalizeCapabilities(spec.Requires),
ReferenceSlots: normalizeReferenceSlots(spec.ReferenceSlots),
Key: strings.TrimSpace(spec.Key),
Stage: spec.Stage,
ExecutionClass: executionClass,
CorrectionProtocol: correctionProtocol,
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
Provides: normalizeCapabilities(spec.Provides),
Requires: normalizeCapabilities(spec.Requires),
ReferenceSlots: normalizeReferenceSlots(spec.ReferenceSlots),
}
}
@@ -70,13 +73,14 @@ func normalizeCapabilities(values []string) []string {
func cloneModuleSpec(spec ModuleSpec) ModuleSpec {
return ModuleSpec{
Key: spec.Key,
Stage: spec.Stage,
ExecutionClass: spec.ExecutionClass,
ArtifactKind: spec.ArtifactKind,
Provides: append([]string(nil), spec.Provides...),
Requires: append([]string(nil), spec.Requires...),
ReferenceSlots: contracts.CloneReferenceSlots(spec.ReferenceSlots),
Key: spec.Key,
Stage: spec.Stage,
ExecutionClass: spec.ExecutionClass,
CorrectionProtocol: spec.CorrectionProtocol,
ArtifactKind: spec.ArtifactKind,
Provides: append([]string(nil), spec.Provides...),
Requires: append([]string(nil), spec.Requires...),
ReferenceSlots: contracts.CloneReferenceSlots(spec.ReferenceSlots),
}
}
@@ -93,6 +97,19 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
if spec.ExecutionClass != contracts.ExecutionClassDeterministic && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
return fmt.Errorf("%s %q has unsupported execution class %q", kind, spec.Key, spec.ExecutionClass)
}
if spec.CorrectionProtocol != "" {
if err := spec.CorrectionProtocol.Validate(); err != nil {
return fmt.Errorf("%s %q correction protocol: %w", kind, spec.Key, err)
}
if spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
return fmt.Errorf("%s %q correction protocol requires an LLM-backed execution class", kind, spec.Key)
}
switch spec.Stage {
case StageChunk, StageExtract, StageMerge, StageNormalize:
default:
return fmt.Errorf("%s %q correction protocol is not supported for %q stage", kind, spec.Key, spec.Stage)
}
}
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
}

View File

@@ -32,14 +32,70 @@ func TestValidateModuleSpecRequiresSupportedExecutionClass(t *testing.T) {
}
}
func TestCloneModuleSpecPreservesExecutionClass(t *testing.T) {
spec := normalizeModuleSpec(ModuleSpec{Key: " module ", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked})
func TestCloneModuleSpecPreservesCorrectionProtocol(t *testing.T) {
spec := normalizeModuleSpec(ModuleSpec{
Key: " module ",
Stage: StageChunk,
ExecutionClass: contracts.ExecutionClassLLMBacked,
CorrectionProtocol: " single_response_v1 ",
})
if spec.CorrectionProtocol != contracts.CorrectionProtocolSingleResponseV1 {
t.Fatalf("normalized CorrectionProtocol = %q, want %q", spec.CorrectionProtocol, contracts.CorrectionProtocolSingleResponseV1)
}
cloned := cloneModuleSpec(spec)
if !reflect.DeepEqual(cloned, spec) {
t.Fatalf("cloneModuleSpec() = %#v, want %#v", cloned, spec)
}
}
func TestValidateModuleSpecCorrectionProtocolEligibility(t *testing.T) {
tests := []struct {
name string
stage ModuleStage
class contracts.ExecutionClass
protocol contracts.CorrectionProtocol
want string
}{
{name: "LLM chunk", stage: StageChunk, class: contracts.ExecutionClassLLMBacked, protocol: contracts.CorrectionProtocolSingleResponseV1},
{name: "LLM extract", stage: StageExtract, class: contracts.ExecutionClassLLMBacked, protocol: contracts.CorrectionProtocolSingleResponseV1},
{name: "LLM merge", stage: StageMerge, class: contracts.ExecutionClassLLMBacked, protocol: contracts.CorrectionProtocolSingleResponseV1},
{name: "LLM normalize", stage: StageNormalize, class: contracts.ExecutionClassLLMBacked, protocol: contracts.CorrectionProtocolSingleResponseV1},
{name: "deterministic chunk", stage: StageChunk, class: contracts.ExecutionClassDeterministic, protocol: contracts.CorrectionProtocolSingleResponseV1, want: "LLM-backed"},
{name: "input", stage: StageInput, class: contracts.ExecutionClassLLMBacked, protocol: contracts.CorrectionProtocolSingleResponseV1, want: "not supported"},
{name: "output", stage: StageOutput, class: contracts.ExecutionClassLLMBacked, protocol: contracts.CorrectionProtocolSingleResponseV1, want: "not supported"},
{name: "unknown protocol", stage: StageChunk, class: contracts.ExecutionClassLLMBacked, protocol: "unsupported", want: "unsupported correction protocol"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
spec := normalizeModuleSpec(ModuleSpec{
Key: "module",
Stage: test.stage,
ExecutionClass: test.class,
CorrectionProtocol: test.protocol,
})
err := validateModuleSpec("module", test.stage, spec)
if test.want == "" && err != nil {
t.Fatalf("validateModuleSpec() error = %v, want nil", err)
}
if test.want != "" && (err == nil || !strings.Contains(err.Error(), test.want)) {
t.Fatalf("validateModuleSpec() error = %v, want %q", err, test.want)
}
})
}
}
func TestNormalizeValidatorSpecRejectsCorrectionProtocol(t *testing.T) {
_, err := normalizeValidatorSpec(ValidatorSpec{
Key: "validator",
ExecutionClass: contracts.ExecutionClassLLMBacked,
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
})
if err == nil || !strings.Contains(err.Error(), "correction protocol") {
t.Fatalf("normalizeValidatorSpec() error = %v, want correction protocol error", err)
}
}
func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
tests := []struct {
name string

View File

@@ -76,11 +76,19 @@ func RegisterNormalizerBuilder[T any](registry *NormalizerRegistry, spec ModuleS
if err != nil {
return erasedTypedResult{}, err
}
result, err := normalizer.Normalize(ctx, contracts.TypedNormalizeRequest[T]{Source: request.Source, LaneID: request.LaneID, MergeOutput: contracts.MergeArtifact[T]{LaneID: request.MergeOutput.LaneID, MergerKey: request.MergeOutput.MergerKey, SourceID: request.MergeOutput.SourceID, Value: value}, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, StructuredOutputRepairAttempts: request.StructuredOutputRepairAttempts, Metadata: request.Metadata})
correction, err := contracts.CloneSemanticCorrection(request.Correction)
if err != nil {
return erasedTypedResult{}, fmt.Errorf("clone normalize correction: %w", err)
}
result, err := normalizer.Normalize(ctx, contracts.TypedNormalizeRequest[T]{Source: request.Source, LaneID: request.LaneID, MergeOutput: contracts.MergeArtifact[T]{LaneID: request.MergeOutput.LaneID, MergerKey: request.MergeOutput.MergerKey, SourceID: request.MergeOutput.SourceID, Value: value}, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, StructuredOutputRepairAttempts: request.StructuredOutputRepairAttempts, Correction: correction, Metadata: request.Metadata})
if err != nil {
return erasedTypedResult{}, err
}
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), Retry: cloneNormalizeRetry(result.Retry)}, nil
candidate, err := contracts.CloneModelCandidate(result.ModelCandidate)
if err != nil {
return erasedTypedResult{}, fmt.Errorf("clone normalize model candidate: %w", err)
}
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), Retry: cloneNormalizeRetry(result.Retry), ModelCandidate: candidate}, nil
},
}
return nil

View File

@@ -13,10 +13,11 @@ import (
// resolved pipeline. Its implementation values are private so execution cannot
// replace or reconfigure them after preparation.
type PreparedPipeline struct {
Input ModuleBinding
Chunk ModuleBinding
Steps []PreparedPipelineStep
Output ModuleBinding
Input ModuleBinding
Chunk ModuleBinding
ChunkCorrectionProtocol contracts.CorrectionProtocol
Steps []PreparedPipelineStep
Output ModuleBinding
resolved ResolvedPipeline
dependencies ModuleDependencies
@@ -36,7 +37,10 @@ type PreparedPipelineStep struct {
}
type PreparedArtifactLane struct {
Resolved ResolvedArtifactLane
Resolved ResolvedArtifactLane
ExtractCorrectionProtocol contracts.CorrectionProtocol
MergeCorrectionProtocol contracts.CorrectionProtocol
NormalizeCorrectionProtocol contracts.CorrectionProtocol
}
type preparedLaneExecutor struct {
@@ -79,6 +83,7 @@ type preparedValidator struct {
typedValidate typedValidateOperation
chunk contracts.ChunkValidator
serialized contracts.SerializedValidator
position int
}
// Prepare validates all configured options and constructs every selected
@@ -87,17 +92,21 @@ func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDepend
if err := validateResolvedPipeline(resolved); err != nil {
return nil, err
}
if err := validateCorrectionRetryCapabilities(resolved); err != nil {
return nil, err
}
if err := validateRegistrySet(resolved, registries); err != nil {
return nil, err
}
stable := cloneResolvedPipeline(resolved)
prepared := &PreparedPipeline{
Input: cloneModuleBinding(stable.Input),
Chunk: cloneModuleBinding(stable.Chunk),
Output: cloneModuleBinding(stable.Output),
resolved: stable,
dependencies: deps,
artifactCodecs: registries.ArtifactCodecs,
Input: cloneModuleBinding(stable.Input),
Chunk: cloneModuleBinding(stable.Chunk),
ChunkCorrectionProtocol: stable.ChunkCorrectionProtocol,
Output: cloneModuleBinding(stable.Output),
resolved: stable,
dependencies: deps,
artifactCodecs: registries.ArtifactCodecs,
}
request := func(binding ModuleBinding, references contracts.ReferenceSet) BuildRequest {
return BuildRequest{Dependencies: deps, Options: binding.Options, References: references}
@@ -131,7 +140,12 @@ func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDepend
if err != nil {
return nil, err
}
preparedStep.ArtifactLanes = append(preparedStep.ArtifactLanes, PreparedArtifactLane{Resolved: cloneResolvedArtifactLane(lane)})
preparedStep.ArtifactLanes = append(preparedStep.ArtifactLanes, PreparedArtifactLane{
Resolved: cloneResolvedArtifactLane(lane),
ExtractCorrectionProtocol: lane.ExtractCorrectionProtocol,
MergeCorrectionProtocol: lane.MergeCorrectionProtocol,
NormalizeCorrectionProtocol: lane.NormalizeCorrectionProtocol,
})
preparedStep.lanes = append(preparedStep.lanes, executor)
}
prepared.Steps[stepIndex] = preparedStep
@@ -199,6 +213,40 @@ func prepareEvidencePlan(resolved ResolvedPipeline, registries Registries, outpu
return plan, nil
}
func validateCorrectionRetryCapabilities(pipeline ResolvedPipeline) error {
validate := func(stage ModuleStage, laneID string, binding ModuleBinding, executionClass contracts.ExecutionClass, protocol contracts.CorrectionProtocol) error {
if executionClass != contracts.ExecutionClassLLMBacked || binding.Retries == 0 {
return nil
}
chain := resolvedValidatorChain(stage, laneID, binding.Module, pipeline.ValidatorChains)
if len(chain.Validators) == 0 || protocol == contracts.CorrectionProtocolSingleResponseV1 {
return nil
}
if laneID == "" {
return fmt.Errorf("pipeline %q %s module %q configures validators and retries but does not declare correction protocol %q", pipeline.ID, stage, binding.Module, contracts.CorrectionProtocolSingleResponseV1)
}
return fmt.Errorf("pipeline %q lane %q %s module %q configures validators and retries but does not declare correction protocol %q", pipeline.ID, laneID, stage, binding.Module, contracts.CorrectionProtocolSingleResponseV1)
}
if err := validate(StageChunk, "", pipeline.Chunk, pipeline.ChunkExecutionClass, pipeline.ChunkCorrectionProtocol); err != nil {
return err
}
for _, step := range pipeline.Steps {
for _, lane := range step.ArtifactLanes {
if err := validate(StageExtract, lane.ID, lane.Extract, lane.ExtractExecutionClass, lane.ExtractCorrectionProtocol); err != nil {
return err
}
if err := validate(StageMerge, lane.ID, lane.Merge, lane.MergeExecutionClass, lane.MergeCorrectionProtocol); err != nil {
return err
}
if err := validate(StageNormalize, lane.ID, lane.Normalize, lane.NormalizeExecutionClass, lane.NormalizeCorrectionProtocol); err != nil {
return err
}
}
}
return nil
}
func prepareLane(pipeline ResolvedPipeline, lane ResolvedArtifactLane, registries Registries, deps ModuleDependencies) (preparedLaneExecutor, error) {
executor := preparedLaneExecutor{resolved: cloneResolvedArtifactLane(lane)}
request := func(binding ModuleBinding, references contracts.ReferenceSet) BuildRequest {
@@ -406,6 +454,7 @@ func validateRegistrySet(resolved ResolvedPipeline, registries Registries) error
func cloneResolvedPipeline(in ResolvedPipeline) ResolvedPipeline {
out := in
out.ConfiguredValidationPolicy = cloneValidationPolicyOverride(in.ConfiguredValidationPolicy)
out.Input = cloneModuleBinding(in.Input)
out.Chunk = cloneModuleBinding(in.Chunk)
out.Output = cloneModuleBinding(in.Output)

View File

@@ -0,0 +1,391 @@
package pipeline
import (
"context"
"errors"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
// producerAttemptKind records why a producer invocation followed the prior
// one. It is deliberately independent of any artifact family.
type producerAttemptKind string
const (
producerAttemptInitial producerAttemptKind = "initial"
producerAttemptOperationalRetry producerAttemptKind = "operational_error_retry"
producerAttemptStructuralRetry producerAttemptKind = "structural_retry"
producerAttemptModuleRetry producerAttemptKind = "module_requested_retry"
producerAttemptSemanticRetry producerAttemptKind = "semantic_correction"
)
type producerAttemptOutcome string
const (
producerAttemptAccepted producerAttemptOutcome = "accepted"
producerAttemptRejected producerAttemptOutcome = "rejected"
producerAttemptIncompleteAccepted producerAttemptOutcome = "incomplete_accepted"
producerAttemptRetried producerAttemptOutcome = "retried"
producerAttemptFailed producerAttemptOutcome = "failed"
)
type producerTerminalAction string
const (
producerTerminalAccepted producerTerminalAction = "accepted"
producerTerminalRejected producerTerminalAction = "reject_output"
producerTerminalIncompleteAccepted producerTerminalAction = "warn_continue"
producerTerminalFailed producerTerminalAction = "fail_run"
)
// producerAttemptRequest contains only attempt-local control material. The
// producer reconstructs its ordinary request from its own durable inputs.
type producerAttemptRequest struct {
Number int
Kind producerAttemptKind
Correction *contracts.SemanticCorrection
}
// producerRetryDirective asks for another producer invocation while retaining
// the current value as a safe fallback if its shared budget is exhausted.
// Artifact-specific adapters are responsible for validating and populating it.
type producerRetryDirective struct {
FallbackWarnings []contracts.Warning
}
func (directive *producerRetryDirective) clone() *producerRetryDirective {
if directive == nil {
return nil
}
return &producerRetryDirective{FallbackWarnings: cloneWarnings(directive.FallbackWarnings)}
}
// producerAttemptOutput is intentionally artifact-neutral. Value remains
// opaque to the state machine; Candidate is the attempt-local response that
// may support semantic correction.
type producerAttemptOutput struct {
Value any
Candidate *contracts.ModelCandidate
Warnings []contracts.Warning
Retry *producerRetryDirective
}
func (output producerAttemptOutput) clone() (producerAttemptOutput, error) {
candidate, err := contracts.CloneModelCandidate(output.Candidate)
if err != nil {
return producerAttemptOutput{}, fmt.Errorf("clone model candidate: %w", err)
}
return producerAttemptOutput{
Value: output.Value,
Candidate: candidate,
Warnings: cloneWarnings(output.Warnings),
Retry: output.Retry.clone(),
}, nil
}
type producerAttemptProducer func(context.Context, producerAttemptRequest) (producerAttemptOutput, error)
type producerAttemptValidator func(context.Context, producerAttemptOutput) (validationReport, error)
type producerAttemptConfig struct {
Retries int
Policy ValidationPolicy
AllowStructuralRetry bool
}
// producerAttemptProvenance is ordered by producer invocation. It retains the
// settled validation report for later debug and durable-provenance adapters.
type producerAttemptProvenance struct {
Number int
Kind producerAttemptKind
Outcome producerAttemptOutcome
Validation validationReport
}
func (provenance producerAttemptProvenance) clone() producerAttemptProvenance {
provenance.Validation = cloneValidationReport(provenance.Validation)
return provenance
}
// producerAttemptTerminal describes the state-machine decision without
// materializing an artifact or persisting stage-specific diagnostics.
type producerAttemptTerminal struct {
Action producerTerminalAction
Value any
Warnings []contracts.Warning
Rejection *contracts.RejectedOutput
Validation validationReport
ValidationIncomplete bool
Provenance []producerAttemptProvenance
}
func (terminal producerAttemptTerminal) clone() producerAttemptTerminal {
terminal.Warnings = cloneWarnings(terminal.Warnings)
if terminal.Rejection != nil {
rejection := *terminal.Rejection
rejection.Validation = cloneValidationSummaryPtr(rejection.Validation)
terminal.Rejection = &rejection
}
terminal.Validation = cloneValidationReport(terminal.Validation)
terminal.Provenance = cloneProducerAttemptProvenance(terminal.Provenance)
return terminal
}
func cloneProducerAttemptProvenance(provenance []producerAttemptProvenance) []producerAttemptProvenance {
if len(provenance) == 0 {
return nil
}
cloned := make([]producerAttemptProvenance, len(provenance))
for index, item := range provenance {
cloned[index] = item.clone()
}
return cloned
}
func cloneValidationReport(report validationReport) validationReport {
return validationReport{records: report.Records()}
}
func runProducerAttempts(ctx context.Context, config producerAttemptConfig, produce producerAttemptProducer, validate producerAttemptValidator) (producerAttemptTerminal, error) {
if ctx == nil {
return producerAttemptTerminal{}, errors.New("producer attempt context must not be nil")
}
if config.Retries < 0 {
return producerAttemptTerminal{}, errors.New("producer retries must not be negative")
}
if produce == nil {
return producerAttemptTerminal{}, errors.New("producer attempt closure must not be nil")
}
if validate == nil {
return producerAttemptTerminal{}, errors.New("producer validation closure must not be nil")
}
attemptLimit := config.Retries + 1
provenance := make([]producerAttemptProvenance, 0, attemptLimit)
kind := producerAttemptInitial
var correction *contracts.SemanticCorrection
for number := 1; number <= attemptLimit; number++ {
if err := ctx.Err(); err != nil {
return failedProducerAttempt(provenance), err
}
requestCorrection, err := contracts.CloneSemanticCorrection(correction)
if err != nil {
return failedProducerAttempt(provenance), fmt.Errorf("clone semantic correction: %w", err)
}
output, err := produce(ctx, producerAttemptRequest{Number: number, Kind: kind, Correction: requestCorrection})
if err != nil {
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed})
if isImmediateProducerFailure(err) {
return failedProducerAttempt(provenance), err
}
if errors.Is(err, contracts.ErrInvalidStructuredOutput) && config.AllowStructuralRetry {
if number < attemptLimit {
kind, correction = producerAttemptStructuralRetry, nil
continue
}
return applyStructuralTerminalPolicy(config.Policy, provenance, number, err)
}
if number < attemptLimit {
kind, correction = producerAttemptOperationalRetry, nil
continue
}
return failedProducerAttempt(provenance), fmt.Errorf("producer failed after %d attempt(s): %w", number, err)
}
output, err = output.clone()
if err != nil {
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed})
return failedProducerAttempt(provenance), err
}
if output.Retry != nil && number < attemptLimit {
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptRetried})
kind, correction = producerAttemptModuleRetry, nil
continue
}
if output.Retry != nil {
output.Warnings = append(output.Warnings, cloneWarnings(output.Retry.FallbackWarnings)...)
}
correctionCandidate, err := contracts.CloneModelCandidate(output.Candidate)
if err != nil {
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed})
return failedProducerAttempt(provenance), fmt.Errorf("clone correction candidate: %w", err)
}
report, err := validate(ctx, output)
if err != nil {
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed})
return failedProducerAttempt(provenance), err
}
report = cloneValidationReport(report)
if err := ctx.Err(); err != nil {
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed, Validation: report})
return failedProducerAttempt(provenance), err
}
if rejection := report.FirstRejection(); rejection != nil {
if correctionCandidate != nil {
if err := correctionCandidate.Validate(); err != nil {
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed, Validation: report})
return failedProducerAttempt(provenance), fmt.Errorf("validate producer model candidate: %w", err)
}
}
if number < attemptLimit && correctionCandidate != nil && correctionCandidate.Protocol == contracts.CorrectionProtocolSingleResponseV1 {
correctionRequest, guidanceErr := report.CorrectionRequest()
if guidanceErr != nil {
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed, Validation: report})
return failedProducerAttempt(provenance), fmt.Errorf("construct semantic correction request: %w", guidanceErr)
}
correction, err = contracts.NewSemanticCorrection(correctionCandidate.Response, correctionRequest)
if err != nil {
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed, Validation: report})
return failedProducerAttempt(provenance), fmt.Errorf("construct semantic correction: %w", err)
}
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptRetried, Validation: report})
kind = producerAttemptSemanticRetry
continue
}
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptRejected, Validation: report})
return applySemanticTerminalPolicy(config.Policy, provenance, number, output, report, *rejection)
}
if incomplete := firstIncompleteValidation(report); incomplete != nil {
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptIncompleteAccepted, Validation: report})
if config.Policy.ValidatorFailure == ValidatorFailureWarnContinue {
warnings := terminalWarnings(output, report)
warnings = append(warnings, incompleteValidationWarnings(report)...)
return producerAttemptTerminal{Action: producerTerminalIncompleteAccepted, Value: output.Value, Warnings: warnings, Validation: report, ValidationIncomplete: true, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
}
return failedProducerAttempt(provenance), validatorFailureError(*incomplete)
}
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptAccepted, Validation: report})
return producerAttemptTerminal{Action: producerTerminalAccepted, Value: output.Value, Warnings: terminalWarnings(output, report), Validation: report, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
}
return failedProducerAttempt(provenance), errors.New("producer attempt budget was not exhausted deterministically")
}
func isImmediateProducerFailure(err error) bool {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return true
}
var debugErr *attemptDebugPersistenceError
return errors.As(err, &debugErr)
}
func applyStructuralTerminalPolicy(policy ValidationPolicy, provenance []producerAttemptProvenance, number int, err error) (producerAttemptTerminal, error) {
switch policy.ProducerStructuralFailure {
case ProducerStructuralFailureRejectOutput:
return producerAttemptTerminal{Action: producerTerminalRejected, Rejection: &contracts.RejectedOutput{ReasonCode: "invalid_structured_output", Message: "producer returned invalid structured output", AttemptCount: number}, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
case ProducerStructuralFailureFailRun:
return failedProducerAttempt(provenance), fmt.Errorf("producer returned invalid structured output after %d attempt(s): %w", number, err)
default:
return failedProducerAttempt(provenance), fmt.Errorf("unknown producer structural-failure action %q", policy.ProducerStructuralFailure)
}
}
func applySemanticTerminalPolicy(policy ValidationPolicy, provenance []producerAttemptProvenance, number int, output producerAttemptOutput, report validationReport, rejection validationRecord) (producerAttemptTerminal, error) {
rejected := contracts.RejectedOutput{ValidatorName: rejection.validatorName, ReasonCode: rejection.reasonCode, Message: rejection.message, AttemptCount: number, DiagnosticArtifactPath: rejection.diagnosticPath}
switch policy.SemanticRejection {
case SemanticRejectionRejectOutput:
return producerAttemptTerminal{Action: producerTerminalRejected, Warnings: terminalWarnings(output, report), Rejection: &rejected, Validation: report, ValidationIncomplete: firstIncompleteValidation(report) != nil, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
case SemanticRejectionFailRun:
return failedProducerAttempt(provenance), fmt.Errorf("producer candidate rejected after %d attempt(s): %s", number, rejection.message)
default:
return failedProducerAttempt(provenance), fmt.Errorf("unknown semantic-rejection action %q", policy.SemanticRejection)
}
}
func firstIncompleteValidation(report validationReport) *validationRecord {
for _, record := range report.records {
if record.outcome == validationFailed || record.outcome == validationSkipped {
clone := record.clone()
return &clone
}
}
return nil
}
func terminalWarnings(output producerAttemptOutput, report validationReport) []contracts.Warning {
warnings := cloneWarnings(output.Warnings)
warnings = append(warnings, report.Warnings()...)
return warnings
}
// incompleteValidationWarnings reports only validators that exhausted their
// execution budget. It never reports rejected candidates, and it uses fixed
// text so provider errors and correction content cannot cross this boundary.
func incompleteValidationWarnings(report validationReport) []contracts.Warning {
warnings := make([]contracts.Warning, 0)
for _, record := range report.records {
if record.outcome != validationFailed {
continue
}
warnings = append(warnings, contracts.Warning{
Scope: record.validatorName,
ReasonCode: "validator_execution_incomplete",
Message: "Validator execution did not complete within its configured budget.",
})
}
return warnings
}
// validationSummary projects a terminal state-machine result into the durable
// bounded representation used by manifests, rejections, and receipts.
func validationSummary(terminal producerAttemptTerminal, stage ModuleStage, stepID, laneID, moduleKey, chunkID string, chunkIndex int) artifacts.ValidationSummary {
summary := artifacts.ValidationSummary{
Stage: string(stage),
StepID: stepID,
LaneID: laneID,
ModuleKey: moduleKey,
ChunkID: chunkID,
ChunkIndex: chunkIndex,
ProducerAttemptCount: len(terminal.Provenance),
TerminalAction: string(terminal.Action),
}
switch {
case terminal.Action == producerTerminalRejected:
summary.Status = "rejected"
case terminal.Action == producerTerminalFailed:
summary.Status = "incomplete"
case terminal.ValidationIncomplete:
summary.Status = "incomplete"
default:
summary.Status = "complete"
}
seenValidators := make(map[string]struct{})
seenReasons := make(map[string]struct{})
seenIncomplete := make(map[string]struct{})
for _, record := range terminal.Validation.records {
if record.reasonCode != "" {
if _, exists := seenReasons[record.reasonCode]; !exists {
seenReasons[record.reasonCode] = struct{}{}
summary.ReasonCodes = append(summary.ReasonCodes, record.reasonCode)
}
}
if record.outcome == validationRejected {
if _, exists := seenValidators[record.validatorName]; !exists {
seenValidators[record.validatorName] = struct{}{}
summary.RejectingValidators = append(summary.RejectingValidators, record.validatorName)
}
}
if record.outcome == validationFailed || record.outcome == validationSkipped {
if _, exists := seenIncomplete[record.validatorName]; !exists {
seenIncomplete[record.validatorName] = struct{}{}
summary.IncompleteValidators = append(summary.IncompleteValidators, record.validatorName)
}
}
}
if terminal.Rejection != nil && terminal.Rejection.ReasonCode != "" {
if _, exists := seenReasons[terminal.Rejection.ReasonCode]; !exists {
summary.ReasonCodes = append(summary.ReasonCodes, terminal.Rejection.ReasonCode)
}
}
return summary
}
func failedProducerAttempt(provenance []producerAttemptProvenance) producerAttemptTerminal {
return producerAttemptTerminal{Action: producerTerminalFailed, Provenance: cloneProducerAttemptProvenance(provenance)}
}

View File

@@ -0,0 +1,414 @@
package pipeline
import (
"context"
"errors"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestRunProducerAttemptsUsesOneSharedBudget(t *testing.T) {
candidate := attemptCandidate(t, "first response")
producerCalls := 0
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 2, Policy: DefaultValidationPolicy(), AllowStructuralRetry: true}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
producerCalls++
switch request.Number {
case 1:
return producerAttemptOutput{}, errors.New("temporary producer failure")
case 2:
return producerAttemptOutput{}, contracts.ErrInvalidStructuredOutput
case 3:
return producerAttemptOutput{Value: "accepted", Candidate: candidate}, nil
default:
t.Fatalf("unexpected producer attempt %d", request.Number)
return producerAttemptOutput{}, nil
}
}, approveAttempt)
if err != nil {
t.Fatalf("runProducerAttempts() error = %v", err)
}
if terminal.Action != producerTerminalAccepted || terminal.Value != "accepted" {
t.Fatalf("terminal = %#v, want accepted value", terminal)
}
if producerCalls != 3 {
t.Fatalf("producer calls = %d, want 3", producerCalls)
}
if got := attemptKinds(terminal.Provenance); !reflect.DeepEqual(got, []producerAttemptKind{producerAttemptInitial, producerAttemptOperationalRetry, producerAttemptStructuralRetry}) {
t.Fatalf("attempt kinds = %v", got)
}
}
func TestRunProducerAttemptsReplacesSemanticCorrectionWithLatestResponse(t *testing.T) {
first := attemptCandidate(t, "first response")
second := attemptCandidate(t, "second response")
var corrections []*contracts.SemanticCorrection
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 2, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
if request.Correction != nil {
corrections = append(corrections, request.Correction)
}
switch request.Number {
case 1:
return producerAttemptOutput{Value: "one", Candidate: first}, nil
case 2:
return producerAttemptOutput{Value: "two", Candidate: second}, nil
case 3:
return producerAttemptOutput{Value: "three", Candidate: attemptCandidate(t, "third response")}, nil
default:
t.Fatalf("unexpected producer attempt %d", request.Number)
return producerAttemptOutput{}, nil
}
}, func(_ context.Context, output producerAttemptOutput) (validationReport, error) {
if output.Value == "three" {
return validationReport{}, nil
}
return rejectedAttemptReport("semantic_defect"), nil
})
if err != nil {
t.Fatalf("runProducerAttempts() error = %v", err)
}
if terminal.Action != producerTerminalAccepted {
t.Fatalf("terminal action = %q, want accepted", terminal.Action)
}
if len(corrections) != 2 {
t.Fatalf("correction count = %d, want 2", len(corrections))
}
if corrections[0] == corrections[1] {
t.Fatal("semantic corrections reused the same pointer")
}
if got := string(corrections[0].AssistantResponse); got != "first response" {
t.Fatalf("first correction response = %q", got)
}
if got := string(corrections[1].AssistantResponse); got != "second response" {
t.Fatalf("second correction response = %q", got)
}
if got := attemptKinds(terminal.Provenance); !reflect.DeepEqual(got, []producerAttemptKind{producerAttemptInitial, producerAttemptSemanticRetry, producerAttemptSemanticRetry}) {
t.Fatalf("attempt kinds = %v", got)
}
}
func TestRunProducerAttemptsAppliesTerminalPolicies(t *testing.T) {
tests := []struct {
name string
config producerAttemptConfig
produce producerAttemptProducer
validate producerAttemptValidator
wantAction producerTerminalAction
wantError bool
wantValue any
}{
{
name: "structural reject output",
config: producerAttemptConfig{Policy: ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureRejectOutput, SemanticRejection: SemanticRejectionFailRun, ValidatorFailure: ValidatorFailureWarnContinue}, AllowStructuralRetry: true},
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
return producerAttemptOutput{}, contracts.ErrInvalidStructuredOutput
},
validate: approveAttempt,
wantAction: producerTerminalRejected,
},
{
name: "structural fail run",
config: producerAttemptConfig{Policy: DefaultValidationPolicy(), AllowStructuralRetry: true},
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
return producerAttemptOutput{}, contracts.ErrInvalidStructuredOutput
},
validate: approveAttempt,
wantAction: producerTerminalFailed,
wantError: true,
},
{
name: "operational failure after retry exhaustion",
config: producerAttemptConfig{Policy: DefaultValidationPolicy()},
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
return producerAttemptOutput{}, errors.New("producer unavailable")
},
validate: approveAttempt,
wantAction: producerTerminalFailed,
wantError: true,
},
{
name: "semantic reject output",
config: producerAttemptConfig{Policy: ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureFailRun, SemanticRejection: SemanticRejectionRejectOutput, ValidatorFailure: ValidatorFailureWarnContinue}},
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
return producerAttemptOutput{Value: "discard", Candidate: attemptCandidate(t, "response")}, nil
},
validate: func(context.Context, producerAttemptOutput) (validationReport, error) {
return rejectedAttemptReport("bad"), nil
},
wantAction: producerTerminalRejected,
},
{
name: "semantic fail run",
config: producerAttemptConfig{Policy: DefaultValidationPolicy()},
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
return producerAttemptOutput{Candidate: attemptCandidate(t, "response")}, nil
},
validate: func(context.Context, producerAttemptOutput) (validationReport, error) {
return rejectedAttemptReport("bad"), nil
},
wantAction: producerTerminalFailed,
wantError: true,
},
{
name: "validator failure warns and continues",
config: producerAttemptConfig{Policy: DefaultValidationPolicy()},
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
return producerAttemptOutput{Value: "kept"}, nil
},
validate: func(context.Context, producerAttemptOutput) (validationReport, error) {
return failedAttemptReport(), nil
},
wantAction: producerTerminalIncompleteAccepted,
wantValue: "kept",
},
{
name: "validator failure fails run",
config: producerAttemptConfig{Policy: ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureFailRun, SemanticRejection: SemanticRejectionFailRun, ValidatorFailure: ValidatorFailureFailRun}},
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
return producerAttemptOutput{}, nil
},
validate: func(context.Context, producerAttemptOutput) (validationReport, error) {
return failedAttemptReport(), nil
},
wantAction: producerTerminalFailed,
wantError: true,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
terminal, err := runProducerAttempts(context.Background(), test.config, test.produce, test.validate)
if (err != nil) != test.wantError {
t.Fatalf("error = %v, want error %t", err, test.wantError)
}
if terminal.Action != test.wantAction {
t.Fatalf("terminal action = %q, want %q", terminal.Action, test.wantAction)
}
if terminal.Value != test.wantValue {
t.Fatalf("terminal value = %#v, want %#v", terminal.Value, test.wantValue)
}
if test.wantAction == producerTerminalRejected && (terminal.Rejection == nil || terminal.Value != nil) {
t.Fatalf("rejected terminal = %#v, want rejection without value", terminal)
}
})
}
}
func TestRunProducerAttemptsDoesNotRetryUncorrectableRejection(t *testing.T) {
calls := 0
policy := DefaultValidationPolicy()
policy.SemanticRejection = SemanticRejectionRejectOutput
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 3, Policy: policy}, func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
calls++
return producerAttemptOutput{Value: "deterministic"}, nil
}, func(context.Context, producerAttemptOutput) (validationReport, error) {
return rejectedAttemptReport("deterministic_rejection"), nil
})
if err != nil {
t.Fatalf("runProducerAttempts() error = %v", err)
}
if calls != 1 || terminal.Action != producerTerminalRejected || len(terminal.Provenance) != 1 {
t.Fatalf("terminal = %#v, calls = %d; want immediate rejection", terminal, calls)
}
}
func TestRunProducerAttemptsUsesModuleRetryBudgetAndFallback(t *testing.T) {
t.Run("retry", func(t *testing.T) {
calls := 0
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
calls++
if request.Number == 1 {
return producerAttemptOutput{Value: "fallback", Retry: &producerRetryDirective{}}, nil
}
return producerAttemptOutput{Value: "replacement"}, nil
}, approveAttempt)
if err != nil {
t.Fatalf("runProducerAttempts() error = %v", err)
}
if terminal.Action != producerTerminalAccepted || terminal.Value != "replacement" || calls != 2 {
t.Fatalf("terminal = %#v, calls = %d", terminal, calls)
}
if got := attemptKinds(terminal.Provenance); !reflect.DeepEqual(got, []producerAttemptKind{producerAttemptInitial, producerAttemptModuleRetry}) {
t.Fatalf("attempt kinds = %v", got)
}
})
t.Run("fallback", func(t *testing.T) {
fallbackWarning := contracts.Warning{ReasonCode: "fallback", Message: "fallback warning"}
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Policy: DefaultValidationPolicy()}, func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
return producerAttemptOutput{Value: "fallback", Retry: &producerRetryDirective{FallbackWarnings: []contracts.Warning{fallbackWarning}}}, nil
}, approveAttempt)
if err != nil {
t.Fatalf("runProducerAttempts() error = %v", err)
}
if terminal.Action != producerTerminalAccepted || terminal.Value != "fallback" || !reflect.DeepEqual(terminal.Warnings, []contracts.Warning{fallbackWarning}) {
t.Fatalf("terminal = %#v", terminal)
}
})
}
func TestRunProducerAttemptsRejectionWinsOverValidatorFailure(t *testing.T) {
firstWarnings := []contracts.Warning{{ReasonCode: "discarded", Message: "discarded warning"}}
secondWarnings := []contracts.Warning{{ReasonCode: "accepted", Message: "accepted warning"}}
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
if request.Number == 1 {
return producerAttemptOutput{Value: "first", Candidate: attemptCandidate(t, "defective"), Warnings: firstWarnings}, nil
}
return producerAttemptOutput{Value: "second", Candidate: attemptCandidate(t, "corrected"), Warnings: secondWarnings}, nil
}, func(_ context.Context, output producerAttemptOutput) (validationReport, error) {
if output.Value == "first" {
report := rejectedAttemptReport("defect")
report.records = append(report.records, validationRecord{validatorName: "unavailable", outcome: validationFailed, failure: errors.New("validator unavailable")})
return report, nil
}
return validationReport{}, nil
})
if err != nil {
t.Fatalf("runProducerAttempts() error = %v", err)
}
if terminal.Action != producerTerminalAccepted {
t.Fatalf("terminal action = %q, want accepted", terminal.Action)
}
if got := attemptKinds(terminal.Provenance); !reflect.DeepEqual(got, []producerAttemptKind{producerAttemptInitial, producerAttemptSemanticRetry}) {
t.Fatalf("attempt kinds = %v", got)
}
if got := terminal.Warnings; !reflect.DeepEqual(got, secondWarnings) {
t.Fatalf("terminal warnings = %#v, want %#v", got, secondWarnings)
}
if len(terminal.Provenance[0].Validation.records) != 2 {
t.Fatalf("first validation records = %#v, want rejection and failure", terminal.Provenance[0].Validation.records)
}
}
func TestValidationSummaryIsBoundedAndCorrectedSuccessIsQuiet(t *testing.T) {
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
if request.Number == 1 {
return producerAttemptOutput{Value: "first", Candidate: attemptCandidate(t, "sensitive defective response")}, nil
}
return producerAttemptOutput{Value: "corrected", Candidate: attemptCandidate(t, "sensitive corrected response")}, nil
}, func(_ context.Context, output producerAttemptOutput) (validationReport, error) {
if output.Value == "first" {
return validationReport{records: []validationRecord{{validatorName: "first", outcome: validationRejected, reasonCode: "needs_fix", message: "sensitive diagnostic", correctionGuidance: "sensitive guidance"}}}, nil
}
return validationReport{}, nil
})
if err != nil {
t.Fatalf("runProducerAttempts() error = %v", err)
}
summary := validationSummary(terminal, StageExtract, "step", "lane", "module", "chunk", 3)
if summary.Status != "complete" || summary.ProducerAttemptCount != 2 || summary.TerminalAction != string(producerTerminalAccepted) {
t.Fatalf("summary = %#v", summary)
}
if len(summary.ReasonCodes) != 0 || len(summary.RejectingValidators) != 0 || len(summary.IncompleteValidators) != 0 {
t.Fatalf("corrected success summary retained prior findings: %#v", summary)
}
if len(terminal.Warnings) != 0 {
t.Fatalf("corrected success warnings = %#v, want none", terminal.Warnings)
}
}
func TestWarnContinueRecordsOneWarningForEachExhaustedValidator(t *testing.T) {
report := validationReport{records: []validationRecord{
{validatorName: "first", outcome: validationFailed, attemptCount: 2, failure: errors.New("provider error with sensitive details")},
{validatorName: "second", outcome: validationSkipped, attemptCount: 1, reasonCode: "missing_prerequisite", message: "sensitive skipped detail"},
{validatorName: "third", outcome: validationFailed, attemptCount: 1, failure: errors.New("other provider error")},
}}
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Policy: DefaultValidationPolicy()}, func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
return producerAttemptOutput{Value: "candidate"}, nil
}, func(context.Context, producerAttemptOutput) (validationReport, error) {
return report, nil
})
if err != nil {
t.Fatalf("runProducerAttempts() error = %v", err)
}
if terminal.Action != producerTerminalIncompleteAccepted {
t.Fatalf("terminal action = %q", terminal.Action)
}
if got, want := terminal.Warnings, []contracts.Warning{
{Scope: "first", ReasonCode: "validator_execution_incomplete", Message: "Validator execution did not complete within its configured budget."},
{Scope: "third", ReasonCode: "validator_execution_incomplete", Message: "Validator execution did not complete within its configured budget."},
}; !reflect.DeepEqual(got, want) {
t.Fatalf("warnings = %#v, want %#v", got, want)
}
summary := validationSummary(terminal, StageNormalize, "step", "lane", "module", "", 0)
if summary.Status != "incomplete" || !reflect.DeepEqual(summary.IncompleteValidators, []string{"first", "second", "third"}) || !reflect.DeepEqual(summary.ReasonCodes, []string{"missing_prerequisite"}) {
t.Fatalf("summary = %#v", summary)
}
}
func TestRunProducerAttemptsStopsForCancellationAndDebugFailure(t *testing.T) {
tests := []struct {
name string
context context.Context
produce producerAttemptProducer
}{
{
name: "cancelled context",
context: cancelledAttemptContext(),
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
return producerAttemptOutput{}, nil
},
},
{
name: "debug persistence failure",
context: context.Background(),
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
return producerAttemptOutput{}, &attemptDebugPersistenceError{label: "attempt", err: errors.New("write debug")}
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
calls := 0
producer := func(ctx context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
calls++
return test.produce(ctx, request)
}
terminal, err := runProducerAttempts(test.context, producerAttemptConfig{Retries: 3, Policy: DefaultValidationPolicy()}, producer, approveAttempt)
if err == nil || terminal.Action != producerTerminalFailed {
t.Fatalf("terminal = %#v, error = %v", terminal, err)
}
if test.name == "cancelled context" && calls != 0 {
t.Fatalf("cancelled producer calls = %d, want 0", calls)
}
if test.name == "debug persistence failure" && calls != 1 {
t.Fatalf("debug failure producer calls = %d, want 1", calls)
}
})
}
}
func attemptCandidate(t *testing.T, response string) *contracts.ModelCandidate {
t.Helper()
candidate, err := contracts.NewModelCandidate([]byte(response), contracts.CorrectionProtocolSingleResponseV1)
if err != nil {
t.Fatalf("NewModelCandidate() error = %v", err)
}
return candidate
}
func approveAttempt(context.Context, producerAttemptOutput) (validationReport, error) {
return validationReport{}, nil
}
func rejectedAttemptReport(reason string) validationReport {
return validationReport{records: []validationRecord{{validatorName: "validator", outcome: validationRejected, reasonCode: reason, message: "candidate rejected", correctionGuidance: "fix the defect"}}}
}
func failedAttemptReport() validationReport {
return validationReport{records: []validationRecord{{validatorName: "validator", outcome: validationFailed, attemptCount: 1, failure: errors.New("validator failure")}}}
}
func attemptKinds(provenance []producerAttemptProvenance) []producerAttemptKind {
kinds := make([]producerAttemptKind, len(provenance))
for index, item := range provenance {
kinds[index] = item.Kind
}
return kinds
}
func cancelledAttemptContext() context.Context {
ctx, cancel := context.WithCancel(context.Background())
cancel()
return ctx
}

View File

@@ -24,6 +24,7 @@ type ModuleBinding struct {
Module string `json:"module"`
LLMProfile string `json:"llm_profile,omitempty"`
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
ValidationPolicy *ValidationPolicyOverride `json:"validation_policy,omitempty"`
Retries int `json:"retries,omitempty"`
Options map[string]any `json:"options,omitempty"`
References map[string]ReferenceSource `json:"references,omitempty"`
@@ -84,6 +85,7 @@ func (binding ModuleBinding) MarshalJSON() ([]byte, error) {
Module string `json:"module"`
LLMProfile string `json:"llm_profile,omitempty"`
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
ValidationPolicy *ValidationPolicyOverride `json:"validation_policy,omitempty"`
Retries int `json:"retries,omitempty"`
Options map[string]any `json:"options,omitempty"`
References map[string]ReferenceSource `json:"references,omitempty"`
@@ -93,6 +95,7 @@ func (binding ModuleBinding) MarshalJSON() ([]byte, error) {
Module: binding.Module,
LLMProfile: binding.LLMProfile,
StructuredOutputRepairAttempts: binding.StructuredOutputRepairAttempts,
ValidationPolicy: cloneValidationPolicyOverride(binding.ValidationPolicy),
Retries: binding.Retries,
Options: binding.Options,
References: binding.References,
@@ -112,6 +115,7 @@ func (binding *ModuleBinding) UnmarshalJSON(data []byte) error {
Module string `json:"module"`
LLMProfile string `json:"llm_profile,omitempty"`
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
ValidationPolicy *ValidationPolicyOverride `json:"validation_policy,omitempty"`
Retries int `json:"retries,omitempty"`
Options map[string]any `json:"options,omitempty"`
References map[string]ReferenceSource `json:"references,omitempty"`
@@ -125,6 +129,7 @@ func (binding *ModuleBinding) UnmarshalJSON(data []byte) error {
Module: decoded.Module,
LLMProfile: decoded.LLMProfile,
StructuredOutputRepairAttempts: decoded.StructuredOutputRepairAttempts,
ValidationPolicy: cloneValidationPolicyOverride(decoded.ValidationPolicy),
Retries: decoded.Retries,
Options: decoded.Options,
References: decoded.References,
@@ -158,6 +163,7 @@ type PipelineProfile struct {
ID string `json:"id"`
LLMProfile string `json:"llm_profile,omitempty"`
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
ValidationPolicy *ValidationPolicyOverride `json:"validation_policy,omitempty"`
Input ModuleBinding `json:"input"`
Chunk ModuleBinding `json:"chunk,omitempty"`
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
@@ -198,23 +204,29 @@ type ResolvedReferenceTarget struct {
}
type ResolvedArtifactLane struct {
StepID string
ID string
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
Extract ModuleBinding
ExtractExecutionClass contracts.ExecutionClass `json:"extract_execution_class"`
Merge ModuleBinding
MergeExecutionClass contracts.ExecutionClass `json:"merge_execution_class"`
Normalize ModuleBinding
NormalizeExecutionClass contracts.ExecutionClass `json:"normalize_execution_class"`
Validators []ModuleBinding
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
StepID string
ID string
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
Extract ModuleBinding
ExtractExecutionClass contracts.ExecutionClass `json:"extract_execution_class"`
ExtractCorrectionProtocol contracts.CorrectionProtocol `json:"extract_correction_protocol,omitempty"`
ExtractValidationPolicy ValidationPolicy `json:"extract_validation_policy"`
Merge ModuleBinding
MergeExecutionClass contracts.ExecutionClass `json:"merge_execution_class"`
MergeCorrectionProtocol contracts.CorrectionProtocol `json:"merge_correction_protocol,omitempty"`
MergeValidationPolicy ValidationPolicy `json:"merge_validation_policy"`
Normalize ModuleBinding
NormalizeExecutionClass contracts.ExecutionClass `json:"normalize_execution_class"`
NormalizeCorrectionProtocol contracts.CorrectionProtocol `json:"normalize_correction_protocol,omitempty"`
NormalizeValidationPolicy ValidationPolicy `json:"normalize_validation_policy"`
Validators []ModuleBinding
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
}
type ResolvedPipelineStep struct {
@@ -237,17 +249,20 @@ type ResolvedValidator struct {
}
type ResolvedPipeline struct {
ID string
Digest string
Input ModuleBinding
InputExecutionClass contracts.ExecutionClass `json:"input_execution_class"`
Chunk ModuleBinding
ChunkExecutionClass contracts.ExecutionClass `json:"chunk_execution_class"`
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
Steps []ResolvedPipelineStep
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
Output ModuleBinding
OutputExecutionClass contracts.ExecutionClass `json:"output_execution_class"`
ID string
Digest string
ConfiguredValidationPolicy *ValidationPolicyOverride `json:"configured_validation_policy,omitempty"`
Input ModuleBinding
InputExecutionClass contracts.ExecutionClass `json:"input_execution_class"`
Chunk ModuleBinding
ChunkExecutionClass contracts.ExecutionClass `json:"chunk_execution_class"`
ChunkCorrectionProtocol contracts.CorrectionProtocol `json:"chunk_correction_protocol,omitempty"`
ChunkValidationPolicy ValidationPolicy `json:"chunk_validation_policy"`
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
Steps []ResolvedPipelineStep
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
Output ModuleBinding
OutputExecutionClass contracts.ExecutionClass `json:"output_execution_class"`
}
// AllArtifactLanes returns lanes in deterministic step order for read-only
@@ -414,13 +429,15 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
return ResolvedPipeline{}, err
}
resolved := ResolvedPipeline{
ID: pipelineID,
Input: input,
InputExecutionClass: inputModuleSpec.ExecutionClass,
Chunk: chunk,
ChunkExecutionClass: chunkSpec.ExecutionClass,
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
Output: output,
ID: pipelineID,
ConfiguredValidationPolicy: cloneValidationPolicyOverride(profile.ValidationPolicy),
Input: input,
InputExecutionClass: inputModuleSpec.ExecutionClass,
Chunk: chunk,
ChunkExecutionClass: chunkSpec.ExecutionClass,
ChunkCorrectionProtocol: chunkSpec.CorrectionProtocol,
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
Output: output,
}
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, "", nil, catalog)
if err != nil {
@@ -493,6 +510,9 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
if err := applyEffectiveStructuredOutputRepairAttempts(&resolved, profile.StructuredOutputRepairAttempts); err != nil {
return ResolvedPipeline{}, err
}
if err := applyEffectiveValidationPolicies(&resolved, profile.ValidationPolicy); err != nil {
return ResolvedPipeline{}, err
}
if err := validateResolvedOptions(resolved, catalog, configuredLaneIDs); err != nil {
return ResolvedPipeline{}, err
}
@@ -600,6 +620,7 @@ func resolveArtifactLane(
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
lane.ExtractExecutionClass = extractSpec.ExecutionClass
lane.ExtractCorrectionProtocol = extractSpec.CorrectionProtocol
capabilities.add(extractSpec.Provides...)
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
@@ -626,6 +647,7 @@ func resolveArtifactLane(
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
lane.MergeReferences.StepID = strings.TrimSpace(stepID)
lane.MergeExecutionClass = mergeSpec.ExecutionClass
lane.MergeCorrectionProtocol = mergeSpec.CorrectionProtocol
capabilities.add(mergeSpec.Provides...)
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
@@ -652,6 +674,7 @@ func resolveArtifactLane(
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
lane.NormalizeExecutionClass = normalizeSpec.ExecutionClass
lane.NormalizeCorrectionProtocol = normalizeSpec.CorrectionProtocol
capabilities.add(normalizeSpec.Provides...)
if len(lane.Validators) > 0 {
@@ -913,6 +936,9 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
if err != nil {
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
}
if validator.Retries > 0 && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator %q retries require an LLM-backed execution class", pipelineID, stage, validator.Module)
}
chain.Validators = append(chain.Validators, ResolvedValidator{
Binding: cloneModuleBinding(validator),
ExecutionClass: spec.ExecutionClass,
@@ -1328,6 +1354,7 @@ func resolveBinding(binding ModuleBinding, defaultModule string, referenceSlotLa
Module: module,
LLMProfile: llmProfile,
StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(binding.StructuredOutputRepairAttempts),
ValidationPolicy: cloneValidationPolicyOverride(binding.ValidationPolicy),
Retries: binding.Retries,
Options: cloneOptions(binding.Options),
References: references,
@@ -1456,6 +1483,72 @@ func applyEffectiveStructuredOutputRepairAttempts(resolved *ResolvedPipeline, pi
return nil
}
func applyEffectiveValidationPolicies(resolved *ResolvedPipeline, pipelinePolicy *ValidationPolicyOverride) error {
apply := func(stage ModuleStage, laneID string, binding ModuleBinding, executionClass contracts.ExecutionClass) (ValidationPolicy, error) {
binding.ValidationPolicy = cloneValidationPolicyOverride(binding.ValidationPolicy)
if binding.ValidationPolicy != nil {
if err := binding.ValidationPolicy.Validate(); err != nil {
return ValidationPolicy{}, fmt.Errorf("pipeline %q %s validation_policy: %w", resolved.ID, stage, err)
}
if executionClass == contracts.ExecutionClassDeterministic && binding.ValidationPolicy.ProducerStructuralFailure != nil {
if laneID == "" {
return ValidationPolicy{}, fmt.Errorf("pipeline %q %s %q assigns producer_structural_failure to deterministic module", resolved.ID, stage, binding.Module)
}
return ValidationPolicy{}, fmt.Errorf("pipeline %q lane %q %s %q assigns producer_structural_failure to deterministic module", resolved.ID, laneID, stage, binding.Module)
}
}
return ResolveValidationPolicy(binding.ValidationPolicy, pipelinePolicy), nil
}
if pipelinePolicy != nil {
if err := pipelinePolicy.Validate(); err != nil {
return fmt.Errorf("pipeline %q validation_policy: %w", resolved.ID, err)
}
}
if resolved.Input.ValidationPolicy != nil {
return fmt.Errorf("pipeline %q input validation_policy is not supported", resolved.ID)
}
if resolved.Output.ValidationPolicy != nil {
return fmt.Errorf("pipeline %q output validation_policy is not supported", resolved.ID)
}
for _, chain := range resolved.ValidatorChains {
for _, validator := range chain.Validators {
if validator.Binding.ValidationPolicy != nil {
if chain.LaneID == "" {
return fmt.Errorf("pipeline %q %s validator %q validation_policy is not supported", resolved.ID, chain.Stage, validator.Binding.Module)
}
return fmt.Errorf("pipeline %q lane %q %s validator %q validation_policy is not supported", resolved.ID, chain.LaneID, chain.Stage, validator.Binding.Module)
}
}
}
policy, err := apply(StageChunk, "", resolved.Chunk, resolved.ChunkExecutionClass)
if err != nil {
return err
}
resolved.ChunkValidationPolicy = policy
for stepIndex := range resolved.Steps {
for laneIndex := range resolved.Steps[stepIndex].ArtifactLanes {
lane := &resolved.Steps[stepIndex].ArtifactLanes[laneIndex]
policy, err = apply(StageExtract, lane.ID, lane.Extract, lane.ExtractExecutionClass)
if err != nil {
return err
}
lane.ExtractValidationPolicy = policy
policy, err = apply(StageMerge, lane.ID, lane.Merge, lane.MergeExecutionClass)
if err != nil {
return err
}
lane.MergeValidationPolicy = policy
policy, err = apply(StageNormalize, lane.ID, lane.Normalize, lane.NormalizeExecutionClass)
if err != nil {
return err
}
lane.NormalizeValidationPolicy = policy
}
}
return nil
}
func resolveBindings(bindings []ModuleBinding, defaultModule string, referenceSlotLabel string) ([]ModuleBinding, error) {
if len(bindings) == 0 {
return nil, nil
@@ -1549,27 +1642,33 @@ func selectedArtifactLanes(pipelineID string, artifacts map[string]ArtifactLaneP
func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
withoutDigest := struct {
ID string
Input ModuleBinding
InputExecutionClass contracts.ExecutionClass
Chunk ModuleBinding
ChunkExecutionClass contracts.ExecutionClass
ChunkReferences ResolvedReferenceTarget
Steps []ResolvedPipelineStep
ValidatorChains []ResolvedValidatorChain
Output ModuleBinding
OutputExecutionClass contracts.ExecutionClass
ID string
ConfiguredValidationPolicy *ValidationPolicyOverride
Input ModuleBinding
InputExecutionClass contracts.ExecutionClass
Chunk ModuleBinding
ChunkExecutionClass contracts.ExecutionClass
ChunkCorrectionProtocol contracts.CorrectionProtocol
ChunkValidationPolicy ValidationPolicy
ChunkReferences ResolvedReferenceTarget
Steps []ResolvedPipelineStep
ValidatorChains []ResolvedValidatorChain
Output ModuleBinding
OutputExecutionClass contracts.ExecutionClass
}{
ID: resolved.ID,
Input: resolved.Input,
InputExecutionClass: resolved.InputExecutionClass,
Chunk: resolved.Chunk,
ChunkExecutionClass: resolved.ChunkExecutionClass,
ChunkReferences: resolved.ChunkReferences,
Steps: resolved.Steps,
ValidatorChains: resolved.ValidatorChains,
Output: resolved.Output,
OutputExecutionClass: resolved.OutputExecutionClass,
ID: resolved.ID,
ConfiguredValidationPolicy: cloneValidationPolicyOverride(resolved.ConfiguredValidationPolicy),
ChunkValidationPolicy: resolved.ChunkValidationPolicy,
Input: resolved.Input,
InputExecutionClass: resolved.InputExecutionClass,
Chunk: resolved.Chunk,
ChunkExecutionClass: resolved.ChunkExecutionClass,
ChunkCorrectionProtocol: resolved.ChunkCorrectionProtocol,
ChunkReferences: resolved.ChunkReferences,
Steps: resolved.Steps,
ValidatorChains: resolved.ValidatorChains,
Output: resolved.Output,
OutputExecutionClass: resolved.OutputExecutionClass,
}
encoded, err := json.Marshal(withoutDigest)
if err != nil {

View File

@@ -126,6 +126,148 @@ func TestModuleCatalogExecutionClassLooksUpRegisteredMetadata(t *testing.T) {
}
}
func TestResolvePipelineCarriesCorrectionProtocolsIntoPreparedMetadata(t *testing.T) {
correction := contracts.CorrectionProtocolSingleResponseV1
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked, Provides: []string{"source"}},
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, CorrectionProtocol: correction, Requires: []string{"source"}, Provides: []string{"chunk"}},
ModuleSpec{Key: "llm-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, CorrectionProtocol: correction, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, CorrectionProtocol: correction, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, CorrectionProtocol: correction, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
)
resolved, err := ResolvePipeline(llmProfilePipeline(), ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if resolved.ChunkCorrectionProtocol != correction {
t.Fatalf("ChunkCorrectionProtocol = %q, want %q", resolved.ChunkCorrectionProtocol, correction)
}
lane := resolved.Steps[0].ArtifactLanes[0]
if lane.ExtractCorrectionProtocol != correction || lane.MergeCorrectionProtocol != correction || lane.NormalizeCorrectionProtocol != correction {
t.Fatalf("lane correction protocols = %q/%q/%q, want %q", lane.ExtractCorrectionProtocol, lane.MergeCorrectionProtocol, lane.NormalizeCorrectionProtocol, correction)
}
withoutCapability := cloneResolvedPipeline(resolved)
withoutCapability.ChunkCorrectionProtocol = ""
withoutCapability.Steps[0].ArtifactLanes[0].ExtractCorrectionProtocol = ""
withoutCapability.Steps[0].ArtifactLanes[0].MergeCorrectionProtocol = ""
withoutCapability.Steps[0].ArtifactLanes[0].NormalizeCorrectionProtocol = ""
withoutCapabilityDigest, err := resolvedPipelineDigest(withoutCapability)
if err != nil {
t.Fatalf("resolvedPipelineDigest() error = %v", err)
}
if withoutCapabilityDigest == resolved.Digest {
t.Fatalf("resolved digest = %q with and without correction capability, want changed", resolved.Digest)
}
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
}
if prepared.ChunkCorrectionProtocol != correction {
t.Fatalf("prepared ChunkCorrectionProtocol = %q, want %q", prepared.ChunkCorrectionProtocol, correction)
}
preparedLane := prepared.Steps[0].ArtifactLanes[0]
if preparedLane.ExtractCorrectionProtocol != correction || preparedLane.MergeCorrectionProtocol != correction || preparedLane.NormalizeCorrectionProtocol != correction {
t.Fatalf("prepared lane correction protocols = %q/%q/%q, want %q", preparedLane.ExtractCorrectionProtocol, preparedLane.MergeCorrectionProtocol, preparedLane.NormalizeCorrectionProtocol, correction)
}
}
func TestPrepareRequiresCorrectionCapabilityForValidatorRetries(t *testing.T) {
for _, test := range []struct {
name string
protocol contracts.CorrectionProtocol
retries int
validators bool
want string
}{
{name: "supported", protocol: contracts.CorrectionProtocolSingleResponseV1, retries: 1, validators: true},
{name: "unsupported", retries: 1, validators: true, want: "does not declare correction protocol"},
{name: "no validators", retries: 1},
{name: "no retries", validators: true},
} {
t.Run(test.name, func(t *testing.T) {
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked, Provides: []string{"source"}},
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, CorrectionProtocol: test.protocol, Requires: []string{"source"}, Provides: []string{"chunk"}},
ModuleSpec{Key: "llm-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
)
if err := RegisterChunkValidator(catalog.Validators, ValidatorSpec{Key: "chunk-validator", ExecutionClass: contracts.ExecutionClassLLMBacked}, func() (contracts.ChunkValidator, error) {
return llmProfileTestChunkValidator{key: "chunk-validator"}, nil
}); err != nil {
t.Fatal(err)
}
profile := llmProfilePipeline()
profile.Chunk.Retries = test.retries
if test.validators {
profile.Chunk.Validators = ValidatorOverride{Set: true, Validators: []ModuleBinding{{Module: "chunk-validator"}}}
}
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
_, err = Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
if test.want != "" {
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("Prepare() error = %v, want %q", err, test.want)
}
return
}
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
}
})
}
}
type llmProfileTestChunkValidator struct{ key string }
func (validator llmProfileTestChunkValidator) Name() string { return validator.key }
func (llmProfileTestChunkValidator) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassLLMBacked
}
func (llmProfileTestChunkValidator) Validate(context.Context, contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: true}, nil
}
func TestResolvePipelineValidatorRetriesRequireLLMBackedValidator(t *testing.T) {
for _, test := range []struct {
name string
class contracts.ExecutionClass
want string
}{
{name: "deterministic validator", class: contracts.ExecutionClassDeterministic, want: "retries require an LLM-backed"},
{name: "LLM validator", class: contracts.ExecutionClassLLMBacked},
} {
t.Run(test.name, func(t *testing.T) {
catalog := newProfileCatalog(t)
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "retrying-validator", ExecutionClass: test.class})
profile := baselineProfile()
lane := profile.Artifacts["events"]
lane.Extract.Validators = ValidatorOverride{Set: true, Validators: []ModuleBinding{{Module: "retrying-validator", Retries: 1}}}
profile.Artifacts["events"] = lane
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
if test.want != "" {
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("ResolvePipeline() error = %v, want %q", err, test.want)
}
return
}
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if got := resolved.ValidatorChains[1].Validators[0].Binding.Retries; got != 1 {
t.Fatalf("validator retries = %d, want 1", got)
}
})
}
}
func TestResolvePipelineAppliesDefaults(t *testing.T) {
resolved, err := ResolvePipeline(PipelineProfile{
ID: "defaulted",

View File

@@ -61,20 +61,24 @@ type RunInput struct {
// single worker so direct framework callers retain deterministic behavior.
ExtractWorkers int
pipeline ResolvedPipeline
llmClient contracts.StructuredLLMClient
stepID string
references map[referenceTargetKey]contracts.ReferenceSet
pipeline ResolvedPipeline
llmClient contracts.StructuredLLMClient
stepID string
references map[referenceTargetKey]contracts.ReferenceSet
referenceReuseEligibility map[referenceTargetKey]bool
}
type RunOutput struct {
Manifest artifacts.RunManifest `json:"manifest"`
ChunkPlan *artifacts.ChunkPlanSummary `json:"chunk_plan,omitempty"`
NormalizeOutputs []contracts.SerializedOutput `json:"normalize_outputs,omitempty"`
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
Warnings []contracts.Warning `json:"warnings,omitempty"`
OutputFiles []contracts.OutputFile `json:"-"`
CheckpointEvents []CheckpointEvent `json:"checkpoint_events,omitempty"`
Manifest artifacts.RunManifest `json:"manifest"`
ChunkPlan *artifacts.ChunkPlanSummary `json:"chunk_plan,omitempty"`
NormalizeOutputs []contracts.SerializedOutput `json:"normalize_outputs,omitempty"`
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
Warnings []contracts.Warning `json:"warnings,omitempty"`
OutputFiles []contracts.OutputFile `json:"-"`
CheckpointEvents []CheckpointEvent `json:"checkpoint_events,omitempty"`
ValidationSummaries []artifacts.ValidationSummary `json:"validation_summaries,omitempty"`
normalizeReuseEligibility map[generatedOutputKey]bool
}
func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err error) {
@@ -241,6 +245,9 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
if chunkResult.rejection != nil {
output.Rejected = append(output.Rejected, *chunkResult.rejection)
}
if chunkResult.validation != nil {
output.ValidationSummaries = append(output.ValidationSummaries, artifacts.CloneValidationSummary(*chunkResult.validation))
}
output.Warnings = append(output.Warnings, chunkResult.warnings...)
chunkDebugPayload := map[string]any{
"cache_mode": chunkMode,
@@ -297,6 +304,8 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
if len(output.Rejected) > 0 {
output.Manifest.ValidationStatus = "rejected"
} else if hasIncompleteValidation(output.ValidationSummaries) {
output.Manifest.ValidationStatus = "incomplete"
} else {
output.Manifest.ValidationStatus = "approved"
}
@@ -401,6 +410,15 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
return output, nil
}
func hasIncompleteValidation(summaries []artifacts.ValidationSummary) bool {
for _, summary := range summaries {
if summary.Status == "incomplete" {
return true
}
}
return false
}
func (r *Runner) runPreparedSteps(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk, output *RunOutput) error {
for _, step := range input.Prepared.Steps {
stepInput := input
@@ -410,6 +428,7 @@ func (r *Runner) runPreparedSteps(ctx context.Context, input RunInput, checkpoin
return fmt.Errorf("prepare generated references for pipeline step %q: %w", step.ID, err)
}
stepInput.references = stepReferences
stepInput.referenceReuseEligibility = buildStepReferenceReuseEligibility(step, output.normalizeReuseEligibility)
output.Manifest.References = append(output.Manifest.References, referenceProvenance...)
laneOutput, laneErr := r.runLanes(ctx, stepInput, step, checkpoints, loader, doc, sourceInput, sessionID, chunks)
if err := mergeLaneOutput(output, laneOutput); err != nil {
@@ -428,7 +447,7 @@ type retryAttemptResult struct {
warnings []contracts.Warning
}
func runWithRetry(ctx context.Context, retries int, run func(attempt int) (retryAttemptResult, error)) (retryAttemptResult, error) {
func runSimpleRetry(ctx context.Context, retries int, run func(attempt int) (retryAttemptResult, error)) (retryAttemptResult, error) {
attempts := retries + 1
var last retryAttemptResult
for attempt := 1; attempt <= attempts; attempt++ {
@@ -470,26 +489,25 @@ func runWithRetry(ctx context.Context, retries int, run func(attempt int) (retry
return last, nil
}
func (r *Runner) validateChunks(ctx context.Context, doc *source.SourceDocument, moduleKey string, chunks []source.Chunk, sourceInput contracts.LLMInputMaterial, sessionID string, references contracts.ReferenceSet, metadata map[string]any, prepared preparedValidatorChain, attempt int, debug DebugRecorder) ([]contracts.Warning, *contracts.RejectedOutput, error) {
func (r *Runner) validateChunkReport(ctx context.Context, doc *source.SourceDocument, moduleKey string, chunks []source.Chunk, sourceInput contracts.LLMInputMaterial, sessionID string, references contracts.ReferenceSet, metadata map[string]any, prepared preparedValidatorChain, attempt int, debug DebugRecorder) (validationReport, error) {
content, err := json.Marshal(chunks)
if err != nil {
return nil, nil, fmt.Errorf("encode canonical chunks for validation: %w", err)
return validationReport{}, fmt.Errorf("encode canonical chunks for validation: %w", err)
}
schema := contracts.ArtifactSchema{ID: "notarius.source.chunks", Name: "notarius_source_chunks", Version: "v1", JSONSchema: []byte(`{"$schema":"https://json-schema.org/draft/2020-12/schema","type":"array"}`)}
var warnings []contracts.Warning
for index, item := range prepared.validators {
report, err := executeValidationChain(ctx, prepared, func(validatorCtx context.Context, item preparedValidator, validatorAttempt int) (validationInvocation, error) {
binding := item.resolved.Binding
started := time.Now().UTC()
attemptPath := path.Join("validate", fileio.EncodePathComponent(string(StageChunk)), "", fileio.EncodePathComponent(moduleKey), fmt.Sprintf("%02d-%s-attempt-%02d", index+1, fileio.EncodePathComponent(binding.Module), attempt))
validatorCtx, llmScope := withIsolatedDebugLLMScope(ctx, attemptPath)
attemptPath := validatorAttemptPath(path.Join("validate", fileio.EncodePathComponent(string(StageChunk)), "", fileio.EncodePathComponent(moduleKey), fmt.Sprintf("%02d-%s-attempt-%02d", item.position, fileio.EncodePathComponent(binding.Module), attempt)), validatorAttempt)
validatorCtx, llmScope := withIsolatedDebugLLMScope(validatorCtx, attemptPath)
var result contracts.ValidationResult
requestMetadata, cloneErr := cloneMetadata(metadata)
if cloneErr != nil {
return nil, nil, fmt.Errorf("clone chunk validation metadata: %w", cloneErr)
return validationInvocation{}, fatalValidationError(fmt.Errorf("clone chunk validation metadata: %w", cloneErr))
}
requestChunks, cloneErr := cloneSourceChunks(chunks)
if cloneErr != nil {
return nil, nil, fmt.Errorf("clone chunks for validation: %w", cloneErr)
return validationInvocation{}, fatalValidationError(fmt.Errorf("clone chunks for validation: %w", cloneErr))
}
switch item.resolved.Target {
case ValidatorTargetChunk:
@@ -497,7 +515,10 @@ func (r *Runner) validateChunks(ctx context.Context, doc *source.SourceDocument,
case ValidatorTargetSerialized:
result, err = item.serialized.Validate(validatorCtx, contracts.SerializedValidationRequest{Stage: string(StageChunk), ModuleKey: moduleKey, Source: doc, SourceID: doc.ID, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(references), LLMProfile: binding.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(binding.StructuredOutputRepairAttempts), Metadata: requestMetadata, Chunks: requestChunks, Schema: contracts.CloneArtifactSchema(schema), MediaType: "application/json", Content: append([]byte(nil), content...)})
default:
return nil, nil, fmt.Errorf("validator %q is incompatible with chunk validation", binding.Module)
return validationInvocation{}, fatalValidationError(fmt.Errorf("validator %q is incompatible with chunk validation", binding.Module))
}
if err == nil {
err = contracts.ValidateValidationResult(result)
}
debugContent := debugContentEnvelope(content, "application/json", nil, nil)
debugContent.ContentDigest = debugContentDigest(content)
@@ -508,27 +529,19 @@ func (r *Runner) validateChunks(ctx context.Context, doc *source.SourceDocument,
if err != nil {
validationErr := fmt.Errorf("validate chunks with validator %q: %w", binding.Module, err)
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(StageChunk), ModuleKey: moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall, Error: debugCall.Error}, llmScope); debugErr != nil {
return warnings, nil, errors.Join(validationErr, fmt.Errorf("write chunk validator attempt debug artifact: %w", debugErr))
return validationInvocation{}, fatalValidationError(errors.Join(validationErr, fmt.Errorf("write chunk validator attempt debug artifact: %w", debugErr)))
}
return warnings, nil, validationErr
return validationInvocation{}, validationErr
}
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(StageChunk), ModuleKey: moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall}, llmScope); debugErr != nil {
return warnings, nil, fmt.Errorf("write chunk validator attempt debug artifact: %w", debugErr)
return validationInvocation{}, fatalValidationError(fmt.Errorf("write chunk validator attempt debug artifact: %w", debugErr))
}
if !result.Approved {
reason := result.ReasonCode
if reason == "" {
reason = "output_rejected"
}
message := result.Message
if message == "" {
message = "output rejected"
}
return warnings, &contracts.RejectedOutput{Stage: string(StageChunk), ModuleKey: moduleKey, ValidatorName: binding.Module, ReasonCode: reason, Message: message, AttemptCount: attempt, DiagnosticArtifactPath: result.DiagnosticArtifactPath}, nil
}
warnings = append(warnings, result.Warnings...)
return validationInvocation{result: result}, nil
})
if err != nil {
return report, err
}
return warnings, nil, nil
return report, nil
}
func resolvedValidatorChain(stage ModuleStage, laneID string, moduleKey string, chains []ResolvedValidatorChain) ResolvedValidatorChain {
@@ -726,6 +739,9 @@ func populateOutputManifest(output *RunOutput) {
}
output.Manifest.NormalizedOutputs = normalizedOutputManifests(output.NormalizeOutputs)
output.Manifest.RejectedOutputs = rejectedOutputManifests(output.Rejected)
if len(output.ValidationSummaries) > 0 {
output.Manifest.ValidationSummaries = cloneValidationSummaries(output.ValidationSummaries)
}
if len(output.CheckpointEvents) > 0 {
decisions := make([]artifacts.CheckpointDecisionManifest, 0, len(output.CheckpointEvents))
for _, event := range output.CheckpointEvents {
@@ -775,6 +791,7 @@ func rejectedOutputManifests(rejected []contracts.RejectedOutput) []artifacts.Re
Message: output.Message,
AttemptCount: output.AttemptCount,
DiagnosticArtifactPath: output.DiagnosticArtifactPath,
Validation: cloneValidationSummaryPtr(output.Validation),
})
}
return manifests
@@ -1033,7 +1050,31 @@ func cloneRejectedOutputs(rejected []contracts.RejectedOutput) []contracts.Rejec
if len(rejected) == 0 {
return nil
}
return append([]contracts.RejectedOutput(nil), rejected...)
cloned := make([]contracts.RejectedOutput, len(rejected))
for index, item := range rejected {
cloned[index] = item
cloned[index].Validation = cloneValidationSummaryPtr(item.Validation)
}
return cloned
}
func cloneValidationSummaryPtr(summary *artifacts.ValidationSummary) *artifacts.ValidationSummary {
if summary == nil {
return nil
}
cloned := artifacts.CloneValidationSummary(*summary)
return &cloned
}
func cloneValidationSummaries(summaries []artifacts.ValidationSummary) []artifacts.ValidationSummary {
if len(summaries) == 0 {
return nil
}
cloned := make([]artifacts.ValidationSummary, len(summaries))
for index, summary := range summaries {
cloned[index] = artifacts.CloneValidationSummary(summary)
}
return cloned
}
func timePtr(t time.Time) *time.Time {

View File

@@ -230,14 +230,14 @@ func TestRunnerRecordsDistinctRetryAttemptsAndPromotesAcceptedWarningsOnly(t *te
if attempts == 1 {
scope = "discarded"
}
return erasedTypedResult{Value: codecNotes{Items: []string{scope}}, Warnings: []contracts.Warning{{Scope: scope, ReasonCode: "observed", Message: scope}}}, nil
return erasedTypedResult{Value: codecNotes{Items: []string{scope}}, Warnings: []contracts.Warning{{Scope: scope, ReasonCode: "observed", Message: scope}}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%s"]}`, scope))}, nil
}
validatorCalls := 0
validator := preparedValidator{
resolved: ResolvedValidator{Binding: Binding("retry-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
validatorCalls++
return contracts.ValidationResult{Approved: validatorCalls > 1, ReasonCode: "retry", Message: "retry candidate"}, nil
return contracts.ValidationResult{Approved: validatorCalls > 1, ReasonCode: "retry", Message: "retry candidate", CorrectionGuidance: "return an acceptable candidate"}, nil
},
}
switch stage {
@@ -280,11 +280,12 @@ func TestRunnerRecordsDistinctRetryAttemptsAndPromotesAcceptedWarningsOnly(t *te
func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
tests := []struct {
name string
configure func(*PreparedPipeline)
path string
wantError string
wantBody string
name string
configure func(*PreparedPipeline)
path string
wantError string
wantBody string
attemptError bool
}{
{
name: "merge module error",
@@ -293,12 +294,14 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
return erasedTypedResult{}, errors.New("merge exploded")
}
},
path: "merge/notes/attempt-01.json",
wantError: "merge exploded",
path: "merge/notes/attempt-01.json",
wantError: "merge exploded",
attemptError: true,
},
{
name: "normalize validator error",
configure: func(prepared *PreparedPipeline) {
prepared.Steps[0].lanes[0].resolved.NormalizeValidationPolicy.ValidatorFailure = ValidatorFailureFailRun
prepared.Steps[0].lanes[0].normalizeValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("error-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
@@ -313,10 +316,11 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
{
name: "merge final rejection",
configure: func(prepared *PreparedPipeline) {
prepared.Steps[0].lanes[0].resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
prepared.Steps[0].lanes[0].mergeValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("reject-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted"}, nil
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted", CorrectionGuidance: "return an acceptable candidate"}, nil
},
}}
},
@@ -330,8 +334,9 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
return erasedTypedResult{Value: "wrong artifact type"}, nil
}
},
path: "normalize/notes/attempt-01.json",
wantError: "serialize normalize candidate",
path: "normalize/notes/attempt-01.json",
wantError: "serialize normalize candidate",
attemptError: true,
},
}
@@ -343,9 +348,15 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
output, runErr := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug})
envelope := debug.envelope(t, tc.path)
if tc.wantError != "" {
if runErr == nil || !strings.Contains(runErr.Error(), tc.wantError) || !strings.Contains(envelope.Error, tc.wantError) {
if runErr == nil || !strings.Contains(runErr.Error(), tc.wantError) {
t.Fatalf("run error = %v, envelope error = %q; want %q", runErr, envelope.Error, tc.wantError)
}
if tc.attemptError && !strings.Contains(envelope.Error, tc.wantError) {
t.Fatalf("attempt envelope error = %q, want %q", envelope.Error, tc.wantError)
}
if !tc.attemptError && envelope.Error != "" {
t.Fatalf("settled validator failure attempt error = %q, want empty", envelope.Error)
}
} else if runErr != nil {
t.Fatalf("Run() error = %v, want nil rejection outcome", runErr)
}

View File

@@ -160,7 +160,7 @@ func setCandidateValidator(prepared *PreparedPipeline, target ModuleStage, appro
validator := preparedValidator{
resolved: ResolvedValidator{Binding: Binding("candidate-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: approved, ReasonCode: "candidate_rejected", Message: "candidate rejected by validator"}, nil
return contracts.ValidationResult{Approved: approved, ReasonCode: "candidate_rejected", Message: "candidate rejected by validator", CorrectionGuidance: "return an acceptable candidate"}, nil
},
}
switch target {
@@ -187,6 +187,11 @@ func TestRunnerIsolatesTypedValidatorCandidates(t *testing.T) {
for _, target := range []ModuleStage{StageExtract, StageMerge, StageNormalize} {
t.Run(string(target), func(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
if target == StageMerge {
prepared.Steps[0].lanes[0].resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
} else {
prepared.Steps[0].lanes[0].resolved.NormalizeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
}
prepared.Steps[0].lanes[0].extractValidators = preparedValidatorChain{}
prepared.Steps[0].lanes[0].mergeValidators = preparedValidatorChain{}
prepared.Steps[0].lanes[0].normalizeValidators = preparedValidatorChain{}
@@ -306,6 +311,11 @@ func TestRunnerRejectsCandidatesBeforeFinalEncoding(t *testing.T) {
for _, target := range []ModuleStage{StageMerge, StageNormalize} {
t.Run(string(target), func(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
if target == StageMerge {
prepared.Steps[0].lanes[0].resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
} else {
prepared.Steps[0].lanes[0].resolved.NormalizeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
}
codec := &observedNotesCodec{}
installObservedNotesCodec(t, prepared, codec)
configureCandidateOperation(prepared, target, codecNotes{Items: []string{"invalid"}})

View File

@@ -12,15 +12,24 @@ import (
)
type chunkPlanExecution struct {
accepted bool
chunks []source.Chunk
plan *source.ChunkPlan
warnings []contracts.Warning
rejection *contracts.RejectedOutput
lookup ChunkPlanDecision
record *ChunkPlanRecord
action string
summary artifacts.ChunkPlanSummary
accepted bool
chunks []source.Chunk
plan *source.ChunkPlan
warnings []contracts.Warning
rejection *contracts.RejectedOutput
lookup ChunkPlanDecision
record *ChunkPlanRecord
action string
summary artifacts.ChunkPlanSummary
validation *artifacts.ValidationSummary
}
type generatedChunkPlanCandidate struct {
plan source.ChunkPlan
chunks []source.Chunk
record ChunkPlanRecord
producerWarnings []contracts.Warning
terminal *attemptTerminalRecorder
}
func effectiveChunkCacheMode(mode ChunkCacheMode) ChunkCacheMode {
@@ -58,17 +67,38 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
case ChunkPlanHit:
plan, chunks, validationErr := validateAndMaterializeChunkPlan(doc, record.Plan)
if validationErr == nil {
if err := result.setCandidate(record, "reused"); err != nil {
return result, fmt.Errorf("clone reused chunk plan record: %w", err)
report, err := r.validateChunkReport(ctx, doc, chunker.Key(), chunks, sourceInput, sessionID, input.pipeline.ChunkReferences.ReferenceSet, input.Metadata, input.Prepared.chunkValidators, 1, input.Debug)
if err != nil {
result.setValidation(report.Warnings(), nil, err)
return result, err
}
validationWarnings, rejection, err := r.validateChunks(ctx, doc, chunker.Key(), chunks, sourceInput, sessionID, input.pipeline.ChunkReferences.ReferenceSet, input.Metadata, input.Prepared.chunkValidators, 1, input.Debug)
result.plan = &plan
result.chunks = chunks
result.warnings = append(cloneWarnings(record.Warnings), validationWarnings...)
result.rejection = rejection
result.accepted = rejection == nil && err == nil
result.setValidation(validationWarnings, rejection, err)
return result, err
rejection := report.FirstRejection()
incomplete := firstIncompleteValidation(report)
if rejection == nil && incomplete == nil {
if err := result.setCandidate(record, "reused"); err != nil {
return result, fmt.Errorf("clone reused chunk plan record: %w", err)
}
result.plan = &plan
result.chunks = chunks
result.warnings = append(cloneWarnings(record.Warnings), report.Warnings()...)
result.accepted = true
result.setValidation(report.Warnings(), nil, nil)
cachedTerminal := producerAttemptTerminal{Action: producerTerminalAccepted, Validation: report}
summary := validationSummary(cachedTerminal, StageChunk, "", "", chunker.Key(), "", 0)
result.validation = &summary
return result, nil
}
if incomplete != nil && input.pipeline.ChunkValidationPolicy.ValidatorFailure == ValidatorFailureFailRun {
failure := validatorFailureError(*incomplete)
result.setValidation(report.Warnings(), nil, failure)
failedTerminal := producerAttemptTerminal{Action: producerTerminalFailed, Validation: report, ValidationIncomplete: true}
summary := validationSummary(failedTerminal, StageChunk, "", "", chunker.Key(), "", 0)
result.validation = &summary
return result, failure
}
// A cache hit is not model material. Its rejection is discarded and
// generation begins with the ordinary initial request below. Warnings
// from this discarded candidate are intentionally not promoted.
}
result.lookup = ChunkPlanDecision{Status: ChunkPlanInvalid, Reason: chunkPlanLookupReason(ChunkPlanInvalid)}
result.summary.LookupStatus = "invalid"
@@ -83,37 +113,41 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
result.summary.PublicationStatus = "not_published"
}
var producerWarnings []contracts.Warning
retryResult, err := runWithRetry(ctx, input.pipeline.Chunk.Retries, func(attempt int) (retryAttemptResult, error) {
terminal, err := runProducerAttempts(ctx, producerAttemptConfig{
Retries: input.pipeline.Chunk.Retries,
Policy: input.pipeline.ChunkValidationPolicy,
AllowStructuralRetry: input.pipeline.ChunkExecutionClass == contracts.ExecutionClassLLMBacked,
}, func(attemptCtx context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
attempt := request.Number
attemptStarted := time.Now().UTC()
attemptPath := path.Join("chunk", fmt.Sprintf("attempt-%02d", attempt))
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "chunk", llmScope, debugTimedEnvelope{Stage: string(StageChunk), ModuleKey: chunker.Key(), Attempt: attempt, StartedAt: attemptStarted})
attemptCtx, llmScope := withDebugLLMScope(attemptCtx, attemptPath)
attemptTerminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "chunk", llmScope, debugTimedEnvelope{Stage: string(StageChunk), ModuleKey: chunker.Key(), Attempt: attempt, AttemptKind: string(request.Kind), StartedAt: attemptStarted})
requestMetadata, metadataErr := cloneMetadata(input.Metadata)
if metadataErr != nil {
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("clone chunk request metadata: %w", metadataErr))
return producerAttemptOutput{}, attemptTerminal.record(nil, fmt.Errorf("clone chunk request metadata: %w", metadataErr))
}
chunkResult, callErr := chunker.Plan(attemptCtx, contracts.ChunkRequest{
Source: doc, SourceInput: sourceInput.Clone(), SessionID: sessionID,
References: CloneReferenceSet(input.pipeline.ChunkReferences.ReferenceSet),
LLMProfile: input.pipeline.Chunk.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(input.pipeline.Chunk.StructuredOutputRepairAttempts), Metadata: requestMetadata,
LLMProfile: input.pipeline.Chunk.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(input.pipeline.Chunk.StructuredOutputRepairAttempts), Correction: request.Correction, Metadata: requestMetadata,
})
if callErr != nil {
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("chunk source with chunker %q: %w", chunker.Key(), callErr))
return producerAttemptOutput{}, attemptTerminal.record(nil, fmt.Errorf("chunk source with chunker %q: %w", chunker.Key(), callErr))
}
plan, chunks, validationErr := validateAndMaterializeChunkPlan(doc, chunkResult.Plan)
if validationErr != nil {
attemptErr := fmt.Errorf("validate chunk plan from chunker %q: %w", chunker.Key(), validationErr)
payload := map[string]any{"plan": debugChunkPlanEnvelope(chunkResult.Plan), "warnings": debugWarningEnvelopes(chunkResult.Warnings)}
return retryAttemptResult{}, terminal.record(payload, attemptErr)
return producerAttemptOutput{}, attemptTerminal.record(payload, fmt.Errorf("%w: %v", contracts.ErrInvalidStructuredOutput, attemptErr))
}
planDigest, digestErr := source.DigestChunkPlan(plan)
if digestErr != nil {
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("digest generated chunk plan: %w", digestErr))
return producerAttemptOutput{}, attemptTerminal.record(nil, fmt.Errorf("digest generated chunk plan: %w", digestErr))
}
producerMetadata, _, metadataErr := moduleManifestMetadata(chunker)
if metadataErr != nil {
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
return producerAttemptOutput{}, attemptTerminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
}
profile := ""
if input.pipeline.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
@@ -129,62 +163,78 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
},
Warnings: cloneWarnings(chunkResult.Warnings), CreatedAt: time.Now().UTC(),
}
action := "generated"
if mode == ChunkCacheRefresh {
action = "refreshed"
return producerAttemptOutput{Value: generatedChunkPlanCandidate{plan: plan, chunks: chunks, record: candidate, producerWarnings: cloneWarnings(chunkResult.Warnings), terminal: &attemptTerminal}, Candidate: chunkResult.ModelCandidate, Warnings: cloneWarnings(chunkResult.Warnings)}, nil
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
candidate, ok := output.Value.(generatedChunkPlanCandidate)
if !ok {
return validationReport{}, fmt.Errorf("chunk attempt candidate has incompatible type")
}
if mode == ChunkCacheBypass {
action = "bypassed"
}
if candidateErr := result.setCandidate(candidate, action); candidateErr != nil {
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("clone generated chunk plan record: %w", candidateErr))
}
validationWarnings, rejected, validationErr := r.validateChunks(attemptCtx, doc, chunker.Key(), chunks, sourceInput, sessionID, input.pipeline.ChunkReferences.ReferenceSet, input.Metadata, input.Prepared.chunkValidators, attempt, input.Debug)
attemptWarnings := append(cloneWarnings(chunkResult.Warnings), validationWarnings...)
report, validationErr := r.validateChunkReport(validationCtx, doc, chunker.Key(), candidate.chunks, sourceInput, sessionID, input.pipeline.ChunkReferences.ReferenceSet, input.Metadata, input.Prepared.chunkValidators, candidate.terminal.envelope.Attempt, input.Debug)
attemptWarnings := append(cloneWarnings(output.Warnings), report.Warnings()...)
payload := map[string]any{
"plan": debugChunkPlanEnvelope(plan), "materialized_chunks": debugSourceChunkEnvelopes(chunks),
"warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected),
"plan": debugChunkPlanEnvelope(candidate.plan), "materialized_chunks": debugSourceChunkEnvelopes(candidate.chunks),
"warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(chunkRejection(report, candidate.terminal.envelope.Attempt, chunker.Key())),
}
if validationErr != nil {
result.setValidation(validationWarnings, rejected, validationErr)
return retryAttemptResult{}, terminal.record(payload, validationErr)
return report, candidate.terminal.record(payload, validationErr)
}
if rejected != nil {
result.setValidation(validationWarnings, rejected, nil)
if debugErr := terminal.record(payload, nil); debugErr != nil {
return retryAttemptResult{}, debugErr
}
return retryAttemptResult{rejection: rejected, warnings: attemptWarnings}, nil
}
result.chunks = chunks
result.plan = &plan
result.warnings = attemptWarnings
producerWarnings = cloneWarnings(chunkResult.Warnings)
result.setValidation(validationWarnings, nil, nil)
if debugErr := terminal.record(payload, nil); debugErr != nil {
return retryAttemptResult{}, debugErr
}
return retryAttemptResult{accepted: true}, nil
return report, candidate.terminal.record(payload, nil)
})
terminalSummary := validationSummary(terminal, StageChunk, "", "", chunker.Key(), "", 0)
if debugErr := writeProducerTerminalDebug(input.Debug, "chunk/terminal.json", terminal, input.pipeline.ChunkValidationPolicy, terminalSummary); debugErr != nil {
return result, debugErr
}
if err != nil {
if result.summary.ValidationStatus == "not_run" {
result.summary.ValidationStatus = "error"
}
return result, err
}
result.accepted = retryResult.accepted
result.rejection = retryResult.rejection
if !retryResult.accepted {
result.warnings = cloneWarnings(retryResult.warnings)
if terminal.Action == producerTerminalRejected {
result.rejection = terminal.Rejection
if result.rejection != nil {
result.rejection.Stage = string(StageChunk)
result.rejection.ModuleKey = chunker.Key()
}
result.warnings = cloneWarnings(terminal.Warnings)
result.validation = &terminalSummary
if result.rejection != nil {
result.rejection.Validation = cloneValidationSummaryPtr(result.validation)
}
result.setValidation(terminal.Validation.Warnings(), result.rejection, nil)
return result, nil
}
candidate, ok := terminal.Value.(generatedChunkPlanCandidate)
if !ok {
return result, fmt.Errorf("chunk attempt terminal has incompatible value")
}
if err := result.setCandidate(candidate.record, "generated"); err != nil {
return result, fmt.Errorf("clone generated chunk plan record: %w", err)
}
if mode == ChunkCacheRefresh {
result.action = "refreshed"
result.summary.Action = "refreshed"
}
if mode == ChunkCacheBypass {
result.action = "bypassed"
result.summary.Action = "bypassed"
}
result.accepted = true
result.plan = &candidate.plan
result.chunks = candidate.chunks
result.warnings = cloneWarnings(terminal.Warnings)
result.validation = &terminalSummary
result.setValidation(terminal.Validation.Warnings(), nil, nil)
if terminal.ValidationIncomplete {
result.summary.ValidationStatus = "incomplete"
}
if mode == ChunkCacheAuto || mode == ChunkCacheRefresh {
if (mode == ChunkCacheAuto || mode == ChunkCacheRefresh) && !terminal.ValidationIncomplete {
record, cloneErr := cloneChunkPlanRecord(*result.record)
if cloneErr != nil {
return result, fmt.Errorf("clone chunk plan record for publication: %w", cloneErr)
}
record.Warnings = cloneWarnings(producerWarnings)
record.Warnings = cloneWarnings(candidate.producerWarnings)
if err := input.ChunkPlans.Save(record); err != nil {
return result, fmt.Errorf("save chunk plan: %w", err)
}
@@ -193,6 +243,14 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
return result, nil
}
func chunkRejection(report validationReport, attempt int, moduleKey string) *contracts.RejectedOutput {
rejection := report.FirstRejection()
if rejection == nil {
return nil
}
return &contracts.RejectedOutput{Stage: string(StageChunk), ModuleKey: moduleKey, ValidatorName: rejection.validatorName, ReasonCode: rejection.reasonCode, Message: rejection.message, AttemptCount: attempt, DiagnosticArtifactPath: rejection.diagnosticPath}
}
func chunkPlanLookupStatus(status ChunkPlanStatus) string {
switch status {
case ChunkPlanHit:

View File

@@ -84,6 +84,35 @@ type retryingChunker struct {
calls int
}
type correctionAwareChunker struct {
key string
plan source.ChunkPlan
calls int
correction *contracts.SemanticCorrection
}
func (c *correctionAwareChunker) Key() string { return c.key }
func (*correctionAwareChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (c *correctionAwareChunker) Plan(_ context.Context, request contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
c.calls++
if request.Correction != nil {
correction, err := contracts.CloneSemanticCorrection(request.Correction)
if err != nil {
return contracts.ChunkPlanResult{}, err
}
c.correction = correction
}
response := "initial chunk response"
if c.calls > 1 {
response = "corrected chunk response"
}
candidate, err := contracts.NewModelCandidate([]byte(response), contracts.CorrectionProtocolSingleResponseV1)
if err != nil {
return contracts.ChunkPlanResult{}, err
}
return contracts.ChunkPlanResult{Plan: source.CloneChunkPlan(c.plan), ModelCandidate: candidate}, nil
}
func (c *retryingChunker) Key() string { return c.key }
func (*retryingChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (c *retryingChunker) Plan(context.Context, contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
@@ -264,7 +293,7 @@ func TestRunnerOmitsChunkMapForRejectedPlan(t *testing.T) {
prepared.output = encoder
prepared.chunkValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("reject/chunk"), Target: ValidatorTargetChunk},
chunk: &countingChunkValidator{result: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted"}},
chunk: &countingChunkValidator{result: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted", CorrectionGuidance: "return an acceptable chunk plan"}},
}}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
if err != nil {
@@ -275,6 +304,139 @@ func TestRunnerOmitsChunkMapForRejectedPlan(t *testing.T) {
}
}
func TestRunnerCorrectsRejectedGeneratedChunkPlan(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t)
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
prepared.resolved.Chunk.Retries = 1
chunker := &correctionAwareChunker{key: prepared.resolved.Chunk.Module, plan: plan}
prepared.chunker = chunker
validatorCalls := 0
prepared.chunkValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("reject-then-approve"), Target: ValidatorTargetChunk},
chunk: chunkValidationFunc{name: "reject-then-approve", validate: func(context.Context, contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
validatorCalls++
if validatorCalls == 1 {
return contracts.ValidationResult{ReasonCode: "chunk_scope", Message: "scope needs correction", CorrectionGuidance: "correct the chunk scope"}, nil
}
return contracts.ValidationResult{Approved: true}, nil
}},
}}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(output.Rejected) != 0 || chunker.calls != 2 || validatorCalls != 2 {
t.Fatalf("output = %#v chunker calls = %d validator calls = %d", output.Rejected, chunker.calls, validatorCalls)
}
if got := string(chunker.correction.AssistantResponse); got != "initial chunk response" {
t.Fatalf("correction response = %q", got)
}
}
func TestRunnerDoesNotPublishValidationIncompleteChunkPlan(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t)
prepared.resolved.ChunkValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
prepared.chunkValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("unavailable"), Target: ValidatorTargetChunk},
chunk: &countingChunkValidator{err: errors.New("validator unavailable")},
}}
store := &recordingChunkPlanStore{decision: ChunkPlanDecision{Status: ChunkPlanMissing}}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: store})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if output.ChunkPlan == nil || output.ChunkPlan.ValidationStatus != "incomplete" || output.ChunkPlan.PublicationStatus != "not_published" || store.saves != 0 {
t.Fatalf("chunk summary = %#v saves = %d", output.ChunkPlan, store.saves)
}
}
func TestRunnerRegeneratesValidationIncompleteChunkPlanHit(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t)
prepared.resolved.ChunkValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
calls := 0
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan, calls: &calls}
validatorCalls := 0
prepared.chunkValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("cache-then-generated"), Target: ValidatorTargetChunk},
chunk: chunkValidationFunc{name: "cache-then-generated", validate: func(context.Context, contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
validatorCalls++
if validatorCalls == 1 {
return contracts.ValidationResult{}, errors.New("cached candidate could not be validated")
}
return contracts.ValidationResult{Approved: true}, nil
}},
}}
record := chunkPlanRecord(t, prepared, plan)
record.Warnings = []contracts.Warning{{Scope: "stored", ReasonCode: "old", Message: "discarded stored warning"}}
store := &recordingChunkPlanStore{record: record, decision: ChunkPlanDecision{Status: ChunkPlanHit}}
debug := newCapturedDebugRecorder()
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: store, Debug: debug})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if calls != 1 || validatorCalls != 2 || store.saves != 1 {
t.Fatalf("calls = producer %d validator %d saves %d, want 1 2 1", calls, validatorCalls, store.saves)
}
if output.ChunkPlan == nil || output.ChunkPlan.Action != "generated" || output.ChunkPlan.LookupStatus != "invalid" || output.ChunkPlan.ValidationStatus != "approved" {
t.Fatalf("chunk plan summary = %#v", output.ChunkPlan)
}
if len(output.Warnings) != 0 {
t.Fatalf("warnings = %#v, want discarded cache warnings omitted", output.Warnings)
}
assertAttemptEnvelopeSequence(t, debug, "chunk", 1)
}
func TestRunnerKeepsStoredPlanWhenCacheAndGeneratedValidationAreIncomplete(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t)
prepared.resolved.ChunkValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
calls := 0
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan, calls: &calls}
validator := &countingChunkValidator{err: errors.New("validator unavailable")}
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
record := chunkPlanRecord(t, prepared, plan)
record.Warnings = []contracts.Warning{{Scope: "stored", ReasonCode: "old", Message: "discarded stored warning"}}
store := &recordingChunkPlanStore{record: record, decision: ChunkPlanDecision{Status: ChunkPlanHit}}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: store})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if calls != 1 || validator.calls != 2 || store.saves != 0 {
t.Fatalf("calls = producer %d validator %d saves %d, want 1 2 0", calls, validator.calls, store.saves)
}
if output.ChunkPlan == nil || output.ChunkPlan.Action != "generated" || output.ChunkPlan.LookupStatus != "invalid" || output.ChunkPlan.ValidationStatus != "incomplete" {
t.Fatalf("chunk plan summary = %#v", output.ChunkPlan)
}
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != "validator_execution_incomplete" {
t.Fatalf("warnings = %#v, want only generated incomplete warning", output.Warnings)
}
if !reflect.DeepEqual(store.record, record) {
t.Fatal("discarded incomplete candidates mutated the stored record")
}
}
func TestRunnerFailsOnValidationIncompleteChunkPlanHitUnderFailRun(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t)
calls := 0
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan, calls: &calls}
validator := &countingChunkValidator{err: errors.New("validator unavailable")}
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
store := &recordingChunkPlanStore{record: chunkPlanRecord(t, prepared, plan), decision: ChunkPlanDecision{Status: ChunkPlanHit}}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: store})
if err == nil || !strings.Contains(err.Error(), "validator unavailable") {
t.Fatalf("Run() error = %v, want cached validation failure", err)
}
if calls != 0 || validator.calls != 1 || store.saves != 0 {
t.Fatalf("calls = producer %d validator %d saves %d, want 0 1 0", calls, validator.calls, store.saves)
}
if output.ChunkPlan == nil || output.ChunkPlan.ValidationStatus != "error" {
t.Fatalf("chunk plan summary = %#v", output.ChunkPlan)
}
}
func TestRunnerRetainsChunkMapAfterLaneRejection(t *testing.T) {
prepared, _ := preparedTerminalDebugPipeline(t)
encoder := &capturingChunkMapOutput{}
@@ -282,7 +444,7 @@ func TestRunnerRetainsChunkMapAfterLaneRejection(t *testing.T) {
prepared.Steps[0].lanes[0].mergeValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("reject/lane"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted"}, nil
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted", CorrectionGuidance: "return an acceptable candidate"}, nil
},
}}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
@@ -316,14 +478,14 @@ func TestRunnerChunkMapRequestDoesNotAliasStoredPlan(t *testing.T) {
func TestRunnerChunkPlanManifestRetainsCandidateOnRejection(t *testing.T) {
prepared, _ := preparedTerminalDebugPipeline(t)
validator := &countingChunkValidator{result: contracts.ValidationResult{Approved: false, ReasonCode: "policy", Message: "no"}}
validator := &countingChunkValidator{result: contracts.ValidationResult{Approved: false, ReasonCode: "policy", Message: "no", CorrectionGuidance: "return a policy-compliant chunk plan"}}
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: &recordingChunkPlanStore{}})
if err != nil {
t.Fatal(err)
}
if output.Manifest.ChunkPlan.Action != "refreshed" || output.Manifest.ChunkPlan.PlanDigest == "" || output.ChunkPlan.ValidationStatus != "rejected" || output.ChunkPlan.PublicationStatus != "not_published" {
t.Fatalf("manifest = %#v summary = %#v", output.Manifest.ChunkPlan, output.ChunkPlan)
if output.Manifest.ChunkPlan.Action != "" || output.Manifest.ChunkPlan.PlanDigest != "" || output.ChunkPlan.ValidationStatus != "rejected" || output.ChunkPlan.PublicationStatus != "not_published" {
t.Fatalf("manifest = %#v summary = %#v; rejected output must not retain a usable chunk candidate", output.Manifest.ChunkPlan, output.ChunkPlan)
}
}
@@ -432,7 +594,7 @@ func TestRunnerAutoHitValidatesOnceWithoutRegenerationOrMutation(t *testing.T) {
wantReject bool
}{
{name: "warning", result: contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{Scope: "current", ReasonCode: "observed", Message: "current warning"}}}},
{name: "rejection", result: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected hit"}, wantReject: true},
{name: "rejection", result: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected hit", CorrectionGuidance: "return an acceptable chunk plan"}, wantReject: true},
{name: "error", validatorErr: errors.New("validator failed"), wantError: "validator failed"},
}
for _, tc := range tests {
@@ -455,20 +617,32 @@ func TestRunnerAutoHitValidatesOnceWithoutRegenerationOrMutation(t *testing.T) {
} else if err != nil {
t.Fatalf("Run() error = %v", err)
}
if validator.calls != 1 || calls != 0 || llmCalls != 0 || store.loads != 1 || store.saves != 0 {
wantProducerCalls := 0
if tc.wantReject {
wantProducerCalls = 1
}
if validator.calls != 1+wantProducerCalls || calls != wantProducerCalls || llmCalls != wantProducerCalls || store.loads != 1 || store.saves != 0 {
t.Fatalf("calls = validator %d module %d llm %d load %d save %d", validator.calls, calls, llmCalls, store.loads, store.saves)
}
assertAttemptEnvelopeSequence(t, debug, "chunk")
if tc.wantReject {
assertAttemptEnvelopeSequence(t, debug, "chunk", 1)
} else {
assertAttemptEnvelopeSequence(t, debug, "chunk")
}
if tc.wantError == "" {
encoded := string(debug.json["chunk/output.json"])
if !strings.Contains(encoded, `"status":"hit"`) || !strings.Contains(encoded, `"plan":`) || !strings.Contains(encoded, `"materialized_chunks":`) {
if tc.wantReject {
if !strings.Contains(encoded, `"status":"invalid"`) || strings.Contains(encoded, `"plan":`) {
t.Fatalf("rejected cache debug = %s", encoded)
}
} else if !strings.Contains(encoded, `"status":"hit"`) || !strings.Contains(encoded, `"plan":`) || !strings.Contains(encoded, `"materialized_chunks":`) {
t.Fatalf("chunk hit debug = %s", encoded)
}
}
if tc.wantReject && (len(output.Rejected) != 1 || output.Rejected[0].ReasonCode != "rejected") {
t.Fatalf("rejected = %#v", output.Rejected)
}
if tc.wantError == "" && len(output.Warnings) != 1+len(tc.result.Warnings) {
if tc.wantError == "" && !tc.wantReject && len(output.Warnings) != 1+len(tc.result.Warnings) {
t.Fatalf("warnings = %#v, want stored warning once plus current warnings", output.Warnings)
}
})
@@ -485,7 +659,7 @@ func TestRunnerPublishesOnlyAcceptedGeneratedPlans(t *testing.T) {
wantReject bool
}{
{name: "module error", moduleErr: errors.New("generation failed"), wantCalls: 2},
{name: "validator rejection", validator: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected generated plan"}, wantCalls: 2, wantReject: true},
{name: "validator rejection", validator: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected generated plan", CorrectionGuidance: "return an acceptable chunk plan"}, wantCalls: 1, wantReject: true},
{name: "cancellation", cancel: true},
} {
t.Run(tc.name, func(t *testing.T) {

View File

@@ -558,6 +558,7 @@ func TestRunnerSelectsMergeErrorBeforeEarlierLaneNormalizeError(t *testing.T) {
func TestRunnerKeepsSuccessfulChunksWhenAnotherExtractIsRejected(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 2)
prepared.Steps[0].lanes[0].resolved.ExtractValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
for laneIndex := range prepared.Steps[0].lanes {
lane := laneIndex
installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
@@ -567,7 +568,7 @@ func TestRunnerKeepsSuccessfulChunksWhenAnotherExtractIsRejected(t *testing.T) {
validator := &prepared.Steps[0].lanes[0].extractValidators.validators[0]
validator.typedValidate = func(_ context.Context, _ any, target typedValidationTarget) (contracts.ValidationResult, error) {
if target.chunk != nil && target.chunk.Index == 0 {
return contracts.ValidationResult{Approved: false, ReasonCode: "expected_rejection", Message: "rejected by test"}, nil
return contracts.ValidationResult{Approved: false, ReasonCode: "expected_rejection", Message: "rejected by test", CorrectionGuidance: "return an acceptable candidate"}, nil
}
return contracts.ValidationResult{Approved: true}, nil
}

View File

@@ -10,33 +10,40 @@ import (
"sync"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/fileio"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type laneExtractState struct {
index int
prepared preparedLaneExecutor
deps []CheckpointFingerprint
decision CheckpointDecision
values []erasedExtractArtifact
serialized []CheckpointArtifact
warnings []contracts.Warning
rejected []contracts.RejectedOutput
results map[int]extractJobResult
remaining int
failed bool
terminal bool
output RunOutput
index int
prepared preparedLaneExecutor
deps []CheckpointFingerprint
decision CheckpointDecision
reuseEligible bool
values []erasedExtractArtifact
serialized []CheckpointArtifact
warnings []contracts.Warning
rejected []contracts.RejectedOutput
incomplete []int
validationSummaries []artifacts.ValidationSummary
results map[int]extractJobResult
remaining int
failed bool
terminal bool
output RunOutput
}
type finalizedExtractResults struct {
accepted []erasedExtractArtifact
serialized []CheckpointArtifact
warnings []contracts.Warning
rejected []contracts.RejectedOutput
decision CheckpointDecision
accepted []erasedExtractArtifact
serialized []CheckpointArtifact
warnings []contracts.Warning
rejected []contracts.RejectedOutput
incomplete []int
validationSummaries []artifacts.ValidationSummary
decision CheckpointDecision
reuseEligible bool
}
func loadExtract(loader CheckpointLoader, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
@@ -56,13 +63,21 @@ type extractJob struct {
}
type extractJobResult struct {
laneIndex int
chunkIndex int
value erasedExtractArtifact
laneIndex int
chunkIndex int
value erasedExtractArtifact
serialized CheckpointArtifact
warnings []contracts.Warning
rejected *contracts.RejectedOutput
validationIncomplete bool
validationSummary *artifacts.ValidationSummary
err error
}
type extractAttemptValue struct {
artifact erasedExtractArtifact
serialized CheckpointArtifact
warnings []contracts.Warning
rejected *contracts.RejectedOutput
err error
terminal *attemptTerminalRecorder
}
type laneCompletion struct {
@@ -110,6 +125,9 @@ func initializeLaneStates(input RunInput, step PreparedPipelineStep, checkpoints
return states, err
}
if input.CheckpointPolicy.requiresReusable(input.stepID, prepared.resolved.ID) && !input.CheckpointPolicy.forced(input.stepID, prepared.resolved.ID) {
if !laneReferencesReuseEligible(input, prepared.resolved) {
return states, fmt.Errorf("required reusable checkpoint unavailable for step %q lane %q: generated input lineage is validation-incomplete", input.stepID, prepared.resolved.ID)
}
state, err := hydrateRequiredLane(input, loader, doc, i, prepared)
states[i] = state
if err != nil {
@@ -121,7 +139,7 @@ func initializeLaneStates(input RunInput, step PreparedPipelineStep, checkpoints
if err != nil {
return states, err
}
if !state.decision.Reused {
if !state.decision.Reused && state.reuseEligible {
if err := checkpointExtractRunning(checkpoints, input.stepID, prepared.resolved.ID, prepared.resolved.Extract.Module, state.deps); err != nil {
return states, fmt.Errorf("write extract checkpoint for lane %q: %w", prepared.resolved.ID, err)
}
@@ -290,7 +308,9 @@ func (e *laneEngine) handleExtractResult(result extractJobResult) {
if result.err != nil {
state.failed = true
e.runErrors = append(e.runErrors, orderedRunError{stage: 0, lane: state.index, chunk: result.chunkIndex, err: result.err})
_ = checkpointExtractFailed(e.checkpoints, e.input.stepID, state.prepared.resolved.ID, state.prepared.resolved.Extract.Module, state.deps, result.err)
if state.reuseEligible {
_ = checkpointExtractFailed(e.checkpoints, e.input.stepID, state.prepared.resolved.ID, state.prepared.resolved.Extract.Module, state.deps, result.err)
}
e.cancel()
} else {
state.results[result.chunkIndex] = result
@@ -328,7 +348,7 @@ func hydrateRequiredLane(input RunInput, loader CheckpointLoader, doc *source.So
}
resolution, err := resolveCheckpointDecision(&local, loader, input.CheckpointPolicy, StageNormalize, input.stepID, lane.ID, lane.Normalize.Module, decision, typed.codec, []CheckpointArtifact{checkpoint.Output})
decision = resolution.decision
state := &laneExtractState{index: index, prepared: prepared, decision: decision, terminal: true, output: local}
state := &laneExtractState{index: index, prepared: prepared, decision: decision, reuseEligible: true, terminal: true, output: local}
if err != nil {
return state, err
}
@@ -341,6 +361,7 @@ func hydrateRequiredLane(input RunInput, loader CheckpointLoader, doc *source.So
SourceID: doc.ID,
Artifact: contracts.CloneSerializedArtifact(hydrated.Artifact),
})
local.normalizeReuseEligibility = map[generatedOutputKey]bool{generatedOutputKeyFor(input.stepID, lane.ID): true}
state.output = local
return state, nil
}
@@ -374,7 +395,12 @@ func prepareLaneExtract(input RunInput, loader CheckpointLoader, doc *source.Sou
}
extractReferences := operationReferenceSet(input, lane.ExtractReferences)
deps := append(digestFingerprints("chunks", digest), generatedReferenceDependencies(extractReferences)...)
state := &laneExtractState{index: index, prepared: prepared, deps: normalizeCheckpointFingerprints(deps), remaining: len(chunks), results: make(map[int]extractJobResult, len(chunks))}
state := &laneExtractState{index: index, prepared: prepared, deps: normalizeCheckpointFingerprints(deps), reuseEligible: referenceTargetReuseEligible(input, lane.ExtractReferences), remaining: len(chunks), results: make(map[int]extractJobResult, len(chunks))}
if !state.reuseEligible {
state.decision = checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonValidationIncompleteLineage)
recordCheckpointEvent(output, loader, string(StageExtract), input.stepID, lane.ID, lane.Extract.Module, state.decision)
return state, nil
}
cp, decision := loadExtract(loader, input.stepID, lane.ID, lane.Extract.Module, state.deps)
resolution, err := resolveCheckpointDecision(output, loader, input.CheckpointPolicy, StageExtract, input.stepID, lane.ID, lane.Extract.Module, decision, typed.codec, cp.Outputs)
if err != nil {
@@ -407,23 +433,21 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
result.err = fmt.Errorf("clone chunk %q for extraction: %w", job.chunk.ID, cloneErr)
return result
}
var accepted erasedExtractArtifact
var serialized CheckpointArtifact
var acceptedWarnings []contracts.Warning
retryResult, err := runWithRetry(ctx, lane.Extract.Retries, func(attempt int) (retryAttemptResult, error) {
terminalResult, err := runProducerAttempts(ctx, producerAttemptConfig{Retries: lane.Extract.Retries, Policy: lane.ExtractValidationPolicy, AllowStructuralRetry: lane.ExtractExecutionClass == contracts.ExecutionClassLLMBacked}, func(attemptCtx context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
attempt := request.Number
started := time.Now().UTC()
attemptPath := path.Join("extract", fileio.EncodePathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), fmt.Sprintf("attempt-%02d", attempt))
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "extract", llmScope, debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started})
attemptCtx, llmScope := withDebugLLMScope(attemptCtx, attemptPath)
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "extract", llmScope, debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, AttemptKind: string(request.Kind), StartedAt: started})
requestMetadata, metadataErr := cloneMetadata(input.Metadata)
if metadataErr != nil {
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("clone extract request metadata: %w", metadataErr))
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("clone extract request metadata: %w", metadataErr))
}
extractReferences := operationReferenceSet(input, lane.ExtractReferences)
extracted, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(extractReferences), LLMProfile: lane.Extract.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(lane.Extract.StructuredOutputRepairAttempts), Metadata: requestMetadata})
extracted, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(extractReferences), LLMProfile: lane.Extract.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(lane.Extract.StructuredOutputRepairAttempts), Correction: request.Correction, Metadata: requestMetadata})
if callErr != nil {
attemptErr := fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, lane.Extract.Module, callErr)
return retryAttemptResult{}, terminal.record(nil, attemptErr)
return producerAttemptOutput{}, terminal.record(nil, attemptErr)
}
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: chunk.ID, ChunkIndex: chunk.Index, ChunkRef: chunk.Ref, Value: extracted.Value}
attemptWarnings := cloneWarnings(extracted.Warnings)
@@ -431,41 +455,65 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
if encodeErr != nil {
attemptErr := fmt.Errorf("serialize extract candidate for lane %q chunk %q: %w", lane.ID, chunk.ID, encodeErr)
payload := map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}
return retryAttemptResult{}, terminal.record(payload, attemptErr)
return producerAttemptOutput{}, terminal.record(payload, attemptErr)
}
serializedCandidate.ChunkID, serializedCandidate.ChunkIndex, serializedCandidate.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageExtract, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: extractReferences, metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: extracted.Value, candidate: &serializedCandidate}, state.prepared.extractValidators, attempt, input.Debug)
attemptWarnings = append(attemptWarnings, warnings...)
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
if validateErr != nil {
return retryAttemptResult{}, terminal.record(payload, validateErr)
return producerAttemptOutput{Value: extractAttemptValue{artifact: artifact, serialized: serializedCandidate, terminal: &terminal}, Candidate: extracted.ModelCandidate, Warnings: attemptWarnings}, nil
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
candidate, ok := output.Value.(extractAttemptValue)
if !ok {
return validationReport{}, fmt.Errorf("extract attempt has incompatible value")
}
if rejected != nil {
if debugErr := terminal.record(payload, nil); debugErr != nil {
return retryAttemptResult{}, debugErr
}
return retryAttemptResult{rejection: rejected, warnings: attemptWarnings}, nil
report, validationErr := r.validateTypedReport(validationCtx, typed.codec, typedValidationTarget{stage: StageExtract, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: operationReferenceSet(input, lane.ExtractReferences), metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: candidate.artifact.Value, candidate: &candidate.serialized}, state.prepared.extractValidators, candidate.terminal.envelope.Attempt, input.Debug)
payload := map[string]any{
"output": debugCheckpointArtifact(candidate.serialized),
"warnings": debugWarningEnvelopes(append(cloneWarnings(output.Warnings), report.Warnings()...)),
"rejection": debugRejectedOutputPtr(typedRejection(report, typedValidationTarget{stage: StageExtract, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Extract.Module, chunk: &chunk}, candidate.terminal.envelope.Attempt)),
}
stored, encodeErr := checkpointArtifact(typed.codec, artifact.LaneID, artifact.ExtractorKey, artifact.SourceID, artifact.Value)
if encodeErr != nil {
attemptErr := fmt.Errorf("serialize accepted extract output for lane %q chunk %q: %w", lane.ID, chunk.ID, encodeErr)
return retryAttemptResult{}, terminal.record(payload, attemptErr)
if validationErr != nil {
return report, candidate.terminal.record(payload, validationErr)
}
stored.ChunkID, stored.ChunkIndex, stored.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
accepted, serialized = artifact, stored
acceptedWarnings = attemptWarnings
if debugErr := terminal.record(payload, nil); debugErr != nil {
return retryAttemptResult{}, debugErr
}
return retryAttemptResult{accepted: true}, nil
return report, candidate.terminal.record(payload, nil)
})
result.err = err
if err == nil && !retryResult.accepted {
result.rejected = retryResult.rejection
result.warnings = cloneWarnings(retryResult.warnings)
summary := validationSummary(terminalResult, StageExtract, input.stepID, lane.ID, lane.Extract.Module, chunk.ID, chunk.Index)
if debugErr := writeProducerTerminalDebug(input.Debug, path.Join("extract", fileio.EncodePathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), "terminal.json"), terminalResult, lane.ExtractValidationPolicy, summary); debugErr != nil {
result.err = errors.Join(result.err, debugErr)
return result
}
result.value, result.serialized, result.warnings = accepted, serialized, acceptedWarnings
if err == nil && terminalResult.Action == producerTerminalRejected {
result.rejected = terminalResult.Rejection
if result.rejected != nil {
result.rejected.Stage, result.rejected.StepID, result.rejected.LaneID, result.rejected.ModuleKey, result.rejected.ChunkID, result.rejected.ChunkIndex = string(StageExtract), input.stepID, lane.ID, lane.Extract.Module, chunk.ID, chunk.Index
result.validationSummary = &summary
result.rejected.Validation = cloneValidationSummaryPtr(result.validationSummary)
}
result.warnings = cloneWarnings(terminalResult.Warnings)
return result
}
if err == nil {
candidate, ok := terminalResult.Value.(extractAttemptValue)
if !ok {
result.err = fmt.Errorf("extract attempt terminal has incompatible value")
return result
}
stored, encodeErr := checkpointArtifact(typed.codec, candidate.artifact.LaneID, candidate.artifact.ExtractorKey, candidate.artifact.SourceID, candidate.artifact.Value)
payload := map[string]any{"output": debugCheckpointArtifact(candidate.serialized), "warnings": debugWarningEnvelopes(terminalResult.Warnings), "rejection": debugRejectedOutputPtr(nil)}
if encodeErr != nil {
attemptErr := fmt.Errorf("serialize accepted extract output for lane %q chunk %q: %w", lane.ID, chunk.ID, encodeErr)
result.err = candidate.terminal.record(payload, attemptErr)
return result
}
stored.ChunkID, stored.ChunkIndex, stored.ChunkRef = candidate.artifact.ChunkID, candidate.artifact.ChunkIndex, candidate.artifact.ChunkRef
if debugErr := candidate.terminal.record(payload, nil); debugErr != nil {
result.err = debugErr
return result
}
result.value, result.serialized = candidate.artifact, stored
result.warnings = cloneWarnings(terminalResult.Warnings)
result.validationIncomplete = terminalResult.ValidationIncomplete
result.validationSummary = &summary
}
return result
}
@@ -478,6 +526,9 @@ func finalizeLaneExtract(checkpoints CheckpointRecorder, stepID string, state *l
sort.Ints(indexes)
for _, index := range indexes {
result := state.results[index]
if result.validationSummary != nil {
state.validationSummaries = append(state.validationSummaries, artifacts.CloneValidationSummary(*result.validationSummary))
}
if result.rejected != nil {
state.rejected = append(state.rejected, *result.rejected)
state.warnings = append(state.warnings, result.warnings...)
@@ -486,11 +537,18 @@ func finalizeLaneExtract(checkpoints CheckpointRecorder, stepID string, state *l
state.values = append(state.values, result.value)
state.serialized = append(state.serialized, result.serialized)
state.warnings = append(state.warnings, result.warnings...)
if result.validationIncomplete {
state.incomplete = append(state.incomplete, result.chunkIndex)
}
}
sort.SliceStable(state.values, func(i, j int) bool { return state.values[i].ChunkIndex < state.values[j].ChunkIndex })
sort.SliceStable(state.serialized, func(i, j int) bool { return state.serialized[i].ChunkIndex < state.serialized[j].ChunkIndex })
sort.SliceStable(state.rejected, func(i, j int) bool { return state.rejected[i].ChunkIndex < state.rejected[j].ChunkIndex })
if !state.decision.Reused {
sort.Ints(state.incomplete)
if len(state.incomplete) > 0 {
state.reuseEligible = false
}
if !state.decision.Reused && state.reuseEligible {
if err := recordExtract(checkpoints, stepID, lane.ID, lane.Extract.Module, state.deps, state.serialized, state.rejected, state.warnings); err != nil {
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
}
@@ -502,18 +560,22 @@ func (r *Runner) continueLane(ctx context.Context, input RunInput, checkpoints C
lane := state.prepared.resolved
local := RunOutput{Manifest: manifestFromPipeline(input)}
results := finalizedExtractResults{
accepted: state.values,
serialized: state.serialized,
warnings: state.warnings,
rejected: state.rejected,
decision: state.decision,
accepted: state.values,
serialized: state.serialized,
warnings: state.warnings,
rejected: state.rejected,
incomplete: state.incomplete,
validationSummaries: state.validationSummaries,
decision: state.decision,
reuseEligible: state.reuseEligible,
}
local.Warnings = append(local.Warnings, cloneWarnings(results.warnings)...)
local.Rejected = append(local.Rejected, cloneRejectedOutputs(results.rejected)...)
local.ValidationSummaries = append(local.ValidationSummaries, cloneValidationSummaries(results.validationSummaries)...)
if err := writeDebugTimed(input.Debug, path.Join("extract", fileio.EncodePathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "decision": results.decision, "source": debugSourceDocumentEnvelope(doc), "chunks": debugSourceChunkEnvelopes(chunks), "options": redactSensitiveMap(lane.Extract.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
return local, &laneRunError{stage: StageExtract, err: err}
}
if err := writeDebugTimed(input.Debug, path.Join("extract", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "outputs": debugCheckpointArtifacts(results.serialized), "rejected": debugRejectedOutputEnvelopes(results.rejected), "warnings": debugWarningEnvelopes(results.warnings)}}); err != nil {
if err := writeDebugTimed(input.Debug, path.Join("extract", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "outputs": debugCheckpointArtifacts(results.serialized), "rejected": debugRejectedOutputEnvelopes(results.rejected), "warnings": debugWarningEnvelopes(results.warnings), "validation_incomplete_chunks": append([]int(nil), results.incomplete...)}}); err != nil {
return local, &laneRunError{stage: StageExtract, err: err}
}
if len(results.accepted) == 0 {
@@ -580,6 +642,15 @@ func mergeLaneOutput(dst *RunOutput, src RunOutput) error {
dst.Rejected = append(dst.Rejected, cloneRejectedOutputs(src.Rejected)...)
dst.Warnings = append(dst.Warnings, cloneWarnings(src.Warnings)...)
dst.CheckpointEvents = append(dst.CheckpointEvents, src.CheckpointEvents...)
dst.ValidationSummaries = append(dst.ValidationSummaries, cloneValidationSummaries(src.ValidationSummaries)...)
if len(src.normalizeReuseEligibility) > 0 {
if dst.normalizeReuseEligibility == nil {
dst.normalizeReuseEligibility = make(map[generatedOutputKey]bool, len(src.normalizeReuseEligibility))
}
for key, eligible := range src.normalizeReuseEligibility {
dst.normalizeReuseEligibility[key] = eligible
}
}
for i := range dst.Manifest.ArtifactLanes {
for j := range src.Manifest.ArtifactLanes {
if dst.Manifest.ArtifactLanes[i].ID == src.Manifest.ArtifactLanes[j].ID && dst.Manifest.ArtifactLanes[i].StepID == src.Manifest.ArtifactLanes[j].StepID && src.Manifest.ArtifactLanes[j].Metadata != nil {

View File

@@ -0,0 +1,139 @@
package pipeline
import (
"context"
"encoding/json"
"fmt"
"reflect"
"strings"
"sync"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestRunnerCorrectsExtractCandidatesIndependentlyPerChunk(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 2)
lane := &prepared.Steps[0].lanes[0]
lane.resolved.Extract.Retries = 1
var mu sync.Mutex
corrections := make(map[int]*contracts.SemanticCorrection)
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
chunkIndex := request.Chunk.Index
if request.Correction != nil {
correction, err := contracts.CloneSemanticCorrection(request.Correction)
if err != nil {
return erasedTypedResult{}, err
}
mu.Lock()
corrections[chunkIndex] = correction
mu.Unlock()
candidate, err := contracts.NewModelCandidate([]byte(fmt.Sprintf("corrected-response-%d", chunkIndex)), contracts.CorrectionProtocolSingleResponseV1)
if err != nil {
return erasedTypedResult{}, err
}
return erasedTypedResult{Value: codecNotes{Items: []string{fmt.Sprintf("corrected-%d", chunkIndex)}}, ModelCandidate: candidate}, nil
}
candidate, err := contracts.NewModelCandidate([]byte(fmt.Sprintf("initial-response-%d", chunkIndex)), contracts.CorrectionProtocolSingleResponseV1)
if err != nil {
return erasedTypedResult{}, err
}
return erasedTypedResult{Value: codecNotes{Items: []string{fmt.Sprintf("invalid-%d", chunkIndex)}}, ModelCandidate: candidate}, nil
})
lane.extractValidators.validators[0].typedValidate = func(_ context.Context, _ any, target typedValidationTarget) (contracts.ValidationResult, error) {
value := target.value.(codecNotes)
if value.Items[0] == fmt.Sprintf("invalid-%d", target.chunk.Index) {
return contracts.ValidationResult{Approved: false, ReasonCode: "incorrect_extract", Message: "candidate needs correction", CorrectionGuidance: fmt.Sprintf("correct chunk %d", target.chunk.Index)}, nil
}
return contracts.ValidationResult{Approved: true}, nil
}
debug := newCapturedDebugRecorder()
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2, Debug: debug})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 2 {
t.Fatalf("run output = %#v, want corrected accepted extracts", output)
}
mu.Lock()
defer mu.Unlock()
if len(corrections) != 2 {
t.Fatalf("corrections = %#v, want one correction per chunk", corrections)
}
for index := 0; index < 2; index++ {
correction := corrections[index]
if correction == nil || string(correction.AssistantResponse) != fmt.Sprintf("initial-response-%d", index) || !strings.Contains(correction.UserGuidance, fmt.Sprintf("correct chunk %d", index)) || !strings.Contains(correction.UserGuidance, "complete corrected replacement") || strings.Contains(correction.UserGuidance, "incorrect_extract") || strings.Contains(correction.UserGuidance, "candidate needs correction") {
t.Fatalf("chunk %d correction = %#v, want its exact initial response and semantic replacement guidance only", index, correction)
}
}
terminal := string(debug.json["extract/notes/chunk-000001/terminal.json"])
if !strings.Contains(terminal, `"kind":"semantic_correction"`) || !strings.Contains(terminal, `"producer_attempt_count":2`) {
t.Fatalf("extract terminal debug = %s, want correction provenance", terminal)
}
if strings.Contains(terminal, "initial-response-0") || strings.Contains(terminal, "corrected-response-0") || strings.Contains(terminal, "correct chunk 0") {
t.Fatalf("extract terminal debug leaked correction content: %s", terminal)
}
manifest, err := json.Marshal(output.Manifest)
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(manifest), "initial-response-0") || strings.Contains(string(manifest), "corrected-response-0") || strings.Contains(string(manifest), "correct chunk 0") {
t.Fatalf("manifest leaked correction content: %s", manifest)
}
}
func containsWarnings(have, want []contracts.Warning) bool {
for index := 0; index+len(want) <= len(have); index++ {
if reflect.DeepEqual(have[index:index+len(want)], want) {
return true
}
}
return false
}
func containsValidationSummary(summaries []artifacts.ValidationSummary, status string) bool {
for _, summary := range summaries {
if summary.Status == status {
return true
}
}
return false
}
func TestRunnerContinuesValidationIncompleteExtractWithoutCheckpoint(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
lane := &prepared.Steps[0].lanes[0]
lane.resolved.ExtractValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
})
lane.extractValidators.validators[0].typedValidate = func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{}, fmt.Errorf("validator unavailable")
}
recorder := &extractCaptureRecorder{CheckpointRecorder: NoopCheckpointRecorder()}
debug := newCapturedDebugRecorder()
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoints: recorder, Debug: debug})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if len(output.NormalizeOutputs) != 1 || len(output.Rejected) != 0 {
t.Fatalf("run output = %#v, want accepted incomplete extract", output)
}
if output.Manifest.ValidationStatus != "incomplete" || !containsValidationSummary(output.Manifest.ValidationSummaries, "incomplete") {
t.Fatalf("manifest validation = %#v, want incomplete extract provenance", output.Manifest)
}
if got, want := output.Warnings, []contracts.Warning{{Scope: "typed/check", ReasonCode: "validator_execution_incomplete", Message: "Validator execution did not complete within its configured budget."}}; !containsWarnings(got, want) {
t.Fatalf("warnings = %#v, want %#v", got, want)
}
if len(recorder.checkpoint.Outputs) != 0 || len(recorder.checkpoint.Rejected) != 0 {
t.Fatalf("extract checkpoint = %#v, want no persisted incomplete output", recorder.checkpoint)
}
encoded := string(debug.json["extract/notes/output.json"])
if !strings.Contains(encoded, `"validation_incomplete_chunks":[0]`) {
t.Fatalf("extract output debug = %s, want incomplete chunk marker", encoded)
}
}

View File

@@ -3,6 +3,7 @@ package pipeline
import (
"context"
"encoding/json"
"fmt"
"reflect"
"strings"
"testing"
@@ -221,14 +222,15 @@ func TestRunnerPromotesOnlyAcceptedExtractRetryWarnings(t *testing.T) {
scope = "accepted"
}
return erasedTypedResult{
Value: typedValueForLane(0, request.Chunk.Index),
Warnings: []contracts.Warning{{Scope: scope, ReasonCode: "observed", Message: scope}},
Value: typedValueForLane(0, request.Chunk.Index),
Warnings: []contracts.Warning{{Scope: scope, ReasonCode: "observed", Message: scope}},
ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%s"]}`, scope)),
}, nil
})
validatorCalls := 0
prepared.Steps[0].lanes[0].extractValidators.validators[0].typedValidate = func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
validatorCalls++
return contracts.ValidationResult{Approved: validatorCalls == 2, ReasonCode: "retry", Message: "retry extract"}, nil
return contracts.ValidationResult{Approved: validatorCalls == 2, ReasonCode: "retry", Message: "retry extract", CorrectionGuidance: "return an acceptable extract"}, nil
}
debug := newCapturedDebugRecorder()
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug})

View File

@@ -70,7 +70,7 @@ func TestRunnerHandlesRetryableNormalizeFallbacks(t *testing.T) {
validator: &preparedValidator{
resolved: ResolvedValidator{Binding: Binding("reject-final-fallback"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "reject fallback"}, nil
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "reject fallback", CorrectionGuidance: "return an acceptable normalized artifact"}, nil
},
},
wantCalls: 2,
@@ -85,6 +85,7 @@ func TestRunnerHandlesRetryableNormalizeFallbacks(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
lane := &prepared.Steps[0].lanes[0]
lane.resolved.Normalize.Retries = tc.retries
lane.resolved.NormalizeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
if tc.validator != nil {
lane.normalizeValidators.validators = []preparedValidator{*tc.validator}
}

View File

@@ -50,13 +50,14 @@ func TestRunnerPromotesOnlyTerminalRejectionWarnings(t *testing.T) {
return contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("validator-%d", attempts), ReasonCode: "validator", Message: "validator warning"}}}
}
reject := func() contracts.ValidationResult {
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected"}
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected", CorrectionGuidance: "return an acceptable candidate"}
}
debug := newCapturedDebugRecorder()
recorder := &extractCaptureRecorder{CheckpointRecorder: NoopCheckpointRecorder()}
switch target {
case StageChunk:
prepared.resolved.ChunkValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
chunker := prepared.chunker.(*typedTestChunker)
prepared.chunker = &warningChunker{key: prepared.resolved.Chunk.Module, plan: source.CloneChunkPlan(chunker.plan)}
prepared.resolved.Chunk.Retries = 1
@@ -74,23 +75,26 @@ func TestRunnerPromotesOnlyTerminalRejectionWarnings(t *testing.T) {
}
case StageExtract:
lane.resolved.Extract.Retries = 1
lane.resolved.ExtractValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
installExtractOperation(prepared, 0, func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
attempts++
return erasedTypedResult{Value: codecNotes{Items: []string{"extract"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}}, nil
return erasedTypedResult{Value: codecNotes{Items: []string{"extract"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%d"]}`, attempts))}, nil
})
lane.extractValidators.validators = rejectionWarningTypedValidators(first, reject)
case StageMerge:
lane.resolved.Merge.Retries = 1
lane.resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
lane.typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
attempts++
return erasedTypedResult{Value: codecNotes{Items: []string{"merge"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}}, nil
return erasedTypedResult{Value: codecNotes{Items: []string{"merge"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%d"]}`, attempts))}, nil
}
lane.mergeValidators.validators = rejectionWarningTypedValidators(first, reject)
case StageNormalize:
lane.resolved.Normalize.Retries = 1
lane.resolved.NormalizeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
lane.typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
attempts++
return erasedTypedResult{Value: codecNotes{Items: []string{"normalize"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}}, nil
return erasedTypedResult{Value: codecNotes{Items: []string{"normalize"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%d"]}`, attempts))}, nil
}
lane.normalizeValidators.validators = rejectionWarningTypedValidators(first, reject)
}
@@ -100,11 +104,16 @@ func TestRunnerPromotesOnlyTerminalRejectionWarnings(t *testing.T) {
t.Fatalf("Run() error = %v, want nil", err)
}
wantScopes := []string{"operation-2", "validator-2"}
wantAttempts := 2
if target == StageChunk {
wantScopes = []string{"operation-1", "validator-1"}
wantAttempts = 1
}
if got := rejectionWarningScopes(output.Warnings); !reflect.DeepEqual(got, wantScopes) {
t.Fatalf("published warning scopes = %#v, want %#v", got, wantScopes)
}
if len(output.Rejected) != 1 || output.Rejected[0].AttemptCount != 2 {
t.Fatalf("rejections = %#v, want final rejection after two attempts", output.Rejected)
if len(output.Rejected) != 1 || output.Rejected[0].AttemptCount != wantAttempts {
t.Fatalf("rejections = %#v, want terminal rejection after %d attempt(s)", output.Rejected, wantAttempts)
}
if target == StageExtract && !reflect.DeepEqual(rejectionWarningScopes(recorder.checkpoint.Warnings), wantScopes) {
t.Fatalf("extract checkpoint warnings = %#v, want %#v", recorder.checkpoint.Warnings, wantScopes)

View File

@@ -0,0 +1,238 @@
package pipeline
import (
"context"
"fmt"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type reuseLineageCheckpointSpy struct {
CheckpointLoader
CheckpointRecorder
loads map[string]int
writes map[string]int
forbid map[string]struct{}
}
func newReuseLineageCheckpointSpy() *reuseLineageCheckpointSpy {
return &reuseLineageCheckpointSpy{
CheckpointLoader: NoopCheckpointLoader(),
CheckpointRecorder: NoopCheckpointRecorder(),
loads: make(map[string]int),
writes: make(map[string]int),
forbid: make(map[string]struct{}),
}
}
func (s *reuseLineageCheckpointSpy) Enabled() bool { return true }
func (s *reuseLineageCheckpointSpy) load(stage, laneID string) {
key := stage + "/" + laneID
if _, forbidden := s.forbid[key]; forbidden {
panic("checkpoint lookup crossed validation-incomplete lineage: " + key)
}
s.loads[key]++
}
func (s *reuseLineageCheckpointSpy) write(stage, action, laneID string) {
s.writes[stage+"/"+action+"/"+laneID]++
}
func (s *reuseLineageCheckpointSpy) Extract(laneID, _ string, _ []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
s.load("extract", laneID)
return ExtractCheckpoint{}, NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing)
}
func (s *reuseLineageCheckpointSpy) Merge(laneID, _ string, _ []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
s.load("merge", laneID)
return MergeCheckpoint{}, NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing)
}
func (s *reuseLineageCheckpointSpy) Normalize(laneID, _ string, _ []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
s.load("normalize", laneID)
return NormalizeCheckpoint{}, NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing)
}
func (s *reuseLineageCheckpointSpy) ExtractRunning(laneID, _ string, _ []CheckpointFingerprint) error {
s.write("extract", "running", laneID)
return nil
}
func (s *reuseLineageCheckpointSpy) ExtractSucceeded(laneID, _ string, _ []CheckpointFingerprint, _ []CheckpointArtifact, _ []contracts.RejectedOutput, _ []contracts.Warning) error {
s.write("extract", "succeeded", laneID)
return nil
}
func (s *reuseLineageCheckpointSpy) ExtractFailed(laneID, _ string, _ []CheckpointFingerprint, _ error) error {
s.write("extract", "failed", laneID)
return nil
}
func (s *reuseLineageCheckpointSpy) MergeRunning(laneID, _ string, _ []CheckpointFingerprint) error {
s.write("merge", "running", laneID)
return nil
}
func (s *reuseLineageCheckpointSpy) MergeSucceeded(laneID, _ string, _ []CheckpointFingerprint, _ CheckpointArtifact, _ []contracts.Warning) error {
s.write("merge", "succeeded", laneID)
return nil
}
func (s *reuseLineageCheckpointSpy) MergeRejected(laneID, _ string, _ []CheckpointFingerprint, _ contracts.RejectedOutput) error {
s.write("merge", "rejected", laneID)
return nil
}
func (s *reuseLineageCheckpointSpy) MergeFailed(laneID, _ string, _ []CheckpointFingerprint, _ error) error {
s.write("merge", "failed", laneID)
return nil
}
func (s *reuseLineageCheckpointSpy) NormalizeRunning(laneID, _ string, _ []CheckpointFingerprint) error {
s.write("normalize", "running", laneID)
return nil
}
func (s *reuseLineageCheckpointSpy) NormalizeSucceeded(laneID, _ string, _ []CheckpointFingerprint, _ CheckpointArtifact, _ []contracts.Warning) error {
s.write("normalize", "succeeded", laneID)
return nil
}
func (s *reuseLineageCheckpointSpy) NormalizeRejected(laneID, _ string, _ []CheckpointFingerprint, _ contracts.RejectedOutput) error {
s.write("normalize", "rejected", laneID)
return nil
}
func (s *reuseLineageCheckpointSpy) NormalizeFailed(laneID, _ string, _ []CheckpointFingerprint, _ error) error {
s.write("normalize", "failed", laneID)
return nil
}
func (s *reuseLineageCheckpointSpy) writesFor(stage, laneID string) int {
total := 0
needle := stage + "/"
suffix := "/" + laneID
for key, count := range s.writes {
if strings.HasPrefix(key, needle) && strings.HasSuffix(key, suffix) {
total += count
}
}
return total
}
func unavailableTypedValidator(kind contracts.ArtifactKind) preparedValidator {
return preparedValidator{
resolved: ResolvedValidator{Binding: Binding("typed/check"), Target: ValidatorTargetTyped, ArtifactKind: kind},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{}, fmt.Errorf("validator unavailable")
},
}
}
func TestValidationIncompleteExtractDisablesDownstreamCheckpointIO(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
lane := &prepared.Steps[0].lanes[0]
lane.resolved.ExtractValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
})
lane.extractValidators.validators[0].typedValidate = func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{}, fmt.Errorf("validator unavailable")
}
spy := newReuseLineageCheckpointSpy()
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: spy, Checkpoints: spy})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(output.NormalizeOutputs) != 1 {
t.Fatalf("normalize outputs = %d, want current-run handoff", len(output.NormalizeOutputs))
}
if spy.loads["merge/notes"] != 0 || spy.loads["normalize/notes"] != 0 {
t.Fatalf("downstream loads = %#v, want none", spy.loads)
}
if spy.writesFor("merge", "notes") != 0 || spy.writesFor("normalize", "notes") != 0 {
t.Fatalf("downstream writes = %#v, want none", spy.writes)
}
if spy.writes["extract/succeeded/notes"] != 0 {
t.Fatalf("extract writes = %#v, want no reusable success", spy.writes)
}
}
func TestValidationIncompleteMergeDisablesNormalizeCheckpointIO(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
lane := &prepared.Steps[0].lanes[0]
lane.resolved.MergeValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
})
lane.mergeValidators.validators = []preparedValidator{unavailableTypedValidator(lane.resolved.ArtifactKind)}
spy := newReuseLineageCheckpointSpy()
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: spy, Checkpoints: spy})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(output.NormalizeOutputs) != 1 {
t.Fatalf("normalize outputs = %d, want current-run handoff", len(output.NormalizeOutputs))
}
if spy.loads["merge/notes"] != 1 || spy.loads["normalize/notes"] != 0 {
t.Fatalf("loads = %#v, want merge lookup only", spy.loads)
}
if spy.writes["merge/succeeded/notes"] != 0 || spy.writesFor("normalize", "notes") != 0 {
t.Fatalf("writes = %#v, want no reusable merge or normalize state", spy.writes)
}
}
func TestValidationIncompleteNormalizeIsNotPublished(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
lane := &prepared.Steps[0].lanes[0]
lane.resolved.NormalizeValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
})
lane.normalizeValidators.validators = []preparedValidator{unavailableTypedValidator(lane.resolved.ArtifactKind)}
spy := newReuseLineageCheckpointSpy()
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: spy, Checkpoints: spy})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(output.NormalizeOutputs) != 1 || spy.writes["normalize/succeeded/notes"] != 0 {
t.Fatalf("output/writes = %d / %#v, want in-memory output without normalize publication", len(output.NormalizeOutputs), spy.writes)
}
}
func TestGeneratedReferenceFromIncompleteValidationDisablesDependentCheckpointIO(t *testing.T) {
input, _, _ := handoffFixture(t, codecNotes{Items: []string{"producer"}})
prepared := input.Prepared
producer := &prepared.Steps[0].lanes[0]
consumer := &prepared.Steps[1].lanes[0]
producer.resolved.NormalizeValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
producer.normalizeValidators.validators = []preparedValidator{unavailableTypedValidator(producer.resolved.ArtifactKind)}
referenceItems := 0
consumer.typed.extract = func(_ context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
referenceItems = len(request.References.Slots["producer-output"].Items)
return erasedTypedResult{Value: codecScore{Value: 1}}, nil
}
spy := newReuseLineageCheckpointSpy()
for _, stage := range []string{"extract", "merge", "normalize"} {
spy.forbid[stage+"/score"] = struct{}{}
}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: spy, Checkpoints: spy})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if referenceItems != 1 || len(output.NormalizeOutputs) != 2 {
t.Fatalf("current-run handoff = items %d outputs %d, want one generated item and two outputs", referenceItems, len(output.NormalizeOutputs))
}
for _, stage := range []string{"extract", "merge", "normalize"} {
if spy.writesFor(stage, "score") != 0 {
t.Fatalf("dependent writes = %#v, want none for %s", spy.writes, stage)
}
}
}

View File

@@ -0,0 +1,126 @@
package pipeline
import (
"context"
"fmt"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestRunnerCorrectsRejectedMergeAndNormalizeCandidates(t *testing.T) {
for _, target := range []ModuleStage{StageMerge, StageNormalize} {
t.Run(string(target), func(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
lane := &prepared.Steps[0].lanes[0]
var correction *contracts.SemanticCorrection
validator := preparedValidator{
resolved: ResolvedValidator{Binding: Binding("correction-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(_ context.Context, _ any, target typedValidationTarget) (contracts.ValidationResult, error) {
value := target.value.(codecNotes)
if firstNote(value) == "invalid" {
return contracts.ValidationResult{Approved: false, ReasonCode: "invalid_candidate", Message: "candidate needs correction", CorrectionGuidance: "produce the accepted value"}, nil
}
return contracts.ValidationResult{Approved: true}, nil
},
}
operation := func(observed *contracts.SemanticCorrection) (erasedTypedResult, error) {
if observed != nil {
cloned, err := contracts.CloneSemanticCorrection(observed)
if err != nil {
return erasedTypedResult{}, err
}
correction = cloned
}
item, response := "invalid", `{"items":["invalid"]}`
if observed != nil {
item, response = "accepted", `{"items":["accepted"]}`
}
candidate, err := contracts.NewModelCandidate([]byte(response), contracts.CorrectionProtocolSingleResponseV1)
if err != nil {
return erasedTypedResult{}, err
}
return erasedTypedResult{Value: codecNotes{Items: []string{item}}, ModelCandidate: candidate}, nil
}
switch target {
case StageMerge:
lane.resolved.Merge.Retries = 1
lane.mergeValidators.validators = []preparedValidator{validator}
lane.typed.merge = func(_ context.Context, _ any, request contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
return operation(request.Correction)
}
case StageNormalize:
lane.resolved.Normalize.Retries = 1
lane.normalizeValidators.validators = []preparedValidator{validator}
lane.typed.normalize = func(_ context.Context, _ any, request contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
return operation(request.Correction)
}
}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if correction == nil || string(correction.AssistantResponse) != `{"items":["invalid"]}` || !strings.Contains(correction.UserGuidance, "produce the accepted value") || !strings.Contains(correction.UserGuidance, "complete corrected replacement") || strings.Contains(correction.UserGuidance, "invalid_candidate") || strings.Contains(correction.UserGuidance, "candidate needs correction") {
t.Fatalf("%s correction = %#v, want exact rejected response and semantic replacement guidance only", target, correction)
}
if len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
t.Fatalf("run output = %#v, want corrected accepted output", output)
}
})
}
}
func TestRunnerRejectsDeterministicMergeCandidateWithoutCorrection(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
lane := &prepared.Steps[0].lanes[0]
lane.resolved.Merge.Retries = 1
lane.resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
calls := 0
lane.typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
calls++
return erasedTypedResult{Value: codecNotes{Items: []string{"invalid"}}}, nil
}
lane.mergeValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("reject"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: false, ReasonCode: "invalid_candidate", Message: "not accepted", CorrectionGuidance: "return an acceptable candidate"}, nil
},
}}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
if err != nil {
t.Fatalf("Run() error = %v, want non-fatal rejection", err)
}
if calls != 1 || len(output.Rejected) != 1 || output.Rejected[0].AttemptCount != 1 {
t.Fatalf("merge calls = %d output = %#v, want one deterministic terminal rejection", calls, output)
}
}
func TestRunnerContinuesNormalizeAfterValidatorFailure(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
lane := &prepared.Steps[0].lanes[0]
lane.resolved.NormalizeValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
lane.normalizeValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("unavailable"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{}, fmt.Errorf("validator unavailable")
},
}}
checkpoints := &candidateCheckpointRecorder{CheckpointRecorder: NoopCheckpointRecorder()}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoints: checkpoints})
if err != nil {
t.Fatalf("Run() error = %v, want incomplete accepted output", err)
}
if len(output.NormalizeOutputs) != 1 || len(output.Rejected) != 0 || checkpoints.normalizeSucceeded != 0 {
t.Fatalf("run output = %#v checkpoints = %#v, want non-checkpointed incomplete normalized output", output, checkpoints)
}
if output.Manifest.ValidationStatus != "incomplete" || !containsValidationSummary(output.Manifest.ValidationSummaries, "incomplete") {
t.Fatalf("manifest validation = %#v, want incomplete normalization provenance", output.Manifest)
}
if got, want := output.Warnings, []contracts.Warning{{Scope: "unavailable", ReasonCode: "validator_execution_incomplete", Message: "Validator execution did not complete within its configured budget."}}; !containsWarnings(got, want) {
t.Fatalf("warnings = %#v, want %#v", got, want)
}
}

View File

@@ -37,6 +37,11 @@ type terminalChunkValidator struct {
err error
}
type retryingLLMChunkValidator struct {
requests []contracts.ChunkValidationRequest
calls int
}
type observingChunkValidator struct {
request contracts.ChunkValidationRequest
}
@@ -73,6 +78,24 @@ func (v terminalChunkValidator) Validate(context.Context, contracts.ChunkValidat
return v.result, v.err
}
func (*retryingLLMChunkValidator) Name() string { return "retrying/llm-chunk-validator" }
func (*retryingLLMChunkValidator) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassLLMBacked
}
func (validator *retryingLLMChunkValidator) Validate(_ context.Context, request contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
validator.calls++
validator.requests = append(validator.requests, request)
if validator.calls == 1 && len(request.Chunks) > 0 {
request.Chunks[0].ID = "mutated"
}
if validator.calls == 1 {
return contracts.ValidationResult{}, errors.New("temporary validator failure")
}
return contracts.ValidationResult{Approved: true}, nil
}
func assertAttemptEnvelopeSequence(t *testing.T, debug *capturedDebugRecorder, prefix string, attempts ...int) {
t.Helper()
marker := strings.TrimSuffix(prefix, "/") + "/attempt-"
@@ -100,6 +123,26 @@ func assertAttemptEnvelopeSequence(t *testing.T, debug *capturedDebugRecorder, p
func preparedTerminalDebugPipeline(t *testing.T) (*PreparedPipeline, source.ChunkPlan) {
t.Helper()
prepared := preparedAttemptDebugPipeline(t)
prepared.resolved.ChunkValidationPolicy = ValidationPolicy{
ProducerStructuralFailure: ProducerStructuralFailureFailRun,
SemanticRejection: SemanticRejectionRejectOutput,
ValidatorFailure: ValidatorFailureFailRun,
}
prepared.Steps[0].lanes[0].resolved.ExtractValidationPolicy = ValidationPolicy{
ProducerStructuralFailure: ProducerStructuralFailureFailRun,
SemanticRejection: SemanticRejectionRejectOutput,
ValidatorFailure: ValidatorFailureFailRun,
}
prepared.Steps[0].lanes[0].resolved.MergeValidationPolicy = ValidationPolicy{
ProducerStructuralFailure: ProducerStructuralFailureFailRun,
SemanticRejection: SemanticRejectionRejectOutput,
ValidatorFailure: ValidatorFailureFailRun,
}
prepared.Steps[0].lanes[0].resolved.NormalizeValidationPolicy = ValidationPolicy{
ProducerStructuralFailure: ProducerStructuralFailureFailRun,
SemanticRejection: SemanticRejectionRejectOutput,
ValidatorFailure: ValidatorFailureFailRun,
}
chunker, ok := prepared.chunker.(*typedTestChunker)
if !ok {
t.Fatalf("prepared chunker = %T, want *typedTestChunker", prepared.chunker)
@@ -114,10 +157,11 @@ func TestRunnerRecordsChunkTerminalOutcomes(t *testing.T) {
validator terminalChunkValidator
wantError string
wantRejection bool
attemptError bool
}{
{name: "accepted", validator: terminalChunkValidator{result: contracts.ValidationResult{Approved: true}}},
{name: "module error", moduleError: errors.New("chunk module failed"), wantError: "chunk module failed"},
{name: "validator rejection", validator: terminalChunkValidator{result: contracts.ValidationResult{Approved: false, ReasonCode: "chunk_rejected", Message: "chunk rejected"}}, wantRejection: true},
{name: "module error", moduleError: errors.New("chunk module failed"), wantError: "chunk module failed", attemptError: true},
{name: "validator rejection", validator: terminalChunkValidator{result: contracts.ValidationResult{Approved: false, ReasonCode: "chunk_rejected", Message: "chunk rejected", CorrectionGuidance: "return an acceptable chunk plan"}}, wantRejection: true},
{name: "validator error", validator: terminalChunkValidator{err: errors.New("chunk validator failed")}, wantError: "chunk validator failed"},
}
for _, tc := range tests {
@@ -131,9 +175,15 @@ func TestRunnerRecordsChunkTerminalOutcomes(t *testing.T) {
assertAttemptEnvelopeSequence(t, debug, "chunk", 1)
envelope := debug.envelope(t, "chunk/attempt-01.json")
if tc.wantError != "" {
if err == nil || !strings.Contains(err.Error(), tc.wantError) || !strings.Contains(envelope.Error, tc.wantError) {
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
t.Fatalf("Run() error = %v, attempt error = %q; want %q", err, envelope.Error, tc.wantError)
}
if tc.attemptError && !strings.Contains(envelope.Error, tc.wantError) {
t.Fatalf("attempt error = %q, want %q", envelope.Error, tc.wantError)
}
if !tc.attemptError && envelope.Error != "" {
t.Fatalf("settled validator failure attempt error = %q, want empty", envelope.Error)
}
} else if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
@@ -152,6 +202,28 @@ func TestRunnerRecordsChunkTerminalOutcomes(t *testing.T) {
}
}
func TestRunnerRetriesLLMValidatorsWithoutRegeneratingChunkCandidate(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t)
chunkCalls := 0
prepared.resolved.Chunk.Retries = 0
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan, calls: &chunkCalls}
validator := &retryingLLMChunkValidator{}
prepared.chunkValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: ModuleBinding{Module: validator.Name(), Retries: 1}, ExecutionClass: contracts.ExecutionClassLLMBacked, Target: ValidatorTargetChunk},
chunk: validator,
}}
if _, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")}); err != nil {
t.Fatalf("Run() error = %v", err)
}
if chunkCalls != 1 || validator.calls != 2 {
t.Fatalf("producer calls = %d validator calls = %d, want 1 and 2", chunkCalls, validator.calls)
}
if len(validator.requests) != 2 || len(validator.requests[0].Chunks) == 0 || len(validator.requests[1].Chunks) == 0 || validator.requests[1].Chunks[0].ID == "mutated" {
t.Fatalf("validator requests = %#v, want independently owned immutable candidates", validator.requests)
}
}
func TestRunnerMaterializesAnnotatedPlanBeforeChunkValidation(t *testing.T) {
prepared, plan := preparedTerminalDebugPipeline(t)
plan.Annotations = source.ChunkAnnotations{"same": []byte(`{"plan":1}`)}
@@ -235,16 +307,22 @@ func TestRunnerRecordsExtractTerminalOutcomes(t *testing.T) {
candidateFail bool
finalFail bool
wantError string
attemptError bool
}{
{name: "terminal rejection", reject: true},
{name: "module error", moduleError: errors.New("extract module failed"), wantError: "extract module failed"},
{name: "module error", moduleError: errors.New("extract module failed"), wantError: "extract module failed", attemptError: true},
{name: "validator error", validatorErr: errors.New("extract validator failed"), wantError: "extract validator failed"},
{name: "candidate codec error", candidateFail: true, wantError: "serialize extract candidate"},
{name: "final codec error", finalFail: true, wantError: "serialize accepted extract output"},
{name: "candidate codec error", candidateFail: true, wantError: "serialize extract candidate", attemptError: true},
{name: "final codec error", finalFail: true, wantError: "serialize accepted extract output", attemptError: true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
prepared.Steps[0].lanes[0].resolved.ExtractValidationPolicy = ValidationPolicy{
ProducerStructuralFailure: ProducerStructuralFailureFailRun,
SemanticRejection: SemanticRejectionRejectOutput,
ValidatorFailure: ValidatorFailureFailRun,
}
value := "extract-terminal"
codec := &observedNotesCodec{}
if tc.candidateFail {
@@ -263,7 +341,7 @@ func TestRunnerRecordsExtractTerminalOutcomes(t *testing.T) {
prepared.Steps[0].lanes[0].extractValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("terminal/extract-validator"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: !tc.reject, ReasonCode: "extract_rejected", Message: "extract rejected"}, tc.validatorErr
return contracts.ValidationResult{Approved: !tc.reject, ReasonCode: "extract_rejected", Message: "extract rejected", CorrectionGuidance: "return an acceptable extract"}, tc.validatorErr
},
}}
debug := newCapturedDebugRecorder()
@@ -273,9 +351,15 @@ func TestRunnerRecordsExtractTerminalOutcomes(t *testing.T) {
attemptPath := "extract/notes/chunk-000001/attempt-01.json"
envelope := debug.envelope(t, attemptPath)
if tc.wantError != "" {
if err == nil || !strings.Contains(err.Error(), tc.wantError) || !strings.Contains(envelope.Error, tc.wantError) {
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
t.Fatalf("Run() error = %v, attempt error = %q; want %q", err, envelope.Error, tc.wantError)
}
if tc.attemptError && !strings.Contains(envelope.Error, tc.wantError) {
t.Fatalf("attempt error = %q, want %q", envelope.Error, tc.wantError)
}
if !tc.attemptError && envelope.Error != "" {
t.Fatalf("settled validator failure attempt error = %q, want empty", envelope.Error)
}
} else {
if err != nil {
t.Fatalf("Run() error = %v, want nil rejection", err)

View File

@@ -11,6 +11,7 @@ import (
"reflect"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/fileio"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
@@ -188,9 +189,24 @@ func (e *laneRunError) Error() string { return e.err.Error() }
func (e *laneRunError) Unwrap() error { return e.err }
type mergeStageResult struct {
artifact erasedMergeArtifact
serialized CheckpointArtifact
terminal bool
artifact erasedMergeArtifact
serialized CheckpointArtifact
terminal bool
validationIncomplete bool
reuseEligible bool
}
type mergeAttemptValue struct {
artifact erasedMergeArtifact
candidate CheckpointArtifact
terminal *attemptTerminalRecorder
}
type normalizeAttemptValue struct {
value any
candidate CheckpointArtifact
terminal *attemptTerminalRecorder
retry map[string]any
}
func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, extracts finalizedExtractResults, output *RunOutput) error {
@@ -208,12 +224,16 @@ func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoi
if merged.terminal {
return nil
}
normalized, err := r.runNormalizeStage(ctx, input, checkpoints, loader, doc, sourceInput, sessionID, prepared, merged.artifact, merged.serialized, output)
normalized, err := r.runNormalizeStage(ctx, input, checkpoints, loader, doc, sourceInput, sessionID, prepared, merged.artifact, merged.serialized, merged.reuseEligible, output)
if err != nil {
return &laneRunError{stage: StageNormalize, err: err}
}
if normalized.accepted {
output.NormalizeOutputs = append(output.NormalizeOutputs, contracts.SerializedOutput{StepID: input.stepID, LaneID: lane.ID, NormalizerKey: lane.Normalize.Module, SourceID: doc.ID, Artifact: contracts.CloneSerializedArtifact(normalized.serialized.Artifact)})
if output.normalizeReuseEligibility == nil {
output.normalizeReuseEligibility = make(map[generatedOutputKey]bool)
}
output.normalizeReuseEligibility[generatedOutputKeyFor(input.stepID, lane.ID)] = normalized.reuseEligible
}
return nil
}
@@ -226,9 +246,14 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
mergeInputs[i] = contracts.ExtractArtifact[any]{LaneID: value.LaneID, ExtractorKey: value.ExtractorKey, SourceID: value.SourceID, ChunkID: value.ChunkID, ChunkIndex: value.ChunkIndex, ChunkRef: value.ChunkRef, Value: value.Value}
}
mergeReferences := operationReferenceSet(input, lane.MergeReferences)
stageResult.reuseEligible = extracts.reuseEligible && referenceTargetReuseEligible(input, lane.MergeReferences)
mergeDeps := append(artifactCheckpointDigests(extracts.serialized), generatedReferenceDependencies(mergeReferences)...)
mergeDeps = normalizeCheckpointFingerprints(mergeDeps)
mergeCP, mergeDecision := loadMerge(loader, input.stepID, lane.ID, lane.Merge.Module, mergeDeps)
var mergeCP MergeCheckpoint
mergeDecision := checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonValidationIncompleteLineage)
if stageResult.reuseEligible {
mergeCP, mergeDecision = loadMerge(loader, input.stepID, lane.ID, lane.Merge.Module, mergeDeps)
}
mergeResolution, err := resolveCheckpointDecision(output, loader, input.CheckpointPolicy, StageMerge, input.stepID, lane.ID, lane.Merge.Module, mergeDecision, typed.codec, []CheckpointArtifact{mergeCP.Output})
if err != nil {
return stageResult, err
@@ -246,70 +271,106 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
mergeWarnings = cloneWarnings(mergeCP.Warnings)
output.Warnings = append(output.Warnings, mergeWarnings...)
} else {
if err := checkpointMergeRunning(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps); err != nil {
return stageResult, err
if stageResult.reuseEligible {
if err := checkpointMergeRunning(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps); err != nil {
return stageResult, err
}
}
retryResult, runErr := runWithRetry(ctx, lane.Merge.Retries, func(attempt int) (retryAttemptResult, error) {
terminalResult, runErr := runProducerAttempts(ctx, producerAttemptConfig{Retries: lane.Merge.Retries, Policy: lane.MergeValidationPolicy, AllowStructuralRetry: lane.MergeExecutionClass == contracts.ExecutionClassLLMBacked}, func(attemptCtx context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
started := time.Now().UTC()
attemptPath := path.Join("merge", fileio.EncodePathComponent(lane.ID), fmt.Sprintf("attempt-%02d", attempt))
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "merge", llmScope, debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, Attempt: attempt, StartedAt: started})
attemptPath := path.Join("merge", fileio.EncodePathComponent(lane.ID), fmt.Sprintf("attempt-%02d", request.Number))
attemptCtx, llmScope := withDebugLLMScope(attemptCtx, attemptPath)
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "merge", llmScope, debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, Attempt: request.Number, AttemptKind: string(request.Kind), StartedAt: started})
requestMetadata, metadataErr := cloneMetadata(input.Metadata)
if metadataErr != nil {
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("clone merge request metadata: %w", metadataErr))
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("clone merge request metadata: %w", metadataErr))
}
result, callErr := typed.merge(attemptCtx, typed.merger, contracts.TypedMergeRequest[any]{Source: doc, LaneID: lane.ID, ExtractOutputs: mergeInputs, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(mergeReferences), LLMProfile: lane.Merge.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(lane.Merge.StructuredOutputRepairAttempts), Metadata: requestMetadata})
result, callErr := typed.merge(attemptCtx, typed.merger, contracts.TypedMergeRequest[any]{Source: doc, LaneID: lane.ID, ExtractOutputs: mergeInputs, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(mergeReferences), LLMProfile: lane.Merge.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(lane.Merge.StructuredOutputRepairAttempts), Correction: request.Correction, Metadata: requestMetadata})
if callErr != nil {
attemptErr := fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, lane.Merge.Module, callErr)
return retryAttemptResult{}, terminal.record(nil, attemptErr)
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, lane.Merge.Module, callErr))
}
candidate := erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: result.Value}
attemptWarnings := cloneWarnings(result.Warnings)
warnings := cloneWarnings(result.Warnings)
serializedCandidate, encodeErr := serializeCandidateArtifact(typed.codec, candidate.LaneID, candidate.MergerKey, candidate.SourceID, candidate.Value)
if encodeErr != nil {
attemptErr := fmt.Errorf("serialize merge candidate for lane %q: %w", lane.ID, encodeErr)
return retryAttemptResult{}, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}, attemptErr)
return producerAttemptOutput{}, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(warnings)}, fmt.Errorf("serialize merge candidate for lane %q: %w", lane.ID, encodeErr))
}
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageMerge, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: mergeReferences, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.mergeValidators, attempt, input.Debug)
attemptWarnings = append(attemptWarnings, warnings...)
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
if validateErr != nil {
return retryAttemptResult{}, terminal.record(payload, validateErr)
return producerAttemptOutput{Value: mergeAttemptValue{artifact: candidate, candidate: serializedCandidate, terminal: &terminal}, Candidate: result.ModelCandidate, Warnings: warnings}, nil
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
candidate, ok := output.Value.(mergeAttemptValue)
if !ok {
return validationReport{}, fmt.Errorf("merge attempt has incompatible value")
}
if rejected != nil {
if debugErr := terminal.record(payload, nil); debugErr != nil {
return retryAttemptResult{}, debugErr
}
return retryAttemptResult{rejection: rejected, warnings: attemptWarnings}, nil
target := typedValidationTarget{stage: StageMerge, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: mergeReferences, metadata: input.Metadata, value: candidate.artifact.Value, candidate: &candidate.candidate}
report, validationErr := r.validateTypedReport(validationCtx, typed.codec, target, prepared.mergeValidators, candidate.terminal.envelope.Attempt, input.Debug)
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "warnings": debugWarningEnvelopes(append(cloneWarnings(output.Warnings), report.Warnings()...)), "rejection": debugRejectedOutputPtr(typedRejection(report, target, candidate.terminal.envelope.Attempt))}
if validationErr != nil {
return report, candidate.terminal.record(payload, validationErr)
}
stored, encodeErr := checkpointArtifact(typed.codec, candidate.LaneID, candidate.MergerKey, candidate.SourceID, candidate.Value)
if encodeErr != nil {
attemptErr := fmt.Errorf("serialize accepted merge output for lane %q: %w", lane.ID, encodeErr)
return retryAttemptResult{}, terminal.record(payload, attemptErr)
}
if debugErr := terminal.record(payload, nil); debugErr != nil {
return retryAttemptResult{}, debugErr
}
merged, serializedMerge = candidate, stored
mergeWarnings = attemptWarnings
return retryAttemptResult{accepted: true}, nil
return report, candidate.terminal.record(payload, nil)
})
terminalSummary := validationSummary(terminalResult, StageMerge, input.stepID, lane.ID, lane.Merge.Module, "", 0)
if debugErr := writeProducerTerminalDebug(input.Debug, path.Join("merge", fileio.EncodePathComponent(lane.ID), "terminal.json"), terminalResult, lane.MergeValidationPolicy, terminalSummary); debugErr != nil {
if stageResult.reuseEligible {
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, debugErr)
}
return stageResult, debugErr
}
if runErr != nil {
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, runErr)
if stageResult.reuseEligible {
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, runErr)
}
return stageResult, runErr
}
if !retryResult.accepted {
output.Warnings = append(output.Warnings, retryResult.warnings...)
output.Rejected = append(output.Rejected, *retryResult.rejection)
if err := checkpointMergeRejected(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, *retryResult.rejection); err != nil {
return stageResult, err
if terminalResult.Action == producerTerminalRejected {
rejected := terminalResult.Rejection
if rejected == nil {
return stageResult, fmt.Errorf("merge attempt terminal is missing rejection")
}
rejected.Stage, rejected.StepID, rejected.LaneID, rejected.ModuleKey = string(StageMerge), input.stepID, lane.ID, lane.Merge.Module
rejected.Validation = &terminalSummary
output.ValidationSummaries = append(output.ValidationSummaries, artifacts.CloneValidationSummary(terminalSummary))
output.Warnings = append(output.Warnings, terminalResult.Warnings...)
output.Rejected = append(output.Rejected, *rejected)
if stageResult.reuseEligible {
if err := checkpointMergeRejected(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, *rejected); err != nil {
return stageResult, err
}
}
stageResult.terminal = true
return stageResult, nil
}
candidate, ok := terminalResult.Value.(mergeAttemptValue)
if !ok {
return stageResult, fmt.Errorf("merge attempt terminal has incompatible value")
}
stored, encodeErr := checkpointArtifact(typed.codec, candidate.artifact.LaneID, candidate.artifact.MergerKey, candidate.artifact.SourceID, candidate.artifact.Value)
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "warnings": debugWarningEnvelopes(terminalResult.Warnings), "rejection": debugRejectedOutputPtr(nil)}
if encodeErr != nil {
attemptErr := fmt.Errorf("serialize accepted merge output for lane %q: %w", lane.ID, encodeErr)
if stageResult.reuseEligible {
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, attemptErr)
}
return stageResult, candidate.terminal.record(payload, attemptErr)
}
if debugErr := candidate.terminal.record(payload, nil); debugErr != nil {
if stageResult.reuseEligible {
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, debugErr)
}
return stageResult, debugErr
}
merged, serializedMerge = candidate.artifact, stored
mergeWarnings = cloneWarnings(terminalResult.Warnings)
output.ValidationSummaries = append(output.ValidationSummaries, terminalSummary)
stageResult.validationIncomplete = terminalResult.ValidationIncomplete
if stageResult.validationIncomplete {
stageResult.reuseEligible = false
}
output.Warnings = append(output.Warnings, mergeWarnings...)
if err := recordMerge(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge, mergeWarnings); err != nil {
return stageResult, err
if stageResult.reuseEligible {
if err := recordMerge(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge, mergeWarnings); err != nil {
return stageResult, err
}
}
}
if err := writeDebugTimed(input.Debug, path.Join("merge", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedMerge), "warnings": debugWarningEnvelopes(mergeWarnings)}}); err != nil {
@@ -321,18 +382,24 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
}
type normalizeStageResult struct {
serialized CheckpointArtifact
warnings []contracts.Warning
accepted bool
serialized CheckpointArtifact
warnings []contracts.Warning
accepted bool
reuseEligible bool
}
func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, merged erasedMergeArtifact, serializedMerge CheckpointArtifact, output *RunOutput) (normalizeStageResult, error) {
func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, merged erasedMergeArtifact, serializedMerge CheckpointArtifact, upstreamReuseEligible bool, output *RunOutput) (normalizeStageResult, error) {
var stageResult normalizeStageResult
lane, typed := prepared.resolved, prepared.typed
normalizeReferences := operationReferenceSet(input, lane.NormalizeReferences)
stageResult.reuseEligible = upstreamReuseEligible && referenceTargetReuseEligible(input, lane.NormalizeReferences)
normalizeDeps := append(artifactCheckpointDigests([]CheckpointArtifact{serializedMerge}), generatedReferenceDependencies(normalizeReferences)...)
normalizeDeps = normalizeCheckpointFingerprints(normalizeDeps)
normalizeCP, normalizeDecision := loadNormalize(loader, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps)
var normalizeCP NormalizeCheckpoint
normalizeDecision := checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonValidationIncompleteLineage)
if stageResult.reuseEligible {
normalizeCP, normalizeDecision = loadNormalize(loader, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps)
}
normalizeResolution, err := resolveCheckpointDecision(output, loader, input.CheckpointPolicy, StageNormalize, input.stepID, lane.ID, lane.Normalize.Module, normalizeDecision, typed.codec, []CheckpointArtifact{normalizeCP.Output})
if err != nil {
return stageResult, err
@@ -348,89 +415,125 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
normalizeWarnings = cloneWarnings(normalizeCP.Warnings)
output.Warnings = append(output.Warnings, normalizeWarnings...)
} else {
if err := checkpointNormalizeRunning(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps); err != nil {
return stageResult, err
if stageResult.reuseEligible {
if err := checkpointNormalizeRunning(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps); err != nil {
return stageResult, err
}
}
retryResult, runErr := runWithRetry(ctx, lane.Normalize.Retries, func(attempt int) (retryAttemptResult, error) {
terminalResult, runErr := runProducerAttempts(ctx, producerAttemptConfig{Retries: lane.Normalize.Retries, Policy: lane.NormalizeValidationPolicy, AllowStructuralRetry: lane.NormalizeExecutionClass == contracts.ExecutionClassLLMBacked}, func(attemptCtx context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
started := time.Now().UTC()
attemptPath := path.Join("normalize", fileio.EncodePathComponent(lane.ID), fmt.Sprintf("attempt-%02d", attempt))
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "normalize", llmScope, debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, Attempt: attempt, StartedAt: started})
attemptPath := path.Join("normalize", fileio.EncodePathComponent(lane.ID), fmt.Sprintf("attempt-%02d", request.Number))
attemptCtx, llmScope := withDebugLLMScope(attemptCtx, attemptPath)
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "normalize", llmScope, debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, Attempt: request.Number, AttemptKind: string(request.Kind), StartedAt: started})
requestMetadata, metadataErr := cloneMetadata(input.Metadata)
if metadataErr != nil {
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("clone normalize request metadata: %w", metadataErr))
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("clone normalize request metadata: %w", metadataErr))
}
result, callErr := typed.normalize(attemptCtx, typed.normalizer, contracts.TypedNormalizeRequest[any]{Source: doc, LaneID: lane.ID, MergeOutput: contracts.MergeArtifact[any]{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: merged.Value}, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(normalizeReferences), LLMProfile: lane.Normalize.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(lane.Normalize.StructuredOutputRepairAttempts), Metadata: requestMetadata})
result, callErr := typed.normalize(attemptCtx, typed.normalizer, contracts.TypedNormalizeRequest[any]{Source: doc, LaneID: lane.ID, MergeOutput: contracts.MergeArtifact[any]{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: merged.Value}, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(normalizeReferences), LLMProfile: lane.Normalize.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(lane.Normalize.StructuredOutputRepairAttempts), Correction: request.Correction, Metadata: requestMetadata})
if callErr != nil {
attemptErr := fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, lane.Normalize.Module, callErr)
return retryAttemptResult{}, terminal.record(nil, attemptErr)
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, lane.Normalize.Module, callErr))
}
attemptWarnings := cloneWarnings(result.Warnings)
warnings := cloneWarnings(result.Warnings)
serializedCandidate, encodeErr := serializeCandidateArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, result.Value)
if encodeErr != nil {
attemptErr := fmt.Errorf("serialize normalize candidate for lane %q: %w", lane.ID, encodeErr)
return retryAttemptResult{}, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}, attemptErr)
return producerAttemptOutput{}, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(warnings)}, fmt.Errorf("serialize normalize candidate for lane %q: %w", lane.ID, encodeErr))
}
var retryPayload map[string]any
attemptValue := normalizeAttemptValue{value: result.Value, candidate: serializedCandidate, terminal: &terminal}
var directive *producerRetryDirective
if result.Retry != nil {
if err := validateNormalizeRetry(result.Retry); err != nil {
return retryAttemptResult{}, terminal.record(map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings)}, fmt.Errorf("normalize lane %q returned invalid retry directive: %w", lane.ID, err))
return producerAttemptOutput{}, terminal.record(map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(warnings)}, fmt.Errorf("normalize lane %q returned invalid retry directive: %w", lane.ID, err))
}
retryRemaining := attempt <= lane.Normalize.Retries
retryPayload = map[string]any{
"reason_code": result.Retry.ReasonCode,
"message": result.Retry.Message,
"another_attempt": retryRemaining,
"fallback_accepted": !retryRemaining,
anotherAttempt := request.Number <= lane.Normalize.Retries
attemptValue.retry = map[string]any{"reason_code": result.Retry.ReasonCode, "message": result.Retry.Message, "another_attempt": anotherAttempt, "fallback_accepted": !anotherAttempt}
directive = &producerRetryDirective{FallbackWarnings: cloneWarnings(result.Retry.FallbackWarnings)}
if anotherAttempt {
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(warnings), "retry": attemptValue.retry}
if debugErr := terminal.record(payload, nil); debugErr != nil {
return producerAttemptOutput{}, debugErr
}
}
if retryRemaining {
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "retry": retryPayload}
return retryAttemptResult{}, terminal.record(payload, nil)
}
attemptWarnings = append(attemptWarnings, cloneWarnings(result.Retry.FallbackWarnings)...)
}
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageNormalize, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: normalizeReferences, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.normalizeValidators, attempt, input.Debug)
attemptWarnings = append(attemptWarnings, warnings...)
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
if retryPayload != nil {
payload["retry"] = retryPayload
return producerAttemptOutput{Value: attemptValue, Candidate: result.ModelCandidate, Warnings: warnings, Retry: directive}, nil
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
candidate, ok := output.Value.(normalizeAttemptValue)
if !ok {
return validationReport{}, fmt.Errorf("normalize attempt has incompatible value")
}
if validateErr != nil {
return retryAttemptResult{}, terminal.record(payload, validateErr)
target := typedValidationTarget{stage: StageNormalize, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: normalizeReferences, metadata: input.Metadata, value: candidate.value, candidate: &candidate.candidate}
report, validationErr := r.validateTypedReport(validationCtx, typed.codec, target, prepared.normalizeValidators, candidate.terminal.envelope.Attempt, input.Debug)
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "warnings": debugWarningEnvelopes(append(cloneWarnings(output.Warnings), report.Warnings()...)), "rejection": debugRejectedOutputPtr(typedRejection(report, target, candidate.terminal.envelope.Attempt))}
if candidate.retry != nil {
payload["retry"] = candidate.retry
}
if rejected != nil {
if debugErr := terminal.record(payload, nil); debugErr != nil {
return retryAttemptResult{}, debugErr
}
return retryAttemptResult{rejection: rejected, warnings: attemptWarnings}, nil
if validationErr != nil {
return report, candidate.terminal.record(payload, validationErr)
}
stored, encodeErr := checkpointArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, result.Value)
if encodeErr != nil {
attemptErr := fmt.Errorf("serialize accepted normalize output for lane %q: %w", lane.ID, encodeErr)
return retryAttemptResult{}, terminal.record(payload, attemptErr)
}
if debugErr := terminal.record(payload, nil); debugErr != nil {
return retryAttemptResult{}, debugErr
}
serializedNormalize = stored
normalizeWarnings = attemptWarnings
return retryAttemptResult{accepted: true}, nil
return report, candidate.terminal.record(payload, nil)
})
terminalSummary := validationSummary(terminalResult, StageNormalize, input.stepID, lane.ID, lane.Normalize.Module, "", 0)
if debugErr := writeProducerTerminalDebug(input.Debug, path.Join("normalize", fileio.EncodePathComponent(lane.ID), "terminal.json"), terminalResult, lane.NormalizeValidationPolicy, terminalSummary); debugErr != nil {
if stageResult.reuseEligible {
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, debugErr)
}
return stageResult, debugErr
}
if runErr != nil {
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, runErr)
if stageResult.reuseEligible {
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, runErr)
}
return stageResult, runErr
}
if !retryResult.accepted {
output.Warnings = append(output.Warnings, retryResult.warnings...)
output.Rejected = append(output.Rejected, *retryResult.rejection)
if err := checkpointNormalizeRejected(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, *retryResult.rejection); err != nil {
return stageResult, err
if terminalResult.Action == producerTerminalRejected {
rejected := terminalResult.Rejection
if rejected == nil {
return stageResult, fmt.Errorf("normalize attempt terminal is missing rejection")
}
rejected.Stage, rejected.StepID, rejected.LaneID, rejected.ModuleKey = string(StageNormalize), input.stepID, lane.ID, lane.Normalize.Module
rejected.Validation = &terminalSummary
output.ValidationSummaries = append(output.ValidationSummaries, artifacts.CloneValidationSummary(terminalSummary))
output.Warnings = append(output.Warnings, terminalResult.Warnings...)
output.Rejected = append(output.Rejected, *rejected)
if stageResult.reuseEligible {
if err := checkpointNormalizeRejected(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, *rejected); err != nil {
return stageResult, err
}
}
return stageResult, nil
}
candidate, ok := terminalResult.Value.(normalizeAttemptValue)
if !ok {
return stageResult, fmt.Errorf("normalize attempt terminal has incompatible value")
}
stored, encodeErr := checkpointArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, candidate.value)
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "warnings": debugWarningEnvelopes(terminalResult.Warnings), "rejection": debugRejectedOutputPtr(nil)}
if candidate.retry != nil {
payload["retry"] = candidate.retry
}
if encodeErr != nil {
attemptErr := fmt.Errorf("serialize accepted normalize output for lane %q: %w", lane.ID, encodeErr)
if stageResult.reuseEligible {
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, attemptErr)
}
return stageResult, candidate.terminal.record(payload, attemptErr)
}
if debugErr := candidate.terminal.record(payload, nil); debugErr != nil {
if stageResult.reuseEligible {
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, debugErr)
}
return stageResult, debugErr
}
serializedNormalize = stored
normalizeWarnings = cloneWarnings(terminalResult.Warnings)
output.ValidationSummaries = append(output.ValidationSummaries, terminalSummary)
output.Warnings = append(output.Warnings, normalizeWarnings...)
if err := recordNormalize(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize, normalizeWarnings); err != nil {
return stageResult, err
if terminalResult.ValidationIncomplete {
stageResult.reuseEligible = false
}
if stageResult.reuseEligible {
if err := recordNormalize(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize, normalizeWarnings); err != nil {
return stageResult, err
}
}
}
if err := writeDebugTimed(input.Debug, path.Join("normalize", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedNormalize), "warnings": debugWarningEnvelopes(normalizeWarnings)}}); err != nil {
@@ -471,59 +574,56 @@ func setTypedLaneManifestMetadata(output *RunOutput, laneID string, extractor, m
return nil
}
func (r *Runner) validateTypedArtifact(ctx context.Context, codec artifactCodecEntry, target typedValidationTarget, chain preparedValidatorChain, attempt int, debug DebugRecorder) ([]contracts.Warning, *contracts.RejectedOutput, error) {
var warnings []contracts.Warning
func (r *Runner) validateTypedReport(ctx context.Context, codec artifactCodecEntry, target typedValidationTarget, chain preparedValidatorChain, attempt int, debug DebugRecorder) (validationReport, error) {
if len(chain.validators) > 0 && target.candidate == nil {
candidate, err := validationCandidateArtifact(codec, target)
if err != nil {
return nil, nil, fmt.Errorf("serialize %s candidate for validation: %w", target.stage, err)
return validationReport{}, fmt.Errorf("serialize %s candidate for validation: %w", target.stage, err)
}
target.candidate = &candidate
}
for index, item := range chain.validators {
report, err := executeValidationChain(ctx, chain, func(validatorCtx context.Context, item preparedValidator, validatorAttempt int) (validationInvocation, error) {
binding := item.resolved.Binding
var result contracts.ValidationResult
var err error
started := time.Now().UTC()
attemptPath := path.Join("validate", fileio.EncodePathComponent(string(target.stage)), fileio.EncodePathComponent(target.laneID), fileio.EncodePathComponent(target.moduleKey), fmt.Sprintf("%02d-%s-attempt-%02d", index+1, fileio.EncodePathComponent(binding.Module), attempt))
validatorCtx, llmScope := withIsolatedDebugLLMScope(ctx, attemptPath)
attemptPath := validatorAttemptPath(path.Join("validate", fileio.EncodePathComponent(string(target.stage)), fileio.EncodePathComponent(target.laneID), fileio.EncodePathComponent(target.moduleKey), fmt.Sprintf("%02d-%s-attempt-%02d", item.position, fileio.EncodePathComponent(binding.Module), attempt)), validatorAttempt)
validatorCtx, llmScope := withIsolatedDebugLLMScope(validatorCtx, attemptPath)
requestTarget := target
requestTarget.sourceInput = target.sourceInput.Clone()
requestTarget.references = CloneReferenceSet(target.references)
requestTarget.metadata, err = cloneMetadata(target.metadata)
if err != nil {
return nil, nil, fmt.Errorf("clone typed validation metadata: %w", err)
return validationInvocation{}, fatalValidationError(fmt.Errorf("clone typed validation metadata: %w", err))
}
requestTarget.chunk, err = cloneSourceChunkPtr(target.chunk)
if err != nil {
return nil, nil, fmt.Errorf("clone typed validation chunk: %w", err)
return validationInvocation{}, fatalValidationError(fmt.Errorf("clone typed validation chunk: %w", err))
}
requestTarget.chunks, err = cloneSourceChunks(target.chunks)
if err != nil {
return nil, nil, fmt.Errorf("clone typed validation chunks: %w", err)
return validationInvocation{}, fatalValidationError(fmt.Errorf("clone typed validation chunks: %w", err))
}
switch item.resolved.Target {
case ValidatorTargetTyped:
candidateValue, decodeErr := decodeTypedValidationCandidate(codec, *target.candidate)
if decodeErr != nil {
err = fmt.Errorf("decode %s candidate for typed validator %q: %w", target.stage, binding.Module, decodeErr)
break
return validationInvocation{}, fatalValidationError(fmt.Errorf("decode %s candidate for typed validator %q: %w", target.stage, binding.Module, decodeErr))
}
requestTarget.value = candidateValue
requestTarget.llmProfile = binding.LLMProfile
requestTarget.structuredOutputRepairAttempts = cloneStructuredOutputRepairAttempts(binding.StructuredOutputRepairAttempts)
result, err = item.typedValidate(validatorCtx, item.typed, requestTarget)
case ValidatorTargetSerialized:
artifact, encodeErr := validationCandidateArtifact(codec, target)
if encodeErr != nil {
err = encodeErr
break
}
artifact := cloneCheckpointArtifact(*target.candidate)
result, err = item.serialized.Validate(validatorCtx, contracts.SerializedValidationRequest{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Source: target.source, SourceID: target.sourceID, SourceInput: requestTarget.sourceInput, SessionID: target.sessionID, References: requestTarget.references, LLMProfile: binding.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(binding.StructuredOutputRepairAttempts), Metadata: requestTarget.metadata, Chunk: requestTarget.chunk, Chunks: requestTarget.chunks, Schema: contracts.CloneArtifactSchema(artifact.Artifact.Schema), MediaType: artifact.Artifact.MediaType, Content: append([]byte(nil), artifact.Artifact.Content...)})
default:
return nil, nil, fmt.Errorf("validator %q is incompatible with typed artifact validation", binding.Module)
return validationInvocation{}, fatalValidationError(fmt.Errorf("validator %q is incompatible with typed artifact validation", binding.Module))
}
artifact, _ := validationCandidateArtifact(codec, target)
if err == nil {
err = contracts.ValidateValidationResult(result)
}
artifact := cloneCheckpointArtifact(*target.candidate)
debugCall := debugValidationCall{ValidatorName: binding.Module, Request: map[string]any{"stage": string(target.stage), "lane_id": target.laneID, "module_key": target.moduleKey, "source_id": target.sourceID, "artifact": debugCheckpointArtifact(artifact), "metadata": redactSensitiveMap(target.metadata)}, Result: debugValidationResultEnvelope(result)}
if err != nil {
debugCall.Error = err.Error()
@@ -531,37 +631,31 @@ func (r *Runner) validateTypedArtifact(ctx context.Context, codec artifactCodecE
if err != nil {
validationErr := fmt.Errorf("validate typed %s output with validator %q: %w", target.stage, binding.Module, err)
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall, Error: debugCall.Error}, llmScope); debugErr != nil {
return warnings, nil, errors.Join(validationErr, fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr))
return validationInvocation{}, fatalValidationError(errors.Join(validationErr, fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr)))
}
return warnings, nil, validationErr
return validationInvocation{}, validationErr
}
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall}, llmScope); debugErr != nil {
return warnings, nil, fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr)
return validationInvocation{}, fatalValidationError(fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr))
}
if !result.Approved {
reason := result.ReasonCode
if reason == "" {
reason = "artifact_rejected"
}
message := result.Message
if message == "" {
message = "artifact rejected"
}
return warnings, &contracts.RejectedOutput{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, ChunkID: func() string {
if target.chunk != nil {
return target.chunk.ID
}
return ""
}(), ChunkIndex: func() int {
if target.chunk != nil {
return target.chunk.Index
}
return 0
}(), ValidatorName: binding.Module, ReasonCode: reason, Message: message, AttemptCount: attempt, DiagnosticArtifactPath: result.DiagnosticArtifactPath}, nil
}
warnings = append(warnings, result.Warnings...)
return validationInvocation{result: result}, nil
})
if err != nil {
return report, err
}
return warnings, nil, nil
return report, nil
}
func typedRejection(report validationReport, target typedValidationTarget, attempt int) *contracts.RejectedOutput {
rejection := report.FirstRejection()
if rejection == nil {
return nil
}
chunkID, chunkIndex := "", 0
if target.chunk != nil {
chunkID, chunkIndex = target.chunk.ID, target.chunk.Index
}
return &contracts.RejectedOutput{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, ChunkID: chunkID, ChunkIndex: chunkIndex, ValidatorName: rejection.validatorName, ReasonCode: rejection.reasonCode, Message: rejection.message, AttemptCount: attempt, DiagnosticArtifactPath: rejection.diagnosticPath}
}
func validationCandidateArtifact(codec artifactCodecEntry, target typedValidationTarget) (CheckpointArtifact, error) {

View File

@@ -22,9 +22,10 @@ type erasedMergeArtifact struct {
}
type erasedTypedResult struct {
Value any
Warnings []contracts.Warning
Retry *contracts.NormalizeRetry
Value any
Warnings []contracts.Warning
Retry *contracts.NormalizeRetry
ModelCandidate *contracts.ModelCandidate
}
type typedValidationTarget struct {

View File

@@ -1,12 +1,70 @@
package pipeline
import (
"bytes"
"context"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type correctionObservingNotesExtractor struct {
correction *contracts.SemanticCorrection
candidate *contracts.ModelCandidate
}
func (*correctionObservingNotesExtractor) Key() string { return "test/correction-observing-extract" }
func (*correctionObservingNotesExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (e *correctionObservingNotesExtractor) Extract(_ context.Context, request contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[codecNotes], error) {
e.correction = request.Correction
candidate, err := contracts.NewModelCandidate([]byte(`{"items":["extract"]}`), contracts.CorrectionProtocolSingleResponseV1)
if err != nil {
return contracts.TypedExtractionResult[codecNotes]{}, err
}
e.candidate = candidate
return contracts.TypedExtractionResult[codecNotes]{Value: codecNotes{Items: []string{"extract"}}, ModelCandidate: candidate}, nil
}
type correctionObservingNotesMerger struct {
correction *contracts.SemanticCorrection
candidate *contracts.ModelCandidate
}
func (*correctionObservingNotesMerger) Key() string { return "test/correction-observing-merge" }
func (m *correctionObservingNotesMerger) Merge(_ context.Context, request contracts.TypedMergeRequest[codecNotes]) (contracts.TypedMergeResult[codecNotes], error) {
m.correction = request.Correction
candidate, err := contracts.NewModelCandidate([]byte(`{"items":["merge"]}`), contracts.CorrectionProtocolSingleResponseV1)
if err != nil {
return contracts.TypedMergeResult[codecNotes]{}, err
}
m.candidate = candidate
return contracts.TypedMergeResult[codecNotes]{Value: codecNotes{Items: []string{"merge"}}, ModelCandidate: candidate}, nil
}
type correctionObservingNotesNormalizer struct {
correction *contracts.SemanticCorrection
candidate *contracts.ModelCandidate
}
func (*correctionObservingNotesNormalizer) Key() string { return "test/correction-observing-normalize" }
func (*correctionObservingNotesNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (n *correctionObservingNotesNormalizer) Normalize(_ context.Context, request contracts.TypedNormalizeRequest[codecNotes]) (contracts.TypedNormalizeResult[codecNotes], error) {
n.correction = request.Correction
candidate, err := contracts.NewModelCandidate([]byte(`{"items":["normalize"]}`), contracts.CorrectionProtocolSingleResponseV1)
if err != nil {
return contracts.TypedNormalizeResult[codecNotes]{}, err
}
n.candidate = candidate
return contracts.TypedNormalizeResult[codecNotes]{Value: codecNotes{Items: []string{"normalize"}}, ModelCandidate: candidate}, nil
}
type repairObservingNotesMerger struct {
attempts *int
}
@@ -93,3 +151,103 @@ func TestTypedRegistryErasurePreservesStructuredOutputRepairAttempts(t *testing.
}
})
}
func TestTypedRegistryErasurePreservesCorrectionAndCandidateOwnership(t *testing.T) {
const artifactKind contracts.ArtifactKind = "test/notes"
t.Run("extract", func(t *testing.T) {
correction := newTestCorrection(t)
implementation := &correctionObservingNotesExtractor{}
registry := NewExtractorRegistry()
if err := RegisterExtractor(registry, ModuleSpec{Key: implementation.Key(), Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: artifactKind}, func() (contracts.Extractor[codecNotes], error) {
return implementation, nil
}); err != nil {
t.Fatalf("RegisterExtractor() error = %v", err)
}
entry, ok := registry.typedEntry(implementation.Key())
if !ok {
t.Fatal("typed extractor entry missing")
}
built, err := entry.builder(BuildRequest{})
if err != nil {
t.Fatalf("builder() error = %v", err)
}
result, err := entry.extract(context.Background(), built, contracts.TypedExtractionRequest{Correction: correction})
if err != nil {
t.Fatalf("extract() error = %v", err)
}
assertCorrectionAndCandidateOwnership(t, correction, implementation.correction, implementation.candidate, result.ModelCandidate)
})
t.Run("merge", func(t *testing.T) {
correction := newTestCorrection(t)
implementation := &correctionObservingNotesMerger{}
registry := NewMergerRegistry()
if err := RegisterMerger(registry, ModuleSpec{Key: implementation.Key(), Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: artifactKind}, func() (contracts.Merger[codecNotes], error) {
return implementation, nil
}); err != nil {
t.Fatalf("RegisterMerger() error = %v", err)
}
entry, ok := registry.typedEntry(implementation.Key(), artifactKind)
if !ok {
t.Fatal("typed merger entry missing")
}
built, err := entry.builder(BuildRequest{})
if err != nil {
t.Fatalf("builder() error = %v", err)
}
result, err := entry.merge(context.Background(), built, contracts.TypedMergeRequest[any]{Correction: correction, ExtractOutputs: []contracts.ExtractArtifact[any]{{Value: codecNotes{Items: []string{"extract"}}}}})
if err != nil {
t.Fatalf("merge() error = %v", err)
}
assertCorrectionAndCandidateOwnership(t, correction, implementation.correction, implementation.candidate, result.ModelCandidate)
})
t.Run("normalize", func(t *testing.T) {
correction := newTestCorrection(t)
implementation := &correctionObservingNotesNormalizer{}
registry := NewNormalizerRegistry()
if err := RegisterNormalizer(registry, ModuleSpec{Key: implementation.Key(), Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: artifactKind}, func() (contracts.Normalizer[codecNotes], error) {
return implementation, nil
}); err != nil {
t.Fatalf("RegisterNormalizer() error = %v", err)
}
entry, ok := registry.typedEntry(implementation.Key(), artifactKind)
if !ok {
t.Fatal("typed normalizer entry missing")
}
built, err := entry.builder(BuildRequest{})
if err != nil {
t.Fatalf("builder() error = %v", err)
}
result, err := entry.normalize(context.Background(), built, contracts.TypedNormalizeRequest[any]{Correction: correction, MergeOutput: contracts.MergeArtifact[any]{Value: codecNotes{Items: []string{"merge"}}}})
if err != nil {
t.Fatalf("normalize() error = %v", err)
}
assertCorrectionAndCandidateOwnership(t, correction, implementation.correction, implementation.candidate, result.ModelCandidate)
})
}
func newTestCorrection(t *testing.T) *contracts.SemanticCorrection {
t.Helper()
correction, err := contracts.NewSemanticCorrection([]byte(`{"items":["original"]}`), "Correct the response.")
if err != nil {
t.Fatalf("NewSemanticCorrection() error = %v", err)
}
return correction
}
func assertCorrectionAndCandidateOwnership(t *testing.T, callerCorrection, observedCorrection *contracts.SemanticCorrection, producerCandidate, returnedCandidate *contracts.ModelCandidate) {
t.Helper()
if observedCorrection == nil || producerCandidate == nil || returnedCandidate == nil {
t.Fatal("correction and candidates must be present")
}
callerCorrection.AssistantResponse[0] = '['
if got := string(observedCorrection.AssistantResponse); got != `{"items":["original"]}` {
t.Fatalf("observed correction = %q, want owned original content", got)
}
producerCandidate.Response[0] = '['
if bytes.Equal(producerCandidate.Response, returnedCandidate.Response) {
t.Fatalf("returned candidate aliases producer candidate: %q", returnedCandidate.Response)
}
}

View File

@@ -0,0 +1,216 @@
package pipeline
import (
"context"
"errors"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type validationOutcome string
const (
validationApproved validationOutcome = "approved"
validationRejected validationOutcome = "rejected"
validationFailed validationOutcome = "failed"
validationSkipped validationOutcome = "skipped"
)
const correctionRequestIntroduction = "The previous response failed semantic validation. Return one complete corrected replacement response, not a patch, explanation, or commentary.\n\nCorrect all of the following:\n"
// validationRecord captures the settled result of one configured validator.
// Its fields remain private so reports cannot expose mutable warning storage.
type validationRecord struct {
validatorName string
outcome validationOutcome
attemptCount int
reasonCode string
message string
diagnosticPath string
warnings []contracts.Warning
correctionGuidance string
failure error
}
func (record validationRecord) clone() validationRecord {
record.warnings = cloneWarnings(record.warnings)
return record
}
// validationReport is the immutable, ordered result of one validator chain.
type validationReport struct {
records []validationRecord
}
func (report validationReport) Records() []validationRecord {
records := make([]validationRecord, len(report.records))
for index, record := range report.records {
records[index] = record.clone()
}
return records
}
func (report validationReport) Warnings() []contracts.Warning {
var warnings []contracts.Warning
for _, record := range report.records {
if record.outcome == validationApproved || record.outcome == validationRejected {
warnings = append(warnings, cloneWarnings(record.warnings)...)
}
}
return warnings
}
func (report validationReport) FirstRejection() *validationRecord {
for _, record := range report.records {
if record.outcome == validationRejected {
clone := record.clone()
return &clone
}
}
return nil
}
func (report validationReport) FirstFailure() *validationRecord {
for _, record := range report.records {
if record.outcome == validationFailed {
clone := record.clone()
return &clone
}
}
return nil
}
func (report validationReport) CorrectionRequest() (string, error) {
seen := make(map[string]struct{})
parts := make([]string, 0, len(report.records))
for _, record := range report.records {
if record.outcome != validationRejected {
continue
}
guidance := strings.TrimSpace(record.correctionGuidance)
if guidance == "" {
return "", fmt.Errorf("validator %q rejected output without correction guidance", record.validatorName)
}
if _, exists := seen[guidance]; exists {
continue
}
seen[guidance] = struct{}{}
parts = append(parts, guidance)
}
if len(parts) == 0 {
return "", errors.New("validation report contains no correction guidance")
}
var builder strings.Builder
builder.WriteString(correctionRequestIntroduction)
for index, guidance := range parts {
fmt.Fprintf(&builder, "%d. %s", index+1, guidance)
if index+1 < len(parts) {
builder.WriteByte('\n')
}
}
request := builder.String()
if len(request) > contracts.MaxCorrectionGuidanceBytes {
return "", fmt.Errorf("aggregate correction request exceeds maximum length of %d bytes", contracts.MaxCorrectionGuidanceBytes)
}
return request, nil
}
type validationInvocation struct {
result contracts.ValidationResult
skipped bool
reason string
message string
}
func skippedValidation(reason, message string) validationInvocation {
return validationInvocation{skipped: true, reason: strings.TrimSpace(reason), message: strings.TrimSpace(message)}
}
type validationInvoker func(context.Context, preparedValidator, int) (validationInvocation, error)
type validationFrameworkError struct{ err error }
func (err validationFrameworkError) Error() string { return err.err.Error() }
func (err validationFrameworkError) Unwrap() error { return err.err }
func fatalValidationError(err error) error {
if err == nil {
return nil
}
return validationFrameworkError{err: err}
}
func executeValidationChain(ctx context.Context, chain preparedValidatorChain, invoke validationInvoker) (validationReport, error) {
if ctx == nil {
return validationReport{}, errors.New("validator execution context must not be nil")
}
if invoke == nil {
return validationReport{}, errors.New("validator invocation must not be nil")
}
report := validationReport{records: make([]validationRecord, 0, len(chain.validators))}
for index, validator := range chain.validators {
validator.position = index + 1
if err := ctx.Err(); err != nil {
return validationReport{}, err
}
binding := validator.resolved.Binding
attemptLimit := 1
if validator.resolved.ExecutionClass == contracts.ExecutionClassLLMBacked {
attemptLimit += binding.Retries
}
for attempt := 1; attempt <= attemptLimit; attempt++ {
if err := ctx.Err(); err != nil {
return validationReport{}, err
}
invocation, err := invoke(ctx, validator, attempt)
if err != nil {
var frameworkErr validationFrameworkError
if errors.As(err, &frameworkErr) || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return validationReport{}, err
}
if attempt == attemptLimit {
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationFailed, attemptCount: attempt, message: "validator failed after retry exhaustion", failure: err})
}
continue
}
if invocation.skipped {
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationSkipped, attemptCount: attempt, reasonCode: invocation.reason, message: invocation.message})
break
}
if err := contracts.ValidateValidationResult(invocation.result); err != nil {
if attempt == attemptLimit {
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationFailed, attemptCount: attempt, message: "validator returned an invalid result", failure: err})
}
continue
}
if invocation.result.Approved {
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationApproved, attemptCount: attempt, warnings: cloneWarnings(invocation.result.Warnings), diagnosticPath: invocation.result.DiagnosticArtifactPath})
break
}
message := invocation.result.Message
if message == "" {
message = "output rejected"
}
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationRejected, attemptCount: attempt, reasonCode: invocation.result.ReasonCode, message: message, diagnosticPath: invocation.result.DiagnosticArtifactPath, warnings: cloneWarnings(invocation.result.Warnings), correctionGuidance: invocation.result.CorrectionGuidance})
break
}
}
return report, nil
}
func validatorFailureError(record validationRecord) error {
if record.failure != nil {
return record.failure
}
return fmt.Errorf("validator %q failed after %d attempt(s)", record.validatorName, record.attemptCount)
}
func validatorAttemptPath(base string, attempt int) string {
if attempt == 1 {
return base
}
return fmt.Sprintf("%s-validator-%02d", base, attempt)
}

View File

@@ -0,0 +1,218 @@
package pipeline
import (
"context"
"errors"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestExecuteValidationChainSettlesEveryValidatorInOrder(t *testing.T) {
tests := []struct {
name string
chain preparedValidatorChain
invoke validationInvoker
want []validationOutcome
wantCalls []string
wantGuidance []string
wantWarnings []string
}{
{
name: "all approved",
chain: validationChain(validationSpec("shape", contracts.ExecutionClassDeterministic, 0), validationSpec("refs", contracts.ExecutionClassDeterministic, 0)),
invoke: validationSequence(map[string][]validationStep{
"shape": {{result: contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{ReasonCode: "shape"}}}}},
"refs": {{result: contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{ReasonCode: "refs"}}}}},
}),
want: []validationOutcome{validationApproved, validationApproved},
wantCalls: []string{"shape:1", "refs:1"},
wantWarnings: []string{"shape", "refs"},
},
{
name: "multiple rejections deduplicate guidance",
chain: validationChain(validationSpec("shape", contracts.ExecutionClassDeterministic, 0), validationSpec("refs", contracts.ExecutionClassDeterministic, 0), validationSpec("coverage", contracts.ExecutionClassDeterministic, 0)),
invoke: validationSequence(map[string][]validationStep{
"shape": {{result: contracts.ValidationResult{ReasonCode: "shape", Message: "invalid", CorrectionGuidance: "repair shape"}}},
"refs": {{result: contracts.ValidationResult{ReasonCode: "refs", Message: "missing", CorrectionGuidance: "repair references"}}},
"coverage": {{result: contracts.ValidationResult{ReasonCode: "coverage", Message: "missing", CorrectionGuidance: "repair shape"}}},
}),
want: []validationOutcome{validationRejected, validationRejected, validationRejected},
wantCalls: []string{"shape:1", "refs:1", "coverage:1"},
wantGuidance: []string{"repair shape", "repair references"},
},
{
name: "rejection and failure both settle",
chain: validationChain(validationSpec("shape", contracts.ExecutionClassDeterministic, 0), validationSpec("remote", contracts.ExecutionClassDeterministic, 0)),
invoke: validationSequence(map[string][]validationStep{
"shape": {{result: contracts.ValidationResult{ReasonCode: "shape", CorrectionGuidance: "repair shape"}}},
"remote": {{err: errors.New("offline")}},
}),
want: []validationOutcome{validationRejected, validationFailed},
wantCalls: []string{"shape:1", "remote:1"},
wantGuidance: []string{"repair shape"},
},
{
name: "failure only has no correction guidance",
chain: validationChain(validationSpec("remote", contracts.ExecutionClassDeterministic, 0)),
invoke: validationSequence(map[string][]validationStep{"remote": {{err: errors.New("offline")}}}),
want: []validationOutcome{validationFailed},
wantCalls: []string{"remote:1"},
},
{
name: "skipped is retained without guidance",
chain: validationChain(validationSpec("optional", contracts.ExecutionClassDeterministic, 0)),
invoke: validationSequence(map[string][]validationStep{"optional": {{invocation: skippedValidation("runtime_unavailable", "optional dependency unavailable")}}}),
want: []validationOutcome{validationSkipped},
wantCalls: []string{"optional:1"},
},
{
name: "LLM failure retries then succeeds",
chain: validationChain(validationSpec("remote", contracts.ExecutionClassLLMBacked, 2)),
invoke: validationSequence(map[string][]validationStep{"remote": {{err: errors.New("first")}, {err: errors.New("second")}, {result: contracts.ValidationResult{Approved: true}}}}),
want: []validationOutcome{validationApproved},
wantCalls: []string{"remote:1", "remote:2", "remote:3"},
},
{
name: "LLM failure retry exhaustion records once",
chain: validationChain(validationSpec("remote", contracts.ExecutionClassLLMBacked, 1)),
invoke: validationSequence(map[string][]validationStep{"remote": {{err: errors.New("first")}, {err: errors.New("second")}}}),
want: []validationOutcome{validationFailed},
wantCalls: []string{"remote:1", "remote:2"},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
calls := []string(nil)
invoke := func(ctx context.Context, validator preparedValidator, attempt int) (validationInvocation, error) {
calls = append(calls, validator.resolved.Binding.Module+":"+string(rune('0'+attempt)))
return test.invoke(ctx, validator, attempt)
}
report, err := executeValidationChain(context.Background(), test.chain, invoke)
if err != nil {
t.Fatalf("executeValidationChain() error = %v", err)
}
records := report.Records()
outcomes := make([]validationOutcome, len(records))
for index, record := range records {
outcomes[index] = record.outcome
}
if !reflect.DeepEqual(outcomes, test.want) || !reflect.DeepEqual(calls, test.wantCalls) {
t.Fatalf("outcomes = %#v calls = %#v, want %#v %#v", outcomes, calls, test.want, test.wantCalls)
}
if len(test.wantGuidance) > 0 {
guidance, err := report.CorrectionRequest()
if err != nil {
t.Fatalf("CorrectionRequest() error = %v", err)
}
if !strings.Contains(guidance, "complete corrected replacement") {
t.Fatalf("CorrectionRequest() = %q, want complete replacement instruction", guidance)
}
for _, item := range test.wantGuidance {
if strings.Count(guidance, item) != 1 {
t.Fatalf("CorrectionRequest() = %q, want one occurrence of %q", guidance, item)
}
}
}
warnings := report.Warnings()
var warningCodes []string
for _, warning := range warnings {
warningCodes = append(warningCodes, warning.ReasonCode)
}
if !reflect.DeepEqual(warningCodes, test.wantWarnings) {
t.Fatalf("Warnings() = %#v, want codes %#v", warnings, test.wantWarnings)
}
})
}
}
func TestExecuteValidationChainKeepsInvocationInputsImmutable(t *testing.T) {
candidate := []byte(`{"events":["one"]}`)
chain := validationChain(validationSpec("remote", contracts.ExecutionClassLLMBacked, 1))
var received [][]byte
report, err := executeValidationChain(context.Background(), chain, func(_ context.Context, _ preparedValidator, attempt int) (validationInvocation, error) {
requestCandidate := append([]byte(nil), candidate...)
received = append(received, requestCandidate)
requestCandidate[0] = 'x'
if attempt == 1 {
return validationInvocation{}, errors.New("retry")
}
return validationInvocation{result: contracts.ValidationResult{Approved: true}}, nil
})
if err != nil || len(report.Records()) != 1 || report.Records()[0].attemptCount != 2 {
t.Fatalf("report = %#v, error = %v", report, err)
}
if string(candidate) != `{"events":["one"]}` || len(received) != 2 || string(received[0]) != string(received[1]) {
t.Fatalf("candidate = %q received = %#v, want immutable repeated request content", candidate, received)
}
}
func TestValidationReportRejectsOversizedGuidanceAndOwnsRecords(t *testing.T) {
guidance := strings.Repeat("x", contracts.MaxValidationCorrectionGuidanceBytes)
report := validationReport{records: make([]validationRecord, 17)}
for index := range report.records {
unique := []byte(guidance)
unique[0] = byte('a' + index)
report.records[index] = validationRecord{validatorName: "validator", outcome: validationRejected, correctionGuidance: string(unique)}
}
report.records[0].warnings = []contracts.Warning{{ReasonCode: "warning"}}
if _, err := report.CorrectionRequest(); err == nil {
t.Fatal("CorrectionRequest() error = nil, want aggregate overflow error")
}
records := report.Records()
records[0].warnings[0].ReasonCode = "changed"
if report.Records()[0].warnings[0].ReasonCode != "warning" {
t.Fatal("Records() exposed mutable warning storage")
}
}
func TestExecuteValidationChainReturnsFrameworkAndCancellationErrors(t *testing.T) {
chain := validationChain(validationSpec("validator", contracts.ExecutionClassLLMBacked, 1))
framework := errors.New("debug persistence failed")
if _, err := executeValidationChain(context.Background(), chain, func(context.Context, preparedValidator, int) (validationInvocation, error) {
return validationInvocation{}, fatalValidationError(framework)
}); !errors.Is(err, framework) {
t.Fatalf("framework error = %v, want %v", err, framework)
}
canceled, cancel := context.WithCancel(context.Background())
cancel()
if _, err := executeValidationChain(canceled, chain, func(context.Context, preparedValidator, int) (validationInvocation, error) {
return validationInvocation{result: contracts.ValidationResult{Approved: true}}, nil
}); !errors.Is(err, context.Canceled) {
t.Fatalf("cancellation error = %v, want context canceled", err)
}
}
type validationStep struct {
result contracts.ValidationResult
invocation validationInvocation
err error
}
func validationSequence(steps map[string][]validationStep) validationInvoker {
positions := make(map[string]int)
return func(_ context.Context, validator preparedValidator, _ int) (validationInvocation, error) {
name := validator.resolved.Binding.Module
index := positions[name]
positions[name]++
if index >= len(steps[name]) {
return validationInvocation{}, errors.New("unexpected validator invocation")
}
step := steps[name][index]
if step.invocation.skipped {
return step.invocation, step.err
}
return validationInvocation{result: step.result}, step.err
}
}
func validationChain(validators ...preparedValidator) preparedValidatorChain {
return preparedValidatorChain{validators: validators}
}
func validationSpec(name string, class contracts.ExecutionClass, retries int) preparedValidator {
return preparedValidator{resolved: ResolvedValidator{Binding: ModuleBinding{Module: name, Retries: retries}, ExecutionClass: class}}
}

View File

@@ -0,0 +1,115 @@
package pipeline
import "fmt"
type ProducerStructuralFailureAction string
const (
ProducerStructuralFailureFailRun ProducerStructuralFailureAction = "fail_run"
ProducerStructuralFailureRejectOutput ProducerStructuralFailureAction = "reject_output"
)
type SemanticRejectionAction string
const (
SemanticRejectionFailRun SemanticRejectionAction = "fail_run"
SemanticRejectionRejectOutput SemanticRejectionAction = "reject_output"
)
type ValidatorFailureAction string
const (
ValidatorFailureWarnContinue ValidatorFailureAction = "warn_continue"
ValidatorFailureFailRun ValidatorFailureAction = "fail_run"
)
// ValidationPolicyOverride records only explicitly configured terminal-policy
// values. Nil fields inherit independently from the pipeline or application.
type ValidationPolicyOverride struct {
ProducerStructuralFailure *ProducerStructuralFailureAction `json:"producer_structural_failure,omitempty"`
SemanticRejection *SemanticRejectionAction `json:"semantic_rejection,omitempty"`
ValidatorFailure *ValidatorFailureAction `json:"validator_failure,omitempty"`
}
// ValidationPolicy is the concrete terminal policy retained by each resolved
// producer. It contains no inherited pointers.
type ValidationPolicy struct {
ProducerStructuralFailure ProducerStructuralFailureAction `json:"producer_structural_failure"`
SemanticRejection SemanticRejectionAction `json:"semantic_rejection"`
ValidatorFailure ValidatorFailureAction `json:"validator_failure"`
}
func DefaultValidationPolicy() ValidationPolicy {
return ValidationPolicy{
ProducerStructuralFailure: ProducerStructuralFailureFailRun,
SemanticRejection: SemanticRejectionFailRun,
ValidatorFailure: ValidatorFailureWarnContinue,
}
}
func (override ValidationPolicyOverride) Validate() error {
if override.ProducerStructuralFailure != nil {
switch *override.ProducerStructuralFailure {
case ProducerStructuralFailureFailRun, ProducerStructuralFailureRejectOutput:
default:
return fmt.Errorf("producer_structural_failure must be fail_run or reject_output")
}
}
if override.SemanticRejection != nil {
switch *override.SemanticRejection {
case SemanticRejectionFailRun, SemanticRejectionRejectOutput:
default:
return fmt.Errorf("semantic_rejection must be fail_run or reject_output")
}
}
if override.ValidatorFailure != nil {
switch *override.ValidatorFailure {
case ValidatorFailureWarnContinue, ValidatorFailureFailRun:
default:
return fmt.Errorf("validator_failure must be warn_continue or fail_run")
}
}
return nil
}
func ResolveValidationPolicy(binding, pipeline *ValidationPolicyOverride) ValidationPolicy {
resolved := DefaultValidationPolicy()
applyValidationPolicyOverride(&resolved, pipeline)
applyValidationPolicyOverride(&resolved, binding)
return resolved
}
func cloneValidationPolicyOverride(override *ValidationPolicyOverride) *ValidationPolicyOverride {
if override == nil {
return nil
}
out := *override
if override.ProducerStructuralFailure != nil {
value := *override.ProducerStructuralFailure
out.ProducerStructuralFailure = &value
}
if override.SemanticRejection != nil {
value := *override.SemanticRejection
out.SemanticRejection = &value
}
if override.ValidatorFailure != nil {
value := *override.ValidatorFailure
out.ValidatorFailure = &value
}
return &out
}
func applyValidationPolicyOverride(policy *ValidationPolicy, override *ValidationPolicyOverride) {
if override == nil {
return
}
if override.ProducerStructuralFailure != nil {
policy.ProducerStructuralFailure = *override.ProducerStructuralFailure
}
if override.SemanticRejection != nil {
policy.SemanticRejection = *override.SemanticRejection
}
if override.ValidatorFailure != nil {
policy.ValidatorFailure = *override.ValidatorFailure
}
}

View File

@@ -0,0 +1,103 @@
package pipeline
import (
"encoding/json"
"strings"
"testing"
)
func TestResolvePipelineAppliesValidationPolicyFieldByField(t *testing.T) {
profile := baselineProfile()
profile.ValidationPolicy = validationPolicyOverride(
ProducerStructuralFailureRejectOutput,
SemanticRejectionRejectOutput,
ValidatorFailureFailRun,
)
profile.Chunk.ValidationPolicy = validationPolicyOverride("", "", ValidatorFailureWarnContinue)
lane := profile.Artifacts["events"]
lane.Extract.ValidationPolicy = validationPolicyOverride("", SemanticRejectionFailRun, "")
lane.Merge.ValidationPolicy = validationPolicyOverride("", "", ValidatorFailureWarnContinue)
profile.Artifacts["events"] = lane
resolved, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
if err != nil {
t.Fatalf("ResolvePipeline() error = %v", err)
}
if got, want := resolved.ChunkValidationPolicy, (ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureRejectOutput, SemanticRejection: SemanticRejectionRejectOutput, ValidatorFailure: ValidatorFailureWarnContinue}); got != want {
t.Fatalf("chunk validation policy = %#v, want %#v", got, want)
}
resolvedLane := resolved.Steps[0].ArtifactLanes[0]
if got, want := resolvedLane.ExtractValidationPolicy, (ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureRejectOutput, SemanticRejection: SemanticRejectionFailRun, ValidatorFailure: ValidatorFailureFailRun}); got != want {
t.Fatalf("extract validation policy = %#v, want %#v", got, want)
}
if got, want := resolvedLane.MergeValidationPolicy, (ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureRejectOutput, SemanticRejection: SemanticRejectionRejectOutput, ValidatorFailure: ValidatorFailureWarnContinue}); got != want {
t.Fatalf("merge validation policy = %#v, want %#v", got, want)
}
if got, want := resolvedLane.NormalizeValidationPolicy, (ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureRejectOutput, SemanticRejection: SemanticRejectionRejectOutput, ValidatorFailure: ValidatorFailureFailRun}); got != want {
t.Fatalf("normalize validation policy = %#v, want %#v", got, want)
}
*profile.ValidationPolicy.SemanticRejection = SemanticRejectionFailRun
if got := resolved.ConfiguredValidationPolicy.SemanticRejection; got == nil || *got != SemanticRejectionRejectOutput {
t.Fatalf("resolved configured policy aliases profile: %#v", resolved.ConfiguredValidationPolicy)
}
}
func TestResolvePipelineRejectsStructuralPolicyOverrideOnDeterministicProducer(t *testing.T) {
profile := baselineProfile()
profile.Chunk.ValidationPolicy = validationPolicyOverride(ProducerStructuralFailureRejectOutput, "", "")
_, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
if err == nil || !strings.Contains(err.Error(), "producer_structural_failure") || !strings.Contains(err.Error(), "deterministic") {
t.Fatalf("ResolvePipeline() error = %v, want deterministic structural-policy rejection", err)
}
}
func TestValidationPolicyChangesResolvedDigestAndRoundTripsBindings(t *testing.T) {
left := baselineProfile()
right := baselineProfile()
right.ValidationPolicy = validationPolicyOverride("", SemanticRejectionRejectOutput, "")
leftResolved, err := ResolvePipeline(left, ResolveOptions{}, newProfileCatalog(t))
if err != nil {
t.Fatalf("ResolvePipeline(left) error = %v", err)
}
rightResolved, err := ResolvePipeline(right, ResolveOptions{}, newProfileCatalog(t))
if err != nil {
t.Fatalf("ResolvePipeline(right) error = %v", err)
}
if leftResolved.Digest == rightResolved.Digest {
t.Fatal("validation policy change did not change resolved digest")
}
binding := ModuleBinding{Module: "producer", ValidationPolicy: validationPolicyOverride(ProducerStructuralFailureRejectOutput, SemanticRejectionRejectOutput, ValidatorFailureFailRun)}
encoded, err := json.Marshal(binding)
if err != nil {
t.Fatalf("marshal binding: %v", err)
}
var decoded ModuleBinding
if err := json.Unmarshal(encoded, &decoded); err != nil {
t.Fatalf("unmarshal binding: %v", err)
}
if decoded.ValidationPolicy == nil || *decoded.ValidationPolicy.ProducerStructuralFailure != ProducerStructuralFailureRejectOutput || *decoded.ValidationPolicy.SemanticRejection != SemanticRejectionRejectOutput || *decoded.ValidationPolicy.ValidatorFailure != ValidatorFailureFailRun {
t.Fatalf("round-tripped validation policy = %#v", decoded.ValidationPolicy)
}
cloned := cloneModuleBinding(binding)
*binding.ValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
if got := *cloned.ValidationPolicy.ValidatorFailure; got != ValidatorFailureFailRun {
t.Fatalf("cloned validation policy aliases binding: %q", got)
}
}
func validationPolicyOverride(structural ProducerStructuralFailureAction, semantic SemanticRejectionAction, validator ValidatorFailureAction) *ValidationPolicyOverride {
policy := &ValidationPolicyOverride{}
if structural != "" {
policy.ProducerStructuralFailure = &structural
}
if semantic != "" {
policy.SemanticRejection = &semantic
}
if validator != "" {
policy.ValidatorFailure = &validator
}
return policy
}

View File

@@ -95,6 +95,7 @@ func cloneModuleBinding(binding ModuleBinding) ModuleBinding {
binding.Module = strings.TrimSpace(binding.Module)
binding.LLMProfile = strings.TrimSpace(binding.LLMProfile)
binding.StructuredOutputRepairAttempts = cloneStructuredOutputRepairAttempts(binding.StructuredOutputRepairAttempts)
binding.ValidationPolicy = cloneValidationPolicyOverride(binding.ValidationPolicy)
binding.Options = cloneOptions(binding.Options)
if len(binding.References) > 0 {
references := make(map[string]ReferenceSource, len(binding.References))

View File

@@ -11,8 +11,9 @@ import (
)
type ValidatorSpec struct {
Key string `json:"key"`
ExecutionClass contracts.ExecutionClass `json:"execution_class"`
Key string `json:"key"`
ExecutionClass contracts.ExecutionClass `json:"execution_class"`
CorrectionProtocol contracts.CorrectionProtocol `json:"correction_protocol,omitempty"`
}
type SerializedValidatorSpec struct {
@@ -321,7 +322,11 @@ func (r *ValidatorRegistry) RegisteredKeys() []string {
}
func normalizeValidatorSpec(spec ValidatorSpec) (ValidatorSpec, error) {
normalized := ValidatorSpec{Key: strings.TrimSpace(spec.Key), ExecutionClass: spec.ExecutionClass}
normalized := ValidatorSpec{
Key: strings.TrimSpace(spec.Key),
ExecutionClass: spec.ExecutionClass,
CorrectionProtocol: contracts.CorrectionProtocol(strings.TrimSpace(string(spec.CorrectionProtocol))),
}
if normalized.Key == "" {
return ValidatorSpec{}, fmt.Errorf("validator key must not be empty")
}
@@ -330,6 +335,9 @@ func normalizeValidatorSpec(spec ValidatorSpec) (ValidatorSpec, error) {
default:
return ValidatorSpec{}, fmt.Errorf("validator %q execution class %q is not supported", normalized.Key, normalized.ExecutionClass)
}
if normalized.CorrectionProtocol != "" {
return ValidatorSpec{}, fmt.Errorf("validator %q correction protocol is not supported", normalized.Key)
}
return normalized, nil
}

View File

@@ -49,6 +49,7 @@ type Request struct {
ProfileID string
StructuredOutputRepairAttempts *int
SessionID string
Correction *contracts.SemanticCorrection
}
// ResultDisposition classifies a provider-neutral reconciliation outcome.
@@ -69,6 +70,7 @@ type Result struct {
issues []Issue
discardedGroupCount int
candidateMappings []CandidateMapping
modelCandidate *contracts.ModelCandidate
}
// Disposition returns the classified outcome.
@@ -88,6 +90,16 @@ func (result Result) CandidateMappings() []CandidateMapping {
return append([]CandidateMapping(nil), result.candidateMappings...)
}
// ModelCandidate returns an owned copy of the proposal response when a model
// completion produced this result.
func (result Result) ModelCandidate() *contracts.ModelCandidate {
candidate, err := contracts.CloneModelCandidate(result.modelCandidate)
if err != nil {
return nil
}
return candidate
}
func (result Result) planCopy() Plan {
return Plan{groups: result.plan.Groups()}
}
@@ -162,8 +174,12 @@ func (engine *Engine) Reconcile(ctx context.Context, request Request) (Result, e
if err := ctx.Err(); err != nil {
return Result{}, fmt.Errorf("semantic reconciliation %q: context error before completion: %w", request.StageName, err)
}
correction, err := contracts.CloneSemanticCorrection(request.Correction)
if err != nil {
return Result{}, fmt.Errorf("semantic reconciliation %q: clone correction: %w", request.StageName, err)
}
var response ProposalResponse
_, err = engine.client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
completion, err := engine.client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: request.StageName,
PromptID: engine.prompt.ID,
PromptVersion: engine.prompt.Version,
@@ -171,6 +187,7 @@ func (engine *Engine) Reconcile(ctx context.Context, request Request) (Result, e
SessionID: request.SessionID,
StructuredOutputRepairAttempts: request.StructuredOutputRepairAttempts,
Inputs: preparation.Materials(),
Correction: correction,
}, &response)
if err != nil {
if errors.Is(err, contracts.ErrInvalidStructuredOutput) {
@@ -179,6 +196,11 @@ func (engine *Engine) Reconcile(ctx context.Context, request Request) (Result, e
}
return Result{}, fmt.Errorf("semantic reconciliation %q: complete structured output: %w", request.StageName, err)
}
candidate, err := contracts.NewModelCandidate(completion.Content, contracts.CorrectionProtocolSingleResponseV1)
if err != nil {
return Result{}, fmt.Errorf("semantic reconciliation %q: own model candidate: %w", request.StageName, err)
}
result.modelCandidate = candidate
assessment := preparation.Assess(response)
result.plan = assessment.Plan()

View File

@@ -56,6 +56,11 @@ func TestEnginePropagatesRequestAndAssessesResponse(t *testing.T) {
request := readyEngineRequest()
request.ProfileID = " profile-as-resolved "
request.SessionID = " session-as-supplied "
correction, err := contracts.NewSemanticCorrection([]byte(`{"duplicate_groups":[]}`), "retain distinct candidates")
if err != nil {
t.Fatal(err)
}
request.Correction = correction
result, err := engine.Reconcile(context.Background(), request)
if err != nil {
@@ -75,24 +80,32 @@ func TestEnginePropagatesRequestAndAssessesResponse(t *testing.T) {
if got.StageName != request.StageName || got.PromptID != engine.prompt.ID || got.PromptVersion != engine.prompt.Version || got.ProfileID != request.ProfileID || got.SessionID != request.SessionID {
t.Fatalf("structured request = %#v, want exact routing values", got)
}
if !reflect.DeepEqual(got.Correction, correction) {
t.Fatalf("structured request correction = %#v, want %#v", got.Correction, correction)
}
if len(got.Inputs) != 2 || got.Inputs["candidates"].Name != "candidates" || got.Inputs["transcript"].Name != "transcript" || len(got.Vars) != 0 {
t.Fatalf("structured request inputs = %#v vars = %#v, want only candidate and transcript materials", got.Inputs, got.Vars)
}
candidate := result.ModelCandidate()
if candidate == nil || candidate.Protocol != contracts.CorrectionProtocolSingleResponseV1 || string(candidate.Response) != `{"duplicate_groups":[]}` {
t.Fatalf("model candidate = %#v, want exact owned completion response", candidate)
}
}
func TestEngineClassifiesSemanticAndTransportOutcomes(t *testing.T) {
transportErr := errors.New("provider unavailable")
tests := []struct {
name string
response ProposalResponse
completion error
want ResultDisposition
wantDiscard int
wantIssues bool
wantError error
name string
response ProposalResponse
completion error
want ResultDisposition
wantDiscard int
wantIssues bool
wantCandidate bool
wantError error
}{
{name: "empty groups complete", response: ProposalResponse{DuplicateGroups: []DuplicateGroup{}}, want: Complete},
{name: "discarded proposal retryable", response: ProposalResponse{DuplicateGroups: []DuplicateGroup{{CandidateIDs: []int{1, 99}, CanonicalCandidateID: 1}}}, want: RetryableDiscardedProposalGroups, wantDiscard: 1, wantIssues: true},
{name: "empty groups complete", response: ProposalResponse{DuplicateGroups: []DuplicateGroup{}}, want: Complete, wantCandidate: true},
{name: "discarded proposal retryable", response: ProposalResponse{DuplicateGroups: []DuplicateGroup{{CandidateIDs: []int{1, 99}, CanonicalCandidateID: 1}}}, want: RetryableDiscardedProposalGroups, wantDiscard: 1, wantIssues: true, wantCandidate: true},
{name: "invalid structured output retryable", completion: fmt.Errorf("decode response: %w", contracts.ErrInvalidStructuredOutput), want: RetryableInvalidStructuredOutput},
{name: "transport failure", completion: transportErr, wantError: transportErr},
}
@@ -112,6 +125,9 @@ func TestEngineClassifiesSemanticAndTransportOutcomes(t *testing.T) {
if result.Disposition() != test.want || result.DiscardedGroupCount() != test.wantDiscard || (len(result.Issues()) > 0) != test.wantIssues {
t.Fatalf("result = disposition %v discarded %d issues %#v", result.Disposition(), result.DiscardedGroupCount(), result.Issues())
}
if (result.ModelCandidate() != nil) != test.wantCandidate {
t.Fatalf("model candidate = %#v, want presence %t", result.ModelCandidate(), test.wantCandidate)
}
if len(client.requests) != 1 {
t.Fatalf("completion calls = %d, want one", len(client.requests))
}
@@ -141,6 +157,9 @@ func TestEngineSkipsDeterministicOutcomesWithoutCompletion(t *testing.T) {
if result.Disposition() != test.want || len(result.CandidateMappings()) != test.mappingLen {
t.Fatalf("result disposition = %v mappings = %#v", result.Disposition(), result.CandidateMappings())
}
if result.ModelCandidate() != nil {
t.Fatalf("model candidate = %#v, want nil for no-call outcome", result.ModelCandidate())
}
if len(client.requests) != 0 {
t.Fatalf("completion calls = %d, want zero", len(client.requests))
}
@@ -209,7 +228,9 @@ func TestEngineCallsAreIndependentAndResultsAreOwned(t *testing.T) {
firstIssues[0].Category = "changed"
firstMappings := first.CandidateMappings()
firstMappings[0].CandidatePosition = 99
if first.Plan().Groups()[0].MemberPositions()[0] != 0 || first.Issues()[0].Category == "changed" || first.CandidateMappings()[0].CandidatePosition != 0 {
firstCandidate := first.ModelCandidate()
firstCandidate.Response[0] = 'x'
if first.Plan().Groups()[0].MemberPositions()[0] != 0 || first.Issues()[0].Category == "changed" || first.CandidateMappings()[0].CandidatePosition != 0 || string(first.ModelCandidate().Response) != `{"duplicate_groups":[]}` {
t.Fatal("result accessors exposed retained state")
}
@@ -217,7 +238,7 @@ func TestEngineCallsAreIndependentAndResultsAreOwned(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if second.Disposition() != Complete || len(second.Plan().Groups()) != 0 || len(second.Issues()) != 0 || second.DiscardedGroupCount() != 0 || len(second.CandidateMappings()) != 2 {
if second.Disposition() != Complete || len(second.Plan().Groups()) != 0 || len(second.Issues()) != 0 || second.DiscardedGroupCount() != 0 || len(second.CandidateMappings()) != 2 || second.ModelCandidate() == nil {
t.Fatalf("second result retained prior call state: disposition %v plan %#v issues %#v discarded %d mappings %#v", second.Disposition(), second.Plan().Groups(), second.Issues(), second.DiscardedGroupCount(), second.CandidateMappings())
}
}
@@ -226,11 +247,15 @@ type recordingReconciliationClient struct {
requests []contracts.StructuredCompletionRequest
responses []ProposalResponse
errors []error
content []byte
}
func (client *recordingReconciliationClient) CompleteStructured(_ context.Context, request contracts.StructuredCompletionRequest, output any) (contracts.StructuredCompletionResponse, error) {
request.Inputs = request.Inputs.Clone()
client.requests = append(client.requests, request)
snapshot, err := contracts.CloneStructuredCompletionRequest(request)
if err != nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("clone recording request: %w", err)
}
client.requests = append(client.requests, snapshot)
index := len(client.requests) - 1
if index < len(client.errors) && client.errors[index] != nil {
return contracts.StructuredCompletionResponse{}, client.errors[index]
@@ -244,7 +269,11 @@ func (client *recordingReconciliationClient) CompleteStructured(_ context.Contex
return contracts.StructuredCompletionResponse{}, fmt.Errorf("output type = %T", output)
}
*target = response
return contracts.StructuredCompletionResponse{}, nil
content := append([]byte(nil), client.content...)
if len(content) == 0 {
content = []byte(`{"duplicate_groups":[]}`)
}
return contracts.StructuredCompletionResponse{Content: content}, nil
}
func cloneProposalResponse(response ProposalResponse) ProposalResponse {

View File

@@ -94,33 +94,40 @@ func (c *Chunker) Plan(ctx context.Context, req contracts.ChunkRequest) (contrac
return contracts.ChunkPlanResult{}, chunkerErrorf("validate source document: %w", err)
}
var response chunkResponse
if _, err := c.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
completion, err := c.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: Key,
PromptID: PromptID,
PromptVersion: ResponseSchemaVersion,
ProfileID: req.LLMProfile,
SessionID: req.SessionID,
StructuredOutputRepairAttempts: req.StructuredOutputRepairAttempts,
Correction: req.Correction,
Inputs: shared.PromptInputs(req.SourceInput, req.References),
}, &response); err != nil {
}, &response)
if err != nil {
return contracts.ChunkPlanResult{}, chunkerErrorf("complete structured output: %w", err)
}
candidate, err := shared.ModelCandidateFromResponse(completion)
if err != nil {
return contracts.ChunkPlanResult{}, chunkerErrorf("capture model candidate: %w", err)
}
plan, err := planFromResponse(req.Source, response)
if err != nil {
return contracts.ChunkPlanResult{}, chunkerErrorf("malformed structured output: %w", err)
}
return contracts.ChunkPlanResult{Plan: plan}, nil
return contracts.ChunkPlanResult{Plan: plan, ModelCandidate: candidate}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),
Key: Key,
Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassLLMBacked,
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),
}
}

View File

@@ -22,12 +22,13 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
}
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: []string{"source.transcript"},
Provides: []string{"chunks"},
ReferenceSlots: wantReferenceSlots(),
Key: Key,
Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassLLMBacked,
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
Requires: []string{"source.transcript"},
Provides: []string{"chunks"},
ReferenceSlots: wantReferenceSlots(),
}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
@@ -181,6 +182,13 @@ func TestPlanReturnsAnnotationFreeSceneRangesFromStructuredOutput(t *testing.T)
if len(result.Warnings) != 0 {
t.Fatalf("warnings = %#v, want absent", result.Warnings)
}
wantCandidate, err := json.Marshal(client.response)
if err != nil {
t.Fatalf("marshal expected candidate: %v", err)
}
if result.ModelCandidate == nil || result.ModelCandidate.Protocol != contracts.CorrectionProtocolSingleResponseV1 || !reflect.DeepEqual(result.ModelCandidate.Response, wantCandidate) {
t.Fatalf("model candidate = %#v, want exact validated response", result.ModelCandidate)
}
}
func TestPlanUsesDocumentOrderForNonconsecutiveUnitIDs(t *testing.T) {
@@ -533,6 +541,11 @@ func (client *fakeScenesLLMClient) CompleteStructured(ctx context.Context, req c
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
req.Inputs = req.Inputs.Clone()
correction, err := contracts.CloneSemanticCorrection(req.Correction)
if err != nil {
panic(err)
}
req.Correction = correction
req.Vars = cloneVars(req.Vars)
return req
}

View File

@@ -208,30 +208,37 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
var response extractionResponse
inputs := shared.PromptInputs(sourceInput, req.References)
inputs[NPCRegistryReferenceSlot] = npcRegistry.PromptInput()
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
completion, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: Key,
PromptID: PromptID,
PromptVersion: SchemaVersion,
ProfileID: req.LLMProfile,
SessionID: req.SessionID,
StructuredOutputRepairAttempts: req.StructuredOutputRepairAttempts,
Correction: req.Correction,
Inputs: inputs,
}, &response); err != nil {
}, &response)
if err != nil {
return contracts.TypedExtractionResult[dnd.CombatTurnList]{}, extractorErrorf("complete structured output: %w", err)
}
candidate, err := shared.ModelCandidateFromResponse(completion)
if err != nil {
return contracts.TypedExtractionResult[dnd.CombatTurnList]{}, extractorErrorf("capture model candidate: %w", err)
}
canonicalizeResponse(&response, order, req.Source.ID)
return contracts.TypedExtractionResult[dnd.CombatTurnList]{Value: canonicalCombatTurnList(response, req.Source.ID)}, nil
return contracts.TypedExtractionResult[dnd.CombatTurnList]{Value: canonicalCombatTurnList(response, req.Source.ID), ModelCandidate: candidate}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.CombatTurnListKind,
ReferenceSlots: referenceSlots(),
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.CombatTurnListKind,
ReferenceSlots: referenceSlots(),
}
}

View File

@@ -448,7 +448,7 @@ func TestExtractorManifestMetadataAndFingerprints(t *testing.T) {
func TestModuleSpecAndRegistration(t *testing.T) {
wantSlots := referenceSlots()
got := ModuleSpec()
if got.Key != Key || got.Stage != pipeline.StageExtract || got.ArtifactKind != dnd.CombatTurnListKind || !reflect.DeepEqual(got.Requires, []string{"chunks", "source.transcript"}) || !reflect.DeepEqual(got.Provides, []string{"dnd.combat_turns"}) || !reflect.DeepEqual(got.ReferenceSlots, wantSlots) {
if got.Key != Key || got.Stage != pipeline.StageExtract || got.CorrectionProtocol != contracts.CorrectionProtocolSingleResponseV1 || got.ArtifactKind != dnd.CombatTurnListKind || !reflect.DeepEqual(got.Requires, []string{"chunks", "source.transcript"}) || !reflect.DeepEqual(got.Provides, []string{"dnd.combat_turns"}) || !reflect.DeepEqual(got.ReferenceSlots, wantSlots) {
t.Fatalf("ModuleSpec() = %#v, want combat extractor contract", got)
}
got.Requires[0] = "changed"
@@ -592,6 +592,11 @@ func (client *fakeCombatTurnsLLMClient) CompleteStructured(_ context.Context, re
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
req.Inputs = req.Inputs.Clone()
correction, err := contracts.CloneSemanticCorrection(req.Correction)
if err != nil {
panic(err)
}
req.Correction = correction
if len(req.Vars) == 0 {
req.Vars = nil
return req

View File

@@ -142,19 +142,26 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
inputs[name] = input
}
var response extractionResponse
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
completion, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: Key,
PromptID: PromptID,
PromptVersion: SchemaVersion,
ProfileID: req.LLMProfile,
SessionID: req.SessionID,
StructuredOutputRepairAttempts: req.StructuredOutputRepairAttempts,
Correction: req.Correction,
Inputs: inputs,
}, &response); err != nil {
}, &response)
if err != nil {
return contracts.TypedExtractionResult[dnd.EnemyEventList]{}, extractorErrorf("complete structured output: %w", err)
}
candidate, err := shared.ModelCandidateFromResponse(completion)
if err != nil {
return contracts.TypedExtractionResult[dnd.EnemyEventList]{}, extractorErrorf("capture model candidate: %w", err)
}
return contracts.TypedExtractionResult[dnd.EnemyEventList]{
Value: canonicalEnemyEventList(response, shared.NewSourceRefOrder(req.Source), req.Source.ID),
Value: canonicalEnemyEventList(response, shared.NewSourceRefOrder(req.Source), req.Source.ID),
ModelCandidate: candidate,
}, nil
}
@@ -174,13 +181,14 @@ func unavailableSceneResult() contracts.TypedExtractionResult[dnd.EnemyEventList
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.EnemyEventListKind,
ReferenceSlots: referenceSlots(),
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.EnemyEventListKind,
ReferenceSlots: referenceSlots(),
}
}

View File

@@ -137,7 +137,7 @@ func TestConstructorSpecOptionsAndSafeMetadata(t *testing.T) {
first.Requires[0] = "changed"
first.ReferenceSlots[0].AcceptedMediaTypes[0] = "changed"
second := ModuleSpec()
if second.Requires[0] != "chunks" || second.ArtifactKind != dnd.EnemyEventListKind || second.ExecutionClass != contracts.ExecutionClassLLMBacked || second.ReferenceSlots[0].AcceptedMediaTypes[0] == "changed" {
if second.Requires[0] != "chunks" || second.ArtifactKind != dnd.EnemyEventListKind || second.ExecutionClass != contracts.ExecutionClassLLMBacked || second.CorrectionProtocol != contracts.CorrectionProtocolSingleResponseV1 || second.ReferenceSlots[0].AcceptedMediaTypes[0] == "changed" {
t.Fatalf("ModuleSpec() reused mutable state: %#v", second)
}
registry := pipeline.NewExtractorRegistry()

View File

@@ -155,28 +155,35 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
var response extractionResponse
inputs := shared.PromptInputs(sourceInput, req.References)
inputs[ItemRegistryReferenceSlot] = registry.PromptInput()
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
completion, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: Key, PromptID: PromptID, PromptVersion: SchemaVersion,
ProfileID: req.LLMProfile, SessionID: req.SessionID, StructuredOutputRepairAttempts: req.StructuredOutputRepairAttempts, Inputs: inputs,
}, &response); err != nil {
ProfileID: req.LLMProfile, SessionID: req.SessionID, StructuredOutputRepairAttempts: req.StructuredOutputRepairAttempts,
Correction: req.Correction, Inputs: inputs,
}, &response)
if err != nil {
return contracts.TypedExtractionResult[dnd.ItemOccurrenceList]{}, extractorErrorf("complete structured output: %w", err)
}
candidate, err := shared.ModelCandidateFromResponse(completion)
if err != nil {
return contracts.TypedExtractionResult[dnd.ItemOccurrenceList]{}, extractorErrorf("capture model candidate: %w", err)
}
value, err := canonicalItemOccurrenceList(response, order, req.Source.ID, registry)
if err != nil {
return contracts.TypedExtractionResult[dnd.ItemOccurrenceList]{}, extractorErrorf("map item occurrence response: %w", err)
}
return contracts.TypedExtractionResult[dnd.ItemOccurrenceList]{Value: value}, nil
return contracts.TypedExtractionResult[dnd.ItemOccurrenceList]{Value: value, ModelCandidate: candidate}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.ItemOccurrenceListKind,
ReferenceSlots: referenceSlots(),
Key: Key,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassLLMBacked,
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.ItemOccurrenceListKind,
ReferenceSlots: referenceSlots(),
}
}

View File

@@ -11,11 +11,15 @@ import (
func TestExtractGroundsOccurrencesInRequiredRegistry(t *testing.T) {
id := itemidentity.DeriveID("Torch")
client := &fakeItemOccurrencesLLMClient{response: extractionResponse{Occurrences: []itemOccurrenceResponse{
{Name: "Torch", Kind: "lost", From: "party", SourceRefs: responseRefs(1, 1)},
}}}
rawResponse := []byte(`{"occurrences":[{"name":"Torch","kind":"lost","from":"party","source_refs":[{"start_unit_id":1,"end_unit_id":1}]}]}`)
client := &fakeItemOccurrencesLLMClient{content: append([]byte(nil), rawResponse...)}
req := extractionRequest()
req.References = itemRegistryReferences(t)
correction, err := contracts.NewSemanticCorrection([]byte(`{"occurrences":[]}`), "Keep the transcript-grounded item occurrence.")
if err != nil {
t.Fatalf("NewSemanticCorrection() error = %v", err)
}
req.Correction = correction
result, err := newExtractor(t, client, req.References).Extract(context.Background(), req)
if err != nil {
t.Fatal(err)
@@ -26,10 +30,30 @@ func TestExtractGroundsOccurrencesInRequiredRegistry(t *testing.T) {
if refs := result.Value.Occurrences[0].SourceRefs; len(refs) != 1 || refs[0].SourceID != req.Source.ID || refs[0].StartUnitID != 1 || refs[0].EndUnitID != 1 {
t.Fatalf("occurrence evidence = %#v, want current-source unit range", refs)
}
input := client.requests[0].Inputs[ItemRegistryReferenceSlot]
request := client.requests[0]
input := request.Inputs[ItemRegistryReferenceSlot]
if input.Name != ItemRegistryReferenceSlot || string(input.Content) != `{"items":[{"name":"Torch"}]}` || strings.Contains(string(input.Content), "item:sha256:") {
t.Fatalf("registry prompt input = %#v, want names-only projection", input)
}
if request.Correction == nil || string(request.Correction.AssistantResponse) != `{"occurrences":[]}` || request.Correction.UserGuidance != "Keep the transcript-grounded item occurrence." {
t.Fatalf("correction = %#v, want exact request correction", request.Correction)
}
for _, material := range []string{string(request.Correction.AssistantResponse), request.Correction.UserGuidance} {
if strings.Contains(material, id) || strings.Contains(material, "item:sha256:") {
t.Fatalf("correction material leaked opaque item identity: %q", material)
}
}
correction.AssistantResponse[0] = '['
if got := string(request.Correction.AssistantResponse); got != `{"occurrences":[]}` {
t.Fatalf("captured correction changed after caller mutation: %q", got)
}
if result.ModelCandidate == nil || result.ModelCandidate.Protocol != contracts.CorrectionProtocolSingleResponseV1 || string(result.ModelCandidate.Response) != string(rawResponse) {
t.Fatalf("model candidate = %#v, want exact validated response", result.ModelCandidate)
}
client.content[0] = '['
if got := string(result.ModelCandidate.Response); got != string(rawResponse) {
t.Fatalf("model candidate changed after provider buffer mutation: %q", got)
}
}
func TestExtractCanonicalizesComparisonEquivalentNames(t *testing.T) {

View File

@@ -36,6 +36,9 @@ func testSourceRefs() []source.SourceRef {
func TestModuleSpecDeclaresRequiredRegistry(t *testing.T) {
spec := ModuleSpec()
if spec.CorrectionProtocol != contracts.CorrectionProtocolSingleResponseV1 {
t.Fatalf("correction protocol = %q, want %q", spec.CorrectionProtocol, contracts.CorrectionProtocolSingleResponseV1)
}
var slot contracts.ReferenceSlot
for _, candidate := range spec.ReferenceSlots {
if candidate.Name == ItemRegistryReferenceSlot {

View File

@@ -58,6 +58,11 @@ func newExtractor(t *testing.T, client contracts.StructuredLLMClient, references
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
req.Inputs = req.Inputs.Clone()
correction, err := contracts.CloneSemanticCorrection(req.Correction)
if err != nil {
panic(err)
}
req.Correction = correction
return req
}

View File

@@ -109,21 +109,28 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
order := shared.NewSourceRefOrder(req.Source)
var response extractionResponse
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
completion, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: Key, PromptID: PromptID, PromptVersion: SchemaVersion,
ProfileID: req.LLMProfile, SessionID: req.SessionID, StructuredOutputRepairAttempts: req.StructuredOutputRepairAttempts,
Inputs: shared.PromptInputs(sourceInput, req.References),
}, &response); err != nil {
Correction: req.Correction,
Inputs: shared.PromptInputs(sourceInput, req.References),
}, &response)
if err != nil {
return contracts.TypedExtractionResult[dnd.ItemRegistry]{}, extractorErrorf("complete structured output: %w", err)
}
candidate, err := shared.ModelCandidateFromResponse(completion)
if err != nil {
return contracts.TypedExtractionResult[dnd.ItemRegistry]{}, extractorErrorf("capture model candidate: %w", err)
}
canonicalizeResponse(&response, order, req.Source.ID)
return contracts.TypedExtractionResult[dnd.ItemRegistry]{Value: canonicalItemRegistry(response, req.Source.ID)}, nil
return contracts.TypedExtractionResult[dnd.ItemRegistry]{Value: canonicalItemRegistry(response, req.Source.ID), ModelCandidate: candidate}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...),
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.ItemRegistryKind, ReferenceSlots: referenceSlots(),
}
}

View File

@@ -19,7 +19,7 @@ func TestModuleRegistrationMetadataAndRedaction(t *testing.T) {
if _, err := New(&fakeItemsLLMClient{}, Options{}, contracts.ReferenceSet{}, contracts.ReferenceSet{}); err == nil || !strings.Contains(err.Error(), "at most one") {
t.Fatalf("New() error = %v, want reference-set rejection", err)
}
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_registry"}, ArtifactKind: dnd.ItemRegistryKind, ReferenceSlots: referenceSlots()}
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_registry"}, ArtifactKind: dnd.ItemRegistryKind, ReferenceSlots: referenceSlots()}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}

View File

@@ -88,5 +88,10 @@ func (client *fakeItemsLLMClient) CompleteStructured(ctx context.Context, req co
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
req.Inputs = req.Inputs.Clone()
correction, err := contracts.CloneSemanticCorrection(req.Correction)
if err != nil {
panic(err)
}
req.Correction = correction
return req
}

Some files were not shown because too many files have changed in this diff Show More