Compare commits
19 Commits
916d9210fd
...
e92bcfa74c
| Author | SHA1 | Date | |
|---|---|---|---|
| e92bcfa74c | |||
| e6b2ae88d2 | |||
| b2e83bd6e7 | |||
| 7f01c3e79e | |||
| 7b5f4ebd42 | |||
| 4bb4582695 | |||
| 7d3434e5f6 | |||
| 29872b2e28 | |||
| b487d93186 | |||
| 8dd7a4324d | |||
| 04ba87e174 | |||
| a26d6ed042 | |||
| 759d32403f | |||
| 1a7b20c766 | |||
| 85a5b52be7 | |||
| 9d0faabf61 | |||
| 1c3da3e869 | |||
| 9abd93502f | |||
| c8a29a5fa2 |
@@ -83,9 +83,7 @@ entity identity, checkpoint contracts, or cross-request correlation. Changes
|
||||
to shared protocol and policy assets must participate in the normal prompt,
|
||||
schema, and checkpoint fingerprint mechanisms.
|
||||
|
||||
Acceptance of this decision does not imply that the shared mechanism or its
|
||||
consumer migrations are implemented. The
|
||||
[feature roadmap](../roadmap/semantic-reconciliation.md) owns target behavior
|
||||
and status, and the
|
||||
[implementation plan](../roadmap/implementation.md) owns delivery sequence
|
||||
until the work is complete.
|
||||
The shared mechanism and its initial D&D registry consumers are now
|
||||
implemented. Current behavior is documented in
|
||||
[Module Internals](../internal/modules.md#semantic-reconciliation) and
|
||||
[D&D Module Internals](../internal/dnd.md#semantic-registry-reconciliation).
|
||||
|
||||
59
docs/adr/0014-feedback-aware-validation-retries.md
Normal file
59
docs/adr/0014-feedback-aware-validation-retries.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# ADR-0014: Use feedback-aware validation retries
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-08-26
|
||||
|
||||
## Context
|
||||
|
||||
Validation can identify a candidate defect after a producer has returned an
|
||||
otherwise well-formed result. Retrying without the validator's deterministic,
|
||||
bounded feedback wastes the useful diagnosis, while treating validator
|
||||
execution failures as defects would ask a producer to repair conditions it
|
||||
cannot control. The mechanism must preserve typed producer ownership,
|
||||
checkpoint safety, and the repository's sensitive-data boundaries.
|
||||
|
||||
## Decision
|
||||
|
||||
The implementation will keep three independent budgets: the producer binding's
|
||||
outer `retries` budget, PromptKit's structured-output repair budget, and each
|
||||
validator's execution-retry budget. Validators will run sequentially in their
|
||||
configured order and aggregate both rejections and execution failures before a
|
||||
candidate disposition is selected.
|
||||
|
||||
A correction-capable producer will provide the exact single LLM response that
|
||||
controlled its candidate using the `single_response_v1` protocol. A correction
|
||||
attempt will reconstruct the ordinary request and append exactly two fresh
|
||||
messages: that latest response as `assistant`, followed by one deterministic
|
||||
aggregate correction request as `user`. Earlier turns will not accumulate.
|
||||
|
||||
Validator failures will not recurse into correction. Pipeline policy owns
|
||||
terminal disposition, with field-by-field producer overrides over pipeline
|
||||
defaults: structural failure and semantic rejection default to `fail_run`, and
|
||||
validator execution failure defaults to `warn_continue`. Validators can report
|
||||
facts and bounded corrective guidance, but never decide disposition.
|
||||
|
||||
Rejected and structurally invalid candidates will not advance. A candidate
|
||||
allowed through after a validator execution failure will retain explicit
|
||||
incomplete-validation provenance and will not be checkpointed. Exact response
|
||||
and correction text remain attempt-local: they are excluded from ordinary
|
||||
errors, warnings, manifests, receipts, caches, checkpoints, and default debug
|
||||
summaries.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- Retry every producer after any validation outcome. This conflates producer
|
||||
defects with validator operational failures and wastes retry budget.
|
||||
- Let validators decide whether to continue. This would distribute pipeline
|
||||
disposition policy across validators and undermine consistent defaults.
|
||||
- Reuse the full prior conversation. Accumulated turns introduce unbounded
|
||||
prompt growth and make correction behavior depend on incidental history.
|
||||
- Persist raw responses to simplify diagnosis. Raw model output and correction
|
||||
guidance may be sensitive and do not belong in durable pipeline records.
|
||||
|
||||
## Consequences
|
||||
|
||||
The framework gains transport-neutral correction and candidate contracts,
|
||||
producer capability checks, policy resolution, aggregated validation outcomes,
|
||||
and conservative checkpoint handling. Prompt construction remains inside the
|
||||
LLM adapter, while modules remain responsible for accurately exposing the
|
||||
single response that directly controlled a candidate.
|
||||
@@ -132,7 +132,7 @@ model: example-model
|
||||
Keep credentials out of the local-backend object. A PromptKit profile may name
|
||||
its credential environment variable through `api_key_env`; set that variable
|
||||
only in the run environment. PromptKit owns the
|
||||
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/formats.md),
|
||||
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md),
|
||||
including `base_profile` inheritance. Notarius passes profiles through without
|
||||
merging them. Filesystem profiles cannot express PromptKit's in-memory
|
||||
`APIKeyRequired` setting; an unset `api_key_env` is optional and may reach the
|
||||
@@ -223,6 +223,7 @@ pipelines:
|
||||
| --- | --- | --- | --- |
|
||||
| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. An explicitly present blank value is invalid. |
|
||||
| **structured_output_repair_attempts** | integer | prompt-owned (1 in maintained production prompts) | Optional structural-repair limit from 0 through 3 for selected LLM-backed bindings and validators. Omission leaves the prompt's declared policy in control; explicit 0 disables structural repair at that scope. |
|
||||
| **validation_policy** | object | see below | Optional terminal policy defaults for producer validation. Its fields inherit independently into chunk, extract, merge, and normalize bindings. |
|
||||
| **input** | module binding | none | Required. |
|
||||
| **chunk** | module binding | **generic** | Optional. |
|
||||
| **output** | module binding | **json** | Optional. |
|
||||
@@ -251,6 +252,33 @@ explicit `null` and non-integer values are invalid. An explicit value on a
|
||||
deterministic binding or validator is invalid, while a pipeline value simply
|
||||
does not apply to deterministic selections.
|
||||
|
||||
`validation_policy` controls terminal disposition for one complete producer
|
||||
attempt and validator chain. It may appear on a pipeline or a **chunk**,
|
||||
**extract**, **merge**, or **normalize** module binding; input, output, and
|
||||
validator bindings reject it. Every field is optional and resolves in binding,
|
||||
pipeline, then application-default order:
|
||||
|
||||
| Field | Values | Default |
|
||||
| --- | --- | --- |
|
||||
| **producer_structural_failure** | **fail_run**, **reject_output** | **fail_run** |
|
||||
| **semantic_rejection** | **fail_run**, **reject_output** | **fail_run** |
|
||||
| **validator_failure** | **warn_continue**, **fail_run** | **warn_continue** |
|
||||
|
||||
The policy object and its fields must be non-null, and unknown fields are
|
||||
rejected. A deterministic producer may not explicitly set
|
||||
**producer_structural_failure** on its binding, although a pipeline-level
|
||||
default remains valid for pipelines that include LLM-backed producers.
|
||||
|
||||
After the producer binding's retry budget is exhausted, an invalid structured
|
||||
response uses **producer_structural_failure**. One or more semantic validator
|
||||
rejections use **semantic_rejection**; rejection takes precedence over an
|
||||
exhausted validator failure or skip. With no rejection, an exhausted validator
|
||||
failure or skip uses **validator_failure**. `reject_output` records the
|
||||
terminal rejection without advancing that candidate. `warn_continue` is valid
|
||||
only for validator execution failure: it advances a structurally valid,
|
||||
otherwise unrejected result with incomplete-validation provenance and without
|
||||
making it reusable checkpoint state.
|
||||
|
||||
A lane has these fields:
|
||||
|
||||
| Field | Type | Default | Rules |
|
||||
@@ -289,7 +317,8 @@ extract:
|
||||
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
|
||||
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
|
||||
| **structured_output_repair_attempts** | integer | pipeline or prompt-owned (1 in maintained production prompts) | Optional structural-repair limit from 0 through 3 for an LLM-backed binding. It overrides the pipeline value; explicit 0 disables structural repair. |
|
||||
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
|
||||
| **validation_policy** | object | pipeline or application defaults | Optional field-by-field terminal-policy override for a chunk, extract, merge, or normalize binding. |
|
||||
| **retries** | integer | 0 | Non-negative additional complete producer attempts for chunk, extract, merge, and normalize bindings. This single budget covers operational errors, invalid structured output, module-requested normalization retry, and semantic correction. |
|
||||
| **options** | object | none | Must satisfy the selected module. |
|
||||
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
||||
| **validators** | list | production chain | Valid only on chunk, extract, merge, and normalize bindings. |
|
||||
@@ -297,11 +326,23 @@ extract:
|
||||
Omitting **validators** uses the registered chain. **validators: []** selects
|
||||
an empty chain; a non-empty list replaces the chain in the listed order.
|
||||
Validator bindings accept only **module**, **llm_profile**,
|
||||
**structured_output_repair_attempts**, and **options**. They reject
|
||||
**references**, **retries**, and nested **validators**. Deterministic validators
|
||||
reject explicit **llm_profile** and **structured_output_repair_attempts**.
|
||||
**structured_output_repair_attempts**, **retries**, and **options**. Their
|
||||
**retries** value is a non-negative additional validator-execution budget and
|
||||
is valid only when the selected validator is LLM-backed. A validator retry
|
||||
rechecks the same immutable candidate; it never regenerates the producer.
|
||||
They reject
|
||||
**validation_policy**, **references**, and nested **validators**. Deterministic
|
||||
validators reject explicit **llm_profile** and
|
||||
**structured_output_repair_attempts**.
|
||||
Deterministic module bindings also reject those explicit fields.
|
||||
|
||||
An LLM-backed chunk, extract, merge, or normalize producer with both a
|
||||
non-empty validator chain and positive **retries** must declare the supported
|
||||
single-response correction capability. Preparation rejects a configuration
|
||||
that could require semantic correction from a producer that cannot provide an
|
||||
exact prior response. A deterministic producer, or an LLM attempt that did
|
||||
not make a model call, cannot consume a semantic retry after rejection.
|
||||
|
||||
The **json** output module accepts optional **include_chunk_map** and
|
||||
**evidence_context** settings:
|
||||
|
||||
|
||||
@@ -101,6 +101,12 @@ root. Do not scan the output root for its newest directory, guess a run ID, or
|
||||
construct a bundle path. Resolve `index_file` beneath `output_directory` and
|
||||
reject an absolute logical path or any result that escapes the bundle root.
|
||||
|
||||
The complete configuration uses the application validation defaults. A caller
|
||||
that requires fully validated D&D artifacts must also require receipt
|
||||
`validation_status: approved`; a successful `incomplete` result reflects the
|
||||
configured validator-failure continuation policy and carries its bounded
|
||||
validator provenance in `validation_summaries`.
|
||||
|
||||
Read `index.json` and locate each requested lane in `output_files` by its exact
|
||||
`lane_id`. Do not guess a lane filename. Before decoding a payload:
|
||||
|
||||
|
||||
@@ -69,6 +69,13 @@ lanes. The caller decides which lane IDs are required for its own work and
|
||||
which are optional; it should make that decision explicitly rather than infer
|
||||
failure from the receipt counts alone.
|
||||
|
||||
When complete validation is required, also require receipt
|
||||
`validation_status: approved` and inspect `validation_summaries`. A successful
|
||||
run with `validation_status: incomplete` contains a structurally valid result
|
||||
that advanced after validator execution could not complete under the configured
|
||||
`warn_continue` policy. It is not reusable checkpoint state and should not be
|
||||
silently treated as fully reviewed by the caller.
|
||||
|
||||
## Preserve Provenance And Handle Data Carefully
|
||||
|
||||
Keep the receipt with the published `manifest.json`, and retain
|
||||
|
||||
@@ -92,7 +92,7 @@ group into the following externally observable summaries:
|
||||
| Run identity and result | `run_id`, `pipeline_id`, `pipeline_digest`, `schema_version`, `validation_status`, `started_at`, `completed_at` |
|
||||
| Resolved components | `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, `output_encoder`, `artifact_lanes`, `validator_chains`, `module_metadata` |
|
||||
| Source and references | `source_digests`, `references` |
|
||||
| Published result summaries | `normalized_outputs`, `rejected_outputs` |
|
||||
| Published result summaries | `normalized_outputs`, `rejected_outputs`, `validation_summaries` |
|
||||
| Execution summaries | `chunk_plan`, `checkpoint_decisions`, `llm_profiles`, `metadata` |
|
||||
|
||||
`references` records provenance such as the target, slot, origin, digest,
|
||||
@@ -102,6 +102,16 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is
|
||||
provenance for the plan used by this run; cache records, debug artifacts, and
|
||||
other operational state are not published as bundle files.
|
||||
|
||||
Each `validation_summaries` entry is a bounded outcome for one producer result.
|
||||
It has required `status`, `producer_attempt_count`, and `terminal_action`;
|
||||
the stage and affected step, lane, module, or chunk identity are present when
|
||||
applicable. `status` is `complete`, `rejected`, or `incomplete`.
|
||||
`rejecting_validators`, `reason_codes`, and `incomplete_validators` preserve
|
||||
configured validator order and omit later duplicates. Entries contain no raw
|
||||
candidate response, correction guidance, validator diagnostic message, or
|
||||
artifact payload. The same shape may appear as `validation` on an affected
|
||||
rejection entry.
|
||||
|
||||
When present, `metadata.session_id` is the effective non-secret routing
|
||||
correlation identifier used for the run. It can be visible to providers and is
|
||||
not a substitute for a cache or checkpoint identity. Its generation and
|
||||
@@ -127,7 +137,10 @@ distinct even when their profile, provider, and model are otherwise equal.
|
||||
`rejected.json` is always an object with a `rejected` array. Each entry has
|
||||
required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`,
|
||||
`chunk_index`, `validator_name`, `reason_code`, `attempt_count`, and
|
||||
`diagnostic_artifact_path` are present only when applicable.
|
||||
`diagnostic_artifact_path` are present only when applicable. An entry may also
|
||||
contain the bounded `validation` summary described above; the existing singular
|
||||
validator and reason fields remain the first configured rejection for
|
||||
compatibility.
|
||||
|
||||
`warnings.json` is always an object with a `warnings` array. Each warning has
|
||||
`reason_code` and `message`; `scope` is optional. Both arrays are empty when
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# PromptKit Integration
|
||||
|
||||
Notarius pins
|
||||
[`gitea.maximumdirect.net/eric/promptkit` v0.8.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0)
|
||||
[`gitea.maximumdirect.net/eric/promptkit` v0.9.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0)
|
||||
as its in-process prompt engine. The upstream
|
||||
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/consumers/pkg-promptkit.md)
|
||||
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/consumers/pkg-promptkit.md)
|
||||
owns the public engine API, and the upstream
|
||||
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/formats.md)
|
||||
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md)
|
||||
owns prompt, profile, and schema file contracts.
|
||||
|
||||
## Supported Boundary
|
||||
@@ -15,7 +15,8 @@ Notarius relies on the root `promptkit` package to:
|
||||
- construct an `Engine` with filesystem-backed prompt, schema, and optional
|
||||
operator and application-fallback profile sources;
|
||||
- prepare one frozen execution from a `RunRequest` with named inline artifacts,
|
||||
variables, a direct session ID, prompt identity, and profile selection, then
|
||||
variables, a direct session ID, prompt identity, profile selection, and
|
||||
optional appended rendered messages, then
|
||||
record credential-redacted details and run that exact execution;
|
||||
- return rendered debug material, validated structured output, selected
|
||||
profile, backend, effective model metadata, and token usage;
|
||||
@@ -26,7 +27,7 @@ Notarius relies on the root `promptkit` package to:
|
||||
admission exhaustion through `ErrCapacityExceeded`.
|
||||
|
||||
The pinned
|
||||
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/backends.go)
|
||||
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/backends.go)
|
||||
owns the registration and backend-capacity contract.
|
||||
|
||||
For one completion, the adapter calls `PrepareExecution`, takes a
|
||||
@@ -52,7 +53,7 @@ Notarius sends one stable effective session through PromptKit's direct session
|
||||
field, which is authoritative for provider session behavior. It also retains
|
||||
the same value as the `session_id` prompt variable for maintained prompt
|
||||
compatibility. The generated identifier is 76 ASCII characters, within
|
||||
PromptKit v0.8.0's 256-code-point session limit. Session IDs are non-secret
|
||||
PromptKit v0.9.0's 256-code-point session limit. Session IDs are non-secret
|
||||
correlation identifiers and may be exposed to providers and provider
|
||||
observability. The CLI contract owns generation and override behavior.
|
||||
|
||||
@@ -85,12 +86,38 @@ configuration and deployment workflow are defined in
|
||||
[Operations](../operations.md#promptkit-profile-deployment).
|
||||
|
||||
PromptKit owns `base_profile` resolution under its
|
||||
[pinned format rules](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/formats.md).
|
||||
[pinned format rules](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md).
|
||||
Notarius records the selected leaf identity and resolved target without parsing
|
||||
or merging inheritance. An unset filesystem `api_key_env` is optional and may
|
||||
reach the provider without authorization, which can result in a 401 or 403.
|
||||
|
||||
Notarius supports this boundary against PromptKit v0.8.0. Its fallback source,
|
||||
PromptKit v0.9.0 accepts only the `developer`, `system`, `user`, and
|
||||
`assistant` text-chat roles after normalizing case and surrounding whitespace.
|
||||
Maintained Notarius prompt definitions use only `system` and `user`.
|
||||
|
||||
For application-owned semantic correction, Notarius uses PromptKit v0.9.0's
|
||||
`RunRequest.AppendedMessages` after the ordinary rendered prompt. It supplies
|
||||
exactly two messages in order: the latest validated producer response with
|
||||
role `assistant`, then deterministic validation guidance with role `user`.
|
||||
It never exposes a general caller-selected role API, accumulates earlier
|
||||
correction turns, or changes the ordinary prompt prefix. Ordinary requests
|
||||
leave appended messages unset.
|
||||
|
||||
PromptKit preserves supplied content but does not own Notarius's correction
|
||||
bounds. Notarius rejects invalid UTF-8, blank, or oversized assistant material
|
||||
(at most 1 MiB), guidance (at most 64 KiB), and combined content (at most
|
||||
1,114,112 bytes) before preparing the request. The transport-neutral
|
||||
application contract owns defensive copying and these limits. Default request
|
||||
and terminal summaries retain only safe counts, digests, identities, and usage;
|
||||
complete appended messages remain limited to the explicitly requested detailed
|
||||
debug trace.
|
||||
|
||||
PromptKit now obtains its maintained OpenRouter and Rakestrawhome backend and
|
||||
profile catalogs from independently versioned transitive modules. Notarius
|
||||
does not import or register either catalog; PromptKit retains catalog source,
|
||||
identity, precedence, credential, and capacity ownership.
|
||||
|
||||
Notarius supports this boundary against PromptKit v0.9.0. Its fallback source,
|
||||
prepared-execution, inspection, and typed capacity APIs are used as public
|
||||
upstream contracts; other PromptKit APIs or file-format behavior are not
|
||||
implicitly supported. A dependency upgrade requires reviewing the adapter,
|
||||
|
||||
@@ -22,11 +22,15 @@ The current schema version is `notarius.run-result.v1`.
|
||||
| `rejected_output_count` | Yes | Number of recorded rejected outputs. |
|
||||
| `warning_count` | Yes | Number of final run warnings. |
|
||||
| `validation_status` | Yes | The final run manifest validation status. |
|
||||
| `validation_summaries` | No | Bounded per-producer validation outcomes; present when producer work ran. |
|
||||
| `debug_directory` | No | Absolute path to the run-specific debug bundle when requested debug capture completed. |
|
||||
|
||||
For the production `json` output module, `index_file` is present only when the
|
||||
completed run returned exactly one logical output file named `index.json`.
|
||||
For another output module, its absence does not indicate a failed run.
|
||||
`validation_status` is `approved`, `rejected`, or `incomplete`; `incomplete`
|
||||
means one or more otherwise accepted results advanced under validator-failure
|
||||
`warn_continue`.
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -38,7 +42,17 @@ For another output module, its absence does not indicate a failed run.
|
||||
"normalized_output_count": 6,
|
||||
"rejected_output_count": 2,
|
||||
"warning_count": 1,
|
||||
"validation_status": "rejected"
|
||||
"validation_status": "incomplete",
|
||||
"validation_summaries": [
|
||||
{
|
||||
"stage": "extract",
|
||||
"lane_id": "spells",
|
||||
"status": "incomplete",
|
||||
"incomplete_validators": ["dnd/spells/source_refs"],
|
||||
"producer_attempt_count": 1,
|
||||
"terminal_action": "warn_continue"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
@@ -48,8 +62,11 @@ For another output module, its absence does not indicate a failed run.
|
||||
paths. They identify the paths used by Notarius and do not resolve symlinks.
|
||||
`output_directory` is the run-specific bundle, not the configured output root.
|
||||
|
||||
The receipt is a summary and discovery document. It does not contain lane
|
||||
descriptors, payloads, manifest data, rejections, warnings, or file contents.
|
||||
The receipt is a summary and discovery document. Its optional validation
|
||||
summaries contain only stable status, identity, validator names, reason codes,
|
||||
attempt counts, and terminal actions. It does not contain lane descriptors,
|
||||
payloads, manifest payloads, rejection messages, warnings, raw model responses,
|
||||
correction guidance, or file contents.
|
||||
For the production JSON output, resolve `index_file` beneath
|
||||
`output_directory`, reject path escapes, and use the
|
||||
[Published JSON Output contract](json-output.md) to discover logical files and
|
||||
|
||||
@@ -74,10 +74,19 @@ profile-free, and no second inheritance decision occurs during execution. The
|
||||
public field definitions and precedence are owned by
|
||||
[Configuration](../config.md#pipelines).
|
||||
|
||||
The resolver retains configured `validation_policy` overrides and derives one
|
||||
detached concrete terminal policy for the chunk producer and every lane's
|
||||
extract, merge, and normalize producers. That field-by-field inheritance is
|
||||
complete before preparation, and the effective values contribute to pipeline
|
||||
and checkpoint identity; execution does not interpret configuration defaults.
|
||||
|
||||
The framework resolver supplies defaults, selects lanes, resolves validator
|
||||
chains, checks registered module and artifact compatibility, validates module
|
||||
options, and returns the fixed ordered pipeline shape. The resulting
|
||||
**EffectiveConfig** retains the selected ID, requested selection and reference
|
||||
options, and returns the fixed ordered pipeline shape. Positive validator retry
|
||||
budgets require an LLM-backed selected validator; deterministic validators are
|
||||
rejected during resolution. Eligible LLM-backed producer specifications also
|
||||
contribute their declared correction protocol to the resolved metadata. The
|
||||
resulting **EffectiveConfig** retains the selected ID, requested selection and reference
|
||||
changes, a clone of the input configuration, and the resolved pipeline.
|
||||
Callers may therefore retain or modify their input slices and maps without
|
||||
changing the resolved result, and later consumers cannot mutate the original
|
||||
@@ -94,8 +103,8 @@ runtime error class described in the [CLI reference](../cli.md#output-streams-an
|
||||
|
||||
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
||||
after defaults, lane selection, module bindings, reference bindings, validator
|
||||
chains, effective LLM profiles, and artifact schema identity have been
|
||||
resolved. The digest excludes
|
||||
chains, selected correction protocols, effective LLM profiles, and artifact
|
||||
schema identity have been resolved. The digest excludes
|
||||
its own stored value. It identifies resolved composition rather than raw YAML
|
||||
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
||||
provenance before execution; cache and checkpoint identity have additional
|
||||
|
||||
@@ -33,7 +33,9 @@ typed builder. Scene chunking, every extractor, and NPC, location, and item-regi
|
||||
normalization are registered as `llm_backed`; the remaining current D&D mergers
|
||||
and normalizers are `deterministic`. The metadata is available to catalog inspection and
|
||||
resolved-pipeline debug data and determines which selected bindings inherit the
|
||||
pipeline profile. Configuration remains the canonical owner of the exact keys,
|
||||
pipeline profile. The registry normalizers use `single_response_v1`, forwarding
|
||||
corrections to their reconciliation completion and retaining the accepted raw
|
||||
proposal only as an owned model candidate. Configuration remains the canonical owner of the exact keys,
|
||||
profile precedence, and validator order.
|
||||
|
||||
Private structured-LLM response schemas are deliberately minimal. They reject
|
||||
@@ -42,6 +44,13 @@ unknown fields, while preserving semantic candidates for deterministic
|
||||
validation. Do not promote a private response envelope into a durable schema;
|
||||
the contracts above define durable data.
|
||||
|
||||
A D&D producer that declares `single_response_v1` forwards any supplied
|
||||
semantic correction to its structured completion and returns an owned copy of
|
||||
that completion's exact validated raw response as its model candidate. It does
|
||||
not serialize normalized artifacts to create that candidate, so deterministic
|
||||
identity, evidence, warning, and durable-schema behavior remains separate from
|
||||
the model transport material.
|
||||
|
||||
## Prompt Construction
|
||||
|
||||
D&D LLM-facing content lives beneath `assets/dnd/`. Each module contributes a
|
||||
@@ -125,6 +134,13 @@ relatedness validators report advisory evidence concerns. The configured order
|
||||
is documented in
|
||||
[Configuration](../config.md#production-validator-keys-and-default-chains).
|
||||
|
||||
Every D&D rejection describes the correction in transcript-grounded domain
|
||||
terms, using contextual names, artifact fields, and source segment ranges when
|
||||
useful. The guidance must not ask the model to reproduce durable entity IDs,
|
||||
hashes, validator module keys, or reason codes. Those identifiers remain in
|
||||
ordinary validation provenance; only the actionable semantic guidance is
|
||||
eligible for the correction prompt.
|
||||
|
||||
Enemy-event extraction additionally rejects a second `engaged` observation for
|
||||
the same comparison identity within one scene-scoped result. Normalization may
|
||||
combine results from distinct scenes, so it intentionally does not apply that
|
||||
|
||||
@@ -42,6 +42,22 @@ observability. The adapter returns PromptKit’s validated raw bytes rather than
|
||||
re-encoding the decoded target. An empty optional material is represented as
|
||||
one space so its named input is retained by PromptKit.
|
||||
|
||||
When a request includes semantic correction material, the adapter validates and
|
||||
defensively copies it before preparation, then appends exactly two messages
|
||||
after the ordinarily rendered prompt: the prior response as an assistant
|
||||
message and the correction guidance as a user message. Requests without a
|
||||
correction do not add messages or introduce caller roles. Ordinary request
|
||||
summaries record correction byte counts and digests only; complete messages are
|
||||
available solely in an explicitly requested debug trace.
|
||||
|
||||
The adapter leaves the ordinary rendered message prefix, named inputs,
|
||||
variables, session, profile, execution overrides, prepared-execution path, and
|
||||
PromptKit repair policy unchanged for a corrected request. It never imports a
|
||||
PromptKit message type into a module or pipeline contract. PromptKit reports
|
||||
actual structural repair count and cumulative token usage per completion; the
|
||||
pipeline's safe terminal debug record projects those values without copying
|
||||
message content.
|
||||
|
||||
Client construction may also receive a run-wide reasoning-effort override from
|
||||
the CLI factory boundary. The adapter copies the caller-owned pointer and
|
||||
creates a fresh PromptKit execution override for each request: a nil pointer
|
||||
@@ -229,10 +245,11 @@ or corrective call remain provider-neutral operational errors with the same
|
||||
redaction boundary.
|
||||
|
||||
Structural repair does not replace pipeline retry behavior: a binding's
|
||||
configured retry count reruns its complete stage attempt after an error or
|
||||
rejection. The pipeline owns attempt lifecycle, validation chains, and retry
|
||||
diagnostics; see [Pipeline Internals](pipeline.md#validation-retries-and-output)
|
||||
and the [binding reference](../config.md#module-bindings-and-validators).
|
||||
configured retry count reruns its complete stage attempt after an operational
|
||||
or structural error, module-requested retry, or actionable semantic rejection.
|
||||
The pipeline owns attempt lifecycle, validation chains, and retry diagnostics;
|
||||
see [Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
||||
[binding reference](../config.md#module-bindings-and-validators).
|
||||
|
||||
## Timeout Ownership
|
||||
|
||||
@@ -262,6 +279,13 @@ surfaced; when the completion already failed, its call error remains the
|
||||
result. Debug-bundle location, retention, and handling are operational concerns
|
||||
documented in [Operations](../operations.md#debug-bundles).
|
||||
|
||||
The attempt-terminal summary is a separate safe trace record: it contains
|
||||
attempt kinds, validator outcome counts and reason codes, effective policy,
|
||||
terminal action, and repair/usage references. It excludes raw assistant
|
||||
responses and correction text. Those values can appear only in the explicitly
|
||||
requested detailed prompt and response artifacts, which require sensitive-data
|
||||
handling.
|
||||
|
||||
Run manifests receive selected profile summaries, including optional effective
|
||||
backend and reasoning provenance, and component identities—not prompt, schema,
|
||||
source, reference, or response content. The published field semantics belong
|
||||
|
||||
@@ -22,6 +22,15 @@ to bindings whose declared execution class is `llm_backed` and rejects a
|
||||
binding-specific profile on a deterministic module. The user-facing precedence
|
||||
contract belongs in [Configuration](../config.md#pipelines).
|
||||
|
||||
An eligible LLM-backed chunk, extract, merge, or normalize producer may also
|
||||
declare correction protocol `single_response_v1`. That declaration is a
|
||||
promise that the implementation accepts one attempt-local semantic correction
|
||||
and returns an owned copy of the exact one model response that directly
|
||||
controlled the candidate. It must forward correction only to its structured
|
||||
completion request; it must not manufacture prior-response material by
|
||||
serializing a normalized artifact or expose opaque application IDs. Input,
|
||||
output, validator, and deterministic specs cannot declare the protocol.
|
||||
|
||||
Implementations that accept options must provide both an option validator and
|
||||
a builder. The validator is used while resolving configuration; the builder
|
||||
decodes the same options and constructs the implementation from the prepared
|
||||
@@ -36,6 +45,15 @@ they need, register each leaf implementation, and add any family-owned assets
|
||||
or default validator chains. They return contextual errors so production
|
||||
composition fails at startup rather than at the first run.
|
||||
|
||||
A validator that returns a completed rejection must supply two separate
|
||||
bounded values: a stable `ReasonCode` for provenance and actionable
|
||||
`CorrectionGuidance` for the producer. Guidance identifies the semantic defect
|
||||
and the constraints on one complete corrected replacement. It must not contain
|
||||
validator keys, diagnostic paths, opaque application IDs, or other internal
|
||||
identifiers. An operator-facing `Message` may explain the same event, but the
|
||||
framework never copies it into a model request. Missing or invalid guidance is
|
||||
a validator contract failure.
|
||||
|
||||
An artifact family can register an optional typed evidence projector alongside
|
||||
its codec. The projector returns defensive copies of the artifact's direct
|
||||
generic source references and must use the codec's exact Go type. It does not
|
||||
@@ -91,6 +109,12 @@ combined-material bound preserves the deterministic result under the family's
|
||||
fallback policy. Provider, transport, cancellation, and context-construction
|
||||
failures remain execution errors.
|
||||
|
||||
When the engine actually makes a proposal call, its typed result carries the
|
||||
owned exact proposal response under the same correction contract as other
|
||||
eligible producers. Deterministic skip, limit, and fallback outcomes carry no
|
||||
model candidate, so a later rejection applies terminal policy without spending
|
||||
an ineffective semantic retry.
|
||||
|
||||
The core supplies a conservative generic prompt and the single private
|
||||
response schema. A domain prompt may substitute its semantic instructions but
|
||||
mounts the core-owned protocol and candidate/transcript presentation assets.
|
||||
@@ -111,9 +135,13 @@ its domain prompt.
|
||||
3. Implement strict option decoding, construction, and the typed stage
|
||||
interface. Preserve caller ownership: do not retain mutable request data
|
||||
and return defensive copies where an implementation exposes stored data.
|
||||
If declaring correction capability, forward the request correction and
|
||||
retain only the exact validated response that controlled the result.
|
||||
4. Register the module through its typed registry helper and add it to the
|
||||
owning family registrar. Add a default validator chain only when that
|
||||
family owns the behavior; otherwise require an explicit compatible chain.
|
||||
Every rejection path in a validator must provide actionable correction
|
||||
guidance while retaining its stable internal reason code.
|
||||
5. Update the selectable-key and chain reference in
|
||||
[Configuration](../config.md#production-module-keys), the applicable
|
||||
integration contract, and focused tests. Keep the configuration document
|
||||
|
||||
@@ -32,8 +32,14 @@ Resolution turns a configured pipeline profile into a **ResolvedPipeline**.
|
||||
It normalizes the pipeline and lane identities, applies stage defaults, selects
|
||||
requested lanes where that is supported, resolves validator chains, checks
|
||||
module capabilities and typed artifact compatibility, validates options, and
|
||||
assigns a deterministic resolved-composition digest. The resolved pipeline
|
||||
contains bindings and declared reference targets, not external reference bytes.
|
||||
assigns a deterministic resolved-composition digest. A correction protocol is
|
||||
selected from each eligible LLM-backed producer specification and becomes part
|
||||
of that resolved identity; only `single_response_v1` is currently supported.
|
||||
Preparation rejects an LLM-backed producer that combines a non-empty validator
|
||||
chain with positive producer retries unless it declares that protocol. Producers
|
||||
without validators or without retries remain valid without correction support.
|
||||
The resolved pipeline contains bindings and declared reference targets, not
|
||||
external reference bytes.
|
||||
After selection, the resolver applies command, binding, and pipeline profile
|
||||
precedence to LLM-backed bindings and validators only; prompt defaults remain
|
||||
an empty resolved binding profile. It resolves structural output repair
|
||||
@@ -59,9 +65,11 @@ remains declared but has no bytes until its producing step completes.
|
||||
|
||||
Preparation is the construction boundary. It validates the resolved shape and
|
||||
registry set, clones the resolved data, then constructs the input adapter,
|
||||
chunker, stage-local validators, every typed lane, and output encoder. Each
|
||||
registered builder receives its own cloned build request immediately before its
|
||||
module-owned code runs. Preparation also collects stable checkpoint
|
||||
chunker, stage-local validators, every typed lane, and output encoder. The
|
||||
prepared producer metadata preserves each selected correction protocol, and
|
||||
the resolved digest carrying that metadata participates in checkpoint identity.
|
||||
Each registered builder receives its own cloned build request immediately
|
||||
before its module-owned code runs. Preparation also collects stable checkpoint
|
||||
fingerprints. Missing registrations, incompatible typed entries, nil
|
||||
implementations, and constructor failures are reported before source parsing
|
||||
or any stage operation begins.
|
||||
@@ -128,14 +136,45 @@ their target: chunks, codec-decoded typed candidates, or serialized codec
|
||||
bytes. Each typed validator receives a newly decoded value from the one
|
||||
candidate serialization for that attempt, while serialized validators receive
|
||||
separately owned representation bytes and schema metadata. They may approve,
|
||||
approve with warnings, reject, or fail. A rejection is an ordinary pipeline
|
||||
result; a validator error is a framework error.
|
||||
approve with warnings, reject, fail, or be skipped when a runtime prerequisite
|
||||
is unavailable. The shared executor settles every configured validator in
|
||||
order. A failed LLM-backed validator retries only itself against the same
|
||||
immutable candidate; it does not regenerate the producer or alter the
|
||||
validator request. Rejections stop that validator, while other configured
|
||||
validators still run. The executor retains ordered results, bounded
|
||||
deduplicated correction guidance from rejections, and only the final exhausted
|
||||
failure outcome for each validator. The correction builder keeps first
|
||||
occurrence order, omits internal reason codes, validator names, and operator
|
||||
messages, and requests one complete replacement. Missing guidance or an
|
||||
oversized aggregate is a framework contract error; guidance is never inferred
|
||||
or truncated.
|
||||
|
||||
The runner applies the binding's retry policy around a stage operation and its
|
||||
complete validation chain. It preserves warnings only from the final accepted
|
||||
or rejected attempt. Cancellation stops retries. Normalizer-specific retry
|
||||
directives consume this same budget and validate any final safe fallback through
|
||||
the normalizer chain.
|
||||
or rejected attempt, plus one fixed warning per validator whose execution
|
||||
budget was exhausted under `warn_continue`. Cancellation stops retries.
|
||||
Normalizer-specific retry directives consume this same budget and validate any
|
||||
final safe fallback through the normalizer chain.
|
||||
|
||||
The artifact-neutral producer-attempt state machine owns that shared budget,
|
||||
attempt provenance, semantic-correction material, and terminal-policy
|
||||
selection. It accepts producer and complete-validation closures, so artifact
|
||||
materialization, cache handling, checkpoints, and debug output stay at the
|
||||
operation boundary. It distinguishes operational, structural, module-requested,
|
||||
and semantic retries. A semantic retry is available only for a valid latest
|
||||
`single_response_v1` candidate; a deterministic or no-model rejection instead
|
||||
settles the semantic policy immediately. Structural-output errors alone use the
|
||||
structural policy, and validation failure without rejection settles the
|
||||
validator-failure policy without regenerating the producer.
|
||||
|
||||
Chunk planning uses this state machine for generated plans. A rejected or
|
||||
validation-incomplete automatic cache hit is not model material and therefore
|
||||
falls through to a fresh initial generation at producer attempt one; it neither
|
||||
receives a correction, consumes retry budget, promotes cached-candidate
|
||||
warnings, nor overwrites the stored record. An incomplete cache validation
|
||||
under `fail_run` terminates instead. Only a newly generated, completely
|
||||
validated plan is published to the chunk-plan store. Rejected plans never
|
||||
advance, and validation-incomplete plans remain unpublishable.
|
||||
|
||||
After terminal lane work, the runner assembles manifest provenance, normalized
|
||||
artifacts, rejections, warnings, and an optional accepted chunk map. When an
|
||||
@@ -149,6 +188,15 @@ The CLI publishes those files only after the runner returns without a framework
|
||||
error. Logical file names and schemas are defined by the [output integration
|
||||
contracts](../integrations/).
|
||||
|
||||
For every completed producer disposition, the runner projects one bounded
|
||||
validation summary to the manifest, the affected rejection when present, and
|
||||
the CLI result receipt. The summary records status, configured-order rejecting
|
||||
validators and reason codes, incomplete validators, producer-attempt count,
|
||||
and terminal action. It contains no operator message, correction guidance, or
|
||||
model response. `complete`, `rejected`, and `incomplete` describe the final
|
||||
candidate disposition; a run-level `incomplete` status indicates at least one
|
||||
current-run output advanced under `warn_continue`.
|
||||
|
||||
## Checkpoint And Debug Hooks
|
||||
|
||||
The runner receives checkpoint and debug interfaces rather than roots. It
|
||||
@@ -158,6 +206,20 @@ handoff. Generated-reference dependencies participate in checkpoint decisions.
|
||||
Selective recomputation can require a canonical accepted normalized predecessor
|
||||
before a dependent lane starts.
|
||||
|
||||
The runner writes successful checkpoint artifacts only after complete accepted
|
||||
validation. Chunk plans follow the same rule for publication. A rejection,
|
||||
invalid structured response, or incomplete validation is never reusable state;
|
||||
the current run may still hand off an otherwise valid `warn_continue` result
|
||||
according to its terminal policy. The runner carries private reuse eligibility
|
||||
through extract, merge, normalize, and generated-reference handoff. Any stage
|
||||
derived from incomplete validation skips both checkpoint lookup and all
|
||||
checkpoint publication even when that stage's own validation completes.
|
||||
External references and fully validated generated references remain eligible.
|
||||
Attempt debug records retain safe kind,
|
||||
validator, repair-usage, policy, and terminal-decision provenance. Full
|
||||
assistant and correction content remains confined to the requested detailed
|
||||
LLM trace.
|
||||
|
||||
Debug recording is attempt-scoped and application-owned. A failure to persist
|
||||
required debug data is a framework error. State roots, persistence, reason-code
|
||||
meanings, resume, and cleanup are intentionally owned by
|
||||
|
||||
@@ -109,6 +109,33 @@ On success, the command reports the output bundle path. A warning-bearing run
|
||||
still succeeds and reports its warning count on standard error. Errors and
|
||||
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).
|
||||
|
||||
## Validation Retries And Terminal Outcomes
|
||||
|
||||
Each producer binding has one outer **retries** budget. It covers complete
|
||||
producer attempts for operational failures, invalid structured output,
|
||||
normalizer fallback retry, and semantic correction. It is independent from
|
||||
PromptKit's structural-repair calls inside one completion and from an
|
||||
LLM-backed validator's own retry budget. A semantic correction rebuilds the
|
||||
ordinary producer request and supplies only the latest rejected model response
|
||||
plus aggregated validator guidance; it is not a conversation replay.
|
||||
|
||||
After the applicable budgets are exhausted, the resolved
|
||||
[`validation_policy`](config.md#pipelines) determines the result. Structural
|
||||
failure and semantic rejection normally fail the run; an explicit
|
||||
`reject_output` records a rejection and allows unrelated work to finish. A
|
||||
validator execution failure normally uses `warn_continue`, which keeps an
|
||||
otherwise accepted result in the current run with `incomplete` validation
|
||||
provenance. It emits one bounded warning for every validator whose execution
|
||||
budget was exhausted. A corrected result that later passes validation does not
|
||||
retain abandoned-attempt warnings.
|
||||
|
||||
Treat a successful process exit as a completed run, not as proof that every
|
||||
candidate was fully validated. Inspect the receipt's `validation_status`,
|
||||
`validation_summaries`, rejection count, and warning count when an orchestrator
|
||||
requires complete validation. The durable fields and their meanings are owned
|
||||
by the [run-result receipt](integrations/run-result.md) and
|
||||
[published JSON output contract](integrations/json-output.md).
|
||||
|
||||
## Output Bundles
|
||||
|
||||
Each successful run receives a generated safe run identifier and writes beneath:
|
||||
@@ -158,7 +185,9 @@ The configured cache mode controls one invocation:
|
||||
A reused plan is still materialized and validated against the current source.
|
||||
If a prior plan no longer gives acceptable results, use a refresh run rather
|
||||
than editing cache files. Deleting a plan is recoverable but can repeat costly
|
||||
chunking work.
|
||||
chunking work. A plan accepted only under incomplete validation is not
|
||||
published, and a rejected cache hit falls through to ordinary generation rather
|
||||
than becoming a correction candidate.
|
||||
|
||||
## Checkpoint Recording, Resume, And Recompute
|
||||
|
||||
@@ -183,6 +212,12 @@ Reasoning-effort inheritance, replacement, and explicit clearing are distinct
|
||||
runtime identities, so checkpoints created under one state are not reused by
|
||||
either of the others.
|
||||
|
||||
Only accepted, completely validated chunk, extract, merge, and normalize
|
||||
results are checkpointed for reuse. Rejected, structurally invalid, and
|
||||
validation-incomplete producer results remain non-reusable, even when a
|
||||
`warn_continue` result advanced during its original run. A resumed invocation
|
||||
therefore reruns that producer rather than treating degraded state as accepted.
|
||||
|
||||
Checkpoint state is confined below an identity-specific path:
|
||||
|
||||
~~~
|
||||
@@ -237,13 +272,16 @@ Only a [debug-enabled run](cli.md#run) creates a bundle:
|
||||
~~~
|
||||
|
||||
The summary contains redacted invocation and resolution information plus run,
|
||||
warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace
|
||||
contains allowlisted application diagnostic records and can include source or
|
||||
derived application data. Neither surface is a cache input. Do not treat a
|
||||
debug bundle as safe to share merely because its configuration summary is
|
||||
redacted. Invocation metadata omits reasoning effort when it is inherited,
|
||||
records the replacement value when one is supplied, and records an empty value
|
||||
when inherited reasoning was explicitly cleared.
|
||||
warning, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt
|
||||
terminal records contain bounded attempt kinds, validator outcomes, policy,
|
||||
decision, PromptKit repair count, and usage; they do not contain assistant
|
||||
responses or complete correction messages. The trace contains allowlisted
|
||||
application diagnostic records and can include source, model, and correction
|
||||
content. Neither surface is a cache input. Do not treat a debug bundle as safe
|
||||
to share merely because its configuration summary is redacted. Invocation
|
||||
metadata omits reasoning effort when it is inherited, records the replacement
|
||||
value when one is supplied, and records an empty value when inherited reasoning
|
||||
was explicitly cleared.
|
||||
|
||||
Notarius never creates debug state without an explicit request and never
|
||||
automatically deletes a requested bundle. If allocation succeeds, the command
|
||||
|
||||
@@ -167,18 +167,29 @@ starting, waits for started work, and prevents output encoding.
|
||||
## Validation
|
||||
|
||||
Validation is a framework-managed boundary around outputs from chunk, extract,
|
||||
merge, and normalize stages. Validators receive immutable stage output
|
||||
and make an explicit whole-output decision: approve, approve with warnings, or
|
||||
reject.
|
||||
merge, and normalize stages. Validators receive immutable stage output and
|
||||
make an explicit whole-output decision: approve, approve with warnings,
|
||||
reject, fail, or skip when a runtime prerequisite is unavailable.
|
||||
|
||||
Typed artifact validators receive the domain value directly. Chunk validators
|
||||
receive source-zone chunks, while serialized validators receive immutable
|
||||
representation bytes and declared schema metadata. A validator registered for
|
||||
one target or artifact kind cannot satisfy an incompatible selection.
|
||||
|
||||
Rejection is a recorded pipeline outcome, not a framework execution error.
|
||||
Validator execution failures are framework errors. Rejected output does not
|
||||
advance to the next stage.
|
||||
The framework runs every applicable validator sequentially in configured order.
|
||||
It aggregates rejections, exhausted validator failures, and skips before the
|
||||
producer policy chooses a disposition. A completed rejection never advances.
|
||||
With no rejection, an exhausted validator failure may fail the run or, under
|
||||
the configured `warn_continue` policy, advance a structurally valid candidate
|
||||
with explicit incomplete-validation provenance. Validators report findings;
|
||||
they do not choose candidate disposition.
|
||||
|
||||
A completed rejection supplies a stable reason code for internal provenance
|
||||
and bounded actionable correction guidance for the candidate producer. Reason
|
||||
codes, validator keys, and operator-facing messages remain diagnostic data;
|
||||
they are not model instructions. The framework constructs model-facing retry
|
||||
text only from the semantic guidance and fails the contract rather than
|
||||
inventing or truncating missing guidance.
|
||||
|
||||
Default validator chains are production composition policy and are registered
|
||||
centrally by stage and module. Configuration may replace a stage-local default,
|
||||
@@ -199,6 +210,15 @@ PromptKit owns bounded structural correction within one structured completion.
|
||||
Notarius owns outer stage attempts, semantic validation, and acceptance policy;
|
||||
the two budgets must remain separate.
|
||||
|
||||
An LLM-backed producer can participate in semantic correction only when it
|
||||
declares `single_response_v1` and returns the exact one response that directly
|
||||
controlled its candidate. On an actionable rejection, the framework rebuilds
|
||||
the ordinary request and appends only the latest defective response as an
|
||||
`assistant` message plus one aggregated `user` correction message. This is a
|
||||
fresh replacement request, not a growing conversation. The retry budgets,
|
||||
terminal policy, and sensitive-data rationale are recorded in
|
||||
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md).
|
||||
|
||||
When a model selects an application entity, callers must supply a contextual
|
||||
selection and deterministically attach the opaque application identity whenever
|
||||
the selection resolves exactly. Models do not receive or reproduce opaque
|
||||
@@ -233,7 +253,8 @@ invalid or incompatible.
|
||||
|
||||
Run manifests record enough resolved pipeline, module, source, reference, and
|
||||
LLM provenance to make a run auditable after configuration changes. Manifests
|
||||
record identities and summaries rather than secret or large payload content.
|
||||
record identities and bounded validation summaries rather than secret, raw
|
||||
model, correction, or large payload content.
|
||||
|
||||
## State, Output, And Safety
|
||||
|
||||
@@ -253,12 +274,23 @@ an invocation that explicitly requests resume. Debug is never a cache input and
|
||||
is never created without an explicit request. Pipeline modules receive
|
||||
collaborator interfaces and never physical roots.
|
||||
|
||||
Only accepted, completely validated producer output is reusable checkpoint or
|
||||
chunk-plan state. Rejected, structurally invalid, and validation-incomplete
|
||||
results cannot become cache or checkpoint inputs, even when a
|
||||
`warn_continue` result is allowed to advance in the current run. This
|
||||
ineligibility follows derived merge and normalize results and generated
|
||||
references for the remainder of the run: current-run handoff remains allowed,
|
||||
but no dependent cache or checkpoint may be loaded or published.
|
||||
|
||||
Writes are atomic where practical. Paths for writes, moves, overwrites, and
|
||||
deletion must be narrow and explicit. Notarius never automatically deletes
|
||||
output or requested debug bundles; cache cleanup is explicit and recoverable.
|
||||
|
||||
Secrets must not appear in errors, logs, output, cache, debug summaries,
|
||||
traces, manifests, documentation, examples, or redacted configuration. Debug
|
||||
traces, manifests, documentation, examples, or redacted configuration. Raw
|
||||
assistant responses and complete correction messages are attempt-local and are
|
||||
excluded from ordinary durable records and summaries; the requested detailed
|
||||
debug trace is the sole diagnostic surface allowed to retain them. Debug
|
||||
collection is allowlisted to application-owned payloads and must not capture
|
||||
unrelated process environment values or filesystem content. Trace data may
|
||||
contain application data and therefore inherits its sensitivity; operators own
|
||||
|
||||
83
docs/releases/v0.4.0.md
Normal file
83
docs/releases/v0.4.0.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# Notarius v0.4.0
|
||||
|
||||
This release strengthens LLM reliability and validation throughout the
|
||||
configured pipeline, upgrades the PromptKit integration, and establishes the
|
||||
source-release and downstream-consumer workflows needed for broader D&D
|
||||
pipeline integration.
|
||||
|
||||
## Summary
|
||||
|
||||
Notarius now distinguishes PromptKit structural-output repair from
|
||||
application-owned semantic validation retries. Producer candidates can run
|
||||
through complete deterministic validator chains, receive bounded semantic
|
||||
correction guidance, and retry under explicit stage policies. Final run
|
||||
receipts and manifests preserve bounded validation provenance, while outputs
|
||||
that advance with incomplete validation remain available to the current run
|
||||
without entering reusable checkpoint state.
|
||||
|
||||
The release also adds a maintained complete D&D subprocess-consumer workflow,
|
||||
diagnostic build versions, and the source-only release procedure used to
|
||||
publish this version.
|
||||
|
||||
## Compatibility
|
||||
|
||||
- Configuration files must use schema version 4. Version 3 is not decoded or
|
||||
rewritten; rename the top-level `scriptorium` section to `promptkit` when
|
||||
migrating. See [Configuration](../config.md#migrating-version-3-configuration).
|
||||
- PromptKit is pinned to v0.9.0. Operator profile files use PromptKit's v0.9.0
|
||||
format and may use its profile-inheritance support. Notarius continues to
|
||||
resolve operator profiles before embedded fallbacks.
|
||||
- Structural-output repair and semantic stage retries are separate bounded
|
||||
mechanisms. Maintained production prompts request one structural repair by
|
||||
default; explicit configuration can override the supported repair count.
|
||||
- Validation policy can now fail a run, reject an output, or permit an
|
||||
otherwise valid candidate to advance with incomplete-validation provenance.
|
||||
The application defaults are documented in
|
||||
[Configuration](../config.md#pipelines).
|
||||
- The `notarius.run-result.v1` receipt remains at schema version 1 and adds
|
||||
optional validation summaries plus a required validation-status field.
|
||||
Consumers of this pre-release contract should follow the current
|
||||
[run-result receipt](../integrations/run-result.md).
|
||||
- Existing D&D artifact schema identities remain unchanged. Validation and
|
||||
producer-policy changes can nevertheless cause previously accepted weak
|
||||
candidates to retry, reject, or fail instead.
|
||||
|
||||
## Upgrade
|
||||
|
||||
1. Migrate every Notarius configuration to version 4 and rename `scriptorium`
|
||||
to `promptkit`.
|
||||
2. Review deployed PromptKit profiles against the pinned v0.9.0 profile format
|
||||
and ensure their credential environment variables are available at run
|
||||
time.
|
||||
3. Run `notarius config validate --config <path> --pipeline <id>` before the
|
||||
first production invocation.
|
||||
4. Review `structured_output_repair_attempts`, producer retry counts, and
|
||||
`validation_policy` wherever the deployment needs behavior different from
|
||||
the documented defaults.
|
||||
5. Update subprocess consumers to inspect receipt `validation_status` and to
|
||||
tolerate the optional bounded `validation_summaries` field. A consumer that
|
||||
requires fully validated artifacts should require `approved`.
|
||||
|
||||
## Changes
|
||||
|
||||
- Upgraded PromptKit from v0.5.0 through v0.9.0 and adopted profile
|
||||
inheritance, structured-output repair, typed error classification, and the
|
||||
correction-aware completion protocol.
|
||||
- Added pipeline and binding configuration for structural repair and terminal
|
||||
validation policy, with strict startup validation and effective-setting
|
||||
provenance.
|
||||
- Added feedback-aware retries for chunking, extraction, merge, normalize, and
|
||||
semantic reconciliation producers. Retry prompts contain the exact defective
|
||||
response and actionable semantic correction guidance without exposing
|
||||
internal reason codes or opaque entity identifiers.
|
||||
- Added complete validator-chain execution, validator retry handling, bounded
|
||||
warnings, terminal dispositions, and durable validation summaries.
|
||||
- Prevented validation-incomplete artifacts and all derived lineage from
|
||||
loading or publishing reusable checkpoints while preserving same-run
|
||||
generated-reference handoff.
|
||||
- Tightened cached chunk-plan validation so only completely validated plans are
|
||||
reused or replace stored plans.
|
||||
- Added a machine-readable subprocess receipt workflow and complete D&D
|
||||
consumer documentation covering all maintained artifacts.
|
||||
- Added source-release checks, immutable lightweight-tag guidance, Linux and
|
||||
Darwin build verification, and diagnostic `notarius --version` output.
|
||||
@@ -1,224 +0,0 @@
|
||||
# D&D Subprocess Consumer Documentation
|
||||
|
||||
## Status
|
||||
|
||||
Completed. The target guide is `docs/consumers/dnd-pipeline.md`.
|
||||
|
||||
## Purpose
|
||||
|
||||
Provide one task-oriented guide for applications that run Notarius as a
|
||||
subprocess to execute the maintained complete D&D pipeline and consume its
|
||||
published artifacts. The initial concrete consumer is Narratio, but the guide
|
||||
must describe the public Notarius workflow rather than depend on Narratio
|
||||
internals.
|
||||
|
||||
The guide should make the safe integration path obvious without duplicating
|
||||
the CLI, input, receipt, output-bundle, or individual artifact contracts that
|
||||
already have canonical documentation.
|
||||
|
||||
## Current State
|
||||
|
||||
The public integration surface is documented accurately but is distributed
|
||||
across several documents:
|
||||
|
||||
- `docs/consumers/subprocess.md` defines the generic subprocess workflow;
|
||||
- `docs/cli.md` owns commands, flags, stream behavior, and exit statuses;
|
||||
- `docs/integrations/seriatim.md` owns the accepted transcript input shape;
|
||||
- `docs/integrations/run-result.md` owns the machine-readable successful-run
|
||||
receipt;
|
||||
- `docs/integrations/json-output.md` owns bundle discovery and logical files;
|
||||
- the D&D integration documents own the individual lane payload contracts;
|
||||
- `examples/dnd-complete.config.yml` is the maintained complete pipeline.
|
||||
|
||||
A consumer can reconstruct the full workflow from those documents, but there
|
||||
is no D&D-focused guide that connects the maintained example to its input,
|
||||
invocation, complete artifact inventory, discovery procedure, and downstream
|
||||
acceptance decisions.
|
||||
|
||||
## Target Documentation Set
|
||||
|
||||
### Create `docs/consumers/dnd-pipeline.md`
|
||||
|
||||
This document should own the end-to-end consumer workflow for the maintained
|
||||
complete D&D configuration. It should be useful to Narratio and to another
|
||||
subprocess orchestrator with the same needs.
|
||||
|
||||
The guide should contain the following sections.
|
||||
|
||||
#### Prerequisites And Deployment Configuration
|
||||
|
||||
- Link to `examples/dnd-complete.config.yml` rather than embedding a second
|
||||
complete configuration.
|
||||
- Explain that a deployment must provide the configured PromptKit profile and
|
||||
campaign reference files.
|
||||
- Recommend absolute paths for a service or orchestrator deployment.
|
||||
- Call out the path-resolution distinction explicitly: YAML reference paths
|
||||
are relative to the Notarius configuration file, while
|
||||
`promptkit.profile_file` is relative to the Notarius process working
|
||||
directory.
|
||||
- Recommend validating the selected configuration and `dnd-session` pipeline
|
||||
before processing sessions.
|
||||
|
||||
#### Transcript Input
|
||||
|
||||
- State that the complete pipeline consumes a Seriatim JSON document.
|
||||
- Link to the canonical Seriatim contract for required fields and validation.
|
||||
- Recommend the caller's final trimmed transcript when the caller maintains
|
||||
transcript tiers. For Narratio, identify the implemented source as
|
||||
`narratio.transcript.final_trimmed`, normally stored at
|
||||
`transcripts/final.trimmed.json`.
|
||||
- Explain that segment IDs must remain stable because D&D source references
|
||||
cite those units.
|
||||
- Explain that Notarius derives its default prompt session from the input
|
||||
module and exact input bytes and that ordinary callers should not supply
|
||||
`--session-id`.
|
||||
|
||||
#### Subprocess Invocation
|
||||
|
||||
- Show one concise invocation using `notarius run dnd-session`, explicit
|
||||
absolute `--config`, `--input`, and `--output-dir` paths, and `--json`.
|
||||
- Direct callers to capture stdout and stderr separately, propagate
|
||||
cancellation, impose an operator-appropriate timeout, and wait for process
|
||||
completion before parsing stdout.
|
||||
- State that only exit status zero permits receipt decoding and link to the CLI
|
||||
contract for the complete exit-status definition.
|
||||
- Recommend retaining stderr and the invocation context for diagnosis without
|
||||
logging secrets or transcript content.
|
||||
|
||||
#### Receipt And Bundle Discovery
|
||||
|
||||
- Require callers to accept only supported run-result schema versions while
|
||||
tolerating unknown fields allowed by that version.
|
||||
- Direct callers to obtain the exact run-specific bundle from the receipt's
|
||||
absolute `output_directory`; they must not scan for the newest run directory
|
||||
or construct a run ID.
|
||||
- Require a confinement check when resolving `index_file` beneath the reported
|
||||
bundle root.
|
||||
- Direct callers to discover lane payloads by `lane_id` in `index.json`, then
|
||||
verify descriptor media type and schema identity before decoding them.
|
||||
- Explain that descriptor paths are untrusted relative paths and require the
|
||||
same confinement discipline.
|
||||
|
||||
#### Complete D&D Artifact Inventory
|
||||
|
||||
Include a compact table for the ten lane IDs selected by the maintained
|
||||
complete configuration:
|
||||
|
||||
- `item-registry`;
|
||||
- `npc-registry`;
|
||||
- `location-registry`;
|
||||
- `scene-descriptions`;
|
||||
- `item-occurrences`;
|
||||
- `spells`;
|
||||
- `combat-turns`;
|
||||
- `npc-occurrences`;
|
||||
- `location-occurrences`;
|
||||
- `enemy-events`.
|
||||
|
||||
For each row, give a one-line purpose and link to the corresponding canonical
|
||||
D&D artifact contract. Do not copy its fields or schema rules into the
|
||||
consumer guide.
|
||||
|
||||
Document the four always-published bundle files—`index.json`, `manifest.json`,
|
||||
`rejected.json`, and `warnings.json`—and the complete example's configured
|
||||
`chunk-map.json` and `evidence-context.json` pipeline-wide artifacts. Link to
|
||||
their canonical contracts and distinguish pipeline-wide artifacts from lane
|
||||
outputs.
|
||||
|
||||
The inventory must say that a file is available only when its corresponding
|
||||
artifact was accepted and published. It must not imply that process success
|
||||
guarantees every configured lane.
|
||||
|
||||
#### Downstream Acceptance And Retention
|
||||
|
||||
- Explain that exit status zero can coexist with rejected outputs, warnings,
|
||||
or absent lane descriptors.
|
||||
- Require the consumer to define its required lane set explicitly. Recommend
|
||||
treating all ten lanes as required when the caller claims to consume the
|
||||
complete D&D workflow, while allowing another consumer to adopt a narrower
|
||||
documented policy.
|
||||
- Recommend retaining the receipt, the complete published bundle, and captured
|
||||
diagnostic streams long enough to support provenance and failure analysis.
|
||||
- Explain that `evidence-context.json` is a reading excerpt; authoritative
|
||||
citations remain in lane payloads.
|
||||
- Treat transcripts, lane artifacts, evidence context, manifests, and logs as
|
||||
sensitive campaign data.
|
||||
|
||||
#### Compatibility Checklist
|
||||
|
||||
End with a concise checklist covering process exit, receipt schema, path
|
||||
confinement, pipeline identity, index decoding, required descriptors,
|
||||
descriptor schema/media compatibility, warnings and rejections, checksums or
|
||||
retention, and secure handling. Compatibility should be based on published
|
||||
receipt and artifact contracts rather than parsing a human version string.
|
||||
|
||||
### Update Existing Navigation
|
||||
|
||||
- Add a short link from `docs/consumers/subprocess.md` to the D&D-specific
|
||||
workflow. Keep generic subprocess policy in the existing document.
|
||||
- Add the guide to the documentation links in `README.md`.
|
||||
- Extend the subprocess-consumer row in `docs/development.md` so maintainers
|
||||
working on the D&D workflow are routed to the new guide and the canonical
|
||||
contracts.
|
||||
|
||||
### Verify Canonical Contract Documents
|
||||
|
||||
Review the linked integration documents and the complete example while writing
|
||||
the guide. Correct an integration document only if repository inspection finds
|
||||
an actual stale contract. Do not move schema definitions, field tables, CLI
|
||||
flags, or configuration semantics into the new guide.
|
||||
|
||||
## Narratio Alignment
|
||||
|
||||
The guide may name Narratio as the motivating consumer and identify its current
|
||||
final-trimmed transcript source. It must not claim that Narratio already has a
|
||||
Notarius adapter or extraction stage. Until that feature is implemented,
|
||||
Narratio-specific architecture, configuration, stage behavior, manifest
|
||||
records, and artifact source IDs belong in Narratio's roadmap.
|
||||
|
||||
Once Narratio implements the integration, its own integration documentation
|
||||
should link to this guide and the durable Notarius contracts instead of
|
||||
repeating them.
|
||||
|
||||
## Validation
|
||||
|
||||
Documentation implementation should include:
|
||||
|
||||
```sh
|
||||
go run ./cmd/notarius config validate \
|
||||
--config examples/dnd-complete.config.yml \
|
||||
--pipeline dnd-session
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Also verify all new and changed relative Markdown links, compare the artifact
|
||||
inventory directly with the maintained complete configuration, and confirm
|
||||
that commands and path semantics match the CLI and configuration references.
|
||||
If the repository still has no automated link checker, record that fact and
|
||||
perform a focused manual link review.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A subprocess integrator can follow one D&D-focused guide from a Seriatim
|
||||
transcript through safe discovery of every artifact configured by the
|
||||
complete example.
|
||||
- The guide makes stdout, stderr, exit-status, receipt, and path-confinement
|
||||
responsibilities unambiguous.
|
||||
- The ten configured D&D lanes and both configured pipeline-wide artifacts are
|
||||
listed and linked to their canonical contracts.
|
||||
- The guide distinguishes process success from the caller's required-artifact
|
||||
policy.
|
||||
- The profile-path and reference-path resolution rules are clearly stated.
|
||||
- Existing navigation makes the guide discoverable.
|
||||
- No volatile contract is defined in two places, and no unimplemented Narratio
|
||||
behavior is presented as current.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Implementing or documenting Narratio's future adapter or stage as current
|
||||
Notarius behavior.
|
||||
- Adding a new Notarius command, receipt version, output format, or artifact
|
||||
schema.
|
||||
- Duplicating the complete configuration or individual D&D payload schemas in
|
||||
prose.
|
||||
- Defining a universal partial-result policy for every Notarius consumer.
|
||||
@@ -7,130 +7,13 @@ not as committed release dates.
|
||||
|
||||
## Near-Term Validation And LLM Reliability
|
||||
|
||||
The following work forms one related program but should be promoted into
|
||||
separate feature roadmaps and implemented in dependency order. PromptKit owns
|
||||
structural output repair within one completion. Notarius owns stage candidates,
|
||||
validator chains, semantic rejection policy, and whether another stage attempt
|
||||
is warranted.
|
||||
PromptKit now owns structural output repair within one completion. Notarius
|
||||
owns stage candidates, validator chains, semantic rejection policy, bounded
|
||||
feedback-aware stage retries, validation provenance, and reusable-state
|
||||
eligibility. The remaining near-term work applies those completed foundations
|
||||
to domain review and operator-facing diagnostics.
|
||||
|
||||
### 1. Upgrade To PromptKit v0.8.0
|
||||
|
||||
This item has been promoted to the standalone
|
||||
[PromptKit v0.8.0 Upgrade](promptkit-v0.8.md) roadmap. That document owns the
|
||||
release-by-release compatibility review, adopted features, structured-repair
|
||||
policy, target integration boundary, acceptance criteria, and settled design
|
||||
decisions.
|
||||
|
||||
### 2. Feedback-Aware Stage Validation Retries
|
||||
|
||||
- Model Notarius's corrective stage-retry conversation explicitly after
|
||||
PromptKit v0.8.0. The first attempt sends the ordinary complete initial
|
||||
prompt. If application validation rejects the resulting LLM-produced
|
||||
candidate and another stage attempt is available, reconstruct that complete
|
||||
initial prompt byte-for-byte and append exactly two messages: an assistant
|
||||
message containing the defective response and an application-owned user
|
||||
message detailing every applicable semantic validation error and requesting
|
||||
one corrected, complete replacement response. This is a freshly constructed
|
||||
correction request, not continuation of an accumulating conversation.
|
||||
- Use the configured stage `retries` value as the one outer retry budget for
|
||||
this loop. `retries: N` continues to mean at most `N` additional complete
|
||||
chunk, extract, merge, or normalize attempts after the initial attempt,
|
||||
whether an attempt is needed because of a producer error or semantic
|
||||
rejection. Do not add a second semantic-correction count. PromptKit's
|
||||
prompt-level `repair_attempts` budget is independent and internal to each
|
||||
individual LLM completion, and does not consume or replenish the Notarius
|
||||
stage budget.
|
||||
- Extend the framework-managed validation boundary for chunk, extract, merge,
|
||||
and normalize stages so a rejected LLM-produced candidate and its exact raw
|
||||
model response remain available to construct the next stage attempt.
|
||||
Deterministic producers cannot improve by repeating the same inputs; a
|
||||
rejection from a deterministic stage is therefore terminal under the
|
||||
configured rejection policy rather than consuming retries mechanically.
|
||||
- Preserve the original session ID, selected profile, structured-output
|
||||
contract, prompt inputs, and reusable prompt prefix. Carry only the latest
|
||||
candidate and latest aggregate feedback; do not build an unbounded retry
|
||||
conversation. Keep model-facing corrective guidance separate from
|
||||
operator-facing diagnostics, and apply explicit size, redaction, and debug
|
||||
disclosure rules to both.
|
||||
- Run every applicable validator in the configured chain before deciding
|
||||
whether to retry. Do not short-circuit merely because an earlier validator
|
||||
rejected the candidate. Aggregate all semantic rejection reason codes and
|
||||
corrective guidance into the retry message so one retry can address the
|
||||
whole candidate. A validator is applicable only when its declared target and
|
||||
prerequisites can be satisfied; record a deterministic skipped diagnostic
|
||||
rather than invoking a validator on an input it cannot interpret. Initially
|
||||
execute the chain sequentially in configured order so results, diagnostics,
|
||||
costs, and feedback ordering remain deterministic; consider validator
|
||||
concurrency only in response to measured latency.
|
||||
- Continue running independent applicable validators after one validator
|
||||
execution failure so the attempt retains as much useful diagnostic
|
||||
information as practical. Do not present validator operational failures as
|
||||
defects in the producer candidate and do not include them in corrective
|
||||
feedback.
|
||||
- Distinguish three terminal conditions and make their policies configurable
|
||||
at a coherent pipeline or binding scope:
|
||||
- **producer structural failure:** PromptKit could not return a usable
|
||||
structured candidate after its repair budget. Default to `fail_run`; an
|
||||
allowed alternative may record a terminal stage or lane rejection where
|
||||
execution can safely continue, but may not accept the invalid output;
|
||||
- **semantic rejection:** one or more validators completed and rejected the
|
||||
candidate. Default to `fail_run` after corrective stage retries are
|
||||
exhausted; allow an explicit alternative that records the existing
|
||||
rejected-output outcome without advancing that output;
|
||||
- **validator execution failure:** a validator could not produce a valid
|
||||
decision because of generation, structural-output, transport, or internal
|
||||
failure. Default to a genuine warning and an explicitly recorded
|
||||
`validation_incomplete` or equivalent degraded state while allowing the
|
||||
candidate to continue; allow strict configuration to fail the run instead.
|
||||
- An LLM-backed validator uses the same scheduled PromptKit boundary as every
|
||||
other LLM-backed module. Its own response may use PromptKit's bounded
|
||||
structural repair. Distinguish its possible output states:
|
||||
- output rejected by PromptKit's structural contract should consume only the
|
||||
validator prompt's configured PromptKit repair budget;
|
||||
- output that is structurally valid but violates a deterministically
|
||||
checkable validator-result invariant should be classified as a validator
|
||||
execution failure;
|
||||
- output that satisfies the complete validator-result contract is the
|
||||
validator's decision, even though an LLM judgment may remain imperfect.
|
||||
Automatically judging that judgment would require another semantic
|
||||
validator and is outside this feature.
|
||||
If the validator cannot return a contract-valid decision, do not recursively
|
||||
create another Notarius semantic-validation loop around it. Apply the
|
||||
configured validator-failure policy. The default warning must identify the
|
||||
validator and affected stage without exposing sensitive content.
|
||||
- Separate validator execution retry from producer correction. A transient
|
||||
validator operational failure must not automatically discard and regenerate
|
||||
an otherwise usable producer candidate. Any bounded retry of the validator
|
||||
itself should reuse that same immutable candidate and remain subordinate to
|
||||
PromptKit and provider retry behavior.
|
||||
- Preserve attempt-level provenance, cumulative token usage, validator
|
||||
outcomes, aggregated correction feedback, and terminal policy decisions in
|
||||
the debug and manifest models without copying raw source material into
|
||||
ordinary errors or durable summaries.
|
||||
- Define terminal-outcome precedence. A semantic rejection dominates a
|
||||
validator execution failure for the same candidate: use the completed
|
||||
rejections to correct the producer while separately recording incomplete
|
||||
validation. If a later candidate has no semantic rejection but one validator
|
||||
still fails, apply the configured validator-failure policy to that candidate.
|
||||
Never allow a known semantic rejection to become accepted through a
|
||||
warn-and-continue setting, and never accept a structurally invalid producer
|
||||
response. Permissive policy may preserve a rejected-output outcome or accept
|
||||
a structurally valid candidate with explicitly incomplete validation; it may
|
||||
not relabel known-invalid output as approved.
|
||||
|
||||
Before implementation, record the generic validation and retry state machine
|
||||
in an ADR. The ADR should own the separation between PromptKit repair and
|
||||
Notarius correction, use of the existing stage-retry budget, reconstruction of
|
||||
correction conversations, all-applicable-validator aggregation, deterministic
|
||||
validator ordering, non-recursive validator failure handling, outcome
|
||||
precedence, default fail-open/fail-closed choices, configurable terminal
|
||||
policies, and provenance and sensitive-data constraints. A dependency-upgrade
|
||||
ADR is not needed for PromptKit v0.8.0 itself. Current behavior remains
|
||||
authoritative until the validation ADR is implemented and the canonical
|
||||
architecture, configuration, operations, and internal documentation are
|
||||
updated.
|
||||
|
||||
### 3. D&D Combat Scene Semantic Validation
|
||||
### D&D Combat Scene Semantic Validation
|
||||
|
||||
- Add an optional production LLM-backed D&D validator that determines whether
|
||||
proposed scene boundaries and classifications represent substantive active
|
||||
@@ -172,7 +55,7 @@ updated.
|
||||
the chunker or otherwise changes stage ownership or the durable chunk-plan
|
||||
contract.
|
||||
|
||||
### 4. Warning Signal And Presentation Reform
|
||||
### Warning Signal And Presentation Reform
|
||||
|
||||
- Audit every warning producer and representative successful runs. Ordinary
|
||||
success producing dozens of warnings is a failed operator experience: the
|
||||
|
||||
@@ -1,782 +0,0 @@
|
||||
# PromptKit v0.8.0 Upgrade Implementation Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
Implement the target state defined by the
|
||||
[PromptKit v0.8.0 Upgrade](promptkit-v0.8.md): adopt the useful PromptKit
|
||||
v0.6.0, v0.7.0, and v0.8.0 changes; enable one bounded structural correction
|
||||
by default; expose pipeline and binding overrides; preserve safe provider
|
||||
diagnostics; and keep PromptKit behind Notarius's transport-neutral LLM
|
||||
boundary.
|
||||
|
||||
This plan is ordered. Each numbered stage is one implementation prompt for a
|
||||
gpt-5.6-terra coding agent. Complete and validate one stage before beginning
|
||||
the next. Read `docs/development.md` and every policy under `docs/policy/` at
|
||||
the start of each stage, inspect the current code and tests named by that
|
||||
stage, preserve unrelated worktree changes, and update current-behavior
|
||||
documentation in the same stage as the behavior it describes.
|
||||
|
||||
Do not retire this plan or `promptkit-v0.8.md` during implementation. Keep both
|
||||
until the completed work has passed a separate review. Do not implement the
|
||||
future Notarius semantic-validation retry loop, D&D combat-scene validator, or
|
||||
warning redesign as part of this plan.
|
||||
|
||||
## Decisions Fixed For Implementation
|
||||
|
||||
- Pin `gitea.maximumdirect.net/eric/promptkit` v0.8.0 directly, with no
|
||||
`replace`, workspace dependency, or vendored source.
|
||||
- Every maintained eligible production prompt defaults to exactly one
|
||||
PromptKit structural repair attempt.
|
||||
- Add the exact configuration key
|
||||
`structured_output_repair_attempts` at pipeline scope and on LLM-backed
|
||||
module and validator bindings.
|
||||
- Effective precedence is binding value, then pipeline value, then the prompt's
|
||||
declared `repair_attempts` value. Omission inherits; explicit zero disables
|
||||
structural repair at that scope.
|
||||
- Accepted values are integers from zero through three. Explicit null and
|
||||
non-integer values are invalid. An explicit binding value on a deterministic
|
||||
module or validator is invalid. A pipeline value is applied only to selected
|
||||
LLM-backed bindings and does not make deterministic bindings invalid.
|
||||
- Keep file configuration version 4. This is an additive pre-release field and
|
||||
does not require parallel versioned behavior.
|
||||
- Use `StructuredOutputRepairAttempts *int` for presence-aware internal Go
|
||||
fields. Clone pointers at every ownership boundary.
|
||||
- A configured override never replaces schema identity, output format, or
|
||||
validation mode. The PromptKit adapter calls `InspectPrompt`, copies the
|
||||
complete normalized prompt-owned output contract, changes only
|
||||
`RepairAttempts`, and supplies the complete replacement on `RunRequest`.
|
||||
Do not add an inspection cache initially.
|
||||
- PromptKit repair is internal to one `CompleteStructured` call and does not
|
||||
consume or replenish a binding's existing `retries` budget.
|
||||
- Add `RepairAttempts int` to Notarius's structured-completion response. It is
|
||||
the actual corrective-call count reported by PromptKit; token usage remains
|
||||
PromptKit's cumulative usage and must not be summed again.
|
||||
- A valid repaired response is successful and produces no warning solely
|
||||
because repair occurred. Exhausted structural validation maps to
|
||||
`ErrInvalidStructuredOutput` with the final candidate and debug material
|
||||
retained.
|
||||
- Add an application-owned generation-error sentinel and typed status-bearing
|
||||
error. PromptKit error types must not cross `internal/framework/llm`.
|
||||
- HTTP status may appear in the application-owned generation error. Provider
|
||||
code, type, and message are excluded from ordinary errors, warnings,
|
||||
manifests, cache, and checkpoint identity; they may appear only in an
|
||||
explicitly requested debug trace after Notarius redaction.
|
||||
- Profile inheritance is owned entirely by PromptKit. Notarius passes sources
|
||||
through, inspects and records the resolved target, and does not parse or merge
|
||||
`base_profile` itself.
|
||||
- PromptKit's built-in `rakestrawhome` backend and
|
||||
`rakestrawhome-gemma-4-31b` profile are available generically. Notarius does
|
||||
not register, shadow, or select them by default.
|
||||
- Missing optional credential environment values are allowed to reach the
|
||||
provider without `Authorization`; Notarius does not recreate v0.5.0's local
|
||||
failure or add provider-specific authentication logic.
|
||||
- No dependency-upgrade ADR is required. Update architecture only with the
|
||||
durable ownership distinction between PromptKit structural repair and
|
||||
Notarius stage/semantic validation policy.
|
||||
|
||||
## Stage 1: Upgrade The Dependency And Establish A Clean v0.8.0 Baseline ✅
|
||||
|
||||
### Goal
|
||||
|
||||
Move the repository to PromptKit v0.8.0, resolve source-compatibility issues,
|
||||
and establish a passing baseline before adopting new behavior.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Re-read the upstream v0.6.0, v0.7.0, and v0.8.0 release guides and the
|
||||
v0.8.0 package consumer and format documentation. Treat the pinned v0.8.0
|
||||
tag, not the sibling checkout's moving branch, as authoritative.
|
||||
2. Update `go.mod` and `go.sum` to PromptKit v0.8.0 and run `go mod tidy` with
|
||||
`GOWORK=off`.
|
||||
3. Compile before making compatibility edits. Correct only actual source or
|
||||
behavior incompatibilities. In particular:
|
||||
- convert any positional `promptkit.Profile` or
|
||||
`promptkit.OpenAICompatibleProfileConfig` literals to keyed literals;
|
||||
- confirm Notarius does not register the newly reserved `rakestrawhome`
|
||||
backend ID; and
|
||||
- preserve `PrepareExecution`/`Details`/`RunPrepared` snapshot ownership,
|
||||
`Discard`, session forwarding, reasoning override, profile preflight,
|
||||
and capacity adaptation.
|
||||
4. Change `promptKitBuiltinProfileCatalogID` in
|
||||
`internal/framework/llm/promptkit_profile_fingerprint.go` from the v0.5.0
|
||||
catalog marker to an opaque v0.8.0 marker. Do not hash PromptKit internal
|
||||
files or include catalog content in manifests.
|
||||
5. Update `docs/integrations/pkg-promptkit.md` to pin and link v0.8.0 and to
|
||||
state that this stage still leaves the production prompt-declared repair
|
||||
budget at its current value. Do not document later configuration or default
|
||||
behavior before it exists.
|
||||
6. Update only those existing tests whose public PromptKit types or stable
|
||||
v0.8.0 behavior genuinely changed. Do not rewrite tests merely to match
|
||||
upstream diagnostic wording.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
```sh
|
||||
GOWORK=off go mod tidy -diff
|
||||
GOWORK=off go test ./internal/framework/llm ./internal/cli
|
||||
GOWORK=off go test ./...
|
||||
GOWORK=off go vet ./...
|
||||
GOWORK=off go build ./cmd/notarius
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- `go list -m gitea.maximumdirect.net/eric/promptkit` reports v0.8.0.
|
||||
- There is no PromptKit `replace`, active Go workspace dependency, or vendor
|
||||
tree.
|
||||
- The adapter still uses one frozen prepared execution and all existing LLM
|
||||
tests pass.
|
||||
- Checkpoint profile identity includes the v0.8.0 built-in catalog marker.
|
||||
- Current integration documentation pins v0.8.0 without claiming that
|
||||
later stages are already active.
|
||||
- The full ordinary test suite, vet, and command build pass.
|
||||
|
||||
## Stage 2: Verify v0.6.0 Compatibility And Hardening ✅
|
||||
|
||||
### Goal
|
||||
|
||||
Audit Notarius's assets and boundary values against PromptKit v0.6.0's stricter
|
||||
source, path, endpoint, JSON, and cancellation contracts, fixing only concrete
|
||||
incompatibilities.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Inspect `internal/framework/llm/asset_registry.go`, prompt/profile source
|
||||
composition, all registered asset roots, the conventional local backend,
|
||||
and their focused tests.
|
||||
2. Exercise every production asset registry through PromptKit engine
|
||||
construction and the existing production composition tests. Confirm that:
|
||||
- YAML IDs and versions, not filenames, select definitions;
|
||||
- every `content_file` path is exact, relative, contained, and points to a
|
||||
regular embedded file;
|
||||
- every schema and JSON asset is one complete JSON value;
|
||||
- every current output contract is valid under v0.8.0; and
|
||||
- unrelated malformed definitions do not create a second Notarius identity
|
||||
or fallback mechanism.
|
||||
3. Review local endpoint parsing and validation. Retain a narrower Notarius
|
||||
rule only if it has independent application value; otherwise rely on
|
||||
PromptKit's absolute HTTP/HTTPS URL contract. Never accept a value that the
|
||||
adapter will later reject.
|
||||
4. Review conversion of Notarius variables, inputs, profile extras, and debug
|
||||
values at the adapter boundary for PromptKit's bounded JSON-compatible-value
|
||||
rules. Do not add a second generic JSON walker or duplicate upstream numeric
|
||||
limits.
|
||||
5. Verify cancellation and deadline identity through existing adapter tests.
|
||||
Add or refine one focused regression only if Notarius currently destroys an
|
||||
`errors.Is`-relevant context or transport error that the application owns.
|
||||
6. Do not add a cross-operation schema cache, artifact cache, provider-body
|
||||
reader, or duplicate JSON framing validation; v0.6.0 owns those mechanisms.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
Run the focused asset, profile-source, and adapter packages, then the ordinary
|
||||
and race-enabled suites:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/framework/llm ./internal/cli
|
||||
GOWORK=off go test ./...
|
||||
GOWORK=off go test -race ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Tests must remain offline and should validate Notarius's assembled boundary,
|
||||
not reproduce PromptKit's internal path, JSON-depth, or response-size matrices.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Every maintained embedded prompt, schema, and fallback profile can be loaded
|
||||
through the assembled v0.8.0 engine.
|
||||
- Current local endpoint and JSON-compatible values either satisfy the stricter
|
||||
upstream contract or fail during preparation with safe diagnostics.
|
||||
- No duplicate PromptKit-owned cache, JSON, or response-bound mechanism is
|
||||
introduced.
|
||||
- Cancellation and deadline behavior remains discoverable at the Notarius
|
||||
boundary.
|
||||
- Ordinary and race-enabled tests pass.
|
||||
|
||||
## Stage 3: Adopt Profile Inheritance, Rakestrawhome, And Optional Credentials ✅
|
||||
|
||||
### Goal
|
||||
|
||||
Make the useful PromptKit v0.7.0 profile and backend behavior work through
|
||||
Notarius's existing generic profile boundary without adding provider-specific
|
||||
composition logic.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Inspect `promptkit_profiles.go`, `asset_registry.go`, profile fingerprinting,
|
||||
CLI profile preflight, profile provenance recording, and their tests before
|
||||
editing.
|
||||
2. Add an offline integration test using a temporary operator profile source
|
||||
whose leaf uses `base_profile`. Prove that:
|
||||
- preflight reports the leaf ID;
|
||||
- the effective backend, model, reasoning, and other inherited values match
|
||||
the resolved PromptKit target;
|
||||
- execution uses the same resolved target as inspection; and
|
||||
- a missing parent or cycle fails before provider generation with a safe
|
||||
profile-load diagnostic.
|
||||
Do not duplicate PromptKit's entire field-by-field merge test matrix.
|
||||
3. Add a checkpoint-safety test showing that changing a parent definition in
|
||||
an operator profile directory changes Notarius's profile-source fingerprint
|
||||
while profile content and paths remain absent from the fingerprint value.
|
||||
Retain the v0.8.0 catalog marker as coverage for built-in-parent changes.
|
||||
4. Verify `rakestrawhome-gemma-4-31b` through the ordinary profile inspector.
|
||||
Assert its selected backend reaches Notarius's application-owned inspection
|
||||
and provenance fields. Use a fake PromptKit client or transport if execution
|
||||
coverage is needed; never contact the live service or require credentials.
|
||||
5. Verify that Notarius registers no `rakestrawhome` override and that the
|
||||
existing `local` registration remains independent.
|
||||
6. Add one `httptest`-backed adapter integration test for a filesystem profile
|
||||
with a missing optional `api_key_env`. The request must reach the test server
|
||||
without an `Authorization` header. Add a focused in-memory PromptKit profile
|
||||
test for `APIKeyRequired` only if needed to prove Notarius preserves upstream
|
||||
preflight behavior; do not expose a new operator profile API.
|
||||
7. Keep `assets/dnd/profiles/dnd-extraction.yaml` standalone and unchanged. No
|
||||
matching v0.8.0 built-in profile owns its `openai/gpt-5.6-luna` target.
|
||||
8. Update the current profile-source, deployment, and pinned-integration
|
||||
sections in `docs/config.md`, `docs/operations.md`,
|
||||
`docs/internal/llm.md`, and `docs/integrations/pkg-promptkit.md`. Link to the
|
||||
pinned PromptKit format rules for inheritance. Explain that filesystem
|
||||
profiles cannot express PromptKit's in-memory `APIKeyRequired` field and
|
||||
that an optional missing credential may result in a provider 401/403.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/framework/llm ./internal/core/config ./internal/cli
|
||||
GOWORK=off go test ./...
|
||||
GOWORK=off go test -race ./internal/framework/llm ./internal/cli
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Inherited operator profiles resolve identically during preflight and
|
||||
execution, with the leaf ID and effective target kept distinct.
|
||||
- Parent changes invalidate checkpoint reuse without leaking profile content or
|
||||
paths.
|
||||
- Rakestrawhome is available through generic PromptKit profile handling and is
|
||||
not selected by default or registered by Notarius.
|
||||
- Missing optional credentials omit authorization and reach the controlled
|
||||
test provider; explicitly required credentials retain upstream behavior.
|
||||
- Current documentation accurately describes the implemented profile and
|
||||
credential behavior without duplicating PromptKit's merge algorithm.
|
||||
|
||||
## Stage 4: Adapt Structured Generation Errors Safely ✅
|
||||
|
||||
### Goal
|
||||
|
||||
Use PromptKit v0.7.0's structured generation errors for stable status
|
||||
classification and debug-only provider diagnostics without leaking PromptKit
|
||||
types or sensitive provider text.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. In `internal/framework/contracts`, add:
|
||||
- `ErrLLMGeneration` as the provider-neutral generation-failure sentinel;
|
||||
- an application-owned `LLMGenerationError` with private status and safe
|
||||
diagnostic fields, `Error`, `Unwrap`, and `StatusCode` methods; and
|
||||
- a constructor that accepts a nonnegative status and an already-redacted
|
||||
diagnostic. Status zero means no HTTP status was available.
|
||||
Ordinary callers may inspect status with `errors.As` and category with
|
||||
`errors.Is`, but cannot obtain provider code, type, or message from the
|
||||
error.
|
||||
2. Add an application-owned `LLMDebugProviderError` with `status_code`, `code`,
|
||||
`type`, and `message` fields, referenced optionally from
|
||||
`LLMDebugResponse`. This is debug material, not a manifest or durable public
|
||||
artifact contract.
|
||||
3. In `PromptKitClient.CompleteStructured`, preserve precedence in this order:
|
||||
caller context cancellation/deadline, PromptKit capacity error, structured
|
||||
PromptKit generation error, then other PromptKit generation failures.
|
||||
Map every generation failure to `ErrLLMGeneration`; map
|
||||
`*promptkit.GenerationError` to `LLMGenerationError` with its status.
|
||||
Never wrap or return the PromptKit error value itself.
|
||||
4. Keep the ordinary diagnostic limited to PromptKit's safe default error
|
||||
formatting after bearer and known-credential redaction. Do not append
|
||||
`ProviderCode`, `ProviderType`, or `ProviderMessage` to it.
|
||||
5. For an explicitly requested debug path, preserve prepared prompt details and
|
||||
attach the PromptKit provider code, type, and message after:
|
||||
- reading only the selected prepared target's `APIKeyEnv`, if any, to obtain
|
||||
the exact known credential solely for redaction;
|
||||
- applying `RedactSecrets` and the existing bearer/key-pattern redaction;
|
||||
- retaining PromptKit's already-normalized bounds; and
|
||||
- discarding the credential value immediately rather than storing it.
|
||||
Do not scan unrelated environment variables.
|
||||
6. Return prompt/debug material alongside the error so the existing debug LLM
|
||||
wrapper can persist it only when debug recording is enabled. Confirm that
|
||||
provider fields do not appear in ordinary error text, warnings, manifests,
|
||||
cache, checkpoint data, or a run without debug output.
|
||||
7. Refactor error mapping into small helpers if needed to keep
|
||||
`CompleteStructured` readable; do not create provider-specific policy in
|
||||
modules or the pipeline runner.
|
||||
8. Update the error and observability sections of `docs/internal/llm.md` and
|
||||
`docs/integrations/pkg-promptkit.md`. Keep operator disclosure rules in
|
||||
`docs/operations.md` concise and link to the internal boundary where useful.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
- Use `httptest.Server` to return representative structured 400 and 503
|
||||
responses. Assert `errors.Is(ErrLLMGeneration)`, `errors.As` to the
|
||||
application-owned type, and the exact status without asserting complete
|
||||
human wording.
|
||||
- Include a provider message containing the selected test credential and a
|
||||
bearer-shaped value. Verify both are absent from the ordinary error and
|
||||
debug artifact, while a non-sensitive marker appears only in the requested
|
||||
debug trace.
|
||||
- Retain existing capacity and context tests to prove their more specific
|
||||
classifications still win.
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/framework/contracts ./internal/framework/llm ./internal/framework/pipeline ./internal/cli
|
||||
GOWORK=off go test ./...
|
||||
GOWORK=off go test -race ./internal/framework/llm ./internal/framework/pipeline
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- PromptKit generation errors never escape the adapter error chain.
|
||||
- All generation failures match `ErrLLMGeneration`; structured non-success
|
||||
responses expose only application-owned HTTP status to ordinary callers.
|
||||
- Provider code, type, and message are available only in an explicitly
|
||||
requested, redacted debug trace.
|
||||
- Capacity and context classifications remain unchanged and more specific.
|
||||
- Security tests prove selected credentials and bearer tokens are not leaked.
|
||||
|
||||
## Stage 5: Add Adapter-Level Structured Repair Support ✅
|
||||
|
||||
### Goal
|
||||
|
||||
Teach the transport-neutral completion boundary and PromptKit adapter to apply
|
||||
an optional repair override and report actual repair behavior, without yet
|
||||
exposing the setting in pipeline configuration.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Add `StructuredOutputRepairAttempts *int` to
|
||||
`contracts.StructuredCompletionRequest`. Copy the pointed-to value wherever
|
||||
requests are cloned or retained.
|
||||
2. Add `RepairAttempts int` to `contracts.StructuredCompletionResponse`. It is
|
||||
the actual number of corrective generation calls, not the configured budget
|
||||
and not the number of total candidates.
|
||||
3. Validate a non-nil request value as zero through three at the adapter
|
||||
boundary so programmatic callers cannot bypass later file/config validation.
|
||||
4. When the request value is nil, leave `promptkit.RunRequest.Validation` nil
|
||||
so the prompt's complete contract remains authoritative.
|
||||
5. When the value is non-nil:
|
||||
- call `Engine.InspectPrompt(ctx, promptID, promptVersion)`;
|
||||
- copy `PromptInspection.OutputContract` by value;
|
||||
- replace only `RepairAttempts`;
|
||||
- pass the complete copied contract as `RunRequest.Validation`; and
|
||||
- prepare and execute exactly as before.
|
||||
Do not infer or hard-code schema paths, validation modes, or formats. Do not
|
||||
cache inspection in this stage.
|
||||
6. Map `result.Validation.RepairAttempts` to the response and leave
|
||||
`result.Usage` cumulative values unchanged. The existing debug validation
|
||||
object and prepared output contract should show actual and configured values
|
||||
respectively.
|
||||
7. Preserve result semantics:
|
||||
- valid initial and repaired candidates decode normally;
|
||||
- repair exhaustion returns the final raw candidate/debug material with an
|
||||
error matching `ErrInvalidStructuredOutput`;
|
||||
- explicit empty or whitespace-only content follows PromptKit validation;
|
||||
- missing/null/non-string content remains a generation/provider failure;
|
||||
- corrective-call generation errors use Stage 4's application-owned mapping;
|
||||
and
|
||||
- context cancellation wins at every error boundary.
|
||||
8. Keep `CompleteStructured` and its helpers provider neutral outside this
|
||||
adapter package. Do not expose PromptKit validation or inspection types.
|
||||
9. Update only the adapter-owned repair behavior in `docs/internal/llm.md` and
|
||||
`docs/integrations/pkg-promptkit.md`. State that public pipeline configuration
|
||||
and the production default are added by later stages of this plan.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
Add adapter-level behavioral tests using a deterministic fake PromptKit LLM:
|
||||
|
||||
- nil override uses the prompt declaration;
|
||||
- explicit zero overrides a positive prompt declaration without dropping its
|
||||
JSON Schema contract;
|
||||
- explicit one turns an invalid first candidate followed by a valid candidate
|
||||
into one successful response with the final raw bytes, actual repair count
|
||||
one, and cumulative usage;
|
||||
- repair exhaustion returns the final candidate and validation diagnostics as
|
||||
`ErrInvalidStructuredOutput`;
|
||||
- explicit empty content is eligible for repair;
|
||||
- a corrective generation failure maps through Stage 4; and
|
||||
- invalid direct values below zero or above three fail before provider work.
|
||||
|
||||
Do not assert PromptKit's exact assistant/user correction prose or copy its
|
||||
full internal repair matrix.
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/framework/contracts ./internal/framework/llm ./internal/framework/pipeline
|
||||
GOWORK=off go test -race ./internal/framework/llm ./internal/framework/pipeline
|
||||
GOWORK=off go test ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- The adapter changes only repair count when applying a request override.
|
||||
- Nil and explicit zero remain distinct.
|
||||
- Repaired success returns final raw output, cumulative usage, and actual count
|
||||
without a warning.
|
||||
- Exhaustion, empty content, corrective generation failure, and cancellation
|
||||
match the target semantics.
|
||||
- No PromptKit type crosses the LLM package boundary.
|
||||
|
||||
## Stage 6: Propagate Repair Policy Through Framework Requests ✅
|
||||
|
||||
### Goal
|
||||
|
||||
Carry an optional effective repair budget from each resolved stage or validator
|
||||
binding to its module request without changing public file configuration yet.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Add `StructuredOutputRepairAttempts *int` alongside `LLMProfile` to every
|
||||
stage request that can belong to an LLM-backed binding:
|
||||
- `ParseRequest`;
|
||||
- `ChunkRequest`;
|
||||
- `TypedExtractionRequest`;
|
||||
- `TypedMergeRequest`;
|
||||
- `TypedNormalizeRequest`;
|
||||
- `OutputRequest`;
|
||||
- `TypedValidationRequest`;
|
||||
- `ChunkValidationRequest`; and
|
||||
- `SerializedValidationRequest`.
|
||||
2. Add the same optional field to the erased/internal request carriers used by
|
||||
registry builders, preparation, runner stage attempts, validator targets,
|
||||
retry closures, and debug wrappers. Copy pointer values; never share a
|
||||
mutable pointer owned by configuration.
|
||||
3. At every runner stage invocation, obtain the value from the exact resolved
|
||||
producer binding. At every validator invocation, obtain it from that exact
|
||||
resolved validator binding. Do not use the producer's value for a validator
|
||||
or vice versa.
|
||||
4. Ensure all retry attempts for the same binding receive the same effective
|
||||
structural-repair value. Do not decrement it in Notarius; PromptKit owns the
|
||||
inner budget independently on each `CompleteStructured` call.
|
||||
5. Extend `semanticreconcile.Request` with the optional field and carry it into
|
||||
each generic reconciliation completion. A batched reconciliation may make
|
||||
several completion calls; each call receives the same effective budget.
|
||||
6. Update registry erasure/adaptation code for typed merge, normalize, and
|
||||
validation requests so no field is lost. Preserve input/output support even
|
||||
though current production input and output modules are deterministic.
|
||||
7. Add focused framework tests for one chunk producer, one extraction
|
||||
producer, one normalizer, and one LLM-backed validator. Verify exact pointer
|
||||
value propagation and separation between producer and validator settings.
|
||||
Do not add repetitive tests for every generic adapter.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/framework/contracts ./internal/framework/pipeline ./internal/framework/semanticreconcile
|
||||
GOWORK=off go test -race ./internal/framework/pipeline ./internal/framework/semanticreconcile
|
||||
GOWORK=off go test ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Every stage and validator request can carry a detached optional repair value.
|
||||
- The runner sources the value from the exact resolved binding.
|
||||
- Producer and validator values cannot overwrite one another.
|
||||
- Stage retries reuse but do not mutate or consume the inner repair budget.
|
||||
- Semantic reconciliation forwards the budget to every one of its completion
|
||||
calls.
|
||||
- Existing behavior remains unchanged while all values are nil.
|
||||
|
||||
## Stage 7: Forward Repair Policy From Every LLM-Backed Module ✅
|
||||
|
||||
### Goal
|
||||
|
||||
Complete the internal end-to-end path by having every production LLM-backed
|
||||
module forward its stage request value to `CompleteStructured`.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Inventory every production `CompleteStructured` call with code search before
|
||||
editing. The expected current owners include:
|
||||
- `dnd/scenes` chunking;
|
||||
- the combat-turn, enemy-event, item-occurrence, item-registry,
|
||||
location-occurrence, location-registry, NPC-occurrence, NPC-registry,
|
||||
scene-description, and spell extractors; and
|
||||
- generic semantic reconciliation used by the item, location, and NPC
|
||||
registry normalizers.
|
||||
Reconcile this list with the actual repository; do not omit a newly added
|
||||
production caller merely because it is not named here.
|
||||
2. In each direct caller, set
|
||||
`StructuredCompletionRequest.StructuredOutputRepairAttempts` from the
|
||||
corresponding stage request. Clone the pointer or use a small shared helper
|
||||
if that reduces repeated ownership mistakes without moving domain logic.
|
||||
3. Ensure D&D registry normalizers pass their typed normalize request value into
|
||||
`semanticreconcile.Request`, and that the generic engine forwards it as
|
||||
established in Stage 6.
|
||||
4. Update existing module prompt-mapping tests that already inspect a captured
|
||||
structured-completion request to assert the new field. Do not create a new
|
||||
one-test-per-module suite solely to memorialize field plumbing; rely on the
|
||||
existing request-contract tests plus a final complete call-site audit.
|
||||
5. Search again after editing for production `CompleteStructured` calls and
|
||||
verify each either forwards the field or documents why it cannot receive a
|
||||
pipeline binding. Test-only fakes need only preserve the field when their
|
||||
contract test depends on it.
|
||||
6. Do not set a module-specific fallback value. Nil must reach the adapter so
|
||||
the prompt declaration remains authoritative.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
Run focused D&D and semantic-reconciliation packages, then the full suite:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/modules/dnd/... ./internal/framework/semanticreconcile
|
||||
GOWORK=off go test -race ./internal/modules/dnd/... ./internal/framework/semanticreconcile
|
||||
GOWORK=off go test ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Every production LLM-backed completion receives the exact stage or validator
|
||||
repair value.
|
||||
- No module invents a default or imports PromptKit.
|
||||
- Registry normalizers preserve the value through semantic reconciliation.
|
||||
- Existing request-contract tests remain concise and pass.
|
||||
- A final call-site audit finds no silent production omission.
|
||||
|
||||
## Stage 8: Add The Public Repair Configuration Contract ✅
|
||||
|
||||
### Goal
|
||||
|
||||
Add presence-aware pipeline and binding configuration for
|
||||
`structured_output_repair_attempts` without yet changing runtime resolution.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Add `StructuredOutputRepairAttempts *int` to
|
||||
`pipeline.PipelineProfile` and `pipeline.ModuleBinding`, using
|
||||
`json:"structured_output_repair_attempts,omitempty"`.
|
||||
2. Add presence-aware YAML support at pipeline and object-binding scope:
|
||||
- exact key `structured_output_repair_attempts`;
|
||||
- integer values zero through three;
|
||||
- explicit null, non-integer, and out-of-range values rejected with scoped
|
||||
diagnostics; and
|
||||
- scalar shorthand bindings continue to omit the binding override.
|
||||
Preserve file configuration version 4.
|
||||
3. Update every configuration clone, conversion, redaction, summary, and JSON
|
||||
round-trip carrier. Copy pointers by value into newly allocated storage so
|
||||
parsed, configured, and redacted values do not alias.
|
||||
4. Preserve omission versus explicit zero through YAML parsing, profile
|
||||
inheritance, module-binding object form, and JSON round trips. Keep scalar
|
||||
shorthand bindings equivalent to omission.
|
||||
5. Do not add a top-level `promptkit.repair_attempts` setting or CLI override.
|
||||
6. Add concise parser and ownership tests. Defer execution-class checks,
|
||||
effective precedence, resolved digests, and runtime forwarding to Stage 9,
|
||||
where module metadata is available.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
At the parser/config boundary, test omitted, explicit zero, positive bounds,
|
||||
negative, above-three, null, non-integer, scalar shorthand, cloning, redaction,
|
||||
and JSON round-trip behavior. Use relational boundary tests for the allowed
|
||||
range and avoid duplicating the same cases at every layer.
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/core/config ./internal/cli
|
||||
GOWORK=off go test -race ./internal/core/config
|
||||
GOWORK=off go test ./...
|
||||
go run ./cmd/notarius config validate \
|
||||
--config examples/dnd-minimal.config.yml \
|
||||
--pipeline dnd-session
|
||||
go run ./cmd/notarius config validate \
|
||||
--config examples/dnd-complete.config.yml \
|
||||
--pipeline dnd-session
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- The exact public field parses at pipeline and object-binding scope with the
|
||||
fixed range.
|
||||
- Nil and explicit zero remain distinguishable through parsing, cloning,
|
||||
inheritance, redaction, summaries, and round trips.
|
||||
- Both maintained configurations remain valid without requiring the new field.
|
||||
- No runtime or prompt default has changed prematurely.
|
||||
|
||||
## Stage 9: Resolve And Apply Repair Configuration ✅
|
||||
|
||||
### Goal
|
||||
|
||||
Resolve the public field against module execution classes, incorporate the
|
||||
effective value into pipeline identity, and connect it to the request plumbing
|
||||
completed in Stages 6 and 7.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. During resolution, compute the effective value for every selected binding:
|
||||
- explicit binding value wins;
|
||||
- otherwise an explicit pipeline value applies to an LLM-backed binding;
|
||||
- otherwise leave nil for prompt-owned policy.
|
||||
Apply the pipeline value to LLM-backed validators as well as producers.
|
||||
2. Reject an explicit binding value on a deterministic module or deterministic
|
||||
validator using the same execution-class knowledge used for `llm_profile`.
|
||||
Do not reject a pipeline-level value merely because a selected pipeline also
|
||||
contains deterministic bindings; simply do not apply it to those bindings.
|
||||
3. Clone every resolved pointer so the parsed pipeline, resolved profile,
|
||||
redacted summaries, and runner requests have distinct ownership.
|
||||
4. Include the effective field in resolved pipeline JSON and digest input. A
|
||||
change between nil, zero, and a positive value must change the resolved
|
||||
digest when it changes an LLM-backed selected binding. Unselected lanes must
|
||||
retain the repository's existing digest and selection semantics.
|
||||
5. Pass the resolved value into the Stage 6 request field for every selected
|
||||
input, chunk, extract, merge, normalize, output, and validator binding.
|
||||
6. Update `docs/config.md` as the canonical field, range, and precedence
|
||||
contract; `docs/internal/pipeline.md` as the resolution owner; and
|
||||
`docs/operations.md` for the distinction from binding `retries`. The
|
||||
prompt-owned production default remains unchanged until Stage 10.
|
||||
7. Add focused resolution and runner tests. Cover representative execution
|
||||
classes rather than repeating the same assertion for every module type.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
Test:
|
||||
|
||||
- binding over pipeline over nil precedence;
|
||||
- inheritance into each selected LLM-backed stage and validator;
|
||||
- no inheritance into deterministic bindings;
|
||||
- explicit deterministic-binding rejection;
|
||||
- detached pointers;
|
||||
- runner forwarding for representative producer and validator bindings; and
|
||||
- digest changes for execution-relevant nil, zero, and positive changes.
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/framework/pipeline ./internal/cli
|
||||
GOWORK=off go test -race ./internal/framework/pipeline
|
||||
GOWORK=off go test ./...
|
||||
go run ./cmd/notarius config validate \
|
||||
--config examples/dnd-minimal.config.yml \
|
||||
--pipeline dnd-session
|
||||
go run ./cmd/notarius config validate \
|
||||
--config examples/dnd-complete.config.yml \
|
||||
--pipeline dnd-session
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- The exact public field has the fixed binding-over-pipeline-over-prompt
|
||||
precedence for every selected LLM-backed producer and validator.
|
||||
- Deterministic binding misuse fails during resolution before execution, while
|
||||
a pipeline value coexists with deterministic bindings.
|
||||
- Nil and explicit zero remain distinguishable through resolution, runtime,
|
||||
summaries, and digests.
|
||||
- A policy change invalidates checkpoint identity when it changes an effective
|
||||
selected binding.
|
||||
- Current configuration, pipeline, and operations documentation matches the
|
||||
implemented behavior.
|
||||
|
||||
## Stage 10: Enable The Default, Finish Documentation, And Verify The Feature ✅
|
||||
|
||||
### Goal
|
||||
|
||||
Set the accepted production default of one repair, reconcile all canonical
|
||||
documentation, and run the full repository verification pass.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Change `repair_attempts: 0` to `repair_attempts: 1` in every maintained
|
||||
production prompt manifest that produces structured output, including the
|
||||
generic semantic-reconciliation prompt and every D&D chunk, extraction, and
|
||||
registry-normalization prompt. Do not mechanically change unrelated test
|
||||
fixtures whose purpose is to exercise zero.
|
||||
2. Inspect every production prompt output contract after the edit. Confirm that
|
||||
each positive budget uses `basic`, `json`, or `json_schema`, remains no
|
||||
greater than three, and retains its existing format and schema path.
|
||||
3. Add or refine the smallest durable assembled-assets test that proves the
|
||||
production engine can prepare the maintained prompts with the activated
|
||||
contracts. Do not add a brittle test that asserts an exact prompt count,
|
||||
file count, message prose, correction text, or asset length. The public
|
||||
default may be tested at one canonical assembled boundary because its
|
||||
literal value is an operational contract.
|
||||
4. Confirm a successful repair does not create a warning and that exhausted
|
||||
repair remains `ErrInvalidStructuredOutput`. Verify the debug prompt records
|
||||
the configured contract, the debug response records actual repair count,
|
||||
and cumulative usage is not double-counted.
|
||||
5. Confirm scheduling behavior with one focused test or existing coverage: the
|
||||
Notarius scheduled client admits one logical `CompleteStructured` operation
|
||||
while PromptKit may make serial corrective provider calls inside it. Do not
|
||||
attempt to reacquire a Notarius permit from inside PromptKit or add a second
|
||||
scheduler.
|
||||
6. Reconcile current-state documentation:
|
||||
- `docs/integrations/pkg-promptkit.md` owns the pinned upstream boundary;
|
||||
- `docs/config.md` owns field names, range, default, and precedence;
|
||||
- `docs/operations.md` owns latency/cost, optional credentials, concurrency,
|
||||
timeout, and the upper-bound formula;
|
||||
- `docs/internal/llm.md` owns inspection-based contract replacement,
|
||||
cumulative usage, actual repair count, generation errors, and debug data;
|
||||
- `docs/internal/pipeline.md` owns effective policy propagation and the
|
||||
separation from stage retries; and
|
||||
- `docs/policy/architecture.md` adds only the durable rule that PromptKit
|
||||
owns deterministic structural repair inside one completion while Notarius
|
||||
owns stage attempts and semantic validation.
|
||||
7. Remove current-behavior claims that PromptKit is v0.5.0, that every
|
||||
production repair budget is zero, or that PromptKit is always single-pass.
|
||||
Do not alter historical release notes or archived roadmaps.
|
||||
8. Keep the maintained minimal and complete examples secret-free and valid.
|
||||
They may omit the new field to demonstrate the default; do not add a
|
||||
redundant complete profile or Rakestrawhome example merely to exercise an
|
||||
upstream catalog entry.
|
||||
9. Review `docs/roadmap/future.md` only for consistency. Leave the future
|
||||
feedback-aware stage retry, combat-scene validator, and warning-reform work
|
||||
unimplemented and clearly separate.
|
||||
|
||||
### Tests And Validation
|
||||
|
||||
Run focused tests first, then all repository checks:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test ./internal/framework/llm ./internal/framework/pipeline ./internal/framework/semanticreconcile ./internal/modules/dnd/...
|
||||
GOWORK=off go test ./...
|
||||
GOWORK=off go test -race ./...
|
||||
GOWORK=off go vet ./...
|
||||
GOWORK=off go build ./cmd/notarius
|
||||
GOWORK=off go mod tidy -diff
|
||||
go run ./cmd/notarius config validate \
|
||||
--config examples/dnd-minimal.config.yml \
|
||||
--pipeline dnd-session
|
||||
go run ./cmd/notarius config validate \
|
||||
--config examples/dnd-complete.config.yml \
|
||||
--pipeline dnd-session
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Also perform focused repository searches that exclude `docs/roadmap/archive/`
|
||||
and historical release notes:
|
||||
|
||||
- no active v0.5.0 PromptKit pins or links remain;
|
||||
- no maintained production prompt still declares `repair_attempts: 0`;
|
||||
- every production `CompleteStructured` caller forwards the repair field; and
|
||||
- no provider code, type, or message is added to ordinary errors, warnings,
|
||||
manifests, cache, or checkpoint schemas.
|
||||
|
||||
If the repository's source-release checker is available and the ordinary
|
||||
checks above pass, run `./scripts/check-release-source.sh v0.0.0` as the final
|
||||
integrated validation. It must not create a tag, release note, or repository
|
||||
artifact.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Every maintained structured prompt defaults to one corrective call and can
|
||||
be overridden to zero through three at pipeline or binding scope.
|
||||
- A real assembled Notarius completion follows the PromptKit v0.8.0 repair
|
||||
contract without changing prompt schema identity or cacheable prefix.
|
||||
- Actual repair count, cumulative usage, error classification, debug-only
|
||||
provider diagnostics, scheduling, and checkpoint identity match the feature
|
||||
roadmap.
|
||||
- Profile inheritance, Rakestrawhome availability, optional credentials, and
|
||||
v0.6.0 hardening remain covered and documented.
|
||||
- All canonical documentation describes implemented v0.8.0 behavior in its
|
||||
assigned home and leaves future semantic validation work in the roadmap.
|
||||
- Maintained examples validate, all ordinary/race/vet/build/module checks pass,
|
||||
and the worktree contains no generated or sensitive artifacts.
|
||||
@@ -1,520 +0,0 @@
|
||||
# PromptKit v0.8.0 Upgrade
|
||||
|
||||
## Status
|
||||
|
||||
Proposed.
|
||||
|
||||
## Purpose
|
||||
|
||||
Upgrade Notarius from PromptKit v0.5.0 to v0.8.0 and deliberately adopt the
|
||||
useful correctness, profile-composition, provider-diagnostic, backend, and
|
||||
structured-output-repair capabilities introduced in PromptKit v0.6.0, v0.7.0,
|
||||
and v0.8.0.
|
||||
|
||||
The upgrade should improve structured-output reliability without confusing
|
||||
PromptKit's bounded deterministic repair with Notarius's existing stage retry
|
||||
budget or the future feedback-aware semantic-validation loop. PromptKit types
|
||||
and provider behavior must remain behind Notarius's transport-neutral LLM
|
||||
boundary.
|
||||
|
||||
## Current State
|
||||
|
||||
Notarius currently pins PromptKit v0.5.0. Its production adapter prepares one
|
||||
frozen execution, records credential-redacted details, and runs that same
|
||||
prepared value. It maps PromptKit capacity failures to an application-owned
|
||||
error, maps failed structured validation to `ErrInvalidStructuredOutput`, and
|
||||
returns PromptKit's raw validated bytes and usage metadata.
|
||||
|
||||
Every maintained production prompt uses JSON Schema validation and currently
|
||||
declares `repair_attempts: 0`. Notarius stage bindings separately expose
|
||||
`retries`, which reruns a complete stage operation after an error or rejected
|
||||
candidate. The two mechanisms have different ownership and must remain
|
||||
independent.
|
||||
|
||||
Notarius also maintains:
|
||||
|
||||
- embedded prompt, schema, and fallback-profile filesystems;
|
||||
- operator profile-file and profile-directory sources;
|
||||
- one optional conventional `local` backend registration;
|
||||
- explicit profile preflight through PromptKit inspection;
|
||||
- one application-wide scheduled LLM client around the PromptKit adapter;
|
||||
- PromptKit profile-source fingerprints for checkpoint safety; and
|
||||
- redacted debug and manifest provenance at application-owned boundaries.
|
||||
|
||||
The upgrade must preserve those established responsibilities while revising
|
||||
the pinned integration contract and any behavior affected by the three
|
||||
intervening releases.
|
||||
|
||||
This roadmap is based on PromptKit's pinned release guides for
|
||||
[v0.6.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/releases/v0.6.0.md),
|
||||
[v0.7.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/releases/v0.7.0.md),
|
||||
and
|
||||
[v0.8.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/releases/v0.8.0.md),
|
||||
plus the public API and format documentation at the v0.8.0 tag.
|
||||
|
||||
## Target End State
|
||||
|
||||
- `go.mod` and `go.sum` pin PromptKit v0.8.0 without a local replacement or
|
||||
vendored copy.
|
||||
- Every maintained PromptKit prompt and profile prepares successfully under
|
||||
v0.8.0's stricter validation and source-loading rules.
|
||||
- Eligible Notarius structured completions use one PromptKit corrective call by
|
||||
default after a structurally invalid response. Operators can explicitly set
|
||||
a value from zero through three for a configured pipeline, with a more local
|
||||
LLM-backed binding override where needed.
|
||||
- PromptKit repair remains an inner operation within one Notarius stage
|
||||
attempt. It never consumes or replenishes the binding's `retries` budget.
|
||||
- A successful repaired result exposes cumulative usage and the actual repair
|
||||
count to Notarius's application-owned response and debug models. A repaired
|
||||
success is not itself a warning.
|
||||
- Exhausted PromptKit validation remains an invalid structured-output result,
|
||||
preserving the final candidate and diagnostics for debug and for any
|
||||
applicable outer Notarius stage policy. Invalid structured output is never
|
||||
accepted merely because the repair budget was exhausted.
|
||||
- Profile inheritance, the built-in Rakestrawhome backend/profile, optional
|
||||
credential behavior, and structured generation errors work through the
|
||||
existing Notarius PromptKit boundary and are accurately documented.
|
||||
- Provider-specific PromptKit types do not escape `internal/framework/llm`.
|
||||
- Checkpoint identity, effective configuration, redacted summaries, and debug
|
||||
provenance reflect every execution-affecting repair or profile change.
|
||||
- Current documentation pins and describes v0.8.0; future Notarius semantic
|
||||
validation retries remain roadmap behavior rather than being conflated with
|
||||
this dependency upgrade.
|
||||
|
||||
## Release-by-Release Adoption
|
||||
|
||||
### PromptKit v0.6.0: Correctness, Safety, And Efficiency
|
||||
|
||||
PromptKit v0.6.0 adds no public declarations, but intentionally rejects several
|
||||
formerly permissive or ambiguous inputs. The upgrade must audit Notarius's
|
||||
embedded and operator-facing integration against these rules:
|
||||
|
||||
- YAML `id` and `version` metadata, rather than filenames, define prompt and
|
||||
profile identity.
|
||||
- Prompt `content_file` paths are exact, relative, contained paths; built-in
|
||||
file artifacts must resolve to regular files.
|
||||
- execution controls, output contracts, and repair budgets must be finite and
|
||||
within their documented ranges;
|
||||
- provider endpoints must be absolute HTTP or HTTPS URLs with a host and no
|
||||
user information, query, or fragment;
|
||||
- JSON documents and successful provider responses contain exactly one value;
|
||||
- successful provider responses are bounded to 16 MiB; and
|
||||
- JSON-compatible values are bounded for depth and expansion.
|
||||
|
||||
Notarius should rely on PromptKit for these rules rather than duplicate its
|
||||
parsers or internal limits. Existing Notarius validation may retain a narrower
|
||||
application rule where it has independent value, but overlapping validation
|
||||
must agree with PromptKit and must not accept a value PromptKit will reject
|
||||
later.
|
||||
|
||||
The upgrade automatically receives operation-local schema-plan reuse,
|
||||
artifact-text memoization, improved cancellation checks, and transport error
|
||||
identity preservation. Notarius should verify these changes through its real
|
||||
adapter boundary and avoid adding a second cache or response-body layer that
|
||||
would duplicate PromptKit's ownership.
|
||||
|
||||
### PromptKit v0.7.0: Profiles, Backend Access, And Generation Errors
|
||||
|
||||
#### Profile Inheritance
|
||||
|
||||
Operator profiles may use `base_profile` to alias or selectively refine a
|
||||
built-in, fallback, or higher-precedence operator profile. Notarius must pass
|
||||
profile sources through unchanged and let PromptKit own parent lookup, merge
|
||||
rules, source precedence, cycle detection, and fully resolved prepared targets.
|
||||
|
||||
Preflight inspection must resolve inherited profiles through the same source
|
||||
and backend composition used at execution. The selected leaf profile ID remains
|
||||
the public profile identity, while effective backend, endpoint, model, and
|
||||
reasoning provenance reflect the resolved chain. Notarius must not implement a
|
||||
second inheritance parser.
|
||||
|
||||
The existing complete `dnd-extraction` fallback remains a standalone profile:
|
||||
PromptKit v0.8.0 does not provide a built-in `openai/gpt-5.6-luna` profile that
|
||||
would be an appropriate parent. Documentation should nevertheless explain how
|
||||
operators can use inheritance for environment-specific workload profiles and
|
||||
should link to PromptKit's pinned format contract rather than duplicate its
|
||||
field-by-field merge algorithm.
|
||||
|
||||
Checkpoint safety must cover inherited behavior. Operator file/directory
|
||||
digests already cover changes to definitions in those sources, fallback asset
|
||||
digests cover application parents, and the PromptKit built-in catalog marker
|
||||
must change from its v0.5.0 identity to v0.8.0 so a changed built-in parent
|
||||
cannot reuse an incompatible checkpoint.
|
||||
|
||||
#### Rakestrawhome Backend And Profile
|
||||
|
||||
PromptKit's reserved `rakestrawhome` backend and
|
||||
`rakestrawhome-gemma-4-31b` profile become available without Notarius-specific
|
||||
registration. Notarius must not register or shadow the reserved backend ID.
|
||||
Profile preflight, backend-capacity reporting, scheduling, generation, and
|
||||
provenance should work for it through the same generic paths used by OpenRouter
|
||||
and `local`.
|
||||
|
||||
The D&D default remains `dnd-extraction`; this upgrade does not silently move a
|
||||
production workload to Rakestrawhome. Operator documentation should identify
|
||||
the built-in profile as an available selection and link to PromptKit for its
|
||||
endpoint, credential environment, model, and capacity defaults.
|
||||
|
||||
#### Optional Credentials
|
||||
|
||||
An absent or blank optional `APIKeyEnv` now causes PromptKit to omit the
|
||||
`Authorization` header and send the request. Notarius must not restore the old
|
||||
failure behavior by pre-reading provider credential environment variables or
|
||||
by adding provider-specific authentication logic.
|
||||
|
||||
Profile inspection may report an explicit `APIKeyRequired` policy without
|
||||
reading the credential, and execution remains the boundary at which that
|
||||
requirement is enforced. For optional profiles, an authentication-requiring
|
||||
provider may instead return a structured 401 or 403 generation failure. The
|
||||
configuration and operations documentation must explain this distinction.
|
||||
Notarius does not currently expose PromptKit's in-memory profile-registration
|
||||
API to operators, and PromptKit's filesystem profile format does not expose
|
||||
`APIKeyRequired`; therefore Notarius must not promise that an operator profile
|
||||
can force local credential preflight. Operators should provision the named
|
||||
environment variable, while Notarius should preserve the provider's structured
|
||||
authentication failure when it is absent.
|
||||
|
||||
Notarius must continue to document mechanisms and environment-variable names,
|
||||
never secret values.
|
||||
|
||||
#### Structured Generation Errors
|
||||
|
||||
The adapter should recognize `*promptkit.GenerationError` with `errors.As` and
|
||||
translate useful information into an immutable, provider-neutral Notarius
|
||||
error classification. At minimum, retain the HTTP status code so callers and
|
||||
future retry policy can distinguish transport success with provider rejection
|
||||
from other generation failures.
|
||||
|
||||
PromptKit's provider code, type, and message accessors are bounded but remain
|
||||
untrusted and potentially sensitive. They must never appear automatically in
|
||||
ordinary CLI output, warnings, manifests, checkpoint identity, or cache data.
|
||||
If retained for an explicitly requested debug trace, they must pass through
|
||||
Notarius's known-secret and bearer redaction and remain clearly identified as
|
||||
untrusted provider diagnostics. Default error formatting should continue to
|
||||
use a bounded, redacted application-owned message.
|
||||
|
||||
Capacity and cancellation retain their current more specific classifications
|
||||
and precedence. This upgrade does not add automatic provider-error retry
|
||||
classification; it only preserves safe structured data needed for diagnosis
|
||||
and later policy.
|
||||
|
||||
### PromptKit v0.8.0: Bounded Structured-Output Repair
|
||||
|
||||
#### Default Policy
|
||||
|
||||
Every maintained production prompt whose output is consumed as structured data
|
||||
should declare one repair attempt. All current production prompts use eligible
|
||||
JSON Schema validation, so no current prompt needs a zero default merely
|
||||
because of its output mode.
|
||||
|
||||
One repair means at most one corrective generation after the initial
|
||||
candidate. PromptKit reconstructs the immutable original conversation and
|
||||
appends only the latest invalid assistant candidate and latest deterministic
|
||||
validation diagnostics. It preserves the selected target, direct session ID,
|
||||
provider-native structured-output contract, and backend capacity policy. This
|
||||
shape preserves the original cacheable prompt prefix and avoids accumulating
|
||||
unbounded failed history.
|
||||
|
||||
The default is deliberately small. A single repair captures the common case in
|
||||
which a capable model can correct malformed JSON or a schema violation after
|
||||
receiving an exact diagnostic, while bounding the extra latency and cost of a
|
||||
single structured completion.
|
||||
|
||||
#### Configuration Contract
|
||||
|
||||
The public configuration is an optional, presence-aware
|
||||
`structured_output_repair_attempts` integer at pipeline scope and at each
|
||||
LLM-backed module or validator binding. Its effective precedence is:
|
||||
|
||||
1. the binding value, when present;
|
||||
2. the pipeline value, when present; and
|
||||
3. the selected prompt's declared `repair_attempts` value.
|
||||
|
||||
The value must be from zero through three. Explicit zero disables PromptKit
|
||||
repair at that scope. A deterministic binding must reject the field because it
|
||||
cannot perform structured LLM repair. Validator bindings may use it only when
|
||||
the selected validator is LLM-backed. Shorthand module bindings continue to
|
||||
inherit the pipeline or prompt default.
|
||||
|
||||
The long, provider-neutral name is intentional: it distinguishes PromptKit's
|
||||
inner structural repair from the existing binding `retries` field, which owns
|
||||
complete stage attempts, without exposing a dependency name in generic
|
||||
pipeline contracts.
|
||||
|
||||
The effective value must survive file parsing, cloning, redacted summaries,
|
||||
pipeline resolution, and pipeline digest construction without pointer aliasing
|
||||
or loss of presence. It must affect checkpoint identity because it can change
|
||||
the selected result, latency, token usage, and provider cost.
|
||||
|
||||
#### Adapter Contract
|
||||
|
||||
The transport-neutral structured-completion request should carry an optional
|
||||
application-owned structural-repair budget. No `promptkit.OutputContract` or
|
||||
other PromptKit type may cross the adapter boundary.
|
||||
|
||||
PromptKit v0.8.0 request validation replaces the complete prompt output
|
||||
contract rather than merging one field. When Notarius has a configured
|
||||
override, the adapter must therefore inspect the selected prompt, copy its
|
||||
normalized declared format, validation mode, and schema path, change only the
|
||||
repair count, and supply that complete contract on the prepared request. A nil
|
||||
override continues to use the prompt declaration directly. Inspection and
|
||||
preparation must use the same immutable engine sources; a small adapter-local
|
||||
cache keyed by normalized prompt ID and version is acceptable but not required
|
||||
without measured need.
|
||||
|
||||
This approach prevents configuration from accidentally dropping JSON Schema
|
||||
validation, avoids duplicating schema paths in pipeline YAML, and keeps prompt
|
||||
assets authoritative for every output-contract field other than the explicit
|
||||
operator override.
|
||||
|
||||
The transport-neutral structured-completion response should report the actual
|
||||
number of PromptKit repair calls. PromptKit's returned token usage is already
|
||||
cumulative and must be passed through without re-summing it. Debug records
|
||||
should distinguish the configured budget from the actual count. Ordinary run
|
||||
manifests need not gain raw prompt or response data merely to report repairs;
|
||||
any durable aggregate should be added only if it has a clear consumer contract.
|
||||
|
||||
#### Result And Failure Semantics
|
||||
|
||||
- A valid initial candidate returns normally with zero actual repairs.
|
||||
- A valid corrected candidate returns normally with cumulative usage and its
|
||||
positive actual repair count. It does not emit a warning solely because a
|
||||
repair occurred.
|
||||
- Exhausting the repair budget returns PromptKit's final candidate and failed
|
||||
validation result. The adapter maps this to
|
||||
`ErrInvalidStructuredOutput`, preserves the response and debug material, and
|
||||
does not decode or accept the candidate.
|
||||
- An explicitly empty or whitespace-only candidate participates in the
|
||||
declared structural validation and repair flow. Missing, `null`, or
|
||||
non-string provider content remains a malformed provider response.
|
||||
- A generation failure during a corrective call is an operational generation
|
||||
failure and uses the same safe structured-error adaptation as an initial
|
||||
generation failure.
|
||||
- Context cancellation remains authoritative throughout the initial and
|
||||
corrective calls.
|
||||
|
||||
PromptKit repair happens inside one scheduled `CompleteStructured` operation.
|
||||
The Notarius scheduler holds one permit for that logical operation while
|
||||
PromptKit performs its initial and serial corrective calls; PromptKit
|
||||
reacquires its own selected-backend capacity for each corrective generation.
|
||||
Because corrective calls are serial, this cannot expand actual concurrent
|
||||
provider work beyond the number of admitted Notarius operations, but
|
||||
documentation must stop describing the Notarius permit as a separate admission
|
||||
event for every internal repair call.
|
||||
|
||||
One `CompleteStructured` invocation with effective PromptKit repair budget `R`
|
||||
may make at most `R + 1` provider calls. If one stage attempt makes `C`
|
||||
structured-completion invocations, a binding with `retries: N` has an upper
|
||||
bound of `(N + 1) * C * (R + 1)` provider calls; `C` may itself be a bounded,
|
||||
data-dependent module property, as it is for batched semantic reconciliation.
|
||||
LLM-backed validators have their own corresponding invocation counts, budgets,
|
||||
and costs. These formulas are upper bounds, not promises that every failure is
|
||||
retryable or that every attempt reaches the provider.
|
||||
|
||||
## Profile And Prompt Source Compatibility
|
||||
|
||||
The upgrade must preserve Notarius's source precedence: an operator source,
|
||||
then registered application fallback profiles, then PromptKit built-ins. A
|
||||
selected malformed definition remains authoritative and fails rather than
|
||||
falling through. Parent resolution introduced by profile inheritance observes
|
||||
that same precedence.
|
||||
|
||||
All embedded prompt manifests, shared content fragments, response schemas, and
|
||||
fallback profiles must be prepared or inspected offline under v0.8.0. The
|
||||
review should specifically catch:
|
||||
|
||||
- IDs inferred accidentally from filenames;
|
||||
- stale or escaping `content_file` paths;
|
||||
- missing or non-regular embedded artifacts;
|
||||
- repair values outside zero through three or paired with ineligible
|
||||
validation;
|
||||
- schemas or examples that are not exact single JSON documents;
|
||||
- unsupported endpoint forms; and
|
||||
- JSON-compatible variables or profile extras that exceed upstream bounds.
|
||||
|
||||
No prompt prose, schema shape, durable D&D artifact contract, or default D&D
|
||||
model should change merely to exercise the dependency. Prompt manifests should
|
||||
change only as needed to enable the adopted repair default and satisfy v0.8.0
|
||||
contracts.
|
||||
|
||||
## Provenance, Debugging, And Security
|
||||
|
||||
- Update the opaque PromptKit built-in profile-catalog identity from v0.5.0 to
|
||||
v0.8.0. Do not hash or publish PromptKit's internal catalog bytes.
|
||||
- Ensure a prompt's repair default remains covered by its existing prompt asset
|
||||
fingerprint and a configured effective override remains covered by the
|
||||
resolved pipeline digest.
|
||||
- Preserve selected leaf profile identity while recording the inherited
|
||||
effective target already exposed by PromptKit inspection and prepared
|
||||
details.
|
||||
- Add actual structural-repair count and, when useful, the configured budget to
|
||||
application-owned debug material. Token totals remain PromptKit's cumulative
|
||||
values.
|
||||
- Do not generate a warning for a successful repair. Repair exhaustion is an
|
||||
invalid-output failure, while provider rejection is a generation failure.
|
||||
- Never expose raw provider diagnostic fields without explicit debug capture
|
||||
and application redaction. Do not place them in normal errors or durable
|
||||
summaries.
|
||||
- Preserve context and transport error identity sufficiently for
|
||||
`errors.Is`-based cancellation and deadline handling after adapting the
|
||||
external error.
|
||||
|
||||
## Documentation And Examples
|
||||
|
||||
Implementation should update current-state documentation only when the new
|
||||
behavior lands:
|
||||
|
||||
- `docs/integrations/pkg-promptkit.md` must pin v0.8.0 and define the revised
|
||||
prepared-execution, repair, profile-inheritance, backend, credential, and
|
||||
error-adaptation boundary.
|
||||
- `docs/config.md` must own the repair configuration fields, precedence,
|
||||
allowed range, explicit-zero behavior, profile inheritance availability, and
|
||||
optional credential semantics.
|
||||
- `docs/operations.md` must explain structural repair cost, timeout and
|
||||
concurrency effects, credential failures, and its distinction from stage
|
||||
retries.
|
||||
- `docs/internal/llm.md` must describe adapter contract replacement, actual
|
||||
repair metadata, error adaptation, source compatibility, and scheduling.
|
||||
- `docs/internal/pipeline.md` must describe how effective repair configuration
|
||||
is resolved and how inner repair differs from outer stage attempts.
|
||||
- `docs/policy/architecture.md` should receive only the durable ownership rule:
|
||||
PromptKit owns bounded deterministic structural repair within one completion,
|
||||
while Notarius owns stage attempts and semantic validation policy. Detailed
|
||||
fields and retry formulas belong in their canonical configuration and
|
||||
operations documents.
|
||||
|
||||
Update maintained configuration examples only if the public Notarius
|
||||
configuration contract changes. A short inheritance illustration may remain in
|
||||
the configuration reference; do not create a complete example solely to copy
|
||||
PromptKit's upstream profile catalog. All upstream links must point to the
|
||||
v0.8.0 tag. Historical release or archived roadmap references should remain
|
||||
historical.
|
||||
|
||||
No ADR is required solely to pin a newer dependency. The durable separation
|
||||
between PromptKit structural repair and Notarius semantic stage retries should
|
||||
be stated in architecture documentation now; the more extensive future
|
||||
validation state machine still warrants the separate ADR already identified in
|
||||
`future.md` when that work is promoted.
|
||||
|
||||
## Validation And Acceptance Criteria
|
||||
|
||||
The implementation is complete when:
|
||||
|
||||
- the repository builds and tests against PromptKit v0.8.0 with no replacement
|
||||
directive, workspace dependency, or vendored source;
|
||||
- every maintained prompt and profile prepares or inspects successfully under
|
||||
the v0.8.0 source, path, endpoint, output-contract, and JSON-value rules;
|
||||
- an invalid first JSON Schema candidate followed by a valid correction returns
|
||||
the valid raw output, cumulative usage, and actual repair count through the
|
||||
Notarius adapter;
|
||||
- repair exhaustion returns the final raw candidate and debug material with an
|
||||
error matching `ErrInvalidStructuredOutput`;
|
||||
- a corrective generation failure retains safe generation classification and
|
||||
provider status without leaking untrusted provider detail;
|
||||
- explicit empty content follows structural validation rather than being
|
||||
misclassified by Notarius;
|
||||
- repair configuration is presence-aware, range checked, rejected on
|
||||
deterministic bindings, resolved with documented precedence, and included in
|
||||
effective pipeline identity;
|
||||
- inherited profiles resolve consistently during preflight and execution, and
|
||||
changes to any relevant operator, fallback, or built-in parent invalidate
|
||||
checkpoint reuse;
|
||||
- the Rakestrawhome built-in profile reaches generic preflight, scheduling, and
|
||||
provenance paths without application-specific registration;
|
||||
- optional missing credentials and explicitly required credentials behave as
|
||||
documented without contacting real providers in tests;
|
||||
- cancellation, timeout, backend capacity, prepared-execution snapshot,
|
||||
session ID, raw-output, debug-redaction, and existing profile provenance
|
||||
behavior remain intact;
|
||||
- maintained examples validate successfully; and
|
||||
- canonical documentation contains no active v0.5.0 pin or claim that PromptKit
|
||||
is always single-pass.
|
||||
|
||||
Tests should follow `docs/policy/testing.md`: exercise observable Notarius
|
||||
contracts with offline fake clients or `httptest` boundaries, and do not copy
|
||||
PromptKit's entire internal repair test suite or assert its exact correction
|
||||
message prose. The dependency's internal wording is not a Notarius contract.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Implementing Notarius's future feedback-aware semantic stage-retry loop.
|
||||
- Adding the D&D combat-scene semantic validator.
|
||||
- Redesigning warning policy or treating successful structural repair as a
|
||||
warning.
|
||||
- Adding provider transport retries or deciding which HTTP statuses should
|
||||
consume a stage retry.
|
||||
- Exposing PromptKit request, response, profile, validation, capacity, or error
|
||||
types outside the LLM adapter.
|
||||
- Changing durable artifact schemas, D&D prompt semantics, the D&D default
|
||||
model, or the fixed pipeline shape.
|
||||
- Reimplementing PromptKit profile inheritance, schema validation, response
|
||||
bounds, repair conversations, backend admission, or provider parsing inside
|
||||
Notarius.
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Default Structured-Output Repair Budget
|
||||
|
||||
**Decision: default to one repair attempt.** Set every maintained
|
||||
eligible production prompt to `repair_attempts: 1`. One corrective call is a
|
||||
strong fit for Notarius because every current production LLM response has a
|
||||
strict JSON Schema contract, smaller cost-effective models are a deliberate
|
||||
deployment target, and a precise structural diagnostic often makes one retry
|
||||
materially more successful. The budget is paid only after a structurally
|
||||
invalid candidate and remains tightly bounded.
|
||||
|
||||
**Alternative considered: retain zero by default.** This preserves single-pass
|
||||
cost and latency and requires operators to opt in. It is preferable for an
|
||||
environment where every additional request is expensive or where upstream
|
||||
provider-native schema enforcement already produces negligible invalid output.
|
||||
It is less suitable as the Notarius default because one malformed response can
|
||||
otherwise discard substantial completed pipeline work.
|
||||
|
||||
**Alternative considered: default to two.** This may improve recovery for
|
||||
weak models, but it doubles the worst-case corrective cost relative to the
|
||||
selected default and compounds with outer stage retries. It should be an
|
||||
operator choice supported by configuration, not the initial default, unless
|
||||
observational evidence shows that the second correction has a worthwhile
|
||||
marginal success rate.
|
||||
|
||||
### 2. Repair Override Scope
|
||||
|
||||
**Decision: support both pipeline and LLM-backed binding overrides.** Use
|
||||
the presence-aware `structured_output_repair_attempts` field and precedence
|
||||
defined above. A pipeline value provides the convenient one-line control the
|
||||
operator requested, while a binding value permits an expensive normalizer or
|
||||
future LLM-backed validator to use a deliberately different budget. This
|
||||
mirrors Notarius's established pipeline/binding profile inheritance and scales
|
||||
without editing embedded prompts.
|
||||
|
||||
**Alternative considered: support only a pipeline override.** This is smaller to
|
||||
implement and document and still permits global enablement or disablement for
|
||||
one pipeline. Its drawback is that one exceptional prompt cannot opt out or
|
||||
request a larger budget without changing an embedded asset for every pipeline.
|
||||
|
||||
**Alternative considered: expose one global value under the top-level
|
||||
`promptkit` configuration.** This makes client construction simple, but applies
|
||||
the same budget to unrelated pipelines and leaks an execution policy into the
|
||||
dependency configuration block. It is less compositional than pipeline-owned
|
||||
policy and therefore not recommended.
|
||||
|
||||
### 3. Retention Of Provider-Supplied Generation Details
|
||||
|
||||
**Decision: retain status in the application-owned error contract and
|
||||
retain redacted provider code, type, and message only in explicitly requested
|
||||
debug traces.** Status is useful for diagnosis and future retry policy without
|
||||
usually containing sensitive data. The other fields can materially explain a
|
||||
400 response but may echo request or schema content, so they belong only in the
|
||||
already-sensitive debug surface after Notarius redaction.
|
||||
|
||||
**Alternative considered: retain only HTTP status and discard all provider fields.**
|
||||
This is the safest and smallest policy and still improves typed failure
|
||||
handling. It sacrifices potentially decisive provider diagnostics, leaving an
|
||||
operator with less information when a provider returns a terse status and the
|
||||
problem cannot be reproduced easily.
|
||||
|
||||
**Alternative considered: include bounded provider code and type in normal
|
||||
errors while keeping message debug-only.** Codes and types are often stable and
|
||||
less sensitive than messages, but PromptKit explicitly classifies every
|
||||
provider field as untrusted. Promoting them to ordinary output creates a
|
||||
disclosure and compatibility burden that is not currently justified.
|
||||
@@ -1,282 +0,0 @@
|
||||
# Source-Only Releases
|
||||
|
||||
## Status
|
||||
|
||||
Implemented. Creating the first release under this procedure remains a
|
||||
separate maintainer operation.
|
||||
|
||||
## Purpose
|
||||
|
||||
Define a repeatable, guarded release process for Notarius without taking on a
|
||||
binary-distribution system that its current operator audience does not need.
|
||||
The process should make an exact source revision, its compatibility impact,
|
||||
and its validation status easy to identify while keeping installation in the
|
||||
hands of technically capable operators and deployment automation.
|
||||
|
||||
The model is adapted from Weatherreporter's release procedure, but its target
|
||||
is deliberately narrower: an immutable source tag and checked-in release note
|
||||
are the release. Notarius does not publish executable archives or support
|
||||
Windows as part of this work.
|
||||
|
||||
## Release Model
|
||||
|
||||
Notarius releases come from commits on `main` and use stable semantic-version
|
||||
tags in the form `vMAJOR.MINOR.PATCH`. Prerelease tags are not part of the
|
||||
initial process.
|
||||
|
||||
Every release has one nonempty, version-matched note at
|
||||
`docs/releases/<tag>.md`. The note and every affected current-state document
|
||||
must be present in the tagged commit. The Git tag and checked-in note together
|
||||
are the durable release record; no separately editable release page is
|
||||
required.
|
||||
|
||||
Published tags are immutable. A maintainer must never move, reuse, or delete a
|
||||
published tag. If a published candidate is defective, the correction is made
|
||||
on `main` and released under a new patch version. An unpublished local tag may
|
||||
be deleted when candidate inspection finds a problem before any remote push.
|
||||
|
||||
Before `v1.0.0`, a minor release may intentionally change a documented CLI,
|
||||
configuration, durable artifact, integration, or operating contract when its
|
||||
release note explains the impact and required operator action. A patch release
|
||||
must not intentionally break those documented contracts within its minor
|
||||
line.
|
||||
|
||||
The existing `v0.1.0`, `v0.2.0`, and `v0.3.0` tags remain unchanged. They
|
||||
predate this procedure and do not need retrospective release notes. The first
|
||||
release made under this process establishes the release-note series.
|
||||
|
||||
## Source-Only Distribution
|
||||
|
||||
Notarius does not publish release binaries, archives, installers, container
|
||||
images, package-manager entries, checksum files, or signatures. A release tag
|
||||
is suitable for Go-native installation and for an operator-controlled build
|
||||
from an exact checkout.
|
||||
|
||||
The primary installation form is:
|
||||
|
||||
```sh
|
||||
GOWORK=off go install \
|
||||
gitea.maximumdirect.net/eric/notarius/cmd/notarius@vMAJOR.MINOR.PATCH
|
||||
```
|
||||
|
||||
Operator documentation should also describe cloning the repository, checking
|
||||
out the tag in detached-head state, and building `./cmd/notarius` with the Go
|
||||
version declared by `go.mod`. Private-module authentication and `GOPRIVATE`
|
||||
configuration belong to the operator environment and must be documented by
|
||||
mechanism rather than with real credentials.
|
||||
|
||||
Consumers such as Narratio should pin the desired Notarius tag in provisioning
|
||||
or deployment configuration. They must continue to decide runtime
|
||||
compatibility from Notarius's published receipt and artifact schema contracts,
|
||||
not merely from the executable's product version.
|
||||
|
||||
Packaged binaries may be reconsidered if distribution demand, installation
|
||||
friction, or a broader user audience justifies their build, signing, retention,
|
||||
and platform-support costs. They are not a prerequisite for a disciplined
|
||||
release process.
|
||||
|
||||
## Platform Policy
|
||||
|
||||
Linux is the supported deployment platform. Release validation must run the
|
||||
test suite and the release build on Linux and must confirm that the command
|
||||
builds with `CGO_ENABLED=0` for Linux `amd64` and `arm64`.
|
||||
|
||||
macOS is a best-effort development and testing platform. Release validation
|
||||
should confirm that the command cross-compiles with `CGO_ENABLED=0` for Darwin
|
||||
`amd64` and `arm64`, but the project does not promise packaged artifacts or a
|
||||
separate runtime test environment for those targets.
|
||||
|
||||
Windows is unsupported. The release process must not require Windows builds,
|
||||
Windows-specific compatibility work, or Windows documentation. Platform-
|
||||
specific implementation may intentionally use Unix facilities when they are
|
||||
important to Notarius's filesystem safety and operational model. Any later
|
||||
decision to support Windows requires its own feature scope and validation
|
||||
policy.
|
||||
|
||||
## Version Reporting
|
||||
|
||||
Add a root `notarius --version` interface for deployment diagnostics. It
|
||||
prints exactly one line:
|
||||
|
||||
```text
|
||||
notarius vMAJOR.MINOR.PATCH
|
||||
```
|
||||
|
||||
when the build has a valid release version, and:
|
||||
|
||||
```text
|
||||
notarius development
|
||||
```
|
||||
|
||||
when no release version is available.
|
||||
|
||||
The implementation must obtain the main-module version from Go build
|
||||
information so `go install ...@vMAJOR.MINOR.PATCH` reports the selected tag. It
|
||||
must also accept an optional link-time version override so controlled builds
|
||||
and release CI can identify an exact tag from a checkout. The override must be
|
||||
validated and must not silently turn arbitrary text into a release version.
|
||||
Ordinary unversioned checkout builds remain `development`; the release process
|
||||
must not modify a tracked source constant for each release.
|
||||
|
||||
Version reporting is an informational product interface. It does not replace
|
||||
receipt, configuration, prompt, or artifact schema versioning, and it must not
|
||||
be used as the sole downstream compatibility check.
|
||||
|
||||
## Release Notes
|
||||
|
||||
Each new `docs/releases/<tag>.md` document has this minimum structure:
|
||||
|
||||
```markdown
|
||||
# Notarius vMAJOR.MINOR.PATCH
|
||||
|
||||
This release ...
|
||||
|
||||
## Summary
|
||||
|
||||
## Compatibility
|
||||
|
||||
## Upgrade
|
||||
|
||||
## Changes
|
||||
```
|
||||
|
||||
The note should concisely explain the release's purpose, compatibility with the
|
||||
preceding release, operator actions, and material user-visible, operational,
|
||||
integration, and maintainer-visible changes. It should link to canonical
|
||||
current-state documentation for exact contracts rather than duplicating those
|
||||
contracts.
|
||||
|
||||
Release notes are durable historical summaries. They must not contain
|
||||
credentials, private infrastructure detail, sensitive campaign material, or
|
||||
claims that are not true of the tagged candidate. A release note does not
|
||||
excuse stale current-state documentation; affected canonical documents are
|
||||
updated in the same candidate.
|
||||
|
||||
## Candidate Validation
|
||||
|
||||
The release procedure must provide copyable POSIX-shell guards that validate
|
||||
the release version, release-note filename and heading, required note sections,
|
||||
repository state, and module hygiene. Validation must be run from the Notarius
|
||||
repository root with Go workspace behavior disabled.
|
||||
|
||||
At minimum, a candidate must pass:
|
||||
|
||||
- no tracked `go.work` or `go.work.sum`, no vendored tree, and no `replace`
|
||||
directive in `go.mod`;
|
||||
- `GOWORK=off go test -count=1 ./...`;
|
||||
- `GOWORK=off go test -race -count=1 ./...`;
|
||||
- `GOWORK=off go vet ./...`;
|
||||
- `GOWORK=off go build ./...`;
|
||||
- `GOWORK=off go mod tidy -diff`;
|
||||
- `gofmt` verification for every tracked Go file;
|
||||
- `git diff --check` and `git diff --cached --check`;
|
||||
- validation of both maintained D&D configuration examples with their selected
|
||||
pipeline;
|
||||
- Linux `amd64` and `arm64` static command builds;
|
||||
- best-effort Darwin `amd64` and `arm64` static command builds; and
|
||||
- a focused manual or automated check that every added or changed local
|
||||
Markdown link resolves.
|
||||
|
||||
The candidate review also checks for generated binaries, test output,
|
||||
credentials, temporary files, module replacements, vendored dependencies, and
|
||||
other unintended source-control content. Tests remain offline and do not call
|
||||
an LLM provider or require live credentials.
|
||||
|
||||
## Candidate Publication
|
||||
|
||||
The release procedure must guard the exact commit immediately before tagging.
|
||||
It requires:
|
||||
|
||||
- the current branch is `main`;
|
||||
- the worktree and index are clean;
|
||||
- the candidate commit has been pushed and exactly matches `origin/main`;
|
||||
- the matching release note exists in that commit;
|
||||
- no local or remote tag already uses the selected version; and
|
||||
- the substantive release checks have passed for that exact candidate.
|
||||
|
||||
The maintainer records the exact candidate commit, creates a lightweight tag
|
||||
bound explicitly to that commit, verifies the local tag target, and pushes only
|
||||
that tag ref. The procedure must not recommend `git push --tags`.
|
||||
|
||||
After publication, the maintainer verifies that the remote tag resolves to the
|
||||
guarded commit and that the release note can be read from the tagged tree. A
|
||||
fresh temporary checkout or `go install ...@<tag>` must build successfully, and
|
||||
the resulting command must report the expected version through `--version`.
|
||||
|
||||
## Validation-Only Release Automation
|
||||
|
||||
Add a tag-triggered Woodpecker pipeline that validates source releases without
|
||||
publishing artifacts. It should:
|
||||
|
||||
- accept only stable semantic-version tags;
|
||||
- require the version-matched release note;
|
||||
- run the same substantive module, test, race, vet, build, formatting, and
|
||||
whitespace checks as the documented local procedure;
|
||||
- validate the maintained configuration examples;
|
||||
- perform the supported and best-effort cross-build checks; and
|
||||
- verify a release-version build's `notarius --version` output on the CI host.
|
||||
|
||||
The pipeline must not upload binaries, create archives or checksums, create or
|
||||
edit a Gitea release object, or require a release API token. Local guards remain
|
||||
authoritative before tag publication because CI begins only after the tag is
|
||||
already remote.
|
||||
|
||||
If tag validation fails, preserve the published tag, fix the cause on `main`,
|
||||
select a new patch version, and repeat the full process. Do not weaken tag
|
||||
immutability merely because the release contains source rather than binaries.
|
||||
|
||||
## Documentation Ownership
|
||||
|
||||
In the target state:
|
||||
|
||||
- `docs/release.md` owns the maintainer release procedure, commands, ordering,
|
||||
publication checks, and failure recovery;
|
||||
- `docs/releases/` owns one historical summary per release made under the new
|
||||
process;
|
||||
- `docs/cli.md` owns the `--version` contract;
|
||||
- `README.md` owns the shortest source-installation example and links to the
|
||||
release procedure where useful;
|
||||
- `docs/development.md` routes release preparation, tagging, and verification
|
||||
work to `docs/release.md`;
|
||||
- `docs/policy/documentation.md` assigns canonical ownership to the release
|
||||
procedure and release notes;
|
||||
- `docs/policy/architecture.md` records Linux support, best-effort macOS
|
||||
development, unsupported Windows, and source-only distribution only if those
|
||||
are judged durable development invariants rather than release mechanics; and
|
||||
- `docs/operations.md` describes only installation or deployment consequences
|
||||
relevant to operators and links to canonical CLI and release contracts.
|
||||
|
||||
Current-state documentation must not describe the new release process,
|
||||
`--version`, or automated validation until the corresponding behavior exists.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A maintainer can prepare, validate, tag, publish, and verify a source release
|
||||
by following `docs/release.md` without relying on undocumented knowledge.
|
||||
- Every new release has an immutable semantic-version tag and matching
|
||||
checked-in release note in the tagged commit.
|
||||
- The guarded candidate is clean, synchronized with `origin/main`, and passes
|
||||
the documented substantive checks before tagging.
|
||||
- Tag-triggered CI independently validates the published source and never
|
||||
publishes binary artifacts.
|
||||
- `go install` of a tagged version succeeds and `notarius --version` reports
|
||||
that version; ordinary unversioned builds report `development`.
|
||||
- Linux is the documented supported deployment platform, macOS has a
|
||||
best-effort development build check, and Windows is explicitly unsupported.
|
||||
- Downstream compatibility remains based on durable Notarius contracts rather
|
||||
than the product version alone.
|
||||
- Existing pre-procedure tags remain untouched and require no invented release
|
||||
history.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Publishing executable archives, installers, container images, checksums,
|
||||
signatures, or package-manager entries.
|
||||
- Supporting or cross-compiling for Windows.
|
||||
- Creating or maintaining a mutable Gitea release page.
|
||||
- Supporting prerelease tag syntax in the initial procedure.
|
||||
- Automating version selection, release-note authorship, commits, or tag
|
||||
creation.
|
||||
- Retrospectively creating release notes for `v0.1.0` through `v0.3.0`.
|
||||
- Treating a product version as a substitute for receipt, configuration,
|
||||
prompt, or artifact schema compatibility.
|
||||
7
go.mod
7
go.mod
@@ -3,9 +3,14 @@ module gitea.maximumdirect.net/eric/notarius
|
||||
go 1.25.5
|
||||
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/promptkit v0.8.0
|
||||
gitea.maximumdirect.net/eric/promptkit v0.9.0
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
require golang.org/x/text v0.40.0
|
||||
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0 // indirect
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0 // indirect
|
||||
)
|
||||
|
||||
8
go.sum
8
go.sum
@@ -1,5 +1,9 @@
|
||||
gitea.maximumdirect.net/eric/promptkit v0.8.0 h1:NGd9hDLu0UMxKbvittMrqM5Ua94eFb+kOE7UIir8l08=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.8.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.9.0 h1:IpvDRC8L6xRxQ9hpuyKOmMc5b6MeLTKYyx+h1YAjy08=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.9.0/go.mod h1:oMJ/WUJImUtwJ5e+6MAGECPYAErAkOaKel0G+3T/b4E=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0 h1:lc062euk2qseO//D762i3JaFyulDNML3eQQX7DkYTho=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0/go.mod h1:AIa7kAu2mfrRQgcspe4L+DW51WqgnALQT60lqkEywJI=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0 h1:j9YY7wsTVjzke2kHH4YAzpU0oUpM+x+nXwl1IeS+2eg=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0/go.mod h1:4RNS+LILDg4JbS4Ts9Lwy1C92wauXJIbeQaalps4Koo=
|
||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
||||
|
||||
@@ -21,6 +21,10 @@ import (
|
||||
|
||||
const assembledSpellExtractorKey = "test/dnd/spell-casts"
|
||||
|
||||
const assembledCorrectingSpellExtractorKey = "test/dnd/correcting-spell-casts"
|
||||
|
||||
const assembledDirectSpellValidatorKey = "test/dnd/direct-spell-correction"
|
||||
|
||||
func TestAssembledSpellPipelineNormalizesMergedCasts(t *testing.T) {
|
||||
registries, resolved, extractor := assembledSpellPipeline(t, assembledSpellPipelineOptions{})
|
||||
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
|
||||
@@ -101,6 +105,30 @@ func TestAssembledSpellPipelineNormalizesMergedCasts(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssembledSpellPipelineCorrectsRejectedDirectExtraction(t *testing.T) {
|
||||
registries, resolved, extractor := assembledCorrectingSpellPipeline(t)
|
||||
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
output, err := pipeline.New().Run(context.Background(), pipeline.RunInput{
|
||||
Prepared: prepared,
|
||||
RawInput: readRepositoryFile(t, "examples", "seriatim-minimal-transcript.json"),
|
||||
ChunkCacheMode: pipeline.ChunkCacheBypass,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v, want nil", err)
|
||||
}
|
||||
if len(output.Rejected) != 0 || output.Manifest.ValidationStatus != "approved" || len(output.NormalizeOutputs) != 1 {
|
||||
t.Fatalf("run output = %#v, want corrected accepted spell output", output)
|
||||
}
|
||||
correction := extractor.correctionSnapshot()
|
||||
if correction == nil || string(correction.AssistantResponse) != `{"spell":"Mysterious Burst"}` || !strings.Contains(correction.UserGuidance, "use a known spell name") || !strings.Contains(correction.UserGuidance, "complete corrected replacement") || strings.Contains(correction.UserGuidance, "unknown_spell") || strings.Contains(correction.UserGuidance, "spell is not in the catalog") {
|
||||
t.Fatalf("extract correction = %#v, want exact rejected model response and semantic replacement guidance only", correction)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssembledSpellPipelineHonorsNormalizeValidatorOverride(t *testing.T) {
|
||||
registries, resolved, _ := assembledSpellPipeline(t, assembledSpellPipelineOptions{normalizeValidatorOverride: true})
|
||||
var normalizeChain *pipeline.ResolvedValidatorChain
|
||||
@@ -136,6 +164,7 @@ func TestAssembledSpellPipelineHonorsNormalizeValidatorOverride(t *testing.T) {
|
||||
|
||||
func TestAssembledSpellPipelinePromotesTerminalUnknownSpellWarning(t *testing.T) {
|
||||
registries, resolved, _ := assembledSpellPipeline(t, assembledSpellPipelineOptions{unknownSpell: true})
|
||||
resolved.Steps[0].ArtifactLanes[0].NormalizeValidationPolicy.SemanticRejection = pipeline.SemanticRejectionRejectOutput
|
||||
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||
@@ -161,8 +190,8 @@ func TestAssembledSpellPipelinePromotesTerminalUnknownSpellWarning(t *testing.T)
|
||||
if !reflect.DeepEqual(rejectedFile.Rejected, output.Rejected) {
|
||||
t.Fatalf("rejected file = %#v, run rejections = %#v, want durable rejection diagnostic", rejectedFile.Rejected, output.Rejected)
|
||||
}
|
||||
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != spellnormalize.ReasonCodeSpellNameUnresolved || output.Warnings[0].Scope != "spell_casts[0]" {
|
||||
t.Fatalf("warnings = %#v, want terminal normalize catalog warning", output.Warnings)
|
||||
if len(output.Warnings) != 2 || output.Warnings[0].ReasonCode != spellnormalize.ReasonCodeSpellNameUnresolved || output.Warnings[0].Scope != "spell_casts[0]" || output.Warnings[1].ReasonCode != "spell_not_near_source" {
|
||||
t.Fatalf("warnings = %#v, want complete terminal normalize validation warnings", output.Warnings)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -206,6 +235,47 @@ type assembledSpellPipelineOptions struct {
|
||||
unknownSpell bool
|
||||
}
|
||||
|
||||
func assembledCorrectingSpellPipeline(t *testing.T) (pipeline.Registries, pipeline.ResolvedPipeline, *assembledCorrectingSpellExtractor) {
|
||||
t.Helper()
|
||||
components := productionTestComponents(t)
|
||||
extractor := &assembledCorrectingSpellExtractor{}
|
||||
if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{
|
||||
Key: assembledCorrectingSpellExtractorKey,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
|
||||
Requires: []string{"chunks", "source.transcript"},
|
||||
Provides: []string{"dnd.spell_casts"},
|
||||
ArtifactKind: dnd.SpellListKind,
|
||||
}, func() (contracts.Extractor[dnd.SpellList], error) {
|
||||
return extractor, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register correcting extractor: %v", err)
|
||||
}
|
||||
if err := pipeline.RegisterTypedValidator[dnd.SpellList](components.registries.Validators, dnd.SpellListKind, pipeline.ValidatorSpec{Key: assembledDirectSpellValidatorKey, ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.TypedValidator[dnd.SpellList], error) {
|
||||
return assembledDirectSpellValidator{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register direct spell validator: %v", err)
|
||||
}
|
||||
|
||||
extract := pipeline.Binding(assembledCorrectingSpellExtractorKey)
|
||||
extract.Retries = 1
|
||||
extract.Validators = pipeline.ValidatorOverride{Set: true, Validators: []pipeline.ModuleBinding{{Module: assembledDirectSpellValidatorKey}}}
|
||||
resolved, err := pipeline.ResolvePipeline(pipeline.PipelineProfile{
|
||||
ID: "assembled-dnd-correcting-spells",
|
||||
Input: pipeline.Binding("seriatim"),
|
||||
Chunk: pipeline.ModuleBinding{Module: "generic", Options: map[string]any{"max_units": 1}},
|
||||
Artifacts: map[string]pipeline.ArtifactLaneProfile{
|
||||
"spells": {Extract: extract, Normalize: pipeline.Binding(spellnormalize.Key)},
|
||||
},
|
||||
Output: pipeline.Binding("json"),
|
||||
}, pipeline.ResolveOptions{}, catalogFromRegistries(components.registries))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
return components.registries, resolved, extractor
|
||||
}
|
||||
|
||||
func assembledSpellPipeline(t *testing.T, options assembledSpellPipelineOptions) (pipeline.Registries, pipeline.ResolvedPipeline, *assembledSpellExtractor) {
|
||||
t.Helper()
|
||||
components := productionTestComponents(t)
|
||||
@@ -251,6 +321,69 @@ type assembledSpellExtractor struct {
|
||||
unknownSpell bool
|
||||
}
|
||||
|
||||
type assembledCorrectingSpellExtractor struct {
|
||||
mu sync.Mutex
|
||||
correction *contracts.SemanticCorrection
|
||||
}
|
||||
|
||||
func (*assembledCorrectingSpellExtractor) Key() string { return assembledCorrectingSpellExtractorKey }
|
||||
|
||||
func (*assembledCorrectingSpellExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||
|
||||
func (e *assembledCorrectingSpellExtractor) Extract(_ context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[dnd.SpellList], error) {
|
||||
if req.Source == nil || req.Chunk == nil {
|
||||
return contracts.TypedExtractionResult[dnd.SpellList]{}, fmt.Errorf("correcting assembled extractor requires source and chunk")
|
||||
}
|
||||
response := `{"spell":"accepted"}`
|
||||
value := dnd.SpellList{SpellCasts: []dnd.SpellCast{}}
|
||||
if req.Chunk.Index == 0 && req.Correction == nil {
|
||||
response = `{"spell":"Mysterious Burst"}`
|
||||
value.SpellCasts = []dnd.SpellCast{{Caster: "Aria", Spell: "Mysterious Burst", SourceRefs: []source.SourceRef{{SourceID: req.Source.ID, StartUnitID: 1, EndUnitID: 1}}}}
|
||||
}
|
||||
if req.Chunk.Index == 0 && req.Correction != nil {
|
||||
correction, err := contracts.CloneSemanticCorrection(req.Correction)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.SpellList]{}, err
|
||||
}
|
||||
e.mu.Lock()
|
||||
e.correction = correction
|
||||
e.mu.Unlock()
|
||||
value.SpellCasts = []dnd.SpellCast{{Caster: "Aria", Spell: "Cure Wounds", SourceRefs: []source.SourceRef{{SourceID: req.Source.ID, StartUnitID: 1, EndUnitID: 1}}}}
|
||||
}
|
||||
candidate, err := contracts.NewModelCandidate([]byte(response), contracts.CorrectionProtocolSingleResponseV1)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.SpellList]{}, err
|
||||
}
|
||||
return contracts.TypedExtractionResult[dnd.SpellList]{Value: value, ModelCandidate: candidate}, nil
|
||||
}
|
||||
|
||||
func (e *assembledCorrectingSpellExtractor) correctionSnapshot() *contracts.SemanticCorrection {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
correction, err := contracts.CloneSemanticCorrection(e.correction)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return correction
|
||||
}
|
||||
|
||||
type assembledDirectSpellValidator struct{}
|
||||
|
||||
func (assembledDirectSpellValidator) Name() string { return assembledDirectSpellValidatorKey }
|
||||
|
||||
func (assembledDirectSpellValidator) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassDeterministic
|
||||
}
|
||||
|
||||
func (assembledDirectSpellValidator) Validate(_ context.Context, req contracts.TypedValidationRequest[dnd.SpellList]) (contracts.ValidationResult, error) {
|
||||
for _, cast := range req.Value.SpellCasts {
|
||||
if cast.Spell == "Mysterious Burst" {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "unknown_spell", Message: "spell is not in the catalog", CorrectionGuidance: "use a known spell name"}, nil
|
||||
}
|
||||
}
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
|
||||
func (e *assembledSpellExtractor) Key() string { return assembledSpellExtractorKey }
|
||||
|
||||
func (*assembledSpellExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||
|
||||
@@ -396,8 +396,12 @@ func (client *enemyEventLLMClient) CompleteStructured(ctx context.Context, reque
|
||||
if err := json.Unmarshal(content, output); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate fake structured target: %w", err)
|
||||
}
|
||||
snapshot, err := contracts.CloneStructuredCompletionRequest(request)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("clone fake request: %w", err)
|
||||
}
|
||||
client.mu.Lock()
|
||||
client.requests = append(client.requests, request)
|
||||
client.requests = append(client.requests, snapshot)
|
||||
client.mu.Unlock()
|
||||
return contracts.StructuredCompletionResponse{Content: content, Provider: "test", Model: "deterministic", ProfileID: request.ProfileID}, nil
|
||||
}
|
||||
@@ -408,7 +412,11 @@ func (client *enemyEventLLMClient) requestsFor(promptID string) []contracts.Stru
|
||||
var requests []contracts.StructuredCompletionRequest
|
||||
for _, request := range client.requests {
|
||||
if request.PromptID == promptID {
|
||||
requests = append(requests, request)
|
||||
snapshot, err := contracts.CloneStructuredCompletionRequest(request)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
requests = append(requests, snapshot)
|
||||
}
|
||||
}
|
||||
return requests
|
||||
|
||||
@@ -343,6 +343,9 @@ func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
|
||||
Inputs []struct {
|
||||
Name string `yaml:"name"`
|
||||
} `yaml:"inputs"`
|
||||
Messages []struct {
|
||||
Role string `yaml:"role"`
|
||||
} `yaml:"messages"`
|
||||
}
|
||||
preparedPrompts := 0
|
||||
if err := fs.WalkDir(promptFS, ".", func(path string, entry fs.DirEntry, walkErr error) error {
|
||||
@@ -360,6 +363,11 @@ func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
|
||||
if err := yaml.Unmarshal(data, &prompt); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, message := range prompt.Messages {
|
||||
if message.Role != promptkit.RoleSystem && message.Role != promptkit.RoleUser {
|
||||
return fmt.Errorf("production prompt %q uses role %q, want system or user", prompt.ID, message.Role)
|
||||
}
|
||||
}
|
||||
inputs := make(map[string]promptkit.ArtifactRef, len(prompt.Inputs))
|
||||
for _, input := range prompt.Inputs {
|
||||
inputs[input.Name] = promptkit.Inline(`{}`)
|
||||
@@ -1091,8 +1099,12 @@ func (client *productionFakeLLMClient) CompleteStructured(ctx context.Context, r
|
||||
if err := json.Unmarshal(content, out); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate fake structured target: %w", err)
|
||||
}
|
||||
snapshot, err := contracts.CloneStructuredCompletionRequest(req)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("clone fake request: %w", err)
|
||||
}
|
||||
client.mu.Lock()
|
||||
client.requests = append(client.requests, req)
|
||||
client.requests = append(client.requests, snapshot)
|
||||
client.mu.Unlock()
|
||||
return contracts.StructuredCompletionResponse{Content: content, Provider: "test", Model: "deterministic", ProfileID: req.ProfileID}, nil
|
||||
}
|
||||
@@ -1103,7 +1115,11 @@ func (client *productionFakeLLMClient) requestsFor(promptID string) []contracts.
|
||||
var requests []contracts.StructuredCompletionRequest
|
||||
for _, req := range client.requests {
|
||||
if req.PromptID == promptID {
|
||||
requests = append(requests, req)
|
||||
snapshot, err := contracts.CloneStructuredCompletionRequest(req)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
requests = append(requests, snapshot)
|
||||
}
|
||||
}
|
||||
return requests
|
||||
|
||||
@@ -7,22 +7,24 @@ import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
)
|
||||
|
||||
const runResultSchemaVersion = "notarius.run-result.v1"
|
||||
|
||||
type runResult struct {
|
||||
SchemaVersion string `json:"schema_version"`
|
||||
RunID string `json:"run_id"`
|
||||
PipelineID string `json:"pipeline_id"`
|
||||
OutputDirectory string `json:"output_directory"`
|
||||
IndexFile string `json:"index_file,omitempty"`
|
||||
NormalizedOutputCount int `json:"normalized_output_count"`
|
||||
RejectedOutputCount int `json:"rejected_output_count"`
|
||||
WarningCount int `json:"warning_count"`
|
||||
ValidationStatus string `json:"validation_status"`
|
||||
DebugDirectory string `json:"debug_directory,omitempty"`
|
||||
SchemaVersion string `json:"schema_version"`
|
||||
RunID string `json:"run_id"`
|
||||
PipelineID string `json:"pipeline_id"`
|
||||
OutputDirectory string `json:"output_directory"`
|
||||
IndexFile string `json:"index_file,omitempty"`
|
||||
NormalizedOutputCount int `json:"normalized_output_count"`
|
||||
RejectedOutputCount int `json:"rejected_output_count"`
|
||||
WarningCount int `json:"warning_count"`
|
||||
ValidationStatus string `json:"validation_status"`
|
||||
ValidationSummaries []artifacts.ValidationSummary `json:"validation_summaries,omitempty"`
|
||||
DebugDirectory string `json:"debug_directory,omitempty"`
|
||||
}
|
||||
|
||||
func newRunResult(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput, outputDirectory, debugDirectory string) (runResult, error) {
|
||||
@@ -59,6 +61,7 @@ func newRunResult(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput,
|
||||
RejectedOutputCount: len(output.Rejected),
|
||||
WarningCount: len(output.Warnings),
|
||||
ValidationStatus: output.Manifest.ValidationStatus,
|
||||
ValidationSummaries: cloneValidationSummaries(output.Manifest.ValidationSummaries),
|
||||
}
|
||||
|
||||
if strings.TrimSpace(debugDirectory) != "" {
|
||||
@@ -85,6 +88,17 @@ func newRunResult(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput,
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func cloneValidationSummaries(summaries []artifacts.ValidationSummary) []artifacts.ValidationSummary {
|
||||
if len(summaries) == 0 {
|
||||
return nil
|
||||
}
|
||||
cloned := make([]artifacts.ValidationSummary, len(summaries))
|
||||
for index, summary := range summaries {
|
||||
cloned[index] = artifacts.CloneValidationSummary(summary)
|
||||
}
|
||||
return cloned
|
||||
}
|
||||
|
||||
func encodeRunResult(result runResult) ([]byte, error) {
|
||||
encoded, err := json.Marshal(result)
|
||||
if err != nil {
|
||||
|
||||
@@ -92,7 +92,7 @@ func TestRunResultReportsSuccessfulRejection(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
configBytes = []byte(replaceRequiredOnce(t, string(configBytes), " normalize: test/normalize\n", " normalize:\n module: test/normalize\n validators:\n - generic/always_reject\n"))
|
||||
configBytes = []byte(replaceRequiredOnce(t, string(configBytes), " normalize: test/normalize\n", " normalize:\n module: test/normalize\n validators:\n - generic/always_reject\n validation_policy:\n semantic_rejection: reject_output\n"))
|
||||
if err := os.WriteFile(roots.config, configBytes, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
@@ -55,6 +55,9 @@ func TestRunResultEncodesRequiredFieldsAndCounts(t *testing.T) {
|
||||
if got := decoded["warning_count"]; got != float64(1) {
|
||||
t.Fatalf("warning_count = %v", got)
|
||||
}
|
||||
if got := decoded["validation_summaries"]; got != nil {
|
||||
t.Fatalf("validation_summaries = %#v, want omitted when empty", got)
|
||||
}
|
||||
if got := decoded["output_directory"]; got != filepath.Join(mustWorkingDirectory(t), "relative-output") {
|
||||
t.Fatalf("output_directory = %q", got)
|
||||
}
|
||||
@@ -112,6 +115,26 @@ func TestRunResultOmitsIndexFileForOtherOutputModules(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunResultProjectsOwnedValidationSummaries(t *testing.T) {
|
||||
output := testRunOutput()
|
||||
output.Manifest.ValidationSummaries = []artifacts.ValidationSummary{{Status: "incomplete", IncompleteValidators: []string{"validator"}, ProducerAttemptCount: 1, TerminalAction: "warn_continue"}}
|
||||
result, err := newRunResult(testResolvedPipeline(pipeline.DefaultOutputModule), output, "output", "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
output.Manifest.ValidationSummaries[0].IncompleteValidators[0] = "caller mutation"
|
||||
if got := result.ValidationSummaries[0].IncompleteValidators; len(got) != 1 || got[0] != "validator" {
|
||||
t.Fatalf("result validation summaries = %#v", result.ValidationSummaries)
|
||||
}
|
||||
encoded, err := encodeRunResult(result)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !bytes.Contains(encoded, []byte(`"validation_summaries":[{"status":"incomplete","incomplete_validators":["validator"],"producer_attempt_count":1,"terminal_action":"warn_continue"}]`)) {
|
||||
t.Fatalf("encoded result = %s", encoded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunResultRequiresOneProductionIndexFile(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
|
||||
@@ -65,6 +65,7 @@ func TestProductionSpellCatalogValidationRetries(t *testing.T) {
|
||||
t.Fatalf("materialize production references: %v", err)
|
||||
}
|
||||
materialized.Steps[0].ArtifactLanes[0].Extract.Retries = retries
|
||||
materialized.Steps[0].ArtifactLanes[0].ExtractValidationPolicy.SemanticRejection = pipeline.SemanticRejectionRejectOutput
|
||||
|
||||
llmClient := &catalogRetryLLMClient{responses: tt.responses}
|
||||
prepared, err := pipeline.Prepare(materialized, components.registries, pipeline.ModuleDependencies{LLM: llmClient})
|
||||
@@ -91,8 +92,8 @@ func TestProductionSpellCatalogValidationRetries(t *testing.T) {
|
||||
if rejection.ReasonCode != "unknown_spell" || rejection.AttemptCount != retries+1 {
|
||||
t.Fatalf("rejection = %#v, want exhausted unknown-spell rejection", rejection)
|
||||
}
|
||||
if len(output.Warnings) != 0 {
|
||||
t.Fatalf("warnings = %#v, want no emitted warnings from rejected attempts", output.Warnings)
|
||||
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != "spell_not_near_source" {
|
||||
t.Fatalf("warnings = %#v, want complete terminal validation warnings", output.Warnings)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
@@ -93,17 +93,44 @@ type NormalizedOutputManifest struct {
|
||||
}
|
||||
|
||||
type RejectedOutputManifest struct {
|
||||
Stage string `json:"stage"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
ChunkID string `json:"chunk_id,omitempty"`
|
||||
ChunkIndex int `json:"chunk_index,omitempty"`
|
||||
ValidatorName string `json:"validator_name,omitempty"`
|
||||
ReasonCode string `json:"reason_code,omitempty"`
|
||||
Message string `json:"message,omitempty"`
|
||||
AttemptCount int `json:"attempt_count,omitempty"`
|
||||
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
|
||||
Stage string `json:"stage"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
ChunkID string `json:"chunk_id,omitempty"`
|
||||
ChunkIndex int `json:"chunk_index,omitempty"`
|
||||
ValidatorName string `json:"validator_name,omitempty"`
|
||||
ReasonCode string `json:"reason_code,omitempty"`
|
||||
Message string `json:"message,omitempty"`
|
||||
AttemptCount int `json:"attempt_count,omitempty"`
|
||||
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
|
||||
Validation *ValidationSummary `json:"validation,omitempty"`
|
||||
}
|
||||
|
||||
// ValidationSummary is the bounded, durable outcome of validating one
|
||||
// producer result. It deliberately contains identities and stable codes, not
|
||||
// model responses, corrective guidance, validator diagnostics, or payloads.
|
||||
type ValidationSummary struct {
|
||||
Stage string `json:"stage,omitempty"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
ChunkID string `json:"chunk_id,omitempty"`
|
||||
ChunkIndex int `json:"chunk_index,omitempty"`
|
||||
Status string `json:"status"`
|
||||
RejectingValidators []string `json:"rejecting_validators,omitempty"`
|
||||
ReasonCodes []string `json:"reason_codes,omitempty"`
|
||||
IncompleteValidators []string `json:"incomplete_validators,omitempty"`
|
||||
ProducerAttemptCount int `json:"producer_attempt_count"`
|
||||
TerminalAction string `json:"terminal_action"`
|
||||
}
|
||||
|
||||
// CloneValidationSummary returns an independently owned durable summary.
|
||||
func CloneValidationSummary(summary ValidationSummary) ValidationSummary {
|
||||
summary.RejectingValidators = append([]string(nil), summary.RejectingValidators...)
|
||||
summary.ReasonCodes = append([]string(nil), summary.ReasonCodes...)
|
||||
summary.IncompleteValidators = append([]string(nil), summary.IncompleteValidators...)
|
||||
return summary
|
||||
}
|
||||
|
||||
type CheckpointDecisionManifest struct {
|
||||
@@ -163,6 +190,7 @@ type RunManifest struct {
|
||||
References []ReferenceProvenance `json:"references,omitempty"`
|
||||
NormalizedOutputs []NormalizedOutputManifest `json:"normalized_outputs,omitempty"`
|
||||
RejectedOutputs []RejectedOutputManifest `json:"rejected_outputs,omitempty"`
|
||||
ValidationSummaries []ValidationSummary `json:"validation_summaries,omitempty"`
|
||||
CheckpointDecisions []CheckpointDecisionManifest `json:"checkpoint_decisions,omitempty"`
|
||||
LLMProfiles []LLMProfileManifest `json:"llm_profiles,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
|
||||
@@ -116,6 +116,7 @@ func (c *ConcurrencyConfig) recomputeStageWorkerDefaults() {
|
||||
|
||||
func clonePipelineProfile(in pipeline.PipelineProfile) pipeline.PipelineProfile {
|
||||
out := in
|
||||
out.ValidationPolicy = cloneValidationPolicyOverride(in.ValidationPolicy)
|
||||
if in.StructuredOutputRepairAttempts != nil {
|
||||
value := *in.StructuredOutputRepairAttempts
|
||||
out.StructuredOutputRepairAttempts = &value
|
||||
@@ -200,6 +201,7 @@ func cloneReferenceSource(in pipeline.ReferenceSource) pipeline.ReferenceSource
|
||||
|
||||
func cloneModuleBinding(in pipeline.ModuleBinding) pipeline.ModuleBinding {
|
||||
out := in
|
||||
out.ValidationPolicy = cloneValidationPolicyOverride(in.ValidationPolicy)
|
||||
if in.StructuredOutputRepairAttempts != nil {
|
||||
value := *in.StructuredOutputRepairAttempts
|
||||
out.StructuredOutputRepairAttempts = &value
|
||||
@@ -212,6 +214,26 @@ func cloneModuleBinding(in pipeline.ModuleBinding) pipeline.ModuleBinding {
|
||||
return out
|
||||
}
|
||||
|
||||
func cloneValidationPolicyOverride(in *pipeline.ValidationPolicyOverride) *pipeline.ValidationPolicyOverride {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := *in
|
||||
if in.ProducerStructuralFailure != nil {
|
||||
value := *in.ProducerStructuralFailure
|
||||
out.ProducerStructuralFailure = &value
|
||||
}
|
||||
if in.SemanticRejection != nil {
|
||||
value := *in.SemanticRejection
|
||||
out.SemanticRejection = &value
|
||||
}
|
||||
if in.ValidatorFailure != nil {
|
||||
value := *in.ValidatorFailure
|
||||
out.ValidatorFailure = &value
|
||||
}
|
||||
return &out
|
||||
}
|
||||
|
||||
func cloneValidatorOverride(in pipeline.ValidatorOverride) pipeline.ValidatorOverride {
|
||||
out := pipeline.ValidatorOverride{Set: in.Set}
|
||||
if len(in.Validators) > 0 {
|
||||
|
||||
@@ -37,6 +37,7 @@ type FilePromptKitLocalBackendConfig struct {
|
||||
type FilePipelineProfile struct {
|
||||
LLMProfile *string `yaml:"llm_profile,omitempty"`
|
||||
StructuredOutputRepairAttempts *int `yaml:"structured_output_repair_attempts,omitempty"`
|
||||
ValidationPolicy *pipeline.ValidationPolicyOverride `yaml:"validation_policy,omitempty"`
|
||||
Input fileModuleBinding `yaml:"input"`
|
||||
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
|
||||
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
|
||||
@@ -55,12 +56,19 @@ func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
||||
type plainFilePipelineProfile FilePipelineProfile
|
||||
var decoded plainFilePipelineProfile
|
||||
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
|
||||
"llm_profile": {}, "structured_output_repair_attempts": {}, "input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
|
||||
"llm_profile": {}, "structured_output_repair_attempts": {}, "validation_policy": {}, "input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
|
||||
}, "pipeline profile")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
*p = FilePipelineProfile(decoded)
|
||||
if validationPolicyNode, ok := mappingValue(node, "validation_policy"); ok {
|
||||
policy, err := parseValidationPolicy(validationPolicyNode, "pipeline profile")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
p.ValidationPolicy = policy
|
||||
}
|
||||
_, p.artifactsSet = seen["artifacts"]
|
||||
_, p.stepsSet = seen["steps"]
|
||||
_, p.llmProfileSet = seen["llm_profile"]
|
||||
@@ -154,6 +162,7 @@ type fileModuleBinding struct {
|
||||
Module string
|
||||
LLMProfile string
|
||||
StructuredOutputRepairAttempts *int
|
||||
ValidationPolicy *pipeline.ValidationPolicyOverride
|
||||
Retries int
|
||||
Options map[string]any
|
||||
References map[string]fileReferenceSource
|
||||
@@ -250,9 +259,14 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
|
||||
b.Module = strings.TrimSpace(module)
|
||||
return nil
|
||||
case yaml.MappingNode:
|
||||
seen := make(map[string]struct{}, len(node.Content)/2)
|
||||
for i := 0; i < len(node.Content); i += 2 {
|
||||
keyNode := node.Content[i]
|
||||
valueNode := node.Content[i+1]
|
||||
if _, exists := seen[keyNode.Value]; exists {
|
||||
return fmt.Errorf("module binding field %q is duplicated", keyNode.Value)
|
||||
}
|
||||
seen[keyNode.Value] = struct{}{}
|
||||
switch keyNode.Value {
|
||||
case "module":
|
||||
var module string
|
||||
@@ -275,6 +289,12 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
|
||||
return err
|
||||
}
|
||||
b.StructuredOutputRepairAttempts = attempts
|
||||
case "validation_policy":
|
||||
policy, err := parseValidationPolicy(valueNode, "module binding")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
b.ValidationPolicy = policy
|
||||
case "retries":
|
||||
var retries int
|
||||
if err := valueNode.Decode(&retries); err != nil {
|
||||
@@ -318,6 +338,7 @@ func (b fileModuleBinding) toPipelineBinding() pipeline.ModuleBinding {
|
||||
Module: strings.TrimSpace(b.Module),
|
||||
LLMProfile: strings.TrimSpace(b.LLMProfile),
|
||||
StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(b.StructuredOutputRepairAttempts),
|
||||
ValidationPolicy: cloneValidationPolicyOverride(b.ValidationPolicy),
|
||||
Retries: b.Retries,
|
||||
Options: cloneOptions(b.Options),
|
||||
References: fileReferenceSourcesToPipeline(b.References),
|
||||
@@ -325,6 +346,51 @@ func (b fileModuleBinding) toPipelineBinding() pipeline.ModuleBinding {
|
||||
}
|
||||
}
|
||||
|
||||
func mappingValue(node *yaml.Node, key string) (*yaml.Node, bool) {
|
||||
for i := 0; i < len(node.Content); i += 2 {
|
||||
if node.Content[i].Value == key {
|
||||
return node.Content[i+1], true
|
||||
}
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
func parseValidationPolicy(node *yaml.Node, context string) (*pipeline.ValidationPolicyOverride, error) {
|
||||
if node == nil || node.Tag == "!!null" || node.Kind != yaml.MappingNode {
|
||||
return nil, fmt.Errorf("%s validation_policy must be an object", context)
|
||||
}
|
||||
policy := &pipeline.ValidationPolicyOverride{}
|
||||
seen := make(map[string]struct{}, len(node.Content)/2)
|
||||
for i := 0; i < len(node.Content); i += 2 {
|
||||
key := node.Content[i].Value
|
||||
value := node.Content[i+1]
|
||||
if _, exists := seen[key]; exists {
|
||||
return nil, fmt.Errorf("%s validation_policy field %q is duplicated", context, key)
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
if value.Tag == "!!null" || value.Kind != yaml.ScalarNode || value.Tag != "!!str" {
|
||||
return nil, fmt.Errorf("%s validation_policy.%s must be a string", context, key)
|
||||
}
|
||||
switch key {
|
||||
case "producer_structural_failure":
|
||||
value := pipeline.ProducerStructuralFailureAction(value.Value)
|
||||
policy.ProducerStructuralFailure = &value
|
||||
case "semantic_rejection":
|
||||
value := pipeline.SemanticRejectionAction(value.Value)
|
||||
policy.SemanticRejection = &value
|
||||
case "validator_failure":
|
||||
value := pipeline.ValidatorFailureAction(value.Value)
|
||||
policy.ValidatorFailure = &value
|
||||
default:
|
||||
return nil, fmt.Errorf("field %s not found in %s validation_policy", key, context)
|
||||
}
|
||||
}
|
||||
if err := policy.Validate(); err != nil {
|
||||
return nil, fmt.Errorf("%s validation_policy: %w", context, err)
|
||||
}
|
||||
return policy, nil
|
||||
}
|
||||
|
||||
func validateStructuredOutputRepairAttemptsNode(node *yaml.Node, context string) error {
|
||||
if node.Kind != yaml.MappingNode {
|
||||
return fmt.Errorf("%s must be an object", context)
|
||||
@@ -573,6 +639,7 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
||||
ID: pipelineID,
|
||||
LLMProfile: llmProfile,
|
||||
StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(filePipeline.StructuredOutputRepairAttempts),
|
||||
ValidationPolicy: cloneValidationPolicyOverride(filePipeline.ValidationPolicy),
|
||||
Input: filePipeline.Input.toPipelineBinding(),
|
||||
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
|
||||
References: fileReferenceSourcesToPipeline(filePipeline.References),
|
||||
|
||||
@@ -32,6 +32,7 @@ func (e EffectiveConfig) RedactedResolvedPipelinePayload() pipeline.ResolvedPipe
|
||||
|
||||
func cloneResolvedPipeline(in pipeline.ResolvedPipeline) pipeline.ResolvedPipeline {
|
||||
out := in
|
||||
out.ConfiguredValidationPolicy = cloneValidationPolicyOverride(in.ConfiguredValidationPolicy)
|
||||
out.Input = redactBinding(cloneModuleBinding(in.Input))
|
||||
out.Chunk = redactBinding(cloneModuleBinding(in.Chunk))
|
||||
out.ChunkReferences = pipeline.CloneReferenceTarget(in.ChunkReferences)
|
||||
|
||||
@@ -225,6 +225,39 @@ func TestRedactedResolvedPipelinePayloadHandlesTypedOptionContainers(t *testing.
|
||||
}
|
||||
}
|
||||
|
||||
func TestRedactedEffectiveConfigPayloadOwnsValidationPolicies(t *testing.T) {
|
||||
semantic := pipeline.SemanticRejectionRejectOutput
|
||||
validator := pipeline.ValidatorFailureFailRun
|
||||
configured := &pipeline.ValidationPolicyOverride{SemanticRejection: &semantic, ValidatorFailure: &validator}
|
||||
effective := EffectiveConfig{
|
||||
Config: Config{Pipelines: map[string]pipeline.PipelineProfile{
|
||||
"main": {ValidationPolicy: configured},
|
||||
}},
|
||||
ResolvedPipeline: pipeline.ResolvedPipeline{
|
||||
ConfiguredValidationPolicy: configured,
|
||||
ChunkValidationPolicy: pipeline.ValidationPolicy{
|
||||
ProducerStructuralFailure: pipeline.ProducerStructuralFailureFailRun,
|
||||
SemanticRejection: pipeline.SemanticRejectionRejectOutput,
|
||||
ValidatorFailure: pipeline.ValidatorFailureFailRun,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
payload := effective.RedactedSummaryPayload().(EffectiveConfig)
|
||||
encoded, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(encoded), `"configured_validation_policy":{"semantic_rejection":"reject_output","validator_failure":"fail_run"}`) || !strings.Contains(string(encoded), `"chunk_validation_policy":{"producer_structural_failure":"fail_run","semantic_rejection":"reject_output","validator_failure":"fail_run"}`) {
|
||||
t.Fatalf("redacted payload omitted validation policy: %s", encoded)
|
||||
}
|
||||
*payload.Config.Pipelines["main"].ValidationPolicy.SemanticRejection = pipeline.SemanticRejectionFailRun
|
||||
*payload.ResolvedPipeline.ConfiguredValidationPolicy.ValidatorFailure = pipeline.ValidatorFailureWarnContinue
|
||||
if *effective.Config.Pipelines["main"].ValidationPolicy.SemanticRejection != pipeline.SemanticRejectionRejectOutput || *effective.ResolvedPipeline.ConfiguredValidationPolicy.ValidatorFailure != pipeline.ValidatorFailureFailRun {
|
||||
t.Fatal("redacted payload aliases validation policy")
|
||||
}
|
||||
}
|
||||
|
||||
func redactionTestBinding(name string) pipeline.ModuleBinding {
|
||||
return pipeline.ModuleBinding{
|
||||
Module: "safe-" + name,
|
||||
|
||||
@@ -119,6 +119,11 @@ func validatePipelineProfiles(profiles map[string]pipeline.PipelineProfile) erro
|
||||
if err := validateStructuredOutputRepairAttempts(fmt.Sprintf("pipeline %q", id), profile.StructuredOutputRepairAttempts); err != nil {
|
||||
return err
|
||||
}
|
||||
if profile.ValidationPolicy != nil {
|
||||
if err := profile.ValidationPolicy.Validate(); err != nil {
|
||||
return fmt.Errorf("pipeline %q validation_policy: %w", id, err)
|
||||
}
|
||||
}
|
||||
if err := validateBinding(id, "", "input", profile.Input, false); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -198,6 +203,16 @@ func validateBinding(
|
||||
binding pipeline.ModuleBinding,
|
||||
referencesAllowed bool,
|
||||
) error {
|
||||
if binding.ValidationPolicy != nil {
|
||||
switch slot {
|
||||
case "chunk", "extract", "merge", "normalize":
|
||||
default:
|
||||
return fmt.Errorf("%s validation_policy is not supported", referenceContext(pipelineID, laneID, slot))
|
||||
}
|
||||
if err := binding.ValidationPolicy.Validate(); err != nil {
|
||||
return fmt.Errorf("%s validation_policy: %w", referenceContext(pipelineID, laneID, slot), err)
|
||||
}
|
||||
}
|
||||
if err := validateStructuredOutputRepairAttempts(referenceContext(pipelineID, laneID, slot), binding.StructuredOutputRepairAttempts); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -255,8 +270,11 @@ func validateValidatorOverride(pipelineID string, laneID string, slot string, ov
|
||||
if validator.Validators.Set {
|
||||
return fmt.Errorf("%s nested validators are not supported", context)
|
||||
}
|
||||
if validator.Retries != 0 {
|
||||
return fmt.Errorf("%s retries are not supported", context)
|
||||
if validator.ValidationPolicy != nil {
|
||||
return fmt.Errorf("%s validation_policy is not supported", context)
|
||||
}
|
||||
if validator.Retries < 0 {
|
||||
return fmt.Errorf("%s retries must be greater than or equal to zero", context)
|
||||
}
|
||||
if validator.LLMProfile != "" && strings.TrimSpace(validator.LLMProfile) == "" {
|
||||
return fmt.Errorf("%s llm_profile must not be empty when set", context)
|
||||
|
||||
@@ -374,17 +374,17 @@ func TestValidateValidatorBindingRules(t *testing.T) {
|
||||
want: "chunk validators[0] module must not be empty",
|
||||
},
|
||||
{
|
||||
name: "validator retries",
|
||||
name: "negative validator retries",
|
||||
setup: func(profile *pipeline.PipelineProfile) {
|
||||
profile.Chunk.Validators = pipeline.ValidatorOverride{
|
||||
Set: true,
|
||||
Validators: []pipeline.ModuleBinding{{
|
||||
Module: "validator",
|
||||
Retries: 1,
|
||||
Retries: -1,
|
||||
}},
|
||||
}
|
||||
},
|
||||
want: "chunk validators[0] retries are not supported",
|
||||
want: "chunk validators[0] retries must be greater than or equal to zero",
|
||||
},
|
||||
{
|
||||
name: "validator references",
|
||||
|
||||
97
internal/core/config/validation_policy_contract_test.go
Normal file
97
internal/core/config/validation_policy_contract_test.go
Normal file
@@ -0,0 +1,97 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
)
|
||||
|
||||
func TestValidationPolicyFileConfigurationIsStrictAndPresenceAware(t *testing.T) {
|
||||
const valid = `version: 4
|
||||
pipelines:
|
||||
main:
|
||||
validation_policy:
|
||||
producer_structural_failure: reject_output
|
||||
semantic_rejection: fail_run
|
||||
input: seriatim
|
||||
chunk:
|
||||
module: generic
|
||||
validation_policy:
|
||||
validator_failure: fail_run
|
||||
artifacts:
|
||||
lane:
|
||||
extract:
|
||||
module: extract
|
||||
validation_policy:
|
||||
semantic_rejection: reject_output
|
||||
`
|
||||
cfg := applyFileConfig(t, valid)
|
||||
profile := cfg.Pipelines["main"]
|
||||
if profile.ValidationPolicy == nil || profile.ValidationPolicy.ProducerStructuralFailure == nil || *profile.ValidationPolicy.ProducerStructuralFailure != pipeline.ProducerStructuralFailureRejectOutput || profile.ValidationPolicy.SemanticRejection == nil || *profile.ValidationPolicy.SemanticRejection != pipeline.SemanticRejectionFailRun || profile.ValidationPolicy.ValidatorFailure != nil {
|
||||
t.Fatalf("pipeline validation policy = %#v", profile.ValidationPolicy)
|
||||
}
|
||||
if profile.Chunk.ValidationPolicy == nil || profile.Chunk.ValidationPolicy.ValidatorFailure == nil || *profile.Chunk.ValidationPolicy.ValidatorFailure != pipeline.ValidatorFailureFailRun {
|
||||
t.Fatalf("chunk validation policy = %#v", profile.Chunk.ValidationPolicy)
|
||||
}
|
||||
lane := profile.Artifacts["lane"]
|
||||
if lane.Extract.ValidationPolicy == nil || lane.Extract.ValidationPolicy.SemanticRejection == nil || *lane.Extract.ValidationPolicy.SemanticRejection != pipeline.SemanticRejectionRejectOutput {
|
||||
t.Fatalf("extract validation policy = %#v", lane.Extract.ValidationPolicy)
|
||||
}
|
||||
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
yaml string
|
||||
}{
|
||||
{"null object", strings.Replace(valid, "validation_policy:\n producer_structural_failure: reject_output\n semantic_rejection: fail_run", "validation_policy: null", 1)},
|
||||
{"null field", strings.Replace(valid, "semantic_rejection: fail_run", "semantic_rejection: null", 1)},
|
||||
{"unknown field", strings.Replace(valid, "semantic_rejection: fail_run", "unknown: fail_run", 1)},
|
||||
{"duplicate field", strings.Replace(valid, "semantic_rejection: fail_run", "semantic_rejection: fail_run\n semantic_rejection: reject_output", 1)},
|
||||
{"invalid enum", strings.Replace(valid, "semantic_rejection: fail_run", "semantic_rejection: continue", 1)},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
if _, err := ParseFileConfigYAML([]byte(test.yaml)); err == nil {
|
||||
t.Fatal("ParseFileConfigYAML() error = nil, want strict validation-policy rejection")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidationPolicyPlacementRules(t *testing.T) {
|
||||
policy := &pipeline.ValidationPolicyOverride{}
|
||||
semantic := pipeline.SemanticRejectionRejectOutput
|
||||
policy.SemanticRejection = &semantic
|
||||
base := pipeline.PipelineProfile{
|
||||
ID: "main",
|
||||
Input: pipeline.Binding("input"),
|
||||
Artifacts: map[string]pipeline.ArtifactLaneProfile{
|
||||
"lane": {Extract: pipeline.Binding("extract")},
|
||||
},
|
||||
}
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
mutate func(*pipeline.PipelineProfile)
|
||||
}{
|
||||
{"input", func(profile *pipeline.PipelineProfile) { profile.Input.ValidationPolicy = policy }},
|
||||
{"output", func(profile *pipeline.PipelineProfile) {
|
||||
profile.Output = pipeline.Binding("output")
|
||||
profile.Output.ValidationPolicy = policy
|
||||
}},
|
||||
{"validator", func(profile *pipeline.PipelineProfile) {
|
||||
lane := profile.Artifacts["lane"]
|
||||
lane.Extract.Validators = pipeline.ValidatorOverride{Set: true, Validators: []pipeline.ModuleBinding{{Module: "validator", ValidationPolicy: policy}}}
|
||||
profile.Artifacts["lane"] = lane
|
||||
}},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
profile := base
|
||||
profile.Artifacts = map[string]pipeline.ArtifactLaneProfile{"lane": base.Artifacts["lane"]}
|
||||
test.mutate(&profile)
|
||||
cfg := Default()
|
||||
cfg.Pipelines = map[string]pipeline.PipelineProfile{"main": profile}
|
||||
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "validation_policy") {
|
||||
t.Fatalf("Config.Validate() error = %v, want placement rejection", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/fileio"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
@@ -344,7 +345,15 @@ func cloneRejectedOutputs(rejected []contracts.RejectedOutput) []contracts.Rejec
|
||||
if len(rejected) == 0 {
|
||||
return nil
|
||||
}
|
||||
return append([]contracts.RejectedOutput(nil), rejected...)
|
||||
cloned := make([]contracts.RejectedOutput, len(rejected))
|
||||
for index, item := range rejected {
|
||||
cloned[index] = item
|
||||
if item.Validation != nil {
|
||||
summary := artifacts.CloneValidationSummary(*item.Validation)
|
||||
cloned[index].Validation = &summary
|
||||
}
|
||||
}
|
||||
return cloned
|
||||
}
|
||||
|
||||
func cloneMetadata(metadata map[string]any) map[string]any {
|
||||
|
||||
84
internal/framework/contracts/completion_request_debug.go
Normal file
84
internal/framework/contracts/completion_request_debug.go
Normal file
@@ -0,0 +1,84 @@
|
||||
package contracts
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// DebugStructuredCompletionRequest is the content-safe representation of a
|
||||
// structured completion request for ordinary diagnostics and summaries.
|
||||
// Detailed prompt material remains available only through the explicitly
|
||||
// requested LLM debug trace.
|
||||
type DebugStructuredCompletionRequest struct {
|
||||
StageName string `json:"stage_name,omitempty"`
|
||||
PromptID string `json:"prompt_id,omitempty"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
ProfileID string `json:"profile_id,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
InputCount int `json:"input_count"`
|
||||
VariableCount int `json:"variable_count"`
|
||||
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
|
||||
Correction *DebugSemanticCorrection `json:"correction,omitempty"`
|
||||
}
|
||||
|
||||
// DebugSemanticCorrection records only safe correction metadata. It never
|
||||
// exposes the assistant response or user guidance text.
|
||||
type DebugSemanticCorrection struct {
|
||||
AssistantResponseBytes int `json:"assistant_response_bytes"`
|
||||
AssistantResponseDigest string `json:"assistant_response_digest"`
|
||||
UserGuidanceBytes int `json:"user_guidance_bytes"`
|
||||
UserGuidanceDigest string `json:"user_guidance_digest"`
|
||||
}
|
||||
|
||||
// DebugSummary returns a content-safe representation suitable for ordinary
|
||||
// diagnostics. It does not validate or retain correction content.
|
||||
func (request StructuredCompletionRequest) DebugSummary() DebugStructuredCompletionRequest {
|
||||
summary := DebugStructuredCompletionRequest{
|
||||
StageName: request.StageName,
|
||||
PromptID: request.PromptID,
|
||||
PromptVersion: request.PromptVersion,
|
||||
ProfileID: request.ProfileID,
|
||||
SessionID: request.SessionID,
|
||||
InputCount: len(request.Inputs),
|
||||
VariableCount: len(request.Vars),
|
||||
}
|
||||
if request.StructuredOutputRepairAttempts != nil {
|
||||
attempts := *request.StructuredOutputRepairAttempts
|
||||
summary.StructuredOutputRepairAttempts = &attempts
|
||||
}
|
||||
if request.Correction != nil {
|
||||
summary.Correction = request.Correction.DebugSummary()
|
||||
}
|
||||
return summary
|
||||
}
|
||||
|
||||
// DebugSummary returns content-safe correction metadata suitable for ordinary
|
||||
// diagnostics.
|
||||
func (correction *SemanticCorrection) DebugSummary() *DebugSemanticCorrection {
|
||||
if correction == nil {
|
||||
return nil
|
||||
}
|
||||
return &DebugSemanticCorrection{
|
||||
AssistantResponseBytes: len(correction.AssistantResponse),
|
||||
AssistantResponseDigest: debugContentDigest(correction.AssistantResponse),
|
||||
UserGuidanceBytes: len(correction.UserGuidance),
|
||||
UserGuidanceDigest: debugContentDigest([]byte(correction.UserGuidance)),
|
||||
}
|
||||
}
|
||||
|
||||
// String prevents ordinary request formatting from exposing correction
|
||||
// content. Use the explicitly requested debug trace for complete messages.
|
||||
func (request StructuredCompletionRequest) String() string {
|
||||
return fmt.Sprintf("%+v", request.DebugSummary())
|
||||
}
|
||||
|
||||
// GoString gives %#v formatting the same content-safe behavior as String.
|
||||
func (request StructuredCompletionRequest) GoString() string {
|
||||
return request.String()
|
||||
}
|
||||
|
||||
func debugContentDigest(content []byte) string {
|
||||
sum := sha256.Sum256(content)
|
||||
return "sha256:" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
@@ -9,14 +9,15 @@ import (
|
||||
)
|
||||
|
||||
type StructuredCompletionRequest struct {
|
||||
StageName string `json:"stage_name"`
|
||||
PromptID string `json:"prompt_id,omitempty"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
ProfileID string `json:"profile_id,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
Inputs LLMInputSet `json:"inputs,omitempty"`
|
||||
Vars map[string]any `json:"vars,omitempty"`
|
||||
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
|
||||
StageName string `json:"stage_name"`
|
||||
PromptID string `json:"prompt_id,omitempty"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
ProfileID string `json:"profile_id,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
Inputs LLMInputSet `json:"inputs,omitempty"`
|
||||
Vars map[string]any `json:"vars,omitempty"`
|
||||
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
|
||||
Correction *SemanticCorrection `json:"-"`
|
||||
}
|
||||
|
||||
type StructuredCompletionResponse struct {
|
||||
@@ -156,12 +157,14 @@ type ChunkRequest struct {
|
||||
References ReferenceSet `json:"references,omitempty"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
|
||||
Correction *SemanticCorrection `json:"-"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type ChunkPlanResult struct {
|
||||
Plan source.ChunkPlan `json:"plan"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
Plan source.ChunkPlan `json:"plan"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
ModelCandidate *ModelCandidate `json:"-"`
|
||||
}
|
||||
|
||||
type Chunker interface {
|
||||
@@ -283,6 +286,7 @@ type ValidationResult struct {
|
||||
Approved bool `json:"approved"`
|
||||
ReasonCode string `json:"reason_code,omitempty"`
|
||||
Message string `json:"message,omitempty"`
|
||||
CorrectionGuidance string `json:"-"`
|
||||
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
@@ -322,17 +326,18 @@ type OutputEncoder interface {
|
||||
}
|
||||
|
||||
type RejectedOutput struct {
|
||||
Stage string `json:"stage"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
ChunkID string `json:"chunk_id,omitempty"`
|
||||
ChunkIndex int `json:"chunk_index,omitempty"`
|
||||
ValidatorName string `json:"validator_name,omitempty"`
|
||||
ReasonCode string `json:"reason_code,omitempty"`
|
||||
Message string `json:"message"`
|
||||
AttemptCount int `json:"attempt_count,omitempty"`
|
||||
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
|
||||
Stage string `json:"stage"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
ChunkID string `json:"chunk_id,omitempty"`
|
||||
ChunkIndex int `json:"chunk_index,omitempty"`
|
||||
ValidatorName string `json:"validator_name,omitempty"`
|
||||
ReasonCode string `json:"reason_code,omitempty"`
|
||||
Message string `json:"message"`
|
||||
AttemptCount int `json:"attempt_count,omitempty"`
|
||||
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
|
||||
Validation *artifacts.ValidationSummary `json:"validation,omitempty"`
|
||||
}
|
||||
|
||||
type ManifestMetadataProvider interface {
|
||||
|
||||
175
internal/framework/contracts/correction.go
Normal file
175
internal/framework/contracts/correction.go
Normal file
@@ -0,0 +1,175 @@
|
||||
package contracts
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
)
|
||||
|
||||
// CorrectionProtocol identifies how a producer can represent the model output
|
||||
// that directly controlled a candidate.
|
||||
type CorrectionProtocol string
|
||||
|
||||
const (
|
||||
CorrectionProtocolSingleResponseV1 CorrectionProtocol = "single_response_v1"
|
||||
)
|
||||
|
||||
const (
|
||||
MaxValidationReasonCodeBytes = 128
|
||||
MaxValidationCorrectionGuidanceBytes = 4 * 1024
|
||||
MaxAssistantResponseBytes = 1 << 20
|
||||
MaxCorrectionGuidanceBytes = 64 * 1024
|
||||
MaxCorrectionContentBytes = MaxAssistantResponseBytes + MaxCorrectionGuidanceBytes
|
||||
)
|
||||
|
||||
func (protocol CorrectionProtocol) Validate() error {
|
||||
if protocol == CorrectionProtocolSingleResponseV1 {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("unsupported correction protocol %q", protocol)
|
||||
}
|
||||
|
||||
// SemanticCorrection carries the latest model response and application-owned
|
||||
// guidance for one fresh corrected request. Its content is sensitive and is
|
||||
// deliberately excluded from ordinary JSON serialization.
|
||||
type SemanticCorrection struct {
|
||||
AssistantResponse []byte `json:"-"`
|
||||
UserGuidance string `json:"-"`
|
||||
}
|
||||
|
||||
func NewSemanticCorrection(assistantResponse []byte, userGuidance string) (*SemanticCorrection, error) {
|
||||
correction := &SemanticCorrection{
|
||||
AssistantResponse: append([]byte(nil), assistantResponse...),
|
||||
UserGuidance: userGuidance,
|
||||
}
|
||||
if err := correction.Validate(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return correction, nil
|
||||
}
|
||||
|
||||
func CloneSemanticCorrection(correction *SemanticCorrection) (*SemanticCorrection, error) {
|
||||
if correction == nil {
|
||||
return nil, nil
|
||||
}
|
||||
return NewSemanticCorrection(correction.AssistantResponse, correction.UserGuidance)
|
||||
}
|
||||
|
||||
// CloneStructuredCompletionRequest returns a request whose mutable values are
|
||||
// owned by the caller. It is suitable for clients that retain requests after
|
||||
// CompleteStructured returns.
|
||||
func CloneStructuredCompletionRequest(request StructuredCompletionRequest) (StructuredCompletionRequest, error) {
|
||||
correction, err := CloneSemanticCorrection(request.Correction)
|
||||
if err != nil {
|
||||
return StructuredCompletionRequest{}, fmt.Errorf("clone correction: %w", err)
|
||||
}
|
||||
vars, err := source.CloneMetadata(request.Vars)
|
||||
if err != nil {
|
||||
return StructuredCompletionRequest{}, fmt.Errorf("clone variables: %w", err)
|
||||
}
|
||||
request.Inputs = request.Inputs.Clone()
|
||||
request.Vars = vars
|
||||
request.Correction = correction
|
||||
if request.StructuredOutputRepairAttempts != nil {
|
||||
attempts := *request.StructuredOutputRepairAttempts
|
||||
request.StructuredOutputRepairAttempts = &attempts
|
||||
}
|
||||
return request, nil
|
||||
}
|
||||
|
||||
func (correction SemanticCorrection) Validate() error {
|
||||
if err := validateAssistantResponse(correction.AssistantResponse); err != nil {
|
||||
return fmt.Errorf("semantic correction assistant response: %w", err)
|
||||
}
|
||||
if err := validateBoundedText(correction.UserGuidance, MaxCorrectionGuidanceBytes, "semantic correction user guidance", false); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(correction.AssistantResponse)+len(correction.UserGuidance) > MaxCorrectionContentBytes {
|
||||
return errors.New("semantic correction content exceeds maximum length")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ModelCandidate preserves the exact single model response that directly
|
||||
// controlled a producer result. It is attempt-local and never durable data.
|
||||
type ModelCandidate struct {
|
||||
Response []byte `json:"-"`
|
||||
Protocol CorrectionProtocol `json:"-"`
|
||||
}
|
||||
|
||||
func NewModelCandidate(response []byte, protocol CorrectionProtocol) (*ModelCandidate, error) {
|
||||
candidate := &ModelCandidate{Response: append([]byte(nil), response...), Protocol: protocol}
|
||||
if err := candidate.Validate(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return candidate, nil
|
||||
}
|
||||
|
||||
func CloneModelCandidate(candidate *ModelCandidate) (*ModelCandidate, error) {
|
||||
if candidate == nil {
|
||||
return nil, nil
|
||||
}
|
||||
return NewModelCandidate(candidate.Response, candidate.Protocol)
|
||||
}
|
||||
|
||||
func (candidate ModelCandidate) Validate() error {
|
||||
if err := candidate.Protocol.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateAssistantResponse(candidate.Response); err != nil {
|
||||
return fmt.Errorf("model candidate response: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func ValidateValidationResult(result ValidationResult) error {
|
||||
if !result.Approved && result.ReasonCode == "" {
|
||||
return errors.New("validation rejection reason code must not be empty")
|
||||
}
|
||||
if result.ReasonCode != "" {
|
||||
if err := validateBoundedText(result.ReasonCode, MaxValidationReasonCodeBytes, "validation reason code", false); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if !result.Approved && result.CorrectionGuidance == "" {
|
||||
return errors.New("validation rejection correction guidance must not be empty")
|
||||
}
|
||||
if result.CorrectionGuidance != "" {
|
||||
if err := validateBoundedText(result.CorrectionGuidance, MaxValidationCorrectionGuidanceBytes, "validation correction guidance", false); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateAssistantResponse(response []byte) error {
|
||||
if len(response) > MaxAssistantResponseBytes {
|
||||
return errors.New("exceeds maximum length")
|
||||
}
|
||||
if !utf8.Valid(response) {
|
||||
return errors.New("must be valid UTF-8")
|
||||
}
|
||||
if len(strings.TrimSpace(string(response))) == 0 {
|
||||
return errors.New("must not be blank")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateBoundedText(value string, maximum int, name string, optional bool) error {
|
||||
if value == "" && optional {
|
||||
return nil
|
||||
}
|
||||
if !utf8.ValidString(value) {
|
||||
return fmt.Errorf("%s must be valid UTF-8", name)
|
||||
}
|
||||
if strings.TrimSpace(value) == "" {
|
||||
return fmt.Errorf("%s must not be blank", name)
|
||||
}
|
||||
if len(value) > maximum {
|
||||
return fmt.Errorf("%s exceeds maximum length", name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
184
internal/framework/contracts/correction_test.go
Normal file
184
internal/framework/contracts/correction_test.go
Normal file
@@ -0,0 +1,184 @@
|
||||
package contracts
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSemanticCorrectionOwnsValidatedContent(t *testing.T) {
|
||||
assistant := []byte(`{"items":["original"]}`)
|
||||
correction, err := NewSemanticCorrection(assistant, "Return one corrected replacement.")
|
||||
if err != nil {
|
||||
t.Fatalf("NewSemanticCorrection() error = %v", err)
|
||||
}
|
||||
assistant[0] = '['
|
||||
if got := string(correction.AssistantResponse); got != `{"items":["original"]}` {
|
||||
t.Fatalf("assistant response = %q, want owned original content", got)
|
||||
}
|
||||
|
||||
clone, err := CloneSemanticCorrection(correction)
|
||||
if err != nil {
|
||||
t.Fatalf("CloneSemanticCorrection() error = %v", err)
|
||||
}
|
||||
clone.AssistantResponse[0] = '['
|
||||
if got := string(correction.AssistantResponse); got != `{"items":["original"]}` {
|
||||
t.Fatalf("source correction changed through clone = %q", got)
|
||||
}
|
||||
if nilClone, err := CloneSemanticCorrection(nil); err != nil || nilClone != nil {
|
||||
t.Fatalf("CloneSemanticCorrection(nil) = %#v, %v; want nil, nil", nilClone, err)
|
||||
}
|
||||
|
||||
encoded, err := json.Marshal(correction)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal correction: %v", err)
|
||||
}
|
||||
if string(encoded) != "{}" {
|
||||
t.Fatalf("correction JSON = %s, want no sensitive content", encoded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCorrectionContractsRejectInvalidContent(t *testing.T) {
|
||||
tooLongAssistant := bytes.Repeat([]byte("a"), MaxAssistantResponseBytes+1)
|
||||
tooLongGuidance := strings.Repeat("a", MaxCorrectionGuidanceBytes+1)
|
||||
tooLongReason := strings.Repeat("a", MaxValidationReasonCodeBytes+1)
|
||||
tooLongValidationGuidance := strings.Repeat("a", MaxValidationCorrectionGuidanceBytes+1)
|
||||
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
call func() error
|
||||
}{
|
||||
{"blank assistant", func() error { _, err := NewSemanticCorrection([]byte(" \n"), "guidance"); return err }},
|
||||
{"invalid assistant utf8", func() error { _, err := NewSemanticCorrection([]byte{0xff}, "guidance"); return err }},
|
||||
{"oversized assistant", func() error { _, err := NewSemanticCorrection(tooLongAssistant, "guidance"); return err }},
|
||||
{"blank guidance", func() error { _, err := NewSemanticCorrection([]byte("response"), " \t"); return err }},
|
||||
{"invalid guidance utf8", func() error { _, err := NewSemanticCorrection([]byte("response"), string([]byte{0xff})); return err }},
|
||||
{"oversized guidance", func() error { _, err := NewSemanticCorrection([]byte("response"), tooLongGuidance); return err }},
|
||||
{"unsupported protocol", func() error { _, err := NewModelCandidate([]byte("response"), "multiple_responses"); return err }},
|
||||
{"missing candidate protocol", func() error { _, err := NewModelCandidate([]byte("response"), ""); return err }},
|
||||
{"blank candidate response", func() error { _, err := NewModelCandidate([]byte(" "), CorrectionProtocolSingleResponseV1); return err }},
|
||||
{"missing rejection reason code", func() error {
|
||||
return ValidateValidationResult(ValidationResult{CorrectionGuidance: "Correct the response."})
|
||||
}},
|
||||
{"missing rejection guidance", func() error { return ValidateValidationResult(ValidationResult{ReasonCode: "invalid"}) }},
|
||||
{"oversized reason code", func() error {
|
||||
return ValidateValidationResult(ValidationResult{ReasonCode: tooLongReason, CorrectionGuidance: "Correct the response."})
|
||||
}},
|
||||
{"blank reason code", func() error {
|
||||
return ValidateValidationResult(ValidationResult{ReasonCode: " \t", CorrectionGuidance: "Correct the response."})
|
||||
}},
|
||||
{"invalid correction guidance utf8", func() error {
|
||||
return ValidateValidationResult(ValidationResult{ReasonCode: "invalid", CorrectionGuidance: string([]byte{0xff})})
|
||||
}},
|
||||
{"oversized correction guidance", func() error {
|
||||
return ValidateValidationResult(ValidationResult{ReasonCode: "invalid", CorrectionGuidance: tooLongValidationGuidance})
|
||||
}},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
if err := test.call(); err == nil {
|
||||
t.Fatal("validation error = nil, want error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestModelCandidateOwnsValidatedResponse(t *testing.T) {
|
||||
response := []byte(`{"items":["original"]}`)
|
||||
candidate, err := NewModelCandidate(response, CorrectionProtocolSingleResponseV1)
|
||||
if err != nil {
|
||||
t.Fatalf("NewModelCandidate() error = %v", err)
|
||||
}
|
||||
response[0] = '['
|
||||
if got := string(candidate.Response); got != `{"items":["original"]}` {
|
||||
t.Fatalf("candidate response = %q, want owned original content", got)
|
||||
}
|
||||
clone, err := CloneModelCandidate(candidate)
|
||||
if err != nil {
|
||||
t.Fatalf("CloneModelCandidate() error = %v", err)
|
||||
}
|
||||
clone.Response[0] = '['
|
||||
if got := string(candidate.Response); got != `{"items":["original"]}` {
|
||||
t.Fatalf("source candidate changed through clone = %q", got)
|
||||
}
|
||||
if nilClone, err := CloneModelCandidate(nil); err != nil || nilClone != nil {
|
||||
t.Fatalf("CloneModelCandidate(nil) = %#v, %v; want nil, nil", nilClone, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidationResultAllowsAbsentOptionalCorrectionFields(t *testing.T) {
|
||||
if err := ValidateValidationResult(ValidationResult{Approved: true}); err != nil {
|
||||
t.Fatalf("ValidateValidationResult() error = %v, want nil", err)
|
||||
}
|
||||
if err := ValidateValidationResult(ValidationResult{ReasonCode: "invalid-evidence", CorrectionGuidance: "Provide source-backed evidence."}); err != nil {
|
||||
t.Fatalf("ValidateValidationResult() error = %v, want nil", err)
|
||||
}
|
||||
if err := CorrectionProtocol("").Validate(); err == nil {
|
||||
t.Fatal("empty correction protocol validation error = nil, want error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCloneStructuredCompletionRequestOwnsCorrection(t *testing.T) {
|
||||
correction, err := NewSemanticCorrection([]byte(`{"value":"original"}`), "Correct the value.")
|
||||
if err != nil {
|
||||
t.Fatalf("NewSemanticCorrection() error = %v", err)
|
||||
}
|
||||
attempts := 2
|
||||
request := StructuredCompletionRequest{
|
||||
Inputs: LLMInputSet{"source": NewLLMInputMaterial("source", "application/json", []byte(`{"source":true}`), "", "")},
|
||||
Vars: map[string]any{"labels": []string{"original"}},
|
||||
StructuredOutputRepairAttempts: &attempts,
|
||||
Correction: correction,
|
||||
}
|
||||
clone, err := CloneStructuredCompletionRequest(request)
|
||||
if err != nil {
|
||||
t.Fatalf("CloneStructuredCompletionRequest() error = %v", err)
|
||||
}
|
||||
correction.AssistantResponse[0] = '['
|
||||
request.Inputs["source"] = NewLLMInputMaterial("source", "application/json", []byte(`{"source":false}`), "", "")
|
||||
*request.StructuredOutputRepairAttempts = 7
|
||||
if got := string(clone.Correction.AssistantResponse); got != `{"value":"original"}` {
|
||||
t.Fatalf("cloned correction response = %q, want owned original content", got)
|
||||
}
|
||||
if got := string(clone.Inputs["source"].Content); got != `{"source":true}` {
|
||||
t.Fatalf("cloned input = %q, want owned original content", got)
|
||||
}
|
||||
if clone.StructuredOutputRepairAttempts == nil || *clone.StructuredOutputRepairAttempts != 2 {
|
||||
t.Fatalf("cloned repair attempts = %v, want 2", clone.StructuredOutputRepairAttempts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStructuredCompletionRequestDebugSummaryOmitsCorrectionContent(t *testing.T) {
|
||||
const assistantResponse = `{"secret":"assistant response"}`
|
||||
const userGuidance = "secret user guidance"
|
||||
correction, err := NewSemanticCorrection([]byte(assistantResponse), userGuidance)
|
||||
if err != nil {
|
||||
t.Fatalf("NewSemanticCorrection() error = %v", err)
|
||||
}
|
||||
request := StructuredCompletionRequest{
|
||||
Inputs: LLMInputSet{"source": NewLLMInputMaterial("source", "application/json", []byte(`{"source":true}`), "", "")},
|
||||
Vars: map[string]any{"custom": "value"},
|
||||
Correction: correction,
|
||||
}
|
||||
|
||||
summary := request.DebugSummary()
|
||||
if summary.InputCount != 1 || summary.VariableCount != 1 || summary.Correction == nil {
|
||||
t.Fatalf("debug summary = %#v, want input, variable, and correction metadata", summary)
|
||||
}
|
||||
if summary.Correction.AssistantResponseBytes != len(assistantResponse) || summary.Correction.UserGuidanceBytes != len(userGuidance) ||
|
||||
summary.Correction.AssistantResponseDigest == "" || summary.Correction.UserGuidanceDigest == "" {
|
||||
t.Fatalf("correction summary = %#v, want byte counts and digests", summary.Correction)
|
||||
}
|
||||
encoded, err := json.Marshal(summary)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal debug summary: %v", err)
|
||||
}
|
||||
for _, rendered := range []string{string(encoded), fmt.Sprintf("%+v", request), fmt.Sprintf("%#v", request)} {
|
||||
for _, secret := range []string{assistantResponse, userGuidance} {
|
||||
if strings.Contains(rendered, secret) {
|
||||
t.Fatalf("content-safe request rendering leaked %q: %s", secret, rendered)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -42,12 +42,14 @@ type TypedExtractionRequest struct {
|
||||
References ReferenceSet
|
||||
LLMProfile string
|
||||
StructuredOutputRepairAttempts *int
|
||||
Correction *SemanticCorrection
|
||||
Metadata map[string]any
|
||||
}
|
||||
|
||||
type TypedExtractionResult[T any] struct {
|
||||
Value T
|
||||
Warnings []Warning
|
||||
Value T
|
||||
Warnings []Warning
|
||||
ModelCandidate *ModelCandidate
|
||||
}
|
||||
|
||||
type Extractor[T any] interface {
|
||||
@@ -65,12 +67,14 @@ type TypedMergeRequest[T any] struct {
|
||||
References ReferenceSet
|
||||
LLMProfile string
|
||||
StructuredOutputRepairAttempts *int
|
||||
Correction *SemanticCorrection
|
||||
Metadata map[string]any
|
||||
}
|
||||
|
||||
type TypedMergeResult[T any] struct {
|
||||
Value T
|
||||
Warnings []Warning
|
||||
Value T
|
||||
Warnings []Warning
|
||||
ModelCandidate *ModelCandidate
|
||||
}
|
||||
|
||||
type Merger[T any] interface {
|
||||
@@ -87,13 +91,15 @@ type TypedNormalizeRequest[T any] struct {
|
||||
References ReferenceSet
|
||||
LLMProfile string
|
||||
StructuredOutputRepairAttempts *int
|
||||
Correction *SemanticCorrection
|
||||
Metadata map[string]any
|
||||
}
|
||||
|
||||
type TypedNormalizeResult[T any] struct {
|
||||
Value T
|
||||
Warnings []Warning
|
||||
Retry *NormalizeRetry
|
||||
Value T
|
||||
Warnings []Warning
|
||||
Retry *NormalizeRetry
|
||||
ModelCandidate *ModelCandidate
|
||||
}
|
||||
|
||||
// Normalize retry diagnostic limits bound module-provided values before the
|
||||
|
||||
@@ -119,6 +119,10 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
|
||||
if req.StructuredOutputRepairAttempts != nil && (*req.StructuredOutputRepairAttempts < 0 || *req.StructuredOutputRepairAttempts > 3) {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured output repair attempts must be between zero and three")
|
||||
}
|
||||
appendedMessages, err := promptKitCorrectionMessages(req.Correction)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion correction: %w", err)
|
||||
}
|
||||
promptID := strings.TrimSpace(req.PromptID)
|
||||
if promptID == "" {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty")
|
||||
@@ -133,13 +137,14 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
|
||||
}
|
||||
|
||||
runReq := promptkit.RunRequest{
|
||||
PromptID: promptID,
|
||||
PromptVersion: strings.TrimSpace(req.PromptVersion),
|
||||
ProfileID: strings.TrimSpace(req.ProfileID),
|
||||
SessionID: sessionID,
|
||||
Inputs: promptKitInputs(req.Inputs),
|
||||
Vars: promptKitVars(req, sessionID),
|
||||
Execution: execution,
|
||||
PromptID: promptID,
|
||||
PromptVersion: strings.TrimSpace(req.PromptVersion),
|
||||
ProfileID: strings.TrimSpace(req.ProfileID),
|
||||
SessionID: sessionID,
|
||||
Inputs: promptKitInputs(req.Inputs),
|
||||
Vars: promptKitVars(req, sessionID),
|
||||
Execution: execution,
|
||||
AppendedMessages: appendedMessages,
|
||||
}
|
||||
if req.StructuredOutputRepairAttempts != nil {
|
||||
inspection, err := c.engine.InspectPrompt(ctx, promptID, strings.TrimSpace(req.PromptVersion))
|
||||
@@ -213,6 +218,20 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts.
|
||||
return response, nil
|
||||
}
|
||||
|
||||
func promptKitCorrectionMessages(correction *contracts.SemanticCorrection) ([]promptkit.RenderedMessage, error) {
|
||||
owned, err := contracts.CloneSemanticCorrection(correction)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if owned == nil {
|
||||
return nil, nil
|
||||
}
|
||||
return []promptkit.RenderedMessage{
|
||||
{Role: promptkit.RoleAssistant, Content: string(owned.AssistantResponse)},
|
||||
{Role: promptkit.RoleUser, Content: owned.UserGuidance},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func promptKitDebugGenerationError(prepared *promptkit.PreparedRun, generationErr *promptkit.GenerationError) *contracts.LLMDebugResponse {
|
||||
if generationErr == nil {
|
||||
return nil
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package llm
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
@@ -116,6 +117,74 @@ func TestPromptKitClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientAppendsSemanticCorrectionAfterRenderedPrompt(t *testing.T) {
|
||||
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||
client := newTestPromptKitClient(t, fake)
|
||||
request := contracts.StructuredCompletionRequest{
|
||||
PromptID: "adapter.direct-session",
|
||||
ProfileID: "explicit-profile",
|
||||
SessionID: "correction-session",
|
||||
Inputs: contracts.LLMInputSet{
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
Vars: map[string]any{"custom": "value"},
|
||||
}
|
||||
|
||||
var ordinary map[string]any
|
||||
if _, err := client.CompleteStructured(context.Background(), request, &ordinary); err != nil {
|
||||
t.Fatalf("ordinary CompleteStructured() error = %v", err)
|
||||
}
|
||||
ordinaryMessages := append([]promptkit.RenderedMessage(nil), fake.lastRequest().Prompt.Messages...)
|
||||
if len(ordinaryMessages) != 1 {
|
||||
t.Fatalf("ordinary rendered messages = %#v, want only the declared prompt message", ordinaryMessages)
|
||||
}
|
||||
|
||||
correction, err := contracts.NewSemanticCorrection([]byte(`{"previous":"response"}`), "Return the corrected JSON object.")
|
||||
if err != nil {
|
||||
t.Fatalf("NewSemanticCorrection() error = %v", err)
|
||||
}
|
||||
request.Correction = correction
|
||||
var corrected map[string]any
|
||||
if _, err := client.CompleteStructured(context.Background(), request, &corrected); err != nil {
|
||||
t.Fatalf("corrected CompleteStructured() error = %v", err)
|
||||
}
|
||||
correctedMessages := fake.lastRequest().Prompt.Messages
|
||||
if len(correctedMessages) != len(ordinaryMessages)+2 {
|
||||
t.Fatalf("corrected message count = %d, want %d", len(correctedMessages), len(ordinaryMessages)+2)
|
||||
}
|
||||
if !reflect.DeepEqual(correctedMessages[:len(ordinaryMessages)], ordinaryMessages) {
|
||||
t.Fatalf("ordinary rendered prefix changed: got %#v, want %#v", correctedMessages[:len(ordinaryMessages)], ordinaryMessages)
|
||||
}
|
||||
if got, want := correctedMessages[len(ordinaryMessages)], (promptkit.RenderedMessage{Role: promptkit.RoleAssistant, Content: `{"previous":"response"}`}); got != want {
|
||||
t.Fatalf("assistant correction message = %#v, want %#v", got, want)
|
||||
}
|
||||
if got, want := correctedMessages[len(ordinaryMessages)+1], (promptkit.RenderedMessage{Role: promptkit.RoleUser, Content: "Return the corrected JSON object."}); got != want {
|
||||
t.Fatalf("user correction message = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientRejectsInvalidCorrectionsBeforePromptPreparation(t *testing.T) {
|
||||
const sensitiveResponse = "assistant-response-must-not-appear-in-errors"
|
||||
for _, correction := range []*contracts.SemanticCorrection{
|
||||
{AssistantResponse: []byte(sensitiveResponse), UserGuidance: " \t"},
|
||||
{AssistantResponse: bytes.Repeat([]byte(sensitiveResponse), contracts.MaxAssistantResponseBytes/len(sensitiveResponse)+1), UserGuidance: "Use a smaller response."},
|
||||
} {
|
||||
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||
client := newTestPromptKitClient(t, fake)
|
||||
var out map[string]any
|
||||
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{Correction: correction}, &out)
|
||||
if err == nil || !strings.Contains(err.Error(), "correction") {
|
||||
t.Fatalf("CompleteStructured() error = %v, want correction validation failure", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), sensitiveResponse) {
|
||||
t.Fatalf("correction validation error leaked response content: %v", err)
|
||||
}
|
||||
if got := atomic.LoadInt32(&fake.calls); got != 0 {
|
||||
t.Fatalf("provider calls = %d, want no provider call after invalid correction", got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptKitClientUsesOnePreparedSnapshotForDebugAndGeneration(t *testing.T) {
|
||||
const initialPrompt = `id: snapshot.test
|
||||
version: "v1"
|
||||
@@ -960,6 +1029,10 @@ func TestPromptKitClientRepairsStructuredOutputAndReportsCumulativeUsage(t *test
|
||||
{Content: `{"ok":true}`, Usage: promptkit.TokenUsage{PromptTokens: 7, CompletionTokens: 11, TotalTokens: 18}},
|
||||
}}
|
||||
client := newTestPromptKitClient(t, fake)
|
||||
correction, err := contracts.NewSemanticCorrection([]byte(`{"bad":true}`), "Return the required ok field.")
|
||||
if err != nil {
|
||||
t.Fatalf("NewSemanticCorrection() error = %v", err)
|
||||
}
|
||||
|
||||
var out struct {
|
||||
OK bool `json:"ok"`
|
||||
@@ -968,6 +1041,7 @@ func TestPromptKitClientRepairsStructuredOutputAndReportsCumulativeUsage(t *test
|
||||
PromptID: "adapter.test",
|
||||
StructuredOutputRepairAttempts: &attempts,
|
||||
SessionID: "repair-test",
|
||||
Correction: correction,
|
||||
Inputs: contracts.LLMInputSet{
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
@@ -978,6 +1052,17 @@ func TestPromptKitClientRepairsStructuredOutputAndReportsCumulativeUsage(t *test
|
||||
if got := atomic.LoadInt32(&fake.calls); got != 2 {
|
||||
t.Fatalf("provider calls = %d, want initial generation and one repair", got)
|
||||
}
|
||||
requests := fake.requestsSnapshot()
|
||||
if len(requests) != 2 || len(requests[0].Prompt.Messages) < 3 {
|
||||
t.Fatalf("repair requests = %#v, want correction messages on the initial prepared request", requests)
|
||||
}
|
||||
messages := requests[0].Prompt.Messages
|
||||
if got, want := messages[len(messages)-2], (promptkit.RenderedMessage{Role: promptkit.RoleAssistant, Content: `{"bad":true}`}); got != want {
|
||||
t.Fatalf("repair assistant correction = %#v, want %#v", got, want)
|
||||
}
|
||||
if got, want := messages[len(messages)-1], (promptkit.RenderedMessage{Role: promptkit.RoleUser, Content: "Return the required ok field."}); got != want {
|
||||
t.Fatalf("repair user correction = %#v, want %#v", got, want)
|
||||
}
|
||||
if response.RepairAttempts != 1 || response.PromptTokens != 10 || response.CompletionTokens != 16 || response.TotalTokens != 26 {
|
||||
t.Fatalf("response repair and usage = %#v, want one repair and PromptKit cumulative usage", response)
|
||||
}
|
||||
|
||||
@@ -17,7 +17,7 @@ const (
|
||||
promptKitLocalBackendMarker = "notarius:promptkit-local-backend:v1"
|
||||
// The built-in profile catalog is compiled into this pinned PromptKit
|
||||
// release. Update this identity when the dependency is upgraded.
|
||||
promptKitBuiltinProfileCatalogID = "promptkit:v0.8.0:builtin-profiles"
|
||||
promptKitBuiltinProfileCatalogID = "promptkit:v0.9.0:builtin-profiles"
|
||||
)
|
||||
|
||||
func promptKitProfileFingerprint(profileDir, profileFile, fallbackProfileDigest string) (CheckpointFingerprint, error) {
|
||||
|
||||
@@ -89,6 +89,7 @@ const (
|
||||
CheckpointReasonReused CheckpointReasonCode = "checkpoint_reused"
|
||||
CheckpointReasonAcceptedArtifactReused CheckpointReasonCode = "accepted_artifact_reused"
|
||||
CheckpointReasonRecomputeStep CheckpointReasonCode = "recompute_step"
|
||||
CheckpointReasonValidationIncompleteLineage CheckpointReasonCode = "validation_incomplete_lineage"
|
||||
)
|
||||
|
||||
type CheckpointDecision struct {
|
||||
@@ -160,6 +161,8 @@ func checkpointDecisionDetail(reasonCode CheckpointReasonCode) string {
|
||||
return "accepted normalized artifact is reusable"
|
||||
case CheckpointReasonRecomputeStep:
|
||||
return "selected step requires execution"
|
||||
case CheckpointReasonValidationIncompleteLineage:
|
||||
return "checkpoint reuse is disabled by validation-incomplete input lineage"
|
||||
default:
|
||||
return "checkpoint decision"
|
||||
}
|
||||
|
||||
@@ -42,6 +42,7 @@ type debugTimedEnvelope struct {
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
Attempt int `json:"attempt,omitempty"`
|
||||
AttemptKind string `json:"attempt_kind,omitempty"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
CompletedAt time.Time `json:"completed_at"`
|
||||
DurationMS int64 `json:"duration_ms"`
|
||||
@@ -110,24 +111,7 @@ type debugSerializedOutput struct {
|
||||
Content debugBinaryEnvelope `json:"content"`
|
||||
}
|
||||
|
||||
type debugLLMInputMaterial struct {
|
||||
Name string `json:"name"`
|
||||
MediaType string `json:"media_type,omitempty"`
|
||||
Content string `json:"content_base64,omitempty"`
|
||||
Digest string `json:"digest,omitempty"`
|
||||
OriginURI string `json:"origin_uri,omitempty"`
|
||||
SizeBytes int64 `json:"size_bytes,omitempty"`
|
||||
}
|
||||
|
||||
type debugStructuredCompletionRequest struct {
|
||||
StageName string `json:"stage_name"`
|
||||
PromptID string `json:"prompt_id,omitempty"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
ProfileID string `json:"profile_id,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
Inputs map[string]debugLLMInputMaterial `json:"inputs,omitempty"`
|
||||
Vars map[string]any `json:"vars,omitempty"`
|
||||
}
|
||||
type debugStructuredCompletionRequest = contracts.DebugStructuredCompletionRequest
|
||||
|
||||
type debugStructuredCompletionResponse struct {
|
||||
Content string `json:"content,omitempty"`
|
||||
@@ -160,9 +144,37 @@ type debugLLMCallReference struct {
|
||||
PromptID string `json:"prompt_id,omitempty"`
|
||||
ProfileID string `json:"profile_id,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
PromptTokens int `json:"prompt_tokens,omitempty"`
|
||||
CompletionTokens int `json:"completion_tokens,omitempty"`
|
||||
TotalTokens int `json:"total_tokens,omitempty"`
|
||||
RepairAttempts int `json:"repair_attempts,omitempty"`
|
||||
Error bool `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type debugValidationOutcome struct {
|
||||
ValidatorName string `json:"validator_name"`
|
||||
Outcome string `json:"outcome"`
|
||||
AttemptCount int `json:"attempt_count"`
|
||||
ReasonCode string `json:"reason_code,omitempty"`
|
||||
}
|
||||
|
||||
type debugProducerAttempt struct {
|
||||
Number int `json:"number"`
|
||||
Kind string `json:"kind"`
|
||||
Outcome string `json:"outcome"`
|
||||
Validation []debugValidationOutcome `json:"validation,omitempty"`
|
||||
}
|
||||
|
||||
type debugProducerTerminal struct {
|
||||
ProducerAttemptCount int `json:"producer_attempt_count"`
|
||||
Attempts []debugProducerAttempt `json:"attempts,omitempty"`
|
||||
AggregateReasonCodes []string `json:"aggregate_reason_codes,omitempty"`
|
||||
ValidationComplete bool `json:"validation_complete"`
|
||||
EffectivePolicy ValidationPolicy `json:"effective_policy"`
|
||||
TerminalAction string `json:"terminal_action"`
|
||||
Summary artifacts.ValidationSummary `json:"validation_summary"`
|
||||
}
|
||||
|
||||
type debugValidationCall struct {
|
||||
ValidatorName string `json:"validator_name"`
|
||||
Request any `json:"request"`
|
||||
@@ -267,6 +279,10 @@ func (client *debugLLMClient) CompleteStructured(ctx context.Context, req contra
|
||||
PromptID: req.PromptID,
|
||||
ProfileID: debugFirstNonEmptyString(response.ProfileID, req.ProfileID),
|
||||
Model: debugFirstNonEmptyString(response.Model, debugResponseModel(response)),
|
||||
PromptTokens: response.PromptTokens,
|
||||
CompletionTokens: response.CompletionTokens,
|
||||
TotalTokens: response.TotalTokens,
|
||||
RepairAttempts: response.RepairAttempts,
|
||||
Error: err != nil,
|
||||
}
|
||||
if scope := debugLLMScopeFromContext(ctx); scope != nil {
|
||||
@@ -423,6 +439,50 @@ func (r attemptTerminalRecorder) record(payload any, terminalErr error) error {
|
||||
return terminalErr
|
||||
}
|
||||
|
||||
func debugValidationOutcomes(report validationReport) []debugValidationOutcome {
|
||||
if len(report.records) == 0 {
|
||||
return nil
|
||||
}
|
||||
outcomes := make([]debugValidationOutcome, 0, len(report.records))
|
||||
for _, record := range report.records {
|
||||
outcomes = append(outcomes, debugValidationOutcome{
|
||||
ValidatorName: record.validatorName,
|
||||
Outcome: string(record.outcome),
|
||||
AttemptCount: record.attemptCount,
|
||||
ReasonCode: record.reasonCode,
|
||||
})
|
||||
}
|
||||
return outcomes
|
||||
}
|
||||
|
||||
func writeProducerTerminalDebug(recorder DebugRecorder, name string, terminal producerAttemptTerminal, policy ValidationPolicy, summary artifacts.ValidationSummary) error {
|
||||
attempts := make([]debugProducerAttempt, 0, len(terminal.Provenance))
|
||||
for _, item := range terminal.Provenance {
|
||||
attempts = append(attempts, debugProducerAttempt{
|
||||
Number: item.Number,
|
||||
Kind: string(item.Kind),
|
||||
Outcome: string(item.Outcome),
|
||||
Validation: debugValidationOutcomes(item.Validation),
|
||||
})
|
||||
}
|
||||
return writeDebugTimed(recorder, name, debugTimedEnvelope{
|
||||
Stage: summary.Stage,
|
||||
StepID: summary.StepID,
|
||||
LaneID: summary.LaneID,
|
||||
ModuleKey: summary.ModuleKey,
|
||||
StartedAt: time.Now().UTC(),
|
||||
Payload: debugProducerTerminal{
|
||||
ProducerAttemptCount: summary.ProducerAttemptCount,
|
||||
Attempts: attempts,
|
||||
AggregateReasonCodes: append([]string(nil), summary.ReasonCodes...),
|
||||
ValidationComplete: summary.Status == "complete",
|
||||
EffectivePolicy: policy,
|
||||
TerminalAction: string(terminal.Action),
|
||||
Summary: artifacts.CloneValidationSummary(summary),
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
func debugContentEnvelope(content []byte, mediaType string, metadata map[string]any, warnings []contracts.Warning) debugBinaryEnvelope {
|
||||
content = redactSecretBytes(content)
|
||||
return debugBinaryEnvelope{
|
||||
@@ -567,29 +627,7 @@ func debugOutputFiles(files []contracts.OutputFile) []debugOutputFile {
|
||||
}
|
||||
|
||||
func debugCompletionRequest(req contracts.StructuredCompletionRequest) debugStructuredCompletionRequest {
|
||||
inputs := make(map[string]debugLLMInputMaterial, len(req.Inputs))
|
||||
for key, material := range req.Inputs {
|
||||
inputs[key] = debugLLMInputMaterial{
|
||||
Name: material.Name,
|
||||
MediaType: material.MediaType,
|
||||
Content: base64.StdEncoding.EncodeToString(redactSecretBytes(material.Content)),
|
||||
Digest: material.Digest,
|
||||
OriginURI: material.OriginURI,
|
||||
SizeBytes: material.SizeBytes,
|
||||
}
|
||||
}
|
||||
if len(inputs) == 0 {
|
||||
inputs = nil
|
||||
}
|
||||
return debugStructuredCompletionRequest{
|
||||
StageName: req.StageName,
|
||||
PromptID: req.PromptID,
|
||||
PromptVersion: req.PromptVersion,
|
||||
ProfileID: req.ProfileID,
|
||||
SessionID: req.SessionID,
|
||||
Inputs: inputs,
|
||||
Vars: redactSensitiveMap(req.Vars),
|
||||
}
|
||||
return req.DebugSummary()
|
||||
}
|
||||
|
||||
func debugCompletionResponse(response contracts.StructuredCompletionResponse) debugStructuredCompletionResponse {
|
||||
@@ -691,6 +729,7 @@ func debugResponseModel(response contracts.StructuredCompletionResponse) string
|
||||
|
||||
func debugValidationResultEnvelope(result contracts.ValidationResult) contracts.ValidationResult {
|
||||
result.Message = string(redactSecretBytes([]byte(result.Message)))
|
||||
result.CorrectionGuidance = ""
|
||||
result.DiagnosticArtifactPath = string(redactSecretBytes([]byte(result.DiagnosticArtifactPath)))
|
||||
for i := range result.Warnings {
|
||||
result.Warnings[i].Message = string(redactSecretBytes([]byte(result.Warnings[i].Message)))
|
||||
|
||||
@@ -47,6 +47,32 @@ func TestDebugLLMPathsKeepDotIdentitiesDistinct(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDebugCompletionRequestOmitsCorrectionContent(t *testing.T) {
|
||||
const assistantResponse = `{"secret":"assistant response"}`
|
||||
const userGuidance = "secret user guidance"
|
||||
correction, err := contracts.NewSemanticCorrection([]byte(assistantResponse), userGuidance)
|
||||
if err != nil {
|
||||
t.Fatalf("NewSemanticCorrection() error = %v", err)
|
||||
}
|
||||
summary := debugCompletionRequest(contracts.StructuredCompletionRequest{
|
||||
Inputs: contracts.LLMInputSet{"source": contracts.NewLLMInputMaterial("source", "application/json", []byte(`{"source":true}`), "", "")},
|
||||
Vars: map[string]any{"custom": "value"},
|
||||
Correction: correction,
|
||||
})
|
||||
encoded, err := json.Marshal(summary)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal completion summary: %v", err)
|
||||
}
|
||||
for _, secret := range []string{assistantResponse, userGuidance} {
|
||||
if strings.Contains(string(encoded), secret) {
|
||||
t.Fatalf("debug completion summary leaked %q: %s", secret, encoded)
|
||||
}
|
||||
}
|
||||
if summary.InputCount != 1 || summary.VariableCount != 1 || summary.Correction == nil {
|
||||
t.Fatalf("debug completion summary = %#v, want counts and correction metadata", summary)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDebugSourceDocumentPreservesUnitReferences(t *testing.T) {
|
||||
doc := validSourceDocument()
|
||||
envelope := debugSourceDocumentEnvelope(doc)
|
||||
|
||||
@@ -114,9 +114,10 @@ func TestRunnerEvidenceContextOmitsAbsentAndRejectedLanes(t *testing.T) {
|
||||
prepared.Steps[1].lanes[0].mergeValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("reject"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted"}, nil
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted", CorrectionGuidance: "return an acceptable candidate"}, nil
|
||||
},
|
||||
}}
|
||||
prepared.Steps[1].lanes[0].resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
installEvidencePlan(prepared, 0, []string{"absent", "present", "rejected"}, func(notes codecNotes) ([]source.SourceRef, error) {
|
||||
if len(notes.Items) > 0 && notes.Items[0] == "present" {
|
||||
return []source.SourceRef{{SourceID: "source", StartUnitID: 1, EndUnitID: 1}}, nil
|
||||
|
||||
@@ -58,11 +58,20 @@ func RegisterExtractorBuilder[T any](registry *ExtractorRegistry, spec ModuleSpe
|
||||
if !ok {
|
||||
return erasedTypedResult{}, fmt.Errorf("extractor %q has incompatible implementation %T", normalized.Key, implementation)
|
||||
}
|
||||
correction, err := contracts.CloneSemanticCorrection(request.Correction)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, fmt.Errorf("clone extraction correction: %w", err)
|
||||
}
|
||||
request.Correction = correction
|
||||
result, err := extractor.Extract(ctx, request)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
|
||||
candidate, err := contracts.CloneModelCandidate(result.ModelCandidate)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, fmt.Errorf("clone extraction model candidate: %w", err)
|
||||
}
|
||||
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), ModelCandidate: candidate}, nil
|
||||
}}
|
||||
if registry.typedEntries == nil {
|
||||
registry.typedEntries = map[string]typedExtractorEntry{}
|
||||
|
||||
@@ -41,6 +41,54 @@ func operationReferenceSet(input RunInput, target ResolvedReferenceTarget) contr
|
||||
return CloneReferenceSet(target.ReferenceSet)
|
||||
}
|
||||
|
||||
// referenceTargetReuseEligible reports whether every generated artifact in a
|
||||
// stage's reference set descends exclusively from fully validated work. Static
|
||||
// references and callers that do not supply lineage metadata are reusable.
|
||||
func referenceTargetReuseEligible(input RunInput, target ResolvedReferenceTarget) bool {
|
||||
if input.referenceReuseEligibility == nil {
|
||||
return true
|
||||
}
|
||||
eligible, ok := input.referenceReuseEligibility[keyForReferenceTarget(target)]
|
||||
return !ok || eligible
|
||||
}
|
||||
|
||||
func laneReferencesReuseEligible(input RunInput, lane ResolvedArtifactLane) bool {
|
||||
return referenceTargetReuseEligible(input, lane.ExtractReferences) &&
|
||||
referenceTargetReuseEligible(input, lane.MergeReferences) &&
|
||||
referenceTargetReuseEligible(input, lane.NormalizeReferences)
|
||||
}
|
||||
|
||||
// buildStepReferenceReuseEligibility carries validation completeness alongside
|
||||
// generated references without exposing the internal lineage flag in artifact
|
||||
// payloads. A target becomes ineligible when any generated input is ineligible.
|
||||
func buildStepReferenceReuseEligibility(step PreparedPipelineStep, outputs map[generatedOutputKey]bool) map[referenceTargetKey]bool {
|
||||
eligibility := make(map[referenceTargetKey]bool)
|
||||
for _, prepared := range step.lanes {
|
||||
lane := prepared.resolved
|
||||
for _, target := range []ResolvedReferenceTarget{lane.ExtractReferences, lane.MergeReferences, lane.NormalizeReferences} {
|
||||
generated := false
|
||||
eligible := true
|
||||
for _, binding := range target.Bindings {
|
||||
if binding.Artifact == nil {
|
||||
continue
|
||||
}
|
||||
generated = true
|
||||
producer := generatedOutputKeyFor(binding.Artifact.Step, binding.Artifact.Lane)
|
||||
if reusable, ok := outputs[producer]; ok && !reusable {
|
||||
eligible = false
|
||||
}
|
||||
}
|
||||
if generated {
|
||||
eligibility[keyForReferenceTarget(target)] = eligible
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(eligibility) == 0 {
|
||||
return nil
|
||||
}
|
||||
return eligibility
|
||||
}
|
||||
|
||||
// buildStepReferenceSets resolves every generated binding for a step before
|
||||
// any lane in that step is allowed to start. Each returned set is a fresh
|
||||
// operation-time view; prepared reference sets are never modified.
|
||||
|
||||
@@ -85,11 +85,19 @@ func RegisterMergerBuilder[T any](registry *MergerRegistry, spec ModuleSpec, val
|
||||
}
|
||||
outputs[i] = contracts.ExtractArtifact[T]{LaneID: output.LaneID, ExtractorKey: output.ExtractorKey, SourceID: output.SourceID, ChunkID: output.ChunkID, ChunkIndex: output.ChunkIndex, ChunkRef: output.ChunkRef, Value: value}
|
||||
}
|
||||
result, err := merger.Merge(ctx, contracts.TypedMergeRequest[T]{Source: request.Source, LaneID: request.LaneID, ExtractOutputs: outputs, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, StructuredOutputRepairAttempts: request.StructuredOutputRepairAttempts, Metadata: request.Metadata})
|
||||
correction, err := contracts.CloneSemanticCorrection(request.Correction)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, fmt.Errorf("clone merge correction: %w", err)
|
||||
}
|
||||
result, err := merger.Merge(ctx, contracts.TypedMergeRequest[T]{Source: request.Source, LaneID: request.LaneID, ExtractOutputs: outputs, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, StructuredOutputRepairAttempts: request.StructuredOutputRepairAttempts, Correction: correction, Metadata: request.Metadata})
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
|
||||
candidate, err := contracts.CloneModelCandidate(result.ModelCandidate)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, fmt.Errorf("clone merge model candidate: %w", err)
|
||||
}
|
||||
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), ModelCandidate: candidate}, nil
|
||||
},
|
||||
}
|
||||
return nil
|
||||
|
||||
@@ -21,25 +21,28 @@ const (
|
||||
)
|
||||
|
||||
type ModuleSpec struct {
|
||||
Key string
|
||||
Stage ModuleStage
|
||||
ExecutionClass contracts.ExecutionClass
|
||||
ArtifactKind contracts.ArtifactKind
|
||||
Provides []string
|
||||
Requires []string
|
||||
ReferenceSlots []contracts.ReferenceSlot
|
||||
Key string
|
||||
Stage ModuleStage
|
||||
ExecutionClass contracts.ExecutionClass
|
||||
CorrectionProtocol contracts.CorrectionProtocol
|
||||
ArtifactKind contracts.ArtifactKind
|
||||
Provides []string
|
||||
Requires []string
|
||||
ReferenceSlots []contracts.ReferenceSlot
|
||||
}
|
||||
|
||||
func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
|
||||
executionClass := contracts.ExecutionClass(strings.TrimSpace(string(spec.ExecutionClass)))
|
||||
correctionProtocol := contracts.CorrectionProtocol(strings.TrimSpace(string(spec.CorrectionProtocol)))
|
||||
return ModuleSpec{
|
||||
Key: strings.TrimSpace(spec.Key),
|
||||
Stage: spec.Stage,
|
||||
ExecutionClass: executionClass,
|
||||
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
|
||||
Provides: normalizeCapabilities(spec.Provides),
|
||||
Requires: normalizeCapabilities(spec.Requires),
|
||||
ReferenceSlots: normalizeReferenceSlots(spec.ReferenceSlots),
|
||||
Key: strings.TrimSpace(spec.Key),
|
||||
Stage: spec.Stage,
|
||||
ExecutionClass: executionClass,
|
||||
CorrectionProtocol: correctionProtocol,
|
||||
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
|
||||
Provides: normalizeCapabilities(spec.Provides),
|
||||
Requires: normalizeCapabilities(spec.Requires),
|
||||
ReferenceSlots: normalizeReferenceSlots(spec.ReferenceSlots),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -70,13 +73,14 @@ func normalizeCapabilities(values []string) []string {
|
||||
|
||||
func cloneModuleSpec(spec ModuleSpec) ModuleSpec {
|
||||
return ModuleSpec{
|
||||
Key: spec.Key,
|
||||
Stage: spec.Stage,
|
||||
ExecutionClass: spec.ExecutionClass,
|
||||
ArtifactKind: spec.ArtifactKind,
|
||||
Provides: append([]string(nil), spec.Provides...),
|
||||
Requires: append([]string(nil), spec.Requires...),
|
||||
ReferenceSlots: contracts.CloneReferenceSlots(spec.ReferenceSlots),
|
||||
Key: spec.Key,
|
||||
Stage: spec.Stage,
|
||||
ExecutionClass: spec.ExecutionClass,
|
||||
CorrectionProtocol: spec.CorrectionProtocol,
|
||||
ArtifactKind: spec.ArtifactKind,
|
||||
Provides: append([]string(nil), spec.Provides...),
|
||||
Requires: append([]string(nil), spec.Requires...),
|
||||
ReferenceSlots: contracts.CloneReferenceSlots(spec.ReferenceSlots),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -93,6 +97,19 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
|
||||
if spec.ExecutionClass != contracts.ExecutionClassDeterministic && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
||||
return fmt.Errorf("%s %q has unsupported execution class %q", kind, spec.Key, spec.ExecutionClass)
|
||||
}
|
||||
if spec.CorrectionProtocol != "" {
|
||||
if err := spec.CorrectionProtocol.Validate(); err != nil {
|
||||
return fmt.Errorf("%s %q correction protocol: %w", kind, spec.Key, err)
|
||||
}
|
||||
if spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
||||
return fmt.Errorf("%s %q correction protocol requires an LLM-backed execution class", kind, spec.Key)
|
||||
}
|
||||
switch spec.Stage {
|
||||
case StageChunk, StageExtract, StageMerge, StageNormalize:
|
||||
default:
|
||||
return fmt.Errorf("%s %q correction protocol is not supported for %q stage", kind, spec.Key, spec.Stage)
|
||||
}
|
||||
}
|
||||
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
|
||||
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
|
||||
}
|
||||
|
||||
@@ -32,14 +32,70 @@ func TestValidateModuleSpecRequiresSupportedExecutionClass(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCloneModuleSpecPreservesExecutionClass(t *testing.T) {
|
||||
spec := normalizeModuleSpec(ModuleSpec{Key: " module ", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked})
|
||||
func TestCloneModuleSpecPreservesCorrectionProtocol(t *testing.T) {
|
||||
spec := normalizeModuleSpec(ModuleSpec{
|
||||
Key: " module ",
|
||||
Stage: StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
CorrectionProtocol: " single_response_v1 ",
|
||||
})
|
||||
if spec.CorrectionProtocol != contracts.CorrectionProtocolSingleResponseV1 {
|
||||
t.Fatalf("normalized CorrectionProtocol = %q, want %q", spec.CorrectionProtocol, contracts.CorrectionProtocolSingleResponseV1)
|
||||
}
|
||||
cloned := cloneModuleSpec(spec)
|
||||
if !reflect.DeepEqual(cloned, spec) {
|
||||
t.Fatalf("cloneModuleSpec() = %#v, want %#v", cloned, spec)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateModuleSpecCorrectionProtocolEligibility(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
stage ModuleStage
|
||||
class contracts.ExecutionClass
|
||||
protocol contracts.CorrectionProtocol
|
||||
want string
|
||||
}{
|
||||
{name: "LLM chunk", stage: StageChunk, class: contracts.ExecutionClassLLMBacked, protocol: contracts.CorrectionProtocolSingleResponseV1},
|
||||
{name: "LLM extract", stage: StageExtract, class: contracts.ExecutionClassLLMBacked, protocol: contracts.CorrectionProtocolSingleResponseV1},
|
||||
{name: "LLM merge", stage: StageMerge, class: contracts.ExecutionClassLLMBacked, protocol: contracts.CorrectionProtocolSingleResponseV1},
|
||||
{name: "LLM normalize", stage: StageNormalize, class: contracts.ExecutionClassLLMBacked, protocol: contracts.CorrectionProtocolSingleResponseV1},
|
||||
{name: "deterministic chunk", stage: StageChunk, class: contracts.ExecutionClassDeterministic, protocol: contracts.CorrectionProtocolSingleResponseV1, want: "LLM-backed"},
|
||||
{name: "input", stage: StageInput, class: contracts.ExecutionClassLLMBacked, protocol: contracts.CorrectionProtocolSingleResponseV1, want: "not supported"},
|
||||
{name: "output", stage: StageOutput, class: contracts.ExecutionClassLLMBacked, protocol: contracts.CorrectionProtocolSingleResponseV1, want: "not supported"},
|
||||
{name: "unknown protocol", stage: StageChunk, class: contracts.ExecutionClassLLMBacked, protocol: "unsupported", want: "unsupported correction protocol"},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
spec := normalizeModuleSpec(ModuleSpec{
|
||||
Key: "module",
|
||||
Stage: test.stage,
|
||||
ExecutionClass: test.class,
|
||||
CorrectionProtocol: test.protocol,
|
||||
})
|
||||
err := validateModuleSpec("module", test.stage, spec)
|
||||
if test.want == "" && err != nil {
|
||||
t.Fatalf("validateModuleSpec() error = %v, want nil", err)
|
||||
}
|
||||
if test.want != "" && (err == nil || !strings.Contains(err.Error(), test.want)) {
|
||||
t.Fatalf("validateModuleSpec() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeValidatorSpecRejectsCorrectionProtocol(t *testing.T) {
|
||||
_, err := normalizeValidatorSpec(ValidatorSpec{
|
||||
Key: "validator",
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "correction protocol") {
|
||||
t.Fatalf("normalizeValidatorSpec() error = %v, want correction protocol error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
|
||||
@@ -76,11 +76,19 @@ func RegisterNormalizerBuilder[T any](registry *NormalizerRegistry, spec ModuleS
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
result, err := normalizer.Normalize(ctx, contracts.TypedNormalizeRequest[T]{Source: request.Source, LaneID: request.LaneID, MergeOutput: contracts.MergeArtifact[T]{LaneID: request.MergeOutput.LaneID, MergerKey: request.MergeOutput.MergerKey, SourceID: request.MergeOutput.SourceID, Value: value}, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, StructuredOutputRepairAttempts: request.StructuredOutputRepairAttempts, Metadata: request.Metadata})
|
||||
correction, err := contracts.CloneSemanticCorrection(request.Correction)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, fmt.Errorf("clone normalize correction: %w", err)
|
||||
}
|
||||
result, err := normalizer.Normalize(ctx, contracts.TypedNormalizeRequest[T]{Source: request.Source, LaneID: request.LaneID, MergeOutput: contracts.MergeArtifact[T]{LaneID: request.MergeOutput.LaneID, MergerKey: request.MergeOutput.MergerKey, SourceID: request.MergeOutput.SourceID, Value: value}, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, StructuredOutputRepairAttempts: request.StructuredOutputRepairAttempts, Correction: correction, Metadata: request.Metadata})
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), Retry: cloneNormalizeRetry(result.Retry)}, nil
|
||||
candidate, err := contracts.CloneModelCandidate(result.ModelCandidate)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, fmt.Errorf("clone normalize model candidate: %w", err)
|
||||
}
|
||||
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), Retry: cloneNormalizeRetry(result.Retry), ModelCandidate: candidate}, nil
|
||||
},
|
||||
}
|
||||
return nil
|
||||
|
||||
@@ -13,10 +13,11 @@ import (
|
||||
// resolved pipeline. Its implementation values are private so execution cannot
|
||||
// replace or reconfigure them after preparation.
|
||||
type PreparedPipeline struct {
|
||||
Input ModuleBinding
|
||||
Chunk ModuleBinding
|
||||
Steps []PreparedPipelineStep
|
||||
Output ModuleBinding
|
||||
Input ModuleBinding
|
||||
Chunk ModuleBinding
|
||||
ChunkCorrectionProtocol contracts.CorrectionProtocol
|
||||
Steps []PreparedPipelineStep
|
||||
Output ModuleBinding
|
||||
|
||||
resolved ResolvedPipeline
|
||||
dependencies ModuleDependencies
|
||||
@@ -36,7 +37,10 @@ type PreparedPipelineStep struct {
|
||||
}
|
||||
|
||||
type PreparedArtifactLane struct {
|
||||
Resolved ResolvedArtifactLane
|
||||
Resolved ResolvedArtifactLane
|
||||
ExtractCorrectionProtocol contracts.CorrectionProtocol
|
||||
MergeCorrectionProtocol contracts.CorrectionProtocol
|
||||
NormalizeCorrectionProtocol contracts.CorrectionProtocol
|
||||
}
|
||||
|
||||
type preparedLaneExecutor struct {
|
||||
@@ -79,6 +83,7 @@ type preparedValidator struct {
|
||||
typedValidate typedValidateOperation
|
||||
chunk contracts.ChunkValidator
|
||||
serialized contracts.SerializedValidator
|
||||
position int
|
||||
}
|
||||
|
||||
// Prepare validates all configured options and constructs every selected
|
||||
@@ -87,17 +92,21 @@ func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDepend
|
||||
if err := validateResolvedPipeline(resolved); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := validateCorrectionRetryCapabilities(resolved); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := validateRegistrySet(resolved, registries); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
stable := cloneResolvedPipeline(resolved)
|
||||
prepared := &PreparedPipeline{
|
||||
Input: cloneModuleBinding(stable.Input),
|
||||
Chunk: cloneModuleBinding(stable.Chunk),
|
||||
Output: cloneModuleBinding(stable.Output),
|
||||
resolved: stable,
|
||||
dependencies: deps,
|
||||
artifactCodecs: registries.ArtifactCodecs,
|
||||
Input: cloneModuleBinding(stable.Input),
|
||||
Chunk: cloneModuleBinding(stable.Chunk),
|
||||
ChunkCorrectionProtocol: stable.ChunkCorrectionProtocol,
|
||||
Output: cloneModuleBinding(stable.Output),
|
||||
resolved: stable,
|
||||
dependencies: deps,
|
||||
artifactCodecs: registries.ArtifactCodecs,
|
||||
}
|
||||
request := func(binding ModuleBinding, references contracts.ReferenceSet) BuildRequest {
|
||||
return BuildRequest{Dependencies: deps, Options: binding.Options, References: references}
|
||||
@@ -131,7 +140,12 @@ func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDepend
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
preparedStep.ArtifactLanes = append(preparedStep.ArtifactLanes, PreparedArtifactLane{Resolved: cloneResolvedArtifactLane(lane)})
|
||||
preparedStep.ArtifactLanes = append(preparedStep.ArtifactLanes, PreparedArtifactLane{
|
||||
Resolved: cloneResolvedArtifactLane(lane),
|
||||
ExtractCorrectionProtocol: lane.ExtractCorrectionProtocol,
|
||||
MergeCorrectionProtocol: lane.MergeCorrectionProtocol,
|
||||
NormalizeCorrectionProtocol: lane.NormalizeCorrectionProtocol,
|
||||
})
|
||||
preparedStep.lanes = append(preparedStep.lanes, executor)
|
||||
}
|
||||
prepared.Steps[stepIndex] = preparedStep
|
||||
@@ -199,6 +213,40 @@ func prepareEvidencePlan(resolved ResolvedPipeline, registries Registries, outpu
|
||||
return plan, nil
|
||||
}
|
||||
|
||||
func validateCorrectionRetryCapabilities(pipeline ResolvedPipeline) error {
|
||||
validate := func(stage ModuleStage, laneID string, binding ModuleBinding, executionClass contracts.ExecutionClass, protocol contracts.CorrectionProtocol) error {
|
||||
if executionClass != contracts.ExecutionClassLLMBacked || binding.Retries == 0 {
|
||||
return nil
|
||||
}
|
||||
chain := resolvedValidatorChain(stage, laneID, binding.Module, pipeline.ValidatorChains)
|
||||
if len(chain.Validators) == 0 || protocol == contracts.CorrectionProtocolSingleResponseV1 {
|
||||
return nil
|
||||
}
|
||||
if laneID == "" {
|
||||
return fmt.Errorf("pipeline %q %s module %q configures validators and retries but does not declare correction protocol %q", pipeline.ID, stage, binding.Module, contracts.CorrectionProtocolSingleResponseV1)
|
||||
}
|
||||
return fmt.Errorf("pipeline %q lane %q %s module %q configures validators and retries but does not declare correction protocol %q", pipeline.ID, laneID, stage, binding.Module, contracts.CorrectionProtocolSingleResponseV1)
|
||||
}
|
||||
|
||||
if err := validate(StageChunk, "", pipeline.Chunk, pipeline.ChunkExecutionClass, pipeline.ChunkCorrectionProtocol); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, step := range pipeline.Steps {
|
||||
for _, lane := range step.ArtifactLanes {
|
||||
if err := validate(StageExtract, lane.ID, lane.Extract, lane.ExtractExecutionClass, lane.ExtractCorrectionProtocol); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validate(StageMerge, lane.ID, lane.Merge, lane.MergeExecutionClass, lane.MergeCorrectionProtocol); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validate(StageNormalize, lane.ID, lane.Normalize, lane.NormalizeExecutionClass, lane.NormalizeCorrectionProtocol); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func prepareLane(pipeline ResolvedPipeline, lane ResolvedArtifactLane, registries Registries, deps ModuleDependencies) (preparedLaneExecutor, error) {
|
||||
executor := preparedLaneExecutor{resolved: cloneResolvedArtifactLane(lane)}
|
||||
request := func(binding ModuleBinding, references contracts.ReferenceSet) BuildRequest {
|
||||
@@ -406,6 +454,7 @@ func validateRegistrySet(resolved ResolvedPipeline, registries Registries) error
|
||||
|
||||
func cloneResolvedPipeline(in ResolvedPipeline) ResolvedPipeline {
|
||||
out := in
|
||||
out.ConfiguredValidationPolicy = cloneValidationPolicyOverride(in.ConfiguredValidationPolicy)
|
||||
out.Input = cloneModuleBinding(in.Input)
|
||||
out.Chunk = cloneModuleBinding(in.Chunk)
|
||||
out.Output = cloneModuleBinding(in.Output)
|
||||
|
||||
391
internal/framework/pipeline/producer_attempts.go
Normal file
391
internal/framework/pipeline/producer_attempts.go
Normal file
@@ -0,0 +1,391 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
// producerAttemptKind records why a producer invocation followed the prior
|
||||
// one. It is deliberately independent of any artifact family.
|
||||
type producerAttemptKind string
|
||||
|
||||
const (
|
||||
producerAttemptInitial producerAttemptKind = "initial"
|
||||
producerAttemptOperationalRetry producerAttemptKind = "operational_error_retry"
|
||||
producerAttemptStructuralRetry producerAttemptKind = "structural_retry"
|
||||
producerAttemptModuleRetry producerAttemptKind = "module_requested_retry"
|
||||
producerAttemptSemanticRetry producerAttemptKind = "semantic_correction"
|
||||
)
|
||||
|
||||
type producerAttemptOutcome string
|
||||
|
||||
const (
|
||||
producerAttemptAccepted producerAttemptOutcome = "accepted"
|
||||
producerAttemptRejected producerAttemptOutcome = "rejected"
|
||||
producerAttemptIncompleteAccepted producerAttemptOutcome = "incomplete_accepted"
|
||||
producerAttemptRetried producerAttemptOutcome = "retried"
|
||||
producerAttemptFailed producerAttemptOutcome = "failed"
|
||||
)
|
||||
|
||||
type producerTerminalAction string
|
||||
|
||||
const (
|
||||
producerTerminalAccepted producerTerminalAction = "accepted"
|
||||
producerTerminalRejected producerTerminalAction = "reject_output"
|
||||
producerTerminalIncompleteAccepted producerTerminalAction = "warn_continue"
|
||||
producerTerminalFailed producerTerminalAction = "fail_run"
|
||||
)
|
||||
|
||||
// producerAttemptRequest contains only attempt-local control material. The
|
||||
// producer reconstructs its ordinary request from its own durable inputs.
|
||||
type producerAttemptRequest struct {
|
||||
Number int
|
||||
Kind producerAttemptKind
|
||||
Correction *contracts.SemanticCorrection
|
||||
}
|
||||
|
||||
// producerRetryDirective asks for another producer invocation while retaining
|
||||
// the current value as a safe fallback if its shared budget is exhausted.
|
||||
// Artifact-specific adapters are responsible for validating and populating it.
|
||||
type producerRetryDirective struct {
|
||||
FallbackWarnings []contracts.Warning
|
||||
}
|
||||
|
||||
func (directive *producerRetryDirective) clone() *producerRetryDirective {
|
||||
if directive == nil {
|
||||
return nil
|
||||
}
|
||||
return &producerRetryDirective{FallbackWarnings: cloneWarnings(directive.FallbackWarnings)}
|
||||
}
|
||||
|
||||
// producerAttemptOutput is intentionally artifact-neutral. Value remains
|
||||
// opaque to the state machine; Candidate is the attempt-local response that
|
||||
// may support semantic correction.
|
||||
type producerAttemptOutput struct {
|
||||
Value any
|
||||
Candidate *contracts.ModelCandidate
|
||||
Warnings []contracts.Warning
|
||||
Retry *producerRetryDirective
|
||||
}
|
||||
|
||||
func (output producerAttemptOutput) clone() (producerAttemptOutput, error) {
|
||||
candidate, err := contracts.CloneModelCandidate(output.Candidate)
|
||||
if err != nil {
|
||||
return producerAttemptOutput{}, fmt.Errorf("clone model candidate: %w", err)
|
||||
}
|
||||
return producerAttemptOutput{
|
||||
Value: output.Value,
|
||||
Candidate: candidate,
|
||||
Warnings: cloneWarnings(output.Warnings),
|
||||
Retry: output.Retry.clone(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
type producerAttemptProducer func(context.Context, producerAttemptRequest) (producerAttemptOutput, error)
|
||||
type producerAttemptValidator func(context.Context, producerAttemptOutput) (validationReport, error)
|
||||
|
||||
type producerAttemptConfig struct {
|
||||
Retries int
|
||||
Policy ValidationPolicy
|
||||
AllowStructuralRetry bool
|
||||
}
|
||||
|
||||
// producerAttemptProvenance is ordered by producer invocation. It retains the
|
||||
// settled validation report for later debug and durable-provenance adapters.
|
||||
type producerAttemptProvenance struct {
|
||||
Number int
|
||||
Kind producerAttemptKind
|
||||
Outcome producerAttemptOutcome
|
||||
Validation validationReport
|
||||
}
|
||||
|
||||
func (provenance producerAttemptProvenance) clone() producerAttemptProvenance {
|
||||
provenance.Validation = cloneValidationReport(provenance.Validation)
|
||||
return provenance
|
||||
}
|
||||
|
||||
// producerAttemptTerminal describes the state-machine decision without
|
||||
// materializing an artifact or persisting stage-specific diagnostics.
|
||||
type producerAttemptTerminal struct {
|
||||
Action producerTerminalAction
|
||||
Value any
|
||||
Warnings []contracts.Warning
|
||||
Rejection *contracts.RejectedOutput
|
||||
Validation validationReport
|
||||
ValidationIncomplete bool
|
||||
Provenance []producerAttemptProvenance
|
||||
}
|
||||
|
||||
func (terminal producerAttemptTerminal) clone() producerAttemptTerminal {
|
||||
terminal.Warnings = cloneWarnings(terminal.Warnings)
|
||||
if terminal.Rejection != nil {
|
||||
rejection := *terminal.Rejection
|
||||
rejection.Validation = cloneValidationSummaryPtr(rejection.Validation)
|
||||
terminal.Rejection = &rejection
|
||||
}
|
||||
terminal.Validation = cloneValidationReport(terminal.Validation)
|
||||
terminal.Provenance = cloneProducerAttemptProvenance(terminal.Provenance)
|
||||
return terminal
|
||||
}
|
||||
|
||||
func cloneProducerAttemptProvenance(provenance []producerAttemptProvenance) []producerAttemptProvenance {
|
||||
if len(provenance) == 0 {
|
||||
return nil
|
||||
}
|
||||
cloned := make([]producerAttemptProvenance, len(provenance))
|
||||
for index, item := range provenance {
|
||||
cloned[index] = item.clone()
|
||||
}
|
||||
return cloned
|
||||
}
|
||||
|
||||
func cloneValidationReport(report validationReport) validationReport {
|
||||
return validationReport{records: report.Records()}
|
||||
}
|
||||
|
||||
func runProducerAttempts(ctx context.Context, config producerAttemptConfig, produce producerAttemptProducer, validate producerAttemptValidator) (producerAttemptTerminal, error) {
|
||||
if ctx == nil {
|
||||
return producerAttemptTerminal{}, errors.New("producer attempt context must not be nil")
|
||||
}
|
||||
if config.Retries < 0 {
|
||||
return producerAttemptTerminal{}, errors.New("producer retries must not be negative")
|
||||
}
|
||||
if produce == nil {
|
||||
return producerAttemptTerminal{}, errors.New("producer attempt closure must not be nil")
|
||||
}
|
||||
if validate == nil {
|
||||
return producerAttemptTerminal{}, errors.New("producer validation closure must not be nil")
|
||||
}
|
||||
|
||||
attemptLimit := config.Retries + 1
|
||||
provenance := make([]producerAttemptProvenance, 0, attemptLimit)
|
||||
kind := producerAttemptInitial
|
||||
var correction *contracts.SemanticCorrection
|
||||
|
||||
for number := 1; number <= attemptLimit; number++ {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return failedProducerAttempt(provenance), err
|
||||
}
|
||||
requestCorrection, err := contracts.CloneSemanticCorrection(correction)
|
||||
if err != nil {
|
||||
return failedProducerAttempt(provenance), fmt.Errorf("clone semantic correction: %w", err)
|
||||
}
|
||||
output, err := produce(ctx, producerAttemptRequest{Number: number, Kind: kind, Correction: requestCorrection})
|
||||
if err != nil {
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed})
|
||||
if isImmediateProducerFailure(err) {
|
||||
return failedProducerAttempt(provenance), err
|
||||
}
|
||||
if errors.Is(err, contracts.ErrInvalidStructuredOutput) && config.AllowStructuralRetry {
|
||||
if number < attemptLimit {
|
||||
kind, correction = producerAttemptStructuralRetry, nil
|
||||
continue
|
||||
}
|
||||
return applyStructuralTerminalPolicy(config.Policy, provenance, number, err)
|
||||
}
|
||||
if number < attemptLimit {
|
||||
kind, correction = producerAttemptOperationalRetry, nil
|
||||
continue
|
||||
}
|
||||
return failedProducerAttempt(provenance), fmt.Errorf("producer failed after %d attempt(s): %w", number, err)
|
||||
}
|
||||
|
||||
output, err = output.clone()
|
||||
if err != nil {
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed})
|
||||
return failedProducerAttempt(provenance), err
|
||||
}
|
||||
if output.Retry != nil && number < attemptLimit {
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptRetried})
|
||||
kind, correction = producerAttemptModuleRetry, nil
|
||||
continue
|
||||
}
|
||||
if output.Retry != nil {
|
||||
output.Warnings = append(output.Warnings, cloneWarnings(output.Retry.FallbackWarnings)...)
|
||||
}
|
||||
correctionCandidate, err := contracts.CloneModelCandidate(output.Candidate)
|
||||
if err != nil {
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed})
|
||||
return failedProducerAttempt(provenance), fmt.Errorf("clone correction candidate: %w", err)
|
||||
}
|
||||
|
||||
report, err := validate(ctx, output)
|
||||
if err != nil {
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed})
|
||||
return failedProducerAttempt(provenance), err
|
||||
}
|
||||
report = cloneValidationReport(report)
|
||||
if err := ctx.Err(); err != nil {
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed, Validation: report})
|
||||
return failedProducerAttempt(provenance), err
|
||||
}
|
||||
|
||||
if rejection := report.FirstRejection(); rejection != nil {
|
||||
if correctionCandidate != nil {
|
||||
if err := correctionCandidate.Validate(); err != nil {
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed, Validation: report})
|
||||
return failedProducerAttempt(provenance), fmt.Errorf("validate producer model candidate: %w", err)
|
||||
}
|
||||
}
|
||||
if number < attemptLimit && correctionCandidate != nil && correctionCandidate.Protocol == contracts.CorrectionProtocolSingleResponseV1 {
|
||||
correctionRequest, guidanceErr := report.CorrectionRequest()
|
||||
if guidanceErr != nil {
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed, Validation: report})
|
||||
return failedProducerAttempt(provenance), fmt.Errorf("construct semantic correction request: %w", guidanceErr)
|
||||
}
|
||||
correction, err = contracts.NewSemanticCorrection(correctionCandidate.Response, correctionRequest)
|
||||
if err != nil {
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed, Validation: report})
|
||||
return failedProducerAttempt(provenance), fmt.Errorf("construct semantic correction: %w", err)
|
||||
}
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptRetried, Validation: report})
|
||||
kind = producerAttemptSemanticRetry
|
||||
continue
|
||||
}
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptRejected, Validation: report})
|
||||
return applySemanticTerminalPolicy(config.Policy, provenance, number, output, report, *rejection)
|
||||
}
|
||||
|
||||
if incomplete := firstIncompleteValidation(report); incomplete != nil {
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptIncompleteAccepted, Validation: report})
|
||||
if config.Policy.ValidatorFailure == ValidatorFailureWarnContinue {
|
||||
warnings := terminalWarnings(output, report)
|
||||
warnings = append(warnings, incompleteValidationWarnings(report)...)
|
||||
return producerAttemptTerminal{Action: producerTerminalIncompleteAccepted, Value: output.Value, Warnings: warnings, Validation: report, ValidationIncomplete: true, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
|
||||
}
|
||||
return failedProducerAttempt(provenance), validatorFailureError(*incomplete)
|
||||
}
|
||||
|
||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptAccepted, Validation: report})
|
||||
return producerAttemptTerminal{Action: producerTerminalAccepted, Value: output.Value, Warnings: terminalWarnings(output, report), Validation: report, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
|
||||
}
|
||||
|
||||
return failedProducerAttempt(provenance), errors.New("producer attempt budget was not exhausted deterministically")
|
||||
}
|
||||
|
||||
func isImmediateProducerFailure(err error) bool {
|
||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
return true
|
||||
}
|
||||
var debugErr *attemptDebugPersistenceError
|
||||
return errors.As(err, &debugErr)
|
||||
}
|
||||
|
||||
func applyStructuralTerminalPolicy(policy ValidationPolicy, provenance []producerAttemptProvenance, number int, err error) (producerAttemptTerminal, error) {
|
||||
switch policy.ProducerStructuralFailure {
|
||||
case ProducerStructuralFailureRejectOutput:
|
||||
return producerAttemptTerminal{Action: producerTerminalRejected, Rejection: &contracts.RejectedOutput{ReasonCode: "invalid_structured_output", Message: "producer returned invalid structured output", AttemptCount: number}, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
|
||||
case ProducerStructuralFailureFailRun:
|
||||
return failedProducerAttempt(provenance), fmt.Errorf("producer returned invalid structured output after %d attempt(s): %w", number, err)
|
||||
default:
|
||||
return failedProducerAttempt(provenance), fmt.Errorf("unknown producer structural-failure action %q", policy.ProducerStructuralFailure)
|
||||
}
|
||||
}
|
||||
|
||||
func applySemanticTerminalPolicy(policy ValidationPolicy, provenance []producerAttemptProvenance, number int, output producerAttemptOutput, report validationReport, rejection validationRecord) (producerAttemptTerminal, error) {
|
||||
rejected := contracts.RejectedOutput{ValidatorName: rejection.validatorName, ReasonCode: rejection.reasonCode, Message: rejection.message, AttemptCount: number, DiagnosticArtifactPath: rejection.diagnosticPath}
|
||||
switch policy.SemanticRejection {
|
||||
case SemanticRejectionRejectOutput:
|
||||
return producerAttemptTerminal{Action: producerTerminalRejected, Warnings: terminalWarnings(output, report), Rejection: &rejected, Validation: report, ValidationIncomplete: firstIncompleteValidation(report) != nil, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
|
||||
case SemanticRejectionFailRun:
|
||||
return failedProducerAttempt(provenance), fmt.Errorf("producer candidate rejected after %d attempt(s): %s", number, rejection.message)
|
||||
default:
|
||||
return failedProducerAttempt(provenance), fmt.Errorf("unknown semantic-rejection action %q", policy.SemanticRejection)
|
||||
}
|
||||
}
|
||||
|
||||
func firstIncompleteValidation(report validationReport) *validationRecord {
|
||||
for _, record := range report.records {
|
||||
if record.outcome == validationFailed || record.outcome == validationSkipped {
|
||||
clone := record.clone()
|
||||
return &clone
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func terminalWarnings(output producerAttemptOutput, report validationReport) []contracts.Warning {
|
||||
warnings := cloneWarnings(output.Warnings)
|
||||
warnings = append(warnings, report.Warnings()...)
|
||||
return warnings
|
||||
}
|
||||
|
||||
// incompleteValidationWarnings reports only validators that exhausted their
|
||||
// execution budget. It never reports rejected candidates, and it uses fixed
|
||||
// text so provider errors and correction content cannot cross this boundary.
|
||||
func incompleteValidationWarnings(report validationReport) []contracts.Warning {
|
||||
warnings := make([]contracts.Warning, 0)
|
||||
for _, record := range report.records {
|
||||
if record.outcome != validationFailed {
|
||||
continue
|
||||
}
|
||||
warnings = append(warnings, contracts.Warning{
|
||||
Scope: record.validatorName,
|
||||
ReasonCode: "validator_execution_incomplete",
|
||||
Message: "Validator execution did not complete within its configured budget.",
|
||||
})
|
||||
}
|
||||
return warnings
|
||||
}
|
||||
|
||||
// validationSummary projects a terminal state-machine result into the durable
|
||||
// bounded representation used by manifests, rejections, and receipts.
|
||||
func validationSummary(terminal producerAttemptTerminal, stage ModuleStage, stepID, laneID, moduleKey, chunkID string, chunkIndex int) artifacts.ValidationSummary {
|
||||
summary := artifacts.ValidationSummary{
|
||||
Stage: string(stage),
|
||||
StepID: stepID,
|
||||
LaneID: laneID,
|
||||
ModuleKey: moduleKey,
|
||||
ChunkID: chunkID,
|
||||
ChunkIndex: chunkIndex,
|
||||
ProducerAttemptCount: len(terminal.Provenance),
|
||||
TerminalAction: string(terminal.Action),
|
||||
}
|
||||
switch {
|
||||
case terminal.Action == producerTerminalRejected:
|
||||
summary.Status = "rejected"
|
||||
case terminal.Action == producerTerminalFailed:
|
||||
summary.Status = "incomplete"
|
||||
case terminal.ValidationIncomplete:
|
||||
summary.Status = "incomplete"
|
||||
default:
|
||||
summary.Status = "complete"
|
||||
}
|
||||
seenValidators := make(map[string]struct{})
|
||||
seenReasons := make(map[string]struct{})
|
||||
seenIncomplete := make(map[string]struct{})
|
||||
for _, record := range terminal.Validation.records {
|
||||
if record.reasonCode != "" {
|
||||
if _, exists := seenReasons[record.reasonCode]; !exists {
|
||||
seenReasons[record.reasonCode] = struct{}{}
|
||||
summary.ReasonCodes = append(summary.ReasonCodes, record.reasonCode)
|
||||
}
|
||||
}
|
||||
if record.outcome == validationRejected {
|
||||
if _, exists := seenValidators[record.validatorName]; !exists {
|
||||
seenValidators[record.validatorName] = struct{}{}
|
||||
summary.RejectingValidators = append(summary.RejectingValidators, record.validatorName)
|
||||
}
|
||||
}
|
||||
if record.outcome == validationFailed || record.outcome == validationSkipped {
|
||||
if _, exists := seenIncomplete[record.validatorName]; !exists {
|
||||
seenIncomplete[record.validatorName] = struct{}{}
|
||||
summary.IncompleteValidators = append(summary.IncompleteValidators, record.validatorName)
|
||||
}
|
||||
}
|
||||
}
|
||||
if terminal.Rejection != nil && terminal.Rejection.ReasonCode != "" {
|
||||
if _, exists := seenReasons[terminal.Rejection.ReasonCode]; !exists {
|
||||
summary.ReasonCodes = append(summary.ReasonCodes, terminal.Rejection.ReasonCode)
|
||||
}
|
||||
}
|
||||
return summary
|
||||
}
|
||||
|
||||
func failedProducerAttempt(provenance []producerAttemptProvenance) producerAttemptTerminal {
|
||||
return producerAttemptTerminal{Action: producerTerminalFailed, Provenance: cloneProducerAttemptProvenance(provenance)}
|
||||
}
|
||||
414
internal/framework/pipeline/producer_attempts_test.go
Normal file
414
internal/framework/pipeline/producer_attempts_test.go
Normal file
@@ -0,0 +1,414 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestRunProducerAttemptsUsesOneSharedBudget(t *testing.T) {
|
||||
candidate := attemptCandidate(t, "first response")
|
||||
producerCalls := 0
|
||||
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 2, Policy: DefaultValidationPolicy(), AllowStructuralRetry: true}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
producerCalls++
|
||||
switch request.Number {
|
||||
case 1:
|
||||
return producerAttemptOutput{}, errors.New("temporary producer failure")
|
||||
case 2:
|
||||
return producerAttemptOutput{}, contracts.ErrInvalidStructuredOutput
|
||||
case 3:
|
||||
return producerAttemptOutput{Value: "accepted", Candidate: candidate}, nil
|
||||
default:
|
||||
t.Fatalf("unexpected producer attempt %d", request.Number)
|
||||
return producerAttemptOutput{}, nil
|
||||
}
|
||||
}, approveAttempt)
|
||||
if err != nil {
|
||||
t.Fatalf("runProducerAttempts() error = %v", err)
|
||||
}
|
||||
if terminal.Action != producerTerminalAccepted || terminal.Value != "accepted" {
|
||||
t.Fatalf("terminal = %#v, want accepted value", terminal)
|
||||
}
|
||||
if producerCalls != 3 {
|
||||
t.Fatalf("producer calls = %d, want 3", producerCalls)
|
||||
}
|
||||
if got := attemptKinds(terminal.Provenance); !reflect.DeepEqual(got, []producerAttemptKind{producerAttemptInitial, producerAttemptOperationalRetry, producerAttemptStructuralRetry}) {
|
||||
t.Fatalf("attempt kinds = %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunProducerAttemptsReplacesSemanticCorrectionWithLatestResponse(t *testing.T) {
|
||||
first := attemptCandidate(t, "first response")
|
||||
second := attemptCandidate(t, "second response")
|
||||
var corrections []*contracts.SemanticCorrection
|
||||
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 2, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
if request.Correction != nil {
|
||||
corrections = append(corrections, request.Correction)
|
||||
}
|
||||
switch request.Number {
|
||||
case 1:
|
||||
return producerAttemptOutput{Value: "one", Candidate: first}, nil
|
||||
case 2:
|
||||
return producerAttemptOutput{Value: "two", Candidate: second}, nil
|
||||
case 3:
|
||||
return producerAttemptOutput{Value: "three", Candidate: attemptCandidate(t, "third response")}, nil
|
||||
default:
|
||||
t.Fatalf("unexpected producer attempt %d", request.Number)
|
||||
return producerAttemptOutput{}, nil
|
||||
}
|
||||
}, func(_ context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||
if output.Value == "three" {
|
||||
return validationReport{}, nil
|
||||
}
|
||||
return rejectedAttemptReport("semantic_defect"), nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("runProducerAttempts() error = %v", err)
|
||||
}
|
||||
if terminal.Action != producerTerminalAccepted {
|
||||
t.Fatalf("terminal action = %q, want accepted", terminal.Action)
|
||||
}
|
||||
if len(corrections) != 2 {
|
||||
t.Fatalf("correction count = %d, want 2", len(corrections))
|
||||
}
|
||||
if corrections[0] == corrections[1] {
|
||||
t.Fatal("semantic corrections reused the same pointer")
|
||||
}
|
||||
if got := string(corrections[0].AssistantResponse); got != "first response" {
|
||||
t.Fatalf("first correction response = %q", got)
|
||||
}
|
||||
if got := string(corrections[1].AssistantResponse); got != "second response" {
|
||||
t.Fatalf("second correction response = %q", got)
|
||||
}
|
||||
if got := attemptKinds(terminal.Provenance); !reflect.DeepEqual(got, []producerAttemptKind{producerAttemptInitial, producerAttemptSemanticRetry, producerAttemptSemanticRetry}) {
|
||||
t.Fatalf("attempt kinds = %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunProducerAttemptsAppliesTerminalPolicies(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
config producerAttemptConfig
|
||||
produce producerAttemptProducer
|
||||
validate producerAttemptValidator
|
||||
wantAction producerTerminalAction
|
||||
wantError bool
|
||||
wantValue any
|
||||
}{
|
||||
{
|
||||
name: "structural reject output",
|
||||
config: producerAttemptConfig{Policy: ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureRejectOutput, SemanticRejection: SemanticRejectionFailRun, ValidatorFailure: ValidatorFailureWarnContinue}, AllowStructuralRetry: true},
|
||||
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
return producerAttemptOutput{}, contracts.ErrInvalidStructuredOutput
|
||||
},
|
||||
validate: approveAttempt,
|
||||
wantAction: producerTerminalRejected,
|
||||
},
|
||||
{
|
||||
name: "structural fail run",
|
||||
config: producerAttemptConfig{Policy: DefaultValidationPolicy(), AllowStructuralRetry: true},
|
||||
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
return producerAttemptOutput{}, contracts.ErrInvalidStructuredOutput
|
||||
},
|
||||
validate: approveAttempt,
|
||||
wantAction: producerTerminalFailed,
|
||||
wantError: true,
|
||||
},
|
||||
{
|
||||
name: "operational failure after retry exhaustion",
|
||||
config: producerAttemptConfig{Policy: DefaultValidationPolicy()},
|
||||
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
return producerAttemptOutput{}, errors.New("producer unavailable")
|
||||
},
|
||||
validate: approveAttempt,
|
||||
wantAction: producerTerminalFailed,
|
||||
wantError: true,
|
||||
},
|
||||
{
|
||||
name: "semantic reject output",
|
||||
config: producerAttemptConfig{Policy: ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureFailRun, SemanticRejection: SemanticRejectionRejectOutput, ValidatorFailure: ValidatorFailureWarnContinue}},
|
||||
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
return producerAttemptOutput{Value: "discard", Candidate: attemptCandidate(t, "response")}, nil
|
||||
},
|
||||
validate: func(context.Context, producerAttemptOutput) (validationReport, error) {
|
||||
return rejectedAttemptReport("bad"), nil
|
||||
},
|
||||
wantAction: producerTerminalRejected,
|
||||
},
|
||||
{
|
||||
name: "semantic fail run",
|
||||
config: producerAttemptConfig{Policy: DefaultValidationPolicy()},
|
||||
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
return producerAttemptOutput{Candidate: attemptCandidate(t, "response")}, nil
|
||||
},
|
||||
validate: func(context.Context, producerAttemptOutput) (validationReport, error) {
|
||||
return rejectedAttemptReport("bad"), nil
|
||||
},
|
||||
wantAction: producerTerminalFailed,
|
||||
wantError: true,
|
||||
},
|
||||
{
|
||||
name: "validator failure warns and continues",
|
||||
config: producerAttemptConfig{Policy: DefaultValidationPolicy()},
|
||||
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
return producerAttemptOutput{Value: "kept"}, nil
|
||||
},
|
||||
validate: func(context.Context, producerAttemptOutput) (validationReport, error) {
|
||||
return failedAttemptReport(), nil
|
||||
},
|
||||
wantAction: producerTerminalIncompleteAccepted,
|
||||
wantValue: "kept",
|
||||
},
|
||||
{
|
||||
name: "validator failure fails run",
|
||||
config: producerAttemptConfig{Policy: ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureFailRun, SemanticRejection: SemanticRejectionFailRun, ValidatorFailure: ValidatorFailureFailRun}},
|
||||
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
return producerAttemptOutput{}, nil
|
||||
},
|
||||
validate: func(context.Context, producerAttemptOutput) (validationReport, error) {
|
||||
return failedAttemptReport(), nil
|
||||
},
|
||||
wantAction: producerTerminalFailed,
|
||||
wantError: true,
|
||||
},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
terminal, err := runProducerAttempts(context.Background(), test.config, test.produce, test.validate)
|
||||
if (err != nil) != test.wantError {
|
||||
t.Fatalf("error = %v, want error %t", err, test.wantError)
|
||||
}
|
||||
if terminal.Action != test.wantAction {
|
||||
t.Fatalf("terminal action = %q, want %q", terminal.Action, test.wantAction)
|
||||
}
|
||||
if terminal.Value != test.wantValue {
|
||||
t.Fatalf("terminal value = %#v, want %#v", terminal.Value, test.wantValue)
|
||||
}
|
||||
if test.wantAction == producerTerminalRejected && (terminal.Rejection == nil || terminal.Value != nil) {
|
||||
t.Fatalf("rejected terminal = %#v, want rejection without value", terminal)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunProducerAttemptsDoesNotRetryUncorrectableRejection(t *testing.T) {
|
||||
calls := 0
|
||||
policy := DefaultValidationPolicy()
|
||||
policy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 3, Policy: policy}, func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
calls++
|
||||
return producerAttemptOutput{Value: "deterministic"}, nil
|
||||
}, func(context.Context, producerAttemptOutput) (validationReport, error) {
|
||||
return rejectedAttemptReport("deterministic_rejection"), nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("runProducerAttempts() error = %v", err)
|
||||
}
|
||||
if calls != 1 || terminal.Action != producerTerminalRejected || len(terminal.Provenance) != 1 {
|
||||
t.Fatalf("terminal = %#v, calls = %d; want immediate rejection", terminal, calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunProducerAttemptsUsesModuleRetryBudgetAndFallback(t *testing.T) {
|
||||
t.Run("retry", func(t *testing.T) {
|
||||
calls := 0
|
||||
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
calls++
|
||||
if request.Number == 1 {
|
||||
return producerAttemptOutput{Value: "fallback", Retry: &producerRetryDirective{}}, nil
|
||||
}
|
||||
return producerAttemptOutput{Value: "replacement"}, nil
|
||||
}, approveAttempt)
|
||||
if err != nil {
|
||||
t.Fatalf("runProducerAttempts() error = %v", err)
|
||||
}
|
||||
if terminal.Action != producerTerminalAccepted || terminal.Value != "replacement" || calls != 2 {
|
||||
t.Fatalf("terminal = %#v, calls = %d", terminal, calls)
|
||||
}
|
||||
if got := attemptKinds(terminal.Provenance); !reflect.DeepEqual(got, []producerAttemptKind{producerAttemptInitial, producerAttemptModuleRetry}) {
|
||||
t.Fatalf("attempt kinds = %v", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("fallback", func(t *testing.T) {
|
||||
fallbackWarning := contracts.Warning{ReasonCode: "fallback", Message: "fallback warning"}
|
||||
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Policy: DefaultValidationPolicy()}, func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
return producerAttemptOutput{Value: "fallback", Retry: &producerRetryDirective{FallbackWarnings: []contracts.Warning{fallbackWarning}}}, nil
|
||||
}, approveAttempt)
|
||||
if err != nil {
|
||||
t.Fatalf("runProducerAttempts() error = %v", err)
|
||||
}
|
||||
if terminal.Action != producerTerminalAccepted || terminal.Value != "fallback" || !reflect.DeepEqual(terminal.Warnings, []contracts.Warning{fallbackWarning}) {
|
||||
t.Fatalf("terminal = %#v", terminal)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestRunProducerAttemptsRejectionWinsOverValidatorFailure(t *testing.T) {
|
||||
firstWarnings := []contracts.Warning{{ReasonCode: "discarded", Message: "discarded warning"}}
|
||||
secondWarnings := []contracts.Warning{{ReasonCode: "accepted", Message: "accepted warning"}}
|
||||
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
if request.Number == 1 {
|
||||
return producerAttemptOutput{Value: "first", Candidate: attemptCandidate(t, "defective"), Warnings: firstWarnings}, nil
|
||||
}
|
||||
return producerAttemptOutput{Value: "second", Candidate: attemptCandidate(t, "corrected"), Warnings: secondWarnings}, nil
|
||||
}, func(_ context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||
if output.Value == "first" {
|
||||
report := rejectedAttemptReport("defect")
|
||||
report.records = append(report.records, validationRecord{validatorName: "unavailable", outcome: validationFailed, failure: errors.New("validator unavailable")})
|
||||
return report, nil
|
||||
}
|
||||
return validationReport{}, nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("runProducerAttempts() error = %v", err)
|
||||
}
|
||||
if terminal.Action != producerTerminalAccepted {
|
||||
t.Fatalf("terminal action = %q, want accepted", terminal.Action)
|
||||
}
|
||||
if got := attemptKinds(terminal.Provenance); !reflect.DeepEqual(got, []producerAttemptKind{producerAttemptInitial, producerAttemptSemanticRetry}) {
|
||||
t.Fatalf("attempt kinds = %v", got)
|
||||
}
|
||||
if got := terminal.Warnings; !reflect.DeepEqual(got, secondWarnings) {
|
||||
t.Fatalf("terminal warnings = %#v, want %#v", got, secondWarnings)
|
||||
}
|
||||
if len(terminal.Provenance[0].Validation.records) != 2 {
|
||||
t.Fatalf("first validation records = %#v, want rejection and failure", terminal.Provenance[0].Validation.records)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidationSummaryIsBoundedAndCorrectedSuccessIsQuiet(t *testing.T) {
|
||||
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
if request.Number == 1 {
|
||||
return producerAttemptOutput{Value: "first", Candidate: attemptCandidate(t, "sensitive defective response")}, nil
|
||||
}
|
||||
return producerAttemptOutput{Value: "corrected", Candidate: attemptCandidate(t, "sensitive corrected response")}, nil
|
||||
}, func(_ context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||
if output.Value == "first" {
|
||||
return validationReport{records: []validationRecord{{validatorName: "first", outcome: validationRejected, reasonCode: "needs_fix", message: "sensitive diagnostic", correctionGuidance: "sensitive guidance"}}}, nil
|
||||
}
|
||||
return validationReport{}, nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("runProducerAttempts() error = %v", err)
|
||||
}
|
||||
summary := validationSummary(terminal, StageExtract, "step", "lane", "module", "chunk", 3)
|
||||
if summary.Status != "complete" || summary.ProducerAttemptCount != 2 || summary.TerminalAction != string(producerTerminalAccepted) {
|
||||
t.Fatalf("summary = %#v", summary)
|
||||
}
|
||||
if len(summary.ReasonCodes) != 0 || len(summary.RejectingValidators) != 0 || len(summary.IncompleteValidators) != 0 {
|
||||
t.Fatalf("corrected success summary retained prior findings: %#v", summary)
|
||||
}
|
||||
if len(terminal.Warnings) != 0 {
|
||||
t.Fatalf("corrected success warnings = %#v, want none", terminal.Warnings)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWarnContinueRecordsOneWarningForEachExhaustedValidator(t *testing.T) {
|
||||
report := validationReport{records: []validationRecord{
|
||||
{validatorName: "first", outcome: validationFailed, attemptCount: 2, failure: errors.New("provider error with sensitive details")},
|
||||
{validatorName: "second", outcome: validationSkipped, attemptCount: 1, reasonCode: "missing_prerequisite", message: "sensitive skipped detail"},
|
||||
{validatorName: "third", outcome: validationFailed, attemptCount: 1, failure: errors.New("other provider error")},
|
||||
}}
|
||||
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Policy: DefaultValidationPolicy()}, func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
return producerAttemptOutput{Value: "candidate"}, nil
|
||||
}, func(context.Context, producerAttemptOutput) (validationReport, error) {
|
||||
return report, nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("runProducerAttempts() error = %v", err)
|
||||
}
|
||||
if terminal.Action != producerTerminalIncompleteAccepted {
|
||||
t.Fatalf("terminal action = %q", terminal.Action)
|
||||
}
|
||||
if got, want := terminal.Warnings, []contracts.Warning{
|
||||
{Scope: "first", ReasonCode: "validator_execution_incomplete", Message: "Validator execution did not complete within its configured budget."},
|
||||
{Scope: "third", ReasonCode: "validator_execution_incomplete", Message: "Validator execution did not complete within its configured budget."},
|
||||
}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("warnings = %#v, want %#v", got, want)
|
||||
}
|
||||
summary := validationSummary(terminal, StageNormalize, "step", "lane", "module", "", 0)
|
||||
if summary.Status != "incomplete" || !reflect.DeepEqual(summary.IncompleteValidators, []string{"first", "second", "third"}) || !reflect.DeepEqual(summary.ReasonCodes, []string{"missing_prerequisite"}) {
|
||||
t.Fatalf("summary = %#v", summary)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunProducerAttemptsStopsForCancellationAndDebugFailure(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
context context.Context
|
||||
produce producerAttemptProducer
|
||||
}{
|
||||
{
|
||||
name: "cancelled context",
|
||||
context: cancelledAttemptContext(),
|
||||
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
return producerAttemptOutput{}, nil
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "debug persistence failure",
|
||||
context: context.Background(),
|
||||
produce: func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
return producerAttemptOutput{}, &attemptDebugPersistenceError{label: "attempt", err: errors.New("write debug")}
|
||||
},
|
||||
},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
calls := 0
|
||||
producer := func(ctx context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
calls++
|
||||
return test.produce(ctx, request)
|
||||
}
|
||||
terminal, err := runProducerAttempts(test.context, producerAttemptConfig{Retries: 3, Policy: DefaultValidationPolicy()}, producer, approveAttempt)
|
||||
if err == nil || terminal.Action != producerTerminalFailed {
|
||||
t.Fatalf("terminal = %#v, error = %v", terminal, err)
|
||||
}
|
||||
if test.name == "cancelled context" && calls != 0 {
|
||||
t.Fatalf("cancelled producer calls = %d, want 0", calls)
|
||||
}
|
||||
if test.name == "debug persistence failure" && calls != 1 {
|
||||
t.Fatalf("debug failure producer calls = %d, want 1", calls)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func attemptCandidate(t *testing.T, response string) *contracts.ModelCandidate {
|
||||
t.Helper()
|
||||
candidate, err := contracts.NewModelCandidate([]byte(response), contracts.CorrectionProtocolSingleResponseV1)
|
||||
if err != nil {
|
||||
t.Fatalf("NewModelCandidate() error = %v", err)
|
||||
}
|
||||
return candidate
|
||||
}
|
||||
|
||||
func approveAttempt(context.Context, producerAttemptOutput) (validationReport, error) {
|
||||
return validationReport{}, nil
|
||||
}
|
||||
|
||||
func rejectedAttemptReport(reason string) validationReport {
|
||||
return validationReport{records: []validationRecord{{validatorName: "validator", outcome: validationRejected, reasonCode: reason, message: "candidate rejected", correctionGuidance: "fix the defect"}}}
|
||||
}
|
||||
|
||||
func failedAttemptReport() validationReport {
|
||||
return validationReport{records: []validationRecord{{validatorName: "validator", outcome: validationFailed, attemptCount: 1, failure: errors.New("validator failure")}}}
|
||||
}
|
||||
|
||||
func attemptKinds(provenance []producerAttemptProvenance) []producerAttemptKind {
|
||||
kinds := make([]producerAttemptKind, len(provenance))
|
||||
for index, item := range provenance {
|
||||
kinds[index] = item.Kind
|
||||
}
|
||||
return kinds
|
||||
}
|
||||
|
||||
func cancelledAttemptContext() context.Context {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
return ctx
|
||||
}
|
||||
@@ -24,6 +24,7 @@ type ModuleBinding struct {
|
||||
Module string `json:"module"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
|
||||
ValidationPolicy *ValidationPolicyOverride `json:"validation_policy,omitempty"`
|
||||
Retries int `json:"retries,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
References map[string]ReferenceSource `json:"references,omitempty"`
|
||||
@@ -84,6 +85,7 @@ func (binding ModuleBinding) MarshalJSON() ([]byte, error) {
|
||||
Module string `json:"module"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
|
||||
ValidationPolicy *ValidationPolicyOverride `json:"validation_policy,omitempty"`
|
||||
Retries int `json:"retries,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
References map[string]ReferenceSource `json:"references,omitempty"`
|
||||
@@ -93,6 +95,7 @@ func (binding ModuleBinding) MarshalJSON() ([]byte, error) {
|
||||
Module: binding.Module,
|
||||
LLMProfile: binding.LLMProfile,
|
||||
StructuredOutputRepairAttempts: binding.StructuredOutputRepairAttempts,
|
||||
ValidationPolicy: cloneValidationPolicyOverride(binding.ValidationPolicy),
|
||||
Retries: binding.Retries,
|
||||
Options: binding.Options,
|
||||
References: binding.References,
|
||||
@@ -112,6 +115,7 @@ func (binding *ModuleBinding) UnmarshalJSON(data []byte) error {
|
||||
Module string `json:"module"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
|
||||
ValidationPolicy *ValidationPolicyOverride `json:"validation_policy,omitempty"`
|
||||
Retries int `json:"retries,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
References map[string]ReferenceSource `json:"references,omitempty"`
|
||||
@@ -125,6 +129,7 @@ func (binding *ModuleBinding) UnmarshalJSON(data []byte) error {
|
||||
Module: decoded.Module,
|
||||
LLMProfile: decoded.LLMProfile,
|
||||
StructuredOutputRepairAttempts: decoded.StructuredOutputRepairAttempts,
|
||||
ValidationPolicy: cloneValidationPolicyOverride(decoded.ValidationPolicy),
|
||||
Retries: decoded.Retries,
|
||||
Options: decoded.Options,
|
||||
References: decoded.References,
|
||||
@@ -158,6 +163,7 @@ type PipelineProfile struct {
|
||||
ID string `json:"id"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
|
||||
ValidationPolicy *ValidationPolicyOverride `json:"validation_policy,omitempty"`
|
||||
Input ModuleBinding `json:"input"`
|
||||
Chunk ModuleBinding `json:"chunk,omitempty"`
|
||||
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
|
||||
@@ -198,23 +204,29 @@ type ResolvedReferenceTarget struct {
|
||||
}
|
||||
|
||||
type ResolvedArtifactLane struct {
|
||||
StepID string
|
||||
ID string
|
||||
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
|
||||
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
|
||||
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
|
||||
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
|
||||
Extract ModuleBinding
|
||||
ExtractExecutionClass contracts.ExecutionClass `json:"extract_execution_class"`
|
||||
Merge ModuleBinding
|
||||
MergeExecutionClass contracts.ExecutionClass `json:"merge_execution_class"`
|
||||
Normalize ModuleBinding
|
||||
NormalizeExecutionClass contracts.ExecutionClass `json:"normalize_execution_class"`
|
||||
Validators []ModuleBinding
|
||||
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
||||
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
||||
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
||||
StepID string
|
||||
ID string
|
||||
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
|
||||
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
|
||||
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
|
||||
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
|
||||
Extract ModuleBinding
|
||||
ExtractExecutionClass contracts.ExecutionClass `json:"extract_execution_class"`
|
||||
ExtractCorrectionProtocol contracts.CorrectionProtocol `json:"extract_correction_protocol,omitempty"`
|
||||
ExtractValidationPolicy ValidationPolicy `json:"extract_validation_policy"`
|
||||
Merge ModuleBinding
|
||||
MergeExecutionClass contracts.ExecutionClass `json:"merge_execution_class"`
|
||||
MergeCorrectionProtocol contracts.CorrectionProtocol `json:"merge_correction_protocol,omitempty"`
|
||||
MergeValidationPolicy ValidationPolicy `json:"merge_validation_policy"`
|
||||
Normalize ModuleBinding
|
||||
NormalizeExecutionClass contracts.ExecutionClass `json:"normalize_execution_class"`
|
||||
NormalizeCorrectionProtocol contracts.CorrectionProtocol `json:"normalize_correction_protocol,omitempty"`
|
||||
NormalizeValidationPolicy ValidationPolicy `json:"normalize_validation_policy"`
|
||||
Validators []ModuleBinding
|
||||
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
||||
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
||||
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
||||
}
|
||||
|
||||
type ResolvedPipelineStep struct {
|
||||
@@ -237,17 +249,20 @@ type ResolvedValidator struct {
|
||||
}
|
||||
|
||||
type ResolvedPipeline struct {
|
||||
ID string
|
||||
Digest string
|
||||
Input ModuleBinding
|
||||
InputExecutionClass contracts.ExecutionClass `json:"input_execution_class"`
|
||||
Chunk ModuleBinding
|
||||
ChunkExecutionClass contracts.ExecutionClass `json:"chunk_execution_class"`
|
||||
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
|
||||
Steps []ResolvedPipelineStep
|
||||
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
|
||||
Output ModuleBinding
|
||||
OutputExecutionClass contracts.ExecutionClass `json:"output_execution_class"`
|
||||
ID string
|
||||
Digest string
|
||||
ConfiguredValidationPolicy *ValidationPolicyOverride `json:"configured_validation_policy,omitempty"`
|
||||
Input ModuleBinding
|
||||
InputExecutionClass contracts.ExecutionClass `json:"input_execution_class"`
|
||||
Chunk ModuleBinding
|
||||
ChunkExecutionClass contracts.ExecutionClass `json:"chunk_execution_class"`
|
||||
ChunkCorrectionProtocol contracts.CorrectionProtocol `json:"chunk_correction_protocol,omitempty"`
|
||||
ChunkValidationPolicy ValidationPolicy `json:"chunk_validation_policy"`
|
||||
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
|
||||
Steps []ResolvedPipelineStep
|
||||
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
|
||||
Output ModuleBinding
|
||||
OutputExecutionClass contracts.ExecutionClass `json:"output_execution_class"`
|
||||
}
|
||||
|
||||
// AllArtifactLanes returns lanes in deterministic step order for read-only
|
||||
@@ -414,13 +429,15 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
resolved := ResolvedPipeline{
|
||||
ID: pipelineID,
|
||||
Input: input,
|
||||
InputExecutionClass: inputModuleSpec.ExecutionClass,
|
||||
Chunk: chunk,
|
||||
ChunkExecutionClass: chunkSpec.ExecutionClass,
|
||||
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
||||
Output: output,
|
||||
ID: pipelineID,
|
||||
ConfiguredValidationPolicy: cloneValidationPolicyOverride(profile.ValidationPolicy),
|
||||
Input: input,
|
||||
InputExecutionClass: inputModuleSpec.ExecutionClass,
|
||||
Chunk: chunk,
|
||||
ChunkExecutionClass: chunkSpec.ExecutionClass,
|
||||
ChunkCorrectionProtocol: chunkSpec.CorrectionProtocol,
|
||||
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
||||
Output: output,
|
||||
}
|
||||
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, "", nil, catalog)
|
||||
if err != nil {
|
||||
@@ -493,6 +510,9 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
||||
if err := applyEffectiveStructuredOutputRepairAttempts(&resolved, profile.StructuredOutputRepairAttempts); err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
if err := applyEffectiveValidationPolicies(&resolved, profile.ValidationPolicy); err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
if err := validateResolvedOptions(resolved, catalog, configuredLaneIDs); err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
@@ -600,6 +620,7 @@ func resolveArtifactLane(
|
||||
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
|
||||
lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
|
||||
lane.ExtractExecutionClass = extractSpec.ExecutionClass
|
||||
lane.ExtractCorrectionProtocol = extractSpec.CorrectionProtocol
|
||||
capabilities.add(extractSpec.Provides...)
|
||||
|
||||
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
|
||||
@@ -626,6 +647,7 @@ func resolveArtifactLane(
|
||||
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
|
||||
lane.MergeReferences.StepID = strings.TrimSpace(stepID)
|
||||
lane.MergeExecutionClass = mergeSpec.ExecutionClass
|
||||
lane.MergeCorrectionProtocol = mergeSpec.CorrectionProtocol
|
||||
capabilities.add(mergeSpec.Provides...)
|
||||
|
||||
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
|
||||
@@ -652,6 +674,7 @@ func resolveArtifactLane(
|
||||
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
|
||||
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
|
||||
lane.NormalizeExecutionClass = normalizeSpec.ExecutionClass
|
||||
lane.NormalizeCorrectionProtocol = normalizeSpec.CorrectionProtocol
|
||||
capabilities.add(normalizeSpec.Provides...)
|
||||
|
||||
if len(lane.Validators) > 0 {
|
||||
@@ -913,6 +936,9 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
|
||||
if err != nil {
|
||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
|
||||
}
|
||||
if validator.Retries > 0 && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator %q retries require an LLM-backed execution class", pipelineID, stage, validator.Module)
|
||||
}
|
||||
chain.Validators = append(chain.Validators, ResolvedValidator{
|
||||
Binding: cloneModuleBinding(validator),
|
||||
ExecutionClass: spec.ExecutionClass,
|
||||
@@ -1328,6 +1354,7 @@ func resolveBinding(binding ModuleBinding, defaultModule string, referenceSlotLa
|
||||
Module: module,
|
||||
LLMProfile: llmProfile,
|
||||
StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(binding.StructuredOutputRepairAttempts),
|
||||
ValidationPolicy: cloneValidationPolicyOverride(binding.ValidationPolicy),
|
||||
Retries: binding.Retries,
|
||||
Options: cloneOptions(binding.Options),
|
||||
References: references,
|
||||
@@ -1456,6 +1483,72 @@ func applyEffectiveStructuredOutputRepairAttempts(resolved *ResolvedPipeline, pi
|
||||
return nil
|
||||
}
|
||||
|
||||
func applyEffectiveValidationPolicies(resolved *ResolvedPipeline, pipelinePolicy *ValidationPolicyOverride) error {
|
||||
apply := func(stage ModuleStage, laneID string, binding ModuleBinding, executionClass contracts.ExecutionClass) (ValidationPolicy, error) {
|
||||
binding.ValidationPolicy = cloneValidationPolicyOverride(binding.ValidationPolicy)
|
||||
if binding.ValidationPolicy != nil {
|
||||
if err := binding.ValidationPolicy.Validate(); err != nil {
|
||||
return ValidationPolicy{}, fmt.Errorf("pipeline %q %s validation_policy: %w", resolved.ID, stage, err)
|
||||
}
|
||||
if executionClass == contracts.ExecutionClassDeterministic && binding.ValidationPolicy.ProducerStructuralFailure != nil {
|
||||
if laneID == "" {
|
||||
return ValidationPolicy{}, fmt.Errorf("pipeline %q %s %q assigns producer_structural_failure to deterministic module", resolved.ID, stage, binding.Module)
|
||||
}
|
||||
return ValidationPolicy{}, fmt.Errorf("pipeline %q lane %q %s %q assigns producer_structural_failure to deterministic module", resolved.ID, laneID, stage, binding.Module)
|
||||
}
|
||||
}
|
||||
return ResolveValidationPolicy(binding.ValidationPolicy, pipelinePolicy), nil
|
||||
}
|
||||
|
||||
if pipelinePolicy != nil {
|
||||
if err := pipelinePolicy.Validate(); err != nil {
|
||||
return fmt.Errorf("pipeline %q validation_policy: %w", resolved.ID, err)
|
||||
}
|
||||
}
|
||||
if resolved.Input.ValidationPolicy != nil {
|
||||
return fmt.Errorf("pipeline %q input validation_policy is not supported", resolved.ID)
|
||||
}
|
||||
if resolved.Output.ValidationPolicy != nil {
|
||||
return fmt.Errorf("pipeline %q output validation_policy is not supported", resolved.ID)
|
||||
}
|
||||
for _, chain := range resolved.ValidatorChains {
|
||||
for _, validator := range chain.Validators {
|
||||
if validator.Binding.ValidationPolicy != nil {
|
||||
if chain.LaneID == "" {
|
||||
return fmt.Errorf("pipeline %q %s validator %q validation_policy is not supported", resolved.ID, chain.Stage, validator.Binding.Module)
|
||||
}
|
||||
return fmt.Errorf("pipeline %q lane %q %s validator %q validation_policy is not supported", resolved.ID, chain.LaneID, chain.Stage, validator.Binding.Module)
|
||||
}
|
||||
}
|
||||
}
|
||||
policy, err := apply(StageChunk, "", resolved.Chunk, resolved.ChunkExecutionClass)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resolved.ChunkValidationPolicy = policy
|
||||
for stepIndex := range resolved.Steps {
|
||||
for laneIndex := range resolved.Steps[stepIndex].ArtifactLanes {
|
||||
lane := &resolved.Steps[stepIndex].ArtifactLanes[laneIndex]
|
||||
policy, err = apply(StageExtract, lane.ID, lane.Extract, lane.ExtractExecutionClass)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
lane.ExtractValidationPolicy = policy
|
||||
policy, err = apply(StageMerge, lane.ID, lane.Merge, lane.MergeExecutionClass)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
lane.MergeValidationPolicy = policy
|
||||
policy, err = apply(StageNormalize, lane.ID, lane.Normalize, lane.NormalizeExecutionClass)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
lane.NormalizeValidationPolicy = policy
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func resolveBindings(bindings []ModuleBinding, defaultModule string, referenceSlotLabel string) ([]ModuleBinding, error) {
|
||||
if len(bindings) == 0 {
|
||||
return nil, nil
|
||||
@@ -1549,27 +1642,33 @@ func selectedArtifactLanes(pipelineID string, artifacts map[string]ArtifactLaneP
|
||||
|
||||
func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
|
||||
withoutDigest := struct {
|
||||
ID string
|
||||
Input ModuleBinding
|
||||
InputExecutionClass contracts.ExecutionClass
|
||||
Chunk ModuleBinding
|
||||
ChunkExecutionClass contracts.ExecutionClass
|
||||
ChunkReferences ResolvedReferenceTarget
|
||||
Steps []ResolvedPipelineStep
|
||||
ValidatorChains []ResolvedValidatorChain
|
||||
Output ModuleBinding
|
||||
OutputExecutionClass contracts.ExecutionClass
|
||||
ID string
|
||||
ConfiguredValidationPolicy *ValidationPolicyOverride
|
||||
Input ModuleBinding
|
||||
InputExecutionClass contracts.ExecutionClass
|
||||
Chunk ModuleBinding
|
||||
ChunkExecutionClass contracts.ExecutionClass
|
||||
ChunkCorrectionProtocol contracts.CorrectionProtocol
|
||||
ChunkValidationPolicy ValidationPolicy
|
||||
ChunkReferences ResolvedReferenceTarget
|
||||
Steps []ResolvedPipelineStep
|
||||
ValidatorChains []ResolvedValidatorChain
|
||||
Output ModuleBinding
|
||||
OutputExecutionClass contracts.ExecutionClass
|
||||
}{
|
||||
ID: resolved.ID,
|
||||
Input: resolved.Input,
|
||||
InputExecutionClass: resolved.InputExecutionClass,
|
||||
Chunk: resolved.Chunk,
|
||||
ChunkExecutionClass: resolved.ChunkExecutionClass,
|
||||
ChunkReferences: resolved.ChunkReferences,
|
||||
Steps: resolved.Steps,
|
||||
ValidatorChains: resolved.ValidatorChains,
|
||||
Output: resolved.Output,
|
||||
OutputExecutionClass: resolved.OutputExecutionClass,
|
||||
ID: resolved.ID,
|
||||
ConfiguredValidationPolicy: cloneValidationPolicyOverride(resolved.ConfiguredValidationPolicy),
|
||||
ChunkValidationPolicy: resolved.ChunkValidationPolicy,
|
||||
Input: resolved.Input,
|
||||
InputExecutionClass: resolved.InputExecutionClass,
|
||||
Chunk: resolved.Chunk,
|
||||
ChunkExecutionClass: resolved.ChunkExecutionClass,
|
||||
ChunkCorrectionProtocol: resolved.ChunkCorrectionProtocol,
|
||||
ChunkReferences: resolved.ChunkReferences,
|
||||
Steps: resolved.Steps,
|
||||
ValidatorChains: resolved.ValidatorChains,
|
||||
Output: resolved.Output,
|
||||
OutputExecutionClass: resolved.OutputExecutionClass,
|
||||
}
|
||||
encoded, err := json.Marshal(withoutDigest)
|
||||
if err != nil {
|
||||
|
||||
@@ -126,6 +126,148 @@ func TestModuleCatalogExecutionClassLooksUpRegisteredMetadata(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineCarriesCorrectionProtocolsIntoPreparedMetadata(t *testing.T) {
|
||||
correction := contracts.CorrectionProtocolSingleResponseV1
|
||||
catalog := newProfileCatalogWithOverrides(t,
|
||||
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked, Provides: []string{"source"}},
|
||||
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, CorrectionProtocol: correction, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||
ModuleSpec{Key: "llm-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, CorrectionProtocol: correction, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, CorrectionProtocol: correction, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, CorrectionProtocol: correction, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||
)
|
||||
resolved, err := ResolvePipeline(llmProfilePipeline(), ResolveOptions{}, catalog)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
if resolved.ChunkCorrectionProtocol != correction {
|
||||
t.Fatalf("ChunkCorrectionProtocol = %q, want %q", resolved.ChunkCorrectionProtocol, correction)
|
||||
}
|
||||
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||
if lane.ExtractCorrectionProtocol != correction || lane.MergeCorrectionProtocol != correction || lane.NormalizeCorrectionProtocol != correction {
|
||||
t.Fatalf("lane correction protocols = %q/%q/%q, want %q", lane.ExtractCorrectionProtocol, lane.MergeCorrectionProtocol, lane.NormalizeCorrectionProtocol, correction)
|
||||
}
|
||||
withoutCapability := cloneResolvedPipeline(resolved)
|
||||
withoutCapability.ChunkCorrectionProtocol = ""
|
||||
withoutCapability.Steps[0].ArtifactLanes[0].ExtractCorrectionProtocol = ""
|
||||
withoutCapability.Steps[0].ArtifactLanes[0].MergeCorrectionProtocol = ""
|
||||
withoutCapability.Steps[0].ArtifactLanes[0].NormalizeCorrectionProtocol = ""
|
||||
withoutCapabilityDigest, err := resolvedPipelineDigest(withoutCapability)
|
||||
if err != nil {
|
||||
t.Fatalf("resolvedPipelineDigest() error = %v", err)
|
||||
}
|
||||
if withoutCapabilityDigest == resolved.Digest {
|
||||
t.Fatalf("resolved digest = %q with and without correction capability, want changed", resolved.Digest)
|
||||
}
|
||||
|
||||
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||
}
|
||||
if prepared.ChunkCorrectionProtocol != correction {
|
||||
t.Fatalf("prepared ChunkCorrectionProtocol = %q, want %q", prepared.ChunkCorrectionProtocol, correction)
|
||||
}
|
||||
preparedLane := prepared.Steps[0].ArtifactLanes[0]
|
||||
if preparedLane.ExtractCorrectionProtocol != correction || preparedLane.MergeCorrectionProtocol != correction || preparedLane.NormalizeCorrectionProtocol != correction {
|
||||
t.Fatalf("prepared lane correction protocols = %q/%q/%q, want %q", preparedLane.ExtractCorrectionProtocol, preparedLane.MergeCorrectionProtocol, preparedLane.NormalizeCorrectionProtocol, correction)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrepareRequiresCorrectionCapabilityForValidatorRetries(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
protocol contracts.CorrectionProtocol
|
||||
retries int
|
||||
validators bool
|
||||
want string
|
||||
}{
|
||||
{name: "supported", protocol: contracts.CorrectionProtocolSingleResponseV1, retries: 1, validators: true},
|
||||
{name: "unsupported", retries: 1, validators: true, want: "does not declare correction protocol"},
|
||||
{name: "no validators", retries: 1},
|
||||
{name: "no retries", validators: true},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
catalog := newProfileCatalogWithOverrides(t,
|
||||
ModuleSpec{Key: "llm-input", Stage: StageInput, ExecutionClass: contracts.ExecutionClassLLMBacked, Provides: []string{"source"}},
|
||||
ModuleSpec{Key: "llm-chunk", Stage: StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, CorrectionProtocol: test.protocol, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||
ModuleSpec{Key: "llm-extractor", Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "llm-merge", Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||
ModuleSpec{Key: "llm-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||
ModuleSpec{Key: "llm-output", Stage: StageOutput, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||
)
|
||||
if err := RegisterChunkValidator(catalog.Validators, ValidatorSpec{Key: "chunk-validator", ExecutionClass: contracts.ExecutionClassLLMBacked}, func() (contracts.ChunkValidator, error) {
|
||||
return llmProfileTestChunkValidator{key: "chunk-validator"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
profile := llmProfilePipeline()
|
||||
profile.Chunk.Retries = test.retries
|
||||
if test.validators {
|
||||
profile.Chunk.Validators = ValidatorOverride{Set: true, Validators: []ModuleBinding{{Module: "chunk-validator"}}}
|
||||
}
|
||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
_, err = Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
|
||||
if test.want != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("Prepare() error = %v, want %q", err, test.want)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type llmProfileTestChunkValidator struct{ key string }
|
||||
|
||||
func (validator llmProfileTestChunkValidator) Name() string { return validator.key }
|
||||
|
||||
func (llmProfileTestChunkValidator) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassLLMBacked
|
||||
}
|
||||
|
||||
func (llmProfileTestChunkValidator) Validate(context.Context, contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
|
||||
func TestResolvePipelineValidatorRetriesRequireLLMBackedValidator(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
class contracts.ExecutionClass
|
||||
want string
|
||||
}{
|
||||
{name: "deterministic validator", class: contracts.ExecutionClassDeterministic, want: "retries require an LLM-backed"},
|
||||
{name: "LLM validator", class: contracts.ExecutionClassLLMBacked},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
catalog := newProfileCatalog(t)
|
||||
registerProfileValidatorSpec(t, catalog, ValidatorSpec{Key: "retrying-validator", ExecutionClass: test.class})
|
||||
profile := baselineProfile()
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.Extract.Validators = ValidatorOverride{Set: true, Validators: []ModuleBinding{{Module: "retrying-validator", Retries: 1}}}
|
||||
profile.Artifacts["events"] = lane
|
||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||
if test.want != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want %q", err, test.want)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
if got := resolved.ValidatorChains[1].Validators[0].Binding.Retries; got != 1 {
|
||||
t.Fatalf("validator retries = %d, want 1", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
||||
resolved, err := ResolvePipeline(PipelineProfile{
|
||||
ID: "defaulted",
|
||||
|
||||
@@ -61,20 +61,24 @@ type RunInput struct {
|
||||
// single worker so direct framework callers retain deterministic behavior.
|
||||
ExtractWorkers int
|
||||
|
||||
pipeline ResolvedPipeline
|
||||
llmClient contracts.StructuredLLMClient
|
||||
stepID string
|
||||
references map[referenceTargetKey]contracts.ReferenceSet
|
||||
pipeline ResolvedPipeline
|
||||
llmClient contracts.StructuredLLMClient
|
||||
stepID string
|
||||
references map[referenceTargetKey]contracts.ReferenceSet
|
||||
referenceReuseEligibility map[referenceTargetKey]bool
|
||||
}
|
||||
|
||||
type RunOutput struct {
|
||||
Manifest artifacts.RunManifest `json:"manifest"`
|
||||
ChunkPlan *artifacts.ChunkPlanSummary `json:"chunk_plan,omitempty"`
|
||||
NormalizeOutputs []contracts.SerializedOutput `json:"normalize_outputs,omitempty"`
|
||||
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
|
||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||
OutputFiles []contracts.OutputFile `json:"-"`
|
||||
CheckpointEvents []CheckpointEvent `json:"checkpoint_events,omitempty"`
|
||||
Manifest artifacts.RunManifest `json:"manifest"`
|
||||
ChunkPlan *artifacts.ChunkPlanSummary `json:"chunk_plan,omitempty"`
|
||||
NormalizeOutputs []contracts.SerializedOutput `json:"normalize_outputs,omitempty"`
|
||||
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
|
||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||
OutputFiles []contracts.OutputFile `json:"-"`
|
||||
CheckpointEvents []CheckpointEvent `json:"checkpoint_events,omitempty"`
|
||||
ValidationSummaries []artifacts.ValidationSummary `json:"validation_summaries,omitempty"`
|
||||
|
||||
normalizeReuseEligibility map[generatedOutputKey]bool
|
||||
}
|
||||
|
||||
func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err error) {
|
||||
@@ -241,6 +245,9 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
||||
if chunkResult.rejection != nil {
|
||||
output.Rejected = append(output.Rejected, *chunkResult.rejection)
|
||||
}
|
||||
if chunkResult.validation != nil {
|
||||
output.ValidationSummaries = append(output.ValidationSummaries, artifacts.CloneValidationSummary(*chunkResult.validation))
|
||||
}
|
||||
output.Warnings = append(output.Warnings, chunkResult.warnings...)
|
||||
chunkDebugPayload := map[string]any{
|
||||
"cache_mode": chunkMode,
|
||||
@@ -297,6 +304,8 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
||||
|
||||
if len(output.Rejected) > 0 {
|
||||
output.Manifest.ValidationStatus = "rejected"
|
||||
} else if hasIncompleteValidation(output.ValidationSummaries) {
|
||||
output.Manifest.ValidationStatus = "incomplete"
|
||||
} else {
|
||||
output.Manifest.ValidationStatus = "approved"
|
||||
}
|
||||
@@ -401,6 +410,15 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
||||
return output, nil
|
||||
}
|
||||
|
||||
func hasIncompleteValidation(summaries []artifacts.ValidationSummary) bool {
|
||||
for _, summary := range summaries {
|
||||
if summary.Status == "incomplete" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (r *Runner) runPreparedSteps(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk, output *RunOutput) error {
|
||||
for _, step := range input.Prepared.Steps {
|
||||
stepInput := input
|
||||
@@ -410,6 +428,7 @@ func (r *Runner) runPreparedSteps(ctx context.Context, input RunInput, checkpoin
|
||||
return fmt.Errorf("prepare generated references for pipeline step %q: %w", step.ID, err)
|
||||
}
|
||||
stepInput.references = stepReferences
|
||||
stepInput.referenceReuseEligibility = buildStepReferenceReuseEligibility(step, output.normalizeReuseEligibility)
|
||||
output.Manifest.References = append(output.Manifest.References, referenceProvenance...)
|
||||
laneOutput, laneErr := r.runLanes(ctx, stepInput, step, checkpoints, loader, doc, sourceInput, sessionID, chunks)
|
||||
if err := mergeLaneOutput(output, laneOutput); err != nil {
|
||||
@@ -428,7 +447,7 @@ type retryAttemptResult struct {
|
||||
warnings []contracts.Warning
|
||||
}
|
||||
|
||||
func runWithRetry(ctx context.Context, retries int, run func(attempt int) (retryAttemptResult, error)) (retryAttemptResult, error) {
|
||||
func runSimpleRetry(ctx context.Context, retries int, run func(attempt int) (retryAttemptResult, error)) (retryAttemptResult, error) {
|
||||
attempts := retries + 1
|
||||
var last retryAttemptResult
|
||||
for attempt := 1; attempt <= attempts; attempt++ {
|
||||
@@ -470,26 +489,25 @@ func runWithRetry(ctx context.Context, retries int, run func(attempt int) (retry
|
||||
return last, nil
|
||||
}
|
||||
|
||||
func (r *Runner) validateChunks(ctx context.Context, doc *source.SourceDocument, moduleKey string, chunks []source.Chunk, sourceInput contracts.LLMInputMaterial, sessionID string, references contracts.ReferenceSet, metadata map[string]any, prepared preparedValidatorChain, attempt int, debug DebugRecorder) ([]contracts.Warning, *contracts.RejectedOutput, error) {
|
||||
func (r *Runner) validateChunkReport(ctx context.Context, doc *source.SourceDocument, moduleKey string, chunks []source.Chunk, sourceInput contracts.LLMInputMaterial, sessionID string, references contracts.ReferenceSet, metadata map[string]any, prepared preparedValidatorChain, attempt int, debug DebugRecorder) (validationReport, error) {
|
||||
content, err := json.Marshal(chunks)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("encode canonical chunks for validation: %w", err)
|
||||
return validationReport{}, fmt.Errorf("encode canonical chunks for validation: %w", err)
|
||||
}
|
||||
schema := contracts.ArtifactSchema{ID: "notarius.source.chunks", Name: "notarius_source_chunks", Version: "v1", JSONSchema: []byte(`{"$schema":"https://json-schema.org/draft/2020-12/schema","type":"array"}`)}
|
||||
var warnings []contracts.Warning
|
||||
for index, item := range prepared.validators {
|
||||
report, err := executeValidationChain(ctx, prepared, func(validatorCtx context.Context, item preparedValidator, validatorAttempt int) (validationInvocation, error) {
|
||||
binding := item.resolved.Binding
|
||||
started := time.Now().UTC()
|
||||
attemptPath := path.Join("validate", fileio.EncodePathComponent(string(StageChunk)), "", fileio.EncodePathComponent(moduleKey), fmt.Sprintf("%02d-%s-attempt-%02d", index+1, fileio.EncodePathComponent(binding.Module), attempt))
|
||||
validatorCtx, llmScope := withIsolatedDebugLLMScope(ctx, attemptPath)
|
||||
attemptPath := validatorAttemptPath(path.Join("validate", fileio.EncodePathComponent(string(StageChunk)), "", fileio.EncodePathComponent(moduleKey), fmt.Sprintf("%02d-%s-attempt-%02d", item.position, fileio.EncodePathComponent(binding.Module), attempt)), validatorAttempt)
|
||||
validatorCtx, llmScope := withIsolatedDebugLLMScope(validatorCtx, attemptPath)
|
||||
var result contracts.ValidationResult
|
||||
requestMetadata, cloneErr := cloneMetadata(metadata)
|
||||
if cloneErr != nil {
|
||||
return nil, nil, fmt.Errorf("clone chunk validation metadata: %w", cloneErr)
|
||||
return validationInvocation{}, fatalValidationError(fmt.Errorf("clone chunk validation metadata: %w", cloneErr))
|
||||
}
|
||||
requestChunks, cloneErr := cloneSourceChunks(chunks)
|
||||
if cloneErr != nil {
|
||||
return nil, nil, fmt.Errorf("clone chunks for validation: %w", cloneErr)
|
||||
return validationInvocation{}, fatalValidationError(fmt.Errorf("clone chunks for validation: %w", cloneErr))
|
||||
}
|
||||
switch item.resolved.Target {
|
||||
case ValidatorTargetChunk:
|
||||
@@ -497,7 +515,10 @@ func (r *Runner) validateChunks(ctx context.Context, doc *source.SourceDocument,
|
||||
case ValidatorTargetSerialized:
|
||||
result, err = item.serialized.Validate(validatorCtx, contracts.SerializedValidationRequest{Stage: string(StageChunk), ModuleKey: moduleKey, Source: doc, SourceID: doc.ID, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(references), LLMProfile: binding.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(binding.StructuredOutputRepairAttempts), Metadata: requestMetadata, Chunks: requestChunks, Schema: contracts.CloneArtifactSchema(schema), MediaType: "application/json", Content: append([]byte(nil), content...)})
|
||||
default:
|
||||
return nil, nil, fmt.Errorf("validator %q is incompatible with chunk validation", binding.Module)
|
||||
return validationInvocation{}, fatalValidationError(fmt.Errorf("validator %q is incompatible with chunk validation", binding.Module))
|
||||
}
|
||||
if err == nil {
|
||||
err = contracts.ValidateValidationResult(result)
|
||||
}
|
||||
debugContent := debugContentEnvelope(content, "application/json", nil, nil)
|
||||
debugContent.ContentDigest = debugContentDigest(content)
|
||||
@@ -508,27 +529,19 @@ func (r *Runner) validateChunks(ctx context.Context, doc *source.SourceDocument,
|
||||
if err != nil {
|
||||
validationErr := fmt.Errorf("validate chunks with validator %q: %w", binding.Module, err)
|
||||
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(StageChunk), ModuleKey: moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall, Error: debugCall.Error}, llmScope); debugErr != nil {
|
||||
return warnings, nil, errors.Join(validationErr, fmt.Errorf("write chunk validator attempt debug artifact: %w", debugErr))
|
||||
return validationInvocation{}, fatalValidationError(errors.Join(validationErr, fmt.Errorf("write chunk validator attempt debug artifact: %w", debugErr)))
|
||||
}
|
||||
return warnings, nil, validationErr
|
||||
return validationInvocation{}, validationErr
|
||||
}
|
||||
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(StageChunk), ModuleKey: moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall}, llmScope); debugErr != nil {
|
||||
return warnings, nil, fmt.Errorf("write chunk validator attempt debug artifact: %w", debugErr)
|
||||
return validationInvocation{}, fatalValidationError(fmt.Errorf("write chunk validator attempt debug artifact: %w", debugErr))
|
||||
}
|
||||
if !result.Approved {
|
||||
reason := result.ReasonCode
|
||||
if reason == "" {
|
||||
reason = "output_rejected"
|
||||
}
|
||||
message := result.Message
|
||||
if message == "" {
|
||||
message = "output rejected"
|
||||
}
|
||||
return warnings, &contracts.RejectedOutput{Stage: string(StageChunk), ModuleKey: moduleKey, ValidatorName: binding.Module, ReasonCode: reason, Message: message, AttemptCount: attempt, DiagnosticArtifactPath: result.DiagnosticArtifactPath}, nil
|
||||
}
|
||||
warnings = append(warnings, result.Warnings...)
|
||||
return validationInvocation{result: result}, nil
|
||||
})
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
return warnings, nil, nil
|
||||
return report, nil
|
||||
}
|
||||
|
||||
func resolvedValidatorChain(stage ModuleStage, laneID string, moduleKey string, chains []ResolvedValidatorChain) ResolvedValidatorChain {
|
||||
@@ -726,6 +739,9 @@ func populateOutputManifest(output *RunOutput) {
|
||||
}
|
||||
output.Manifest.NormalizedOutputs = normalizedOutputManifests(output.NormalizeOutputs)
|
||||
output.Manifest.RejectedOutputs = rejectedOutputManifests(output.Rejected)
|
||||
if len(output.ValidationSummaries) > 0 {
|
||||
output.Manifest.ValidationSummaries = cloneValidationSummaries(output.ValidationSummaries)
|
||||
}
|
||||
if len(output.CheckpointEvents) > 0 {
|
||||
decisions := make([]artifacts.CheckpointDecisionManifest, 0, len(output.CheckpointEvents))
|
||||
for _, event := range output.CheckpointEvents {
|
||||
@@ -775,6 +791,7 @@ func rejectedOutputManifests(rejected []contracts.RejectedOutput) []artifacts.Re
|
||||
Message: output.Message,
|
||||
AttemptCount: output.AttemptCount,
|
||||
DiagnosticArtifactPath: output.DiagnosticArtifactPath,
|
||||
Validation: cloneValidationSummaryPtr(output.Validation),
|
||||
})
|
||||
}
|
||||
return manifests
|
||||
@@ -1033,7 +1050,31 @@ func cloneRejectedOutputs(rejected []contracts.RejectedOutput) []contracts.Rejec
|
||||
if len(rejected) == 0 {
|
||||
return nil
|
||||
}
|
||||
return append([]contracts.RejectedOutput(nil), rejected...)
|
||||
cloned := make([]contracts.RejectedOutput, len(rejected))
|
||||
for index, item := range rejected {
|
||||
cloned[index] = item
|
||||
cloned[index].Validation = cloneValidationSummaryPtr(item.Validation)
|
||||
}
|
||||
return cloned
|
||||
}
|
||||
|
||||
func cloneValidationSummaryPtr(summary *artifacts.ValidationSummary) *artifacts.ValidationSummary {
|
||||
if summary == nil {
|
||||
return nil
|
||||
}
|
||||
cloned := artifacts.CloneValidationSummary(*summary)
|
||||
return &cloned
|
||||
}
|
||||
|
||||
func cloneValidationSummaries(summaries []artifacts.ValidationSummary) []artifacts.ValidationSummary {
|
||||
if len(summaries) == 0 {
|
||||
return nil
|
||||
}
|
||||
cloned := make([]artifacts.ValidationSummary, len(summaries))
|
||||
for index, summary := range summaries {
|
||||
cloned[index] = artifacts.CloneValidationSummary(summary)
|
||||
}
|
||||
return cloned
|
||||
}
|
||||
|
||||
func timePtr(t time.Time) *time.Time {
|
||||
|
||||
@@ -230,14 +230,14 @@ func TestRunnerRecordsDistinctRetryAttemptsAndPromotesAcceptedWarningsOnly(t *te
|
||||
if attempts == 1 {
|
||||
scope = "discarded"
|
||||
}
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{scope}}, Warnings: []contracts.Warning{{Scope: scope, ReasonCode: "observed", Message: scope}}}, nil
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{scope}}, Warnings: []contracts.Warning{{Scope: scope, ReasonCode: "observed", Message: scope}}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%s"]}`, scope))}, nil
|
||||
}
|
||||
validatorCalls := 0
|
||||
validator := preparedValidator{
|
||||
resolved: ResolvedValidator{Binding: Binding("retry-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
validatorCalls++
|
||||
return contracts.ValidationResult{Approved: validatorCalls > 1, ReasonCode: "retry", Message: "retry candidate"}, nil
|
||||
return contracts.ValidationResult{Approved: validatorCalls > 1, ReasonCode: "retry", Message: "retry candidate", CorrectionGuidance: "return an acceptable candidate"}, nil
|
||||
},
|
||||
}
|
||||
switch stage {
|
||||
@@ -280,11 +280,12 @@ func TestRunnerRecordsDistinctRetryAttemptsAndPromotesAcceptedWarningsOnly(t *te
|
||||
|
||||
func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
configure func(*PreparedPipeline)
|
||||
path string
|
||||
wantError string
|
||||
wantBody string
|
||||
name string
|
||||
configure func(*PreparedPipeline)
|
||||
path string
|
||||
wantError string
|
||||
wantBody string
|
||||
attemptError bool
|
||||
}{
|
||||
{
|
||||
name: "merge module error",
|
||||
@@ -293,12 +294,14 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
|
||||
return erasedTypedResult{}, errors.New("merge exploded")
|
||||
}
|
||||
},
|
||||
path: "merge/notes/attempt-01.json",
|
||||
wantError: "merge exploded",
|
||||
path: "merge/notes/attempt-01.json",
|
||||
wantError: "merge exploded",
|
||||
attemptError: true,
|
||||
},
|
||||
{
|
||||
name: "normalize validator error",
|
||||
configure: func(prepared *PreparedPipeline) {
|
||||
prepared.Steps[0].lanes[0].resolved.NormalizeValidationPolicy.ValidatorFailure = ValidatorFailureFailRun
|
||||
prepared.Steps[0].lanes[0].normalizeValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("error-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
@@ -313,10 +316,11 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
|
||||
{
|
||||
name: "merge final rejection",
|
||||
configure: func(prepared *PreparedPipeline) {
|
||||
prepared.Steps[0].lanes[0].resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
prepared.Steps[0].lanes[0].mergeValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("reject-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted"}, nil
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted", CorrectionGuidance: "return an acceptable candidate"}, nil
|
||||
},
|
||||
}}
|
||||
},
|
||||
@@ -330,8 +334,9 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
|
||||
return erasedTypedResult{Value: "wrong artifact type"}, nil
|
||||
}
|
||||
},
|
||||
path: "normalize/notes/attempt-01.json",
|
||||
wantError: "serialize normalize candidate",
|
||||
path: "normalize/notes/attempt-01.json",
|
||||
wantError: "serialize normalize candidate",
|
||||
attemptError: true,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -343,9 +348,15 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
|
||||
output, runErr := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug})
|
||||
envelope := debug.envelope(t, tc.path)
|
||||
if tc.wantError != "" {
|
||||
if runErr == nil || !strings.Contains(runErr.Error(), tc.wantError) || !strings.Contains(envelope.Error, tc.wantError) {
|
||||
if runErr == nil || !strings.Contains(runErr.Error(), tc.wantError) {
|
||||
t.Fatalf("run error = %v, envelope error = %q; want %q", runErr, envelope.Error, tc.wantError)
|
||||
}
|
||||
if tc.attemptError && !strings.Contains(envelope.Error, tc.wantError) {
|
||||
t.Fatalf("attempt envelope error = %q, want %q", envelope.Error, tc.wantError)
|
||||
}
|
||||
if !tc.attemptError && envelope.Error != "" {
|
||||
t.Fatalf("settled validator failure attempt error = %q, want empty", envelope.Error)
|
||||
}
|
||||
} else if runErr != nil {
|
||||
t.Fatalf("Run() error = %v, want nil rejection outcome", runErr)
|
||||
}
|
||||
|
||||
@@ -160,7 +160,7 @@ func setCandidateValidator(prepared *PreparedPipeline, target ModuleStage, appro
|
||||
validator := preparedValidator{
|
||||
resolved: ResolvedValidator{Binding: Binding("candidate-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: approved, ReasonCode: "candidate_rejected", Message: "candidate rejected by validator"}, nil
|
||||
return contracts.ValidationResult{Approved: approved, ReasonCode: "candidate_rejected", Message: "candidate rejected by validator", CorrectionGuidance: "return an acceptable candidate"}, nil
|
||||
},
|
||||
}
|
||||
switch target {
|
||||
@@ -187,6 +187,11 @@ func TestRunnerIsolatesTypedValidatorCandidates(t *testing.T) {
|
||||
for _, target := range []ModuleStage{StageExtract, StageMerge, StageNormalize} {
|
||||
t.Run(string(target), func(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
if target == StageMerge {
|
||||
prepared.Steps[0].lanes[0].resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
} else {
|
||||
prepared.Steps[0].lanes[0].resolved.NormalizeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
}
|
||||
prepared.Steps[0].lanes[0].extractValidators = preparedValidatorChain{}
|
||||
prepared.Steps[0].lanes[0].mergeValidators = preparedValidatorChain{}
|
||||
prepared.Steps[0].lanes[0].normalizeValidators = preparedValidatorChain{}
|
||||
@@ -306,6 +311,11 @@ func TestRunnerRejectsCandidatesBeforeFinalEncoding(t *testing.T) {
|
||||
for _, target := range []ModuleStage{StageMerge, StageNormalize} {
|
||||
t.Run(string(target), func(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
if target == StageMerge {
|
||||
prepared.Steps[0].lanes[0].resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
} else {
|
||||
prepared.Steps[0].lanes[0].resolved.NormalizeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
}
|
||||
codec := &observedNotesCodec{}
|
||||
installObservedNotesCodec(t, prepared, codec)
|
||||
configureCandidateOperation(prepared, target, codecNotes{Items: []string{"invalid"}})
|
||||
|
||||
@@ -12,15 +12,24 @@ import (
|
||||
)
|
||||
|
||||
type chunkPlanExecution struct {
|
||||
accepted bool
|
||||
chunks []source.Chunk
|
||||
plan *source.ChunkPlan
|
||||
warnings []contracts.Warning
|
||||
rejection *contracts.RejectedOutput
|
||||
lookup ChunkPlanDecision
|
||||
record *ChunkPlanRecord
|
||||
action string
|
||||
summary artifacts.ChunkPlanSummary
|
||||
accepted bool
|
||||
chunks []source.Chunk
|
||||
plan *source.ChunkPlan
|
||||
warnings []contracts.Warning
|
||||
rejection *contracts.RejectedOutput
|
||||
lookup ChunkPlanDecision
|
||||
record *ChunkPlanRecord
|
||||
action string
|
||||
summary artifacts.ChunkPlanSummary
|
||||
validation *artifacts.ValidationSummary
|
||||
}
|
||||
|
||||
type generatedChunkPlanCandidate struct {
|
||||
plan source.ChunkPlan
|
||||
chunks []source.Chunk
|
||||
record ChunkPlanRecord
|
||||
producerWarnings []contracts.Warning
|
||||
terminal *attemptTerminalRecorder
|
||||
}
|
||||
|
||||
func effectiveChunkCacheMode(mode ChunkCacheMode) ChunkCacheMode {
|
||||
@@ -58,17 +67,38 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
||||
case ChunkPlanHit:
|
||||
plan, chunks, validationErr := validateAndMaterializeChunkPlan(doc, record.Plan)
|
||||
if validationErr == nil {
|
||||
if err := result.setCandidate(record, "reused"); err != nil {
|
||||
return result, fmt.Errorf("clone reused chunk plan record: %w", err)
|
||||
report, err := r.validateChunkReport(ctx, doc, chunker.Key(), chunks, sourceInput, sessionID, input.pipeline.ChunkReferences.ReferenceSet, input.Metadata, input.Prepared.chunkValidators, 1, input.Debug)
|
||||
if err != nil {
|
||||
result.setValidation(report.Warnings(), nil, err)
|
||||
return result, err
|
||||
}
|
||||
validationWarnings, rejection, err := r.validateChunks(ctx, doc, chunker.Key(), chunks, sourceInput, sessionID, input.pipeline.ChunkReferences.ReferenceSet, input.Metadata, input.Prepared.chunkValidators, 1, input.Debug)
|
||||
result.plan = &plan
|
||||
result.chunks = chunks
|
||||
result.warnings = append(cloneWarnings(record.Warnings), validationWarnings...)
|
||||
result.rejection = rejection
|
||||
result.accepted = rejection == nil && err == nil
|
||||
result.setValidation(validationWarnings, rejection, err)
|
||||
return result, err
|
||||
rejection := report.FirstRejection()
|
||||
incomplete := firstIncompleteValidation(report)
|
||||
if rejection == nil && incomplete == nil {
|
||||
if err := result.setCandidate(record, "reused"); err != nil {
|
||||
return result, fmt.Errorf("clone reused chunk plan record: %w", err)
|
||||
}
|
||||
result.plan = &plan
|
||||
result.chunks = chunks
|
||||
result.warnings = append(cloneWarnings(record.Warnings), report.Warnings()...)
|
||||
result.accepted = true
|
||||
result.setValidation(report.Warnings(), nil, nil)
|
||||
cachedTerminal := producerAttemptTerminal{Action: producerTerminalAccepted, Validation: report}
|
||||
summary := validationSummary(cachedTerminal, StageChunk, "", "", chunker.Key(), "", 0)
|
||||
result.validation = &summary
|
||||
return result, nil
|
||||
}
|
||||
if incomplete != nil && input.pipeline.ChunkValidationPolicy.ValidatorFailure == ValidatorFailureFailRun {
|
||||
failure := validatorFailureError(*incomplete)
|
||||
result.setValidation(report.Warnings(), nil, failure)
|
||||
failedTerminal := producerAttemptTerminal{Action: producerTerminalFailed, Validation: report, ValidationIncomplete: true}
|
||||
summary := validationSummary(failedTerminal, StageChunk, "", "", chunker.Key(), "", 0)
|
||||
result.validation = &summary
|
||||
return result, failure
|
||||
}
|
||||
// A cache hit is not model material. Its rejection is discarded and
|
||||
// generation begins with the ordinary initial request below. Warnings
|
||||
// from this discarded candidate are intentionally not promoted.
|
||||
}
|
||||
result.lookup = ChunkPlanDecision{Status: ChunkPlanInvalid, Reason: chunkPlanLookupReason(ChunkPlanInvalid)}
|
||||
result.summary.LookupStatus = "invalid"
|
||||
@@ -83,37 +113,41 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
||||
result.summary.PublicationStatus = "not_published"
|
||||
}
|
||||
|
||||
var producerWarnings []contracts.Warning
|
||||
retryResult, err := runWithRetry(ctx, input.pipeline.Chunk.Retries, func(attempt int) (retryAttemptResult, error) {
|
||||
terminal, err := runProducerAttempts(ctx, producerAttemptConfig{
|
||||
Retries: input.pipeline.Chunk.Retries,
|
||||
Policy: input.pipeline.ChunkValidationPolicy,
|
||||
AllowStructuralRetry: input.pipeline.ChunkExecutionClass == contracts.ExecutionClassLLMBacked,
|
||||
}, func(attemptCtx context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
attempt := request.Number
|
||||
attemptStarted := time.Now().UTC()
|
||||
attemptPath := path.Join("chunk", fmt.Sprintf("attempt-%02d", attempt))
|
||||
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "chunk", llmScope, debugTimedEnvelope{Stage: string(StageChunk), ModuleKey: chunker.Key(), Attempt: attempt, StartedAt: attemptStarted})
|
||||
attemptCtx, llmScope := withDebugLLMScope(attemptCtx, attemptPath)
|
||||
attemptTerminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "chunk", llmScope, debugTimedEnvelope{Stage: string(StageChunk), ModuleKey: chunker.Key(), Attempt: attempt, AttemptKind: string(request.Kind), StartedAt: attemptStarted})
|
||||
requestMetadata, metadataErr := cloneMetadata(input.Metadata)
|
||||
if metadataErr != nil {
|
||||
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("clone chunk request metadata: %w", metadataErr))
|
||||
return producerAttemptOutput{}, attemptTerminal.record(nil, fmt.Errorf("clone chunk request metadata: %w", metadataErr))
|
||||
}
|
||||
chunkResult, callErr := chunker.Plan(attemptCtx, contracts.ChunkRequest{
|
||||
Source: doc, SourceInput: sourceInput.Clone(), SessionID: sessionID,
|
||||
References: CloneReferenceSet(input.pipeline.ChunkReferences.ReferenceSet),
|
||||
LLMProfile: input.pipeline.Chunk.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(input.pipeline.Chunk.StructuredOutputRepairAttempts), Metadata: requestMetadata,
|
||||
LLMProfile: input.pipeline.Chunk.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(input.pipeline.Chunk.StructuredOutputRepairAttempts), Correction: request.Correction, Metadata: requestMetadata,
|
||||
})
|
||||
if callErr != nil {
|
||||
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("chunk source with chunker %q: %w", chunker.Key(), callErr))
|
||||
return producerAttemptOutput{}, attemptTerminal.record(nil, fmt.Errorf("chunk source with chunker %q: %w", chunker.Key(), callErr))
|
||||
}
|
||||
plan, chunks, validationErr := validateAndMaterializeChunkPlan(doc, chunkResult.Plan)
|
||||
if validationErr != nil {
|
||||
attemptErr := fmt.Errorf("validate chunk plan from chunker %q: %w", chunker.Key(), validationErr)
|
||||
payload := map[string]any{"plan": debugChunkPlanEnvelope(chunkResult.Plan), "warnings": debugWarningEnvelopes(chunkResult.Warnings)}
|
||||
return retryAttemptResult{}, terminal.record(payload, attemptErr)
|
||||
return producerAttemptOutput{}, attemptTerminal.record(payload, fmt.Errorf("%w: %v", contracts.ErrInvalidStructuredOutput, attemptErr))
|
||||
}
|
||||
planDigest, digestErr := source.DigestChunkPlan(plan)
|
||||
if digestErr != nil {
|
||||
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("digest generated chunk plan: %w", digestErr))
|
||||
return producerAttemptOutput{}, attemptTerminal.record(nil, fmt.Errorf("digest generated chunk plan: %w", digestErr))
|
||||
}
|
||||
producerMetadata, _, metadataErr := moduleManifestMetadata(chunker)
|
||||
if metadataErr != nil {
|
||||
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
|
||||
return producerAttemptOutput{}, attemptTerminal.record(nil, fmt.Errorf("clone chunker manifest metadata: %w", metadataErr))
|
||||
}
|
||||
profile := ""
|
||||
if input.pipeline.ChunkExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||
@@ -129,62 +163,78 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
||||
},
|
||||
Warnings: cloneWarnings(chunkResult.Warnings), CreatedAt: time.Now().UTC(),
|
||||
}
|
||||
action := "generated"
|
||||
if mode == ChunkCacheRefresh {
|
||||
action = "refreshed"
|
||||
return producerAttemptOutput{Value: generatedChunkPlanCandidate{plan: plan, chunks: chunks, record: candidate, producerWarnings: cloneWarnings(chunkResult.Warnings), terminal: &attemptTerminal}, Candidate: chunkResult.ModelCandidate, Warnings: cloneWarnings(chunkResult.Warnings)}, nil
|
||||
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||
candidate, ok := output.Value.(generatedChunkPlanCandidate)
|
||||
if !ok {
|
||||
return validationReport{}, fmt.Errorf("chunk attempt candidate has incompatible type")
|
||||
}
|
||||
if mode == ChunkCacheBypass {
|
||||
action = "bypassed"
|
||||
}
|
||||
if candidateErr := result.setCandidate(candidate, action); candidateErr != nil {
|
||||
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("clone generated chunk plan record: %w", candidateErr))
|
||||
}
|
||||
validationWarnings, rejected, validationErr := r.validateChunks(attemptCtx, doc, chunker.Key(), chunks, sourceInput, sessionID, input.pipeline.ChunkReferences.ReferenceSet, input.Metadata, input.Prepared.chunkValidators, attempt, input.Debug)
|
||||
attemptWarnings := append(cloneWarnings(chunkResult.Warnings), validationWarnings...)
|
||||
report, validationErr := r.validateChunkReport(validationCtx, doc, chunker.Key(), candidate.chunks, sourceInput, sessionID, input.pipeline.ChunkReferences.ReferenceSet, input.Metadata, input.Prepared.chunkValidators, candidate.terminal.envelope.Attempt, input.Debug)
|
||||
attemptWarnings := append(cloneWarnings(output.Warnings), report.Warnings()...)
|
||||
payload := map[string]any{
|
||||
"plan": debugChunkPlanEnvelope(plan), "materialized_chunks": debugSourceChunkEnvelopes(chunks),
|
||||
"warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected),
|
||||
"plan": debugChunkPlanEnvelope(candidate.plan), "materialized_chunks": debugSourceChunkEnvelopes(candidate.chunks),
|
||||
"warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(chunkRejection(report, candidate.terminal.envelope.Attempt, chunker.Key())),
|
||||
}
|
||||
if validationErr != nil {
|
||||
result.setValidation(validationWarnings, rejected, validationErr)
|
||||
return retryAttemptResult{}, terminal.record(payload, validationErr)
|
||||
return report, candidate.terminal.record(payload, validationErr)
|
||||
}
|
||||
if rejected != nil {
|
||||
result.setValidation(validationWarnings, rejected, nil)
|
||||
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
||||
return retryAttemptResult{}, debugErr
|
||||
}
|
||||
return retryAttemptResult{rejection: rejected, warnings: attemptWarnings}, nil
|
||||
}
|
||||
result.chunks = chunks
|
||||
result.plan = &plan
|
||||
result.warnings = attemptWarnings
|
||||
producerWarnings = cloneWarnings(chunkResult.Warnings)
|
||||
result.setValidation(validationWarnings, nil, nil)
|
||||
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
||||
return retryAttemptResult{}, debugErr
|
||||
}
|
||||
return retryAttemptResult{accepted: true}, nil
|
||||
return report, candidate.terminal.record(payload, nil)
|
||||
})
|
||||
terminalSummary := validationSummary(terminal, StageChunk, "", "", chunker.Key(), "", 0)
|
||||
if debugErr := writeProducerTerminalDebug(input.Debug, "chunk/terminal.json", terminal, input.pipeline.ChunkValidationPolicy, terminalSummary); debugErr != nil {
|
||||
return result, debugErr
|
||||
}
|
||||
if err != nil {
|
||||
if result.summary.ValidationStatus == "not_run" {
|
||||
result.summary.ValidationStatus = "error"
|
||||
}
|
||||
return result, err
|
||||
}
|
||||
result.accepted = retryResult.accepted
|
||||
result.rejection = retryResult.rejection
|
||||
if !retryResult.accepted {
|
||||
result.warnings = cloneWarnings(retryResult.warnings)
|
||||
if terminal.Action == producerTerminalRejected {
|
||||
result.rejection = terminal.Rejection
|
||||
if result.rejection != nil {
|
||||
result.rejection.Stage = string(StageChunk)
|
||||
result.rejection.ModuleKey = chunker.Key()
|
||||
}
|
||||
result.warnings = cloneWarnings(terminal.Warnings)
|
||||
result.validation = &terminalSummary
|
||||
if result.rejection != nil {
|
||||
result.rejection.Validation = cloneValidationSummaryPtr(result.validation)
|
||||
}
|
||||
result.setValidation(terminal.Validation.Warnings(), result.rejection, nil)
|
||||
return result, nil
|
||||
}
|
||||
candidate, ok := terminal.Value.(generatedChunkPlanCandidate)
|
||||
if !ok {
|
||||
return result, fmt.Errorf("chunk attempt terminal has incompatible value")
|
||||
}
|
||||
if err := result.setCandidate(candidate.record, "generated"); err != nil {
|
||||
return result, fmt.Errorf("clone generated chunk plan record: %w", err)
|
||||
}
|
||||
if mode == ChunkCacheRefresh {
|
||||
result.action = "refreshed"
|
||||
result.summary.Action = "refreshed"
|
||||
}
|
||||
if mode == ChunkCacheBypass {
|
||||
result.action = "bypassed"
|
||||
result.summary.Action = "bypassed"
|
||||
}
|
||||
result.accepted = true
|
||||
result.plan = &candidate.plan
|
||||
result.chunks = candidate.chunks
|
||||
result.warnings = cloneWarnings(terminal.Warnings)
|
||||
result.validation = &terminalSummary
|
||||
result.setValidation(terminal.Validation.Warnings(), nil, nil)
|
||||
if terminal.ValidationIncomplete {
|
||||
result.summary.ValidationStatus = "incomplete"
|
||||
}
|
||||
|
||||
if mode == ChunkCacheAuto || mode == ChunkCacheRefresh {
|
||||
if (mode == ChunkCacheAuto || mode == ChunkCacheRefresh) && !terminal.ValidationIncomplete {
|
||||
record, cloneErr := cloneChunkPlanRecord(*result.record)
|
||||
if cloneErr != nil {
|
||||
return result, fmt.Errorf("clone chunk plan record for publication: %w", cloneErr)
|
||||
}
|
||||
record.Warnings = cloneWarnings(producerWarnings)
|
||||
record.Warnings = cloneWarnings(candidate.producerWarnings)
|
||||
if err := input.ChunkPlans.Save(record); err != nil {
|
||||
return result, fmt.Errorf("save chunk plan: %w", err)
|
||||
}
|
||||
@@ -193,6 +243,14 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func chunkRejection(report validationReport, attempt int, moduleKey string) *contracts.RejectedOutput {
|
||||
rejection := report.FirstRejection()
|
||||
if rejection == nil {
|
||||
return nil
|
||||
}
|
||||
return &contracts.RejectedOutput{Stage: string(StageChunk), ModuleKey: moduleKey, ValidatorName: rejection.validatorName, ReasonCode: rejection.reasonCode, Message: rejection.message, AttemptCount: attempt, DiagnosticArtifactPath: rejection.diagnosticPath}
|
||||
}
|
||||
|
||||
func chunkPlanLookupStatus(status ChunkPlanStatus) string {
|
||||
switch status {
|
||||
case ChunkPlanHit:
|
||||
|
||||
@@ -84,6 +84,35 @@ type retryingChunker struct {
|
||||
calls int
|
||||
}
|
||||
|
||||
type correctionAwareChunker struct {
|
||||
key string
|
||||
plan source.ChunkPlan
|
||||
calls int
|
||||
correction *contracts.SemanticCorrection
|
||||
}
|
||||
|
||||
func (c *correctionAwareChunker) Key() string { return c.key }
|
||||
func (*correctionAwareChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||
func (c *correctionAwareChunker) Plan(_ context.Context, request contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
||||
c.calls++
|
||||
if request.Correction != nil {
|
||||
correction, err := contracts.CloneSemanticCorrection(request.Correction)
|
||||
if err != nil {
|
||||
return contracts.ChunkPlanResult{}, err
|
||||
}
|
||||
c.correction = correction
|
||||
}
|
||||
response := "initial chunk response"
|
||||
if c.calls > 1 {
|
||||
response = "corrected chunk response"
|
||||
}
|
||||
candidate, err := contracts.NewModelCandidate([]byte(response), contracts.CorrectionProtocolSingleResponseV1)
|
||||
if err != nil {
|
||||
return contracts.ChunkPlanResult{}, err
|
||||
}
|
||||
return contracts.ChunkPlanResult{Plan: source.CloneChunkPlan(c.plan), ModelCandidate: candidate}, nil
|
||||
}
|
||||
|
||||
func (c *retryingChunker) Key() string { return c.key }
|
||||
func (*retryingChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||
func (c *retryingChunker) Plan(context.Context, contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
||||
@@ -264,7 +293,7 @@ func TestRunnerOmitsChunkMapForRejectedPlan(t *testing.T) {
|
||||
prepared.output = encoder
|
||||
prepared.chunkValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("reject/chunk"), Target: ValidatorTargetChunk},
|
||||
chunk: &countingChunkValidator{result: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted"}},
|
||||
chunk: &countingChunkValidator{result: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted", CorrectionGuidance: "return an acceptable chunk plan"}},
|
||||
}}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
|
||||
if err != nil {
|
||||
@@ -275,6 +304,139 @@ func TestRunnerOmitsChunkMapForRejectedPlan(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerCorrectsRejectedGeneratedChunkPlan(t *testing.T) {
|
||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
|
||||
prepared.resolved.Chunk.Retries = 1
|
||||
chunker := &correctionAwareChunker{key: prepared.resolved.Chunk.Module, plan: plan}
|
||||
prepared.chunker = chunker
|
||||
validatorCalls := 0
|
||||
prepared.chunkValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("reject-then-approve"), Target: ValidatorTargetChunk},
|
||||
chunk: chunkValidationFunc{name: "reject-then-approve", validate: func(context.Context, contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
|
||||
validatorCalls++
|
||||
if validatorCalls == 1 {
|
||||
return contracts.ValidationResult{ReasonCode: "chunk_scope", Message: "scope needs correction", CorrectionGuidance: "correct the chunk scope"}, nil
|
||||
}
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}},
|
||||
}}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(output.Rejected) != 0 || chunker.calls != 2 || validatorCalls != 2 {
|
||||
t.Fatalf("output = %#v chunker calls = %d validator calls = %d", output.Rejected, chunker.calls, validatorCalls)
|
||||
}
|
||||
if got := string(chunker.correction.AssistantResponse); got != "initial chunk response" {
|
||||
t.Fatalf("correction response = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerDoesNotPublishValidationIncompleteChunkPlan(t *testing.T) {
|
||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||
prepared.resolved.ChunkValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
|
||||
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan}
|
||||
prepared.chunkValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("unavailable"), Target: ValidatorTargetChunk},
|
||||
chunk: &countingChunkValidator{err: errors.New("validator unavailable")},
|
||||
}}
|
||||
store := &recordingChunkPlanStore{decision: ChunkPlanDecision{Status: ChunkPlanMissing}}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: store})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if output.ChunkPlan == nil || output.ChunkPlan.ValidationStatus != "incomplete" || output.ChunkPlan.PublicationStatus != "not_published" || store.saves != 0 {
|
||||
t.Fatalf("chunk summary = %#v saves = %d", output.ChunkPlan, store.saves)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerRegeneratesValidationIncompleteChunkPlanHit(t *testing.T) {
|
||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||
prepared.resolved.ChunkValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
|
||||
calls := 0
|
||||
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan, calls: &calls}
|
||||
validatorCalls := 0
|
||||
prepared.chunkValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("cache-then-generated"), Target: ValidatorTargetChunk},
|
||||
chunk: chunkValidationFunc{name: "cache-then-generated", validate: func(context.Context, contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
|
||||
validatorCalls++
|
||||
if validatorCalls == 1 {
|
||||
return contracts.ValidationResult{}, errors.New("cached candidate could not be validated")
|
||||
}
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}},
|
||||
}}
|
||||
record := chunkPlanRecord(t, prepared, plan)
|
||||
record.Warnings = []contracts.Warning{{Scope: "stored", ReasonCode: "old", Message: "discarded stored warning"}}
|
||||
store := &recordingChunkPlanStore{record: record, decision: ChunkPlanDecision{Status: ChunkPlanHit}}
|
||||
debug := newCapturedDebugRecorder()
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: store, Debug: debug})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if calls != 1 || validatorCalls != 2 || store.saves != 1 {
|
||||
t.Fatalf("calls = producer %d validator %d saves %d, want 1 2 1", calls, validatorCalls, store.saves)
|
||||
}
|
||||
if output.ChunkPlan == nil || output.ChunkPlan.Action != "generated" || output.ChunkPlan.LookupStatus != "invalid" || output.ChunkPlan.ValidationStatus != "approved" {
|
||||
t.Fatalf("chunk plan summary = %#v", output.ChunkPlan)
|
||||
}
|
||||
if len(output.Warnings) != 0 {
|
||||
t.Fatalf("warnings = %#v, want discarded cache warnings omitted", output.Warnings)
|
||||
}
|
||||
assertAttemptEnvelopeSequence(t, debug, "chunk", 1)
|
||||
}
|
||||
|
||||
func TestRunnerKeepsStoredPlanWhenCacheAndGeneratedValidationAreIncomplete(t *testing.T) {
|
||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||
prepared.resolved.ChunkValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
|
||||
calls := 0
|
||||
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan, calls: &calls}
|
||||
validator := &countingChunkValidator{err: errors.New("validator unavailable")}
|
||||
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
|
||||
record := chunkPlanRecord(t, prepared, plan)
|
||||
record.Warnings = []contracts.Warning{{Scope: "stored", ReasonCode: "old", Message: "discarded stored warning"}}
|
||||
store := &recordingChunkPlanStore{record: record, decision: ChunkPlanDecision{Status: ChunkPlanHit}}
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: store})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if calls != 1 || validator.calls != 2 || store.saves != 0 {
|
||||
t.Fatalf("calls = producer %d validator %d saves %d, want 1 2 0", calls, validator.calls, store.saves)
|
||||
}
|
||||
if output.ChunkPlan == nil || output.ChunkPlan.Action != "generated" || output.ChunkPlan.LookupStatus != "invalid" || output.ChunkPlan.ValidationStatus != "incomplete" {
|
||||
t.Fatalf("chunk plan summary = %#v", output.ChunkPlan)
|
||||
}
|
||||
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != "validator_execution_incomplete" {
|
||||
t.Fatalf("warnings = %#v, want only generated incomplete warning", output.Warnings)
|
||||
}
|
||||
if !reflect.DeepEqual(store.record, record) {
|
||||
t.Fatal("discarded incomplete candidates mutated the stored record")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerFailsOnValidationIncompleteChunkPlanHitUnderFailRun(t *testing.T) {
|
||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||
calls := 0
|
||||
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan, calls: &calls}
|
||||
validator := &countingChunkValidator{err: errors.New("validator unavailable")}
|
||||
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
|
||||
store := &recordingChunkPlanStore{record: chunkPlanRecord(t, prepared, plan), decision: ChunkPlanDecision{Status: ChunkPlanHit}}
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: store})
|
||||
if err == nil || !strings.Contains(err.Error(), "validator unavailable") {
|
||||
t.Fatalf("Run() error = %v, want cached validation failure", err)
|
||||
}
|
||||
if calls != 0 || validator.calls != 1 || store.saves != 0 {
|
||||
t.Fatalf("calls = producer %d validator %d saves %d, want 0 1 0", calls, validator.calls, store.saves)
|
||||
}
|
||||
if output.ChunkPlan == nil || output.ChunkPlan.ValidationStatus != "error" {
|
||||
t.Fatalf("chunk plan summary = %#v", output.ChunkPlan)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerRetainsChunkMapAfterLaneRejection(t *testing.T) {
|
||||
prepared, _ := preparedTerminalDebugPipeline(t)
|
||||
encoder := &capturingChunkMapOutput{}
|
||||
@@ -282,7 +444,7 @@ func TestRunnerRetainsChunkMapAfterLaneRejection(t *testing.T) {
|
||||
prepared.Steps[0].lanes[0].mergeValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("reject/lane"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted"}, nil
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted", CorrectionGuidance: "return an acceptable candidate"}, nil
|
||||
},
|
||||
}}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
|
||||
@@ -316,14 +478,14 @@ func TestRunnerChunkMapRequestDoesNotAliasStoredPlan(t *testing.T) {
|
||||
|
||||
func TestRunnerChunkPlanManifestRetainsCandidateOnRejection(t *testing.T) {
|
||||
prepared, _ := preparedTerminalDebugPipeline(t)
|
||||
validator := &countingChunkValidator{result: contracts.ValidationResult{Approved: false, ReasonCode: "policy", Message: "no"}}
|
||||
validator := &countingChunkValidator{result: contracts.ValidationResult{Approved: false, ReasonCode: "policy", Message: "no", CorrectionGuidance: "return a policy-compliant chunk plan"}}
|
||||
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: &recordingChunkPlanStore{}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if output.Manifest.ChunkPlan.Action != "refreshed" || output.Manifest.ChunkPlan.PlanDigest == "" || output.ChunkPlan.ValidationStatus != "rejected" || output.ChunkPlan.PublicationStatus != "not_published" {
|
||||
t.Fatalf("manifest = %#v summary = %#v", output.Manifest.ChunkPlan, output.ChunkPlan)
|
||||
if output.Manifest.ChunkPlan.Action != "" || output.Manifest.ChunkPlan.PlanDigest != "" || output.ChunkPlan.ValidationStatus != "rejected" || output.ChunkPlan.PublicationStatus != "not_published" {
|
||||
t.Fatalf("manifest = %#v summary = %#v; rejected output must not retain a usable chunk candidate", output.Manifest.ChunkPlan, output.ChunkPlan)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -432,7 +594,7 @@ func TestRunnerAutoHitValidatesOnceWithoutRegenerationOrMutation(t *testing.T) {
|
||||
wantReject bool
|
||||
}{
|
||||
{name: "warning", result: contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{Scope: "current", ReasonCode: "observed", Message: "current warning"}}}},
|
||||
{name: "rejection", result: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected hit"}, wantReject: true},
|
||||
{name: "rejection", result: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected hit", CorrectionGuidance: "return an acceptable chunk plan"}, wantReject: true},
|
||||
{name: "error", validatorErr: errors.New("validator failed"), wantError: "validator failed"},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
@@ -455,20 +617,32 @@ func TestRunnerAutoHitValidatesOnceWithoutRegenerationOrMutation(t *testing.T) {
|
||||
} else if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if validator.calls != 1 || calls != 0 || llmCalls != 0 || store.loads != 1 || store.saves != 0 {
|
||||
wantProducerCalls := 0
|
||||
if tc.wantReject {
|
||||
wantProducerCalls = 1
|
||||
}
|
||||
if validator.calls != 1+wantProducerCalls || calls != wantProducerCalls || llmCalls != wantProducerCalls || store.loads != 1 || store.saves != 0 {
|
||||
t.Fatalf("calls = validator %d module %d llm %d load %d save %d", validator.calls, calls, llmCalls, store.loads, store.saves)
|
||||
}
|
||||
assertAttemptEnvelopeSequence(t, debug, "chunk")
|
||||
if tc.wantReject {
|
||||
assertAttemptEnvelopeSequence(t, debug, "chunk", 1)
|
||||
} else {
|
||||
assertAttemptEnvelopeSequence(t, debug, "chunk")
|
||||
}
|
||||
if tc.wantError == "" {
|
||||
encoded := string(debug.json["chunk/output.json"])
|
||||
if !strings.Contains(encoded, `"status":"hit"`) || !strings.Contains(encoded, `"plan":`) || !strings.Contains(encoded, `"materialized_chunks":`) {
|
||||
if tc.wantReject {
|
||||
if !strings.Contains(encoded, `"status":"invalid"`) || strings.Contains(encoded, `"plan":`) {
|
||||
t.Fatalf("rejected cache debug = %s", encoded)
|
||||
}
|
||||
} else if !strings.Contains(encoded, `"status":"hit"`) || !strings.Contains(encoded, `"plan":`) || !strings.Contains(encoded, `"materialized_chunks":`) {
|
||||
t.Fatalf("chunk hit debug = %s", encoded)
|
||||
}
|
||||
}
|
||||
if tc.wantReject && (len(output.Rejected) != 1 || output.Rejected[0].ReasonCode != "rejected") {
|
||||
t.Fatalf("rejected = %#v", output.Rejected)
|
||||
}
|
||||
if tc.wantError == "" && len(output.Warnings) != 1+len(tc.result.Warnings) {
|
||||
if tc.wantError == "" && !tc.wantReject && len(output.Warnings) != 1+len(tc.result.Warnings) {
|
||||
t.Fatalf("warnings = %#v, want stored warning once plus current warnings", output.Warnings)
|
||||
}
|
||||
})
|
||||
@@ -485,7 +659,7 @@ func TestRunnerPublishesOnlyAcceptedGeneratedPlans(t *testing.T) {
|
||||
wantReject bool
|
||||
}{
|
||||
{name: "module error", moduleErr: errors.New("generation failed"), wantCalls: 2},
|
||||
{name: "validator rejection", validator: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected generated plan"}, wantCalls: 2, wantReject: true},
|
||||
{name: "validator rejection", validator: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected generated plan", CorrectionGuidance: "return an acceptable chunk plan"}, wantCalls: 1, wantReject: true},
|
||||
{name: "cancellation", cancel: true},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
|
||||
@@ -558,6 +558,7 @@ func TestRunnerSelectsMergeErrorBeforeEarlierLaneNormalizeError(t *testing.T) {
|
||||
|
||||
func TestRunnerKeepsSuccessfulChunksWhenAnotherExtractIsRejected(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 2)
|
||||
prepared.Steps[0].lanes[0].resolved.ExtractValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
for laneIndex := range prepared.Steps[0].lanes {
|
||||
lane := laneIndex
|
||||
installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
@@ -567,7 +568,7 @@ func TestRunnerKeepsSuccessfulChunksWhenAnotherExtractIsRejected(t *testing.T) {
|
||||
validator := &prepared.Steps[0].lanes[0].extractValidators.validators[0]
|
||||
validator.typedValidate = func(_ context.Context, _ any, target typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
if target.chunk != nil && target.chunk.Index == 0 {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "expected_rejection", Message: "rejected by test"}, nil
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "expected_rejection", Message: "rejected by test", CorrectionGuidance: "return an acceptable candidate"}, nil
|
||||
}
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
|
||||
@@ -10,33 +10,40 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/fileio"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type laneExtractState struct {
|
||||
index int
|
||||
prepared preparedLaneExecutor
|
||||
deps []CheckpointFingerprint
|
||||
decision CheckpointDecision
|
||||
values []erasedExtractArtifact
|
||||
serialized []CheckpointArtifact
|
||||
warnings []contracts.Warning
|
||||
rejected []contracts.RejectedOutput
|
||||
results map[int]extractJobResult
|
||||
remaining int
|
||||
failed bool
|
||||
terminal bool
|
||||
output RunOutput
|
||||
index int
|
||||
prepared preparedLaneExecutor
|
||||
deps []CheckpointFingerprint
|
||||
decision CheckpointDecision
|
||||
reuseEligible bool
|
||||
values []erasedExtractArtifact
|
||||
serialized []CheckpointArtifact
|
||||
warnings []contracts.Warning
|
||||
rejected []contracts.RejectedOutput
|
||||
incomplete []int
|
||||
validationSummaries []artifacts.ValidationSummary
|
||||
results map[int]extractJobResult
|
||||
remaining int
|
||||
failed bool
|
||||
terminal bool
|
||||
output RunOutput
|
||||
}
|
||||
|
||||
type finalizedExtractResults struct {
|
||||
accepted []erasedExtractArtifact
|
||||
serialized []CheckpointArtifact
|
||||
warnings []contracts.Warning
|
||||
rejected []contracts.RejectedOutput
|
||||
decision CheckpointDecision
|
||||
accepted []erasedExtractArtifact
|
||||
serialized []CheckpointArtifact
|
||||
warnings []contracts.Warning
|
||||
rejected []contracts.RejectedOutput
|
||||
incomplete []int
|
||||
validationSummaries []artifacts.ValidationSummary
|
||||
decision CheckpointDecision
|
||||
reuseEligible bool
|
||||
}
|
||||
|
||||
func loadExtract(loader CheckpointLoader, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||
@@ -56,13 +63,21 @@ type extractJob struct {
|
||||
}
|
||||
|
||||
type extractJobResult struct {
|
||||
laneIndex int
|
||||
chunkIndex int
|
||||
value erasedExtractArtifact
|
||||
laneIndex int
|
||||
chunkIndex int
|
||||
value erasedExtractArtifact
|
||||
serialized CheckpointArtifact
|
||||
warnings []contracts.Warning
|
||||
rejected *contracts.RejectedOutput
|
||||
validationIncomplete bool
|
||||
validationSummary *artifacts.ValidationSummary
|
||||
err error
|
||||
}
|
||||
|
||||
type extractAttemptValue struct {
|
||||
artifact erasedExtractArtifact
|
||||
serialized CheckpointArtifact
|
||||
warnings []contracts.Warning
|
||||
rejected *contracts.RejectedOutput
|
||||
err error
|
||||
terminal *attemptTerminalRecorder
|
||||
}
|
||||
|
||||
type laneCompletion struct {
|
||||
@@ -110,6 +125,9 @@ func initializeLaneStates(input RunInput, step PreparedPipelineStep, checkpoints
|
||||
return states, err
|
||||
}
|
||||
if input.CheckpointPolicy.requiresReusable(input.stepID, prepared.resolved.ID) && !input.CheckpointPolicy.forced(input.stepID, prepared.resolved.ID) {
|
||||
if !laneReferencesReuseEligible(input, prepared.resolved) {
|
||||
return states, fmt.Errorf("required reusable checkpoint unavailable for step %q lane %q: generated input lineage is validation-incomplete", input.stepID, prepared.resolved.ID)
|
||||
}
|
||||
state, err := hydrateRequiredLane(input, loader, doc, i, prepared)
|
||||
states[i] = state
|
||||
if err != nil {
|
||||
@@ -121,7 +139,7 @@ func initializeLaneStates(input RunInput, step PreparedPipelineStep, checkpoints
|
||||
if err != nil {
|
||||
return states, err
|
||||
}
|
||||
if !state.decision.Reused {
|
||||
if !state.decision.Reused && state.reuseEligible {
|
||||
if err := checkpointExtractRunning(checkpoints, input.stepID, prepared.resolved.ID, prepared.resolved.Extract.Module, state.deps); err != nil {
|
||||
return states, fmt.Errorf("write extract checkpoint for lane %q: %w", prepared.resolved.ID, err)
|
||||
}
|
||||
@@ -290,7 +308,9 @@ func (e *laneEngine) handleExtractResult(result extractJobResult) {
|
||||
if result.err != nil {
|
||||
state.failed = true
|
||||
e.runErrors = append(e.runErrors, orderedRunError{stage: 0, lane: state.index, chunk: result.chunkIndex, err: result.err})
|
||||
_ = checkpointExtractFailed(e.checkpoints, e.input.stepID, state.prepared.resolved.ID, state.prepared.resolved.Extract.Module, state.deps, result.err)
|
||||
if state.reuseEligible {
|
||||
_ = checkpointExtractFailed(e.checkpoints, e.input.stepID, state.prepared.resolved.ID, state.prepared.resolved.Extract.Module, state.deps, result.err)
|
||||
}
|
||||
e.cancel()
|
||||
} else {
|
||||
state.results[result.chunkIndex] = result
|
||||
@@ -328,7 +348,7 @@ func hydrateRequiredLane(input RunInput, loader CheckpointLoader, doc *source.So
|
||||
}
|
||||
resolution, err := resolveCheckpointDecision(&local, loader, input.CheckpointPolicy, StageNormalize, input.stepID, lane.ID, lane.Normalize.Module, decision, typed.codec, []CheckpointArtifact{checkpoint.Output})
|
||||
decision = resolution.decision
|
||||
state := &laneExtractState{index: index, prepared: prepared, decision: decision, terminal: true, output: local}
|
||||
state := &laneExtractState{index: index, prepared: prepared, decision: decision, reuseEligible: true, terminal: true, output: local}
|
||||
if err != nil {
|
||||
return state, err
|
||||
}
|
||||
@@ -341,6 +361,7 @@ func hydrateRequiredLane(input RunInput, loader CheckpointLoader, doc *source.So
|
||||
SourceID: doc.ID,
|
||||
Artifact: contracts.CloneSerializedArtifact(hydrated.Artifact),
|
||||
})
|
||||
local.normalizeReuseEligibility = map[generatedOutputKey]bool{generatedOutputKeyFor(input.stepID, lane.ID): true}
|
||||
state.output = local
|
||||
return state, nil
|
||||
}
|
||||
@@ -374,7 +395,12 @@ func prepareLaneExtract(input RunInput, loader CheckpointLoader, doc *source.Sou
|
||||
}
|
||||
extractReferences := operationReferenceSet(input, lane.ExtractReferences)
|
||||
deps := append(digestFingerprints("chunks", digest), generatedReferenceDependencies(extractReferences)...)
|
||||
state := &laneExtractState{index: index, prepared: prepared, deps: normalizeCheckpointFingerprints(deps), remaining: len(chunks), results: make(map[int]extractJobResult, len(chunks))}
|
||||
state := &laneExtractState{index: index, prepared: prepared, deps: normalizeCheckpointFingerprints(deps), reuseEligible: referenceTargetReuseEligible(input, lane.ExtractReferences), remaining: len(chunks), results: make(map[int]extractJobResult, len(chunks))}
|
||||
if !state.reuseEligible {
|
||||
state.decision = checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonValidationIncompleteLineage)
|
||||
recordCheckpointEvent(output, loader, string(StageExtract), input.stepID, lane.ID, lane.Extract.Module, state.decision)
|
||||
return state, nil
|
||||
}
|
||||
cp, decision := loadExtract(loader, input.stepID, lane.ID, lane.Extract.Module, state.deps)
|
||||
resolution, err := resolveCheckpointDecision(output, loader, input.CheckpointPolicy, StageExtract, input.stepID, lane.ID, lane.Extract.Module, decision, typed.codec, cp.Outputs)
|
||||
if err != nil {
|
||||
@@ -407,23 +433,21 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
|
||||
result.err = fmt.Errorf("clone chunk %q for extraction: %w", job.chunk.ID, cloneErr)
|
||||
return result
|
||||
}
|
||||
var accepted erasedExtractArtifact
|
||||
var serialized CheckpointArtifact
|
||||
var acceptedWarnings []contracts.Warning
|
||||
retryResult, err := runWithRetry(ctx, lane.Extract.Retries, func(attempt int) (retryAttemptResult, error) {
|
||||
terminalResult, err := runProducerAttempts(ctx, producerAttemptConfig{Retries: lane.Extract.Retries, Policy: lane.ExtractValidationPolicy, AllowStructuralRetry: lane.ExtractExecutionClass == contracts.ExecutionClassLLMBacked}, func(attemptCtx context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
attempt := request.Number
|
||||
started := time.Now().UTC()
|
||||
attemptPath := path.Join("extract", fileio.EncodePathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), fmt.Sprintf("attempt-%02d", attempt))
|
||||
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "extract", llmScope, debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started})
|
||||
attemptCtx, llmScope := withDebugLLMScope(attemptCtx, attemptPath)
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "extract", llmScope, debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, AttemptKind: string(request.Kind), StartedAt: started})
|
||||
requestMetadata, metadataErr := cloneMetadata(input.Metadata)
|
||||
if metadataErr != nil {
|
||||
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("clone extract request metadata: %w", metadataErr))
|
||||
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("clone extract request metadata: %w", metadataErr))
|
||||
}
|
||||
extractReferences := operationReferenceSet(input, lane.ExtractReferences)
|
||||
extracted, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(extractReferences), LLMProfile: lane.Extract.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(lane.Extract.StructuredOutputRepairAttempts), Metadata: requestMetadata})
|
||||
extracted, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(extractReferences), LLMProfile: lane.Extract.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(lane.Extract.StructuredOutputRepairAttempts), Correction: request.Correction, Metadata: requestMetadata})
|
||||
if callErr != nil {
|
||||
attemptErr := fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, lane.Extract.Module, callErr)
|
||||
return retryAttemptResult{}, terminal.record(nil, attemptErr)
|
||||
return producerAttemptOutput{}, terminal.record(nil, attemptErr)
|
||||
}
|
||||
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: chunk.ID, ChunkIndex: chunk.Index, ChunkRef: chunk.Ref, Value: extracted.Value}
|
||||
attemptWarnings := cloneWarnings(extracted.Warnings)
|
||||
@@ -431,41 +455,65 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
|
||||
if encodeErr != nil {
|
||||
attemptErr := fmt.Errorf("serialize extract candidate for lane %q chunk %q: %w", lane.ID, chunk.ID, encodeErr)
|
||||
payload := map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}
|
||||
return retryAttemptResult{}, terminal.record(payload, attemptErr)
|
||||
return producerAttemptOutput{}, terminal.record(payload, attemptErr)
|
||||
}
|
||||
serializedCandidate.ChunkID, serializedCandidate.ChunkIndex, serializedCandidate.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageExtract, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: extractReferences, metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: extracted.Value, candidate: &serializedCandidate}, state.prepared.extractValidators, attempt, input.Debug)
|
||||
attemptWarnings = append(attemptWarnings, warnings...)
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
|
||||
if validateErr != nil {
|
||||
return retryAttemptResult{}, terminal.record(payload, validateErr)
|
||||
return producerAttemptOutput{Value: extractAttemptValue{artifact: artifact, serialized: serializedCandidate, terminal: &terminal}, Candidate: extracted.ModelCandidate, Warnings: attemptWarnings}, nil
|
||||
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||
candidate, ok := output.Value.(extractAttemptValue)
|
||||
if !ok {
|
||||
return validationReport{}, fmt.Errorf("extract attempt has incompatible value")
|
||||
}
|
||||
if rejected != nil {
|
||||
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
||||
return retryAttemptResult{}, debugErr
|
||||
}
|
||||
return retryAttemptResult{rejection: rejected, warnings: attemptWarnings}, nil
|
||||
report, validationErr := r.validateTypedReport(validationCtx, typed.codec, typedValidationTarget{stage: StageExtract, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: operationReferenceSet(input, lane.ExtractReferences), metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: candidate.artifact.Value, candidate: &candidate.serialized}, state.prepared.extractValidators, candidate.terminal.envelope.Attempt, input.Debug)
|
||||
payload := map[string]any{
|
||||
"output": debugCheckpointArtifact(candidate.serialized),
|
||||
"warnings": debugWarningEnvelopes(append(cloneWarnings(output.Warnings), report.Warnings()...)),
|
||||
"rejection": debugRejectedOutputPtr(typedRejection(report, typedValidationTarget{stage: StageExtract, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Extract.Module, chunk: &chunk}, candidate.terminal.envelope.Attempt)),
|
||||
}
|
||||
stored, encodeErr := checkpointArtifact(typed.codec, artifact.LaneID, artifact.ExtractorKey, artifact.SourceID, artifact.Value)
|
||||
if encodeErr != nil {
|
||||
attemptErr := fmt.Errorf("serialize accepted extract output for lane %q chunk %q: %w", lane.ID, chunk.ID, encodeErr)
|
||||
return retryAttemptResult{}, terminal.record(payload, attemptErr)
|
||||
if validationErr != nil {
|
||||
return report, candidate.terminal.record(payload, validationErr)
|
||||
}
|
||||
stored.ChunkID, stored.ChunkIndex, stored.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
|
||||
accepted, serialized = artifact, stored
|
||||
acceptedWarnings = attemptWarnings
|
||||
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
||||
return retryAttemptResult{}, debugErr
|
||||
}
|
||||
return retryAttemptResult{accepted: true}, nil
|
||||
return report, candidate.terminal.record(payload, nil)
|
||||
})
|
||||
result.err = err
|
||||
if err == nil && !retryResult.accepted {
|
||||
result.rejected = retryResult.rejection
|
||||
result.warnings = cloneWarnings(retryResult.warnings)
|
||||
summary := validationSummary(terminalResult, StageExtract, input.stepID, lane.ID, lane.Extract.Module, chunk.ID, chunk.Index)
|
||||
if debugErr := writeProducerTerminalDebug(input.Debug, path.Join("extract", fileio.EncodePathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), "terminal.json"), terminalResult, lane.ExtractValidationPolicy, summary); debugErr != nil {
|
||||
result.err = errors.Join(result.err, debugErr)
|
||||
return result
|
||||
}
|
||||
result.value, result.serialized, result.warnings = accepted, serialized, acceptedWarnings
|
||||
if err == nil && terminalResult.Action == producerTerminalRejected {
|
||||
result.rejected = terminalResult.Rejection
|
||||
if result.rejected != nil {
|
||||
result.rejected.Stage, result.rejected.StepID, result.rejected.LaneID, result.rejected.ModuleKey, result.rejected.ChunkID, result.rejected.ChunkIndex = string(StageExtract), input.stepID, lane.ID, lane.Extract.Module, chunk.ID, chunk.Index
|
||||
result.validationSummary = &summary
|
||||
result.rejected.Validation = cloneValidationSummaryPtr(result.validationSummary)
|
||||
}
|
||||
result.warnings = cloneWarnings(terminalResult.Warnings)
|
||||
return result
|
||||
}
|
||||
if err == nil {
|
||||
candidate, ok := terminalResult.Value.(extractAttemptValue)
|
||||
if !ok {
|
||||
result.err = fmt.Errorf("extract attempt terminal has incompatible value")
|
||||
return result
|
||||
}
|
||||
stored, encodeErr := checkpointArtifact(typed.codec, candidate.artifact.LaneID, candidate.artifact.ExtractorKey, candidate.artifact.SourceID, candidate.artifact.Value)
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(candidate.serialized), "warnings": debugWarningEnvelopes(terminalResult.Warnings), "rejection": debugRejectedOutputPtr(nil)}
|
||||
if encodeErr != nil {
|
||||
attemptErr := fmt.Errorf("serialize accepted extract output for lane %q chunk %q: %w", lane.ID, chunk.ID, encodeErr)
|
||||
result.err = candidate.terminal.record(payload, attemptErr)
|
||||
return result
|
||||
}
|
||||
stored.ChunkID, stored.ChunkIndex, stored.ChunkRef = candidate.artifact.ChunkID, candidate.artifact.ChunkIndex, candidate.artifact.ChunkRef
|
||||
if debugErr := candidate.terminal.record(payload, nil); debugErr != nil {
|
||||
result.err = debugErr
|
||||
return result
|
||||
}
|
||||
result.value, result.serialized = candidate.artifact, stored
|
||||
result.warnings = cloneWarnings(terminalResult.Warnings)
|
||||
result.validationIncomplete = terminalResult.ValidationIncomplete
|
||||
result.validationSummary = &summary
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
@@ -478,6 +526,9 @@ func finalizeLaneExtract(checkpoints CheckpointRecorder, stepID string, state *l
|
||||
sort.Ints(indexes)
|
||||
for _, index := range indexes {
|
||||
result := state.results[index]
|
||||
if result.validationSummary != nil {
|
||||
state.validationSummaries = append(state.validationSummaries, artifacts.CloneValidationSummary(*result.validationSummary))
|
||||
}
|
||||
if result.rejected != nil {
|
||||
state.rejected = append(state.rejected, *result.rejected)
|
||||
state.warnings = append(state.warnings, result.warnings...)
|
||||
@@ -486,11 +537,18 @@ func finalizeLaneExtract(checkpoints CheckpointRecorder, stepID string, state *l
|
||||
state.values = append(state.values, result.value)
|
||||
state.serialized = append(state.serialized, result.serialized)
|
||||
state.warnings = append(state.warnings, result.warnings...)
|
||||
if result.validationIncomplete {
|
||||
state.incomplete = append(state.incomplete, result.chunkIndex)
|
||||
}
|
||||
}
|
||||
sort.SliceStable(state.values, func(i, j int) bool { return state.values[i].ChunkIndex < state.values[j].ChunkIndex })
|
||||
sort.SliceStable(state.serialized, func(i, j int) bool { return state.serialized[i].ChunkIndex < state.serialized[j].ChunkIndex })
|
||||
sort.SliceStable(state.rejected, func(i, j int) bool { return state.rejected[i].ChunkIndex < state.rejected[j].ChunkIndex })
|
||||
if !state.decision.Reused {
|
||||
sort.Ints(state.incomplete)
|
||||
if len(state.incomplete) > 0 {
|
||||
state.reuseEligible = false
|
||||
}
|
||||
if !state.decision.Reused && state.reuseEligible {
|
||||
if err := recordExtract(checkpoints, stepID, lane.ID, lane.Extract.Module, state.deps, state.serialized, state.rejected, state.warnings); err != nil {
|
||||
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
|
||||
}
|
||||
@@ -502,18 +560,22 @@ func (r *Runner) continueLane(ctx context.Context, input RunInput, checkpoints C
|
||||
lane := state.prepared.resolved
|
||||
local := RunOutput{Manifest: manifestFromPipeline(input)}
|
||||
results := finalizedExtractResults{
|
||||
accepted: state.values,
|
||||
serialized: state.serialized,
|
||||
warnings: state.warnings,
|
||||
rejected: state.rejected,
|
||||
decision: state.decision,
|
||||
accepted: state.values,
|
||||
serialized: state.serialized,
|
||||
warnings: state.warnings,
|
||||
rejected: state.rejected,
|
||||
incomplete: state.incomplete,
|
||||
validationSummaries: state.validationSummaries,
|
||||
decision: state.decision,
|
||||
reuseEligible: state.reuseEligible,
|
||||
}
|
||||
local.Warnings = append(local.Warnings, cloneWarnings(results.warnings)...)
|
||||
local.Rejected = append(local.Rejected, cloneRejectedOutputs(results.rejected)...)
|
||||
local.ValidationSummaries = append(local.ValidationSummaries, cloneValidationSummaries(results.validationSummaries)...)
|
||||
if err := writeDebugTimed(input.Debug, path.Join("extract", fileio.EncodePathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "decision": results.decision, "source": debugSourceDocumentEnvelope(doc), "chunks": debugSourceChunkEnvelopes(chunks), "options": redactSensitiveMap(lane.Extract.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||
return local, &laneRunError{stage: StageExtract, err: err}
|
||||
}
|
||||
if err := writeDebugTimed(input.Debug, path.Join("extract", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "outputs": debugCheckpointArtifacts(results.serialized), "rejected": debugRejectedOutputEnvelopes(results.rejected), "warnings": debugWarningEnvelopes(results.warnings)}}); err != nil {
|
||||
if err := writeDebugTimed(input.Debug, path.Join("extract", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "outputs": debugCheckpointArtifacts(results.serialized), "rejected": debugRejectedOutputEnvelopes(results.rejected), "warnings": debugWarningEnvelopes(results.warnings), "validation_incomplete_chunks": append([]int(nil), results.incomplete...)}}); err != nil {
|
||||
return local, &laneRunError{stage: StageExtract, err: err}
|
||||
}
|
||||
if len(results.accepted) == 0 {
|
||||
@@ -580,6 +642,15 @@ func mergeLaneOutput(dst *RunOutput, src RunOutput) error {
|
||||
dst.Rejected = append(dst.Rejected, cloneRejectedOutputs(src.Rejected)...)
|
||||
dst.Warnings = append(dst.Warnings, cloneWarnings(src.Warnings)...)
|
||||
dst.CheckpointEvents = append(dst.CheckpointEvents, src.CheckpointEvents...)
|
||||
dst.ValidationSummaries = append(dst.ValidationSummaries, cloneValidationSummaries(src.ValidationSummaries)...)
|
||||
if len(src.normalizeReuseEligibility) > 0 {
|
||||
if dst.normalizeReuseEligibility == nil {
|
||||
dst.normalizeReuseEligibility = make(map[generatedOutputKey]bool, len(src.normalizeReuseEligibility))
|
||||
}
|
||||
for key, eligible := range src.normalizeReuseEligibility {
|
||||
dst.normalizeReuseEligibility[key] = eligible
|
||||
}
|
||||
}
|
||||
for i := range dst.Manifest.ArtifactLanes {
|
||||
for j := range src.Manifest.ArtifactLanes {
|
||||
if dst.Manifest.ArtifactLanes[i].ID == src.Manifest.ArtifactLanes[j].ID && dst.Manifest.ArtifactLanes[i].StepID == src.Manifest.ArtifactLanes[j].StepID && src.Manifest.ArtifactLanes[j].Metadata != nil {
|
||||
|
||||
139
internal/framework/pipeline/runner_extract_correction_test.go
Normal file
139
internal/framework/pipeline/runner_extract_correction_test.go
Normal file
@@ -0,0 +1,139 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestRunnerCorrectsExtractCandidatesIndependentlyPerChunk(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 2)
|
||||
lane := &prepared.Steps[0].lanes[0]
|
||||
lane.resolved.Extract.Retries = 1
|
||||
|
||||
var mu sync.Mutex
|
||||
corrections := make(map[int]*contracts.SemanticCorrection)
|
||||
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
chunkIndex := request.Chunk.Index
|
||||
if request.Correction != nil {
|
||||
correction, err := contracts.CloneSemanticCorrection(request.Correction)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
mu.Lock()
|
||||
corrections[chunkIndex] = correction
|
||||
mu.Unlock()
|
||||
candidate, err := contracts.NewModelCandidate([]byte(fmt.Sprintf("corrected-response-%d", chunkIndex)), contracts.CorrectionProtocolSingleResponseV1)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{fmt.Sprintf("corrected-%d", chunkIndex)}}, ModelCandidate: candidate}, nil
|
||||
}
|
||||
candidate, err := contracts.NewModelCandidate([]byte(fmt.Sprintf("initial-response-%d", chunkIndex)), contracts.CorrectionProtocolSingleResponseV1)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{fmt.Sprintf("invalid-%d", chunkIndex)}}, ModelCandidate: candidate}, nil
|
||||
})
|
||||
lane.extractValidators.validators[0].typedValidate = func(_ context.Context, _ any, target typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
value := target.value.(codecNotes)
|
||||
if value.Items[0] == fmt.Sprintf("invalid-%d", target.chunk.Index) {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "incorrect_extract", Message: "candidate needs correction", CorrectionGuidance: fmt.Sprintf("correct chunk %d", target.chunk.Index)}, nil
|
||||
}
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
|
||||
debug := newCapturedDebugRecorder()
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2, Debug: debug})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v, want nil", err)
|
||||
}
|
||||
if len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 2 {
|
||||
t.Fatalf("run output = %#v, want corrected accepted extracts", output)
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if len(corrections) != 2 {
|
||||
t.Fatalf("corrections = %#v, want one correction per chunk", corrections)
|
||||
}
|
||||
for index := 0; index < 2; index++ {
|
||||
correction := corrections[index]
|
||||
if correction == nil || string(correction.AssistantResponse) != fmt.Sprintf("initial-response-%d", index) || !strings.Contains(correction.UserGuidance, fmt.Sprintf("correct chunk %d", index)) || !strings.Contains(correction.UserGuidance, "complete corrected replacement") || strings.Contains(correction.UserGuidance, "incorrect_extract") || strings.Contains(correction.UserGuidance, "candidate needs correction") {
|
||||
t.Fatalf("chunk %d correction = %#v, want its exact initial response and semantic replacement guidance only", index, correction)
|
||||
}
|
||||
}
|
||||
terminal := string(debug.json["extract/notes/chunk-000001/terminal.json"])
|
||||
if !strings.Contains(terminal, `"kind":"semantic_correction"`) || !strings.Contains(terminal, `"producer_attempt_count":2`) {
|
||||
t.Fatalf("extract terminal debug = %s, want correction provenance", terminal)
|
||||
}
|
||||
if strings.Contains(terminal, "initial-response-0") || strings.Contains(terminal, "corrected-response-0") || strings.Contains(terminal, "correct chunk 0") {
|
||||
t.Fatalf("extract terminal debug leaked correction content: %s", terminal)
|
||||
}
|
||||
manifest, err := json.Marshal(output.Manifest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(string(manifest), "initial-response-0") || strings.Contains(string(manifest), "corrected-response-0") || strings.Contains(string(manifest), "correct chunk 0") {
|
||||
t.Fatalf("manifest leaked correction content: %s", manifest)
|
||||
}
|
||||
}
|
||||
|
||||
func containsWarnings(have, want []contracts.Warning) bool {
|
||||
for index := 0; index+len(want) <= len(have); index++ {
|
||||
if reflect.DeepEqual(have[index:index+len(want)], want) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func containsValidationSummary(summaries []artifacts.ValidationSummary, status string) bool {
|
||||
for _, summary := range summaries {
|
||||
if summary.Status == status {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func TestRunnerContinuesValidationIncompleteExtractWithoutCheckpoint(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
lane := &prepared.Steps[0].lanes[0]
|
||||
lane.resolved.ExtractValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
|
||||
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
|
||||
})
|
||||
lane.extractValidators.validators[0].typedValidate = func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{}, fmt.Errorf("validator unavailable")
|
||||
}
|
||||
recorder := &extractCaptureRecorder{CheckpointRecorder: NoopCheckpointRecorder()}
|
||||
debug := newCapturedDebugRecorder()
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoints: recorder, Debug: debug})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v, want nil", err)
|
||||
}
|
||||
if len(output.NormalizeOutputs) != 1 || len(output.Rejected) != 0 {
|
||||
t.Fatalf("run output = %#v, want accepted incomplete extract", output)
|
||||
}
|
||||
if output.Manifest.ValidationStatus != "incomplete" || !containsValidationSummary(output.Manifest.ValidationSummaries, "incomplete") {
|
||||
t.Fatalf("manifest validation = %#v, want incomplete extract provenance", output.Manifest)
|
||||
}
|
||||
if got, want := output.Warnings, []contracts.Warning{{Scope: "typed/check", ReasonCode: "validator_execution_incomplete", Message: "Validator execution did not complete within its configured budget."}}; !containsWarnings(got, want) {
|
||||
t.Fatalf("warnings = %#v, want %#v", got, want)
|
||||
}
|
||||
if len(recorder.checkpoint.Outputs) != 0 || len(recorder.checkpoint.Rejected) != 0 {
|
||||
t.Fatalf("extract checkpoint = %#v, want no persisted incomplete output", recorder.checkpoint)
|
||||
}
|
||||
encoded := string(debug.json["extract/notes/output.json"])
|
||||
if !strings.Contains(encoded, `"validation_incomplete_chunks":[0]`) {
|
||||
t.Fatalf("extract output debug = %s, want incomplete chunk marker", encoded)
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package pipeline
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
@@ -221,14 +222,15 @@ func TestRunnerPromotesOnlyAcceptedExtractRetryWarnings(t *testing.T) {
|
||||
scope = "accepted"
|
||||
}
|
||||
return erasedTypedResult{
|
||||
Value: typedValueForLane(0, request.Chunk.Index),
|
||||
Warnings: []contracts.Warning{{Scope: scope, ReasonCode: "observed", Message: scope}},
|
||||
Value: typedValueForLane(0, request.Chunk.Index),
|
||||
Warnings: []contracts.Warning{{Scope: scope, ReasonCode: "observed", Message: scope}},
|
||||
ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%s"]}`, scope)),
|
||||
}, nil
|
||||
})
|
||||
validatorCalls := 0
|
||||
prepared.Steps[0].lanes[0].extractValidators.validators[0].typedValidate = func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
validatorCalls++
|
||||
return contracts.ValidationResult{Approved: validatorCalls == 2, ReasonCode: "retry", Message: "retry extract"}, nil
|
||||
return contracts.ValidationResult{Approved: validatorCalls == 2, ReasonCode: "retry", Message: "retry extract", CorrectionGuidance: "return an acceptable extract"}, nil
|
||||
}
|
||||
debug := newCapturedDebugRecorder()
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug})
|
||||
|
||||
@@ -70,7 +70,7 @@ func TestRunnerHandlesRetryableNormalizeFallbacks(t *testing.T) {
|
||||
validator: &preparedValidator{
|
||||
resolved: ResolvedValidator{Binding: Binding("reject-final-fallback"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "reject fallback"}, nil
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "reject fallback", CorrectionGuidance: "return an acceptable normalized artifact"}, nil
|
||||
},
|
||||
},
|
||||
wantCalls: 2,
|
||||
@@ -85,6 +85,7 @@ func TestRunnerHandlesRetryableNormalizeFallbacks(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
lane := &prepared.Steps[0].lanes[0]
|
||||
lane.resolved.Normalize.Retries = tc.retries
|
||||
lane.resolved.NormalizeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
if tc.validator != nil {
|
||||
lane.normalizeValidators.validators = []preparedValidator{*tc.validator}
|
||||
}
|
||||
|
||||
@@ -50,13 +50,14 @@ func TestRunnerPromotesOnlyTerminalRejectionWarnings(t *testing.T) {
|
||||
return contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("validator-%d", attempts), ReasonCode: "validator", Message: "validator warning"}}}
|
||||
}
|
||||
reject := func() contracts.ValidationResult {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected"}
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected", CorrectionGuidance: "return an acceptable candidate"}
|
||||
}
|
||||
debug := newCapturedDebugRecorder()
|
||||
recorder := &extractCaptureRecorder{CheckpointRecorder: NoopCheckpointRecorder()}
|
||||
|
||||
switch target {
|
||||
case StageChunk:
|
||||
prepared.resolved.ChunkValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
chunker := prepared.chunker.(*typedTestChunker)
|
||||
prepared.chunker = &warningChunker{key: prepared.resolved.Chunk.Module, plan: source.CloneChunkPlan(chunker.plan)}
|
||||
prepared.resolved.Chunk.Retries = 1
|
||||
@@ -74,23 +75,26 @@ func TestRunnerPromotesOnlyTerminalRejectionWarnings(t *testing.T) {
|
||||
}
|
||||
case StageExtract:
|
||||
lane.resolved.Extract.Retries = 1
|
||||
lane.resolved.ExtractValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
installExtractOperation(prepared, 0, func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
attempts++
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"extract"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}}, nil
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"extract"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%d"]}`, attempts))}, nil
|
||||
})
|
||||
lane.extractValidators.validators = rejectionWarningTypedValidators(first, reject)
|
||||
case StageMerge:
|
||||
lane.resolved.Merge.Retries = 1
|
||||
lane.resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
lane.typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
attempts++
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"merge"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}}, nil
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"merge"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%d"]}`, attempts))}, nil
|
||||
}
|
||||
lane.mergeValidators.validators = rejectionWarningTypedValidators(first, reject)
|
||||
case StageNormalize:
|
||||
lane.resolved.Normalize.Retries = 1
|
||||
lane.resolved.NormalizeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
lane.typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
attempts++
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"normalize"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}}, nil
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"normalize"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%d"]}`, attempts))}, nil
|
||||
}
|
||||
lane.normalizeValidators.validators = rejectionWarningTypedValidators(first, reject)
|
||||
}
|
||||
@@ -100,11 +104,16 @@ func TestRunnerPromotesOnlyTerminalRejectionWarnings(t *testing.T) {
|
||||
t.Fatalf("Run() error = %v, want nil", err)
|
||||
}
|
||||
wantScopes := []string{"operation-2", "validator-2"}
|
||||
wantAttempts := 2
|
||||
if target == StageChunk {
|
||||
wantScopes = []string{"operation-1", "validator-1"}
|
||||
wantAttempts = 1
|
||||
}
|
||||
if got := rejectionWarningScopes(output.Warnings); !reflect.DeepEqual(got, wantScopes) {
|
||||
t.Fatalf("published warning scopes = %#v, want %#v", got, wantScopes)
|
||||
}
|
||||
if len(output.Rejected) != 1 || output.Rejected[0].AttemptCount != 2 {
|
||||
t.Fatalf("rejections = %#v, want final rejection after two attempts", output.Rejected)
|
||||
if len(output.Rejected) != 1 || output.Rejected[0].AttemptCount != wantAttempts {
|
||||
t.Fatalf("rejections = %#v, want terminal rejection after %d attempt(s)", output.Rejected, wantAttempts)
|
||||
}
|
||||
if target == StageExtract && !reflect.DeepEqual(rejectionWarningScopes(recorder.checkpoint.Warnings), wantScopes) {
|
||||
t.Fatalf("extract checkpoint warnings = %#v, want %#v", recorder.checkpoint.Warnings, wantScopes)
|
||||
|
||||
238
internal/framework/pipeline/runner_reuse_lineage_test.go
Normal file
238
internal/framework/pipeline/runner_reuse_lineage_test.go
Normal file
@@ -0,0 +1,238 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type reuseLineageCheckpointSpy struct {
|
||||
CheckpointLoader
|
||||
CheckpointRecorder
|
||||
loads map[string]int
|
||||
writes map[string]int
|
||||
forbid map[string]struct{}
|
||||
}
|
||||
|
||||
func newReuseLineageCheckpointSpy() *reuseLineageCheckpointSpy {
|
||||
return &reuseLineageCheckpointSpy{
|
||||
CheckpointLoader: NoopCheckpointLoader(),
|
||||
CheckpointRecorder: NoopCheckpointRecorder(),
|
||||
loads: make(map[string]int),
|
||||
writes: make(map[string]int),
|
||||
forbid: make(map[string]struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) Enabled() bool { return true }
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) load(stage, laneID string) {
|
||||
key := stage + "/" + laneID
|
||||
if _, forbidden := s.forbid[key]; forbidden {
|
||||
panic("checkpoint lookup crossed validation-incomplete lineage: " + key)
|
||||
}
|
||||
s.loads[key]++
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) write(stage, action, laneID string) {
|
||||
s.writes[stage+"/"+action+"/"+laneID]++
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) Extract(laneID, _ string, _ []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||
s.load("extract", laneID)
|
||||
return ExtractCheckpoint{}, NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing)
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) Merge(laneID, _ string, _ []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
|
||||
s.load("merge", laneID)
|
||||
return MergeCheckpoint{}, NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing)
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) Normalize(laneID, _ string, _ []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
s.load("normalize", laneID)
|
||||
return NormalizeCheckpoint{}, NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing)
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) ExtractRunning(laneID, _ string, _ []CheckpointFingerprint) error {
|
||||
s.write("extract", "running", laneID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) ExtractSucceeded(laneID, _ string, _ []CheckpointFingerprint, _ []CheckpointArtifact, _ []contracts.RejectedOutput, _ []contracts.Warning) error {
|
||||
s.write("extract", "succeeded", laneID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) ExtractFailed(laneID, _ string, _ []CheckpointFingerprint, _ error) error {
|
||||
s.write("extract", "failed", laneID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) MergeRunning(laneID, _ string, _ []CheckpointFingerprint) error {
|
||||
s.write("merge", "running", laneID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) MergeSucceeded(laneID, _ string, _ []CheckpointFingerprint, _ CheckpointArtifact, _ []contracts.Warning) error {
|
||||
s.write("merge", "succeeded", laneID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) MergeRejected(laneID, _ string, _ []CheckpointFingerprint, _ contracts.RejectedOutput) error {
|
||||
s.write("merge", "rejected", laneID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) MergeFailed(laneID, _ string, _ []CheckpointFingerprint, _ error) error {
|
||||
s.write("merge", "failed", laneID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) NormalizeRunning(laneID, _ string, _ []CheckpointFingerprint) error {
|
||||
s.write("normalize", "running", laneID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) NormalizeSucceeded(laneID, _ string, _ []CheckpointFingerprint, _ CheckpointArtifact, _ []contracts.Warning) error {
|
||||
s.write("normalize", "succeeded", laneID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) NormalizeRejected(laneID, _ string, _ []CheckpointFingerprint, _ contracts.RejectedOutput) error {
|
||||
s.write("normalize", "rejected", laneID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) NormalizeFailed(laneID, _ string, _ []CheckpointFingerprint, _ error) error {
|
||||
s.write("normalize", "failed", laneID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *reuseLineageCheckpointSpy) writesFor(stage, laneID string) int {
|
||||
total := 0
|
||||
needle := stage + "/"
|
||||
suffix := "/" + laneID
|
||||
for key, count := range s.writes {
|
||||
if strings.HasPrefix(key, needle) && strings.HasSuffix(key, suffix) {
|
||||
total += count
|
||||
}
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
func unavailableTypedValidator(kind contracts.ArtifactKind) preparedValidator {
|
||||
return preparedValidator{
|
||||
resolved: ResolvedValidator{Binding: Binding("typed/check"), Target: ValidatorTargetTyped, ArtifactKind: kind},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{}, fmt.Errorf("validator unavailable")
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidationIncompleteExtractDisablesDownstreamCheckpointIO(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
lane := &prepared.Steps[0].lanes[0]
|
||||
lane.resolved.ExtractValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
|
||||
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
|
||||
})
|
||||
lane.extractValidators.validators[0].typedValidate = func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{}, fmt.Errorf("validator unavailable")
|
||||
}
|
||||
spy := newReuseLineageCheckpointSpy()
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: spy, Checkpoints: spy})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(output.NormalizeOutputs) != 1 {
|
||||
t.Fatalf("normalize outputs = %d, want current-run handoff", len(output.NormalizeOutputs))
|
||||
}
|
||||
if spy.loads["merge/notes"] != 0 || spy.loads["normalize/notes"] != 0 {
|
||||
t.Fatalf("downstream loads = %#v, want none", spy.loads)
|
||||
}
|
||||
if spy.writesFor("merge", "notes") != 0 || spy.writesFor("normalize", "notes") != 0 {
|
||||
t.Fatalf("downstream writes = %#v, want none", spy.writes)
|
||||
}
|
||||
if spy.writes["extract/succeeded/notes"] != 0 {
|
||||
t.Fatalf("extract writes = %#v, want no reusable success", spy.writes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidationIncompleteMergeDisablesNormalizeCheckpointIO(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
lane := &prepared.Steps[0].lanes[0]
|
||||
lane.resolved.MergeValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
|
||||
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
|
||||
})
|
||||
lane.mergeValidators.validators = []preparedValidator{unavailableTypedValidator(lane.resolved.ArtifactKind)}
|
||||
spy := newReuseLineageCheckpointSpy()
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: spy, Checkpoints: spy})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(output.NormalizeOutputs) != 1 {
|
||||
t.Fatalf("normalize outputs = %d, want current-run handoff", len(output.NormalizeOutputs))
|
||||
}
|
||||
if spy.loads["merge/notes"] != 1 || spy.loads["normalize/notes"] != 0 {
|
||||
t.Fatalf("loads = %#v, want merge lookup only", spy.loads)
|
||||
}
|
||||
if spy.writes["merge/succeeded/notes"] != 0 || spy.writesFor("normalize", "notes") != 0 {
|
||||
t.Fatalf("writes = %#v, want no reusable merge or normalize state", spy.writes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidationIncompleteNormalizeIsNotPublished(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
lane := &prepared.Steps[0].lanes[0]
|
||||
lane.resolved.NormalizeValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
|
||||
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
|
||||
})
|
||||
lane.normalizeValidators.validators = []preparedValidator{unavailableTypedValidator(lane.resolved.ArtifactKind)}
|
||||
spy := newReuseLineageCheckpointSpy()
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: spy, Checkpoints: spy})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(output.NormalizeOutputs) != 1 || spy.writes["normalize/succeeded/notes"] != 0 {
|
||||
t.Fatalf("output/writes = %d / %#v, want in-memory output without normalize publication", len(output.NormalizeOutputs), spy.writes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGeneratedReferenceFromIncompleteValidationDisablesDependentCheckpointIO(t *testing.T) {
|
||||
input, _, _ := handoffFixture(t, codecNotes{Items: []string{"producer"}})
|
||||
prepared := input.Prepared
|
||||
producer := &prepared.Steps[0].lanes[0]
|
||||
consumer := &prepared.Steps[1].lanes[0]
|
||||
producer.resolved.NormalizeValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
|
||||
producer.normalizeValidators.validators = []preparedValidator{unavailableTypedValidator(producer.resolved.ArtifactKind)}
|
||||
referenceItems := 0
|
||||
consumer.typed.extract = func(_ context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
referenceItems = len(request.References.Slots["producer-output"].Items)
|
||||
return erasedTypedResult{Value: codecScore{Value: 1}}, nil
|
||||
}
|
||||
spy := newReuseLineageCheckpointSpy()
|
||||
for _, stage := range []string{"extract", "merge", "normalize"} {
|
||||
spy.forbid[stage+"/score"] = struct{}{}
|
||||
}
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: spy, Checkpoints: spy})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if referenceItems != 1 || len(output.NormalizeOutputs) != 2 {
|
||||
t.Fatalf("current-run handoff = items %d outputs %d, want one generated item and two outputs", referenceItems, len(output.NormalizeOutputs))
|
||||
}
|
||||
for _, stage := range []string{"extract", "merge", "normalize"} {
|
||||
if spy.writesFor(stage, "score") != 0 {
|
||||
t.Fatalf("dependent writes = %#v, want none for %s", spy.writes, stage)
|
||||
}
|
||||
}
|
||||
}
|
||||
126
internal/framework/pipeline/runner_semantic_correction_test.go
Normal file
126
internal/framework/pipeline/runner_semantic_correction_test.go
Normal file
@@ -0,0 +1,126 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestRunnerCorrectsRejectedMergeAndNormalizeCandidates(t *testing.T) {
|
||||
for _, target := range []ModuleStage{StageMerge, StageNormalize} {
|
||||
t.Run(string(target), func(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
lane := &prepared.Steps[0].lanes[0]
|
||||
var correction *contracts.SemanticCorrection
|
||||
validator := preparedValidator{
|
||||
resolved: ResolvedValidator{Binding: Binding("correction-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(_ context.Context, _ any, target typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
value := target.value.(codecNotes)
|
||||
if firstNote(value) == "invalid" {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "invalid_candidate", Message: "candidate needs correction", CorrectionGuidance: "produce the accepted value"}, nil
|
||||
}
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
},
|
||||
}
|
||||
operation := func(observed *contracts.SemanticCorrection) (erasedTypedResult, error) {
|
||||
if observed != nil {
|
||||
cloned, err := contracts.CloneSemanticCorrection(observed)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
correction = cloned
|
||||
}
|
||||
item, response := "invalid", `{"items":["invalid"]}`
|
||||
if observed != nil {
|
||||
item, response = "accepted", `{"items":["accepted"]}`
|
||||
}
|
||||
candidate, err := contracts.NewModelCandidate([]byte(response), contracts.CorrectionProtocolSingleResponseV1)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{item}}, ModelCandidate: candidate}, nil
|
||||
}
|
||||
switch target {
|
||||
case StageMerge:
|
||||
lane.resolved.Merge.Retries = 1
|
||||
lane.mergeValidators.validators = []preparedValidator{validator}
|
||||
lane.typed.merge = func(_ context.Context, _ any, request contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
return operation(request.Correction)
|
||||
}
|
||||
case StageNormalize:
|
||||
lane.resolved.Normalize.Retries = 1
|
||||
lane.normalizeValidators.validators = []preparedValidator{validator}
|
||||
lane.typed.normalize = func(_ context.Context, _ any, request contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
return operation(request.Correction)
|
||||
}
|
||||
}
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v, want nil", err)
|
||||
}
|
||||
if correction == nil || string(correction.AssistantResponse) != `{"items":["invalid"]}` || !strings.Contains(correction.UserGuidance, "produce the accepted value") || !strings.Contains(correction.UserGuidance, "complete corrected replacement") || strings.Contains(correction.UserGuidance, "invalid_candidate") || strings.Contains(correction.UserGuidance, "candidate needs correction") {
|
||||
t.Fatalf("%s correction = %#v, want exact rejected response and semantic replacement guidance only", target, correction)
|
||||
}
|
||||
if len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
|
||||
t.Fatalf("run output = %#v, want corrected accepted output", output)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerRejectsDeterministicMergeCandidateWithoutCorrection(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
lane := &prepared.Steps[0].lanes[0]
|
||||
lane.resolved.Merge.Retries = 1
|
||||
lane.resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||
calls := 0
|
||||
lane.typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
calls++
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"invalid"}}}, nil
|
||||
}
|
||||
lane.mergeValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("reject"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "invalid_candidate", Message: "not accepted", CorrectionGuidance: "return an acceptable candidate"}, nil
|
||||
},
|
||||
}}
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v, want non-fatal rejection", err)
|
||||
}
|
||||
if calls != 1 || len(output.Rejected) != 1 || output.Rejected[0].AttemptCount != 1 {
|
||||
t.Fatalf("merge calls = %d output = %#v, want one deterministic terminal rejection", calls, output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerContinuesNormalizeAfterValidatorFailure(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
lane := &prepared.Steps[0].lanes[0]
|
||||
lane.resolved.NormalizeValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
|
||||
lane.normalizeValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("unavailable"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{}, fmt.Errorf("validator unavailable")
|
||||
},
|
||||
}}
|
||||
checkpoints := &candidateCheckpointRecorder{CheckpointRecorder: NoopCheckpointRecorder()}
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoints: checkpoints})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v, want incomplete accepted output", err)
|
||||
}
|
||||
if len(output.NormalizeOutputs) != 1 || len(output.Rejected) != 0 || checkpoints.normalizeSucceeded != 0 {
|
||||
t.Fatalf("run output = %#v checkpoints = %#v, want non-checkpointed incomplete normalized output", output, checkpoints)
|
||||
}
|
||||
if output.Manifest.ValidationStatus != "incomplete" || !containsValidationSummary(output.Manifest.ValidationSummaries, "incomplete") {
|
||||
t.Fatalf("manifest validation = %#v, want incomplete normalization provenance", output.Manifest)
|
||||
}
|
||||
if got, want := output.Warnings, []contracts.Warning{{Scope: "unavailable", ReasonCode: "validator_execution_incomplete", Message: "Validator execution did not complete within its configured budget."}}; !containsWarnings(got, want) {
|
||||
t.Fatalf("warnings = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
@@ -37,6 +37,11 @@ type terminalChunkValidator struct {
|
||||
err error
|
||||
}
|
||||
|
||||
type retryingLLMChunkValidator struct {
|
||||
requests []contracts.ChunkValidationRequest
|
||||
calls int
|
||||
}
|
||||
|
||||
type observingChunkValidator struct {
|
||||
request contracts.ChunkValidationRequest
|
||||
}
|
||||
@@ -73,6 +78,24 @@ func (v terminalChunkValidator) Validate(context.Context, contracts.ChunkValidat
|
||||
return v.result, v.err
|
||||
}
|
||||
|
||||
func (*retryingLLMChunkValidator) Name() string { return "retrying/llm-chunk-validator" }
|
||||
|
||||
func (*retryingLLMChunkValidator) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassLLMBacked
|
||||
}
|
||||
|
||||
func (validator *retryingLLMChunkValidator) Validate(_ context.Context, request contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
|
||||
validator.calls++
|
||||
validator.requests = append(validator.requests, request)
|
||||
if validator.calls == 1 && len(request.Chunks) > 0 {
|
||||
request.Chunks[0].ID = "mutated"
|
||||
}
|
||||
if validator.calls == 1 {
|
||||
return contracts.ValidationResult{}, errors.New("temporary validator failure")
|
||||
}
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
|
||||
func assertAttemptEnvelopeSequence(t *testing.T, debug *capturedDebugRecorder, prefix string, attempts ...int) {
|
||||
t.Helper()
|
||||
marker := strings.TrimSuffix(prefix, "/") + "/attempt-"
|
||||
@@ -100,6 +123,26 @@ func assertAttemptEnvelopeSequence(t *testing.T, debug *capturedDebugRecorder, p
|
||||
func preparedTerminalDebugPipeline(t *testing.T) (*PreparedPipeline, source.ChunkPlan) {
|
||||
t.Helper()
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
prepared.resolved.ChunkValidationPolicy = ValidationPolicy{
|
||||
ProducerStructuralFailure: ProducerStructuralFailureFailRun,
|
||||
SemanticRejection: SemanticRejectionRejectOutput,
|
||||
ValidatorFailure: ValidatorFailureFailRun,
|
||||
}
|
||||
prepared.Steps[0].lanes[0].resolved.ExtractValidationPolicy = ValidationPolicy{
|
||||
ProducerStructuralFailure: ProducerStructuralFailureFailRun,
|
||||
SemanticRejection: SemanticRejectionRejectOutput,
|
||||
ValidatorFailure: ValidatorFailureFailRun,
|
||||
}
|
||||
prepared.Steps[0].lanes[0].resolved.MergeValidationPolicy = ValidationPolicy{
|
||||
ProducerStructuralFailure: ProducerStructuralFailureFailRun,
|
||||
SemanticRejection: SemanticRejectionRejectOutput,
|
||||
ValidatorFailure: ValidatorFailureFailRun,
|
||||
}
|
||||
prepared.Steps[0].lanes[0].resolved.NormalizeValidationPolicy = ValidationPolicy{
|
||||
ProducerStructuralFailure: ProducerStructuralFailureFailRun,
|
||||
SemanticRejection: SemanticRejectionRejectOutput,
|
||||
ValidatorFailure: ValidatorFailureFailRun,
|
||||
}
|
||||
chunker, ok := prepared.chunker.(*typedTestChunker)
|
||||
if !ok {
|
||||
t.Fatalf("prepared chunker = %T, want *typedTestChunker", prepared.chunker)
|
||||
@@ -114,10 +157,11 @@ func TestRunnerRecordsChunkTerminalOutcomes(t *testing.T) {
|
||||
validator terminalChunkValidator
|
||||
wantError string
|
||||
wantRejection bool
|
||||
attemptError bool
|
||||
}{
|
||||
{name: "accepted", validator: terminalChunkValidator{result: contracts.ValidationResult{Approved: true}}},
|
||||
{name: "module error", moduleError: errors.New("chunk module failed"), wantError: "chunk module failed"},
|
||||
{name: "validator rejection", validator: terminalChunkValidator{result: contracts.ValidationResult{Approved: false, ReasonCode: "chunk_rejected", Message: "chunk rejected"}}, wantRejection: true},
|
||||
{name: "module error", moduleError: errors.New("chunk module failed"), wantError: "chunk module failed", attemptError: true},
|
||||
{name: "validator rejection", validator: terminalChunkValidator{result: contracts.ValidationResult{Approved: false, ReasonCode: "chunk_rejected", Message: "chunk rejected", CorrectionGuidance: "return an acceptable chunk plan"}}, wantRejection: true},
|
||||
{name: "validator error", validator: terminalChunkValidator{err: errors.New("chunk validator failed")}, wantError: "chunk validator failed"},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
@@ -131,9 +175,15 @@ func TestRunnerRecordsChunkTerminalOutcomes(t *testing.T) {
|
||||
assertAttemptEnvelopeSequence(t, debug, "chunk", 1)
|
||||
envelope := debug.envelope(t, "chunk/attempt-01.json")
|
||||
if tc.wantError != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tc.wantError) || !strings.Contains(envelope.Error, tc.wantError) {
|
||||
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
|
||||
t.Fatalf("Run() error = %v, attempt error = %q; want %q", err, envelope.Error, tc.wantError)
|
||||
}
|
||||
if tc.attemptError && !strings.Contains(envelope.Error, tc.wantError) {
|
||||
t.Fatalf("attempt error = %q, want %q", envelope.Error, tc.wantError)
|
||||
}
|
||||
if !tc.attemptError && envelope.Error != "" {
|
||||
t.Fatalf("settled validator failure attempt error = %q, want empty", envelope.Error)
|
||||
}
|
||||
} else if err != nil {
|
||||
t.Fatalf("Run() error = %v, want nil", err)
|
||||
}
|
||||
@@ -152,6 +202,28 @@ func TestRunnerRecordsChunkTerminalOutcomes(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerRetriesLLMValidatorsWithoutRegeneratingChunkCandidate(t *testing.T) {
|
||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||
chunkCalls := 0
|
||||
prepared.resolved.Chunk.Retries = 0
|
||||
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan, calls: &chunkCalls}
|
||||
validator := &retryingLLMChunkValidator{}
|
||||
prepared.chunkValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: ModuleBinding{Module: validator.Name(), Retries: 1}, ExecutionClass: contracts.ExecutionClassLLMBacked, Target: ValidatorTargetChunk},
|
||||
chunk: validator,
|
||||
}}
|
||||
|
||||
if _, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")}); err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if chunkCalls != 1 || validator.calls != 2 {
|
||||
t.Fatalf("producer calls = %d validator calls = %d, want 1 and 2", chunkCalls, validator.calls)
|
||||
}
|
||||
if len(validator.requests) != 2 || len(validator.requests[0].Chunks) == 0 || len(validator.requests[1].Chunks) == 0 || validator.requests[1].Chunks[0].ID == "mutated" {
|
||||
t.Fatalf("validator requests = %#v, want independently owned immutable candidates", validator.requests)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerMaterializesAnnotatedPlanBeforeChunkValidation(t *testing.T) {
|
||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||
plan.Annotations = source.ChunkAnnotations{"same": []byte(`{"plan":1}`)}
|
||||
@@ -235,16 +307,22 @@ func TestRunnerRecordsExtractTerminalOutcomes(t *testing.T) {
|
||||
candidateFail bool
|
||||
finalFail bool
|
||||
wantError string
|
||||
attemptError bool
|
||||
}{
|
||||
{name: "terminal rejection", reject: true},
|
||||
{name: "module error", moduleError: errors.New("extract module failed"), wantError: "extract module failed"},
|
||||
{name: "module error", moduleError: errors.New("extract module failed"), wantError: "extract module failed", attemptError: true},
|
||||
{name: "validator error", validatorErr: errors.New("extract validator failed"), wantError: "extract validator failed"},
|
||||
{name: "candidate codec error", candidateFail: true, wantError: "serialize extract candidate"},
|
||||
{name: "final codec error", finalFail: true, wantError: "serialize accepted extract output"},
|
||||
{name: "candidate codec error", candidateFail: true, wantError: "serialize extract candidate", attemptError: true},
|
||||
{name: "final codec error", finalFail: true, wantError: "serialize accepted extract output", attemptError: true},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
prepared.Steps[0].lanes[0].resolved.ExtractValidationPolicy = ValidationPolicy{
|
||||
ProducerStructuralFailure: ProducerStructuralFailureFailRun,
|
||||
SemanticRejection: SemanticRejectionRejectOutput,
|
||||
ValidatorFailure: ValidatorFailureFailRun,
|
||||
}
|
||||
value := "extract-terminal"
|
||||
codec := &observedNotesCodec{}
|
||||
if tc.candidateFail {
|
||||
@@ -263,7 +341,7 @@ func TestRunnerRecordsExtractTerminalOutcomes(t *testing.T) {
|
||||
prepared.Steps[0].lanes[0].extractValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("terminal/extract-validator"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: !tc.reject, ReasonCode: "extract_rejected", Message: "extract rejected"}, tc.validatorErr
|
||||
return contracts.ValidationResult{Approved: !tc.reject, ReasonCode: "extract_rejected", Message: "extract rejected", CorrectionGuidance: "return an acceptable extract"}, tc.validatorErr
|
||||
},
|
||||
}}
|
||||
debug := newCapturedDebugRecorder()
|
||||
@@ -273,9 +351,15 @@ func TestRunnerRecordsExtractTerminalOutcomes(t *testing.T) {
|
||||
attemptPath := "extract/notes/chunk-000001/attempt-01.json"
|
||||
envelope := debug.envelope(t, attemptPath)
|
||||
if tc.wantError != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tc.wantError) || !strings.Contains(envelope.Error, tc.wantError) {
|
||||
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
|
||||
t.Fatalf("Run() error = %v, attempt error = %q; want %q", err, envelope.Error, tc.wantError)
|
||||
}
|
||||
if tc.attemptError && !strings.Contains(envelope.Error, tc.wantError) {
|
||||
t.Fatalf("attempt error = %q, want %q", envelope.Error, tc.wantError)
|
||||
}
|
||||
if !tc.attemptError && envelope.Error != "" {
|
||||
t.Fatalf("settled validator failure attempt error = %q, want empty", envelope.Error)
|
||||
}
|
||||
} else {
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v, want nil rejection", err)
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"reflect"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/fileio"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
@@ -188,9 +189,24 @@ func (e *laneRunError) Error() string { return e.err.Error() }
|
||||
func (e *laneRunError) Unwrap() error { return e.err }
|
||||
|
||||
type mergeStageResult struct {
|
||||
artifact erasedMergeArtifact
|
||||
serialized CheckpointArtifact
|
||||
terminal bool
|
||||
artifact erasedMergeArtifact
|
||||
serialized CheckpointArtifact
|
||||
terminal bool
|
||||
validationIncomplete bool
|
||||
reuseEligible bool
|
||||
}
|
||||
|
||||
type mergeAttemptValue struct {
|
||||
artifact erasedMergeArtifact
|
||||
candidate CheckpointArtifact
|
||||
terminal *attemptTerminalRecorder
|
||||
}
|
||||
|
||||
type normalizeAttemptValue struct {
|
||||
value any
|
||||
candidate CheckpointArtifact
|
||||
terminal *attemptTerminalRecorder
|
||||
retry map[string]any
|
||||
}
|
||||
|
||||
func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, extracts finalizedExtractResults, output *RunOutput) error {
|
||||
@@ -208,12 +224,16 @@ func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoi
|
||||
if merged.terminal {
|
||||
return nil
|
||||
}
|
||||
normalized, err := r.runNormalizeStage(ctx, input, checkpoints, loader, doc, sourceInput, sessionID, prepared, merged.artifact, merged.serialized, output)
|
||||
normalized, err := r.runNormalizeStage(ctx, input, checkpoints, loader, doc, sourceInput, sessionID, prepared, merged.artifact, merged.serialized, merged.reuseEligible, output)
|
||||
if err != nil {
|
||||
return &laneRunError{stage: StageNormalize, err: err}
|
||||
}
|
||||
if normalized.accepted {
|
||||
output.NormalizeOutputs = append(output.NormalizeOutputs, contracts.SerializedOutput{StepID: input.stepID, LaneID: lane.ID, NormalizerKey: lane.Normalize.Module, SourceID: doc.ID, Artifact: contracts.CloneSerializedArtifact(normalized.serialized.Artifact)})
|
||||
if output.normalizeReuseEligibility == nil {
|
||||
output.normalizeReuseEligibility = make(map[generatedOutputKey]bool)
|
||||
}
|
||||
output.normalizeReuseEligibility[generatedOutputKeyFor(input.stepID, lane.ID)] = normalized.reuseEligible
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -226,9 +246,14 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
|
||||
mergeInputs[i] = contracts.ExtractArtifact[any]{LaneID: value.LaneID, ExtractorKey: value.ExtractorKey, SourceID: value.SourceID, ChunkID: value.ChunkID, ChunkIndex: value.ChunkIndex, ChunkRef: value.ChunkRef, Value: value.Value}
|
||||
}
|
||||
mergeReferences := operationReferenceSet(input, lane.MergeReferences)
|
||||
stageResult.reuseEligible = extracts.reuseEligible && referenceTargetReuseEligible(input, lane.MergeReferences)
|
||||
mergeDeps := append(artifactCheckpointDigests(extracts.serialized), generatedReferenceDependencies(mergeReferences)...)
|
||||
mergeDeps = normalizeCheckpointFingerprints(mergeDeps)
|
||||
mergeCP, mergeDecision := loadMerge(loader, input.stepID, lane.ID, lane.Merge.Module, mergeDeps)
|
||||
var mergeCP MergeCheckpoint
|
||||
mergeDecision := checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonValidationIncompleteLineage)
|
||||
if stageResult.reuseEligible {
|
||||
mergeCP, mergeDecision = loadMerge(loader, input.stepID, lane.ID, lane.Merge.Module, mergeDeps)
|
||||
}
|
||||
mergeResolution, err := resolveCheckpointDecision(output, loader, input.CheckpointPolicy, StageMerge, input.stepID, lane.ID, lane.Merge.Module, mergeDecision, typed.codec, []CheckpointArtifact{mergeCP.Output})
|
||||
if err != nil {
|
||||
return stageResult, err
|
||||
@@ -246,70 +271,106 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
|
||||
mergeWarnings = cloneWarnings(mergeCP.Warnings)
|
||||
output.Warnings = append(output.Warnings, mergeWarnings...)
|
||||
} else {
|
||||
if err := checkpointMergeRunning(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps); err != nil {
|
||||
return stageResult, err
|
||||
if stageResult.reuseEligible {
|
||||
if err := checkpointMergeRunning(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
}
|
||||
retryResult, runErr := runWithRetry(ctx, lane.Merge.Retries, func(attempt int) (retryAttemptResult, error) {
|
||||
terminalResult, runErr := runProducerAttempts(ctx, producerAttemptConfig{Retries: lane.Merge.Retries, Policy: lane.MergeValidationPolicy, AllowStructuralRetry: lane.MergeExecutionClass == contracts.ExecutionClassLLMBacked}, func(attemptCtx context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
started := time.Now().UTC()
|
||||
attemptPath := path.Join("merge", fileio.EncodePathComponent(lane.ID), fmt.Sprintf("attempt-%02d", attempt))
|
||||
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "merge", llmScope, debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, Attempt: attempt, StartedAt: started})
|
||||
attemptPath := path.Join("merge", fileio.EncodePathComponent(lane.ID), fmt.Sprintf("attempt-%02d", request.Number))
|
||||
attemptCtx, llmScope := withDebugLLMScope(attemptCtx, attemptPath)
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "merge", llmScope, debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, Attempt: request.Number, AttemptKind: string(request.Kind), StartedAt: started})
|
||||
requestMetadata, metadataErr := cloneMetadata(input.Metadata)
|
||||
if metadataErr != nil {
|
||||
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("clone merge request metadata: %w", metadataErr))
|
||||
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("clone merge request metadata: %w", metadataErr))
|
||||
}
|
||||
result, callErr := typed.merge(attemptCtx, typed.merger, contracts.TypedMergeRequest[any]{Source: doc, LaneID: lane.ID, ExtractOutputs: mergeInputs, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(mergeReferences), LLMProfile: lane.Merge.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(lane.Merge.StructuredOutputRepairAttempts), Metadata: requestMetadata})
|
||||
result, callErr := typed.merge(attemptCtx, typed.merger, contracts.TypedMergeRequest[any]{Source: doc, LaneID: lane.ID, ExtractOutputs: mergeInputs, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(mergeReferences), LLMProfile: lane.Merge.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(lane.Merge.StructuredOutputRepairAttempts), Correction: request.Correction, Metadata: requestMetadata})
|
||||
if callErr != nil {
|
||||
attemptErr := fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, lane.Merge.Module, callErr)
|
||||
return retryAttemptResult{}, terminal.record(nil, attemptErr)
|
||||
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, lane.Merge.Module, callErr))
|
||||
}
|
||||
candidate := erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: result.Value}
|
||||
attemptWarnings := cloneWarnings(result.Warnings)
|
||||
warnings := cloneWarnings(result.Warnings)
|
||||
serializedCandidate, encodeErr := serializeCandidateArtifact(typed.codec, candidate.LaneID, candidate.MergerKey, candidate.SourceID, candidate.Value)
|
||||
if encodeErr != nil {
|
||||
attemptErr := fmt.Errorf("serialize merge candidate for lane %q: %w", lane.ID, encodeErr)
|
||||
return retryAttemptResult{}, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}, attemptErr)
|
||||
return producerAttemptOutput{}, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(warnings)}, fmt.Errorf("serialize merge candidate for lane %q: %w", lane.ID, encodeErr))
|
||||
}
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageMerge, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: mergeReferences, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.mergeValidators, attempt, input.Debug)
|
||||
attemptWarnings = append(attemptWarnings, warnings...)
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
|
||||
if validateErr != nil {
|
||||
return retryAttemptResult{}, terminal.record(payload, validateErr)
|
||||
return producerAttemptOutput{Value: mergeAttemptValue{artifact: candidate, candidate: serializedCandidate, terminal: &terminal}, Candidate: result.ModelCandidate, Warnings: warnings}, nil
|
||||
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||
candidate, ok := output.Value.(mergeAttemptValue)
|
||||
if !ok {
|
||||
return validationReport{}, fmt.Errorf("merge attempt has incompatible value")
|
||||
}
|
||||
if rejected != nil {
|
||||
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
||||
return retryAttemptResult{}, debugErr
|
||||
}
|
||||
return retryAttemptResult{rejection: rejected, warnings: attemptWarnings}, nil
|
||||
target := typedValidationTarget{stage: StageMerge, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: mergeReferences, metadata: input.Metadata, value: candidate.artifact.Value, candidate: &candidate.candidate}
|
||||
report, validationErr := r.validateTypedReport(validationCtx, typed.codec, target, prepared.mergeValidators, candidate.terminal.envelope.Attempt, input.Debug)
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "warnings": debugWarningEnvelopes(append(cloneWarnings(output.Warnings), report.Warnings()...)), "rejection": debugRejectedOutputPtr(typedRejection(report, target, candidate.terminal.envelope.Attempt))}
|
||||
if validationErr != nil {
|
||||
return report, candidate.terminal.record(payload, validationErr)
|
||||
}
|
||||
stored, encodeErr := checkpointArtifact(typed.codec, candidate.LaneID, candidate.MergerKey, candidate.SourceID, candidate.Value)
|
||||
if encodeErr != nil {
|
||||
attemptErr := fmt.Errorf("serialize accepted merge output for lane %q: %w", lane.ID, encodeErr)
|
||||
return retryAttemptResult{}, terminal.record(payload, attemptErr)
|
||||
}
|
||||
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
||||
return retryAttemptResult{}, debugErr
|
||||
}
|
||||
merged, serializedMerge = candidate, stored
|
||||
mergeWarnings = attemptWarnings
|
||||
return retryAttemptResult{accepted: true}, nil
|
||||
return report, candidate.terminal.record(payload, nil)
|
||||
})
|
||||
terminalSummary := validationSummary(terminalResult, StageMerge, input.stepID, lane.ID, lane.Merge.Module, "", 0)
|
||||
if debugErr := writeProducerTerminalDebug(input.Debug, path.Join("merge", fileio.EncodePathComponent(lane.ID), "terminal.json"), terminalResult, lane.MergeValidationPolicy, terminalSummary); debugErr != nil {
|
||||
if stageResult.reuseEligible {
|
||||
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, debugErr)
|
||||
}
|
||||
return stageResult, debugErr
|
||||
}
|
||||
if runErr != nil {
|
||||
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, runErr)
|
||||
if stageResult.reuseEligible {
|
||||
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, runErr)
|
||||
}
|
||||
return stageResult, runErr
|
||||
}
|
||||
if !retryResult.accepted {
|
||||
output.Warnings = append(output.Warnings, retryResult.warnings...)
|
||||
output.Rejected = append(output.Rejected, *retryResult.rejection)
|
||||
if err := checkpointMergeRejected(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, *retryResult.rejection); err != nil {
|
||||
return stageResult, err
|
||||
if terminalResult.Action == producerTerminalRejected {
|
||||
rejected := terminalResult.Rejection
|
||||
if rejected == nil {
|
||||
return stageResult, fmt.Errorf("merge attempt terminal is missing rejection")
|
||||
}
|
||||
rejected.Stage, rejected.StepID, rejected.LaneID, rejected.ModuleKey = string(StageMerge), input.stepID, lane.ID, lane.Merge.Module
|
||||
rejected.Validation = &terminalSummary
|
||||
output.ValidationSummaries = append(output.ValidationSummaries, artifacts.CloneValidationSummary(terminalSummary))
|
||||
output.Warnings = append(output.Warnings, terminalResult.Warnings...)
|
||||
output.Rejected = append(output.Rejected, *rejected)
|
||||
if stageResult.reuseEligible {
|
||||
if err := checkpointMergeRejected(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, *rejected); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
}
|
||||
stageResult.terminal = true
|
||||
return stageResult, nil
|
||||
}
|
||||
candidate, ok := terminalResult.Value.(mergeAttemptValue)
|
||||
if !ok {
|
||||
return stageResult, fmt.Errorf("merge attempt terminal has incompatible value")
|
||||
}
|
||||
stored, encodeErr := checkpointArtifact(typed.codec, candidate.artifact.LaneID, candidate.artifact.MergerKey, candidate.artifact.SourceID, candidate.artifact.Value)
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "warnings": debugWarningEnvelopes(terminalResult.Warnings), "rejection": debugRejectedOutputPtr(nil)}
|
||||
if encodeErr != nil {
|
||||
attemptErr := fmt.Errorf("serialize accepted merge output for lane %q: %w", lane.ID, encodeErr)
|
||||
if stageResult.reuseEligible {
|
||||
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, attemptErr)
|
||||
}
|
||||
return stageResult, candidate.terminal.record(payload, attemptErr)
|
||||
}
|
||||
if debugErr := candidate.terminal.record(payload, nil); debugErr != nil {
|
||||
if stageResult.reuseEligible {
|
||||
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, debugErr)
|
||||
}
|
||||
return stageResult, debugErr
|
||||
}
|
||||
merged, serializedMerge = candidate.artifact, stored
|
||||
mergeWarnings = cloneWarnings(terminalResult.Warnings)
|
||||
output.ValidationSummaries = append(output.ValidationSummaries, terminalSummary)
|
||||
stageResult.validationIncomplete = terminalResult.ValidationIncomplete
|
||||
if stageResult.validationIncomplete {
|
||||
stageResult.reuseEligible = false
|
||||
}
|
||||
output.Warnings = append(output.Warnings, mergeWarnings...)
|
||||
if err := recordMerge(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge, mergeWarnings); err != nil {
|
||||
return stageResult, err
|
||||
if stageResult.reuseEligible {
|
||||
if err := recordMerge(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge, mergeWarnings); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := writeDebugTimed(input.Debug, path.Join("merge", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedMerge), "warnings": debugWarningEnvelopes(mergeWarnings)}}); err != nil {
|
||||
@@ -321,18 +382,24 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
|
||||
}
|
||||
|
||||
type normalizeStageResult struct {
|
||||
serialized CheckpointArtifact
|
||||
warnings []contracts.Warning
|
||||
accepted bool
|
||||
serialized CheckpointArtifact
|
||||
warnings []contracts.Warning
|
||||
accepted bool
|
||||
reuseEligible bool
|
||||
}
|
||||
|
||||
func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, merged erasedMergeArtifact, serializedMerge CheckpointArtifact, output *RunOutput) (normalizeStageResult, error) {
|
||||
func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, merged erasedMergeArtifact, serializedMerge CheckpointArtifact, upstreamReuseEligible bool, output *RunOutput) (normalizeStageResult, error) {
|
||||
var stageResult normalizeStageResult
|
||||
lane, typed := prepared.resolved, prepared.typed
|
||||
normalizeReferences := operationReferenceSet(input, lane.NormalizeReferences)
|
||||
stageResult.reuseEligible = upstreamReuseEligible && referenceTargetReuseEligible(input, lane.NormalizeReferences)
|
||||
normalizeDeps := append(artifactCheckpointDigests([]CheckpointArtifact{serializedMerge}), generatedReferenceDependencies(normalizeReferences)...)
|
||||
normalizeDeps = normalizeCheckpointFingerprints(normalizeDeps)
|
||||
normalizeCP, normalizeDecision := loadNormalize(loader, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps)
|
||||
var normalizeCP NormalizeCheckpoint
|
||||
normalizeDecision := checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonValidationIncompleteLineage)
|
||||
if stageResult.reuseEligible {
|
||||
normalizeCP, normalizeDecision = loadNormalize(loader, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps)
|
||||
}
|
||||
normalizeResolution, err := resolveCheckpointDecision(output, loader, input.CheckpointPolicy, StageNormalize, input.stepID, lane.ID, lane.Normalize.Module, normalizeDecision, typed.codec, []CheckpointArtifact{normalizeCP.Output})
|
||||
if err != nil {
|
||||
return stageResult, err
|
||||
@@ -348,89 +415,125 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
|
||||
normalizeWarnings = cloneWarnings(normalizeCP.Warnings)
|
||||
output.Warnings = append(output.Warnings, normalizeWarnings...)
|
||||
} else {
|
||||
if err := checkpointNormalizeRunning(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps); err != nil {
|
||||
return stageResult, err
|
||||
if stageResult.reuseEligible {
|
||||
if err := checkpointNormalizeRunning(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
}
|
||||
retryResult, runErr := runWithRetry(ctx, lane.Normalize.Retries, func(attempt int) (retryAttemptResult, error) {
|
||||
terminalResult, runErr := runProducerAttempts(ctx, producerAttemptConfig{Retries: lane.Normalize.Retries, Policy: lane.NormalizeValidationPolicy, AllowStructuralRetry: lane.NormalizeExecutionClass == contracts.ExecutionClassLLMBacked}, func(attemptCtx context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||
started := time.Now().UTC()
|
||||
attemptPath := path.Join("normalize", fileio.EncodePathComponent(lane.ID), fmt.Sprintf("attempt-%02d", attempt))
|
||||
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "normalize", llmScope, debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, Attempt: attempt, StartedAt: started})
|
||||
attemptPath := path.Join("normalize", fileio.EncodePathComponent(lane.ID), fmt.Sprintf("attempt-%02d", request.Number))
|
||||
attemptCtx, llmScope := withDebugLLMScope(attemptCtx, attemptPath)
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "normalize", llmScope, debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, Attempt: request.Number, AttemptKind: string(request.Kind), StartedAt: started})
|
||||
requestMetadata, metadataErr := cloneMetadata(input.Metadata)
|
||||
if metadataErr != nil {
|
||||
return retryAttemptResult{}, terminal.record(nil, fmt.Errorf("clone normalize request metadata: %w", metadataErr))
|
||||
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("clone normalize request metadata: %w", metadataErr))
|
||||
}
|
||||
result, callErr := typed.normalize(attemptCtx, typed.normalizer, contracts.TypedNormalizeRequest[any]{Source: doc, LaneID: lane.ID, MergeOutput: contracts.MergeArtifact[any]{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: merged.Value}, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(normalizeReferences), LLMProfile: lane.Normalize.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(lane.Normalize.StructuredOutputRepairAttempts), Metadata: requestMetadata})
|
||||
result, callErr := typed.normalize(attemptCtx, typed.normalizer, contracts.TypedNormalizeRequest[any]{Source: doc, LaneID: lane.ID, MergeOutput: contracts.MergeArtifact[any]{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: merged.Value}, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(normalizeReferences), LLMProfile: lane.Normalize.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(lane.Normalize.StructuredOutputRepairAttempts), Correction: request.Correction, Metadata: requestMetadata})
|
||||
if callErr != nil {
|
||||
attemptErr := fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, lane.Normalize.Module, callErr)
|
||||
return retryAttemptResult{}, terminal.record(nil, attemptErr)
|
||||
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, lane.Normalize.Module, callErr))
|
||||
}
|
||||
attemptWarnings := cloneWarnings(result.Warnings)
|
||||
warnings := cloneWarnings(result.Warnings)
|
||||
serializedCandidate, encodeErr := serializeCandidateArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, result.Value)
|
||||
if encodeErr != nil {
|
||||
attemptErr := fmt.Errorf("serialize normalize candidate for lane %q: %w", lane.ID, encodeErr)
|
||||
return retryAttemptResult{}, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}, attemptErr)
|
||||
return producerAttemptOutput{}, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(warnings)}, fmt.Errorf("serialize normalize candidate for lane %q: %w", lane.ID, encodeErr))
|
||||
}
|
||||
var retryPayload map[string]any
|
||||
attemptValue := normalizeAttemptValue{value: result.Value, candidate: serializedCandidate, terminal: &terminal}
|
||||
var directive *producerRetryDirective
|
||||
if result.Retry != nil {
|
||||
if err := validateNormalizeRetry(result.Retry); err != nil {
|
||||
return retryAttemptResult{}, terminal.record(map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings)}, fmt.Errorf("normalize lane %q returned invalid retry directive: %w", lane.ID, err))
|
||||
return producerAttemptOutput{}, terminal.record(map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(warnings)}, fmt.Errorf("normalize lane %q returned invalid retry directive: %w", lane.ID, err))
|
||||
}
|
||||
retryRemaining := attempt <= lane.Normalize.Retries
|
||||
retryPayload = map[string]any{
|
||||
"reason_code": result.Retry.ReasonCode,
|
||||
"message": result.Retry.Message,
|
||||
"another_attempt": retryRemaining,
|
||||
"fallback_accepted": !retryRemaining,
|
||||
anotherAttempt := request.Number <= lane.Normalize.Retries
|
||||
attemptValue.retry = map[string]any{"reason_code": result.Retry.ReasonCode, "message": result.Retry.Message, "another_attempt": anotherAttempt, "fallback_accepted": !anotherAttempt}
|
||||
directive = &producerRetryDirective{FallbackWarnings: cloneWarnings(result.Retry.FallbackWarnings)}
|
||||
if anotherAttempt {
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(warnings), "retry": attemptValue.retry}
|
||||
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
||||
return producerAttemptOutput{}, debugErr
|
||||
}
|
||||
}
|
||||
if retryRemaining {
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "retry": retryPayload}
|
||||
return retryAttemptResult{}, terminal.record(payload, nil)
|
||||
}
|
||||
attemptWarnings = append(attemptWarnings, cloneWarnings(result.Retry.FallbackWarnings)...)
|
||||
}
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageNormalize, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: normalizeReferences, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.normalizeValidators, attempt, input.Debug)
|
||||
attemptWarnings = append(attemptWarnings, warnings...)
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
|
||||
if retryPayload != nil {
|
||||
payload["retry"] = retryPayload
|
||||
return producerAttemptOutput{Value: attemptValue, Candidate: result.ModelCandidate, Warnings: warnings, Retry: directive}, nil
|
||||
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||
candidate, ok := output.Value.(normalizeAttemptValue)
|
||||
if !ok {
|
||||
return validationReport{}, fmt.Errorf("normalize attempt has incompatible value")
|
||||
}
|
||||
if validateErr != nil {
|
||||
return retryAttemptResult{}, terminal.record(payload, validateErr)
|
||||
target := typedValidationTarget{stage: StageNormalize, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: normalizeReferences, metadata: input.Metadata, value: candidate.value, candidate: &candidate.candidate}
|
||||
report, validationErr := r.validateTypedReport(validationCtx, typed.codec, target, prepared.normalizeValidators, candidate.terminal.envelope.Attempt, input.Debug)
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "warnings": debugWarningEnvelopes(append(cloneWarnings(output.Warnings), report.Warnings()...)), "rejection": debugRejectedOutputPtr(typedRejection(report, target, candidate.terminal.envelope.Attempt))}
|
||||
if candidate.retry != nil {
|
||||
payload["retry"] = candidate.retry
|
||||
}
|
||||
if rejected != nil {
|
||||
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
||||
return retryAttemptResult{}, debugErr
|
||||
}
|
||||
return retryAttemptResult{rejection: rejected, warnings: attemptWarnings}, nil
|
||||
if validationErr != nil {
|
||||
return report, candidate.terminal.record(payload, validationErr)
|
||||
}
|
||||
stored, encodeErr := checkpointArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, result.Value)
|
||||
if encodeErr != nil {
|
||||
attemptErr := fmt.Errorf("serialize accepted normalize output for lane %q: %w", lane.ID, encodeErr)
|
||||
return retryAttemptResult{}, terminal.record(payload, attemptErr)
|
||||
}
|
||||
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
||||
return retryAttemptResult{}, debugErr
|
||||
}
|
||||
serializedNormalize = stored
|
||||
normalizeWarnings = attemptWarnings
|
||||
return retryAttemptResult{accepted: true}, nil
|
||||
return report, candidate.terminal.record(payload, nil)
|
||||
})
|
||||
terminalSummary := validationSummary(terminalResult, StageNormalize, input.stepID, lane.ID, lane.Normalize.Module, "", 0)
|
||||
if debugErr := writeProducerTerminalDebug(input.Debug, path.Join("normalize", fileio.EncodePathComponent(lane.ID), "terminal.json"), terminalResult, lane.NormalizeValidationPolicy, terminalSummary); debugErr != nil {
|
||||
if stageResult.reuseEligible {
|
||||
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, debugErr)
|
||||
}
|
||||
return stageResult, debugErr
|
||||
}
|
||||
if runErr != nil {
|
||||
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, runErr)
|
||||
if stageResult.reuseEligible {
|
||||
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, runErr)
|
||||
}
|
||||
return stageResult, runErr
|
||||
}
|
||||
if !retryResult.accepted {
|
||||
output.Warnings = append(output.Warnings, retryResult.warnings...)
|
||||
output.Rejected = append(output.Rejected, *retryResult.rejection)
|
||||
if err := checkpointNormalizeRejected(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, *retryResult.rejection); err != nil {
|
||||
return stageResult, err
|
||||
if terminalResult.Action == producerTerminalRejected {
|
||||
rejected := terminalResult.Rejection
|
||||
if rejected == nil {
|
||||
return stageResult, fmt.Errorf("normalize attempt terminal is missing rejection")
|
||||
}
|
||||
rejected.Stage, rejected.StepID, rejected.LaneID, rejected.ModuleKey = string(StageNormalize), input.stepID, lane.ID, lane.Normalize.Module
|
||||
rejected.Validation = &terminalSummary
|
||||
output.ValidationSummaries = append(output.ValidationSummaries, artifacts.CloneValidationSummary(terminalSummary))
|
||||
output.Warnings = append(output.Warnings, terminalResult.Warnings...)
|
||||
output.Rejected = append(output.Rejected, *rejected)
|
||||
if stageResult.reuseEligible {
|
||||
if err := checkpointNormalizeRejected(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, *rejected); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
}
|
||||
return stageResult, nil
|
||||
}
|
||||
candidate, ok := terminalResult.Value.(normalizeAttemptValue)
|
||||
if !ok {
|
||||
return stageResult, fmt.Errorf("normalize attempt terminal has incompatible value")
|
||||
}
|
||||
stored, encodeErr := checkpointArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, candidate.value)
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "warnings": debugWarningEnvelopes(terminalResult.Warnings), "rejection": debugRejectedOutputPtr(nil)}
|
||||
if candidate.retry != nil {
|
||||
payload["retry"] = candidate.retry
|
||||
}
|
||||
if encodeErr != nil {
|
||||
attemptErr := fmt.Errorf("serialize accepted normalize output for lane %q: %w", lane.ID, encodeErr)
|
||||
if stageResult.reuseEligible {
|
||||
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, attemptErr)
|
||||
}
|
||||
return stageResult, candidate.terminal.record(payload, attemptErr)
|
||||
}
|
||||
if debugErr := candidate.terminal.record(payload, nil); debugErr != nil {
|
||||
if stageResult.reuseEligible {
|
||||
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, debugErr)
|
||||
}
|
||||
return stageResult, debugErr
|
||||
}
|
||||
serializedNormalize = stored
|
||||
normalizeWarnings = cloneWarnings(terminalResult.Warnings)
|
||||
output.ValidationSummaries = append(output.ValidationSummaries, terminalSummary)
|
||||
output.Warnings = append(output.Warnings, normalizeWarnings...)
|
||||
if err := recordNormalize(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize, normalizeWarnings); err != nil {
|
||||
return stageResult, err
|
||||
if terminalResult.ValidationIncomplete {
|
||||
stageResult.reuseEligible = false
|
||||
}
|
||||
if stageResult.reuseEligible {
|
||||
if err := recordNormalize(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize, normalizeWarnings); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := writeDebugTimed(input.Debug, path.Join("normalize", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedNormalize), "warnings": debugWarningEnvelopes(normalizeWarnings)}}); err != nil {
|
||||
@@ -471,59 +574,56 @@ func setTypedLaneManifestMetadata(output *RunOutput, laneID string, extractor, m
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *Runner) validateTypedArtifact(ctx context.Context, codec artifactCodecEntry, target typedValidationTarget, chain preparedValidatorChain, attempt int, debug DebugRecorder) ([]contracts.Warning, *contracts.RejectedOutput, error) {
|
||||
var warnings []contracts.Warning
|
||||
func (r *Runner) validateTypedReport(ctx context.Context, codec artifactCodecEntry, target typedValidationTarget, chain preparedValidatorChain, attempt int, debug DebugRecorder) (validationReport, error) {
|
||||
if len(chain.validators) > 0 && target.candidate == nil {
|
||||
candidate, err := validationCandidateArtifact(codec, target)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("serialize %s candidate for validation: %w", target.stage, err)
|
||||
return validationReport{}, fmt.Errorf("serialize %s candidate for validation: %w", target.stage, err)
|
||||
}
|
||||
target.candidate = &candidate
|
||||
}
|
||||
for index, item := range chain.validators {
|
||||
report, err := executeValidationChain(ctx, chain, func(validatorCtx context.Context, item preparedValidator, validatorAttempt int) (validationInvocation, error) {
|
||||
binding := item.resolved.Binding
|
||||
var result contracts.ValidationResult
|
||||
var err error
|
||||
started := time.Now().UTC()
|
||||
attemptPath := path.Join("validate", fileio.EncodePathComponent(string(target.stage)), fileio.EncodePathComponent(target.laneID), fileio.EncodePathComponent(target.moduleKey), fmt.Sprintf("%02d-%s-attempt-%02d", index+1, fileio.EncodePathComponent(binding.Module), attempt))
|
||||
validatorCtx, llmScope := withIsolatedDebugLLMScope(ctx, attemptPath)
|
||||
attemptPath := validatorAttemptPath(path.Join("validate", fileio.EncodePathComponent(string(target.stage)), fileio.EncodePathComponent(target.laneID), fileio.EncodePathComponent(target.moduleKey), fmt.Sprintf("%02d-%s-attempt-%02d", item.position, fileio.EncodePathComponent(binding.Module), attempt)), validatorAttempt)
|
||||
validatorCtx, llmScope := withIsolatedDebugLLMScope(validatorCtx, attemptPath)
|
||||
requestTarget := target
|
||||
requestTarget.sourceInput = target.sourceInput.Clone()
|
||||
requestTarget.references = CloneReferenceSet(target.references)
|
||||
requestTarget.metadata, err = cloneMetadata(target.metadata)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("clone typed validation metadata: %w", err)
|
||||
return validationInvocation{}, fatalValidationError(fmt.Errorf("clone typed validation metadata: %w", err))
|
||||
}
|
||||
requestTarget.chunk, err = cloneSourceChunkPtr(target.chunk)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("clone typed validation chunk: %w", err)
|
||||
return validationInvocation{}, fatalValidationError(fmt.Errorf("clone typed validation chunk: %w", err))
|
||||
}
|
||||
requestTarget.chunks, err = cloneSourceChunks(target.chunks)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("clone typed validation chunks: %w", err)
|
||||
return validationInvocation{}, fatalValidationError(fmt.Errorf("clone typed validation chunks: %w", err))
|
||||
}
|
||||
switch item.resolved.Target {
|
||||
case ValidatorTargetTyped:
|
||||
candidateValue, decodeErr := decodeTypedValidationCandidate(codec, *target.candidate)
|
||||
if decodeErr != nil {
|
||||
err = fmt.Errorf("decode %s candidate for typed validator %q: %w", target.stage, binding.Module, decodeErr)
|
||||
break
|
||||
return validationInvocation{}, fatalValidationError(fmt.Errorf("decode %s candidate for typed validator %q: %w", target.stage, binding.Module, decodeErr))
|
||||
}
|
||||
requestTarget.value = candidateValue
|
||||
requestTarget.llmProfile = binding.LLMProfile
|
||||
requestTarget.structuredOutputRepairAttempts = cloneStructuredOutputRepairAttempts(binding.StructuredOutputRepairAttempts)
|
||||
result, err = item.typedValidate(validatorCtx, item.typed, requestTarget)
|
||||
case ValidatorTargetSerialized:
|
||||
artifact, encodeErr := validationCandidateArtifact(codec, target)
|
||||
if encodeErr != nil {
|
||||
err = encodeErr
|
||||
break
|
||||
}
|
||||
artifact := cloneCheckpointArtifact(*target.candidate)
|
||||
result, err = item.serialized.Validate(validatorCtx, contracts.SerializedValidationRequest{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Source: target.source, SourceID: target.sourceID, SourceInput: requestTarget.sourceInput, SessionID: target.sessionID, References: requestTarget.references, LLMProfile: binding.LLMProfile, StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(binding.StructuredOutputRepairAttempts), Metadata: requestTarget.metadata, Chunk: requestTarget.chunk, Chunks: requestTarget.chunks, Schema: contracts.CloneArtifactSchema(artifact.Artifact.Schema), MediaType: artifact.Artifact.MediaType, Content: append([]byte(nil), artifact.Artifact.Content...)})
|
||||
default:
|
||||
return nil, nil, fmt.Errorf("validator %q is incompatible with typed artifact validation", binding.Module)
|
||||
return validationInvocation{}, fatalValidationError(fmt.Errorf("validator %q is incompatible with typed artifact validation", binding.Module))
|
||||
}
|
||||
artifact, _ := validationCandidateArtifact(codec, target)
|
||||
if err == nil {
|
||||
err = contracts.ValidateValidationResult(result)
|
||||
}
|
||||
artifact := cloneCheckpointArtifact(*target.candidate)
|
||||
debugCall := debugValidationCall{ValidatorName: binding.Module, Request: map[string]any{"stage": string(target.stage), "lane_id": target.laneID, "module_key": target.moduleKey, "source_id": target.sourceID, "artifact": debugCheckpointArtifact(artifact), "metadata": redactSensitiveMap(target.metadata)}, Result: debugValidationResultEnvelope(result)}
|
||||
if err != nil {
|
||||
debugCall.Error = err.Error()
|
||||
@@ -531,37 +631,31 @@ func (r *Runner) validateTypedArtifact(ctx context.Context, codec artifactCodecE
|
||||
if err != nil {
|
||||
validationErr := fmt.Errorf("validate typed %s output with validator %q: %w", target.stage, binding.Module, err)
|
||||
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall, Error: debugCall.Error}, llmScope); debugErr != nil {
|
||||
return warnings, nil, errors.Join(validationErr, fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr))
|
||||
return validationInvocation{}, fatalValidationError(errors.Join(validationErr, fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr)))
|
||||
}
|
||||
return warnings, nil, validationErr
|
||||
return validationInvocation{}, validationErr
|
||||
}
|
||||
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall}, llmScope); debugErr != nil {
|
||||
return warnings, nil, fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr)
|
||||
return validationInvocation{}, fatalValidationError(fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr))
|
||||
}
|
||||
if !result.Approved {
|
||||
reason := result.ReasonCode
|
||||
if reason == "" {
|
||||
reason = "artifact_rejected"
|
||||
}
|
||||
message := result.Message
|
||||
if message == "" {
|
||||
message = "artifact rejected"
|
||||
}
|
||||
return warnings, &contracts.RejectedOutput{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, ChunkID: func() string {
|
||||
if target.chunk != nil {
|
||||
return target.chunk.ID
|
||||
}
|
||||
return ""
|
||||
}(), ChunkIndex: func() int {
|
||||
if target.chunk != nil {
|
||||
return target.chunk.Index
|
||||
}
|
||||
return 0
|
||||
}(), ValidatorName: binding.Module, ReasonCode: reason, Message: message, AttemptCount: attempt, DiagnosticArtifactPath: result.DiagnosticArtifactPath}, nil
|
||||
}
|
||||
warnings = append(warnings, result.Warnings...)
|
||||
return validationInvocation{result: result}, nil
|
||||
})
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
return warnings, nil, nil
|
||||
return report, nil
|
||||
}
|
||||
|
||||
func typedRejection(report validationReport, target typedValidationTarget, attempt int) *contracts.RejectedOutput {
|
||||
rejection := report.FirstRejection()
|
||||
if rejection == nil {
|
||||
return nil
|
||||
}
|
||||
chunkID, chunkIndex := "", 0
|
||||
if target.chunk != nil {
|
||||
chunkID, chunkIndex = target.chunk.ID, target.chunk.Index
|
||||
}
|
||||
return &contracts.RejectedOutput{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, ChunkID: chunkID, ChunkIndex: chunkIndex, ValidatorName: rejection.validatorName, ReasonCode: rejection.reasonCode, Message: rejection.message, AttemptCount: attempt, DiagnosticArtifactPath: rejection.diagnosticPath}
|
||||
}
|
||||
|
||||
func validationCandidateArtifact(codec artifactCodecEntry, target typedValidationTarget) (CheckpointArtifact, error) {
|
||||
|
||||
@@ -22,9 +22,10 @@ type erasedMergeArtifact struct {
|
||||
}
|
||||
|
||||
type erasedTypedResult struct {
|
||||
Value any
|
||||
Warnings []contracts.Warning
|
||||
Retry *contracts.NormalizeRetry
|
||||
Value any
|
||||
Warnings []contracts.Warning
|
||||
Retry *contracts.NormalizeRetry
|
||||
ModelCandidate *contracts.ModelCandidate
|
||||
}
|
||||
|
||||
type typedValidationTarget struct {
|
||||
|
||||
@@ -1,12 +1,70 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type correctionObservingNotesExtractor struct {
|
||||
correction *contracts.SemanticCorrection
|
||||
candidate *contracts.ModelCandidate
|
||||
}
|
||||
|
||||
func (*correctionObservingNotesExtractor) Key() string { return "test/correction-observing-extract" }
|
||||
func (*correctionObservingNotesExtractor) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (e *correctionObservingNotesExtractor) Extract(_ context.Context, request contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[codecNotes], error) {
|
||||
e.correction = request.Correction
|
||||
candidate, err := contracts.NewModelCandidate([]byte(`{"items":["extract"]}`), contracts.CorrectionProtocolSingleResponseV1)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[codecNotes]{}, err
|
||||
}
|
||||
e.candidate = candidate
|
||||
return contracts.TypedExtractionResult[codecNotes]{Value: codecNotes{Items: []string{"extract"}}, ModelCandidate: candidate}, nil
|
||||
}
|
||||
|
||||
type correctionObservingNotesMerger struct {
|
||||
correction *contracts.SemanticCorrection
|
||||
candidate *contracts.ModelCandidate
|
||||
}
|
||||
|
||||
func (*correctionObservingNotesMerger) Key() string { return "test/correction-observing-merge" }
|
||||
|
||||
func (m *correctionObservingNotesMerger) Merge(_ context.Context, request contracts.TypedMergeRequest[codecNotes]) (contracts.TypedMergeResult[codecNotes], error) {
|
||||
m.correction = request.Correction
|
||||
candidate, err := contracts.NewModelCandidate([]byte(`{"items":["merge"]}`), contracts.CorrectionProtocolSingleResponseV1)
|
||||
if err != nil {
|
||||
return contracts.TypedMergeResult[codecNotes]{}, err
|
||||
}
|
||||
m.candidate = candidate
|
||||
return contracts.TypedMergeResult[codecNotes]{Value: codecNotes{Items: []string{"merge"}}, ModelCandidate: candidate}, nil
|
||||
}
|
||||
|
||||
type correctionObservingNotesNormalizer struct {
|
||||
correction *contracts.SemanticCorrection
|
||||
candidate *contracts.ModelCandidate
|
||||
}
|
||||
|
||||
func (*correctionObservingNotesNormalizer) Key() string { return "test/correction-observing-normalize" }
|
||||
func (*correctionObservingNotesNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (n *correctionObservingNotesNormalizer) Normalize(_ context.Context, request contracts.TypedNormalizeRequest[codecNotes]) (contracts.TypedNormalizeResult[codecNotes], error) {
|
||||
n.correction = request.Correction
|
||||
candidate, err := contracts.NewModelCandidate([]byte(`{"items":["normalize"]}`), contracts.CorrectionProtocolSingleResponseV1)
|
||||
if err != nil {
|
||||
return contracts.TypedNormalizeResult[codecNotes]{}, err
|
||||
}
|
||||
n.candidate = candidate
|
||||
return contracts.TypedNormalizeResult[codecNotes]{Value: codecNotes{Items: []string{"normalize"}}, ModelCandidate: candidate}, nil
|
||||
}
|
||||
|
||||
type repairObservingNotesMerger struct {
|
||||
attempts *int
|
||||
}
|
||||
@@ -93,3 +151,103 @@ func TestTypedRegistryErasurePreservesStructuredOutputRepairAttempts(t *testing.
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestTypedRegistryErasurePreservesCorrectionAndCandidateOwnership(t *testing.T) {
|
||||
const artifactKind contracts.ArtifactKind = "test/notes"
|
||||
|
||||
t.Run("extract", func(t *testing.T) {
|
||||
correction := newTestCorrection(t)
|
||||
implementation := &correctionObservingNotesExtractor{}
|
||||
registry := NewExtractorRegistry()
|
||||
if err := RegisterExtractor(registry, ModuleSpec{Key: implementation.Key(), Stage: StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: artifactKind}, func() (contracts.Extractor[codecNotes], error) {
|
||||
return implementation, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterExtractor() error = %v", err)
|
||||
}
|
||||
entry, ok := registry.typedEntry(implementation.Key())
|
||||
if !ok {
|
||||
t.Fatal("typed extractor entry missing")
|
||||
}
|
||||
built, err := entry.builder(BuildRequest{})
|
||||
if err != nil {
|
||||
t.Fatalf("builder() error = %v", err)
|
||||
}
|
||||
result, err := entry.extract(context.Background(), built, contracts.TypedExtractionRequest{Correction: correction})
|
||||
if err != nil {
|
||||
t.Fatalf("extract() error = %v", err)
|
||||
}
|
||||
assertCorrectionAndCandidateOwnership(t, correction, implementation.correction, implementation.candidate, result.ModelCandidate)
|
||||
})
|
||||
|
||||
t.Run("merge", func(t *testing.T) {
|
||||
correction := newTestCorrection(t)
|
||||
implementation := &correctionObservingNotesMerger{}
|
||||
registry := NewMergerRegistry()
|
||||
if err := RegisterMerger(registry, ModuleSpec{Key: implementation.Key(), Stage: StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: artifactKind}, func() (contracts.Merger[codecNotes], error) {
|
||||
return implementation, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterMerger() error = %v", err)
|
||||
}
|
||||
entry, ok := registry.typedEntry(implementation.Key(), artifactKind)
|
||||
if !ok {
|
||||
t.Fatal("typed merger entry missing")
|
||||
}
|
||||
built, err := entry.builder(BuildRequest{})
|
||||
if err != nil {
|
||||
t.Fatalf("builder() error = %v", err)
|
||||
}
|
||||
result, err := entry.merge(context.Background(), built, contracts.TypedMergeRequest[any]{Correction: correction, ExtractOutputs: []contracts.ExtractArtifact[any]{{Value: codecNotes{Items: []string{"extract"}}}}})
|
||||
if err != nil {
|
||||
t.Fatalf("merge() error = %v", err)
|
||||
}
|
||||
assertCorrectionAndCandidateOwnership(t, correction, implementation.correction, implementation.candidate, result.ModelCandidate)
|
||||
})
|
||||
|
||||
t.Run("normalize", func(t *testing.T) {
|
||||
correction := newTestCorrection(t)
|
||||
implementation := &correctionObservingNotesNormalizer{}
|
||||
registry := NewNormalizerRegistry()
|
||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: implementation.Key(), Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: artifactKind}, func() (contracts.Normalizer[codecNotes], error) {
|
||||
return implementation, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||
}
|
||||
entry, ok := registry.typedEntry(implementation.Key(), artifactKind)
|
||||
if !ok {
|
||||
t.Fatal("typed normalizer entry missing")
|
||||
}
|
||||
built, err := entry.builder(BuildRequest{})
|
||||
if err != nil {
|
||||
t.Fatalf("builder() error = %v", err)
|
||||
}
|
||||
result, err := entry.normalize(context.Background(), built, contracts.TypedNormalizeRequest[any]{Correction: correction, MergeOutput: contracts.MergeArtifact[any]{Value: codecNotes{Items: []string{"merge"}}}})
|
||||
if err != nil {
|
||||
t.Fatalf("normalize() error = %v", err)
|
||||
}
|
||||
assertCorrectionAndCandidateOwnership(t, correction, implementation.correction, implementation.candidate, result.ModelCandidate)
|
||||
})
|
||||
}
|
||||
|
||||
func newTestCorrection(t *testing.T) *contracts.SemanticCorrection {
|
||||
t.Helper()
|
||||
correction, err := contracts.NewSemanticCorrection([]byte(`{"items":["original"]}`), "Correct the response.")
|
||||
if err != nil {
|
||||
t.Fatalf("NewSemanticCorrection() error = %v", err)
|
||||
}
|
||||
return correction
|
||||
}
|
||||
|
||||
func assertCorrectionAndCandidateOwnership(t *testing.T, callerCorrection, observedCorrection *contracts.SemanticCorrection, producerCandidate, returnedCandidate *contracts.ModelCandidate) {
|
||||
t.Helper()
|
||||
if observedCorrection == nil || producerCandidate == nil || returnedCandidate == nil {
|
||||
t.Fatal("correction and candidates must be present")
|
||||
}
|
||||
callerCorrection.AssistantResponse[0] = '['
|
||||
if got := string(observedCorrection.AssistantResponse); got != `{"items":["original"]}` {
|
||||
t.Fatalf("observed correction = %q, want owned original content", got)
|
||||
}
|
||||
producerCandidate.Response[0] = '['
|
||||
if bytes.Equal(producerCandidate.Response, returnedCandidate.Response) {
|
||||
t.Fatalf("returned candidate aliases producer candidate: %q", returnedCandidate.Response)
|
||||
}
|
||||
}
|
||||
|
||||
216
internal/framework/pipeline/validation_executor.go
Normal file
216
internal/framework/pipeline/validation_executor.go
Normal file
@@ -0,0 +1,216 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type validationOutcome string
|
||||
|
||||
const (
|
||||
validationApproved validationOutcome = "approved"
|
||||
validationRejected validationOutcome = "rejected"
|
||||
validationFailed validationOutcome = "failed"
|
||||
validationSkipped validationOutcome = "skipped"
|
||||
)
|
||||
|
||||
const correctionRequestIntroduction = "The previous response failed semantic validation. Return one complete corrected replacement response, not a patch, explanation, or commentary.\n\nCorrect all of the following:\n"
|
||||
|
||||
// validationRecord captures the settled result of one configured validator.
|
||||
// Its fields remain private so reports cannot expose mutable warning storage.
|
||||
type validationRecord struct {
|
||||
validatorName string
|
||||
outcome validationOutcome
|
||||
attemptCount int
|
||||
reasonCode string
|
||||
message string
|
||||
diagnosticPath string
|
||||
warnings []contracts.Warning
|
||||
correctionGuidance string
|
||||
failure error
|
||||
}
|
||||
|
||||
func (record validationRecord) clone() validationRecord {
|
||||
record.warnings = cloneWarnings(record.warnings)
|
||||
return record
|
||||
}
|
||||
|
||||
// validationReport is the immutable, ordered result of one validator chain.
|
||||
type validationReport struct {
|
||||
records []validationRecord
|
||||
}
|
||||
|
||||
func (report validationReport) Records() []validationRecord {
|
||||
records := make([]validationRecord, len(report.records))
|
||||
for index, record := range report.records {
|
||||
records[index] = record.clone()
|
||||
}
|
||||
return records
|
||||
}
|
||||
|
||||
func (report validationReport) Warnings() []contracts.Warning {
|
||||
var warnings []contracts.Warning
|
||||
for _, record := range report.records {
|
||||
if record.outcome == validationApproved || record.outcome == validationRejected {
|
||||
warnings = append(warnings, cloneWarnings(record.warnings)...)
|
||||
}
|
||||
}
|
||||
return warnings
|
||||
}
|
||||
|
||||
func (report validationReport) FirstRejection() *validationRecord {
|
||||
for _, record := range report.records {
|
||||
if record.outcome == validationRejected {
|
||||
clone := record.clone()
|
||||
return &clone
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (report validationReport) FirstFailure() *validationRecord {
|
||||
for _, record := range report.records {
|
||||
if record.outcome == validationFailed {
|
||||
clone := record.clone()
|
||||
return &clone
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (report validationReport) CorrectionRequest() (string, error) {
|
||||
seen := make(map[string]struct{})
|
||||
parts := make([]string, 0, len(report.records))
|
||||
for _, record := range report.records {
|
||||
if record.outcome != validationRejected {
|
||||
continue
|
||||
}
|
||||
guidance := strings.TrimSpace(record.correctionGuidance)
|
||||
if guidance == "" {
|
||||
return "", fmt.Errorf("validator %q rejected output without correction guidance", record.validatorName)
|
||||
}
|
||||
if _, exists := seen[guidance]; exists {
|
||||
continue
|
||||
}
|
||||
seen[guidance] = struct{}{}
|
||||
parts = append(parts, guidance)
|
||||
}
|
||||
if len(parts) == 0 {
|
||||
return "", errors.New("validation report contains no correction guidance")
|
||||
}
|
||||
var builder strings.Builder
|
||||
builder.WriteString(correctionRequestIntroduction)
|
||||
for index, guidance := range parts {
|
||||
fmt.Fprintf(&builder, "%d. %s", index+1, guidance)
|
||||
if index+1 < len(parts) {
|
||||
builder.WriteByte('\n')
|
||||
}
|
||||
}
|
||||
request := builder.String()
|
||||
if len(request) > contracts.MaxCorrectionGuidanceBytes {
|
||||
return "", fmt.Errorf("aggregate correction request exceeds maximum length of %d bytes", contracts.MaxCorrectionGuidanceBytes)
|
||||
}
|
||||
return request, nil
|
||||
}
|
||||
|
||||
type validationInvocation struct {
|
||||
result contracts.ValidationResult
|
||||
skipped bool
|
||||
reason string
|
||||
message string
|
||||
}
|
||||
|
||||
func skippedValidation(reason, message string) validationInvocation {
|
||||
return validationInvocation{skipped: true, reason: strings.TrimSpace(reason), message: strings.TrimSpace(message)}
|
||||
}
|
||||
|
||||
type validationInvoker func(context.Context, preparedValidator, int) (validationInvocation, error)
|
||||
|
||||
type validationFrameworkError struct{ err error }
|
||||
|
||||
func (err validationFrameworkError) Error() string { return err.err.Error() }
|
||||
|
||||
func (err validationFrameworkError) Unwrap() error { return err.err }
|
||||
|
||||
func fatalValidationError(err error) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
return validationFrameworkError{err: err}
|
||||
}
|
||||
|
||||
func executeValidationChain(ctx context.Context, chain preparedValidatorChain, invoke validationInvoker) (validationReport, error) {
|
||||
if ctx == nil {
|
||||
return validationReport{}, errors.New("validator execution context must not be nil")
|
||||
}
|
||||
if invoke == nil {
|
||||
return validationReport{}, errors.New("validator invocation must not be nil")
|
||||
}
|
||||
report := validationReport{records: make([]validationRecord, 0, len(chain.validators))}
|
||||
for index, validator := range chain.validators {
|
||||
validator.position = index + 1
|
||||
if err := ctx.Err(); err != nil {
|
||||
return validationReport{}, err
|
||||
}
|
||||
binding := validator.resolved.Binding
|
||||
attemptLimit := 1
|
||||
if validator.resolved.ExecutionClass == contracts.ExecutionClassLLMBacked {
|
||||
attemptLimit += binding.Retries
|
||||
}
|
||||
for attempt := 1; attempt <= attemptLimit; attempt++ {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return validationReport{}, err
|
||||
}
|
||||
invocation, err := invoke(ctx, validator, attempt)
|
||||
if err != nil {
|
||||
var frameworkErr validationFrameworkError
|
||||
if errors.As(err, &frameworkErr) || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
return validationReport{}, err
|
||||
}
|
||||
if attempt == attemptLimit {
|
||||
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationFailed, attemptCount: attempt, message: "validator failed after retry exhaustion", failure: err})
|
||||
}
|
||||
continue
|
||||
}
|
||||
if invocation.skipped {
|
||||
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationSkipped, attemptCount: attempt, reasonCode: invocation.reason, message: invocation.message})
|
||||
break
|
||||
}
|
||||
if err := contracts.ValidateValidationResult(invocation.result); err != nil {
|
||||
if attempt == attemptLimit {
|
||||
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationFailed, attemptCount: attempt, message: "validator returned an invalid result", failure: err})
|
||||
}
|
||||
continue
|
||||
}
|
||||
if invocation.result.Approved {
|
||||
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationApproved, attemptCount: attempt, warnings: cloneWarnings(invocation.result.Warnings), diagnosticPath: invocation.result.DiagnosticArtifactPath})
|
||||
break
|
||||
}
|
||||
message := invocation.result.Message
|
||||
if message == "" {
|
||||
message = "output rejected"
|
||||
}
|
||||
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationRejected, attemptCount: attempt, reasonCode: invocation.result.ReasonCode, message: message, diagnosticPath: invocation.result.DiagnosticArtifactPath, warnings: cloneWarnings(invocation.result.Warnings), correctionGuidance: invocation.result.CorrectionGuidance})
|
||||
break
|
||||
}
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
|
||||
func validatorFailureError(record validationRecord) error {
|
||||
if record.failure != nil {
|
||||
return record.failure
|
||||
}
|
||||
return fmt.Errorf("validator %q failed after %d attempt(s)", record.validatorName, record.attemptCount)
|
||||
}
|
||||
|
||||
func validatorAttemptPath(base string, attempt int) string {
|
||||
if attempt == 1 {
|
||||
return base
|
||||
}
|
||||
return fmt.Sprintf("%s-validator-%02d", base, attempt)
|
||||
}
|
||||
218
internal/framework/pipeline/validation_executor_test.go
Normal file
218
internal/framework/pipeline/validation_executor_test.go
Normal file
@@ -0,0 +1,218 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestExecuteValidationChainSettlesEveryValidatorInOrder(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
chain preparedValidatorChain
|
||||
invoke validationInvoker
|
||||
want []validationOutcome
|
||||
wantCalls []string
|
||||
wantGuidance []string
|
||||
wantWarnings []string
|
||||
}{
|
||||
{
|
||||
name: "all approved",
|
||||
chain: validationChain(validationSpec("shape", contracts.ExecutionClassDeterministic, 0), validationSpec("refs", contracts.ExecutionClassDeterministic, 0)),
|
||||
invoke: validationSequence(map[string][]validationStep{
|
||||
"shape": {{result: contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{ReasonCode: "shape"}}}}},
|
||||
"refs": {{result: contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{ReasonCode: "refs"}}}}},
|
||||
}),
|
||||
want: []validationOutcome{validationApproved, validationApproved},
|
||||
wantCalls: []string{"shape:1", "refs:1"},
|
||||
wantWarnings: []string{"shape", "refs"},
|
||||
},
|
||||
{
|
||||
name: "multiple rejections deduplicate guidance",
|
||||
chain: validationChain(validationSpec("shape", contracts.ExecutionClassDeterministic, 0), validationSpec("refs", contracts.ExecutionClassDeterministic, 0), validationSpec("coverage", contracts.ExecutionClassDeterministic, 0)),
|
||||
invoke: validationSequence(map[string][]validationStep{
|
||||
"shape": {{result: contracts.ValidationResult{ReasonCode: "shape", Message: "invalid", CorrectionGuidance: "repair shape"}}},
|
||||
"refs": {{result: contracts.ValidationResult{ReasonCode: "refs", Message: "missing", CorrectionGuidance: "repair references"}}},
|
||||
"coverage": {{result: contracts.ValidationResult{ReasonCode: "coverage", Message: "missing", CorrectionGuidance: "repair shape"}}},
|
||||
}),
|
||||
want: []validationOutcome{validationRejected, validationRejected, validationRejected},
|
||||
wantCalls: []string{"shape:1", "refs:1", "coverage:1"},
|
||||
wantGuidance: []string{"repair shape", "repair references"},
|
||||
},
|
||||
{
|
||||
name: "rejection and failure both settle",
|
||||
chain: validationChain(validationSpec("shape", contracts.ExecutionClassDeterministic, 0), validationSpec("remote", contracts.ExecutionClassDeterministic, 0)),
|
||||
invoke: validationSequence(map[string][]validationStep{
|
||||
"shape": {{result: contracts.ValidationResult{ReasonCode: "shape", CorrectionGuidance: "repair shape"}}},
|
||||
"remote": {{err: errors.New("offline")}},
|
||||
}),
|
||||
want: []validationOutcome{validationRejected, validationFailed},
|
||||
wantCalls: []string{"shape:1", "remote:1"},
|
||||
wantGuidance: []string{"repair shape"},
|
||||
},
|
||||
{
|
||||
name: "failure only has no correction guidance",
|
||||
chain: validationChain(validationSpec("remote", contracts.ExecutionClassDeterministic, 0)),
|
||||
invoke: validationSequence(map[string][]validationStep{"remote": {{err: errors.New("offline")}}}),
|
||||
want: []validationOutcome{validationFailed},
|
||||
wantCalls: []string{"remote:1"},
|
||||
},
|
||||
{
|
||||
name: "skipped is retained without guidance",
|
||||
chain: validationChain(validationSpec("optional", contracts.ExecutionClassDeterministic, 0)),
|
||||
invoke: validationSequence(map[string][]validationStep{"optional": {{invocation: skippedValidation("runtime_unavailable", "optional dependency unavailable")}}}),
|
||||
want: []validationOutcome{validationSkipped},
|
||||
wantCalls: []string{"optional:1"},
|
||||
},
|
||||
{
|
||||
name: "LLM failure retries then succeeds",
|
||||
chain: validationChain(validationSpec("remote", contracts.ExecutionClassLLMBacked, 2)),
|
||||
invoke: validationSequence(map[string][]validationStep{"remote": {{err: errors.New("first")}, {err: errors.New("second")}, {result: contracts.ValidationResult{Approved: true}}}}),
|
||||
want: []validationOutcome{validationApproved},
|
||||
wantCalls: []string{"remote:1", "remote:2", "remote:3"},
|
||||
},
|
||||
{
|
||||
name: "LLM failure retry exhaustion records once",
|
||||
chain: validationChain(validationSpec("remote", contracts.ExecutionClassLLMBacked, 1)),
|
||||
invoke: validationSequence(map[string][]validationStep{"remote": {{err: errors.New("first")}, {err: errors.New("second")}}}),
|
||||
want: []validationOutcome{validationFailed},
|
||||
wantCalls: []string{"remote:1", "remote:2"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
calls := []string(nil)
|
||||
invoke := func(ctx context.Context, validator preparedValidator, attempt int) (validationInvocation, error) {
|
||||
calls = append(calls, validator.resolved.Binding.Module+":"+string(rune('0'+attempt)))
|
||||
return test.invoke(ctx, validator, attempt)
|
||||
}
|
||||
report, err := executeValidationChain(context.Background(), test.chain, invoke)
|
||||
if err != nil {
|
||||
t.Fatalf("executeValidationChain() error = %v", err)
|
||||
}
|
||||
records := report.Records()
|
||||
outcomes := make([]validationOutcome, len(records))
|
||||
for index, record := range records {
|
||||
outcomes[index] = record.outcome
|
||||
}
|
||||
if !reflect.DeepEqual(outcomes, test.want) || !reflect.DeepEqual(calls, test.wantCalls) {
|
||||
t.Fatalf("outcomes = %#v calls = %#v, want %#v %#v", outcomes, calls, test.want, test.wantCalls)
|
||||
}
|
||||
if len(test.wantGuidance) > 0 {
|
||||
guidance, err := report.CorrectionRequest()
|
||||
if err != nil {
|
||||
t.Fatalf("CorrectionRequest() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(guidance, "complete corrected replacement") {
|
||||
t.Fatalf("CorrectionRequest() = %q, want complete replacement instruction", guidance)
|
||||
}
|
||||
for _, item := range test.wantGuidance {
|
||||
if strings.Count(guidance, item) != 1 {
|
||||
t.Fatalf("CorrectionRequest() = %q, want one occurrence of %q", guidance, item)
|
||||
}
|
||||
}
|
||||
}
|
||||
warnings := report.Warnings()
|
||||
var warningCodes []string
|
||||
for _, warning := range warnings {
|
||||
warningCodes = append(warningCodes, warning.ReasonCode)
|
||||
}
|
||||
if !reflect.DeepEqual(warningCodes, test.wantWarnings) {
|
||||
t.Fatalf("Warnings() = %#v, want codes %#v", warnings, test.wantWarnings)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteValidationChainKeepsInvocationInputsImmutable(t *testing.T) {
|
||||
candidate := []byte(`{"events":["one"]}`)
|
||||
chain := validationChain(validationSpec("remote", contracts.ExecutionClassLLMBacked, 1))
|
||||
var received [][]byte
|
||||
report, err := executeValidationChain(context.Background(), chain, func(_ context.Context, _ preparedValidator, attempt int) (validationInvocation, error) {
|
||||
requestCandidate := append([]byte(nil), candidate...)
|
||||
received = append(received, requestCandidate)
|
||||
requestCandidate[0] = 'x'
|
||||
if attempt == 1 {
|
||||
return validationInvocation{}, errors.New("retry")
|
||||
}
|
||||
return validationInvocation{result: contracts.ValidationResult{Approved: true}}, nil
|
||||
})
|
||||
if err != nil || len(report.Records()) != 1 || report.Records()[0].attemptCount != 2 {
|
||||
t.Fatalf("report = %#v, error = %v", report, err)
|
||||
}
|
||||
if string(candidate) != `{"events":["one"]}` || len(received) != 2 || string(received[0]) != string(received[1]) {
|
||||
t.Fatalf("candidate = %q received = %#v, want immutable repeated request content", candidate, received)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidationReportRejectsOversizedGuidanceAndOwnsRecords(t *testing.T) {
|
||||
guidance := strings.Repeat("x", contracts.MaxValidationCorrectionGuidanceBytes)
|
||||
report := validationReport{records: make([]validationRecord, 17)}
|
||||
for index := range report.records {
|
||||
unique := []byte(guidance)
|
||||
unique[0] = byte('a' + index)
|
||||
report.records[index] = validationRecord{validatorName: "validator", outcome: validationRejected, correctionGuidance: string(unique)}
|
||||
}
|
||||
report.records[0].warnings = []contracts.Warning{{ReasonCode: "warning"}}
|
||||
if _, err := report.CorrectionRequest(); err == nil {
|
||||
t.Fatal("CorrectionRequest() error = nil, want aggregate overflow error")
|
||||
}
|
||||
records := report.Records()
|
||||
records[0].warnings[0].ReasonCode = "changed"
|
||||
if report.Records()[0].warnings[0].ReasonCode != "warning" {
|
||||
t.Fatal("Records() exposed mutable warning storage")
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteValidationChainReturnsFrameworkAndCancellationErrors(t *testing.T) {
|
||||
chain := validationChain(validationSpec("validator", contracts.ExecutionClassLLMBacked, 1))
|
||||
framework := errors.New("debug persistence failed")
|
||||
if _, err := executeValidationChain(context.Background(), chain, func(context.Context, preparedValidator, int) (validationInvocation, error) {
|
||||
return validationInvocation{}, fatalValidationError(framework)
|
||||
}); !errors.Is(err, framework) {
|
||||
t.Fatalf("framework error = %v, want %v", err, framework)
|
||||
}
|
||||
canceled, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if _, err := executeValidationChain(canceled, chain, func(context.Context, preparedValidator, int) (validationInvocation, error) {
|
||||
return validationInvocation{result: contracts.ValidationResult{Approved: true}}, nil
|
||||
}); !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("cancellation error = %v, want context canceled", err)
|
||||
}
|
||||
}
|
||||
|
||||
type validationStep struct {
|
||||
result contracts.ValidationResult
|
||||
invocation validationInvocation
|
||||
err error
|
||||
}
|
||||
|
||||
func validationSequence(steps map[string][]validationStep) validationInvoker {
|
||||
positions := make(map[string]int)
|
||||
return func(_ context.Context, validator preparedValidator, _ int) (validationInvocation, error) {
|
||||
name := validator.resolved.Binding.Module
|
||||
index := positions[name]
|
||||
positions[name]++
|
||||
if index >= len(steps[name]) {
|
||||
return validationInvocation{}, errors.New("unexpected validator invocation")
|
||||
}
|
||||
step := steps[name][index]
|
||||
if step.invocation.skipped {
|
||||
return step.invocation, step.err
|
||||
}
|
||||
return validationInvocation{result: step.result}, step.err
|
||||
}
|
||||
}
|
||||
|
||||
func validationChain(validators ...preparedValidator) preparedValidatorChain {
|
||||
return preparedValidatorChain{validators: validators}
|
||||
}
|
||||
|
||||
func validationSpec(name string, class contracts.ExecutionClass, retries int) preparedValidator {
|
||||
return preparedValidator{resolved: ResolvedValidator{Binding: ModuleBinding{Module: name, Retries: retries}, ExecutionClass: class}}
|
||||
}
|
||||
115
internal/framework/pipeline/validation_policy.go
Normal file
115
internal/framework/pipeline/validation_policy.go
Normal file
@@ -0,0 +1,115 @@
|
||||
package pipeline
|
||||
|
||||
import "fmt"
|
||||
|
||||
type ProducerStructuralFailureAction string
|
||||
|
||||
const (
|
||||
ProducerStructuralFailureFailRun ProducerStructuralFailureAction = "fail_run"
|
||||
ProducerStructuralFailureRejectOutput ProducerStructuralFailureAction = "reject_output"
|
||||
)
|
||||
|
||||
type SemanticRejectionAction string
|
||||
|
||||
const (
|
||||
SemanticRejectionFailRun SemanticRejectionAction = "fail_run"
|
||||
SemanticRejectionRejectOutput SemanticRejectionAction = "reject_output"
|
||||
)
|
||||
|
||||
type ValidatorFailureAction string
|
||||
|
||||
const (
|
||||
ValidatorFailureWarnContinue ValidatorFailureAction = "warn_continue"
|
||||
ValidatorFailureFailRun ValidatorFailureAction = "fail_run"
|
||||
)
|
||||
|
||||
// ValidationPolicyOverride records only explicitly configured terminal-policy
|
||||
// values. Nil fields inherit independently from the pipeline or application.
|
||||
type ValidationPolicyOverride struct {
|
||||
ProducerStructuralFailure *ProducerStructuralFailureAction `json:"producer_structural_failure,omitempty"`
|
||||
SemanticRejection *SemanticRejectionAction `json:"semantic_rejection,omitempty"`
|
||||
ValidatorFailure *ValidatorFailureAction `json:"validator_failure,omitempty"`
|
||||
}
|
||||
|
||||
// ValidationPolicy is the concrete terminal policy retained by each resolved
|
||||
// producer. It contains no inherited pointers.
|
||||
type ValidationPolicy struct {
|
||||
ProducerStructuralFailure ProducerStructuralFailureAction `json:"producer_structural_failure"`
|
||||
SemanticRejection SemanticRejectionAction `json:"semantic_rejection"`
|
||||
ValidatorFailure ValidatorFailureAction `json:"validator_failure"`
|
||||
}
|
||||
|
||||
func DefaultValidationPolicy() ValidationPolicy {
|
||||
return ValidationPolicy{
|
||||
ProducerStructuralFailure: ProducerStructuralFailureFailRun,
|
||||
SemanticRejection: SemanticRejectionFailRun,
|
||||
ValidatorFailure: ValidatorFailureWarnContinue,
|
||||
}
|
||||
}
|
||||
|
||||
func (override ValidationPolicyOverride) Validate() error {
|
||||
if override.ProducerStructuralFailure != nil {
|
||||
switch *override.ProducerStructuralFailure {
|
||||
case ProducerStructuralFailureFailRun, ProducerStructuralFailureRejectOutput:
|
||||
default:
|
||||
return fmt.Errorf("producer_structural_failure must be fail_run or reject_output")
|
||||
}
|
||||
}
|
||||
if override.SemanticRejection != nil {
|
||||
switch *override.SemanticRejection {
|
||||
case SemanticRejectionFailRun, SemanticRejectionRejectOutput:
|
||||
default:
|
||||
return fmt.Errorf("semantic_rejection must be fail_run or reject_output")
|
||||
}
|
||||
}
|
||||
if override.ValidatorFailure != nil {
|
||||
switch *override.ValidatorFailure {
|
||||
case ValidatorFailureWarnContinue, ValidatorFailureFailRun:
|
||||
default:
|
||||
return fmt.Errorf("validator_failure must be warn_continue or fail_run")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func ResolveValidationPolicy(binding, pipeline *ValidationPolicyOverride) ValidationPolicy {
|
||||
resolved := DefaultValidationPolicy()
|
||||
applyValidationPolicyOverride(&resolved, pipeline)
|
||||
applyValidationPolicyOverride(&resolved, binding)
|
||||
return resolved
|
||||
}
|
||||
|
||||
func cloneValidationPolicyOverride(override *ValidationPolicyOverride) *ValidationPolicyOverride {
|
||||
if override == nil {
|
||||
return nil
|
||||
}
|
||||
out := *override
|
||||
if override.ProducerStructuralFailure != nil {
|
||||
value := *override.ProducerStructuralFailure
|
||||
out.ProducerStructuralFailure = &value
|
||||
}
|
||||
if override.SemanticRejection != nil {
|
||||
value := *override.SemanticRejection
|
||||
out.SemanticRejection = &value
|
||||
}
|
||||
if override.ValidatorFailure != nil {
|
||||
value := *override.ValidatorFailure
|
||||
out.ValidatorFailure = &value
|
||||
}
|
||||
return &out
|
||||
}
|
||||
|
||||
func applyValidationPolicyOverride(policy *ValidationPolicy, override *ValidationPolicyOverride) {
|
||||
if override == nil {
|
||||
return
|
||||
}
|
||||
if override.ProducerStructuralFailure != nil {
|
||||
policy.ProducerStructuralFailure = *override.ProducerStructuralFailure
|
||||
}
|
||||
if override.SemanticRejection != nil {
|
||||
policy.SemanticRejection = *override.SemanticRejection
|
||||
}
|
||||
if override.ValidatorFailure != nil {
|
||||
policy.ValidatorFailure = *override.ValidatorFailure
|
||||
}
|
||||
}
|
||||
103
internal/framework/pipeline/validation_policy_test.go
Normal file
103
internal/framework/pipeline/validation_policy_test.go
Normal file
@@ -0,0 +1,103 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestResolvePipelineAppliesValidationPolicyFieldByField(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
profile.ValidationPolicy = validationPolicyOverride(
|
||||
ProducerStructuralFailureRejectOutput,
|
||||
SemanticRejectionRejectOutput,
|
||||
ValidatorFailureFailRun,
|
||||
)
|
||||
profile.Chunk.ValidationPolicy = validationPolicyOverride("", "", ValidatorFailureWarnContinue)
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.Extract.ValidationPolicy = validationPolicyOverride("", SemanticRejectionFailRun, "")
|
||||
lane.Merge.ValidationPolicy = validationPolicyOverride("", "", ValidatorFailureWarnContinue)
|
||||
profile.Artifacts["events"] = lane
|
||||
|
||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v", err)
|
||||
}
|
||||
if got, want := resolved.ChunkValidationPolicy, (ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureRejectOutput, SemanticRejection: SemanticRejectionRejectOutput, ValidatorFailure: ValidatorFailureWarnContinue}); got != want {
|
||||
t.Fatalf("chunk validation policy = %#v, want %#v", got, want)
|
||||
}
|
||||
resolvedLane := resolved.Steps[0].ArtifactLanes[0]
|
||||
if got, want := resolvedLane.ExtractValidationPolicy, (ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureRejectOutput, SemanticRejection: SemanticRejectionFailRun, ValidatorFailure: ValidatorFailureFailRun}); got != want {
|
||||
t.Fatalf("extract validation policy = %#v, want %#v", got, want)
|
||||
}
|
||||
if got, want := resolvedLane.MergeValidationPolicy, (ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureRejectOutput, SemanticRejection: SemanticRejectionRejectOutput, ValidatorFailure: ValidatorFailureWarnContinue}); got != want {
|
||||
t.Fatalf("merge validation policy = %#v, want %#v", got, want)
|
||||
}
|
||||
if got, want := resolvedLane.NormalizeValidationPolicy, (ValidationPolicy{ProducerStructuralFailure: ProducerStructuralFailureRejectOutput, SemanticRejection: SemanticRejectionRejectOutput, ValidatorFailure: ValidatorFailureFailRun}); got != want {
|
||||
t.Fatalf("normalize validation policy = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
*profile.ValidationPolicy.SemanticRejection = SemanticRejectionFailRun
|
||||
if got := resolved.ConfiguredValidationPolicy.SemanticRejection; got == nil || *got != SemanticRejectionRejectOutput {
|
||||
t.Fatalf("resolved configured policy aliases profile: %#v", resolved.ConfiguredValidationPolicy)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineRejectsStructuralPolicyOverrideOnDeterministicProducer(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
profile.Chunk.ValidationPolicy = validationPolicyOverride(ProducerStructuralFailureRejectOutput, "", "")
|
||||
_, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
|
||||
if err == nil || !strings.Contains(err.Error(), "producer_structural_failure") || !strings.Contains(err.Error(), "deterministic") {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want deterministic structural-policy rejection", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidationPolicyChangesResolvedDigestAndRoundTripsBindings(t *testing.T) {
|
||||
left := baselineProfile()
|
||||
right := baselineProfile()
|
||||
right.ValidationPolicy = validationPolicyOverride("", SemanticRejectionRejectOutput, "")
|
||||
|
||||
leftResolved, err := ResolvePipeline(left, ResolveOptions{}, newProfileCatalog(t))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline(left) error = %v", err)
|
||||
}
|
||||
rightResolved, err := ResolvePipeline(right, ResolveOptions{}, newProfileCatalog(t))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline(right) error = %v", err)
|
||||
}
|
||||
if leftResolved.Digest == rightResolved.Digest {
|
||||
t.Fatal("validation policy change did not change resolved digest")
|
||||
}
|
||||
|
||||
binding := ModuleBinding{Module: "producer", ValidationPolicy: validationPolicyOverride(ProducerStructuralFailureRejectOutput, SemanticRejectionRejectOutput, ValidatorFailureFailRun)}
|
||||
encoded, err := json.Marshal(binding)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal binding: %v", err)
|
||||
}
|
||||
var decoded ModuleBinding
|
||||
if err := json.Unmarshal(encoded, &decoded); err != nil {
|
||||
t.Fatalf("unmarshal binding: %v", err)
|
||||
}
|
||||
if decoded.ValidationPolicy == nil || *decoded.ValidationPolicy.ProducerStructuralFailure != ProducerStructuralFailureRejectOutput || *decoded.ValidationPolicy.SemanticRejection != SemanticRejectionRejectOutput || *decoded.ValidationPolicy.ValidatorFailure != ValidatorFailureFailRun {
|
||||
t.Fatalf("round-tripped validation policy = %#v", decoded.ValidationPolicy)
|
||||
}
|
||||
cloned := cloneModuleBinding(binding)
|
||||
*binding.ValidationPolicy.ValidatorFailure = ValidatorFailureWarnContinue
|
||||
if got := *cloned.ValidationPolicy.ValidatorFailure; got != ValidatorFailureFailRun {
|
||||
t.Fatalf("cloned validation policy aliases binding: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func validationPolicyOverride(structural ProducerStructuralFailureAction, semantic SemanticRejectionAction, validator ValidatorFailureAction) *ValidationPolicyOverride {
|
||||
policy := &ValidationPolicyOverride{}
|
||||
if structural != "" {
|
||||
policy.ProducerStructuralFailure = &structural
|
||||
}
|
||||
if semantic != "" {
|
||||
policy.SemanticRejection = &semantic
|
||||
}
|
||||
if validator != "" {
|
||||
policy.ValidatorFailure = &validator
|
||||
}
|
||||
return policy
|
||||
}
|
||||
@@ -95,6 +95,7 @@ func cloneModuleBinding(binding ModuleBinding) ModuleBinding {
|
||||
binding.Module = strings.TrimSpace(binding.Module)
|
||||
binding.LLMProfile = strings.TrimSpace(binding.LLMProfile)
|
||||
binding.StructuredOutputRepairAttempts = cloneStructuredOutputRepairAttempts(binding.StructuredOutputRepairAttempts)
|
||||
binding.ValidationPolicy = cloneValidationPolicyOverride(binding.ValidationPolicy)
|
||||
binding.Options = cloneOptions(binding.Options)
|
||||
if len(binding.References) > 0 {
|
||||
references := make(map[string]ReferenceSource, len(binding.References))
|
||||
|
||||
@@ -11,8 +11,9 @@ import (
|
||||
)
|
||||
|
||||
type ValidatorSpec struct {
|
||||
Key string `json:"key"`
|
||||
ExecutionClass contracts.ExecutionClass `json:"execution_class"`
|
||||
Key string `json:"key"`
|
||||
ExecutionClass contracts.ExecutionClass `json:"execution_class"`
|
||||
CorrectionProtocol contracts.CorrectionProtocol `json:"correction_protocol,omitempty"`
|
||||
}
|
||||
|
||||
type SerializedValidatorSpec struct {
|
||||
@@ -321,7 +322,11 @@ func (r *ValidatorRegistry) RegisteredKeys() []string {
|
||||
}
|
||||
|
||||
func normalizeValidatorSpec(spec ValidatorSpec) (ValidatorSpec, error) {
|
||||
normalized := ValidatorSpec{Key: strings.TrimSpace(spec.Key), ExecutionClass: spec.ExecutionClass}
|
||||
normalized := ValidatorSpec{
|
||||
Key: strings.TrimSpace(spec.Key),
|
||||
ExecutionClass: spec.ExecutionClass,
|
||||
CorrectionProtocol: contracts.CorrectionProtocol(strings.TrimSpace(string(spec.CorrectionProtocol))),
|
||||
}
|
||||
if normalized.Key == "" {
|
||||
return ValidatorSpec{}, fmt.Errorf("validator key must not be empty")
|
||||
}
|
||||
@@ -330,6 +335,9 @@ func normalizeValidatorSpec(spec ValidatorSpec) (ValidatorSpec, error) {
|
||||
default:
|
||||
return ValidatorSpec{}, fmt.Errorf("validator %q execution class %q is not supported", normalized.Key, normalized.ExecutionClass)
|
||||
}
|
||||
if normalized.CorrectionProtocol != "" {
|
||||
return ValidatorSpec{}, fmt.Errorf("validator %q correction protocol is not supported", normalized.Key)
|
||||
}
|
||||
return normalized, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -49,6 +49,7 @@ type Request struct {
|
||||
ProfileID string
|
||||
StructuredOutputRepairAttempts *int
|
||||
SessionID string
|
||||
Correction *contracts.SemanticCorrection
|
||||
}
|
||||
|
||||
// ResultDisposition classifies a provider-neutral reconciliation outcome.
|
||||
@@ -69,6 +70,7 @@ type Result struct {
|
||||
issues []Issue
|
||||
discardedGroupCount int
|
||||
candidateMappings []CandidateMapping
|
||||
modelCandidate *contracts.ModelCandidate
|
||||
}
|
||||
|
||||
// Disposition returns the classified outcome.
|
||||
@@ -88,6 +90,16 @@ func (result Result) CandidateMappings() []CandidateMapping {
|
||||
return append([]CandidateMapping(nil), result.candidateMappings...)
|
||||
}
|
||||
|
||||
// ModelCandidate returns an owned copy of the proposal response when a model
|
||||
// completion produced this result.
|
||||
func (result Result) ModelCandidate() *contracts.ModelCandidate {
|
||||
candidate, err := contracts.CloneModelCandidate(result.modelCandidate)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return candidate
|
||||
}
|
||||
|
||||
func (result Result) planCopy() Plan {
|
||||
return Plan{groups: result.plan.Groups()}
|
||||
}
|
||||
@@ -162,8 +174,12 @@ func (engine *Engine) Reconcile(ctx context.Context, request Request) (Result, e
|
||||
if err := ctx.Err(); err != nil {
|
||||
return Result{}, fmt.Errorf("semantic reconciliation %q: context error before completion: %w", request.StageName, err)
|
||||
}
|
||||
correction, err := contracts.CloneSemanticCorrection(request.Correction)
|
||||
if err != nil {
|
||||
return Result{}, fmt.Errorf("semantic reconciliation %q: clone correction: %w", request.StageName, err)
|
||||
}
|
||||
var response ProposalResponse
|
||||
_, err = engine.client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
completion, err := engine.client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: request.StageName,
|
||||
PromptID: engine.prompt.ID,
|
||||
PromptVersion: engine.prompt.Version,
|
||||
@@ -171,6 +187,7 @@ func (engine *Engine) Reconcile(ctx context.Context, request Request) (Result, e
|
||||
SessionID: request.SessionID,
|
||||
StructuredOutputRepairAttempts: request.StructuredOutputRepairAttempts,
|
||||
Inputs: preparation.Materials(),
|
||||
Correction: correction,
|
||||
}, &response)
|
||||
if err != nil {
|
||||
if errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
||||
@@ -179,6 +196,11 @@ func (engine *Engine) Reconcile(ctx context.Context, request Request) (Result, e
|
||||
}
|
||||
return Result{}, fmt.Errorf("semantic reconciliation %q: complete structured output: %w", request.StageName, err)
|
||||
}
|
||||
candidate, err := contracts.NewModelCandidate(completion.Content, contracts.CorrectionProtocolSingleResponseV1)
|
||||
if err != nil {
|
||||
return Result{}, fmt.Errorf("semantic reconciliation %q: own model candidate: %w", request.StageName, err)
|
||||
}
|
||||
result.modelCandidate = candidate
|
||||
|
||||
assessment := preparation.Assess(response)
|
||||
result.plan = assessment.Plan()
|
||||
|
||||
@@ -56,6 +56,11 @@ func TestEnginePropagatesRequestAndAssessesResponse(t *testing.T) {
|
||||
request := readyEngineRequest()
|
||||
request.ProfileID = " profile-as-resolved "
|
||||
request.SessionID = " session-as-supplied "
|
||||
correction, err := contracts.NewSemanticCorrection([]byte(`{"duplicate_groups":[]}`), "retain distinct candidates")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
request.Correction = correction
|
||||
|
||||
result, err := engine.Reconcile(context.Background(), request)
|
||||
if err != nil {
|
||||
@@ -75,24 +80,32 @@ func TestEnginePropagatesRequestAndAssessesResponse(t *testing.T) {
|
||||
if got.StageName != request.StageName || got.PromptID != engine.prompt.ID || got.PromptVersion != engine.prompt.Version || got.ProfileID != request.ProfileID || got.SessionID != request.SessionID {
|
||||
t.Fatalf("structured request = %#v, want exact routing values", got)
|
||||
}
|
||||
if !reflect.DeepEqual(got.Correction, correction) {
|
||||
t.Fatalf("structured request correction = %#v, want %#v", got.Correction, correction)
|
||||
}
|
||||
if len(got.Inputs) != 2 || got.Inputs["candidates"].Name != "candidates" || got.Inputs["transcript"].Name != "transcript" || len(got.Vars) != 0 {
|
||||
t.Fatalf("structured request inputs = %#v vars = %#v, want only candidate and transcript materials", got.Inputs, got.Vars)
|
||||
}
|
||||
candidate := result.ModelCandidate()
|
||||
if candidate == nil || candidate.Protocol != contracts.CorrectionProtocolSingleResponseV1 || string(candidate.Response) != `{"duplicate_groups":[]}` {
|
||||
t.Fatalf("model candidate = %#v, want exact owned completion response", candidate)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEngineClassifiesSemanticAndTransportOutcomes(t *testing.T) {
|
||||
transportErr := errors.New("provider unavailable")
|
||||
tests := []struct {
|
||||
name string
|
||||
response ProposalResponse
|
||||
completion error
|
||||
want ResultDisposition
|
||||
wantDiscard int
|
||||
wantIssues bool
|
||||
wantError error
|
||||
name string
|
||||
response ProposalResponse
|
||||
completion error
|
||||
want ResultDisposition
|
||||
wantDiscard int
|
||||
wantIssues bool
|
||||
wantCandidate bool
|
||||
wantError error
|
||||
}{
|
||||
{name: "empty groups complete", response: ProposalResponse{DuplicateGroups: []DuplicateGroup{}}, want: Complete},
|
||||
{name: "discarded proposal retryable", response: ProposalResponse{DuplicateGroups: []DuplicateGroup{{CandidateIDs: []int{1, 99}, CanonicalCandidateID: 1}}}, want: RetryableDiscardedProposalGroups, wantDiscard: 1, wantIssues: true},
|
||||
{name: "empty groups complete", response: ProposalResponse{DuplicateGroups: []DuplicateGroup{}}, want: Complete, wantCandidate: true},
|
||||
{name: "discarded proposal retryable", response: ProposalResponse{DuplicateGroups: []DuplicateGroup{{CandidateIDs: []int{1, 99}, CanonicalCandidateID: 1}}}, want: RetryableDiscardedProposalGroups, wantDiscard: 1, wantIssues: true, wantCandidate: true},
|
||||
{name: "invalid structured output retryable", completion: fmt.Errorf("decode response: %w", contracts.ErrInvalidStructuredOutput), want: RetryableInvalidStructuredOutput},
|
||||
{name: "transport failure", completion: transportErr, wantError: transportErr},
|
||||
}
|
||||
@@ -112,6 +125,9 @@ func TestEngineClassifiesSemanticAndTransportOutcomes(t *testing.T) {
|
||||
if result.Disposition() != test.want || result.DiscardedGroupCount() != test.wantDiscard || (len(result.Issues()) > 0) != test.wantIssues {
|
||||
t.Fatalf("result = disposition %v discarded %d issues %#v", result.Disposition(), result.DiscardedGroupCount(), result.Issues())
|
||||
}
|
||||
if (result.ModelCandidate() != nil) != test.wantCandidate {
|
||||
t.Fatalf("model candidate = %#v, want presence %t", result.ModelCandidate(), test.wantCandidate)
|
||||
}
|
||||
if len(client.requests) != 1 {
|
||||
t.Fatalf("completion calls = %d, want one", len(client.requests))
|
||||
}
|
||||
@@ -141,6 +157,9 @@ func TestEngineSkipsDeterministicOutcomesWithoutCompletion(t *testing.T) {
|
||||
if result.Disposition() != test.want || len(result.CandidateMappings()) != test.mappingLen {
|
||||
t.Fatalf("result disposition = %v mappings = %#v", result.Disposition(), result.CandidateMappings())
|
||||
}
|
||||
if result.ModelCandidate() != nil {
|
||||
t.Fatalf("model candidate = %#v, want nil for no-call outcome", result.ModelCandidate())
|
||||
}
|
||||
if len(client.requests) != 0 {
|
||||
t.Fatalf("completion calls = %d, want zero", len(client.requests))
|
||||
}
|
||||
@@ -209,7 +228,9 @@ func TestEngineCallsAreIndependentAndResultsAreOwned(t *testing.T) {
|
||||
firstIssues[0].Category = "changed"
|
||||
firstMappings := first.CandidateMappings()
|
||||
firstMappings[0].CandidatePosition = 99
|
||||
if first.Plan().Groups()[0].MemberPositions()[0] != 0 || first.Issues()[0].Category == "changed" || first.CandidateMappings()[0].CandidatePosition != 0 {
|
||||
firstCandidate := first.ModelCandidate()
|
||||
firstCandidate.Response[0] = 'x'
|
||||
if first.Plan().Groups()[0].MemberPositions()[0] != 0 || first.Issues()[0].Category == "changed" || first.CandidateMappings()[0].CandidatePosition != 0 || string(first.ModelCandidate().Response) != `{"duplicate_groups":[]}` {
|
||||
t.Fatal("result accessors exposed retained state")
|
||||
}
|
||||
|
||||
@@ -217,7 +238,7 @@ func TestEngineCallsAreIndependentAndResultsAreOwned(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if second.Disposition() != Complete || len(second.Plan().Groups()) != 0 || len(second.Issues()) != 0 || second.DiscardedGroupCount() != 0 || len(second.CandidateMappings()) != 2 {
|
||||
if second.Disposition() != Complete || len(second.Plan().Groups()) != 0 || len(second.Issues()) != 0 || second.DiscardedGroupCount() != 0 || len(second.CandidateMappings()) != 2 || second.ModelCandidate() == nil {
|
||||
t.Fatalf("second result retained prior call state: disposition %v plan %#v issues %#v discarded %d mappings %#v", second.Disposition(), second.Plan().Groups(), second.Issues(), second.DiscardedGroupCount(), second.CandidateMappings())
|
||||
}
|
||||
}
|
||||
@@ -226,11 +247,15 @@ type recordingReconciliationClient struct {
|
||||
requests []contracts.StructuredCompletionRequest
|
||||
responses []ProposalResponse
|
||||
errors []error
|
||||
content []byte
|
||||
}
|
||||
|
||||
func (client *recordingReconciliationClient) CompleteStructured(_ context.Context, request contracts.StructuredCompletionRequest, output any) (contracts.StructuredCompletionResponse, error) {
|
||||
request.Inputs = request.Inputs.Clone()
|
||||
client.requests = append(client.requests, request)
|
||||
snapshot, err := contracts.CloneStructuredCompletionRequest(request)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("clone recording request: %w", err)
|
||||
}
|
||||
client.requests = append(client.requests, snapshot)
|
||||
index := len(client.requests) - 1
|
||||
if index < len(client.errors) && client.errors[index] != nil {
|
||||
return contracts.StructuredCompletionResponse{}, client.errors[index]
|
||||
@@ -244,7 +269,11 @@ func (client *recordingReconciliationClient) CompleteStructured(_ context.Contex
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("output type = %T", output)
|
||||
}
|
||||
*target = response
|
||||
return contracts.StructuredCompletionResponse{}, nil
|
||||
content := append([]byte(nil), client.content...)
|
||||
if len(content) == 0 {
|
||||
content = []byte(`{"duplicate_groups":[]}`)
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{Content: content}, nil
|
||||
}
|
||||
|
||||
func cloneProposalResponse(response ProposalResponse) ProposalResponse {
|
||||
|
||||
@@ -94,33 +94,40 @@ func (c *Chunker) Plan(ctx context.Context, req contracts.ChunkRequest) (contrac
|
||||
return contracts.ChunkPlanResult{}, chunkerErrorf("validate source document: %w", err)
|
||||
}
|
||||
var response chunkResponse
|
||||
if _, err := c.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
completion, err := c.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key,
|
||||
PromptID: PromptID,
|
||||
PromptVersion: ResponseSchemaVersion,
|
||||
ProfileID: req.LLMProfile,
|
||||
SessionID: req.SessionID,
|
||||
StructuredOutputRepairAttempts: req.StructuredOutputRepairAttempts,
|
||||
Correction: req.Correction,
|
||||
Inputs: shared.PromptInputs(req.SourceInput, req.References),
|
||||
}, &response); err != nil {
|
||||
}, &response)
|
||||
if err != nil {
|
||||
return contracts.ChunkPlanResult{}, chunkerErrorf("complete structured output: %w", err)
|
||||
}
|
||||
candidate, err := shared.ModelCandidateFromResponse(completion)
|
||||
if err != nil {
|
||||
return contracts.ChunkPlanResult{}, chunkerErrorf("capture model candidate: %w", err)
|
||||
}
|
||||
|
||||
plan, err := planFromResponse(req.Source, response)
|
||||
if err != nil {
|
||||
return contracts.ChunkPlanResult{}, chunkerErrorf("malformed structured output: %w", err)
|
||||
}
|
||||
return contracts.ChunkPlanResult{Plan: plan}, nil
|
||||
return contracts.ChunkPlanResult{Plan: plan, ModelCandidate: candidate}, nil
|
||||
}
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),
|
||||
Key: Key,
|
||||
Stage: pipeline.StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -22,12 +22,13 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
|
||||
}
|
||||
|
||||
want := pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: []string{"source.transcript"},
|
||||
Provides: []string{"chunks"},
|
||||
ReferenceSlots: wantReferenceSlots(),
|
||||
Key: Key,
|
||||
Stage: pipeline.StageChunk,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
|
||||
Requires: []string{"source.transcript"},
|
||||
Provides: []string{"chunks"},
|
||||
ReferenceSlots: wantReferenceSlots(),
|
||||
}
|
||||
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||
@@ -181,6 +182,13 @@ func TestPlanReturnsAnnotationFreeSceneRangesFromStructuredOutput(t *testing.T)
|
||||
if len(result.Warnings) != 0 {
|
||||
t.Fatalf("warnings = %#v, want absent", result.Warnings)
|
||||
}
|
||||
wantCandidate, err := json.Marshal(client.response)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal expected candidate: %v", err)
|
||||
}
|
||||
if result.ModelCandidate == nil || result.ModelCandidate.Protocol != contracts.CorrectionProtocolSingleResponseV1 || !reflect.DeepEqual(result.ModelCandidate.Response, wantCandidate) {
|
||||
t.Fatalf("model candidate = %#v, want exact validated response", result.ModelCandidate)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlanUsesDocumentOrderForNonconsecutiveUnitIDs(t *testing.T) {
|
||||
@@ -533,6 +541,11 @@ func (client *fakeScenesLLMClient) CompleteStructured(ctx context.Context, req c
|
||||
|
||||
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
|
||||
req.Inputs = req.Inputs.Clone()
|
||||
correction, err := contracts.CloneSemanticCorrection(req.Correction)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
req.Correction = correction
|
||||
req.Vars = cloneVars(req.Vars)
|
||||
return req
|
||||
}
|
||||
|
||||
@@ -208,30 +208,37 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
|
||||
var response extractionResponse
|
||||
inputs := shared.PromptInputs(sourceInput, req.References)
|
||||
inputs[NPCRegistryReferenceSlot] = npcRegistry.PromptInput()
|
||||
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
completion, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key,
|
||||
PromptID: PromptID,
|
||||
PromptVersion: SchemaVersion,
|
||||
ProfileID: req.LLMProfile,
|
||||
SessionID: req.SessionID,
|
||||
StructuredOutputRepairAttempts: req.StructuredOutputRepairAttempts,
|
||||
Correction: req.Correction,
|
||||
Inputs: inputs,
|
||||
}, &response); err != nil {
|
||||
}, &response)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.CombatTurnList]{}, extractorErrorf("complete structured output: %w", err)
|
||||
}
|
||||
candidate, err := shared.ModelCandidateFromResponse(completion)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.CombatTurnList]{}, extractorErrorf("capture model candidate: %w", err)
|
||||
}
|
||||
canonicalizeResponse(&response, order, req.Source.ID)
|
||||
return contracts.TypedExtractionResult[dnd.CombatTurnList]{Value: canonicalCombatTurnList(response, req.Source.ID)}, nil
|
||||
return contracts.TypedExtractionResult[dnd.CombatTurnList]{Value: canonicalCombatTurnList(response, req.Source.ID), ModelCandidate: candidate}, nil
|
||||
}
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.CombatTurnListKind,
|
||||
ReferenceSlots: referenceSlots(),
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.CombatTurnListKind,
|
||||
ReferenceSlots: referenceSlots(),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -448,7 +448,7 @@ func TestExtractorManifestMetadataAndFingerprints(t *testing.T) {
|
||||
func TestModuleSpecAndRegistration(t *testing.T) {
|
||||
wantSlots := referenceSlots()
|
||||
got := ModuleSpec()
|
||||
if got.Key != Key || got.Stage != pipeline.StageExtract || got.ArtifactKind != dnd.CombatTurnListKind || !reflect.DeepEqual(got.Requires, []string{"chunks", "source.transcript"}) || !reflect.DeepEqual(got.Provides, []string{"dnd.combat_turns"}) || !reflect.DeepEqual(got.ReferenceSlots, wantSlots) {
|
||||
if got.Key != Key || got.Stage != pipeline.StageExtract || got.CorrectionProtocol != contracts.CorrectionProtocolSingleResponseV1 || got.ArtifactKind != dnd.CombatTurnListKind || !reflect.DeepEqual(got.Requires, []string{"chunks", "source.transcript"}) || !reflect.DeepEqual(got.Provides, []string{"dnd.combat_turns"}) || !reflect.DeepEqual(got.ReferenceSlots, wantSlots) {
|
||||
t.Fatalf("ModuleSpec() = %#v, want combat extractor contract", got)
|
||||
}
|
||||
got.Requires[0] = "changed"
|
||||
@@ -592,6 +592,11 @@ func (client *fakeCombatTurnsLLMClient) CompleteStructured(_ context.Context, re
|
||||
|
||||
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
|
||||
req.Inputs = req.Inputs.Clone()
|
||||
correction, err := contracts.CloneSemanticCorrection(req.Correction)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
req.Correction = correction
|
||||
if len(req.Vars) == 0 {
|
||||
req.Vars = nil
|
||||
return req
|
||||
|
||||
@@ -142,19 +142,26 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
|
||||
inputs[name] = input
|
||||
}
|
||||
var response extractionResponse
|
||||
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
completion, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key,
|
||||
PromptID: PromptID,
|
||||
PromptVersion: SchemaVersion,
|
||||
ProfileID: req.LLMProfile,
|
||||
SessionID: req.SessionID,
|
||||
StructuredOutputRepairAttempts: req.StructuredOutputRepairAttempts,
|
||||
Correction: req.Correction,
|
||||
Inputs: inputs,
|
||||
}, &response); err != nil {
|
||||
}, &response)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.EnemyEventList]{}, extractorErrorf("complete structured output: %w", err)
|
||||
}
|
||||
candidate, err := shared.ModelCandidateFromResponse(completion)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.EnemyEventList]{}, extractorErrorf("capture model candidate: %w", err)
|
||||
}
|
||||
return contracts.TypedExtractionResult[dnd.EnemyEventList]{
|
||||
Value: canonicalEnemyEventList(response, shared.NewSourceRefOrder(req.Source), req.Source.ID),
|
||||
Value: canonicalEnemyEventList(response, shared.NewSourceRefOrder(req.Source), req.Source.ID),
|
||||
ModelCandidate: candidate,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -174,13 +181,14 @@ func unavailableSceneResult() contracts.TypedExtractionResult[dnd.EnemyEventList
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.EnemyEventListKind,
|
||||
ReferenceSlots: referenceSlots(),
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.EnemyEventListKind,
|
||||
ReferenceSlots: referenceSlots(),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -137,7 +137,7 @@ func TestConstructorSpecOptionsAndSafeMetadata(t *testing.T) {
|
||||
first.Requires[0] = "changed"
|
||||
first.ReferenceSlots[0].AcceptedMediaTypes[0] = "changed"
|
||||
second := ModuleSpec()
|
||||
if second.Requires[0] != "chunks" || second.ArtifactKind != dnd.EnemyEventListKind || second.ExecutionClass != contracts.ExecutionClassLLMBacked || second.ReferenceSlots[0].AcceptedMediaTypes[0] == "changed" {
|
||||
if second.Requires[0] != "chunks" || second.ArtifactKind != dnd.EnemyEventListKind || second.ExecutionClass != contracts.ExecutionClassLLMBacked || second.CorrectionProtocol != contracts.CorrectionProtocolSingleResponseV1 || second.ReferenceSlots[0].AcceptedMediaTypes[0] == "changed" {
|
||||
t.Fatalf("ModuleSpec() reused mutable state: %#v", second)
|
||||
}
|
||||
registry := pipeline.NewExtractorRegistry()
|
||||
|
||||
@@ -155,28 +155,35 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
|
||||
var response extractionResponse
|
||||
inputs := shared.PromptInputs(sourceInput, req.References)
|
||||
inputs[ItemRegistryReferenceSlot] = registry.PromptInput()
|
||||
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
completion, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key, PromptID: PromptID, PromptVersion: SchemaVersion,
|
||||
ProfileID: req.LLMProfile, SessionID: req.SessionID, StructuredOutputRepairAttempts: req.StructuredOutputRepairAttempts, Inputs: inputs,
|
||||
}, &response); err != nil {
|
||||
ProfileID: req.LLMProfile, SessionID: req.SessionID, StructuredOutputRepairAttempts: req.StructuredOutputRepairAttempts,
|
||||
Correction: req.Correction, Inputs: inputs,
|
||||
}, &response)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.ItemOccurrenceList]{}, extractorErrorf("complete structured output: %w", err)
|
||||
}
|
||||
candidate, err := shared.ModelCandidateFromResponse(completion)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.ItemOccurrenceList]{}, extractorErrorf("capture model candidate: %w", err)
|
||||
}
|
||||
value, err := canonicalItemOccurrenceList(response, order, req.Source.ID, registry)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.ItemOccurrenceList]{}, extractorErrorf("map item occurrence response: %w", err)
|
||||
}
|
||||
return contracts.TypedExtractionResult[dnd.ItemOccurrenceList]{Value: value}, nil
|
||||
return contracts.TypedExtractionResult[dnd.ItemOccurrenceList]{Value: value, ModelCandidate: candidate}, nil
|
||||
}
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.ItemOccurrenceListKind,
|
||||
ReferenceSlots: referenceSlots(),
|
||||
Key: Key,
|
||||
Stage: pipeline.StageExtract,
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.ItemOccurrenceListKind,
|
||||
ReferenceSlots: referenceSlots(),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -11,11 +11,15 @@ import (
|
||||
|
||||
func TestExtractGroundsOccurrencesInRequiredRegistry(t *testing.T) {
|
||||
id := itemidentity.DeriveID("Torch")
|
||||
client := &fakeItemOccurrencesLLMClient{response: extractionResponse{Occurrences: []itemOccurrenceResponse{
|
||||
{Name: "Torch", Kind: "lost", From: "party", SourceRefs: responseRefs(1, 1)},
|
||||
}}}
|
||||
rawResponse := []byte(`{"occurrences":[{"name":"Torch","kind":"lost","from":"party","source_refs":[{"start_unit_id":1,"end_unit_id":1}]}]}`)
|
||||
client := &fakeItemOccurrencesLLMClient{content: append([]byte(nil), rawResponse...)}
|
||||
req := extractionRequest()
|
||||
req.References = itemRegistryReferences(t)
|
||||
correction, err := contracts.NewSemanticCorrection([]byte(`{"occurrences":[]}`), "Keep the transcript-grounded item occurrence.")
|
||||
if err != nil {
|
||||
t.Fatalf("NewSemanticCorrection() error = %v", err)
|
||||
}
|
||||
req.Correction = correction
|
||||
result, err := newExtractor(t, client, req.References).Extract(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -26,10 +30,30 @@ func TestExtractGroundsOccurrencesInRequiredRegistry(t *testing.T) {
|
||||
if refs := result.Value.Occurrences[0].SourceRefs; len(refs) != 1 || refs[0].SourceID != req.Source.ID || refs[0].StartUnitID != 1 || refs[0].EndUnitID != 1 {
|
||||
t.Fatalf("occurrence evidence = %#v, want current-source unit range", refs)
|
||||
}
|
||||
input := client.requests[0].Inputs[ItemRegistryReferenceSlot]
|
||||
request := client.requests[0]
|
||||
input := request.Inputs[ItemRegistryReferenceSlot]
|
||||
if input.Name != ItemRegistryReferenceSlot || string(input.Content) != `{"items":[{"name":"Torch"}]}` || strings.Contains(string(input.Content), "item:sha256:") {
|
||||
t.Fatalf("registry prompt input = %#v, want names-only projection", input)
|
||||
}
|
||||
if request.Correction == nil || string(request.Correction.AssistantResponse) != `{"occurrences":[]}` || request.Correction.UserGuidance != "Keep the transcript-grounded item occurrence." {
|
||||
t.Fatalf("correction = %#v, want exact request correction", request.Correction)
|
||||
}
|
||||
for _, material := range []string{string(request.Correction.AssistantResponse), request.Correction.UserGuidance} {
|
||||
if strings.Contains(material, id) || strings.Contains(material, "item:sha256:") {
|
||||
t.Fatalf("correction material leaked opaque item identity: %q", material)
|
||||
}
|
||||
}
|
||||
correction.AssistantResponse[0] = '['
|
||||
if got := string(request.Correction.AssistantResponse); got != `{"occurrences":[]}` {
|
||||
t.Fatalf("captured correction changed after caller mutation: %q", got)
|
||||
}
|
||||
if result.ModelCandidate == nil || result.ModelCandidate.Protocol != contracts.CorrectionProtocolSingleResponseV1 || string(result.ModelCandidate.Response) != string(rawResponse) {
|
||||
t.Fatalf("model candidate = %#v, want exact validated response", result.ModelCandidate)
|
||||
}
|
||||
client.content[0] = '['
|
||||
if got := string(result.ModelCandidate.Response); got != string(rawResponse) {
|
||||
t.Fatalf("model candidate changed after provider buffer mutation: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractCanonicalizesComparisonEquivalentNames(t *testing.T) {
|
||||
|
||||
@@ -36,6 +36,9 @@ func testSourceRefs() []source.SourceRef {
|
||||
|
||||
func TestModuleSpecDeclaresRequiredRegistry(t *testing.T) {
|
||||
spec := ModuleSpec()
|
||||
if spec.CorrectionProtocol != contracts.CorrectionProtocolSingleResponseV1 {
|
||||
t.Fatalf("correction protocol = %q, want %q", spec.CorrectionProtocol, contracts.CorrectionProtocolSingleResponseV1)
|
||||
}
|
||||
var slot contracts.ReferenceSlot
|
||||
for _, candidate := range spec.ReferenceSlots {
|
||||
if candidate.Name == ItemRegistryReferenceSlot {
|
||||
|
||||
@@ -58,6 +58,11 @@ func newExtractor(t *testing.T, client contracts.StructuredLLMClient, references
|
||||
|
||||
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
|
||||
req.Inputs = req.Inputs.Clone()
|
||||
correction, err := contracts.CloneSemanticCorrection(req.Correction)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
req.Correction = correction
|
||||
return req
|
||||
}
|
||||
|
||||
|
||||
@@ -109,21 +109,28 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
|
||||
order := shared.NewSourceRefOrder(req.Source)
|
||||
|
||||
var response extractionResponse
|
||||
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
completion, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key, PromptID: PromptID, PromptVersion: SchemaVersion,
|
||||
ProfileID: req.LLMProfile, SessionID: req.SessionID, StructuredOutputRepairAttempts: req.StructuredOutputRepairAttempts,
|
||||
Inputs: shared.PromptInputs(sourceInput, req.References),
|
||||
}, &response); err != nil {
|
||||
Correction: req.Correction,
|
||||
Inputs: shared.PromptInputs(sourceInput, req.References),
|
||||
}, &response)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.ItemRegistry]{}, extractorErrorf("complete structured output: %w", err)
|
||||
}
|
||||
candidate, err := shared.ModelCandidateFromResponse(completion)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.ItemRegistry]{}, extractorErrorf("capture model candidate: %w", err)
|
||||
}
|
||||
canonicalizeResponse(&response, order, req.Source.ID)
|
||||
return contracts.TypedExtractionResult[dnd.ItemRegistry]{Value: canonicalItemRegistry(response, req.Source.ID)}, nil
|
||||
return contracts.TypedExtractionResult[dnd.ItemRegistry]{Value: canonicalItemRegistry(response, req.Source.ID), ModelCandidate: candidate}, nil
|
||||
}
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...),
|
||||
CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1,
|
||||
Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.ItemRegistryKind, ReferenceSlots: referenceSlots(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,7 +19,7 @@ func TestModuleRegistrationMetadataAndRedaction(t *testing.T) {
|
||||
if _, err := New(&fakeItemsLLMClient{}, Options{}, contracts.ReferenceSet{}, contracts.ReferenceSet{}); err == nil || !strings.Contains(err.Error(), "at most one") {
|
||||
t.Fatalf("New() error = %v, want reference-set rejection", err)
|
||||
}
|
||||
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_registry"}, ArtifactKind: dnd.ItemRegistryKind, ReferenceSlots: referenceSlots()}
|
||||
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, CorrectionProtocol: contracts.CorrectionProtocolSingleResponseV1, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.item_registry"}, ArtifactKind: dnd.ItemRegistryKind, ReferenceSlots: referenceSlots()}
|
||||
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
@@ -88,5 +88,10 @@ func (client *fakeItemsLLMClient) CompleteStructured(ctx context.Context, req co
|
||||
|
||||
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
|
||||
req.Inputs = req.Inputs.Clone()
|
||||
correction, err := contracts.CloneSemanticCorrection(req.Correction)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
req.Correction = correction
|
||||
return req
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user