Compare commits
16 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 61436d7c18 | |||
| 079d5af337 | |||
| 1025001f20 | |||
| da14924a02 | |||
| 2065a8288b | |||
| af0119cc1d | |||
| 54de2b816a | |||
| 4dbbf68051 | |||
| 480680b257 | |||
| 6a1fd7bdb6 | |||
| ccba2ce3f9 | |||
| 1f1967c8d2 | |||
| 5175cb0722 | |||
| ba569594a1 | |||
| acb04954eb | |||
| 610dd3d7c3 |
@@ -0,0 +1,67 @@
|
|||||||
|
# ADR-0015: Separate process warnings from quality diagnostics
|
||||||
|
|
||||||
|
**Status:** Accepted
|
||||||
|
**Date:** 2026-08-27
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Notarius currently represents process degradation, incomplete validation,
|
||||||
|
extraction-quality doubt, and routine normalization with one flat warning
|
||||||
|
record. That makes ordinary successful runs noisy, loses the framework context
|
||||||
|
needed to explain a finding, and gives `warning_count` no stable operational
|
||||||
|
meaning. It also permits output encoders to add a warning after the durable
|
||||||
|
warning file has already been written.
|
||||||
|
|
||||||
|
The application needs one bounded diagnostic model that preserves exact
|
||||||
|
occurrence counts while retaining only safe, representative samples. Fresh and
|
||||||
|
resumed logical runs must present the same groups. The model must not alter
|
||||||
|
validation decisions, retry budgets, rejected-output behavior, or process exit
|
||||||
|
policy.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Warnings are reserved for a completed run that advanced under an allowed
|
||||||
|
process-level degradation or incomplete-work policy. Extraction-quality signals
|
||||||
|
are advisories, and routine accepted transformations are observations. A
|
||||||
|
non-degraded successful run therefore has zero actionable warnings.
|
||||||
|
|
||||||
|
Modules and validators own a diagnostic's disposition, category, reason code,
|
||||||
|
scope, and safe message. The framework adds pipeline origin, including stage,
|
||||||
|
step, lane, module, validator, and chunk context where applicable. It then
|
||||||
|
aggregates deterministically by disposition, category, reason code, and full
|
||||||
|
origin. Chunk context remains on representative samples so equivalent findings
|
||||||
|
across chunks aggregate together.
|
||||||
|
|
||||||
|
Diagnostics carry exact occurrence counts, at most three distinct samples, and
|
||||||
|
numeric omitted-sample metadata. Producers and validators are bounded to 64
|
||||||
|
local groups. Final actionable warning groups are bounded without truncation;
|
||||||
|
the non-warning collection may truncate represented groups while preserving an
|
||||||
|
exact total occurrence count and explicit truncation metadata.
|
||||||
|
|
||||||
|
The public contracts will be versioned: grouped actionable warnings use
|
||||||
|
`notarius.warnings.v2`, grouped advisories and observations use
|
||||||
|
`notarius.diagnostics.v1`, and the run receipt uses
|
||||||
|
`notarius.run-result.v2`. Successful output encoders return logical files or
|
||||||
|
an error; they do not add post-encoding warnings.
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
|
||||||
|
- Keep one warning list and filter only CLI output. This would leave durable
|
||||||
|
consumers with the same semantically mixed, unbounded contract.
|
||||||
|
- Map reason codes to severity in a central framework registry. This would
|
||||||
|
split module-owned meaning between synchronized policy tables and make new
|
||||||
|
diagnostic meaning implicit.
|
||||||
|
- Preserve local omission warning records. They inflate visible group counts
|
||||||
|
and lose exact occurrence semantics.
|
||||||
|
- Keep output-encoder warnings. A one-pass encoder cannot include those
|
||||||
|
records consistently in files it has already serialized; a two-phase encoder
|
||||||
|
protocol is deferred until a demonstrated need exists.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
The framework gains validated diagnostic primitives, local collection,
|
||||||
|
origin-aware aggregation, and versioned durable presentation. Existing warning
|
||||||
|
transport remains temporarily while producers migrate. Current architecture,
|
||||||
|
operator, integration, and internal documentation will describe the behavior
|
||||||
|
only as each implementation step lands; this accepted decision does not claim
|
||||||
|
that the migration is complete.
|
||||||
@@ -103,11 +103,14 @@ names, requiredness, and configured bindings are part of the
|
|||||||
Without **--json**, standard output contains the completed pipeline ID, counts
|
Without **--json**, standard output contains the completed pipeline ID, counts
|
||||||
of normalized and rejected outputs, and the output directory. A debug-enabled
|
of normalized and rejected outputs, and the output directory. A debug-enabled
|
||||||
run also prints its debug-bundle path to standard output. A successful run with
|
run also prints its debug-bundle path to standard output. A successful run with
|
||||||
warnings reports the warning count to standard error. The published JSON bundle
|
actionable process warnings reports their group and occurrence counts to
|
||||||
|
standard error. When the selected output module publishes `warnings.json`, the
|
||||||
|
summary also reports that durable file's path. Advisory and observation findings
|
||||||
|
do not produce a warning line. The published JSON bundle
|
||||||
is defined by the [JSON output contract](integrations/json-output.md).
|
is defined by the [JSON output contract](integrations/json-output.md).
|
||||||
|
|
||||||
With **--json**, successful standard output is exactly one
|
With **--json**, successful standard output is exactly one
|
||||||
`notarius.run-result.v1` JSON document followed by a newline, with no
|
`notarius.run-result.v2` JSON document followed by a newline, with no
|
||||||
human-oriented status or debug-path line. Its fields and compatibility policy
|
human-oriented status or debug-path line. Its fields and compatibility policy
|
||||||
are defined by the [run-result contract](integrations/run-result.md). A caller
|
are defined by the [run-result contract](integrations/run-result.md). A caller
|
||||||
must check for exit status 0 before decoding this output; a failed write can
|
must check for exit status 0 before decoding this output; a failed write can
|
||||||
@@ -171,7 +174,7 @@ go run ./cmd/notarius pipelines list \
|
|||||||
Successful commands write their primary result to standard output. Warnings and
|
Successful commands write their primary result to standard output. Warnings and
|
||||||
errors are written to standard error.
|
errors are written to standard error.
|
||||||
|
|
||||||
For **run --json**, warnings remain on standard error and standard output is a
|
For **run --json**, actionable process warnings remain on standard error and standard output is a
|
||||||
machine-readable success result only. Syntax and runtime diagnostics remain on
|
machine-readable success result only. Syntax and runtime diagnostics remain on
|
||||||
standard error. Parse the result only after the process exits with status 0.
|
standard error. Parse the result only after the process exits with status 0.
|
||||||
|
|
||||||
|
|||||||
@@ -92,7 +92,7 @@ stream and exit-status contract.
|
|||||||
## Discover The Published Bundle
|
## Discover The Published Bundle
|
||||||
|
|
||||||
Decode the successful stdout document as a supported run-result schema. For
|
Decode the successful stdout document as a supported run-result schema. For
|
||||||
the current contract, `schema_version` is `notarius.run-result.v1`. Tolerate
|
the current contract, `schema_version` is `notarius.run-result.v2`. Tolerate
|
||||||
unknown fields allowed by that version, but reject an unsupported schema
|
unknown fields allowed by that version, but reject an unsupported schema
|
||||||
version.
|
version.
|
||||||
|
|
||||||
@@ -145,7 +145,8 @@ The JSON encoder always publishes these bundle-management files:
|
|||||||
| `index.json` | Discovery document for lane and pipeline-wide artifacts. |
|
| `index.json` | Discovery document for lane and pipeline-wide artifacts. |
|
||||||
| `manifest.json` | Run provenance and result summaries. |
|
| `manifest.json` | Run provenance and result summaries. |
|
||||||
| `rejected.json` | Rejected pipeline outputs. |
|
| `rejected.json` | Rejected pipeline outputs. |
|
||||||
| `warnings.json` | Accepted-output and run warnings. |
|
| `warnings.json` | Actionable process-degradation warnings. |
|
||||||
|
| `diagnostics.json` | Advisory and observation findings for accepted artifacts. |
|
||||||
|
|
||||||
The complete configuration also requests two pipeline-wide artifacts:
|
The complete configuration also requests two pipeline-wide artifacts:
|
||||||
|
|
||||||
|
|||||||
@@ -79,7 +79,7 @@ silently treated as fully reviewed by the caller.
|
|||||||
## Preserve Provenance And Handle Data Carefully
|
## Preserve Provenance And Handle Data Carefully
|
||||||
|
|
||||||
Keep the receipt with the published `manifest.json`, and retain
|
Keep the receipt with the published `manifest.json`, and retain
|
||||||
`rejected.json` and `warnings.json` when review or later provenance requires
|
`rejected.json`, `warnings.json`, and `diagnostics.json` when review or later provenance requires
|
||||||
them. Treat the input, output bundle, cache, debug bundle, and captured process
|
them. Treat the input, output bundle, cache, debug bundle, and captured process
|
||||||
logs as potentially sensitive data. Apply the caller's access controls and
|
logs as potentially sensitive data. Apply the caller's access controls and
|
||||||
retention policy, and avoid copying secrets into arguments, logs, or
|
retention policy, and avoid copying secrets into arguments, logs, or
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ Output configuration, including chunk-map and evidence-context publication, belo
|
|||||||
## Bundle Layout
|
## Bundle Layout
|
||||||
|
|
||||||
All paths below are logical, relative, slash-separated bundle paths. The
|
All paths below are logical, relative, slash-separated bundle paths. The
|
||||||
encoder always emits the first four JSON files below and adds lane or
|
encoder always emits the first five JSON files below and adds lane or
|
||||||
pipeline-wide artifact files when their corresponding artifacts are available:
|
pipeline-wide artifact files when their corresponding artifacts are available:
|
||||||
|
|
||||||
A subprocess caller first obtains the physical bundle root from the
|
A subprocess caller first obtains the physical bundle root from the
|
||||||
@@ -21,7 +21,8 @@ root for the logical discovery described here.
|
|||||||
| `index.json` | Entry point that names the other published files and lane payloads. |
|
| `index.json` | Entry point that names the other published files and lane payloads. |
|
||||||
| `manifest.json` | Run provenance and result summaries. |
|
| `manifest.json` | Run provenance and result summaries. |
|
||||||
| `rejected.json` | Rejected pipeline outputs. |
|
| `rejected.json` | Rejected pipeline outputs. |
|
||||||
| `warnings.json` | Accepted-output and run warnings. |
|
| `warnings.json` | Actionable process-degradation warnings. |
|
||||||
|
| `diagnostics.json` | Accepted-artifact quality advisories and normalization observations. |
|
||||||
| `lanes/<safe-lane-id>.json` | One normalized artifact payload for each lane. |
|
| `lanes/<safe-lane-id>.json` | One normalized artifact payload for each lane. |
|
||||||
| `chunk-map.json` | Optional accepted chunk map, when its export is enabled and available. |
|
| `chunk-map.json` | Optional accepted chunk map, when its export is enabled and available. |
|
||||||
| `evidence-context.json` | Optional selected source-unit excerpt, when evidence publication is enabled. |
|
| `evidence-context.json` | Optional selected source-unit excerpt, when evidence publication is enabled. |
|
||||||
@@ -39,7 +40,8 @@ normalized lanes has this valid minimal index:
|
|||||||
"manifest_file": "manifest.json",
|
"manifest_file": "manifest.json",
|
||||||
"output_files": [],
|
"output_files": [],
|
||||||
"rejected_file": "rejected.json",
|
"rejected_file": "rejected.json",
|
||||||
"warnings_file": "warnings.json"
|
"warnings_file": "warnings.json",
|
||||||
|
"diagnostics_file": "diagnostics.json"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -49,6 +51,7 @@ normalized lanes has this valid minimal index:
|
|||||||
| `output_files` | Yes | Lane descriptors sorted by `lane_id`. |
|
| `output_files` | Yes | Lane descriptors sorted by `lane_id`. |
|
||||||
| `rejected_file` | Yes | Always `rejected.json`. |
|
| `rejected_file` | Yes | Always `rejected.json`. |
|
||||||
| `warnings_file` | Yes | Always `warnings.json`. |
|
| `warnings_file` | Yes | Always `warnings.json`. |
|
||||||
|
| `diagnostics_file` | Yes | Always `diagnostics.json`. |
|
||||||
| `chunk_map` | No | Descriptor for the pipeline-wide `chunk-map.json`; never a lane descriptor. |
|
| `chunk_map` | No | Descriptor for the pipeline-wide `chunk-map.json`; never a lane descriptor. |
|
||||||
| `evidence_context` | No | Descriptor for the pipeline-wide `evidence-context.json`; never a lane descriptor. |
|
| `evidence_context` | No | Descriptor for the pipeline-wide `evidence-context.json`; never a lane descriptor. |
|
||||||
|
|
||||||
@@ -132,7 +135,7 @@ These values describe observed execution; they are not a backend-registration
|
|||||||
interface. Entries that differ by backend or effective reasoning remain
|
interface. Entries that differ by backend or effective reasoning remain
|
||||||
distinct even when their profile, provider, and model are otherwise equal.
|
distinct even when their profile, provider, and model are otherwise equal.
|
||||||
|
|
||||||
## Rejections And Warnings
|
## Rejections, Warnings, And Diagnostics
|
||||||
|
|
||||||
`rejected.json` is always an object with a `rejected` array. Each entry has
|
`rejected.json` is always an object with a `rejected` array. Each entry has
|
||||||
required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`,
|
required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`,
|
||||||
@@ -142,9 +145,47 @@ contain the bounded `validation` summary described above; the existing singular
|
|||||||
validator and reason fields remain the first configured rejection for
|
validator and reason fields remain the first configured rejection for
|
||||||
compatibility.
|
compatibility.
|
||||||
|
|
||||||
`warnings.json` is always an object with a `warnings` array. Each warning has
|
`warnings.json` is always the `notarius.warnings.v2` envelope:
|
||||||
`reason_code` and `message`; `scope` is optional. Both arrays are empty when
|
|
||||||
there is nothing to report.
|
```json
|
||||||
|
{
|
||||||
|
"schema_version": "notarius.warnings.v2",
|
||||||
|
"group_count": 0,
|
||||||
|
"occurrence_count": 0,
|
||||||
|
"groups": []
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
It contains only process warnings. `group_count` is exact, and
|
||||||
|
`occurrence_count` is the exact sum of its group occurrence counts.
|
||||||
|
|
||||||
|
`diagnostics.json` is always the `notarius.diagnostics.v1` envelope:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"schema_version": "notarius.diagnostics.v1",
|
||||||
|
"group_count": 0,
|
||||||
|
"occurrence_count": 0,
|
||||||
|
"truncated": false,
|
||||||
|
"unrepresented_occurrence_count": 0,
|
||||||
|
"groups": []
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
It contains only advisory and observation groups. `group_count` counts groups
|
||||||
|
represented in `groups`; `occurrence_count` includes both represented and
|
||||||
|
unrepresented occurrences. When `truncated` is true,
|
||||||
|
`unrepresented_occurrence_count` is the exact number omitted from group
|
||||||
|
representation.
|
||||||
|
|
||||||
|
Each group has `disposition`, `category`, `reason_code`, framework-owned
|
||||||
|
`origin`, exact `occurrence_count`, bounded `samples`, and
|
||||||
|
`omitted_sample_count`. Samples carry safe `scope` and `message`, plus a chunk
|
||||||
|
ID and zero-based chunk index when applicable. A group retains at most three
|
||||||
|
distinct samples. The framework fails rather than truncating actionable
|
||||||
|
warnings beyond 128 groups; it represents at most 256 advisory/observation
|
||||||
|
groups and records further occurrences through the diagnostic truncation
|
||||||
|
fields above.
|
||||||
|
|
||||||
## Compatibility
|
## Compatibility
|
||||||
|
|
||||||
|
|||||||
@@ -9,18 +9,22 @@ Command syntax, streams, and exit statuses are defined in the
|
|||||||
|
|
||||||
## Schema
|
## Schema
|
||||||
|
|
||||||
The current schema version is `notarius.run-result.v1`.
|
The current schema version is `notarius.run-result.v2`.
|
||||||
|
|
||||||
| Field | Required | Meaning |
|
| Field | Required | Meaning |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `schema_version` | Yes | Exactly `notarius.run-result.v1`. |
|
| `schema_version` | Yes | Exactly `notarius.run-result.v2`. |
|
||||||
| `run_id` | Yes | The finalized Notarius run identifier. |
|
| `run_id` | Yes | The finalized Notarius run identifier. |
|
||||||
| `pipeline_id` | Yes | The effective pipeline identifier. |
|
| `pipeline_id` | Yes | The effective pipeline identifier. |
|
||||||
| `output_directory` | Yes | Absolute path to the published, run-specific output bundle. |
|
| `output_directory` | Yes | Absolute path to the published, run-specific output bundle. |
|
||||||
| `index_file` | For the production JSON output | Logical path `index.json`; omitted for other output modules. |
|
| `index_file` | For the production JSON output | Logical path `index.json`; omitted for other output modules. |
|
||||||
| `normalized_output_count` | Yes | Number of final normalized outputs. |
|
| `normalized_output_count` | Yes | Number of final normalized outputs. |
|
||||||
| `rejected_output_count` | Yes | Number of recorded rejected outputs. |
|
| `rejected_output_count` | Yes | Number of recorded rejected outputs. |
|
||||||
| `warning_count` | Yes | Number of final run warnings. |
|
| `warning_group_count` | Yes | Exact number of actionable warning groups. |
|
||||||
|
| `warning_occurrence_count` | Yes | Exact occurrences represented by actionable warning groups. |
|
||||||
|
| `diagnostic_group_count` | Yes | Number of represented advisory and observation groups. |
|
||||||
|
| `diagnostic_occurrence_count` | Yes | Advisory and observation occurrences, including unrepresented occurrences. |
|
||||||
|
| `diagnostics_truncated` | Yes | Whether advisory/observation group representation was truncated. |
|
||||||
| `validation_status` | Yes | The final run manifest validation status. |
|
| `validation_status` | Yes | The final run manifest validation status. |
|
||||||
| `validation_summaries` | No | Bounded per-producer validation outcomes; present when producer work ran. |
|
| `validation_summaries` | No | Bounded per-producer validation outcomes; present when producer work ran. |
|
||||||
| `debug_directory` | No | Absolute path to the run-specific debug bundle when requested debug capture completed. |
|
| `debug_directory` | No | Absolute path to the run-specific debug bundle when requested debug capture completed. |
|
||||||
@@ -34,14 +38,18 @@ means one or more otherwise accepted results advanced under validator-failure
|
|||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"schema_version": "notarius.run-result.v1",
|
"schema_version": "notarius.run-result.v2",
|
||||||
"run_id": "run-1770000000000000000-0123456789abcdef0123456789abcdef",
|
"run_id": "run-1770000000000000000-0123456789abcdef0123456789abcdef",
|
||||||
"pipeline_id": "dnd-session",
|
"pipeline_id": "dnd-session",
|
||||||
"output_directory": "/work/results/run-1770000000000000000-0123456789abcdef0123456789abcdef",
|
"output_directory": "/work/results/run-1770000000000000000-0123456789abcdef0123456789abcdef",
|
||||||
"index_file": "index.json",
|
"index_file": "index.json",
|
||||||
"normalized_output_count": 6,
|
"normalized_output_count": 6,
|
||||||
"rejected_output_count": 2,
|
"rejected_output_count": 2,
|
||||||
"warning_count": 1,
|
"warning_group_count": 1,
|
||||||
|
"warning_occurrence_count": 2,
|
||||||
|
"diagnostic_group_count": 3,
|
||||||
|
"diagnostic_occurrence_count": 5,
|
||||||
|
"diagnostics_truncated": false,
|
||||||
"validation_status": "incomplete",
|
"validation_status": "incomplete",
|
||||||
"validation_summaries": [
|
"validation_summaries": [
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -149,7 +149,7 @@ rule. Configuration owns the exact validator key and chain position.
|
|||||||
Normalizers are deterministic for spells, combat turns, item occurrences, NPC
|
Normalizers are deterministic for spells, combat turns, item occurrences, NPC
|
||||||
occurrences, scene descriptions, enemy events, and location occurrences. They
|
occurrences, scene descriptions, enemy events, and location occurrences. They
|
||||||
canonicalize display values and evidence, use source-document order for stable
|
canonicalize display values and evidence, use source-document order for stable
|
||||||
output, and issue bounded warnings for changes or collapsed duplicates. NPC,
|
output, and emit bounded normalization observations for changes or collapsed duplicates. NPC,
|
||||||
item, and location registry normalizers are intentional exceptions: each first
|
item, and location registry normalizers are intentional exceptions: each first
|
||||||
produces a deterministic candidate set, then may use a bounded structured-LLM
|
produces a deterministic candidate set, then may use a bounded structured-LLM
|
||||||
proposal to reconcile identity groups.
|
proposal to reconcile identity groups.
|
||||||
|
|||||||
@@ -102,7 +102,7 @@ returned duplicate groups into a stable non-overlapping plan.
|
|||||||
The normalizer then applies that plan through a typed `ApplicationPolicy`. The
|
The normalizer then applies that plan through a typed `ApplicationPolicy`. The
|
||||||
core preserves ungrouped records, contribution order, and provenance while the
|
core preserves ungrouped records, contribution order, and provenance while the
|
||||||
artifact family owns group guards, field and evidence consolidation, durable
|
artifact family owns group guards, field and evidence consolidation, durable
|
||||||
ID derivation, retry and fallback presentation, warnings, and postconditions.
|
ID derivation, retry and fallback presentation, classified diagnostics, and postconditions.
|
||||||
Request-local handles do not enter the typed value or durable artifact. Fewer
|
Request-local handles do not enter the typed value or durable artifact. Fewer
|
||||||
than two eligible candidates skips model invocation; exceeding a candidate or
|
than two eligible candidates skips model invocation; exceeding a candidate or
|
||||||
combined-material bound preserves the deterministic result under the family's
|
combined-material bound preserves the deterministic result under the family's
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ own durable output shapes. Concrete production extensions are covered by
|
|||||||
The pipeline framework accepts a resolved composition, registries, shared
|
The pipeline framework accepts a resolved composition, registries, shared
|
||||||
dependencies, input bytes, a supplied prompt session, and state/debug
|
dependencies, input bytes, a supplied prompt session, and state/debug
|
||||||
collaborators. It returns logical output files, normalized artifacts, recorded
|
collaborators. It returns logical output files, normalized artifacts, recorded
|
||||||
rejections and warnings, manifest provenance, and checkpoint decisions. The
|
rejections, grouped diagnostics, manifest provenance, and checkpoint decisions. The
|
||||||
CLI owns process arguments, configuration discovery, session resolution,
|
CLI owns process arguments, configuration discovery, session resolution,
|
||||||
physical roots, and placement of returned output files.
|
physical roots, and placement of returned output files.
|
||||||
|
|
||||||
@@ -150,8 +150,8 @@ oversized aggregate is a framework contract error; guidance is never inferred
|
|||||||
or truncated.
|
or truncated.
|
||||||
|
|
||||||
The runner applies the binding's retry policy around a stage operation and its
|
The runner applies the binding's retry policy around a stage operation and its
|
||||||
complete validation chain. It preserves warnings only from the final accepted
|
complete validation chain. It preserves terminal diagnostics only from the final accepted
|
||||||
or rejected attempt, plus one fixed warning per validator whose execution
|
or rejected attempt, plus one fixed validation-incomplete warning per validator whose execution
|
||||||
budget was exhausted under `warn_continue`. Cancellation stops retries.
|
budget was exhausted under `warn_continue`. Cancellation stops retries.
|
||||||
Normalizer-specific retry directives consume this same budget and validate any
|
Normalizer-specific retry directives consume this same budget and validate any
|
||||||
final safe fallback through the normalizer chain.
|
final safe fallback through the normalizer chain.
|
||||||
@@ -171,13 +171,13 @@ Chunk planning uses this state machine for generated plans. A rejected or
|
|||||||
validation-incomplete automatic cache hit is not model material and therefore
|
validation-incomplete automatic cache hit is not model material and therefore
|
||||||
falls through to a fresh initial generation at producer attempt one; it neither
|
falls through to a fresh initial generation at producer attempt one; it neither
|
||||||
receives a correction, consumes retry budget, promotes cached-candidate
|
receives a correction, consumes retry budget, promotes cached-candidate
|
||||||
warnings, nor overwrites the stored record. An incomplete cache validation
|
diagnostics, nor overwrites the stored record. An incomplete cache validation
|
||||||
under `fail_run` terminates instead. Only a newly generated, completely
|
under `fail_run` terminates instead. Only a newly generated, completely
|
||||||
validated plan is published to the chunk-plan store. Rejected plans never
|
validated plan is published to the chunk-plan store. Rejected plans never
|
||||||
advance, and validation-incomplete plans remain unpublishable.
|
advance, and validation-incomplete plans remain unpublishable.
|
||||||
|
|
||||||
After terminal lane work, the runner assembles manifest provenance, normalized
|
After terminal lane work, the runner assembles manifest provenance, normalized
|
||||||
artifacts, rejections, warnings, and an optional accepted chunk map. When an
|
artifacts, rejections, final grouped diagnostics, and an optional accepted chunk map. When an
|
||||||
output policy selected evidence lanes, it decodes accepted serialized normalize
|
output policy selected evidence lanes, it decodes accepted serialized normalize
|
||||||
outputs through their registered codecs and invokes the prepared typed
|
outputs through their registered codecs and invokes the prepared typed
|
||||||
projectors. Rejected or absent lanes contribute nothing. This reconstruction is
|
projectors. Rejected or absent lanes contribute nothing. This reconstruction is
|
||||||
|
|||||||
@@ -64,12 +64,13 @@ Ordinary resume loads extract, merge, and normalize checkpoints progressively
|
|||||||
and may execute later lane stages after an earlier cache miss. Selective
|
and may execute later lane stages after an earlier cache miss. Selective
|
||||||
recomputation instead asks the loader for the required producer's accepted
|
recomputation instead asks the loader for the required producer's accepted
|
||||||
normalize artifact. That lookup reuses the existing normalize files, requires
|
normalize artifact. That lookup reuses the existing normalize files, requires
|
||||||
workspace schema v3 plus an exact non-empty invocation identity, and deliberately
|
workspace schema v4 plus an exact non-empty invocation identity, and deliberately
|
||||||
does not require extract or merge checkpoint files or dependency fingerprints.
|
does not require extract or merge checkpoint files or dependency fingerprints.
|
||||||
The runner performs canonical codec and producer-provenance validation before
|
The runner performs canonical codec and producer-provenance validation before
|
||||||
cloning the artifact into normal step output. Success restores only stored
|
cloning the artifact into normal step output. Success restores only stored
|
||||||
normalize warnings and emits one normalize decision; failure retains the files,
|
normalize diagnostics and emits one normalize decision; failure retains the
|
||||||
records the decision, and stops without executing the producer or consumer.
|
files, records the decision, and stops without executing the producer or
|
||||||
|
consumer.
|
||||||
|
|
||||||
The loader assigns a typed category and reason code at each validation site;
|
The loader assigns a typed category and reason code at each validation site;
|
||||||
diagnostic prose is not classified after the fact. The runner then applies
|
diagnostic prose is not classified after the fact. The runner then applies
|
||||||
@@ -97,7 +98,7 @@ owns the operator workflow and stable reason-code meanings.
|
|||||||
|
|
||||||
`internal/core/debugbundle` allocates an explicitly requested per-run bundle
|
`internal/core/debugbundle` allocates an explicitly requested per-run bundle
|
||||||
with `summary/` and `trace/` roots. `SummaryWriter` persists redacted command,
|
with `summary/` and `trace/` roots. `SummaryWriter` persists redacted command,
|
||||||
resolution, run, warning, and failure artifacts. `internal/framework/debug`
|
resolution, run, final grouped diagnostic, and failure artifacts. `internal/framework/debug`
|
||||||
implements the pipeline-facing trace recorder under the trace root.
|
implements the pipeline-facing trace recorder under the trace root.
|
||||||
|
|
||||||
The CLI allocates a bundle before pipeline resolution and treats requested
|
The CLI allocates a bundle before pipeline resolution and treats requested
|
||||||
|
|||||||
@@ -105,8 +105,10 @@ and resolves configuration before module preparation and source parsing. It
|
|||||||
then performs any permitted cache lookup, executes the pipeline, and publishes
|
then performs any permitted cache lookup, executes the pipeline, and publishes
|
||||||
logical output files only after a successful runner result.
|
logical output files only after a successful runner result.
|
||||||
|
|
||||||
On success, the command reports the output bundle path. A warning-bearing run
|
On success, the command reports the output bundle path. A run with actionable
|
||||||
still succeeds and reports its warning count on standard error. Errors and
|
process warnings still succeeds and reports warning-group and occurrence counts
|
||||||
|
on standard error; advisory and observation findings do not produce a warning
|
||||||
|
line. Errors and
|
||||||
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).
|
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).
|
||||||
|
|
||||||
## Validation Retries And Terminal Outcomes
|
## Validation Retries And Terminal Outcomes
|
||||||
@@ -127,13 +129,14 @@ validator execution failure normally uses `warn_continue`, which keeps an
|
|||||||
otherwise accepted result in the current run with `incomplete` validation
|
otherwise accepted result in the current run with `incomplete` validation
|
||||||
provenance. It emits one bounded warning for every validator whose execution
|
provenance. It emits one bounded warning for every validator whose execution
|
||||||
budget was exhausted. A corrected result that later passes validation does not
|
budget was exhausted. A corrected result that later passes validation does not
|
||||||
retain abandoned-attempt warnings.
|
retain diagnostics from abandoned attempts.
|
||||||
|
|
||||||
Treat a successful process exit as a completed run, not as proof that every
|
Treat a successful process exit as a completed run, not as proof that every
|
||||||
candidate was fully validated. Inspect the receipt's `validation_status`,
|
candidate was fully validated. Inspect the receipt's `validation_status`,
|
||||||
`validation_summaries`, rejection count, and warning count when an orchestrator
|
`validation_summaries`, rejection count, and warning group and occurrence
|
||||||
requires complete validation. The durable fields and their meanings are owned
|
counts when an orchestrator requires complete validation. The durable fields
|
||||||
by the [run-result receipt](integrations/run-result.md) and
|
and their meanings are owned by the
|
||||||
|
[run-result receipt](integrations/run-result.md) and
|
||||||
[published JSON output contract](integrations/json-output.md).
|
[published JSON output contract](integrations/json-output.md).
|
||||||
|
|
||||||
## Output Bundles
|
## Output Bundles
|
||||||
@@ -272,7 +275,7 @@ Only a [debug-enabled run](cli.md#run) creates a bundle:
|
|||||||
~~~
|
~~~
|
||||||
|
|
||||||
The summary contains redacted invocation and resolution information plus run,
|
The summary contains redacted invocation and resolution information plus run,
|
||||||
warning, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt
|
final grouped diagnostic, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt
|
||||||
terminal records contain bounded attempt kinds, validator outcomes, policy,
|
terminal records contain bounded attempt kinds, validator outcomes, policy,
|
||||||
decision, PromptKit repair count, and usage; they do not contain assistant
|
decision, PromptKit repair count, and usage; they do not contain assistant
|
||||||
responses or complete correction messages. The trace contains allowlisted
|
responses or complete correction messages. The trace contains allowlisted
|
||||||
|
|||||||
@@ -146,8 +146,9 @@ lanes, validators, and LLM profile: the canonical source digest selects the
|
|||||||
plan, while the current run still applies its configured chunk validators to
|
plan, while the current run still applies its configured chunk validators to
|
||||||
the materialized chunks.
|
the materialized chunks.
|
||||||
|
|
||||||
The framework owns orchestration and handoff provenance. Modules return logical
|
The framework owns orchestration, origin enrichment, aggregation, and handoff
|
||||||
results and warnings; they do not own CLI reporting, physical output, cache, or
|
provenance. Modules return logical results and classified diagnostics; they do
|
||||||
|
not own CLI reporting, physical output, cache, or
|
||||||
debug roots, durable file placement, or checkpoint and debug lifecycle.
|
debug roots, durable file placement, or checkpoint and debug lifecycle.
|
||||||
|
|
||||||
After pipeline-wide chunking, extraction uses bounded framework concurrency.
|
After pipeline-wide chunking, extraction uses bounded framework concurrency.
|
||||||
@@ -159,17 +160,26 @@ may overlap. The framework must not create unbounded goroutines per lane or
|
|||||||
chunk.
|
chunk.
|
||||||
|
|
||||||
Completion timing does not choose public ordering or errors. The coordinator
|
Completion timing does not choose public ordering or errors. The coordinator
|
||||||
orders accepted artifacts, warnings, rejections, checkpoint events, and
|
orders accepted artifacts, grouped diagnostics, rejections, checkpoint events, and
|
||||||
framework errors by stable pipeline scope. Rejections do not cancel unrelated
|
framework errors by stable pipeline scope. Rejections do not cancel unrelated
|
||||||
work. A framework error cancels derived work, prevents undispatched work from
|
work. A framework error cancels derived work, prevents undispatched work from
|
||||||
starting, waits for started work, and prevents output encoding.
|
starting, waits for started work, and prevents output encoding.
|
||||||
|
|
||||||
|
Warnings are process-only signals: configuration degradation, approved fallback,
|
||||||
|
or incomplete configured validation. Quality uncertainty and grounding findings
|
||||||
|
are advisories; successful canonicalization and cleanup are observations.
|
||||||
|
Modules choose that semantic classification, while the framework attaches
|
||||||
|
origin, aggregates groups, enforces bounds, and presents final collections.
|
||||||
|
An ordinary successful run therefore has zero warnings. See
|
||||||
|
[ADR-0015](../adr/0015-separate-process-warnings-from-quality-diagnostics.md)
|
||||||
|
for the decision rationale.
|
||||||
|
|
||||||
## Validation
|
## Validation
|
||||||
|
|
||||||
Validation is a framework-managed boundary around outputs from chunk, extract,
|
Validation is a framework-managed boundary around outputs from chunk, extract,
|
||||||
merge, and normalize stages. Validators receive immutable stage output and
|
merge, and normalize stages. Validators receive immutable stage output and
|
||||||
make an explicit whole-output decision: approve, approve with warnings,
|
make an explicit whole-output decision: approve, reject, fail, or skip when a
|
||||||
reject, fail, or skip when a runtime prerequisite is unavailable.
|
runtime prerequisite is unavailable.
|
||||||
|
|
||||||
Typed artifact validators receive the domain value directly. Chunk validators
|
Typed artifact validators receive the domain value directly. Chunk validators
|
||||||
receive source-zone chunks, while serialized validators receive immutable
|
receive source-zone chunks, while serialized validators receive immutable
|
||||||
|
|||||||
85
docs/releases/v0.5.0.md
Normal file
85
docs/releases/v0.5.0.md
Normal file
@@ -0,0 +1,85 @@
|
|||||||
|
# Notarius v0.5.0
|
||||||
|
|
||||||
|
This release separates actionable process warnings from extraction-quality
|
||||||
|
advisories and routine normalization observations, giving operators a quiet
|
||||||
|
warning channel without discarding durable diagnostic detail.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Notarius now carries one validated, origin-aware diagnostic contract from
|
||||||
|
producers and validators through retries, reusable state, output publication,
|
||||||
|
debug summaries, run receipts, and CLI presentation. Warnings are reserved for
|
||||||
|
process degradation or incomplete configured work. Data-quality findings are
|
||||||
|
advisories, and successful deterministic cleanup is recorded as observations.
|
||||||
|
An ordinary successful run therefore reports zero warnings while retaining
|
||||||
|
bounded diagnostic provenance for later review.
|
||||||
|
|
||||||
|
The framework aggregates findings deterministically by their stable identity
|
||||||
|
and complete pipeline origin, preserves exact occurrence counts, and retains
|
||||||
|
bounded representative samples. Warning groups fail rather than truncate;
|
||||||
|
advisory and observation representation is bounded with explicit truncation
|
||||||
|
metadata and exact unrepresented-occurrence counts.
|
||||||
|
|
||||||
|
## Compatibility
|
||||||
|
|
||||||
|
- `warnings.json` now uses the incompatible grouped
|
||||||
|
`notarius.warnings.v2` envelope and contains process warnings only. Consumers
|
||||||
|
of the former flat warning payload must migrate to the current
|
||||||
|
[JSON output contract](../integrations/json-output.md).
|
||||||
|
- The new `diagnostics.json` file uses `notarius.diagnostics.v1` and contains
|
||||||
|
advisory and observation groups. Production `index.json` files always expose
|
||||||
|
both `warnings_file` and `diagnostics_file`.
|
||||||
|
- The machine-readable run receipt is now `notarius.run-result.v2`. It replaces
|
||||||
|
`warning_count` with exact warning group and occurrence counts and adds
|
||||||
|
advisory/observation group, occurrence, and truncation fields. See the
|
||||||
|
current [run-result receipt](../integrations/run-result.md).
|
||||||
|
- Custom output modules must return their complete logical file set or an
|
||||||
|
error. The former `OutputResult.Warnings` field has been removed; an output
|
||||||
|
module cannot report a warning after serializing its output.
|
||||||
|
- Reusable state now uses `notarius.workspace.v4` and chunk-plan records use
|
||||||
|
`notarius.chunk-plan.v3` so they can preserve structured diagnostics. Older
|
||||||
|
pre-release reusable state is not reused under these contracts; start with
|
||||||
|
clean state when deterministic continuity with an older workspace is not
|
||||||
|
required.
|
||||||
|
- Validation acceptance, semantic retry budgets, rejection policy, and D&D
|
||||||
|
artifact schema identities are unchanged by this release.
|
||||||
|
|
||||||
|
## Upgrade
|
||||||
|
|
||||||
|
1. Update subprocess consumers to require `notarius.run-result.v2` and read
|
||||||
|
`warning_group_count`, `warning_occurrence_count`,
|
||||||
|
`diagnostic_group_count`, `diagnostic_occurrence_count`, and
|
||||||
|
`diagnostics_truncated`.
|
||||||
|
2. Update output-bundle consumers to decode `notarius.warnings.v2`, discover
|
||||||
|
`diagnostics.json` through `index.json`, and treat diagnostics as review
|
||||||
|
information rather than process warnings.
|
||||||
|
3. Update any custom output module for the removal of
|
||||||
|
`OutputResult.Warnings`; return an error when encoding cannot complete.
|
||||||
|
4. Clear pre-release reusable state before the first upgraded production run
|
||||||
|
when deterministic continuity with an older workspace is not required.
|
||||||
|
5. Run `notarius config validate --config <path> --pipeline <id>` and perform
|
||||||
|
one representative run before promoting the release in an automated
|
||||||
|
pipeline.
|
||||||
|
|
||||||
|
## Changes
|
||||||
|
|
||||||
|
- Added validated diagnostic dispositions, categories, origins, stable reason
|
||||||
|
codes, exact occurrence counts, and bounded representative samples.
|
||||||
|
- Added deterministic run-level aggregation with separate limits for
|
||||||
|
actionable warning groups and advisory/observation groups.
|
||||||
|
- Reclassified D&D source-relatedness and unresolved-identity findings as
|
||||||
|
data-quality advisories and routine normalization changes as observations.
|
||||||
|
- Preserved structured diagnostics across producer retries, validation,
|
||||||
|
generated-reference handoff, checkpoints, chunk-plan reuse, and debug
|
||||||
|
summaries while discarding superseded-attempt findings.
|
||||||
|
- Added grouped `warnings.json`, a new grouped `diagnostics.json`, and the
|
||||||
|
corresponding production index entries.
|
||||||
|
- Upgraded the machine-readable run receipt and human CLI summary to report
|
||||||
|
exact warning and diagnostic counts without allowing advisory volume to
|
||||||
|
create warning output.
|
||||||
|
- Removed post-encoding output warnings and hardened diagnostic validation,
|
||||||
|
overflow handling, aggregate memory bounds, and warning-file path
|
||||||
|
presentation.
|
||||||
|
- Documented diagnostic ownership, classification, operator interpretation,
|
||||||
|
durable contracts, and architectural invariants in ADR-0015 and the
|
||||||
|
canonical CLI, operations, integration, and internal documentation.
|
||||||
648
docs/roadmap/archive/warning-signal-and-presentation-audit.md
Normal file
648
docs/roadmap/archive/warning-signal-and-presentation-audit.md
Normal file
@@ -0,0 +1,648 @@
|
|||||||
|
# Warning Signal And Presentation Audit
|
||||||
|
|
||||||
|
## Executive Assessment
|
||||||
|
|
||||||
|
Notarius warning execution is mechanically stronger than its operator-facing
|
||||||
|
presentation. Terminal-attempt promotion, stable ordering after concurrent
|
||||||
|
work, checkpoint replay, validation summaries, and debug retention are all
|
||||||
|
substantially correct. The audit found no general duplicate-append defect in
|
||||||
|
the extract, merge, or normalize handoffs and no leakage of abandoned-attempt
|
||||||
|
warnings into a successful result.
|
||||||
|
|
||||||
|
The warning channel itself is not coherent. One flat `contracts.Warning` type
|
||||||
|
currently represents at least four materially different concepts:
|
||||||
|
|
||||||
|
- actionable degradation or incomplete validation;
|
||||||
|
- heuristic data-quality doubt;
|
||||||
|
- successful but potentially reviewable fallback; and
|
||||||
|
- routine canonicalization, ordering, and deduplication observations.
|
||||||
|
|
||||||
|
That conflation is the primary reason successful runs produce a count that is
|
||||||
|
large but operationally weak. The maintained complete D&D example demonstrates
|
||||||
|
the problem without a live provider: an approved run with no rejected outputs
|
||||||
|
published 12 warning records, all from three advisory relatedness checks. An
|
||||||
|
operator separately reported a successful complete D&D run with 10 outputs,
|
||||||
|
one rejection, and 85 warnings. The production bundle for that run was not
|
||||||
|
available in this environment, so its reason-code distribution could not be
|
||||||
|
measured.
|
||||||
|
|
||||||
|
The current implementation also has four correctness or robustness gaps:
|
||||||
|
|
||||||
|
1. warning records lose stage, step, lane, module, validator, and chunk
|
||||||
|
provenance when promoted, which makes safe aggregation and diagnosis
|
||||||
|
impossible from `warnings.json` alone;
|
||||||
|
2. there is no framework-level validation or aggregate bound, and the NPC- and
|
||||||
|
spell-relatedness validators bypass the D&D warning limiter entirely;
|
||||||
|
3. warnings returned by an output encoder are added after `warnings.json` has
|
||||||
|
already been encoded, so the receipt, stderr, debug bundle, and published
|
||||||
|
warning file can disagree; and
|
||||||
|
4. a skipped validator contributes to `incomplete` validation but does not
|
||||||
|
receive the warning generated for an exhausted validator failure.
|
||||||
|
|
||||||
|
The recommended end state is a structured diagnostic contract with explicit
|
||||||
|
disposition, category, origin, occurrence count, and bounded samples. Warnings
|
||||||
|
are reserved for process-level degradation or incompleteness. LLM-judged or
|
||||||
|
deterministically inferred extraction-quality signals are advisories, never
|
||||||
|
warnings, and routine normalization observations remain inspectable without
|
||||||
|
being reported as top-level warnings. An ordinary successful run in which all
|
||||||
|
configured work completes normally should therefore report zero warnings. This
|
||||||
|
is an architectural and durable-contract change, not merely revised CLI prose.
|
||||||
|
|
||||||
|
## Evidence And Limits
|
||||||
|
|
||||||
|
The audit used:
|
||||||
|
|
||||||
|
- a complete static search of production `contracts.Warning` constructors,
|
||||||
|
reason-code constants, result fields, and promotion sites under `internal/`;
|
||||||
|
- call-path inspection through producer attempts, validators, lane
|
||||||
|
coordination, chunk-plan reuse, checkpoints, output encoding, debug output,
|
||||||
|
CLI presentation, and run-result construction;
|
||||||
|
- the maintained complete and minimal D&D examples with offline fake LLMs;
|
||||||
|
- focused deterministic tests for warning bounds, semantic-reconciliation
|
||||||
|
fallback, `warn_continue`, semantic retries, terminal rejection, concurrency
|
||||||
|
ordering, and checkpoint reuse; and
|
||||||
|
- the operator-provided observation of an 85-warning complete D&D run.
|
||||||
|
|
||||||
|
No provider-backed production run was attempted because this environment has
|
||||||
|
no API key. Consequently, the audit can establish warning mechanics, possible
|
||||||
|
multiplicity, synthetic volume, and obvious heuristic limitations, but cannot
|
||||||
|
estimate production frequency or the real false-positive rate of individual
|
||||||
|
D&D advisories. Those measurements are not required to choose the recommended
|
||||||
|
architecture; they are required before strengthening any heuristic advisory
|
||||||
|
into a rejection or setting a numerical production acceptance target.
|
||||||
|
|
||||||
|
## Complete Warning-Producer Inventory
|
||||||
|
|
||||||
|
### Framework And Generic Boundaries
|
||||||
|
|
||||||
|
| Producer | Reason code | Trigger and consequence | Multiplicity and bound | Current surfaces and coverage |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| Reference materialization in `internal/framework/pipeline/references.go` | `empty_reference` | A bound external reference is a valid, accepted media type but contains zero bytes. The prompt may receive materially incomplete context. | One per empty bound file; finite by configuration but no shared run-level cap. | Enters `RunInput.Warnings`; reference tests protect contextual scope. |
|
||||||
|
| Producer-attempt policy in `internal/framework/pipeline/producer_attempts.go` | `validator_execution_incomplete` | An applicable validator exhausted its execution budget and `warn_continue` accepted the otherwise valid candidate. | One per failed validator on each terminal candidate. An extract chain can multiply this by chunks and lanes. There is no global cap. | Durable warning, receipt count, stderr, debug, and validation summary. `TestWarnContinueRecordsOneWarningForEachExhaustedValidator` covers failures. |
|
||||||
|
| Chunk, extract, merge, normalize, and output module result contracts | Module-defined | A module may return arbitrary warnings with its successful candidate. | No contract validation, message limit, per-result cap, or global cap. Current production modules are inventoried below. | Terminal-attempt filtering and concurrency ordering are well tested. |
|
||||||
|
| Production JSON output encoder | None | The encoder copies incoming warnings into `warnings.json`; it does not currently create warnings. | Same count as its input. | JSON encoder and assembled-pipeline tests compare the incoming run warnings with the published file. |
|
||||||
|
| Output encoder result contract | Module-defined | Any output encoder may return warnings discovered during encoding. | Unbounded by contract. No production encoder currently exercises this capability. | Appended to final `RunOutput` only after logical files were encoded; this is the cross-surface defect described in AUD-WARN-004. |
|
||||||
|
|
||||||
|
Input adapters and production mergers do not currently have independent
|
||||||
|
warning producers. Chunk-plan and checkpoint decisions are structured manifest
|
||||||
|
or debug provenance rather than warnings. Cancellation and hard persistence,
|
||||||
|
reference, parsing, serialization, and provider failures remain errors.
|
||||||
|
|
||||||
|
### D&D Extraction Gates
|
||||||
|
|
||||||
|
| Producer | Reason code | Trigger and consequence | Multiplicity and bound |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `dnd/combat-turns` extractor | `scene_classification_unavailable` | The chunk has no exact matching scene-description classification. The extractor returns an empty accepted result and skips the LLM, so combat-turn output may be incomplete. | At most one per chunk for this lane. |
|
||||||
|
| `dnd/enemy-events` extractor | `scene_classification_unavailable` | The same missing or mismatched scene gate causes accepted empty enemy-event output. | At most one per chunk for this lane. |
|
||||||
|
|
||||||
|
An exact non-combat classification produces an intentional empty result without
|
||||||
|
a warning. An exact combat classification proceeds normally. The two producers
|
||||||
|
share a code and operator consequence but use different messages; their module
|
||||||
|
origins are not retained in the final warning record.
|
||||||
|
|
||||||
|
### D&D Source-Relatedness Validators
|
||||||
|
|
||||||
|
All ten relatedness validators are deterministic advisories: they approve the
|
||||||
|
candidate and warn when contextual prose or an entity name is not lexically
|
||||||
|
present in cited text. Shape and source-reference failures are deliberately
|
||||||
|
left to blocking validators earlier in the chain. The same relatedness
|
||||||
|
validator is registered in both the extract and normalize default chain for
|
||||||
|
each artifact family in `internal/modules/dnd/register/chains.go`.
|
||||||
|
|
||||||
|
| Artifact family | Warning reason | Per-record trigger | Local bound | Omission reason |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| Combat turns | `combat_turn_not_near_source` | Actor token sequence absent | 20 per validator invocation | `combat_turn_relatedness_warnings_omitted` |
|
||||||
|
| Enemy events | `enemy_event_not_near_source` | Subject token sequence absent | 20 | `enemy_event_relatedness_warnings_omitted` |
|
||||||
|
| Item occurrences | `item_occurrence_source_unrelated` | Item name token sequence absent | 20 | `item_occurrence_relatedness_warnings_omitted` |
|
||||||
|
| Item registry | `item_not_near_source` | Item name token sequence absent | 20 | `item_relatedness_warnings_omitted` |
|
||||||
|
| Location occurrences | `location_occurrence_not_near_source` | Location name token sequence absent | 20 | `location_occurrence_relatedness_warnings_omitted` |
|
||||||
|
| Location registry | `location_not_near_source` | Location name token sequence absent | 20 | `location_relatedness_warnings_omitted` |
|
||||||
|
| NPC occurrences | `npc_occurrence_not_near_source` | NPC name token sequence absent | 20 | `npc_occurrence_relatedness_warnings_omitted` |
|
||||||
|
| NPC registry | `npc_not_near_source` | NPC name token sequence absent | **Unbounded** | None |
|
||||||
|
| Scene descriptions | `scene_description_not_near_source` | No significant title or summary token appears; up to two findings per scene | 20 | `scene_description_relatedness_warnings_omitted` |
|
||||||
|
| Spells | `spell_not_near_source` | Spell-name token sequence absent | **Unbounded** | None |
|
||||||
|
|
||||||
|
The eight limiter-generated omission records are presentation artifacts, not
|
||||||
|
new source-relatedness conditions. They occupy a warning slot and make list
|
||||||
|
length differ from the actual occurrence count.
|
||||||
|
|
||||||
|
### D&D Normalizers
|
||||||
|
|
||||||
|
Every production D&D normalizer bounds its returned warning slice to 20 through
|
||||||
|
`internal/modules/dnd/shared/diagnostics`, including a final omission record
|
||||||
|
when needed. Registry semantic retries reserve space for their fallback
|
||||||
|
warning. The following table is complete by semantically distinct condition;
|
||||||
|
codes listed together are parallel artifact-family variants.
|
||||||
|
|
||||||
|
| Condition | Reason codes | Result impact | Current classification assessment |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| Display or field whitespace/name canonicalization | `npc_fields_normalized`, `item_fields_normalized`, `location_fields_normalized`, `spell_name_canonicalized`, `combat_actor_canonicalized`, `enemy_event_name_canonicalized`, `item_occurrence_name_canonicalized`, `location_occurrence_name_canonicalized`, `scene_description_prose_normalized` | Deterministic successful mutation. The item-occurrence code can also describe `from`/`to` whitespace, not only the item name. | Routine observation. |
|
||||||
|
| Durable ID recomputation | `npc_id_recomputed`, `item_id_recomputed`, `location_id_recomputed` | Restores the deterministic name-derived ID. | Routine observation; invalid identity is separately rejected by default chains. |
|
||||||
|
| Source-reference sorting or deduplication | `source_references_normalized` | Sorts and removes exact duplicate references while deliberately preserving invalid references for their validators. | Routine observation. Shared code is useful but ambiguous without producer origin. |
|
||||||
|
| Canonical record ordering | `combat_turns_reordered`, `enemy_events_reordered`, `item_occurrences_reordered`, `location_occurrences_reordered`, `npc_occurrences_reordered`, `scene_description_order_normalized` | Deterministic order changes only. | Routine observation. |
|
||||||
|
| Exact or approved semantic duplicate consolidation | `duplicate_npc_collapsed`, `duplicate_item_collapsed`, `duplicate_location_collapsed`, `duplicate_spell_cast_collapsed`, `duplicate_combat_turn_collapsed`, `duplicate_enemy_event_collapsed`, `duplicate_item_occurrence_collapsed`, `duplicate_location_occurrence_collapsed`, `duplicate_npc_occurrence_collapsed`, `scene_description_duplicate_collapsed` | Removes duplicate records and preserves or combines canonical evidence according to the artifact policy. Registry codes cover both exact and accepted semantic consolidation. | Durable normalization observation; not normally operator-actionable. |
|
||||||
|
| Unresolved external membership | `spell_name_unresolved`, `item_occurrence_unknown_item_id`, `location_occurrence_unknown_location_id` | The value is preserved but is not grounded in the effective catalog or registry. Default chains normally reject the same condition before normalization; it remains reachable with validator overrides or defensive direct use. | Actionable data-quality warning. |
|
||||||
|
| Unsafe currency consolidation proposal | `item_semantic_proposal_invalid` | The proposed group is rejected and all records are preserved because denominations or currency/non-currency members are incompatible. The same code is also used internally as a retry reason. | Advisory about model proposal quality; no accepted-data loss. The control and diagnostic meanings should be separated. |
|
||||||
|
| Semantic reconciliation unavailable or exhausted | `npc_semantic_reconciliation_exhausted`, `item_semantic_reconciliation_exhausted`, `location_semantic_reconciliation_exhausted` | The safe deterministic result is accepted, but possible semantic duplicates remain. | Actionable fallback warning. |
|
||||||
|
| Local warning truncation | `npc_normalization_warnings_omitted`, `item_normalization_warnings_omitted`, `location_normalization_warnings_omitted`, `spell_normalization_warnings_omitted`, `combat_turn_normalization_warnings_omitted`, `enemy_event_normalization_warnings_omitted`, `item_occurrence_normalization_warnings_omitted`, `location_occurrence_normalization_warnings_omitted`, `npc_occurrence_normalization_warnings_omitted`, `scene_description_normalization_warnings_omitted` | Reports that individual records were omitted from presentation. | Group metadata, not an independent warning. |
|
||||||
|
|
||||||
|
No production D&D normalization warning exposes raw model responses,
|
||||||
|
correction guidance, or provider errors. Most dynamic names are quoted and
|
||||||
|
truncated by the shared helper. That local discipline is not enforced by the
|
||||||
|
generic warning contract, and the spell relatedness message does not use the
|
||||||
|
shared truncation helper.
|
||||||
|
|
||||||
|
## Warning Propagation And Surface Map
|
||||||
|
|
||||||
|
```text
|
||||||
|
external-reference warnings -----------------------------+
|
||||||
|
|
|
||||||
|
module candidate warnings -> validation chain warnings |
|
||||||
|
| | |
|
||||||
|
+---- producer-attempt terminal policy -------+
|
||||||
|
| |
|
||||||
|
accepted / terminal rejection only |
|
||||||
|
| |
|
||||||
|
chunk or lane result in canonical order |
|
||||||
|
| |
|
||||||
|
checkpoint record/replay and ordered step merge |
|
||||||
|
| |
|
||||||
|
RunOutput.Warnings <------------+
|
||||||
|
|
|
||||||
|
OutputRequest -> output encoder
|
||||||
|
| |
|
||||||
|
warnings.json OutputResult.Warnings
|
||||||
|
|
|
||||||
|
appended to final RunOutput only
|
||||||
|
|
|
||||||
|
receipt, stderr, final debug warning summary
|
||||||
|
```
|
||||||
|
|
||||||
|
### Attempts And Validation
|
||||||
|
|
||||||
|
- `runProducerAttempts` promotes only the terminal accepted or terminal
|
||||||
|
rejected candidate's module and completed-validator warnings. Operational,
|
||||||
|
structural, semantic, and module-directed attempts that are superseded are
|
||||||
|
retained in attempt debug artifacts but not in the final collection.
|
||||||
|
- A module-directed semantic-reconciliation retry adds its fallback warning
|
||||||
|
only when no retry remains. Earlier attempt warnings are discarded.
|
||||||
|
- On `warn_continue`, warnings from the otherwise accepted candidate and
|
||||||
|
completed approved or rejected validators are retained. One fixed,
|
||||||
|
non-sensitive `validator_execution_incomplete` warning is added for every
|
||||||
|
failed validator. Skipped validators affect the validation summary and final
|
||||||
|
`incomplete` status but do not receive such a warning.
|
||||||
|
- A semantic terminal rejection retains only warnings from that rejected
|
||||||
|
attempt. Structural rejection after producer failure cannot retain a
|
||||||
|
candidate warning because no valid candidate result exists.
|
||||||
|
|
||||||
|
These behaviors are protected by the producer-attempt, extract-handoff,
|
||||||
|
rejection-warning, normalize-retry, and attempt-debug tests. They are the right
|
||||||
|
foundation for the redesign and should not be replaced with early-exit or
|
||||||
|
all-attempt accumulation.
|
||||||
|
|
||||||
|
### Concurrency And Ordering
|
||||||
|
|
||||||
|
Extract jobs are dispatched chunk-first and lane-second. Results are stored by
|
||||||
|
chunk index, finalized in ascending chunk order, and lane continuations are
|
||||||
|
merged into a slice indexed by configured lane order. Pipeline steps run in
|
||||||
|
configured order. The resulting public order is therefore:
|
||||||
|
|
||||||
|
1. pre-run reference warnings;
|
||||||
|
2. chunk-stage warnings;
|
||||||
|
3. step order;
|
||||||
|
4. configured lane order within each step;
|
||||||
|
5. chunk order within extract;
|
||||||
|
6. merge warnings; then
|
||||||
|
7. normalize warnings; followed by any output-result warnings.
|
||||||
|
|
||||||
|
`TestRunnerBoundsExtractJobsAndStabilizesReverseCompletion` exercises warning
|
||||||
|
order under reversed completion. No completion-order leak was found.
|
||||||
|
|
||||||
|
### Chunk Plans And Checkpoints
|
||||||
|
|
||||||
|
- A reusable chunk plan stores producer warnings only. Current validators run
|
||||||
|
again, and their current warnings are appended. Warnings from a cached plan
|
||||||
|
candidate that fails current validation are discarded before regeneration.
|
||||||
|
- Accepted extract, merge, and normalize checkpoints store the terminal
|
||||||
|
warnings for their stage. Reuse loads and appends those warnings once at the
|
||||||
|
same logical handoff. Tests compare fresh and resumed warning collections and
|
||||||
|
preserve their order.
|
||||||
|
- Validation-incomplete accepted outputs are not reusable, preventing a later
|
||||||
|
run from silently treating incomplete validation as complete.
|
||||||
|
- Required accepted-normalize hydration replays that normalize checkpoint's
|
||||||
|
warnings; checkpoint decisions separately expose that reuse occurred.
|
||||||
|
|
||||||
|
The recommended redesign should keep fresh and resumed logical diagnostics
|
||||||
|
equivalent. Whether a result was reused belongs in checkpoint provenance, not
|
||||||
|
in the diagnostic grouping key; adding a `reused` distinction would fragment
|
||||||
|
groups and make equivalent runs present differently.
|
||||||
|
|
||||||
|
### Terminal Surfaces
|
||||||
|
|
||||||
|
| Surface | Current content | Audience | Audit result |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `RunOutput.Warnings` | Flat final slice | Framework and CLI | Canonical in-memory list, but lacks origin and bounds. |
|
||||||
|
| Published `warnings.json` | Object containing the warnings passed into the output encoder | Durable consumers | Exact for the production JSON encoder unless the encoder itself returns warnings. |
|
||||||
|
| `index.json` | Path to `warnings.json` | Durable consumers | Stable discovery path; no separate diagnostic-detail path. |
|
||||||
|
| Run-result v1 | `warning_count = len(final RunOutput.Warnings)` | Subprocess callers | Count only; no group/occurrence distinction. |
|
||||||
|
| Human stderr | `run completed with N warning(s)` | Operators | Count only and no direct detail path. Successful exit remains zero. |
|
||||||
|
| Manifest | Validation and rejection summaries, no warning collection | Durable provenance | Correctly avoids duplicating the flat list. |
|
||||||
|
| Debug summary `warnings.json` | Raw final warning array | Operators/developers | Includes final output-result warnings and can therefore differ from published `warnings.json`. |
|
||||||
|
| Debug run report | Final warning count | Operators/developers | Same final slice length as receipt and stderr. |
|
||||||
|
| Attempt/stage debug | Candidate-local warning detail and origin in path/envelope | Forensics | Sufficient to diagnose provenance, but debug capture is optional and is not a durable consumer contract. |
|
||||||
|
|
||||||
|
## Empirical Measurements
|
||||||
|
|
||||||
|
### Offline And Synthetic Runs
|
||||||
|
|
||||||
|
| Scenario | Result | What it establishes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Maintained complete D&D config and transcript with the repository's offline fake LLM | Approved, 10 normalized outputs, 0 rejected outputs, 12 warnings; receipt, stderr, and published file all reported 12 | An ordinary structurally successful workflow can be noisy without fallback or incomplete validation. |
|
||||||
|
| Same complete run, grouped after publication | Three reason codes, seven exact `(reason, scope, message)` tuples, maximum exact-tuple repetition of three | The flat count materially overstates distinct operator conditions. Scope resets within chunks and does not identify origin. |
|
||||||
|
| Maintained focused scene-description workflow | Approved, one normalized output, 0 warnings | The warning channel can be quiet when synthetic model text is lexically grounded. |
|
||||||
|
| Generic warning publication contract | One warning reaches successful stderr, durable output, and debug summary | The ordinary pre-output path is consistent. |
|
||||||
|
| NPC semantic-reconciliation candidate-limit fallback | No LLM call, all records preserved, one exhaustion warning, total warnings no greater than 20 | Fallback is bounded and materially different from routine normalization. |
|
||||||
|
| `warn_continue` with two failed validators and one skipped validator | Validation status contains all three incomplete validators; warning slice contains two execution-incomplete records | Current warning count does not describe all incomplete validation. |
|
||||||
|
| Retrying extract candidate | Two producer attempts; only the accepted attempt's one warning is final | Retry does not amplify abandoned warnings. |
|
||||||
|
| Terminal semantic rejection | Only the final rejected attempt's operation and validator warnings are final | Rejection diagnostics are retained without retaining superseded warnings. |
|
||||||
|
| Fresh versus reused extract checkpoint | Warning collections are deeply equal | Checkpoint replay does not itself amplify warnings. |
|
||||||
|
| Spell normalizer with 21 unresolved entries | 20 records: 19 samples plus one omission record saying two additional warnings were omitted | `warning_count` is neither exact occurrence count nor distinct-condition count. |
|
||||||
|
|
||||||
|
The focused audit tests passed in `internal/cli`,
|
||||||
|
`internal/framework/pipeline`, the NPC-registry and spell normalizers, and all
|
||||||
|
D&D packages.
|
||||||
|
|
||||||
|
### Bounded Sample Review
|
||||||
|
|
||||||
|
The complete offline D&D run produced:
|
||||||
|
|
||||||
|
| Reason | Count | Sample | Review |
|
||||||
|
| --- | ---: | --- | --- |
|
||||||
|
| `location_not_near_source` | 4 | `Moon Gate` was absent from cited text | Correctly identifies deliberately unsupported fake output. Three records shared the same exact tuple because chunk and stage origin were lost. |
|
||||||
|
| `location_occurrence_not_near_source` | 4 | A `Moon Gate` visit was absent from cited text | Correctly identifies the same unsupported registry-driven occurrence, but repeats the same operator concern across extraction and normalization. |
|
||||||
|
| `scene_description_not_near_source` | 4 | A title or summary had no significant exact token in cited text | Mixed value. Generic `session scene` prose is ungrounded, while `Arrival` versus transcript `arrive` illustrates an expected lexical false positive. |
|
||||||
|
|
||||||
|
This fake workflow is an integration fixture, not a model-quality benchmark.
|
||||||
|
It nonetheless proves that the checks carry useful evidence while being too
|
||||||
|
imprecise and repetitive to serve as one-warning-per-record operator alerts.
|
||||||
|
|
||||||
|
### Production Evidence Still Needed
|
||||||
|
|
||||||
|
The reported 85-warning run establishes that high volume occurs in practice,
|
||||||
|
but the following remain unknown:
|
||||||
|
|
||||||
|
- dominant production reason codes and stage/lane sources;
|
||||||
|
- unique group count versus repeated occurrence count;
|
||||||
|
- false-positive rate for each relatedness family;
|
||||||
|
- how much volume comes from normalization observations versus advisories;
|
||||||
|
- whether fresh and resumed production runs remain equivalent; and
|
||||||
|
- a defensible numerical acceptance target.
|
||||||
|
|
||||||
|
If further data is worthwhile, the operator can supply the v1 receipt,
|
||||||
|
`warnings.json`, and manifest validation summaries without supplying transcript
|
||||||
|
or lane artifacts. An initial privacy-preserving report should group by reason
|
||||||
|
code and normalized scope family, count exact repeated tuples, and omit message
|
||||||
|
text. Reviewing heuristic precision requires a separately approved bounded
|
||||||
|
sample with its cited source context.
|
||||||
|
|
||||||
|
## Classification Of Current Conditions
|
||||||
|
|
||||||
|
| Target disposition | Current families | Result impact | Operator action | Durable placement |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| **Warning** | Empty reference; validator failure or skip accepted under `warn_continue`; unavailable required scene classification; exhausted semantic reconciliation | A configured process completed under an allowed degraded or incomplete policy rather than completing normally | Correct reference/configuration, inspect provider/validator, or rerun | Actionable `warnings.json`, receipt summary, stderr summary, debug |
|
||||||
|
| **Advisory** | Source-relatedness heuristics; unresolved spell or registry membership; guarded invalid semantic proposal; any future LLM-judged uncertainty or extraction-quality signal | Uncertain data quality or poor model proposal, but accepted data is structurally valid and deterministic guards prevented unsafe mutation | Optional model/source review; no routine action for every record | Durable diagnostic detail and debug; never a top-level warning |
|
||||||
|
| **Observation** | Whitespace/name/ID/source-reference canonicalization; canonical ordering; exact and approved semantic duplicate consolidation | Successful intended normalization | None under normal operation | Durable bounded normalization diagnostics or debug; no stderr warning |
|
||||||
|
| **Not a diagnostic** | Rejection, invalid structure, cancellation, persistence error, provider failure under fail-run policy | Candidate or run did not complete according to policy | Inspect rejection/error and retry or correct input/configuration | Existing rejection, validation summary, error, and debug contracts |
|
||||||
|
|
||||||
|
Exact and semantic duplicate consolidation should remain distinguishable in
|
||||||
|
category or reason metadata even though both are observations. Semantic
|
||||||
|
reconciliation exhaustion remains a warning because a capability was not
|
||||||
|
applied; successful approved consolidation is an observation because it is the
|
||||||
|
normalizer's intended work.
|
||||||
|
|
||||||
|
## Ranked Findings
|
||||||
|
|
||||||
|
### AUD-WARN-001 — The Flat Warning Type Destroys Signal Quality
|
||||||
|
|
||||||
|
- **Priority:** High operator impact; high implementation leverage.
|
||||||
|
- **Evidence:** `contracts.Warning` has only scope, reason, and message. Routine
|
||||||
|
normalizer changes, heuristic doubt, fallback, and incomplete validation all
|
||||||
|
enter the same slice and the same CLI count. The offline complete run's 12
|
||||||
|
records were all advisories; the operator observed 85 records in a successful
|
||||||
|
real run.
|
||||||
|
- **Impact:** Operators cannot tell whether a warning requires a rerun, a
|
||||||
|
configuration repair, optional review, or no action. Repeated routine output
|
||||||
|
trains them to ignore the channel.
|
||||||
|
- **Recommendation:** Replace the flat semantic contract with explicit
|
||||||
|
`warning`, `advisory`, and `observation` dispositions plus a small category
|
||||||
|
vocabulary. Do not infer disposition from message text or require every
|
||||||
|
downstream consumer to maintain a reason-code policy table.
|
||||||
|
|
||||||
|
### AUD-WARN-002 — Warning Records Lose The Origin Needed For Diagnosis And Aggregation
|
||||||
|
|
||||||
|
- **Priority:** High correctness and usability impact.
|
||||||
|
- **Evidence:** The runner knows stage, step, lane, module, validator, chunk ID,
|
||||||
|
and chunk index at promotion time, but `terminalWarnings` flattens module and
|
||||||
|
validator records into `[]contracts.Warning`. Per-chunk scopes such as
|
||||||
|
`locations[0]` and `occurrences[0]` then repeat without identifying their
|
||||||
|
chunk or producer. `source_references_normalized` is intentionally shared
|
||||||
|
across families and is therefore especially ambiguous.
|
||||||
|
- **Impact:** `warnings.json` cannot answer which stage or module produced a
|
||||||
|
record. Message- or scope-based deduplication would merge unrelated findings
|
||||||
|
or retain accidental duplicates.
|
||||||
|
- **Recommendation:** Keep module findings free of framework context, then have
|
||||||
|
the framework add a structured origin envelope before promotion. Validator
|
||||||
|
findings must retain validator identity instead of passing through
|
||||||
|
`validationReport.Warnings()` as a flat slice.
|
||||||
|
|
||||||
|
### AUD-WARN-003 — Warning Volume Is Not End-To-End Bounded Or Validated
|
||||||
|
|
||||||
|
- **Priority:** High robustness impact; medium immediate likelihood.
|
||||||
|
- **Evidence:** D&D's `LimitWarnings` caps most individual producers at 20, but
|
||||||
|
NPC- and spell-relatedness return one warning per record without the helper.
|
||||||
|
Every extract validator is invoked per chunk, all ten relatedness checks run
|
||||||
|
again after normalization, and there is no run-level collector. The generic
|
||||||
|
contract validates neither disposition nor reason/message size, UTF-8,
|
||||||
|
blankness, or total records.
|
||||||
|
- **Impact:** Warning memory and output grow with chunks, lanes, configured
|
||||||
|
validators, and record counts. Local omission records lose exact occurrence
|
||||||
|
semantics while still incrementing `warning_count`.
|
||||||
|
- **Recommendation:** Add a generic bounded diagnostic collector that preserves
|
||||||
|
exact occurrence counts and bounded samples. Validate all diagnostic fields
|
||||||
|
at the module/framework boundary. Immediately bring NPC and spell
|
||||||
|
relatedness under the existing cap if the full redesign is staged.
|
||||||
|
|
||||||
|
### AUD-WARN-004 — Output Encoder Warnings Make Durable Surfaces Disagree
|
||||||
|
|
||||||
|
- **Priority:** Medium current impact; high contract correctness risk.
|
||||||
|
- **Evidence:** `Runner.Run` passes existing warnings to `encoder.Encode`, then
|
||||||
|
the production encoder serializes `warnings.json`. Only after encoding does
|
||||||
|
the runner append `OutputResult.Warnings`. The receipt, stderr, debug summary,
|
||||||
|
and debug run report see the final slice; the already-created published file
|
||||||
|
cannot. No production encoder currently returns a warning, so ordinary JSON
|
||||||
|
runs do not trigger the defect.
|
||||||
|
- **Impact:** A valid output-module implementation can violate the documented
|
||||||
|
claim that `warning_count` describes the published warning collection.
|
||||||
|
- **Recommendation:** Remove successful output warnings from the output-module
|
||||||
|
contract unless a demonstrated use case requires them; encoding failures
|
||||||
|
should be errors and optional encoder observations should be debug data. A
|
||||||
|
two-phase finalize API is the viable but more complex alternative.
|
||||||
|
|
||||||
|
### AUD-WARN-005 — Validation Skips Are Incomplete But Not Warned
|
||||||
|
|
||||||
|
- **Priority:** Medium operator/correctness impact.
|
||||||
|
- **Evidence:** `firstIncompleteValidation` treats failed and skipped validators
|
||||||
|
alike, and validation summaries include both. `incompleteValidationWarnings`
|
||||||
|
emits records only for `validationFailed`. The focused test demonstrates
|
||||||
|
three incomplete validators but two warnings.
|
||||||
|
- **Impact:** A successful run can have `validation_status: incomplete` while
|
||||||
|
its warning count understates or even omits the affected validators. A caller
|
||||||
|
that checks only warnings receives a weaker signal than the manifest and
|
||||||
|
receipt status.
|
||||||
|
- **Recommendation:** Produce one aggregated incomplete-validation warning
|
||||||
|
group whose occurrences cover both failure and skip, while retaining typed
|
||||||
|
outcome and safe reason metadata in the validation summary. Do not expose
|
||||||
|
provider errors or arbitrary skip prose in model or operator messages.
|
||||||
|
|
||||||
|
### AUD-WARN-006 — Relatedness Checks Are Useful But Repetitive And Lexically Weak
|
||||||
|
|
||||||
|
- **Priority:** Medium operator impact; low acceptance-policy urgency.
|
||||||
|
- **Evidence:** Every family runs the advisory in both extract and normalize
|
||||||
|
chains. The complete fixture contains exact repeated tuples, and the checks
|
||||||
|
rely on exact normalized token sequences or a minimal significant-token
|
||||||
|
overlap. Reason naming drifts between `*_not_near_source` and
|
||||||
|
`*_source_unrelated`.
|
||||||
|
- **Impact:** The checks can catch unsupported entities, but aliases, pronouns,
|
||||||
|
inflection, and generic scene prose create predictable false positives. Flat
|
||||||
|
per-record presentation magnifies them.
|
||||||
|
- **Recommendation:** Retain the validators and their stage-local execution,
|
||||||
|
but classify and aggregate them as advisories. Normalize reason-code naming
|
||||||
|
when the diagnostic contract changes. Do not strengthen them into rejection
|
||||||
|
rules without a human-reviewed production evaluation.
|
||||||
|
|
||||||
|
### AUD-WARN-007 — `warning_count` Has No Stable Operational Meaning
|
||||||
|
|
||||||
|
- **Priority:** High downstream-contract impact.
|
||||||
|
- **Evidence:** The receipt and CLI use `len(output.Warnings)`. One list element
|
||||||
|
can be an omission summary representing several hidden occurrences; repeated
|
||||||
|
records can represent the same condition; and skipped validators can be
|
||||||
|
absent. A 21-occurrence spell test produces a list length of 20.
|
||||||
|
- **Impact:** The value is neither an exact occurrence count nor a distinct
|
||||||
|
warning-group count. Consumers cannot set policy or present a trustworthy
|
||||||
|
summary from it.
|
||||||
|
- **Recommendation:** Introduce explicit warning-group and warning-occurrence
|
||||||
|
counts in a versioned receipt. Do not silently redefine the v1 field.
|
||||||
|
|
||||||
|
## Recommended Target Contract And Presentation Model
|
||||||
|
|
||||||
|
### Diagnostic Model
|
||||||
|
|
||||||
|
Use one validated internal diagnostic model with these concepts:
|
||||||
|
|
||||||
|
- **disposition:** `warning`, `advisory`, or `observation`;
|
||||||
|
- **category:** a small enum such as `configuration`, `degradation`,
|
||||||
|
`validation_incomplete`, `data_quality`, `fallback`, or `normalization`;
|
||||||
|
- **reason code:** stable semantic identity owned by the producer;
|
||||||
|
- **origin:** framework-added phase/stage, step ID, lane ID, module key,
|
||||||
|
validator name, chunk ID, and chunk index when applicable;
|
||||||
|
- **occurrence count:** exact number of matching findings;
|
||||||
|
- **samples:** a small deterministic list of bounded scope/message pairs; and
|
||||||
|
- **omitted sample count:** `occurrence_count - len(samples)`, represented as
|
||||||
|
metadata rather than another diagnostic record.
|
||||||
|
|
||||||
|
Errors and rejected outputs must not become diagnostic dispositions. A warning
|
||||||
|
means that the run completed under policy despite a process-level degradation
|
||||||
|
or incomplete configured operation. Advisory and observation dispositions can
|
||||||
|
describe accepted artifact quality and transformation provenance, but no
|
||||||
|
LLM-judged extraction-quality signal may be promoted to a warning. Validation
|
||||||
|
status remains authoritative for approval, rejection, and incomplete
|
||||||
|
validation.
|
||||||
|
|
||||||
|
### Aggregation
|
||||||
|
|
||||||
|
The framework runner should own aggregation after it enriches findings with
|
||||||
|
origin and before public output construction. Modules and validators retain
|
||||||
|
semantic ownership of disposition, category, reason, scope, and message; they
|
||||||
|
must not own CLI or file presentation.
|
||||||
|
|
||||||
|
The default stable key should be:
|
||||||
|
|
||||||
|
```text
|
||||||
|
disposition + category + reason_code
|
||||||
|
+ phase/stage + step_id + lane_id + module_key + validator_name
|
||||||
|
```
|
||||||
|
|
||||||
|
Chunk, record scope, and message text belong in samples and must not be part of
|
||||||
|
the group key. This groups repeated per-chunk findings without merging the same
|
||||||
|
code across distinct producers or pipeline locations. Group order should be
|
||||||
|
the first occurrence in the runner's existing canonical order; sample order
|
||||||
|
should follow the same order. A final canonical sort by the complete origin key
|
||||||
|
is also viable, but completion timing must never choose either order.
|
||||||
|
|
||||||
|
Aggregation must be incremental and bounded. Producers should use a shared
|
||||||
|
collector that counts every occurrence while retaining only bounded samples;
|
||||||
|
the framework then merges producer groups without reconstructing counts from
|
||||||
|
omission prose. A global maximum group count is also required, with overflow
|
||||||
|
represented by structured aggregate metadata and with actionable groups given
|
||||||
|
priority over lower dispositions.
|
||||||
|
|
||||||
|
### Durable Files
|
||||||
|
|
||||||
|
Keep one canonical home for each class:
|
||||||
|
|
||||||
|
- `warnings.json` should contain versioned, grouped actionable warnings only;
|
||||||
|
- a new `diagnostics.json` should contain versioned advisory and observation
|
||||||
|
groups only, avoiding duplication of warning groups;
|
||||||
|
- `index.json` should link both files;
|
||||||
|
- `rejected.json` and manifest validation summaries should retain their current
|
||||||
|
separate responsibilities; and
|
||||||
|
- debug bundles should retain candidate-attempt detail plus the final grouped
|
||||||
|
projections.
|
||||||
|
|
||||||
|
This is preferable to keeping all detail in `warnings.json` and filtering only
|
||||||
|
the CLI: downstream consumers would otherwise continue to receive a semantically
|
||||||
|
mixed warning contract, and routine observations would still dominate the
|
||||||
|
durable file.
|
||||||
|
|
||||||
|
### CLI And Receipt
|
||||||
|
|
||||||
|
For a successful human run with actionable warnings, print a concise summary
|
||||||
|
such as:
|
||||||
|
|
||||||
|
```text
|
||||||
|
notarius: run completed with 2 warning groups (7 occurrences); details=/.../warnings.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Advisories and observations should not produce the warning line. Their durable
|
||||||
|
path remains discoverable through `index.json`; a concise non-warning count can
|
||||||
|
be added to the ordinary success line only if operator testing shows value. An
|
||||||
|
ordinary successful run with no process degradation should write nothing to
|
||||||
|
the warning stream even when it publishes quality advisories or normalization
|
||||||
|
observations.
|
||||||
|
|
||||||
|
Create `notarius.run-result.v2` rather than redefining v1. It should expose at
|
||||||
|
least:
|
||||||
|
|
||||||
|
- `warning_group_count`;
|
||||||
|
- `warning_occurrence_count`; and
|
||||||
|
- `diagnostic_group_count` for non-warning durable groups.
|
||||||
|
|
||||||
|
The receipt should continue to expose validation status, validation summaries,
|
||||||
|
and rejected-output count independently. Process exit behavior should not
|
||||||
|
change as part of warning presentation reform.
|
||||||
|
|
||||||
|
### Checkpoint Semantics
|
||||||
|
|
||||||
|
Store the structured terminal diagnostic groups with accepted checkpoints and
|
||||||
|
replay them exactly once at their logical stage. Fresh and reused runs should
|
||||||
|
produce the same public groups and counts. Checkpoint events and debug records,
|
||||||
|
not diagnostic identity, should disclose whether computation was reused.
|
||||||
|
|
||||||
|
### Output Encoder Boundary
|
||||||
|
|
||||||
|
Prefer removing `OutputResult.Warnings`. A successful output encoder should
|
||||||
|
either return the complete logical files or fail. If future encoders genuinely
|
||||||
|
need to produce durable post-encoding warnings, introduce an explicit
|
||||||
|
two-phase prepare/finalize contract so those warnings can be included in the
|
||||||
|
same published collection. Do not retain the current self-inconsistent
|
||||||
|
one-pass capability.
|
||||||
|
|
||||||
|
## Resolution Of Required Design Questions
|
||||||
|
|
||||||
|
| Question | Recommendation | Viable alternative and tradeoff |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Explicit severity/disposition or external reason mapping? | Put validated disposition and category in the contract. | A central reason-code registry avoids payload fields but makes new modules depend on a second synchronized policy table and leaves downstream meaning implicit. |
|
||||||
|
| Keep all detail in `warnings.json` or separate it? | Separate grouped actionable warnings from grouped advisories/observations in `diagnostics.json`. | Keep the flat durable list and aggregate only CLI output; simpler migration, but it preserves the noisy downstream contract and ambiguous count. |
|
||||||
|
| Who owns aggregation? | Framework runner/coordinator after origin enrichment. | Output module aggregation keeps framework types smaller but duplicates policy across encoders and cannot repair missing validator origin. |
|
||||||
|
| Stable aggregation key? | Disposition, category, reason code, and full producer origin; exclude chunk/scope/message. | Explicit producer-supplied grouping keys offer flexibility but add another identity that can drift from reason codes. Message-template grouping is brittle and unsafe. |
|
||||||
|
| Samples and omissions? | Exact occurrence count plus deterministic bounded samples and numeric omitted-sample count. | Omission warning records preserve the current representation but inflate group counts and require prose parsing. |
|
||||||
|
| `warning_count` semantics? | Version receipt and replace ambiguity with group and occurrence counts. | Keep v1 count as published record length and add optional fields; compatible, but two competing warning counts remain easy to misuse. |
|
||||||
|
| Which normalization changes remain warnings? | Only exhausted process fallback. Unresolved membership is a data-quality advisory; successful canonicalization, reordering, ID repair, source-ref dedupe, and duplicate consolidation are observations. | Treat unresolved membership or semantic duplicate consolidation as warnings because they affect grounding or cardinality; more conservative, but it violates the process-only warning rule and reports accepted artifact quality as an operational failure. |
|
||||||
|
| Source-relatedness disposition? | Grouped advisory by default; preserve current approve behavior. | Retain warning disposition or make rejection configurable. Rejection requires production precision evidence; current lexical rules are not strong enough. |
|
||||||
|
| Checkpoint-loaded warnings? | Present the same logical groups as fresh execution and use checkpoint events for reuse provenance. | Mark groups as replayed; aids forensics but fragments aggregation and makes semantically equivalent runs differ. |
|
||||||
|
| ADR and schema versions? | Add an ADR and version the run receipt and diagnostic files. | Treat the work as CLI-only presentation and avoid an ADR; insufficient because module contracts, output files, checkpoint payloads, and downstream fields change. |
|
||||||
|
|
||||||
|
## Compatibility, Documentation, And ADR Implications
|
||||||
|
|
||||||
|
The target alters public and internal contracts enough to require a new ADR.
|
||||||
|
It should record:
|
||||||
|
|
||||||
|
- the distinction among warnings, advisories, observations, rejections, and
|
||||||
|
errors;
|
||||||
|
- the invariant that warnings are process-level signals, LLM-judged extraction
|
||||||
|
quality is never a warning, and ordinary non-degraded success has zero
|
||||||
|
warnings;
|
||||||
|
- module semantic ownership versus framework origin/aggregation ownership;
|
||||||
|
- bounded group and sample semantics;
|
||||||
|
- fresh/checkpoint equivalence; and
|
||||||
|
- the output-encoder decision.
|
||||||
|
|
||||||
|
Implementation should introduce `notarius.run-result.v2`. The grouped warning
|
||||||
|
and diagnostic envelopes should each carry their own schema version. Because
|
||||||
|
the content of `warnings.json` changes incompatibly from a flat array wrapper
|
||||||
|
to groups, release notes and the published JSON integration contract must call
|
||||||
|
out the migration. `index.json` gains the diagnostic file path.
|
||||||
|
|
||||||
|
Canonical documentation updates belong in:
|
||||||
|
|
||||||
|
- `docs/cli.md` for stderr presentation only;
|
||||||
|
- `docs/operations.md` for operator review and debug workflow;
|
||||||
|
- `docs/integrations/json-output.md` for warning and diagnostic file schemas;
|
||||||
|
- `docs/integrations/run-result.md` for v2 fields and compatibility;
|
||||||
|
- `docs/consumers/subprocess.md` and `docs/consumers/dnd-pipeline.md` for
|
||||||
|
downstream policy checks;
|
||||||
|
- `docs/internal/pipeline.md` for promotion, aggregation, retry, and checkpoint
|
||||||
|
mechanics;
|
||||||
|
- `docs/internal/modules.md` and `docs/internal/dnd.md` for producer rules and
|
||||||
|
the D&D classification matrix; and
|
||||||
|
- `docs/policy/architecture.md` for the durable ownership invariant after the
|
||||||
|
ADR is accepted and implemented.
|
||||||
|
|
||||||
|
No configuration knob is required for the first implementation. A fixed,
|
||||||
|
well-documented taxonomy is easier to reason about than per-reason display
|
||||||
|
overrides. Configurable escalation or suppression can be considered only after
|
||||||
|
production review demonstrates a concrete operator need.
|
||||||
|
|
||||||
|
## Test Coverage Assessment
|
||||||
|
|
||||||
|
Existing coverage worth preserving includes:
|
||||||
|
|
||||||
|
- accepted-attempt and terminal-rejection warning promotion;
|
||||||
|
- validator failure retry exhaustion and `warn_continue`;
|
||||||
|
- module semantic retry fallback;
|
||||||
|
- deterministic warning order under concurrent lane completion;
|
||||||
|
- chunk-plan invalidation and discarded-cache warning behavior;
|
||||||
|
- fresh/checkpoint warning equivalence;
|
||||||
|
- local D&D warning caps and safe dynamic-message quoting;
|
||||||
|
- JSON warning-file publication; and
|
||||||
|
- CLI stderr, debug, and receipt counts.
|
||||||
|
|
||||||
|
Material gaps are:
|
||||||
|
|
||||||
|
- no bound test for NPC- or spell-relatedness warnings;
|
||||||
|
- no generic warning-field or result-size validation;
|
||||||
|
- no test for output encoder warnings versus published `warnings.json`;
|
||||||
|
- no operator-level aggregation or bounded-sample tests;
|
||||||
|
- no fresh/resume test for grouped counts because groups do not yet exist; and
|
||||||
|
- no production evaluation of advisory precision.
|
||||||
|
|
||||||
|
Tests should protect the semantic relationships: exact occurrence counts,
|
||||||
|
bounded samples, deterministic group order, actionable-only warning
|
||||||
|
presentation, and cross-surface equality. They should not assert one exact
|
||||||
|
warning count for every complete D&D run or treat message wording as a public
|
||||||
|
API unless the wording itself enforces a security boundary.
|
||||||
|
|
||||||
|
## Audit Conclusion
|
||||||
|
|
||||||
|
The application is in a good position for warning reform. Its retry,
|
||||||
|
validation, checkpoint, and concurrency mechanics provide reliable points at
|
||||||
|
which to attach structured diagnostics. The most valuable change is not to
|
||||||
|
suppress individual reason codes; it is to replace the semantically flat,
|
||||||
|
origin-free collection with bounded typed groups and to reserve the word
|
||||||
|
“warning” for conditions that merit operator attention.
|
||||||
|
|
||||||
|
Provider-backed runs would improve prioritization and help tune the D&D
|
||||||
|
advisories, but they are not necessary to conclude that routine normalization
|
||||||
|
and heuristic doubt should not dominate stderr or the durable warning
|
||||||
|
contract. They should be gathered before changing heuristic acceptance policy
|
||||||
|
or adopting a numerical production warning-volume target.
|
||||||
@@ -10,8 +10,9 @@ not as committed release dates.
|
|||||||
PromptKit now owns structural output repair within one completion. Notarius
|
PromptKit now owns structural output repair within one completion. Notarius
|
||||||
owns stage candidates, validator chains, semantic rejection policy, bounded
|
owns stage candidates, validator chains, semantic rejection policy, bounded
|
||||||
feedback-aware stage retries, validation provenance, and reusable-state
|
feedback-aware stage retries, validation provenance, and reusable-state
|
||||||
eligibility. The remaining near-term work applies those completed foundations
|
eligibility, and the separation of actionable process warnings from quality
|
||||||
to domain review and operator-facing diagnostics.
|
diagnostics. The remaining near-term work applies those completed foundations
|
||||||
|
to domain review and empirical evaluation.
|
||||||
|
|
||||||
### D&D Combat Scene Semantic Validation
|
### D&D Combat Scene Semantic Validation
|
||||||
|
|
||||||
@@ -55,33 +56,6 @@ to domain review and operator-facing diagnostics.
|
|||||||
the chunker or otherwise changes stage ownership or the durable chunk-plan
|
the chunker or otherwise changes stage ownership or the durable chunk-plan
|
||||||
contract.
|
contract.
|
||||||
|
|
||||||
### Warning Signal And Presentation Reform
|
|
||||||
|
|
||||||
- Audit every warning producer and representative successful runs. Ordinary
|
|
||||||
success producing dozens of warnings is a failed operator experience: the
|
|
||||||
volume obscures actionable problems and trains operators to ignore the
|
|
||||||
warning channel.
|
|
||||||
- Define a small warning taxonomy that distinguishes actionable degradation,
|
|
||||||
incomplete validation, lossy fallback, and data-quality risk from routine
|
|
||||||
normalization observations or informational diagnostics. Preserve detailed
|
|
||||||
traceability in debug or manifest data without promoting every observation
|
|
||||||
to a top-level CLI warning.
|
|
||||||
- Consider stable deduplication and aggregation by scope and reason code,
|
|
||||||
bounded samples plus omitted counts, and a concise CLI summary with a path to
|
|
||||||
detailed diagnostics. Do not suppress genuine validator execution failures
|
|
||||||
merely to reduce the count.
|
|
||||||
- Decide which warnings affect process status, rejection summaries, durable run
|
|
||||||
receipts, or only debug output. Ensure warning ordering and aggregation are
|
|
||||||
deterministic across concurrent execution.
|
|
||||||
- Establish a representative warning-volume acceptance target and human review
|
|
||||||
workflow before changing individual producers piecemeal. The intended result
|
|
||||||
is not zero warnings; it is a small set in which every surfaced warning merits
|
|
||||||
operator attention.
|
|
||||||
- This work does not require an ADR unless it changes validation acceptance,
|
|
||||||
failure, or durable contract semantics. CLI presentation and diagnostic
|
|
||||||
taxonomy otherwise belong in a feature roadmap followed by updates to their
|
|
||||||
canonical configuration, operations, integration, and internal documents.
|
|
||||||
|
|
||||||
## Near-Term D&D Pipeline
|
## Near-Term D&D Pipeline
|
||||||
|
|
||||||
### Evaluate Spell Extraction And Normalization
|
### Evaluate Spell Extraction And Normalization
|
||||||
|
|||||||
@@ -71,28 +71,39 @@ func TestAssembledSpellPipelineNormalizesMergedCasts(t *testing.T) {
|
|||||||
t.Fatalf("distinct cast = %#v, want separate evidence event", distinct)
|
t.Fatalf("distinct cast = %#v, want separate evidence event", distinct)
|
||||||
}
|
}
|
||||||
|
|
||||||
wantWarningReasons := []string{
|
wantDiagnosticReasons := []string{
|
||||||
spellnormalize.ReasonCodeSpellNameCanonicalized,
|
spellnormalize.ReasonCodeSpellNameCanonicalized,
|
||||||
spellnormalize.ReasonCodeSourceReferencesNormalized,
|
spellnormalize.ReasonCodeSourceReferencesNormalized,
|
||||||
spellnormalize.ReasonCodeDuplicateSpellCastCollapsed,
|
spellnormalize.ReasonCodeDuplicateSpellCastCollapsed,
|
||||||
"spell_not_near_source",
|
"spell_not_near_source",
|
||||||
}
|
}
|
||||||
gotWarningReasons := make([]string, len(output.Warnings))
|
gotDiagnosticReasons := make([]string, len(output.Diagnostics.Groups))
|
||||||
for index, warning := range output.Warnings {
|
for index, group := range output.Diagnostics.Groups {
|
||||||
gotWarningReasons[index] = warning.ReasonCode
|
gotDiagnosticReasons[index] = group.ReasonCode
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(gotWarningReasons, wantWarningReasons) {
|
if !reflect.DeepEqual(gotDiagnosticReasons, wantDiagnosticReasons) {
|
||||||
t.Fatalf("warnings = %#v, want deterministic normalize and validation warnings", output.Warnings)
|
t.Fatalf("diagnostics = %#v, want deterministic normalize and validation diagnostics", output.Diagnostics)
|
||||||
}
|
}
|
||||||
if output.Warnings[2].Scope != "spell_casts[0]" || !strings.Contains(output.Warnings[2].Message, "retained input index 0") || !strings.Contains(output.Warnings[2].Message, "removed input indices [1]") {
|
if output.Diagnostics.Groups[2].Samples[0].Scope != "spell_casts[0]" || !strings.Contains(output.Diagnostics.Groups[2].Samples[0].Message, "retained input index 0") || !strings.Contains(output.Diagnostics.Groups[2].Samples[0].Message, "removed input indices [1]") {
|
||||||
t.Fatalf("duplicate warning = %#v, want retained and removed merged indices", output.Warnings[2])
|
t.Fatalf("duplicate diagnostic = %#v, want retained and removed merged indices", output.Diagnostics.Groups[2])
|
||||||
}
|
}
|
||||||
|
|
||||||
warningsFile := decodeAssembledOutput[struct {
|
warningsFile := decodeAssembledOutput[struct {
|
||||||
Warnings []contracts.Warning `json:"warnings"`
|
Groups []contracts.DiagnosticGroup `json:"groups"`
|
||||||
}](t, output.OutputFiles, "warnings.json")
|
}](t, output.OutputFiles, "warnings.json")
|
||||||
if !reflect.DeepEqual(warningsFile.Warnings, output.Warnings) {
|
if len(warningsFile.Groups) != 0 {
|
||||||
t.Fatalf("warnings file = %#v, run warnings = %#v, want manifest output path to preserve warnings", warningsFile.Warnings, output.Warnings)
|
t.Fatalf("warnings file = %#v, want no process warnings for advisory-only diagnostics", warningsFile.Groups)
|
||||||
|
}
|
||||||
|
diagnosticsFile := decodeAssembledOutput[struct {
|
||||||
|
SchemaVersion string `json:"schema_version"`
|
||||||
|
GroupCount int `json:"group_count"`
|
||||||
|
OccurrenceCount int `json:"occurrence_count"`
|
||||||
|
Truncated bool `json:"truncated"`
|
||||||
|
UnrepresentedOccurrenceCount int `json:"unrepresented_occurrence_count"`
|
||||||
|
Groups []contracts.DiagnosticGroup `json:"groups"`
|
||||||
|
}](t, output.OutputFiles, "diagnostics.json")
|
||||||
|
if diagnosticsFile.SchemaVersion != "notarius.diagnostics.v1" || diagnosticsFile.GroupCount != len(output.Diagnostics.Groups) || !reflect.DeepEqual(diagnosticsFile.Groups, output.Diagnostics.Groups) || diagnosticsFile.OccurrenceCount != diagnosticGroupOccurrences(output.Diagnostics.Groups)+output.Diagnostics.UnrepresentedOccurrenceCount || diagnosticsFile.Truncated != output.Diagnostics.Truncated || diagnosticsFile.UnrepresentedOccurrenceCount != output.Diagnostics.UnrepresentedOccurrenceCount {
|
||||||
|
t.Fatalf("diagnostics file = %#v, run diagnostics = %#v", diagnosticsFile, output.Diagnostics)
|
||||||
}
|
}
|
||||||
manifest := decodeAssembledOutput[artifacts.RunManifest](t, output.OutputFiles, "manifest.json")
|
manifest := decodeAssembledOutput[artifacts.RunManifest](t, output.OutputFiles, "manifest.json")
|
||||||
if len(manifest.ArtifactLanes) != 1 || manifest.ArtifactLanes[0].Normalizer != spellnormalize.Key {
|
if len(manifest.ArtifactLanes) != 1 || manifest.ArtifactLanes[0].Normalizer != spellnormalize.Key {
|
||||||
@@ -155,14 +166,14 @@ func TestAssembledSpellPipelineHonorsNormalizeValidatorOverride(t *testing.T) {
|
|||||||
if err != nil || output.Manifest.ValidationStatus != "approved" || len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
|
if err != nil || output.Manifest.ValidationStatus != "approved" || len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
|
||||||
t.Fatalf("Run() error = %v output = %#v, want approved override run", err, output)
|
t.Fatalf("Run() error = %v output = %#v, want approved override run", err, output)
|
||||||
}
|
}
|
||||||
for _, warning := range output.Warnings {
|
for _, group := range output.Diagnostics.Groups {
|
||||||
if warning.ReasonCode == "spell_not_near_source" {
|
if group.ReasonCode == "spell_not_near_source" {
|
||||||
t.Fatalf("warnings = %#v, want explicit validator override to replace default relatedness chain", output.Warnings)
|
t.Fatalf("diagnostics = %#v, want explicit validator override to replace default relatedness chain", output.Diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestAssembledSpellPipelinePromotesTerminalUnknownSpellWarning(t *testing.T) {
|
func TestAssembledSpellPipelinePromotesTerminalUnknownSpellDiagnostics(t *testing.T) {
|
||||||
registries, resolved, _ := assembledSpellPipeline(t, assembledSpellPipelineOptions{unknownSpell: true})
|
registries, resolved, _ := assembledSpellPipeline(t, assembledSpellPipelineOptions{unknownSpell: true})
|
||||||
resolved.Steps[0].ArtifactLanes[0].NormalizeValidationPolicy.SemanticRejection = pipeline.SemanticRejectionRejectOutput
|
resolved.Steps[0].ArtifactLanes[0].NormalizeValidationPolicy.SemanticRejection = pipeline.SemanticRejectionRejectOutput
|
||||||
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
|
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
|
||||||
@@ -190,12 +201,12 @@ func TestAssembledSpellPipelinePromotesTerminalUnknownSpellWarning(t *testing.T)
|
|||||||
if !reflect.DeepEqual(rejectedFile.Rejected, output.Rejected) {
|
if !reflect.DeepEqual(rejectedFile.Rejected, output.Rejected) {
|
||||||
t.Fatalf("rejected file = %#v, run rejections = %#v, want durable rejection diagnostic", rejectedFile.Rejected, output.Rejected)
|
t.Fatalf("rejected file = %#v, run rejections = %#v, want durable rejection diagnostic", rejectedFile.Rejected, output.Rejected)
|
||||||
}
|
}
|
||||||
if len(output.Warnings) != 2 || output.Warnings[0].ReasonCode != spellnormalize.ReasonCodeSpellNameUnresolved || output.Warnings[0].Scope != "spell_casts[0]" || output.Warnings[1].ReasonCode != "spell_not_near_source" {
|
if len(output.Diagnostics.Groups) != 2 || output.Diagnostics.Groups[0].ReasonCode != spellnormalize.ReasonCodeSpellNameUnresolved || output.Diagnostics.Groups[0].Samples[0].Scope != "spell_casts[0]" || output.Diagnostics.Groups[1].ReasonCode != "spell_not_near_source" {
|
||||||
t.Fatalf("warnings = %#v, want complete terminal normalize validation warnings", output.Warnings)
|
t.Fatalf("diagnostics = %#v, want complete terminal normalize validation diagnostics", output.Diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestAssembledSpellPipelinePromotesUnknownSpellWarningWhenOverrideAccepts(t *testing.T) {
|
func TestAssembledSpellPipelinePromotesUnknownSpellAdvisoryWhenOverrideAccepts(t *testing.T) {
|
||||||
registries, resolved, _ := assembledSpellPipeline(t, assembledSpellPipelineOptions{normalizeValidatorOverride: true, unknownSpell: true})
|
registries, resolved, _ := assembledSpellPipeline(t, assembledSpellPipelineOptions{normalizeValidatorOverride: true, unknownSpell: true})
|
||||||
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
|
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -219,14 +230,20 @@ func TestAssembledSpellPipelinePromotesUnknownSpellWarningWhenOverrideAccepts(t
|
|||||||
if len(normalized.SpellCasts) != 1 || normalized.SpellCasts[0].Spell != "Mysterious Burst" {
|
if len(normalized.SpellCasts) != 1 || normalized.SpellCasts[0].Spell != "Mysterious Burst" {
|
||||||
t.Fatalf("normalized casts = %#v, want unresolved name preserved", normalized.SpellCasts)
|
t.Fatalf("normalized casts = %#v, want unresolved name preserved", normalized.SpellCasts)
|
||||||
}
|
}
|
||||||
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != spellnormalize.ReasonCodeSpellNameUnresolved || output.Warnings[0].Scope != "spell_casts[0]" {
|
if len(output.Diagnostics.Groups) != 1 || output.Diagnostics.Groups[0].ReasonCode != spellnormalize.ReasonCodeSpellNameUnresolved || output.Diagnostics.Groups[0].Samples[0].Scope != "spell_casts[0]" {
|
||||||
t.Fatalf("warnings = %#v, want promoted scoped unresolved-name warning", output.Warnings)
|
t.Fatalf("diagnostics = %#v, want promoted scoped unresolved-name diagnostic", output.Diagnostics)
|
||||||
}
|
}
|
||||||
warningsFile := decodeAssembledOutput[struct {
|
warningsFile := decodeAssembledOutput[struct {
|
||||||
Warnings []contracts.Warning `json:"warnings"`
|
Groups []contracts.DiagnosticGroup `json:"groups"`
|
||||||
}](t, output.OutputFiles, "warnings.json")
|
}](t, output.OutputFiles, "warnings.json")
|
||||||
if !reflect.DeepEqual(warningsFile.Warnings, output.Warnings) {
|
if len(warningsFile.Groups) != 0 {
|
||||||
t.Fatalf("warnings file = %#v, run warnings = %#v, want durable unresolved-name warning", warningsFile.Warnings, output.Warnings)
|
t.Fatalf("warnings file = %#v, want no process warnings for an advisory diagnostic", warningsFile.Groups)
|
||||||
|
}
|
||||||
|
diagnosticsFile := decodeAssembledOutput[struct {
|
||||||
|
Groups []contracts.DiagnosticGroup `json:"groups"`
|
||||||
|
}](t, output.OutputFiles, "diagnostics.json")
|
||||||
|
if !reflect.DeepEqual(diagnosticsFile.Groups, output.Diagnostics.Groups) {
|
||||||
|
t.Fatalf("diagnostics file = %#v, run diagnostics = %#v", diagnosticsFile.Groups, output.Diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -429,6 +446,14 @@ func (e *assembledSpellExtractor) chunkIndexesSnapshot() []int {
|
|||||||
return append([]int(nil), e.chunkIndexes...)
|
return append([]int(nil), e.chunkIndexes...)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func diagnosticGroupOccurrences(groups []contracts.DiagnosticGroup) int {
|
||||||
|
count := 0
|
||||||
|
for _, group := range groups {
|
||||||
|
count += group.OccurrenceCount
|
||||||
|
}
|
||||||
|
return count
|
||||||
|
}
|
||||||
|
|
||||||
func decodeAssembledOutput[T any](t *testing.T, files []contracts.OutputFile, name string) T {
|
func decodeAssembledOutput[T any](t *testing.T, files []contracts.OutputFile, name string) T {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
for _, file := range files {
|
for _, file := range files {
|
||||||
|
|||||||
@@ -91,8 +91,8 @@ func TestProductionSceneDescriptionWorkflow(t *testing.T) {
|
|||||||
if !reflect.DeepEqual(durable, want) {
|
if !reflect.DeepEqual(durable, want) {
|
||||||
t.Fatalf("durable output payload = %#v, want %#v", durable, want)
|
t.Fatalf("durable output payload = %#v, want %#v", durable, want)
|
||||||
}
|
}
|
||||||
if len(output.Warnings) != 0 {
|
if len(output.Diagnostics.Groups) != 0 {
|
||||||
t.Fatalf("warnings = %#v, want grounded descriptions without warnings", output.Warnings)
|
t.Fatalf("diagnostics = %#v, want grounded descriptions without diagnostics", output.Diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -239,10 +239,10 @@ func TestMaintainedMinimalInvocationProducesJSONBundle(t *testing.T) {
|
|||||||
t.Fatalf("rejected = %#v, want empty rejection list", rejected.Rejected)
|
t.Fatalf("rejected = %#v, want empty rejection list", rejected.Rejected)
|
||||||
}
|
}
|
||||||
warnings := readProductionJSON[struct {
|
warnings := readProductionJSON[struct {
|
||||||
Warnings []json.RawMessage `json:"warnings"`
|
Groups []json.RawMessage `json:"groups"`
|
||||||
}](t, filepath.Join(runRoot, "warnings.json"))
|
}](t, filepath.Join(runRoot, "warnings.json"))
|
||||||
if len(warnings.Warnings) != 0 {
|
if len(warnings.Groups) != 0 {
|
||||||
t.Fatalf("warnings = %#v, want empty warning list", warnings.Warnings)
|
t.Fatalf("warnings = %#v, want empty warning list", warnings.Groups)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -852,10 +852,10 @@ func TestProductionSceneRunRecordsAnnotationFreeChunkPlanAndProvenance(t *testin
|
|||||||
t.Fatalf("chunk map range annotations = %#v, want none", chunkMap.Chunks[0].Annotations)
|
t.Fatalf("chunk map range annotations = %#v, want none", chunkMap.Chunks[0].Annotations)
|
||||||
}
|
}
|
||||||
warnings := readProductionJSON[struct {
|
warnings := readProductionJSON[struct {
|
||||||
Warnings []contracts.Warning `json:"warnings"`
|
Groups []contracts.DiagnosticGroup `json:"groups"`
|
||||||
}](t, filepath.Join(outputRoot, productionRunID, "warnings.json"))
|
}](t, filepath.Join(outputRoot, productionRunID, "warnings.json"))
|
||||||
if len(warnings.Warnings) != 0 {
|
if len(warnings.Groups) != 0 {
|
||||||
t.Fatalf("warnings = %#v, want none", warnings.Warnings)
|
t.Fatalf("warnings = %#v, want none", warnings.Groups)
|
||||||
}
|
}
|
||||||
if len(fake.requestsFor(scenes.PromptID)) != 1 || len(fake.requestsFor(spells.PromptID)) != 1 || len(fake.requestsFor(itemoccurrenceextract.PromptID)) != 1 {
|
if len(fake.requestsFor(scenes.PromptID)) != 1 || len(fake.requestsFor(spells.PromptID)) != 1 || len(fake.requestsFor(itemoccurrenceextract.PromptID)) != 1 {
|
||||||
t.Fatalf("fake prompt requests = %#v, want one scene, spell, and item-occurrence request", fake.requestPrompts())
|
t.Fatalf("fake prompt requests = %#v, want one scene, spell, and item-occurrence request", fake.requestPrompts())
|
||||||
|
|||||||
@@ -375,7 +375,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("resolve working directory: %w", err))
|
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("resolve working directory: %w", err))
|
||||||
}
|
}
|
||||||
materialized, referenceWarnings, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalog, pipeline.ReferenceMaterializationOptions{
|
materialized, referenceDiagnostics, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalog, pipeline.ReferenceMaterializationOptions{
|
||||||
ConfigPath: loadedConfigPath,
|
ConfigPath: loadedConfigPath,
|
||||||
WorkingDir: workingDir,
|
WorkingDir: workingDir,
|
||||||
})
|
})
|
||||||
@@ -462,7 +462,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
StartedAt: startedAt,
|
StartedAt: startedAt,
|
||||||
LLMProfiles: llmProfiles,
|
LLMProfiles: llmProfiles,
|
||||||
Metadata: runMetadata(effective.Config.Output.Directory, debugPath),
|
Metadata: runMetadata(effective.Config.Output.Directory, debugPath),
|
||||||
Warnings: referenceWarnings,
|
Diagnostics: referenceDiagnostics,
|
||||||
ChunkCacheMode: effective.Config.Cache.ChunkPlans.Mode,
|
ChunkCacheMode: effective.Config.Cache.ChunkPlans.Mode,
|
||||||
ChunkPlans: chunkPlans,
|
ChunkPlans: chunkPlans,
|
||||||
Checkpoints: checkpointRecorder,
|
Checkpoints: checkpointRecorder,
|
||||||
@@ -471,7 +471,10 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
Debug: debugRecorder,
|
Debug: debugRecorder,
|
||||||
ExtractWorkers: cfg.Concurrency.StageWorkers["extract"],
|
ExtractWorkers: cfg.Concurrency.StageWorkers["extract"],
|
||||||
})
|
})
|
||||||
commandState.observeOutput(output)
|
diagnosticProjection, diagnosticErr := contracts.ProjectDiagnosticCollection(output.Diagnostics)
|
||||||
|
if diagnosticErr == nil {
|
||||||
|
commandState.observeOutput(output, diagnosticProjection)
|
||||||
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
primaryErr := fmt.Errorf("run pipeline %q: %w", pipelineID, err)
|
primaryErr := fmt.Errorf("run pipeline %q: %w", pipelineID, err)
|
||||||
if output.Manifest.PipelineID != "" {
|
if output.Manifest.PipelineID != "" {
|
||||||
@@ -479,15 +482,21 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
return failPipelineCommand(stderr, commandState, terminalWriter, primaryErr, fmt.Errorf("write debug summary: %w", summaryErr))
|
return failPipelineCommand(stderr, commandState, terminalWriter, primaryErr, fmt.Errorf("write debug summary: %w", summaryErr))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if diagnosticErr != nil {
|
||||||
|
return failPipelineCommand(stderr, commandState, terminalWriter, primaryErr, fmt.Errorf("summarize run diagnostics: %w", diagnosticErr))
|
||||||
|
}
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, primaryErr)
|
return failPipelineCommand(stderr, commandState, terminalWriter, primaryErr)
|
||||||
}
|
}
|
||||||
|
if diagnosticErr != nil {
|
||||||
|
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("summarize run diagnostics: %w", diagnosticErr))
|
||||||
|
}
|
||||||
|
|
||||||
if err := writePartialSummary(summary, output); err != nil {
|
if err := writePartialSummary(summary, output); err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug summary: %w", err))
|
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug summary: %w", err))
|
||||||
}
|
}
|
||||||
var encodedResult []byte
|
var encodedResult []byte
|
||||||
if *machineOutput {
|
if *machineOutput {
|
||||||
result, err := newRunResult(effective.ResolvedPipeline, output, runOutputDir, debugPath)
|
result, err := newRunResultWithDiagnostics(effective.ResolvedPipeline, output, runOutputDir, debugPath, diagnosticProjection)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
@@ -513,12 +522,25 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
fmt.Fprintf(stdout, "debug=%s\n", debugPath)
|
fmt.Fprintf(stdout, "debug=%s\n", debugPath)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if len(output.Warnings) > 0 {
|
if warningGroups := len(diagnosticProjection.Warnings); warningGroups > 0 {
|
||||||
fmt.Fprintf(stderr, "notarius: run completed with %d warning(s)\n", len(output.Warnings))
|
fmt.Fprintf(stderr, "notarius: run completed with %d warning group(s), %d occurrence(s)", warningGroups, diagnosticProjection.WarningOccurrenceCount)
|
||||||
|
if warningFile, ok := logicalOutputFile(output.OutputFiles, "warnings.json"); ok {
|
||||||
|
fmt.Fprintf(stderr, "; details=%s", filepath.Join(runOutputDir, warningFile))
|
||||||
|
}
|
||||||
|
fmt.Fprintln(stderr)
|
||||||
}
|
}
|
||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func logicalOutputFile(files []contracts.OutputFile, name string) (string, bool) {
|
||||||
|
for _, file := range files {
|
||||||
|
if file.Name == name {
|
||||||
|
return file.Name, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|
||||||
func writeSummary(summary *debugbundle.SummaryWriter, write func() error) error {
|
func writeSummary(summary *debugbundle.SummaryWriter, write func() error) error {
|
||||||
if summary == nil {
|
if summary == nil {
|
||||||
return nil
|
return nil
|
||||||
@@ -538,7 +560,7 @@ func writePartialSummary(summary *debugbundle.SummaryWriter, output pipeline.Run
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err := summary.WriteWarnings(output.Warnings); err != nil {
|
if err := summary.WriteDiagnostics(output.Diagnostics); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return summary.WriteCheckpointEvents(output.CheckpointEvents)
|
return summary.WriteCheckpointEvents(output.CheckpointEvents)
|
||||||
|
|||||||
@@ -590,10 +590,11 @@ func TestRunWarningsRemainSuccessfulAndReachDurableSurfaces(t *testing.T) {
|
|||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
harness := newStateTestHarness()
|
harness := newStateTestHarness()
|
||||||
harness.includeWarnings = true
|
harness.includeWarnings = true
|
||||||
harness.chunkWarnings = []contracts.Warning{{Scope: "chunk", ReasonCode: "contract-warning", Message: "warning retained"}}
|
harness.includeWarningFile = true
|
||||||
|
harness.chunkDiagnostics = []contracts.ProducerDiagnostic{stateTestDiagnostic("chunk", "contract-warning", "warning retained")}
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass", "--debug"}, &stdout, &stderr, harness.options())
|
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass", "--debug"}, &stdout, &stderr, harness.options())
|
||||||
if code != 0 || !strings.Contains(stdout.String(), "outputs=1") || !strings.Contains(stderr.String(), "1 warning(s)") {
|
if code != 0 || !strings.Contains(stdout.String(), "outputs=1") || !strings.Contains(stderr.String(), "1 warning group(s), 1 occurrence(s)") || !strings.Contains(stderr.String(), "warnings.json") {
|
||||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
}
|
}
|
||||||
outputPath := filepath.Join(onlyChildDir(t, roots.output), "result.json")
|
outputPath := filepath.Join(onlyChildDir(t, roots.output), "result.json")
|
||||||
@@ -601,11 +602,12 @@ func TestRunWarningsRemainSuccessfulAndReachDurableSurfaces(t *testing.T) {
|
|||||||
if err != nil || !strings.Contains(string(output), "contract-warning") {
|
if err != nil || !strings.Contains(string(output), "contract-warning") {
|
||||||
t.Fatalf("durable output = %q, %v", output, err)
|
t.Fatalf("durable output = %q, %v", output, err)
|
||||||
}
|
}
|
||||||
|
assertFile(t, filepath.Join(filepath.Dir(outputPath), "warnings.json"))
|
||||||
bundle := onlyChildDir(t, roots.debug)
|
bundle := onlyChildDir(t, roots.debug)
|
||||||
var warnings []contracts.Warning
|
var diagnostics contracts.DiagnosticCollection
|
||||||
readStateTestSummaryJSON(t, bundle, "warnings.json", &warnings)
|
readStateTestSummaryJSON(t, bundle, "final-diagnostics.json", &diagnostics)
|
||||||
if len(warnings) != 1 || warnings[0].ReasonCode != "contract-warning" {
|
if len(diagnostics.Groups) != 1 || diagnostics.Groups[0].ReasonCode != "contract-warning" {
|
||||||
t.Fatalf("debug warnings = %#v", warnings)
|
t.Fatalf("debug diagnostics = %#v", diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -8,26 +8,39 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
)
|
)
|
||||||
|
|
||||||
const runResultSchemaVersion = "notarius.run-result.v1"
|
const runResultSchemaVersion = "notarius.run-result.v2"
|
||||||
|
|
||||||
type runResult struct {
|
type runResult struct {
|
||||||
SchemaVersion string `json:"schema_version"`
|
SchemaVersion string `json:"schema_version"`
|
||||||
RunID string `json:"run_id"`
|
RunID string `json:"run_id"`
|
||||||
PipelineID string `json:"pipeline_id"`
|
PipelineID string `json:"pipeline_id"`
|
||||||
OutputDirectory string `json:"output_directory"`
|
OutputDirectory string `json:"output_directory"`
|
||||||
IndexFile string `json:"index_file,omitempty"`
|
IndexFile string `json:"index_file,omitempty"`
|
||||||
NormalizedOutputCount int `json:"normalized_output_count"`
|
NormalizedOutputCount int `json:"normalized_output_count"`
|
||||||
RejectedOutputCount int `json:"rejected_output_count"`
|
RejectedOutputCount int `json:"rejected_output_count"`
|
||||||
WarningCount int `json:"warning_count"`
|
WarningGroupCount int `json:"warning_group_count"`
|
||||||
ValidationStatus string `json:"validation_status"`
|
WarningOccurrenceCount int `json:"warning_occurrence_count"`
|
||||||
ValidationSummaries []artifacts.ValidationSummary `json:"validation_summaries,omitempty"`
|
DiagnosticGroupCount int `json:"diagnostic_group_count"`
|
||||||
DebugDirectory string `json:"debug_directory,omitempty"`
|
DiagnosticOccurrenceCount int `json:"diagnostic_occurrence_count"`
|
||||||
|
DiagnosticsTruncated bool `json:"diagnostics_truncated"`
|
||||||
|
ValidationStatus string `json:"validation_status"`
|
||||||
|
ValidationSummaries []artifacts.ValidationSummary `json:"validation_summaries,omitempty"`
|
||||||
|
DebugDirectory string `json:"debug_directory,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func newRunResult(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput, outputDirectory, debugDirectory string) (runResult, error) {
|
func newRunResult(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput, outputDirectory, debugDirectory string) (runResult, error) {
|
||||||
|
diagnosticProjection, err := contracts.ProjectDiagnosticCollection(output.Diagnostics)
|
||||||
|
if err != nil {
|
||||||
|
return runResult{}, fmt.Errorf("summarize run diagnostics: %w", err)
|
||||||
|
}
|
||||||
|
return newRunResultWithDiagnostics(resolved, output, outputDirectory, debugDirectory, diagnosticProjection)
|
||||||
|
}
|
||||||
|
|
||||||
|
func newRunResultWithDiagnostics(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput, outputDirectory, debugDirectory string, diagnosticProjection contracts.DiagnosticProjection) (runResult, error) {
|
||||||
if strings.TrimSpace(output.Manifest.RunID) == "" {
|
if strings.TrimSpace(output.Manifest.RunID) == "" {
|
||||||
return runResult{}, fmt.Errorf("run result requires a run ID")
|
return runResult{}, fmt.Errorf("run result requires a run ID")
|
||||||
}
|
}
|
||||||
@@ -53,15 +66,19 @@ func newRunResult(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput,
|
|||||||
}
|
}
|
||||||
|
|
||||||
result := runResult{
|
result := runResult{
|
||||||
SchemaVersion: runResultSchemaVersion,
|
SchemaVersion: runResultSchemaVersion,
|
||||||
RunID: output.Manifest.RunID,
|
RunID: output.Manifest.RunID,
|
||||||
PipelineID: output.Manifest.PipelineID,
|
PipelineID: output.Manifest.PipelineID,
|
||||||
OutputDirectory: absOutputDirectory,
|
OutputDirectory: absOutputDirectory,
|
||||||
NormalizedOutputCount: len(output.NormalizeOutputs),
|
NormalizedOutputCount: len(output.NormalizeOutputs),
|
||||||
RejectedOutputCount: len(output.Rejected),
|
RejectedOutputCount: len(output.Rejected),
|
||||||
WarningCount: len(output.Warnings),
|
WarningGroupCount: len(diagnosticProjection.Warnings),
|
||||||
ValidationStatus: output.Manifest.ValidationStatus,
|
WarningOccurrenceCount: diagnosticProjection.WarningOccurrenceCount,
|
||||||
ValidationSummaries: cloneValidationSummaries(output.Manifest.ValidationSummaries),
|
DiagnosticGroupCount: len(diagnosticProjection.Diagnostics),
|
||||||
|
DiagnosticOccurrenceCount: diagnosticProjection.DiagnosticOccurrenceCount,
|
||||||
|
DiagnosticsTruncated: output.Diagnostics.Truncated,
|
||||||
|
ValidationStatus: output.Manifest.ValidationStatus,
|
||||||
|
ValidationSummaries: cloneValidationSummaries(output.Manifest.ValidationSummaries),
|
||||||
}
|
}
|
||||||
|
|
||||||
if strings.TrimSpace(debugDirectory) != "" {
|
if strings.TrimSpace(debugDirectory) != "" {
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ func TestMaintainedMinimalInvocationEmitsRunResult(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
receipt := decodeRunResultDocument(t, stdout.String())
|
receipt := decodeRunResultDocument(t, stdout.String())
|
||||||
if got := receipt["schema_version"]; got != "notarius.run-result.v1" {
|
if got := receipt["schema_version"]; got != "notarius.run-result.v2" {
|
||||||
t.Fatalf("schema_version = %q", got)
|
t.Fatalf("schema_version = %q", got)
|
||||||
}
|
}
|
||||||
if got := receipt["run_id"]; got != productionRunID {
|
if got := receipt["run_id"]; got != productionRunID {
|
||||||
@@ -45,8 +45,8 @@ func TestMaintainedMinimalInvocationEmitsRunResult(t *testing.T) {
|
|||||||
if got := receipt["rejected_output_count"]; got != float64(0) {
|
if got := receipt["rejected_output_count"]; got != float64(0) {
|
||||||
t.Fatalf("rejected_output_count = %v", got)
|
t.Fatalf("rejected_output_count = %v", got)
|
||||||
}
|
}
|
||||||
if got := receipt["warning_count"]; got != float64(0) {
|
if got := receipt["warning_group_count"]; got != float64(0) || receipt["warning_occurrence_count"] != float64(0) || receipt["diagnostic_group_count"] != float64(0) || receipt["diagnostic_occurrence_count"] != float64(0) || receipt["diagnostics_truncated"] != false {
|
||||||
t.Fatalf("warning_count = %v", got)
|
t.Fatalf("diagnostic counts = %#v", receipt)
|
||||||
}
|
}
|
||||||
if got := receipt["validation_status"]; got != "approved" {
|
if got := receipt["validation_status"]; got != "approved" {
|
||||||
t.Fatalf("validation_status = %q", got)
|
t.Fatalf("validation_status = %q", got)
|
||||||
@@ -63,19 +63,19 @@ func TestMaintainedMinimalInvocationEmitsRunResult(t *testing.T) {
|
|||||||
func TestRunResultReportsWarningsAndDebugBundle(t *testing.T) {
|
func TestRunResultReportsWarningsAndDebugBundle(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
harness := newStateTestHarness()
|
harness := newStateTestHarness()
|
||||||
harness.chunkWarnings = []contracts.Warning{{Scope: "chunk", ReasonCode: "contract-warning", Message: "warning retained"}}
|
harness.chunkDiagnostics = []contracts.ProducerDiagnostic{stateTestDiagnostic("chunk", "contract-warning", "warning retained")}
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
code := RunWithOptions([]string{
|
code := RunWithOptions([]string{
|
||||||
"run", "sample", "--config", roots.config, "--input", roots.input,
|
"run", "sample", "--config", roots.config, "--input", roots.input,
|
||||||
"--chunk_cache", "bypass", "--debug", "--json",
|
"--chunk_cache", "bypass", "--debug", "--json",
|
||||||
}, &stdout, &stderr, harness.options())
|
}, &stdout, &stderr, harness.options())
|
||||||
if code != 0 || !strings.Contains(stderr.String(), "1 warning(s)") {
|
if code != 0 || !strings.Contains(stderr.String(), "1 warning group(s), 1 occurrence(s)") || strings.Contains(stderr.String(), "warnings.json") {
|
||||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
receipt := decodeRunResultDocument(t, stdout.String())
|
receipt := decodeRunResultDocument(t, stdout.String())
|
||||||
if got := receipt["warning_count"]; got != float64(1) {
|
if got := receipt["warning_group_count"]; got != float64(1) || receipt["warning_occurrence_count"] != float64(1) || receipt["diagnostic_group_count"] != float64(0) || receipt["diagnostic_occurrence_count"] != float64(0) || receipt["diagnostics_truncated"] != false {
|
||||||
t.Fatalf("warning_count = %v", got)
|
t.Fatalf("diagnostic counts = %#v", receipt)
|
||||||
}
|
}
|
||||||
debugDirectory, ok := receipt["debug_directory"].(string)
|
debugDirectory, ok := receipt["debug_directory"].(string)
|
||||||
if !ok || !filepath.IsAbs(debugDirectory) || debugDirectory != onlyChildDir(t, roots.debug) {
|
if !ok || !filepath.IsAbs(debugDirectory) || debugDirectory != onlyChildDir(t, roots.debug) {
|
||||||
|
|||||||
@@ -52,8 +52,8 @@ func TestRunResultEncodesRequiredFieldsAndCounts(t *testing.T) {
|
|||||||
if got := decoded["rejected_output_count"]; got != float64(1) {
|
if got := decoded["rejected_output_count"]; got != float64(1) {
|
||||||
t.Fatalf("rejected_output_count = %v", got)
|
t.Fatalf("rejected_output_count = %v", got)
|
||||||
}
|
}
|
||||||
if got := decoded["warning_count"]; got != float64(1) {
|
if got := decoded["warning_group_count"]; got != float64(1) || decoded["warning_occurrence_count"] != float64(1) || decoded["diagnostic_group_count"] != float64(0) || decoded["diagnostic_occurrence_count"] != float64(0) || decoded["diagnostics_truncated"] != false {
|
||||||
t.Fatalf("warning_count = %v", got)
|
t.Fatalf("diagnostic counts = %#v", decoded)
|
||||||
}
|
}
|
||||||
if got := decoded["validation_summaries"]; got != nil {
|
if got := decoded["validation_summaries"]; got != nil {
|
||||||
t.Fatalf("validation_summaries = %#v, want omitted when empty", got)
|
t.Fatalf("validation_summaries = %#v, want omitted when empty", got)
|
||||||
@@ -182,8 +182,15 @@ func testRunOutput() pipeline.RunOutput {
|
|||||||
Manifest: artifacts.RunManifest{RunID: "run-123", PipelineID: "sample", ValidationStatus: "rejected"},
|
Manifest: artifacts.RunManifest{RunID: "run-123", PipelineID: "sample", ValidationStatus: "rejected"},
|
||||||
NormalizeOutputs: []contracts.SerializedOutput{{}, {}},
|
NormalizeOutputs: []contracts.SerializedOutput{{}, {}},
|
||||||
Rejected: []contracts.RejectedOutput{{}},
|
Rejected: []contracts.RejectedOutput{{}},
|
||||||
Warnings: []contracts.Warning{{}},
|
Diagnostics: contracts.DiagnosticCollection{Groups: []contracts.DiagnosticGroup{{
|
||||||
OutputFiles: []contracts.OutputFile{{Name: "index.json"}},
|
Disposition: contracts.DiagnosticDispositionWarning,
|
||||||
|
Category: contracts.DiagnosticCategoryFallback,
|
||||||
|
ReasonCode: "fallback",
|
||||||
|
Origin: contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageNormalize, StepID: "step", LaneID: "lane", ModuleKey: "module"},
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []contracts.DiagnosticSample{{Scope: "scope", Message: "message"}},
|
||||||
|
}}},
|
||||||
|
OutputFiles: []contracts.OutputFile{{Name: "index.json"}},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/debugbundle"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/debugbundle"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -33,13 +34,17 @@ func (s *pipelineCommandState) setDebugPath(debugPath string) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *pipelineCommandState) observeOutput(output pipeline.RunOutput) {
|
func (s *pipelineCommandState) observeOutput(output pipeline.RunOutput, diagnostics contracts.DiagnosticProjection) {
|
||||||
if s == nil {
|
if s == nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
s.report.OutputCount = len(output.NormalizeOutputs)
|
s.report.OutputCount = len(output.NormalizeOutputs)
|
||||||
s.report.RejectedCount = len(output.Rejected)
|
s.report.RejectedCount = len(output.Rejected)
|
||||||
s.report.WarningCount = len(output.Warnings)
|
s.report.WarningGroupCount = len(diagnostics.Warnings)
|
||||||
|
s.report.WarningOccurrenceCount = diagnostics.WarningOccurrenceCount
|
||||||
|
s.report.DiagnosticGroupCount = len(diagnostics.Diagnostics)
|
||||||
|
s.report.DiagnosticOccurrenceCount = diagnostics.DiagnosticOccurrenceCount
|
||||||
|
s.report.DiagnosticsTruncated = output.Diagnostics.Truncated
|
||||||
s.report.ValidationStatus = output.Manifest.ValidationStatus
|
s.report.ValidationStatus = output.Manifest.ValidationStatus
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -238,7 +238,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
Kind: dnd.SpellListKind, Schema: normalizeSchema, MediaType: "application/json", Content: []byte(`{"spell_casts":[]}`),
|
Kind: dnd.SpellListKind, Schema: normalizeSchema, MediaType: "application/json", Content: []byte(`{"spell_casts":[]}`),
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
if err := recorder.NormalizeSucceeded("spells", spellnormalize.Key, normalizeDependencies, normalizeArtifact, nil); err != nil {
|
if err := recorder.NormalizeSucceeded("spells", spellnormalize.Key, normalizeDependencies, normalizeArtifact); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -19,12 +19,12 @@ import (
|
|||||||
func TestProductionSpellCatalogValidationRetries(t *testing.T) {
|
func TestProductionSpellCatalogValidationRetries(t *testing.T) {
|
||||||
const retries = 2
|
const retries = 2
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
responses []string
|
responses []string
|
||||||
wantCalls int
|
wantCalls int
|
||||||
wantRejected bool
|
wantRejected bool
|
||||||
wantSpell string
|
wantSpell string
|
||||||
wantWarningCode string
|
wantAdvisoryCode string
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "unknown spell remains rejected after exhaustion",
|
name: "unknown spell remains rejected after exhaustion",
|
||||||
@@ -42,9 +42,9 @@ func TestProductionSpellCatalogValidationRetries(t *testing.T) {
|
|||||||
productionSpellResponse("Unknown Spell"),
|
productionSpellResponse("Unknown Spell"),
|
||||||
productionSpellResponse("Aegis of Emberfall"),
|
productionSpellResponse("Aegis of Emberfall"),
|
||||||
},
|
},
|
||||||
wantCalls: 2,
|
wantCalls: 2,
|
||||||
wantSpell: "Aegis of Emberfall",
|
wantSpell: "Aegis of Emberfall",
|
||||||
wantWarningCode: "spell_not_near_source",
|
wantAdvisoryCode: "spell_not_near_source",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -92,8 +92,8 @@ func TestProductionSpellCatalogValidationRetries(t *testing.T) {
|
|||||||
if rejection.ReasonCode != "unknown_spell" || rejection.AttemptCount != retries+1 {
|
if rejection.ReasonCode != "unknown_spell" || rejection.AttemptCount != retries+1 {
|
||||||
t.Fatalf("rejection = %#v, want exhausted unknown-spell rejection", rejection)
|
t.Fatalf("rejection = %#v, want exhausted unknown-spell rejection", rejection)
|
||||||
}
|
}
|
||||||
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != "spell_not_near_source" {
|
if len(output.Diagnostics.Groups) != 1 || output.Diagnostics.Groups[0].ReasonCode != "spell_not_near_source" {
|
||||||
t.Fatalf("warnings = %#v, want complete terminal validation warnings", output.Warnings)
|
t.Fatalf("diagnostics = %#v, want complete terminal validation diagnostic", output.Diagnostics)
|
||||||
}
|
}
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -108,8 +108,16 @@ func TestProductionSpellCatalogValidationRetries(t *testing.T) {
|
|||||||
if len(value.SpellCasts) != 1 || value.SpellCasts[0].Spell != tt.wantSpell {
|
if len(value.SpellCasts) != 1 || value.SpellCasts[0].Spell != tt.wantSpell {
|
||||||
t.Fatalf("normalized spell list = %#v, want accepted overlay spell", value)
|
t.Fatalf("normalized spell list = %#v, want accepted overlay spell", value)
|
||||||
}
|
}
|
||||||
if len(output.Warnings) != 2 || output.Warnings[0].ReasonCode != tt.wantWarningCode || output.Warnings[1].ReasonCode != tt.wantWarningCode {
|
if len(output.Diagnostics.Groups) != 2 || output.Diagnostics.Groups[0].ReasonCode != tt.wantAdvisoryCode {
|
||||||
t.Fatalf("warnings = %#v, want accepted-attempt warnings from extract and normalize validation", output.Warnings)
|
t.Fatalf("diagnostics = %#v, want terminal extract and normalize diagnostics", output.Diagnostics)
|
||||||
|
}
|
||||||
|
if len(output.Diagnostics.Groups) != 2 {
|
||||||
|
t.Fatalf("diagnostics = %#v, want extract and normalize data-quality advisories", output.Diagnostics)
|
||||||
|
}
|
||||||
|
for _, diagnostic := range output.Diagnostics.Groups {
|
||||||
|
if diagnostic.Disposition != contracts.DiagnosticDispositionAdvisory || diagnostic.ReasonCode != tt.wantAdvisoryCode {
|
||||||
|
t.Fatalf("diagnostics = %#v, want only data-quality advisories", output.Diagnostics)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -349,7 +349,7 @@ func TestRunUsesOneInjectedIdentityForDebugOutputAndManifest(t *testing.T) {
|
|||||||
t.Fatalf("debug invocation session = %q, want %q", invocation.SessionID, wantSessionID)
|
t.Fatalf("debug invocation session = %q, want %q", invocation.SessionID, wantSessionID)
|
||||||
}
|
}
|
||||||
report := readStateTestRunReport(t, debugPath)
|
report := readStateTestRunReport(t, debugPath)
|
||||||
if !report.Succeeded || report.RunID != runID || report.PipelineID != "sample" || report.OutputPath != outputPath || report.DebugPath != debugPath || report.OutputCount != 1 || report.RejectedCount != 0 || report.WarningCount != 0 || report.ValidationStatus != "approved" {
|
if !report.Succeeded || report.RunID != runID || report.PipelineID != "sample" || report.OutputPath != outputPath || report.DebugPath != debugPath || report.OutputCount != 1 || report.RejectedCount != 0 || report.WarningGroupCount != 0 || report.WarningOccurrenceCount != 0 || report.DiagnosticGroupCount != 0 || report.DiagnosticOccurrenceCount != 0 || report.DiagnosticsTruncated || report.ValidationStatus != "approved" {
|
||||||
t.Fatalf("success report = %#v", report)
|
t.Fatalf("success report = %#v", report)
|
||||||
}
|
}
|
||||||
if !strings.Contains(result.stdout, "outputs=1 rejected=0") {
|
if !strings.Contains(result.stdout, "outputs=1 rejected=0") {
|
||||||
@@ -431,7 +431,7 @@ func TestRunWritesTerminalArtifactsForResolutionPipelineAndOutputFailures(t *tes
|
|||||||
bundlePath := onlyChildDir(t, roots.debug)
|
bundlePath := onlyChildDir(t, roots.debug)
|
||||||
runID := filepath.Base(bundlePath)
|
runID := filepath.Base(bundlePath)
|
||||||
report := readStateTestRunReport(t, bundlePath)
|
report := readStateTestRunReport(t, bundlePath)
|
||||||
if report.Succeeded || report.RunID != runID || report.PipelineID != tc.pipelineID || report.OutputPath != filepath.Join(roots.output, runID) || report.DebugPath != bundlePath || report.OutputCount != tc.wantOutputs || report.RejectedCount != 0 || report.WarningCount != 0 || report.ValidationStatus != tc.wantValidation {
|
if report.Succeeded || report.RunID != runID || report.PipelineID != tc.pipelineID || report.OutputPath != filepath.Join(roots.output, runID) || report.DebugPath != bundlePath || report.OutputCount != tc.wantOutputs || report.RejectedCount != 0 || report.WarningGroupCount != 0 || report.WarningOccurrenceCount != 0 || report.DiagnosticGroupCount != 0 || report.DiagnosticOccurrenceCount != 0 || report.DiagnosticsTruncated || report.ValidationStatus != tc.wantValidation {
|
||||||
t.Fatalf("failure report = %#v", report)
|
t.Fatalf("failure report = %#v", report)
|
||||||
}
|
}
|
||||||
errorLog, err := os.ReadFile(filepath.Join(bundlePath, "summary", "error.log"))
|
errorLog, err := os.ReadFile(filepath.Join(bundlePath, "summary", "error.log"))
|
||||||
@@ -445,7 +445,7 @@ func TestRunWritesTerminalArtifactsForResolutionPipelineAndOutputFailures(t *tes
|
|||||||
func TestRunRetainsPartialPipelineOutcomeInFailureSummary(t *testing.T) {
|
func TestRunRetainsPartialPipelineOutcomeInFailureSummary(t *testing.T) {
|
||||||
roots := newStateTestRoots(t)
|
roots := newStateTestRoots(t)
|
||||||
harness := newStateTestHarness()
|
harness := newStateTestHarness()
|
||||||
harness.chunkWarnings = []contracts.Warning{{Scope: "chunk", ReasonCode: "partial-warning", Message: "warning retained before failure"}}
|
harness.chunkDiagnostics = []contracts.ProducerDiagnostic{stateTestDiagnostic("chunk", "partial-warning", "warning retained before failure")}
|
||||||
harness.extractErr = errors.New("synthetic partial pipeline failure")
|
harness.extractErr = errors.New("synthetic partial pipeline failure")
|
||||||
|
|
||||||
result := runStateTest(t, roots, harness.options(), true, true, "bypass")
|
result := runStateTest(t, roots, harness.options(), true, true, "bypass")
|
||||||
@@ -454,7 +454,7 @@ func TestRunRetainsPartialPipelineOutcomeInFailureSummary(t *testing.T) {
|
|||||||
}
|
}
|
||||||
bundlePath := onlyChildDir(t, roots.debug)
|
bundlePath := onlyChildDir(t, roots.debug)
|
||||||
report := readStateTestRunReport(t, bundlePath)
|
report := readStateTestRunReport(t, bundlePath)
|
||||||
if report.Succeeded || report.OutputCount != 0 || report.RejectedCount != 0 || report.WarningCount != 1 || report.ValidationStatus != "failed" {
|
if report.Succeeded || report.OutputCount != 0 || report.RejectedCount != 0 || report.WarningGroupCount != 1 || report.WarningOccurrenceCount != 1 || report.DiagnosticGroupCount != 0 || report.DiagnosticOccurrenceCount != 0 || report.DiagnosticsTruncated || report.ValidationStatus != "failed" {
|
||||||
t.Fatalf("partial failure report = %#v", report)
|
t.Fatalf("partial failure report = %#v", report)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -463,10 +463,10 @@ func TestRunRetainsPartialPipelineOutcomeInFailureSummary(t *testing.T) {
|
|||||||
if manifest.RunID != report.RunID || manifest.PipelineID != "sample" || manifest.ValidationStatus != "failed" {
|
if manifest.RunID != report.RunID || manifest.PipelineID != "sample" || manifest.ValidationStatus != "failed" {
|
||||||
t.Fatalf("partial manifest = %#v", manifest)
|
t.Fatalf("partial manifest = %#v", manifest)
|
||||||
}
|
}
|
||||||
var warnings []contracts.Warning
|
var diagnostics contracts.DiagnosticCollection
|
||||||
readStateTestSummaryJSON(t, bundlePath, "warnings.json", &warnings)
|
readStateTestSummaryJSON(t, bundlePath, "final-diagnostics.json", &diagnostics)
|
||||||
if len(warnings) != 1 || warnings[0].ReasonCode != "partial-warning" {
|
if len(diagnostics.Groups) != 1 || diagnostics.Groups[0].ReasonCode != "partial-warning" {
|
||||||
t.Fatalf("partial warnings = %#v", warnings)
|
t.Fatalf("partial diagnostics = %#v", diagnostics)
|
||||||
}
|
}
|
||||||
var events []pipeline.CheckpointEvent
|
var events []pipeline.CheckpointEvent
|
||||||
readStateTestSummaryJSON(t, bundlePath, "checkpoint-events.json", &events)
|
readStateTestSummaryJSON(t, bundlePath, "checkpoint-events.json", &events)
|
||||||
@@ -863,11 +863,12 @@ type stateTestHarness struct {
|
|||||||
chunkCalls, extractCalls int
|
chunkCalls, extractCalls int
|
||||||
runIDCalls uint64
|
runIDCalls uint64
|
||||||
extractErr error
|
extractErr error
|
||||||
chunkWarnings []contracts.Warning
|
chunkDiagnostics []contracts.ProducerDiagnostic
|
||||||
moduleProfiles []string
|
moduleProfiles []string
|
||||||
sessionIDs []string
|
sessionIDs []string
|
||||||
outputWarnings []contracts.Warning
|
outputDiagnostics contracts.DiagnosticCollection
|
||||||
includeWarnings bool
|
includeWarnings bool
|
||||||
|
includeWarningFile bool
|
||||||
}
|
}
|
||||||
|
|
||||||
func newStateTestHarness() *stateTestHarness { return &stateTestHarness{} }
|
func newStateTestHarness() *stateTestHarness { return &stateTestHarness{} }
|
||||||
@@ -892,7 +893,7 @@ func (h *stateTestHarness) options() Options {
|
|||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
||||||
return stateTestOutput{harness: h, includeWarnings: h.includeWarnings}, nil
|
return stateTestOutput{harness: h, includeDiagnostics: h.includeWarnings}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
panic(err)
|
panic(err)
|
||||||
}
|
}
|
||||||
@@ -931,7 +932,7 @@ func (c stateTestChunker) Plan(_ context.Context, req contracts.ChunkRequest) (c
|
|||||||
c.harness.mu.Lock()
|
c.harness.mu.Lock()
|
||||||
c.harness.chunkCalls++
|
c.harness.chunkCalls++
|
||||||
c.harness.mu.Unlock()
|
c.harness.mu.Unlock()
|
||||||
return contracts.ChunkPlanResult{Plan: source.ChunkPlan{SourceDigest: req.Source.Digest, Ranges: []source.ChunkRange{{StartUnitID: 1, EndUnitID: 1}}}, Warnings: append([]contracts.Warning(nil), c.harness.chunkWarnings...)}, nil
|
return contracts.ChunkPlanResult{Plan: source.ChunkPlan{SourceDigest: req.Source.Digest, Ranges: []source.ChunkRange{{StartUnitID: 1, EndUnitID: 1}}}, Diagnostics: contracts.CloneProducerDiagnostics(c.harness.chunkDiagnostics)}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
const stateTestArtifactKind contracts.ArtifactKind = "test/artifact"
|
const stateTestArtifactKind contracts.ArtifactKind = "test/artifact"
|
||||||
@@ -999,20 +1000,28 @@ func (n stateTestNormalizer) Normalize(_ context.Context, req contracts.TypedNor
|
|||||||
}
|
}
|
||||||
|
|
||||||
type stateTestOutput struct {
|
type stateTestOutput struct {
|
||||||
harness *stateTestHarness
|
harness *stateTestHarness
|
||||||
includeWarnings bool
|
includeDiagnostics bool
|
||||||
}
|
}
|
||||||
|
|
||||||
func (o stateTestOutput) Key() string { return "test/output" }
|
func (o stateTestOutput) Key() string { return "test/output" }
|
||||||
func (o stateTestOutput) Encode(_ context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
func (o stateTestOutput) Encode(_ context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||||
o.harness.mu.Lock()
|
o.harness.mu.Lock()
|
||||||
o.harness.outputWarnings = append([]contracts.Warning(nil), req.Warnings...)
|
o.harness.outputDiagnostics = contracts.CloneDiagnosticCollection(req.Diagnostics)
|
||||||
o.harness.mu.Unlock()
|
o.harness.mu.Unlock()
|
||||||
data := []byte("{\"ok\":true}\n")
|
data := []byte("{\"ok\":true}\n")
|
||||||
if o.includeWarnings && len(req.Warnings) > 0 {
|
if o.includeDiagnostics && len(req.Diagnostics.Groups) > 0 {
|
||||||
data = []byte(fmt.Sprintf("{\"ok\":true,\"warnings\":%q}\n", req.Warnings[0].ReasonCode))
|
data = []byte(fmt.Sprintf("{\"ok\":true,\"diagnostics\":%q}\n", req.Diagnostics.Groups[0].ReasonCode))
|
||||||
}
|
}
|
||||||
return contracts.OutputResult{Files: []contracts.OutputFile{{Name: "result.json", Bytes: data}}}, nil
|
files := []contracts.OutputFile{{Name: "result.json", Bytes: data}}
|
||||||
|
if o.harness.includeWarningFile {
|
||||||
|
files = append(files, contracts.OutputFile{Name: "warnings.json", Bytes: []byte("{\"warnings\":true}\n")})
|
||||||
|
}
|
||||||
|
return contracts.OutputResult{Files: files}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func stateTestDiagnostic(scope, reasonCode, message string) contracts.ProducerDiagnostic {
|
||||||
|
return contracts.ProducerDiagnostic{Disposition: contracts.DiagnosticDispositionWarning, Category: contracts.DiagnosticCategoryDegradation, ReasonCode: reasonCode, OccurrenceCount: 1, Samples: []contracts.DiagnosticSample{{Scope: scope, Message: message}}}
|
||||||
}
|
}
|
||||||
|
|
||||||
type failingDebugRecorder struct{}
|
type failingDebugRecorder struct{}
|
||||||
|
|||||||
@@ -110,7 +110,7 @@ func TestSummaryWriterWritesEverySummaryArtifact(t *testing.T) {
|
|||||||
if err := summary.WriteRunReport(RunReport{RunID: bundle.RunID(), PipelineID: "test"}); err != nil {
|
if err := summary.WriteRunReport(RunReport{RunID: bundle.RunID(), PipelineID: "test"}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := summary.WriteWarnings([]contracts.Warning{{ReasonCode: "test"}}); err != nil {
|
if err := summary.WriteDiagnostics(contracts.DiagnosticCollection{}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := summary.WriteError("failed"); err != nil {
|
if err := summary.WriteError("failed"); err != nil {
|
||||||
@@ -126,7 +126,7 @@ func TestSummaryWriterWritesEverySummaryArtifact(t *testing.T) {
|
|||||||
ArtifactRunManifest,
|
ArtifactRunManifest,
|
||||||
ArtifactChunkPlan,
|
ArtifactChunkPlan,
|
||||||
ArtifactRunReport,
|
ArtifactRunReport,
|
||||||
ArtifactWarnings,
|
ArtifactDiagnostics,
|
||||||
ArtifactErrorLog,
|
ArtifactErrorLog,
|
||||||
} {
|
} {
|
||||||
info, err := os.Stat(filepath.Join(bundle.SummaryRoot(), name))
|
info, err := os.Stat(filepath.Join(bundle.SummaryRoot(), name))
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ const (
|
|||||||
ArtifactRunManifest = "run-manifest.json"
|
ArtifactRunManifest = "run-manifest.json"
|
||||||
ArtifactChunkPlan = "chunk-plan.json"
|
ArtifactChunkPlan = "chunk-plan.json"
|
||||||
ArtifactRunReport = "run-report.json"
|
ArtifactRunReport = "run-report.json"
|
||||||
ArtifactWarnings = "warnings.json"
|
ArtifactDiagnostics = "final-diagnostics.json"
|
||||||
ArtifactErrorLog = "error.log"
|
ArtifactErrorLog = "error.log"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -45,15 +45,19 @@ type Invocation struct {
|
|||||||
StartedAt time.Time `json:"started_at"`
|
StartedAt time.Time `json:"started_at"`
|
||||||
}
|
}
|
||||||
type RunReport struct {
|
type RunReport struct {
|
||||||
RunID string `json:"run_id"`
|
RunID string `json:"run_id"`
|
||||||
PipelineID string `json:"pipeline_id"`
|
PipelineID string `json:"pipeline_id"`
|
||||||
OutputPath string `json:"output_path,omitempty"`
|
OutputPath string `json:"output_path,omitempty"`
|
||||||
DebugPath string `json:"debug_path,omitempty"`
|
DebugPath string `json:"debug_path,omitempty"`
|
||||||
Succeeded bool `json:"succeeded"`
|
Succeeded bool `json:"succeeded"`
|
||||||
OutputCount int `json:"output_count"`
|
OutputCount int `json:"output_count"`
|
||||||
RejectedCount int `json:"rejected_count"`
|
RejectedCount int `json:"rejected_count"`
|
||||||
WarningCount int `json:"warning_count"`
|
WarningGroupCount int `json:"warning_group_count"`
|
||||||
ValidationStatus string `json:"validation_status,omitempty"`
|
WarningOccurrenceCount int `json:"warning_occurrence_count"`
|
||||||
|
DiagnosticGroupCount int `json:"diagnostic_group_count"`
|
||||||
|
DiagnosticOccurrenceCount int `json:"diagnostic_occurrence_count"`
|
||||||
|
DiagnosticsTruncated bool `json:"diagnostics_truncated"`
|
||||||
|
ValidationStatus string `json:"validation_status,omitempty"`
|
||||||
}
|
}
|
||||||
type SummaryWriter struct {
|
type SummaryWriter struct {
|
||||||
root, runID string
|
root, runID string
|
||||||
@@ -101,8 +105,8 @@ func (w *SummaryWriter) WriteChunkPlan(v artifacts.ChunkPlanSummary) error {
|
|||||||
return w.writeJSON(ArtifactChunkPlan, v)
|
return w.writeJSON(ArtifactChunkPlan, v)
|
||||||
}
|
}
|
||||||
func (w *SummaryWriter) WriteRunReport(v RunReport) error { return w.writeJSON(ArtifactRunReport, v) }
|
func (w *SummaryWriter) WriteRunReport(v RunReport) error { return w.writeJSON(ArtifactRunReport, v) }
|
||||||
func (w *SummaryWriter) WriteWarnings(v []contracts.Warning) error {
|
func (w *SummaryWriter) WriteDiagnostics(v contracts.DiagnosticCollection) error {
|
||||||
return w.writeJSON(ArtifactWarnings, v)
|
return w.writeJSON(ArtifactDiagnostics, contracts.CloneDiagnosticCollection(v))
|
||||||
}
|
}
|
||||||
func (w *SummaryWriter) WriteError(message string) error {
|
func (w *SummaryWriter) WriteError(message string) error {
|
||||||
return w.writeBytes(ArtifactErrorLog, []byte(message+"\n"))
|
return w.writeBytes(ArtifactErrorLog, []byte(message+"\n"))
|
||||||
|
|||||||
@@ -24,7 +24,6 @@ type filesystemCheckpointFixture struct {
|
|||||||
merge pipeline.CheckpointArtifact
|
merge pipeline.CheckpointArtifact
|
||||||
normalize pipeline.CheckpointArtifact
|
normalize pipeline.CheckpointArtifact
|
||||||
dependencies []pipeline.CheckpointFingerprint
|
dependencies []pipeline.CheckpointFingerprint
|
||||||
warnings []contracts.Warning
|
|
||||||
rejected []contracts.RejectedOutput
|
rejected []contracts.RejectedOutput
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -36,9 +35,9 @@ func TestFilesystemCheckpointRoundTripsAllStages(t *testing.T) {
|
|||||||
fixture.doc.Metadata["owner"] = "caller mutation"
|
fixture.doc.Metadata["owner"] = "caller mutation"
|
||||||
fixture.extract.Artifact.Content[0] = 'x'
|
fixture.extract.Artifact.Content[0] = 'x'
|
||||||
fixture.extract.Artifact.Metadata["content"] = "caller mutation"
|
fixture.extract.Artifact.Metadata["content"] = "caller mutation"
|
||||||
|
fixture.extract.Diagnostics[0].Diagnostic.Samples[0].Message = "caller mutation"
|
||||||
fixture.merge.Artifact.Content[0] = 'x'
|
fixture.merge.Artifact.Content[0] = 'x'
|
||||||
fixture.normalize.Artifact.Content[0] = 'x'
|
fixture.normalize.Artifact.Content[0] = 'x'
|
||||||
fixture.warnings[0].Message = "caller mutation"
|
|
||||||
fixture.rejected[0].Message = "caller mutation"
|
fixture.rejected[0].Message = "caller mutation"
|
||||||
|
|
||||||
t.Run("source", func(t *testing.T) {
|
t.Run("source", func(t *testing.T) {
|
||||||
@@ -60,11 +59,11 @@ func TestFilesystemCheckpointRoundTripsAllStages(t *testing.T) {
|
|||||||
|
|
||||||
t.Run("extract", func(t *testing.T) {
|
t.Run("extract", func(t *testing.T) {
|
||||||
got, decision := fixture.loader.Extract("lane-a", "extract-module", fixture.dependencies)
|
got, decision := fixture.loader.Extract("lane-a", "extract-module", fixture.dependencies)
|
||||||
if !decision.Reused || len(got.Outputs) != 1 || len(got.Rejected) != 1 || len(got.Warnings) != 1 {
|
if !decision.Reused || len(got.Outputs) != 1 || len(got.Rejected) != 1 || len(got.Outputs[0].Diagnostics) != 1 {
|
||||||
t.Fatalf("extract result=%#v decision=%#v", got, decision)
|
t.Fatalf("extract result=%#v decision=%#v", got, decision)
|
||||||
}
|
}
|
||||||
output := got.Outputs[0]
|
output := got.Outputs[0]
|
||||||
if !bytes.Equal(output.Artifact.Content, []byte(`{"spell":"fire"}`)) || output.Artifact.Kind != "spell" || output.Artifact.Schema.ID != "spell-schema" || output.Artifact.Schema.Version != "1" || output.Artifact.MediaType != "application/json" || output.Artifact.Metadata["chunk"] != "chunk-a" || output.ChunkRef.StartUnitID != 1 || got.Warnings[0].ReasonCode != "partial" || got.Rejected[0].ReasonCode != "invalid_source" {
|
if !bytes.Equal(output.Artifact.Content, []byte(`{"spell":"fire"}`)) || output.Artifact.Kind != "spell" || output.Artifact.Schema.ID != "spell-schema" || output.Artifact.Schema.Version != "1" || output.Artifact.MediaType != "application/json" || output.Artifact.Metadata["chunk"] != "chunk-a" || output.ChunkRef.StartUnitID != 1 || output.Diagnostics[0].Diagnostic.ReasonCode != "normalized_record" || got.Rejected[0].ReasonCode != "invalid_source" {
|
||||||
t.Fatalf("extract values were not restored: %#v", got)
|
t.Fatalf("extract values were not restored: %#v", got)
|
||||||
}
|
}
|
||||||
manifest := readManifest[ExtractLaneManifest](t, filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), "extract", "lane-a", "manifest.json"))
|
manifest := readManifest[ExtractLaneManifest](t, filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), "extract", "lane-a", "manifest.json"))
|
||||||
@@ -73,37 +72,36 @@ func TestFilesystemCheckpointRoundTripsAllStages(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
got.Outputs[0].Artifact.Content[0] = 'y'
|
got.Outputs[0].Artifact.Content[0] = 'y'
|
||||||
got.Warnings[0].Message = "loaded mutation"
|
|
||||||
reloaded, decision := fixture.loader.Extract("lane-a", "extract-module", fixture.dependencies)
|
reloaded, decision := fixture.loader.Extract("lane-a", "extract-module", fixture.dependencies)
|
||||||
if !decision.Reused || !bytes.Equal(reloaded.Outputs[0].Artifact.Content, []byte(`{"spell":"fire"}`)) || reloaded.Warnings[0].Message != "partial output" {
|
if !decision.Reused || !bytes.Equal(reloaded.Outputs[0].Artifact.Content, []byte(`{"spell":"fire"}`)) {
|
||||||
t.Fatalf("extract reload changed after loaded mutation: %#v decision=%#v", reloaded, decision)
|
t.Fatalf("extract reload changed after loaded mutation: %#v decision=%#v", reloaded, decision)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
for _, tt := range []struct {
|
for _, tt := range []struct {
|
||||||
name string
|
name string
|
||||||
load func() (pipeline.CheckpointArtifact, []contracts.Warning, pipeline.CheckpointDecision)
|
load func() (pipeline.CheckpointArtifact, pipeline.CheckpointDecision)
|
||||||
want []byte
|
want []byte
|
||||||
}{
|
}{
|
||||||
{name: "merge", load: func() (pipeline.CheckpointArtifact, []contracts.Warning, pipeline.CheckpointDecision) {
|
{name: "merge", load: func() (pipeline.CheckpointArtifact, pipeline.CheckpointDecision) {
|
||||||
got, decision := fixture.loader.Merge("lane-a", "merge-module", fixture.dependencies)
|
got, decision := fixture.loader.Merge("lane-a", "merge-module", fixture.dependencies)
|
||||||
return got.Output, got.Warnings, decision
|
return got.Output, decision
|
||||||
}, want: []byte(`{"spells":["fire"]}`)},
|
}, want: []byte(`{"spells":["fire"]}`)},
|
||||||
{name: "normalize", load: func() (pipeline.CheckpointArtifact, []contracts.Warning, pipeline.CheckpointDecision) {
|
{name: "normalize", load: func() (pipeline.CheckpointArtifact, pipeline.CheckpointDecision) {
|
||||||
got, decision := fixture.loader.Normalize("lane-a", "normalize-module", fixture.dependencies)
|
got, decision := fixture.loader.Normalize("lane-a", "normalize-module", fixture.dependencies)
|
||||||
return got.Output, got.Warnings, decision
|
return got.Output, decision
|
||||||
}, want: []byte(`{"spells":["fire"],"normalized":true}`)},
|
}, want: []byte(`{"spells":["fire"],"normalized":true}`)},
|
||||||
} {
|
} {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
got, warnings, decision := tt.load()
|
got, decision := tt.load()
|
||||||
if !decision.Reused || !bytes.Equal(got.Artifact.Content, tt.want) || got.Artifact.Kind != "spell" || got.Artifact.Schema.ID != "spell-schema" || got.Artifact.Schema.Version != "1" || got.Artifact.Metadata["lane"] != "lane-a" || len(warnings) != 1 || warnings[0].ReasonCode != "review" {
|
if !decision.Reused || !bytes.Equal(got.Artifact.Content, tt.want) || got.Artifact.Kind != "spell" || got.Artifact.Schema.ID != "spell-schema" || got.Artifact.Schema.Version != "1" || got.Artifact.Metadata["lane"] != "lane-a" || len(got.Diagnostics) != 1 || got.Diagnostics[0].Diagnostic.ReasonCode != "normalized_record" {
|
||||||
t.Fatalf("%s result=%#v warnings=%#v decision=%#v", tt.name, got, warnings, decision)
|
t.Fatalf("%s result=%#v decision=%#v", tt.name, got, decision)
|
||||||
}
|
}
|
||||||
|
|
||||||
got.Artifact.Content[0] = 'z'
|
got.Artifact.Content[0] = 'z'
|
||||||
reloaded, warnings, decision := tt.load()
|
reloaded, decision := tt.load()
|
||||||
if !decision.Reused || !bytes.Equal(reloaded.Artifact.Content, tt.want) || warnings[0].Message != "review manually" {
|
if !decision.Reused || !bytes.Equal(reloaded.Artifact.Content, tt.want) {
|
||||||
t.Fatalf("%s reload changed after loaded mutation: %#v warnings=%#v decision=%#v", tt.name, reloaded, warnings, decision)
|
t.Fatalf("%s reload changed after loaded mutation: %#v decision=%#v", tt.name, reloaded, decision)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -159,6 +157,7 @@ func TestFilesystemCheckpointRejectsIncompatibleManifests(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{"v1 schema", func(m map[string]any) { m["workspace_schema_version"] = WorkspaceSchemaVersionV1 }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
|
{"v1 schema", func(m map[string]any) { m["workspace_schema_version"] = WorkspaceSchemaVersionV1 }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
|
||||||
{"v2 schema", func(m map[string]any) { m["workspace_schema_version"] = WorkspaceSchemaVersionV2 }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
|
{"v2 schema", func(m map[string]any) { m["workspace_schema_version"] = WorkspaceSchemaVersionV2 }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
|
||||||
|
{"v3 schema", func(m map[string]any) { m["workspace_schema_version"] = WorkspaceSchemaVersionV3 }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
|
||||||
{"unknown schema", func(m map[string]any) { m["workspace_schema_version"] = "notarius.workspace.future" }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
|
{"unknown schema", func(m map[string]any) { m["workspace_schema_version"] = "notarius.workspace.future" }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
|
||||||
{"identity", func(m map[string]any) { m["metadata"].(map[string]any)["checkpoint_identity_digest"] = "sha256:other" }, pipeline.CheckpointReasonIdentityMismatch},
|
{"identity", func(m map[string]any) { m["metadata"].(map[string]any)["checkpoint_identity_digest"] = "sha256:other" }, pipeline.CheckpointReasonIdentityMismatch},
|
||||||
{"stage", func(m map[string]any) { m["stage"] = string(StageMerge) }, pipeline.CheckpointReasonStageMismatch},
|
{"stage", func(m map[string]any) { m["stage"] = string(StageMerge) }, pipeline.CheckpointReasonStageMismatch},
|
||||||
@@ -204,6 +203,11 @@ func TestFilesystemCheckpointRejectsIncompleteArtifactsAndContent(t *testing.T)
|
|||||||
{"content digest", func(m map[string]any) {
|
{"content digest", func(m map[string]any) {
|
||||||
m["outputs"].([]any)[0].(map[string]any)["content"].(map[string]any)["content_digest"] = "sha256:other"
|
m["outputs"].([]any)[0].(map[string]any)["content"].(map[string]any)["content_digest"] = "sha256:other"
|
||||||
}, pipeline.CheckpointReasonArtifactDigestMismatch},
|
}, pipeline.CheckpointReasonArtifactDigestMismatch},
|
||||||
|
{"diagnostics", func(m map[string]any) {
|
||||||
|
m["outputs"].([]any)[0].(map[string]any)["diagnostics"] = []any{map[string]any{
|
||||||
|
"diagnostic": map[string]any{"disposition": "warning", "category": "configuration", "reason_code": "invalid", "occurrence_count": float64(0)},
|
||||||
|
}}
|
||||||
|
}, pipeline.CheckpointReasonArtifactPayloadInvalid},
|
||||||
} {
|
} {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
fixture := seedFilesystemCheckpoints(t)
|
fixture := seedFilesystemCheckpoints(t)
|
||||||
@@ -337,9 +341,6 @@ func TestFilesystemLoaderReadsAcceptedNormalizeWithoutStageDependencies(t *testi
|
|||||||
if checkpoint.Output.Artifact.Content == nil || string(checkpoint.Output.Artifact.Content) != string(fixture.normalize.Artifact.Content) {
|
if checkpoint.Output.Artifact.Content == nil || string(checkpoint.Output.Artifact.Content) != string(fixture.normalize.Artifact.Content) {
|
||||||
t.Fatalf("accepted normalize output = %#v, want recorded artifact", checkpoint.Output)
|
t.Fatalf("accepted normalize output = %#v, want recorded artifact", checkpoint.Output)
|
||||||
}
|
}
|
||||||
if len(checkpoint.Warnings) != 1 || checkpoint.Warnings[0].ReasonCode != "normalized" {
|
|
||||||
t.Fatalf("accepted normalize warnings = %#v", checkpoint.Warnings)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFilesystemLoaderRejectsInvalidAcceptedNormalize(t *testing.T) {
|
func TestFilesystemLoaderRejectsInvalidAcceptedNormalize(t *testing.T) {
|
||||||
@@ -402,8 +403,7 @@ func seedAcceptedNormalizeCheckpoint(t *testing.T) filesystemCheckpointFixture {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
stepRecorder := recorder.(pipeline.StepCheckpointRecorder)
|
stepRecorder := recorder.(pipeline.StepCheckpointRecorder)
|
||||||
warnings := []contracts.Warning{{Scope: "normalize", ReasonCode: "normalized", Message: "normalized warning"}}
|
if err := stepRecorder.NormalizeSucceededForStep("step-1", "lane-a", "normalize-module", []pipeline.CheckpointFingerprint{{Name: "merge", Value: "sha256:unavailable"}}, artifact); err != nil {
|
||||||
if err := stepRecorder.NormalizeSucceededForStep("step-1", "lane-a", "normalize-module", []pipeline.CheckpointFingerprint{{Name: "merge", Value: "sha256:unavailable"}}, artifact, warnings); err != nil {
|
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
loader, err := NewFilesystemLoader(root, identity)
|
loader, err := NewFilesystemLoader(root, identity)
|
||||||
@@ -493,20 +493,18 @@ func seedFilesystemCheckpoints(t *testing.T) filesystemCheckpointFixture {
|
|||||||
merge: checkpointArtifact("merge", `{"spells":["fire"]}`),
|
merge: checkpointArtifact("merge", `{"spells":["fire"]}`),
|
||||||
normalize: checkpointArtifact("normalize", `{"spells":["fire"],"normalized":true}`),
|
normalize: checkpointArtifact("normalize", `{"spells":["fire"],"normalized":true}`),
|
||||||
dependencies: []pipeline.CheckpointFingerprint{{Name: "source", Value: "sha256:source"}, {Name: "chunk-plan", Value: "sha256:plan"}},
|
dependencies: []pipeline.CheckpointFingerprint{{Name: "source", Value: "sha256:source"}, {Name: "chunk-plan", Value: "sha256:plan"}},
|
||||||
warnings: []contracts.Warning{{Scope: "extract", ReasonCode: "partial", Message: "partial output"}},
|
|
||||||
rejected: []contracts.RejectedOutput{{Stage: "extract", LaneID: "lane-a", ModuleKey: "extract-module", ChunkID: "chunk-a", ValidatorName: "source_refs", ReasonCode: "invalid_source", Message: "source reference is invalid", AttemptCount: 1}},
|
rejected: []contracts.RejectedOutput{{Stage: "extract", LaneID: "lane-a", ModuleKey: "extract-module", ChunkID: "chunk-a", ValidatorName: "source_refs", ReasonCode: "invalid_source", Message: "source reference is invalid", AttemptCount: 1}},
|
||||||
}
|
}
|
||||||
if err := recorder.SourceSucceeded("source-module", &fixture.doc); err != nil {
|
if err := recorder.SourceSucceeded("source-module", &fixture.doc); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := recorder.ExtractSucceeded("lane-a", "extract-module", fixture.dependencies, []pipeline.CheckpointArtifact{fixture.extract}, fixture.rejected, fixture.warnings); err != nil {
|
if err := recorder.ExtractSucceeded("lane-a", "extract-module", fixture.dependencies, []pipeline.CheckpointArtifact{fixture.extract}, fixture.rejected); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
mergeWarnings := []contracts.Warning{{Scope: "merge", ReasonCode: "review", Message: "review manually"}}
|
if err := recorder.MergeSucceeded("lane-a", "merge-module", fixture.dependencies, fixture.merge); err != nil {
|
||||||
if err := recorder.MergeSucceeded("lane-a", "merge-module", fixture.dependencies, fixture.merge, mergeWarnings); err != nil {
|
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := recorder.NormalizeSucceeded("lane-a", "normalize-module", fixture.dependencies, fixture.normalize, mergeWarnings); err != nil {
|
if err := recorder.NormalizeSucceeded("lane-a", "normalize-module", fixture.dependencies, fixture.normalize); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
fixture.loader, err = NewFilesystemLoader(root, identity)
|
fixture.loader, err = NewFilesystemLoader(root, identity)
|
||||||
@@ -527,7 +525,8 @@ func checkpointArtifact(module, content string) pipeline.CheckpointArtifact {
|
|||||||
return pipeline.CheckpointArtifact{
|
return pipeline.CheckpointArtifact{
|
||||||
LaneID: "lane-a", ModuleKey: module, SourceID: "document-1", ChunkID: "chunk-a", ChunkIndex: 0,
|
LaneID: "lane-a", ModuleKey: module, SourceID: "document-1", ChunkID: "chunk-a", ChunkIndex: 0,
|
||||||
ChunkRef: source.SourceRef{SourceID: "document-1", StartUnitID: 1, EndUnitID: 1}, SchemaDigest: "sha256:schema",
|
ChunkRef: source.SourceRef{SourceID: "document-1", StartUnitID: 1, EndUnitID: 1}, SchemaDigest: "sha256:schema",
|
||||||
Artifact: contracts.SerializedArtifact{Kind: "spell", Schema: contracts.ArtifactSchema{ID: "spell-schema", Name: "Spell", Version: "1", JSONSchema: []byte(`{"type":"object"}`)}, MediaType: "application/json", Content: []byte(content), Metadata: map[string]any{"chunk": "chunk-a", "lane": "lane-a"}},
|
Artifact: contracts.SerializedArtifact{Kind: "spell", Schema: contracts.ArtifactSchema{ID: "spell-schema", Name: "Spell", Version: "1", JSONSchema: []byte(`{"type":"object"}`)}, MediaType: "application/json", Content: []byte(content), Metadata: map[string]any{"chunk": "chunk-a", "lane": "lane-a"}},
|
||||||
|
Diagnostics: []pipeline.CheckpointDiagnostic{{Diagnostic: contracts.ProducerDiagnostic{Disposition: contracts.DiagnosticDispositionObservation, Category: contracts.DiagnosticCategoryNormalization, ReasonCode: "normalized_record", OccurrenceCount: 1, Samples: []contracts.DiagnosticSample{{Scope: "fixture", Message: "record normalized"}}}}},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -86,7 +86,7 @@ func (l *FilesystemLoader) ExtractForStep(stepID, laneID, moduleKey string, depe
|
|||||||
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(outputs)) {
|
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(outputs)) {
|
||||||
return pipeline.ExtractCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch)
|
return pipeline.ExtractCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch)
|
||||||
}
|
}
|
||||||
return pipeline.ExtractCheckpoint{Outputs: outputs, Rejected: cloneRejectedOutputs(payload.Rejected), Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
return pipeline.ExtractCheckpoint{Outputs: outputs, Rejected: cloneRejectedOutputs(payload.Rejected)}, reusedDecision()
|
||||||
}
|
}
|
||||||
|
|
||||||
func (l *FilesystemLoader) Merge(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
|
func (l *FilesystemLoader) Merge(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
|
||||||
@@ -112,7 +112,7 @@ func (l *FilesystemLoader) MergeForStep(stepID, laneID, moduleKey string, depend
|
|||||||
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
||||||
return pipeline.MergeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch)
|
return pipeline.MergeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch)
|
||||||
}
|
}
|
||||||
return pipeline.MergeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
return pipeline.MergeCheckpoint{Output: values[0]}, reusedDecision()
|
||||||
}
|
}
|
||||||
|
|
||||||
func (l *FilesystemLoader) Normalize(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
|
func (l *FilesystemLoader) Normalize(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
|
||||||
@@ -138,7 +138,7 @@ func (l *FilesystemLoader) NormalizeForStep(stepID, laneID, moduleKey string, de
|
|||||||
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
||||||
return pipeline.NormalizeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch)
|
return pipeline.NormalizeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch)
|
||||||
}
|
}
|
||||||
return pipeline.NormalizeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
return pipeline.NormalizeCheckpoint{Output: values[0]}, reusedDecision()
|
||||||
}
|
}
|
||||||
|
|
||||||
func (l *FilesystemLoader) AcceptedNormalize(stepID, laneID, moduleKey string) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
|
func (l *FilesystemLoader) AcceptedNormalize(stepID, laneID, moduleKey string) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
|
||||||
@@ -163,7 +163,7 @@ func (l *FilesystemLoader) AcceptedNormalize(stepID, laneID, moduleKey string) (
|
|||||||
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
||||||
return pipeline.NormalizeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch)
|
return pipeline.NormalizeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch)
|
||||||
}
|
}
|
||||||
return pipeline.NormalizeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, decision(pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonAcceptedArtifactReused)
|
return pipeline.NormalizeCheckpoint{Output: values[0]}, decision(pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonAcceptedArtifactReused)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (l *FilesystemLoader) validateAcceptedNormalizeManifest(manifest StageManifest, stepID, laneID, moduleKey string) pipeline.CheckpointDecision {
|
func (l *FilesystemLoader) validateAcceptedNormalizeManifest(manifest StageManifest, stepID, laneID, moduleKey string) pipeline.CheckpointDecision {
|
||||||
@@ -205,11 +205,33 @@ func artifactCheckpointOutputs(values []artifactCheckpointEnvelope) ([]pipeline.
|
|||||||
if strings.TrimSpace(string(v.Kind)) == "" || strings.TrimSpace(v.Schema.ID) == "" || strings.TrimSpace(v.Schema.Version) == "" || strings.TrimSpace(v.SchemaDigest) == "" {
|
if strings.TrimSpace(string(v.Kind)) == "" || strings.TrimSpace(v.Schema.ID) == "" || strings.TrimSpace(v.Schema.Version) == "" || strings.TrimSpace(v.SchemaDigest) == "" {
|
||||||
return nil, &artifactPayloadError{code: pipeline.CheckpointReasonArtifactCodecIncompatible, err: fmt.Errorf("artifact codec identity is incomplete")}
|
return nil, &artifactPayloadError{code: pipeline.CheckpointReasonArtifactCodecIncompatible, err: fmt.Errorf("artifact codec identity is incomplete")}
|
||||||
}
|
}
|
||||||
out = append(out, pipeline.CheckpointArtifact{LaneID: v.LaneID, ModuleKey: v.ModuleKey, SourceID: v.SourceID, ChunkID: v.ChunkID, ChunkIndex: v.ChunkIndex, ChunkRef: v.ChunkRef, SchemaDigest: v.SchemaDigest, Artifact: contracts.SerializedArtifact{Kind: v.Kind, Schema: v.Schema, MediaType: v.Content.MediaType, Content: content, Metadata: cloneMetadata(v.Content.Metadata)}})
|
diagnostics, err := cloneAndValidateCheckpointDiagnostics(v.Diagnostics)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, pipeline.CheckpointArtifact{LaneID: v.LaneID, ModuleKey: v.ModuleKey, SourceID: v.SourceID, ChunkID: v.ChunkID, ChunkIndex: v.ChunkIndex, ChunkRef: v.ChunkRef, SchemaDigest: v.SchemaDigest, Artifact: contracts.SerializedArtifact{Kind: v.Kind, Schema: v.Schema, MediaType: v.Content.MediaType, Content: content, Metadata: cloneMetadata(v.Content.Metadata)}, Diagnostics: diagnostics})
|
||||||
}
|
}
|
||||||
return out, nil
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func cloneAndValidateCheckpointDiagnostics(values []pipeline.CheckpointDiagnostic) ([]pipeline.CheckpointDiagnostic, error) {
|
||||||
|
if len(values) == 0 {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
diagnostics := make([]contracts.ProducerDiagnostic, len(values))
|
||||||
|
for index, value := range values {
|
||||||
|
diagnostics[index] = value.Diagnostic
|
||||||
|
}
|
||||||
|
if err := contracts.ValidateProducerDiagnostics(diagnostics); err != nil {
|
||||||
|
return nil, &artifactPayloadError{code: pipeline.CheckpointReasonArtifactPayloadInvalid, err: fmt.Errorf("checkpoint diagnostics: %w", err)}
|
||||||
|
}
|
||||||
|
cloned := make([]pipeline.CheckpointDiagnostic, len(values))
|
||||||
|
for index, value := range values {
|
||||||
|
cloned[index] = pipeline.CheckpointDiagnostic{Diagnostic: contracts.CloneProducerDiagnostics([]contracts.ProducerDiagnostic{value.Diagnostic})[0], ValidatorKey: value.ValidatorKey}
|
||||||
|
}
|
||||||
|
return cloned, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (l *FilesystemLoader) readJSON(name string, out any) pipeline.CheckpointDecision {
|
func (l *FilesystemLoader) readJSON(name string, out any) pipeline.CheckpointDecision {
|
||||||
if !l.Enabled() {
|
if !l.Enabled() {
|
||||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonLoadingDisabled)
|
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonLoadingDisabled)
|
||||||
|
|||||||
@@ -3,7 +3,8 @@ package checkpoint
|
|||||||
import "time"
|
import "time"
|
||||||
|
|
||||||
const (
|
const (
|
||||||
WorkspaceSchemaVersion = "notarius.workspace.v3"
|
WorkspaceSchemaVersion = "notarius.workspace.v4"
|
||||||
|
WorkspaceSchemaVersionV3 = "notarius.workspace.v3"
|
||||||
WorkspaceSchemaVersionV2 = "notarius.workspace.v2"
|
WorkspaceSchemaVersionV2 = "notarius.workspace.v2"
|
||||||
WorkspaceSchemaVersionV1 = "notarius.workspace.v1"
|
WorkspaceSchemaVersionV1 = "notarius.workspace.v1"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -81,18 +81,18 @@ func (r *FilesystemRecorder) ExtractRunningForStep(stepID, laneID string, module
|
|||||||
return r.writeManifest(laneManifestPath("extract", stepID, laneID), ExtractLaneManifest{StageManifest: manifest})
|
return r.writeManifest(laneManifestPath("extract", stepID, laneID), ExtractLaneManifest{StageManifest: manifest})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *FilesystemRecorder) ExtractSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
func (r *FilesystemRecorder) ExtractSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput) error {
|
||||||
return r.ExtractSucceededForStep("", laneID, moduleKey, dependencies, outputs, rejected, warnings)
|
return r.ExtractSucceededForStep("", laneID, moduleKey, dependencies, outputs, rejected)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *FilesystemRecorder) ExtractSucceededForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
func (r *FilesystemRecorder) ExtractSucceededForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput) error {
|
||||||
payload := artifactExtractEnvelope{Outputs: artifactCheckpointEnvelopes(outputs), Rejected: cloneRejectedOutputs(rejected), Warnings: cloneWarnings(warnings)}
|
payload := artifactExtractEnvelope{Outputs: artifactCheckpointEnvelopes(outputs), Rejected: cloneRejectedOutputs(rejected)}
|
||||||
if err := r.writePayload(lanePayloadPath("extract", stepID, laneID, "outputs.json"), payload); err != nil {
|
if err := r.writePayload(lanePayloadPath("extract", stepID, laneID, "outputs.json"), payload); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
manifest := r.laneManifest(StageExtract, statusForRejected(rejected), stepID, laneID, moduleKey, dependencies)
|
manifest := r.laneManifest(StageExtract, statusForRejected(rejected), stepID, laneID, moduleKey, dependencies)
|
||||||
manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests(outputs))
|
manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests(outputs))
|
||||||
manifest.ValidationStatus = validationStatusString(warnings, rejected)
|
manifest.ValidationStatus = validationStatusString(rejected)
|
||||||
manifest.Rejections = rejectionSummaries(rejected)
|
manifest.Rejections = rejectionSummaries(rejected)
|
||||||
manifest.CompletedAt = timePtr(r.timestamp())
|
manifest.CompletedAt = timePtr(r.timestamp())
|
||||||
return r.writeManifest(laneManifestPath("extract", stepID, laneID), ExtractLaneManifest{StageManifest: manifest, ChunkCount: len(outputs) + len(rejected), OutputCount: len(outputs)})
|
return r.writeManifest(laneManifestPath("extract", stepID, laneID), ExtractLaneManifest{StageManifest: manifest, ChunkCount: len(outputs) + len(rejected), OutputCount: len(outputs)})
|
||||||
@@ -119,17 +119,17 @@ func (r *FilesystemRecorder) MergeRunningForStep(stepID, laneID string, moduleKe
|
|||||||
return r.writeManifest(laneManifestPath("merge", stepID, laneID), MergeLaneManifest{StageManifest: manifest})
|
return r.writeManifest(laneManifestPath("merge", stepID, laneID), MergeLaneManifest{StageManifest: manifest})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *FilesystemRecorder) MergeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
|
func (r *FilesystemRecorder) MergeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact) error {
|
||||||
return r.MergeSucceededForStep("", laneID, moduleKey, dependencies, output, warnings)
|
return r.MergeSucceededForStep("", laneID, moduleKey, dependencies, output)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *FilesystemRecorder) MergeSucceededForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
|
func (r *FilesystemRecorder) MergeSucceededForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact) error {
|
||||||
if err := r.writePayload(lanePayloadPath("merge", stepID, laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
|
if err := r.writePayload(lanePayloadPath("merge", stepID, laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output)}); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
manifest := r.laneManifest(StageMerge, StatusSucceeded, stepID, laneID, moduleKey, dependencies)
|
manifest := r.laneManifest(StageMerge, StatusSucceeded, stepID, laneID, moduleKey, dependencies)
|
||||||
manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
|
manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
|
||||||
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
manifest.ValidationStatus = validationStatusString(nil)
|
||||||
manifest.CompletedAt = timePtr(r.timestamp())
|
manifest.CompletedAt = timePtr(r.timestamp())
|
||||||
return r.writeManifest(laneManifestPath("merge", stepID, laneID), MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
return r.writeManifest(laneManifestPath("merge", stepID, laneID), MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||||
}
|
}
|
||||||
@@ -167,17 +167,17 @@ func (r *FilesystemRecorder) NormalizeRunningForStep(stepID, laneID string, modu
|
|||||||
return r.writeManifest(laneManifestPath("normalize", stepID, laneID), NormalizeLaneManifest{StageManifest: manifest})
|
return r.writeManifest(laneManifestPath("normalize", stepID, laneID), NormalizeLaneManifest{StageManifest: manifest})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *FilesystemRecorder) NormalizeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
|
func (r *FilesystemRecorder) NormalizeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact) error {
|
||||||
return r.NormalizeSucceededForStep("", laneID, moduleKey, dependencies, output, warnings)
|
return r.NormalizeSucceededForStep("", laneID, moduleKey, dependencies, output)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *FilesystemRecorder) NormalizeSucceededForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
|
func (r *FilesystemRecorder) NormalizeSucceededForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact) error {
|
||||||
if err := r.writePayload(lanePayloadPath("normalize", stepID, laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
|
if err := r.writePayload(lanePayloadPath("normalize", stepID, laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output)}); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
manifest := r.laneManifest(StageNormalize, StatusSucceeded, stepID, laneID, moduleKey, dependencies)
|
manifest := r.laneManifest(StageNormalize, StatusSucceeded, stepID, laneID, moduleKey, dependencies)
|
||||||
manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
|
manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
|
||||||
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
manifest.ValidationStatus = validationStatusString(nil)
|
||||||
manifest.CompletedAt = timePtr(r.timestamp())
|
manifest.CompletedAt = timePtr(r.timestamp())
|
||||||
return r.writeManifest(laneManifestPath("normalize", stepID, laneID), NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
return r.writeManifest(laneManifestPath("normalize", stepID, laneID), NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||||
}
|
}
|
||||||
@@ -249,39 +249,37 @@ type sourceDocumentEnvelope struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type binaryEnvelope struct {
|
type binaryEnvelope struct {
|
||||||
ContentBase64 string `json:"content_base64,omitempty"`
|
ContentBase64 string `json:"content_base64,omitempty"`
|
||||||
ContentDigest string `json:"content_digest,omitempty"`
|
ContentDigest string `json:"content_digest,omitempty"`
|
||||||
MediaType string `json:"media_type,omitempty"`
|
MediaType string `json:"media_type,omitempty"`
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type artifactCheckpointEnvelope struct {
|
type artifactCheckpointEnvelope struct {
|
||||||
LaneID string `json:"lane_id"`
|
LaneID string `json:"lane_id"`
|
||||||
ModuleKey string `json:"module_key"`
|
ModuleKey string `json:"module_key"`
|
||||||
SourceID string `json:"source_id,omitempty"`
|
SourceID string `json:"source_id,omitempty"`
|
||||||
ChunkID string `json:"chunk_id,omitempty"`
|
ChunkID string `json:"chunk_id,omitempty"`
|
||||||
ChunkIndex int `json:"chunk_index,omitempty"`
|
ChunkIndex int `json:"chunk_index,omitempty"`
|
||||||
ChunkRef source.SourceRef `json:"chunk_ref,omitempty"`
|
ChunkRef source.SourceRef `json:"chunk_ref,omitempty"`
|
||||||
Kind contracts.ArtifactKind `json:"artifact_kind"`
|
Kind contracts.ArtifactKind `json:"artifact_kind"`
|
||||||
Schema contracts.ArtifactSchema `json:"schema"`
|
Schema contracts.ArtifactSchema `json:"schema"`
|
||||||
SchemaDigest string `json:"schema_digest"`
|
SchemaDigest string `json:"schema_digest"`
|
||||||
Content binaryEnvelope `json:"content"`
|
Content binaryEnvelope `json:"content"`
|
||||||
|
Diagnostics []pipeline.CheckpointDiagnostic `json:"diagnostics,omitempty"`
|
||||||
}
|
}
|
||||||
type artifactExtractEnvelope struct {
|
type artifactExtractEnvelope struct {
|
||||||
Outputs []artifactCheckpointEnvelope `json:"outputs"`
|
Outputs []artifactCheckpointEnvelope `json:"outputs"`
|
||||||
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
|
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
|
||||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
|
||||||
}
|
}
|
||||||
type artifactSingleEnvelope struct {
|
type artifactSingleEnvelope struct {
|
||||||
Output artifactCheckpointEnvelope `json:"output"`
|
Output artifactCheckpointEnvelope `json:"output"`
|
||||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func artifactCheckpointEnvelopeFromOutput(output pipeline.CheckpointArtifact) artifactCheckpointEnvelope {
|
func artifactCheckpointEnvelopeFromOutput(output pipeline.CheckpointArtifact) artifactCheckpointEnvelope {
|
||||||
schema := contracts.CloneArtifactSchema(output.Artifact.Schema)
|
schema := contracts.CloneArtifactSchema(output.Artifact.Schema)
|
||||||
schema.JSONSchema = nil
|
schema.JSONSchema = nil
|
||||||
return artifactCheckpointEnvelope{LaneID: output.LaneID, ModuleKey: output.ModuleKey, SourceID: output.SourceID, ChunkID: output.ChunkID, ChunkIndex: output.ChunkIndex, ChunkRef: output.ChunkRef, Kind: output.Artifact.Kind, Schema: schema, SchemaDigest: output.SchemaDigest, Content: binaryEnvelopeFromContent(output.Artifact.Content, output.Artifact.MediaType, output.Artifact.Metadata, nil)}
|
return artifactCheckpointEnvelope{LaneID: output.LaneID, ModuleKey: output.ModuleKey, SourceID: output.SourceID, ChunkID: output.ChunkID, ChunkIndex: output.ChunkIndex, ChunkRef: output.ChunkRef, Kind: output.Artifact.Kind, Schema: schema, SchemaDigest: output.SchemaDigest, Content: binaryEnvelopeFromContent(output.Artifact.Content, output.Artifact.MediaType, output.Artifact.Metadata), Diagnostics: cloneCheckpointDiagnostics(output.Diagnostics)}
|
||||||
}
|
}
|
||||||
func artifactCheckpointEnvelopes(outputs []pipeline.CheckpointArtifact) []artifactCheckpointEnvelope {
|
func artifactCheckpointEnvelopes(outputs []pipeline.CheckpointArtifact) []artifactCheckpointEnvelope {
|
||||||
if len(outputs) == 0 {
|
if len(outputs) == 0 {
|
||||||
@@ -293,6 +291,20 @@ func artifactCheckpointEnvelopes(outputs []pipeline.CheckpointArtifact) []artifa
|
|||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func cloneCheckpointDiagnostics(values []pipeline.CheckpointDiagnostic) []pipeline.CheckpointDiagnostic {
|
||||||
|
if len(values) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
cloned := make([]pipeline.CheckpointDiagnostic, len(values))
|
||||||
|
for index, value := range values {
|
||||||
|
cloned[index] = pipeline.CheckpointDiagnostic{
|
||||||
|
Diagnostic: contracts.CloneProducerDiagnostics([]contracts.ProducerDiagnostic{value.Diagnostic})[0],
|
||||||
|
ValidatorKey: value.ValidatorKey,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return cloned
|
||||||
|
}
|
||||||
func artifactOutputDigests(outputs []pipeline.CheckpointArtifact) []pipeline.CheckpointFingerprint {
|
func artifactOutputDigests(outputs []pipeline.CheckpointArtifact) []pipeline.CheckpointFingerprint {
|
||||||
values := make([]pipeline.CheckpointFingerprint, 0, len(outputs))
|
values := make([]pipeline.CheckpointFingerprint, 0, len(outputs))
|
||||||
for i, v := range outputs {
|
for i, v := range outputs {
|
||||||
@@ -301,13 +313,12 @@ func artifactOutputDigests(outputs []pipeline.CheckpointArtifact) []pipeline.Che
|
|||||||
return normalizeFingerprints(values)
|
return normalizeFingerprints(values)
|
||||||
}
|
}
|
||||||
|
|
||||||
func binaryEnvelopeFromContent(content []byte, mediaType string, metadata map[string]any, warnings []contracts.Warning) binaryEnvelope {
|
func binaryEnvelopeFromContent(content []byte, mediaType string, metadata map[string]any) binaryEnvelope {
|
||||||
return binaryEnvelope{
|
return binaryEnvelope{
|
||||||
ContentBase64: base64.StdEncoding.EncodeToString(content),
|
ContentBase64: base64.StdEncoding.EncodeToString(content),
|
||||||
ContentDigest: contentDigest(content),
|
ContentDigest: contentDigest(content),
|
||||||
MediaType: mediaType,
|
MediaType: mediaType,
|
||||||
Metadata: cloneMetadata(metadata),
|
Metadata: cloneMetadata(metadata),
|
||||||
Warnings: cloneWarnings(warnings),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -334,13 +345,6 @@ func cloneSourceUnits(units []source.SourceUnit) []source.SourceUnit {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func cloneWarnings(warnings []contracts.Warning) []contracts.Warning {
|
|
||||||
if len(warnings) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
return append([]contracts.Warning(nil), warnings...)
|
|
||||||
}
|
|
||||||
|
|
||||||
func cloneRejectedOutputs(rejected []contracts.RejectedOutput) []contracts.RejectedOutput {
|
func cloneRejectedOutputs(rejected []contracts.RejectedOutput) []contracts.RejectedOutput {
|
||||||
if len(rejected) == 0 {
|
if len(rejected) == 0 {
|
||||||
return nil
|
return nil
|
||||||
@@ -461,13 +465,10 @@ func statusForRejected(rejected []contracts.RejectedOutput) StageStatus {
|
|||||||
return StatusSucceeded
|
return StatusSucceeded
|
||||||
}
|
}
|
||||||
|
|
||||||
func validationStatusString(warnings []contracts.Warning, rejected []contracts.RejectedOutput) string {
|
func validationStatusString(rejected []contracts.RejectedOutput) string {
|
||||||
if len(rejected) > 0 {
|
if len(rejected) > 0 {
|
||||||
return "rejected"
|
return "rejected"
|
||||||
}
|
}
|
||||||
if len(warnings) > 0 {
|
|
||||||
return "approved_with_warnings"
|
|
||||||
}
|
|
||||||
return "approved"
|
return "approved"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ func TestRootBasedRecorderOutputIsReusable(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := recorder.ExtractSucceeded("lane", "module", nil, nil, nil, nil); err != nil {
|
if err := recorder.ExtractSucceeded("lane", "module", nil, nil, nil); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
loader, err := NewFilesystemLoader(root, identity)
|
loader, err := NewFilesystemLoader(root, identity)
|
||||||
@@ -44,7 +44,7 @@ func TestStepAwareRecorderAndLoaderIsolateLaneState(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := recorder.(pipeline.StepCheckpointRecorder).ExtractSucceededForStep("step-a", "lane", "module", nil, nil, nil, nil); err != nil {
|
if err := recorder.(pipeline.StepCheckpointRecorder).ExtractSucceededForStep("step-a", "lane", "module", nil, nil, nil); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
loader, err := NewFilesystemLoader(root, identity)
|
loader, err := NewFilesystemLoader(root, identity)
|
||||||
@@ -87,7 +87,7 @@ func TestStepAwareCheckpointPreservesDistinctDotIdentities(t *testing.T) {
|
|||||||
LaneID: "lane", ModuleKey: "normalize-module", SourceID: "source", ChunkID: "chunk", ChunkRef: source.SourceRef{SourceID: "source", StartUnitID: 1, EndUnitID: 1}, SchemaDigest: "sha256:schema",
|
LaneID: "lane", ModuleKey: "normalize-module", SourceID: "source", ChunkID: "chunk", ChunkRef: source.SourceRef{SourceID: "source", StartUnitID: 1, EndUnitID: 1}, SchemaDigest: "sha256:schema",
|
||||||
Artifact: contracts.SerializedArtifact{Kind: "kind", Schema: contracts.ArtifactSchema{ID: "schema", Name: "Schema", Version: "1"}, MediaType: "application/json", Content: []byte(test.content)},
|
Artifact: contracts.SerializedArtifact{Kind: "kind", Schema: contracts.ArtifactSchema{ID: "schema", Name: "Schema", Version: "1"}, MediaType: "application/json", Content: []byte(test.content)},
|
||||||
}
|
}
|
||||||
if err := stepRecorder.NormalizeSucceededForStep(test.stepID, "lane", "normalize-module", nil, artifact, nil); err != nil {
|
if err := stepRecorder.NormalizeSucceededForStep(test.stepID, "lane", "normalize-module", nil, artifact); err != nil {
|
||||||
t.Fatalf("record %q: %v", test.stepID, err)
|
t.Fatalf("record %q: %v", test.stepID, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -119,7 +119,7 @@ func TestStepAwareCheckpointPreservesDistinctDotIdentities(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestCheckpointSchemaCompatibilityIdentifiers(t *testing.T) {
|
func TestCheckpointSchemaCompatibilityIdentifiers(t *testing.T) {
|
||||||
if WorkspaceSchemaVersion != "notarius.workspace.v3" || WorkspaceSchemaVersionV2 != "notarius.workspace.v2" || WorkspaceSchemaVersionV1 != "notarius.workspace.v1" {
|
if WorkspaceSchemaVersion != "notarius.workspace.v4" || WorkspaceSchemaVersionV3 != "notarius.workspace.v3" || WorkspaceSchemaVersionV2 != "notarius.workspace.v2" || WorkspaceSchemaVersionV1 != "notarius.workspace.v1" {
|
||||||
t.Fatal("checkpoint schema identifiers are incorrect")
|
t.Fatal("checkpoint schema identifiers are incorrect")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ import (
|
|||||||
"syscall"
|
"syscall"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -340,6 +341,9 @@ func validateRecord(record pipeline.ChunkPlanRecord, requestedDigest string) err
|
|||||||
if record.SchemaVersion != SchemaVersion {
|
if record.SchemaVersion != SchemaVersion {
|
||||||
return fmt.Errorf("schema_version %q is not supported", record.SchemaVersion)
|
return fmt.Errorf("schema_version %q is not supported", record.SchemaVersion)
|
||||||
}
|
}
|
||||||
|
if err := contracts.ValidateProducerDiagnostics(record.Diagnostics); err != nil {
|
||||||
|
return fmt.Errorf("diagnostics: %w", err)
|
||||||
|
}
|
||||||
if _, err := digestPathSegment(requestedDigest); err != nil {
|
if _, err := digestPathSegment(requestedDigest); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -281,7 +281,7 @@ func TestFilesystemStoreReportsInvalidRecordsAsRecoverable(t *testing.T) {
|
|||||||
return bytes.Replace(data, []byte(`{"schema_version"`), []byte(`{"SENTINEL_UNKNOWN_FIELD":true,"schema_version"`), 1)
|
return bytes.Replace(data, []byte(`{"schema_version"`), []byte(`{"SENTINEL_UNKNOWN_FIELD":true,"schema_version"`), 1)
|
||||||
}},
|
}},
|
||||||
{name: "truncated JSON", mutate: func(data []byte) []byte { return data[:len(data)/2] }},
|
{name: "truncated JSON", mutate: func(data []byte) []byte { return data[:len(data)/2] }},
|
||||||
{name: "legacy v1 record", mutate: replaceJSON(`notarius.chunk-plan.v2`, `notarius.chunk-plan.v1`)},
|
{name: "legacy v2 record", mutate: replaceJSON(`notarius.chunk-plan.v3`, `notarius.chunk-plan.v2`)},
|
||||||
{name: "source mismatch", mutate: replaceJSON(testSourceDigest, "sha256:"+strings.Repeat("b", 64))},
|
{name: "source mismatch", mutate: replaceJSON(testSourceDigest, "sha256:"+strings.Repeat("b", 64))},
|
||||||
{name: "plan digest mismatch", mutate: func(data []byte) []byte {
|
{name: "plan digest mismatch", mutate: func(data []byte) []byte {
|
||||||
prefix := []byte(`"plan_digest":"sha256:`)
|
prefix := []byte(`"plan_digest":"sha256:`)
|
||||||
@@ -498,7 +498,13 @@ func testRecord(t *testing.T, value int) pipeline.ChunkPlanRecord {
|
|||||||
References: []artifacts.ReferenceProvenance{{Stage: "chunk", SlotName: "guide", OriginType: "file", OriginURI: "file:///guide.txt", Digest: "sha256:reference"}},
|
References: []artifacts.ReferenceProvenance{{Stage: "chunk", SlotName: "guide", OriginType: "file", OriginURI: "file:///guide.txt", Digest: "sha256:reference"}},
|
||||||
Metadata: map[string]any{"prompt_id": "test/prompt", "enabled": true},
|
Metadata: map[string]any{"prompt_id": "test/prompt", "enabled": true},
|
||||||
},
|
},
|
||||||
Warnings: []contracts.Warning{{Scope: "chunk/test", ReasonCode: "observed", Message: "warning"}},
|
Diagnostics: []contracts.ProducerDiagnostic{{
|
||||||
|
Disposition: contracts.DiagnosticDispositionWarning,
|
||||||
|
Category: contracts.DiagnosticCategoryConfiguration,
|
||||||
|
ReasonCode: "empty_reference",
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []contracts.DiagnosticSample{{Scope: "reference", Message: "Reference was empty."}},
|
||||||
|
}},
|
||||||
CreatedAt: time.Date(2026, 7, 18, 12, 0, 0, 0, time.UTC),
|
CreatedAt: time.Date(2026, 7, 18, 12, 0, 0, 0, time.UTC),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -162,9 +162,9 @@ type ChunkRequest struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type ChunkPlanResult struct {
|
type ChunkPlanResult struct {
|
||||||
Plan source.ChunkPlan `json:"plan"`
|
Plan source.ChunkPlan `json:"plan"`
|
||||||
Warnings []Warning `json:"warnings,omitempty"`
|
Diagnostics []ProducerDiagnostic `json:"diagnostics,omitempty"`
|
||||||
ModelCandidate *ModelCandidate `json:"-"`
|
ModelCandidate *ModelCandidate `json:"-"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type Chunker interface {
|
type Chunker interface {
|
||||||
@@ -283,25 +283,19 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type ValidationResult struct {
|
type ValidationResult struct {
|
||||||
Approved bool `json:"approved"`
|
Approved bool `json:"approved"`
|
||||||
ReasonCode string `json:"reason_code,omitempty"`
|
ReasonCode string `json:"reason_code,omitempty"`
|
||||||
Message string `json:"message,omitempty"`
|
Message string `json:"message,omitempty"`
|
||||||
CorrectionGuidance string `json:"-"`
|
CorrectionGuidance string `json:"-"`
|
||||||
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
|
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
|
||||||
Warnings []Warning `json:"warnings,omitempty"`
|
Diagnostics []ProducerDiagnostic `json:"diagnostics,omitempty"`
|
||||||
}
|
|
||||||
|
|
||||||
type Warning struct {
|
|
||||||
Scope string `json:"scope,omitempty"`
|
|
||||||
ReasonCode string `json:"reason_code"`
|
|
||||||
Message string `json:"message"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type OutputRequest struct {
|
type OutputRequest struct {
|
||||||
Manifest artifacts.RunManifest `json:"manifest"`
|
Manifest artifacts.RunManifest `json:"manifest"`
|
||||||
NormalizeOutputs []SerializedOutput `json:"normalize_outputs,omitempty"`
|
NormalizeOutputs []SerializedOutput `json:"normalize_outputs,omitempty"`
|
||||||
Rejected []RejectedOutput `json:"rejected,omitempty"`
|
Rejected []RejectedOutput `json:"rejected,omitempty"`
|
||||||
Warnings []Warning `json:"warnings,omitempty"`
|
Diagnostics DiagnosticCollection `json:"diagnostics,omitempty"`
|
||||||
LLMProfile string `json:"llm_profile,omitempty"`
|
LLMProfile string `json:"llm_profile,omitempty"`
|
||||||
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
|
StructuredOutputRepairAttempts *int `json:"structured_output_repair_attempts,omitempty"`
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
@@ -316,8 +310,7 @@ type OutputFile struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type OutputResult struct {
|
type OutputResult struct {
|
||||||
Files []OutputFile `json:"files,omitempty"`
|
Files []OutputFile `json:"files,omitempty"`
|
||||||
Warnings []Warning `json:"warnings,omitempty"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type OutputEncoder interface {
|
type OutputEncoder interface {
|
||||||
|
|||||||
@@ -126,6 +126,9 @@ func (candidate ModelCandidate) Validate() error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func ValidateValidationResult(result ValidationResult) error {
|
func ValidateValidationResult(result ValidationResult) error {
|
||||||
|
if err := ValidateProducerDiagnostics(result.Diagnostics); err != nil {
|
||||||
|
return fmt.Errorf("validation diagnostics: %w", err)
|
||||||
|
}
|
||||||
if !result.Approved && result.ReasonCode == "" {
|
if !result.Approved && result.ReasonCode == "" {
|
||||||
return errors.New("validation rejection reason code must not be empty")
|
return errors.New("validation rejection reason code must not be empty")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -75,6 +75,9 @@ func TestCorrectionContractsRejectInvalidContent(t *testing.T) {
|
|||||||
{"oversized correction guidance", func() error {
|
{"oversized correction guidance", func() error {
|
||||||
return ValidateValidationResult(ValidationResult{ReasonCode: "invalid", CorrectionGuidance: tooLongValidationGuidance})
|
return ValidateValidationResult(ValidationResult{ReasonCode: "invalid", CorrectionGuidance: tooLongValidationGuidance})
|
||||||
}},
|
}},
|
||||||
|
{"invalid diagnostics", func() error {
|
||||||
|
return ValidateValidationResult(ValidationResult{Approved: true, Diagnostics: []ProducerDiagnostic{{}}})
|
||||||
|
}},
|
||||||
} {
|
} {
|
||||||
t.Run(test.name, func(t *testing.T) {
|
t.Run(test.name, func(t *testing.T) {
|
||||||
if err := test.call(); err == nil {
|
if err := test.call(); err == nil {
|
||||||
|
|||||||
401
internal/framework/contracts/diagnostics.go
Normal file
401
internal/framework/contracts/diagnostics.go
Normal file
@@ -0,0 +1,401 @@
|
|||||||
|
package contracts
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"unicode/utf8"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
MaxDiagnosticReasonCodeBytes = 128
|
||||||
|
MaxDiagnosticScopeBytes = 512
|
||||||
|
MaxDiagnosticMessageBytes = 4 * 1024
|
||||||
|
MaxDiagnosticSamples = 3
|
||||||
|
MaxProducerDiagnosticGroups = 64
|
||||||
|
)
|
||||||
|
|
||||||
|
// DiagnosticDisposition identifies the operator significance of a producer
|
||||||
|
// finding. Warnings are reserved for process-level degradation or incomplete
|
||||||
|
// configured work.
|
||||||
|
type DiagnosticDisposition string
|
||||||
|
|
||||||
|
const (
|
||||||
|
DiagnosticDispositionWarning DiagnosticDisposition = "warning"
|
||||||
|
DiagnosticDispositionAdvisory DiagnosticDisposition = "advisory"
|
||||||
|
DiagnosticDispositionObservation DiagnosticDisposition = "observation"
|
||||||
|
)
|
||||||
|
|
||||||
|
// DiagnosticCategory gives a stable, bounded classification for a producer
|
||||||
|
// finding.
|
||||||
|
type DiagnosticCategory string
|
||||||
|
|
||||||
|
const (
|
||||||
|
DiagnosticCategoryConfiguration DiagnosticCategory = "configuration"
|
||||||
|
DiagnosticCategoryDegradation DiagnosticCategory = "degradation"
|
||||||
|
DiagnosticCategoryValidationIncomplete DiagnosticCategory = "validation_incomplete"
|
||||||
|
DiagnosticCategoryFallback DiagnosticCategory = "fallback"
|
||||||
|
DiagnosticCategoryDataQuality DiagnosticCategory = "data_quality"
|
||||||
|
DiagnosticCategoryNormalization DiagnosticCategory = "normalization"
|
||||||
|
)
|
||||||
|
|
||||||
|
// DiagnosticOriginStage identifies the framework operation that promoted a
|
||||||
|
// diagnostic. It is framework-owned rather than producer-owned.
|
||||||
|
type DiagnosticOriginStage string
|
||||||
|
|
||||||
|
const (
|
||||||
|
DiagnosticOriginStageReferences DiagnosticOriginStage = "references"
|
||||||
|
DiagnosticOriginStageChunk DiagnosticOriginStage = "chunk"
|
||||||
|
DiagnosticOriginStageExtract DiagnosticOriginStage = "extract"
|
||||||
|
DiagnosticOriginStageMerge DiagnosticOriginStage = "merge"
|
||||||
|
DiagnosticOriginStageNormalize DiagnosticOriginStage = "normalize"
|
||||||
|
)
|
||||||
|
|
||||||
|
// DiagnosticSample is a bounded, safe example of a diagnostic occurrence.
|
||||||
|
// Chunk identity is attached by the framework when it promotes a producer
|
||||||
|
// diagnostic into a final group.
|
||||||
|
type DiagnosticSample struct {
|
||||||
|
Scope string `json:"scope"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
ChunkID string `json:"chunk_id,omitempty"`
|
||||||
|
ChunkIndex *int `json:"chunk_index,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ProducerDiagnostic is the locally grouped form returned by one producer or
|
||||||
|
// validator. It intentionally has no pipeline origin.
|
||||||
|
type ProducerDiagnostic struct {
|
||||||
|
Disposition DiagnosticDisposition `json:"disposition"`
|
||||||
|
Category DiagnosticCategory `json:"category"`
|
||||||
|
ReasonCode string `json:"reason_code"`
|
||||||
|
OccurrenceCount int `json:"occurrence_count"`
|
||||||
|
Samples []DiagnosticSample `json:"samples"`
|
||||||
|
OmittedSampleCount int `json:"omitted_sample_count"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// DiagnosticOrigin is framework-owned context used to distinguish findings
|
||||||
|
// from different pipeline locations during final aggregation.
|
||||||
|
type DiagnosticOrigin struct {
|
||||||
|
Stage DiagnosticOriginStage `json:"stage"`
|
||||||
|
StepID string `json:"step_id,omitempty"`
|
||||||
|
LaneID string `json:"lane_id,omitempty"`
|
||||||
|
ModuleKey string `json:"module_key,omitempty"`
|
||||||
|
ValidatorKey string `json:"validator_key,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// DiagnosticGroup is a producer diagnostic after framework origin enrichment.
|
||||||
|
type DiagnosticGroup struct {
|
||||||
|
Disposition DiagnosticDisposition `json:"disposition"`
|
||||||
|
Category DiagnosticCategory `json:"category"`
|
||||||
|
ReasonCode string `json:"reason_code"`
|
||||||
|
Origin DiagnosticOrigin `json:"origin"`
|
||||||
|
OccurrenceCount int `json:"occurrence_count"`
|
||||||
|
Samples []DiagnosticSample `json:"samples"`
|
||||||
|
OmittedSampleCount int `json:"omitted_sample_count"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// DiagnosticCollection is the grouped collection supplied to later durable
|
||||||
|
// and presentation boundaries. Global aggregation policy is applied by the
|
||||||
|
// framework before it reaches those boundaries.
|
||||||
|
type DiagnosticCollection struct {
|
||||||
|
Groups []DiagnosticGroup `json:"groups"`
|
||||||
|
Truncated bool `json:"truncated"`
|
||||||
|
UnrepresentedOccurrenceCount int `json:"unrepresented_occurrence_count"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// DiagnosticProjection is the validated warning/non-warning view used by
|
||||||
|
// durable and presentation boundaries. Occurrence totals are checked before
|
||||||
|
// they leave the framework contract.
|
||||||
|
type DiagnosticProjection struct {
|
||||||
|
Warnings []DiagnosticGroup
|
||||||
|
Diagnostics []DiagnosticGroup
|
||||||
|
WarningOccurrenceCount int
|
||||||
|
DiagnosticOccurrenceCount int
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate checks a producer-local diagnostic against the public safety and
|
||||||
|
// classification contract.
|
||||||
|
func (diagnostic ProducerDiagnostic) Validate() error {
|
||||||
|
return validateDiagnostic(
|
||||||
|
diagnostic.Disposition,
|
||||||
|
diagnostic.Category,
|
||||||
|
diagnostic.ReasonCode,
|
||||||
|
diagnostic.OccurrenceCount,
|
||||||
|
diagnostic.Samples,
|
||||||
|
diagnostic.OmittedSampleCount,
|
||||||
|
false,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ValidateProducerDiagnostics validates the complete set returned by one
|
||||||
|
// producer or validator result.
|
||||||
|
func ValidateProducerDiagnostics(diagnostics []ProducerDiagnostic) error {
|
||||||
|
if len(diagnostics) > MaxProducerDiagnosticGroups {
|
||||||
|
return errors.New("producer diagnostics exceed maximum group count")
|
||||||
|
}
|
||||||
|
for index, diagnostic := range diagnostics {
|
||||||
|
if err := diagnostic.Validate(); err != nil {
|
||||||
|
return fmt.Errorf("producer diagnostic %d: %w", index, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate checks framework-owned origin fields.
|
||||||
|
func (origin DiagnosticOrigin) Validate() error {
|
||||||
|
switch origin.Stage {
|
||||||
|
case DiagnosticOriginStageReferences, DiagnosticOriginStageChunk, DiagnosticOriginStageExtract, DiagnosticOriginStageMerge, DiagnosticOriginStageNormalize:
|
||||||
|
default:
|
||||||
|
return errors.New("diagnostic origin stage is invalid")
|
||||||
|
}
|
||||||
|
for _, field := range []struct {
|
||||||
|
name string
|
||||||
|
value string
|
||||||
|
}{
|
||||||
|
{name: "diagnostic origin step ID", value: origin.StepID},
|
||||||
|
{name: "diagnostic origin lane ID", value: origin.LaneID},
|
||||||
|
{name: "diagnostic origin module key", value: origin.ModuleKey},
|
||||||
|
{name: "diagnostic origin validator key", value: origin.ValidatorKey},
|
||||||
|
} {
|
||||||
|
if field.value != "" && (!utf8.ValidString(field.value) || strings.TrimSpace(field.value) == "") {
|
||||||
|
return fmt.Errorf("%s must be valid nonblank UTF-8 when present", field.name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate checks a final origin-enriched group.
|
||||||
|
func (group DiagnosticGroup) Validate() error {
|
||||||
|
if err := group.Origin.Validate(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return validateDiagnostic(
|
||||||
|
group.Disposition,
|
||||||
|
group.Category,
|
||||||
|
group.ReasonCode,
|
||||||
|
group.OccurrenceCount,
|
||||||
|
group.Samples,
|
||||||
|
group.OmittedSampleCount,
|
||||||
|
true,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate checks the collection shape without imposing later global
|
||||||
|
// aggregation limits.
|
||||||
|
func (collection DiagnosticCollection) Validate() error {
|
||||||
|
if collection.UnrepresentedOccurrenceCount < 0 {
|
||||||
|
return errors.New("diagnostic collection unrepresented occurrence count must not be negative")
|
||||||
|
}
|
||||||
|
if !collection.Truncated && collection.UnrepresentedOccurrenceCount != 0 {
|
||||||
|
return errors.New("diagnostic collection has unrepresented occurrences without truncation")
|
||||||
|
}
|
||||||
|
seen := make(map[diagnosticGroupKey]struct{}, len(collection.Groups))
|
||||||
|
for index, group := range collection.Groups {
|
||||||
|
if err := group.Validate(); err != nil {
|
||||||
|
return fmt.Errorf("diagnostic group %d: %w", index, err)
|
||||||
|
}
|
||||||
|
key := diagnosticGroupKeyFromGroup(group)
|
||||||
|
if _, exists := seen[key]; exists {
|
||||||
|
return errors.New("diagnostic collection contains duplicate group identity")
|
||||||
|
}
|
||||||
|
seen[key] = struct{}{}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ProjectDiagnosticCollection validates, partitions, and totals one finalized
|
||||||
|
// collection. Unrepresented occurrences belong to the non-warning projection.
|
||||||
|
func ProjectDiagnosticCollection(collection DiagnosticCollection) (DiagnosticProjection, error) {
|
||||||
|
if err := collection.Validate(); err != nil {
|
||||||
|
return DiagnosticProjection{}, err
|
||||||
|
}
|
||||||
|
projection := DiagnosticProjection{
|
||||||
|
Warnings: make([]DiagnosticGroup, 0),
|
||||||
|
Diagnostics: make([]DiagnosticGroup, 0),
|
||||||
|
}
|
||||||
|
for _, group := range collection.Groups {
|
||||||
|
if group.Disposition == DiagnosticDispositionWarning {
|
||||||
|
count, err := addDiagnosticOccurrences(projection.WarningOccurrenceCount, group.OccurrenceCount)
|
||||||
|
if err != nil {
|
||||||
|
return DiagnosticProjection{}, fmt.Errorf("warning occurrences: %w", err)
|
||||||
|
}
|
||||||
|
projection.WarningOccurrenceCount = count
|
||||||
|
projection.Warnings = append(projection.Warnings, cloneDiagnosticGroup(group))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
count, err := addDiagnosticOccurrences(projection.DiagnosticOccurrenceCount, group.OccurrenceCount)
|
||||||
|
if err != nil {
|
||||||
|
return DiagnosticProjection{}, fmt.Errorf("diagnostic occurrences: %w", err)
|
||||||
|
}
|
||||||
|
projection.DiagnosticOccurrenceCount = count
|
||||||
|
projection.Diagnostics = append(projection.Diagnostics, cloneDiagnosticGroup(group))
|
||||||
|
}
|
||||||
|
count, err := addDiagnosticOccurrences(projection.DiagnosticOccurrenceCount, collection.UnrepresentedOccurrenceCount)
|
||||||
|
if err != nil {
|
||||||
|
return DiagnosticProjection{}, fmt.Errorf("diagnostic occurrences: %w", err)
|
||||||
|
}
|
||||||
|
projection.DiagnosticOccurrenceCount = count
|
||||||
|
return projection, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// CloneProducerDiagnostics returns independent diagnostic slice ownership.
|
||||||
|
func CloneProducerDiagnostics(diagnostics []ProducerDiagnostic) []ProducerDiagnostic {
|
||||||
|
if len(diagnostics) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
cloned := make([]ProducerDiagnostic, len(diagnostics))
|
||||||
|
for index, diagnostic := range diagnostics {
|
||||||
|
cloned[index] = cloneProducerDiagnostic(diagnostic)
|
||||||
|
}
|
||||||
|
return cloned
|
||||||
|
}
|
||||||
|
|
||||||
|
// CloneDiagnosticCollection returns independent collection ownership.
|
||||||
|
func CloneDiagnosticCollection(collection DiagnosticCollection) DiagnosticCollection {
|
||||||
|
groups := collection.Groups
|
||||||
|
collection.Groups = make([]DiagnosticGroup, len(groups))
|
||||||
|
for index, group := range groups {
|
||||||
|
collection.Groups[index] = cloneDiagnosticGroup(group)
|
||||||
|
}
|
||||||
|
return collection
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateDiagnostic(disposition DiagnosticDisposition, category DiagnosticCategory, reasonCode string, occurrenceCount int, samples []DiagnosticSample, omittedSampleCount int, allowChunkContext bool) error {
|
||||||
|
if !diagnosticCategoryAllowed(disposition, category) {
|
||||||
|
return errors.New("diagnostic disposition and category combination is invalid")
|
||||||
|
}
|
||||||
|
if err := validateDiagnosticText(reasonCode, MaxDiagnosticReasonCodeBytes, "diagnostic reason code"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if occurrenceCount <= 0 {
|
||||||
|
return errors.New("diagnostic occurrence count must be positive")
|
||||||
|
}
|
||||||
|
if len(samples) == 0 {
|
||||||
|
return errors.New("diagnostic samples must not be empty")
|
||||||
|
}
|
||||||
|
if len(samples) > MaxDiagnosticSamples {
|
||||||
|
return errors.New("diagnostic samples exceed maximum count")
|
||||||
|
}
|
||||||
|
seen := make(map[diagnosticSampleKey]struct{}, len(samples))
|
||||||
|
for index, sample := range samples {
|
||||||
|
if err := validateDiagnosticSample(sample, allowChunkContext); err != nil {
|
||||||
|
return fmt.Errorf("diagnostic sample %d: %w", index, err)
|
||||||
|
}
|
||||||
|
key := diagnosticSampleKeyFromSample(sample)
|
||||||
|
if _, exists := seen[key]; exists {
|
||||||
|
return errors.New("diagnostic samples must be distinct")
|
||||||
|
}
|
||||||
|
seen[key] = struct{}{}
|
||||||
|
}
|
||||||
|
if occurrenceCount < len(samples) {
|
||||||
|
return errors.New("diagnostic occurrence count is smaller than sample count")
|
||||||
|
}
|
||||||
|
if omittedSampleCount != occurrenceCount-len(samples) {
|
||||||
|
return errors.New("diagnostic omitted sample count is inconsistent")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func diagnosticCategoryAllowed(disposition DiagnosticDisposition, category DiagnosticCategory) bool {
|
||||||
|
switch disposition {
|
||||||
|
case DiagnosticDispositionWarning:
|
||||||
|
return category == DiagnosticCategoryConfiguration || category == DiagnosticCategoryDegradation || category == DiagnosticCategoryValidationIncomplete || category == DiagnosticCategoryFallback
|
||||||
|
case DiagnosticDispositionAdvisory:
|
||||||
|
return category == DiagnosticCategoryDataQuality
|
||||||
|
case DiagnosticDispositionObservation:
|
||||||
|
return category == DiagnosticCategoryNormalization
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateDiagnosticSample(sample DiagnosticSample, allowChunkContext bool) error {
|
||||||
|
if err := validateDiagnosticText(sample.Scope, MaxDiagnosticScopeBytes, "diagnostic sample scope"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := validateDiagnosticText(sample.Message, MaxDiagnosticMessageBytes, "diagnostic sample message"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !allowChunkContext && (sample.ChunkID != "" || sample.ChunkIndex != nil) {
|
||||||
|
return errors.New("producer diagnostic sample must not include framework chunk context")
|
||||||
|
}
|
||||||
|
if sample.ChunkID != "" && (!utf8.ValidString(sample.ChunkID) || strings.TrimSpace(sample.ChunkID) == "") {
|
||||||
|
return errors.New("diagnostic sample chunk ID must be valid nonblank UTF-8 when present")
|
||||||
|
}
|
||||||
|
if sample.ChunkIndex != nil && *sample.ChunkIndex < 0 {
|
||||||
|
return errors.New("diagnostic sample chunk index must not be negative")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateDiagnosticText(value string, maximum int, name string) error {
|
||||||
|
if !utf8.ValidString(value) {
|
||||||
|
return fmt.Errorf("%s must be valid UTF-8", name)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(value) == "" {
|
||||||
|
return fmt.Errorf("%s must not be blank", name)
|
||||||
|
}
|
||||||
|
if len(value) > maximum {
|
||||||
|
return fmt.Errorf("%s exceeds maximum length", name)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneProducerDiagnostic(diagnostic ProducerDiagnostic) ProducerDiagnostic {
|
||||||
|
diagnostic.Samples = cloneDiagnosticSamples(diagnostic.Samples)
|
||||||
|
return diagnostic
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneDiagnosticGroup(group DiagnosticGroup) DiagnosticGroup {
|
||||||
|
group.Samples = cloneDiagnosticSamples(group.Samples)
|
||||||
|
return group
|
||||||
|
}
|
||||||
|
|
||||||
|
func addDiagnosticOccurrences(current, incoming int) (int, error) {
|
||||||
|
if incoming > int(^uint(0)>>1)-current {
|
||||||
|
return 0, errors.New("occurrence count overflow")
|
||||||
|
}
|
||||||
|
return current + incoming, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneDiagnosticSamples(samples []DiagnosticSample) []DiagnosticSample {
|
||||||
|
if len(samples) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
cloned := make([]DiagnosticSample, len(samples))
|
||||||
|
for index, sample := range samples {
|
||||||
|
if sample.ChunkIndex != nil {
|
||||||
|
chunkIndex := *sample.ChunkIndex
|
||||||
|
sample.ChunkIndex = &chunkIndex
|
||||||
|
}
|
||||||
|
cloned[index] = sample
|
||||||
|
}
|
||||||
|
return cloned
|
||||||
|
}
|
||||||
|
|
||||||
|
type diagnosticSampleKey struct {
|
||||||
|
scope string
|
||||||
|
message string
|
||||||
|
chunkID string
|
||||||
|
chunkIndex int
|
||||||
|
hasChunkIndex bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func diagnosticSampleKeyFromSample(sample DiagnosticSample) diagnosticSampleKey {
|
||||||
|
key := diagnosticSampleKey{scope: sample.Scope, message: sample.Message, chunkID: sample.ChunkID}
|
||||||
|
if sample.ChunkIndex != nil {
|
||||||
|
key.chunkIndex = *sample.ChunkIndex
|
||||||
|
key.hasChunkIndex = true
|
||||||
|
}
|
||||||
|
return key
|
||||||
|
}
|
||||||
|
|
||||||
|
type diagnosticGroupKey struct {
|
||||||
|
disposition DiagnosticDisposition
|
||||||
|
category DiagnosticCategory
|
||||||
|
reasonCode string
|
||||||
|
origin DiagnosticOrigin
|
||||||
|
}
|
||||||
|
|
||||||
|
func diagnosticGroupKeyFromGroup(group DiagnosticGroup) diagnosticGroupKey {
|
||||||
|
return diagnosticGroupKey{disposition: group.Disposition, category: group.Category, reasonCode: group.ReasonCode, origin: group.Origin}
|
||||||
|
}
|
||||||
198
internal/framework/contracts/diagnostics_test.go
Normal file
198
internal/framework/contracts/diagnostics_test.go
Normal file
@@ -0,0 +1,198 @@
|
|||||||
|
package contracts
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestProducerDiagnosticValidationAcceptsClassificationMatrix(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
disposition DiagnosticDisposition
|
||||||
|
category DiagnosticCategory
|
||||||
|
}{
|
||||||
|
{name: "configuration warning", disposition: DiagnosticDispositionWarning, category: DiagnosticCategoryConfiguration},
|
||||||
|
{name: "degradation warning", disposition: DiagnosticDispositionWarning, category: DiagnosticCategoryDegradation},
|
||||||
|
{name: "incomplete validation warning", disposition: DiagnosticDispositionWarning, category: DiagnosticCategoryValidationIncomplete},
|
||||||
|
{name: "fallback warning", disposition: DiagnosticDispositionWarning, category: DiagnosticCategoryFallback},
|
||||||
|
{name: "quality advisory", disposition: DiagnosticDispositionAdvisory, category: DiagnosticCategoryDataQuality},
|
||||||
|
{name: "normalization observation", disposition: DiagnosticDispositionObservation, category: DiagnosticCategoryNormalization},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
diagnostic := validProducerDiagnostic()
|
||||||
|
diagnostic.Disposition = test.disposition
|
||||||
|
diagnostic.Category = test.category
|
||||||
|
if err := diagnostic.Validate(); err != nil {
|
||||||
|
t.Fatalf("Validate() error = %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProducerDiagnosticValidationRejectsInvalidFieldsAndCounts(t *testing.T) {
|
||||||
|
tooLongReason := strings.Repeat("r", MaxDiagnosticReasonCodeBytes+1)
|
||||||
|
tooLongScope := strings.Repeat("s", MaxDiagnosticScopeBytes+1)
|
||||||
|
tooLongMessage := strings.Repeat("m", MaxDiagnosticMessageBytes+1)
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*ProducerDiagnostic)
|
||||||
|
}{
|
||||||
|
{name: "invalid classification", mutate: func(diagnostic *ProducerDiagnostic) { diagnostic.Category = DiagnosticCategoryDataQuality }},
|
||||||
|
{name: "blank reason", mutate: func(diagnostic *ProducerDiagnostic) { diagnostic.ReasonCode = " \t" }},
|
||||||
|
{name: "invalid reason UTF-8", mutate: func(diagnostic *ProducerDiagnostic) { diagnostic.ReasonCode = string([]byte{0xff}) }},
|
||||||
|
{name: "oversized reason", mutate: func(diagnostic *ProducerDiagnostic) { diagnostic.ReasonCode = tooLongReason }},
|
||||||
|
{name: "blank scope", mutate: func(diagnostic *ProducerDiagnostic) { diagnostic.Samples[0].Scope = "\n" }},
|
||||||
|
{name: "invalid scope UTF-8", mutate: func(diagnostic *ProducerDiagnostic) { diagnostic.Samples[0].Scope = string([]byte{0xff}) }},
|
||||||
|
{name: "oversized scope", mutate: func(diagnostic *ProducerDiagnostic) { diagnostic.Samples[0].Scope = tooLongScope }},
|
||||||
|
{name: "blank message", mutate: func(diagnostic *ProducerDiagnostic) { diagnostic.Samples[0].Message = " " }},
|
||||||
|
{name: "invalid message UTF-8", mutate: func(diagnostic *ProducerDiagnostic) { diagnostic.Samples[0].Message = string([]byte{0xff}) }},
|
||||||
|
{name: "oversized message", mutate: func(diagnostic *ProducerDiagnostic) { diagnostic.Samples[0].Message = tooLongMessage }},
|
||||||
|
{name: "zero occurrences", mutate: func(diagnostic *ProducerDiagnostic) { diagnostic.OccurrenceCount = 0 }},
|
||||||
|
{name: "missing samples", mutate: func(diagnostic *ProducerDiagnostic) {
|
||||||
|
diagnostic.Samples = nil
|
||||||
|
diagnostic.OmittedSampleCount = diagnostic.OccurrenceCount
|
||||||
|
}},
|
||||||
|
{name: "too many samples", mutate: func(diagnostic *ProducerDiagnostic) {
|
||||||
|
diagnostic.OccurrenceCount = 4
|
||||||
|
diagnostic.Samples = []DiagnosticSample{{Scope: "one", Message: "one"}, {Scope: "two", Message: "two"}, {Scope: "three", Message: "three"}, {Scope: "four", Message: "four"}}
|
||||||
|
diagnostic.OmittedSampleCount = 0
|
||||||
|
}},
|
||||||
|
{name: "duplicate samples", mutate: func(diagnostic *ProducerDiagnostic) {
|
||||||
|
diagnostic.OccurrenceCount = 2
|
||||||
|
diagnostic.Samples = []DiagnosticSample{{Scope: "scope", Message: "message"}, {Scope: "scope", Message: "message"}}
|
||||||
|
diagnostic.OmittedSampleCount = 0
|
||||||
|
}},
|
||||||
|
{name: "inconsistent omission", mutate: func(diagnostic *ProducerDiagnostic) { diagnostic.OmittedSampleCount = 1 }},
|
||||||
|
{name: "producer chunk context", mutate: func(diagnostic *ProducerDiagnostic) { chunkIndex := 0; diagnostic.Samples[0].ChunkIndex = &chunkIndex }},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
diagnostic := validProducerDiagnostic()
|
||||||
|
test.mutate(&diagnostic)
|
||||||
|
if err := diagnostic.Validate(); err == nil {
|
||||||
|
t.Fatal("Validate() error = nil, want invalid diagnostic error")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateProducerDiagnosticsEnforcesLocalGroupBound(t *testing.T) {
|
||||||
|
diagnostics := make([]ProducerDiagnostic, MaxProducerDiagnosticGroups)
|
||||||
|
for index := range diagnostics {
|
||||||
|
diagnostics[index] = validProducerDiagnostic()
|
||||||
|
diagnostics[index].ReasonCode = "reason-" + string(rune('a'+index))
|
||||||
|
}
|
||||||
|
if err := ValidateProducerDiagnostics(diagnostics); err != nil {
|
||||||
|
t.Fatalf("ValidateProducerDiagnostics() error = %v", err)
|
||||||
|
}
|
||||||
|
diagnostics = append(diagnostics, validProducerDiagnostic())
|
||||||
|
if err := ValidateProducerDiagnostics(diagnostics); err == nil {
|
||||||
|
t.Fatal("ValidateProducerDiagnostics() error = nil, want excessive-group error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDiagnosticGroupValidationPreservesChunkIndexZero(t *testing.T) {
|
||||||
|
chunkIndex := 0
|
||||||
|
group := DiagnosticGroup{
|
||||||
|
Disposition: DiagnosticDispositionAdvisory,
|
||||||
|
Category: DiagnosticCategoryDataQuality,
|
||||||
|
ReasonCode: "unresolved",
|
||||||
|
Origin: DiagnosticOrigin{Stage: DiagnosticOriginStageExtract, StepID: "extract", LaneID: "spells", ModuleKey: "dnd/spells", ValidatorKey: "dnd/spells/source-relatedness"},
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []DiagnosticSample{{Scope: "spells[0]", Message: "Spell was not found", ChunkID: "chunk-1", ChunkIndex: &chunkIndex}},
|
||||||
|
}
|
||||||
|
if err := group.Validate(); err != nil {
|
||||||
|
t.Fatalf("Validate() error = %v", err)
|
||||||
|
}
|
||||||
|
collection := DiagnosticCollection{Groups: []DiagnosticGroup{group}}
|
||||||
|
if err := collection.Validate(); err != nil {
|
||||||
|
t.Fatalf("DiagnosticCollection.Validate() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDiagnosticGroupRejectsRepeatedSampleWithEqualChunkIndex(t *testing.T) {
|
||||||
|
firstIndex := 0
|
||||||
|
secondIndex := 0
|
||||||
|
group := DiagnosticGroup{
|
||||||
|
Disposition: DiagnosticDispositionAdvisory,
|
||||||
|
Category: DiagnosticCategoryDataQuality,
|
||||||
|
ReasonCode: "unresolved",
|
||||||
|
Origin: DiagnosticOrigin{Stage: DiagnosticOriginStageExtract, StepID: "extract", LaneID: "spells", ModuleKey: "dnd/spells"},
|
||||||
|
OccurrenceCount: 2,
|
||||||
|
Samples: []DiagnosticSample{
|
||||||
|
{Scope: "spells[0]", Message: "Spell was not found", ChunkID: "chunk-1", ChunkIndex: &firstIndex},
|
||||||
|
{Scope: "spells[0]", Message: "Spell was not found", ChunkID: "chunk-1", ChunkIndex: &secondIndex},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if err := group.Validate(); err == nil {
|
||||||
|
t.Fatal("Validate() error = nil, want duplicate sample error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCloneDiagnosticCollectionOwnsGroupsAndChunkIndex(t *testing.T) {
|
||||||
|
chunkIndex := 0
|
||||||
|
collection := DiagnosticCollection{Groups: []DiagnosticGroup{{
|
||||||
|
Disposition: DiagnosticDispositionAdvisory,
|
||||||
|
Category: DiagnosticCategoryDataQuality,
|
||||||
|
ReasonCode: "unresolved",
|
||||||
|
Origin: DiagnosticOrigin{Stage: DiagnosticOriginStageExtract, StepID: "extract", LaneID: "spells", ModuleKey: "dnd/spells"},
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []DiagnosticSample{{Scope: "spells[0]", Message: "Spell was not found", ChunkIndex: &chunkIndex}},
|
||||||
|
}}}
|
||||||
|
cloned := CloneDiagnosticCollection(collection)
|
||||||
|
collection.Groups[0].Samples[0].Message = "changed"
|
||||||
|
*collection.Groups[0].Samples[0].ChunkIndex = 1
|
||||||
|
if got := cloned.Groups[0].Samples[0]; got.Message != "Spell was not found" || got.ChunkIndex == nil || *got.ChunkIndex != 0 {
|
||||||
|
t.Fatalf("cloned sample = %#v, want independently owned original", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProjectDiagnosticCollectionPartitionsAndChecksTotals(t *testing.T) {
|
||||||
|
warning := validDiagnosticGroup(DiagnosticDispositionWarning, DiagnosticCategoryFallback, "fallback", 2)
|
||||||
|
diagnostic := validDiagnosticGroup(DiagnosticDispositionAdvisory, DiagnosticCategoryDataQuality, "quality", 3)
|
||||||
|
collection := DiagnosticCollection{
|
||||||
|
Groups: []DiagnosticGroup{warning, diagnostic},
|
||||||
|
Truncated: true,
|
||||||
|
UnrepresentedOccurrenceCount: 4,
|
||||||
|
}
|
||||||
|
projection, err := ProjectDiagnosticCollection(collection)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(projection.Warnings) != 1 || len(projection.Diagnostics) != 1 || projection.WarningOccurrenceCount != 2 || projection.DiagnosticOccurrenceCount != 7 {
|
||||||
|
t.Fatalf("projection = %#v, want partitioned exact totals", projection)
|
||||||
|
}
|
||||||
|
collection.Groups[0].Samples[0].Message = "mutated"
|
||||||
|
if projection.Warnings[0].Samples[0].Message != "message" {
|
||||||
|
t.Fatal("projection retained caller-owned sample storage")
|
||||||
|
}
|
||||||
|
|
||||||
|
overflow := DiagnosticCollection{Groups: []DiagnosticGroup{
|
||||||
|
validDiagnosticGroup(DiagnosticDispositionWarning, DiagnosticCategoryFallback, "first", int(^uint(0)>>1)),
|
||||||
|
validDiagnosticGroup(DiagnosticDispositionWarning, DiagnosticCategoryFallback, "second", 1),
|
||||||
|
}}
|
||||||
|
if _, err := ProjectDiagnosticCollection(overflow); err == nil {
|
||||||
|
t.Fatal("ProjectDiagnosticCollection() overflow error = nil")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validDiagnosticGroup(disposition DiagnosticDisposition, category DiagnosticCategory, reason string, occurrences int) DiagnosticGroup {
|
||||||
|
return DiagnosticGroup{
|
||||||
|
Disposition: disposition,
|
||||||
|
Category: category,
|
||||||
|
ReasonCode: reason,
|
||||||
|
Origin: DiagnosticOrigin{Stage: DiagnosticOriginStageNormalize, StepID: "step", LaneID: "lane", ModuleKey: "module"},
|
||||||
|
OccurrenceCount: occurrences,
|
||||||
|
Samples: []DiagnosticSample{{Scope: "scope", Message: "message"}},
|
||||||
|
OmittedSampleCount: occurrences - 1,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validProducerDiagnostic() ProducerDiagnostic {
|
||||||
|
return ProducerDiagnostic{
|
||||||
|
Disposition: DiagnosticDispositionWarning,
|
||||||
|
Category: DiagnosticCategoryConfiguration,
|
||||||
|
ReasonCode: "empty_reference",
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []DiagnosticSample{{Scope: "references.glossary", Message: "Reference is empty"}},
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -48,7 +48,7 @@ type TypedExtractionRequest struct {
|
|||||||
|
|
||||||
type TypedExtractionResult[T any] struct {
|
type TypedExtractionResult[T any] struct {
|
||||||
Value T
|
Value T
|
||||||
Warnings []Warning
|
Diagnostics []ProducerDiagnostic
|
||||||
ModelCandidate *ModelCandidate
|
ModelCandidate *ModelCandidate
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -73,7 +73,7 @@ type TypedMergeRequest[T any] struct {
|
|||||||
|
|
||||||
type TypedMergeResult[T any] struct {
|
type TypedMergeResult[T any] struct {
|
||||||
Value T
|
Value T
|
||||||
Warnings []Warning
|
Diagnostics []ProducerDiagnostic
|
||||||
ModelCandidate *ModelCandidate
|
ModelCandidate *ModelCandidate
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -97,7 +97,7 @@ type TypedNormalizeRequest[T any] struct {
|
|||||||
|
|
||||||
type TypedNormalizeResult[T any] struct {
|
type TypedNormalizeResult[T any] struct {
|
||||||
Value T
|
Value T
|
||||||
Warnings []Warning
|
Diagnostics []ProducerDiagnostic
|
||||||
Retry *NormalizeRetry
|
Retry *NormalizeRetry
|
||||||
ModelCandidate *ModelCandidate
|
ModelCandidate *ModelCandidate
|
||||||
}
|
}
|
||||||
@@ -112,9 +112,9 @@ const (
|
|||||||
// NormalizeRetry asks the framework to retry normalization while retaining a
|
// NormalizeRetry asks the framework to retry normalization while retaining a
|
||||||
// safe candidate for acceptance if the retry budget is exhausted.
|
// safe candidate for acceptance if the retry budget is exhausted.
|
||||||
type NormalizeRetry struct {
|
type NormalizeRetry struct {
|
||||||
ReasonCode string
|
ReasonCode string
|
||||||
Message string
|
Message string
|
||||||
FallbackWarnings []Warning
|
FallbackDiagnostics []ProducerDiagnostic
|
||||||
}
|
}
|
||||||
|
|
||||||
type Normalizer[T any] interface {
|
type Normalizer[T any] interface {
|
||||||
|
|||||||
164
internal/framework/diagnostics/aggregator.go
Normal file
164
internal/framework/diagnostics/aggregator.go
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
package diagnostics
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
MaxWarningGroups = 128
|
||||||
|
MaxNonWarningGroups = 256
|
||||||
|
)
|
||||||
|
|
||||||
|
// Aggregator merges origin-enriched diagnostics in caller-supplied canonical
|
||||||
|
// order. Its zero value is ready for use.
|
||||||
|
type Aggregator struct {
|
||||||
|
groups []contracts.DiagnosticGroup
|
||||||
|
indices map[groupKey]int
|
||||||
|
warningGroups int
|
||||||
|
nonWarningGroups int
|
||||||
|
warningOccurrences int
|
||||||
|
nonWarningOccurrences int
|
||||||
|
unrepresentedOccurrences int
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add validates and incorporates one final diagnostic group. Actionable
|
||||||
|
// warnings cannot overflow; later non-warning groups are represented by exact
|
||||||
|
// unrepresented-occurrence metadata once their fixed bound is reached.
|
||||||
|
func (aggregator *Aggregator) Add(group contracts.DiagnosticGroup) error {
|
||||||
|
if err := group.Validate(); err != nil {
|
||||||
|
return fmt.Errorf("diagnostic group: %w", err)
|
||||||
|
}
|
||||||
|
if aggregator.indices == nil {
|
||||||
|
aggregator.indices = make(map[groupKey]int)
|
||||||
|
}
|
||||||
|
key := groupKeyFromGroup(group)
|
||||||
|
if index, exists := aggregator.indices[key]; exists {
|
||||||
|
if err := aggregator.checkOccurrenceTotal(group.Disposition, group.OccurrenceCount); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := aggregator.merge(index, group); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
aggregator.addOccurrenceTotal(group.Disposition, group.OccurrenceCount)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if group.Disposition == contracts.DiagnosticDispositionWarning {
|
||||||
|
if aggregator.warningGroups >= MaxWarningGroups {
|
||||||
|
return errors.New("diagnostic warning groups exceed maximum count")
|
||||||
|
}
|
||||||
|
} else if aggregator.nonWarningGroups >= MaxNonWarningGroups {
|
||||||
|
if err := aggregator.checkOccurrenceTotal(group.Disposition, group.OccurrenceCount); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
aggregator.addOccurrenceTotal(group.Disposition, group.OccurrenceCount)
|
||||||
|
return aggregator.addUnrepresented(group.OccurrenceCount)
|
||||||
|
}
|
||||||
|
if err := aggregator.checkOccurrenceTotal(group.Disposition, group.OccurrenceCount); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if group.Disposition == contracts.DiagnosticDispositionWarning {
|
||||||
|
aggregator.warningGroups++
|
||||||
|
} else {
|
||||||
|
aggregator.nonWarningGroups++
|
||||||
|
}
|
||||||
|
aggregator.addOccurrenceTotal(group.Disposition, group.OccurrenceCount)
|
||||||
|
aggregator.indices[key] = len(aggregator.groups)
|
||||||
|
aggregator.groups = append(aggregator.groups, contracts.CloneDiagnosticCollection(contracts.DiagnosticCollection{Groups: []contracts.DiagnosticGroup{group}}).Groups[0])
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (aggregator *Aggregator) checkOccurrenceTotal(disposition contracts.DiagnosticDisposition, count int) error {
|
||||||
|
current := aggregator.nonWarningOccurrences
|
||||||
|
if disposition == contracts.DiagnosticDispositionWarning {
|
||||||
|
current = aggregator.warningOccurrences
|
||||||
|
}
|
||||||
|
if count > maximumInt()-current {
|
||||||
|
return errors.New("diagnostic occurrence count overflow")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (aggregator *Aggregator) addOccurrenceTotal(disposition contracts.DiagnosticDisposition, count int) {
|
||||||
|
if disposition == contracts.DiagnosticDispositionWarning {
|
||||||
|
aggregator.warningOccurrences += count
|
||||||
|
return
|
||||||
|
}
|
||||||
|
aggregator.nonWarningOccurrences += count
|
||||||
|
}
|
||||||
|
|
||||||
|
// Collection returns an independently owned grouped result in first-occurrence
|
||||||
|
// order.
|
||||||
|
func (aggregator *Aggregator) Collection() contracts.DiagnosticCollection {
|
||||||
|
if aggregator == nil {
|
||||||
|
return contracts.DiagnosticCollection{}
|
||||||
|
}
|
||||||
|
return contracts.CloneDiagnosticCollection(contracts.DiagnosticCollection{
|
||||||
|
Groups: aggregator.groups,
|
||||||
|
Truncated: aggregator.unrepresentedOccurrences > 0,
|
||||||
|
UnrepresentedOccurrenceCount: aggregator.unrepresentedOccurrences,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (aggregator *Aggregator) merge(index int, incoming contracts.DiagnosticGroup) error {
|
||||||
|
current := &aggregator.groups[index]
|
||||||
|
if incoming.OccurrenceCount > maximumInt()-current.OccurrenceCount {
|
||||||
|
return errors.New("diagnostic occurrence count overflow")
|
||||||
|
}
|
||||||
|
current.OccurrenceCount += incoming.OccurrenceCount
|
||||||
|
for _, sample := range incoming.Samples {
|
||||||
|
if len(current.Samples) == contracts.MaxDiagnosticSamples || containsGroupSample(current.Samples, sample) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
current.Samples = append(current.Samples, cloneGroupSample(sample))
|
||||||
|
}
|
||||||
|
current.OmittedSampleCount = current.OccurrenceCount - len(current.Samples)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (aggregator *Aggregator) addUnrepresented(count int) error {
|
||||||
|
if count > maximumInt()-aggregator.unrepresentedOccurrences {
|
||||||
|
return errors.New("diagnostic unrepresented occurrence count overflow")
|
||||||
|
}
|
||||||
|
aggregator.unrepresentedOccurrences += count
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func containsGroupSample(samples []contracts.DiagnosticSample, candidate contracts.DiagnosticSample) bool {
|
||||||
|
for _, sample := range samples {
|
||||||
|
if sample.Scope != candidate.Scope || sample.Message != candidate.Message || sample.ChunkID != candidate.ChunkID {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if sample.ChunkIndex == nil || candidate.ChunkIndex == nil {
|
||||||
|
if sample.ChunkIndex == candidate.ChunkIndex {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if *sample.ChunkIndex == *candidate.ChunkIndex {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneGroupSample(sample contracts.DiagnosticSample) contracts.DiagnosticSample {
|
||||||
|
if sample.ChunkIndex != nil {
|
||||||
|
value := *sample.ChunkIndex
|
||||||
|
sample.ChunkIndex = &value
|
||||||
|
}
|
||||||
|
return sample
|
||||||
|
}
|
||||||
|
|
||||||
|
type groupKey struct {
|
||||||
|
disposition contracts.DiagnosticDisposition
|
||||||
|
category contracts.DiagnosticCategory
|
||||||
|
reasonCode string
|
||||||
|
origin contracts.DiagnosticOrigin
|
||||||
|
}
|
||||||
|
|
||||||
|
func groupKeyFromGroup(group contracts.DiagnosticGroup) groupKey {
|
||||||
|
return groupKey{disposition: group.Disposition, category: group.Category, reasonCode: group.ReasonCode, origin: group.Origin}
|
||||||
|
}
|
||||||
89
internal/framework/diagnostics/aggregator_test.go
Normal file
89
internal/framework/diagnostics/aggregator_test.go
Normal file
@@ -0,0 +1,89 @@
|
|||||||
|
package diagnostics
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAggregatorMergesByOriginAndPreservesFirstOccurrenceOrder(t *testing.T) {
|
||||||
|
aggregator := Aggregator{}
|
||||||
|
for _, group := range []contracts.DiagnosticGroup{
|
||||||
|
groupForAggregation("first", "message one", contracts.DiagnosticDispositionAdvisory, contracts.DiagnosticCategoryDataQuality, "dnd/spells"),
|
||||||
|
groupForAggregation("second", "message two", contracts.DiagnosticDispositionWarning, contracts.DiagnosticCategoryFallback, "dnd/items"),
|
||||||
|
groupForAggregation("third", "message three", contracts.DiagnosticDispositionAdvisory, contracts.DiagnosticCategoryDataQuality, "dnd/spells"),
|
||||||
|
} {
|
||||||
|
if err := aggregator.Add(group); err != nil {
|
||||||
|
t.Fatalf("Add() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
collection := aggregator.Collection()
|
||||||
|
if len(collection.Groups) != 2 {
|
||||||
|
t.Fatalf("group count = %d, want 2", len(collection.Groups))
|
||||||
|
}
|
||||||
|
if got := []string{collection.Groups[0].Origin.ModuleKey, collection.Groups[1].Origin.ModuleKey}; !equalStrings(got, []string{"dnd/spells", "dnd/items"}) {
|
||||||
|
t.Fatalf("group order = %#v, want first occurrence order", got)
|
||||||
|
}
|
||||||
|
if collection.Groups[0].OccurrenceCount != 2 || collection.Groups[0].OmittedSampleCount != 0 {
|
||||||
|
t.Fatalf("merged group = %#v, want two represented occurrences", collection.Groups[0])
|
||||||
|
}
|
||||||
|
if got := []string{collection.Groups[0].Samples[0].Scope, collection.Groups[0].Samples[1].Scope}; !equalStrings(got, []string{"first", "third"}) {
|
||||||
|
t.Fatalf("merged samples = %#v, want first occurrence order", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAggregatorEnforcesWarningBoundAndTruncatesOnlyNonWarnings(t *testing.T) {
|
||||||
|
warnings := Aggregator{}
|
||||||
|
for index := 0; index < MaxWarningGroups; index++ {
|
||||||
|
group := groupForAggregation("scope", "message", contracts.DiagnosticDispositionWarning, contracts.DiagnosticCategoryFallback, "module")
|
||||||
|
group.ReasonCode = "warning-" + string(rune('a'+index))
|
||||||
|
if err := warnings.Add(group); err != nil {
|
||||||
|
t.Fatalf("warning Add(%d) error = %v", index, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := warnings.Add(groupForAggregation("overflow", "overflow", contracts.DiagnosticDispositionWarning, contracts.DiagnosticCategoryFallback, "overflow")); err == nil {
|
||||||
|
t.Fatal("warning overflow error = nil, want error")
|
||||||
|
}
|
||||||
|
|
||||||
|
nonWarnings := Aggregator{}
|
||||||
|
for index := 0; index < MaxNonWarningGroups+3; index++ {
|
||||||
|
group := groupForAggregation("scope", "message", contracts.DiagnosticDispositionAdvisory, contracts.DiagnosticCategoryDataQuality, "module")
|
||||||
|
group.ReasonCode = "advisory-" + string(rune('a'+index))
|
||||||
|
if err := nonWarnings.Add(group); err != nil {
|
||||||
|
t.Fatalf("non-warning Add(%d) error = %v", index, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
collection := nonWarnings.Collection()
|
||||||
|
if len(collection.Groups) != MaxNonWarningGroups || !collection.Truncated || collection.UnrepresentedOccurrenceCount != 3 {
|
||||||
|
t.Fatalf("collection = %#v, want bounded non-warning groups and three unrepresented occurrences", collection)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAggregatorRejectsOccurrenceTotalOverflow(t *testing.T) {
|
||||||
|
aggregator := Aggregator{}
|
||||||
|
first := groupForAggregation("first", "first", contracts.DiagnosticDispositionAdvisory, contracts.DiagnosticCategoryDataQuality, "first")
|
||||||
|
first.OccurrenceCount = maximumInt()
|
||||||
|
first.OmittedSampleCount = maximumInt() - 1
|
||||||
|
if err := aggregator.Add(first); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
second := groupForAggregation("second", "second", contracts.DiagnosticDispositionAdvisory, contracts.DiagnosticCategoryDataQuality, "second")
|
||||||
|
if err := aggregator.Add(second); err == nil {
|
||||||
|
t.Fatal("Add() occurrence total overflow error = nil")
|
||||||
|
}
|
||||||
|
collection := aggregator.Collection()
|
||||||
|
if len(collection.Groups) != 1 || collection.Groups[0].OccurrenceCount != maximumInt() {
|
||||||
|
t.Fatalf("collection changed after rejected overflow = %#v", collection)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func groupForAggregation(scope string, message string, disposition contracts.DiagnosticDisposition, category contracts.DiagnosticCategory, module string) contracts.DiagnosticGroup {
|
||||||
|
return contracts.DiagnosticGroup{
|
||||||
|
Disposition: disposition,
|
||||||
|
Category: category,
|
||||||
|
ReasonCode: "source_unrelated",
|
||||||
|
Origin: contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageExtract, StepID: "extract", LaneID: "spells", ModuleKey: module},
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []contracts.DiagnosticSample{{Scope: scope, Message: message}},
|
||||||
|
}
|
||||||
|
}
|
||||||
93
internal/framework/diagnostics/collector.go
Normal file
93
internal/framework/diagnostics/collector.go
Normal file
@@ -0,0 +1,93 @@
|
|||||||
|
// Package diagnostics provides bounded local grouping for producer and
|
||||||
|
// validator diagnostic results.
|
||||||
|
package diagnostics
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Collector merges local producer diagnostics by their semantic identity. Its
|
||||||
|
// zero value is ready for use.
|
||||||
|
type Collector struct {
|
||||||
|
diagnostics []contracts.ProducerDiagnostic
|
||||||
|
indices map[key]int
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewCollector returns an empty local diagnostic collector.
|
||||||
|
func NewCollector() *Collector {
|
||||||
|
return &Collector{}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add validates and merges one producer diagnostic. Every occurrence remains
|
||||||
|
// counted, while the first three distinct samples in input order are retained.
|
||||||
|
func (collector *Collector) Add(diagnostic contracts.ProducerDiagnostic) error {
|
||||||
|
if err := diagnostic.Validate(); err != nil {
|
||||||
|
return fmt.Errorf("producer diagnostic: %w", err)
|
||||||
|
}
|
||||||
|
if collector.indices == nil {
|
||||||
|
collector.indices = make(map[key]int)
|
||||||
|
}
|
||||||
|
diagnosticKey := key{disposition: diagnostic.Disposition, category: diagnostic.Category, reasonCode: diagnostic.ReasonCode}
|
||||||
|
index, exists := collector.indices[diagnosticKey]
|
||||||
|
if !exists {
|
||||||
|
if len(collector.diagnostics) >= contracts.MaxProducerDiagnosticGroups {
|
||||||
|
return errors.New("producer diagnostics exceed maximum group count")
|
||||||
|
}
|
||||||
|
collector.indices[diagnosticKey] = len(collector.diagnostics)
|
||||||
|
collector.diagnostics = append(collector.diagnostics, contracts.CloneProducerDiagnostics([]contracts.ProducerDiagnostic{diagnostic})[0])
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
current := &collector.diagnostics[index]
|
||||||
|
if diagnostic.OccurrenceCount > maximumInt()-current.OccurrenceCount {
|
||||||
|
return errors.New("producer diagnostic occurrence count overflow")
|
||||||
|
}
|
||||||
|
current.OccurrenceCount += diagnostic.OccurrenceCount
|
||||||
|
for _, sample := range diagnostic.Samples {
|
||||||
|
if len(current.Samples) == contracts.MaxDiagnosticSamples || containsSample(current.Samples, sample) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
current.Samples = append(current.Samples, cloneSample(sample))
|
||||||
|
}
|
||||||
|
current.OmittedSampleCount = current.OccurrenceCount - len(current.Samples)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Diagnostics returns an independently owned snapshot in first-occurrence
|
||||||
|
// order.
|
||||||
|
func (collector *Collector) Diagnostics() []contracts.ProducerDiagnostic {
|
||||||
|
if collector == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return contracts.CloneProducerDiagnostics(collector.diagnostics)
|
||||||
|
}
|
||||||
|
|
||||||
|
func containsSample(samples []contracts.DiagnosticSample, candidate contracts.DiagnosticSample) bool {
|
||||||
|
for _, sample := range samples {
|
||||||
|
if sample.Scope == candidate.Scope && sample.Message == candidate.Message {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneSample(sample contracts.DiagnosticSample) contracts.DiagnosticSample {
|
||||||
|
if sample.ChunkIndex != nil {
|
||||||
|
chunkIndex := *sample.ChunkIndex
|
||||||
|
sample.ChunkIndex = &chunkIndex
|
||||||
|
}
|
||||||
|
return sample
|
||||||
|
}
|
||||||
|
|
||||||
|
func maximumInt() int {
|
||||||
|
return int(^uint(0) >> 1)
|
||||||
|
}
|
||||||
|
|
||||||
|
type key struct {
|
||||||
|
disposition contracts.DiagnosticDisposition
|
||||||
|
category contracts.DiagnosticCategory
|
||||||
|
reasonCode string
|
||||||
|
}
|
||||||
103
internal/framework/diagnostics/collector_test.go
Normal file
103
internal/framework/diagnostics/collector_test.go
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
package diagnostics
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestCollectorCountsOccurrencesAndRetainsDistinctSamplesInOrder(t *testing.T) {
|
||||||
|
collector := NewCollector()
|
||||||
|
for _, diagnostic := range []contracts.ProducerDiagnostic{
|
||||||
|
advisory("one", "first"),
|
||||||
|
advisory("one", "first"),
|
||||||
|
advisory("two", "second"),
|
||||||
|
advisory("three", "third"),
|
||||||
|
advisory("four", "fourth"),
|
||||||
|
} {
|
||||||
|
if err := collector.Add(diagnostic); err != nil {
|
||||||
|
t.Fatalf("Add() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
diagnostics := collector.Diagnostics()
|
||||||
|
if len(diagnostics) != 1 {
|
||||||
|
t.Fatalf("group count = %d, want 1", len(diagnostics))
|
||||||
|
}
|
||||||
|
group := diagnostics[0]
|
||||||
|
if group.OccurrenceCount != 5 || group.OmittedSampleCount != 2 {
|
||||||
|
t.Fatalf("group counts = %#v, want five occurrences and two omitted samples", group)
|
||||||
|
}
|
||||||
|
if got := []string{group.Samples[0].Scope, group.Samples[1].Scope, group.Samples[2].Scope}; !equalStrings(got, []string{"one", "two", "three"}) {
|
||||||
|
t.Fatalf("sample order = %#v, want first three distinct samples", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCollectorSeparatesGroupsAndRejectsInvalidOrExcessiveGroups(t *testing.T) {
|
||||||
|
collector := NewCollector()
|
||||||
|
if err := collector.Add(advisory("one", "first")); err != nil {
|
||||||
|
t.Fatalf("Add() error = %v", err)
|
||||||
|
}
|
||||||
|
warning := advisory("two", "second")
|
||||||
|
warning.Disposition = contracts.DiagnosticDispositionWarning
|
||||||
|
warning.Category = contracts.DiagnosticCategoryFallback
|
||||||
|
if err := collector.Add(warning); err != nil {
|
||||||
|
t.Fatalf("Add() error = %v", err)
|
||||||
|
}
|
||||||
|
if got := len(collector.Diagnostics()); got != 2 {
|
||||||
|
t.Fatalf("group count = %d, want 2", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
invalid := advisory("bad", "bad")
|
||||||
|
invalid.ReasonCode = ""
|
||||||
|
if err := collector.Add(invalid); err == nil {
|
||||||
|
t.Fatal("Add() error = nil, want invalid diagnostic error")
|
||||||
|
}
|
||||||
|
|
||||||
|
limited := NewCollector()
|
||||||
|
for index := 0; index < contracts.MaxProducerDiagnosticGroups; index++ {
|
||||||
|
diagnostic := advisory("scope", "message")
|
||||||
|
diagnostic.ReasonCode = "reason-" + string(rune('a'+index))
|
||||||
|
if err := limited.Add(diagnostic); err != nil {
|
||||||
|
t.Fatalf("Add(%d) error = %v", index, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := limited.Add(advisory("overflow", "overflow")); err == nil {
|
||||||
|
t.Fatal("Add() error = nil, want local group limit error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCollectorReturnsIndependentSnapshots(t *testing.T) {
|
||||||
|
collector := NewCollector()
|
||||||
|
if err := collector.Add(advisory("scope", "message")); err != nil {
|
||||||
|
t.Fatalf("Add() error = %v", err)
|
||||||
|
}
|
||||||
|
first := collector.Diagnostics()
|
||||||
|
first[0].Samples[0].Message = "changed"
|
||||||
|
second := collector.Diagnostics()
|
||||||
|
if second[0].Samples[0].Message != "message" {
|
||||||
|
t.Fatalf("collector snapshot changed = %#v", second)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func advisory(scope string, message string) contracts.ProducerDiagnostic {
|
||||||
|
return contracts.ProducerDiagnostic{
|
||||||
|
Disposition: contracts.DiagnosticDispositionAdvisory,
|
||||||
|
Category: contracts.DiagnosticCategoryDataQuality,
|
||||||
|
ReasonCode: "source_unrelated",
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []contracts.DiagnosticSample{{Scope: scope, Message: message}},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func equalStrings(left []string, right []string) bool {
|
||||||
|
if len(left) != len(right) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for index := range left {
|
||||||
|
if left[index] != right[index] {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
@@ -28,14 +28,14 @@ type CheckpointRecorder interface {
|
|||||||
SourceSucceeded(moduleKey string, doc *source.SourceDocument) error
|
SourceSucceeded(moduleKey string, doc *source.SourceDocument) error
|
||||||
SourceFailed(moduleKey string, err error) error
|
SourceFailed(moduleKey string, err error) error
|
||||||
ExtractRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
ExtractRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||||
ExtractSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error
|
ExtractSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput) error
|
||||||
ExtractFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
ExtractFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||||
MergeRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
MergeRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||||
MergeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
|
MergeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact) error
|
||||||
MergeRejected(laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
MergeRejected(laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
||||||
MergeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
MergeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||||
NormalizeRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
NormalizeRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||||
NormalizeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
|
NormalizeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact) error
|
||||||
NormalizeRejected(laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
NormalizeRejected(laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
||||||
NormalizeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
NormalizeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||||
}
|
}
|
||||||
@@ -45,14 +45,14 @@ type CheckpointRecorder interface {
|
|||||||
// remain available for callers that do not have step context.
|
// remain available for callers that do not have step context.
|
||||||
type StepCheckpointRecorder interface {
|
type StepCheckpointRecorder interface {
|
||||||
ExtractRunningForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
ExtractRunningForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||||
ExtractSucceededForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error
|
ExtractSucceededForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput) error
|
||||||
ExtractFailedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
ExtractFailedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||||
MergeRunningForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
MergeRunningForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||||
MergeSucceededForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
|
MergeSucceededForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact) error
|
||||||
MergeRejectedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
MergeRejectedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
||||||
MergeFailedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
MergeFailedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||||
NormalizeRunningForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
NormalizeRunningForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||||
NormalizeSucceededForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
|
NormalizeSucceededForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact) error
|
||||||
NormalizeRejectedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
NormalizeRejectedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
||||||
NormalizeFailedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
NormalizeFailedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||||
}
|
}
|
||||||
@@ -279,21 +279,27 @@ type CheckpointArtifact struct {
|
|||||||
ChunkRef source.SourceRef
|
ChunkRef source.SourceRef
|
||||||
Artifact contracts.SerializedArtifact
|
Artifact contracts.SerializedArtifact
|
||||||
SchemaDigest string
|
SchemaDigest string
|
||||||
|
Diagnostics []CheckpointDiagnostic
|
||||||
|
}
|
||||||
|
|
||||||
|
// CheckpointDiagnostic stores a producer-local diagnostic alongside a
|
||||||
|
// reusable artifact. The runner supplies the current run's origin when it
|
||||||
|
// promotes this value into a diagnostic group.
|
||||||
|
type CheckpointDiagnostic struct {
|
||||||
|
Diagnostic contracts.ProducerDiagnostic `json:"diagnostic"`
|
||||||
|
ValidatorKey string `json:"validator_key,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ExtractCheckpoint struct {
|
type ExtractCheckpoint struct {
|
||||||
Outputs []CheckpointArtifact
|
Outputs []CheckpointArtifact
|
||||||
Rejected []contracts.RejectedOutput
|
Rejected []contracts.RejectedOutput
|
||||||
Warnings []contracts.Warning
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type MergeCheckpoint struct {
|
type MergeCheckpoint struct {
|
||||||
Output CheckpointArtifact
|
Output CheckpointArtifact
|
||||||
Warnings []contracts.Warning
|
|
||||||
}
|
}
|
||||||
type NormalizeCheckpoint struct {
|
type NormalizeCheckpoint struct {
|
||||||
Output CheckpointArtifact
|
Output CheckpointArtifact
|
||||||
Warnings []contracts.Warning
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type CheckpointLoader interface {
|
type CheckpointLoader interface {
|
||||||
@@ -326,14 +332,14 @@ func (noopCheckpointRecorder) SourceFailed(string, error) error
|
|||||||
func (noopCheckpointRecorder) ExtractRunning(string, string, []CheckpointFingerprint) error {
|
func (noopCheckpointRecorder) ExtractRunning(string, string, []CheckpointFingerprint) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) ExtractSucceeded(string, string, []CheckpointFingerprint, []CheckpointArtifact, []contracts.RejectedOutput, []contracts.Warning) error {
|
func (noopCheckpointRecorder) ExtractSucceeded(string, string, []CheckpointFingerprint, []CheckpointArtifact, []contracts.RejectedOutput) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) ExtractFailed(string, string, []CheckpointFingerprint, error) error {
|
func (noopCheckpointRecorder) ExtractFailed(string, string, []CheckpointFingerprint, error) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) MergeRunning(string, string, []CheckpointFingerprint) error { return nil }
|
func (noopCheckpointRecorder) MergeRunning(string, string, []CheckpointFingerprint) error { return nil }
|
||||||
func (noopCheckpointRecorder) MergeSucceeded(string, string, []CheckpointFingerprint, CheckpointArtifact, []contracts.Warning) error {
|
func (noopCheckpointRecorder) MergeSucceeded(string, string, []CheckpointFingerprint, CheckpointArtifact) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) MergeRejected(string, string, []CheckpointFingerprint, contracts.RejectedOutput) error {
|
func (noopCheckpointRecorder) MergeRejected(string, string, []CheckpointFingerprint, contracts.RejectedOutput) error {
|
||||||
@@ -345,7 +351,7 @@ func (noopCheckpointRecorder) MergeFailed(string, string, []CheckpointFingerprin
|
|||||||
func (noopCheckpointRecorder) NormalizeRunning(string, string, []CheckpointFingerprint) error {
|
func (noopCheckpointRecorder) NormalizeRunning(string, string, []CheckpointFingerprint) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) NormalizeSucceeded(string, string, []CheckpointFingerprint, CheckpointArtifact, []contracts.Warning) error {
|
func (noopCheckpointRecorder) NormalizeSucceeded(string, string, []CheckpointFingerprint, CheckpointArtifact) error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
func (noopCheckpointRecorder) NormalizeRejected(string, string, []CheckpointFingerprint, contracts.RejectedOutput) error {
|
func (noopCheckpointRecorder) NormalizeRejected(string, string, []CheckpointFingerprint, contracts.RejectedOutput) error {
|
||||||
@@ -378,11 +384,11 @@ func checkpointExtractRunning(recorder CheckpointRecorder, stepID, laneID, modul
|
|||||||
}
|
}
|
||||||
return recorder.ExtractRunning(laneID, moduleKey, deps)
|
return recorder.ExtractRunning(laneID, moduleKey, deps)
|
||||||
}
|
}
|
||||||
func checkpointExtractSucceeded(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
func checkpointExtractSucceeded(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput) error {
|
||||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||||
return stepAware.ExtractSucceededForStep(stepID, laneID, moduleKey, deps, outputs, rejected, warnings)
|
return stepAware.ExtractSucceededForStep(stepID, laneID, moduleKey, deps, outputs, rejected)
|
||||||
}
|
}
|
||||||
return recorder.ExtractSucceeded(laneID, moduleKey, deps, outputs, rejected, warnings)
|
return recorder.ExtractSucceeded(laneID, moduleKey, deps, outputs, rejected)
|
||||||
}
|
}
|
||||||
func checkpointExtractFailed(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, err error) error {
|
func checkpointExtractFailed(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, err error) error {
|
||||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||||
@@ -396,11 +402,11 @@ func checkpointMergeRunning(recorder CheckpointRecorder, stepID, laneID, moduleK
|
|||||||
}
|
}
|
||||||
return recorder.MergeRunning(laneID, moduleKey, deps)
|
return recorder.MergeRunning(laneID, moduleKey, deps)
|
||||||
}
|
}
|
||||||
func checkpointMergeSucceeded(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
func checkpointMergeSucceeded(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact) error {
|
||||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||||
return stepAware.MergeSucceededForStep(stepID, laneID, moduleKey, deps, output, warnings)
|
return stepAware.MergeSucceededForStep(stepID, laneID, moduleKey, deps, output)
|
||||||
}
|
}
|
||||||
return recorder.MergeSucceeded(laneID, moduleKey, deps, output, warnings)
|
return recorder.MergeSucceeded(laneID, moduleKey, deps, output)
|
||||||
}
|
}
|
||||||
func checkpointMergeRejected(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
func checkpointMergeRejected(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||||
@@ -420,11 +426,11 @@ func checkpointNormalizeRunning(recorder CheckpointRecorder, stepID, laneID, mod
|
|||||||
}
|
}
|
||||||
return recorder.NormalizeRunning(laneID, moduleKey, deps)
|
return recorder.NormalizeRunning(laneID, moduleKey, deps)
|
||||||
}
|
}
|
||||||
func checkpointNormalizeSucceeded(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
func checkpointNormalizeSucceeded(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact) error {
|
||||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||||
return stepAware.NormalizeSucceededForStep(stepID, laneID, moduleKey, deps, output, warnings)
|
return stepAware.NormalizeSucceededForStep(stepID, laneID, moduleKey, deps, output)
|
||||||
}
|
}
|
||||||
return recorder.NormalizeSucceeded(laneID, moduleKey, deps, output, warnings)
|
return recorder.NormalizeSucceeded(laneID, moduleKey, deps, output)
|
||||||
}
|
}
|
||||||
func checkpointNormalizeRejected(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
func checkpointNormalizeRejected(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
const ChunkPlanSchemaVersion = "notarius.chunk-plan.v2"
|
const ChunkPlanSchemaVersion = "notarius.chunk-plan.v3"
|
||||||
|
|
||||||
type ChunkPlanProducer struct {
|
type ChunkPlanProducer struct {
|
||||||
InputModule string `json:"input_module"`
|
InputModule string `json:"input_module"`
|
||||||
@@ -19,13 +19,13 @@ type ChunkPlanProducer struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type ChunkPlanRecord struct {
|
type ChunkPlanRecord struct {
|
||||||
SchemaVersion string `json:"schema_version"`
|
SchemaVersion string `json:"schema_version"`
|
||||||
SourceDigest string `json:"source_digest"`
|
SourceDigest string `json:"source_digest"`
|
||||||
PlanDigest string `json:"plan_digest"`
|
PlanDigest string `json:"plan_digest"`
|
||||||
Plan source.ChunkPlan `json:"plan"`
|
Plan source.ChunkPlan `json:"plan"`
|
||||||
Producer ChunkPlanProducer `json:"producer"`
|
Producer ChunkPlanProducer `json:"producer"`
|
||||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
Diagnostics []contracts.ProducerDiagnostic `json:"diagnostics,omitempty"`
|
||||||
CreatedAt time.Time `json:"created_at"`
|
CreatedAt time.Time `json:"created_at"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type ChunkPlanStore interface {
|
type ChunkPlanStore interface {
|
||||||
|
|||||||
@@ -52,11 +52,10 @@ type debugTimedEnvelope struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type debugBinaryEnvelope struct {
|
type debugBinaryEnvelope struct {
|
||||||
ContentBase64 string `json:"content_base64,omitempty"`
|
ContentBase64 string `json:"content_base64,omitempty"`
|
||||||
ContentDigest string `json:"content_digest,omitempty"`
|
ContentDigest string `json:"content_digest,omitempty"`
|
||||||
MediaType string `json:"media_type,omitempty"`
|
MediaType string `json:"media_type,omitempty"`
|
||||||
Metadata map[string]any `json:"metadata,omitempty"`
|
Metadata map[string]any `json:"metadata,omitempty"`
|
||||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type debugSourceInput struct {
|
type debugSourceInput struct {
|
||||||
@@ -483,14 +482,13 @@ func writeProducerTerminalDebug(recorder DebugRecorder, name string, terminal pr
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func debugContentEnvelope(content []byte, mediaType string, metadata map[string]any, warnings []contracts.Warning) debugBinaryEnvelope {
|
func debugContentEnvelope(content []byte, mediaType string, metadata map[string]any, _ any) debugBinaryEnvelope {
|
||||||
content = redactSecretBytes(content)
|
content = redactSecretBytes(content)
|
||||||
return debugBinaryEnvelope{
|
return debugBinaryEnvelope{
|
||||||
ContentBase64: base64.StdEncoding.EncodeToString(content),
|
ContentBase64: base64.StdEncoding.EncodeToString(content),
|
||||||
ContentDigest: debugContentDigest(content),
|
ContentDigest: debugContentDigest(content),
|
||||||
MediaType: mediaType,
|
MediaType: mediaType,
|
||||||
Metadata: redactSensitiveMap(metadata),
|
Metadata: redactSensitiveMap(metadata),
|
||||||
Warnings: cloneWarnings(warnings),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -731,20 +729,9 @@ func debugValidationResultEnvelope(result contracts.ValidationResult) contracts.
|
|||||||
result.Message = string(redactSecretBytes([]byte(result.Message)))
|
result.Message = string(redactSecretBytes([]byte(result.Message)))
|
||||||
result.CorrectionGuidance = ""
|
result.CorrectionGuidance = ""
|
||||||
result.DiagnosticArtifactPath = string(redactSecretBytes([]byte(result.DiagnosticArtifactPath)))
|
result.DiagnosticArtifactPath = string(redactSecretBytes([]byte(result.DiagnosticArtifactPath)))
|
||||||
for i := range result.Warnings {
|
|
||||||
result.Warnings[i].Message = string(redactSecretBytes([]byte(result.Warnings[i].Message)))
|
|
||||||
}
|
|
||||||
return result
|
return result
|
||||||
}
|
}
|
||||||
|
|
||||||
func debugWarningEnvelopes(warnings []contracts.Warning) []contracts.Warning {
|
|
||||||
out := cloneWarnings(warnings)
|
|
||||||
for i := range out {
|
|
||||||
out[i].Message = string(redactSecretBytes([]byte(out[i].Message)))
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
func debugRejectedOutputEnvelope(rejected contracts.RejectedOutput) contracts.RejectedOutput {
|
func debugRejectedOutputEnvelope(rejected contracts.RejectedOutput) contracts.RejectedOutput {
|
||||||
rejected.Message = string(redactSecretBytes([]byte(rejected.Message)))
|
rejected.Message = string(redactSecretBytes([]byte(rejected.Message)))
|
||||||
rejected.DiagnosticArtifactPath = string(redactSecretBytes([]byte(rejected.DiagnosticArtifactPath)))
|
rejected.DiagnosticArtifactPath = string(redactSecretBytes([]byte(rejected.DiagnosticArtifactPath)))
|
||||||
|
|||||||
124
internal/framework/pipeline/diagnostics.go
Normal file
124
internal/framework/pipeline/diagnostics.go
Normal file
@@ -0,0 +1,124 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
frameworkdiagnostics "gitea.maximumdirect.net/eric/notarius/internal/framework/diagnostics"
|
||||||
|
)
|
||||||
|
|
||||||
|
func terminalDiagnosticGroups(terminal producerAttemptTerminal, origin contracts.DiagnosticOrigin, chunk *source.Chunk) ([]contracts.DiagnosticGroup, error) {
|
||||||
|
return promoteCheckpointDiagnostics(terminalCheckpointDiagnostics(terminal), origin, chunk)
|
||||||
|
}
|
||||||
|
|
||||||
|
func terminalCheckpointDiagnostics(terminal producerAttemptTerminal) []CheckpointDiagnostic {
|
||||||
|
diagnostics := make([]CheckpointDiagnostic, 0, len(terminal.Diagnostics))
|
||||||
|
for _, diagnostic := range terminal.Diagnostics {
|
||||||
|
diagnostics = append(diagnostics, CheckpointDiagnostic{Diagnostic: diagnostic})
|
||||||
|
}
|
||||||
|
for _, record := range terminal.Validation.Diagnostics() {
|
||||||
|
diagnostics = append(diagnostics, CheckpointDiagnostic{Diagnostic: record.diagnostic, ValidatorKey: record.validatorName})
|
||||||
|
}
|
||||||
|
if terminal.Action == producerTerminalIncompleteAccepted {
|
||||||
|
for _, record := range incompleteValidationDiagnostics(terminal.Validation) {
|
||||||
|
diagnostics = append(diagnostics, CheckpointDiagnostic{Diagnostic: record.diagnostic, ValidatorKey: record.validatorName})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return cloneCheckpointDiagnostics(diagnostics)
|
||||||
|
}
|
||||||
|
|
||||||
|
func promoteCheckpointDiagnostics(diagnostics []CheckpointDiagnostic, origin contracts.DiagnosticOrigin, chunk *source.Chunk) ([]contracts.DiagnosticGroup, error) {
|
||||||
|
groups := make([]contracts.DiagnosticGroup, 0, len(diagnostics))
|
||||||
|
for _, record := range diagnostics {
|
||||||
|
validatorOrigin := origin
|
||||||
|
validatorOrigin.ValidatorKey = record.ValidatorKey
|
||||||
|
promoted, err := promoteProducerDiagnostics([]contracts.ProducerDiagnostic{record.Diagnostic}, validatorOrigin, chunk)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("checkpoint diagnostic: %w", err)
|
||||||
|
}
|
||||||
|
groups = append(groups, promoted...)
|
||||||
|
}
|
||||||
|
return groups, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneCheckpointDiagnostics(diagnostics []CheckpointDiagnostic) []CheckpointDiagnostic {
|
||||||
|
if len(diagnostics) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
cloned := make([]CheckpointDiagnostic, len(diagnostics))
|
||||||
|
for index, diagnostic := range diagnostics {
|
||||||
|
cloned[index] = CheckpointDiagnostic{
|
||||||
|
Diagnostic: contracts.CloneProducerDiagnostics([]contracts.ProducerDiagnostic{diagnostic.Diagnostic})[0],
|
||||||
|
ValidatorKey: diagnostic.ValidatorKey,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return cloned
|
||||||
|
}
|
||||||
|
|
||||||
|
func promoteProducerDiagnostics(diagnostics []contracts.ProducerDiagnostic, origin contracts.DiagnosticOrigin, chunk *source.Chunk) ([]contracts.DiagnosticGroup, error) {
|
||||||
|
if err := contracts.ValidateProducerDiagnostics(diagnostics); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if len(diagnostics) == 0 {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
groups := make([]contracts.DiagnosticGroup, len(diagnostics))
|
||||||
|
for index, diagnostic := range diagnostics {
|
||||||
|
group := contracts.DiagnosticGroup{
|
||||||
|
Disposition: diagnostic.Disposition,
|
||||||
|
Category: diagnostic.Category,
|
||||||
|
ReasonCode: diagnostic.ReasonCode,
|
||||||
|
Origin: origin,
|
||||||
|
OccurrenceCount: diagnostic.OccurrenceCount,
|
||||||
|
Samples: cloneDiagnosticSamples(diagnostic.Samples, chunk),
|
||||||
|
OmittedSampleCount: diagnostic.OmittedSampleCount,
|
||||||
|
}
|
||||||
|
if err := group.Validate(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
groups[index] = group
|
||||||
|
}
|
||||||
|
return groups, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneDiagnosticSamples(samples []contracts.DiagnosticSample, chunk *source.Chunk) []contracts.DiagnosticSample {
|
||||||
|
if len(samples) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
cloned := make([]contracts.DiagnosticSample, len(samples))
|
||||||
|
for index, sample := range samples {
|
||||||
|
if chunk != nil {
|
||||||
|
chunkIndex := chunk.Index
|
||||||
|
sample.ChunkID = chunk.ID
|
||||||
|
sample.ChunkIndex = &chunkIndex
|
||||||
|
} else if sample.ChunkIndex != nil {
|
||||||
|
chunkIndex := *sample.ChunkIndex
|
||||||
|
sample.ChunkIndex = &chunkIndex
|
||||||
|
}
|
||||||
|
cloned[index] = sample
|
||||||
|
}
|
||||||
|
return cloned
|
||||||
|
}
|
||||||
|
|
||||||
|
func appendDiagnosticGroups(output *RunOutput, groups []contracts.DiagnosticGroup) {
|
||||||
|
if output == nil || len(groups) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
cloned := contracts.CloneDiagnosticCollection(contracts.DiagnosticCollection{Groups: groups}).Groups
|
||||||
|
output.diagnosticGroups = append(output.diagnosticGroups, cloned...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func finalizeDiagnostics(output *RunOutput) error {
|
||||||
|
if output == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
aggregator := frameworkdiagnostics.Aggregator{}
|
||||||
|
for _, group := range output.diagnosticGroups {
|
||||||
|
if err := aggregator.Add(group); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
output.Diagnostics = aggregator.Collection()
|
||||||
|
return nil
|
||||||
|
}
|
||||||
75
internal/framework/pipeline/diagnostics_test.go
Normal file
75
internal/framework/pipeline/diagnostics_test.go
Normal file
@@ -0,0 +1,75 @@
|
|||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestTerminalDiagnosticGroupsDiscardSupersededAttemptDiagnostics(t *testing.T) {
|
||||||
|
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||||
|
if request.Number == 1 {
|
||||||
|
return producerAttemptOutput{Value: "first", Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("discarded", "discarded")}, Candidate: attemptCandidate(t, "first")}, nil
|
||||||
|
}
|
||||||
|
return producerAttemptOutput{Value: "second", Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("terminal", "terminal")}}, nil
|
||||||
|
}, func(_ context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||||
|
if output.Value == "first" {
|
||||||
|
return validationReport{records: []validationRecord{{validatorName: "validator", outcome: validationRejected, reasonCode: "invalid", correctionGuidance: "Correct it."}}}, nil
|
||||||
|
}
|
||||||
|
return validationReport{}, nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("runProducerAttempts() error = %v", err)
|
||||||
|
}
|
||||||
|
groups, err := terminalDiagnosticGroups(terminal, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageMerge, StepID: "step", LaneID: "lane", ModuleKey: "module"}, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("terminalDiagnosticGroups() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(groups) != 1 || groups[0].Samples[0].Scope != "terminal" {
|
||||||
|
t.Fatalf("groups = %#v, want only terminal attempt diagnostics", groups)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTerminalDiagnosticGroupsPreserveValidatorOrigin(t *testing.T) {
|
||||||
|
terminal := producerAttemptTerminal{Validation: validationReport{records: []validationRecord{{
|
||||||
|
validatorName: "dnd/spells/source-relatedness",
|
||||||
|
outcome: validationApproved,
|
||||||
|
diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("spells[0]", "not found")},
|
||||||
|
}}}}
|
||||||
|
groups, err := terminalDiagnosticGroups(terminal, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageExtract, StepID: "extract", LaneID: "spells", ModuleKey: "dnd/spells"}, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("terminalDiagnosticGroups() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(groups) != 1 || groups[0].Origin.ValidatorKey != "dnd/spells/source-relatedness" {
|
||||||
|
t.Fatalf("groups = %#v, want validator origin", groups)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAppendDiagnosticGroupsRetainsOnlyStructuredDiagnostics(t *testing.T) {
|
||||||
|
output := RunOutput{}
|
||||||
|
appendDiagnosticGroups(&output, []contracts.DiagnosticGroup{{
|
||||||
|
Disposition: contracts.DiagnosticDispositionAdvisory,
|
||||||
|
Category: contracts.DiagnosticCategoryDataQuality,
|
||||||
|
ReasonCode: "source_unrelated",
|
||||||
|
Origin: contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageExtract, StepID: "extract", LaneID: "spells", ModuleKey: "dnd/spells"},
|
||||||
|
OccurrenceCount: 2,
|
||||||
|
Samples: []contracts.DiagnosticSample{{Scope: "spells[0]", Message: "not found"}, {Scope: "spells[1]", Message: "also not found"}},
|
||||||
|
}})
|
||||||
|
if err := finalizeDiagnostics(&output); err != nil {
|
||||||
|
t.Fatalf("finalizeDiagnostics() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(output.Diagnostics.Groups) != 1 || output.Diagnostics.Groups[0].OccurrenceCount != 2 {
|
||||||
|
t.Fatalf("diagnostics = %#v, want grouped collection", output.Diagnostics)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func producerDiagnostic(scope string, message string) contracts.ProducerDiagnostic {
|
||||||
|
return contracts.ProducerDiagnostic{
|
||||||
|
Disposition: contracts.DiagnosticDispositionAdvisory,
|
||||||
|
Category: contracts.DiagnosticCategoryDataQuality,
|
||||||
|
ReasonCode: "source_unrelated",
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []contracts.DiagnosticSample{{Scope: scope, Message: message}},
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -71,7 +71,7 @@ func RegisterExtractorBuilder[T any](registry *ExtractorRegistry, spec ModuleSpe
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return erasedTypedResult{}, fmt.Errorf("clone extraction model candidate: %w", err)
|
return erasedTypedResult{}, fmt.Errorf("clone extraction model candidate: %w", err)
|
||||||
}
|
}
|
||||||
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), ModelCandidate: candidate}, nil
|
return erasedTypedResult{Value: result.Value, Diagnostics: contracts.CloneProducerDiagnostics(result.Diagnostics), ModelCandidate: candidate}, nil
|
||||||
}}
|
}}
|
||||||
if registry.typedEntries == nil {
|
if registry.typedEntries == nil {
|
||||||
registry.typedEntries = map[string]typedExtractorEntry{}
|
registry.typedEntries = map[string]typedExtractorEntry{}
|
||||||
|
|||||||
@@ -97,7 +97,7 @@ func RegisterMergerBuilder[T any](registry *MergerRegistry, spec ModuleSpec, val
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return erasedTypedResult{}, fmt.Errorf("clone merge model candidate: %w", err)
|
return erasedTypedResult{}, fmt.Errorf("clone merge model candidate: %w", err)
|
||||||
}
|
}
|
||||||
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), ModelCandidate: candidate}, nil
|
return erasedTypedResult{Value: result.Value, Diagnostics: contracts.CloneProducerDiagnostics(result.Diagnostics), ModelCandidate: candidate}, nil
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -88,7 +88,7 @@ func RegisterNormalizerBuilder[T any](registry *NormalizerRegistry, spec ModuleS
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return erasedTypedResult{}, fmt.Errorf("clone normalize model candidate: %w", err)
|
return erasedTypedResult{}, fmt.Errorf("clone normalize model candidate: %w", err)
|
||||||
}
|
}
|
||||||
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), Retry: cloneNormalizeRetry(result.Retry), ModelCandidate: candidate}, nil
|
return erasedTypedResult{Value: result.Value, Diagnostics: contracts.CloneProducerDiagnostics(result.Diagnostics), Retry: cloneNormalizeRetry(result.Retry), ModelCandidate: candidate}, nil
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
@@ -99,9 +99,9 @@ func cloneNormalizeRetry(retry *contracts.NormalizeRetry) *contracts.NormalizeRe
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return &contracts.NormalizeRetry{
|
return &contracts.NormalizeRetry{
|
||||||
ReasonCode: retry.ReasonCode,
|
ReasonCode: retry.ReasonCode,
|
||||||
Message: retry.Message,
|
Message: retry.Message,
|
||||||
FallbackWarnings: cloneWarnings(retry.FallbackWarnings),
|
FallbackDiagnostics: contracts.CloneProducerDiagnostics(retry.FallbackDiagnostics),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -8,26 +8,23 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type retryingNotesNormalizer struct {
|
type retryingNotesNormalizer struct {
|
||||||
warnings []contracts.Warning
|
retry *contracts.NormalizeRetry
|
||||||
retry *contracts.NormalizeRetry
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (retryingNotesNormalizer) Key() string { return "test/retry-normalize" }
|
func (retryingNotesNormalizer) Key() string { return "test/retry-normalize" }
|
||||||
func (retryingNotesNormalizer) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
func (retryingNotesNormalizer) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||||
func (n retryingNotesNormalizer) Normalize(_ context.Context, req contracts.TypedNormalizeRequest[codecNotes]) (contracts.TypedNormalizeResult[codecNotes], error) {
|
func (n retryingNotesNormalizer) Normalize(_ context.Context, req contracts.TypedNormalizeRequest[codecNotes]) (contracts.TypedNormalizeResult[codecNotes], error) {
|
||||||
return contracts.TypedNormalizeResult[codecNotes]{Value: req.MergeOutput.Value, Warnings: n.warnings, Retry: n.retry}, nil
|
return contracts.TypedNormalizeResult[codecNotes]{Value: req.MergeOutput.Value, Retry: n.retry}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
|
func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
|
||||||
warnings := []contracts.Warning{{Scope: "attempt", ReasonCode: "ordinary", Message: "ordinary warning"}}
|
|
||||||
retry := &contracts.NormalizeRetry{
|
retry := &contracts.NormalizeRetry{
|
||||||
ReasonCode: "retryable",
|
ReasonCode: "retryable",
|
||||||
Message: "safe fallback available",
|
Message: "safe fallback available",
|
||||||
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}},
|
|
||||||
}
|
}
|
||||||
registry := NewNormalizerRegistry()
|
registry := NewNormalizerRegistry()
|
||||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||||
return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil
|
return retryingNotesNormalizer{retry: retry}, nil
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -43,10 +40,8 @@ func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("normalize() error = %v", err)
|
t.Fatalf("normalize() error = %v", err)
|
||||||
}
|
}
|
||||||
warnings[0].Message = "mutated"
|
|
||||||
retry.Message = "mutated"
|
retry.Message = "mutated"
|
||||||
retry.FallbackWarnings[0].Message = "mutated"
|
if result.Retry == nil || result.Retry.Message != "safe fallback available" {
|
||||||
if result.Retry == nil || result.Warnings[0].Message != "ordinary warning" || result.Retry.Message != "safe fallback available" || result.Retry.FallbackWarnings[0].Message != "fallback warning" {
|
t.Fatalf("erased retry result = %#v, want independent retry data", result)
|
||||||
t.Fatalf("erased retry result = %#v, want independent warning data", result)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -52,24 +52,24 @@ type producerAttemptRequest struct {
|
|||||||
// the current value as a safe fallback if its shared budget is exhausted.
|
// the current value as a safe fallback if its shared budget is exhausted.
|
||||||
// Artifact-specific adapters are responsible for validating and populating it.
|
// Artifact-specific adapters are responsible for validating and populating it.
|
||||||
type producerRetryDirective struct {
|
type producerRetryDirective struct {
|
||||||
FallbackWarnings []contracts.Warning
|
FallbackDiagnostics []contracts.ProducerDiagnostic
|
||||||
}
|
}
|
||||||
|
|
||||||
func (directive *producerRetryDirective) clone() *producerRetryDirective {
|
func (directive *producerRetryDirective) clone() *producerRetryDirective {
|
||||||
if directive == nil {
|
if directive == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return &producerRetryDirective{FallbackWarnings: cloneWarnings(directive.FallbackWarnings)}
|
return &producerRetryDirective{FallbackDiagnostics: contracts.CloneProducerDiagnostics(directive.FallbackDiagnostics)}
|
||||||
}
|
}
|
||||||
|
|
||||||
// producerAttemptOutput is intentionally artifact-neutral. Value remains
|
// producerAttemptOutput is intentionally artifact-neutral. Value remains
|
||||||
// opaque to the state machine; Candidate is the attempt-local response that
|
// opaque to the state machine; Candidate is the attempt-local response that
|
||||||
// may support semantic correction.
|
// may support semantic correction.
|
||||||
type producerAttemptOutput struct {
|
type producerAttemptOutput struct {
|
||||||
Value any
|
Value any
|
||||||
Candidate *contracts.ModelCandidate
|
Candidate *contracts.ModelCandidate
|
||||||
Warnings []contracts.Warning
|
Diagnostics []contracts.ProducerDiagnostic
|
||||||
Retry *producerRetryDirective
|
Retry *producerRetryDirective
|
||||||
}
|
}
|
||||||
|
|
||||||
func (output producerAttemptOutput) clone() (producerAttemptOutput, error) {
|
func (output producerAttemptOutput) clone() (producerAttemptOutput, error) {
|
||||||
@@ -78,10 +78,10 @@ func (output producerAttemptOutput) clone() (producerAttemptOutput, error) {
|
|||||||
return producerAttemptOutput{}, fmt.Errorf("clone model candidate: %w", err)
|
return producerAttemptOutput{}, fmt.Errorf("clone model candidate: %w", err)
|
||||||
}
|
}
|
||||||
return producerAttemptOutput{
|
return producerAttemptOutput{
|
||||||
Value: output.Value,
|
Value: output.Value,
|
||||||
Candidate: candidate,
|
Candidate: candidate,
|
||||||
Warnings: cloneWarnings(output.Warnings),
|
Diagnostics: contracts.CloneProducerDiagnostics(output.Diagnostics),
|
||||||
Retry: output.Retry.clone(),
|
Retry: output.Retry.clone(),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -113,7 +113,7 @@ func (provenance producerAttemptProvenance) clone() producerAttemptProvenance {
|
|||||||
type producerAttemptTerminal struct {
|
type producerAttemptTerminal struct {
|
||||||
Action producerTerminalAction
|
Action producerTerminalAction
|
||||||
Value any
|
Value any
|
||||||
Warnings []contracts.Warning
|
Diagnostics []contracts.ProducerDiagnostic
|
||||||
Rejection *contracts.RejectedOutput
|
Rejection *contracts.RejectedOutput
|
||||||
Validation validationReport
|
Validation validationReport
|
||||||
ValidationIncomplete bool
|
ValidationIncomplete bool
|
||||||
@@ -121,7 +121,7 @@ type producerAttemptTerminal struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (terminal producerAttemptTerminal) clone() producerAttemptTerminal {
|
func (terminal producerAttemptTerminal) clone() producerAttemptTerminal {
|
||||||
terminal.Warnings = cloneWarnings(terminal.Warnings)
|
terminal.Diagnostics = contracts.CloneProducerDiagnostics(terminal.Diagnostics)
|
||||||
if terminal.Rejection != nil {
|
if terminal.Rejection != nil {
|
||||||
rejection := *terminal.Rejection
|
rejection := *terminal.Rejection
|
||||||
rejection.Validation = cloneValidationSummaryPtr(rejection.Validation)
|
rejection.Validation = cloneValidationSummaryPtr(rejection.Validation)
|
||||||
@@ -193,6 +193,10 @@ func runProducerAttempts(ctx context.Context, config producerAttemptConfig, prod
|
|||||||
}
|
}
|
||||||
return failedProducerAttempt(provenance), fmt.Errorf("producer failed after %d attempt(s): %w", number, err)
|
return failedProducerAttempt(provenance), fmt.Errorf("producer failed after %d attempt(s): %w", number, err)
|
||||||
}
|
}
|
||||||
|
if err := validateProducerAttemptDiagnostics(output); err != nil {
|
||||||
|
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptFailed})
|
||||||
|
return failedProducerAttempt(provenance), err
|
||||||
|
}
|
||||||
|
|
||||||
output, err = output.clone()
|
output, err = output.clone()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -205,7 +209,7 @@ func runProducerAttempts(ctx context.Context, config producerAttemptConfig, prod
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if output.Retry != nil {
|
if output.Retry != nil {
|
||||||
output.Warnings = append(output.Warnings, cloneWarnings(output.Retry.FallbackWarnings)...)
|
output.Diagnostics = append(output.Diagnostics, contracts.CloneProducerDiagnostics(output.Retry.FallbackDiagnostics)...)
|
||||||
}
|
}
|
||||||
correctionCandidate, err := contracts.CloneModelCandidate(output.Candidate)
|
correctionCandidate, err := contracts.CloneModelCandidate(output.Candidate)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -253,20 +257,30 @@ func runProducerAttempts(ctx context.Context, config producerAttemptConfig, prod
|
|||||||
if incomplete := firstIncompleteValidation(report); incomplete != nil {
|
if incomplete := firstIncompleteValidation(report); incomplete != nil {
|
||||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptIncompleteAccepted, Validation: report})
|
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptIncompleteAccepted, Validation: report})
|
||||||
if config.Policy.ValidatorFailure == ValidatorFailureWarnContinue {
|
if config.Policy.ValidatorFailure == ValidatorFailureWarnContinue {
|
||||||
warnings := terminalWarnings(output, report)
|
return producerAttemptTerminal{Action: producerTerminalIncompleteAccepted, Value: output.Value, Diagnostics: cloneProducerDiagnostics(output.Diagnostics), Validation: report, ValidationIncomplete: true, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
|
||||||
warnings = append(warnings, incompleteValidationWarnings(report)...)
|
|
||||||
return producerAttemptTerminal{Action: producerTerminalIncompleteAccepted, Value: output.Value, Warnings: warnings, Validation: report, ValidationIncomplete: true, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
|
|
||||||
}
|
}
|
||||||
return failedProducerAttempt(provenance), validatorFailureError(*incomplete)
|
return failedProducerAttempt(provenance), validatorFailureError(*incomplete)
|
||||||
}
|
}
|
||||||
|
|
||||||
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptAccepted, Validation: report})
|
provenance = append(provenance, producerAttemptProvenance{Number: number, Kind: kind, Outcome: producerAttemptAccepted, Validation: report})
|
||||||
return producerAttemptTerminal{Action: producerTerminalAccepted, Value: output.Value, Warnings: terminalWarnings(output, report), Validation: report, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
|
return producerAttemptTerminal{Action: producerTerminalAccepted, Value: output.Value, Diagnostics: cloneProducerDiagnostics(output.Diagnostics), Validation: report, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return failedProducerAttempt(provenance), errors.New("producer attempt budget was not exhausted deterministically")
|
return failedProducerAttempt(provenance), errors.New("producer attempt budget was not exhausted deterministically")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateProducerAttemptDiagnostics(output producerAttemptOutput) error {
|
||||||
|
if err := contracts.ValidateProducerDiagnostics(output.Diagnostics); err != nil {
|
||||||
|
return fmt.Errorf("producer returned invalid diagnostics: %w", err)
|
||||||
|
}
|
||||||
|
if output.Retry != nil {
|
||||||
|
if err := contracts.ValidateProducerDiagnostics(output.Retry.FallbackDiagnostics); err != nil {
|
||||||
|
return fmt.Errorf("producer returned invalid retry fallback diagnostics: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func isImmediateProducerFailure(err error) bool {
|
func isImmediateProducerFailure(err error) bool {
|
||||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||||
return true
|
return true
|
||||||
@@ -290,7 +304,7 @@ func applySemanticTerminalPolicy(policy ValidationPolicy, provenance []producerA
|
|||||||
rejected := contracts.RejectedOutput{ValidatorName: rejection.validatorName, ReasonCode: rejection.reasonCode, Message: rejection.message, AttemptCount: number, DiagnosticArtifactPath: rejection.diagnosticPath}
|
rejected := contracts.RejectedOutput{ValidatorName: rejection.validatorName, ReasonCode: rejection.reasonCode, Message: rejection.message, AttemptCount: number, DiagnosticArtifactPath: rejection.diagnosticPath}
|
||||||
switch policy.SemanticRejection {
|
switch policy.SemanticRejection {
|
||||||
case SemanticRejectionRejectOutput:
|
case SemanticRejectionRejectOutput:
|
||||||
return producerAttemptTerminal{Action: producerTerminalRejected, Warnings: terminalWarnings(output, report), Rejection: &rejected, Validation: report, ValidationIncomplete: firstIncompleteValidation(report) != nil, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
|
return producerAttemptTerminal{Action: producerTerminalRejected, Diagnostics: cloneProducerDiagnostics(output.Diagnostics), Rejection: &rejected, Validation: report, ValidationIncomplete: firstIncompleteValidation(report) != nil, Provenance: cloneProducerAttemptProvenance(provenance)}, nil
|
||||||
case SemanticRejectionFailRun:
|
case SemanticRejectionFailRun:
|
||||||
return failedProducerAttempt(provenance), fmt.Errorf("producer candidate rejected after %d attempt(s): %s", number, rejection.message)
|
return failedProducerAttempt(provenance), fmt.Errorf("producer candidate rejected after %d attempt(s): %s", number, rejection.message)
|
||||||
default:
|
default:
|
||||||
@@ -308,28 +322,31 @@ func firstIncompleteValidation(report validationReport) *validationRecord {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func terminalWarnings(output producerAttemptOutput, report validationReport) []contracts.Warning {
|
func cloneProducerDiagnostics(diagnostics []contracts.ProducerDiagnostic) []contracts.ProducerDiagnostic {
|
||||||
warnings := cloneWarnings(output.Warnings)
|
return contracts.CloneProducerDiagnostics(diagnostics)
|
||||||
warnings = append(warnings, report.Warnings()...)
|
|
||||||
return warnings
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// incompleteValidationWarnings reports only validators that exhausted their
|
// incompleteValidationDiagnostics reports every applicable validator that
|
||||||
// execution budget. It never reports rejected candidates, and it uses fixed
|
// could not complete under warn_continue. It uses fixed text so provider
|
||||||
// text so provider errors and correction content cannot cross this boundary.
|
// errors and arbitrary validator prose cannot cross this boundary.
|
||||||
func incompleteValidationWarnings(report validationReport) []contracts.Warning {
|
func incompleteValidationDiagnostics(report validationReport) []validationDiagnosticRecord {
|
||||||
warnings := make([]contracts.Warning, 0)
|
diagnostics := make([]validationDiagnosticRecord, 0)
|
||||||
for _, record := range report.records {
|
for _, record := range report.records {
|
||||||
if record.outcome != validationFailed {
|
if record.outcome != validationFailed && record.outcome != validationSkipped {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
warnings = append(warnings, contracts.Warning{
|
diagnostics = append(diagnostics, validationDiagnosticRecord{validatorName: record.validatorName, diagnostic: contracts.ProducerDiagnostic{
|
||||||
Scope: record.validatorName,
|
Disposition: contracts.DiagnosticDispositionWarning,
|
||||||
ReasonCode: "validator_execution_incomplete",
|
Category: contracts.DiagnosticCategoryValidationIncomplete,
|
||||||
Message: "Validator execution did not complete within its configured budget.",
|
ReasonCode: "validator_execution_incomplete",
|
||||||
})
|
OccurrenceCount: 1,
|
||||||
|
Samples: []contracts.DiagnosticSample{{
|
||||||
|
Scope: record.validatorName,
|
||||||
|
Message: "Validator execution did not complete within its configured budget.",
|
||||||
|
}},
|
||||||
|
}})
|
||||||
}
|
}
|
||||||
return warnings
|
return diagnostics
|
||||||
}
|
}
|
||||||
|
|
||||||
// validationSummary projects a terminal state-machine result into the durable
|
// validationSummary projects a terminal state-machine result into the durable
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"reflect"
|
"reflect"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
@@ -234,27 +235,66 @@ func TestRunProducerAttemptsUsesModuleRetryBudgetAndFallback(t *testing.T) {
|
|||||||
})
|
})
|
||||||
|
|
||||||
t.Run("fallback", func(t *testing.T) {
|
t.Run("fallback", func(t *testing.T) {
|
||||||
fallbackWarning := contracts.Warning{ReasonCode: "fallback", Message: "fallback warning"}
|
fallbackDiagnostic := contracts.ProducerDiagnostic{Disposition: contracts.DiagnosticDispositionWarning, Category: contracts.DiagnosticCategoryFallback, ReasonCode: "fallback", OccurrenceCount: 1, Samples: []contracts.DiagnosticSample{{Scope: "fallback", Message: "fallback warning"}}}
|
||||||
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Policy: DefaultValidationPolicy()}, func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Policy: DefaultValidationPolicy()}, func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||||
return producerAttemptOutput{Value: "fallback", Retry: &producerRetryDirective{FallbackWarnings: []contracts.Warning{fallbackWarning}}}, nil
|
return producerAttemptOutput{Value: "fallback", Retry: &producerRetryDirective{FallbackDiagnostics: []contracts.ProducerDiagnostic{fallbackDiagnostic}}}, nil
|
||||||
}, approveAttempt)
|
}, approveAttempt)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("runProducerAttempts() error = %v", err)
|
t.Fatalf("runProducerAttempts() error = %v", err)
|
||||||
}
|
}
|
||||||
if terminal.Action != producerTerminalAccepted || terminal.Value != "fallback" || !reflect.DeepEqual(terminal.Warnings, []contracts.Warning{fallbackWarning}) {
|
if terminal.Action != producerTerminalAccepted || terminal.Value != "fallback" || !reflect.DeepEqual(terminal.Diagnostics, []contracts.ProducerDiagnostic{fallbackDiagnostic}) {
|
||||||
t.Fatalf("terminal = %#v", terminal)
|
t.Fatalf("terminal = %#v", terminal)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunProducerAttemptsRejectsInvalidDiagnosticsWithoutRetry(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
output producerAttemptOutput
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "producer diagnostics",
|
||||||
|
output: producerAttemptOutput{Diagnostics: []contracts.ProducerDiagnostic{{}}},
|
||||||
|
want: "producer returned invalid diagnostics",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "retry fallback diagnostics",
|
||||||
|
output: producerAttemptOutput{Retry: &producerRetryDirective{FallbackDiagnostics: []contracts.ProducerDiagnostic{{}}}},
|
||||||
|
want: "producer returned invalid retry fallback diagnostics",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
producerCalls := 0
|
||||||
|
validatorCalls := 0
|
||||||
|
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 3, Policy: DefaultValidationPolicy()}, func(context.Context, producerAttemptRequest) (producerAttemptOutput, error) {
|
||||||
|
producerCalls++
|
||||||
|
return test.output, nil
|
||||||
|
}, func(context.Context, producerAttemptOutput) (validationReport, error) {
|
||||||
|
validatorCalls++
|
||||||
|
return validationReport{}, nil
|
||||||
|
})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("runProducerAttempts() error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
if terminal.Action != producerTerminalFailed || producerCalls != 1 || validatorCalls != 0 {
|
||||||
|
t.Fatalf("terminal = %#v, producer calls = %d, validator calls = %d; want immediate framework failure", terminal, producerCalls, validatorCalls)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunProducerAttemptsRejectionWinsOverValidatorFailure(t *testing.T) {
|
func TestRunProducerAttemptsRejectionWinsOverValidatorFailure(t *testing.T) {
|
||||||
firstWarnings := []contracts.Warning{{ReasonCode: "discarded", Message: "discarded warning"}}
|
firstDiagnostics := []contracts.ProducerDiagnostic{producerDiagnostic("discarded", "discarded warning")}
|
||||||
secondWarnings := []contracts.Warning{{ReasonCode: "accepted", Message: "accepted warning"}}
|
secondDiagnostics := []contracts.ProducerDiagnostic{producerDiagnostic("accepted", "accepted warning")}
|
||||||
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
terminal, err := runProducerAttempts(context.Background(), producerAttemptConfig{Retries: 1, Policy: DefaultValidationPolicy()}, func(_ context.Context, request producerAttemptRequest) (producerAttemptOutput, error) {
|
||||||
if request.Number == 1 {
|
if request.Number == 1 {
|
||||||
return producerAttemptOutput{Value: "first", Candidate: attemptCandidate(t, "defective"), Warnings: firstWarnings}, nil
|
return producerAttemptOutput{Value: "first", Candidate: attemptCandidate(t, "defective"), Diagnostics: firstDiagnostics}, nil
|
||||||
}
|
}
|
||||||
return producerAttemptOutput{Value: "second", Candidate: attemptCandidate(t, "corrected"), Warnings: secondWarnings}, nil
|
return producerAttemptOutput{Value: "second", Candidate: attemptCandidate(t, "corrected"), Diagnostics: secondDiagnostics}, nil
|
||||||
}, func(_ context.Context, output producerAttemptOutput) (validationReport, error) {
|
}, func(_ context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||||
if output.Value == "first" {
|
if output.Value == "first" {
|
||||||
report := rejectedAttemptReport("defect")
|
report := rejectedAttemptReport("defect")
|
||||||
@@ -272,8 +312,8 @@ func TestRunProducerAttemptsRejectionWinsOverValidatorFailure(t *testing.T) {
|
|||||||
if got := attemptKinds(terminal.Provenance); !reflect.DeepEqual(got, []producerAttemptKind{producerAttemptInitial, producerAttemptSemanticRetry}) {
|
if got := attemptKinds(terminal.Provenance); !reflect.DeepEqual(got, []producerAttemptKind{producerAttemptInitial, producerAttemptSemanticRetry}) {
|
||||||
t.Fatalf("attempt kinds = %v", got)
|
t.Fatalf("attempt kinds = %v", got)
|
||||||
}
|
}
|
||||||
if got := terminal.Warnings; !reflect.DeepEqual(got, secondWarnings) {
|
if got := terminal.Diagnostics; !reflect.DeepEqual(got, secondDiagnostics) {
|
||||||
t.Fatalf("terminal warnings = %#v, want %#v", got, secondWarnings)
|
t.Fatalf("terminal diagnostics = %#v, want %#v", got, secondDiagnostics)
|
||||||
}
|
}
|
||||||
if len(terminal.Provenance[0].Validation.records) != 2 {
|
if len(terminal.Provenance[0].Validation.records) != 2 {
|
||||||
t.Fatalf("first validation records = %#v, want rejection and failure", terminal.Provenance[0].Validation.records)
|
t.Fatalf("first validation records = %#v, want rejection and failure", terminal.Provenance[0].Validation.records)
|
||||||
@@ -302,9 +342,6 @@ func TestValidationSummaryIsBoundedAndCorrectedSuccessIsQuiet(t *testing.T) {
|
|||||||
if len(summary.ReasonCodes) != 0 || len(summary.RejectingValidators) != 0 || len(summary.IncompleteValidators) != 0 {
|
if len(summary.ReasonCodes) != 0 || len(summary.RejectingValidators) != 0 || len(summary.IncompleteValidators) != 0 {
|
||||||
t.Fatalf("corrected success summary retained prior findings: %#v", summary)
|
t.Fatalf("corrected success summary retained prior findings: %#v", summary)
|
||||||
}
|
}
|
||||||
if len(terminal.Warnings) != 0 {
|
|
||||||
t.Fatalf("corrected success warnings = %#v, want none", terminal.Warnings)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestWarnContinueRecordsOneWarningForEachExhaustedValidator(t *testing.T) {
|
func TestWarnContinueRecordsOneWarningForEachExhaustedValidator(t *testing.T) {
|
||||||
@@ -324,11 +361,17 @@ func TestWarnContinueRecordsOneWarningForEachExhaustedValidator(t *testing.T) {
|
|||||||
if terminal.Action != producerTerminalIncompleteAccepted {
|
if terminal.Action != producerTerminalIncompleteAccepted {
|
||||||
t.Fatalf("terminal action = %q", terminal.Action)
|
t.Fatalf("terminal action = %q", terminal.Action)
|
||||||
}
|
}
|
||||||
if got, want := terminal.Warnings, []contracts.Warning{
|
groups, groupErr := terminalDiagnosticGroups(terminal, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageNormalize, StepID: "step", LaneID: "lane", ModuleKey: "module"}, nil)
|
||||||
{Scope: "first", ReasonCode: "validator_execution_incomplete", Message: "Validator execution did not complete within its configured budget."},
|
if groupErr != nil {
|
||||||
{Scope: "third", ReasonCode: "validator_execution_incomplete", Message: "Validator execution did not complete within its configured budget."},
|
t.Fatalf("terminalDiagnosticGroups() error = %v", groupErr)
|
||||||
}; !reflect.DeepEqual(got, want) {
|
}
|
||||||
t.Fatalf("warnings = %#v, want %#v", got, want)
|
if len(groups) != 3 {
|
||||||
|
t.Fatalf("diagnostic groups = %#v, want every failed and skipped validator", groups)
|
||||||
|
}
|
||||||
|
for index, validator := range []string{"first", "second", "third"} {
|
||||||
|
if group := groups[index]; group.Origin.ValidatorKey != validator || group.Disposition != contracts.DiagnosticDispositionWarning || group.Category != contracts.DiagnosticCategoryValidationIncomplete || group.ReasonCode != "validator_execution_incomplete" {
|
||||||
|
t.Fatalf("diagnostic group %d = %#v, want incomplete warning for %q", index, group, validator)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
summary := validationSummary(terminal, StageNormalize, "step", "lane", "module", "", 0)
|
summary := validationSummary(terminal, StageNormalize, "step", "lane", "module", "", 0)
|
||||||
if summary.Status != "incomplete" || !reflect.DeepEqual(summary.IncompleteValidators, []string{"first", "second", "third"}) || !reflect.DeepEqual(summary.ReasonCodes, []string{"missing_prerequisite"}) {
|
if summary.Status != "incomplete" || !reflect.DeepEqual(summary.IncompleteValidators, []string{"first", "second", "third"}) || !reflect.DeepEqual(summary.ReasonCodes, []string{"missing_prerequisite"}) {
|
||||||
|
|||||||
@@ -30,45 +30,45 @@ type ReferenceMaterializationOptions struct {
|
|||||||
WorkingDir string
|
WorkingDir string
|
||||||
}
|
}
|
||||||
|
|
||||||
func MaterializeReferences(resolved ResolvedPipeline, catalog ModuleCatalog, options ReferenceMaterializationOptions) (ResolvedPipeline, []contracts.Warning, error) {
|
func MaterializeReferences(resolved ResolvedPipeline, catalog ModuleCatalog, options ReferenceMaterializationOptions) (ResolvedPipeline, []contracts.ProducerDiagnostic, error) {
|
||||||
out := resolved
|
out := resolved
|
||||||
out.ChunkReferences = CloneReferenceTarget(resolved.ChunkReferences)
|
out.ChunkReferences = CloneReferenceTarget(resolved.ChunkReferences)
|
||||||
chunkReferenceSet, chunkWarnings, err := materializeReferenceTarget(resolved.ID, resolved.ChunkReferences, "", catalog, options)
|
chunkReferenceSet, chunkDiagnostics, err := materializeReferenceTarget(resolved.ID, resolved.ChunkReferences, "", catalog, options)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedPipeline{}, nil, err
|
return ResolvedPipeline{}, nil, err
|
||||||
}
|
}
|
||||||
out.ChunkReferences.ReferenceSet = chunkReferenceSet
|
out.ChunkReferences.ReferenceSet = chunkReferenceSet
|
||||||
warnings := append([]contracts.Warning(nil), chunkWarnings...)
|
diagnostics := contracts.CloneProducerDiagnostics(chunkDiagnostics)
|
||||||
if len(resolved.Steps) == 0 {
|
if len(resolved.Steps) == 0 {
|
||||||
return out, warnings, nil
|
return out, diagnostics, nil
|
||||||
}
|
}
|
||||||
materializeLane := func(lane ResolvedArtifactLane) (ResolvedArtifactLane, []contracts.Warning, error) {
|
materializeLane := func(lane ResolvedArtifactLane) (ResolvedArtifactLane, []contracts.ProducerDiagnostic, error) {
|
||||||
materializedLane := lane
|
materializedLane := lane
|
||||||
var allWarnings []contracts.Warning
|
var allDiagnostics []contracts.ProducerDiagnostic
|
||||||
materializedLane.ExtractReferences = CloneReferenceTarget(lane.ExtractReferences)
|
materializedLane.ExtractReferences = CloneReferenceTarget(lane.ExtractReferences)
|
||||||
materializedLane.MergeReferences = CloneReferenceTarget(lane.MergeReferences)
|
materializedLane.MergeReferences = CloneReferenceTarget(lane.MergeReferences)
|
||||||
materializedLane.NormalizeReferences = CloneReferenceTarget(lane.NormalizeReferences)
|
materializedLane.NormalizeReferences = CloneReferenceTarget(lane.NormalizeReferences)
|
||||||
extractReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.ExtractReferences, lane.ArtifactKind, catalog, options)
|
extractReferenceSet, laneDiagnostics, err := materializeReferenceTarget(resolved.ID, lane.ExtractReferences, lane.ArtifactKind, catalog, options)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedArtifactLane{}, nil, err
|
return ResolvedArtifactLane{}, nil, err
|
||||||
}
|
}
|
||||||
materializedLane.ExtractReferences.ReferenceSet = extractReferenceSet
|
materializedLane.ExtractReferences.ReferenceSet = extractReferenceSet
|
||||||
allWarnings = append(allWarnings, laneWarnings...)
|
allDiagnostics = append(allDiagnostics, laneDiagnostics...)
|
||||||
|
|
||||||
mergeReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.MergeReferences, lane.ArtifactKind, catalog, options)
|
mergeReferenceSet, laneDiagnostics, err := materializeReferenceTarget(resolved.ID, lane.MergeReferences, lane.ArtifactKind, catalog, options)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedArtifactLane{}, nil, err
|
return ResolvedArtifactLane{}, nil, err
|
||||||
}
|
}
|
||||||
materializedLane.MergeReferences.ReferenceSet = mergeReferenceSet
|
materializedLane.MergeReferences.ReferenceSet = mergeReferenceSet
|
||||||
allWarnings = append(allWarnings, laneWarnings...)
|
allDiagnostics = append(allDiagnostics, laneDiagnostics...)
|
||||||
|
|
||||||
normalizeReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.NormalizeReferences, lane.ArtifactKind, catalog, options)
|
normalizeReferenceSet, laneDiagnostics, err := materializeReferenceTarget(resolved.ID, lane.NormalizeReferences, lane.ArtifactKind, catalog, options)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedArtifactLane{}, nil, err
|
return ResolvedArtifactLane{}, nil, err
|
||||||
}
|
}
|
||||||
materializedLane.NormalizeReferences.ReferenceSet = normalizeReferenceSet
|
materializedLane.NormalizeReferences.ReferenceSet = normalizeReferenceSet
|
||||||
allWarnings = append(allWarnings, laneWarnings...)
|
allDiagnostics = append(allDiagnostics, laneDiagnostics...)
|
||||||
return materializedLane, allWarnings, nil
|
return materializedLane, allDiagnostics, nil
|
||||||
}
|
}
|
||||||
if len(resolved.Steps) > 0 {
|
if len(resolved.Steps) > 0 {
|
||||||
out.Steps = make([]ResolvedPipelineStep, len(resolved.Steps))
|
out.Steps = make([]ResolvedPipelineStep, len(resolved.Steps))
|
||||||
@@ -76,16 +76,16 @@ func MaterializeReferences(resolved ResolvedPipeline, catalog ModuleCatalog, opt
|
|||||||
out.Steps[i].ID = step.ID
|
out.Steps[i].ID = step.ID
|
||||||
out.Steps[i].ArtifactLanes = make([]ResolvedArtifactLane, len(step.ArtifactLanes))
|
out.Steps[i].ArtifactLanes = make([]ResolvedArtifactLane, len(step.ArtifactLanes))
|
||||||
for j, lane := range step.ArtifactLanes {
|
for j, lane := range step.ArtifactLanes {
|
||||||
materializedLane, laneWarnings, err := materializeLane(lane)
|
materializedLane, laneDiagnostics, err := materializeLane(lane)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ResolvedPipeline{}, nil, err
|
return ResolvedPipeline{}, nil, err
|
||||||
}
|
}
|
||||||
out.Steps[i].ArtifactLanes[j] = materializedLane
|
out.Steps[i].ArtifactLanes[j] = materializedLane
|
||||||
warnings = append(warnings, laneWarnings...)
|
diagnostics = append(diagnostics, laneDiagnostics...)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return out, warnings, nil
|
return out, diagnostics, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func materializeReferenceTarget(
|
func materializeReferenceTarget(
|
||||||
@@ -94,7 +94,7 @@ func materializeReferenceTarget(
|
|||||||
artifactKind contracts.ArtifactKind,
|
artifactKind contracts.ArtifactKind,
|
||||||
catalog ModuleCatalog,
|
catalog ModuleCatalog,
|
||||||
options ReferenceMaterializationOptions,
|
options ReferenceMaterializationOptions,
|
||||||
) (contracts.ReferenceSet, []contracts.Warning, error) {
|
) (contracts.ReferenceSet, []contracts.ProducerDiagnostic, error) {
|
||||||
if len(target.Bindings) == 0 {
|
if len(target.Bindings) == 0 {
|
||||||
return contracts.ReferenceSet{}, nil, nil
|
return contracts.ReferenceSet{}, nil, nil
|
||||||
}
|
}
|
||||||
@@ -109,7 +109,7 @@ func materializeReferenceTarget(
|
|||||||
}
|
}
|
||||||
|
|
||||||
set := contracts.ReferenceSet{Slots: make(map[string]contracts.ResolvedReferenceSlot, len(target.Bindings))}
|
set := contracts.ReferenceSet{Slots: make(map[string]contracts.ResolvedReferenceSlot, len(target.Bindings))}
|
||||||
var warnings []contracts.Warning
|
var diagnostics []contracts.ProducerDiagnostic
|
||||||
for _, binding := range target.Bindings {
|
for _, binding := range target.Bindings {
|
||||||
slotName := strings.TrimSpace(binding.SlotName)
|
slotName := strings.TrimSpace(binding.SlotName)
|
||||||
slot, ok := slotByName[slotName]
|
slot, ok := slotByName[slotName]
|
||||||
@@ -159,11 +159,7 @@ func materializeReferenceTarget(
|
|||||||
return contracts.ReferenceSet{}, nil, fmt.Errorf("%s reference slot %q path %q media type %q is not accepted", referenceTargetContext(pipelineID, target), slotName, path, mediaType)
|
return contracts.ReferenceSet{}, nil, fmt.Errorf("%s reference slot %q path %q media type %q is not accepted", referenceTargetContext(pipelineID, target), slotName, path, mediaType)
|
||||||
}
|
}
|
||||||
if len(content) == 0 {
|
if len(content) == 0 {
|
||||||
warnings = append(warnings, contracts.Warning{
|
diagnostics = append(diagnostics, contracts.ProducerDiagnostic{Disposition: contracts.DiagnosticDispositionWarning, Category: contracts.DiagnosticCategoryConfiguration, ReasonCode: "empty_reference", OccurrenceCount: 1, Samples: []contracts.DiagnosticSample{{Scope: referenceWarningScope(pipelineID, target, slotName), Message: fmt.Sprintf("reference slot %q for %s is bound to an empty file", slotName, referenceTargetLabel(target))}}})
|
||||||
Scope: referenceWarningScope(pipelineID, target, slotName),
|
|
||||||
ReasonCode: "empty_reference",
|
|
||||||
Message: fmt.Sprintf("reference slot %q for %s is bound to an empty file", slotName, referenceTargetLabel(target)),
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
item := contracts.ReferenceItem{
|
item := contracts.ReferenceItem{
|
||||||
@@ -180,7 +176,7 @@ func materializeReferenceTarget(
|
|||||||
Items: []contracts.ReferenceItem{item},
|
Items: []contracts.ReferenceItem{item},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return set, warnings, nil
|
return set, diagnostics, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type referenceSizeLimitError struct {
|
type referenceSizeLimitError struct {
|
||||||
|
|||||||
@@ -445,14 +445,14 @@ func TestMaterializeReferencesWarnsForEmptyFiles(t *testing.T) {
|
|||||||
writeReferenceFile(t, path, nil)
|
writeReferenceFile(t, path, nil)
|
||||||
|
|
||||||
resolved := resolvedPipelineWithReference(t, "roster", "empty.txt", contracts.ReferenceBindingSourceConfig, contracts.ReferenceSlot{Name: "roster"})
|
resolved := resolvedPipelineWithReference(t, "roster", "empty.txt", contracts.ReferenceBindingSourceConfig, contracts.ReferenceSlot{Name: "roster"})
|
||||||
materialized, warnings, err := MaterializeReferences(resolved, referenceCatalog(t, []contracts.ReferenceSlot{{Name: "roster"}}), ReferenceMaterializationOptions{
|
materialized, diagnostics, err := MaterializeReferences(resolved, referenceCatalog(t, []contracts.ReferenceSlot{{Name: "roster"}}), ReferenceMaterializationOptions{
|
||||||
ConfigPath: filepath.Join(configDir, "config.yml"),
|
ConfigPath: filepath.Join(configDir, "config.yml"),
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
|
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
if len(warnings) != 1 || warnings[0].ReasonCode != "empty_reference" {
|
if len(diagnostics) != 1 || diagnostics[0].Disposition != contracts.DiagnosticDispositionWarning || diagnostics[0].Category != contracts.DiagnosticCategoryConfiguration || diagnostics[0].ReasonCode != "empty_reference" || diagnostics[0].OccurrenceCount != 1 || len(diagnostics[0].Samples) != 1 {
|
||||||
t.Fatalf("warnings = %#v, want empty reference warning", warnings)
|
t.Fatalf("diagnostics = %#v, want structured empty-reference signal", diagnostics)
|
||||||
}
|
}
|
||||||
item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||||
if item.SizeBytes != 0 || item.Digest != referenceDigest(nil) {
|
if item.SizeBytes != 0 || item.Digest != referenceDigest(nil) {
|
||||||
@@ -483,13 +483,13 @@ func TestMaterializeReferencesWarningScopesIncludeTargetContext(t *testing.T) {
|
|||||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
_, warnings, err := MaterializeReferences(resolved, catalog, ReferenceMaterializationOptions{
|
_, diagnostics, err := MaterializeReferences(resolved, catalog, ReferenceMaterializationOptions{
|
||||||
ConfigPath: filepath.Join(configDir, "config.yml"),
|
ConfigPath: filepath.Join(configDir, "config.yml"),
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
|
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
got := warningScopes(warnings)
|
got := diagnosticScopes(diagnostics)
|
||||||
want := []string{
|
want := []string{
|
||||||
"pipeline.baseline.chunk.reference.scene_guide",
|
"pipeline.baseline.chunk.reference.scene_guide",
|
||||||
"pipeline.baseline.lane.events.extract.reference.roster",
|
"pipeline.baseline.lane.events.extract.reference.roster",
|
||||||
@@ -621,10 +621,10 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlo
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
func warningScopes(warnings []contracts.Warning) []string {
|
func diagnosticScopes(diagnostics []contracts.ProducerDiagnostic) []string {
|
||||||
scopes := make([]string, 0, len(warnings))
|
scopes := make([]string, 0, len(diagnostics))
|
||||||
for _, warning := range warnings {
|
for _, diagnostic := range diagnostics {
|
||||||
scopes = append(scopes, warning.Scope)
|
scopes = append(scopes, diagnostic.Samples[0].Scope)
|
||||||
}
|
}
|
||||||
sort.Strings(scopes)
|
sort.Strings(scopes)
|
||||||
return scopes
|
return scopes
|
||||||
|
|||||||
@@ -50,7 +50,7 @@ type RunInput struct {
|
|||||||
StartedAt time.Time
|
StartedAt time.Time
|
||||||
LLMProfiles []artifacts.LLMProfileManifest
|
LLMProfiles []artifacts.LLMProfileManifest
|
||||||
Metadata map[string]any
|
Metadata map[string]any
|
||||||
Warnings []contracts.Warning
|
Diagnostics []contracts.ProducerDiagnostic
|
||||||
ChunkCacheMode ChunkCacheMode
|
ChunkCacheMode ChunkCacheMode
|
||||||
ChunkPlans ChunkPlanStore
|
ChunkPlans ChunkPlanStore
|
||||||
Checkpoints CheckpointRecorder
|
Checkpoints CheckpointRecorder
|
||||||
@@ -69,16 +69,17 @@ type RunInput struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type RunOutput struct {
|
type RunOutput struct {
|
||||||
Manifest artifacts.RunManifest `json:"manifest"`
|
Manifest artifacts.RunManifest `json:"manifest"`
|
||||||
ChunkPlan *artifacts.ChunkPlanSummary `json:"chunk_plan,omitempty"`
|
ChunkPlan *artifacts.ChunkPlanSummary `json:"chunk_plan,omitempty"`
|
||||||
NormalizeOutputs []contracts.SerializedOutput `json:"normalize_outputs,omitempty"`
|
NormalizeOutputs []contracts.SerializedOutput `json:"normalize_outputs,omitempty"`
|
||||||
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
|
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
|
||||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
Diagnostics contracts.DiagnosticCollection `json:"diagnostics,omitempty"`
|
||||||
OutputFiles []contracts.OutputFile `json:"-"`
|
OutputFiles []contracts.OutputFile `json:"-"`
|
||||||
CheckpointEvents []CheckpointEvent `json:"checkpoint_events,omitempty"`
|
CheckpointEvents []CheckpointEvent `json:"checkpoint_events,omitempty"`
|
||||||
ValidationSummaries []artifacts.ValidationSummary `json:"validation_summaries,omitempty"`
|
ValidationSummaries []artifacts.ValidationSummary `json:"validation_summaries,omitempty"`
|
||||||
|
|
||||||
normalizeReuseEligibility map[generatedOutputKey]bool
|
normalizeReuseEligibility map[generatedOutputKey]bool
|
||||||
|
diagnosticGroups []contracts.DiagnosticGroup
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err error) {
|
func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err error) {
|
||||||
@@ -129,7 +130,11 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
|||||||
defer func() {
|
defer func() {
|
||||||
output.Manifest.LLMProfiles = mergeLLMProfileManifests(input.LLMProfiles, llmProfileManifests(input.llmClient))
|
output.Manifest.LLMProfiles = mergeLLMProfileManifests(input.LLMProfiles, llmProfileManifests(input.llmClient))
|
||||||
}()
|
}()
|
||||||
output.Warnings = append(output.Warnings, cloneWarnings(input.Warnings)...)
|
inputDiagnostics, diagnosticErr := promoteProducerDiagnostics(input.Diagnostics, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageReferences}, nil)
|
||||||
|
if diagnosticErr != nil {
|
||||||
|
return failOutput(output), fmt.Errorf("promote input diagnostics: %w", diagnosticErr)
|
||||||
|
}
|
||||||
|
appendDiagnosticGroups(&output, inputDiagnostics)
|
||||||
if err := writeDebugTimed(debugRecorder, "run.json", debugTimedEnvelope{
|
if err := writeDebugTimed(debugRecorder, "run.json", debugTimedEnvelope{
|
||||||
Stage: "run",
|
Stage: "run",
|
||||||
StartedAt: startedTime(input.StartedAt),
|
StartedAt: startedTime(input.StartedAt),
|
||||||
@@ -248,13 +253,12 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
|||||||
if chunkResult.validation != nil {
|
if chunkResult.validation != nil {
|
||||||
output.ValidationSummaries = append(output.ValidationSummaries, artifacts.CloneValidationSummary(*chunkResult.validation))
|
output.ValidationSummaries = append(output.ValidationSummaries, artifacts.CloneValidationSummary(*chunkResult.validation))
|
||||||
}
|
}
|
||||||
output.Warnings = append(output.Warnings, chunkResult.warnings...)
|
appendDiagnosticGroups(&output, chunkResult.diagnostics)
|
||||||
chunkDebugPayload := map[string]any{
|
chunkDebugPayload := map[string]any{
|
||||||
"cache_mode": chunkMode,
|
"cache_mode": chunkMode,
|
||||||
"lookup": chunkResult.lookup,
|
"lookup": chunkResult.lookup,
|
||||||
"accepted": chunkResult.accepted,
|
"accepted": chunkResult.accepted,
|
||||||
"materialized_chunks": debugSourceChunkEnvelopes(chunkResult.chunks),
|
"materialized_chunks": debugSourceChunkEnvelopes(chunkResult.chunks),
|
||||||
"warnings": chunkResult.warnings,
|
|
||||||
}
|
}
|
||||||
if chunkResult.plan != nil {
|
if chunkResult.plan != nil {
|
||||||
chunkDebugPayload["plan"] = debugChunkPlanEnvelope(*chunkResult.plan)
|
chunkDebugPayload["plan"] = debugChunkPlanEnvelope(*chunkResult.plan)
|
||||||
@@ -311,6 +315,9 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
|||||||
}
|
}
|
||||||
populateOutputManifest(&output)
|
populateOutputManifest(&output)
|
||||||
output.Manifest.CompletedAt = timePtr(time.Now().UTC())
|
output.Manifest.CompletedAt = timePtr(time.Now().UTC())
|
||||||
|
if err := finalizeDiagnostics(&output); err != nil {
|
||||||
|
return failOutput(output), fmt.Errorf("aggregate diagnostics: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
encoder := input.Prepared.output
|
encoder := input.Prepared.output
|
||||||
if err := attachModuleManifestMetadata(&output, "output", encoder); err != nil {
|
if err := attachModuleManifestMetadata(&output, "output", encoder); err != nil {
|
||||||
@@ -344,7 +351,6 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
|||||||
"manifest": output.Manifest,
|
"manifest": output.Manifest,
|
||||||
"normalize_outputs": debugSerializedOutputEnvelopes(output.NormalizeOutputs),
|
"normalize_outputs": debugSerializedOutputEnvelopes(output.NormalizeOutputs),
|
||||||
"rejected": debugRejectedOutputEnvelopes(output.Rejected),
|
"rejected": debugRejectedOutputEnvelopes(output.Rejected),
|
||||||
"warnings": output.Warnings,
|
|
||||||
"options": redactSensitiveMap(input.pipeline.Output.Options),
|
"options": redactSensitiveMap(input.pipeline.Output.Options),
|
||||||
"metadata": redactSensitiveMap(input.Metadata),
|
"metadata": redactSensitiveMap(input.Metadata),
|
||||||
}
|
}
|
||||||
@@ -373,7 +379,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
|||||||
Manifest: output.Manifest,
|
Manifest: output.Manifest,
|
||||||
NormalizeOutputs: cloneSerializedOutputs(output.NormalizeOutputs),
|
NormalizeOutputs: cloneSerializedOutputs(output.NormalizeOutputs),
|
||||||
Rejected: cloneRejectedOutputs(output.Rejected),
|
Rejected: cloneRejectedOutputs(output.Rejected),
|
||||||
Warnings: output.Warnings,
|
Diagnostics: contracts.CloneDiagnosticCollection(output.Diagnostics),
|
||||||
LLMProfile: input.pipeline.Output.LLMProfile,
|
LLMProfile: input.pipeline.Output.LLMProfile,
|
||||||
StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(input.pipeline.Output.StructuredOutputRepairAttempts),
|
StructuredOutputRepairAttempts: cloneStructuredOutputRepairAttempts(input.pipeline.Output.StructuredOutputRepairAttempts),
|
||||||
Metadata: outputMetadata,
|
Metadata: outputMetadata,
|
||||||
@@ -395,8 +401,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
|||||||
ModuleKey: encoder.Key(),
|
ModuleKey: encoder.Key(),
|
||||||
StartedAt: outputStarted,
|
StartedAt: outputStarted,
|
||||||
Payload: map[string]any{
|
Payload: map[string]any{
|
||||||
"files": debugOutputFiles(files),
|
"files": debugOutputFiles(files),
|
||||||
"warnings": encoded.Warnings,
|
|
||||||
},
|
},
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
return failOutput(output), fmt.Errorf("write output debug artifact: %w", err)
|
return failOutput(output), fmt.Errorf("write output debug artifact: %w", err)
|
||||||
@@ -404,7 +409,6 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
|||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return failOutput(output), err
|
return failOutput(output), err
|
||||||
}
|
}
|
||||||
output.Warnings = append(output.Warnings, encoded.Warnings...)
|
|
||||||
output.OutputFiles = files
|
output.OutputFiles = files
|
||||||
|
|
||||||
return output, nil
|
return output, nil
|
||||||
@@ -444,7 +448,6 @@ func (r *Runner) runPreparedSteps(ctx context.Context, input RunInput, checkpoin
|
|||||||
type retryAttemptResult struct {
|
type retryAttemptResult struct {
|
||||||
accepted bool
|
accepted bool
|
||||||
rejection *contracts.RejectedOutput
|
rejection *contracts.RejectedOutput
|
||||||
warnings []contracts.Warning
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func runSimpleRetry(ctx context.Context, retries int, run func(attempt int) (retryAttemptResult, error)) (retryAttemptResult, error) {
|
func runSimpleRetry(ctx context.Context, retries int, run func(attempt int) (retryAttemptResult, error)) (retryAttemptResult, error) {
|
||||||
@@ -474,7 +477,7 @@ func runSimpleRetry(ctx context.Context, retries int, run func(attempt int) (ret
|
|||||||
if result.rejection != nil {
|
if result.rejection != nil {
|
||||||
rejection := *result.rejection
|
rejection := *result.rejection
|
||||||
rejection.AttemptCount = attempt
|
rejection.AttemptCount = attempt
|
||||||
last = retryAttemptResult{rejection: &rejection, warnings: cloneWarnings(result.warnings)}
|
last = retryAttemptResult{rejection: &rejection}
|
||||||
}
|
}
|
||||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
return retryAttemptResult{}, ctxErr
|
return retryAttemptResult{}, ctxErr
|
||||||
@@ -697,6 +700,10 @@ func validatorChainManifests(chains []ResolvedValidatorChain) []artifacts.Valida
|
|||||||
}
|
}
|
||||||
|
|
||||||
func failOutput(output RunOutput) RunOutput {
|
func failOutput(output RunOutput) RunOutput {
|
||||||
|
// Preserve any diagnostics already accepted by the pipeline when a later
|
||||||
|
// operation fails. Producers and validators validate each diagnostic before
|
||||||
|
// it is appended, so finalization here cannot introduce a new failure path.
|
||||||
|
_ = finalizeDiagnostics(&output)
|
||||||
if output.Manifest.PipelineID != "" {
|
if output.Manifest.PipelineID != "" {
|
||||||
populateOutputManifest(&output)
|
populateOutputManifest(&output)
|
||||||
output.Manifest.ValidationStatus = "failed"
|
output.Manifest.ValidationStatus = "failed"
|
||||||
@@ -971,13 +978,6 @@ func manifestMetadataWithSessionID(metadata map[string]any, sessionID string) (m
|
|||||||
return out, nil
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func cloneWarnings(warnings []contracts.Warning) []contracts.Warning {
|
|
||||||
if len(warnings) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
return append([]contracts.Warning(nil), warnings...)
|
|
||||||
}
|
|
||||||
|
|
||||||
func cloneSourceChunkPtr(chunk *source.Chunk) (*source.Chunk, error) {
|
func cloneSourceChunkPtr(chunk *source.Chunk) (*source.Chunk, error) {
|
||||||
if chunk == nil {
|
if chunk == nil {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
|
|||||||
@@ -84,7 +84,8 @@ func TestRunnerHydratesRequiredNormalizedArtifact(t *testing.T) {
|
|||||||
loader := newAcceptedCheckpointLoader()
|
loader := newAcceptedCheckpointLoader()
|
||||||
producerKey := CheckpointLaneKey(producer.resolved.StepID, producer.resolved.ID)
|
producerKey := CheckpointLaneKey(producer.resolved.StepID, producer.resolved.ID)
|
||||||
consumerKey := CheckpointLaneKey(consumer.resolved.StepID, consumer.resolved.ID)
|
consumerKey := CheckpointLaneKey(consumer.resolved.StepID, consumer.resolved.ID)
|
||||||
loader.accepted[producerKey] = NormalizeCheckpoint{Output: stored, Warnings: []contracts.Warning{{Scope: "normalize", ReasonCode: "stored-warning", Message: "stored normalize warning"}}}
|
stored.Diagnostics = []CheckpointDiagnostic{{Diagnostic: contracts.ProducerDiagnostic{Disposition: contracts.DiagnosticDispositionWarning, Category: contracts.DiagnosticCategoryFallback, ReasonCode: "stored-warning", OccurrenceCount: 1, Samples: []contracts.DiagnosticSample{{Scope: "normalize", Message: "stored normalize warning"}}}}}
|
||||||
|
loader.accepted[producerKey] = NormalizeCheckpoint{Output: stored}
|
||||||
loader.acceptedDecision[producerKey] = NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused)
|
loader.acceptedDecision[producerKey] = NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused)
|
||||||
policy := CheckpointExecutionPolicy{
|
policy := CheckpointExecutionPolicy{
|
||||||
RequireReusableLanes: map[string]struct{}{producerKey: {}},
|
RequireReusableLanes: map[string]struct{}{producerKey: {}},
|
||||||
@@ -101,8 +102,8 @@ func TestRunnerHydratesRequiredNormalizedArtifact(t *testing.T) {
|
|||||||
if string(item.Content) != string(stored.Artifact.Content) || item.Producer.StepID != producer.resolved.StepID || item.Producer.LaneID != producer.resolved.ID {
|
if string(item.Content) != string(stored.Artifact.Content) || item.Producer.StepID != producer.resolved.StepID || item.Producer.LaneID != producer.resolved.ID {
|
||||||
t.Fatalf("consumer generated reference = %#v, want exact hydrated producer bytes and identity", item)
|
t.Fatalf("consumer generated reference = %#v, want exact hydrated producer bytes and identity", item)
|
||||||
}
|
}
|
||||||
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != "stored-warning" {
|
if len(output.Diagnostics.Groups) != 1 || output.Diagnostics.Groups[0].ReasonCode != "stored-warning" {
|
||||||
t.Fatalf("hydrated warnings = %#v, want normalize checkpoint warnings only", output.Warnings)
|
t.Fatalf("hydrated diagnostics = %#v, want normalize checkpoint diagnostics only", output.Diagnostics)
|
||||||
}
|
}
|
||||||
assertAcceptedNormalizeEvent(t, output.CheckpointEvents, producer.resolved.StepID, producer.resolved.ID, CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused)
|
assertAcceptedNormalizeEvent(t, output.CheckpointEvents, producer.resolved.StepID, producer.resolved.ID, CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused)
|
||||||
for _, event := range output.CheckpointEvents {
|
for _, event := range output.CheckpointEvents {
|
||||||
@@ -225,7 +226,8 @@ func TestRunnerRetainsEarlierHydratedProducerWhenLaterRequiredProducerFails(t *t
|
|||||||
loader := newAcceptedCheckpointLoader()
|
loader := newAcceptedCheckpointLoader()
|
||||||
firstKey := CheckpointLaneKey(first.resolved.StepID, first.resolved.ID)
|
firstKey := CheckpointLaneKey(first.resolved.StepID, first.resolved.ID)
|
||||||
secondKey := CheckpointLaneKey(second.resolved.StepID, second.resolved.ID)
|
secondKey := CheckpointLaneKey(second.resolved.StepID, second.resolved.ID)
|
||||||
loader.accepted[firstKey] = NormalizeCheckpoint{Output: stored, Warnings: []contracts.Warning{{Scope: "normalize", ReasonCode: "retained-warning", Message: "retained warning"}}}
|
stored.Diagnostics = []CheckpointDiagnostic{{Diagnostic: contracts.ProducerDiagnostic{Disposition: contracts.DiagnosticDispositionWarning, Category: contracts.DiagnosticCategoryFallback, ReasonCode: "retained-warning", OccurrenceCount: 1, Samples: []contracts.DiagnosticSample{{Scope: "normalize", Message: "retained warning"}}}}}
|
||||||
|
loader.accepted[firstKey] = NormalizeCheckpoint{Output: stored}
|
||||||
loader.acceptedDecision[firstKey] = NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused)
|
loader.acceptedDecision[firstKey] = NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused)
|
||||||
loader.acceptedDecision[secondKey] = NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing)
|
loader.acceptedDecision[secondKey] = NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing)
|
||||||
policy := CheckpointExecutionPolicy{RequireReusableLanes: map[string]struct{}{firstKey: {}, secondKey: {}}}
|
policy := CheckpointExecutionPolicy{RequireReusableLanes: map[string]struct{}{firstKey: {}, secondKey: {}}}
|
||||||
@@ -240,8 +242,8 @@ func TestRunnerRetainsEarlierHydratedProducerWhenLaterRequiredProducerFails(t *t
|
|||||||
if len(output.NormalizeOutputs) != 1 || output.NormalizeOutputs[0].LaneID != first.resolved.ID || string(output.NormalizeOutputs[0].Artifact.Content) != string(stored.Artifact.Content) {
|
if len(output.NormalizeOutputs) != 1 || output.NormalizeOutputs[0].LaneID != first.resolved.ID || string(output.NormalizeOutputs[0].Artifact.Content) != string(stored.Artifact.Content) {
|
||||||
t.Fatalf("retained normalize outputs = %#v, want first producer", output.NormalizeOutputs)
|
t.Fatalf("retained normalize outputs = %#v, want first producer", output.NormalizeOutputs)
|
||||||
}
|
}
|
||||||
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != "retained-warning" {
|
if len(output.Diagnostics.Groups) != 1 || output.Diagnostics.Groups[0].ReasonCode != "retained-warning" {
|
||||||
t.Fatalf("retained warnings = %#v", output.Warnings)
|
t.Fatalf("retained diagnostics = %#v", output.Diagnostics)
|
||||||
}
|
}
|
||||||
type decisionExpectation struct {
|
type decisionExpectation struct {
|
||||||
step, lane string
|
step, lane string
|
||||||
|
|||||||
@@ -162,13 +162,13 @@ func TestRunnerWritesAttemptScopedMergeAndNormalizeDebug(t *testing.T) {
|
|||||||
if err := callAttemptDebugLLM(ctx, client, "merge"); err != nil {
|
if err := callAttemptDebugLLM(ctx, client, "merge"); err != nil {
|
||||||
return erasedTypedResult{}, err
|
return erasedTypedResult{}, err
|
||||||
}
|
}
|
||||||
return erasedTypedResult{Value: codecNotes{Items: []string{"merged"}}, Warnings: []contracts.Warning{{Scope: "merge", ReasonCode: "observed", Message: "merge warning"}}}, nil
|
return erasedTypedResult{Value: codecNotes{Items: []string{"merged"}}, Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("merge", "merge diagnostic")}}, nil
|
||||||
}
|
}
|
||||||
prepared.Steps[0].lanes[0].typed.normalize = func(ctx context.Context, _ any, _ contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
prepared.Steps[0].lanes[0].typed.normalize = func(ctx context.Context, _ any, _ contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||||
if err := callAttemptDebugLLM(ctx, client, "normalize"); err != nil {
|
if err := callAttemptDebugLLM(ctx, client, "normalize"); err != nil {
|
||||||
return erasedTypedResult{}, err
|
return erasedTypedResult{}, err
|
||||||
}
|
}
|
||||||
return erasedTypedResult{Value: codecNotes{Items: []string{"normalized"}}, Warnings: []contracts.Warning{{Scope: "normalize", ReasonCode: "observed", Message: "normalize warning"}}}, nil
|
return erasedTypedResult{Value: codecNotes{Items: []string{"normalized"}}, Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("normalize", "normalize diagnostic")}}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug})
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug})
|
||||||
@@ -213,7 +213,7 @@ func TestRunnerWritesAttemptScopedMergeAndNormalizeDebug(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunnerRecordsDistinctRetryAttemptsAndPromotesAcceptedWarningsOnly(t *testing.T) {
|
func TestRunnerRecordsDistinctRetryAttemptsAndPromotesAcceptedDiagnosticsOnly(t *testing.T) {
|
||||||
for _, stage := range []ModuleStage{StageMerge, StageNormalize} {
|
for _, stage := range []ModuleStage{StageMerge, StageNormalize} {
|
||||||
t.Run(string(stage), func(t *testing.T) {
|
t.Run(string(stage), func(t *testing.T) {
|
||||||
prepared := preparedAttemptDebugPipeline(t)
|
prepared := preparedAttemptDebugPipeline(t)
|
||||||
@@ -230,7 +230,7 @@ func TestRunnerRecordsDistinctRetryAttemptsAndPromotesAcceptedWarningsOnly(t *te
|
|||||||
if attempts == 1 {
|
if attempts == 1 {
|
||||||
scope = "discarded"
|
scope = "discarded"
|
||||||
}
|
}
|
||||||
return erasedTypedResult{Value: codecNotes{Items: []string{scope}}, Warnings: []contracts.Warning{{Scope: scope, ReasonCode: "observed", Message: scope}}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%s"]}`, scope))}, nil
|
return erasedTypedResult{Value: codecNotes{Items: []string{scope}}, Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic(scope, scope)}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%s"]}`, scope))}, nil
|
||||||
}
|
}
|
||||||
validatorCalls := 0
|
validatorCalls := 0
|
||||||
validator := preparedValidator{
|
validator := preparedValidator{
|
||||||
@@ -268,11 +268,11 @@ func TestRunnerRecordsDistinctRetryAttemptsAndPromotesAcceptedWarningsOnly(t *te
|
|||||||
if len(first.LLMCalls) != 1 || len(second.LLMCalls) != 1 || first.LLMCalls[0].CallID == second.LLMCalls[0].CallID {
|
if len(first.LLMCalls) != 1 || len(second.LLMCalls) != 1 || first.LLMCalls[0].CallID == second.LLMCalls[0].CallID {
|
||||||
t.Fatalf("retry LLM calls = first %#v, second %#v; want distinct calls", first.LLMCalls, second.LLMCalls)
|
t.Fatalf("retry LLM calls = first %#v, second %#v; want distinct calls", first.LLMCalls, second.LLMCalls)
|
||||||
}
|
}
|
||||||
if !strings.Contains(string(debug.json[firstPath]), "discarded") || !strings.Contains(string(debug.json[firstPath]), "rejection") {
|
if !strings.Contains(string(debug.json[firstPath]), "rejection") {
|
||||||
t.Fatalf("first attempt envelope = %s, want discarded warning and rejection", debug.json[firstPath])
|
t.Fatalf("first attempt envelope = %s, want rejection", debug.json[firstPath])
|
||||||
}
|
}
|
||||||
if len(output.Warnings) != 1 || output.Warnings[0].Scope != "accepted" {
|
if len(output.Diagnostics.Groups) != 1 || output.Diagnostics.Groups[0].Samples[0].Scope != "accepted" {
|
||||||
t.Fatalf("promoted warnings = %#v, want accepted attempt only", output.Warnings)
|
t.Fatalf("promoted diagnostics = %#v, want accepted attempt only", output.Diagnostics)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -101,13 +101,13 @@ type candidateCheckpointRecorder struct {
|
|||||||
normalizeOutput CheckpointArtifact
|
normalizeOutput CheckpointArtifact
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *candidateCheckpointRecorder) MergeSucceeded(_ string, _ string, _ []CheckpointFingerprint, output CheckpointArtifact, _ []contracts.Warning) error {
|
func (r *candidateCheckpointRecorder) MergeSucceeded(_ string, _ string, _ []CheckpointFingerprint, output CheckpointArtifact) error {
|
||||||
r.mergeSucceeded++
|
r.mergeSucceeded++
|
||||||
r.mergeOutput = cloneCheckpointArtifact(output)
|
r.mergeOutput = cloneCheckpointArtifact(output)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *candidateCheckpointRecorder) NormalizeSucceeded(_ string, _ string, _ []CheckpointFingerprint, output CheckpointArtifact, _ []contracts.Warning) error {
|
func (r *candidateCheckpointRecorder) NormalizeSucceeded(_ string, _ string, _ []CheckpointFingerprint, output CheckpointArtifact) error {
|
||||||
r.normalizeSucceeded++
|
r.normalizeSucceeded++
|
||||||
r.normalizeOutput = cloneCheckpointArtifact(output)
|
r.normalizeOutput = cloneCheckpointArtifact(output)
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -12,24 +12,24 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type chunkPlanExecution struct {
|
type chunkPlanExecution struct {
|
||||||
accepted bool
|
accepted bool
|
||||||
chunks []source.Chunk
|
chunks []source.Chunk
|
||||||
plan *source.ChunkPlan
|
plan *source.ChunkPlan
|
||||||
warnings []contracts.Warning
|
diagnostics []contracts.DiagnosticGroup
|
||||||
rejection *contracts.RejectedOutput
|
rejection *contracts.RejectedOutput
|
||||||
lookup ChunkPlanDecision
|
lookup ChunkPlanDecision
|
||||||
record *ChunkPlanRecord
|
record *ChunkPlanRecord
|
||||||
action string
|
action string
|
||||||
summary artifacts.ChunkPlanSummary
|
summary artifacts.ChunkPlanSummary
|
||||||
validation *artifacts.ValidationSummary
|
validation *artifacts.ValidationSummary
|
||||||
}
|
}
|
||||||
|
|
||||||
type generatedChunkPlanCandidate struct {
|
type generatedChunkPlanCandidate struct {
|
||||||
plan source.ChunkPlan
|
plan source.ChunkPlan
|
||||||
chunks []source.Chunk
|
chunks []source.Chunk
|
||||||
record ChunkPlanRecord
|
record ChunkPlanRecord
|
||||||
producerWarnings []contracts.Warning
|
producerDiagnostics []contracts.ProducerDiagnostic
|
||||||
terminal *attemptTerminalRecorder
|
terminal *attemptTerminalRecorder
|
||||||
}
|
}
|
||||||
|
|
||||||
func effectiveChunkCacheMode(mode ChunkCacheMode) ChunkCacheMode {
|
func effectiveChunkCacheMode(mode ChunkCacheMode) ChunkCacheMode {
|
||||||
@@ -69,7 +69,7 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
|||||||
if validationErr == nil {
|
if validationErr == nil {
|
||||||
report, err := r.validateChunkReport(ctx, doc, chunker.Key(), chunks, sourceInput, sessionID, input.pipeline.ChunkReferences.ReferenceSet, input.Metadata, input.Prepared.chunkValidators, 1, input.Debug)
|
report, err := r.validateChunkReport(ctx, doc, chunker.Key(), chunks, sourceInput, sessionID, input.pipeline.ChunkReferences.ReferenceSet, input.Metadata, input.Prepared.chunkValidators, 1, input.Debug)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
result.setValidation(report.Warnings(), nil, err)
|
result.setValidation(nil, err)
|
||||||
return result, err
|
return result, err
|
||||||
}
|
}
|
||||||
rejection := report.FirstRejection()
|
rejection := report.FirstRejection()
|
||||||
@@ -80,24 +80,28 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
|||||||
}
|
}
|
||||||
result.plan = &plan
|
result.plan = &plan
|
||||||
result.chunks = chunks
|
result.chunks = chunks
|
||||||
result.warnings = append(cloneWarnings(record.Warnings), report.Warnings()...)
|
|
||||||
result.accepted = true
|
result.accepted = true
|
||||||
result.setValidation(report.Warnings(), nil, nil)
|
result.setValidation(nil, nil)
|
||||||
cachedTerminal := producerAttemptTerminal{Action: producerTerminalAccepted, Validation: report}
|
cachedTerminal := producerAttemptTerminal{Action: producerTerminalAccepted, Diagnostics: contracts.CloneProducerDiagnostics(record.Diagnostics), Validation: report}
|
||||||
|
diagnostics, diagnosticErr := terminalDiagnosticGroups(cachedTerminal, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageChunk, ModuleKey: chunker.Key()}, nil)
|
||||||
|
if diagnosticErr != nil {
|
||||||
|
return result, fmt.Errorf("promote reused chunk diagnostics: %w", diagnosticErr)
|
||||||
|
}
|
||||||
|
result.diagnostics = diagnostics
|
||||||
summary := validationSummary(cachedTerminal, StageChunk, "", "", chunker.Key(), "", 0)
|
summary := validationSummary(cachedTerminal, StageChunk, "", "", chunker.Key(), "", 0)
|
||||||
result.validation = &summary
|
result.validation = &summary
|
||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
if incomplete != nil && input.pipeline.ChunkValidationPolicy.ValidatorFailure == ValidatorFailureFailRun {
|
if incomplete != nil && input.pipeline.ChunkValidationPolicy.ValidatorFailure == ValidatorFailureFailRun {
|
||||||
failure := validatorFailureError(*incomplete)
|
failure := validatorFailureError(*incomplete)
|
||||||
result.setValidation(report.Warnings(), nil, failure)
|
result.setValidation(nil, failure)
|
||||||
failedTerminal := producerAttemptTerminal{Action: producerTerminalFailed, Validation: report, ValidationIncomplete: true}
|
failedTerminal := producerAttemptTerminal{Action: producerTerminalFailed, Validation: report, ValidationIncomplete: true}
|
||||||
summary := validationSummary(failedTerminal, StageChunk, "", "", chunker.Key(), "", 0)
|
summary := validationSummary(failedTerminal, StageChunk, "", "", chunker.Key(), "", 0)
|
||||||
result.validation = &summary
|
result.validation = &summary
|
||||||
return result, failure
|
return result, failure
|
||||||
}
|
}
|
||||||
// A cache hit is not model material. Its rejection is discarded and
|
// A cache hit is not model material. Its rejection is discarded and
|
||||||
// generation begins with the ordinary initial request below. Warnings
|
// generation begins with the ordinary initial request below. Diagnostics
|
||||||
// from this discarded candidate are intentionally not promoted.
|
// from this discarded candidate are intentionally not promoted.
|
||||||
}
|
}
|
||||||
result.lookup = ChunkPlanDecision{Status: ChunkPlanInvalid, Reason: chunkPlanLookupReason(ChunkPlanInvalid)}
|
result.lookup = ChunkPlanDecision{Status: ChunkPlanInvalid, Reason: chunkPlanLookupReason(ChunkPlanInvalid)}
|
||||||
@@ -138,7 +142,7 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
|||||||
plan, chunks, validationErr := validateAndMaterializeChunkPlan(doc, chunkResult.Plan)
|
plan, chunks, validationErr := validateAndMaterializeChunkPlan(doc, chunkResult.Plan)
|
||||||
if validationErr != nil {
|
if validationErr != nil {
|
||||||
attemptErr := fmt.Errorf("validate chunk plan from chunker %q: %w", chunker.Key(), validationErr)
|
attemptErr := fmt.Errorf("validate chunk plan from chunker %q: %w", chunker.Key(), validationErr)
|
||||||
payload := map[string]any{"plan": debugChunkPlanEnvelope(chunkResult.Plan), "warnings": debugWarningEnvelopes(chunkResult.Warnings)}
|
payload := map[string]any{"plan": debugChunkPlanEnvelope(chunkResult.Plan)}
|
||||||
return producerAttemptOutput{}, attemptTerminal.record(payload, fmt.Errorf("%w: %v", contracts.ErrInvalidStructuredOutput, attemptErr))
|
return producerAttemptOutput{}, attemptTerminal.record(payload, fmt.Errorf("%w: %v", contracts.ErrInvalidStructuredOutput, attemptErr))
|
||||||
}
|
}
|
||||||
planDigest, digestErr := source.DigestChunkPlan(plan)
|
planDigest, digestErr := source.DigestChunkPlan(plan)
|
||||||
@@ -161,19 +165,18 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
|||||||
References: append([]artifacts.ReferenceProvenance(nil), referenceTargetProvenance(input.pipeline.ChunkReferences)...),
|
References: append([]artifacts.ReferenceProvenance(nil), referenceTargetProvenance(input.pipeline.ChunkReferences)...),
|
||||||
Metadata: producerMetadata,
|
Metadata: producerMetadata,
|
||||||
},
|
},
|
||||||
Warnings: cloneWarnings(chunkResult.Warnings), CreatedAt: time.Now().UTC(),
|
CreatedAt: time.Now().UTC(),
|
||||||
}
|
}
|
||||||
return producerAttemptOutput{Value: generatedChunkPlanCandidate{plan: plan, chunks: chunks, record: candidate, producerWarnings: cloneWarnings(chunkResult.Warnings), terminal: &attemptTerminal}, Candidate: chunkResult.ModelCandidate, Warnings: cloneWarnings(chunkResult.Warnings)}, nil
|
return producerAttemptOutput{Value: generatedChunkPlanCandidate{plan: plan, chunks: chunks, record: candidate, producerDiagnostics: contracts.CloneProducerDiagnostics(chunkResult.Diagnostics), terminal: &attemptTerminal}, Candidate: chunkResult.ModelCandidate, Diagnostics: contracts.CloneProducerDiagnostics(chunkResult.Diagnostics)}, nil
|
||||||
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
|
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||||
candidate, ok := output.Value.(generatedChunkPlanCandidate)
|
candidate, ok := output.Value.(generatedChunkPlanCandidate)
|
||||||
if !ok {
|
if !ok {
|
||||||
return validationReport{}, fmt.Errorf("chunk attempt candidate has incompatible type")
|
return validationReport{}, fmt.Errorf("chunk attempt candidate has incompatible type")
|
||||||
}
|
}
|
||||||
report, validationErr := r.validateChunkReport(validationCtx, doc, chunker.Key(), candidate.chunks, sourceInput, sessionID, input.pipeline.ChunkReferences.ReferenceSet, input.Metadata, input.Prepared.chunkValidators, candidate.terminal.envelope.Attempt, input.Debug)
|
report, validationErr := r.validateChunkReport(validationCtx, doc, chunker.Key(), candidate.chunks, sourceInput, sessionID, input.pipeline.ChunkReferences.ReferenceSet, input.Metadata, input.Prepared.chunkValidators, candidate.terminal.envelope.Attempt, input.Debug)
|
||||||
attemptWarnings := append(cloneWarnings(output.Warnings), report.Warnings()...)
|
|
||||||
payload := map[string]any{
|
payload := map[string]any{
|
||||||
"plan": debugChunkPlanEnvelope(candidate.plan), "materialized_chunks": debugSourceChunkEnvelopes(candidate.chunks),
|
"plan": debugChunkPlanEnvelope(candidate.plan), "materialized_chunks": debugSourceChunkEnvelopes(candidate.chunks),
|
||||||
"warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(chunkRejection(report, candidate.terminal.envelope.Attempt, chunker.Key())),
|
"rejection": debugRejectedOutputPtr(chunkRejection(report, candidate.terminal.envelope.Attempt, chunker.Key())),
|
||||||
}
|
}
|
||||||
if validationErr != nil {
|
if validationErr != nil {
|
||||||
return report, candidate.terminal.record(payload, validationErr)
|
return report, candidate.terminal.record(payload, validationErr)
|
||||||
@@ -190,18 +193,22 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
|||||||
}
|
}
|
||||||
return result, err
|
return result, err
|
||||||
}
|
}
|
||||||
|
diagnostics, diagnosticErr := terminalDiagnosticGroups(terminal, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageChunk, ModuleKey: chunker.Key()}, nil)
|
||||||
|
if diagnosticErr != nil {
|
||||||
|
return result, fmt.Errorf("promote chunk diagnostics: %w", diagnosticErr)
|
||||||
|
}
|
||||||
|
result.diagnostics = diagnostics
|
||||||
if terminal.Action == producerTerminalRejected {
|
if terminal.Action == producerTerminalRejected {
|
||||||
result.rejection = terminal.Rejection
|
result.rejection = terminal.Rejection
|
||||||
if result.rejection != nil {
|
if result.rejection != nil {
|
||||||
result.rejection.Stage = string(StageChunk)
|
result.rejection.Stage = string(StageChunk)
|
||||||
result.rejection.ModuleKey = chunker.Key()
|
result.rejection.ModuleKey = chunker.Key()
|
||||||
}
|
}
|
||||||
result.warnings = cloneWarnings(terminal.Warnings)
|
|
||||||
result.validation = &terminalSummary
|
result.validation = &terminalSummary
|
||||||
if result.rejection != nil {
|
if result.rejection != nil {
|
||||||
result.rejection.Validation = cloneValidationSummaryPtr(result.validation)
|
result.rejection.Validation = cloneValidationSummaryPtr(result.validation)
|
||||||
}
|
}
|
||||||
result.setValidation(terminal.Validation.Warnings(), result.rejection, nil)
|
result.setValidation(result.rejection, nil)
|
||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
candidate, ok := terminal.Value.(generatedChunkPlanCandidate)
|
candidate, ok := terminal.Value.(generatedChunkPlanCandidate)
|
||||||
@@ -222,9 +229,8 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
|||||||
result.accepted = true
|
result.accepted = true
|
||||||
result.plan = &candidate.plan
|
result.plan = &candidate.plan
|
||||||
result.chunks = candidate.chunks
|
result.chunks = candidate.chunks
|
||||||
result.warnings = cloneWarnings(terminal.Warnings)
|
|
||||||
result.validation = &terminalSummary
|
result.validation = &terminalSummary
|
||||||
result.setValidation(terminal.Validation.Warnings(), nil, nil)
|
result.setValidation(nil, nil)
|
||||||
if terminal.ValidationIncomplete {
|
if terminal.ValidationIncomplete {
|
||||||
result.summary.ValidationStatus = "incomplete"
|
result.summary.ValidationStatus = "incomplete"
|
||||||
}
|
}
|
||||||
@@ -234,7 +240,7 @@ func (r *Runner) runChunkPlan(ctx context.Context, input RunInput, doc *source.S
|
|||||||
if cloneErr != nil {
|
if cloneErr != nil {
|
||||||
return result, fmt.Errorf("clone chunk plan record for publication: %w", cloneErr)
|
return result, fmt.Errorf("clone chunk plan record for publication: %w", cloneErr)
|
||||||
}
|
}
|
||||||
record.Warnings = cloneWarnings(candidate.producerWarnings)
|
record.Diagnostics = contracts.CloneProducerDiagnostics(candidate.producerDiagnostics)
|
||||||
if err := input.ChunkPlans.Save(record); err != nil {
|
if err := input.ChunkPlans.Save(record); err != nil {
|
||||||
return result, fmt.Errorf("save chunk plan: %w", err)
|
return result, fmt.Errorf("save chunk plan: %w", err)
|
||||||
}
|
}
|
||||||
@@ -290,14 +296,12 @@ func (result *chunkPlanExecution) setCandidate(record ChunkPlanRecord, action st
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (result *chunkPlanExecution) setValidation(warnings []contracts.Warning, rejection *contracts.RejectedOutput, err error) {
|
func (result *chunkPlanExecution) setValidation(rejection *contracts.RejectedOutput, err error) {
|
||||||
switch {
|
switch {
|
||||||
case err != nil:
|
case err != nil:
|
||||||
result.summary.ValidationStatus = "error"
|
result.summary.ValidationStatus = "error"
|
||||||
case rejection != nil:
|
case rejection != nil:
|
||||||
result.summary.ValidationStatus = "rejected"
|
result.summary.ValidationStatus = "rejected"
|
||||||
case len(warnings) > 0:
|
|
||||||
result.summary.ValidationStatus = "approved_with_warnings"
|
|
||||||
default:
|
default:
|
||||||
result.summary.ValidationStatus = "approved"
|
result.summary.ValidationStatus = "approved"
|
||||||
}
|
}
|
||||||
@@ -311,7 +315,7 @@ func cloneChunkPlanRecord(record ChunkPlanRecord) (ChunkPlanRecord, error) {
|
|||||||
return ChunkPlanRecord{}, fmt.Errorf("clone chunk plan producer metadata: %w", err)
|
return ChunkPlanRecord{}, fmt.Errorf("clone chunk plan producer metadata: %w", err)
|
||||||
}
|
}
|
||||||
record.Producer.Metadata = metadata
|
record.Producer.Metadata = metadata
|
||||||
record.Warnings = cloneWarnings(record.Warnings)
|
record.Diagnostics = contracts.CloneProducerDiagnostics(record.Diagnostics)
|
||||||
return record, nil
|
return record, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -118,9 +118,9 @@ func (*retryingChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil
|
|||||||
func (c *retryingChunker) Plan(context.Context, contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
func (c *retryingChunker) Plan(context.Context, contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
||||||
c.calls++
|
c.calls++
|
||||||
if c.calls == 1 {
|
if c.calls == 1 {
|
||||||
return contracts.ChunkPlanResult{Warnings: []contracts.Warning{{Scope: "discarded", ReasonCode: "retry", Message: "discarded warning"}}}, errors.New("retry generation")
|
return contracts.ChunkPlanResult{Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("discarded", "discarded diagnostic")}}, errors.New("retry generation")
|
||||||
}
|
}
|
||||||
return contracts.ChunkPlanResult{Plan: source.CloneChunkPlan(c.plan), Warnings: []contracts.Warning{{Scope: "accepted", ReasonCode: "observed", Message: "accepted warning"}}}, nil
|
return contracts.ChunkPlanResult{Plan: source.CloneChunkPlan(c.plan), Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("accepted", "accepted diagnostic")}}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type dependencyLoader struct {
|
type dependencyLoader struct {
|
||||||
@@ -233,6 +233,37 @@ func TestRunnerChunkPlanHitUsesStoredProducerProvenance(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunnerReusesChunkPlanDiagnosticsWithCurrentValidatorDiagnostics(t *testing.T) {
|
||||||
|
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||||
|
record := chunkPlanRecord(t, prepared, plan)
|
||||||
|
record.Diagnostics = []contracts.ProducerDiagnostic{{
|
||||||
|
Disposition: contracts.DiagnosticDispositionWarning,
|
||||||
|
Category: contracts.DiagnosticCategoryConfiguration,
|
||||||
|
ReasonCode: "stored_chunk_diagnostic",
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []contracts.DiagnosticSample{{Scope: "reference", Message: "Stored configuration signal."}},
|
||||||
|
}}
|
||||||
|
validator := &countingChunkValidator{result: contracts.ValidationResult{Approved: true, Diagnostics: []contracts.ProducerDiagnostic{{
|
||||||
|
Disposition: contracts.DiagnosticDispositionAdvisory,
|
||||||
|
Category: contracts.DiagnosticCategoryDataQuality,
|
||||||
|
ReasonCode: "current_validator_diagnostic",
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []contracts.DiagnosticSample{{Scope: "chunk", Message: "Current validator finding."}},
|
||||||
|
}}}}
|
||||||
|
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
|
||||||
|
|
||||||
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: &recordingChunkPlanStore{record: record, decision: ChunkPlanDecision{Status: ChunkPlanHit}}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if validator.calls != 1 || len(output.Diagnostics.Groups) != 2 {
|
||||||
|
t.Fatalf("validator calls = %d diagnostics = %#v", validator.calls, output.Diagnostics)
|
||||||
|
}
|
||||||
|
if output.Diagnostics.Groups[0].ReasonCode != "stored_chunk_diagnostic" || output.Diagnostics.Groups[1].ReasonCode != "current_validator_diagnostic" || output.Diagnostics.Groups[1].Origin.ValidatorKey != validator.Name() {
|
||||||
|
t.Fatalf("diagnostic groups = %#v", output.Diagnostics.Groups)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunnerProvidesAcceptedChunkMapToOutput(t *testing.T) {
|
func TestRunnerProvidesAcceptedChunkMapToOutput(t *testing.T) {
|
||||||
for _, test := range []struct {
|
for _, test := range []struct {
|
||||||
name string
|
name string
|
||||||
@@ -368,7 +399,7 @@ func TestRunnerRegeneratesValidationIncompleteChunkPlanHit(t *testing.T) {
|
|||||||
}},
|
}},
|
||||||
}}
|
}}
|
||||||
record := chunkPlanRecord(t, prepared, plan)
|
record := chunkPlanRecord(t, prepared, plan)
|
||||||
record.Warnings = []contracts.Warning{{Scope: "stored", ReasonCode: "old", Message: "discarded stored warning"}}
|
record.Diagnostics = []contracts.ProducerDiagnostic{producerDiagnostic("stored", "discarded stored diagnostic")}
|
||||||
store := &recordingChunkPlanStore{record: record, decision: ChunkPlanDecision{Status: ChunkPlanHit}}
|
store := &recordingChunkPlanStore{record: record, decision: ChunkPlanDecision{Status: ChunkPlanHit}}
|
||||||
debug := newCapturedDebugRecorder()
|
debug := newCapturedDebugRecorder()
|
||||||
|
|
||||||
@@ -382,8 +413,8 @@ func TestRunnerRegeneratesValidationIncompleteChunkPlanHit(t *testing.T) {
|
|||||||
if output.ChunkPlan == nil || output.ChunkPlan.Action != "generated" || output.ChunkPlan.LookupStatus != "invalid" || output.ChunkPlan.ValidationStatus != "approved" {
|
if output.ChunkPlan == nil || output.ChunkPlan.Action != "generated" || output.ChunkPlan.LookupStatus != "invalid" || output.ChunkPlan.ValidationStatus != "approved" {
|
||||||
t.Fatalf("chunk plan summary = %#v", output.ChunkPlan)
|
t.Fatalf("chunk plan summary = %#v", output.ChunkPlan)
|
||||||
}
|
}
|
||||||
if len(output.Warnings) != 0 {
|
if len(output.Diagnostics.Groups) != 0 {
|
||||||
t.Fatalf("warnings = %#v, want discarded cache warnings omitted", output.Warnings)
|
t.Fatalf("diagnostics = %#v, want discarded cache diagnostics omitted", output.Diagnostics)
|
||||||
}
|
}
|
||||||
assertAttemptEnvelopeSequence(t, debug, "chunk", 1)
|
assertAttemptEnvelopeSequence(t, debug, "chunk", 1)
|
||||||
}
|
}
|
||||||
@@ -396,7 +427,7 @@ func TestRunnerKeepsStoredPlanWhenCacheAndGeneratedValidationAreIncomplete(t *te
|
|||||||
validator := &countingChunkValidator{err: errors.New("validator unavailable")}
|
validator := &countingChunkValidator{err: errors.New("validator unavailable")}
|
||||||
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
|
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
|
||||||
record := chunkPlanRecord(t, prepared, plan)
|
record := chunkPlanRecord(t, prepared, plan)
|
||||||
record.Warnings = []contracts.Warning{{Scope: "stored", ReasonCode: "old", Message: "discarded stored warning"}}
|
record.Diagnostics = []contracts.ProducerDiagnostic{producerDiagnostic("stored", "discarded stored diagnostic")}
|
||||||
store := &recordingChunkPlanStore{record: record, decision: ChunkPlanDecision{Status: ChunkPlanHit}}
|
store := &recordingChunkPlanStore{record: record, decision: ChunkPlanDecision{Status: ChunkPlanHit}}
|
||||||
|
|
||||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: store})
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: store})
|
||||||
@@ -409,8 +440,8 @@ func TestRunnerKeepsStoredPlanWhenCacheAndGeneratedValidationAreIncomplete(t *te
|
|||||||
if output.ChunkPlan == nil || output.ChunkPlan.Action != "generated" || output.ChunkPlan.LookupStatus != "invalid" || output.ChunkPlan.ValidationStatus != "incomplete" {
|
if output.ChunkPlan == nil || output.ChunkPlan.Action != "generated" || output.ChunkPlan.LookupStatus != "invalid" || output.ChunkPlan.ValidationStatus != "incomplete" {
|
||||||
t.Fatalf("chunk plan summary = %#v", output.ChunkPlan)
|
t.Fatalf("chunk plan summary = %#v", output.ChunkPlan)
|
||||||
}
|
}
|
||||||
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != "validator_execution_incomplete" {
|
if len(output.Diagnostics.Groups) != 1 || output.Diagnostics.Groups[0].ReasonCode != "validator_execution_incomplete" {
|
||||||
t.Fatalf("warnings = %#v, want only generated incomplete warning", output.Warnings)
|
t.Fatalf("diagnostics = %#v, want only generated incomplete diagnostic", output.Diagnostics)
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(store.record, record) {
|
if !reflect.DeepEqual(store.record, record) {
|
||||||
t.Fatal("discarded incomplete candidates mutated the stored record")
|
t.Fatal("discarded incomplete candidates mutated the stored record")
|
||||||
@@ -580,8 +611,8 @@ func TestRunnerRetriesBeforePublishingAcceptedPlan(t *testing.T) {
|
|||||||
if chunker.calls != 2 || store.saves != 1 {
|
if chunker.calls != 2 || store.saves != 1 {
|
||||||
t.Fatalf("module calls = %d saves = %d, want 2 and 1", chunker.calls, store.saves)
|
t.Fatalf("module calls = %d saves = %d, want 2 and 1", chunker.calls, store.saves)
|
||||||
}
|
}
|
||||||
if len(output.Warnings) != 1 || output.Warnings[0].Scope != "accepted" || len(store.saved.Warnings) != 1 || store.saved.Warnings[0].Scope != "accepted" {
|
if len(output.Diagnostics.Groups) != 1 || output.Diagnostics.Groups[0].Samples[0].Scope != "accepted" || len(store.saved.Diagnostics) != 1 || store.saved.Diagnostics[0].Samples[0].Scope != "accepted" {
|
||||||
t.Fatalf("output warnings = %#v stored warnings = %#v", output.Warnings, store.saved.Warnings)
|
t.Fatalf("output diagnostics = %#v stored diagnostics = %#v", output.Diagnostics, store.saved.Diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -593,7 +624,7 @@ func TestRunnerAutoHitValidatesOnceWithoutRegenerationOrMutation(t *testing.T) {
|
|||||||
wantError string
|
wantError string
|
||||||
wantReject bool
|
wantReject bool
|
||||||
}{
|
}{
|
||||||
{name: "warning", result: contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{Scope: "current", ReasonCode: "observed", Message: "current warning"}}}},
|
{name: "diagnostic", result: contracts.ValidationResult{Approved: true, Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("current", "current diagnostic")}}},
|
||||||
{name: "rejection", result: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected hit", CorrectionGuidance: "return an acceptable chunk plan"}, wantReject: true},
|
{name: "rejection", result: contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected hit", CorrectionGuidance: "return an acceptable chunk plan"}, wantReject: true},
|
||||||
{name: "error", validatorErr: errors.New("validator failed"), wantError: "validator failed"},
|
{name: "error", validatorErr: errors.New("validator failed"), wantError: "validator failed"},
|
||||||
}
|
}
|
||||||
@@ -606,7 +637,7 @@ func TestRunnerAutoHitValidatesOnceWithoutRegenerationOrMutation(t *testing.T) {
|
|||||||
validator := &countingChunkValidator{result: tc.result, err: tc.validatorErr}
|
validator := &countingChunkValidator{result: tc.result, err: tc.validatorErr}
|
||||||
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
|
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
|
||||||
record := chunkPlanRecord(t, prepared, plan)
|
record := chunkPlanRecord(t, prepared, plan)
|
||||||
record.Warnings = []contracts.Warning{{Scope: "stored", ReasonCode: "observed", Message: "stored warning"}}
|
record.Diagnostics = []contracts.ProducerDiagnostic{producerDiagnostic("stored", "stored diagnostic")}
|
||||||
store := &recordingChunkPlanStore{record: record, decision: ChunkPlanDecision{Status: ChunkPlanHit}}
|
store := &recordingChunkPlanStore{record: record, decision: ChunkPlanDecision{Status: ChunkPlanHit}}
|
||||||
debug := newCapturedDebugRecorder()
|
debug := newCapturedDebugRecorder()
|
||||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: store, Debug: debug})
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheAuto, ChunkPlans: store, Debug: debug})
|
||||||
@@ -642,8 +673,8 @@ func TestRunnerAutoHitValidatesOnceWithoutRegenerationOrMutation(t *testing.T) {
|
|||||||
if tc.wantReject && (len(output.Rejected) != 1 || output.Rejected[0].ReasonCode != "rejected") {
|
if tc.wantReject && (len(output.Rejected) != 1 || output.Rejected[0].ReasonCode != "rejected") {
|
||||||
t.Fatalf("rejected = %#v", output.Rejected)
|
t.Fatalf("rejected = %#v", output.Rejected)
|
||||||
}
|
}
|
||||||
if tc.wantError == "" && !tc.wantReject && len(output.Warnings) != 1+len(tc.result.Warnings) {
|
if tc.wantError == "" && !tc.wantReject && len(output.Diagnostics.Groups) != 1+len(tc.result.Diagnostics) {
|
||||||
t.Fatalf("warnings = %#v, want stored warning once plus current warnings", output.Warnings)
|
t.Fatalf("diagnostics = %#v, want stored diagnostic once plus current diagnostics", output.Diagnostics)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -693,7 +724,7 @@ func TestRunnerPublishesOnlyAcceptedGeneratedPlans(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) {
|
func TestRunnerStoresProducerProvenanceAndDiagnostics(t *testing.T) {
|
||||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||||
prepared.resolved.Chunk.LLMProfile = "chunk-profile"
|
prepared.resolved.Chunk.LLMProfile = "chunk-profile"
|
||||||
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
|
prepared.resolved.ChunkExecutionClass = contracts.ExecutionClassLLMBacked
|
||||||
@@ -703,8 +734,8 @@ func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) {
|
|||||||
"guide": {Items: []contracts.ReferenceItem{{SlotName: "guide", Digest: "sha256:guide", Origin: contracts.ReferenceOrigin{Type: "file", URI: "file:///guide.txt"}, Content: []byte("sensitive")}}},
|
"guide": {Items: []contracts.ReferenceItem{{SlotName: "guide", Digest: "sha256:guide", Origin: contracts.ReferenceOrigin{Type: "file", URI: "file:///guide.txt"}, Content: []byte("sensitive")}}},
|
||||||
}},
|
}},
|
||||||
}
|
}
|
||||||
prepared.chunker = manifestChunker{terminalChunker: terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan, warnings: []contracts.Warning{{Scope: "producer", ReasonCode: "observed", Message: "producer warning"}}}, metadata: map[string]any{"prompt_id": "chunk/prompt"}}
|
prepared.chunker = manifestChunker{terminalChunker: terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan, diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("producer", "producer diagnostic")}}, metadata: map[string]any{"prompt_id": "chunk/prompt"}}
|
||||||
validator := &countingChunkValidator{result: contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{Scope: "validator", ReasonCode: "observed", Message: "validator warning"}}}}
|
validator := &countingChunkValidator{result: contracts.ValidationResult{Approved: true, Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("validator", "validator diagnostic")}}}
|
||||||
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
|
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding(validator.Name()), Target: ValidatorTargetChunk}, chunk: validator}}
|
||||||
store := &recordingChunkPlanStore{}
|
store := &recordingChunkPlanStore{}
|
||||||
if _, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store}); err != nil {
|
if _, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ChunkCacheMode: ChunkCacheRefresh, ChunkPlans: store}); err != nil {
|
||||||
@@ -717,8 +748,8 @@ func TestRunnerStoresProducerProvenanceAndProducerWarnings(t *testing.T) {
|
|||||||
if len(producer.References) != 1 || producer.References[0].Digest != "sha256:guide" {
|
if len(producer.References) != 1 || producer.References[0].Digest != "sha256:guide" {
|
||||||
t.Fatalf("producer references = %#v", producer.References)
|
t.Fatalf("producer references = %#v", producer.References)
|
||||||
}
|
}
|
||||||
if len(store.saved.Warnings) != 1 || store.saved.Warnings[0].Scope != "producer" {
|
if len(store.saved.Diagnostics) != 1 || store.saved.Diagnostics[0].Samples[0].Scope != "producer" {
|
||||||
t.Fatalf("stored warnings = %#v, want only producer warning", store.saved.Warnings)
|
t.Fatalf("stored diagnostics = %#v, want only producer diagnostic", store.saved.Diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -146,8 +146,7 @@ func (e *cancelingOutputEncoder) Encode(context.Context, contracts.OutputRequest
|
|||||||
e.calls.Add(1)
|
e.calls.Add(1)
|
||||||
e.cancel()
|
e.cancel()
|
||||||
return contracts.OutputResult{
|
return contracts.OutputResult{
|
||||||
Files: []contracts.OutputFile{{Name: "result.txt", ContentType: "text/plain", Bytes: []byte("result")}},
|
Files: []contracts.OutputFile{{Name: "result.txt", ContentType: "text/plain", Bytes: []byte("result")}},
|
||||||
Warnings: []contracts.Warning{{ReasonCode: "returned-after-cancel"}},
|
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -202,9 +201,9 @@ func TestRunnerDiscardsOutputReturnedAfterCancellation(t *testing.T) {
|
|||||||
if len(output.OutputFiles) != 0 {
|
if len(output.OutputFiles) != 0 {
|
||||||
t.Fatalf("output files = %#v, want none", output.OutputFiles)
|
t.Fatalf("output files = %#v, want none", output.OutputFiles)
|
||||||
}
|
}
|
||||||
for _, warning := range output.Warnings {
|
for _, group := range output.Diagnostics.Groups {
|
||||||
if warning.ReasonCode == "returned-after-cancel" {
|
if group.ReasonCode == "returned-after-cancel" {
|
||||||
t.Fatalf("output warnings include encoder warning after cancellation")
|
t.Fatalf("output diagnostics include encoder diagnostic after cancellation")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -367,7 +366,8 @@ func TestRunnerBoundsExtractJobsAndStabilizesReverseCompletion(t *testing.T) {
|
|||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return erasedTypedResult{}, ctx.Err()
|
return erasedTypedResult{}, ctx.Err()
|
||||||
}
|
}
|
||||||
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index), Warnings: []contracts.Warning{{Scope: fmt.Sprintf("lane-%d/chunk-%d", lane, request.Chunk.Index), ReasonCode: "observed", Message: "ordered"}}}, nil
|
scope := fmt.Sprintf("lane-%d/chunk-%d", lane, request.Chunk.Index)
|
||||||
|
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index), Diagnostics: []contracts.ProducerDiagnostic{{Disposition: contracts.DiagnosticDispositionObservation, Category: contracts.DiagnosticCategoryNormalization, ReasonCode: "observed", OccurrenceCount: 1, Samples: []contracts.DiagnosticSample{{Scope: scope, Message: "ordered"}}}}}, nil
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -396,13 +396,23 @@ func TestRunnerBoundsExtractJobsAndStabilizesReverseCompletion(t *testing.T) {
|
|||||||
if got := maximum.Load(); got != 2 {
|
if got := maximum.Load(); got != 2 {
|
||||||
t.Fatalf("maximum concurrent extract jobs = %d, want 2", got)
|
t.Fatalf("maximum concurrent extract jobs = %d, want 2", got)
|
||||||
}
|
}
|
||||||
wantScopes := []string{"lane-0/chunk-0", "lane-0/chunk-1", "lane-0/chunk-2", "lane-1/chunk-0", "lane-1/chunk-1", "lane-1/chunk-2"}
|
if len(result.output.Diagnostics.Groups) != 2 {
|
||||||
gotScopes := make([]string, len(result.output.Warnings))
|
t.Fatalf("diagnostic groups = %#v, want one deterministic group per lane", result.output.Diagnostics.Groups)
|
||||||
for i := range result.output.Warnings {
|
|
||||||
gotScopes[i] = result.output.Warnings[i].Scope
|
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(gotScopes, wantScopes) {
|
for lane, group := range result.output.Diagnostics.Groups {
|
||||||
t.Fatalf("warning order = %#v, want %#v", gotScopes, wantScopes)
|
if group.Origin.LaneID != prepared.Steps[0].lanes[lane].resolved.ID {
|
||||||
|
t.Fatalf("group origin = %#v, want configured lane %q", group.Origin, prepared.Steps[0].lanes[lane].resolved.ID)
|
||||||
|
}
|
||||||
|
indexes := make([]int, len(group.Samples))
|
||||||
|
for index, sample := range group.Samples {
|
||||||
|
if sample.ChunkIndex == nil {
|
||||||
|
t.Fatalf("sample = %#v, want chunk index", sample)
|
||||||
|
}
|
||||||
|
indexes[index] = *sample.ChunkIndex
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(indexes, []int{0, 1, 2}) {
|
||||||
|
t.Fatalf("group sample chunk order = %#v, want canonical chunk order", indexes)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ type laneExtractState struct {
|
|||||||
reuseEligible bool
|
reuseEligible bool
|
||||||
values []erasedExtractArtifact
|
values []erasedExtractArtifact
|
||||||
serialized []CheckpointArtifact
|
serialized []CheckpointArtifact
|
||||||
warnings []contracts.Warning
|
diagnostics []contracts.DiagnosticGroup
|
||||||
rejected []contracts.RejectedOutput
|
rejected []contracts.RejectedOutput
|
||||||
incomplete []int
|
incomplete []int
|
||||||
validationSummaries []artifacts.ValidationSummary
|
validationSummaries []artifacts.ValidationSummary
|
||||||
@@ -38,7 +38,7 @@ type laneExtractState struct {
|
|||||||
type finalizedExtractResults struct {
|
type finalizedExtractResults struct {
|
||||||
accepted []erasedExtractArtifact
|
accepted []erasedExtractArtifact
|
||||||
serialized []CheckpointArtifact
|
serialized []CheckpointArtifact
|
||||||
warnings []contracts.Warning
|
diagnostics []contracts.DiagnosticGroup
|
||||||
rejected []contracts.RejectedOutput
|
rejected []contracts.RejectedOutput
|
||||||
incomplete []int
|
incomplete []int
|
||||||
validationSummaries []artifacts.ValidationSummary
|
validationSummaries []artifacts.ValidationSummary
|
||||||
@@ -53,8 +53,8 @@ func loadExtract(loader CheckpointLoader, stepID, laneID, moduleKey string, deps
|
|||||||
return loader.Extract(laneID, moduleKey, deps)
|
return loader.Extract(laneID, moduleKey, deps)
|
||||||
}
|
}
|
||||||
|
|
||||||
func recordExtract(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
func recordExtract(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput) error {
|
||||||
return checkpointExtractSucceeded(recorder, stepID, laneID, moduleKey, deps, outputs, rejected, warnings)
|
return checkpointExtractSucceeded(recorder, stepID, laneID, moduleKey, deps, outputs, rejected)
|
||||||
}
|
}
|
||||||
|
|
||||||
type extractJob struct {
|
type extractJob struct {
|
||||||
@@ -67,7 +67,7 @@ type extractJobResult struct {
|
|||||||
chunkIndex int
|
chunkIndex int
|
||||||
value erasedExtractArtifact
|
value erasedExtractArtifact
|
||||||
serialized CheckpointArtifact
|
serialized CheckpointArtifact
|
||||||
warnings []contracts.Warning
|
diagnostics []contracts.DiagnosticGroup
|
||||||
rejected *contracts.RejectedOutput
|
rejected *contracts.RejectedOutput
|
||||||
validationIncomplete bool
|
validationIncomplete bool
|
||||||
validationSummary *artifacts.ValidationSummary
|
validationSummary *artifacts.ValidationSummary
|
||||||
@@ -353,7 +353,11 @@ func hydrateRequiredLane(input RunInput, loader CheckpointLoader, doc *source.So
|
|||||||
return state, err
|
return state, err
|
||||||
}
|
}
|
||||||
hydrated := resolution.artifacts[0]
|
hydrated := resolution.artifacts[0]
|
||||||
local.Warnings = append(local.Warnings, cloneWarnings(checkpoint.Warnings)...)
|
diagnostics, diagnosticErr := promoteCheckpointDiagnostics(hydrated.Diagnostics, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageNormalize, StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module}, nil)
|
||||||
|
if diagnosticErr != nil {
|
||||||
|
return state, fmt.Errorf("promote reused accepted diagnostics: %w", diagnosticErr)
|
||||||
|
}
|
||||||
|
appendDiagnosticGroups(&local, diagnostics)
|
||||||
local.NormalizeOutputs = append(local.NormalizeOutputs, contracts.SerializedOutput{
|
local.NormalizeOutputs = append(local.NormalizeOutputs, contracts.SerializedOutput{
|
||||||
StepID: input.stepID,
|
StepID: input.stepID,
|
||||||
LaneID: lane.ID,
|
LaneID: lane.ID,
|
||||||
@@ -418,8 +422,14 @@ func prepareLaneExtract(input RunInput, loader CheckpointLoader, doc *source.Sou
|
|||||||
}
|
}
|
||||||
state.values = append(state.values, artifact)
|
state.values = append(state.values, artifact)
|
||||||
state.serialized = append(state.serialized, cloneCheckpointArtifact(stored))
|
state.serialized = append(state.serialized, cloneCheckpointArtifact(stored))
|
||||||
|
chunk := source.Chunk{ID: stored.ChunkID, Index: stored.ChunkIndex}
|
||||||
|
diagnostics, diagnosticErr := promoteCheckpointDiagnostics(stored.Diagnostics, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageExtract, StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module}, &chunk)
|
||||||
|
if diagnosticErr != nil {
|
||||||
|
return nil, fmt.Errorf("promote reused extract diagnostics: %w", diagnosticErr)
|
||||||
|
}
|
||||||
|
state.diagnostics = append(state.diagnostics, diagnostics...)
|
||||||
}
|
}
|
||||||
state.warnings, state.rejected = cloneWarnings(cp.Warnings), cloneRejectedOutputs(cp.Rejected)
|
state.rejected = cloneRejectedOutputs(cp.Rejected)
|
||||||
}
|
}
|
||||||
return state, nil
|
return state, nil
|
||||||
}
|
}
|
||||||
@@ -450,15 +460,14 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
|
|||||||
return producerAttemptOutput{}, terminal.record(nil, attemptErr)
|
return producerAttemptOutput{}, terminal.record(nil, attemptErr)
|
||||||
}
|
}
|
||||||
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: chunk.ID, ChunkIndex: chunk.Index, ChunkRef: chunk.Ref, Value: extracted.Value}
|
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: chunk.ID, ChunkIndex: chunk.Index, ChunkRef: chunk.Ref, Value: extracted.Value}
|
||||||
attemptWarnings := cloneWarnings(extracted.Warnings)
|
|
||||||
serializedCandidate, encodeErr := serializeCandidateArtifact(typed.codec, artifact.LaneID, artifact.ExtractorKey, artifact.SourceID, artifact.Value)
|
serializedCandidate, encodeErr := serializeCandidateArtifact(typed.codec, artifact.LaneID, artifact.ExtractorKey, artifact.SourceID, artifact.Value)
|
||||||
if encodeErr != nil {
|
if encodeErr != nil {
|
||||||
attemptErr := fmt.Errorf("serialize extract candidate for lane %q chunk %q: %w", lane.ID, chunk.ID, encodeErr)
|
attemptErr := fmt.Errorf("serialize extract candidate for lane %q chunk %q: %w", lane.ID, chunk.ID, encodeErr)
|
||||||
payload := map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}
|
payload := map[string]any{}
|
||||||
return producerAttemptOutput{}, terminal.record(payload, attemptErr)
|
return producerAttemptOutput{}, terminal.record(payload, attemptErr)
|
||||||
}
|
}
|
||||||
serializedCandidate.ChunkID, serializedCandidate.ChunkIndex, serializedCandidate.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
|
serializedCandidate.ChunkID, serializedCandidate.ChunkIndex, serializedCandidate.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
|
||||||
return producerAttemptOutput{Value: extractAttemptValue{artifact: artifact, serialized: serializedCandidate, terminal: &terminal}, Candidate: extracted.ModelCandidate, Warnings: attemptWarnings}, nil
|
return producerAttemptOutput{Value: extractAttemptValue{artifact: artifact, serialized: serializedCandidate, terminal: &terminal}, Candidate: extracted.ModelCandidate, Diagnostics: contracts.CloneProducerDiagnostics(extracted.Diagnostics)}, nil
|
||||||
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
|
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||||
candidate, ok := output.Value.(extractAttemptValue)
|
candidate, ok := output.Value.(extractAttemptValue)
|
||||||
if !ok {
|
if !ok {
|
||||||
@@ -467,7 +476,6 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
|
|||||||
report, validationErr := r.validateTypedReport(validationCtx, typed.codec, typedValidationTarget{stage: StageExtract, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: operationReferenceSet(input, lane.ExtractReferences), metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: candidate.artifact.Value, candidate: &candidate.serialized}, state.prepared.extractValidators, candidate.terminal.envelope.Attempt, input.Debug)
|
report, validationErr := r.validateTypedReport(validationCtx, typed.codec, typedValidationTarget{stage: StageExtract, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: operationReferenceSet(input, lane.ExtractReferences), metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: candidate.artifact.Value, candidate: &candidate.serialized}, state.prepared.extractValidators, candidate.terminal.envelope.Attempt, input.Debug)
|
||||||
payload := map[string]any{
|
payload := map[string]any{
|
||||||
"output": debugCheckpointArtifact(candidate.serialized),
|
"output": debugCheckpointArtifact(candidate.serialized),
|
||||||
"warnings": debugWarningEnvelopes(append(cloneWarnings(output.Warnings), report.Warnings()...)),
|
|
||||||
"rejection": debugRejectedOutputPtr(typedRejection(report, typedValidationTarget{stage: StageExtract, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Extract.Module, chunk: &chunk}, candidate.terminal.envelope.Attempt)),
|
"rejection": debugRejectedOutputPtr(typedRejection(report, typedValidationTarget{stage: StageExtract, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Extract.Module, chunk: &chunk}, candidate.terminal.envelope.Attempt)),
|
||||||
}
|
}
|
||||||
if validationErr != nil {
|
if validationErr != nil {
|
||||||
@@ -481,6 +489,14 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
|
|||||||
result.err = errors.Join(result.err, debugErr)
|
result.err = errors.Join(result.err, debugErr)
|
||||||
return result
|
return result
|
||||||
}
|
}
|
||||||
|
if err == nil {
|
||||||
|
diagnostics, diagnosticErr := terminalDiagnosticGroups(terminalResult, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageExtract, StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module}, &chunk)
|
||||||
|
if diagnosticErr != nil {
|
||||||
|
result.err = fmt.Errorf("promote extract diagnostics: %w", diagnosticErr)
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
result.diagnostics = diagnostics
|
||||||
|
}
|
||||||
if err == nil && terminalResult.Action == producerTerminalRejected {
|
if err == nil && terminalResult.Action == producerTerminalRejected {
|
||||||
result.rejected = terminalResult.Rejection
|
result.rejected = terminalResult.Rejection
|
||||||
if result.rejected != nil {
|
if result.rejected != nil {
|
||||||
@@ -488,7 +504,6 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
|
|||||||
result.validationSummary = &summary
|
result.validationSummary = &summary
|
||||||
result.rejected.Validation = cloneValidationSummaryPtr(result.validationSummary)
|
result.rejected.Validation = cloneValidationSummaryPtr(result.validationSummary)
|
||||||
}
|
}
|
||||||
result.warnings = cloneWarnings(terminalResult.Warnings)
|
|
||||||
return result
|
return result
|
||||||
}
|
}
|
||||||
if err == nil {
|
if err == nil {
|
||||||
@@ -498,19 +513,19 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
|
|||||||
return result
|
return result
|
||||||
}
|
}
|
||||||
stored, encodeErr := checkpointArtifact(typed.codec, candidate.artifact.LaneID, candidate.artifact.ExtractorKey, candidate.artifact.SourceID, candidate.artifact.Value)
|
stored, encodeErr := checkpointArtifact(typed.codec, candidate.artifact.LaneID, candidate.artifact.ExtractorKey, candidate.artifact.SourceID, candidate.artifact.Value)
|
||||||
payload := map[string]any{"output": debugCheckpointArtifact(candidate.serialized), "warnings": debugWarningEnvelopes(terminalResult.Warnings), "rejection": debugRejectedOutputPtr(nil)}
|
payload := map[string]any{"output": debugCheckpointArtifact(candidate.serialized), "rejection": debugRejectedOutputPtr(nil)}
|
||||||
if encodeErr != nil {
|
if encodeErr != nil {
|
||||||
attemptErr := fmt.Errorf("serialize accepted extract output for lane %q chunk %q: %w", lane.ID, chunk.ID, encodeErr)
|
attemptErr := fmt.Errorf("serialize accepted extract output for lane %q chunk %q: %w", lane.ID, chunk.ID, encodeErr)
|
||||||
result.err = candidate.terminal.record(payload, attemptErr)
|
result.err = candidate.terminal.record(payload, attemptErr)
|
||||||
return result
|
return result
|
||||||
}
|
}
|
||||||
stored.ChunkID, stored.ChunkIndex, stored.ChunkRef = candidate.artifact.ChunkID, candidate.artifact.ChunkIndex, candidate.artifact.ChunkRef
|
stored.ChunkID, stored.ChunkIndex, stored.ChunkRef = candidate.artifact.ChunkID, candidate.artifact.ChunkIndex, candidate.artifact.ChunkRef
|
||||||
|
stored.Diagnostics = terminalCheckpointDiagnostics(terminalResult)
|
||||||
if debugErr := candidate.terminal.record(payload, nil); debugErr != nil {
|
if debugErr := candidate.terminal.record(payload, nil); debugErr != nil {
|
||||||
result.err = debugErr
|
result.err = debugErr
|
||||||
return result
|
return result
|
||||||
}
|
}
|
||||||
result.value, result.serialized = candidate.artifact, stored
|
result.value, result.serialized = candidate.artifact, stored
|
||||||
result.warnings = cloneWarnings(terminalResult.Warnings)
|
|
||||||
result.validationIncomplete = terminalResult.ValidationIncomplete
|
result.validationIncomplete = terminalResult.ValidationIncomplete
|
||||||
result.validationSummary = &summary
|
result.validationSummary = &summary
|
||||||
}
|
}
|
||||||
@@ -531,12 +546,12 @@ func finalizeLaneExtract(checkpoints CheckpointRecorder, stepID string, state *l
|
|||||||
}
|
}
|
||||||
if result.rejected != nil {
|
if result.rejected != nil {
|
||||||
state.rejected = append(state.rejected, *result.rejected)
|
state.rejected = append(state.rejected, *result.rejected)
|
||||||
state.warnings = append(state.warnings, result.warnings...)
|
state.diagnostics = append(state.diagnostics, result.diagnostics...)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
state.values = append(state.values, result.value)
|
state.values = append(state.values, result.value)
|
||||||
state.serialized = append(state.serialized, result.serialized)
|
state.serialized = append(state.serialized, result.serialized)
|
||||||
state.warnings = append(state.warnings, result.warnings...)
|
state.diagnostics = append(state.diagnostics, result.diagnostics...)
|
||||||
if result.validationIncomplete {
|
if result.validationIncomplete {
|
||||||
state.incomplete = append(state.incomplete, result.chunkIndex)
|
state.incomplete = append(state.incomplete, result.chunkIndex)
|
||||||
}
|
}
|
||||||
@@ -549,7 +564,7 @@ func finalizeLaneExtract(checkpoints CheckpointRecorder, stepID string, state *l
|
|||||||
state.reuseEligible = false
|
state.reuseEligible = false
|
||||||
}
|
}
|
||||||
if !state.decision.Reused && state.reuseEligible {
|
if !state.decision.Reused && state.reuseEligible {
|
||||||
if err := recordExtract(checkpoints, stepID, lane.ID, lane.Extract.Module, state.deps, state.serialized, state.rejected, state.warnings); err != nil {
|
if err := recordExtract(checkpoints, stepID, lane.ID, lane.Extract.Module, state.deps, state.serialized, state.rejected); err != nil {
|
||||||
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
|
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -562,20 +577,20 @@ func (r *Runner) continueLane(ctx context.Context, input RunInput, checkpoints C
|
|||||||
results := finalizedExtractResults{
|
results := finalizedExtractResults{
|
||||||
accepted: state.values,
|
accepted: state.values,
|
||||||
serialized: state.serialized,
|
serialized: state.serialized,
|
||||||
warnings: state.warnings,
|
diagnostics: state.diagnostics,
|
||||||
rejected: state.rejected,
|
rejected: state.rejected,
|
||||||
incomplete: state.incomplete,
|
incomplete: state.incomplete,
|
||||||
validationSummaries: state.validationSummaries,
|
validationSummaries: state.validationSummaries,
|
||||||
decision: state.decision,
|
decision: state.decision,
|
||||||
reuseEligible: state.reuseEligible,
|
reuseEligible: state.reuseEligible,
|
||||||
}
|
}
|
||||||
local.Warnings = append(local.Warnings, cloneWarnings(results.warnings)...)
|
local.diagnosticGroups = append(local.diagnosticGroups, contracts.CloneDiagnosticCollection(contracts.DiagnosticCollection{Groups: results.diagnostics}).Groups...)
|
||||||
local.Rejected = append(local.Rejected, cloneRejectedOutputs(results.rejected)...)
|
local.Rejected = append(local.Rejected, cloneRejectedOutputs(results.rejected)...)
|
||||||
local.ValidationSummaries = append(local.ValidationSummaries, cloneValidationSummaries(results.validationSummaries)...)
|
local.ValidationSummaries = append(local.ValidationSummaries, cloneValidationSummaries(results.validationSummaries)...)
|
||||||
if err := writeDebugTimed(input.Debug, path.Join("extract", fileio.EncodePathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "decision": results.decision, "source": debugSourceDocumentEnvelope(doc), "chunks": debugSourceChunkEnvelopes(chunks), "options": redactSensitiveMap(lane.Extract.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
if err := writeDebugTimed(input.Debug, path.Join("extract", fileio.EncodePathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "decision": results.decision, "source": debugSourceDocumentEnvelope(doc), "chunks": debugSourceChunkEnvelopes(chunks), "options": redactSensitiveMap(lane.Extract.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||||
return local, &laneRunError{stage: StageExtract, err: err}
|
return local, &laneRunError{stage: StageExtract, err: err}
|
||||||
}
|
}
|
||||||
if err := writeDebugTimed(input.Debug, path.Join("extract", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "outputs": debugCheckpointArtifacts(results.serialized), "rejected": debugRejectedOutputEnvelopes(results.rejected), "warnings": debugWarningEnvelopes(results.warnings), "validation_incomplete_chunks": append([]int(nil), results.incomplete...)}}); err != nil {
|
if err := writeDebugTimed(input.Debug, path.Join("extract", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "outputs": debugCheckpointArtifacts(results.serialized), "rejected": debugRejectedOutputEnvelopes(results.rejected), "validation_incomplete_chunks": append([]int(nil), results.incomplete...)}}); err != nil {
|
||||||
return local, &laneRunError{stage: StageExtract, err: err}
|
return local, &laneRunError{stage: StageExtract, err: err}
|
||||||
}
|
}
|
||||||
if len(results.accepted) == 0 {
|
if len(results.accepted) == 0 {
|
||||||
@@ -640,7 +655,7 @@ func mergeLaneOutput(dst *RunOutput, src RunOutput) error {
|
|||||||
}
|
}
|
||||||
dst.NormalizeOutputs = append(dst.NormalizeOutputs, cloneSerializedOutputs(src.NormalizeOutputs)...)
|
dst.NormalizeOutputs = append(dst.NormalizeOutputs, cloneSerializedOutputs(src.NormalizeOutputs)...)
|
||||||
dst.Rejected = append(dst.Rejected, cloneRejectedOutputs(src.Rejected)...)
|
dst.Rejected = append(dst.Rejected, cloneRejectedOutputs(src.Rejected)...)
|
||||||
dst.Warnings = append(dst.Warnings, cloneWarnings(src.Warnings)...)
|
appendDiagnosticGroups(dst, src.diagnosticGroups)
|
||||||
dst.CheckpointEvents = append(dst.CheckpointEvents, src.CheckpointEvents...)
|
dst.CheckpointEvents = append(dst.CheckpointEvents, src.CheckpointEvents...)
|
||||||
dst.ValidationSummaries = append(dst.ValidationSummaries, cloneValidationSummaries(src.ValidationSummaries)...)
|
dst.ValidationSummaries = append(dst.ValidationSummaries, cloneValidationSummaries(src.ValidationSummaries)...)
|
||||||
if len(src.normalizeReuseEligibility) > 0 {
|
if len(src.normalizeReuseEligibility) > 0 {
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"reflect"
|
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -85,15 +84,6 @@ func TestRunnerCorrectsExtractCandidatesIndependentlyPerChunk(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func containsWarnings(have, want []contracts.Warning) bool {
|
|
||||||
for index := 0; index+len(want) <= len(have); index++ {
|
|
||||||
if reflect.DeepEqual(have[index:index+len(want)], want) {
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
func containsValidationSummary(summaries []artifacts.ValidationSummary, status string) bool {
|
func containsValidationSummary(summaries []artifacts.ValidationSummary, status string) bool {
|
||||||
for _, summary := range summaries {
|
for _, summary := range summaries {
|
||||||
if summary.Status == status {
|
if summary.Status == status {
|
||||||
@@ -126,8 +116,8 @@ func TestRunnerContinuesValidationIncompleteExtractWithoutCheckpoint(t *testing.
|
|||||||
if output.Manifest.ValidationStatus != "incomplete" || !containsValidationSummary(output.Manifest.ValidationSummaries, "incomplete") {
|
if output.Manifest.ValidationStatus != "incomplete" || !containsValidationSummary(output.Manifest.ValidationSummaries, "incomplete") {
|
||||||
t.Fatalf("manifest validation = %#v, want incomplete extract provenance", output.Manifest)
|
t.Fatalf("manifest validation = %#v, want incomplete extract provenance", output.Manifest)
|
||||||
}
|
}
|
||||||
if got, want := output.Warnings, []contracts.Warning{{Scope: "typed/check", ReasonCode: "validator_execution_incomplete", Message: "Validator execution did not complete within its configured budget."}}; !containsWarnings(got, want) {
|
if len(output.Diagnostics.Groups) != 1 || output.Diagnostics.Groups[0].ReasonCode != "validator_execution_incomplete" {
|
||||||
t.Fatalf("warnings = %#v, want %#v", got, want)
|
t.Fatalf("diagnostics = %#v, want validation-incomplete group", output.Diagnostics)
|
||||||
}
|
}
|
||||||
if len(recorder.checkpoint.Outputs) != 0 || len(recorder.checkpoint.Rejected) != 0 {
|
if len(recorder.checkpoint.Outputs) != 0 || len(recorder.checkpoint.Rejected) != 0 {
|
||||||
t.Fatalf("extract checkpoint = %#v, want no persisted incomplete output", recorder.checkpoint)
|
t.Fatalf("extract checkpoint = %#v, want no persisted incomplete output", recorder.checkpoint)
|
||||||
|
|||||||
@@ -35,11 +35,10 @@ func (v *mutatingMetadataValidator) Validate(_ context.Context, request contract
|
|||||||
return contracts.ValidationResult{Approved: true}, nil
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *extractCaptureRecorder) ExtractSucceeded(_ string, _ string, _ []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
func (r *extractCaptureRecorder) ExtractSucceeded(_ string, _ string, _ []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput) error {
|
||||||
r.checkpoint = ExtractCheckpoint{
|
r.checkpoint = ExtractCheckpoint{
|
||||||
Outputs: cloneCheckpointArtifacts(outputs),
|
Outputs: cloneCheckpointArtifacts(outputs),
|
||||||
Rejected: cloneRejectedOutputs(rejected),
|
Rejected: cloneRejectedOutputs(rejected),
|
||||||
Warnings: cloneWarnings(warnings),
|
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -56,7 +55,6 @@ func (l *extractResultLoader) Extract(string, string, []CheckpointFingerprint) (
|
|||||||
return ExtractCheckpoint{
|
return ExtractCheckpoint{
|
||||||
Outputs: cloneCheckpointArtifacts(l.checkpoint.Outputs),
|
Outputs: cloneCheckpointArtifacts(l.checkpoint.Outputs),
|
||||||
Rejected: cloneRejectedOutputs(l.checkpoint.Rejected),
|
Rejected: cloneRejectedOutputs(l.checkpoint.Rejected),
|
||||||
Warnings: cloneWarnings(l.checkpoint.Warnings),
|
|
||||||
}, l.decision
|
}, l.decision
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -156,8 +154,14 @@ func TestRunnerContinuesFromFreshAndReusedExtractResults(t *testing.T) {
|
|||||||
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
extractCalls++
|
extractCalls++
|
||||||
return erasedTypedResult{
|
return erasedTypedResult{
|
||||||
Value: typedValueForLane(0, request.Chunk.Index),
|
Value: typedValueForLane(0, request.Chunk.Index),
|
||||||
Warnings: []contracts.Warning{{Scope: "extract", ReasonCode: "observed", Message: "accepted extract"}},
|
Diagnostics: []contracts.ProducerDiagnostic{{
|
||||||
|
Disposition: contracts.DiagnosticDispositionObservation,
|
||||||
|
Category: contracts.DiagnosticCategoryNormalization,
|
||||||
|
ReasonCode: "accepted_extract_normalized",
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []contracts.DiagnosticSample{{Scope: "extract", Message: "accepted extract"}},
|
||||||
|
}},
|
||||||
}, nil
|
}, nil
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -206,12 +210,12 @@ func TestRunnerContinuesFromFreshAndReusedExtractResults(t *testing.T) {
|
|||||||
if !reflect.DeepEqual(reused.NormalizeOutputs, fresh.NormalizeOutputs) {
|
if !reflect.DeepEqual(reused.NormalizeOutputs, fresh.NormalizeOutputs) {
|
||||||
t.Fatalf("reused normalize outputs = %#v, want fresh outputs %#v", reused.NormalizeOutputs, fresh.NormalizeOutputs)
|
t.Fatalf("reused normalize outputs = %#v, want fresh outputs %#v", reused.NormalizeOutputs, fresh.NormalizeOutputs)
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(reused.Warnings, fresh.Warnings) {
|
if !reflect.DeepEqual(reused.Diagnostics, fresh.Diagnostics) {
|
||||||
t.Fatalf("reused warnings = %#v, want fresh warnings %#v", reused.Warnings, fresh.Warnings)
|
t.Fatalf("reused diagnostics = %#v, want fresh diagnostics %#v", reused.Diagnostics, fresh.Diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunnerPromotesOnlyAcceptedExtractRetryWarnings(t *testing.T) {
|
func TestRunnerPromotesOnlyAcceptedExtractRetryDiagnostics(t *testing.T) {
|
||||||
prepared := preparedAttemptDebugPipeline(t)
|
prepared := preparedAttemptDebugPipeline(t)
|
||||||
prepared.Steps[0].lanes[0].resolved.Extract.Retries = 1
|
prepared.Steps[0].lanes[0].resolved.Extract.Retries = 1
|
||||||
attempts := 0
|
attempts := 0
|
||||||
@@ -223,7 +227,7 @@ func TestRunnerPromotesOnlyAcceptedExtractRetryWarnings(t *testing.T) {
|
|||||||
}
|
}
|
||||||
return erasedTypedResult{
|
return erasedTypedResult{
|
||||||
Value: typedValueForLane(0, request.Chunk.Index),
|
Value: typedValueForLane(0, request.Chunk.Index),
|
||||||
Warnings: []contracts.Warning{{Scope: scope, ReasonCode: "observed", Message: scope}},
|
Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic(scope, scope)},
|
||||||
ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%s"]}`, scope)),
|
ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%s"]}`, scope)),
|
||||||
}, nil
|
}, nil
|
||||||
})
|
})
|
||||||
@@ -240,8 +244,8 @@ func TestRunnerPromotesOnlyAcceptedExtractRetryWarnings(t *testing.T) {
|
|||||||
if attempts != 2 {
|
if attempts != 2 {
|
||||||
t.Fatalf("extract attempts = %d, want 2", attempts)
|
t.Fatalf("extract attempts = %d, want 2", attempts)
|
||||||
}
|
}
|
||||||
if len(output.Warnings) != 1 || output.Warnings[0].Scope != "accepted" {
|
if len(output.Diagnostics.Groups) != 1 || output.Diagnostics.Groups[0].Samples[0].Scope != "accepted" {
|
||||||
t.Fatalf("promoted warnings = %#v, want accepted attempt only", output.Warnings)
|
t.Fatalf("promoted diagnostics = %#v, want accepted attempt only", output.Diagnostics)
|
||||||
}
|
}
|
||||||
assertAttemptEnvelopeSequence(t, debug, "extract/notes/chunk-000001", 1, 2)
|
assertAttemptEnvelopeSequence(t, debug, "extract/notes/chunk-000001", 1, 2)
|
||||||
first := debug.envelope(t, "extract/notes/chunk-000001/attempt-01.json")
|
first := debug.envelope(t, "extract/notes/chunk-000001/attempt-01.json")
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ func TestRunnerHandlesRetryableNormalizeFallbacks(t *testing.T) {
|
|||||||
validator *preparedValidator
|
validator *preparedValidator
|
||||||
wantCalls int
|
wantCalls int
|
||||||
wantItem string
|
wantItem string
|
||||||
wantWarnings []string
|
wantMessages []string
|
||||||
wantRejected int
|
wantRejected int
|
||||||
wantDebug []string
|
wantDebug []string
|
||||||
wantCheckpoint int
|
wantCheckpoint int
|
||||||
@@ -30,7 +30,7 @@ func TestRunnerHandlesRetryableNormalizeFallbacks(t *testing.T) {
|
|||||||
},
|
},
|
||||||
wantCalls: 1,
|
wantCalls: 1,
|
||||||
wantItem: "fallback",
|
wantItem: "fallback",
|
||||||
wantWarnings: []string{"ordinary", "fallback-warning"},
|
wantMessages: []string{"ordinary", "fallback-warning"},
|
||||||
wantDebug: []string{`"another_attempt":false`, `"fallback_accepted":true`},
|
wantDebug: []string{`"another_attempt":false`, `"fallback_accepted":true`},
|
||||||
wantCheckpoint: 1,
|
wantCheckpoint: 1,
|
||||||
},
|
},
|
||||||
@@ -41,11 +41,11 @@ func TestRunnerHandlesRetryableNormalizeFallbacks(t *testing.T) {
|
|||||||
if attempt == 1 {
|
if attempt == 1 {
|
||||||
return retryableNormalizeResult("discarded", "discarded-ordinary", "discarded-fallback")
|
return retryableNormalizeResult("discarded", "discarded-ordinary", "discarded-fallback")
|
||||||
}
|
}
|
||||||
return erasedTypedResult{Value: codecNotes{Items: []string{"accepted"}}, Warnings: []contracts.Warning{{Scope: "accepted", ReasonCode: "ordinary", Message: "accepted-warning"}}}
|
return erasedTypedResult{Value: codecNotes{Items: []string{"accepted"}}, Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("accepted", "accepted-warning")}}
|
||||||
},
|
},
|
||||||
wantCalls: 2,
|
wantCalls: 2,
|
||||||
wantItem: "accepted",
|
wantItem: "accepted",
|
||||||
wantWarnings: []string{"accepted-warning"},
|
wantMessages: []string{"accepted-warning"},
|
||||||
wantDebug: []string{`"another_attempt":true`, `"fallback_accepted":false`},
|
wantDebug: []string{`"another_attempt":true`, `"fallback_accepted":false`},
|
||||||
wantCheckpoint: 1,
|
wantCheckpoint: 1,
|
||||||
},
|
},
|
||||||
@@ -57,7 +57,7 @@ func TestRunnerHandlesRetryableNormalizeFallbacks(t *testing.T) {
|
|||||||
},
|
},
|
||||||
wantCalls: 2,
|
wantCalls: 2,
|
||||||
wantItem: "fallback-2",
|
wantItem: "fallback-2",
|
||||||
wantWarnings: []string{"ordinary-2", "fallback-warning-2"},
|
wantMessages: []string{"ordinary-2", "fallback-warning-2"},
|
||||||
wantDebug: []string{`"another_attempt":false`, `"fallback_accepted":true`},
|
wantDebug: []string{`"another_attempt":false`, `"fallback_accepted":true`},
|
||||||
wantCheckpoint: 1,
|
wantCheckpoint: 1,
|
||||||
},
|
},
|
||||||
@@ -140,12 +140,14 @@ func TestRunnerHandlesRetryableNormalizeFallbacks(t *testing.T) {
|
|||||||
if got := firstNote(normalized); got != tc.wantItem {
|
if got := firstNote(normalized); got != tc.wantItem {
|
||||||
t.Fatalf("normalized item = %q, want %q", got, tc.wantItem)
|
t.Fatalf("normalized item = %q, want %q", got, tc.wantItem)
|
||||||
}
|
}
|
||||||
gotWarnings := make([]string, len(output.Warnings))
|
var gotMessages []string
|
||||||
for index, warning := range output.Warnings {
|
for _, group := range output.Diagnostics.Groups {
|
||||||
gotWarnings[index] = warning.Message
|
for _, sample := range group.Samples {
|
||||||
|
gotMessages = append(gotMessages, sample.Message)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if strings.Join(gotWarnings, "|") != strings.Join(tc.wantWarnings, "|") {
|
if strings.Join(gotMessages, "|") != strings.Join(tc.wantMessages, "|") {
|
||||||
t.Fatalf("durable warnings = %#v, want %#v", gotWarnings, tc.wantWarnings)
|
t.Fatalf("durable diagnostics = %#v, want %#v", gotMessages, tc.wantMessages)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -262,12 +264,12 @@ func TestRunnerValidatesNormalizeRetryDiagnostics(t *testing.T) {
|
|||||||
|
|
||||||
func retryableNormalizeResult(item, ordinary, fallback string) erasedTypedResult {
|
func retryableNormalizeResult(item, ordinary, fallback string) erasedTypedResult {
|
||||||
return erasedTypedResult{
|
return erasedTypedResult{
|
||||||
Value: codecNotes{Items: []string{item}},
|
Value: codecNotes{Items: []string{item}},
|
||||||
Warnings: []contracts.Warning{{Scope: "attempt", ReasonCode: "ordinary", Message: ordinary}},
|
Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("attempt", ordinary)},
|
||||||
Retry: &contracts.NormalizeRetry{
|
Retry: &contracts.NormalizeRetry{
|
||||||
ReasonCode: "retryable_normalization",
|
ReasonCode: "retryable_normalization",
|
||||||
Message: "safe fallback is available",
|
Message: "safe fallback is available",
|
||||||
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "fallback", Message: fallback}},
|
FallbackDiagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("fallback", fallback)},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -11,19 +11,19 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
)
|
)
|
||||||
|
|
||||||
type warningChunker struct {
|
type diagnosticChunker struct {
|
||||||
key string
|
key string
|
||||||
plan source.ChunkPlan
|
plan source.ChunkPlan
|
||||||
calls int
|
calls int
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *warningChunker) Key() string { return c.key }
|
func (c *diagnosticChunker) Key() string { return c.key }
|
||||||
func (*warningChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
func (*diagnosticChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||||
func (c *warningChunker) Plan(context.Context, contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
func (c *diagnosticChunker) Plan(context.Context, contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
|
||||||
c.calls++
|
c.calls++
|
||||||
return contracts.ChunkPlanResult{
|
return contracts.ChunkPlanResult{
|
||||||
Plan: source.CloneChunkPlan(c.plan),
|
Plan: source.CloneChunkPlan(c.plan),
|
||||||
Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", c.calls), ReasonCode: "operation", Message: "operation warning"}},
|
Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic(fmt.Sprintf("operation-%d", c.calls), "operation diagnostic")},
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -40,14 +40,14 @@ func (v chunkValidationFunc) Validate(ctx context.Context, request contracts.Chu
|
|||||||
return v.validate(ctx, request)
|
return v.validate(ctx, request)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunnerPromotesOnlyTerminalRejectionWarnings(t *testing.T) {
|
func TestRunnerPromotesOnlyTerminalRejectionDiagnostics(t *testing.T) {
|
||||||
for _, target := range []ModuleStage{StageChunk, StageExtract, StageMerge, StageNormalize} {
|
for _, target := range []ModuleStage{StageChunk, StageExtract, StageMerge, StageNormalize} {
|
||||||
t.Run(string(target), func(t *testing.T) {
|
t.Run(string(target), func(t *testing.T) {
|
||||||
prepared := preparedAttemptDebugPipeline(t)
|
prepared := preparedAttemptDebugPipeline(t)
|
||||||
lane := &prepared.Steps[0].lanes[0]
|
lane := &prepared.Steps[0].lanes[0]
|
||||||
attempts := 0
|
attempts := 0
|
||||||
first := func() contracts.ValidationResult {
|
first := func() contracts.ValidationResult {
|
||||||
return contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("validator-%d", attempts), ReasonCode: "validator", Message: "validator warning"}}}
|
return contracts.ValidationResult{Approved: true, Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic(fmt.Sprintf("validator-%d", attempts), "validator diagnostic")}}
|
||||||
}
|
}
|
||||||
reject := func() contracts.ValidationResult {
|
reject := func() contracts.ValidationResult {
|
||||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected", CorrectionGuidance: "return an acceptable candidate"}
|
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "rejected", CorrectionGuidance: "return an acceptable candidate"}
|
||||||
@@ -59,7 +59,7 @@ func TestRunnerPromotesOnlyTerminalRejectionWarnings(t *testing.T) {
|
|||||||
case StageChunk:
|
case StageChunk:
|
||||||
prepared.resolved.ChunkValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
prepared.resolved.ChunkValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||||
chunker := prepared.chunker.(*typedTestChunker)
|
chunker := prepared.chunker.(*typedTestChunker)
|
||||||
prepared.chunker = &warningChunker{key: prepared.resolved.Chunk.Module, plan: source.CloneChunkPlan(chunker.plan)}
|
prepared.chunker = &diagnosticChunker{key: prepared.resolved.Chunk.Module, plan: source.CloneChunkPlan(chunker.plan)}
|
||||||
prepared.resolved.Chunk.Retries = 1
|
prepared.resolved.Chunk.Retries = 1
|
||||||
prepared.chunkValidators.validators = []preparedValidator{
|
prepared.chunkValidators.validators = []preparedValidator{
|
||||||
{resolved: ResolvedValidator{Binding: Binding("warning-approval"), Target: ValidatorTargetChunk}, chunk: chunkValidationFunc{name: "warning-approval", validate: func(context.Context, contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
|
{resolved: ResolvedValidator{Binding: Binding("warning-approval"), Target: ValidatorTargetChunk}, chunk: chunkValidationFunc{name: "warning-approval", validate: func(context.Context, contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
|
||||||
@@ -69,34 +69,34 @@ func TestRunnerPromotesOnlyTerminalRejectionWarnings(t *testing.T) {
|
|||||||
return reject(), nil
|
return reject(), nil
|
||||||
}}},
|
}}},
|
||||||
}
|
}
|
||||||
chunkerWithWarnings := prepared.chunker.(*warningChunker)
|
chunkerWithDiagnostics := prepared.chunker.(*diagnosticChunker)
|
||||||
first = func() contracts.ValidationResult {
|
first = func() contracts.ValidationResult {
|
||||||
return contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("validator-%d", chunkerWithWarnings.calls), ReasonCode: "validator", Message: "validator warning"}}}
|
return contracts.ValidationResult{Approved: true, Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic(fmt.Sprintf("validator-%d", chunkerWithDiagnostics.calls), "validator diagnostic")}}
|
||||||
}
|
}
|
||||||
case StageExtract:
|
case StageExtract:
|
||||||
lane.resolved.Extract.Retries = 1
|
lane.resolved.Extract.Retries = 1
|
||||||
lane.resolved.ExtractValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
lane.resolved.ExtractValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||||
installExtractOperation(prepared, 0, func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
installExtractOperation(prepared, 0, func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||||
attempts++
|
attempts++
|
||||||
return erasedTypedResult{Value: codecNotes{Items: []string{"extract"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%d"]}`, attempts))}, nil
|
return erasedTypedResult{Value: codecNotes{Items: []string{"extract"}}, Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic(fmt.Sprintf("operation-%d", attempts), "operation diagnostic")}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%d"]}`, attempts))}, nil
|
||||||
})
|
})
|
||||||
lane.extractValidators.validators = rejectionWarningTypedValidators(first, reject)
|
lane.extractValidators.validators = rejectionDiagnosticTypedValidators(first, reject)
|
||||||
case StageMerge:
|
case StageMerge:
|
||||||
lane.resolved.Merge.Retries = 1
|
lane.resolved.Merge.Retries = 1
|
||||||
lane.resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
lane.resolved.MergeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||||
lane.typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
lane.typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||||
attempts++
|
attempts++
|
||||||
return erasedTypedResult{Value: codecNotes{Items: []string{"merge"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%d"]}`, attempts))}, nil
|
return erasedTypedResult{Value: codecNotes{Items: []string{"merge"}}, Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic(fmt.Sprintf("operation-%d", attempts), "operation diagnostic")}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%d"]}`, attempts))}, nil
|
||||||
}
|
}
|
||||||
lane.mergeValidators.validators = rejectionWarningTypedValidators(first, reject)
|
lane.mergeValidators.validators = rejectionDiagnosticTypedValidators(first, reject)
|
||||||
case StageNormalize:
|
case StageNormalize:
|
||||||
lane.resolved.Normalize.Retries = 1
|
lane.resolved.Normalize.Retries = 1
|
||||||
lane.resolved.NormalizeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
lane.resolved.NormalizeValidationPolicy.SemanticRejection = SemanticRejectionRejectOutput
|
||||||
lane.typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
lane.typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||||
attempts++
|
attempts++
|
||||||
return erasedTypedResult{Value: codecNotes{Items: []string{"normalize"}}, Warnings: []contracts.Warning{{Scope: fmt.Sprintf("operation-%d", attempts), ReasonCode: "operation", Message: "operation warning"}}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%d"]}`, attempts))}, nil
|
return erasedTypedResult{Value: codecNotes{Items: []string{"normalize"}}, Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic(fmt.Sprintf("operation-%d", attempts), "operation diagnostic")}, ModelCandidate: attemptCandidate(t, fmt.Sprintf(`{"items":["%d"]}`, attempts))}, nil
|
||||||
}
|
}
|
||||||
lane.normalizeValidators.validators = rejectionWarningTypedValidators(first, reject)
|
lane.normalizeValidators.validators = rejectionDiagnosticTypedValidators(first, reject)
|
||||||
}
|
}
|
||||||
|
|
||||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoints: recorder, Debug: debug})
|
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoints: recorder, Debug: debug})
|
||||||
@@ -109,29 +109,26 @@ func TestRunnerPromotesOnlyTerminalRejectionWarnings(t *testing.T) {
|
|||||||
wantScopes = []string{"operation-1", "validator-1"}
|
wantScopes = []string{"operation-1", "validator-1"}
|
||||||
wantAttempts = 1
|
wantAttempts = 1
|
||||||
}
|
}
|
||||||
if got := rejectionWarningScopes(output.Warnings); !reflect.DeepEqual(got, wantScopes) {
|
if got := rejectionDiagnosticScopes(output.Diagnostics.Groups); !reflect.DeepEqual(got, wantScopes) {
|
||||||
t.Fatalf("published warning scopes = %#v, want %#v", got, wantScopes)
|
t.Fatalf("published diagnostic scopes = %#v, want %#v", got, wantScopes)
|
||||||
}
|
}
|
||||||
if len(output.Rejected) != 1 || output.Rejected[0].AttemptCount != wantAttempts {
|
if len(output.Rejected) != 1 || output.Rejected[0].AttemptCount != wantAttempts {
|
||||||
t.Fatalf("rejections = %#v, want terminal rejection after %d attempt(s)", output.Rejected, wantAttempts)
|
t.Fatalf("rejections = %#v, want terminal rejection after %d attempt(s)", output.Rejected, wantAttempts)
|
||||||
}
|
}
|
||||||
if target == StageExtract && !reflect.DeepEqual(rejectionWarningScopes(recorder.checkpoint.Warnings), wantScopes) {
|
|
||||||
t.Fatalf("extract checkpoint warnings = %#v, want %#v", recorder.checkpoint.Warnings, wantScopes)
|
|
||||||
}
|
|
||||||
attemptPath := fmt.Sprintf("%s/notes/attempt-01.json", target)
|
attemptPath := fmt.Sprintf("%s/notes/attempt-01.json", target)
|
||||||
if target == StageChunk {
|
if target == StageChunk {
|
||||||
attemptPath = "chunk/attempt-01.json"
|
attemptPath = "chunk/attempt-01.json"
|
||||||
} else if target == StageExtract {
|
} else if target == StageExtract {
|
||||||
attemptPath = "extract/notes/chunk-000001/attempt-01.json"
|
attemptPath = "extract/notes/chunk-000001/attempt-01.json"
|
||||||
}
|
}
|
||||||
if !strings.Contains(string(debug.json[attemptPath]), "operation-1") || !strings.Contains(string(debug.json[attemptPath]), "validator-1") {
|
if !strings.Contains(string(debug.json[attemptPath]), "rejection") {
|
||||||
t.Fatalf("first attempt debug = %s, want discarded warnings", debug.json[attemptPath])
|
t.Fatalf("first attempt debug = %s, want rejection", debug.json[attemptPath])
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func rejectionWarningTypedValidators(first func() contracts.ValidationResult, reject func() contracts.ValidationResult) []preparedValidator {
|
func rejectionDiagnosticTypedValidators(first func() contracts.ValidationResult, reject func() contracts.ValidationResult) []preparedValidator {
|
||||||
return []preparedValidator{
|
return []preparedValidator{
|
||||||
{resolved: ResolvedValidator{Binding: Binding("warning-approval"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"}, typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
{resolved: ResolvedValidator{Binding: Binding("warning-approval"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"}, typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||||
return first(), nil
|
return first(), nil
|
||||||
@@ -142,10 +139,10 @@ func rejectionWarningTypedValidators(first func() contracts.ValidationResult, re
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func rejectionWarningScopes(warnings []contracts.Warning) []string {
|
func rejectionDiagnosticScopes(groups []contracts.DiagnosticGroup) []string {
|
||||||
scopes := make([]string, len(warnings))
|
scopes := make([]string, len(groups))
|
||||||
for index := range warnings {
|
for index := range groups {
|
||||||
scopes[index] = warnings[index].Scope
|
scopes[index] = groups[index].Samples[0].Scope
|
||||||
}
|
}
|
||||||
return scopes
|
return scopes
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -61,7 +61,7 @@ func (s *reuseLineageCheckpointSpy) ExtractRunning(laneID, _ string, _ []Checkpo
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *reuseLineageCheckpointSpy) ExtractSucceeded(laneID, _ string, _ []CheckpointFingerprint, _ []CheckpointArtifact, _ []contracts.RejectedOutput, _ []contracts.Warning) error {
|
func (s *reuseLineageCheckpointSpy) ExtractSucceeded(laneID, _ string, _ []CheckpointFingerprint, _ []CheckpointArtifact, _ []contracts.RejectedOutput) error {
|
||||||
s.write("extract", "succeeded", laneID)
|
s.write("extract", "succeeded", laneID)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -76,7 +76,7 @@ func (s *reuseLineageCheckpointSpy) MergeRunning(laneID, _ string, _ []Checkpoin
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *reuseLineageCheckpointSpy) MergeSucceeded(laneID, _ string, _ []CheckpointFingerprint, _ CheckpointArtifact, _ []contracts.Warning) error {
|
func (s *reuseLineageCheckpointSpy) MergeSucceeded(laneID, _ string, _ []CheckpointFingerprint, _ CheckpointArtifact) error {
|
||||||
s.write("merge", "succeeded", laneID)
|
s.write("merge", "succeeded", laneID)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -96,7 +96,7 @@ func (s *reuseLineageCheckpointSpy) NormalizeRunning(laneID, _ string, _ []Check
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *reuseLineageCheckpointSpy) NormalizeSucceeded(laneID, _ string, _ []CheckpointFingerprint, _ CheckpointArtifact, _ []contracts.Warning) error {
|
func (s *reuseLineageCheckpointSpy) NormalizeSucceeded(laneID, _ string, _ []CheckpointFingerprint, _ CheckpointArtifact) error {
|
||||||
s.write("normalize", "succeeded", laneID)
|
s.write("normalize", "succeeded", laneID)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -120,7 +120,7 @@ func TestRunnerContinuesNormalizeAfterValidatorFailure(t *testing.T) {
|
|||||||
if output.Manifest.ValidationStatus != "incomplete" || !containsValidationSummary(output.Manifest.ValidationSummaries, "incomplete") {
|
if output.Manifest.ValidationStatus != "incomplete" || !containsValidationSummary(output.Manifest.ValidationSummaries, "incomplete") {
|
||||||
t.Fatalf("manifest validation = %#v, want incomplete normalization provenance", output.Manifest)
|
t.Fatalf("manifest validation = %#v, want incomplete normalization provenance", output.Manifest)
|
||||||
}
|
}
|
||||||
if got, want := output.Warnings, []contracts.Warning{{Scope: "unavailable", ReasonCode: "validator_execution_incomplete", Message: "Validator execution did not complete within its configured budget."}}; !containsWarnings(got, want) {
|
if len(output.Diagnostics.Groups) != 1 || output.Diagnostics.Groups[0].ReasonCode != "validator_execution_incomplete" {
|
||||||
t.Fatalf("warnings = %#v, want %#v", got, want)
|
t.Fatalf("diagnostics = %#v, want validation-incomplete group", output.Diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,11 +14,11 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type terminalChunker struct {
|
type terminalChunker struct {
|
||||||
key string
|
key string
|
||||||
plan source.ChunkPlan
|
plan source.ChunkPlan
|
||||||
warnings []contracts.Warning
|
diagnostics []contracts.ProducerDiagnostic
|
||||||
err error
|
err error
|
||||||
calls *int
|
calls *int
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c terminalChunker) Key() string { return c.key }
|
func (c terminalChunker) Key() string { return c.key }
|
||||||
@@ -29,7 +29,7 @@ func (c terminalChunker) Plan(context.Context, contracts.ChunkRequest) (contract
|
|||||||
if c.calls != nil {
|
if c.calls != nil {
|
||||||
(*c.calls)++
|
(*c.calls)++
|
||||||
}
|
}
|
||||||
return contracts.ChunkPlanResult{Plan: source.CloneChunkPlan(c.plan), Warnings: cloneWarnings(c.warnings)}, c.err
|
return contracts.ChunkPlanResult{Plan: source.CloneChunkPlan(c.plan), Diagnostics: contracts.CloneProducerDiagnostics(c.diagnostics)}, c.err
|
||||||
}
|
}
|
||||||
|
|
||||||
type terminalChunkValidator struct {
|
type terminalChunkValidator struct {
|
||||||
@@ -167,7 +167,7 @@ func TestRunnerRecordsChunkTerminalOutcomes(t *testing.T) {
|
|||||||
for _, tc := range tests {
|
for _, tc := range tests {
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
prepared, plan := preparedTerminalDebugPipeline(t)
|
prepared, plan := preparedTerminalDebugPipeline(t)
|
||||||
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan, warnings: []contracts.Warning{{Scope: "chunk", ReasonCode: "observed", Message: "chunk warning"}}, err: tc.moduleError}
|
prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, plan: plan, diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("chunk", "chunk diagnostic")}, err: tc.moduleError}
|
||||||
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding("terminal/chunk-validator"), Target: ValidatorTargetChunk}, chunk: tc.validator}}
|
prepared.chunkValidators.validators = []preparedValidator{{resolved: ResolvedValidator{Binding: Binding("terminal/chunk-validator"), Target: ValidatorTargetChunk}, chunk: tc.validator}}
|
||||||
debug := newCapturedDebugRecorder()
|
debug := newCapturedDebugRecorder()
|
||||||
|
|
||||||
@@ -336,7 +336,7 @@ func TestRunnerRecordsExtractTerminalOutcomes(t *testing.T) {
|
|||||||
if tc.moduleError != nil {
|
if tc.moduleError != nil {
|
||||||
return erasedTypedResult{}, tc.moduleError
|
return erasedTypedResult{}, tc.moduleError
|
||||||
}
|
}
|
||||||
return erasedTypedResult{Value: codecNotes{Items: []string{value}}, Warnings: []contracts.Warning{{Scope: "extract", ReasonCode: "observed", Message: "extract warning"}}}, nil
|
return erasedTypedResult{Value: codecNotes{Items: []string{value}}, Diagnostics: []contracts.ProducerDiagnostic{producerDiagnostic("extract", "extract diagnostic")}}, nil
|
||||||
})
|
})
|
||||||
prepared.Steps[0].lanes[0].extractValidators.validators = []preparedValidator{{
|
prepared.Steps[0].lanes[0].extractValidators.validators = []preparedValidator{{
|
||||||
resolved: ResolvedValidator{Binding: Binding("terminal/extract-validator"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
resolved: ResolvedValidator{Binding: Binding("terminal/extract-validator"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||||
@@ -368,9 +368,6 @@ func TestRunnerRecordsExtractTerminalOutcomes(t *testing.T) {
|
|||||||
t.Fatalf("extract rejection = envelope %#v, outputs %#v", envelope, output.Rejected)
|
t.Fatalf("extract rejection = envelope %#v, outputs %#v", envelope, output.Rejected)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if tc.moduleError == nil && !strings.Contains(string(debug.json[attemptPath]), "extract warning") {
|
|
||||||
t.Fatalf("attempt envelope = %s, want attempt warning", debug.json[attemptPath])
|
|
||||||
}
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -29,14 +29,15 @@ func loadNormalize(loader CheckpointLoader, stepID, laneID, moduleKey string, de
|
|||||||
}
|
}
|
||||||
return loader.Normalize(laneID, moduleKey, deps)
|
return loader.Normalize(laneID, moduleKey, deps)
|
||||||
}
|
}
|
||||||
func recordMerge(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
func recordMerge(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact) error {
|
||||||
return checkpointMergeSucceeded(recorder, stepID, laneID, moduleKey, deps, output, warnings)
|
return checkpointMergeSucceeded(recorder, stepID, laneID, moduleKey, deps, output)
|
||||||
}
|
}
|
||||||
func recordNormalize(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
func recordNormalize(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact) error {
|
||||||
return checkpointNormalizeSucceeded(recorder, stepID, laneID, moduleKey, deps, output, warnings)
|
return checkpointNormalizeSucceeded(recorder, stepID, laneID, moduleKey, deps, output)
|
||||||
}
|
}
|
||||||
func cloneCheckpointArtifact(output CheckpointArtifact) CheckpointArtifact {
|
func cloneCheckpointArtifact(output CheckpointArtifact) CheckpointArtifact {
|
||||||
output.Artifact = contracts.CloneSerializedArtifact(output.Artifact)
|
output.Artifact = contracts.CloneSerializedArtifact(output.Artifact)
|
||||||
|
output.Diagnostics = cloneCheckpointDiagnostics(output.Diagnostics)
|
||||||
return output
|
return output
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -264,12 +265,14 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
|
|||||||
}
|
}
|
||||||
var merged erasedMergeArtifact
|
var merged erasedMergeArtifact
|
||||||
var serializedMerge CheckpointArtifact
|
var serializedMerge CheckpointArtifact
|
||||||
var mergeWarnings []contracts.Warning
|
|
||||||
if mergeDecision.Reused {
|
if mergeDecision.Reused {
|
||||||
merged = erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: mergeResolution.values[0]}
|
merged = erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: mergeResolution.values[0]}
|
||||||
serializedMerge = mergeResolution.artifacts[0]
|
serializedMerge = mergeResolution.artifacts[0]
|
||||||
mergeWarnings = cloneWarnings(mergeCP.Warnings)
|
diagnostics, diagnosticErr := promoteCheckpointDiagnostics(serializedMerge.Diagnostics, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageMerge, StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module}, nil)
|
||||||
output.Warnings = append(output.Warnings, mergeWarnings...)
|
if diagnosticErr != nil {
|
||||||
|
return stageResult, fmt.Errorf("promote reused merge diagnostics: %w", diagnosticErr)
|
||||||
|
}
|
||||||
|
appendDiagnosticGroups(output, diagnostics)
|
||||||
} else {
|
} else {
|
||||||
if stageResult.reuseEligible {
|
if stageResult.reuseEligible {
|
||||||
if err := checkpointMergeRunning(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps); err != nil {
|
if err := checkpointMergeRunning(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps); err != nil {
|
||||||
@@ -290,12 +293,11 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
|
|||||||
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, lane.Merge.Module, callErr))
|
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, lane.Merge.Module, callErr))
|
||||||
}
|
}
|
||||||
candidate := erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: result.Value}
|
candidate := erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: result.Value}
|
||||||
warnings := cloneWarnings(result.Warnings)
|
|
||||||
serializedCandidate, encodeErr := serializeCandidateArtifact(typed.codec, candidate.LaneID, candidate.MergerKey, candidate.SourceID, candidate.Value)
|
serializedCandidate, encodeErr := serializeCandidateArtifact(typed.codec, candidate.LaneID, candidate.MergerKey, candidate.SourceID, candidate.Value)
|
||||||
if encodeErr != nil {
|
if encodeErr != nil {
|
||||||
return producerAttemptOutput{}, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(warnings)}, fmt.Errorf("serialize merge candidate for lane %q: %w", lane.ID, encodeErr))
|
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("serialize merge candidate for lane %q: %w", lane.ID, encodeErr))
|
||||||
}
|
}
|
||||||
return producerAttemptOutput{Value: mergeAttemptValue{artifact: candidate, candidate: serializedCandidate, terminal: &terminal}, Candidate: result.ModelCandidate, Warnings: warnings}, nil
|
return producerAttemptOutput{Value: mergeAttemptValue{artifact: candidate, candidate: serializedCandidate, terminal: &terminal}, Candidate: result.ModelCandidate, Diagnostics: contracts.CloneProducerDiagnostics(result.Diagnostics)}, nil
|
||||||
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
|
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||||
candidate, ok := output.Value.(mergeAttemptValue)
|
candidate, ok := output.Value.(mergeAttemptValue)
|
||||||
if !ok {
|
if !ok {
|
||||||
@@ -303,7 +305,7 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
|
|||||||
}
|
}
|
||||||
target := typedValidationTarget{stage: StageMerge, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: mergeReferences, metadata: input.Metadata, value: candidate.artifact.Value, candidate: &candidate.candidate}
|
target := typedValidationTarget{stage: StageMerge, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: mergeReferences, metadata: input.Metadata, value: candidate.artifact.Value, candidate: &candidate.candidate}
|
||||||
report, validationErr := r.validateTypedReport(validationCtx, typed.codec, target, prepared.mergeValidators, candidate.terminal.envelope.Attempt, input.Debug)
|
report, validationErr := r.validateTypedReport(validationCtx, typed.codec, target, prepared.mergeValidators, candidate.terminal.envelope.Attempt, input.Debug)
|
||||||
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "warnings": debugWarningEnvelopes(append(cloneWarnings(output.Warnings), report.Warnings()...)), "rejection": debugRejectedOutputPtr(typedRejection(report, target, candidate.terminal.envelope.Attempt))}
|
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "rejection": debugRejectedOutputPtr(typedRejection(report, target, candidate.terminal.envelope.Attempt))}
|
||||||
if validationErr != nil {
|
if validationErr != nil {
|
||||||
return report, candidate.terminal.record(payload, validationErr)
|
return report, candidate.terminal.record(payload, validationErr)
|
||||||
}
|
}
|
||||||
@@ -322,6 +324,10 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
|
|||||||
}
|
}
|
||||||
return stageResult, runErr
|
return stageResult, runErr
|
||||||
}
|
}
|
||||||
|
terminalDiagnostics, diagnosticErr := terminalDiagnosticGroups(terminalResult, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageMerge, StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module}, nil)
|
||||||
|
if diagnosticErr != nil {
|
||||||
|
return stageResult, fmt.Errorf("promote merge diagnostics: %w", diagnosticErr)
|
||||||
|
}
|
||||||
if terminalResult.Action == producerTerminalRejected {
|
if terminalResult.Action == producerTerminalRejected {
|
||||||
rejected := terminalResult.Rejection
|
rejected := terminalResult.Rejection
|
||||||
if rejected == nil {
|
if rejected == nil {
|
||||||
@@ -330,7 +336,7 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
|
|||||||
rejected.Stage, rejected.StepID, rejected.LaneID, rejected.ModuleKey = string(StageMerge), input.stepID, lane.ID, lane.Merge.Module
|
rejected.Stage, rejected.StepID, rejected.LaneID, rejected.ModuleKey = string(StageMerge), input.stepID, lane.ID, lane.Merge.Module
|
||||||
rejected.Validation = &terminalSummary
|
rejected.Validation = &terminalSummary
|
||||||
output.ValidationSummaries = append(output.ValidationSummaries, artifacts.CloneValidationSummary(terminalSummary))
|
output.ValidationSummaries = append(output.ValidationSummaries, artifacts.CloneValidationSummary(terminalSummary))
|
||||||
output.Warnings = append(output.Warnings, terminalResult.Warnings...)
|
appendDiagnosticGroups(output, terminalDiagnostics)
|
||||||
output.Rejected = append(output.Rejected, *rejected)
|
output.Rejected = append(output.Rejected, *rejected)
|
||||||
if stageResult.reuseEligible {
|
if stageResult.reuseEligible {
|
||||||
if err := checkpointMergeRejected(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, *rejected); err != nil {
|
if err := checkpointMergeRejected(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, *rejected); err != nil {
|
||||||
@@ -345,7 +351,7 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
|
|||||||
return stageResult, fmt.Errorf("merge attempt terminal has incompatible value")
|
return stageResult, fmt.Errorf("merge attempt terminal has incompatible value")
|
||||||
}
|
}
|
||||||
stored, encodeErr := checkpointArtifact(typed.codec, candidate.artifact.LaneID, candidate.artifact.MergerKey, candidate.artifact.SourceID, candidate.artifact.Value)
|
stored, encodeErr := checkpointArtifact(typed.codec, candidate.artifact.LaneID, candidate.artifact.MergerKey, candidate.artifact.SourceID, candidate.artifact.Value)
|
||||||
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "warnings": debugWarningEnvelopes(terminalResult.Warnings), "rejection": debugRejectedOutputPtr(nil)}
|
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "rejection": debugRejectedOutputPtr(nil)}
|
||||||
if encodeErr != nil {
|
if encodeErr != nil {
|
||||||
attemptErr := fmt.Errorf("serialize accepted merge output for lane %q: %w", lane.ID, encodeErr)
|
attemptErr := fmt.Errorf("serialize accepted merge output for lane %q: %w", lane.ID, encodeErr)
|
||||||
if stageResult.reuseEligible {
|
if stageResult.reuseEligible {
|
||||||
@@ -353,6 +359,7 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
|
|||||||
}
|
}
|
||||||
return stageResult, candidate.terminal.record(payload, attemptErr)
|
return stageResult, candidate.terminal.record(payload, attemptErr)
|
||||||
}
|
}
|
||||||
|
stored.Diagnostics = terminalCheckpointDiagnostics(terminalResult)
|
||||||
if debugErr := candidate.terminal.record(payload, nil); debugErr != nil {
|
if debugErr := candidate.terminal.record(payload, nil); debugErr != nil {
|
||||||
if stageResult.reuseEligible {
|
if stageResult.reuseEligible {
|
||||||
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, debugErr)
|
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, debugErr)
|
||||||
@@ -360,20 +367,19 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
|
|||||||
return stageResult, debugErr
|
return stageResult, debugErr
|
||||||
}
|
}
|
||||||
merged, serializedMerge = candidate.artifact, stored
|
merged, serializedMerge = candidate.artifact, stored
|
||||||
mergeWarnings = cloneWarnings(terminalResult.Warnings)
|
|
||||||
output.ValidationSummaries = append(output.ValidationSummaries, terminalSummary)
|
output.ValidationSummaries = append(output.ValidationSummaries, terminalSummary)
|
||||||
stageResult.validationIncomplete = terminalResult.ValidationIncomplete
|
stageResult.validationIncomplete = terminalResult.ValidationIncomplete
|
||||||
if stageResult.validationIncomplete {
|
if stageResult.validationIncomplete {
|
||||||
stageResult.reuseEligible = false
|
stageResult.reuseEligible = false
|
||||||
}
|
}
|
||||||
output.Warnings = append(output.Warnings, mergeWarnings...)
|
appendDiagnosticGroups(output, terminalDiagnostics)
|
||||||
if stageResult.reuseEligible {
|
if stageResult.reuseEligible {
|
||||||
if err := recordMerge(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge, mergeWarnings); err != nil {
|
if err := recordMerge(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge); err != nil {
|
||||||
return stageResult, err
|
return stageResult, err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err := writeDebugTimed(input.Debug, path.Join("merge", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedMerge), "warnings": debugWarningEnvelopes(mergeWarnings)}}); err != nil {
|
if err := writeDebugTimed(input.Debug, path.Join("merge", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedMerge)}}); err != nil {
|
||||||
return stageResult, err
|
return stageResult, err
|
||||||
}
|
}
|
||||||
stageResult.artifact = merged
|
stageResult.artifact = merged
|
||||||
@@ -383,7 +389,6 @@ func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints
|
|||||||
|
|
||||||
type normalizeStageResult struct {
|
type normalizeStageResult struct {
|
||||||
serialized CheckpointArtifact
|
serialized CheckpointArtifact
|
||||||
warnings []contracts.Warning
|
|
||||||
accepted bool
|
accepted bool
|
||||||
reuseEligible bool
|
reuseEligible bool
|
||||||
}
|
}
|
||||||
@@ -409,11 +414,13 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
|
|||||||
return stageResult, err
|
return stageResult, err
|
||||||
}
|
}
|
||||||
var serializedNormalize CheckpointArtifact
|
var serializedNormalize CheckpointArtifact
|
||||||
var normalizeWarnings []contracts.Warning
|
|
||||||
if normalizeDecision.Reused {
|
if normalizeDecision.Reused {
|
||||||
serializedNormalize = normalizeResolution.artifacts[0]
|
serializedNormalize = normalizeResolution.artifacts[0]
|
||||||
normalizeWarnings = cloneWarnings(normalizeCP.Warnings)
|
diagnostics, diagnosticErr := promoteCheckpointDiagnostics(serializedNormalize.Diagnostics, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageNormalize, StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module}, nil)
|
||||||
output.Warnings = append(output.Warnings, normalizeWarnings...)
|
if diagnosticErr != nil {
|
||||||
|
return stageResult, fmt.Errorf("promote reused normalize diagnostics: %w", diagnosticErr)
|
||||||
|
}
|
||||||
|
appendDiagnosticGroups(output, diagnostics)
|
||||||
} else {
|
} else {
|
||||||
if stageResult.reuseEligible {
|
if stageResult.reuseEligible {
|
||||||
if err := checkpointNormalizeRunning(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps); err != nil {
|
if err := checkpointNormalizeRunning(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps); err != nil {
|
||||||
@@ -433,28 +440,27 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
|
|||||||
if callErr != nil {
|
if callErr != nil {
|
||||||
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, lane.Normalize.Module, callErr))
|
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, lane.Normalize.Module, callErr))
|
||||||
}
|
}
|
||||||
warnings := cloneWarnings(result.Warnings)
|
|
||||||
serializedCandidate, encodeErr := serializeCandidateArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, result.Value)
|
serializedCandidate, encodeErr := serializeCandidateArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, result.Value)
|
||||||
if encodeErr != nil {
|
if encodeErr != nil {
|
||||||
return producerAttemptOutput{}, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(warnings)}, fmt.Errorf("serialize normalize candidate for lane %q: %w", lane.ID, encodeErr))
|
return producerAttemptOutput{}, terminal.record(nil, fmt.Errorf("serialize normalize candidate for lane %q: %w", lane.ID, encodeErr))
|
||||||
}
|
}
|
||||||
attemptValue := normalizeAttemptValue{value: result.Value, candidate: serializedCandidate, terminal: &terminal}
|
attemptValue := normalizeAttemptValue{value: result.Value, candidate: serializedCandidate, terminal: &terminal}
|
||||||
var directive *producerRetryDirective
|
var directive *producerRetryDirective
|
||||||
if result.Retry != nil {
|
if result.Retry != nil {
|
||||||
if err := validateNormalizeRetry(result.Retry); err != nil {
|
if err := validateNormalizeRetry(result.Retry); err != nil {
|
||||||
return producerAttemptOutput{}, terminal.record(map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(warnings)}, fmt.Errorf("normalize lane %q returned invalid retry directive: %w", lane.ID, err))
|
return producerAttemptOutput{}, terminal.record(map[string]any{"output": debugCheckpointArtifact(serializedCandidate)}, fmt.Errorf("normalize lane %q returned invalid retry directive: %w", lane.ID, err))
|
||||||
}
|
}
|
||||||
anotherAttempt := request.Number <= lane.Normalize.Retries
|
anotherAttempt := request.Number <= lane.Normalize.Retries
|
||||||
attemptValue.retry = map[string]any{"reason_code": result.Retry.ReasonCode, "message": result.Retry.Message, "another_attempt": anotherAttempt, "fallback_accepted": !anotherAttempt}
|
attemptValue.retry = map[string]any{"reason_code": result.Retry.ReasonCode, "message": result.Retry.Message, "another_attempt": anotherAttempt, "fallback_accepted": !anotherAttempt}
|
||||||
directive = &producerRetryDirective{FallbackWarnings: cloneWarnings(result.Retry.FallbackWarnings)}
|
directive = &producerRetryDirective{FallbackDiagnostics: contracts.CloneProducerDiagnostics(result.Retry.FallbackDiagnostics)}
|
||||||
if anotherAttempt {
|
if anotherAttempt {
|
||||||
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(warnings), "retry": attemptValue.retry}
|
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "retry": attemptValue.retry}
|
||||||
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
if debugErr := terminal.record(payload, nil); debugErr != nil {
|
||||||
return producerAttemptOutput{}, debugErr
|
return producerAttemptOutput{}, debugErr
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return producerAttemptOutput{Value: attemptValue, Candidate: result.ModelCandidate, Warnings: warnings, Retry: directive}, nil
|
return producerAttemptOutput{Value: attemptValue, Candidate: result.ModelCandidate, Diagnostics: contracts.CloneProducerDiagnostics(result.Diagnostics), Retry: directive}, nil
|
||||||
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
|
}, func(validationCtx context.Context, output producerAttemptOutput) (validationReport, error) {
|
||||||
candidate, ok := output.Value.(normalizeAttemptValue)
|
candidate, ok := output.Value.(normalizeAttemptValue)
|
||||||
if !ok {
|
if !ok {
|
||||||
@@ -462,7 +468,7 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
|
|||||||
}
|
}
|
||||||
target := typedValidationTarget{stage: StageNormalize, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: normalizeReferences, metadata: input.Metadata, value: candidate.value, candidate: &candidate.candidate}
|
target := typedValidationTarget{stage: StageNormalize, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: normalizeReferences, metadata: input.Metadata, value: candidate.value, candidate: &candidate.candidate}
|
||||||
report, validationErr := r.validateTypedReport(validationCtx, typed.codec, target, prepared.normalizeValidators, candidate.terminal.envelope.Attempt, input.Debug)
|
report, validationErr := r.validateTypedReport(validationCtx, typed.codec, target, prepared.normalizeValidators, candidate.terminal.envelope.Attempt, input.Debug)
|
||||||
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "warnings": debugWarningEnvelopes(append(cloneWarnings(output.Warnings), report.Warnings()...)), "rejection": debugRejectedOutputPtr(typedRejection(report, target, candidate.terminal.envelope.Attempt))}
|
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "rejection": debugRejectedOutputPtr(typedRejection(report, target, candidate.terminal.envelope.Attempt))}
|
||||||
if candidate.retry != nil {
|
if candidate.retry != nil {
|
||||||
payload["retry"] = candidate.retry
|
payload["retry"] = candidate.retry
|
||||||
}
|
}
|
||||||
@@ -484,6 +490,10 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
|
|||||||
}
|
}
|
||||||
return stageResult, runErr
|
return stageResult, runErr
|
||||||
}
|
}
|
||||||
|
terminalDiagnostics, diagnosticErr := terminalDiagnosticGroups(terminalResult, contracts.DiagnosticOrigin{Stage: contracts.DiagnosticOriginStageNormalize, StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module}, nil)
|
||||||
|
if diagnosticErr != nil {
|
||||||
|
return stageResult, fmt.Errorf("promote normalize diagnostics: %w", diagnosticErr)
|
||||||
|
}
|
||||||
if terminalResult.Action == producerTerminalRejected {
|
if terminalResult.Action == producerTerminalRejected {
|
||||||
rejected := terminalResult.Rejection
|
rejected := terminalResult.Rejection
|
||||||
if rejected == nil {
|
if rejected == nil {
|
||||||
@@ -492,7 +502,7 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
|
|||||||
rejected.Stage, rejected.StepID, rejected.LaneID, rejected.ModuleKey = string(StageNormalize), input.stepID, lane.ID, lane.Normalize.Module
|
rejected.Stage, rejected.StepID, rejected.LaneID, rejected.ModuleKey = string(StageNormalize), input.stepID, lane.ID, lane.Normalize.Module
|
||||||
rejected.Validation = &terminalSummary
|
rejected.Validation = &terminalSummary
|
||||||
output.ValidationSummaries = append(output.ValidationSummaries, artifacts.CloneValidationSummary(terminalSummary))
|
output.ValidationSummaries = append(output.ValidationSummaries, artifacts.CloneValidationSummary(terminalSummary))
|
||||||
output.Warnings = append(output.Warnings, terminalResult.Warnings...)
|
appendDiagnosticGroups(output, terminalDiagnostics)
|
||||||
output.Rejected = append(output.Rejected, *rejected)
|
output.Rejected = append(output.Rejected, *rejected)
|
||||||
if stageResult.reuseEligible {
|
if stageResult.reuseEligible {
|
||||||
if err := checkpointNormalizeRejected(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, *rejected); err != nil {
|
if err := checkpointNormalizeRejected(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, *rejected); err != nil {
|
||||||
@@ -506,7 +516,7 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
|
|||||||
return stageResult, fmt.Errorf("normalize attempt terminal has incompatible value")
|
return stageResult, fmt.Errorf("normalize attempt terminal has incompatible value")
|
||||||
}
|
}
|
||||||
stored, encodeErr := checkpointArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, candidate.value)
|
stored, encodeErr := checkpointArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, candidate.value)
|
||||||
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "warnings": debugWarningEnvelopes(terminalResult.Warnings), "rejection": debugRejectedOutputPtr(nil)}
|
payload := map[string]any{"output": debugCheckpointArtifact(candidate.candidate), "rejection": debugRejectedOutputPtr(nil)}
|
||||||
if candidate.retry != nil {
|
if candidate.retry != nil {
|
||||||
payload["retry"] = candidate.retry
|
payload["retry"] = candidate.retry
|
||||||
}
|
}
|
||||||
@@ -517,6 +527,7 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
|
|||||||
}
|
}
|
||||||
return stageResult, candidate.terminal.record(payload, attemptErr)
|
return stageResult, candidate.terminal.record(payload, attemptErr)
|
||||||
}
|
}
|
||||||
|
stored.Diagnostics = terminalCheckpointDiagnostics(terminalResult)
|
||||||
if debugErr := candidate.terminal.record(payload, nil); debugErr != nil {
|
if debugErr := candidate.terminal.record(payload, nil); debugErr != nil {
|
||||||
if stageResult.reuseEligible {
|
if stageResult.reuseEligible {
|
||||||
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, debugErr)
|
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, debugErr)
|
||||||
@@ -524,23 +535,21 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
|
|||||||
return stageResult, debugErr
|
return stageResult, debugErr
|
||||||
}
|
}
|
||||||
serializedNormalize = stored
|
serializedNormalize = stored
|
||||||
normalizeWarnings = cloneWarnings(terminalResult.Warnings)
|
|
||||||
output.ValidationSummaries = append(output.ValidationSummaries, terminalSummary)
|
output.ValidationSummaries = append(output.ValidationSummaries, terminalSummary)
|
||||||
output.Warnings = append(output.Warnings, normalizeWarnings...)
|
appendDiagnosticGroups(output, terminalDiagnostics)
|
||||||
if terminalResult.ValidationIncomplete {
|
if terminalResult.ValidationIncomplete {
|
||||||
stageResult.reuseEligible = false
|
stageResult.reuseEligible = false
|
||||||
}
|
}
|
||||||
if stageResult.reuseEligible {
|
if stageResult.reuseEligible {
|
||||||
if err := recordNormalize(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize, normalizeWarnings); err != nil {
|
if err := recordNormalize(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize); err != nil {
|
||||||
return stageResult, err
|
return stageResult, err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err := writeDebugTimed(input.Debug, path.Join("normalize", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedNormalize), "warnings": debugWarningEnvelopes(normalizeWarnings)}}); err != nil {
|
if err := writeDebugTimed(input.Debug, path.Join("normalize", fileio.EncodePathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedNormalize)}}); err != nil {
|
||||||
return stageResult, err
|
return stageResult, err
|
||||||
}
|
}
|
||||||
stageResult.serialized = serializedNormalize
|
stageResult.serialized = serializedNormalize
|
||||||
stageResult.warnings = normalizeWarnings
|
|
||||||
stageResult.accepted = true
|
stageResult.accepted = true
|
||||||
return stageResult, nil
|
return stageResult, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -134,8 +134,8 @@ func (r *lockedCheckpointRecorder) SourceFailed(key string, err error) error {
|
|||||||
func (r *lockedCheckpointRecorder) ExtractRunning(lane, key string, deps []CheckpointFingerprint) error {
|
func (r *lockedCheckpointRecorder) ExtractRunning(lane, key string, deps []CheckpointFingerprint) error {
|
||||||
return r.call(func() error { return r.inner.ExtractRunning(lane, key, deps) })
|
return r.call(func() error { return r.inner.ExtractRunning(lane, key, deps) })
|
||||||
}
|
}
|
||||||
func (r *lockedCheckpointRecorder) ExtractSucceeded(lane, key string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
func (r *lockedCheckpointRecorder) ExtractSucceeded(lane, key string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput) error {
|
||||||
return r.call(func() error { return r.inner.ExtractSucceeded(lane, key, deps, outputs, rejected, warnings) })
|
return r.call(func() error { return r.inner.ExtractSucceeded(lane, key, deps, outputs, rejected) })
|
||||||
}
|
}
|
||||||
func (r *lockedCheckpointRecorder) ExtractFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
|
func (r *lockedCheckpointRecorder) ExtractFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
|
||||||
return r.call(func() error { return r.inner.ExtractFailed(lane, key, deps, err) })
|
return r.call(func() error { return r.inner.ExtractFailed(lane, key, deps, err) })
|
||||||
@@ -143,8 +143,8 @@ func (r *lockedCheckpointRecorder) ExtractFailed(lane, key string, deps []Checkp
|
|||||||
func (r *lockedCheckpointRecorder) MergeRunning(lane, key string, deps []CheckpointFingerprint) error {
|
func (r *lockedCheckpointRecorder) MergeRunning(lane, key string, deps []CheckpointFingerprint) error {
|
||||||
return r.call(func() error { return r.inner.MergeRunning(lane, key, deps) })
|
return r.call(func() error { return r.inner.MergeRunning(lane, key, deps) })
|
||||||
}
|
}
|
||||||
func (r *lockedCheckpointRecorder) MergeSucceeded(lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
func (r *lockedCheckpointRecorder) MergeSucceeded(lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact) error {
|
||||||
return r.call(func() error { return r.inner.MergeSucceeded(lane, key, deps, output, warnings) })
|
return r.call(func() error { return r.inner.MergeSucceeded(lane, key, deps, output) })
|
||||||
}
|
}
|
||||||
func (r *lockedCheckpointRecorder) MergeRejected(lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
func (r *lockedCheckpointRecorder) MergeRejected(lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||||
return r.call(func() error { return r.inner.MergeRejected(lane, key, deps, rejected) })
|
return r.call(func() error { return r.inner.MergeRejected(lane, key, deps, rejected) })
|
||||||
@@ -155,8 +155,8 @@ func (r *lockedCheckpointRecorder) MergeFailed(lane, key string, deps []Checkpoi
|
|||||||
func (r *lockedCheckpointRecorder) NormalizeRunning(lane, key string, deps []CheckpointFingerprint) error {
|
func (r *lockedCheckpointRecorder) NormalizeRunning(lane, key string, deps []CheckpointFingerprint) error {
|
||||||
return r.call(func() error { return r.inner.NormalizeRunning(lane, key, deps) })
|
return r.call(func() error { return r.inner.NormalizeRunning(lane, key, deps) })
|
||||||
}
|
}
|
||||||
func (r *lockedCheckpointRecorder) NormalizeSucceeded(lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
func (r *lockedCheckpointRecorder) NormalizeSucceeded(lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact) error {
|
||||||
return r.call(func() error { return r.inner.NormalizeSucceeded(lane, key, deps, output, warnings) })
|
return r.call(func() error { return r.inner.NormalizeSucceeded(lane, key, deps, output) })
|
||||||
}
|
}
|
||||||
func (r *lockedCheckpointRecorder) NormalizeRejected(lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
func (r *lockedCheckpointRecorder) NormalizeRejected(lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||||
return r.call(func() error { return r.inner.NormalizeRejected(lane, key, deps, rejected) })
|
return r.call(func() error { return r.inner.NormalizeRejected(lane, key, deps, rejected) })
|
||||||
@@ -173,12 +173,12 @@ func (r *lockedCheckpointRecorder) ExtractRunningForStep(step, lane, key string,
|
|||||||
return r.inner.ExtractRunning(lane, key, deps)
|
return r.inner.ExtractRunning(lane, key, deps)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
func (r *lockedCheckpointRecorder) ExtractSucceededForStep(step, lane, key string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
func (r *lockedCheckpointRecorder) ExtractSucceededForStep(step, lane, key string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput) error {
|
||||||
return r.call(func() error {
|
return r.call(func() error {
|
||||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||||
return v.ExtractSucceededForStep(step, lane, key, deps, outputs, rejected, warnings)
|
return v.ExtractSucceededForStep(step, lane, key, deps, outputs, rejected)
|
||||||
}
|
}
|
||||||
return r.inner.ExtractSucceeded(lane, key, deps, outputs, rejected, warnings)
|
return r.inner.ExtractSucceeded(lane, key, deps, outputs, rejected)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
func (r *lockedCheckpointRecorder) ExtractFailedForStep(step, lane, key string, deps []CheckpointFingerprint, err error) error {
|
func (r *lockedCheckpointRecorder) ExtractFailedForStep(step, lane, key string, deps []CheckpointFingerprint, err error) error {
|
||||||
@@ -197,12 +197,12 @@ func (r *lockedCheckpointRecorder) MergeRunningForStep(step, lane, key string, d
|
|||||||
return r.inner.MergeRunning(lane, key, deps)
|
return r.inner.MergeRunning(lane, key, deps)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
func (r *lockedCheckpointRecorder) MergeSucceededForStep(step, lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
func (r *lockedCheckpointRecorder) MergeSucceededForStep(step, lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact) error {
|
||||||
return r.call(func() error {
|
return r.call(func() error {
|
||||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||||
return v.MergeSucceededForStep(step, lane, key, deps, output, warnings)
|
return v.MergeSucceededForStep(step, lane, key, deps, output)
|
||||||
}
|
}
|
||||||
return r.inner.MergeSucceeded(lane, key, deps, output, warnings)
|
return r.inner.MergeSucceeded(lane, key, deps, output)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
func (r *lockedCheckpointRecorder) MergeRejectedForStep(step, lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
func (r *lockedCheckpointRecorder) MergeRejectedForStep(step, lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||||
@@ -229,12 +229,12 @@ func (r *lockedCheckpointRecorder) NormalizeRunningForStep(step, lane, key strin
|
|||||||
return r.inner.NormalizeRunning(lane, key, deps)
|
return r.inner.NormalizeRunning(lane, key, deps)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
func (r *lockedCheckpointRecorder) NormalizeSucceededForStep(step, lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
func (r *lockedCheckpointRecorder) NormalizeSucceededForStep(step, lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact) error {
|
||||||
return r.call(func() error {
|
return r.call(func() error {
|
||||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||||
return v.NormalizeSucceededForStep(step, lane, key, deps, output, warnings)
|
return v.NormalizeSucceededForStep(step, lane, key, deps, output)
|
||||||
}
|
}
|
||||||
return r.inner.NormalizeSucceeded(lane, key, deps, output, warnings)
|
return r.inner.NormalizeSucceeded(lane, key, deps, output)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
func (r *lockedCheckpointRecorder) NormalizeRejectedForStep(step, lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
func (r *lockedCheckpointRecorder) NormalizeRejectedForStep(step, lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ type erasedMergeArtifact struct {
|
|||||||
|
|
||||||
type erasedTypedResult struct {
|
type erasedTypedResult struct {
|
||||||
Value any
|
Value any
|
||||||
Warnings []contracts.Warning
|
Diagnostics []contracts.ProducerDiagnostic
|
||||||
Retry *contracts.NormalizeRetry
|
Retry *contracts.NormalizeRetry
|
||||||
ModelCandidate *contracts.ModelCandidate
|
ModelCandidate *contracts.ModelCandidate
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ const (
|
|||||||
const correctionRequestIntroduction = "The previous response failed semantic validation. Return one complete corrected replacement response, not a patch, explanation, or commentary.\n\nCorrect all of the following:\n"
|
const correctionRequestIntroduction = "The previous response failed semantic validation. Return one complete corrected replacement response, not a patch, explanation, or commentary.\n\nCorrect all of the following:\n"
|
||||||
|
|
||||||
// validationRecord captures the settled result of one configured validator.
|
// validationRecord captures the settled result of one configured validator.
|
||||||
// Its fields remain private so reports cannot expose mutable warning storage.
|
// Its fields remain private so reports cannot expose mutable diagnostic storage.
|
||||||
type validationRecord struct {
|
type validationRecord struct {
|
||||||
validatorName string
|
validatorName string
|
||||||
outcome validationOutcome
|
outcome validationOutcome
|
||||||
@@ -29,13 +29,13 @@ type validationRecord struct {
|
|||||||
reasonCode string
|
reasonCode string
|
||||||
message string
|
message string
|
||||||
diagnosticPath string
|
diagnosticPath string
|
||||||
warnings []contracts.Warning
|
diagnostics []contracts.ProducerDiagnostic
|
||||||
correctionGuidance string
|
correctionGuidance string
|
||||||
failure error
|
failure error
|
||||||
}
|
}
|
||||||
|
|
||||||
func (record validationRecord) clone() validationRecord {
|
func (record validationRecord) clone() validationRecord {
|
||||||
record.warnings = cloneWarnings(record.warnings)
|
record.diagnostics = contracts.CloneProducerDiagnostics(record.diagnostics)
|
||||||
return record
|
return record
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -52,14 +52,22 @@ func (report validationReport) Records() []validationRecord {
|
|||||||
return records
|
return records
|
||||||
}
|
}
|
||||||
|
|
||||||
func (report validationReport) Warnings() []contracts.Warning {
|
func (report validationReport) Diagnostics() []validationDiagnosticRecord {
|
||||||
var warnings []contracts.Warning
|
var diagnostics []validationDiagnosticRecord
|
||||||
for _, record := range report.records {
|
for _, record := range report.records {
|
||||||
if record.outcome == validationApproved || record.outcome == validationRejected {
|
if record.outcome != validationApproved && record.outcome != validationRejected {
|
||||||
warnings = append(warnings, cloneWarnings(record.warnings)...)
|
continue
|
||||||
|
}
|
||||||
|
for _, diagnostic := range record.diagnostics {
|
||||||
|
diagnostics = append(diagnostics, validationDiagnosticRecord{validatorName: record.validatorName, diagnostic: cloneProducerDiagnostics([]contracts.ProducerDiagnostic{diagnostic})[0]})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return warnings
|
return diagnostics
|
||||||
|
}
|
||||||
|
|
||||||
|
type validationDiagnosticRecord struct {
|
||||||
|
validatorName string
|
||||||
|
diagnostic contracts.ProducerDiagnostic
|
||||||
}
|
}
|
||||||
|
|
||||||
func (report validationReport) FirstRejection() *validationRecord {
|
func (report validationReport) FirstRejection() *validationRecord {
|
||||||
@@ -180,6 +188,9 @@ func executeValidationChain(ctx context.Context, chain preparedValidatorChain, i
|
|||||||
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationSkipped, attemptCount: attempt, reasonCode: invocation.reason, message: invocation.message})
|
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationSkipped, attemptCount: attempt, reasonCode: invocation.reason, message: invocation.message})
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
|
if err := contracts.ValidateProducerDiagnostics(invocation.result.Diagnostics); err != nil {
|
||||||
|
return validationReport{}, fatalValidationError(fmt.Errorf("validator %q returned invalid diagnostics: %w", binding.Module, err))
|
||||||
|
}
|
||||||
if err := contracts.ValidateValidationResult(invocation.result); err != nil {
|
if err := contracts.ValidateValidationResult(invocation.result); err != nil {
|
||||||
if attempt == attemptLimit {
|
if attempt == attemptLimit {
|
||||||
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationFailed, attemptCount: attempt, message: "validator returned an invalid result", failure: err})
|
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationFailed, attemptCount: attempt, message: "validator returned an invalid result", failure: err})
|
||||||
@@ -187,14 +198,14 @@ func executeValidationChain(ctx context.Context, chain preparedValidatorChain, i
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if invocation.result.Approved {
|
if invocation.result.Approved {
|
||||||
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationApproved, attemptCount: attempt, warnings: cloneWarnings(invocation.result.Warnings), diagnosticPath: invocation.result.DiagnosticArtifactPath})
|
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationApproved, attemptCount: attempt, diagnostics: cloneProducerDiagnostics(invocation.result.Diagnostics), diagnosticPath: invocation.result.DiagnosticArtifactPath})
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
message := invocation.result.Message
|
message := invocation.result.Message
|
||||||
if message == "" {
|
if message == "" {
|
||||||
message = "output rejected"
|
message = "output rejected"
|
||||||
}
|
}
|
||||||
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationRejected, attemptCount: attempt, reasonCode: invocation.result.ReasonCode, message: message, diagnosticPath: invocation.result.DiagnosticArtifactPath, warnings: cloneWarnings(invocation.result.Warnings), correctionGuidance: invocation.result.CorrectionGuidance})
|
report.records = append(report.records, validationRecord{validatorName: binding.Module, outcome: validationRejected, attemptCount: attempt, reasonCode: invocation.result.ReasonCode, message: message, diagnosticPath: invocation.result.DiagnosticArtifactPath, diagnostics: cloneProducerDiagnostics(invocation.result.Diagnostics), correctionGuidance: invocation.result.CorrectionGuidance})
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,24 +12,24 @@ import (
|
|||||||
|
|
||||||
func TestExecuteValidationChainSettlesEveryValidatorInOrder(t *testing.T) {
|
func TestExecuteValidationChainSettlesEveryValidatorInOrder(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
chain preparedValidatorChain
|
chain preparedValidatorChain
|
||||||
invoke validationInvoker
|
invoke validationInvoker
|
||||||
want []validationOutcome
|
want []validationOutcome
|
||||||
wantCalls []string
|
wantCalls []string
|
||||||
wantGuidance []string
|
wantGuidance []string
|
||||||
wantWarnings []string
|
wantDiagnostics []string
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "all approved",
|
name: "all approved",
|
||||||
chain: validationChain(validationSpec("shape", contracts.ExecutionClassDeterministic, 0), validationSpec("refs", contracts.ExecutionClassDeterministic, 0)),
|
chain: validationChain(validationSpec("shape", contracts.ExecutionClassDeterministic, 0), validationSpec("refs", contracts.ExecutionClassDeterministic, 0)),
|
||||||
invoke: validationSequence(map[string][]validationStep{
|
invoke: validationSequence(map[string][]validationStep{
|
||||||
"shape": {{result: contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{ReasonCode: "shape"}}}}},
|
"shape": {{result: contracts.ValidationResult{Approved: true, Diagnostics: []contracts.ProducerDiagnostic{validationDiagnostic("shape")}}}},
|
||||||
"refs": {{result: contracts.ValidationResult{Approved: true, Warnings: []contracts.Warning{{ReasonCode: "refs"}}}}},
|
"refs": {{result: contracts.ValidationResult{Approved: true, Diagnostics: []contracts.ProducerDiagnostic{validationDiagnostic("refs")}}}},
|
||||||
}),
|
}),
|
||||||
want: []validationOutcome{validationApproved, validationApproved},
|
want: []validationOutcome{validationApproved, validationApproved},
|
||||||
wantCalls: []string{"shape:1", "refs:1"},
|
wantCalls: []string{"shape:1", "refs:1"},
|
||||||
wantWarnings: []string{"shape", "refs"},
|
wantDiagnostics: []string{"shape", "refs"},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "multiple rejections deduplicate guidance",
|
name: "multiple rejections deduplicate guidance",
|
||||||
@@ -117,13 +117,13 @@ func TestExecuteValidationChainSettlesEveryValidatorInOrder(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
warnings := report.Warnings()
|
diagnostics := report.Diagnostics()
|
||||||
var warningCodes []string
|
var diagnosticCodes []string
|
||||||
for _, warning := range warnings {
|
for _, record := range diagnostics {
|
||||||
warningCodes = append(warningCodes, warning.ReasonCode)
|
diagnosticCodes = append(diagnosticCodes, record.diagnostic.ReasonCode)
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(warningCodes, test.wantWarnings) {
|
if !reflect.DeepEqual(diagnosticCodes, test.wantDiagnostics) {
|
||||||
t.Fatalf("Warnings() = %#v, want codes %#v", warnings, test.wantWarnings)
|
t.Fatalf("Diagnostics() = %#v, want codes %#v", diagnostics, test.wantDiagnostics)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -158,14 +158,24 @@ func TestValidationReportRejectsOversizedGuidanceAndOwnsRecords(t *testing.T) {
|
|||||||
unique[0] = byte('a' + index)
|
unique[0] = byte('a' + index)
|
||||||
report.records[index] = validationRecord{validatorName: "validator", outcome: validationRejected, correctionGuidance: string(unique)}
|
report.records[index] = validationRecord{validatorName: "validator", outcome: validationRejected, correctionGuidance: string(unique)}
|
||||||
}
|
}
|
||||||
report.records[0].warnings = []contracts.Warning{{ReasonCode: "warning"}}
|
report.records[0].diagnostics = []contracts.ProducerDiagnostic{validationDiagnostic("diagnostic")}
|
||||||
if _, err := report.CorrectionRequest(); err == nil {
|
if _, err := report.CorrectionRequest(); err == nil {
|
||||||
t.Fatal("CorrectionRequest() error = nil, want aggregate overflow error")
|
t.Fatal("CorrectionRequest() error = nil, want aggregate overflow error")
|
||||||
}
|
}
|
||||||
records := report.Records()
|
records := report.Records()
|
||||||
records[0].warnings[0].ReasonCode = "changed"
|
records[0].diagnostics[0].ReasonCode = "changed"
|
||||||
if report.Records()[0].warnings[0].ReasonCode != "warning" {
|
if report.Records()[0].diagnostics[0].ReasonCode != "diagnostic" {
|
||||||
t.Fatal("Records() exposed mutable warning storage")
|
t.Fatal("Records() exposed mutable diagnostic storage")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validationDiagnostic(reasonCode string) contracts.ProducerDiagnostic {
|
||||||
|
return contracts.ProducerDiagnostic{
|
||||||
|
Disposition: contracts.DiagnosticDispositionObservation,
|
||||||
|
Category: contracts.DiagnosticCategoryNormalization,
|
||||||
|
ReasonCode: reasonCode,
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []contracts.DiagnosticSample{{Scope: "validator", Message: reasonCode}},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -186,6 +196,19 @@ func TestExecuteValidationChainReturnsFrameworkAndCancellationErrors(t *testing.
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteValidationChainRejectsInvalidDiagnosticsWithoutRetry(t *testing.T) {
|
||||||
|
chain := validationChain(validationSpec("validator", contracts.ExecutionClassLLMBacked, 2))
|
||||||
|
calls := 0
|
||||||
|
_, err := executeValidationChain(context.Background(), chain, func(context.Context, preparedValidator, int) (validationInvocation, error) {
|
||||||
|
calls++
|
||||||
|
return validationInvocation{result: contracts.ValidationResult{Approved: true, Diagnostics: []contracts.ProducerDiagnostic{{}}}}, nil
|
||||||
|
})
|
||||||
|
var frameworkErr validationFrameworkError
|
||||||
|
if !errors.As(err, &frameworkErr) || calls != 1 || !strings.Contains(err.Error(), "validator \"validator\" returned invalid diagnostics") {
|
||||||
|
t.Fatalf("error = %v calls = %d, want one fatal invalid-diagnostics result", err, calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
type validationStep struct {
|
type validationStep struct {
|
||||||
result contracts.ValidationResult
|
result contracts.ValidationResult
|
||||||
invocation validationInvocation
|
invocation validationInvocation
|
||||||
|
|||||||
@@ -70,7 +70,7 @@ type ApplicationPolicy[T any] struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// GroupProvenance identifies the complete input contribution of one plan
|
// GroupProvenance identifies the complete input contribution of one plan
|
||||||
// group without prescribing domain warning or retry policy.
|
// group without prescribing domain diagnostic or retry policy.
|
||||||
type GroupProvenance struct {
|
type GroupProvenance struct {
|
||||||
memberPositions []int
|
memberPositions []int
|
||||||
canonicalPosition int
|
canonicalPosition int
|
||||||
@@ -108,7 +108,7 @@ func (event AppliedGroup) Provenance() GroupProvenance {
|
|||||||
return cloneGroupProvenance(event.provenance)
|
return cloneGroupProvenance(event.provenance)
|
||||||
}
|
}
|
||||||
|
|
||||||
// RejectedGroup records a typed guard decision while leaving warning and retry
|
// RejectedGroup records a typed guard decision while leaving diagnostic and retry
|
||||||
// construction to the consuming domain.
|
// construction to the consuming domain.
|
||||||
type RejectedGroup struct {
|
type RejectedGroup struct {
|
||||||
category RejectionCategory
|
category RejectionCategory
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// Issue identifies an unsafe proposal category at its original response group
|
// Issue identifies an unsafe proposal category at its original response group
|
||||||
// index without prescribing caller warning text.
|
// index without prescribing caller diagnostic text.
|
||||||
type Issue struct {
|
type Issue struct {
|
||||||
GroupIndex int
|
GroupIndex int
|
||||||
Category IssueCategory
|
Category IssueCategory
|
||||||
|
|||||||
@@ -179,9 +179,6 @@ func TestPlanReturnsAnnotationFreeSceneRangesFromStructuredOutput(t *testing.T)
|
|||||||
if len(result.Plan.Annotations) != 0 {
|
if len(result.Plan.Annotations) != 0 {
|
||||||
t.Fatalf("plan annotations = %#v, want absent", result.Plan.Annotations)
|
t.Fatalf("plan annotations = %#v, want absent", result.Plan.Annotations)
|
||||||
}
|
}
|
||||||
if len(result.Warnings) != 0 {
|
|
||||||
t.Fatalf("warnings = %#v, want absent", result.Warnings)
|
|
||||||
}
|
|
||||||
wantCandidate, err := json.Marshal(client.response)
|
wantCandidate, err := json.Marshal(client.response)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("marshal expected candidate: %v", err)
|
t.Fatalf("marshal expected candidate: %v", err)
|
||||||
|
|||||||
@@ -190,10 +190,12 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
|
|||||||
case sceneregistry.MatchMissing, sceneregistry.MatchMismatched:
|
case sceneregistry.MatchMissing, sceneregistry.MatchMismatched:
|
||||||
return contracts.TypedExtractionResult[dnd.CombatTurnList]{
|
return contracts.TypedExtractionResult[dnd.CombatTurnList]{
|
||||||
Value: dnd.CombatTurnList{CombatTurns: []dnd.CombatTurn{}},
|
Value: dnd.CombatTurnList{CombatTurns: []dnd.CombatTurn{}},
|
||||||
Warnings: []contracts.Warning{{
|
Diagnostics: []contracts.ProducerDiagnostic{{
|
||||||
Scope: SceneDescriptionReferenceSlot,
|
Disposition: contracts.DiagnosticDispositionWarning,
|
||||||
ReasonCode: "scene_classification_unavailable",
|
Category: contracts.DiagnosticCategoryDegradation,
|
||||||
Message: "No exact scene classification was available; combat extraction was skipped.",
|
ReasonCode: "scene_classification_unavailable",
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []contracts.DiagnosticSample{{Scope: SceneDescriptionReferenceSlot, Message: "No exact scene classification was available; combat extraction was skipped."}},
|
||||||
}},
|
}},
|
||||||
}, nil
|
}, nil
|
||||||
default:
|
default:
|
||||||
|
|||||||
@@ -310,11 +310,11 @@ func TestExtractAppliesSceneEligibilityBeforePromptConstruction(t *testing.T) {
|
|||||||
t.Fatal("CombatTurns = nil, want accepted non-nil empty list")
|
t.Fatal("CombatTurns = nil, want accepted non-nil empty list")
|
||||||
}
|
}
|
||||||
if test.wantWarning != "" {
|
if test.wantWarning != "" {
|
||||||
if len(result.Warnings) != 1 || result.Warnings[0].Scope != SceneDescriptionReferenceSlot || result.Warnings[0].ReasonCode != test.wantWarning {
|
if len(result.Diagnostics) != 1 || result.Diagnostics[0].Samples[0].Scope != SceneDescriptionReferenceSlot || result.Diagnostics[0].ReasonCode != test.wantWarning || result.Diagnostics[0].Disposition != contracts.DiagnosticDispositionWarning || result.Diagnostics[0].Category != contracts.DiagnosticCategoryDegradation {
|
||||||
t.Fatalf("warnings = %#v", result.Warnings)
|
t.Fatalf("diagnostics = %#v", result.Diagnostics)
|
||||||
}
|
}
|
||||||
} else if len(result.Warnings) != 0 {
|
} else if len(result.Diagnostics) != 0 {
|
||||||
t.Fatalf("warnings = %#v, want none", result.Warnings)
|
t.Fatalf("diagnostics = %#v, want none", result.Diagnostics)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -171,10 +171,12 @@ func emptyResult() contracts.TypedExtractionResult[dnd.EnemyEventList] {
|
|||||||
|
|
||||||
func unavailableSceneResult() contracts.TypedExtractionResult[dnd.EnemyEventList] {
|
func unavailableSceneResult() contracts.TypedExtractionResult[dnd.EnemyEventList] {
|
||||||
result := emptyResult()
|
result := emptyResult()
|
||||||
result.Warnings = []contracts.Warning{{
|
result.Diagnostics = []contracts.ProducerDiagnostic{{
|
||||||
Scope: SceneDescriptionReferenceSlot,
|
Disposition: contracts.DiagnosticDispositionWarning,
|
||||||
ReasonCode: "scene_classification_unavailable",
|
Category: contracts.DiagnosticCategoryDegradation,
|
||||||
Message: "No exact scene classification was available; enemy-event extraction was skipped.",
|
ReasonCode: "scene_classification_unavailable",
|
||||||
|
OccurrenceCount: 1,
|
||||||
|
Samples: []contracts.DiagnosticSample{{Scope: SceneDescriptionReferenceSlot, Message: "No exact scene classification was available; enemy-event extraction was skipped."}},
|
||||||
}}
|
}}
|
||||||
return result
|
return result
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -98,11 +98,11 @@ func TestExtractSkipsModelForIneligibleScenes(t *testing.T) {
|
|||||||
t.Fatalf("result = %#v, calls = %d", result, len(client.requests))
|
t.Fatalf("result = %#v, calls = %d", result, len(client.requests))
|
||||||
}
|
}
|
||||||
if test.wantWarning {
|
if test.wantWarning {
|
||||||
if len(result.Warnings) != 1 || result.Warnings[0].ReasonCode != "scene_classification_unavailable" || result.Warnings[0].Scope != SceneDescriptionReferenceSlot {
|
if len(result.Diagnostics) != 1 || result.Diagnostics[0].ReasonCode != "scene_classification_unavailable" || result.Diagnostics[0].Disposition != contracts.DiagnosticDispositionWarning || result.Diagnostics[0].Category != contracts.DiagnosticCategoryDegradation || result.Diagnostics[0].Samples[0].Scope != SceneDescriptionReferenceSlot {
|
||||||
t.Fatalf("warnings = %#v", result.Warnings)
|
t.Fatalf("diagnostics = %#v", result.Diagnostics)
|
||||||
}
|
}
|
||||||
} else if len(result.Warnings) != 0 {
|
} else if len(result.Diagnostics) != 0 {
|
||||||
t.Fatalf("warnings = %#v", result.Warnings)
|
t.Fatalf("diagnostics = %#v", result.Diagnostics)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -40,10 +40,6 @@ func TestExtractReturnsCanonicalSpellListFromPrivateResponse(t *testing.T) {
|
|||||||
if !reflect.DeepEqual(result.Value, want) {
|
if !reflect.DeepEqual(result.Value, want) {
|
||||||
t.Fatalf("Value = %#v, want %#v", result.Value, want)
|
t.Fatalf("Value = %#v, want %#v", result.Value, want)
|
||||||
}
|
}
|
||||||
if len(result.Warnings) != 0 {
|
|
||||||
t.Fatalf("Warnings = %#v, want none", result.Warnings)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(client.requests) != 1 {
|
if len(client.requests) != 1 {
|
||||||
t.Fatalf("LLM calls = %d, want 1", len(client.requests))
|
t.Fatalf("LLM calls = %d, want 1", len(client.requests))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -27,7 +27,6 @@ const (
|
|||||||
ReasonCodeSourceRefsNormalized = "source_references_normalized"
|
ReasonCodeSourceRefsNormalized = "source_references_normalized"
|
||||||
ReasonCodeTurnsReordered = "combat_turns_reordered"
|
ReasonCodeTurnsReordered = "combat_turns_reordered"
|
||||||
ReasonCodeDuplicateCollapsed = "duplicate_combat_turn_collapsed"
|
ReasonCodeDuplicateCollapsed = "duplicate_combat_turn_collapsed"
|
||||||
ReasonCodeWarningsOmitted = "combat_turn_normalization_warnings_omitted"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -113,11 +112,12 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
}
|
}
|
||||||
index := source.NewDocumentIndex(req.Source)
|
index := source.NewDocumentIndex(req.Source)
|
||||||
order := shared.NewSourceRefOrderFromIndex(index)
|
order := shared.NewSourceRefOrderFromIndex(index)
|
||||||
value, warnings := normalizeList(req.MergeOutput.Value, index, order, npcRegistry)
|
value, findings := normalizeList(req.MergeOutput.Value, index, order, npcRegistry)
|
||||||
return contracts.TypedNormalizeResult[dnd.CombatTurnList]{
|
diagnosticGroups, err := diagnostics.NormalizationDiagnostics(findings)
|
||||||
Value: value,
|
if err != nil {
|
||||||
Warnings: diagnostics.LimitWarnings(warnings, "combat_turns", ReasonCodeWarningsOmitted),
|
return contracts.TypedNormalizeResult[dnd.CombatTurnList]{}, normalizerErrorf("collect diagnostics: %w", err)
|
||||||
}, nil
|
}
|
||||||
|
return contracts.TypedNormalizeResult[dnd.CombatTurnList]{Value: value, Diagnostics: diagnosticGroups}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type normalizedRecord struct {
|
type normalizedRecord struct {
|
||||||
@@ -132,13 +132,13 @@ type actorCanonicalization struct {
|
|||||||
to string
|
to string
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeList(input dnd.CombatTurnList, documentIndex source.DocumentIndex, order shared.SourceRefOrder, registry *npcregistry.Registry) (dnd.CombatTurnList, []contracts.Warning) {
|
func normalizeList(input dnd.CombatTurnList, documentIndex source.DocumentIndex, order shared.SourceRefOrder, registry *npcregistry.Registry) (dnd.CombatTurnList, []diagnostics.Finding) {
|
||||||
if input.CombatTurns == nil {
|
if input.CombatTurns == nil {
|
||||||
return dnd.CombatTurnList{}, nil
|
return dnd.CombatTurnList{}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
records := make([]normalizedRecord, len(input.CombatTurns))
|
records := make([]normalizedRecord, len(input.CombatTurns))
|
||||||
warnings := make([]contracts.Warning, 0)
|
findings := make([]diagnostics.Finding, 0)
|
||||||
for index, inputTurn := range input.CombatTurns {
|
for index, inputTurn := range input.CombatTurns {
|
||||||
turn, actorChange, refsChanged := normalizeTurn(inputTurn, order, registry)
|
turn, actorChange, refsChanged := normalizeTurn(inputTurn, order, registry)
|
||||||
earliest, hasEvidence := order.EarliestValid(turn.SourceRefs)
|
earliest, hasEvidence := order.EarliestValid(turn.SourceRefs)
|
||||||
@@ -149,7 +149,7 @@ func normalizeList(input dnd.CombatTurnList, documentIndex source.DocumentIndex,
|
|||||||
hasEvidence: hasEvidence,
|
hasEvidence: hasEvidence,
|
||||||
}
|
}
|
||||||
if actorChange != nil {
|
if actorChange != nil {
|
||||||
warnings = append(warnings, contracts.Warning{
|
findings = append(findings, diagnostics.Finding{
|
||||||
Scope: turnScope(index),
|
Scope: turnScope(index),
|
||||||
ReasonCode: ReasonCodeActorCanonicalized,
|
ReasonCode: ReasonCodeActorCanonicalized,
|
||||||
Message: fmt.Sprintf("input index %d: actor canonicalized from %s to %s",
|
Message: fmt.Sprintf("input index %d: actor canonicalized from %s to %s",
|
||||||
@@ -157,7 +157,7 @@ func normalizeList(input dnd.CombatTurnList, documentIndex source.DocumentIndex,
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
if refsChanged {
|
if refsChanged {
|
||||||
warnings = append(warnings, contracts.Warning{
|
findings = append(findings, diagnostics.Finding{
|
||||||
Scope: turnScope(index),
|
Scope: turnScope(index),
|
||||||
ReasonCode: ReasonCodeSourceRefsNormalized,
|
ReasonCode: ReasonCodeSourceRefsNormalized,
|
||||||
Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)",
|
Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)",
|
||||||
@@ -179,7 +179,7 @@ func normalizeList(input dnd.CombatTurnList, documentIndex source.DocumentIndex,
|
|||||||
if position == record.inputIndex {
|
if position == record.inputIndex {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
warnings = append(warnings, contracts.Warning{
|
findings = append(findings, diagnostics.Finding{
|
||||||
Scope: turnScope(record.inputIndex),
|
Scope: turnScope(record.inputIndex),
|
||||||
ReasonCode: ReasonCodeTurnsReordered,
|
ReasonCode: ReasonCodeTurnsReordered,
|
||||||
Message: fmt.Sprintf("input index %d moved to normalized position %d by source chronology",
|
Message: fmt.Sprintf("input index %d moved to normalized position %d by source chronology",
|
||||||
@@ -187,9 +187,9 @@ func normalizeList(input dnd.CombatTurnList, documentIndex source.DocumentIndex,
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
output, duplicateWarnings := collapseDuplicates(records, documentIndex)
|
output, duplicateFindings := collapseDuplicates(records, documentIndex)
|
||||||
warnings = append(warnings, duplicateWarnings...)
|
findings = append(findings, duplicateFindings...)
|
||||||
return dnd.CombatTurnList{CombatTurns: output}, warnings
|
return dnd.CombatTurnList{CombatTurns: output}, findings
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeTurn(input dnd.CombatTurn, order shared.SourceRefOrder, registry *npcregistry.Registry) (dnd.CombatTurn, *actorCanonicalization, bool) {
|
func normalizeTurn(input dnd.CombatTurn, order shared.SourceRefOrder, registry *npcregistry.Registry) (dnd.CombatTurn, *actorCanonicalization, bool) {
|
||||||
@@ -236,7 +236,7 @@ type duplicateGroup struct {
|
|||||||
removed []int
|
removed []int
|
||||||
}
|
}
|
||||||
|
|
||||||
func collapseDuplicates(records []normalizedRecord, documentIndex source.DocumentIndex) ([]dnd.CombatTurn, []contracts.Warning) {
|
func collapseDuplicates(records []normalizedRecord, documentIndex source.DocumentIndex) ([]dnd.CombatTurn, []diagnostics.Finding) {
|
||||||
if len(records) == 0 {
|
if len(records) == 0 {
|
||||||
return make([]dnd.CombatTurn, 0), nil
|
return make([]dnd.CombatTurn, 0), nil
|
||||||
}
|
}
|
||||||
@@ -267,14 +267,14 @@ func collapseDuplicates(records []normalizedRecord, documentIndex source.Documen
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
warnings := make([]contracts.Warning, 0)
|
findings := make([]diagnostics.Finding, 0)
|
||||||
for _, group := range groups {
|
for _, group := range groups {
|
||||||
if len(group.removed) == 0 {
|
if len(group.removed) == 0 {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
warnings = append(warnings, duplicateWarning(group.retainedIndex, group.removed))
|
findings = append(findings, duplicateFinding(group.retainedIndex, group.removed))
|
||||||
}
|
}
|
||||||
return output, warnings
|
return output, findings
|
||||||
}
|
}
|
||||||
|
|
||||||
func duplicateKey(turn dnd.CombatTurn, documentIndex source.DocumentIndex) (string, bool) {
|
func duplicateKey(turn dnd.CombatTurn, documentIndex source.DocumentIndex) (string, bool) {
|
||||||
@@ -309,12 +309,12 @@ func writeKeyInt(builder *strings.Builder, value int) {
|
|||||||
builder.WriteByte(';')
|
builder.WriteByte(';')
|
||||||
}
|
}
|
||||||
|
|
||||||
func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
|
func duplicateFinding(retainedIndex int, removed []int) diagnostics.Finding {
|
||||||
issues := make([]string, len(removed))
|
issues := make([]string, len(removed))
|
||||||
for index, removedIndex := range removed {
|
for index, removedIndex := range removed {
|
||||||
issues[index] = fmt.Sprintf("removed input index %d", removedIndex)
|
issues[index] = fmt.Sprintf("removed input index %d", removedIndex)
|
||||||
}
|
}
|
||||||
return contracts.Warning{
|
return diagnostics.Finding{
|
||||||
Scope: turnScope(retainedIndex),
|
Scope: turnScope(retainedIndex),
|
||||||
ReasonCode: ReasonCodeDuplicateCollapsed,
|
ReasonCode: ReasonCodeDuplicateCollapsed,
|
||||||
Message: diagnostics.Aggregate(
|
Message: diagnostics.Aggregate(
|
||||||
|
|||||||
@@ -15,7 +15,6 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcregistry"
|
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcregistry"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestNormalizeCanonicalizesFieldsAndRegistryIdentities(t *testing.T) {
|
func TestNormalizeCanonicalizesFieldsAndRegistryIdentities(t *testing.T) {
|
||||||
@@ -47,8 +46,8 @@ func TestNormalizeCanonicalizesFieldsAndRegistryIdentities(t *testing.T) {
|
|||||||
t.Fatalf("normalized refs = %#v, want %#v", result.Value.CombatTurns[0].SourceRefs, wantRefs)
|
t.Fatalf("normalized refs = %#v, want %#v", result.Value.CombatTurns[0].SourceRefs, wantRefs)
|
||||||
}
|
}
|
||||||
for _, reason := range []string{ReasonCodeActorCanonicalized, ReasonCodeSourceRefsNormalized} {
|
for _, reason := range []string{ReasonCodeActorCanonicalized, ReasonCodeSourceRefsNormalized} {
|
||||||
if !hasWarningReason(result.Warnings, reason) {
|
if !hasDiagnosticReason(result.Diagnostics, reason) {
|
||||||
t.Fatalf("warnings = %#v, missing reason %q", result.Warnings, reason)
|
t.Fatalf("diagnostics = %#v, missing reason %q", result.Diagnostics, reason)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(input.CombatTurns[0], original) {
|
if !reflect.DeepEqual(input.CombatTurns[0], original) {
|
||||||
@@ -60,8 +59,8 @@ func TestNormalizeCanonicalizesFieldsAndRegistryIdentities(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestNormalizeLimitsWarningsWithoutChangingCombatTurnValues(t *testing.T) {
|
func TestNormalizeLimitsFindingsWithoutChangingCombatTurnValues(t *testing.T) {
|
||||||
units := make([]source.SourceUnit, diagnostics.MaxWarnings+1)
|
units := make([]source.SourceUnit, contracts.MaxDiagnosticSamples+1)
|
||||||
turns := make([]dnd.CombatTurn, len(units))
|
turns := make([]dnd.CombatTurn, len(units))
|
||||||
for index := range units {
|
for index := range units {
|
||||||
unitID := index + 1
|
unitID := index + 1
|
||||||
@@ -82,15 +81,8 @@ func TestNormalizeLimitsWarningsWithoutChangingCombatTurnValues(t *testing.T) {
|
|||||||
if len(result.Value.CombatTurns) != len(turns) || result.Value.CombatTurns[0].Actor != "Aria" {
|
if len(result.Value.CombatTurns) != len(turns) || result.Value.CombatTurns[0].Actor != "Aria" {
|
||||||
t.Fatalf("normalized turns = %#v, want canonicalized values", result.Value.CombatTurns)
|
t.Fatalf("normalized turns = %#v, want canonicalized values", result.Value.CombatTurns)
|
||||||
}
|
}
|
||||||
if len(result.Warnings) != diagnostics.MaxWarnings {
|
if len(result.Diagnostics) == 0 || result.Diagnostics[0].Disposition != contracts.DiagnosticDispositionObservation || result.Diagnostics[0].OccurrenceCount != len(turns) || len(result.Diagnostics[0].Samples) != contracts.MaxDiagnosticSamples {
|
||||||
t.Fatalf("warning count = %d, want %d", len(result.Warnings), diagnostics.MaxWarnings)
|
t.Fatalf("diagnostics = %#v", result.Diagnostics)
|
||||||
}
|
|
||||||
if first := result.Warnings[0]; first.Scope != "combat_turns[0]" || first.ReasonCode != ReasonCodeActorCanonicalized {
|
|
||||||
t.Fatalf("first warning = %#v, want first input warning", first)
|
|
||||||
}
|
|
||||||
summary := result.Warnings[len(result.Warnings)-1]
|
|
||||||
if summary.Scope != "combat_turns" || summary.ReasonCode != ReasonCodeWarningsOmitted || summary.Message != "2 additional warning(s) omitted" {
|
|
||||||
t.Fatalf("warning summary = %#v", summary)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -147,12 +139,12 @@ func TestNormalizeOrdersBySourcePositionAndCollapsesExactDuplicates(t *testing.T
|
|||||||
if result.Value.CombatTurns[0].SourceRefs[0].StartUnitID != 50 || result.Value.CombatTurns[1].SourceRefs[0].StartUnitID != 90 || result.Value.CombatTurns[2].Actor != "Unknown" {
|
if result.Value.CombatTurns[0].SourceRefs[0].StartUnitID != 50 || result.Value.CombatTurns[1].SourceRefs[0].StartUnitID != 90 || result.Value.CombatTurns[2].Actor != "Unknown" {
|
||||||
t.Fatalf("normalized order/value = %#v, want chronology then invalid evidence", result.Value.CombatTurns)
|
t.Fatalf("normalized order/value = %#v, want chronology then invalid evidence", result.Value.CombatTurns)
|
||||||
}
|
}
|
||||||
if !hasWarningReason(result.Warnings, ReasonCodeTurnsReordered) || !hasWarningReason(result.Warnings, ReasonCodeDuplicateCollapsed) {
|
if !hasDiagnosticReason(result.Diagnostics, ReasonCodeTurnsReordered) || !hasDiagnosticReason(result.Diagnostics, ReasonCodeDuplicateCollapsed) {
|
||||||
t.Fatalf("warnings = %#v, want reorder and duplicate warnings", result.Warnings)
|
t.Fatalf("diagnostics = %#v, want reorder and duplicate observations", result.Diagnostics)
|
||||||
}
|
}
|
||||||
for _, warning := range result.Warnings {
|
for _, diagnostic := range result.Diagnostics {
|
||||||
if warning.ReasonCode == ReasonCodeDuplicateCollapsed && warning.Scope != "combat_turns[1]" {
|
if diagnostic.ReasonCode == ReasonCodeDuplicateCollapsed && diagnostic.Samples[0].Scope != "combat_turns[1]" {
|
||||||
t.Fatalf("duplicate warning = %#v, want retained input scope combat_turns[1]", warning)
|
t.Fatalf("duplicate diagnostic = %#v, want retained input scope combat_turns[1]", diagnostic)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -202,8 +194,8 @@ func TestNormalizePreservesStableOrderForEqualEvidencePositions(t *testing.T) {
|
|||||||
if got := []string{result.Value.CombatTurns[0].Actor, result.Value.CombatTurns[1].Actor}; !reflect.DeepEqual(got, []string{"Aria", "Borin"}) {
|
if got := []string{result.Value.CombatTurns[0].Actor, result.Value.CombatTurns[1].Actor}; !reflect.DeepEqual(got, []string{"Aria", "Borin"}) {
|
||||||
t.Fatalf("equal-position order = %#v, want stable input order", got)
|
t.Fatalf("equal-position order = %#v, want stable input order", got)
|
||||||
}
|
}
|
||||||
if hasWarningReason(result.Warnings, ReasonCodeTurnsReordered) {
|
if hasDiagnosticReason(result.Diagnostics, ReasonCodeTurnsReordered) {
|
||||||
t.Fatalf("warnings = %#v, equal-position stable sort should not warn", result.Warnings)
|
t.Fatalf("diagnostics = %#v, equal-position stable sort should not report reordering", result.Diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -296,23 +288,23 @@ func TestNormalizerPreparationMetadataFingerprintsAndModuleContract(t *testing.T
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestDuplicateWarningBoundsDisplayedIndexesAndReportsAllOmissions(t *testing.T) {
|
func TestDuplicateFindingBoundsDisplayedIndexesAndReportsAllOmissions(t *testing.T) {
|
||||||
removed := make([]int, 25)
|
removed := make([]int, 25)
|
||||||
for index := range removed {
|
for index := range removed {
|
||||||
removed[index] = math.MaxInt - index
|
removed[index] = math.MaxInt - index
|
||||||
}
|
}
|
||||||
warning := duplicateWarning(7, removed)
|
finding := duplicateFinding(7, removed)
|
||||||
if warning.Scope != "combat_turns[7]" || warning.ReasonCode != ReasonCodeDuplicateCollapsed {
|
if finding.Scope != "combat_turns[7]" || finding.ReasonCode != ReasonCodeDuplicateCollapsed {
|
||||||
t.Fatalf("duplicate warning = %#v, want retained-record scope and reason", warning)
|
t.Fatalf("duplicate finding = %#v, want retained-record scope and reason", finding)
|
||||||
}
|
}
|
||||||
if !strings.Contains(warning.Message, "retained input index 7") || !strings.Contains(warning.Message, fmt.Sprintf("removed input index %d", removed[0])) {
|
if !strings.Contains(finding.Message, "retained input index 7") || !strings.Contains(finding.Message, fmt.Sprintf("removed input index %d", removed[0])) {
|
||||||
t.Fatalf("duplicate warning = %q, want retained and displayed removed indexes", warning.Message)
|
t.Fatalf("duplicate finding = %q, want retained and displayed removed indexes", finding.Message)
|
||||||
}
|
}
|
||||||
if !strings.Contains(warning.Message, "5 additional issue(s) omitted") {
|
if !strings.Contains(finding.Message, "5 additional issue(s) omitted") {
|
||||||
t.Fatalf("duplicate warning = %q, want exact omitted count", warning.Message)
|
t.Fatalf("duplicate finding = %q, want exact omitted count", finding.Message)
|
||||||
}
|
}
|
||||||
if !utf8.ValidString(warning.Message) || len([]byte(warning.Message)) > 4096 {
|
if !utf8.ValidString(finding.Message) || len([]byte(finding.Message)) > 4096 {
|
||||||
t.Fatalf("duplicate warning length/encoding = %d/%t", len([]byte(warning.Message)), utf8.ValidString(warning.Message))
|
t.Fatalf("duplicate finding length/encoding = %d/%t", len([]byte(finding.Message)), utf8.ValidString(finding.Message))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -361,9 +353,9 @@ func npcReferences(t *testing.T) contracts.ReferenceSet {
|
|||||||
}}
|
}}
|
||||||
}
|
}
|
||||||
|
|
||||||
func hasWarningReason(warnings []contracts.Warning, reason string) bool {
|
func hasDiagnosticReason(diagnostics []contracts.ProducerDiagnostic, reason string) bool {
|
||||||
for _, warning := range warnings {
|
for _, diagnostic := range diagnostics {
|
||||||
if warning.ReasonCode == reason {
|
if diagnostic.ReasonCode == reason && diagnostic.Disposition == contracts.DiagnosticDispositionObservation {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -25,7 +25,6 @@ const (
|
|||||||
ReasonCodeSourceRefsNormalized = "source_references_normalized"
|
ReasonCodeSourceRefsNormalized = "source_references_normalized"
|
||||||
ReasonCodeEventsReordered = "enemy_events_reordered"
|
ReasonCodeEventsReordered = "enemy_events_reordered"
|
||||||
ReasonCodeDuplicateCollapsed = "duplicate_enemy_event_collapsed"
|
ReasonCodeDuplicateCollapsed = "duplicate_enemy_event_collapsed"
|
||||||
ReasonCodeWarningsOmitted = "enemy_event_normalization_warnings_omitted"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -105,8 +104,12 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
}
|
}
|
||||||
index := source.NewDocumentIndex(req.Source)
|
index := source.NewDocumentIndex(req.Source)
|
||||||
order := shared.NewSourceRefOrderFromIndex(index)
|
order := shared.NewSourceRefOrderFromIndex(index)
|
||||||
value, warnings := normalizeList(req.MergeOutput.Value, order, registry)
|
value, findings := normalizeList(req.MergeOutput.Value, order, registry)
|
||||||
return contracts.TypedNormalizeResult[dnd.EnemyEventList]{Value: value, Warnings: warnings}, nil
|
diagnosticGroups, err := diagnostics.NormalizationDiagnostics(findings)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.EnemyEventList]{}, normalizerErrorf("collect diagnostics: %w", err)
|
||||||
|
}
|
||||||
|
return contracts.TypedNormalizeResult[dnd.EnemyEventList]{Value: value, Diagnostics: diagnosticGroups}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type normalizedRecord struct {
|
type normalizedRecord struct {
|
||||||
@@ -120,17 +123,17 @@ type nameCanonicalization struct {
|
|||||||
to string
|
to string
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeList(input dnd.EnemyEventList, order shared.SourceRefOrder, registry *npcregistry.Registry) (dnd.EnemyEventList, []contracts.Warning) {
|
func normalizeList(input dnd.EnemyEventList, order shared.SourceRefOrder, registry *npcregistry.Registry) (dnd.EnemyEventList, []diagnostics.Finding) {
|
||||||
if input.Events == nil {
|
if input.Events == nil {
|
||||||
return dnd.EnemyEventList{}, nil
|
return dnd.EnemyEventList{}, nil
|
||||||
}
|
}
|
||||||
records := make([]normalizedRecord, len(input.Events))
|
records := make([]normalizedRecord, len(input.Events))
|
||||||
warnings := make([]contracts.Warning, 0)
|
findings := make([]diagnostics.Finding, 0)
|
||||||
for index, inputEvent := range input.Events {
|
for index, inputEvent := range input.Events {
|
||||||
event, nameChange, refsChanged := normalizeEvent(inputEvent, order, registry)
|
event, nameChange, refsChanged := normalizeEvent(inputEvent, order, registry)
|
||||||
records[index] = normalizedRecord{event: event, identity: enemyeventmodel.CanonicalIdentity(event), inputIndex: index}
|
records[index] = normalizedRecord{event: event, identity: enemyeventmodel.CanonicalIdentity(event), inputIndex: index}
|
||||||
if nameChange != nil {
|
if nameChange != nil {
|
||||||
warnings = append(warnings, contracts.Warning{
|
findings = append(findings, diagnostics.Finding{
|
||||||
Scope: eventScope(index),
|
Scope: eventScope(index),
|
||||||
ReasonCode: ReasonCodeNameCanonicalized,
|
ReasonCode: ReasonCodeNameCanonicalized,
|
||||||
Message: fmt.Sprintf("input index %d: subject canonicalized from %s to %s",
|
Message: fmt.Sprintf("input index %d: subject canonicalized from %s to %s",
|
||||||
@@ -138,7 +141,7 @@ func normalizeList(input dnd.EnemyEventList, order shared.SourceRefOrder, regist
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
if refsChanged {
|
if refsChanged {
|
||||||
warnings = append(warnings, contracts.Warning{
|
findings = append(findings, diagnostics.Finding{
|
||||||
Scope: eventScope(index),
|
Scope: eventScope(index),
|
||||||
ReasonCode: ReasonCodeSourceRefsNormalized,
|
ReasonCode: ReasonCodeSourceRefsNormalized,
|
||||||
Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)",
|
Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)",
|
||||||
@@ -154,16 +157,16 @@ func normalizeList(input dnd.EnemyEventList, order shared.SourceRefOrder, regist
|
|||||||
if position == record.inputIndex {
|
if position == record.inputIndex {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
warnings = append(warnings, contracts.Warning{
|
findings = append(findings, diagnostics.Finding{
|
||||||
Scope: eventScope(record.inputIndex),
|
Scope: eventScope(record.inputIndex),
|
||||||
ReasonCode: ReasonCodeEventsReordered,
|
ReasonCode: ReasonCodeEventsReordered,
|
||||||
Message: fmt.Sprintf("input index %d moved to normalized position %d", record.inputIndex, position),
|
Message: fmt.Sprintf("input index %d moved to normalized position %d", record.inputIndex, position),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
output, duplicateWarnings := collapseDuplicates(records)
|
output, duplicateFindings := collapseDuplicates(records)
|
||||||
warnings = append(warnings, duplicateWarnings...)
|
findings = append(findings, duplicateFindings...)
|
||||||
return dnd.EnemyEventList{Events: output}, diagnostics.LimitWarnings(warnings, "enemy_events", ReasonCodeWarningsOmitted)
|
return dnd.EnemyEventList{Events: output}, findings
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeEvent(input dnd.EnemyEvent, order shared.SourceRefOrder, registry *npcregistry.Registry) (dnd.EnemyEvent, *nameCanonicalization, bool) {
|
func normalizeEvent(input dnd.EnemyEvent, order shared.SourceRefOrder, registry *npcregistry.Registry) (dnd.EnemyEvent, *nameCanonicalization, bool) {
|
||||||
@@ -205,7 +208,7 @@ type duplicateGroup struct {
|
|||||||
removed []int
|
removed []int
|
||||||
}
|
}
|
||||||
|
|
||||||
func collapseDuplicates(records []normalizedRecord) ([]dnd.EnemyEvent, []contracts.Warning) {
|
func collapseDuplicates(records []normalizedRecord) ([]dnd.EnemyEvent, []diagnostics.Finding) {
|
||||||
if len(records) == 0 {
|
if len(records) == 0 {
|
||||||
return make([]dnd.EnemyEvent, 0), nil
|
return make([]dnd.EnemyEvent, 0), nil
|
||||||
}
|
}
|
||||||
@@ -232,7 +235,7 @@ func collapseDuplicates(records []normalizedRecord) ([]dnd.EnemyEvent, []contrac
|
|||||||
for index, record := range kept {
|
for index, record := range kept {
|
||||||
output[index] = cloneEvent(record.event)
|
output[index] = cloneEvent(record.event)
|
||||||
}
|
}
|
||||||
warnings := make([]contracts.Warning, 0)
|
findings := make([]diagnostics.Finding, 0)
|
||||||
for _, group := range groups {
|
for _, group := range groups {
|
||||||
if len(group.removed) == 0 {
|
if len(group.removed) == 0 {
|
||||||
continue
|
continue
|
||||||
@@ -241,14 +244,14 @@ func collapseDuplicates(records []normalizedRecord) ([]dnd.EnemyEvent, []contrac
|
|||||||
for index, removed := range group.removed {
|
for index, removed := range group.removed {
|
||||||
issues[index] = fmt.Sprintf("removed input index %d", removed)
|
issues[index] = fmt.Sprintf("removed input index %d", removed)
|
||||||
}
|
}
|
||||||
warnings = append(warnings, contracts.Warning{
|
findings = append(findings, diagnostics.Finding{
|
||||||
Scope: eventScope(group.retainedIndex),
|
Scope: eventScope(group.retainedIndex),
|
||||||
ReasonCode: ReasonCodeDuplicateCollapsed,
|
ReasonCode: ReasonCodeDuplicateCollapsed,
|
||||||
Message: diagnostics.Aggregate(
|
Message: diagnostics.Aggregate(
|
||||||
fmt.Sprintf("duplicate enemy event collapsed; retained input index %d", group.retainedIndex), issues),
|
fmt.Sprintf("duplicate enemy event collapsed; retained input index %d", group.retainedIndex), issues),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
return output, warnings
|
return output, findings
|
||||||
}
|
}
|
||||||
|
|
||||||
func eventScope(index int) string { return fmt.Sprintf("events[%d]", index) }
|
func eventScope(index int) string { return fmt.Sprintf("events[%d]", index) }
|
||||||
|
|||||||
@@ -13,7 +13,6 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcregistry"
|
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcregistry"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestNormalizeCanonicalizesSubjectsEvidenceOrderAndDuplicates(t *testing.T) {
|
func TestNormalizeCanonicalizesSubjectsEvidenceOrderAndDuplicates(t *testing.T) {
|
||||||
@@ -38,8 +37,8 @@ func TestNormalizeCanonicalizesSubjectsEvidenceOrderAndDuplicates(t *testing.T)
|
|||||||
t.Fatalf("normalized events = %#v", got)
|
t.Fatalf("normalized events = %#v", got)
|
||||||
}
|
}
|
||||||
for _, reason := range []string{ReasonCodeNameCanonicalized, ReasonCodeSourceRefsNormalized, ReasonCodeEventsReordered, ReasonCodeDuplicateCollapsed} {
|
for _, reason := range []string{ReasonCodeNameCanonicalized, ReasonCodeSourceRefsNormalized, ReasonCodeEventsReordered, ReasonCodeDuplicateCollapsed} {
|
||||||
if !hasWarning(result.Warnings, reason) {
|
if !hasDiagnostic(result.Diagnostics, reason) {
|
||||||
t.Fatalf("warnings = %#v, missing %q", result.Warnings, reason)
|
t.Fatalf("diagnostics = %#v, missing %q", result.Diagnostics, reason)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(input.Events[0].SourceRefs, originalRefs) {
|
if !reflect.DeepEqual(input.Events[0].SourceRefs, originalRefs) {
|
||||||
@@ -93,7 +92,7 @@ func TestNormalizePreservesDistinctEvidenceWhenEncodedKeysCoincide(t *testing.T)
|
|||||||
}}
|
}}
|
||||||
|
|
||||||
result, err := newNormalizer(t, npcReferences(t)).Normalize(context.Background(), normalizeRequest(document, input, contracts.ReferenceSet{}))
|
result, err := newNormalizer(t, npcReferences(t)).Normalize(context.Background(), normalizeRequest(document, input, contracts.ReferenceSet{}))
|
||||||
if err != nil || len(result.Value.Events) != 2 || hasWarning(result.Warnings, ReasonCodeDuplicateCollapsed) {
|
if err != nil || len(result.Value.Events) != 2 || hasDiagnostic(result.Diagnostics, ReasonCodeDuplicateCollapsed) {
|
||||||
t.Fatalf("Normalize() = %#v, %v; want distinct evidence observations retained", result, err)
|
t.Fatalf("Normalize() = %#v, %v; want distinct evidence observations retained", result, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -112,7 +111,7 @@ func TestNormalizeIsIdempotentAndPreservesEmptyRepresentation(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
second, err := normalizer.Normalize(context.Background(), normalizeRequest(testDocument(), first.Value, contracts.ReferenceSet{}))
|
second, err := normalizer.Normalize(context.Background(), normalizeRequest(testDocument(), first.Value, contracts.ReferenceSet{}))
|
||||||
if err != nil || !reflect.DeepEqual(second.Value, first.Value) || len(second.Warnings) != 0 {
|
if err != nil || !reflect.DeepEqual(second.Value, first.Value) || len(second.Diagnostics) != 0 {
|
||||||
t.Fatalf("second normalization = %#v, %v", second, err)
|
t.Fatalf("second normalization = %#v, %v", second, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -135,7 +134,7 @@ func TestNormalizeRequiresRegistryAndKeepsOperationContentOutOfMetadata(t *testi
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestNormalizerContractAndWarningBound(t *testing.T) {
|
func TestNormalizerContractAndFindingBound(t *testing.T) {
|
||||||
normalizer := newNormalizer(t, npcReferences(t))
|
normalizer := newNormalizer(t, npcReferences(t))
|
||||||
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
||||||
t.Fatal("DecodeOptions() accepted an unknown option")
|
t.Fatal("DecodeOptions() accepted an unknown option")
|
||||||
@@ -162,7 +161,7 @@ func TestNormalizerContractAndWarningBound(t *testing.T) {
|
|||||||
t.Fatalf("unsafe metadata = %s, %v", encoded, err)
|
t.Fatalf("unsafe metadata = %s, %v", encoded, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
count := diagnostics.MaxWarnings + 5
|
count := contracts.MaxDiagnosticSamples + 5
|
||||||
document := &source.SourceDocument{ID: "session", Units: make([]source.SourceUnit, count)}
|
document := &source.SourceDocument{ID: "session", Units: make([]source.SourceUnit, count)}
|
||||||
input := dnd.EnemyEventList{Events: make([]dnd.EnemyEvent, count)}
|
input := dnd.EnemyEventList{Events: make([]dnd.EnemyEvent, count)}
|
||||||
for index := range document.Units {
|
for index := range document.Units {
|
||||||
@@ -170,8 +169,8 @@ func TestNormalizerContractAndWarningBound(t *testing.T) {
|
|||||||
input.Events[index] = dnd.EnemyEvent{Name: " ÁRIA ", Kind: dnd.EnemyEventKindEngaged, SourceRefs: []source.SourceRef{{SourceID: document.ID, StartUnitID: count - index, EndUnitID: count - index}}}
|
input.Events[index] = dnd.EnemyEvent{Name: " ÁRIA ", Kind: dnd.EnemyEventKindEngaged, SourceRefs: []source.SourceRef{{SourceID: document.ID, StartUnitID: count - index, EndUnitID: count - index}}}
|
||||||
}
|
}
|
||||||
result, err := normalizer.Normalize(context.Background(), normalizeRequest(document, input, contracts.ReferenceSet{}))
|
result, err := normalizer.Normalize(context.Background(), normalizeRequest(document, input, contracts.ReferenceSet{}))
|
||||||
if err != nil || len(result.Warnings) != diagnostics.MaxWarnings || result.Warnings[len(result.Warnings)-1].ReasonCode != ReasonCodeWarningsOmitted {
|
if err != nil || len(result.Diagnostics) == 0 {
|
||||||
t.Fatalf("warnings = %#v, %v", result.Warnings, err)
|
t.Fatalf("diagnostics = %#v, %v", result.Diagnostics, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -212,9 +211,9 @@ func npcReferences(t *testing.T) contracts.ReferenceSet {
|
|||||||
}}}
|
}}}
|
||||||
}
|
}
|
||||||
|
|
||||||
func hasWarning(warnings []contracts.Warning, reason string) bool {
|
func hasDiagnostic(diagnostics []contracts.ProducerDiagnostic, reason string) bool {
|
||||||
for _, warning := range warnings {
|
for _, diagnostic := range diagnostics {
|
||||||
if warning.ReasonCode == reason {
|
if diagnostic.ReasonCode == reason && diagnostic.Disposition == contracts.DiagnosticDispositionObservation {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -26,7 +26,6 @@ const (
|
|||||||
ReasonCodeSourceRefsNormalized = "source_references_normalized"
|
ReasonCodeSourceRefsNormalized = "source_references_normalized"
|
||||||
ReasonCodeOccurrencesReordered = "item_occurrences_reordered"
|
ReasonCodeOccurrencesReordered = "item_occurrences_reordered"
|
||||||
ReasonCodeDuplicateCollapsed = "duplicate_item_occurrence_collapsed"
|
ReasonCodeDuplicateCollapsed = "duplicate_item_occurrence_collapsed"
|
||||||
ReasonCodeWarningsOmitted = "item_occurrence_normalization_warnings_omitted"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -107,8 +106,12 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
return contracts.TypedNormalizeResult[dnd.ItemOccurrenceList]{}, normalizerErrorf("item registry reference is required")
|
return contracts.TypedNormalizeResult[dnd.ItemOccurrenceList]{}, normalizerErrorf("item registry reference is required")
|
||||||
}
|
}
|
||||||
index := source.NewDocumentIndex(req.Source)
|
index := source.NewDocumentIndex(req.Source)
|
||||||
value, warnings := normalizeList(req.MergeOutput.Value, index, shared.NewSourceRefOrderFromIndex(index), registry)
|
value, findings := normalizeList(req.MergeOutput.Value, index, shared.NewSourceRefOrderFromIndex(index), registry)
|
||||||
return contracts.TypedNormalizeResult[dnd.ItemOccurrenceList]{Value: value, Warnings: warnings}, nil
|
diagnosticGroups, err := diagnostics.NormalizationDiagnostics(findings, ReasonCodeUnknownItemID)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.ItemOccurrenceList]{}, normalizerErrorf("collect diagnostics: %w", err)
|
||||||
|
}
|
||||||
|
return contracts.TypedNormalizeResult[dnd.ItemOccurrenceList]{Value: value, Diagnostics: diagnosticGroups}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type normalizedRecord struct {
|
type normalizedRecord struct {
|
||||||
@@ -117,18 +120,18 @@ type normalizedRecord struct {
|
|||||||
inputIndex int
|
inputIndex int
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeList(input dnd.ItemOccurrenceList, index source.DocumentIndex, order shared.SourceRefOrder, registry *itemregistry.Registry) (dnd.ItemOccurrenceList, []contracts.Warning) {
|
func normalizeList(input dnd.ItemOccurrenceList, index source.DocumentIndex, order shared.SourceRefOrder, registry *itemregistry.Registry) (dnd.ItemOccurrenceList, []diagnostics.Finding) {
|
||||||
if input.Occurrences == nil {
|
if input.Occurrences == nil {
|
||||||
return dnd.ItemOccurrenceList{}, nil
|
return dnd.ItemOccurrenceList{}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
records := make([]normalizedRecord, len(input.Occurrences))
|
records := make([]normalizedRecord, len(input.Occurrences))
|
||||||
warnings := make([]contracts.Warning, 0)
|
findings := make([]diagnostics.Finding, 0)
|
||||||
for index, inputOccurrence := range input.Occurrences {
|
for index, inputOccurrence := range input.Occurrences {
|
||||||
occurrence, changedFields, found, refsChanged := normalizeOccurrence(inputOccurrence, order, registry)
|
occurrence, changedFields, found, refsChanged := normalizeOccurrence(inputOccurrence, order, registry)
|
||||||
records[index] = normalizedRecord{occurrence: occurrence, identity: itemoccurrencemodel.CanonicalExactIdentity(occurrence), inputIndex: index}
|
records[index] = normalizedRecord{occurrence: occurrence, identity: itemoccurrencemodel.CanonicalExactIdentity(occurrence), inputIndex: index}
|
||||||
if len(changedFields) != 0 {
|
if len(changedFields) != 0 {
|
||||||
warnings = append(warnings, contracts.Warning{
|
findings = append(findings, diagnostics.Finding{
|
||||||
Scope: occurrenceScope(index),
|
Scope: occurrenceScope(index),
|
||||||
ReasonCode: ReasonCodeNameCanonicalized,
|
ReasonCode: ReasonCodeNameCanonicalized,
|
||||||
Message: fmt.Sprintf("input index %d: normalized display whitespace in %s", index,
|
Message: fmt.Sprintf("input index %d: normalized display whitespace in %s", index,
|
||||||
@@ -136,15 +139,15 @@ func normalizeList(input dnd.ItemOccurrenceList, index source.DocumentIndex, ord
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
if found && inputOccurrence.Name != occurrence.Name {
|
if found && inputOccurrence.Name != occurrence.Name {
|
||||||
warnings = append(warnings, contracts.Warning{Scope: occurrenceScope(index), ReasonCode: ReasonCodeNameCanonicalized,
|
findings = append(findings, diagnostics.Finding{Scope: occurrenceScope(index), ReasonCode: ReasonCodeNameCanonicalized,
|
||||||
Message: fmt.Sprintf("input index %d: item name canonicalized from %s to %s", index, diagnostics.Quote(inputOccurrence.Name), diagnostics.Quote(occurrence.Name))})
|
Message: fmt.Sprintf("input index %d: item name canonicalized from %s to %s", index, diagnostics.Quote(inputOccurrence.Name), diagnostics.Quote(occurrence.Name))})
|
||||||
}
|
}
|
||||||
if !found {
|
if !found {
|
||||||
warnings = append(warnings, contracts.Warning{Scope: occurrenceScope(index), ReasonCode: ReasonCodeUnknownItemID,
|
findings = append(findings, diagnostics.Finding{Scope: occurrenceScope(index), ReasonCode: ReasonCodeUnknownItemID,
|
||||||
Message: fmt.Sprintf("input index %d: item ID %s is not in the supplied registry", index, diagnostics.Quote(inputOccurrence.ItemID))})
|
Message: fmt.Sprintf("input index %d: item ID %s is not in the supplied registry", index, diagnostics.Quote(inputOccurrence.ItemID))})
|
||||||
}
|
}
|
||||||
if refsChanged {
|
if refsChanged {
|
||||||
warnings = append(warnings, contracts.Warning{
|
findings = append(findings, diagnostics.Finding{
|
||||||
Scope: occurrenceScope(index),
|
Scope: occurrenceScope(index),
|
||||||
ReasonCode: ReasonCodeSourceRefsNormalized,
|
ReasonCode: ReasonCodeSourceRefsNormalized,
|
||||||
Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)",
|
Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)",
|
||||||
@@ -160,16 +163,16 @@ func normalizeList(input dnd.ItemOccurrenceList, index source.DocumentIndex, ord
|
|||||||
if position == record.inputIndex {
|
if position == record.inputIndex {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
warnings = append(warnings, contracts.Warning{
|
findings = append(findings, diagnostics.Finding{
|
||||||
Scope: occurrenceScope(record.inputIndex),
|
Scope: occurrenceScope(record.inputIndex),
|
||||||
ReasonCode: ReasonCodeOccurrencesReordered,
|
ReasonCode: ReasonCodeOccurrencesReordered,
|
||||||
Message: fmt.Sprintf("input index %d moved to normalized position %d", record.inputIndex, position),
|
Message: fmt.Sprintf("input index %d moved to normalized position %d", record.inputIndex, position),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
output, duplicateWarnings := collapseDuplicates(records, index)
|
output, duplicateFindings := collapseDuplicates(records, index)
|
||||||
warnings = append(warnings, duplicateWarnings...)
|
findings = append(findings, duplicateFindings...)
|
||||||
return dnd.ItemOccurrenceList{Occurrences: output}, diagnostics.LimitWarnings(warnings, "item_occurrences", ReasonCodeWarningsOmitted)
|
return dnd.ItemOccurrenceList{Occurrences: output}, findings
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeOccurrence(input dnd.ItemOccurrence, order shared.SourceRefOrder, registry *itemregistry.Registry) (dnd.ItemOccurrence, []string, bool, bool) {
|
func normalizeOccurrence(input dnd.ItemOccurrence, order shared.SourceRefOrder, registry *itemregistry.Registry) (dnd.ItemOccurrence, []string, bool, bool) {
|
||||||
@@ -214,7 +217,7 @@ type duplicateGroup struct {
|
|||||||
removed []int
|
removed []int
|
||||||
}
|
}
|
||||||
|
|
||||||
func collapseDuplicates(records []normalizedRecord, index source.DocumentIndex) ([]dnd.ItemOccurrence, []contracts.Warning) {
|
func collapseDuplicates(records []normalizedRecord, index source.DocumentIndex) ([]dnd.ItemOccurrence, []diagnostics.Finding) {
|
||||||
if len(records) == 0 {
|
if len(records) == 0 {
|
||||||
return make([]dnd.ItemOccurrence, 0), nil
|
return make([]dnd.ItemOccurrence, 0), nil
|
||||||
}
|
}
|
||||||
@@ -250,21 +253,21 @@ func collapseDuplicates(records []normalizedRecord, index source.DocumentIndex)
|
|||||||
output = append(output, cloneOccurrence(record.occurrence))
|
output = append(output, cloneOccurrence(record.occurrence))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
warnings := make([]contracts.Warning, 0)
|
findings := make([]diagnostics.Finding, 0)
|
||||||
for _, group := range groups {
|
for _, group := range groups {
|
||||||
if len(group.removed) != 0 {
|
if len(group.removed) != 0 {
|
||||||
warnings = append(warnings, duplicateWarning(group.retainedIndex, group.removed))
|
findings = append(findings, duplicateFinding(group.retainedIndex, group.removed))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return output, warnings
|
return output, findings
|
||||||
}
|
}
|
||||||
|
|
||||||
func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
|
func duplicateFinding(retainedIndex int, removed []int) diagnostics.Finding {
|
||||||
issues := make([]string, len(removed))
|
issues := make([]string, len(removed))
|
||||||
for index, removedIndex := range removed {
|
for index, removedIndex := range removed {
|
||||||
issues[index] = fmt.Sprintf("removed input index %d", removedIndex)
|
issues[index] = fmt.Sprintf("removed input index %d", removedIndex)
|
||||||
}
|
}
|
||||||
return contracts.Warning{
|
return diagnostics.Finding{
|
||||||
Scope: occurrenceScope(retainedIndex),
|
Scope: occurrenceScope(retainedIndex),
|
||||||
ReasonCode: ReasonCodeDuplicateCollapsed,
|
ReasonCode: ReasonCodeDuplicateCollapsed,
|
||||||
Message: diagnostics.Aggregate(
|
Message: diagnostics.Aggregate(
|
||||||
|
|||||||
@@ -40,8 +40,8 @@ func TestNormalizeCanonicalizesRegistryNameAndRetainsUnknownValues(t *testing.T)
|
|||||||
if result.Value.Occurrences[1].Name != "Torch" || result.Value.Occurrences[0].Name != "Unknown" {
|
if result.Value.Occurrences[1].Name != "Torch" || result.Value.Occurrences[0].Name != "Unknown" {
|
||||||
t.Fatalf("occurrences = %#v", result.Value.Occurrences)
|
t.Fatalf("occurrences = %#v", result.Value.Occurrences)
|
||||||
}
|
}
|
||||||
if !hasWarning(result.Warnings, ReasonCodeNameCanonicalized) || !hasWarning(result.Warnings, ReasonCodeUnknownItemID) {
|
if !hasDiagnostic(result.Diagnostics, ReasonCodeNameCanonicalized, contracts.DiagnosticDispositionObservation) || !hasDiagnostic(result.Diagnostics, ReasonCodeUnknownItemID, contracts.DiagnosticDispositionAdvisory) {
|
||||||
t.Fatalf("warnings = %#v", result.Warnings)
|
t.Fatalf("diagnostics = %#v", result.Diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -63,9 +63,9 @@ func TestNormalizeRequiresRegistryAndRegistersSlot(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func hasWarning(warnings []contracts.Warning, reason string) bool {
|
func hasDiagnostic(diagnostics []contracts.ProducerDiagnostic, reason string, disposition contracts.DiagnosticDisposition) bool {
|
||||||
for _, warning := range warnings {
|
for _, diagnostic := range diagnostics {
|
||||||
if warning.ReasonCode == reason {
|
if diagnostic.ReasonCode == reason && diagnostic.Disposition == disposition {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -31,8 +31,8 @@ const (
|
|||||||
ReasonCodeSourceReferencesNormalized = "source_references_normalized"
|
ReasonCodeSourceReferencesNormalized = "source_references_normalized"
|
||||||
ReasonCodeDuplicateItemCollapsed = "duplicate_item_collapsed"
|
ReasonCodeDuplicateItemCollapsed = "duplicate_item_collapsed"
|
||||||
ReasonCodeItemSemanticProposalInvalid = "item_semantic_proposal_invalid"
|
ReasonCodeItemSemanticProposalInvalid = "item_semantic_proposal_invalid"
|
||||||
|
ReasonCodeItemSemanticRetryProposalInvalid = "item_semantic_retry_proposal_invalid"
|
||||||
ReasonCodeItemSemanticReconciliationExhausted = "item_semantic_reconciliation_exhausted"
|
ReasonCodeItemSemanticReconciliationExhausted = "item_semantic_reconciliation_exhausted"
|
||||||
ReasonCodeItemNormalizationWarningsOmitted = "item_normalization_warnings_omitted"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
var requiredCapabilities = []string{"merged"}
|
var requiredCapabilities = []string{"merged"}
|
||||||
@@ -107,10 +107,10 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
}
|
}
|
||||||
|
|
||||||
order := shared.NewSourceRefOrder(req.Source)
|
order := shared.NewSourceRefOrder(req.Source)
|
||||||
records, warnings := preprocessRecords(req.MergeOutput.Value, order)
|
records, findings := preprocessRecords(req.MergeOutput.Value, order)
|
||||||
deterministic := recordList(records)
|
deterministic := recordList(records)
|
||||||
if len(records) < 2 {
|
if len(records) < 2 {
|
||||||
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{Value: deterministic, Warnings: limitWarnings(warnings)}, nil
|
return normalizationResult(deterministic, findings, nil, nil)
|
||||||
}
|
}
|
||||||
|
|
||||||
candidates, envelopes, err := reconciliationInputs(records)
|
candidates, envelopes, err := reconciliationInputs(records)
|
||||||
@@ -127,73 +127,86 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
|
|
||||||
switch reconciliation.Disposition() {
|
switch reconciliation.Disposition() {
|
||||||
case semanticreconcile.SkippedInsufficientCandidates:
|
case semanticreconcile.SkippedInsufficientCandidates:
|
||||||
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{Value: deterministic, Warnings: limitWarnings(warnings)}, nil
|
return normalizationResult(deterministic, findings, nil, nil)
|
||||||
case semanticreconcile.SkippedLimitExceeded:
|
case semanticreconcile.SkippedLimitExceeded:
|
||||||
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{Value: deterministic, Warnings: limitWarningsWithSemanticFallback(warnings)}, nil
|
return fallbackResult(deterministic, findings, nil, semanticFallbackFinding(-1))
|
||||||
case semanticreconcile.RetryableInvalidStructuredOutput:
|
case semanticreconcile.RetryableInvalidStructuredOutput:
|
||||||
return n.invalidStructuredResult(deterministic, warnings), nil
|
return n.invalidStructuredResult(deterministic, findings)
|
||||||
case semanticreconcile.Complete, semanticreconcile.RetryableDiscardedProposalGroups:
|
case semanticreconcile.Complete, semanticreconcile.RetryableDiscardedProposalGroups:
|
||||||
default:
|
default:
|
||||||
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, normalizerErrorf("unknown semantic reconciliation disposition %d", reconciliation.Disposition())
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, normalizerErrorf("unknown semantic reconciliation disposition %d", reconciliation.Disposition())
|
||||||
}
|
}
|
||||||
|
|
||||||
applied, semanticWarnings, rejectedGroups, err := applyReconciliationPlan(reconciliation.Plan(), records, envelopes, order)
|
applied, semanticFindings, advisoryFindings, rejectedGroups, err := applyReconciliationPlan(reconciliation.Plan(), records, envelopes, order)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, normalizerErrorf("apply semantic reconciliation plan: %w", err)
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, normalizerErrorf("apply semantic reconciliation plan: %w", err)
|
||||||
}
|
}
|
||||||
warnings = append(warnings, semanticWarnings...)
|
findings = append(findings, semanticFindings...)
|
||||||
discardedGroups := reconciliation.DiscardedGroupCount() + rejectedGroups
|
discardedGroups := reconciliation.DiscardedGroupCount() + rejectedGroups
|
||||||
if discardedGroups == 0 {
|
if discardedGroups == 0 {
|
||||||
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{Value: recordList(applied), Warnings: limitWarnings(warnings), ModelCandidate: reconciliation.ModelCandidate()}, nil
|
return normalizationResult(recordList(applied), findings, advisoryFindings, reconciliation.ModelCandidate())
|
||||||
}
|
}
|
||||||
return retryResult(recordList(applied), warnings, reconciliation, rejectedGroups), nil
|
return retryResult(recordList(applied), findings, advisoryFindings, reconciliation, rejectedGroups)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (n *Normalizer) invalidStructuredResult(value dnd.ItemRegistry, warnings []contracts.Warning) contracts.TypedNormalizeResult[dnd.ItemRegistry] {
|
func (n *Normalizer) invalidStructuredResult(value dnd.ItemRegistry, findings []diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.ItemRegistry], error) {
|
||||||
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{Value: value, Warnings: limitWarningsForRetry(warnings), Retry: &contracts.NormalizeRetry{
|
return retryResultWithFallback(value, findings, nil, nil, ReasonCodeItemSemanticRetryProposalInvalid, "semantic proposal requires retry: invalid structured output", semanticFallbackFinding(-1))
|
||||||
ReasonCode: ReasonCodeItemSemanticProposalInvalid, Message: "semantic proposal requires retry: invalid structured output",
|
|
||||||
FallbackWarnings: []contracts.Warning{semanticFallbackWarning(-1)},
|
|
||||||
}}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func retryResult(value dnd.ItemRegistry, warnings []contracts.Warning, reconciliation semanticreconcile.Result, rejectedGroups int) contracts.TypedNormalizeResult[dnd.ItemRegistry] {
|
func retryResult(value dnd.ItemRegistry, findings, advisoryFindings []diagnostics.Finding, reconciliation semanticreconcile.Result, rejectedGroups int) (contracts.TypedNormalizeResult[dnd.ItemRegistry], error) {
|
||||||
details := semanticreconcile.IssueDetails(reconciliation.Issues())
|
details := semanticreconcile.IssueDetails(reconciliation.Issues())
|
||||||
if rejectedGroups > 0 {
|
if rejectedGroups > 0 {
|
||||||
details = append(details, "currency may only be consolidated with aliases of one denomination")
|
details = append(details, "currency may only be consolidated with aliases of one denomination")
|
||||||
}
|
}
|
||||||
discardedGroups := reconciliation.DiscardedGroupCount() + rejectedGroups
|
discardedGroups := reconciliation.DiscardedGroupCount() + rejectedGroups
|
||||||
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{Value: value, Warnings: limitWarningsForRetry(warnings), ModelCandidate: reconciliation.ModelCandidate(), Retry: &contracts.NormalizeRetry{
|
return retryResultWithFallback(value, findings, advisoryFindings, reconciliation.ModelCandidate(), ReasonCodeItemSemanticRetryProposalInvalid, diagnostics.Aggregate("semantic proposal requires retry", details), semanticFallbackFinding(discardedGroups))
|
||||||
ReasonCode: ReasonCodeItemSemanticProposalInvalid, Message: diagnostics.Aggregate("semantic proposal requires retry", details),
|
|
||||||
FallbackWarnings: []contracts.Warning{semanticFallbackWarning(discardedGroups)},
|
|
||||||
}}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func semanticFallbackWarning(discarded int) contracts.Warning {
|
func semanticFallbackFinding(discarded int) diagnostics.Finding {
|
||||||
message := "semantic proposal could not be applied"
|
message := "semantic proposal could not be applied"
|
||||||
if discarded >= 0 {
|
if discarded >= 0 {
|
||||||
message = fmt.Sprintf("%d proposal group(s) omitted after semantic proposal retry exhaustion", discarded)
|
message = fmt.Sprintf("%d proposal group(s) omitted after semantic proposal retry exhaustion", discarded)
|
||||||
}
|
}
|
||||||
return contracts.Warning{Scope: "items", ReasonCode: ReasonCodeItemSemanticReconciliationExhausted, Message: message}
|
return diagnostics.Finding{Scope: "items", ReasonCode: ReasonCodeItemSemanticReconciliationExhausted, Message: message}
|
||||||
}
|
}
|
||||||
|
|
||||||
func limitWarnings(warnings []contracts.Warning) []contracts.Warning {
|
func normalizationResult(value dnd.ItemRegistry, findings, advisoryFindings []diagnostics.Finding, candidate *contracts.ModelCandidate) (contracts.TypedNormalizeResult[dnd.ItemRegistry], error) {
|
||||||
return diagnostics.LimitWarnings(warnings, "items", ReasonCodeItemNormalizationWarningsOmitted)
|
diagnosticGroups, err := diagnostics.Collect(findings, contracts.DiagnosticDispositionObservation, contracts.DiagnosticCategoryNormalization)
|
||||||
}
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, normalizerErrorf("collect normalization diagnostics: %w", err)
|
||||||
func limitWarningsForRetry(warnings []contracts.Warning) []contracts.Warning {
|
|
||||||
if warnings == nil {
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
if len(warnings) < diagnostics.MaxWarnings {
|
advisoryGroups, err := diagnostics.Collect(advisoryFindings, contracts.DiagnosticDispositionAdvisory, contracts.DiagnosticCategoryDataQuality)
|
||||||
return append([]contracts.Warning(nil), warnings...)
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, normalizerErrorf("collect data-quality diagnostics: %w", err)
|
||||||
}
|
}
|
||||||
displayed := diagnostics.MaxWarnings - 2
|
diagnosticGroups = append(diagnosticGroups, advisoryGroups...)
|
||||||
bounded := append([]contracts.Warning(nil), warnings[:displayed]...)
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{Value: value, Diagnostics: diagnosticGroups, ModelCandidate: candidate}, nil
|
||||||
return append(bounded, contracts.Warning{Scope: "items", ReasonCode: ReasonCodeItemNormalizationWarningsOmitted, Message: fmt.Sprintf("%d additional warning(s) omitted", len(warnings)-displayed)})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func limitWarningsWithSemanticFallback(warnings []contracts.Warning) []contracts.Warning {
|
func fallbackResult(value dnd.ItemRegistry, findings, advisoryFindings []diagnostics.Finding, fallback diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.ItemRegistry], error) {
|
||||||
return append(limitWarningsForRetry(warnings), semanticFallbackWarning(-1))
|
result, err := normalizationResult(value, findings, advisoryFindings, nil)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, err
|
||||||
|
}
|
||||||
|
fallbackGroups, err := diagnostics.Collect([]diagnostics.Finding{fallback}, contracts.DiagnosticDispositionWarning, contracts.DiagnosticCategoryFallback)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, normalizerErrorf("collect fallback diagnostic: %w", err)
|
||||||
|
}
|
||||||
|
result.Diagnostics = append(result.Diagnostics, fallbackGroups...)
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func retryResultWithFallback(value dnd.ItemRegistry, findings, advisoryFindings []diagnostics.Finding, candidate *contracts.ModelCandidate, reasonCode, message string, fallback diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.ItemRegistry], error) {
|
||||||
|
result, err := normalizationResult(value, findings, advisoryFindings, candidate)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, err
|
||||||
|
}
|
||||||
|
fallbackGroups, err := diagnostics.Collect([]diagnostics.Finding{fallback}, contracts.DiagnosticDispositionWarning, contracts.DiagnosticCategoryFallback)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.ItemRegistry]{}, normalizerErrorf("collect fallback diagnostic: %w", err)
|
||||||
|
}
|
||||||
|
result.Retry = &contracts.NormalizeRetry{ReasonCode: reasonCode, Message: message, FallbackDiagnostics: fallbackGroups}
|
||||||
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type normalizedRecord struct {
|
type normalizedRecord struct {
|
||||||
@@ -202,23 +215,23 @@ type normalizedRecord struct {
|
|||||||
earliest int
|
earliest int
|
||||||
}
|
}
|
||||||
|
|
||||||
func preprocessRecords(input dnd.ItemRegistry, order shared.SourceRefOrder) ([]normalizedRecord, []contracts.Warning) {
|
func preprocessRecords(input dnd.ItemRegistry, order shared.SourceRefOrder) ([]normalizedRecord, []diagnostics.Finding) {
|
||||||
if input.Items == nil {
|
if input.Items == nil {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
records := make([]normalizedRecord, len(input.Items))
|
records := make([]normalizedRecord, len(input.Items))
|
||||||
warnings := make([]contracts.Warning, 0)
|
findings := make([]diagnostics.Finding, 0)
|
||||||
for index, inputItem := range input.Items {
|
for index, inputItem := range input.Items {
|
||||||
item, fieldsChanged, refsChanged := normalizeRecord(inputItem, order)
|
item, fieldsChanged, refsChanged := normalizeRecord(inputItem, order)
|
||||||
records[index] = normalizedRecord{item: item, inputIndexes: []int{index}, earliest: index}
|
records[index] = normalizedRecord{item: item, inputIndexes: []int{index}, earliest: index}
|
||||||
if fieldsChanged {
|
if fieldsChanged {
|
||||||
warnings = append(warnings, contracts.Warning{Scope: itemScope(index), ReasonCode: ReasonCodeItemFieldsNormalized, Message: fmt.Sprintf("input index %d: item name normalized for %s", index, diagnostics.Quote(inputItem.Name))})
|
findings = append(findings, diagnostics.Finding{Scope: itemScope(index), ReasonCode: ReasonCodeItemFieldsNormalized, Message: fmt.Sprintf("input index %d: item name normalized for %s", index, diagnostics.Quote(inputItem.Name))})
|
||||||
}
|
}
|
||||||
if refsChanged {
|
if refsChanged {
|
||||||
warnings = append(warnings, contracts.Warning{Scope: itemScope(index), ReasonCode: ReasonCodeSourceReferencesNormalized, Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)", index, len(inputItem.SourceRefs), len(item.SourceRefs))})
|
findings = append(findings, diagnostics.Finding{Scope: itemScope(index), ReasonCode: ReasonCodeSourceReferencesNormalized, Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)", index, len(inputItem.SourceRefs), len(item.SourceRefs))})
|
||||||
}
|
}
|
||||||
if inputItem.ID != item.ID {
|
if inputItem.ID != item.ID {
|
||||||
warnings = append(warnings, contracts.Warning{Scope: itemScope(index), ReasonCode: ReasonCodeItemIDRecomputed, Message: fmt.Sprintf("input index %d: item ID recomputed from %s", index, diagnostics.Quote(item.Name))})
|
findings = append(findings, diagnostics.Finding{Scope: itemScope(index), ReasonCode: ReasonCodeItemIDRecomputed, Message: fmt.Sprintf("input index %d: item ID recomputed from %s", index, diagnostics.Quote(item.Name))})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
groups := comparisonNameGroups(records)
|
groups := comparisonNameGroups(records)
|
||||||
@@ -234,10 +247,10 @@ func preprocessRecords(input dnd.ItemRegistry, order shared.SourceRefOrder) ([]n
|
|||||||
retained.inputIndexes = sortedUniqueIndexes(retained.inputIndexes)
|
retained.inputIndexes = sortedUniqueIndexes(retained.inputIndexes)
|
||||||
output = append(output, retained)
|
output = append(output, retained)
|
||||||
if len(members) > 1 {
|
if len(members) > 1 {
|
||||||
warnings = append(warnings, duplicateWarning(retained.earliest, memberInputIndexes(records, members[1:])))
|
findings = append(findings, duplicateFinding(retained.earliest, memberInputIndexes(records, members[1:])))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return output, warnings
|
return output, findings
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeRecord(input dnd.Item, order shared.SourceRefOrder) (dnd.Item, bool, bool) {
|
func normalizeRecord(input dnd.Item, order shared.SourceRefOrder) (dnd.Item, bool, bool) {
|
||||||
@@ -315,7 +328,7 @@ func recordList(records []normalizedRecord) dnd.ItemRegistry {
|
|||||||
return dnd.ItemRegistry{Items: recordValues(records)}
|
return dnd.ItemRegistry{Items: recordValues(records)}
|
||||||
}
|
}
|
||||||
|
|
||||||
func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
|
func duplicateFinding(retainedIndex int, removed []int) diagnostics.Finding {
|
||||||
const maxDisplayedIndices = 20
|
const maxDisplayedIndices = 20
|
||||||
displayed := removed
|
displayed := removed
|
||||||
if len(displayed) > maxDisplayedIndices {
|
if len(displayed) > maxDisplayedIndices {
|
||||||
@@ -329,7 +342,7 @@ func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
|
|||||||
if omitted := len(removed) - len(displayed); omitted > 0 {
|
if omitted := len(removed) - len(displayed); omitted > 0 {
|
||||||
message += fmt.Sprintf("; %d additional removed input indices omitted", omitted)
|
message += fmt.Sprintf("; %d additional removed input indices omitted", omitted)
|
||||||
}
|
}
|
||||||
return contracts.Warning{Scope: itemScope(retainedIndex), ReasonCode: ReasonCodeDuplicateItemCollapsed, Message: message}
|
return diagnostics.Finding{Scope: itemScope(retainedIndex), ReasonCode: ReasonCodeDuplicateItemCollapsed, Message: message}
|
||||||
}
|
}
|
||||||
func itemScope(index int) string { return fmt.Sprintf("items[%d]", index) }
|
func itemScope(index int) string { return fmt.Sprintf("items[%d]", index) }
|
||||||
|
|
||||||
|
|||||||
@@ -18,7 +18,6 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/semanticreconcile"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/semanticreconcile"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/items/identity"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/items/identity"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
|
||||||
identityvalidator "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/itemregistry/identity"
|
identityvalidator "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/itemregistry/identity"
|
||||||
"gitea.maximumdirect.net/eric/promptkit"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
@@ -81,8 +80,8 @@ func TestNormalizeConsolidatesEqualNamesAcrossEvidenceWithoutMutation(t *testing
|
|||||||
}
|
}
|
||||||
rope := result.Value.Items[0]
|
rope := result.Value.Items[0]
|
||||||
wantRefs := []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}, {SourceID: "session", StartUnitID: 2, EndUnitID: 2}, {SourceID: "session", StartUnitID: 3, EndUnitID: 3}}
|
wantRefs := []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}, {SourceID: "session", StartUnitID: 2, EndUnitID: 2}, {SourceID: "session", StartUnitID: 3, EndUnitID: 3}}
|
||||||
if rope.Name != "Rope" || rope.ID != identity.DeriveID("Rope") || !reflect.DeepEqual(rope.SourceRefs, wantRefs) || result.Value.Items[1].Name != "Lantern" || !hasWarning(result.Warnings, ReasonCodeDuplicateItemCollapsed) {
|
if rope.Name != "Rope" || rope.ID != identity.DeriveID("Rope") || !reflect.DeepEqual(rope.SourceRefs, wantRefs) || result.Value.Items[1].Name != "Lantern" || !hasDiagnostic(result.Diagnostics, ReasonCodeDuplicateItemCollapsed, contracts.DiagnosticDispositionObservation) {
|
||||||
t.Fatalf("items = %#v, warnings = %#v; want earliest display name, canonical evidence union, and stable placement", result.Value.Items, result.Warnings)
|
t.Fatalf("items = %#v, diagnostics = %#v; want earliest display name, canonical evidence union, and stable placement", result.Value.Items, result.Diagnostics)
|
||||||
}
|
}
|
||||||
validation, validationErr := identityvalidator.New(identityvalidator.Options{}).Validate(context.Background(), contracts.TypedValidationRequest[dnd.ItemRegistry]{Value: result.Value})
|
validation, validationErr := identityvalidator.New(identityvalidator.Options{}).Validate(context.Background(), contracts.TypedValidationRequest[dnd.ItemRegistry]{Value: result.Value})
|
||||||
if validationErr != nil || !validation.Approved {
|
if validationErr != nil || !validation.Approved {
|
||||||
@@ -129,8 +128,8 @@ func TestNormalizeAppliesSafeAliasProposal(t *testing.T) {
|
|||||||
t.Fatalf("Normalize() = %#v, %v", result, err)
|
t.Fatalf("Normalize() = %#v, %v", result, err)
|
||||||
}
|
}
|
||||||
merged := result.Value.Items[0]
|
merged := result.Value.Items[0]
|
||||||
if merged.Name != "Compass of the Stars" || merged.ID != identity.DeriveID(merged.Name) || len(merged.SourceRefs) != 2 || !hasWarning(result.Warnings, ReasonCodeDuplicateItemCollapsed) {
|
if merged.Name != "Compass of the Stars" || merged.ID != identity.DeriveID(merged.Name) || len(merged.SourceRefs) != 2 || !hasDiagnostic(result.Diagnostics, ReasonCodeDuplicateItemCollapsed, contracts.DiagnosticDispositionObservation) {
|
||||||
t.Fatalf("merged item = %#v, warnings = %#v", merged, result.Warnings)
|
t.Fatalf("merged item = %#v, diagnostics = %#v", merged, result.Diagnostics)
|
||||||
}
|
}
|
||||||
encoded := string(client.requests[0].Inputs["candidates"].Content) + string(client.requests[0].Inputs["transcript"].Content)
|
encoded := string(client.requests[0].Inputs["candidates"].Content) + string(client.requests[0].Inputs["transcript"].Content)
|
||||||
if strings.Contains(encoded, doc.ID) || strings.Contains(encoded, "candidate-") || strings.Contains(encoded, merged.ID) || !strings.Contains(encoded, `"source_refs"`) {
|
if strings.Contains(encoded, doc.ID) || strings.Contains(encoded, "candidate-") || strings.Contains(encoded, merged.ID) || !strings.Contains(encoded, `"source_refs"`) {
|
||||||
@@ -150,7 +149,8 @@ func TestNormalizeAppliesCurrencyReconciliationSafely(t *testing.T) {
|
|||||||
wantNames []string
|
wantNames []string
|
||||||
wantRefCounts []int
|
wantRefCounts []int
|
||||||
wantRetry bool
|
wantRetry bool
|
||||||
warning string
|
reasonCode string
|
||||||
|
disposition contracts.DiagnosticDisposition
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "same denomination aliases",
|
name: "same denomination aliases",
|
||||||
@@ -162,7 +162,8 @@ func TestNormalizeAppliesCurrencyReconciliationSafely(t *testing.T) {
|
|||||||
response: `{"duplicate_groups":[{"candidate_ids":[1,2,3],"canonical_candidate_id":2}]}`,
|
response: `{"duplicate_groups":[{"candidate_ids":[1,2,3],"canonical_candidate_id":2}]}`,
|
||||||
wantNames: []string{"Gold Piece"},
|
wantNames: []string{"Gold Piece"},
|
||||||
wantRefCounts: []int{3},
|
wantRefCounts: []int{3},
|
||||||
warning: ReasonCodeDuplicateItemCollapsed,
|
reasonCode: ReasonCodeDuplicateItemCollapsed,
|
||||||
|
disposition: contracts.DiagnosticDispositionObservation,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "different denominations",
|
name: "different denominations",
|
||||||
@@ -174,7 +175,8 @@ func TestNormalizeAppliesCurrencyReconciliationSafely(t *testing.T) {
|
|||||||
wantNames: []string{"Gold Pieces", "Silver Pieces"},
|
wantNames: []string{"Gold Pieces", "Silver Pieces"},
|
||||||
wantRefCounts: []int{1, 1},
|
wantRefCounts: []int{1, 1},
|
||||||
wantRetry: true,
|
wantRetry: true,
|
||||||
warning: ReasonCodeItemSemanticProposalInvalid,
|
reasonCode: ReasonCodeItemSemanticProposalInvalid,
|
||||||
|
disposition: contracts.DiagnosticDispositionAdvisory,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "currency plus ordinary item",
|
name: "currency plus ordinary item",
|
||||||
@@ -186,7 +188,8 @@ func TestNormalizeAppliesCurrencyReconciliationSafely(t *testing.T) {
|
|||||||
wantNames: []string{"Gold Pieces", "Longsword"},
|
wantNames: []string{"Gold Pieces", "Longsword"},
|
||||||
wantRefCounts: []int{1, 1},
|
wantRefCounts: []int{1, 1},
|
||||||
wantRetry: true,
|
wantRetry: true,
|
||||||
warning: ReasonCodeItemSemanticProposalInvalid,
|
reasonCode: ReasonCodeItemSemanticProposalInvalid,
|
||||||
|
disposition: contracts.DiagnosticDispositionAdvisory,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "ordinary items",
|
name: "ordinary items",
|
||||||
@@ -197,7 +200,8 @@ func TestNormalizeAppliesCurrencyReconciliationSafely(t *testing.T) {
|
|||||||
response: `{"duplicate_groups":[{"candidate_ids":[1,2],"canonical_candidate_id":2}]}`,
|
response: `{"duplicate_groups":[{"candidate_ids":[1,2],"canonical_candidate_id":2}]}`,
|
||||||
wantNames: []string{"Compass of the Stars"},
|
wantNames: []string{"Compass of the Stars"},
|
||||||
wantRefCounts: []int{2},
|
wantRefCounts: []int{2},
|
||||||
warning: ReasonCodeDuplicateItemCollapsed,
|
reasonCode: ReasonCodeDuplicateItemCollapsed,
|
||||||
|
disposition: contracts.DiagnosticDispositionObservation,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "currency plus ordinary canonical item",
|
name: "currency plus ordinary canonical item",
|
||||||
@@ -209,7 +213,8 @@ func TestNormalizeAppliesCurrencyReconciliationSafely(t *testing.T) {
|
|||||||
wantNames: []string{"Gold Pieces", "Longsword"},
|
wantNames: []string{"Gold Pieces", "Longsword"},
|
||||||
wantRefCounts: []int{1, 1},
|
wantRefCounts: []int{1, 1},
|
||||||
wantRetry: true,
|
wantRetry: true,
|
||||||
warning: ReasonCodeItemSemanticProposalInvalid,
|
reasonCode: ReasonCodeItemSemanticProposalInvalid,
|
||||||
|
disposition: contracts.DiagnosticDispositionAdvisory,
|
||||||
},
|
},
|
||||||
} {
|
} {
|
||||||
t.Run(test.name, func(t *testing.T) {
|
t.Run(test.name, func(t *testing.T) {
|
||||||
@@ -223,7 +228,7 @@ func TestNormalizeAppliesCurrencyReconciliationSafely(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
result, err := newNormalizer(t, &recordingNormalizerClient{response: test.response}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
result, err := newNormalizer(t, &recordingNormalizerClient{response: test.response}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||||
if err != nil || (result.Retry != nil) != test.wantRetry || !reflect.DeepEqual(input, before) || !hasWarning(result.Warnings, test.warning) {
|
if err != nil || (result.Retry != nil) != test.wantRetry || !reflect.DeepEqual(input, before) || !hasDiagnostic(result.Diagnostics, test.reasonCode, test.disposition) {
|
||||||
t.Fatalf("Normalize() = %#v, %v", result, err)
|
t.Fatalf("Normalize() = %#v, %v", result, err)
|
||||||
}
|
}
|
||||||
if len(result.Value.Items) != len(test.wantNames) {
|
if len(result.Value.Items) != len(test.wantNames) {
|
||||||
@@ -234,7 +239,7 @@ func TestNormalizeAppliesCurrencyReconciliationSafely(t *testing.T) {
|
|||||||
t.Fatalf("item %d = %#v, want name %q with %d source refs", index, item, test.wantNames[index], test.wantRefCounts[index])
|
t.Fatalf("item %d = %#v, want name %q with %d source refs", index, item, test.wantNames[index], test.wantRefCounts[index])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if test.wantRetry && (len(result.Retry.FallbackWarnings) != 1 || result.Retry.FallbackWarnings[0].ReasonCode != ReasonCodeItemSemanticReconciliationExhausted) {
|
if test.wantRetry && (len(result.Retry.FallbackDiagnostics) != 1 || result.Retry.FallbackDiagnostics[0].ReasonCode != ReasonCodeItemSemanticReconciliationExhausted || result.Retry.ReasonCode != ReasonCodeItemSemanticRetryProposalInvalid) {
|
||||||
t.Fatalf("retry = %#v, want preserved-group fallback", result.Retry)
|
t.Fatalf("retry = %#v, want preserved-group fallback", result.Retry)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
@@ -281,10 +286,10 @@ func TestNormalizeAppliesIndependentGroupAndCountsAllOmissions(t *testing.T) {
|
|||||||
t.Fatalf("item %d = %#v, want %q", index, result.Value.Items[index], name)
|
t.Fatalf("item %d = %#v, want %q", index, result.Value.Items[index], name)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !hasWarning(result.Warnings, ReasonCodeDuplicateItemCollapsed) || !hasWarning(result.Warnings, ReasonCodeItemSemanticProposalInvalid) {
|
if !hasDiagnostic(result.Diagnostics, ReasonCodeDuplicateItemCollapsed, contracts.DiagnosticDispositionObservation) || !hasDiagnostic(result.Diagnostics, ReasonCodeItemSemanticProposalInvalid, contracts.DiagnosticDispositionAdvisory) {
|
||||||
t.Fatalf("warnings = %#v, want accepted and guarded-group diagnostics", result.Warnings)
|
t.Fatalf("diagnostics = %#v, want accepted and guarded-group diagnostics", result.Diagnostics)
|
||||||
}
|
}
|
||||||
if len(result.Retry.FallbackWarnings) != 1 || !strings.Contains(result.Retry.FallbackWarnings[0].Message, "2 proposal group(s)") {
|
if len(result.Retry.FallbackDiagnostics) != 1 || !strings.Contains(result.Retry.FallbackDiagnostics[0].Samples[0].Message, "2 proposal group(s)") {
|
||||||
t.Fatalf("retry = %#v, want one guarded and one malformed group counted", result.Retry)
|
t.Fatalf("retry = %#v, want one guarded and one malformed group counted", result.Retry)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -304,25 +309,22 @@ func TestNormalizeLimitSkipDoesNotCallLLMAndAddsBoundedFallbackWarning(t *testin
|
|||||||
if len(client.requests) != 0 || len(result.Value.Items) != limit+1 {
|
if len(client.requests) != 0 || len(result.Value.Items) != limit+1 {
|
||||||
t.Fatalf("completion calls = %d, items = %d; want no call and all records", len(client.requests), len(result.Value.Items))
|
t.Fatalf("completion calls = %d, items = %d; want no call and all records", len(client.requests), len(result.Value.Items))
|
||||||
}
|
}
|
||||||
if !hasWarning(result.Warnings, ReasonCodeItemSemanticReconciliationExhausted) || len(result.Warnings) > diagnostics.MaxWarnings {
|
if !hasDiagnostic(result.Diagnostics, ReasonCodeItemSemanticReconciliationExhausted, contracts.DiagnosticDispositionWarning) {
|
||||||
t.Fatalf("warnings = %#v, want bounded reconciliation fallback", result.Warnings)
|
t.Fatalf("diagnostics = %#v, want reconciliation fallback warning", result.Diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestNormalizeRetryFallbackErrorsWarningsAndIdempotence(t *testing.T) {
|
func TestNormalizeRetryFallbackErrorsAndIdempotence(t *testing.T) {
|
||||||
doc := semanticDocument()
|
doc := semanticDocument()
|
||||||
input := dnd.ItemRegistry{Items: []dnd.Item{{Name: "Star Compass", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}}, {Name: "Compass", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}}}}
|
input := dnd.ItemRegistry{Items: []dnd.Item{{Name: "Star Compass", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}}, {Name: "Compass", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}}}}
|
||||||
invalid, err := newNormalizer(t, &recordingNormalizerClient{err: contracts.ErrInvalidStructuredOutput}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
invalid, err := newNormalizer(t, &recordingNormalizerClient{err: contracts.ErrInvalidStructuredOutput}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||||
if err != nil || invalid.Retry == nil || invalid.Retry.ReasonCode != ReasonCodeItemSemanticProposalInvalid {
|
if err != nil || invalid.Retry == nil || invalid.Retry.ReasonCode != ReasonCodeItemSemanticRetryProposalInvalid {
|
||||||
t.Fatalf("invalid result = %#v, %v", invalid, err)
|
t.Fatalf("invalid result = %#v, %v", invalid, err)
|
||||||
}
|
}
|
||||||
_, err = newNormalizer(t, &recordingNormalizerClient{err: errors.New("provider unavailable")}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
_, err = newNormalizer(t, &recordingNormalizerClient{err: errors.New("provider unavailable")}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||||
if err == nil || !strings.Contains(err.Error(), "provider unavailable") {
|
if err == nil || !strings.Contains(err.Error(), "provider unavailable") {
|
||||||
t.Fatalf("provider error = %v", err)
|
t.Fatalf("provider error = %v", err)
|
||||||
}
|
}
|
||||||
if bounded := limitWarningsForRetry(make([]contracts.Warning, diagnostics.MaxWarnings+5)); len(bounded) != diagnostics.MaxWarnings-1 || bounded[len(bounded)-1].ReasonCode != ReasonCodeItemNormalizationWarningsOmitted {
|
|
||||||
t.Fatalf("retry warning limit = %#v", bounded)
|
|
||||||
}
|
|
||||||
first, err := newNormalizer(t, &recordingNormalizerClient{}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
first, err := newNormalizer(t, &recordingNormalizerClient{}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||||
second, secondErr := newNormalizer(t, &recordingNormalizerClient{}).Normalize(context.Background(), normalizeRequestWithSource(first.Value, doc))
|
second, secondErr := newNormalizer(t, &recordingNormalizerClient{}).Normalize(context.Background(), normalizeRequestWithSource(first.Value, doc))
|
||||||
if err != nil || secondErr != nil || !reflect.DeepEqual(first.Value, second.Value) {
|
if err != nil || secondErr != nil || !reflect.DeepEqual(first.Value, second.Value) {
|
||||||
@@ -407,9 +409,9 @@ func normalizeRequestWithSource(value dnd.ItemRegistry, doc *source.SourceDocume
|
|||||||
func semanticDocument() *source.SourceDocument {
|
func semanticDocument() *source.SourceDocument {
|
||||||
return &source.SourceDocument{ID: "item-session", Units: []source.SourceUnit{{ID: 10, Text: "The Star Compass points north."}, {ID: 20, Text: "The compass of the stars glows."}, {ID: 30, Text: "The chest holds gold pieces."}}}
|
return &source.SourceDocument{ID: "item-session", Units: []source.SourceUnit{{ID: 10, Text: "The Star Compass points north."}, {ID: 20, Text: "The compass of the stars glows."}, {ID: 30, Text: "The chest holds gold pieces."}}}
|
||||||
}
|
}
|
||||||
func hasWarning(warnings []contracts.Warning, reason string) bool {
|
func hasDiagnostic(diagnostics []contracts.ProducerDiagnostic, reason string, disposition contracts.DiagnosticDisposition) bool {
|
||||||
for _, warning := range warnings {
|
for _, diagnostic := range diagnostics {
|
||||||
if warning.ReasonCode == reason {
|
if diagnostic.ReasonCode == reason && diagnostic.Disposition == disposition {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"sort"
|
"sort"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/semanticreconcile"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/semanticreconcile"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/items/identity"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/items/identity"
|
||||||
@@ -31,7 +30,7 @@ func reconciliationInputs(records []normalizedRecord) ([]semanticreconcile.Candi
|
|||||||
return candidates, envelopes, nil
|
return candidates, envelopes, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func applyReconciliationPlan(plan semanticreconcile.Plan, records []normalizedRecord, envelopes []semanticreconcile.Record[dnd.Item], order shared.SourceRefOrder) ([]normalizedRecord, []contracts.Warning, int, error) {
|
func applyReconciliationPlan(plan semanticreconcile.Plan, records []normalizedRecord, envelopes []semanticreconcile.Record[dnd.Item], order shared.SourceRefOrder) ([]normalizedRecord, []diagnostics.Finding, []diagnostics.Finding, int, error) {
|
||||||
application, err := semanticreconcile.ApplyPlan(plan, envelopes, semanticreconcile.ApplicationPolicy[dnd.Item]{
|
application, err := semanticreconcile.ApplyPlan(plan, envelopes, semanticreconcile.ApplicationPolicy[dnd.Item]{
|
||||||
CloneValue: cloneItem,
|
CloneValue: cloneItem,
|
||||||
RejectGroup: func(members []dnd.Item, _ dnd.Item) semanticreconcile.RejectionCategory {
|
RejectGroup: func(members []dnd.Item, _ dnd.Item) semanticreconcile.RejectionCategory {
|
||||||
@@ -52,7 +51,7 @@ func applyReconciliationPlan(plan semanticreconcile.Plan, records []normalizedRe
|
|||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, 0, err
|
return nil, nil, nil, 0, err
|
||||||
}
|
}
|
||||||
|
|
||||||
applied := application.Records()
|
applied := application.Records()
|
||||||
@@ -64,35 +63,41 @@ func applyReconciliationPlan(plan semanticreconcile.Plan, records []normalizedRe
|
|||||||
earliest: record.EarliestInputPosition(),
|
earliest: record.EarliestInputPosition(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
type orderedWarning struct {
|
type orderedFinding struct {
|
||||||
position int
|
position int
|
||||||
warning contracts.Warning
|
finding diagnostics.Finding
|
||||||
}
|
}
|
||||||
orderedWarnings := make([]orderedWarning, 0, len(application.AppliedGroups())+len(application.RejectedGroups()))
|
observations := make([]orderedFinding, 0, len(application.AppliedGroups()))
|
||||||
|
advisories := make([]orderedFinding, 0, len(application.RejectedGroups()))
|
||||||
for _, event := range application.AppliedGroups() {
|
for _, event := range application.AppliedGroups() {
|
||||||
provenance := event.Provenance()
|
provenance := event.Provenance()
|
||||||
orderedWarnings = append(orderedWarnings, orderedWarning{
|
observations = append(observations, orderedFinding{
|
||||||
position: provenance.EarliestInputPosition(),
|
position: provenance.EarliestInputPosition(),
|
||||||
warning: semanticDuplicateWarning(provenance, records[provenance.CanonicalPosition()]),
|
finding: semanticDuplicateFinding(provenance, records[provenance.CanonicalPosition()]),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
for _, event := range application.RejectedGroups() {
|
for _, event := range application.RejectedGroups() {
|
||||||
provenance := event.Provenance()
|
provenance := event.Provenance()
|
||||||
orderedWarnings = append(orderedWarnings, orderedWarning{
|
advisories = append(advisories, orderedFinding{
|
||||||
position: provenance.EarliestInputPosition(),
|
position: provenance.EarliestInputPosition(),
|
||||||
warning: contracts.Warning{
|
finding: diagnostics.Finding{
|
||||||
Scope: itemScope(provenance.EarliestInputPosition()),
|
Scope: itemScope(provenance.EarliestInputPosition()),
|
||||||
ReasonCode: ReasonCodeItemSemanticProposalInvalid,
|
ReasonCode: ReasonCodeItemSemanticProposalInvalid,
|
||||||
Message: "proposal group preserved because currency may only be consolidated with aliases of one denomination",
|
Message: "proposal group preserved because currency may only be consolidated with aliases of one denomination",
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
sort.SliceStable(orderedWarnings, func(left, right int) bool { return orderedWarnings[left].position < orderedWarnings[right].position })
|
sort.SliceStable(observations, func(left, right int) bool { return observations[left].position < observations[right].position })
|
||||||
warnings := make([]contracts.Warning, len(orderedWarnings))
|
sort.SliceStable(advisories, func(left, right int) bool { return advisories[left].position < advisories[right].position })
|
||||||
for index, entry := range orderedWarnings {
|
observationFindings := make([]diagnostics.Finding, len(observations))
|
||||||
warnings[index] = entry.warning
|
for index, entry := range observations {
|
||||||
|
observationFindings[index] = entry.finding
|
||||||
}
|
}
|
||||||
return output, warnings, len(application.RejectedGroups()), nil
|
advisoryFindings := make([]diagnostics.Finding, len(advisories))
|
||||||
|
for index, entry := range advisories {
|
||||||
|
advisoryFindings[index] = entry.finding
|
||||||
|
}
|
||||||
|
return output, observationFindings, advisoryFindings, len(application.RejectedGroups()), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func canConsolidate(items []dnd.Item) bool {
|
func canConsolidate(items []dnd.Item) bool {
|
||||||
@@ -133,7 +138,7 @@ func currencyDenomination(name string) string {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func semanticDuplicateWarning(provenance semanticreconcile.GroupProvenance, canonical normalizedRecord) contracts.Warning {
|
func semanticDuplicateFinding(provenance semanticreconcile.GroupProvenance, canonical normalizedRecord) diagnostics.Finding {
|
||||||
inputIndexes := provenance.OriginalInputIndexes()
|
inputIndexes := provenance.OriginalInputIndexes()
|
||||||
details := make([]string, 0, len(inputIndexes)+1)
|
details := make([]string, 0, len(inputIndexes)+1)
|
||||||
for _, inputIndex := range inputIndexes {
|
for _, inputIndex := range inputIndexes {
|
||||||
@@ -142,7 +147,7 @@ func semanticDuplicateWarning(provenance semanticreconcile.GroupProvenance, cano
|
|||||||
if canonical.earliest != provenance.EarliestInputPosition() {
|
if canonical.earliest != provenance.EarliestInputPosition() {
|
||||||
details = append(details, fmt.Sprintf("canonical display name from input index %d", canonical.earliest))
|
details = append(details, fmt.Sprintf("canonical display name from input index %d", canonical.earliest))
|
||||||
}
|
}
|
||||||
return contracts.Warning{
|
return diagnostics.Finding{
|
||||||
Scope: itemScope(provenance.EarliestInputPosition()),
|
Scope: itemScope(provenance.EarliestInputPosition()),
|
||||||
ReasonCode: ReasonCodeDuplicateItemCollapsed,
|
ReasonCode: ReasonCodeDuplicateItemCollapsed,
|
||||||
Message: diagnostics.Aggregate("semantic duplicate consolidation", details),
|
Message: diagnostics.Aggregate("semantic duplicate consolidation", details),
|
||||||
|
|||||||
@@ -27,7 +27,6 @@ const (
|
|||||||
ReasonCodeSourceRefsNormalized = "source_references_normalized"
|
ReasonCodeSourceRefsNormalized = "source_references_normalized"
|
||||||
ReasonCodeOccurrencesReordered = "location_occurrences_reordered"
|
ReasonCodeOccurrencesReordered = "location_occurrences_reordered"
|
||||||
ReasonCodeDuplicateCollapsed = "duplicate_location_occurrence_collapsed"
|
ReasonCodeDuplicateCollapsed = "duplicate_location_occurrence_collapsed"
|
||||||
ReasonCodeWarningsOmitted = "location_occurrence_normalization_warnings_omitted"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -108,8 +107,12 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
return contracts.TypedNormalizeResult[dnd.LocationOccurrenceList]{}, normalizerErrorf("location registry reference is required")
|
return contracts.TypedNormalizeResult[dnd.LocationOccurrenceList]{}, normalizerErrorf("location registry reference is required")
|
||||||
}
|
}
|
||||||
index := source.NewDocumentIndex(req.Source)
|
index := source.NewDocumentIndex(req.Source)
|
||||||
value, warnings := normalizeList(req.MergeOutput.Value, index, shared.NewSourceRefOrderFromIndex(index), registry)
|
value, findings := normalizeList(req.MergeOutput.Value, index, shared.NewSourceRefOrderFromIndex(index), registry)
|
||||||
return contracts.TypedNormalizeResult[dnd.LocationOccurrenceList]{Value: value, Warnings: warnings}, nil
|
diagnosticGroups, err := diagnostics.NormalizationDiagnostics(findings, ReasonCodeUnknownLocationID)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.LocationOccurrenceList]{}, normalizerErrorf("collect diagnostics: %w", err)
|
||||||
|
}
|
||||||
|
return contracts.TypedNormalizeResult[dnd.LocationOccurrenceList]{Value: value, Diagnostics: diagnosticGroups}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type normalizedRecord struct {
|
type normalizedRecord struct {
|
||||||
@@ -119,25 +122,25 @@ type normalizedRecord struct {
|
|||||||
|
|
||||||
type nameCanonicalization struct{ from, to string }
|
type nameCanonicalization struct{ from, to string }
|
||||||
|
|
||||||
func normalizeList(input dnd.LocationOccurrenceList, documentIndex source.DocumentIndex, order shared.SourceRefOrder, registry *locationregistry.Registry) (dnd.LocationOccurrenceList, []contracts.Warning) {
|
func normalizeList(input dnd.LocationOccurrenceList, documentIndex source.DocumentIndex, order shared.SourceRefOrder, registry *locationregistry.Registry) (dnd.LocationOccurrenceList, []diagnostics.Finding) {
|
||||||
if input.Occurrences == nil {
|
if input.Occurrences == nil {
|
||||||
return dnd.LocationOccurrenceList{}, nil
|
return dnd.LocationOccurrenceList{}, nil
|
||||||
}
|
}
|
||||||
records := make([]normalizedRecord, len(input.Occurrences))
|
records := make([]normalizedRecord, len(input.Occurrences))
|
||||||
warnings := make([]contracts.Warning, 0)
|
findings := make([]diagnostics.Finding, 0)
|
||||||
for index, inputOccurrence := range input.Occurrences {
|
for index, inputOccurrence := range input.Occurrences {
|
||||||
occurrence, change, found, refsChanged := normalizeOccurrence(inputOccurrence, order, registry)
|
occurrence, change, found, refsChanged := normalizeOccurrence(inputOccurrence, order, registry)
|
||||||
records[index] = normalizedRecord{occurrence: occurrence, inputIndex: index}
|
records[index] = normalizedRecord{occurrence: occurrence, inputIndex: index}
|
||||||
if change != nil {
|
if change != nil {
|
||||||
warnings = append(warnings, contracts.Warning{Scope: occurrenceScope(index), ReasonCode: ReasonCodeNameCanonicalized,
|
findings = append(findings, diagnostics.Finding{Scope: occurrenceScope(index), ReasonCode: ReasonCodeNameCanonicalized,
|
||||||
Message: fmt.Sprintf("input index %d: location name canonicalized from %s to %s", index, diagnostics.Quote(change.from), diagnostics.Quote(change.to))})
|
Message: fmt.Sprintf("input index %d: location name canonicalized from %s to %s", index, diagnostics.Quote(change.from), diagnostics.Quote(change.to))})
|
||||||
}
|
}
|
||||||
if !found {
|
if !found {
|
||||||
warnings = append(warnings, contracts.Warning{Scope: occurrenceScope(index), ReasonCode: ReasonCodeUnknownLocationID,
|
findings = append(findings, diagnostics.Finding{Scope: occurrenceScope(index), ReasonCode: ReasonCodeUnknownLocationID,
|
||||||
Message: fmt.Sprintf("input index %d: location ID %s is not in the supplied registry", index, diagnostics.Quote(inputOccurrence.LocationID))})
|
Message: fmt.Sprintf("input index %d: location ID %s is not in the supplied registry", index, diagnostics.Quote(inputOccurrence.LocationID))})
|
||||||
}
|
}
|
||||||
if refsChanged {
|
if refsChanged {
|
||||||
warnings = append(warnings, contracts.Warning{Scope: occurrenceScope(index), ReasonCode: ReasonCodeSourceRefsNormalized,
|
findings = append(findings, diagnostics.Finding{Scope: occurrenceScope(index), ReasonCode: ReasonCodeSourceRefsNormalized,
|
||||||
Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)", index, len(inputOccurrence.SourceRefs), len(occurrence.SourceRefs))})
|
Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)", index, len(inputOccurrence.SourceRefs), len(occurrence.SourceRefs))})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -146,13 +149,13 @@ func normalizeList(input dnd.LocationOccurrenceList, documentIndex source.Docume
|
|||||||
})
|
})
|
||||||
for position, record := range records {
|
for position, record := range records {
|
||||||
if position != record.inputIndex {
|
if position != record.inputIndex {
|
||||||
warnings = append(warnings, contracts.Warning{Scope: occurrenceScope(record.inputIndex), ReasonCode: ReasonCodeOccurrencesReordered,
|
findings = append(findings, diagnostics.Finding{Scope: occurrenceScope(record.inputIndex), ReasonCode: ReasonCodeOccurrencesReordered,
|
||||||
Message: fmt.Sprintf("input index %d moved to normalized position %d by canonical occurrence order", record.inputIndex, position)})
|
Message: fmt.Sprintf("input index %d moved to normalized position %d by canonical occurrence order", record.inputIndex, position)})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
output, duplicateWarnings := collapseDuplicates(records, documentIndex)
|
output, duplicateFindings := collapseDuplicates(records, documentIndex)
|
||||||
warnings = append(warnings, duplicateWarnings...)
|
findings = append(findings, duplicateFindings...)
|
||||||
return dnd.LocationOccurrenceList{Occurrences: output}, diagnostics.LimitWarnings(warnings, "location_occurrences", ReasonCodeWarningsOmitted)
|
return dnd.LocationOccurrenceList{Occurrences: output}, findings
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeOccurrence(input dnd.LocationOccurrence, order shared.SourceRefOrder, registry *locationregistry.Registry) (dnd.LocationOccurrence, *nameCanonicalization, bool, bool) {
|
func normalizeOccurrence(input dnd.LocationOccurrence, order shared.SourceRefOrder, registry *locationregistry.Registry) (dnd.LocationOccurrence, *nameCanonicalization, bool, bool) {
|
||||||
@@ -191,7 +194,7 @@ type duplicateGroup struct {
|
|||||||
removed []int
|
removed []int
|
||||||
}
|
}
|
||||||
|
|
||||||
func collapseDuplicates(records []normalizedRecord, documentIndex source.DocumentIndex) ([]dnd.LocationOccurrence, []contracts.Warning) {
|
func collapseDuplicates(records []normalizedRecord, documentIndex source.DocumentIndex) ([]dnd.LocationOccurrence, []diagnostics.Finding) {
|
||||||
if len(records) == 0 {
|
if len(records) == 0 {
|
||||||
return make([]dnd.LocationOccurrence, 0), nil
|
return make([]dnd.LocationOccurrence, 0), nil
|
||||||
}
|
}
|
||||||
@@ -219,13 +222,13 @@ func collapseDuplicates(records []normalizedRecord, documentIndex source.Documen
|
|||||||
output = append(output, cloneOccurrence(record.occurrence))
|
output = append(output, cloneOccurrence(record.occurrence))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
warnings := make([]contracts.Warning, 0)
|
findings := make([]diagnostics.Finding, 0)
|
||||||
for _, group := range groups {
|
for _, group := range groups {
|
||||||
if len(group.removed) > 0 {
|
if len(group.removed) > 0 {
|
||||||
warnings = append(warnings, duplicateWarning(group.retainedIndex, group.removed))
|
findings = append(findings, duplicateFinding(group.retainedIndex, group.removed))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return output, warnings
|
return output, findings
|
||||||
}
|
}
|
||||||
|
|
||||||
func validSourceRefs(index source.DocumentIndex, refs []source.SourceRef) bool {
|
func validSourceRefs(index source.DocumentIndex, refs []source.SourceRef) bool {
|
||||||
@@ -313,12 +316,12 @@ func sourceRefsLess(order shared.SourceRefOrder, left, right []source.SourceRef)
|
|||||||
return len(left) < len(right)
|
return len(left) < len(right)
|
||||||
}
|
}
|
||||||
|
|
||||||
func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
|
func duplicateFinding(retainedIndex int, removed []int) diagnostics.Finding {
|
||||||
issues := make([]string, len(removed))
|
issues := make([]string, len(removed))
|
||||||
for index, removedIndex := range removed {
|
for index, removedIndex := range removed {
|
||||||
issues[index] = fmt.Sprintf("removed input index %d", removedIndex)
|
issues[index] = fmt.Sprintf("removed input index %d", removedIndex)
|
||||||
}
|
}
|
||||||
return contracts.Warning{Scope: occurrenceScope(retainedIndex), ReasonCode: ReasonCodeDuplicateCollapsed,
|
return diagnostics.Finding{Scope: occurrenceScope(retainedIndex), ReasonCode: ReasonCodeDuplicateCollapsed,
|
||||||
Message: diagnostics.Aggregate(fmt.Sprintf("duplicate location occurrence collapsed; retained input index %d", retainedIndex), issues)}
|
Message: diagnostics.Aggregate(fmt.Sprintf("duplicate location occurrence collapsed; retained input index %d", retainedIndex), issues)}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -12,7 +12,6 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
locationcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/locationregistry"
|
locationcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/locationregistry"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestNormalizeCanonicalizesNamesByIDAndClonesInputs(t *testing.T) {
|
func TestNormalizeCanonicalizesNamesByIDAndClonesInputs(t *testing.T) {
|
||||||
@@ -32,11 +31,11 @@ func TestNormalizeCanonicalizesNamesByIDAndClonesInputs(t *testing.T) {
|
|||||||
if occurrence.Name != locations.Locations[1].Name || !reflect.DeepEqual(occurrence.SourceRefs, []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}, {SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}) {
|
if occurrence.Name != locations.Locations[1].Name || !reflect.DeepEqual(occurrence.SourceRefs, []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}, {SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}) {
|
||||||
t.Fatalf("normalized occurrence = %#v", occurrence)
|
t.Fatalf("normalized occurrence = %#v", occurrence)
|
||||||
}
|
}
|
||||||
if !hasWarning(result.Warnings, ReasonCodeNameCanonicalized) || !hasWarning(result.Warnings, ReasonCodeSourceRefsNormalized) || !reflect.DeepEqual(input, before) {
|
if !hasDiagnostic(result.Diagnostics, ReasonCodeNameCanonicalized, contracts.DiagnosticDispositionObservation) || !hasDiagnostic(result.Diagnostics, ReasonCodeSourceRefsNormalized, contracts.DiagnosticDispositionObservation) || !reflect.DeepEqual(input, before) {
|
||||||
t.Fatalf("warnings/input = %#v/%#v", result.Warnings, input)
|
t.Fatalf("diagnostics/input = %#v/%#v", result.Diagnostics, input)
|
||||||
}
|
}
|
||||||
second, err := normalizer.Normalize(context.Background(), normalizeRequest(result.Value, doc, contracts.ReferenceSet{}))
|
second, err := normalizer.Normalize(context.Background(), normalizeRequest(result.Value, doc, contracts.ReferenceSet{}))
|
||||||
if err != nil || !reflect.DeepEqual(second.Value, result.Value) || len(second.Warnings) != 0 {
|
if err != nil || !reflect.DeepEqual(second.Value, result.Value) || len(second.Diagnostics) != 0 {
|
||||||
t.Fatalf("second normalization = %#v, %v", second, err)
|
t.Fatalf("second normalization = %#v, %v", second, err)
|
||||||
}
|
}
|
||||||
result.Value.Occurrences[0].SourceRefs[0].StartUnitID = 999
|
result.Value.Occurrences[0].SourceRefs[0].StartUnitID = 999
|
||||||
@@ -71,8 +70,8 @@ func TestNormalizeKeepsSameNamedIDsAndDistinctEvidence(t *testing.T) {
|
|||||||
if len(got) != 7 || got[0].Kind != dnd.LocationOccurrenceKindVisited || got[1].Kind != dnd.LocationOccurrenceKindPlanned || got[2].Kind != dnd.LocationOccurrenceKindRecalled || got[3].Kind != dnd.LocationOccurrenceKindMentioned || got[3].SourceRefs[0].StartUnitID != 50 || got[4].SourceRefs[0].StartUnitID != 10 || got[5].LocationID != second.ID || got[6].SourceRefs[0].StartUnitID != 999 {
|
if len(got) != 7 || got[0].Kind != dnd.LocationOccurrenceKindVisited || got[1].Kind != dnd.LocationOccurrenceKindPlanned || got[2].Kind != dnd.LocationOccurrenceKindRecalled || got[3].Kind != dnd.LocationOccurrenceKindMentioned || got[3].SourceRefs[0].StartUnitID != 50 || got[4].SourceRefs[0].StartUnitID != 10 || got[5].LocationID != second.ID || got[6].SourceRefs[0].StartUnitID != 999 {
|
||||||
t.Fatalf("canonical occurrences = %#v", got)
|
t.Fatalf("canonical occurrences = %#v", got)
|
||||||
}
|
}
|
||||||
if !hasWarning(result.Warnings, ReasonCodeDuplicateCollapsed) || !hasWarning(result.Warnings, ReasonCodeOccurrencesReordered) {
|
if !hasDiagnostic(result.Diagnostics, ReasonCodeDuplicateCollapsed, contracts.DiagnosticDispositionObservation) || !hasDiagnostic(result.Diagnostics, ReasonCodeOccurrencesReordered, contracts.DiagnosticDispositionObservation) {
|
||||||
t.Fatalf("warnings = %#v", result.Warnings)
|
t.Fatalf("diagnostics = %#v", result.Diagnostics)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -81,7 +80,7 @@ func TestNormalizePreservesUnknownIDsAndMalformedOperationRegistry(t *testing.T)
|
|||||||
locations := registryLocations("The Mill")
|
locations := registryLocations("The Mill")
|
||||||
unknown := dnd.LocationOccurrence{LocationID: "location:sha256:unknown", Name: "The Mill", Kind: "unexpected", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}}
|
unknown := dnd.LocationOccurrence{LocationID: "location:sha256:unknown", Name: "The Mill", Kind: "unexpected", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}}
|
||||||
result, err := newNormalizer(t, registryReferences(t, locations)).Normalize(context.Background(), normalizeRequest(dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{unknown}}, doc, contracts.ReferenceSet{}))
|
result, err := newNormalizer(t, registryReferences(t, locations)).Normalize(context.Background(), normalizeRequest(dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{unknown}}, doc, contracts.ReferenceSet{}))
|
||||||
if err != nil || !reflect.DeepEqual(result.Value.Occurrences[0], unknown) || !hasWarning(result.Warnings, ReasonCodeUnknownLocationID) {
|
if err != nil || !reflect.DeepEqual(result.Value.Occurrences[0], unknown) || !hasDiagnostic(result.Diagnostics, ReasonCodeUnknownLocationID, contracts.DiagnosticDispositionAdvisory) {
|
||||||
t.Fatalf("unknown normalization = %#v, %v", result, err)
|
t.Fatalf("unknown normalization = %#v, %v", result, err)
|
||||||
}
|
}
|
||||||
malformed := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{LocationRegistryReferenceSlot: {
|
malformed := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{LocationRegistryReferenceSlot: {
|
||||||
@@ -92,7 +91,7 @@ func TestNormalizePreservesUnknownIDsAndMalformedOperationRegistry(t *testing.T)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestNormalizerContractsRequiredRegistryAndWarningBounds(t *testing.T) {
|
func TestNormalizerContractsRequiredRegistryAndFindingBounds(t *testing.T) {
|
||||||
if _, err := New(Options{}, contracts.ReferenceSet{}, contracts.ReferenceSet{}); err == nil || !strings.Contains(err.Error(), "at most one reference set") {
|
if _, err := New(Options{}, contracts.ReferenceSet{}, contracts.ReferenceSet{}); err == nil || !strings.Contains(err.Error(), "at most one reference set") {
|
||||||
t.Fatalf("New() error = %v", err)
|
t.Fatalf("New() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -125,7 +124,7 @@ func TestNormalizerContractsRequiredRegistryAndWarningBounds(t *testing.T) {
|
|||||||
t.Fatalf("unbound registry error = %v", err)
|
t.Fatalf("unbound registry error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
count := diagnostics.MaxWarnings + 5
|
count := contracts.MaxDiagnosticSamples + 5
|
||||||
doc := &source.SourceDocument{ID: "session", Units: make([]source.SourceUnit, count)}
|
doc := &source.SourceDocument{ID: "session", Units: make([]source.SourceUnit, count)}
|
||||||
input := dnd.LocationOccurrenceList{Occurrences: make([]dnd.LocationOccurrence, count)}
|
input := dnd.LocationOccurrenceList{Occurrences: make([]dnd.LocationOccurrence, count)}
|
||||||
location := registryLocations("The Mill").Locations[0]
|
location := registryLocations("The Mill").Locations[0]
|
||||||
@@ -134,8 +133,8 @@ func TestNormalizerContractsRequiredRegistryAndWarningBounds(t *testing.T) {
|
|||||||
input.Occurrences[index] = dnd.LocationOccurrence{LocationID: location.ID, Name: "not canonical", Kind: dnd.LocationOccurrenceKindMentioned, SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: count - index, EndUnitID: count - index}}}
|
input.Occurrences[index] = dnd.LocationOccurrence{LocationID: location.ID, Name: "not canonical", Kind: dnd.LocationOccurrenceKindMentioned, SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: count - index, EndUnitID: count - index}}}
|
||||||
}
|
}
|
||||||
bounded, err := newNormalizer(t, registryReferences(t, dnd.LocationRegistry{Locations: []dnd.Location{location}})).Normalize(context.Background(), normalizeRequest(input, doc, contracts.ReferenceSet{}))
|
bounded, err := newNormalizer(t, registryReferences(t, dnd.LocationRegistry{Locations: []dnd.Location{location}})).Normalize(context.Background(), normalizeRequest(input, doc, contracts.ReferenceSet{}))
|
||||||
if err != nil || len(bounded.Warnings) != diagnostics.MaxWarnings || bounded.Warnings[len(bounded.Warnings)-1].ReasonCode != ReasonCodeWarningsOmitted {
|
if err != nil || len(bounded.Diagnostics) == 0 {
|
||||||
t.Fatalf("bounded warnings = %#v, %v", bounded.Warnings, err)
|
t.Fatalf("bounded diagnostics = %#v, %v", bounded.Diagnostics, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -204,9 +203,9 @@ func cloneList(input dnd.LocationOccurrenceList) dnd.LocationOccurrenceList {
|
|||||||
return output
|
return output
|
||||||
}
|
}
|
||||||
|
|
||||||
func hasWarning(warnings []contracts.Warning, code string) bool {
|
func hasDiagnostic(diagnostics []contracts.ProducerDiagnostic, code string, disposition contracts.DiagnosticDisposition) bool {
|
||||||
for _, warning := range warnings {
|
for _, diagnostic := range diagnostics {
|
||||||
if warning.ReasonCode == code {
|
if diagnostic.ReasonCode == code && diagnostic.Disposition == disposition {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -33,7 +33,6 @@ const (
|
|||||||
ReasonCodeDuplicateLocationCollapsed = "duplicate_location_collapsed"
|
ReasonCodeDuplicateLocationCollapsed = "duplicate_location_collapsed"
|
||||||
ReasonCodeLocationSemanticProposalInvalid = "location_semantic_proposal_invalid"
|
ReasonCodeLocationSemanticProposalInvalid = "location_semantic_proposal_invalid"
|
||||||
ReasonCodeLocationSemanticReconciliationExhausted = "location_semantic_reconciliation_exhausted"
|
ReasonCodeLocationSemanticReconciliationExhausted = "location_semantic_reconciliation_exhausted"
|
||||||
ReasonCodeLocationNormalizationWarningsOmitted = "location_normalization_warnings_omitted"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
var requiredCapabilities = []string{"merged"}
|
var requiredCapabilities = []string{"merged"}
|
||||||
@@ -108,10 +107,10 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
}
|
}
|
||||||
|
|
||||||
order := shared.NewSourceRefOrder(req.Source)
|
order := shared.NewSourceRefOrder(req.Source)
|
||||||
records, warnings := preprocessRecords(req.MergeOutput.Value, order)
|
records, findings := preprocessRecords(req.MergeOutput.Value, order)
|
||||||
deterministic := recordList(records)
|
deterministic := recordList(records)
|
||||||
if len(records) < 2 {
|
if len(records) < 2 {
|
||||||
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{Value: deterministic, Warnings: limitWarnings(warnings)}, nil
|
return normalizationResult(deterministic, findings, nil)
|
||||||
}
|
}
|
||||||
|
|
||||||
candidates, envelopes, err := reconciliationInputs(records)
|
candidates, envelopes, err := reconciliationInputs(records)
|
||||||
@@ -128,67 +127,75 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
|||||||
|
|
||||||
switch reconciliation.Disposition() {
|
switch reconciliation.Disposition() {
|
||||||
case semanticreconcile.SkippedInsufficientCandidates:
|
case semanticreconcile.SkippedInsufficientCandidates:
|
||||||
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{Value: deterministic, Warnings: limitWarnings(warnings)}, nil
|
return normalizationResult(deterministic, findings, nil)
|
||||||
case semanticreconcile.SkippedLimitExceeded:
|
case semanticreconcile.SkippedLimitExceeded:
|
||||||
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{Value: deterministic, Warnings: limitWarningsWithSemanticFallback(warnings)}, nil
|
return fallbackResult(deterministic, findings, semanticFallbackFinding(-1))
|
||||||
case semanticreconcile.RetryableInvalidStructuredOutput:
|
case semanticreconcile.RetryableInvalidStructuredOutput:
|
||||||
return n.invalidStructuredResult(deterministic, warnings), nil
|
return n.invalidStructuredResult(deterministic, findings)
|
||||||
case semanticreconcile.Complete, semanticreconcile.RetryableDiscardedProposalGroups:
|
case semanticreconcile.Complete, semanticreconcile.RetryableDiscardedProposalGroups:
|
||||||
default:
|
default:
|
||||||
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, normalizerErrorf("unknown semantic reconciliation disposition %d", reconciliation.Disposition())
|
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, normalizerErrorf("unknown semantic reconciliation disposition %d", reconciliation.Disposition())
|
||||||
}
|
}
|
||||||
|
|
||||||
applied, semanticWarnings, err := applyReconciliationPlan(reconciliation.Plan(), records, envelopes, order)
|
applied, semanticFindings, err := applyReconciliationPlan(reconciliation.Plan(), records, envelopes, order)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, normalizerErrorf("apply semantic reconciliation plan: %w", err)
|
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, normalizerErrorf("apply semantic reconciliation plan: %w", err)
|
||||||
}
|
}
|
||||||
warnings = append(warnings, semanticWarnings...)
|
findings = append(findings, semanticFindings...)
|
||||||
if reconciliation.Disposition() == semanticreconcile.Complete {
|
if reconciliation.Disposition() == semanticreconcile.Complete {
|
||||||
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{Value: recordList(applied), Warnings: limitWarnings(warnings), ModelCandidate: reconciliation.ModelCandidate()}, nil
|
return normalizationResult(recordList(applied), findings, reconciliation.ModelCandidate())
|
||||||
}
|
}
|
||||||
return retryResult(recordList(applied), warnings, reconciliation), nil
|
return retryResult(recordList(applied), findings, reconciliation)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (n *Normalizer) invalidStructuredResult(value dnd.LocationRegistry, warnings []contracts.Warning) contracts.TypedNormalizeResult[dnd.LocationRegistry] {
|
func (n *Normalizer) invalidStructuredResult(value dnd.LocationRegistry, findings []diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.LocationRegistry], error) {
|
||||||
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{Value: value, Warnings: limitWarningsForRetry(warnings), Retry: &contracts.NormalizeRetry{
|
return retryResultWithFallback(value, findings, nil, "semantic proposal requires retry: invalid structured output", semanticFallbackFinding(-1))
|
||||||
ReasonCode: ReasonCodeLocationSemanticProposalInvalid, Message: "semantic proposal requires retry: invalid structured output",
|
|
||||||
FallbackWarnings: []contracts.Warning{semanticFallbackWarning(-1)},
|
|
||||||
}}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func retryResult(value dnd.LocationRegistry, warnings []contracts.Warning, reconciliation semanticreconcile.Result) contracts.TypedNormalizeResult[dnd.LocationRegistry] {
|
func retryResult(value dnd.LocationRegistry, findings []diagnostics.Finding, reconciliation semanticreconcile.Result) (contracts.TypedNormalizeResult[dnd.LocationRegistry], error) {
|
||||||
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{Value: value, Warnings: limitWarningsForRetry(warnings), ModelCandidate: reconciliation.ModelCandidate(), Retry: &contracts.NormalizeRetry{
|
return retryResultWithFallback(value, findings, reconciliation.ModelCandidate(), diagnostics.Aggregate("semantic proposal requires retry", semanticreconcile.IssueDetails(reconciliation.Issues())), semanticFallbackFinding(reconciliation.DiscardedGroupCount()))
|
||||||
ReasonCode: ReasonCodeLocationSemanticProposalInvalid, Message: diagnostics.Aggregate("semantic proposal requires retry", semanticreconcile.IssueDetails(reconciliation.Issues())),
|
|
||||||
FallbackWarnings: []contracts.Warning{semanticFallbackWarning(reconciliation.DiscardedGroupCount())},
|
|
||||||
}}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func semanticFallbackWarning(discarded int) contracts.Warning {
|
func semanticFallbackFinding(discarded int) diagnostics.Finding {
|
||||||
message := "semantic proposal could not be applied"
|
message := "semantic proposal could not be applied"
|
||||||
if discarded >= 0 {
|
if discarded >= 0 {
|
||||||
message = fmt.Sprintf("%d proposal group(s) omitted after semantic proposal retry exhaustion", discarded)
|
message = fmt.Sprintf("%d proposal group(s) omitted after semantic proposal retry exhaustion", discarded)
|
||||||
}
|
}
|
||||||
return contracts.Warning{Scope: "locations", ReasonCode: ReasonCodeLocationSemanticReconciliationExhausted, Message: message}
|
return diagnostics.Finding{Scope: "locations", ReasonCode: ReasonCodeLocationSemanticReconciliationExhausted, Message: message}
|
||||||
}
|
}
|
||||||
|
|
||||||
func limitWarnings(warnings []contracts.Warning) []contracts.Warning {
|
func normalizationResult(value dnd.LocationRegistry, findings []diagnostics.Finding, candidate *contracts.ModelCandidate) (contracts.TypedNormalizeResult[dnd.LocationRegistry], error) {
|
||||||
return diagnostics.LimitWarnings(warnings, "locations", ReasonCodeLocationNormalizationWarningsOmitted)
|
diagnosticGroups, err := diagnostics.Collect(findings, contracts.DiagnosticDispositionObservation, contracts.DiagnosticCategoryNormalization)
|
||||||
}
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, normalizerErrorf("collect normalization diagnostics: %w", err)
|
||||||
func limitWarningsForRetry(warnings []contracts.Warning) []contracts.Warning {
|
|
||||||
if warnings == nil {
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
if len(warnings) < diagnostics.MaxWarnings {
|
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{Value: value, Diagnostics: diagnosticGroups, ModelCandidate: candidate}, nil
|
||||||
return append([]contracts.Warning(nil), warnings...)
|
|
||||||
}
|
|
||||||
displayed := diagnostics.MaxWarnings - 2
|
|
||||||
bounded := append([]contracts.Warning(nil), warnings[:displayed]...)
|
|
||||||
return append(bounded, contracts.Warning{Scope: "locations", ReasonCode: ReasonCodeLocationNormalizationWarningsOmitted, Message: fmt.Sprintf("%d additional warning(s) omitted", len(warnings)-displayed)})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func limitWarningsWithSemanticFallback(warnings []contracts.Warning) []contracts.Warning {
|
func fallbackResult(value dnd.LocationRegistry, findings []diagnostics.Finding, fallback diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.LocationRegistry], error) {
|
||||||
return append(limitWarningsForRetry(warnings), semanticFallbackWarning(-1))
|
result, err := normalizationResult(value, findings, nil)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, err
|
||||||
|
}
|
||||||
|
fallbackGroups, err := diagnostics.Collect([]diagnostics.Finding{fallback}, contracts.DiagnosticDispositionWarning, contracts.DiagnosticCategoryFallback)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, normalizerErrorf("collect fallback diagnostic: %w", err)
|
||||||
|
}
|
||||||
|
result.Diagnostics = append(result.Diagnostics, fallbackGroups...)
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func retryResultWithFallback(value dnd.LocationRegistry, findings []diagnostics.Finding, candidate *contracts.ModelCandidate, message string, fallback diagnostics.Finding) (contracts.TypedNormalizeResult[dnd.LocationRegistry], error) {
|
||||||
|
result, err := normalizationResult(value, findings, candidate)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, err
|
||||||
|
}
|
||||||
|
fallbackGroups, err := diagnostics.Collect([]diagnostics.Finding{fallback}, contracts.DiagnosticDispositionWarning, contracts.DiagnosticCategoryFallback)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.LocationRegistry]{}, normalizerErrorf("collect fallback diagnostic: %w", err)
|
||||||
|
}
|
||||||
|
result.Retry = &contracts.NormalizeRetry{ReasonCode: ReasonCodeLocationSemanticProposalInvalid, Message: message, FallbackDiagnostics: fallbackGroups}
|
||||||
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
type normalizedRecord struct {
|
type normalizedRecord struct {
|
||||||
@@ -197,23 +204,23 @@ type normalizedRecord struct {
|
|||||||
earliest int
|
earliest int
|
||||||
}
|
}
|
||||||
|
|
||||||
func preprocessRecords(input dnd.LocationRegistry, order shared.SourceRefOrder) ([]normalizedRecord, []contracts.Warning) {
|
func preprocessRecords(input dnd.LocationRegistry, order shared.SourceRefOrder) ([]normalizedRecord, []diagnostics.Finding) {
|
||||||
if input.Locations == nil {
|
if input.Locations == nil {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
records := make([]normalizedRecord, len(input.Locations))
|
records := make([]normalizedRecord, len(input.Locations))
|
||||||
warnings := make([]contracts.Warning, 0)
|
findings := make([]diagnostics.Finding, 0)
|
||||||
for index, inputLocation := range input.Locations {
|
for index, inputLocation := range input.Locations {
|
||||||
location, fieldsChanged, refsChanged := normalizeRecord(inputLocation, order)
|
location, fieldsChanged, refsChanged := normalizeRecord(inputLocation, order)
|
||||||
records[index] = normalizedRecord{location: location, inputIndexes: []int{index}, earliest: index}
|
records[index] = normalizedRecord{location: location, inputIndexes: []int{index}, earliest: index}
|
||||||
if fieldsChanged {
|
if fieldsChanged {
|
||||||
warnings = append(warnings, contracts.Warning{Scope: locationScope(index), ReasonCode: ReasonCodeLocationFieldsNormalized, Message: fmt.Sprintf("input index %d: location name normalized for %s", index, diagnostics.Quote(inputLocation.Name))})
|
findings = append(findings, diagnostics.Finding{Scope: locationScope(index), ReasonCode: ReasonCodeLocationFieldsNormalized, Message: fmt.Sprintf("input index %d: location name normalized for %s", index, diagnostics.Quote(inputLocation.Name))})
|
||||||
}
|
}
|
||||||
if refsChanged {
|
if refsChanged {
|
||||||
warnings = append(warnings, contracts.Warning{Scope: locationScope(index), ReasonCode: ReasonCodeSourceReferencesNormalized, Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)", index, len(inputLocation.SourceRefs), len(location.SourceRefs))})
|
findings = append(findings, diagnostics.Finding{Scope: locationScope(index), ReasonCode: ReasonCodeSourceReferencesNormalized, Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)", index, len(inputLocation.SourceRefs), len(location.SourceRefs))})
|
||||||
}
|
}
|
||||||
if inputLocation.ID != location.ID {
|
if inputLocation.ID != location.ID {
|
||||||
warnings = append(warnings, contracts.Warning{Scope: locationScope(index), ReasonCode: ReasonCodeLocationIDRecomputed, Message: fmt.Sprintf("input index %d: location ID recomputed from %s", index, diagnostics.Quote(location.Name))})
|
findings = append(findings, diagnostics.Finding{Scope: locationScope(index), ReasonCode: ReasonCodeLocationIDRecomputed, Message: fmt.Sprintf("input index %d: location ID recomputed from %s", index, diagnostics.Quote(location.Name))})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
groups := exactDuplicateGroups(records)
|
groups := exactDuplicateGroups(records)
|
||||||
@@ -226,10 +233,10 @@ func preprocessRecords(input dnd.LocationRegistry, order shared.SourceRefOrder)
|
|||||||
retained.inputIndexes = sortedUniqueIndexes(retained.inputIndexes)
|
retained.inputIndexes = sortedUniqueIndexes(retained.inputIndexes)
|
||||||
output = append(output, retained)
|
output = append(output, retained)
|
||||||
if len(members) > 1 {
|
if len(members) > 1 {
|
||||||
warnings = append(warnings, duplicateWarning(retained.earliest, memberInputIndexes(records, members[1:])))
|
findings = append(findings, duplicateFinding(retained.earliest, memberInputIndexes(records, members[1:])))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return output, warnings
|
return output, findings
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeRecord(input dnd.Location, order shared.SourceRefOrder) (dnd.Location, bool, bool) {
|
func normalizeRecord(input dnd.Location, order shared.SourceRefOrder) (dnd.Location, bool, bool) {
|
||||||
@@ -335,7 +342,7 @@ func recordList(records []normalizedRecord) dnd.LocationRegistry {
|
|||||||
return dnd.LocationRegistry{Locations: recordValues(records)}
|
return dnd.LocationRegistry{Locations: recordValues(records)}
|
||||||
}
|
}
|
||||||
|
|
||||||
func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
|
func duplicateFinding(retainedIndex int, removed []int) diagnostics.Finding {
|
||||||
const maxDisplayedIndices = 20
|
const maxDisplayedIndices = 20
|
||||||
displayed := removed
|
displayed := removed
|
||||||
if len(displayed) > maxDisplayedIndices {
|
if len(displayed) > maxDisplayedIndices {
|
||||||
@@ -349,7 +356,7 @@ func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
|
|||||||
if omitted := len(removed) - len(displayed); omitted > 0 {
|
if omitted := len(removed) - len(displayed); omitted > 0 {
|
||||||
message += fmt.Sprintf("; %d additional removed input indices omitted", omitted)
|
message += fmt.Sprintf("; %d additional removed input indices omitted", omitted)
|
||||||
}
|
}
|
||||||
return contracts.Warning{Scope: locationScope(retainedIndex), ReasonCode: ReasonCodeDuplicateLocationCollapsed, Message: message}
|
return diagnostics.Finding{Scope: locationScope(retainedIndex), ReasonCode: ReasonCodeDuplicateLocationCollapsed, Message: message}
|
||||||
}
|
}
|
||||||
func locationScope(index int) string { return fmt.Sprintf("locations[%d]", index) }
|
func locationScope(index int) string { return fmt.Sprintf("locations[%d]", index) }
|
||||||
|
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user