Compare commits
13 Commits
f94ab0a6bf
...
906d97b391
| Author | SHA1 | Date | |
|---|---|---|---|
| 906d97b391 | |||
| f15fd4f9c1 | |||
| 7aadb088a6 | |||
| 9de399432e | |||
| 5bdd56cfb1 | |||
| 64ea23c21f | |||
| 7071102ab7 | |||
| 9184072839 | |||
| c437682407 | |||
| 22d4f29670 | |||
| afb7ed3cf1 | |||
| f846f252c0 | |||
| f5618d1f0c |
@@ -30,4 +30,5 @@ Useful references:
|
||||
- [Developer guide](docs/development.md)
|
||||
- [Internal implementation docs](docs/internal/overview.md)
|
||||
- [Maintained example config](examples/dnd-spells.config.yml)
|
||||
- [NPC-grounded example config](examples/dnd-npc-grounded.config.yml)
|
||||
- [Maintained example input](examples/seriatim-minimal-transcript.json)
|
||||
|
||||
50
docs/adr/0008-ordered-pipeline-steps.md
Normal file
50
docs/adr/0008-ordered-pipeline-steps.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# ADR-0008: Bounded ordered pipeline steps and explicit artifact references
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-07-21
|
||||
|
||||
## Context
|
||||
|
||||
Notarius currently models one pipeline-wide input, chunking plan, artifact
|
||||
lanes, and output boundary. Some workflows need a deterministic handoff from
|
||||
one set of normalized artifacts to a later set of artifacts, such as using
|
||||
extracted NPC records while grounding later combat events. The workflow needs
|
||||
an explicit topology without turning the pipeline into a general-purpose
|
||||
workflow engine.
|
||||
|
||||
## Decision
|
||||
|
||||
Add an ordered collection of pipeline steps. Each step owns one or more
|
||||
artifact lanes, and lanes within a step retain the existing independent
|
||||
execution model. The pipeline continues to have one input, chunk plan, output,
|
||||
and failure boundary. Steps are barriers: a later step may consume only
|
||||
normalized artifacts from an earlier step.
|
||||
|
||||
Generated references use an explicit step-and-lane selector. Reference slots
|
||||
declare the generated artifact kinds and media types they accept. The resolver
|
||||
validates the topology, ordering, lane identity, artifact kind, schema, and
|
||||
codec compatibility before execution. External references remain supported as
|
||||
path sources, and the legacy top-level artifact map is interpreted as an
|
||||
implicit `default` step.
|
||||
|
||||
Pipeline-level references may not select generated artifacts. General DAGs,
|
||||
branches, loops, conditional execution, joins, and inferred dependencies are
|
||||
not part of this model.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- A general DAG would provide more flexibility but would also require a new
|
||||
scheduler, lifecycle model, failure semantics, and provenance model.
|
||||
- Separate pipeline runs connected through filesystem paths would lose the
|
||||
static topology and typed compatibility checks.
|
||||
- Inferring dependencies from module or lane names would make ordering and
|
||||
configuration errors difficult to detect reliably.
|
||||
|
||||
## Consequences
|
||||
|
||||
The resolved pipeline has a deterministic, inspectable topology and can
|
||||
include it in its identity digest. Configuration validation can reject invalid
|
||||
generated bindings before any work begins. Existing single-step profiles keep
|
||||
their behavior through the implicit `default` step. Execution handoff and
|
||||
multi-step scheduling require follow-up work in the runner and checkpoint
|
||||
layers.
|
||||
65
docs/cli.md
65
docs/cli.md
@@ -9,7 +9,7 @@ For the minimal end-to-end invocation, see the [README](../README.md).
|
||||
|
||||
```text
|
||||
notarius help
|
||||
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--chunk_cache auto|bypass|refresh] [--output-dir path] [--resume] [--debug [--debug-dir path]] [--llm-profile id] [--session-id id] [--reference selector=path] [--without-reference selector]
|
||||
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--chunk_cache auto|bypass|refresh] [--output-dir path] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--llm-profile id] [--session-id id] [--reference selector=path] [--without-reference selector]
|
||||
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
|
||||
notarius pipelines list [--config path/to/config.yml] [--json]
|
||||
```
|
||||
@@ -28,11 +28,18 @@ Flags:
|
||||
- `--config path`: config file path. If omitted, Notarius uses the discovery
|
||||
rules in [Configuration](config.md#discovery).
|
||||
- `--only lane-a,lane-b`: run only the named artifact lanes. Values are
|
||||
comma-separated and must be non-empty.
|
||||
comma-separated and must be non-empty. This retains its existing behavior for
|
||||
implicit single-step pipelines; explicit multi-step pipelines reject it
|
||||
rather than inferring dependency closure.
|
||||
- `--resume`: request checkpoint reuse for this invocation. Checkpoint recording
|
||||
must be enabled in configuration. See
|
||||
[Operations](operations.md#checkpoint-cache) for prerequisites and reuse
|
||||
behavior.
|
||||
- `--recompute-step step-id`: with `--resume` and checkpoint recording enabled,
|
||||
force the named ordered step and every transitive dependent lane to execute.
|
||||
Compatible required predecessors and unrelated lanes remain reusable. The
|
||||
value may identify an explicit step or the implicit single-step ID `default`;
|
||||
it cannot be combined with `--only`.
|
||||
- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
|
||||
invocation. `auto` reuses a valid plan by canonical source digest, `bypass`
|
||||
performs no plan-cache I/O, and `refresh` regenerates and replaces a valid
|
||||
@@ -58,8 +65,10 @@ rejected output counts, and the output directory. A debug-enabled run also
|
||||
prints `debug=<bundle-path>`. If the run completes with warnings, the warning
|
||||
count is printed to stderr.
|
||||
|
||||
Reference flags are resolved against selected chunk, extractor, merger, and
|
||||
normalizer targets before the run starts. Flat slot names are accepted only
|
||||
Reference flags are external file bindings resolved against selected chunk,
|
||||
extractor, merger, and normalizer targets before the run starts. Generated
|
||||
artifact bindings are configured in ordered steps and cannot be introduced by a
|
||||
CLI path flag. Flat slot names are accepted only
|
||||
when exactly one selected target declares that slot. For configured reference
|
||||
bindings, precedence, path resolution, and validation, see
|
||||
[Configuration](config.md#pipelines).
|
||||
@@ -94,30 +103,21 @@ go run ./cmd/notarius run dnd-session \
|
||||
--reference spells.extract.glossary=./campaign-glossary.txt
|
||||
```
|
||||
|
||||
For the operator-driven NPC-to-spell workflow, bind the normalized NPC lane
|
||||
payload from the completed NPC run to the spell extractor:
|
||||
For the maintained NPC-grounded workflow, use the explicit ordered pipeline.
|
||||
The first step produces the normalized NPC artifact; the second step receives
|
||||
it in memory and fans it out to spell extraction, combat extraction, and combat
|
||||
normalization:
|
||||
|
||||
```sh
|
||||
go run ./cmd/notarius run dnd-spells \
|
||||
--config examples/dnd-npc-spell-sequential.config.yml \
|
||||
go run ./cmd/notarius run dnd-npc-grounded \
|
||||
--config examples/dnd-npc-grounded.config.yml \
|
||||
--input examples/seriatim-minimal-transcript.json \
|
||||
--reference spells.extract.npcs=./npc-run/lanes/npcs.json
|
||||
--output-dir ./npc-grounded-output
|
||||
```
|
||||
|
||||
For the independent NPC-to-combat workflow, bind the same completed NPC lane
|
||||
to both combat stages explicitly:
|
||||
|
||||
```sh
|
||||
go run ./cmd/notarius run dnd-combat \
|
||||
--config examples/dnd-npc-combat-sequential.config.yml \
|
||||
--input examples/seriatim-minimal-transcript.json \
|
||||
--reference combat.extract.npcs=./npc-run/lanes/npcs.json \
|
||||
--reference combat.normalize.npcs=./npc-run/lanes/npcs.json
|
||||
```
|
||||
|
||||
The two selectors are independent stage-local bindings. Binding extraction
|
||||
does not implicitly bind normalization, and Notarius does not discover or
|
||||
schedule the preceding NPC run.
|
||||
The generated NPC content remains contextual grounding, not spell or combat
|
||||
evidence. It is represented in manifests and debug summaries by bounded
|
||||
identity and producer provenance, not by payload content or a filesystem path.
|
||||
|
||||
The same grammar can target chunk, merge, and normalize slots when the configured
|
||||
modules declare them:
|
||||
@@ -161,6 +161,25 @@ go run ./cmd/notarius run dnd-session \
|
||||
--resume
|
||||
```
|
||||
|
||||
To selectively rerun one ordered step and its dependent lanes, use the step ID
|
||||
from the configuration. The selected step and dependents are reported as
|
||||
`forced_recompute`; reusable predecessors are reported as `reused`:
|
||||
|
||||
```sh
|
||||
go run ./cmd/notarius run dnd-npc-grounded \
|
||||
--config examples/dnd-npc-grounded.config.yml \
|
||||
--input examples/seriatim-minimal-transcript.json \
|
||||
--resume --recompute-step grounded-events
|
||||
```
|
||||
|
||||
Checkpoint decisions use these categories: `reused`, `executed`,
|
||||
`forced_recompute`, and `dependency_invalidated`. The reason code and bounded
|
||||
detail identify the decision without exposing reference content, local paths,
|
||||
or secrets. `--recompute-step` requires checkpoint recording and `--resume`;
|
||||
unknown step IDs, empty values, and combinations with `--only` are rejected.
|
||||
The operator meanings of checkpoint reason codes are maintained in
|
||||
[Operations](operations.md#resume-and-selective-recompute).
|
||||
|
||||
Use `--debug` to retain the redacted summary and trace bundle for one run. The
|
||||
bundle is allocated before pipeline resolution; once allocated, its path is
|
||||
also printed to stderr if the command fails. Debug-write failures cause exit
|
||||
|
||||
159
docs/config.md
159
docs/config.md
@@ -22,9 +22,8 @@ The explicit-path option is defined in the [CLI reference](cli.md).
|
||||
- [Minimal D&D spell configuration](../examples/dnd-spells.config.yml)
|
||||
- [Production-oriented D&D spell configuration](../examples/dnd-spells-production.config.yml)
|
||||
- [D&D NPC configuration](../examples/dnd-npcs.config.yml)
|
||||
- [Sequential D&D NPC and spell configuration](../examples/dnd-npc-spell-sequential.config.yml)
|
||||
- [D&D combat-turn configuration](../examples/dnd-combat-turns.config.yml)
|
||||
- [Sequential D&D NPC and combat-turn configuration](../examples/dnd-npc-combat-sequential.config.yml)
|
||||
- [D&D NPC-grounded spell and combat configuration](../examples/dnd-npc-grounded.config.yml)
|
||||
|
||||
All are complete version 3 files. The fragments below illustrate individual
|
||||
fields and are not alternate complete configurations.
|
||||
@@ -140,13 +139,64 @@ Pipeline fields:
|
||||
|
||||
- `input`: required module binding.
|
||||
- `chunk`: optional module binding. Default module is `generic`.
|
||||
- `artifacts`: required for pipeline resolution. It maps artifact lane IDs to
|
||||
lane definitions.
|
||||
- `artifacts`: the artifact lane map for a single-step pipeline. It is treated
|
||||
as an implicit step with the stable ID `default`.
|
||||
- `steps`: an ordered, non-empty list of step definitions. A pipeline may use
|
||||
`steps` or `artifacts`, but not both. Step IDs must be unique after trimming.
|
||||
- `output`: optional module binding. Default module is `json`.
|
||||
- `references`: optional map of reference slot names to reference paths. These
|
||||
bindings are defaults for eligible pipeline targets that declare the matching
|
||||
slot.
|
||||
|
||||
Each explicit step contains an `id`, an optional `references` map, and a
|
||||
non-empty `artifacts` map. Steps share the pipeline input, chunk plan, worker
|
||||
budget, output encoder, manifest, and failure boundary. Lanes within a step
|
||||
retain the fixed extract, validate, merge, validate, normalize, and validate
|
||||
workflow; the next step starts only after the current step is terminal.
|
||||
|
||||
Generated references use the structured `artifact` source form to identify one
|
||||
accepted normalized lane from an earlier step:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
dnd-npc-grounded:
|
||||
input: seriatim
|
||||
steps:
|
||||
- id: identify-npcs
|
||||
artifacts:
|
||||
npcs:
|
||||
extract: dnd/npcs
|
||||
normalize: dnd/npcs
|
||||
- id: grounded-events
|
||||
references:
|
||||
npcs:
|
||||
artifact:
|
||||
step: identify-npcs
|
||||
lane: npcs
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
normalize: dnd/spells
|
||||
combat:
|
||||
extract: dnd/combat-turns
|
||||
normalize: dnd/combat-turns
|
||||
```
|
||||
|
||||
The generated binding is explicit and typed; it is not inferred from module,
|
||||
lane, or slot names. It may be declared at step scope, applying to every
|
||||
selected target in that step that declares the slot, or at one target's
|
||||
`references` map. A producer may fan out to compatible target slots, but a
|
||||
slot accepts only one producer. A producer must be in an earlier step, and a
|
||||
configured generated dependency is required even when the consumer slot is
|
||||
otherwise optional. Aggregating several producer artifacts is unsupported.
|
||||
|
||||
The producer codec supplies the artifact kind, complete schema identity, media
|
||||
type, canonical content digest, and size used for compatibility and checkpoint
|
||||
dependency checks. Only one accepted normalized artifact crosses the boundary;
|
||||
raw extraction results, rejected output, intermediate values, and validator
|
||||
diagnostics do not. Generated content is supplied in memory and is never
|
||||
represented by a filesystem path.
|
||||
|
||||
Artifact lane fields:
|
||||
|
||||
- `extract`: required module binding.
|
||||
@@ -164,19 +214,23 @@ See [CLI Reference](cli.md) for command syntax.
|
||||
Reference bindings are validated against reference slots declared by eligible
|
||||
chunk, extract, merge, and normalize targets during pipeline resolution. Required slots
|
||||
must be bound after config defaults, target-local references, lane-level
|
||||
compatibility bindings, and command-line reference overrides are applied.
|
||||
Config-relative paths are resolved relative to the config file; command-line
|
||||
reference paths are resolved relative to the current working directory. Bound
|
||||
files must be UTF-8 text. Reference media types are inferred from file
|
||||
extensions and checked when a module restricts accepted types; unknown
|
||||
extensions use `application/octet-stream`. See [CLI Reference](cli.md#run) for
|
||||
command-line selectors and [Operations](operations.md) for recorded provenance
|
||||
and sensitive-data handling.
|
||||
compatibility bindings, step-local references, and command-line reference
|
||||
overrides are applied. Config-relative paths are resolved relative to the
|
||||
config file; command-line reference paths are resolved relative to the current
|
||||
working directory. Bound files must be UTF-8 text. Reference media types are
|
||||
inferred from file extensions and checked when a module restricts accepted
|
||||
types; unknown extensions use `application/octet-stream`. See [CLI Reference](cli.md#run)
|
||||
for command-line selectors and [Operations](operations.md) for recorded
|
||||
provenance and sensitive-data handling.
|
||||
|
||||
Pipeline-level `references` are defaults. They are valid when at least one
|
||||
eligible target in the full configured pipeline declares the slot, including
|
||||
chunk, extractor, merger, and normalizer targets. During a run, they apply only
|
||||
to the selected targets that declare the slot:
|
||||
to the selected targets that declare the slot. For external bindings, a
|
||||
step-local binding overrides a pipeline-level default and a target-local
|
||||
binding retains the existing most-specific precedence. Generated and external
|
||||
bindings may not resolve to the same target slot, and a step-scoped generated
|
||||
binding may not be duplicated by a target-local generated binding.
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
@@ -226,6 +280,18 @@ Target-local reference fields use the same map shape at:
|
||||
|
||||
Each binding is valid only when that target module declares the slot.
|
||||
|
||||
Reference source forms are:
|
||||
|
||||
- a scalar string, which is an external file path; or
|
||||
- an object with only `artifact`, containing trimmed `step` and `lane` IDs for
|
||||
an earlier producer lane.
|
||||
|
||||
Pipeline-level references accept only external paths. Generated references are
|
||||
valid at step scope or on a target-local `references` map. Their producer's
|
||||
registered codec is authoritative for schema, media type, and canonical
|
||||
content identity; an external file is not treated as generated merely because
|
||||
its bytes decode as the same artifact.
|
||||
|
||||
## Module Bindings
|
||||
|
||||
Every module binding may use shorthand:
|
||||
@@ -400,15 +466,17 @@ The extractor uses campaign references only as supporting disambiguation
|
||||
material; spell casts still must be present in the source transcript.
|
||||
|
||||
It also declares an optional `npcs` slot for a normalized NPC artifact. The
|
||||
slot accepts exactly one `application/json` file no larger than 1 MiB. During
|
||||
extractor preparation Notarius strictly decodes and identity-validates the
|
||||
artifact, then gives the model canonical JSON for caster-name grounding.
|
||||
Registry source references may belong to the NPC-producing session and are
|
||||
provenance only; they are not spell evidence. The bound registry contributes a
|
||||
semantic digest and NPC count to extractor metadata and checkpoint identity,
|
||||
while its names, aliases, content, and path do not appear there. When absent,
|
||||
the prompt receives the exact empty value `{"npcs":[]}` and no registry
|
||||
provenance or fingerprint is recorded.
|
||||
slot accepts exactly one `application/json` artifact no larger than 1 MiB. An
|
||||
external file is decoded and identity-validated during preparation. A
|
||||
generated binding is validated at the step handoff and is provided to the
|
||||
operation through the same reference contract. In both cases, the model
|
||||
receives canonical JSON for caster-name grounding. Registry source references
|
||||
may belong to the NPC-producing session and are provenance only; they are not
|
||||
spell evidence. Generated reference identity and bounded producer provenance
|
||||
are recorded by the framework; NPC names, aliases, content, and paths are not
|
||||
copied into manifests or checkpoint decisions. When absent, the prompt receives
|
||||
the exact empty value `{"npcs":[]}` and no registry provenance or fingerprint
|
||||
is recorded.
|
||||
|
||||
The `dnd/spells` normalizer declares the same optional `spell_catalog` slot.
|
||||
When an overlay is used, bind it independently under
|
||||
@@ -419,29 +487,34 @@ bound.
|
||||
|
||||
The `dnd/npcs` extractor declares the same optional campaign slots as the spell
|
||||
extractor, but it does not declare the `npcs` registry slot. Its normalizer
|
||||
accepts no references. To pass an NPC result to a later spell run, bind the
|
||||
normalized payload explicitly at runtime; the maintained sequential example
|
||||
documents that operator workflow.
|
||||
accepts no references. The maintained
|
||||
[NPC-grounded example](../examples/dnd-npc-grounded.config.yml) binds its
|
||||
accepted normalized output to the later spell and combat targets through an
|
||||
explicit ordered step.
|
||||
|
||||
The `dnd/combat-turns` extractor declares the optional campaign slots and the
|
||||
structured `npcs` slot. Campaign references guide only the LLM extraction
|
||||
stage. The deterministic normalizer declares only `npcs`, whose prepared
|
||||
immutable registry supports the same actor and target canonicalization. Each
|
||||
`npcs` slot accepts exactly one UTF-8 `application/json` file no larger than 1
|
||||
MiB. The registry's source ranges remain provenance for the reference and never
|
||||
become combat evidence. Binding `npcs` to extraction and normalization is
|
||||
stage-local, so an operator-driven combat run uses two explicit selectors:
|
||||
stage. The deterministic normalizer declares only `npcs`, whose operation-time
|
||||
registry supports the same actor and target canonicalization. Each `npcs` slot
|
||||
accepts exactly one UTF-8 `application/json` artifact no larger than 1 MiB. The
|
||||
registry's source ranges remain provenance for the reference and never become
|
||||
combat evidence. An ordered step binding fans the same generated NPC artifact
|
||||
out to extraction and normalization:
|
||||
|
||||
```text
|
||||
combat.extract.npcs=<npc-run>/lanes/npcs.json
|
||||
combat.normalize.npcs=<npc-run>/lanes/npcs.json
|
||||
```yaml
|
||||
references:
|
||||
npcs:
|
||||
artifact:
|
||||
step: identify-npcs
|
||||
lane: npcs
|
||||
```
|
||||
|
||||
When bound, the combat extractor and normalizer record only the registry's
|
||||
semantic digest and count in their metadata and checkpoint fingerprints; names,
|
||||
aliases, content, and paths are not recorded there. When absent, the combat
|
||||
prompt receives the exact empty registry value `{"npcs":[]}` and no registry
|
||||
provenance or fingerprint is recorded.
|
||||
When bound, the combat extractor and normalizer receive the generated registry
|
||||
at operation time. Framework provenance and checkpoint dependencies contain its
|
||||
kind, schema identity, media type, canonical digest, size, and bounded producer
|
||||
identity; names, aliases, content, and paths are not recorded there. When
|
||||
absent, the combat prompt receives the exact empty registry value
|
||||
`{"npcs":[]}` and no registry provenance or fingerprint is recorded.
|
||||
|
||||
## State Surfaces
|
||||
|
||||
@@ -572,8 +645,11 @@ Configuration validation checks:
|
||||
Pipeline resolution additionally checks:
|
||||
|
||||
- the pipeline ID exists;
|
||||
- at least one artifact lane is declared and selected;
|
||||
- at least one artifact lane is declared and selected in each explicit step;
|
||||
- `artifacts` and `steps` are mutually exclusive, explicit steps are non-empty,
|
||||
and step IDs are unique after trimming;
|
||||
- lanes selected through the CLI exist in the resolved pipeline;
|
||||
- lane IDs are globally unique across ordered steps;
|
||||
- required module keys are present;
|
||||
- module keys are registered for the expected slot;
|
||||
- module capability requirements are satisfied;
|
||||
@@ -583,4 +659,7 @@ Pipeline resolution additionally checks:
|
||||
Scriptorium profile IDs;
|
||||
- bound reference slots are declared by selected chunk, extractor, merger, or
|
||||
normalizer targets;
|
||||
- generated references identify one lane in an earlier step, use a declared
|
||||
compatible artifact kind, and do not conflict with external or target-local
|
||||
generated bindings;
|
||||
- required reference slots are bound for selected targets.
|
||||
|
||||
@@ -92,6 +92,9 @@ chunk-scoped transcript plus the existing `players`, `party`, and `glossary`
|
||||
inputs, and optionally the deprecated `roster` reference through the shared
|
||||
party mapping. The optional `npcs` reference is an approved normalized NPC
|
||||
artifact used only for identity grounding; it never supplies combat evidence.
|
||||
An external file is validated during preparation. In an ordered pipeline, the
|
||||
same slot may receive the producer's canonical generated artifact at the step
|
||||
handoff.
|
||||
|
||||
The private response envelope has the same fields and JSON types as the durable
|
||||
turn/action shape except that source references contain only `start_unit_id`
|
||||
@@ -130,8 +133,10 @@ schema, combat shape, source references, then source relatedness.
|
||||
The standalone normalizer uses key `dnd/combat-turns`, requires `merged`,
|
||||
provides `normalized`, accepts no options, and accepts only the optional
|
||||
structured `npcs` reference. Campaign references are LLM extraction context and
|
||||
are not normalizer inputs. The NPC registry is resolved during preparation;
|
||||
runtime normalization uses that immutable prepared view.
|
||||
are not normalizer inputs. For an external file, the NPC registry is resolved
|
||||
during preparation; for a generated binding, it is resolved at the operation-
|
||||
time handoff. Runtime normalization uses that immutable prepared or handed-off
|
||||
view.
|
||||
|
||||
Normalization policy is `dnd.combat_turns.normalize.v1`. It display-normalizes
|
||||
actor, summary, declarations, targets, and non-null resolutions; canonicalizes
|
||||
@@ -143,9 +148,11 @@ merging its actions or prose. Invalid evidence is never eligible for duplicate
|
||||
collapse. Every mutation and collapse emits a bounded warning using the merged
|
||||
input index in its scope.
|
||||
|
||||
The normalizer reports `normalization_policy` and `identity_policy` metadata and
|
||||
fingerprints, plus `npc_registry_digest`, `npc_count`, and `npc_registry` only
|
||||
when a registry is bound. The normalized-invariants validator is
|
||||
The normalizer reports `normalization_policy` and `identity_policy` metadata
|
||||
and fingerprints. An external registry may additionally contribute
|
||||
`npc_registry_digest` and `npc_count`; generated registry identity is retained
|
||||
in framework handoff provenance and dependency fingerprints. The
|
||||
normalized-invariants validator is
|
||||
`normalize/dnd/combat-turns/invariants`; it defers shape and source-reference
|
||||
failures, then checks display normalization, target identity uniqueness,
|
||||
canonical evidence ordering, chronology, and duplicate identity. It rejects
|
||||
@@ -162,9 +169,10 @@ merge validator chain.
|
||||
|
||||
The selectable lane uses extractor and normalizer key `dnd/combat-turns`,
|
||||
`appendorder` for the typed merger, and the durable codec above. A bound `npcs`
|
||||
reference contributes raw-file provenance to the run manifest. Prepared combat
|
||||
extractor and normalizer metadata and checkpoint fingerprints contain only the
|
||||
NPC registry's semantic digest and count; the registry content, path, and NPC
|
||||
source ranges are not copied into combat output. The normalized lane is emitted
|
||||
as `lanes/<lane-id>.json` by the JSON output module, and warnings and rejection
|
||||
summaries remain in their shared companion files.
|
||||
reference contributes raw-file provenance to the run manifest. A generated
|
||||
binding contributes artifact kind, schema identity, media type, canonical
|
||||
digest, size, and bounded producer provenance. Consumer metadata and checkpoint
|
||||
fingerprints contain no registry names, aliases, content, paths, or NPC source
|
||||
ranges. The normalized lane is emitted as `lanes/<lane-id>.json` by the JSON
|
||||
output module, and warnings and rejection summaries remain in their shared
|
||||
companion files.
|
||||
|
||||
@@ -85,23 +85,43 @@ checks. Relatedness emits bounded warnings when an NPC canonical name or
|
||||
alias is not present near its cited transcript text; opaque campaign
|
||||
references may explain such a warning but do not become evidence.
|
||||
|
||||
## Manifest And Sequential Consumption
|
||||
## Manifest And Artifact Handoff
|
||||
|
||||
The NPC extractor records prompt and response-schema identities. The durable
|
||||
codec records only `npc_count`; raw names, aliases, descriptions, source
|
||||
references, and payload bytes stay in the lane file rather than manifest
|
||||
metadata. The normalized lane is independently reusable as a file reference:
|
||||
metadata. The normalized lane can be consumed by a later ordered step through
|
||||
the registered canonical codec:
|
||||
|
||||
```sh
|
||||
go run ./cmd/notarius run dnd-spells \
|
||||
--config examples/dnd-npc-spell-sequential.config.yml \
|
||||
--input examples/seriatim-minimal-transcript.json \
|
||||
--reference spells.extract.npcs=./npc-output/<run-id>/lanes/npcs.json
|
||||
```yaml
|
||||
steps:
|
||||
- id: identify-npcs
|
||||
artifacts:
|
||||
npcs:
|
||||
extract: dnd/npcs
|
||||
normalize: dnd/npcs
|
||||
- id: grounded-events
|
||||
references:
|
||||
npcs:
|
||||
artifact:
|
||||
step: identify-npcs
|
||||
lane: npcs
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
normalize: dnd/spells
|
||||
combat:
|
||||
extract: dnd/combat-turns
|
||||
normalize: dnd/combat-turns
|
||||
```
|
||||
|
||||
The spell extractor strictly decodes and identity-validates this file, accepts
|
||||
source references belonging to another session as registry provenance, and
|
||||
uses only canonical names and aliases for caster grounding. Those NPC source
|
||||
references are never accepted as spell evidence. The spell run's manifest
|
||||
keeps raw file provenance under `references` and records only the prepared
|
||||
registry's semantic digest and count in extractor metadata.
|
||||
The framework hands only an accepted normalized artifact across the barrier. It
|
||||
validates the canonical bytes against each consumer slot and clones the
|
||||
operation-time reference for the spell and combat consumers. Generated
|
||||
provenance records the artifact kind, schema identity, media type, canonical
|
||||
digest, size, and producer step/lane/module, but not names, aliases, source
|
||||
ranges, or payload bytes. External normalized files remain supported as
|
||||
explicit references and retain their file provenance.
|
||||
|
||||
NPC source references are registry provenance and are never accepted as spell
|
||||
or combat evidence. Current transcript units remain the only event evidence.
|
||||
|
||||
@@ -97,18 +97,20 @@ addressable through `source_refs`.
|
||||
## Optional NPC Grounding
|
||||
|
||||
The `dnd/spells` extractor accepts an optional `npcs` reference containing one
|
||||
normalized NPC artifact as `application/json`, up to 1 MiB. Preparation uses
|
||||
the approved NPC codec and identity policy to validate the file, re-encodes
|
||||
canonical durable JSON, and supplies that JSON as a spell-owned prompt input.
|
||||
It helps the model prefer canonical caster names and recognize aliases; it
|
||||
does not establish that a spell was cast.
|
||||
normalized NPC artifact as `application/json`, up to 1 MiB. An external file is
|
||||
validated during preparation; an ordered generated binding is validated at the
|
||||
step handoff. Both paths use the approved NPC codec and identity policy,
|
||||
re-encode canonical durable JSON, and supply that JSON as an operation-time
|
||||
spell prompt input. It helps the model prefer canonical caster names and
|
||||
recognize aliases; it does not establish that a spell was cast.
|
||||
|
||||
NPC source references may identify the run that produced the registry or any
|
||||
other session. They remain registry provenance and are never copied into a
|
||||
spell cast's `source_refs`; every spell evidence range must still identify the
|
||||
current transcript. When the slot is absent, the prompt receives exactly
|
||||
`{"npcs":[]}` and the run has no NPC reference provenance or NPC checkpoint
|
||||
fingerprint.
|
||||
current transcript. Generated provenance records producer and canonical
|
||||
artifact identity without payload content or a path. When the slot is absent,
|
||||
the prompt receives exactly `{"npcs":[]}` and the run has no NPC reference
|
||||
provenance or NPC checkpoint fingerprint.
|
||||
|
||||
## Normalization Behavior
|
||||
|
||||
@@ -190,15 +192,20 @@ identity fields when that module is selected. Overlay origin, media type, byte
|
||||
size, and raw digest are recorded separately in the manifest's reference
|
||||
provenance; see the [JSON output contract](json-output.md#manifestjson).
|
||||
|
||||
The `npc_registry_digest` and `npc_count` fields in the example are present only
|
||||
when the optional NPC registry is bound. They contain no NPC names, aliases,
|
||||
source references, paths, or raw bytes.
|
||||
The `npc_registry_digest` and `npc_count` fields in the example are present for
|
||||
an external NPC registry when the extractor publishes its prepared module
|
||||
metadata. They contain no NPC names, aliases, source references, paths, or raw
|
||||
bytes. A generated registry's identity is instead represented by the framework
|
||||
handoff provenance and dependency fingerprint, so the consumer module metadata
|
||||
does not duplicate it.
|
||||
|
||||
The extractor's prompt hash, private response-schema hash, and effective catalog
|
||||
digest also contribute independently scoped semantic checkpoint fingerprints.
|
||||
Changing any of those prepared contracts intentionally produces a cold
|
||||
checkpoint miss. Fingerprints contain only digests, never prompt, schema,
|
||||
catalog, or reference content. When an NPC registry is bound, its semantic
|
||||
digest contributes an additional local `npc_registry` fingerprint; the
|
||||
manifest metadata contains only that digest and `npc_count`. Raw NPC file
|
||||
provenance remains independently recorded in the manifest's `references` list.
|
||||
digest contributes an additional local `npc_registry` fingerprint for an
|
||||
external binding; the manifest metadata contains only that digest and
|
||||
`npc_count`. Raw NPC file provenance remains independently recorded in the
|
||||
manifest's `references` list. Generated bindings contribute the canonical
|
||||
artifact dependency fingerprint and bounded producer provenance instead.
|
||||
|
||||
@@ -104,11 +104,12 @@ The NPC identity package owns Unicode comparison keys, deterministic
|
||||
The registry package resolves one optional normalized artifact through the
|
||||
strict codec, validates whole-registry identity, canonicalizes its JSON, and
|
||||
provides immutable records, prompt input, semantic digest, count, and exact
|
||||
canonical-name/alias lookup. It owns the `npcs` slot and its bounded,
|
||||
content-safe preparation failures. NPC source references are durable
|
||||
provenance and are not treated as evidence for a consuming pipeline. The NPC
|
||||
codec owns the strict durable `dnd/npc-list` JSON boundary and exposes
|
||||
candidate versus approved encode/decode operations.
|
||||
canonical-name/alias lookup. External files cross this boundary during
|
||||
preparation; generated artifacts cross it at the ordered step handoff. It owns
|
||||
the `npcs` slot and its bounded, content-safe validation failures. NPC source
|
||||
references are durable provenance and are not treated as evidence for a
|
||||
consuming pipeline. The NPC codec owns the strict durable `dnd/npc-list` JSON
|
||||
boundary and exposes candidate versus approved encode/decode operations.
|
||||
|
||||
The `internal/modules/dnd/codec/combatturns` package owns the durable
|
||||
`dnd/combat-turn-list` schema and candidate versus approved JSON boundary. It
|
||||
@@ -220,9 +221,12 @@ Shared D&D helpers keep prompt input
|
||||
names and source-unit reference conversion consistent with the scene chunker.
|
||||
|
||||
The extractor also declares the optional `npcs` registry slot and consumes the
|
||||
prepared immutable registry boundary from `internal/modules/dnd/npcs/registry`.
|
||||
A bound registry adds only `npc_registry_digest` and `npc_count` to manifest
|
||||
metadata and an `npc_registry` checkpoint fingerprint. The unbound prompt
|
||||
immutable registry boundary from `internal/modules/dnd/npcs/registry`. An
|
||||
external registry is prepared before execution; a generated registry is
|
||||
validated and supplied at operation time. External bindings may add only
|
||||
`npc_registry_digest` and `npc_count` to module metadata and an
|
||||
`npc_registry` checkpoint fingerprint. Generated bindings are represented by
|
||||
framework handoff provenance and dependency fingerprints. The unbound prompt
|
||||
input is exactly `{"npcs":[]}` and has no registry provenance or fingerprint.
|
||||
The shared NPC grounding fragment is placed immediately after the common
|
||||
campaign reference message and is included in the spell prompt fingerprint.
|
||||
@@ -310,16 +314,18 @@ independently for extraction and normalization.
|
||||
|
||||
### `internal/modules/dnd/normalize/combatturns`
|
||||
|
||||
The combat normalizer prepares the optional NPC registry once and uses the
|
||||
immutable prepared view during runtime. It display-normalizes combat fields,
|
||||
The combat normalizer prepares an external NPC registry before execution or
|
||||
receives a generated registry at the ordered step handoff, then uses the
|
||||
immutable view during runtime. It display-normalizes combat fields,
|
||||
rewrites exact canonical-name or alias matches for actors and targets, orders
|
||||
and deduplicates source references, stable-sorts records by source-document
|
||||
position, and collapses only exact duplicate identities with fully valid
|
||||
evidence. It deep-clones output storage and emits bounded warnings scoped to
|
||||
merged input indexes. Its metadata and fingerprints identify the normalization
|
||||
and NPC identity policies, with registry digest/count only when bound. The
|
||||
normalizer is included in the production D&D registrar with the default combat
|
||||
normalization chain.
|
||||
and NPC identity policies. External bindings may contribute registry
|
||||
digest/count metadata; generated identity is retained in framework provenance
|
||||
and dependency fingerprints. The normalizer is included in the production D&D
|
||||
registrar with the default combat normalization chain.
|
||||
|
||||
## Output Encoder
|
||||
|
||||
|
||||
@@ -14,11 +14,13 @@ collaborators, invokes `internal/framework/pipeline`, and places the logical
|
||||
output files returned by the runner. Cache and debug collaborators are supplied
|
||||
at this boundary.
|
||||
|
||||
Resolution produces a fixed ordered workflow and a sorted set of artifact
|
||||
lanes. Preparation constructs the complete module and validator set before the
|
||||
runner receives source bytes. Source parsing and chunking are serial; extraction
|
||||
uses a bounded run-wide worker pool, followed by serial per-lane merge and
|
||||
normalize continuations that may overlap across lanes.
|
||||
Resolution produces a fixed ordered workflow of steps and globally unique,
|
||||
sorted artifact lanes. Preparation constructs the complete module and validator
|
||||
set before the runner receives source bytes. Source parsing and chunking are
|
||||
serial. Each step then uses a bounded run-wide extraction pool followed by
|
||||
serial per-lane merge and normalize continuations. A step barrier prevents
|
||||
later consumers from starting until all earlier lanes are terminal and their
|
||||
required normalized artifacts have crossed the typed handoff.
|
||||
|
||||
## Application Boundary
|
||||
|
||||
@@ -42,7 +44,7 @@ normalize continuations that may overlap across lanes.
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
|
||||
| `internal/framework/pipeline` | Module and artifact-codec registries, option validation, profile resolution, capability checks, reference materialization, complete pipeline preparation, retries, orchestration, warnings, and manifest population. |
|
||||
| `internal/framework/pipeline` | Module and artifact-codec registries, ordered-step and generated-reference resolution, option validation, profile resolution, capability checks, external reference materialization, complete pipeline preparation, typed handoff, retries, orchestration, warnings, checkpoint decisions, and manifest population. |
|
||||
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
|
||||
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
|
||||
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
|
||||
@@ -111,12 +113,15 @@ this package. Domain-neutral prompt filesystem composition lives in
|
||||
`internal/framework/promptfs`.
|
||||
|
||||
The `dnd/npcs/registry` package owns the optional `npcs` registry boundary.
|
||||
Preparation strictly decodes and identity-validates one normalized JSON
|
||||
artifact, emits canonical registry JSON to the spell prompt, and records only
|
||||
its semantic digest and count in prepared metadata. The raw reference remains
|
||||
independently tracked by pipeline provenance. An absent registry is represented
|
||||
only by the empty prompt value `{"npcs":[]}`. Spell extraction consumes this
|
||||
shared registry boundary without changing its public module contract.
|
||||
External references are strictly decoded and identity-validated during
|
||||
preparation; generated references are decoded and identity-validated at the
|
||||
ordered step handoff. Both paths emit canonical registry JSON to operation-time
|
||||
spell and combat prompt or normalization requests. The framework records
|
||||
generated identity and bounded producer provenance, while the raw external
|
||||
reference remains independently tracked by pipeline provenance. An absent
|
||||
registry is represented only by the empty prompt value `{"npcs":[]}`. Spell
|
||||
and combat consumers use this shared boundary without changing their public
|
||||
module contracts.
|
||||
|
||||
Generic validators under `internal/modules/generic/validate` provide
|
||||
unconditional test decisions, JSON syntax validation, and JSON Schema
|
||||
@@ -144,12 +149,12 @@ Implementation details for all production extensions are in
|
||||
| Surface | Implemented owners | Internal purpose |
|
||||
| --- | --- | --- |
|
||||
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
|
||||
| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes. |
|
||||
| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes, including ordered-step scope and generated-artifact dependency decisions. |
|
||||
| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. |
|
||||
| Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. |
|
||||
|
||||
Physical layout, cleanup, recovery, and sensitive-data handling are defined
|
||||
in [Operations](../operations.md). Concrete stage modules receive recorder
|
||||
in [Operations](../operations.md). Concrete modules receive recorder
|
||||
interfaces and request data, not physical state roots.
|
||||
|
||||
## Focused Documentation
|
||||
|
||||
@@ -6,13 +6,15 @@ Their fixed workflow and ownership boundaries are defined by
|
||||
defaults, and selectable keys are defined in
|
||||
[Configuration](../config.md#pipelines).
|
||||
|
||||
Resolution fixes the selected lanes and all stage bindings; preparation
|
||||
constructs every selected implementation before the runner begins source work.
|
||||
After serial input parsing and plan selection or generation, the runner
|
||||
materializes chunks and dispatches extract work to
|
||||
one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
|
||||
merge and normalize operations remain serial and may overlap other lanes once
|
||||
all extracts for that lane are terminal.
|
||||
Resolution fixes the ordered steps, selected lanes, and all stage bindings;
|
||||
preparation constructs every selected implementation before the runner begins
|
||||
source work. After serial input parsing and plan selection or generation, the
|
||||
runner materializes chunks and executes one step at a time. Within a step,
|
||||
extract work uses one bounded run-wide worker pool in chunk-first, lane-second
|
||||
order. Each lane's merge and normalize operations remain serial, and lanes in
|
||||
the same step may overlap once their extracts are terminal. A later step cannot
|
||||
start across its barrier until every earlier lane is terminal and each required
|
||||
generated artifact has been accepted and handed off.
|
||||
|
||||
## Resolution
|
||||
|
||||
@@ -22,38 +24,48 @@ calls `pipeline.ResolvePipeline`.
|
||||
|
||||
`ResolvePipeline`:
|
||||
|
||||
1. selects and sorts artifact lanes;
|
||||
2. completes omitted bindings using the documented configuration defaults;
|
||||
3. looks up each module and validator spec without constructing it;
|
||||
4. for a typed extractor, derives its artifact kind, requires the codec, and
|
||||
1. selects the explicit ordered steps, or creates the implicit `default` step
|
||||
from the legacy top-level `artifacts` map;
|
||||
2. selects and sorts artifact lanes within each step while enforcing global lane
|
||||
identity;
|
||||
3. completes omitted bindings using the documented configuration defaults;
|
||||
4. looks up each module and validator spec without constructing it;
|
||||
5. for a typed extractor, derives its artifact kind, requires the codec, and
|
||||
selects exact-type merger, normalizer, and validator variants;
|
||||
5. checks required and provided capabilities in workflow order;
|
||||
6. resolves target-aware reference bindings and validator chains;
|
||||
7. validates each selected module and validator option set through its registry
|
||||
6. checks required and provided capabilities in workflow order;
|
||||
7. resolves external and generated target-aware reference bindings and
|
||||
validates producer order, consumer slot declarations, and artifact-kind
|
||||
compatibility;
|
||||
8. validates each selected module and validator option set through its registry
|
||||
entry; and
|
||||
8. calculates a digest over the resolved structure, including typed artifact
|
||||
kind and schema identity and the effective validator policy in its resolved
|
||||
execution order.
|
||||
9. calculates a digest over the resolved structure, including step order, step
|
||||
IDs, lane membership, generated topology, producer and consumer identities,
|
||||
typed artifact kind and schema identity, and the effective validator policy
|
||||
in its resolved execution order.
|
||||
|
||||
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
|
||||
bindings, validator chains, reference targets, and the digest. It does not read
|
||||
reference bytes or construct runtime modules. CLI lane and reference selector
|
||||
syntax is defined in the [CLI reference](../cli.md#run).
|
||||
Resolution returns a `ResolvedPipeline` containing ordered steps, lanes,
|
||||
concrete bindings, validator chains, reference targets, and the digest. It does
|
||||
not read external reference bytes or construct runtime modules. CLI lane and
|
||||
reference selector syntax is defined in the [CLI reference](../cli.md#run).
|
||||
|
||||
The digest includes each resolved validator chain's stage, lane, owning module,
|
||||
ordered validator bindings, execution classes, targets, and artifact kinds.
|
||||
Changing a default chain or an explicit override therefore changes pipeline
|
||||
identity whenever it changes the effective validator policy.
|
||||
The digest includes each resolved step's ID and lane membership, generated
|
||||
producer/consumer topology, and each validator chain's stage, lane, owning
|
||||
module, ordered validator bindings, execution classes, targets, and artifact
|
||||
kinds. Changing step order, a dependency, a default chain, or an explicit
|
||||
override therefore changes pipeline identity whenever it changes effective
|
||||
execution policy.
|
||||
|
||||
## Reference Materialization
|
||||
|
||||
The CLI calls `MaterializeReferences` after resolution and before constructing
|
||||
the LLM client or running the pipeline. The materializer checks each binding
|
||||
against its resolved target declaration, reads and validates the file, and
|
||||
builds both a `contracts.ReferenceSet` and provenance-only metadata on the
|
||||
corresponding `ResolvedReferenceTarget`.
|
||||
the LLM client or running the pipeline. For external bindings, the materializer
|
||||
checks each binding against its resolved target declaration, reads and validates
|
||||
the file, and builds both a `contracts.ReferenceSet` and provenance-only
|
||||
metadata on the corresponding `ResolvedReferenceTarget`. A structured
|
||||
generated binding is declaration-only at this point: its producer bytes do not
|
||||
exist until the producer lane reaches an accepted normalized result.
|
||||
|
||||
Preparation delivers the materialized set for each target through
|
||||
Preparation delivers the materialized external set for each target through
|
||||
`pipeline.BuildRequest`: chunkers and chunk validators receive the chunk target;
|
||||
extractors and extract validators receive the lane extract target; mergers and
|
||||
merge validators receive the lane merge target; and normalizers and normalize
|
||||
@@ -62,10 +74,20 @@ an empty set because those stages cannot declare references. Every builder gets
|
||||
an isolated deep clone of its target set, so construction-time mutation cannot
|
||||
change another builder, the resolved pipeline, or later runtime requests.
|
||||
|
||||
Prepared consumers do not need to be reconstructed when generated content is
|
||||
available. At the step boundary, the runner encodes the accepted producer value
|
||||
through its registered canonical codec, validates the generated bytes against
|
||||
each target slot's kind, schema, media type, and size, and clones one immutable
|
||||
reference item into the operation request. The item includes canonical digest,
|
||||
size, and bounded producer provenance but no filesystem URI. A handoff failure
|
||||
is a framework dependency error and prevents every consumer in that step from
|
||||
starting.
|
||||
|
||||
The runner continues to clone the resulting set into the chunk, extract, merge,
|
||||
or normalize request that owns the target. LLM-backed extensions may convert
|
||||
those items into named prompt inputs. Reference content remains separate from
|
||||
source evidence and source digests.
|
||||
source evidence and source digests, whether the item came from a file or a
|
||||
generated handoff.
|
||||
|
||||
Binding precedence, path resolution, accepted content, and media-type behavior
|
||||
are configuration contracts; see [Configuration](../config.md#pipelines).
|
||||
@@ -120,10 +142,12 @@ The current production catalog and default chain are listed only in
|
||||
## Preparation And Runner Boundary
|
||||
|
||||
`pipeline.Prepare` receives a resolved pipeline, the registries, and shared
|
||||
module dependencies. It constructs input; chunk and its validators; each lane's
|
||||
extract, merge, and normalize modules and validator chains in resolved order;
|
||||
then output. It stops at the first error with pipeline, stage, lane, module, and
|
||||
validator context as applicable. It never invokes an operation method.
|
||||
module dependencies. It constructs input; chunk and its validators; every
|
||||
step's lane extract, merge, and normalize modules and validator chains in
|
||||
resolved order; then output. It stops at the first error with pipeline, step,
|
||||
stage, lane, module, and validator context as applicable. It never invokes an
|
||||
operation method. Generated references are not available during preparation;
|
||||
the operation request is the handoff boundary.
|
||||
|
||||
`PreparedPipeline` keeps private constructed executors and exposes cloned
|
||||
resolved input, chunk, lane, and output identities. Prepared components may
|
||||
@@ -136,8 +160,8 @@ stable and must not contain source content, credentials, local paths,
|
||||
timestamps, or other invocation-specific values.
|
||||
|
||||
`pipeline.RunInput` carries that prepared pipeline, raw source input, run identity and timing, optional
|
||||
session and profile metadata, a chunk-plan store and mode, and checkpoint/debug
|
||||
collaborators. The runner
|
||||
session and profile metadata, a chunk-plan store and mode, a checkpoint
|
||||
execution policy, and checkpoint/debug collaborators. The runner
|
||||
parses source bytes through the already constructed input adapter. Later stage
|
||||
requests receive the generic source model; extract requests receive
|
||||
chunk-scoped input material, while chunk, merge, and normalize requests retain
|
||||
@@ -145,7 +169,9 @@ access to the original source material. Input, chunk, and output operation
|
||||
requests do not carry raw module options. The chunk request also does not carry
|
||||
an LLM client; an LLM-backed chunker receives the shared client during
|
||||
preparation. Their operation requests retain run-specific source, reference,
|
||||
profile, session, and metadata context as applicable.
|
||||
profile, session, metadata, and step-handoff context as applicable. A generated
|
||||
reference is cloned into each compatible consumer request and is never exposed
|
||||
as a path.
|
||||
|
||||
Prepared lanes retain exact-type-checked erased operation closures. The runner
|
||||
uses those closures to keep each value typed through extraction, validation,
|
||||
@@ -165,6 +191,13 @@ the runner returns.
|
||||
|
||||
## Execution Flow
|
||||
|
||||
The pipeline-wide coordinator owns the ordered step loop, generated-reference
|
||||
sets at each barrier, and deterministic merging of step outcomes. For one step,
|
||||
the lane engine initializes checkpoint state in lane order, dispatches bounded
|
||||
extract work, advances terminal lanes through serial merge and normalize work,
|
||||
selects failures by stable pipeline scope, and merges lane-local outcomes back
|
||||
in resolved order. Completion timing never becomes public ordering.
|
||||
|
||||
The runner:
|
||||
|
||||
1. validates its prepared input;
|
||||
@@ -173,10 +206,15 @@ The runner:
|
||||
3. selects a stored plan or executes the configured chunker's `Plan` operation;
|
||||
4. canonicalizes and materializes the plan, then validates the resulting
|
||||
chunks;
|
||||
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
|
||||
bounded lane continuation when all extracts for that lane are terminal;
|
||||
6. invokes the prepared output encoder and validates its logical file results;
|
||||
7. returns the assembled manifest, outcomes, warnings, and files.
|
||||
5. executes each resolved step in configuration order. For one step, it
|
||||
dispatches extract jobs in source-chunk then resolved-lane order, starts a
|
||||
bounded lane continuation when all extracts for that lane are terminal, and
|
||||
waits for every lane to become terminal;
|
||||
6. encodes and validates each accepted normalized producer artifact, then
|
||||
builds the immutable generated reference sets for the next step;
|
||||
7. invokes the prepared output encoder only after every step succeeds and
|
||||
validates its logical file results;
|
||||
8. returns the assembled manifest, outcomes, warnings, and files.
|
||||
|
||||
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
|
||||
and validators while performing these transitions:
|
||||
@@ -190,6 +228,13 @@ and validators while performing these transitions:
|
||||
6. normalize the accepted merge result;
|
||||
7. validate and append the accepted normalized result.
|
||||
|
||||
At a step barrier, a lane with no accepted normalized output is still a regular
|
||||
rejection unless a later generated binding names that lane as a required
|
||||
producer. In that case the runner raises a deterministic dependency error and
|
||||
does not start the consumer step. One accepted typed artifact may fan out to
|
||||
multiple compatible target slots. Consumers in the same step may run
|
||||
concurrently after the handoff; no work crosses the barrier early.
|
||||
|
||||
Module-provided warnings and payload warnings are promoted only from attempts
|
||||
whose results are accepted and used.
|
||||
|
||||
@@ -268,13 +313,42 @@ implementations when collaborators are absent. Each checkpointed workflow
|
||||
boundary records a running, succeeded, or failed transition. Reuse decisions
|
||||
are consulted in workflow order and accepted payloads are cloned before
|
||||
entering the normal handoff path. Typed extract, merge, and normalize
|
||||
checkpoints store codec bytes with artifact kind, schema ID and version, exact
|
||||
schema digest, and media type. Reuse compares that identity with the prepared
|
||||
checkpoints store codec bytes with artifact kind, schema ID, name, version and
|
||||
exact digest, and media type. Reuse compares that identity with the prepared
|
||||
codec and decodes through the codec; missing identity, mismatches, corrupt
|
||||
bytes, and decode failures become explicit reuse misses and execute the step
|
||||
bytes, and decode failures become explicit reuse misses and execute the lane
|
||||
normally. Dependency fingerprints and debug content digests use the same stable
|
||||
codec bytes that cross those boundaries.
|
||||
|
||||
That progressive extract, merge, and normalize reuse is the ordinary resume
|
||||
path. A lane marked as a required predecessor for selective recomputation takes
|
||||
a separate accepted-output path before extract scheduling. The loader reads the
|
||||
existing successful normalize manifest and payload by step, lane, and
|
||||
normalizer, without consulting extract or merge dependencies. It requires the
|
||||
current non-empty checkpoint identity to match, so the invocation identity
|
||||
still binds the input, resolved topology and configuration, references, runtime
|
||||
overrides, profiles, and component fingerprints.
|
||||
|
||||
The runner decodes that accepted normalized artifact with the prepared codec,
|
||||
re-encodes it, and requires exact kind, schema identity and digest, media type,
|
||||
canonical bytes, content digest, and producer provenance. A valid result becomes
|
||||
a runner-owned cloned normalized output, restores only normalize-checkpoint
|
||||
warnings, and records one `accepted_artifact_reused` normalize decision. It does
|
||||
not invoke or record extract, merge, normalize, or their validators. Invalid or
|
||||
unavailable accepted state records its decision and fails the producer step;
|
||||
the dependent step never starts and the producer is not implicitly rerun.
|
||||
|
||||
Generated references add downstream dependencies containing the producer's
|
||||
artifact kind, complete schema identity, media type, canonical content digest,
|
||||
and size. Compatible accepted producer outputs may therefore feed a later step
|
||||
without re-executing the producer. Forced lanes bypass accepted-output
|
||||
hydration and execute normally. A missing, rejected, corrupt, incompatible, or
|
||||
changed producer blocks its dependent while leaving independent work eligible
|
||||
for reuse. The runner records bounded decision
|
||||
categories: `reused`, `executed`, `forced_recompute`, and
|
||||
`dependency_invalidated`. Operator meanings for the stable reason codes belong
|
||||
to [Operations](../operations.md#resume-and-selective-recompute).
|
||||
|
||||
The CLI includes prepared-component fingerprints in the run-wide checkpoint
|
||||
identity alongside resolved configuration, raw input, reference provenance,
|
||||
runtime overrides, and LLM-profile fingerprints. Module metadata is not used
|
||||
@@ -283,6 +357,11 @@ values that can change accepted output. Adding or changing a component
|
||||
fingerprint intentionally produces a cold cache miss. Existing checkpoint
|
||||
schemas and paths remain unchanged.
|
||||
|
||||
The CLI's `--recompute-step` policy forces the selected step and all transitive
|
||||
dependents, but requires accepted normalized artifacts for every unselected
|
||||
producer on which that closure depends. It changes execution policy only; it
|
||||
does not alter persistent checkpoint identity.
|
||||
|
||||
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
|
||||
boundaries. Every executed chunk, extract, merge, and normalize attempt writes
|
||||
one terminal envelope for acceptance, validator rejection, module or validator
|
||||
@@ -347,7 +426,9 @@ stage, resolved lane, and source chunk rather than completion time.
|
||||
- `internal/framework/pipeline/references_test.go`: target resolution and
|
||||
materialization.
|
||||
- `internal/cli/run_contract_test.go`: production run transitions, retries,
|
||||
rejections, warnings, debug hooks, and manifests.
|
||||
rejections, warnings, CLI recomputation controls, debug hooks, and manifests.
|
||||
- `internal/cli/recompute_execution_contract_test.go`: filesystem-backed
|
||||
selective recomputation and accepted-producer recovery.
|
||||
- `internal/cli/production_contract_test.go`: production composition and
|
||||
configuration-resolution smoke coverage.
|
||||
- `internal/cli/example_contract_test.go`: maintained example resolution and
|
||||
|
||||
@@ -32,6 +32,40 @@ The serialized
|
||||
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
|
||||
they do not describe a current public state surface.
|
||||
|
||||
Ordered-step lane checkpoints include the step identity in their storage scope.
|
||||
When a later lane consumes a generated artifact, its dependency fingerprints
|
||||
include the producer's artifact kind, complete schema identity, media type,
|
||||
canonical content digest, and size. Ordinary resume compares those fingerprints
|
||||
when progressively loading consumer stage checkpoints, so changed producer
|
||||
content produces `dependency_invalidated` rather than stale downstream reuse.
|
||||
Selective recomputation instead requires each unselected producer's accepted
|
||||
normalized artifact; invalid accepted state records its specific bounded reason
|
||||
and stops before the dependent. The selected step and its transitive dependents
|
||||
record `forced_recompute`.
|
||||
|
||||
Ordinary resume loads extract, merge, and normalize checkpoints progressively
|
||||
and may execute later lane stages after an earlier cache miss. Selective
|
||||
recomputation instead asks the loader for the required producer's accepted
|
||||
normalize artifact. That lookup reuses the existing normalize files, requires
|
||||
workspace schema v3 plus an exact non-empty invocation identity, and deliberately
|
||||
does not require extract or merge checkpoint files or dependency fingerprints.
|
||||
The runner performs canonical codec and producer-provenance validation before
|
||||
cloning the artifact into normal step output. Success restores only stored
|
||||
normalize warnings and emits one normalize decision; failure retains the files,
|
||||
records the decision, and stops without executing the producer or consumer.
|
||||
|
||||
The loader assigns a typed category and reason code at each validation site;
|
||||
diagnostic prose is not classified after the fact. The runner then applies
|
||||
forced-execution policy, validates reusable artifact bytes through the prepared
|
||||
codec, and records the final decision before enforcing a required-predecessor
|
||||
failure. That failure names only the step, lane, and stable reason code. Decision
|
||||
detail passes through one UTF-8-safe bounded sanitizer and contains only
|
||||
allowlisted diagnostic context, never payloads, references, credentials,
|
||||
environment values, or physical paths. Typed categories and codes remain intact
|
||||
through pipeline events and become strings only in manifest and debug-summary
|
||||
JSON. [Operations](../operations.md#resume-and-selective-recompute) is the
|
||||
canonical operator-facing reason-code reference.
|
||||
|
||||
`internal/core/fileio` provides confined atomic file writes used by state
|
||||
collaborators. The chunk-plan store retains its stronger entry validation.
|
||||
|
||||
@@ -48,6 +82,11 @@ boundaries redact sensitive metadata and credential-shaped bytes while allowing
|
||||
application-owned trace material. Debug data is never a checkpoint source or
|
||||
cache input.
|
||||
|
||||
Generated reference bytes exist only in cloned operation requests and are not
|
||||
written as paths into checkpoints, manifests, or debug summaries. Those state
|
||||
surfaces retain canonical identities and bounded producer provenance so that a
|
||||
resume decision can be explained without copying generated campaign content.
|
||||
|
||||
After allocation, one CLI-owned state value accumulates the known report paths,
|
||||
pipeline outcome counts, and validation status. A single guarded terminalization
|
||||
operation writes the success report, or makes one attempt each to write the
|
||||
@@ -60,6 +99,8 @@ separately and never replace the command's primary error.
|
||||
terminalization, and output/report boundaries.
|
||||
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
|
||||
permissions, cleanup, and redaction.
|
||||
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
|
||||
filesystem recovery, deterministic decisions, and failed predecessor state.
|
||||
- `internal/cli/production_contract_test.go`: production composition and
|
||||
configuration validation at the CLI boundary.
|
||||
- `internal/cli/example_contract_test.go`: maintained example ownership.
|
||||
|
||||
@@ -40,63 +40,33 @@ names, schemas, and media types inside a run directory.
|
||||
Remove an output run directory only after its consumer data is no longer
|
||||
needed. This is data deletion, not cache cleanup.
|
||||
|
||||
## Sequential NPC And Spell Runs
|
||||
## Ordered D&D Workflow
|
||||
|
||||
The maintained [sequential configuration](../examples/dnd-npc-spell-sequential.config.yml)
|
||||
contains two independent pipelines over the same Seriatim input shape. Run the
|
||||
NPC pipeline first and retain its normalized payload:
|
||||
The maintained [NPC-grounded configuration](../examples/dnd-npc-grounded.config.yml)
|
||||
contains one pipeline with two ordered steps. The first step extracts and
|
||||
normalizes NPCs. Only after that lane reaches an accepted terminal result does
|
||||
the second step begin; its generated NPC reference is supplied in memory to
|
||||
spell extraction, combat extraction, and combat normalization.
|
||||
|
||||
```sh
|
||||
go run ./cmd/notarius run dnd-npcs \
|
||||
--config examples/dnd-npc-spell-sequential.config.yml \
|
||||
go run ./cmd/notarius run dnd-npc-grounded \
|
||||
--config examples/dnd-npc-grounded.config.yml \
|
||||
--input examples/seriatim-minimal-transcript.json \
|
||||
--output-dir ./npc-output
|
||||
--output-dir ./npc-grounded-output
|
||||
```
|
||||
|
||||
Then bind that completed run's `lanes/npcs.json` file to the spell extractor:
|
||||
The NPC artifact grounds canonical names and aliases, not spell or combat
|
||||
evidence. Current-transcript source ranges remain the only event evidence. The
|
||||
manifest records generated-reference identity and bounded producer provenance;
|
||||
it does not record generated payload content, and no generated content is
|
||||
exposed through a filesystem path. The same producer artifact may fan out to
|
||||
compatible consumers, while a missing or rejected producer prevents the later
|
||||
step from starting.
|
||||
|
||||
```sh
|
||||
go run ./cmd/notarius run dnd-spells \
|
||||
--config examples/dnd-npc-spell-sequential.config.yml \
|
||||
--input examples/seriatim-minimal-transcript.json \
|
||||
--reference spells.extract.npcs=./npc-output/<run-id>/lanes/npcs.json
|
||||
```
|
||||
|
||||
The NPC file is a reference for canonical caster names and aliases, not spell
|
||||
evidence. The spell manifest records the bound file's raw reference provenance
|
||||
and the prepared registry's count and semantic digest separately. The NPC
|
||||
payload, names, aliases, source references, and file bytes can be sensitive
|
||||
campaign data; protect both output roots and any checkpoint or debug roots that
|
||||
retain derived application data. A registry from another session is allowed,
|
||||
but its source references are never copied into spell output evidence.
|
||||
|
||||
## Sequential NPC And Combat Runs
|
||||
|
||||
The maintained [sequential NPC and combat configuration](../examples/dnd-npc-combat-sequential.config.yml)
|
||||
also represents two independent runs. Run `dnd-npcs` first, then bind its
|
||||
normalized `lanes/npcs.json` payload to both combat stages:
|
||||
|
||||
```sh
|
||||
go run ./cmd/notarius run dnd-npcs \
|
||||
--config examples/dnd-npc-combat-sequential.config.yml \
|
||||
--input examples/seriatim-minimal-transcript.json \
|
||||
--output-dir ./npc-output
|
||||
|
||||
go run ./cmd/notarius run dnd-combat \
|
||||
--config examples/dnd-npc-combat-sequential.config.yml \
|
||||
--input examples/seriatim-minimal-transcript.json \
|
||||
--reference combat.extract.npcs=./npc-output/<run-id>/lanes/npcs.json \
|
||||
--reference combat.normalize.npcs=./npc-output/<run-id>/lanes/npcs.json
|
||||
```
|
||||
|
||||
Extraction and normalization bindings are stage-local and are intentionally
|
||||
specified separately. Notarius does not discover the NPC run, copy its source
|
||||
ranges into combat evidence, or compose the two runs into one workflow. The
|
||||
combat manifest records both reference bindings and the prepared registry's
|
||||
semantic digest/count. Changing the referenced NPC payload, prompt or schema,
|
||||
normalization policy, or registry digest makes affected checkpoint state
|
||||
incompatible; output, checkpoint, and debug roots remain independent sensitive
|
||||
state surfaces.
|
||||
Standalone module configurations continue to support external NPC files when a
|
||||
workflow intentionally crosses a process or session boundary. Those files are
|
||||
validated against the consumer slot and must be protected as sensitive
|
||||
campaign data. They are not part of the maintained ordered handoff workflow.
|
||||
|
||||
## Chunk-Plan Cache
|
||||
|
||||
@@ -177,6 +147,72 @@ cache:
|
||||
Remove an exact checkpoint identity directory or the configured root only when
|
||||
recomputation is acceptable.
|
||||
|
||||
### Resume And Selective Recompute
|
||||
|
||||
`--resume` loads compatible accepted work only when checkpoint recording is
|
||||
enabled. A normal resumed run may reuse source, extract, merge, and normalize
|
||||
checkpoints independently and may recompute a stage after a cache miss.
|
||||
Generated references add a dependency fingerprint
|
||||
for the producer's artifact kind, schema identity, media type, canonical
|
||||
content digest, and size. If that fingerprint changes or the producer is
|
||||
missing, dependent checkpoints are invalidated; unrelated work remains eligible
|
||||
for reuse.
|
||||
|
||||
`--recompute-step <step-id>` requires both `--resume` and
|
||||
`cache.checkpoints.enabled: true`. It forces the named step and all transitive
|
||||
dependents to execute, while compatible predecessors and unrelated lanes remain
|
||||
reusable. The ID may be an explicit configured step or `default` for an
|
||||
implicit single-step pipeline. It cannot be combined with `--only`, and it does
|
||||
not change the persistent identity of otherwise identical checkpoints.
|
||||
Decisions are bounded and categorized as `reused`, `executed`,
|
||||
`forced_recompute`, or `dependency_invalidated`.
|
||||
|
||||
For an unselected producer required by a recomputed step, Notarius loads the
|
||||
accepted normalized artifact directly. Valid normalize state is sufficient even
|
||||
when that producer's extract or merge checkpoint is missing or corrupt. The
|
||||
normalize manifest must be successful and match workspace schema v3, the exact
|
||||
current invocation identity, step, lane, and normalizer; its payload digest and
|
||||
canonical codec representation must also validate. A forced producer bypasses
|
||||
this lookup and executes.
|
||||
|
||||
If a required predecessor's accepted normalized artifact is missing, rejected,
|
||||
corrupt, non-canonical, or incompatible, the run fails before the dependent
|
||||
step starts. It does not fall back to rerunning that predecessor. The failure
|
||||
manifest retains completed upstream outcomes and dependency context but not
|
||||
generated reference content. For diagnosis, first check the producer step and
|
||||
lane in the manifest, then inspect checkpoint decision categories and reason
|
||||
codes. Rerun the producer explicitly rather than copying an artifact into the
|
||||
checkpoint root.
|
||||
|
||||
The decision that caused a required-predecessor failure is retained before the
|
||||
run returns, and the CLI error identifies its step, lane, and reason code.
|
||||
|
||||
Checkpoint reason codes are stable diagnostic identifiers:
|
||||
|
||||
| Reason code | Operator meaning |
|
||||
| --- | --- |
|
||||
| `loading_disabled` | This invocation did not enable checkpoint loading. |
|
||||
| `checkpoint_missing` | The requested checkpoint file does not exist. |
|
||||
| `checkpoint_path_invalid` | The requested checkpoint location failed confinement validation. |
|
||||
| `checkpoint_read_failed` | An existing checkpoint could not be read. |
|
||||
| `checkpoint_decode_failed` | Checkpoint JSON could not be decoded. |
|
||||
| `workspace_schema_incompatible` | The stored workspace schema is not supported by this build. |
|
||||
| `identity_mismatch` | The stored invocation identity differs from the current invocation. |
|
||||
| `stage_mismatch`, `step_mismatch`, `lane_mismatch`, `module_mismatch` | Stored scope does not match the requested pipeline scope. |
|
||||
| `status_not_reusable` | The stored operation did not finish in a reusable status. |
|
||||
| `dependency_mismatch` | Stored dependencies differ; the category is `dependency_invalidated`. |
|
||||
| `artifact_payload_invalid` | Stored artifact payload structure or encoding is invalid. |
|
||||
| `artifact_digest_mismatch` | Stored artifact bytes do not match their recorded digest. |
|
||||
| `artifact_codec_incompatible` | Stored artifact identity is incomplete or incompatible with the codec contract. |
|
||||
| `artifact_not_canonical` | The codec can decode the artifact, but its bytes are not canonical. |
|
||||
| `checkpoint_reused` | The stored checkpoint passed validation and was reused. |
|
||||
| `accepted_artifact_reused` | A required producer's accepted normalized artifact was canonically validated and hydrated. |
|
||||
| `recompute_step` | Selective recomputation forced execution of this lane. |
|
||||
|
||||
Decision detail is bounded explanatory text, not a data-recovery channel. It
|
||||
never contains checkpoint paths, artifact or reference content, source content,
|
||||
credentials, or environment values.
|
||||
|
||||
## Debug Bundles
|
||||
|
||||
Only `notarius run --debug` enables debug collection. The selected root contains
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
# D&D Module Harmonization And Prompt Reuse
|
||||
|
||||
This roadmap tracked harmonization of the spell, NPC, and combat-turn lanes.
|
||||
The implemented behavior is now owned by [Module Internals](../internal/modules.md)
|
||||
and [LLM Runtime Internals](../internal/llm.md); this document records status
|
||||
and the one remaining external prerequisite rather than duplicating those
|
||||
current-behavior references.
|
||||
|
||||
## Implementation Status
|
||||
|
||||
The offline/runtime work is complete:
|
||||
|
||||
- shared prompt assets have explicit per-prompt manifests that drive both
|
||||
mounting and prompt fingerprints;
|
||||
- shared prompt inputs, deterministic chunk material preparation, and cited
|
||||
source traversal are centralized;
|
||||
- prompt ordering and cache-control policy are aligned across the D&D
|
||||
extraction prompts, while scene chunking retains its distinct prompt shape;
|
||||
- validator prerequisite, bounded-diagnostic, and checkpoint policies are
|
||||
aligned without changing compatibility-sensitive identifiers;
|
||||
- production registration remains explicit and typed, with grouped composition,
|
||||
central default-chain ownership, and artifact-specific merge behavior;
|
||||
- private LLM response schemas remain package-owned and separate from durable
|
||||
codec schemas; and
|
||||
- focused tests and repository-wide test, vet, build, and diff checks pass.
|
||||
|
||||
## Deferred evaluation
|
||||
|
||||
An optional live before/after provider comparison remains deferred because it
|
||||
requires credentials and a maintained human-reviewed transcript or fixture
|
||||
set. It is not part of the default test suite or merge gate. If those
|
||||
prerequisites become available, record only aggregate extraction-review
|
||||
results, prompt token counts, cache-hit/cache-write metrics, and non-secret
|
||||
prompt hashes. Never commit transcript content, rendered prompts, credentials,
|
||||
endpoints, or private reference material.
|
||||
@@ -5,122 +5,6 @@ configuration, operations, internal, and integration docs. This roadmap records
|
||||
future work only. Items are ordered roughly by current value and specificity,
|
||||
not as committed release dates.
|
||||
|
||||
## Near-Term: Ordered Pipeline Steps
|
||||
|
||||
Allow one configured pipeline to contain multiple ordered execution steps so
|
||||
accepted artifacts from an earlier step can become generated references for
|
||||
later steps in the same run. This is a bounded extension of the fixed pipeline
|
||||
model, not an arbitrary DAG or general workflow language.
|
||||
|
||||
Input parsing and chunk planning remain pipeline-wide. Each step selects one or
|
||||
more artifact lanes; every selected lane completes extraction, validation,
|
||||
merge, normalization, and validation before dependent later steps begin. Lanes
|
||||
within the same step remain independent and may execute concurrently. The
|
||||
runner exposes only accepted normalized artifacts across a step boundary; raw
|
||||
extracts, rejected outputs, and intermediate merge results cannot become
|
||||
downstream references.
|
||||
|
||||
Generated-reference bindings must be explicit in resolved configuration. A
|
||||
binding identifies an earlier producing lane and one declared reference slot on
|
||||
a later consuming lane. Resolution must reject missing producers, references to
|
||||
the same or a later step, incompatible artifact kinds or media types, undeclared
|
||||
consumer slots, cycles, and ambiguous bindings. A configured external reference
|
||||
and a generated reference cannot bind the same effective target slot; reject
|
||||
that pipeline or runtime override instead of applying a precedence rule.
|
||||
Each effective target slot accepts at most one producer. One generated artifact
|
||||
may fan out to multiple compatible target slots in a later step; aggregation
|
||||
from multiple producers into one slot is deferred until a concrete use case
|
||||
defines deterministic semantics.
|
||||
|
||||
Step-scoped reference defaults mirror existing pipeline-level reference
|
||||
defaults. A step binds an earlier artifact once, and the binding automatically
|
||||
applies to every target in that step that declares the named slot. Target-local
|
||||
bindings remain available when only one module should consume the artifact; do
|
||||
not combine a target-local and step-scoped binding for the same effective slot.
|
||||
A generated source uses a structured, unambiguous form rather than encoding a
|
||||
producer into a path-like string. The target configuration shape is:
|
||||
|
||||
```yaml
|
||||
steps:
|
||||
- id: identify-npcs
|
||||
artifacts:
|
||||
npcs:
|
||||
extract: dnd/npcs
|
||||
normalize: dnd/npcs
|
||||
- id: grounded-events
|
||||
references:
|
||||
npcs:
|
||||
artifact:
|
||||
step: identify-npcs
|
||||
lane: npcs
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
normalize: dnd/spells
|
||||
combat:
|
||||
extract: dnd/combat-turns
|
||||
normalize: dnd/combat-turns
|
||||
```
|
||||
|
||||
This snippet shows the step and reference portion of a pipeline; pipeline-wide
|
||||
input, chunk, output, and other references are omitted. Existing scalar
|
||||
reference values continue to mean external file paths; the structured
|
||||
`artifact` form means an accepted normalized artifact from the named earlier
|
||||
step and lane. Do not infer generated bindings from module keys, matching lane
|
||||
names, or D&D-specific knowledge in the framework.
|
||||
|
||||
The handoff should use the producer artifact's canonical codec representation
|
||||
and retain its artifact kind, schema identity, media type, content digest, and
|
||||
producer provenance. Generated references provide context or disambiguation,
|
||||
not source evidence. They use the existing module-facing reference contract
|
||||
where possible; typed or domain-specific adapters may validate and prepare a
|
||||
reference without moving domain concepts into the pipeline framework.
|
||||
|
||||
Pipeline identity, manifests, checkpoint dependencies, debug records, and
|
||||
errors must include step identity and generated-reference provenance. A
|
||||
downstream checkpoint is reusable only when the upstream artifact identity and
|
||||
content digest match. Resume should reconstruct an accepted upstream artifact
|
||||
through its registered codec instead of requiring the producing lane to run
|
||||
again when its checkpoint is reusable.
|
||||
|
||||
### Dependency-aware resume and recomputation
|
||||
|
||||
Treat generated-reference bindings as checkpoint dependencies. Reusing an
|
||||
earlier step is safe only when its existing checkpoint and codec identity are
|
||||
valid. Reusing a dependent step additionally requires an exact match for every
|
||||
upstream artifact kind, schema identity, media type, and canonical content
|
||||
digest it consumed. A changed, missing, rejected, corrupt, or incompatible
|
||||
producer artifact invalidates all transitive dependent checkpoints; the runner
|
||||
must never combine a newly produced upstream artifact with stale downstream
|
||||
output.
|
||||
|
||||
Support selectively recomputing one configured step and all of its transitive
|
||||
dependents while retaining reusable independent and predecessor work. The
|
||||
operator-facing selection mechanism should identify a stable configured step,
|
||||
not individual internal stage operations. Resolution must reject a selection
|
||||
that would omit a required predecessor without a reusable accepted artifact.
|
||||
|
||||
Manifests, checkpoint events, and diagnostics should distinguish work that was
|
||||
executed, reused, or invalidated and record a bounded non-secret reason for
|
||||
dependency-driven invalidation. Completion order must not affect invalidation,
|
||||
public artifact ordering, or the set of dependent steps selected for rerun.
|
||||
|
||||
If a required producer finishes without an accepted normalized artifact, fail
|
||||
the entire run with a deterministic dependency error. Do not start any
|
||||
dependent step. Preserve the upstream rejection or empty-result outcome and
|
||||
step provenance in the failed run manifest so the cause remains auditable.
|
||||
|
||||
The first production workflow is D&D NPC grounding:
|
||||
|
||||
1. the first step runs the NPC lane through accepted normalized output; and
|
||||
2. the second step runs spell and combat-turn lanes, binding that NPC artifact
|
||||
to the spell extractor and to the combat extractor and normalizer through
|
||||
their existing `npcs` reference slots.
|
||||
|
||||
Spell and combat-turn extraction may run concurrently after the NPC handoff is
|
||||
available. The NPC artifact may disambiguate participant identity, but it does
|
||||
not prove that a spell cast or combat turn occurred.
|
||||
|
||||
## Near-Term D&D Pipeline
|
||||
|
||||
### Evaluate Spell Extraction And Normalization
|
||||
|
||||
@@ -1,573 +1,399 @@
|
||||
# D&D Module Harmonization Implementation Plan
|
||||
# Implementation Plan: Ordered Pipeline Follow-Up
|
||||
|
||||
This document is the executable implementation plan for the target state in
|
||||
[D&D Module Harmonization And Prompt Reuse](dnd.md). It is written for a coding
|
||||
agent and must be followed in stage order. Each stage must leave the repository
|
||||
building and its focused tests passing before the next stage begins.
|
||||
## Status
|
||||
|
||||
Current behavior is documented in the
|
||||
[module internals](../internal/modules.md) and
|
||||
[LLM runtime internals](../internal/llm.md). The policies in
|
||||
[Architecture](../policy/architecture.md),
|
||||
[Testing](../policy/testing.md), and
|
||||
[Documentation](../policy/documentation.md) govern all stages.
|
||||
Completed on 2026-07-22. This document retains the implementation sequence for
|
||||
historical context; current contracts are maintained in the canonical CLI,
|
||||
configuration, operations, integration, and internal documentation linked from
|
||||
the [development guide](../development.md).
|
||||
|
||||
## Fixed Decisions And Constraints
|
||||
## Purpose
|
||||
|
||||
The following decisions are complete and are not implementation-time choices:
|
||||
Record the work that closed the correctness, observability, test, and
|
||||
maintainability gaps in the implemented
|
||||
[Ordered Pipeline Steps](ordered-pipeline-steps.md) feature. That feature
|
||||
roadmap remains the authority for product intent, policy choices, acceptance
|
||||
criteria, and exclusions. This document preserves the ordered,
|
||||
decision-complete implementation sequence for the follow-up work.
|
||||
|
||||
- Keep spell, NPC, and combat-turn response DTOs, schemas, canonicalization,
|
||||
codecs, normalizers, and domain validators in their current domain packages.
|
||||
- Put D&D-only reuse in `internal/modules/dnd/shared`; do not move D&D concepts
|
||||
into `internal/framework`.
|
||||
- Keep private LLM response schemas separate from durable codec schemas. Do not
|
||||
introduce shared JSON Schema fragments or a schema-generation step in this
|
||||
work.
|
||||
- Preserve all public module keys, artifact kinds, prompt IDs, schema IDs,
|
||||
schema versions, media types, reference slot names, durable JSON fields, and
|
||||
existing reason-code strings. In particular, retain the spell validator's
|
||||
existing `invalid_source_refs` reason code as a compatibility exception.
|
||||
- Preserve central ownership and ordering of default validator chains in the
|
||||
D&D registrar.
|
||||
- Do not create a generic extractor framework, use reflection for lane
|
||||
registration, or erase typed artifact relationships outside existing
|
||||
framework boundaries.
|
||||
- Exact cache identity means equal ordered message roles, content bytes, and
|
||||
cache-control values after Scriptorium has rendered the prompt. Enforce this
|
||||
primarily through one canonical shared asset per shared message and document
|
||||
the expected order and cache-control policy.
|
||||
- Scriptorium and provider-specific types remain confined to the LLM runtime,
|
||||
prompt-asset wiring, and their tests. Production extractors continue to use
|
||||
only Notarius contracts.
|
||||
- Default tests remain offline, deterministic, and credential-free. Live model
|
||||
evaluation is an explicit manual acceptance activity, not part of
|
||||
`go test ./...`.
|
||||
- Do not add message-count, common-prefix-length, or complete rendered-prompt
|
||||
snapshots. They are change detectors rather than durable behavioral tests.
|
||||
- Update current-behavior documentation only in the stage that changes that
|
||||
behavior. Do not describe a partially implemented later stage as complete.
|
||||
## Background
|
||||
|
||||
## Target Prompt Layout
|
||||
The original implementation is complete in broad architecture. Pipelines now
|
||||
resolve and prepare one canonical ordered-step model, execute hard barriers
|
||||
between steps, hand accepted normalized artifacts to later operations as
|
||||
canonical generated references, include generated identity in checkpoint and
|
||||
output provenance, support `--recompute-step`, and use generated NPC output at
|
||||
operation time in the D&D spell and combat consumers. Current documentation and
|
||||
maintained examples describe that model.
|
||||
|
||||
Stage 3 must produce the following ordered messages. All listed shared entries
|
||||
must refer to one shared embedded file rather than package-local copies.
|
||||
The completed follow-up addressed these narrower gaps:
|
||||
|
||||
| Index | Spell | NPC | Combat turn | Role | Cache control |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 0 | common system | common system | common system | system | none |
|
||||
| 1 | common extraction evidence | common extraction evidence | common extraction evidence | user | none |
|
||||
| 2 | common in-world identity | common in-world identity | common in-world identity | user | `ephemeral` |
|
||||
| 3 | common transcript | common transcript | common transcript | user | `ephemeral` |
|
||||
| 4 | common campaign references | common campaign references | common campaign references | user | `ephemeral` |
|
||||
| 5 | common immediate resolution | NPC task | common immediate resolution | user | none |
|
||||
| 6 | common NPC registry | NPC instructions | common NPC registry | user | `ephemeral` for spell/combat |
|
||||
| 7 | spell catalog | — | combat task | user | none |
|
||||
| 8 | spell task | — | combat instructions | user | none |
|
||||
| 9 | spell instructions | — | — | user | none |
|
||||
- selective recomputation now hydrates an unselected predecessor from its
|
||||
accepted normalized artifact without requiring its extract and merge state;
|
||||
- the failed checkpoint decision is recorded before a required-predecessor
|
||||
error returns;
|
||||
- checkpoint reason codes are assigned explicitly rather than inferred from
|
||||
human-readable prose;
|
||||
- runner lane and checkpoint orchestration has explicit responsibility seams;
|
||||
and
|
||||
- CLI and resumed-producer acceptance coverage exercises the complete recovery
|
||||
contract.
|
||||
|
||||
The common extraction-evidence asset must state, in artifact-neutral language,
|
||||
that:
|
||||
No new product feature is introduced by this plan. Preserve configuration file
|
||||
version 3 and checkpoint workspace schema `notarius.workspace.v3`; the fixes do
|
||||
not require a new persistent format.
|
||||
|
||||
- transcript units are the only event evidence;
|
||||
- campaign and registry references may disambiguate but are not evidence;
|
||||
- every reported factual claim is supported by cited transcript units;
|
||||
- references use integer `start_unit_id` and `end_unit_id` values;
|
||||
- `source_id` is omitted because Notarius assigns the current source identity;
|
||||
- non-contiguous evidence uses multiple narrow ranges rather than a broad
|
||||
bridge over unrelated conversation; and
|
||||
- output contains only the configured JSON object and schema-defined fields.
|
||||
## Instructions For Every Stage
|
||||
|
||||
The common in-world-identity asset must require the most specific supported
|
||||
in-world character or creature identity instead of a human player, transcript
|
||||
speaker, or the GM as an out-of-world person. It may permit campaign references
|
||||
to disambiguate an identity, but must not establish participation from a
|
||||
reference alone. NPC-only exclusion rules and relationship rules remain in the
|
||||
NPC task or instructions.
|
||||
Before changing code, read `docs/development.md`, all documents under
|
||||
`docs/policy/`, and the task-specific internal documents named there. Use the
|
||||
repository's code knowledge graph for code discovery before falling back to
|
||||
text search.
|
||||
|
||||
The common immediate-resolution asset, used only by spells and combat turns,
|
||||
must limit an artifact to a declaration/action and its immediate observed
|
||||
resolution. It must exclude consequences on later turns or elsewhere in the
|
||||
scene. Spell-only persistent-effect language and combat-only classification
|
||||
rules remain local.
|
||||
Implement the stages in order. Each stage must leave the repository formatted,
|
||||
building, and passing its focused tests. Preserve these invariants throughout:
|
||||
|
||||
Remove equivalent prose from package-local task and instruction files after it
|
||||
has moved to a shared asset. Do not retain paraphrased copies. Read each final
|
||||
prompt as a whole to remove contradictions and preserve all artifact-specific
|
||||
requirements.
|
||||
- The pipeline retains one input, chunk plan, output encoder, run identity,
|
||||
checkpoint identity, failure boundary, worker budget, and provider scheduler.
|
||||
- Every selected module and validator is constructed before source parsing.
|
||||
- Steps schedule fixed extract, merge, and normalize lanes; this work must not
|
||||
introduce module-to-module calls or a general DAG scheduler.
|
||||
- Generated artifacts remain cloned operation-time context, never source
|
||||
evidence. Never emit their content, source material, credentials, or local
|
||||
paths in decisions, manifests, logs, or summaries.
|
||||
- Public ordering remains step order, lane ID, and source/chunk order as
|
||||
applicable. Refactoring must not expose completion order.
|
||||
- Tests must follow `docs/policy/testing.md`: protect observable recovery,
|
||||
orchestration, and CLI contracts without asserting private helper calls,
|
||||
exact prose, goroutine choreography, or full-document snapshots.
|
||||
- Update current-behavior documentation in the same stage that changes the
|
||||
corresponding behavior. Keep detailed implementation sequencing only here.
|
||||
|
||||
The cache-control choices above create three all-lane breakpoints and one
|
||||
spell/combat breakpoint. Use no more than these four markers so the request
|
||||
remains portable across the configured backends; do not add module-specific
|
||||
markers in this work.
|
||||
## Target Checkpoint Semantics
|
||||
|
||||
## Stage 1: Protect Shared Input Behavior And Baseline Prompt Wiring
|
||||
Use these definitions consistently in all stages:
|
||||
|
||||
### Goal
|
||||
- A **stage checkpoint** is internal resumable state for extract, merge, or
|
||||
normalize. Ordinary resume may continue to reuse this state progressively.
|
||||
- An **accepted lane artifact** is the one successful normalized artifact for a
|
||||
`(step ID, lane ID, normalizer module)` under the current checkpoint identity.
|
||||
It is the dependency exposed to a later step.
|
||||
- An unselected required predecessor satisfies selective recomputation when its
|
||||
accepted lane artifact can be validated and hydrated. Its extract and merge
|
||||
stage checkpoints are not prerequisites for that handoff.
|
||||
- A selected lane and its transitive dependents execute. They must never use
|
||||
accepted-artifact hydration to bypass forced execution.
|
||||
- If a required predecessor's accepted lane artifact is missing, rejected,
|
||||
corrupt, non-canonical, or incompatible, fail the run before any dependent
|
||||
lane starts. Do not implicitly rerun that predecessor.
|
||||
- Ordinary resume without `--recompute-step` keeps its existing progressive
|
||||
stage-reuse and cold-miss behavior.
|
||||
|
||||
Protect deterministic shared input rendering and confirm the existing prompt
|
||||
assets prepare successfully before changing prompt composition.
|
||||
## Stage 1: Decompose Runner And Checkpoint Orchestration
|
||||
|
||||
### Implementation
|
||||
### Objective
|
||||
|
||||
1. Retain one offline prompt preparation test in each extractor package. Each
|
||||
test should prove registration succeeds, the prompt selects its package-owned
|
||||
response schema, required dynamic inputs render, and no provider call or
|
||||
credentials are required. Do not assert total message count, shared-prefix
|
||||
length, or the complete rendered prompt.
|
||||
2. Extend `internal/modules/dnd/shared/prompt_inputs_test.go` with one
|
||||
table-driven identity test covering:
|
||||
- identical source material and references produce deeply equal common input
|
||||
materials;
|
||||
- reference item insertion order does not change rendered reference bytes;
|
||||
- `roster` fallback produces the canonical `party` material;
|
||||
- an explicit non-empty `party` wins over `roster`; and
|
||||
- missing optional slots render the existing single-space placeholder.
|
||||
3. Do not add a full HTTP/provider test for cross-module prompt equality.
|
||||
Scriptorium owns provider request serialization, and shared asset ownership
|
||||
removes the duplicated content that would otherwise need equality testing.
|
||||
Create explicit, testable ownership seams for ordered-step coordination,
|
||||
per-lane execution, and per-stage checkpoint handling without changing
|
||||
observable behavior.
|
||||
|
||||
### Verification
|
||||
### Changes
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/combatturns
|
||||
```
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Shared input tests protect deterministic serialization and canonical slot
|
||||
handling.
|
||||
- Each extractor's existing behavior-level asset test prepares successfully
|
||||
without pinning message count or prefix boundaries.
|
||||
- No production behavior changes in this stage.
|
||||
|
||||
## Stage 2: Make Prompt Asset Manifests Exact
|
||||
|
||||
### Goal
|
||||
|
||||
Use one explicit asset manifest for prompt mounting and prompt fingerprinting,
|
||||
and stop fingerprinting shared assets a prompt does not render.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Replace the broad `sharedPromptFiles`, `CommonHashParts`, and
|
||||
`ReferenceHashParts` grouping in `internal/modules/dnd/shared/assets.go`
|
||||
with an explicit manifest abstraction:
|
||||
|
||||
```go
|
||||
type PromptAssetManifest struct {
|
||||
ModuleDir string
|
||||
ModuleFiles []promptfs.ModulePromptFile
|
||||
SharedFiles []string
|
||||
}
|
||||
```
|
||||
|
||||
Exact formatting may follow `gofmt`, but retain these fields and meanings.
|
||||
2. Give the manifest two operations:
|
||||
- `PromptFS(moduleFS fs.FS) (fs.FS, error)`, which resolves only the named
|
||||
shared files and delegates composition to `promptfs.ModulePromptFS`; and
|
||||
- `Hash(moduleFS fs.FS) (string, error)`, which hashes the same module and
|
||||
shared files in manifest order through `llm.HashAssets`.
|
||||
3. Keep the shared-name-to-embedded-path mapping private to the shared package.
|
||||
Reject unknown, duplicate, empty, or path-containing shared names. Return
|
||||
fresh slices so callers cannot mutate package state.
|
||||
4. In each `scriptorium_assets.go`, declare one package-local manifest that
|
||||
includes its YAML definition and every Markdown file referenced by that
|
||||
definition. Keep ordering stable within the module and shared lists. Use
|
||||
that manifest for both `RegisterPromptAssets` and
|
||||
`scriptoriumPromptMetadata`.
|
||||
5. Migrate the scene chunker to the same API because it uses the shared prompt
|
||||
helper. Its behavior and prompt ordering remain unchanged in this stage.
|
||||
6. Mount only shared assets actually referenced by each prompt. Before Stage 3:
|
||||
- NPC and scene prompts must not include or hash `common-dnd-npcs.md`;
|
||||
- spell and combat prompts must include and hash it; and
|
||||
- all four prompts must include and hash the shared system, transcript, and
|
||||
campaign-reference files they render.
|
||||
7. Retain the response-schema fingerprint as its existing independent
|
||||
fingerprint. Do not include response schema bytes in the prompt manifest.
|
||||
1. Keep `Runner.Run` as the pipeline-wide coordinator. Extract a small
|
||||
step-coordination helper responsible only for iterating prepared steps,
|
||||
building generated reference sets at each barrier, invoking the lane engine,
|
||||
and merging deterministic outcomes.
|
||||
2. Split `runLanes` in `internal/framework/pipeline/runner_concurrent.go` into
|
||||
helpers with these responsibilities:
|
||||
- initialize lane state and resolve extract checkpoint state in lane order;
|
||||
- run the bounded extract worker/continuation engine;
|
||||
- collect terminal lane results and choose failures deterministically; and
|
||||
- merge lane-local output into step output.
|
||||
Preserve the existing bounded channels, cancellation, drain behavior,
|
||||
chunk-first dispatch, lane ordering, and step barrier.
|
||||
3. Split `continueTypedLane` in
|
||||
`internal/framework/pipeline/runner_typed.go` into stage-specific merge and
|
||||
normalize helpers. Each helper should own dependency construction, checkpoint
|
||||
loading and canonical validation, execution/retry/validation when needed,
|
||||
checkpoint recording, debug envelopes, and its typed result. Use small
|
||||
result structs rather than long parallel return lists.
|
||||
4. Centralize repeated checkpoint-decision flow in one pipeline helper that can
|
||||
apply forced-execution policy, validate canonical stored artifacts, record
|
||||
the final observable decision, and return a contextual error. Do not yet
|
||||
change categories, reason codes, or required-predecessor semantics; Stages 2
|
||||
and 3 will change those deliberately.
|
||||
5. Keep stage-specific code where the data shapes genuinely differ. Do not
|
||||
introduce reflection, a generic stage state machine, or callbacks that hide
|
||||
the fixed extract/merge/normalize lifecycle.
|
||||
|
||||
### Tests
|
||||
|
||||
- Replace broad shared asset tests with table-driven manifest tests for valid
|
||||
composition, exact mounted files, unknown names, duplicate names, invalid
|
||||
names, missing module files, missing shared files, and defensive copying.
|
||||
- Independently construct the expected `llm.AssetHashPart` list in manifest
|
||||
tests and assert that `Hash` equals `llm.HashAssets` over exactly that list.
|
||||
This proves unused shared assets are excluded and listed assets participate
|
||||
without adding mutation hooks for the embedded filesystem.
|
||||
- Keep one package-level registration test per prompt; remove redundant
|
||||
per-file mounting assertions when the shared manifest tests already own that
|
||||
behavior.
|
||||
- Existing runner, checkpoint, barrier, ordering, cancellation, retry, debug,
|
||||
and D&D integration tests must pass without expectation changes except moves
|
||||
required by renamed private test fixtures.
|
||||
- Add no tests for helper boundaries or collaborator call counts. Add a narrow
|
||||
regression assertion only if the refactor exposes an observable behavior not
|
||||
already protected.
|
||||
|
||||
### Verification
|
||||
### Completion Gate
|
||||
|
||||
Run:
|
||||
Run `go test ./internal/framework/pipeline ./internal/framework/checkpoint
|
||||
./internal/modules/integration` and the pipeline race tests. Review the diff to
|
||||
confirm this stage changes structure only: serialized output, checkpoint state,
|
||||
decision values, failure selection, and module invocation behavior must remain
|
||||
unchanged.
|
||||
|
||||
```sh
|
||||
go test ./internal/framework/promptfs ./internal/modules/dnd/shared ./internal/modules/dnd/chunk/scenes ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/combatturns
|
||||
```
|
||||
## Stage 2: Make Checkpoint Decisions Typed And Observable
|
||||
|
||||
### Completion Criteria
|
||||
### Objective
|
||||
|
||||
- Mounting and hashing are driven by the same ordered manifest.
|
||||
- No prompt fingerprints an unused shared asset or omits a rendered asset.
|
||||
- Prepared prompt messages are unchanged from Stage 1.
|
||||
Assign stable decision categories and reason codes explicitly, and retain the
|
||||
decision that causes a required-dependency failure.
|
||||
|
||||
## Stage 3: Factor And Reorder Shared Prompt Messages
|
||||
### Changes
|
||||
|
||||
### Goal
|
||||
|
||||
Implement the target prompt layout defined above using canonical shared assets
|
||||
and maximize the reusable all-lane and spell/combat content.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Add these embedded assets under
|
||||
`internal/modules/dnd/shared/assets/prompts`:
|
||||
- `common-dnd-extraction-evidence.md`;
|
||||
- `common-dnd-identity.md`; and
|
||||
- `common-dnd-immediate-resolution.md`.
|
||||
2. Write their content according to the fixed semantic boundaries in
|
||||
`Target Prompt Layout`. Use no template inputs in the identity or immediate
|
||||
assets. The evidence asset also needs no new input; it refers generically to
|
||||
the transcript and references already presented later.
|
||||
3. Update the three extractor YAML files to use the exact order, roles, and
|
||||
cache-control values in the target table. Do not change prompt IDs, versions,
|
||||
default profiles, input declarations, output schema paths, or repair counts.
|
||||
4. Update each package's prompt manifest from Stage 2 so it lists exactly the
|
||||
new shared dependencies in rendered order.
|
||||
5. Delete duplicated policy prose from local `task.md` and `instructions.md`
|
||||
files while retaining every module-specific rule. Preserve one and only one
|
||||
`Return exactly one JSON object` rule through the common evidence message.
|
||||
6. Keep the spell catalog input and NPC registry input byte generation
|
||||
unchanged. Keep the NPC extractor free of an `npcs` input.
|
||||
7. Update package asset tests only for behavioral wiring: prompt preparation,
|
||||
response schema selection, and required dynamic input rendering. Do not add
|
||||
assertions for total message count, exact shared-prefix length, or the end
|
||||
index of a shared section. The shared manifest tests own canonical asset
|
||||
selection.
|
||||
8. Update `docs/internal/modules.md` and `docs/internal/llm.md` in the same
|
||||
change to describe the implemented common-prefix composition, exact prompt
|
||||
manifest fingerprinting, and cache-control placement. Describe current
|
||||
behavior, not this staged plan.
|
||||
|
||||
### Verification
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/combatturns ./internal/modules/dnd/register
|
||||
go test ./internal/framework/llm
|
||||
```
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Shared rules exist in one embedded file and have no local paraphrased copy.
|
||||
- Every applicable prompt references those shared assets in the documented
|
||||
order and with the documented cache-control policy.
|
||||
- Prompt fingerprints change for the intentional prompt contract change and
|
||||
include every newly rendered asset.
|
||||
- No artifact schema, durable representation, module selection, or reference
|
||||
contract changes.
|
||||
|
||||
## Stage 4: Centralize Chunk Prompt Material
|
||||
|
||||
### Goal
|
||||
|
||||
Remove the three identical `chunkSourceInput` implementations and make common
|
||||
transcript input preparation a single D&D-owned behavior.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Add `internal/modules/dnd/shared/extraction_inputs.go` with:
|
||||
|
||||
```go
|
||||
func ChunkPromptMaterial(req contracts.TypedExtractionRequest) (contracts.LLMInputMaterial, error)
|
||||
```
|
||||
|
||||
2. Preserve the existing behavior exactly: clone `req.SourceInput`; fall back
|
||||
to chunk content and media type when content is empty; require content bytes
|
||||
to equal `req.Chunk.Content`; default the material name to `source`; default
|
||||
media type from the chunk; and populate `SizeBytes` when zero.
|
||||
3. The helper may assume the caller has already checked `req.Chunk != nil`.
|
||||
Return a D&D-shared error without an extractor name. Each extractor wraps it
|
||||
with its existing `extractorErrorf`, retaining module context.
|
||||
4. Replace all three local helpers and remove now-unused `bytes` imports.
|
||||
5. Do not centralize the remaining request checks. Their typed result handling
|
||||
and module-specific errors make the small duplication clearer than a
|
||||
callback- or generic-heavy abstraction.
|
||||
1. In `internal/framework/pipeline/checkpoint.go`, introduce string-backed
|
||||
internal types and constants for checkpoint decision categories and reason
|
||||
codes. Keep the current JSON strings and public artifact fields compatible.
|
||||
Categories remain exactly `executed`, `reused`, `forced_recompute`, and
|
||||
`dependency_invalidated`.
|
||||
2. Replace prose inspection in `internal/framework/checkpoint/loader.go` with an
|
||||
explicit decision constructor accepting category, reason code, and optional
|
||||
detail. Remove every `strings.Contains` classification branch. Assign a code
|
||||
at the validation site using this bounded vocabulary:
|
||||
- `loading_disabled`, `checkpoint_missing`, `checkpoint_path_invalid`,
|
||||
`checkpoint_read_failed`, and `checkpoint_decode_failed`;
|
||||
- `workspace_schema_incompatible` and `identity_mismatch`;
|
||||
- `stage_mismatch`, `step_mismatch`, `lane_mismatch`, `module_mismatch`, and
|
||||
`status_not_reusable`;
|
||||
- `dependency_mismatch` for the `dependency_invalidated` category;
|
||||
- `artifact_payload_invalid`, `artifact_digest_mismatch`,
|
||||
`artifact_codec_incompatible`, and `artifact_not_canonical`; and
|
||||
- `checkpoint_reused` and `accepted_artifact_reused` for successful reuse,
|
||||
and `recompute_step` for forced execution.
|
||||
Retain an existing code not listed here only when a current external or
|
||||
documented contract already depends on it.
|
||||
3. Keep detail human-readable, UTF-8, bounded, and sanitized through one helper.
|
||||
Detail may identify stage, step, lane, module, expected status, or schema
|
||||
version, but must not include artifact/reference content, source content,
|
||||
credentials, environment values, or local paths. Tests must assert codes and
|
||||
safety properties, not exact detail prose.
|
||||
4. Change the centralized runner decision flow from Stage 1 so the final loader
|
||||
or canonical-validation decision is recorded before returning a
|
||||
required-predecessor error. The contextual error must identify the step and
|
||||
lane and include the stable reason code; it must not interpolate unsafe
|
||||
loader detail.
|
||||
5. Propagate typed values without lossy conversion through checkpoint events,
|
||||
run manifests, debug summaries, and CLI diagnostics. Convert to strings only
|
||||
at existing serialized boundaries unless changing an internal field type is
|
||||
simpler and wire-compatible.
|
||||
6. Update the canonical current-behavior owners in the same change:
|
||||
`docs/internal/state.md` owns the internal decision flow, while
|
||||
`docs/operations.md` owns operator diagnosis and any operator-visible reason
|
||||
code table. Link rather than duplicate the table elsewhere.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add one table-driven shared helper test covering fallback, clone isolation,
|
||||
mismatch, default fields, and preservation of explicit metadata.
|
||||
- Remove duplicate extractor tests only when the shared test fully owns the
|
||||
behavior. Retain one extractor-level test per lane proving helper errors are
|
||||
wrapped with that module's context.
|
||||
- Keep existing tests proving all three extractors pass equal common prompt
|
||||
inputs to the LLM contract.
|
||||
- Add a table in `internal/framework/checkpoint` that exercises one
|
||||
representative input per reason-code family and asserts category, code,
|
||||
bounded valid UTF-8 detail, and absence of supplied secret/path sentinels.
|
||||
- Add pipeline behavior tests proving a required-predecessor failure records the
|
||||
underlying missing, corrupt, incompatible, or dependency-invalidated
|
||||
decision before returning.
|
||||
- Retain focused manifest/debug serialization assertions for category and code;
|
||||
do not add exact-detail or full-manifest snapshots.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Run `go test ./internal/framework/checkpoint ./internal/framework/pipeline
|
||||
./internal/core/artifacts ./internal/core/debugbundle ./internal/cli`. Search the
|
||||
production checkpoint package to confirm no decision category or reason code is
|
||||
derived from diagnostic prose.
|
||||
|
||||
## Stage 3: Hydrate Required Predecessors From Accepted Normalized Artifacts
|
||||
|
||||
### Objective
|
||||
|
||||
Make selective recomputation enforce the lane-level accepted-artifact contract:
|
||||
a valid normalized producer artifact is sufficient even when its extract or
|
||||
merge stage cache is unavailable.
|
||||
|
||||
### Changes
|
||||
|
||||
1. Extend the checkpoint loader contract with a dedicated accepted-normalized-
|
||||
artifact lookup. Implement it in the filesystem loader and every no-op or
|
||||
test implementation. The lookup receives step ID, lane ID, and normalizer
|
||||
module key and returns the normalized artifact, its warnings, and an explicit
|
||||
checkpoint decision. It must not require caller-supplied extract or merge
|
||||
dependency fingerprints.
|
||||
2. The filesystem lookup may reuse the existing normalize manifest and payload;
|
||||
do not add a second persistent copy. It is reusable only when all of the
|
||||
following hold:
|
||||
- workspace schema is v3 and the non-empty stored checkpoint identity matches
|
||||
the current invocation identity;
|
||||
- stage, step, lane, normalizer module, and successful status match;
|
||||
- the manifest output digest matches the payload; and
|
||||
- the payload can subsequently be validated through the registered artifact
|
||||
codec.
|
||||
Skipping caller-supplied merge dependencies is safe only because the matched
|
||||
non-empty checkpoint identity already binds the current input, resolved
|
||||
topology and configuration, external references, runtime overrides, LLM
|
||||
profiles, and component semantic fingerprints. Do not weaken or omit that
|
||||
identity check.
|
||||
A loader lacking a verifiable current identity must return an unavailable
|
||||
decision rather than perform accepted-artifact reuse.
|
||||
3. Add a pipeline hydration helper that decodes the returned serialized
|
||||
artifact through the producer's registered codec, re-encodes it canonically,
|
||||
and requires exact artifact kind, schema ID/name/version/digest, media type,
|
||||
content bytes, and content digest. Return a runner-owned clone with producer
|
||||
step, lane, module, and source identity. Any mismatch is an explicit bounded
|
||||
decision and the bytes never reach a consumer.
|
||||
4. Before normal execution of a lane marked in `RequireReusableLanes`, use the
|
||||
accepted-artifact lookup:
|
||||
- on success, mark the lane terminal without invoking extract, merge,
|
||||
normalize, or their validators;
|
||||
- append the accepted normalized output in the normal deterministic location
|
||||
and restore only the warnings stored with that normalized checkpoint;
|
||||
- record one `reused` normalize decision with a stable accepted-output reuse
|
||||
reason code; do not synthesize extract/merge decisions, warnings, or
|
||||
rejections that were not loaded; and
|
||||
- allow the ordinary step barrier and generated handoff code to consume that
|
||||
output exactly as it consumes a freshly executed output.
|
||||
5. On missing, rejected, corrupt, non-canonical, or incompatible accepted state,
|
||||
record the decision and fail the run before the dependent step begins. Do
|
||||
not fall back to stage execution. Forced lanes must bypass this hydration
|
||||
path and execute normally.
|
||||
6. Leave ordinary resume unchanged when no lane is marked
|
||||
`RequireReusableLanes`: it may reuse or recompute extract, merge, and
|
||||
normalize progressively under the existing cold-miss rules.
|
||||
7. Keep generated-reference fingerprints and provenance unchanged. Hydrating a
|
||||
byte-identical producer must yield the same canonical handoff digest and
|
||||
downstream dependency fingerprint as fresh execution.
|
||||
8. Update `docs/internal/pipeline.md`, `docs/internal/state.md`, and
|
||||
`docs/operations.md` in this stage to distinguish progressive stage reuse
|
||||
from accepted normalized-artifact hydration and to document the fail-rather-
|
||||
than-rerun rule for invalid required predecessors.
|
||||
|
||||
### Tests
|
||||
|
||||
- At the pipeline/checkpoint boundary, create a valid producer normalize
|
||||
checkpoint while omitting or corrupting its extract and merge checkpoints.
|
||||
Select a later step for recomputation and prove the producer hydrates, no
|
||||
producer operation or validator runs, and the dependent receives the exact
|
||||
canonical artifact.
|
||||
- Cover missing, rejected-status, corrupt, non-canonical, wrong-codec-identity,
|
||||
and wrong-content-digest normalize state. Assert failure and the recorded
|
||||
stable decision before any consumer invocation.
|
||||
- Prove forced producers execute instead of hydrating, while a reusable
|
||||
predecessor and an unrelated lane remain reusable.
|
||||
- Prove fresh and hydrated producer outputs create identical generated
|
||||
provenance and downstream checkpoint fingerprints without exposing content
|
||||
or paths.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Run `go test ./internal/framework/checkpoint ./internal/framework/pipeline
|
||||
./internal/modules/integration` and the corresponding race tests. Manually
|
||||
inspect one failed test fixture to confirm the accepted artifact remains on
|
||||
disk but its bytes do not appear in the manifest, decision detail, debug
|
||||
summary, or error.
|
||||
|
||||
## Stage 4: Complete Recompute CLI And Recovery Acceptance Coverage
|
||||
|
||||
### Objective
|
||||
|
||||
Exercise the operator-visible recomputation contract through stable boundaries
|
||||
and close the acceptance-test omissions from the original plan.
|
||||
|
||||
### Changes
|
||||
|
||||
1. Add one table-driven CLI contract test for `--recompute-step` covering:
|
||||
- a valid explicit step and the implicit `default` step;
|
||||
- repeated flags and an empty or unknown step ID;
|
||||
- use without `--resume`;
|
||||
- use when checkpoint recording is disabled; and
|
||||
- combination with `--only`.
|
||||
Assert exit classification and stable identifying fragments or reason codes,
|
||||
not complete prose.
|
||||
2. Add one filesystem-backed CLI execution test using deterministic fake
|
||||
modules and a three-step generated dependency chain plus one unrelated lane:
|
||||
- perform a fresh checkpointed run;
|
||||
- remove or corrupt only the required producer's extract and merge state
|
||||
inside `t.TempDir()`, leaving its normalize artifact valid;
|
||||
- resume with the middle step selected;
|
||||
- assert the selected lane and transitive dependents execute, the predecessor
|
||||
hydrates without module calls, the unrelated lane reuses, and output and
|
||||
decision ordering are deterministic; and
|
||||
- invalidate the producer normalize state in a subcase and assert the command
|
||||
fails before dependent execution with the bounded decision preserved.
|
||||
3. Add or extend one generic two-step pipeline test proving handoff succeeds
|
||||
both from fresh producer execution and from accepted normalized-artifact
|
||||
hydration. Keep this at the pipeline boundary if the CLI execution test
|
||||
already proves flag wiring; do not duplicate every CLI case end to end.
|
||||
4. Review existing recomputation policy tests. Retain the pure closure test
|
||||
because it protects transitive selection, but remove or consolidate any new
|
||||
test that merely repeats the CLI or pipeline behavior above.
|
||||
5. Make only production changes revealed as necessary by these contract tests;
|
||||
do not add new flag semantics or broaden the feature roadmap.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Run `go test ./internal/cli ./internal/framework/pipeline
|
||||
./internal/framework/checkpoint`, then run the CLI, pipeline, and checkpoint
|
||||
packages under the race detector. Confirm no test asserts internal loader call
|
||||
counts, exact decision detail, filesystem layout outside a temp workspace, or
|
||||
the exact length of any prompt or prefix.
|
||||
|
||||
## Stage 5: Current Documentation And Release Verification
|
||||
|
||||
### Objective
|
||||
|
||||
Reconcile all canonical documentation after the staged changes and verify the
|
||||
complete feature against policy and roadmap.
|
||||
|
||||
### Changes
|
||||
|
||||
1. Re-read `docs/internal/pipeline.md`, `docs/internal/state.md`,
|
||||
`docs/operations.md`, and `docs/cli.md` against the final code. Correct any
|
||||
stale statements left by Stages 1 through 4 without duplicating their
|
||||
canonical contracts.
|
||||
2. Ensure internal documentation describes the coordinator, lane engine, stage
|
||||
checkpoint flow, and accepted-artifact validation at the responsibility
|
||||
level without listing volatile private helper names.
|
||||
3. Confirm the operator-visible reason-code table has one canonical owner and
|
||||
other documents link to it rather than maintaining parallel copies.
|
||||
4. Remove stale implementation claims exposed by this work and ensure the
|
||||
feature roadmap continues to describe target policy rather than task
|
||||
sequencing.
|
||||
|
||||
### Verification
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/combatturns
|
||||
```
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Only one production implementation prepares chunk prompt material.
|
||||
- Extractor behavior and public errors retain useful module context.
|
||||
- Prompt preparation and shared input behavior tests still pass.
|
||||
|
||||
## Stage 5: Share Cited Source Traversal
|
||||
|
||||
### Goal
|
||||
|
||||
Give all relatedness validators one deterministic implementation for resolving,
|
||||
ordering, and deduplicating cited source units while keeping matching semantics
|
||||
artifact-specific.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Add `internal/modules/dnd/shared/citations.go` with:
|
||||
|
||||
```go
|
||||
func CitedText(doc *source.SourceDocument, refs []source.SourceRef) (string, error)
|
||||
```
|
||||
|
||||
2. Validate every range with `source.ValidateRef`. Resolve ranges against
|
||||
document order, include each covered source unit once even when ranges
|
||||
overlap, and join included unit text with a single newline. Return an error
|
||||
for a nil document or any invalid range. Return `"", nil` for an empty ref
|
||||
slice with a non-nil document. Do not mutate the document or refs.
|
||||
3. Add table-driven tests for nil documents, empty refs, invalid source IDs,
|
||||
unknown/reversed unit IDs, disjoint ranges supplied out of order, adjacent
|
||||
ranges, and overlapping/duplicate ranges. Output must always follow document
|
||||
order.
|
||||
4. Replace spell, NPC, and combat relatedness validators' local cited-text
|
||||
traversal with `shared.CitedText`.
|
||||
5. When shape is invalid or `CitedText` returns an error, relatedness validators
|
||||
approve without relatedness warnings so shape/source-reference validators
|
||||
remain the sole owners of those defects.
|
||||
6. Keep matching local:
|
||||
- spells compare the canonical spell name case-insensitively against combined
|
||||
cited text;
|
||||
- NPCs use `identity.ComparisonKey` for names and aliases; and
|
||||
- combat uses its existing comparison-key actor logic and declaration token
|
||||
heuristic.
|
||||
7. Strengthen matching tests with Unicode/apostrophe variants, multiword names,
|
||||
overlapping ranges, and a short-name substring false-positive case. For
|
||||
names and actors, require token/word-boundary-aware matching rather than raw
|
||||
substring matching. For multiword values, match the consecutive normalized
|
||||
token sequence. Keep the combat declaration rule of at least one normalized
|
||||
token of four or more runes.
|
||||
8. Put shared normalized tokenization and consecutive-token matching in
|
||||
`internal/modules/dnd/shared` only if at least two validators use it after
|
||||
the change. Otherwise leave the matching helper local; do not create a
|
||||
single configurable matching engine.
|
||||
|
||||
### Verification
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/modules/dnd/shared ./internal/modules/dnd/validate/spells/source_relatedness ./internal/modules/dnd/validate/npcs/source_relatedness ./internal/modules/dnd/validate/combatturns/source_relatedness
|
||||
```
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Cited range validation, ordering, overlap handling, and text assembly have
|
||||
one production owner.
|
||||
- Relatedness remains warning-only and ignores invalid prerequisite data.
|
||||
- Artifact-specific matching policies remain understandable in their validator
|
||||
packages.
|
||||
|
||||
## Stage 6: Align Validator Policy And Diagnostics
|
||||
|
||||
### Goal
|
||||
|
||||
Make validator checkpoint identity, prerequisite handling, and diagnostic
|
||||
bounding consistent without changing validator-chain order or durable reason
|
||||
codes.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Add these local policy constants and `CheckpointFingerprintProvider`
|
||||
implementations:
|
||||
- spell shape: `dnd.spells.validator.shape.v1`;
|
||||
- spell source refs: `dnd.spells.validator.source_refs.v1`; and
|
||||
- spell source relatedness: `dnd.spells.validator.source_relatedness.v1`.
|
||||
2. Do not add a separate policy fingerprint to the spell catalog validator; its
|
||||
effective catalog digest remains its existing semantic checkpoint identity.
|
||||
If its non-catalog validation policy changes during this work, add a second
|
||||
`policy` fingerprint rather than replacing `effective_catalog`.
|
||||
3. Change spell source-reference validation to approve when spell shape is
|
||||
invalid, matching NPC and combat prerequisite behavior.
|
||||
4. Change spell source-reference validation to collect all reference issues,
|
||||
truncate individual errors with `shared/diagnostics.Truncate`, and return a
|
||||
bounded aggregate with `shared/diagnostics.Aggregate`. Preserve
|
||||
`invalid_source_refs`.
|
||||
5. Confirm NPC and combat source-reference validators follow the same
|
||||
prerequisite and bounded-aggregate policy; refactor only enough to share
|
||||
obvious local structure. Do not introduce a generic typed validator builder.
|
||||
6. Keep shape validators responsible for malformed artifact fields and source
|
||||
reference validators responsible for document/range validity.
|
||||
7. Add or consolidate package-level tests for policy fingerprints, invalid-shape
|
||||
deferral, aggregation of multiple invalid refs, bounded diagnostics, strict
|
||||
options, registration, and deterministic execution class. Prefer a small
|
||||
table of behavioral expectations in each typed package over a reflection-
|
||||
based cross-package harness.
|
||||
8. Update `docs/internal/modules.md` to describe the aligned prerequisite and
|
||||
checkpoint policy after it is implemented.
|
||||
|
||||
### Verification
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/modules/dnd/validate/...
|
||||
go test ./internal/modules/dnd/register
|
||||
```
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Every deterministic validator policy affecting checkpoint reuse has an
|
||||
explicit semantic fingerprint.
|
||||
- Later validators do not duplicate shape rejection.
|
||||
- All source-reference diagnostic output is bounded.
|
||||
- Existing reason codes and validator-chain order are unchanged.
|
||||
|
||||
## Stage 7: Clarify D&D Registration And Naming
|
||||
|
||||
### Goal
|
||||
|
||||
Make production composition easier to audit without introducing heterogeneous
|
||||
generic descriptors or changing registration behavior.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Keep package `internal/modules/dnd/register`, but split its current concerns
|
||||
into focused files:
|
||||
- `register.go`: public `Register`, registry validation, and ordered execution
|
||||
of named registration functions;
|
||||
- `modules.go`: codecs, chunker, extractors, mergers, normalizers, no-op
|
||||
normalizers, and prompt asset registrations;
|
||||
- `validators.go`: production and generic test validator registrations;
|
||||
- `chains.go`: the six default extract/normalize chain mappings; and
|
||||
- `merge.go`: typed append functions and deep-clone helpers.
|
||||
2. Use small private functions such as `registerModules`,
|
||||
`registerValidators`, `registerPromptAssets`, and
|
||||
`registerDefaultChains`. Keep the existing ordered `registration{name,
|
||||
register}` error-context pattern within each group.
|
||||
3. Do not create one slice containing generic lane descriptors; Go cannot retain
|
||||
the heterogeneous typed codec and module relationships there without
|
||||
erasure or callbacks that obscure more than they clarify.
|
||||
4. Keep append and clone behavior in the `register` package for this change.
|
||||
Moving it would require a new lane-ownership package with no independent
|
||||
domain responsibility.
|
||||
5. Normalize import aliases in the registrar to the pattern
|
||||
`spellextract`, `npcextract`, `combatextract`, `spellnormalize`,
|
||||
`npcnormalize`, and `combatnormalize`, with corresponding validator aliases.
|
||||
This is internal naming only.
|
||||
6. Preserve registration order, error prefixes, default chain contents and
|
||||
order, reference-slot/spec behavior, and prompt asset collection.
|
||||
7. Update `register_test.go` only as required by file movement. Tests should
|
||||
continue asserting observable registry contents and chain policy, not the
|
||||
new private helper call graph.
|
||||
|
||||
### Verification
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/modules/dnd/register
|
||||
go test ./internal/modules/dnd/...
|
||||
```
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Production composition is grouped by responsibility and remains explicit.
|
||||
- The refactor produces no registry, chain, capability, or error behavior
|
||||
change.
|
||||
- Tests do not couple to private registration helpers.
|
||||
|
||||
## Stage 8: Final Integration, Documentation, And Evaluation
|
||||
|
||||
### Goal
|
||||
|
||||
Verify the complete target state, update canonical current-behavior documents,
|
||||
and gather quality/cache evidence without making live services part of the
|
||||
default test suite.
|
||||
|
||||
### Implementation And Review
|
||||
|
||||
1. Re-read `docs/roadmap/dnd.md` and verify every completion criterion against
|
||||
production code and tests. Do not mark an item complete based only on this
|
||||
implementation plan.
|
||||
2. Review naming across the three lanes. Harmonize internal aliases and private
|
||||
policy constant names, but do not rename compatibility-sensitive identifiers
|
||||
listed in `Fixed Decisions And Constraints`.
|
||||
3. Review prompt and durable schemas for accidental duplication. Make no schema
|
||||
refactor unless composition already exists and the change is behavior-free;
|
||||
the fixed decision for this plan is to leave them package-owned and separate.
|
||||
4. Consolidate redundant tests created by intermediate stages. Retain:
|
||||
- one shared manifest test suite;
|
||||
- behavior-level prompt preparation tests without prefix-length snapshots;
|
||||
- focused package tests for artifact-specific prompts and validators; and
|
||||
- existing production registration contract coverage.
|
||||
5. Update `docs/internal/modules.md` and `docs/internal/llm.md` so they are the
|
||||
canonical description of the final implemented behavior. Remove superseded
|
||||
implementation details rather than appending a second description.
|
||||
6. Update `docs/roadmap/dnd.md` to record implementation status. Remove completed
|
||||
future-work details that are fully owned by current internal documentation,
|
||||
leaving only genuinely deferred outcomes. Do not turn the feature roadmap
|
||||
into a second current-behavior reference.
|
||||
7. If credentials and the maintained human-reviewed transcript set are
|
||||
available, run an explicitly opt-in comparison using identical profiles and
|
||||
inputs before and after the prompt change. Record only aggregate extraction
|
||||
review results, prompt token counts, cached-token/cache-write metrics, and
|
||||
non-secret prompt hashes. Never commit transcript content, rendered prompts,
|
||||
credentials, endpoints, or private reference material.
|
||||
8. Live evaluation is not a merge gate when credentials or reviewed fixtures
|
||||
are unavailable. In that case, record the missing external prerequisite in
|
||||
the remaining roadmap item; do not add a fake cache-hit claim and do not
|
||||
weaken offline identity tests.
|
||||
|
||||
### Repository Verification
|
||||
|
||||
Run all required checks:
|
||||
Run the repository-prescribed commands from `docs/development.md`:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Also inspect the final diff for:
|
||||
Also run race tests for the pipeline, checkpoint, CLI state, D&D NPC registry,
|
||||
and D&D integration packages. Validate maintained examples/configurations
|
||||
through the production catalog. Review one fresh run, ordinary resumed run,
|
||||
forced-recompute run, hydrated-predecessor run, and invalid-predecessor failure
|
||||
for deterministic ordering, correct decisions, bounded provenance, and absence
|
||||
of generated content, secrets, or local paths in manifests and diagnostics.
|
||||
|
||||
- unintended public identifier or durable schema changes;
|
||||
- prompt rules duplicated between shared and local assets;
|
||||
- prompt assets rendered but absent from fingerprints, or fingerprinted but
|
||||
not rendered;
|
||||
- provider-specific types outside allowed boundaries;
|
||||
- tests containing real transcript/reference material or credentials; and
|
||||
- unrelated changes in a pre-existing dirty worktree.
|
||||
### Completion Gate
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Every completion criterion in the feature roadmap is either implemented and
|
||||
documented in its canonical current-behavior owner or explicitly retained as
|
||||
deferred roadmap work.
|
||||
- All focused and repository-wide checks pass.
|
||||
- The final test suite protects behavioral contracts without retaining
|
||||
redundant implementation snapshots.
|
||||
The follow-up is complete only when every finding summarized above is protected
|
||||
by a stable behavioral test, the current documentation matches the corrected
|
||||
implementation, the full verification suite passes, and the worktree contains
|
||||
no temporary adapters or TODOs introduced by these stages.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The plan fixes all choices required for implementation. Live provider
|
||||
evaluation may depend on credentials and reviewed fixtures, but that is an
|
||||
external acceptance prerequisite rather than an unresolved design decision.
|
||||
None. The feature roadmap and this plan fix the required product and
|
||||
architecture choices. If implementation reveals that accepted normalized
|
||||
artifacts cannot be validated safely without a persistent format change, stop
|
||||
and amend this plan rather than weakening identity, codec, or content
|
||||
validation.
|
||||
|
||||
339
docs/roadmap/ordered-pipeline-steps.md
Normal file
339
docs/roadmap/ordered-pipeline-steps.md
Normal file
@@ -0,0 +1,339 @@
|
||||
# Scope: Ordered Pipeline Steps
|
||||
|
||||
## Status
|
||||
|
||||
Implemented. This document preserves the bounded feature policy, architecture
|
||||
choices, acceptance criteria, and exclusions. Current behavior belongs in the
|
||||
canonical [CLI](../cli.md), [Configuration](../config.md),
|
||||
[Operations](../operations.md), and [internal pipeline](../internal/pipeline.md)
|
||||
documentation rather than in this roadmap.
|
||||
|
||||
## Policy Recommendation
|
||||
|
||||
Treat ordered pipeline steps, generated artifact references, and
|
||||
dependency-aware checkpoint reuse as one coherent platform capability. The D&D
|
||||
proving workflow produces accepted normalized NPC output first and then supplies
|
||||
it to spell extraction, combat-turn extraction, and combat-turn normalization.
|
||||
|
||||
## Intended Outcome
|
||||
|
||||
A configured pipeline may contain multiple ordered steps while retaining one
|
||||
pipeline-wide input, chunk plan, output, worker budget, LLM scheduler, run
|
||||
manifest, and failure boundary. Every artifact lane still follows the fixed
|
||||
extract, validate, merge, validate, normalize, and validate lifecycle. Steps
|
||||
add explicit barriers between groups of lanes; they do not create arbitrary
|
||||
stage graphs.
|
||||
|
||||
An accepted normalized artifact from an earlier step may be bound explicitly
|
||||
to declared reference slots in a later step. The framework remains
|
||||
domain-neutral, and generated references remain contextual material rather than
|
||||
source evidence.
|
||||
|
||||
## Fixed Product And Architecture Decisions
|
||||
|
||||
### Pipeline shape
|
||||
|
||||
- Input parsing and chunk planning remain pipeline-wide and execute once.
|
||||
- A step contains one or more artifact lanes. Step order is configuration order.
|
||||
- Lanes within a step remain independent and may use the existing bounded
|
||||
concurrency model.
|
||||
- A later step cannot begin until every lane in the current step is terminal
|
||||
and every generated artifact it requires is accepted and available.
|
||||
- Public artifact and failure ordering is step order followed by deterministic
|
||||
lane and source-chunk order, never completion order.
|
||||
- Output encoding occurs once, after every step succeeds.
|
||||
- This is not an arbitrary DAG, a general workflow language, concurrent
|
||||
cross-lane reconciliation, or permission for modules to invoke other modules.
|
||||
|
||||
### Configuration model
|
||||
|
||||
Existing single-step pipelines remain valid. A top-level `artifacts` map is
|
||||
treated as an implicit step with stable ID `default`. A pipeline may configure
|
||||
either `artifacts` or `steps`, but not both. Explicit steps must be non-empty
|
||||
and have unique, trimmed, non-empty IDs. Artifact lane IDs must remain unique
|
||||
across the entire pipeline so output paths, selectors, manifests, errors, and
|
||||
checkpoint scopes remain unambiguous.
|
||||
|
||||
The target configuration shape is:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
dnd-session:
|
||||
input: seriatim
|
||||
chunk: generic
|
||||
steps:
|
||||
- id: identify-npcs
|
||||
artifacts:
|
||||
npcs:
|
||||
extract: dnd/npcs
|
||||
normalize: dnd/npcs
|
||||
- id: grounded-events
|
||||
references:
|
||||
npcs:
|
||||
artifact:
|
||||
step: identify-npcs
|
||||
lane: npcs
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
normalize: dnd/spells
|
||||
combat:
|
||||
extract: dnd/combat-turns
|
||||
normalize: dnd/combat-turns
|
||||
output: json
|
||||
```
|
||||
|
||||
Existing scalar reference values continue to represent external file paths.
|
||||
The structured `artifact` form identifies accepted normalized output from one
|
||||
earlier step and lane. Generated artifact bindings are allowed at step scope or
|
||||
at an individual module target; they are not inferred from module keys, lane
|
||||
names, slot names, or domain knowledge.
|
||||
|
||||
A step-scoped reference applies automatically to every selected target in that
|
||||
step that declares the slot. In the example, one `npcs` binding reaches the
|
||||
spell extractor plus the combat extractor and normalizer. A target-local
|
||||
binding is used when only one module should consume the artifact.
|
||||
|
||||
Pipeline-level external references remain defaults. Step-local external
|
||||
references override pipeline-level external defaults, and target-local
|
||||
external references retain their existing precedence. A generated reference
|
||||
and an external reference may not resolve to the same effective target slot;
|
||||
configuration or a runtime override that creates that conflict is invalid.
|
||||
Likewise, a step-scoped and target-local generated binding cannot both target
|
||||
the same effective slot.
|
||||
|
||||
Each effective target slot accepts at most one producer. One producer may fan
|
||||
out to multiple compatible slots in a later step. Aggregating several producer
|
||||
artifacts into one slot is outside this scope.
|
||||
|
||||
Reference-slot specs gain optional generated-artifact compatibility metadata.
|
||||
A generated binding is allowed only when the consumer slot declares the
|
||||
producer's artifact kind; the producer's registered codec supplies the exact
|
||||
schema identity and media type used for the handoff. The D&D `npcs` consumer
|
||||
slots declare the normalized NPC-list artifact kind. Existing external-file
|
||||
slots and bindings retain their current behavior and do not acquire an artifact
|
||||
kind merely because their bytes happen to decode as one.
|
||||
|
||||
### Resolution and preparation
|
||||
|
||||
Resolution validates the complete ordered structure before source processing.
|
||||
It must reject duplicate identities, missing producers, same-step or forward
|
||||
references, undeclared slots, reference conflicts, and incompatible artifact
|
||||
kind, schema, media type, or cardinality constraints that are statically
|
||||
discoverable. Size is checked when canonical producer bytes exist at handoff.
|
||||
Ordered steps make cycles structurally impossible; resolution must not
|
||||
introduce a general graph scheduler to rediscover their order.
|
||||
|
||||
The resolved pipeline and its digest include step order, step IDs, lane
|
||||
membership, generated-reference topology, producer identity, consumer targets,
|
||||
and existing module and validator policy. Cloning, redaction, canonical JSON,
|
||||
debug summaries, and manifests preserve the same structure without reference
|
||||
content or secrets.
|
||||
|
||||
All modules and validators are still selected, option-validated, and
|
||||
constructed before source parsing. Generated content cannot be supplied during
|
||||
construction because it does not exist yet. The framework therefore augments
|
||||
the existing operation-request `References` at the step boundary. Consumers
|
||||
that currently assume an NPC registry is construction-only must accept the
|
||||
generated registry from their operation request without deferring general
|
||||
module construction until after upstream work.
|
||||
|
||||
Only validation that inherently depends on generated bytes may occur at the
|
||||
handoff. A handoff validation failure is a contextual framework error and fails
|
||||
the run before any consumer in that step begins.
|
||||
|
||||
### Generated artifact handoff
|
||||
|
||||
Only accepted normalized output may cross a step boundary. Raw extraction
|
||||
responses, rejected artifacts, merge intermediates, and validator diagnostics
|
||||
cannot be bound as references.
|
||||
|
||||
The framework serializes the producer through its registered canonical artifact
|
||||
codec and constructs one immutable reference item containing:
|
||||
|
||||
- the declared target slot;
|
||||
- canonical artifact bytes and media type;
|
||||
- artifact kind and schema ID, name, version, and schema digest;
|
||||
- canonical content digest and size; and
|
||||
- producer pipeline, step, lane, and module provenance.
|
||||
|
||||
A generated binding requires exactly one accepted normalized artifact from its
|
||||
producer lane. No artifact is a missing dependency, while more than one is a
|
||||
cardinality error; a typed collection such as an NPC list remains one artifact.
|
||||
Combining several normalized outputs into one reference is aggregation and is
|
||||
outside this scope.
|
||||
|
||||
The existing slot contract remains authoritative for accepted media types,
|
||||
maximum size, and cardinality. Generated content is cloned at ownership
|
||||
boundaries and never exposed through a filesystem path. Manifests and debug
|
||||
summaries record identities and bounded provenance, not artifact content.
|
||||
|
||||
Configuring a generated binding makes that dependency required even when the
|
||||
consumer module declares the underlying slot optional. An accepted artifact
|
||||
whose domain collection is empty is still a valid artifact and may be handed
|
||||
off. If the producer has no accepted normalized artifact, the entire run fails
|
||||
with a deterministic dependency error and no later step begins.
|
||||
|
||||
### Checkpoint reuse and selective recomputation
|
||||
|
||||
Generated references participate in downstream checkpoint dependencies by
|
||||
artifact kind, complete schema identity, media type, and canonical content
|
||||
digest. The pipeline digest protects topology; stage dependency fingerprints
|
||||
protect the exact upstream artifact consumed. The runner must never combine a
|
||||
new or changed producer with stale dependent output.
|
||||
|
||||
Ordinary resume may progressively decode compatible producer and consumer stage
|
||||
checkpoints through the registered codec. Selective recomputation may hydrate a
|
||||
required unselected producer directly from its accepted normalized artifact;
|
||||
its extract and merge state are not prerequisites. Missing, rejected, corrupt,
|
||||
incompatible, or changed accepted state stops the run before dependent
|
||||
execution rather than implicitly rerunning the producer. Independent work
|
||||
remains reusable.
|
||||
|
||||
The operator control `--recompute-step <step-id>` has these semantics:
|
||||
|
||||
- it requires checkpoint recording and `--resume`;
|
||||
- the selected step and all transitive dependents execute rather than reuse
|
||||
their checkpoints;
|
||||
- valid required predecessors and unrelated work remain reusable;
|
||||
- the recompute selection affects loader decisions, not the persistent
|
||||
checkpoint identity of otherwise identical work; and
|
||||
- the command fails before dependent execution if a required predecessor has
|
||||
no reusable accepted artifact.
|
||||
|
||||
Existing `--only` behavior remains unchanged for implicit single-step
|
||||
pipelines. Combining `--only` with explicit multi-step pipelines is outside
|
||||
this scope and should be rejected with actionable guidance rather than given
|
||||
implicit dependency-expansion semantics.
|
||||
|
||||
Checkpoint events, manifests, and diagnostics distinguish executed, reused,
|
||||
forced-recomputed, and dependency-invalidated work. Invalidation reasons are
|
||||
bounded, deterministic, and free of reference content, local paths, or secrets.
|
||||
Old checkpoint state need not be migrated; it must produce a safe, explicit
|
||||
cold miss rather than an error or unsafe reuse.
|
||||
|
||||
### Failure, cancellation, and concurrency
|
||||
|
||||
The existing run-wide worker and provider-call limits apply across every step.
|
||||
Workers may be reused between steps, but concurrency cannot cross a step
|
||||
barrier. A framework error cancels started work using the existing bounded
|
||||
drain behavior and prevents later steps and output encoding. Rejections remain
|
||||
recorded outcomes, but failure to produce a normalized artifact required by a
|
||||
generated binding escalates to the run-level dependency error described above.
|
||||
|
||||
The failed manifest retains completed upstream outcomes, step and lane
|
||||
provenance, rejections, checkpoint events, and the dependency failure without
|
||||
embedding generated artifact content.
|
||||
|
||||
## D&D Proving Workflow
|
||||
|
||||
The production acceptance workflow has two explicit steps:
|
||||
|
||||
1. `identify-npcs` runs the NPC lane through normalization and its complete
|
||||
validator policy.
|
||||
2. `grounded-events` receives the canonical NPC artifact in its step-scoped
|
||||
`npcs` reference and runs spell and combat-turn lanes. The binding reaches
|
||||
spell extraction, combat-turn extraction, and combat-turn normalization.
|
||||
|
||||
Spell and combat-turn lanes may execute concurrently after the handoff. NPC
|
||||
content may ground names and identities but cannot establish a spell cast or
|
||||
combat event; source units remain the only event evidence.
|
||||
|
||||
The maintained workflow uses one ordered-pipeline example instead of a manual
|
||||
two-run NPC-to-spell or NPC-to-combat handoff. Existing module keys, artifact
|
||||
contracts, reference slot names, prompt IDs, and D&D evidence policy remain
|
||||
unchanged.
|
||||
|
||||
## Included Work
|
||||
|
||||
- Configuration parsing, validation, cloning, defaults, redaction, and
|
||||
documentation for explicit steps and structured generated references.
|
||||
- Domain-neutral resolved step, dependency, producer, and consumer identities.
|
||||
- Generated-artifact compatibility metadata on reference-slot contracts,
|
||||
including D&D NPC-list declarations for every `npcs` consumer.
|
||||
- Step-aware preparation metadata and runner orchestration.
|
||||
- Canonical codec handoff into existing reference request contracts.
|
||||
- Required-dependency failure and bounded provenance behavior.
|
||||
- Dependency-aware checkpoint reuse, invalidation, events, and selective step
|
||||
recomputation.
|
||||
- D&D NPC-first production composition for spell and combat-turn consumers.
|
||||
- Refactoring the affected D&D consumers so generated NPC references are
|
||||
available at operation time while retaining early static construction.
|
||||
- Maintained examples and current architecture, configuration, CLI, operations,
|
||||
internal, integration, and testing documentation.
|
||||
- An ADR recording the bounded ordered-step extension to the fixed pipeline
|
||||
architecture and its explicit rejection of a general DAG.
|
||||
|
||||
## Explicitly Out Of Scope
|
||||
|
||||
- D&D item extraction or any other new artifact lane.
|
||||
- Cross-artifact NPC ID fields or artifact-schema migration machinery.
|
||||
- Arbitrary DAGs, conditional branches, loops, joins, dynamic step creation, or
|
||||
module-controlled scheduling.
|
||||
- Multiple source inputs, per-step input adapters, per-step chunk plans, or
|
||||
per-step output encoders.
|
||||
- Aggregating multiple generated artifacts into one reference slot.
|
||||
- Optional or best-effort generated dependencies; a configured dependency is
|
||||
required in this scope.
|
||||
- Prior-run or cross-pipeline generated references.
|
||||
- `--only` dependency closure for explicit multi-step pipelines.
|
||||
- Cross-lane reconciliation or domain concepts in the generic framework.
|
||||
- Live-provider tests or model-quality changes to D&D prompts.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
The scope is complete when:
|
||||
|
||||
- all existing single-step configurations retain their current behavior;
|
||||
- explicit step order and dependency topology resolve deterministically and
|
||||
affect pipeline identity;
|
||||
- invalid producer, consumer, conflict, ordering, type, schema, media, and
|
||||
cardinality configurations fail before source processing when statically
|
||||
discoverable, while content-size violations fail at handoff;
|
||||
- no consumer step begins before all required generated artifacts are accepted,
|
||||
canonicalized, and validated for its target slots;
|
||||
- one producer artifact fans out safely to every compatible target selected by
|
||||
a step-scoped binding;
|
||||
- missing required producer output fails the complete run before dependent work;
|
||||
- changing NPC output invalidates spell and combat-turn checkpoints while
|
||||
leaving compatible independent work reusable;
|
||||
- selective step recomputation executes exactly the selected dependency closure
|
||||
and reports why work was executed, reused, or invalidated;
|
||||
- the D&D ordered workflow supplies NPC content to spell extraction, combat-turn
|
||||
extraction, and combat-turn normalization without treating it as evidence;
|
||||
- completion timing cannot change public ordering, failure selection, or
|
||||
dependency behavior;
|
||||
- output, manifests, checkpoints, and debug artifacts contain the required
|
||||
identities and provenance without leaking generated reference content; and
|
||||
- repository-wide tests, vet, build, maintained-example checks, and
|
||||
documentation validation pass.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Tests should protect behavior and invariants rather than the implementation's
|
||||
internal scheduler shape.
|
||||
|
||||
- Configuration contract tests own legacy shorthand, explicit step parsing,
|
||||
source-form discrimination, conflicts, and redaction.
|
||||
- Resolution tests own ordering, global lane uniqueness, dependency validation,
|
||||
slot compatibility, fan-out, cloning, canonical JSON, and digest changes.
|
||||
- Runner tests own step barriers, within-step bounded concurrency, stable
|
||||
ordering, cancellation, required-producer failure, and immutable handoff.
|
||||
- Checkpoint tests own producer decoding, exact dependency matching, transitive
|
||||
invalidation, forced recomputation, cold misses, and bounded decisions.
|
||||
- One CLI contract test should cover the recompute control and its invalid
|
||||
combinations.
|
||||
- One D&D integration test with offline fake LLM responses should prove the
|
||||
complete NPC-to-spell-and-combat handoff, including combat normalization.
|
||||
- Maintained configuration examples should be parsed and resolved through the
|
||||
production catalog.
|
||||
|
||||
Do not add scheduler choreography tests, exact goroutine-count assertions,
|
||||
complete manifest snapshots, exact diagnostic strings, or duplicated tests for
|
||||
every invalid configuration at every layer. No test may require credentials or
|
||||
a live model provider.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None required to define this scope. Exact internal type names and implementation
|
||||
decomposition are intentionally not feature-policy decisions.
|
||||
@@ -1,31 +0,0 @@
|
||||
version: 3
|
||||
output:
|
||||
directory: ./notarius-output
|
||||
cache:
|
||||
chunk_plans:
|
||||
mode: bypass
|
||||
directory: ./notarius-cache/chunk-plans
|
||||
checkpoints:
|
||||
enabled: false
|
||||
directory: ./notarius-cache/checkpoints
|
||||
debug:
|
||||
directory: ./notarius-debug
|
||||
pipelines:
|
||||
dnd-npcs:
|
||||
input: seriatim
|
||||
chunk: generic
|
||||
artifacts:
|
||||
npcs:
|
||||
extract:
|
||||
module: dnd/npcs
|
||||
retries: 2
|
||||
normalize: dnd/npcs
|
||||
dnd-combat:
|
||||
input: seriatim
|
||||
chunk: generic
|
||||
artifacts:
|
||||
combat:
|
||||
extract:
|
||||
module: dnd/combat-turns
|
||||
retries: 2
|
||||
normalize: dnd/combat-turns
|
||||
37
examples/dnd-npc-grounded.config.yml
Normal file
37
examples/dnd-npc-grounded.config.yml
Normal file
@@ -0,0 +1,37 @@
|
||||
version: 3
|
||||
output:
|
||||
directory: ./notarius-output
|
||||
cache:
|
||||
chunk_plans:
|
||||
mode: bypass
|
||||
checkpoints:
|
||||
enabled: false
|
||||
directory: ""
|
||||
debug:
|
||||
directory: ./notarius-debug
|
||||
pipelines:
|
||||
dnd-npc-grounded:
|
||||
input: seriatim
|
||||
steps:
|
||||
- id: identify-npcs
|
||||
artifacts:
|
||||
npcs:
|
||||
extract:
|
||||
module: dnd/npcs
|
||||
retries: 2
|
||||
normalize: dnd/npcs
|
||||
- id: grounded-events
|
||||
references:
|
||||
npcs:
|
||||
artifact:
|
||||
step: identify-npcs
|
||||
lane: npcs
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
normalize: dnd/spells
|
||||
combat:
|
||||
extract:
|
||||
module: dnd/combat-turns
|
||||
retries: 2
|
||||
normalize: dnd/combat-turns
|
||||
@@ -1,28 +0,0 @@
|
||||
version: 3
|
||||
output:
|
||||
directory: ./notarius-output
|
||||
cache:
|
||||
chunk_plans:
|
||||
mode: bypass
|
||||
checkpoints:
|
||||
enabled: false
|
||||
directory: ""
|
||||
debug:
|
||||
directory: ./notarius-debug
|
||||
pipelines:
|
||||
dnd-npcs:
|
||||
input: seriatim
|
||||
chunk: generic
|
||||
artifacts:
|
||||
npcs:
|
||||
extract:
|
||||
module: dnd/npcs
|
||||
retries: 2
|
||||
normalize: dnd/npcs
|
||||
dnd-spells:
|
||||
input: seriatim
|
||||
chunk: generic
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
normalize: dnd/spells
|
||||
@@ -24,10 +24,10 @@ func TestProductionCombatConfigurationResolvesTypedLane(t *testing.T) {
|
||||
if effective.ResolvedPipeline.Chunk.Module != pipeline.DefaultChunkModule {
|
||||
t.Fatalf("chunk module = %q, want %q", effective.ResolvedPipeline.Chunk.Module, pipeline.DefaultChunkModule)
|
||||
}
|
||||
if len(effective.ResolvedPipeline.ArtifactLanes) != 1 {
|
||||
t.Fatalf("artifact lanes = %#v, want one combat lane", effective.ResolvedPipeline.ArtifactLanes)
|
||||
if len(effective.ResolvedPipeline.Steps[0].ArtifactLanes) != 1 {
|
||||
t.Fatalf("artifact lanes = %#v, want one combat lane", effective.ResolvedPipeline.Steps[0].ArtifactLanes)
|
||||
}
|
||||
lane := effective.ResolvedPipeline.ArtifactLanes[0]
|
||||
lane := effective.ResolvedPipeline.Steps[0].ArtifactLanes[0]
|
||||
if lane.ID != "combat" || lane.ArtifactKind != dnd.CombatTurnListKind || lane.Extract.Module != combatextract.Key || lane.Extract.Retries != 2 || lane.Merge.Module != pipeline.DefaultMergeModule || lane.Normalize.Module != combatnormalize.Key {
|
||||
t.Fatalf("resolved combat lane = %#v, want typed production composition", lane)
|
||||
}
|
||||
@@ -89,7 +89,7 @@ func TestProductionCombatConfigurationResolvesTypedLane(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve(bound references) error = %v, want nil", err)
|
||||
}
|
||||
boundLane := bound.ResolvedPipeline.ArtifactLanes[0]
|
||||
boundLane := bound.ResolvedPipeline.Steps[0].ArtifactLanes[0]
|
||||
if len(boundLane.ExtractReferences.Bindings) != 1 || len(boundLane.NormalizeReferences.Bindings) != 1 || boundLane.ExtractReferences.Bindings[0].SlotName != "npcs" || boundLane.NormalizeReferences.Bindings[0].SlotName != "npcs" {
|
||||
t.Fatalf("bound combat references = %#v / %#v, want one independent NPC binding per stage", boundLane.ExtractReferences, boundLane.NormalizeReferences)
|
||||
}
|
||||
|
||||
@@ -24,10 +24,10 @@ func TestProductionNPCConfigurationResolvesTypedLane(t *testing.T) {
|
||||
if effective.ResolvedPipeline.Chunk.Module != pipeline.DefaultChunkModule {
|
||||
t.Fatalf("chunk module = %q, want %q", effective.ResolvedPipeline.Chunk.Module, pipeline.DefaultChunkModule)
|
||||
}
|
||||
if len(effective.ResolvedPipeline.ArtifactLanes) != 1 {
|
||||
t.Fatalf("artifact lanes = %#v, want one NPC lane", effective.ResolvedPipeline.ArtifactLanes)
|
||||
if len(effective.ResolvedPipeline.Steps[0].ArtifactLanes) != 1 {
|
||||
t.Fatalf("artifact lanes = %#v, want one NPC lane", effective.ResolvedPipeline.Steps[0].ArtifactLanes)
|
||||
}
|
||||
lane := effective.ResolvedPipeline.ArtifactLanes[0]
|
||||
lane := effective.ResolvedPipeline.Steps[0].ArtifactLanes[0]
|
||||
if lane.ID != "npcs" || lane.ArtifactKind != dnd.NPCListKind || lane.Extract.Module != npcextract.Key || lane.Extract.Retries != 2 || lane.Merge.Module != pipeline.DefaultMergeModule || lane.Normalize.Module != npcnormalize.Key {
|
||||
t.Fatalf("resolved NPC lane = %#v, want typed production composition", lane)
|
||||
}
|
||||
@@ -101,11 +101,11 @@ func TestProductionNPCConfigurationValidatesOptionsReferencesAndPlacement(t *tes
|
||||
t.Fatalf("unknown normalizer option error = %v, want strict option rejection", err)
|
||||
}
|
||||
if err := resolve(func(profile *pipeline.PipelineProfile) {
|
||||
profile.References = map[string]string{
|
||||
profile.References = pipeline.ExternalReferenceMap(map[string]string{
|
||||
"players": "players.txt",
|
||||
"party": "party.txt",
|
||||
"glossary": "glossary.txt",
|
||||
}
|
||||
})
|
||||
}); err != nil {
|
||||
t.Fatalf("optional NPC references error = %v, want resolution success", err)
|
||||
}
|
||||
|
||||
@@ -33,10 +33,10 @@ func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
|
||||
t.Fatalf("materialize maintained example references for %q: %v", pipelineID, err)
|
||||
}
|
||||
if example.name == "production" {
|
||||
if len(materialized.ArtifactLanes) != 1 ||
|
||||
len(materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["spell_catalog"].Items) != 1 ||
|
||||
len(materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"].Items) != 1 {
|
||||
t.Fatalf("production spell catalog reference was not materialized: %#v", materialized.ArtifactLanes)
|
||||
if len(materialized.Steps[0].ArtifactLanes) != 1 ||
|
||||
len(materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["spell_catalog"].Items) != 1 ||
|
||||
len(materialized.Steps[0].ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"].Items) != 1 {
|
||||
t.Fatalf("production spell catalog reference was not materialized: %#v", materialized.Steps[0].ArtifactLanes)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -22,7 +22,7 @@ func TestOversizedNPCRegistryFailsBeforeRuntimeAndCheckpointConstruction(t *test
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
|
||||
content := string(readRepositoryFile(t, "examples", "dnd-npc-spell-sequential.config.yml"))
|
||||
content := string(readRepositoryFile(t, "examples", "dnd-spells.config.yml"))
|
||||
content = replaceRequiredOnce(t, content, " extract: dnd/spells", " extract:\n module: dnd/spells\n references:\n npcs: "+npcPath)
|
||||
content = replaceRequiredOnce(t, content, " enabled: false\n directory: \"\"", " enabled: true\n directory: "+checkpointRoot)
|
||||
configPath := filepath.Join(t.TempDir(), "config.yml")
|
||||
@@ -46,11 +46,11 @@ func TestOversizedNPCRegistryFailsBeforeRuntimeAndCheckpointConstruction(t *test
|
||||
}
|
||||
var stdout, stderr strings.Builder
|
||||
code := RunWithOptions([]string{
|
||||
"run", "dnd-spells", "--config", configPath,
|
||||
"run", "dnd-session", "--config", configPath,
|
||||
"--input", repositoryPath("examples", "seriatim-minimal-transcript.json"),
|
||||
"--chunk_cache", "bypass", "--output-dir", t.TempDir(),
|
||||
}, &stdout, &stderr, options)
|
||||
for _, fragment := range []string{`pipeline "dnd-spells"`, `reference slot "npcs"`, "1048577 bytes", "limit 1048576"} {
|
||||
for _, fragment := range []string{`pipeline "dnd-session"`, `reference slot "npcs"`, "1048577 bytes", "limit 1048576"} {
|
||||
if code == 0 || !strings.Contains(stderr.String(), fragment) {
|
||||
t.Fatalf("RunWithOptions() code = %d stderr = %q, want context fragment %q", code, stderr.String(), fragment)
|
||||
}
|
||||
|
||||
@@ -187,8 +187,8 @@ func TestProductionSpellValidatorsPrepareFromMaterializedCatalog(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("materialize production spell references: %v", err)
|
||||
}
|
||||
extractItems := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["spell_catalog"].Items
|
||||
normalizeItems := materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"].Items
|
||||
extractItems := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["spell_catalog"].Items
|
||||
normalizeItems := materialized.Steps[0].ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"].Items
|
||||
if len(extractItems) != 1 || extractItems[0].MediaType != "application/json" || len(extractItems[0].Content) == 0 {
|
||||
t.Fatalf("materialized extract spell catalog items = %#v, want one JSON item", extractItems)
|
||||
}
|
||||
@@ -243,9 +243,9 @@ func TestProductionSpellNormalizerRejectsInvalidCatalogReferencesBeforeExecution
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
slot := materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"]
|
||||
slot := materialized.Steps[0].ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"]
|
||||
slot.Items = append(slot.Items, slot.Items[0])
|
||||
materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"] = slot
|
||||
materialized.Steps[0].ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"] = slot
|
||||
_, err = pipeline.Prepare(materialized, components.registries, pipeline.ModuleDependencies{LLM: &productionFakeLLMClient{}})
|
||||
for _, fragment := range []string{"normalize", `module "dnd/spells"`, "zero or one item"} {
|
||||
if err == nil || !strings.Contains(err.Error(), fragment) {
|
||||
@@ -307,10 +307,10 @@ func TestProductionSpellNormalizerRejectsInvalidCatalogReferencesBeforeExecution
|
||||
|
||||
func setNormalizeSpellCatalogSource(t *testing.T, resolved *pipeline.ResolvedPipeline, sourcePath string) {
|
||||
t.Helper()
|
||||
if resolved == nil || len(resolved.ArtifactLanes) != 1 {
|
||||
if resolved == nil || len(resolved.Steps[0].ArtifactLanes) != 1 {
|
||||
t.Fatalf("resolved pipeline = %#v, want one artifact lane", resolved)
|
||||
}
|
||||
bindings := resolved.ArtifactLanes[0].NormalizeReferences.Bindings
|
||||
bindings := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings
|
||||
matches := 0
|
||||
for index := range bindings {
|
||||
if bindings[index].SlotName == "spell_catalog" {
|
||||
@@ -321,7 +321,7 @@ func setNormalizeSpellCatalogSource(t *testing.T, resolved *pipeline.ResolvedPip
|
||||
if matches != 1 {
|
||||
t.Fatalf("normalize reference bindings = %#v, want exactly one spell_catalog binding", bindings)
|
||||
}
|
||||
resolved.ArtifactLanes[0].NormalizeReferences.Bindings = bindings
|
||||
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings = bindings
|
||||
}
|
||||
|
||||
func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
|
||||
@@ -504,9 +504,8 @@ func maintainedExampleFiles(t *testing.T) []maintainedExample {
|
||||
{name: "minimal", path: repositoryPath("examples", "dnd-spells.config.yml"), pipelineIDs: []string{"dnd-session"}},
|
||||
{name: "production", path: repositoryPath("examples", "dnd-spells-production.config.yml"), pipelineIDs: []string{"dnd-session"}},
|
||||
{name: "npcs", path: repositoryPath("examples", "dnd-npcs.config.yml"), pipelineIDs: []string{"dnd-session"}},
|
||||
{name: "sequential", path: repositoryPath("examples", "dnd-npc-spell-sequential.config.yml"), pipelineIDs: []string{"dnd-npcs", "dnd-spells"}},
|
||||
{name: "combat", path: repositoryPath("examples", "dnd-combat-turns.config.yml"), pipelineIDs: []string{"dnd-combat"}},
|
||||
{name: "npc-combat-sequential", path: repositoryPath("examples", "dnd-npc-combat-sequential.config.yml"), pipelineIDs: []string{"dnd-combat", "dnd-npcs"}},
|
||||
{name: "npc-grounded", path: repositoryPath("examples", "dnd-npc-grounded.config.yml"), pipelineIDs: []string{"dnd-npc-grounded"}},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
323
internal/cli/recompute_execution_contract_test.go
Normal file
323
internal/cli/recompute_execution_contract_test.go
Normal file
@@ -0,0 +1,323 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
)
|
||||
|
||||
func TestRecomputeStepRecoversThroughFilesystemCheckpoints(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
invalidateOutput bool
|
||||
wantCode int
|
||||
}{
|
||||
{name: "accepted producer is hydrated", wantCode: 0},
|
||||
{name: "invalid producer stops dependents", invalidateOutput: true, wantCode: 1},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
roots := newRecomputeTestRoots(t)
|
||||
harness := newRecomputeTestHarness()
|
||||
fresh := runRecomputeCommand(roots, harness.options(), false)
|
||||
if fresh.code != 0 {
|
||||
t.Fatalf("fresh run code=%d stderr=%q", fresh.code, fresh.stderr)
|
||||
}
|
||||
removeCheckpointLaneStage(t, roots.checkpoints, "extract", "first", "producer")
|
||||
removeCheckpointLaneStage(t, roots.checkpoints, "merge", "first", "producer")
|
||||
if tt.invalidateOutput {
|
||||
path := findCheckpointFile(t, roots.checkpoints, "normalize", "first", "producer", "output.json")
|
||||
if err := os.WriteFile(path, []byte("{"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
harness.resetCalls()
|
||||
|
||||
resumed := runRecomputeCommand(roots, harness.options(), true)
|
||||
if resumed.code != tt.wantCode {
|
||||
t.Fatalf("resumed code=%d stdout=%q stderr=%q", resumed.code, resumed.stdout, resumed.stderr)
|
||||
}
|
||||
events := readLatestCheckpointEvents(t, roots.debug)
|
||||
if tt.invalidateOutput {
|
||||
if harness.callsFor("test/extract/middle") != 0 || harness.callsFor("test/extract/dependent") != 0 {
|
||||
t.Fatalf("dependent calls after invalid producer = %#v", harness.callsSnapshot())
|
||||
}
|
||||
if !strings.Contains(resumed.stderr, string(pipeline.CheckpointReasonDecodeFailed)) {
|
||||
t.Fatalf("stderr=%q, want stable checkpoint reason", resumed.stderr)
|
||||
}
|
||||
assertNormalizeDecisionSequence(t, events, []checkpointDecisionExpectation{{"first", "producer", pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonDecodeFailed}})
|
||||
return
|
||||
}
|
||||
|
||||
if got := harness.callsSnapshot(); !reflect.DeepEqual(got, map[string]int{"test/extract/dependent": 1, "test/extract/middle": 1}) {
|
||||
t.Fatalf("resumed extractor calls = %#v", got)
|
||||
}
|
||||
outputPath := filepath.Join(latestChildDir(t, roots.output), "result.json")
|
||||
data, err := os.ReadFile(outputPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(data) != "[\"producer\",\"unrelated\",\"middle\",\"dependent\"]\n" {
|
||||
t.Fatalf("ordered output = %q", data)
|
||||
}
|
||||
assertNormalizeDecisionSequence(t, events, []checkpointDecisionExpectation{
|
||||
{"first", "producer", pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonAcceptedArtifactReused},
|
||||
{"first", "unrelated", pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonReused},
|
||||
{"second", "middle", pipeline.CheckpointDecisionForcedRecompute, pipeline.CheckpointReasonRecomputeStep},
|
||||
{"third", "dependent", pipeline.CheckpointDecisionForcedRecompute, pipeline.CheckpointReasonRecomputeStep},
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type checkpointDecisionExpectation struct {
|
||||
step, lane string
|
||||
action pipeline.CheckpointDecisionCategory
|
||||
reason pipeline.CheckpointReasonCode
|
||||
}
|
||||
|
||||
func assertNormalizeDecisionSequence(t *testing.T, events []pipeline.CheckpointEvent, want []checkpointDecisionExpectation) {
|
||||
t.Helper()
|
||||
var got []checkpointDecisionExpectation
|
||||
for _, event := range events {
|
||||
if event.Stage == string(pipeline.StageNormalize) {
|
||||
got = append(got, checkpointDecisionExpectation{event.StepID, event.LaneID, event.Action, event.ReasonCode})
|
||||
}
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("normalize decisions = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
type recomputeTestHarness struct {
|
||||
base *stateTestHarness
|
||||
mu sync.Mutex
|
||||
calls map[string]int
|
||||
}
|
||||
|
||||
func newRecomputeTestHarness() *recomputeTestHarness {
|
||||
return &recomputeTestHarness{base: newStateTestHarness(), calls: make(map[string]int)}
|
||||
}
|
||||
|
||||
func (h *recomputeTestHarness) options() Options {
|
||||
opts := h.base.options()
|
||||
for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} {
|
||||
moduleKey := key
|
||||
spec := pipeline.ModuleSpec{
|
||||
Key: moduleKey, Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}},
|
||||
}
|
||||
if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) {
|
||||
return recomputeTestExtractor{key: moduleKey, harness: h}, nil
|
||||
}); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
|
||||
return recomputeTestOutput{}, nil
|
||||
}); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
opts.Catalog = catalogFromRegistries(opts.Registries)
|
||||
return opts
|
||||
}
|
||||
|
||||
func (h *recomputeTestHarness) record(key string) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
h.calls[key]++
|
||||
}
|
||||
|
||||
func (h *recomputeTestHarness) resetCalls() {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
h.calls = make(map[string]int)
|
||||
}
|
||||
|
||||
func (h *recomputeTestHarness) callsFor(key string) int {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
return h.calls[key]
|
||||
}
|
||||
|
||||
func (h *recomputeTestHarness) callsSnapshot() map[string]int {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
result := make(map[string]int, len(h.calls))
|
||||
for key, value := range h.calls {
|
||||
result[key] = value
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
type recomputeTestExtractor struct {
|
||||
key string
|
||||
harness *recomputeTestHarness
|
||||
}
|
||||
|
||||
func (e recomputeTestExtractor) Key() string { return e.key }
|
||||
func (e recomputeTestExtractor) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}}
|
||||
}
|
||||
func (e recomputeTestExtractor) Extract(context.Context, contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[stateTestArtifact], error) {
|
||||
e.harness.record(e.key)
|
||||
return contracts.TypedExtractionResult[stateTestArtifact]{Value: stateTestArtifact{Value: e.key}}, nil
|
||||
}
|
||||
|
||||
type recomputeTestOutput struct{}
|
||||
|
||||
func (recomputeTestOutput) Key() string { return "test/recompute-output" }
|
||||
func (recomputeTestOutput) Encode(_ context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||
lanes := make([]string, 0, len(req.NormalizeOutputs))
|
||||
for _, output := range req.NormalizeOutputs {
|
||||
lanes = append(lanes, output.LaneID)
|
||||
}
|
||||
data, err := json.Marshal(lanes)
|
||||
if err != nil {
|
||||
return contracts.OutputResult{}, err
|
||||
}
|
||||
return contracts.OutputResult{Files: []contracts.OutputFile{{Name: "result.json", Bytes: append(data, '\n')}}}, nil
|
||||
}
|
||||
|
||||
func newRecomputeTestRoots(t *testing.T) stateTestRoots {
|
||||
t.Helper()
|
||||
roots := newStateTestRoots(t)
|
||||
config := fmt.Sprintf(`version: 3
|
||||
output:
|
||||
directory: %q
|
||||
cache:
|
||||
chunk_plans:
|
||||
directory: %q
|
||||
mode: bypass
|
||||
checkpoints:
|
||||
enabled: true
|
||||
directory: %q
|
||||
debug:
|
||||
directory: %q
|
||||
pipelines:
|
||||
sample:
|
||||
input: test/input
|
||||
chunk: test/chunk
|
||||
steps:
|
||||
- id: first
|
||||
artifacts:
|
||||
producer:
|
||||
extract: test/extract/producer
|
||||
merge: test/merge
|
||||
normalize: test/normalize
|
||||
unrelated:
|
||||
extract: test/extract/unrelated
|
||||
merge: test/merge
|
||||
normalize: test/normalize
|
||||
- id: second
|
||||
references:
|
||||
upstream:
|
||||
artifact:
|
||||
step: first
|
||||
lane: producer
|
||||
artifacts:
|
||||
middle:
|
||||
extract: test/extract/middle
|
||||
merge: test/merge
|
||||
normalize: test/normalize
|
||||
- id: third
|
||||
references:
|
||||
upstream:
|
||||
artifact:
|
||||
step: second
|
||||
lane: middle
|
||||
artifacts:
|
||||
dependent:
|
||||
extract: test/extract/dependent
|
||||
merge: test/merge
|
||||
normalize: test/normalize
|
||||
output: test/recompute-output
|
||||
`, roots.output, roots.plans, roots.checkpoints, roots.debug)
|
||||
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return roots
|
||||
}
|
||||
|
||||
func runRecomputeCommand(roots stateTestRoots, opts Options, recompute bool) stateTestResult {
|
||||
args := []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}
|
||||
if recompute {
|
||||
args = append(args, "--resume", "--recompute-step", "second", "--debug")
|
||||
}
|
||||
var stdout, stderr bytes.Buffer
|
||||
return stateTestResult{code: RunWithOptions(args, &stdout, &stderr, opts), stdout: stdout.String(), stderr: stderr.String()}
|
||||
}
|
||||
|
||||
func removeCheckpointLaneStage(t *testing.T, root, stage, step, lane string) {
|
||||
t.Helper()
|
||||
dir := filepath.Dir(findCheckpointFile(t, root, stage, step, lane, "manifest.json"))
|
||||
if err := os.RemoveAll(dir); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func findCheckpointFile(t *testing.T, root, stage, step, lane, name string) string {
|
||||
t.Helper()
|
||||
want := filepath.Join(stage, step, lane, name)
|
||||
var matches []string
|
||||
err := filepath.WalkDir(root, func(path string, entry os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !entry.IsDir() && strings.HasSuffix(path, want) {
|
||||
matches = append(matches, path)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(matches) != 1 {
|
||||
t.Fatalf("checkpoint files ending in %q = %v", want, matches)
|
||||
}
|
||||
return matches[0]
|
||||
}
|
||||
|
||||
func latestChildDir(t *testing.T, root string) string {
|
||||
t.Helper()
|
||||
entries, err := os.ReadDir(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var dirs []string
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() {
|
||||
dirs = append(dirs, filepath.Join(root, entry.Name()))
|
||||
}
|
||||
}
|
||||
if len(dirs) == 0 {
|
||||
t.Fatal("no child directory")
|
||||
}
|
||||
sort.Strings(dirs)
|
||||
return dirs[len(dirs)-1]
|
||||
}
|
||||
|
||||
func readLatestCheckpointEvents(t *testing.T, root string) []pipeline.CheckpointEvent {
|
||||
t.Helper()
|
||||
var events []pipeline.CheckpointEvent
|
||||
data, err := os.ReadFile(filepath.Join(latestChildDir(t, root), "summary", "checkpoint-events.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := json.Unmarshal(data, &events); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return events
|
||||
}
|
||||
56
internal/cli/recompute_policy_test.go
Normal file
56
internal/cli/recompute_policy_test.go
Normal file
@@ -0,0 +1,56 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
)
|
||||
|
||||
func TestRecomputePolicyIncludesDependentLanesAndReusablePredecessors(t *testing.T) {
|
||||
producer := pipeline.ResolvedArtifactLane{StepID: "first", ID: "producer"}
|
||||
unrelated := pipeline.ResolvedArtifactLane{StepID: "first", ID: "unrelated"}
|
||||
consumer := pipeline.ResolvedArtifactLane{
|
||||
StepID: "second", ID: "consumer",
|
||||
ExtractReferences: pipeline.ResolvedReferenceTarget{Bindings: []pipeline.ReferenceBinding{{Artifact: &pipeline.ArtifactReference{Step: "first", Lane: "producer"}}}},
|
||||
}
|
||||
downstream := pipeline.ResolvedArtifactLane{
|
||||
StepID: "third", ID: "downstream",
|
||||
ExtractReferences: pipeline.ResolvedReferenceTarget{Bindings: []pipeline.ReferenceBinding{{Artifact: &pipeline.ArtifactReference{Step: "second", Lane: "consumer"}}}},
|
||||
}
|
||||
independent := pipeline.ResolvedArtifactLane{StepID: "third", ID: "independent"}
|
||||
resolved := pipeline.ResolvedPipeline{Steps: []pipeline.ResolvedPipelineStep{
|
||||
{ID: "first", ArtifactLanes: []pipeline.ResolvedArtifactLane{producer, unrelated}},
|
||||
{ID: "second", ArtifactLanes: []pipeline.ResolvedArtifactLane{consumer}},
|
||||
{ID: "third", ArtifactLanes: []pipeline.ResolvedArtifactLane{downstream, independent}},
|
||||
}}
|
||||
|
||||
policy, err := recomputePolicy(resolved, "second")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("second", "consumer")]; !ok {
|
||||
t.Fatal("selected lane was not forced")
|
||||
}
|
||||
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("third", "downstream")]; !ok {
|
||||
t.Fatal("transitive dependent lane was not forced")
|
||||
}
|
||||
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("first", "producer")]; ok {
|
||||
t.Fatal("predecessor was implicitly forced")
|
||||
}
|
||||
if _, ok := policy.RequireReusableLanes[pipeline.CheckpointLaneKey("first", "producer")]; !ok {
|
||||
t.Fatal("required predecessor was not marked reusable")
|
||||
}
|
||||
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("first", "unrelated")]; ok {
|
||||
t.Fatal("unrelated lane was forced")
|
||||
}
|
||||
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("third", "independent")]; ok {
|
||||
t.Fatal("unrelated later lane was forced")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRecomputePolicyRejectsUnknownStep(t *testing.T) {
|
||||
_, err := recomputePolicy(pipeline.ResolvedPipeline{Steps: []pipeline.ResolvedPipelineStep{{ID: "known"}}}, "missing")
|
||||
if err == nil {
|
||||
t.Fatal("unknown step was accepted")
|
||||
}
|
||||
}
|
||||
@@ -319,28 +319,28 @@ func referenceContractConfig() config.Config {
|
||||
Extract: pipeline.Binding("reference/extract-alpha"),
|
||||
Merge: pipeline.Binding("reference/shared-merge"),
|
||||
Normalize: pipeline.Binding("reference/shared-normalize"),
|
||||
References: map[string]string{"required-extract": "required.txt"},
|
||||
References: pipeline.ExternalReferenceMap(map[string]string{"required-extract": "required.txt"}),
|
||||
},
|
||||
"beta": {
|
||||
Extract: pipeline.Binding("reference/extract-beta"),
|
||||
Merge: pipeline.Binding("reference/shared-merge"),
|
||||
Normalize: pipeline.Binding("reference/shared-normalize"),
|
||||
References: map[string]string{"required-extract": "required.txt"},
|
||||
References: pipeline.ExternalReferenceMap(map[string]string{"required-extract": "required.txt"}),
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
profile := cfg.Pipelines["demo"]
|
||||
profile.Chunk.References = map[string]string{"required-chunk": "required.txt"}
|
||||
profile.Chunk.References = pipeline.ExternalReferenceMap(map[string]string{"required-chunk": "required.txt"})
|
||||
alpha := profile.Artifacts["alpha"]
|
||||
alpha.Extract.References = map[string]string{"required-extract": "required.txt", "alpha-slot": "optional.txt"}
|
||||
alpha.Merge.References = map[string]string{"required-merge": "required.txt"}
|
||||
alpha.Normalize.References = map[string]string{"required-normalize": "required.txt"}
|
||||
alpha.Extract.References = pipeline.ExternalReferenceMap(map[string]string{"required-extract": "required.txt", "alpha-slot": "optional.txt"})
|
||||
alpha.Merge.References = pipeline.ExternalReferenceMap(map[string]string{"required-merge": "required.txt"})
|
||||
alpha.Normalize.References = pipeline.ExternalReferenceMap(map[string]string{"required-normalize": "required.txt"})
|
||||
profile.Artifacts["alpha"] = alpha
|
||||
beta := profile.Artifacts["beta"]
|
||||
beta.Extract.References = map[string]string{"required-extract": "required.txt"}
|
||||
beta.Merge.References = map[string]string{"required-merge": "required.txt"}
|
||||
beta.Normalize.References = map[string]string{"required-normalize": "required.txt"}
|
||||
beta.Extract.References = pipeline.ExternalReferenceMap(map[string]string{"required-extract": "required.txt"})
|
||||
beta.Merge.References = pipeline.ExternalReferenceMap(map[string]string{"required-merge": "required.txt"})
|
||||
beta.Normalize.References = pipeline.ExternalReferenceMap(map[string]string{"required-normalize": "required.txt"})
|
||||
profile.Artifacts["beta"] = beta
|
||||
cfg.Pipelines["demo"] = profile
|
||||
return cfg
|
||||
@@ -418,7 +418,7 @@ func (referenceContractCodecB) Decode([]byte) (stateTestArtifact, error) {
|
||||
|
||||
func referenceContractLane(t *testing.T, resolved pipeline.ResolvedPipeline, id string) pipeline.ResolvedArtifactLane {
|
||||
t.Helper()
|
||||
for _, lane := range resolved.ArtifactLanes {
|
||||
for _, lane := range resolved.Steps[0].ArtifactLanes {
|
||||
if lane.ID == id {
|
||||
return lane
|
||||
}
|
||||
|
||||
@@ -28,7 +28,7 @@ import (
|
||||
const defaultConfigPath = "/usr/local/etc/notarius/config.yml"
|
||||
const usage = `Usage:
|
||||
notarius help
|
||||
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--output-dir path] [--chunk_cache auto|bypass|refresh] [--resume] [--debug [--debug-dir path]] [--only lane-a,lane-b] [--session-id id] [--reference selector=path] [--without-reference selector]
|
||||
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--output-dir path] [--chunk_cache auto|bypass|refresh] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--only lane-a,lane-b] [--session-id id] [--reference selector=path] [--without-reference selector]
|
||||
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
|
||||
notarius pipelines list --config path/to/config.yml [--json]
|
||||
`
|
||||
@@ -136,6 +136,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
debugDir := fs.String("debug-dir", "", "debug bundle directory")
|
||||
llmProfile := fs.String("llm-profile", "", "LLM profile override")
|
||||
resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints")
|
||||
recomputeStep := singleValueFlag{}
|
||||
chunkCache := chunkCacheFlag{}
|
||||
sessionID := sessionIDFlag{}
|
||||
referenceFlags := stringListFlag{}
|
||||
@@ -144,6 +145,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh")
|
||||
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path")
|
||||
fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference")
|
||||
fs.Var(&recomputeStep, "recompute-step", "recompute one ordered pipeline step and dependent lanes")
|
||||
if err := validateRunFlagValues(args); err != nil {
|
||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||
return 2
|
||||
@@ -190,6 +192,14 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||
return 2
|
||||
}
|
||||
if recomputeStep.set && strings.TrimSpace(recomputeStep.value) == "" {
|
||||
fmt.Fprintln(stderr, "notarius: --recompute-step must not be empty")
|
||||
return 2
|
||||
}
|
||||
if recomputeStep.set && len(only) > 0 {
|
||||
fmt.Fprintln(stderr, "notarius: --recompute-step cannot be combined with --only")
|
||||
return 2
|
||||
}
|
||||
referenceRequests, err := parseReferenceFlags(referenceFlags)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||
@@ -223,6 +233,14 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
fmt.Fprintln(stderr, "notarius: --resume requires cache.checkpoints.enabled: true")
|
||||
return 1
|
||||
}
|
||||
if recomputeStep.set && !*resume {
|
||||
fmt.Fprintln(stderr, "notarius: --recompute-step requires --resume")
|
||||
return 2
|
||||
}
|
||||
if recomputeStep.set && !cfg.Cache.Checkpoints.Enabled {
|
||||
fmt.Fprintln(stderr, "notarius: --recompute-step requires cache.checkpoints.enabled: true")
|
||||
return 1
|
||||
}
|
||||
|
||||
startedAt := opts.Now().UTC()
|
||||
runID, err := opts.RunIDGenerator(startedAt)
|
||||
@@ -267,6 +285,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
OnlyLanes: append([]string(nil), only...),
|
||||
ChunkCacheOverride: chunkCache.explicitValue(),
|
||||
Resume: *resume,
|
||||
RecomputeStep: strings.TrimSpace(recomputeStep.value),
|
||||
RunID: runID,
|
||||
StartedAt: startedAt,
|
||||
}
|
||||
@@ -309,6 +328,13 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||
}
|
||||
effective.ResolvedPipeline = materialized
|
||||
checkpointPolicy := pipeline.CheckpointExecutionPolicy{}
|
||||
if recomputeStep.set {
|
||||
checkpointPolicy, err = recomputePolicy(effective.ResolvedPipeline, recomputeStep.value)
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||
}
|
||||
}
|
||||
invocation.PipelineDigest = effective.ResolvedPipeline.Digest
|
||||
if err := writeSummary(summary, func() error { return summary.WriteInvocation(invocation) }); err != nil {
|
||||
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug invocation metadata: %w", err))
|
||||
@@ -358,21 +384,22 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
}
|
||||
|
||||
output, err := pipeline.New().Run(ctx, pipeline.RunInput{
|
||||
Prepared: prepared,
|
||||
Path: strings.TrimSpace(*inputPath),
|
||||
RawInput: rawInput,
|
||||
SessionID: strings.TrimSpace(sessionID.value),
|
||||
RunID: runID,
|
||||
StartedAt: startedAt,
|
||||
LLMProfiles: llmProfiles,
|
||||
Metadata: runMetadata(effective.Config.Output.Directory, debugPath),
|
||||
Warnings: referenceWarnings,
|
||||
ChunkCacheMode: effective.Config.Cache.ChunkPlans.Mode,
|
||||
ChunkPlans: chunkPlans,
|
||||
Checkpoints: checkpointRecorder,
|
||||
Checkpoint: checkpointLoader,
|
||||
Debug: debugRecorder,
|
||||
ExtractWorkers: cfg.Concurrency.StageWorkers["extract"],
|
||||
Prepared: prepared,
|
||||
Path: strings.TrimSpace(*inputPath),
|
||||
RawInput: rawInput,
|
||||
SessionID: strings.TrimSpace(sessionID.value),
|
||||
RunID: runID,
|
||||
StartedAt: startedAt,
|
||||
LLMProfiles: llmProfiles,
|
||||
Metadata: runMetadata(effective.Config.Output.Directory, debugPath),
|
||||
Warnings: referenceWarnings,
|
||||
ChunkCacheMode: effective.Config.Cache.ChunkPlans.Mode,
|
||||
ChunkPlans: chunkPlans,
|
||||
Checkpoints: checkpointRecorder,
|
||||
Checkpoint: checkpointLoader,
|
||||
CheckpointPolicy: checkpointPolicy,
|
||||
Debug: debugRecorder,
|
||||
ExtractWorkers: cfg.Concurrency.StageWorkers["extract"],
|
||||
})
|
||||
commandState.observeOutput(output)
|
||||
if err != nil {
|
||||
@@ -481,6 +508,95 @@ func checkpointHandlersForRun(
|
||||
return recorder, loader, nil
|
||||
}
|
||||
|
||||
func recomputePolicy(resolved pipeline.ResolvedPipeline, requestedStep string) (pipeline.CheckpointExecutionPolicy, error) {
|
||||
requestedStep = strings.TrimSpace(requestedStep)
|
||||
if requestedStep == "" {
|
||||
return pipeline.CheckpointExecutionPolicy{}, fmt.Errorf("--recompute-step must not be empty")
|
||||
}
|
||||
var selected *pipeline.ResolvedPipelineStep
|
||||
for index := range resolved.Steps {
|
||||
if strings.TrimSpace(resolved.Steps[index].ID) == requestedStep {
|
||||
selected = &resolved.Steps[index]
|
||||
break
|
||||
}
|
||||
}
|
||||
if selected == nil {
|
||||
return pipeline.CheckpointExecutionPolicy{}, fmt.Errorf("unknown pipeline step %q", requestedStep)
|
||||
}
|
||||
|
||||
policy := pipeline.CheckpointExecutionPolicy{ForcedLanes: make(map[string]struct{}), RequireReusableLanes: make(map[string]struct{})}
|
||||
var forced []pipeline.ResolvedArtifactLane
|
||||
for _, lane := range selected.ArtifactLanes {
|
||||
lane.StepID = selected.ID
|
||||
policy.ForcedLanes[pipeline.CheckpointLaneKey(selected.ID, lane.ID)] = struct{}{}
|
||||
forced = append(forced, lane)
|
||||
}
|
||||
|
||||
lanes := make(map[string]pipeline.ResolvedArtifactLane)
|
||||
dependents := make(map[string][]pipeline.ResolvedArtifactLane)
|
||||
for _, step := range resolved.Steps {
|
||||
for _, lane := range step.ArtifactLanes {
|
||||
lane.StepID = step.ID
|
||||
lanes[pipeline.CheckpointLaneKey(step.ID, lane.ID)] = lane
|
||||
for _, target := range []pipeline.ResolvedReferenceTarget{lane.ExtractReferences, lane.MergeReferences, lane.NormalizeReferences} {
|
||||
for _, binding := range target.Bindings {
|
||||
if binding.Artifact != nil {
|
||||
producerKey := pipeline.CheckpointLaneKey(binding.Artifact.Step, binding.Artifact.Lane)
|
||||
dependents[producerKey] = append(dependents[producerKey], lane)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
processed := make(map[string]struct{})
|
||||
queue := append([]pipeline.ResolvedArtifactLane(nil), forced...)
|
||||
for len(queue) > 0 {
|
||||
lane := queue[0]
|
||||
queue = queue[1:]
|
||||
key := pipeline.CheckpointLaneKey(lane.StepID, lane.ID)
|
||||
if _, ok := processed[key]; ok {
|
||||
continue
|
||||
}
|
||||
processed[key] = struct{}{}
|
||||
for _, dependent := range dependents[key] {
|
||||
dependentKey := pipeline.CheckpointLaneKey(dependent.StepID, dependent.ID)
|
||||
if _, alreadyForced := policy.ForcedLanes[dependentKey]; alreadyForced {
|
||||
continue
|
||||
}
|
||||
policy.ForcedLanes[dependentKey] = struct{}{}
|
||||
forced = append(forced, dependent)
|
||||
queue = append(queue, dependent)
|
||||
}
|
||||
}
|
||||
|
||||
processed = make(map[string]struct{})
|
||||
queue = append([]pipeline.ResolvedArtifactLane(nil), forced...)
|
||||
for len(queue) > 0 {
|
||||
lane := queue[0]
|
||||
queue = queue[1:]
|
||||
key := pipeline.CheckpointLaneKey(lane.StepID, lane.ID)
|
||||
if _, ok := processed[key]; ok {
|
||||
continue
|
||||
}
|
||||
processed[key] = struct{}{}
|
||||
for _, target := range []pipeline.ResolvedReferenceTarget{lane.ExtractReferences, lane.MergeReferences, lane.NormalizeReferences} {
|
||||
for _, binding := range target.Bindings {
|
||||
if binding.Artifact == nil {
|
||||
continue
|
||||
}
|
||||
producerKey := pipeline.CheckpointLaneKey(binding.Artifact.Step, binding.Artifact.Lane)
|
||||
if _, forcedAlready := policy.ForcedLanes[producerKey]; !forcedAlready {
|
||||
policy.RequireReusableLanes[producerKey] = struct{}{}
|
||||
}
|
||||
if producer, ok := lanes[producerKey]; ok {
|
||||
queue = append(queue, producer)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return policy, nil
|
||||
}
|
||||
|
||||
func checkpointIdentityFingerprints(values []pipeline.CheckpointFingerprint) []checkpoint.Fingerprint {
|
||||
if len(values) == 0 {
|
||||
return nil
|
||||
@@ -661,7 +777,7 @@ func reorderRunArgs(args []string) []string {
|
||||
|
||||
func runFlagTakesValue(arg string) bool {
|
||||
switch arg {
|
||||
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--chunk_cache", "--reference", "--without-reference":
|
||||
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
@@ -749,7 +865,7 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
|
||||
}
|
||||
}
|
||||
add(resolved.Chunk)
|
||||
for _, lane := range resolved.ArtifactLanes {
|
||||
for _, lane := range resolved.AllArtifactLanes() {
|
||||
add(lane.Extract)
|
||||
add(lane.Merge)
|
||||
add(lane.Normalize)
|
||||
@@ -1008,6 +1124,27 @@ type sessionIDFlag struct {
|
||||
set bool
|
||||
}
|
||||
|
||||
type singleValueFlag struct {
|
||||
value string
|
||||
set bool
|
||||
}
|
||||
|
||||
func (flag *singleValueFlag) String() string {
|
||||
if flag == nil {
|
||||
return ""
|
||||
}
|
||||
return flag.value
|
||||
}
|
||||
|
||||
func (flag *singleValueFlag) Set(value string) error {
|
||||
if flag.set {
|
||||
return fmt.Errorf("--recompute-step may be specified only once")
|
||||
}
|
||||
flag.value = value
|
||||
flag.set = true
|
||||
return nil
|
||||
}
|
||||
|
||||
func (flag *sessionIDFlag) String() string {
|
||||
if flag == nil {
|
||||
return ""
|
||||
@@ -1178,17 +1315,34 @@ func selectedReferenceTargets(cfg config.Config, pipelineID string, only []strin
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("pipeline %q is not configured", strings.TrimSpace(pipelineID))
|
||||
}
|
||||
if profile.Steps != nil && len(only) > 0 {
|
||||
return nil, fmt.Errorf("pipeline %q --only is not supported for explicit ordered steps", strings.TrimSpace(pipelineID))
|
||||
}
|
||||
|
||||
lanesByID := make(map[string]pipeline.ArtifactLaneProfile, len(profile.Artifacts))
|
||||
for rawLaneID, lane := range profile.Artifacts {
|
||||
laneID := strings.TrimSpace(rawLaneID)
|
||||
if laneID == "" {
|
||||
return nil, fmt.Errorf("pipeline %q artifact lane id must not be empty", strings.TrimSpace(pipelineID))
|
||||
addLanes := func(values map[string]pipeline.ArtifactLaneProfile) error {
|
||||
for rawLaneID, lane := range values {
|
||||
laneID := strings.TrimSpace(rawLaneID)
|
||||
if laneID == "" {
|
||||
return fmt.Errorf("pipeline %q artifact lane id must not be empty", strings.TrimSpace(pipelineID))
|
||||
}
|
||||
if _, ok := lanesByID[laneID]; ok {
|
||||
return fmt.Errorf("pipeline %q artifact lane %q is duplicated after trimming", strings.TrimSpace(pipelineID), laneID)
|
||||
}
|
||||
lanesByID[laneID] = lane
|
||||
}
|
||||
if _, ok := lanesByID[laneID]; ok {
|
||||
return nil, fmt.Errorf("pipeline %q artifact lane %q is duplicated after trimming", strings.TrimSpace(pipelineID), laneID)
|
||||
return nil
|
||||
}
|
||||
if profile.Steps == nil {
|
||||
if err := addLanes(profile.Artifacts); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
} else {
|
||||
for _, step := range profile.Steps {
|
||||
if err := addLanes(step.Artifacts); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
lanesByID[laneID] = lane
|
||||
}
|
||||
|
||||
selectedIDs := make([]string, 0, len(lanesByID))
|
||||
|
||||
@@ -60,6 +60,56 @@ func TestRunControlsRejectSyntaxWithoutAllocatingState(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRecomputeStepCLIContract(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
configure func(*testing.T, stateTestRoots)
|
||||
flags []string
|
||||
wantCode int
|
||||
wantOutput string
|
||||
wantError string
|
||||
}{
|
||||
{
|
||||
name: "explicit step",
|
||||
configure: func(t *testing.T, roots stateTestRoots) {
|
||||
replaceStateTestConfigLine(t, roots.config, " artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n", " steps:\n - id: chosen\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n")
|
||||
},
|
||||
flags: []string{"--resume", "--recompute-step", "chosen"},
|
||||
wantCode: 0,
|
||||
wantOutput: "outputs=1",
|
||||
},
|
||||
{name: "implicit default step", flags: []string{"--resume", "--recompute-step", "default"}, wantCode: 0, wantOutput: "outputs=1"},
|
||||
{name: "repeated flag", flags: []string{"--resume", "--recompute-step", "default", "--recompute-step", "default"}, wantCode: 2, wantError: "specified only once"},
|
||||
{name: "empty step", flags: []string{"--resume", "--recompute-step", ""}, wantCode: 2, wantError: "must not be empty"},
|
||||
{name: "unknown step", flags: []string{"--resume", "--recompute-step", "missing"}, wantCode: 1, wantError: "unknown pipeline step"},
|
||||
{name: "without resume", flags: []string{"--recompute-step", "default"}, wantCode: 2, wantError: "requires --resume"},
|
||||
{
|
||||
name: "checkpoint recording disabled",
|
||||
configure: func(t *testing.T, roots stateTestRoots) {
|
||||
replaceStateTestConfigLine(t, roots.config, " enabled: true\n", " enabled: false\n")
|
||||
},
|
||||
flags: []string{"--resume", "--recompute-step", "default"}, wantCode: 1, wantError: "cache.checkpoints.enabled",
|
||||
},
|
||||
{name: "with only", flags: []string{"--resume", "--recompute-step", "default", "--only", "items"}, wantCode: 2, wantError: "cannot be combined with --only"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
roots := newStateTestRoots(t)
|
||||
if tt.configure != nil {
|
||||
tt.configure(t, roots)
|
||||
}
|
||||
args := []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}
|
||||
args = append(args, tt.flags...)
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := RunWithOptions(args, &stdout, &stderr, newStateTestHarness().options())
|
||||
if code != tt.wantCode || (tt.wantOutput != "" && !strings.Contains(stdout.String(), tt.wantOutput)) || (tt.wantError != "" && !strings.Contains(stderr.String(), tt.wantError)) {
|
||||
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunValidFailuresClassifyAndReportDebug(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
@@ -268,13 +318,14 @@ func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
|
||||
resolved := pipeline.ResolvedPipeline{
|
||||
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
|
||||
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
|
||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{
|
||||
{
|
||||
Steps: []pipeline.ResolvedPipelineStep{{
|
||||
ID: "default",
|
||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
||||
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
|
||||
Merge: pipeline.ModuleBinding{LLMProfile: "zeta"},
|
||||
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
|
||||
},
|
||||
},
|
||||
}},
|
||||
}},
|
||||
ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{
|
||||
{Binding: pipeline.ModuleBinding{LLMProfile: "deterministic-profile"}, ExecutionClass: contracts.ExecutionClassDeterministic},
|
||||
{Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked},
|
||||
|
||||
@@ -31,7 +31,7 @@ func TestSpellCatalogBytesAffectCheckpointIdentityButNotSemanticDigest(t *testin
|
||||
}
|
||||
overlayPath := filepath.Join(t.TempDir(), "catalog.json")
|
||||
resolved := effective.ResolvedPipeline
|
||||
bindings := resolved.ArtifactLanes[0].ExtractReferences.Bindings
|
||||
bindings := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings
|
||||
catalogBindingIndex := -1
|
||||
for index, binding := range bindings {
|
||||
if binding.SlotName == spellcatalog.SpellCatalogReferenceSlot {
|
||||
@@ -42,8 +42,8 @@ func TestSpellCatalogBytesAffectCheckpointIdentityButNotSemanticDigest(t *testin
|
||||
if catalogBindingIndex < 0 {
|
||||
t.Fatalf("spell catalog bindings = %#v, want catalog binding", bindings)
|
||||
}
|
||||
resolved.ArtifactLanes[0].ExtractReferences.Bindings[catalogBindingIndex].Source = overlayPath
|
||||
normalizeBindings := resolved.ArtifactLanes[0].NormalizeReferences.Bindings
|
||||
resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings[catalogBindingIndex].Source = overlayPath
|
||||
normalizeBindings := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings
|
||||
normalizeCatalogBindingIndex := -1
|
||||
for index, binding := range normalizeBindings {
|
||||
if binding.SlotName == spellcatalog.SpellCatalogReferenceSlot {
|
||||
@@ -54,7 +54,7 @@ func TestSpellCatalogBytesAffectCheckpointIdentityButNotSemanticDigest(t *testin
|
||||
if normalizeCatalogBindingIndex < 0 {
|
||||
t.Fatalf("normalize spell catalog bindings = %#v, want catalog binding", normalizeBindings)
|
||||
}
|
||||
resolved.ArtifactLanes[0].NormalizeReferences.Bindings[normalizeCatalogBindingIndex].Source = overlayPath
|
||||
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings[normalizeCatalogBindingIndex].Source = overlayPath
|
||||
|
||||
if err := os.WriteFile(overlayPath, []byte(reorderedOverlayA), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -415,7 +415,7 @@ func catalogCheckpointIdentity(t *testing.T, resolved pipeline.ResolvedPipeline)
|
||||
|
||||
func catalogExtractorMetadata(t *testing.T, resolved pipeline.ResolvedPipeline) map[string]any {
|
||||
t.Helper()
|
||||
lane := resolved.ArtifactLanes[0]
|
||||
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||
extractor, err := spells.New(&productionFakeLLMClient{}, spells.Options{}, lane.ExtractReferences.ReferenceSet)
|
||||
if err != nil {
|
||||
t.Fatalf("construct extractor: %v", err)
|
||||
@@ -425,7 +425,7 @@ func catalogExtractorMetadata(t *testing.T, resolved pipeline.ResolvedPipeline)
|
||||
|
||||
func catalogNormalizerMetadata(t *testing.T, resolved pipeline.ResolvedPipeline) map[string]any {
|
||||
t.Helper()
|
||||
lane := resolved.ArtifactLanes[0]
|
||||
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||
normalizer, err := spellnormalize.New(spellnormalize.Options{}, lane.NormalizeReferences.ReferenceSet)
|
||||
if err != nil {
|
||||
t.Fatalf("construct normalizer: %v", err)
|
||||
|
||||
@@ -63,7 +63,7 @@ func TestProductionSpellCatalogValidationRetries(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("materialize production references: %v", err)
|
||||
}
|
||||
materialized.ArtifactLanes[0].Extract.Retries = retries
|
||||
materialized.Steps[0].ArtifactLanes[0].Extract.Retries = retries
|
||||
|
||||
llmClient := &catalogRetryLLMClient{responses: tt.responses}
|
||||
prepared, err := pipeline.Prepare(materialized, components.registries, pipeline.ModuleDependencies{LLM: llmClient})
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
)
|
||||
|
||||
type ArtifactLaneManifest struct {
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
ID string `json:"id"`
|
||||
Extractor string `json:"extractor"`
|
||||
Merger string `json:"merger"`
|
||||
@@ -31,15 +32,25 @@ type LLMProfileManifest struct {
|
||||
}
|
||||
|
||||
type ReferenceProvenance struct {
|
||||
Stage string `json:"stage,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
SlotName string `json:"slot_name"`
|
||||
OriginType string `json:"origin_type"`
|
||||
OriginURI string `json:"origin_uri,omitempty"`
|
||||
Digest string `json:"digest,omitempty"`
|
||||
MediaType string `json:"media_type,omitempty"`
|
||||
SizeBytes int64 `json:"size_bytes,omitempty"`
|
||||
BindingSource string `json:"binding_source,omitempty"`
|
||||
Stage string `json:"stage,omitempty"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
SlotName string `json:"slot_name"`
|
||||
OriginType string `json:"origin_type"`
|
||||
OriginURI string `json:"origin_uri,omitempty"`
|
||||
Digest string `json:"digest,omitempty"`
|
||||
MediaType string `json:"media_type,omitempty"`
|
||||
SizeBytes int64 `json:"size_bytes,omitempty"`
|
||||
BindingSource string `json:"binding_source,omitempty"`
|
||||
ArtifactKind string `json:"artifact_kind,omitempty"`
|
||||
SchemaID string `json:"schema_id,omitempty"`
|
||||
SchemaName string `json:"schema_name,omitempty"`
|
||||
SchemaVersion string `json:"schema_version,omitempty"`
|
||||
SchemaDigest string `json:"schema_digest,omitempty"`
|
||||
ProducerPipeline string `json:"producer_pipeline_id,omitempty"`
|
||||
ProducerStep string `json:"producer_step_id,omitempty"`
|
||||
ProducerLane string `json:"producer_lane_id,omitempty"`
|
||||
ProducerModule string `json:"producer_module_key,omitempty"`
|
||||
}
|
||||
|
||||
type OutputSchemaProvenance struct {
|
||||
@@ -49,6 +60,7 @@ type OutputSchemaProvenance struct {
|
||||
}
|
||||
|
||||
type NormalizedOutputManifest struct {
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
@@ -58,6 +70,7 @@ type NormalizedOutputManifest struct {
|
||||
|
||||
type RejectedOutputManifest struct {
|
||||
Stage string `json:"stage"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
ChunkID string `json:"chunk_id,omitempty"`
|
||||
@@ -69,6 +82,16 @@ type RejectedOutputManifest struct {
|
||||
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
|
||||
}
|
||||
|
||||
type CheckpointDecisionManifest struct {
|
||||
Stage string `json:"stage"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
Category string `json:"category"`
|
||||
ReasonCode string `json:"reason_code,omitempty"`
|
||||
Detail string `json:"detail,omitempty"`
|
||||
}
|
||||
|
||||
type ChunkPlanManifest struct {
|
||||
Mode string `json:"mode"`
|
||||
Action string `json:"action,omitempty"`
|
||||
@@ -99,27 +122,28 @@ type ChunkPlanSummary struct {
|
||||
}
|
||||
|
||||
type RunManifest struct {
|
||||
RunID string `json:"run_id,omitempty"`
|
||||
PipelineID string `json:"pipeline_id,omitempty"`
|
||||
PipelineDigest string `json:"pipeline_digest,omitempty"`
|
||||
InputModule string `json:"input_module,omitempty"`
|
||||
Chunker string `json:"chunker,omitempty"`
|
||||
ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"`
|
||||
SourceDigests []string `json:"source_digests,omitempty"`
|
||||
Extractors []string `json:"extractors,omitempty"`
|
||||
Merger string `json:"merger,omitempty"`
|
||||
Normalizer string `json:"normalizer,omitempty"`
|
||||
OutputEncoder string `json:"output_encoder,omitempty"`
|
||||
ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"`
|
||||
ArtifactLanes []ArtifactLaneManifest `json:"artifact_lanes,omitempty"`
|
||||
ValidatorChains []ValidatorChainManifest `json:"validator_chains,omitempty"`
|
||||
References []ReferenceProvenance `json:"references,omitempty"`
|
||||
NormalizedOutputs []NormalizedOutputManifest `json:"normalized_outputs,omitempty"`
|
||||
RejectedOutputs []RejectedOutputManifest `json:"rejected_outputs,omitempty"`
|
||||
LLMProfiles []LLMProfileManifest `json:"llm_profiles,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
SchemaVersion string `json:"schema_version,omitempty"`
|
||||
ValidationStatus string `json:"validation_status,omitempty"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
CompletedAt *time.Time `json:"completed_at,omitempty"`
|
||||
RunID string `json:"run_id,omitempty"`
|
||||
PipelineID string `json:"pipeline_id,omitempty"`
|
||||
PipelineDigest string `json:"pipeline_digest,omitempty"`
|
||||
InputModule string `json:"input_module,omitempty"`
|
||||
Chunker string `json:"chunker,omitempty"`
|
||||
ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"`
|
||||
SourceDigests []string `json:"source_digests,omitempty"`
|
||||
Extractors []string `json:"extractors,omitempty"`
|
||||
Merger string `json:"merger,omitempty"`
|
||||
Normalizer string `json:"normalizer,omitempty"`
|
||||
OutputEncoder string `json:"output_encoder,omitempty"`
|
||||
ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"`
|
||||
ArtifactLanes []ArtifactLaneManifest `json:"artifact_lanes,omitempty"`
|
||||
ValidatorChains []ValidatorChainManifest `json:"validator_chains,omitempty"`
|
||||
References []ReferenceProvenance `json:"references,omitempty"`
|
||||
NormalizedOutputs []NormalizedOutputManifest `json:"normalized_outputs,omitempty"`
|
||||
RejectedOutputs []RejectedOutputManifest `json:"rejected_outputs,omitempty"`
|
||||
CheckpointDecisions []CheckpointDecisionManifest `json:"checkpoint_decisions,omitempty"`
|
||||
LLMProfiles []LLMProfileManifest `json:"llm_profiles,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
SchemaVersion string `json:"schema_version,omitempty"`
|
||||
ValidationStatus string `json:"validation_status,omitempty"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
CompletedAt *time.Time `json:"completed_at,omitempty"`
|
||||
}
|
||||
|
||||
@@ -107,7 +107,26 @@ func clonePipelineProfile(in pipeline.PipelineProfile) pipeline.PipelineProfile
|
||||
out.Input = cloneModuleBinding(in.Input)
|
||||
out.Chunk = cloneModuleBinding(in.Chunk)
|
||||
out.Output = cloneModuleBinding(in.Output)
|
||||
out.References = cloneStringMap(in.References)
|
||||
out.References = cloneReferenceSourceMap(in.References)
|
||||
if len(in.Artifacts) > 0 {
|
||||
out.Artifacts = make(map[string]pipeline.ArtifactLaneProfile, len(in.Artifacts))
|
||||
for key, lane := range in.Artifacts {
|
||||
out.Artifacts[key] = cloneArtifactLaneProfile(lane)
|
||||
}
|
||||
}
|
||||
if in.Steps != nil {
|
||||
out.Steps = make([]pipeline.PipelineStepProfile, len(in.Steps))
|
||||
for i, step := range in.Steps {
|
||||
out.Steps[i] = clonePipelineStepProfile(step)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func clonePipelineStepProfile(in pipeline.PipelineStepProfile) pipeline.PipelineStepProfile {
|
||||
out := in
|
||||
out.ID = in.ID
|
||||
out.References = cloneReferenceSourceMap(in.References)
|
||||
if len(in.Artifacts) > 0 {
|
||||
out.Artifacts = make(map[string]pipeline.ArtifactLaneProfile, len(in.Artifacts))
|
||||
for key, lane := range in.Artifacts {
|
||||
@@ -122,7 +141,7 @@ func cloneArtifactLaneProfile(in pipeline.ArtifactLaneProfile) pipeline.Artifact
|
||||
out.Extract = cloneModuleBinding(in.Extract)
|
||||
out.Merge = cloneModuleBinding(in.Merge)
|
||||
out.Normalize = cloneModuleBinding(in.Normalize)
|
||||
out.References = cloneStringMap(in.References)
|
||||
out.References = cloneReferenceSourceMap(in.References)
|
||||
if len(in.Validators) > 0 {
|
||||
out.Validators = make([]pipeline.ModuleBinding, len(in.Validators))
|
||||
for i, binding := range in.Validators {
|
||||
@@ -143,12 +162,32 @@ func cloneStringMap(in map[string]string) map[string]string {
|
||||
return out
|
||||
}
|
||||
|
||||
func cloneReferenceSourceMap(in map[string]pipeline.ReferenceSource) map[string]pipeline.ReferenceSource {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]pipeline.ReferenceSource, len(in))
|
||||
for key, source := range in {
|
||||
out[key] = cloneReferenceSource(source)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func cloneReferenceSource(in pipeline.ReferenceSource) pipeline.ReferenceSource {
|
||||
out := in
|
||||
if in.Artifact != nil {
|
||||
artifact := *in.Artifact
|
||||
out.Artifact = &artifact
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func cloneModuleBinding(in pipeline.ModuleBinding) pipeline.ModuleBinding {
|
||||
out := in
|
||||
if len(in.Options) > 0 {
|
||||
out.Options = cloneOptions(in.Options)
|
||||
}
|
||||
out.References = cloneStringMap(in.References)
|
||||
out.References = cloneReferenceSourceMap(in.References)
|
||||
out.Validators = cloneValidatorOverride(in.Validators)
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -67,11 +67,17 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
||||
|
||||
func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) {
|
||||
profile.Chunk.LLMProfile = profileID
|
||||
for laneID, lane := range profile.Artifacts {
|
||||
lane.Extract.LLMProfile = profileID
|
||||
lane.Merge.LLMProfile = profileID
|
||||
lane.Normalize.LLMProfile = profileID
|
||||
profile.Artifacts[laneID] = lane
|
||||
apply := func(artifacts map[string]pipeline.ArtifactLaneProfile) {
|
||||
for laneID, lane := range artifacts {
|
||||
lane.Extract.LLMProfile = profileID
|
||||
lane.Merge.LLMProfile = profileID
|
||||
lane.Normalize.LLMProfile = profileID
|
||||
artifacts[laneID] = lane
|
||||
}
|
||||
}
|
||||
apply(profile.Artifacts)
|
||||
for index := range profile.Steps {
|
||||
apply(profile.Steps[index].Artifacts)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -53,8 +53,8 @@ func TestEffectiveConfigOnlySelectsRequestedLanesWithoutMutatingSource(t *testin
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve() error = %v", err)
|
||||
}
|
||||
if len(effective.ResolvedPipeline.ArtifactLanes) != 1 || effective.ResolvedPipeline.ArtifactLanes[0].ID != "other" {
|
||||
t.Fatalf("resolved lanes = %#v", effective.ResolvedPipeline.ArtifactLanes)
|
||||
if len(effective.ResolvedPipeline.Steps[0].ArtifactLanes) != 1 || effective.ResolvedPipeline.Steps[0].ArtifactLanes[0].ID != "other" {
|
||||
t.Fatalf("resolved lanes = %#v", effective.ResolvedPipeline.Steps[0].ArtifactLanes)
|
||||
}
|
||||
if len(cfg.Pipelines["main"].Artifacts) != 2 {
|
||||
t.Fatalf("source lanes were mutated: %#v", cfg.Pipelines["main"].Artifacts)
|
||||
@@ -79,8 +79,8 @@ func TestEffectiveConfigMaterializesDefaultBindingsThroughCatalog(t *testing.T)
|
||||
if resolved.Chunk.Module != pipeline.DefaultChunkModule || resolved.Output.Module != pipeline.DefaultOutputModule {
|
||||
t.Fatalf("default pipeline bindings = %#v, %#v", resolved.Chunk, resolved.Output)
|
||||
}
|
||||
if len(resolved.ArtifactLanes) != 1 || resolved.ArtifactLanes[0].Merge.Module != pipeline.DefaultMergeModule || resolved.ArtifactLanes[0].Normalize.Module != pipeline.DefaultNormalizeModule {
|
||||
t.Fatalf("default lane bindings = %#v", resolved.ArtifactLanes)
|
||||
if len(resolved.Steps[0].ArtifactLanes) != 1 || resolved.Steps[0].ArtifactLanes[0].Merge.Module != pipeline.DefaultMergeModule || resolved.Steps[0].ArtifactLanes[0].Normalize.Module != pipeline.DefaultNormalizeModule {
|
||||
t.Fatalf("default lane bindings = %#v", resolved.Steps[0].ArtifactLanes)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -182,8 +182,8 @@ func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidato
|
||||
t.Fatal("LLM profile override did not change the pipeline digest")
|
||||
}
|
||||
resolved := overridden.ResolvedPipeline
|
||||
if resolved.Chunk.LLMProfile != "override-profile" || resolved.ArtifactLanes[0].Extract.LLMProfile != "override-profile" ||
|
||||
resolved.ArtifactLanes[0].Merge.LLMProfile != "override-profile" || resolved.ArtifactLanes[0].Normalize.LLMProfile != "override-profile" {
|
||||
if resolved.Chunk.LLMProfile != "override-profile" || resolved.Steps[0].ArtifactLanes[0].Extract.LLMProfile != "override-profile" ||
|
||||
resolved.Steps[0].ArtifactLanes[0].Merge.LLMProfile != "override-profile" || resolved.Steps[0].ArtifactLanes[0].Normalize.LLMProfile != "override-profile" {
|
||||
t.Fatalf("pipeline profile override was not applied: %#v", resolved)
|
||||
}
|
||||
validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane")
|
||||
@@ -249,7 +249,7 @@ func TestEffectiveConfigValidatorOverridesRemainDistinctAndOrdered(t *testing.T)
|
||||
func TestEffectiveConfigAndResolutionInputsDoNotAliasSource(t *testing.T) {
|
||||
profile := effectiveProfile()
|
||||
profile.Chunk.Options = map[string]any{"nested": map[string]any{"safe": "source"}}
|
||||
profile.Chunk.References = map[string]string{"chunk-ref": "chunk.txt"}
|
||||
profile.Chunk.References = pipeline.ExternalReferenceMap(map[string]string{"chunk-ref": "chunk.txt"})
|
||||
lane := profile.Artifacts["lane"]
|
||||
lane.Extract.Validators = pipeline.ValidatorOverride{
|
||||
Set: true,
|
||||
@@ -282,7 +282,7 @@ func TestEffectiveConfigAndResolutionInputsDoNotAliasSource(t *testing.T) {
|
||||
if got := cfg.Pipelines["main"].Chunk.Options["nested"].(map[string]any)["safe"]; got != "source" {
|
||||
t.Fatalf("source config option was aliased: %v", got)
|
||||
}
|
||||
if got := cfg.Pipelines["main"].Chunk.References["chunk-ref"]; got != "chunk.txt" {
|
||||
if got := cfg.Pipelines["main"].Chunk.References["chunk-ref"].Path; got != "chunk.txt" {
|
||||
t.Fatalf("source config references were aliased: %v", got)
|
||||
}
|
||||
if only[0] != "lane" || overrides[0].Source != "source.txt" {
|
||||
|
||||
@@ -28,19 +28,88 @@ type FileScriptoriumConfig struct {
|
||||
}
|
||||
|
||||
type FilePipelineProfile struct {
|
||||
Input fileModuleBinding `yaml:"input"`
|
||||
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
|
||||
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
|
||||
Output *fileModuleBinding `yaml:"output,omitempty"`
|
||||
References map[string]string `yaml:"references,omitempty"`
|
||||
Input fileModuleBinding `yaml:"input"`
|
||||
Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
|
||||
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
|
||||
Steps []FilePipelineStepProfile `yaml:"steps,omitempty"`
|
||||
Output *fileModuleBinding `yaml:"output,omitempty"`
|
||||
References map[string]fileReferenceSource `yaml:"references,omitempty"`
|
||||
artifactsSet bool `yaml:"-"`
|
||||
stepsSet bool `yaml:"-"`
|
||||
}
|
||||
|
||||
func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
|
||||
type plainFilePipelineProfile FilePipelineProfile
|
||||
var decoded plainFilePipelineProfile
|
||||
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
|
||||
"input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
|
||||
}, "pipeline profile")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
*p = FilePipelineProfile(decoded)
|
||||
_, p.artifactsSet = seen["artifacts"]
|
||||
_, p.stepsSet = seen["steps"]
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *FilePipelineStepProfile) UnmarshalYAML(node *yaml.Node) error {
|
||||
type plainFilePipelineStepProfile FilePipelineStepProfile
|
||||
var decoded plainFilePipelineStepProfile
|
||||
if _, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
|
||||
"id": {}, "artifacts": {}, "references": {},
|
||||
}, "pipeline step"); err != nil {
|
||||
return err
|
||||
}
|
||||
*s = FilePipelineStepProfile(decoded)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (l *FileArtifactLaneProfile) UnmarshalYAML(node *yaml.Node) error {
|
||||
type plainFileArtifactLaneProfile FileArtifactLaneProfile
|
||||
var decoded plainFileArtifactLaneProfile
|
||||
if _, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
|
||||
"extract": {}, "merge": {}, "normalize": {}, "validators": {}, "references": {},
|
||||
}, "artifact lane"); err != nil {
|
||||
return err
|
||||
}
|
||||
*l = FileArtifactLaneProfile(decoded)
|
||||
return nil
|
||||
}
|
||||
|
||||
func decodeKnownMapping(node *yaml.Node, target any, allowed map[string]struct{}, context string) (map[string]struct{}, error) {
|
||||
if node.Kind != yaml.MappingNode {
|
||||
return nil, fmt.Errorf("%s must be an object", context)
|
||||
}
|
||||
if err := node.Decode(target); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
seen := make(map[string]struct{}, len(node.Content)/2)
|
||||
for i := 0; i < len(node.Content); i += 2 {
|
||||
key := node.Content[i].Value
|
||||
if _, exists := seen[key]; exists {
|
||||
return nil, fmt.Errorf("%s field %q is duplicated", context, key)
|
||||
}
|
||||
if _, ok := allowed[key]; !ok {
|
||||
return nil, fmt.Errorf("field %s not found in %s", key, context)
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
}
|
||||
return seen, nil
|
||||
}
|
||||
|
||||
type FilePipelineStepProfile struct {
|
||||
ID string `yaml:"id"`
|
||||
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts"`
|
||||
References map[string]fileReferenceSource `yaml:"references,omitempty"`
|
||||
}
|
||||
|
||||
type FileArtifactLaneProfile struct {
|
||||
Extract fileModuleBinding `yaml:"extract"`
|
||||
Merge *fileModuleBinding `yaml:"merge,omitempty"`
|
||||
Normalize *fileModuleBinding `yaml:"normalize,omitempty"`
|
||||
Validators []fileModuleBinding `yaml:"validators,omitempty"`
|
||||
References map[string]string `yaml:"references,omitempty"`
|
||||
Extract fileModuleBinding `yaml:"extract"`
|
||||
Merge *fileModuleBinding `yaml:"merge,omitempty"`
|
||||
Normalize *fileModuleBinding `yaml:"normalize,omitempty"`
|
||||
Validators []fileModuleBinding `yaml:"validators,omitempty"`
|
||||
References map[string]fileReferenceSource `yaml:"references,omitempty"`
|
||||
}
|
||||
|
||||
type FileConcurrencyConfig struct {
|
||||
@@ -72,10 +141,90 @@ type fileModuleBinding struct {
|
||||
LLMProfile string
|
||||
Retries int
|
||||
Options map[string]any
|
||||
References map[string]string
|
||||
References map[string]fileReferenceSource
|
||||
Validators pipeline.ValidatorOverride
|
||||
}
|
||||
|
||||
type fileReferenceSource struct {
|
||||
path string
|
||||
artifact *pipeline.ArtifactReference
|
||||
}
|
||||
|
||||
func (source *fileReferenceSource) UnmarshalYAML(node *yaml.Node) error {
|
||||
if source == nil {
|
||||
return fmt.Errorf("reference source must not be nil")
|
||||
}
|
||||
switch node.Kind {
|
||||
case yaml.ScalarNode:
|
||||
if node.Tag != "!!str" {
|
||||
return fmt.Errorf("external reference path must be a string")
|
||||
}
|
||||
path := strings.TrimSpace(node.Value)
|
||||
if path == "" {
|
||||
return fmt.Errorf("external reference path must not be empty")
|
||||
}
|
||||
source.path = path
|
||||
source.artifact = nil
|
||||
return nil
|
||||
case yaml.MappingNode:
|
||||
if len(node.Content) != 2 || node.Content[0].Value != "artifact" {
|
||||
return fmt.Errorf("reference source mapping must contain only artifact")
|
||||
}
|
||||
artifactNode := node.Content[1]
|
||||
if artifactNode.Kind != yaml.MappingNode {
|
||||
return fmt.Errorf("artifact reference must be an object")
|
||||
}
|
||||
var step, lane string
|
||||
seen := map[string]bool{}
|
||||
for i := 0; i < len(artifactNode.Content); i += 2 {
|
||||
key := artifactNode.Content[i].Value
|
||||
value := artifactNode.Content[i+1]
|
||||
if seen[key] {
|
||||
return fmt.Errorf("artifact reference field %q is duplicated", key)
|
||||
}
|
||||
seen[key] = true
|
||||
if value.Tag != "!!str" {
|
||||
return fmt.Errorf("artifact reference field %q must be a string", key)
|
||||
}
|
||||
switch key {
|
||||
case "step":
|
||||
step = strings.TrimSpace(value.Value)
|
||||
case "lane":
|
||||
lane = strings.TrimSpace(value.Value)
|
||||
default:
|
||||
return fmt.Errorf("field %s not found in artifact reference", key)
|
||||
}
|
||||
}
|
||||
if step == "" || lane == "" {
|
||||
return fmt.Errorf("artifact reference step and lane must not be empty")
|
||||
}
|
||||
source.path = ""
|
||||
source.artifact = &pipeline.ArtifactReference{Step: step, Lane: lane}
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("reference source must be a string or object")
|
||||
}
|
||||
}
|
||||
|
||||
func (source fileReferenceSource) toPipelineSource() pipeline.ReferenceSource {
|
||||
if source.artifact != nil {
|
||||
artifact := *source.artifact
|
||||
return pipeline.ReferenceSource{Artifact: &artifact}
|
||||
}
|
||||
return pipeline.ExternalReference(source.path)
|
||||
}
|
||||
|
||||
func fileReferenceSourcesToPipeline(values map[string]fileReferenceSource) map[string]pipeline.ReferenceSource {
|
||||
if len(values) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]pipeline.ReferenceSource, len(values))
|
||||
for key, value := range values {
|
||||
out[strings.TrimSpace(key)] = value.toPipelineSource()
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
|
||||
switch node.Kind {
|
||||
case yaml.ScalarNode:
|
||||
@@ -115,7 +264,7 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
|
||||
}
|
||||
b.Options = normalizeOptions(options)
|
||||
case "references":
|
||||
var references map[string]string
|
||||
var references map[string]fileReferenceSource
|
||||
if err := valueNode.Decode(&references); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -146,7 +295,7 @@ func (b fileModuleBinding) toPipelineBinding() pipeline.ModuleBinding {
|
||||
LLMProfile: strings.TrimSpace(b.LLMProfile),
|
||||
Retries: b.Retries,
|
||||
Options: cloneOptions(b.Options),
|
||||
References: normalizedStringMap(b.References),
|
||||
References: fileReferenceSourcesToPipeline(b.References),
|
||||
Validators: b.Validators,
|
||||
}
|
||||
}
|
||||
@@ -214,10 +363,49 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
||||
}
|
||||
for _, pipelineID := range pipelineIDs {
|
||||
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
|
||||
hasArtifacts := filePipeline.artifactsSet || filePipeline.Artifacts != nil
|
||||
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
|
||||
if hasArtifacts && hasSteps {
|
||||
return fmt.Errorf("pipeline %q must not declare both artifacts and steps", pipelineID)
|
||||
}
|
||||
if hasSteps && len(filePipeline.Steps) == 0 {
|
||||
return fmt.Errorf("pipeline %q must declare at least one ordered step", pipelineID)
|
||||
}
|
||||
if hasSteps {
|
||||
seenSteps := make(map[string]struct{}, len(filePipeline.Steps))
|
||||
seenLanes := make(map[string]struct{})
|
||||
for index, step := range filePipeline.Steps {
|
||||
stepID := strings.TrimSpace(step.ID)
|
||||
if stepID == "" {
|
||||
return fmt.Errorf("pipeline %q step[%d] id must not be empty", pipelineID, index)
|
||||
}
|
||||
if _, ok := seenSteps[stepID]; ok {
|
||||
return fmt.Errorf("pipeline %q step id %q is duplicated after trimming", pipelineID, stepID)
|
||||
}
|
||||
seenSteps[stepID] = struct{}{}
|
||||
laneIDs, rawLaneIDs, err := normalizedMapKeys(step.Artifacts, fmt.Sprintf("pipeline %q step %q artifact lane id", pipelineID, stepID))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, laneID := range laneIDs {
|
||||
if _, ok := seenLanes[laneID]; ok {
|
||||
return fmt.Errorf("pipeline %q artifact lane id %q is duplicated across steps", pipelineID, laneID)
|
||||
}
|
||||
seenLanes[laneID] = struct{}{}
|
||||
fileLane := step.Artifacts[rawLaneIDs[laneID]]
|
||||
if err := validateFileLaneReferences(pipelineID, stepID, laneID, fileLane); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := validateFileReferenceSources(step.References, fmt.Sprintf("pipeline %q step %q reference slot", pipelineID, stepID)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if _, _, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID)); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, _, err := normalizedMapKeys(filePipeline.References, fmt.Sprintf("pipeline %q reference slot", pipelineID)); err != nil {
|
||||
if err := validateFileReferenceSources(filePipeline.References, fmt.Sprintf("pipeline %q reference slot", pipelineID)); err != nil {
|
||||
return err
|
||||
}
|
||||
if filePipeline.Chunk != nil {
|
||||
@@ -281,6 +469,7 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
||||
|
||||
for _, pipelineID := range pipelineIDs {
|
||||
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
|
||||
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
|
||||
laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID))
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -289,7 +478,7 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
||||
ID: pipelineID,
|
||||
Input: filePipeline.Input.toPipelineBinding(),
|
||||
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
|
||||
References: normalizedStringMap(filePipeline.References),
|
||||
References: fileReferenceSourcesToPipeline(filePipeline.References),
|
||||
}
|
||||
if filePipeline.Chunk != nil {
|
||||
profile.Chunk = filePipeline.Chunk.toPipelineBinding()
|
||||
@@ -300,10 +489,10 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
||||
for _, laneID := range laneIDs {
|
||||
fileLane := filePipeline.Artifacts[rawLaneIDs[laneID]]
|
||||
extract := fileLane.Extract.toPipelineBinding()
|
||||
extract.References = mergeStringMaps(normalizedStringMap(fileLane.References), extract.References)
|
||||
extract.References = mergeReferenceSources(fileReferenceSourcesToPipeline(fileLane.References), extract.References)
|
||||
lane := pipeline.ArtifactLaneProfile{
|
||||
Extract: extract,
|
||||
References: normalizedStringMap(fileLane.References),
|
||||
References: fileReferenceSourcesToPipeline(fileLane.References),
|
||||
}
|
||||
if fileLane.Merge != nil {
|
||||
lane.Merge = fileLane.Merge.toPipelineBinding()
|
||||
@@ -319,6 +508,42 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
||||
}
|
||||
profile.Artifacts[laneID] = lane
|
||||
}
|
||||
if hasSteps {
|
||||
profile.Artifacts = nil
|
||||
profile.Steps = make([]pipeline.PipelineStepProfile, len(filePipeline.Steps))
|
||||
for i, fileStep := range filePipeline.Steps {
|
||||
stepID := strings.TrimSpace(fileStep.ID)
|
||||
step := pipeline.PipelineStepProfile{
|
||||
ID: stepID,
|
||||
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(fileStep.Artifacts)),
|
||||
References: fileReferenceSourcesToPipeline(fileStep.References),
|
||||
}
|
||||
stepLaneIDs, stepRawLaneIDs, err := normalizedMapKeys(fileStep.Artifacts, fmt.Sprintf("pipeline %q step %q artifact lane id", pipelineID, stepID))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, laneID := range stepLaneIDs {
|
||||
fileLane := fileStep.Artifacts[stepRawLaneIDs[laneID]]
|
||||
extract := fileLane.Extract.toPipelineBinding()
|
||||
extract.References = mergeReferenceSources(fileReferenceSourcesToPipeline(fileLane.References), extract.References)
|
||||
lane := pipeline.ArtifactLaneProfile{Extract: extract, References: fileReferenceSourcesToPipeline(fileLane.References)}
|
||||
if fileLane.Merge != nil {
|
||||
lane.Merge = fileLane.Merge.toPipelineBinding()
|
||||
}
|
||||
if fileLane.Normalize != nil {
|
||||
lane.Normalize = fileLane.Normalize.toPipelineBinding()
|
||||
}
|
||||
if len(fileLane.Validators) > 0 {
|
||||
lane.Validators = make([]pipeline.ModuleBinding, len(fileLane.Validators))
|
||||
for index, validator := range fileLane.Validators {
|
||||
lane.Validators[index] = validator.toPipelineBinding()
|
||||
}
|
||||
}
|
||||
step.Artifacts[laneID] = lane
|
||||
}
|
||||
profile.Steps[i] = step
|
||||
}
|
||||
}
|
||||
c.Pipelines[pipelineID] = profile
|
||||
}
|
||||
|
||||
@@ -430,30 +655,72 @@ func normalizedMapKeys[T any](values map[string]T, keyName string) ([]string, ma
|
||||
return keys, rawByNormalized, nil
|
||||
}
|
||||
|
||||
func normalizedStringMap(values map[string]string) map[string]string {
|
||||
if len(values) == 0 {
|
||||
return nil
|
||||
func validateFileReferenceSources(values map[string]fileReferenceSource, context string) error {
|
||||
seen := make(map[string]struct{}, len(values))
|
||||
for rawSlot, source := range values {
|
||||
slot := strings.TrimSpace(rawSlot)
|
||||
if slot == "" {
|
||||
return fmt.Errorf("%s must not be empty", context)
|
||||
}
|
||||
if _, ok := seen[slot]; ok {
|
||||
return fmt.Errorf("%s %q is duplicated after trimming", context, slot)
|
||||
}
|
||||
seen[slot] = struct{}{}
|
||||
if source.artifact != nil {
|
||||
if strings.TrimSpace(source.artifact.Step) == "" || strings.TrimSpace(source.artifact.Lane) == "" {
|
||||
return fmt.Errorf("%s %q artifact selector step and lane must not be empty", context, slot)
|
||||
}
|
||||
if strings.TrimSpace(source.path) != "" {
|
||||
return fmt.Errorf("%s %q must contain either an external path or artifact selector", context, slot)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if strings.TrimSpace(source.path) == "" {
|
||||
return fmt.Errorf("%s %q source must not be empty", context, slot)
|
||||
}
|
||||
}
|
||||
out := make(map[string]string, len(values))
|
||||
keys := make([]string, 0, len(values))
|
||||
rawByNormalized := make(map[string]string, len(values))
|
||||
for rawKey := range values {
|
||||
key := strings.TrimSpace(rawKey)
|
||||
rawByNormalized[key] = rawKey
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, key := range keys {
|
||||
out[key] = strings.TrimSpace(values[rawByNormalized[key]])
|
||||
}
|
||||
return out
|
||||
return nil
|
||||
}
|
||||
|
||||
func mergeStringMaps(base map[string]string, override map[string]string) map[string]string {
|
||||
func validateFileLaneReferences(pipelineID, stepID, laneID string, lane FileArtifactLaneProfile) error {
|
||||
prefix := fmt.Sprintf("pipeline %q step %q lane %q", pipelineID, stepID, laneID)
|
||||
references := []struct {
|
||||
label string
|
||||
values map[string]fileReferenceSource
|
||||
}{
|
||||
{label: "reference slot", values: lane.References},
|
||||
{label: "extract reference slot", values: lane.Extract.References},
|
||||
}
|
||||
if lane.Merge != nil {
|
||||
references = append(references, struct {
|
||||
label string
|
||||
values map[string]fileReferenceSource
|
||||
}{label: "merge reference slot", values: lane.Merge.References})
|
||||
}
|
||||
if lane.Normalize != nil {
|
||||
references = append(references, struct {
|
||||
label string
|
||||
values map[string]fileReferenceSource
|
||||
}{label: "normalize reference slot", values: lane.Normalize.References})
|
||||
}
|
||||
for _, item := range references {
|
||||
if err := validateFileReferenceSources(item.values, prefix+" "+item.label); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for index, validator := range lane.Validators {
|
||||
if err := validateFileReferenceSources(validator.References, fmt.Sprintf("%s validator[%d] reference slot", prefix, index)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func mergeReferenceSources(base map[string]pipeline.ReferenceSource, override map[string]pipeline.ReferenceSource) map[string]pipeline.ReferenceSource {
|
||||
if len(base) == 0 && len(override) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]string, len(base)+len(override))
|
||||
out := make(map[string]pipeline.ReferenceSource, len(base)+len(override))
|
||||
for key, value := range base {
|
||||
out[key] = value
|
||||
}
|
||||
|
||||
@@ -142,7 +142,7 @@ pipelines:
|
||||
}
|
||||
if profile.Chunk.Module != "generic" || profile.Chunk.LLMProfile != "chunk-profile" || profile.Chunk.Retries != 2 ||
|
||||
!reflect.DeepEqual(profile.Chunk.Options, map[string]any{"max_units": 25}) ||
|
||||
!reflect.DeepEqual(profile.Chunk.References, map[string]string{"glossary": "./glossary.md"}) {
|
||||
!reflect.DeepEqual(profile.Chunk.References, pipeline.ExternalReferenceMap(map[string]string{"glossary": "./glossary.md"})) {
|
||||
t.Fatalf("object binding = %#v", profile.Chunk)
|
||||
}
|
||||
if !profile.Chunk.Validators.Set || len(profile.Chunk.Validators.Validators) != 0 {
|
||||
@@ -192,33 +192,33 @@ pipelines:
|
||||
normalize-only: ./normalize.txt
|
||||
`)
|
||||
profile := cfg.Pipelines["main"]
|
||||
if !reflect.DeepEqual(profile.References, map[string]string{
|
||||
if !reflect.DeepEqual(profile.References, pipeline.ExternalReferenceMap(map[string]string{
|
||||
"pipeline-only": "./pipeline.txt",
|
||||
"shared": "./pipeline-shared.txt",
|
||||
}) {
|
||||
})) {
|
||||
t.Fatalf("pipeline references = %#v", profile.References)
|
||||
}
|
||||
if !reflect.DeepEqual(profile.Chunk.References, map[string]string{"chunk-only": "./chunk.txt"}) {
|
||||
if !reflect.DeepEqual(profile.Chunk.References, pipeline.ExternalReferenceMap(map[string]string{"chunk-only": "./chunk.txt"})) {
|
||||
t.Fatalf("chunk references = %#v", profile.Chunk.References)
|
||||
}
|
||||
lane := profile.Artifacts["spells"]
|
||||
if !reflect.DeepEqual(lane.References, map[string]string{
|
||||
if !reflect.DeepEqual(lane.References, pipeline.ExternalReferenceMap(map[string]string{
|
||||
"lane-only": "./lane.txt",
|
||||
"shared": "./lane-shared.txt",
|
||||
"overridden": "./lane.txt",
|
||||
}) {
|
||||
})) {
|
||||
t.Fatalf("lane compatibility references = %#v", lane.References)
|
||||
}
|
||||
if !reflect.DeepEqual(lane.Extract.References, map[string]string{
|
||||
if !reflect.DeepEqual(lane.Extract.References, pipeline.ExternalReferenceMap(map[string]string{
|
||||
"lane-only": "./lane.txt",
|
||||
"shared": "./lane-shared.txt",
|
||||
"overridden": "./extract-overridden.txt",
|
||||
"extract-only": "./extract.txt",
|
||||
}) {
|
||||
})) {
|
||||
t.Fatalf("extract references = %#v", lane.Extract.References)
|
||||
}
|
||||
if !reflect.DeepEqual(lane.Merge.References, map[string]string{"merge-only": "./merge.txt"}) ||
|
||||
!reflect.DeepEqual(lane.Normalize.References, map[string]string{"normalize-only": "./normalize.txt"}) {
|
||||
if !reflect.DeepEqual(lane.Merge.References, pipeline.ExternalReferenceMap(map[string]string{"merge-only": "./merge.txt"})) ||
|
||||
!reflect.DeepEqual(lane.Normalize.References, pipeline.ExternalReferenceMap(map[string]string{"normalize-only": "./normalize.txt"})) {
|
||||
t.Fatalf("merge/normalize references = %#v, %#v", lane.Merge.References, lane.Normalize.References)
|
||||
}
|
||||
}
|
||||
@@ -357,6 +357,82 @@ func TestFileConfigRejectsTrimmedKeyCollisions(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileConfigParsesOrderedStepsAndReferenceSources(t *testing.T) {
|
||||
file := parseFileConfig(t, `version: 3
|
||||
pipelines:
|
||||
session:
|
||||
input: seriatim
|
||||
steps:
|
||||
- id: identify-npcs
|
||||
artifacts:
|
||||
npcs:
|
||||
extract: dnd/npcs
|
||||
- id: grounded-events
|
||||
references:
|
||||
npcs:
|
||||
artifact:
|
||||
step: identify-npcs
|
||||
lane: npcs
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
`)
|
||||
cfg := Default()
|
||||
if err := cfg.ApplyFileConfig(file); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
profile := cfg.Pipelines["session"]
|
||||
if len(profile.Steps) != 2 || profile.Steps[0].ID != "identify-npcs" || profile.Steps[1].ID != "grounded-events" {
|
||||
t.Fatalf("steps = %#v", profile.Steps)
|
||||
}
|
||||
source := profile.Steps[1].References["npcs"]
|
||||
if source.Artifact == nil || source.Artifact.Step != "identify-npcs" || source.Artifact.Lane != "npcs" {
|
||||
t.Fatalf("generated source = %#v", source)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileConfigRejectsAmbiguousReferenceSourceForms(t *testing.T) {
|
||||
for _, source := range []string{
|
||||
"artifact: {step: a, lane: b, extra: c}",
|
||||
"artifact: {step: 1, lane: b}",
|
||||
"1",
|
||||
} {
|
||||
_, err := ParseFileConfigYAML([]byte("version: 3\npipelines:\n p:\n input: text\n references:\n slot: " + source + "\n"))
|
||||
if err == nil {
|
||||
t.Fatalf("ParseFileConfigYAML(%q) error = nil", source)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileConfigRejectsEmptyAndAmbiguousPipelineShapes(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
yaml string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "empty steps",
|
||||
yaml: "version: 3\npipelines:\n p:\n input: text\n steps: []\n",
|
||||
want: "at least one ordered step",
|
||||
},
|
||||
{
|
||||
name: "both forms",
|
||||
yaml: "version: 3\npipelines:\n p:\n input: text\n artifacts: {}\n steps: []\n",
|
||||
want: "both artifacts and steps",
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
file := parseFileConfig(t, tt.yaml)
|
||||
cfg := Default()
|
||||
err := cfg.ApplyFileConfig(file)
|
||||
if err == nil || !strings.Contains(err.Error(), tt.want) {
|
||||
t.Fatalf("ApplyFileConfig() error = %v, want context %q", err, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFileConfigReportsPathAndOperationContext(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
missing := filepath.Join(dir, "missing.yml")
|
||||
|
||||
@@ -42,10 +42,16 @@ func cloneResolvedPipeline(in pipeline.ResolvedPipeline) pipeline.ResolvedPipeli
|
||||
out.ValidatorChains[i] = cloneResolvedValidatorChain(chain)
|
||||
}
|
||||
}
|
||||
if len(in.ArtifactLanes) > 0 {
|
||||
out.ArtifactLanes = make([]pipeline.ResolvedArtifactLane, len(in.ArtifactLanes))
|
||||
for i, lane := range in.ArtifactLanes {
|
||||
out.ArtifactLanes[i] = cloneResolvedArtifactLane(lane)
|
||||
if len(in.Steps) > 0 {
|
||||
out.Steps = make([]pipeline.ResolvedPipelineStep, len(in.Steps))
|
||||
for i, step := range in.Steps {
|
||||
out.Steps[i] = pipeline.ResolvedPipelineStep{ID: step.ID}
|
||||
if len(step.ArtifactLanes) > 0 {
|
||||
out.Steps[i].ArtifactLanes = make([]pipeline.ResolvedArtifactLane, len(step.ArtifactLanes))
|
||||
for j, lane := range step.ArtifactLanes {
|
||||
out.Steps[i].ArtifactLanes[j] = cloneResolvedArtifactLane(lane)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
@@ -89,14 +95,20 @@ func redactConfig(cfg Config) Config {
|
||||
profile.Input = redactBinding(profile.Input)
|
||||
profile.Chunk = redactBinding(profile.Chunk)
|
||||
profile.Output = redactBinding(profile.Output)
|
||||
for laneID, lane := range profile.Artifacts {
|
||||
lane.Extract = redactBinding(lane.Extract)
|
||||
lane.Merge = redactBinding(lane.Merge)
|
||||
lane.Normalize = redactBinding(lane.Normalize)
|
||||
for i := range lane.Validators {
|
||||
lane.Validators[i] = redactBinding(lane.Validators[i])
|
||||
redactLanes := func(lanes map[string]pipeline.ArtifactLaneProfile) {
|
||||
for laneID, lane := range lanes {
|
||||
lane.Extract = redactBinding(lane.Extract)
|
||||
lane.Merge = redactBinding(lane.Merge)
|
||||
lane.Normalize = redactBinding(lane.Normalize)
|
||||
for i := range lane.Validators {
|
||||
lane.Validators[i] = redactBinding(lane.Validators[i])
|
||||
}
|
||||
lanes[laneID] = lane
|
||||
}
|
||||
profile.Artifacts[laneID] = lane
|
||||
}
|
||||
redactLanes(profile.Artifacts)
|
||||
for i := range profile.Steps {
|
||||
redactLanes(profile.Steps[i].Artifacts)
|
||||
}
|
||||
cfg.Pipelines[id] = profile
|
||||
}
|
||||
|
||||
@@ -24,16 +24,19 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
|
||||
Input: bindings["input"],
|
||||
Chunk: bindings["chunk"],
|
||||
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
|
||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
||||
ID: "safe-lane",
|
||||
ArtifactKind: "safe/artifact",
|
||||
Extract: bindings["extract"],
|
||||
Merge: bindings["merge"],
|
||||
Normalize: bindings["normalize"],
|
||||
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
|
||||
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
|
||||
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
|
||||
NormalizeReferences: redactionTestReferenceTarget(pipeline.StageNormalize, "safe-lane", "normalize-reference-content"),
|
||||
Steps: []pipeline.ResolvedPipelineStep{{
|
||||
ID: "default",
|
||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
||||
ID: "safe-lane",
|
||||
ArtifactKind: "safe/artifact",
|
||||
Extract: bindings["extract"],
|
||||
Merge: bindings["merge"],
|
||||
Normalize: bindings["normalize"],
|
||||
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
|
||||
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
|
||||
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
|
||||
NormalizeReferences: redactionTestReferenceTarget(pipeline.StageNormalize, "safe-lane", "normalize-reference-content"),
|
||||
}},
|
||||
}},
|
||||
ValidatorChains: []pipeline.ResolvedValidatorChain{{
|
||||
Stage: pipeline.StageExtract,
|
||||
@@ -142,12 +145,15 @@ func TestRedactedSummaryPayloadsCoverEveryEffectiveConfigBinding(t *testing.T) {
|
||||
Input: bindings["input"],
|
||||
Chunk: bindings["chunk"],
|
||||
Output: bindings["output"],
|
||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
||||
ID: "safe-lane",
|
||||
Extract: bindings["extract"],
|
||||
Merge: bindings["merge"],
|
||||
Normalize: bindings["normalize"],
|
||||
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
|
||||
Steps: []pipeline.ResolvedPipelineStep{{
|
||||
ID: "default",
|
||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
|
||||
ID: "safe-lane",
|
||||
Extract: bindings["extract"],
|
||||
Merge: bindings["merge"],
|
||||
Normalize: bindings["normalize"],
|
||||
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
|
||||
}},
|
||||
}},
|
||||
},
|
||||
}
|
||||
|
||||
@@ -106,30 +106,60 @@ func validatePipelineProfiles(profiles map[string]pipeline.PipelineProfile) erro
|
||||
if err := validateReferenceMap(id, "", profile.References); err != nil {
|
||||
return err
|
||||
}
|
||||
seenLanes := make(map[string]struct{}, len(profile.Artifacts))
|
||||
for rawLaneID, lane := range profile.Artifacts {
|
||||
laneID := strings.TrimSpace(rawLaneID)
|
||||
if laneID == "" {
|
||||
return fmt.Errorf("pipeline %q artifact lane id must not be empty", id)
|
||||
explicitSteps := profile.Steps != nil
|
||||
if len(profile.Artifacts) > 0 && explicitSteps {
|
||||
return fmt.Errorf("pipeline %q must not declare both artifacts and steps", id)
|
||||
}
|
||||
steps := profile.Steps
|
||||
if !explicitSteps {
|
||||
steps = []pipeline.PipelineStepProfile{{ID: "default", Artifacts: profile.Artifacts}}
|
||||
}
|
||||
if explicitSteps && len(steps) == 0 {
|
||||
return fmt.Errorf("pipeline %q must declare at least one ordered step", id)
|
||||
}
|
||||
seenSteps := make(map[string]struct{}, len(steps))
|
||||
seenLanes := make(map[string]struct{})
|
||||
for index, step := range steps {
|
||||
stepID := strings.TrimSpace(step.ID)
|
||||
if stepID == "" {
|
||||
return fmt.Errorf("pipeline %q step[%d] id must not be empty", id, index)
|
||||
}
|
||||
if _, ok := seenLanes[laneID]; ok {
|
||||
return fmt.Errorf("pipeline %q artifact lane id %q is duplicated after trimming", id, laneID)
|
||||
if _, ok := seenSteps[stepID]; ok {
|
||||
return fmt.Errorf("pipeline %q step id %q is duplicated after trimming", id, stepID)
|
||||
}
|
||||
seenLanes[laneID] = struct{}{}
|
||||
if err := validateReferenceMap(id, laneID, lane.References); err != nil {
|
||||
return err
|
||||
seenSteps[stepID] = struct{}{}
|
||||
if explicitSteps {
|
||||
if err := validateReferenceMapForContext(id, "", "step "+stepID, step.References, true); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := validateBinding(id, laneID, "extract", lane.Extract, true); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateBinding(id, laneID, "merge", lane.Merge, true); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateBinding(id, laneID, "normalize", lane.Normalize, true); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(lane.Validators) > 0 {
|
||||
return fmt.Errorf("pipeline %q lane %q validators are not supported at artifact lane level; use extract.validators, merge.validators, or normalize.validators", id, laneID)
|
||||
for rawLaneID, lane := range step.Artifacts {
|
||||
laneID := strings.TrimSpace(rawLaneID)
|
||||
if laneID == "" {
|
||||
return fmt.Errorf("pipeline %q artifact lane id must not be empty", id)
|
||||
}
|
||||
if _, ok := seenLanes[laneID]; ok {
|
||||
if !explicitSteps {
|
||||
return fmt.Errorf("pipeline %q artifact lane id %q is duplicated after trimming", id, laneID)
|
||||
}
|
||||
return fmt.Errorf("pipeline %q artifact lane id %q is duplicated across steps", id, laneID)
|
||||
}
|
||||
seenLanes[laneID] = struct{}{}
|
||||
if err := validateReferenceMapForContext(id, laneID, "", lane.References, true); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateBinding(id, laneID, "extract", lane.Extract, true); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateBinding(id, laneID, "merge", lane.Merge, true); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateBinding(id, laneID, "normalize", lane.Normalize, true); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(lane.Validators) > 0 {
|
||||
return fmt.Errorf("pipeline %q lane %q validators are not supported at artifact lane level; use extract.validators, merge.validators, or normalize.validators", id, laneID)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -164,7 +194,7 @@ func validateBinding(
|
||||
}
|
||||
return fmt.Errorf("pipeline %q %s references are not supported", pipelineID, slot)
|
||||
}
|
||||
return validateReferenceMapForContext(pipelineID, laneID, slot, binding.References)
|
||||
return validateReferenceMapForContext(pipelineID, laneID, slot, binding.References, true)
|
||||
}
|
||||
|
||||
func validateValidatorOverride(pipelineID string, laneID string, slot string, override pipeline.ValidatorOverride) error {
|
||||
@@ -197,13 +227,13 @@ func validateValidatorOverride(pipelineID string, laneID string, slot string, ov
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateReferenceMap(pipelineID string, laneID string, references map[string]string) error {
|
||||
return validateReferenceMapForContext(pipelineID, laneID, "", references)
|
||||
func validateReferenceMap(pipelineID string, laneID string, references map[string]pipeline.ReferenceSource) error {
|
||||
return validateReferenceMapForContext(pipelineID, laneID, "", references, false)
|
||||
}
|
||||
|
||||
func validateReferenceMapForContext(pipelineID string, laneID string, slot string, references map[string]string) error {
|
||||
func validateReferenceMapForContext(pipelineID string, laneID string, slot string, references map[string]pipeline.ReferenceSource, generatedAllowed bool) error {
|
||||
seen := make(map[string]struct{}, len(references))
|
||||
for rawSlotName, rawSource := range references {
|
||||
for rawSlotName, source := range references {
|
||||
slotName := strings.TrimSpace(rawSlotName)
|
||||
if slotName == "" {
|
||||
return fmt.Errorf("%s reference slot name must not be empty", referenceContext(pipelineID, laneID, slot))
|
||||
@@ -212,7 +242,19 @@ func validateReferenceMapForContext(pipelineID string, laneID string, slot strin
|
||||
return fmt.Errorf("%s reference slot %q is duplicated after trimming", referenceContext(pipelineID, laneID, slot), slotName)
|
||||
}
|
||||
seen[slotName] = struct{}{}
|
||||
if strings.TrimSpace(rawSource) == "" {
|
||||
if source.Artifact != nil {
|
||||
if !generatedAllowed {
|
||||
return fmt.Errorf("%s reference slot %q must use an external path", referenceContext(pipelineID, laneID, slot), slotName)
|
||||
}
|
||||
if strings.TrimSpace(source.Artifact.Step) == "" || strings.TrimSpace(source.Artifact.Lane) == "" {
|
||||
return fmt.Errorf("%s reference slot %q artifact selector step and lane must not be empty", referenceContext(pipelineID, laneID, slot), slotName)
|
||||
}
|
||||
if strings.TrimSpace(source.Path) != "" {
|
||||
return fmt.Errorf("%s reference slot %q must contain exactly one source form", referenceContext(pipelineID, laneID, slot), slotName)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if strings.TrimSpace(source.Path) == "" {
|
||||
return fmt.Errorf("%s reference slot %q source must not be empty", referenceContext(pipelineID, laneID, slot), slotName)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -200,7 +200,7 @@ func TestValidateIdentifiersAfterTrimming(t *testing.T) {
|
||||
name: "empty reference slot",
|
||||
setup: func(cfg *Config) {
|
||||
profile := validationProfile()
|
||||
profile.References = map[string]string{" ": "source.txt"}
|
||||
profile.References = pipeline.ExternalReferenceMap(map[string]string{" ": "source.txt"})
|
||||
cfg.Pipelines = map[string]pipeline.PipelineProfile{"main": profile}
|
||||
},
|
||||
want: "reference slot name must not be empty",
|
||||
@@ -209,7 +209,7 @@ func TestValidateIdentifiersAfterTrimming(t *testing.T) {
|
||||
name: "duplicate reference slots",
|
||||
setup: func(cfg *Config) {
|
||||
profile := validationProfile()
|
||||
profile.References = map[string]string{"slot": "one.txt", " slot ": "two.txt"}
|
||||
profile.References = pipeline.ExternalReferenceMap(map[string]string{"slot": "one.txt", " slot ": "two.txt"})
|
||||
cfg.Pipelines = map[string]pipeline.PipelineProfile{"main": profile}
|
||||
},
|
||||
want: "reference slot \"slot\" is duplicated after trimming",
|
||||
@@ -267,14 +267,14 @@ func TestValidateReferencesAreUnsupportedOnInputAndOutput(t *testing.T) {
|
||||
{
|
||||
name: "input references",
|
||||
set: func(profile *pipeline.PipelineProfile) {
|
||||
profile.Input.References = map[string]string{"slot": "source.txt"}
|
||||
profile.Input.References = pipeline.ExternalReferenceMap(map[string]string{"slot": "source.txt"})
|
||||
},
|
||||
want: "input references are not supported",
|
||||
},
|
||||
{
|
||||
name: "output references",
|
||||
set: func(profile *pipeline.PipelineProfile) {
|
||||
profile.Output.References = map[string]string{"slot": "source.txt"}
|
||||
profile.Output.References = pipeline.ExternalReferenceMap(map[string]string{"slot": "source.txt"})
|
||||
},
|
||||
want: "output references are not supported",
|
||||
},
|
||||
@@ -326,7 +326,7 @@ func TestValidateValidatorBindingRules(t *testing.T) {
|
||||
Set: true,
|
||||
Validators: []pipeline.ModuleBinding{{
|
||||
Module: "validator",
|
||||
References: map[string]string{"slot": "source.txt"},
|
||||
References: pipeline.ExternalReferenceMap(map[string]string{"slot": "source.txt"}),
|
||||
}},
|
||||
}
|
||||
},
|
||||
|
||||
@@ -33,6 +33,7 @@ type Invocation struct {
|
||||
PipelineID string `json:"pipeline_id,omitempty"`
|
||||
PipelineDigest string `json:"pipeline_digest,omitempty"`
|
||||
Resume bool `json:"resume,omitempty"`
|
||||
RecomputeStep string `json:"recompute_step,omitempty"`
|
||||
InputPath string `json:"input_path,omitempty"`
|
||||
ConfigPath string `json:"config_path,omitempty"`
|
||||
ConfigSource string `json:"config_source,omitempty"`
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
@@ -137,7 +138,7 @@ func TestFilesystemCheckpointRejectsMissingAndCorruptState(t *testing.T) {
|
||||
if err := os.Remove(stage.manifest(fixture)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
assertStageNotReused(t, stage, fixture, "missing")
|
||||
assertStageNotReused(t, stage, fixture, pipeline.CheckpointReasonMissing)
|
||||
})
|
||||
|
||||
t.Run(stage.name+" malformed manifest", func(t *testing.T) {
|
||||
@@ -145,7 +146,7 @@ func TestFilesystemCheckpointRejectsMissingAndCorruptState(t *testing.T) {
|
||||
if err := os.WriteFile(stage.manifest(fixture), []byte("{"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
assertStageNotReused(t, stage, fixture, "decode")
|
||||
assertStageNotReused(t, stage, fixture, pipeline.CheckpointReasonDecodeFailed)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -154,17 +155,18 @@ func TestFilesystemCheckpointRejectsIncompatibleManifests(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
edit func(map[string]any)
|
||||
want string
|
||||
want pipeline.CheckpointReasonCode
|
||||
}{
|
||||
{"v1 schema", func(m map[string]any) { m["workspace_schema_version"] = WorkspaceSchemaVersionV1 }, "workspace schema"},
|
||||
{"unknown schema", func(m map[string]any) { m["workspace_schema_version"] = "notarius.workspace.future" }, "workspace schema"},
|
||||
{"identity", func(m map[string]any) { m["metadata"].(map[string]any)["checkpoint_identity_digest"] = "sha256:other" }, "identity"},
|
||||
{"stage", func(m map[string]any) { m["stage"] = string(StageMerge) }, "stage"},
|
||||
{"lane", func(m map[string]any) { m["lane_id"] = "lane-other" }, "lane"},
|
||||
{"module", func(m map[string]any) { m["module_key"] = "module-other" }, "module"},
|
||||
{"v1 schema", func(m map[string]any) { m["workspace_schema_version"] = WorkspaceSchemaVersionV1 }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
|
||||
{"v2 schema", func(m map[string]any) { m["workspace_schema_version"] = WorkspaceSchemaVersionV2 }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
|
||||
{"unknown schema", func(m map[string]any) { m["workspace_schema_version"] = "notarius.workspace.future" }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
|
||||
{"identity", func(m map[string]any) { m["metadata"].(map[string]any)["checkpoint_identity_digest"] = "sha256:other" }, pipeline.CheckpointReasonIdentityMismatch},
|
||||
{"stage", func(m map[string]any) { m["stage"] = string(StageMerge) }, pipeline.CheckpointReasonStageMismatch},
|
||||
{"lane", func(m map[string]any) { m["lane_id"] = "lane-other" }, pipeline.CheckpointReasonLaneMismatch},
|
||||
{"module", func(m map[string]any) { m["module_key"] = "module-other" }, pipeline.CheckpointReasonModuleMismatch},
|
||||
{"dependency", func(m map[string]any) {
|
||||
m["dependency_fingerprints"] = []map[string]string{{"name": "input", "value": "other"}}
|
||||
}, "dependency"},
|
||||
}, pipeline.CheckpointReasonDependencyMismatch},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
fixture := seedFilesystemCheckpoints(t)
|
||||
@@ -179,7 +181,7 @@ func TestFilesystemCheckpointRejectsNonTerminalStatuses(t *testing.T) {
|
||||
t.Run(string(status), func(t *testing.T) {
|
||||
fixture := seedFilesystemCheckpoints(t)
|
||||
editManifest(t, checkpointStages()[1].manifest(fixture), func(m map[string]any) { m["status"] = string(status) })
|
||||
assertStageNotReused(t, checkpointStages()[1], fixture, "status")
|
||||
assertStageNotReused(t, checkpointStages()[1], fixture, pipeline.CheckpointReasonStatusNotReusable)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -188,20 +190,20 @@ func TestFilesystemCheckpointRejectsIncompleteArtifactsAndContent(t *testing.T)
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
edit func(map[string]any)
|
||||
want string
|
||||
want pipeline.CheckpointReasonCode
|
||||
}{
|
||||
{"artifact kind", func(m map[string]any) { m["outputs"].([]any)[0].(map[string]any)["artifact_kind"] = "" }, "artifact codec identity"},
|
||||
{"schema id", func(m map[string]any) { m["outputs"].([]any)[0].(map[string]any)["schema"].(map[string]any)["id"] = "" }, "artifact codec identity"},
|
||||
{"artifact kind", func(m map[string]any) { m["outputs"].([]any)[0].(map[string]any)["artifact_kind"] = "" }, pipeline.CheckpointReasonArtifactCodecIncompatible},
|
||||
{"schema id", func(m map[string]any) { m["outputs"].([]any)[0].(map[string]any)["schema"].(map[string]any)["id"] = "" }, pipeline.CheckpointReasonArtifactCodecIncompatible},
|
||||
{"schema version", func(m map[string]any) {
|
||||
m["outputs"].([]any)[0].(map[string]any)["schema"].(map[string]any)["version"] = ""
|
||||
}, "artifact codec identity"},
|
||||
{"schema digest", func(m map[string]any) { m["outputs"].([]any)[0].(map[string]any)["schema_digest"] = "" }, "artifact codec identity"},
|
||||
}, pipeline.CheckpointReasonArtifactCodecIncompatible},
|
||||
{"schema digest", func(m map[string]any) { m["outputs"].([]any)[0].(map[string]any)["schema_digest"] = "" }, pipeline.CheckpointReasonArtifactCodecIncompatible},
|
||||
{"base64", func(m map[string]any) {
|
||||
m["outputs"].([]any)[0].(map[string]any)["content"].(map[string]any)["content_base64"] = "%"
|
||||
}, "base64"},
|
||||
}, pipeline.CheckpointReasonArtifactPayloadInvalid},
|
||||
{"content digest", func(m map[string]any) {
|
||||
m["outputs"].([]any)[0].(map[string]any)["content"].(map[string]any)["content_digest"] = "sha256:other"
|
||||
}, "content digest"},
|
||||
}, pipeline.CheckpointReasonArtifactDigestMismatch},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
fixture := seedFilesystemCheckpoints(t)
|
||||
@@ -217,7 +219,7 @@ func TestFilesystemCheckpointRejectsSourceAndOutputDigestMismatches(t *testing.T
|
||||
editJSON(t, filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), "source", "source-document.json"), func(m map[string]any) {
|
||||
m["document"].(map[string]any)["units"].([]any)[0].(map[string]any)["text"] = ""
|
||||
})
|
||||
assertStageNotReused(t, checkpointStages()[0], fixture, "source checkpoint document")
|
||||
assertStageNotReused(t, checkpointStages()[0], fixture, pipeline.CheckpointReasonArtifactPayloadInvalid)
|
||||
})
|
||||
|
||||
t.Run("source output digest", func(t *testing.T) {
|
||||
@@ -225,7 +227,7 @@ func TestFilesystemCheckpointRejectsSourceAndOutputDigestMismatches(t *testing.T
|
||||
editJSON(t, filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), "source", "source-document.json"), func(m map[string]any) {
|
||||
m["document"].(map[string]any)["digest"] = "sha256:other"
|
||||
})
|
||||
assertStageNotReused(t, checkpointStages()[0], fixture, "output digest")
|
||||
assertStageNotReused(t, checkpointStages()[0], fixture, pipeline.CheckpointReasonArtifactDigestMismatch)
|
||||
})
|
||||
|
||||
for _, stage := range checkpointStages()[1:] {
|
||||
@@ -234,7 +236,7 @@ func TestFilesystemCheckpointRejectsSourceAndOutputDigestMismatches(t *testing.T
|
||||
editManifest(t, stage.manifest(fixture), func(m map[string]any) {
|
||||
m["output_digests"].([]any)[0].(map[string]any)["value"] = "sha256:other"
|
||||
})
|
||||
assertStageNotReused(t, stage, fixture, "output digest")
|
||||
assertStageNotReused(t, stage, fixture, pipeline.CheckpointReasonArtifactDigestMismatch)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -247,6 +249,180 @@ func TestFilesystemCheckpointReusesExtractWithRejections(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilesystemCheckpointDependencyDecisionIsBoundedAndCategorized(t *testing.T) {
|
||||
fixture := seedFilesystemCheckpoints(t)
|
||||
_, decision := fixture.loader.Extract("lane-a", "extract-module", []pipeline.CheckpointFingerprint{{Name: "source", Value: "sha256:changed"}})
|
||||
if decision.Reused || decision.Category != "dependency_invalidated" || decision.ReasonCode != "dependency_mismatch" {
|
||||
t.Fatalf("dependency decision = %#v", decision)
|
||||
}
|
||||
if strings.Contains(decision.Reason, fixture.root) || strings.Contains(decision.Detail, fixture.root) {
|
||||
t.Fatalf("dependency decision leaked checkpoint path: %#v", decision)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilesystemCheckpointDecisionFamiliesAreStableAndSafe(t *testing.T) {
|
||||
const secretSentinel = "do-not-expose-checkpoint-secret"
|
||||
tests := []struct {
|
||||
name string
|
||||
prepare func(*testing.T, filesystemCheckpointFixture) pipeline.CheckpointDecision
|
||||
category pipeline.CheckpointDecisionCategory
|
||||
code pipeline.CheckpointReasonCode
|
||||
}{
|
||||
{"loading disabled", func(t *testing.T, _ filesystemCheckpointFixture) pipeline.CheckpointDecision {
|
||||
_, decision := pipeline.NoopCheckpointLoader().Source("source")
|
||||
return decision
|
||||
}, pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonLoadingDisabled},
|
||||
{"checkpoint unavailable", func(t *testing.T, fixture filesystemCheckpointFixture) pipeline.CheckpointDecision {
|
||||
if err := os.Remove(checkpointStages()[1].manifest(fixture)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return checkpointStages()[1].load(fixture)
|
||||
}, pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonMissing},
|
||||
{"workspace identity", func(t *testing.T, fixture filesystemCheckpointFixture) pipeline.CheckpointDecision {
|
||||
editManifest(t, checkpointStages()[1].manifest(fixture), func(m map[string]any) {
|
||||
m["workspace_schema_version"] = secretSentinel
|
||||
})
|
||||
return checkpointStages()[1].load(fixture)
|
||||
}, pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
|
||||
{"manifest scope", func(t *testing.T, fixture filesystemCheckpointFixture) pipeline.CheckpointDecision {
|
||||
editManifest(t, checkpointStages()[1].manifest(fixture), func(m map[string]any) { m["module_key"] = secretSentinel })
|
||||
return checkpointStages()[1].load(fixture)
|
||||
}, pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonModuleMismatch},
|
||||
{"dependency invalidated", func(t *testing.T, fixture filesystemCheckpointFixture) pipeline.CheckpointDecision {
|
||||
_, decision := fixture.loader.Extract("lane-a", "extract-module", []pipeline.CheckpointFingerprint{{Name: "input", Value: secretSentinel}})
|
||||
return decision
|
||||
}, pipeline.CheckpointDecisionDependencyInvalidated, pipeline.CheckpointReasonDependencyMismatch},
|
||||
{"artifact payload", func(t *testing.T, fixture filesystemCheckpointFixture) pipeline.CheckpointDecision {
|
||||
editJSON(t, filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), "extract", "lane-a", "outputs.json"), func(m map[string]any) {
|
||||
m["outputs"].([]any)[0].(map[string]any)["artifact_kind"] = ""
|
||||
m["outputs"].([]any)[0].(map[string]any)["source_id"] = secretSentinel
|
||||
})
|
||||
return checkpointStages()[1].load(fixture)
|
||||
}, pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactCodecIncompatible},
|
||||
{"checkpoint reused", func(t *testing.T, fixture filesystemCheckpointFixture) pipeline.CheckpointDecision {
|
||||
return checkpointStages()[1].load(fixture)
|
||||
}, pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonReused},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
fixture := seedFilesystemCheckpoints(t)
|
||||
decision := test.prepare(t, fixture)
|
||||
if decision.Category != test.category || decision.ReasonCode != test.code {
|
||||
t.Fatalf("decision = %#v, want category %q and code %q", decision, test.category, test.code)
|
||||
}
|
||||
if len([]byte(decision.Detail)) > 512 || !utf8.ValidString(decision.Detail) {
|
||||
t.Fatalf("decision detail is not bounded valid UTF-8: %#v", decision)
|
||||
}
|
||||
for _, forbidden := range []string{secretSentinel, fixture.root} {
|
||||
if strings.Contains(decision.Detail, forbidden) || strings.Contains(decision.Reason, forbidden) {
|
||||
t.Fatalf("decision leaked %q: %#v", forbidden, decision)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilesystemLoaderReadsAcceptedNormalizeWithoutStageDependencies(t *testing.T) {
|
||||
fixture := seedAcceptedNormalizeCheckpoint(t)
|
||||
identityRoot := filepath.Join(fixture.root, mustRelativePath(t, fixture.identity))
|
||||
for _, stage := range []string{"extract", "merge"} {
|
||||
if _, err := os.Stat(filepath.Join(identityRoot, laneManifestPath(stage, "step-1", "lane-a"))); !os.IsNotExist(err) {
|
||||
t.Fatalf("%s checkpoint stat error = %v, want absent prerequisite", stage, err)
|
||||
}
|
||||
}
|
||||
checkpoint, decision := fixture.loader.AcceptedNormalize("step-1", "lane-a", "normalize-module")
|
||||
if !decision.Reused || decision.Category != pipeline.CheckpointDecisionReused || decision.ReasonCode != pipeline.CheckpointReasonAcceptedArtifactReused {
|
||||
t.Fatalf("accepted normalize decision = %#v", decision)
|
||||
}
|
||||
if checkpoint.Output.Artifact.Content == nil || string(checkpoint.Output.Artifact.Content) != string(fixture.normalize.Artifact.Content) {
|
||||
t.Fatalf("accepted normalize output = %#v, want recorded artifact", checkpoint.Output)
|
||||
}
|
||||
if len(checkpoint.Warnings) != 1 || checkpoint.Warnings[0].ReasonCode != "normalized" {
|
||||
t.Fatalf("accepted normalize warnings = %#v", checkpoint.Warnings)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilesystemLoaderRejectsInvalidAcceptedNormalize(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*testing.T, *filesystemCheckpointFixture)
|
||||
code pipeline.CheckpointReasonCode
|
||||
}{
|
||||
{"missing", func(t *testing.T, fixture *filesystemCheckpointFixture) {
|
||||
if err := os.Remove(acceptedNormalizeManifest(t, *fixture)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}, pipeline.CheckpointReasonMissing},
|
||||
{"rejected status", func(t *testing.T, fixture *filesystemCheckpointFixture) {
|
||||
editManifest(t, acceptedNormalizeManifest(t, *fixture), func(m map[string]any) { m["status"] = string(StatusSucceededWithRejections) })
|
||||
}, pipeline.CheckpointReasonStatusNotReusable},
|
||||
{"corrupt payload", func(t *testing.T, fixture *filesystemCheckpointFixture) {
|
||||
if err := os.WriteFile(acceptedNormalizePayload(t, *fixture), []byte("{"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}, pipeline.CheckpointReasonDecodeFailed},
|
||||
{"wrong codec identity", func(t *testing.T, fixture *filesystemCheckpointFixture) {
|
||||
editJSON(t, acceptedNormalizePayload(t, *fixture), func(m map[string]any) { m["output"].(map[string]any)["artifact_kind"] = "" })
|
||||
}, pipeline.CheckpointReasonArtifactCodecIncompatible},
|
||||
{"wrong content digest", func(t *testing.T, fixture *filesystemCheckpointFixture) {
|
||||
editJSON(t, acceptedNormalizePayload(t, *fixture), func(m map[string]any) {
|
||||
m["output"].(map[string]any)["content"].(map[string]any)["content_digest"] = "sha256:wrong"
|
||||
})
|
||||
}, pipeline.CheckpointReasonArtifactDigestMismatch},
|
||||
{"unverifiable identity", func(t *testing.T, fixture *filesystemCheckpointFixture) {
|
||||
loader := fixture.loader.(*FilesystemLoader)
|
||||
fixture.loader = &FilesystemLoader{root: loader.root}
|
||||
}, pipeline.CheckpointReasonIdentityMismatch},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
fixture := seedAcceptedNormalizeCheckpoint(t)
|
||||
test.mutate(t, &fixture)
|
||||
_, decision := fixture.loader.AcceptedNormalize("step-1", "lane-a", "normalize-module")
|
||||
if decision.Reused || decision.ReasonCode != test.code {
|
||||
t.Fatalf("accepted normalize decision = %#v, want %q", decision, test.code)
|
||||
}
|
||||
if strings.Contains(decision.Detail, fixture.root) || strings.Contains(decision.Detail, string(fixture.normalize.Artifact.Content)) {
|
||||
t.Fatalf("accepted normalize decision leaked path or content: %#v", decision)
|
||||
}
|
||||
if _, err := os.Stat(acceptedNormalizePayload(t, fixture)); err != nil {
|
||||
t.Fatalf("accepted normalize payload was removed after rejection: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func seedAcceptedNormalizeCheckpoint(t *testing.T) filesystemCheckpointFixture {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
identity := testIdentity(t)
|
||||
artifact := checkpointArtifact("normalize-module", `{"accepted":true}`)
|
||||
recorder, err := NewFilesystemRecorder(root, identity)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stepRecorder := recorder.(pipeline.StepCheckpointRecorder)
|
||||
warnings := []contracts.Warning{{Scope: "normalize", ReasonCode: "normalized", Message: "normalized warning"}}
|
||||
if err := stepRecorder.NormalizeSucceededForStep("step-1", "lane-a", "normalize-module", []pipeline.CheckpointFingerprint{{Name: "merge", Value: "sha256:unavailable"}}, artifact, warnings); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
loader, err := NewFilesystemLoader(root, identity)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return filesystemCheckpointFixture{root: root, identity: identity, loader: loader, normalize: artifact}
|
||||
}
|
||||
|
||||
func acceptedNormalizeManifest(t *testing.T, fixture filesystemCheckpointFixture) string {
|
||||
t.Helper()
|
||||
return filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), laneManifestPath("normalize", "step-1", "lane-a"))
|
||||
}
|
||||
|
||||
func acceptedNormalizePayload(t *testing.T, fixture filesystemCheckpointFixture) string {
|
||||
t.Helper()
|
||||
return filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), lanePayloadPath("normalize", "step-1", "lane-a", "output.json"))
|
||||
}
|
||||
|
||||
type checkpointStage struct {
|
||||
name string
|
||||
manifest func(filesystemCheckpointFixture) string
|
||||
@@ -355,11 +531,11 @@ func checkpointArtifact(module, content string) pipeline.CheckpointArtifact {
|
||||
}
|
||||
}
|
||||
|
||||
func assertStageNotReused(t *testing.T, stage checkpointStage, fixture filesystemCheckpointFixture, want string) {
|
||||
func assertStageNotReused(t *testing.T, stage checkpointStage, fixture filesystemCheckpointFixture, want pipeline.CheckpointReasonCode) {
|
||||
t.Helper()
|
||||
decision := stage.load(fixture)
|
||||
if decision.Reused || !strings.Contains(strings.ToLower(decision.Reason), strings.ToLower(want)) {
|
||||
t.Fatalf("%s decision=%#v, want non-reused reason containing %q", stage.name, decision, want)
|
||||
if decision.Reused || decision.ReasonCode != want {
|
||||
t.Fatalf("%s decision=%#v, want non-reused reason code %q", stage.name, decision, want)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -31,6 +31,7 @@ type Identity struct {
|
||||
Digest string `json:"digest"`
|
||||
PipelineID string `json:"pipeline_id"`
|
||||
PipelineDigest string `json:"pipeline_digest"`
|
||||
PipelineTopology []string `json:"pipeline_topology,omitempty"`
|
||||
InputKey string `json:"input_key"`
|
||||
RawInputDigest string `json:"raw_input_digest,omitempty"`
|
||||
SourceDigest string `json:"source_digest,omitempty"`
|
||||
@@ -57,8 +58,8 @@ func NewIdentity(input IdentityInput) (Identity, error) {
|
||||
if strings.TrimSpace(input.RawInputDigest) == "" && strings.TrimSpace(input.SourceDigest) == "" {
|
||||
return Identity{}, fmt.Errorf("checkpoint identity raw input digest or source digest must be set")
|
||||
}
|
||||
v := Identity{PipelineID: pipelineID, PipelineDigest: pipelineDigest, InputKey: inputKey, RawInputDigest: strings.TrimSpace(input.RawInputDigest), SourceDigest: strings.TrimSpace(input.SourceDigest), SelectedLanes: normalizedLanes(input.SelectedLanes, input.Pipeline.ArtifactLanes), RuntimeOverrides: normalizeIdentityFingerprints(input.RuntimeOverrides), ReferenceDigests: referenceFingerprints(input.References), ProvenanceFingerprints: normalizeIdentityFingerprints(input.ProvenanceFingerprints)}
|
||||
data, err := json.Marshal(Identity{PipelineID: v.PipelineID, PipelineDigest: v.PipelineDigest, InputKey: v.InputKey, RawInputDigest: v.RawInputDigest, SourceDigest: v.SourceDigest, SelectedLanes: v.SelectedLanes, RuntimeOverrides: v.RuntimeOverrides, ReferenceDigests: v.ReferenceDigests, ProvenanceFingerprints: v.ProvenanceFingerprints})
|
||||
v := Identity{PipelineID: pipelineID, PipelineDigest: pipelineDigest, PipelineTopology: pipelineTopology(input.Pipeline), InputKey: inputKey, RawInputDigest: strings.TrimSpace(input.RawInputDigest), SourceDigest: strings.TrimSpace(input.SourceDigest), SelectedLanes: normalizedLanes(input.SelectedLanes, input.Pipeline.AllArtifactLanes()), RuntimeOverrides: normalizeIdentityFingerprints(input.RuntimeOverrides), ReferenceDigests: referenceFingerprints(input.References), ProvenanceFingerprints: normalizeIdentityFingerprints(input.ProvenanceFingerprints)}
|
||||
data, err := json.Marshal(Identity{PipelineID: v.PipelineID, PipelineDigest: v.PipelineDigest, PipelineTopology: v.PipelineTopology, InputKey: v.InputKey, RawInputDigest: v.RawInputDigest, SourceDigest: v.SourceDigest, SelectedLanes: v.SelectedLanes, RuntimeOverrides: v.RuntimeOverrides, ReferenceDigests: v.ReferenceDigests, ProvenanceFingerprints: v.ProvenanceFingerprints})
|
||||
if err != nil {
|
||||
return Identity{}, fmt.Errorf("marshal checkpoint identity: %w", err)
|
||||
}
|
||||
@@ -66,6 +67,19 @@ func NewIdentity(input IdentityInput) (Identity, error) {
|
||||
v.Digest = "sha256:" + hex.EncodeToString(sum[:])
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func pipelineTopology(resolved pipeline.ResolvedPipeline) []string {
|
||||
var topology []string
|
||||
for _, step := range resolved.Steps {
|
||||
for _, lane := range step.ArtifactLanes {
|
||||
topology = append(topology, strings.TrimSpace(step.ID)+"\x00"+strings.TrimSpace(lane.ID))
|
||||
}
|
||||
}
|
||||
if len(topology) == 0 {
|
||||
return nil
|
||||
}
|
||||
return topology
|
||||
}
|
||||
func (i Identity) RelativePath() (string, error) {
|
||||
p, err := safeComponent(i.PipelineID)
|
||||
if err != nil {
|
||||
|
||||
@@ -32,7 +32,9 @@ func TestNewIdentityNormalizesOrderAndEmptyValues(t *testing.T) {
|
||||
v.ProvenanceFingerprints = []Fingerprint{{Name: "source", Value: "v2"}, {Name: "runner", Value: "v1"}}
|
||||
}},
|
||||
{"resolved lanes", func(v *IdentityInput) {
|
||||
v.Pipeline.ArtifactLanes = []pipeline.ResolvedArtifactLane{v.Pipeline.ArtifactLanes[1], v.Pipeline.ArtifactLanes[0]}
|
||||
steps := append([]pipeline.ResolvedPipelineStep(nil), v.Pipeline.Steps...)
|
||||
steps[0].ArtifactLanes = []pipeline.ResolvedArtifactLane{steps[0].ArtifactLanes[1], steps[0].ArtifactLanes[0]}
|
||||
v.Pipeline.Steps = steps
|
||||
}},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
@@ -42,6 +44,12 @@ func TestNewIdentityNormalizesOrderAndEmptyValues(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if tt.name == "resolved lanes" {
|
||||
if reflect.DeepEqual(identity, got) {
|
||||
t.Fatal("reordered topology did not change checkpoint identity")
|
||||
}
|
||||
return
|
||||
}
|
||||
if !reflect.DeepEqual(identity, got) {
|
||||
t.Fatalf("reordered identity differs:\nbase=%#v\ngot=%#v", identity, got)
|
||||
}
|
||||
@@ -174,10 +182,13 @@ func representativeIdentityInput() IdentityInput {
|
||||
ID: "pipeline",
|
||||
Digest: "sha256:pipeline-digest-000000000000",
|
||||
Input: pipeline.Binding("input"),
|
||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{
|
||||
{ID: "lane-b"},
|
||||
{ID: "lane-a"},
|
||||
},
|
||||
Steps: []pipeline.ResolvedPipelineStep{{
|
||||
ID: "default",
|
||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{
|
||||
{ID: "lane-b"},
|
||||
{ID: "lane-a"},
|
||||
},
|
||||
}},
|
||||
},
|
||||
InputKey: "input",
|
||||
RawInputDigest: "sha256:raw-input-000000000000",
|
||||
@@ -197,6 +208,6 @@ func cloneIdentityInput(input IdentityInput) IdentityInput {
|
||||
input.RuntimeOverrides = append([]Fingerprint(nil), input.RuntimeOverrides...)
|
||||
input.References = append([]artifacts.ReferenceProvenance(nil), input.References...)
|
||||
input.ProvenanceFingerprints = append([]Fingerprint(nil), input.ProvenanceFingerprints...)
|
||||
input.Pipeline.ArtifactLanes = append([]pipeline.ResolvedArtifactLane(nil), input.Pipeline.ArtifactLanes...)
|
||||
input.Pipeline.Steps[0].ArtifactLanes = append([]pipeline.ResolvedArtifactLane(nil), input.Pipeline.Steps[0].ArtifactLanes...)
|
||||
return input
|
||||
}
|
||||
|
||||
@@ -52,83 +52,146 @@ func (l *FilesystemLoader) Source(moduleKey string) (pipeline.SourceCheckpoint,
|
||||
}
|
||||
doc := cloneSourceDocument(payload.Document)
|
||||
if err := source.ValidateDocument(&doc); err != nil {
|
||||
return pipeline.SourceCheckpoint{}, invalidDecision("source checkpoint document is invalid: %v", err)
|
||||
return pipeline.SourceCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactPayloadInvalid, "source checkpoint document is invalid")
|
||||
}
|
||||
if strings.TrimSpace(manifest.SourceID) != "" && manifest.SourceID != doc.ID {
|
||||
return pipeline.SourceCheckpoint{}, invalidDecision("source checkpoint source id does not match payload")
|
||||
return pipeline.SourceCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactPayloadInvalid, "source checkpoint identity does not match its payload")
|
||||
}
|
||||
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), digestFingerprints("source_document", doc.Digest)) {
|
||||
return pipeline.SourceCheckpoint{}, invalidDecision("source checkpoint output digest does not match payload")
|
||||
return pipeline.SourceCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch, "source checkpoint output digest does not match its payload")
|
||||
}
|
||||
return pipeline.SourceCheckpoint{Document: &doc}, reusedDecision()
|
||||
}
|
||||
|
||||
func (l *FilesystemLoader) Extract(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.ExtractCheckpoint, pipeline.CheckpointDecision) {
|
||||
return l.ExtractForStep("", laneID, moduleKey, dependencies)
|
||||
}
|
||||
|
||||
func (l *FilesystemLoader) ExtractForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.ExtractCheckpoint, pipeline.CheckpointDecision) {
|
||||
var manifest ExtractLaneManifest
|
||||
if d := l.readJSON(laneManifestPath("extract", laneID), &manifest); !d.Reused {
|
||||
if d := l.readJSON(laneManifestPath("extract", stepID, laneID), &manifest); !d.Reused {
|
||||
return pipeline.ExtractCheckpoint{}, d
|
||||
}
|
||||
if d := l.validateLaneManifest(manifest.StageManifest, StageExtract, laneID, moduleKey, dependencies, StatusSucceeded, StatusSucceededWithRejections); !d.Reused {
|
||||
if d := l.validateLaneManifest(manifest.StageManifest, StageExtract, stepID, laneID, moduleKey, dependencies, StatusSucceeded, StatusSucceededWithRejections); !d.Reused {
|
||||
return pipeline.ExtractCheckpoint{}, d
|
||||
}
|
||||
var payload artifactExtractEnvelope
|
||||
if d := l.readJSON(lanePayloadPath("extract", laneID, "outputs.json"), &payload); !d.Reused {
|
||||
if d := l.readJSON(lanePayloadPath("extract", stepID, laneID, "outputs.json"), &payload); !d.Reused {
|
||||
return pipeline.ExtractCheckpoint{}, d
|
||||
}
|
||||
outputs, err := artifactCheckpointOutputs(payload.Outputs)
|
||||
if err != nil {
|
||||
return pipeline.ExtractCheckpoint{}, invalidDecision("extract artifact checkpoint payload is invalid: %v", err)
|
||||
return pipeline.ExtractCheckpoint{}, artifactDecision(err, "extract checkpoint artifact is invalid")
|
||||
}
|
||||
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(outputs)) {
|
||||
return pipeline.ExtractCheckpoint{}, invalidDecision("extract artifact checkpoint output digests do not match payload")
|
||||
return pipeline.ExtractCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch, "extract checkpoint output digest does not match its payload")
|
||||
}
|
||||
return pipeline.ExtractCheckpoint{Outputs: outputs, Rejected: cloneRejectedOutputs(payload.Rejected), Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||
}
|
||||
|
||||
func (l *FilesystemLoader) Merge(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
|
||||
return l.MergeForStep("", laneID, moduleKey, dependencies)
|
||||
}
|
||||
|
||||
func (l *FilesystemLoader) MergeForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
|
||||
var manifest MergeLaneManifest
|
||||
if d := l.readJSON(laneManifestPath("merge", laneID), &manifest); !d.Reused {
|
||||
if d := l.readJSON(laneManifestPath("merge", stepID, laneID), &manifest); !d.Reused {
|
||||
return pipeline.MergeCheckpoint{}, d
|
||||
}
|
||||
if d := l.validateLaneManifest(manifest.StageManifest, StageMerge, laneID, moduleKey, dependencies, StatusSucceeded); !d.Reused {
|
||||
if d := l.validateLaneManifest(manifest.StageManifest, StageMerge, stepID, laneID, moduleKey, dependencies, StatusSucceeded); !d.Reused {
|
||||
return pipeline.MergeCheckpoint{}, d
|
||||
}
|
||||
var payload artifactSingleEnvelope
|
||||
if d := l.readJSON(lanePayloadPath("merge", laneID, "output.json"), &payload); !d.Reused {
|
||||
if d := l.readJSON(lanePayloadPath("merge", stepID, laneID, "output.json"), &payload); !d.Reused {
|
||||
return pipeline.MergeCheckpoint{}, d
|
||||
}
|
||||
values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
|
||||
if err != nil {
|
||||
return pipeline.MergeCheckpoint{}, invalidDecision("merge artifact checkpoint payload is invalid: %v", err)
|
||||
return pipeline.MergeCheckpoint{}, artifactDecision(err, "merge checkpoint artifact is invalid")
|
||||
}
|
||||
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
||||
return pipeline.MergeCheckpoint{}, invalidDecision("merge artifact checkpoint output digest does not match payload")
|
||||
return pipeline.MergeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch, "merge checkpoint output digest does not match its payload")
|
||||
}
|
||||
return pipeline.MergeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||
}
|
||||
|
||||
func (l *FilesystemLoader) Normalize(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
|
||||
return l.NormalizeForStep("", laneID, moduleKey, dependencies)
|
||||
}
|
||||
|
||||
func (l *FilesystemLoader) NormalizeForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
|
||||
var manifest NormalizeLaneManifest
|
||||
if d := l.readJSON(laneManifestPath("normalize", laneID), &manifest); !d.Reused {
|
||||
if d := l.readJSON(laneManifestPath("normalize", stepID, laneID), &manifest); !d.Reused {
|
||||
return pipeline.NormalizeCheckpoint{}, d
|
||||
}
|
||||
if d := l.validateLaneManifest(manifest.StageManifest, StageNormalize, laneID, moduleKey, dependencies, StatusSucceeded); !d.Reused {
|
||||
if d := l.validateLaneManifest(manifest.StageManifest, StageNormalize, stepID, laneID, moduleKey, dependencies, StatusSucceeded); !d.Reused {
|
||||
return pipeline.NormalizeCheckpoint{}, d
|
||||
}
|
||||
var payload artifactSingleEnvelope
|
||||
if d := l.readJSON(lanePayloadPath("normalize", laneID, "output.json"), &payload); !d.Reused {
|
||||
if d := l.readJSON(lanePayloadPath("normalize", stepID, laneID, "output.json"), &payload); !d.Reused {
|
||||
return pipeline.NormalizeCheckpoint{}, d
|
||||
}
|
||||
values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
|
||||
if err != nil {
|
||||
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize artifact checkpoint payload is invalid: %v", err)
|
||||
return pipeline.NormalizeCheckpoint{}, artifactDecision(err, "normalize checkpoint artifact is invalid")
|
||||
}
|
||||
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
||||
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize artifact checkpoint output digest does not match payload")
|
||||
return pipeline.NormalizeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch, "normalize checkpoint output digest does not match its payload")
|
||||
}
|
||||
return pipeline.NormalizeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||
}
|
||||
|
||||
func (l *FilesystemLoader) AcceptedNormalize(stepID, laneID, moduleKey string) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
|
||||
var manifest NormalizeLaneManifest
|
||||
if d := l.readJSON(laneManifestPath("normalize", stepID, laneID), &manifest); !d.Reused {
|
||||
return pipeline.NormalizeCheckpoint{}, d
|
||||
}
|
||||
if d := l.validateAcceptedNormalizeManifest(manifest.StageManifest, stepID, laneID, moduleKey); !d.Reused {
|
||||
return pipeline.NormalizeCheckpoint{}, d
|
||||
}
|
||||
var payload artifactSingleEnvelope
|
||||
if d := l.readJSON(lanePayloadPath("normalize", stepID, laneID, "output.json"), &payload); !d.Reused {
|
||||
return pipeline.NormalizeCheckpoint{}, d
|
||||
}
|
||||
values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
|
||||
if err != nil {
|
||||
return pipeline.NormalizeCheckpoint{}, artifactDecision(err, "accepted normalize checkpoint artifact is invalid")
|
||||
}
|
||||
if len(values) != 1 {
|
||||
return pipeline.NormalizeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactPayloadInvalid, "accepted normalize checkpoint payload is invalid")
|
||||
}
|
||||
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
||||
return pipeline.NormalizeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch, "accepted normalize checkpoint digest does not match its payload")
|
||||
}
|
||||
return pipeline.NormalizeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, decision(pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonAcceptedArtifactReused, "accepted normalized artifact is reusable")
|
||||
}
|
||||
|
||||
func (l *FilesystemLoader) validateAcceptedNormalizeManifest(manifest StageManifest, stepID, laneID, moduleKey string) pipeline.CheckpointDecision {
|
||||
if manifest.WorkspaceSchemaVersion != WorkspaceSchemaVersion {
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonWorkspaceSchemaIncompatible, "checkpoint workspace schema is incompatible")
|
||||
}
|
||||
identity := strings.TrimSpace(l.identityDigest)
|
||||
if identity == "" || strings.TrimSpace(manifest.Metadata["checkpoint_identity_digest"]) == "" || manifest.Metadata["checkpoint_identity_digest"] != identity {
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonIdentityMismatch, "checkpoint identity is unavailable or does not match the current invocation")
|
||||
}
|
||||
if manifest.Stage != StageNormalize {
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonStageMismatch, "checkpoint stage does not match normalize")
|
||||
}
|
||||
if strings.TrimSpace(stepID) == "" || manifest.StepID != stepID {
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonStepMismatch, "checkpoint step does not match the requested step")
|
||||
}
|
||||
if strings.TrimSpace(laneID) == "" || manifest.LaneID != laneID {
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonLaneMismatch, "checkpoint lane does not match the requested lane")
|
||||
}
|
||||
if strings.TrimSpace(moduleKey) == "" || manifest.ModuleKey != moduleKey {
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonModuleMismatch, "checkpoint module does not match the requested normalizer")
|
||||
}
|
||||
if manifest.Status != StatusSucceeded {
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonStatusNotReusable, "checkpoint status cannot provide an accepted normalized artifact")
|
||||
}
|
||||
return reusedDecision()
|
||||
}
|
||||
|
||||
func artifactCheckpointOutputs(values []artifactCheckpointEnvelope) ([]pipeline.CheckpointArtifact, error) {
|
||||
if len(values) == 0 {
|
||||
return nil, nil
|
||||
@@ -140,7 +203,7 @@ func artifactCheckpointOutputs(values []artifactCheckpointEnvelope) ([]pipeline.
|
||||
return nil, err
|
||||
}
|
||||
if strings.TrimSpace(string(v.Kind)) == "" || strings.TrimSpace(v.Schema.ID) == "" || strings.TrimSpace(v.Schema.Version) == "" || strings.TrimSpace(v.SchemaDigest) == "" {
|
||||
return nil, fmt.Errorf("artifact codec identity is incomplete")
|
||||
return nil, &artifactPayloadError{code: pipeline.CheckpointReasonArtifactCodecIncompatible, err: fmt.Errorf("artifact codec identity is incomplete")}
|
||||
}
|
||||
out = append(out, pipeline.CheckpointArtifact{LaneID: v.LaneID, ModuleKey: v.ModuleKey, SourceID: v.SourceID, ChunkID: v.ChunkID, ChunkIndex: v.ChunkIndex, ChunkRef: v.ChunkRef, SchemaDigest: v.SchemaDigest, Artifact: contracts.SerializedArtifact{Kind: v.Kind, Schema: v.Schema, MediaType: v.Content.MediaType, Content: content, Metadata: cloneMetadata(v.Content.Metadata)}})
|
||||
}
|
||||
@@ -149,47 +212,53 @@ func artifactCheckpointOutputs(values []artifactCheckpointEnvelope) ([]pipeline.
|
||||
|
||||
func (l *FilesystemLoader) readJSON(name string, out any) pipeline.CheckpointDecision {
|
||||
if !l.Enabled() {
|
||||
return pipeline.CheckpointDecision{Reason: "checkpoint loading disabled"}
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonLoadingDisabled, "checkpoint loading disabled")
|
||||
}
|
||||
target, err := fileio.SafePath(l.root, name)
|
||||
if err != nil {
|
||||
return invalidDecision("checkpoint path is invalid: %v", err)
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonPathInvalid, "checkpoint path is invalid")
|
||||
}
|
||||
data, err := os.ReadFile(target)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return pipeline.CheckpointDecision{Reason: "checkpoint artifact is missing"}
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonMissing, "checkpoint artifact is missing")
|
||||
}
|
||||
return invalidDecision("read checkpoint artifact: %v", err)
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonReadFailed, "checkpoint artifact could not be read")
|
||||
}
|
||||
if err := json.Unmarshal(data, out); err != nil {
|
||||
return invalidDecision("decode checkpoint artifact: %v", err)
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonDecodeFailed, "checkpoint artifact could not be decoded")
|
||||
}
|
||||
return reusedDecision()
|
||||
}
|
||||
|
||||
func (l *FilesystemLoader) validateManifest(manifest StageManifest, stage StageName, laneID string, moduleKey string, status StageStatus, dependencies []pipeline.CheckpointFingerprint) pipeline.CheckpointDecision {
|
||||
return l.validateLaneManifest(manifest, stage, laneID, moduleKey, dependencies, status)
|
||||
return l.validateLaneManifest(manifest, stage, "", laneID, moduleKey, dependencies, status)
|
||||
}
|
||||
|
||||
func (l *FilesystemLoader) validateLaneManifest(manifest StageManifest, stage StageName, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, statuses ...StageStatus) pipeline.CheckpointDecision {
|
||||
func (l *FilesystemLoader) validateLaneManifest(manifest StageManifest, stage StageName, stepID string, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, statuses ...StageStatus) pipeline.CheckpointDecision {
|
||||
if manifest.WorkspaceSchemaVersion == WorkspaceSchemaVersionV1 {
|
||||
return invalidDecision("checkpoint workspace schema version %q is incompatible with %q and must be recomputed", manifest.WorkspaceSchemaVersion, WorkspaceSchemaVersion)
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonWorkspaceSchemaIncompatible, "checkpoint workspace schema is incompatible")
|
||||
}
|
||||
if manifest.WorkspaceSchemaVersion == WorkspaceSchemaVersionV2 {
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonWorkspaceSchemaIncompatible, "checkpoint workspace schema is incompatible")
|
||||
}
|
||||
if manifest.WorkspaceSchemaVersion != WorkspaceSchemaVersion {
|
||||
return invalidDecision("checkpoint workspace schema version %q is not supported", manifest.WorkspaceSchemaVersion)
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonWorkspaceSchemaIncompatible, "checkpoint workspace schema is incompatible")
|
||||
}
|
||||
if strings.TrimSpace(l.identityDigest) != "" && manifest.Metadata["checkpoint_identity_digest"] != l.identityDigest {
|
||||
return invalidDecision("checkpoint identity digest does not match current invocation")
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonIdentityMismatch, "checkpoint identity does not match the current invocation")
|
||||
}
|
||||
if manifest.Stage != stage {
|
||||
return invalidDecision("checkpoint stage %q does not match %q", manifest.Stage, stage)
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonStageMismatch, "checkpoint stage does not match the requested stage")
|
||||
}
|
||||
if strings.TrimSpace(stepID) != "" && manifest.StepID != stepID {
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonStepMismatch, "checkpoint step does not match the requested step")
|
||||
}
|
||||
if strings.TrimSpace(laneID) != "" && manifest.LaneID != laneID {
|
||||
return invalidDecision("checkpoint lane %q does not match %q", manifest.LaneID, laneID)
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonLaneMismatch, "checkpoint lane does not match the requested lane")
|
||||
}
|
||||
if strings.TrimSpace(moduleKey) != "" && manifest.ModuleKey != moduleKey {
|
||||
return invalidDecision("checkpoint module %q does not match %q", manifest.ModuleKey, moduleKey)
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonModuleMismatch, "checkpoint module does not match the requested module")
|
||||
}
|
||||
statusOK := false
|
||||
for _, status := range statuses {
|
||||
@@ -199,10 +268,10 @@ func (l *FilesystemLoader) validateLaneManifest(manifest StageManifest, stage St
|
||||
}
|
||||
}
|
||||
if !statusOK {
|
||||
return invalidDecision("checkpoint status %q cannot be reused", manifest.Status)
|
||||
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonStatusNotReusable, "checkpoint status cannot be reused")
|
||||
}
|
||||
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.DependencyFingerprints), dependencies) {
|
||||
return invalidDecision("checkpoint dependency fingerprints do not match")
|
||||
return decision(pipeline.CheckpointDecisionDependencyInvalidated, pipeline.CheckpointReasonDependencyMismatch, "checkpoint dependencies do not match")
|
||||
}
|
||||
return reusedDecision()
|
||||
}
|
||||
@@ -210,10 +279,10 @@ func (l *FilesystemLoader) validateLaneManifest(manifest StageManifest, stage St
|
||||
func contentFromEnvelope(value binaryEnvelope) ([]byte, error) {
|
||||
content, err := base64.StdEncoding.DecodeString(value.ContentBase64)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("decode content_base64: %w", err)
|
||||
return nil, &artifactPayloadError{code: pipeline.CheckpointReasonArtifactPayloadInvalid, err: fmt.Errorf("decode content_base64: %w", err)}
|
||||
}
|
||||
if digest := strings.TrimSpace(value.ContentDigest); digest != "" && digest != contentDigest(content) {
|
||||
return nil, fmt.Errorf("content digest mismatch")
|
||||
return nil, &artifactPayloadError{code: pipeline.CheckpointReasonArtifactDigestMismatch, err: fmt.Errorf("content digest mismatch")}
|
||||
}
|
||||
return content, nil
|
||||
}
|
||||
@@ -244,9 +313,24 @@ func fingerprintsEqual(a []pipeline.CheckpointFingerprint, b []pipeline.Checkpoi
|
||||
}
|
||||
|
||||
func reusedDecision() pipeline.CheckpointDecision {
|
||||
return pipeline.CheckpointDecision{Reused: true, Reason: "checkpoint is valid"}
|
||||
return decision(pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonReused, "checkpoint is reusable")
|
||||
}
|
||||
|
||||
func invalidDecision(format string, args ...any) pipeline.CheckpointDecision {
|
||||
return pipeline.CheckpointDecision{Reason: fmt.Sprintf(format, args...)}
|
||||
func decision(category pipeline.CheckpointDecisionCategory, code pipeline.CheckpointReasonCode, detail string) pipeline.CheckpointDecision {
|
||||
return pipeline.NewCheckpointDecision(category, code, detail)
|
||||
}
|
||||
|
||||
type artifactPayloadError struct {
|
||||
code pipeline.CheckpointReasonCode
|
||||
err error
|
||||
}
|
||||
|
||||
func (e *artifactPayloadError) Error() string { return e.err.Error() }
|
||||
|
||||
func artifactDecision(err error, detail string) pipeline.CheckpointDecision {
|
||||
code := pipeline.CheckpointReasonArtifactPayloadInvalid
|
||||
if payloadErr, ok := err.(*artifactPayloadError); ok {
|
||||
code = payloadErr.code
|
||||
}
|
||||
return decision(pipeline.CheckpointDecisionExecuted, code, detail)
|
||||
}
|
||||
|
||||
@@ -3,9 +3,8 @@ package checkpoint
|
||||
import "time"
|
||||
|
||||
const (
|
||||
// These names and values are frozen checkpoint wire-compatibility
|
||||
// identifiers. They intentionally retain the former terminology.
|
||||
WorkspaceSchemaVersion = "notarius.workspace.v2"
|
||||
WorkspaceSchemaVersion = "notarius.workspace.v3"
|
||||
WorkspaceSchemaVersionV2 = "notarius.workspace.v2"
|
||||
WorkspaceSchemaVersionV1 = "notarius.workspace.v1"
|
||||
)
|
||||
|
||||
@@ -32,6 +31,7 @@ const (
|
||||
type StageManifest struct {
|
||||
WorkspaceSchemaVersion string `json:"workspace_schema_version"`
|
||||
Stage StageName `json:"stage"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
DependencyFingerprints []Fingerprint `json:"dependency_fingerprints,omitempty"`
|
||||
|
||||
@@ -71,93 +71,137 @@ func (r *FilesystemRecorder) SourceFailed(moduleKey string, err error) error {
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) ExtractRunning(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
|
||||
manifest := r.laneManifest(StageExtract, StatusRunning, laneID, moduleKey, dependencies)
|
||||
return r.ExtractRunningForStep("", laneID, moduleKey, dependencies)
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) ExtractRunningForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
|
||||
manifest := r.laneManifest(StageExtract, StatusRunning, stepID, laneID, moduleKey, dependencies)
|
||||
manifest.StartedAt = timePtr(r.timestamp())
|
||||
return r.writeManifest(laneManifestPath("extract", laneID), ExtractLaneManifest{StageManifest: manifest})
|
||||
return r.writeManifest(laneManifestPath("extract", stepID, laneID), ExtractLaneManifest{StageManifest: manifest})
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) ExtractSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||
return r.ExtractSucceededForStep("", laneID, moduleKey, dependencies, outputs, rejected, warnings)
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) ExtractSucceededForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||
payload := artifactExtractEnvelope{Outputs: artifactCheckpointEnvelopes(outputs), Rejected: cloneRejectedOutputs(rejected), Warnings: cloneWarnings(warnings)}
|
||||
if err := r.writePayload(lanePayloadPath("extract", laneID, "outputs.json"), payload); err != nil {
|
||||
if err := r.writePayload(lanePayloadPath("extract", stepID, laneID, "outputs.json"), payload); err != nil {
|
||||
return err
|
||||
}
|
||||
manifest := r.laneManifest(StageExtract, statusForRejected(rejected), laneID, moduleKey, dependencies)
|
||||
manifest := r.laneManifest(StageExtract, statusForRejected(rejected), stepID, laneID, moduleKey, dependencies)
|
||||
manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests(outputs))
|
||||
manifest.ValidationStatus = validationStatusString(warnings, rejected)
|
||||
manifest.Rejections = rejectionSummaries(rejected)
|
||||
manifest.CompletedAt = timePtr(r.timestamp())
|
||||
return r.writeManifest(laneManifestPath("extract", laneID), ExtractLaneManifest{StageManifest: manifest, ChunkCount: len(outputs) + len(rejected), OutputCount: len(outputs)})
|
||||
return r.writeManifest(laneManifestPath("extract", stepID, laneID), ExtractLaneManifest{StageManifest: manifest, ChunkCount: len(outputs) + len(rejected), OutputCount: len(outputs)})
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) ExtractFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
|
||||
manifest := r.laneManifest(StageExtract, StatusFailed, laneID, moduleKey, dependencies)
|
||||
return r.ExtractFailedForStep("", laneID, moduleKey, dependencies, err)
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) ExtractFailedForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
|
||||
manifest := r.laneManifest(StageExtract, StatusFailed, stepID, laneID, moduleKey, dependencies)
|
||||
manifest.CompletedAt = timePtr(r.timestamp())
|
||||
manifest.Metadata = errorMetadata(err)
|
||||
return r.writeManifest(laneManifestPath("extract", laneID), ExtractLaneManifest{StageManifest: manifest})
|
||||
return r.writeManifest(laneManifestPath("extract", stepID, laneID), ExtractLaneManifest{StageManifest: manifest})
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) MergeRunning(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
|
||||
manifest := r.laneManifest(StageMerge, StatusRunning, laneID, moduleKey, dependencies)
|
||||
return r.MergeRunningForStep("", laneID, moduleKey, dependencies)
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) MergeRunningForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
|
||||
manifest := r.laneManifest(StageMerge, StatusRunning, stepID, laneID, moduleKey, dependencies)
|
||||
manifest.StartedAt = timePtr(r.timestamp())
|
||||
return r.writeManifest(laneManifestPath("merge", laneID), MergeLaneManifest{StageManifest: manifest})
|
||||
return r.writeManifest(laneManifestPath("merge", stepID, laneID), MergeLaneManifest{StageManifest: manifest})
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) MergeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
if err := r.writePayload(lanePayloadPath("merge", laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
|
||||
return r.MergeSucceededForStep("", laneID, moduleKey, dependencies, output, warnings)
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) MergeSucceededForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
if err := r.writePayload(lanePayloadPath("merge", stepID, laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
|
||||
return err
|
||||
}
|
||||
manifest := r.laneManifest(StageMerge, StatusSucceeded, laneID, moduleKey, dependencies)
|
||||
manifest := r.laneManifest(StageMerge, StatusSucceeded, stepID, laneID, moduleKey, dependencies)
|
||||
manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
|
||||
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
||||
manifest.CompletedAt = timePtr(r.timestamp())
|
||||
return r.writeManifest(laneManifestPath("merge", laneID), MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||
return r.writeManifest(laneManifestPath("merge", stepID, laneID), MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) MergeRejected(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||
manifest := r.laneManifest(StageMerge, StatusSucceededWithRejections, laneID, moduleKey, dependencies)
|
||||
return r.MergeRejectedForStep("", laneID, moduleKey, dependencies, rejected)
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) MergeRejectedForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||
manifest := r.laneManifest(StageMerge, StatusSucceededWithRejections, stepID, laneID, moduleKey, dependencies)
|
||||
manifest.ValidationStatus = "rejected"
|
||||
manifest.Rejections = rejectionSummaries([]contracts.RejectedOutput{rejected})
|
||||
manifest.CompletedAt = timePtr(r.timestamp())
|
||||
return r.writeManifest(laneManifestPath("merge", laneID), MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||
return r.writeManifest(laneManifestPath("merge", stepID, laneID), MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) MergeFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
|
||||
manifest := r.laneManifest(StageMerge, StatusFailed, laneID, moduleKey, dependencies)
|
||||
return r.MergeFailedForStep("", laneID, moduleKey, dependencies, err)
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) MergeFailedForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
|
||||
manifest := r.laneManifest(StageMerge, StatusFailed, stepID, laneID, moduleKey, dependencies)
|
||||
manifest.CompletedAt = timePtr(r.timestamp())
|
||||
manifest.Metadata = errorMetadata(err)
|
||||
return r.writeManifest(laneManifestPath("merge", laneID), MergeLaneManifest{StageManifest: manifest})
|
||||
return r.writeManifest(laneManifestPath("merge", stepID, laneID), MergeLaneManifest{StageManifest: manifest})
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) NormalizeRunning(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
|
||||
manifest := r.laneManifest(StageNormalize, StatusRunning, laneID, moduleKey, dependencies)
|
||||
return r.NormalizeRunningForStep("", laneID, moduleKey, dependencies)
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) NormalizeRunningForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
|
||||
manifest := r.laneManifest(StageNormalize, StatusRunning, stepID, laneID, moduleKey, dependencies)
|
||||
manifest.StartedAt = timePtr(r.timestamp())
|
||||
return r.writeManifest(laneManifestPath("normalize", laneID), NormalizeLaneManifest{StageManifest: manifest})
|
||||
return r.writeManifest(laneManifestPath("normalize", stepID, laneID), NormalizeLaneManifest{StageManifest: manifest})
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) NormalizeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
if err := r.writePayload(lanePayloadPath("normalize", laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
|
||||
return r.NormalizeSucceededForStep("", laneID, moduleKey, dependencies, output, warnings)
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) NormalizeSucceededForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
if err := r.writePayload(lanePayloadPath("normalize", stepID, laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
|
||||
return err
|
||||
}
|
||||
manifest := r.laneManifest(StageNormalize, StatusSucceeded, laneID, moduleKey, dependencies)
|
||||
manifest := r.laneManifest(StageNormalize, StatusSucceeded, stepID, laneID, moduleKey, dependencies)
|
||||
manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
|
||||
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
||||
manifest.CompletedAt = timePtr(r.timestamp())
|
||||
return r.writeManifest(laneManifestPath("normalize", laneID), NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||
return r.writeManifest(laneManifestPath("normalize", stepID, laneID), NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) NormalizeRejected(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||
manifest := r.laneManifest(StageNormalize, StatusSucceededWithRejections, laneID, moduleKey, dependencies)
|
||||
return r.NormalizeRejectedForStep("", laneID, moduleKey, dependencies, rejected)
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) NormalizeRejectedForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||
manifest := r.laneManifest(StageNormalize, StatusSucceededWithRejections, stepID, laneID, moduleKey, dependencies)
|
||||
manifest.ValidationStatus = "rejected"
|
||||
manifest.Rejections = rejectionSummaries([]contracts.RejectedOutput{rejected})
|
||||
manifest.CompletedAt = timePtr(r.timestamp())
|
||||
return r.writeManifest(laneManifestPath("normalize", laneID), NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||
return r.writeManifest(laneManifestPath("normalize", stepID, laneID), NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) NormalizeFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
|
||||
manifest := r.laneManifest(StageNormalize, StatusFailed, laneID, moduleKey, dependencies)
|
||||
return r.NormalizeFailedForStep("", laneID, moduleKey, dependencies, err)
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) NormalizeFailedForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
|
||||
manifest := r.laneManifest(StageNormalize, StatusFailed, stepID, laneID, moduleKey, dependencies)
|
||||
manifest.CompletedAt = timePtr(r.timestamp())
|
||||
manifest.Metadata = errorMetadata(err)
|
||||
return r.writeManifest(laneManifestPath("normalize", laneID), NormalizeLaneManifest{StageManifest: manifest})
|
||||
return r.writeManifest(laneManifestPath("normalize", stepID, laneID), NormalizeLaneManifest{StageManifest: manifest})
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) writeManifest(name string, payload any) error {
|
||||
@@ -190,8 +234,9 @@ func (r *FilesystemRecorder) newStageManifest(stage StageName, status StageStatu
|
||||
return manifest
|
||||
}
|
||||
|
||||
func (r *FilesystemRecorder) laneManifest(stage StageName, status StageStatus, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) StageManifest {
|
||||
func (r *FilesystemRecorder) laneManifest(stage StageName, status StageStatus, stepID string, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) StageManifest {
|
||||
manifest := r.newStageManifest(stage, status)
|
||||
manifest.StepID = stepID
|
||||
manifest.LaneID = laneID
|
||||
manifest.ModuleKey = moduleKey
|
||||
manifest.DependencyFingerprints = checkpointFingerprints(dependencies)
|
||||
@@ -424,12 +469,15 @@ func errorMetadata(err error) map[string]string {
|
||||
return map[string]string{"error": err.Error()}
|
||||
}
|
||||
|
||||
func laneManifestPath(stage string, laneID string) string {
|
||||
return lanePayloadPath(stage, laneID, "manifest.json")
|
||||
func laneManifestPath(stage string, stepID string, laneID string) string {
|
||||
return lanePayloadPath(stage, stepID, laneID, "manifest.json")
|
||||
}
|
||||
|
||||
func lanePayloadPath(stage string, laneID string, file string) string {
|
||||
return path.Join(stage, checkpointPathComponent(laneID), file)
|
||||
func lanePayloadPath(stage string, stepID string, laneID string, file string) string {
|
||||
if strings.TrimSpace(stepID) == "" {
|
||||
return path.Join(stage, checkpointPathComponent(laneID), file)
|
||||
}
|
||||
return path.Join(stage, checkpointPathComponent(stepID), checkpointPathComponent(laneID), file)
|
||||
}
|
||||
|
||||
func checkpointPathComponent(value string) string {
|
||||
|
||||
@@ -35,9 +35,40 @@ func TestRootBasedRecorderOutputIsReusable(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckpointSchemaCompatibilityIsUnchanged(t *testing.T) {
|
||||
if WorkspaceSchemaVersion != "notarius.workspace.v2" || WorkspaceSchemaVersionV1 != "notarius.workspace.v1" {
|
||||
t.Fatal("checkpoint schema identifiers changed")
|
||||
func TestStepAwareRecorderAndLoaderIsolateLaneState(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
identity := testIdentity(t)
|
||||
recorder, err := NewFilesystemRecorder(root, identity)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := recorder.(pipeline.StepCheckpointRecorder).ExtractSucceededForStep("step-a", "lane", "module", nil, nil, nil, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
loader, err := NewFilesystemLoader(root, identity)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stepLoader := loader.(pipeline.StepCheckpointLoader)
|
||||
if _, decision := stepLoader.ExtractForStep("step-b", "lane", "module", nil); decision.Reused {
|
||||
t.Fatal("checkpoint from another step was reused")
|
||||
}
|
||||
loaded, decision := stepLoader.ExtractForStep("step-a", "lane", "module", nil)
|
||||
if !decision.Reused || len(loaded.Outputs) != 0 {
|
||||
t.Fatalf("step-aware load = %#v, decision=%#v", loaded, decision)
|
||||
}
|
||||
relative, err := identity.RelativePath()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(root, relative, "extract", "step-a", "lane", "manifest.json")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckpointSchemaCompatibilityIdentifiers(t *testing.T) {
|
||||
if WorkspaceSchemaVersion != "notarius.workspace.v3" || WorkspaceSchemaVersionV2 != "notarius.workspace.v2" || WorkspaceSchemaVersionV1 != "notarius.workspace.v1" {
|
||||
t.Fatal("checkpoint schema identifiers are incorrect")
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -30,6 +30,7 @@ type SerializedArtifact struct {
|
||||
// SerializedOutput associates a domain-neutral artifact with the pipeline
|
||||
// operation that produced it. Provenance remains outside codec-owned bytes.
|
||||
type SerializedOutput struct {
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id"`
|
||||
NormalizerKey string `json:"normalizer_key"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
|
||||
@@ -163,12 +163,13 @@ const (
|
||||
)
|
||||
|
||||
type ReferenceSlot struct {
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Required bool `json:"required,omitempty"`
|
||||
AcceptedMediaTypes []string `json:"accepted_media_types,omitempty"`
|
||||
Multiple bool `json:"multiple,omitempty"`
|
||||
MaxBytes int64 `json:"max_bytes,omitempty"`
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Required bool `json:"required,omitempty"`
|
||||
AcceptedMediaTypes []string `json:"accepted_media_types,omitempty"`
|
||||
AcceptedArtifactKinds []ArtifactKind `json:"accepted_artifact_kinds,omitempty"`
|
||||
Multiple bool `json:"multiple,omitempty"`
|
||||
MaxBytes int64 `json:"max_bytes,omitempty"`
|
||||
}
|
||||
|
||||
func CloneReferenceSlots(slots []ReferenceSlot) []ReferenceSlot {
|
||||
@@ -178,6 +179,7 @@ func CloneReferenceSlots(slots []ReferenceSlot) []ReferenceSlot {
|
||||
out := make([]ReferenceSlot, len(slots))
|
||||
for i, slot := range slots {
|
||||
slot.AcceptedMediaTypes = append([]string(nil), slot.AcceptedMediaTypes...)
|
||||
slot.AcceptedArtifactKinds = append([]ArtifactKind(nil), slot.AcceptedArtifactKinds...)
|
||||
out[i] = slot
|
||||
}
|
||||
return out
|
||||
@@ -188,14 +190,64 @@ type ReferenceOrigin struct {
|
||||
URI string `json:"uri,omitempty"`
|
||||
}
|
||||
|
||||
// ReferenceProducer identifies the operation that produced a generated
|
||||
// reference. It contains provenance only; referenced bytes remain in Content.
|
||||
type ReferenceProducer struct {
|
||||
PipelineID string `json:"pipeline_id,omitempty"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
}
|
||||
|
||||
type ReferenceItem struct {
|
||||
SlotName string `json:"slot_name"`
|
||||
MediaType string `json:"media_type,omitempty"`
|
||||
Content []byte `json:"-"`
|
||||
Digest string `json:"digest,omitempty"`
|
||||
Origin ReferenceOrigin `json:"origin"`
|
||||
SizeBytes int64 `json:"size_bytes,omitempty"`
|
||||
BindingSource string `json:"binding_source,omitempty"`
|
||||
SlotName string `json:"slot_name"`
|
||||
MediaType string `json:"media_type,omitempty"`
|
||||
Content []byte `json:"-"`
|
||||
Digest string `json:"digest,omitempty"`
|
||||
Origin ReferenceOrigin `json:"origin"`
|
||||
SizeBytes int64 `json:"size_bytes,omitempty"`
|
||||
BindingSource string `json:"binding_source,omitempty"`
|
||||
ArtifactKind ArtifactKind `json:"artifact_kind,omitempty"`
|
||||
ArtifactSchema ArtifactSchema `json:"artifact_schema,omitempty"`
|
||||
Producer ReferenceProducer `json:"producer,omitempty"`
|
||||
}
|
||||
|
||||
func CloneReferenceItem(item ReferenceItem) ReferenceItem {
|
||||
item.Content = append([]byte(nil), item.Content...)
|
||||
item.ArtifactSchema = CloneArtifactSchema(item.ArtifactSchema)
|
||||
return item
|
||||
}
|
||||
|
||||
func (item ReferenceItem) MarshalJSON() ([]byte, error) {
|
||||
type referenceItemJSON struct {
|
||||
SlotName string `json:"slot_name"`
|
||||
MediaType string `json:"media_type,omitempty"`
|
||||
Digest string `json:"digest,omitempty"`
|
||||
Origin ReferenceOrigin `json:"origin"`
|
||||
SizeBytes int64 `json:"size_bytes,omitempty"`
|
||||
BindingSource string `json:"binding_source,omitempty"`
|
||||
ArtifactKind ArtifactKind `json:"artifact_kind,omitempty"`
|
||||
ArtifactSchema *ArtifactSchema `json:"artifact_schema,omitempty"`
|
||||
Producer *ReferenceProducer `json:"producer,omitempty"`
|
||||
}
|
||||
encoded := referenceItemJSON{
|
||||
SlotName: item.SlotName,
|
||||
MediaType: item.MediaType,
|
||||
Digest: item.Digest,
|
||||
Origin: item.Origin,
|
||||
SizeBytes: item.SizeBytes,
|
||||
BindingSource: item.BindingSource,
|
||||
ArtifactKind: item.ArtifactKind,
|
||||
}
|
||||
if item.ArtifactSchema.ID != "" || item.ArtifactSchema.Name != "" || item.ArtifactSchema.Version != "" || len(item.ArtifactSchema.JSONSchema) > 0 {
|
||||
schema := CloneArtifactSchema(item.ArtifactSchema)
|
||||
encoded.ArtifactSchema = &schema
|
||||
}
|
||||
if item.Producer.PipelineID != "" || item.Producer.StepID != "" || item.Producer.LaneID != "" || item.Producer.ModuleKey != "" {
|
||||
producer := item.Producer
|
||||
encoded.Producer = &producer
|
||||
}
|
||||
return json.Marshal(encoded)
|
||||
}
|
||||
|
||||
type ResolvedReferenceSlot struct {
|
||||
@@ -259,6 +311,7 @@ type OutputEncoder interface {
|
||||
|
||||
type RejectedOutput struct {
|
||||
Stage string `json:"stage"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
ChunkID string `json:"chunk_id,omitempty"`
|
||||
|
||||
@@ -70,6 +70,15 @@ func TestCloneReferenceSlotsCopiesAcceptedMediaTypes(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCloneReferenceSlotsCopiesAcceptedArtifactKinds(t *testing.T) {
|
||||
slots := []ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []ArtifactKind{"dnd/npc-list"}}}
|
||||
clone := CloneReferenceSlots(slots)
|
||||
clone[0].AcceptedArtifactKinds[0] = "changed"
|
||||
if slots[0].AcceptedArtifactKinds[0] != "dnd/npc-list" {
|
||||
t.Fatalf("source AcceptedArtifactKinds aliased clone: %#v", slots[0].AcceptedArtifactKinds)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReferenceItemJSONOmitsContent(t *testing.T) {
|
||||
item := ReferenceItem{
|
||||
SlotName: "roster",
|
||||
|
||||
@@ -40,17 +40,167 @@ type CheckpointRecorder interface {
|
||||
NormalizeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||
}
|
||||
|
||||
// StepCheckpointRecorder is implemented by checkpoint stores that isolate
|
||||
// lane artifacts by their ordered pipeline step. The legacy recorder methods
|
||||
// remain available for callers that do not have step context.
|
||||
type StepCheckpointRecorder interface {
|
||||
ExtractRunningForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||
ExtractSucceededForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error
|
||||
ExtractFailedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||
MergeRunningForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||
MergeSucceededForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
|
||||
MergeRejectedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
||||
MergeFailedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||
NormalizeRunningForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||
NormalizeSucceededForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
|
||||
NormalizeRejectedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
||||
NormalizeFailedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||
}
|
||||
|
||||
type CheckpointDecisionCategory string
|
||||
|
||||
const (
|
||||
CheckpointDecisionExecuted CheckpointDecisionCategory = "executed"
|
||||
CheckpointDecisionReused CheckpointDecisionCategory = "reused"
|
||||
CheckpointDecisionForcedRecompute CheckpointDecisionCategory = "forced_recompute"
|
||||
CheckpointDecisionDependencyInvalidated CheckpointDecisionCategory = "dependency_invalidated"
|
||||
)
|
||||
|
||||
type CheckpointReasonCode string
|
||||
|
||||
const (
|
||||
CheckpointReasonLoadingDisabled CheckpointReasonCode = "loading_disabled"
|
||||
CheckpointReasonMissing CheckpointReasonCode = "checkpoint_missing"
|
||||
CheckpointReasonPathInvalid CheckpointReasonCode = "checkpoint_path_invalid"
|
||||
CheckpointReasonReadFailed CheckpointReasonCode = "checkpoint_read_failed"
|
||||
CheckpointReasonDecodeFailed CheckpointReasonCode = "checkpoint_decode_failed"
|
||||
CheckpointReasonWorkspaceSchemaIncompatible CheckpointReasonCode = "workspace_schema_incompatible"
|
||||
CheckpointReasonIdentityMismatch CheckpointReasonCode = "identity_mismatch"
|
||||
CheckpointReasonStageMismatch CheckpointReasonCode = "stage_mismatch"
|
||||
CheckpointReasonStepMismatch CheckpointReasonCode = "step_mismatch"
|
||||
CheckpointReasonLaneMismatch CheckpointReasonCode = "lane_mismatch"
|
||||
CheckpointReasonModuleMismatch CheckpointReasonCode = "module_mismatch"
|
||||
CheckpointReasonStatusNotReusable CheckpointReasonCode = "status_not_reusable"
|
||||
CheckpointReasonDependencyMismatch CheckpointReasonCode = "dependency_mismatch"
|
||||
CheckpointReasonArtifactPayloadInvalid CheckpointReasonCode = "artifact_payload_invalid"
|
||||
CheckpointReasonArtifactDigestMismatch CheckpointReasonCode = "artifact_digest_mismatch"
|
||||
CheckpointReasonArtifactCodecIncompatible CheckpointReasonCode = "artifact_codec_incompatible"
|
||||
CheckpointReasonArtifactNotCanonical CheckpointReasonCode = "artifact_not_canonical"
|
||||
CheckpointReasonReused CheckpointReasonCode = "checkpoint_reused"
|
||||
CheckpointReasonAcceptedArtifactReused CheckpointReasonCode = "accepted_artifact_reused"
|
||||
CheckpointReasonRecomputeStep CheckpointReasonCode = "recompute_step"
|
||||
)
|
||||
|
||||
type CheckpointDecision struct {
|
||||
Reused bool `json:"reused"`
|
||||
Reused bool `json:"reused"`
|
||||
Category CheckpointDecisionCategory `json:"category,omitempty"`
|
||||
ReasonCode CheckpointReasonCode `json:"reason_code,omitempty"`
|
||||
Detail string `json:"detail,omitempty"`
|
||||
// Reason is retained as a compatibility/debug field for existing callers.
|
||||
// New checkpoint stores should put bounded, non-sensitive text in Detail.
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
const checkpointDecisionDetailLimit = 512
|
||||
|
||||
func NewCheckpointDecision(category CheckpointDecisionCategory, reasonCode CheckpointReasonCode, detail string) CheckpointDecision {
|
||||
return checkpointDecision(category, reasonCode, detail)
|
||||
}
|
||||
|
||||
func checkpointDecision(category CheckpointDecisionCategory, reasonCode CheckpointReasonCode, detail string) CheckpointDecision {
|
||||
detail = sanitizeCheckpointDecisionDetail(detail)
|
||||
return CheckpointDecision{
|
||||
Reused: category == CheckpointDecisionReused,
|
||||
Category: category,
|
||||
ReasonCode: reasonCode,
|
||||
Detail: detail,
|
||||
Reason: detail,
|
||||
}
|
||||
}
|
||||
|
||||
func sanitizeCheckpointDecisionDetail(detail string) string {
|
||||
detail = strings.TrimSpace(strings.ToValidUTF8(detail, "?"))
|
||||
lower := strings.ToLower(detail)
|
||||
if strings.ContainsAny(detail, `/\\`) || strings.Contains(lower, "secret") || strings.Contains(lower, "token") || strings.Contains(lower, "password") || strings.Contains(lower, "credential") || strings.Contains(lower, "environment") {
|
||||
return "checkpoint decision detail redacted"
|
||||
}
|
||||
var b strings.Builder
|
||||
for _, r := range detail {
|
||||
if r < 0x20 || r == 0x7f {
|
||||
r = ' '
|
||||
}
|
||||
if b.Len()+len(string(r)) > checkpointDecisionDetailLimit {
|
||||
break
|
||||
}
|
||||
b.WriteRune(r)
|
||||
}
|
||||
return strings.Join(strings.Fields(b.String()), " ")
|
||||
}
|
||||
|
||||
type CheckpointEvent struct {
|
||||
Stage string `json:"stage"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
Action string `json:"action"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Stage string `json:"stage"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
Action CheckpointDecisionCategory `json:"action"`
|
||||
Category CheckpointDecisionCategory `json:"category,omitempty"`
|
||||
ReasonCode CheckpointReasonCode `json:"reason_code,omitempty"`
|
||||
Detail string `json:"detail,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
type CheckpointExecutionPolicy struct {
|
||||
ForcedLanes map[string]struct{}
|
||||
RequireReusableLanes map[string]struct{}
|
||||
}
|
||||
|
||||
func CheckpointLaneKey(stepID, laneID string) string {
|
||||
return strings.TrimSpace(stepID) + "\x00" + strings.TrimSpace(laneID)
|
||||
}
|
||||
|
||||
func (policy CheckpointExecutionPolicy) forced(stepID, laneID string) bool {
|
||||
_, ok := policy.ForcedLanes[CheckpointLaneKey(stepID, laneID)]
|
||||
return ok
|
||||
}
|
||||
|
||||
func (policy CheckpointExecutionPolicy) requiresReusable(stepID, laneID string) bool {
|
||||
_, ok := policy.RequireReusableLanes[CheckpointLaneKey(stepID, laneID)]
|
||||
return ok
|
||||
}
|
||||
|
||||
func forceCheckpointDecision(policy CheckpointExecutionPolicy, stepID, laneID string, decision CheckpointDecision) CheckpointDecision {
|
||||
if policy.forced(stepID, laneID) {
|
||||
return checkpointDecision(CheckpointDecisionForcedRecompute, CheckpointReasonRecomputeStep, "selected step requires execution")
|
||||
}
|
||||
return decision
|
||||
}
|
||||
|
||||
func requireReusableCheckpoint(policy CheckpointExecutionPolicy, stepID, laneID string, decision CheckpointDecision) error {
|
||||
if policy.requiresReusable(stepID, laneID) && !policy.forced(stepID, laneID) && !decision.Reused {
|
||||
return fmt.Errorf("required reusable checkpoint unavailable for step %q lane %q (%s)", strings.TrimSpace(stepID), strings.TrimSpace(laneID), decision.ReasonCode)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// resolveCheckpointDecision applies runner policy and canonical payload
|
||||
// validation at the single point where a stage's observable decision is made.
|
||||
func resolveCheckpointDecision(output *RunOutput, loader CheckpointLoader, policy CheckpointExecutionPolicy, stage ModuleStage, stepID, laneID, moduleKey string, decision CheckpointDecision, codec artifactCodecEntry, artifacts []CheckpointArtifact) (CheckpointDecision, error) {
|
||||
decision = forceCheckpointDecision(policy, stepID, laneID, decision)
|
||||
if decision.Reused {
|
||||
for _, artifact := range artifacts {
|
||||
if _, _, err := decodeCanonicalCheckpointArtifact(codec, artifact); err != nil {
|
||||
decision = checkpointDecision(CheckpointDecisionExecuted, checkpointArtifactReasonCode(err), "stored "+string(stage)+" artifact failed canonical codec validation")
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if output != nil {
|
||||
recordCheckpointEvent(output, loader, string(stage), stepID, laneID, moduleKey, decision)
|
||||
}
|
||||
if err := requireReusableCheckpoint(policy, stepID, laneID, decision); err != nil {
|
||||
return decision, err
|
||||
}
|
||||
return decision, nil
|
||||
}
|
||||
|
||||
type SourceCheckpoint struct {
|
||||
@@ -91,6 +241,16 @@ type CheckpointLoader interface {
|
||||
Extract(laneID string, moduleKey string, dependencies []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision)
|
||||
Merge(laneID string, moduleKey string, dependencies []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision)
|
||||
Normalize(laneID string, moduleKey string, dependencies []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision)
|
||||
AcceptedNormalize(stepID, laneID, moduleKey string) (NormalizeCheckpoint, CheckpointDecision)
|
||||
}
|
||||
|
||||
// StepCheckpointLoader is the step-aware counterpart used by the persistent
|
||||
// checkpoint implementation. Loaders without this optional interface remain
|
||||
// usable by framework callers and test doubles through the legacy methods.
|
||||
type StepCheckpointLoader interface {
|
||||
ExtractForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision)
|
||||
MergeForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision)
|
||||
NormalizeForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision)
|
||||
}
|
||||
|
||||
type noopCheckpointRecorder struct{}
|
||||
@@ -136,16 +296,86 @@ func (noopCheckpointRecorder) NormalizeFailed(string, string, []CheckpointFinger
|
||||
|
||||
func (noopCheckpointLoader) Enabled() bool { return false }
|
||||
func (noopCheckpointLoader) Source(string) (SourceCheckpoint, CheckpointDecision) {
|
||||
return SourceCheckpoint{}, CheckpointDecision{Reason: "checkpoint loading disabled"}
|
||||
return SourceCheckpoint{}, checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonLoadingDisabled, "checkpoint loading disabled")
|
||||
}
|
||||
func (noopCheckpointLoader) Extract(string, string, []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||
return ExtractCheckpoint{}, CheckpointDecision{Reason: "checkpoint loading disabled"}
|
||||
return ExtractCheckpoint{}, checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonLoadingDisabled, "checkpoint loading disabled")
|
||||
}
|
||||
func (noopCheckpointLoader) Merge(string, string, []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
|
||||
return MergeCheckpoint{}, CheckpointDecision{Reason: "checkpoint loading disabled"}
|
||||
return MergeCheckpoint{}, checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonLoadingDisabled, "checkpoint loading disabled")
|
||||
}
|
||||
func (noopCheckpointLoader) Normalize(string, string, []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
return NormalizeCheckpoint{}, CheckpointDecision{Reason: "checkpoint loading disabled"}
|
||||
return NormalizeCheckpoint{}, checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonLoadingDisabled, "checkpoint loading disabled")
|
||||
}
|
||||
func (noopCheckpointLoader) AcceptedNormalize(string, string, string) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
return NormalizeCheckpoint{}, checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonLoadingDisabled, "checkpoint loading disabled")
|
||||
}
|
||||
|
||||
func checkpointExtractRunning(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) error {
|
||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||
return stepAware.ExtractRunningForStep(stepID, laneID, moduleKey, deps)
|
||||
}
|
||||
return recorder.ExtractRunning(laneID, moduleKey, deps)
|
||||
}
|
||||
func checkpointExtractSucceeded(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||
return stepAware.ExtractSucceededForStep(stepID, laneID, moduleKey, deps, outputs, rejected, warnings)
|
||||
}
|
||||
return recorder.ExtractSucceeded(laneID, moduleKey, deps, outputs, rejected, warnings)
|
||||
}
|
||||
func checkpointExtractFailed(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, err error) error {
|
||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||
return stepAware.ExtractFailedForStep(stepID, laneID, moduleKey, deps, err)
|
||||
}
|
||||
return recorder.ExtractFailed(laneID, moduleKey, deps, err)
|
||||
}
|
||||
func checkpointMergeRunning(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) error {
|
||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||
return stepAware.MergeRunningForStep(stepID, laneID, moduleKey, deps)
|
||||
}
|
||||
return recorder.MergeRunning(laneID, moduleKey, deps)
|
||||
}
|
||||
func checkpointMergeSucceeded(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||
return stepAware.MergeSucceededForStep(stepID, laneID, moduleKey, deps, output, warnings)
|
||||
}
|
||||
return recorder.MergeSucceeded(laneID, moduleKey, deps, output, warnings)
|
||||
}
|
||||
func checkpointMergeRejected(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||
return stepAware.MergeRejectedForStep(stepID, laneID, moduleKey, deps, rejected)
|
||||
}
|
||||
return recorder.MergeRejected(laneID, moduleKey, deps, rejected)
|
||||
}
|
||||
func checkpointMergeFailed(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, err error) error {
|
||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||
return stepAware.MergeFailedForStep(stepID, laneID, moduleKey, deps, err)
|
||||
}
|
||||
return recorder.MergeFailed(laneID, moduleKey, deps, err)
|
||||
}
|
||||
func checkpointNormalizeRunning(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) error {
|
||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||
return stepAware.NormalizeRunningForStep(stepID, laneID, moduleKey, deps)
|
||||
}
|
||||
return recorder.NormalizeRunning(laneID, moduleKey, deps)
|
||||
}
|
||||
func checkpointNormalizeSucceeded(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||
return stepAware.NormalizeSucceededForStep(stepID, laneID, moduleKey, deps, output, warnings)
|
||||
}
|
||||
return recorder.NormalizeSucceeded(laneID, moduleKey, deps, output, warnings)
|
||||
}
|
||||
func checkpointNormalizeRejected(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||
return stepAware.NormalizeRejectedForStep(stepID, laneID, moduleKey, deps, rejected)
|
||||
}
|
||||
return recorder.NormalizeRejected(laneID, moduleKey, deps, rejected)
|
||||
}
|
||||
func checkpointNormalizeFailed(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, err error) error {
|
||||
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
|
||||
return stepAware.NormalizeFailedForStep(stepID, laneID, moduleKey, deps, err)
|
||||
}
|
||||
return recorder.NormalizeFailed(laneID, moduleKey, deps, err)
|
||||
}
|
||||
|
||||
func digestFingerprints(name string, digest string) []CheckpointFingerprint {
|
||||
|
||||
@@ -63,6 +63,7 @@ func debugPathComponent(value string) string {
|
||||
|
||||
type debugTimedEnvelope struct {
|
||||
Stage string `json:"stage,omitempty"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key,omitempty"`
|
||||
Attempt int `json:"attempt,omitempty"`
|
||||
@@ -124,6 +125,7 @@ type debugChunkRange struct {
|
||||
}
|
||||
|
||||
type debugSerializedOutput struct {
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id"`
|
||||
NormalizerKey string `json:"normalizer_key"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
@@ -556,7 +558,7 @@ func debugSerializedOutputEnvelope(output contracts.SerializedOutput) debugSeria
|
||||
content := debugContentEnvelope(output.Artifact.Content, output.Artifact.MediaType, output.Artifact.Metadata, nil)
|
||||
content.ContentDigest = debugContentDigest(output.Artifact.Content)
|
||||
return debugSerializedOutput{
|
||||
LaneID: output.LaneID, NormalizerKey: output.NormalizerKey, SourceID: output.SourceID,
|
||||
StepID: output.StepID, LaneID: output.LaneID, NormalizerKey: output.NormalizerKey, SourceID: output.SourceID,
|
||||
Kind: output.Artifact.Kind, Schema: schema, SchemaDigest: digest,
|
||||
Content: content,
|
||||
}
|
||||
|
||||
287
internal/framework/pipeline/handoff.go
Normal file
287
internal/framework/pipeline/handoff.go
Normal file
@@ -0,0 +1,287 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type referenceTargetKey struct {
|
||||
StepID string
|
||||
LaneID string
|
||||
Stage ModuleStage
|
||||
}
|
||||
|
||||
func keyForReferenceTarget(target ResolvedReferenceTarget) referenceTargetKey {
|
||||
return referenceTargetKey{StepID: target.StepID, LaneID: target.LaneID, Stage: target.Stage}
|
||||
}
|
||||
|
||||
func operationReferenceSet(input RunInput, target ResolvedReferenceTarget) contracts.ReferenceSet {
|
||||
if input.references != nil {
|
||||
if set, ok := input.references[keyForReferenceTarget(target)]; ok {
|
||||
return CloneReferenceSet(set)
|
||||
}
|
||||
}
|
||||
return CloneReferenceSet(target.ReferenceSet)
|
||||
}
|
||||
|
||||
// buildStepReferenceSets resolves every generated binding for a step before
|
||||
// any lane in that step is allowed to start. Each returned set is a fresh
|
||||
// operation-time view; prepared reference sets are never modified.
|
||||
func buildStepReferenceSets(input RunInput, step PreparedPipelineStep, outputs []contracts.SerializedOutput) (map[referenceTargetKey]contracts.ReferenceSet, []artifacts.ReferenceProvenance, error) {
|
||||
if len(step.lanes) == 0 {
|
||||
return nil, nil, nil
|
||||
}
|
||||
sets := make(map[referenceTargetKey]contracts.ReferenceSet)
|
||||
var provenance []artifacts.ReferenceProvenance
|
||||
canonical := make(map[string]contracts.ReferenceItem)
|
||||
for _, prepared := range step.lanes {
|
||||
lane := prepared.resolved
|
||||
for _, target := range []ResolvedReferenceTarget{lane.ExtractReferences, lane.MergeReferences, lane.NormalizeReferences} {
|
||||
generated := false
|
||||
resolved := CloneReferenceSet(target.ReferenceSet)
|
||||
for _, binding := range target.Bindings {
|
||||
if binding.Artifact == nil {
|
||||
continue
|
||||
}
|
||||
generated = true
|
||||
item, err := generatedReferenceItem(input, binding, outputs, canonical)
|
||||
if err != nil {
|
||||
return nil, nil, contextualHandoffError(input, target, binding, err)
|
||||
}
|
||||
slotName := strings.TrimSpace(binding.SlotName)
|
||||
slot, ok := resolved.Slots[slotName]
|
||||
if !ok {
|
||||
return nil, nil, contextualHandoffError(input, target, binding, fmt.Errorf("reference slot %q has no resolved declaration", slotName))
|
||||
}
|
||||
if len(slot.Items) > 0 {
|
||||
return nil, nil, contextualHandoffError(input, target, binding, fmt.Errorf("reference slot %q already contains materialized items", slotName))
|
||||
}
|
||||
if len(slot.Slot.AcceptedArtifactKinds) == 0 || !containsArtifactKind(slot.Slot.AcceptedArtifactKinds, item.ArtifactKind) {
|
||||
return nil, nil, contextualHandoffError(input, target, binding, fmt.Errorf("reference slot %q does not accept artifact kind %q", slotName, item.ArtifactKind))
|
||||
}
|
||||
if !referenceMediaTypeAccepted(item.MediaType, slot.Slot.AcceptedMediaTypes) {
|
||||
return nil, nil, contextualHandoffError(input, target, binding, fmt.Errorf("reference slot %q does not accept media type %q", slotName, item.MediaType))
|
||||
}
|
||||
if slot.Slot.MaxBytes > 0 && item.SizeBytes > slot.Slot.MaxBytes {
|
||||
return nil, nil, contextualHandoffError(input, target, binding, fmt.Errorf("reference slot %q is %d bytes, limit %d", slotName, item.SizeBytes, slot.Slot.MaxBytes))
|
||||
}
|
||||
slot.Items = []contracts.ReferenceItem{contracts.CloneReferenceItem(item)}
|
||||
resolved.Slots[slotName] = slot
|
||||
}
|
||||
if !generated {
|
||||
continue
|
||||
}
|
||||
key := keyForReferenceTarget(target)
|
||||
sets[key] = resolved
|
||||
for _, item := range generatedItems(resolved) {
|
||||
provenance = append(provenance, referenceProvenanceForItem(target, item))
|
||||
}
|
||||
}
|
||||
}
|
||||
return sets, provenance, nil
|
||||
}
|
||||
|
||||
func containsArtifactKind(kinds []contracts.ArtifactKind, want contracts.ArtifactKind) bool {
|
||||
for _, kind := range kinds {
|
||||
if kind == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func contextualHandoffError(input RunInput, target ResolvedReferenceTarget, binding ReferenceBinding, err error) error {
|
||||
return fmt.Errorf("pipeline %q step %q lane %q %s generated dependency %q from %q/%q: %w", input.pipeline.ID, target.StepID, target.LaneID, target.Stage, binding.SlotName, binding.Artifact.Step, binding.Artifact.Lane, err)
|
||||
}
|
||||
|
||||
func generatedReferenceItem(input RunInput, binding ReferenceBinding, outputs []contracts.SerializedOutput, cache map[string]contracts.ReferenceItem) (contracts.ReferenceItem, error) {
|
||||
selector := binding.Artifact
|
||||
if selector == nil {
|
||||
return contracts.ReferenceItem{}, fmt.Errorf("generated reference selector must not be nil")
|
||||
}
|
||||
stepID := strings.TrimSpace(selector.Step)
|
||||
laneID := strings.TrimSpace(selector.Lane)
|
||||
cacheKey := stepID + "\x00" + laneID
|
||||
if item, ok := cache[cacheKey]; ok {
|
||||
item.SlotName = strings.TrimSpace(binding.SlotName)
|
||||
item.BindingSource = strings.TrimSpace(binding.BindingSource)
|
||||
return contracts.CloneReferenceItem(item), nil
|
||||
}
|
||||
matches := make([]contracts.SerializedOutput, 0, 1)
|
||||
for _, output := range outputs {
|
||||
if strings.TrimSpace(output.StepID) == stepID && strings.TrimSpace(output.LaneID) == laneID {
|
||||
matches = append(matches, output)
|
||||
}
|
||||
}
|
||||
if len(matches) == 0 {
|
||||
return contracts.ReferenceItem{}, fmt.Errorf("producer has no accepted normalized output")
|
||||
}
|
||||
if len(matches) > 1 {
|
||||
return contracts.ReferenceItem{}, fmt.Errorf("producer has %d accepted normalized outputs; exactly one is required", len(matches))
|
||||
}
|
||||
producer, ok := findResolvedLane(input.pipeline, stepID, laneID)
|
||||
if !ok {
|
||||
return contracts.ReferenceItem{}, fmt.Errorf("producer lane is not present in the resolved pipeline")
|
||||
}
|
||||
if input.Prepared == nil || input.Prepared.artifactCodecs == nil {
|
||||
return contracts.ReferenceItem{}, fmt.Errorf("artifact codec registry is unavailable")
|
||||
}
|
||||
serialized := contracts.CloneSerializedArtifact(matches[0].Artifact)
|
||||
if serialized.Kind != producer.ArtifactKind {
|
||||
return contracts.ReferenceItem{}, fmt.Errorf("producer artifact kind %q does not match resolved kind %q", serialized.Kind, producer.ArtifactKind)
|
||||
}
|
||||
value, err := input.Prepared.artifactCodecs.Decode(serialized)
|
||||
if err != nil {
|
||||
return contracts.ReferenceItem{}, fmt.Errorf("decode producer artifact: %w", err)
|
||||
}
|
||||
canonical, err := input.Prepared.artifactCodecs.Encode(producer.ArtifactKind, value)
|
||||
if err != nil {
|
||||
return contracts.ReferenceItem{}, fmt.Errorf("canonicalize producer artifact: %w", err)
|
||||
}
|
||||
expected, ok := input.Prepared.artifactCodecs.Spec(producer.ArtifactKind)
|
||||
if !ok {
|
||||
return contracts.ReferenceItem{}, fmt.Errorf("producer artifact codec %q is not registered", producer.ArtifactKind)
|
||||
}
|
||||
if canonical.Kind != expected.Kind || canonical.Schema.ID != expected.Schema.ID || canonical.Schema.Name != expected.Schema.Name || canonical.Schema.Version != expected.Schema.Version || contracts.DigestArtifactSchema(canonical.Schema) != expected.SchemaDigest || canonical.MediaType != expected.MediaType {
|
||||
return contracts.ReferenceItem{}, fmt.Errorf("canonical producer artifact does not match registered codec identity")
|
||||
}
|
||||
item := contracts.ReferenceItem{
|
||||
SlotName: strings.TrimSpace(binding.SlotName),
|
||||
MediaType: canonical.MediaType,
|
||||
Content: append([]byte(nil), canonical.Content...),
|
||||
Digest: referenceDigest(canonical.Content),
|
||||
Origin: contracts.ReferenceOrigin{Type: "generated"},
|
||||
SizeBytes: int64(len(canonical.Content)),
|
||||
BindingSource: strings.TrimSpace(binding.BindingSource),
|
||||
ArtifactKind: canonical.Kind,
|
||||
ArtifactSchema: contracts.CloneArtifactSchema(canonical.Schema),
|
||||
Producer: contracts.ReferenceProducer{
|
||||
PipelineID: input.pipeline.ID,
|
||||
StepID: stepID,
|
||||
LaneID: laneID,
|
||||
ModuleKey: producer.Normalize.Module,
|
||||
},
|
||||
}
|
||||
cacheItem := contracts.CloneReferenceItem(item)
|
||||
cacheItem.SlotName = ""
|
||||
cacheItem.BindingSource = ""
|
||||
cache[cacheKey] = cacheItem
|
||||
return item, nil
|
||||
}
|
||||
|
||||
func findResolvedLane(pipeline ResolvedPipeline, stepID, laneID string) (ResolvedArtifactLane, bool) {
|
||||
for _, step := range pipeline.Steps {
|
||||
if step.ID != stepID {
|
||||
continue
|
||||
}
|
||||
for _, lane := range step.ArtifactLanes {
|
||||
if lane.ID == laneID {
|
||||
return lane, true
|
||||
}
|
||||
}
|
||||
}
|
||||
return ResolvedArtifactLane{}, false
|
||||
}
|
||||
|
||||
func generatedItems(set contracts.ReferenceSet) []contracts.ReferenceItem {
|
||||
var items []contracts.ReferenceItem
|
||||
keys := make([]string, 0, len(set.Slots))
|
||||
for key := range set.Slots {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, key := range keys {
|
||||
for _, item := range set.Slots[key].Items {
|
||||
if item.Origin.Type == "generated" {
|
||||
items = append(items, item)
|
||||
}
|
||||
}
|
||||
}
|
||||
return items
|
||||
}
|
||||
|
||||
func referenceProvenanceForItem(target ResolvedReferenceTarget, item contracts.ReferenceItem) artifacts.ReferenceProvenance {
|
||||
return artifacts.ReferenceProvenance{
|
||||
Stage: string(target.Stage),
|
||||
StepID: target.StepID,
|
||||
LaneID: target.LaneID,
|
||||
SlotName: item.SlotName,
|
||||
OriginType: item.Origin.Type,
|
||||
OriginURI: item.Origin.URI,
|
||||
Digest: item.Digest,
|
||||
MediaType: item.MediaType,
|
||||
SizeBytes: item.SizeBytes,
|
||||
BindingSource: item.BindingSource,
|
||||
ArtifactKind: string(item.ArtifactKind),
|
||||
SchemaID: item.ArtifactSchema.ID,
|
||||
SchemaName: item.ArtifactSchema.Name,
|
||||
SchemaVersion: item.ArtifactSchema.Version,
|
||||
SchemaDigest: contracts.DigestArtifactSchema(item.ArtifactSchema),
|
||||
ProducerPipeline: item.Producer.PipelineID,
|
||||
ProducerStep: item.Producer.StepID,
|
||||
ProducerLane: item.Producer.LaneID,
|
||||
ProducerModule: item.Producer.ModuleKey,
|
||||
}
|
||||
}
|
||||
|
||||
type generatedReferenceFingerprintIdentity struct {
|
||||
ProducerPipeline string `json:"producer_pipeline_id"`
|
||||
ProducerStep string `json:"producer_step_id"`
|
||||
ProducerLane string `json:"producer_lane_id"`
|
||||
ProducerModule string `json:"producer_module_key"`
|
||||
ArtifactKind string `json:"artifact_kind"`
|
||||
SchemaID string `json:"schema_id"`
|
||||
SchemaName string `json:"schema_name"`
|
||||
SchemaVersion string `json:"schema_version"`
|
||||
SchemaDigest string `json:"schema_digest"`
|
||||
MediaType string `json:"media_type"`
|
||||
ContentDigest string `json:"content_digest"`
|
||||
}
|
||||
|
||||
// generatedReferenceDependencies returns the canonical semantic dependency
|
||||
// identity that receiving operation checkpoints must include.
|
||||
func generatedReferenceDependencies(set contracts.ReferenceSet) []CheckpointFingerprint {
|
||||
var fingerprints []CheckpointFingerprint
|
||||
keys := make([]string, 0, len(set.Slots))
|
||||
for key := range set.Slots {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, slotName := range keys {
|
||||
for index, item := range set.Slots[slotName].Items {
|
||||
if item.Origin.Type != "generated" {
|
||||
continue
|
||||
}
|
||||
identity := generatedReferenceFingerprintIdentity{
|
||||
ProducerPipeline: item.Producer.PipelineID,
|
||||
ProducerStep: item.Producer.StepID,
|
||||
ProducerLane: item.Producer.LaneID,
|
||||
ProducerModule: item.Producer.ModuleKey,
|
||||
ArtifactKind: string(item.ArtifactKind),
|
||||
SchemaID: item.ArtifactSchema.ID,
|
||||
SchemaName: item.ArtifactSchema.Name,
|
||||
SchemaVersion: item.ArtifactSchema.Version,
|
||||
SchemaDigest: contracts.DigestArtifactSchema(item.ArtifactSchema),
|
||||
MediaType: item.MediaType,
|
||||
ContentDigest: item.Digest,
|
||||
}
|
||||
encoded, err := json.Marshal(identity)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
sum := sha256.Sum256(encoded)
|
||||
fingerprints = append(fingerprints, CheckpointFingerprint{
|
||||
Name: fmt.Sprintf("generated-reference:%s:%d", slotName, index),
|
||||
Value: "sha256:" + hex.EncodeToString(sum[:]),
|
||||
})
|
||||
}
|
||||
}
|
||||
return normalizeCheckpointFingerprints(fingerprints)
|
||||
}
|
||||
287
internal/framework/pipeline/handoff_test.go
Normal file
287
internal/framework/pipeline/handoff_test.go
Normal file
@@ -0,0 +1,287 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func handoffFixture(t *testing.T, value codecNotes) (RunInput, PreparedPipelineStep, contracts.SerializedOutput) {
|
||||
t.Helper()
|
||||
prepared := preparedOrderedPipeline(t, 1,
|
||||
orderedLaneSpec{id: "notes", profile: "notes"},
|
||||
orderedLaneSpec{id: "score", profile: "score"},
|
||||
)
|
||||
consumer := &prepared.Steps[1].lanes[0]
|
||||
installGeneratedTarget := func(target *ResolvedReferenceTarget, stage ModuleStage, module string) {
|
||||
*target = ResolvedReferenceTarget{
|
||||
Stage: stage,
|
||||
StepID: consumer.resolved.StepID,
|
||||
LaneID: consumer.resolved.ID,
|
||||
Module: module,
|
||||
Bindings: []ReferenceBinding{{
|
||||
Stage: stage,
|
||||
LaneID: consumer.resolved.ID,
|
||||
SlotName: "producer-output",
|
||||
Artifact: &ArtifactReference{Step: "step-1", Lane: "notes"},
|
||||
}},
|
||||
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||
"producer-output": {Slot: contracts.ReferenceSlot{
|
||||
Name: "producer-output",
|
||||
AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"},
|
||||
AcceptedMediaTypes: []string{"application/json"},
|
||||
}},
|
||||
}},
|
||||
}
|
||||
}
|
||||
installGeneratedTarget(&consumer.resolved.ExtractReferences, StageExtract, consumer.resolved.Extract.Module)
|
||||
installGeneratedTarget(&consumer.resolved.MergeReferences, StageMerge, consumer.resolved.Merge.Module)
|
||||
installGeneratedTarget(&consumer.resolved.NormalizeReferences, StageNormalize, consumer.resolved.Normalize.Module)
|
||||
|
||||
artifact, err := checkpointArtifact(prepared.Steps[0].lanes[0].typed.codec, "notes", "typed/normalize", "source", value)
|
||||
if err != nil {
|
||||
t.Fatalf("checkpointArtifact() error = %v", err)
|
||||
}
|
||||
input := RunInput{Prepared: prepared}
|
||||
input.pipeline = prepared.resolved
|
||||
return input, prepared.Steps[1], contracts.SerializedOutput{
|
||||
StepID: "step-1",
|
||||
LaneID: "notes",
|
||||
NormalizerKey: "typed/normalize",
|
||||
SourceID: "source",
|
||||
Artifact: contracts.CloneSerializedArtifact(artifact.Artifact),
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildStepReferenceSetsCanonicalizesAndClonesFanout(t *testing.T) {
|
||||
input, step, producerOutput := handoffFixture(t, codecNotes{Items: []string{"first"}})
|
||||
sets, provenance, err := buildStepReferenceSets(input, step, []contracts.SerializedOutput{producerOutput})
|
||||
if err != nil {
|
||||
t.Fatalf("buildStepReferenceSets() error = %v", err)
|
||||
}
|
||||
if got, want := len(sets), 3; got != want {
|
||||
t.Fatalf("reference target set count = %d, want %d", got, want)
|
||||
}
|
||||
if got, want := len(provenance), 3; got != want {
|
||||
t.Fatalf("generated provenance count = %d, want %d", got, want)
|
||||
}
|
||||
|
||||
var first []byte
|
||||
for key, set := range sets {
|
||||
item := set.Slots["producer-output"].Items[0]
|
||||
if item.Origin.Type != "generated" || item.Origin.URI != "" {
|
||||
t.Fatalf("target %v origin = %#v, want generated origin without URI", key, item.Origin)
|
||||
}
|
||||
if item.Producer.StepID != "step-1" || item.Producer.LaneID != "notes" || item.Producer.ModuleKey != "typed/normalize" {
|
||||
t.Fatalf("target %v producer = %#v, want producer provenance", key, item.Producer)
|
||||
}
|
||||
if first == nil {
|
||||
first = item.Content
|
||||
continue
|
||||
}
|
||||
if &first[0] == &item.Content[0] {
|
||||
t.Fatalf("target %v shares generated content backing storage", key)
|
||||
}
|
||||
}
|
||||
if got := len(step.lanes[0].resolved.ExtractReferences.ReferenceSet.Slots["producer-output"].Items); got != 0 {
|
||||
t.Fatalf("prepared extract reference items = %d, want zero", got)
|
||||
}
|
||||
|
||||
encoded, err := json.Marshal(provenance)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal generated provenance: %v", err)
|
||||
}
|
||||
text := string(encoded)
|
||||
for _, forbidden := range []string{"first", "file://", "content_base64"} {
|
||||
if strings.Contains(text, forbidden) {
|
||||
t.Fatalf("generated provenance contains forbidden %q: %s", forbidden, text)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(text, "producer_pipeline_id") || !strings.Contains(text, "schema_digest") {
|
||||
t.Fatalf("generated provenance = %s, want bounded producer and schema identity", text)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGeneratedReferenceAcceptsTypedEmptyCollection(t *testing.T) {
|
||||
input, step, producerOutput := handoffFixture(t, codecNotes{})
|
||||
sets, _, err := buildStepReferenceSets(input, step, []contracts.SerializedOutput{producerOutput})
|
||||
if err != nil {
|
||||
t.Fatalf("buildStepReferenceSets() error = %v, want nil for accepted empty collection", err)
|
||||
}
|
||||
if got := len(sets[keyForReferenceTarget(step.lanes[0].resolved.ExtractReferences)].Slots["producer-output"].Items); got != 1 {
|
||||
t.Fatalf("generated empty collection item count = %d, want one artifact", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGeneratedReferenceRejectsInvalidProducerOutputsDeterministically(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
outputs func(contracts.SerializedOutput) []contracts.SerializedOutput
|
||||
mutate func(*ResolvedReferenceTarget, *contracts.SerializedOutput)
|
||||
want string
|
||||
}{
|
||||
{name: "missing", outputs: func(contracts.SerializedOutput) []contracts.SerializedOutput { return nil }, want: "no accepted normalized output"},
|
||||
{name: "multiple", outputs: func(output contracts.SerializedOutput) []contracts.SerializedOutput {
|
||||
return []contracts.SerializedOutput{output, output}
|
||||
}, want: "exactly one is required"},
|
||||
{name: "kind mismatch", outputs: func(output contracts.SerializedOutput) []contracts.SerializedOutput {
|
||||
return []contracts.SerializedOutput{output}
|
||||
}, mutate: func(target *ResolvedReferenceTarget, _ *contracts.SerializedOutput) {
|
||||
target.ReferenceSet.Slots["producer-output"] = contracts.ResolvedReferenceSlot{Slot: contracts.ReferenceSlot{Name: "producer-output", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/score"}}}
|
||||
}, want: "does not accept artifact kind"},
|
||||
{name: "schema mismatch", outputs: func(output contracts.SerializedOutput) []contracts.SerializedOutput {
|
||||
return []contracts.SerializedOutput{output}
|
||||
}, mutate: func(_ *ResolvedReferenceTarget, output *contracts.SerializedOutput) {
|
||||
output.Artifact.Schema.ID = "wrong.schema"
|
||||
}, want: "schema identity"},
|
||||
{name: "media mismatch", outputs: func(output contracts.SerializedOutput) []contracts.SerializedOutput {
|
||||
return []contracts.SerializedOutput{output}
|
||||
}, mutate: func(_ *ResolvedReferenceTarget, output *contracts.SerializedOutput) {
|
||||
output.Artifact.MediaType = "text/plain"
|
||||
}, want: "media type"},
|
||||
{name: "size mismatch", outputs: func(output contracts.SerializedOutput) []contracts.SerializedOutput {
|
||||
return []contracts.SerializedOutput{output}
|
||||
}, mutate: func(target *ResolvedReferenceTarget, _ *contracts.SerializedOutput) {
|
||||
slot := target.ReferenceSet.Slots["producer-output"]
|
||||
slot.Slot.MaxBytes = 1
|
||||
target.ReferenceSet.Slots["producer-output"] = slot
|
||||
}, want: "limit"},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
input, step, output := handoffFixture(t, codecNotes{Items: []string{"first"}})
|
||||
output = contracts.CloneSerializedOutput(output)
|
||||
if test.mutate != nil {
|
||||
target := &step.lanes[0].resolved.ExtractReferences
|
||||
test.mutate(target, &output)
|
||||
}
|
||||
_, _, err := buildStepReferenceSets(input, step, test.outputs(output))
|
||||
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("buildStepReferenceSets() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGeneratedReferenceFingerprintChangesWithCanonicalContent(t *testing.T) {
|
||||
input, step, first := handoffFixture(t, codecNotes{Items: []string{"first"}})
|
||||
firstSets, _, err := buildStepReferenceSets(input, step, []contracts.SerializedOutput{first})
|
||||
if err != nil {
|
||||
t.Fatalf("build first reference set: %v", err)
|
||||
}
|
||||
_, _, second := handoffFixture(t, codecNotes{Items: []string{"second"}})
|
||||
secondSets, _, err := buildStepReferenceSets(input, step, []contracts.SerializedOutput{second})
|
||||
if err != nil {
|
||||
t.Fatalf("build second reference set: %v", err)
|
||||
}
|
||||
firstDeps := generatedReferenceDependencies(firstSets[keyForReferenceTarget(step.lanes[0].resolved.ExtractReferences)])
|
||||
secondDeps := generatedReferenceDependencies(secondSets[keyForReferenceTarget(step.lanes[0].resolved.ExtractReferences)])
|
||||
if reflect.DeepEqual(firstDeps, secondDeps) {
|
||||
t.Fatalf("generated dependencies = %#v, want content-sensitive fingerprint", firstDeps)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerHandsOffAcceptedNormalizedOutputBeforeConsumerLanes(t *testing.T) {
|
||||
input, _, _ := handoffFixture(t, codecNotes{Items: []string{"first"}})
|
||||
prepared := input.Prepared
|
||||
var received contracts.ReferenceSet
|
||||
prepared.Steps[0].lanes[0].typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"first"}}}, nil
|
||||
}
|
||||
prepared.Steps[1].lanes[0].typed.extract = func(_ context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
received = CloneReferenceSet(request.References)
|
||||
return erasedTypedResult{Value: codecScore{Value: 3}}, nil
|
||||
}
|
||||
|
||||
output, err := New().Run(context.Background(), input)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if got, want := len(output.NormalizeOutputs), 2; got != want {
|
||||
t.Fatalf("normalized output count = %d, want %d", got, want)
|
||||
}
|
||||
if output.NormalizeOutputs[0].StepID != "step-1" || output.NormalizeOutputs[1].StepID != "step-2" {
|
||||
t.Fatalf("normalized output step IDs = %#v, want step-1 and step-2", output.NormalizeOutputs)
|
||||
}
|
||||
item := received.Slots["producer-output"].Items[0]
|
||||
if item.Origin.Type != "generated" || item.Producer.StepID != "step-1" || item.Producer.LaneID != "notes" {
|
||||
t.Fatalf("consumer reference = %#v, want generated producer reference", item)
|
||||
}
|
||||
if len(output.Manifest.References) == 0 || output.Manifest.References[len(output.Manifest.References)-1].OriginType != "generated" {
|
||||
t.Fatalf("manifest references = %#v, want generated provenance", output.Manifest.References)
|
||||
}
|
||||
}
|
||||
|
||||
type handoffDependencyLoader struct {
|
||||
CheckpointLoader
|
||||
extract [][]CheckpointFingerprint
|
||||
merge [][]CheckpointFingerprint
|
||||
normalize [][]CheckpointFingerprint
|
||||
}
|
||||
|
||||
func (l *handoffDependencyLoader) Enabled() bool { return true }
|
||||
|
||||
func (l *handoffDependencyLoader) Extract(_ string, _ string, dependencies []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||
l.extract = append(l.extract, append([]CheckpointFingerprint(nil), dependencies...))
|
||||
return ExtractCheckpoint{}, CheckpointDecision{Reason: "not found"}
|
||||
}
|
||||
|
||||
func (l *handoffDependencyLoader) Merge(_ string, _ string, dependencies []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
|
||||
l.merge = append(l.merge, append([]CheckpointFingerprint(nil), dependencies...))
|
||||
return MergeCheckpoint{}, CheckpointDecision{Reason: "not found"}
|
||||
}
|
||||
|
||||
func (l *handoffDependencyLoader) Normalize(_ string, _ string, dependencies []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
l.normalize = append(l.normalize, append([]CheckpointFingerprint(nil), dependencies...))
|
||||
return NormalizeCheckpoint{}, CheckpointDecision{Reason: "not found"}
|
||||
}
|
||||
|
||||
func runHandoffWithProducerValue(t *testing.T, value codecNotes) *handoffDependencyLoader {
|
||||
t.Helper()
|
||||
input, _, _ := handoffFixture(t, value)
|
||||
prepared := input.Prepared
|
||||
prepared.Steps[0].lanes[0].typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: value}, nil
|
||||
}
|
||||
prepared.Steps[0].lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: value}, nil
|
||||
}
|
||||
prepared.Steps[1].lanes[0].typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: codecScore{Value: 3}}, nil
|
||||
}
|
||||
loader := &handoffDependencyLoader{CheckpointLoader: NoopCheckpointLoader()}
|
||||
if _, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: loader}); err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
return loader
|
||||
}
|
||||
|
||||
func TestRunnerAddsGeneratedDependencyToEveryReceivingCheckpoint(t *testing.T) {
|
||||
first := runHandoffWithProducerValue(t, codecNotes{Items: []string{"first"}})
|
||||
second := runHandoffWithProducerValue(t, codecNotes{Items: []string{"second"}})
|
||||
if len(first.extract) != 2 || len(first.merge) != 2 || len(first.normalize) != 2 {
|
||||
t.Fatalf("checkpoint load calls = extract %d merge %d normalize %d, want two each", len(first.extract), len(first.merge), len(first.normalize))
|
||||
}
|
||||
for name, calls := range map[string][][]CheckpointFingerprint{"extract": first.extract, "merge": first.merge, "normalize": first.normalize} {
|
||||
if got := generatedFingerprintCount(calls[1]); got != 1 {
|
||||
t.Fatalf("%s consumer dependency count = %d, want one", name, got)
|
||||
}
|
||||
}
|
||||
if reflect.DeepEqual(first.extract[1], second.extract[1]) || reflect.DeepEqual(first.merge[1], second.merge[1]) || reflect.DeepEqual(first.normalize[1], second.normalize[1]) {
|
||||
t.Fatalf("consumer checkpoint dependencies did not change with producer content")
|
||||
}
|
||||
}
|
||||
|
||||
func generatedFingerprintCount(dependencies []CheckpointFingerprint) int {
|
||||
count := 0
|
||||
for _, dependency := range dependencies {
|
||||
if strings.HasPrefix(dependency.Name, "generated-reference:") {
|
||||
count++
|
||||
}
|
||||
}
|
||||
return count
|
||||
}
|
||||
@@ -133,6 +133,7 @@ func normalizeReferenceSlots(slots []contracts.ReferenceSlot) []contracts.Refere
|
||||
slot.Name = strings.TrimSpace(slot.Name)
|
||||
slot.Description = strings.TrimSpace(slot.Description)
|
||||
slot.AcceptedMediaTypes = normalizeStringSet(slot.AcceptedMediaTypes)
|
||||
slot.AcceptedArtifactKinds = normalizeArtifactKinds(slot.AcceptedArtifactKinds)
|
||||
normalized = append(normalized, slot)
|
||||
}
|
||||
sort.SliceStable(normalized, func(i, j int) bool {
|
||||
@@ -141,6 +142,34 @@ func normalizeReferenceSlots(slots []contracts.ReferenceSlot) []contracts.Refere
|
||||
return normalized
|
||||
}
|
||||
|
||||
func normalizeArtifactKinds(values []contracts.ArtifactKind) []contracts.ArtifactKind {
|
||||
if len(values) == 0 {
|
||||
return nil
|
||||
}
|
||||
seen := make(map[contracts.ArtifactKind]struct{}, len(values))
|
||||
containsEmpty := false
|
||||
for _, value := range values {
|
||||
value = normalizeArtifactKind(value)
|
||||
if value != "" {
|
||||
seen[value] = struct{}{}
|
||||
} else {
|
||||
containsEmpty = true
|
||||
}
|
||||
}
|
||||
if len(seen) == 0 && !containsEmpty {
|
||||
return nil
|
||||
}
|
||||
result := make([]contracts.ArtifactKind, 0, len(seen))
|
||||
for value := range seen {
|
||||
result = append(result, value)
|
||||
}
|
||||
if containsEmpty {
|
||||
result = append(result, "")
|
||||
}
|
||||
sort.Slice(result, func(i, j int) bool { return result[i] < result[j] })
|
||||
return result
|
||||
}
|
||||
|
||||
func normalizeStringSet(values []string) []string {
|
||||
if len(values) == 0 {
|
||||
return nil
|
||||
@@ -178,6 +207,11 @@ func validateReferenceSlots(slots []contracts.ReferenceSlot) error {
|
||||
if slot.MaxBytes < 0 {
|
||||
return fmt.Errorf("slot %q max_bytes must not be negative", slot.Name)
|
||||
}
|
||||
for _, kind := range slot.AcceptedArtifactKinds {
|
||||
if normalizeArtifactKind(kind) == "" {
|
||||
return fmt.Errorf("slot %q accepted artifact kind must not be empty", slot.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -13,7 +13,7 @@ func validateResolvedOptions(resolved ResolvedPipeline, catalog ModuleCatalog) e
|
||||
return err
|
||||
}
|
||||
|
||||
for _, lane := range resolved.ArtifactLanes {
|
||||
validateLane := func(lane ResolvedArtifactLane) error {
|
||||
if err := catalog.Extractors.validateOptions(lane.Extract.Module, lane.Extract.Options); err != nil {
|
||||
return moduleOptionsError(resolved.ID, lane.ID, StageExtract, lane.Extract.Module, err)
|
||||
}
|
||||
@@ -29,8 +29,13 @@ func validateResolvedOptions(resolved ResolvedPipeline, catalog ModuleCatalog) e
|
||||
if err := catalog.Normalizers.validateOptions(lane.Normalize.Module, lane.ArtifactKind, lane.Normalize.Options); err != nil {
|
||||
return moduleOptionsError(resolved.ID, lane.ID, StageNormalize, lane.Normalize.Module, err)
|
||||
}
|
||||
if err := validateChainOptions(resolved, catalog, StageNormalize, lane.ID, lane.Normalize.Module); err != nil {
|
||||
return err
|
||||
return validateChainOptions(resolved, catalog, StageNormalize, lane.ID, lane.Normalize.Module)
|
||||
}
|
||||
for _, step := range resolved.Steps {
|
||||
for _, lane := range step.ArtifactLanes {
|
||||
if err := validateLane(lane); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -67,11 +67,71 @@ func TestPrepareConstructsEverythingInStableOrder(t *testing.T) {
|
||||
if !reflect.DeepEqual(built, want) {
|
||||
t.Fatalf("construction order = %#v, want %#v", built, want)
|
||||
}
|
||||
if prepared.Input.Module != "input" || prepared.Chunk.Module != "chunk" || prepared.Output.Module != "output" || len(prepared.ArtifactLanes) != 1 {
|
||||
if prepared.Input.Module != "input" || prepared.Chunk.Module != "chunk" || prepared.Output.Module != "output" || len(prepared.Steps) != 1 || len(prepared.Steps[0].ArtifactLanes) != 1 {
|
||||
t.Fatalf("PreparedPipeline = %#v, want explicit resolved components", prepared)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrepareBuildsEveryOrderedStepBeforeExecution(t *testing.T) {
|
||||
var built []string
|
||||
registries, input := constructionRegistries(t, &built, nil)
|
||||
profile := constructionProfile()
|
||||
lane := profile.Artifacts["artifact"]
|
||||
profile.Artifacts = nil
|
||||
profile.Steps = []PipelineStepProfile{
|
||||
{ID: "first", Artifacts: map[string]ArtifactLaneProfile{"first-artifact": lane}},
|
||||
{ID: "second", Artifacts: map[string]ArtifactLaneProfile{"second-artifact": lane}},
|
||||
}
|
||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, registries.catalog())
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
prepared, err := Prepare(resolved, registries, ModuleDependencies{})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||
}
|
||||
want := []string{
|
||||
"input", "chunk", "validator",
|
||||
"extract", "validator", "merge", "validator", "normalize", "validator",
|
||||
"extract", "validator", "merge", "validator", "normalize", "validator",
|
||||
"output",
|
||||
}
|
||||
if !reflect.DeepEqual(built, want) {
|
||||
t.Fatalf("construction order = %#v, want %#v", built, want)
|
||||
}
|
||||
if len(prepared.Steps) != 2 || len(prepared.Steps[0].lanes) != 1 || len(prepared.Steps[1].lanes) != 1 {
|
||||
t.Fatalf("prepared steps = %#v, want two constructed steps", prepared.Steps)
|
||||
}
|
||||
if len(input.requests) != 0 {
|
||||
t.Fatalf("input Parse calls = %d, want zero during preparation", len(input.requests))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrepareRetainsGeneratedSelectorsWithoutReferenceBytes(t *testing.T) {
|
||||
var built []string
|
||||
registries, _ := constructionRegistries(t, &built, nil)
|
||||
resolved, err := ResolvePipeline(constructionProfile(), ResolveOptions{}, registries.catalog())
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings = []ReferenceBinding{{
|
||||
Stage: StageExtract,
|
||||
LaneID: "artifact",
|
||||
SlotName: "generated",
|
||||
Artifact: &ArtifactReference{Step: "producer", Lane: "artifact"},
|
||||
}}
|
||||
prepared, err := Prepare(resolved, registries, ModuleDependencies{})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||
}
|
||||
if len(prepared.Steps[0].ArtifactLanes[0].Resolved.ExtractReferences.ReferenceSet.Slots) != 0 {
|
||||
t.Fatalf("prepared generated reference set = %#v, want no reference bytes", prepared.Steps[0].ArtifactLanes[0].Resolved.ExtractReferences.ReferenceSet)
|
||||
}
|
||||
if len(built) == 0 {
|
||||
t.Fatal("constructed modules = empty, want preparation to construct the selected components")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPreparedCheckpointFingerprintsCollectEveryComponentInStableScopeOrder(t *testing.T) {
|
||||
provider := func(value string) checkpointFingerprintTestProvider {
|
||||
return checkpointFingerprintTestProvider{{Name: "identity", Value: value}}
|
||||
@@ -90,6 +150,7 @@ func TestPreparedCheckpointFingerprintsCollectEveryComponentInStableScopeOrder(t
|
||||
chunk: provider("chunk-validator"),
|
||||
}}},
|
||||
output: provider("output"),
|
||||
Steps: []PreparedPipelineStep{{ID: "default"}},
|
||||
}
|
||||
lane := preparedLaneExecutor{
|
||||
resolved: ResolvedArtifactLane{
|
||||
@@ -107,7 +168,7 @@ func TestPreparedCheckpointFingerprintsCollectEveryComponentInStableScopeOrder(t
|
||||
mergeValidators: fingerprintTestValidatorChain("merge-validator", provider("merge-validator")),
|
||||
normalizeValidators: fingerprintTestValidatorChain("normalize-validator", provider("normalize-validator")),
|
||||
}
|
||||
prepared.lanes = []preparedLaneExecutor{lane}
|
||||
prepared.Steps[0].lanes = []preparedLaneExecutor{lane}
|
||||
|
||||
first, err := collectPreparedCheckpointFingerprints(prepared)
|
||||
if err != nil {
|
||||
@@ -254,9 +315,9 @@ func TestPrepareDeliversTargetReferencesAsIndependentBuildInputs(t *testing.T) {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
resolved.ChunkReferences.ReferenceSet = constructionReferenceSet("chunk", "chunk reference")
|
||||
resolved.ArtifactLanes[0].ExtractReferences.ReferenceSet = constructionReferenceSet("extract", "extract reference")
|
||||
resolved.ArtifactLanes[0].MergeReferences.ReferenceSet = constructionReferenceSet("merge", "merge reference")
|
||||
resolved.ArtifactLanes[0].NormalizeReferences.ReferenceSet = constructionReferenceSet("normalize", "normalize reference")
|
||||
resolved.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet = constructionReferenceSet("extract", "extract reference")
|
||||
resolved.Steps[0].ArtifactLanes[0].MergeReferences.ReferenceSet = constructionReferenceSet("merge", "merge reference")
|
||||
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.ReferenceSet = constructionReferenceSet("normalize", "normalize reference")
|
||||
|
||||
prepared, err := Prepare(resolved, registries, ModuleDependencies{})
|
||||
if err != nil {
|
||||
@@ -275,12 +336,12 @@ func TestPrepareDeliversTargetReferencesAsIndependentBuildInputs(t *testing.T) {
|
||||
t.Errorf("build request %d (%s) reference content = %q, want %q", i, observations[i].Name, got, want)
|
||||
}
|
||||
}
|
||||
if got := constructionReferenceContent(resolved.ArtifactLanes[0].ExtractReferences.ReferenceSet); got != "extract reference" {
|
||||
if got := constructionReferenceContent(resolved.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet); got != "extract reference" {
|
||||
t.Fatalf("resolved extract references = %q, want original content", got)
|
||||
}
|
||||
|
||||
_, err = prepared.lanes[0].typed.extract(context.Background(), prepared.lanes[0].typed.extractor, contracts.TypedExtractionRequest{
|
||||
References: CloneReferenceSet(resolved.ArtifactLanes[0].ExtractReferences.ReferenceSet),
|
||||
_, err = prepared.Steps[0].lanes[0].typed.extract(context.Background(), prepared.Steps[0].lanes[0].typed.extractor, contracts.TypedExtractionRequest{
|
||||
References: CloneReferenceSet(resolved.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("prepared extractor operation error = %v, want nil", err)
|
||||
|
||||
@@ -12,21 +12,27 @@ import (
|
||||
// resolved pipeline. Its implementation values are private so execution cannot
|
||||
// replace or reconfigure them after preparation.
|
||||
type PreparedPipeline struct {
|
||||
Input ModuleBinding
|
||||
Chunk ModuleBinding
|
||||
ArtifactLanes []PreparedArtifactLane
|
||||
Output ModuleBinding
|
||||
Input ModuleBinding
|
||||
Chunk ModuleBinding
|
||||
Steps []PreparedPipelineStep
|
||||
Output ModuleBinding
|
||||
|
||||
resolved ResolvedPipeline
|
||||
dependencies ModuleDependencies
|
||||
input contracts.InputAdapter
|
||||
chunker contracts.Chunker
|
||||
chunkValidators preparedValidatorChain
|
||||
lanes []preparedLaneExecutor
|
||||
output contracts.OutputEncoder
|
||||
artifactCodecs *ArtifactCodecRegistry
|
||||
checkpointFingerprints []CheckpointFingerprint
|
||||
}
|
||||
|
||||
type PreparedPipelineStep struct {
|
||||
ID string
|
||||
ArtifactLanes []PreparedArtifactLane
|
||||
lanes []preparedLaneExecutor
|
||||
}
|
||||
|
||||
type PreparedArtifactLane struct {
|
||||
Resolved ResolvedArtifactLane
|
||||
}
|
||||
@@ -73,11 +79,12 @@ func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDepend
|
||||
}
|
||||
stable := cloneResolvedPipeline(resolved)
|
||||
prepared := &PreparedPipeline{
|
||||
Input: cloneModuleBinding(stable.Input),
|
||||
Chunk: cloneModuleBinding(stable.Chunk),
|
||||
Output: cloneModuleBinding(stable.Output),
|
||||
resolved: stable,
|
||||
dependencies: deps,
|
||||
Input: cloneModuleBinding(stable.Input),
|
||||
Chunk: cloneModuleBinding(stable.Chunk),
|
||||
Output: cloneModuleBinding(stable.Output),
|
||||
resolved: stable,
|
||||
dependencies: deps,
|
||||
artifactCodecs: registries.ArtifactCodecs,
|
||||
}
|
||||
request := func(binding ModuleBinding, references contracts.ReferenceSet) BuildRequest {
|
||||
return BuildRequest{Dependencies: deps, Options: cloneOptions(binding.Options), References: references}
|
||||
@@ -99,15 +106,22 @@ func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDepend
|
||||
return nil, err
|
||||
}
|
||||
|
||||
prepared.ArtifactLanes = make([]PreparedArtifactLane, 0, len(stable.ArtifactLanes))
|
||||
prepared.lanes = make([]preparedLaneExecutor, 0, len(stable.ArtifactLanes))
|
||||
for _, lane := range stable.ArtifactLanes {
|
||||
executor, err := prepareLane(stable, lane, registries, deps)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
prepared.Steps = make([]PreparedPipelineStep, len(stable.Steps))
|
||||
for stepIndex, step := range stable.Steps {
|
||||
preparedStep := PreparedPipelineStep{
|
||||
ID: step.ID,
|
||||
ArtifactLanes: make([]PreparedArtifactLane, 0, len(step.ArtifactLanes)),
|
||||
lanes: make([]preparedLaneExecutor, 0, len(step.ArtifactLanes)),
|
||||
}
|
||||
prepared.ArtifactLanes = append(prepared.ArtifactLanes, PreparedArtifactLane{Resolved: cloneResolvedArtifactLane(lane)})
|
||||
prepared.lanes = append(prepared.lanes, executor)
|
||||
for _, lane := range step.ArtifactLanes {
|
||||
executor, err := prepareLane(stable, lane, registries, deps)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
preparedStep.ArtifactLanes = append(preparedStep.ArtifactLanes, PreparedArtifactLane{Resolved: cloneResolvedArtifactLane(lane)})
|
||||
preparedStep.lanes = append(preparedStep.lanes, executor)
|
||||
}
|
||||
prepared.Steps[stepIndex] = preparedStep
|
||||
}
|
||||
|
||||
output, err := registries.Outputs.BuildWithRequest(stable.Output.Module, request(stable.Output, contracts.ReferenceSet{}))
|
||||
@@ -302,6 +316,9 @@ func validateRegistrySet(resolved ResolvedPipeline, registries Registries) error
|
||||
if registries.Chunkers == nil {
|
||||
return fmt.Errorf("chunker registry must not be nil")
|
||||
}
|
||||
if registries.ArtifactCodecs == nil {
|
||||
return fmt.Errorf("artifact codec registry must not be nil")
|
||||
}
|
||||
if registries.Extractors == nil {
|
||||
return fmt.Errorf("extractor registry must not be nil")
|
||||
}
|
||||
@@ -331,10 +348,16 @@ func cloneResolvedPipeline(in ResolvedPipeline) ResolvedPipeline {
|
||||
out.Output = cloneModuleBinding(in.Output)
|
||||
out.ChunkReferences = CloneReferenceTarget(in.ChunkReferences)
|
||||
out.ValidatorChains = cloneResolvedValidatorChains(in.ValidatorChains)
|
||||
if len(in.ArtifactLanes) > 0 {
|
||||
out.ArtifactLanes = make([]ResolvedArtifactLane, len(in.ArtifactLanes))
|
||||
for i, lane := range in.ArtifactLanes {
|
||||
out.ArtifactLanes[i] = cloneResolvedArtifactLane(lane)
|
||||
if len(in.Steps) > 0 {
|
||||
out.Steps = make([]ResolvedPipelineStep, len(in.Steps))
|
||||
for i, step := range in.Steps {
|
||||
out.Steps[i] = ResolvedPipelineStep{ID: step.ID}
|
||||
if len(step.ArtifactLanes) > 0 {
|
||||
out.Steps[i].ArtifactLanes = make([]ResolvedArtifactLane, len(step.ArtifactLanes))
|
||||
for j, lane := range step.ArtifactLanes {
|
||||
out.Steps[i].ArtifactLanes[j] = cloneResolvedArtifactLane(lane)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
|
||||
@@ -26,16 +26,18 @@ func collectPreparedCheckpointFingerprints(prepared *PreparedPipeline) ([]Checkp
|
||||
{scope: "chunk:" + prepared.resolved.Chunk.Module, module: prepared.chunker},
|
||||
}
|
||||
components = appendValidatorFingerprintComponents(components, "chunk:"+prepared.resolved.Chunk.Module, prepared.chunkValidators)
|
||||
for _, lane := range prepared.lanes {
|
||||
laneScope := func(stage ModuleStage, moduleKey string) string {
|
||||
return string(stage) + ":" + lane.resolved.ID + ":" + moduleKey
|
||||
for _, step := range prepared.Steps {
|
||||
for _, lane := range step.lanes {
|
||||
laneScope := func(stage ModuleStage, moduleKey string) string {
|
||||
return string(stage) + ":" + lane.resolved.ID + ":" + moduleKey
|
||||
}
|
||||
components = append(components, checkpointFingerprintComponent{scope: laneScope(StageExtract, lane.resolved.Extract.Module), module: lane.typed.extractor})
|
||||
components = appendValidatorFingerprintComponents(components, laneScope(StageExtract, lane.resolved.Extract.Module), lane.extractValidators)
|
||||
components = append(components, checkpointFingerprintComponent{scope: laneScope(StageMerge, lane.resolved.Merge.Module), module: lane.typed.merger})
|
||||
components = appendValidatorFingerprintComponents(components, laneScope(StageMerge, lane.resolved.Merge.Module), lane.mergeValidators)
|
||||
components = append(components, checkpointFingerprintComponent{scope: laneScope(StageNormalize, lane.resolved.Normalize.Module), module: lane.typed.normalizer})
|
||||
components = appendValidatorFingerprintComponents(components, laneScope(StageNormalize, lane.resolved.Normalize.Module), lane.normalizeValidators)
|
||||
}
|
||||
components = append(components, checkpointFingerprintComponent{scope: laneScope(StageExtract, lane.resolved.Extract.Module), module: lane.typed.extractor})
|
||||
components = appendValidatorFingerprintComponents(components, laneScope(StageExtract, lane.resolved.Extract.Module), lane.extractValidators)
|
||||
components = append(components, checkpointFingerprintComponent{scope: laneScope(StageMerge, lane.resolved.Merge.Module), module: lane.typed.merger})
|
||||
components = appendValidatorFingerprintComponents(components, laneScope(StageMerge, lane.resolved.Merge.Module), lane.mergeValidators)
|
||||
components = append(components, checkpointFingerprintComponent{scope: laneScope(StageNormalize, lane.resolved.Normalize.Module), module: lane.typed.normalizer})
|
||||
components = appendValidatorFingerprintComponents(components, laneScope(StageNormalize, lane.resolved.Normalize.Module), lane.normalizeValidators)
|
||||
}
|
||||
components = append(components, checkpointFingerprintComponent{scope: "output:" + prepared.resolved.Output.Module, module: prepared.output})
|
||||
|
||||
|
||||
@@ -21,12 +21,56 @@ const (
|
||||
)
|
||||
|
||||
type ModuleBinding struct {
|
||||
Module string `json:"module"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Retries int `json:"retries,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
References map[string]string `json:"references,omitempty"`
|
||||
Validators ValidatorOverride `json:"validators,omitempty"`
|
||||
Module string `json:"module"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Retries int `json:"retries,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
References map[string]ReferenceSource `json:"references,omitempty"`
|
||||
Validators ValidatorOverride `json:"validators,omitempty"`
|
||||
}
|
||||
|
||||
// ArtifactReference identifies a normalized artifact produced by an earlier
|
||||
// ordered step. It is a selector only; it does not contain artifact bytes.
|
||||
type ArtifactReference struct {
|
||||
Step string `json:"step"`
|
||||
Lane string `json:"lane"`
|
||||
}
|
||||
|
||||
// ReferenceSource is the discriminated source form used by configured
|
||||
// reference maps. Exactly one of Path and Artifact is set after resolution.
|
||||
type ReferenceSource struct {
|
||||
Path string `json:"path,omitempty"`
|
||||
Artifact *ArtifactReference `json:"artifact,omitempty"`
|
||||
}
|
||||
|
||||
func ExternalReference(path string) ReferenceSource {
|
||||
return ReferenceSource{Path: strings.TrimSpace(path)}
|
||||
}
|
||||
|
||||
func ExternalReferenceMap(values map[string]string) map[string]ReferenceSource {
|
||||
out := make(map[string]ReferenceSource, len(values))
|
||||
for key, value := range values {
|
||||
out[key] = ExternalReference(value)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func GeneratedReference(step, lane string) ReferenceSource {
|
||||
return ReferenceSource{Artifact: &ArtifactReference{Step: strings.TrimSpace(step), Lane: strings.TrimSpace(lane)}}
|
||||
}
|
||||
|
||||
func (source ReferenceSource) IsGenerated() bool { return source.Artifact != nil }
|
||||
|
||||
func (source ReferenceSource) MarshalJSON() ([]byte, error) {
|
||||
if source.Path != "" && source.Artifact != nil {
|
||||
return nil, fmt.Errorf("reference source must not contain both an external path and an artifact selector")
|
||||
}
|
||||
if source.Artifact != nil {
|
||||
return json.Marshal(struct {
|
||||
Artifact *ArtifactReference `json:"artifact"`
|
||||
}{Artifact: source.Artifact})
|
||||
}
|
||||
return json.Marshal(source.Path)
|
||||
}
|
||||
|
||||
type ValidatorOverride struct {
|
||||
@@ -36,12 +80,12 @@ type ValidatorOverride struct {
|
||||
|
||||
func (binding ModuleBinding) MarshalJSON() ([]byte, error) {
|
||||
type moduleBindingJSON struct {
|
||||
Module string `json:"module"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Retries int `json:"retries,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
References map[string]string `json:"references,omitempty"`
|
||||
Validators *[]ModuleBinding `json:"validators,omitempty"`
|
||||
Module string `json:"module"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Retries int `json:"retries,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
References map[string]ReferenceSource `json:"references,omitempty"`
|
||||
Validators *[]ModuleBinding `json:"validators,omitempty"`
|
||||
}
|
||||
out := moduleBindingJSON{
|
||||
Module: binding.Module,
|
||||
@@ -58,11 +102,17 @@ func (binding ModuleBinding) MarshalJSON() ([]byte, error) {
|
||||
}
|
||||
|
||||
type ArtifactLaneProfile struct {
|
||||
Extract ModuleBinding `json:"extract"`
|
||||
Merge ModuleBinding `json:"merge,omitempty"`
|
||||
Normalize ModuleBinding `json:"normalize,omitempty"`
|
||||
Validators []ModuleBinding `json:"validators,omitempty"`
|
||||
References map[string]string `json:"references,omitempty"`
|
||||
Extract ModuleBinding `json:"extract"`
|
||||
Merge ModuleBinding `json:"merge,omitempty"`
|
||||
Normalize ModuleBinding `json:"normalize,omitempty"`
|
||||
Validators []ModuleBinding `json:"validators,omitempty"`
|
||||
References map[string]ReferenceSource `json:"references,omitempty"`
|
||||
}
|
||||
|
||||
type PipelineStepProfile struct {
|
||||
ID string `json:"id"`
|
||||
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
|
||||
References map[string]ReferenceSource `json:"references,omitempty"`
|
||||
}
|
||||
|
||||
type PipelineProfile struct {
|
||||
@@ -70,8 +120,9 @@ type PipelineProfile struct {
|
||||
Input ModuleBinding `json:"input"`
|
||||
Chunk ModuleBinding `json:"chunk,omitempty"`
|
||||
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
|
||||
Steps []PipelineStepProfile `json:"steps,omitempty"`
|
||||
Output ModuleBinding `json:"output,omitempty"`
|
||||
References map[string]string `json:"references,omitempty"`
|
||||
References map[string]ReferenceSource `json:"references,omitempty"`
|
||||
}
|
||||
|
||||
type ResolveOptions struct {
|
||||
@@ -81,11 +132,12 @@ type ResolveOptions struct {
|
||||
}
|
||||
|
||||
type ReferenceBinding struct {
|
||||
Stage ModuleStage `json:"stage,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
SlotName string `json:"slot_name"`
|
||||
Source string `json:"source"`
|
||||
BindingSource string `json:"binding_source,omitempty"`
|
||||
Stage ModuleStage `json:"stage,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
SlotName string `json:"slot_name"`
|
||||
Source string `json:"source,omitempty"`
|
||||
Artifact *ArtifactReference `json:"artifact,omitempty"`
|
||||
BindingSource string `json:"binding_source,omitempty"`
|
||||
}
|
||||
|
||||
type ReferenceUnbind struct {
|
||||
@@ -96,6 +148,7 @@ type ReferenceUnbind struct {
|
||||
|
||||
type ResolvedReferenceTarget struct {
|
||||
Stage ModuleStage `json:"stage"`
|
||||
StepID string `json:"step_id,omitempty"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
Module string `json:"module"`
|
||||
Bindings []ReferenceBinding `json:"bindings,omitempty"`
|
||||
@@ -103,6 +156,7 @@ type ResolvedReferenceTarget struct {
|
||||
}
|
||||
|
||||
type ResolvedArtifactLane struct {
|
||||
StepID string
|
||||
ID string
|
||||
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
|
||||
@@ -118,6 +172,11 @@ type ResolvedArtifactLane struct {
|
||||
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
||||
}
|
||||
|
||||
type ResolvedPipelineStep struct {
|
||||
ID string `json:"id"`
|
||||
ArtifactLanes []ResolvedArtifactLane `json:"artifact_lanes"`
|
||||
}
|
||||
|
||||
type ResolvedValidatorChain struct {
|
||||
Stage ModuleStage `json:"stage"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
@@ -138,11 +197,21 @@ type ResolvedPipeline struct {
|
||||
Input ModuleBinding
|
||||
Chunk ModuleBinding
|
||||
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
|
||||
ArtifactLanes []ResolvedArtifactLane
|
||||
Steps []ResolvedPipelineStep
|
||||
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
|
||||
Output ModuleBinding
|
||||
}
|
||||
|
||||
// AllArtifactLanes returns lanes in deterministic step order for read-only
|
||||
// discovery and identity construction.
|
||||
func (resolved ResolvedPipeline) AllArtifactLanes() []ResolvedArtifactLane {
|
||||
var lanes []ResolvedArtifactLane
|
||||
for _, step := range resolved.Steps {
|
||||
lanes = append(lanes, step.ArtifactLanes...)
|
||||
}
|
||||
return lanes
|
||||
}
|
||||
|
||||
type ModuleCatalog struct {
|
||||
Inputs *InputAdapterRegistry
|
||||
Chunkers *ChunkerRegistry
|
||||
@@ -165,7 +234,21 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
||||
return ResolvedPipeline{}, fmt.Errorf("pipeline id must not be empty")
|
||||
}
|
||||
|
||||
if len(profile.Artifacts) == 0 {
|
||||
explicitSteps := profile.Steps != nil
|
||||
if len(profile.Artifacts) > 0 && explicitSteps {
|
||||
return ResolvedPipeline{}, fmt.Errorf("pipeline %q must not declare both artifacts and steps", pipelineID)
|
||||
}
|
||||
if explicitSteps && len(options.Only) > 0 {
|
||||
return ResolvedPipeline{}, fmt.Errorf("pipeline %q --only is not supported for explicit ordered steps", pipelineID)
|
||||
}
|
||||
steps := profile.Steps
|
||||
if !explicitSteps {
|
||||
steps = []PipelineStepProfile{{ID: "default", Artifacts: profile.Artifacts}}
|
||||
}
|
||||
if explicitSteps && len(steps) == 0 {
|
||||
return ResolvedPipeline{}, fmt.Errorf("pipeline %q must declare at least one ordered step", pipelineID)
|
||||
}
|
||||
if len(steps) == 0 {
|
||||
return ResolvedPipeline{}, fmt.Errorf("pipeline %q must declare at least one artifact lane", pipelineID)
|
||||
}
|
||||
|
||||
@@ -194,15 +277,25 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
||||
}
|
||||
capabilities.add(chunkSpec.Provides...)
|
||||
|
||||
lanesByID, selectedLaneIDs, err := selectedArtifactLanes(pipelineID, profile.Artifacts, options)
|
||||
if err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
if !explicitSteps {
|
||||
if err := validatePipelineReferenceDefaults(pipelineID, profile.References, chunkSpec, profile.Artifacts, catalog); err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
} else {
|
||||
allLanes := make(map[string]ArtifactLaneProfile)
|
||||
for _, step := range profile.Steps {
|
||||
for laneID, lane := range step.Artifacts {
|
||||
allLanes[strings.TrimSpace(laneID)] = lane
|
||||
}
|
||||
}
|
||||
if err := validatePipelineReferenceDefaults(pipelineID, profile.References, chunkSpec, allLanes, catalog); err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
}
|
||||
if len(selectedLaneIDs) == 0 {
|
||||
return ResolvedPipeline{}, fmt.Errorf("pipeline %q must select at least one artifact lane", pipelineID)
|
||||
}
|
||||
if err := validatePipelineReferenceDefaults(pipelineID, profile.References, chunkSpec, lanesByID, catalog); err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
for _, source := range profile.References {
|
||||
if source.Artifact != nil {
|
||||
return ResolvedPipeline{}, fmt.Errorf("pipeline %q pipeline-level references may not use generated artifact selectors", pipelineID)
|
||||
}
|
||||
}
|
||||
|
||||
chunkReferences, err := resolveReferenceTargetBindings(referenceResolutionTarget{
|
||||
@@ -230,16 +323,55 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
||||
}
|
||||
resolved.ValidatorChains = append(resolved.ValidatorChains, chunkValidatorChain)
|
||||
outputCapabilities := capabilities.clone()
|
||||
seenResolvedLanes := make(map[string]string)
|
||||
seenResolvedSteps := make(map[string]struct{}, len(steps))
|
||||
|
||||
for _, laneID := range selectedLaneIDs {
|
||||
laneProfile := lanesByID[laneID]
|
||||
lane, validatorChains, laneCapabilities, err := resolveArtifactLane(pipelineID, laneID, laneProfile, profile.References, options, capabilities, catalog)
|
||||
for _, step := range steps {
|
||||
stepID := strings.TrimSpace(step.ID)
|
||||
if stepID == "" {
|
||||
return ResolvedPipeline{}, fmt.Errorf("pipeline %q step id must not be empty", pipelineID)
|
||||
}
|
||||
if _, exists := seenResolvedSteps[stepID]; exists {
|
||||
return ResolvedPipeline{}, fmt.Errorf("pipeline %q step id %q is duplicated after trimming", pipelineID, stepID)
|
||||
}
|
||||
seenResolvedSteps[stepID] = struct{}{}
|
||||
laneOptions := ResolveOptions{}
|
||||
if !explicitSteps {
|
||||
laneOptions = options
|
||||
}
|
||||
lanesByID, selectedLaneIDs, err := selectedArtifactLanes(pipelineID, step.Artifacts, laneOptions)
|
||||
if err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
resolved.ArtifactLanes = append(resolved.ArtifactLanes, lane)
|
||||
resolved.ValidatorChains = append(resolved.ValidatorChains, validatorChains...)
|
||||
outputCapabilities.addSet(laneCapabilities)
|
||||
if len(selectedLaneIDs) == 0 {
|
||||
return ResolvedPipeline{}, fmt.Errorf("pipeline %q step %q must declare at least one artifact lane", pipelineID, stepID)
|
||||
}
|
||||
if referenceSourceMapsConflict(profile.References, step.References) {
|
||||
return ResolvedPipeline{}, fmt.Errorf("pipeline %q step %q has a generated and external reference collision", pipelineID, stepID)
|
||||
}
|
||||
if err := validatePipelineReferenceDefaults(pipelineID, step.References, chunkSpec, lanesByID, catalog); err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
stepReferences := mergeReferenceMaps(profile.References, step.References)
|
||||
resolvedStep := ResolvedPipelineStep{ID: stepID}
|
||||
for _, laneID := range selectedLaneIDs {
|
||||
if previousStep, exists := seenResolvedLanes[laneID]; exists {
|
||||
return ResolvedPipeline{}, fmt.Errorf("pipeline %q artifact lane id %q is duplicated across steps %q and %q", pipelineID, laneID, previousStep, stepID)
|
||||
}
|
||||
seenResolvedLanes[laneID] = stepID
|
||||
laneProfile := lanesByID[laneID]
|
||||
lane, validatorChains, laneCapabilities, err := resolveArtifactLane(pipelineID, stepID, laneID, laneProfile, stepReferences, options, capabilities, catalog)
|
||||
if err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
resolvedStep.ArtifactLanes = append(resolvedStep.ArtifactLanes, lane)
|
||||
resolved.ValidatorChains = append(resolved.ValidatorChains, validatorChains...)
|
||||
outputCapabilities.addSet(laneCapabilities)
|
||||
}
|
||||
resolved.Steps = append(resolved.Steps, resolvedStep)
|
||||
}
|
||||
if err := validateGeneratedBindings(pipelineID, resolved, catalog); err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
|
||||
outputSpec, err := outputSpec(catalog, resolved.Output.Module)
|
||||
@@ -263,14 +395,16 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
||||
|
||||
func resolveArtifactLane(
|
||||
pipelineID string,
|
||||
stepID string,
|
||||
laneID string,
|
||||
profile ArtifactLaneProfile,
|
||||
pipelineReferences map[string]string,
|
||||
pipelineReferences map[string]ReferenceSource,
|
||||
options ResolveOptions,
|
||||
inherited capabilitySet,
|
||||
catalog ModuleCatalog,
|
||||
) (ResolvedArtifactLane, []ResolvedValidatorChain, capabilitySet, error) {
|
||||
lane := ResolvedArtifactLane{
|
||||
StepID: strings.TrimSpace(stepID),
|
||||
ID: laneID,
|
||||
Extract: resolveBinding(profile.Extract, ""),
|
||||
Merge: resolveBinding(profile.Merge, DefaultMergeModule),
|
||||
@@ -294,6 +428,9 @@ func resolveArtifactLane(
|
||||
if err != nil {
|
||||
return ResolvedArtifactLane{}, nil, nil, err
|
||||
}
|
||||
if referenceSourceMapsConflict(profile.References, lane.Extract.References) {
|
||||
return ResolvedArtifactLane{}, nil, nil, fmt.Errorf("pipeline %q step %q lane %q extract has a generated and external reference collision", pipelineID, stepID, laneID)
|
||||
}
|
||||
extractReferences := mergeReferenceMaps(profile.References, lane.Extract.References)
|
||||
references, err := resolveReferenceTargetBindings(referenceResolutionTarget{
|
||||
PipelineID: pipelineID,
|
||||
@@ -309,6 +446,7 @@ func resolveArtifactLane(
|
||||
return ResolvedArtifactLane{}, nil, nil, err
|
||||
}
|
||||
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
|
||||
lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
|
||||
capabilities.add(extractSpec.Provides...)
|
||||
|
||||
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
|
||||
@@ -332,6 +470,7 @@ func resolveArtifactLane(
|
||||
return ResolvedArtifactLane{}, nil, nil, err
|
||||
}
|
||||
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
|
||||
lane.MergeReferences.StepID = strings.TrimSpace(stepID)
|
||||
capabilities.add(mergeSpec.Provides...)
|
||||
|
||||
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
|
||||
@@ -355,6 +494,7 @@ func resolveArtifactLane(
|
||||
return ResolvedArtifactLane{}, nil, nil, err
|
||||
}
|
||||
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
|
||||
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
|
||||
capabilities.add(normalizeSpec.Provides...)
|
||||
|
||||
if len(lane.Validators) > 0 {
|
||||
@@ -382,6 +522,83 @@ func configuredValidatorsError(pipelineID string, laneID string) error {
|
||||
return fmt.Errorf("pipeline %q lane %q validators are not supported at artifact lane level; use extract.validators, merge.validators, or normalize.validators", pipelineID, laneID)
|
||||
}
|
||||
|
||||
func validateGeneratedBindings(pipelineID string, resolved ResolvedPipeline, catalog ModuleCatalog) error {
|
||||
stepIndex := make(map[string]int, len(resolved.Steps))
|
||||
laneIndex := make(map[string]ResolvedArtifactLane)
|
||||
for index, step := range resolved.Steps {
|
||||
if _, exists := stepIndex[step.ID]; exists {
|
||||
return fmt.Errorf("pipeline %q step id %q is ambiguous", pipelineID, step.ID)
|
||||
}
|
||||
stepIndex[step.ID] = index
|
||||
for _, lane := range step.ArtifactLanes {
|
||||
laneIndex[step.ID+"\x00"+lane.ID] = lane
|
||||
}
|
||||
}
|
||||
for consumerIndex, step := range resolved.Steps {
|
||||
for _, lane := range step.ArtifactLanes {
|
||||
for _, target := range []ResolvedReferenceTarget{lane.ExtractReferences, lane.MergeReferences, lane.NormalizeReferences} {
|
||||
slots, err := resolvedReferenceSlots(target, lane.ArtifactKind, catalog)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pipeline %q step %q lane %q: %w", pipelineID, step.ID, lane.ID, err)
|
||||
}
|
||||
slotByName := make(map[string]contracts.ReferenceSlot, len(slots))
|
||||
for _, slot := range slots {
|
||||
slotByName[slot.Name] = slot
|
||||
}
|
||||
for _, binding := range target.Bindings {
|
||||
if binding.Artifact == nil {
|
||||
continue
|
||||
}
|
||||
producerStep, ok := stepIndex[strings.TrimSpace(binding.Artifact.Step)]
|
||||
if !ok {
|
||||
return fmt.Errorf("pipeline %q step %q lane %q %s reference slot %q producer step %q is not declared", pipelineID, step.ID, lane.ID, target.Stage, binding.SlotName, binding.Artifact.Step)
|
||||
}
|
||||
if producerStep >= consumerIndex {
|
||||
return fmt.Errorf("pipeline %q step %q lane %q reference slot %q producer %q must be an earlier step", pipelineID, step.ID, lane.ID, binding.SlotName, binding.Artifact.Step)
|
||||
}
|
||||
producer, ok := laneIndex[strings.TrimSpace(binding.Artifact.Step)+"\x00"+strings.TrimSpace(binding.Artifact.Lane)]
|
||||
if !ok {
|
||||
return fmt.Errorf("pipeline %q step %q lane %q reference slot %q producer lane %q is not selected", pipelineID, step.ID, lane.ID, binding.SlotName, binding.Artifact.Lane)
|
||||
}
|
||||
slot, ok := slotByName[strings.TrimSpace(binding.SlotName)]
|
||||
if !ok {
|
||||
return fmt.Errorf("pipeline %q step %q lane %q %s reference slot %q is not declared", pipelineID, step.ID, lane.ID, target.Stage, binding.SlotName)
|
||||
}
|
||||
if len(slot.AcceptedArtifactKinds) == 0 {
|
||||
return fmt.Errorf("pipeline %q step %q lane %q reference slot %q does not accept generated artifacts", pipelineID, step.ID, lane.ID, binding.SlotName)
|
||||
}
|
||||
accepted := false
|
||||
for _, kind := range slot.AcceptedArtifactKinds {
|
||||
if kind == producer.ArtifactKind {
|
||||
accepted = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !accepted {
|
||||
return fmt.Errorf("pipeline %q step %q lane %q reference slot %q does not accept artifact kind %q", pipelineID, step.ID, lane.ID, binding.SlotName, producer.ArtifactKind)
|
||||
}
|
||||
codecSpec, ok := catalog.ArtifactCodecs.Spec(producer.ArtifactKind)
|
||||
if !ok {
|
||||
return fmt.Errorf("pipeline %q step %q lane %q producer %q artifact codec %q is not registered", pipelineID, step.ID, lane.ID, producer.ID, producer.ArtifactKind)
|
||||
}
|
||||
if !referenceMediaTypeAccepted(codecSpec.MediaType, slot.AcceptedMediaTypes) {
|
||||
return fmt.Errorf("pipeline %q step %q lane %q reference slot %q does not accept producer media type %q", pipelineID, step.ID, lane.ID, binding.SlotName, codecSpec.MediaType)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func resolvedReferenceSlots(target ResolvedReferenceTarget, artifactKind contracts.ArtifactKind, catalog ModuleCatalog) ([]contracts.ReferenceSlot, error) {
|
||||
spec, err := referenceTargetSpec(target, artifactKind, catalog)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return contracts.CloneReferenceSlots(spec.ReferenceSlots), nil
|
||||
}
|
||||
|
||||
func resolveArtifactIdentity(pipelineID, laneID string, lane *ResolvedArtifactLane, extractSpec ModuleSpec, catalog ModuleCatalog) (reflect.Type, error) {
|
||||
if extractSpec.ArtifactKind == "" {
|
||||
return nil, fmt.Errorf("pipeline %q lane %q extract module %q does not declare an artifact kind", pipelineID, laneID, lane.Extract.Module)
|
||||
@@ -593,23 +810,35 @@ func referenceTarget(stage ModuleStage, laneID string, module string, bindings [
|
||||
}
|
||||
}
|
||||
|
||||
func mergeReferenceMaps(base map[string]string, override map[string]string) map[string]string {
|
||||
func mergeReferenceMaps(base map[string]ReferenceSource, override map[string]ReferenceSource) map[string]ReferenceSource {
|
||||
if len(base) == 0 && len(override) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]string, len(base)+len(override))
|
||||
out := make(map[string]ReferenceSource, len(base)+len(override))
|
||||
for key, value := range base {
|
||||
out[key] = value
|
||||
out[key] = cloneReferenceSource(value)
|
||||
}
|
||||
for key, value := range override {
|
||||
out[key] = value
|
||||
out[key] = cloneReferenceSource(value)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func referenceSourceMapsConflict(base, override map[string]ReferenceSource) bool {
|
||||
for rawKey, left := range base {
|
||||
key := strings.TrimSpace(rawKey)
|
||||
for rawOverrideKey, right := range override {
|
||||
if key == strings.TrimSpace(rawOverrideKey) && left.IsGenerated() != right.IsGenerated() {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func validatePipelineReferenceDefaults(
|
||||
pipelineID string,
|
||||
pipelineReferences map[string]string,
|
||||
pipelineReferences map[string]ReferenceSource,
|
||||
chunkSpec ModuleSpec,
|
||||
lanesByID map[string]ArtifactLaneProfile,
|
||||
catalog ModuleCatalog,
|
||||
@@ -680,8 +909,8 @@ type referenceResolutionTarget struct {
|
||||
Stage ModuleStage
|
||||
Module string
|
||||
Slots []contracts.ReferenceSlot
|
||||
PipelineReferences map[string]string
|
||||
LocalReferences map[string]string
|
||||
PipelineReferences map[string]ReferenceSource
|
||||
LocalReferences map[string]ReferenceSource
|
||||
Options ResolveOptions
|
||||
}
|
||||
|
||||
@@ -692,24 +921,46 @@ func resolveReferenceTargetBindings(target referenceResolutionTarget) ([]Referen
|
||||
}
|
||||
|
||||
bindings := make(map[string]ReferenceBinding)
|
||||
addBinding := func(slotName, source, bindingSource string) error {
|
||||
addBinding := func(slotName string, source ReferenceSource, bindingSource string) error {
|
||||
slotName = strings.TrimSpace(slotName)
|
||||
source = strings.TrimSpace(source)
|
||||
if slotName == "" {
|
||||
return fmt.Errorf("%s reference slot name must not be empty", referenceTargetErrorContext(target))
|
||||
}
|
||||
if source == "" {
|
||||
if source.Artifact == nil {
|
||||
source.Path = strings.TrimSpace(source.Path)
|
||||
}
|
||||
if source.Artifact == nil && source.Path == "" {
|
||||
return fmt.Errorf("%s reference slot %q source must not be empty", referenceTargetErrorContext(target), slotName)
|
||||
}
|
||||
if source.Artifact != nil {
|
||||
if strings.TrimSpace(source.Path) != "" {
|
||||
return fmt.Errorf("%s reference slot %q must contain exactly one source form", referenceTargetErrorContext(target), slotName)
|
||||
}
|
||||
artifact := *source.Artifact
|
||||
artifact.Step = strings.TrimSpace(artifact.Step)
|
||||
artifact.Lane = strings.TrimSpace(artifact.Lane)
|
||||
if artifact.Step == "" || artifact.Lane == "" {
|
||||
return fmt.Errorf("%s reference slot %q artifact selector step and lane must not be empty", referenceTargetErrorContext(target), slotName)
|
||||
}
|
||||
source.Artifact = &artifact
|
||||
}
|
||||
if _, ok := slotByName[slotName]; !ok {
|
||||
return fmt.Errorf("%s reference slot %q is not declared by %s module %q", referenceTargetErrorContext(target), slotName, target.Stage, target.Module)
|
||||
}
|
||||
bindings[slotName] = ReferenceBinding{
|
||||
if previous, exists := bindings[slotName]; exists && (previous.Artifact != nil || source.Artifact != nil) {
|
||||
return fmt.Errorf("%s reference slot %q has conflicting generated and external bindings", referenceTargetErrorContext(target), slotName)
|
||||
}
|
||||
binding := ReferenceBinding{
|
||||
LaneID: target.LaneID,
|
||||
SlotName: slotName,
|
||||
Source: source,
|
||||
BindingSource: bindingSource,
|
||||
}
|
||||
if source.Artifact != nil {
|
||||
binding.Artifact = source.Artifact
|
||||
} else {
|
||||
binding.Source = source.Path
|
||||
}
|
||||
bindings[slotName] = binding
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -748,7 +999,7 @@ func resolveReferenceTargetBindings(target referenceResolutionTarget) ([]Referen
|
||||
if strings.TrimSpace(source) == "" {
|
||||
source = contracts.ReferenceBindingSourceCLI
|
||||
}
|
||||
if err := addBinding(override.SlotName, override.Source, strings.TrimSpace(source)); err != nil {
|
||||
if err := addBinding(override.SlotName, ExternalReference(override.Source), strings.TrimSpace(source)); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
@@ -768,7 +1019,9 @@ func resolveReferenceTargetBindings(target referenceResolutionTarget) ([]Referen
|
||||
if _, ok := slotByName[slotName]; !ok {
|
||||
return nil, fmt.Errorf("%s reference slot %q is not declared", referenceTargetErrorContext(target), slotName)
|
||||
}
|
||||
delete(bindings, slotName)
|
||||
if binding, ok := bindings[slotName]; ok && binding.Artifact == nil {
|
||||
delete(bindings, slotName)
|
||||
}
|
||||
}
|
||||
|
||||
for _, slot := range target.Slots {
|
||||
@@ -843,11 +1096,11 @@ func referenceTargetSlotLabel(target referenceResolutionTarget) string {
|
||||
return fmt.Sprintf("pipeline %q %s reference slot", target.PipelineID, target.Stage)
|
||||
}
|
||||
|
||||
func normalizedReferenceMap(values map[string]string, keyName string) (map[string]string, error) {
|
||||
func normalizedReferenceMap(values map[string]ReferenceSource, keyName string) (map[string]ReferenceSource, error) {
|
||||
if len(values) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
out := make(map[string]string, len(values))
|
||||
out := make(map[string]ReferenceSource, len(values))
|
||||
for rawSlotName, rawSource := range values {
|
||||
slotName := strings.TrimSpace(rawSlotName)
|
||||
if slotName == "" {
|
||||
@@ -856,8 +1109,8 @@ func normalizedReferenceMap(values map[string]string, keyName string) (map[strin
|
||||
if _, ok := out[slotName]; ok {
|
||||
return nil, fmt.Errorf("%s %q is duplicated after trimming", keyName, slotName)
|
||||
}
|
||||
source := strings.TrimSpace(rawSource)
|
||||
if source == "" {
|
||||
source := cloneReferenceSource(rawSource)
|
||||
if source.Artifact == nil && strings.TrimSpace(source.Path) == "" {
|
||||
return nil, fmt.Errorf("%s %q source must not be empty", keyName, slotName)
|
||||
}
|
||||
out[slotName] = source
|
||||
@@ -877,7 +1130,7 @@ func sortedArtifactLaneProfileKeys(values map[string]ArtifactLaneProfile) []stri
|
||||
return keys
|
||||
}
|
||||
|
||||
func sortedStringMapKeys(values map[string]string) []string {
|
||||
func sortedStringMapKeys(values map[string]ReferenceSource) []string {
|
||||
if len(values) == 0 {
|
||||
return nil
|
||||
}
|
||||
@@ -942,11 +1195,11 @@ func cloneOptions(options map[string]any) map[string]any {
|
||||
return copied
|
||||
}
|
||||
|
||||
func normalizeReferenceMap(values map[string]string) map[string]string {
|
||||
func normalizeReferenceMap(values map[string]ReferenceSource) map[string]ReferenceSource {
|
||||
if len(values) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]string, len(values))
|
||||
out := make(map[string]ReferenceSource, len(values))
|
||||
keys := make([]string, 0, len(values))
|
||||
rawByNormalized := make(map[string]string, len(values))
|
||||
for rawKey := range values {
|
||||
@@ -956,7 +1209,7 @@ func normalizeReferenceMap(values map[string]string) map[string]string {
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, key := range keys {
|
||||
out[key] = strings.TrimSpace(values[rawByNormalized[key]])
|
||||
out[key] = cloneReferenceSource(values[rawByNormalized[key]])
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -1009,7 +1262,7 @@ func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
|
||||
Input ModuleBinding
|
||||
Chunk ModuleBinding
|
||||
ChunkReferences ResolvedReferenceTarget
|
||||
ArtifactLanes []ResolvedArtifactLane
|
||||
Steps []ResolvedPipelineStep
|
||||
ValidatorChains []ResolvedValidatorChain
|
||||
Output ModuleBinding
|
||||
}{
|
||||
@@ -1017,7 +1270,7 @@ func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
|
||||
Input: resolved.Input,
|
||||
Chunk: resolved.Chunk,
|
||||
ChunkReferences: resolved.ChunkReferences,
|
||||
ArtifactLanes: resolved.ArtifactLanes,
|
||||
Steps: resolved.Steps,
|
||||
ValidatorChains: resolved.ValidatorChains,
|
||||
Output: resolved.Output,
|
||||
}
|
||||
|
||||
@@ -53,10 +53,10 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
|
||||
if resolved.Chunk.Options["size"] != 10 {
|
||||
t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options)
|
||||
}
|
||||
if len(resolved.ArtifactLanes) != 1 {
|
||||
t.Fatalf("len(ArtifactLanes) = %d, want 1", len(resolved.ArtifactLanes))
|
||||
if len(resolved.Steps) != 1 || resolved.Steps[0].ID != "default" || len(resolved.Steps[0].ArtifactLanes) != 1 {
|
||||
t.Fatalf("resolved steps = %#v, want one default step with one lane", resolved.Steps)
|
||||
}
|
||||
lane := resolved.ArtifactLanes[0]
|
||||
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||
if lane.ID != "records" {
|
||||
t.Fatalf("lane.ID = %q, want records", lane.ID)
|
||||
}
|
||||
@@ -98,7 +98,7 @@ func TestResolvePipelineAppliesDefaults(t *testing.T) {
|
||||
if !reflect.DeepEqual(resolved.Output, ModuleBinding{Module: DefaultOutputModule}) {
|
||||
t.Fatalf("Output = %#v, want default output binding", resolved.Output)
|
||||
}
|
||||
lane := resolved.ArtifactLanes[0]
|
||||
lane := resolved.Steps[0].ArtifactLanes[0]
|
||||
if !reflect.DeepEqual(lane.Merge, ModuleBinding{Module: DefaultMergeModule}) {
|
||||
t.Fatalf("Merge = %#v, want default merge binding", lane.Merge)
|
||||
}
|
||||
@@ -326,23 +326,77 @@ func TestResolvePipelineSelectsOnlyRequestedLanes(t *testing.T) {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
got := laneIDs(resolved.ArtifactLanes)
|
||||
got := laneIDs(resolved.Steps[0].ArtifactLanes)
|
||||
want := []string{"events", "summaries"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("lane IDs = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *testing.T) {
|
||||
catalog := newProfileCatalogWithOverrides(t,
|
||||
ModuleSpec{
|
||||
Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes",
|
||||
Requires: []string{"chunk"}, Provides: []string{"candidate"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
|
||||
})
|
||||
resolved, err := ResolvePipeline(PipelineProfile{
|
||||
ID: "ordered",
|
||||
Input: Binding("text"),
|
||||
Steps: []PipelineStepProfile{
|
||||
{ID: "produce", Artifacts: map[string]ArtifactLaneProfile{"npcs": {Extract: Binding("event-extractor")}}},
|
||||
{ID: "consume", References: map[string]ReferenceSource{"npcs": GeneratedReference("produce", "npcs")}, Artifacts: map[string]ArtifactLaneProfile{"events": {Extract: Binding("note-extractor")}}},
|
||||
},
|
||||
}, ResolveOptions{}, catalog)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
if got := []string{resolved.Steps[0].ID, resolved.Steps[1].ID}; !reflect.DeepEqual(got, []string{"produce", "consume"}) {
|
||||
t.Fatalf("step order = %#v", got)
|
||||
}
|
||||
binding := resolved.Steps[1].ArtifactLanes[0].ExtractReferences.Bindings[0]
|
||||
if binding.Artifact == nil || binding.Artifact.Step != "produce" || binding.Artifact.Lane != "npcs" {
|
||||
t.Fatalf("generated binding = %#v", binding)
|
||||
}
|
||||
if len(resolved.AllArtifactLanes()) != 2 || resolved.Steps[1].ArtifactLanes[0].StepID != "consume" {
|
||||
t.Fatalf("resolved lane topology = %#v", resolved.Steps)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineRejectsGeneratedBindingOrderingAndKind(t *testing.T) {
|
||||
catalog := newProfileCatalogWithOverrides(t,
|
||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
|
||||
)
|
||||
_, err := ResolvePipeline(PipelineProfile{
|
||||
ID: "invalid-order", Input: Binding("text"), Steps: []PipelineStepProfile{
|
||||
{ID: "first", References: map[string]ReferenceSource{"npcs": GeneratedReference("later", "npcs")}, Artifacts: map[string]ArtifactLaneProfile{"events": {Extract: Binding("note-extractor")}}},
|
||||
{ID: "later", Artifacts: map[string]ArtifactLaneProfile{"npcs": {Extract: Binding("event-extractor")}}},
|
||||
},
|
||||
}, ResolveOptions{}, catalog)
|
||||
if err == nil || !strings.Contains(err.Error(), "earlier step") {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want ordering failure", err)
|
||||
}
|
||||
_, err = ResolvePipeline(PipelineProfile{
|
||||
ID: "invalid-kind", Input: Binding("text"), Steps: []PipelineStepProfile{
|
||||
{ID: "produce", Artifacts: map[string]ArtifactLaneProfile{"npcs": {Extract: Binding("event-extractor")}}},
|
||||
{ID: "consume", References: map[string]ReferenceSource{"npcs": GeneratedReference("produce", "npcs")}, Artifacts: map[string]ArtifactLaneProfile{"events": {Extract: Binding("note-extractor")}}},
|
||||
},
|
||||
}, ResolveOptions{}, catalog)
|
||||
if err == nil || !strings.Contains(err.Error(), "does not accept artifact kind") {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want generated kind failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
|
||||
profile := multiLaneProfile()
|
||||
profile.References = map[string]string{
|
||||
profile.References = ExternalReferenceMap(map[string]string{
|
||||
" roster ": " ./shared-roster.yml ",
|
||||
}
|
||||
})
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.References = map[string]string{
|
||||
lane.References = ExternalReferenceMap(map[string]string{
|
||||
"roster": "./lane-roster.yml",
|
||||
" lore ": " ./lore.md ",
|
||||
}
|
||||
})
|
||||
profile.Artifacts["events"] = lane
|
||||
|
||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||
@@ -360,7 +414,7 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
events := resolvedLane(t, resolved.ArtifactLanes, "events")
|
||||
events := resolvedLane(t, resolved.Steps[0].ArtifactLanes, "events")
|
||||
if events.ExtractReferences.Stage != StageExtract || events.ExtractReferences.LaneID != "events" || events.ExtractReferences.Module != "event-extractor" {
|
||||
t.Fatalf("extract reference target = %#v, want event extractor target", events.ExtractReferences)
|
||||
}
|
||||
@@ -380,7 +434,7 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
|
||||
if !reflect.DeepEqual(events.ExtractReferences.Bindings, want) {
|
||||
t.Fatalf("events references = %#v, want %#v", events.ExtractReferences.Bindings, want)
|
||||
}
|
||||
summaries := resolvedLane(t, resolved.ArtifactLanes, "summaries")
|
||||
summaries := resolvedLane(t, resolved.Steps[0].ArtifactLanes, "summaries")
|
||||
if len(summaries.ExtractReferences.Bindings) != 0 {
|
||||
t.Fatalf("summaries references = %#v, want none", summaries.ExtractReferences.Bindings)
|
||||
}
|
||||
@@ -388,7 +442,7 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
|
||||
|
||||
func TestResolvePipelineAppliesPipelineReferenceDefaultToChunkTarget(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
profile.References = map[string]string{"scene_guide": "./scenes.md"}
|
||||
profile.References = ExternalReferenceMap(map[string]string{"scene_guide": "./scenes.md"})
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "generic",
|
||||
Stage: StageChunk,
|
||||
@@ -406,17 +460,17 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToChunkTarget(t *testing.
|
||||
if !reflect.DeepEqual(resolved.ChunkReferences.Bindings, want) {
|
||||
t.Fatalf("chunk references = %#v, want %#v", resolved.ChunkReferences.Bindings, want)
|
||||
}
|
||||
if refs := resolved.ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
|
||||
if refs := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
|
||||
t.Fatalf("extract references = %#v, want none", refs)
|
||||
}
|
||||
if refs := resolved.ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
|
||||
if refs := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
|
||||
t.Fatalf("normalize references = %#v, want none", refs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineAppliesPipelineReferenceDefaultToExtractorTarget(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
profile.References = map[string]string{"roster": "./roster.yml"}
|
||||
profile.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
@@ -431,20 +485,20 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToExtractorTarget(t *test
|
||||
}
|
||||
|
||||
want := []ReferenceBinding{{LaneID: "events", SlotName: "roster", Source: "./roster.yml", BindingSource: contracts.ReferenceBindingSourceConfig}}
|
||||
if !reflect.DeepEqual(resolved.ArtifactLanes[0].ExtractReferences.Bindings, want) {
|
||||
t.Fatalf("extract references = %#v, want %#v", resolved.ArtifactLanes[0].ExtractReferences.Bindings, want)
|
||||
if !reflect.DeepEqual(resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings, want) {
|
||||
t.Fatalf("extract references = %#v, want %#v", resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings, want)
|
||||
}
|
||||
if refs := resolved.ChunkReferences.Bindings; len(refs) != 0 {
|
||||
t.Fatalf("chunk references = %#v, want none", refs)
|
||||
}
|
||||
if refs := resolved.ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
|
||||
if refs := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
|
||||
t.Fatalf("normalize references = %#v, want none", refs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineAppliesPipelineReferenceDefaultToNormalizerTarget(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
profile.References = map[string]string{"normalization_notes": "./normalize.md"}
|
||||
profile.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./normalize.md"})
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "noop",
|
||||
Stage: StageNormalize,
|
||||
@@ -459,20 +513,20 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToNormalizerTarget(t *tes
|
||||
}
|
||||
|
||||
want := []ReferenceBinding{{LaneID: "events", SlotName: "normalization_notes", Source: "./normalize.md", BindingSource: contracts.ReferenceBindingSourceConfig}}
|
||||
if !reflect.DeepEqual(resolved.ArtifactLanes[0].NormalizeReferences.Bindings, want) {
|
||||
t.Fatalf("normalize references = %#v, want %#v", resolved.ArtifactLanes[0].NormalizeReferences.Bindings, want)
|
||||
if !reflect.DeepEqual(resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings, want) {
|
||||
t.Fatalf("normalize references = %#v, want %#v", resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings, want)
|
||||
}
|
||||
if refs := resolved.ChunkReferences.Bindings; len(refs) != 0 {
|
||||
t.Fatalf("chunk references = %#v, want none", refs)
|
||||
}
|
||||
if refs := resolved.ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
|
||||
if refs := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
|
||||
t.Fatalf("extract references = %#v, want none", refs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineAppliesPipelineReferenceDefaultToMergeTarget(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
profile.References = map[string]string{"merge_notes": "./merge.md"}
|
||||
profile.References = ExternalReferenceMap(map[string]string{"merge_notes": "./merge.md"})
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "appendorder",
|
||||
Stage: StageMerge,
|
||||
@@ -487,23 +541,23 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToMergeTarget(t *testing.
|
||||
}
|
||||
|
||||
want := []ReferenceBinding{{LaneID: "events", SlotName: "merge_notes", Source: "./merge.md", BindingSource: contracts.ReferenceBindingSourceConfig}}
|
||||
if !reflect.DeepEqual(resolved.ArtifactLanes[0].MergeReferences.Bindings, want) {
|
||||
t.Fatalf("merge references = %#v, want %#v", resolved.ArtifactLanes[0].MergeReferences.Bindings, want)
|
||||
if !reflect.DeepEqual(resolved.Steps[0].ArtifactLanes[0].MergeReferences.Bindings, want) {
|
||||
t.Fatalf("merge references = %#v, want %#v", resolved.Steps[0].ArtifactLanes[0].MergeReferences.Bindings, want)
|
||||
}
|
||||
if refs := resolved.ChunkReferences.Bindings; len(refs) != 0 {
|
||||
t.Fatalf("chunk references = %#v, want none", refs)
|
||||
}
|
||||
if refs := resolved.ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
|
||||
if refs := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
|
||||
t.Fatalf("extract references = %#v, want none", refs)
|
||||
}
|
||||
if refs := resolved.ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
|
||||
if refs := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
|
||||
t.Fatalf("normalize references = %#v, want none", refs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
profile.References = map[string]string{"context": "./context.md"}
|
||||
profile.References = ExternalReferenceMap(map[string]string{"context": "./context.md"})
|
||||
catalog := newProfileCatalogWithOverrides(t,
|
||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
@@ -517,14 +571,14 @@ func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *t
|
||||
}
|
||||
|
||||
assertBindingSource(t, resolved.ChunkReferences.Bindings, "context", "./context.md")
|
||||
assertBindingSource(t, resolved.ArtifactLanes[0].ExtractReferences.Bindings, "context", "./context.md")
|
||||
assertBindingSource(t, resolved.ArtifactLanes[0].MergeReferences.Bindings, "context", "./context.md")
|
||||
assertBindingSource(t, resolved.ArtifactLanes[0].NormalizeReferences.Bindings, "context", "./context.md")
|
||||
assertBindingSource(t, resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings, "context", "./context.md")
|
||||
assertBindingSource(t, resolved.Steps[0].ArtifactLanes[0].MergeReferences.Bindings, "context", "./context.md")
|
||||
assertBindingSource(t, resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings, "context", "./context.md")
|
||||
}
|
||||
|
||||
func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedLane(t *testing.T) {
|
||||
profile := multiLaneProfile()
|
||||
profile.References = map[string]string{"notes_context": "./notes.md"}
|
||||
profile.References = ExternalReferenceMap(map[string]string{"notes_context": "./notes.md"})
|
||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||
Key: "note-extractor",
|
||||
Stage: StageExtract,
|
||||
@@ -539,14 +593,14 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedLane(t *t
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
if refs := resolved.ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
|
||||
if refs := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
|
||||
t.Fatalf("selected lane references = %#v, want none", refs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedNormalizer(t *testing.T) {
|
||||
profile := multiLaneProfile()
|
||||
profile.References = map[string]string{"notes_context": "./notes.md"}
|
||||
profile.References = ExternalReferenceMap(map[string]string{"notes_context": "./notes.md"})
|
||||
lane := profile.Artifacts["notes"]
|
||||
lane.Normalize = Binding("note-normalizer")
|
||||
profile.Artifacts["notes"] = lane
|
||||
@@ -564,17 +618,17 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedNormalize
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
if refs := resolved.ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
|
||||
if refs := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
|
||||
t.Fatalf("selected extract references = %#v, want none", refs)
|
||||
}
|
||||
if refs := resolved.ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
|
||||
if refs := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
|
||||
t.Fatalf("selected normalize references = %#v, want none", refs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePipelineRejectsPipelineReferenceNotDeclaredByAnyLane(t *testing.T) {
|
||||
profile := multiLaneProfile()
|
||||
profile.References = map[string]string{"missing": "./missing.md"}
|
||||
profile.References = ExternalReferenceMap(map[string]string{"missing": "./missing.md"})
|
||||
|
||||
_, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"events"}}, newProfileCatalog(t))
|
||||
if err == nil {
|
||||
@@ -586,7 +640,7 @@ func TestResolvePipelineRejectsPipelineReferenceNotDeclaredByAnyLane(t *testing.
|
||||
func TestResolvePipelineRejectsUndeclaredReferenceSlot(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.References = map[string]string{"missing": "./missing.yml"}
|
||||
lane.References = ExternalReferenceMap(map[string]string{"missing": "./missing.yml"})
|
||||
profile.Artifacts["events"] = lane
|
||||
|
||||
_, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
|
||||
@@ -599,7 +653,7 @@ func TestResolvePipelineRejectsUndeclaredReferenceSlot(t *testing.T) {
|
||||
func TestResolvePipelineRejectsExtractLocalReferenceDeclaredOnlyByNormalizer(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.Extract.References = map[string]string{"normalization_notes": "./normalize.md"}
|
||||
lane.Extract.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./normalize.md"})
|
||||
profile.Artifacts["events"] = lane
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "noop",
|
||||
@@ -619,7 +673,7 @@ func TestResolvePipelineRejectsExtractLocalReferenceDeclaredOnlyByNormalizer(t *
|
||||
func TestResolvePipelineRejectsMergeLocalReferenceDeclaredOnlyByNormalizer(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.Merge.References = map[string]string{"normalization_notes": "./normalize.md"}
|
||||
lane.Merge.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./normalize.md"})
|
||||
profile.Artifacts["events"] = lane
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "noop",
|
||||
@@ -641,7 +695,7 @@ func TestResolvePipelineRejectsMergeLocalReferenceDeclaredOnlyByNormalizer(t *te
|
||||
func TestResolvePipelineRejectsNormalizeLocalReferenceDeclaredOnlyByExtractor(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.Normalize.References = map[string]string{"roster": "./roster.yml"}
|
||||
lane.Normalize.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
|
||||
profile.Artifacts["events"] = lane
|
||||
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
|
||||
Key: "event-extractor",
|
||||
@@ -692,15 +746,15 @@ func TestResolvePipelineRequiresBoundNormalizeReference(t *testing.T) {
|
||||
|
||||
func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTargets(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
profile.References = map[string]string{
|
||||
profile.References = ExternalReferenceMap(map[string]string{
|
||||
"context": "./shared-context.md",
|
||||
"roster": "./shared-roster.yml",
|
||||
"normalization_notes": "./shared-normalize.md",
|
||||
}
|
||||
profile.Chunk.References = map[string]string{"context": "./chunk-context.md"}
|
||||
})
|
||||
profile.Chunk.References = ExternalReferenceMap(map[string]string{"context": "./chunk-context.md"})
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.Extract.References = map[string]string{"roster": "./extract-roster.yml"}
|
||||
lane.Normalize.References = map[string]string{"normalization_notes": "./local-normalize.md"}
|
||||
lane.Extract.References = ExternalReferenceMap(map[string]string{"roster": "./extract-roster.yml"})
|
||||
lane.Normalize.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./local-normalize.md"})
|
||||
profile.Artifacts["events"] = lane
|
||||
catalog := newProfileCatalogWithOverrides(t,
|
||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
|
||||
@@ -714,8 +768,8 @@ func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTarget
|
||||
}
|
||||
|
||||
assertBindingSource(t, resolved.ChunkReferences.Bindings, "context", "./chunk-context.md")
|
||||
assertBindingSource(t, resolved.ArtifactLanes[0].ExtractReferences.Bindings, "roster", "./extract-roster.yml")
|
||||
assertBindingSource(t, resolved.ArtifactLanes[0].NormalizeReferences.Bindings, "normalization_notes", "./local-normalize.md")
|
||||
assertBindingSource(t, resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings, "roster", "./extract-roster.yml")
|
||||
assertBindingSource(t, resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings, "normalization_notes", "./local-normalize.md")
|
||||
}
|
||||
|
||||
func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T) {
|
||||
@@ -742,7 +796,7 @@ func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T
|
||||
|
||||
func TestResolvePipelineReferenceUnbindCanLeaveRequiredSlotMissing(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
profile.References = map[string]string{"roster": "./roster.yml"}
|
||||
profile.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
|
||||
catalog := newProfileCatalogWithOverride(t, ModuleSpec{
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
@@ -764,7 +818,7 @@ func TestResolvePipelineReferenceUnbindCanLeaveRequiredSlotMissing(t *testing.T)
|
||||
|
||||
func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t *testing.T) {
|
||||
profile := baselineProfile()
|
||||
profile.References = map[string]string{"roster": "./roster.yml"}
|
||||
profile.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
|
||||
catalog := emptyProfileCatalog()
|
||||
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, notesCodec())
|
||||
for _, spec := range defaultProfileSpecs() {
|
||||
@@ -791,7 +845,7 @@ func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
if got := resolved.ArtifactLanes[0].ExtractReferences.Bindings[0].Source; got != "./roster.yml" {
|
||||
if got := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings[0].Source; got != "./roster.yml" {
|
||||
t.Fatalf("reference source = %q, want ./roster.yml", got)
|
||||
}
|
||||
}
|
||||
@@ -998,7 +1052,7 @@ func TestResolvePipelineOrdersLanesDeterministically(t *testing.T) {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
got := laneIDs(resolved.ArtifactLanes)
|
||||
got := laneIDs(resolved.Steps[0].ArtifactLanes)
|
||||
want := []string{"events", "notes", "summaries"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("lane IDs = %#v, want %#v", got, want)
|
||||
@@ -1162,7 +1216,7 @@ func TestResolvedPipelineDigestIncludesCompleteValidatorPolicy(t *testing.T) {
|
||||
value.ValidatorChains[0].Validators[0].Binding.Options["threshold"] = 0.75
|
||||
}},
|
||||
{name: "references", mutate: func(value *ResolvedPipeline) {
|
||||
value.ValidatorChains[0].Validators[0].Binding.References["rules"] = "other.md"
|
||||
value.ValidatorChains[0].Validators[0].Binding.References["rules"] = ExternalReference("other.md")
|
||||
}},
|
||||
{name: "execution class", mutate: func(value *ResolvedPipeline) {
|
||||
value.ValidatorChains[0].Validators[0].ExecutionClass = contracts.ExecutionClassDeterministic
|
||||
@@ -1194,9 +1248,9 @@ func TestResolvedPipelineDigestCanonicalizesValidatorBindingMaps(t *testing.T) {
|
||||
right.ValidatorChains[0].Validators[0].Binding.Options = map[string]any{}
|
||||
right.ValidatorChains[0].Validators[0].Binding.Options["threshold"] = 0.5
|
||||
right.ValidatorChains[0].Validators[0].Binding.Options["mode"] = "strict"
|
||||
right.ValidatorChains[0].Validators[0].Binding.References = map[string]string{}
|
||||
right.ValidatorChains[0].Validators[0].Binding.References["examples"] = "examples.md"
|
||||
right.ValidatorChains[0].Validators[0].Binding.References["rules"] = "rules.md"
|
||||
right.ValidatorChains[0].Validators[0].Binding.References = ExternalReferenceMap(map[string]string{})
|
||||
right.ValidatorChains[0].Validators[0].Binding.References["examples"] = ExternalReference("examples.md")
|
||||
right.ValidatorChains[0].Validators[0].Binding.References["rules"] = ExternalReference("rules.md")
|
||||
|
||||
leftDigest, err := resolvedPipelineDigest(left)
|
||||
if err != nil {
|
||||
@@ -1225,7 +1279,7 @@ func validatorDigestFixture() ResolvedPipeline {
|
||||
LLMProfile: "careful",
|
||||
Retries: 2,
|
||||
Options: map[string]any{"mode": "strict", "threshold": 0.5},
|
||||
References: map[string]string{"rules": "rules.md", "examples": "examples.md"},
|
||||
References: ExternalReferenceMap(map[string]string{"rules": "rules.md", "examples": "examples.md"}),
|
||||
},
|
||||
ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Target: ValidatorTargetTyped,
|
||||
|
||||
@@ -36,37 +36,51 @@ func MaterializeReferences(resolved ResolvedPipeline, catalog ModuleCatalog, opt
|
||||
}
|
||||
out.ChunkReferences.ReferenceSet = chunkReferenceSet
|
||||
warnings := append([]contracts.Warning(nil), chunkWarnings...)
|
||||
if len(resolved.ArtifactLanes) == 0 {
|
||||
if len(resolved.Steps) == 0 {
|
||||
return out, warnings, nil
|
||||
}
|
||||
|
||||
out.ArtifactLanes = make([]ResolvedArtifactLane, len(resolved.ArtifactLanes))
|
||||
for i, lane := range resolved.ArtifactLanes {
|
||||
materializeLane := func(lane ResolvedArtifactLane) (ResolvedArtifactLane, []contracts.Warning, error) {
|
||||
materializedLane := lane
|
||||
var allWarnings []contracts.Warning
|
||||
materializedLane.ExtractReferences = CloneReferenceTarget(lane.ExtractReferences)
|
||||
materializedLane.MergeReferences = CloneReferenceTarget(lane.MergeReferences)
|
||||
materializedLane.NormalizeReferences = CloneReferenceTarget(lane.NormalizeReferences)
|
||||
extractReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.ExtractReferences, lane.ArtifactKind, catalog, options)
|
||||
if err != nil {
|
||||
return ResolvedPipeline{}, nil, err
|
||||
return ResolvedArtifactLane{}, nil, err
|
||||
}
|
||||
materializedLane.ExtractReferences.ReferenceSet = extractReferenceSet
|
||||
warnings = append(warnings, laneWarnings...)
|
||||
allWarnings = append(allWarnings, laneWarnings...)
|
||||
|
||||
mergeReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.MergeReferences, lane.ArtifactKind, catalog, options)
|
||||
if err != nil {
|
||||
return ResolvedPipeline{}, nil, err
|
||||
return ResolvedArtifactLane{}, nil, err
|
||||
}
|
||||
materializedLane.MergeReferences.ReferenceSet = mergeReferenceSet
|
||||
warnings = append(warnings, laneWarnings...)
|
||||
allWarnings = append(allWarnings, laneWarnings...)
|
||||
|
||||
normalizeReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.NormalizeReferences, lane.ArtifactKind, catalog, options)
|
||||
if err != nil {
|
||||
return ResolvedPipeline{}, nil, err
|
||||
return ResolvedArtifactLane{}, nil, err
|
||||
}
|
||||
materializedLane.NormalizeReferences.ReferenceSet = normalizeReferenceSet
|
||||
warnings = append(warnings, laneWarnings...)
|
||||
out.ArtifactLanes[i] = materializedLane
|
||||
allWarnings = append(allWarnings, laneWarnings...)
|
||||
return materializedLane, allWarnings, nil
|
||||
}
|
||||
if len(resolved.Steps) > 0 {
|
||||
out.Steps = make([]ResolvedPipelineStep, len(resolved.Steps))
|
||||
for i, step := range resolved.Steps {
|
||||
out.Steps[i].ID = step.ID
|
||||
out.Steps[i].ArtifactLanes = make([]ResolvedArtifactLane, len(step.ArtifactLanes))
|
||||
for j, lane := range step.ArtifactLanes {
|
||||
materializedLane, laneWarnings, err := materializeLane(lane)
|
||||
if err != nil {
|
||||
return ResolvedPipeline{}, nil, err
|
||||
}
|
||||
out.Steps[i].ArtifactLanes[j] = materializedLane
|
||||
warnings = append(warnings, laneWarnings...)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, warnings, nil
|
||||
}
|
||||
@@ -99,6 +113,12 @@ func materializeReferenceTarget(
|
||||
if !ok {
|
||||
return contracts.ReferenceSet{}, nil, fmt.Errorf("%s reference slot %q is not declared by %s module %q", referenceTargetContext(pipelineID, target), slotName, target.Stage, target.Module)
|
||||
}
|
||||
if binding.Artifact != nil {
|
||||
// Generated selectors are resolved at the step handoff. Keep the
|
||||
// declared slot and its constraints, but do not materialize bytes yet.
|
||||
set.Slots[slotName] = contracts.ResolvedReferenceSlot{Slot: cloneReferenceSlot(slot)}
|
||||
continue
|
||||
}
|
||||
|
||||
path, err := referencePath(binding, options)
|
||||
if err != nil {
|
||||
@@ -273,6 +293,7 @@ func fileURI(path string) string {
|
||||
|
||||
func cloneReferenceSlot(slot contracts.ReferenceSlot) contracts.ReferenceSlot {
|
||||
slot.AcceptedMediaTypes = append([]string(nil), slot.AcceptedMediaTypes...)
|
||||
slot.AcceptedArtifactKinds = append([]contracts.ArtifactKind(nil), slot.AcceptedArtifactKinds...)
|
||||
return slot
|
||||
}
|
||||
|
||||
@@ -292,8 +313,7 @@ func CloneReferenceSet(in contracts.ReferenceSet) contracts.ReferenceSet {
|
||||
if len(slot.Items) > 0 {
|
||||
items := make([]contracts.ReferenceItem, len(slot.Items))
|
||||
for i, item := range slot.Items {
|
||||
item.Content = append([]byte(nil), item.Content...)
|
||||
items[i] = item
|
||||
items[i] = contracts.CloneReferenceItem(item)
|
||||
}
|
||||
slot.Items = items
|
||||
}
|
||||
@@ -304,7 +324,16 @@ func CloneReferenceSet(in contracts.ReferenceSet) contracts.ReferenceSet {
|
||||
|
||||
func CloneReferenceTarget(in ResolvedReferenceTarget) ResolvedReferenceTarget {
|
||||
out := in
|
||||
out.Bindings = append([]ReferenceBinding(nil), in.Bindings...)
|
||||
if len(in.Bindings) > 0 {
|
||||
out.Bindings = make([]ReferenceBinding, len(in.Bindings))
|
||||
for i, binding := range in.Bindings {
|
||||
out.Bindings[i] = binding
|
||||
if binding.Artifact != nil {
|
||||
artifact := *binding.Artifact
|
||||
out.Bindings[i].Artifact = &artifact
|
||||
}
|
||||
}
|
||||
}
|
||||
out.ReferenceSet = CloneReferenceSet(in.ReferenceSet)
|
||||
return out
|
||||
}
|
||||
@@ -312,10 +341,14 @@ func CloneReferenceTarget(in ResolvedReferenceTarget) ResolvedReferenceTarget {
|
||||
func ReferenceProvenance(resolved ResolvedPipeline) []artifacts.ReferenceProvenance {
|
||||
provenance := []artifacts.ReferenceProvenance{}
|
||||
provenance = append(provenance, referenceTargetProvenance(resolved.ChunkReferences)...)
|
||||
for _, lane := range resolved.ArtifactLanes {
|
||||
provenance = append(provenance, referenceTargetProvenance(lane.ExtractReferences)...)
|
||||
provenance = append(provenance, referenceTargetProvenance(lane.MergeReferences)...)
|
||||
provenance = append(provenance, referenceTargetProvenance(lane.NormalizeReferences)...)
|
||||
if len(resolved.Steps) > 0 {
|
||||
for _, step := range resolved.Steps {
|
||||
for _, lane := range step.ArtifactLanes {
|
||||
provenance = append(provenance, referenceTargetProvenance(lane.ExtractReferences)...)
|
||||
provenance = append(provenance, referenceTargetProvenance(lane.MergeReferences)...)
|
||||
provenance = append(provenance, referenceTargetProvenance(lane.NormalizeReferences)...)
|
||||
}
|
||||
}
|
||||
}
|
||||
return provenance
|
||||
}
|
||||
@@ -333,16 +366,30 @@ func referenceTargetProvenance(target ResolvedReferenceTarget) []artifacts.Refer
|
||||
for _, slotName := range slotNames {
|
||||
slot := target.ReferenceSet.Slots[slotName]
|
||||
for _, item := range slot.Items {
|
||||
schemaDigest := ""
|
||||
if item.ArtifactSchema.ID != "" || item.ArtifactSchema.Name != "" || item.ArtifactSchema.Version != "" || len(item.ArtifactSchema.JSONSchema) > 0 {
|
||||
schemaDigest = contracts.DigestArtifactSchema(item.ArtifactSchema)
|
||||
}
|
||||
provenance = append(provenance, artifacts.ReferenceProvenance{
|
||||
Stage: string(target.Stage),
|
||||
LaneID: target.LaneID,
|
||||
SlotName: item.SlotName,
|
||||
OriginType: item.Origin.Type,
|
||||
OriginURI: item.Origin.URI,
|
||||
Digest: item.Digest,
|
||||
MediaType: item.MediaType,
|
||||
SizeBytes: item.SizeBytes,
|
||||
BindingSource: item.BindingSource,
|
||||
Stage: string(target.Stage),
|
||||
StepID: target.StepID,
|
||||
LaneID: target.LaneID,
|
||||
SlotName: item.SlotName,
|
||||
OriginType: item.Origin.Type,
|
||||
OriginURI: item.Origin.URI,
|
||||
Digest: item.Digest,
|
||||
MediaType: item.MediaType,
|
||||
SizeBytes: item.SizeBytes,
|
||||
BindingSource: item.BindingSource,
|
||||
ArtifactKind: string(item.ArtifactKind),
|
||||
SchemaID: item.ArtifactSchema.ID,
|
||||
SchemaName: item.ArtifactSchema.Name,
|
||||
SchemaVersion: item.ArtifactSchema.Version,
|
||||
SchemaDigest: schemaDigest,
|
||||
ProducerPipeline: item.Producer.PipelineID,
|
||||
ProducerStep: item.Producer.StepID,
|
||||
ProducerLane: item.Producer.LaneID,
|
||||
ProducerModule: item.Producer.ModuleKey,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -21,9 +21,9 @@ func TestMaterializeReferencesResolvesPathsAndDigestsContent(t *testing.T) {
|
||||
writeReferenceFile(t, cliReference, []byte("cli text"))
|
||||
|
||||
pipeline := baselineProfile()
|
||||
pipeline.References = map[string]string{"roster": "config-reference.txt"}
|
||||
pipeline.References = ExternalReferenceMap(map[string]string{"roster": "config-reference.txt"})
|
||||
lane := pipeline.Artifacts["events"]
|
||||
lane.References = map[string]string{"glossary": "cli-reference.txt"}
|
||||
lane.References = ExternalReferenceMap(map[string]string{"glossary": "cli-reference.txt"})
|
||||
pipeline.Artifacts["events"] = lane
|
||||
catalog := referenceCatalog(t, []contracts.ReferenceSlot{
|
||||
{Name: "roster"},
|
||||
@@ -56,12 +56,12 @@ func TestMaterializeReferencesResolvesPathsAndDigestsContent(t *testing.T) {
|
||||
t.Fatalf("MaterializeReferences(second) error = %v, want nil", err)
|
||||
}
|
||||
|
||||
referenceSet := first.ArtifactLanes[0].ExtractReferences.ReferenceSet
|
||||
referenceSet := first.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet
|
||||
roster := referenceSet.Slots["roster"].Items[0]
|
||||
if string(roster.Content) != "config text" {
|
||||
t.Fatalf("roster content = %q, want config text", roster.Content)
|
||||
}
|
||||
if roster.Digest != referenceDigest([]byte("config text")) || roster.Digest != second.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0].Digest {
|
||||
if roster.Digest != referenceDigest([]byte("config text")) || roster.Digest != second.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0].Digest {
|
||||
t.Fatalf("roster digest = %q, want stable digest", roster.Digest)
|
||||
}
|
||||
if roster.BindingSource != contracts.ReferenceBindingSourceConfig {
|
||||
@@ -113,12 +113,12 @@ func TestMaterializeReferencesStoresSetsAndProvenanceForAllTargets(t *testing.T)
|
||||
writeReferenceFile(t, filepath.Join(configDir, "normalize.txt"), []byte("normalize text"))
|
||||
|
||||
profile := baselineProfile()
|
||||
profile.References = map[string]string{
|
||||
profile.References = ExternalReferenceMap(map[string]string{
|
||||
"scene_guide": "chunk.txt",
|
||||
"roster": "extract.txt",
|
||||
"merge_notes": "merge.txt",
|
||||
"normalization_notes": "normalize.txt",
|
||||
}
|
||||
})
|
||||
catalog := referenceCatalogForTargets(t,
|
||||
[]contracts.ReferenceSlot{{Name: "scene_guide"}},
|
||||
[]contracts.ReferenceSlot{{Name: "roster"}},
|
||||
@@ -144,15 +144,15 @@ func TestMaterializeReferencesStoresSetsAndProvenanceForAllTargets(t *testing.T)
|
||||
if string(chunkItem.Content) != "chunk text" {
|
||||
t.Fatalf("chunk content = %q, want chunk text", chunkItem.Content)
|
||||
}
|
||||
extractItem := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||
extractItem := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||
if string(extractItem.Content) != "extract text" {
|
||||
t.Fatalf("extract content = %q, want extract text", extractItem.Content)
|
||||
}
|
||||
mergeItem := materialized.ArtifactLanes[0].MergeReferences.ReferenceSet.Slots["merge_notes"].Items[0]
|
||||
mergeItem := materialized.Steps[0].ArtifactLanes[0].MergeReferences.ReferenceSet.Slots["merge_notes"].Items[0]
|
||||
if string(mergeItem.Content) != "merge text" {
|
||||
t.Fatalf("merge content = %q, want merge text", mergeItem.Content)
|
||||
}
|
||||
normalizeItem := materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["normalization_notes"].Items[0]
|
||||
normalizeItem := materialized.Steps[0].ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["normalization_notes"].Items[0]
|
||||
if string(normalizeItem.Content) != "normalize text" {
|
||||
t.Fatalf("normalize content = %q, want normalize text", normalizeItem.Content)
|
||||
}
|
||||
@@ -202,22 +202,25 @@ func TestMaterializeReferencesUsesLaneArtifactVariant(t *testing.T) {
|
||||
}
|
||||
resolved := ResolvedPipeline{
|
||||
ID: "variants",
|
||||
ArtifactLanes: []ResolvedArtifactLane{{
|
||||
ID: "alpha",
|
||||
ArtifactKind: "test/alpha",
|
||||
MergeReferences: referenceTarget(StageMerge, "alpha", "shared/merge", []ReferenceBinding{{
|
||||
Stage: StageMerge, LaneID: "alpha", SlotName: "alpha_merge", Source: "merge.txt", BindingSource: contracts.ReferenceBindingSourceConfig,
|
||||
}}),
|
||||
NormalizeReferences: referenceTarget(StageNormalize, "alpha", "shared/normalize", []ReferenceBinding{{
|
||||
Stage: StageNormalize, LaneID: "alpha", SlotName: "alpha_normalize", Source: "normalize.txt", BindingSource: contracts.ReferenceBindingSourceConfig,
|
||||
}}),
|
||||
Steps: []ResolvedPipelineStep{{
|
||||
ID: "default",
|
||||
ArtifactLanes: []ResolvedArtifactLane{{
|
||||
ID: "alpha",
|
||||
ArtifactKind: "test/alpha",
|
||||
MergeReferences: referenceTarget(StageMerge, "alpha", "shared/merge", []ReferenceBinding{{
|
||||
Stage: StageMerge, LaneID: "alpha", SlotName: "alpha_merge", Source: "merge.txt", BindingSource: contracts.ReferenceBindingSourceConfig,
|
||||
}}),
|
||||
NormalizeReferences: referenceTarget(StageNormalize, "alpha", "shared/normalize", []ReferenceBinding{{
|
||||
Stage: StageNormalize, LaneID: "alpha", SlotName: "alpha_normalize", Source: "normalize.txt", BindingSource: contracts.ReferenceBindingSourceConfig,
|
||||
}}),
|
||||
}},
|
||||
}},
|
||||
}
|
||||
materialized, _, err := MaterializeReferences(resolved, ModuleCatalog{Mergers: mergers, Normalizers: normalizers}, ReferenceMaterializationOptions{ConfigPath: filepath.Join(configDir, "notarius.yml")})
|
||||
if err != nil {
|
||||
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
|
||||
}
|
||||
lane := materialized.ArtifactLanes[0]
|
||||
lane := materialized.Steps[0].ArtifactLanes[0]
|
||||
if got := string(lane.MergeReferences.ReferenceSet.Slots["alpha_merge"].Items[0].Content); got != "merge alpha" {
|
||||
t.Fatalf("merge reference = %q", got)
|
||||
}
|
||||
@@ -266,7 +269,7 @@ func TestMaterializeReferencesAllowsAnyMediaTypeWhenSlotDoesNotRestrictIt(t *tes
|
||||
if err != nil {
|
||||
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
|
||||
}
|
||||
item := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||
item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||
if item.MediaType != unknownMediaType {
|
||||
t.Fatalf("MediaType = %q, want %q", item.MediaType, unknownMediaType)
|
||||
}
|
||||
@@ -285,7 +288,7 @@ func TestMaterializeReferencesAcceptsDeclaredMarkdownMediaType(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
|
||||
}
|
||||
item := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["glossary"].Items[0]
|
||||
item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["glossary"].Items[0]
|
||||
if item.MediaType != "text/markdown" {
|
||||
t.Fatalf("MediaType = %q, want text/markdown", item.MediaType)
|
||||
}
|
||||
@@ -304,7 +307,7 @@ func TestMaterializeReferencesAcceptsDeclaredJSONMediaType(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
|
||||
}
|
||||
item := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||
item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||
if item.MediaType != "application/json" {
|
||||
t.Fatalf("MediaType = %q, want application/json", item.MediaType)
|
||||
}
|
||||
@@ -323,7 +326,7 @@ func TestMaterializeReferencesAcceptsDeclaredYAMLMediaType(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
|
||||
}
|
||||
item := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||
item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||
if item.MediaType != "application/yaml" {
|
||||
t.Fatalf("MediaType = %q, want application/yaml", item.MediaType)
|
||||
}
|
||||
@@ -357,7 +360,7 @@ func TestMaterializeReferencesMatchesAcceptedMediaTypesIgnoringParameters(t *tes
|
||||
if err != nil {
|
||||
t.Fatalf("MaterializeReferences() error = %v, want nil", err)
|
||||
}
|
||||
item := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||
item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||
if item.MediaType != referenceMediaType {
|
||||
t.Fatalf("MediaType = %q, want %q", item.MediaType, referenceMediaType)
|
||||
}
|
||||
@@ -393,7 +396,7 @@ func TestMaterializeReferencesWarnsForEmptyFiles(t *testing.T) {
|
||||
if len(warnings) != 1 || warnings[0].ReasonCode != "empty_reference" {
|
||||
t.Fatalf("warnings = %#v, want empty reference warning", warnings)
|
||||
}
|
||||
item := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||
item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
|
||||
if item.SizeBytes != 0 || item.Digest != referenceDigest(nil) {
|
||||
t.Fatalf("empty item = %#v, want zero size and empty digest", item)
|
||||
}
|
||||
@@ -406,11 +409,11 @@ func TestMaterializeReferencesWarningScopesIncludeTargetContext(t *testing.T) {
|
||||
writeReferenceFile(t, filepath.Join(configDir, "normalize.txt"), nil)
|
||||
|
||||
profile := baselineProfile()
|
||||
profile.References = map[string]string{
|
||||
profile.References = ExternalReferenceMap(map[string]string{
|
||||
"scene_guide": "chunk.txt",
|
||||
"roster": "extract.txt",
|
||||
"normalization_notes": "normalize.txt",
|
||||
}
|
||||
})
|
||||
catalog := referenceCatalogForTargets(t,
|
||||
[]contracts.ReferenceSlot{{Name: "scene_guide"}},
|
||||
[]contracts.ReferenceSlot{{Name: "roster"}},
|
||||
@@ -464,18 +467,18 @@ func resolvedPipelineWithTargetReference(t *testing.T, stage ModuleStage, laneID
|
||||
profile := baselineProfile()
|
||||
switch stage {
|
||||
case StageChunk:
|
||||
profile.Chunk.References = map[string]string{slotName: source}
|
||||
profile.Chunk.References = ExternalReferenceMap(map[string]string{slotName: source})
|
||||
case StageExtract:
|
||||
lane := profile.Artifacts[laneID]
|
||||
lane.References = map[string]string{slotName: source}
|
||||
lane.References = ExternalReferenceMap(map[string]string{slotName: source})
|
||||
profile.Artifacts[laneID] = lane
|
||||
case StageMerge:
|
||||
lane := profile.Artifacts[laneID]
|
||||
lane.Merge.References = map[string]string{slotName: source}
|
||||
lane.Merge.References = ExternalReferenceMap(map[string]string{slotName: source})
|
||||
profile.Artifacts[laneID] = lane
|
||||
case StageNormalize:
|
||||
lane := profile.Artifacts[laneID]
|
||||
lane.Normalize.References = map[string]string{slotName: source}
|
||||
lane.Normalize.References = ExternalReferenceMap(map[string]string{slotName: source})
|
||||
profile.Artifacts[laneID] = lane
|
||||
default:
|
||||
t.Fatalf("unsupported reference target stage %q", stage)
|
||||
@@ -490,11 +493,11 @@ func resolvedPipelineWithTargetReference(t *testing.T, stage ModuleStage, laneID
|
||||
case StageChunk:
|
||||
resolved.ChunkReferences.Bindings[0].BindingSource = bindingSource
|
||||
case StageExtract:
|
||||
resolved.ArtifactLanes[0].ExtractReferences.Bindings[0].BindingSource = bindingSource
|
||||
resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings[0].BindingSource = bindingSource
|
||||
case StageMerge:
|
||||
resolved.ArtifactLanes[0].MergeReferences.Bindings[0].BindingSource = bindingSource
|
||||
resolved.Steps[0].ArtifactLanes[0].MergeReferences.Bindings[0].BindingSource = bindingSource
|
||||
case StageNormalize:
|
||||
resolved.ArtifactLanes[0].NormalizeReferences.Bindings[0].BindingSource = bindingSource
|
||||
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings[0].BindingSource = bindingSource
|
||||
}
|
||||
}
|
||||
return resolved
|
||||
|
||||
@@ -38,27 +38,30 @@ func New() *Runner {
|
||||
}
|
||||
|
||||
type RunInput struct {
|
||||
Prepared *PreparedPipeline
|
||||
SourceID string
|
||||
Path string
|
||||
RawInput []byte
|
||||
SessionID string
|
||||
RunID string
|
||||
StartedAt time.Time
|
||||
LLMProfiles []artifacts.LLMProfileManifest
|
||||
Metadata map[string]any
|
||||
Warnings []contracts.Warning
|
||||
ChunkCacheMode ChunkCacheMode
|
||||
ChunkPlans ChunkPlanStore
|
||||
Checkpoints CheckpointRecorder
|
||||
Checkpoint CheckpointLoader
|
||||
Debug DebugRecorder
|
||||
Prepared *PreparedPipeline
|
||||
SourceID string
|
||||
Path string
|
||||
RawInput []byte
|
||||
SessionID string
|
||||
RunID string
|
||||
StartedAt time.Time
|
||||
LLMProfiles []artifacts.LLMProfileManifest
|
||||
Metadata map[string]any
|
||||
Warnings []contracts.Warning
|
||||
ChunkCacheMode ChunkCacheMode
|
||||
ChunkPlans ChunkPlanStore
|
||||
Checkpoints CheckpointRecorder
|
||||
Checkpoint CheckpointLoader
|
||||
CheckpointPolicy CheckpointExecutionPolicy
|
||||
Debug DebugRecorder
|
||||
// ExtractWorkers bounds run-wide extract jobs. Values less than one use a
|
||||
// single worker so direct framework callers retain deterministic behavior.
|
||||
ExtractWorkers int
|
||||
|
||||
pipeline ResolvedPipeline
|
||||
llmClient contracts.StructuredLLMClient
|
||||
pipeline ResolvedPipeline
|
||||
llmClient contracts.StructuredLLMClient
|
||||
stepID string
|
||||
references map[referenceTargetKey]contracts.ReferenceSet
|
||||
}
|
||||
|
||||
type RunOutput struct {
|
||||
@@ -129,7 +132,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
||||
return failOutput(output), err
|
||||
}
|
||||
sourceCheckpoint, sourceDecision := checkpointLoader.Source(adapter.Key())
|
||||
recordCheckpointEvent(&output, checkpointLoader, "source", "", adapter.Key(), sourceDecision)
|
||||
recordCheckpointEvent(&output, checkpointLoader, "source", "", "", adapter.Key(), sourceDecision)
|
||||
doc := sourceCheckpoint.Document
|
||||
sourceStarted := time.Now().UTC()
|
||||
if err := writeDebugTimed(debugRecorder, "source/input.json", debugTimedEnvelope{
|
||||
@@ -249,13 +252,9 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
||||
}
|
||||
|
||||
if chunkResult.accepted {
|
||||
laneOutput, laneErr := r.runLanes(ctx, input, checkpoints, checkpointLoader, doc, sourceInput, sessionID, chunkResult.chunks)
|
||||
if err := mergeLaneOutput(&output, laneOutput); err != nil {
|
||||
if err := r.runPreparedSteps(ctx, input, checkpoints, checkpointLoader, doc, sourceInput, sessionID, chunkResult.chunks, &output); err != nil {
|
||||
return failOutput(output), err
|
||||
}
|
||||
if laneErr != nil {
|
||||
return failOutput(output), laneErr
|
||||
}
|
||||
}
|
||||
|
||||
if len(output.Rejected) > 0 {
|
||||
@@ -322,6 +321,27 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
|
||||
return output, nil
|
||||
}
|
||||
|
||||
func (r *Runner) runPreparedSteps(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk, output *RunOutput) error {
|
||||
for _, step := range input.Prepared.Steps {
|
||||
stepInput := input
|
||||
stepInput.stepID = step.ID
|
||||
stepReferences, referenceProvenance, err := buildStepReferenceSets(input, step, output.NormalizeOutputs)
|
||||
if err != nil {
|
||||
return fmt.Errorf("prepare generated references for pipeline step %q: %w", step.ID, err)
|
||||
}
|
||||
stepInput.references = stepReferences
|
||||
output.Manifest.References = append(output.Manifest.References, referenceProvenance...)
|
||||
laneOutput, laneErr := r.runLanes(ctx, stepInput, step, checkpoints, loader, doc, sourceInput, sessionID, chunks)
|
||||
if err := mergeLaneOutput(output, laneOutput); err != nil {
|
||||
return err
|
||||
}
|
||||
if laneErr != nil {
|
||||
return fmt.Errorf("execute pipeline step %q: %w", step.ID, laneErr)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func runWithRetry(ctx context.Context, retries int, run func(attempt int) (bool, *contracts.RejectedOutput, error)) (bool, *contracts.RejectedOutput, error) {
|
||||
attempts := retries + 1
|
||||
var last *contracts.RejectedOutput
|
||||
@@ -460,7 +480,10 @@ func validateRunInput(input RunInput) error {
|
||||
if mode != ChunkCacheBypass && input.ChunkPlans == nil {
|
||||
return fmt.Errorf("chunk plan store is required for %q mode", mode)
|
||||
}
|
||||
return validateResolvedPipeline(input.Prepared.resolved)
|
||||
if err := validateResolvedPipeline(input.Prepared.resolved); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateResolvedPipeline(pipeline ResolvedPipeline) error {
|
||||
@@ -479,10 +502,10 @@ func validateResolvedPipeline(pipeline ResolvedPipeline) error {
|
||||
if pipeline.Output.Module == "" {
|
||||
return fmt.Errorf("resolved pipeline output module must not be empty")
|
||||
}
|
||||
if len(pipeline.ArtifactLanes) == 0 {
|
||||
if len(pipeline.Steps) == 0 {
|
||||
return fmt.Errorf("resolved pipeline artifact lanes must not be empty")
|
||||
}
|
||||
for _, lane := range pipeline.ArtifactLanes {
|
||||
for _, lane := range pipeline.AllArtifactLanes() {
|
||||
if lane.ID == "" {
|
||||
return fmt.Errorf("resolved pipeline artifact lane id must not be empty")
|
||||
}
|
||||
@@ -530,15 +553,16 @@ func manifestFromPipeline(input RunInput) artifacts.RunManifest {
|
||||
RequestedModule: pipeline.Chunk.Module,
|
||||
},
|
||||
OutputEncoder: pipeline.Output.Module,
|
||||
ArtifactLanes: make([]artifacts.ArtifactLaneManifest, 0, len(pipeline.ArtifactLanes)),
|
||||
ArtifactLanes: make([]artifacts.ArtifactLaneManifest, 0, len(pipeline.AllArtifactLanes())),
|
||||
ValidatorChains: validatorChainManifests(pipeline.ValidatorChains),
|
||||
RunID: runID,
|
||||
StartedAt: timePtr(startedAt),
|
||||
References: ReferenceProvenance(pipeline),
|
||||
LLMProfiles: cloneLLMProfiles(input.LLMProfiles),
|
||||
}
|
||||
for _, lane := range pipeline.ArtifactLanes {
|
||||
for _, lane := range pipeline.AllArtifactLanes() {
|
||||
laneManifest := artifacts.ArtifactLaneManifest{
|
||||
StepID: lane.StepID,
|
||||
ID: lane.ID,
|
||||
Extractor: lane.Extract.Module,
|
||||
Merger: lane.Merge.Module,
|
||||
@@ -581,29 +605,47 @@ func failOutput(output RunOutput) RunOutput {
|
||||
return output
|
||||
}
|
||||
|
||||
func recordCheckpointEvent(output *RunOutput, loader CheckpointLoader, stage string, laneID string, moduleKey string, decision CheckpointDecision) {
|
||||
func recordCheckpointEvent(output *RunOutput, loader CheckpointLoader, stage string, stepID string, laneID string, moduleKey string, decision CheckpointDecision) {
|
||||
if output == nil || loader == nil || !loader.Enabled() {
|
||||
return
|
||||
}
|
||||
action := "executed"
|
||||
if decision.Reused {
|
||||
action = "reused"
|
||||
}
|
||||
action := checkpointDecisionCategory(decision)
|
||||
output.CheckpointEvents = append(output.CheckpointEvents, CheckpointEvent{
|
||||
Stage: stage,
|
||||
LaneID: laneID,
|
||||
ModuleKey: moduleKey,
|
||||
Action: action,
|
||||
Reason: decision.Reason,
|
||||
Stage: stage,
|
||||
StepID: stepID,
|
||||
LaneID: laneID,
|
||||
ModuleKey: moduleKey,
|
||||
Action: action,
|
||||
Category: checkpointDecisionCategory(decision),
|
||||
ReasonCode: decision.ReasonCode,
|
||||
Detail: decision.Detail,
|
||||
Reason: decision.Reason,
|
||||
})
|
||||
}
|
||||
|
||||
func checkpointDecisionCategory(decision CheckpointDecision) CheckpointDecisionCategory {
|
||||
if decision.Category != "" {
|
||||
return decision.Category
|
||||
}
|
||||
if decision.Reused {
|
||||
return CheckpointDecisionReused
|
||||
}
|
||||
return CheckpointDecisionExecuted
|
||||
}
|
||||
|
||||
func populateOutputManifest(output *RunOutput) {
|
||||
if output == nil {
|
||||
return
|
||||
}
|
||||
output.Manifest.NormalizedOutputs = normalizedOutputManifests(output.NormalizeOutputs)
|
||||
output.Manifest.RejectedOutputs = rejectedOutputManifests(output.Rejected)
|
||||
if len(output.CheckpointEvents) > 0 {
|
||||
decisions := make([]artifacts.CheckpointDecisionManifest, 0, len(output.CheckpointEvents))
|
||||
for _, event := range output.CheckpointEvents {
|
||||
decisions = append(decisions, artifacts.CheckpointDecisionManifest{Stage: event.Stage, StepID: event.StepID, LaneID: event.LaneID, ModuleKey: event.ModuleKey, Category: string(event.Category), ReasonCode: string(event.ReasonCode), Detail: event.Detail})
|
||||
}
|
||||
output.Manifest.CheckpointDecisions = decisions
|
||||
}
|
||||
}
|
||||
|
||||
func normalizedOutputManifests(outputs []contracts.SerializedOutput) []artifacts.NormalizedOutputManifest {
|
||||
@@ -613,6 +655,7 @@ func normalizedOutputManifests(outputs []contracts.SerializedOutput) []artifacts
|
||||
manifests := make([]artifacts.NormalizedOutputManifest, 0, len(outputs))
|
||||
for _, output := range outputs {
|
||||
manifests = append(manifests, artifacts.NormalizedOutputManifest{
|
||||
StepID: output.StepID,
|
||||
LaneID: output.LaneID,
|
||||
ModuleKey: output.NormalizerKey,
|
||||
SourceID: output.SourceID,
|
||||
@@ -635,6 +678,7 @@ func rejectedOutputManifests(rejected []contracts.RejectedOutput) []artifacts.Re
|
||||
for _, output := range rejected {
|
||||
manifests = append(manifests, artifacts.RejectedOutputManifest{
|
||||
Stage: output.Stage,
|
||||
StepID: output.StepID,
|
||||
LaneID: output.LaneID,
|
||||
ModuleKey: output.ModuleKey,
|
||||
ChunkID: output.ChunkID,
|
||||
|
||||
300
internal/framework/pipeline/runner_accepted_checkpoint_test.go
Normal file
300
internal/framework/pipeline/runner_accepted_checkpoint_test.go
Normal file
@@ -0,0 +1,300 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type acceptedCheckpointLoader struct {
|
||||
CheckpointLoader
|
||||
accepted map[string]NormalizeCheckpoint
|
||||
acceptedDecision map[string]CheckpointDecision
|
||||
extractDeps map[string][]CheckpointFingerprint
|
||||
}
|
||||
|
||||
func newAcceptedCheckpointLoader() *acceptedCheckpointLoader {
|
||||
return &acceptedCheckpointLoader{
|
||||
CheckpointLoader: NoopCheckpointLoader(),
|
||||
accepted: make(map[string]NormalizeCheckpoint),
|
||||
acceptedDecision: make(map[string]CheckpointDecision),
|
||||
extractDeps: make(map[string][]CheckpointFingerprint),
|
||||
}
|
||||
}
|
||||
|
||||
func (l *acceptedCheckpointLoader) Enabled() bool { return true }
|
||||
func (l *acceptedCheckpointLoader) AcceptedNormalize(stepID, laneID, _ string) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
key := CheckpointLaneKey(stepID, laneID)
|
||||
return l.accepted[key], l.acceptedDecision[key]
|
||||
}
|
||||
func (l *acceptedCheckpointLoader) Extract(laneID string, _ string, dependencies []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||
l.extractDeps[laneID] = append([]CheckpointFingerprint(nil), dependencies...)
|
||||
return ExtractCheckpoint{}, NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing, "checkpoint missing")
|
||||
}
|
||||
func (l *acceptedCheckpointLoader) Merge(_ string, _ string, _ []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
|
||||
return MergeCheckpoint{}, NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing, "checkpoint missing")
|
||||
}
|
||||
func (l *acceptedCheckpointLoader) Normalize(_ string, _ string, _ []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
return NormalizeCheckpoint{}, NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing, "checkpoint missing")
|
||||
}
|
||||
|
||||
func TestRunnerHydratesRequiredNormalizedArtifact(t *testing.T) {
|
||||
value := codecNotes{Items: []string{"canonical producer value"}}
|
||||
input, _, _ := handoffFixture(t, value)
|
||||
prepared := input.Prepared
|
||||
producer := &prepared.Steps[0].lanes[0]
|
||||
consumer := &prepared.Steps[1].lanes[0]
|
||||
doc := prepared.input.(*typedTestInput).doc
|
||||
stored, err := checkpointArtifact(producer.typed.codec, producer.resolved.ID, producer.resolved.Normalize.Module, doc.ID, value)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
operationCalls := 0
|
||||
producer.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
operationCalls++
|
||||
return erasedTypedResult{Value: value}, nil
|
||||
}
|
||||
producer.typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
operationCalls++
|
||||
return erasedTypedResult{Value: value}, nil
|
||||
}
|
||||
producer.typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
operationCalls++
|
||||
return erasedTypedResult{Value: value}, nil
|
||||
}
|
||||
validatorCalls := 0
|
||||
validator := preparedValidator{typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
validatorCalls++
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}}
|
||||
producer.extractValidators.validators = []preparedValidator{validator}
|
||||
producer.mergeValidators.validators = []preparedValidator{validator}
|
||||
producer.normalizeValidators.validators = []preparedValidator{validator}
|
||||
|
||||
var received contracts.ReferenceSet
|
||||
consumer.typed.extract = func(_ context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
received = CloneReferenceSet(request.References)
|
||||
return erasedTypedResult{Value: codecScore{Value: 3}}, nil
|
||||
}
|
||||
loader := newAcceptedCheckpointLoader()
|
||||
producerKey := CheckpointLaneKey(producer.resolved.StepID, producer.resolved.ID)
|
||||
consumerKey := CheckpointLaneKey(consumer.resolved.StepID, consumer.resolved.ID)
|
||||
loader.accepted[producerKey] = NormalizeCheckpoint{Output: stored, Warnings: []contracts.Warning{{Scope: "normalize", ReasonCode: "stored-warning", Message: "stored normalize warning"}}}
|
||||
loader.acceptedDecision[producerKey] = NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted artifact reusable")
|
||||
policy := CheckpointExecutionPolicy{
|
||||
RequireReusableLanes: map[string]struct{}{producerKey: {}},
|
||||
ForcedLanes: map[string]struct{}{consumerKey: {}},
|
||||
}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: loader, CheckpointPolicy: policy})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if operationCalls != 0 || validatorCalls != 0 {
|
||||
t.Fatalf("hydrated producer calls = operations %d validators %d, want zero", operationCalls, validatorCalls)
|
||||
}
|
||||
item := received.Slots["producer-output"].Items[0]
|
||||
if string(item.Content) != string(stored.Artifact.Content) || item.Producer.StepID != producer.resolved.StepID || item.Producer.LaneID != producer.resolved.ID {
|
||||
t.Fatalf("consumer generated reference = %#v, want exact hydrated producer bytes and identity", item)
|
||||
}
|
||||
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != "stored-warning" {
|
||||
t.Fatalf("hydrated warnings = %#v, want normalize checkpoint warnings only", output.Warnings)
|
||||
}
|
||||
assertAcceptedNormalizeEvent(t, output.CheckpointEvents, producer.resolved.StepID, producer.resolved.ID, CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused)
|
||||
for _, event := range output.CheckpointEvents {
|
||||
if event.StepID == producer.resolved.StepID && event.LaneID == producer.resolved.ID && event.Stage != string(StageNormalize) {
|
||||
t.Fatalf("hydrated producer synthesized checkpoint event: %#v", event)
|
||||
}
|
||||
}
|
||||
|
||||
freshInput, _, _ := handoffFixture(t, value)
|
||||
freshPrepared := freshInput.Prepared
|
||||
freshPrepared.Steps[0].lanes[0].typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: value}, nil
|
||||
}
|
||||
freshPrepared.Steps[0].lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: value}, nil
|
||||
}
|
||||
freshPrepared.Steps[1].lanes[0].typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: codecScore{Value: 3}}, nil
|
||||
}
|
||||
freshLoader := &handoffDependencyLoader{CheckpointLoader: NoopCheckpointLoader()}
|
||||
freshOutput, err := New().Run(context.Background(), RunInput{Prepared: freshPrepared, RawInput: []byte("input"), Checkpoint: freshLoader})
|
||||
if err != nil {
|
||||
t.Fatalf("fresh Run() error = %v", err)
|
||||
}
|
||||
hydratedDependencies := loader.extractDeps[consumer.resolved.ID]
|
||||
if generatedFingerprintCount(hydratedDependencies) != 1 {
|
||||
t.Fatalf("hydrated consumer dependencies = %#v, want generated producer fingerprint", hydratedDependencies)
|
||||
}
|
||||
var matchedFreshDependencies bool
|
||||
for _, dependencies := range freshLoader.extract {
|
||||
if reflect.DeepEqual(dependencies, hydratedDependencies) {
|
||||
matchedFreshDependencies = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !matchedFreshDependencies {
|
||||
t.Fatalf("consumer dependencies differ: fresh %#v hydrated %#v", freshLoader.extract, hydratedDependencies)
|
||||
}
|
||||
if !reflect.DeepEqual(freshOutput.Manifest.References, output.Manifest.References) {
|
||||
t.Fatalf("generated provenance differs: fresh %#v hydrated %#v", freshOutput.Manifest.References, output.Manifest.References)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerRejectsInvalidRequiredNormalizedArtifactBeforeConsumer(t *testing.T) {
|
||||
const contentSentinel = "sensitive-campaign-payload-74291"
|
||||
tests := []struct {
|
||||
name string
|
||||
decision CheckpointDecision
|
||||
mutate func(*CheckpointArtifact)
|
||||
wantCode CheckpointReasonCode
|
||||
}{
|
||||
{"missing", NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing, "checkpoint missing"), nil, CheckpointReasonMissing},
|
||||
{"rejected status", NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonStatusNotReusable, "status rejected"), nil, CheckpointReasonStatusNotReusable},
|
||||
{"corrupt payload", NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted artifact reusable"), func(v *CheckpointArtifact) { v.Artifact.Content = []byte(`{"items":[`) }, CheckpointReasonArtifactPayloadInvalid},
|
||||
{"non canonical", NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted artifact reusable"), func(v *CheckpointArtifact) { v.Artifact.Content = []byte(`{"items": ["stored"]}`) }, CheckpointReasonArtifactNotCanonical},
|
||||
{"wrong codec identity", NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted artifact reusable"), func(v *CheckpointArtifact) { v.Artifact.Kind = "test/score" }, CheckpointReasonArtifactCodecIncompatible},
|
||||
{"wrong content digest", NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonArtifactDigestMismatch, "content digest mismatch"), nil, CheckpointReasonArtifactDigestMismatch},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
input, _, _ := handoffFixture(t, codecNotes{Items: []string{contentSentinel}})
|
||||
prepared := input.Prepared
|
||||
producer := &prepared.Steps[0].lanes[0]
|
||||
consumer := &prepared.Steps[1].lanes[0]
|
||||
doc := prepared.input.(*typedTestInput).doc
|
||||
stored, err := checkpointArtifact(producer.typed.codec, producer.resolved.ID, producer.resolved.Normalize.Module, doc.ID, codecNotes{Items: []string{contentSentinel}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if test.mutate != nil {
|
||||
test.mutate(&stored)
|
||||
}
|
||||
consumerCalls := 0
|
||||
consumer.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
consumerCalls++
|
||||
return erasedTypedResult{Value: codecScore{Value: 1}}, nil
|
||||
}
|
||||
loader := newAcceptedCheckpointLoader()
|
||||
producerKey := CheckpointLaneKey(producer.resolved.StepID, producer.resolved.ID)
|
||||
loader.accepted[producerKey] = NormalizeCheckpoint{Output: stored}
|
||||
loader.acceptedDecision[producerKey] = test.decision
|
||||
policy := CheckpointExecutionPolicy{RequireReusableLanes: map[string]struct{}{producerKey: {}}}
|
||||
output, runErr := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: loader, CheckpointPolicy: policy})
|
||||
if runErr == nil || !strings.Contains(runErr.Error(), string(test.wantCode)) || consumerCalls != 0 {
|
||||
t.Fatalf("Run() error = %v consumer calls = %d, want %q before consumer", runErr, consumerCalls, test.wantCode)
|
||||
}
|
||||
assertAcceptedNormalizeEvent(t, output.CheckpointEvents, producer.resolved.StepID, producer.resolved.ID, CheckpointDecisionExecuted, test.wantCode)
|
||||
encoded, err := json.Marshal(struct {
|
||||
Manifest any
|
||||
Events any
|
||||
Error string
|
||||
}{output.Manifest, output.CheckpointEvents, runErr.Error()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(string(encoded), contentSentinel) || strings.Contains(string(encoded), string(stored.Artifact.Content)) {
|
||||
t.Fatalf("failed hydration diagnostics leaked artifact content: %s", encoded)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestForcedRequiredLaneExecutesInsteadOfHydrating(t *testing.T) {
|
||||
prepared := preparedOrderedPipeline(t, 1,
|
||||
orderedLaneSpec{id: "unrelated", profile: "score"},
|
||||
orderedLaneSpec{id: "producer", profile: "notes"},
|
||||
orderedLaneSpec{id: "consumer", profile: "score"},
|
||||
)
|
||||
unrelated := &prepared.Steps[0].lanes[0]
|
||||
producer := &prepared.Steps[1].lanes[0]
|
||||
consumer := &prepared.Steps[2].lanes[0]
|
||||
installGeneratedReferenceTarget(&consumer.resolved.ExtractReferences, StageExtract, consumer, "step-2", "producer")
|
||||
|
||||
doc := prepared.input.(*typedTestInput).doc
|
||||
unrelatedArtifact, err := checkpointArtifact(unrelated.typed.codec, unrelated.resolved.ID, unrelated.resolved.Normalize.Module, doc.ID, codecScore{Value: 9})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
producerArtifact, err := checkpointArtifact(producer.typed.codec, producer.resolved.ID, producer.resolved.Normalize.Module, doc.ID, codecNotes{Items: []string{"stale"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
unrelatedCalls, producerCalls := 0, 0
|
||||
unrelated.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
unrelatedCalls++
|
||||
return erasedTypedResult{Value: codecScore{Value: 9}}, nil
|
||||
}
|
||||
producer.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
producerCalls++
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"fresh"}}}, nil
|
||||
}
|
||||
producer.typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"fresh"}}}, nil
|
||||
}
|
||||
var consumerReferences contracts.ReferenceSet
|
||||
consumer.typed.extract = func(_ context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
consumerReferences = CloneReferenceSet(request.References)
|
||||
return erasedTypedResult{Value: codecScore{Value: 1}}, nil
|
||||
}
|
||||
loader := newAcceptedCheckpointLoader()
|
||||
unrelatedKey := CheckpointLaneKey(unrelated.resolved.StepID, unrelated.resolved.ID)
|
||||
producerKey := CheckpointLaneKey(producer.resolved.StepID, producer.resolved.ID)
|
||||
consumerKey := CheckpointLaneKey(consumer.resolved.StepID, consumer.resolved.ID)
|
||||
loader.accepted[unrelatedKey] = NormalizeCheckpoint{Output: unrelatedArtifact}
|
||||
loader.acceptedDecision[unrelatedKey] = NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted artifact reusable")
|
||||
loader.accepted[producerKey] = NormalizeCheckpoint{Output: producerArtifact}
|
||||
loader.acceptedDecision[producerKey] = NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted artifact reusable")
|
||||
policy := CheckpointExecutionPolicy{
|
||||
ForcedLanes: map[string]struct{}{producerKey: {}, consumerKey: {}},
|
||||
RequireReusableLanes: map[string]struct{}{unrelatedKey: {}, producerKey: {}},
|
||||
}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: loader, CheckpointPolicy: policy})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if producerCalls == 0 || unrelatedCalls != 0 {
|
||||
t.Fatalf("calls producer=%d unrelated=%d, want forced producer execution and unrelated hydration", producerCalls, unrelatedCalls)
|
||||
}
|
||||
if got := string(consumerReferences.Slots["producer-output"].Items[0].Content); !strings.Contains(got, "fresh") || strings.Contains(got, "stale") {
|
||||
t.Fatalf("forced producer reference = %q, want freshly executed output", got)
|
||||
}
|
||||
assertAcceptedNormalizeEvent(t, output.CheckpointEvents, unrelated.resolved.StepID, unrelated.resolved.ID, CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused)
|
||||
assertAcceptedNormalizeEvent(t, output.CheckpointEvents, producer.resolved.StepID, producer.resolved.ID, CheckpointDecisionForcedRecompute, CheckpointReasonRecomputeStep)
|
||||
}
|
||||
|
||||
func installGeneratedReferenceTarget(target *ResolvedReferenceTarget, stage ModuleStage, consumer *preparedLaneExecutor, producerStep, producerLane string) {
|
||||
*target = ResolvedReferenceTarget{
|
||||
Stage: stage,
|
||||
StepID: consumer.resolved.StepID,
|
||||
LaneID: consumer.resolved.ID,
|
||||
Module: consumer.resolved.Extract.Module,
|
||||
Bindings: []ReferenceBinding{{
|
||||
Stage: stage,
|
||||
LaneID: consumer.resolved.ID,
|
||||
SlotName: "producer-output",
|
||||
Artifact: &ArtifactReference{Step: producerStep, Lane: producerLane},
|
||||
}},
|
||||
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||
"producer-output": {Slot: contracts.ReferenceSlot{Name: "producer-output", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
|
||||
}},
|
||||
}
|
||||
}
|
||||
|
||||
func assertAcceptedNormalizeEvent(t *testing.T, events []CheckpointEvent, stepID, laneID string, category CheckpointDecisionCategory, code CheckpointReasonCode) {
|
||||
t.Helper()
|
||||
for _, event := range events {
|
||||
if event.Stage == string(StageNormalize) && event.StepID == stepID && event.LaneID == laneID {
|
||||
if event.Category != category || event.ReasonCode != code {
|
||||
t.Fatalf("accepted normalize event = %#v, want %q/%q", event, category, code)
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Fatalf("accepted normalize event missing from %#v", events)
|
||||
}
|
||||
@@ -109,11 +109,11 @@ func (attemptDebugLLM) CompleteStructured(_ context.Context, request contracts.S
|
||||
func preparedAttemptDebugPipeline(t *testing.T) *PreparedPipeline {
|
||||
t.Helper()
|
||||
prepared := preparedConcurrentPipeline(t, 1)
|
||||
prepared.lanes = prepared.lanes[:1]
|
||||
prepared.resolved.ArtifactLanes = prepared.resolved.ArtifactLanes[:1]
|
||||
prepared.ArtifactLanes = prepared.ArtifactLanes[:1]
|
||||
prepared.lanes[0].mergeValidators = preparedValidatorChain{}
|
||||
prepared.lanes[0].normalizeValidators = preparedValidatorChain{}
|
||||
prepared.Steps[0].lanes = prepared.Steps[0].lanes[:1]
|
||||
prepared.resolved.Steps[0].ArtifactLanes = prepared.resolved.Steps[0].ArtifactLanes[:1]
|
||||
prepared.Steps[0].ArtifactLanes = prepared.Steps[0].ArtifactLanes[:1]
|
||||
prepared.Steps[0].lanes[0].mergeValidators = preparedValidatorChain{}
|
||||
prepared.Steps[0].lanes[0].normalizeValidators = preparedValidatorChain{}
|
||||
return prepared
|
||||
}
|
||||
|
||||
@@ -126,13 +126,13 @@ func TestRunnerWritesAttemptScopedMergeAndNormalizeDebug(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
debug := newCapturedDebugRecorder()
|
||||
client := WithDebugLLMRecording(attemptDebugLLM{}, debug)
|
||||
prepared.lanes[0].typed.merge = func(ctx context.Context, _ any, _ contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
prepared.Steps[0].lanes[0].typed.merge = func(ctx context.Context, _ any, _ contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
if err := callAttemptDebugLLM(ctx, client, "merge"); err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"merged"}}, Warnings: []contracts.Warning{{Scope: "merge", ReasonCode: "observed", Message: "merge warning"}}}, nil
|
||||
}
|
||||
prepared.lanes[0].typed.normalize = func(ctx context.Context, _ any, _ contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
prepared.Steps[0].lanes[0].typed.normalize = func(ctx context.Context, _ any, _ contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
if err := callAttemptDebugLLM(ctx, client, "normalize"); err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
@@ -187,7 +187,7 @@ func TestRunnerRecordsDistinctRetryAttemptsAndPromotesAcceptedWarningsOnly(t *te
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
debug := newCapturedDebugRecorder()
|
||||
client := WithDebugLLMRecording(attemptDebugLLM{}, debug)
|
||||
lane := &prepared.lanes[0]
|
||||
lane := &prepared.Steps[0].lanes[0]
|
||||
attempts := 0
|
||||
operation := func(ctx context.Context) (erasedTypedResult, error) {
|
||||
attempts++
|
||||
@@ -257,7 +257,7 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
|
||||
{
|
||||
name: "merge module error",
|
||||
configure: func(prepared *PreparedPipeline) {
|
||||
prepared.lanes[0].typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
prepared.Steps[0].lanes[0].typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{}, errors.New("merge exploded")
|
||||
}
|
||||
},
|
||||
@@ -267,7 +267,7 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
|
||||
{
|
||||
name: "normalize validator error",
|
||||
configure: func(prepared *PreparedPipeline) {
|
||||
prepared.lanes[0].normalizeValidators.validators = []preparedValidator{{
|
||||
prepared.Steps[0].lanes[0].normalizeValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("error-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{}, errors.New("validator exploded")
|
||||
@@ -281,7 +281,7 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
|
||||
{
|
||||
name: "merge final rejection",
|
||||
configure: func(prepared *PreparedPipeline) {
|
||||
prepared.lanes[0].mergeValidators.validators = []preparedValidator{{
|
||||
prepared.Steps[0].lanes[0].mergeValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("reject-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted"}, nil
|
||||
@@ -294,7 +294,7 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
|
||||
{
|
||||
name: "normalize serialization error",
|
||||
configure: func(prepared *PreparedPipeline) {
|
||||
prepared.lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
prepared.Steps[0].lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: "wrong artifact type"}, nil
|
||||
}
|
||||
},
|
||||
@@ -331,13 +331,13 @@ func TestRunnerKeepsValidatorLLMCallsOutOfModuleAttempt(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
debug := newCapturedDebugRecorder()
|
||||
client := WithDebugLLMRecording(attemptDebugLLM{}, debug)
|
||||
prepared.lanes[0].typed.merge = func(ctx context.Context, _ any, _ contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
prepared.Steps[0].lanes[0].typed.merge = func(ctx context.Context, _ any, _ contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
if err := callAttemptDebugLLM(ctx, client, "merge-module"); err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"merged"}}}, nil
|
||||
}
|
||||
prepared.lanes[0].mergeValidators.validators = []preparedValidator{{
|
||||
prepared.Steps[0].lanes[0].mergeValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("llm-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(ctx context.Context, _ any, _ typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
if err := callAttemptDebugLLM(ctx, client, "merge-validator"); err != nil {
|
||||
@@ -389,7 +389,7 @@ func (l attemptReuseLoader) Normalize(laneID, _ string, _ []CheckpointFingerprin
|
||||
|
||||
func TestRunnerCheckpointReuseDoesNotSynthesizeModuleAttempts(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
lane := prepared.lanes[0]
|
||||
lane := prepared.Steps[0].lanes[0]
|
||||
merge, err := checkpointArtifact(lane.typed.codec, lane.resolved.ID, lane.resolved.Merge.Module, "source", codecNotes{Items: []string{"merged"}})
|
||||
if err != nil {
|
||||
t.Fatalf("checkpointArtifact(merge): %v", err)
|
||||
|
||||
@@ -104,11 +104,11 @@ func installObservedNotesCodec(t *testing.T, prepared *PreparedPipeline, codec *
|
||||
if err != nil {
|
||||
t.Fatalf("codec entry error = %v", err)
|
||||
}
|
||||
prepared.lanes[0].typed.codec = entry
|
||||
prepared.Steps[0].lanes[0].typed.codec = entry
|
||||
}
|
||||
|
||||
func configureCandidateOperation(prepared *PreparedPipeline, target ModuleStage, value codecNotes) {
|
||||
lane := &prepared.lanes[0]
|
||||
lane := &prepared.Steps[0].lanes[0]
|
||||
switch target {
|
||||
case StageMerge:
|
||||
lane.typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
@@ -136,9 +136,9 @@ func setCandidateValidator(prepared *PreparedPipeline, target ModuleStage, appro
|
||||
}
|
||||
switch target {
|
||||
case StageMerge:
|
||||
prepared.lanes[0].mergeValidators.validators = []preparedValidator{validator}
|
||||
prepared.Steps[0].lanes[0].mergeValidators.validators = []preparedValidator{validator}
|
||||
case StageNormalize:
|
||||
prepared.lanes[0].normalizeValidators.validators = []preparedValidator{validator}
|
||||
prepared.Steps[0].lanes[0].normalizeValidators.validators = []preparedValidator{validator}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -469,9 +469,9 @@ func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
|
||||
func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) {
|
||||
doc := typedTestDocumentWithUnits(2)
|
||||
prepared := preparedConcurrentPipeline(t, 1)
|
||||
prepared.lanes = prepared.lanes[:1]
|
||||
prepared.resolved.ArtifactLanes = prepared.resolved.ArtifactLanes[:1]
|
||||
prepared.ArtifactLanes = prepared.ArtifactLanes[:1]
|
||||
prepared.Steps[0].lanes = prepared.Steps[0].lanes[:1]
|
||||
prepared.resolved.Steps[0].ArtifactLanes = prepared.resolved.Steps[0].ArtifactLanes[:1]
|
||||
prepared.Steps[0].ArtifactLanes = prepared.Steps[0].ArtifactLanes[:1]
|
||||
prepared.input = &typedTestInput{key: prepared.resolved.Input.Module, doc: doc}
|
||||
|
||||
run := func(plan source.ChunkPlan) []CheckpointFingerprint {
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
)
|
||||
@@ -60,8 +61,184 @@ func preparedConcurrentPipeline(t *testing.T, chunkCount int) *PreparedPipeline
|
||||
return prepared
|
||||
}
|
||||
|
||||
type orderedLaneSpec struct {
|
||||
id string
|
||||
profile string
|
||||
}
|
||||
|
||||
func preparedOrderedPipeline(t *testing.T, chunkCount int, specs ...orderedLaneSpec) *PreparedPipeline {
|
||||
t.Helper()
|
||||
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
|
||||
profile := typedResolutionProfile()
|
||||
profile.Artifacts = nil
|
||||
for index, spec := range specs {
|
||||
lane, ok := typedResolutionProfile().Artifacts[spec.profile]
|
||||
if !ok {
|
||||
t.Fatalf("typed lane profile %q is not defined", spec.profile)
|
||||
}
|
||||
profile.Steps = append(profile.Steps, PipelineStepProfile{
|
||||
ID: fmt.Sprintf("step-%d", index+1),
|
||||
Artifacts: map[string]ArtifactLaneProfile{spec.id: lane},
|
||||
})
|
||||
}
|
||||
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v", err)
|
||||
}
|
||||
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v", err)
|
||||
}
|
||||
doc := typedTestDocumentWithUnits(chunkCount)
|
||||
if adapter, ok := prepared.input.(*typedTestInput); ok {
|
||||
adapter.doc = doc
|
||||
} else {
|
||||
t.Fatalf("prepared input = %T, want *typedTestInput", prepared.input)
|
||||
}
|
||||
prepared.chunker = &typedTestChunker{key: "typed/chunk", plan: typedTestPlan(doc)}
|
||||
return prepared
|
||||
}
|
||||
|
||||
type countingOrderedOutput struct {
|
||||
calls atomic.Int32
|
||||
}
|
||||
|
||||
func (o *countingOrderedOutput) Key() string { return "typed/output" }
|
||||
func (o *countingOrderedOutput) Encode(context.Context, contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||
o.calls.Add(1)
|
||||
return contracts.OutputResult{}, nil
|
||||
}
|
||||
|
||||
type countingInputAdapter struct {
|
||||
contracts.InputAdapter
|
||||
calls atomic.Int32
|
||||
}
|
||||
|
||||
func (a *countingInputAdapter) Parse(ctx context.Context, request contracts.ParseRequest) (*source.SourceDocument, error) {
|
||||
a.calls.Add(1)
|
||||
return a.InputAdapter.Parse(ctx, request)
|
||||
}
|
||||
|
||||
func TestRunnerFailsGeneratedHandoffBeforeConsumerExtraction(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 1)
|
||||
input := &countingInputAdapter{InputAdapter: prepared.input}
|
||||
prepared.input = input
|
||||
preparedLane := &prepared.Steps[0].lanes[0]
|
||||
lane := &preparedLane.resolved
|
||||
binding := ReferenceBinding{
|
||||
Stage: StageExtract,
|
||||
LaneID: lane.ID,
|
||||
SlotName: "generated",
|
||||
Artifact: &ArtifactReference{Step: "producer", Lane: "source"},
|
||||
}
|
||||
lane.ExtractReferences.Bindings = []ReferenceBinding{binding}
|
||||
lane.ExtractReferences.ReferenceSet = contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||
"generated": {Slot: contracts.ReferenceSlot{Name: "generated", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}}},
|
||||
}}
|
||||
prepared.resolved.Steps[0].ArtifactLanes[0].ExtractReferences = CloneReferenceTarget(lane.ExtractReferences)
|
||||
var extractCalls atomic.Int32
|
||||
preparedLane.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
extractCalls.Add(1)
|
||||
return erasedTypedResult{Value: codecNotes{}}, nil
|
||||
}
|
||||
|
||||
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
|
||||
if err == nil || !strings.Contains(err.Error(), "generated dependency") {
|
||||
t.Fatalf("Run() error = %v, want generated dependency failure", err)
|
||||
}
|
||||
if got := input.calls.Load(); got != 1 {
|
||||
t.Fatalf("input Parse calls = %d, want one before handoff", got)
|
||||
}
|
||||
if got := extractCalls.Load(); got != 0 {
|
||||
t.Fatalf("consumer extract calls = %d, want zero", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerExecutesOrderedStepsWithHardBarriers(t *testing.T) {
|
||||
prepared := preparedOrderedPipeline(t, 1,
|
||||
orderedLaneSpec{id: "notes", profile: "notes"},
|
||||
orderedLaneSpec{id: "score", profile: "score"},
|
||||
)
|
||||
var mu sync.Mutex
|
||||
var events []string
|
||||
record := func(event string) {
|
||||
mu.Lock()
|
||||
events = append(events, event)
|
||||
mu.Unlock()
|
||||
}
|
||||
first := &prepared.Steps[0].lanes[0]
|
||||
second := &prepared.Steps[1].lanes[0]
|
||||
first.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"first"}}}, nil
|
||||
}
|
||||
originalFirstNormalize := first.typed.normalize
|
||||
first.typed.normalize = func(ctx context.Context, implementation any, request contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
result, err := originalFirstNormalize(ctx, implementation, request)
|
||||
if err == nil {
|
||||
record("first-normalize-done")
|
||||
}
|
||||
return result, err
|
||||
}
|
||||
second.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
record("second-extract-started")
|
||||
return erasedTypedResult{Value: codecScore{Value: 2}}, nil
|
||||
}
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
mu.Lock()
|
||||
gotEvents := append([]string(nil), events...)
|
||||
mu.Unlock()
|
||||
if want := []string{"first-normalize-done", "second-extract-started"}; !reflect.DeepEqual(gotEvents, want) {
|
||||
t.Fatalf("ordered events = %#v, want %#v", gotEvents, want)
|
||||
}
|
||||
if got := []string{output.NormalizeOutputs[0].LaneID, output.NormalizeOutputs[1].LaneID}; !reflect.DeepEqual(got, []string{"notes", "score"}) {
|
||||
t.Fatalf("normalized lane order = %#v, want notes then score", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerStopsLaterOrderedStepsAfterFailure(t *testing.T) {
|
||||
prepared := preparedOrderedPipeline(t, 1,
|
||||
orderedLaneSpec{id: "first", profile: "notes"},
|
||||
orderedLaneSpec{id: "failure", profile: "score"},
|
||||
orderedLaneSpec{id: "later", profile: "notes"},
|
||||
)
|
||||
first := &prepared.Steps[0].lanes[0]
|
||||
second := &prepared.Steps[1].lanes[0]
|
||||
third := &prepared.Steps[2].lanes[0]
|
||||
first.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"first"}}}, nil
|
||||
}
|
||||
second.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{}, errors.New("ordered step extraction failed")
|
||||
}
|
||||
var thirdCalls atomic.Int32
|
||||
third.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
thirdCalls.Add(1)
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"later"}}}, nil
|
||||
}
|
||||
encoder := &countingOrderedOutput{}
|
||||
prepared.output = encoder
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||
if err == nil || !strings.Contains(err.Error(), `execute pipeline step "step-2"`) {
|
||||
t.Fatalf("Run() error = %v, want step-scoped failure", err)
|
||||
}
|
||||
if len(output.NormalizeOutputs) != 1 || output.NormalizeOutputs[0].LaneID != "first" {
|
||||
t.Fatalf("completed outputs = %#v, want only the first step output", output.NormalizeOutputs)
|
||||
}
|
||||
if got := thirdCalls.Load(); got != 0 {
|
||||
t.Fatalf("later step extract calls = %d, want zero", got)
|
||||
}
|
||||
if got := encoder.calls.Load(); got != 0 {
|
||||
t.Fatalf("output encoder calls = %d, want zero after failure", got)
|
||||
}
|
||||
}
|
||||
|
||||
func installExtractOperation(prepared *PreparedPipeline, laneIndex int, operation func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error)) {
|
||||
prepared.lanes[laneIndex].typed.extract = func(ctx context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
prepared.Steps[0].lanes[laneIndex].typed.extract = func(ctx context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return operation(ctx, request)
|
||||
}
|
||||
}
|
||||
@@ -82,7 +259,7 @@ func TestRunnerBoundsExtractJobsAndStabilizesReverseCompletion(t *testing.T) {
|
||||
}
|
||||
var active atomic.Int32
|
||||
var maximum atomic.Int32
|
||||
for laneIndex := range prepared.lanes {
|
||||
for laneIndex := range prepared.Steps[0].lanes {
|
||||
lane := laneIndex
|
||||
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
jobIndex := request.Chunk.Index*2 + lane
|
||||
@@ -154,8 +331,8 @@ func TestRunnerStartsLaneContinuationWhileOtherLaneExtractsRemain(t *testing.T)
|
||||
return erasedTypedResult{}, ctx.Err()
|
||||
}
|
||||
})
|
||||
originalMerge := prepared.lanes[0].typed.merge
|
||||
prepared.lanes[0].typed.merge = func(ctx context.Context, implementation any, request contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
originalMerge := prepared.Steps[0].lanes[0].typed.merge
|
||||
prepared.Steps[0].lanes[0].typed.merge = func(ctx context.Context, implementation any, request contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
select {
|
||||
case mergeStarted <- struct{}{}:
|
||||
default:
|
||||
@@ -180,7 +357,7 @@ func TestRunnerStartsLaneContinuationWhileOtherLaneExtractsRemain(t *testing.T)
|
||||
|
||||
func TestRunnerBoundsConcurrentLaneContinuations(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 1)
|
||||
base := prepared.lanes[0]
|
||||
base := prepared.Steps[0].lanes[0]
|
||||
lanes := make([]preparedLaneExecutor, 8)
|
||||
resolvedLanes := make([]ResolvedArtifactLane, len(lanes))
|
||||
publicLanes := make([]PreparedArtifactLane, len(lanes))
|
||||
@@ -215,9 +392,9 @@ func TestRunnerBoundsConcurrentLaneContinuations(t *testing.T) {
|
||||
resolvedLanes[i] = lane.resolved
|
||||
publicLanes[i] = PreparedArtifactLane{Resolved: lane.resolved}
|
||||
}
|
||||
prepared.lanes = lanes
|
||||
prepared.resolved.ArtifactLanes = resolvedLanes
|
||||
prepared.ArtifactLanes = publicLanes
|
||||
prepared.Steps[0].lanes = lanes
|
||||
prepared.resolved.Steps[0].ArtifactLanes = resolvedLanes
|
||||
prepared.Steps[0].ArtifactLanes = publicLanes
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||
if err != nil {
|
||||
@@ -235,7 +412,7 @@ func TestRunnerSelectsFrameworkErrorByStableLaneOrder(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 1)
|
||||
ready := make(chan struct{}, 2)
|
||||
release := make(chan struct{})
|
||||
for laneIndex := range prepared.lanes {
|
||||
for laneIndex := range prepared.Steps[0].lanes {
|
||||
lane := laneIndex
|
||||
installExtractOperation(prepared, lane, func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
ready <- struct{}{}
|
||||
@@ -258,7 +435,7 @@ func TestRunnerSelectsFrameworkErrorByStableLaneOrder(t *testing.T) {
|
||||
|
||||
func TestRunnerSelectsMergeErrorBeforeEarlierLaneNormalizeError(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 1)
|
||||
for laneIndex := range prepared.lanes {
|
||||
for laneIndex := range prepared.Steps[0].lanes {
|
||||
lane := laneIndex
|
||||
installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
|
||||
@@ -266,12 +443,12 @@ func TestRunnerSelectsMergeErrorBeforeEarlierLaneNormalizeError(t *testing.T) {
|
||||
}
|
||||
ready := make(chan struct{}, 2)
|
||||
release := make(chan struct{})
|
||||
prepared.lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
prepared.Steps[0].lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
ready <- struct{}{}
|
||||
<-release
|
||||
return erasedTypedResult{}, errors.New("earlier lane normalize failure")
|
||||
}
|
||||
prepared.lanes[1].typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
prepared.Steps[0].lanes[1].typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
ready <- struct{}{}
|
||||
<-release
|
||||
return erasedTypedResult{}, errors.New("later lane merge failure")
|
||||
@@ -291,13 +468,13 @@ func TestRunnerSelectsMergeErrorBeforeEarlierLaneNormalizeError(t *testing.T) {
|
||||
|
||||
func TestRunnerKeepsSuccessfulChunksWhenAnotherExtractIsRejected(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 2)
|
||||
for laneIndex := range prepared.lanes {
|
||||
for laneIndex := range prepared.Steps[0].lanes {
|
||||
lane := laneIndex
|
||||
installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
|
||||
})
|
||||
}
|
||||
validator := &prepared.lanes[0].extractValidators.validators[0]
|
||||
validator := &prepared.Steps[0].lanes[0].extractValidators.validators[0]
|
||||
validator.typedValidate = func(_ context.Context, _ any, target typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
if target.chunk != nil && target.chunk.Index == 0 {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "expected_rejection", Message: "rejected by test"}, nil
|
||||
@@ -308,7 +485,7 @@ func TestRunnerKeepsSuccessfulChunksWhenAnotherExtractIsRejected(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(output.Rejected) != 1 || output.Rejected[0].LaneID != prepared.lanes[0].resolved.ID || output.Rejected[0].ChunkIndex != 0 {
|
||||
if len(output.Rejected) != 1 || output.Rejected[0].LaneID != prepared.Steps[0].lanes[0].resolved.ID || output.Rejected[0].ChunkIndex != 0 {
|
||||
t.Fatalf("rejections = %#v, want the first lane's first chunk", output.Rejected)
|
||||
}
|
||||
if len(output.NormalizeOutputs) != 2 {
|
||||
@@ -330,9 +507,9 @@ func TestRunnerSeparatelyBoundsExtractJobsAndSharedProviderCalls(t *testing.T) {
|
||||
var jobActive atomic.Int32
|
||||
var jobMaximum atomic.Int32
|
||||
var attempts sync.Map
|
||||
for laneIndex := range prepared.lanes {
|
||||
for laneIndex := range prepared.Steps[0].lanes {
|
||||
lane := laneIndex
|
||||
prepared.lanes[lane].resolved.Extract.Retries = 1
|
||||
prepared.Steps[0].lanes[lane].resolved.Extract.Retries = 1
|
||||
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
current := jobActive.Add(1)
|
||||
defer jobActive.Add(-1)
|
||||
@@ -353,7 +530,7 @@ func TestRunnerSeparatelyBoundsExtractJobsAndSharedProviderCalls(t *testing.T) {
|
||||
}
|
||||
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
|
||||
})
|
||||
validator := &prepared.lanes[lane].extractValidators.validators[0]
|
||||
validator := &prepared.Steps[0].lanes[lane].extractValidators.validators[0]
|
||||
validator.typedValidate = func(ctx context.Context, _ any, _ typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
if _, callErr := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{StageName: "validate"}, nil); callErr != nil {
|
||||
return contracts.ValidationResult{}, callErr
|
||||
@@ -382,7 +559,7 @@ func TestRunnerSeparatelyBoundsExtractJobsAndSharedProviderCalls(t *testing.T) {
|
||||
func TestRunnerReturnsParentCancellationAndStopsQueuedExtracts(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 4)
|
||||
started := make(chan struct{}, 8)
|
||||
for laneIndex := range prepared.lanes {
|
||||
for laneIndex := range prepared.Steps[0].lanes {
|
||||
lane := laneIndex
|
||||
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
started <- struct{}{}
|
||||
|
||||
@@ -26,6 +26,8 @@ type laneExtractState struct {
|
||||
results map[int]extractJobResult
|
||||
remaining int
|
||||
failed bool
|
||||
terminal bool
|
||||
output RunOutput
|
||||
}
|
||||
|
||||
type finalizedExtractResults struct {
|
||||
@@ -36,12 +38,15 @@ type finalizedExtractResults struct {
|
||||
decision CheckpointDecision
|
||||
}
|
||||
|
||||
func loadExtract(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||
func loadExtract(loader CheckpointLoader, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||
if stepAware, ok := loader.(StepCheckpointLoader); ok {
|
||||
return stepAware.ExtractForStep(stepID, laneID, moduleKey, deps)
|
||||
}
|
||||
return loader.Extract(laneID, moduleKey, deps)
|
||||
}
|
||||
|
||||
func recordExtract(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||
return recorder.ExtractSucceeded(laneID, moduleKey, deps, outputs, rejected, warnings)
|
||||
func recordExtract(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||
return checkpointExtractSucceeded(recorder, stepID, laneID, moduleKey, deps, outputs, rejected, warnings)
|
||||
}
|
||||
|
||||
type extractJob struct {
|
||||
@@ -72,30 +77,56 @@ type orderedRunError struct {
|
||||
err error
|
||||
}
|
||||
|
||||
func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk) (RunOutput, error) {
|
||||
func (r *Runner) runLanes(parent context.Context, input RunInput, step PreparedPipelineStep, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk) (RunOutput, error) {
|
||||
output := RunOutput{Manifest: manifestFromPipeline(input)}
|
||||
states := make([]*laneExtractState, len(input.Prepared.lanes))
|
||||
for i, prepared := range input.Prepared.lanes {
|
||||
states, err := initializeLaneStates(input, step, checkpoints, loader, doc, chunks, &output)
|
||||
if err != nil {
|
||||
return output, err
|
||||
}
|
||||
completedOutputs, runErrors := r.runLaneEngine(parent, input, checkpoints, loader, doc, sourceInput, sessionID, chunks, states)
|
||||
if err := mergeCompletedLanes(&output, completedOutputs); err != nil {
|
||||
return output, err
|
||||
}
|
||||
if err := selectRunError(parent, runErrors); err != nil {
|
||||
return output, err
|
||||
}
|
||||
return output, nil
|
||||
}
|
||||
|
||||
func initializeLaneStates(input RunInput, step PreparedPipelineStep, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, chunks []source.Chunk, output *RunOutput) ([]*laneExtractState, error) {
|
||||
states := make([]*laneExtractState, len(step.lanes))
|
||||
for i, prepared := range step.lanes {
|
||||
if prepared.typed == nil {
|
||||
return output, fmt.Errorf("typed lane %q executor is not prepared", prepared.resolved.ID)
|
||||
return nil, fmt.Errorf("typed lane %q executor is not prepared", prepared.resolved.ID)
|
||||
}
|
||||
if err := setTypedLaneManifestMetadata(&output, prepared.resolved.ID, prepared.typed.extractor, prepared.typed.merger, prepared.typed.normalizer); err != nil {
|
||||
return output, err
|
||||
if err := setTypedLaneManifestMetadata(output, prepared.resolved.ID, prepared.typed.extractor, prepared.typed.merger, prepared.typed.normalizer); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
state, err := prepareLaneExtract(input, loader, doc, chunks, i, prepared)
|
||||
if input.CheckpointPolicy.requiresReusable(input.stepID, prepared.resolved.ID) && !input.CheckpointPolicy.forced(input.stepID, prepared.resolved.ID) {
|
||||
state, err := hydrateRequiredLane(input, loader, doc, i, prepared, output)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
states[i] = state
|
||||
continue
|
||||
}
|
||||
state, err := prepareLaneExtract(input, loader, doc, chunks, i, prepared, output)
|
||||
if err != nil {
|
||||
return output, err
|
||||
return nil, err
|
||||
}
|
||||
if !state.decision.Reused {
|
||||
if err := checkpoints.ExtractRunning(prepared.resolved.ID, prepared.resolved.Extract.Module, state.deps); err != nil {
|
||||
return output, fmt.Errorf("write extract checkpoint for lane %q: %w", prepared.resolved.ID, err)
|
||||
if err := checkpointExtractRunning(checkpoints, input.stepID, prepared.resolved.ID, prepared.resolved.Extract.Module, state.deps); err != nil {
|
||||
return nil, fmt.Errorf("write extract checkpoint for lane %q: %w", prepared.resolved.ID, err)
|
||||
}
|
||||
} else if err := finalizeLaneExtract(checkpoints, state); err != nil {
|
||||
return output, err
|
||||
} else if err := finalizeLaneExtract(checkpoints, input.stepID, state); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
states[i] = state
|
||||
}
|
||||
return states, nil
|
||||
}
|
||||
|
||||
func (r *Runner) runLaneEngine(parent context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk, states []*laneExtractState) ([]RunOutput, []orderedRunError) {
|
||||
workerCount := input.ExtractWorkers
|
||||
if workerCount < 1 {
|
||||
workerCount = 1
|
||||
@@ -126,7 +157,7 @@ func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints Ch
|
||||
for chunkIndex := range chunks {
|
||||
for laneIndex := range states {
|
||||
state := states[laneIndex]
|
||||
if state.decision.Reused {
|
||||
if state.terminal || state.decision.Reused {
|
||||
continue
|
||||
}
|
||||
select {
|
||||
@@ -154,16 +185,24 @@ func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints Ch
|
||||
}()
|
||||
}
|
||||
|
||||
completedOutputs, runErrors := collectLaneResults(ctx, cancel, input, checkpoints, chunks, states, results, completions, continuations)
|
||||
close(continuations)
|
||||
continuationWorkers.Wait()
|
||||
return completedOutputs, runErrors
|
||||
}
|
||||
|
||||
func collectLaneResults(ctx context.Context, cancel context.CancelFunc, input RunInput, checkpoints CheckpointRecorder, chunks []source.Chunk, states []*laneExtractState, results <-chan extractJobResult, completions <-chan laneCompletion, continuations chan<- *laneExtractState) ([]RunOutput, []orderedRunError) {
|
||||
completedOutputs := make([]RunOutput, len(states))
|
||||
var runErrors []orderedRunError
|
||||
var pendingContinuations []*laneExtractState
|
||||
launched, completed := 0, 0
|
||||
for _, state := range states {
|
||||
if state.decision.Reused {
|
||||
if state.terminal {
|
||||
completedOutputs[state.index] = state.output
|
||||
} else if state.decision.Reused {
|
||||
pendingContinuations = append(pendingContinuations, state)
|
||||
}
|
||||
}
|
||||
|
||||
resultChannel := results
|
||||
for resultChannel != nil || len(pendingContinuations) > 0 || completed < launched {
|
||||
var continuationChannel chan<- *laneExtractState
|
||||
@@ -188,13 +227,13 @@ func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints Ch
|
||||
if result.err != nil {
|
||||
state.failed = true
|
||||
runErrors = append(runErrors, orderedRunError{stage: 0, lane: state.index, chunk: result.chunkIndex, err: result.err})
|
||||
_ = checkpoints.ExtractFailed(state.prepared.resolved.ID, state.prepared.resolved.Extract.Module, state.deps, result.err)
|
||||
_ = checkpointExtractFailed(checkpoints, input.stepID, state.prepared.resolved.ID, state.prepared.resolved.Extract.Module, state.deps, result.err)
|
||||
cancel()
|
||||
} else {
|
||||
state.results[result.chunkIndex] = result
|
||||
}
|
||||
if state.remaining == 0 && !state.failed && ctx.Err() == nil {
|
||||
if err := finalizeLaneExtract(checkpoints, state); err != nil {
|
||||
if err := finalizeLaneExtract(checkpoints, input.stepID, state); err != nil {
|
||||
state.failed = true
|
||||
runErrors = append(runErrors, orderedRunError{stage: 0, lane: state.index, chunk: len(chunks), err: err})
|
||||
cancel()
|
||||
@@ -211,47 +250,73 @@ func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints Ch
|
||||
}
|
||||
}
|
||||
}
|
||||
close(continuations)
|
||||
continuationWorkers.Wait()
|
||||
for i := range completedOutputs {
|
||||
if err := mergeLaneOutput(&output, completedOutputs[i]); err != nil {
|
||||
return output, err
|
||||
}
|
||||
}
|
||||
if err := selectRunError(parent, runErrors); err != nil {
|
||||
return output, err
|
||||
}
|
||||
return output, nil
|
||||
return completedOutputs, runErrors
|
||||
}
|
||||
|
||||
func prepareLaneExtract(input RunInput, loader CheckpointLoader, doc *source.SourceDocument, chunks []source.Chunk, index int, prepared preparedLaneExecutor) (*laneExtractState, error) {
|
||||
func hydrateRequiredLane(input RunInput, loader CheckpointLoader, doc *source.SourceDocument, index int, prepared preparedLaneExecutor, output *RunOutput) (*laneExtractState, error) {
|
||||
lane, typed := prepared.resolved, prepared.typed
|
||||
local := RunOutput{Manifest: manifestFromPipeline(input)}
|
||||
checkpoint, decision := loader.AcceptedNormalize(input.stepID, lane.ID, lane.Normalize.Module)
|
||||
if decision.Reused {
|
||||
decision = checkpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted normalized artifact is reusable")
|
||||
if checkpoint.Output.LaneID != lane.ID || checkpoint.Output.ModuleKey != lane.Normalize.Module || checkpoint.Output.SourceID != doc.ID {
|
||||
decision = checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonArtifactPayloadInvalid, "accepted normalized artifact provenance does not match the producer lane")
|
||||
}
|
||||
}
|
||||
decision, err := resolveCheckpointDecision(&local, loader, input.CheckpointPolicy, StageNormalize, input.stepID, lane.ID, lane.Normalize.Module, decision, typed.codec, []CheckpointArtifact{checkpoint.Output})
|
||||
if err != nil {
|
||||
if mergeErr := mergeLaneOutput(output, local); mergeErr != nil {
|
||||
return nil, mergeErr
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
_, hydrated, err := decodeCanonicalCheckpointArtifact(typed.codec, checkpoint.Output)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("hydrate accepted normalized artifact for step %q lane %q: %w", input.stepID, lane.ID, err)
|
||||
}
|
||||
local.Warnings = append(local.Warnings, cloneWarnings(checkpoint.Warnings)...)
|
||||
local.NormalizeOutputs = append(local.NormalizeOutputs, contracts.SerializedOutput{
|
||||
StepID: input.stepID,
|
||||
LaneID: lane.ID,
|
||||
NormalizerKey: lane.Normalize.Module,
|
||||
SourceID: doc.ID,
|
||||
Artifact: contracts.CloneSerializedArtifact(hydrated.Artifact),
|
||||
})
|
||||
return &laneExtractState{index: index, prepared: prepared, decision: decision, terminal: true, output: local}, nil
|
||||
}
|
||||
|
||||
func mergeCompletedLanes(output *RunOutput, completedOutputs []RunOutput) error {
|
||||
for i := range completedOutputs {
|
||||
if err := mergeLaneOutput(output, completedOutputs[i]); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func prepareLaneExtract(input RunInput, loader CheckpointLoader, doc *source.SourceDocument, chunks []source.Chunk, index int, prepared preparedLaneExecutor, output *RunOutput) (*laneExtractState, error) {
|
||||
lane, typed := prepared.resolved, prepared.typed
|
||||
digest, err := joinedChunkDigest(chunks)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("digest chunks for lane %q: %w", lane.ID, err)
|
||||
}
|
||||
state := &laneExtractState{index: index, prepared: prepared, deps: digestFingerprints("chunks", digest), remaining: len(chunks), results: make(map[int]extractJobResult, len(chunks))}
|
||||
cp, decision := loadExtract(loader, lane.ID, lane.Extract.Module, state.deps)
|
||||
if decision.Reused {
|
||||
for _, stored := range cp.Outputs {
|
||||
if _, decodeErr := decodeCheckpointArtifact(typed.codec, stored); decodeErr != nil {
|
||||
decision = CheckpointDecision{Reason: "extract artifact checkpoint codec is incompatible: " + decodeErr.Error()}
|
||||
break
|
||||
}
|
||||
}
|
||||
extractReferences := operationReferenceSet(input, lane.ExtractReferences)
|
||||
deps := append(digestFingerprints("chunks", digest), generatedReferenceDependencies(extractReferences)...)
|
||||
state := &laneExtractState{index: index, prepared: prepared, deps: normalizeCheckpointFingerprints(deps), remaining: len(chunks), results: make(map[int]extractJobResult, len(chunks))}
|
||||
cp, decision := loadExtract(loader, input.stepID, lane.ID, lane.Extract.Module, state.deps)
|
||||
decision, err = resolveCheckpointDecision(output, loader, input.CheckpointPolicy, StageExtract, input.stepID, lane.ID, lane.Extract.Module, decision, typed.codec, cp.Outputs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
state.decision = decision
|
||||
if decision.Reused {
|
||||
state.remaining = 0
|
||||
for _, stored := range cp.Outputs {
|
||||
value, decodeErr := decodeCheckpointArtifact(typed.codec, stored)
|
||||
value, hydrated, decodeErr := decodeCanonicalCheckpointArtifact(typed.codec, stored)
|
||||
if decodeErr != nil {
|
||||
return nil, fmt.Errorf("decode extract checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||
}
|
||||
stored, decodeErr = hydrateCheckpointArtifact(typed.codec, stored, value)
|
||||
if decodeErr != nil {
|
||||
return nil, fmt.Errorf("hydrate extract checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||
}
|
||||
stored = hydrated
|
||||
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: stored.ChunkID, ChunkIndex: stored.ChunkIndex, ChunkRef: stored.ChunkRef, Value: value}
|
||||
if stored.ChunkIndex >= 0 && stored.ChunkIndex < len(chunks) && artifact.ChunkRef == (source.SourceRef{}) {
|
||||
artifact.ChunkRef = chunks[stored.ChunkIndex].Ref
|
||||
@@ -280,12 +345,13 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
|
||||
started := time.Now().UTC()
|
||||
attemptPath := path.Join("extract", debugPathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), fmt.Sprintf("attempt-%02d", attempt))
|
||||
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "extract", llmScope, debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started})
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "extract", llmScope, debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started})
|
||||
requestMetadata, metadataErr := cloneMetadata(input.Metadata)
|
||||
if metadataErr != nil {
|
||||
return false, nil, terminal.record(nil, fmt.Errorf("clone extract request metadata: %w", metadataErr))
|
||||
}
|
||||
extracted, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(lane.ExtractReferences.ReferenceSet), LLMProfile: lane.Extract.LLMProfile, Metadata: requestMetadata})
|
||||
extractReferences := operationReferenceSet(input, lane.ExtractReferences)
|
||||
extracted, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(extractReferences), LLMProfile: lane.Extract.LLMProfile, Metadata: requestMetadata})
|
||||
if callErr != nil {
|
||||
attemptErr := fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, lane.Extract.Module, callErr)
|
||||
return false, nil, terminal.record(nil, attemptErr)
|
||||
@@ -299,7 +365,7 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
|
||||
return false, nil, terminal.record(payload, attemptErr)
|
||||
}
|
||||
serializedCandidate.ChunkID, serializedCandidate.ChunkIndex, serializedCandidate.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageExtract, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: lane.ExtractReferences.ReferenceSet, metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: extracted.Value, candidate: &serializedCandidate}, state.prepared.extractValidators, attempt, input.Debug)
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageExtract, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: extractReferences, metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: extracted.Value, candidate: &serializedCandidate}, state.prepared.extractValidators, attempt, input.Debug)
|
||||
attemptWarnings = append(attemptWarnings, warnings...)
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
|
||||
if validateErr != nil || rejected != nil {
|
||||
@@ -327,7 +393,7 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
|
||||
return result
|
||||
}
|
||||
|
||||
func finalizeLaneExtract(checkpoints CheckpointRecorder, state *laneExtractState) error {
|
||||
func finalizeLaneExtract(checkpoints CheckpointRecorder, stepID string, state *laneExtractState) error {
|
||||
lane := state.prepared.resolved
|
||||
indexes := make([]int, 0, len(state.results))
|
||||
for index := range state.results {
|
||||
@@ -348,7 +414,7 @@ func finalizeLaneExtract(checkpoints CheckpointRecorder, state *laneExtractState
|
||||
sort.SliceStable(state.serialized, func(i, j int) bool { return state.serialized[i].ChunkIndex < state.serialized[j].ChunkIndex })
|
||||
sort.SliceStable(state.rejected, func(i, j int) bool { return state.rejected[i].ChunkIndex < state.rejected[j].ChunkIndex })
|
||||
if !state.decision.Reused {
|
||||
if err := recordExtract(checkpoints, lane.ID, lane.Extract.Module, state.deps, state.serialized, state.rejected, state.warnings); err != nil {
|
||||
if err := recordExtract(checkpoints, stepID, lane.ID, lane.Extract.Module, state.deps, state.serialized, state.rejected, state.warnings); err != nil {
|
||||
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
|
||||
}
|
||||
}
|
||||
@@ -367,11 +433,10 @@ func (r *Runner) continueLane(ctx context.Context, input RunInput, checkpoints C
|
||||
}
|
||||
local.Warnings = append(local.Warnings, cloneWarnings(results.warnings)...)
|
||||
local.Rejected = append(local.Rejected, cloneRejectedOutputs(results.rejected)...)
|
||||
recordCheckpointEvent(&local, loader, string(StageExtract), lane.ID, lane.Extract.Module, results.decision)
|
||||
if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "decision": results.decision, "source": debugSourceDocumentEnvelope(doc), "chunks": debugSourceChunkEnvelopes(chunks), "options": redactSensitiveMap(lane.Extract.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||
if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "decision": results.decision, "source": debugSourceDocumentEnvelope(doc), "chunks": debugSourceChunkEnvelopes(chunks), "options": redactSensitiveMap(lane.Extract.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||
return local, &laneRunError{stage: StageExtract, err: err}
|
||||
}
|
||||
if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "outputs": debugCheckpointArtifacts(results.serialized), "rejected": debugRejectedOutputEnvelopes(results.rejected), "warnings": debugWarningEnvelopes(results.warnings)}}); err != nil {
|
||||
if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "outputs": debugCheckpointArtifacts(results.serialized), "rejected": debugRejectedOutputEnvelopes(results.rejected), "warnings": debugWarningEnvelopes(results.warnings)}}); err != nil {
|
||||
return local, &laneRunError{stage: StageExtract, err: err}
|
||||
}
|
||||
if len(results.accepted) == 0 {
|
||||
@@ -440,7 +505,7 @@ func mergeLaneOutput(dst *RunOutput, src RunOutput) error {
|
||||
dst.CheckpointEvents = append(dst.CheckpointEvents, src.CheckpointEvents...)
|
||||
for i := range dst.Manifest.ArtifactLanes {
|
||||
for j := range src.Manifest.ArtifactLanes {
|
||||
if dst.Manifest.ArtifactLanes[i].ID == src.Manifest.ArtifactLanes[j].ID && src.Manifest.ArtifactLanes[j].Metadata != nil {
|
||||
if dst.Manifest.ArtifactLanes[i].ID == src.Manifest.ArtifactLanes[j].ID && dst.Manifest.ArtifactLanes[i].StepID == src.Manifest.ArtifactLanes[j].StepID && src.Manifest.ArtifactLanes[j].Metadata != nil {
|
||||
metadata, err := cloneMetadata(src.Manifest.ArtifactLanes[j].Metadata)
|
||||
if err != nil {
|
||||
return fmt.Errorf("clone lane %q manifest metadata: %w", dst.Manifest.ArtifactLanes[i].ID, err)
|
||||
|
||||
@@ -212,7 +212,7 @@ func TestRunnerContinuesFromFreshAndReusedExtractResults(t *testing.T) {
|
||||
|
||||
func TestRunnerPromotesOnlyAcceptedExtractRetryWarnings(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
prepared.lanes[0].resolved.Extract.Retries = 1
|
||||
prepared.Steps[0].lanes[0].resolved.Extract.Retries = 1
|
||||
attempts := 0
|
||||
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
attempts++
|
||||
@@ -226,7 +226,7 @@ func TestRunnerPromotesOnlyAcceptedExtractRetryWarnings(t *testing.T) {
|
||||
}, nil
|
||||
})
|
||||
validatorCalls := 0
|
||||
prepared.lanes[0].extractValidators.validators[0].typedValidate = func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
prepared.Steps[0].lanes[0].extractValidators.validators[0].typedValidate = func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
validatorCalls++
|
||||
return contracts.ValidationResult{Approved: validatorCalls == 2, ReasonCode: "retry", Message: "retry extract"}, nil
|
||||
}
|
||||
@@ -252,7 +252,7 @@ func TestRunnerPromotesOnlyAcceptedExtractRetryWarnings(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func assertExtractDecision(t *testing.T, events []CheckpointEvent, action string, reason string) {
|
||||
func assertExtractDecision(t *testing.T, events []CheckpointEvent, action CheckpointDecisionCategory, reason string) {
|
||||
t.Helper()
|
||||
for _, event := range events {
|
||||
if event.Stage == string(StageExtract) {
|
||||
|
||||
@@ -220,7 +220,7 @@ func TestRunnerRecordsExtractTerminalOutcomes(t *testing.T) {
|
||||
}
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{value}}, Warnings: []contracts.Warning{{Scope: "extract", ReasonCode: "observed", Message: "extract warning"}}}, nil
|
||||
})
|
||||
prepared.lanes[0].extractValidators.validators = []preparedValidator{{
|
||||
prepared.Steps[0].lanes[0].extractValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("terminal/extract-validator"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: !tc.reject, ReasonCode: "extract_rejected", Message: "extract rejected"}, tc.validatorErr
|
||||
@@ -315,7 +315,7 @@ func TestRunnerKeepsExtractModuleAndValidatorLLMCallsIsolated(t *testing.T) {
|
||||
}
|
||||
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
|
||||
})
|
||||
prepared.lanes[0].extractValidators.validators = []preparedValidator{{
|
||||
prepared.Steps[0].lanes[0].extractValidators.validators = []preparedValidator{{
|
||||
resolved: ResolvedValidator{Binding: Binding("llm-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(ctx context.Context, _ any, _ typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
if err := callAttemptDebugLLM(ctx, client, "extract-validator"); err != nil {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
@@ -13,17 +14,23 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func loadMerge(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
|
||||
func loadMerge(loader CheckpointLoader, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
|
||||
if stepAware, ok := loader.(StepCheckpointLoader); ok {
|
||||
return stepAware.MergeForStep(stepID, laneID, moduleKey, deps)
|
||||
}
|
||||
return loader.Merge(laneID, moduleKey, deps)
|
||||
}
|
||||
func loadNormalize(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
func loadNormalize(loader CheckpointLoader, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
if stepAware, ok := loader.(StepCheckpointLoader); ok {
|
||||
return stepAware.NormalizeForStep(stepID, laneID, moduleKey, deps)
|
||||
}
|
||||
return loader.Normalize(laneID, moduleKey, deps)
|
||||
}
|
||||
func recordMerge(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
return recorder.MergeSucceeded(laneID, moduleKey, deps, output, warnings)
|
||||
func recordMerge(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
return checkpointMergeSucceeded(recorder, stepID, laneID, moduleKey, deps, output, warnings)
|
||||
}
|
||||
func recordNormalize(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
return recorder.NormalizeSucceeded(laneID, moduleKey, deps, output, warnings)
|
||||
func recordNormalize(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
return checkpointNormalizeSucceeded(recorder, stepID, laneID, moduleKey, deps, output, warnings)
|
||||
}
|
||||
func cloneCheckpointArtifact(output CheckpointArtifact) CheckpointArtifact {
|
||||
output.Artifact = contracts.CloneSerializedArtifact(output.Artifact)
|
||||
@@ -94,21 +101,64 @@ func serializeArtifact(codec artifactCodecEntry, value any, candidate bool) (con
|
||||
return contracts.SerializedArtifact{Kind: codec.spec.Kind, Schema: contracts.CloneArtifactSchema(schema), MediaType: codec.spec.MediaType, Content: append([]byte(nil), content...), Metadata: metadata}, nil
|
||||
}
|
||||
|
||||
type checkpointArtifactValidationError struct {
|
||||
code CheckpointReasonCode
|
||||
err error
|
||||
}
|
||||
|
||||
func (e *checkpointArtifactValidationError) Error() string { return e.err.Error() }
|
||||
func (e *checkpointArtifactValidationError) Unwrap() error { return e.err }
|
||||
|
||||
func checkpointArtifactValidationFailure(code CheckpointReasonCode, format string, args ...any) error {
|
||||
return &checkpointArtifactValidationError{code: code, err: fmt.Errorf(format, args...)}
|
||||
}
|
||||
|
||||
func checkpointArtifactReasonCode(err error) CheckpointReasonCode {
|
||||
var validationErr *checkpointArtifactValidationError
|
||||
if errors.As(err, &validationErr) {
|
||||
return validationErr.code
|
||||
}
|
||||
return CheckpointReasonArtifactPayloadInvalid
|
||||
}
|
||||
|
||||
func decodeCheckpointArtifact(codec artifactCodecEntry, artifact CheckpointArtifact) (any, error) {
|
||||
expectedDigest := contracts.DigestArtifactSchema(codec.spec.Schema)
|
||||
if artifact.Artifact.Kind != codec.spec.Kind {
|
||||
return nil, fmt.Errorf("artifact kind %q does not match codec %q", artifact.Artifact.Kind, codec.spec.Kind)
|
||||
return nil, checkpointArtifactValidationFailure(CheckpointReasonArtifactCodecIncompatible, "artifact kind %q does not match codec %q", artifact.Artifact.Kind, codec.spec.Kind)
|
||||
}
|
||||
if artifact.Artifact.Schema.ID != codec.spec.Schema.ID || artifact.Artifact.Schema.Version != codec.spec.Schema.Version {
|
||||
return nil, fmt.Errorf("artifact schema %q version %q does not match codec schema %q version %q", artifact.Artifact.Schema.ID, artifact.Artifact.Schema.Version, codec.spec.Schema.ID, codec.spec.Schema.Version)
|
||||
if artifact.Artifact.Schema.ID != codec.spec.Schema.ID || artifact.Artifact.Schema.Name != codec.spec.Schema.Name || artifact.Artifact.Schema.Version != codec.spec.Schema.Version {
|
||||
return nil, checkpointArtifactValidationFailure(CheckpointReasonArtifactCodecIncompatible, "artifact schema %q version %q does not match codec schema %q version %q", artifact.Artifact.Schema.ID, artifact.Artifact.Schema.Version, codec.spec.Schema.ID, codec.spec.Schema.Version)
|
||||
}
|
||||
if artifact.SchemaDigest != expectedDigest {
|
||||
return nil, fmt.Errorf("artifact schema digest %q does not match codec schema digest %q", artifact.SchemaDigest, expectedDigest)
|
||||
return nil, checkpointArtifactValidationFailure(CheckpointReasonArtifactCodecIncompatible, "artifact schema digest %q does not match codec schema digest %q", artifact.SchemaDigest, expectedDigest)
|
||||
}
|
||||
if artifact.Artifact.MediaType != codec.spec.MediaType {
|
||||
return nil, fmt.Errorf("artifact media type %q does not match codec media type %q", artifact.Artifact.MediaType, codec.spec.MediaType)
|
||||
return nil, checkpointArtifactValidationFailure(CheckpointReasonArtifactCodecIncompatible, "artifact media type %q does not match codec media type %q", artifact.Artifact.MediaType, codec.spec.MediaType)
|
||||
}
|
||||
return codec.decode(append([]byte(nil), artifact.Artifact.Content...))
|
||||
value, err := codec.decode(append([]byte(nil), artifact.Artifact.Content...))
|
||||
if err != nil {
|
||||
return nil, checkpointArtifactValidationFailure(CheckpointReasonArtifactPayloadInvalid, "decode artifact payload: %w", err)
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
|
||||
func decodeCanonicalCheckpointArtifact(codec artifactCodecEntry, artifact CheckpointArtifact) (any, CheckpointArtifact, error) {
|
||||
value, err := decodeCheckpointArtifact(codec, artifact)
|
||||
if err != nil {
|
||||
return nil, CheckpointArtifact{}, err
|
||||
}
|
||||
canonical, err := serializeArtifact(codec, value, false)
|
||||
if err != nil {
|
||||
return nil, CheckpointArtifact{}, checkpointArtifactValidationFailure(CheckpointReasonArtifactPayloadInvalid, "encode canonical artifact: %w", err)
|
||||
}
|
||||
if canonical.Kind != artifact.Artifact.Kind || canonical.Schema.ID != artifact.Artifact.Schema.ID || canonical.Schema.Name != artifact.Artifact.Schema.Name || canonical.Schema.Version != artifact.Artifact.Schema.Version || canonical.MediaType != artifact.Artifact.MediaType || !bytes.Equal(canonical.Content, artifact.Artifact.Content) || checkpointContentDigest(canonical.Content) != checkpointContentDigest(artifact.Artifact.Content) {
|
||||
return nil, CheckpointArtifact{}, checkpointArtifactValidationFailure(CheckpointReasonArtifactNotCanonical, "stored artifact is not canonical")
|
||||
}
|
||||
hydrated, err := hydrateCheckpointArtifact(codec, cloneCheckpointArtifact(artifact), value)
|
||||
if err != nil {
|
||||
return nil, CheckpointArtifact{}, checkpointArtifactValidationFailure(CheckpointReasonArtifactPayloadInvalid, "hydrate checkpoint artifact: %w", err)
|
||||
}
|
||||
return value, hydrated, nil
|
||||
}
|
||||
|
||||
func checkpointArtifact(codec artifactCodecEntry, laneID, moduleKey, sourceID string, value any) (CheckpointArtifact, error) {
|
||||
@@ -135,13 +185,13 @@ type laneRunError struct {
|
||||
func (e *laneRunError) Error() string { return e.err.Error() }
|
||||
func (e *laneRunError) Unwrap() error { return e.err }
|
||||
|
||||
func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, extracts finalizedExtractResults, output *RunOutput) (err error) {
|
||||
activeStage := StageMerge
|
||||
defer func() {
|
||||
if err != nil {
|
||||
err = &laneRunError{stage: activeStage, err: err}
|
||||
}
|
||||
}()
|
||||
type mergeStageResult struct {
|
||||
artifact erasedMergeArtifact
|
||||
serialized CheckpointArtifact
|
||||
terminal bool
|
||||
}
|
||||
|
||||
func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, extracts finalizedExtractResults, output *RunOutput) error {
|
||||
lane, typed := prepared.resolved, prepared.typed
|
||||
if typed == nil {
|
||||
return fmt.Errorf("typed lane %q executor is not prepared", lane.ID)
|
||||
@@ -149,51 +199,67 @@ func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoi
|
||||
if err := setTypedLaneManifestMetadata(output, lane.ID, typed.extractor, typed.merger, typed.normalizer); err != nil {
|
||||
return err
|
||||
}
|
||||
merged, err := r.runMergeStage(ctx, input, checkpoints, loader, doc, sourceInput, sessionID, prepared, extracts, output)
|
||||
if err != nil {
|
||||
return &laneRunError{stage: StageMerge, err: err}
|
||||
}
|
||||
if merged.terminal {
|
||||
return nil
|
||||
}
|
||||
normalized, err := r.runNormalizeStage(ctx, input, checkpoints, loader, doc, sourceInput, sessionID, prepared, merged.artifact, merged.serialized, output)
|
||||
if err != nil {
|
||||
return &laneRunError{stage: StageNormalize, err: err}
|
||||
}
|
||||
if normalized.accepted {
|
||||
output.NormalizeOutputs = append(output.NormalizeOutputs, contracts.SerializedOutput{StepID: input.stepID, LaneID: lane.ID, NormalizerKey: lane.Normalize.Module, SourceID: doc.ID, Artifact: contracts.CloneSerializedArtifact(normalized.serialized.Artifact)})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, extracts finalizedExtractResults, output *RunOutput) (mergeStageResult, error) {
|
||||
var stageResult mergeStageResult
|
||||
lane, typed := prepared.resolved, prepared.typed
|
||||
mergeInputs := make([]contracts.ExtractArtifact[any], len(extracts.accepted))
|
||||
for i, value := range extracts.accepted {
|
||||
mergeInputs[i] = contracts.ExtractArtifact[any]{LaneID: value.LaneID, ExtractorKey: value.ExtractorKey, SourceID: value.SourceID, ChunkID: value.ChunkID, ChunkIndex: value.ChunkIndex, ChunkRef: value.ChunkRef, Value: value.Value}
|
||||
}
|
||||
mergeDeps := artifactCheckpointDigests(extracts.serialized)
|
||||
mergeCP, mergeDecision := loadMerge(loader, lane.ID, lane.Merge.Module, mergeDeps)
|
||||
if mergeDecision.Reused {
|
||||
if _, decodeErr := decodeCheckpointArtifact(typed.codec, mergeCP.Output); decodeErr != nil {
|
||||
mergeDecision = CheckpointDecision{Reason: "merge artifact checkpoint codec is incompatible: " + decodeErr.Error()}
|
||||
}
|
||||
mergeReferences := operationReferenceSet(input, lane.MergeReferences)
|
||||
mergeDeps := append(artifactCheckpointDigests(extracts.serialized), generatedReferenceDependencies(mergeReferences)...)
|
||||
mergeDeps = normalizeCheckpointFingerprints(mergeDeps)
|
||||
mergeCP, mergeDecision := loadMerge(loader, input.stepID, lane.ID, lane.Merge.Module, mergeDeps)
|
||||
mergeDecision, err := resolveCheckpointDecision(output, loader, input.CheckpointPolicy, StageMerge, input.stepID, lane.ID, lane.Merge.Module, mergeDecision, typed.codec, []CheckpointArtifact{mergeCP.Output})
|
||||
if err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
recordCheckpointEvent(output, loader, string(StageMerge), lane.ID, lane.Merge.Module, mergeDecision)
|
||||
if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageMerge), LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "decision": mergeDecision, "source": debugSourceDocumentEnvelope(doc), "extract_outputs": debugCheckpointArtifacts(extracts.serialized), "options": redactSensitiveMap(lane.Merge.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||
return err
|
||||
if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "decision": mergeDecision, "source": debugSourceDocumentEnvelope(doc), "extract_outputs": debugCheckpointArtifacts(extracts.serialized), "options": redactSensitiveMap(lane.Merge.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
var merged erasedMergeArtifact
|
||||
var serializedMerge CheckpointArtifact
|
||||
var mergeWarnings []contracts.Warning
|
||||
if mergeDecision.Reused {
|
||||
value, decodeErr := decodeCheckpointArtifact(typed.codec, mergeCP.Output)
|
||||
value, hydrated, decodeErr := decodeCanonicalCheckpointArtifact(typed.codec, mergeCP.Output)
|
||||
if decodeErr != nil {
|
||||
return fmt.Errorf("decode merge checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||
return stageResult, fmt.Errorf("decode merge checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||
}
|
||||
merged = erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: value}
|
||||
serializedMerge, decodeErr = hydrateCheckpointArtifact(typed.codec, cloneCheckpointArtifact(mergeCP.Output), value)
|
||||
if decodeErr != nil {
|
||||
return fmt.Errorf("hydrate merge checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||
}
|
||||
serializedMerge = hydrated
|
||||
mergeWarnings = cloneWarnings(mergeCP.Warnings)
|
||||
output.Warnings = append(output.Warnings, mergeWarnings...)
|
||||
} else {
|
||||
if err := checkpoints.MergeRunning(lane.ID, lane.Merge.Module, mergeDeps); err != nil {
|
||||
return err
|
||||
if err := checkpointMergeRunning(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
ok, rejection, runErr := runWithRetry(ctx, lane.Merge.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
|
||||
started := time.Now().UTC()
|
||||
attemptPath := path.Join("merge", debugPathComponent(lane.ID), fmt.Sprintf("attempt-%02d", attempt))
|
||||
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "merge", llmScope, debugTimedEnvelope{Stage: string(StageMerge), LaneID: lane.ID, ModuleKey: lane.Merge.Module, Attempt: attempt, StartedAt: started})
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "merge", llmScope, debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, Attempt: attempt, StartedAt: started})
|
||||
requestMetadata, metadataErr := cloneMetadata(input.Metadata)
|
||||
if metadataErr != nil {
|
||||
return false, nil, terminal.record(nil, fmt.Errorf("clone merge request metadata: %w", metadataErr))
|
||||
}
|
||||
result, callErr := typed.merge(attemptCtx, typed.merger, contracts.TypedMergeRequest[any]{Source: doc, LaneID: lane.ID, ExtractOutputs: mergeInputs, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(lane.MergeReferences.ReferenceSet), LLMProfile: lane.Merge.LLMProfile, Metadata: requestMetadata})
|
||||
result, callErr := typed.merge(attemptCtx, typed.merger, contracts.TypedMergeRequest[any]{Source: doc, LaneID: lane.ID, ExtractOutputs: mergeInputs, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(mergeReferences), LLMProfile: lane.Merge.LLMProfile, Metadata: requestMetadata})
|
||||
if callErr != nil {
|
||||
attemptErr := fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, lane.Merge.Module, callErr)
|
||||
return false, nil, terminal.record(nil, attemptErr)
|
||||
@@ -205,7 +271,7 @@ func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoi
|
||||
attemptErr := fmt.Errorf("serialize merge candidate for lane %q: %w", lane.ID, encodeErr)
|
||||
return false, nil, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}, attemptErr)
|
||||
}
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageMerge, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: lane.MergeReferences.ReferenceSet, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.mergeValidators, attempt, input.Debug)
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageMerge, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: mergeReferences, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.mergeValidators, attempt, input.Debug)
|
||||
attemptWarnings = append(attemptWarnings, warnings...)
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
|
||||
if validateErr != nil || rejected != nil {
|
||||
@@ -224,64 +290,74 @@ func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoi
|
||||
return true, nil, nil
|
||||
})
|
||||
if runErr != nil {
|
||||
_ = checkpoints.MergeFailed(lane.ID, lane.Merge.Module, mergeDeps, runErr)
|
||||
return runErr
|
||||
_ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, runErr)
|
||||
return stageResult, runErr
|
||||
}
|
||||
if !ok {
|
||||
output.Rejected = append(output.Rejected, *rejection)
|
||||
if err := checkpoints.MergeRejected(lane.ID, lane.Merge.Module, mergeDeps, *rejection); err != nil {
|
||||
return err
|
||||
if err := checkpointMergeRejected(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, *rejection); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
return nil
|
||||
stageResult.terminal = true
|
||||
return stageResult, nil
|
||||
}
|
||||
output.Warnings = append(output.Warnings, mergeWarnings...)
|
||||
if err := recordMerge(checkpoints, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge, mergeWarnings); err != nil {
|
||||
return err
|
||||
if err := recordMerge(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge, mergeWarnings); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
}
|
||||
if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageMerge), LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedMerge), "warnings": debugWarningEnvelopes(mergeWarnings)}}); err != nil {
|
||||
return err
|
||||
if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedMerge), "warnings": debugWarningEnvelopes(mergeWarnings)}}); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
stageResult.artifact = merged
|
||||
stageResult.serialized = serializedMerge
|
||||
return stageResult, nil
|
||||
}
|
||||
|
||||
activeStage = StageNormalize
|
||||
normalizeDeps := artifactCheckpointDigests([]CheckpointArtifact{serializedMerge})
|
||||
normalizeCP, normalizeDecision := loadNormalize(loader, lane.ID, lane.Normalize.Module, normalizeDeps)
|
||||
if normalizeDecision.Reused {
|
||||
if _, decodeErr := decodeCheckpointArtifact(typed.codec, normalizeCP.Output); decodeErr != nil {
|
||||
normalizeDecision = CheckpointDecision{Reason: "normalize artifact checkpoint codec is incompatible: " + decodeErr.Error()}
|
||||
}
|
||||
type normalizeStageResult struct {
|
||||
serialized CheckpointArtifact
|
||||
warnings []contracts.Warning
|
||||
accepted bool
|
||||
}
|
||||
|
||||
func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, merged erasedMergeArtifact, serializedMerge CheckpointArtifact, output *RunOutput) (normalizeStageResult, error) {
|
||||
var stageResult normalizeStageResult
|
||||
lane, typed := prepared.resolved, prepared.typed
|
||||
normalizeReferences := operationReferenceSet(input, lane.NormalizeReferences)
|
||||
normalizeDeps := append(artifactCheckpointDigests([]CheckpointArtifact{serializedMerge}), generatedReferenceDependencies(normalizeReferences)...)
|
||||
normalizeDeps = normalizeCheckpointFingerprints(normalizeDeps)
|
||||
normalizeCP, normalizeDecision := loadNormalize(loader, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps)
|
||||
normalizeDecision, err := resolveCheckpointDecision(output, loader, input.CheckpointPolicy, StageNormalize, input.stepID, lane.ID, lane.Normalize.Module, normalizeDecision, typed.codec, []CheckpointArtifact{normalizeCP.Output})
|
||||
if err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
recordCheckpointEvent(output, loader, string(StageNormalize), lane.ID, lane.Normalize.Module, normalizeDecision)
|
||||
if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageNormalize), LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "decision": normalizeDecision, "source": debugSourceDocumentEnvelope(doc), "merge_output": debugCheckpointArtifact(serializedMerge), "options": redactSensitiveMap(lane.Normalize.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||
return err
|
||||
if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "decision": normalizeDecision, "source": debugSourceDocumentEnvelope(doc), "merge_output": debugCheckpointArtifact(serializedMerge), "options": redactSensitiveMap(lane.Normalize.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
var serializedNormalize CheckpointArtifact
|
||||
var normalizeWarnings []contracts.Warning
|
||||
if normalizeDecision.Reused {
|
||||
value, decodeErr := decodeCheckpointArtifact(typed.codec, normalizeCP.Output)
|
||||
_, hydrated, decodeErr := decodeCanonicalCheckpointArtifact(typed.codec, normalizeCP.Output)
|
||||
if decodeErr != nil {
|
||||
return fmt.Errorf("decode normalize checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||
}
|
||||
serializedNormalize, decodeErr = hydrateCheckpointArtifact(typed.codec, cloneCheckpointArtifact(normalizeCP.Output), value)
|
||||
if decodeErr != nil {
|
||||
return fmt.Errorf("hydrate normalize checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||
return stageResult, fmt.Errorf("decode normalize checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||
}
|
||||
serializedNormalize = hydrated
|
||||
normalizeWarnings = cloneWarnings(normalizeCP.Warnings)
|
||||
output.Warnings = append(output.Warnings, normalizeWarnings...)
|
||||
} else {
|
||||
if err := checkpoints.NormalizeRunning(lane.ID, lane.Normalize.Module, normalizeDeps); err != nil {
|
||||
return err
|
||||
if err := checkpointNormalizeRunning(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
ok, rejection, runErr := runWithRetry(ctx, lane.Normalize.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
|
||||
started := time.Now().UTC()
|
||||
attemptPath := path.Join("normalize", debugPathComponent(lane.ID), fmt.Sprintf("attempt-%02d", attempt))
|
||||
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "normalize", llmScope, debugTimedEnvelope{Stage: string(StageNormalize), LaneID: lane.ID, ModuleKey: lane.Normalize.Module, Attempt: attempt, StartedAt: started})
|
||||
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "normalize", llmScope, debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, Attempt: attempt, StartedAt: started})
|
||||
requestMetadata, metadataErr := cloneMetadata(input.Metadata)
|
||||
if metadataErr != nil {
|
||||
return false, nil, terminal.record(nil, fmt.Errorf("clone normalize request metadata: %w", metadataErr))
|
||||
}
|
||||
result, callErr := typed.normalize(attemptCtx, typed.normalizer, contracts.TypedNormalizeRequest[any]{Source: doc, LaneID: lane.ID, MergeOutput: contracts.MergeArtifact[any]{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: merged.Value}, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(lane.NormalizeReferences.ReferenceSet), LLMProfile: lane.Normalize.LLMProfile, Metadata: requestMetadata})
|
||||
result, callErr := typed.normalize(attemptCtx, typed.normalizer, contracts.TypedNormalizeRequest[any]{Source: doc, LaneID: lane.ID, MergeOutput: contracts.MergeArtifact[any]{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: merged.Value}, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(normalizeReferences), LLMProfile: lane.Normalize.LLMProfile, Metadata: requestMetadata})
|
||||
if callErr != nil {
|
||||
attemptErr := fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, lane.Normalize.Module, callErr)
|
||||
return false, nil, terminal.record(nil, attemptErr)
|
||||
@@ -292,7 +368,7 @@ func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoi
|
||||
attemptErr := fmt.Errorf("serialize normalize candidate for lane %q: %w", lane.ID, encodeErr)
|
||||
return false, nil, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}, attemptErr)
|
||||
}
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageNormalize, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: lane.NormalizeReferences.ReferenceSet, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.normalizeValidators, attempt, input.Debug)
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageNormalize, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: normalizeReferences, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.normalizeValidators, attempt, input.Debug)
|
||||
attemptWarnings = append(attemptWarnings, warnings...)
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
|
||||
if validateErr != nil || rejected != nil {
|
||||
@@ -311,26 +387,28 @@ func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoi
|
||||
return true, nil, nil
|
||||
})
|
||||
if runErr != nil {
|
||||
_ = checkpoints.NormalizeFailed(lane.ID, lane.Normalize.Module, normalizeDeps, runErr)
|
||||
return runErr
|
||||
_ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, runErr)
|
||||
return stageResult, runErr
|
||||
}
|
||||
if !ok {
|
||||
output.Rejected = append(output.Rejected, *rejection)
|
||||
if err := checkpoints.NormalizeRejected(lane.ID, lane.Normalize.Module, normalizeDeps, *rejection); err != nil {
|
||||
return err
|
||||
if err := checkpointNormalizeRejected(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, *rejection); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
return nil
|
||||
return stageResult, nil
|
||||
}
|
||||
output.Warnings = append(output.Warnings, normalizeWarnings...)
|
||||
if err := recordNormalize(checkpoints, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize, normalizeWarnings); err != nil {
|
||||
return err
|
||||
if err := recordNormalize(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize, normalizeWarnings); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
}
|
||||
if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageNormalize), LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedNormalize), "warnings": debugWarningEnvelopes(normalizeWarnings)}}); err != nil {
|
||||
return err
|
||||
if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedNormalize), "warnings": debugWarningEnvelopes(normalizeWarnings)}}); err != nil {
|
||||
return stageResult, err
|
||||
}
|
||||
output.NormalizeOutputs = append(output.NormalizeOutputs, contracts.SerializedOutput{LaneID: lane.ID, NormalizerKey: lane.Normalize.Module, SourceID: doc.ID, Artifact: contracts.CloneSerializedArtifact(serializedNormalize.Artifact)})
|
||||
return nil
|
||||
stageResult.serialized = serializedNormalize
|
||||
stageResult.warnings = normalizeWarnings
|
||||
stageResult.accepted = true
|
||||
return stageResult, nil
|
||||
}
|
||||
|
||||
func setTypedLaneManifestMetadata(output *RunOutput, laneID string, extractor, merger, normalizer any) error {
|
||||
@@ -414,12 +492,12 @@ func (r *Runner) validateTypedArtifact(ctx context.Context, codec artifactCodecE
|
||||
}
|
||||
if err != nil {
|
||||
validationErr := fmt.Errorf("validate typed %s output with validator %q: %w", target.stage, binding.Module, err)
|
||||
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall, Error: debugCall.Error}, llmScope); debugErr != nil {
|
||||
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall, Error: debugCall.Error}, llmScope); debugErr != nil {
|
||||
return warnings, nil, errors.Join(validationErr, fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr))
|
||||
}
|
||||
return warnings, nil, validationErr
|
||||
}
|
||||
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall}, llmScope); debugErr != nil {
|
||||
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall}, llmScope); debugErr != nil {
|
||||
return warnings, nil, fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr)
|
||||
}
|
||||
if !result.Approved {
|
||||
@@ -431,7 +509,7 @@ func (r *Runner) validateTypedArtifact(ctx context.Context, codec artifactCodecE
|
||||
if message == "" {
|
||||
message = "artifact rejected"
|
||||
}
|
||||
return warnings, &contracts.RejectedOutput{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, ChunkID: func() string {
|
||||
return warnings, &contracts.RejectedOutput{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, ChunkID: func() string {
|
||||
if target.chunk != nil {
|
||||
return target.chunk.ID
|
||||
}
|
||||
|
||||
@@ -1,12 +1,102 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type requiredCheckpointLoader struct {
|
||||
CheckpointLoader
|
||||
checkpoint ExtractCheckpoint
|
||||
decision CheckpointDecision
|
||||
}
|
||||
|
||||
func (l requiredCheckpointLoader) Enabled() bool { return true }
|
||||
func (l requiredCheckpointLoader) Extract(string, string, []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||
return l.checkpoint, l.decision
|
||||
}
|
||||
func (l requiredCheckpointLoader) AcceptedNormalize(string, string, string) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
output := CheckpointArtifact{}
|
||||
if len(l.checkpoint.Outputs) > 0 {
|
||||
output = l.checkpoint.Outputs[0]
|
||||
}
|
||||
return NormalizeCheckpoint{Output: output}, l.decision
|
||||
}
|
||||
|
||||
func TestRequiredCheckpointFailureRetainsDecision(t *testing.T) {
|
||||
const unsafeDetail = "unsafe-loader-detail-/private/checkpoint/path"
|
||||
tests := []struct {
|
||||
name string
|
||||
decision CheckpointDecision
|
||||
corrupt bool
|
||||
wantCode CheckpointReasonCode
|
||||
wantAction CheckpointDecisionCategory
|
||||
}{
|
||||
{"missing", NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing, unsafeDetail), false, CheckpointReasonMissing, CheckpointDecisionExecuted},
|
||||
{"corrupt", NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonReused, "checkpoint reusable"), true, CheckpointReasonArtifactNotCanonical, CheckpointDecisionExecuted},
|
||||
{"incompatible", NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonArtifactCodecIncompatible, unsafeDetail), false, CheckpointReasonArtifactCodecIncompatible, CheckpointDecisionExecuted},
|
||||
{"dependency invalidated", NewCheckpointDecision(CheckpointDecisionDependencyInvalidated, CheckpointReasonDependencyMismatch, unsafeDetail), false, CheckpointReasonDependencyMismatch, CheckpointDecisionDependencyInvalidated},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 1)
|
||||
step := prepared.Steps[0]
|
||||
lane := step.lanes[0]
|
||||
doc := prepared.input.(*typedTestInput).doc
|
||||
checkpoint := ExtractCheckpoint{}
|
||||
if test.corrupt {
|
||||
stored, err := checkpointArtifact(lane.typed.codec, lane.resolved.ID, lane.resolved.Normalize.Module, doc.ID, codecNotes{Items: []string{"stored"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stored.Artifact.Content = []byte(`{"items": ["stored"]}`)
|
||||
checkpoint.Outputs = []CheckpointArtifact{stored}
|
||||
}
|
||||
loader := requiredCheckpointLoader{CheckpointLoader: NoopCheckpointLoader(), checkpoint: checkpoint, decision: test.decision}
|
||||
policy := CheckpointExecutionPolicy{RequireReusableLanes: map[string]struct{}{CheckpointLaneKey(step.ID, lane.resolved.ID): {}}}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: loader, CheckpointPolicy: policy})
|
||||
if err == nil || !strings.Contains(err.Error(), step.ID) || !strings.Contains(err.Error(), lane.resolved.ID) || !strings.Contains(err.Error(), string(test.wantCode)) {
|
||||
t.Fatalf("Run() error = %v, want step, lane, and reason code %q", err, test.wantCode)
|
||||
}
|
||||
if strings.Contains(err.Error(), unsafeDetail) {
|
||||
t.Fatalf("Run() error leaked loader detail: %v", err)
|
||||
}
|
||||
var found bool
|
||||
for _, event := range output.CheckpointEvents {
|
||||
if event.Stage == string(StageNormalize) && event.StepID == step.ID && event.LaneID == lane.resolved.ID {
|
||||
found = true
|
||||
if event.Action != test.wantAction || event.ReasonCode != test.wantCode {
|
||||
t.Fatalf("checkpoint event = %#v, want action %q and reason %q", event, test.wantAction, test.wantCode)
|
||||
}
|
||||
if strings.Contains(event.Detail, unsafeDetail) {
|
||||
t.Fatalf("checkpoint event leaked loader detail: %#v", event)
|
||||
}
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("required checkpoint decision missing from %#v", output.CheckpointEvents)
|
||||
}
|
||||
var manifestFound bool
|
||||
for _, decision := range output.Manifest.CheckpointDecisions {
|
||||
if decision.Stage == string(StageNormalize) && decision.StepID == step.ID && decision.LaneID == lane.resolved.ID {
|
||||
manifestFound = decision.Category == string(test.wantAction) && decision.ReasonCode == string(test.wantCode)
|
||||
}
|
||||
}
|
||||
if !manifestFound {
|
||||
t.Fatalf("manifest checkpoint decision missing category %q and code %q: %#v", test.wantAction, test.wantCode, output.Manifest.CheckpointDecisions)
|
||||
}
|
||||
encoded, marshalErr := json.Marshal(output.CheckpointEvents)
|
||||
if marshalErr != nil || !strings.Contains(string(encoded), `"category":"`+string(test.wantAction)+`"`) || !strings.Contains(string(encoded), `"reason_code":"`+string(test.wantCode)+`"`) {
|
||||
t.Fatalf("checkpoint event JSON = %s, error = %v", encoded, marshalErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDecodeCheckpointArtifactRejectsIncompatibleCodecIdentityAndBytes(t *testing.T) {
|
||||
registry := NewArtifactCodecRegistry()
|
||||
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
|
||||
@@ -43,3 +133,26 @@ func TestDecodeCheckpointArtifactRejectsIncompatibleCodecIdentityAndBytes(t *tes
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDecodeCanonicalCheckpointArtifactRejectsNonCanonicalBytes(t *testing.T) {
|
||||
registry := NewArtifactCodecRegistry()
|
||||
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
|
||||
t.Fatalf("RegisterArtifactCodec: %v", err)
|
||||
}
|
||||
codec, _, err := registry.entry("test/notes")
|
||||
if err != nil {
|
||||
t.Fatalf("entry: %v", err)
|
||||
}
|
||||
artifact, err := serializeArtifact(codec, codecNotes{Items: []string{"one"}}, false)
|
||||
if err != nil {
|
||||
t.Fatalf("serializeArtifact: %v", err)
|
||||
}
|
||||
stored := CheckpointArtifact{Artifact: artifact, SchemaDigest: contracts.DigestArtifactSchema(artifact.Schema)}
|
||||
if _, _, err := decodeCanonicalCheckpointArtifact(codec, stored); err != nil {
|
||||
t.Fatalf("canonical artifact rejected: %v", err)
|
||||
}
|
||||
stored.Artifact.Content = []byte(`{"items": ["one"]}`)
|
||||
if _, _, err := decodeCanonicalCheckpointArtifact(codec, stored); err == nil {
|
||||
t.Fatal("non-canonical artifact was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -76,6 +76,35 @@ func (l *lockedCheckpointLoader) Normalize(lane, key string, deps []CheckpointFi
|
||||
defer l.mu.Unlock()
|
||||
return l.inner.Normalize(lane, key, deps)
|
||||
}
|
||||
func (l *lockedCheckpointLoader) AcceptedNormalize(step, lane, key string) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.inner.AcceptedNormalize(step, lane, key)
|
||||
}
|
||||
func (l *lockedCheckpointLoader) ExtractForStep(step, lane, key string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
if stepAware, ok := l.inner.(StepCheckpointLoader); ok {
|
||||
return stepAware.ExtractForStep(step, lane, key, deps)
|
||||
}
|
||||
return l.inner.Extract(lane, key, deps)
|
||||
}
|
||||
func (l *lockedCheckpointLoader) MergeForStep(step, lane, key string, deps []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
if stepAware, ok := l.inner.(StepCheckpointLoader); ok {
|
||||
return stepAware.MergeForStep(step, lane, key, deps)
|
||||
}
|
||||
return l.inner.Merge(lane, key, deps)
|
||||
}
|
||||
func (l *lockedCheckpointLoader) NormalizeForStep(step, lane, key string, deps []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
if stepAware, ok := l.inner.(StepCheckpointLoader); ok {
|
||||
return stepAware.NormalizeForStep(step, lane, key, deps)
|
||||
}
|
||||
return l.inner.Normalize(lane, key, deps)
|
||||
}
|
||||
|
||||
type lockedCheckpointRecorder struct {
|
||||
inner CheckpointRecorder
|
||||
@@ -135,3 +164,92 @@ func (r *lockedCheckpointRecorder) NormalizeRejected(lane, key string, deps []Ch
|
||||
func (r *lockedCheckpointRecorder) NormalizeFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
|
||||
return r.call(func() error { return r.inner.NormalizeFailed(lane, key, deps, err) })
|
||||
}
|
||||
|
||||
func (r *lockedCheckpointRecorder) ExtractRunningForStep(step, lane, key string, deps []CheckpointFingerprint) error {
|
||||
return r.call(func() error {
|
||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||
return v.ExtractRunningForStep(step, lane, key, deps)
|
||||
}
|
||||
return r.inner.ExtractRunning(lane, key, deps)
|
||||
})
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) ExtractSucceededForStep(step, lane, key string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||
return r.call(func() error {
|
||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||
return v.ExtractSucceededForStep(step, lane, key, deps, outputs, rejected, warnings)
|
||||
}
|
||||
return r.inner.ExtractSucceeded(lane, key, deps, outputs, rejected, warnings)
|
||||
})
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) ExtractFailedForStep(step, lane, key string, deps []CheckpointFingerprint, err error) error {
|
||||
return r.call(func() error {
|
||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||
return v.ExtractFailedForStep(step, lane, key, deps, err)
|
||||
}
|
||||
return r.inner.ExtractFailed(lane, key, deps, err)
|
||||
})
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) MergeRunningForStep(step, lane, key string, deps []CheckpointFingerprint) error {
|
||||
return r.call(func() error {
|
||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||
return v.MergeRunningForStep(step, lane, key, deps)
|
||||
}
|
||||
return r.inner.MergeRunning(lane, key, deps)
|
||||
})
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) MergeSucceededForStep(step, lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
return r.call(func() error {
|
||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||
return v.MergeSucceededForStep(step, lane, key, deps, output, warnings)
|
||||
}
|
||||
return r.inner.MergeSucceeded(lane, key, deps, output, warnings)
|
||||
})
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) MergeRejectedForStep(step, lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||
return r.call(func() error {
|
||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||
return v.MergeRejectedForStep(step, lane, key, deps, rejected)
|
||||
}
|
||||
return r.inner.MergeRejected(lane, key, deps, rejected)
|
||||
})
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) MergeFailedForStep(step, lane, key string, deps []CheckpointFingerprint, err error) error {
|
||||
return r.call(func() error {
|
||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||
return v.MergeFailedForStep(step, lane, key, deps, err)
|
||||
}
|
||||
return r.inner.MergeFailed(lane, key, deps, err)
|
||||
})
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) NormalizeRunningForStep(step, lane, key string, deps []CheckpointFingerprint) error {
|
||||
return r.call(func() error {
|
||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||
return v.NormalizeRunningForStep(step, lane, key, deps)
|
||||
}
|
||||
return r.inner.NormalizeRunning(lane, key, deps)
|
||||
})
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) NormalizeSucceededForStep(step, lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
return r.call(func() error {
|
||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||
return v.NormalizeSucceededForStep(step, lane, key, deps, output, warnings)
|
||||
}
|
||||
return r.inner.NormalizeSucceeded(lane, key, deps, output, warnings)
|
||||
})
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) NormalizeRejectedForStep(step, lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||
return r.call(func() error {
|
||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||
return v.NormalizeRejectedForStep(step, lane, key, deps, rejected)
|
||||
}
|
||||
return r.inner.NormalizeRejected(lane, key, deps, rejected)
|
||||
})
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) NormalizeFailedForStep(step, lane, key string, deps []CheckpointFingerprint, err error) error {
|
||||
return r.call(func() error {
|
||||
if v, ok := r.inner.(StepCheckpointRecorder); ok {
|
||||
return v.NormalizeFailedForStep(step, lane, key, deps, err)
|
||||
}
|
||||
return r.inner.NormalizeFailed(lane, key, deps, err)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -28,6 +28,7 @@ type erasedTypedResult struct {
|
||||
|
||||
type typedValidationTarget struct {
|
||||
stage ModuleStage
|
||||
stepID string
|
||||
laneID string
|
||||
moduleKey string
|
||||
source *source.SourceDocument
|
||||
|
||||
@@ -147,11 +147,11 @@ func TestResolveTypedHeterogeneousLanes(t *testing.T) {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
if got, want := resolvedLaneIDs(resolved.ArtifactLanes), []string{"notes", "score"}; !reflect.DeepEqual(got, want) {
|
||||
if got, want := resolvedLaneIDs(resolved.Steps[0].ArtifactLanes), []string{"notes", "score"}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("lane order = %#v, want %#v", got, want)
|
||||
}
|
||||
assertResolvedArtifactIdentity(t, resolved.ArtifactLanes[0], "test/notes", "notes.v1")
|
||||
assertResolvedArtifactIdentity(t, resolved.ArtifactLanes[1], "test/score", "score.v1")
|
||||
assertResolvedArtifactIdentity(t, resolved.Steps[0].ArtifactLanes[0], "test/notes", "notes.v1")
|
||||
assertResolvedArtifactIdentity(t, resolved.Steps[0].ArtifactLanes[1], "test/score", "score.v1")
|
||||
|
||||
chunkChain := resolved.ValidatorChains[0]
|
||||
if got := resolvedValidatorTargets(chunkChain.Validators); !reflect.DeepEqual(got, []ValidatorTarget{ValidatorTargetChunk, ValidatorTargetSerialized}) {
|
||||
@@ -176,8 +176,8 @@ func TestPrepareConstructsHeterogeneousTypedLanes(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||
}
|
||||
if len(prepared.ArtifactLanes) != 2 || prepared.lanes[0].typed == nil || prepared.lanes[1].typed == nil {
|
||||
t.Fatalf("PreparedPipeline lanes = %#v, want two typed executors", prepared.ArtifactLanes)
|
||||
if len(prepared.Steps[0].ArtifactLanes) != 2 || prepared.Steps[0].lanes[0].typed == nil || prepared.Steps[0].lanes[1].typed == nil {
|
||||
t.Fatalf("PreparedPipeline lanes = %#v, want two typed executors", prepared.Steps[0].ArtifactLanes)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -92,9 +92,9 @@ func cloneModuleBinding(binding ModuleBinding) ModuleBinding {
|
||||
binding.LLMProfile = strings.TrimSpace(binding.LLMProfile)
|
||||
binding.Options = cloneOptions(binding.Options)
|
||||
if len(binding.References) > 0 {
|
||||
references := make(map[string]string, len(binding.References))
|
||||
references := make(map[string]ReferenceSource, len(binding.References))
|
||||
for key, value := range binding.References {
|
||||
references[key] = value
|
||||
references[key] = cloneReferenceSource(value)
|
||||
}
|
||||
binding.References = references
|
||||
}
|
||||
@@ -102,6 +102,15 @@ func cloneModuleBinding(binding ModuleBinding) ModuleBinding {
|
||||
return binding
|
||||
}
|
||||
|
||||
func cloneReferenceSource(source ReferenceSource) ReferenceSource {
|
||||
out := source
|
||||
if source.Artifact != nil {
|
||||
artifact := *source.Artifact
|
||||
out.Artifact = &artifact
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func cloneValidatorOverride(override ValidatorOverride) ValidatorOverride {
|
||||
return ValidatorOverride{
|
||||
Set: override.Set,
|
||||
|
||||
@@ -42,10 +42,11 @@ var referenceSlotDescriptions = shared.ReferenceSlotDescriptions{
|
||||
func referenceSlots() []contracts.ReferenceSlot {
|
||||
slots := shared.ReferenceSlots(referenceSlotDescriptions)
|
||||
slots = append(slots, contracts.ReferenceSlot{
|
||||
Name: NPCRegistryReferenceSlot,
|
||||
Description: "Optional normalized NPC registry used for canonical actor and target grounding.",
|
||||
AcceptedMediaTypes: []string{"application/json"},
|
||||
MaxBytes: NPCRegistryMaxBytes,
|
||||
Name: NPCRegistryReferenceSlot,
|
||||
Description: "Optional normalized NPC registry used for canonical actor and target grounding.",
|
||||
AcceptedMediaTypes: []string{"application/json"},
|
||||
AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.NPCListKind},
|
||||
MaxBytes: NPCRegistryMaxBytes,
|
||||
})
|
||||
sort.Slice(slots, func(i, j int) bool { return slots[i].Name < slots[j].Name })
|
||||
return slots
|
||||
@@ -59,7 +60,7 @@ type Options struct{}
|
||||
|
||||
type Extractor struct {
|
||||
llm contracts.StructuredLLMClient
|
||||
npcRegistry *npcregistry.Registry
|
||||
npcResolver *npcregistry.Resolver
|
||||
promptSHA string
|
||||
responseSchemaSHA string
|
||||
}
|
||||
@@ -75,7 +76,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
||||
if len(references) == 1 {
|
||||
referenceSet = references[0]
|
||||
}
|
||||
npcRegistry, err := npcregistry.Resolve(referenceSet)
|
||||
npcResolver, err := npcregistry.NewResolver(referenceSet)
|
||||
if err != nil {
|
||||
return nil, extractorErrorf("prepare NPC registry prompt input: %w", err)
|
||||
}
|
||||
@@ -89,7 +90,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
||||
}
|
||||
return &Extractor{
|
||||
llm: llmClient,
|
||||
npcRegistry: npcRegistry,
|
||||
npcResolver: npcResolver,
|
||||
promptSHA: promptSHA,
|
||||
responseSchemaSHA: responseSchema.SHA256,
|
||||
}, nil
|
||||
@@ -114,9 +115,10 @@ func (e *Extractor) ManifestMetadata() map[string]any {
|
||||
"response_schema_version": SchemaVersion,
|
||||
"response_schema_sha256": e.responseSchemaSHA,
|
||||
}
|
||||
if e.npcRegistry.Bound() {
|
||||
metadata["npc_registry_digest"] = e.npcRegistry.Digest()
|
||||
metadata["npc_count"] = e.npcRegistry.Count()
|
||||
seeded := e.npcResolver.Seeded()
|
||||
if seeded.Bound() {
|
||||
metadata["npc_registry_digest"] = seeded.Digest()
|
||||
metadata["npc_count"] = seeded.Count()
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
@@ -130,8 +132,9 @@ func (e *Extractor) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||
{Name: "response_schema", Value: e.responseSchemaSHA},
|
||||
{Name: "mapping_policy", Value: mappingPolicy},
|
||||
}
|
||||
if e.npcRegistry.Bound() {
|
||||
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: e.npcRegistry.Digest()})
|
||||
seeded := e.npcResolver.Seeded()
|
||||
if seeded.Bound() {
|
||||
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: seeded.Digest()})
|
||||
}
|
||||
return fingerprints
|
||||
}
|
||||
@@ -162,10 +165,14 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.CombatTurnList]{}, extractorErrorf("%w", err)
|
||||
}
|
||||
npcRegistry, err := e.npcResolver.Resolve(req.References)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.CombatTurnList]{}, extractorErrorf("resolve NPC registry: %w", err)
|
||||
}
|
||||
|
||||
var response extractionResponse
|
||||
inputs := shared.PromptInputs(sourceInput, req.References)
|
||||
inputs[NPCRegistryReferenceSlot] = e.npcRegistry.PromptInput()
|
||||
inputs[NPCRegistryReferenceSlot] = npcRegistry.PromptInput()
|
||||
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key,
|
||||
PromptID: PromptID,
|
||||
|
||||
@@ -160,6 +160,29 @@ func TestExtractUnboundRegistryUsesExactEmptyPromptAndOmitsIdentity(t *testing.T
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorResolvesOperationNPCOverrideWithoutSingletonMetadata(t *testing.T) {
|
||||
content := npcRegistryJSON(t)
|
||||
client := &fakeCombatTurnsLLMClient{response: extractionResponse{CombatTurns: []combatTurnResponse{}}}
|
||||
extractor := newExtractor(t, client)
|
||||
request := extractionRequest()
|
||||
request.References = contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||
NPCRegistryReferenceSlot: {
|
||||
Slot: contracts.ReferenceSlot{Name: NPCRegistryReferenceSlot},
|
||||
Items: []contracts.ReferenceItem{{SlotName: NPCRegistryReferenceSlot, MediaType: npccodec.MediaType, Content: append([]byte(nil), content...), Origin: contracts.ReferenceOrigin{Type: "generated"}}},
|
||||
},
|
||||
}}
|
||||
if _, err := extractor.Extract(context.Background(), request); err != nil {
|
||||
t.Fatalf("Extract() error = %v", err)
|
||||
}
|
||||
input := client.requests[0].Inputs[NPCRegistryReferenceSlot]
|
||||
if input.Digest == "" || string(input.Content) != string(content) || input.OriginURI != "" {
|
||||
t.Fatalf("operation NPC input = %#v, want generated canonical grounding without provenance", input)
|
||||
}
|
||||
if metadata := extractor.ManifestMetadata(); metadata["npc_registry_digest"] != nil || metadata["npc_count"] != nil {
|
||||
t.Fatalf("singleton metadata = %#v, want no operation-varying NPC identity", metadata)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewRejectsMalformedNPCRegistryBeforeLLMCallWithoutContent(t *testing.T) {
|
||||
client := &fakeCombatTurnsLLMClient{}
|
||||
references := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||
|
||||
@@ -46,10 +46,11 @@ func referenceSlots() []contracts.ReferenceSlot {
|
||||
AcceptedMediaTypes: []string{"application/json"},
|
||||
MaxBytes: 1048576,
|
||||
}, contracts.ReferenceSlot{
|
||||
Name: NPCRegistryReferenceSlot,
|
||||
Description: "Optional normalized NPC registry used for canonical caster-name grounding.",
|
||||
AcceptedMediaTypes: []string{"application/json"},
|
||||
MaxBytes: NPCRegistryMaxBytes,
|
||||
Name: NPCRegistryReferenceSlot,
|
||||
Description: "Optional normalized NPC registry used for canonical caster-name grounding.",
|
||||
AcceptedMediaTypes: []string{"application/json"},
|
||||
AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.NPCListKind},
|
||||
MaxBytes: NPCRegistryMaxBytes,
|
||||
})
|
||||
sort.Slice(slots, func(i, j int) bool { return slots[i].Name < slots[j].Name })
|
||||
return slots
|
||||
@@ -64,7 +65,7 @@ type Extractor struct {
|
||||
llm contracts.StructuredLLMClient
|
||||
effectiveCatalog spellcatalog.EffectiveCatalog
|
||||
catalogPromptInput contracts.LLMInputMaterial
|
||||
npcRegistry *npcregistry.Registry
|
||||
npcResolver *npcregistry.Resolver
|
||||
promptSHA string
|
||||
responseSchemaSHA string
|
||||
}
|
||||
@@ -88,7 +89,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
||||
if err != nil {
|
||||
return nil, extractorErrorf("prepare spell catalog prompt input: %w", err)
|
||||
}
|
||||
npcRegistry, err := npcregistry.Resolve(referenceSet)
|
||||
npcResolver, err := npcregistry.NewResolver(referenceSet)
|
||||
if err != nil {
|
||||
return nil, extractorErrorf("prepare NPC registry prompt input: %w", err)
|
||||
}
|
||||
@@ -104,7 +105,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
||||
llm: llmClient,
|
||||
effectiveCatalog: effectiveCatalog,
|
||||
catalogPromptInput: catalogPromptInput,
|
||||
npcRegistry: npcRegistry,
|
||||
npcResolver: npcResolver,
|
||||
promptSHA: promptSHA,
|
||||
responseSchemaSHA: responseSchema.SHA256,
|
||||
}, nil
|
||||
@@ -132,9 +133,10 @@ func (e *Extractor) ManifestMetadata() map[string]any {
|
||||
"response_schema_version": SchemaVersion,
|
||||
"response_schema_sha256": e.responseSchemaSHA,
|
||||
}
|
||||
if e.npcRegistry.Bound() {
|
||||
metadata["npc_registry_digest"] = e.npcRegistry.Digest()
|
||||
metadata["npc_count"] = e.npcRegistry.Count()
|
||||
seeded := e.npcResolver.Seeded()
|
||||
if seeded.Bound() {
|
||||
metadata["npc_registry_digest"] = seeded.Digest()
|
||||
metadata["npc_count"] = seeded.Count()
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
@@ -148,8 +150,9 @@ func (e *Extractor) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||
{Name: "prompt", Value: e.promptSHA},
|
||||
{Name: "response_schema", Value: e.responseSchemaSHA},
|
||||
}
|
||||
if e.npcRegistry.Bound() {
|
||||
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: e.npcRegistry.Digest()})
|
||||
seeded := e.npcResolver.Seeded()
|
||||
if seeded.Bound() {
|
||||
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: seeded.Digest()})
|
||||
}
|
||||
return fingerprints
|
||||
}
|
||||
@@ -180,11 +183,15 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.SpellList]{}, extractorErrorf("%w", err)
|
||||
}
|
||||
npcRegistry, err := e.npcResolver.Resolve(req.References)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.SpellList]{}, extractorErrorf("resolve NPC registry: %w", err)
|
||||
}
|
||||
|
||||
var response extractionResponse
|
||||
inputs := shared.PromptInputs(sourceInput, req.References)
|
||||
inputs[spellcatalog.SpellCatalogReferenceSlot] = e.catalogPromptInput.Clone()
|
||||
inputs[NPCRegistryReferenceSlot] = e.npcRegistry.PromptInput()
|
||||
inputs[NPCRegistryReferenceSlot] = npcRegistry.PromptInput()
|
||||
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key,
|
||||
PromptID: PromptID,
|
||||
|
||||
@@ -84,6 +84,32 @@ func TestSpellExtractorPreservesSemanticNPCRegistryFingerprintAndPromptWiring(t
|
||||
}
|
||||
}
|
||||
|
||||
func TestSpellExtractorResolvesOperationNPCOverrideWithoutSingletonMetadata(t *testing.T) {
|
||||
canonical, err := npccodec.New().Encode(registryFixture())
|
||||
if err != nil {
|
||||
t.Fatalf("encode NPC registry: %v", err)
|
||||
}
|
||||
client := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{}}}
|
||||
extractor := newExtractor(t, client)
|
||||
request := extractionRequest()
|
||||
request.References = spellNPCRegistryReference(canonical, "file:///generated.json")
|
||||
if _, err := extractor.Extract(context.Background(), request); err != nil {
|
||||
t.Fatalf("Extract() error = %v", err)
|
||||
}
|
||||
input := client.requests[0].Inputs[NPCRegistryReferenceSlot]
|
||||
if input.Digest == "" || string(input.Content) != string(canonical) || input.OriginURI != "" {
|
||||
t.Fatalf("operation NPC input = %#v, want generated canonical grounding without provenance", input)
|
||||
}
|
||||
if metadata := extractor.ManifestMetadata(); metadata["npc_registry_digest"] != nil || metadata["npc_count"] != nil {
|
||||
t.Fatalf("singleton metadata = %#v, want no operation-varying NPC identity", metadata)
|
||||
}
|
||||
for _, fingerprint := range extractor.CheckpointFingerprints() {
|
||||
if fingerprint.Name == "npc_registry" {
|
||||
t.Fatalf("singleton fingerprints = %#v, want no operation-varying NPC identity", extractor.CheckpointFingerprints())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func registryFixture() dnd.NPCList {
|
||||
return dnd.NPCList{NPCs: []dnd.NPC{{
|
||||
ID: identity.DeriveID("Mira Thorn"),
|
||||
|
||||
@@ -40,10 +40,11 @@ func TestModuleSpec(t *testing.T) {
|
||||
AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"},
|
||||
},
|
||||
{
|
||||
Name: NPCRegistryReferenceSlot,
|
||||
Description: "Optional normalized NPC registry used for canonical caster-name grounding.",
|
||||
AcceptedMediaTypes: []string{"application/json"},
|
||||
MaxBytes: NPCRegistryMaxBytes,
|
||||
Name: NPCRegistryReferenceSlot,
|
||||
Description: "Optional normalized NPC registry used for canonical caster-name grounding.",
|
||||
AcceptedMediaTypes: []string{"application/json"},
|
||||
AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.NPCListKind},
|
||||
MaxBytes: NPCRegistryMaxBytes,
|
||||
},
|
||||
{
|
||||
Name: "party",
|
||||
|
||||
@@ -45,7 +45,7 @@ var _ pipeline.CheckpointFingerprintProvider = (*Normalizer)(nil)
|
||||
type Options struct{}
|
||||
|
||||
type Normalizer struct {
|
||||
npcRegistry *npcregistry.Registry
|
||||
npcResolver *npcregistry.Resolver
|
||||
}
|
||||
|
||||
func New(_ Options, references ...contracts.ReferenceSet) (*Normalizer, error) {
|
||||
@@ -56,11 +56,11 @@ func New(_ Options, references ...contracts.ReferenceSet) (*Normalizer, error) {
|
||||
if len(references) == 1 {
|
||||
referenceSet = references[0]
|
||||
}
|
||||
npcRegistry, err := npcregistry.Resolve(referenceSet)
|
||||
npcResolver, err := npcregistry.NewResolver(referenceSet)
|
||||
if err != nil {
|
||||
return nil, normalizerErrorf("prepare NPC registry: %w", err)
|
||||
}
|
||||
return &Normalizer{npcRegistry: npcRegistry}, nil
|
||||
return &Normalizer{npcResolver: npcResolver}, nil
|
||||
}
|
||||
|
||||
func (n *Normalizer) Key() string { return Key }
|
||||
@@ -75,9 +75,10 @@ func (n *Normalizer) ManifestMetadata() map[string]any {
|
||||
"normalization_policy": normalizationPolicy,
|
||||
"identity_policy": identity.Policy,
|
||||
}
|
||||
if n.npcRegistry.Bound() {
|
||||
metadata["npc_registry_digest"] = n.npcRegistry.Digest()
|
||||
metadata["npc_count"] = n.npcRegistry.Count()
|
||||
seeded := n.npcResolver.Seeded()
|
||||
if seeded.Bound() {
|
||||
metadata["npc_registry_digest"] = seeded.Digest()
|
||||
metadata["npc_count"] = seeded.Count()
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
@@ -90,8 +91,9 @@ func (n *Normalizer) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||
{Name: "normalization_policy", Value: normalizationPolicy},
|
||||
{Name: "identity_policy", Value: identity.Policy},
|
||||
}
|
||||
if n.npcRegistry.Bound() {
|
||||
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: n.npcRegistry.Digest()})
|
||||
seeded := n.npcResolver.Seeded()
|
||||
if seeded.Bound() {
|
||||
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: seeded.Digest()})
|
||||
}
|
||||
return fingerprints
|
||||
}
|
||||
@@ -107,7 +109,11 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
||||
return contracts.TypedNormalizeResult[dnd.CombatTurnList]{}, normalizerErrorf("context error before normalize: %w", err)
|
||||
}
|
||||
|
||||
value, warnings := normalizeList(req.MergeOutput.Value, req.Source, n.npcRegistry)
|
||||
npcRegistry, err := n.npcResolver.Resolve(req.References)
|
||||
if err != nil {
|
||||
return contracts.TypedNormalizeResult[dnd.CombatTurnList]{}, normalizerErrorf("resolve NPC registry: %w", err)
|
||||
}
|
||||
value, warnings := normalizeList(req.MergeOutput.Value, req.Source, npcRegistry)
|
||||
return contracts.TypedNormalizeResult[dnd.CombatTurnList]{Value: value, Warnings: warnings}, nil
|
||||
}
|
||||
|
||||
@@ -494,10 +500,11 @@ func turnScope(index int) string { return fmt.Sprintf("combat_turns[%d]", index)
|
||||
|
||||
func referenceSlots() []contracts.ReferenceSlot {
|
||||
return []contracts.ReferenceSlot{{
|
||||
Name: NPCRegistryReferenceSlot,
|
||||
Description: "Optional normalized NPC registry used for canonical actor and target grounding.",
|
||||
AcceptedMediaTypes: []string{"application/json"},
|
||||
MaxBytes: NPCRegistryMaxBytes,
|
||||
Name: NPCRegistryReferenceSlot,
|
||||
Description: "Optional normalized NPC registry used for canonical actor and target grounding.",
|
||||
AcceptedMediaTypes: []string{"application/json"},
|
||||
AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.NPCListKind},
|
||||
MaxBytes: NPCRegistryMaxBytes,
|
||||
}}
|
||||
}
|
||||
|
||||
|
||||
@@ -70,6 +70,36 @@ func TestNormalizeCanonicalizesFieldsAndRegistryIdentities(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeResolvesOperationNPCOverrideWithoutSingletonMetadata(t *testing.T) {
|
||||
doc := testDocument()
|
||||
normalizer, err := New(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
input := dnd.CombatTurnList{CombatTurns: []dnd.CombatTurn{{
|
||||
Actor: "Storm",
|
||||
TurnKind: dnd.CombatTurnKindTurn,
|
||||
Actions: []dnd.CombatAction{{Category: dnd.CombatActionCategoryAttack, Declaration: "attacks", Targets: []string{"Minion"}}},
|
||||
Summary: "Storm attacks",
|
||||
SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}},
|
||||
}}}
|
||||
result, err := normalizer.Normalize(context.Background(), contracts.TypedNormalizeRequest[dnd.CombatTurnList]{
|
||||
Source: doc,
|
||||
MergeOutput: contracts.MergeArtifact[dnd.CombatTurnList]{Value: input},
|
||||
References: npcReferences(t),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Normalize() error = %v", err)
|
||||
}
|
||||
turn := result.Value.CombatTurns[0]
|
||||
if turn.Actor != "Aria" || turn.Actions[0].Targets[0] != "Goblin" {
|
||||
t.Fatalf("operation-normalized turn = %#v, want Aria/Goblin", turn)
|
||||
}
|
||||
if metadata := normalizer.ManifestMetadata(); metadata["npc_registry_digest"] != nil || metadata["npc_count"] != nil {
|
||||
t.Fatalf("singleton metadata = %#v, want no operation-varying NPC identity", metadata)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeOrdersBySourcePositionAndCollapsesExactDuplicates(t *testing.T) {
|
||||
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{{ID: 50}, {ID: 10}, {ID: 90}}}
|
||||
first := validTurn("Aria", source.SourceRef{SourceID: doc.ID, StartUnitID: 50, EndUnitID: 50})
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"fmt"
|
||||
"mime"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
@@ -34,6 +35,106 @@ type Registry struct {
|
||||
lookupByKey map[string]int
|
||||
}
|
||||
|
||||
// Resolver retains only the validated construction-time registry and immutable
|
||||
// canonical registries keyed by their semantic digest. Operation references
|
||||
// are resolved on demand; caller-owned reference bytes are never retained.
|
||||
type Resolver struct {
|
||||
seeded *Registry
|
||||
|
||||
mu sync.Mutex
|
||||
cache map[string]*Registry
|
||||
rawCache map[string]*Registry
|
||||
}
|
||||
|
||||
// NewResolver validates the optional construction-time NPC reference and
|
||||
// prepares the operation-time registry cache. A malformed static reference
|
||||
// therefore fails before any operation starts.
|
||||
func NewResolver(references contracts.ReferenceSet) (*Resolver, error) {
|
||||
seeded, err := Resolve(constructionReferences(references))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Resolver{seeded: seeded, cache: make(map[string]*Registry), rawCache: make(map[string]*Registry)}, nil
|
||||
}
|
||||
|
||||
func constructionReferences(references contracts.ReferenceSet) contracts.ReferenceSet {
|
||||
slot, ok := references.Slots[ReferenceSlot]
|
||||
if !ok || len(slot.Items) > 0 {
|
||||
return references
|
||||
}
|
||||
cloned := contracts.ReferenceSet{Slots: make(map[string]contracts.ResolvedReferenceSlot, len(references.Slots))}
|
||||
for name, value := range references.Slots {
|
||||
cloned.Slots[name] = value
|
||||
}
|
||||
delete(cloned.Slots, ReferenceSlot)
|
||||
return cloned
|
||||
}
|
||||
|
||||
// Seeded returns the immutable construction-time registry. Its accessors are
|
||||
// defensive, so callers may safely use the returned view for static metadata.
|
||||
func (r *Resolver) Seeded() *Registry {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
return r.seeded
|
||||
}
|
||||
|
||||
// Resolve returns the effective registry for one operation. An operation
|
||||
// without an NPC item uses the construction-time registry. A canonical item
|
||||
// matching that registry reuses it; other canonical registries are cached by
|
||||
// digest for concurrent chunk operations.
|
||||
func (r *Resolver) Resolve(references contracts.ReferenceSet) (*Registry, error) {
|
||||
if r == nil {
|
||||
return Resolve(references)
|
||||
}
|
||||
if _, ok := references.Slots[ReferenceSlot]; !ok {
|
||||
return r.seeded, nil
|
||||
}
|
||||
|
||||
slot := references.Slots[ReferenceSlot]
|
||||
rawKey := ""
|
||||
if len(slot.Items) == 1 {
|
||||
rawKey = strings.ToLower(strings.TrimSpace(slot.Items[0].MediaType)) + "\x00" + semanticDigest(slot.Items[0].Content)
|
||||
}
|
||||
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if rawKey != "" {
|
||||
if cached, ok := r.rawCache[rawKey]; ok {
|
||||
return cached, nil
|
||||
}
|
||||
}
|
||||
resolved, err := Resolve(references)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if sameRegistryIdentity(r.seeded, resolved) {
|
||||
if rawKey != "" {
|
||||
r.rawCache[rawKey] = r.seeded
|
||||
}
|
||||
return r.seeded, nil
|
||||
}
|
||||
|
||||
if cached, ok := r.cache[resolved.Digest()]; ok {
|
||||
if rawKey != "" {
|
||||
r.rawCache[rawKey] = cached
|
||||
}
|
||||
return cached, nil
|
||||
}
|
||||
r.cache[resolved.Digest()] = resolved
|
||||
if rawKey != "" {
|
||||
r.rawCache[rawKey] = resolved
|
||||
}
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
func sameRegistryIdentity(first, second *Registry) bool {
|
||||
if first == nil || second == nil {
|
||||
return first == second
|
||||
}
|
||||
return first.bound == second.bound && first.digest == second.digest
|
||||
}
|
||||
|
||||
// Resolve prepares the optional NPC registry reference. An absent slot
|
||||
// produces the exact empty prompt input and no semantic registry identity.
|
||||
func Resolve(references contracts.ReferenceSet) (*Registry, error) {
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
|
||||
@@ -168,6 +169,95 @@ func TestRegistryAccessorsAndLookupAreDefensive(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolverUsesSeededFallbackAndCachesGeneratedCanonicalRegistry(t *testing.T) {
|
||||
staticContent := encodeRegistry(t, validRegistryList())
|
||||
resolver, err := NewResolver(registryReference(staticContent, "file:///static.json"))
|
||||
if err != nil {
|
||||
t.Fatalf("NewResolver() error = %v", err)
|
||||
}
|
||||
if got, err := resolver.Resolve(contracts.ReferenceSet{}); err != nil || got != resolver.Seeded() {
|
||||
t.Fatalf("Resolve(absent) = %p, %v, want seeded %p", got, err, resolver.Seeded())
|
||||
}
|
||||
|
||||
generated := registryReference(append([]byte("\n"), staticContent...), "file:///generated.json")
|
||||
first, err := resolver.Resolve(generated)
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve(generated) error = %v", err)
|
||||
}
|
||||
second, err := resolver.Resolve(generated)
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve(generated second) error = %v", err)
|
||||
}
|
||||
if first != resolver.Seeded() || second != first {
|
||||
t.Fatalf("resolved registries = %p, %p, seeded %p; want seeded reuse", first, second, resolver.Seeded())
|
||||
}
|
||||
|
||||
changed := validRegistryList()
|
||||
changed.NPCs[0].Description = "A changed generated description."
|
||||
changedContent := encodeRegistry(t, changed)
|
||||
changedReferences := registryReference(changedContent, "file:///changed.json")
|
||||
resolved, err := resolver.Resolve(changedReferences)
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve(changed) error = %v", err)
|
||||
}
|
||||
changedReferences.Slots[ReferenceSlot].Items[0].Content[0] = 'X'
|
||||
if resolved == first || string(resolved.CanonicalBytes()) != string(changedContent) {
|
||||
t.Fatalf("changed registry = %p/%s, want independent canonical cache entry", resolved, resolved.CanonicalBytes())
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolverSharesOneCachedRegistryAcrossConcurrentOperations(t *testing.T) {
|
||||
content := encodeRegistry(t, validRegistryList())
|
||||
resolver, err := NewResolver(contracts.ReferenceSet{})
|
||||
if err != nil {
|
||||
t.Fatalf("NewResolver() error = %v", err)
|
||||
}
|
||||
references := registryReference(content, "file:///generated.json")
|
||||
const callers = 32
|
||||
results := make(chan *Registry, callers)
|
||||
errors := make(chan error, callers)
|
||||
var wait sync.WaitGroup
|
||||
for index := 0; index < callers; index++ {
|
||||
wait.Add(1)
|
||||
go func() {
|
||||
defer wait.Done()
|
||||
resolved, resolveErr := resolver.Resolve(references)
|
||||
if resolveErr != nil {
|
||||
errors <- resolveErr
|
||||
return
|
||||
}
|
||||
results <- resolved
|
||||
}()
|
||||
}
|
||||
wait.Wait()
|
||||
close(results)
|
||||
close(errors)
|
||||
for err := range errors {
|
||||
t.Fatalf("concurrent Resolve() error = %v", err)
|
||||
}
|
||||
var first *Registry
|
||||
for resolved := range results {
|
||||
if first == nil {
|
||||
first = resolved
|
||||
} else if resolved != first {
|
||||
t.Fatalf("concurrent resolved registry %p differs from cached %p", resolved, first)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewResolverAllowsGeneratedDeclarationButRejectsMalformedStaticItem(t *testing.T) {
|
||||
placeholder := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||
ReferenceSlot: {Slot: contracts.ReferenceSlot{Name: ReferenceSlot, AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.NPCListKind}}},
|
||||
}}
|
||||
if _, err := NewResolver(placeholder); err != nil {
|
||||
t.Fatalf("NewResolver(generated declaration) error = %v, want nil", err)
|
||||
}
|
||||
malformed := registryReference([]byte(`{"npcs":[`), "file:///runtime.json")
|
||||
if _, err := NewResolver(malformed); err == nil || !strings.Contains(err.Error(), "invalid approved NPC JSON") {
|
||||
t.Fatalf("NewResolver(malformed) error = %v, want bounded decode failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func validRegistryList() dnd.NPCList {
|
||||
return dnd.NPCList{NPCs: []dnd.NPC{{
|
||||
ID: identity.DeriveID("Mira Thorn"),
|
||||
|
||||
@@ -152,79 +152,6 @@ func TestCombatNormalizerRejectsCampaignReferenceBinding(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSequentialNPCOutputGroundsCombatAtBothStageLocalReferences(t *testing.T) {
|
||||
registries := productionNPCRegistries(t)
|
||||
catalog := moduleCatalog(registries)
|
||||
configValue := loadSequentialCombatPipelineConfig(t)
|
||||
npcEffective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-npcs", Catalog: catalog})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve(NPC) error = %v", err)
|
||||
}
|
||||
npcClient := &fakeNPCProductionLLMClient{response: npcProductionResponse{NPCs: []npcProductionRecord{
|
||||
{Name: "Mira Thorn", Aliases: []string{"The Greencloak", "Mira"}, Description: "A ranger.", Relationships: []npcProductionRelationship{}, SourceRefs: []npcProductionSourceRef{{StartUnitID: 1, EndUnitID: 2}}},
|
||||
{Name: "Hooded Guard", Aliases: []string{}, Description: "A sentry.", Relationships: []npcProductionRelationship{}, SourceRefs: []npcProductionSourceRef{{StartUnitID: 3, EndUnitID: 3}}},
|
||||
}}}
|
||||
npcOutput, err := runPreparedPipeline(t, registries, npcEffective.ResolvedPipeline, npcClient, pipeline.RunInput{RawInput: readNPCFixture(t)})
|
||||
if err != nil || len(npcOutput.NormalizeOutputs) != 1 {
|
||||
t.Fatalf("NPC run error = %v output = %#v, want one normalized NPC lane", err, npcOutput.NormalizeOutputs)
|
||||
}
|
||||
npcPayload := npcOutput.NormalizeOutputs[0].Artifact.Content
|
||||
if _, err := npccodec.New().Decode(npcPayload); err != nil {
|
||||
t.Fatalf("Decode(NPC output) error = %v", err)
|
||||
}
|
||||
npcPath := filepath.Join(t.TempDir(), "npc-run", "lanes", "npcs.json")
|
||||
if err := os.MkdirAll(filepath.Dir(npcPath), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(npcPath, npcPayload, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
combatEffective, err := configValue.Resolve(config.ResolveInput{
|
||||
PipelineID: "dnd-combat",
|
||||
Catalog: catalog,
|
||||
ReferenceOverrides: []pipeline.ReferenceBinding{
|
||||
{Stage: pipeline.StageExtract, LaneID: "combat", SlotName: "npcs", Source: npcPath, BindingSource: contracts.ReferenceBindingSourceCLI},
|
||||
{Stage: pipeline.StageNormalize, LaneID: "combat", SlotName: "npcs", Source: npcPath, BindingSource: contracts.ReferenceBindingSourceCLI},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve(combat) error = %v", err)
|
||||
}
|
||||
materialized, warnings, err := pipeline.MaterializeReferences(combatEffective.ResolvedPipeline, catalog, pipeline.ReferenceMaterializationOptions{})
|
||||
if err != nil || len(warnings) != 0 {
|
||||
t.Fatalf("MaterializeReferences() error = %v warnings = %#v", err, warnings)
|
||||
}
|
||||
client := &fakeCombatLLMClient{responses: []string{combatTestTurnResponse("The Greencloak", "turn", "watches", "Hooded Guard", 1, 1)}}
|
||||
output, err := runPreparedPipeline(t, registries, materialized, client, pipeline.RunInput{RawInput: readNPCFixture(t)})
|
||||
if err != nil {
|
||||
t.Fatalf("Run(combat) error = %v", err)
|
||||
}
|
||||
value, err := combatcodec.New().Decode(output.NormalizeOutputs[0].Artifact.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("Decode(combat output) error = %v", err)
|
||||
}
|
||||
if len(value.CombatTurns) != 1 || value.CombatTurns[0].Actor != "Mira Thorn" || value.CombatTurns[0].Actions[0].Targets[0] != "Hooded Guard" {
|
||||
t.Fatalf("combat value = %#v, want canonical actor and target", value)
|
||||
}
|
||||
for _, ref := range value.CombatTurns[0].SourceRefs {
|
||||
if ref.SourceID != "npc-session" {
|
||||
t.Fatalf("combat source ref = %#v, want transcript evidence only", ref)
|
||||
}
|
||||
}
|
||||
if len(output.Manifest.References) != 2 {
|
||||
t.Fatalf("manifest references = %#v, want both stage-local NPC provenance entries", output.Manifest.References)
|
||||
}
|
||||
for _, provenance := range output.Manifest.References {
|
||||
if provenance.SlotName != "npcs" || provenance.LaneID != "combat" || !strings.Contains(provenance.OriginURI, "npcs.json") || provenance.BindingSource != contracts.ReferenceBindingSourceCLI {
|
||||
t.Fatalf("NPC provenance = %#v, want combat stage-local CLI binding", provenance)
|
||||
}
|
||||
}
|
||||
if len(client.requests) != 1 || string(client.requests[0].Inputs[combatextract.NPCRegistryReferenceSlot].Content) != string(npcPayload) || client.requests[0].Inputs[combatextract.NPCRegistryReferenceSlot].OriginURI != "" {
|
||||
t.Fatalf("combat NPC prompt input = %#v, want canonical payload without path provenance", client.requests)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCombatPreparationRejectsMalformedOrOversizedNPCReferencesBeforeExecution(t *testing.T) {
|
||||
registries := productionNPCRegistries(t)
|
||||
catalog := moduleCatalog(registries)
|
||||
@@ -313,19 +240,6 @@ func combatOnlyConfig() config.Config {
|
||||
return cfg
|
||||
}
|
||||
|
||||
func loadSequentialCombatPipelineConfig(t *testing.T) config.Config {
|
||||
t.Helper()
|
||||
fileConfig, err := config.LoadFileConfig(repositoryPathForIntegration("examples", "dnd-npc-combat-sequential.config.yml"))
|
||||
if err != nil {
|
||||
t.Fatalf("LoadFileConfig() error = %v", err)
|
||||
}
|
||||
cfg := config.Default()
|
||||
if err := cfg.ApplyFileConfig(fileConfig); err != nil {
|
||||
t.Fatalf("ApplyFileConfig() error = %v", err)
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
func repositoryPathForIntegration(parts ...string) string {
|
||||
_, file, _, _ := runtime.Caller(0)
|
||||
return filepath.Join(append([]string{filepath.Dir(file), "..", "..", ".."}, parts...)...)
|
||||
|
||||
222
internal/modules/integration/dnd_npc_grounded_test.go
Normal file
222
internal/modules/integration/dnd_npc_grounded_test.go
Normal file
@@ -0,0 +1,222 @@
|
||||
package integration_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
combatcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/combatturns"
|
||||
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
|
||||
combatextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/combatturns"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
)
|
||||
|
||||
func TestNPCOutputGroundsSpellAndCombatConsumersThroughOneOperation(t *testing.T) {
|
||||
registries := productionNPCRegistries(t)
|
||||
catalog := moduleCatalog(registries)
|
||||
configValue := loadGroundedPipelineConfig(t)
|
||||
effective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-npc-grounded", Catalog: catalog})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve() error = %v", err)
|
||||
}
|
||||
materialized, warnings, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalog, pipeline.ReferenceMaterializationOptions{})
|
||||
if err != nil || len(warnings) != 0 {
|
||||
t.Fatalf("MaterializeReferences() error = %v warnings = %#v", err, warnings)
|
||||
}
|
||||
|
||||
client := &groundedDNDLLMClient{}
|
||||
output, err := runPreparedPipeline(t, registries, materialized, client, pipeline.RunInput{
|
||||
RawInput: readNPCFixture(t),
|
||||
ExtractWorkers: 1,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 3 {
|
||||
t.Fatalf("run outputs = %#v rejected = %#v, want NPC, spell, and combat outputs", output.NormalizeOutputs, output.Rejected)
|
||||
}
|
||||
|
||||
var npcPayload []byte
|
||||
for _, serialized := range output.NormalizeOutputs {
|
||||
if serialized.LaneID == "npcs" {
|
||||
npcPayload = append([]byte(nil), serialized.Artifact.Content...)
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(npcPayload) == 0 {
|
||||
t.Fatal("NPC producer did not publish a canonical payload")
|
||||
}
|
||||
npcValue, err := npccodec.New().Decode(npcPayload)
|
||||
if err != nil {
|
||||
t.Fatalf("decode NPC producer payload: %v", err)
|
||||
}
|
||||
npcPayload, err = npccodec.New().Encode(npcValue)
|
||||
if err != nil {
|
||||
t.Fatalf("encode canonical NPC producer payload: %v", err)
|
||||
}
|
||||
canonicalDigest := digestBytes(npcPayload)
|
||||
seenConsumers := map[string]bool{}
|
||||
for _, request := range client.requestsSnapshot() {
|
||||
if request.PromptID != spells.PromptID && request.PromptID != combatextract.PromptID {
|
||||
continue
|
||||
}
|
||||
input := request.Inputs["npcs"]
|
||||
if input.MediaType != npccodec.MediaType || input.Digest != canonicalDigest || string(input.Content) != string(npcPayload) || input.OriginURI != "" {
|
||||
t.Fatalf("%s NPC prompt input = %#v, want canonical generated registry without provenance", request.PromptID, input)
|
||||
}
|
||||
seenConsumers[request.PromptID] = true
|
||||
}
|
||||
if !seenConsumers[spells.PromptID] || !seenConsumers[combatextract.PromptID] {
|
||||
t.Fatalf("consumer prompt IDs = %#v, want spell and combat requests", seenConsumers)
|
||||
}
|
||||
|
||||
provenanceCount := 0
|
||||
for _, reference := range output.Manifest.References {
|
||||
if reference.SlotName != "npcs" {
|
||||
continue
|
||||
}
|
||||
provenanceCount++
|
||||
if reference.Digest != canonicalDigest || reference.OriginType != "generated" {
|
||||
t.Fatalf("NPC generated provenance = %#v, want canonical digest and generated origin", reference)
|
||||
}
|
||||
}
|
||||
if provenanceCount != 3 {
|
||||
t.Fatalf("NPC generated provenance count = %d, want spell extract plus combat extract/normalize", provenanceCount)
|
||||
}
|
||||
for _, lane := range output.Manifest.ArtifactLanes {
|
||||
for _, component := range []string{"extractor", "normalizer"} {
|
||||
metadata, ok := lane.Metadata[component].(map[string]any)
|
||||
if ok && metadata["npc_registry_digest"] != nil {
|
||||
t.Fatalf("%s %s metadata = %#v, want generated identity only in framework provenance", lane.ID, component, metadata)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var combatValue dnd.CombatTurnList
|
||||
for _, serialized := range output.NormalizeOutputs {
|
||||
switch serialized.LaneID {
|
||||
case "spells":
|
||||
spellValue := decodeRunnerSpellResponse(t, serialized.Artifact.Content)
|
||||
if len(spellValue.SpellCasts) != 1 || spellValue.SpellCasts[0].Caster != "The Greencloak" {
|
||||
t.Fatalf("spell output = %#v, want one registry-grounded-context spell", spellValue)
|
||||
}
|
||||
assertSpellEvidence(t, spellValue.SpellCasts[0].SourceRefs)
|
||||
case "combat":
|
||||
decoded, decodeErr := combatcodec.New().Decode(serialized.Artifact.Content)
|
||||
if decodeErr != nil {
|
||||
t.Fatalf("decode combat output: %v", decodeErr)
|
||||
}
|
||||
combatValue = decoded
|
||||
}
|
||||
}
|
||||
if len(combatValue.CombatTurns) != 1 || combatValue.CombatTurns[0].Actor != "Mira Thorn" || combatValue.CombatTurns[0].Actions[0].Targets[0] != "Hooded Guard" {
|
||||
t.Fatalf("combat output = %#v, want registry-normalized actor and target", combatValue)
|
||||
}
|
||||
assertCurrentEvidence(t, combatValue.CombatTurns[0].SourceRefs)
|
||||
}
|
||||
|
||||
func assertSpellEvidence(t *testing.T, references []shared.SourceRefResponse) {
|
||||
t.Helper()
|
||||
for _, reference := range references {
|
||||
if reference.SourceID != "npc-session" {
|
||||
t.Fatalf("spell evidence reference = %#v, want current source only", reference)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func assertCurrentEvidence(t *testing.T, references []source.SourceRef) {
|
||||
t.Helper()
|
||||
for _, reference := range references {
|
||||
if reference.SourceID != "npc-session" {
|
||||
t.Fatalf("evidence reference = %#v, want current source only", reference)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func loadGroundedPipelineConfig(t *testing.T) config.Config {
|
||||
t.Helper()
|
||||
fileConfig, err := config.LoadFileConfig(repositoryPathForIntegration("internal", "modules", "integration", "testdata", "dnd_npc_grounded_pipeline.yml"))
|
||||
if err != nil {
|
||||
t.Fatalf("LoadFileConfig() error = %v", err)
|
||||
}
|
||||
cfg := config.Default()
|
||||
if err := cfg.ApplyFileConfig(fileConfig); err != nil {
|
||||
t.Fatalf("ApplyFileConfig() error = %v", err)
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
type groundedDNDLLMClient struct {
|
||||
mu sync.Mutex
|
||||
requests []contracts.StructuredCompletionRequest
|
||||
}
|
||||
|
||||
func (client *groundedDNDLLMClient) CompleteStructured(ctx context.Context, request contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
client.mu.Lock()
|
||||
client.requests = append(client.requests, cloneStructuredCompletionRequest(request))
|
||||
client.mu.Unlock()
|
||||
|
||||
var payload any
|
||||
switch request.PromptID {
|
||||
case npcs.PromptID:
|
||||
payload = map[string]any{"npcs": []any{
|
||||
map[string]any{
|
||||
"name": "Mira Thorn", "aliases": []string{"The Greencloak"}, "description": "A guarded ranger.",
|
||||
"relationships": []any{}, "source_refs": []any{map[string]int{"start_unit_id": 1, "end_unit_id": 1}},
|
||||
},
|
||||
map[string]any{
|
||||
"name": "Hooded Guard", "aliases": []string{}, "description": "A sentry.",
|
||||
"relationships": []any{}, "source_refs": []any{map[string]int{"start_unit_id": 3, "end_unit_id": 3}},
|
||||
},
|
||||
}}
|
||||
case spells.PromptID:
|
||||
payload = map[string]any{"spell_casts": []any{map[string]any{
|
||||
"caster": "The Greencloak", "spell": "Cure Wounds", "effect": "Restores an ally.",
|
||||
"narrative_description": "The Greencloak restores an ally.",
|
||||
"source_refs": []any{map[string]int{"start_unit_id": 1, "end_unit_id": 1}},
|
||||
}}}
|
||||
case combatextract.PromptID:
|
||||
payload = map[string]any{"combat_turns": []any{map[string]any{
|
||||
"actor": "The Greencloak", "turn_kind": "turn", "round": 1,
|
||||
"actions": []any{map[string]any{"category": "attack", "declaration": "watches", "targets": []string{"Hooded Guard"}, "resolution": "observed"}},
|
||||
"summary": "The Greencloak watches the gate.",
|
||||
"source_refs": []any{map[string]int{"start_unit_id": 1, "end_unit_id": 1}},
|
||||
}}}
|
||||
default:
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("unexpected grounded prompt %q", request.PromptID)
|
||||
}
|
||||
content, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
if err := json.Unmarshal(content, out); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate grounded response: %w", err)
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{Content: content, Provider: "test", Model: "grounded-fake"}, nil
|
||||
}
|
||||
|
||||
func (client *groundedDNDLLMClient) requestsSnapshot() []contracts.StructuredCompletionRequest {
|
||||
client.mu.Lock()
|
||||
defer client.mu.Unlock()
|
||||
return append([]contracts.StructuredCompletionRequest(nil), client.requests...)
|
||||
}
|
||||
|
||||
func digestBytes(content []byte) string {
|
||||
sum := sha256.Sum256(content)
|
||||
return fmt.Sprintf("sha256:%x", sum[:])
|
||||
}
|
||||
|
||||
var _ contracts.StructuredLLMClient = (*groundedDNDLLMClient)(nil)
|
||||
@@ -1,130 +0,0 @@
|
||||
package integration_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
||||
)
|
||||
|
||||
func TestSequentialNPCOutputCanGroundIndependentSpellRun(t *testing.T) {
|
||||
registries := productionNPCRegistries(t)
|
||||
catalog := moduleCatalog(registries)
|
||||
configValue := loadSequentialPipelineConfig(t)
|
||||
|
||||
npcEffective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-npcs", Catalog: catalog})
|
||||
if err != nil {
|
||||
t.Fatalf("resolve NPC pipeline: %v", err)
|
||||
}
|
||||
npcClient := &fakeNPCProductionLLMClient{response: npcProductionResponse{NPCs: []npcProductionRecord{{
|
||||
Name: "Mira Thorn",
|
||||
Aliases: []string{"The Greencloak"},
|
||||
Description: "A guarded ranger who watches the northern road.",
|
||||
Relationships: []npcProductionRelationship{{
|
||||
Target: "Captain Vale", Relationship: "reports to",
|
||||
}},
|
||||
SourceRefs: []npcProductionSourceRef{{StartUnitID: 1, EndUnitID: 2}},
|
||||
}}}}
|
||||
npcOutput, err := runPreparedPipeline(t, registries, npcEffective.ResolvedPipeline, npcClient, pipeline.RunInput{RawInput: readNPCFixture(t)})
|
||||
if err != nil {
|
||||
t.Fatalf("run NPC pipeline: %v", err)
|
||||
}
|
||||
if len(npcOutput.NormalizeOutputs) != 1 || npcOutput.NormalizeOutputs[0].LaneID != "npcs" {
|
||||
t.Fatalf("NPC normalized outputs = %#v, want one npcs lane", npcOutput.NormalizeOutputs)
|
||||
}
|
||||
npcPayload := npcOutput.NormalizeOutputs[0].Artifact.Content
|
||||
if _, err := npccodec.New().Decode(npcPayload); err != nil {
|
||||
t.Fatalf("decode normalized NPC payload: %v", err)
|
||||
}
|
||||
|
||||
npcRunDir := t.TempDir()
|
||||
npcPath := filepath.Join(npcRunDir, "lanes", "npcs.json")
|
||||
if err := os.MkdirAll(filepath.Dir(npcPath), 0o700); err != nil {
|
||||
t.Fatalf("create NPC output directory: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(npcPath, npcPayload, 0o600); err != nil {
|
||||
t.Fatalf("write NPC output payload: %v", err)
|
||||
}
|
||||
|
||||
spellEffective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-spells", Catalog: catalog})
|
||||
if err != nil {
|
||||
t.Fatalf("resolve spell pipeline: %v", err)
|
||||
}
|
||||
spellEffective.ResolvedPipeline.ArtifactLanes[0].ExtractReferences.Bindings = []pipeline.ReferenceBinding{{
|
||||
Stage: pipeline.StageExtract,
|
||||
LaneID: "spells",
|
||||
SlotName: spells.NPCRegistryReferenceSlot,
|
||||
Source: npcPath,
|
||||
BindingSource: contracts.ReferenceBindingSourceCLI,
|
||||
}}
|
||||
materialized, warnings, err := pipeline.MaterializeReferences(spellEffective.ResolvedPipeline, catalog, pipeline.ReferenceMaterializationOptions{WorkingDir: npcRunDir})
|
||||
if err != nil {
|
||||
t.Fatalf("materialize NPC registry reference: %v", err)
|
||||
}
|
||||
if len(warnings) != 0 {
|
||||
t.Fatalf("reference materialization warnings = %#v, want none", warnings)
|
||||
}
|
||||
|
||||
spellClient := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{{
|
||||
Caster: "Mira Thorn",
|
||||
Spell: "Cure Wounds",
|
||||
Effect: "Restores the injured ally.",
|
||||
NarrativeDescription: "Mira Thorn restores the ally after the fight.",
|
||||
SourceRefs: responseSourceRefs("spell-session", 1, 1),
|
||||
}}}}
|
||||
spellOutput, err := runPreparedPipeline(t, registries, materialized, spellClient, pipeline.RunInput{RawInput: readDNDSpellsFixture(t)})
|
||||
if err != nil {
|
||||
t.Fatalf("run spell pipeline: %v", err)
|
||||
}
|
||||
if len(spellOutput.NormalizeOutputs) != 1 || spellOutput.NormalizeOutputs[0].LaneID != "spells" {
|
||||
t.Fatalf("spell normalized outputs = %#v, want one spells lane", spellOutput.NormalizeOutputs)
|
||||
}
|
||||
spellValue := decodeRunnerSpellResponse(t, spellOutput.NormalizeOutputs[0].Artifact.Content)
|
||||
if len(spellValue.SpellCasts) != 1 || spellValue.SpellCasts[0].Caster != "Mira Thorn" {
|
||||
t.Fatalf("spell output = %#v, want one registry-grounded caster", spellValue)
|
||||
}
|
||||
if len(spellValue.SpellCasts[0].SourceRefs) != 1 || spellValue.SpellCasts[0].SourceRefs[0].SourceID != "spell-session" {
|
||||
t.Fatalf("spell source refs = %#v, want current spell session only", spellValue.SpellCasts[0].SourceRefs)
|
||||
}
|
||||
if len(spellClient.requests) != 1 {
|
||||
t.Fatalf("spell LLM requests = %d, want one", len(spellClient.requests))
|
||||
}
|
||||
registryInput := spellClient.requests[0].Inputs[spells.NPCRegistryReferenceSlot]
|
||||
if string(registryInput.Content) != string(npcPayload) || registryInput.MediaType != npccodec.MediaType || registryInput.OriginURI != "" {
|
||||
t.Fatalf("spell NPC prompt input = %#v, want canonical payload without origin", registryInput)
|
||||
}
|
||||
if len(spellOutput.Manifest.References) != 1 {
|
||||
t.Fatalf("spell manifest references = %#v, want one NPC provenance entry", spellOutput.Manifest.References)
|
||||
}
|
||||
provenance := spellOutput.Manifest.References[0]
|
||||
if provenance.Stage != "extract" || provenance.LaneID != "spells" || provenance.SlotName != spells.NPCRegistryReferenceSlot || provenance.BindingSource != contracts.ReferenceBindingSourceCLI || !strings.Contains(provenance.OriginURI, "npcs.json") {
|
||||
t.Fatalf("spell NPC provenance = %#v, want extract CLI reference provenance", provenance)
|
||||
}
|
||||
metadata, ok := spellOutput.Manifest.ArtifactLanes[0].Metadata["extractor"].(map[string]any)
|
||||
if !ok || metadata["npc_count"] != 1 || metadata["npc_registry_digest"] != registryInput.Digest {
|
||||
t.Fatalf("spell extractor metadata = %#v, want NPC count and semantic digest", spellOutput.Manifest.ArtifactLanes[0].Metadata)
|
||||
}
|
||||
}
|
||||
|
||||
func loadSequentialPipelineConfig(t *testing.T) config.Config {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile("testdata/dnd_npc_spell_sequential_pipeline.yml")
|
||||
if err != nil {
|
||||
t.Fatalf("read sequential pipeline config: %v", err)
|
||||
}
|
||||
fileConfig, err := config.ParseFileConfigYAML(data)
|
||||
if err != nil {
|
||||
t.Fatalf("parse sequential pipeline config: %v", err)
|
||||
}
|
||||
configValue := config.Default()
|
||||
if err := configValue.ApplyFileConfig(fileConfig); err != nil {
|
||||
t.Fatalf("apply sequential pipeline config: %v", err)
|
||||
}
|
||||
return configValue
|
||||
}
|
||||
@@ -118,7 +118,7 @@ func TestRunnerPassesPartyAndGlossaryReferencesToDNDSpellsPrompt(t *testing.T) {
|
||||
raw := readDNDSpellsFixture(t)
|
||||
expectedDoc := parseDNDSpellsFixture(t, raw)
|
||||
resolved := resolveDNDSpellsPipeline(t)
|
||||
resolved.ResolvedPipeline.ArtifactLanes[0].ExtractReferences.ReferenceSet = dndSpellsReferenceSet(
|
||||
resolved.ResolvedPipeline.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet = dndSpellsReferenceSet(
|
||||
"Aria: party cleric\nBorin: fighter",
|
||||
"Fire Bolt: evocation cantrip",
|
||||
)
|
||||
|
||||
31
internal/modules/integration/testdata/dnd_npc_grounded_pipeline.yml
vendored
Normal file
31
internal/modules/integration/testdata/dnd_npc_grounded_pipeline.yml
vendored
Normal file
@@ -0,0 +1,31 @@
|
||||
version: 3
|
||||
output:
|
||||
directory: ./notarius-output
|
||||
cache:
|
||||
chunk_plans:
|
||||
mode: bypass
|
||||
checkpoints: {}
|
||||
debug:
|
||||
directory: ./notarius-debug
|
||||
pipelines:
|
||||
dnd-npc-grounded:
|
||||
input: seriatim
|
||||
steps:
|
||||
- id: identify-npcs
|
||||
artifacts:
|
||||
npcs:
|
||||
extract: dnd/npcs
|
||||
normalize: dnd/npcs
|
||||
- id: grounded-events
|
||||
references:
|
||||
npcs:
|
||||
artifact:
|
||||
step: identify-npcs
|
||||
lane: npcs
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
normalize: dnd/spells
|
||||
combat:
|
||||
extract: dnd/combat-turns
|
||||
normalize: dnd/combat-turns
|
||||
@@ -1,25 +0,0 @@
|
||||
version: 3
|
||||
output:
|
||||
directory: ./notarius-output
|
||||
cache:
|
||||
chunk_plans:
|
||||
mode: bypass
|
||||
checkpoints: {}
|
||||
debug:
|
||||
directory: ./notarius-debug
|
||||
pipelines:
|
||||
dnd-npcs:
|
||||
input: seriatim
|
||||
chunk: generic
|
||||
artifacts:
|
||||
npcs:
|
||||
extract:
|
||||
module: dnd/npcs
|
||||
normalize: dnd/npcs
|
||||
dnd-spells:
|
||||
input: seriatim
|
||||
chunk: generic
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
normalize: dnd/spells
|
||||
Reference in New Issue
Block a user