Compare commits

...

13 Commits

95 changed files with 6375 additions and 2050 deletions

View File

@@ -30,4 +30,5 @@ Useful references:
- [Developer guide](docs/development.md) - [Developer guide](docs/development.md)
- [Internal implementation docs](docs/internal/overview.md) - [Internal implementation docs](docs/internal/overview.md)
- [Maintained example config](examples/dnd-spells.config.yml) - [Maintained example config](examples/dnd-spells.config.yml)
- [NPC-grounded example config](examples/dnd-npc-grounded.config.yml)
- [Maintained example input](examples/seriatim-minimal-transcript.json) - [Maintained example input](examples/seriatim-minimal-transcript.json)

View File

@@ -0,0 +1,50 @@
# ADR-0008: Bounded ordered pipeline steps and explicit artifact references
**Status:** Accepted
**Date:** 2026-07-21
## Context
Notarius currently models one pipeline-wide input, chunking plan, artifact
lanes, and output boundary. Some workflows need a deterministic handoff from
one set of normalized artifacts to a later set of artifacts, such as using
extracted NPC records while grounding later combat events. The workflow needs
an explicit topology without turning the pipeline into a general-purpose
workflow engine.
## Decision
Add an ordered collection of pipeline steps. Each step owns one or more
artifact lanes, and lanes within a step retain the existing independent
execution model. The pipeline continues to have one input, chunk plan, output,
and failure boundary. Steps are barriers: a later step may consume only
normalized artifacts from an earlier step.
Generated references use an explicit step-and-lane selector. Reference slots
declare the generated artifact kinds and media types they accept. The resolver
validates the topology, ordering, lane identity, artifact kind, schema, and
codec compatibility before execution. External references remain supported as
path sources, and the legacy top-level artifact map is interpreted as an
implicit `default` step.
Pipeline-level references may not select generated artifacts. General DAGs,
branches, loops, conditional execution, joins, and inferred dependencies are
not part of this model.
## Alternatives considered
- A general DAG would provide more flexibility but would also require a new
scheduler, lifecycle model, failure semantics, and provenance model.
- Separate pipeline runs connected through filesystem paths would lose the
static topology and typed compatibility checks.
- Inferring dependencies from module or lane names would make ordering and
configuration errors difficult to detect reliably.
## Consequences
The resolved pipeline has a deterministic, inspectable topology and can
include it in its identity digest. Configuration validation can reject invalid
generated bindings before any work begins. Existing single-step profiles keep
their behavior through the implicit `default` step. Execution handoff and
multi-step scheduling require follow-up work in the runner and checkpoint
layers.

View File

@@ -9,7 +9,7 @@ For the minimal end-to-end invocation, see the [README](../README.md).
```text ```text
notarius help notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--chunk_cache auto|bypass|refresh] [--output-dir path] [--resume] [--debug [--debug-dir path]] [--llm-profile id] [--session-id id] [--reference selector=path] [--without-reference selector] notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--chunk_cache auto|bypass|refresh] [--output-dir path] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--llm-profile id] [--session-id id] [--reference selector=path] [--without-reference selector]
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b] notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list [--config path/to/config.yml] [--json] notarius pipelines list [--config path/to/config.yml] [--json]
``` ```
@@ -28,11 +28,18 @@ Flags:
- `--config path`: config file path. If omitted, Notarius uses the discovery - `--config path`: config file path. If omitted, Notarius uses the discovery
rules in [Configuration](config.md#discovery). rules in [Configuration](config.md#discovery).
- `--only lane-a,lane-b`: run only the named artifact lanes. Values are - `--only lane-a,lane-b`: run only the named artifact lanes. Values are
comma-separated and must be non-empty. comma-separated and must be non-empty. This retains its existing behavior for
implicit single-step pipelines; explicit multi-step pipelines reject it
rather than inferring dependency closure.
- `--resume`: request checkpoint reuse for this invocation. Checkpoint recording - `--resume`: request checkpoint reuse for this invocation. Checkpoint recording
must be enabled in configuration. See must be enabled in configuration. See
[Operations](operations.md#checkpoint-cache) for prerequisites and reuse [Operations](operations.md#checkpoint-cache) for prerequisites and reuse
behavior. behavior.
- `--recompute-step step-id`: with `--resume` and checkpoint recording enabled,
force the named ordered step and every transitive dependent lane to execute.
Compatible required predecessors and unrelated lanes remain reusable. The
value may identify an explicit step or the implicit single-step ID `default`;
it cannot be combined with `--only`.
- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this - `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
invocation. `auto` reuses a valid plan by canonical source digest, `bypass` invocation. `auto` reuses a valid plan by canonical source digest, `bypass`
performs no plan-cache I/O, and `refresh` regenerates and replaces a valid performs no plan-cache I/O, and `refresh` regenerates and replaces a valid
@@ -58,8 +65,10 @@ rejected output counts, and the output directory. A debug-enabled run also
prints `debug=<bundle-path>`. If the run completes with warnings, the warning prints `debug=<bundle-path>`. If the run completes with warnings, the warning
count is printed to stderr. count is printed to stderr.
Reference flags are resolved against selected chunk, extractor, merger, and Reference flags are external file bindings resolved against selected chunk,
normalizer targets before the run starts. Flat slot names are accepted only extractor, merger, and normalizer targets before the run starts. Generated
artifact bindings are configured in ordered steps and cannot be introduced by a
CLI path flag. Flat slot names are accepted only
when exactly one selected target declares that slot. For configured reference when exactly one selected target declares that slot. For configured reference
bindings, precedence, path resolution, and validation, see bindings, precedence, path resolution, and validation, see
[Configuration](config.md#pipelines). [Configuration](config.md#pipelines).
@@ -94,30 +103,21 @@ go run ./cmd/notarius run dnd-session \
--reference spells.extract.glossary=./campaign-glossary.txt --reference spells.extract.glossary=./campaign-glossary.txt
``` ```
For the operator-driven NPC-to-spell workflow, bind the normalized NPC lane For the maintained NPC-grounded workflow, use the explicit ordered pipeline.
payload from the completed NPC run to the spell extractor: The first step produces the normalized NPC artifact; the second step receives
it in memory and fans it out to spell extraction, combat extraction, and combat
normalization:
```sh ```sh
go run ./cmd/notarius run dnd-spells \ go run ./cmd/notarius run dnd-npc-grounded \
--config examples/dnd-npc-spell-sequential.config.yml \ --config examples/dnd-npc-grounded.config.yml \
--input examples/seriatim-minimal-transcript.json \ --input examples/seriatim-minimal-transcript.json \
--reference spells.extract.npcs=./npc-run/lanes/npcs.json --output-dir ./npc-grounded-output
``` ```
For the independent NPC-to-combat workflow, bind the same completed NPC lane The generated NPC content remains contextual grounding, not spell or combat
to both combat stages explicitly: evidence. It is represented in manifests and debug summaries by bounded
identity and producer provenance, not by payload content or a filesystem path.
```sh
go run ./cmd/notarius run dnd-combat \
--config examples/dnd-npc-combat-sequential.config.yml \
--input examples/seriatim-minimal-transcript.json \
--reference combat.extract.npcs=./npc-run/lanes/npcs.json \
--reference combat.normalize.npcs=./npc-run/lanes/npcs.json
```
The two selectors are independent stage-local bindings. Binding extraction
does not implicitly bind normalization, and Notarius does not discover or
schedule the preceding NPC run.
The same grammar can target chunk, merge, and normalize slots when the configured The same grammar can target chunk, merge, and normalize slots when the configured
modules declare them: modules declare them:
@@ -161,6 +161,25 @@ go run ./cmd/notarius run dnd-session \
--resume --resume
``` ```
To selectively rerun one ordered step and its dependent lanes, use the step ID
from the configuration. The selected step and dependents are reported as
`forced_recompute`; reusable predecessors are reported as `reused`:
```sh
go run ./cmd/notarius run dnd-npc-grounded \
--config examples/dnd-npc-grounded.config.yml \
--input examples/seriatim-minimal-transcript.json \
--resume --recompute-step grounded-events
```
Checkpoint decisions use these categories: `reused`, `executed`,
`forced_recompute`, and `dependency_invalidated`. The reason code and bounded
detail identify the decision without exposing reference content, local paths,
or secrets. `--recompute-step` requires checkpoint recording and `--resume`;
unknown step IDs, empty values, and combinations with `--only` are rejected.
The operator meanings of checkpoint reason codes are maintained in
[Operations](operations.md#resume-and-selective-recompute).
Use `--debug` to retain the redacted summary and trace bundle for one run. The Use `--debug` to retain the redacted summary and trace bundle for one run. The
bundle is allocated before pipeline resolution; once allocated, its path is bundle is allocated before pipeline resolution; once allocated, its path is
also printed to stderr if the command fails. Debug-write failures cause exit also printed to stderr if the command fails. Debug-write failures cause exit

View File

@@ -22,9 +22,8 @@ The explicit-path option is defined in the [CLI reference](cli.md).
- [Minimal D&D spell configuration](../examples/dnd-spells.config.yml) - [Minimal D&D spell configuration](../examples/dnd-spells.config.yml)
- [Production-oriented D&D spell configuration](../examples/dnd-spells-production.config.yml) - [Production-oriented D&D spell configuration](../examples/dnd-spells-production.config.yml)
- [D&D NPC configuration](../examples/dnd-npcs.config.yml) - [D&D NPC configuration](../examples/dnd-npcs.config.yml)
- [Sequential D&D NPC and spell configuration](../examples/dnd-npc-spell-sequential.config.yml)
- [D&D combat-turn configuration](../examples/dnd-combat-turns.config.yml) - [D&D combat-turn configuration](../examples/dnd-combat-turns.config.yml)
- [Sequential D&D NPC and combat-turn configuration](../examples/dnd-npc-combat-sequential.config.yml) - [D&D NPC-grounded spell and combat configuration](../examples/dnd-npc-grounded.config.yml)
All are complete version 3 files. The fragments below illustrate individual All are complete version 3 files. The fragments below illustrate individual
fields and are not alternate complete configurations. fields and are not alternate complete configurations.
@@ -140,13 +139,64 @@ Pipeline fields:
- `input`: required module binding. - `input`: required module binding.
- `chunk`: optional module binding. Default module is `generic`. - `chunk`: optional module binding. Default module is `generic`.
- `artifacts`: required for pipeline resolution. It maps artifact lane IDs to - `artifacts`: the artifact lane map for a single-step pipeline. It is treated
lane definitions. as an implicit step with the stable ID `default`.
- `steps`: an ordered, non-empty list of step definitions. A pipeline may use
`steps` or `artifacts`, but not both. Step IDs must be unique after trimming.
- `output`: optional module binding. Default module is `json`. - `output`: optional module binding. Default module is `json`.
- `references`: optional map of reference slot names to reference paths. These - `references`: optional map of reference slot names to reference paths. These
bindings are defaults for eligible pipeline targets that declare the matching bindings are defaults for eligible pipeline targets that declare the matching
slot. slot.
Each explicit step contains an `id`, an optional `references` map, and a
non-empty `artifacts` map. Steps share the pipeline input, chunk plan, worker
budget, output encoder, manifest, and failure boundary. Lanes within a step
retain the fixed extract, validate, merge, validate, normalize, and validate
workflow; the next step starts only after the current step is terminal.
Generated references use the structured `artifact` source form to identify one
accepted normalized lane from an earlier step:
```yaml
pipelines:
dnd-npc-grounded:
input: seriatim
steps:
- id: identify-npcs
artifacts:
npcs:
extract: dnd/npcs
normalize: dnd/npcs
- id: grounded-events
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells
combat:
extract: dnd/combat-turns
normalize: dnd/combat-turns
```
The generated binding is explicit and typed; it is not inferred from module,
lane, or slot names. It may be declared at step scope, applying to every
selected target in that step that declares the slot, or at one target's
`references` map. A producer may fan out to compatible target slots, but a
slot accepts only one producer. A producer must be in an earlier step, and a
configured generated dependency is required even when the consumer slot is
otherwise optional. Aggregating several producer artifacts is unsupported.
The producer codec supplies the artifact kind, complete schema identity, media
type, canonical content digest, and size used for compatibility and checkpoint
dependency checks. Only one accepted normalized artifact crosses the boundary;
raw extraction results, rejected output, intermediate values, and validator
diagnostics do not. Generated content is supplied in memory and is never
represented by a filesystem path.
Artifact lane fields: Artifact lane fields:
- `extract`: required module binding. - `extract`: required module binding.
@@ -164,19 +214,23 @@ See [CLI Reference](cli.md) for command syntax.
Reference bindings are validated against reference slots declared by eligible Reference bindings are validated against reference slots declared by eligible
chunk, extract, merge, and normalize targets during pipeline resolution. Required slots chunk, extract, merge, and normalize targets during pipeline resolution. Required slots
must be bound after config defaults, target-local references, lane-level must be bound after config defaults, target-local references, lane-level
compatibility bindings, and command-line reference overrides are applied. compatibility bindings, step-local references, and command-line reference
Config-relative paths are resolved relative to the config file; command-line overrides are applied. Config-relative paths are resolved relative to the
reference paths are resolved relative to the current working directory. Bound config file; command-line reference paths are resolved relative to the current
files must be UTF-8 text. Reference media types are inferred from file working directory. Bound files must be UTF-8 text. Reference media types are
extensions and checked when a module restricts accepted types; unknown inferred from file extensions and checked when a module restricts accepted
extensions use `application/octet-stream`. See [CLI Reference](cli.md#run) for types; unknown extensions use `application/octet-stream`. See [CLI Reference](cli.md#run)
command-line selectors and [Operations](operations.md) for recorded provenance for command-line selectors and [Operations](operations.md) for recorded
and sensitive-data handling. provenance and sensitive-data handling.
Pipeline-level `references` are defaults. They are valid when at least one Pipeline-level `references` are defaults. They are valid when at least one
eligible target in the full configured pipeline declares the slot, including eligible target in the full configured pipeline declares the slot, including
chunk, extractor, merger, and normalizer targets. During a run, they apply only chunk, extractor, merger, and normalizer targets. During a run, they apply only
to the selected targets that declare the slot: to the selected targets that declare the slot. For external bindings, a
step-local binding overrides a pipeline-level default and a target-local
binding retains the existing most-specific precedence. Generated and external
bindings may not resolve to the same target slot, and a step-scoped generated
binding may not be duplicated by a target-local generated binding.
```yaml ```yaml
pipelines: pipelines:
@@ -226,6 +280,18 @@ Target-local reference fields use the same map shape at:
Each binding is valid only when that target module declares the slot. Each binding is valid only when that target module declares the slot.
Reference source forms are:
- a scalar string, which is an external file path; or
- an object with only `artifact`, containing trimmed `step` and `lane` IDs for
an earlier producer lane.
Pipeline-level references accept only external paths. Generated references are
valid at step scope or on a target-local `references` map. Their producer's
registered codec is authoritative for schema, media type, and canonical
content identity; an external file is not treated as generated merely because
its bytes decode as the same artifact.
## Module Bindings ## Module Bindings
Every module binding may use shorthand: Every module binding may use shorthand:
@@ -400,15 +466,17 @@ The extractor uses campaign references only as supporting disambiguation
material; spell casts still must be present in the source transcript. material; spell casts still must be present in the source transcript.
It also declares an optional `npcs` slot for a normalized NPC artifact. The It also declares an optional `npcs` slot for a normalized NPC artifact. The
slot accepts exactly one `application/json` file no larger than 1 MiB. During slot accepts exactly one `application/json` artifact no larger than 1 MiB. An
extractor preparation Notarius strictly decodes and identity-validates the external file is decoded and identity-validated during preparation. A
artifact, then gives the model canonical JSON for caster-name grounding. generated binding is validated at the step handoff and is provided to the
Registry source references may belong to the NPC-producing session and are operation through the same reference contract. In both cases, the model
provenance only; they are not spell evidence. The bound registry contributes a receives canonical JSON for caster-name grounding. Registry source references
semantic digest and NPC count to extractor metadata and checkpoint identity, may belong to the NPC-producing session and are provenance only; they are not
while its names, aliases, content, and path do not appear there. When absent, spell evidence. Generated reference identity and bounded producer provenance
the prompt receives the exact empty value `{"npcs":[]}` and no registry are recorded by the framework; NPC names, aliases, content, and paths are not
provenance or fingerprint is recorded. copied into manifests or checkpoint decisions. When absent, the prompt receives
the exact empty value `{"npcs":[]}` and no registry provenance or fingerprint
is recorded.
The `dnd/spells` normalizer declares the same optional `spell_catalog` slot. The `dnd/spells` normalizer declares the same optional `spell_catalog` slot.
When an overlay is used, bind it independently under When an overlay is used, bind it independently under
@@ -419,29 +487,34 @@ bound.
The `dnd/npcs` extractor declares the same optional campaign slots as the spell The `dnd/npcs` extractor declares the same optional campaign slots as the spell
extractor, but it does not declare the `npcs` registry slot. Its normalizer extractor, but it does not declare the `npcs` registry slot. Its normalizer
accepts no references. To pass an NPC result to a later spell run, bind the accepts no references. The maintained
normalized payload explicitly at runtime; the maintained sequential example [NPC-grounded example](../examples/dnd-npc-grounded.config.yml) binds its
documents that operator workflow. accepted normalized output to the later spell and combat targets through an
explicit ordered step.
The `dnd/combat-turns` extractor declares the optional campaign slots and the The `dnd/combat-turns` extractor declares the optional campaign slots and the
structured `npcs` slot. Campaign references guide only the LLM extraction structured `npcs` slot. Campaign references guide only the LLM extraction
stage. The deterministic normalizer declares only `npcs`, whose prepared stage. The deterministic normalizer declares only `npcs`, whose operation-time
immutable registry supports the same actor and target canonicalization. Each registry supports the same actor and target canonicalization. Each `npcs` slot
`npcs` slot accepts exactly one UTF-8 `application/json` file no larger than 1 accepts exactly one UTF-8 `application/json` artifact no larger than 1 MiB. The
MiB. The registry's source ranges remain provenance for the reference and never registry's source ranges remain provenance for the reference and never become
become combat evidence. Binding `npcs` to extraction and normalization is combat evidence. An ordered step binding fans the same generated NPC artifact
stage-local, so an operator-driven combat run uses two explicit selectors: out to extraction and normalization:
```text ```yaml
combat.extract.npcs=<npc-run>/lanes/npcs.json references:
combat.normalize.npcs=<npc-run>/lanes/npcs.json npcs:
artifact:
step: identify-npcs
lane: npcs
``` ```
When bound, the combat extractor and normalizer record only the registry's When bound, the combat extractor and normalizer receive the generated registry
semantic digest and count in their metadata and checkpoint fingerprints; names, at operation time. Framework provenance and checkpoint dependencies contain its
aliases, content, and paths are not recorded there. When absent, the combat kind, schema identity, media type, canonical digest, size, and bounded producer
prompt receives the exact empty registry value `{"npcs":[]}` and no registry identity; names, aliases, content, and paths are not recorded there. When
provenance or fingerprint is recorded. absent, the combat prompt receives the exact empty registry value
`{"npcs":[]}` and no registry provenance or fingerprint is recorded.
## State Surfaces ## State Surfaces
@@ -572,8 +645,11 @@ Configuration validation checks:
Pipeline resolution additionally checks: Pipeline resolution additionally checks:
- the pipeline ID exists; - the pipeline ID exists;
- at least one artifact lane is declared and selected; - at least one artifact lane is declared and selected in each explicit step;
- `artifacts` and `steps` are mutually exclusive, explicit steps are non-empty,
and step IDs are unique after trimming;
- lanes selected through the CLI exist in the resolved pipeline; - lanes selected through the CLI exist in the resolved pipeline;
- lane IDs are globally unique across ordered steps;
- required module keys are present; - required module keys are present;
- module keys are registered for the expected slot; - module keys are registered for the expected slot;
- module capability requirements are satisfied; - module capability requirements are satisfied;
@@ -583,4 +659,7 @@ Pipeline resolution additionally checks:
Scriptorium profile IDs; Scriptorium profile IDs;
- bound reference slots are declared by selected chunk, extractor, merger, or - bound reference slots are declared by selected chunk, extractor, merger, or
normalizer targets; normalizer targets;
- generated references identify one lane in an earlier step, use a declared
compatible artifact kind, and do not conflict with external or target-local
generated bindings;
- required reference slots are bound for selected targets. - required reference slots are bound for selected targets.

View File

@@ -92,6 +92,9 @@ chunk-scoped transcript plus the existing `players`, `party`, and `glossary`
inputs, and optionally the deprecated `roster` reference through the shared inputs, and optionally the deprecated `roster` reference through the shared
party mapping. The optional `npcs` reference is an approved normalized NPC party mapping. The optional `npcs` reference is an approved normalized NPC
artifact used only for identity grounding; it never supplies combat evidence. artifact used only for identity grounding; it never supplies combat evidence.
An external file is validated during preparation. In an ordered pipeline, the
same slot may receive the producer's canonical generated artifact at the step
handoff.
The private response envelope has the same fields and JSON types as the durable The private response envelope has the same fields and JSON types as the durable
turn/action shape except that source references contain only `start_unit_id` turn/action shape except that source references contain only `start_unit_id`
@@ -130,8 +133,10 @@ schema, combat shape, source references, then source relatedness.
The standalone normalizer uses key `dnd/combat-turns`, requires `merged`, The standalone normalizer uses key `dnd/combat-turns`, requires `merged`,
provides `normalized`, accepts no options, and accepts only the optional provides `normalized`, accepts no options, and accepts only the optional
structured `npcs` reference. Campaign references are LLM extraction context and structured `npcs` reference. Campaign references are LLM extraction context and
are not normalizer inputs. The NPC registry is resolved during preparation; are not normalizer inputs. For an external file, the NPC registry is resolved
runtime normalization uses that immutable prepared view. during preparation; for a generated binding, it is resolved at the operation-
time handoff. Runtime normalization uses that immutable prepared or handed-off
view.
Normalization policy is `dnd.combat_turns.normalize.v1`. It display-normalizes Normalization policy is `dnd.combat_turns.normalize.v1`. It display-normalizes
actor, summary, declarations, targets, and non-null resolutions; canonicalizes actor, summary, declarations, targets, and non-null resolutions; canonicalizes
@@ -143,9 +148,11 @@ merging its actions or prose. Invalid evidence is never eligible for duplicate
collapse. Every mutation and collapse emits a bounded warning using the merged collapse. Every mutation and collapse emits a bounded warning using the merged
input index in its scope. input index in its scope.
The normalizer reports `normalization_policy` and `identity_policy` metadata and The normalizer reports `normalization_policy` and `identity_policy` metadata
fingerprints, plus `npc_registry_digest`, `npc_count`, and `npc_registry` only and fingerprints. An external registry may additionally contribute
when a registry is bound. The normalized-invariants validator is `npc_registry_digest` and `npc_count`; generated registry identity is retained
in framework handoff provenance and dependency fingerprints. The
normalized-invariants validator is
`normalize/dnd/combat-turns/invariants`; it defers shape and source-reference `normalize/dnd/combat-turns/invariants`; it defers shape and source-reference
failures, then checks display normalization, target identity uniqueness, failures, then checks display normalization, target identity uniqueness,
canonical evidence ordering, chronology, and duplicate identity. It rejects canonical evidence ordering, chronology, and duplicate identity. It rejects
@@ -162,9 +169,10 @@ merge validator chain.
The selectable lane uses extractor and normalizer key `dnd/combat-turns`, The selectable lane uses extractor and normalizer key `dnd/combat-turns`,
`appendorder` for the typed merger, and the durable codec above. A bound `npcs` `appendorder` for the typed merger, and the durable codec above. A bound `npcs`
reference contributes raw-file provenance to the run manifest. Prepared combat reference contributes raw-file provenance to the run manifest. A generated
extractor and normalizer metadata and checkpoint fingerprints contain only the binding contributes artifact kind, schema identity, media type, canonical
NPC registry's semantic digest and count; the registry content, path, and NPC digest, size, and bounded producer provenance. Consumer metadata and checkpoint
source ranges are not copied into combat output. The normalized lane is emitted fingerprints contain no registry names, aliases, content, paths, or NPC source
as `lanes/<lane-id>.json` by the JSON output module, and warnings and rejection ranges. The normalized lane is emitted as `lanes/<lane-id>.json` by the JSON
summaries remain in their shared companion files. output module, and warnings and rejection summaries remain in their shared
companion files.

View File

@@ -85,23 +85,43 @@ checks. Relatedness emits bounded warnings when an NPC canonical name or
alias is not present near its cited transcript text; opaque campaign alias is not present near its cited transcript text; opaque campaign
references may explain such a warning but do not become evidence. references may explain such a warning but do not become evidence.
## Manifest And Sequential Consumption ## Manifest And Artifact Handoff
The NPC extractor records prompt and response-schema identities. The durable The NPC extractor records prompt and response-schema identities. The durable
codec records only `npc_count`; raw names, aliases, descriptions, source codec records only `npc_count`; raw names, aliases, descriptions, source
references, and payload bytes stay in the lane file rather than manifest references, and payload bytes stay in the lane file rather than manifest
metadata. The normalized lane is independently reusable as a file reference: metadata. The normalized lane can be consumed by a later ordered step through
the registered canonical codec:
```sh ```yaml
go run ./cmd/notarius run dnd-spells \ steps:
--config examples/dnd-npc-spell-sequential.config.yml \ - id: identify-npcs
--input examples/seriatim-minimal-transcript.json \ artifacts:
--reference spells.extract.npcs=./npc-output/<run-id>/lanes/npcs.json npcs:
extract: dnd/npcs
normalize: dnd/npcs
- id: grounded-events
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells
combat:
extract: dnd/combat-turns
normalize: dnd/combat-turns
``` ```
The spell extractor strictly decodes and identity-validates this file, accepts The framework hands only an accepted normalized artifact across the barrier. It
source references belonging to another session as registry provenance, and validates the canonical bytes against each consumer slot and clones the
uses only canonical names and aliases for caster grounding. Those NPC source operation-time reference for the spell and combat consumers. Generated
references are never accepted as spell evidence. The spell run's manifest provenance records the artifact kind, schema identity, media type, canonical
keeps raw file provenance under `references` and records only the prepared digest, size, and producer step/lane/module, but not names, aliases, source
registry's semantic digest and count in extractor metadata. ranges, or payload bytes. External normalized files remain supported as
explicit references and retain their file provenance.
NPC source references are registry provenance and are never accepted as spell
or combat evidence. Current transcript units remain the only event evidence.

View File

@@ -97,18 +97,20 @@ addressable through `source_refs`.
## Optional NPC Grounding ## Optional NPC Grounding
The `dnd/spells` extractor accepts an optional `npcs` reference containing one The `dnd/spells` extractor accepts an optional `npcs` reference containing one
normalized NPC artifact as `application/json`, up to 1 MiB. Preparation uses normalized NPC artifact as `application/json`, up to 1 MiB. An external file is
the approved NPC codec and identity policy to validate the file, re-encodes validated during preparation; an ordered generated binding is validated at the
canonical durable JSON, and supplies that JSON as a spell-owned prompt input. step handoff. Both paths use the approved NPC codec and identity policy,
It helps the model prefer canonical caster names and recognize aliases; it re-encode canonical durable JSON, and supply that JSON as an operation-time
does not establish that a spell was cast. spell prompt input. It helps the model prefer canonical caster names and
recognize aliases; it does not establish that a spell was cast.
NPC source references may identify the run that produced the registry or any NPC source references may identify the run that produced the registry or any
other session. They remain registry provenance and are never copied into a other session. They remain registry provenance and are never copied into a
spell cast's `source_refs`; every spell evidence range must still identify the spell cast's `source_refs`; every spell evidence range must still identify the
current transcript. When the slot is absent, the prompt receives exactly current transcript. Generated provenance records producer and canonical
`{"npcs":[]}` and the run has no NPC reference provenance or NPC checkpoint artifact identity without payload content or a path. When the slot is absent,
fingerprint. the prompt receives exactly `{"npcs":[]}` and the run has no NPC reference
provenance or NPC checkpoint fingerprint.
## Normalization Behavior ## Normalization Behavior
@@ -190,15 +192,20 @@ identity fields when that module is selected. Overlay origin, media type, byte
size, and raw digest are recorded separately in the manifest's reference size, and raw digest are recorded separately in the manifest's reference
provenance; see the [JSON output contract](json-output.md#manifestjson). provenance; see the [JSON output contract](json-output.md#manifestjson).
The `npc_registry_digest` and `npc_count` fields in the example are present only The `npc_registry_digest` and `npc_count` fields in the example are present for
when the optional NPC registry is bound. They contain no NPC names, aliases, an external NPC registry when the extractor publishes its prepared module
source references, paths, or raw bytes. metadata. They contain no NPC names, aliases, source references, paths, or raw
bytes. A generated registry's identity is instead represented by the framework
handoff provenance and dependency fingerprint, so the consumer module metadata
does not duplicate it.
The extractor's prompt hash, private response-schema hash, and effective catalog The extractor's prompt hash, private response-schema hash, and effective catalog
digest also contribute independently scoped semantic checkpoint fingerprints. digest also contribute independently scoped semantic checkpoint fingerprints.
Changing any of those prepared contracts intentionally produces a cold Changing any of those prepared contracts intentionally produces a cold
checkpoint miss. Fingerprints contain only digests, never prompt, schema, checkpoint miss. Fingerprints contain only digests, never prompt, schema,
catalog, or reference content. When an NPC registry is bound, its semantic catalog, or reference content. When an NPC registry is bound, its semantic
digest contributes an additional local `npc_registry` fingerprint; the digest contributes an additional local `npc_registry` fingerprint for an
manifest metadata contains only that digest and `npc_count`. Raw NPC file external binding; the manifest metadata contains only that digest and
provenance remains independently recorded in the manifest's `references` list. `npc_count`. Raw NPC file provenance remains independently recorded in the
manifest's `references` list. Generated bindings contribute the canonical
artifact dependency fingerprint and bounded producer provenance instead.

View File

@@ -104,11 +104,12 @@ The NPC identity package owns Unicode comparison keys, deterministic
The registry package resolves one optional normalized artifact through the The registry package resolves one optional normalized artifact through the
strict codec, validates whole-registry identity, canonicalizes its JSON, and strict codec, validates whole-registry identity, canonicalizes its JSON, and
provides immutable records, prompt input, semantic digest, count, and exact provides immutable records, prompt input, semantic digest, count, and exact
canonical-name/alias lookup. It owns the `npcs` slot and its bounded, canonical-name/alias lookup. External files cross this boundary during
content-safe preparation failures. NPC source references are durable preparation; generated artifacts cross it at the ordered step handoff. It owns
provenance and are not treated as evidence for a consuming pipeline. The NPC the `npcs` slot and its bounded, content-safe validation failures. NPC source
codec owns the strict durable `dnd/npc-list` JSON boundary and exposes references are durable provenance and are not treated as evidence for a
candidate versus approved encode/decode operations. consuming pipeline. The NPC codec owns the strict durable `dnd/npc-list` JSON
boundary and exposes candidate versus approved encode/decode operations.
The `internal/modules/dnd/codec/combatturns` package owns the durable The `internal/modules/dnd/codec/combatturns` package owns the durable
`dnd/combat-turn-list` schema and candidate versus approved JSON boundary. It `dnd/combat-turn-list` schema and candidate versus approved JSON boundary. It
@@ -220,9 +221,12 @@ Shared D&D helpers keep prompt input
names and source-unit reference conversion consistent with the scene chunker. names and source-unit reference conversion consistent with the scene chunker.
The extractor also declares the optional `npcs` registry slot and consumes the The extractor also declares the optional `npcs` registry slot and consumes the
prepared immutable registry boundary from `internal/modules/dnd/npcs/registry`. immutable registry boundary from `internal/modules/dnd/npcs/registry`. An
A bound registry adds only `npc_registry_digest` and `npc_count` to manifest external registry is prepared before execution; a generated registry is
metadata and an `npc_registry` checkpoint fingerprint. The unbound prompt validated and supplied at operation time. External bindings may add only
`npc_registry_digest` and `npc_count` to module metadata and an
`npc_registry` checkpoint fingerprint. Generated bindings are represented by
framework handoff provenance and dependency fingerprints. The unbound prompt
input is exactly `{"npcs":[]}` and has no registry provenance or fingerprint. input is exactly `{"npcs":[]}` and has no registry provenance or fingerprint.
The shared NPC grounding fragment is placed immediately after the common The shared NPC grounding fragment is placed immediately after the common
campaign reference message and is included in the spell prompt fingerprint. campaign reference message and is included in the spell prompt fingerprint.
@@ -310,16 +314,18 @@ independently for extraction and normalization.
### `internal/modules/dnd/normalize/combatturns` ### `internal/modules/dnd/normalize/combatturns`
The combat normalizer prepares the optional NPC registry once and uses the The combat normalizer prepares an external NPC registry before execution or
immutable prepared view during runtime. It display-normalizes combat fields, receives a generated registry at the ordered step handoff, then uses the
immutable view during runtime. It display-normalizes combat fields,
rewrites exact canonical-name or alias matches for actors and targets, orders rewrites exact canonical-name or alias matches for actors and targets, orders
and deduplicates source references, stable-sorts records by source-document and deduplicates source references, stable-sorts records by source-document
position, and collapses only exact duplicate identities with fully valid position, and collapses only exact duplicate identities with fully valid
evidence. It deep-clones output storage and emits bounded warnings scoped to evidence. It deep-clones output storage and emits bounded warnings scoped to
merged input indexes. Its metadata and fingerprints identify the normalization merged input indexes. Its metadata and fingerprints identify the normalization
and NPC identity policies, with registry digest/count only when bound. The and NPC identity policies. External bindings may contribute registry
normalizer is included in the production D&D registrar with the default combat digest/count metadata; generated identity is retained in framework provenance
normalization chain. and dependency fingerprints. The normalizer is included in the production D&D
registrar with the default combat normalization chain.
## Output Encoder ## Output Encoder

View File

@@ -14,11 +14,13 @@ collaborators, invokes `internal/framework/pipeline`, and places the logical
output files returned by the runner. Cache and debug collaborators are supplied output files returned by the runner. Cache and debug collaborators are supplied
at this boundary. at this boundary.
Resolution produces a fixed ordered workflow and a sorted set of artifact Resolution produces a fixed ordered workflow of steps and globally unique,
lanes. Preparation constructs the complete module and validator set before the sorted artifact lanes. Preparation constructs the complete module and validator
runner receives source bytes. Source parsing and chunking are serial; extraction set before the runner receives source bytes. Source parsing and chunking are
uses a bounded run-wide worker pool, followed by serial per-lane merge and serial. Each step then uses a bounded run-wide extraction pool followed by
normalize continuations that may overlap across lanes. serial per-lane merge and normalize continuations. A step barrier prevents
later consumers from starting until all earlier lanes are terminal and their
required normalized artifacts have crossed the typed handoff.
## Application Boundary ## Application Boundary
@@ -42,7 +44,7 @@ normalize continuations that may overlap across lanes.
| Package | Implemented responsibility | | Package | Implemented responsibility |
| --- | --- | | --- | --- |
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. | | `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
| `internal/framework/pipeline` | Module and artifact-codec registries, option validation, profile resolution, capability checks, reference materialization, complete pipeline preparation, retries, orchestration, warnings, and manifest population. | | `internal/framework/pipeline` | Module and artifact-codec registries, ordered-step and generated-reference resolution, option validation, profile resolution, capability checks, external reference materialization, complete pipeline preparation, typed handoff, retries, orchestration, warnings, checkpoint decisions, and manifest population. |
| `internal/framework/validate` | Shared validator decision and cardinality helpers. | | `internal/framework/validate` | Shared validator decision and cardinality helpers. |
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. | | `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. | | `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
@@ -111,12 +113,15 @@ this package. Domain-neutral prompt filesystem composition lives in
`internal/framework/promptfs`. `internal/framework/promptfs`.
The `dnd/npcs/registry` package owns the optional `npcs` registry boundary. The `dnd/npcs/registry` package owns the optional `npcs` registry boundary.
Preparation strictly decodes and identity-validates one normalized JSON External references are strictly decoded and identity-validated during
artifact, emits canonical registry JSON to the spell prompt, and records only preparation; generated references are decoded and identity-validated at the
its semantic digest and count in prepared metadata. The raw reference remains ordered step handoff. Both paths emit canonical registry JSON to operation-time
independently tracked by pipeline provenance. An absent registry is represented spell and combat prompt or normalization requests. The framework records
only by the empty prompt value `{"npcs":[]}`. Spell extraction consumes this generated identity and bounded producer provenance, while the raw external
shared registry boundary without changing its public module contract. reference remains independently tracked by pipeline provenance. An absent
registry is represented only by the empty prompt value `{"npcs":[]}`. Spell
and combat consumers use this shared boundary without changing their public
module contracts.
Generic validators under `internal/modules/generic/validate` provide Generic validators under `internal/modules/generic/validate` provide
unconditional test decisions, JSON syntax validation, and JSON Schema unconditional test decisions, JSON syntax validation, and JSON Schema
@@ -144,12 +149,12 @@ Implementation details for all production extensions are in
| Surface | Implemented owners | Internal purpose | | Surface | Implemented owners | Internal purpose |
| --- | --- | --- | | --- | --- | --- |
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. | | Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes. | | Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes, including ordered-step scope and generated-artifact dependency decisions. |
| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. | | Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. |
| Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. | | Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. |
Physical layout, cleanup, recovery, and sensitive-data handling are defined Physical layout, cleanup, recovery, and sensitive-data handling are defined
in [Operations](../operations.md). Concrete stage modules receive recorder in [Operations](../operations.md). Concrete modules receive recorder
interfaces and request data, not physical state roots. interfaces and request data, not physical state roots.
## Focused Documentation ## Focused Documentation

View File

@@ -6,13 +6,15 @@ Their fixed workflow and ownership boundaries are defined by
defaults, and selectable keys are defined in defaults, and selectable keys are defined in
[Configuration](../config.md#pipelines). [Configuration](../config.md#pipelines).
Resolution fixes the selected lanes and all stage bindings; preparation Resolution fixes the ordered steps, selected lanes, and all stage bindings;
constructs every selected implementation before the runner begins source work. preparation constructs every selected implementation before the runner begins
After serial input parsing and plan selection or generation, the runner source work. After serial input parsing and plan selection or generation, the
materializes chunks and dispatches extract work to runner materializes chunks and executes one step at a time. Within a step,
one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's extract work uses one bounded run-wide worker pool in chunk-first, lane-second
merge and normalize operations remain serial and may overlap other lanes once order. Each lane's merge and normalize operations remain serial, and lanes in
all extracts for that lane are terminal. the same step may overlap once their extracts are terminal. A later step cannot
start across its barrier until every earlier lane is terminal and each required
generated artifact has been accepted and handed off.
## Resolution ## Resolution
@@ -22,38 +24,48 @@ calls `pipeline.ResolvePipeline`.
`ResolvePipeline`: `ResolvePipeline`:
1. selects and sorts artifact lanes; 1. selects the explicit ordered steps, or creates the implicit `default` step
2. completes omitted bindings using the documented configuration defaults; from the legacy top-level `artifacts` map;
3. looks up each module and validator spec without constructing it; 2. selects and sorts artifact lanes within each step while enforcing global lane
4. for a typed extractor, derives its artifact kind, requires the codec, and identity;
3. completes omitted bindings using the documented configuration defaults;
4. looks up each module and validator spec without constructing it;
5. for a typed extractor, derives its artifact kind, requires the codec, and
selects exact-type merger, normalizer, and validator variants; selects exact-type merger, normalizer, and validator variants;
5. checks required and provided capabilities in workflow order; 6. checks required and provided capabilities in workflow order;
6. resolves target-aware reference bindings and validator chains; 7. resolves external and generated target-aware reference bindings and
7. validates each selected module and validator option set through its registry validates producer order, consumer slot declarations, and artifact-kind
compatibility;
8. validates each selected module and validator option set through its registry
entry; and entry; and
8. calculates a digest over the resolved structure, including typed artifact 9. calculates a digest over the resolved structure, including step order, step
kind and schema identity and the effective validator policy in its resolved IDs, lane membership, generated topology, producer and consumer identities,
execution order. typed artifact kind and schema identity, and the effective validator policy
in its resolved execution order.
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete Resolution returns a `ResolvedPipeline` containing ordered steps, lanes,
bindings, validator chains, reference targets, and the digest. It does not read concrete bindings, validator chains, reference targets, and the digest. It does
reference bytes or construct runtime modules. CLI lane and reference selector not read external reference bytes or construct runtime modules. CLI lane and
syntax is defined in the [CLI reference](../cli.md#run). reference selector syntax is defined in the [CLI reference](../cli.md#run).
The digest includes each resolved validator chain's stage, lane, owning module, The digest includes each resolved step's ID and lane membership, generated
ordered validator bindings, execution classes, targets, and artifact kinds. producer/consumer topology, and each validator chain's stage, lane, owning
Changing a default chain or an explicit override therefore changes pipeline module, ordered validator bindings, execution classes, targets, and artifact
identity whenever it changes the effective validator policy. kinds. Changing step order, a dependency, a default chain, or an explicit
override therefore changes pipeline identity whenever it changes effective
execution policy.
## Reference Materialization ## Reference Materialization
The CLI calls `MaterializeReferences` after resolution and before constructing The CLI calls `MaterializeReferences` after resolution and before constructing
the LLM client or running the pipeline. The materializer checks each binding the LLM client or running the pipeline. For external bindings, the materializer
against its resolved target declaration, reads and validates the file, and checks each binding against its resolved target declaration, reads and validates
builds both a `contracts.ReferenceSet` and provenance-only metadata on the the file, and builds both a `contracts.ReferenceSet` and provenance-only
corresponding `ResolvedReferenceTarget`. metadata on the corresponding `ResolvedReferenceTarget`. A structured
generated binding is declaration-only at this point: its producer bytes do not
exist until the producer lane reaches an accepted normalized result.
Preparation delivers the materialized set for each target through Preparation delivers the materialized external set for each target through
`pipeline.BuildRequest`: chunkers and chunk validators receive the chunk target; `pipeline.BuildRequest`: chunkers and chunk validators receive the chunk target;
extractors and extract validators receive the lane extract target; mergers and extractors and extract validators receive the lane extract target; mergers and
merge validators receive the lane merge target; and normalizers and normalize merge validators receive the lane merge target; and normalizers and normalize
@@ -62,10 +74,20 @@ an empty set because those stages cannot declare references. Every builder gets
an isolated deep clone of its target set, so construction-time mutation cannot an isolated deep clone of its target set, so construction-time mutation cannot
change another builder, the resolved pipeline, or later runtime requests. change another builder, the resolved pipeline, or later runtime requests.
Prepared consumers do not need to be reconstructed when generated content is
available. At the step boundary, the runner encodes the accepted producer value
through its registered canonical codec, validates the generated bytes against
each target slot's kind, schema, media type, and size, and clones one immutable
reference item into the operation request. The item includes canonical digest,
size, and bounded producer provenance but no filesystem URI. A handoff failure
is a framework dependency error and prevents every consumer in that step from
starting.
The runner continues to clone the resulting set into the chunk, extract, merge, The runner continues to clone the resulting set into the chunk, extract, merge,
or normalize request that owns the target. LLM-backed extensions may convert or normalize request that owns the target. LLM-backed extensions may convert
those items into named prompt inputs. Reference content remains separate from those items into named prompt inputs. Reference content remains separate from
source evidence and source digests. source evidence and source digests, whether the item came from a file or a
generated handoff.
Binding precedence, path resolution, accepted content, and media-type behavior Binding precedence, path resolution, accepted content, and media-type behavior
are configuration contracts; see [Configuration](../config.md#pipelines). are configuration contracts; see [Configuration](../config.md#pipelines).
@@ -120,10 +142,12 @@ The current production catalog and default chain are listed only in
## Preparation And Runner Boundary ## Preparation And Runner Boundary
`pipeline.Prepare` receives a resolved pipeline, the registries, and shared `pipeline.Prepare` receives a resolved pipeline, the registries, and shared
module dependencies. It constructs input; chunk and its validators; each lane's module dependencies. It constructs input; chunk and its validators; every
extract, merge, and normalize modules and validator chains in resolved order; step's lane extract, merge, and normalize modules and validator chains in
then output. It stops at the first error with pipeline, stage, lane, module, and resolved order; then output. It stops at the first error with pipeline, step,
validator context as applicable. It never invokes an operation method. stage, lane, module, and validator context as applicable. It never invokes an
operation method. Generated references are not available during preparation;
the operation request is the handoff boundary.
`PreparedPipeline` keeps private constructed executors and exposes cloned `PreparedPipeline` keeps private constructed executors and exposes cloned
resolved input, chunk, lane, and output identities. Prepared components may resolved input, chunk, lane, and output identities. Prepared components may
@@ -136,8 +160,8 @@ stable and must not contain source content, credentials, local paths,
timestamps, or other invocation-specific values. timestamps, or other invocation-specific values.
`pipeline.RunInput` carries that prepared pipeline, raw source input, run identity and timing, optional `pipeline.RunInput` carries that prepared pipeline, raw source input, run identity and timing, optional
session and profile metadata, a chunk-plan store and mode, and checkpoint/debug session and profile metadata, a chunk-plan store and mode, a checkpoint
collaborators. The runner execution policy, and checkpoint/debug collaborators. The runner
parses source bytes through the already constructed input adapter. Later stage parses source bytes through the already constructed input adapter. Later stage
requests receive the generic source model; extract requests receive requests receive the generic source model; extract requests receive
chunk-scoped input material, while chunk, merge, and normalize requests retain chunk-scoped input material, while chunk, merge, and normalize requests retain
@@ -145,7 +169,9 @@ access to the original source material. Input, chunk, and output operation
requests do not carry raw module options. The chunk request also does not carry requests do not carry raw module options. The chunk request also does not carry
an LLM client; an LLM-backed chunker receives the shared client during an LLM client; an LLM-backed chunker receives the shared client during
preparation. Their operation requests retain run-specific source, reference, preparation. Their operation requests retain run-specific source, reference,
profile, session, and metadata context as applicable. profile, session, metadata, and step-handoff context as applicable. A generated
reference is cloned into each compatible consumer request and is never exposed
as a path.
Prepared lanes retain exact-type-checked erased operation closures. The runner Prepared lanes retain exact-type-checked erased operation closures. The runner
uses those closures to keep each value typed through extraction, validation, uses those closures to keep each value typed through extraction, validation,
@@ -165,6 +191,13 @@ the runner returns.
## Execution Flow ## Execution Flow
The pipeline-wide coordinator owns the ordered step loop, generated-reference
sets at each barrier, and deterministic merging of step outcomes. For one step,
the lane engine initializes checkpoint state in lane order, dispatches bounded
extract work, advances terminal lanes through serial merge and normalize work,
selects failures by stable pipeline scope, and merges lane-local outcomes back
in resolved order. Completion timing never becomes public ordering.
The runner: The runner:
1. validates its prepared input; 1. validates its prepared input;
@@ -173,10 +206,15 @@ The runner:
3. selects a stored plan or executes the configured chunker's `Plan` operation; 3. selects a stored plan or executes the configured chunker's `Plan` operation;
4. canonicalizes and materializes the plan, then validates the resulting 4. canonicalizes and materializes the plan, then validates the resulting
chunks; chunks;
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a 5. executes each resolved step in configuration order. For one step, it
bounded lane continuation when all extracts for that lane are terminal; dispatches extract jobs in source-chunk then resolved-lane order, starts a
6. invokes the prepared output encoder and validates its logical file results; bounded lane continuation when all extracts for that lane are terminal, and
7. returns the assembled manifest, outcomes, warnings, and files. waits for every lane to become terminal;
6. encodes and validates each accepted normalized producer artifact, then
builds the immutable generated reference sets for the next step;
7. invokes the prepared output encoder only after every step succeeds and
validates its logical file results;
8. returns the assembled manifest, outcomes, warnings, and files.
Within each artifact lane, it reuses the prepared extractor, merger, normalizer, Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
and validators while performing these transitions: and validators while performing these transitions:
@@ -190,6 +228,13 @@ and validators while performing these transitions:
6. normalize the accepted merge result; 6. normalize the accepted merge result;
7. validate and append the accepted normalized result. 7. validate and append the accepted normalized result.
At a step barrier, a lane with no accepted normalized output is still a regular
rejection unless a later generated binding names that lane as a required
producer. In that case the runner raises a deterministic dependency error and
does not start the consumer step. One accepted typed artifact may fan out to
multiple compatible target slots. Consumers in the same step may run
concurrently after the handoff; no work crosses the barrier early.
Module-provided warnings and payload warnings are promoted only from attempts Module-provided warnings and payload warnings are promoted only from attempts
whose results are accepted and used. whose results are accepted and used.
@@ -268,13 +313,42 @@ implementations when collaborators are absent. Each checkpointed workflow
boundary records a running, succeeded, or failed transition. Reuse decisions boundary records a running, succeeded, or failed transition. Reuse decisions
are consulted in workflow order and accepted payloads are cloned before are consulted in workflow order and accepted payloads are cloned before
entering the normal handoff path. Typed extract, merge, and normalize entering the normal handoff path. Typed extract, merge, and normalize
checkpoints store codec bytes with artifact kind, schema ID and version, exact checkpoints store codec bytes with artifact kind, schema ID, name, version and
schema digest, and media type. Reuse compares that identity with the prepared exact digest, and media type. Reuse compares that identity with the prepared
codec and decodes through the codec; missing identity, mismatches, corrupt codec and decodes through the codec; missing identity, mismatches, corrupt
bytes, and decode failures become explicit reuse misses and execute the step bytes, and decode failures become explicit reuse misses and execute the lane
normally. Dependency fingerprints and debug content digests use the same stable normally. Dependency fingerprints and debug content digests use the same stable
codec bytes that cross those boundaries. codec bytes that cross those boundaries.
That progressive extract, merge, and normalize reuse is the ordinary resume
path. A lane marked as a required predecessor for selective recomputation takes
a separate accepted-output path before extract scheduling. The loader reads the
existing successful normalize manifest and payload by step, lane, and
normalizer, without consulting extract or merge dependencies. It requires the
current non-empty checkpoint identity to match, so the invocation identity
still binds the input, resolved topology and configuration, references, runtime
overrides, profiles, and component fingerprints.
The runner decodes that accepted normalized artifact with the prepared codec,
re-encodes it, and requires exact kind, schema identity and digest, media type,
canonical bytes, content digest, and producer provenance. A valid result becomes
a runner-owned cloned normalized output, restores only normalize-checkpoint
warnings, and records one `accepted_artifact_reused` normalize decision. It does
not invoke or record extract, merge, normalize, or their validators. Invalid or
unavailable accepted state records its decision and fails the producer step;
the dependent step never starts and the producer is not implicitly rerun.
Generated references add downstream dependencies containing the producer's
artifact kind, complete schema identity, media type, canonical content digest,
and size. Compatible accepted producer outputs may therefore feed a later step
without re-executing the producer. Forced lanes bypass accepted-output
hydration and execute normally. A missing, rejected, corrupt, incompatible, or
changed producer blocks its dependent while leaving independent work eligible
for reuse. The runner records bounded decision
categories: `reused`, `executed`, `forced_recompute`, and
`dependency_invalidated`. Operator meanings for the stable reason codes belong
to [Operations](../operations.md#resume-and-selective-recompute).
The CLI includes prepared-component fingerprints in the run-wide checkpoint The CLI includes prepared-component fingerprints in the run-wide checkpoint
identity alongside resolved configuration, raw input, reference provenance, identity alongside resolved configuration, raw input, reference provenance,
runtime overrides, and LLM-profile fingerprints. Module metadata is not used runtime overrides, and LLM-profile fingerprints. Module metadata is not used
@@ -283,6 +357,11 @@ values that can change accepted output. Adding or changing a component
fingerprint intentionally produces a cold cache miss. Existing checkpoint fingerprint intentionally produces a cold cache miss. Existing checkpoint
schemas and paths remain unchanged. schemas and paths remain unchanged.
The CLI's `--recompute-step` policy forces the selected step and all transitive
dependents, but requires accepted normalized artifacts for every unselected
producer on which that closure depends. It changes execution policy only; it
does not alter persistent checkpoint identity.
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
boundaries. Every executed chunk, extract, merge, and normalize attempt writes boundaries. Every executed chunk, extract, merge, and normalize attempt writes
one terminal envelope for acceptance, validator rejection, module or validator one terminal envelope for acceptance, validator rejection, module or validator
@@ -347,7 +426,9 @@ stage, resolved lane, and source chunk rather than completion time.
- `internal/framework/pipeline/references_test.go`: target resolution and - `internal/framework/pipeline/references_test.go`: target resolution and
materialization. materialization.
- `internal/cli/run_contract_test.go`: production run transitions, retries, - `internal/cli/run_contract_test.go`: production run transitions, retries,
rejections, warnings, debug hooks, and manifests. rejections, warnings, CLI recomputation controls, debug hooks, and manifests.
- `internal/cli/recompute_execution_contract_test.go`: filesystem-backed
selective recomputation and accepted-producer recovery.
- `internal/cli/production_contract_test.go`: production composition and - `internal/cli/production_contract_test.go`: production composition and
configuration-resolution smoke coverage. configuration-resolution smoke coverage.
- `internal/cli/example_contract_test.go`: maintained example resolution and - `internal/cli/example_contract_test.go`: maintained example resolution and

View File

@@ -32,6 +32,40 @@ The serialized
`workspace_schema_version` identifiers are frozen wire-compatibility fields; `workspace_schema_version` identifiers are frozen wire-compatibility fields;
they do not describe a current public state surface. they do not describe a current public state surface.
Ordered-step lane checkpoints include the step identity in their storage scope.
When a later lane consumes a generated artifact, its dependency fingerprints
include the producer's artifact kind, complete schema identity, media type,
canonical content digest, and size. Ordinary resume compares those fingerprints
when progressively loading consumer stage checkpoints, so changed producer
content produces `dependency_invalidated` rather than stale downstream reuse.
Selective recomputation instead requires each unselected producer's accepted
normalized artifact; invalid accepted state records its specific bounded reason
and stops before the dependent. The selected step and its transitive dependents
record `forced_recompute`.
Ordinary resume loads extract, merge, and normalize checkpoints progressively
and may execute later lane stages after an earlier cache miss. Selective
recomputation instead asks the loader for the required producer's accepted
normalize artifact. That lookup reuses the existing normalize files, requires
workspace schema v3 plus an exact non-empty invocation identity, and deliberately
does not require extract or merge checkpoint files or dependency fingerprints.
The runner performs canonical codec and producer-provenance validation before
cloning the artifact into normal step output. Success restores only stored
normalize warnings and emits one normalize decision; failure retains the files,
records the decision, and stops without executing the producer or consumer.
The loader assigns a typed category and reason code at each validation site;
diagnostic prose is not classified after the fact. The runner then applies
forced-execution policy, validates reusable artifact bytes through the prepared
codec, and records the final decision before enforcing a required-predecessor
failure. That failure names only the step, lane, and stable reason code. Decision
detail passes through one UTF-8-safe bounded sanitizer and contains only
allowlisted diagnostic context, never payloads, references, credentials,
environment values, or physical paths. Typed categories and codes remain intact
through pipeline events and become strings only in manifest and debug-summary
JSON. [Operations](../operations.md#resume-and-selective-recompute) is the
canonical operator-facing reason-code reference.
`internal/core/fileio` provides confined atomic file writes used by state `internal/core/fileio` provides confined atomic file writes used by state
collaborators. The chunk-plan store retains its stronger entry validation. collaborators. The chunk-plan store retains its stronger entry validation.
@@ -48,6 +82,11 @@ boundaries redact sensitive metadata and credential-shaped bytes while allowing
application-owned trace material. Debug data is never a checkpoint source or application-owned trace material. Debug data is never a checkpoint source or
cache input. cache input.
Generated reference bytes exist only in cloned operation requests and are not
written as paths into checkpoints, manifests, or debug summaries. Those state
surfaces retain canonical identities and bounded producer provenance so that a
resume decision can be explained without copying generated campaign content.
After allocation, one CLI-owned state value accumulates the known report paths, After allocation, one CLI-owned state value accumulates the known report paths,
pipeline outcome counts, and validation status. A single guarded terminalization pipeline outcome counts, and validation status. A single guarded terminalization
operation writes the success report, or makes one attempt each to write the operation writes the success report, or makes one attempt each to write the
@@ -60,6 +99,8 @@ separately and never replace the command's primary error.
terminalization, and output/report boundaries. terminalization, and output/report boundaries.
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures, - `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
permissions, cleanup, and redaction. permissions, cleanup, and redaction.
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
filesystem recovery, deterministic decisions, and failed predecessor state.
- `internal/cli/production_contract_test.go`: production composition and - `internal/cli/production_contract_test.go`: production composition and
configuration validation at the CLI boundary. configuration validation at the CLI boundary.
- `internal/cli/example_contract_test.go`: maintained example ownership. - `internal/cli/example_contract_test.go`: maintained example ownership.

View File

@@ -40,63 +40,33 @@ names, schemas, and media types inside a run directory.
Remove an output run directory only after its consumer data is no longer Remove an output run directory only after its consumer data is no longer
needed. This is data deletion, not cache cleanup. needed. This is data deletion, not cache cleanup.
## Sequential NPC And Spell Runs ## Ordered D&D Workflow
The maintained [sequential configuration](../examples/dnd-npc-spell-sequential.config.yml) The maintained [NPC-grounded configuration](../examples/dnd-npc-grounded.config.yml)
contains two independent pipelines over the same Seriatim input shape. Run the contains one pipeline with two ordered steps. The first step extracts and
NPC pipeline first and retain its normalized payload: normalizes NPCs. Only after that lane reaches an accepted terminal result does
the second step begin; its generated NPC reference is supplied in memory to
spell extraction, combat extraction, and combat normalization.
```sh ```sh
go run ./cmd/notarius run dnd-npcs \ go run ./cmd/notarius run dnd-npc-grounded \
--config examples/dnd-npc-spell-sequential.config.yml \ --config examples/dnd-npc-grounded.config.yml \
--input examples/seriatim-minimal-transcript.json \ --input examples/seriatim-minimal-transcript.json \
--output-dir ./npc-output --output-dir ./npc-grounded-output
``` ```
Then bind that completed run's `lanes/npcs.json` file to the spell extractor: The NPC artifact grounds canonical names and aliases, not spell or combat
evidence. Current-transcript source ranges remain the only event evidence. The
manifest records generated-reference identity and bounded producer provenance;
it does not record generated payload content, and no generated content is
exposed through a filesystem path. The same producer artifact may fan out to
compatible consumers, while a missing or rejected producer prevents the later
step from starting.
```sh Standalone module configurations continue to support external NPC files when a
go run ./cmd/notarius run dnd-spells \ workflow intentionally crosses a process or session boundary. Those files are
--config examples/dnd-npc-spell-sequential.config.yml \ validated against the consumer slot and must be protected as sensitive
--input examples/seriatim-minimal-transcript.json \ campaign data. They are not part of the maintained ordered handoff workflow.
--reference spells.extract.npcs=./npc-output/<run-id>/lanes/npcs.json
```
The NPC file is a reference for canonical caster names and aliases, not spell
evidence. The spell manifest records the bound file's raw reference provenance
and the prepared registry's count and semantic digest separately. The NPC
payload, names, aliases, source references, and file bytes can be sensitive
campaign data; protect both output roots and any checkpoint or debug roots that
retain derived application data. A registry from another session is allowed,
but its source references are never copied into spell output evidence.
## Sequential NPC And Combat Runs
The maintained [sequential NPC and combat configuration](../examples/dnd-npc-combat-sequential.config.yml)
also represents two independent runs. Run `dnd-npcs` first, then bind its
normalized `lanes/npcs.json` payload to both combat stages:
```sh
go run ./cmd/notarius run dnd-npcs \
--config examples/dnd-npc-combat-sequential.config.yml \
--input examples/seriatim-minimal-transcript.json \
--output-dir ./npc-output
go run ./cmd/notarius run dnd-combat \
--config examples/dnd-npc-combat-sequential.config.yml \
--input examples/seriatim-minimal-transcript.json \
--reference combat.extract.npcs=./npc-output/<run-id>/lanes/npcs.json \
--reference combat.normalize.npcs=./npc-output/<run-id>/lanes/npcs.json
```
Extraction and normalization bindings are stage-local and are intentionally
specified separately. Notarius does not discover the NPC run, copy its source
ranges into combat evidence, or compose the two runs into one workflow. The
combat manifest records both reference bindings and the prepared registry's
semantic digest/count. Changing the referenced NPC payload, prompt or schema,
normalization policy, or registry digest makes affected checkpoint state
incompatible; output, checkpoint, and debug roots remain independent sensitive
state surfaces.
## Chunk-Plan Cache ## Chunk-Plan Cache
@@ -177,6 +147,72 @@ cache:
Remove an exact checkpoint identity directory or the configured root only when Remove an exact checkpoint identity directory or the configured root only when
recomputation is acceptable. recomputation is acceptable.
### Resume And Selective Recompute
`--resume` loads compatible accepted work only when checkpoint recording is
enabled. A normal resumed run may reuse source, extract, merge, and normalize
checkpoints independently and may recompute a stage after a cache miss.
Generated references add a dependency fingerprint
for the producer's artifact kind, schema identity, media type, canonical
content digest, and size. If that fingerprint changes or the producer is
missing, dependent checkpoints are invalidated; unrelated work remains eligible
for reuse.
`--recompute-step <step-id>` requires both `--resume` and
`cache.checkpoints.enabled: true`. It forces the named step and all transitive
dependents to execute, while compatible predecessors and unrelated lanes remain
reusable. The ID may be an explicit configured step or `default` for an
implicit single-step pipeline. It cannot be combined with `--only`, and it does
not change the persistent identity of otherwise identical checkpoints.
Decisions are bounded and categorized as `reused`, `executed`,
`forced_recompute`, or `dependency_invalidated`.
For an unselected producer required by a recomputed step, Notarius loads the
accepted normalized artifact directly. Valid normalize state is sufficient even
when that producer's extract or merge checkpoint is missing or corrupt. The
normalize manifest must be successful and match workspace schema v3, the exact
current invocation identity, step, lane, and normalizer; its payload digest and
canonical codec representation must also validate. A forced producer bypasses
this lookup and executes.
If a required predecessor's accepted normalized artifact is missing, rejected,
corrupt, non-canonical, or incompatible, the run fails before the dependent
step starts. It does not fall back to rerunning that predecessor. The failure
manifest retains completed upstream outcomes and dependency context but not
generated reference content. For diagnosis, first check the producer step and
lane in the manifest, then inspect checkpoint decision categories and reason
codes. Rerun the producer explicitly rather than copying an artifact into the
checkpoint root.
The decision that caused a required-predecessor failure is retained before the
run returns, and the CLI error identifies its step, lane, and reason code.
Checkpoint reason codes are stable diagnostic identifiers:
| Reason code | Operator meaning |
| --- | --- |
| `loading_disabled` | This invocation did not enable checkpoint loading. |
| `checkpoint_missing` | The requested checkpoint file does not exist. |
| `checkpoint_path_invalid` | The requested checkpoint location failed confinement validation. |
| `checkpoint_read_failed` | An existing checkpoint could not be read. |
| `checkpoint_decode_failed` | Checkpoint JSON could not be decoded. |
| `workspace_schema_incompatible` | The stored workspace schema is not supported by this build. |
| `identity_mismatch` | The stored invocation identity differs from the current invocation. |
| `stage_mismatch`, `step_mismatch`, `lane_mismatch`, `module_mismatch` | Stored scope does not match the requested pipeline scope. |
| `status_not_reusable` | The stored operation did not finish in a reusable status. |
| `dependency_mismatch` | Stored dependencies differ; the category is `dependency_invalidated`. |
| `artifact_payload_invalid` | Stored artifact payload structure or encoding is invalid. |
| `artifact_digest_mismatch` | Stored artifact bytes do not match their recorded digest. |
| `artifact_codec_incompatible` | Stored artifact identity is incomplete or incompatible with the codec contract. |
| `artifact_not_canonical` | The codec can decode the artifact, but its bytes are not canonical. |
| `checkpoint_reused` | The stored checkpoint passed validation and was reused. |
| `accepted_artifact_reused` | A required producer's accepted normalized artifact was canonically validated and hydrated. |
| `recompute_step` | Selective recomputation forced execution of this lane. |
Decision detail is bounded explanatory text, not a data-recovery channel. It
never contains checkpoint paths, artifact or reference content, source content,
credentials, or environment values.
## Debug Bundles ## Debug Bundles
Only `notarius run --debug` enables debug collection. The selected root contains Only `notarius run --debug` enables debug collection. The selected root contains

View File

@@ -1,35 +0,0 @@
# D&D Module Harmonization And Prompt Reuse
This roadmap tracked harmonization of the spell, NPC, and combat-turn lanes.
The implemented behavior is now owned by [Module Internals](../internal/modules.md)
and [LLM Runtime Internals](../internal/llm.md); this document records status
and the one remaining external prerequisite rather than duplicating those
current-behavior references.
## Implementation Status
The offline/runtime work is complete:
- shared prompt assets have explicit per-prompt manifests that drive both
mounting and prompt fingerprints;
- shared prompt inputs, deterministic chunk material preparation, and cited
source traversal are centralized;
- prompt ordering and cache-control policy are aligned across the D&D
extraction prompts, while scene chunking retains its distinct prompt shape;
- validator prerequisite, bounded-diagnostic, and checkpoint policies are
aligned without changing compatibility-sensitive identifiers;
- production registration remains explicit and typed, with grouped composition,
central default-chain ownership, and artifact-specific merge behavior;
- private LLM response schemas remain package-owned and separate from durable
codec schemas; and
- focused tests and repository-wide test, vet, build, and diff checks pass.
## Deferred evaluation
An optional live before/after provider comparison remains deferred because it
requires credentials and a maintained human-reviewed transcript or fixture
set. It is not part of the default test suite or merge gate. If those
prerequisites become available, record only aggregate extraction-review
results, prompt token counts, cache-hit/cache-write metrics, and non-secret
prompt hashes. Never commit transcript content, rendered prompts, credentials,
endpoints, or private reference material.

View File

@@ -5,122 +5,6 @@ configuration, operations, internal, and integration docs. This roadmap records
future work only. Items are ordered roughly by current value and specificity, future work only. Items are ordered roughly by current value and specificity,
not as committed release dates. not as committed release dates.
## Near-Term: Ordered Pipeline Steps
Allow one configured pipeline to contain multiple ordered execution steps so
accepted artifacts from an earlier step can become generated references for
later steps in the same run. This is a bounded extension of the fixed pipeline
model, not an arbitrary DAG or general workflow language.
Input parsing and chunk planning remain pipeline-wide. Each step selects one or
more artifact lanes; every selected lane completes extraction, validation,
merge, normalization, and validation before dependent later steps begin. Lanes
within the same step remain independent and may execute concurrently. The
runner exposes only accepted normalized artifacts across a step boundary; raw
extracts, rejected outputs, and intermediate merge results cannot become
downstream references.
Generated-reference bindings must be explicit in resolved configuration. A
binding identifies an earlier producing lane and one declared reference slot on
a later consuming lane. Resolution must reject missing producers, references to
the same or a later step, incompatible artifact kinds or media types, undeclared
consumer slots, cycles, and ambiguous bindings. A configured external reference
and a generated reference cannot bind the same effective target slot; reject
that pipeline or runtime override instead of applying a precedence rule.
Each effective target slot accepts at most one producer. One generated artifact
may fan out to multiple compatible target slots in a later step; aggregation
from multiple producers into one slot is deferred until a concrete use case
defines deterministic semantics.
Step-scoped reference defaults mirror existing pipeline-level reference
defaults. A step binds an earlier artifact once, and the binding automatically
applies to every target in that step that declares the named slot. Target-local
bindings remain available when only one module should consume the artifact; do
not combine a target-local and step-scoped binding for the same effective slot.
A generated source uses a structured, unambiguous form rather than encoding a
producer into a path-like string. The target configuration shape is:
```yaml
steps:
- id: identify-npcs
artifacts:
npcs:
extract: dnd/npcs
normalize: dnd/npcs
- id: grounded-events
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells
combat:
extract: dnd/combat-turns
normalize: dnd/combat-turns
```
This snippet shows the step and reference portion of a pipeline; pipeline-wide
input, chunk, output, and other references are omitted. Existing scalar
reference values continue to mean external file paths; the structured
`artifact` form means an accepted normalized artifact from the named earlier
step and lane. Do not infer generated bindings from module keys, matching lane
names, or D&D-specific knowledge in the framework.
The handoff should use the producer artifact's canonical codec representation
and retain its artifact kind, schema identity, media type, content digest, and
producer provenance. Generated references provide context or disambiguation,
not source evidence. They use the existing module-facing reference contract
where possible; typed or domain-specific adapters may validate and prepare a
reference without moving domain concepts into the pipeline framework.
Pipeline identity, manifests, checkpoint dependencies, debug records, and
errors must include step identity and generated-reference provenance. A
downstream checkpoint is reusable only when the upstream artifact identity and
content digest match. Resume should reconstruct an accepted upstream artifact
through its registered codec instead of requiring the producing lane to run
again when its checkpoint is reusable.
### Dependency-aware resume and recomputation
Treat generated-reference bindings as checkpoint dependencies. Reusing an
earlier step is safe only when its existing checkpoint and codec identity are
valid. Reusing a dependent step additionally requires an exact match for every
upstream artifact kind, schema identity, media type, and canonical content
digest it consumed. A changed, missing, rejected, corrupt, or incompatible
producer artifact invalidates all transitive dependent checkpoints; the runner
must never combine a newly produced upstream artifact with stale downstream
output.
Support selectively recomputing one configured step and all of its transitive
dependents while retaining reusable independent and predecessor work. The
operator-facing selection mechanism should identify a stable configured step,
not individual internal stage operations. Resolution must reject a selection
that would omit a required predecessor without a reusable accepted artifact.
Manifests, checkpoint events, and diagnostics should distinguish work that was
executed, reused, or invalidated and record a bounded non-secret reason for
dependency-driven invalidation. Completion order must not affect invalidation,
public artifact ordering, or the set of dependent steps selected for rerun.
If a required producer finishes without an accepted normalized artifact, fail
the entire run with a deterministic dependency error. Do not start any
dependent step. Preserve the upstream rejection or empty-result outcome and
step provenance in the failed run manifest so the cause remains auditable.
The first production workflow is D&D NPC grounding:
1. the first step runs the NPC lane through accepted normalized output; and
2. the second step runs spell and combat-turn lanes, binding that NPC artifact
to the spell extractor and to the combat extractor and normalizer through
their existing `npcs` reference slots.
Spell and combat-turn extraction may run concurrently after the NPC handoff is
available. The NPC artifact may disambiguate participant identity, but it does
not prove that a spell cast or combat turn occurred.
## Near-Term D&D Pipeline ## Near-Term D&D Pipeline
### Evaluate Spell Extraction And Normalization ### Evaluate Spell Extraction And Normalization

View File

@@ -1,573 +1,399 @@
# D&D Module Harmonization Implementation Plan # Implementation Plan: Ordered Pipeline Follow-Up
This document is the executable implementation plan for the target state in ## Status
[D&D Module Harmonization And Prompt Reuse](dnd.md). It is written for a coding
agent and must be followed in stage order. Each stage must leave the repository
building and its focused tests passing before the next stage begins.
Current behavior is documented in the Completed on 2026-07-22. This document retains the implementation sequence for
[module internals](../internal/modules.md) and historical context; current contracts are maintained in the canonical CLI,
[LLM runtime internals](../internal/llm.md). The policies in configuration, operations, integration, and internal documentation linked from
[Architecture](../policy/architecture.md), the [development guide](../development.md).
[Testing](../policy/testing.md), and
[Documentation](../policy/documentation.md) govern all stages.
## Fixed Decisions And Constraints ## Purpose
The following decisions are complete and are not implementation-time choices: Record the work that closed the correctness, observability, test, and
maintainability gaps in the implemented
[Ordered Pipeline Steps](ordered-pipeline-steps.md) feature. That feature
roadmap remains the authority for product intent, policy choices, acceptance
criteria, and exclusions. This document preserves the ordered,
decision-complete implementation sequence for the follow-up work.
- Keep spell, NPC, and combat-turn response DTOs, schemas, canonicalization, ## Background
codecs, normalizers, and domain validators in their current domain packages.
- Put D&D-only reuse in `internal/modules/dnd/shared`; do not move D&D concepts
into `internal/framework`.
- Keep private LLM response schemas separate from durable codec schemas. Do not
introduce shared JSON Schema fragments or a schema-generation step in this
work.
- Preserve all public module keys, artifact kinds, prompt IDs, schema IDs,
schema versions, media types, reference slot names, durable JSON fields, and
existing reason-code strings. In particular, retain the spell validator's
existing `invalid_source_refs` reason code as a compatibility exception.
- Preserve central ownership and ordering of default validator chains in the
D&D registrar.
- Do not create a generic extractor framework, use reflection for lane
registration, or erase typed artifact relationships outside existing
framework boundaries.
- Exact cache identity means equal ordered message roles, content bytes, and
cache-control values after Scriptorium has rendered the prompt. Enforce this
primarily through one canonical shared asset per shared message and document
the expected order and cache-control policy.
- Scriptorium and provider-specific types remain confined to the LLM runtime,
prompt-asset wiring, and their tests. Production extractors continue to use
only Notarius contracts.
- Default tests remain offline, deterministic, and credential-free. Live model
evaluation is an explicit manual acceptance activity, not part of
`go test ./...`.
- Do not add message-count, common-prefix-length, or complete rendered-prompt
snapshots. They are change detectors rather than durable behavioral tests.
- Update current-behavior documentation only in the stage that changes that
behavior. Do not describe a partially implemented later stage as complete.
## Target Prompt Layout The original implementation is complete in broad architecture. Pipelines now
resolve and prepare one canonical ordered-step model, execute hard barriers
between steps, hand accepted normalized artifacts to later operations as
canonical generated references, include generated identity in checkpoint and
output provenance, support `--recompute-step`, and use generated NPC output at
operation time in the D&D spell and combat consumers. Current documentation and
maintained examples describe that model.
Stage 3 must produce the following ordered messages. All listed shared entries The completed follow-up addressed these narrower gaps:
must refer to one shared embedded file rather than package-local copies.
| Index | Spell | NPC | Combat turn | Role | Cache control | - selective recomputation now hydrates an unselected predecessor from its
| --- | --- | --- | --- | --- | --- | accepted normalized artifact without requiring its extract and merge state;
| 0 | common system | common system | common system | system | none | - the failed checkpoint decision is recorded before a required-predecessor
| 1 | common extraction evidence | common extraction evidence | common extraction evidence | user | none | error returns;
| 2 | common in-world identity | common in-world identity | common in-world identity | user | `ephemeral` | - checkpoint reason codes are assigned explicitly rather than inferred from
| 3 | common transcript | common transcript | common transcript | user | `ephemeral` | human-readable prose;
| 4 | common campaign references | common campaign references | common campaign references | user | `ephemeral` | - runner lane and checkpoint orchestration has explicit responsibility seams;
| 5 | common immediate resolution | NPC task | common immediate resolution | user | none | and
| 6 | common NPC registry | NPC instructions | common NPC registry | user | `ephemeral` for spell/combat | - CLI and resumed-producer acceptance coverage exercises the complete recovery
| 7 | spell catalog | — | combat task | user | none | contract.
| 8 | spell task | — | combat instructions | user | none |
| 9 | spell instructions | — | — | user | none |
The common extraction-evidence asset must state, in artifact-neutral language, No new product feature is introduced by this plan. Preserve configuration file
that: version 3 and checkpoint workspace schema `notarius.workspace.v3`; the fixes do
not require a new persistent format.
- transcript units are the only event evidence; ## Instructions For Every Stage
- campaign and registry references may disambiguate but are not evidence;
- every reported factual claim is supported by cited transcript units;
- references use integer `start_unit_id` and `end_unit_id` values;
- `source_id` is omitted because Notarius assigns the current source identity;
- non-contiguous evidence uses multiple narrow ranges rather than a broad
bridge over unrelated conversation; and
- output contains only the configured JSON object and schema-defined fields.
The common in-world-identity asset must require the most specific supported Before changing code, read `docs/development.md`, all documents under
in-world character or creature identity instead of a human player, transcript `docs/policy/`, and the task-specific internal documents named there. Use the
speaker, or the GM as an out-of-world person. It may permit campaign references repository's code knowledge graph for code discovery before falling back to
to disambiguate an identity, but must not establish participation from a text search.
reference alone. NPC-only exclusion rules and relationship rules remain in the
NPC task or instructions.
The common immediate-resolution asset, used only by spells and combat turns, Implement the stages in order. Each stage must leave the repository formatted,
must limit an artifact to a declaration/action and its immediate observed building, and passing its focused tests. Preserve these invariants throughout:
resolution. It must exclude consequences on later turns or elsewhere in the
scene. Spell-only persistent-effect language and combat-only classification
rules remain local.
Remove equivalent prose from package-local task and instruction files after it - The pipeline retains one input, chunk plan, output encoder, run identity,
has moved to a shared asset. Do not retain paraphrased copies. Read each final checkpoint identity, failure boundary, worker budget, and provider scheduler.
prompt as a whole to remove contradictions and preserve all artifact-specific - Every selected module and validator is constructed before source parsing.
requirements. - Steps schedule fixed extract, merge, and normalize lanes; this work must not
introduce module-to-module calls or a general DAG scheduler.
- Generated artifacts remain cloned operation-time context, never source
evidence. Never emit their content, source material, credentials, or local
paths in decisions, manifests, logs, or summaries.
- Public ordering remains step order, lane ID, and source/chunk order as
applicable. Refactoring must not expose completion order.
- Tests must follow `docs/policy/testing.md`: protect observable recovery,
orchestration, and CLI contracts without asserting private helper calls,
exact prose, goroutine choreography, or full-document snapshots.
- Update current-behavior documentation in the same stage that changes the
corresponding behavior. Keep detailed implementation sequencing only here.
The cache-control choices above create three all-lane breakpoints and one ## Target Checkpoint Semantics
spell/combat breakpoint. Use no more than these four markers so the request
remains portable across the configured backends; do not add module-specific
markers in this work.
## Stage 1: Protect Shared Input Behavior And Baseline Prompt Wiring Use these definitions consistently in all stages:
### Goal - A **stage checkpoint** is internal resumable state for extract, merge, or
normalize. Ordinary resume may continue to reuse this state progressively.
- An **accepted lane artifact** is the one successful normalized artifact for a
`(step ID, lane ID, normalizer module)` under the current checkpoint identity.
It is the dependency exposed to a later step.
- An unselected required predecessor satisfies selective recomputation when its
accepted lane artifact can be validated and hydrated. Its extract and merge
stage checkpoints are not prerequisites for that handoff.
- A selected lane and its transitive dependents execute. They must never use
accepted-artifact hydration to bypass forced execution.
- If a required predecessor's accepted lane artifact is missing, rejected,
corrupt, non-canonical, or incompatible, fail the run before any dependent
lane starts. Do not implicitly rerun that predecessor.
- Ordinary resume without `--recompute-step` keeps its existing progressive
stage-reuse and cold-miss behavior.
Protect deterministic shared input rendering and confirm the existing prompt ## Stage 1: Decompose Runner And Checkpoint Orchestration
assets prepare successfully before changing prompt composition.
### Implementation ### Objective
1. Retain one offline prompt preparation test in each extractor package. Each Create explicit, testable ownership seams for ordered-step coordination,
test should prove registration succeeds, the prompt selects its package-owned per-lane execution, and per-stage checkpoint handling without changing
response schema, required dynamic inputs render, and no provider call or observable behavior.
credentials are required. Do not assert total message count, shared-prefix
length, or the complete rendered prompt.
2. Extend `internal/modules/dnd/shared/prompt_inputs_test.go` with one
table-driven identity test covering:
- identical source material and references produce deeply equal common input
materials;
- reference item insertion order does not change rendered reference bytes;
- `roster` fallback produces the canonical `party` material;
- an explicit non-empty `party` wins over `roster`; and
- missing optional slots render the existing single-space placeholder.
3. Do not add a full HTTP/provider test for cross-module prompt equality.
Scriptorium owns provider request serialization, and shared asset ownership
removes the duplicated content that would otherwise need equality testing.
### Verification ### Changes
Run: 1. Keep `Runner.Run` as the pipeline-wide coordinator. Extract a small
step-coordination helper responsible only for iterating prepared steps,
```sh building generated reference sets at each barrier, invoking the lane engine,
go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/combatturns and merging deterministic outcomes.
``` 2. Split `runLanes` in `internal/framework/pipeline/runner_concurrent.go` into
helpers with these responsibilities:
### Completion Criteria - initialize lane state and resolve extract checkpoint state in lane order;
- run the bounded extract worker/continuation engine;
- Shared input tests protect deterministic serialization and canonical slot - collect terminal lane results and choose failures deterministically; and
handling. - merge lane-local output into step output.
- Each extractor's existing behavior-level asset test prepares successfully Preserve the existing bounded channels, cancellation, drain behavior,
without pinning message count or prefix boundaries. chunk-first dispatch, lane ordering, and step barrier.
- No production behavior changes in this stage. 3. Split `continueTypedLane` in
`internal/framework/pipeline/runner_typed.go` into stage-specific merge and
## Stage 2: Make Prompt Asset Manifests Exact normalize helpers. Each helper should own dependency construction, checkpoint
loading and canonical validation, execution/retry/validation when needed,
### Goal checkpoint recording, debug envelopes, and its typed result. Use small
result structs rather than long parallel return lists.
Use one explicit asset manifest for prompt mounting and prompt fingerprinting, 4. Centralize repeated checkpoint-decision flow in one pipeline helper that can
and stop fingerprinting shared assets a prompt does not render. apply forced-execution policy, validate canonical stored artifacts, record
the final observable decision, and return a contextual error. Do not yet
### Implementation change categories, reason codes, or required-predecessor semantics; Stages 2
and 3 will change those deliberately.
1. Replace the broad `sharedPromptFiles`, `CommonHashParts`, and 5. Keep stage-specific code where the data shapes genuinely differ. Do not
`ReferenceHashParts` grouping in `internal/modules/dnd/shared/assets.go` introduce reflection, a generic stage state machine, or callbacks that hide
with an explicit manifest abstraction: the fixed extract/merge/normalize lifecycle.
```go
type PromptAssetManifest struct {
ModuleDir string
ModuleFiles []promptfs.ModulePromptFile
SharedFiles []string
}
```
Exact formatting may follow `gofmt`, but retain these fields and meanings.
2. Give the manifest two operations:
- `PromptFS(moduleFS fs.FS) (fs.FS, error)`, which resolves only the named
shared files and delegates composition to `promptfs.ModulePromptFS`; and
- `Hash(moduleFS fs.FS) (string, error)`, which hashes the same module and
shared files in manifest order through `llm.HashAssets`.
3. Keep the shared-name-to-embedded-path mapping private to the shared package.
Reject unknown, duplicate, empty, or path-containing shared names. Return
fresh slices so callers cannot mutate package state.
4. In each `scriptorium_assets.go`, declare one package-local manifest that
includes its YAML definition and every Markdown file referenced by that
definition. Keep ordering stable within the module and shared lists. Use
that manifest for both `RegisterPromptAssets` and
`scriptoriumPromptMetadata`.
5. Migrate the scene chunker to the same API because it uses the shared prompt
helper. Its behavior and prompt ordering remain unchanged in this stage.
6. Mount only shared assets actually referenced by each prompt. Before Stage 3:
- NPC and scene prompts must not include or hash `common-dnd-npcs.md`;
- spell and combat prompts must include and hash it; and
- all four prompts must include and hash the shared system, transcript, and
campaign-reference files they render.
7. Retain the response-schema fingerprint as its existing independent
fingerprint. Do not include response schema bytes in the prompt manifest.
### Tests ### Tests
- Replace broad shared asset tests with table-driven manifest tests for valid - Existing runner, checkpoint, barrier, ordering, cancellation, retry, debug,
composition, exact mounted files, unknown names, duplicate names, invalid and D&D integration tests must pass without expectation changes except moves
names, missing module files, missing shared files, and defensive copying. required by renamed private test fixtures.
- Independently construct the expected `llm.AssetHashPart` list in manifest - Add no tests for helper boundaries or collaborator call counts. Add a narrow
tests and assert that `Hash` equals `llm.HashAssets` over exactly that list. regression assertion only if the refactor exposes an observable behavior not
This proves unused shared assets are excluded and listed assets participate already protected.
without adding mutation hooks for the embedded filesystem.
- Keep one package-level registration test per prompt; remove redundant
per-file mounting assertions when the shared manifest tests already own that
behavior.
### Verification ### Completion Gate
Run: Run `go test ./internal/framework/pipeline ./internal/framework/checkpoint
./internal/modules/integration` and the pipeline race tests. Review the diff to
confirm this stage changes structure only: serialized output, checkpoint state,
decision values, failure selection, and module invocation behavior must remain
unchanged.
```sh ## Stage 2: Make Checkpoint Decisions Typed And Observable
go test ./internal/framework/promptfs ./internal/modules/dnd/shared ./internal/modules/dnd/chunk/scenes ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/combatturns
```
### Completion Criteria ### Objective
- Mounting and hashing are driven by the same ordered manifest. Assign stable decision categories and reason codes explicitly, and retain the
- No prompt fingerprints an unused shared asset or omits a rendered asset. decision that causes a required-dependency failure.
- Prepared prompt messages are unchanged from Stage 1.
## Stage 3: Factor And Reorder Shared Prompt Messages ### Changes
### Goal 1. In `internal/framework/pipeline/checkpoint.go`, introduce string-backed
internal types and constants for checkpoint decision categories and reason
Implement the target prompt layout defined above using canonical shared assets codes. Keep the current JSON strings and public artifact fields compatible.
and maximize the reusable all-lane and spell/combat content. Categories remain exactly `executed`, `reused`, `forced_recompute`, and
`dependency_invalidated`.
### Implementation 2. Replace prose inspection in `internal/framework/checkpoint/loader.go` with an
explicit decision constructor accepting category, reason code, and optional
1. Add these embedded assets under detail. Remove every `strings.Contains` classification branch. Assign a code
`internal/modules/dnd/shared/assets/prompts`: at the validation site using this bounded vocabulary:
- `common-dnd-extraction-evidence.md`; - `loading_disabled`, `checkpoint_missing`, `checkpoint_path_invalid`,
- `common-dnd-identity.md`; and `checkpoint_read_failed`, and `checkpoint_decode_failed`;
- `common-dnd-immediate-resolution.md`. - `workspace_schema_incompatible` and `identity_mismatch`;
2. Write their content according to the fixed semantic boundaries in - `stage_mismatch`, `step_mismatch`, `lane_mismatch`, `module_mismatch`, and
`Target Prompt Layout`. Use no template inputs in the identity or immediate `status_not_reusable`;
assets. The evidence asset also needs no new input; it refers generically to - `dependency_mismatch` for the `dependency_invalidated` category;
the transcript and references already presented later. - `artifact_payload_invalid`, `artifact_digest_mismatch`,
3. Update the three extractor YAML files to use the exact order, roles, and `artifact_codec_incompatible`, and `artifact_not_canonical`; and
cache-control values in the target table. Do not change prompt IDs, versions, - `checkpoint_reused` and `accepted_artifact_reused` for successful reuse,
default profiles, input declarations, output schema paths, or repair counts. and `recompute_step` for forced execution.
4. Update each package's prompt manifest from Stage 2 so it lists exactly the Retain an existing code not listed here only when a current external or
new shared dependencies in rendered order. documented contract already depends on it.
5. Delete duplicated policy prose from local `task.md` and `instructions.md` 3. Keep detail human-readable, UTF-8, bounded, and sanitized through one helper.
files while retaining every module-specific rule. Preserve one and only one Detail may identify stage, step, lane, module, expected status, or schema
`Return exactly one JSON object` rule through the common evidence message. version, but must not include artifact/reference content, source content,
6. Keep the spell catalog input and NPC registry input byte generation credentials, environment values, or local paths. Tests must assert codes and
unchanged. Keep the NPC extractor free of an `npcs` input. safety properties, not exact detail prose.
7. Update package asset tests only for behavioral wiring: prompt preparation, 4. Change the centralized runner decision flow from Stage 1 so the final loader
response schema selection, and required dynamic input rendering. Do not add or canonical-validation decision is recorded before returning a
assertions for total message count, exact shared-prefix length, or the end required-predecessor error. The contextual error must identify the step and
index of a shared section. The shared manifest tests own canonical asset lane and include the stable reason code; it must not interpolate unsafe
selection. loader detail.
8. Update `docs/internal/modules.md` and `docs/internal/llm.md` in the same 5. Propagate typed values without lossy conversion through checkpoint events,
change to describe the implemented common-prefix composition, exact prompt run manifests, debug summaries, and CLI diagnostics. Convert to strings only
manifest fingerprinting, and cache-control placement. Describe current at existing serialized boundaries unless changing an internal field type is
behavior, not this staged plan. simpler and wire-compatible.
6. Update the canonical current-behavior owners in the same change:
### Verification `docs/internal/state.md` owns the internal decision flow, while
`docs/operations.md` owns operator diagnosis and any operator-visible reason
Run: code table. Link rather than duplicate the table elsewhere.
```sh
go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/combatturns ./internal/modules/dnd/register
go test ./internal/framework/llm
```
### Completion Criteria
- Shared rules exist in one embedded file and have no local paraphrased copy.
- Every applicable prompt references those shared assets in the documented
order and with the documented cache-control policy.
- Prompt fingerprints change for the intentional prompt contract change and
include every newly rendered asset.
- No artifact schema, durable representation, module selection, or reference
contract changes.
## Stage 4: Centralize Chunk Prompt Material
### Goal
Remove the three identical `chunkSourceInput` implementations and make common
transcript input preparation a single D&D-owned behavior.
### Implementation
1. Add `internal/modules/dnd/shared/extraction_inputs.go` with:
```go
func ChunkPromptMaterial(req contracts.TypedExtractionRequest) (contracts.LLMInputMaterial, error)
```
2. Preserve the existing behavior exactly: clone `req.SourceInput`; fall back
to chunk content and media type when content is empty; require content bytes
to equal `req.Chunk.Content`; default the material name to `source`; default
media type from the chunk; and populate `SizeBytes` when zero.
3. The helper may assume the caller has already checked `req.Chunk != nil`.
Return a D&D-shared error without an extractor name. Each extractor wraps it
with its existing `extractorErrorf`, retaining module context.
4. Replace all three local helpers and remove now-unused `bytes` imports.
5. Do not centralize the remaining request checks. Their typed result handling
and module-specific errors make the small duplication clearer than a
callback- or generic-heavy abstraction.
### Tests ### Tests
- Add one table-driven shared helper test covering fallback, clone isolation, - Add a table in `internal/framework/checkpoint` that exercises one
mismatch, default fields, and preservation of explicit metadata. representative input per reason-code family and asserts category, code,
- Remove duplicate extractor tests only when the shared test fully owns the bounded valid UTF-8 detail, and absence of supplied secret/path sentinels.
behavior. Retain one extractor-level test per lane proving helper errors are - Add pipeline behavior tests proving a required-predecessor failure records the
wrapped with that module's context. underlying missing, corrupt, incompatible, or dependency-invalidated
- Keep existing tests proving all three extractors pass equal common prompt decision before returning.
inputs to the LLM contract. - Retain focused manifest/debug serialization assertions for category and code;
do not add exact-detail or full-manifest snapshots.
### Completion Gate
Run `go test ./internal/framework/checkpoint ./internal/framework/pipeline
./internal/core/artifacts ./internal/core/debugbundle ./internal/cli`. Search the
production checkpoint package to confirm no decision category or reason code is
derived from diagnostic prose.
## Stage 3: Hydrate Required Predecessors From Accepted Normalized Artifacts
### Objective
Make selective recomputation enforce the lane-level accepted-artifact contract:
a valid normalized producer artifact is sufficient even when its extract or
merge stage cache is unavailable.
### Changes
1. Extend the checkpoint loader contract with a dedicated accepted-normalized-
artifact lookup. Implement it in the filesystem loader and every no-op or
test implementation. The lookup receives step ID, lane ID, and normalizer
module key and returns the normalized artifact, its warnings, and an explicit
checkpoint decision. It must not require caller-supplied extract or merge
dependency fingerprints.
2. The filesystem lookup may reuse the existing normalize manifest and payload;
do not add a second persistent copy. It is reusable only when all of the
following hold:
- workspace schema is v3 and the non-empty stored checkpoint identity matches
the current invocation identity;
- stage, step, lane, normalizer module, and successful status match;
- the manifest output digest matches the payload; and
- the payload can subsequently be validated through the registered artifact
codec.
Skipping caller-supplied merge dependencies is safe only because the matched
non-empty checkpoint identity already binds the current input, resolved
topology and configuration, external references, runtime overrides, LLM
profiles, and component semantic fingerprints. Do not weaken or omit that
identity check.
A loader lacking a verifiable current identity must return an unavailable
decision rather than perform accepted-artifact reuse.
3. Add a pipeline hydration helper that decodes the returned serialized
artifact through the producer's registered codec, re-encodes it canonically,
and requires exact artifact kind, schema ID/name/version/digest, media type,
content bytes, and content digest. Return a runner-owned clone with producer
step, lane, module, and source identity. Any mismatch is an explicit bounded
decision and the bytes never reach a consumer.
4. Before normal execution of a lane marked in `RequireReusableLanes`, use the
accepted-artifact lookup:
- on success, mark the lane terminal without invoking extract, merge,
normalize, or their validators;
- append the accepted normalized output in the normal deterministic location
and restore only the warnings stored with that normalized checkpoint;
- record one `reused` normalize decision with a stable accepted-output reuse
reason code; do not synthesize extract/merge decisions, warnings, or
rejections that were not loaded; and
- allow the ordinary step barrier and generated handoff code to consume that
output exactly as it consumes a freshly executed output.
5. On missing, rejected, corrupt, non-canonical, or incompatible accepted state,
record the decision and fail the run before the dependent step begins. Do
not fall back to stage execution. Forced lanes must bypass this hydration
path and execute normally.
6. Leave ordinary resume unchanged when no lane is marked
`RequireReusableLanes`: it may reuse or recompute extract, merge, and
normalize progressively under the existing cold-miss rules.
7. Keep generated-reference fingerprints and provenance unchanged. Hydrating a
byte-identical producer must yield the same canonical handoff digest and
downstream dependency fingerprint as fresh execution.
8. Update `docs/internal/pipeline.md`, `docs/internal/state.md`, and
`docs/operations.md` in this stage to distinguish progressive stage reuse
from accepted normalized-artifact hydration and to document the fail-rather-
than-rerun rule for invalid required predecessors.
### Tests
- At the pipeline/checkpoint boundary, create a valid producer normalize
checkpoint while omitting or corrupting its extract and merge checkpoints.
Select a later step for recomputation and prove the producer hydrates, no
producer operation or validator runs, and the dependent receives the exact
canonical artifact.
- Cover missing, rejected-status, corrupt, non-canonical, wrong-codec-identity,
and wrong-content-digest normalize state. Assert failure and the recorded
stable decision before any consumer invocation.
- Prove forced producers execute instead of hydrating, while a reusable
predecessor and an unrelated lane remain reusable.
- Prove fresh and hydrated producer outputs create identical generated
provenance and downstream checkpoint fingerprints without exposing content
or paths.
### Completion Gate
Run `go test ./internal/framework/checkpoint ./internal/framework/pipeline
./internal/modules/integration` and the corresponding race tests. Manually
inspect one failed test fixture to confirm the accepted artifact remains on
disk but its bytes do not appear in the manifest, decision detail, debug
summary, or error.
## Stage 4: Complete Recompute CLI And Recovery Acceptance Coverage
### Objective
Exercise the operator-visible recomputation contract through stable boundaries
and close the acceptance-test omissions from the original plan.
### Changes
1. Add one table-driven CLI contract test for `--recompute-step` covering:
- a valid explicit step and the implicit `default` step;
- repeated flags and an empty or unknown step ID;
- use without `--resume`;
- use when checkpoint recording is disabled; and
- combination with `--only`.
Assert exit classification and stable identifying fragments or reason codes,
not complete prose.
2. Add one filesystem-backed CLI execution test using deterministic fake
modules and a three-step generated dependency chain plus one unrelated lane:
- perform a fresh checkpointed run;
- remove or corrupt only the required producer's extract and merge state
inside `t.TempDir()`, leaving its normalize artifact valid;
- resume with the middle step selected;
- assert the selected lane and transitive dependents execute, the predecessor
hydrates without module calls, the unrelated lane reuses, and output and
decision ordering are deterministic; and
- invalidate the producer normalize state in a subcase and assert the command
fails before dependent execution with the bounded decision preserved.
3. Add or extend one generic two-step pipeline test proving handoff succeeds
both from fresh producer execution and from accepted normalized-artifact
hydration. Keep this at the pipeline boundary if the CLI execution test
already proves flag wiring; do not duplicate every CLI case end to end.
4. Review existing recomputation policy tests. Retain the pure closure test
because it protects transitive selection, but remove or consolidate any new
test that merely repeats the CLI or pipeline behavior above.
5. Make only production changes revealed as necessary by these contract tests;
do not add new flag semantics or broaden the feature roadmap.
### Completion Gate
Run `go test ./internal/cli ./internal/framework/pipeline
./internal/framework/checkpoint`, then run the CLI, pipeline, and checkpoint
packages under the race detector. Confirm no test asserts internal loader call
counts, exact decision detail, filesystem layout outside a temp workspace, or
the exact length of any prompt or prefix.
## Stage 5: Current Documentation And Release Verification
### Objective
Reconcile all canonical documentation after the staged changes and verify the
complete feature against policy and roadmap.
### Changes
1. Re-read `docs/internal/pipeline.md`, `docs/internal/state.md`,
`docs/operations.md`, and `docs/cli.md` against the final code. Correct any
stale statements left by Stages 1 through 4 without duplicating their
canonical contracts.
2. Ensure internal documentation describes the coordinator, lane engine, stage
checkpoint flow, and accepted-artifact validation at the responsibility
level without listing volatile private helper names.
3. Confirm the operator-visible reason-code table has one canonical owner and
other documents link to it rather than maintaining parallel copies.
4. Remove stale implementation claims exposed by this work and ensure the
feature roadmap continues to describe target policy rather than task
sequencing.
### Verification ### Verification
Run: Run the repository-prescribed commands from `docs/development.md`:
```sh
go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./internal/modules/dnd/extract/npcs ./internal/modules/dnd/extract/combatturns
```
### Completion Criteria
- Only one production implementation prepares chunk prompt material.
- Extractor behavior and public errors retain useful module context.
- Prompt preparation and shared input behavior tests still pass.
## Stage 5: Share Cited Source Traversal
### Goal
Give all relatedness validators one deterministic implementation for resolving,
ordering, and deduplicating cited source units while keeping matching semantics
artifact-specific.
### Implementation
1. Add `internal/modules/dnd/shared/citations.go` with:
```go
func CitedText(doc *source.SourceDocument, refs []source.SourceRef) (string, error)
```
2. Validate every range with `source.ValidateRef`. Resolve ranges against
document order, include each covered source unit once even when ranges
overlap, and join included unit text with a single newline. Return an error
for a nil document or any invalid range. Return `"", nil` for an empty ref
slice with a non-nil document. Do not mutate the document or refs.
3. Add table-driven tests for nil documents, empty refs, invalid source IDs,
unknown/reversed unit IDs, disjoint ranges supplied out of order, adjacent
ranges, and overlapping/duplicate ranges. Output must always follow document
order.
4. Replace spell, NPC, and combat relatedness validators' local cited-text
traversal with `shared.CitedText`.
5. When shape is invalid or `CitedText` returns an error, relatedness validators
approve without relatedness warnings so shape/source-reference validators
remain the sole owners of those defects.
6. Keep matching local:
- spells compare the canonical spell name case-insensitively against combined
cited text;
- NPCs use `identity.ComparisonKey` for names and aliases; and
- combat uses its existing comparison-key actor logic and declaration token
heuristic.
7. Strengthen matching tests with Unicode/apostrophe variants, multiword names,
overlapping ranges, and a short-name substring false-positive case. For
names and actors, require token/word-boundary-aware matching rather than raw
substring matching. For multiword values, match the consecutive normalized
token sequence. Keep the combat declaration rule of at least one normalized
token of four or more runes.
8. Put shared normalized tokenization and consecutive-token matching in
`internal/modules/dnd/shared` only if at least two validators use it after
the change. Otherwise leave the matching helper local; do not create a
single configurable matching engine.
### Verification
Run:
```sh
go test ./internal/modules/dnd/shared ./internal/modules/dnd/validate/spells/source_relatedness ./internal/modules/dnd/validate/npcs/source_relatedness ./internal/modules/dnd/validate/combatturns/source_relatedness
```
### Completion Criteria
- Cited range validation, ordering, overlap handling, and text assembly have
one production owner.
- Relatedness remains warning-only and ignores invalid prerequisite data.
- Artifact-specific matching policies remain understandable in their validator
packages.
## Stage 6: Align Validator Policy And Diagnostics
### Goal
Make validator checkpoint identity, prerequisite handling, and diagnostic
bounding consistent without changing validator-chain order or durable reason
codes.
### Implementation
1. Add these local policy constants and `CheckpointFingerprintProvider`
implementations:
- spell shape: `dnd.spells.validator.shape.v1`;
- spell source refs: `dnd.spells.validator.source_refs.v1`; and
- spell source relatedness: `dnd.spells.validator.source_relatedness.v1`.
2. Do not add a separate policy fingerprint to the spell catalog validator; its
effective catalog digest remains its existing semantic checkpoint identity.
If its non-catalog validation policy changes during this work, add a second
`policy` fingerprint rather than replacing `effective_catalog`.
3. Change spell source-reference validation to approve when spell shape is
invalid, matching NPC and combat prerequisite behavior.
4. Change spell source-reference validation to collect all reference issues,
truncate individual errors with `shared/diagnostics.Truncate`, and return a
bounded aggregate with `shared/diagnostics.Aggregate`. Preserve
`invalid_source_refs`.
5. Confirm NPC and combat source-reference validators follow the same
prerequisite and bounded-aggregate policy; refactor only enough to share
obvious local structure. Do not introduce a generic typed validator builder.
6. Keep shape validators responsible for malformed artifact fields and source
reference validators responsible for document/range validity.
7. Add or consolidate package-level tests for policy fingerprints, invalid-shape
deferral, aggregation of multiple invalid refs, bounded diagnostics, strict
options, registration, and deterministic execution class. Prefer a small
table of behavioral expectations in each typed package over a reflection-
based cross-package harness.
8. Update `docs/internal/modules.md` to describe the aligned prerequisite and
checkpoint policy after it is implemented.
### Verification
Run:
```sh
go test ./internal/modules/dnd/validate/...
go test ./internal/modules/dnd/register
```
### Completion Criteria
- Every deterministic validator policy affecting checkpoint reuse has an
explicit semantic fingerprint.
- Later validators do not duplicate shape rejection.
- All source-reference diagnostic output is bounded.
- Existing reason codes and validator-chain order are unchanged.
## Stage 7: Clarify D&D Registration And Naming
### Goal
Make production composition easier to audit without introducing heterogeneous
generic descriptors or changing registration behavior.
### Implementation
1. Keep package `internal/modules/dnd/register`, but split its current concerns
into focused files:
- `register.go`: public `Register`, registry validation, and ordered execution
of named registration functions;
- `modules.go`: codecs, chunker, extractors, mergers, normalizers, no-op
normalizers, and prompt asset registrations;
- `validators.go`: production and generic test validator registrations;
- `chains.go`: the six default extract/normalize chain mappings; and
- `merge.go`: typed append functions and deep-clone helpers.
2. Use small private functions such as `registerModules`,
`registerValidators`, `registerPromptAssets`, and
`registerDefaultChains`. Keep the existing ordered `registration{name,
register}` error-context pattern within each group.
3. Do not create one slice containing generic lane descriptors; Go cannot retain
the heterogeneous typed codec and module relationships there without
erasure or callbacks that obscure more than they clarify.
4. Keep append and clone behavior in the `register` package for this change.
Moving it would require a new lane-ownership package with no independent
domain responsibility.
5. Normalize import aliases in the registrar to the pattern
`spellextract`, `npcextract`, `combatextract`, `spellnormalize`,
`npcnormalize`, and `combatnormalize`, with corresponding validator aliases.
This is internal naming only.
6. Preserve registration order, error prefixes, default chain contents and
order, reference-slot/spec behavior, and prompt asset collection.
7. Update `register_test.go` only as required by file movement. Tests should
continue asserting observable registry contents and chain policy, not the
new private helper call graph.
### Verification
Run:
```sh
go test ./internal/modules/dnd/register
go test ./internal/modules/dnd/...
```
### Completion Criteria
- Production composition is grouped by responsibility and remains explicit.
- The refactor produces no registry, chain, capability, or error behavior
change.
- Tests do not couple to private registration helpers.
## Stage 8: Final Integration, Documentation, And Evaluation
### Goal
Verify the complete target state, update canonical current-behavior documents,
and gather quality/cache evidence without making live services part of the
default test suite.
### Implementation And Review
1. Re-read `docs/roadmap/dnd.md` and verify every completion criterion against
production code and tests. Do not mark an item complete based only on this
implementation plan.
2. Review naming across the three lanes. Harmonize internal aliases and private
policy constant names, but do not rename compatibility-sensitive identifiers
listed in `Fixed Decisions And Constraints`.
3. Review prompt and durable schemas for accidental duplication. Make no schema
refactor unless composition already exists and the change is behavior-free;
the fixed decision for this plan is to leave them package-owned and separate.
4. Consolidate redundant tests created by intermediate stages. Retain:
- one shared manifest test suite;
- behavior-level prompt preparation tests without prefix-length snapshots;
- focused package tests for artifact-specific prompts and validators; and
- existing production registration contract coverage.
5. Update `docs/internal/modules.md` and `docs/internal/llm.md` so they are the
canonical description of the final implemented behavior. Remove superseded
implementation details rather than appending a second description.
6. Update `docs/roadmap/dnd.md` to record implementation status. Remove completed
future-work details that are fully owned by current internal documentation,
leaving only genuinely deferred outcomes. Do not turn the feature roadmap
into a second current-behavior reference.
7. If credentials and the maintained human-reviewed transcript set are
available, run an explicitly opt-in comparison using identical profiles and
inputs before and after the prompt change. Record only aggregate extraction
review results, prompt token counts, cached-token/cache-write metrics, and
non-secret prompt hashes. Never commit transcript content, rendered prompts,
credentials, endpoints, or private reference material.
8. Live evaluation is not a merge gate when credentials or reviewed fixtures
are unavailable. In that case, record the missing external prerequisite in
the remaining roadmap item; do not add a fake cache-hit claim and do not
weaken offline identity tests.
### Repository Verification
Run all required checks:
```sh ```sh
go test ./... go test ./...
go vet ./... go vet ./...
go build ./cmd/notarius go build ./cmd/notarius
git diff --check
``` ```
Also inspect the final diff for: Also run race tests for the pipeline, checkpoint, CLI state, D&D NPC registry,
and D&D integration packages. Validate maintained examples/configurations
through the production catalog. Review one fresh run, ordinary resumed run,
forced-recompute run, hydrated-predecessor run, and invalid-predecessor failure
for deterministic ordering, correct decisions, bounded provenance, and absence
of generated content, secrets, or local paths in manifests and diagnostics.
- unintended public identifier or durable schema changes; ### Completion Gate
- prompt rules duplicated between shared and local assets;
- prompt assets rendered but absent from fingerprints, or fingerprinted but
not rendered;
- provider-specific types outside allowed boundaries;
- tests containing real transcript/reference material or credentials; and
- unrelated changes in a pre-existing dirty worktree.
### Completion Criteria The follow-up is complete only when every finding summarized above is protected
by a stable behavioral test, the current documentation matches the corrected
- Every completion criterion in the feature roadmap is either implemented and implementation, the full verification suite passes, and the worktree contains
documented in its canonical current-behavior owner or explicitly retained as no temporary adapters or TODOs introduced by these stages.
deferred roadmap work.
- All focused and repository-wide checks pass.
- The final test suite protects behavioral contracts without retaining
redundant implementation snapshots.
## Open Questions ## Open Questions
None. The plan fixes all choices required for implementation. Live provider None. The feature roadmap and this plan fix the required product and
evaluation may depend on credentials and reviewed fixtures, but that is an architecture choices. If implementation reveals that accepted normalized
external acceptance prerequisite rather than an unresolved design decision. artifacts cannot be validated safely without a persistent format change, stop
and amend this plan rather than weakening identity, codec, or content
validation.

View File

@@ -0,0 +1,339 @@
# Scope: Ordered Pipeline Steps
## Status
Implemented. This document preserves the bounded feature policy, architecture
choices, acceptance criteria, and exclusions. Current behavior belongs in the
canonical [CLI](../cli.md), [Configuration](../config.md),
[Operations](../operations.md), and [internal pipeline](../internal/pipeline.md)
documentation rather than in this roadmap.
## Policy Recommendation
Treat ordered pipeline steps, generated artifact references, and
dependency-aware checkpoint reuse as one coherent platform capability. The D&D
proving workflow produces accepted normalized NPC output first and then supplies
it to spell extraction, combat-turn extraction, and combat-turn normalization.
## Intended Outcome
A configured pipeline may contain multiple ordered steps while retaining one
pipeline-wide input, chunk plan, output, worker budget, LLM scheduler, run
manifest, and failure boundary. Every artifact lane still follows the fixed
extract, validate, merge, validate, normalize, and validate lifecycle. Steps
add explicit barriers between groups of lanes; they do not create arbitrary
stage graphs.
An accepted normalized artifact from an earlier step may be bound explicitly
to declared reference slots in a later step. The framework remains
domain-neutral, and generated references remain contextual material rather than
source evidence.
## Fixed Product And Architecture Decisions
### Pipeline shape
- Input parsing and chunk planning remain pipeline-wide and execute once.
- A step contains one or more artifact lanes. Step order is configuration order.
- Lanes within a step remain independent and may use the existing bounded
concurrency model.
- A later step cannot begin until every lane in the current step is terminal
and every generated artifact it requires is accepted and available.
- Public artifact and failure ordering is step order followed by deterministic
lane and source-chunk order, never completion order.
- Output encoding occurs once, after every step succeeds.
- This is not an arbitrary DAG, a general workflow language, concurrent
cross-lane reconciliation, or permission for modules to invoke other modules.
### Configuration model
Existing single-step pipelines remain valid. A top-level `artifacts` map is
treated as an implicit step with stable ID `default`. A pipeline may configure
either `artifacts` or `steps`, but not both. Explicit steps must be non-empty
and have unique, trimmed, non-empty IDs. Artifact lane IDs must remain unique
across the entire pipeline so output paths, selectors, manifests, errors, and
checkpoint scopes remain unambiguous.
The target configuration shape is:
```yaml
pipelines:
dnd-session:
input: seriatim
chunk: generic
steps:
- id: identify-npcs
artifacts:
npcs:
extract: dnd/npcs
normalize: dnd/npcs
- id: grounded-events
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells
combat:
extract: dnd/combat-turns
normalize: dnd/combat-turns
output: json
```
Existing scalar reference values continue to represent external file paths.
The structured `artifact` form identifies accepted normalized output from one
earlier step and lane. Generated artifact bindings are allowed at step scope or
at an individual module target; they are not inferred from module keys, lane
names, slot names, or domain knowledge.
A step-scoped reference applies automatically to every selected target in that
step that declares the slot. In the example, one `npcs` binding reaches the
spell extractor plus the combat extractor and normalizer. A target-local
binding is used when only one module should consume the artifact.
Pipeline-level external references remain defaults. Step-local external
references override pipeline-level external defaults, and target-local
external references retain their existing precedence. A generated reference
and an external reference may not resolve to the same effective target slot;
configuration or a runtime override that creates that conflict is invalid.
Likewise, a step-scoped and target-local generated binding cannot both target
the same effective slot.
Each effective target slot accepts at most one producer. One producer may fan
out to multiple compatible slots in a later step. Aggregating several producer
artifacts into one slot is outside this scope.
Reference-slot specs gain optional generated-artifact compatibility metadata.
A generated binding is allowed only when the consumer slot declares the
producer's artifact kind; the producer's registered codec supplies the exact
schema identity and media type used for the handoff. The D&D `npcs` consumer
slots declare the normalized NPC-list artifact kind. Existing external-file
slots and bindings retain their current behavior and do not acquire an artifact
kind merely because their bytes happen to decode as one.
### Resolution and preparation
Resolution validates the complete ordered structure before source processing.
It must reject duplicate identities, missing producers, same-step or forward
references, undeclared slots, reference conflicts, and incompatible artifact
kind, schema, media type, or cardinality constraints that are statically
discoverable. Size is checked when canonical producer bytes exist at handoff.
Ordered steps make cycles structurally impossible; resolution must not
introduce a general graph scheduler to rediscover their order.
The resolved pipeline and its digest include step order, step IDs, lane
membership, generated-reference topology, producer identity, consumer targets,
and existing module and validator policy. Cloning, redaction, canonical JSON,
debug summaries, and manifests preserve the same structure without reference
content or secrets.
All modules and validators are still selected, option-validated, and
constructed before source parsing. Generated content cannot be supplied during
construction because it does not exist yet. The framework therefore augments
the existing operation-request `References` at the step boundary. Consumers
that currently assume an NPC registry is construction-only must accept the
generated registry from their operation request without deferring general
module construction until after upstream work.
Only validation that inherently depends on generated bytes may occur at the
handoff. A handoff validation failure is a contextual framework error and fails
the run before any consumer in that step begins.
### Generated artifact handoff
Only accepted normalized output may cross a step boundary. Raw extraction
responses, rejected artifacts, merge intermediates, and validator diagnostics
cannot be bound as references.
The framework serializes the producer through its registered canonical artifact
codec and constructs one immutable reference item containing:
- the declared target slot;
- canonical artifact bytes and media type;
- artifact kind and schema ID, name, version, and schema digest;
- canonical content digest and size; and
- producer pipeline, step, lane, and module provenance.
A generated binding requires exactly one accepted normalized artifact from its
producer lane. No artifact is a missing dependency, while more than one is a
cardinality error; a typed collection such as an NPC list remains one artifact.
Combining several normalized outputs into one reference is aggregation and is
outside this scope.
The existing slot contract remains authoritative for accepted media types,
maximum size, and cardinality. Generated content is cloned at ownership
boundaries and never exposed through a filesystem path. Manifests and debug
summaries record identities and bounded provenance, not artifact content.
Configuring a generated binding makes that dependency required even when the
consumer module declares the underlying slot optional. An accepted artifact
whose domain collection is empty is still a valid artifact and may be handed
off. If the producer has no accepted normalized artifact, the entire run fails
with a deterministic dependency error and no later step begins.
### Checkpoint reuse and selective recomputation
Generated references participate in downstream checkpoint dependencies by
artifact kind, complete schema identity, media type, and canonical content
digest. The pipeline digest protects topology; stage dependency fingerprints
protect the exact upstream artifact consumed. The runner must never combine a
new or changed producer with stale dependent output.
Ordinary resume may progressively decode compatible producer and consumer stage
checkpoints through the registered codec. Selective recomputation may hydrate a
required unselected producer directly from its accepted normalized artifact;
its extract and merge state are not prerequisites. Missing, rejected, corrupt,
incompatible, or changed accepted state stops the run before dependent
execution rather than implicitly rerunning the producer. Independent work
remains reusable.
The operator control `--recompute-step <step-id>` has these semantics:
- it requires checkpoint recording and `--resume`;
- the selected step and all transitive dependents execute rather than reuse
their checkpoints;
- valid required predecessors and unrelated work remain reusable;
- the recompute selection affects loader decisions, not the persistent
checkpoint identity of otherwise identical work; and
- the command fails before dependent execution if a required predecessor has
no reusable accepted artifact.
Existing `--only` behavior remains unchanged for implicit single-step
pipelines. Combining `--only` with explicit multi-step pipelines is outside
this scope and should be rejected with actionable guidance rather than given
implicit dependency-expansion semantics.
Checkpoint events, manifests, and diagnostics distinguish executed, reused,
forced-recomputed, and dependency-invalidated work. Invalidation reasons are
bounded, deterministic, and free of reference content, local paths, or secrets.
Old checkpoint state need not be migrated; it must produce a safe, explicit
cold miss rather than an error or unsafe reuse.
### Failure, cancellation, and concurrency
The existing run-wide worker and provider-call limits apply across every step.
Workers may be reused between steps, but concurrency cannot cross a step
barrier. A framework error cancels started work using the existing bounded
drain behavior and prevents later steps and output encoding. Rejections remain
recorded outcomes, but failure to produce a normalized artifact required by a
generated binding escalates to the run-level dependency error described above.
The failed manifest retains completed upstream outcomes, step and lane
provenance, rejections, checkpoint events, and the dependency failure without
embedding generated artifact content.
## D&D Proving Workflow
The production acceptance workflow has two explicit steps:
1. `identify-npcs` runs the NPC lane through normalization and its complete
validator policy.
2. `grounded-events` receives the canonical NPC artifact in its step-scoped
`npcs` reference and runs spell and combat-turn lanes. The binding reaches
spell extraction, combat-turn extraction, and combat-turn normalization.
Spell and combat-turn lanes may execute concurrently after the handoff. NPC
content may ground names and identities but cannot establish a spell cast or
combat event; source units remain the only event evidence.
The maintained workflow uses one ordered-pipeline example instead of a manual
two-run NPC-to-spell or NPC-to-combat handoff. Existing module keys, artifact
contracts, reference slot names, prompt IDs, and D&D evidence policy remain
unchanged.
## Included Work
- Configuration parsing, validation, cloning, defaults, redaction, and
documentation for explicit steps and structured generated references.
- Domain-neutral resolved step, dependency, producer, and consumer identities.
- Generated-artifact compatibility metadata on reference-slot contracts,
including D&D NPC-list declarations for every `npcs` consumer.
- Step-aware preparation metadata and runner orchestration.
- Canonical codec handoff into existing reference request contracts.
- Required-dependency failure and bounded provenance behavior.
- Dependency-aware checkpoint reuse, invalidation, events, and selective step
recomputation.
- D&D NPC-first production composition for spell and combat-turn consumers.
- Refactoring the affected D&D consumers so generated NPC references are
available at operation time while retaining early static construction.
- Maintained examples and current architecture, configuration, CLI, operations,
internal, integration, and testing documentation.
- An ADR recording the bounded ordered-step extension to the fixed pipeline
architecture and its explicit rejection of a general DAG.
## Explicitly Out Of Scope
- D&D item extraction or any other new artifact lane.
- Cross-artifact NPC ID fields or artifact-schema migration machinery.
- Arbitrary DAGs, conditional branches, loops, joins, dynamic step creation, or
module-controlled scheduling.
- Multiple source inputs, per-step input adapters, per-step chunk plans, or
per-step output encoders.
- Aggregating multiple generated artifacts into one reference slot.
- Optional or best-effort generated dependencies; a configured dependency is
required in this scope.
- Prior-run or cross-pipeline generated references.
- `--only` dependency closure for explicit multi-step pipelines.
- Cross-lane reconciliation or domain concepts in the generic framework.
- Live-provider tests or model-quality changes to D&D prompts.
## Acceptance Criteria
The scope is complete when:
- all existing single-step configurations retain their current behavior;
- explicit step order and dependency topology resolve deterministically and
affect pipeline identity;
- invalid producer, consumer, conflict, ordering, type, schema, media, and
cardinality configurations fail before source processing when statically
discoverable, while content-size violations fail at handoff;
- no consumer step begins before all required generated artifacts are accepted,
canonicalized, and validated for its target slots;
- one producer artifact fans out safely to every compatible target selected by
a step-scoped binding;
- missing required producer output fails the complete run before dependent work;
- changing NPC output invalidates spell and combat-turn checkpoints while
leaving compatible independent work reusable;
- selective step recomputation executes exactly the selected dependency closure
and reports why work was executed, reused, or invalidated;
- the D&D ordered workflow supplies NPC content to spell extraction, combat-turn
extraction, and combat-turn normalization without treating it as evidence;
- completion timing cannot change public ordering, failure selection, or
dependency behavior;
- output, manifests, checkpoints, and debug artifacts contain the required
identities and provenance without leaking generated reference content; and
- repository-wide tests, vet, build, maintained-example checks, and
documentation validation pass.
## Testing Strategy
Tests should protect behavior and invariants rather than the implementation's
internal scheduler shape.
- Configuration contract tests own legacy shorthand, explicit step parsing,
source-form discrimination, conflicts, and redaction.
- Resolution tests own ordering, global lane uniqueness, dependency validation,
slot compatibility, fan-out, cloning, canonical JSON, and digest changes.
- Runner tests own step barriers, within-step bounded concurrency, stable
ordering, cancellation, required-producer failure, and immutable handoff.
- Checkpoint tests own producer decoding, exact dependency matching, transitive
invalidation, forced recomputation, cold misses, and bounded decisions.
- One CLI contract test should cover the recompute control and its invalid
combinations.
- One D&D integration test with offline fake LLM responses should prove the
complete NPC-to-spell-and-combat handoff, including combat normalization.
- Maintained configuration examples should be parsed and resolved through the
production catalog.
Do not add scheduler choreography tests, exact goroutine-count assertions,
complete manifest snapshots, exact diagnostic strings, or duplicated tests for
every invalid configuration at every layer. No test may require credentials or
a live model provider.
## Open Questions
None required to define this scope. Exact internal type names and implementation
decomposition are intentionally not feature-policy decisions.

View File

@@ -1,31 +0,0 @@
version: 3
output:
directory: ./notarius-output
cache:
chunk_plans:
mode: bypass
directory: ./notarius-cache/chunk-plans
checkpoints:
enabled: false
directory: ./notarius-cache/checkpoints
debug:
directory: ./notarius-debug
pipelines:
dnd-npcs:
input: seriatim
chunk: generic
artifacts:
npcs:
extract:
module: dnd/npcs
retries: 2
normalize: dnd/npcs
dnd-combat:
input: seriatim
chunk: generic
artifacts:
combat:
extract:
module: dnd/combat-turns
retries: 2
normalize: dnd/combat-turns

View File

@@ -0,0 +1,37 @@
version: 3
output:
directory: ./notarius-output
cache:
chunk_plans:
mode: bypass
checkpoints:
enabled: false
directory: ""
debug:
directory: ./notarius-debug
pipelines:
dnd-npc-grounded:
input: seriatim
steps:
- id: identify-npcs
artifacts:
npcs:
extract:
module: dnd/npcs
retries: 2
normalize: dnd/npcs
- id: grounded-events
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells
combat:
extract:
module: dnd/combat-turns
retries: 2
normalize: dnd/combat-turns

View File

@@ -1,28 +0,0 @@
version: 3
output:
directory: ./notarius-output
cache:
chunk_plans:
mode: bypass
checkpoints:
enabled: false
directory: ""
debug:
directory: ./notarius-debug
pipelines:
dnd-npcs:
input: seriatim
chunk: generic
artifacts:
npcs:
extract:
module: dnd/npcs
retries: 2
normalize: dnd/npcs
dnd-spells:
input: seriatim
chunk: generic
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells

View File

@@ -24,10 +24,10 @@ func TestProductionCombatConfigurationResolvesTypedLane(t *testing.T) {
if effective.ResolvedPipeline.Chunk.Module != pipeline.DefaultChunkModule { if effective.ResolvedPipeline.Chunk.Module != pipeline.DefaultChunkModule {
t.Fatalf("chunk module = %q, want %q", effective.ResolvedPipeline.Chunk.Module, pipeline.DefaultChunkModule) t.Fatalf("chunk module = %q, want %q", effective.ResolvedPipeline.Chunk.Module, pipeline.DefaultChunkModule)
} }
if len(effective.ResolvedPipeline.ArtifactLanes) != 1 { if len(effective.ResolvedPipeline.Steps[0].ArtifactLanes) != 1 {
t.Fatalf("artifact lanes = %#v, want one combat lane", effective.ResolvedPipeline.ArtifactLanes) t.Fatalf("artifact lanes = %#v, want one combat lane", effective.ResolvedPipeline.Steps[0].ArtifactLanes)
} }
lane := effective.ResolvedPipeline.ArtifactLanes[0] lane := effective.ResolvedPipeline.Steps[0].ArtifactLanes[0]
if lane.ID != "combat" || lane.ArtifactKind != dnd.CombatTurnListKind || lane.Extract.Module != combatextract.Key || lane.Extract.Retries != 2 || lane.Merge.Module != pipeline.DefaultMergeModule || lane.Normalize.Module != combatnormalize.Key { if lane.ID != "combat" || lane.ArtifactKind != dnd.CombatTurnListKind || lane.Extract.Module != combatextract.Key || lane.Extract.Retries != 2 || lane.Merge.Module != pipeline.DefaultMergeModule || lane.Normalize.Module != combatnormalize.Key {
t.Fatalf("resolved combat lane = %#v, want typed production composition", lane) t.Fatalf("resolved combat lane = %#v, want typed production composition", lane)
} }
@@ -89,7 +89,7 @@ func TestProductionCombatConfigurationResolvesTypedLane(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("Resolve(bound references) error = %v, want nil", err) t.Fatalf("Resolve(bound references) error = %v, want nil", err)
} }
boundLane := bound.ResolvedPipeline.ArtifactLanes[0] boundLane := bound.ResolvedPipeline.Steps[0].ArtifactLanes[0]
if len(boundLane.ExtractReferences.Bindings) != 1 || len(boundLane.NormalizeReferences.Bindings) != 1 || boundLane.ExtractReferences.Bindings[0].SlotName != "npcs" || boundLane.NormalizeReferences.Bindings[0].SlotName != "npcs" { if len(boundLane.ExtractReferences.Bindings) != 1 || len(boundLane.NormalizeReferences.Bindings) != 1 || boundLane.ExtractReferences.Bindings[0].SlotName != "npcs" || boundLane.NormalizeReferences.Bindings[0].SlotName != "npcs" {
t.Fatalf("bound combat references = %#v / %#v, want one independent NPC binding per stage", boundLane.ExtractReferences, boundLane.NormalizeReferences) t.Fatalf("bound combat references = %#v / %#v, want one independent NPC binding per stage", boundLane.ExtractReferences, boundLane.NormalizeReferences)
} }

View File

@@ -24,10 +24,10 @@ func TestProductionNPCConfigurationResolvesTypedLane(t *testing.T) {
if effective.ResolvedPipeline.Chunk.Module != pipeline.DefaultChunkModule { if effective.ResolvedPipeline.Chunk.Module != pipeline.DefaultChunkModule {
t.Fatalf("chunk module = %q, want %q", effective.ResolvedPipeline.Chunk.Module, pipeline.DefaultChunkModule) t.Fatalf("chunk module = %q, want %q", effective.ResolvedPipeline.Chunk.Module, pipeline.DefaultChunkModule)
} }
if len(effective.ResolvedPipeline.ArtifactLanes) != 1 { if len(effective.ResolvedPipeline.Steps[0].ArtifactLanes) != 1 {
t.Fatalf("artifact lanes = %#v, want one NPC lane", effective.ResolvedPipeline.ArtifactLanes) t.Fatalf("artifact lanes = %#v, want one NPC lane", effective.ResolvedPipeline.Steps[0].ArtifactLanes)
} }
lane := effective.ResolvedPipeline.ArtifactLanes[0] lane := effective.ResolvedPipeline.Steps[0].ArtifactLanes[0]
if lane.ID != "npcs" || lane.ArtifactKind != dnd.NPCListKind || lane.Extract.Module != npcextract.Key || lane.Extract.Retries != 2 || lane.Merge.Module != pipeline.DefaultMergeModule || lane.Normalize.Module != npcnormalize.Key { if lane.ID != "npcs" || lane.ArtifactKind != dnd.NPCListKind || lane.Extract.Module != npcextract.Key || lane.Extract.Retries != 2 || lane.Merge.Module != pipeline.DefaultMergeModule || lane.Normalize.Module != npcnormalize.Key {
t.Fatalf("resolved NPC lane = %#v, want typed production composition", lane) t.Fatalf("resolved NPC lane = %#v, want typed production composition", lane)
} }
@@ -101,11 +101,11 @@ func TestProductionNPCConfigurationValidatesOptionsReferencesAndPlacement(t *tes
t.Fatalf("unknown normalizer option error = %v, want strict option rejection", err) t.Fatalf("unknown normalizer option error = %v, want strict option rejection", err)
} }
if err := resolve(func(profile *pipeline.PipelineProfile) { if err := resolve(func(profile *pipeline.PipelineProfile) {
profile.References = map[string]string{ profile.References = pipeline.ExternalReferenceMap(map[string]string{
"players": "players.txt", "players": "players.txt",
"party": "party.txt", "party": "party.txt",
"glossary": "glossary.txt", "glossary": "glossary.txt",
} })
}); err != nil { }); err != nil {
t.Fatalf("optional NPC references error = %v, want resolution success", err) t.Fatalf("optional NPC references error = %v, want resolution success", err)
} }

View File

@@ -33,10 +33,10 @@ func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
t.Fatalf("materialize maintained example references for %q: %v", pipelineID, err) t.Fatalf("materialize maintained example references for %q: %v", pipelineID, err)
} }
if example.name == "production" { if example.name == "production" {
if len(materialized.ArtifactLanes) != 1 || if len(materialized.Steps[0].ArtifactLanes) != 1 ||
len(materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["spell_catalog"].Items) != 1 || len(materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["spell_catalog"].Items) != 1 ||
len(materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"].Items) != 1 { len(materialized.Steps[0].ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"].Items) != 1 {
t.Fatalf("production spell catalog reference was not materialized: %#v", materialized.ArtifactLanes) t.Fatalf("production spell catalog reference was not materialized: %#v", materialized.Steps[0].ArtifactLanes)
} }
} }
} }

View File

@@ -22,7 +22,7 @@ func TestOversizedNPCRegistryFailsBeforeRuntimeAndCheckpointConstruction(t *test
t.Fatal(err) t.Fatal(err)
} }
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints") checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
content := string(readRepositoryFile(t, "examples", "dnd-npc-spell-sequential.config.yml")) content := string(readRepositoryFile(t, "examples", "dnd-spells.config.yml"))
content = replaceRequiredOnce(t, content, " extract: dnd/spells", " extract:\n module: dnd/spells\n references:\n npcs: "+npcPath) content = replaceRequiredOnce(t, content, " extract: dnd/spells", " extract:\n module: dnd/spells\n references:\n npcs: "+npcPath)
content = replaceRequiredOnce(t, content, " enabled: false\n directory: \"\"", " enabled: true\n directory: "+checkpointRoot) content = replaceRequiredOnce(t, content, " enabled: false\n directory: \"\"", " enabled: true\n directory: "+checkpointRoot)
configPath := filepath.Join(t.TempDir(), "config.yml") configPath := filepath.Join(t.TempDir(), "config.yml")
@@ -46,11 +46,11 @@ func TestOversizedNPCRegistryFailsBeforeRuntimeAndCheckpointConstruction(t *test
} }
var stdout, stderr strings.Builder var stdout, stderr strings.Builder
code := RunWithOptions([]string{ code := RunWithOptions([]string{
"run", "dnd-spells", "--config", configPath, "run", "dnd-session", "--config", configPath,
"--input", repositoryPath("examples", "seriatim-minimal-transcript.json"), "--input", repositoryPath("examples", "seriatim-minimal-transcript.json"),
"--chunk_cache", "bypass", "--output-dir", t.TempDir(), "--chunk_cache", "bypass", "--output-dir", t.TempDir(),
}, &stdout, &stderr, options) }, &stdout, &stderr, options)
for _, fragment := range []string{`pipeline "dnd-spells"`, `reference slot "npcs"`, "1048577 bytes", "limit 1048576"} { for _, fragment := range []string{`pipeline "dnd-session"`, `reference slot "npcs"`, "1048577 bytes", "limit 1048576"} {
if code == 0 || !strings.Contains(stderr.String(), fragment) { if code == 0 || !strings.Contains(stderr.String(), fragment) {
t.Fatalf("RunWithOptions() code = %d stderr = %q, want context fragment %q", code, stderr.String(), fragment) t.Fatalf("RunWithOptions() code = %d stderr = %q, want context fragment %q", code, stderr.String(), fragment)
} }

View File

@@ -187,8 +187,8 @@ func TestProductionSpellValidatorsPrepareFromMaterializedCatalog(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("materialize production spell references: %v", err) t.Fatalf("materialize production spell references: %v", err)
} }
extractItems := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["spell_catalog"].Items extractItems := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["spell_catalog"].Items
normalizeItems := materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"].Items normalizeItems := materialized.Steps[0].ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"].Items
if len(extractItems) != 1 || extractItems[0].MediaType != "application/json" || len(extractItems[0].Content) == 0 { if len(extractItems) != 1 || extractItems[0].MediaType != "application/json" || len(extractItems[0].Content) == 0 {
t.Fatalf("materialized extract spell catalog items = %#v, want one JSON item", extractItems) t.Fatalf("materialized extract spell catalog items = %#v, want one JSON item", extractItems)
} }
@@ -243,9 +243,9 @@ func TestProductionSpellNormalizerRejectsInvalidCatalogReferencesBeforeExecution
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
slot := materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"] slot := materialized.Steps[0].ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"]
slot.Items = append(slot.Items, slot.Items[0]) slot.Items = append(slot.Items, slot.Items[0])
materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"] = slot materialized.Steps[0].ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["spell_catalog"] = slot
_, err = pipeline.Prepare(materialized, components.registries, pipeline.ModuleDependencies{LLM: &productionFakeLLMClient{}}) _, err = pipeline.Prepare(materialized, components.registries, pipeline.ModuleDependencies{LLM: &productionFakeLLMClient{}})
for _, fragment := range []string{"normalize", `module "dnd/spells"`, "zero or one item"} { for _, fragment := range []string{"normalize", `module "dnd/spells"`, "zero or one item"} {
if err == nil || !strings.Contains(err.Error(), fragment) { if err == nil || !strings.Contains(err.Error(), fragment) {
@@ -307,10 +307,10 @@ func TestProductionSpellNormalizerRejectsInvalidCatalogReferencesBeforeExecution
func setNormalizeSpellCatalogSource(t *testing.T, resolved *pipeline.ResolvedPipeline, sourcePath string) { func setNormalizeSpellCatalogSource(t *testing.T, resolved *pipeline.ResolvedPipeline, sourcePath string) {
t.Helper() t.Helper()
if resolved == nil || len(resolved.ArtifactLanes) != 1 { if resolved == nil || len(resolved.Steps[0].ArtifactLanes) != 1 {
t.Fatalf("resolved pipeline = %#v, want one artifact lane", resolved) t.Fatalf("resolved pipeline = %#v, want one artifact lane", resolved)
} }
bindings := resolved.ArtifactLanes[0].NormalizeReferences.Bindings bindings := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings
matches := 0 matches := 0
for index := range bindings { for index := range bindings {
if bindings[index].SlotName == "spell_catalog" { if bindings[index].SlotName == "spell_catalog" {
@@ -321,7 +321,7 @@ func setNormalizeSpellCatalogSource(t *testing.T, resolved *pipeline.ResolvedPip
if matches != 1 { if matches != 1 {
t.Fatalf("normalize reference bindings = %#v, want exactly one spell_catalog binding", bindings) t.Fatalf("normalize reference bindings = %#v, want exactly one spell_catalog binding", bindings)
} }
resolved.ArtifactLanes[0].NormalizeReferences.Bindings = bindings resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings = bindings
} }
func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) { func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
@@ -504,9 +504,8 @@ func maintainedExampleFiles(t *testing.T) []maintainedExample {
{name: "minimal", path: repositoryPath("examples", "dnd-spells.config.yml"), pipelineIDs: []string{"dnd-session"}}, {name: "minimal", path: repositoryPath("examples", "dnd-spells.config.yml"), pipelineIDs: []string{"dnd-session"}},
{name: "production", path: repositoryPath("examples", "dnd-spells-production.config.yml"), pipelineIDs: []string{"dnd-session"}}, {name: "production", path: repositoryPath("examples", "dnd-spells-production.config.yml"), pipelineIDs: []string{"dnd-session"}},
{name: "npcs", path: repositoryPath("examples", "dnd-npcs.config.yml"), pipelineIDs: []string{"dnd-session"}}, {name: "npcs", path: repositoryPath("examples", "dnd-npcs.config.yml"), pipelineIDs: []string{"dnd-session"}},
{name: "sequential", path: repositoryPath("examples", "dnd-npc-spell-sequential.config.yml"), pipelineIDs: []string{"dnd-npcs", "dnd-spells"}},
{name: "combat", path: repositoryPath("examples", "dnd-combat-turns.config.yml"), pipelineIDs: []string{"dnd-combat"}}, {name: "combat", path: repositoryPath("examples", "dnd-combat-turns.config.yml"), pipelineIDs: []string{"dnd-combat"}},
{name: "npc-combat-sequential", path: repositoryPath("examples", "dnd-npc-combat-sequential.config.yml"), pipelineIDs: []string{"dnd-combat", "dnd-npcs"}}, {name: "npc-grounded", path: repositoryPath("examples", "dnd-npc-grounded.config.yml"), pipelineIDs: []string{"dnd-npc-grounded"}},
} }
} }

View File

@@ -0,0 +1,323 @@
package cli
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"sort"
"strings"
"sync"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestRecomputeStepRecoversThroughFilesystemCheckpoints(t *testing.T) {
tests := []struct {
name string
invalidateOutput bool
wantCode int
}{
{name: "accepted producer is hydrated", wantCode: 0},
{name: "invalid producer stops dependents", invalidateOutput: true, wantCode: 1},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newRecomputeTestRoots(t)
harness := newRecomputeTestHarness()
fresh := runRecomputeCommand(roots, harness.options(), false)
if fresh.code != 0 {
t.Fatalf("fresh run code=%d stderr=%q", fresh.code, fresh.stderr)
}
removeCheckpointLaneStage(t, roots.checkpoints, "extract", "first", "producer")
removeCheckpointLaneStage(t, roots.checkpoints, "merge", "first", "producer")
if tt.invalidateOutput {
path := findCheckpointFile(t, roots.checkpoints, "normalize", "first", "producer", "output.json")
if err := os.WriteFile(path, []byte("{"), 0o600); err != nil {
t.Fatal(err)
}
}
harness.resetCalls()
resumed := runRecomputeCommand(roots, harness.options(), true)
if resumed.code != tt.wantCode {
t.Fatalf("resumed code=%d stdout=%q stderr=%q", resumed.code, resumed.stdout, resumed.stderr)
}
events := readLatestCheckpointEvents(t, roots.debug)
if tt.invalidateOutput {
if harness.callsFor("test/extract/middle") != 0 || harness.callsFor("test/extract/dependent") != 0 {
t.Fatalf("dependent calls after invalid producer = %#v", harness.callsSnapshot())
}
if !strings.Contains(resumed.stderr, string(pipeline.CheckpointReasonDecodeFailed)) {
t.Fatalf("stderr=%q, want stable checkpoint reason", resumed.stderr)
}
assertNormalizeDecisionSequence(t, events, []checkpointDecisionExpectation{{"first", "producer", pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonDecodeFailed}})
return
}
if got := harness.callsSnapshot(); !reflect.DeepEqual(got, map[string]int{"test/extract/dependent": 1, "test/extract/middle": 1}) {
t.Fatalf("resumed extractor calls = %#v", got)
}
outputPath := filepath.Join(latestChildDir(t, roots.output), "result.json")
data, err := os.ReadFile(outputPath)
if err != nil {
t.Fatal(err)
}
if string(data) != "[\"producer\",\"unrelated\",\"middle\",\"dependent\"]\n" {
t.Fatalf("ordered output = %q", data)
}
assertNormalizeDecisionSequence(t, events, []checkpointDecisionExpectation{
{"first", "producer", pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonAcceptedArtifactReused},
{"first", "unrelated", pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonReused},
{"second", "middle", pipeline.CheckpointDecisionForcedRecompute, pipeline.CheckpointReasonRecomputeStep},
{"third", "dependent", pipeline.CheckpointDecisionForcedRecompute, pipeline.CheckpointReasonRecomputeStep},
})
})
}
}
type checkpointDecisionExpectation struct {
step, lane string
action pipeline.CheckpointDecisionCategory
reason pipeline.CheckpointReasonCode
}
func assertNormalizeDecisionSequence(t *testing.T, events []pipeline.CheckpointEvent, want []checkpointDecisionExpectation) {
t.Helper()
var got []checkpointDecisionExpectation
for _, event := range events {
if event.Stage == string(pipeline.StageNormalize) {
got = append(got, checkpointDecisionExpectation{event.StepID, event.LaneID, event.Action, event.ReasonCode})
}
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("normalize decisions = %#v, want %#v", got, want)
}
}
type recomputeTestHarness struct {
base *stateTestHarness
mu sync.Mutex
calls map[string]int
}
func newRecomputeTestHarness() *recomputeTestHarness {
return &recomputeTestHarness{base: newStateTestHarness(), calls: make(map[string]int)}
}
func (h *recomputeTestHarness) options() Options {
opts := h.base.options()
for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} {
moduleKey := key
spec := pipeline.ModuleSpec{
Key: moduleKey, Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}},
}
if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) {
return recomputeTestExtractor{key: moduleKey, harness: h}, nil
}); err != nil {
panic(err)
}
}
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
return recomputeTestOutput{}, nil
}); err != nil {
panic(err)
}
opts.Catalog = catalogFromRegistries(opts.Registries)
return opts
}
func (h *recomputeTestHarness) record(key string) {
h.mu.Lock()
defer h.mu.Unlock()
h.calls[key]++
}
func (h *recomputeTestHarness) resetCalls() {
h.mu.Lock()
defer h.mu.Unlock()
h.calls = make(map[string]int)
}
func (h *recomputeTestHarness) callsFor(key string) int {
h.mu.Lock()
defer h.mu.Unlock()
return h.calls[key]
}
func (h *recomputeTestHarness) callsSnapshot() map[string]int {
h.mu.Lock()
defer h.mu.Unlock()
result := make(map[string]int, len(h.calls))
for key, value := range h.calls {
result[key] = value
}
return result
}
type recomputeTestExtractor struct {
key string
harness *recomputeTestHarness
}
func (e recomputeTestExtractor) Key() string { return e.key }
func (e recomputeTestExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}}
}
func (e recomputeTestExtractor) Extract(context.Context, contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[stateTestArtifact], error) {
e.harness.record(e.key)
return contracts.TypedExtractionResult[stateTestArtifact]{Value: stateTestArtifact{Value: e.key}}, nil
}
type recomputeTestOutput struct{}
func (recomputeTestOutput) Key() string { return "test/recompute-output" }
func (recomputeTestOutput) Encode(_ context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
lanes := make([]string, 0, len(req.NormalizeOutputs))
for _, output := range req.NormalizeOutputs {
lanes = append(lanes, output.LaneID)
}
data, err := json.Marshal(lanes)
if err != nil {
return contracts.OutputResult{}, err
}
return contracts.OutputResult{Files: []contracts.OutputFile{{Name: "result.json", Bytes: append(data, '\n')}}}, nil
}
func newRecomputeTestRoots(t *testing.T) stateTestRoots {
t.Helper()
roots := newStateTestRoots(t)
config := fmt.Sprintf(`version: 3
output:
directory: %q
cache:
chunk_plans:
directory: %q
mode: bypass
checkpoints:
enabled: true
directory: %q
debug:
directory: %q
pipelines:
sample:
input: test/input
chunk: test/chunk
steps:
- id: first
artifacts:
producer:
extract: test/extract/producer
merge: test/merge
normalize: test/normalize
unrelated:
extract: test/extract/unrelated
merge: test/merge
normalize: test/normalize
- id: second
references:
upstream:
artifact:
step: first
lane: producer
artifacts:
middle:
extract: test/extract/middle
merge: test/merge
normalize: test/normalize
- id: third
references:
upstream:
artifact:
step: second
lane: middle
artifacts:
dependent:
extract: test/extract/dependent
merge: test/merge
normalize: test/normalize
output: test/recompute-output
`, roots.output, roots.plans, roots.checkpoints, roots.debug)
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
t.Fatal(err)
}
return roots
}
func runRecomputeCommand(roots stateTestRoots, opts Options, recompute bool) stateTestResult {
args := []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}
if recompute {
args = append(args, "--resume", "--recompute-step", "second", "--debug")
}
var stdout, stderr bytes.Buffer
return stateTestResult{code: RunWithOptions(args, &stdout, &stderr, opts), stdout: stdout.String(), stderr: stderr.String()}
}
func removeCheckpointLaneStage(t *testing.T, root, stage, step, lane string) {
t.Helper()
dir := filepath.Dir(findCheckpointFile(t, root, stage, step, lane, "manifest.json"))
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
func findCheckpointFile(t *testing.T, root, stage, step, lane, name string) string {
t.Helper()
want := filepath.Join(stage, step, lane, name)
var matches []string
err := filepath.WalkDir(root, func(path string, entry os.DirEntry, err error) error {
if err != nil {
return err
}
if !entry.IsDir() && strings.HasSuffix(path, want) {
matches = append(matches, path)
}
return nil
})
if err != nil {
t.Fatal(err)
}
if len(matches) != 1 {
t.Fatalf("checkpoint files ending in %q = %v", want, matches)
}
return matches[0]
}
func latestChildDir(t *testing.T, root string) string {
t.Helper()
entries, err := os.ReadDir(root)
if err != nil {
t.Fatal(err)
}
var dirs []string
for _, entry := range entries {
if entry.IsDir() {
dirs = append(dirs, filepath.Join(root, entry.Name()))
}
}
if len(dirs) == 0 {
t.Fatal("no child directory")
}
sort.Strings(dirs)
return dirs[len(dirs)-1]
}
func readLatestCheckpointEvents(t *testing.T, root string) []pipeline.CheckpointEvent {
t.Helper()
var events []pipeline.CheckpointEvent
data, err := os.ReadFile(filepath.Join(latestChildDir(t, root), "summary", "checkpoint-events.json"))
if err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(data, &events); err != nil {
t.Fatal(err)
}
return events
}

View File

@@ -0,0 +1,56 @@
package cli
import (
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestRecomputePolicyIncludesDependentLanesAndReusablePredecessors(t *testing.T) {
producer := pipeline.ResolvedArtifactLane{StepID: "first", ID: "producer"}
unrelated := pipeline.ResolvedArtifactLane{StepID: "first", ID: "unrelated"}
consumer := pipeline.ResolvedArtifactLane{
StepID: "second", ID: "consumer",
ExtractReferences: pipeline.ResolvedReferenceTarget{Bindings: []pipeline.ReferenceBinding{{Artifact: &pipeline.ArtifactReference{Step: "first", Lane: "producer"}}}},
}
downstream := pipeline.ResolvedArtifactLane{
StepID: "third", ID: "downstream",
ExtractReferences: pipeline.ResolvedReferenceTarget{Bindings: []pipeline.ReferenceBinding{{Artifact: &pipeline.ArtifactReference{Step: "second", Lane: "consumer"}}}},
}
independent := pipeline.ResolvedArtifactLane{StepID: "third", ID: "independent"}
resolved := pipeline.ResolvedPipeline{Steps: []pipeline.ResolvedPipelineStep{
{ID: "first", ArtifactLanes: []pipeline.ResolvedArtifactLane{producer, unrelated}},
{ID: "second", ArtifactLanes: []pipeline.ResolvedArtifactLane{consumer}},
{ID: "third", ArtifactLanes: []pipeline.ResolvedArtifactLane{downstream, independent}},
}}
policy, err := recomputePolicy(resolved, "second")
if err != nil {
t.Fatal(err)
}
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("second", "consumer")]; !ok {
t.Fatal("selected lane was not forced")
}
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("third", "downstream")]; !ok {
t.Fatal("transitive dependent lane was not forced")
}
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("first", "producer")]; ok {
t.Fatal("predecessor was implicitly forced")
}
if _, ok := policy.RequireReusableLanes[pipeline.CheckpointLaneKey("first", "producer")]; !ok {
t.Fatal("required predecessor was not marked reusable")
}
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("first", "unrelated")]; ok {
t.Fatal("unrelated lane was forced")
}
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("third", "independent")]; ok {
t.Fatal("unrelated later lane was forced")
}
}
func TestRecomputePolicyRejectsUnknownStep(t *testing.T) {
_, err := recomputePolicy(pipeline.ResolvedPipeline{Steps: []pipeline.ResolvedPipelineStep{{ID: "known"}}}, "missing")
if err == nil {
t.Fatal("unknown step was accepted")
}
}

View File

@@ -319,28 +319,28 @@ func referenceContractConfig() config.Config {
Extract: pipeline.Binding("reference/extract-alpha"), Extract: pipeline.Binding("reference/extract-alpha"),
Merge: pipeline.Binding("reference/shared-merge"), Merge: pipeline.Binding("reference/shared-merge"),
Normalize: pipeline.Binding("reference/shared-normalize"), Normalize: pipeline.Binding("reference/shared-normalize"),
References: map[string]string{"required-extract": "required.txt"}, References: pipeline.ExternalReferenceMap(map[string]string{"required-extract": "required.txt"}),
}, },
"beta": { "beta": {
Extract: pipeline.Binding("reference/extract-beta"), Extract: pipeline.Binding("reference/extract-beta"),
Merge: pipeline.Binding("reference/shared-merge"), Merge: pipeline.Binding("reference/shared-merge"),
Normalize: pipeline.Binding("reference/shared-normalize"), Normalize: pipeline.Binding("reference/shared-normalize"),
References: map[string]string{"required-extract": "required.txt"}, References: pipeline.ExternalReferenceMap(map[string]string{"required-extract": "required.txt"}),
}, },
}, },
}, },
} }
profile := cfg.Pipelines["demo"] profile := cfg.Pipelines["demo"]
profile.Chunk.References = map[string]string{"required-chunk": "required.txt"} profile.Chunk.References = pipeline.ExternalReferenceMap(map[string]string{"required-chunk": "required.txt"})
alpha := profile.Artifacts["alpha"] alpha := profile.Artifacts["alpha"]
alpha.Extract.References = map[string]string{"required-extract": "required.txt", "alpha-slot": "optional.txt"} alpha.Extract.References = pipeline.ExternalReferenceMap(map[string]string{"required-extract": "required.txt", "alpha-slot": "optional.txt"})
alpha.Merge.References = map[string]string{"required-merge": "required.txt"} alpha.Merge.References = pipeline.ExternalReferenceMap(map[string]string{"required-merge": "required.txt"})
alpha.Normalize.References = map[string]string{"required-normalize": "required.txt"} alpha.Normalize.References = pipeline.ExternalReferenceMap(map[string]string{"required-normalize": "required.txt"})
profile.Artifacts["alpha"] = alpha profile.Artifacts["alpha"] = alpha
beta := profile.Artifacts["beta"] beta := profile.Artifacts["beta"]
beta.Extract.References = map[string]string{"required-extract": "required.txt"} beta.Extract.References = pipeline.ExternalReferenceMap(map[string]string{"required-extract": "required.txt"})
beta.Merge.References = map[string]string{"required-merge": "required.txt"} beta.Merge.References = pipeline.ExternalReferenceMap(map[string]string{"required-merge": "required.txt"})
beta.Normalize.References = map[string]string{"required-normalize": "required.txt"} beta.Normalize.References = pipeline.ExternalReferenceMap(map[string]string{"required-normalize": "required.txt"})
profile.Artifacts["beta"] = beta profile.Artifacts["beta"] = beta
cfg.Pipelines["demo"] = profile cfg.Pipelines["demo"] = profile
return cfg return cfg
@@ -418,7 +418,7 @@ func (referenceContractCodecB) Decode([]byte) (stateTestArtifact, error) {
func referenceContractLane(t *testing.T, resolved pipeline.ResolvedPipeline, id string) pipeline.ResolvedArtifactLane { func referenceContractLane(t *testing.T, resolved pipeline.ResolvedPipeline, id string) pipeline.ResolvedArtifactLane {
t.Helper() t.Helper()
for _, lane := range resolved.ArtifactLanes { for _, lane := range resolved.Steps[0].ArtifactLanes {
if lane.ID == id { if lane.ID == id {
return lane return lane
} }

View File

@@ -28,7 +28,7 @@ import (
const defaultConfigPath = "/usr/local/etc/notarius/config.yml" const defaultConfigPath = "/usr/local/etc/notarius/config.yml"
const usage = `Usage: const usage = `Usage:
notarius help notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--output-dir path] [--chunk_cache auto|bypass|refresh] [--resume] [--debug [--debug-dir path]] [--only lane-a,lane-b] [--session-id id] [--reference selector=path] [--without-reference selector] notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--output-dir path] [--chunk_cache auto|bypass|refresh] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--only lane-a,lane-b] [--session-id id] [--reference selector=path] [--without-reference selector]
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b] notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list --config path/to/config.yml [--json] notarius pipelines list --config path/to/config.yml [--json]
` `
@@ -136,6 +136,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
debugDir := fs.String("debug-dir", "", "debug bundle directory") debugDir := fs.String("debug-dir", "", "debug bundle directory")
llmProfile := fs.String("llm-profile", "", "LLM profile override") llmProfile := fs.String("llm-profile", "", "LLM profile override")
resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints") resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints")
recomputeStep := singleValueFlag{}
chunkCache := chunkCacheFlag{} chunkCache := chunkCacheFlag{}
sessionID := sessionIDFlag{} sessionID := sessionIDFlag{}
referenceFlags := stringListFlag{} referenceFlags := stringListFlag{}
@@ -144,6 +145,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh") fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh")
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path") fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path")
fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference") fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference")
fs.Var(&recomputeStep, "recompute-step", "recompute one ordered pipeline step and dependent lanes")
if err := validateRunFlagValues(args); err != nil { if err := validateRunFlagValues(args); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err) fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2 return 2
@@ -190,6 +192,14 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
fmt.Fprintf(stderr, "notarius: %v\n", err) fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2 return 2
} }
if recomputeStep.set && strings.TrimSpace(recomputeStep.value) == "" {
fmt.Fprintln(stderr, "notarius: --recompute-step must not be empty")
return 2
}
if recomputeStep.set && len(only) > 0 {
fmt.Fprintln(stderr, "notarius: --recompute-step cannot be combined with --only")
return 2
}
referenceRequests, err := parseReferenceFlags(referenceFlags) referenceRequests, err := parseReferenceFlags(referenceFlags)
if err != nil { if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err) fmt.Fprintf(stderr, "notarius: %v\n", err)
@@ -223,6 +233,14 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
fmt.Fprintln(stderr, "notarius: --resume requires cache.checkpoints.enabled: true") fmt.Fprintln(stderr, "notarius: --resume requires cache.checkpoints.enabled: true")
return 1 return 1
} }
if recomputeStep.set && !*resume {
fmt.Fprintln(stderr, "notarius: --recompute-step requires --resume")
return 2
}
if recomputeStep.set && !cfg.Cache.Checkpoints.Enabled {
fmt.Fprintln(stderr, "notarius: --recompute-step requires cache.checkpoints.enabled: true")
return 1
}
startedAt := opts.Now().UTC() startedAt := opts.Now().UTC()
runID, err := opts.RunIDGenerator(startedAt) runID, err := opts.RunIDGenerator(startedAt)
@@ -267,6 +285,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
OnlyLanes: append([]string(nil), only...), OnlyLanes: append([]string(nil), only...),
ChunkCacheOverride: chunkCache.explicitValue(), ChunkCacheOverride: chunkCache.explicitValue(),
Resume: *resume, Resume: *resume,
RecomputeStep: strings.TrimSpace(recomputeStep.value),
RunID: runID, RunID: runID,
StartedAt: startedAt, StartedAt: startedAt,
} }
@@ -309,6 +328,13 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
return failPipelineCommand(stderr, commandState, terminalWriter, err) return failPipelineCommand(stderr, commandState, terminalWriter, err)
} }
effective.ResolvedPipeline = materialized effective.ResolvedPipeline = materialized
checkpointPolicy := pipeline.CheckpointExecutionPolicy{}
if recomputeStep.set {
checkpointPolicy, err = recomputePolicy(effective.ResolvedPipeline, recomputeStep.value)
if err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, err)
}
}
invocation.PipelineDigest = effective.ResolvedPipeline.Digest invocation.PipelineDigest = effective.ResolvedPipeline.Digest
if err := writeSummary(summary, func() error { return summary.WriteInvocation(invocation) }); err != nil { if err := writeSummary(summary, func() error { return summary.WriteInvocation(invocation) }); err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug invocation metadata: %w", err)) return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug invocation metadata: %w", err))
@@ -358,21 +384,22 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
} }
output, err := pipeline.New().Run(ctx, pipeline.RunInput{ output, err := pipeline.New().Run(ctx, pipeline.RunInput{
Prepared: prepared, Prepared: prepared,
Path: strings.TrimSpace(*inputPath), Path: strings.TrimSpace(*inputPath),
RawInput: rawInput, RawInput: rawInput,
SessionID: strings.TrimSpace(sessionID.value), SessionID: strings.TrimSpace(sessionID.value),
RunID: runID, RunID: runID,
StartedAt: startedAt, StartedAt: startedAt,
LLMProfiles: llmProfiles, LLMProfiles: llmProfiles,
Metadata: runMetadata(effective.Config.Output.Directory, debugPath), Metadata: runMetadata(effective.Config.Output.Directory, debugPath),
Warnings: referenceWarnings, Warnings: referenceWarnings,
ChunkCacheMode: effective.Config.Cache.ChunkPlans.Mode, ChunkCacheMode: effective.Config.Cache.ChunkPlans.Mode,
ChunkPlans: chunkPlans, ChunkPlans: chunkPlans,
Checkpoints: checkpointRecorder, Checkpoints: checkpointRecorder,
Checkpoint: checkpointLoader, Checkpoint: checkpointLoader,
Debug: debugRecorder, CheckpointPolicy: checkpointPolicy,
ExtractWorkers: cfg.Concurrency.StageWorkers["extract"], Debug: debugRecorder,
ExtractWorkers: cfg.Concurrency.StageWorkers["extract"],
}) })
commandState.observeOutput(output) commandState.observeOutput(output)
if err != nil { if err != nil {
@@ -481,6 +508,95 @@ func checkpointHandlersForRun(
return recorder, loader, nil return recorder, loader, nil
} }
func recomputePolicy(resolved pipeline.ResolvedPipeline, requestedStep string) (pipeline.CheckpointExecutionPolicy, error) {
requestedStep = strings.TrimSpace(requestedStep)
if requestedStep == "" {
return pipeline.CheckpointExecutionPolicy{}, fmt.Errorf("--recompute-step must not be empty")
}
var selected *pipeline.ResolvedPipelineStep
for index := range resolved.Steps {
if strings.TrimSpace(resolved.Steps[index].ID) == requestedStep {
selected = &resolved.Steps[index]
break
}
}
if selected == nil {
return pipeline.CheckpointExecutionPolicy{}, fmt.Errorf("unknown pipeline step %q", requestedStep)
}
policy := pipeline.CheckpointExecutionPolicy{ForcedLanes: make(map[string]struct{}), RequireReusableLanes: make(map[string]struct{})}
var forced []pipeline.ResolvedArtifactLane
for _, lane := range selected.ArtifactLanes {
lane.StepID = selected.ID
policy.ForcedLanes[pipeline.CheckpointLaneKey(selected.ID, lane.ID)] = struct{}{}
forced = append(forced, lane)
}
lanes := make(map[string]pipeline.ResolvedArtifactLane)
dependents := make(map[string][]pipeline.ResolvedArtifactLane)
for _, step := range resolved.Steps {
for _, lane := range step.ArtifactLanes {
lane.StepID = step.ID
lanes[pipeline.CheckpointLaneKey(step.ID, lane.ID)] = lane
for _, target := range []pipeline.ResolvedReferenceTarget{lane.ExtractReferences, lane.MergeReferences, lane.NormalizeReferences} {
for _, binding := range target.Bindings {
if binding.Artifact != nil {
producerKey := pipeline.CheckpointLaneKey(binding.Artifact.Step, binding.Artifact.Lane)
dependents[producerKey] = append(dependents[producerKey], lane)
}
}
}
}
}
processed := make(map[string]struct{})
queue := append([]pipeline.ResolvedArtifactLane(nil), forced...)
for len(queue) > 0 {
lane := queue[0]
queue = queue[1:]
key := pipeline.CheckpointLaneKey(lane.StepID, lane.ID)
if _, ok := processed[key]; ok {
continue
}
processed[key] = struct{}{}
for _, dependent := range dependents[key] {
dependentKey := pipeline.CheckpointLaneKey(dependent.StepID, dependent.ID)
if _, alreadyForced := policy.ForcedLanes[dependentKey]; alreadyForced {
continue
}
policy.ForcedLanes[dependentKey] = struct{}{}
forced = append(forced, dependent)
queue = append(queue, dependent)
}
}
processed = make(map[string]struct{})
queue = append([]pipeline.ResolvedArtifactLane(nil), forced...)
for len(queue) > 0 {
lane := queue[0]
queue = queue[1:]
key := pipeline.CheckpointLaneKey(lane.StepID, lane.ID)
if _, ok := processed[key]; ok {
continue
}
processed[key] = struct{}{}
for _, target := range []pipeline.ResolvedReferenceTarget{lane.ExtractReferences, lane.MergeReferences, lane.NormalizeReferences} {
for _, binding := range target.Bindings {
if binding.Artifact == nil {
continue
}
producerKey := pipeline.CheckpointLaneKey(binding.Artifact.Step, binding.Artifact.Lane)
if _, forcedAlready := policy.ForcedLanes[producerKey]; !forcedAlready {
policy.RequireReusableLanes[producerKey] = struct{}{}
}
if producer, ok := lanes[producerKey]; ok {
queue = append(queue, producer)
}
}
}
}
return policy, nil
}
func checkpointIdentityFingerprints(values []pipeline.CheckpointFingerprint) []checkpoint.Fingerprint { func checkpointIdentityFingerprints(values []pipeline.CheckpointFingerprint) []checkpoint.Fingerprint {
if len(values) == 0 { if len(values) == 0 {
return nil return nil
@@ -661,7 +777,7 @@ func reorderRunArgs(args []string) []string {
func runFlagTakesValue(arg string) bool { func runFlagTakesValue(arg string) bool {
switch arg { switch arg {
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--chunk_cache", "--reference", "--without-reference": case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
return true return true
default: default:
return false return false
@@ -749,7 +865,7 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
} }
} }
add(resolved.Chunk) add(resolved.Chunk)
for _, lane := range resolved.ArtifactLanes { for _, lane := range resolved.AllArtifactLanes() {
add(lane.Extract) add(lane.Extract)
add(lane.Merge) add(lane.Merge)
add(lane.Normalize) add(lane.Normalize)
@@ -1008,6 +1124,27 @@ type sessionIDFlag struct {
set bool set bool
} }
type singleValueFlag struct {
value string
set bool
}
func (flag *singleValueFlag) String() string {
if flag == nil {
return ""
}
return flag.value
}
func (flag *singleValueFlag) Set(value string) error {
if flag.set {
return fmt.Errorf("--recompute-step may be specified only once")
}
flag.value = value
flag.set = true
return nil
}
func (flag *sessionIDFlag) String() string { func (flag *sessionIDFlag) String() string {
if flag == nil { if flag == nil {
return "" return ""
@@ -1178,17 +1315,34 @@ func selectedReferenceTargets(cfg config.Config, pipelineID string, only []strin
if !ok { if !ok {
return nil, fmt.Errorf("pipeline %q is not configured", strings.TrimSpace(pipelineID)) return nil, fmt.Errorf("pipeline %q is not configured", strings.TrimSpace(pipelineID))
} }
if profile.Steps != nil && len(only) > 0 {
return nil, fmt.Errorf("pipeline %q --only is not supported for explicit ordered steps", strings.TrimSpace(pipelineID))
}
lanesByID := make(map[string]pipeline.ArtifactLaneProfile, len(profile.Artifacts)) lanesByID := make(map[string]pipeline.ArtifactLaneProfile, len(profile.Artifacts))
for rawLaneID, lane := range profile.Artifacts { addLanes := func(values map[string]pipeline.ArtifactLaneProfile) error {
laneID := strings.TrimSpace(rawLaneID) for rawLaneID, lane := range values {
if laneID == "" { laneID := strings.TrimSpace(rawLaneID)
return nil, fmt.Errorf("pipeline %q artifact lane id must not be empty", strings.TrimSpace(pipelineID)) if laneID == "" {
return fmt.Errorf("pipeline %q artifact lane id must not be empty", strings.TrimSpace(pipelineID))
}
if _, ok := lanesByID[laneID]; ok {
return fmt.Errorf("pipeline %q artifact lane %q is duplicated after trimming", strings.TrimSpace(pipelineID), laneID)
}
lanesByID[laneID] = lane
} }
if _, ok := lanesByID[laneID]; ok { return nil
return nil, fmt.Errorf("pipeline %q artifact lane %q is duplicated after trimming", strings.TrimSpace(pipelineID), laneID) }
if profile.Steps == nil {
if err := addLanes(profile.Artifacts); err != nil {
return nil, err
}
} else {
for _, step := range profile.Steps {
if err := addLanes(step.Artifacts); err != nil {
return nil, err
}
} }
lanesByID[laneID] = lane
} }
selectedIDs := make([]string, 0, len(lanesByID)) selectedIDs := make([]string, 0, len(lanesByID))

View File

@@ -60,6 +60,56 @@ func TestRunControlsRejectSyntaxWithoutAllocatingState(t *testing.T) {
} }
} }
func TestRecomputeStepCLIContract(t *testing.T) {
tests := []struct {
name string
configure func(*testing.T, stateTestRoots)
flags []string
wantCode int
wantOutput string
wantError string
}{
{
name: "explicit step",
configure: func(t *testing.T, roots stateTestRoots) {
replaceStateTestConfigLine(t, roots.config, " artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n", " steps:\n - id: chosen\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n")
},
flags: []string{"--resume", "--recompute-step", "chosen"},
wantCode: 0,
wantOutput: "outputs=1",
},
{name: "implicit default step", flags: []string{"--resume", "--recompute-step", "default"}, wantCode: 0, wantOutput: "outputs=1"},
{name: "repeated flag", flags: []string{"--resume", "--recompute-step", "default", "--recompute-step", "default"}, wantCode: 2, wantError: "specified only once"},
{name: "empty step", flags: []string{"--resume", "--recompute-step", ""}, wantCode: 2, wantError: "must not be empty"},
{name: "unknown step", flags: []string{"--resume", "--recompute-step", "missing"}, wantCode: 1, wantError: "unknown pipeline step"},
{name: "without resume", flags: []string{"--recompute-step", "default"}, wantCode: 2, wantError: "requires --resume"},
{
name: "checkpoint recording disabled",
configure: func(t *testing.T, roots stateTestRoots) {
replaceStateTestConfigLine(t, roots.config, " enabled: true\n", " enabled: false\n")
},
flags: []string{"--resume", "--recompute-step", "default"}, wantCode: 1, wantError: "cache.checkpoints.enabled",
},
{name: "with only", flags: []string{"--resume", "--recompute-step", "default", "--only", "items"}, wantCode: 2, wantError: "cannot be combined with --only"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
if tt.configure != nil {
tt.configure(t, roots)
}
args := []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}
args = append(args, tt.flags...)
var stdout, stderr bytes.Buffer
code := RunWithOptions(args, &stdout, &stderr, newStateTestHarness().options())
if code != tt.wantCode || (tt.wantOutput != "" && !strings.Contains(stdout.String(), tt.wantOutput)) || (tt.wantError != "" && !strings.Contains(stderr.String(), tt.wantError)) {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
}
}
func TestRunValidFailuresClassifyAndReportDebug(t *testing.T) { func TestRunValidFailuresClassifyAndReportDebug(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
@@ -268,13 +318,14 @@ func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
resolved := pipeline.ResolvedPipeline{ resolved := pipeline.ResolvedPipeline{
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"}, Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "}, Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
ArtifactLanes: []pipeline.ResolvedArtifactLane{ Steps: []pipeline.ResolvedPipelineStep{{
{ ID: "default",
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"}, Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
Merge: pipeline.ModuleBinding{LLMProfile: "zeta"}, Merge: pipeline.ModuleBinding{LLMProfile: "zeta"},
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "}, Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
}, }},
}, }},
ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{ ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{
{Binding: pipeline.ModuleBinding{LLMProfile: "deterministic-profile"}, ExecutionClass: contracts.ExecutionClassDeterministic}, {Binding: pipeline.ModuleBinding{LLMProfile: "deterministic-profile"}, ExecutionClass: contracts.ExecutionClassDeterministic},
{Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked}, {Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked},

View File

@@ -31,7 +31,7 @@ func TestSpellCatalogBytesAffectCheckpointIdentityButNotSemanticDigest(t *testin
} }
overlayPath := filepath.Join(t.TempDir(), "catalog.json") overlayPath := filepath.Join(t.TempDir(), "catalog.json")
resolved := effective.ResolvedPipeline resolved := effective.ResolvedPipeline
bindings := resolved.ArtifactLanes[0].ExtractReferences.Bindings bindings := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings
catalogBindingIndex := -1 catalogBindingIndex := -1
for index, binding := range bindings { for index, binding := range bindings {
if binding.SlotName == spellcatalog.SpellCatalogReferenceSlot { if binding.SlotName == spellcatalog.SpellCatalogReferenceSlot {
@@ -42,8 +42,8 @@ func TestSpellCatalogBytesAffectCheckpointIdentityButNotSemanticDigest(t *testin
if catalogBindingIndex < 0 { if catalogBindingIndex < 0 {
t.Fatalf("spell catalog bindings = %#v, want catalog binding", bindings) t.Fatalf("spell catalog bindings = %#v, want catalog binding", bindings)
} }
resolved.ArtifactLanes[0].ExtractReferences.Bindings[catalogBindingIndex].Source = overlayPath resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings[catalogBindingIndex].Source = overlayPath
normalizeBindings := resolved.ArtifactLanes[0].NormalizeReferences.Bindings normalizeBindings := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings
normalizeCatalogBindingIndex := -1 normalizeCatalogBindingIndex := -1
for index, binding := range normalizeBindings { for index, binding := range normalizeBindings {
if binding.SlotName == spellcatalog.SpellCatalogReferenceSlot { if binding.SlotName == spellcatalog.SpellCatalogReferenceSlot {
@@ -54,7 +54,7 @@ func TestSpellCatalogBytesAffectCheckpointIdentityButNotSemanticDigest(t *testin
if normalizeCatalogBindingIndex < 0 { if normalizeCatalogBindingIndex < 0 {
t.Fatalf("normalize spell catalog bindings = %#v, want catalog binding", normalizeBindings) t.Fatalf("normalize spell catalog bindings = %#v, want catalog binding", normalizeBindings)
} }
resolved.ArtifactLanes[0].NormalizeReferences.Bindings[normalizeCatalogBindingIndex].Source = overlayPath resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings[normalizeCatalogBindingIndex].Source = overlayPath
if err := os.WriteFile(overlayPath, []byte(reorderedOverlayA), 0o600); err != nil { if err := os.WriteFile(overlayPath, []byte(reorderedOverlayA), 0o600); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -415,7 +415,7 @@ func catalogCheckpointIdentity(t *testing.T, resolved pipeline.ResolvedPipeline)
func catalogExtractorMetadata(t *testing.T, resolved pipeline.ResolvedPipeline) map[string]any { func catalogExtractorMetadata(t *testing.T, resolved pipeline.ResolvedPipeline) map[string]any {
t.Helper() t.Helper()
lane := resolved.ArtifactLanes[0] lane := resolved.Steps[0].ArtifactLanes[0]
extractor, err := spells.New(&productionFakeLLMClient{}, spells.Options{}, lane.ExtractReferences.ReferenceSet) extractor, err := spells.New(&productionFakeLLMClient{}, spells.Options{}, lane.ExtractReferences.ReferenceSet)
if err != nil { if err != nil {
t.Fatalf("construct extractor: %v", err) t.Fatalf("construct extractor: %v", err)
@@ -425,7 +425,7 @@ func catalogExtractorMetadata(t *testing.T, resolved pipeline.ResolvedPipeline)
func catalogNormalizerMetadata(t *testing.T, resolved pipeline.ResolvedPipeline) map[string]any { func catalogNormalizerMetadata(t *testing.T, resolved pipeline.ResolvedPipeline) map[string]any {
t.Helper() t.Helper()
lane := resolved.ArtifactLanes[0] lane := resolved.Steps[0].ArtifactLanes[0]
normalizer, err := spellnormalize.New(spellnormalize.Options{}, lane.NormalizeReferences.ReferenceSet) normalizer, err := spellnormalize.New(spellnormalize.Options{}, lane.NormalizeReferences.ReferenceSet)
if err != nil { if err != nil {
t.Fatalf("construct normalizer: %v", err) t.Fatalf("construct normalizer: %v", err)

View File

@@ -63,7 +63,7 @@ func TestProductionSpellCatalogValidationRetries(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("materialize production references: %v", err) t.Fatalf("materialize production references: %v", err)
} }
materialized.ArtifactLanes[0].Extract.Retries = retries materialized.Steps[0].ArtifactLanes[0].Extract.Retries = retries
llmClient := &catalogRetryLLMClient{responses: tt.responses} llmClient := &catalogRetryLLMClient{responses: tt.responses}
prepared, err := pipeline.Prepare(materialized, components.registries, pipeline.ModuleDependencies{LLM: llmClient}) prepared, err := pipeline.Prepare(materialized, components.registries, pipeline.ModuleDependencies{LLM: llmClient})

View File

@@ -5,6 +5,7 @@ import (
) )
type ArtifactLaneManifest struct { type ArtifactLaneManifest struct {
StepID string `json:"step_id,omitempty"`
ID string `json:"id"` ID string `json:"id"`
Extractor string `json:"extractor"` Extractor string `json:"extractor"`
Merger string `json:"merger"` Merger string `json:"merger"`
@@ -31,15 +32,25 @@ type LLMProfileManifest struct {
} }
type ReferenceProvenance struct { type ReferenceProvenance struct {
Stage string `json:"stage,omitempty"` Stage string `json:"stage,omitempty"`
LaneID string `json:"lane_id,omitempty"` StepID string `json:"step_id,omitempty"`
SlotName string `json:"slot_name"` LaneID string `json:"lane_id,omitempty"`
OriginType string `json:"origin_type"` SlotName string `json:"slot_name"`
OriginURI string `json:"origin_uri,omitempty"` OriginType string `json:"origin_type"`
Digest string `json:"digest,omitempty"` OriginURI string `json:"origin_uri,omitempty"`
MediaType string `json:"media_type,omitempty"` Digest string `json:"digest,omitempty"`
SizeBytes int64 `json:"size_bytes,omitempty"` MediaType string `json:"media_type,omitempty"`
BindingSource string `json:"binding_source,omitempty"` SizeBytes int64 `json:"size_bytes,omitempty"`
BindingSource string `json:"binding_source,omitempty"`
ArtifactKind string `json:"artifact_kind,omitempty"`
SchemaID string `json:"schema_id,omitempty"`
SchemaName string `json:"schema_name,omitempty"`
SchemaVersion string `json:"schema_version,omitempty"`
SchemaDigest string `json:"schema_digest,omitempty"`
ProducerPipeline string `json:"producer_pipeline_id,omitempty"`
ProducerStep string `json:"producer_step_id,omitempty"`
ProducerLane string `json:"producer_lane_id,omitempty"`
ProducerModule string `json:"producer_module_key,omitempty"`
} }
type OutputSchemaProvenance struct { type OutputSchemaProvenance struct {
@@ -49,6 +60,7 @@ type OutputSchemaProvenance struct {
} }
type NormalizedOutputManifest struct { type NormalizedOutputManifest struct {
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id"` LaneID string `json:"lane_id"`
ModuleKey string `json:"module_key,omitempty"` ModuleKey string `json:"module_key,omitempty"`
SourceID string `json:"source_id,omitempty"` SourceID string `json:"source_id,omitempty"`
@@ -58,6 +70,7 @@ type NormalizedOutputManifest struct {
type RejectedOutputManifest struct { type RejectedOutputManifest struct {
Stage string `json:"stage"` Stage string `json:"stage"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"` LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"` ModuleKey string `json:"module_key,omitempty"`
ChunkID string `json:"chunk_id,omitempty"` ChunkID string `json:"chunk_id,omitempty"`
@@ -69,6 +82,16 @@ type RejectedOutputManifest struct {
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"` DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
} }
type CheckpointDecisionManifest struct {
Stage string `json:"stage"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"`
Category string `json:"category"`
ReasonCode string `json:"reason_code,omitempty"`
Detail string `json:"detail,omitempty"`
}
type ChunkPlanManifest struct { type ChunkPlanManifest struct {
Mode string `json:"mode"` Mode string `json:"mode"`
Action string `json:"action,omitempty"` Action string `json:"action,omitempty"`
@@ -99,27 +122,28 @@ type ChunkPlanSummary struct {
} }
type RunManifest struct { type RunManifest struct {
RunID string `json:"run_id,omitempty"` RunID string `json:"run_id,omitempty"`
PipelineID string `json:"pipeline_id,omitempty"` PipelineID string `json:"pipeline_id,omitempty"`
PipelineDigest string `json:"pipeline_digest,omitempty"` PipelineDigest string `json:"pipeline_digest,omitempty"`
InputModule string `json:"input_module,omitempty"` InputModule string `json:"input_module,omitempty"`
Chunker string `json:"chunker,omitempty"` Chunker string `json:"chunker,omitempty"`
ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"` ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"`
SourceDigests []string `json:"source_digests,omitempty"` SourceDigests []string `json:"source_digests,omitempty"`
Extractors []string `json:"extractors,omitempty"` Extractors []string `json:"extractors,omitempty"`
Merger string `json:"merger,omitempty"` Merger string `json:"merger,omitempty"`
Normalizer string `json:"normalizer,omitempty"` Normalizer string `json:"normalizer,omitempty"`
OutputEncoder string `json:"output_encoder,omitempty"` OutputEncoder string `json:"output_encoder,omitempty"`
ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"` ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"`
ArtifactLanes []ArtifactLaneManifest `json:"artifact_lanes,omitempty"` ArtifactLanes []ArtifactLaneManifest `json:"artifact_lanes,omitempty"`
ValidatorChains []ValidatorChainManifest `json:"validator_chains,omitempty"` ValidatorChains []ValidatorChainManifest `json:"validator_chains,omitempty"`
References []ReferenceProvenance `json:"references,omitempty"` References []ReferenceProvenance `json:"references,omitempty"`
NormalizedOutputs []NormalizedOutputManifest `json:"normalized_outputs,omitempty"` NormalizedOutputs []NormalizedOutputManifest `json:"normalized_outputs,omitempty"`
RejectedOutputs []RejectedOutputManifest `json:"rejected_outputs,omitempty"` RejectedOutputs []RejectedOutputManifest `json:"rejected_outputs,omitempty"`
LLMProfiles []LLMProfileManifest `json:"llm_profiles,omitempty"` CheckpointDecisions []CheckpointDecisionManifest `json:"checkpoint_decisions,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"` LLMProfiles []LLMProfileManifest `json:"llm_profiles,omitempty"`
SchemaVersion string `json:"schema_version,omitempty"` Metadata map[string]any `json:"metadata,omitempty"`
ValidationStatus string `json:"validation_status,omitempty"` SchemaVersion string `json:"schema_version,omitempty"`
StartedAt *time.Time `json:"started_at,omitempty"` ValidationStatus string `json:"validation_status,omitempty"`
CompletedAt *time.Time `json:"completed_at,omitempty"` StartedAt *time.Time `json:"started_at,omitempty"`
CompletedAt *time.Time `json:"completed_at,omitempty"`
} }

View File

@@ -107,7 +107,26 @@ func clonePipelineProfile(in pipeline.PipelineProfile) pipeline.PipelineProfile
out.Input = cloneModuleBinding(in.Input) out.Input = cloneModuleBinding(in.Input)
out.Chunk = cloneModuleBinding(in.Chunk) out.Chunk = cloneModuleBinding(in.Chunk)
out.Output = cloneModuleBinding(in.Output) out.Output = cloneModuleBinding(in.Output)
out.References = cloneStringMap(in.References) out.References = cloneReferenceSourceMap(in.References)
if len(in.Artifacts) > 0 {
out.Artifacts = make(map[string]pipeline.ArtifactLaneProfile, len(in.Artifacts))
for key, lane := range in.Artifacts {
out.Artifacts[key] = cloneArtifactLaneProfile(lane)
}
}
if in.Steps != nil {
out.Steps = make([]pipeline.PipelineStepProfile, len(in.Steps))
for i, step := range in.Steps {
out.Steps[i] = clonePipelineStepProfile(step)
}
}
return out
}
func clonePipelineStepProfile(in pipeline.PipelineStepProfile) pipeline.PipelineStepProfile {
out := in
out.ID = in.ID
out.References = cloneReferenceSourceMap(in.References)
if len(in.Artifacts) > 0 { if len(in.Artifacts) > 0 {
out.Artifacts = make(map[string]pipeline.ArtifactLaneProfile, len(in.Artifacts)) out.Artifacts = make(map[string]pipeline.ArtifactLaneProfile, len(in.Artifacts))
for key, lane := range in.Artifacts { for key, lane := range in.Artifacts {
@@ -122,7 +141,7 @@ func cloneArtifactLaneProfile(in pipeline.ArtifactLaneProfile) pipeline.Artifact
out.Extract = cloneModuleBinding(in.Extract) out.Extract = cloneModuleBinding(in.Extract)
out.Merge = cloneModuleBinding(in.Merge) out.Merge = cloneModuleBinding(in.Merge)
out.Normalize = cloneModuleBinding(in.Normalize) out.Normalize = cloneModuleBinding(in.Normalize)
out.References = cloneStringMap(in.References) out.References = cloneReferenceSourceMap(in.References)
if len(in.Validators) > 0 { if len(in.Validators) > 0 {
out.Validators = make([]pipeline.ModuleBinding, len(in.Validators)) out.Validators = make([]pipeline.ModuleBinding, len(in.Validators))
for i, binding := range in.Validators { for i, binding := range in.Validators {
@@ -143,12 +162,32 @@ func cloneStringMap(in map[string]string) map[string]string {
return out return out
} }
func cloneReferenceSourceMap(in map[string]pipeline.ReferenceSource) map[string]pipeline.ReferenceSource {
if len(in) == 0 {
return nil
}
out := make(map[string]pipeline.ReferenceSource, len(in))
for key, source := range in {
out[key] = cloneReferenceSource(source)
}
return out
}
func cloneReferenceSource(in pipeline.ReferenceSource) pipeline.ReferenceSource {
out := in
if in.Artifact != nil {
artifact := *in.Artifact
out.Artifact = &artifact
}
return out
}
func cloneModuleBinding(in pipeline.ModuleBinding) pipeline.ModuleBinding { func cloneModuleBinding(in pipeline.ModuleBinding) pipeline.ModuleBinding {
out := in out := in
if len(in.Options) > 0 { if len(in.Options) > 0 {
out.Options = cloneOptions(in.Options) out.Options = cloneOptions(in.Options)
} }
out.References = cloneStringMap(in.References) out.References = cloneReferenceSourceMap(in.References)
out.Validators = cloneValidatorOverride(in.Validators) out.Validators = cloneValidatorOverride(in.Validators)
return out return out
} }

View File

@@ -67,11 +67,17 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) { func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) {
profile.Chunk.LLMProfile = profileID profile.Chunk.LLMProfile = profileID
for laneID, lane := range profile.Artifacts { apply := func(artifacts map[string]pipeline.ArtifactLaneProfile) {
lane.Extract.LLMProfile = profileID for laneID, lane := range artifacts {
lane.Merge.LLMProfile = profileID lane.Extract.LLMProfile = profileID
lane.Normalize.LLMProfile = profileID lane.Merge.LLMProfile = profileID
profile.Artifacts[laneID] = lane lane.Normalize.LLMProfile = profileID
artifacts[laneID] = lane
}
}
apply(profile.Artifacts)
for index := range profile.Steps {
apply(profile.Steps[index].Artifacts)
} }
} }

View File

@@ -53,8 +53,8 @@ func TestEffectiveConfigOnlySelectsRequestedLanesWithoutMutatingSource(t *testin
if err != nil { if err != nil {
t.Fatalf("Resolve() error = %v", err) t.Fatalf("Resolve() error = %v", err)
} }
if len(effective.ResolvedPipeline.ArtifactLanes) != 1 || effective.ResolvedPipeline.ArtifactLanes[0].ID != "other" { if len(effective.ResolvedPipeline.Steps[0].ArtifactLanes) != 1 || effective.ResolvedPipeline.Steps[0].ArtifactLanes[0].ID != "other" {
t.Fatalf("resolved lanes = %#v", effective.ResolvedPipeline.ArtifactLanes) t.Fatalf("resolved lanes = %#v", effective.ResolvedPipeline.Steps[0].ArtifactLanes)
} }
if len(cfg.Pipelines["main"].Artifacts) != 2 { if len(cfg.Pipelines["main"].Artifacts) != 2 {
t.Fatalf("source lanes were mutated: %#v", cfg.Pipelines["main"].Artifacts) t.Fatalf("source lanes were mutated: %#v", cfg.Pipelines["main"].Artifacts)
@@ -79,8 +79,8 @@ func TestEffectiveConfigMaterializesDefaultBindingsThroughCatalog(t *testing.T)
if resolved.Chunk.Module != pipeline.DefaultChunkModule || resolved.Output.Module != pipeline.DefaultOutputModule { if resolved.Chunk.Module != pipeline.DefaultChunkModule || resolved.Output.Module != pipeline.DefaultOutputModule {
t.Fatalf("default pipeline bindings = %#v, %#v", resolved.Chunk, resolved.Output) t.Fatalf("default pipeline bindings = %#v, %#v", resolved.Chunk, resolved.Output)
} }
if len(resolved.ArtifactLanes) != 1 || resolved.ArtifactLanes[0].Merge.Module != pipeline.DefaultMergeModule || resolved.ArtifactLanes[0].Normalize.Module != pipeline.DefaultNormalizeModule { if len(resolved.Steps[0].ArtifactLanes) != 1 || resolved.Steps[0].ArtifactLanes[0].Merge.Module != pipeline.DefaultMergeModule || resolved.Steps[0].ArtifactLanes[0].Normalize.Module != pipeline.DefaultNormalizeModule {
t.Fatalf("default lane bindings = %#v", resolved.ArtifactLanes) t.Fatalf("default lane bindings = %#v", resolved.Steps[0].ArtifactLanes)
} }
} }
@@ -182,8 +182,8 @@ func TestEffectiveConfigLLMProfileOverrideChangesDigestWithoutOverridingValidato
t.Fatal("LLM profile override did not change the pipeline digest") t.Fatal("LLM profile override did not change the pipeline digest")
} }
resolved := overridden.ResolvedPipeline resolved := overridden.ResolvedPipeline
if resolved.Chunk.LLMProfile != "override-profile" || resolved.ArtifactLanes[0].Extract.LLMProfile != "override-profile" || if resolved.Chunk.LLMProfile != "override-profile" || resolved.Steps[0].ArtifactLanes[0].Extract.LLMProfile != "override-profile" ||
resolved.ArtifactLanes[0].Merge.LLMProfile != "override-profile" || resolved.ArtifactLanes[0].Normalize.LLMProfile != "override-profile" { resolved.Steps[0].ArtifactLanes[0].Merge.LLMProfile != "override-profile" || resolved.Steps[0].ArtifactLanes[0].Normalize.LLMProfile != "override-profile" {
t.Fatalf("pipeline profile override was not applied: %#v", resolved) t.Fatalf("pipeline profile override was not applied: %#v", resolved)
} }
validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane") validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane")
@@ -249,7 +249,7 @@ func TestEffectiveConfigValidatorOverridesRemainDistinctAndOrdered(t *testing.T)
func TestEffectiveConfigAndResolutionInputsDoNotAliasSource(t *testing.T) { func TestEffectiveConfigAndResolutionInputsDoNotAliasSource(t *testing.T) {
profile := effectiveProfile() profile := effectiveProfile()
profile.Chunk.Options = map[string]any{"nested": map[string]any{"safe": "source"}} profile.Chunk.Options = map[string]any{"nested": map[string]any{"safe": "source"}}
profile.Chunk.References = map[string]string{"chunk-ref": "chunk.txt"} profile.Chunk.References = pipeline.ExternalReferenceMap(map[string]string{"chunk-ref": "chunk.txt"})
lane := profile.Artifacts["lane"] lane := profile.Artifacts["lane"]
lane.Extract.Validators = pipeline.ValidatorOverride{ lane.Extract.Validators = pipeline.ValidatorOverride{
Set: true, Set: true,
@@ -282,7 +282,7 @@ func TestEffectiveConfigAndResolutionInputsDoNotAliasSource(t *testing.T) {
if got := cfg.Pipelines["main"].Chunk.Options["nested"].(map[string]any)["safe"]; got != "source" { if got := cfg.Pipelines["main"].Chunk.Options["nested"].(map[string]any)["safe"]; got != "source" {
t.Fatalf("source config option was aliased: %v", got) t.Fatalf("source config option was aliased: %v", got)
} }
if got := cfg.Pipelines["main"].Chunk.References["chunk-ref"]; got != "chunk.txt" { if got := cfg.Pipelines["main"].Chunk.References["chunk-ref"].Path; got != "chunk.txt" {
t.Fatalf("source config references were aliased: %v", got) t.Fatalf("source config references were aliased: %v", got)
} }
if only[0] != "lane" || overrides[0].Source != "source.txt" { if only[0] != "lane" || overrides[0].Source != "source.txt" {

View File

@@ -28,19 +28,88 @@ type FileScriptoriumConfig struct {
} }
type FilePipelineProfile struct { type FilePipelineProfile struct {
Input fileModuleBinding `yaml:"input"` Input fileModuleBinding `yaml:"input"`
Chunk *fileModuleBinding `yaml:"chunk,omitempty"` Chunk *fileModuleBinding `yaml:"chunk,omitempty"`
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"` Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
Output *fileModuleBinding `yaml:"output,omitempty"` Steps []FilePipelineStepProfile `yaml:"steps,omitempty"`
References map[string]string `yaml:"references,omitempty"` Output *fileModuleBinding `yaml:"output,omitempty"`
References map[string]fileReferenceSource `yaml:"references,omitempty"`
artifactsSet bool `yaml:"-"`
stepsSet bool `yaml:"-"`
}
func (p *FilePipelineProfile) UnmarshalYAML(node *yaml.Node) error {
type plainFilePipelineProfile FilePipelineProfile
var decoded plainFilePipelineProfile
seen, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
"input": {}, "chunk": {}, "artifacts": {}, "steps": {}, "output": {}, "references": {},
}, "pipeline profile")
if err != nil {
return err
}
*p = FilePipelineProfile(decoded)
_, p.artifactsSet = seen["artifacts"]
_, p.stepsSet = seen["steps"]
return nil
}
func (s *FilePipelineStepProfile) UnmarshalYAML(node *yaml.Node) error {
type plainFilePipelineStepProfile FilePipelineStepProfile
var decoded plainFilePipelineStepProfile
if _, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
"id": {}, "artifacts": {}, "references": {},
}, "pipeline step"); err != nil {
return err
}
*s = FilePipelineStepProfile(decoded)
return nil
}
func (l *FileArtifactLaneProfile) UnmarshalYAML(node *yaml.Node) error {
type plainFileArtifactLaneProfile FileArtifactLaneProfile
var decoded plainFileArtifactLaneProfile
if _, err := decodeKnownMapping(node, &decoded, map[string]struct{}{
"extract": {}, "merge": {}, "normalize": {}, "validators": {}, "references": {},
}, "artifact lane"); err != nil {
return err
}
*l = FileArtifactLaneProfile(decoded)
return nil
}
func decodeKnownMapping(node *yaml.Node, target any, allowed map[string]struct{}, context string) (map[string]struct{}, error) {
if node.Kind != yaml.MappingNode {
return nil, fmt.Errorf("%s must be an object", context)
}
if err := node.Decode(target); err != nil {
return nil, err
}
seen := make(map[string]struct{}, len(node.Content)/2)
for i := 0; i < len(node.Content); i += 2 {
key := node.Content[i].Value
if _, exists := seen[key]; exists {
return nil, fmt.Errorf("%s field %q is duplicated", context, key)
}
if _, ok := allowed[key]; !ok {
return nil, fmt.Errorf("field %s not found in %s", key, context)
}
seen[key] = struct{}{}
}
return seen, nil
}
type FilePipelineStepProfile struct {
ID string `yaml:"id"`
Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts"`
References map[string]fileReferenceSource `yaml:"references,omitempty"`
} }
type FileArtifactLaneProfile struct { type FileArtifactLaneProfile struct {
Extract fileModuleBinding `yaml:"extract"` Extract fileModuleBinding `yaml:"extract"`
Merge *fileModuleBinding `yaml:"merge,omitempty"` Merge *fileModuleBinding `yaml:"merge,omitempty"`
Normalize *fileModuleBinding `yaml:"normalize,omitempty"` Normalize *fileModuleBinding `yaml:"normalize,omitempty"`
Validators []fileModuleBinding `yaml:"validators,omitempty"` Validators []fileModuleBinding `yaml:"validators,omitempty"`
References map[string]string `yaml:"references,omitempty"` References map[string]fileReferenceSource `yaml:"references,omitempty"`
} }
type FileConcurrencyConfig struct { type FileConcurrencyConfig struct {
@@ -72,10 +141,90 @@ type fileModuleBinding struct {
LLMProfile string LLMProfile string
Retries int Retries int
Options map[string]any Options map[string]any
References map[string]string References map[string]fileReferenceSource
Validators pipeline.ValidatorOverride Validators pipeline.ValidatorOverride
} }
type fileReferenceSource struct {
path string
artifact *pipeline.ArtifactReference
}
func (source *fileReferenceSource) UnmarshalYAML(node *yaml.Node) error {
if source == nil {
return fmt.Errorf("reference source must not be nil")
}
switch node.Kind {
case yaml.ScalarNode:
if node.Tag != "!!str" {
return fmt.Errorf("external reference path must be a string")
}
path := strings.TrimSpace(node.Value)
if path == "" {
return fmt.Errorf("external reference path must not be empty")
}
source.path = path
source.artifact = nil
return nil
case yaml.MappingNode:
if len(node.Content) != 2 || node.Content[0].Value != "artifact" {
return fmt.Errorf("reference source mapping must contain only artifact")
}
artifactNode := node.Content[1]
if artifactNode.Kind != yaml.MappingNode {
return fmt.Errorf("artifact reference must be an object")
}
var step, lane string
seen := map[string]bool{}
for i := 0; i < len(artifactNode.Content); i += 2 {
key := artifactNode.Content[i].Value
value := artifactNode.Content[i+1]
if seen[key] {
return fmt.Errorf("artifact reference field %q is duplicated", key)
}
seen[key] = true
if value.Tag != "!!str" {
return fmt.Errorf("artifact reference field %q must be a string", key)
}
switch key {
case "step":
step = strings.TrimSpace(value.Value)
case "lane":
lane = strings.TrimSpace(value.Value)
default:
return fmt.Errorf("field %s not found in artifact reference", key)
}
}
if step == "" || lane == "" {
return fmt.Errorf("artifact reference step and lane must not be empty")
}
source.path = ""
source.artifact = &pipeline.ArtifactReference{Step: step, Lane: lane}
return nil
default:
return fmt.Errorf("reference source must be a string or object")
}
}
func (source fileReferenceSource) toPipelineSource() pipeline.ReferenceSource {
if source.artifact != nil {
artifact := *source.artifact
return pipeline.ReferenceSource{Artifact: &artifact}
}
return pipeline.ExternalReference(source.path)
}
func fileReferenceSourcesToPipeline(values map[string]fileReferenceSource) map[string]pipeline.ReferenceSource {
if len(values) == 0 {
return nil
}
out := make(map[string]pipeline.ReferenceSource, len(values))
for key, value := range values {
out[strings.TrimSpace(key)] = value.toPipelineSource()
}
return out
}
func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error { func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
switch node.Kind { switch node.Kind {
case yaml.ScalarNode: case yaml.ScalarNode:
@@ -115,7 +264,7 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
} }
b.Options = normalizeOptions(options) b.Options = normalizeOptions(options)
case "references": case "references":
var references map[string]string var references map[string]fileReferenceSource
if err := valueNode.Decode(&references); err != nil { if err := valueNode.Decode(&references); err != nil {
return err return err
} }
@@ -146,7 +295,7 @@ func (b fileModuleBinding) toPipelineBinding() pipeline.ModuleBinding {
LLMProfile: strings.TrimSpace(b.LLMProfile), LLMProfile: strings.TrimSpace(b.LLMProfile),
Retries: b.Retries, Retries: b.Retries,
Options: cloneOptions(b.Options), Options: cloneOptions(b.Options),
References: normalizedStringMap(b.References), References: fileReferenceSourcesToPipeline(b.References),
Validators: b.Validators, Validators: b.Validators,
} }
} }
@@ -214,10 +363,49 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
} }
for _, pipelineID := range pipelineIDs { for _, pipelineID := range pipelineIDs {
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]] filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
hasArtifacts := filePipeline.artifactsSet || filePipeline.Artifacts != nil
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
if hasArtifacts && hasSteps {
return fmt.Errorf("pipeline %q must not declare both artifacts and steps", pipelineID)
}
if hasSteps && len(filePipeline.Steps) == 0 {
return fmt.Errorf("pipeline %q must declare at least one ordered step", pipelineID)
}
if hasSteps {
seenSteps := make(map[string]struct{}, len(filePipeline.Steps))
seenLanes := make(map[string]struct{})
for index, step := range filePipeline.Steps {
stepID := strings.TrimSpace(step.ID)
if stepID == "" {
return fmt.Errorf("pipeline %q step[%d] id must not be empty", pipelineID, index)
}
if _, ok := seenSteps[stepID]; ok {
return fmt.Errorf("pipeline %q step id %q is duplicated after trimming", pipelineID, stepID)
}
seenSteps[stepID] = struct{}{}
laneIDs, rawLaneIDs, err := normalizedMapKeys(step.Artifacts, fmt.Sprintf("pipeline %q step %q artifact lane id", pipelineID, stepID))
if err != nil {
return err
}
for _, laneID := range laneIDs {
if _, ok := seenLanes[laneID]; ok {
return fmt.Errorf("pipeline %q artifact lane id %q is duplicated across steps", pipelineID, laneID)
}
seenLanes[laneID] = struct{}{}
fileLane := step.Artifacts[rawLaneIDs[laneID]]
if err := validateFileLaneReferences(pipelineID, stepID, laneID, fileLane); err != nil {
return err
}
}
if err := validateFileReferenceSources(step.References, fmt.Sprintf("pipeline %q step %q reference slot", pipelineID, stepID)); err != nil {
return err
}
}
}
if _, _, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID)); err != nil { if _, _, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID)); err != nil {
return err return err
} }
if _, _, err := normalizedMapKeys(filePipeline.References, fmt.Sprintf("pipeline %q reference slot", pipelineID)); err != nil { if err := validateFileReferenceSources(filePipeline.References, fmt.Sprintf("pipeline %q reference slot", pipelineID)); err != nil {
return err return err
} }
if filePipeline.Chunk != nil { if filePipeline.Chunk != nil {
@@ -281,6 +469,7 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
for _, pipelineID := range pipelineIDs { for _, pipelineID := range pipelineIDs {
filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]] filePipeline := fileCfg.Pipelines[rawPipelineIDs[pipelineID]]
hasSteps := filePipeline.stepsSet || filePipeline.Steps != nil
laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID)) laneIDs, rawLaneIDs, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID))
if err != nil { if err != nil {
return err return err
@@ -289,7 +478,7 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
ID: pipelineID, ID: pipelineID,
Input: filePipeline.Input.toPipelineBinding(), Input: filePipeline.Input.toPipelineBinding(),
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)), Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
References: normalizedStringMap(filePipeline.References), References: fileReferenceSourcesToPipeline(filePipeline.References),
} }
if filePipeline.Chunk != nil { if filePipeline.Chunk != nil {
profile.Chunk = filePipeline.Chunk.toPipelineBinding() profile.Chunk = filePipeline.Chunk.toPipelineBinding()
@@ -300,10 +489,10 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
for _, laneID := range laneIDs { for _, laneID := range laneIDs {
fileLane := filePipeline.Artifacts[rawLaneIDs[laneID]] fileLane := filePipeline.Artifacts[rawLaneIDs[laneID]]
extract := fileLane.Extract.toPipelineBinding() extract := fileLane.Extract.toPipelineBinding()
extract.References = mergeStringMaps(normalizedStringMap(fileLane.References), extract.References) extract.References = mergeReferenceSources(fileReferenceSourcesToPipeline(fileLane.References), extract.References)
lane := pipeline.ArtifactLaneProfile{ lane := pipeline.ArtifactLaneProfile{
Extract: extract, Extract: extract,
References: normalizedStringMap(fileLane.References), References: fileReferenceSourcesToPipeline(fileLane.References),
} }
if fileLane.Merge != nil { if fileLane.Merge != nil {
lane.Merge = fileLane.Merge.toPipelineBinding() lane.Merge = fileLane.Merge.toPipelineBinding()
@@ -319,6 +508,42 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
} }
profile.Artifacts[laneID] = lane profile.Artifacts[laneID] = lane
} }
if hasSteps {
profile.Artifacts = nil
profile.Steps = make([]pipeline.PipelineStepProfile, len(filePipeline.Steps))
for i, fileStep := range filePipeline.Steps {
stepID := strings.TrimSpace(fileStep.ID)
step := pipeline.PipelineStepProfile{
ID: stepID,
Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(fileStep.Artifacts)),
References: fileReferenceSourcesToPipeline(fileStep.References),
}
stepLaneIDs, stepRawLaneIDs, err := normalizedMapKeys(fileStep.Artifacts, fmt.Sprintf("pipeline %q step %q artifact lane id", pipelineID, stepID))
if err != nil {
return err
}
for _, laneID := range stepLaneIDs {
fileLane := fileStep.Artifacts[stepRawLaneIDs[laneID]]
extract := fileLane.Extract.toPipelineBinding()
extract.References = mergeReferenceSources(fileReferenceSourcesToPipeline(fileLane.References), extract.References)
lane := pipeline.ArtifactLaneProfile{Extract: extract, References: fileReferenceSourcesToPipeline(fileLane.References)}
if fileLane.Merge != nil {
lane.Merge = fileLane.Merge.toPipelineBinding()
}
if fileLane.Normalize != nil {
lane.Normalize = fileLane.Normalize.toPipelineBinding()
}
if len(fileLane.Validators) > 0 {
lane.Validators = make([]pipeline.ModuleBinding, len(fileLane.Validators))
for index, validator := range fileLane.Validators {
lane.Validators[index] = validator.toPipelineBinding()
}
}
step.Artifacts[laneID] = lane
}
profile.Steps[i] = step
}
}
c.Pipelines[pipelineID] = profile c.Pipelines[pipelineID] = profile
} }
@@ -430,30 +655,72 @@ func normalizedMapKeys[T any](values map[string]T, keyName string) ([]string, ma
return keys, rawByNormalized, nil return keys, rawByNormalized, nil
} }
func normalizedStringMap(values map[string]string) map[string]string { func validateFileReferenceSources(values map[string]fileReferenceSource, context string) error {
if len(values) == 0 { seen := make(map[string]struct{}, len(values))
return nil for rawSlot, source := range values {
slot := strings.TrimSpace(rawSlot)
if slot == "" {
return fmt.Errorf("%s must not be empty", context)
}
if _, ok := seen[slot]; ok {
return fmt.Errorf("%s %q is duplicated after trimming", context, slot)
}
seen[slot] = struct{}{}
if source.artifact != nil {
if strings.TrimSpace(source.artifact.Step) == "" || strings.TrimSpace(source.artifact.Lane) == "" {
return fmt.Errorf("%s %q artifact selector step and lane must not be empty", context, slot)
}
if strings.TrimSpace(source.path) != "" {
return fmt.Errorf("%s %q must contain either an external path or artifact selector", context, slot)
}
continue
}
if strings.TrimSpace(source.path) == "" {
return fmt.Errorf("%s %q source must not be empty", context, slot)
}
} }
out := make(map[string]string, len(values)) return nil
keys := make([]string, 0, len(values))
rawByNormalized := make(map[string]string, len(values))
for rawKey := range values {
key := strings.TrimSpace(rawKey)
rawByNormalized[key] = rawKey
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
out[key] = strings.TrimSpace(values[rawByNormalized[key]])
}
return out
} }
func mergeStringMaps(base map[string]string, override map[string]string) map[string]string { func validateFileLaneReferences(pipelineID, stepID, laneID string, lane FileArtifactLaneProfile) error {
prefix := fmt.Sprintf("pipeline %q step %q lane %q", pipelineID, stepID, laneID)
references := []struct {
label string
values map[string]fileReferenceSource
}{
{label: "reference slot", values: lane.References},
{label: "extract reference slot", values: lane.Extract.References},
}
if lane.Merge != nil {
references = append(references, struct {
label string
values map[string]fileReferenceSource
}{label: "merge reference slot", values: lane.Merge.References})
}
if lane.Normalize != nil {
references = append(references, struct {
label string
values map[string]fileReferenceSource
}{label: "normalize reference slot", values: lane.Normalize.References})
}
for _, item := range references {
if err := validateFileReferenceSources(item.values, prefix+" "+item.label); err != nil {
return err
}
}
for index, validator := range lane.Validators {
if err := validateFileReferenceSources(validator.References, fmt.Sprintf("%s validator[%d] reference slot", prefix, index)); err != nil {
return err
}
}
return nil
}
func mergeReferenceSources(base map[string]pipeline.ReferenceSource, override map[string]pipeline.ReferenceSource) map[string]pipeline.ReferenceSource {
if len(base) == 0 && len(override) == 0 { if len(base) == 0 && len(override) == 0 {
return nil return nil
} }
out := make(map[string]string, len(base)+len(override)) out := make(map[string]pipeline.ReferenceSource, len(base)+len(override))
for key, value := range base { for key, value := range base {
out[key] = value out[key] = value
} }

View File

@@ -142,7 +142,7 @@ pipelines:
} }
if profile.Chunk.Module != "generic" || profile.Chunk.LLMProfile != "chunk-profile" || profile.Chunk.Retries != 2 || if profile.Chunk.Module != "generic" || profile.Chunk.LLMProfile != "chunk-profile" || profile.Chunk.Retries != 2 ||
!reflect.DeepEqual(profile.Chunk.Options, map[string]any{"max_units": 25}) || !reflect.DeepEqual(profile.Chunk.Options, map[string]any{"max_units": 25}) ||
!reflect.DeepEqual(profile.Chunk.References, map[string]string{"glossary": "./glossary.md"}) { !reflect.DeepEqual(profile.Chunk.References, pipeline.ExternalReferenceMap(map[string]string{"glossary": "./glossary.md"})) {
t.Fatalf("object binding = %#v", profile.Chunk) t.Fatalf("object binding = %#v", profile.Chunk)
} }
if !profile.Chunk.Validators.Set || len(profile.Chunk.Validators.Validators) != 0 { if !profile.Chunk.Validators.Set || len(profile.Chunk.Validators.Validators) != 0 {
@@ -192,33 +192,33 @@ pipelines:
normalize-only: ./normalize.txt normalize-only: ./normalize.txt
`) `)
profile := cfg.Pipelines["main"] profile := cfg.Pipelines["main"]
if !reflect.DeepEqual(profile.References, map[string]string{ if !reflect.DeepEqual(profile.References, pipeline.ExternalReferenceMap(map[string]string{
"pipeline-only": "./pipeline.txt", "pipeline-only": "./pipeline.txt",
"shared": "./pipeline-shared.txt", "shared": "./pipeline-shared.txt",
}) { })) {
t.Fatalf("pipeline references = %#v", profile.References) t.Fatalf("pipeline references = %#v", profile.References)
} }
if !reflect.DeepEqual(profile.Chunk.References, map[string]string{"chunk-only": "./chunk.txt"}) { if !reflect.DeepEqual(profile.Chunk.References, pipeline.ExternalReferenceMap(map[string]string{"chunk-only": "./chunk.txt"})) {
t.Fatalf("chunk references = %#v", profile.Chunk.References) t.Fatalf("chunk references = %#v", profile.Chunk.References)
} }
lane := profile.Artifacts["spells"] lane := profile.Artifacts["spells"]
if !reflect.DeepEqual(lane.References, map[string]string{ if !reflect.DeepEqual(lane.References, pipeline.ExternalReferenceMap(map[string]string{
"lane-only": "./lane.txt", "lane-only": "./lane.txt",
"shared": "./lane-shared.txt", "shared": "./lane-shared.txt",
"overridden": "./lane.txt", "overridden": "./lane.txt",
}) { })) {
t.Fatalf("lane compatibility references = %#v", lane.References) t.Fatalf("lane compatibility references = %#v", lane.References)
} }
if !reflect.DeepEqual(lane.Extract.References, map[string]string{ if !reflect.DeepEqual(lane.Extract.References, pipeline.ExternalReferenceMap(map[string]string{
"lane-only": "./lane.txt", "lane-only": "./lane.txt",
"shared": "./lane-shared.txt", "shared": "./lane-shared.txt",
"overridden": "./extract-overridden.txt", "overridden": "./extract-overridden.txt",
"extract-only": "./extract.txt", "extract-only": "./extract.txt",
}) { })) {
t.Fatalf("extract references = %#v", lane.Extract.References) t.Fatalf("extract references = %#v", lane.Extract.References)
} }
if !reflect.DeepEqual(lane.Merge.References, map[string]string{"merge-only": "./merge.txt"}) || if !reflect.DeepEqual(lane.Merge.References, pipeline.ExternalReferenceMap(map[string]string{"merge-only": "./merge.txt"})) ||
!reflect.DeepEqual(lane.Normalize.References, map[string]string{"normalize-only": "./normalize.txt"}) { !reflect.DeepEqual(lane.Normalize.References, pipeline.ExternalReferenceMap(map[string]string{"normalize-only": "./normalize.txt"})) {
t.Fatalf("merge/normalize references = %#v, %#v", lane.Merge.References, lane.Normalize.References) t.Fatalf("merge/normalize references = %#v, %#v", lane.Merge.References, lane.Normalize.References)
} }
} }
@@ -357,6 +357,82 @@ func TestFileConfigRejectsTrimmedKeyCollisions(t *testing.T) {
} }
} }
func TestFileConfigParsesOrderedStepsAndReferenceSources(t *testing.T) {
file := parseFileConfig(t, `version: 3
pipelines:
session:
input: seriatim
steps:
- id: identify-npcs
artifacts:
npcs:
extract: dnd/npcs
- id: grounded-events
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
artifacts:
spells:
extract: dnd/spells
`)
cfg := Default()
if err := cfg.ApplyFileConfig(file); err != nil {
t.Fatal(err)
}
profile := cfg.Pipelines["session"]
if len(profile.Steps) != 2 || profile.Steps[0].ID != "identify-npcs" || profile.Steps[1].ID != "grounded-events" {
t.Fatalf("steps = %#v", profile.Steps)
}
source := profile.Steps[1].References["npcs"]
if source.Artifact == nil || source.Artifact.Step != "identify-npcs" || source.Artifact.Lane != "npcs" {
t.Fatalf("generated source = %#v", source)
}
}
func TestFileConfigRejectsAmbiguousReferenceSourceForms(t *testing.T) {
for _, source := range []string{
"artifact: {step: a, lane: b, extra: c}",
"artifact: {step: 1, lane: b}",
"1",
} {
_, err := ParseFileConfigYAML([]byte("version: 3\npipelines:\n p:\n input: text\n references:\n slot: " + source + "\n"))
if err == nil {
t.Fatalf("ParseFileConfigYAML(%q) error = nil", source)
}
}
}
func TestFileConfigRejectsEmptyAndAmbiguousPipelineShapes(t *testing.T) {
tests := []struct {
name string
yaml string
want string
}{
{
name: "empty steps",
yaml: "version: 3\npipelines:\n p:\n input: text\n steps: []\n",
want: "at least one ordered step",
},
{
name: "both forms",
yaml: "version: 3\npipelines:\n p:\n input: text\n artifacts: {}\n steps: []\n",
want: "both artifacts and steps",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
file := parseFileConfig(t, tt.yaml)
cfg := Default()
err := cfg.ApplyFileConfig(file)
if err == nil || !strings.Contains(err.Error(), tt.want) {
t.Fatalf("ApplyFileConfig() error = %v, want context %q", err, tt.want)
}
})
}
}
func TestLoadFileConfigReportsPathAndOperationContext(t *testing.T) { func TestLoadFileConfigReportsPathAndOperationContext(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
missing := filepath.Join(dir, "missing.yml") missing := filepath.Join(dir, "missing.yml")

View File

@@ -42,10 +42,16 @@ func cloneResolvedPipeline(in pipeline.ResolvedPipeline) pipeline.ResolvedPipeli
out.ValidatorChains[i] = cloneResolvedValidatorChain(chain) out.ValidatorChains[i] = cloneResolvedValidatorChain(chain)
} }
} }
if len(in.ArtifactLanes) > 0 { if len(in.Steps) > 0 {
out.ArtifactLanes = make([]pipeline.ResolvedArtifactLane, len(in.ArtifactLanes)) out.Steps = make([]pipeline.ResolvedPipelineStep, len(in.Steps))
for i, lane := range in.ArtifactLanes { for i, step := range in.Steps {
out.ArtifactLanes[i] = cloneResolvedArtifactLane(lane) out.Steps[i] = pipeline.ResolvedPipelineStep{ID: step.ID}
if len(step.ArtifactLanes) > 0 {
out.Steps[i].ArtifactLanes = make([]pipeline.ResolvedArtifactLane, len(step.ArtifactLanes))
for j, lane := range step.ArtifactLanes {
out.Steps[i].ArtifactLanes[j] = cloneResolvedArtifactLane(lane)
}
}
} }
} }
return out return out
@@ -89,14 +95,20 @@ func redactConfig(cfg Config) Config {
profile.Input = redactBinding(profile.Input) profile.Input = redactBinding(profile.Input)
profile.Chunk = redactBinding(profile.Chunk) profile.Chunk = redactBinding(profile.Chunk)
profile.Output = redactBinding(profile.Output) profile.Output = redactBinding(profile.Output)
for laneID, lane := range profile.Artifacts { redactLanes := func(lanes map[string]pipeline.ArtifactLaneProfile) {
lane.Extract = redactBinding(lane.Extract) for laneID, lane := range lanes {
lane.Merge = redactBinding(lane.Merge) lane.Extract = redactBinding(lane.Extract)
lane.Normalize = redactBinding(lane.Normalize) lane.Merge = redactBinding(lane.Merge)
for i := range lane.Validators { lane.Normalize = redactBinding(lane.Normalize)
lane.Validators[i] = redactBinding(lane.Validators[i]) for i := range lane.Validators {
lane.Validators[i] = redactBinding(lane.Validators[i])
}
lanes[laneID] = lane
} }
profile.Artifacts[laneID] = lane }
redactLanes(profile.Artifacts)
for i := range profile.Steps {
redactLanes(profile.Steps[i].Artifacts)
} }
cfg.Pipelines[id] = profile cfg.Pipelines[id] = profile
} }

View File

@@ -24,16 +24,19 @@ func TestRedactedResolvedPipelinePayloadRedactsEveryBinding(t *testing.T) {
Input: bindings["input"], Input: bindings["input"],
Chunk: bindings["chunk"], Chunk: bindings["chunk"],
ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"), ChunkReferences: redactionTestReferenceTarget(pipeline.StageChunk, "", "chunk-reference-content"),
ArtifactLanes: []pipeline.ResolvedArtifactLane{{ Steps: []pipeline.ResolvedPipelineStep{{
ID: "safe-lane", ID: "default",
ArtifactKind: "safe/artifact", ArtifactLanes: []pipeline.ResolvedArtifactLane{{
Extract: bindings["extract"], ID: "safe-lane",
Merge: bindings["merge"], ArtifactKind: "safe/artifact",
Normalize: bindings["normalize"], Extract: bindings["extract"],
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]}, Merge: bindings["merge"],
ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"), Normalize: bindings["normalize"],
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"), Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
NormalizeReferences: redactionTestReferenceTarget(pipeline.StageNormalize, "safe-lane", "normalize-reference-content"), ExtractReferences: redactionTestReferenceTarget(pipeline.StageExtract, "safe-lane", "extract-reference-content"),
MergeReferences: redactionTestReferenceTarget(pipeline.StageMerge, "safe-lane", "merge-reference-content"),
NormalizeReferences: redactionTestReferenceTarget(pipeline.StageNormalize, "safe-lane", "normalize-reference-content"),
}},
}}, }},
ValidatorChains: []pipeline.ResolvedValidatorChain{{ ValidatorChains: []pipeline.ResolvedValidatorChain{{
Stage: pipeline.StageExtract, Stage: pipeline.StageExtract,
@@ -142,12 +145,15 @@ func TestRedactedSummaryPayloadsCoverEveryEffectiveConfigBinding(t *testing.T) {
Input: bindings["input"], Input: bindings["input"],
Chunk: bindings["chunk"], Chunk: bindings["chunk"],
Output: bindings["output"], Output: bindings["output"],
ArtifactLanes: []pipeline.ResolvedArtifactLane{{ Steps: []pipeline.ResolvedPipelineStep{{
ID: "safe-lane", ID: "default",
Extract: bindings["extract"], ArtifactLanes: []pipeline.ResolvedArtifactLane{{
Merge: bindings["merge"], ID: "safe-lane",
Normalize: bindings["normalize"], Extract: bindings["extract"],
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]}, Merge: bindings["merge"],
Normalize: bindings["normalize"],
Validators: []pipeline.ModuleBinding{bindings["lane-validator"]},
}},
}}, }},
}, },
} }

View File

@@ -106,30 +106,60 @@ func validatePipelineProfiles(profiles map[string]pipeline.PipelineProfile) erro
if err := validateReferenceMap(id, "", profile.References); err != nil { if err := validateReferenceMap(id, "", profile.References); err != nil {
return err return err
} }
seenLanes := make(map[string]struct{}, len(profile.Artifacts)) explicitSteps := profile.Steps != nil
for rawLaneID, lane := range profile.Artifacts { if len(profile.Artifacts) > 0 && explicitSteps {
laneID := strings.TrimSpace(rawLaneID) return fmt.Errorf("pipeline %q must not declare both artifacts and steps", id)
if laneID == "" { }
return fmt.Errorf("pipeline %q artifact lane id must not be empty", id) steps := profile.Steps
if !explicitSteps {
steps = []pipeline.PipelineStepProfile{{ID: "default", Artifacts: profile.Artifacts}}
}
if explicitSteps && len(steps) == 0 {
return fmt.Errorf("pipeline %q must declare at least one ordered step", id)
}
seenSteps := make(map[string]struct{}, len(steps))
seenLanes := make(map[string]struct{})
for index, step := range steps {
stepID := strings.TrimSpace(step.ID)
if stepID == "" {
return fmt.Errorf("pipeline %q step[%d] id must not be empty", id, index)
} }
if _, ok := seenLanes[laneID]; ok { if _, ok := seenSteps[stepID]; ok {
return fmt.Errorf("pipeline %q artifact lane id %q is duplicated after trimming", id, laneID) return fmt.Errorf("pipeline %q step id %q is duplicated after trimming", id, stepID)
} }
seenLanes[laneID] = struct{}{} seenSteps[stepID] = struct{}{}
if err := validateReferenceMap(id, laneID, lane.References); err != nil { if explicitSteps {
return err if err := validateReferenceMapForContext(id, "", "step "+stepID, step.References, true); err != nil {
return err
}
} }
if err := validateBinding(id, laneID, "extract", lane.Extract, true); err != nil { for rawLaneID, lane := range step.Artifacts {
return err laneID := strings.TrimSpace(rawLaneID)
} if laneID == "" {
if err := validateBinding(id, laneID, "merge", lane.Merge, true); err != nil { return fmt.Errorf("pipeline %q artifact lane id must not be empty", id)
return err }
} if _, ok := seenLanes[laneID]; ok {
if err := validateBinding(id, laneID, "normalize", lane.Normalize, true); err != nil { if !explicitSteps {
return err return fmt.Errorf("pipeline %q artifact lane id %q is duplicated after trimming", id, laneID)
} }
if len(lane.Validators) > 0 { return fmt.Errorf("pipeline %q artifact lane id %q is duplicated across steps", id, laneID)
return fmt.Errorf("pipeline %q lane %q validators are not supported at artifact lane level; use extract.validators, merge.validators, or normalize.validators", id, laneID) }
seenLanes[laneID] = struct{}{}
if err := validateReferenceMapForContext(id, laneID, "", lane.References, true); err != nil {
return err
}
if err := validateBinding(id, laneID, "extract", lane.Extract, true); err != nil {
return err
}
if err := validateBinding(id, laneID, "merge", lane.Merge, true); err != nil {
return err
}
if err := validateBinding(id, laneID, "normalize", lane.Normalize, true); err != nil {
return err
}
if len(lane.Validators) > 0 {
return fmt.Errorf("pipeline %q lane %q validators are not supported at artifact lane level; use extract.validators, merge.validators, or normalize.validators", id, laneID)
}
} }
} }
} }
@@ -164,7 +194,7 @@ func validateBinding(
} }
return fmt.Errorf("pipeline %q %s references are not supported", pipelineID, slot) return fmt.Errorf("pipeline %q %s references are not supported", pipelineID, slot)
} }
return validateReferenceMapForContext(pipelineID, laneID, slot, binding.References) return validateReferenceMapForContext(pipelineID, laneID, slot, binding.References, true)
} }
func validateValidatorOverride(pipelineID string, laneID string, slot string, override pipeline.ValidatorOverride) error { func validateValidatorOverride(pipelineID string, laneID string, slot string, override pipeline.ValidatorOverride) error {
@@ -197,13 +227,13 @@ func validateValidatorOverride(pipelineID string, laneID string, slot string, ov
return nil return nil
} }
func validateReferenceMap(pipelineID string, laneID string, references map[string]string) error { func validateReferenceMap(pipelineID string, laneID string, references map[string]pipeline.ReferenceSource) error {
return validateReferenceMapForContext(pipelineID, laneID, "", references) return validateReferenceMapForContext(pipelineID, laneID, "", references, false)
} }
func validateReferenceMapForContext(pipelineID string, laneID string, slot string, references map[string]string) error { func validateReferenceMapForContext(pipelineID string, laneID string, slot string, references map[string]pipeline.ReferenceSource, generatedAllowed bool) error {
seen := make(map[string]struct{}, len(references)) seen := make(map[string]struct{}, len(references))
for rawSlotName, rawSource := range references { for rawSlotName, source := range references {
slotName := strings.TrimSpace(rawSlotName) slotName := strings.TrimSpace(rawSlotName)
if slotName == "" { if slotName == "" {
return fmt.Errorf("%s reference slot name must not be empty", referenceContext(pipelineID, laneID, slot)) return fmt.Errorf("%s reference slot name must not be empty", referenceContext(pipelineID, laneID, slot))
@@ -212,7 +242,19 @@ func validateReferenceMapForContext(pipelineID string, laneID string, slot strin
return fmt.Errorf("%s reference slot %q is duplicated after trimming", referenceContext(pipelineID, laneID, slot), slotName) return fmt.Errorf("%s reference slot %q is duplicated after trimming", referenceContext(pipelineID, laneID, slot), slotName)
} }
seen[slotName] = struct{}{} seen[slotName] = struct{}{}
if strings.TrimSpace(rawSource) == "" { if source.Artifact != nil {
if !generatedAllowed {
return fmt.Errorf("%s reference slot %q must use an external path", referenceContext(pipelineID, laneID, slot), slotName)
}
if strings.TrimSpace(source.Artifact.Step) == "" || strings.TrimSpace(source.Artifact.Lane) == "" {
return fmt.Errorf("%s reference slot %q artifact selector step and lane must not be empty", referenceContext(pipelineID, laneID, slot), slotName)
}
if strings.TrimSpace(source.Path) != "" {
return fmt.Errorf("%s reference slot %q must contain exactly one source form", referenceContext(pipelineID, laneID, slot), slotName)
}
continue
}
if strings.TrimSpace(source.Path) == "" {
return fmt.Errorf("%s reference slot %q source must not be empty", referenceContext(pipelineID, laneID, slot), slotName) return fmt.Errorf("%s reference slot %q source must not be empty", referenceContext(pipelineID, laneID, slot), slotName)
} }
} }

View File

@@ -200,7 +200,7 @@ func TestValidateIdentifiersAfterTrimming(t *testing.T) {
name: "empty reference slot", name: "empty reference slot",
setup: func(cfg *Config) { setup: func(cfg *Config) {
profile := validationProfile() profile := validationProfile()
profile.References = map[string]string{" ": "source.txt"} profile.References = pipeline.ExternalReferenceMap(map[string]string{" ": "source.txt"})
cfg.Pipelines = map[string]pipeline.PipelineProfile{"main": profile} cfg.Pipelines = map[string]pipeline.PipelineProfile{"main": profile}
}, },
want: "reference slot name must not be empty", want: "reference slot name must not be empty",
@@ -209,7 +209,7 @@ func TestValidateIdentifiersAfterTrimming(t *testing.T) {
name: "duplicate reference slots", name: "duplicate reference slots",
setup: func(cfg *Config) { setup: func(cfg *Config) {
profile := validationProfile() profile := validationProfile()
profile.References = map[string]string{"slot": "one.txt", " slot ": "two.txt"} profile.References = pipeline.ExternalReferenceMap(map[string]string{"slot": "one.txt", " slot ": "two.txt"})
cfg.Pipelines = map[string]pipeline.PipelineProfile{"main": profile} cfg.Pipelines = map[string]pipeline.PipelineProfile{"main": profile}
}, },
want: "reference slot \"slot\" is duplicated after trimming", want: "reference slot \"slot\" is duplicated after trimming",
@@ -267,14 +267,14 @@ func TestValidateReferencesAreUnsupportedOnInputAndOutput(t *testing.T) {
{ {
name: "input references", name: "input references",
set: func(profile *pipeline.PipelineProfile) { set: func(profile *pipeline.PipelineProfile) {
profile.Input.References = map[string]string{"slot": "source.txt"} profile.Input.References = pipeline.ExternalReferenceMap(map[string]string{"slot": "source.txt"})
}, },
want: "input references are not supported", want: "input references are not supported",
}, },
{ {
name: "output references", name: "output references",
set: func(profile *pipeline.PipelineProfile) { set: func(profile *pipeline.PipelineProfile) {
profile.Output.References = map[string]string{"slot": "source.txt"} profile.Output.References = pipeline.ExternalReferenceMap(map[string]string{"slot": "source.txt"})
}, },
want: "output references are not supported", want: "output references are not supported",
}, },
@@ -326,7 +326,7 @@ func TestValidateValidatorBindingRules(t *testing.T) {
Set: true, Set: true,
Validators: []pipeline.ModuleBinding{{ Validators: []pipeline.ModuleBinding{{
Module: "validator", Module: "validator",
References: map[string]string{"slot": "source.txt"}, References: pipeline.ExternalReferenceMap(map[string]string{"slot": "source.txt"}),
}}, }},
} }
}, },

View File

@@ -33,6 +33,7 @@ type Invocation struct {
PipelineID string `json:"pipeline_id,omitempty"` PipelineID string `json:"pipeline_id,omitempty"`
PipelineDigest string `json:"pipeline_digest,omitempty"` PipelineDigest string `json:"pipeline_digest,omitempty"`
Resume bool `json:"resume,omitempty"` Resume bool `json:"resume,omitempty"`
RecomputeStep string `json:"recompute_step,omitempty"`
InputPath string `json:"input_path,omitempty"` InputPath string `json:"input_path,omitempty"`
ConfigPath string `json:"config_path,omitempty"` ConfigPath string `json:"config_path,omitempty"`
ConfigSource string `json:"config_source,omitempty"` ConfigSource string `json:"config_source,omitempty"`

View File

@@ -8,6 +8,7 @@ import (
"runtime" "runtime"
"strings" "strings"
"testing" "testing"
"unicode/utf8"
"gitea.maximumdirect.net/eric/notarius/internal/core/source" "gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
@@ -137,7 +138,7 @@ func TestFilesystemCheckpointRejectsMissingAndCorruptState(t *testing.T) {
if err := os.Remove(stage.manifest(fixture)); err != nil { if err := os.Remove(stage.manifest(fixture)); err != nil {
t.Fatal(err) t.Fatal(err)
} }
assertStageNotReused(t, stage, fixture, "missing") assertStageNotReused(t, stage, fixture, pipeline.CheckpointReasonMissing)
}) })
t.Run(stage.name+" malformed manifest", func(t *testing.T) { t.Run(stage.name+" malformed manifest", func(t *testing.T) {
@@ -145,7 +146,7 @@ func TestFilesystemCheckpointRejectsMissingAndCorruptState(t *testing.T) {
if err := os.WriteFile(stage.manifest(fixture), []byte("{"), 0o600); err != nil { if err := os.WriteFile(stage.manifest(fixture), []byte("{"), 0o600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
assertStageNotReused(t, stage, fixture, "decode") assertStageNotReused(t, stage, fixture, pipeline.CheckpointReasonDecodeFailed)
}) })
} }
} }
@@ -154,17 +155,18 @@ func TestFilesystemCheckpointRejectsIncompatibleManifests(t *testing.T) {
for _, tt := range []struct { for _, tt := range []struct {
name string name string
edit func(map[string]any) edit func(map[string]any)
want string want pipeline.CheckpointReasonCode
}{ }{
{"v1 schema", func(m map[string]any) { m["workspace_schema_version"] = WorkspaceSchemaVersionV1 }, "workspace schema"}, {"v1 schema", func(m map[string]any) { m["workspace_schema_version"] = WorkspaceSchemaVersionV1 }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
{"unknown schema", func(m map[string]any) { m["workspace_schema_version"] = "notarius.workspace.future" }, "workspace schema"}, {"v2 schema", func(m map[string]any) { m["workspace_schema_version"] = WorkspaceSchemaVersionV2 }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
{"identity", func(m map[string]any) { m["metadata"].(map[string]any)["checkpoint_identity_digest"] = "sha256:other" }, "identity"}, {"unknown schema", func(m map[string]any) { m["workspace_schema_version"] = "notarius.workspace.future" }, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
{"stage", func(m map[string]any) { m["stage"] = string(StageMerge) }, "stage"}, {"identity", func(m map[string]any) { m["metadata"].(map[string]any)["checkpoint_identity_digest"] = "sha256:other" }, pipeline.CheckpointReasonIdentityMismatch},
{"lane", func(m map[string]any) { m["lane_id"] = "lane-other" }, "lane"}, {"stage", func(m map[string]any) { m["stage"] = string(StageMerge) }, pipeline.CheckpointReasonStageMismatch},
{"module", func(m map[string]any) { m["module_key"] = "module-other" }, "module"}, {"lane", func(m map[string]any) { m["lane_id"] = "lane-other" }, pipeline.CheckpointReasonLaneMismatch},
{"module", func(m map[string]any) { m["module_key"] = "module-other" }, pipeline.CheckpointReasonModuleMismatch},
{"dependency", func(m map[string]any) { {"dependency", func(m map[string]any) {
m["dependency_fingerprints"] = []map[string]string{{"name": "input", "value": "other"}} m["dependency_fingerprints"] = []map[string]string{{"name": "input", "value": "other"}}
}, "dependency"}, }, pipeline.CheckpointReasonDependencyMismatch},
} { } {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
fixture := seedFilesystemCheckpoints(t) fixture := seedFilesystemCheckpoints(t)
@@ -179,7 +181,7 @@ func TestFilesystemCheckpointRejectsNonTerminalStatuses(t *testing.T) {
t.Run(string(status), func(t *testing.T) { t.Run(string(status), func(t *testing.T) {
fixture := seedFilesystemCheckpoints(t) fixture := seedFilesystemCheckpoints(t)
editManifest(t, checkpointStages()[1].manifest(fixture), func(m map[string]any) { m["status"] = string(status) }) editManifest(t, checkpointStages()[1].manifest(fixture), func(m map[string]any) { m["status"] = string(status) })
assertStageNotReused(t, checkpointStages()[1], fixture, "status") assertStageNotReused(t, checkpointStages()[1], fixture, pipeline.CheckpointReasonStatusNotReusable)
}) })
} }
} }
@@ -188,20 +190,20 @@ func TestFilesystemCheckpointRejectsIncompleteArtifactsAndContent(t *testing.T)
for _, tt := range []struct { for _, tt := range []struct {
name string name string
edit func(map[string]any) edit func(map[string]any)
want string want pipeline.CheckpointReasonCode
}{ }{
{"artifact kind", func(m map[string]any) { m["outputs"].([]any)[0].(map[string]any)["artifact_kind"] = "" }, "artifact codec identity"}, {"artifact kind", func(m map[string]any) { m["outputs"].([]any)[0].(map[string]any)["artifact_kind"] = "" }, pipeline.CheckpointReasonArtifactCodecIncompatible},
{"schema id", func(m map[string]any) { m["outputs"].([]any)[0].(map[string]any)["schema"].(map[string]any)["id"] = "" }, "artifact codec identity"}, {"schema id", func(m map[string]any) { m["outputs"].([]any)[0].(map[string]any)["schema"].(map[string]any)["id"] = "" }, pipeline.CheckpointReasonArtifactCodecIncompatible},
{"schema version", func(m map[string]any) { {"schema version", func(m map[string]any) {
m["outputs"].([]any)[0].(map[string]any)["schema"].(map[string]any)["version"] = "" m["outputs"].([]any)[0].(map[string]any)["schema"].(map[string]any)["version"] = ""
}, "artifact codec identity"}, }, pipeline.CheckpointReasonArtifactCodecIncompatible},
{"schema digest", func(m map[string]any) { m["outputs"].([]any)[0].(map[string]any)["schema_digest"] = "" }, "artifact codec identity"}, {"schema digest", func(m map[string]any) { m["outputs"].([]any)[0].(map[string]any)["schema_digest"] = "" }, pipeline.CheckpointReasonArtifactCodecIncompatible},
{"base64", func(m map[string]any) { {"base64", func(m map[string]any) {
m["outputs"].([]any)[0].(map[string]any)["content"].(map[string]any)["content_base64"] = "%" m["outputs"].([]any)[0].(map[string]any)["content"].(map[string]any)["content_base64"] = "%"
}, "base64"}, }, pipeline.CheckpointReasonArtifactPayloadInvalid},
{"content digest", func(m map[string]any) { {"content digest", func(m map[string]any) {
m["outputs"].([]any)[0].(map[string]any)["content"].(map[string]any)["content_digest"] = "sha256:other" m["outputs"].([]any)[0].(map[string]any)["content"].(map[string]any)["content_digest"] = "sha256:other"
}, "content digest"}, }, pipeline.CheckpointReasonArtifactDigestMismatch},
} { } {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
fixture := seedFilesystemCheckpoints(t) fixture := seedFilesystemCheckpoints(t)
@@ -217,7 +219,7 @@ func TestFilesystemCheckpointRejectsSourceAndOutputDigestMismatches(t *testing.T
editJSON(t, filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), "source", "source-document.json"), func(m map[string]any) { editJSON(t, filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), "source", "source-document.json"), func(m map[string]any) {
m["document"].(map[string]any)["units"].([]any)[0].(map[string]any)["text"] = "" m["document"].(map[string]any)["units"].([]any)[0].(map[string]any)["text"] = ""
}) })
assertStageNotReused(t, checkpointStages()[0], fixture, "source checkpoint document") assertStageNotReused(t, checkpointStages()[0], fixture, pipeline.CheckpointReasonArtifactPayloadInvalid)
}) })
t.Run("source output digest", func(t *testing.T) { t.Run("source output digest", func(t *testing.T) {
@@ -225,7 +227,7 @@ func TestFilesystemCheckpointRejectsSourceAndOutputDigestMismatches(t *testing.T
editJSON(t, filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), "source", "source-document.json"), func(m map[string]any) { editJSON(t, filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), "source", "source-document.json"), func(m map[string]any) {
m["document"].(map[string]any)["digest"] = "sha256:other" m["document"].(map[string]any)["digest"] = "sha256:other"
}) })
assertStageNotReused(t, checkpointStages()[0], fixture, "output digest") assertStageNotReused(t, checkpointStages()[0], fixture, pipeline.CheckpointReasonArtifactDigestMismatch)
}) })
for _, stage := range checkpointStages()[1:] { for _, stage := range checkpointStages()[1:] {
@@ -234,7 +236,7 @@ func TestFilesystemCheckpointRejectsSourceAndOutputDigestMismatches(t *testing.T
editManifest(t, stage.manifest(fixture), func(m map[string]any) { editManifest(t, stage.manifest(fixture), func(m map[string]any) {
m["output_digests"].([]any)[0].(map[string]any)["value"] = "sha256:other" m["output_digests"].([]any)[0].(map[string]any)["value"] = "sha256:other"
}) })
assertStageNotReused(t, stage, fixture, "output digest") assertStageNotReused(t, stage, fixture, pipeline.CheckpointReasonArtifactDigestMismatch)
}) })
} }
} }
@@ -247,6 +249,180 @@ func TestFilesystemCheckpointReusesExtractWithRejections(t *testing.T) {
} }
} }
func TestFilesystemCheckpointDependencyDecisionIsBoundedAndCategorized(t *testing.T) {
fixture := seedFilesystemCheckpoints(t)
_, decision := fixture.loader.Extract("lane-a", "extract-module", []pipeline.CheckpointFingerprint{{Name: "source", Value: "sha256:changed"}})
if decision.Reused || decision.Category != "dependency_invalidated" || decision.ReasonCode != "dependency_mismatch" {
t.Fatalf("dependency decision = %#v", decision)
}
if strings.Contains(decision.Reason, fixture.root) || strings.Contains(decision.Detail, fixture.root) {
t.Fatalf("dependency decision leaked checkpoint path: %#v", decision)
}
}
func TestFilesystemCheckpointDecisionFamiliesAreStableAndSafe(t *testing.T) {
const secretSentinel = "do-not-expose-checkpoint-secret"
tests := []struct {
name string
prepare func(*testing.T, filesystemCheckpointFixture) pipeline.CheckpointDecision
category pipeline.CheckpointDecisionCategory
code pipeline.CheckpointReasonCode
}{
{"loading disabled", func(t *testing.T, _ filesystemCheckpointFixture) pipeline.CheckpointDecision {
_, decision := pipeline.NoopCheckpointLoader().Source("source")
return decision
}, pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonLoadingDisabled},
{"checkpoint unavailable", func(t *testing.T, fixture filesystemCheckpointFixture) pipeline.CheckpointDecision {
if err := os.Remove(checkpointStages()[1].manifest(fixture)); err != nil {
t.Fatal(err)
}
return checkpointStages()[1].load(fixture)
}, pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonMissing},
{"workspace identity", func(t *testing.T, fixture filesystemCheckpointFixture) pipeline.CheckpointDecision {
editManifest(t, checkpointStages()[1].manifest(fixture), func(m map[string]any) {
m["workspace_schema_version"] = secretSentinel
})
return checkpointStages()[1].load(fixture)
}, pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonWorkspaceSchemaIncompatible},
{"manifest scope", func(t *testing.T, fixture filesystemCheckpointFixture) pipeline.CheckpointDecision {
editManifest(t, checkpointStages()[1].manifest(fixture), func(m map[string]any) { m["module_key"] = secretSentinel })
return checkpointStages()[1].load(fixture)
}, pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonModuleMismatch},
{"dependency invalidated", func(t *testing.T, fixture filesystemCheckpointFixture) pipeline.CheckpointDecision {
_, decision := fixture.loader.Extract("lane-a", "extract-module", []pipeline.CheckpointFingerprint{{Name: "input", Value: secretSentinel}})
return decision
}, pipeline.CheckpointDecisionDependencyInvalidated, pipeline.CheckpointReasonDependencyMismatch},
{"artifact payload", func(t *testing.T, fixture filesystemCheckpointFixture) pipeline.CheckpointDecision {
editJSON(t, filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), "extract", "lane-a", "outputs.json"), func(m map[string]any) {
m["outputs"].([]any)[0].(map[string]any)["artifact_kind"] = ""
m["outputs"].([]any)[0].(map[string]any)["source_id"] = secretSentinel
})
return checkpointStages()[1].load(fixture)
}, pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactCodecIncompatible},
{"checkpoint reused", func(t *testing.T, fixture filesystemCheckpointFixture) pipeline.CheckpointDecision {
return checkpointStages()[1].load(fixture)
}, pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonReused},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
fixture := seedFilesystemCheckpoints(t)
decision := test.prepare(t, fixture)
if decision.Category != test.category || decision.ReasonCode != test.code {
t.Fatalf("decision = %#v, want category %q and code %q", decision, test.category, test.code)
}
if len([]byte(decision.Detail)) > 512 || !utf8.ValidString(decision.Detail) {
t.Fatalf("decision detail is not bounded valid UTF-8: %#v", decision)
}
for _, forbidden := range []string{secretSentinel, fixture.root} {
if strings.Contains(decision.Detail, forbidden) || strings.Contains(decision.Reason, forbidden) {
t.Fatalf("decision leaked %q: %#v", forbidden, decision)
}
}
})
}
}
func TestFilesystemLoaderReadsAcceptedNormalizeWithoutStageDependencies(t *testing.T) {
fixture := seedAcceptedNormalizeCheckpoint(t)
identityRoot := filepath.Join(fixture.root, mustRelativePath(t, fixture.identity))
for _, stage := range []string{"extract", "merge"} {
if _, err := os.Stat(filepath.Join(identityRoot, laneManifestPath(stage, "step-1", "lane-a"))); !os.IsNotExist(err) {
t.Fatalf("%s checkpoint stat error = %v, want absent prerequisite", stage, err)
}
}
checkpoint, decision := fixture.loader.AcceptedNormalize("step-1", "lane-a", "normalize-module")
if !decision.Reused || decision.Category != pipeline.CheckpointDecisionReused || decision.ReasonCode != pipeline.CheckpointReasonAcceptedArtifactReused {
t.Fatalf("accepted normalize decision = %#v", decision)
}
if checkpoint.Output.Artifact.Content == nil || string(checkpoint.Output.Artifact.Content) != string(fixture.normalize.Artifact.Content) {
t.Fatalf("accepted normalize output = %#v, want recorded artifact", checkpoint.Output)
}
if len(checkpoint.Warnings) != 1 || checkpoint.Warnings[0].ReasonCode != "normalized" {
t.Fatalf("accepted normalize warnings = %#v", checkpoint.Warnings)
}
}
func TestFilesystemLoaderRejectsInvalidAcceptedNormalize(t *testing.T) {
tests := []struct {
name string
mutate func(*testing.T, *filesystemCheckpointFixture)
code pipeline.CheckpointReasonCode
}{
{"missing", func(t *testing.T, fixture *filesystemCheckpointFixture) {
if err := os.Remove(acceptedNormalizeManifest(t, *fixture)); err != nil {
t.Fatal(err)
}
}, pipeline.CheckpointReasonMissing},
{"rejected status", func(t *testing.T, fixture *filesystemCheckpointFixture) {
editManifest(t, acceptedNormalizeManifest(t, *fixture), func(m map[string]any) { m["status"] = string(StatusSucceededWithRejections) })
}, pipeline.CheckpointReasonStatusNotReusable},
{"corrupt payload", func(t *testing.T, fixture *filesystemCheckpointFixture) {
if err := os.WriteFile(acceptedNormalizePayload(t, *fixture), []byte("{"), 0o600); err != nil {
t.Fatal(err)
}
}, pipeline.CheckpointReasonDecodeFailed},
{"wrong codec identity", func(t *testing.T, fixture *filesystemCheckpointFixture) {
editJSON(t, acceptedNormalizePayload(t, *fixture), func(m map[string]any) { m["output"].(map[string]any)["artifact_kind"] = "" })
}, pipeline.CheckpointReasonArtifactCodecIncompatible},
{"wrong content digest", func(t *testing.T, fixture *filesystemCheckpointFixture) {
editJSON(t, acceptedNormalizePayload(t, *fixture), func(m map[string]any) {
m["output"].(map[string]any)["content"].(map[string]any)["content_digest"] = "sha256:wrong"
})
}, pipeline.CheckpointReasonArtifactDigestMismatch},
{"unverifiable identity", func(t *testing.T, fixture *filesystemCheckpointFixture) {
loader := fixture.loader.(*FilesystemLoader)
fixture.loader = &FilesystemLoader{root: loader.root}
}, pipeline.CheckpointReasonIdentityMismatch},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
fixture := seedAcceptedNormalizeCheckpoint(t)
test.mutate(t, &fixture)
_, decision := fixture.loader.AcceptedNormalize("step-1", "lane-a", "normalize-module")
if decision.Reused || decision.ReasonCode != test.code {
t.Fatalf("accepted normalize decision = %#v, want %q", decision, test.code)
}
if strings.Contains(decision.Detail, fixture.root) || strings.Contains(decision.Detail, string(fixture.normalize.Artifact.Content)) {
t.Fatalf("accepted normalize decision leaked path or content: %#v", decision)
}
if _, err := os.Stat(acceptedNormalizePayload(t, fixture)); err != nil {
t.Fatalf("accepted normalize payload was removed after rejection: %v", err)
}
})
}
}
func seedAcceptedNormalizeCheckpoint(t *testing.T) filesystemCheckpointFixture {
t.Helper()
root := t.TempDir()
identity := testIdentity(t)
artifact := checkpointArtifact("normalize-module", `{"accepted":true}`)
recorder, err := NewFilesystemRecorder(root, identity)
if err != nil {
t.Fatal(err)
}
stepRecorder := recorder.(pipeline.StepCheckpointRecorder)
warnings := []contracts.Warning{{Scope: "normalize", ReasonCode: "normalized", Message: "normalized warning"}}
if err := stepRecorder.NormalizeSucceededForStep("step-1", "lane-a", "normalize-module", []pipeline.CheckpointFingerprint{{Name: "merge", Value: "sha256:unavailable"}}, artifact, warnings); err != nil {
t.Fatal(err)
}
loader, err := NewFilesystemLoader(root, identity)
if err != nil {
t.Fatal(err)
}
return filesystemCheckpointFixture{root: root, identity: identity, loader: loader, normalize: artifact}
}
func acceptedNormalizeManifest(t *testing.T, fixture filesystemCheckpointFixture) string {
t.Helper()
return filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), laneManifestPath("normalize", "step-1", "lane-a"))
}
func acceptedNormalizePayload(t *testing.T, fixture filesystemCheckpointFixture) string {
t.Helper()
return filepath.Join(fixture.root, mustRelativePath(t, fixture.identity), lanePayloadPath("normalize", "step-1", "lane-a", "output.json"))
}
type checkpointStage struct { type checkpointStage struct {
name string name string
manifest func(filesystemCheckpointFixture) string manifest func(filesystemCheckpointFixture) string
@@ -355,11 +531,11 @@ func checkpointArtifact(module, content string) pipeline.CheckpointArtifact {
} }
} }
func assertStageNotReused(t *testing.T, stage checkpointStage, fixture filesystemCheckpointFixture, want string) { func assertStageNotReused(t *testing.T, stage checkpointStage, fixture filesystemCheckpointFixture, want pipeline.CheckpointReasonCode) {
t.Helper() t.Helper()
decision := stage.load(fixture) decision := stage.load(fixture)
if decision.Reused || !strings.Contains(strings.ToLower(decision.Reason), strings.ToLower(want)) { if decision.Reused || decision.ReasonCode != want {
t.Fatalf("%s decision=%#v, want non-reused reason containing %q", stage.name, decision, want) t.Fatalf("%s decision=%#v, want non-reused reason code %q", stage.name, decision, want)
} }
} }

View File

@@ -31,6 +31,7 @@ type Identity struct {
Digest string `json:"digest"` Digest string `json:"digest"`
PipelineID string `json:"pipeline_id"` PipelineID string `json:"pipeline_id"`
PipelineDigest string `json:"pipeline_digest"` PipelineDigest string `json:"pipeline_digest"`
PipelineTopology []string `json:"pipeline_topology,omitempty"`
InputKey string `json:"input_key"` InputKey string `json:"input_key"`
RawInputDigest string `json:"raw_input_digest,omitempty"` RawInputDigest string `json:"raw_input_digest,omitempty"`
SourceDigest string `json:"source_digest,omitempty"` SourceDigest string `json:"source_digest,omitempty"`
@@ -57,8 +58,8 @@ func NewIdentity(input IdentityInput) (Identity, error) {
if strings.TrimSpace(input.RawInputDigest) == "" && strings.TrimSpace(input.SourceDigest) == "" { if strings.TrimSpace(input.RawInputDigest) == "" && strings.TrimSpace(input.SourceDigest) == "" {
return Identity{}, fmt.Errorf("checkpoint identity raw input digest or source digest must be set") return Identity{}, fmt.Errorf("checkpoint identity raw input digest or source digest must be set")
} }
v := Identity{PipelineID: pipelineID, PipelineDigest: pipelineDigest, InputKey: inputKey, RawInputDigest: strings.TrimSpace(input.RawInputDigest), SourceDigest: strings.TrimSpace(input.SourceDigest), SelectedLanes: normalizedLanes(input.SelectedLanes, input.Pipeline.ArtifactLanes), RuntimeOverrides: normalizeIdentityFingerprints(input.RuntimeOverrides), ReferenceDigests: referenceFingerprints(input.References), ProvenanceFingerprints: normalizeIdentityFingerprints(input.ProvenanceFingerprints)} v := Identity{PipelineID: pipelineID, PipelineDigest: pipelineDigest, PipelineTopology: pipelineTopology(input.Pipeline), InputKey: inputKey, RawInputDigest: strings.TrimSpace(input.RawInputDigest), SourceDigest: strings.TrimSpace(input.SourceDigest), SelectedLanes: normalizedLanes(input.SelectedLanes, input.Pipeline.AllArtifactLanes()), RuntimeOverrides: normalizeIdentityFingerprints(input.RuntimeOverrides), ReferenceDigests: referenceFingerprints(input.References), ProvenanceFingerprints: normalizeIdentityFingerprints(input.ProvenanceFingerprints)}
data, err := json.Marshal(Identity{PipelineID: v.PipelineID, PipelineDigest: v.PipelineDigest, InputKey: v.InputKey, RawInputDigest: v.RawInputDigest, SourceDigest: v.SourceDigest, SelectedLanes: v.SelectedLanes, RuntimeOverrides: v.RuntimeOverrides, ReferenceDigests: v.ReferenceDigests, ProvenanceFingerprints: v.ProvenanceFingerprints}) data, err := json.Marshal(Identity{PipelineID: v.PipelineID, PipelineDigest: v.PipelineDigest, PipelineTopology: v.PipelineTopology, InputKey: v.InputKey, RawInputDigest: v.RawInputDigest, SourceDigest: v.SourceDigest, SelectedLanes: v.SelectedLanes, RuntimeOverrides: v.RuntimeOverrides, ReferenceDigests: v.ReferenceDigests, ProvenanceFingerprints: v.ProvenanceFingerprints})
if err != nil { if err != nil {
return Identity{}, fmt.Errorf("marshal checkpoint identity: %w", err) return Identity{}, fmt.Errorf("marshal checkpoint identity: %w", err)
} }
@@ -66,6 +67,19 @@ func NewIdentity(input IdentityInput) (Identity, error) {
v.Digest = "sha256:" + hex.EncodeToString(sum[:]) v.Digest = "sha256:" + hex.EncodeToString(sum[:])
return v, nil return v, nil
} }
func pipelineTopology(resolved pipeline.ResolvedPipeline) []string {
var topology []string
for _, step := range resolved.Steps {
for _, lane := range step.ArtifactLanes {
topology = append(topology, strings.TrimSpace(step.ID)+"\x00"+strings.TrimSpace(lane.ID))
}
}
if len(topology) == 0 {
return nil
}
return topology
}
func (i Identity) RelativePath() (string, error) { func (i Identity) RelativePath() (string, error) {
p, err := safeComponent(i.PipelineID) p, err := safeComponent(i.PipelineID)
if err != nil { if err != nil {

View File

@@ -32,7 +32,9 @@ func TestNewIdentityNormalizesOrderAndEmptyValues(t *testing.T) {
v.ProvenanceFingerprints = []Fingerprint{{Name: "source", Value: "v2"}, {Name: "runner", Value: "v1"}} v.ProvenanceFingerprints = []Fingerprint{{Name: "source", Value: "v2"}, {Name: "runner", Value: "v1"}}
}}, }},
{"resolved lanes", func(v *IdentityInput) { {"resolved lanes", func(v *IdentityInput) {
v.Pipeline.ArtifactLanes = []pipeline.ResolvedArtifactLane{v.Pipeline.ArtifactLanes[1], v.Pipeline.ArtifactLanes[0]} steps := append([]pipeline.ResolvedPipelineStep(nil), v.Pipeline.Steps...)
steps[0].ArtifactLanes = []pipeline.ResolvedArtifactLane{steps[0].ArtifactLanes[1], steps[0].ArtifactLanes[0]}
v.Pipeline.Steps = steps
}}, }},
} { } {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
@@ -42,6 +44,12 @@ func TestNewIdentityNormalizesOrderAndEmptyValues(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if tt.name == "resolved lanes" {
if reflect.DeepEqual(identity, got) {
t.Fatal("reordered topology did not change checkpoint identity")
}
return
}
if !reflect.DeepEqual(identity, got) { if !reflect.DeepEqual(identity, got) {
t.Fatalf("reordered identity differs:\nbase=%#v\ngot=%#v", identity, got) t.Fatalf("reordered identity differs:\nbase=%#v\ngot=%#v", identity, got)
} }
@@ -174,10 +182,13 @@ func representativeIdentityInput() IdentityInput {
ID: "pipeline", ID: "pipeline",
Digest: "sha256:pipeline-digest-000000000000", Digest: "sha256:pipeline-digest-000000000000",
Input: pipeline.Binding("input"), Input: pipeline.Binding("input"),
ArtifactLanes: []pipeline.ResolvedArtifactLane{ Steps: []pipeline.ResolvedPipelineStep{{
{ID: "lane-b"}, ID: "default",
{ID: "lane-a"}, ArtifactLanes: []pipeline.ResolvedArtifactLane{
}, {ID: "lane-b"},
{ID: "lane-a"},
},
}},
}, },
InputKey: "input", InputKey: "input",
RawInputDigest: "sha256:raw-input-000000000000", RawInputDigest: "sha256:raw-input-000000000000",
@@ -197,6 +208,6 @@ func cloneIdentityInput(input IdentityInput) IdentityInput {
input.RuntimeOverrides = append([]Fingerprint(nil), input.RuntimeOverrides...) input.RuntimeOverrides = append([]Fingerprint(nil), input.RuntimeOverrides...)
input.References = append([]artifacts.ReferenceProvenance(nil), input.References...) input.References = append([]artifacts.ReferenceProvenance(nil), input.References...)
input.ProvenanceFingerprints = append([]Fingerprint(nil), input.ProvenanceFingerprints...) input.ProvenanceFingerprints = append([]Fingerprint(nil), input.ProvenanceFingerprints...)
input.Pipeline.ArtifactLanes = append([]pipeline.ResolvedArtifactLane(nil), input.Pipeline.ArtifactLanes...) input.Pipeline.Steps[0].ArtifactLanes = append([]pipeline.ResolvedArtifactLane(nil), input.Pipeline.Steps[0].ArtifactLanes...)
return input return input
} }

View File

@@ -52,83 +52,146 @@ func (l *FilesystemLoader) Source(moduleKey string) (pipeline.SourceCheckpoint,
} }
doc := cloneSourceDocument(payload.Document) doc := cloneSourceDocument(payload.Document)
if err := source.ValidateDocument(&doc); err != nil { if err := source.ValidateDocument(&doc); err != nil {
return pipeline.SourceCheckpoint{}, invalidDecision("source checkpoint document is invalid: %v", err) return pipeline.SourceCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactPayloadInvalid, "source checkpoint document is invalid")
} }
if strings.TrimSpace(manifest.SourceID) != "" && manifest.SourceID != doc.ID { if strings.TrimSpace(manifest.SourceID) != "" && manifest.SourceID != doc.ID {
return pipeline.SourceCheckpoint{}, invalidDecision("source checkpoint source id does not match payload") return pipeline.SourceCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactPayloadInvalid, "source checkpoint identity does not match its payload")
} }
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), digestFingerprints("source_document", doc.Digest)) { if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), digestFingerprints("source_document", doc.Digest)) {
return pipeline.SourceCheckpoint{}, invalidDecision("source checkpoint output digest does not match payload") return pipeline.SourceCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch, "source checkpoint output digest does not match its payload")
} }
return pipeline.SourceCheckpoint{Document: &doc}, reusedDecision() return pipeline.SourceCheckpoint{Document: &doc}, reusedDecision()
} }
func (l *FilesystemLoader) Extract(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.ExtractCheckpoint, pipeline.CheckpointDecision) { func (l *FilesystemLoader) Extract(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.ExtractCheckpoint, pipeline.CheckpointDecision) {
return l.ExtractForStep("", laneID, moduleKey, dependencies)
}
func (l *FilesystemLoader) ExtractForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.ExtractCheckpoint, pipeline.CheckpointDecision) {
var manifest ExtractLaneManifest var manifest ExtractLaneManifest
if d := l.readJSON(laneManifestPath("extract", laneID), &manifest); !d.Reused { if d := l.readJSON(laneManifestPath("extract", stepID, laneID), &manifest); !d.Reused {
return pipeline.ExtractCheckpoint{}, d return pipeline.ExtractCheckpoint{}, d
} }
if d := l.validateLaneManifest(manifest.StageManifest, StageExtract, laneID, moduleKey, dependencies, StatusSucceeded, StatusSucceededWithRejections); !d.Reused { if d := l.validateLaneManifest(manifest.StageManifest, StageExtract, stepID, laneID, moduleKey, dependencies, StatusSucceeded, StatusSucceededWithRejections); !d.Reused {
return pipeline.ExtractCheckpoint{}, d return pipeline.ExtractCheckpoint{}, d
} }
var payload artifactExtractEnvelope var payload artifactExtractEnvelope
if d := l.readJSON(lanePayloadPath("extract", laneID, "outputs.json"), &payload); !d.Reused { if d := l.readJSON(lanePayloadPath("extract", stepID, laneID, "outputs.json"), &payload); !d.Reused {
return pipeline.ExtractCheckpoint{}, d return pipeline.ExtractCheckpoint{}, d
} }
outputs, err := artifactCheckpointOutputs(payload.Outputs) outputs, err := artifactCheckpointOutputs(payload.Outputs)
if err != nil { if err != nil {
return pipeline.ExtractCheckpoint{}, invalidDecision("extract artifact checkpoint payload is invalid: %v", err) return pipeline.ExtractCheckpoint{}, artifactDecision(err, "extract checkpoint artifact is invalid")
} }
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(outputs)) { if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(outputs)) {
return pipeline.ExtractCheckpoint{}, invalidDecision("extract artifact checkpoint output digests do not match payload") return pipeline.ExtractCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch, "extract checkpoint output digest does not match its payload")
} }
return pipeline.ExtractCheckpoint{Outputs: outputs, Rejected: cloneRejectedOutputs(payload.Rejected), Warnings: cloneWarnings(payload.Warnings)}, reusedDecision() return pipeline.ExtractCheckpoint{Outputs: outputs, Rejected: cloneRejectedOutputs(payload.Rejected), Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
} }
func (l *FilesystemLoader) Merge(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) { func (l *FilesystemLoader) Merge(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
return l.MergeForStep("", laneID, moduleKey, dependencies)
}
func (l *FilesystemLoader) MergeForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
var manifest MergeLaneManifest var manifest MergeLaneManifest
if d := l.readJSON(laneManifestPath("merge", laneID), &manifest); !d.Reused { if d := l.readJSON(laneManifestPath("merge", stepID, laneID), &manifest); !d.Reused {
return pipeline.MergeCheckpoint{}, d return pipeline.MergeCheckpoint{}, d
} }
if d := l.validateLaneManifest(manifest.StageManifest, StageMerge, laneID, moduleKey, dependencies, StatusSucceeded); !d.Reused { if d := l.validateLaneManifest(manifest.StageManifest, StageMerge, stepID, laneID, moduleKey, dependencies, StatusSucceeded); !d.Reused {
return pipeline.MergeCheckpoint{}, d return pipeline.MergeCheckpoint{}, d
} }
var payload artifactSingleEnvelope var payload artifactSingleEnvelope
if d := l.readJSON(lanePayloadPath("merge", laneID, "output.json"), &payload); !d.Reused { if d := l.readJSON(lanePayloadPath("merge", stepID, laneID, "output.json"), &payload); !d.Reused {
return pipeline.MergeCheckpoint{}, d return pipeline.MergeCheckpoint{}, d
} }
values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output}) values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
if err != nil { if err != nil {
return pipeline.MergeCheckpoint{}, invalidDecision("merge artifact checkpoint payload is invalid: %v", err) return pipeline.MergeCheckpoint{}, artifactDecision(err, "merge checkpoint artifact is invalid")
} }
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) { if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
return pipeline.MergeCheckpoint{}, invalidDecision("merge artifact checkpoint output digest does not match payload") return pipeline.MergeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch, "merge checkpoint output digest does not match its payload")
} }
return pipeline.MergeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision() return pipeline.MergeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
} }
func (l *FilesystemLoader) Normalize(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) { func (l *FilesystemLoader) Normalize(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
return l.NormalizeForStep("", laneID, moduleKey, dependencies)
}
func (l *FilesystemLoader) NormalizeForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
var manifest NormalizeLaneManifest var manifest NormalizeLaneManifest
if d := l.readJSON(laneManifestPath("normalize", laneID), &manifest); !d.Reused { if d := l.readJSON(laneManifestPath("normalize", stepID, laneID), &manifest); !d.Reused {
return pipeline.NormalizeCheckpoint{}, d return pipeline.NormalizeCheckpoint{}, d
} }
if d := l.validateLaneManifest(manifest.StageManifest, StageNormalize, laneID, moduleKey, dependencies, StatusSucceeded); !d.Reused { if d := l.validateLaneManifest(manifest.StageManifest, StageNormalize, stepID, laneID, moduleKey, dependencies, StatusSucceeded); !d.Reused {
return pipeline.NormalizeCheckpoint{}, d return pipeline.NormalizeCheckpoint{}, d
} }
var payload artifactSingleEnvelope var payload artifactSingleEnvelope
if d := l.readJSON(lanePayloadPath("normalize", laneID, "output.json"), &payload); !d.Reused { if d := l.readJSON(lanePayloadPath("normalize", stepID, laneID, "output.json"), &payload); !d.Reused {
return pipeline.NormalizeCheckpoint{}, d return pipeline.NormalizeCheckpoint{}, d
} }
values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output}) values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
if err != nil { if err != nil {
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize artifact checkpoint payload is invalid: %v", err) return pipeline.NormalizeCheckpoint{}, artifactDecision(err, "normalize checkpoint artifact is invalid")
} }
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) { if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize artifact checkpoint output digest does not match payload") return pipeline.NormalizeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch, "normalize checkpoint output digest does not match its payload")
} }
return pipeline.NormalizeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision() return pipeline.NormalizeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
} }
func (l *FilesystemLoader) AcceptedNormalize(stepID, laneID, moduleKey string) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
var manifest NormalizeLaneManifest
if d := l.readJSON(laneManifestPath("normalize", stepID, laneID), &manifest); !d.Reused {
return pipeline.NormalizeCheckpoint{}, d
}
if d := l.validateAcceptedNormalizeManifest(manifest.StageManifest, stepID, laneID, moduleKey); !d.Reused {
return pipeline.NormalizeCheckpoint{}, d
}
var payload artifactSingleEnvelope
if d := l.readJSON(lanePayloadPath("normalize", stepID, laneID, "output.json"), &payload); !d.Reused {
return pipeline.NormalizeCheckpoint{}, d
}
values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
if err != nil {
return pipeline.NormalizeCheckpoint{}, artifactDecision(err, "accepted normalize checkpoint artifact is invalid")
}
if len(values) != 1 {
return pipeline.NormalizeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactPayloadInvalid, "accepted normalize checkpoint payload is invalid")
}
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
return pipeline.NormalizeCheckpoint{}, decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonArtifactDigestMismatch, "accepted normalize checkpoint digest does not match its payload")
}
return pipeline.NormalizeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, decision(pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonAcceptedArtifactReused, "accepted normalized artifact is reusable")
}
func (l *FilesystemLoader) validateAcceptedNormalizeManifest(manifest StageManifest, stepID, laneID, moduleKey string) pipeline.CheckpointDecision {
if manifest.WorkspaceSchemaVersion != WorkspaceSchemaVersion {
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonWorkspaceSchemaIncompatible, "checkpoint workspace schema is incompatible")
}
identity := strings.TrimSpace(l.identityDigest)
if identity == "" || strings.TrimSpace(manifest.Metadata["checkpoint_identity_digest"]) == "" || manifest.Metadata["checkpoint_identity_digest"] != identity {
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonIdentityMismatch, "checkpoint identity is unavailable or does not match the current invocation")
}
if manifest.Stage != StageNormalize {
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonStageMismatch, "checkpoint stage does not match normalize")
}
if strings.TrimSpace(stepID) == "" || manifest.StepID != stepID {
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonStepMismatch, "checkpoint step does not match the requested step")
}
if strings.TrimSpace(laneID) == "" || manifest.LaneID != laneID {
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonLaneMismatch, "checkpoint lane does not match the requested lane")
}
if strings.TrimSpace(moduleKey) == "" || manifest.ModuleKey != moduleKey {
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonModuleMismatch, "checkpoint module does not match the requested normalizer")
}
if manifest.Status != StatusSucceeded {
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonStatusNotReusable, "checkpoint status cannot provide an accepted normalized artifact")
}
return reusedDecision()
}
func artifactCheckpointOutputs(values []artifactCheckpointEnvelope) ([]pipeline.CheckpointArtifact, error) { func artifactCheckpointOutputs(values []artifactCheckpointEnvelope) ([]pipeline.CheckpointArtifact, error) {
if len(values) == 0 { if len(values) == 0 {
return nil, nil return nil, nil
@@ -140,7 +203,7 @@ func artifactCheckpointOutputs(values []artifactCheckpointEnvelope) ([]pipeline.
return nil, err return nil, err
} }
if strings.TrimSpace(string(v.Kind)) == "" || strings.TrimSpace(v.Schema.ID) == "" || strings.TrimSpace(v.Schema.Version) == "" || strings.TrimSpace(v.SchemaDigest) == "" { if strings.TrimSpace(string(v.Kind)) == "" || strings.TrimSpace(v.Schema.ID) == "" || strings.TrimSpace(v.Schema.Version) == "" || strings.TrimSpace(v.SchemaDigest) == "" {
return nil, fmt.Errorf("artifact codec identity is incomplete") return nil, &artifactPayloadError{code: pipeline.CheckpointReasonArtifactCodecIncompatible, err: fmt.Errorf("artifact codec identity is incomplete")}
} }
out = append(out, pipeline.CheckpointArtifact{LaneID: v.LaneID, ModuleKey: v.ModuleKey, SourceID: v.SourceID, ChunkID: v.ChunkID, ChunkIndex: v.ChunkIndex, ChunkRef: v.ChunkRef, SchemaDigest: v.SchemaDigest, Artifact: contracts.SerializedArtifact{Kind: v.Kind, Schema: v.Schema, MediaType: v.Content.MediaType, Content: content, Metadata: cloneMetadata(v.Content.Metadata)}}) out = append(out, pipeline.CheckpointArtifact{LaneID: v.LaneID, ModuleKey: v.ModuleKey, SourceID: v.SourceID, ChunkID: v.ChunkID, ChunkIndex: v.ChunkIndex, ChunkRef: v.ChunkRef, SchemaDigest: v.SchemaDigest, Artifact: contracts.SerializedArtifact{Kind: v.Kind, Schema: v.Schema, MediaType: v.Content.MediaType, Content: content, Metadata: cloneMetadata(v.Content.Metadata)}})
} }
@@ -149,47 +212,53 @@ func artifactCheckpointOutputs(values []artifactCheckpointEnvelope) ([]pipeline.
func (l *FilesystemLoader) readJSON(name string, out any) pipeline.CheckpointDecision { func (l *FilesystemLoader) readJSON(name string, out any) pipeline.CheckpointDecision {
if !l.Enabled() { if !l.Enabled() {
return pipeline.CheckpointDecision{Reason: "checkpoint loading disabled"} return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonLoadingDisabled, "checkpoint loading disabled")
} }
target, err := fileio.SafePath(l.root, name) target, err := fileio.SafePath(l.root, name)
if err != nil { if err != nil {
return invalidDecision("checkpoint path is invalid: %v", err) return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonPathInvalid, "checkpoint path is invalid")
} }
data, err := os.ReadFile(target) data, err := os.ReadFile(target)
if err != nil { if err != nil {
if os.IsNotExist(err) { if os.IsNotExist(err) {
return pipeline.CheckpointDecision{Reason: "checkpoint artifact is missing"} return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonMissing, "checkpoint artifact is missing")
} }
return invalidDecision("read checkpoint artifact: %v", err) return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonReadFailed, "checkpoint artifact could not be read")
} }
if err := json.Unmarshal(data, out); err != nil { if err := json.Unmarshal(data, out); err != nil {
return invalidDecision("decode checkpoint artifact: %v", err) return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonDecodeFailed, "checkpoint artifact could not be decoded")
} }
return reusedDecision() return reusedDecision()
} }
func (l *FilesystemLoader) validateManifest(manifest StageManifest, stage StageName, laneID string, moduleKey string, status StageStatus, dependencies []pipeline.CheckpointFingerprint) pipeline.CheckpointDecision { func (l *FilesystemLoader) validateManifest(manifest StageManifest, stage StageName, laneID string, moduleKey string, status StageStatus, dependencies []pipeline.CheckpointFingerprint) pipeline.CheckpointDecision {
return l.validateLaneManifest(manifest, stage, laneID, moduleKey, dependencies, status) return l.validateLaneManifest(manifest, stage, "", laneID, moduleKey, dependencies, status)
} }
func (l *FilesystemLoader) validateLaneManifest(manifest StageManifest, stage StageName, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, statuses ...StageStatus) pipeline.CheckpointDecision { func (l *FilesystemLoader) validateLaneManifest(manifest StageManifest, stage StageName, stepID string, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, statuses ...StageStatus) pipeline.CheckpointDecision {
if manifest.WorkspaceSchemaVersion == WorkspaceSchemaVersionV1 { if manifest.WorkspaceSchemaVersion == WorkspaceSchemaVersionV1 {
return invalidDecision("checkpoint workspace schema version %q is incompatible with %q and must be recomputed", manifest.WorkspaceSchemaVersion, WorkspaceSchemaVersion) return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonWorkspaceSchemaIncompatible, "checkpoint workspace schema is incompatible")
}
if manifest.WorkspaceSchemaVersion == WorkspaceSchemaVersionV2 {
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonWorkspaceSchemaIncompatible, "checkpoint workspace schema is incompatible")
} }
if manifest.WorkspaceSchemaVersion != WorkspaceSchemaVersion { if manifest.WorkspaceSchemaVersion != WorkspaceSchemaVersion {
return invalidDecision("checkpoint workspace schema version %q is not supported", manifest.WorkspaceSchemaVersion) return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonWorkspaceSchemaIncompatible, "checkpoint workspace schema is incompatible")
} }
if strings.TrimSpace(l.identityDigest) != "" && manifest.Metadata["checkpoint_identity_digest"] != l.identityDigest { if strings.TrimSpace(l.identityDigest) != "" && manifest.Metadata["checkpoint_identity_digest"] != l.identityDigest {
return invalidDecision("checkpoint identity digest does not match current invocation") return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonIdentityMismatch, "checkpoint identity does not match the current invocation")
} }
if manifest.Stage != stage { if manifest.Stage != stage {
return invalidDecision("checkpoint stage %q does not match %q", manifest.Stage, stage) return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonStageMismatch, "checkpoint stage does not match the requested stage")
}
if strings.TrimSpace(stepID) != "" && manifest.StepID != stepID {
return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonStepMismatch, "checkpoint step does not match the requested step")
} }
if strings.TrimSpace(laneID) != "" && manifest.LaneID != laneID { if strings.TrimSpace(laneID) != "" && manifest.LaneID != laneID {
return invalidDecision("checkpoint lane %q does not match %q", manifest.LaneID, laneID) return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonLaneMismatch, "checkpoint lane does not match the requested lane")
} }
if strings.TrimSpace(moduleKey) != "" && manifest.ModuleKey != moduleKey { if strings.TrimSpace(moduleKey) != "" && manifest.ModuleKey != moduleKey {
return invalidDecision("checkpoint module %q does not match %q", manifest.ModuleKey, moduleKey) return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonModuleMismatch, "checkpoint module does not match the requested module")
} }
statusOK := false statusOK := false
for _, status := range statuses { for _, status := range statuses {
@@ -199,10 +268,10 @@ func (l *FilesystemLoader) validateLaneManifest(manifest StageManifest, stage St
} }
} }
if !statusOK { if !statusOK {
return invalidDecision("checkpoint status %q cannot be reused", manifest.Status) return decision(pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonStatusNotReusable, "checkpoint status cannot be reused")
} }
if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.DependencyFingerprints), dependencies) { if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.DependencyFingerprints), dependencies) {
return invalidDecision("checkpoint dependency fingerprints do not match") return decision(pipeline.CheckpointDecisionDependencyInvalidated, pipeline.CheckpointReasonDependencyMismatch, "checkpoint dependencies do not match")
} }
return reusedDecision() return reusedDecision()
} }
@@ -210,10 +279,10 @@ func (l *FilesystemLoader) validateLaneManifest(manifest StageManifest, stage St
func contentFromEnvelope(value binaryEnvelope) ([]byte, error) { func contentFromEnvelope(value binaryEnvelope) ([]byte, error) {
content, err := base64.StdEncoding.DecodeString(value.ContentBase64) content, err := base64.StdEncoding.DecodeString(value.ContentBase64)
if err != nil { if err != nil {
return nil, fmt.Errorf("decode content_base64: %w", err) return nil, &artifactPayloadError{code: pipeline.CheckpointReasonArtifactPayloadInvalid, err: fmt.Errorf("decode content_base64: %w", err)}
} }
if digest := strings.TrimSpace(value.ContentDigest); digest != "" && digest != contentDigest(content) { if digest := strings.TrimSpace(value.ContentDigest); digest != "" && digest != contentDigest(content) {
return nil, fmt.Errorf("content digest mismatch") return nil, &artifactPayloadError{code: pipeline.CheckpointReasonArtifactDigestMismatch, err: fmt.Errorf("content digest mismatch")}
} }
return content, nil return content, nil
} }
@@ -244,9 +313,24 @@ func fingerprintsEqual(a []pipeline.CheckpointFingerprint, b []pipeline.Checkpoi
} }
func reusedDecision() pipeline.CheckpointDecision { func reusedDecision() pipeline.CheckpointDecision {
return pipeline.CheckpointDecision{Reused: true, Reason: "checkpoint is valid"} return decision(pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonReused, "checkpoint is reusable")
} }
func invalidDecision(format string, args ...any) pipeline.CheckpointDecision { func decision(category pipeline.CheckpointDecisionCategory, code pipeline.CheckpointReasonCode, detail string) pipeline.CheckpointDecision {
return pipeline.CheckpointDecision{Reason: fmt.Sprintf(format, args...)} return pipeline.NewCheckpointDecision(category, code, detail)
}
type artifactPayloadError struct {
code pipeline.CheckpointReasonCode
err error
}
func (e *artifactPayloadError) Error() string { return e.err.Error() }
func artifactDecision(err error, detail string) pipeline.CheckpointDecision {
code := pipeline.CheckpointReasonArtifactPayloadInvalid
if payloadErr, ok := err.(*artifactPayloadError); ok {
code = payloadErr.code
}
return decision(pipeline.CheckpointDecisionExecuted, code, detail)
} }

View File

@@ -3,9 +3,8 @@ package checkpoint
import "time" import "time"
const ( const (
// These names and values are frozen checkpoint wire-compatibility WorkspaceSchemaVersion = "notarius.workspace.v3"
// identifiers. They intentionally retain the former terminology. WorkspaceSchemaVersionV2 = "notarius.workspace.v2"
WorkspaceSchemaVersion = "notarius.workspace.v2"
WorkspaceSchemaVersionV1 = "notarius.workspace.v1" WorkspaceSchemaVersionV1 = "notarius.workspace.v1"
) )
@@ -32,6 +31,7 @@ const (
type StageManifest struct { type StageManifest struct {
WorkspaceSchemaVersion string `json:"workspace_schema_version"` WorkspaceSchemaVersion string `json:"workspace_schema_version"`
Stage StageName `json:"stage"` Stage StageName `json:"stage"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"` LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"` ModuleKey string `json:"module_key,omitempty"`
DependencyFingerprints []Fingerprint `json:"dependency_fingerprints,omitempty"` DependencyFingerprints []Fingerprint `json:"dependency_fingerprints,omitempty"`

View File

@@ -71,93 +71,137 @@ func (r *FilesystemRecorder) SourceFailed(moduleKey string, err error) error {
} }
func (r *FilesystemRecorder) ExtractRunning(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error { func (r *FilesystemRecorder) ExtractRunning(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
manifest := r.laneManifest(StageExtract, StatusRunning, laneID, moduleKey, dependencies) return r.ExtractRunningForStep("", laneID, moduleKey, dependencies)
}
func (r *FilesystemRecorder) ExtractRunningForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
manifest := r.laneManifest(StageExtract, StatusRunning, stepID, laneID, moduleKey, dependencies)
manifest.StartedAt = timePtr(r.timestamp()) manifest.StartedAt = timePtr(r.timestamp())
return r.writeManifest(laneManifestPath("extract", laneID), ExtractLaneManifest{StageManifest: manifest}) return r.writeManifest(laneManifestPath("extract", stepID, laneID), ExtractLaneManifest{StageManifest: manifest})
} }
func (r *FilesystemRecorder) ExtractSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error { func (r *FilesystemRecorder) ExtractSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
return r.ExtractSucceededForStep("", laneID, moduleKey, dependencies, outputs, rejected, warnings)
}
func (r *FilesystemRecorder) ExtractSucceededForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
payload := artifactExtractEnvelope{Outputs: artifactCheckpointEnvelopes(outputs), Rejected: cloneRejectedOutputs(rejected), Warnings: cloneWarnings(warnings)} payload := artifactExtractEnvelope{Outputs: artifactCheckpointEnvelopes(outputs), Rejected: cloneRejectedOutputs(rejected), Warnings: cloneWarnings(warnings)}
if err := r.writePayload(lanePayloadPath("extract", laneID, "outputs.json"), payload); err != nil { if err := r.writePayload(lanePayloadPath("extract", stepID, laneID, "outputs.json"), payload); err != nil {
return err return err
} }
manifest := r.laneManifest(StageExtract, statusForRejected(rejected), laneID, moduleKey, dependencies) manifest := r.laneManifest(StageExtract, statusForRejected(rejected), stepID, laneID, moduleKey, dependencies)
manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests(outputs)) manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests(outputs))
manifest.ValidationStatus = validationStatusString(warnings, rejected) manifest.ValidationStatus = validationStatusString(warnings, rejected)
manifest.Rejections = rejectionSummaries(rejected) manifest.Rejections = rejectionSummaries(rejected)
manifest.CompletedAt = timePtr(r.timestamp()) manifest.CompletedAt = timePtr(r.timestamp())
return r.writeManifest(laneManifestPath("extract", laneID), ExtractLaneManifest{StageManifest: manifest, ChunkCount: len(outputs) + len(rejected), OutputCount: len(outputs)}) return r.writeManifest(laneManifestPath("extract", stepID, laneID), ExtractLaneManifest{StageManifest: manifest, ChunkCount: len(outputs) + len(rejected), OutputCount: len(outputs)})
} }
func (r *FilesystemRecorder) ExtractFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error { func (r *FilesystemRecorder) ExtractFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
manifest := r.laneManifest(StageExtract, StatusFailed, laneID, moduleKey, dependencies) return r.ExtractFailedForStep("", laneID, moduleKey, dependencies, err)
}
func (r *FilesystemRecorder) ExtractFailedForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
manifest := r.laneManifest(StageExtract, StatusFailed, stepID, laneID, moduleKey, dependencies)
manifest.CompletedAt = timePtr(r.timestamp()) manifest.CompletedAt = timePtr(r.timestamp())
manifest.Metadata = errorMetadata(err) manifest.Metadata = errorMetadata(err)
return r.writeManifest(laneManifestPath("extract", laneID), ExtractLaneManifest{StageManifest: manifest}) return r.writeManifest(laneManifestPath("extract", stepID, laneID), ExtractLaneManifest{StageManifest: manifest})
} }
func (r *FilesystemRecorder) MergeRunning(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error { func (r *FilesystemRecorder) MergeRunning(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
manifest := r.laneManifest(StageMerge, StatusRunning, laneID, moduleKey, dependencies) return r.MergeRunningForStep("", laneID, moduleKey, dependencies)
}
func (r *FilesystemRecorder) MergeRunningForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
manifest := r.laneManifest(StageMerge, StatusRunning, stepID, laneID, moduleKey, dependencies)
manifest.StartedAt = timePtr(r.timestamp()) manifest.StartedAt = timePtr(r.timestamp())
return r.writeManifest(laneManifestPath("merge", laneID), MergeLaneManifest{StageManifest: manifest}) return r.writeManifest(laneManifestPath("merge", stepID, laneID), MergeLaneManifest{StageManifest: manifest})
} }
func (r *FilesystemRecorder) MergeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error { func (r *FilesystemRecorder) MergeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
if err := r.writePayload(lanePayloadPath("merge", laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil { return r.MergeSucceededForStep("", laneID, moduleKey, dependencies, output, warnings)
}
func (r *FilesystemRecorder) MergeSucceededForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
if err := r.writePayload(lanePayloadPath("merge", stepID, laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
return err return err
} }
manifest := r.laneManifest(StageMerge, StatusSucceeded, laneID, moduleKey, dependencies) manifest := r.laneManifest(StageMerge, StatusSucceeded, stepID, laneID, moduleKey, dependencies)
manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output})) manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
manifest.ValidationStatus = validationStatusString(warnings, nil) manifest.ValidationStatus = validationStatusString(warnings, nil)
manifest.CompletedAt = timePtr(r.timestamp()) manifest.CompletedAt = timePtr(r.timestamp())
return r.writeManifest(laneManifestPath("merge", laneID), MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)}) return r.writeManifest(laneManifestPath("merge", stepID, laneID), MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
} }
func (r *FilesystemRecorder) MergeRejected(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error { func (r *FilesystemRecorder) MergeRejected(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
manifest := r.laneManifest(StageMerge, StatusSucceededWithRejections, laneID, moduleKey, dependencies) return r.MergeRejectedForStep("", laneID, moduleKey, dependencies, rejected)
}
func (r *FilesystemRecorder) MergeRejectedForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
manifest := r.laneManifest(StageMerge, StatusSucceededWithRejections, stepID, laneID, moduleKey, dependencies)
manifest.ValidationStatus = "rejected" manifest.ValidationStatus = "rejected"
manifest.Rejections = rejectionSummaries([]contracts.RejectedOutput{rejected}) manifest.Rejections = rejectionSummaries([]contracts.RejectedOutput{rejected})
manifest.CompletedAt = timePtr(r.timestamp()) manifest.CompletedAt = timePtr(r.timestamp())
return r.writeManifest(laneManifestPath("merge", laneID), MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)}) return r.writeManifest(laneManifestPath("merge", stepID, laneID), MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
} }
func (r *FilesystemRecorder) MergeFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error { func (r *FilesystemRecorder) MergeFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
manifest := r.laneManifest(StageMerge, StatusFailed, laneID, moduleKey, dependencies) return r.MergeFailedForStep("", laneID, moduleKey, dependencies, err)
}
func (r *FilesystemRecorder) MergeFailedForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
manifest := r.laneManifest(StageMerge, StatusFailed, stepID, laneID, moduleKey, dependencies)
manifest.CompletedAt = timePtr(r.timestamp()) manifest.CompletedAt = timePtr(r.timestamp())
manifest.Metadata = errorMetadata(err) manifest.Metadata = errorMetadata(err)
return r.writeManifest(laneManifestPath("merge", laneID), MergeLaneManifest{StageManifest: manifest}) return r.writeManifest(laneManifestPath("merge", stepID, laneID), MergeLaneManifest{StageManifest: manifest})
} }
func (r *FilesystemRecorder) NormalizeRunning(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error { func (r *FilesystemRecorder) NormalizeRunning(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
manifest := r.laneManifest(StageNormalize, StatusRunning, laneID, moduleKey, dependencies) return r.NormalizeRunningForStep("", laneID, moduleKey, dependencies)
}
func (r *FilesystemRecorder) NormalizeRunningForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
manifest := r.laneManifest(StageNormalize, StatusRunning, stepID, laneID, moduleKey, dependencies)
manifest.StartedAt = timePtr(r.timestamp()) manifest.StartedAt = timePtr(r.timestamp())
return r.writeManifest(laneManifestPath("normalize", laneID), NormalizeLaneManifest{StageManifest: manifest}) return r.writeManifest(laneManifestPath("normalize", stepID, laneID), NormalizeLaneManifest{StageManifest: manifest})
} }
func (r *FilesystemRecorder) NormalizeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error { func (r *FilesystemRecorder) NormalizeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
if err := r.writePayload(lanePayloadPath("normalize", laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil { return r.NormalizeSucceededForStep("", laneID, moduleKey, dependencies, output, warnings)
}
func (r *FilesystemRecorder) NormalizeSucceededForStep(stepID, laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
if err := r.writePayload(lanePayloadPath("normalize", stepID, laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
return err return err
} }
manifest := r.laneManifest(StageNormalize, StatusSucceeded, laneID, moduleKey, dependencies) manifest := r.laneManifest(StageNormalize, StatusSucceeded, stepID, laneID, moduleKey, dependencies)
manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output})) manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
manifest.ValidationStatus = validationStatusString(warnings, nil) manifest.ValidationStatus = validationStatusString(warnings, nil)
manifest.CompletedAt = timePtr(r.timestamp()) manifest.CompletedAt = timePtr(r.timestamp())
return r.writeManifest(laneManifestPath("normalize", laneID), NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)}) return r.writeManifest(laneManifestPath("normalize", stepID, laneID), NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
} }
func (r *FilesystemRecorder) NormalizeRejected(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error { func (r *FilesystemRecorder) NormalizeRejected(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
manifest := r.laneManifest(StageNormalize, StatusSucceededWithRejections, laneID, moduleKey, dependencies) return r.NormalizeRejectedForStep("", laneID, moduleKey, dependencies, rejected)
}
func (r *FilesystemRecorder) NormalizeRejectedForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
manifest := r.laneManifest(StageNormalize, StatusSucceededWithRejections, stepID, laneID, moduleKey, dependencies)
manifest.ValidationStatus = "rejected" manifest.ValidationStatus = "rejected"
manifest.Rejections = rejectionSummaries([]contracts.RejectedOutput{rejected}) manifest.Rejections = rejectionSummaries([]contracts.RejectedOutput{rejected})
manifest.CompletedAt = timePtr(r.timestamp()) manifest.CompletedAt = timePtr(r.timestamp())
return r.writeManifest(laneManifestPath("normalize", laneID), NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)}) return r.writeManifest(laneManifestPath("normalize", stepID, laneID), NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
} }
func (r *FilesystemRecorder) NormalizeFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error { func (r *FilesystemRecorder) NormalizeFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
manifest := r.laneManifest(StageNormalize, StatusFailed, laneID, moduleKey, dependencies) return r.NormalizeFailedForStep("", laneID, moduleKey, dependencies, err)
}
func (r *FilesystemRecorder) NormalizeFailedForStep(stepID, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
manifest := r.laneManifest(StageNormalize, StatusFailed, stepID, laneID, moduleKey, dependencies)
manifest.CompletedAt = timePtr(r.timestamp()) manifest.CompletedAt = timePtr(r.timestamp())
manifest.Metadata = errorMetadata(err) manifest.Metadata = errorMetadata(err)
return r.writeManifest(laneManifestPath("normalize", laneID), NormalizeLaneManifest{StageManifest: manifest}) return r.writeManifest(laneManifestPath("normalize", stepID, laneID), NormalizeLaneManifest{StageManifest: manifest})
} }
func (r *FilesystemRecorder) writeManifest(name string, payload any) error { func (r *FilesystemRecorder) writeManifest(name string, payload any) error {
@@ -190,8 +234,9 @@ func (r *FilesystemRecorder) newStageManifest(stage StageName, status StageStatu
return manifest return manifest
} }
func (r *FilesystemRecorder) laneManifest(stage StageName, status StageStatus, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) StageManifest { func (r *FilesystemRecorder) laneManifest(stage StageName, status StageStatus, stepID string, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) StageManifest {
manifest := r.newStageManifest(stage, status) manifest := r.newStageManifest(stage, status)
manifest.StepID = stepID
manifest.LaneID = laneID manifest.LaneID = laneID
manifest.ModuleKey = moduleKey manifest.ModuleKey = moduleKey
manifest.DependencyFingerprints = checkpointFingerprints(dependencies) manifest.DependencyFingerprints = checkpointFingerprints(dependencies)
@@ -424,12 +469,15 @@ func errorMetadata(err error) map[string]string {
return map[string]string{"error": err.Error()} return map[string]string{"error": err.Error()}
} }
func laneManifestPath(stage string, laneID string) string { func laneManifestPath(stage string, stepID string, laneID string) string {
return lanePayloadPath(stage, laneID, "manifest.json") return lanePayloadPath(stage, stepID, laneID, "manifest.json")
} }
func lanePayloadPath(stage string, laneID string, file string) string { func lanePayloadPath(stage string, stepID string, laneID string, file string) string {
return path.Join(stage, checkpointPathComponent(laneID), file) if strings.TrimSpace(stepID) == "" {
return path.Join(stage, checkpointPathComponent(laneID), file)
}
return path.Join(stage, checkpointPathComponent(stepID), checkpointPathComponent(laneID), file)
} }
func checkpointPathComponent(value string) string { func checkpointPathComponent(value string) string {

View File

@@ -35,9 +35,40 @@ func TestRootBasedRecorderOutputIsReusable(t *testing.T) {
} }
} }
func TestCheckpointSchemaCompatibilityIsUnchanged(t *testing.T) { func TestStepAwareRecorderAndLoaderIsolateLaneState(t *testing.T) {
if WorkspaceSchemaVersion != "notarius.workspace.v2" || WorkspaceSchemaVersionV1 != "notarius.workspace.v1" { root := t.TempDir()
t.Fatal("checkpoint schema identifiers changed") identity := testIdentity(t)
recorder, err := NewFilesystemRecorder(root, identity)
if err != nil {
t.Fatal(err)
}
if err := recorder.(pipeline.StepCheckpointRecorder).ExtractSucceededForStep("step-a", "lane", "module", nil, nil, nil, nil); err != nil {
t.Fatal(err)
}
loader, err := NewFilesystemLoader(root, identity)
if err != nil {
t.Fatal(err)
}
stepLoader := loader.(pipeline.StepCheckpointLoader)
if _, decision := stepLoader.ExtractForStep("step-b", "lane", "module", nil); decision.Reused {
t.Fatal("checkpoint from another step was reused")
}
loaded, decision := stepLoader.ExtractForStep("step-a", "lane", "module", nil)
if !decision.Reused || len(loaded.Outputs) != 0 {
t.Fatalf("step-aware load = %#v, decision=%#v", loaded, decision)
}
relative, err := identity.RelativePath()
if err != nil {
t.Fatal(err)
}
if _, err := os.Stat(filepath.Join(root, relative, "extract", "step-a", "lane", "manifest.json")); err != nil {
t.Fatal(err)
}
}
func TestCheckpointSchemaCompatibilityIdentifiers(t *testing.T) {
if WorkspaceSchemaVersion != "notarius.workspace.v3" || WorkspaceSchemaVersionV2 != "notarius.workspace.v2" || WorkspaceSchemaVersionV1 != "notarius.workspace.v1" {
t.Fatal("checkpoint schema identifiers are incorrect")
} }
} }

View File

@@ -30,6 +30,7 @@ type SerializedArtifact struct {
// SerializedOutput associates a domain-neutral artifact with the pipeline // SerializedOutput associates a domain-neutral artifact with the pipeline
// operation that produced it. Provenance remains outside codec-owned bytes. // operation that produced it. Provenance remains outside codec-owned bytes.
type SerializedOutput struct { type SerializedOutput struct {
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id"` LaneID string `json:"lane_id"`
NormalizerKey string `json:"normalizer_key"` NormalizerKey string `json:"normalizer_key"`
SourceID string `json:"source_id,omitempty"` SourceID string `json:"source_id,omitempty"`

View File

@@ -163,12 +163,13 @@ const (
) )
type ReferenceSlot struct { type ReferenceSlot struct {
Name string `json:"name"` Name string `json:"name"`
Description string `json:"description,omitempty"` Description string `json:"description,omitempty"`
Required bool `json:"required,omitempty"` Required bool `json:"required,omitempty"`
AcceptedMediaTypes []string `json:"accepted_media_types,omitempty"` AcceptedMediaTypes []string `json:"accepted_media_types,omitempty"`
Multiple bool `json:"multiple,omitempty"` AcceptedArtifactKinds []ArtifactKind `json:"accepted_artifact_kinds,omitempty"`
MaxBytes int64 `json:"max_bytes,omitempty"` Multiple bool `json:"multiple,omitempty"`
MaxBytes int64 `json:"max_bytes,omitempty"`
} }
func CloneReferenceSlots(slots []ReferenceSlot) []ReferenceSlot { func CloneReferenceSlots(slots []ReferenceSlot) []ReferenceSlot {
@@ -178,6 +179,7 @@ func CloneReferenceSlots(slots []ReferenceSlot) []ReferenceSlot {
out := make([]ReferenceSlot, len(slots)) out := make([]ReferenceSlot, len(slots))
for i, slot := range slots { for i, slot := range slots {
slot.AcceptedMediaTypes = append([]string(nil), slot.AcceptedMediaTypes...) slot.AcceptedMediaTypes = append([]string(nil), slot.AcceptedMediaTypes...)
slot.AcceptedArtifactKinds = append([]ArtifactKind(nil), slot.AcceptedArtifactKinds...)
out[i] = slot out[i] = slot
} }
return out return out
@@ -188,14 +190,64 @@ type ReferenceOrigin struct {
URI string `json:"uri,omitempty"` URI string `json:"uri,omitempty"`
} }
// ReferenceProducer identifies the operation that produced a generated
// reference. It contains provenance only; referenced bytes remain in Content.
type ReferenceProducer struct {
PipelineID string `json:"pipeline_id,omitempty"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"`
}
type ReferenceItem struct { type ReferenceItem struct {
SlotName string `json:"slot_name"` SlotName string `json:"slot_name"`
MediaType string `json:"media_type,omitempty"` MediaType string `json:"media_type,omitempty"`
Content []byte `json:"-"` Content []byte `json:"-"`
Digest string `json:"digest,omitempty"` Digest string `json:"digest,omitempty"`
Origin ReferenceOrigin `json:"origin"` Origin ReferenceOrigin `json:"origin"`
SizeBytes int64 `json:"size_bytes,omitempty"` SizeBytes int64 `json:"size_bytes,omitempty"`
BindingSource string `json:"binding_source,omitempty"` BindingSource string `json:"binding_source,omitempty"`
ArtifactKind ArtifactKind `json:"artifact_kind,omitempty"`
ArtifactSchema ArtifactSchema `json:"artifact_schema,omitempty"`
Producer ReferenceProducer `json:"producer,omitempty"`
}
func CloneReferenceItem(item ReferenceItem) ReferenceItem {
item.Content = append([]byte(nil), item.Content...)
item.ArtifactSchema = CloneArtifactSchema(item.ArtifactSchema)
return item
}
func (item ReferenceItem) MarshalJSON() ([]byte, error) {
type referenceItemJSON struct {
SlotName string `json:"slot_name"`
MediaType string `json:"media_type,omitempty"`
Digest string `json:"digest,omitempty"`
Origin ReferenceOrigin `json:"origin"`
SizeBytes int64 `json:"size_bytes,omitempty"`
BindingSource string `json:"binding_source,omitempty"`
ArtifactKind ArtifactKind `json:"artifact_kind,omitempty"`
ArtifactSchema *ArtifactSchema `json:"artifact_schema,omitempty"`
Producer *ReferenceProducer `json:"producer,omitempty"`
}
encoded := referenceItemJSON{
SlotName: item.SlotName,
MediaType: item.MediaType,
Digest: item.Digest,
Origin: item.Origin,
SizeBytes: item.SizeBytes,
BindingSource: item.BindingSource,
ArtifactKind: item.ArtifactKind,
}
if item.ArtifactSchema.ID != "" || item.ArtifactSchema.Name != "" || item.ArtifactSchema.Version != "" || len(item.ArtifactSchema.JSONSchema) > 0 {
schema := CloneArtifactSchema(item.ArtifactSchema)
encoded.ArtifactSchema = &schema
}
if item.Producer.PipelineID != "" || item.Producer.StepID != "" || item.Producer.LaneID != "" || item.Producer.ModuleKey != "" {
producer := item.Producer
encoded.Producer = &producer
}
return json.Marshal(encoded)
} }
type ResolvedReferenceSlot struct { type ResolvedReferenceSlot struct {
@@ -259,6 +311,7 @@ type OutputEncoder interface {
type RejectedOutput struct { type RejectedOutput struct {
Stage string `json:"stage"` Stage string `json:"stage"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"` LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"` ModuleKey string `json:"module_key,omitempty"`
ChunkID string `json:"chunk_id,omitempty"` ChunkID string `json:"chunk_id,omitempty"`

View File

@@ -70,6 +70,15 @@ func TestCloneReferenceSlotsCopiesAcceptedMediaTypes(t *testing.T) {
} }
} }
func TestCloneReferenceSlotsCopiesAcceptedArtifactKinds(t *testing.T) {
slots := []ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []ArtifactKind{"dnd/npc-list"}}}
clone := CloneReferenceSlots(slots)
clone[0].AcceptedArtifactKinds[0] = "changed"
if slots[0].AcceptedArtifactKinds[0] != "dnd/npc-list" {
t.Fatalf("source AcceptedArtifactKinds aliased clone: %#v", slots[0].AcceptedArtifactKinds)
}
}
func TestReferenceItemJSONOmitsContent(t *testing.T) { func TestReferenceItemJSONOmitsContent(t *testing.T) {
item := ReferenceItem{ item := ReferenceItem{
SlotName: "roster", SlotName: "roster",

View File

@@ -40,17 +40,167 @@ type CheckpointRecorder interface {
NormalizeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error NormalizeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
} }
// StepCheckpointRecorder is implemented by checkpoint stores that isolate
// lane artifacts by their ordered pipeline step. The legacy recorder methods
// remain available for callers that do not have step context.
type StepCheckpointRecorder interface {
ExtractRunningForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
ExtractSucceededForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error
ExtractFailedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
MergeRunningForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
MergeSucceededForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
MergeRejectedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
MergeFailedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
NormalizeRunningForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
NormalizeSucceededForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
NormalizeRejectedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
NormalizeFailedForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
}
type CheckpointDecisionCategory string
const (
CheckpointDecisionExecuted CheckpointDecisionCategory = "executed"
CheckpointDecisionReused CheckpointDecisionCategory = "reused"
CheckpointDecisionForcedRecompute CheckpointDecisionCategory = "forced_recompute"
CheckpointDecisionDependencyInvalidated CheckpointDecisionCategory = "dependency_invalidated"
)
type CheckpointReasonCode string
const (
CheckpointReasonLoadingDisabled CheckpointReasonCode = "loading_disabled"
CheckpointReasonMissing CheckpointReasonCode = "checkpoint_missing"
CheckpointReasonPathInvalid CheckpointReasonCode = "checkpoint_path_invalid"
CheckpointReasonReadFailed CheckpointReasonCode = "checkpoint_read_failed"
CheckpointReasonDecodeFailed CheckpointReasonCode = "checkpoint_decode_failed"
CheckpointReasonWorkspaceSchemaIncompatible CheckpointReasonCode = "workspace_schema_incompatible"
CheckpointReasonIdentityMismatch CheckpointReasonCode = "identity_mismatch"
CheckpointReasonStageMismatch CheckpointReasonCode = "stage_mismatch"
CheckpointReasonStepMismatch CheckpointReasonCode = "step_mismatch"
CheckpointReasonLaneMismatch CheckpointReasonCode = "lane_mismatch"
CheckpointReasonModuleMismatch CheckpointReasonCode = "module_mismatch"
CheckpointReasonStatusNotReusable CheckpointReasonCode = "status_not_reusable"
CheckpointReasonDependencyMismatch CheckpointReasonCode = "dependency_mismatch"
CheckpointReasonArtifactPayloadInvalid CheckpointReasonCode = "artifact_payload_invalid"
CheckpointReasonArtifactDigestMismatch CheckpointReasonCode = "artifact_digest_mismatch"
CheckpointReasonArtifactCodecIncompatible CheckpointReasonCode = "artifact_codec_incompatible"
CheckpointReasonArtifactNotCanonical CheckpointReasonCode = "artifact_not_canonical"
CheckpointReasonReused CheckpointReasonCode = "checkpoint_reused"
CheckpointReasonAcceptedArtifactReused CheckpointReasonCode = "accepted_artifact_reused"
CheckpointReasonRecomputeStep CheckpointReasonCode = "recompute_step"
)
type CheckpointDecision struct { type CheckpointDecision struct {
Reused bool `json:"reused"` Reused bool `json:"reused"`
Category CheckpointDecisionCategory `json:"category,omitempty"`
ReasonCode CheckpointReasonCode `json:"reason_code,omitempty"`
Detail string `json:"detail,omitempty"`
// Reason is retained as a compatibility/debug field for existing callers.
// New checkpoint stores should put bounded, non-sensitive text in Detail.
Reason string `json:"reason,omitempty"` Reason string `json:"reason,omitempty"`
} }
const checkpointDecisionDetailLimit = 512
func NewCheckpointDecision(category CheckpointDecisionCategory, reasonCode CheckpointReasonCode, detail string) CheckpointDecision {
return checkpointDecision(category, reasonCode, detail)
}
func checkpointDecision(category CheckpointDecisionCategory, reasonCode CheckpointReasonCode, detail string) CheckpointDecision {
detail = sanitizeCheckpointDecisionDetail(detail)
return CheckpointDecision{
Reused: category == CheckpointDecisionReused,
Category: category,
ReasonCode: reasonCode,
Detail: detail,
Reason: detail,
}
}
func sanitizeCheckpointDecisionDetail(detail string) string {
detail = strings.TrimSpace(strings.ToValidUTF8(detail, "?"))
lower := strings.ToLower(detail)
if strings.ContainsAny(detail, `/\\`) || strings.Contains(lower, "secret") || strings.Contains(lower, "token") || strings.Contains(lower, "password") || strings.Contains(lower, "credential") || strings.Contains(lower, "environment") {
return "checkpoint decision detail redacted"
}
var b strings.Builder
for _, r := range detail {
if r < 0x20 || r == 0x7f {
r = ' '
}
if b.Len()+len(string(r)) > checkpointDecisionDetailLimit {
break
}
b.WriteRune(r)
}
return strings.Join(strings.Fields(b.String()), " ")
}
type CheckpointEvent struct { type CheckpointEvent struct {
Stage string `json:"stage"` Stage string `json:"stage"`
LaneID string `json:"lane_id,omitempty"` StepID string `json:"step_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"` LaneID string `json:"lane_id,omitempty"`
Action string `json:"action"` ModuleKey string `json:"module_key,omitempty"`
Reason string `json:"reason,omitempty"` Action CheckpointDecisionCategory `json:"action"`
Category CheckpointDecisionCategory `json:"category,omitempty"`
ReasonCode CheckpointReasonCode `json:"reason_code,omitempty"`
Detail string `json:"detail,omitempty"`
Reason string `json:"reason,omitempty"`
}
type CheckpointExecutionPolicy struct {
ForcedLanes map[string]struct{}
RequireReusableLanes map[string]struct{}
}
func CheckpointLaneKey(stepID, laneID string) string {
return strings.TrimSpace(stepID) + "\x00" + strings.TrimSpace(laneID)
}
func (policy CheckpointExecutionPolicy) forced(stepID, laneID string) bool {
_, ok := policy.ForcedLanes[CheckpointLaneKey(stepID, laneID)]
return ok
}
func (policy CheckpointExecutionPolicy) requiresReusable(stepID, laneID string) bool {
_, ok := policy.RequireReusableLanes[CheckpointLaneKey(stepID, laneID)]
return ok
}
func forceCheckpointDecision(policy CheckpointExecutionPolicy, stepID, laneID string, decision CheckpointDecision) CheckpointDecision {
if policy.forced(stepID, laneID) {
return checkpointDecision(CheckpointDecisionForcedRecompute, CheckpointReasonRecomputeStep, "selected step requires execution")
}
return decision
}
func requireReusableCheckpoint(policy CheckpointExecutionPolicy, stepID, laneID string, decision CheckpointDecision) error {
if policy.requiresReusable(stepID, laneID) && !policy.forced(stepID, laneID) && !decision.Reused {
return fmt.Errorf("required reusable checkpoint unavailable for step %q lane %q (%s)", strings.TrimSpace(stepID), strings.TrimSpace(laneID), decision.ReasonCode)
}
return nil
}
// resolveCheckpointDecision applies runner policy and canonical payload
// validation at the single point where a stage's observable decision is made.
func resolveCheckpointDecision(output *RunOutput, loader CheckpointLoader, policy CheckpointExecutionPolicy, stage ModuleStage, stepID, laneID, moduleKey string, decision CheckpointDecision, codec artifactCodecEntry, artifacts []CheckpointArtifact) (CheckpointDecision, error) {
decision = forceCheckpointDecision(policy, stepID, laneID, decision)
if decision.Reused {
for _, artifact := range artifacts {
if _, _, err := decodeCanonicalCheckpointArtifact(codec, artifact); err != nil {
decision = checkpointDecision(CheckpointDecisionExecuted, checkpointArtifactReasonCode(err), "stored "+string(stage)+" artifact failed canonical codec validation")
break
}
}
}
if output != nil {
recordCheckpointEvent(output, loader, string(stage), stepID, laneID, moduleKey, decision)
}
if err := requireReusableCheckpoint(policy, stepID, laneID, decision); err != nil {
return decision, err
}
return decision, nil
} }
type SourceCheckpoint struct { type SourceCheckpoint struct {
@@ -91,6 +241,16 @@ type CheckpointLoader interface {
Extract(laneID string, moduleKey string, dependencies []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) Extract(laneID string, moduleKey string, dependencies []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision)
Merge(laneID string, moduleKey string, dependencies []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) Merge(laneID string, moduleKey string, dependencies []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision)
Normalize(laneID string, moduleKey string, dependencies []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) Normalize(laneID string, moduleKey string, dependencies []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision)
AcceptedNormalize(stepID, laneID, moduleKey string) (NormalizeCheckpoint, CheckpointDecision)
}
// StepCheckpointLoader is the step-aware counterpart used by the persistent
// checkpoint implementation. Loaders without this optional interface remain
// usable by framework callers and test doubles through the legacy methods.
type StepCheckpointLoader interface {
ExtractForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision)
MergeForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision)
NormalizeForStep(stepID, laneID string, moduleKey string, dependencies []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision)
} }
type noopCheckpointRecorder struct{} type noopCheckpointRecorder struct{}
@@ -136,16 +296,86 @@ func (noopCheckpointRecorder) NormalizeFailed(string, string, []CheckpointFinger
func (noopCheckpointLoader) Enabled() bool { return false } func (noopCheckpointLoader) Enabled() bool { return false }
func (noopCheckpointLoader) Source(string) (SourceCheckpoint, CheckpointDecision) { func (noopCheckpointLoader) Source(string) (SourceCheckpoint, CheckpointDecision) {
return SourceCheckpoint{}, CheckpointDecision{Reason: "checkpoint loading disabled"} return SourceCheckpoint{}, checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonLoadingDisabled, "checkpoint loading disabled")
} }
func (noopCheckpointLoader) Extract(string, string, []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) { func (noopCheckpointLoader) Extract(string, string, []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
return ExtractCheckpoint{}, CheckpointDecision{Reason: "checkpoint loading disabled"} return ExtractCheckpoint{}, checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonLoadingDisabled, "checkpoint loading disabled")
} }
func (noopCheckpointLoader) Merge(string, string, []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) { func (noopCheckpointLoader) Merge(string, string, []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
return MergeCheckpoint{}, CheckpointDecision{Reason: "checkpoint loading disabled"} return MergeCheckpoint{}, checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonLoadingDisabled, "checkpoint loading disabled")
} }
func (noopCheckpointLoader) Normalize(string, string, []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) { func (noopCheckpointLoader) Normalize(string, string, []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
return NormalizeCheckpoint{}, CheckpointDecision{Reason: "checkpoint loading disabled"} return NormalizeCheckpoint{}, checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonLoadingDisabled, "checkpoint loading disabled")
}
func (noopCheckpointLoader) AcceptedNormalize(string, string, string) (NormalizeCheckpoint, CheckpointDecision) {
return NormalizeCheckpoint{}, checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonLoadingDisabled, "checkpoint loading disabled")
}
func checkpointExtractRunning(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) error {
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
return stepAware.ExtractRunningForStep(stepID, laneID, moduleKey, deps)
}
return recorder.ExtractRunning(laneID, moduleKey, deps)
}
func checkpointExtractSucceeded(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
return stepAware.ExtractSucceededForStep(stepID, laneID, moduleKey, deps, outputs, rejected, warnings)
}
return recorder.ExtractSucceeded(laneID, moduleKey, deps, outputs, rejected, warnings)
}
func checkpointExtractFailed(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, err error) error {
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
return stepAware.ExtractFailedForStep(stepID, laneID, moduleKey, deps, err)
}
return recorder.ExtractFailed(laneID, moduleKey, deps, err)
}
func checkpointMergeRunning(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) error {
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
return stepAware.MergeRunningForStep(stepID, laneID, moduleKey, deps)
}
return recorder.MergeRunning(laneID, moduleKey, deps)
}
func checkpointMergeSucceeded(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
return stepAware.MergeSucceededForStep(stepID, laneID, moduleKey, deps, output, warnings)
}
return recorder.MergeSucceeded(laneID, moduleKey, deps, output, warnings)
}
func checkpointMergeRejected(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
return stepAware.MergeRejectedForStep(stepID, laneID, moduleKey, deps, rejected)
}
return recorder.MergeRejected(laneID, moduleKey, deps, rejected)
}
func checkpointMergeFailed(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, err error) error {
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
return stepAware.MergeFailedForStep(stepID, laneID, moduleKey, deps, err)
}
return recorder.MergeFailed(laneID, moduleKey, deps, err)
}
func checkpointNormalizeRunning(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) error {
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
return stepAware.NormalizeRunningForStep(stepID, laneID, moduleKey, deps)
}
return recorder.NormalizeRunning(laneID, moduleKey, deps)
}
func checkpointNormalizeSucceeded(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
return stepAware.NormalizeSucceededForStep(stepID, laneID, moduleKey, deps, output, warnings)
}
return recorder.NormalizeSucceeded(laneID, moduleKey, deps, output, warnings)
}
func checkpointNormalizeRejected(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
return stepAware.NormalizeRejectedForStep(stepID, laneID, moduleKey, deps, rejected)
}
return recorder.NormalizeRejected(laneID, moduleKey, deps, rejected)
}
func checkpointNormalizeFailed(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, err error) error {
if stepAware, ok := recorder.(StepCheckpointRecorder); ok {
return stepAware.NormalizeFailedForStep(stepID, laneID, moduleKey, deps, err)
}
return recorder.NormalizeFailed(laneID, moduleKey, deps, err)
} }
func digestFingerprints(name string, digest string) []CheckpointFingerprint { func digestFingerprints(name string, digest string) []CheckpointFingerprint {

View File

@@ -63,6 +63,7 @@ func debugPathComponent(value string) string {
type debugTimedEnvelope struct { type debugTimedEnvelope struct {
Stage string `json:"stage,omitempty"` Stage string `json:"stage,omitempty"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"` LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"` ModuleKey string `json:"module_key,omitempty"`
Attempt int `json:"attempt,omitempty"` Attempt int `json:"attempt,omitempty"`
@@ -124,6 +125,7 @@ type debugChunkRange struct {
} }
type debugSerializedOutput struct { type debugSerializedOutput struct {
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id"` LaneID string `json:"lane_id"`
NormalizerKey string `json:"normalizer_key"` NormalizerKey string `json:"normalizer_key"`
SourceID string `json:"source_id,omitempty"` SourceID string `json:"source_id,omitempty"`
@@ -556,7 +558,7 @@ func debugSerializedOutputEnvelope(output contracts.SerializedOutput) debugSeria
content := debugContentEnvelope(output.Artifact.Content, output.Artifact.MediaType, output.Artifact.Metadata, nil) content := debugContentEnvelope(output.Artifact.Content, output.Artifact.MediaType, output.Artifact.Metadata, nil)
content.ContentDigest = debugContentDigest(output.Artifact.Content) content.ContentDigest = debugContentDigest(output.Artifact.Content)
return debugSerializedOutput{ return debugSerializedOutput{
LaneID: output.LaneID, NormalizerKey: output.NormalizerKey, SourceID: output.SourceID, StepID: output.StepID, LaneID: output.LaneID, NormalizerKey: output.NormalizerKey, SourceID: output.SourceID,
Kind: output.Artifact.Kind, Schema: schema, SchemaDigest: digest, Kind: output.Artifact.Kind, Schema: schema, SchemaDigest: digest,
Content: content, Content: content,
} }

View File

@@ -0,0 +1,287 @@
package pipeline
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"sort"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type referenceTargetKey struct {
StepID string
LaneID string
Stage ModuleStage
}
func keyForReferenceTarget(target ResolvedReferenceTarget) referenceTargetKey {
return referenceTargetKey{StepID: target.StepID, LaneID: target.LaneID, Stage: target.Stage}
}
func operationReferenceSet(input RunInput, target ResolvedReferenceTarget) contracts.ReferenceSet {
if input.references != nil {
if set, ok := input.references[keyForReferenceTarget(target)]; ok {
return CloneReferenceSet(set)
}
}
return CloneReferenceSet(target.ReferenceSet)
}
// buildStepReferenceSets resolves every generated binding for a step before
// any lane in that step is allowed to start. Each returned set is a fresh
// operation-time view; prepared reference sets are never modified.
func buildStepReferenceSets(input RunInput, step PreparedPipelineStep, outputs []contracts.SerializedOutput) (map[referenceTargetKey]contracts.ReferenceSet, []artifacts.ReferenceProvenance, error) {
if len(step.lanes) == 0 {
return nil, nil, nil
}
sets := make(map[referenceTargetKey]contracts.ReferenceSet)
var provenance []artifacts.ReferenceProvenance
canonical := make(map[string]contracts.ReferenceItem)
for _, prepared := range step.lanes {
lane := prepared.resolved
for _, target := range []ResolvedReferenceTarget{lane.ExtractReferences, lane.MergeReferences, lane.NormalizeReferences} {
generated := false
resolved := CloneReferenceSet(target.ReferenceSet)
for _, binding := range target.Bindings {
if binding.Artifact == nil {
continue
}
generated = true
item, err := generatedReferenceItem(input, binding, outputs, canonical)
if err != nil {
return nil, nil, contextualHandoffError(input, target, binding, err)
}
slotName := strings.TrimSpace(binding.SlotName)
slot, ok := resolved.Slots[slotName]
if !ok {
return nil, nil, contextualHandoffError(input, target, binding, fmt.Errorf("reference slot %q has no resolved declaration", slotName))
}
if len(slot.Items) > 0 {
return nil, nil, contextualHandoffError(input, target, binding, fmt.Errorf("reference slot %q already contains materialized items", slotName))
}
if len(slot.Slot.AcceptedArtifactKinds) == 0 || !containsArtifactKind(slot.Slot.AcceptedArtifactKinds, item.ArtifactKind) {
return nil, nil, contextualHandoffError(input, target, binding, fmt.Errorf("reference slot %q does not accept artifact kind %q", slotName, item.ArtifactKind))
}
if !referenceMediaTypeAccepted(item.MediaType, slot.Slot.AcceptedMediaTypes) {
return nil, nil, contextualHandoffError(input, target, binding, fmt.Errorf("reference slot %q does not accept media type %q", slotName, item.MediaType))
}
if slot.Slot.MaxBytes > 0 && item.SizeBytes > slot.Slot.MaxBytes {
return nil, nil, contextualHandoffError(input, target, binding, fmt.Errorf("reference slot %q is %d bytes, limit %d", slotName, item.SizeBytes, slot.Slot.MaxBytes))
}
slot.Items = []contracts.ReferenceItem{contracts.CloneReferenceItem(item)}
resolved.Slots[slotName] = slot
}
if !generated {
continue
}
key := keyForReferenceTarget(target)
sets[key] = resolved
for _, item := range generatedItems(resolved) {
provenance = append(provenance, referenceProvenanceForItem(target, item))
}
}
}
return sets, provenance, nil
}
func containsArtifactKind(kinds []contracts.ArtifactKind, want contracts.ArtifactKind) bool {
for _, kind := range kinds {
if kind == want {
return true
}
}
return false
}
func contextualHandoffError(input RunInput, target ResolvedReferenceTarget, binding ReferenceBinding, err error) error {
return fmt.Errorf("pipeline %q step %q lane %q %s generated dependency %q from %q/%q: %w", input.pipeline.ID, target.StepID, target.LaneID, target.Stage, binding.SlotName, binding.Artifact.Step, binding.Artifact.Lane, err)
}
func generatedReferenceItem(input RunInput, binding ReferenceBinding, outputs []contracts.SerializedOutput, cache map[string]contracts.ReferenceItem) (contracts.ReferenceItem, error) {
selector := binding.Artifact
if selector == nil {
return contracts.ReferenceItem{}, fmt.Errorf("generated reference selector must not be nil")
}
stepID := strings.TrimSpace(selector.Step)
laneID := strings.TrimSpace(selector.Lane)
cacheKey := stepID + "\x00" + laneID
if item, ok := cache[cacheKey]; ok {
item.SlotName = strings.TrimSpace(binding.SlotName)
item.BindingSource = strings.TrimSpace(binding.BindingSource)
return contracts.CloneReferenceItem(item), nil
}
matches := make([]contracts.SerializedOutput, 0, 1)
for _, output := range outputs {
if strings.TrimSpace(output.StepID) == stepID && strings.TrimSpace(output.LaneID) == laneID {
matches = append(matches, output)
}
}
if len(matches) == 0 {
return contracts.ReferenceItem{}, fmt.Errorf("producer has no accepted normalized output")
}
if len(matches) > 1 {
return contracts.ReferenceItem{}, fmt.Errorf("producer has %d accepted normalized outputs; exactly one is required", len(matches))
}
producer, ok := findResolvedLane(input.pipeline, stepID, laneID)
if !ok {
return contracts.ReferenceItem{}, fmt.Errorf("producer lane is not present in the resolved pipeline")
}
if input.Prepared == nil || input.Prepared.artifactCodecs == nil {
return contracts.ReferenceItem{}, fmt.Errorf("artifact codec registry is unavailable")
}
serialized := contracts.CloneSerializedArtifact(matches[0].Artifact)
if serialized.Kind != producer.ArtifactKind {
return contracts.ReferenceItem{}, fmt.Errorf("producer artifact kind %q does not match resolved kind %q", serialized.Kind, producer.ArtifactKind)
}
value, err := input.Prepared.artifactCodecs.Decode(serialized)
if err != nil {
return contracts.ReferenceItem{}, fmt.Errorf("decode producer artifact: %w", err)
}
canonical, err := input.Prepared.artifactCodecs.Encode(producer.ArtifactKind, value)
if err != nil {
return contracts.ReferenceItem{}, fmt.Errorf("canonicalize producer artifact: %w", err)
}
expected, ok := input.Prepared.artifactCodecs.Spec(producer.ArtifactKind)
if !ok {
return contracts.ReferenceItem{}, fmt.Errorf("producer artifact codec %q is not registered", producer.ArtifactKind)
}
if canonical.Kind != expected.Kind || canonical.Schema.ID != expected.Schema.ID || canonical.Schema.Name != expected.Schema.Name || canonical.Schema.Version != expected.Schema.Version || contracts.DigestArtifactSchema(canonical.Schema) != expected.SchemaDigest || canonical.MediaType != expected.MediaType {
return contracts.ReferenceItem{}, fmt.Errorf("canonical producer artifact does not match registered codec identity")
}
item := contracts.ReferenceItem{
SlotName: strings.TrimSpace(binding.SlotName),
MediaType: canonical.MediaType,
Content: append([]byte(nil), canonical.Content...),
Digest: referenceDigest(canonical.Content),
Origin: contracts.ReferenceOrigin{Type: "generated"},
SizeBytes: int64(len(canonical.Content)),
BindingSource: strings.TrimSpace(binding.BindingSource),
ArtifactKind: canonical.Kind,
ArtifactSchema: contracts.CloneArtifactSchema(canonical.Schema),
Producer: contracts.ReferenceProducer{
PipelineID: input.pipeline.ID,
StepID: stepID,
LaneID: laneID,
ModuleKey: producer.Normalize.Module,
},
}
cacheItem := contracts.CloneReferenceItem(item)
cacheItem.SlotName = ""
cacheItem.BindingSource = ""
cache[cacheKey] = cacheItem
return item, nil
}
func findResolvedLane(pipeline ResolvedPipeline, stepID, laneID string) (ResolvedArtifactLane, bool) {
for _, step := range pipeline.Steps {
if step.ID != stepID {
continue
}
for _, lane := range step.ArtifactLanes {
if lane.ID == laneID {
return lane, true
}
}
}
return ResolvedArtifactLane{}, false
}
func generatedItems(set contracts.ReferenceSet) []contracts.ReferenceItem {
var items []contracts.ReferenceItem
keys := make([]string, 0, len(set.Slots))
for key := range set.Slots {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
for _, item := range set.Slots[key].Items {
if item.Origin.Type == "generated" {
items = append(items, item)
}
}
}
return items
}
func referenceProvenanceForItem(target ResolvedReferenceTarget, item contracts.ReferenceItem) artifacts.ReferenceProvenance {
return artifacts.ReferenceProvenance{
Stage: string(target.Stage),
StepID: target.StepID,
LaneID: target.LaneID,
SlotName: item.SlotName,
OriginType: item.Origin.Type,
OriginURI: item.Origin.URI,
Digest: item.Digest,
MediaType: item.MediaType,
SizeBytes: item.SizeBytes,
BindingSource: item.BindingSource,
ArtifactKind: string(item.ArtifactKind),
SchemaID: item.ArtifactSchema.ID,
SchemaName: item.ArtifactSchema.Name,
SchemaVersion: item.ArtifactSchema.Version,
SchemaDigest: contracts.DigestArtifactSchema(item.ArtifactSchema),
ProducerPipeline: item.Producer.PipelineID,
ProducerStep: item.Producer.StepID,
ProducerLane: item.Producer.LaneID,
ProducerModule: item.Producer.ModuleKey,
}
}
type generatedReferenceFingerprintIdentity struct {
ProducerPipeline string `json:"producer_pipeline_id"`
ProducerStep string `json:"producer_step_id"`
ProducerLane string `json:"producer_lane_id"`
ProducerModule string `json:"producer_module_key"`
ArtifactKind string `json:"artifact_kind"`
SchemaID string `json:"schema_id"`
SchemaName string `json:"schema_name"`
SchemaVersion string `json:"schema_version"`
SchemaDigest string `json:"schema_digest"`
MediaType string `json:"media_type"`
ContentDigest string `json:"content_digest"`
}
// generatedReferenceDependencies returns the canonical semantic dependency
// identity that receiving operation checkpoints must include.
func generatedReferenceDependencies(set contracts.ReferenceSet) []CheckpointFingerprint {
var fingerprints []CheckpointFingerprint
keys := make([]string, 0, len(set.Slots))
for key := range set.Slots {
keys = append(keys, key)
}
sort.Strings(keys)
for _, slotName := range keys {
for index, item := range set.Slots[slotName].Items {
if item.Origin.Type != "generated" {
continue
}
identity := generatedReferenceFingerprintIdentity{
ProducerPipeline: item.Producer.PipelineID,
ProducerStep: item.Producer.StepID,
ProducerLane: item.Producer.LaneID,
ProducerModule: item.Producer.ModuleKey,
ArtifactKind: string(item.ArtifactKind),
SchemaID: item.ArtifactSchema.ID,
SchemaName: item.ArtifactSchema.Name,
SchemaVersion: item.ArtifactSchema.Version,
SchemaDigest: contracts.DigestArtifactSchema(item.ArtifactSchema),
MediaType: item.MediaType,
ContentDigest: item.Digest,
}
encoded, err := json.Marshal(identity)
if err != nil {
continue
}
sum := sha256.Sum256(encoded)
fingerprints = append(fingerprints, CheckpointFingerprint{
Name: fmt.Sprintf("generated-reference:%s:%d", slotName, index),
Value: "sha256:" + hex.EncodeToString(sum[:]),
})
}
}
return normalizeCheckpointFingerprints(fingerprints)
}

View File

@@ -0,0 +1,287 @@
package pipeline
import (
"context"
"encoding/json"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func handoffFixture(t *testing.T, value codecNotes) (RunInput, PreparedPipelineStep, contracts.SerializedOutput) {
t.Helper()
prepared := preparedOrderedPipeline(t, 1,
orderedLaneSpec{id: "notes", profile: "notes"},
orderedLaneSpec{id: "score", profile: "score"},
)
consumer := &prepared.Steps[1].lanes[0]
installGeneratedTarget := func(target *ResolvedReferenceTarget, stage ModuleStage, module string) {
*target = ResolvedReferenceTarget{
Stage: stage,
StepID: consumer.resolved.StepID,
LaneID: consumer.resolved.ID,
Module: module,
Bindings: []ReferenceBinding{{
Stage: stage,
LaneID: consumer.resolved.ID,
SlotName: "producer-output",
Artifact: &ArtifactReference{Step: "step-1", Lane: "notes"},
}},
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
"producer-output": {Slot: contracts.ReferenceSlot{
Name: "producer-output",
AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"},
AcceptedMediaTypes: []string{"application/json"},
}},
}},
}
}
installGeneratedTarget(&consumer.resolved.ExtractReferences, StageExtract, consumer.resolved.Extract.Module)
installGeneratedTarget(&consumer.resolved.MergeReferences, StageMerge, consumer.resolved.Merge.Module)
installGeneratedTarget(&consumer.resolved.NormalizeReferences, StageNormalize, consumer.resolved.Normalize.Module)
artifact, err := checkpointArtifact(prepared.Steps[0].lanes[0].typed.codec, "notes", "typed/normalize", "source", value)
if err != nil {
t.Fatalf("checkpointArtifact() error = %v", err)
}
input := RunInput{Prepared: prepared}
input.pipeline = prepared.resolved
return input, prepared.Steps[1], contracts.SerializedOutput{
StepID: "step-1",
LaneID: "notes",
NormalizerKey: "typed/normalize",
SourceID: "source",
Artifact: contracts.CloneSerializedArtifact(artifact.Artifact),
}
}
func TestBuildStepReferenceSetsCanonicalizesAndClonesFanout(t *testing.T) {
input, step, producerOutput := handoffFixture(t, codecNotes{Items: []string{"first"}})
sets, provenance, err := buildStepReferenceSets(input, step, []contracts.SerializedOutput{producerOutput})
if err != nil {
t.Fatalf("buildStepReferenceSets() error = %v", err)
}
if got, want := len(sets), 3; got != want {
t.Fatalf("reference target set count = %d, want %d", got, want)
}
if got, want := len(provenance), 3; got != want {
t.Fatalf("generated provenance count = %d, want %d", got, want)
}
var first []byte
for key, set := range sets {
item := set.Slots["producer-output"].Items[0]
if item.Origin.Type != "generated" || item.Origin.URI != "" {
t.Fatalf("target %v origin = %#v, want generated origin without URI", key, item.Origin)
}
if item.Producer.StepID != "step-1" || item.Producer.LaneID != "notes" || item.Producer.ModuleKey != "typed/normalize" {
t.Fatalf("target %v producer = %#v, want producer provenance", key, item.Producer)
}
if first == nil {
first = item.Content
continue
}
if &first[0] == &item.Content[0] {
t.Fatalf("target %v shares generated content backing storage", key)
}
}
if got := len(step.lanes[0].resolved.ExtractReferences.ReferenceSet.Slots["producer-output"].Items); got != 0 {
t.Fatalf("prepared extract reference items = %d, want zero", got)
}
encoded, err := json.Marshal(provenance)
if err != nil {
t.Fatalf("marshal generated provenance: %v", err)
}
text := string(encoded)
for _, forbidden := range []string{"first", "file://", "content_base64"} {
if strings.Contains(text, forbidden) {
t.Fatalf("generated provenance contains forbidden %q: %s", forbidden, text)
}
}
if !strings.Contains(text, "producer_pipeline_id") || !strings.Contains(text, "schema_digest") {
t.Fatalf("generated provenance = %s, want bounded producer and schema identity", text)
}
}
func TestGeneratedReferenceAcceptsTypedEmptyCollection(t *testing.T) {
input, step, producerOutput := handoffFixture(t, codecNotes{})
sets, _, err := buildStepReferenceSets(input, step, []contracts.SerializedOutput{producerOutput})
if err != nil {
t.Fatalf("buildStepReferenceSets() error = %v, want nil for accepted empty collection", err)
}
if got := len(sets[keyForReferenceTarget(step.lanes[0].resolved.ExtractReferences)].Slots["producer-output"].Items); got != 1 {
t.Fatalf("generated empty collection item count = %d, want one artifact", got)
}
}
func TestGeneratedReferenceRejectsInvalidProducerOutputsDeterministically(t *testing.T) {
tests := []struct {
name string
outputs func(contracts.SerializedOutput) []contracts.SerializedOutput
mutate func(*ResolvedReferenceTarget, *contracts.SerializedOutput)
want string
}{
{name: "missing", outputs: func(contracts.SerializedOutput) []contracts.SerializedOutput { return nil }, want: "no accepted normalized output"},
{name: "multiple", outputs: func(output contracts.SerializedOutput) []contracts.SerializedOutput {
return []contracts.SerializedOutput{output, output}
}, want: "exactly one is required"},
{name: "kind mismatch", outputs: func(output contracts.SerializedOutput) []contracts.SerializedOutput {
return []contracts.SerializedOutput{output}
}, mutate: func(target *ResolvedReferenceTarget, _ *contracts.SerializedOutput) {
target.ReferenceSet.Slots["producer-output"] = contracts.ResolvedReferenceSlot{Slot: contracts.ReferenceSlot{Name: "producer-output", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/score"}}}
}, want: "does not accept artifact kind"},
{name: "schema mismatch", outputs: func(output contracts.SerializedOutput) []contracts.SerializedOutput {
return []contracts.SerializedOutput{output}
}, mutate: func(_ *ResolvedReferenceTarget, output *contracts.SerializedOutput) {
output.Artifact.Schema.ID = "wrong.schema"
}, want: "schema identity"},
{name: "media mismatch", outputs: func(output contracts.SerializedOutput) []contracts.SerializedOutput {
return []contracts.SerializedOutput{output}
}, mutate: func(_ *ResolvedReferenceTarget, output *contracts.SerializedOutput) {
output.Artifact.MediaType = "text/plain"
}, want: "media type"},
{name: "size mismatch", outputs: func(output contracts.SerializedOutput) []contracts.SerializedOutput {
return []contracts.SerializedOutput{output}
}, mutate: func(target *ResolvedReferenceTarget, _ *contracts.SerializedOutput) {
slot := target.ReferenceSet.Slots["producer-output"]
slot.Slot.MaxBytes = 1
target.ReferenceSet.Slots["producer-output"] = slot
}, want: "limit"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
input, step, output := handoffFixture(t, codecNotes{Items: []string{"first"}})
output = contracts.CloneSerializedOutput(output)
if test.mutate != nil {
target := &step.lanes[0].resolved.ExtractReferences
test.mutate(target, &output)
}
_, _, err := buildStepReferenceSets(input, step, test.outputs(output))
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("buildStepReferenceSets() error = %v, want %q", err, test.want)
}
})
}
}
func TestGeneratedReferenceFingerprintChangesWithCanonicalContent(t *testing.T) {
input, step, first := handoffFixture(t, codecNotes{Items: []string{"first"}})
firstSets, _, err := buildStepReferenceSets(input, step, []contracts.SerializedOutput{first})
if err != nil {
t.Fatalf("build first reference set: %v", err)
}
_, _, second := handoffFixture(t, codecNotes{Items: []string{"second"}})
secondSets, _, err := buildStepReferenceSets(input, step, []contracts.SerializedOutput{second})
if err != nil {
t.Fatalf("build second reference set: %v", err)
}
firstDeps := generatedReferenceDependencies(firstSets[keyForReferenceTarget(step.lanes[0].resolved.ExtractReferences)])
secondDeps := generatedReferenceDependencies(secondSets[keyForReferenceTarget(step.lanes[0].resolved.ExtractReferences)])
if reflect.DeepEqual(firstDeps, secondDeps) {
t.Fatalf("generated dependencies = %#v, want content-sensitive fingerprint", firstDeps)
}
}
func TestRunnerHandsOffAcceptedNormalizedOutputBeforeConsumerLanes(t *testing.T) {
input, _, _ := handoffFixture(t, codecNotes{Items: []string{"first"}})
prepared := input.Prepared
var received contracts.ReferenceSet
prepared.Steps[0].lanes[0].typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: codecNotes{Items: []string{"first"}}}, nil
}
prepared.Steps[1].lanes[0].typed.extract = func(_ context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
received = CloneReferenceSet(request.References)
return erasedTypedResult{Value: codecScore{Value: 3}}, nil
}
output, err := New().Run(context.Background(), input)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if got, want := len(output.NormalizeOutputs), 2; got != want {
t.Fatalf("normalized output count = %d, want %d", got, want)
}
if output.NormalizeOutputs[0].StepID != "step-1" || output.NormalizeOutputs[1].StepID != "step-2" {
t.Fatalf("normalized output step IDs = %#v, want step-1 and step-2", output.NormalizeOutputs)
}
item := received.Slots["producer-output"].Items[0]
if item.Origin.Type != "generated" || item.Producer.StepID != "step-1" || item.Producer.LaneID != "notes" {
t.Fatalf("consumer reference = %#v, want generated producer reference", item)
}
if len(output.Manifest.References) == 0 || output.Manifest.References[len(output.Manifest.References)-1].OriginType != "generated" {
t.Fatalf("manifest references = %#v, want generated provenance", output.Manifest.References)
}
}
type handoffDependencyLoader struct {
CheckpointLoader
extract [][]CheckpointFingerprint
merge [][]CheckpointFingerprint
normalize [][]CheckpointFingerprint
}
func (l *handoffDependencyLoader) Enabled() bool { return true }
func (l *handoffDependencyLoader) Extract(_ string, _ string, dependencies []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
l.extract = append(l.extract, append([]CheckpointFingerprint(nil), dependencies...))
return ExtractCheckpoint{}, CheckpointDecision{Reason: "not found"}
}
func (l *handoffDependencyLoader) Merge(_ string, _ string, dependencies []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
l.merge = append(l.merge, append([]CheckpointFingerprint(nil), dependencies...))
return MergeCheckpoint{}, CheckpointDecision{Reason: "not found"}
}
func (l *handoffDependencyLoader) Normalize(_ string, _ string, dependencies []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
l.normalize = append(l.normalize, append([]CheckpointFingerprint(nil), dependencies...))
return NormalizeCheckpoint{}, CheckpointDecision{Reason: "not found"}
}
func runHandoffWithProducerValue(t *testing.T, value codecNotes) *handoffDependencyLoader {
t.Helper()
input, _, _ := handoffFixture(t, value)
prepared := input.Prepared
prepared.Steps[0].lanes[0].typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: value}, nil
}
prepared.Steps[0].lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
return erasedTypedResult{Value: value}, nil
}
prepared.Steps[1].lanes[0].typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: codecScore{Value: 3}}, nil
}
loader := &handoffDependencyLoader{CheckpointLoader: NoopCheckpointLoader()}
if _, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: loader}); err != nil {
t.Fatalf("Run() error = %v", err)
}
return loader
}
func TestRunnerAddsGeneratedDependencyToEveryReceivingCheckpoint(t *testing.T) {
first := runHandoffWithProducerValue(t, codecNotes{Items: []string{"first"}})
second := runHandoffWithProducerValue(t, codecNotes{Items: []string{"second"}})
if len(first.extract) != 2 || len(first.merge) != 2 || len(first.normalize) != 2 {
t.Fatalf("checkpoint load calls = extract %d merge %d normalize %d, want two each", len(first.extract), len(first.merge), len(first.normalize))
}
for name, calls := range map[string][][]CheckpointFingerprint{"extract": first.extract, "merge": first.merge, "normalize": first.normalize} {
if got := generatedFingerprintCount(calls[1]); got != 1 {
t.Fatalf("%s consumer dependency count = %d, want one", name, got)
}
}
if reflect.DeepEqual(first.extract[1], second.extract[1]) || reflect.DeepEqual(first.merge[1], second.merge[1]) || reflect.DeepEqual(first.normalize[1], second.normalize[1]) {
t.Fatalf("consumer checkpoint dependencies did not change with producer content")
}
}
func generatedFingerprintCount(dependencies []CheckpointFingerprint) int {
count := 0
for _, dependency := range dependencies {
if strings.HasPrefix(dependency.Name, "generated-reference:") {
count++
}
}
return count
}

View File

@@ -133,6 +133,7 @@ func normalizeReferenceSlots(slots []contracts.ReferenceSlot) []contracts.Refere
slot.Name = strings.TrimSpace(slot.Name) slot.Name = strings.TrimSpace(slot.Name)
slot.Description = strings.TrimSpace(slot.Description) slot.Description = strings.TrimSpace(slot.Description)
slot.AcceptedMediaTypes = normalizeStringSet(slot.AcceptedMediaTypes) slot.AcceptedMediaTypes = normalizeStringSet(slot.AcceptedMediaTypes)
slot.AcceptedArtifactKinds = normalizeArtifactKinds(slot.AcceptedArtifactKinds)
normalized = append(normalized, slot) normalized = append(normalized, slot)
} }
sort.SliceStable(normalized, func(i, j int) bool { sort.SliceStable(normalized, func(i, j int) bool {
@@ -141,6 +142,34 @@ func normalizeReferenceSlots(slots []contracts.ReferenceSlot) []contracts.Refere
return normalized return normalized
} }
func normalizeArtifactKinds(values []contracts.ArtifactKind) []contracts.ArtifactKind {
if len(values) == 0 {
return nil
}
seen := make(map[contracts.ArtifactKind]struct{}, len(values))
containsEmpty := false
for _, value := range values {
value = normalizeArtifactKind(value)
if value != "" {
seen[value] = struct{}{}
} else {
containsEmpty = true
}
}
if len(seen) == 0 && !containsEmpty {
return nil
}
result := make([]contracts.ArtifactKind, 0, len(seen))
for value := range seen {
result = append(result, value)
}
if containsEmpty {
result = append(result, "")
}
sort.Slice(result, func(i, j int) bool { return result[i] < result[j] })
return result
}
func normalizeStringSet(values []string) []string { func normalizeStringSet(values []string) []string {
if len(values) == 0 { if len(values) == 0 {
return nil return nil
@@ -178,6 +207,11 @@ func validateReferenceSlots(slots []contracts.ReferenceSlot) error {
if slot.MaxBytes < 0 { if slot.MaxBytes < 0 {
return fmt.Errorf("slot %q max_bytes must not be negative", slot.Name) return fmt.Errorf("slot %q max_bytes must not be negative", slot.Name)
} }
for _, kind := range slot.AcceptedArtifactKinds {
if normalizeArtifactKind(kind) == "" {
return fmt.Errorf("slot %q accepted artifact kind must not be empty", slot.Name)
}
}
} }
return nil return nil
} }

View File

@@ -13,7 +13,7 @@ func validateResolvedOptions(resolved ResolvedPipeline, catalog ModuleCatalog) e
return err return err
} }
for _, lane := range resolved.ArtifactLanes { validateLane := func(lane ResolvedArtifactLane) error {
if err := catalog.Extractors.validateOptions(lane.Extract.Module, lane.Extract.Options); err != nil { if err := catalog.Extractors.validateOptions(lane.Extract.Module, lane.Extract.Options); err != nil {
return moduleOptionsError(resolved.ID, lane.ID, StageExtract, lane.Extract.Module, err) return moduleOptionsError(resolved.ID, lane.ID, StageExtract, lane.Extract.Module, err)
} }
@@ -29,8 +29,13 @@ func validateResolvedOptions(resolved ResolvedPipeline, catalog ModuleCatalog) e
if err := catalog.Normalizers.validateOptions(lane.Normalize.Module, lane.ArtifactKind, lane.Normalize.Options); err != nil { if err := catalog.Normalizers.validateOptions(lane.Normalize.Module, lane.ArtifactKind, lane.Normalize.Options); err != nil {
return moduleOptionsError(resolved.ID, lane.ID, StageNormalize, lane.Normalize.Module, err) return moduleOptionsError(resolved.ID, lane.ID, StageNormalize, lane.Normalize.Module, err)
} }
if err := validateChainOptions(resolved, catalog, StageNormalize, lane.ID, lane.Normalize.Module); err != nil { return validateChainOptions(resolved, catalog, StageNormalize, lane.ID, lane.Normalize.Module)
return err }
for _, step := range resolved.Steps {
for _, lane := range step.ArtifactLanes {
if err := validateLane(lane); err != nil {
return err
}
} }
} }

View File

@@ -67,11 +67,71 @@ func TestPrepareConstructsEverythingInStableOrder(t *testing.T) {
if !reflect.DeepEqual(built, want) { if !reflect.DeepEqual(built, want) {
t.Fatalf("construction order = %#v, want %#v", built, want) t.Fatalf("construction order = %#v, want %#v", built, want)
} }
if prepared.Input.Module != "input" || prepared.Chunk.Module != "chunk" || prepared.Output.Module != "output" || len(prepared.ArtifactLanes) != 1 { if prepared.Input.Module != "input" || prepared.Chunk.Module != "chunk" || prepared.Output.Module != "output" || len(prepared.Steps) != 1 || len(prepared.Steps[0].ArtifactLanes) != 1 {
t.Fatalf("PreparedPipeline = %#v, want explicit resolved components", prepared) t.Fatalf("PreparedPipeline = %#v, want explicit resolved components", prepared)
} }
} }
func TestPrepareBuildsEveryOrderedStepBeforeExecution(t *testing.T) {
var built []string
registries, input := constructionRegistries(t, &built, nil)
profile := constructionProfile()
lane := profile.Artifacts["artifact"]
profile.Artifacts = nil
profile.Steps = []PipelineStepProfile{
{ID: "first", Artifacts: map[string]ArtifactLaneProfile{"first-artifact": lane}},
{ID: "second", Artifacts: map[string]ArtifactLaneProfile{"second-artifact": lane}},
}
resolved, err := ResolvePipeline(profile, ResolveOptions{}, registries.catalog())
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
prepared, err := Prepare(resolved, registries, ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
}
want := []string{
"input", "chunk", "validator",
"extract", "validator", "merge", "validator", "normalize", "validator",
"extract", "validator", "merge", "validator", "normalize", "validator",
"output",
}
if !reflect.DeepEqual(built, want) {
t.Fatalf("construction order = %#v, want %#v", built, want)
}
if len(prepared.Steps) != 2 || len(prepared.Steps[0].lanes) != 1 || len(prepared.Steps[1].lanes) != 1 {
t.Fatalf("prepared steps = %#v, want two constructed steps", prepared.Steps)
}
if len(input.requests) != 0 {
t.Fatalf("input Parse calls = %d, want zero during preparation", len(input.requests))
}
}
func TestPrepareRetainsGeneratedSelectorsWithoutReferenceBytes(t *testing.T) {
var built []string
registries, _ := constructionRegistries(t, &built, nil)
resolved, err := ResolvePipeline(constructionProfile(), ResolveOptions{}, registries.catalog())
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings = []ReferenceBinding{{
Stage: StageExtract,
LaneID: "artifact",
SlotName: "generated",
Artifact: &ArtifactReference{Step: "producer", Lane: "artifact"},
}}
prepared, err := Prepare(resolved, registries, ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
}
if len(prepared.Steps[0].ArtifactLanes[0].Resolved.ExtractReferences.ReferenceSet.Slots) != 0 {
t.Fatalf("prepared generated reference set = %#v, want no reference bytes", prepared.Steps[0].ArtifactLanes[0].Resolved.ExtractReferences.ReferenceSet)
}
if len(built) == 0 {
t.Fatal("constructed modules = empty, want preparation to construct the selected components")
}
}
func TestPreparedCheckpointFingerprintsCollectEveryComponentInStableScopeOrder(t *testing.T) { func TestPreparedCheckpointFingerprintsCollectEveryComponentInStableScopeOrder(t *testing.T) {
provider := func(value string) checkpointFingerprintTestProvider { provider := func(value string) checkpointFingerprintTestProvider {
return checkpointFingerprintTestProvider{{Name: "identity", Value: value}} return checkpointFingerprintTestProvider{{Name: "identity", Value: value}}
@@ -90,6 +150,7 @@ func TestPreparedCheckpointFingerprintsCollectEveryComponentInStableScopeOrder(t
chunk: provider("chunk-validator"), chunk: provider("chunk-validator"),
}}}, }}},
output: provider("output"), output: provider("output"),
Steps: []PreparedPipelineStep{{ID: "default"}},
} }
lane := preparedLaneExecutor{ lane := preparedLaneExecutor{
resolved: ResolvedArtifactLane{ resolved: ResolvedArtifactLane{
@@ -107,7 +168,7 @@ func TestPreparedCheckpointFingerprintsCollectEveryComponentInStableScopeOrder(t
mergeValidators: fingerprintTestValidatorChain("merge-validator", provider("merge-validator")), mergeValidators: fingerprintTestValidatorChain("merge-validator", provider("merge-validator")),
normalizeValidators: fingerprintTestValidatorChain("normalize-validator", provider("normalize-validator")), normalizeValidators: fingerprintTestValidatorChain("normalize-validator", provider("normalize-validator")),
} }
prepared.lanes = []preparedLaneExecutor{lane} prepared.Steps[0].lanes = []preparedLaneExecutor{lane}
first, err := collectPreparedCheckpointFingerprints(prepared) first, err := collectPreparedCheckpointFingerprints(prepared)
if err != nil { if err != nil {
@@ -254,9 +315,9 @@ func TestPrepareDeliversTargetReferencesAsIndependentBuildInputs(t *testing.T) {
t.Fatalf("ResolvePipeline() error = %v, want nil", err) t.Fatalf("ResolvePipeline() error = %v, want nil", err)
} }
resolved.ChunkReferences.ReferenceSet = constructionReferenceSet("chunk", "chunk reference") resolved.ChunkReferences.ReferenceSet = constructionReferenceSet("chunk", "chunk reference")
resolved.ArtifactLanes[0].ExtractReferences.ReferenceSet = constructionReferenceSet("extract", "extract reference") resolved.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet = constructionReferenceSet("extract", "extract reference")
resolved.ArtifactLanes[0].MergeReferences.ReferenceSet = constructionReferenceSet("merge", "merge reference") resolved.Steps[0].ArtifactLanes[0].MergeReferences.ReferenceSet = constructionReferenceSet("merge", "merge reference")
resolved.ArtifactLanes[0].NormalizeReferences.ReferenceSet = constructionReferenceSet("normalize", "normalize reference") resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.ReferenceSet = constructionReferenceSet("normalize", "normalize reference")
prepared, err := Prepare(resolved, registries, ModuleDependencies{}) prepared, err := Prepare(resolved, registries, ModuleDependencies{})
if err != nil { if err != nil {
@@ -275,12 +336,12 @@ func TestPrepareDeliversTargetReferencesAsIndependentBuildInputs(t *testing.T) {
t.Errorf("build request %d (%s) reference content = %q, want %q", i, observations[i].Name, got, want) t.Errorf("build request %d (%s) reference content = %q, want %q", i, observations[i].Name, got, want)
} }
} }
if got := constructionReferenceContent(resolved.ArtifactLanes[0].ExtractReferences.ReferenceSet); got != "extract reference" { if got := constructionReferenceContent(resolved.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet); got != "extract reference" {
t.Fatalf("resolved extract references = %q, want original content", got) t.Fatalf("resolved extract references = %q, want original content", got)
} }
_, err = prepared.lanes[0].typed.extract(context.Background(), prepared.lanes[0].typed.extractor, contracts.TypedExtractionRequest{ _, err = prepared.Steps[0].lanes[0].typed.extract(context.Background(), prepared.Steps[0].lanes[0].typed.extractor, contracts.TypedExtractionRequest{
References: CloneReferenceSet(resolved.ArtifactLanes[0].ExtractReferences.ReferenceSet), References: CloneReferenceSet(resolved.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet),
}) })
if err != nil { if err != nil {
t.Fatalf("prepared extractor operation error = %v, want nil", err) t.Fatalf("prepared extractor operation error = %v, want nil", err)

View File

@@ -12,21 +12,27 @@ import (
// resolved pipeline. Its implementation values are private so execution cannot // resolved pipeline. Its implementation values are private so execution cannot
// replace or reconfigure them after preparation. // replace or reconfigure them after preparation.
type PreparedPipeline struct { type PreparedPipeline struct {
Input ModuleBinding Input ModuleBinding
Chunk ModuleBinding Chunk ModuleBinding
ArtifactLanes []PreparedArtifactLane Steps []PreparedPipelineStep
Output ModuleBinding Output ModuleBinding
resolved ResolvedPipeline resolved ResolvedPipeline
dependencies ModuleDependencies dependencies ModuleDependencies
input contracts.InputAdapter input contracts.InputAdapter
chunker contracts.Chunker chunker contracts.Chunker
chunkValidators preparedValidatorChain chunkValidators preparedValidatorChain
lanes []preparedLaneExecutor
output contracts.OutputEncoder output contracts.OutputEncoder
artifactCodecs *ArtifactCodecRegistry
checkpointFingerprints []CheckpointFingerprint checkpointFingerprints []CheckpointFingerprint
} }
type PreparedPipelineStep struct {
ID string
ArtifactLanes []PreparedArtifactLane
lanes []preparedLaneExecutor
}
type PreparedArtifactLane struct { type PreparedArtifactLane struct {
Resolved ResolvedArtifactLane Resolved ResolvedArtifactLane
} }
@@ -73,11 +79,12 @@ func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDepend
} }
stable := cloneResolvedPipeline(resolved) stable := cloneResolvedPipeline(resolved)
prepared := &PreparedPipeline{ prepared := &PreparedPipeline{
Input: cloneModuleBinding(stable.Input), Input: cloneModuleBinding(stable.Input),
Chunk: cloneModuleBinding(stable.Chunk), Chunk: cloneModuleBinding(stable.Chunk),
Output: cloneModuleBinding(stable.Output), Output: cloneModuleBinding(stable.Output),
resolved: stable, resolved: stable,
dependencies: deps, dependencies: deps,
artifactCodecs: registries.ArtifactCodecs,
} }
request := func(binding ModuleBinding, references contracts.ReferenceSet) BuildRequest { request := func(binding ModuleBinding, references contracts.ReferenceSet) BuildRequest {
return BuildRequest{Dependencies: deps, Options: cloneOptions(binding.Options), References: references} return BuildRequest{Dependencies: deps, Options: cloneOptions(binding.Options), References: references}
@@ -99,15 +106,22 @@ func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDepend
return nil, err return nil, err
} }
prepared.ArtifactLanes = make([]PreparedArtifactLane, 0, len(stable.ArtifactLanes)) prepared.Steps = make([]PreparedPipelineStep, len(stable.Steps))
prepared.lanes = make([]preparedLaneExecutor, 0, len(stable.ArtifactLanes)) for stepIndex, step := range stable.Steps {
for _, lane := range stable.ArtifactLanes { preparedStep := PreparedPipelineStep{
executor, err := prepareLane(stable, lane, registries, deps) ID: step.ID,
if err != nil { ArtifactLanes: make([]PreparedArtifactLane, 0, len(step.ArtifactLanes)),
return nil, err lanes: make([]preparedLaneExecutor, 0, len(step.ArtifactLanes)),
} }
prepared.ArtifactLanes = append(prepared.ArtifactLanes, PreparedArtifactLane{Resolved: cloneResolvedArtifactLane(lane)}) for _, lane := range step.ArtifactLanes {
prepared.lanes = append(prepared.lanes, executor) executor, err := prepareLane(stable, lane, registries, deps)
if err != nil {
return nil, err
}
preparedStep.ArtifactLanes = append(preparedStep.ArtifactLanes, PreparedArtifactLane{Resolved: cloneResolvedArtifactLane(lane)})
preparedStep.lanes = append(preparedStep.lanes, executor)
}
prepared.Steps[stepIndex] = preparedStep
} }
output, err := registries.Outputs.BuildWithRequest(stable.Output.Module, request(stable.Output, contracts.ReferenceSet{})) output, err := registries.Outputs.BuildWithRequest(stable.Output.Module, request(stable.Output, contracts.ReferenceSet{}))
@@ -302,6 +316,9 @@ func validateRegistrySet(resolved ResolvedPipeline, registries Registries) error
if registries.Chunkers == nil { if registries.Chunkers == nil {
return fmt.Errorf("chunker registry must not be nil") return fmt.Errorf("chunker registry must not be nil")
} }
if registries.ArtifactCodecs == nil {
return fmt.Errorf("artifact codec registry must not be nil")
}
if registries.Extractors == nil { if registries.Extractors == nil {
return fmt.Errorf("extractor registry must not be nil") return fmt.Errorf("extractor registry must not be nil")
} }
@@ -331,10 +348,16 @@ func cloneResolvedPipeline(in ResolvedPipeline) ResolvedPipeline {
out.Output = cloneModuleBinding(in.Output) out.Output = cloneModuleBinding(in.Output)
out.ChunkReferences = CloneReferenceTarget(in.ChunkReferences) out.ChunkReferences = CloneReferenceTarget(in.ChunkReferences)
out.ValidatorChains = cloneResolvedValidatorChains(in.ValidatorChains) out.ValidatorChains = cloneResolvedValidatorChains(in.ValidatorChains)
if len(in.ArtifactLanes) > 0 { if len(in.Steps) > 0 {
out.ArtifactLanes = make([]ResolvedArtifactLane, len(in.ArtifactLanes)) out.Steps = make([]ResolvedPipelineStep, len(in.Steps))
for i, lane := range in.ArtifactLanes { for i, step := range in.Steps {
out.ArtifactLanes[i] = cloneResolvedArtifactLane(lane) out.Steps[i] = ResolvedPipelineStep{ID: step.ID}
if len(step.ArtifactLanes) > 0 {
out.Steps[i].ArtifactLanes = make([]ResolvedArtifactLane, len(step.ArtifactLanes))
for j, lane := range step.ArtifactLanes {
out.Steps[i].ArtifactLanes[j] = cloneResolvedArtifactLane(lane)
}
}
} }
} }
return out return out

View File

@@ -26,16 +26,18 @@ func collectPreparedCheckpointFingerprints(prepared *PreparedPipeline) ([]Checkp
{scope: "chunk:" + prepared.resolved.Chunk.Module, module: prepared.chunker}, {scope: "chunk:" + prepared.resolved.Chunk.Module, module: prepared.chunker},
} }
components = appendValidatorFingerprintComponents(components, "chunk:"+prepared.resolved.Chunk.Module, prepared.chunkValidators) components = appendValidatorFingerprintComponents(components, "chunk:"+prepared.resolved.Chunk.Module, prepared.chunkValidators)
for _, lane := range prepared.lanes { for _, step := range prepared.Steps {
laneScope := func(stage ModuleStage, moduleKey string) string { for _, lane := range step.lanes {
return string(stage) + ":" + lane.resolved.ID + ":" + moduleKey laneScope := func(stage ModuleStage, moduleKey string) string {
return string(stage) + ":" + lane.resolved.ID + ":" + moduleKey
}
components = append(components, checkpointFingerprintComponent{scope: laneScope(StageExtract, lane.resolved.Extract.Module), module: lane.typed.extractor})
components = appendValidatorFingerprintComponents(components, laneScope(StageExtract, lane.resolved.Extract.Module), lane.extractValidators)
components = append(components, checkpointFingerprintComponent{scope: laneScope(StageMerge, lane.resolved.Merge.Module), module: lane.typed.merger})
components = appendValidatorFingerprintComponents(components, laneScope(StageMerge, lane.resolved.Merge.Module), lane.mergeValidators)
components = append(components, checkpointFingerprintComponent{scope: laneScope(StageNormalize, lane.resolved.Normalize.Module), module: lane.typed.normalizer})
components = appendValidatorFingerprintComponents(components, laneScope(StageNormalize, lane.resolved.Normalize.Module), lane.normalizeValidators)
} }
components = append(components, checkpointFingerprintComponent{scope: laneScope(StageExtract, lane.resolved.Extract.Module), module: lane.typed.extractor})
components = appendValidatorFingerprintComponents(components, laneScope(StageExtract, lane.resolved.Extract.Module), lane.extractValidators)
components = append(components, checkpointFingerprintComponent{scope: laneScope(StageMerge, lane.resolved.Merge.Module), module: lane.typed.merger})
components = appendValidatorFingerprintComponents(components, laneScope(StageMerge, lane.resolved.Merge.Module), lane.mergeValidators)
components = append(components, checkpointFingerprintComponent{scope: laneScope(StageNormalize, lane.resolved.Normalize.Module), module: lane.typed.normalizer})
components = appendValidatorFingerprintComponents(components, laneScope(StageNormalize, lane.resolved.Normalize.Module), lane.normalizeValidators)
} }
components = append(components, checkpointFingerprintComponent{scope: "output:" + prepared.resolved.Output.Module, module: prepared.output}) components = append(components, checkpointFingerprintComponent{scope: "output:" + prepared.resolved.Output.Module, module: prepared.output})

View File

@@ -21,12 +21,56 @@ const (
) )
type ModuleBinding struct { type ModuleBinding struct {
Module string `json:"module"` Module string `json:"module"`
LLMProfile string `json:"llm_profile,omitempty"` LLMProfile string `json:"llm_profile,omitempty"`
Retries int `json:"retries,omitempty"` Retries int `json:"retries,omitempty"`
Options map[string]any `json:"options,omitempty"` Options map[string]any `json:"options,omitempty"`
References map[string]string `json:"references,omitempty"` References map[string]ReferenceSource `json:"references,omitempty"`
Validators ValidatorOverride `json:"validators,omitempty"` Validators ValidatorOverride `json:"validators,omitempty"`
}
// ArtifactReference identifies a normalized artifact produced by an earlier
// ordered step. It is a selector only; it does not contain artifact bytes.
type ArtifactReference struct {
Step string `json:"step"`
Lane string `json:"lane"`
}
// ReferenceSource is the discriminated source form used by configured
// reference maps. Exactly one of Path and Artifact is set after resolution.
type ReferenceSource struct {
Path string `json:"path,omitempty"`
Artifact *ArtifactReference `json:"artifact,omitempty"`
}
func ExternalReference(path string) ReferenceSource {
return ReferenceSource{Path: strings.TrimSpace(path)}
}
func ExternalReferenceMap(values map[string]string) map[string]ReferenceSource {
out := make(map[string]ReferenceSource, len(values))
for key, value := range values {
out[key] = ExternalReference(value)
}
return out
}
func GeneratedReference(step, lane string) ReferenceSource {
return ReferenceSource{Artifact: &ArtifactReference{Step: strings.TrimSpace(step), Lane: strings.TrimSpace(lane)}}
}
func (source ReferenceSource) IsGenerated() bool { return source.Artifact != nil }
func (source ReferenceSource) MarshalJSON() ([]byte, error) {
if source.Path != "" && source.Artifact != nil {
return nil, fmt.Errorf("reference source must not contain both an external path and an artifact selector")
}
if source.Artifact != nil {
return json.Marshal(struct {
Artifact *ArtifactReference `json:"artifact"`
}{Artifact: source.Artifact})
}
return json.Marshal(source.Path)
} }
type ValidatorOverride struct { type ValidatorOverride struct {
@@ -36,12 +80,12 @@ type ValidatorOverride struct {
func (binding ModuleBinding) MarshalJSON() ([]byte, error) { func (binding ModuleBinding) MarshalJSON() ([]byte, error) {
type moduleBindingJSON struct { type moduleBindingJSON struct {
Module string `json:"module"` Module string `json:"module"`
LLMProfile string `json:"llm_profile,omitempty"` LLMProfile string `json:"llm_profile,omitempty"`
Retries int `json:"retries,omitempty"` Retries int `json:"retries,omitempty"`
Options map[string]any `json:"options,omitempty"` Options map[string]any `json:"options,omitempty"`
References map[string]string `json:"references,omitempty"` References map[string]ReferenceSource `json:"references,omitempty"`
Validators *[]ModuleBinding `json:"validators,omitempty"` Validators *[]ModuleBinding `json:"validators,omitempty"`
} }
out := moduleBindingJSON{ out := moduleBindingJSON{
Module: binding.Module, Module: binding.Module,
@@ -58,11 +102,17 @@ func (binding ModuleBinding) MarshalJSON() ([]byte, error) {
} }
type ArtifactLaneProfile struct { type ArtifactLaneProfile struct {
Extract ModuleBinding `json:"extract"` Extract ModuleBinding `json:"extract"`
Merge ModuleBinding `json:"merge,omitempty"` Merge ModuleBinding `json:"merge,omitempty"`
Normalize ModuleBinding `json:"normalize,omitempty"` Normalize ModuleBinding `json:"normalize,omitempty"`
Validators []ModuleBinding `json:"validators,omitempty"` Validators []ModuleBinding `json:"validators,omitempty"`
References map[string]string `json:"references,omitempty"` References map[string]ReferenceSource `json:"references,omitempty"`
}
type PipelineStepProfile struct {
ID string `json:"id"`
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
References map[string]ReferenceSource `json:"references,omitempty"`
} }
type PipelineProfile struct { type PipelineProfile struct {
@@ -70,8 +120,9 @@ type PipelineProfile struct {
Input ModuleBinding `json:"input"` Input ModuleBinding `json:"input"`
Chunk ModuleBinding `json:"chunk,omitempty"` Chunk ModuleBinding `json:"chunk,omitempty"`
Artifacts map[string]ArtifactLaneProfile `json:"artifacts"` Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
Steps []PipelineStepProfile `json:"steps,omitempty"`
Output ModuleBinding `json:"output,omitempty"` Output ModuleBinding `json:"output,omitempty"`
References map[string]string `json:"references,omitempty"` References map[string]ReferenceSource `json:"references,omitempty"`
} }
type ResolveOptions struct { type ResolveOptions struct {
@@ -81,11 +132,12 @@ type ResolveOptions struct {
} }
type ReferenceBinding struct { type ReferenceBinding struct {
Stage ModuleStage `json:"stage,omitempty"` Stage ModuleStage `json:"stage,omitempty"`
LaneID string `json:"lane_id,omitempty"` LaneID string `json:"lane_id,omitempty"`
SlotName string `json:"slot_name"` SlotName string `json:"slot_name"`
Source string `json:"source"` Source string `json:"source,omitempty"`
BindingSource string `json:"binding_source,omitempty"` Artifact *ArtifactReference `json:"artifact,omitempty"`
BindingSource string `json:"binding_source,omitempty"`
} }
type ReferenceUnbind struct { type ReferenceUnbind struct {
@@ -96,6 +148,7 @@ type ReferenceUnbind struct {
type ResolvedReferenceTarget struct { type ResolvedReferenceTarget struct {
Stage ModuleStage `json:"stage"` Stage ModuleStage `json:"stage"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"` LaneID string `json:"lane_id,omitempty"`
Module string `json:"module"` Module string `json:"module"`
Bindings []ReferenceBinding `json:"bindings,omitempty"` Bindings []ReferenceBinding `json:"bindings,omitempty"`
@@ -103,6 +156,7 @@ type ResolvedReferenceTarget struct {
} }
type ResolvedArtifactLane struct { type ResolvedArtifactLane struct {
StepID string
ID string ID string
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"` ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"` ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
@@ -118,6 +172,11 @@ type ResolvedArtifactLane struct {
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"` NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
} }
type ResolvedPipelineStep struct {
ID string `json:"id"`
ArtifactLanes []ResolvedArtifactLane `json:"artifact_lanes"`
}
type ResolvedValidatorChain struct { type ResolvedValidatorChain struct {
Stage ModuleStage `json:"stage"` Stage ModuleStage `json:"stage"`
LaneID string `json:"lane_id,omitempty"` LaneID string `json:"lane_id,omitempty"`
@@ -138,11 +197,21 @@ type ResolvedPipeline struct {
Input ModuleBinding Input ModuleBinding
Chunk ModuleBinding Chunk ModuleBinding
ChunkReferences ResolvedReferenceTarget `json:"chunk_references"` ChunkReferences ResolvedReferenceTarget `json:"chunk_references"`
ArtifactLanes []ResolvedArtifactLane Steps []ResolvedPipelineStep
ValidatorChains []ResolvedValidatorChain `json:"validator_chains"` ValidatorChains []ResolvedValidatorChain `json:"validator_chains"`
Output ModuleBinding Output ModuleBinding
} }
// AllArtifactLanes returns lanes in deterministic step order for read-only
// discovery and identity construction.
func (resolved ResolvedPipeline) AllArtifactLanes() []ResolvedArtifactLane {
var lanes []ResolvedArtifactLane
for _, step := range resolved.Steps {
lanes = append(lanes, step.ArtifactLanes...)
}
return lanes
}
type ModuleCatalog struct { type ModuleCatalog struct {
Inputs *InputAdapterRegistry Inputs *InputAdapterRegistry
Chunkers *ChunkerRegistry Chunkers *ChunkerRegistry
@@ -165,7 +234,21 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
return ResolvedPipeline{}, fmt.Errorf("pipeline id must not be empty") return ResolvedPipeline{}, fmt.Errorf("pipeline id must not be empty")
} }
if len(profile.Artifacts) == 0 { explicitSteps := profile.Steps != nil
if len(profile.Artifacts) > 0 && explicitSteps {
return ResolvedPipeline{}, fmt.Errorf("pipeline %q must not declare both artifacts and steps", pipelineID)
}
if explicitSteps && len(options.Only) > 0 {
return ResolvedPipeline{}, fmt.Errorf("pipeline %q --only is not supported for explicit ordered steps", pipelineID)
}
steps := profile.Steps
if !explicitSteps {
steps = []PipelineStepProfile{{ID: "default", Artifacts: profile.Artifacts}}
}
if explicitSteps && len(steps) == 0 {
return ResolvedPipeline{}, fmt.Errorf("pipeline %q must declare at least one ordered step", pipelineID)
}
if len(steps) == 0 {
return ResolvedPipeline{}, fmt.Errorf("pipeline %q must declare at least one artifact lane", pipelineID) return ResolvedPipeline{}, fmt.Errorf("pipeline %q must declare at least one artifact lane", pipelineID)
} }
@@ -194,15 +277,25 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
} }
capabilities.add(chunkSpec.Provides...) capabilities.add(chunkSpec.Provides...)
lanesByID, selectedLaneIDs, err := selectedArtifactLanes(pipelineID, profile.Artifacts, options) if !explicitSteps {
if err != nil { if err := validatePipelineReferenceDefaults(pipelineID, profile.References, chunkSpec, profile.Artifacts, catalog); err != nil {
return ResolvedPipeline{}, err return ResolvedPipeline{}, err
}
} else {
allLanes := make(map[string]ArtifactLaneProfile)
for _, step := range profile.Steps {
for laneID, lane := range step.Artifacts {
allLanes[strings.TrimSpace(laneID)] = lane
}
}
if err := validatePipelineReferenceDefaults(pipelineID, profile.References, chunkSpec, allLanes, catalog); err != nil {
return ResolvedPipeline{}, err
}
} }
if len(selectedLaneIDs) == 0 { for _, source := range profile.References {
return ResolvedPipeline{}, fmt.Errorf("pipeline %q must select at least one artifact lane", pipelineID) if source.Artifact != nil {
} return ResolvedPipeline{}, fmt.Errorf("pipeline %q pipeline-level references may not use generated artifact selectors", pipelineID)
if err := validatePipelineReferenceDefaults(pipelineID, profile.References, chunkSpec, lanesByID, catalog); err != nil { }
return ResolvedPipeline{}, err
} }
chunkReferences, err := resolveReferenceTargetBindings(referenceResolutionTarget{ chunkReferences, err := resolveReferenceTargetBindings(referenceResolutionTarget{
@@ -230,16 +323,55 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
} }
resolved.ValidatorChains = append(resolved.ValidatorChains, chunkValidatorChain) resolved.ValidatorChains = append(resolved.ValidatorChains, chunkValidatorChain)
outputCapabilities := capabilities.clone() outputCapabilities := capabilities.clone()
seenResolvedLanes := make(map[string]string)
seenResolvedSteps := make(map[string]struct{}, len(steps))
for _, laneID := range selectedLaneIDs { for _, step := range steps {
laneProfile := lanesByID[laneID] stepID := strings.TrimSpace(step.ID)
lane, validatorChains, laneCapabilities, err := resolveArtifactLane(pipelineID, laneID, laneProfile, profile.References, options, capabilities, catalog) if stepID == "" {
return ResolvedPipeline{}, fmt.Errorf("pipeline %q step id must not be empty", pipelineID)
}
if _, exists := seenResolvedSteps[stepID]; exists {
return ResolvedPipeline{}, fmt.Errorf("pipeline %q step id %q is duplicated after trimming", pipelineID, stepID)
}
seenResolvedSteps[stepID] = struct{}{}
laneOptions := ResolveOptions{}
if !explicitSteps {
laneOptions = options
}
lanesByID, selectedLaneIDs, err := selectedArtifactLanes(pipelineID, step.Artifacts, laneOptions)
if err != nil { if err != nil {
return ResolvedPipeline{}, err return ResolvedPipeline{}, err
} }
resolved.ArtifactLanes = append(resolved.ArtifactLanes, lane) if len(selectedLaneIDs) == 0 {
resolved.ValidatorChains = append(resolved.ValidatorChains, validatorChains...) return ResolvedPipeline{}, fmt.Errorf("pipeline %q step %q must declare at least one artifact lane", pipelineID, stepID)
outputCapabilities.addSet(laneCapabilities) }
if referenceSourceMapsConflict(profile.References, step.References) {
return ResolvedPipeline{}, fmt.Errorf("pipeline %q step %q has a generated and external reference collision", pipelineID, stepID)
}
if err := validatePipelineReferenceDefaults(pipelineID, step.References, chunkSpec, lanesByID, catalog); err != nil {
return ResolvedPipeline{}, err
}
stepReferences := mergeReferenceMaps(profile.References, step.References)
resolvedStep := ResolvedPipelineStep{ID: stepID}
for _, laneID := range selectedLaneIDs {
if previousStep, exists := seenResolvedLanes[laneID]; exists {
return ResolvedPipeline{}, fmt.Errorf("pipeline %q artifact lane id %q is duplicated across steps %q and %q", pipelineID, laneID, previousStep, stepID)
}
seenResolvedLanes[laneID] = stepID
laneProfile := lanesByID[laneID]
lane, validatorChains, laneCapabilities, err := resolveArtifactLane(pipelineID, stepID, laneID, laneProfile, stepReferences, options, capabilities, catalog)
if err != nil {
return ResolvedPipeline{}, err
}
resolvedStep.ArtifactLanes = append(resolvedStep.ArtifactLanes, lane)
resolved.ValidatorChains = append(resolved.ValidatorChains, validatorChains...)
outputCapabilities.addSet(laneCapabilities)
}
resolved.Steps = append(resolved.Steps, resolvedStep)
}
if err := validateGeneratedBindings(pipelineID, resolved, catalog); err != nil {
return ResolvedPipeline{}, err
} }
outputSpec, err := outputSpec(catalog, resolved.Output.Module) outputSpec, err := outputSpec(catalog, resolved.Output.Module)
@@ -263,14 +395,16 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
func resolveArtifactLane( func resolveArtifactLane(
pipelineID string, pipelineID string,
stepID string,
laneID string, laneID string,
profile ArtifactLaneProfile, profile ArtifactLaneProfile,
pipelineReferences map[string]string, pipelineReferences map[string]ReferenceSource,
options ResolveOptions, options ResolveOptions,
inherited capabilitySet, inherited capabilitySet,
catalog ModuleCatalog, catalog ModuleCatalog,
) (ResolvedArtifactLane, []ResolvedValidatorChain, capabilitySet, error) { ) (ResolvedArtifactLane, []ResolvedValidatorChain, capabilitySet, error) {
lane := ResolvedArtifactLane{ lane := ResolvedArtifactLane{
StepID: strings.TrimSpace(stepID),
ID: laneID, ID: laneID,
Extract: resolveBinding(profile.Extract, ""), Extract: resolveBinding(profile.Extract, ""),
Merge: resolveBinding(profile.Merge, DefaultMergeModule), Merge: resolveBinding(profile.Merge, DefaultMergeModule),
@@ -294,6 +428,9 @@ func resolveArtifactLane(
if err != nil { if err != nil {
return ResolvedArtifactLane{}, nil, nil, err return ResolvedArtifactLane{}, nil, nil, err
} }
if referenceSourceMapsConflict(profile.References, lane.Extract.References) {
return ResolvedArtifactLane{}, nil, nil, fmt.Errorf("pipeline %q step %q lane %q extract has a generated and external reference collision", pipelineID, stepID, laneID)
}
extractReferences := mergeReferenceMaps(profile.References, lane.Extract.References) extractReferences := mergeReferenceMaps(profile.References, lane.Extract.References)
references, err := resolveReferenceTargetBindings(referenceResolutionTarget{ references, err := resolveReferenceTargetBindings(referenceResolutionTarget{
PipelineID: pipelineID, PipelineID: pipelineID,
@@ -309,6 +446,7 @@ func resolveArtifactLane(
return ResolvedArtifactLane{}, nil, nil, err return ResolvedArtifactLane{}, nil, nil, err
} }
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references) lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
lane.ExtractReferences.StepID = strings.TrimSpace(stepID)
capabilities.add(extractSpec.Provides...) capabilities.add(extractSpec.Provides...)
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType) mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
@@ -332,6 +470,7 @@ func resolveArtifactLane(
return ResolvedArtifactLane{}, nil, nil, err return ResolvedArtifactLane{}, nil, nil, err
} }
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences) lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
lane.MergeReferences.StepID = strings.TrimSpace(stepID)
capabilities.add(mergeSpec.Provides...) capabilities.add(mergeSpec.Provides...)
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType) normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
@@ -355,6 +494,7 @@ func resolveArtifactLane(
return ResolvedArtifactLane{}, nil, nil, err return ResolvedArtifactLane{}, nil, nil, err
} }
lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences) lane.NormalizeReferences = referenceTarget(StageNormalize, laneID, lane.Normalize.Module, normalizeReferences)
lane.NormalizeReferences.StepID = strings.TrimSpace(stepID)
capabilities.add(normalizeSpec.Provides...) capabilities.add(normalizeSpec.Provides...)
if len(lane.Validators) > 0 { if len(lane.Validators) > 0 {
@@ -382,6 +522,83 @@ func configuredValidatorsError(pipelineID string, laneID string) error {
return fmt.Errorf("pipeline %q lane %q validators are not supported at artifact lane level; use extract.validators, merge.validators, or normalize.validators", pipelineID, laneID) return fmt.Errorf("pipeline %q lane %q validators are not supported at artifact lane level; use extract.validators, merge.validators, or normalize.validators", pipelineID, laneID)
} }
func validateGeneratedBindings(pipelineID string, resolved ResolvedPipeline, catalog ModuleCatalog) error {
stepIndex := make(map[string]int, len(resolved.Steps))
laneIndex := make(map[string]ResolvedArtifactLane)
for index, step := range resolved.Steps {
if _, exists := stepIndex[step.ID]; exists {
return fmt.Errorf("pipeline %q step id %q is ambiguous", pipelineID, step.ID)
}
stepIndex[step.ID] = index
for _, lane := range step.ArtifactLanes {
laneIndex[step.ID+"\x00"+lane.ID] = lane
}
}
for consumerIndex, step := range resolved.Steps {
for _, lane := range step.ArtifactLanes {
for _, target := range []ResolvedReferenceTarget{lane.ExtractReferences, lane.MergeReferences, lane.NormalizeReferences} {
slots, err := resolvedReferenceSlots(target, lane.ArtifactKind, catalog)
if err != nil {
return fmt.Errorf("pipeline %q step %q lane %q: %w", pipelineID, step.ID, lane.ID, err)
}
slotByName := make(map[string]contracts.ReferenceSlot, len(slots))
for _, slot := range slots {
slotByName[slot.Name] = slot
}
for _, binding := range target.Bindings {
if binding.Artifact == nil {
continue
}
producerStep, ok := stepIndex[strings.TrimSpace(binding.Artifact.Step)]
if !ok {
return fmt.Errorf("pipeline %q step %q lane %q %s reference slot %q producer step %q is not declared", pipelineID, step.ID, lane.ID, target.Stage, binding.SlotName, binding.Artifact.Step)
}
if producerStep >= consumerIndex {
return fmt.Errorf("pipeline %q step %q lane %q reference slot %q producer %q must be an earlier step", pipelineID, step.ID, lane.ID, binding.SlotName, binding.Artifact.Step)
}
producer, ok := laneIndex[strings.TrimSpace(binding.Artifact.Step)+"\x00"+strings.TrimSpace(binding.Artifact.Lane)]
if !ok {
return fmt.Errorf("pipeline %q step %q lane %q reference slot %q producer lane %q is not selected", pipelineID, step.ID, lane.ID, binding.SlotName, binding.Artifact.Lane)
}
slot, ok := slotByName[strings.TrimSpace(binding.SlotName)]
if !ok {
return fmt.Errorf("pipeline %q step %q lane %q %s reference slot %q is not declared", pipelineID, step.ID, lane.ID, target.Stage, binding.SlotName)
}
if len(slot.AcceptedArtifactKinds) == 0 {
return fmt.Errorf("pipeline %q step %q lane %q reference slot %q does not accept generated artifacts", pipelineID, step.ID, lane.ID, binding.SlotName)
}
accepted := false
for _, kind := range slot.AcceptedArtifactKinds {
if kind == producer.ArtifactKind {
accepted = true
break
}
}
if !accepted {
return fmt.Errorf("pipeline %q step %q lane %q reference slot %q does not accept artifact kind %q", pipelineID, step.ID, lane.ID, binding.SlotName, producer.ArtifactKind)
}
codecSpec, ok := catalog.ArtifactCodecs.Spec(producer.ArtifactKind)
if !ok {
return fmt.Errorf("pipeline %q step %q lane %q producer %q artifact codec %q is not registered", pipelineID, step.ID, lane.ID, producer.ID, producer.ArtifactKind)
}
if !referenceMediaTypeAccepted(codecSpec.MediaType, slot.AcceptedMediaTypes) {
return fmt.Errorf("pipeline %q step %q lane %q reference slot %q does not accept producer media type %q", pipelineID, step.ID, lane.ID, binding.SlotName, codecSpec.MediaType)
}
}
}
}
}
return nil
}
func resolvedReferenceSlots(target ResolvedReferenceTarget, artifactKind contracts.ArtifactKind, catalog ModuleCatalog) ([]contracts.ReferenceSlot, error) {
spec, err := referenceTargetSpec(target, artifactKind, catalog)
if err != nil {
return nil, err
}
return contracts.CloneReferenceSlots(spec.ReferenceSlots), nil
}
func resolveArtifactIdentity(pipelineID, laneID string, lane *ResolvedArtifactLane, extractSpec ModuleSpec, catalog ModuleCatalog) (reflect.Type, error) { func resolveArtifactIdentity(pipelineID, laneID string, lane *ResolvedArtifactLane, extractSpec ModuleSpec, catalog ModuleCatalog) (reflect.Type, error) {
if extractSpec.ArtifactKind == "" { if extractSpec.ArtifactKind == "" {
return nil, fmt.Errorf("pipeline %q lane %q extract module %q does not declare an artifact kind", pipelineID, laneID, lane.Extract.Module) return nil, fmt.Errorf("pipeline %q lane %q extract module %q does not declare an artifact kind", pipelineID, laneID, lane.Extract.Module)
@@ -593,23 +810,35 @@ func referenceTarget(stage ModuleStage, laneID string, module string, bindings [
} }
} }
func mergeReferenceMaps(base map[string]string, override map[string]string) map[string]string { func mergeReferenceMaps(base map[string]ReferenceSource, override map[string]ReferenceSource) map[string]ReferenceSource {
if len(base) == 0 && len(override) == 0 { if len(base) == 0 && len(override) == 0 {
return nil return nil
} }
out := make(map[string]string, len(base)+len(override)) out := make(map[string]ReferenceSource, len(base)+len(override))
for key, value := range base { for key, value := range base {
out[key] = value out[key] = cloneReferenceSource(value)
} }
for key, value := range override { for key, value := range override {
out[key] = value out[key] = cloneReferenceSource(value)
} }
return out return out
} }
func referenceSourceMapsConflict(base, override map[string]ReferenceSource) bool {
for rawKey, left := range base {
key := strings.TrimSpace(rawKey)
for rawOverrideKey, right := range override {
if key == strings.TrimSpace(rawOverrideKey) && left.IsGenerated() != right.IsGenerated() {
return true
}
}
}
return false
}
func validatePipelineReferenceDefaults( func validatePipelineReferenceDefaults(
pipelineID string, pipelineID string,
pipelineReferences map[string]string, pipelineReferences map[string]ReferenceSource,
chunkSpec ModuleSpec, chunkSpec ModuleSpec,
lanesByID map[string]ArtifactLaneProfile, lanesByID map[string]ArtifactLaneProfile,
catalog ModuleCatalog, catalog ModuleCatalog,
@@ -680,8 +909,8 @@ type referenceResolutionTarget struct {
Stage ModuleStage Stage ModuleStage
Module string Module string
Slots []contracts.ReferenceSlot Slots []contracts.ReferenceSlot
PipelineReferences map[string]string PipelineReferences map[string]ReferenceSource
LocalReferences map[string]string LocalReferences map[string]ReferenceSource
Options ResolveOptions Options ResolveOptions
} }
@@ -692,24 +921,46 @@ func resolveReferenceTargetBindings(target referenceResolutionTarget) ([]Referen
} }
bindings := make(map[string]ReferenceBinding) bindings := make(map[string]ReferenceBinding)
addBinding := func(slotName, source, bindingSource string) error { addBinding := func(slotName string, source ReferenceSource, bindingSource string) error {
slotName = strings.TrimSpace(slotName) slotName = strings.TrimSpace(slotName)
source = strings.TrimSpace(source)
if slotName == "" { if slotName == "" {
return fmt.Errorf("%s reference slot name must not be empty", referenceTargetErrorContext(target)) return fmt.Errorf("%s reference slot name must not be empty", referenceTargetErrorContext(target))
} }
if source == "" { if source.Artifact == nil {
source.Path = strings.TrimSpace(source.Path)
}
if source.Artifact == nil && source.Path == "" {
return fmt.Errorf("%s reference slot %q source must not be empty", referenceTargetErrorContext(target), slotName) return fmt.Errorf("%s reference slot %q source must not be empty", referenceTargetErrorContext(target), slotName)
} }
if source.Artifact != nil {
if strings.TrimSpace(source.Path) != "" {
return fmt.Errorf("%s reference slot %q must contain exactly one source form", referenceTargetErrorContext(target), slotName)
}
artifact := *source.Artifact
artifact.Step = strings.TrimSpace(artifact.Step)
artifact.Lane = strings.TrimSpace(artifact.Lane)
if artifact.Step == "" || artifact.Lane == "" {
return fmt.Errorf("%s reference slot %q artifact selector step and lane must not be empty", referenceTargetErrorContext(target), slotName)
}
source.Artifact = &artifact
}
if _, ok := slotByName[slotName]; !ok { if _, ok := slotByName[slotName]; !ok {
return fmt.Errorf("%s reference slot %q is not declared by %s module %q", referenceTargetErrorContext(target), slotName, target.Stage, target.Module) return fmt.Errorf("%s reference slot %q is not declared by %s module %q", referenceTargetErrorContext(target), slotName, target.Stage, target.Module)
} }
bindings[slotName] = ReferenceBinding{ if previous, exists := bindings[slotName]; exists && (previous.Artifact != nil || source.Artifact != nil) {
return fmt.Errorf("%s reference slot %q has conflicting generated and external bindings", referenceTargetErrorContext(target), slotName)
}
binding := ReferenceBinding{
LaneID: target.LaneID, LaneID: target.LaneID,
SlotName: slotName, SlotName: slotName,
Source: source,
BindingSource: bindingSource, BindingSource: bindingSource,
} }
if source.Artifact != nil {
binding.Artifact = source.Artifact
} else {
binding.Source = source.Path
}
bindings[slotName] = binding
return nil return nil
} }
@@ -748,7 +999,7 @@ func resolveReferenceTargetBindings(target referenceResolutionTarget) ([]Referen
if strings.TrimSpace(source) == "" { if strings.TrimSpace(source) == "" {
source = contracts.ReferenceBindingSourceCLI source = contracts.ReferenceBindingSourceCLI
} }
if err := addBinding(override.SlotName, override.Source, strings.TrimSpace(source)); err != nil { if err := addBinding(override.SlotName, ExternalReference(override.Source), strings.TrimSpace(source)); err != nil {
return nil, err return nil, err
} }
} }
@@ -768,7 +1019,9 @@ func resolveReferenceTargetBindings(target referenceResolutionTarget) ([]Referen
if _, ok := slotByName[slotName]; !ok { if _, ok := slotByName[slotName]; !ok {
return nil, fmt.Errorf("%s reference slot %q is not declared", referenceTargetErrorContext(target), slotName) return nil, fmt.Errorf("%s reference slot %q is not declared", referenceTargetErrorContext(target), slotName)
} }
delete(bindings, slotName) if binding, ok := bindings[slotName]; ok && binding.Artifact == nil {
delete(bindings, slotName)
}
} }
for _, slot := range target.Slots { for _, slot := range target.Slots {
@@ -843,11 +1096,11 @@ func referenceTargetSlotLabel(target referenceResolutionTarget) string {
return fmt.Sprintf("pipeline %q %s reference slot", target.PipelineID, target.Stage) return fmt.Sprintf("pipeline %q %s reference slot", target.PipelineID, target.Stage)
} }
func normalizedReferenceMap(values map[string]string, keyName string) (map[string]string, error) { func normalizedReferenceMap(values map[string]ReferenceSource, keyName string) (map[string]ReferenceSource, error) {
if len(values) == 0 { if len(values) == 0 {
return nil, nil return nil, nil
} }
out := make(map[string]string, len(values)) out := make(map[string]ReferenceSource, len(values))
for rawSlotName, rawSource := range values { for rawSlotName, rawSource := range values {
slotName := strings.TrimSpace(rawSlotName) slotName := strings.TrimSpace(rawSlotName)
if slotName == "" { if slotName == "" {
@@ -856,8 +1109,8 @@ func normalizedReferenceMap(values map[string]string, keyName string) (map[strin
if _, ok := out[slotName]; ok { if _, ok := out[slotName]; ok {
return nil, fmt.Errorf("%s %q is duplicated after trimming", keyName, slotName) return nil, fmt.Errorf("%s %q is duplicated after trimming", keyName, slotName)
} }
source := strings.TrimSpace(rawSource) source := cloneReferenceSource(rawSource)
if source == "" { if source.Artifact == nil && strings.TrimSpace(source.Path) == "" {
return nil, fmt.Errorf("%s %q source must not be empty", keyName, slotName) return nil, fmt.Errorf("%s %q source must not be empty", keyName, slotName)
} }
out[slotName] = source out[slotName] = source
@@ -877,7 +1130,7 @@ func sortedArtifactLaneProfileKeys(values map[string]ArtifactLaneProfile) []stri
return keys return keys
} }
func sortedStringMapKeys(values map[string]string) []string { func sortedStringMapKeys(values map[string]ReferenceSource) []string {
if len(values) == 0 { if len(values) == 0 {
return nil return nil
} }
@@ -942,11 +1195,11 @@ func cloneOptions(options map[string]any) map[string]any {
return copied return copied
} }
func normalizeReferenceMap(values map[string]string) map[string]string { func normalizeReferenceMap(values map[string]ReferenceSource) map[string]ReferenceSource {
if len(values) == 0 { if len(values) == 0 {
return nil return nil
} }
out := make(map[string]string, len(values)) out := make(map[string]ReferenceSource, len(values))
keys := make([]string, 0, len(values)) keys := make([]string, 0, len(values))
rawByNormalized := make(map[string]string, len(values)) rawByNormalized := make(map[string]string, len(values))
for rawKey := range values { for rawKey := range values {
@@ -956,7 +1209,7 @@ func normalizeReferenceMap(values map[string]string) map[string]string {
} }
sort.Strings(keys) sort.Strings(keys)
for _, key := range keys { for _, key := range keys {
out[key] = strings.TrimSpace(values[rawByNormalized[key]]) out[key] = cloneReferenceSource(values[rawByNormalized[key]])
} }
return out return out
} }
@@ -1009,7 +1262,7 @@ func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
Input ModuleBinding Input ModuleBinding
Chunk ModuleBinding Chunk ModuleBinding
ChunkReferences ResolvedReferenceTarget ChunkReferences ResolvedReferenceTarget
ArtifactLanes []ResolvedArtifactLane Steps []ResolvedPipelineStep
ValidatorChains []ResolvedValidatorChain ValidatorChains []ResolvedValidatorChain
Output ModuleBinding Output ModuleBinding
}{ }{
@@ -1017,7 +1270,7 @@ func resolvedPipelineDigest(resolved ResolvedPipeline) (string, error) {
Input: resolved.Input, Input: resolved.Input,
Chunk: resolved.Chunk, Chunk: resolved.Chunk,
ChunkReferences: resolved.ChunkReferences, ChunkReferences: resolved.ChunkReferences,
ArtifactLanes: resolved.ArtifactLanes, Steps: resolved.Steps,
ValidatorChains: resolved.ValidatorChains, ValidatorChains: resolved.ValidatorChains,
Output: resolved.Output, Output: resolved.Output,
} }

View File

@@ -53,10 +53,10 @@ func TestResolvePipelineWithExplicitModules(t *testing.T) {
if resolved.Chunk.Options["size"] != 10 { if resolved.Chunk.Options["size"] != 10 {
t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options) t.Fatalf("Chunk.Options = %#v, want size option", resolved.Chunk.Options)
} }
if len(resolved.ArtifactLanes) != 1 { if len(resolved.Steps) != 1 || resolved.Steps[0].ID != "default" || len(resolved.Steps[0].ArtifactLanes) != 1 {
t.Fatalf("len(ArtifactLanes) = %d, want 1", len(resolved.ArtifactLanes)) t.Fatalf("resolved steps = %#v, want one default step with one lane", resolved.Steps)
} }
lane := resolved.ArtifactLanes[0] lane := resolved.Steps[0].ArtifactLanes[0]
if lane.ID != "records" { if lane.ID != "records" {
t.Fatalf("lane.ID = %q, want records", lane.ID) t.Fatalf("lane.ID = %q, want records", lane.ID)
} }
@@ -98,7 +98,7 @@ func TestResolvePipelineAppliesDefaults(t *testing.T) {
if !reflect.DeepEqual(resolved.Output, ModuleBinding{Module: DefaultOutputModule}) { if !reflect.DeepEqual(resolved.Output, ModuleBinding{Module: DefaultOutputModule}) {
t.Fatalf("Output = %#v, want default output binding", resolved.Output) t.Fatalf("Output = %#v, want default output binding", resolved.Output)
} }
lane := resolved.ArtifactLanes[0] lane := resolved.Steps[0].ArtifactLanes[0]
if !reflect.DeepEqual(lane.Merge, ModuleBinding{Module: DefaultMergeModule}) { if !reflect.DeepEqual(lane.Merge, ModuleBinding{Module: DefaultMergeModule}) {
t.Fatalf("Merge = %#v, want default merge binding", lane.Merge) t.Fatalf("Merge = %#v, want default merge binding", lane.Merge)
} }
@@ -326,23 +326,77 @@ func TestResolvePipelineSelectsOnlyRequestedLanes(t *testing.T) {
t.Fatalf("ResolvePipeline() error = %v, want nil", err) t.Fatalf("ResolvePipeline() error = %v, want nil", err)
} }
got := laneIDs(resolved.ArtifactLanes) got := laneIDs(resolved.Steps[0].ArtifactLanes)
want := []string{"events", "summaries"} want := []string{"events", "summaries"}
if !reflect.DeepEqual(got, want) { if !reflect.DeepEqual(got, want) {
t.Fatalf("lane IDs = %#v, want %#v", got, want) t.Fatalf("lane IDs = %#v, want %#v", got, want)
} }
} }
func TestResolvePipelinePreservesOrderedStepsAndExpandsGeneratedBindings(t *testing.T) {
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{
Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes",
Requires: []string{"chunk"}, Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
})
resolved, err := ResolvePipeline(PipelineProfile{
ID: "ordered",
Input: Binding("text"),
Steps: []PipelineStepProfile{
{ID: "produce", Artifacts: map[string]ArtifactLaneProfile{"npcs": {Extract: Binding("event-extractor")}}},
{ID: "consume", References: map[string]ReferenceSource{"npcs": GeneratedReference("produce", "npcs")}, Artifacts: map[string]ArtifactLaneProfile{"events": {Extract: Binding("note-extractor")}}},
},
}, ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if got := []string{resolved.Steps[0].ID, resolved.Steps[1].ID}; !reflect.DeepEqual(got, []string{"produce", "consume"}) {
t.Fatalf("step order = %#v", got)
}
binding := resolved.Steps[1].ArtifactLanes[0].ExtractReferences.Bindings[0]
if binding.Artifact == nil || binding.Artifact.Step != "produce" || binding.Artifact.Lane != "npcs" {
t.Fatalf("generated binding = %#v", binding)
}
if len(resolved.AllArtifactLanes()) != 2 || resolved.Steps[1].ArtifactLanes[0].StepID != "consume" {
t.Fatalf("resolved lane topology = %#v", resolved.Steps)
}
}
func TestResolvePipelineRejectsGeneratedBindingOrderingAndKind(t *testing.T) {
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "npcs", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/other"}}}},
)
_, err := ResolvePipeline(PipelineProfile{
ID: "invalid-order", Input: Binding("text"), Steps: []PipelineStepProfile{
{ID: "first", References: map[string]ReferenceSource{"npcs": GeneratedReference("later", "npcs")}, Artifacts: map[string]ArtifactLaneProfile{"events": {Extract: Binding("note-extractor")}}},
{ID: "later", Artifacts: map[string]ArtifactLaneProfile{"npcs": {Extract: Binding("event-extractor")}}},
},
}, ResolveOptions{}, catalog)
if err == nil || !strings.Contains(err.Error(), "earlier step") {
t.Fatalf("ResolvePipeline() error = %v, want ordering failure", err)
}
_, err = ResolvePipeline(PipelineProfile{
ID: "invalid-kind", Input: Binding("text"), Steps: []PipelineStepProfile{
{ID: "produce", Artifacts: map[string]ArtifactLaneProfile{"npcs": {Extract: Binding("event-extractor")}}},
{ID: "consume", References: map[string]ReferenceSource{"npcs": GeneratedReference("produce", "npcs")}, Artifacts: map[string]ArtifactLaneProfile{"events": {Extract: Binding("note-extractor")}}},
},
}, ResolveOptions{}, catalog)
if err == nil || !strings.Contains(err.Error(), "does not accept artifact kind") {
t.Fatalf("ResolvePipeline() error = %v, want generated kind failure", err)
}
}
func TestResolvePipelineAppliesReferenceBindings(t *testing.T) { func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
profile := multiLaneProfile() profile := multiLaneProfile()
profile.References = map[string]string{ profile.References = ExternalReferenceMap(map[string]string{
" roster ": " ./shared-roster.yml ", " roster ": " ./shared-roster.yml ",
} })
lane := profile.Artifacts["events"] lane := profile.Artifacts["events"]
lane.References = map[string]string{ lane.References = ExternalReferenceMap(map[string]string{
"roster": "./lane-roster.yml", "roster": "./lane-roster.yml",
" lore ": " ./lore.md ", " lore ": " ./lore.md ",
} })
profile.Artifacts["events"] = lane profile.Artifacts["events"] = lane
catalog := newProfileCatalogWithOverride(t, ModuleSpec{ catalog := newProfileCatalogWithOverride(t, ModuleSpec{
@@ -360,7 +414,7 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
t.Fatalf("ResolvePipeline() error = %v, want nil", err) t.Fatalf("ResolvePipeline() error = %v, want nil", err)
} }
events := resolvedLane(t, resolved.ArtifactLanes, "events") events := resolvedLane(t, resolved.Steps[0].ArtifactLanes, "events")
if events.ExtractReferences.Stage != StageExtract || events.ExtractReferences.LaneID != "events" || events.ExtractReferences.Module != "event-extractor" { if events.ExtractReferences.Stage != StageExtract || events.ExtractReferences.LaneID != "events" || events.ExtractReferences.Module != "event-extractor" {
t.Fatalf("extract reference target = %#v, want event extractor target", events.ExtractReferences) t.Fatalf("extract reference target = %#v, want event extractor target", events.ExtractReferences)
} }
@@ -380,7 +434,7 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
if !reflect.DeepEqual(events.ExtractReferences.Bindings, want) { if !reflect.DeepEqual(events.ExtractReferences.Bindings, want) {
t.Fatalf("events references = %#v, want %#v", events.ExtractReferences.Bindings, want) t.Fatalf("events references = %#v, want %#v", events.ExtractReferences.Bindings, want)
} }
summaries := resolvedLane(t, resolved.ArtifactLanes, "summaries") summaries := resolvedLane(t, resolved.Steps[0].ArtifactLanes, "summaries")
if len(summaries.ExtractReferences.Bindings) != 0 { if len(summaries.ExtractReferences.Bindings) != 0 {
t.Fatalf("summaries references = %#v, want none", summaries.ExtractReferences.Bindings) t.Fatalf("summaries references = %#v, want none", summaries.ExtractReferences.Bindings)
} }
@@ -388,7 +442,7 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
func TestResolvePipelineAppliesPipelineReferenceDefaultToChunkTarget(t *testing.T) { func TestResolvePipelineAppliesPipelineReferenceDefaultToChunkTarget(t *testing.T) {
profile := baselineProfile() profile := baselineProfile()
profile.References = map[string]string{"scene_guide": "./scenes.md"} profile.References = ExternalReferenceMap(map[string]string{"scene_guide": "./scenes.md"})
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "generic", Key: "generic",
Stage: StageChunk, Stage: StageChunk,
@@ -406,17 +460,17 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToChunkTarget(t *testing.
if !reflect.DeepEqual(resolved.ChunkReferences.Bindings, want) { if !reflect.DeepEqual(resolved.ChunkReferences.Bindings, want) {
t.Fatalf("chunk references = %#v, want %#v", resolved.ChunkReferences.Bindings, want) t.Fatalf("chunk references = %#v, want %#v", resolved.ChunkReferences.Bindings, want)
} }
if refs := resolved.ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 { if refs := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
t.Fatalf("extract references = %#v, want none", refs) t.Fatalf("extract references = %#v, want none", refs)
} }
if refs := resolved.ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 { if refs := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
t.Fatalf("normalize references = %#v, want none", refs) t.Fatalf("normalize references = %#v, want none", refs)
} }
} }
func TestResolvePipelineAppliesPipelineReferenceDefaultToExtractorTarget(t *testing.T) { func TestResolvePipelineAppliesPipelineReferenceDefaultToExtractorTarget(t *testing.T) {
profile := baselineProfile() profile := baselineProfile()
profile.References = map[string]string{"roster": "./roster.yml"} profile.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "event-extractor", Key: "event-extractor",
Stage: StageExtract, Stage: StageExtract,
@@ -431,20 +485,20 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToExtractorTarget(t *test
} }
want := []ReferenceBinding{{LaneID: "events", SlotName: "roster", Source: "./roster.yml", BindingSource: contracts.ReferenceBindingSourceConfig}} want := []ReferenceBinding{{LaneID: "events", SlotName: "roster", Source: "./roster.yml", BindingSource: contracts.ReferenceBindingSourceConfig}}
if !reflect.DeepEqual(resolved.ArtifactLanes[0].ExtractReferences.Bindings, want) { if !reflect.DeepEqual(resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings, want) {
t.Fatalf("extract references = %#v, want %#v", resolved.ArtifactLanes[0].ExtractReferences.Bindings, want) t.Fatalf("extract references = %#v, want %#v", resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings, want)
} }
if refs := resolved.ChunkReferences.Bindings; len(refs) != 0 { if refs := resolved.ChunkReferences.Bindings; len(refs) != 0 {
t.Fatalf("chunk references = %#v, want none", refs) t.Fatalf("chunk references = %#v, want none", refs)
} }
if refs := resolved.ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 { if refs := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
t.Fatalf("normalize references = %#v, want none", refs) t.Fatalf("normalize references = %#v, want none", refs)
} }
} }
func TestResolvePipelineAppliesPipelineReferenceDefaultToNormalizerTarget(t *testing.T) { func TestResolvePipelineAppliesPipelineReferenceDefaultToNormalizerTarget(t *testing.T) {
profile := baselineProfile() profile := baselineProfile()
profile.References = map[string]string{"normalization_notes": "./normalize.md"} profile.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./normalize.md"})
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "noop", Key: "noop",
Stage: StageNormalize, Stage: StageNormalize,
@@ -459,20 +513,20 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToNormalizerTarget(t *tes
} }
want := []ReferenceBinding{{LaneID: "events", SlotName: "normalization_notes", Source: "./normalize.md", BindingSource: contracts.ReferenceBindingSourceConfig}} want := []ReferenceBinding{{LaneID: "events", SlotName: "normalization_notes", Source: "./normalize.md", BindingSource: contracts.ReferenceBindingSourceConfig}}
if !reflect.DeepEqual(resolved.ArtifactLanes[0].NormalizeReferences.Bindings, want) { if !reflect.DeepEqual(resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings, want) {
t.Fatalf("normalize references = %#v, want %#v", resolved.ArtifactLanes[0].NormalizeReferences.Bindings, want) t.Fatalf("normalize references = %#v, want %#v", resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings, want)
} }
if refs := resolved.ChunkReferences.Bindings; len(refs) != 0 { if refs := resolved.ChunkReferences.Bindings; len(refs) != 0 {
t.Fatalf("chunk references = %#v, want none", refs) t.Fatalf("chunk references = %#v, want none", refs)
} }
if refs := resolved.ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 { if refs := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
t.Fatalf("extract references = %#v, want none", refs) t.Fatalf("extract references = %#v, want none", refs)
} }
} }
func TestResolvePipelineAppliesPipelineReferenceDefaultToMergeTarget(t *testing.T) { func TestResolvePipelineAppliesPipelineReferenceDefaultToMergeTarget(t *testing.T) {
profile := baselineProfile() profile := baselineProfile()
profile.References = map[string]string{"merge_notes": "./merge.md"} profile.References = ExternalReferenceMap(map[string]string{"merge_notes": "./merge.md"})
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "appendorder", Key: "appendorder",
Stage: StageMerge, Stage: StageMerge,
@@ -487,23 +541,23 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToMergeTarget(t *testing.
} }
want := []ReferenceBinding{{LaneID: "events", SlotName: "merge_notes", Source: "./merge.md", BindingSource: contracts.ReferenceBindingSourceConfig}} want := []ReferenceBinding{{LaneID: "events", SlotName: "merge_notes", Source: "./merge.md", BindingSource: contracts.ReferenceBindingSourceConfig}}
if !reflect.DeepEqual(resolved.ArtifactLanes[0].MergeReferences.Bindings, want) { if !reflect.DeepEqual(resolved.Steps[0].ArtifactLanes[0].MergeReferences.Bindings, want) {
t.Fatalf("merge references = %#v, want %#v", resolved.ArtifactLanes[0].MergeReferences.Bindings, want) t.Fatalf("merge references = %#v, want %#v", resolved.Steps[0].ArtifactLanes[0].MergeReferences.Bindings, want)
} }
if refs := resolved.ChunkReferences.Bindings; len(refs) != 0 { if refs := resolved.ChunkReferences.Bindings; len(refs) != 0 {
t.Fatalf("chunk references = %#v, want none", refs) t.Fatalf("chunk references = %#v, want none", refs)
} }
if refs := resolved.ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 { if refs := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
t.Fatalf("extract references = %#v, want none", refs) t.Fatalf("extract references = %#v, want none", refs)
} }
if refs := resolved.ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 { if refs := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
t.Fatalf("normalize references = %#v, want none", refs) t.Fatalf("normalize references = %#v, want none", refs)
} }
} }
func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *testing.T) { func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *testing.T) {
profile := baselineProfile() profile := baselineProfile()
profile.References = map[string]string{"context": "./context.md"} profile.References = ExternalReferenceMap(map[string]string{"context": "./context.md"})
catalog := newProfileCatalogWithOverrides(t, catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}}, ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}}, ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
@@ -517,14 +571,14 @@ func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *t
} }
assertBindingSource(t, resolved.ChunkReferences.Bindings, "context", "./context.md") assertBindingSource(t, resolved.ChunkReferences.Bindings, "context", "./context.md")
assertBindingSource(t, resolved.ArtifactLanes[0].ExtractReferences.Bindings, "context", "./context.md") assertBindingSource(t, resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings, "context", "./context.md")
assertBindingSource(t, resolved.ArtifactLanes[0].MergeReferences.Bindings, "context", "./context.md") assertBindingSource(t, resolved.Steps[0].ArtifactLanes[0].MergeReferences.Bindings, "context", "./context.md")
assertBindingSource(t, resolved.ArtifactLanes[0].NormalizeReferences.Bindings, "context", "./context.md") assertBindingSource(t, resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings, "context", "./context.md")
} }
func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedLane(t *testing.T) { func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedLane(t *testing.T) {
profile := multiLaneProfile() profile := multiLaneProfile()
profile.References = map[string]string{"notes_context": "./notes.md"} profile.References = ExternalReferenceMap(map[string]string{"notes_context": "./notes.md"})
catalog := newProfileCatalogWithOverride(t, ModuleSpec{ catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "note-extractor", Key: "note-extractor",
Stage: StageExtract, Stage: StageExtract,
@@ -539,14 +593,14 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedLane(t *t
if err != nil { if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err) t.Fatalf("ResolvePipeline() error = %v, want nil", err)
} }
if refs := resolved.ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 { if refs := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
t.Fatalf("selected lane references = %#v, want none", refs) t.Fatalf("selected lane references = %#v, want none", refs)
} }
} }
func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedNormalizer(t *testing.T) { func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedNormalizer(t *testing.T) {
profile := multiLaneProfile() profile := multiLaneProfile()
profile.References = map[string]string{"notes_context": "./notes.md"} profile.References = ExternalReferenceMap(map[string]string{"notes_context": "./notes.md"})
lane := profile.Artifacts["notes"] lane := profile.Artifacts["notes"]
lane.Normalize = Binding("note-normalizer") lane.Normalize = Binding("note-normalizer")
profile.Artifacts["notes"] = lane profile.Artifacts["notes"] = lane
@@ -564,17 +618,17 @@ func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedNormalize
if err != nil { if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err) t.Fatalf("ResolvePipeline() error = %v, want nil", err)
} }
if refs := resolved.ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 { if refs := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
t.Fatalf("selected extract references = %#v, want none", refs) t.Fatalf("selected extract references = %#v, want none", refs)
} }
if refs := resolved.ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 { if refs := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
t.Fatalf("selected normalize references = %#v, want none", refs) t.Fatalf("selected normalize references = %#v, want none", refs)
} }
} }
func TestResolvePipelineRejectsPipelineReferenceNotDeclaredByAnyLane(t *testing.T) { func TestResolvePipelineRejectsPipelineReferenceNotDeclaredByAnyLane(t *testing.T) {
profile := multiLaneProfile() profile := multiLaneProfile()
profile.References = map[string]string{"missing": "./missing.md"} profile.References = ExternalReferenceMap(map[string]string{"missing": "./missing.md"})
_, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"events"}}, newProfileCatalog(t)) _, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"events"}}, newProfileCatalog(t))
if err == nil { if err == nil {
@@ -586,7 +640,7 @@ func TestResolvePipelineRejectsPipelineReferenceNotDeclaredByAnyLane(t *testing.
func TestResolvePipelineRejectsUndeclaredReferenceSlot(t *testing.T) { func TestResolvePipelineRejectsUndeclaredReferenceSlot(t *testing.T) {
profile := baselineProfile() profile := baselineProfile()
lane := profile.Artifacts["events"] lane := profile.Artifacts["events"]
lane.References = map[string]string{"missing": "./missing.yml"} lane.References = ExternalReferenceMap(map[string]string{"missing": "./missing.yml"})
profile.Artifacts["events"] = lane profile.Artifacts["events"] = lane
_, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t)) _, err := ResolvePipeline(profile, ResolveOptions{}, newProfileCatalog(t))
@@ -599,7 +653,7 @@ func TestResolvePipelineRejectsUndeclaredReferenceSlot(t *testing.T) {
func TestResolvePipelineRejectsExtractLocalReferenceDeclaredOnlyByNormalizer(t *testing.T) { func TestResolvePipelineRejectsExtractLocalReferenceDeclaredOnlyByNormalizer(t *testing.T) {
profile := baselineProfile() profile := baselineProfile()
lane := profile.Artifacts["events"] lane := profile.Artifacts["events"]
lane.Extract.References = map[string]string{"normalization_notes": "./normalize.md"} lane.Extract.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./normalize.md"})
profile.Artifacts["events"] = lane profile.Artifacts["events"] = lane
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "noop", Key: "noop",
@@ -619,7 +673,7 @@ func TestResolvePipelineRejectsExtractLocalReferenceDeclaredOnlyByNormalizer(t *
func TestResolvePipelineRejectsMergeLocalReferenceDeclaredOnlyByNormalizer(t *testing.T) { func TestResolvePipelineRejectsMergeLocalReferenceDeclaredOnlyByNormalizer(t *testing.T) {
profile := baselineProfile() profile := baselineProfile()
lane := profile.Artifacts["events"] lane := profile.Artifacts["events"]
lane.Merge.References = map[string]string{"normalization_notes": "./normalize.md"} lane.Merge.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./normalize.md"})
profile.Artifacts["events"] = lane profile.Artifacts["events"] = lane
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "noop", Key: "noop",
@@ -641,7 +695,7 @@ func TestResolvePipelineRejectsMergeLocalReferenceDeclaredOnlyByNormalizer(t *te
func TestResolvePipelineRejectsNormalizeLocalReferenceDeclaredOnlyByExtractor(t *testing.T) { func TestResolvePipelineRejectsNormalizeLocalReferenceDeclaredOnlyByExtractor(t *testing.T) {
profile := baselineProfile() profile := baselineProfile()
lane := profile.Artifacts["events"] lane := profile.Artifacts["events"]
lane.Normalize.References = map[string]string{"roster": "./roster.yml"} lane.Normalize.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
profile.Artifacts["events"] = lane profile.Artifacts["events"] = lane
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{ catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "event-extractor", Key: "event-extractor",
@@ -692,15 +746,15 @@ func TestResolvePipelineRequiresBoundNormalizeReference(t *testing.T) {
func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTargets(t *testing.T) { func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTargets(t *testing.T) {
profile := baselineProfile() profile := baselineProfile()
profile.References = map[string]string{ profile.References = ExternalReferenceMap(map[string]string{
"context": "./shared-context.md", "context": "./shared-context.md",
"roster": "./shared-roster.yml", "roster": "./shared-roster.yml",
"normalization_notes": "./shared-normalize.md", "normalization_notes": "./shared-normalize.md",
} })
profile.Chunk.References = map[string]string{"context": "./chunk-context.md"} profile.Chunk.References = ExternalReferenceMap(map[string]string{"context": "./chunk-context.md"})
lane := profile.Artifacts["events"] lane := profile.Artifacts["events"]
lane.Extract.References = map[string]string{"roster": "./extract-roster.yml"} lane.Extract.References = ExternalReferenceMap(map[string]string{"roster": "./extract-roster.yml"})
lane.Normalize.References = map[string]string{"normalization_notes": "./local-normalize.md"} lane.Normalize.References = ExternalReferenceMap(map[string]string{"normalization_notes": "./local-normalize.md"})
profile.Artifacts["events"] = lane profile.Artifacts["events"] = lane
catalog := newProfileCatalogWithOverrides(t, catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}}, ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
@@ -714,8 +768,8 @@ func TestResolvePipelineLocalReferencesOverridePipelineDefaultsForEligibleTarget
} }
assertBindingSource(t, resolved.ChunkReferences.Bindings, "context", "./chunk-context.md") assertBindingSource(t, resolved.ChunkReferences.Bindings, "context", "./chunk-context.md")
assertBindingSource(t, resolved.ArtifactLanes[0].ExtractReferences.Bindings, "roster", "./extract-roster.yml") assertBindingSource(t, resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings, "roster", "./extract-roster.yml")
assertBindingSource(t, resolved.ArtifactLanes[0].NormalizeReferences.Bindings, "normalization_notes", "./local-normalize.md") assertBindingSource(t, resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings, "normalization_notes", "./local-normalize.md")
} }
func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T) { func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T) {
@@ -742,7 +796,7 @@ func TestResolvePipelineRequiresBoundReferenceSlotsForSelectedLanes(t *testing.T
func TestResolvePipelineReferenceUnbindCanLeaveRequiredSlotMissing(t *testing.T) { func TestResolvePipelineReferenceUnbindCanLeaveRequiredSlotMissing(t *testing.T) {
profile := baselineProfile() profile := baselineProfile()
profile.References = map[string]string{"roster": "./roster.yml"} profile.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
catalog := newProfileCatalogWithOverride(t, ModuleSpec{ catalog := newProfileCatalogWithOverride(t, ModuleSpec{
Key: "event-extractor", Key: "event-extractor",
Stage: StageExtract, Stage: StageExtract,
@@ -764,7 +818,7 @@ func TestResolvePipelineReferenceUnbindCanLeaveRequiredSlotMissing(t *testing.T)
func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t *testing.T) { func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t *testing.T) {
profile := baselineProfile() profile := baselineProfile()
profile.References = map[string]string{"roster": "./roster.yml"} profile.References = ExternalReferenceMap(map[string]string{"roster": "./roster.yml"})
catalog := emptyProfileCatalog() catalog := emptyProfileCatalog()
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, notesCodec()) mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, notesCodec())
for _, spec := range defaultProfileSpecs() { for _, spec := range defaultProfileSpecs() {
@@ -791,7 +845,7 @@ func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t
if err != nil { if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err) t.Fatalf("ResolvePipeline() error = %v, want nil", err)
} }
if got := resolved.ArtifactLanes[0].ExtractReferences.Bindings[0].Source; got != "./roster.yml" { if got := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings[0].Source; got != "./roster.yml" {
t.Fatalf("reference source = %q, want ./roster.yml", got) t.Fatalf("reference source = %q, want ./roster.yml", got)
} }
} }
@@ -998,7 +1052,7 @@ func TestResolvePipelineOrdersLanesDeterministically(t *testing.T) {
t.Fatalf("ResolvePipeline() error = %v, want nil", err) t.Fatalf("ResolvePipeline() error = %v, want nil", err)
} }
got := laneIDs(resolved.ArtifactLanes) got := laneIDs(resolved.Steps[0].ArtifactLanes)
want := []string{"events", "notes", "summaries"} want := []string{"events", "notes", "summaries"}
if !reflect.DeepEqual(got, want) { if !reflect.DeepEqual(got, want) {
t.Fatalf("lane IDs = %#v, want %#v", got, want) t.Fatalf("lane IDs = %#v, want %#v", got, want)
@@ -1162,7 +1216,7 @@ func TestResolvedPipelineDigestIncludesCompleteValidatorPolicy(t *testing.T) {
value.ValidatorChains[0].Validators[0].Binding.Options["threshold"] = 0.75 value.ValidatorChains[0].Validators[0].Binding.Options["threshold"] = 0.75
}}, }},
{name: "references", mutate: func(value *ResolvedPipeline) { {name: "references", mutate: func(value *ResolvedPipeline) {
value.ValidatorChains[0].Validators[0].Binding.References["rules"] = "other.md" value.ValidatorChains[0].Validators[0].Binding.References["rules"] = ExternalReference("other.md")
}}, }},
{name: "execution class", mutate: func(value *ResolvedPipeline) { {name: "execution class", mutate: func(value *ResolvedPipeline) {
value.ValidatorChains[0].Validators[0].ExecutionClass = contracts.ExecutionClassDeterministic value.ValidatorChains[0].Validators[0].ExecutionClass = contracts.ExecutionClassDeterministic
@@ -1194,9 +1248,9 @@ func TestResolvedPipelineDigestCanonicalizesValidatorBindingMaps(t *testing.T) {
right.ValidatorChains[0].Validators[0].Binding.Options = map[string]any{} right.ValidatorChains[0].Validators[0].Binding.Options = map[string]any{}
right.ValidatorChains[0].Validators[0].Binding.Options["threshold"] = 0.5 right.ValidatorChains[0].Validators[0].Binding.Options["threshold"] = 0.5
right.ValidatorChains[0].Validators[0].Binding.Options["mode"] = "strict" right.ValidatorChains[0].Validators[0].Binding.Options["mode"] = "strict"
right.ValidatorChains[0].Validators[0].Binding.References = map[string]string{} right.ValidatorChains[0].Validators[0].Binding.References = ExternalReferenceMap(map[string]string{})
right.ValidatorChains[0].Validators[0].Binding.References["examples"] = "examples.md" right.ValidatorChains[0].Validators[0].Binding.References["examples"] = ExternalReference("examples.md")
right.ValidatorChains[0].Validators[0].Binding.References["rules"] = "rules.md" right.ValidatorChains[0].Validators[0].Binding.References["rules"] = ExternalReference("rules.md")
leftDigest, err := resolvedPipelineDigest(left) leftDigest, err := resolvedPipelineDigest(left)
if err != nil { if err != nil {
@@ -1225,7 +1279,7 @@ func validatorDigestFixture() ResolvedPipeline {
LLMProfile: "careful", LLMProfile: "careful",
Retries: 2, Retries: 2,
Options: map[string]any{"mode": "strict", "threshold": 0.5}, Options: map[string]any{"mode": "strict", "threshold": 0.5},
References: map[string]string{"rules": "rules.md", "examples": "examples.md"}, References: ExternalReferenceMap(map[string]string{"rules": "rules.md", "examples": "examples.md"}),
}, },
ExecutionClass: contracts.ExecutionClassLLMBacked, ExecutionClass: contracts.ExecutionClassLLMBacked,
Target: ValidatorTargetTyped, Target: ValidatorTargetTyped,

View File

@@ -36,37 +36,51 @@ func MaterializeReferences(resolved ResolvedPipeline, catalog ModuleCatalog, opt
} }
out.ChunkReferences.ReferenceSet = chunkReferenceSet out.ChunkReferences.ReferenceSet = chunkReferenceSet
warnings := append([]contracts.Warning(nil), chunkWarnings...) warnings := append([]contracts.Warning(nil), chunkWarnings...)
if len(resolved.ArtifactLanes) == 0 { if len(resolved.Steps) == 0 {
return out, warnings, nil return out, warnings, nil
} }
materializeLane := func(lane ResolvedArtifactLane) (ResolvedArtifactLane, []contracts.Warning, error) {
out.ArtifactLanes = make([]ResolvedArtifactLane, len(resolved.ArtifactLanes))
for i, lane := range resolved.ArtifactLanes {
materializedLane := lane materializedLane := lane
var allWarnings []contracts.Warning
materializedLane.ExtractReferences = CloneReferenceTarget(lane.ExtractReferences) materializedLane.ExtractReferences = CloneReferenceTarget(lane.ExtractReferences)
materializedLane.MergeReferences = CloneReferenceTarget(lane.MergeReferences) materializedLane.MergeReferences = CloneReferenceTarget(lane.MergeReferences)
materializedLane.NormalizeReferences = CloneReferenceTarget(lane.NormalizeReferences) materializedLane.NormalizeReferences = CloneReferenceTarget(lane.NormalizeReferences)
extractReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.ExtractReferences, lane.ArtifactKind, catalog, options) extractReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.ExtractReferences, lane.ArtifactKind, catalog, options)
if err != nil { if err != nil {
return ResolvedPipeline{}, nil, err return ResolvedArtifactLane{}, nil, err
} }
materializedLane.ExtractReferences.ReferenceSet = extractReferenceSet materializedLane.ExtractReferences.ReferenceSet = extractReferenceSet
warnings = append(warnings, laneWarnings...) allWarnings = append(allWarnings, laneWarnings...)
mergeReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.MergeReferences, lane.ArtifactKind, catalog, options) mergeReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.MergeReferences, lane.ArtifactKind, catalog, options)
if err != nil { if err != nil {
return ResolvedPipeline{}, nil, err return ResolvedArtifactLane{}, nil, err
} }
materializedLane.MergeReferences.ReferenceSet = mergeReferenceSet materializedLane.MergeReferences.ReferenceSet = mergeReferenceSet
warnings = append(warnings, laneWarnings...) allWarnings = append(allWarnings, laneWarnings...)
normalizeReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.NormalizeReferences, lane.ArtifactKind, catalog, options) normalizeReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.NormalizeReferences, lane.ArtifactKind, catalog, options)
if err != nil { if err != nil {
return ResolvedPipeline{}, nil, err return ResolvedArtifactLane{}, nil, err
} }
materializedLane.NormalizeReferences.ReferenceSet = normalizeReferenceSet materializedLane.NormalizeReferences.ReferenceSet = normalizeReferenceSet
warnings = append(warnings, laneWarnings...) allWarnings = append(allWarnings, laneWarnings...)
out.ArtifactLanes[i] = materializedLane return materializedLane, allWarnings, nil
}
if len(resolved.Steps) > 0 {
out.Steps = make([]ResolvedPipelineStep, len(resolved.Steps))
for i, step := range resolved.Steps {
out.Steps[i].ID = step.ID
out.Steps[i].ArtifactLanes = make([]ResolvedArtifactLane, len(step.ArtifactLanes))
for j, lane := range step.ArtifactLanes {
materializedLane, laneWarnings, err := materializeLane(lane)
if err != nil {
return ResolvedPipeline{}, nil, err
}
out.Steps[i].ArtifactLanes[j] = materializedLane
warnings = append(warnings, laneWarnings...)
}
}
} }
return out, warnings, nil return out, warnings, nil
} }
@@ -99,6 +113,12 @@ func materializeReferenceTarget(
if !ok { if !ok {
return contracts.ReferenceSet{}, nil, fmt.Errorf("%s reference slot %q is not declared by %s module %q", referenceTargetContext(pipelineID, target), slotName, target.Stage, target.Module) return contracts.ReferenceSet{}, nil, fmt.Errorf("%s reference slot %q is not declared by %s module %q", referenceTargetContext(pipelineID, target), slotName, target.Stage, target.Module)
} }
if binding.Artifact != nil {
// Generated selectors are resolved at the step handoff. Keep the
// declared slot and its constraints, but do not materialize bytes yet.
set.Slots[slotName] = contracts.ResolvedReferenceSlot{Slot: cloneReferenceSlot(slot)}
continue
}
path, err := referencePath(binding, options) path, err := referencePath(binding, options)
if err != nil { if err != nil {
@@ -273,6 +293,7 @@ func fileURI(path string) string {
func cloneReferenceSlot(slot contracts.ReferenceSlot) contracts.ReferenceSlot { func cloneReferenceSlot(slot contracts.ReferenceSlot) contracts.ReferenceSlot {
slot.AcceptedMediaTypes = append([]string(nil), slot.AcceptedMediaTypes...) slot.AcceptedMediaTypes = append([]string(nil), slot.AcceptedMediaTypes...)
slot.AcceptedArtifactKinds = append([]contracts.ArtifactKind(nil), slot.AcceptedArtifactKinds...)
return slot return slot
} }
@@ -292,8 +313,7 @@ func CloneReferenceSet(in contracts.ReferenceSet) contracts.ReferenceSet {
if len(slot.Items) > 0 { if len(slot.Items) > 0 {
items := make([]contracts.ReferenceItem, len(slot.Items)) items := make([]contracts.ReferenceItem, len(slot.Items))
for i, item := range slot.Items { for i, item := range slot.Items {
item.Content = append([]byte(nil), item.Content...) items[i] = contracts.CloneReferenceItem(item)
items[i] = item
} }
slot.Items = items slot.Items = items
} }
@@ -304,7 +324,16 @@ func CloneReferenceSet(in contracts.ReferenceSet) contracts.ReferenceSet {
func CloneReferenceTarget(in ResolvedReferenceTarget) ResolvedReferenceTarget { func CloneReferenceTarget(in ResolvedReferenceTarget) ResolvedReferenceTarget {
out := in out := in
out.Bindings = append([]ReferenceBinding(nil), in.Bindings...) if len(in.Bindings) > 0 {
out.Bindings = make([]ReferenceBinding, len(in.Bindings))
for i, binding := range in.Bindings {
out.Bindings[i] = binding
if binding.Artifact != nil {
artifact := *binding.Artifact
out.Bindings[i].Artifact = &artifact
}
}
}
out.ReferenceSet = CloneReferenceSet(in.ReferenceSet) out.ReferenceSet = CloneReferenceSet(in.ReferenceSet)
return out return out
} }
@@ -312,10 +341,14 @@ func CloneReferenceTarget(in ResolvedReferenceTarget) ResolvedReferenceTarget {
func ReferenceProvenance(resolved ResolvedPipeline) []artifacts.ReferenceProvenance { func ReferenceProvenance(resolved ResolvedPipeline) []artifacts.ReferenceProvenance {
provenance := []artifacts.ReferenceProvenance{} provenance := []artifacts.ReferenceProvenance{}
provenance = append(provenance, referenceTargetProvenance(resolved.ChunkReferences)...) provenance = append(provenance, referenceTargetProvenance(resolved.ChunkReferences)...)
for _, lane := range resolved.ArtifactLanes { if len(resolved.Steps) > 0 {
provenance = append(provenance, referenceTargetProvenance(lane.ExtractReferences)...) for _, step := range resolved.Steps {
provenance = append(provenance, referenceTargetProvenance(lane.MergeReferences)...) for _, lane := range step.ArtifactLanes {
provenance = append(provenance, referenceTargetProvenance(lane.NormalizeReferences)...) provenance = append(provenance, referenceTargetProvenance(lane.ExtractReferences)...)
provenance = append(provenance, referenceTargetProvenance(lane.MergeReferences)...)
provenance = append(provenance, referenceTargetProvenance(lane.NormalizeReferences)...)
}
}
} }
return provenance return provenance
} }
@@ -333,16 +366,30 @@ func referenceTargetProvenance(target ResolvedReferenceTarget) []artifacts.Refer
for _, slotName := range slotNames { for _, slotName := range slotNames {
slot := target.ReferenceSet.Slots[slotName] slot := target.ReferenceSet.Slots[slotName]
for _, item := range slot.Items { for _, item := range slot.Items {
schemaDigest := ""
if item.ArtifactSchema.ID != "" || item.ArtifactSchema.Name != "" || item.ArtifactSchema.Version != "" || len(item.ArtifactSchema.JSONSchema) > 0 {
schemaDigest = contracts.DigestArtifactSchema(item.ArtifactSchema)
}
provenance = append(provenance, artifacts.ReferenceProvenance{ provenance = append(provenance, artifacts.ReferenceProvenance{
Stage: string(target.Stage), Stage: string(target.Stage),
LaneID: target.LaneID, StepID: target.StepID,
SlotName: item.SlotName, LaneID: target.LaneID,
OriginType: item.Origin.Type, SlotName: item.SlotName,
OriginURI: item.Origin.URI, OriginType: item.Origin.Type,
Digest: item.Digest, OriginURI: item.Origin.URI,
MediaType: item.MediaType, Digest: item.Digest,
SizeBytes: item.SizeBytes, MediaType: item.MediaType,
BindingSource: item.BindingSource, SizeBytes: item.SizeBytes,
BindingSource: item.BindingSource,
ArtifactKind: string(item.ArtifactKind),
SchemaID: item.ArtifactSchema.ID,
SchemaName: item.ArtifactSchema.Name,
SchemaVersion: item.ArtifactSchema.Version,
SchemaDigest: schemaDigest,
ProducerPipeline: item.Producer.PipelineID,
ProducerStep: item.Producer.StepID,
ProducerLane: item.Producer.LaneID,
ProducerModule: item.Producer.ModuleKey,
}) })
} }
} }

View File

@@ -21,9 +21,9 @@ func TestMaterializeReferencesResolvesPathsAndDigestsContent(t *testing.T) {
writeReferenceFile(t, cliReference, []byte("cli text")) writeReferenceFile(t, cliReference, []byte("cli text"))
pipeline := baselineProfile() pipeline := baselineProfile()
pipeline.References = map[string]string{"roster": "config-reference.txt"} pipeline.References = ExternalReferenceMap(map[string]string{"roster": "config-reference.txt"})
lane := pipeline.Artifacts["events"] lane := pipeline.Artifacts["events"]
lane.References = map[string]string{"glossary": "cli-reference.txt"} lane.References = ExternalReferenceMap(map[string]string{"glossary": "cli-reference.txt"})
pipeline.Artifacts["events"] = lane pipeline.Artifacts["events"] = lane
catalog := referenceCatalog(t, []contracts.ReferenceSlot{ catalog := referenceCatalog(t, []contracts.ReferenceSlot{
{Name: "roster"}, {Name: "roster"},
@@ -56,12 +56,12 @@ func TestMaterializeReferencesResolvesPathsAndDigestsContent(t *testing.T) {
t.Fatalf("MaterializeReferences(second) error = %v, want nil", err) t.Fatalf("MaterializeReferences(second) error = %v, want nil", err)
} }
referenceSet := first.ArtifactLanes[0].ExtractReferences.ReferenceSet referenceSet := first.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet
roster := referenceSet.Slots["roster"].Items[0] roster := referenceSet.Slots["roster"].Items[0]
if string(roster.Content) != "config text" { if string(roster.Content) != "config text" {
t.Fatalf("roster content = %q, want config text", roster.Content) t.Fatalf("roster content = %q, want config text", roster.Content)
} }
if roster.Digest != referenceDigest([]byte("config text")) || roster.Digest != second.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0].Digest { if roster.Digest != referenceDigest([]byte("config text")) || roster.Digest != second.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0].Digest {
t.Fatalf("roster digest = %q, want stable digest", roster.Digest) t.Fatalf("roster digest = %q, want stable digest", roster.Digest)
} }
if roster.BindingSource != contracts.ReferenceBindingSourceConfig { if roster.BindingSource != contracts.ReferenceBindingSourceConfig {
@@ -113,12 +113,12 @@ func TestMaterializeReferencesStoresSetsAndProvenanceForAllTargets(t *testing.T)
writeReferenceFile(t, filepath.Join(configDir, "normalize.txt"), []byte("normalize text")) writeReferenceFile(t, filepath.Join(configDir, "normalize.txt"), []byte("normalize text"))
profile := baselineProfile() profile := baselineProfile()
profile.References = map[string]string{ profile.References = ExternalReferenceMap(map[string]string{
"scene_guide": "chunk.txt", "scene_guide": "chunk.txt",
"roster": "extract.txt", "roster": "extract.txt",
"merge_notes": "merge.txt", "merge_notes": "merge.txt",
"normalization_notes": "normalize.txt", "normalization_notes": "normalize.txt",
} })
catalog := referenceCatalogForTargets(t, catalog := referenceCatalogForTargets(t,
[]contracts.ReferenceSlot{{Name: "scene_guide"}}, []contracts.ReferenceSlot{{Name: "scene_guide"}},
[]contracts.ReferenceSlot{{Name: "roster"}}, []contracts.ReferenceSlot{{Name: "roster"}},
@@ -144,15 +144,15 @@ func TestMaterializeReferencesStoresSetsAndProvenanceForAllTargets(t *testing.T)
if string(chunkItem.Content) != "chunk text" { if string(chunkItem.Content) != "chunk text" {
t.Fatalf("chunk content = %q, want chunk text", chunkItem.Content) t.Fatalf("chunk content = %q, want chunk text", chunkItem.Content)
} }
extractItem := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0] extractItem := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
if string(extractItem.Content) != "extract text" { if string(extractItem.Content) != "extract text" {
t.Fatalf("extract content = %q, want extract text", extractItem.Content) t.Fatalf("extract content = %q, want extract text", extractItem.Content)
} }
mergeItem := materialized.ArtifactLanes[0].MergeReferences.ReferenceSet.Slots["merge_notes"].Items[0] mergeItem := materialized.Steps[0].ArtifactLanes[0].MergeReferences.ReferenceSet.Slots["merge_notes"].Items[0]
if string(mergeItem.Content) != "merge text" { if string(mergeItem.Content) != "merge text" {
t.Fatalf("merge content = %q, want merge text", mergeItem.Content) t.Fatalf("merge content = %q, want merge text", mergeItem.Content)
} }
normalizeItem := materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["normalization_notes"].Items[0] normalizeItem := materialized.Steps[0].ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["normalization_notes"].Items[0]
if string(normalizeItem.Content) != "normalize text" { if string(normalizeItem.Content) != "normalize text" {
t.Fatalf("normalize content = %q, want normalize text", normalizeItem.Content) t.Fatalf("normalize content = %q, want normalize text", normalizeItem.Content)
} }
@@ -202,22 +202,25 @@ func TestMaterializeReferencesUsesLaneArtifactVariant(t *testing.T) {
} }
resolved := ResolvedPipeline{ resolved := ResolvedPipeline{
ID: "variants", ID: "variants",
ArtifactLanes: []ResolvedArtifactLane{{ Steps: []ResolvedPipelineStep{{
ID: "alpha", ID: "default",
ArtifactKind: "test/alpha", ArtifactLanes: []ResolvedArtifactLane{{
MergeReferences: referenceTarget(StageMerge, "alpha", "shared/merge", []ReferenceBinding{{ ID: "alpha",
Stage: StageMerge, LaneID: "alpha", SlotName: "alpha_merge", Source: "merge.txt", BindingSource: contracts.ReferenceBindingSourceConfig, ArtifactKind: "test/alpha",
}}), MergeReferences: referenceTarget(StageMerge, "alpha", "shared/merge", []ReferenceBinding{{
NormalizeReferences: referenceTarget(StageNormalize, "alpha", "shared/normalize", []ReferenceBinding{{ Stage: StageMerge, LaneID: "alpha", SlotName: "alpha_merge", Source: "merge.txt", BindingSource: contracts.ReferenceBindingSourceConfig,
Stage: StageNormalize, LaneID: "alpha", SlotName: "alpha_normalize", Source: "normalize.txt", BindingSource: contracts.ReferenceBindingSourceConfig, }}),
}}), NormalizeReferences: referenceTarget(StageNormalize, "alpha", "shared/normalize", []ReferenceBinding{{
Stage: StageNormalize, LaneID: "alpha", SlotName: "alpha_normalize", Source: "normalize.txt", BindingSource: contracts.ReferenceBindingSourceConfig,
}}),
}},
}}, }},
} }
materialized, _, err := MaterializeReferences(resolved, ModuleCatalog{Mergers: mergers, Normalizers: normalizers}, ReferenceMaterializationOptions{ConfigPath: filepath.Join(configDir, "notarius.yml")}) materialized, _, err := MaterializeReferences(resolved, ModuleCatalog{Mergers: mergers, Normalizers: normalizers}, ReferenceMaterializationOptions{ConfigPath: filepath.Join(configDir, "notarius.yml")})
if err != nil { if err != nil {
t.Fatalf("MaterializeReferences() error = %v, want nil", err) t.Fatalf("MaterializeReferences() error = %v, want nil", err)
} }
lane := materialized.ArtifactLanes[0] lane := materialized.Steps[0].ArtifactLanes[0]
if got := string(lane.MergeReferences.ReferenceSet.Slots["alpha_merge"].Items[0].Content); got != "merge alpha" { if got := string(lane.MergeReferences.ReferenceSet.Slots["alpha_merge"].Items[0].Content); got != "merge alpha" {
t.Fatalf("merge reference = %q", got) t.Fatalf("merge reference = %q", got)
} }
@@ -266,7 +269,7 @@ func TestMaterializeReferencesAllowsAnyMediaTypeWhenSlotDoesNotRestrictIt(t *tes
if err != nil { if err != nil {
t.Fatalf("MaterializeReferences() error = %v, want nil", err) t.Fatalf("MaterializeReferences() error = %v, want nil", err)
} }
item := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0] item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
if item.MediaType != unknownMediaType { if item.MediaType != unknownMediaType {
t.Fatalf("MediaType = %q, want %q", item.MediaType, unknownMediaType) t.Fatalf("MediaType = %q, want %q", item.MediaType, unknownMediaType)
} }
@@ -285,7 +288,7 @@ func TestMaterializeReferencesAcceptsDeclaredMarkdownMediaType(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("MaterializeReferences() error = %v, want nil", err) t.Fatalf("MaterializeReferences() error = %v, want nil", err)
} }
item := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["glossary"].Items[0] item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["glossary"].Items[0]
if item.MediaType != "text/markdown" { if item.MediaType != "text/markdown" {
t.Fatalf("MediaType = %q, want text/markdown", item.MediaType) t.Fatalf("MediaType = %q, want text/markdown", item.MediaType)
} }
@@ -304,7 +307,7 @@ func TestMaterializeReferencesAcceptsDeclaredJSONMediaType(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("MaterializeReferences() error = %v, want nil", err) t.Fatalf("MaterializeReferences() error = %v, want nil", err)
} }
item := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0] item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
if item.MediaType != "application/json" { if item.MediaType != "application/json" {
t.Fatalf("MediaType = %q, want application/json", item.MediaType) t.Fatalf("MediaType = %q, want application/json", item.MediaType)
} }
@@ -323,7 +326,7 @@ func TestMaterializeReferencesAcceptsDeclaredYAMLMediaType(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("MaterializeReferences() error = %v, want nil", err) t.Fatalf("MaterializeReferences() error = %v, want nil", err)
} }
item := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0] item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
if item.MediaType != "application/yaml" { if item.MediaType != "application/yaml" {
t.Fatalf("MediaType = %q, want application/yaml", item.MediaType) t.Fatalf("MediaType = %q, want application/yaml", item.MediaType)
} }
@@ -357,7 +360,7 @@ func TestMaterializeReferencesMatchesAcceptedMediaTypesIgnoringParameters(t *tes
if err != nil { if err != nil {
t.Fatalf("MaterializeReferences() error = %v, want nil", err) t.Fatalf("MaterializeReferences() error = %v, want nil", err)
} }
item := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0] item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
if item.MediaType != referenceMediaType { if item.MediaType != referenceMediaType {
t.Fatalf("MediaType = %q, want %q", item.MediaType, referenceMediaType) t.Fatalf("MediaType = %q, want %q", item.MediaType, referenceMediaType)
} }
@@ -393,7 +396,7 @@ func TestMaterializeReferencesWarnsForEmptyFiles(t *testing.T) {
if len(warnings) != 1 || warnings[0].ReasonCode != "empty_reference" { if len(warnings) != 1 || warnings[0].ReasonCode != "empty_reference" {
t.Fatalf("warnings = %#v, want empty reference warning", warnings) t.Fatalf("warnings = %#v, want empty reference warning", warnings)
} }
item := materialized.ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0] item := materialized.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet.Slots["roster"].Items[0]
if item.SizeBytes != 0 || item.Digest != referenceDigest(nil) { if item.SizeBytes != 0 || item.Digest != referenceDigest(nil) {
t.Fatalf("empty item = %#v, want zero size and empty digest", item) t.Fatalf("empty item = %#v, want zero size and empty digest", item)
} }
@@ -406,11 +409,11 @@ func TestMaterializeReferencesWarningScopesIncludeTargetContext(t *testing.T) {
writeReferenceFile(t, filepath.Join(configDir, "normalize.txt"), nil) writeReferenceFile(t, filepath.Join(configDir, "normalize.txt"), nil)
profile := baselineProfile() profile := baselineProfile()
profile.References = map[string]string{ profile.References = ExternalReferenceMap(map[string]string{
"scene_guide": "chunk.txt", "scene_guide": "chunk.txt",
"roster": "extract.txt", "roster": "extract.txt",
"normalization_notes": "normalize.txt", "normalization_notes": "normalize.txt",
} })
catalog := referenceCatalogForTargets(t, catalog := referenceCatalogForTargets(t,
[]contracts.ReferenceSlot{{Name: "scene_guide"}}, []contracts.ReferenceSlot{{Name: "scene_guide"}},
[]contracts.ReferenceSlot{{Name: "roster"}}, []contracts.ReferenceSlot{{Name: "roster"}},
@@ -464,18 +467,18 @@ func resolvedPipelineWithTargetReference(t *testing.T, stage ModuleStage, laneID
profile := baselineProfile() profile := baselineProfile()
switch stage { switch stage {
case StageChunk: case StageChunk:
profile.Chunk.References = map[string]string{slotName: source} profile.Chunk.References = ExternalReferenceMap(map[string]string{slotName: source})
case StageExtract: case StageExtract:
lane := profile.Artifacts[laneID] lane := profile.Artifacts[laneID]
lane.References = map[string]string{slotName: source} lane.References = ExternalReferenceMap(map[string]string{slotName: source})
profile.Artifacts[laneID] = lane profile.Artifacts[laneID] = lane
case StageMerge: case StageMerge:
lane := profile.Artifacts[laneID] lane := profile.Artifacts[laneID]
lane.Merge.References = map[string]string{slotName: source} lane.Merge.References = ExternalReferenceMap(map[string]string{slotName: source})
profile.Artifacts[laneID] = lane profile.Artifacts[laneID] = lane
case StageNormalize: case StageNormalize:
lane := profile.Artifacts[laneID] lane := profile.Artifacts[laneID]
lane.Normalize.References = map[string]string{slotName: source} lane.Normalize.References = ExternalReferenceMap(map[string]string{slotName: source})
profile.Artifacts[laneID] = lane profile.Artifacts[laneID] = lane
default: default:
t.Fatalf("unsupported reference target stage %q", stage) t.Fatalf("unsupported reference target stage %q", stage)
@@ -490,11 +493,11 @@ func resolvedPipelineWithTargetReference(t *testing.T, stage ModuleStage, laneID
case StageChunk: case StageChunk:
resolved.ChunkReferences.Bindings[0].BindingSource = bindingSource resolved.ChunkReferences.Bindings[0].BindingSource = bindingSource
case StageExtract: case StageExtract:
resolved.ArtifactLanes[0].ExtractReferences.Bindings[0].BindingSource = bindingSource resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings[0].BindingSource = bindingSource
case StageMerge: case StageMerge:
resolved.ArtifactLanes[0].MergeReferences.Bindings[0].BindingSource = bindingSource resolved.Steps[0].ArtifactLanes[0].MergeReferences.Bindings[0].BindingSource = bindingSource
case StageNormalize: case StageNormalize:
resolved.ArtifactLanes[0].NormalizeReferences.Bindings[0].BindingSource = bindingSource resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings[0].BindingSource = bindingSource
} }
} }
return resolved return resolved

View File

@@ -38,27 +38,30 @@ func New() *Runner {
} }
type RunInput struct { type RunInput struct {
Prepared *PreparedPipeline Prepared *PreparedPipeline
SourceID string SourceID string
Path string Path string
RawInput []byte RawInput []byte
SessionID string SessionID string
RunID string RunID string
StartedAt time.Time StartedAt time.Time
LLMProfiles []artifacts.LLMProfileManifest LLMProfiles []artifacts.LLMProfileManifest
Metadata map[string]any Metadata map[string]any
Warnings []contracts.Warning Warnings []contracts.Warning
ChunkCacheMode ChunkCacheMode ChunkCacheMode ChunkCacheMode
ChunkPlans ChunkPlanStore ChunkPlans ChunkPlanStore
Checkpoints CheckpointRecorder Checkpoints CheckpointRecorder
Checkpoint CheckpointLoader Checkpoint CheckpointLoader
Debug DebugRecorder CheckpointPolicy CheckpointExecutionPolicy
Debug DebugRecorder
// ExtractWorkers bounds run-wide extract jobs. Values less than one use a // ExtractWorkers bounds run-wide extract jobs. Values less than one use a
// single worker so direct framework callers retain deterministic behavior. // single worker so direct framework callers retain deterministic behavior.
ExtractWorkers int ExtractWorkers int
pipeline ResolvedPipeline pipeline ResolvedPipeline
llmClient contracts.StructuredLLMClient llmClient contracts.StructuredLLMClient
stepID string
references map[referenceTargetKey]contracts.ReferenceSet
} }
type RunOutput struct { type RunOutput struct {
@@ -129,7 +132,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
return failOutput(output), err return failOutput(output), err
} }
sourceCheckpoint, sourceDecision := checkpointLoader.Source(adapter.Key()) sourceCheckpoint, sourceDecision := checkpointLoader.Source(adapter.Key())
recordCheckpointEvent(&output, checkpointLoader, "source", "", adapter.Key(), sourceDecision) recordCheckpointEvent(&output, checkpointLoader, "source", "", "", adapter.Key(), sourceDecision)
doc := sourceCheckpoint.Document doc := sourceCheckpoint.Document
sourceStarted := time.Now().UTC() sourceStarted := time.Now().UTC()
if err := writeDebugTimed(debugRecorder, "source/input.json", debugTimedEnvelope{ if err := writeDebugTimed(debugRecorder, "source/input.json", debugTimedEnvelope{
@@ -249,13 +252,9 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
} }
if chunkResult.accepted { if chunkResult.accepted {
laneOutput, laneErr := r.runLanes(ctx, input, checkpoints, checkpointLoader, doc, sourceInput, sessionID, chunkResult.chunks) if err := r.runPreparedSteps(ctx, input, checkpoints, checkpointLoader, doc, sourceInput, sessionID, chunkResult.chunks, &output); err != nil {
if err := mergeLaneOutput(&output, laneOutput); err != nil {
return failOutput(output), err return failOutput(output), err
} }
if laneErr != nil {
return failOutput(output), laneErr
}
} }
if len(output.Rejected) > 0 { if len(output.Rejected) > 0 {
@@ -322,6 +321,27 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
return output, nil return output, nil
} }
func (r *Runner) runPreparedSteps(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk, output *RunOutput) error {
for _, step := range input.Prepared.Steps {
stepInput := input
stepInput.stepID = step.ID
stepReferences, referenceProvenance, err := buildStepReferenceSets(input, step, output.NormalizeOutputs)
if err != nil {
return fmt.Errorf("prepare generated references for pipeline step %q: %w", step.ID, err)
}
stepInput.references = stepReferences
output.Manifest.References = append(output.Manifest.References, referenceProvenance...)
laneOutput, laneErr := r.runLanes(ctx, stepInput, step, checkpoints, loader, doc, sourceInput, sessionID, chunks)
if err := mergeLaneOutput(output, laneOutput); err != nil {
return err
}
if laneErr != nil {
return fmt.Errorf("execute pipeline step %q: %w", step.ID, laneErr)
}
}
return nil
}
func runWithRetry(ctx context.Context, retries int, run func(attempt int) (bool, *contracts.RejectedOutput, error)) (bool, *contracts.RejectedOutput, error) { func runWithRetry(ctx context.Context, retries int, run func(attempt int) (bool, *contracts.RejectedOutput, error)) (bool, *contracts.RejectedOutput, error) {
attempts := retries + 1 attempts := retries + 1
var last *contracts.RejectedOutput var last *contracts.RejectedOutput
@@ -460,7 +480,10 @@ func validateRunInput(input RunInput) error {
if mode != ChunkCacheBypass && input.ChunkPlans == nil { if mode != ChunkCacheBypass && input.ChunkPlans == nil {
return fmt.Errorf("chunk plan store is required for %q mode", mode) return fmt.Errorf("chunk plan store is required for %q mode", mode)
} }
return validateResolvedPipeline(input.Prepared.resolved) if err := validateResolvedPipeline(input.Prepared.resolved); err != nil {
return err
}
return nil
} }
func validateResolvedPipeline(pipeline ResolvedPipeline) error { func validateResolvedPipeline(pipeline ResolvedPipeline) error {
@@ -479,10 +502,10 @@ func validateResolvedPipeline(pipeline ResolvedPipeline) error {
if pipeline.Output.Module == "" { if pipeline.Output.Module == "" {
return fmt.Errorf("resolved pipeline output module must not be empty") return fmt.Errorf("resolved pipeline output module must not be empty")
} }
if len(pipeline.ArtifactLanes) == 0 { if len(pipeline.Steps) == 0 {
return fmt.Errorf("resolved pipeline artifact lanes must not be empty") return fmt.Errorf("resolved pipeline artifact lanes must not be empty")
} }
for _, lane := range pipeline.ArtifactLanes { for _, lane := range pipeline.AllArtifactLanes() {
if lane.ID == "" { if lane.ID == "" {
return fmt.Errorf("resolved pipeline artifact lane id must not be empty") return fmt.Errorf("resolved pipeline artifact lane id must not be empty")
} }
@@ -530,15 +553,16 @@ func manifestFromPipeline(input RunInput) artifacts.RunManifest {
RequestedModule: pipeline.Chunk.Module, RequestedModule: pipeline.Chunk.Module,
}, },
OutputEncoder: pipeline.Output.Module, OutputEncoder: pipeline.Output.Module,
ArtifactLanes: make([]artifacts.ArtifactLaneManifest, 0, len(pipeline.ArtifactLanes)), ArtifactLanes: make([]artifacts.ArtifactLaneManifest, 0, len(pipeline.AllArtifactLanes())),
ValidatorChains: validatorChainManifests(pipeline.ValidatorChains), ValidatorChains: validatorChainManifests(pipeline.ValidatorChains),
RunID: runID, RunID: runID,
StartedAt: timePtr(startedAt), StartedAt: timePtr(startedAt),
References: ReferenceProvenance(pipeline), References: ReferenceProvenance(pipeline),
LLMProfiles: cloneLLMProfiles(input.LLMProfiles), LLMProfiles: cloneLLMProfiles(input.LLMProfiles),
} }
for _, lane := range pipeline.ArtifactLanes { for _, lane := range pipeline.AllArtifactLanes() {
laneManifest := artifacts.ArtifactLaneManifest{ laneManifest := artifacts.ArtifactLaneManifest{
StepID: lane.StepID,
ID: lane.ID, ID: lane.ID,
Extractor: lane.Extract.Module, Extractor: lane.Extract.Module,
Merger: lane.Merge.Module, Merger: lane.Merge.Module,
@@ -581,29 +605,47 @@ func failOutput(output RunOutput) RunOutput {
return output return output
} }
func recordCheckpointEvent(output *RunOutput, loader CheckpointLoader, stage string, laneID string, moduleKey string, decision CheckpointDecision) { func recordCheckpointEvent(output *RunOutput, loader CheckpointLoader, stage string, stepID string, laneID string, moduleKey string, decision CheckpointDecision) {
if output == nil || loader == nil || !loader.Enabled() { if output == nil || loader == nil || !loader.Enabled() {
return return
} }
action := "executed" action := checkpointDecisionCategory(decision)
if decision.Reused {
action = "reused"
}
output.CheckpointEvents = append(output.CheckpointEvents, CheckpointEvent{ output.CheckpointEvents = append(output.CheckpointEvents, CheckpointEvent{
Stage: stage, Stage: stage,
LaneID: laneID, StepID: stepID,
ModuleKey: moduleKey, LaneID: laneID,
Action: action, ModuleKey: moduleKey,
Reason: decision.Reason, Action: action,
Category: checkpointDecisionCategory(decision),
ReasonCode: decision.ReasonCode,
Detail: decision.Detail,
Reason: decision.Reason,
}) })
} }
func checkpointDecisionCategory(decision CheckpointDecision) CheckpointDecisionCategory {
if decision.Category != "" {
return decision.Category
}
if decision.Reused {
return CheckpointDecisionReused
}
return CheckpointDecisionExecuted
}
func populateOutputManifest(output *RunOutput) { func populateOutputManifest(output *RunOutput) {
if output == nil { if output == nil {
return return
} }
output.Manifest.NormalizedOutputs = normalizedOutputManifests(output.NormalizeOutputs) output.Manifest.NormalizedOutputs = normalizedOutputManifests(output.NormalizeOutputs)
output.Manifest.RejectedOutputs = rejectedOutputManifests(output.Rejected) output.Manifest.RejectedOutputs = rejectedOutputManifests(output.Rejected)
if len(output.CheckpointEvents) > 0 {
decisions := make([]artifacts.CheckpointDecisionManifest, 0, len(output.CheckpointEvents))
for _, event := range output.CheckpointEvents {
decisions = append(decisions, artifacts.CheckpointDecisionManifest{Stage: event.Stage, StepID: event.StepID, LaneID: event.LaneID, ModuleKey: event.ModuleKey, Category: string(event.Category), ReasonCode: string(event.ReasonCode), Detail: event.Detail})
}
output.Manifest.CheckpointDecisions = decisions
}
} }
func normalizedOutputManifests(outputs []contracts.SerializedOutput) []artifacts.NormalizedOutputManifest { func normalizedOutputManifests(outputs []contracts.SerializedOutput) []artifacts.NormalizedOutputManifest {
@@ -613,6 +655,7 @@ func normalizedOutputManifests(outputs []contracts.SerializedOutput) []artifacts
manifests := make([]artifacts.NormalizedOutputManifest, 0, len(outputs)) manifests := make([]artifacts.NormalizedOutputManifest, 0, len(outputs))
for _, output := range outputs { for _, output := range outputs {
manifests = append(manifests, artifacts.NormalizedOutputManifest{ manifests = append(manifests, artifacts.NormalizedOutputManifest{
StepID: output.StepID,
LaneID: output.LaneID, LaneID: output.LaneID,
ModuleKey: output.NormalizerKey, ModuleKey: output.NormalizerKey,
SourceID: output.SourceID, SourceID: output.SourceID,
@@ -635,6 +678,7 @@ func rejectedOutputManifests(rejected []contracts.RejectedOutput) []artifacts.Re
for _, output := range rejected { for _, output := range rejected {
manifests = append(manifests, artifacts.RejectedOutputManifest{ manifests = append(manifests, artifacts.RejectedOutputManifest{
Stage: output.Stage, Stage: output.Stage,
StepID: output.StepID,
LaneID: output.LaneID, LaneID: output.LaneID,
ModuleKey: output.ModuleKey, ModuleKey: output.ModuleKey,
ChunkID: output.ChunkID, ChunkID: output.ChunkID,

View File

@@ -0,0 +1,300 @@
package pipeline
import (
"context"
"encoding/json"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type acceptedCheckpointLoader struct {
CheckpointLoader
accepted map[string]NormalizeCheckpoint
acceptedDecision map[string]CheckpointDecision
extractDeps map[string][]CheckpointFingerprint
}
func newAcceptedCheckpointLoader() *acceptedCheckpointLoader {
return &acceptedCheckpointLoader{
CheckpointLoader: NoopCheckpointLoader(),
accepted: make(map[string]NormalizeCheckpoint),
acceptedDecision: make(map[string]CheckpointDecision),
extractDeps: make(map[string][]CheckpointFingerprint),
}
}
func (l *acceptedCheckpointLoader) Enabled() bool { return true }
func (l *acceptedCheckpointLoader) AcceptedNormalize(stepID, laneID, _ string) (NormalizeCheckpoint, CheckpointDecision) {
key := CheckpointLaneKey(stepID, laneID)
return l.accepted[key], l.acceptedDecision[key]
}
func (l *acceptedCheckpointLoader) Extract(laneID string, _ string, dependencies []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
l.extractDeps[laneID] = append([]CheckpointFingerprint(nil), dependencies...)
return ExtractCheckpoint{}, NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing, "checkpoint missing")
}
func (l *acceptedCheckpointLoader) Merge(_ string, _ string, _ []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
return MergeCheckpoint{}, NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing, "checkpoint missing")
}
func (l *acceptedCheckpointLoader) Normalize(_ string, _ string, _ []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
return NormalizeCheckpoint{}, NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing, "checkpoint missing")
}
func TestRunnerHydratesRequiredNormalizedArtifact(t *testing.T) {
value := codecNotes{Items: []string{"canonical producer value"}}
input, _, _ := handoffFixture(t, value)
prepared := input.Prepared
producer := &prepared.Steps[0].lanes[0]
consumer := &prepared.Steps[1].lanes[0]
doc := prepared.input.(*typedTestInput).doc
stored, err := checkpointArtifact(producer.typed.codec, producer.resolved.ID, producer.resolved.Normalize.Module, doc.ID, value)
if err != nil {
t.Fatal(err)
}
operationCalls := 0
producer.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
operationCalls++
return erasedTypedResult{Value: value}, nil
}
producer.typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
operationCalls++
return erasedTypedResult{Value: value}, nil
}
producer.typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
operationCalls++
return erasedTypedResult{Value: value}, nil
}
validatorCalls := 0
validator := preparedValidator{typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
validatorCalls++
return contracts.ValidationResult{Approved: true}, nil
}}
producer.extractValidators.validators = []preparedValidator{validator}
producer.mergeValidators.validators = []preparedValidator{validator}
producer.normalizeValidators.validators = []preparedValidator{validator}
var received contracts.ReferenceSet
consumer.typed.extract = func(_ context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
received = CloneReferenceSet(request.References)
return erasedTypedResult{Value: codecScore{Value: 3}}, nil
}
loader := newAcceptedCheckpointLoader()
producerKey := CheckpointLaneKey(producer.resolved.StepID, producer.resolved.ID)
consumerKey := CheckpointLaneKey(consumer.resolved.StepID, consumer.resolved.ID)
loader.accepted[producerKey] = NormalizeCheckpoint{Output: stored, Warnings: []contracts.Warning{{Scope: "normalize", ReasonCode: "stored-warning", Message: "stored normalize warning"}}}
loader.acceptedDecision[producerKey] = NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted artifact reusable")
policy := CheckpointExecutionPolicy{
RequireReusableLanes: map[string]struct{}{producerKey: {}},
ForcedLanes: map[string]struct{}{consumerKey: {}},
}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: loader, CheckpointPolicy: policy})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if operationCalls != 0 || validatorCalls != 0 {
t.Fatalf("hydrated producer calls = operations %d validators %d, want zero", operationCalls, validatorCalls)
}
item := received.Slots["producer-output"].Items[0]
if string(item.Content) != string(stored.Artifact.Content) || item.Producer.StepID != producer.resolved.StepID || item.Producer.LaneID != producer.resolved.ID {
t.Fatalf("consumer generated reference = %#v, want exact hydrated producer bytes and identity", item)
}
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != "stored-warning" {
t.Fatalf("hydrated warnings = %#v, want normalize checkpoint warnings only", output.Warnings)
}
assertAcceptedNormalizeEvent(t, output.CheckpointEvents, producer.resolved.StepID, producer.resolved.ID, CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused)
for _, event := range output.CheckpointEvents {
if event.StepID == producer.resolved.StepID && event.LaneID == producer.resolved.ID && event.Stage != string(StageNormalize) {
t.Fatalf("hydrated producer synthesized checkpoint event: %#v", event)
}
}
freshInput, _, _ := handoffFixture(t, value)
freshPrepared := freshInput.Prepared
freshPrepared.Steps[0].lanes[0].typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: value}, nil
}
freshPrepared.Steps[0].lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
return erasedTypedResult{Value: value}, nil
}
freshPrepared.Steps[1].lanes[0].typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: codecScore{Value: 3}}, nil
}
freshLoader := &handoffDependencyLoader{CheckpointLoader: NoopCheckpointLoader()}
freshOutput, err := New().Run(context.Background(), RunInput{Prepared: freshPrepared, RawInput: []byte("input"), Checkpoint: freshLoader})
if err != nil {
t.Fatalf("fresh Run() error = %v", err)
}
hydratedDependencies := loader.extractDeps[consumer.resolved.ID]
if generatedFingerprintCount(hydratedDependencies) != 1 {
t.Fatalf("hydrated consumer dependencies = %#v, want generated producer fingerprint", hydratedDependencies)
}
var matchedFreshDependencies bool
for _, dependencies := range freshLoader.extract {
if reflect.DeepEqual(dependencies, hydratedDependencies) {
matchedFreshDependencies = true
break
}
}
if !matchedFreshDependencies {
t.Fatalf("consumer dependencies differ: fresh %#v hydrated %#v", freshLoader.extract, hydratedDependencies)
}
if !reflect.DeepEqual(freshOutput.Manifest.References, output.Manifest.References) {
t.Fatalf("generated provenance differs: fresh %#v hydrated %#v", freshOutput.Manifest.References, output.Manifest.References)
}
}
func TestRunnerRejectsInvalidRequiredNormalizedArtifactBeforeConsumer(t *testing.T) {
const contentSentinel = "sensitive-campaign-payload-74291"
tests := []struct {
name string
decision CheckpointDecision
mutate func(*CheckpointArtifact)
wantCode CheckpointReasonCode
}{
{"missing", NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing, "checkpoint missing"), nil, CheckpointReasonMissing},
{"rejected status", NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonStatusNotReusable, "status rejected"), nil, CheckpointReasonStatusNotReusable},
{"corrupt payload", NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted artifact reusable"), func(v *CheckpointArtifact) { v.Artifact.Content = []byte(`{"items":[`) }, CheckpointReasonArtifactPayloadInvalid},
{"non canonical", NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted artifact reusable"), func(v *CheckpointArtifact) { v.Artifact.Content = []byte(`{"items": ["stored"]}`) }, CheckpointReasonArtifactNotCanonical},
{"wrong codec identity", NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted artifact reusable"), func(v *CheckpointArtifact) { v.Artifact.Kind = "test/score" }, CheckpointReasonArtifactCodecIncompatible},
{"wrong content digest", NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonArtifactDigestMismatch, "content digest mismatch"), nil, CheckpointReasonArtifactDigestMismatch},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
input, _, _ := handoffFixture(t, codecNotes{Items: []string{contentSentinel}})
prepared := input.Prepared
producer := &prepared.Steps[0].lanes[0]
consumer := &prepared.Steps[1].lanes[0]
doc := prepared.input.(*typedTestInput).doc
stored, err := checkpointArtifact(producer.typed.codec, producer.resolved.ID, producer.resolved.Normalize.Module, doc.ID, codecNotes{Items: []string{contentSentinel}})
if err != nil {
t.Fatal(err)
}
if test.mutate != nil {
test.mutate(&stored)
}
consumerCalls := 0
consumer.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
consumerCalls++
return erasedTypedResult{Value: codecScore{Value: 1}}, nil
}
loader := newAcceptedCheckpointLoader()
producerKey := CheckpointLaneKey(producer.resolved.StepID, producer.resolved.ID)
loader.accepted[producerKey] = NormalizeCheckpoint{Output: stored}
loader.acceptedDecision[producerKey] = test.decision
policy := CheckpointExecutionPolicy{RequireReusableLanes: map[string]struct{}{producerKey: {}}}
output, runErr := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: loader, CheckpointPolicy: policy})
if runErr == nil || !strings.Contains(runErr.Error(), string(test.wantCode)) || consumerCalls != 0 {
t.Fatalf("Run() error = %v consumer calls = %d, want %q before consumer", runErr, consumerCalls, test.wantCode)
}
assertAcceptedNormalizeEvent(t, output.CheckpointEvents, producer.resolved.StepID, producer.resolved.ID, CheckpointDecisionExecuted, test.wantCode)
encoded, err := json.Marshal(struct {
Manifest any
Events any
Error string
}{output.Manifest, output.CheckpointEvents, runErr.Error()})
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(encoded), contentSentinel) || strings.Contains(string(encoded), string(stored.Artifact.Content)) {
t.Fatalf("failed hydration diagnostics leaked artifact content: %s", encoded)
}
})
}
}
func TestForcedRequiredLaneExecutesInsteadOfHydrating(t *testing.T) {
prepared := preparedOrderedPipeline(t, 1,
orderedLaneSpec{id: "unrelated", profile: "score"},
orderedLaneSpec{id: "producer", profile: "notes"},
orderedLaneSpec{id: "consumer", profile: "score"},
)
unrelated := &prepared.Steps[0].lanes[0]
producer := &prepared.Steps[1].lanes[0]
consumer := &prepared.Steps[2].lanes[0]
installGeneratedReferenceTarget(&consumer.resolved.ExtractReferences, StageExtract, consumer, "step-2", "producer")
doc := prepared.input.(*typedTestInput).doc
unrelatedArtifact, err := checkpointArtifact(unrelated.typed.codec, unrelated.resolved.ID, unrelated.resolved.Normalize.Module, doc.ID, codecScore{Value: 9})
if err != nil {
t.Fatal(err)
}
producerArtifact, err := checkpointArtifact(producer.typed.codec, producer.resolved.ID, producer.resolved.Normalize.Module, doc.ID, codecNotes{Items: []string{"stale"}})
if err != nil {
t.Fatal(err)
}
unrelatedCalls, producerCalls := 0, 0
unrelated.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
unrelatedCalls++
return erasedTypedResult{Value: codecScore{Value: 9}}, nil
}
producer.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
producerCalls++
return erasedTypedResult{Value: codecNotes{Items: []string{"fresh"}}}, nil
}
producer.typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
return erasedTypedResult{Value: codecNotes{Items: []string{"fresh"}}}, nil
}
var consumerReferences contracts.ReferenceSet
consumer.typed.extract = func(_ context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
consumerReferences = CloneReferenceSet(request.References)
return erasedTypedResult{Value: codecScore{Value: 1}}, nil
}
loader := newAcceptedCheckpointLoader()
unrelatedKey := CheckpointLaneKey(unrelated.resolved.StepID, unrelated.resolved.ID)
producerKey := CheckpointLaneKey(producer.resolved.StepID, producer.resolved.ID)
consumerKey := CheckpointLaneKey(consumer.resolved.StepID, consumer.resolved.ID)
loader.accepted[unrelatedKey] = NormalizeCheckpoint{Output: unrelatedArtifact}
loader.acceptedDecision[unrelatedKey] = NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted artifact reusable")
loader.accepted[producerKey] = NormalizeCheckpoint{Output: producerArtifact}
loader.acceptedDecision[producerKey] = NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted artifact reusable")
policy := CheckpointExecutionPolicy{
ForcedLanes: map[string]struct{}{producerKey: {}, consumerKey: {}},
RequireReusableLanes: map[string]struct{}{unrelatedKey: {}, producerKey: {}},
}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: loader, CheckpointPolicy: policy})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if producerCalls == 0 || unrelatedCalls != 0 {
t.Fatalf("calls producer=%d unrelated=%d, want forced producer execution and unrelated hydration", producerCalls, unrelatedCalls)
}
if got := string(consumerReferences.Slots["producer-output"].Items[0].Content); !strings.Contains(got, "fresh") || strings.Contains(got, "stale") {
t.Fatalf("forced producer reference = %q, want freshly executed output", got)
}
assertAcceptedNormalizeEvent(t, output.CheckpointEvents, unrelated.resolved.StepID, unrelated.resolved.ID, CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused)
assertAcceptedNormalizeEvent(t, output.CheckpointEvents, producer.resolved.StepID, producer.resolved.ID, CheckpointDecisionForcedRecompute, CheckpointReasonRecomputeStep)
}
func installGeneratedReferenceTarget(target *ResolvedReferenceTarget, stage ModuleStage, consumer *preparedLaneExecutor, producerStep, producerLane string) {
*target = ResolvedReferenceTarget{
Stage: stage,
StepID: consumer.resolved.StepID,
LaneID: consumer.resolved.ID,
Module: consumer.resolved.Extract.Module,
Bindings: []ReferenceBinding{{
Stage: stage,
LaneID: consumer.resolved.ID,
SlotName: "producer-output",
Artifact: &ArtifactReference{Step: producerStep, Lane: producerLane},
}},
ReferenceSet: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
"producer-output": {Slot: contracts.ReferenceSlot{Name: "producer-output", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}, AcceptedMediaTypes: []string{"application/json"}}},
}},
}
}
func assertAcceptedNormalizeEvent(t *testing.T, events []CheckpointEvent, stepID, laneID string, category CheckpointDecisionCategory, code CheckpointReasonCode) {
t.Helper()
for _, event := range events {
if event.Stage == string(StageNormalize) && event.StepID == stepID && event.LaneID == laneID {
if event.Category != category || event.ReasonCode != code {
t.Fatalf("accepted normalize event = %#v, want %q/%q", event, category, code)
}
return
}
}
t.Fatalf("accepted normalize event missing from %#v", events)
}

View File

@@ -109,11 +109,11 @@ func (attemptDebugLLM) CompleteStructured(_ context.Context, request contracts.S
func preparedAttemptDebugPipeline(t *testing.T) *PreparedPipeline { func preparedAttemptDebugPipeline(t *testing.T) *PreparedPipeline {
t.Helper() t.Helper()
prepared := preparedConcurrentPipeline(t, 1) prepared := preparedConcurrentPipeline(t, 1)
prepared.lanes = prepared.lanes[:1] prepared.Steps[0].lanes = prepared.Steps[0].lanes[:1]
prepared.resolved.ArtifactLanes = prepared.resolved.ArtifactLanes[:1] prepared.resolved.Steps[0].ArtifactLanes = prepared.resolved.Steps[0].ArtifactLanes[:1]
prepared.ArtifactLanes = prepared.ArtifactLanes[:1] prepared.Steps[0].ArtifactLanes = prepared.Steps[0].ArtifactLanes[:1]
prepared.lanes[0].mergeValidators = preparedValidatorChain{} prepared.Steps[0].lanes[0].mergeValidators = preparedValidatorChain{}
prepared.lanes[0].normalizeValidators = preparedValidatorChain{} prepared.Steps[0].lanes[0].normalizeValidators = preparedValidatorChain{}
return prepared return prepared
} }
@@ -126,13 +126,13 @@ func TestRunnerWritesAttemptScopedMergeAndNormalizeDebug(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t) prepared := preparedAttemptDebugPipeline(t)
debug := newCapturedDebugRecorder() debug := newCapturedDebugRecorder()
client := WithDebugLLMRecording(attemptDebugLLM{}, debug) client := WithDebugLLMRecording(attemptDebugLLM{}, debug)
prepared.lanes[0].typed.merge = func(ctx context.Context, _ any, _ contracts.TypedMergeRequest[any]) (erasedTypedResult, error) { prepared.Steps[0].lanes[0].typed.merge = func(ctx context.Context, _ any, _ contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
if err := callAttemptDebugLLM(ctx, client, "merge"); err != nil { if err := callAttemptDebugLLM(ctx, client, "merge"); err != nil {
return erasedTypedResult{}, err return erasedTypedResult{}, err
} }
return erasedTypedResult{Value: codecNotes{Items: []string{"merged"}}, Warnings: []contracts.Warning{{Scope: "merge", ReasonCode: "observed", Message: "merge warning"}}}, nil return erasedTypedResult{Value: codecNotes{Items: []string{"merged"}}, Warnings: []contracts.Warning{{Scope: "merge", ReasonCode: "observed", Message: "merge warning"}}}, nil
} }
prepared.lanes[0].typed.normalize = func(ctx context.Context, _ any, _ contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) { prepared.Steps[0].lanes[0].typed.normalize = func(ctx context.Context, _ any, _ contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
if err := callAttemptDebugLLM(ctx, client, "normalize"); err != nil { if err := callAttemptDebugLLM(ctx, client, "normalize"); err != nil {
return erasedTypedResult{}, err return erasedTypedResult{}, err
} }
@@ -187,7 +187,7 @@ func TestRunnerRecordsDistinctRetryAttemptsAndPromotesAcceptedWarningsOnly(t *te
prepared := preparedAttemptDebugPipeline(t) prepared := preparedAttemptDebugPipeline(t)
debug := newCapturedDebugRecorder() debug := newCapturedDebugRecorder()
client := WithDebugLLMRecording(attemptDebugLLM{}, debug) client := WithDebugLLMRecording(attemptDebugLLM{}, debug)
lane := &prepared.lanes[0] lane := &prepared.Steps[0].lanes[0]
attempts := 0 attempts := 0
operation := func(ctx context.Context) (erasedTypedResult, error) { operation := func(ctx context.Context) (erasedTypedResult, error) {
attempts++ attempts++
@@ -257,7 +257,7 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
{ {
name: "merge module error", name: "merge module error",
configure: func(prepared *PreparedPipeline) { configure: func(prepared *PreparedPipeline) {
prepared.lanes[0].typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) { prepared.Steps[0].lanes[0].typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
return erasedTypedResult{}, errors.New("merge exploded") return erasedTypedResult{}, errors.New("merge exploded")
} }
}, },
@@ -267,7 +267,7 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
{ {
name: "normalize validator error", name: "normalize validator error",
configure: func(prepared *PreparedPipeline) { configure: func(prepared *PreparedPipeline) {
prepared.lanes[0].normalizeValidators.validators = []preparedValidator{{ prepared.Steps[0].lanes[0].normalizeValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("error-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"}, resolved: ResolvedValidator{Binding: Binding("error-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) { typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{}, errors.New("validator exploded") return contracts.ValidationResult{}, errors.New("validator exploded")
@@ -281,7 +281,7 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
{ {
name: "merge final rejection", name: "merge final rejection",
configure: func(prepared *PreparedPipeline) { configure: func(prepared *PreparedPipeline) {
prepared.lanes[0].mergeValidators.validators = []preparedValidator{{ prepared.Steps[0].lanes[0].mergeValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("reject-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"}, resolved: ResolvedValidator{Binding: Binding("reject-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) { typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted"}, nil return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "not accepted"}, nil
@@ -294,7 +294,7 @@ func TestRunnerAttemptDebugRepresentsFailuresAndFinalRejection(t *testing.T) {
{ {
name: "normalize serialization error", name: "normalize serialization error",
configure: func(prepared *PreparedPipeline) { configure: func(prepared *PreparedPipeline) {
prepared.lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) { prepared.Steps[0].lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
return erasedTypedResult{Value: "wrong artifact type"}, nil return erasedTypedResult{Value: "wrong artifact type"}, nil
} }
}, },
@@ -331,13 +331,13 @@ func TestRunnerKeepsValidatorLLMCallsOutOfModuleAttempt(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t) prepared := preparedAttemptDebugPipeline(t)
debug := newCapturedDebugRecorder() debug := newCapturedDebugRecorder()
client := WithDebugLLMRecording(attemptDebugLLM{}, debug) client := WithDebugLLMRecording(attemptDebugLLM{}, debug)
prepared.lanes[0].typed.merge = func(ctx context.Context, _ any, _ contracts.TypedMergeRequest[any]) (erasedTypedResult, error) { prepared.Steps[0].lanes[0].typed.merge = func(ctx context.Context, _ any, _ contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
if err := callAttemptDebugLLM(ctx, client, "merge-module"); err != nil { if err := callAttemptDebugLLM(ctx, client, "merge-module"); err != nil {
return erasedTypedResult{}, err return erasedTypedResult{}, err
} }
return erasedTypedResult{Value: codecNotes{Items: []string{"merged"}}}, nil return erasedTypedResult{Value: codecNotes{Items: []string{"merged"}}}, nil
} }
prepared.lanes[0].mergeValidators.validators = []preparedValidator{{ prepared.Steps[0].lanes[0].mergeValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("llm-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"}, resolved: ResolvedValidator{Binding: Binding("llm-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(ctx context.Context, _ any, _ typedValidationTarget) (contracts.ValidationResult, error) { typedValidate: func(ctx context.Context, _ any, _ typedValidationTarget) (contracts.ValidationResult, error) {
if err := callAttemptDebugLLM(ctx, client, "merge-validator"); err != nil { if err := callAttemptDebugLLM(ctx, client, "merge-validator"); err != nil {
@@ -389,7 +389,7 @@ func (l attemptReuseLoader) Normalize(laneID, _ string, _ []CheckpointFingerprin
func TestRunnerCheckpointReuseDoesNotSynthesizeModuleAttempts(t *testing.T) { func TestRunnerCheckpointReuseDoesNotSynthesizeModuleAttempts(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t) prepared := preparedAttemptDebugPipeline(t)
lane := prepared.lanes[0] lane := prepared.Steps[0].lanes[0]
merge, err := checkpointArtifact(lane.typed.codec, lane.resolved.ID, lane.resolved.Merge.Module, "source", codecNotes{Items: []string{"merged"}}) merge, err := checkpointArtifact(lane.typed.codec, lane.resolved.ID, lane.resolved.Merge.Module, "source", codecNotes{Items: []string{"merged"}})
if err != nil { if err != nil {
t.Fatalf("checkpointArtifact(merge): %v", err) t.Fatalf("checkpointArtifact(merge): %v", err)

View File

@@ -104,11 +104,11 @@ func installObservedNotesCodec(t *testing.T, prepared *PreparedPipeline, codec *
if err != nil { if err != nil {
t.Fatalf("codec entry error = %v", err) t.Fatalf("codec entry error = %v", err)
} }
prepared.lanes[0].typed.codec = entry prepared.Steps[0].lanes[0].typed.codec = entry
} }
func configureCandidateOperation(prepared *PreparedPipeline, target ModuleStage, value codecNotes) { func configureCandidateOperation(prepared *PreparedPipeline, target ModuleStage, value codecNotes) {
lane := &prepared.lanes[0] lane := &prepared.Steps[0].lanes[0]
switch target { switch target {
case StageMerge: case StageMerge:
lane.typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) { lane.typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
@@ -136,9 +136,9 @@ func setCandidateValidator(prepared *PreparedPipeline, target ModuleStage, appro
} }
switch target { switch target {
case StageMerge: case StageMerge:
prepared.lanes[0].mergeValidators.validators = []preparedValidator{validator} prepared.Steps[0].lanes[0].mergeValidators.validators = []preparedValidator{validator}
case StageNormalize: case StageNormalize:
prepared.lanes[0].normalizeValidators.validators = []preparedValidator{validator} prepared.Steps[0].lanes[0].normalizeValidators.validators = []preparedValidator{validator}
} }
} }

View File

@@ -469,9 +469,9 @@ func TestRunnerOmitsProducerProfileForDeterministicChunker(t *testing.T) {
func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) { func TestRunnerRefreshChangesDownstreamChunkFingerprint(t *testing.T) {
doc := typedTestDocumentWithUnits(2) doc := typedTestDocumentWithUnits(2)
prepared := preparedConcurrentPipeline(t, 1) prepared := preparedConcurrentPipeline(t, 1)
prepared.lanes = prepared.lanes[:1] prepared.Steps[0].lanes = prepared.Steps[0].lanes[:1]
prepared.resolved.ArtifactLanes = prepared.resolved.ArtifactLanes[:1] prepared.resolved.Steps[0].ArtifactLanes = prepared.resolved.Steps[0].ArtifactLanes[:1]
prepared.ArtifactLanes = prepared.ArtifactLanes[:1] prepared.Steps[0].ArtifactLanes = prepared.Steps[0].ArtifactLanes[:1]
prepared.input = &typedTestInput{key: prepared.resolved.Input.Module, doc: doc} prepared.input = &typedTestInput{key: prepared.resolved.Input.Module, doc: doc}
run := func(plan source.ChunkPlan) []CheckpointFingerprint { run := func(plan source.ChunkPlan) []CheckpointFingerprint {

View File

@@ -11,6 +11,7 @@ import (
"testing" "testing"
"time" "time"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm" frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
) )
@@ -60,8 +61,184 @@ func preparedConcurrentPipeline(t *testing.T, chunkCount int) *PreparedPipeline
return prepared return prepared
} }
type orderedLaneSpec struct {
id string
profile string
}
func preparedOrderedPipeline(t *testing.T, chunkCount int, specs ...orderedLaneSpec) *PreparedPipeline {
t.Helper()
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
profile := typedResolutionProfile()
profile.Artifacts = nil
for index, spec := range specs {
lane, ok := typedResolutionProfile().Artifacts[spec.profile]
if !ok {
t.Fatalf("typed lane profile %q is not defined", spec.profile)
}
profile.Steps = append(profile.Steps, PipelineStepProfile{
ID: fmt.Sprintf("step-%d", index+1),
Artifacts: map[string]ArtifactLaneProfile{spec.id: lane},
})
}
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v", err)
}
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v", err)
}
doc := typedTestDocumentWithUnits(chunkCount)
if adapter, ok := prepared.input.(*typedTestInput); ok {
adapter.doc = doc
} else {
t.Fatalf("prepared input = %T, want *typedTestInput", prepared.input)
}
prepared.chunker = &typedTestChunker{key: "typed/chunk", plan: typedTestPlan(doc)}
return prepared
}
type countingOrderedOutput struct {
calls atomic.Int32
}
func (o *countingOrderedOutput) Key() string { return "typed/output" }
func (o *countingOrderedOutput) Encode(context.Context, contracts.OutputRequest) (contracts.OutputResult, error) {
o.calls.Add(1)
return contracts.OutputResult{}, nil
}
type countingInputAdapter struct {
contracts.InputAdapter
calls atomic.Int32
}
func (a *countingInputAdapter) Parse(ctx context.Context, request contracts.ParseRequest) (*source.SourceDocument, error) {
a.calls.Add(1)
return a.InputAdapter.Parse(ctx, request)
}
func TestRunnerFailsGeneratedHandoffBeforeConsumerExtraction(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 1)
input := &countingInputAdapter{InputAdapter: prepared.input}
prepared.input = input
preparedLane := &prepared.Steps[0].lanes[0]
lane := &preparedLane.resolved
binding := ReferenceBinding{
Stage: StageExtract,
LaneID: lane.ID,
SlotName: "generated",
Artifact: &ArtifactReference{Step: "producer", Lane: "source"},
}
lane.ExtractReferences.Bindings = []ReferenceBinding{binding}
lane.ExtractReferences.ReferenceSet = contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
"generated": {Slot: contracts.ReferenceSlot{Name: "generated", AcceptedArtifactKinds: []contracts.ArtifactKind{"test/notes"}}},
}}
prepared.resolved.Steps[0].ArtifactLanes[0].ExtractReferences = CloneReferenceTarget(lane.ExtractReferences)
var extractCalls atomic.Int32
preparedLane.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
extractCalls.Add(1)
return erasedTypedResult{Value: codecNotes{}}, nil
}
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input")})
if err == nil || !strings.Contains(err.Error(), "generated dependency") {
t.Fatalf("Run() error = %v, want generated dependency failure", err)
}
if got := input.calls.Load(); got != 1 {
t.Fatalf("input Parse calls = %d, want one before handoff", got)
}
if got := extractCalls.Load(); got != 0 {
t.Fatalf("consumer extract calls = %d, want zero", got)
}
}
func TestRunnerExecutesOrderedStepsWithHardBarriers(t *testing.T) {
prepared := preparedOrderedPipeline(t, 1,
orderedLaneSpec{id: "notes", profile: "notes"},
orderedLaneSpec{id: "score", profile: "score"},
)
var mu sync.Mutex
var events []string
record := func(event string) {
mu.Lock()
events = append(events, event)
mu.Unlock()
}
first := &prepared.Steps[0].lanes[0]
second := &prepared.Steps[1].lanes[0]
first.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: codecNotes{Items: []string{"first"}}}, nil
}
originalFirstNormalize := first.typed.normalize
first.typed.normalize = func(ctx context.Context, implementation any, request contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
result, err := originalFirstNormalize(ctx, implementation, request)
if err == nil {
record("first-normalize-done")
}
return result, err
}
second.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
record("second-extract-started")
return erasedTypedResult{Value: codecScore{Value: 2}}, nil
}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
mu.Lock()
gotEvents := append([]string(nil), events...)
mu.Unlock()
if want := []string{"first-normalize-done", "second-extract-started"}; !reflect.DeepEqual(gotEvents, want) {
t.Fatalf("ordered events = %#v, want %#v", gotEvents, want)
}
if got := []string{output.NormalizeOutputs[0].LaneID, output.NormalizeOutputs[1].LaneID}; !reflect.DeepEqual(got, []string{"notes", "score"}) {
t.Fatalf("normalized lane order = %#v, want notes then score", got)
}
}
func TestRunnerStopsLaterOrderedStepsAfterFailure(t *testing.T) {
prepared := preparedOrderedPipeline(t, 1,
orderedLaneSpec{id: "first", profile: "notes"},
orderedLaneSpec{id: "failure", profile: "score"},
orderedLaneSpec{id: "later", profile: "notes"},
)
first := &prepared.Steps[0].lanes[0]
second := &prepared.Steps[1].lanes[0]
third := &prepared.Steps[2].lanes[0]
first.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: codecNotes{Items: []string{"first"}}}, nil
}
second.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{}, errors.New("ordered step extraction failed")
}
var thirdCalls atomic.Int32
third.typed.extract = func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
thirdCalls.Add(1)
return erasedTypedResult{Value: codecNotes{Items: []string{"later"}}}, nil
}
encoder := &countingOrderedOutput{}
prepared.output = encoder
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
if err == nil || !strings.Contains(err.Error(), `execute pipeline step "step-2"`) {
t.Fatalf("Run() error = %v, want step-scoped failure", err)
}
if len(output.NormalizeOutputs) != 1 || output.NormalizeOutputs[0].LaneID != "first" {
t.Fatalf("completed outputs = %#v, want only the first step output", output.NormalizeOutputs)
}
if got := thirdCalls.Load(); got != 0 {
t.Fatalf("later step extract calls = %d, want zero", got)
}
if got := encoder.calls.Load(); got != 0 {
t.Fatalf("output encoder calls = %d, want zero after failure", got)
}
}
func installExtractOperation(prepared *PreparedPipeline, laneIndex int, operation func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error)) { func installExtractOperation(prepared *PreparedPipeline, laneIndex int, operation func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error)) {
prepared.lanes[laneIndex].typed.extract = func(ctx context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) { prepared.Steps[0].lanes[laneIndex].typed.extract = func(ctx context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return operation(ctx, request) return operation(ctx, request)
} }
} }
@@ -82,7 +259,7 @@ func TestRunnerBoundsExtractJobsAndStabilizesReverseCompletion(t *testing.T) {
} }
var active atomic.Int32 var active atomic.Int32
var maximum atomic.Int32 var maximum atomic.Int32
for laneIndex := range prepared.lanes { for laneIndex := range prepared.Steps[0].lanes {
lane := laneIndex lane := laneIndex
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) { installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
jobIndex := request.Chunk.Index*2 + lane jobIndex := request.Chunk.Index*2 + lane
@@ -154,8 +331,8 @@ func TestRunnerStartsLaneContinuationWhileOtherLaneExtractsRemain(t *testing.T)
return erasedTypedResult{}, ctx.Err() return erasedTypedResult{}, ctx.Err()
} }
}) })
originalMerge := prepared.lanes[0].typed.merge originalMerge := prepared.Steps[0].lanes[0].typed.merge
prepared.lanes[0].typed.merge = func(ctx context.Context, implementation any, request contracts.TypedMergeRequest[any]) (erasedTypedResult, error) { prepared.Steps[0].lanes[0].typed.merge = func(ctx context.Context, implementation any, request contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
select { select {
case mergeStarted <- struct{}{}: case mergeStarted <- struct{}{}:
default: default:
@@ -180,7 +357,7 @@ func TestRunnerStartsLaneContinuationWhileOtherLaneExtractsRemain(t *testing.T)
func TestRunnerBoundsConcurrentLaneContinuations(t *testing.T) { func TestRunnerBoundsConcurrentLaneContinuations(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 1) prepared := preparedConcurrentPipeline(t, 1)
base := prepared.lanes[0] base := prepared.Steps[0].lanes[0]
lanes := make([]preparedLaneExecutor, 8) lanes := make([]preparedLaneExecutor, 8)
resolvedLanes := make([]ResolvedArtifactLane, len(lanes)) resolvedLanes := make([]ResolvedArtifactLane, len(lanes))
publicLanes := make([]PreparedArtifactLane, len(lanes)) publicLanes := make([]PreparedArtifactLane, len(lanes))
@@ -215,9 +392,9 @@ func TestRunnerBoundsConcurrentLaneContinuations(t *testing.T) {
resolvedLanes[i] = lane.resolved resolvedLanes[i] = lane.resolved
publicLanes[i] = PreparedArtifactLane{Resolved: lane.resolved} publicLanes[i] = PreparedArtifactLane{Resolved: lane.resolved}
} }
prepared.lanes = lanes prepared.Steps[0].lanes = lanes
prepared.resolved.ArtifactLanes = resolvedLanes prepared.resolved.Steps[0].ArtifactLanes = resolvedLanes
prepared.ArtifactLanes = publicLanes prepared.Steps[0].ArtifactLanes = publicLanes
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2}) output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
if err != nil { if err != nil {
@@ -235,7 +412,7 @@ func TestRunnerSelectsFrameworkErrorByStableLaneOrder(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 1) prepared := preparedConcurrentPipeline(t, 1)
ready := make(chan struct{}, 2) ready := make(chan struct{}, 2)
release := make(chan struct{}) release := make(chan struct{})
for laneIndex := range prepared.lanes { for laneIndex := range prepared.Steps[0].lanes {
lane := laneIndex lane := laneIndex
installExtractOperation(prepared, lane, func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error) { installExtractOperation(prepared, lane, func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
ready <- struct{}{} ready <- struct{}{}
@@ -258,7 +435,7 @@ func TestRunnerSelectsFrameworkErrorByStableLaneOrder(t *testing.T) {
func TestRunnerSelectsMergeErrorBeforeEarlierLaneNormalizeError(t *testing.T) { func TestRunnerSelectsMergeErrorBeforeEarlierLaneNormalizeError(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 1) prepared := preparedConcurrentPipeline(t, 1)
for laneIndex := range prepared.lanes { for laneIndex := range prepared.Steps[0].lanes {
lane := laneIndex lane := laneIndex
installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) { installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
@@ -266,12 +443,12 @@ func TestRunnerSelectsMergeErrorBeforeEarlierLaneNormalizeError(t *testing.T) {
} }
ready := make(chan struct{}, 2) ready := make(chan struct{}, 2)
release := make(chan struct{}) release := make(chan struct{})
prepared.lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) { prepared.Steps[0].lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
ready <- struct{}{} ready <- struct{}{}
<-release <-release
return erasedTypedResult{}, errors.New("earlier lane normalize failure") return erasedTypedResult{}, errors.New("earlier lane normalize failure")
} }
prepared.lanes[1].typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) { prepared.Steps[0].lanes[1].typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
ready <- struct{}{} ready <- struct{}{}
<-release <-release
return erasedTypedResult{}, errors.New("later lane merge failure") return erasedTypedResult{}, errors.New("later lane merge failure")
@@ -291,13 +468,13 @@ func TestRunnerSelectsMergeErrorBeforeEarlierLaneNormalizeError(t *testing.T) {
func TestRunnerKeepsSuccessfulChunksWhenAnotherExtractIsRejected(t *testing.T) { func TestRunnerKeepsSuccessfulChunksWhenAnotherExtractIsRejected(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 2) prepared := preparedConcurrentPipeline(t, 2)
for laneIndex := range prepared.lanes { for laneIndex := range prepared.Steps[0].lanes {
lane := laneIndex lane := laneIndex
installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) { installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
}) })
} }
validator := &prepared.lanes[0].extractValidators.validators[0] validator := &prepared.Steps[0].lanes[0].extractValidators.validators[0]
validator.typedValidate = func(_ context.Context, _ any, target typedValidationTarget) (contracts.ValidationResult, error) { validator.typedValidate = func(_ context.Context, _ any, target typedValidationTarget) (contracts.ValidationResult, error) {
if target.chunk != nil && target.chunk.Index == 0 { if target.chunk != nil && target.chunk.Index == 0 {
return contracts.ValidationResult{Approved: false, ReasonCode: "expected_rejection", Message: "rejected by test"}, nil return contracts.ValidationResult{Approved: false, ReasonCode: "expected_rejection", Message: "rejected by test"}, nil
@@ -308,7 +485,7 @@ func TestRunnerKeepsSuccessfulChunksWhenAnotherExtractIsRejected(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("Run() error = %v", err) t.Fatalf("Run() error = %v", err)
} }
if len(output.Rejected) != 1 || output.Rejected[0].LaneID != prepared.lanes[0].resolved.ID || output.Rejected[0].ChunkIndex != 0 { if len(output.Rejected) != 1 || output.Rejected[0].LaneID != prepared.Steps[0].lanes[0].resolved.ID || output.Rejected[0].ChunkIndex != 0 {
t.Fatalf("rejections = %#v, want the first lane's first chunk", output.Rejected) t.Fatalf("rejections = %#v, want the first lane's first chunk", output.Rejected)
} }
if len(output.NormalizeOutputs) != 2 { if len(output.NormalizeOutputs) != 2 {
@@ -330,9 +507,9 @@ func TestRunnerSeparatelyBoundsExtractJobsAndSharedProviderCalls(t *testing.T) {
var jobActive atomic.Int32 var jobActive atomic.Int32
var jobMaximum atomic.Int32 var jobMaximum atomic.Int32
var attempts sync.Map var attempts sync.Map
for laneIndex := range prepared.lanes { for laneIndex := range prepared.Steps[0].lanes {
lane := laneIndex lane := laneIndex
prepared.lanes[lane].resolved.Extract.Retries = 1 prepared.Steps[0].lanes[lane].resolved.Extract.Retries = 1
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) { installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
current := jobActive.Add(1) current := jobActive.Add(1)
defer jobActive.Add(-1) defer jobActive.Add(-1)
@@ -353,7 +530,7 @@ func TestRunnerSeparatelyBoundsExtractJobsAndSharedProviderCalls(t *testing.T) {
} }
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
}) })
validator := &prepared.lanes[lane].extractValidators.validators[0] validator := &prepared.Steps[0].lanes[lane].extractValidators.validators[0]
validator.typedValidate = func(ctx context.Context, _ any, _ typedValidationTarget) (contracts.ValidationResult, error) { validator.typedValidate = func(ctx context.Context, _ any, _ typedValidationTarget) (contracts.ValidationResult, error) {
if _, callErr := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{StageName: "validate"}, nil); callErr != nil { if _, callErr := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{StageName: "validate"}, nil); callErr != nil {
return contracts.ValidationResult{}, callErr return contracts.ValidationResult{}, callErr
@@ -382,7 +559,7 @@ func TestRunnerSeparatelyBoundsExtractJobsAndSharedProviderCalls(t *testing.T) {
func TestRunnerReturnsParentCancellationAndStopsQueuedExtracts(t *testing.T) { func TestRunnerReturnsParentCancellationAndStopsQueuedExtracts(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 4) prepared := preparedConcurrentPipeline(t, 4)
started := make(chan struct{}, 8) started := make(chan struct{}, 8)
for laneIndex := range prepared.lanes { for laneIndex := range prepared.Steps[0].lanes {
lane := laneIndex lane := laneIndex
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) { installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
started <- struct{}{} started <- struct{}{}

View File

@@ -26,6 +26,8 @@ type laneExtractState struct {
results map[int]extractJobResult results map[int]extractJobResult
remaining int remaining int
failed bool failed bool
terminal bool
output RunOutput
} }
type finalizedExtractResults struct { type finalizedExtractResults struct {
@@ -36,12 +38,15 @@ type finalizedExtractResults struct {
decision CheckpointDecision decision CheckpointDecision
} }
func loadExtract(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) { func loadExtract(loader CheckpointLoader, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
if stepAware, ok := loader.(StepCheckpointLoader); ok {
return stepAware.ExtractForStep(stepID, laneID, moduleKey, deps)
}
return loader.Extract(laneID, moduleKey, deps) return loader.Extract(laneID, moduleKey, deps)
} }
func recordExtract(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error { func recordExtract(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
return recorder.ExtractSucceeded(laneID, moduleKey, deps, outputs, rejected, warnings) return checkpointExtractSucceeded(recorder, stepID, laneID, moduleKey, deps, outputs, rejected, warnings)
} }
type extractJob struct { type extractJob struct {
@@ -72,30 +77,56 @@ type orderedRunError struct {
err error err error
} }
func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk) (RunOutput, error) { func (r *Runner) runLanes(parent context.Context, input RunInput, step PreparedPipelineStep, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk) (RunOutput, error) {
output := RunOutput{Manifest: manifestFromPipeline(input)} output := RunOutput{Manifest: manifestFromPipeline(input)}
states := make([]*laneExtractState, len(input.Prepared.lanes)) states, err := initializeLaneStates(input, step, checkpoints, loader, doc, chunks, &output)
for i, prepared := range input.Prepared.lanes { if err != nil {
return output, err
}
completedOutputs, runErrors := r.runLaneEngine(parent, input, checkpoints, loader, doc, sourceInput, sessionID, chunks, states)
if err := mergeCompletedLanes(&output, completedOutputs); err != nil {
return output, err
}
if err := selectRunError(parent, runErrors); err != nil {
return output, err
}
return output, nil
}
func initializeLaneStates(input RunInput, step PreparedPipelineStep, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, chunks []source.Chunk, output *RunOutput) ([]*laneExtractState, error) {
states := make([]*laneExtractState, len(step.lanes))
for i, prepared := range step.lanes {
if prepared.typed == nil { if prepared.typed == nil {
return output, fmt.Errorf("typed lane %q executor is not prepared", prepared.resolved.ID) return nil, fmt.Errorf("typed lane %q executor is not prepared", prepared.resolved.ID)
} }
if err := setTypedLaneManifestMetadata(&output, prepared.resolved.ID, prepared.typed.extractor, prepared.typed.merger, prepared.typed.normalizer); err != nil { if err := setTypedLaneManifestMetadata(output, prepared.resolved.ID, prepared.typed.extractor, prepared.typed.merger, prepared.typed.normalizer); err != nil {
return output, err return nil, err
} }
state, err := prepareLaneExtract(input, loader, doc, chunks, i, prepared) if input.CheckpointPolicy.requiresReusable(input.stepID, prepared.resolved.ID) && !input.CheckpointPolicy.forced(input.stepID, prepared.resolved.ID) {
state, err := hydrateRequiredLane(input, loader, doc, i, prepared, output)
if err != nil {
return nil, err
}
states[i] = state
continue
}
state, err := prepareLaneExtract(input, loader, doc, chunks, i, prepared, output)
if err != nil { if err != nil {
return output, err return nil, err
} }
if !state.decision.Reused { if !state.decision.Reused {
if err := checkpoints.ExtractRunning(prepared.resolved.ID, prepared.resolved.Extract.Module, state.deps); err != nil { if err := checkpointExtractRunning(checkpoints, input.stepID, prepared.resolved.ID, prepared.resolved.Extract.Module, state.deps); err != nil {
return output, fmt.Errorf("write extract checkpoint for lane %q: %w", prepared.resolved.ID, err) return nil, fmt.Errorf("write extract checkpoint for lane %q: %w", prepared.resolved.ID, err)
} }
} else if err := finalizeLaneExtract(checkpoints, state); err != nil { } else if err := finalizeLaneExtract(checkpoints, input.stepID, state); err != nil {
return output, err return nil, err
} }
states[i] = state states[i] = state
} }
return states, nil
}
func (r *Runner) runLaneEngine(parent context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk, states []*laneExtractState) ([]RunOutput, []orderedRunError) {
workerCount := input.ExtractWorkers workerCount := input.ExtractWorkers
if workerCount < 1 { if workerCount < 1 {
workerCount = 1 workerCount = 1
@@ -126,7 +157,7 @@ func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints Ch
for chunkIndex := range chunks { for chunkIndex := range chunks {
for laneIndex := range states { for laneIndex := range states {
state := states[laneIndex] state := states[laneIndex]
if state.decision.Reused { if state.terminal || state.decision.Reused {
continue continue
} }
select { select {
@@ -154,16 +185,24 @@ func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints Ch
}() }()
} }
completedOutputs, runErrors := collectLaneResults(ctx, cancel, input, checkpoints, chunks, states, results, completions, continuations)
close(continuations)
continuationWorkers.Wait()
return completedOutputs, runErrors
}
func collectLaneResults(ctx context.Context, cancel context.CancelFunc, input RunInput, checkpoints CheckpointRecorder, chunks []source.Chunk, states []*laneExtractState, results <-chan extractJobResult, completions <-chan laneCompletion, continuations chan<- *laneExtractState) ([]RunOutput, []orderedRunError) {
completedOutputs := make([]RunOutput, len(states)) completedOutputs := make([]RunOutput, len(states))
var runErrors []orderedRunError var runErrors []orderedRunError
var pendingContinuations []*laneExtractState var pendingContinuations []*laneExtractState
launched, completed := 0, 0 launched, completed := 0, 0
for _, state := range states { for _, state := range states {
if state.decision.Reused { if state.terminal {
completedOutputs[state.index] = state.output
} else if state.decision.Reused {
pendingContinuations = append(pendingContinuations, state) pendingContinuations = append(pendingContinuations, state)
} }
} }
resultChannel := results resultChannel := results
for resultChannel != nil || len(pendingContinuations) > 0 || completed < launched { for resultChannel != nil || len(pendingContinuations) > 0 || completed < launched {
var continuationChannel chan<- *laneExtractState var continuationChannel chan<- *laneExtractState
@@ -188,13 +227,13 @@ func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints Ch
if result.err != nil { if result.err != nil {
state.failed = true state.failed = true
runErrors = append(runErrors, orderedRunError{stage: 0, lane: state.index, chunk: result.chunkIndex, err: result.err}) runErrors = append(runErrors, orderedRunError{stage: 0, lane: state.index, chunk: result.chunkIndex, err: result.err})
_ = checkpoints.ExtractFailed(state.prepared.resolved.ID, state.prepared.resolved.Extract.Module, state.deps, result.err) _ = checkpointExtractFailed(checkpoints, input.stepID, state.prepared.resolved.ID, state.prepared.resolved.Extract.Module, state.deps, result.err)
cancel() cancel()
} else { } else {
state.results[result.chunkIndex] = result state.results[result.chunkIndex] = result
} }
if state.remaining == 0 && !state.failed && ctx.Err() == nil { if state.remaining == 0 && !state.failed && ctx.Err() == nil {
if err := finalizeLaneExtract(checkpoints, state); err != nil { if err := finalizeLaneExtract(checkpoints, input.stepID, state); err != nil {
state.failed = true state.failed = true
runErrors = append(runErrors, orderedRunError{stage: 0, lane: state.index, chunk: len(chunks), err: err}) runErrors = append(runErrors, orderedRunError{stage: 0, lane: state.index, chunk: len(chunks), err: err})
cancel() cancel()
@@ -211,47 +250,73 @@ func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints Ch
} }
} }
} }
close(continuations) return completedOutputs, runErrors
continuationWorkers.Wait()
for i := range completedOutputs {
if err := mergeLaneOutput(&output, completedOutputs[i]); err != nil {
return output, err
}
}
if err := selectRunError(parent, runErrors); err != nil {
return output, err
}
return output, nil
} }
func prepareLaneExtract(input RunInput, loader CheckpointLoader, doc *source.SourceDocument, chunks []source.Chunk, index int, prepared preparedLaneExecutor) (*laneExtractState, error) { func hydrateRequiredLane(input RunInput, loader CheckpointLoader, doc *source.SourceDocument, index int, prepared preparedLaneExecutor, output *RunOutput) (*laneExtractState, error) {
lane, typed := prepared.resolved, prepared.typed
local := RunOutput{Manifest: manifestFromPipeline(input)}
checkpoint, decision := loader.AcceptedNormalize(input.stepID, lane.ID, lane.Normalize.Module)
if decision.Reused {
decision = checkpointDecision(CheckpointDecisionReused, CheckpointReasonAcceptedArtifactReused, "accepted normalized artifact is reusable")
if checkpoint.Output.LaneID != lane.ID || checkpoint.Output.ModuleKey != lane.Normalize.Module || checkpoint.Output.SourceID != doc.ID {
decision = checkpointDecision(CheckpointDecisionExecuted, CheckpointReasonArtifactPayloadInvalid, "accepted normalized artifact provenance does not match the producer lane")
}
}
decision, err := resolveCheckpointDecision(&local, loader, input.CheckpointPolicy, StageNormalize, input.stepID, lane.ID, lane.Normalize.Module, decision, typed.codec, []CheckpointArtifact{checkpoint.Output})
if err != nil {
if mergeErr := mergeLaneOutput(output, local); mergeErr != nil {
return nil, mergeErr
}
return nil, err
}
_, hydrated, err := decodeCanonicalCheckpointArtifact(typed.codec, checkpoint.Output)
if err != nil {
return nil, fmt.Errorf("hydrate accepted normalized artifact for step %q lane %q: %w", input.stepID, lane.ID, err)
}
local.Warnings = append(local.Warnings, cloneWarnings(checkpoint.Warnings)...)
local.NormalizeOutputs = append(local.NormalizeOutputs, contracts.SerializedOutput{
StepID: input.stepID,
LaneID: lane.ID,
NormalizerKey: lane.Normalize.Module,
SourceID: doc.ID,
Artifact: contracts.CloneSerializedArtifact(hydrated.Artifact),
})
return &laneExtractState{index: index, prepared: prepared, decision: decision, terminal: true, output: local}, nil
}
func mergeCompletedLanes(output *RunOutput, completedOutputs []RunOutput) error {
for i := range completedOutputs {
if err := mergeLaneOutput(output, completedOutputs[i]); err != nil {
return err
}
}
return nil
}
func prepareLaneExtract(input RunInput, loader CheckpointLoader, doc *source.SourceDocument, chunks []source.Chunk, index int, prepared preparedLaneExecutor, output *RunOutput) (*laneExtractState, error) {
lane, typed := prepared.resolved, prepared.typed lane, typed := prepared.resolved, prepared.typed
digest, err := joinedChunkDigest(chunks) digest, err := joinedChunkDigest(chunks)
if err != nil { if err != nil {
return nil, fmt.Errorf("digest chunks for lane %q: %w", lane.ID, err) return nil, fmt.Errorf("digest chunks for lane %q: %w", lane.ID, err)
} }
state := &laneExtractState{index: index, prepared: prepared, deps: digestFingerprints("chunks", digest), remaining: len(chunks), results: make(map[int]extractJobResult, len(chunks))} extractReferences := operationReferenceSet(input, lane.ExtractReferences)
cp, decision := loadExtract(loader, lane.ID, lane.Extract.Module, state.deps) deps := append(digestFingerprints("chunks", digest), generatedReferenceDependencies(extractReferences)...)
if decision.Reused { state := &laneExtractState{index: index, prepared: prepared, deps: normalizeCheckpointFingerprints(deps), remaining: len(chunks), results: make(map[int]extractJobResult, len(chunks))}
for _, stored := range cp.Outputs { cp, decision := loadExtract(loader, input.stepID, lane.ID, lane.Extract.Module, state.deps)
if _, decodeErr := decodeCheckpointArtifact(typed.codec, stored); decodeErr != nil { decision, err = resolveCheckpointDecision(output, loader, input.CheckpointPolicy, StageExtract, input.stepID, lane.ID, lane.Extract.Module, decision, typed.codec, cp.Outputs)
decision = CheckpointDecision{Reason: "extract artifact checkpoint codec is incompatible: " + decodeErr.Error()} if err != nil {
break return nil, err
}
}
} }
state.decision = decision state.decision = decision
if decision.Reused { if decision.Reused {
state.remaining = 0 state.remaining = 0
for _, stored := range cp.Outputs { for _, stored := range cp.Outputs {
value, decodeErr := decodeCheckpointArtifact(typed.codec, stored) value, hydrated, decodeErr := decodeCanonicalCheckpointArtifact(typed.codec, stored)
if decodeErr != nil { if decodeErr != nil {
return nil, fmt.Errorf("decode extract checkpoint for lane %q: %w", lane.ID, decodeErr) return nil, fmt.Errorf("decode extract checkpoint for lane %q: %w", lane.ID, decodeErr)
} }
stored, decodeErr = hydrateCheckpointArtifact(typed.codec, stored, value) stored = hydrated
if decodeErr != nil {
return nil, fmt.Errorf("hydrate extract checkpoint for lane %q: %w", lane.ID, decodeErr)
}
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: stored.ChunkID, ChunkIndex: stored.ChunkIndex, ChunkRef: stored.ChunkRef, Value: value} artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: stored.ChunkID, ChunkIndex: stored.ChunkIndex, ChunkRef: stored.ChunkRef, Value: value}
if stored.ChunkIndex >= 0 && stored.ChunkIndex < len(chunks) && artifact.ChunkRef == (source.SourceRef{}) { if stored.ChunkIndex >= 0 && stored.ChunkIndex < len(chunks) && artifact.ChunkRef == (source.SourceRef{}) {
artifact.ChunkRef = chunks[stored.ChunkIndex].Ref artifact.ChunkRef = chunks[stored.ChunkIndex].Ref
@@ -280,12 +345,13 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
started := time.Now().UTC() started := time.Now().UTC()
attemptPath := path.Join("extract", debugPathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), fmt.Sprintf("attempt-%02d", attempt)) attemptPath := path.Join("extract", debugPathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), fmt.Sprintf("attempt-%02d", attempt))
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath) attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "extract", llmScope, debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started}) terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "extract", llmScope, debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started})
requestMetadata, metadataErr := cloneMetadata(input.Metadata) requestMetadata, metadataErr := cloneMetadata(input.Metadata)
if metadataErr != nil { if metadataErr != nil {
return false, nil, terminal.record(nil, fmt.Errorf("clone extract request metadata: %w", metadataErr)) return false, nil, terminal.record(nil, fmt.Errorf("clone extract request metadata: %w", metadataErr))
} }
extracted, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(lane.ExtractReferences.ReferenceSet), LLMProfile: lane.Extract.LLMProfile, Metadata: requestMetadata}) extractReferences := operationReferenceSet(input, lane.ExtractReferences)
extracted, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(extractReferences), LLMProfile: lane.Extract.LLMProfile, Metadata: requestMetadata})
if callErr != nil { if callErr != nil {
attemptErr := fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, lane.Extract.Module, callErr) attemptErr := fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, lane.Extract.Module, callErr)
return false, nil, terminal.record(nil, attemptErr) return false, nil, terminal.record(nil, attemptErr)
@@ -299,7 +365,7 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
return false, nil, terminal.record(payload, attemptErr) return false, nil, terminal.record(payload, attemptErr)
} }
serializedCandidate.ChunkID, serializedCandidate.ChunkIndex, serializedCandidate.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef serializedCandidate.ChunkID, serializedCandidate.ChunkIndex, serializedCandidate.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageExtract, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: lane.ExtractReferences.ReferenceSet, metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: extracted.Value, candidate: &serializedCandidate}, state.prepared.extractValidators, attempt, input.Debug) warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageExtract, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: extractReferences, metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: extracted.Value, candidate: &serializedCandidate}, state.prepared.extractValidators, attempt, input.Debug)
attemptWarnings = append(attemptWarnings, warnings...) attemptWarnings = append(attemptWarnings, warnings...)
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)} payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
if validateErr != nil || rejected != nil { if validateErr != nil || rejected != nil {
@@ -327,7 +393,7 @@ func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.
return result return result
} }
func finalizeLaneExtract(checkpoints CheckpointRecorder, state *laneExtractState) error { func finalizeLaneExtract(checkpoints CheckpointRecorder, stepID string, state *laneExtractState) error {
lane := state.prepared.resolved lane := state.prepared.resolved
indexes := make([]int, 0, len(state.results)) indexes := make([]int, 0, len(state.results))
for index := range state.results { for index := range state.results {
@@ -348,7 +414,7 @@ func finalizeLaneExtract(checkpoints CheckpointRecorder, state *laneExtractState
sort.SliceStable(state.serialized, func(i, j int) bool { return state.serialized[i].ChunkIndex < state.serialized[j].ChunkIndex }) sort.SliceStable(state.serialized, func(i, j int) bool { return state.serialized[i].ChunkIndex < state.serialized[j].ChunkIndex })
sort.SliceStable(state.rejected, func(i, j int) bool { return state.rejected[i].ChunkIndex < state.rejected[j].ChunkIndex }) sort.SliceStable(state.rejected, func(i, j int) bool { return state.rejected[i].ChunkIndex < state.rejected[j].ChunkIndex })
if !state.decision.Reused { if !state.decision.Reused {
if err := recordExtract(checkpoints, lane.ID, lane.Extract.Module, state.deps, state.serialized, state.rejected, state.warnings); err != nil { if err := recordExtract(checkpoints, stepID, lane.ID, lane.Extract.Module, state.deps, state.serialized, state.rejected, state.warnings); err != nil {
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err) return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
} }
} }
@@ -367,11 +433,10 @@ func (r *Runner) continueLane(ctx context.Context, input RunInput, checkpoints C
} }
local.Warnings = append(local.Warnings, cloneWarnings(results.warnings)...) local.Warnings = append(local.Warnings, cloneWarnings(results.warnings)...)
local.Rejected = append(local.Rejected, cloneRejectedOutputs(results.rejected)...) local.Rejected = append(local.Rejected, cloneRejectedOutputs(results.rejected)...)
recordCheckpointEvent(&local, loader, string(StageExtract), lane.ID, lane.Extract.Module, results.decision) if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "decision": results.decision, "source": debugSourceDocumentEnvelope(doc), "chunks": debugSourceChunkEnvelopes(chunks), "options": redactSensitiveMap(lane.Extract.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "decision": results.decision, "source": debugSourceDocumentEnvelope(doc), "chunks": debugSourceChunkEnvelopes(chunks), "options": redactSensitiveMap(lane.Extract.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
return local, &laneRunError{stage: StageExtract, err: err} return local, &laneRunError{stage: StageExtract, err: err}
} }
if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "outputs": debugCheckpointArtifacts(results.serialized), "rejected": debugRejectedOutputEnvelopes(results.rejected), "warnings": debugWarningEnvelopes(results.warnings)}}); err != nil { if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": results.decision.Reused, "outputs": debugCheckpointArtifacts(results.serialized), "rejected": debugRejectedOutputEnvelopes(results.rejected), "warnings": debugWarningEnvelopes(results.warnings)}}); err != nil {
return local, &laneRunError{stage: StageExtract, err: err} return local, &laneRunError{stage: StageExtract, err: err}
} }
if len(results.accepted) == 0 { if len(results.accepted) == 0 {
@@ -440,7 +505,7 @@ func mergeLaneOutput(dst *RunOutput, src RunOutput) error {
dst.CheckpointEvents = append(dst.CheckpointEvents, src.CheckpointEvents...) dst.CheckpointEvents = append(dst.CheckpointEvents, src.CheckpointEvents...)
for i := range dst.Manifest.ArtifactLanes { for i := range dst.Manifest.ArtifactLanes {
for j := range src.Manifest.ArtifactLanes { for j := range src.Manifest.ArtifactLanes {
if dst.Manifest.ArtifactLanes[i].ID == src.Manifest.ArtifactLanes[j].ID && src.Manifest.ArtifactLanes[j].Metadata != nil { if dst.Manifest.ArtifactLanes[i].ID == src.Manifest.ArtifactLanes[j].ID && dst.Manifest.ArtifactLanes[i].StepID == src.Manifest.ArtifactLanes[j].StepID && src.Manifest.ArtifactLanes[j].Metadata != nil {
metadata, err := cloneMetadata(src.Manifest.ArtifactLanes[j].Metadata) metadata, err := cloneMetadata(src.Manifest.ArtifactLanes[j].Metadata)
if err != nil { if err != nil {
return fmt.Errorf("clone lane %q manifest metadata: %w", dst.Manifest.ArtifactLanes[i].ID, err) return fmt.Errorf("clone lane %q manifest metadata: %w", dst.Manifest.ArtifactLanes[i].ID, err)

View File

@@ -212,7 +212,7 @@ func TestRunnerContinuesFromFreshAndReusedExtractResults(t *testing.T) {
func TestRunnerPromotesOnlyAcceptedExtractRetryWarnings(t *testing.T) { func TestRunnerPromotesOnlyAcceptedExtractRetryWarnings(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t) prepared := preparedAttemptDebugPipeline(t)
prepared.lanes[0].resolved.Extract.Retries = 1 prepared.Steps[0].lanes[0].resolved.Extract.Retries = 1
attempts := 0 attempts := 0
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) { installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
attempts++ attempts++
@@ -226,7 +226,7 @@ func TestRunnerPromotesOnlyAcceptedExtractRetryWarnings(t *testing.T) {
}, nil }, nil
}) })
validatorCalls := 0 validatorCalls := 0
prepared.lanes[0].extractValidators.validators[0].typedValidate = func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) { prepared.Steps[0].lanes[0].extractValidators.validators[0].typedValidate = func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
validatorCalls++ validatorCalls++
return contracts.ValidationResult{Approved: validatorCalls == 2, ReasonCode: "retry", Message: "retry extract"}, nil return contracts.ValidationResult{Approved: validatorCalls == 2, ReasonCode: "retry", Message: "retry extract"}, nil
} }
@@ -252,7 +252,7 @@ func TestRunnerPromotesOnlyAcceptedExtractRetryWarnings(t *testing.T) {
} }
} }
func assertExtractDecision(t *testing.T, events []CheckpointEvent, action string, reason string) { func assertExtractDecision(t *testing.T, events []CheckpointEvent, action CheckpointDecisionCategory, reason string) {
t.Helper() t.Helper()
for _, event := range events { for _, event := range events {
if event.Stage == string(StageExtract) { if event.Stage == string(StageExtract) {

View File

@@ -220,7 +220,7 @@ func TestRunnerRecordsExtractTerminalOutcomes(t *testing.T) {
} }
return erasedTypedResult{Value: codecNotes{Items: []string{value}}, Warnings: []contracts.Warning{{Scope: "extract", ReasonCode: "observed", Message: "extract warning"}}}, nil return erasedTypedResult{Value: codecNotes{Items: []string{value}}, Warnings: []contracts.Warning{{Scope: "extract", ReasonCode: "observed", Message: "extract warning"}}}, nil
}) })
prepared.lanes[0].extractValidators.validators = []preparedValidator{{ prepared.Steps[0].lanes[0].extractValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("terminal/extract-validator"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"}, resolved: ResolvedValidator{Binding: Binding("terminal/extract-validator"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) { typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: !tc.reject, ReasonCode: "extract_rejected", Message: "extract rejected"}, tc.validatorErr return contracts.ValidationResult{Approved: !tc.reject, ReasonCode: "extract_rejected", Message: "extract rejected"}, tc.validatorErr
@@ -315,7 +315,7 @@ func TestRunnerKeepsExtractModuleAndValidatorLLMCallsIsolated(t *testing.T) {
} }
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
}) })
prepared.lanes[0].extractValidators.validators = []preparedValidator{{ prepared.Steps[0].lanes[0].extractValidators.validators = []preparedValidator{{
resolved: ResolvedValidator{Binding: Binding("llm-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"}, resolved: ResolvedValidator{Binding: Binding("llm-check"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(ctx context.Context, _ any, _ typedValidationTarget) (contracts.ValidationResult, error) { typedValidate: func(ctx context.Context, _ any, _ typedValidationTarget) (contracts.ValidationResult, error) {
if err := callAttemptDebugLLM(ctx, client, "extract-validator"); err != nil { if err := callAttemptDebugLLM(ctx, client, "extract-validator"); err != nil {

View File

@@ -1,6 +1,7 @@
package pipeline package pipeline
import ( import (
"bytes"
"context" "context"
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
@@ -13,17 +14,23 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
) )
func loadMerge(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) { func loadMerge(loader CheckpointLoader, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
if stepAware, ok := loader.(StepCheckpointLoader); ok {
return stepAware.MergeForStep(stepID, laneID, moduleKey, deps)
}
return loader.Merge(laneID, moduleKey, deps) return loader.Merge(laneID, moduleKey, deps)
} }
func loadNormalize(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) { func loadNormalize(loader CheckpointLoader, stepID, laneID, moduleKey string, deps []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
if stepAware, ok := loader.(StepCheckpointLoader); ok {
return stepAware.NormalizeForStep(stepID, laneID, moduleKey, deps)
}
return loader.Normalize(laneID, moduleKey, deps) return loader.Normalize(laneID, moduleKey, deps)
} }
func recordMerge(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error { func recordMerge(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
return recorder.MergeSucceeded(laneID, moduleKey, deps, output, warnings) return checkpointMergeSucceeded(recorder, stepID, laneID, moduleKey, deps, output, warnings)
} }
func recordNormalize(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error { func recordNormalize(recorder CheckpointRecorder, stepID, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
return recorder.NormalizeSucceeded(laneID, moduleKey, deps, output, warnings) return checkpointNormalizeSucceeded(recorder, stepID, laneID, moduleKey, deps, output, warnings)
} }
func cloneCheckpointArtifact(output CheckpointArtifact) CheckpointArtifact { func cloneCheckpointArtifact(output CheckpointArtifact) CheckpointArtifact {
output.Artifact = contracts.CloneSerializedArtifact(output.Artifact) output.Artifact = contracts.CloneSerializedArtifact(output.Artifact)
@@ -94,21 +101,64 @@ func serializeArtifact(codec artifactCodecEntry, value any, candidate bool) (con
return contracts.SerializedArtifact{Kind: codec.spec.Kind, Schema: contracts.CloneArtifactSchema(schema), MediaType: codec.spec.MediaType, Content: append([]byte(nil), content...), Metadata: metadata}, nil return contracts.SerializedArtifact{Kind: codec.spec.Kind, Schema: contracts.CloneArtifactSchema(schema), MediaType: codec.spec.MediaType, Content: append([]byte(nil), content...), Metadata: metadata}, nil
} }
type checkpointArtifactValidationError struct {
code CheckpointReasonCode
err error
}
func (e *checkpointArtifactValidationError) Error() string { return e.err.Error() }
func (e *checkpointArtifactValidationError) Unwrap() error { return e.err }
func checkpointArtifactValidationFailure(code CheckpointReasonCode, format string, args ...any) error {
return &checkpointArtifactValidationError{code: code, err: fmt.Errorf(format, args...)}
}
func checkpointArtifactReasonCode(err error) CheckpointReasonCode {
var validationErr *checkpointArtifactValidationError
if errors.As(err, &validationErr) {
return validationErr.code
}
return CheckpointReasonArtifactPayloadInvalid
}
func decodeCheckpointArtifact(codec artifactCodecEntry, artifact CheckpointArtifact) (any, error) { func decodeCheckpointArtifact(codec artifactCodecEntry, artifact CheckpointArtifact) (any, error) {
expectedDigest := contracts.DigestArtifactSchema(codec.spec.Schema) expectedDigest := contracts.DigestArtifactSchema(codec.spec.Schema)
if artifact.Artifact.Kind != codec.spec.Kind { if artifact.Artifact.Kind != codec.spec.Kind {
return nil, fmt.Errorf("artifact kind %q does not match codec %q", artifact.Artifact.Kind, codec.spec.Kind) return nil, checkpointArtifactValidationFailure(CheckpointReasonArtifactCodecIncompatible, "artifact kind %q does not match codec %q", artifact.Artifact.Kind, codec.spec.Kind)
} }
if artifact.Artifact.Schema.ID != codec.spec.Schema.ID || artifact.Artifact.Schema.Version != codec.spec.Schema.Version { if artifact.Artifact.Schema.ID != codec.spec.Schema.ID || artifact.Artifact.Schema.Name != codec.spec.Schema.Name || artifact.Artifact.Schema.Version != codec.spec.Schema.Version {
return nil, fmt.Errorf("artifact schema %q version %q does not match codec schema %q version %q", artifact.Artifact.Schema.ID, artifact.Artifact.Schema.Version, codec.spec.Schema.ID, codec.spec.Schema.Version) return nil, checkpointArtifactValidationFailure(CheckpointReasonArtifactCodecIncompatible, "artifact schema %q version %q does not match codec schema %q version %q", artifact.Artifact.Schema.ID, artifact.Artifact.Schema.Version, codec.spec.Schema.ID, codec.spec.Schema.Version)
} }
if artifact.SchemaDigest != expectedDigest { if artifact.SchemaDigest != expectedDigest {
return nil, fmt.Errorf("artifact schema digest %q does not match codec schema digest %q", artifact.SchemaDigest, expectedDigest) return nil, checkpointArtifactValidationFailure(CheckpointReasonArtifactCodecIncompatible, "artifact schema digest %q does not match codec schema digest %q", artifact.SchemaDigest, expectedDigest)
} }
if artifact.Artifact.MediaType != codec.spec.MediaType { if artifact.Artifact.MediaType != codec.spec.MediaType {
return nil, fmt.Errorf("artifact media type %q does not match codec media type %q", artifact.Artifact.MediaType, codec.spec.MediaType) return nil, checkpointArtifactValidationFailure(CheckpointReasonArtifactCodecIncompatible, "artifact media type %q does not match codec media type %q", artifact.Artifact.MediaType, codec.spec.MediaType)
} }
return codec.decode(append([]byte(nil), artifact.Artifact.Content...)) value, err := codec.decode(append([]byte(nil), artifact.Artifact.Content...))
if err != nil {
return nil, checkpointArtifactValidationFailure(CheckpointReasonArtifactPayloadInvalid, "decode artifact payload: %w", err)
}
return value, nil
}
func decodeCanonicalCheckpointArtifact(codec artifactCodecEntry, artifact CheckpointArtifact) (any, CheckpointArtifact, error) {
value, err := decodeCheckpointArtifact(codec, artifact)
if err != nil {
return nil, CheckpointArtifact{}, err
}
canonical, err := serializeArtifact(codec, value, false)
if err != nil {
return nil, CheckpointArtifact{}, checkpointArtifactValidationFailure(CheckpointReasonArtifactPayloadInvalid, "encode canonical artifact: %w", err)
}
if canonical.Kind != artifact.Artifact.Kind || canonical.Schema.ID != artifact.Artifact.Schema.ID || canonical.Schema.Name != artifact.Artifact.Schema.Name || canonical.Schema.Version != artifact.Artifact.Schema.Version || canonical.MediaType != artifact.Artifact.MediaType || !bytes.Equal(canonical.Content, artifact.Artifact.Content) || checkpointContentDigest(canonical.Content) != checkpointContentDigest(artifact.Artifact.Content) {
return nil, CheckpointArtifact{}, checkpointArtifactValidationFailure(CheckpointReasonArtifactNotCanonical, "stored artifact is not canonical")
}
hydrated, err := hydrateCheckpointArtifact(codec, cloneCheckpointArtifact(artifact), value)
if err != nil {
return nil, CheckpointArtifact{}, checkpointArtifactValidationFailure(CheckpointReasonArtifactPayloadInvalid, "hydrate checkpoint artifact: %w", err)
}
return value, hydrated, nil
} }
func checkpointArtifact(codec artifactCodecEntry, laneID, moduleKey, sourceID string, value any) (CheckpointArtifact, error) { func checkpointArtifact(codec artifactCodecEntry, laneID, moduleKey, sourceID string, value any) (CheckpointArtifact, error) {
@@ -135,13 +185,13 @@ type laneRunError struct {
func (e *laneRunError) Error() string { return e.err.Error() } func (e *laneRunError) Error() string { return e.err.Error() }
func (e *laneRunError) Unwrap() error { return e.err } func (e *laneRunError) Unwrap() error { return e.err }
func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, extracts finalizedExtractResults, output *RunOutput) (err error) { type mergeStageResult struct {
activeStage := StageMerge artifact erasedMergeArtifact
defer func() { serialized CheckpointArtifact
if err != nil { terminal bool
err = &laneRunError{stage: activeStage, err: err} }
}
}() func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, extracts finalizedExtractResults, output *RunOutput) error {
lane, typed := prepared.resolved, prepared.typed lane, typed := prepared.resolved, prepared.typed
if typed == nil { if typed == nil {
return fmt.Errorf("typed lane %q executor is not prepared", lane.ID) return fmt.Errorf("typed lane %q executor is not prepared", lane.ID)
@@ -149,51 +199,67 @@ func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoi
if err := setTypedLaneManifestMetadata(output, lane.ID, typed.extractor, typed.merger, typed.normalizer); err != nil { if err := setTypedLaneManifestMetadata(output, lane.ID, typed.extractor, typed.merger, typed.normalizer); err != nil {
return err return err
} }
merged, err := r.runMergeStage(ctx, input, checkpoints, loader, doc, sourceInput, sessionID, prepared, extracts, output)
if err != nil {
return &laneRunError{stage: StageMerge, err: err}
}
if merged.terminal {
return nil
}
normalized, err := r.runNormalizeStage(ctx, input, checkpoints, loader, doc, sourceInput, sessionID, prepared, merged.artifact, merged.serialized, output)
if err != nil {
return &laneRunError{stage: StageNormalize, err: err}
}
if normalized.accepted {
output.NormalizeOutputs = append(output.NormalizeOutputs, contracts.SerializedOutput{StepID: input.stepID, LaneID: lane.ID, NormalizerKey: lane.Normalize.Module, SourceID: doc.ID, Artifact: contracts.CloneSerializedArtifact(normalized.serialized.Artifact)})
}
return nil
}
func (r *Runner) runMergeStage(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, extracts finalizedExtractResults, output *RunOutput) (mergeStageResult, error) {
var stageResult mergeStageResult
lane, typed := prepared.resolved, prepared.typed
mergeInputs := make([]contracts.ExtractArtifact[any], len(extracts.accepted)) mergeInputs := make([]contracts.ExtractArtifact[any], len(extracts.accepted))
for i, value := range extracts.accepted { for i, value := range extracts.accepted {
mergeInputs[i] = contracts.ExtractArtifact[any]{LaneID: value.LaneID, ExtractorKey: value.ExtractorKey, SourceID: value.SourceID, ChunkID: value.ChunkID, ChunkIndex: value.ChunkIndex, ChunkRef: value.ChunkRef, Value: value.Value} mergeInputs[i] = contracts.ExtractArtifact[any]{LaneID: value.LaneID, ExtractorKey: value.ExtractorKey, SourceID: value.SourceID, ChunkID: value.ChunkID, ChunkIndex: value.ChunkIndex, ChunkRef: value.ChunkRef, Value: value.Value}
} }
mergeDeps := artifactCheckpointDigests(extracts.serialized) mergeReferences := operationReferenceSet(input, lane.MergeReferences)
mergeCP, mergeDecision := loadMerge(loader, lane.ID, lane.Merge.Module, mergeDeps) mergeDeps := append(artifactCheckpointDigests(extracts.serialized), generatedReferenceDependencies(mergeReferences)...)
if mergeDecision.Reused { mergeDeps = normalizeCheckpointFingerprints(mergeDeps)
if _, decodeErr := decodeCheckpointArtifact(typed.codec, mergeCP.Output); decodeErr != nil { mergeCP, mergeDecision := loadMerge(loader, input.stepID, lane.ID, lane.Merge.Module, mergeDeps)
mergeDecision = CheckpointDecision{Reason: "merge artifact checkpoint codec is incompatible: " + decodeErr.Error()} mergeDecision, err := resolveCheckpointDecision(output, loader, input.CheckpointPolicy, StageMerge, input.stepID, lane.ID, lane.Merge.Module, mergeDecision, typed.codec, []CheckpointArtifact{mergeCP.Output})
} if err != nil {
return stageResult, err
} }
recordCheckpointEvent(output, loader, string(StageMerge), lane.ID, lane.Merge.Module, mergeDecision) if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "decision": mergeDecision, "source": debugSourceDocumentEnvelope(doc), "extract_outputs": debugCheckpointArtifacts(extracts.serialized), "options": redactSensitiveMap(lane.Merge.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageMerge), LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "decision": mergeDecision, "source": debugSourceDocumentEnvelope(doc), "extract_outputs": debugCheckpointArtifacts(extracts.serialized), "options": redactSensitiveMap(lane.Merge.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil { return stageResult, err
return err
} }
var merged erasedMergeArtifact var merged erasedMergeArtifact
var serializedMerge CheckpointArtifact var serializedMerge CheckpointArtifact
var mergeWarnings []contracts.Warning var mergeWarnings []contracts.Warning
if mergeDecision.Reused { if mergeDecision.Reused {
value, decodeErr := decodeCheckpointArtifact(typed.codec, mergeCP.Output) value, hydrated, decodeErr := decodeCanonicalCheckpointArtifact(typed.codec, mergeCP.Output)
if decodeErr != nil { if decodeErr != nil {
return fmt.Errorf("decode merge checkpoint for lane %q: %w", lane.ID, decodeErr) return stageResult, fmt.Errorf("decode merge checkpoint for lane %q: %w", lane.ID, decodeErr)
} }
merged = erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: value} merged = erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: value}
serializedMerge, decodeErr = hydrateCheckpointArtifact(typed.codec, cloneCheckpointArtifact(mergeCP.Output), value) serializedMerge = hydrated
if decodeErr != nil {
return fmt.Errorf("hydrate merge checkpoint for lane %q: %w", lane.ID, decodeErr)
}
mergeWarnings = cloneWarnings(mergeCP.Warnings) mergeWarnings = cloneWarnings(mergeCP.Warnings)
output.Warnings = append(output.Warnings, mergeWarnings...) output.Warnings = append(output.Warnings, mergeWarnings...)
} else { } else {
if err := checkpoints.MergeRunning(lane.ID, lane.Merge.Module, mergeDeps); err != nil { if err := checkpointMergeRunning(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps); err != nil {
return err return stageResult, err
} }
ok, rejection, runErr := runWithRetry(ctx, lane.Merge.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) { ok, rejection, runErr := runWithRetry(ctx, lane.Merge.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
started := time.Now().UTC() started := time.Now().UTC()
attemptPath := path.Join("merge", debugPathComponent(lane.ID), fmt.Sprintf("attempt-%02d", attempt)) attemptPath := path.Join("merge", debugPathComponent(lane.ID), fmt.Sprintf("attempt-%02d", attempt))
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath) attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "merge", llmScope, debugTimedEnvelope{Stage: string(StageMerge), LaneID: lane.ID, ModuleKey: lane.Merge.Module, Attempt: attempt, StartedAt: started}) terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "merge", llmScope, debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, Attempt: attempt, StartedAt: started})
requestMetadata, metadataErr := cloneMetadata(input.Metadata) requestMetadata, metadataErr := cloneMetadata(input.Metadata)
if metadataErr != nil { if metadataErr != nil {
return false, nil, terminal.record(nil, fmt.Errorf("clone merge request metadata: %w", metadataErr)) return false, nil, terminal.record(nil, fmt.Errorf("clone merge request metadata: %w", metadataErr))
} }
result, callErr := typed.merge(attemptCtx, typed.merger, contracts.TypedMergeRequest[any]{Source: doc, LaneID: lane.ID, ExtractOutputs: mergeInputs, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(lane.MergeReferences.ReferenceSet), LLMProfile: lane.Merge.LLMProfile, Metadata: requestMetadata}) result, callErr := typed.merge(attemptCtx, typed.merger, contracts.TypedMergeRequest[any]{Source: doc, LaneID: lane.ID, ExtractOutputs: mergeInputs, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(mergeReferences), LLMProfile: lane.Merge.LLMProfile, Metadata: requestMetadata})
if callErr != nil { if callErr != nil {
attemptErr := fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, lane.Merge.Module, callErr) attemptErr := fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, lane.Merge.Module, callErr)
return false, nil, terminal.record(nil, attemptErr) return false, nil, terminal.record(nil, attemptErr)
@@ -205,7 +271,7 @@ func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoi
attemptErr := fmt.Errorf("serialize merge candidate for lane %q: %w", lane.ID, encodeErr) attemptErr := fmt.Errorf("serialize merge candidate for lane %q: %w", lane.ID, encodeErr)
return false, nil, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}, attemptErr) return false, nil, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}, attemptErr)
} }
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageMerge, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: lane.MergeReferences.ReferenceSet, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.mergeValidators, attempt, input.Debug) warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageMerge, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: mergeReferences, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.mergeValidators, attempt, input.Debug)
attemptWarnings = append(attemptWarnings, warnings...) attemptWarnings = append(attemptWarnings, warnings...)
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)} payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
if validateErr != nil || rejected != nil { if validateErr != nil || rejected != nil {
@@ -224,64 +290,74 @@ func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoi
return true, nil, nil return true, nil, nil
}) })
if runErr != nil { if runErr != nil {
_ = checkpoints.MergeFailed(lane.ID, lane.Merge.Module, mergeDeps, runErr) _ = checkpointMergeFailed(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, runErr)
return runErr return stageResult, runErr
} }
if !ok { if !ok {
output.Rejected = append(output.Rejected, *rejection) output.Rejected = append(output.Rejected, *rejection)
if err := checkpoints.MergeRejected(lane.ID, lane.Merge.Module, mergeDeps, *rejection); err != nil { if err := checkpointMergeRejected(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, *rejection); err != nil {
return err return stageResult, err
} }
return nil stageResult.terminal = true
return stageResult, nil
} }
output.Warnings = append(output.Warnings, mergeWarnings...) output.Warnings = append(output.Warnings, mergeWarnings...)
if err := recordMerge(checkpoints, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge, mergeWarnings); err != nil { if err := recordMerge(checkpoints, input.stepID, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge, mergeWarnings); err != nil {
return err return stageResult, err
} }
} }
if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageMerge), LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedMerge), "warnings": debugWarningEnvelopes(mergeWarnings)}}); err != nil { if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageMerge), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedMerge), "warnings": debugWarningEnvelopes(mergeWarnings)}}); err != nil {
return err return stageResult, err
} }
stageResult.artifact = merged
stageResult.serialized = serializedMerge
return stageResult, nil
}
activeStage = StageNormalize type normalizeStageResult struct {
normalizeDeps := artifactCheckpointDigests([]CheckpointArtifact{serializedMerge}) serialized CheckpointArtifact
normalizeCP, normalizeDecision := loadNormalize(loader, lane.ID, lane.Normalize.Module, normalizeDeps) warnings []contracts.Warning
if normalizeDecision.Reused { accepted bool
if _, decodeErr := decodeCheckpointArtifact(typed.codec, normalizeCP.Output); decodeErr != nil { }
normalizeDecision = CheckpointDecision{Reason: "normalize artifact checkpoint codec is incompatible: " + decodeErr.Error()}
} func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, prepared preparedLaneExecutor, merged erasedMergeArtifact, serializedMerge CheckpointArtifact, output *RunOutput) (normalizeStageResult, error) {
var stageResult normalizeStageResult
lane, typed := prepared.resolved, prepared.typed
normalizeReferences := operationReferenceSet(input, lane.NormalizeReferences)
normalizeDeps := append(artifactCheckpointDigests([]CheckpointArtifact{serializedMerge}), generatedReferenceDependencies(normalizeReferences)...)
normalizeDeps = normalizeCheckpointFingerprints(normalizeDeps)
normalizeCP, normalizeDecision := loadNormalize(loader, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps)
normalizeDecision, err := resolveCheckpointDecision(output, loader, input.CheckpointPolicy, StageNormalize, input.stepID, lane.ID, lane.Normalize.Module, normalizeDecision, typed.codec, []CheckpointArtifact{normalizeCP.Output})
if err != nil {
return stageResult, err
} }
recordCheckpointEvent(output, loader, string(StageNormalize), lane.ID, lane.Normalize.Module, normalizeDecision) if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "decision": normalizeDecision, "source": debugSourceDocumentEnvelope(doc), "merge_output": debugCheckpointArtifact(serializedMerge), "options": redactSensitiveMap(lane.Normalize.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageNormalize), LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "decision": normalizeDecision, "source": debugSourceDocumentEnvelope(doc), "merge_output": debugCheckpointArtifact(serializedMerge), "options": redactSensitiveMap(lane.Normalize.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil { return stageResult, err
return err
} }
var serializedNormalize CheckpointArtifact var serializedNormalize CheckpointArtifact
var normalizeWarnings []contracts.Warning var normalizeWarnings []contracts.Warning
if normalizeDecision.Reused { if normalizeDecision.Reused {
value, decodeErr := decodeCheckpointArtifact(typed.codec, normalizeCP.Output) _, hydrated, decodeErr := decodeCanonicalCheckpointArtifact(typed.codec, normalizeCP.Output)
if decodeErr != nil { if decodeErr != nil {
return fmt.Errorf("decode normalize checkpoint for lane %q: %w", lane.ID, decodeErr) return stageResult, fmt.Errorf("decode normalize checkpoint for lane %q: %w", lane.ID, decodeErr)
}
serializedNormalize, decodeErr = hydrateCheckpointArtifact(typed.codec, cloneCheckpointArtifact(normalizeCP.Output), value)
if decodeErr != nil {
return fmt.Errorf("hydrate normalize checkpoint for lane %q: %w", lane.ID, decodeErr)
} }
serializedNormalize = hydrated
normalizeWarnings = cloneWarnings(normalizeCP.Warnings) normalizeWarnings = cloneWarnings(normalizeCP.Warnings)
output.Warnings = append(output.Warnings, normalizeWarnings...) output.Warnings = append(output.Warnings, normalizeWarnings...)
} else { } else {
if err := checkpoints.NormalizeRunning(lane.ID, lane.Normalize.Module, normalizeDeps); err != nil { if err := checkpointNormalizeRunning(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps); err != nil {
return err return stageResult, err
} }
ok, rejection, runErr := runWithRetry(ctx, lane.Normalize.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) { ok, rejection, runErr := runWithRetry(ctx, lane.Normalize.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
started := time.Now().UTC() started := time.Now().UTC()
attemptPath := path.Join("normalize", debugPathComponent(lane.ID), fmt.Sprintf("attempt-%02d", attempt)) attemptPath := path.Join("normalize", debugPathComponent(lane.ID), fmt.Sprintf("attempt-%02d", attempt))
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath) attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "normalize", llmScope, debugTimedEnvelope{Stage: string(StageNormalize), LaneID: lane.ID, ModuleKey: lane.Normalize.Module, Attempt: attempt, StartedAt: started}) terminal := newAttemptTerminalRecorder(input.Debug, attemptPath, "normalize", llmScope, debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, Attempt: attempt, StartedAt: started})
requestMetadata, metadataErr := cloneMetadata(input.Metadata) requestMetadata, metadataErr := cloneMetadata(input.Metadata)
if metadataErr != nil { if metadataErr != nil {
return false, nil, terminal.record(nil, fmt.Errorf("clone normalize request metadata: %w", metadataErr)) return false, nil, terminal.record(nil, fmt.Errorf("clone normalize request metadata: %w", metadataErr))
} }
result, callErr := typed.normalize(attemptCtx, typed.normalizer, contracts.TypedNormalizeRequest[any]{Source: doc, LaneID: lane.ID, MergeOutput: contracts.MergeArtifact[any]{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: merged.Value}, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(lane.NormalizeReferences.ReferenceSet), LLMProfile: lane.Normalize.LLMProfile, Metadata: requestMetadata}) result, callErr := typed.normalize(attemptCtx, typed.normalizer, contracts.TypedNormalizeRequest[any]{Source: doc, LaneID: lane.ID, MergeOutput: contracts.MergeArtifact[any]{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: merged.Value}, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(normalizeReferences), LLMProfile: lane.Normalize.LLMProfile, Metadata: requestMetadata})
if callErr != nil { if callErr != nil {
attemptErr := fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, lane.Normalize.Module, callErr) attemptErr := fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, lane.Normalize.Module, callErr)
return false, nil, terminal.record(nil, attemptErr) return false, nil, terminal.record(nil, attemptErr)
@@ -292,7 +368,7 @@ func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoi
attemptErr := fmt.Errorf("serialize normalize candidate for lane %q: %w", lane.ID, encodeErr) attemptErr := fmt.Errorf("serialize normalize candidate for lane %q: %w", lane.ID, encodeErr)
return false, nil, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}, attemptErr) return false, nil, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}, attemptErr)
} }
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageNormalize, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: lane.NormalizeReferences.ReferenceSet, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.normalizeValidators, attempt, input.Debug) warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageNormalize, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: normalizeReferences, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.normalizeValidators, attempt, input.Debug)
attemptWarnings = append(attemptWarnings, warnings...) attemptWarnings = append(attemptWarnings, warnings...)
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)} payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
if validateErr != nil || rejected != nil { if validateErr != nil || rejected != nil {
@@ -311,26 +387,28 @@ func (r *Runner) continueTypedLane(ctx context.Context, input RunInput, checkpoi
return true, nil, nil return true, nil, nil
}) })
if runErr != nil { if runErr != nil {
_ = checkpoints.NormalizeFailed(lane.ID, lane.Normalize.Module, normalizeDeps, runErr) _ = checkpointNormalizeFailed(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, runErr)
return runErr return stageResult, runErr
} }
if !ok { if !ok {
output.Rejected = append(output.Rejected, *rejection) output.Rejected = append(output.Rejected, *rejection)
if err := checkpoints.NormalizeRejected(lane.ID, lane.Normalize.Module, normalizeDeps, *rejection); err != nil { if err := checkpointNormalizeRejected(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, *rejection); err != nil {
return err return stageResult, err
} }
return nil return stageResult, nil
} }
output.Warnings = append(output.Warnings, normalizeWarnings...) output.Warnings = append(output.Warnings, normalizeWarnings...)
if err := recordNormalize(checkpoints, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize, normalizeWarnings); err != nil { if err := recordNormalize(checkpoints, input.stepID, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize, normalizeWarnings); err != nil {
return err return stageResult, err
} }
} }
if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageNormalize), LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedNormalize), "warnings": debugWarningEnvelopes(normalizeWarnings)}}); err != nil { if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageNormalize), StepID: input.stepID, LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedNormalize), "warnings": debugWarningEnvelopes(normalizeWarnings)}}); err != nil {
return err return stageResult, err
} }
output.NormalizeOutputs = append(output.NormalizeOutputs, contracts.SerializedOutput{LaneID: lane.ID, NormalizerKey: lane.Normalize.Module, SourceID: doc.ID, Artifact: contracts.CloneSerializedArtifact(serializedNormalize.Artifact)}) stageResult.serialized = serializedNormalize
return nil stageResult.warnings = normalizeWarnings
stageResult.accepted = true
return stageResult, nil
} }
func setTypedLaneManifestMetadata(output *RunOutput, laneID string, extractor, merger, normalizer any) error { func setTypedLaneManifestMetadata(output *RunOutput, laneID string, extractor, merger, normalizer any) error {
@@ -414,12 +492,12 @@ func (r *Runner) validateTypedArtifact(ctx context.Context, codec artifactCodecE
} }
if err != nil { if err != nil {
validationErr := fmt.Errorf("validate typed %s output with validator %q: %w", target.stage, binding.Module, err) validationErr := fmt.Errorf("validate typed %s output with validator %q: %w", target.stage, binding.Module, err)
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall, Error: debugCall.Error}, llmScope); debugErr != nil { if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall, Error: debugCall.Error}, llmScope); debugErr != nil {
return warnings, nil, errors.Join(validationErr, fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr)) return warnings, nil, errors.Join(validationErr, fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr))
} }
return warnings, nil, validationErr return warnings, nil, validationErr
} }
if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall}, llmScope); debugErr != nil { if debugErr := writeDebugAttempt(debug, attemptPath, debugTimedEnvelope{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall}, llmScope); debugErr != nil {
return warnings, nil, fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr) return warnings, nil, fmt.Errorf("write typed validator attempt debug artifact: %w", debugErr)
} }
if !result.Approved { if !result.Approved {
@@ -431,7 +509,7 @@ func (r *Runner) validateTypedArtifact(ctx context.Context, codec artifactCodecE
if message == "" { if message == "" {
message = "artifact rejected" message = "artifact rejected"
} }
return warnings, &contracts.RejectedOutput{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, ChunkID: func() string { return warnings, &contracts.RejectedOutput{Stage: string(target.stage), StepID: target.stepID, LaneID: target.laneID, ModuleKey: target.moduleKey, ChunkID: func() string {
if target.chunk != nil { if target.chunk != nil {
return target.chunk.ID return target.chunk.ID
} }

View File

@@ -1,12 +1,102 @@
package pipeline package pipeline
import ( import (
"context"
"encoding/json"
"strings" "strings"
"testing" "testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
) )
type requiredCheckpointLoader struct {
CheckpointLoader
checkpoint ExtractCheckpoint
decision CheckpointDecision
}
func (l requiredCheckpointLoader) Enabled() bool { return true }
func (l requiredCheckpointLoader) Extract(string, string, []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
return l.checkpoint, l.decision
}
func (l requiredCheckpointLoader) AcceptedNormalize(string, string, string) (NormalizeCheckpoint, CheckpointDecision) {
output := CheckpointArtifact{}
if len(l.checkpoint.Outputs) > 0 {
output = l.checkpoint.Outputs[0]
}
return NormalizeCheckpoint{Output: output}, l.decision
}
func TestRequiredCheckpointFailureRetainsDecision(t *testing.T) {
const unsafeDetail = "unsafe-loader-detail-/private/checkpoint/path"
tests := []struct {
name string
decision CheckpointDecision
corrupt bool
wantCode CheckpointReasonCode
wantAction CheckpointDecisionCategory
}{
{"missing", NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonMissing, unsafeDetail), false, CheckpointReasonMissing, CheckpointDecisionExecuted},
{"corrupt", NewCheckpointDecision(CheckpointDecisionReused, CheckpointReasonReused, "checkpoint reusable"), true, CheckpointReasonArtifactNotCanonical, CheckpointDecisionExecuted},
{"incompatible", NewCheckpointDecision(CheckpointDecisionExecuted, CheckpointReasonArtifactCodecIncompatible, unsafeDetail), false, CheckpointReasonArtifactCodecIncompatible, CheckpointDecisionExecuted},
{"dependency invalidated", NewCheckpointDecision(CheckpointDecisionDependencyInvalidated, CheckpointReasonDependencyMismatch, unsafeDetail), false, CheckpointReasonDependencyMismatch, CheckpointDecisionDependencyInvalidated},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 1)
step := prepared.Steps[0]
lane := step.lanes[0]
doc := prepared.input.(*typedTestInput).doc
checkpoint := ExtractCheckpoint{}
if test.corrupt {
stored, err := checkpointArtifact(lane.typed.codec, lane.resolved.ID, lane.resolved.Normalize.Module, doc.ID, codecNotes{Items: []string{"stored"}})
if err != nil {
t.Fatal(err)
}
stored.Artifact.Content = []byte(`{"items": ["stored"]}`)
checkpoint.Outputs = []CheckpointArtifact{stored}
}
loader := requiredCheckpointLoader{CheckpointLoader: NoopCheckpointLoader(), checkpoint: checkpoint, decision: test.decision}
policy := CheckpointExecutionPolicy{RequireReusableLanes: map[string]struct{}{CheckpointLaneKey(step.ID, lane.resolved.ID): {}}}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Checkpoint: loader, CheckpointPolicy: policy})
if err == nil || !strings.Contains(err.Error(), step.ID) || !strings.Contains(err.Error(), lane.resolved.ID) || !strings.Contains(err.Error(), string(test.wantCode)) {
t.Fatalf("Run() error = %v, want step, lane, and reason code %q", err, test.wantCode)
}
if strings.Contains(err.Error(), unsafeDetail) {
t.Fatalf("Run() error leaked loader detail: %v", err)
}
var found bool
for _, event := range output.CheckpointEvents {
if event.Stage == string(StageNormalize) && event.StepID == step.ID && event.LaneID == lane.resolved.ID {
found = true
if event.Action != test.wantAction || event.ReasonCode != test.wantCode {
t.Fatalf("checkpoint event = %#v, want action %q and reason %q", event, test.wantAction, test.wantCode)
}
if strings.Contains(event.Detail, unsafeDetail) {
t.Fatalf("checkpoint event leaked loader detail: %#v", event)
}
}
}
if !found {
t.Fatalf("required checkpoint decision missing from %#v", output.CheckpointEvents)
}
var manifestFound bool
for _, decision := range output.Manifest.CheckpointDecisions {
if decision.Stage == string(StageNormalize) && decision.StepID == step.ID && decision.LaneID == lane.resolved.ID {
manifestFound = decision.Category == string(test.wantAction) && decision.ReasonCode == string(test.wantCode)
}
}
if !manifestFound {
t.Fatalf("manifest checkpoint decision missing category %q and code %q: %#v", test.wantAction, test.wantCode, output.Manifest.CheckpointDecisions)
}
encoded, marshalErr := json.Marshal(output.CheckpointEvents)
if marshalErr != nil || !strings.Contains(string(encoded), `"category":"`+string(test.wantAction)+`"`) || !strings.Contains(string(encoded), `"reason_code":"`+string(test.wantCode)+`"`) {
t.Fatalf("checkpoint event JSON = %s, error = %v", encoded, marshalErr)
}
})
}
}
func TestDecodeCheckpointArtifactRejectsIncompatibleCodecIdentityAndBytes(t *testing.T) { func TestDecodeCheckpointArtifactRejectsIncompatibleCodecIdentityAndBytes(t *testing.T) {
registry := NewArtifactCodecRegistry() registry := NewArtifactCodecRegistry()
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil { if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
@@ -43,3 +133,26 @@ func TestDecodeCheckpointArtifactRejectsIncompatibleCodecIdentityAndBytes(t *tes
}) })
} }
} }
func TestDecodeCanonicalCheckpointArtifactRejectsNonCanonicalBytes(t *testing.T) {
registry := NewArtifactCodecRegistry()
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
t.Fatalf("RegisterArtifactCodec: %v", err)
}
codec, _, err := registry.entry("test/notes")
if err != nil {
t.Fatalf("entry: %v", err)
}
artifact, err := serializeArtifact(codec, codecNotes{Items: []string{"one"}}, false)
if err != nil {
t.Fatalf("serializeArtifact: %v", err)
}
stored := CheckpointArtifact{Artifact: artifact, SchemaDigest: contracts.DigestArtifactSchema(artifact.Schema)}
if _, _, err := decodeCanonicalCheckpointArtifact(codec, stored); err != nil {
t.Fatalf("canonical artifact rejected: %v", err)
}
stored.Artifact.Content = []byte(`{"items": ["one"]}`)
if _, _, err := decodeCanonicalCheckpointArtifact(codec, stored); err == nil {
t.Fatal("non-canonical artifact was accepted")
}
}

View File

@@ -76,6 +76,35 @@ func (l *lockedCheckpointLoader) Normalize(lane, key string, deps []CheckpointFi
defer l.mu.Unlock() defer l.mu.Unlock()
return l.inner.Normalize(lane, key, deps) return l.inner.Normalize(lane, key, deps)
} }
func (l *lockedCheckpointLoader) AcceptedNormalize(step, lane, key string) (NormalizeCheckpoint, CheckpointDecision) {
l.mu.Lock()
defer l.mu.Unlock()
return l.inner.AcceptedNormalize(step, lane, key)
}
func (l *lockedCheckpointLoader) ExtractForStep(step, lane, key string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
l.mu.Lock()
defer l.mu.Unlock()
if stepAware, ok := l.inner.(StepCheckpointLoader); ok {
return stepAware.ExtractForStep(step, lane, key, deps)
}
return l.inner.Extract(lane, key, deps)
}
func (l *lockedCheckpointLoader) MergeForStep(step, lane, key string, deps []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
l.mu.Lock()
defer l.mu.Unlock()
if stepAware, ok := l.inner.(StepCheckpointLoader); ok {
return stepAware.MergeForStep(step, lane, key, deps)
}
return l.inner.Merge(lane, key, deps)
}
func (l *lockedCheckpointLoader) NormalizeForStep(step, lane, key string, deps []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
l.mu.Lock()
defer l.mu.Unlock()
if stepAware, ok := l.inner.(StepCheckpointLoader); ok {
return stepAware.NormalizeForStep(step, lane, key, deps)
}
return l.inner.Normalize(lane, key, deps)
}
type lockedCheckpointRecorder struct { type lockedCheckpointRecorder struct {
inner CheckpointRecorder inner CheckpointRecorder
@@ -135,3 +164,92 @@ func (r *lockedCheckpointRecorder) NormalizeRejected(lane, key string, deps []Ch
func (r *lockedCheckpointRecorder) NormalizeFailed(lane, key string, deps []CheckpointFingerprint, err error) error { func (r *lockedCheckpointRecorder) NormalizeFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
return r.call(func() error { return r.inner.NormalizeFailed(lane, key, deps, err) }) return r.call(func() error { return r.inner.NormalizeFailed(lane, key, deps, err) })
} }
func (r *lockedCheckpointRecorder) ExtractRunningForStep(step, lane, key string, deps []CheckpointFingerprint) error {
return r.call(func() error {
if v, ok := r.inner.(StepCheckpointRecorder); ok {
return v.ExtractRunningForStep(step, lane, key, deps)
}
return r.inner.ExtractRunning(lane, key, deps)
})
}
func (r *lockedCheckpointRecorder) ExtractSucceededForStep(step, lane, key string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
return r.call(func() error {
if v, ok := r.inner.(StepCheckpointRecorder); ok {
return v.ExtractSucceededForStep(step, lane, key, deps, outputs, rejected, warnings)
}
return r.inner.ExtractSucceeded(lane, key, deps, outputs, rejected, warnings)
})
}
func (r *lockedCheckpointRecorder) ExtractFailedForStep(step, lane, key string, deps []CheckpointFingerprint, err error) error {
return r.call(func() error {
if v, ok := r.inner.(StepCheckpointRecorder); ok {
return v.ExtractFailedForStep(step, lane, key, deps, err)
}
return r.inner.ExtractFailed(lane, key, deps, err)
})
}
func (r *lockedCheckpointRecorder) MergeRunningForStep(step, lane, key string, deps []CheckpointFingerprint) error {
return r.call(func() error {
if v, ok := r.inner.(StepCheckpointRecorder); ok {
return v.MergeRunningForStep(step, lane, key, deps)
}
return r.inner.MergeRunning(lane, key, deps)
})
}
func (r *lockedCheckpointRecorder) MergeSucceededForStep(step, lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
return r.call(func() error {
if v, ok := r.inner.(StepCheckpointRecorder); ok {
return v.MergeSucceededForStep(step, lane, key, deps, output, warnings)
}
return r.inner.MergeSucceeded(lane, key, deps, output, warnings)
})
}
func (r *lockedCheckpointRecorder) MergeRejectedForStep(step, lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
return r.call(func() error {
if v, ok := r.inner.(StepCheckpointRecorder); ok {
return v.MergeRejectedForStep(step, lane, key, deps, rejected)
}
return r.inner.MergeRejected(lane, key, deps, rejected)
})
}
func (r *lockedCheckpointRecorder) MergeFailedForStep(step, lane, key string, deps []CheckpointFingerprint, err error) error {
return r.call(func() error {
if v, ok := r.inner.(StepCheckpointRecorder); ok {
return v.MergeFailedForStep(step, lane, key, deps, err)
}
return r.inner.MergeFailed(lane, key, deps, err)
})
}
func (r *lockedCheckpointRecorder) NormalizeRunningForStep(step, lane, key string, deps []CheckpointFingerprint) error {
return r.call(func() error {
if v, ok := r.inner.(StepCheckpointRecorder); ok {
return v.NormalizeRunningForStep(step, lane, key, deps)
}
return r.inner.NormalizeRunning(lane, key, deps)
})
}
func (r *lockedCheckpointRecorder) NormalizeSucceededForStep(step, lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
return r.call(func() error {
if v, ok := r.inner.(StepCheckpointRecorder); ok {
return v.NormalizeSucceededForStep(step, lane, key, deps, output, warnings)
}
return r.inner.NormalizeSucceeded(lane, key, deps, output, warnings)
})
}
func (r *lockedCheckpointRecorder) NormalizeRejectedForStep(step, lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
return r.call(func() error {
if v, ok := r.inner.(StepCheckpointRecorder); ok {
return v.NormalizeRejectedForStep(step, lane, key, deps, rejected)
}
return r.inner.NormalizeRejected(lane, key, deps, rejected)
})
}
func (r *lockedCheckpointRecorder) NormalizeFailedForStep(step, lane, key string, deps []CheckpointFingerprint, err error) error {
return r.call(func() error {
if v, ok := r.inner.(StepCheckpointRecorder); ok {
return v.NormalizeFailedForStep(step, lane, key, deps, err)
}
return r.inner.NormalizeFailed(lane, key, deps, err)
})
}

View File

@@ -28,6 +28,7 @@ type erasedTypedResult struct {
type typedValidationTarget struct { type typedValidationTarget struct {
stage ModuleStage stage ModuleStage
stepID string
laneID string laneID string
moduleKey string moduleKey string
source *source.SourceDocument source *source.SourceDocument

View File

@@ -147,11 +147,11 @@ func TestResolveTypedHeterogeneousLanes(t *testing.T) {
t.Fatalf("ResolvePipeline() error = %v, want nil", err) t.Fatalf("ResolvePipeline() error = %v, want nil", err)
} }
if got, want := resolvedLaneIDs(resolved.ArtifactLanes), []string{"notes", "score"}; !reflect.DeepEqual(got, want) { if got, want := resolvedLaneIDs(resolved.Steps[0].ArtifactLanes), []string{"notes", "score"}; !reflect.DeepEqual(got, want) {
t.Fatalf("lane order = %#v, want %#v", got, want) t.Fatalf("lane order = %#v, want %#v", got, want)
} }
assertResolvedArtifactIdentity(t, resolved.ArtifactLanes[0], "test/notes", "notes.v1") assertResolvedArtifactIdentity(t, resolved.Steps[0].ArtifactLanes[0], "test/notes", "notes.v1")
assertResolvedArtifactIdentity(t, resolved.ArtifactLanes[1], "test/score", "score.v1") assertResolvedArtifactIdentity(t, resolved.Steps[0].ArtifactLanes[1], "test/score", "score.v1")
chunkChain := resolved.ValidatorChains[0] chunkChain := resolved.ValidatorChains[0]
if got := resolvedValidatorTargets(chunkChain.Validators); !reflect.DeepEqual(got, []ValidatorTarget{ValidatorTargetChunk, ValidatorTargetSerialized}) { if got := resolvedValidatorTargets(chunkChain.Validators); !reflect.DeepEqual(got, []ValidatorTarget{ValidatorTargetChunk, ValidatorTargetSerialized}) {
@@ -176,8 +176,8 @@ func TestPrepareConstructsHeterogeneousTypedLanes(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err) t.Fatalf("Prepare() error = %v, want nil", err)
} }
if len(prepared.ArtifactLanes) != 2 || prepared.lanes[0].typed == nil || prepared.lanes[1].typed == nil { if len(prepared.Steps[0].ArtifactLanes) != 2 || prepared.Steps[0].lanes[0].typed == nil || prepared.Steps[0].lanes[1].typed == nil {
t.Fatalf("PreparedPipeline lanes = %#v, want two typed executors", prepared.ArtifactLanes) t.Fatalf("PreparedPipeline lanes = %#v, want two typed executors", prepared.Steps[0].ArtifactLanes)
} }
} }

View File

@@ -92,9 +92,9 @@ func cloneModuleBinding(binding ModuleBinding) ModuleBinding {
binding.LLMProfile = strings.TrimSpace(binding.LLMProfile) binding.LLMProfile = strings.TrimSpace(binding.LLMProfile)
binding.Options = cloneOptions(binding.Options) binding.Options = cloneOptions(binding.Options)
if len(binding.References) > 0 { if len(binding.References) > 0 {
references := make(map[string]string, len(binding.References)) references := make(map[string]ReferenceSource, len(binding.References))
for key, value := range binding.References { for key, value := range binding.References {
references[key] = value references[key] = cloneReferenceSource(value)
} }
binding.References = references binding.References = references
} }
@@ -102,6 +102,15 @@ func cloneModuleBinding(binding ModuleBinding) ModuleBinding {
return binding return binding
} }
func cloneReferenceSource(source ReferenceSource) ReferenceSource {
out := source
if source.Artifact != nil {
artifact := *source.Artifact
out.Artifact = &artifact
}
return out
}
func cloneValidatorOverride(override ValidatorOverride) ValidatorOverride { func cloneValidatorOverride(override ValidatorOverride) ValidatorOverride {
return ValidatorOverride{ return ValidatorOverride{
Set: override.Set, Set: override.Set,

View File

@@ -42,10 +42,11 @@ var referenceSlotDescriptions = shared.ReferenceSlotDescriptions{
func referenceSlots() []contracts.ReferenceSlot { func referenceSlots() []contracts.ReferenceSlot {
slots := shared.ReferenceSlots(referenceSlotDescriptions) slots := shared.ReferenceSlots(referenceSlotDescriptions)
slots = append(slots, contracts.ReferenceSlot{ slots = append(slots, contracts.ReferenceSlot{
Name: NPCRegistryReferenceSlot, Name: NPCRegistryReferenceSlot,
Description: "Optional normalized NPC registry used for canonical actor and target grounding.", Description: "Optional normalized NPC registry used for canonical actor and target grounding.",
AcceptedMediaTypes: []string{"application/json"}, AcceptedMediaTypes: []string{"application/json"},
MaxBytes: NPCRegistryMaxBytes, AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.NPCListKind},
MaxBytes: NPCRegistryMaxBytes,
}) })
sort.Slice(slots, func(i, j int) bool { return slots[i].Name < slots[j].Name }) sort.Slice(slots, func(i, j int) bool { return slots[i].Name < slots[j].Name })
return slots return slots
@@ -59,7 +60,7 @@ type Options struct{}
type Extractor struct { type Extractor struct {
llm contracts.StructuredLLMClient llm contracts.StructuredLLMClient
npcRegistry *npcregistry.Registry npcResolver *npcregistry.Resolver
promptSHA string promptSHA string
responseSchemaSHA string responseSchemaSHA string
} }
@@ -75,7 +76,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
if len(references) == 1 { if len(references) == 1 {
referenceSet = references[0] referenceSet = references[0]
} }
npcRegistry, err := npcregistry.Resolve(referenceSet) npcResolver, err := npcregistry.NewResolver(referenceSet)
if err != nil { if err != nil {
return nil, extractorErrorf("prepare NPC registry prompt input: %w", err) return nil, extractorErrorf("prepare NPC registry prompt input: %w", err)
} }
@@ -89,7 +90,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
} }
return &Extractor{ return &Extractor{
llm: llmClient, llm: llmClient,
npcRegistry: npcRegistry, npcResolver: npcResolver,
promptSHA: promptSHA, promptSHA: promptSHA,
responseSchemaSHA: responseSchema.SHA256, responseSchemaSHA: responseSchema.SHA256,
}, nil }, nil
@@ -114,9 +115,10 @@ func (e *Extractor) ManifestMetadata() map[string]any {
"response_schema_version": SchemaVersion, "response_schema_version": SchemaVersion,
"response_schema_sha256": e.responseSchemaSHA, "response_schema_sha256": e.responseSchemaSHA,
} }
if e.npcRegistry.Bound() { seeded := e.npcResolver.Seeded()
metadata["npc_registry_digest"] = e.npcRegistry.Digest() if seeded.Bound() {
metadata["npc_count"] = e.npcRegistry.Count() metadata["npc_registry_digest"] = seeded.Digest()
metadata["npc_count"] = seeded.Count()
} }
return metadata return metadata
} }
@@ -130,8 +132,9 @@ func (e *Extractor) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
{Name: "response_schema", Value: e.responseSchemaSHA}, {Name: "response_schema", Value: e.responseSchemaSHA},
{Name: "mapping_policy", Value: mappingPolicy}, {Name: "mapping_policy", Value: mappingPolicy},
} }
if e.npcRegistry.Bound() { seeded := e.npcResolver.Seeded()
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: e.npcRegistry.Digest()}) if seeded.Bound() {
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: seeded.Digest()})
} }
return fingerprints return fingerprints
} }
@@ -162,10 +165,14 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
if err != nil { if err != nil {
return contracts.TypedExtractionResult[dnd.CombatTurnList]{}, extractorErrorf("%w", err) return contracts.TypedExtractionResult[dnd.CombatTurnList]{}, extractorErrorf("%w", err)
} }
npcRegistry, err := e.npcResolver.Resolve(req.References)
if err != nil {
return contracts.TypedExtractionResult[dnd.CombatTurnList]{}, extractorErrorf("resolve NPC registry: %w", err)
}
var response extractionResponse var response extractionResponse
inputs := shared.PromptInputs(sourceInput, req.References) inputs := shared.PromptInputs(sourceInput, req.References)
inputs[NPCRegistryReferenceSlot] = e.npcRegistry.PromptInput() inputs[NPCRegistryReferenceSlot] = npcRegistry.PromptInput()
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{ if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: Key, StageName: Key,
PromptID: PromptID, PromptID: PromptID,

View File

@@ -160,6 +160,29 @@ func TestExtractUnboundRegistryUsesExactEmptyPromptAndOmitsIdentity(t *testing.T
} }
} }
func TestExtractorResolvesOperationNPCOverrideWithoutSingletonMetadata(t *testing.T) {
content := npcRegistryJSON(t)
client := &fakeCombatTurnsLLMClient{response: extractionResponse{CombatTurns: []combatTurnResponse{}}}
extractor := newExtractor(t, client)
request := extractionRequest()
request.References = contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
NPCRegistryReferenceSlot: {
Slot: contracts.ReferenceSlot{Name: NPCRegistryReferenceSlot},
Items: []contracts.ReferenceItem{{SlotName: NPCRegistryReferenceSlot, MediaType: npccodec.MediaType, Content: append([]byte(nil), content...), Origin: contracts.ReferenceOrigin{Type: "generated"}}},
},
}}
if _, err := extractor.Extract(context.Background(), request); err != nil {
t.Fatalf("Extract() error = %v", err)
}
input := client.requests[0].Inputs[NPCRegistryReferenceSlot]
if input.Digest == "" || string(input.Content) != string(content) || input.OriginURI != "" {
t.Fatalf("operation NPC input = %#v, want generated canonical grounding without provenance", input)
}
if metadata := extractor.ManifestMetadata(); metadata["npc_registry_digest"] != nil || metadata["npc_count"] != nil {
t.Fatalf("singleton metadata = %#v, want no operation-varying NPC identity", metadata)
}
}
func TestNewRejectsMalformedNPCRegistryBeforeLLMCallWithoutContent(t *testing.T) { func TestNewRejectsMalformedNPCRegistryBeforeLLMCallWithoutContent(t *testing.T) {
client := &fakeCombatTurnsLLMClient{} client := &fakeCombatTurnsLLMClient{}
references := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{ references := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{

View File

@@ -46,10 +46,11 @@ func referenceSlots() []contracts.ReferenceSlot {
AcceptedMediaTypes: []string{"application/json"}, AcceptedMediaTypes: []string{"application/json"},
MaxBytes: 1048576, MaxBytes: 1048576,
}, contracts.ReferenceSlot{ }, contracts.ReferenceSlot{
Name: NPCRegistryReferenceSlot, Name: NPCRegistryReferenceSlot,
Description: "Optional normalized NPC registry used for canonical caster-name grounding.", Description: "Optional normalized NPC registry used for canonical caster-name grounding.",
AcceptedMediaTypes: []string{"application/json"}, AcceptedMediaTypes: []string{"application/json"},
MaxBytes: NPCRegistryMaxBytes, AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.NPCListKind},
MaxBytes: NPCRegistryMaxBytes,
}) })
sort.Slice(slots, func(i, j int) bool { return slots[i].Name < slots[j].Name }) sort.Slice(slots, func(i, j int) bool { return slots[i].Name < slots[j].Name })
return slots return slots
@@ -64,7 +65,7 @@ type Extractor struct {
llm contracts.StructuredLLMClient llm contracts.StructuredLLMClient
effectiveCatalog spellcatalog.EffectiveCatalog effectiveCatalog spellcatalog.EffectiveCatalog
catalogPromptInput contracts.LLMInputMaterial catalogPromptInput contracts.LLMInputMaterial
npcRegistry *npcregistry.Registry npcResolver *npcregistry.Resolver
promptSHA string promptSHA string
responseSchemaSHA string responseSchemaSHA string
} }
@@ -88,7 +89,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
if err != nil { if err != nil {
return nil, extractorErrorf("prepare spell catalog prompt input: %w", err) return nil, extractorErrorf("prepare spell catalog prompt input: %w", err)
} }
npcRegistry, err := npcregistry.Resolve(referenceSet) npcResolver, err := npcregistry.NewResolver(referenceSet)
if err != nil { if err != nil {
return nil, extractorErrorf("prepare NPC registry prompt input: %w", err) return nil, extractorErrorf("prepare NPC registry prompt input: %w", err)
} }
@@ -104,7 +105,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
llm: llmClient, llm: llmClient,
effectiveCatalog: effectiveCatalog, effectiveCatalog: effectiveCatalog,
catalogPromptInput: catalogPromptInput, catalogPromptInput: catalogPromptInput,
npcRegistry: npcRegistry, npcResolver: npcResolver,
promptSHA: promptSHA, promptSHA: promptSHA,
responseSchemaSHA: responseSchema.SHA256, responseSchemaSHA: responseSchema.SHA256,
}, nil }, nil
@@ -132,9 +133,10 @@ func (e *Extractor) ManifestMetadata() map[string]any {
"response_schema_version": SchemaVersion, "response_schema_version": SchemaVersion,
"response_schema_sha256": e.responseSchemaSHA, "response_schema_sha256": e.responseSchemaSHA,
} }
if e.npcRegistry.Bound() { seeded := e.npcResolver.Seeded()
metadata["npc_registry_digest"] = e.npcRegistry.Digest() if seeded.Bound() {
metadata["npc_count"] = e.npcRegistry.Count() metadata["npc_registry_digest"] = seeded.Digest()
metadata["npc_count"] = seeded.Count()
} }
return metadata return metadata
} }
@@ -148,8 +150,9 @@ func (e *Extractor) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
{Name: "prompt", Value: e.promptSHA}, {Name: "prompt", Value: e.promptSHA},
{Name: "response_schema", Value: e.responseSchemaSHA}, {Name: "response_schema", Value: e.responseSchemaSHA},
} }
if e.npcRegistry.Bound() { seeded := e.npcResolver.Seeded()
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: e.npcRegistry.Digest()}) if seeded.Bound() {
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: seeded.Digest()})
} }
return fingerprints return fingerprints
} }
@@ -180,11 +183,15 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
if err != nil { if err != nil {
return contracts.TypedExtractionResult[dnd.SpellList]{}, extractorErrorf("%w", err) return contracts.TypedExtractionResult[dnd.SpellList]{}, extractorErrorf("%w", err)
} }
npcRegistry, err := e.npcResolver.Resolve(req.References)
if err != nil {
return contracts.TypedExtractionResult[dnd.SpellList]{}, extractorErrorf("resolve NPC registry: %w", err)
}
var response extractionResponse var response extractionResponse
inputs := shared.PromptInputs(sourceInput, req.References) inputs := shared.PromptInputs(sourceInput, req.References)
inputs[spellcatalog.SpellCatalogReferenceSlot] = e.catalogPromptInput.Clone() inputs[spellcatalog.SpellCatalogReferenceSlot] = e.catalogPromptInput.Clone()
inputs[NPCRegistryReferenceSlot] = e.npcRegistry.PromptInput() inputs[NPCRegistryReferenceSlot] = npcRegistry.PromptInput()
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{ if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: Key, StageName: Key,
PromptID: PromptID, PromptID: PromptID,

View File

@@ -84,6 +84,32 @@ func TestSpellExtractorPreservesSemanticNPCRegistryFingerprintAndPromptWiring(t
} }
} }
func TestSpellExtractorResolvesOperationNPCOverrideWithoutSingletonMetadata(t *testing.T) {
canonical, err := npccodec.New().Encode(registryFixture())
if err != nil {
t.Fatalf("encode NPC registry: %v", err)
}
client := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{}}}
extractor := newExtractor(t, client)
request := extractionRequest()
request.References = spellNPCRegistryReference(canonical, "file:///generated.json")
if _, err := extractor.Extract(context.Background(), request); err != nil {
t.Fatalf("Extract() error = %v", err)
}
input := client.requests[0].Inputs[NPCRegistryReferenceSlot]
if input.Digest == "" || string(input.Content) != string(canonical) || input.OriginURI != "" {
t.Fatalf("operation NPC input = %#v, want generated canonical grounding without provenance", input)
}
if metadata := extractor.ManifestMetadata(); metadata["npc_registry_digest"] != nil || metadata["npc_count"] != nil {
t.Fatalf("singleton metadata = %#v, want no operation-varying NPC identity", metadata)
}
for _, fingerprint := range extractor.CheckpointFingerprints() {
if fingerprint.Name == "npc_registry" {
t.Fatalf("singleton fingerprints = %#v, want no operation-varying NPC identity", extractor.CheckpointFingerprints())
}
}
}
func registryFixture() dnd.NPCList { func registryFixture() dnd.NPCList {
return dnd.NPCList{NPCs: []dnd.NPC{{ return dnd.NPCList{NPCs: []dnd.NPC{{
ID: identity.DeriveID("Mira Thorn"), ID: identity.DeriveID("Mira Thorn"),

View File

@@ -40,10 +40,11 @@ func TestModuleSpec(t *testing.T) {
AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}, AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"},
}, },
{ {
Name: NPCRegistryReferenceSlot, Name: NPCRegistryReferenceSlot,
Description: "Optional normalized NPC registry used for canonical caster-name grounding.", Description: "Optional normalized NPC registry used for canonical caster-name grounding.",
AcceptedMediaTypes: []string{"application/json"}, AcceptedMediaTypes: []string{"application/json"},
MaxBytes: NPCRegistryMaxBytes, AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.NPCListKind},
MaxBytes: NPCRegistryMaxBytes,
}, },
{ {
Name: "party", Name: "party",

View File

@@ -45,7 +45,7 @@ var _ pipeline.CheckpointFingerprintProvider = (*Normalizer)(nil)
type Options struct{} type Options struct{}
type Normalizer struct { type Normalizer struct {
npcRegistry *npcregistry.Registry npcResolver *npcregistry.Resolver
} }
func New(_ Options, references ...contracts.ReferenceSet) (*Normalizer, error) { func New(_ Options, references ...contracts.ReferenceSet) (*Normalizer, error) {
@@ -56,11 +56,11 @@ func New(_ Options, references ...contracts.ReferenceSet) (*Normalizer, error) {
if len(references) == 1 { if len(references) == 1 {
referenceSet = references[0] referenceSet = references[0]
} }
npcRegistry, err := npcregistry.Resolve(referenceSet) npcResolver, err := npcregistry.NewResolver(referenceSet)
if err != nil { if err != nil {
return nil, normalizerErrorf("prepare NPC registry: %w", err) return nil, normalizerErrorf("prepare NPC registry: %w", err)
} }
return &Normalizer{npcRegistry: npcRegistry}, nil return &Normalizer{npcResolver: npcResolver}, nil
} }
func (n *Normalizer) Key() string { return Key } func (n *Normalizer) Key() string { return Key }
@@ -75,9 +75,10 @@ func (n *Normalizer) ManifestMetadata() map[string]any {
"normalization_policy": normalizationPolicy, "normalization_policy": normalizationPolicy,
"identity_policy": identity.Policy, "identity_policy": identity.Policy,
} }
if n.npcRegistry.Bound() { seeded := n.npcResolver.Seeded()
metadata["npc_registry_digest"] = n.npcRegistry.Digest() if seeded.Bound() {
metadata["npc_count"] = n.npcRegistry.Count() metadata["npc_registry_digest"] = seeded.Digest()
metadata["npc_count"] = seeded.Count()
} }
return metadata return metadata
} }
@@ -90,8 +91,9 @@ func (n *Normalizer) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
{Name: "normalization_policy", Value: normalizationPolicy}, {Name: "normalization_policy", Value: normalizationPolicy},
{Name: "identity_policy", Value: identity.Policy}, {Name: "identity_policy", Value: identity.Policy},
} }
if n.npcRegistry.Bound() { seeded := n.npcResolver.Seeded()
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: n.npcRegistry.Digest()}) if seeded.Bound() {
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: seeded.Digest()})
} }
return fingerprints return fingerprints
} }
@@ -107,7 +109,11 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
return contracts.TypedNormalizeResult[dnd.CombatTurnList]{}, normalizerErrorf("context error before normalize: %w", err) return contracts.TypedNormalizeResult[dnd.CombatTurnList]{}, normalizerErrorf("context error before normalize: %w", err)
} }
value, warnings := normalizeList(req.MergeOutput.Value, req.Source, n.npcRegistry) npcRegistry, err := n.npcResolver.Resolve(req.References)
if err != nil {
return contracts.TypedNormalizeResult[dnd.CombatTurnList]{}, normalizerErrorf("resolve NPC registry: %w", err)
}
value, warnings := normalizeList(req.MergeOutput.Value, req.Source, npcRegistry)
return contracts.TypedNormalizeResult[dnd.CombatTurnList]{Value: value, Warnings: warnings}, nil return contracts.TypedNormalizeResult[dnd.CombatTurnList]{Value: value, Warnings: warnings}, nil
} }
@@ -494,10 +500,11 @@ func turnScope(index int) string { return fmt.Sprintf("combat_turns[%d]", index)
func referenceSlots() []contracts.ReferenceSlot { func referenceSlots() []contracts.ReferenceSlot {
return []contracts.ReferenceSlot{{ return []contracts.ReferenceSlot{{
Name: NPCRegistryReferenceSlot, Name: NPCRegistryReferenceSlot,
Description: "Optional normalized NPC registry used for canonical actor and target grounding.", Description: "Optional normalized NPC registry used for canonical actor and target grounding.",
AcceptedMediaTypes: []string{"application/json"}, AcceptedMediaTypes: []string{"application/json"},
MaxBytes: NPCRegistryMaxBytes, AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.NPCListKind},
MaxBytes: NPCRegistryMaxBytes,
}} }}
} }

View File

@@ -70,6 +70,36 @@ func TestNormalizeCanonicalizesFieldsAndRegistryIdentities(t *testing.T) {
} }
} }
func TestNormalizeResolvesOperationNPCOverrideWithoutSingletonMetadata(t *testing.T) {
doc := testDocument()
normalizer, err := New(Options{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
input := dnd.CombatTurnList{CombatTurns: []dnd.CombatTurn{{
Actor: "Storm",
TurnKind: dnd.CombatTurnKindTurn,
Actions: []dnd.CombatAction{{Category: dnd.CombatActionCategoryAttack, Declaration: "attacks", Targets: []string{"Minion"}}},
Summary: "Storm attacks",
SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}},
}}}
result, err := normalizer.Normalize(context.Background(), contracts.TypedNormalizeRequest[dnd.CombatTurnList]{
Source: doc,
MergeOutput: contracts.MergeArtifact[dnd.CombatTurnList]{Value: input},
References: npcReferences(t),
})
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
turn := result.Value.CombatTurns[0]
if turn.Actor != "Aria" || turn.Actions[0].Targets[0] != "Goblin" {
t.Fatalf("operation-normalized turn = %#v, want Aria/Goblin", turn)
}
if metadata := normalizer.ManifestMetadata(); metadata["npc_registry_digest"] != nil || metadata["npc_count"] != nil {
t.Fatalf("singleton metadata = %#v, want no operation-varying NPC identity", metadata)
}
}
func TestNormalizeOrdersBySourcePositionAndCollapsesExactDuplicates(t *testing.T) { func TestNormalizeOrdersBySourcePositionAndCollapsesExactDuplicates(t *testing.T) {
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{{ID: 50}, {ID: 10}, {ID: 90}}} doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{{ID: 50}, {ID: 10}, {ID: 90}}}
first := validTurn("Aria", source.SourceRef{SourceID: doc.ID, StartUnitID: 50, EndUnitID: 50}) first := validTurn("Aria", source.SourceRef{SourceID: doc.ID, StartUnitID: 50, EndUnitID: 50})

View File

@@ -8,6 +8,7 @@ import (
"fmt" "fmt"
"mime" "mime"
"strings" "strings"
"sync"
"gitea.maximumdirect.net/eric/notarius/internal/core/source" "gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
@@ -34,6 +35,106 @@ type Registry struct {
lookupByKey map[string]int lookupByKey map[string]int
} }
// Resolver retains only the validated construction-time registry and immutable
// canonical registries keyed by their semantic digest. Operation references
// are resolved on demand; caller-owned reference bytes are never retained.
type Resolver struct {
seeded *Registry
mu sync.Mutex
cache map[string]*Registry
rawCache map[string]*Registry
}
// NewResolver validates the optional construction-time NPC reference and
// prepares the operation-time registry cache. A malformed static reference
// therefore fails before any operation starts.
func NewResolver(references contracts.ReferenceSet) (*Resolver, error) {
seeded, err := Resolve(constructionReferences(references))
if err != nil {
return nil, err
}
return &Resolver{seeded: seeded, cache: make(map[string]*Registry), rawCache: make(map[string]*Registry)}, nil
}
func constructionReferences(references contracts.ReferenceSet) contracts.ReferenceSet {
slot, ok := references.Slots[ReferenceSlot]
if !ok || len(slot.Items) > 0 {
return references
}
cloned := contracts.ReferenceSet{Slots: make(map[string]contracts.ResolvedReferenceSlot, len(references.Slots))}
for name, value := range references.Slots {
cloned.Slots[name] = value
}
delete(cloned.Slots, ReferenceSlot)
return cloned
}
// Seeded returns the immutable construction-time registry. Its accessors are
// defensive, so callers may safely use the returned view for static metadata.
func (r *Resolver) Seeded() *Registry {
if r == nil {
return nil
}
return r.seeded
}
// Resolve returns the effective registry for one operation. An operation
// without an NPC item uses the construction-time registry. A canonical item
// matching that registry reuses it; other canonical registries are cached by
// digest for concurrent chunk operations.
func (r *Resolver) Resolve(references contracts.ReferenceSet) (*Registry, error) {
if r == nil {
return Resolve(references)
}
if _, ok := references.Slots[ReferenceSlot]; !ok {
return r.seeded, nil
}
slot := references.Slots[ReferenceSlot]
rawKey := ""
if len(slot.Items) == 1 {
rawKey = strings.ToLower(strings.TrimSpace(slot.Items[0].MediaType)) + "\x00" + semanticDigest(slot.Items[0].Content)
}
r.mu.Lock()
defer r.mu.Unlock()
if rawKey != "" {
if cached, ok := r.rawCache[rawKey]; ok {
return cached, nil
}
}
resolved, err := Resolve(references)
if err != nil {
return nil, err
}
if sameRegistryIdentity(r.seeded, resolved) {
if rawKey != "" {
r.rawCache[rawKey] = r.seeded
}
return r.seeded, nil
}
if cached, ok := r.cache[resolved.Digest()]; ok {
if rawKey != "" {
r.rawCache[rawKey] = cached
}
return cached, nil
}
r.cache[resolved.Digest()] = resolved
if rawKey != "" {
r.rawCache[rawKey] = resolved
}
return resolved, nil
}
func sameRegistryIdentity(first, second *Registry) bool {
if first == nil || second == nil {
return first == second
}
return first.bound == second.bound && first.digest == second.digest
}
// Resolve prepares the optional NPC registry reference. An absent slot // Resolve prepares the optional NPC registry reference. An absent slot
// produces the exact empty prompt input and no semantic registry identity. // produces the exact empty prompt input and no semantic registry identity.
func Resolve(references contracts.ReferenceSet) (*Registry, error) { func Resolve(references contracts.ReferenceSet) (*Registry, error) {

View File

@@ -5,6 +5,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"strings" "strings"
"sync"
"testing" "testing"
"unicode/utf8" "unicode/utf8"
@@ -168,6 +169,95 @@ func TestRegistryAccessorsAndLookupAreDefensive(t *testing.T) {
} }
} }
func TestResolverUsesSeededFallbackAndCachesGeneratedCanonicalRegistry(t *testing.T) {
staticContent := encodeRegistry(t, validRegistryList())
resolver, err := NewResolver(registryReference(staticContent, "file:///static.json"))
if err != nil {
t.Fatalf("NewResolver() error = %v", err)
}
if got, err := resolver.Resolve(contracts.ReferenceSet{}); err != nil || got != resolver.Seeded() {
t.Fatalf("Resolve(absent) = %p, %v, want seeded %p", got, err, resolver.Seeded())
}
generated := registryReference(append([]byte("\n"), staticContent...), "file:///generated.json")
first, err := resolver.Resolve(generated)
if err != nil {
t.Fatalf("Resolve(generated) error = %v", err)
}
second, err := resolver.Resolve(generated)
if err != nil {
t.Fatalf("Resolve(generated second) error = %v", err)
}
if first != resolver.Seeded() || second != first {
t.Fatalf("resolved registries = %p, %p, seeded %p; want seeded reuse", first, second, resolver.Seeded())
}
changed := validRegistryList()
changed.NPCs[0].Description = "A changed generated description."
changedContent := encodeRegistry(t, changed)
changedReferences := registryReference(changedContent, "file:///changed.json")
resolved, err := resolver.Resolve(changedReferences)
if err != nil {
t.Fatalf("Resolve(changed) error = %v", err)
}
changedReferences.Slots[ReferenceSlot].Items[0].Content[0] = 'X'
if resolved == first || string(resolved.CanonicalBytes()) != string(changedContent) {
t.Fatalf("changed registry = %p/%s, want independent canonical cache entry", resolved, resolved.CanonicalBytes())
}
}
func TestResolverSharesOneCachedRegistryAcrossConcurrentOperations(t *testing.T) {
content := encodeRegistry(t, validRegistryList())
resolver, err := NewResolver(contracts.ReferenceSet{})
if err != nil {
t.Fatalf("NewResolver() error = %v", err)
}
references := registryReference(content, "file:///generated.json")
const callers = 32
results := make(chan *Registry, callers)
errors := make(chan error, callers)
var wait sync.WaitGroup
for index := 0; index < callers; index++ {
wait.Add(1)
go func() {
defer wait.Done()
resolved, resolveErr := resolver.Resolve(references)
if resolveErr != nil {
errors <- resolveErr
return
}
results <- resolved
}()
}
wait.Wait()
close(results)
close(errors)
for err := range errors {
t.Fatalf("concurrent Resolve() error = %v", err)
}
var first *Registry
for resolved := range results {
if first == nil {
first = resolved
} else if resolved != first {
t.Fatalf("concurrent resolved registry %p differs from cached %p", resolved, first)
}
}
}
func TestNewResolverAllowsGeneratedDeclarationButRejectsMalformedStaticItem(t *testing.T) {
placeholder := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
ReferenceSlot: {Slot: contracts.ReferenceSlot{Name: ReferenceSlot, AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.NPCListKind}}},
}}
if _, err := NewResolver(placeholder); err != nil {
t.Fatalf("NewResolver(generated declaration) error = %v, want nil", err)
}
malformed := registryReference([]byte(`{"npcs":[`), "file:///runtime.json")
if _, err := NewResolver(malformed); err == nil || !strings.Contains(err.Error(), "invalid approved NPC JSON") {
t.Fatalf("NewResolver(malformed) error = %v, want bounded decode failure", err)
}
}
func validRegistryList() dnd.NPCList { func validRegistryList() dnd.NPCList {
return dnd.NPCList{NPCs: []dnd.NPC{{ return dnd.NPCList{NPCs: []dnd.NPC{{
ID: identity.DeriveID("Mira Thorn"), ID: identity.DeriveID("Mira Thorn"),

View File

@@ -152,79 +152,6 @@ func TestCombatNormalizerRejectsCampaignReferenceBinding(t *testing.T) {
} }
} }
func TestSequentialNPCOutputGroundsCombatAtBothStageLocalReferences(t *testing.T) {
registries := productionNPCRegistries(t)
catalog := moduleCatalog(registries)
configValue := loadSequentialCombatPipelineConfig(t)
npcEffective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-npcs", Catalog: catalog})
if err != nil {
t.Fatalf("Resolve(NPC) error = %v", err)
}
npcClient := &fakeNPCProductionLLMClient{response: npcProductionResponse{NPCs: []npcProductionRecord{
{Name: "Mira Thorn", Aliases: []string{"The Greencloak", "Mira"}, Description: "A ranger.", Relationships: []npcProductionRelationship{}, SourceRefs: []npcProductionSourceRef{{StartUnitID: 1, EndUnitID: 2}}},
{Name: "Hooded Guard", Aliases: []string{}, Description: "A sentry.", Relationships: []npcProductionRelationship{}, SourceRefs: []npcProductionSourceRef{{StartUnitID: 3, EndUnitID: 3}}},
}}}
npcOutput, err := runPreparedPipeline(t, registries, npcEffective.ResolvedPipeline, npcClient, pipeline.RunInput{RawInput: readNPCFixture(t)})
if err != nil || len(npcOutput.NormalizeOutputs) != 1 {
t.Fatalf("NPC run error = %v output = %#v, want one normalized NPC lane", err, npcOutput.NormalizeOutputs)
}
npcPayload := npcOutput.NormalizeOutputs[0].Artifact.Content
if _, err := npccodec.New().Decode(npcPayload); err != nil {
t.Fatalf("Decode(NPC output) error = %v", err)
}
npcPath := filepath.Join(t.TempDir(), "npc-run", "lanes", "npcs.json")
if err := os.MkdirAll(filepath.Dir(npcPath), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(npcPath, npcPayload, 0o600); err != nil {
t.Fatal(err)
}
combatEffective, err := configValue.Resolve(config.ResolveInput{
PipelineID: "dnd-combat",
Catalog: catalog,
ReferenceOverrides: []pipeline.ReferenceBinding{
{Stage: pipeline.StageExtract, LaneID: "combat", SlotName: "npcs", Source: npcPath, BindingSource: contracts.ReferenceBindingSourceCLI},
{Stage: pipeline.StageNormalize, LaneID: "combat", SlotName: "npcs", Source: npcPath, BindingSource: contracts.ReferenceBindingSourceCLI},
},
})
if err != nil {
t.Fatalf("Resolve(combat) error = %v", err)
}
materialized, warnings, err := pipeline.MaterializeReferences(combatEffective.ResolvedPipeline, catalog, pipeline.ReferenceMaterializationOptions{})
if err != nil || len(warnings) != 0 {
t.Fatalf("MaterializeReferences() error = %v warnings = %#v", err, warnings)
}
client := &fakeCombatLLMClient{responses: []string{combatTestTurnResponse("The Greencloak", "turn", "watches", "Hooded Guard", 1, 1)}}
output, err := runPreparedPipeline(t, registries, materialized, client, pipeline.RunInput{RawInput: readNPCFixture(t)})
if err != nil {
t.Fatalf("Run(combat) error = %v", err)
}
value, err := combatcodec.New().Decode(output.NormalizeOutputs[0].Artifact.Content)
if err != nil {
t.Fatalf("Decode(combat output) error = %v", err)
}
if len(value.CombatTurns) != 1 || value.CombatTurns[0].Actor != "Mira Thorn" || value.CombatTurns[0].Actions[0].Targets[0] != "Hooded Guard" {
t.Fatalf("combat value = %#v, want canonical actor and target", value)
}
for _, ref := range value.CombatTurns[0].SourceRefs {
if ref.SourceID != "npc-session" {
t.Fatalf("combat source ref = %#v, want transcript evidence only", ref)
}
}
if len(output.Manifest.References) != 2 {
t.Fatalf("manifest references = %#v, want both stage-local NPC provenance entries", output.Manifest.References)
}
for _, provenance := range output.Manifest.References {
if provenance.SlotName != "npcs" || provenance.LaneID != "combat" || !strings.Contains(provenance.OriginURI, "npcs.json") || provenance.BindingSource != contracts.ReferenceBindingSourceCLI {
t.Fatalf("NPC provenance = %#v, want combat stage-local CLI binding", provenance)
}
}
if len(client.requests) != 1 || string(client.requests[0].Inputs[combatextract.NPCRegistryReferenceSlot].Content) != string(npcPayload) || client.requests[0].Inputs[combatextract.NPCRegistryReferenceSlot].OriginURI != "" {
t.Fatalf("combat NPC prompt input = %#v, want canonical payload without path provenance", client.requests)
}
}
func TestCombatPreparationRejectsMalformedOrOversizedNPCReferencesBeforeExecution(t *testing.T) { func TestCombatPreparationRejectsMalformedOrOversizedNPCReferencesBeforeExecution(t *testing.T) {
registries := productionNPCRegistries(t) registries := productionNPCRegistries(t)
catalog := moduleCatalog(registries) catalog := moduleCatalog(registries)
@@ -313,19 +240,6 @@ func combatOnlyConfig() config.Config {
return cfg return cfg
} }
func loadSequentialCombatPipelineConfig(t *testing.T) config.Config {
t.Helper()
fileConfig, err := config.LoadFileConfig(repositoryPathForIntegration("examples", "dnd-npc-combat-sequential.config.yml"))
if err != nil {
t.Fatalf("LoadFileConfig() error = %v", err)
}
cfg := config.Default()
if err := cfg.ApplyFileConfig(fileConfig); err != nil {
t.Fatalf("ApplyFileConfig() error = %v", err)
}
return cfg
}
func repositoryPathForIntegration(parts ...string) string { func repositoryPathForIntegration(parts ...string) string {
_, file, _, _ := runtime.Caller(0) _, file, _, _ := runtime.Caller(0)
return filepath.Join(append([]string{filepath.Dir(file), "..", "..", ".."}, parts...)...) return filepath.Join(append([]string{filepath.Dir(file), "..", "..", ".."}, parts...)...)

View File

@@ -0,0 +1,222 @@
package integration_test
import (
"context"
"crypto/sha256"
"encoding/json"
"fmt"
"sync"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
combatcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/combatturns"
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
combatextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/combatturns"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
)
func TestNPCOutputGroundsSpellAndCombatConsumersThroughOneOperation(t *testing.T) {
registries := productionNPCRegistries(t)
catalog := moduleCatalog(registries)
configValue := loadGroundedPipelineConfig(t)
effective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-npc-grounded", Catalog: catalog})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
materialized, warnings, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalog, pipeline.ReferenceMaterializationOptions{})
if err != nil || len(warnings) != 0 {
t.Fatalf("MaterializeReferences() error = %v warnings = %#v", err, warnings)
}
client := &groundedDNDLLMClient{}
output, err := runPreparedPipeline(t, registries, materialized, client, pipeline.RunInput{
RawInput: readNPCFixture(t),
ExtractWorkers: 1,
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 3 {
t.Fatalf("run outputs = %#v rejected = %#v, want NPC, spell, and combat outputs", output.NormalizeOutputs, output.Rejected)
}
var npcPayload []byte
for _, serialized := range output.NormalizeOutputs {
if serialized.LaneID == "npcs" {
npcPayload = append([]byte(nil), serialized.Artifact.Content...)
break
}
}
if len(npcPayload) == 0 {
t.Fatal("NPC producer did not publish a canonical payload")
}
npcValue, err := npccodec.New().Decode(npcPayload)
if err != nil {
t.Fatalf("decode NPC producer payload: %v", err)
}
npcPayload, err = npccodec.New().Encode(npcValue)
if err != nil {
t.Fatalf("encode canonical NPC producer payload: %v", err)
}
canonicalDigest := digestBytes(npcPayload)
seenConsumers := map[string]bool{}
for _, request := range client.requestsSnapshot() {
if request.PromptID != spells.PromptID && request.PromptID != combatextract.PromptID {
continue
}
input := request.Inputs["npcs"]
if input.MediaType != npccodec.MediaType || input.Digest != canonicalDigest || string(input.Content) != string(npcPayload) || input.OriginURI != "" {
t.Fatalf("%s NPC prompt input = %#v, want canonical generated registry without provenance", request.PromptID, input)
}
seenConsumers[request.PromptID] = true
}
if !seenConsumers[spells.PromptID] || !seenConsumers[combatextract.PromptID] {
t.Fatalf("consumer prompt IDs = %#v, want spell and combat requests", seenConsumers)
}
provenanceCount := 0
for _, reference := range output.Manifest.References {
if reference.SlotName != "npcs" {
continue
}
provenanceCount++
if reference.Digest != canonicalDigest || reference.OriginType != "generated" {
t.Fatalf("NPC generated provenance = %#v, want canonical digest and generated origin", reference)
}
}
if provenanceCount != 3 {
t.Fatalf("NPC generated provenance count = %d, want spell extract plus combat extract/normalize", provenanceCount)
}
for _, lane := range output.Manifest.ArtifactLanes {
for _, component := range []string{"extractor", "normalizer"} {
metadata, ok := lane.Metadata[component].(map[string]any)
if ok && metadata["npc_registry_digest"] != nil {
t.Fatalf("%s %s metadata = %#v, want generated identity only in framework provenance", lane.ID, component, metadata)
}
}
}
var combatValue dnd.CombatTurnList
for _, serialized := range output.NormalizeOutputs {
switch serialized.LaneID {
case "spells":
spellValue := decodeRunnerSpellResponse(t, serialized.Artifact.Content)
if len(spellValue.SpellCasts) != 1 || spellValue.SpellCasts[0].Caster != "The Greencloak" {
t.Fatalf("spell output = %#v, want one registry-grounded-context spell", spellValue)
}
assertSpellEvidence(t, spellValue.SpellCasts[0].SourceRefs)
case "combat":
decoded, decodeErr := combatcodec.New().Decode(serialized.Artifact.Content)
if decodeErr != nil {
t.Fatalf("decode combat output: %v", decodeErr)
}
combatValue = decoded
}
}
if len(combatValue.CombatTurns) != 1 || combatValue.CombatTurns[0].Actor != "Mira Thorn" || combatValue.CombatTurns[0].Actions[0].Targets[0] != "Hooded Guard" {
t.Fatalf("combat output = %#v, want registry-normalized actor and target", combatValue)
}
assertCurrentEvidence(t, combatValue.CombatTurns[0].SourceRefs)
}
func assertSpellEvidence(t *testing.T, references []shared.SourceRefResponse) {
t.Helper()
for _, reference := range references {
if reference.SourceID != "npc-session" {
t.Fatalf("spell evidence reference = %#v, want current source only", reference)
}
}
}
func assertCurrentEvidence(t *testing.T, references []source.SourceRef) {
t.Helper()
for _, reference := range references {
if reference.SourceID != "npc-session" {
t.Fatalf("evidence reference = %#v, want current source only", reference)
}
}
}
func loadGroundedPipelineConfig(t *testing.T) config.Config {
t.Helper()
fileConfig, err := config.LoadFileConfig(repositoryPathForIntegration("internal", "modules", "integration", "testdata", "dnd_npc_grounded_pipeline.yml"))
if err != nil {
t.Fatalf("LoadFileConfig() error = %v", err)
}
cfg := config.Default()
if err := cfg.ApplyFileConfig(fileConfig); err != nil {
t.Fatalf("ApplyFileConfig() error = %v", err)
}
return cfg
}
type groundedDNDLLMClient struct {
mu sync.Mutex
requests []contracts.StructuredCompletionRequest
}
func (client *groundedDNDLLMClient) CompleteStructured(ctx context.Context, request contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
if err := ctx.Err(); err != nil {
return contracts.StructuredCompletionResponse{}, err
}
client.mu.Lock()
client.requests = append(client.requests, cloneStructuredCompletionRequest(request))
client.mu.Unlock()
var payload any
switch request.PromptID {
case npcs.PromptID:
payload = map[string]any{"npcs": []any{
map[string]any{
"name": "Mira Thorn", "aliases": []string{"The Greencloak"}, "description": "A guarded ranger.",
"relationships": []any{}, "source_refs": []any{map[string]int{"start_unit_id": 1, "end_unit_id": 1}},
},
map[string]any{
"name": "Hooded Guard", "aliases": []string{}, "description": "A sentry.",
"relationships": []any{}, "source_refs": []any{map[string]int{"start_unit_id": 3, "end_unit_id": 3}},
},
}}
case spells.PromptID:
payload = map[string]any{"spell_casts": []any{map[string]any{
"caster": "The Greencloak", "spell": "Cure Wounds", "effect": "Restores an ally.",
"narrative_description": "The Greencloak restores an ally.",
"source_refs": []any{map[string]int{"start_unit_id": 1, "end_unit_id": 1}},
}}}
case combatextract.PromptID:
payload = map[string]any{"combat_turns": []any{map[string]any{
"actor": "The Greencloak", "turn_kind": "turn", "round": 1,
"actions": []any{map[string]any{"category": "attack", "declaration": "watches", "targets": []string{"Hooded Guard"}, "resolution": "observed"}},
"summary": "The Greencloak watches the gate.",
"source_refs": []any{map[string]int{"start_unit_id": 1, "end_unit_id": 1}},
}}}
default:
return contracts.StructuredCompletionResponse{}, fmt.Errorf("unexpected grounded prompt %q", request.PromptID)
}
content, err := json.Marshal(payload)
if err != nil {
return contracts.StructuredCompletionResponse{}, err
}
if err := json.Unmarshal(content, out); err != nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate grounded response: %w", err)
}
return contracts.StructuredCompletionResponse{Content: content, Provider: "test", Model: "grounded-fake"}, nil
}
func (client *groundedDNDLLMClient) requestsSnapshot() []contracts.StructuredCompletionRequest {
client.mu.Lock()
defer client.mu.Unlock()
return append([]contracts.StructuredCompletionRequest(nil), client.requests...)
}
func digestBytes(content []byte) string {
sum := sha256.Sum256(content)
return fmt.Sprintf("sha256:%x", sum[:])
}
var _ contracts.StructuredLLMClient = (*groundedDNDLLMClient)(nil)

View File

@@ -1,130 +0,0 @@
package integration_test
import (
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
)
func TestSequentialNPCOutputCanGroundIndependentSpellRun(t *testing.T) {
registries := productionNPCRegistries(t)
catalog := moduleCatalog(registries)
configValue := loadSequentialPipelineConfig(t)
npcEffective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-npcs", Catalog: catalog})
if err != nil {
t.Fatalf("resolve NPC pipeline: %v", err)
}
npcClient := &fakeNPCProductionLLMClient{response: npcProductionResponse{NPCs: []npcProductionRecord{{
Name: "Mira Thorn",
Aliases: []string{"The Greencloak"},
Description: "A guarded ranger who watches the northern road.",
Relationships: []npcProductionRelationship{{
Target: "Captain Vale", Relationship: "reports to",
}},
SourceRefs: []npcProductionSourceRef{{StartUnitID: 1, EndUnitID: 2}},
}}}}
npcOutput, err := runPreparedPipeline(t, registries, npcEffective.ResolvedPipeline, npcClient, pipeline.RunInput{RawInput: readNPCFixture(t)})
if err != nil {
t.Fatalf("run NPC pipeline: %v", err)
}
if len(npcOutput.NormalizeOutputs) != 1 || npcOutput.NormalizeOutputs[0].LaneID != "npcs" {
t.Fatalf("NPC normalized outputs = %#v, want one npcs lane", npcOutput.NormalizeOutputs)
}
npcPayload := npcOutput.NormalizeOutputs[0].Artifact.Content
if _, err := npccodec.New().Decode(npcPayload); err != nil {
t.Fatalf("decode normalized NPC payload: %v", err)
}
npcRunDir := t.TempDir()
npcPath := filepath.Join(npcRunDir, "lanes", "npcs.json")
if err := os.MkdirAll(filepath.Dir(npcPath), 0o700); err != nil {
t.Fatalf("create NPC output directory: %v", err)
}
if err := os.WriteFile(npcPath, npcPayload, 0o600); err != nil {
t.Fatalf("write NPC output payload: %v", err)
}
spellEffective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-spells", Catalog: catalog})
if err != nil {
t.Fatalf("resolve spell pipeline: %v", err)
}
spellEffective.ResolvedPipeline.ArtifactLanes[0].ExtractReferences.Bindings = []pipeline.ReferenceBinding{{
Stage: pipeline.StageExtract,
LaneID: "spells",
SlotName: spells.NPCRegistryReferenceSlot,
Source: npcPath,
BindingSource: contracts.ReferenceBindingSourceCLI,
}}
materialized, warnings, err := pipeline.MaterializeReferences(spellEffective.ResolvedPipeline, catalog, pipeline.ReferenceMaterializationOptions{WorkingDir: npcRunDir})
if err != nil {
t.Fatalf("materialize NPC registry reference: %v", err)
}
if len(warnings) != 0 {
t.Fatalf("reference materialization warnings = %#v, want none", warnings)
}
spellClient := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{{
Caster: "Mira Thorn",
Spell: "Cure Wounds",
Effect: "Restores the injured ally.",
NarrativeDescription: "Mira Thorn restores the ally after the fight.",
SourceRefs: responseSourceRefs("spell-session", 1, 1),
}}}}
spellOutput, err := runPreparedPipeline(t, registries, materialized, spellClient, pipeline.RunInput{RawInput: readDNDSpellsFixture(t)})
if err != nil {
t.Fatalf("run spell pipeline: %v", err)
}
if len(spellOutput.NormalizeOutputs) != 1 || spellOutput.NormalizeOutputs[0].LaneID != "spells" {
t.Fatalf("spell normalized outputs = %#v, want one spells lane", spellOutput.NormalizeOutputs)
}
spellValue := decodeRunnerSpellResponse(t, spellOutput.NormalizeOutputs[0].Artifact.Content)
if len(spellValue.SpellCasts) != 1 || spellValue.SpellCasts[0].Caster != "Mira Thorn" {
t.Fatalf("spell output = %#v, want one registry-grounded caster", spellValue)
}
if len(spellValue.SpellCasts[0].SourceRefs) != 1 || spellValue.SpellCasts[0].SourceRefs[0].SourceID != "spell-session" {
t.Fatalf("spell source refs = %#v, want current spell session only", spellValue.SpellCasts[0].SourceRefs)
}
if len(spellClient.requests) != 1 {
t.Fatalf("spell LLM requests = %d, want one", len(spellClient.requests))
}
registryInput := spellClient.requests[0].Inputs[spells.NPCRegistryReferenceSlot]
if string(registryInput.Content) != string(npcPayload) || registryInput.MediaType != npccodec.MediaType || registryInput.OriginURI != "" {
t.Fatalf("spell NPC prompt input = %#v, want canonical payload without origin", registryInput)
}
if len(spellOutput.Manifest.References) != 1 {
t.Fatalf("spell manifest references = %#v, want one NPC provenance entry", spellOutput.Manifest.References)
}
provenance := spellOutput.Manifest.References[0]
if provenance.Stage != "extract" || provenance.LaneID != "spells" || provenance.SlotName != spells.NPCRegistryReferenceSlot || provenance.BindingSource != contracts.ReferenceBindingSourceCLI || !strings.Contains(provenance.OriginURI, "npcs.json") {
t.Fatalf("spell NPC provenance = %#v, want extract CLI reference provenance", provenance)
}
metadata, ok := spellOutput.Manifest.ArtifactLanes[0].Metadata["extractor"].(map[string]any)
if !ok || metadata["npc_count"] != 1 || metadata["npc_registry_digest"] != registryInput.Digest {
t.Fatalf("spell extractor metadata = %#v, want NPC count and semantic digest", spellOutput.Manifest.ArtifactLanes[0].Metadata)
}
}
func loadSequentialPipelineConfig(t *testing.T) config.Config {
t.Helper()
data, err := os.ReadFile("testdata/dnd_npc_spell_sequential_pipeline.yml")
if err != nil {
t.Fatalf("read sequential pipeline config: %v", err)
}
fileConfig, err := config.ParseFileConfigYAML(data)
if err != nil {
t.Fatalf("parse sequential pipeline config: %v", err)
}
configValue := config.Default()
if err := configValue.ApplyFileConfig(fileConfig); err != nil {
t.Fatalf("apply sequential pipeline config: %v", err)
}
return configValue
}

View File

@@ -118,7 +118,7 @@ func TestRunnerPassesPartyAndGlossaryReferencesToDNDSpellsPrompt(t *testing.T) {
raw := readDNDSpellsFixture(t) raw := readDNDSpellsFixture(t)
expectedDoc := parseDNDSpellsFixture(t, raw) expectedDoc := parseDNDSpellsFixture(t, raw)
resolved := resolveDNDSpellsPipeline(t) resolved := resolveDNDSpellsPipeline(t)
resolved.ResolvedPipeline.ArtifactLanes[0].ExtractReferences.ReferenceSet = dndSpellsReferenceSet( resolved.ResolvedPipeline.Steps[0].ArtifactLanes[0].ExtractReferences.ReferenceSet = dndSpellsReferenceSet(
"Aria: party cleric\nBorin: fighter", "Aria: party cleric\nBorin: fighter",
"Fire Bolt: evocation cantrip", "Fire Bolt: evocation cantrip",
) )

View File

@@ -0,0 +1,31 @@
version: 3
output:
directory: ./notarius-output
cache:
chunk_plans:
mode: bypass
checkpoints: {}
debug:
directory: ./notarius-debug
pipelines:
dnd-npc-grounded:
input: seriatim
steps:
- id: identify-npcs
artifacts:
npcs:
extract: dnd/npcs
normalize: dnd/npcs
- id: grounded-events
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells
combat:
extract: dnd/combat-turns
normalize: dnd/combat-turns

View File

@@ -1,25 +0,0 @@
version: 3
output:
directory: ./notarius-output
cache:
chunk_plans:
mode: bypass
checkpoints: {}
debug:
directory: ./notarius-debug
pipelines:
dnd-npcs:
input: seriatim
chunk: generic
artifacts:
npcs:
extract:
module: dnd/npcs
normalize: dnd/npcs
dnd-spells:
input: seriatim
chunk: generic
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells