Make command line references pipeline scoped

This commit is contained in:
2026-08-29 12:34:31 +00:00
parent f208dbe954
commit deebc89255
9 changed files with 824 additions and 238 deletions

View File

@@ -55,8 +55,8 @@ pipeline ID and **--input** are required.
| **--session-id id** | Override the generated prompt session identifier with a non-empty value for LLM-backed module calls. | | **--session-id id** | Override the generated prompt session identifier with a non-empty value for LLM-backed module calls. |
| **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. | | **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
| **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. | | **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |
| **--reference selector=path** | Add or replace a file reference binding. Repeatable. | | **--reference selector=path** | Add or replace external file reference bindings at pipeline, lane, chunk, or binding scope. Repeatable. |
| **--without-reference selector** | Remove a configured optional reference binding. Repeatable. | | **--without-reference selector** | Remove matching configured external reference bindings. Repeatable. |
**--chunk_cache** accepts only **auto**, **bypass**, or **refresh**. **--chunk_cache** accepts only **auto**, **bypass**, or **refresh**.
**--debug-dir**, **--output-dir**, **--session-id**, and **--debug-dir**, **--output-dir**, **--session-id**, and
@@ -81,23 +81,49 @@ guidance.
### Reference selectors ### Reference selectors
Use **--reference** only for a reference slot declared by the selected Use **--reference** only for reference slots declared by the selected
configured target. The accepted selector forms are: configured targets. Qualification narrows the scope of an override:
| Form | Target | | Form | Target |
| --- | --- | | --- | --- |
| slot=path | The unique selected target that declares slot. | | slot=path | Every selected target that declares slot. |
| chunk.slot=path | The chunker. | | chunk.slot=path | The chunker. |
| merge.slot=path | The unique selected merger that declares slot. | | lane.slot=path | Every extractor, merger, or normalizer in lane that declares slot. |
| lane.slot=path | The unique extractor, merger, or normalizer in lane that declares slot. |
| lane.extract.slot=path | The extractor in lane. | | lane.extract.slot=path | The extractor in lane. |
| lane.merge.slot=path | The merger in lane. | | lane.merge.slot=path | The merger in lane. |
| lane.normalize.slot=path | The normalizer in lane. | | lane.normalize.slot=path | The normalizer in lane. |
**--without-reference** uses the same selector forms without =path. Slot Pipeline- and lane-scoped selectors are expected to match multiple targets and
names, requiredness, and configured bindings are part of the fail if they match none. A stage-specific selector fails when its lane is not
selected or its target does not declare the slot. There is no stage-wide
`merge.slot` shorthand; name the lane when targeting a merger.
CLI bindings override configured external paths. For overlapping CLI
selectors, a binding-specific or chunk selector overrides a lane selector, and
a lane selector overrides a pipeline selector. The last occurrence wins at
equal scope. Binding and unbinding the same concrete target at equal scope is
an error; a narrower bind or unbind may create an intentional exception to a
broader action.
**--without-reference** uses the same selector forms without `=path` and
removes external bindings only. Neither flag replaces or removes a generated
artifact handoff; an external/generated collision is a resolution error.
Required slots are checked after all effective changes. CLI reference paths
are resolved relative to the process working directory, so subprocess and
service callers should use absolute paths. Slot names, accepted media types,
size limits, requiredness, and configured generated bindings are part of the
[configuration contract](config.md). [configuration contract](config.md).
For example, one shared campaign reference can reach every compatible target,
with an optional lane-specific exception:
~~~
notarius run dnd-session \
--input /data/transcript.json \
--reference party=/data/references/party.txt \
--reference npc-registry.party=/data/references/npc-party-context.txt
~~~
### Run output ### Run output
Without **--json**, standard output contains the completed pipeline ID, counts Without **--json**, standard output contains the completed pipeline ID, counts

View File

@@ -411,10 +411,15 @@ slot. A generated binding supplies one accepted normalized artifact; it does
not name a file. A configured generated dependency remains required even when not name a file. A configured generated dependency remains required even when
that consumer slot is otherwise optional. that consumer slot is otherwise optional.
Pipeline references are defaults. A matching step-local or binding-local Pipeline references are configuration defaults. A matching step-local or
external path overrides a pipeline default. Required slots must be bound after binding-local external path overrides a pipeline default. CLI reference
these configuration values and any CLI reference overrides are applied. bindings are then operational overrides of configured external paths; their
Reference paths in YAML are resolved relative to the configuration file. pipeline, lane, and binding scopes and precedence are defined by the
[CLI reference](cli.md#reference-selectors). A CLI file reference cannot
replace a configured generated artifact handoff. Required slots must be bound
after configuration and CLI reference actions are applied. Reference paths in
YAML are resolved relative to the configuration file; CLI reference paths are
resolved relative to the process working directory.
### D&D Reference Slots ### D&D Reference Slots

View File

@@ -24,7 +24,9 @@ files. Use absolute paths for service and subprocess deployments. In
particular, observe these different resolution rules: particular, observe these different resolution rules:
- reference paths in YAML are resolved relative to the Notarius configuration - reference paths in YAML are resolved relative to the Notarius configuration
file; and file;
- reference paths passed with `--reference` are resolved relative to the
Notarius process working directory; and
- `promptkit.profile_file` is resolved relative to the Notarius process working - `promptkit.profile_file` is resolved relative to the Notarius process working
directory. directory.
@@ -73,9 +75,21 @@ notarius run dnd-session \
--config /absolute/path/to/notarius.yml \ --config /absolute/path/to/notarius.yml \
--input /absolute/path/to/transcripts/final.trimmed.json \ --input /absolute/path/to/transcripts/final.trimmed.json \
--output-dir /absolute/path/to/notarius-output \ --output-dir /absolute/path/to/notarius-output \
--reference party=/absolute/path/to/references/party.txt \
--reference players=/absolute/path/to/references/players.txt \
--reference glossary=/absolute/path/to/references/glossary.txt \
--reference spell_catalog=/absolute/path/to/references/spells.json \
--json --json
``` ```
Each unqualified reference is pipeline-scoped: Notarius supplies it to every
selected D&D target that declares the slot. A deployment may omit an optional
reference it does not maintain, and may use the lane- or binding-qualified
forms from the [CLI reference](../cli.md#reference-selectors) for an exceptional
override. The registry, scene-description, combat-turn, and NPC-occurrence
references declared between ordered steps in the complete configuration are
generated artifacts. Do not pass those handoffs on the CLI.
The caller should: The caller should:
- capture stdout and stderr separately; - capture stdout and stderr separately;

View File

@@ -31,13 +31,36 @@ notarius run pipeline-id \
``` ```
Use absolute paths for supplied input, configuration, output-root, and Use absolute paths for supplied input, configuration, output-root, and
reference files. Notarius generates a stable prompt session for the resolved reference files. Pass each external reference as its own argument-vector pair;
input module and exact input bytes. Pass **--session-id** only when intentionally do not construct and invoke a shell command. An unqualified reference selector
grouping different invocations under a different session. Supply credentials supplies that file to every compatible selected target. Lane and stage
through Notarius's documented configuration and environment mechanisms, never qualification are available for exceptional overrides, while generated
as command-line arguments or generated secret-bearing configuration. In same-run references remain part of configured pipeline composition. The
particular, a session identifier is provider-visible and is not a credential [CLI reference](../cli.md#reference-selectors) owns the exact selector and
mechanism. precedence contract.
The maintained D&D subprocess workflow uses this facility for campaign context:
```sh
notarius run dnd-session \
--config /absolute/path/to/notarius.yml \
--input /absolute/path/to/transcripts/final.trimmed.json \
--output-dir /absolute/path/to/notarius-output \
--reference party=/absolute/path/to/references/party.txt \
--reference players=/absolute/path/to/references/players.txt \
--reference glossary=/absolute/path/to/references/glossary.txt \
--reference spell_catalog=/absolute/path/to/references/spells.json \
--json
```
Only pass the external references available to and desired by the deployment.
Notarius generates a stable prompt session for the resolved input module and
exact input bytes; reference changes do not change it. Pass **--session-id**
only when intentionally grouping different invocations under a different
session. Supply credentials through Notarius's documented configuration and
environment mechanisms, never as command-line arguments or generated
secret-bearing configuration. In particular, a session identifier is
provider-visible and is not a credential mechanism.
Wait for the process before interpreting standard output. Only an exit status Wait for the process before interpreting standard output. Only an exit status
of 0 permits decoding the receipt. On a nonzero exit, retain standard error for of 0 permits decoding the receipt. On a nonzero exit, retain standard error for

View File

@@ -115,6 +115,16 @@ to checkpoint identity and `pipeline.RunInput`. The public flag and stability
contract are defined by the [CLI reference](../cli.md#run); framework and LLM contract are defined by the [CLI reference](../cli.md#run); framework and LLM
packages only transport the supplied value. packages only transport the supplied value.
The CLI also owns the scope grammar for reference flags. It enumerates the
selected chunk and lane targets from registered module specifications, expands
pipeline- and lane-scoped actions into exact stage-and-lane bindings, and
resolves overlapping bind and unbind actions by specificity before calling
configuration resolution. The generic pipeline therefore receives only exact
`ReferenceBinding` and `ReferenceUnbind` values and has no knowledge of CLI
selector syntax. Configuration resolution retains ownership of configured
external/generated conflicts, required slots, and module compatibility; file
materialization still occurs afterward.
For `run --json`, the CLI constructs and encodes its private run-result receipt For `run --json`, the CLI constructs and encodes its private run-result receipt
after a successful runner result is available, before it publishes logical after a successful runner result is available, before it publishes logical
output files. It writes the prepared receipt to standard output only after output files. It writes the prepared receipt to standard output only after
@@ -166,8 +176,9 @@ is discoverable.
- **internal/cli/production_contract_test.go** covers registrar composition, - **internal/cli/production_contract_test.go** covers registrar composition,
production catalog contents, assets, and representative configuration production catalog contents, assets, and representative configuration
validation. validation.
- **internal/cli/reference_contract_test.go** covers CLI reference overrides, - **internal/cli/reference_contract_test.go** covers scoped CLI reference
origin separation, and materialization boundaries. expansion, specificity, bind/unbind conflicts, generated-reference
protection, origin separation, and materialization boundaries.
- **internal/cli/state_hardening_test.go** covers safe run identity, state - **internal/cli/state_hardening_test.go** covers safe run identity, state
roots, and failure ordering. roots, and failure ordering.

View File

@@ -97,6 +97,13 @@ codec, checks its complete schema and media identity, and records a content
digest plus bounded producer provenance. A missing, ambiguous, invalid, or digest plus bounded producer provenance. A missing, ambiguous, invalid, or
incompatible producer prevents the consumer step from starting. incompatible producer prevents the consumer step from starting.
Resolution receives only exact stage-and-lane operational reference overrides.
The CLI may offer broader pipeline- or lane-scoped selectors, but expands and
arbitrates those before entering the framework. External overrides are applied
after configured external defaults and local bindings. They cannot coexist
with a generated binding for the same target and slot, and external unbinds do
not remove generated handoffs.
## Execution And Ordering ## Execution And Ordering
The runner validates its input, installs no-op state collaborators when none The runner validates its input, installs no-op state collaborators when none

View File

@@ -0,0 +1,258 @@
# Pipeline-Scoped CLI References
## Purpose
Make command-line reference binding match the pipeline-level mental model used
by configuration and by subprocess callers. A caller should be able to supply
each shared external reference once, while retaining explicit syntax for the
less common case in which one lane or one stage binding needs a different
source.
## Motivation And Current Problem
Notarius already accepts repeatable `--reference selector=path` and
`--without-reference selector` flags. The current unqualified form,
`--reference slot=path`, succeeds only when exactly one selected chunk, extract,
merge, or normalize target declares that slot. If several targets consume a
shared reference such as `party`, `glossary`, or `spell_catalog`, the command is
rejected as ambiguous and the caller must repeat stage-qualified bindings.
That behavior is internally precise but does not match the public configuration
model. A reference declared at pipeline scope is shared with every compatible
target, while target-local configuration provides the exceptional override.
It is therefore surprising for the least-qualified CLI syntax to mean "find one
unique target" rather than "supply this reference to the pipeline."
The mismatch is especially costly for subprocess use. The primary complete D&D
workflow expects an orchestrator to provide one transcript, one output root,
and a small collection of campaign reference files. Requiring the orchestrator
to know and enumerate every internal consumer of those shared files couples it
to lane composition, makes commands needlessly long, and creates maintenance
work whenever another compatible D&D module is added.
## Target CLI Contract
`--reference` remains repeatable, but selector qualification expresses scope:
| Form | Target scope |
| --- | --- |
| `slot=path` | Every selected pipeline target that declares `slot`. |
| `chunk.slot=path` | The selected chunk binding. |
| `lane.slot=path` | Every extract, merge, or normalize binding in `lane` that declares `slot`. |
| `lane.extract.slot=path` | The extract binding in `lane`. |
| `lane.merge.slot=path` | The merge binding in `lane`. |
| `lane.normalize.slot=path` | The normalize binding in `lane`. |
The current stage-wide `merge.slot=path` shorthand is removed. A merge-specific
override must name its lane as `lane.merge.slot=path`; this keeps the grammar
hierarchical and avoids another uniqueness-dependent selector.
`--without-reference` uses the same selector forms without `=path`. It removes
matching external bindings only. It never removes a generated artifact handoff,
and resolution continues to reject a missing required reference.
Broad selectors are expected to match multiple compatible targets. They fail
when they match no selected target, when the named lane is not selected, or
when a stage-qualified target does not declare the slot. Errors should identify
the selector and relevant scope without requiring callers to understand private
resolver structures.
## Precedence And Conflict Policy
Command-line references are operational overrides and take precedence over
external file references supplied at pipeline, step, lane, or binding scope in
configuration. Among CLI selectors that affect the same concrete target:
1. an exact `lane.stage.slot` or `chunk.slot` selector wins over a lane-scoped
selector;
2. a lane-scoped selector wins over a pipeline-scoped selector; and
3. the last occurrence wins among selectors with equal scope and action.
This ordering lets an orchestrator provide shared defaults once and express
only genuine exceptions:
```sh
--reference party=/refs/party.txt \
--reference npc-registry.party=/refs/npc-party-context.txt
```
Binding and unbinding the same concrete target at the same specificity is a
configuration error rather than an argument-order-dependent result. A
more-specific unbind may carve an exception out of a broader binding, and a
more-specific binding may restore an exception to a broader unbind.
Generated artifact references remain a distinct source form. A CLI file
reference must not silently replace, remove, or coexist with a generated
handoff for the same concrete target and slot. Resolution fails with a
target-specific conflict and directs the caller to narrow or remove the CLI
selector. Same-run generated registry and eligibility handoffs in the complete
D&D pipeline therefore remain controlled by pipeline composition.
CLI reference paths continue to resolve relative to the process working
directory and retain CLI provenance. Subprocess guidance must recommend
absolute paths. The reference source is auxiliary context and does not affect
the generated prompt session identifier.
## Subprocess Target State
Passing shared references is a first-class part of the documented subprocess
workflow. The complete D&D invocation should have this shape:
```sh
notarius run dnd-session \
--config /absolute/path/to/notarius.yml \
--input /absolute/path/to/transcripts/final.trimmed.json \
--output-dir /absolute/path/to/notarius-output \
--reference party=/absolute/path/to/references/party.txt \
--reference players=/absolute/path/to/references/players.txt \
--reference glossary=/absolute/path/to/references/glossary.txt \
--reference spell_catalog=/absolute/path/to/references/spells.json \
--json
```
Only references actually available to and desired by the deployment need to be
passed. The selected modules continue to determine accepted slot names, media
types, requiredness, and size limits. The orchestrator passes references as
separate argument-vector elements rather than building a shell command, captures
standard output and standard error separately, checks exit status, and uses the
JSON receipt's `output_directory` to locate the generated run bundle.
The documentation must clearly distinguish:
- the input file supplied by `--input`;
- the output *root* supplied by `--output-dir` and the run-specific output path
returned in the receipt;
- external file references supplied by `--reference`; and
- generated references produced and consumed within the configured ordered
pipeline.
## Required Implementation Changes
### CLI selector model and parsing
- Replace the unique-target interpretation of unqualified and lane-qualified
selectors with explicit pipeline, lane, chunk, and exact-binding scopes.
- Remove parsing and help text for the stage-wide `merge.slot` form.
- Preserve repeatable flag handling, non-empty selector/path checks, supported
stage names, and syntax-error exit classification.
- Represent selector scope explicitly enough that precedence and diagnostics do
not depend on inferring intent from empty fields.
### Target expansion and precedence
- Refactor CLI reference resolution to return every compatible selected target
for a broad selector rather than requiring uniqueness.
- Resolve overlapping bind and unbind requests into one deterministic action
per concrete target and slot using the precedence policy above.
- Produce exact `pipeline.ReferenceBinding` and `pipeline.ReferenceUnbind`
values after CLI scoping is resolved. Keep broad-selector policy at the CLI
boundary rather than adding CLI grammar or D&D knowledge to the generic
pipeline framework.
- Preserve CLI provenance and the existing rule that command-line paths are
materialized relative to the working directory.
- Detect generated-reference conflicts before file access or stage execution
and retain required-slot validation after effective bindings are known.
- Ensure lane selection is honored for compact pipelines. Explicit ordered
pipelines continue to use their complete configured lane set because they do
not support `--only`.
### Public and internal documentation
- Rewrite the reference-selector section of `docs/cli.md`, which is the
canonical owner of flag syntax, selector semantics, precedence, and command
errors.
- Update `docs/consumers/subprocess.md` so reference-bearing invocation is part
of the primary subprocess workflow rather than an unillustrated aside.
- Update `docs/consumers/dnd-pipeline.md` with the complete D&D subprocess
invocation and explain which references are external versus generated.
- Lightly update `docs/config.md` to link the configuration hierarchy to the
CLI override contract without duplicating CLI syntax.
- Update `docs/internal/cli.md` and, only where necessary, the reference
resolution discussion in `docs/internal/pipeline.md` to describe the
implemented expansion and precedence boundary.
- Do not expand the README quickstart or duplicate complete configuration
files in prose. The maintained example configuration remains the canonical
copyable pipeline definition.
No ADR is required. This is a deliberate public CLI usability correction that
fits the existing architectural decisions: configuration remains centralized,
operational overrides remain explicit, generated handoffs remain part of
pipeline composition, and broad CLI syntax is translated into existing exact
framework bindings at the application boundary. The CLI and configuration
references are the durable owners of the resulting current behavior once it is
implemented.
## Testing And Validation
Add lean offline behavioral coverage at the CLI contract boundary for:
- one pipeline-scoped reference reaching multiple compatible chunk and lane
targets;
- one lane-scoped reference reaching both extract and normalize bindings;
- exact-binding and lane-scoped exceptions overriding broader values;
- final-occurrence behavior at equal specificity;
- broad selectors with no matches and selectors naming unselected lanes or
undeclared slots;
- bind/unbind specificity and same-specificity conflicts;
- rejection of external/generated conflicts and protection of required
generated handoffs;
- working-directory resolution and CLI provenance for expanded bindings; and
- removal of the `merge.slot` shorthand.
Prefer package-level resolution and representative `RunWithOptions` tests over
duplicating the full selector matrix through end-to-end fixtures. Existing
tests that encode unique-target ambiguity should be rewritten or removed rather
than retained as historical change detectors. Do not assert complete error
strings when stable classification and a concise semantic fragment provide
sufficient confidence.
Validation for the completed change should include:
```sh
go test ./internal/cli ./internal/core/config ./internal/framework/pipeline
go test ./...
go vet ./...
go build ./cmd/notarius
```
Also validate both maintained example configurations against their selected
pipelines, verify every changed documentation link, and manually review the
D&D subprocess command against the actual flag parser and complete example
configuration.
## Non-Goals
- Passing reference contents directly in command-line arguments or environment
variables.
- Adding a reference-manifest file, glob syntax, wildcard selectors, or a
second pipeline-reference flag.
- Allowing the CLI to define undeclared reference slots or bypass module media
type, size, UTF-8, requiredness, or materialization validation.
- Replacing generated same-run artifacts with external files or changing
ordered pipeline dependencies.
- Changing configuration-file reference path resolution, prompt-session
derivation, output directory allocation, or the run-result receipt.
- Adding D&D-specific selector behavior to generic CLI or pipeline packages.
- Preserving uniqueness-dependent `slot` or `lane.slot` behavior as an alias.
## Acceptance Criteria
- A subprocess caller can supply `party`, `players`, `glossary`, and
`spell_catalog` once each and have every compatible selected D&D target
receive the corresponding external file.
- `slot=path` has pipeline scope and never fails merely because several selected
targets declare the slot.
- Lane and exact-binding selectors provide deterministic, documented
exceptions with specificity-based precedence.
- A broad selector that matches nothing is rejected before source parsing or
LLM work.
- CLI external references override configured external paths but never silently
replace or remove generated artifact references.
- All effective CLI bindings retain correct provenance and absolute-path
materialization behavior.
- `docs/cli.md` owns the exact public syntax, and both subprocess guides present
shared CLI references as the primary orchestration workflow.
- Current behavior outside `docs/roadmap/` is not documented until the code
lands, and no complete example is duplicated outside `examples/`.
- Focused tests, repository-wide tests, vetting, build, maintained configuration
validation, and documentation-link review pass.

View File

@@ -4,6 +4,7 @@ import (
"bytes" "bytes"
"os" "os"
"path/filepath" "path/filepath"
"slices"
"strings" "strings"
"testing" "testing"
@@ -16,18 +17,14 @@ func TestReferenceSelectorsParseAndApplyAllDocumentedForms(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
selector string selector string
only []string want []string
wantStage pipeline.ModuleStage
wantLane string
wantSlot string
}{ }{
{name: "flat", selector: "alpha-slot", wantStage: pipeline.StageExtract, wantLane: "alpha", wantSlot: "alpha-slot"}, {name: "pipeline", selector: "shared", want: []string{"alpha.extract.shared", "alpha.merge.shared", "alpha.normalize.shared", "beta.extract.shared", "beta.merge.shared", "beta.normalize.shared"}},
{name: "chunk", selector: "chunk.chunk-slot", wantStage: pipeline.StageChunk, wantSlot: "chunk-slot"}, {name: "chunk", selector: "chunk.chunk-slot", want: []string{"chunk.chunk-slot"}},
{name: "merge", selector: "merge.alpha-merge", only: []string{"alpha"}, wantStage: pipeline.StageMerge, wantLane: "alpha", wantSlot: "alpha-merge"}, {name: "lane", selector: "alpha.shared", want: []string{"alpha.extract.shared", "alpha.merge.shared", "alpha.normalize.shared"}},
{name: "lane", selector: "alpha.alpha-slot", wantStage: pipeline.StageExtract, wantLane: "alpha", wantSlot: "alpha-slot"}, {name: "lane extract", selector: "alpha.extract.alpha-slot", want: []string{"alpha.extract.alpha-slot"}},
{name: "lane extract", selector: "alpha.extract.alpha-slot", wantStage: pipeline.StageExtract, wantLane: "alpha", wantSlot: "alpha-slot"}, {name: "lane merge", selector: "alpha.merge.alpha-merge", want: []string{"alpha.merge.alpha-merge"}},
{name: "lane merge", selector: "alpha.merge.alpha-merge", wantStage: pipeline.StageMerge, wantLane: "alpha", wantSlot: "alpha-merge"}, {name: "lane normalize", selector: "alpha.normalize.alpha-normalize", want: []string{"alpha.normalize.alpha-normalize"}},
{name: "lane normalize", selector: "alpha.normalize.alpha-normalize", wantStage: pipeline.StageNormalize, wantLane: "alpha", wantSlot: "alpha-normalize"},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
@@ -37,70 +34,135 @@ func TestReferenceSelectorsParseAndApplyAllDocumentedForms(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
overrides, _, err := resolveCLIReferenceRequests(cfg, "demo", tt.only, catalog, []cliReferenceRequest{{Selector: selector, Source: "reference.txt"}}, nil) overrides, _, err := resolveCLIReferenceRequests(cfg, "demo", nil, catalog, []cliReferenceRequest{{Selector: selector, Source: "reference.txt"}}, nil)
if err != nil { if err != nil {
t.Fatalf("resolve selector: %v", err) t.Fatalf("resolve selector: %v", err)
} }
if len(overrides) != 1 { if got := referenceContractBindingLabels(overrides); !slices.Equal(got, tt.want) {
t.Fatalf("overrides = %#v, want one binding", overrides) t.Fatalf("binding targets = %#v, want %#v", got, tt.want)
}
for _, binding := range overrides {
if binding.Source != "reference.txt" || binding.BindingSource != contracts.ReferenceBindingSourceCLI {
t.Fatalf("binding = %#v, want CLI source", binding)
} }
got := overrides[0]
if got.Stage != tt.wantStage || got.LaneID != tt.wantLane || got.SlotName != tt.wantSlot || got.BindingSource != contracts.ReferenceBindingSourceCLI {
t.Fatalf("binding = %#v, want %s/%s/%s from CLI", got, tt.wantStage, tt.wantLane, tt.wantSlot)
} }
}) })
} }
} }
func TestReferenceSelectorsRejectAmbiguityWithSpecificSuggestions(t *testing.T) { func TestReferenceSelectorSpecificityAndFinalOccurrenceChooseConcreteBindings(t *testing.T) {
cfg := referenceContractConfig() cfg := referenceContractConfig()
catalog := referenceContractCatalog(t, true, true) catalog := referenceContractCatalog(t, true, true)
for _, tt := range []struct { requests := []cliReferenceRequest{
name string {Selector: mustParseReferenceSelector(t, "shared", "--reference"), Source: "pipeline-first.txt"},
selector string {Selector: mustParseReferenceSelector(t, "shared", "--reference"), Source: "pipeline-final.txt"},
want []string {Selector: mustParseReferenceSelector(t, "alpha.shared", "--reference"), Source: "lane.txt"},
}{ {Selector: mustParseReferenceSelector(t, "alpha.extract.shared", "--reference"), Source: "binding.txt"},
{name: "flat shared slot", selector: "shared", want: []string{"alpha.extract.shared", "beta.extract.shared"}}, }
{name: "lane shared slot", selector: "alpha.shared", want: []string{"alpha.extract.shared", "alpha.merge.shared", "alpha.normalize.shared"}}, overrides, unbinds, err := resolveCLIReferenceRequests(cfg, "demo", nil, catalog, requests, nil)
{name: "all mergers", selector: "merge.shared", want: []string{"alpha.merge.shared", "beta.merge.shared"}},
} {
t.Run(tt.name, func(t *testing.T) {
selector, err := parseReferenceSelector(tt.selector, "--reference")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
_, _, err = resolveCLIReferenceRequests(cfg, "demo", nil, catalog, []cliReferenceRequest{{Selector: selector, Source: "reference.txt"}}, nil) if len(unbinds) != 0 {
if err == nil { t.Fatalf("unbinds = %#v, want none", unbinds)
t.Fatal("resolve selector succeeded, want ambiguity error")
} }
for _, fragment := range tt.want { want := map[string]string{
if !strings.Contains(err.Error(), fragment) { "alpha.extract.shared": "binding.txt",
t.Fatalf("error = %q, want suggestion %q", err, fragment) "alpha.merge.shared": "lane.txt",
"alpha.normalize.shared": "lane.txt",
"beta.extract.shared": "pipeline-final.txt",
"beta.merge.shared": "pipeline-final.txt",
"beta.normalize.shared": "pipeline-final.txt",
} }
for _, binding := range overrides {
label := referenceContractBindingLabel(binding)
if binding.Source != want[label] {
t.Fatalf("binding %s source = %q, want %q", label, binding.Source, want[label])
} }
}) delete(want, label)
}
if len(want) != 0 {
t.Fatalf("missing bindings: %#v", want)
} }
} }
func TestReferenceSelectorsRespectSelectedLanesBeforeMaterialization(t *testing.T) { func TestCompleteDNDSharedCLIReferencesExpandAcrossCompatibleTargets(t *testing.T) {
cfg := loadMaintainedExample(t, repositoryPath("examples", "dnd-complete.config.yml"))
catalog := catalogFromRegistries(productionTestComponents(t).registries)
sources := map[string]string{
"party": "/references/party.txt",
"players": "/references/players.txt",
"glossary": "/references/glossary.txt",
"spell_catalog": "/references/spells.json",
}
requests := make([]cliReferenceRequest, 0, len(sources))
for _, slot := range []string{"party", "players", "glossary", "spell_catalog"} {
requests = append(requests, cliReferenceRequest{
Selector: mustParseReferenceSelector(t, slot, "--reference"),
Source: sources[slot],
})
}
overrides, unbinds, err := resolveCLIReferenceRequests(cfg, "dnd-session", nil, catalog, requests, nil)
if err != nil {
t.Fatalf("expand complete D&D references: %v", err)
}
if len(unbinds) != 0 {
t.Fatalf("unbinds = %#v, want none", unbinds)
}
actual := make(map[string]pipeline.ReferenceBinding, len(overrides))
for _, binding := range overrides {
actual[referenceContractBindingLabel(binding)] = binding
}
targets, err := selectedReferenceTargets(cfg, "dnd-session", nil, catalog)
if err != nil {
t.Fatal(err)
}
matched := make(map[string]int, len(sources))
for _, target := range targets {
for slot, sourcePath := range sources {
if _, ok := target.slots[slot]; !ok {
continue
}
matched[slot]++
label := targetLabel(target) + "." + slot
binding, ok := actual[label]
if !ok || binding.Source != sourcePath || binding.BindingSource != contracts.ReferenceBindingSourceCLI {
t.Fatalf("binding %q = %#v, want CLI source %q", label, binding, sourcePath)
}
}
}
for slot := range sources {
if matched[slot] < 2 {
t.Fatalf("reference %q matched %d target(s), want a shared D&D reference", slot, matched[slot])
}
}
if _, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalog, ReferenceOverrides: overrides}); err != nil {
t.Fatalf("resolve complete D&D CLI references: %v", err)
}
}
func TestReferenceSelectorsRejectInvalidOrUnselectedScopesBeforeMaterialization(t *testing.T) {
cfg := referenceContractConfig() cfg := referenceContractConfig()
catalog := referenceContractCatalog(t, true, true) catalog := referenceContractCatalog(t, true, true)
for _, tt := range []struct { for _, tt := range []struct {
name string name string
selector string selector string
only []string
want string want string
}{ }{
{name: "unselected lane", selector: "beta.extract.beta-slot", want: `reference lane "beta" is not selected`}, {name: "pipeline slot", selector: "missing", want: `reference slot "missing" is not declared by any selected target`},
{name: "lane slot", selector: "alpha.missing", want: `reference slot "missing" is not declared by selected lane "alpha"`},
{name: "binding slot", selector: "alpha.extract.missing", want: `reference slot "missing" is not declared`},
{name: "former merge shorthand", selector: "merge.shared", want: `reference lane "merge" is not selected`},
{name: "unselected lane", selector: "beta.extract.beta-slot", only: []string{"alpha"}, want: `reference lane "beta" is not selected`},
{name: "unknown lane", selector: "missing.extract.beta-slot", want: `reference lane "missing" is not selected`}, {name: "unknown lane", selector: "missing.extract.beta-slot", want: `reference lane "missing" is not selected`},
} { } {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
selector, err := parseReferenceSelector(tt.selector, "--reference") selector := mustParseReferenceSelector(t, tt.selector, "--reference")
if err != nil { _, _, err := resolveCLIReferenceRequests(cfg, "demo", tt.only, catalog, []cliReferenceRequest{{Selector: selector, Source: filepath.Join(t.TempDir(), "missing.txt")}}, nil)
t.Fatal(err)
}
_, _, err = resolveCLIReferenceRequests(cfg, "demo", []string{"alpha"}, catalog, []cliReferenceRequest{{Selector: selector, Source: filepath.Join(t.TempDir(), "missing.txt")}}, nil)
if err == nil || !strings.Contains(err.Error(), tt.want) || strings.Contains(err.Error(), "missing.txt") { if err == nil || !strings.Contains(err.Error(), tt.want) || strings.Contains(err.Error(), "missing.txt") {
t.Fatalf("error = %v, want selection failure before file access", err) t.Fatalf("error = %v, want selection failure containing %q before file access", err, tt.want)
} }
}) })
} }
@@ -131,40 +193,50 @@ func TestReferenceSyntaxErrorsReturnTwo(t *testing.T) {
} }
} }
func TestReferenceOverridesUseFinalExactTargetBinding(t *testing.T) { func TestReferenceBindAndUnbindSpecificity(t *testing.T) {
cfg := referenceContractConfig() cfg := referenceContractConfig()
catalog := referenceContractCatalog(t, true, true) catalog := referenceContractCatalog(t, true, true)
alphaShared, err := parseReferenceSelector("alpha.extract.shared", "--reference") t.Run("specific unbind carves out broad binding", func(t *testing.T) {
overrides, unbinds, err := resolveCLIReferenceRequests(cfg, "demo", nil, catalog,
[]cliReferenceRequest{{Selector: mustParseReferenceSelector(t, "shared", "--reference"), Source: "shared.txt"}},
[]cliReferenceUnbindRequest{{Selector: mustParseReferenceSelector(t, "alpha.extract.shared", "--without-reference")}},
)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
betaShared, err := parseReferenceSelector("beta.extract.shared", "--reference") if got := referenceContractBindingLabels(overrides); slices.Contains(got, "alpha.extract.shared") || len(got) != 5 {
t.Fatalf("overrides = %#v, want all shared targets except alpha extract", got)
}
if got := referenceContractUnbindLabels(unbinds); !slices.Equal(got, []string{"alpha.extract.shared"}) {
t.Fatalf("unbinds = %#v, want alpha extract", got)
}
})
t.Run("specific binding restores broad unbind", func(t *testing.T) {
overrides, unbinds, err := resolveCLIReferenceRequests(cfg, "demo", nil, catalog,
[]cliReferenceRequest{{Selector: mustParseReferenceSelector(t, "alpha.extract.shared", "--reference"), Source: "alpha.txt"}},
[]cliReferenceUnbindRequest{{Selector: mustParseReferenceSelector(t, "shared", "--without-reference")}},
)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
overrides, unbinds, err := resolveCLIReferenceRequests(cfg, "demo", nil, catalog, []cliReferenceRequest{ if got := referenceContractBindingLabels(overrides); !slices.Equal(got, []string{"alpha.extract.shared"}) {
{Selector: alphaShared, Source: "alpha-first.txt"}, t.Fatalf("overrides = %#v, want alpha extract", got)
{Selector: alphaShared, Source: "alpha-final.txt"},
{Selector: betaShared, Source: "beta-only.txt"},
}, nil)
if err != nil {
t.Fatal(err)
} }
if len(unbinds) != 0 { if got := referenceContractUnbindLabels(unbinds); slices.Contains(got, "alpha.extract.shared") || len(got) != 5 {
t.Fatalf("unbinds = %#v, want none", unbinds) t.Fatalf("unbinds = %#v, want all shared targets except alpha extract", got)
} }
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "demo", Catalog: catalog, ReferenceOverrides: overrides}) })
if err != nil {
t.Fatalf("resolve pipeline: %v", err) t.Run("same specificity conflicts", func(t *testing.T) {
} _, _, err := resolveCLIReferenceRequests(cfg, "demo", nil, catalog,
alpha := referenceContractLane(t, effective.ResolvedPipeline, "alpha") []cliReferenceRequest{{Selector: mustParseReferenceSelector(t, "alpha.shared", "--reference"), Source: "alpha.txt"}},
beta := referenceContractLane(t, effective.ResolvedPipeline, "beta") []cliReferenceUnbindRequest{{Selector: mustParseReferenceSelector(t, "alpha.shared", "--without-reference")}},
if source := referenceContractBindingSource(alpha.ExtractReferences.Bindings, "shared"); source != "alpha-final.txt" { )
t.Fatalf("alpha shared source = %q, want final exact-target override", source) if err == nil || !strings.Contains(err.Error(), "same specificity") {
} t.Fatalf("error = %v, want same-specificity conflict", err)
if source := referenceContractBindingSource(beta.ExtractReferences.Bindings, "shared"); source != "beta-only.txt" {
t.Fatalf("beta shared source = %q, want target-specific override", source)
} }
})
} }
func TestReferenceUnbindsRemoveOptionalAndProtectRequiredSlots(t *testing.T) { func TestReferenceUnbindsRemoveOptionalAndProtectRequiredSlots(t *testing.T) {
@@ -257,6 +329,52 @@ func TestReferenceMaterializationSeparatesCLIAndConfigPathOrigins(t *testing.T)
} }
} }
func TestPipelineScopedCLIReferenceProtectsGeneratedHandoff(t *testing.T) {
cfg := referenceContractConfig()
profile := cfg.Pipelines["demo"]
alpha := profile.Artifacts["alpha"]
beta := profile.Artifacts["beta"]
alpha.Extract.References["shared"] = pipeline.GeneratedReference("produce", "beta")
profile.Artifacts = nil
profile.Steps = []pipeline.PipelineStepProfile{
{ID: "produce", Artifacts: map[string]pipeline.ArtifactLaneProfile{"beta": beta}},
{ID: "consume", Artifacts: map[string]pipeline.ArtifactLaneProfile{"alpha": alpha}},
}
cfg.Pipelines["demo"] = profile
catalog := referenceContractCatalog(t, true, true)
t.Run("binding conflicts before file access", func(t *testing.T) {
overrides, _, err := resolveCLIReferenceRequests(cfg, "demo", nil, catalog, []cliReferenceRequest{{
Selector: mustParseReferenceSelector(t, "shared", "--reference"),
Source: filepath.Join(t.TempDir(), "never-read.json"),
}}, nil)
if err != nil {
t.Fatalf("expand CLI reference: %v", err)
}
_, err = cfg.Resolve(config.ResolveInput{PipelineID: "demo", Catalog: catalog, ReferenceOverrides: overrides})
if err == nil || !strings.Contains(err.Error(), "conflicting generated and external bindings") || strings.Contains(err.Error(), "never-read.json") {
t.Fatalf("resolve error = %v, want generated/external conflict before file access", err)
}
})
t.Run("unbind leaves generated source intact", func(t *testing.T) {
_, unbinds, err := resolveCLIReferenceRequests(cfg, "demo", nil, catalog, nil, []cliReferenceUnbindRequest{{
Selector: mustParseReferenceSelector(t, "shared", "--without-reference"),
}})
if err != nil {
t.Fatalf("expand CLI unbind: %v", err)
}
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "demo", Catalog: catalog, ReferenceUnbinds: unbinds})
if err != nil {
t.Fatalf("resolve generated reference with CLI unbind: %v", err)
}
binding := referenceContractFindBinding(referenceContractLane(t, effective.ResolvedPipeline, "alpha").ExtractReferences.Bindings, "shared")
if binding == nil || binding.Artifact == nil || binding.Artifact.Step != "produce" || binding.Artifact.Lane != "beta" {
t.Fatalf("generated binding = %#v, want preserved produce/beta handoff", binding)
}
})
}
func TestReferenceTargetLookupUsesArtifactVariantsAndReportsMissingContext(t *testing.T) { func TestReferenceTargetLookupUsesArtifactVariantsAndReportsMissingContext(t *testing.T) {
cfg := referenceContractConfig() cfg := referenceContractConfig()
full := referenceContractCatalog(t, true, true) full := referenceContractCatalog(t, true, true)
@@ -368,7 +486,7 @@ func referenceContractCatalog(t *testing.T, includeBetaMerger, includeBetaNormal
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil })) register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{})) register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{}))
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{})) register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{}))
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil })) register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared", AcceptedArtifactKinds: []contracts.ArtifactKind{referenceContractKindBeta}, AcceptedMediaTypes: []string{"application/json"}}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil })) register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil })) register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
if includeBetaMerger { if includeBetaMerger {
@@ -444,6 +562,42 @@ func referenceContractBindingSource(bindings []pipeline.ReferenceBinding, slot s
return "" return ""
} }
func mustParseReferenceSelector(t *testing.T, value, flagName string) cliReferenceSelector {
t.Helper()
selector, err := parseReferenceSelector(value, flagName)
if err != nil {
t.Fatal(err)
}
return selector
}
func referenceContractBindingLabels(bindings []pipeline.ReferenceBinding) []string {
labels := make([]string, 0, len(bindings))
for _, binding := range bindings {
labels = append(labels, referenceContractBindingLabel(binding))
}
return labels
}
func referenceContractBindingLabel(binding pipeline.ReferenceBinding) string {
if binding.Stage == pipeline.StageChunk {
return "chunk." + binding.SlotName
}
return binding.LaneID + "." + string(binding.Stage) + "." + binding.SlotName
}
func referenceContractUnbindLabels(unbinds []pipeline.ReferenceUnbind) []string {
labels := make([]string, 0, len(unbinds))
for _, unbind := range unbinds {
labels = append(labels, referenceContractBindingLabel(pipeline.ReferenceBinding{
Stage: unbind.Stage,
LaneID: unbind.LaneID,
SlotName: unbind.SlotName,
}))
}
return labels
}
func referenceContractFindBinding(bindings []pipeline.ReferenceBinding, slot string) *pipeline.ReferenceBinding { func referenceContractFindBinding(bindings []pipeline.ReferenceBinding, slot string) *pipeline.ReferenceBinding {
for i := range bindings { for i := range bindings {
if bindings[i].SlotName == slot { if bindings[i].SlotName == slot {

View File

@@ -177,7 +177,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
fs.Var(&llmProfile, "llm-profile", "LLM profile override") fs.Var(&llmProfile, "llm-profile", "LLM profile override")
fs.Var(&reasoningEffort, "reasoning-effort", "reasoning effort override") fs.Var(&reasoningEffort, "reasoning-effort", "reasoning effort override")
fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh") fs.Var(&chunkCache, "chunk_cache", "chunk plan cache mode: auto, bypass, or refresh")
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path") fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path")
fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference") fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference")
fs.Var(&recomputeStep, "recompute-step", "recompute one ordered pipeline step and dependent lanes") fs.Var(&recomputeStep, "recompute-step", "recompute one ordered pipeline step and dependent lanes")
if err := validateRunFlagValues(args); err != nil { if err := validateRunFlagValues(args); err != nil {
@@ -1287,11 +1287,21 @@ type cliReferenceUnbindRequest struct {
} }
type cliReferenceSelector struct { type cliReferenceSelector struct {
Scope cliReferenceSelectorScope
LaneID string LaneID string
Stage pipeline.ModuleStage Stage pipeline.ModuleStage
SlotName string SlotName string
} }
type cliReferenceSelectorScope uint8
const (
cliReferenceScopePipeline cliReferenceSelectorScope = iota
cliReferenceScopeLane
cliReferenceScopeChunk
cliReferenceScopeBinding
)
func parseReferenceFlags(values []string) ([]cliReferenceRequest, error) { func parseReferenceFlags(values []string) ([]cliReferenceRequest, error) {
if len(values) == 0 { if len(values) == 0 {
return nil, nil return nil, nil
@@ -1300,7 +1310,7 @@ func parseReferenceFlags(values []string) ([]cliReferenceRequest, error) {
for _, raw := range values { for _, raw := range values {
name, source, ok := strings.Cut(raw, "=") name, source, ok := strings.Cut(raw, "=")
if !ok { if !ok {
return nil, fmt.Errorf("--reference must use slot=path or lane.slot=path") return nil, fmt.Errorf("--reference must use slot=path, lane.slot=path, or lane.stage.slot=path")
} }
if strings.TrimSpace(source) == "" { if strings.TrimSpace(source) == "" {
return nil, fmt.Errorf("--reference path must not be empty; use --without-reference to unbind") return nil, fmt.Errorf("--reference path must not be empty; use --without-reference to unbind")
@@ -1350,17 +1360,14 @@ func parseReferenceSelector(raw string, flagName string) (cliReferenceSelector,
} }
switch len(parts) { switch len(parts) {
case 1: case 1:
return cliReferenceSelector{SlotName: strings.TrimSpace(parts[0])}, nil return cliReferenceSelector{Scope: cliReferenceScopePipeline, SlotName: strings.TrimSpace(parts[0])}, nil
case 2: case 2:
first := strings.TrimSpace(parts[0]) first := strings.TrimSpace(parts[0])
slotName := strings.TrimSpace(parts[1]) slotName := strings.TrimSpace(parts[1])
if first == string(pipeline.StageChunk) { if first == string(pipeline.StageChunk) {
return cliReferenceSelector{Stage: pipeline.StageChunk, SlotName: slotName}, nil return cliReferenceSelector{Scope: cliReferenceScopeChunk, Stage: pipeline.StageChunk, SlotName: slotName}, nil
} }
if first == string(pipeline.StageMerge) { return cliReferenceSelector{Scope: cliReferenceScopeLane, LaneID: first, SlotName: slotName}, nil
return cliReferenceSelector{Stage: pipeline.StageMerge, SlotName: slotName}, nil
}
return cliReferenceSelector{LaneID: first, SlotName: slotName}, nil
case 3: case 3:
laneID := strings.TrimSpace(parts[0]) laneID := strings.TrimSpace(parts[0])
stage := pipeline.ModuleStage(strings.TrimSpace(parts[1])) stage := pipeline.ModuleStage(strings.TrimSpace(parts[1]))
@@ -1368,9 +1375,9 @@ func parseReferenceSelector(raw string, flagName string) (cliReferenceSelector,
if stage != pipeline.StageExtract && stage != pipeline.StageMerge && stage != pipeline.StageNormalize { if stage != pipeline.StageExtract && stage != pipeline.StageMerge && stage != pipeline.StageNormalize {
return cliReferenceSelector{}, fmt.Errorf("%s lane-qualified selector must use lane.extract.slot, lane.merge.slot, or lane.normalize.slot", flagName) return cliReferenceSelector{}, fmt.Errorf("%s lane-qualified selector must use lane.extract.slot, lane.merge.slot, or lane.normalize.slot", flagName)
} }
return cliReferenceSelector{LaneID: laneID, Stage: stage, SlotName: slotName}, nil return cliReferenceSelector{Scope: cliReferenceScopeBinding, LaneID: laneID, Stage: stage, SlotName: slotName}, nil
default: default:
return cliReferenceSelector{}, fmt.Errorf("%s must use slot, chunk.slot, merge.slot, lane.slot, lane.extract.slot, lane.merge.slot, or lane.normalize.slot", flagName) return cliReferenceSelector{}, fmt.Errorf("%s must use slot, chunk.slot, lane.slot, lane.extract.slot, lane.merge.slot, or lane.normalize.slot", flagName)
} }
} }
@@ -1391,37 +1398,153 @@ func resolveCLIReferenceRequests(
return nil, nil, err return nil, nil, err
} }
overrides := make([]pipeline.ReferenceBinding, 0, len(referenceRequests)) // Broad CLI selectors are only presentation syntax. Collapse them into one
// highest-specificity action per concrete framework target before pipeline
// resolution so the generic reference contract stays stage-and-lane exact.
actions := make(map[cliReferenceTargetKey]resolvedCLIReferenceAction)
for _, request := range referenceRequests { for _, request := range referenceRequests {
target, err := resolveCLIReferenceTarget(targets, request.Selector) matches, err := resolveCLIReferenceTargets(targets, request.Selector)
if err != nil { if err != nil {
return nil, nil, err return nil, nil, err
} }
for _, target := range matches {
candidate := resolvedCLIReferenceAction{
kind: cliReferenceActionBind,
selector: request.Selector,
target: target,
slotName: request.Selector.SlotName,
source: request.Source,
specificity: request.Selector.specificity(),
}
if err := mergeCLIReferenceAction(actions, candidate); err != nil {
return nil, nil, err
}
}
}
for _, request := range unbindRequests {
matches, err := resolveCLIReferenceTargets(targets, request.Selector)
if err != nil {
return nil, nil, err
}
for _, target := range matches {
candidate := resolvedCLIReferenceAction{
kind: cliReferenceActionUnbind,
selector: request.Selector,
target: target,
slotName: request.Selector.SlotName,
specificity: request.Selector.specificity(),
}
if err := mergeCLIReferenceAction(actions, candidate); err != nil {
return nil, nil, err
}
}
}
resolved := make([]resolvedCLIReferenceAction, 0, len(actions))
for _, action := range actions {
resolved = append(resolved, action)
}
sort.Slice(resolved, func(i, j int) bool {
left, right := resolved[i], resolved[j]
if left.target.laneID != right.target.laneID {
return left.target.laneID < right.target.laneID
}
if left.target.stage != right.target.stage {
return referenceStageOrder(left.target.stage) < referenceStageOrder(right.target.stage)
}
return left.slotName < right.slotName
})
overrides := make([]pipeline.ReferenceBinding, 0, len(resolved))
unbinds := make([]pipeline.ReferenceUnbind, 0, len(resolved))
for _, action := range resolved {
switch action.kind {
case cliReferenceActionBind:
overrides = append(overrides, pipeline.ReferenceBinding{ overrides = append(overrides, pipeline.ReferenceBinding{
Stage: target.stage, Stage: action.target.stage,
LaneID: target.laneID, LaneID: action.target.laneID,
SlotName: request.Selector.SlotName, SlotName: action.slotName,
Source: request.Source, Source: action.source,
BindingSource: contracts.ReferenceBindingSourceCLI, BindingSource: contracts.ReferenceBindingSourceCLI,
}) })
} case cliReferenceActionUnbind:
unbinds := make([]pipeline.ReferenceUnbind, 0, len(unbindRequests))
for _, request := range unbindRequests {
target, err := resolveCLIReferenceTarget(targets, request.Selector)
if err != nil {
return nil, nil, err
}
unbinds = append(unbinds, pipeline.ReferenceUnbind{ unbinds = append(unbinds, pipeline.ReferenceUnbind{
Stage: target.stage, Stage: action.target.stage,
LaneID: target.laneID, LaneID: action.target.laneID,
SlotName: request.Selector.SlotName, SlotName: action.slotName,
}) })
} }
}
return overrides, unbinds, nil return overrides, unbinds, nil
} }
type cliReferenceActionKind uint8
const (
cliReferenceActionBind cliReferenceActionKind = iota
cliReferenceActionUnbind
)
type cliReferenceTargetKey struct {
stage pipeline.ModuleStage
laneID string
slotName string
}
type resolvedCLIReferenceAction struct {
kind cliReferenceActionKind
selector cliReferenceSelector
target selectedReferenceTarget
slotName string
source string
specificity int
}
func mergeCLIReferenceAction(actions map[cliReferenceTargetKey]resolvedCLIReferenceAction, candidate resolvedCLIReferenceAction) error {
key := cliReferenceTargetKey{stage: candidate.target.stage, laneID: candidate.target.laneID, slotName: candidate.slotName}
current, ok := actions[key]
if !ok || candidate.specificity > current.specificity {
actions[key] = candidate
return nil
}
if candidate.specificity < current.specificity {
return nil
}
if candidate.kind != current.kind {
return fmt.Errorf("reference target %q slot %q is both bound by %q and unbound by %q at the same specificity", targetLabel(candidate.target), candidate.slotName, current.selector.String(), candidate.selector.String())
}
actions[key] = candidate
return nil
}
func (selector cliReferenceSelector) specificity() int {
switch selector.Scope {
case cliReferenceScopePipeline:
return 0
case cliReferenceScopeLane:
return 1
case cliReferenceScopeChunk, cliReferenceScopeBinding:
return 2
default:
return -1
}
}
func (selector cliReferenceSelector) String() string {
switch selector.Scope {
case cliReferenceScopePipeline:
return selector.SlotName
case cliReferenceScopeLane:
return selector.LaneID + "." + selector.SlotName
case cliReferenceScopeChunk:
return "chunk." + selector.SlotName
case cliReferenceScopeBinding:
return selector.LaneID + "." + string(selector.Stage) + "." + selector.SlotName
default:
return selector.SlotName
}
}
type selectedReferenceTarget struct { type selectedReferenceTarget struct {
laneID string laneID string
stage pipeline.ModuleStage stage pipeline.ModuleStage
@@ -1638,68 +1761,39 @@ func referenceSlotSet(slots []contracts.ReferenceSlot) map[string]struct{} {
return slotSet return slotSet
} }
func resolveCLIReferenceTarget(targets []selectedReferenceTarget, selector cliReferenceSelector) (selectedReferenceTarget, error) { func resolveCLIReferenceTargets(targets []selectedReferenceTarget, selector cliReferenceSelector) ([]selectedReferenceTarget, error) {
slotName := strings.TrimSpace(selector.SlotName) slotName := strings.TrimSpace(selector.SlotName)
if slotName == "" { if slotName == "" {
return selectedReferenceTarget{}, fmt.Errorf("reference slot must not be empty") return nil, fmt.Errorf("reference slot must not be empty")
} }
if selector.Stage == pipeline.StageChunk { switch selector.Scope {
case cliReferenceScopePipeline:
matches := make([]selectedReferenceTarget, 0, len(targets))
for _, target := range targets {
if _, ok := target.slots[slotName]; ok {
matches = append(matches, target)
}
}
if len(matches) == 0 {
return nil, fmt.Errorf("reference slot %q is not declared by any selected target", slotName)
}
return matches, nil
case cliReferenceScopeChunk:
for _, target := range targets { for _, target := range targets {
if target.stage != pipeline.StageChunk { if target.stage != pipeline.StageChunk {
continue continue
} }
if _, ok := target.slots[slotName]; !ok { if _, ok := target.slots[slotName]; !ok {
return selectedReferenceTarget{}, fmt.Errorf("reference slot %q is not declared by chunk module %q", slotName, target.module) return nil, fmt.Errorf("reference slot %q is not declared by chunk module %q", slotName, target.module)
} }
return target, nil return []selectedReferenceTarget{target}, nil
} }
return selectedReferenceTarget{}, fmt.Errorf("reference chunk target is not selected") return nil, fmt.Errorf("reference chunk target is not selected")
} case cliReferenceScopeLane:
if selector.Stage == pipeline.StageExtract || selector.Stage == pipeline.StageMerge || selector.Stage == pipeline.StageNormalize {
if selector.LaneID == "" && selector.Stage == pipeline.StageMerge {
return resolveCLIReferenceStageTarget(targets, selector.Stage, slotName)
}
for _, target := range targets {
if target.laneID == selector.LaneID && target.stage == selector.Stage {
if _, ok := target.slots[slotName]; !ok {
return selectedReferenceTarget{}, fmt.Errorf("reference slot %q is not declared by selected %s target %q", slotName, selector.Stage, targetLabel(target))
}
return target, nil
}
}
return selectedReferenceTarget{}, fmt.Errorf("reference lane %q is not selected", selector.LaneID)
}
if strings.TrimSpace(selector.LaneID) != "" {
return resolveCLIReferenceLaneTarget(targets, strings.TrimSpace(selector.LaneID), slotName)
}
return resolveCLIReferenceFlatTarget(targets, slotName)
}
func resolveCLIReferenceStageTarget(targets []selectedReferenceTarget, stage pipeline.ModuleStage, slotName string) (selectedReferenceTarget, error) {
matches := make([]selectedReferenceTarget, 0, 2)
for _, target := range targets {
if target.stage != stage {
continue
}
if _, ok := target.slots[slotName]; ok {
matches = append(matches, target)
}
}
switch len(matches) {
case 0:
return selectedReferenceTarget{}, fmt.Errorf("reference slot %q is not declared by any selected %s target", slotName, stage)
case 1:
return matches[0], nil
default:
return selectedReferenceTarget{}, fmt.Errorf("reference slot %q is declared by multiple selected %s targets (%s); use a more specific selector such as %s", slotName, stage, targetList(matches), selectorSuggestions(matches, slotName))
}
}
func resolveCLIReferenceLaneTarget(targets []selectedReferenceTarget, laneID string, slotName string) (selectedReferenceTarget, error) {
laneSelected := false laneSelected := false
matches := make([]selectedReferenceTarget, 0, 2) matches := make([]selectedReferenceTarget, 0, 3)
for _, target := range targets { for _, target := range targets {
if target.laneID != laneID { if target.laneID != selector.LaneID {
continue continue
} }
laneSelected = true laneSelected = true
@@ -1708,44 +1802,36 @@ func resolveCLIReferenceLaneTarget(targets []selectedReferenceTarget, laneID str
} }
} }
if !laneSelected { if !laneSelected {
return selectedReferenceTarget{}, fmt.Errorf("reference lane %q is not selected", laneID) return nil, fmt.Errorf("reference lane %q is not selected", selector.LaneID)
} }
switch len(matches) { if len(matches) == 0 {
case 0: return nil, fmt.Errorf("reference slot %q is not declared by selected lane %q", slotName, selector.LaneID)
return selectedReferenceTarget{}, fmt.Errorf("reference slot %q is not declared by selected lane %q", slotName, laneID)
case 1:
return matches[0], nil
default:
return selectedReferenceTarget{}, fmt.Errorf("reference slot %q is declared by multiple selected targets in lane %q (%s); use a more specific selector such as %s", slotName, laneID, targetList(matches), selectorSuggestions(matches, slotName))
} }
} return matches, nil
case cliReferenceScopeBinding:
func resolveCLIReferenceFlatTarget(targets []selectedReferenceTarget, slotName string) (selectedReferenceTarget, error) { laneSelected := false
matches := make([]selectedReferenceTarget, 0, 2)
for _, target := range targets { for _, target := range targets {
if _, ok := target.slots[slotName]; ok { if target.laneID != selector.LaneID {
matches = append(matches, target) continue
} }
laneSelected = true
if target.stage != selector.Stage {
continue
} }
switch len(matches) { if _, ok := target.slots[slotName]; !ok {
case 0: return nil, fmt.Errorf("reference slot %q is not declared by selected %s target %q", slotName, selector.Stage, targetLabel(target))
return selectedReferenceTarget{}, fmt.Errorf("reference slot %q is not declared by any selected reference target", slotName) }
case 1: return []selectedReferenceTarget{target}, nil
return matches[0], nil }
if !laneSelected {
return nil, fmt.Errorf("reference lane %q is not selected", selector.LaneID)
}
return nil, fmt.Errorf("reference %s target is not selected for lane %q", selector.Stage, selector.LaneID)
default: default:
return selectedReferenceTarget{}, fmt.Errorf("reference slot %q is declared by multiple selected targets (%s); use a more specific selector such as %s", slotName, targetList(matches), selectorSuggestions(matches, slotName)) return nil, fmt.Errorf("reference selector has unknown scope")
} }
} }
func targetList(targets []selectedReferenceTarget) string {
labels := make([]string, 0, len(targets))
for _, target := range targets {
labels = append(labels, targetLabel(target))
}
sort.Strings(labels)
return strings.Join(labels, ", ")
}
func targetLabel(target selectedReferenceTarget) string { func targetLabel(target selectedReferenceTarget) string {
if target.stage == pipeline.StageChunk { if target.stage == pipeline.StageChunk {
return "chunk" return "chunk"
@@ -1753,17 +1839,19 @@ func targetLabel(target selectedReferenceTarget) string {
return target.laneID + "." + string(target.stage) return target.laneID + "." + string(target.stage)
} }
func selectorSuggestions(targets []selectedReferenceTarget, slotName string) string { func referenceStageOrder(stage pipeline.ModuleStage) int {
suggestions := make([]string, 0, len(targets)) switch stage {
for _, target := range targets { case pipeline.StageChunk:
if target.stage == pipeline.StageChunk { return 0
suggestions = append(suggestions, "chunk."+slotName) case pipeline.StageExtract:
continue return 1
case pipeline.StageMerge:
return 2
case pipeline.StageNormalize:
return 3
default:
return 4
} }
suggestions = append(suggestions, target.laneID+"."+string(target.stage)+"."+slotName)
}
sort.Strings(suggestions)
return strings.Join(suggestions, " or ")
} }
func sortedPipelineIDs(cfg config.Config) []string { func sortedPipelineIDs(cfg config.Config) []string {