Make command line references pipeline scoped

2026-08-29 12:34:31 +00:00
parent f208dbe954
commit deebc89255
9 changed files with 824 additions and 238 deletions
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -55,8 +55,8 @@ pipeline ID and **--input** are required.
 | **--session-id id** | Override the generated prompt session identifier with a non-empty value for LLM-backed module calls. |
 | **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
 | **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |
-| **--reference selector=path** | Add or replace a file reference binding. Repeatable. |
-| **--without-reference selector** | Remove a configured optional reference binding. Repeatable. |
+| **--reference selector=path** | Add or replace external file reference bindings at pipeline, lane, chunk, or binding scope. Repeatable. |
+| **--without-reference selector** | Remove matching configured external reference bindings. Repeatable. |

 **--chunk_cache** accepts only **auto**, **bypass**, or **refresh**.
 **--debug-dir**, **--output-dir**, **--session-id**, and
@@ -81,23 +81,49 @@ guidance.

 ### Reference selectors

-Use **--reference** only for a reference slot declared by the selected
-configured target. The accepted selector forms are:
+Use **--reference** only for reference slots declared by the selected
+configured targets. Qualification narrows the scope of an override:

 | Form | Target |
 | --- | --- |
-| slot=path | The unique selected target that declares slot. |
+| slot=path | Every selected target that declares slot. |
 | chunk.slot=path | The chunker. |
-| merge.slot=path | The unique selected merger that declares slot. |
-| lane.slot=path | The unique extractor, merger, or normalizer in lane that declares slot. |
+| lane.slot=path | Every extractor, merger, or normalizer in lane that declares slot. |
 | lane.extract.slot=path | The extractor in lane. |
 | lane.merge.slot=path | The merger in lane. |
 | lane.normalize.slot=path | The normalizer in lane. |

-**--without-reference** uses the same selector forms without =path. Slot
-names, requiredness, and configured bindings are part of the
+Pipeline- and lane-scoped selectors are expected to match multiple targets and
+fail if they match none. A stage-specific selector fails when its lane is not
+selected or its target does not declare the slot. There is no stage-wide
+`merge.slot` shorthand; name the lane when targeting a merger.
+
+CLI bindings override configured external paths. For overlapping CLI
+selectors, a binding-specific or chunk selector overrides a lane selector, and
+a lane selector overrides a pipeline selector. The last occurrence wins at
+equal scope. Binding and unbinding the same concrete target at equal scope is
+an error; a narrower bind or unbind may create an intentional exception to a
+broader action.
+
+**--without-reference** uses the same selector forms without `=path` and
+removes external bindings only. Neither flag replaces or removes a generated
+artifact handoff; an external/generated collision is a resolution error.
+Required slots are checked after all effective changes. CLI reference paths
+are resolved relative to the process working directory, so subprocess and
+service callers should use absolute paths. Slot names, accepted media types,
+size limits, requiredness, and configured generated bindings are part of the
 [configuration contract](config.md).

+For example, one shared campaign reference can reach every compatible target,
+with an optional lane-specific exception:
+
+~~~
+notarius run dnd-session \
+  --input /data/transcript.json \
+  --reference party=/data/references/party.txt \
+  --reference npc-registry.party=/data/references/npc-party-context.txt
+~~~
+
 ### Run output

 Without **--json**, standard output contains the completed pipeline ID, counts
--- a/docs/config.md
+++ b/docs/config.md
@@ -411,10 +411,15 @@ slot. A generated binding supplies one accepted normalized artifact; it does
 not name a file. A configured generated dependency remains required even when
 that consumer slot is otherwise optional.

-Pipeline references are defaults. A matching step-local or binding-local
-external path overrides a pipeline default. Required slots must be bound after
-these configuration values and any CLI reference overrides are applied.
-Reference paths in YAML are resolved relative to the configuration file.
+Pipeline references are configuration defaults. A matching step-local or
+binding-local external path overrides a pipeline default. CLI reference
+bindings are then operational overrides of configured external paths; their
+pipeline, lane, and binding scopes and precedence are defined by the
+[CLI reference](cli.md#reference-selectors). A CLI file reference cannot
+replace a configured generated artifact handoff. Required slots must be bound
+after configuration and CLI reference actions are applied. Reference paths in
+YAML are resolved relative to the configuration file; CLI reference paths are
+resolved relative to the process working directory.

 ### D&D Reference Slots

--- a/docs/consumers/dnd-pipeline.md
+++ b/docs/consumers/dnd-pipeline.md
@@ -24,7 +24,9 @@ files. Use absolute paths for service and subprocess deployments. In
 particular, observe these different resolution rules:

 - reference paths in YAML are resolved relative to the Notarius configuration
-  file; and
+  file;
+- reference paths passed with `--reference` are resolved relative to the
+  Notarius process working directory; and
 - `promptkit.profile_file` is resolved relative to the Notarius process working
  directory.

@@ -73,9 +75,21 @@ notarius run dnd-session \
  --config /absolute/path/to/notarius.yml \
  --input /absolute/path/to/transcripts/final.trimmed.json \
  --output-dir /absolute/path/to/notarius-output \
+  --reference party=/absolute/path/to/references/party.txt \
+  --reference players=/absolute/path/to/references/players.txt \
+  --reference glossary=/absolute/path/to/references/glossary.txt \
+  --reference spell_catalog=/absolute/path/to/references/spells.json \
  --json
 ```

+Each unqualified reference is pipeline-scoped: Notarius supplies it to every
+selected D&D target that declares the slot. A deployment may omit an optional
+reference it does not maintain, and may use the lane- or binding-qualified
+forms from the [CLI reference](../cli.md#reference-selectors) for an exceptional
+override. The registry, scene-description, combat-turn, and NPC-occurrence
+references declared between ordered steps in the complete configuration are
+generated artifacts. Do not pass those handoffs on the CLI.
+
 The caller should:

 - capture stdout and stderr separately;
--- a/docs/consumers/subprocess.md
+++ b/docs/consumers/subprocess.md
@@ -31,13 +31,36 @@ notarius run pipeline-id \
 ```

 Use absolute paths for supplied input, configuration, output-root, and
-reference files. Notarius generates a stable prompt session for the resolved
-input module and exact input bytes. Pass **--session-id** only when intentionally
-grouping different invocations under a different session. Supply credentials
-through Notarius's documented configuration and environment mechanisms, never
-as command-line arguments or generated secret-bearing configuration. In
-particular, a session identifier is provider-visible and is not a credential
-mechanism.
+reference files. Pass each external reference as its own argument-vector pair;
+do not construct and invoke a shell command. An unqualified reference selector
+supplies that file to every compatible selected target. Lane and stage
+qualification are available for exceptional overrides, while generated
+same-run references remain part of configured pipeline composition. The
+[CLI reference](../cli.md#reference-selectors) owns the exact selector and
+precedence contract.
+
+The maintained D&D subprocess workflow uses this facility for campaign context:
+
+```sh
+notarius run dnd-session \
+  --config /absolute/path/to/notarius.yml \
+  --input /absolute/path/to/transcripts/final.trimmed.json \
+  --output-dir /absolute/path/to/notarius-output \
+  --reference party=/absolute/path/to/references/party.txt \
+  --reference players=/absolute/path/to/references/players.txt \
+  --reference glossary=/absolute/path/to/references/glossary.txt \
+  --reference spell_catalog=/absolute/path/to/references/spells.json \
+  --json
+```
+
+Only pass the external references available to and desired by the deployment.
+Notarius generates a stable prompt session for the resolved input module and
+exact input bytes; reference changes do not change it. Pass **--session-id**
+only when intentionally grouping different invocations under a different
+session. Supply credentials through Notarius's documented configuration and
+environment mechanisms, never as command-line arguments or generated
+secret-bearing configuration. In particular, a session identifier is
+provider-visible and is not a credential mechanism.

 Wait for the process before interpreting standard output. Only an exit status
 of 0 permits decoding the receipt. On a nonzero exit, retain standard error for
--- a/docs/internal/cli.md
+++ b/docs/internal/cli.md
@@ -115,6 +115,16 @@ to checkpoint identity and `pipeline.RunInput`. The public flag and stability
 contract are defined by the [CLI reference](../cli.md#run); framework and LLM
 packages only transport the supplied value.

+The CLI also owns the scope grammar for reference flags. It enumerates the
+selected chunk and lane targets from registered module specifications, expands
+pipeline- and lane-scoped actions into exact stage-and-lane bindings, and
+resolves overlapping bind and unbind actions by specificity before calling
+configuration resolution. The generic pipeline therefore receives only exact
+`ReferenceBinding` and `ReferenceUnbind` values and has no knowledge of CLI
+selector syntax. Configuration resolution retains ownership of configured
+external/generated conflicts, required slots, and module compatibility; file
+materialization still occurs afterward.
+
 For `run --json`, the CLI constructs and encodes its private run-result receipt
 after a successful runner result is available, before it publishes logical
 output files. It writes the prepared receipt to standard output only after
@@ -166,8 +176,9 @@ is discoverable.
 - **internal/cli/production_contract_test.go** covers registrar composition,
  production catalog contents, assets, and representative configuration
  validation.
- **internal/cli/reference_contract_test.go** covers CLI reference overrides,
-  origin separation, and materialization boundaries.
+- **internal/cli/reference_contract_test.go** covers scoped CLI reference
+  expansion, specificity, bind/unbind conflicts, generated-reference
+  protection, origin separation, and materialization boundaries.
 - **internal/cli/state_hardening_test.go** covers safe run identity, state
  roots, and failure ordering.

--- a/docs/internal/pipeline.md
+++ b/docs/internal/pipeline.md
@@ -97,6 +97,13 @@ codec, checks its complete schema and media identity, and records a content
 digest plus bounded producer provenance. A missing, ambiguous, invalid, or
 incompatible producer prevents the consumer step from starting.

+Resolution receives only exact stage-and-lane operational reference overrides.
+The CLI may offer broader pipeline- or lane-scoped selectors, but expands and
+arbitrates those before entering the framework. External overrides are applied
+after configured external defaults and local bindings. They cannot coexist
+with a generated binding for the same target and slot, and external unbinds do
+not remove generated handoffs.
+
 ## Execution And Ordering

 The runner validates its input, installs no-op state collaborators when none
--- a/docs/roadmap/cli-pipeline-references.md
+++ b/docs/roadmap/cli-pipeline-references.md
@@ -0,0 +1,258 @@
+# Pipeline-Scoped CLI References
+
+## Purpose
+
+Make command-line reference binding match the pipeline-level mental model used
+by configuration and by subprocess callers. A caller should be able to supply
+each shared external reference once, while retaining explicit syntax for the
+less common case in which one lane or one stage binding needs a different
+source.
+
+## Motivation And Current Problem
+
+Notarius already accepts repeatable `--reference selector=path` and
+`--without-reference selector` flags. The current unqualified form,
+`--reference slot=path`, succeeds only when exactly one selected chunk, extract,
+merge, or normalize target declares that slot. If several targets consume a
+shared reference such as `party`, `glossary`, or `spell_catalog`, the command is
+rejected as ambiguous and the caller must repeat stage-qualified bindings.
+
+That behavior is internally precise but does not match the public configuration
+model. A reference declared at pipeline scope is shared with every compatible
+target, while target-local configuration provides the exceptional override.
+It is therefore surprising for the least-qualified CLI syntax to mean "find one
+unique target" rather than "supply this reference to the pipeline."
+
+The mismatch is especially costly for subprocess use. The primary complete D&D
+workflow expects an orchestrator to provide one transcript, one output root,
+and a small collection of campaign reference files. Requiring the orchestrator
+to know and enumerate every internal consumer of those shared files couples it
+to lane composition, makes commands needlessly long, and creates maintenance
+work whenever another compatible D&D module is added.
+
+## Target CLI Contract
+
+`--reference` remains repeatable, but selector qualification expresses scope:
+
+| Form | Target scope |
+| --- | --- |
+| `slot=path` | Every selected pipeline target that declares `slot`. |
+| `chunk.slot=path` | The selected chunk binding. |
+| `lane.slot=path` | Every extract, merge, or normalize binding in `lane` that declares `slot`. |
+| `lane.extract.slot=path` | The extract binding in `lane`. |
+| `lane.merge.slot=path` | The merge binding in `lane`. |
+| `lane.normalize.slot=path` | The normalize binding in `lane`. |
+
+The current stage-wide `merge.slot=path` shorthand is removed. A merge-specific
+override must name its lane as `lane.merge.slot=path`; this keeps the grammar
+hierarchical and avoids another uniqueness-dependent selector.
+
+`--without-reference` uses the same selector forms without `=path`. It removes
+matching external bindings only. It never removes a generated artifact handoff,
+and resolution continues to reject a missing required reference.
+
+Broad selectors are expected to match multiple compatible targets. They fail
+when they match no selected target, when the named lane is not selected, or
+when a stage-qualified target does not declare the slot. Errors should identify
+the selector and relevant scope without requiring callers to understand private
+resolver structures.
+
+## Precedence And Conflict Policy
+
+Command-line references are operational overrides and take precedence over
+external file references supplied at pipeline, step, lane, or binding scope in
+configuration. Among CLI selectors that affect the same concrete target:
+
+1. an exact `lane.stage.slot` or `chunk.slot` selector wins over a lane-scoped
+   selector;
+2. a lane-scoped selector wins over a pipeline-scoped selector; and
+3. the last occurrence wins among selectors with equal scope and action.
+
+This ordering lets an orchestrator provide shared defaults once and express
+only genuine exceptions:
+
+```sh
+--reference party=/refs/party.txt \
+--reference npc-registry.party=/refs/npc-party-context.txt
+```
+
+Binding and unbinding the same concrete target at the same specificity is a
+configuration error rather than an argument-order-dependent result. A
+more-specific unbind may carve an exception out of a broader binding, and a
+more-specific binding may restore an exception to a broader unbind.
+
+Generated artifact references remain a distinct source form. A CLI file
+reference must not silently replace, remove, or coexist with a generated
+handoff for the same concrete target and slot. Resolution fails with a
+target-specific conflict and directs the caller to narrow or remove the CLI
+selector. Same-run generated registry and eligibility handoffs in the complete
+D&D pipeline therefore remain controlled by pipeline composition.
+
+CLI reference paths continue to resolve relative to the process working
+directory and retain CLI provenance. Subprocess guidance must recommend
+absolute paths. The reference source is auxiliary context and does not affect
+the generated prompt session identifier.
+
+## Subprocess Target State
+
+Passing shared references is a first-class part of the documented subprocess
+workflow. The complete D&D invocation should have this shape:
+
+```sh
+notarius run dnd-session \
+  --config /absolute/path/to/notarius.yml \
+  --input /absolute/path/to/transcripts/final.trimmed.json \
+  --output-dir /absolute/path/to/notarius-output \
+  --reference party=/absolute/path/to/references/party.txt \
+  --reference players=/absolute/path/to/references/players.txt \
+  --reference glossary=/absolute/path/to/references/glossary.txt \
+  --reference spell_catalog=/absolute/path/to/references/spells.json \
+  --json
+```
+
+Only references actually available to and desired by the deployment need to be
+passed. The selected modules continue to determine accepted slot names, media
+types, requiredness, and size limits. The orchestrator passes references as
+separate argument-vector elements rather than building a shell command, captures
+standard output and standard error separately, checks exit status, and uses the
+JSON receipt's `output_directory` to locate the generated run bundle.
+
+The documentation must clearly distinguish:
+
+- the input file supplied by `--input`;
+- the output *root* supplied by `--output-dir` and the run-specific output path
+  returned in the receipt;
+- external file references supplied by `--reference`; and
+- generated references produced and consumed within the configured ordered
+  pipeline.
+
+## Required Implementation Changes
+
+### CLI selector model and parsing
+
+- Replace the unique-target interpretation of unqualified and lane-qualified
+  selectors with explicit pipeline, lane, chunk, and exact-binding scopes.
+- Remove parsing and help text for the stage-wide `merge.slot` form.
+- Preserve repeatable flag handling, non-empty selector/path checks, supported
+  stage names, and syntax-error exit classification.
+- Represent selector scope explicitly enough that precedence and diagnostics do
+  not depend on inferring intent from empty fields.
+
+### Target expansion and precedence
+
+- Refactor CLI reference resolution to return every compatible selected target
+  for a broad selector rather than requiring uniqueness.
+- Resolve overlapping bind and unbind requests into one deterministic action
+  per concrete target and slot using the precedence policy above.
+- Produce exact `pipeline.ReferenceBinding` and `pipeline.ReferenceUnbind`
+  values after CLI scoping is resolved. Keep broad-selector policy at the CLI
+  boundary rather than adding CLI grammar or D&D knowledge to the generic
+  pipeline framework.
+- Preserve CLI provenance and the existing rule that command-line paths are
+  materialized relative to the working directory.
+- Detect generated-reference conflicts before file access or stage execution
+  and retain required-slot validation after effective bindings are known.
+- Ensure lane selection is honored for compact pipelines. Explicit ordered
+  pipelines continue to use their complete configured lane set because they do
+  not support `--only`.
+
+### Public and internal documentation
+
+- Rewrite the reference-selector section of `docs/cli.md`, which is the
+  canonical owner of flag syntax, selector semantics, precedence, and command
+  errors.
+- Update `docs/consumers/subprocess.md` so reference-bearing invocation is part
+  of the primary subprocess workflow rather than an unillustrated aside.
+- Update `docs/consumers/dnd-pipeline.md` with the complete D&D subprocess
+  invocation and explain which references are external versus generated.
+- Lightly update `docs/config.md` to link the configuration hierarchy to the
+  CLI override contract without duplicating CLI syntax.
+- Update `docs/internal/cli.md` and, only where necessary, the reference
+  resolution discussion in `docs/internal/pipeline.md` to describe the
+  implemented expansion and precedence boundary.
+- Do not expand the README quickstart or duplicate complete configuration
+  files in prose. The maintained example configuration remains the canonical
+  copyable pipeline definition.
+
+No ADR is required. This is a deliberate public CLI usability correction that
+fits the existing architectural decisions: configuration remains centralized,
+operational overrides remain explicit, generated handoffs remain part of
+pipeline composition, and broad CLI syntax is translated into existing exact
+framework bindings at the application boundary. The CLI and configuration
+references are the durable owners of the resulting current behavior once it is
+implemented.
+
+## Testing And Validation
+
+Add lean offline behavioral coverage at the CLI contract boundary for:
+
+- one pipeline-scoped reference reaching multiple compatible chunk and lane
+  targets;
+- one lane-scoped reference reaching both extract and normalize bindings;
+- exact-binding and lane-scoped exceptions overriding broader values;
+- final-occurrence behavior at equal specificity;
+- broad selectors with no matches and selectors naming unselected lanes or
+  undeclared slots;
+- bind/unbind specificity and same-specificity conflicts;
+- rejection of external/generated conflicts and protection of required
+  generated handoffs;
+- working-directory resolution and CLI provenance for expanded bindings; and
+- removal of the `merge.slot` shorthand.
+
+Prefer package-level resolution and representative `RunWithOptions` tests over
+duplicating the full selector matrix through end-to-end fixtures. Existing
+tests that encode unique-target ambiguity should be rewritten or removed rather
+than retained as historical change detectors. Do not assert complete error
+strings when stable classification and a concise semantic fragment provide
+sufficient confidence.
+
+Validation for the completed change should include:
+
+```sh
+go test ./internal/cli ./internal/core/config ./internal/framework/pipeline
+go test ./...
+go vet ./...
+go build ./cmd/notarius
+```
+
+Also validate both maintained example configurations against their selected
+pipelines, verify every changed documentation link, and manually review the
+D&D subprocess command against the actual flag parser and complete example
+configuration.
+
+## Non-Goals
+
+- Passing reference contents directly in command-line arguments or environment
+  variables.
+- Adding a reference-manifest file, glob syntax, wildcard selectors, or a
+  second pipeline-reference flag.
+- Allowing the CLI to define undeclared reference slots or bypass module media
+  type, size, UTF-8, requiredness, or materialization validation.
+- Replacing generated same-run artifacts with external files or changing
+  ordered pipeline dependencies.
+- Changing configuration-file reference path resolution, prompt-session
+  derivation, output directory allocation, or the run-result receipt.
+- Adding D&D-specific selector behavior to generic CLI or pipeline packages.
+- Preserving uniqueness-dependent `slot` or `lane.slot` behavior as an alias.
+
+## Acceptance Criteria
+
+- A subprocess caller can supply `party`, `players`, `glossary`, and
+  `spell_catalog` once each and have every compatible selected D&D target
+  receive the corresponding external file.
+- `slot=path` has pipeline scope and never fails merely because several selected
+  targets declare the slot.
+- Lane and exact-binding selectors provide deterministic, documented
+  exceptions with specificity-based precedence.
+- A broad selector that matches nothing is rejected before source parsing or
+  LLM work.
+- CLI external references override configured external paths but never silently
+  replace or remove generated artifact references.
+- All effective CLI bindings retain correct provenance and absolute-path
+  materialization behavior.
+- `docs/cli.md` owns the exact public syntax, and both subprocess guides present
+  shared CLI references as the primary orchestration workflow.
+- Current behavior outside `docs/roadmap/` is not documented until the code
+  lands, and no complete example is duplicated outside `examples/`.
+- Focused tests, repository-wide tests, vetting, build, maintained configuration
+  validation, and documentation-link review pass.