Make command line references pipeline scoped

2026-08-29 12:34:31 +00:00
parent f208dbe954
commit deebc89255
9 changed files with 824 additions and 238 deletions
--- a/docs/roadmap/cli-pipeline-references.md
+++ b/docs/roadmap/cli-pipeline-references.md
@@ -0,0 +1,258 @@
+# Pipeline-Scoped CLI References
+
+## Purpose
+
+Make command-line reference binding match the pipeline-level mental model used
+by configuration and by subprocess callers. A caller should be able to supply
+each shared external reference once, while retaining explicit syntax for the
+less common case in which one lane or one stage binding needs a different
+source.
+
+## Motivation And Current Problem
+
+Notarius already accepts repeatable `--reference selector=path` and
+`--without-reference selector` flags. The current unqualified form,
+`--reference slot=path`, succeeds only when exactly one selected chunk, extract,
+merge, or normalize target declares that slot. If several targets consume a
+shared reference such as `party`, `glossary`, or `spell_catalog`, the command is
+rejected as ambiguous and the caller must repeat stage-qualified bindings.
+
+That behavior is internally precise but does not match the public configuration
+model. A reference declared at pipeline scope is shared with every compatible
+target, while target-local configuration provides the exceptional override.
+It is therefore surprising for the least-qualified CLI syntax to mean "find one
+unique target" rather than "supply this reference to the pipeline."
+
+The mismatch is especially costly for subprocess use. The primary complete D&D
+workflow expects an orchestrator to provide one transcript, one output root,
+and a small collection of campaign reference files. Requiring the orchestrator
+to know and enumerate every internal consumer of those shared files couples it
+to lane composition, makes commands needlessly long, and creates maintenance
+work whenever another compatible D&D module is added.
+
+## Target CLI Contract
+
+`--reference` remains repeatable, but selector qualification expresses scope:
+
+| Form | Target scope |
+| --- | --- |
+| `slot=path` | Every selected pipeline target that declares `slot`. |
+| `chunk.slot=path` | The selected chunk binding. |
+| `lane.slot=path` | Every extract, merge, or normalize binding in `lane` that declares `slot`. |
+| `lane.extract.slot=path` | The extract binding in `lane`. |
+| `lane.merge.slot=path` | The merge binding in `lane`. |
+| `lane.normalize.slot=path` | The normalize binding in `lane`. |
+
+The current stage-wide `merge.slot=path` shorthand is removed. A merge-specific
+override must name its lane as `lane.merge.slot=path`; this keeps the grammar
+hierarchical and avoids another uniqueness-dependent selector.
+
+`--without-reference` uses the same selector forms without `=path`. It removes
+matching external bindings only. It never removes a generated artifact handoff,
+and resolution continues to reject a missing required reference.
+
+Broad selectors are expected to match multiple compatible targets. They fail
+when they match no selected target, when the named lane is not selected, or
+when a stage-qualified target does not declare the slot. Errors should identify
+the selector and relevant scope without requiring callers to understand private
+resolver structures.
+
+## Precedence And Conflict Policy
+
+Command-line references are operational overrides and take precedence over
+external file references supplied at pipeline, step, lane, or binding scope in
+configuration. Among CLI selectors that affect the same concrete target:
+
+1. an exact `lane.stage.slot` or `chunk.slot` selector wins over a lane-scoped
+   selector;
+2. a lane-scoped selector wins over a pipeline-scoped selector; and
+3. the last occurrence wins among selectors with equal scope and action.
+
+This ordering lets an orchestrator provide shared defaults once and express
+only genuine exceptions:
+
+```sh
+--reference party=/refs/party.txt \
+--reference npc-registry.party=/refs/npc-party-context.txt
+```
+
+Binding and unbinding the same concrete target at the same specificity is a
+configuration error rather than an argument-order-dependent result. A
+more-specific unbind may carve an exception out of a broader binding, and a
+more-specific binding may restore an exception to a broader unbind.
+
+Generated artifact references remain a distinct source form. A CLI file
+reference must not silently replace, remove, or coexist with a generated
+handoff for the same concrete target and slot. Resolution fails with a
+target-specific conflict and directs the caller to narrow or remove the CLI
+selector. Same-run generated registry and eligibility handoffs in the complete
+D&D pipeline therefore remain controlled by pipeline composition.
+
+CLI reference paths continue to resolve relative to the process working
+directory and retain CLI provenance. Subprocess guidance must recommend
+absolute paths. The reference source is auxiliary context and does not affect
+the generated prompt session identifier.
+
+## Subprocess Target State
+
+Passing shared references is a first-class part of the documented subprocess
+workflow. The complete D&D invocation should have this shape:
+
+```sh
+notarius run dnd-session \
+  --config /absolute/path/to/notarius.yml \
+  --input /absolute/path/to/transcripts/final.trimmed.json \
+  --output-dir /absolute/path/to/notarius-output \
+  --reference party=/absolute/path/to/references/party.txt \
+  --reference players=/absolute/path/to/references/players.txt \
+  --reference glossary=/absolute/path/to/references/glossary.txt \
+  --reference spell_catalog=/absolute/path/to/references/spells.json \
+  --json
+```
+
+Only references actually available to and desired by the deployment need to be
+passed. The selected modules continue to determine accepted slot names, media
+types, requiredness, and size limits. The orchestrator passes references as
+separate argument-vector elements rather than building a shell command, captures
+standard output and standard error separately, checks exit status, and uses the
+JSON receipt's `output_directory` to locate the generated run bundle.
+
+The documentation must clearly distinguish:
+
+- the input file supplied by `--input`;
+- the output *root* supplied by `--output-dir` and the run-specific output path
+  returned in the receipt;
+- external file references supplied by `--reference`; and
+- generated references produced and consumed within the configured ordered
+  pipeline.
+
+## Required Implementation Changes
+
+### CLI selector model and parsing
+
+- Replace the unique-target interpretation of unqualified and lane-qualified
+  selectors with explicit pipeline, lane, chunk, and exact-binding scopes.
+- Remove parsing and help text for the stage-wide `merge.slot` form.
+- Preserve repeatable flag handling, non-empty selector/path checks, supported
+  stage names, and syntax-error exit classification.
+- Represent selector scope explicitly enough that precedence and diagnostics do
+  not depend on inferring intent from empty fields.
+
+### Target expansion and precedence
+
+- Refactor CLI reference resolution to return every compatible selected target
+  for a broad selector rather than requiring uniqueness.
+- Resolve overlapping bind and unbind requests into one deterministic action
+  per concrete target and slot using the precedence policy above.
+- Produce exact `pipeline.ReferenceBinding` and `pipeline.ReferenceUnbind`
+  values after CLI scoping is resolved. Keep broad-selector policy at the CLI
+  boundary rather than adding CLI grammar or D&D knowledge to the generic
+  pipeline framework.
+- Preserve CLI provenance and the existing rule that command-line paths are
+  materialized relative to the working directory.
+- Detect generated-reference conflicts before file access or stage execution
+  and retain required-slot validation after effective bindings are known.
+- Ensure lane selection is honored for compact pipelines. Explicit ordered
+  pipelines continue to use their complete configured lane set because they do
+  not support `--only`.
+
+### Public and internal documentation
+
+- Rewrite the reference-selector section of `docs/cli.md`, which is the
+  canonical owner of flag syntax, selector semantics, precedence, and command
+  errors.
+- Update `docs/consumers/subprocess.md` so reference-bearing invocation is part
+  of the primary subprocess workflow rather than an unillustrated aside.
+- Update `docs/consumers/dnd-pipeline.md` with the complete D&D subprocess
+  invocation and explain which references are external versus generated.
+- Lightly update `docs/config.md` to link the configuration hierarchy to the
+  CLI override contract without duplicating CLI syntax.
+- Update `docs/internal/cli.md` and, only where necessary, the reference
+  resolution discussion in `docs/internal/pipeline.md` to describe the
+  implemented expansion and precedence boundary.
+- Do not expand the README quickstart or duplicate complete configuration
+  files in prose. The maintained example configuration remains the canonical
+  copyable pipeline definition.
+
+No ADR is required. This is a deliberate public CLI usability correction that
+fits the existing architectural decisions: configuration remains centralized,
+operational overrides remain explicit, generated handoffs remain part of
+pipeline composition, and broad CLI syntax is translated into existing exact
+framework bindings at the application boundary. The CLI and configuration
+references are the durable owners of the resulting current behavior once it is
+implemented.
+
+## Testing And Validation
+
+Add lean offline behavioral coverage at the CLI contract boundary for:
+
+- one pipeline-scoped reference reaching multiple compatible chunk and lane
+  targets;
+- one lane-scoped reference reaching both extract and normalize bindings;
+- exact-binding and lane-scoped exceptions overriding broader values;
+- final-occurrence behavior at equal specificity;
+- broad selectors with no matches and selectors naming unselected lanes or
+  undeclared slots;
+- bind/unbind specificity and same-specificity conflicts;
+- rejection of external/generated conflicts and protection of required
+  generated handoffs;
+- working-directory resolution and CLI provenance for expanded bindings; and
+- removal of the `merge.slot` shorthand.
+
+Prefer package-level resolution and representative `RunWithOptions` tests over
+duplicating the full selector matrix through end-to-end fixtures. Existing
+tests that encode unique-target ambiguity should be rewritten or removed rather
+than retained as historical change detectors. Do not assert complete error
+strings when stable classification and a concise semantic fragment provide
+sufficient confidence.
+
+Validation for the completed change should include:
+
+```sh
+go test ./internal/cli ./internal/core/config ./internal/framework/pipeline
+go test ./...
+go vet ./...
+go build ./cmd/notarius
+```
+
+Also validate both maintained example configurations against their selected
+pipelines, verify every changed documentation link, and manually review the
+D&D subprocess command against the actual flag parser and complete example
+configuration.
+
+## Non-Goals
+
+- Passing reference contents directly in command-line arguments or environment
+  variables.
+- Adding a reference-manifest file, glob syntax, wildcard selectors, or a
+  second pipeline-reference flag.
+- Allowing the CLI to define undeclared reference slots or bypass module media
+  type, size, UTF-8, requiredness, or materialization validation.
+- Replacing generated same-run artifacts with external files or changing
+  ordered pipeline dependencies.
+- Changing configuration-file reference path resolution, prompt-session
+  derivation, output directory allocation, or the run-result receipt.
+- Adding D&D-specific selector behavior to generic CLI or pipeline packages.
+- Preserving uniqueness-dependent `slot` or `lane.slot` behavior as an alias.
+
+## Acceptance Criteria
+
+- A subprocess caller can supply `party`, `players`, `glossary`, and
+  `spell_catalog` once each and have every compatible selected D&D target
+  receive the corresponding external file.
+- `slot=path` has pipeline scope and never fails merely because several selected
+  targets declare the slot.
+- Lane and exact-binding selectors provide deterministic, documented
+  exceptions with specificity-based precedence.
+- A broad selector that matches nothing is rejected before source parsing or
+  LLM work.
+- CLI external references override configured external paths but never silently
+  replace or remove generated artifact references.
+- All effective CLI bindings retain correct provenance and absolute-path
+  materialization behavior.
+- `docs/cli.md` owns the exact public syntax, and both subprocess guides present
+  shared CLI references as the primary orchestration workflow.
+- Current behavior outside `docs/roadmap/` is not documented until the code
+  lands, and no complete example is duplicated outside `examples/`.
+- Focused tests, repository-wide tests, vetting, build, maintained configuration
+  validation, and documentation-link review pass.