notarius/docs/roadmap/cli-pipeline-references.md

# Pipeline-Scoped CLI References

## Purpose

Make command-line reference binding match the pipeline-level mental model used
by configuration and by subprocess callers. A caller should be able to supply
each shared external reference once, while retaining explicit syntax for the
less common case in which one lane or one stage binding needs a different
source.

## Motivation And Current Problem

Notarius already accepts repeatable `--reference selector=path` and
`--without-reference selector` flags. The current unqualified form,
`--reference slot=path`, succeeds only when exactly one selected chunk, extract,
merge, or normalize target declares that slot. If several targets consume a
shared reference such as `party`, `glossary`, or `spell_catalog`, the command is
rejected as ambiguous and the caller must repeat stage-qualified bindings.

That behavior is internally precise but does not match the public configuration
model. A reference declared at pipeline scope is shared with every compatible
target, while target-local configuration provides the exceptional override.
It is therefore surprising for the least-qualified CLI syntax to mean "find one
unique target" rather than "supply this reference to the pipeline."

The mismatch is especially costly for subprocess use. The primary complete D&D
workflow expects an orchestrator to provide one transcript, one output root,
and a small collection of campaign reference files. Requiring the orchestrator
to know and enumerate every internal consumer of those shared files couples it
to lane composition, makes commands needlessly long, and creates maintenance
work whenever another compatible D&D module is added.

## Target CLI Contract

`--reference` remains repeatable, but selector qualification expresses scope:

| Form | Target scope |
| --- | --- |
| `slot=path` | Every selected pipeline target that declares `slot`. |
| `chunk.slot=path` | The selected chunk binding. |
| `lane.slot=path` | Every extract, merge, or normalize binding in `lane` that declares `slot`. |
| `lane.extract.slot=path` | The extract binding in `lane`. |
| `lane.merge.slot=path` | The merge binding in `lane`. |
| `lane.normalize.slot=path` | The normalize binding in `lane`. |

The current stage-wide `merge.slot=path` shorthand is removed. A merge-specific
override must name its lane as `lane.merge.slot=path`; this keeps the grammar
hierarchical and avoids another uniqueness-dependent selector.

`--without-reference` uses the same selector forms without `=path`. It removes
matching external bindings only. It never removes a generated artifact handoff,
and resolution continues to reject a missing required reference.

Broad selectors are expected to match multiple compatible targets. They fail
when they match no selected target, when the named lane is not selected, or
when a stage-qualified target does not declare the slot. Errors should identify
the selector and relevant scope without requiring callers to understand private
resolver structures.

## Precedence And Conflict Policy

Command-line references are operational overrides and take precedence over
external file references supplied at pipeline, step, lane, or binding scope in
configuration. Among CLI selectors that affect the same concrete target:

1. an exact `lane.stage.slot` or `chunk.slot` selector wins over a lane-scoped
   selector;
2. a lane-scoped selector wins over a pipeline-scoped selector; and
3. the last occurrence wins among selectors with equal scope and action.

This ordering lets an orchestrator provide shared defaults once and express
only genuine exceptions:

```sh
--reference party=/refs/party.txt \
--reference npc-registry.party=/refs/npc-party-context.txt
```

Binding and unbinding the same concrete target at the same specificity is a
configuration error rather than an argument-order-dependent result. A
more-specific unbind may carve an exception out of a broader binding, and a
more-specific binding may restore an exception to a broader unbind.

Generated artifact references remain a distinct source form. A CLI file
reference must not silently replace, remove, or coexist with a generated
handoff for the same concrete target and slot. Resolution fails with a
target-specific conflict and directs the caller to narrow or remove the CLI
selector. Same-run generated registry and eligibility handoffs in the complete
D&D pipeline therefore remain controlled by pipeline composition.

CLI reference paths continue to resolve relative to the process working
directory and retain CLI provenance. Subprocess guidance must recommend
absolute paths. The reference source is auxiliary context and does not affect
the generated prompt session identifier.

## Subprocess Target State

Passing shared references is a first-class part of the documented subprocess
workflow. The complete D&D invocation should have this shape:

```sh
notarius run dnd-session \
  --config /absolute/path/to/notarius.yml \
  --input /absolute/path/to/transcripts/final.trimmed.json \
  --output-dir /absolute/path/to/notarius-output \
  --reference party=/absolute/path/to/references/party.txt \
  --reference players=/absolute/path/to/references/players.txt \
  --reference glossary=/absolute/path/to/references/glossary.txt \
  --reference spell_catalog=/absolute/path/to/references/spells.json \
  --json
```

Only references actually available to and desired by the deployment need to be
passed. The selected modules continue to determine accepted slot names, media
types, requiredness, and size limits. The orchestrator passes references as
separate argument-vector elements rather than building a shell command, captures
standard output and standard error separately, checks exit status, and uses the
JSON receipt's `output_directory` to locate the generated run bundle.

The documentation must clearly distinguish:

- the input file supplied by `--input`;
- the output *root* supplied by `--output-dir` and the run-specific output path
  returned in the receipt;
- external file references supplied by `--reference`; and
- generated references produced and consumed within the configured ordered
  pipeline.

## Required Implementation Changes

### CLI selector model and parsing

- Replace the unique-target interpretation of unqualified and lane-qualified
  selectors with explicit pipeline, lane, chunk, and exact-binding scopes.
- Remove parsing and help text for the stage-wide `merge.slot` form.
- Preserve repeatable flag handling, non-empty selector/path checks, supported
  stage names, and syntax-error exit classification.
- Represent selector scope explicitly enough that precedence and diagnostics do
  not depend on inferring intent from empty fields.

### Target expansion and precedence

- Refactor CLI reference resolution to return every compatible selected target
  for a broad selector rather than requiring uniqueness.
- Resolve overlapping bind and unbind requests into one deterministic action
  per concrete target and slot using the precedence policy above.
- Produce exact `pipeline.ReferenceBinding` and `pipeline.ReferenceUnbind`
  values after CLI scoping is resolved. Keep broad-selector policy at the CLI
  boundary rather than adding CLI grammar or D&D knowledge to the generic
  pipeline framework.
- Preserve CLI provenance and the existing rule that command-line paths are
  materialized relative to the working directory.
- Detect generated-reference conflicts before file access or stage execution
  and retain required-slot validation after effective bindings are known.
- Ensure lane selection is honored for compact pipelines. Explicit ordered
  pipelines continue to use their complete configured lane set because they do
  not support `--only`.

### Public and internal documentation

- Rewrite the reference-selector section of `docs/cli.md`, which is the
  canonical owner of flag syntax, selector semantics, precedence, and command
  errors.
- Update `docs/consumers/subprocess.md` so reference-bearing invocation is part
  of the primary subprocess workflow rather than an unillustrated aside.
- Update `docs/consumers/dnd-pipeline.md` with the complete D&D subprocess
  invocation and explain which references are external versus generated.
- Lightly update `docs/config.md` to link the configuration hierarchy to the
  CLI override contract without duplicating CLI syntax.
- Update `docs/internal/cli.md` and, only where necessary, the reference
  resolution discussion in `docs/internal/pipeline.md` to describe the
  implemented expansion and precedence boundary.
- Do not expand the README quickstart or duplicate complete configuration
  files in prose. The maintained example configuration remains the canonical
  copyable pipeline definition.

No ADR is required. This is a deliberate public CLI usability correction that
fits the existing architectural decisions: configuration remains centralized,
operational overrides remain explicit, generated handoffs remain part of
pipeline composition, and broad CLI syntax is translated into existing exact
framework bindings at the application boundary. The CLI and configuration
references are the durable owners of the resulting current behavior once it is
implemented.

## Testing And Validation

Add lean offline behavioral coverage at the CLI contract boundary for:

- one pipeline-scoped reference reaching multiple compatible chunk and lane
  targets;
- one lane-scoped reference reaching both extract and normalize bindings;
- exact-binding and lane-scoped exceptions overriding broader values;
- final-occurrence behavior at equal specificity;
- broad selectors with no matches and selectors naming unselected lanes or
  undeclared slots;
- bind/unbind specificity and same-specificity conflicts;
- rejection of external/generated conflicts and protection of required
  generated handoffs;
- working-directory resolution and CLI provenance for expanded bindings; and
- removal of the `merge.slot` shorthand.

Prefer package-level resolution and representative `RunWithOptions` tests over
duplicating the full selector matrix through end-to-end fixtures. Existing
tests that encode unique-target ambiguity should be rewritten or removed rather
than retained as historical change detectors. Do not assert complete error
strings when stable classification and a concise semantic fragment provide
sufficient confidence.

Validation for the completed change should include:

```sh
go test ./internal/cli ./internal/core/config ./internal/framework/pipeline
go test ./...
go vet ./...
go build ./cmd/notarius
```

Also validate both maintained example configurations against their selected
pipelines, verify every changed documentation link, and manually review the
D&D subprocess command against the actual flag parser and complete example
configuration.

## Non-Goals

- Passing reference contents directly in command-line arguments or environment
  variables.
- Adding a reference-manifest file, glob syntax, wildcard selectors, or a
  second pipeline-reference flag.
- Allowing the CLI to define undeclared reference slots or bypass module media
  type, size, UTF-8, requiredness, or materialization validation.
- Replacing generated same-run artifacts with external files or changing
  ordered pipeline dependencies.
- Changing configuration-file reference path resolution, prompt-session
  derivation, output directory allocation, or the run-result receipt.
- Adding D&D-specific selector behavior to generic CLI or pipeline packages.
- Preserving uniqueness-dependent `slot` or `lane.slot` behavior as an alias.

## Acceptance Criteria

- A subprocess caller can supply `party`, `players`, `glossary`, and
  `spell_catalog` once each and have every compatible selected D&D target
  receive the corresponding external file.
- `slot=path` has pipeline scope and never fails merely because several selected
  targets declare the slot.
- Lane and exact-binding selectors provide deterministic, documented
  exceptions with specificity-based precedence.
- A broad selector that matches nothing is rejected before source parsing or
  LLM work.
- CLI external references override configured external paths but never silently
  replace or remove generated artifact references.
- All effective CLI bindings retain correct provenance and absolute-path
  materialization behavior.
- `docs/cli.md` owns the exact public syntax, and both subprocess guides present
  shared CLI references as the primary orchestration workflow.
- Current behavior outside `docs/roadmap/` is not documented until the code
  lands, and no complete example is duplicated outside `examples/`.
- Focused tests, repository-wide tests, vetting, build, maintained configuration
  validation, and documentation-link review pass.