Add a staged implementation plan to extend first-class LLM access and reference support across all chunk, extract, and normalize stages

2026-07-05 11:01:32 -05:00
parent eab640aa21
commit 84c4c06712
1 changed files with 463 additions and 0 deletions
--- a/docs/roadmap/implementation.md
+++ b/docs/roadmap/implementation.md
@@ -0,0 +1,463 @@
+# LLM And Stage Reference Expansion Implementation Plan
+
+## Purpose
+
+Implement these future roadmap items:
+
+- First-class LLM access for all LLM-eligible pipeline stages: `chunk`,
+  `extract`, and `normalize`.
+- Expand reference support beyond extractors to `chunk` and `normalize`
+  modules, using the same declared-slot model and preserving stage-specific
+  ownership boundaries.
+
+This is a decision-complete implementation plan for an LLM coding agent. Follow
+the stages in order. Keep current-behavior docs unchanged until the
+corresponding behavior exists. Do not implement token budgeting, non-file
+reference producers, structured references, retrieval, or multiple effective LLM
+profiles in this pass.
+
+## Target Behavior
+
+LLM access:
+
+- `contracts.ChunkRequest`, `contracts.ExtractionRequest`, and
+  `contracts.NormalizeRequest` all carry `LLMClient`, `LLMProfile`, `Options`,
+  and `Metadata`.
+- The runner passes the configured structured LLM client to chunkers,
+  extractors, and normalizers.
+- The existing single-effective-LLM-profile CLI restriction remains in place.
+
+References:
+
+- `chunk`, `extract`, and `normalize` modules can declare reference slots.
+- The framework treats references as opaque UTF-8 text plus provenance. Stage
+  modules own slot names and semantics.
+- Reference slots are declared in both runtime module contracts and
+  `ModuleSpec.ReferenceSlots`, so config and pipeline validation can inspect
+  slots without constructing modules.
+- `ModuleSpec.ReferenceSlots` is valid only for `chunk`, `extract`, and
+  `normalize` modules. `input`, `merge`, `validate`, and `output` modules must
+  not declare reference slots.
+- Pipeline-level `references` remain defaults. A pipeline-level binding is
+  valid when any declared eligible module in the pipeline declares the slot. In
+  a run, it is applied only to selected targets that declare the slot.
+- Stage-local reference bindings override pipeline-level defaults for that
+  target.
+- CLI `--reference` and `--without-reference` can target chunk, extract, and
+  normalize references explicitly, while preserving existing unambiguous flat
+  forms.
+- Reference materialization still accepts file producers only, validates UTF-8,
+  infers media types from extensions, enforces optional `AcceptedMediaTypes` and
+  `MaxBytes`, warns for empty files, and never writes content to diagnostics,
+  logs, errors, or manifests.
+- Manifest and diagnostics provenance identifies the reference target stage and,
+  for lane-owned stages, the lane ID.
+
+## Target Config Shape
+
+Keep existing config compatibility and add stage-specific binding locations:
+
+```yaml
+pipelines:
+  dnd-session:
+    input: seriatim
+    references:
+      campaign_context: ./campaign/context.md
+    chunk:
+      module: dnd/scenes
+      references:
+        scene_guide: ./campaign/scenes.md
+    artifacts:
+      spells:
+        extract: dnd/spells
+        references:
+          glossary: ./campaign/glossary.md
+        normalize:
+          module: noop
+          references:
+            normalization_notes: ./campaign/normalization.md
+```
+
+Compatibility rule:
+
+- Existing `artifacts.<lane>.references` remains supported as an alias for
+  extractor references.
+- New `artifacts.<lane>.extract.references` is the canonical extract-stage
+  location.
+- If both the legacy lane-level `references` map and
+  `extract.references` bind the same slot, `extract.references` wins. This keeps
+  migration deterministic and avoids breaking existing configs.
+
+`chunk.references` and `normalize.references` are supported only when their
+module binding uses object form. Scalar bindings continue to work when no
+stage-local references are needed.
+
+## Target CLI Reference Selectors
+
+Preserve existing valid forms and add explicit stage selectors:
+
+- `slot=path`: valid only when exactly one selected reference target across
+  chunk, selected extractors, and selected normalizers declares `slot`.
+- `chunk.slot=path`: targets the singleton chunk module.
+- `lane.slot=path`: valid only when exactly one selected module in that lane,
+  extract or normalize, declares `slot`.
+- `lane.extract.slot=path`: targets a lane extractor.
+- `lane.normalize.slot=path`: targets a lane normalizer.
+
+`--without-reference` accepts the same selector forms without `=path`.
+
+Ambiguous selectors must fail before any LLM call and should name the ambiguous
+targets and the more specific selector forms.
+
+## Stage 1: Contract And Registry Expansion
+
+Add reference declaration support to chunkers and normalizers and LLM delivery
+to normalizers. This stage should not change config shape or runtime reference
+resolution yet.
+
+Implementation steps:
+
+- Add `References contracts.ReferenceSet` to `contracts.ChunkRequest`.
+- Add `LLMClient contracts.StructuredLLMClient` and
+  `References contracts.ReferenceSet` to `contracts.NormalizeRequest`.
+- Add `ReferenceSlots() []ReferenceSlot` to `contracts.Chunker` and
+  `contracts.Normalizer`.
+- Update every concrete chunker and normalizer plus all test fakes to implement
+  `ReferenceSlots()`. Existing modules without references should return `nil`.
+- Change `validateModuleSpec` so `ModuleSpec.ReferenceSlots` is allowed for
+  `StageChunk`, `StageExtract`, and `StageNormalize`, and rejected for all other
+  stages.
+- Add registry/module tests proving:
+  - chunker specs may declare valid reference slots;
+  - normalizer specs may declare valid reference slots;
+  - extractor specs still may declare valid reference slots;
+  - input, merge, validate, and output specs reject reference slots;
+  - invalid slot names, duplicate slots, and negative `MaxBytes` are rejected
+    for all eligible stages.
+- Add production registration tests comparing runtime `ReferenceSlots()` with
+  registered `ModuleSpec.ReferenceSlots` for chunkers, extractors, and
+  normalizers that declare slots. Modules with no slots should be covered by
+  simple nil/empty assertions.
+
+Verification:
+
+- `go test ./internal/framework/contracts`
+- `go test ./internal/framework/pipeline`
+- `go test ./internal/modules/chunk/...`
+- `go test ./internal/modules/normalize/...`
+- `go test ./...`
+
+## Stage 2: Resolved Reference Target Model
+
+Replace extractor-only resolved reference storage with a target-aware model.
+Keep external config and CLI behavior unchanged in this stage where possible.
+
+Implementation steps:
+
+- Introduce a target model in `internal/framework/pipeline`, for example:
+  - target stage: `chunk`, `extract`, or `normalize`;
+  - optional lane ID for lane-owned targets;
+  - module key;
+  - reference bindings;
+  - materialized `ReferenceSet`.
+- Add a resolved chunk reference holder to `ResolvedPipeline`.
+- Split lane reference storage into extract and normalize holders on
+  `ResolvedArtifactLane`.
+- Keep JSON field names deterministic and redaction-friendly. Do not expose
+  reference content through resolved pipeline JSON.
+- Update `CloneReferenceSet`, `ReferenceProvenance`, redacted effective config,
+  resolved-pipeline diagnostics, and any affected tests for the target-aware
+  model.
+- Preserve existing extractor behavior by mapping old lane extractor references
+  into the new extract target.
+- Keep the pipeline digest deterministic and based on resolved reference
+  bindings, not materialized reference bytes. Materialized reference digests
+  remain manifest provenance.
+
+Verification:
+
+- `go test ./internal/framework/pipeline`
+- `go test ./internal/core/config`
+- `go test ./internal/core/diagnostics`
+- `go test ./...`
+
+## Stage 3: Config Parsing And Validation
+
+Add stage-local reference fields while preserving existing configs.
+
+Implementation steps:
+
+- Extend the file config module binding object to accept `references`.
+- Add `References map[string]string` to `pipeline.ModuleBinding`, or use a
+  narrower equivalent if the implementation can do so without duplicating
+  binding parsing. If added to `ModuleBinding`, validation must reject
+  references on non-eligible stages.
+- Apply file config references from:
+  - `pipelines.<id>.chunk.references`;
+  - `pipelines.<id>.artifacts.<lane>.extract.references`;
+  - `pipelines.<id>.artifacts.<lane>.normalize.references`;
+  - existing `pipelines.<id>.artifacts.<lane>.references` as an extract-stage
+    compatibility alias.
+- Validate reference maps for empty names, duplicate-after-trim names, and empty
+  sources at each new location.
+- Reject references on `input`, `merge`, validators, and `output` bindings with
+  clear config errors.
+- Preserve existing `llm_profile` validation for chunk, extract, and normalize
+  bindings. No new LLM profile semantics are introduced in this pass.
+- Add config tests for:
+  - parsing `chunk.references`;
+  - parsing `extract.references`;
+  - parsing `normalize.references`;
+  - legacy lane `references` compatibility;
+  - duplicate conflicts where `extract.references` overrides legacy lane
+    references for the same slot;
+  - rejection of references on unsupported module binding locations;
+  - redaction/effective config cloning for all new reference maps.
+
+Verification:
+
+- `go test ./internal/core/config`
+- `go test ./internal/framework/pipeline`
+- `go test ./...`
+
+## Stage 4: Pipeline Resolution
+
+Resolve reference bindings against all eligible stage targets and selected lanes
+without constructing modules.
+
+Implementation steps:
+
+- During `ResolvePipeline`, collect declared reference slots from:
+  - the configured chunker spec;
+  - every declared artifact lane extractor spec;
+  - every declared artifact lane normalizer spec.
+- Keep pipeline-level reference validation broad: a pipeline-level binding is
+  valid if any declared eligible target in the full pipeline declares that slot,
+  even when `--only` selects lanes that do not consume it.
+- During a selected run, apply pipeline-level defaults only to:
+  - the singleton chunk target if its chunker declares the slot;
+  - selected extractor targets that declare the slot;
+  - selected normalizer targets that declare the slot.
+- Apply stage-local references after pipeline defaults:
+  - chunk-local references override chunk pipeline defaults;
+  - extract-local references override extract pipeline defaults;
+  - normalize-local references override normalize pipeline defaults.
+- Enforce required slots independently per selected target after defaults,
+  local references, CLI overrides, and CLI unbinds.
+- Keep lane-local strictness: extract-local references must be declared by that
+  lane's extractor, and normalize-local references must be declared by that
+  lane's normalizer.
+- Return clear errors that name pipeline ID, stage, lane ID when present, slot,
+  and module key.
+- Add pipeline tests for:
+  - pipeline default consumed by chunk only;
+  - pipeline default consumed by extract only;
+  - pipeline default consumed by normalize only;
+  - one pipeline default consumed by several eligible targets;
+  - valid pipeline default declared only by an unselected lane target;
+  - unknown pipeline-level slot rejected when no declared target has it;
+  - extract-local reference rejected by normalizer-only slot;
+  - normalize-local reference rejected by extractor-only slot;
+  - required chunk reference missing;
+  - required normalize reference missing;
+  - local overrides over pipeline defaults for each eligible stage.
+
+Verification:
+
+- `go test ./internal/framework/pipeline`
+- `go test ./internal/core/config`
+- `go test ./...`
+
+## Stage 5: CLI Reference Selectors
+
+Extend CLI reference override and unbind resolution to target chunk, extract,
+and normalize references.
+
+Implementation steps:
+
+- Replace extractor-only CLI reference target lookup with target-aware lookup
+  over the resolved eligible targets for the selected run.
+- Support selector forms listed in "Target CLI Reference Selectors".
+- Preserve existing behavior for unambiguous `slot` and `lane.slot` forms.
+- Reject ambiguous `slot` and `lane.slot` forms with a message naming matching
+  targets and suggesting `chunk.slot`, `lane.extract.slot`, or
+  `lane.normalize.slot` as appropriate.
+- Reject selectors that target an unselected lane during `--only` runs.
+- Reject selectors for stages that do not declare the slot.
+- Keep `--reference` empty-path rejection and `--without-reference` optional
+  unbinding semantics.
+- Add CLI tests for:
+  - `chunk.slot=path`;
+  - `lane.extract.slot=path`;
+  - `lane.normalize.slot=path`;
+  - flat slot resolution across exactly one target;
+  - lane slot resolution across exactly one lane-local target;
+  - ambiguous flat slot across chunk and extract;
+  - ambiguous lane slot across extract and normalize;
+  - unbind forms for chunk, extract, and normalize;
+  - unbind leaving required chunk or normalize slot missing;
+  - `--only` with an unselected lane selector.
+
+Verification:
+
+- `go test ./internal/cli`
+- `go test ./internal/framework/pipeline`
+- `go test ./...`
+
+## Stage 6: Materialization And Provenance
+
+Materialize references for chunk, extract, and normalize targets with one shared
+implementation.
+
+Implementation steps:
+
+- Refactor materialization so it accepts a resolved target plus that target's
+  declared slots, rather than assuming an extractor lane.
+- Reuse the existing file behavior:
+  - config paths relative to the config file;
+  - CLI paths relative to the current working directory;
+  - UTF-8 validation for all files;
+  - extension-based media type inference;
+  - optional `AcceptedMediaTypes` enforcement;
+  - optional `MaxBytes` enforcement;
+  - empty-file warning;
+  - `sha256:` content digest.
+- Store materialized reference sets on the resolved chunk target, extract
+  targets, and normalize targets.
+- Update warnings to include target context in `Scope`, such as:
+  - `pipeline.<id>.chunk.reference.<slot>`;
+  - `pipeline.<id>.lane.<lane>.extract.reference.<slot>`;
+  - `pipeline.<id>.lane.<lane>.normalize.reference.<slot>`.
+- Update manifest and diagnostics provenance to include:
+  - `stage`: `chunk`, `extract`, or `normalize`;
+  - `lane_id`: omitted for chunk and present for extract/normalize;
+  - `slot_name`;
+  - origin type and URI;
+  - digest;
+  - media type;
+  - size bytes;
+  - binding source.
+- Keep reference content out of manifests, diagnostics, logs, and errors.
+- Add tests for materialization and provenance for all three eligible stages,
+  including media-type rejection and UTF-8 rejection on non-extractor targets.
+
+Verification:
+
+- `go test ./internal/framework/pipeline`
+- `go test ./internal/core/artifacts`
+- `go test ./internal/core/diagnostics`
+- `go test ./internal/modules/output/json`
+- `go test ./internal/cli`
+- `go test ./...`
+
+## Stage 7: Runner Delivery
+
+Deliver LLM clients and references to eligible stage modules at runtime.
+
+Implementation steps:
+
+- Pass `CloneReferenceSet` of the materialized chunk reference set into
+  `contracts.ChunkRequest.References`.
+- Continue passing `LLMClient` into `contracts.ChunkRequest`.
+- Continue passing extract references and `LLMClient` into
+  `contracts.ExtractionRequest`.
+- Pass `LLMClient` and `CloneReferenceSet` of the materialized normalize
+  reference set into `contracts.NormalizeRequest`.
+- Ensure normalizers receive their configured `LLMProfile`, options, and
+  metadata as they do today.
+- Add runner tests proving:
+  - chunkers receive LLM client and chunk references;
+  - extractors still receive LLM client and extract references;
+  - normalizers receive LLM client and normalize references;
+  - reference sets are cloned before delivery so modules cannot mutate resolved
+    pipeline state;
+  - missing LLM client still fails only when a module actually uses it, matching
+    existing chunk/extract behavior.
+- Update the walking skeleton to exercise LLM-backed normalize access if a small
+  fake normalizer can do so without making the fixture noisy.
+
+Verification:
+
+- `go test ./internal/framework/pipeline`
+- `go test ./...`
+
+## Stage 8: Prompt Helpers And Module Usage
+
+Make prompt reference helpers usable by chunk and normalize modules, then update
+production modules only where needed.
+
+Implementation steps:
+
+- Confirm `internal/framework/prompt` reference rendering is contract-based and
+  not extractor-specific. If any naming or error text says extractor-only,
+  generalize it to module/stage language.
+- Add prompt tests rendering references for a fake chunker or normalizer prompt
+  bundle if existing tests cover only extractor usage.
+- Update production chunkers and normalizers:
+  - `generic` chunker: no reference slots.
+  - `dnd/scenes` chunker: no reference slots in this pass unless a clear
+    module-owned slot already exists.
+  - `noop` normalizer: no reference slots.
+- Do not invent a new production normalize module in this pass. The goal is
+  framework support, not a domain normalizer.
+
+Verification:
+
+- `go test ./internal/framework/prompt`
+- `go test ./internal/modules/chunk/...`
+- `go test ./internal/modules/normalize/...`
+- `go test ./...`
+
+## Stage 9: Documentation And Examples
+
+After behavior is implemented and tested, update canonical docs and examples.
+
+Implementation steps:
+
+- Update `docs/cli.md` with the expanded `--reference` and
+  `--without-reference` selector grammar and examples.
+- Update `docs/config.md` with `chunk.references`,
+  `artifacts.<lane>.extract.references`, `artifacts.<lane>.normalize.references`,
+  and the legacy `artifacts.<lane>.references` compatibility rule.
+- Update `docs/internal/modules.md` with module-author guidance for
+  `ReferenceSlots()` on chunkers, extractors, and normalizers, plus LLM access
+  expectations for chunk, extract, and normalize.
+- Update `docs/internal/pipeline.md` with target-aware reference resolution,
+  materialization, provenance, and runtime delivery.
+- Update `docs/integrations/json-output.md` with the new reference provenance
+  shape including `stage` and optional `lane_id`.
+- Update `docs/troubleshooting.md` with ambiguous selector examples for chunk,
+  extract, and normalize targets.
+- Update maintained examples only if they can remain small and validated. Do
+  not add speculative reference files for chunk or normalize modules without a
+  production module that actually declares those slots.
+- Update `docs/roadmap/future.md` by removing or revising the two completed
+  items while leaving still-deferred work such as token budgeting and non-file
+  producers.
+- Replace this implementation plan with a concise completed-note document after
+  implementation is finished.
+
+Verification:
+
+- `rg -n "chunk.references|normalize.references|lane.extract.slot|lane.normalize.slot" docs/cli.md docs/config.md docs/internal docs/integrations docs/troubleshooting.md`
+- `go test ./...`
+- `go vet ./...`
+- `go build ./cmd/notarius`
+
+## Final Acceptance Criteria
+
+The feature is complete when:
+
+- chunkers, extractors, and normalizers can declare reference slots through
+  runtime contracts and registry specs;
+- normalizers receive the structured LLM client at runtime;
+- config can bind references at pipeline, chunk, extract, and normalize levels;
+- CLI reference overrides and unbinds can target chunk, extract, and normalize
+  references without ambiguous selector behavior;
+- pipeline resolution validates all reference bindings before any LLM call;
+- materialization and provenance work uniformly for chunk, extract, and
+  normalize references;
+- reference manifest entries identify stage and lane context;
+- existing extractor reference behavior remains backward compatible;
+- canonical docs describe only implemented behavior;
+- `go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` pass.