From 84c4c06712b565bb233957395f6440f5a6cd03d8 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 5 Jul 2026 11:01:32 -0500 Subject: [PATCH] Add a staged implementation plan to extend first-class LLM access and reference support across all chunk, extract, and normalize stages --- docs/roadmap/implementation.md | 463 +++++++++++++++++++++++++++++++++ 1 file changed, 463 insertions(+) create mode 100644 docs/roadmap/implementation.md diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..131e3c3 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,463 @@ +# LLM And Stage Reference Expansion Implementation Plan + +## Purpose + +Implement these future roadmap items: + +- First-class LLM access for all LLM-eligible pipeline stages: `chunk`, + `extract`, and `normalize`. +- Expand reference support beyond extractors to `chunk` and `normalize` + modules, using the same declared-slot model and preserving stage-specific + ownership boundaries. + +This is a decision-complete implementation plan for an LLM coding agent. Follow +the stages in order. Keep current-behavior docs unchanged until the +corresponding behavior exists. Do not implement token budgeting, non-file +reference producers, structured references, retrieval, or multiple effective LLM +profiles in this pass. + +## Target Behavior + +LLM access: + +- `contracts.ChunkRequest`, `contracts.ExtractionRequest`, and + `contracts.NormalizeRequest` all carry `LLMClient`, `LLMProfile`, `Options`, + and `Metadata`. +- The runner passes the configured structured LLM client to chunkers, + extractors, and normalizers. +- The existing single-effective-LLM-profile CLI restriction remains in place. + +References: + +- `chunk`, `extract`, and `normalize` modules can declare reference slots. +- The framework treats references as opaque UTF-8 text plus provenance. Stage + modules own slot names and semantics. +- Reference slots are declared in both runtime module contracts and + `ModuleSpec.ReferenceSlots`, so config and pipeline validation can inspect + slots without constructing modules. +- `ModuleSpec.ReferenceSlots` is valid only for `chunk`, `extract`, and + `normalize` modules. `input`, `merge`, `validate`, and `output` modules must + not declare reference slots. +- Pipeline-level `references` remain defaults. A pipeline-level binding is + valid when any declared eligible module in the pipeline declares the slot. In + a run, it is applied only to selected targets that declare the slot. +- Stage-local reference bindings override pipeline-level defaults for that + target. +- CLI `--reference` and `--without-reference` can target chunk, extract, and + normalize references explicitly, while preserving existing unambiguous flat + forms. +- Reference materialization still accepts file producers only, validates UTF-8, + infers media types from extensions, enforces optional `AcceptedMediaTypes` and + `MaxBytes`, warns for empty files, and never writes content to diagnostics, + logs, errors, or manifests. +- Manifest and diagnostics provenance identifies the reference target stage and, + for lane-owned stages, the lane ID. + +## Target Config Shape + +Keep existing config compatibility and add stage-specific binding locations: + +```yaml +pipelines: + dnd-session: + input: seriatim + references: + campaign_context: ./campaign/context.md + chunk: + module: dnd/scenes + references: + scene_guide: ./campaign/scenes.md + artifacts: + spells: + extract: dnd/spells + references: + glossary: ./campaign/glossary.md + normalize: + module: noop + references: + normalization_notes: ./campaign/normalization.md +``` + +Compatibility rule: + +- Existing `artifacts..references` remains supported as an alias for + extractor references. +- New `artifacts..extract.references` is the canonical extract-stage + location. +- If both the legacy lane-level `references` map and + `extract.references` bind the same slot, `extract.references` wins. This keeps + migration deterministic and avoids breaking existing configs. + +`chunk.references` and `normalize.references` are supported only when their +module binding uses object form. Scalar bindings continue to work when no +stage-local references are needed. + +## Target CLI Reference Selectors + +Preserve existing valid forms and add explicit stage selectors: + +- `slot=path`: valid only when exactly one selected reference target across + chunk, selected extractors, and selected normalizers declares `slot`. +- `chunk.slot=path`: targets the singleton chunk module. +- `lane.slot=path`: valid only when exactly one selected module in that lane, + extract or normalize, declares `slot`. +- `lane.extract.slot=path`: targets a lane extractor. +- `lane.normalize.slot=path`: targets a lane normalizer. + +`--without-reference` accepts the same selector forms without `=path`. + +Ambiguous selectors must fail before any LLM call and should name the ambiguous +targets and the more specific selector forms. + +## Stage 1: Contract And Registry Expansion + +Add reference declaration support to chunkers and normalizers and LLM delivery +to normalizers. This stage should not change config shape or runtime reference +resolution yet. + +Implementation steps: + +- Add `References contracts.ReferenceSet` to `contracts.ChunkRequest`. +- Add `LLMClient contracts.StructuredLLMClient` and + `References contracts.ReferenceSet` to `contracts.NormalizeRequest`. +- Add `ReferenceSlots() []ReferenceSlot` to `contracts.Chunker` and + `contracts.Normalizer`. +- Update every concrete chunker and normalizer plus all test fakes to implement + `ReferenceSlots()`. Existing modules without references should return `nil`. +- Change `validateModuleSpec` so `ModuleSpec.ReferenceSlots` is allowed for + `StageChunk`, `StageExtract`, and `StageNormalize`, and rejected for all other + stages. +- Add registry/module tests proving: + - chunker specs may declare valid reference slots; + - normalizer specs may declare valid reference slots; + - extractor specs still may declare valid reference slots; + - input, merge, validate, and output specs reject reference slots; + - invalid slot names, duplicate slots, and negative `MaxBytes` are rejected + for all eligible stages. +- Add production registration tests comparing runtime `ReferenceSlots()` with + registered `ModuleSpec.ReferenceSlots` for chunkers, extractors, and + normalizers that declare slots. Modules with no slots should be covered by + simple nil/empty assertions. + +Verification: + +- `go test ./internal/framework/contracts` +- `go test ./internal/framework/pipeline` +- `go test ./internal/modules/chunk/...` +- `go test ./internal/modules/normalize/...` +- `go test ./...` + +## Stage 2: Resolved Reference Target Model + +Replace extractor-only resolved reference storage with a target-aware model. +Keep external config and CLI behavior unchanged in this stage where possible. + +Implementation steps: + +- Introduce a target model in `internal/framework/pipeline`, for example: + - target stage: `chunk`, `extract`, or `normalize`; + - optional lane ID for lane-owned targets; + - module key; + - reference bindings; + - materialized `ReferenceSet`. +- Add a resolved chunk reference holder to `ResolvedPipeline`. +- Split lane reference storage into extract and normalize holders on + `ResolvedArtifactLane`. +- Keep JSON field names deterministic and redaction-friendly. Do not expose + reference content through resolved pipeline JSON. +- Update `CloneReferenceSet`, `ReferenceProvenance`, redacted effective config, + resolved-pipeline diagnostics, and any affected tests for the target-aware + model. +- Preserve existing extractor behavior by mapping old lane extractor references + into the new extract target. +- Keep the pipeline digest deterministic and based on resolved reference + bindings, not materialized reference bytes. Materialized reference digests + remain manifest provenance. + +Verification: + +- `go test ./internal/framework/pipeline` +- `go test ./internal/core/config` +- `go test ./internal/core/diagnostics` +- `go test ./...` + +## Stage 3: Config Parsing And Validation + +Add stage-local reference fields while preserving existing configs. + +Implementation steps: + +- Extend the file config module binding object to accept `references`. +- Add `References map[string]string` to `pipeline.ModuleBinding`, or use a + narrower equivalent if the implementation can do so without duplicating + binding parsing. If added to `ModuleBinding`, validation must reject + references on non-eligible stages. +- Apply file config references from: + - `pipelines..chunk.references`; + - `pipelines..artifacts..extract.references`; + - `pipelines..artifacts..normalize.references`; + - existing `pipelines..artifacts..references` as an extract-stage + compatibility alias. +- Validate reference maps for empty names, duplicate-after-trim names, and empty + sources at each new location. +- Reject references on `input`, `merge`, validators, and `output` bindings with + clear config errors. +- Preserve existing `llm_profile` validation for chunk, extract, and normalize + bindings. No new LLM profile semantics are introduced in this pass. +- Add config tests for: + - parsing `chunk.references`; + - parsing `extract.references`; + - parsing `normalize.references`; + - legacy lane `references` compatibility; + - duplicate conflicts where `extract.references` overrides legacy lane + references for the same slot; + - rejection of references on unsupported module binding locations; + - redaction/effective config cloning for all new reference maps. + +Verification: + +- `go test ./internal/core/config` +- `go test ./internal/framework/pipeline` +- `go test ./...` + +## Stage 4: Pipeline Resolution + +Resolve reference bindings against all eligible stage targets and selected lanes +without constructing modules. + +Implementation steps: + +- During `ResolvePipeline`, collect declared reference slots from: + - the configured chunker spec; + - every declared artifact lane extractor spec; + - every declared artifact lane normalizer spec. +- Keep pipeline-level reference validation broad: a pipeline-level binding is + valid if any declared eligible target in the full pipeline declares that slot, + even when `--only` selects lanes that do not consume it. +- During a selected run, apply pipeline-level defaults only to: + - the singleton chunk target if its chunker declares the slot; + - selected extractor targets that declare the slot; + - selected normalizer targets that declare the slot. +- Apply stage-local references after pipeline defaults: + - chunk-local references override chunk pipeline defaults; + - extract-local references override extract pipeline defaults; + - normalize-local references override normalize pipeline defaults. +- Enforce required slots independently per selected target after defaults, + local references, CLI overrides, and CLI unbinds. +- Keep lane-local strictness: extract-local references must be declared by that + lane's extractor, and normalize-local references must be declared by that + lane's normalizer. +- Return clear errors that name pipeline ID, stage, lane ID when present, slot, + and module key. +- Add pipeline tests for: + - pipeline default consumed by chunk only; + - pipeline default consumed by extract only; + - pipeline default consumed by normalize only; + - one pipeline default consumed by several eligible targets; + - valid pipeline default declared only by an unselected lane target; + - unknown pipeline-level slot rejected when no declared target has it; + - extract-local reference rejected by normalizer-only slot; + - normalize-local reference rejected by extractor-only slot; + - required chunk reference missing; + - required normalize reference missing; + - local overrides over pipeline defaults for each eligible stage. + +Verification: + +- `go test ./internal/framework/pipeline` +- `go test ./internal/core/config` +- `go test ./...` + +## Stage 5: CLI Reference Selectors + +Extend CLI reference override and unbind resolution to target chunk, extract, +and normalize references. + +Implementation steps: + +- Replace extractor-only CLI reference target lookup with target-aware lookup + over the resolved eligible targets for the selected run. +- Support selector forms listed in "Target CLI Reference Selectors". +- Preserve existing behavior for unambiguous `slot` and `lane.slot` forms. +- Reject ambiguous `slot` and `lane.slot` forms with a message naming matching + targets and suggesting `chunk.slot`, `lane.extract.slot`, or + `lane.normalize.slot` as appropriate. +- Reject selectors that target an unselected lane during `--only` runs. +- Reject selectors for stages that do not declare the slot. +- Keep `--reference` empty-path rejection and `--without-reference` optional + unbinding semantics. +- Add CLI tests for: + - `chunk.slot=path`; + - `lane.extract.slot=path`; + - `lane.normalize.slot=path`; + - flat slot resolution across exactly one target; + - lane slot resolution across exactly one lane-local target; + - ambiguous flat slot across chunk and extract; + - ambiguous lane slot across extract and normalize; + - unbind forms for chunk, extract, and normalize; + - unbind leaving required chunk or normalize slot missing; + - `--only` with an unselected lane selector. + +Verification: + +- `go test ./internal/cli` +- `go test ./internal/framework/pipeline` +- `go test ./...` + +## Stage 6: Materialization And Provenance + +Materialize references for chunk, extract, and normalize targets with one shared +implementation. + +Implementation steps: + +- Refactor materialization so it accepts a resolved target plus that target's + declared slots, rather than assuming an extractor lane. +- Reuse the existing file behavior: + - config paths relative to the config file; + - CLI paths relative to the current working directory; + - UTF-8 validation for all files; + - extension-based media type inference; + - optional `AcceptedMediaTypes` enforcement; + - optional `MaxBytes` enforcement; + - empty-file warning; + - `sha256:` content digest. +- Store materialized reference sets on the resolved chunk target, extract + targets, and normalize targets. +- Update warnings to include target context in `Scope`, such as: + - `pipeline..chunk.reference.`; + - `pipeline..lane..extract.reference.`; + - `pipeline..lane..normalize.reference.`. +- Update manifest and diagnostics provenance to include: + - `stage`: `chunk`, `extract`, or `normalize`; + - `lane_id`: omitted for chunk and present for extract/normalize; + - `slot_name`; + - origin type and URI; + - digest; + - media type; + - size bytes; + - binding source. +- Keep reference content out of manifests, diagnostics, logs, and errors. +- Add tests for materialization and provenance for all three eligible stages, + including media-type rejection and UTF-8 rejection on non-extractor targets. + +Verification: + +- `go test ./internal/framework/pipeline` +- `go test ./internal/core/artifacts` +- `go test ./internal/core/diagnostics` +- `go test ./internal/modules/output/json` +- `go test ./internal/cli` +- `go test ./...` + +## Stage 7: Runner Delivery + +Deliver LLM clients and references to eligible stage modules at runtime. + +Implementation steps: + +- Pass `CloneReferenceSet` of the materialized chunk reference set into + `contracts.ChunkRequest.References`. +- Continue passing `LLMClient` into `contracts.ChunkRequest`. +- Continue passing extract references and `LLMClient` into + `contracts.ExtractionRequest`. +- Pass `LLMClient` and `CloneReferenceSet` of the materialized normalize + reference set into `contracts.NormalizeRequest`. +- Ensure normalizers receive their configured `LLMProfile`, options, and + metadata as they do today. +- Add runner tests proving: + - chunkers receive LLM client and chunk references; + - extractors still receive LLM client and extract references; + - normalizers receive LLM client and normalize references; + - reference sets are cloned before delivery so modules cannot mutate resolved + pipeline state; + - missing LLM client still fails only when a module actually uses it, matching + existing chunk/extract behavior. +- Update the walking skeleton to exercise LLM-backed normalize access if a small + fake normalizer can do so without making the fixture noisy. + +Verification: + +- `go test ./internal/framework/pipeline` +- `go test ./...` + +## Stage 8: Prompt Helpers And Module Usage + +Make prompt reference helpers usable by chunk and normalize modules, then update +production modules only where needed. + +Implementation steps: + +- Confirm `internal/framework/prompt` reference rendering is contract-based and + not extractor-specific. If any naming or error text says extractor-only, + generalize it to module/stage language. +- Add prompt tests rendering references for a fake chunker or normalizer prompt + bundle if existing tests cover only extractor usage. +- Update production chunkers and normalizers: + - `generic` chunker: no reference slots. + - `dnd/scenes` chunker: no reference slots in this pass unless a clear + module-owned slot already exists. + - `noop` normalizer: no reference slots. +- Do not invent a new production normalize module in this pass. The goal is + framework support, not a domain normalizer. + +Verification: + +- `go test ./internal/framework/prompt` +- `go test ./internal/modules/chunk/...` +- `go test ./internal/modules/normalize/...` +- `go test ./...` + +## Stage 9: Documentation And Examples + +After behavior is implemented and tested, update canonical docs and examples. + +Implementation steps: + +- Update `docs/cli.md` with the expanded `--reference` and + `--without-reference` selector grammar and examples. +- Update `docs/config.md` with `chunk.references`, + `artifacts..extract.references`, `artifacts..normalize.references`, + and the legacy `artifacts..references` compatibility rule. +- Update `docs/internal/modules.md` with module-author guidance for + `ReferenceSlots()` on chunkers, extractors, and normalizers, plus LLM access + expectations for chunk, extract, and normalize. +- Update `docs/internal/pipeline.md` with target-aware reference resolution, + materialization, provenance, and runtime delivery. +- Update `docs/integrations/json-output.md` with the new reference provenance + shape including `stage` and optional `lane_id`. +- Update `docs/troubleshooting.md` with ambiguous selector examples for chunk, + extract, and normalize targets. +- Update maintained examples only if they can remain small and validated. Do + not add speculative reference files for chunk or normalize modules without a + production module that actually declares those slots. +- Update `docs/roadmap/future.md` by removing or revising the two completed + items while leaving still-deferred work such as token budgeting and non-file + producers. +- Replace this implementation plan with a concise completed-note document after + implementation is finished. + +Verification: + +- `rg -n "chunk.references|normalize.references|lane.extract.slot|lane.normalize.slot" docs/cli.md docs/config.md docs/internal docs/integrations docs/troubleshooting.md` +- `go test ./...` +- `go vet ./...` +- `go build ./cmd/notarius` + +## Final Acceptance Criteria + +The feature is complete when: + +- chunkers, extractors, and normalizers can declare reference slots through + runtime contracts and registry specs; +- normalizers receive the structured LLM client at runtime; +- config can bind references at pipeline, chunk, extract, and normalize levels; +- CLI reference overrides and unbinds can target chunk, extract, and normalize + references without ambiguous selector behavior; +- pipeline resolution validates all reference bindings before any LLM call; +- materialization and provenance work uniformly for chunk, extract, and + normalize references; +- reference manifest entries identify stage and lane context; +- existing extractor reference behavior remains backward compatible; +- canonical docs describe only implemented behavior; +- `go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` pass.