diff --git a/docs/cli.md b/docs/cli.md index e4d9cff..a12ae85 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -102,6 +102,17 @@ go run ./cmd/notarius run dnd-session \ --reference spells.extract.glossary=./campaign-glossary.txt ``` +The same grammar can target chunk and normalize slots when the configured +modules declare them: + +```sh +go run ./cmd/notarius run dnd-session \ + --config path/to/config.yml \ + --input examples/seriatim-minimal-transcript.json \ + --reference chunk.scene_guide=./campaign-scenes.txt \ + --reference spells.normalize.normalization_notes=./normalization-notes.txt +``` + Use `--without-reference` to remove a configured optional binding for a run: ```sh diff --git a/docs/config.md b/docs/config.md index b3eaf7d..8805c42 100644 --- a/docs/config.md +++ b/docs/config.md @@ -199,6 +199,14 @@ bindings. They override pipeline-level defaults for slots declared by the chunk or normalizer module. Extractor-local references apply only to the extractor, and normalizer-local references apply only to the normalizer. +Stage-local reference fields use the same map shape at: + +- `pipelines..chunk.references` +- `pipelines..artifacts..extract.references` +- `pipelines..artifacts..normalize.references` + +Each binding is valid only when that target module declares the slot. + ## Module Bindings Every module binding may use shorthand: diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md index 573b91e..5a3cc07 100644 --- a/docs/internal/pipeline.md +++ b/docs/internal/pipeline.md @@ -58,7 +58,7 @@ extractor, or normalizer request. Prompt bundles can declare reference slots and use `reference` and `hasreference` template functions. Bundle loading validates string-literal slot -names against the declaration. Rendering receives a lane reference set from the +names against the declaration. Rendering receives a target reference set from the caller; unbound optional slots render as empty strings, and `hasreference` returns true only when at least one bound item has content. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index b70a137..520d7d7 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -8,11 +8,6 @@ future work only. - Additional input adapters, such as Markdown or note-export formats. - Additional D&D extractors beyond spell casts. -- First-class LLM access for all LLM-eligible pipeline stages: `chunk`, - `extract`, and `normalize`. -- Expand reference support beyond extractors to `chunk` and `normalize` - modules, using the same declared-slot model and preserving stage-specific - ownership boundaries. - Add non-file reference producers, such as prior-run artifacts, derived summaries, or entity registries, without changing module-facing reference item contracts. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 131e3c3..9bf33de 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,463 +1,21 @@ -# LLM And Stage Reference Expansion Implementation Plan +# LLM And Reference Expansion Completed -## Purpose +This roadmap item is implemented. -Implement these future roadmap items: +Notarius now supports first-class LLM access for chunk, extract, and normalize +modules, plus target-aware reference declaration, binding, CLI override, +materialization, provenance, and runtime delivery for chunker, extractor, and +normalizer targets. -- First-class LLM access for all LLM-eligible pipeline stages: `chunk`, - `extract`, and `normalize`. -- Expand reference support beyond extractors to `chunk` and `normalize` - modules, using the same declared-slot model and preserving stage-specific - ownership boundaries. +Current behavior is documented in: -This is a decision-complete implementation plan for an LLM coding agent. Follow -the stages in order. Keep current-behavior docs unchanged until the -corresponding behavior exists. Do not implement token budgeting, non-file -reference producers, structured references, retrieval, or multiple effective LLM -profiles in this pass. +- [CLI Reference](../cli.md) +- [Configuration](../config.md) +- [Pipeline Internals](../internal/pipeline.md) +- [Module Internals](../internal/modules.md) +- [JSON Output](../integrations/json-output.md) +- [Troubleshooting](../troubleshooting.md) -## Target Behavior - -LLM access: - -- `contracts.ChunkRequest`, `contracts.ExtractionRequest`, and - `contracts.NormalizeRequest` all carry `LLMClient`, `LLMProfile`, `Options`, - and `Metadata`. -- The runner passes the configured structured LLM client to chunkers, - extractors, and normalizers. -- The existing single-effective-LLM-profile CLI restriction remains in place. - -References: - -- `chunk`, `extract`, and `normalize` modules can declare reference slots. -- The framework treats references as opaque UTF-8 text plus provenance. Stage - modules own slot names and semantics. -- Reference slots are declared in both runtime module contracts and - `ModuleSpec.ReferenceSlots`, so config and pipeline validation can inspect - slots without constructing modules. -- `ModuleSpec.ReferenceSlots` is valid only for `chunk`, `extract`, and - `normalize` modules. `input`, `merge`, `validate`, and `output` modules must - not declare reference slots. -- Pipeline-level `references` remain defaults. A pipeline-level binding is - valid when any declared eligible module in the pipeline declares the slot. In - a run, it is applied only to selected targets that declare the slot. -- Stage-local reference bindings override pipeline-level defaults for that - target. -- CLI `--reference` and `--without-reference` can target chunk, extract, and - normalize references explicitly, while preserving existing unambiguous flat - forms. -- Reference materialization still accepts file producers only, validates UTF-8, - infers media types from extensions, enforces optional `AcceptedMediaTypes` and - `MaxBytes`, warns for empty files, and never writes content to diagnostics, - logs, errors, or manifests. -- Manifest and diagnostics provenance identifies the reference target stage and, - for lane-owned stages, the lane ID. - -## Target Config Shape - -Keep existing config compatibility and add stage-specific binding locations: - -```yaml -pipelines: - dnd-session: - input: seriatim - references: - campaign_context: ./campaign/context.md - chunk: - module: dnd/scenes - references: - scene_guide: ./campaign/scenes.md - artifacts: - spells: - extract: dnd/spells - references: - glossary: ./campaign/glossary.md - normalize: - module: noop - references: - normalization_notes: ./campaign/normalization.md -``` - -Compatibility rule: - -- Existing `artifacts..references` remains supported as an alias for - extractor references. -- New `artifacts..extract.references` is the canonical extract-stage - location. -- If both the legacy lane-level `references` map and - `extract.references` bind the same slot, `extract.references` wins. This keeps - migration deterministic and avoids breaking existing configs. - -`chunk.references` and `normalize.references` are supported only when their -module binding uses object form. Scalar bindings continue to work when no -stage-local references are needed. - -## Target CLI Reference Selectors - -Preserve existing valid forms and add explicit stage selectors: - -- `slot=path`: valid only when exactly one selected reference target across - chunk, selected extractors, and selected normalizers declares `slot`. -- `chunk.slot=path`: targets the singleton chunk module. -- `lane.slot=path`: valid only when exactly one selected module in that lane, - extract or normalize, declares `slot`. -- `lane.extract.slot=path`: targets a lane extractor. -- `lane.normalize.slot=path`: targets a lane normalizer. - -`--without-reference` accepts the same selector forms without `=path`. - -Ambiguous selectors must fail before any LLM call and should name the ambiguous -targets and the more specific selector forms. - -## Stage 1: Contract And Registry Expansion - -Add reference declaration support to chunkers and normalizers and LLM delivery -to normalizers. This stage should not change config shape or runtime reference -resolution yet. - -Implementation steps: - -- Add `References contracts.ReferenceSet` to `contracts.ChunkRequest`. -- Add `LLMClient contracts.StructuredLLMClient` and - `References contracts.ReferenceSet` to `contracts.NormalizeRequest`. -- Add `ReferenceSlots() []ReferenceSlot` to `contracts.Chunker` and - `contracts.Normalizer`. -- Update every concrete chunker and normalizer plus all test fakes to implement - `ReferenceSlots()`. Existing modules without references should return `nil`. -- Change `validateModuleSpec` so `ModuleSpec.ReferenceSlots` is allowed for - `StageChunk`, `StageExtract`, and `StageNormalize`, and rejected for all other - stages. -- Add registry/module tests proving: - - chunker specs may declare valid reference slots; - - normalizer specs may declare valid reference slots; - - extractor specs still may declare valid reference slots; - - input, merge, validate, and output specs reject reference slots; - - invalid slot names, duplicate slots, and negative `MaxBytes` are rejected - for all eligible stages. -- Add production registration tests comparing runtime `ReferenceSlots()` with - registered `ModuleSpec.ReferenceSlots` for chunkers, extractors, and - normalizers that declare slots. Modules with no slots should be covered by - simple nil/empty assertions. - -Verification: - -- `go test ./internal/framework/contracts` -- `go test ./internal/framework/pipeline` -- `go test ./internal/modules/chunk/...` -- `go test ./internal/modules/normalize/...` -- `go test ./...` - -## Stage 2: Resolved Reference Target Model - -Replace extractor-only resolved reference storage with a target-aware model. -Keep external config and CLI behavior unchanged in this stage where possible. - -Implementation steps: - -- Introduce a target model in `internal/framework/pipeline`, for example: - - target stage: `chunk`, `extract`, or `normalize`; - - optional lane ID for lane-owned targets; - - module key; - - reference bindings; - - materialized `ReferenceSet`. -- Add a resolved chunk reference holder to `ResolvedPipeline`. -- Split lane reference storage into extract and normalize holders on - `ResolvedArtifactLane`. -- Keep JSON field names deterministic and redaction-friendly. Do not expose - reference content through resolved pipeline JSON. -- Update `CloneReferenceSet`, `ReferenceProvenance`, redacted effective config, - resolved-pipeline diagnostics, and any affected tests for the target-aware - model. -- Preserve existing extractor behavior by mapping old lane extractor references - into the new extract target. -- Keep the pipeline digest deterministic and based on resolved reference - bindings, not materialized reference bytes. Materialized reference digests - remain manifest provenance. - -Verification: - -- `go test ./internal/framework/pipeline` -- `go test ./internal/core/config` -- `go test ./internal/core/diagnostics` -- `go test ./...` - -## Stage 3: Config Parsing And Validation - -Add stage-local reference fields while preserving existing configs. - -Implementation steps: - -- Extend the file config module binding object to accept `references`. -- Add `References map[string]string` to `pipeline.ModuleBinding`, or use a - narrower equivalent if the implementation can do so without duplicating - binding parsing. If added to `ModuleBinding`, validation must reject - references on non-eligible stages. -- Apply file config references from: - - `pipelines..chunk.references`; - - `pipelines..artifacts..extract.references`; - - `pipelines..artifacts..normalize.references`; - - existing `pipelines..artifacts..references` as an extract-stage - compatibility alias. -- Validate reference maps for empty names, duplicate-after-trim names, and empty - sources at each new location. -- Reject references on `input`, `merge`, validators, and `output` bindings with - clear config errors. -- Preserve existing `llm_profile` validation for chunk, extract, and normalize - bindings. No new LLM profile semantics are introduced in this pass. -- Add config tests for: - - parsing `chunk.references`; - - parsing `extract.references`; - - parsing `normalize.references`; - - legacy lane `references` compatibility; - - duplicate conflicts where `extract.references` overrides legacy lane - references for the same slot; - - rejection of references on unsupported module binding locations; - - redaction/effective config cloning for all new reference maps. - -Verification: - -- `go test ./internal/core/config` -- `go test ./internal/framework/pipeline` -- `go test ./...` - -## Stage 4: Pipeline Resolution - -Resolve reference bindings against all eligible stage targets and selected lanes -without constructing modules. - -Implementation steps: - -- During `ResolvePipeline`, collect declared reference slots from: - - the configured chunker spec; - - every declared artifact lane extractor spec; - - every declared artifact lane normalizer spec. -- Keep pipeline-level reference validation broad: a pipeline-level binding is - valid if any declared eligible target in the full pipeline declares that slot, - even when `--only` selects lanes that do not consume it. -- During a selected run, apply pipeline-level defaults only to: - - the singleton chunk target if its chunker declares the slot; - - selected extractor targets that declare the slot; - - selected normalizer targets that declare the slot. -- Apply stage-local references after pipeline defaults: - - chunk-local references override chunk pipeline defaults; - - extract-local references override extract pipeline defaults; - - normalize-local references override normalize pipeline defaults. -- Enforce required slots independently per selected target after defaults, - local references, CLI overrides, and CLI unbinds. -- Keep lane-local strictness: extract-local references must be declared by that - lane's extractor, and normalize-local references must be declared by that - lane's normalizer. -- Return clear errors that name pipeline ID, stage, lane ID when present, slot, - and module key. -- Add pipeline tests for: - - pipeline default consumed by chunk only; - - pipeline default consumed by extract only; - - pipeline default consumed by normalize only; - - one pipeline default consumed by several eligible targets; - - valid pipeline default declared only by an unselected lane target; - - unknown pipeline-level slot rejected when no declared target has it; - - extract-local reference rejected by normalizer-only slot; - - normalize-local reference rejected by extractor-only slot; - - required chunk reference missing; - - required normalize reference missing; - - local overrides over pipeline defaults for each eligible stage. - -Verification: - -- `go test ./internal/framework/pipeline` -- `go test ./internal/core/config` -- `go test ./...` - -## Stage 5: CLI Reference Selectors - -Extend CLI reference override and unbind resolution to target chunk, extract, -and normalize references. - -Implementation steps: - -- Replace extractor-only CLI reference target lookup with target-aware lookup - over the resolved eligible targets for the selected run. -- Support selector forms listed in "Target CLI Reference Selectors". -- Preserve existing behavior for unambiguous `slot` and `lane.slot` forms. -- Reject ambiguous `slot` and `lane.slot` forms with a message naming matching - targets and suggesting `chunk.slot`, `lane.extract.slot`, or - `lane.normalize.slot` as appropriate. -- Reject selectors that target an unselected lane during `--only` runs. -- Reject selectors for stages that do not declare the slot. -- Keep `--reference` empty-path rejection and `--without-reference` optional - unbinding semantics. -- Add CLI tests for: - - `chunk.slot=path`; - - `lane.extract.slot=path`; - - `lane.normalize.slot=path`; - - flat slot resolution across exactly one target; - - lane slot resolution across exactly one lane-local target; - - ambiguous flat slot across chunk and extract; - - ambiguous lane slot across extract and normalize; - - unbind forms for chunk, extract, and normalize; - - unbind leaving required chunk or normalize slot missing; - - `--only` with an unselected lane selector. - -Verification: - -- `go test ./internal/cli` -- `go test ./internal/framework/pipeline` -- `go test ./...` - -## Stage 6: Materialization And Provenance - -Materialize references for chunk, extract, and normalize targets with one shared -implementation. - -Implementation steps: - -- Refactor materialization so it accepts a resolved target plus that target's - declared slots, rather than assuming an extractor lane. -- Reuse the existing file behavior: - - config paths relative to the config file; - - CLI paths relative to the current working directory; - - UTF-8 validation for all files; - - extension-based media type inference; - - optional `AcceptedMediaTypes` enforcement; - - optional `MaxBytes` enforcement; - - empty-file warning; - - `sha256:` content digest. -- Store materialized reference sets on the resolved chunk target, extract - targets, and normalize targets. -- Update warnings to include target context in `Scope`, such as: - - `pipeline..chunk.reference.`; - - `pipeline..lane..extract.reference.`; - - `pipeline..lane..normalize.reference.`. -- Update manifest and diagnostics provenance to include: - - `stage`: `chunk`, `extract`, or `normalize`; - - `lane_id`: omitted for chunk and present for extract/normalize; - - `slot_name`; - - origin type and URI; - - digest; - - media type; - - size bytes; - - binding source. -- Keep reference content out of manifests, diagnostics, logs, and errors. -- Add tests for materialization and provenance for all three eligible stages, - including media-type rejection and UTF-8 rejection on non-extractor targets. - -Verification: - -- `go test ./internal/framework/pipeline` -- `go test ./internal/core/artifacts` -- `go test ./internal/core/diagnostics` -- `go test ./internal/modules/output/json` -- `go test ./internal/cli` -- `go test ./...` - -## Stage 7: Runner Delivery - -Deliver LLM clients and references to eligible stage modules at runtime. - -Implementation steps: - -- Pass `CloneReferenceSet` of the materialized chunk reference set into - `contracts.ChunkRequest.References`. -- Continue passing `LLMClient` into `contracts.ChunkRequest`. -- Continue passing extract references and `LLMClient` into - `contracts.ExtractionRequest`. -- Pass `LLMClient` and `CloneReferenceSet` of the materialized normalize - reference set into `contracts.NormalizeRequest`. -- Ensure normalizers receive their configured `LLMProfile`, options, and - metadata as they do today. -- Add runner tests proving: - - chunkers receive LLM client and chunk references; - - extractors still receive LLM client and extract references; - - normalizers receive LLM client and normalize references; - - reference sets are cloned before delivery so modules cannot mutate resolved - pipeline state; - - missing LLM client still fails only when a module actually uses it, matching - existing chunk/extract behavior. -- Update the walking skeleton to exercise LLM-backed normalize access if a small - fake normalizer can do so without making the fixture noisy. - -Verification: - -- `go test ./internal/framework/pipeline` -- `go test ./...` - -## Stage 8: Prompt Helpers And Module Usage - -Make prompt reference helpers usable by chunk and normalize modules, then update -production modules only where needed. - -Implementation steps: - -- Confirm `internal/framework/prompt` reference rendering is contract-based and - not extractor-specific. If any naming or error text says extractor-only, - generalize it to module/stage language. -- Add prompt tests rendering references for a fake chunker or normalizer prompt - bundle if existing tests cover only extractor usage. -- Update production chunkers and normalizers: - - `generic` chunker: no reference slots. - - `dnd/scenes` chunker: no reference slots in this pass unless a clear - module-owned slot already exists. - - `noop` normalizer: no reference slots. -- Do not invent a new production normalize module in this pass. The goal is - framework support, not a domain normalizer. - -Verification: - -- `go test ./internal/framework/prompt` -- `go test ./internal/modules/chunk/...` -- `go test ./internal/modules/normalize/...` -- `go test ./...` - -## Stage 9: Documentation And Examples - -After behavior is implemented and tested, update canonical docs and examples. - -Implementation steps: - -- Update `docs/cli.md` with the expanded `--reference` and - `--without-reference` selector grammar and examples. -- Update `docs/config.md` with `chunk.references`, - `artifacts..extract.references`, `artifacts..normalize.references`, - and the legacy `artifacts..references` compatibility rule. -- Update `docs/internal/modules.md` with module-author guidance for - `ReferenceSlots()` on chunkers, extractors, and normalizers, plus LLM access - expectations for chunk, extract, and normalize. -- Update `docs/internal/pipeline.md` with target-aware reference resolution, - materialization, provenance, and runtime delivery. -- Update `docs/integrations/json-output.md` with the new reference provenance - shape including `stage` and optional `lane_id`. -- Update `docs/troubleshooting.md` with ambiguous selector examples for chunk, - extract, and normalize targets. -- Update maintained examples only if they can remain small and validated. Do - not add speculative reference files for chunk or normalize modules without a - production module that actually declares those slots. -- Update `docs/roadmap/future.md` by removing or revising the two completed - items while leaving still-deferred work such as token budgeting and non-file - producers. -- Replace this implementation plan with a concise completed-note document after - implementation is finished. - -Verification: - -- `rg -n "chunk.references|normalize.references|lane.extract.slot|lane.normalize.slot" docs/cli.md docs/config.md docs/internal docs/integrations docs/troubleshooting.md` -- `go test ./...` -- `go vet ./...` -- `go build ./cmd/notarius` - -## Final Acceptance Criteria - -The feature is complete when: - -- chunkers, extractors, and normalizers can declare reference slots through - runtime contracts and registry specs; -- normalizers receive the structured LLM client at runtime; -- config can bind references at pipeline, chunk, extract, and normalize levels; -- CLI reference overrides and unbinds can target chunk, extract, and normalize - references without ambiguous selector behavior; -- pipeline resolution validates all reference bindings before any LLM call; -- materialization and provenance work uniformly for chunk, extract, and - normalize references; -- reference manifest entries identify stage and lane context; -- existing extractor reference behavior remains backward compatible; -- canonical docs describe only implemented behavior; -- `go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` pass. +Deferred reference work remains tracked in [Future Work](future.md), including +non-file reference producers, token budgeting, structured references, and +reference preprocessing or retrieval. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index fce0578..53ec9e2 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -107,10 +107,10 @@ go run ./cmd/notarius run dnd-session \ Symptoms include: - `reference slot "..." is not declared` -- `reference slot "..." is declared by multiple selected lanes` +- `reference slot "..." is declared by multiple selected targets` - `required reference slot "..." is not bound` -- `--reference must use slot=path or lane.slot=path` -- `--without-reference must use slot or lane.slot` +- `--reference must use slot=path` +- `--without-reference must use a reference selector without =path` - `read "...": no such file` - `must be UTF-8 text` - `media type "..." is not accepted` @@ -118,17 +118,24 @@ Symptoms include: Fix: -- Confirm the selected extractor declares the slot. The implemented - `dnd/spells` extractor declares optional `roster` and `glossary` slots. -- Use `lane.slot=path` when more than one selected lane declares the same slot. -- Use `--without-reference slot` to remove optional config bindings; do not pass - an empty `--reference slot=`. +- Confirm the selected chunker, extractor, or normalizer declares the slot. The + implemented `dnd/spells` extractor declares optional `roster` and `glossary` + slots. +- Use a specific selector when more than one selected target declares the same + slot. Examples include `chunk.context=./context.txt`, + `spells.extract.context=./extract-context.txt`, and + `spells.normalize.context=./normalize-context.txt`. +- `lane.slot=path` is valid only when exactly one selected extractor or + normalizer in that lane declares the slot. If both do, use + `lane.extract.slot=path` or `lane.normalize.slot=path`. +- Use `--without-reference selector` to remove optional config bindings; do not + pass an empty `--reference selector=`. - Check whether a path came from config or CLI. Config paths are relative to the config file. CLI reference paths are relative to the current working directory. - Ensure the file is readable UTF-8 text and within any byte limit declared by - the extractor. -- If the extractor narrows accepted media types, use a file extension that + the declaring module. +- If the declaring module narrows accepted media types, use a file extension that infers an accepted type such as `text/markdown` or `application/json`. Unknown extensions infer `application/octet-stream`. - If diagnostics are retained, inspect `resolved-pipeline.json`,