notarius/docs/roadmap/implementation.md

LLM And Stage Reference Expansion Implementation Plan

Purpose

Implement these future roadmap items:

• First-class LLM access for all LLM-eligible pipeline stages: chunk, extract, and normalize.
• Expand reference support beyond extractors to chunk and normalize modules, using the same declared-slot model and preserving stage-specific ownership boundaries.

This is a decision-complete implementation plan for an LLM coding agent. Follow the stages in order. Keep current-behavior docs unchanged until the corresponding behavior exists. Do not implement token budgeting, non-file reference producers, structured references, retrieval, or multiple effective LLM profiles in this pass.

Target Behavior

LLM access:

• contracts.ChunkRequest, contracts.ExtractionRequest, and contracts.NormalizeRequest all carry LLMClient, LLMProfile, Options, and Metadata.
• The runner passes the configured structured LLM client to chunkers, extractors, and normalizers.
• The existing single-effective-LLM-profile CLI restriction remains in place.

References:

• chunk, extract, and normalize modules can declare reference slots.
• The framework treats references as opaque UTF-8 text plus provenance. Stage modules own slot names and semantics.
• Reference slots are declared in both runtime module contracts and ModuleSpec.ReferenceSlots, so config and pipeline validation can inspect slots without constructing modules.
• ModuleSpec.ReferenceSlots is valid only for chunk, extract, and normalize modules. input, merge, validate, and output modules must not declare reference slots.
• Pipeline-level references remain defaults. A pipeline-level binding is valid when any declared eligible module in the pipeline declares the slot. In a run, it is applied only to selected targets that declare the slot.
• Stage-local reference bindings override pipeline-level defaults for that target.
• CLI --reference and --without-reference can target chunk, extract, and normalize references explicitly, while preserving existing unambiguous flat forms.
• Reference materialization still accepts file producers only, validates UTF-8, infers media types from extensions, enforces optional AcceptedMediaTypes and MaxBytes, warns for empty files, and never writes content to diagnostics, logs, errors, or manifests.
• Manifest and diagnostics provenance identifies the reference target stage and, for lane-owned stages, the lane ID.

Target Config Shape

Keep existing config compatibility and add stage-specific binding locations:

pipelines:
  dnd-session:
    input: seriatim
    references:
      campaign_context: ./campaign/context.md
    chunk:
      module: dnd/scenes
      references:
        scene_guide: ./campaign/scenes.md
    artifacts:
      spells:
        extract: dnd/spells
        references:
          glossary: ./campaign/glossary.md
        normalize:
          module: noop
          references:
            normalization_notes: ./campaign/normalization.md

Compatibility rule:

• Existing artifacts.<lane>.references remains supported as an alias for extractor references.
• New artifacts.<lane>.extract.references is the canonical extract-stage location.
• If both the legacy lane-level references map and extract.references bind the same slot, extract.references wins. This keeps migration deterministic and avoids breaking existing configs.

chunk.references and normalize.references are supported only when their module binding uses object form. Scalar bindings continue to work when no stage-local references are needed.

Target CLI Reference Selectors

Preserve existing valid forms and add explicit stage selectors:

• slot=path: valid only when exactly one selected reference target across chunk, selected extractors, and selected normalizers declares slot.
• chunk.slot=path: targets the singleton chunk module.
• lane.slot=path: valid only when exactly one selected module in that lane, extract or normalize, declares slot.
• lane.extract.slot=path: targets a lane extractor.
• lane.normalize.slot=path: targets a lane normalizer.

--without-reference accepts the same selector forms without =path.

Ambiguous selectors must fail before any LLM call and should name the ambiguous targets and the more specific selector forms.

Stage 1: Contract And Registry Expansion

Add reference declaration support to chunkers and normalizers and LLM delivery to normalizers. This stage should not change config shape or runtime reference resolution yet.

Implementation steps:

• Add References contracts.ReferenceSet to contracts.ChunkRequest.
• Add LLMClient contracts.StructuredLLMClient and References contracts.ReferenceSet to contracts.NormalizeRequest.
• Add ReferenceSlots() []ReferenceSlot to contracts.Chunker and contracts.Normalizer.
• Update every concrete chunker and normalizer plus all test fakes to implement ReferenceSlots(). Existing modules without references should return nil.
• Change validateModuleSpec so ModuleSpec.ReferenceSlots is allowed for StageChunk, StageExtract, and StageNormalize, and rejected for all other stages.
• Add registry/module tests proving:
  • chunker specs may declare valid reference slots;
  • normalizer specs may declare valid reference slots;
  • extractor specs still may declare valid reference slots;
  • input, merge, validate, and output specs reject reference slots;
  • invalid slot names, duplicate slots, and negative MaxBytes are rejected for all eligible stages.
• Add production registration tests comparing runtime ReferenceSlots() with registered ModuleSpec.ReferenceSlots for chunkers, extractors, and normalizers that declare slots. Modules with no slots should be covered by simple nil/empty assertions.

Verification:

• go test ./internal/framework/contracts
• go test ./internal/framework/pipeline
• go test ./internal/modules/chunk/...
• go test ./internal/modules/normalize/...
• go test ./...

Stage 2: Resolved Reference Target Model

Replace extractor-only resolved reference storage with a target-aware model. Keep external config and CLI behavior unchanged in this stage where possible.

Implementation steps:

• Introduce a target model in internal/framework/pipeline, for example:
  • target stage: chunk, extract, or normalize;
  • optional lane ID for lane-owned targets;
  • module key;
  • reference bindings;
  • materialized ReferenceSet.
• Add a resolved chunk reference holder to ResolvedPipeline.
• Split lane reference storage into extract and normalize holders on ResolvedArtifactLane.
• Keep JSON field names deterministic and redaction-friendly. Do not expose reference content through resolved pipeline JSON.
• Update CloneReferenceSet, ReferenceProvenance, redacted effective config, resolved-pipeline diagnostics, and any affected tests for the target-aware model.
• Preserve existing extractor behavior by mapping old lane extractor references into the new extract target.
• Keep the pipeline digest deterministic and based on resolved reference bindings, not materialized reference bytes. Materialized reference digests remain manifest provenance.

Verification:

• go test ./internal/framework/pipeline
• go test ./internal/core/config
• go test ./internal/core/diagnostics
• go test ./...

Stage 3: Config Parsing And Validation

Add stage-local reference fields while preserving existing configs.

Implementation steps:

• Extend the file config module binding object to accept references.
• Add References map[string]string to pipeline.ModuleBinding, or use a narrower equivalent if the implementation can do so without duplicating binding parsing. If added to ModuleBinding, validation must reject references on non-eligible stages.
• Apply file config references from:
  • pipelines.<id>.chunk.references;
  • pipelines.<id>.artifacts.<lane>.extract.references;
  • pipelines.<id>.artifacts.<lane>.normalize.references;
  • existing pipelines.<id>.artifacts.<lane>.references as an extract-stage compatibility alias.
• Validate reference maps for empty names, duplicate-after-trim names, and empty sources at each new location.
• Reject references on input, merge, validators, and output bindings with clear config errors.
• Preserve existing llm_profile validation for chunk, extract, and normalize bindings. No new LLM profile semantics are introduced in this pass.
• Add config tests for:
  • parsing chunk.references;
  • parsing extract.references;
  • parsing normalize.references;
  • legacy lane references compatibility;
  • duplicate conflicts where extract.references overrides legacy lane references for the same slot;
  • rejection of references on unsupported module binding locations;
  • redaction/effective config cloning for all new reference maps.

Verification:

• go test ./internal/core/config
• go test ./internal/framework/pipeline
• go test ./...

Stage 4: Pipeline Resolution

Resolve reference bindings against all eligible stage targets and selected lanes without constructing modules.

Implementation steps:

• During ResolvePipeline, collect declared reference slots from:
  • the configured chunker spec;
  • every declared artifact lane extractor spec;
  • every declared artifact lane normalizer spec.
• Keep pipeline-level reference validation broad: a pipeline-level binding is valid if any declared eligible target in the full pipeline declares that slot, even when --only selects lanes that do not consume it.
• During a selected run, apply pipeline-level defaults only to:
  • the singleton chunk target if its chunker declares the slot;
  • selected extractor targets that declare the slot;
  • selected normalizer targets that declare the slot.
• Apply stage-local references after pipeline defaults:
  • chunk-local references override chunk pipeline defaults;
  • extract-local references override extract pipeline defaults;
  • normalize-local references override normalize pipeline defaults.
• Enforce required slots independently per selected target after defaults, local references, CLI overrides, and CLI unbinds.
• Keep lane-local strictness: extract-local references must be declared by that lane's extractor, and normalize-local references must be declared by that lane's normalizer.
• Return clear errors that name pipeline ID, stage, lane ID when present, slot, and module key.
• Add pipeline tests for:
  • pipeline default consumed by chunk only;
  • pipeline default consumed by extract only;
  • pipeline default consumed by normalize only;
  • one pipeline default consumed by several eligible targets;
  • valid pipeline default declared only by an unselected lane target;
  • unknown pipeline-level slot rejected when no declared target has it;
  • extract-local reference rejected by normalizer-only slot;
  • normalize-local reference rejected by extractor-only slot;
  • required chunk reference missing;
  • required normalize reference missing;
  • local overrides over pipeline defaults for each eligible stage.

Verification:

• go test ./internal/framework/pipeline
• go test ./internal/core/config
• go test ./...

Stage 5: CLI Reference Selectors

Extend CLI reference override and unbind resolution to target chunk, extract, and normalize references.

Implementation steps:

• Replace extractor-only CLI reference target lookup with target-aware lookup over the resolved eligible targets for the selected run.
• Support selector forms listed in "Target CLI Reference Selectors".
• Preserve existing behavior for unambiguous slot and lane.slot forms.
• Reject ambiguous slot and lane.slot forms with a message naming matching targets and suggesting chunk.slot, lane.extract.slot, or lane.normalize.slot as appropriate.
• Reject selectors that target an unselected lane during --only runs.
• Reject selectors for stages that do not declare the slot.
• Keep --reference empty-path rejection and --without-reference optional unbinding semantics.
• Add CLI tests for:
  • chunk.slot=path;
  • lane.extract.slot=path;
  • lane.normalize.slot=path;
  • flat slot resolution across exactly one target;
  • lane slot resolution across exactly one lane-local target;
  • ambiguous flat slot across chunk and extract;
  • ambiguous lane slot across extract and normalize;
  • unbind forms for chunk, extract, and normalize;
  • unbind leaving required chunk or normalize slot missing;
  • --only with an unselected lane selector.

Verification:

• go test ./internal/cli
• go test ./internal/framework/pipeline
• go test ./...

Stage 6: Materialization And Provenance

Materialize references for chunk, extract, and normalize targets with one shared implementation.

Implementation steps:

• Refactor materialization so it accepts a resolved target plus that target's declared slots, rather than assuming an extractor lane.
• Reuse the existing file behavior:
  • config paths relative to the config file;
  • CLI paths relative to the current working directory;
  • UTF-8 validation for all files;
  • extension-based media type inference;
  • optional AcceptedMediaTypes enforcement;
  • optional MaxBytes enforcement;
  • empty-file warning;
  • sha256: content digest.
• Store materialized reference sets on the resolved chunk target, extract targets, and normalize targets.
• Update warnings to include target context in Scope, such as:
  • pipeline.<id>.chunk.reference.<slot>;
  • pipeline.<id>.lane.<lane>.extract.reference.<slot>;
  • pipeline.<id>.lane.<lane>.normalize.reference.<slot>.
• Update manifest and diagnostics provenance to include:
  • stage: chunk, extract, or normalize;
  • lane_id: omitted for chunk and present for extract/normalize;
  • slot_name;
  • origin type and URI;
  • digest;
  • media type;
  • size bytes;
  • binding source.
• Keep reference content out of manifests, diagnostics, logs, and errors.
• Add tests for materialization and provenance for all three eligible stages, including media-type rejection and UTF-8 rejection on non-extractor targets.

Verification:

• go test ./internal/framework/pipeline
• go test ./internal/core/artifacts
• go test ./internal/core/diagnostics
• go test ./internal/modules/output/json
• go test ./internal/cli
• go test ./...

Stage 7: Runner Delivery

Deliver LLM clients and references to eligible stage modules at runtime.

Implementation steps:

• Pass CloneReferenceSet of the materialized chunk reference set into contracts.ChunkRequest.References.
• Continue passing LLMClient into contracts.ChunkRequest.
• Continue passing extract references and LLMClient into contracts.ExtractionRequest.
• Pass LLMClient and CloneReferenceSet of the materialized normalize reference set into contracts.NormalizeRequest.
• Ensure normalizers receive their configured LLMProfile, options, and metadata as they do today.
• Add runner tests proving:
  • chunkers receive LLM client and chunk references;
  • extractors still receive LLM client and extract references;
  • normalizers receive LLM client and normalize references;
  • reference sets are cloned before delivery so modules cannot mutate resolved pipeline state;
  • missing LLM client still fails only when a module actually uses it, matching existing chunk/extract behavior.
• Update the walking skeleton to exercise LLM-backed normalize access if a small fake normalizer can do so without making the fixture noisy.

Verification:

• go test ./internal/framework/pipeline
• go test ./...

Stage 8: Prompt Helpers And Module Usage

Make prompt reference helpers usable by chunk and normalize modules, then update production modules only where needed.

Implementation steps:

• Confirm internal/framework/prompt reference rendering is contract-based and not extractor-specific. If any naming or error text says extractor-only, generalize it to module/stage language.
• Add prompt tests rendering references for a fake chunker or normalizer prompt bundle if existing tests cover only extractor usage.
• Update production chunkers and normalizers:
  • generic chunker: no reference slots.
  • dnd/scenes chunker: no reference slots in this pass unless a clear module-owned slot already exists.
  • noop normalizer: no reference slots.
• Do not invent a new production normalize module in this pass. The goal is framework support, not a domain normalizer.

Verification:

• go test ./internal/framework/prompt
• go test ./internal/modules/chunk/...
• go test ./internal/modules/normalize/...
• go test ./...

Stage 9: Documentation And Examples

After behavior is implemented and tested, update canonical docs and examples.

Implementation steps:

• Update docs/cli.md with the expanded --reference and --without-reference selector grammar and examples.
• Update docs/config.md with chunk.references, artifacts.<lane>.extract.references, artifacts.<lane>.normalize.references, and the legacy artifacts.<lane>.references compatibility rule.
• Update docs/internal/modules.md with module-author guidance for ReferenceSlots() on chunkers, extractors, and normalizers, plus LLM access expectations for chunk, extract, and normalize.
• Update docs/internal/pipeline.md with target-aware reference resolution, materialization, provenance, and runtime delivery.
• Update docs/integrations/json-output.md with the new reference provenance shape including stage and optional lane_id.
• Update docs/troubleshooting.md with ambiguous selector examples for chunk, extract, and normalize targets.
• Update maintained examples only if they can remain small and validated. Do not add speculative reference files for chunk or normalize modules without a production module that actually declares those slots.
• Update docs/roadmap/future.md by removing or revising the two completed items while leaving still-deferred work such as token budgeting and non-file producers.
• Replace this implementation plan with a concise completed-note document after implementation is finished.

Verification:

• rg -n "chunk.references|normalize.references|lane.extract.slot|lane.normalize.slot" docs/cli.md docs/config.md docs/internal docs/integrations docs/troubleshooting.md
• go test ./...
• go vet ./...
• go build ./cmd/notarius

Final Acceptance Criteria

The feature is complete when:

• chunkers, extractors, and normalizers can declare reference slots through runtime contracts and registry specs;
• normalizers receive the structured LLM client at runtime;
• config can bind references at pipeline, chunk, extract, and normalize levels;
• CLI reference overrides and unbinds can target chunk, extract, and normalize references without ambiguous selector behavior;
• pipeline resolution validates all reference bindings before any LLM call;
• materialization and provenance work uniformly for chunk, extract, and normalize references;
• reference manifest entries identify stage and lane context;
• existing extractor reference behavior remains backward compatible;
• canonical docs describe only implemented behavior;
• go test ./..., go vet ./..., and go build ./cmd/notarius pass.