notarius/docs/roadmap/implementation.md

Extraction References Implementation Plan

Purpose

Implement the extraction-reference feature described in references.md. This plan is decision-complete for an LLM coding agent: implement each stage in order, keep the repository compiling after each stage, and do not move planned behavior into non-roadmap docs until the relevant behavior exists.

Core decisions to preserve:

• references are opaque framework inputs and domain semantics stay in extract modules;
• references are not evidence and must not be addressable through SourceRef;
• reference binding is lane-scoped;
• extractors expose ReferenceSlots() directly on the first-class extractor contract;
• token budgeting is deferred; enforce only UTF-8 text handling, empty-file warnings, and declared MaxBytes;
• run manifests record references in a dedicated section, separate from source_digests;
• CLI unbinding uses --without-reference.

Stage 1: Contracts and Mechanical Adoption

Add the framework contracts needed to describe references without changing runtime behavior. Slot declarations must be available without constructing extractor modules, because pipeline/config validation should use registry metadata rather than runtime module instances.

Implementation steps:

• In internal/framework/contracts, add reference model types: ReferenceSlot, ReferenceOrigin, ReferenceItem, ResolvedReferenceSlot, ReferenceSet, and a binding-source enum or string constants for config and cli.
• Include Name, Description, Required, AcceptedMediaTypes, Multiple, and MaxBytes on ReferenceSlot.
• Include slot name, media type, content bytes, digest, origin, size bytes, and binding source on ReferenceItem.
• Add References ReferenceSet to contracts.ExtractionRequest.
• Add ReferenceSlots() []ReferenceSlot to contracts.Extractor.
• Extend extractor registration metadata so reference slots are also declared through the extractor's registry spec. Prefer the smallest idiomatic change to the existing registry model, such as adding ReferenceSlots to ModuleSpec with validation that non-extractor modules leave it empty, unless the codebase shape clearly supports a narrower extractor-specific spec.
• Update every concrete extractor and all extractor fakes/test doubles to implement ReferenceSlots(). Existing extractors without references should return nil.
• Add tests that compare a production extractor's runtime ReferenceSlots() with its registered spec slots so the two declarations cannot drift.
• Add contract tests for empty reference sets, slot copying expectations if helpers are introduced, and compile-time coverage through existing fakes.

Verification:

• go test ./internal/framework/contracts
• go test ./internal/framework/pipeline
• go test ./...

Stage 2: Config Shape and Pipeline-Level Resolution

Add unresolved reference bindings to config and resolved lane bindings to the pipeline model. Do not read reference files in this stage.

Implementation steps:

• Add references maps to file config parsing at both pipeline and artifact lane level.
• Add corresponding fields to pipeline.PipelineProfile and pipeline.ArtifactLaneProfile.
• Preserve deterministic map handling and duplicate-after-trim validation.
• Extend config cloning, effective config, redaction, validation, and tests for the new fields.
• Add resolved reference binding structures to internal/framework/pipeline. They should represent lane ID, slot name, source URI/path, and binding source, but not file bytes.
• During pipeline.ResolvePipeline, collect selected lanes, read each lane extractor's declared slots from registry metadata, and validate without building extractor instances:
  • every bound slot is declared by the lane extractor;
  • required slots are bound after applying pipeline-level and lane-level config;
  • required slots remain bound after any CLI unbinds supplied to resolution;
  • selected lanes under --only are the only lanes considered.
• Apply pipeline-level bindings as defaults only to lanes whose extractor declares the matching slot.
• Apply lane-level bindings as overrides or additions for that lane.
• Keep reference bindings out of source digests and artifact source references.
• Add tests proving reference-slot validation works through registry specs even when extractor constructors would fail if called.

Verification:

• go test ./internal/core/config
• go test ./internal/framework/pipeline
• go test ./...

Stage 3: CLI Reference Overrides and Unbinds

Add run-time CLI syntax for reference binding overrides and optional unbinding.

Implementation steps:

• Add repeatable --reference flags to notarius run. Accepted forms:
  • slot=path for unambiguous slot names across selected lanes;
  • lane.slot=path for explicit lane-scoped binding.
• Add repeatable --without-reference flags to notarius run. Accepted forms:
  • slot;
  • lane.slot.
• Reject empty paths for --reference; use --without-reference for unbinding.
• Reject malformed values with concise CLI errors before expensive work.
• Pass parsed override/unbind requests into config/pipeline resolution.
• Resolve flat CLI names only when exactly one selected lane declares the slot. If multiple selected lanes declare the same slot, fail and instruct the user to use lane.slot.
• Let CLI bindings override config bindings for the same lane and slot.
• Let CLI unbinds remove config-bound optional slots for the same lane and slot.
• Fail if unbinding leaves a required slot unbound.
• Add CLI tests for flat binding, lane-qualified binding, ambiguous flat binding, malformed syntax, optional unbind, and required-slot unbind failure.

Verification:

• go test ./internal/cli
• go test ./internal/core/config
• go test ./internal/framework/pipeline
• go test ./...

Stage 4: Run Preparation and Reference Materialization

Read, validate, digest, and materialize resolved file references before any LLM call.

Implementation steps:

• Add a reference resolver/materializer near pipeline run preparation. Keep file I/O out of pure config parsing.
• Ensure run preparation receives the loaded config path or config directory so config-relative reference paths can be resolved after pure config parsing.
• Resolve config-relative paths relative to the config file path and CLI-relative paths relative to the current working directory.
• For MVP, accept only UTF-8 text files. Reject non-UTF-8 content with an error naming pipeline, lane, slot, and path.
• Compute sha256: content digests over the raw reference bytes.
• Populate ReferenceItem values with content bytes, media type, digest, origin type file, normalized origin URI/path, size bytes, and binding source.
• Enforce declared MaxBytes when greater than zero. The error should name the pipeline, lane, slot, actual size, limit, and path.
• Emit a warning for empty bound files, but do not fail.
• Add ReferenceSet values to the runner input or resolved pipeline path in a way that keeps lane-scoped references available when calling each extractor.
• Pass the correct lane-specific ReferenceSet into contracts.ExtractionRequest.
• Ensure no reference content is written to ordinary diagnostics, logs, errors, or manifests.

Verification:

• Focused resolver/materializer tests for path resolution, digest stability, UTF-8 rejection, empty-file warning, MaxBytes, and binding source.
• go test ./internal/cli
• go test ./internal/framework/pipeline
• go test ./...

Stage 5: Prompt Template Reference Functions

Make references available to module-owned prompt templates.

Implementation steps:

• Extend internal/framework/prompt so prompt bundles can be compiled with declared reference slots.
• Add reference and hasreference template functions.
• Validate at bundle build time, or the earliest feasible equivalent, that templates reference only declared slots.
• Render a declared but unbound optional slot as an empty string.
• Ensure hasreference returns true only when the slot has at least one bound item with content.
• Render multiple items deterministically if future Multiple support is enabled; for MVP, reject multiple bindings unless the slot declares Multiple.
• Keep prompt metadata hashes based on template source. Do not include rendered reference content in prompt identity.
• Add deterministic rendering tests proving byte-identical output across runs with the same reference bytes and config.

Verification:

• go test ./internal/framework/prompt
• go test ./internal/modules/extract/dnd/spells
• go test ./...

Stage 6: Manifest and Diagnostics Provenance

Record reference provenance separately from source provenance.

Implementation steps:

• Add a dedicated references section to artifacts.RunManifest. The shape should be lane-scoped and include lane ID, slot name, origin type, origin URI/path, digest, media type, size bytes, and binding source.
• Do not add reference digests to source_digests.
• Include reference digests in any cache/idempotency key if such a key exists. If no cache/idempotency key exists, add a test or comment documenting that no additional key needs updating yet.
• Write a diagnostics artifact for resolved references that contains provenance only, not full content, consistent with redacted effective config behavior.
• Ensure durable JSON output manifests include the new manifest section.
• Add manifest round-trip tests and a CLI/run test where two runs that differ only in reference bytes produce distinguishable manifests.

Verification:

• go test ./internal/core/artifacts
• go test ./internal/core/diagnostics
• go test ./internal/modules/output/json
• go test ./internal/cli
• go test ./...

Stage 7: D&D Spells Consumer

Use the new reference feature in the first production extractor.

Implementation steps:

• Declare optional roster and glossary slots on dnd/spells.
• Set accepted media type to text/UTF-8. Add conservative MaxBytes limits only if a clear module-owned limit is chosen; otherwise leave MaxBytes unset.
• Update the D&D spells prompt bundle to include conditional reference sections using hasreference and reference.
• Frame references as supporting material only. The prompt must instruct the model to extract only spell-cast events present in the source transcript and use references only for disambiguation.
• Update prompt metadata tests as needed while preserving template-hash semantics.
• Add fixture coverage with no references, with roster/glossary references, and with a roster that mentions a spell never cast in the transcript. The last case must assert no spell-cast artifact is produced for the uncast spell.
• If existing deterministic source-reference validation can be extended cleanly, add warning-level relatedness checks for spell names or close variants near cited source text. If this becomes large, defer that validator enhancement to a separate roadmap item and keep the prompt/regression fixture guard in this stage.

Verification:

• go test ./internal/modules/extract/dnd/spells
• go test ./internal/framework/pipeline
• go test ./internal/cli
• go test ./...

Stage 8: Canonical Documentation and Examples

Move implemented behavior out of roadmap-only status once code exists.

Implementation steps:

• Update docs/cli.md with --reference and --without-reference syntax, precedence, ambiguity behavior, and examples.
• Update docs/config.md with pipeline-level and lane-level references blocks.
• Update docs/internal/modules.md or the most appropriate internal docs with module-author guidance for ReferenceSlots(), reference request delivery, prompt functions, evidence exclusion, and provenance.
• Update docs/internal/pipeline.md with reference resolution lifecycle and lane-scoped delivery.
• Update docs/integrations/json-output.md with the manifest reference provenance shape.
• Update docs/operations.md or docs/troubleshooting.md for common reference errors such as unknown slot, ambiguous flat override, missing required slot, unreadable file, non-UTF-8 content, and MaxBytes failures.
• Add maintained example reference files and update examples/dnd-spells.config.yml only after the CLI/config behavior is implemented and covered by tests.
• Keep future-only material in docs/roadmap/references.md; do not duplicate canonical current behavior there after implementation.

Verification:

• rg -n "references:|--reference|--without-reference|ReferenceSlots|reference \"|hasreference" docs examples
• go test ./...
• go vet ./...
• go build ./cmd/notarius

Final Acceptance Criteria

The feature is complete when:

• extractor modules can declare reference slots through the first-class extractor contract;
• config and CLI can bind and unbind lane-scoped file references;
• selected-pipeline validation catches unknown, ambiguous, or missing required references before any LLM call;
• run preparation materializes UTF-8 text references with digests, size checks, and empty-file warnings;
• extractors receive lane-scoped resolved references;
• prompt templates can render reference and hasreference deterministically;
• D&D spell extraction uses optional roster and glossary references;
• run manifests and diagnostics record reference provenance without recording full content or treating references as source evidence;
• canonical docs and maintained examples describe only implemented behavior;
• go test ./..., go vet ./..., and go build ./cmd/notarius pass.