306 lines
13 KiB
Markdown
306 lines
13 KiB
Markdown
# Extraction References Implementation Plan
|
|
|
|
## Purpose
|
|
|
|
Implement the extraction-reference feature described in
|
|
[references.md](references.md). This plan is decision-complete for an LLM coding
|
|
agent: implement each stage in order, keep the repository compiling after each
|
|
stage, and do not move planned behavior into non-roadmap docs until the relevant
|
|
behavior exists.
|
|
|
|
Core decisions to preserve:
|
|
|
|
- references are opaque framework inputs and domain semantics stay in extract
|
|
modules;
|
|
- references are not evidence and must not be addressable through `SourceRef`;
|
|
- reference binding is lane-scoped;
|
|
- extractors expose `ReferenceSlots()` directly on the first-class extractor
|
|
contract;
|
|
- token budgeting is deferred; enforce only UTF-8 text handling, empty-file
|
|
warnings, and declared `MaxBytes`;
|
|
- run manifests record references in a dedicated section, separate from
|
|
`source_digests`;
|
|
- CLI unbinding uses `--without-reference`.
|
|
|
|
## Stage 1: Contracts and Mechanical Adoption
|
|
|
|
Add the framework contracts needed to describe references without changing
|
|
runtime behavior. Slot declarations must be available without constructing
|
|
extractor modules, because pipeline/config validation should use registry
|
|
metadata rather than runtime module instances.
|
|
|
|
Implementation steps:
|
|
|
|
- In `internal/framework/contracts`, add reference model types:
|
|
`ReferenceSlot`, `ReferenceOrigin`, `ReferenceItem`,
|
|
`ResolvedReferenceSlot`, `ReferenceSet`, and a binding-source enum or string
|
|
constants for `config` and `cli`.
|
|
- Include `Name`, `Description`, `Required`, `AcceptedMediaTypes`, `Multiple`,
|
|
and `MaxBytes` on `ReferenceSlot`.
|
|
- Include slot name, media type, content bytes, digest, origin, size bytes, and
|
|
binding source on `ReferenceItem`.
|
|
- Add `References ReferenceSet` to `contracts.ExtractionRequest`.
|
|
- Add `ReferenceSlots() []ReferenceSlot` to `contracts.Extractor`.
|
|
- Extend extractor registration metadata so reference slots are also declared
|
|
through the extractor's registry spec. Prefer the smallest idiomatic change to
|
|
the existing registry model, such as adding `ReferenceSlots` to `ModuleSpec`
|
|
with validation that non-extractor modules leave it empty, unless the codebase
|
|
shape clearly supports a narrower extractor-specific spec.
|
|
- Update every concrete extractor and all extractor fakes/test doubles to
|
|
implement `ReferenceSlots()`. Existing extractors without references should
|
|
return `nil`.
|
|
- Add tests that compare a production extractor's runtime `ReferenceSlots()`
|
|
with its registered spec slots so the two declarations cannot drift.
|
|
- Add contract tests for empty reference sets, slot copying expectations if
|
|
helpers are introduced, and compile-time coverage through existing fakes.
|
|
|
|
Verification:
|
|
|
|
- `go test ./internal/framework/contracts`
|
|
- `go test ./internal/framework/pipeline`
|
|
- `go test ./...`
|
|
|
|
## Stage 2: Config Shape and Pipeline-Level Resolution
|
|
|
|
Add unresolved reference bindings to config and resolved lane bindings to the
|
|
pipeline model. Do not read reference files in this stage.
|
|
|
|
Implementation steps:
|
|
|
|
- Add `references` maps to file config parsing at both pipeline and artifact
|
|
lane level.
|
|
- Add corresponding fields to `pipeline.PipelineProfile` and
|
|
`pipeline.ArtifactLaneProfile`.
|
|
- Preserve deterministic map handling and duplicate-after-trim validation.
|
|
- Extend config cloning, effective config, redaction, validation, and tests for
|
|
the new fields.
|
|
- Add resolved reference binding structures to `internal/framework/pipeline`.
|
|
They should represent lane ID, slot name, source URI/path, and binding source,
|
|
but not file bytes.
|
|
- During `pipeline.ResolvePipeline`, collect selected lanes, read each lane
|
|
extractor's declared slots from registry metadata, and validate without
|
|
building extractor instances:
|
|
- every bound slot is declared by the lane extractor;
|
|
- required slots are bound after applying pipeline-level and lane-level config;
|
|
- required slots remain bound after any CLI unbinds supplied to resolution;
|
|
- selected lanes under `--only` are the only lanes considered.
|
|
- Apply pipeline-level bindings as defaults only to lanes whose extractor
|
|
declares the matching slot.
|
|
- Apply lane-level bindings as overrides or additions for that lane.
|
|
- Keep reference bindings out of source digests and artifact source references.
|
|
- Add tests proving reference-slot validation works through registry specs even
|
|
when extractor constructors would fail if called.
|
|
|
|
Verification:
|
|
|
|
- `go test ./internal/core/config`
|
|
- `go test ./internal/framework/pipeline`
|
|
- `go test ./...`
|
|
|
|
## Stage 3: CLI Reference Overrides and Unbinds
|
|
|
|
Add run-time CLI syntax for reference binding overrides and optional unbinding.
|
|
|
|
Implementation steps:
|
|
|
|
- Add repeatable `--reference` flags to `notarius run`.
|
|
Accepted forms:
|
|
- `slot=path` for unambiguous slot names across selected lanes;
|
|
- `lane.slot=path` for explicit lane-scoped binding.
|
|
- Add repeatable `--without-reference` flags to `notarius run`.
|
|
Accepted forms:
|
|
- `slot`;
|
|
- `lane.slot`.
|
|
- Reject empty paths for `--reference`; use `--without-reference` for unbinding.
|
|
- Reject malformed values with concise CLI errors before expensive work.
|
|
- Pass parsed override/unbind requests into config/pipeline resolution.
|
|
- Resolve flat CLI names only when exactly one selected lane declares the slot.
|
|
If multiple selected lanes declare the same slot, fail and instruct the user
|
|
to use `lane.slot`.
|
|
- Let CLI bindings override config bindings for the same lane and slot.
|
|
- Let CLI unbinds remove config-bound optional slots for the same lane and slot.
|
|
- Fail if unbinding leaves a required slot unbound.
|
|
- Add CLI tests for flat binding, lane-qualified binding, ambiguous flat
|
|
binding, malformed syntax, optional unbind, and required-slot unbind failure.
|
|
|
|
Verification:
|
|
|
|
- `go test ./internal/cli`
|
|
- `go test ./internal/core/config`
|
|
- `go test ./internal/framework/pipeline`
|
|
- `go test ./...`
|
|
|
|
## Stage 4: Run Preparation and Reference Materialization
|
|
|
|
Read, validate, digest, and materialize resolved file references before any LLM
|
|
call.
|
|
|
|
Implementation steps:
|
|
|
|
- Add a reference resolver/materializer near pipeline run preparation. Keep file
|
|
I/O out of pure config parsing.
|
|
- Ensure run preparation receives the loaded config path or config directory so
|
|
config-relative reference paths can be resolved after pure config parsing.
|
|
- Resolve config-relative paths relative to the config file path and
|
|
CLI-relative paths relative to the current working directory.
|
|
- For MVP, accept only UTF-8 text files. Reject non-UTF-8 content with an error
|
|
naming pipeline, lane, slot, and path.
|
|
- Compute `sha256:` content digests over the raw reference bytes.
|
|
- Populate `ReferenceItem` values with content bytes, media type, digest,
|
|
origin type `file`, normalized origin URI/path, size bytes, and binding source.
|
|
- Enforce declared `MaxBytes` when greater than zero. The error should name the
|
|
pipeline, lane, slot, actual size, limit, and path.
|
|
- Emit a warning for empty bound files, but do not fail.
|
|
- Add `ReferenceSet` values to the runner input or resolved pipeline path in a
|
|
way that keeps lane-scoped references available when calling each extractor.
|
|
- Pass the correct lane-specific `ReferenceSet` into
|
|
`contracts.ExtractionRequest`.
|
|
- Ensure no reference content is written to ordinary diagnostics, logs, errors,
|
|
or manifests.
|
|
|
|
Verification:
|
|
|
|
- Focused resolver/materializer tests for path resolution, digest stability,
|
|
UTF-8 rejection, empty-file warning, `MaxBytes`, and binding source.
|
|
- `go test ./internal/cli`
|
|
- `go test ./internal/framework/pipeline`
|
|
- `go test ./...`
|
|
|
|
## Stage 5: Prompt Template Reference Functions
|
|
|
|
Make references available to module-owned prompt templates.
|
|
|
|
Implementation steps:
|
|
|
|
- Extend `internal/framework/prompt` so prompt bundles can be compiled with
|
|
declared reference slots.
|
|
- Add `reference` and `hasreference` template functions.
|
|
- Validate at bundle build time, or the earliest feasible equivalent, that
|
|
templates reference only declared slots.
|
|
- Render a declared but unbound optional slot as an empty string.
|
|
- Ensure `hasreference` returns true only when the slot has at least one bound
|
|
item with content.
|
|
- Render multiple items deterministically if future `Multiple` support is
|
|
enabled; for MVP, reject multiple bindings unless the slot declares
|
|
`Multiple`.
|
|
- Keep prompt metadata hashes based on template source. Do not include rendered
|
|
reference content in prompt identity.
|
|
- Add deterministic rendering tests proving byte-identical output across runs
|
|
with the same reference bytes and config.
|
|
|
|
Verification:
|
|
|
|
- `go test ./internal/framework/prompt`
|
|
- `go test ./internal/modules/extract/dnd/spells`
|
|
- `go test ./...`
|
|
|
|
## Stage 6: Manifest and Diagnostics Provenance
|
|
|
|
Record reference provenance separately from source provenance.
|
|
|
|
Implementation steps:
|
|
|
|
- Add a dedicated references section to `artifacts.RunManifest`.
|
|
The shape should be lane-scoped and include lane ID, slot name, origin type,
|
|
origin URI/path, digest, media type, size bytes, and binding source.
|
|
- Do not add reference digests to `source_digests`.
|
|
- Include reference digests in any cache/idempotency key if such a key exists.
|
|
If no cache/idempotency key exists, add a test or comment documenting that no
|
|
additional key needs updating yet.
|
|
- Write a diagnostics artifact for resolved references that contains provenance
|
|
only, not full content, consistent with redacted effective config behavior.
|
|
- Ensure durable JSON output manifests include the new manifest section.
|
|
- Add manifest round-trip tests and a CLI/run test where two runs that differ
|
|
only in reference bytes produce distinguishable manifests.
|
|
|
|
Verification:
|
|
|
|
- `go test ./internal/core/artifacts`
|
|
- `go test ./internal/core/diagnostics`
|
|
- `go test ./internal/modules/output/json`
|
|
- `go test ./internal/cli`
|
|
- `go test ./...`
|
|
|
|
## Stage 7: D&D Spells Consumer
|
|
|
|
Use the new reference feature in the first production extractor.
|
|
|
|
Implementation steps:
|
|
|
|
- Declare optional `roster` and `glossary` slots on `dnd/spells`.
|
|
- Set accepted media type to text/UTF-8. Add conservative `MaxBytes` limits only
|
|
if a clear module-owned limit is chosen; otherwise leave `MaxBytes` unset.
|
|
- Update the D&D spells prompt bundle to include conditional reference sections
|
|
using `hasreference` and `reference`.
|
|
- Frame references as supporting material only. The prompt must instruct the
|
|
model to extract only spell-cast events present in the source transcript and
|
|
use references only for disambiguation.
|
|
- Update prompt metadata tests as needed while preserving template-hash
|
|
semantics.
|
|
- Add fixture coverage with no references, with roster/glossary references, and
|
|
with a roster that mentions a spell never cast in the transcript. The last
|
|
case must assert no spell-cast artifact is produced for the uncast spell.
|
|
- If existing deterministic source-reference validation can be extended
|
|
cleanly, add warning-level relatedness checks for spell names or close
|
|
variants near cited source text. If this becomes large, defer that validator
|
|
enhancement to a separate roadmap item and keep the prompt/regression fixture
|
|
guard in this stage.
|
|
|
|
Verification:
|
|
|
|
- `go test ./internal/modules/extract/dnd/spells`
|
|
- `go test ./internal/framework/pipeline`
|
|
- `go test ./internal/cli`
|
|
- `go test ./...`
|
|
|
|
## Stage 8: Canonical Documentation and Examples
|
|
|
|
Move implemented behavior out of roadmap-only status once code exists.
|
|
|
|
Implementation steps:
|
|
|
|
- Update `docs/cli.md` with `--reference` and `--without-reference` syntax,
|
|
precedence, ambiguity behavior, and examples.
|
|
- Update `docs/config.md` with pipeline-level and lane-level `references`
|
|
blocks.
|
|
- Update `docs/internal/modules.md` or the most appropriate internal docs with
|
|
module-author guidance for `ReferenceSlots()`, reference request delivery,
|
|
prompt functions, evidence exclusion, and provenance.
|
|
- Update `docs/internal/pipeline.md` with reference resolution lifecycle and
|
|
lane-scoped delivery.
|
|
- Update `docs/integrations/json-output.md` with the manifest reference
|
|
provenance shape.
|
|
- Update `docs/operations.md` or `docs/troubleshooting.md` for common reference
|
|
errors such as unknown slot, ambiguous flat override, missing required slot,
|
|
unreadable file, non-UTF-8 content, and `MaxBytes` failures.
|
|
- Add maintained example reference files and update `examples/dnd-spells.config.yml`
|
|
only after the CLI/config behavior is implemented and covered by tests.
|
|
- Keep future-only material in `docs/roadmap/references.md`; do not duplicate
|
|
canonical current behavior there after implementation.
|
|
|
|
Verification:
|
|
|
|
- `rg -n "references:|--reference|--without-reference|ReferenceSlots|reference \"|hasreference" docs examples`
|
|
- `go test ./...`
|
|
- `go vet ./...`
|
|
- `go build ./cmd/notarius`
|
|
|
|
## Final Acceptance Criteria
|
|
|
|
The feature is complete when:
|
|
|
|
- extractor modules can declare reference slots through the first-class
|
|
extractor contract;
|
|
- config and CLI can bind and unbind lane-scoped file references;
|
|
- selected-pipeline validation catches unknown, ambiguous, or missing required
|
|
references before any LLM call;
|
|
- run preparation materializes UTF-8 text references with digests, size checks,
|
|
and empty-file warnings;
|
|
- extractors receive lane-scoped resolved references;
|
|
- prompt templates can render `reference` and `hasreference` deterministically;
|
|
- D&D spell extraction uses optional roster and glossary references;
|
|
- run manifests and diagnostics record reference provenance without recording
|
|
full content or treating references as source evidence;
|
|
- canonical docs and maintained examples describe only implemented behavior;
|
|
- `go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` pass.
|