eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add planning roadmaps for large-scale refactors of the pipleine and the validator registry, and add a staged imnplementation plan for the pipeline refactor

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-07 13:20:10 -05:00
parent aec807fcb0
commit 4f057b99ac
4 changed files with 1238 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
docs/roadmap/dnd.md
View File

@@ -1,22 +0,0 @@
# D&D Shared Module Helper Refactor
The D&D shared module helper refactor is complete.
The implemented package split is:
- `internal/modules/sharedassets`: generic shared prompt filesystem composition
and domain-neutral shared asset plumbing.
- `internal/modules/sharedassets/dnd`: D&D shared prompt fragments, accepted
reference media types, `players`/`party`/`glossary` reference slot helpers,
deprecated `roster` alias handling, prompt input assembly, deterministic
reference rendering, and D&D shared prompt hash parts.
- Concrete D&D modules: stage contracts and registration, module keys,
provided/required capabilities, prompt IDs and versions, module-local prompt
definitions, response schemas, validators, artifact or scene interpretation,
and manifest metadata.
The `dnd/scenes` chunker and `dnd/spells` extractor now use the shared D&D
helper package for common prompt and reference behavior while keeping
stage-specific semantics in their concrete module packages. D&D shared prompt
fragments are exposed to Scriptorium through each module's local
`./sharedassets/` prompt paths.

430
docs/roadmap/implementation.md
View File

@@ -1,19 +1,419 @@
# D&D Shared Helper Refactor Completed
# Raw Pipeline Data Model Implementation Plan
The D&D shared helper refactor is implemented.
This plan implements the target state in
[`pipeline.md`](pipeline.md). It is intentionally staged for multiple coding
passes. Do not skip stages: later work assumes the contracts and tests from
earlier stages are already in place.
Implemented package ownership:
The desired end state is a fixed-shape pipeline:
- `internal/modules/sharedassets`: generic prompt filesystem composition and
domain-neutral shared asset plumbing.
- `internal/modules/sharedassets/dnd`: reusable D&D prompt fragments, prompt
filesystem composition for D&D modules, D&D reference slot helpers, prompt
input assembly, reference rendering, and D&D shared prompt hash parts.
- Concrete D&D modules: stage contracts, module registration, module-local
prompt definitions, response schemas, validators, request validation,
response conversion, and manifest metadata ownership.
```text
input -> chunk -> extract -> merge -> normalize -> output
```
The refactor preserved module keys, prompt IDs, prompt versions,
Scriptorium-visible message order, response schema IDs and names, CLI/config
semantics, manifest shape, diagnostics redaction policy, and D&D scene/spell
interpretation behavior.
`chunk` produces typed chunk envelopes. `extract`, `merge`, and `normalize`
produce raw byte payload envelopes with media type and provenance. Rejected
outputs do not pass to the next stage.
## Stage 1: Integer Source Units And Chunk Payloads
Update the core source and chunk contracts before changing downstream stages.
Implementation tasks:
- Change `internal/core/source.SourceUnit.ID` from `string` to `int`.
- Change `internal/core/source.SourceRef.StartUnitID` and `EndUnitID` from
`string` to `int`.
- Treat source-unit IDs as positive, stable, input-adapter-owned integers.
`0` and negative IDs should be invalid.
- Update source validation helpers, including unit lookup and source-ref
ordering checks, to use integer IDs.
- Update `contracts.SourceChunk` to include:
- `StartUnitID int`;
- `EndUnitID int`;
- `Content []byte`;
- `MediaType string`.
- Keep `SourceChunk.Units []source.SourceUnit` for framework scheduling,
diagnostics, and modules that need unit metadata.
- Update chunk canonicalization in `internal/framework/pipeline` to enforce
minimal scheduling invariants:
- at least one chunk;
- non-empty chunk ID;
- source ID matches the source document;
- indexes are deterministic and contiguous from zero;
- start/end unit IDs exist and are ordered;
- `Units` are canonical copies from the source document;
- content is non-empty;
- media type is non-empty.
- Update the Seriatim input adapter to parse segment IDs as integers.
Accept JSON numbers and numeric strings only when they are positive integers.
Reject non-numeric IDs such as `seg-001`.
- Update Seriatim examples, fixtures, tests, and integration docs to use
integer segment IDs.
- Update D&D source-reference helper code under
`internal/modules/sharedassets/dnd` so source refs remain integer-valued all
the way into `source.SourceRef`.
- Remove fallback behavior that converts integer LLM unit refs back into string
source-unit IDs.
- Update the generic chunker and D&D scenes chunker to populate
`StartUnitID`, `EndUnitID`, `Content`, and `MediaType`.
- For current transcript chunkers, use `application/json` content containing a
canonical JSON encoding of the chunk's source units. Keep the original
`Units` field populated as framework-owned typed metadata.
Tests to update or add:
- `go test ./internal/core/source`
- `go test ./internal/modules/input/seriatim`
- `go test ./internal/modules/chunk/generic`
- `go test ./internal/modules/chunk/dnd/scenes`
- `go test ./internal/framework/contracts`
- `go test ./internal/framework/pipeline`
Completion criteria:
- All source refs and source-unit IDs in core/framework types are integers.
- Chunkers return extraction-ready content bytes plus media type.
- Framework chunk validation still allows partial coverage and overlap unless a
validator rejects them.
## Stage 2: Raw Module Output Contracts
Replace artifact-candidate stage contracts with raw payload envelopes.
Implementation tasks:
- Add contract types in `internal/framework/contracts` for raw stage payloads.
The exact names may differ, but the contracts must preserve:
- raw bytes;
- media type;
- lane ID;
- stage module key;
- source ID;
- chunk ID and chunk index where applicable;
- response schema ID, name, and version when applicable;
- metadata;
- warnings.
- Recommended shape:
- `ExtractOutput`
- `MergeOutput`
- `NormalizeOutput`
- a small shared payload/provenance helper if it reduces duplication.
- Update `Extractor` so `Extract` returns one raw `ExtractOutput` for one input
chunk instead of `[]artifacts.ArtifactCandidate`.
- Remove `Extractor.ArtifactType`, `Extractor.SchemaVersion`, and
`Extractor.Validators` from the generic extractor contract. Schema and prompt
provenance should come from the returned output and manifest metadata, not
artifact-specific methods.
- Update `Merger` so `Merge` receives ordered accepted `[]ExtractOutput` and
returns one raw `MergeOutput`.
- Update `Normalizer` so `Normalize` receives one accepted `MergeOutput` and
returns one raw `NormalizeOutput`.
- Update `OutputRequest` so output encoders receive the validated normalized
lane outputs rather than approved `Artifact` values.
- Add a raw rejected-output record type for manifests/output files. It should
identify stage, lane, module, chunk when relevant, validator or error reason,
message, attempt count, and diagnostic artifact path if present. It must not
include large raw content by default.
- Leave old artifact/candidate types in `internal/core/artifacts` only if
temporary compatibility code still needs them during the migration. They
should not remain in the final stage contracts.
Tests to update or add:
- Contract composition tests proving fake chunk, extract, merge, normalize, and
output modules compose using raw envelopes.
- Defensive-copy tests for raw content and metadata maps.
- Registry integration tests proving the new interfaces build and run.
Completion criteria:
- Framework contracts no longer require extractors, mergers, normalizers, or
output encoders to use `ArtifactCandidate` or `Artifact`.
- Stage outputs carry raw bytes and media type.
## Stage 3: Config, References, Profiles, And Retries
Make `merge` a first-class LLM/profile/reference stage and add retry policy.
Implementation tasks:
- Update `pipeline.referenceSlotStage` so `merge` modules may declare reference
slots.
- Add `MergeReferences` to `ResolvedArtifactLane`.
- Resolve and materialize merge references using the same declared-slot model
used by `chunk`, `extract`, and `normalize`.
- Update reference provenance so merge-stage references appear in manifests and
resolved reference diagnostics.
- Update config parsing, config validation, redaction, cloning, and examples so
`artifacts.<lane>.merge.references` is valid.
- Update CLI `--reference` selector parsing and ambiguity checks to support:
- `merge.<slot>=<path>` where unambiguous;
- `<lane>.merge.<slot>=<path>`;
- lane-qualified shorthand only when it resolves unambiguously.
- Update `--without-reference` behavior to support merge-stage slots.
- Update `ModuleStage` LLM-capable helpers so `chunk`, `extract`, `merge`, and
`normalize` are all considered LLM-capable.
- Update `--llm-profile` override handling so it applies to `chunk`, every lane
`extract`, every lane `merge`, and every lane `normalize`.
- Update explicit Scriptorium profile validation so it inspects only those four
LLM-capable stages.
- Add `retries` to `pipeline.ModuleBinding` as a non-negative integer count of
extra attempts after the first attempt. Default is `0`.
- Apply `retries` only to `chunk`, `extract`, `merge`, and `normalize` runtime
execution. Keep input and output retry behavior out of scope.
- Validate `retries >= 0` in config validation.
- Ensure redacted config preserves retry counts.
Tests to update or add:
- Config accepts merge references and rejects unknown/undeclared merge slots.
- CLI reference selectors work for merge and remain strict for ambiguous slots.
- `--llm-profile` overrides merge bindings as well as chunk/extract/normalize.
- Explicit profile validation includes merge and ignores non-LLM stages.
- Negative retries are rejected.
Completion criteria:
- Merge has the same runtime plumbing class as chunk/extract/normalize.
- Retry policy is representable in pipeline config and resolved bindings.
## Stage 4: Runner Orchestration And Retry Semantics
Rewrite pipeline execution around raw envelopes.
Implementation tasks:
- Introduce a small retry helper in `internal/framework/pipeline` that reruns
the same module with the same input when:
- the module returns a framework-level error;
- the module returns output that is rejected by that stage's validator chain.
- Preserve context cancellation: never retry after `ctx.Err()` is non-nil.
- Record attempt count and compact attempt diagnostics for manifests/output.
- Do not log or manifest raw payload bytes by default.
- Run chunk once per source input, with retries from the chunk binding.
- After successful chunk execution, run chunk validation when validator mapping
support exists. A rejected chunk result after final retry should stop
downstream execution and produce a rejected run record.
- Run extract once per accepted chunk, with per-chunk retries from the extract
binding.
- Omit rejected extract outputs from merge input.
- If a lane has no accepted extract outputs after retries, skip merge and
normalize for that lane, record rejected output details, and continue to the
next lane.
- Run merge once per lane with accepted extract outputs, with retries from the
merge binding.
- If merge output is rejected after final retry, omit that lane from normalize
and output.
- Run normalize once per accepted merge output, with retries from the normalize
binding.
- If normalize output is rejected after final retry, omit that lane from output.
- Treat validator rejection as non-fatal run outcome. Treat unrecovered
framework-level execution errors as run failures.
- Preserve deterministic ordering:
- chunks by chunk index;
- extract outputs by chunk index;
- lane outputs by resolved lane order.
- Continue to collect warnings from every successful module attempt whose output
is used. For failed attempts, record compact attempt diagnostics rather than
promoting warnings as final-stage warnings unless the implementation already
has a clear warning policy.
- Set manifest validation status to:
- `approved` when all produced normalized lane outputs pass;
- `rejected` when one or more module outputs are rejected;
- failed-run status through existing failure handling for unrecovered runtime
errors.
Validation seam scope:
- Add the runner-side raw validation seam needed by this data-model refactor.
- Represent the validation boundary as raw module output plus stage provenance,
consistent with [`validation.md`](validation.md).
- Support empty validator chains as approval.
- Add fake validator/test-only coverage for approval, rejection, and
retry-on-rejection behavior.
- Do not implement the full production validator package migration, default
validator mappings, or concrete generic validators in this plan. Those remain
owned by the validation roadmap.
Tests to update or add:
- Runner passes chunk content and media type to extractors.
- Runner passes ordered accepted extract outputs to merge.
- Runner passes accepted merge output to normalize.
- Rejected extract output is omitted from merge input.
- A lane with no accepted extracts is omitted and recorded.
- Rejected merge output prevents normalize for that lane.
- Rejected normalize output prevents output for that lane.
- Retry reruns the same module input after framework error.
- Retry reruns the same module input after validator rejection.
- Retry stops after configured attempts and records attempt count.
- Context cancellation stops retries.
Completion criteria:
- The runner no longer depends on candidate materialization between extract,
merge, normalize, and output.
- Rejected outputs never pass to the next stage.
## Stage 5: Production Module Migration
Update concrete modules to the new contracts.
Implementation tasks:
- Update `internal/modules/extract/dnd/spells`:
- call Scriptorium with the existing prompt/schema;
- capture the returned raw JSON response bytes;
- return one `ExtractOutput` with `application/json` media type and response
schema provenance;
- do not convert spell casts into `ArtifactCandidate`;
- do not run shape/source-ref/source-relatedness validation inside the module.
- Keep D&D spell prompt and schema asset metadata in manifest metadata.
- Update `internal/modules/merge/appendorder` to merge raw JSON extract outputs
deterministically:
- preserve input order by chunk index;
- if all extract outputs are JSON objects with one common top-level array
field, concatenate those arrays into one JSON object under that field;
- otherwise produce a JSON array containing each extract output's decoded JSON
value in order;
- reject non-JSON media types with a clear error unless a future option
deliberately supports them.
- Update `internal/modules/normalize/noop` to pass accepted merge output through
unchanged as normalized output.
- Update chunk modules from Stage 1 as needed after interface changes.
- Remove or quarantine old candidate validators from D&D module packages if
they no longer compile. Their replacement belongs under the validation
roadmap in `internal/validators`.
- Update fake/test modules throughout the repository to the raw contracts.
Tests to update or add:
- D&D spells extractor returns raw JSON with schema provenance and no candidate
materialization.
- Append-order merger concatenates common top-level JSON arrays.
- Append-order merger falls back to ordered JSON value arrays when shapes differ.
- Append-order merger rejects invalid JSON and non-JSON media types clearly.
- No-op normalizer returns defensive copies of merge output bytes and metadata.
- Production catalog still registers all modules successfully.
Completion criteria:
- Production modules compile against raw contracts.
- D&D spell extraction no longer rejects spell candidates inside the extractor.
## Stage 6: Output, Manifest, Diagnostics, And Files
Update durable output around normalized lane outputs.
Implementation tasks:
- Update `RunOutput` to carry normalized lane outputs and rejected module-output
records instead of approved/rejected artifacts.
- Update `artifacts.RunManifest` or introduce a more accurately named manifest
package if the old artifact naming becomes misleading. Prefer the smallest
rename that keeps manifest output clear and avoids a broad unrelated refactor.
- Preserve existing manifest fields that remain meaningful:
- pipeline ID and digest;
- module keys;
- lane IDs;
- source digest;
- LLM profile provenance;
- reference provenance;
- module metadata;
- validation status.
- Add manifest/reporting fields needed for raw outputs:
- normalized lane output media type;
- output schema provenance where present;
- rejected stage/lane/chunk/module information;
- retry attempt counts.
- Update `internal/modules/output/json`:
- write `manifest.json`;
- write `index.json`;
- write `warnings.json`;
- write `rejected.json`;
- write one file per normalized lane output.
- For the JSON output encoder, accept `application/json` normalized outputs and
write them as raw JSON files. Recommended file path:
`lanes/<safe-lane-id>.json`.
- Reject non-JSON normalized output media types in the JSON encoder with a clear
error. Future text/Markdown/binary encoders can support other media types.
- Keep output file name validation strict.
- Ensure diagnostics redaction still prevents raw source, prompt, reference,
schema, and payload bytes from appearing in ordinary errors or manifests.
Tests to update or add:
- JSON output writes lane output files and index entries deterministically.
- JSON output rejects invalid JSON bytes and unsupported media types.
- Rejected output records are written without raw payload bytes.
- Manifest includes raw-output provenance and retry attempt counts.
- Diagnostics tests still prove sensitive/large content is not emitted.
Completion criteria:
- Durable output no longer assumes artifact candidates.
- JSON output remains deterministic and safe.
## Stage 7: Documentation, Examples, And Full Validation
After behavior is implemented, update canonical current-behavior docs.
Implementation tasks:
- Update `docs/internal/pipeline.md` for the raw data model.
- Update `docs/internal/modules.md` for new module contracts.
- Update `docs/internal/llm.md` for merge-stage LLM/profile plumbing.
- Update `docs/config.md` for:
- integer source-unit expectations where config examples include source refs;
- merge references;
- module `retries`;
- LLM profile override scope.
- Update `docs/cli.md` for merge reference selectors and profile behavior.
- Update `docs/integrations/json-output.md` for lane output files.
- Update `docs/integrations/dnd-spell-artifacts.md`; if the artifact-specific
contract is no longer durable, rename or replace it with a D&D spell raw-output
integration doc.
- Update `docs/troubleshooting.md` for common raw-output, media-type, retry, and
validation failures.
- Update examples so Seriatim segment IDs are positive integers and expected
output shape matches lane-normalized output.
- Replace `docs/roadmap/implementation.md` with a completed-note document only
after this implementation is finished and reviewed.
- Leave `docs/roadmap/pipeline.md` as target-state roadmap context until the
feature is implemented, then move implemented behavior into canonical docs and
remove stale roadmap language.
Validation commands:
```sh
go test ./internal/core/source
go test ./internal/framework/contracts
go test ./internal/framework/pipeline
go test ./internal/core/config
go test ./internal/cli
go test ./internal/modules/input/seriatim
go test ./internal/modules/chunk/generic
go test ./internal/modules/chunk/dnd/scenes
go test ./internal/modules/extract/dnd/spells
go test ./internal/modules/merge/appendorder
go test ./internal/modules/normalize/noop
go test ./internal/modules/output/json
go test ./...
go vet ./...
go build ./cmd/notarius
```
Documentation inspection:
```sh
rg -n "ArtifactCandidate|approved artifacts|spell artifact|start_unit_id\": \"|end_unit_id\": \"" docs examples internal
rg -n "chunk`, `extract`, and `normalize|deterministic-only stage|json.RawMessage" docs/roadmap docs/internal docs/config.md docs/cli.md
```
Expected result:
- Remaining `ArtifactCandidate` mentions are only in deliberately retained
compatibility code or historical roadmap context.
- Current-behavior docs describe raw lane outputs, integer source-unit IDs,
merge LLM/reference support, and retry behavior.

403
docs/roadmap/pipeline.md Normal file
View File

@@ -0,0 +1,403 @@
# Raw Pipeline Data Model
This roadmap defines the target data model for the pipeline stages after input.
It is paired with the validation system roadmap in
[`validation.md`](validation.md): validators should evaluate module outputs, but
the pipeline first needs a raw-output handoff model that does not require every
module response shape to be represented by bespoke Go structs.
## Goals
- Make typed chunk envelopes and raw module-output envelopes first-class
pipeline handoffs.
- Keep framework-owned provenance around raw payloads so runs remain
deterministic, ordered, and auditable.
- Avoid requiring each extractor, merger, or normalizer response schema to have
matching Go structs.
- Let extract modules produce one raw output per input chunk.
- Let merge modules receive accepted extract outputs and merge them into one raw
payload.
- Let normalize modules optionally post-process accepted merge output.
- Let output modules write normalized output bytes directly, along with
manifests, warnings, rejected outputs, and diagnostics as appropriate.
- Preserve the fixed workflow shape:
```text
input -> chunk -> extract -> merge -> normalize -> output
```
## Non-Goals
- Do not turn the pipeline into an arbitrary DAG or general workflow language.
- Do not make validators mutate, materialize, or rewrite module outputs.
- Do not require a generic source-reference convention for every raw payload in
this pass.
- Do not eliminate typed framework data where the framework genuinely needs it,
such as input source documents and chunk boundaries.
## Cross-Stage Runtime Plumbing
LLM/profile/reference plumbing should be available to every stage that may need
LLM-backed or reference-aware module behavior: `chunk`, `extract`, `merge`, and
`normalize`.
For all four stages, the framework should provide the same categories of runtime
support where the concrete module contract needs them:
- configured LLM profile and profile override handling;
- Scriptorium client access through framework-owned LLM contracts;
- session ID propagation;
- declared reference slots and resolved reference bindings;
- module options and metadata;
- prompt, schema, profile, and reference provenance for manifests and
diagnostics.
`merge` must not be treated as a deterministic-only stage. A merge module may be
simple and deterministic, but it may also be LLM-backed, reference-aware, and
validator-gated in the same way as `chunk`, `extract`, and `normalize`.
## Chunk Stage Target Model
The chunk stage receives the canonical `SourceDocument` from the input stage,
plus any original source input material needed for prompt construction. The
`SourceDocument` remains the source of truth for source identity, source-unit
ordering, source-unit IDs, and source provenance. Original raw input material
may be supplied to LLM-backed chunkers, but it should not replace the
`SourceDocument` as the framework handoff.
Input adapters are responsible for assigning stable integer source-unit IDs. How
those IDs are assigned depends on the input format. A numbered JSON transcript
may map source units directly to transcript segment numbers; a PDF input adapter
may assign page or extracted-text unit numbers; an adapter for unordered source
material may assign deterministic IDs as part of input parsing. Downstream
framework code should treat those IDs as opaque integers owned by the input
adapter.
The chunk module returns one or more ordered `SourceChunk` envelopes. A single
chunk representing the whole source is valid and should be supported.
Each chunk should include framework-readable provenance and ordering metadata,
plus content suitable for extraction. The content may be JSON, plain text,
Markdown, PDF page text, or another module/input-specific representation, as
long as the framework can still associate the chunk with the source and preserve
deterministic order.
Conceptually:
```go
type SourceChunk struct {
ID string
SourceID string
Index int
StartUnitID int
EndUnitID int
Content []byte
MediaType string
Units []source.SourceUnit
Metadata map[string]any
}
```
The exact implementation shape can differ, but it should preserve:
- chunk identity;
- source identity;
- deterministic chunk order;
- source locator or range, normally integer start/end unit IDs;
- chunk content and media type;
- optional source-unit projection when useful;
- metadata needed by extractors, validators, manifests, diagnostics, and output
encoders.
The framework owns the minimal invariants required to schedule extract work:
- the chunker returns at least one chunk;
- chunk IDs and indexes are stable and non-empty;
- chunk source IDs match the source document;
- chunk ordering is deterministic;
- chunk provenance is sufficient to trace the chunk back to the input source.
Domain-specific chunk acceptability belongs in validators. For example, full
coverage, no gaps, no overlap, scene metadata quality, expected media type, and
D&D scene-boundary policy should be explicit validator concerns rather than
hidden framework rules, except where a minimal invariant is required for
extraction to run safely.
## Extract Stage Target Model
The extract stage receives one input chunk from the chunk stage and produces one
raw extract output for that chunk.
The extractor owns:
- prompt selection;
- input material assembly;
- Scriptorium request construction;
- response-schema selection;
- LLM profile and session usage;
- returned raw payload metadata.
The framework owns:
- chunk iteration;
- deterministic ordering;
- association between each extract output and its input chunk;
- execution errors when an extractor does not return output;
- handoff of returned output to validation and later stages.
An extract output should be an envelope, not just raw bytes. Conceptually:
```go
type ExtractOutput struct {
LaneID string
ExtractorKey string
ChunkID string
ChunkIndex int
SourceID string
RawContent []byte
MediaType string
SchemaID string
SchemaName string
SchemaVersion string
Metadata map[string]any
Warnings []contracts.Warning
}
```
The exact implementation shape can differ, but it should preserve:
- raw returned content;
- chunk provenance;
- chunk order;
- source identity;
- module identity;
- response schema provenance;
- metadata needed by validators, mergers, manifests, diagnostics, and output
encoders.
Extractor modules should not be required to convert raw LLM output into
`ArtifactCandidate` values. Domain-specific Go projection may still exist for
specific modules when it is useful, but it should not be the generic pipeline
contract.
Rejected extract outputs are not passed to merge. This keeps downstream
contracts simple and makes rejection behavior explicit. Support for passing
rejected outputs forward as marked data is deferred future work.
## Merge Stage Target Model
The merge stage receives the accepted extract outputs for a lane. Each extract
output corresponds to one input chunk and carries enough metadata to recover the
original chunk order.
The merger owns:
- merge/reconciliation strategy;
- deterministic or LLM-backed merge logic;
- prompt and schema usage when LLM-backed;
- the shape of merged raw output.
The framework owns:
- passing the ordered accepted extract-output set to the merger;
- lane identity;
- source context;
- references;
- runtime LLM plumbing;
- validation handoff for merged output;
- manifest provenance.
Simple merge modules may concatenate raw extract outputs in chunk order. Other
merge modules may deterministically merge JSON documents, reconcile duplicates,
or use an LLM to produce a more coherent merged result.
Conceptually:
```go
type MergeRequest struct {
LaneID string
Source *source.SourceDocument
ExtractOutputs []ExtractOutput
SourceInput contracts.LLMInputMaterial
SessionID string
References contracts.ReferenceSet
LLMClient contracts.StructuredLLMClient
LLMProfile string
Options map[string]any
Metadata map[string]any
}
type MergeOutput struct {
LaneID string
MergerKey string
RawContent []byte
MediaType string
SchemaID string
SchemaName string
SchemaVersion string
Metadata map[string]any
Warnings []contracts.Warning
}
```
The exact implementation shape can differ, but the key contract is that
merge consumes ordered accepted extract outputs and produces one merged raw
output for the lane.
If no accepted extract outputs remain for a lane, the framework should not pass
rejected outputs to the merger. The lane should produce no merge output and
should be reported as rejected or omitted according to run reporting policy.
## Normalize Stage Target Model
The normalize stage receives accepted merge output for a lane and optionally
post-processes it into the final normalized raw payload for that lane.
Normalization may be a no-op, deterministic cleanup, schema conversion, or an
LLM-backed post-processing pass.
The normalizer owns:
- post-merge processing strategy;
- deterministic or LLM-backed normalization logic;
- prompt and schema usage when LLM-backed;
- the shape of normalized raw output.
The framework owns:
- passing accepted merge output to the normalizer;
- lane identity;
- source context;
- references;
- runtime LLM plumbing;
- validation handoff for normalized output;
- manifest provenance.
Conceptually:
```go
type NormalizeRequest struct {
LaneID string
Source *source.SourceDocument
MergeOutput MergeOutput
SourceInput contracts.LLMInputMaterial
SessionID string
References contracts.ReferenceSet
LLMClient contracts.StructuredLLMClient
LLMProfile string
Options map[string]any
Metadata map[string]any
}
type NormalizeOutput struct {
LaneID string
NormalizerKey string
RawContent []byte
MediaType string
SchemaID string
SchemaName string
SchemaVersion string
Metadata map[string]any
Warnings []contracts.Warning
}
```
The exact implementation shape can differ, but the key contract is that
normalize consumes one accepted merged output and produces one normalized raw
output for the lane.
## Output Stage Target Model
The output stage receives validated normalized output bytes and writes them to
its configured destination.
Output modules should not require normalized output to be converted into
`Artifact` values. Output modules should use the normalized output media type to
decide how to serialize or wrap the payload. A JSON output module can write raw
normalized JSON directly, while a text or Markdown output module can write text
payloads directly. Output modules may also write run manifests, warnings,
rejected outputs, and indexes.
Output modules may still choose to provide convenience layouts, grouping, or file
naming conventions, but those should be output concerns rather than constraints
on extractor or normalizer response schemas.
## Validation Relationship
Validation chains should attach to returned module outputs, not to hidden
module-internal conversions.
For chunk:
```text
chunk(input) -> SourceChunk set -> chunk validators -> extract input
```
For extract:
```text
extract(chunk) -> raw ExtractOutput -> extract validators -> merge input
```
For merge:
```text
merge(extract outputs) -> raw MergeOutput -> merge validators -> normalize input
```
For normalize:
```text
normalize(merge output) -> raw NormalizeOutput -> normalize validators -> output
```
Validators must be read-only. They inspect raw output and metadata, return
accept/reject decisions and warnings, and do not rewrite output.
An empty validator chain approves returned output for that validation point.
Framework/runtime errors remain separate from validation rejections: if a module
or Scriptorium call fails before output is returned, the pipeline reports an
execution error rather than asking validators to evaluate nonexistent output.
Rejected outputs do not pass to the next stage. An empty validator chain still
approves returned output for that validation point.
## Retry Policy
A retry means re-running the same module with the same input after that module
fails to produce valid output. Failure to produce valid output includes both:
- framework-level execution errors, such as module errors, Scriptorium errors,
provider errors, or missing returned output;
- validator rejection of returned output.
Retries should be configurable at the pipeline or lane level. Chunk retries are
per source input. Extract retries are per chunk. Merge and normalize retries,
when configured, are per lane. Retry attempts should preserve deterministic
reporting: the final accepted or rejected output should record attempt count and
enough diagnostics/provenance to understand prior failures without leaking
secrets or large payloads by default.
## Relationship To Validation Roadmap
This roadmap defines the data model that validation should evaluate. The
validation system roadmap in [`validation.md`](validation.md) defines validator
registration, mapping, execution classes, and concrete validator behavior.
The shared boundary between the roadmaps is a returned module output: validators
inspect typed chunk output or raw module-output envelopes and decide whether
that output may continue through the pipeline.

420
docs/roadmap/validation.md Normal file
View File

@@ -0,0 +1,420 @@
# Validation System Refactor
This roadmap defines the target state for making validation a first-class,
composable pipeline concern. Current validation behavior is partly module-owned:
the `dnd/spells` extractor defines built-in validators inside the module package,
and the runner falls back to extractor-provided validators when a lane does not
configure validators. The desired end state is that validator implementations,
validator registration, and default module-to-validator mappings are explicit,
reviewable, and independent of concrete module packages.
## Goals
- Move artifact and module-output validation behavior out of `internal/modules`
and into `internal/validators`.
- Keep each validator in its own package.
- Mirror the stage and domain shape of `internal/modules` where a validator is
module-specific.
- Support deterministic and LLM-backed validators through the same framework
contract.
- Allow validators to be mapped to modules at any pipeline stage that returns
module output for validation: `chunk`, `extract`, `merge`, or `normalize`.
- Make default production module-to-validator mappings centralized and
human-readable.
- Allow pipeline configuration to override default mappings for advanced use.
- Treat an empty validator set as valid and equivalent to approval.
- Preserve the rule that module output passes forward unless a validator rejects
it.
- Make successfully returned module output the explicit validation boundary:
questions about output syntax, media type, schema conformance, and domain
acceptability should be answered by validators.
## Non-Goals
- Do not create a general workflow engine or arbitrary validation DAG.
- Do not enforce validator compatibility with a module or stage in this pass.
- Do not move ordinary runtime invariant checks into validator packages.
- Do not require every module to have validators.
- Do not require LLM-backed validators for modules that can be checked
deterministically.
- Do not silently reorder configured validator chains unless that behavior is
introduced deliberately and documented as part of the validator contract.
## Validation Boundary
Validation packages should own approve/reject/warning evaluation of successfully
returned module outputs. This means logic that decides whether a chunk result,
raw extract output, raw merge output, raw normalize output, or raw LLM response
should continue through the pipeline belongs in `internal/validators`.
The boundary is:
- no module output was returned: execution failed, and the pipeline should report
a module or runtime error;
- module output was returned: the validator chain decides whether that output is
acceptable, and an empty validator chain approves it.
Scriptorium and provider errors are execution failures rather than validator
rejections. This includes provider timeouts, authentication failures,
transport/runtime failures, Scriptorium structured-output retry exhaustion, and
malformed responses that Scriptorium rejects before returning module output.
Other validation-like checks should remain with their owning packages:
- input parsing and source-format validation stay in input modules;
- source document and source reference invariants stay in `internal/core/source`;
- config validation stays in `internal/core/config`;
- registry, profile, and pipeline consistency checks stay in framework and CLI
code;
- response schema loading stays in module asset code;
- Scriptorium runtime errors stay in LLM runtime code.
Domain validators may call reusable core helpers such as `source.ValidateRef`,
but the module-output approval or rejection decision should be made by a
validator.
Validators should answer module-output questions such as:
- is returned content syntactically valid JSON;
- does returned JSON conform to the module's declared schema;
- are required domain fields present and non-empty;
- are source references valid and appropriately grounded;
- does domain-specific output satisfy the configured policy.
## Audita Patterns To Adapt
The validator architecture should adapt useful patterns from
[`audita`](https://gitea.maximumdirect.net/eric/audita) without copying its
narrower transcript-correction shape directly.
Useful patterns:
- concrete validators live under `internal/validators`;
- shared validator runtime mechanics live under a framework package;
- built-in validator keys are stable and centrally registered;
- built-in chains are centrally reviewable;
- validators carry execution-class metadata;
- deterministic and LLM-backed validators implement one contract;
- LLM-backed validator runtime can share batching, diagnostics, structured
response handling, and malformed-response policy;
- reports and manifests can classify validator decisions by execution class.
Important Notarius differences:
- mappings must be keyed by stage and module key, not module key alone;
- mappings should be owned by the central production catalog, not resolved inside
concrete module constructors;
- configured mapping order should be authoritative unless the config explicitly
opts into a different ordering policy;
- validators must support chunk, extract, merge, and normalize outputs rather
than only one proposal shape.
## Validator Package Layout
Concrete validators should live under `internal/validators`. Module-specific
validators should mirror the module tree and use one package per validator:
```text
internal/validators/extract/dnd/spells/shape
internal/validators/extract/dnd/spells/source_refs
internal/validators/extract/dnd/spells/source_relatedness
```
Generic validators may live under stage-specific generic paths when they operate
on a particular stage output shape:
```text
internal/validators/chunk/generic/...
internal/validators/extract/generic/...
internal/validators/merge/generic/...
internal/validators/normalize/generic/...
```
Truly stage-independent validators may live under `internal/validators/generic`
once there is a real shared validator that justifies that location. Generic JSON
syntax and JSON schema validators are likely candidates for
`internal/validators/generic/valid_json` and
`internal/validators/generic/valid_json_schema`.
Each validator package should expose:
- a stable validator key;
- execution-class metadata;
- a constructor;
- a validator spec suitable for registration;
- a `Register` function;
- focused tests for decisions, warnings, errors, and diagnostics behavior.
Reusable validator runtime mechanics should live in framework code, such as
`internal/framework/validators`, not in concrete validator packages. This package
can own shared helpers for decision cardinality, approval/rejection construction,
LLM validator batching, validator diagnostics, and Scriptorium request plumbing.
The concrete validator packages should own policy: what they inspect, what they
approve or reject, what warning reason codes they emit, and how they interpret
domain-specific data.
## Validator Design Policy
Validators should follow a small-tool model: each validator should do one thing
well. If a validator both rejects output and emits unrelated warnings, split
those concerns into separate validators so production mappings can include,
exclude, and order them independently.
Validators are read-only. A validator must not mutate pipeline state, rewrite
module output, materialize raw output into typed stage output, or enrich the
`ModuleOutput` passed to later validators. A validator returns an
accept/reject verdict for the output it evaluates, plus any warnings or
diagnostic references. Any conversion from raw module output into a downstream
representation is a separate materialization concern and must not be hidden
inside a validator.
The initial generic validator set should include:
- `generic/always_accept`: accepts returned module output unchanged. This is
functionally equivalent to a no-op validator and is primarily useful for tests,
demonstrations, and explicit pass-through configurations.
- `generic/always_reject`: rejects returned module output without inspecting it.
This is primarily useful for tests and for proving rejection plumbing,
manifests, and diagnostics.
- `generic/valid_json`: inspects raw returned module output and accepts only
syntactically valid JSON.
- `generic/valid_json_schema`: compares raw returned JSON with the module's
configured response schema and accepts only schema-conformant output.
The exact keys may be adjusted during implementation to match local naming
conventions, but the validator set should preserve these four behaviors.
For the current D&D spell behavior, the target split is:
- `generic/valid_json`: rejects returned module output that is not syntactically
valid JSON.
- `generic/valid_json_schema`: rejects returned JSON that does not conform to
the configured response schema.
- `extract/dnd/spells/shape`: rejects malformed spell-cast payloads and missing
required spell fields.
- `extract/dnd/spells/source_refs`: rejects missing or invalid source
references.
- `extract/dnd/spells/source_relatedness`: warning-only validator that reports
when a spell name is not found in the cited source text.
Production default mappings should generally list deterministic validators
before LLM-backed validators. This keeps cheap structural failures from consuming
model calls and keeps diagnostics easier to interpret. Pipeline-configured order
should still be authoritative; if a user explicitly lists an LLM-backed
validator before a deterministic validator, the framework should honor that
order rather than silently reshuffling it.
## Execution Classes
Validator specs should declare an execution class:
```go
type ExecutionClass string
const (
ExecutionClassDeterministic ExecutionClass = "deterministic"
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
)
```
Execution class should be metadata on the validator spec or registered
definition, not an ad hoc convention inferred from package paths. It should be
used for:
- human-readable catalog and manifest reporting;
- diagnostics and timing summaries;
- operational policy such as concurrency budgeting for LLM-backed validators;
- default mapping review, where deterministic validators should usually appear
before LLM-backed validators.
Execution class should not by itself imply compatibility with a stage or module.
## Validator Contract
The validator framework should support validation of outputs from `chunk`,
`extract`, `merge`, and `normalize` stages. The contract should be generalized
enough for stage-specific validators to inspect the output they care about while
ignoring irrelevant fields.
The request should carry:
- stage name;
- module key;
- raw module output content when available;
- response schema metadata when the module declares one;
- source document;
- source input material;
- session ID;
- references;
- LLM client and profile for LLM-backed validators;
- options and metadata;
- chunk output when validating a chunk module;
- stage-specific typed envelopes when the stage owns them, such as chunk
envelopes for chunk validation.
A shared `ModuleOutput` envelope should represent the validation boundary.
Validators may inspect raw returned content and any already-existing typed
stage output, but they must not modify it.
Conceptually:
```go
type ModuleOutput struct {
Stage pipeline.Stage
ModuleKey string
RawContent []byte
MediaType string
ResponseSchema *llm.ResponseSchemaMetadata
Chunks []contracts.SourceChunk
Warnings []contracts.Warning
}
```
The final implementation does not need to use this exact shape, but it should
preserve the boundary: returned raw module output can enter validation before
any separate materialization step converts it into a stage-specific typed
representation.
The result should continue to express validator identity, warnings, and explicit
decisions. For output collections, the implementation should define an explicit
decision shape rather than silently mutating lists. Validator decisions reject or
approve output; validators do not rewrite output.
An empty validator list is always valid. With no validators, the framework should
pass module output forward unchanged and treat the output as approved for that
validation point. If the approved output cannot be consumed by a later stage
because its media type or envelope shape is unsuitable, that failure should be
reported at the downstream boundary that requires a different shape, not as an
implicit pre-validation rejection.
## Module Development Workflow
The validation system should make iterative module development easier. A module
author should be able to start with an explicit empty validator mapping and
inspect returned raw LLM output without first satisfying JSON syntax, schema,
media-type, or domain validators.
A typical development path should be:
1. Configure an empty validator set for the module and inspect raw returned
output.
2. Add `generic/valid_json` and adjust prompts until the model reliably returns
syntactically valid JSON.
3. Add `generic/valid_json_schema` and iterate on prompt/schema alignment.
4. Add media-type or schema validators appropriate to the module's intended
output format.
5. Add domain-specific validators one at a time until production policy is
represented explicitly in the chain.
This workflow is a central reason for making output validation explicit and
composable rather than hiding schema, shape, or domain checks inside module
implementation code.
## Central Production Mappings
Production defaults should be defined in a central, human-readable location near
the production module and validator registries. The mapping should be keyed by
stage and module key, not only by module key, so future modules can share keys
only when stage context makes their ownership unambiguous.
Conceptually:
```go
{
Stage: pipeline.StageExtract,
Module: "dnd/spells",
Validators: []pipeline.ModuleBinding{
pipeline.Binding("generic/valid_json"),
pipeline.Binding("generic/valid_json_schema"),
pipeline.Binding("extract/dnd/spells/shape"),
pipeline.Binding("extract/dnd/spells/source_refs"),
pipeline.Binding("extract/dnd/spells/source_relatedness"),
},
}
```
The production catalog should expose three related surfaces together:
- available modules;
- available validators;
- default module-to-validator mappings.
This makes production validation policy reviewable without constructing concrete
modules or searching inside module implementation packages.
The validator registry should expose registered validator specs without building
validators, including key and execution class. Building a validator should still
be available for runtime execution.
## Pipeline Overrides
Pipeline configuration should be able to override the central default mapping
for a module binding. Override semantics should distinguish three states:
- unset validators: use the central production/default mapping;
- explicit empty validators: run no validators and pass output forward;
- explicit non-empty validators: run exactly the configured validators in the
configured order.
This keeps the happy path concise while preserving advanced control for
experimentation, debugging, and custom deployments.
The run manifest should record the resolved validator chain for each validation
point so completed runs remain auditable after defaults or configuration change.
Each manifest entry should include at least validator key and execution class,
and should preserve the resolved order actually used for the run.
## LLM-Backed Validator Runtime
LLM-backed validators should use the same validator contract as deterministic
validators. Shared framework runtime should provide common support for:
- Scriptorium request construction;
- validator prompt and schema provenance;
- diagnostics redaction;
- optional batching or context-window controls when validator inputs are large;
- mapping successful LLM validator responses into validator decisions and
warnings;
- consistent handling of Scriptorium/runtime errors.
Scriptorium errors raised during validator execution should be treated as
validator execution errors unless a specific validator deliberately converts a
successful response into reject/warn decisions. This keeps provider/runtime
failure distinct from a validator's semantic rejection of module output.
## Stage Coverage
Validators should be composable across all LLM-eligible stages:
- `chunk`: validators can evaluate chunk boundaries, coverage, overlap, metadata,
or module-specific chunk quality.
- `extract`: validators can evaluate raw extracted output, source references,
payload shape, evidence quality, media type, or domain constraints.
- `merge`: validators can evaluate merged output, cross-chunk consistency,
deduplication results, media type, or domain-specific reconciliation.
- `normalize`: validators can evaluate normalized output, final shape,
post-processing results, media type, or domain-specific policy.
The framework should not require compatibility declarations in this pass. A
validator mapped to an unsuitable output shape should return a clear error, or
approve unchanged only when that is explicitly the validator's documented
behavior.
## Documentation Impact
When implemented, current-behavior docs and policy should be updated together:
- `docs/policy/architecture.md` should describe centralized validator mappings
rather than module-owned validator chains.
- `docs/internal/modules.md` should remove claims that concrete modules own
validator defaults.
- Internal validation docs should describe validator package ownership, mapping
precedence, empty-chain approval behavior, and LLM-backed validator support.
- User/config docs should describe how pipeline validator overrides work once the
syntax is implemented.
Roadmap docs should not remain the canonical description of implemented
validation behavior after the refactor is complete.