Update planning roadmap and add a staged imnplementation plan for the validator registry
This commit is contained in:
@@ -1,17 +1,29 @@
|
||||
# Validation System Refactor
|
||||
|
||||
This roadmap defines the target state for making validation a first-class,
|
||||
composable pipeline concern. Current validation behavior is partly module-owned:
|
||||
the `dnd/spells` extractor defines built-in validators inside the module package,
|
||||
and the runner falls back to extractor-provided validators when a lane does not
|
||||
configure validators. The desired end state is that validator implementations,
|
||||
validator registration, and default module-to-validator mappings are explicit,
|
||||
reviewable, and independent of concrete module packages.
|
||||
composable pipeline concern.
|
||||
|
||||
Current pipeline behavior is raw-output based. The runner can execute
|
||||
`contracts.RawValidator` chains from `pipeline.RawValidationRegistry` for
|
||||
`chunk`, `extract`, `merge`, and `normalize` outputs. Empty chains approve by
|
||||
default, validator rejection records a rejected raw output, and rejected output
|
||||
does not pass to the next stage. Production currently registers no raw
|
||||
validators, and non-empty pipeline-configured validator lists are rejected so
|
||||
they cannot appear in manifests without executing.
|
||||
|
||||
Legacy candidate validator contracts and D&D spell validators still exist under
|
||||
`internal/modules/extract/dnd/spells`, but they are not part of the current
|
||||
runner path. The desired end state is that validator implementations, validator
|
||||
registration, and default module-to-validator mappings are explicit, reviewable,
|
||||
and independent of concrete module packages.
|
||||
|
||||
## Goals
|
||||
|
||||
- Move artifact and module-output validation behavior out of `internal/modules`
|
||||
and into `internal/validators`.
|
||||
- Retire or replace the legacy candidate-oriented `contracts.Validator`,
|
||||
`ValidationRequest`, and `ValidationResult` path after equivalent raw-output
|
||||
validators exist.
|
||||
- Keep each validator in its own package.
|
||||
- Mirror the stage and domain shape of `internal/modules` where a validator is
|
||||
module-specific.
|
||||
@@ -22,6 +34,8 @@ reviewable, and independent of concrete module packages.
|
||||
- Make default production module-to-validator mappings centralized and
|
||||
human-readable.
|
||||
- Allow pipeline configuration to override default mappings for advanced use.
|
||||
- Preserve the distinction between an unset validator override and an explicit
|
||||
empty validator override.
|
||||
- Treat an empty validator set as valid and equivalent to approval.
|
||||
- Preserve the rule that module output passes forward unless a validator rejects
|
||||
it.
|
||||
@@ -32,6 +46,8 @@ reviewable, and independent of concrete module packages.
|
||||
## Non-Goals
|
||||
|
||||
- Do not create a general workflow engine or arbitrary validation DAG.
|
||||
- Do not revive the legacy artifact-candidate validation model as the primary
|
||||
runner path.
|
||||
- Do not enforce validator compatibility with a module or stage in this pass.
|
||||
- Do not move ordinary runtime invariant checks into validator packages.
|
||||
- Do not require every module to have validators.
|
||||
@@ -43,9 +59,9 @@ reviewable, and independent of concrete module packages.
|
||||
## Validation Boundary
|
||||
|
||||
Validation packages should own approve/reject/warning evaluation of successfully
|
||||
returned module outputs. This means logic that decides whether a chunk result,
|
||||
raw extract output, raw merge output, raw normalize output, or raw LLM response
|
||||
should continue through the pipeline belongs in `internal/validators`.
|
||||
returned module outputs. This means logic that decides whether a chunk result or
|
||||
raw extract, merge, or normalize payload should continue through the pipeline
|
||||
belongs in `internal/validators`.
|
||||
|
||||
The boundary is:
|
||||
|
||||
@@ -63,6 +79,9 @@ Other validation-like checks should remain with their owning packages:
|
||||
|
||||
- input parsing and source-format validation stay in input modules;
|
||||
- source document and source reference invariants stay in `internal/core/source`;
|
||||
- generic framework chunk invariants that make extraction possible stay in the
|
||||
runner, such as non-empty chunk content, valid unit ranges, and canonical
|
||||
source-unit ordering;
|
||||
- config validation stays in `internal/core/config`;
|
||||
- registry, profile, and pipeline consistency checks stay in framework and CLI
|
||||
code;
|
||||
@@ -77,6 +96,7 @@ Validators should answer module-output questions such as:
|
||||
|
||||
- is returned content syntactically valid JSON;
|
||||
- does returned JSON conform to the module's declared schema;
|
||||
- does the returned media type match the module or pipeline policy;
|
||||
- are required domain fields present and non-empty;
|
||||
- are source references valid and appropriately grounded;
|
||||
- does domain-specific output satisfy the configured policy.
|
||||
@@ -120,6 +140,10 @@ internal/validators/extract/dnd/spells/source_refs
|
||||
internal/validators/extract/dnd/spells/source_relatedness
|
||||
```
|
||||
|
||||
D&D spell validation policy belongs under
|
||||
`internal/validators/extract/dnd/spells`, split by concern rather than bundled
|
||||
inside the extractor module.
|
||||
|
||||
Generic validators may live under stage-specific generic paths when they operate
|
||||
on a particular stage output shape:
|
||||
|
||||
@@ -237,46 +261,53 @@ The validator framework should support validation of outputs from `chunk`,
|
||||
enough for stage-specific validators to inspect the output they care about while
|
||||
ignoring irrelevant fields.
|
||||
|
||||
The request should carry:
|
||||
The current `contracts.RawValidationRequest` is the right starting point. It
|
||||
already carries stage, lane, module, source, source and chunk provenance,
|
||||
response schema metadata, raw payload, and run metadata. The final contract
|
||||
should evolve from that raw-output shape rather than from the legacy
|
||||
artifact-candidate `ValidationRequest`.
|
||||
|
||||
- stage name;
|
||||
- module key;
|
||||
- raw module output content when available;
|
||||
- response schema metadata when the module declares one;
|
||||
- source document;
|
||||
- source input material;
|
||||
Additional fields needed for the full validator system include:
|
||||
|
||||
- source input material when a validator needs to compare module output to the
|
||||
original source payload;
|
||||
- session ID;
|
||||
- references;
|
||||
- resolved references for the validated target;
|
||||
- LLM client and profile for LLM-backed validators;
|
||||
- options and metadata;
|
||||
- chunk output when validating a chunk module;
|
||||
- stage-specific typed envelopes when the stage owns them, such as chunk
|
||||
envelopes for chunk validation.
|
||||
- validator options;
|
||||
- chunk output collections when validating a chunk module;
|
||||
- ordered upstream output envelopes when validating merge or normalize behavior.
|
||||
|
||||
A shared `ModuleOutput` envelope should represent the validation boundary.
|
||||
Validators may inspect raw returned content and any already-existing typed
|
||||
stage output, but they must not modify it.
|
||||
A shared module-output envelope should represent the validation boundary.
|
||||
Validators may inspect raw returned content and any already-existing typed stage
|
||||
envelope, such as `SourceChunk` values for chunk validation, but they must not
|
||||
modify it.
|
||||
|
||||
Conceptually:
|
||||
|
||||
```go
|
||||
type ModuleOutput struct {
|
||||
Stage pipeline.Stage
|
||||
Stage pipeline.ModuleStage
|
||||
ModuleKey string
|
||||
LaneID string
|
||||
|
||||
RawContent []byte
|
||||
MediaType string
|
||||
ResponseSchema *llm.ResponseSchemaMetadata
|
||||
ResponseSchema contracts.ResponseSchema
|
||||
|
||||
Chunks []contracts.SourceChunk
|
||||
Warnings []contracts.Warning
|
||||
SourceID string
|
||||
ChunkID string
|
||||
ChunkIndex int
|
||||
Chunks []contracts.SourceChunk
|
||||
Warnings []contracts.Warning
|
||||
}
|
||||
```
|
||||
|
||||
The final implementation does not need to use this exact shape, but it should
|
||||
preserve the boundary: returned raw module output can enter validation before
|
||||
any separate materialization step converts it into a stage-specific typed
|
||||
representation.
|
||||
preserve the boundary: returned raw module output enters validation as immutable
|
||||
module output. Validators may parse raw bytes internally to decide approve,
|
||||
reject, or warn, but parsing inside a validator must not create or replace the
|
||||
payload passed to later stages.
|
||||
|
||||
The result should continue to express validator identity, warnings, and explicit
|
||||
decisions. For output collections, the implementation should define an explicit
|
||||
@@ -292,10 +323,10 @@ implicit pre-validation rejection.
|
||||
|
||||
## Module Development Workflow
|
||||
|
||||
The validation system should make iterative module development easier. A module
|
||||
author should be able to start with an explicit empty validator mapping and
|
||||
inspect returned raw LLM output without first satisfying JSON syntax, schema,
|
||||
media-type, or domain validators.
|
||||
The validation system should make iterative module development easier. Once
|
||||
pipeline overrides are implemented, a module author should be able to start with
|
||||
an explicit empty validator mapping and inspect returned raw LLM output without
|
||||
first satisfying JSON syntax, schema, media-type, or domain validators.
|
||||
|
||||
A typical development path should be:
|
||||
|
||||
@@ -349,10 +380,20 @@ The validator registry should expose registered validator specs without building
|
||||
validators, including key and execution class. Building a validator should still
|
||||
be available for runtime execution.
|
||||
|
||||
The mapping surface should preserve the current useful behavior of stage/module
|
||||
lookup and empty-chain approval while adding validator specs, execution-class
|
||||
metadata, production registration, config override integration, and manifest
|
||||
reporting of the resolved chain.
|
||||
|
||||
## Pipeline Overrides
|
||||
|
||||
Pipeline configuration should be able to override the central default mapping
|
||||
for a module binding. Override semantics should distinguish three states:
|
||||
for a module binding. Current configuration validation rejects non-empty
|
||||
validator lists, and the current config/profile structs do not preserve whether
|
||||
an empty list was explicitly configured or simply omitted. The target config
|
||||
model must preserve that distinction.
|
||||
|
||||
Override semantics should distinguish three states:
|
||||
|
||||
- unset validators: use the central production/default mapping;
|
||||
- explicit empty validators: run no validators and pass output forward;
|
||||
@@ -405,7 +446,8 @@ behavior.
|
||||
|
||||
## Documentation Impact
|
||||
|
||||
When implemented, current-behavior docs and policy should be updated together:
|
||||
Current-behavior docs and policy should describe the implemented validation
|
||||
system once the refactor is complete:
|
||||
|
||||
- `docs/policy/architecture.md` should describe centralized validator mappings
|
||||
rather than module-owned validator chains.
|
||||
@@ -413,8 +455,9 @@ When implemented, current-behavior docs and policy should be updated together:
|
||||
validator defaults.
|
||||
- Internal validation docs should describe validator package ownership, mapping
|
||||
precedence, empty-chain approval behavior, and LLM-backed validator support.
|
||||
- User/config docs should describe how pipeline validator overrides work once the
|
||||
syntax is implemented.
|
||||
- User/config docs should replace the current "configured validators are
|
||||
reserved and rejected" language with the implemented pipeline override
|
||||
contract.
|
||||
|
||||
Roadmap docs should not remain the canonical description of implemented
|
||||
validation behavior after the refactor is complete.
|
||||
|
||||
Reference in New Issue
Block a user