diff --git a/docs/roadmap/dnd.md b/docs/roadmap/dnd.md deleted file mode 100644 index 785d88e..0000000 --- a/docs/roadmap/dnd.md +++ /dev/null @@ -1,22 +0,0 @@ -# D&D Shared Module Helper Refactor - -The D&D shared module helper refactor is complete. - -The implemented package split is: - -- `internal/modules/sharedassets`: generic shared prompt filesystem composition - and domain-neutral shared asset plumbing. -- `internal/modules/sharedassets/dnd`: D&D shared prompt fragments, accepted - reference media types, `players`/`party`/`glossary` reference slot helpers, - deprecated `roster` alias handling, prompt input assembly, deterministic - reference rendering, and D&D shared prompt hash parts. -- Concrete D&D modules: stage contracts and registration, module keys, - provided/required capabilities, prompt IDs and versions, module-local prompt - definitions, response schemas, validators, artifact or scene interpretation, - and manifest metadata. - -The `dnd/scenes` chunker and `dnd/spells` extractor now use the shared D&D -helper package for common prompt and reference behavior while keeping -stage-specific semantics in their concrete module packages. D&D shared prompt -fragments are exposed to Scriptorium through each module's local -`./sharedassets/` prompt paths. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 7ccf168..886497d 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,19 +1,419 @@ -# D&D Shared Helper Refactor Completed +# Raw Pipeline Data Model Implementation Plan -The D&D shared helper refactor is implemented. +This plan implements the target state in +[`pipeline.md`](pipeline.md). It is intentionally staged for multiple coding +passes. Do not skip stages: later work assumes the contracts and tests from +earlier stages are already in place. -Implemented package ownership: +The desired end state is a fixed-shape pipeline: -- `internal/modules/sharedassets`: generic prompt filesystem composition and - domain-neutral shared asset plumbing. -- `internal/modules/sharedassets/dnd`: reusable D&D prompt fragments, prompt - filesystem composition for D&D modules, D&D reference slot helpers, prompt - input assembly, reference rendering, and D&D shared prompt hash parts. -- Concrete D&D modules: stage contracts, module registration, module-local - prompt definitions, response schemas, validators, request validation, - response conversion, and manifest metadata ownership. +```text +input -> chunk -> extract -> merge -> normalize -> output +``` -The refactor preserved module keys, prompt IDs, prompt versions, -Scriptorium-visible message order, response schema IDs and names, CLI/config -semantics, manifest shape, diagnostics redaction policy, and D&D scene/spell -interpretation behavior. +`chunk` produces typed chunk envelopes. `extract`, `merge`, and `normalize` +produce raw byte payload envelopes with media type and provenance. Rejected +outputs do not pass to the next stage. + +## Stage 1: Integer Source Units And Chunk Payloads + +Update the core source and chunk contracts before changing downstream stages. + +Implementation tasks: + +- Change `internal/core/source.SourceUnit.ID` from `string` to `int`. +- Change `internal/core/source.SourceRef.StartUnitID` and `EndUnitID` from + `string` to `int`. +- Treat source-unit IDs as positive, stable, input-adapter-owned integers. + `0` and negative IDs should be invalid. +- Update source validation helpers, including unit lookup and source-ref + ordering checks, to use integer IDs. +- Update `contracts.SourceChunk` to include: + - `StartUnitID int`; + - `EndUnitID int`; + - `Content []byte`; + - `MediaType string`. +- Keep `SourceChunk.Units []source.SourceUnit` for framework scheduling, + diagnostics, and modules that need unit metadata. +- Update chunk canonicalization in `internal/framework/pipeline` to enforce + minimal scheduling invariants: + - at least one chunk; + - non-empty chunk ID; + - source ID matches the source document; + - indexes are deterministic and contiguous from zero; + - start/end unit IDs exist and are ordered; + - `Units` are canonical copies from the source document; + - content is non-empty; + - media type is non-empty. +- Update the Seriatim input adapter to parse segment IDs as integers. + Accept JSON numbers and numeric strings only when they are positive integers. + Reject non-numeric IDs such as `seg-001`. +- Update Seriatim examples, fixtures, tests, and integration docs to use + integer segment IDs. +- Update D&D source-reference helper code under + `internal/modules/sharedassets/dnd` so source refs remain integer-valued all + the way into `source.SourceRef`. +- Remove fallback behavior that converts integer LLM unit refs back into string + source-unit IDs. +- Update the generic chunker and D&D scenes chunker to populate + `StartUnitID`, `EndUnitID`, `Content`, and `MediaType`. +- For current transcript chunkers, use `application/json` content containing a + canonical JSON encoding of the chunk's source units. Keep the original + `Units` field populated as framework-owned typed metadata. + +Tests to update or add: + +- `go test ./internal/core/source` +- `go test ./internal/modules/input/seriatim` +- `go test ./internal/modules/chunk/generic` +- `go test ./internal/modules/chunk/dnd/scenes` +- `go test ./internal/framework/contracts` +- `go test ./internal/framework/pipeline` + +Completion criteria: + +- All source refs and source-unit IDs in core/framework types are integers. +- Chunkers return extraction-ready content bytes plus media type. +- Framework chunk validation still allows partial coverage and overlap unless a + validator rejects them. + +## Stage 2: Raw Module Output Contracts + +Replace artifact-candidate stage contracts with raw payload envelopes. + +Implementation tasks: + +- Add contract types in `internal/framework/contracts` for raw stage payloads. + The exact names may differ, but the contracts must preserve: + - raw bytes; + - media type; + - lane ID; + - stage module key; + - source ID; + - chunk ID and chunk index where applicable; + - response schema ID, name, and version when applicable; + - metadata; + - warnings. +- Recommended shape: + - `ExtractOutput` + - `MergeOutput` + - `NormalizeOutput` + - a small shared payload/provenance helper if it reduces duplication. +- Update `Extractor` so `Extract` returns one raw `ExtractOutput` for one input + chunk instead of `[]artifacts.ArtifactCandidate`. +- Remove `Extractor.ArtifactType`, `Extractor.SchemaVersion`, and + `Extractor.Validators` from the generic extractor contract. Schema and prompt + provenance should come from the returned output and manifest metadata, not + artifact-specific methods. +- Update `Merger` so `Merge` receives ordered accepted `[]ExtractOutput` and + returns one raw `MergeOutput`. +- Update `Normalizer` so `Normalize` receives one accepted `MergeOutput` and + returns one raw `NormalizeOutput`. +- Update `OutputRequest` so output encoders receive the validated normalized + lane outputs rather than approved `Artifact` values. +- Add a raw rejected-output record type for manifests/output files. It should + identify stage, lane, module, chunk when relevant, validator or error reason, + message, attempt count, and diagnostic artifact path if present. It must not + include large raw content by default. +- Leave old artifact/candidate types in `internal/core/artifacts` only if + temporary compatibility code still needs them during the migration. They + should not remain in the final stage contracts. + +Tests to update or add: + +- Contract composition tests proving fake chunk, extract, merge, normalize, and + output modules compose using raw envelopes. +- Defensive-copy tests for raw content and metadata maps. +- Registry integration tests proving the new interfaces build and run. + +Completion criteria: + +- Framework contracts no longer require extractors, mergers, normalizers, or + output encoders to use `ArtifactCandidate` or `Artifact`. +- Stage outputs carry raw bytes and media type. + +## Stage 3: Config, References, Profiles, And Retries + +Make `merge` a first-class LLM/profile/reference stage and add retry policy. + +Implementation tasks: + +- Update `pipeline.referenceSlotStage` so `merge` modules may declare reference + slots. +- Add `MergeReferences` to `ResolvedArtifactLane`. +- Resolve and materialize merge references using the same declared-slot model + used by `chunk`, `extract`, and `normalize`. +- Update reference provenance so merge-stage references appear in manifests and + resolved reference diagnostics. +- Update config parsing, config validation, redaction, cloning, and examples so + `artifacts..merge.references` is valid. +- Update CLI `--reference` selector parsing and ambiguity checks to support: + - `merge.=` where unambiguous; + - `.merge.=`; + - lane-qualified shorthand only when it resolves unambiguously. +- Update `--without-reference` behavior to support merge-stage slots. +- Update `ModuleStage` LLM-capable helpers so `chunk`, `extract`, `merge`, and + `normalize` are all considered LLM-capable. +- Update `--llm-profile` override handling so it applies to `chunk`, every lane + `extract`, every lane `merge`, and every lane `normalize`. +- Update explicit Scriptorium profile validation so it inspects only those four + LLM-capable stages. +- Add `retries` to `pipeline.ModuleBinding` as a non-negative integer count of + extra attempts after the first attempt. Default is `0`. +- Apply `retries` only to `chunk`, `extract`, `merge`, and `normalize` runtime + execution. Keep input and output retry behavior out of scope. +- Validate `retries >= 0` in config validation. +- Ensure redacted config preserves retry counts. + +Tests to update or add: + +- Config accepts merge references and rejects unknown/undeclared merge slots. +- CLI reference selectors work for merge and remain strict for ambiguous slots. +- `--llm-profile` overrides merge bindings as well as chunk/extract/normalize. +- Explicit profile validation includes merge and ignores non-LLM stages. +- Negative retries are rejected. + +Completion criteria: + +- Merge has the same runtime plumbing class as chunk/extract/normalize. +- Retry policy is representable in pipeline config and resolved bindings. + +## Stage 4: Runner Orchestration And Retry Semantics + +Rewrite pipeline execution around raw envelopes. + +Implementation tasks: + +- Introduce a small retry helper in `internal/framework/pipeline` that reruns + the same module with the same input when: + - the module returns a framework-level error; + - the module returns output that is rejected by that stage's validator chain. +- Preserve context cancellation: never retry after `ctx.Err()` is non-nil. +- Record attempt count and compact attempt diagnostics for manifests/output. +- Do not log or manifest raw payload bytes by default. +- Run chunk once per source input, with retries from the chunk binding. +- After successful chunk execution, run chunk validation when validator mapping + support exists. A rejected chunk result after final retry should stop + downstream execution and produce a rejected run record. +- Run extract once per accepted chunk, with per-chunk retries from the extract + binding. +- Omit rejected extract outputs from merge input. +- If a lane has no accepted extract outputs after retries, skip merge and + normalize for that lane, record rejected output details, and continue to the + next lane. +- Run merge once per lane with accepted extract outputs, with retries from the + merge binding. +- If merge output is rejected after final retry, omit that lane from normalize + and output. +- Run normalize once per accepted merge output, with retries from the normalize + binding. +- If normalize output is rejected after final retry, omit that lane from output. +- Treat validator rejection as non-fatal run outcome. Treat unrecovered + framework-level execution errors as run failures. +- Preserve deterministic ordering: + - chunks by chunk index; + - extract outputs by chunk index; + - lane outputs by resolved lane order. +- Continue to collect warnings from every successful module attempt whose output + is used. For failed attempts, record compact attempt diagnostics rather than + promoting warnings as final-stage warnings unless the implementation already + has a clear warning policy. +- Set manifest validation status to: + - `approved` when all produced normalized lane outputs pass; + - `rejected` when one or more module outputs are rejected; + - failed-run status through existing failure handling for unrecovered runtime + errors. + +Validation seam scope: + +- Add the runner-side raw validation seam needed by this data-model refactor. +- Represent the validation boundary as raw module output plus stage provenance, + consistent with [`validation.md`](validation.md). +- Support empty validator chains as approval. +- Add fake validator/test-only coverage for approval, rejection, and + retry-on-rejection behavior. +- Do not implement the full production validator package migration, default + validator mappings, or concrete generic validators in this plan. Those remain + owned by the validation roadmap. + +Tests to update or add: + +- Runner passes chunk content and media type to extractors. +- Runner passes ordered accepted extract outputs to merge. +- Runner passes accepted merge output to normalize. +- Rejected extract output is omitted from merge input. +- A lane with no accepted extracts is omitted and recorded. +- Rejected merge output prevents normalize for that lane. +- Rejected normalize output prevents output for that lane. +- Retry reruns the same module input after framework error. +- Retry reruns the same module input after validator rejection. +- Retry stops after configured attempts and records attempt count. +- Context cancellation stops retries. + +Completion criteria: + +- The runner no longer depends on candidate materialization between extract, + merge, normalize, and output. +- Rejected outputs never pass to the next stage. + +## Stage 5: Production Module Migration + +Update concrete modules to the new contracts. + +Implementation tasks: + +- Update `internal/modules/extract/dnd/spells`: + - call Scriptorium with the existing prompt/schema; + - capture the returned raw JSON response bytes; + - return one `ExtractOutput` with `application/json` media type and response + schema provenance; + - do not convert spell casts into `ArtifactCandidate`; + - do not run shape/source-ref/source-relatedness validation inside the module. +- Keep D&D spell prompt and schema asset metadata in manifest metadata. +- Update `internal/modules/merge/appendorder` to merge raw JSON extract outputs + deterministically: + - preserve input order by chunk index; + - if all extract outputs are JSON objects with one common top-level array + field, concatenate those arrays into one JSON object under that field; + - otherwise produce a JSON array containing each extract output's decoded JSON + value in order; + - reject non-JSON media types with a clear error unless a future option + deliberately supports them. +- Update `internal/modules/normalize/noop` to pass accepted merge output through + unchanged as normalized output. +- Update chunk modules from Stage 1 as needed after interface changes. +- Remove or quarantine old candidate validators from D&D module packages if + they no longer compile. Their replacement belongs under the validation + roadmap in `internal/validators`. +- Update fake/test modules throughout the repository to the raw contracts. + +Tests to update or add: + +- D&D spells extractor returns raw JSON with schema provenance and no candidate + materialization. +- Append-order merger concatenates common top-level JSON arrays. +- Append-order merger falls back to ordered JSON value arrays when shapes differ. +- Append-order merger rejects invalid JSON and non-JSON media types clearly. +- No-op normalizer returns defensive copies of merge output bytes and metadata. +- Production catalog still registers all modules successfully. + +Completion criteria: + +- Production modules compile against raw contracts. +- D&D spell extraction no longer rejects spell candidates inside the extractor. + +## Stage 6: Output, Manifest, Diagnostics, And Files + +Update durable output around normalized lane outputs. + +Implementation tasks: + +- Update `RunOutput` to carry normalized lane outputs and rejected module-output + records instead of approved/rejected artifacts. +- Update `artifacts.RunManifest` or introduce a more accurately named manifest + package if the old artifact naming becomes misleading. Prefer the smallest + rename that keeps manifest output clear and avoids a broad unrelated refactor. +- Preserve existing manifest fields that remain meaningful: + - pipeline ID and digest; + - module keys; + - lane IDs; + - source digest; + - LLM profile provenance; + - reference provenance; + - module metadata; + - validation status. +- Add manifest/reporting fields needed for raw outputs: + - normalized lane output media type; + - output schema provenance where present; + - rejected stage/lane/chunk/module information; + - retry attempt counts. +- Update `internal/modules/output/json`: + - write `manifest.json`; + - write `index.json`; + - write `warnings.json`; + - write `rejected.json`; + - write one file per normalized lane output. +- For the JSON output encoder, accept `application/json` normalized outputs and + write them as raw JSON files. Recommended file path: + `lanes/.json`. +- Reject non-JSON normalized output media types in the JSON encoder with a clear + error. Future text/Markdown/binary encoders can support other media types. +- Keep output file name validation strict. +- Ensure diagnostics redaction still prevents raw source, prompt, reference, + schema, and payload bytes from appearing in ordinary errors or manifests. + +Tests to update or add: + +- JSON output writes lane output files and index entries deterministically. +- JSON output rejects invalid JSON bytes and unsupported media types. +- Rejected output records are written without raw payload bytes. +- Manifest includes raw-output provenance and retry attempt counts. +- Diagnostics tests still prove sensitive/large content is not emitted. + +Completion criteria: + +- Durable output no longer assumes artifact candidates. +- JSON output remains deterministic and safe. + +## Stage 7: Documentation, Examples, And Full Validation + +After behavior is implemented, update canonical current-behavior docs. + +Implementation tasks: + +- Update `docs/internal/pipeline.md` for the raw data model. +- Update `docs/internal/modules.md` for new module contracts. +- Update `docs/internal/llm.md` for merge-stage LLM/profile plumbing. +- Update `docs/config.md` for: + - integer source-unit expectations where config examples include source refs; + - merge references; + - module `retries`; + - LLM profile override scope. +- Update `docs/cli.md` for merge reference selectors and profile behavior. +- Update `docs/integrations/json-output.md` for lane output files. +- Update `docs/integrations/dnd-spell-artifacts.md`; if the artifact-specific + contract is no longer durable, rename or replace it with a D&D spell raw-output + integration doc. +- Update `docs/troubleshooting.md` for common raw-output, media-type, retry, and + validation failures. +- Update examples so Seriatim segment IDs are positive integers and expected + output shape matches lane-normalized output. +- Replace `docs/roadmap/implementation.md` with a completed-note document only + after this implementation is finished and reviewed. +- Leave `docs/roadmap/pipeline.md` as target-state roadmap context until the + feature is implemented, then move implemented behavior into canonical docs and + remove stale roadmap language. + +Validation commands: + +```sh +go test ./internal/core/source +go test ./internal/framework/contracts +go test ./internal/framework/pipeline +go test ./internal/core/config +go test ./internal/cli +go test ./internal/modules/input/seriatim +go test ./internal/modules/chunk/generic +go test ./internal/modules/chunk/dnd/scenes +go test ./internal/modules/extract/dnd/spells +go test ./internal/modules/merge/appendorder +go test ./internal/modules/normalize/noop +go test ./internal/modules/output/json +go test ./... +go vet ./... +go build ./cmd/notarius +``` + +Documentation inspection: + +```sh +rg -n "ArtifactCandidate|approved artifacts|spell artifact|start_unit_id\": \"|end_unit_id\": \"" docs examples internal +rg -n "chunk`, `extract`, and `normalize|deterministic-only stage|json.RawMessage" docs/roadmap docs/internal docs/config.md docs/cli.md +``` + +Expected result: + +- Remaining `ArtifactCandidate` mentions are only in deliberately retained + compatibility code or historical roadmap context. +- Current-behavior docs describe raw lane outputs, integer source-unit IDs, + merge LLM/reference support, and retry behavior. diff --git a/docs/roadmap/pipeline.md b/docs/roadmap/pipeline.md new file mode 100644 index 0000000..2b8aac2 --- /dev/null +++ b/docs/roadmap/pipeline.md @@ -0,0 +1,403 @@ +# Raw Pipeline Data Model + +This roadmap defines the target data model for the pipeline stages after input. +It is paired with the validation system roadmap in +[`validation.md`](validation.md): validators should evaluate module outputs, but +the pipeline first needs a raw-output handoff model that does not require every +module response shape to be represented by bespoke Go structs. + +## Goals + +- Make typed chunk envelopes and raw module-output envelopes first-class + pipeline handoffs. +- Keep framework-owned provenance around raw payloads so runs remain + deterministic, ordered, and auditable. +- Avoid requiring each extractor, merger, or normalizer response schema to have + matching Go structs. +- Let extract modules produce one raw output per input chunk. +- Let merge modules receive accepted extract outputs and merge them into one raw + payload. +- Let normalize modules optionally post-process accepted merge output. +- Let output modules write normalized output bytes directly, along with + manifests, warnings, rejected outputs, and diagnostics as appropriate. +- Preserve the fixed workflow shape: + +```text +input -> chunk -> extract -> merge -> normalize -> output +``` + +## Non-Goals + +- Do not turn the pipeline into an arbitrary DAG or general workflow language. +- Do not make validators mutate, materialize, or rewrite module outputs. +- Do not require a generic source-reference convention for every raw payload in + this pass. +- Do not eliminate typed framework data where the framework genuinely needs it, + such as input source documents and chunk boundaries. + +## Cross-Stage Runtime Plumbing + +LLM/profile/reference plumbing should be available to every stage that may need +LLM-backed or reference-aware module behavior: `chunk`, `extract`, `merge`, and +`normalize`. + +For all four stages, the framework should provide the same categories of runtime +support where the concrete module contract needs them: + +- configured LLM profile and profile override handling; +- Scriptorium client access through framework-owned LLM contracts; +- session ID propagation; +- declared reference slots and resolved reference bindings; +- module options and metadata; +- prompt, schema, profile, and reference provenance for manifests and + diagnostics. + +`merge` must not be treated as a deterministic-only stage. A merge module may be +simple and deterministic, but it may also be LLM-backed, reference-aware, and +validator-gated in the same way as `chunk`, `extract`, and `normalize`. + +## Chunk Stage Target Model + +The chunk stage receives the canonical `SourceDocument` from the input stage, +plus any original source input material needed for prompt construction. The +`SourceDocument` remains the source of truth for source identity, source-unit +ordering, source-unit IDs, and source provenance. Original raw input material +may be supplied to LLM-backed chunkers, but it should not replace the +`SourceDocument` as the framework handoff. + +Input adapters are responsible for assigning stable integer source-unit IDs. How +those IDs are assigned depends on the input format. A numbered JSON transcript +may map source units directly to transcript segment numbers; a PDF input adapter +may assign page or extracted-text unit numbers; an adapter for unordered source +material may assign deterministic IDs as part of input parsing. Downstream +framework code should treat those IDs as opaque integers owned by the input +adapter. + +The chunk module returns one or more ordered `SourceChunk` envelopes. A single +chunk representing the whole source is valid and should be supported. + +Each chunk should include framework-readable provenance and ordering metadata, +plus content suitable for extraction. The content may be JSON, plain text, +Markdown, PDF page text, or another module/input-specific representation, as +long as the framework can still associate the chunk with the source and preserve +deterministic order. + +Conceptually: + +```go +type SourceChunk struct { + ID string + SourceID string + Index int + + StartUnitID int + EndUnitID int + + Content []byte + MediaType string + + Units []source.SourceUnit + Metadata map[string]any +} +``` + +The exact implementation shape can differ, but it should preserve: + +- chunk identity; +- source identity; +- deterministic chunk order; +- source locator or range, normally integer start/end unit IDs; +- chunk content and media type; +- optional source-unit projection when useful; +- metadata needed by extractors, validators, manifests, diagnostics, and output + encoders. + +The framework owns the minimal invariants required to schedule extract work: + +- the chunker returns at least one chunk; +- chunk IDs and indexes are stable and non-empty; +- chunk source IDs match the source document; +- chunk ordering is deterministic; +- chunk provenance is sufficient to trace the chunk back to the input source. + +Domain-specific chunk acceptability belongs in validators. For example, full +coverage, no gaps, no overlap, scene metadata quality, expected media type, and +D&D scene-boundary policy should be explicit validator concerns rather than +hidden framework rules, except where a minimal invariant is required for +extraction to run safely. + +## Extract Stage Target Model + +The extract stage receives one input chunk from the chunk stage and produces one +raw extract output for that chunk. + +The extractor owns: + +- prompt selection; +- input material assembly; +- Scriptorium request construction; +- response-schema selection; +- LLM profile and session usage; +- returned raw payload metadata. + +The framework owns: + +- chunk iteration; +- deterministic ordering; +- association between each extract output and its input chunk; +- execution errors when an extractor does not return output; +- handoff of returned output to validation and later stages. + +An extract output should be an envelope, not just raw bytes. Conceptually: + +```go +type ExtractOutput struct { + LaneID string + ExtractorKey string + + ChunkID string + ChunkIndex int + SourceID string + + RawContent []byte + MediaType string + + SchemaID string + SchemaName string + SchemaVersion string + + Metadata map[string]any + Warnings []contracts.Warning +} +``` + +The exact implementation shape can differ, but it should preserve: + +- raw returned content; +- chunk provenance; +- chunk order; +- source identity; +- module identity; +- response schema provenance; +- metadata needed by validators, mergers, manifests, diagnostics, and output + encoders. + +Extractor modules should not be required to convert raw LLM output into +`ArtifactCandidate` values. Domain-specific Go projection may still exist for +specific modules when it is useful, but it should not be the generic pipeline +contract. + +Rejected extract outputs are not passed to merge. This keeps downstream +contracts simple and makes rejection behavior explicit. Support for passing +rejected outputs forward as marked data is deferred future work. + +## Merge Stage Target Model + +The merge stage receives the accepted extract outputs for a lane. Each extract +output corresponds to one input chunk and carries enough metadata to recover the +original chunk order. + +The merger owns: + +- merge/reconciliation strategy; +- deterministic or LLM-backed merge logic; +- prompt and schema usage when LLM-backed; +- the shape of merged raw output. + +The framework owns: + +- passing the ordered accepted extract-output set to the merger; +- lane identity; +- source context; +- references; +- runtime LLM plumbing; +- validation handoff for merged output; +- manifest provenance. + +Simple merge modules may concatenate raw extract outputs in chunk order. Other +merge modules may deterministically merge JSON documents, reconcile duplicates, +or use an LLM to produce a more coherent merged result. + +Conceptually: + +```go +type MergeRequest struct { + LaneID string + Source *source.SourceDocument + ExtractOutputs []ExtractOutput + + SourceInput contracts.LLMInputMaterial + SessionID string + References contracts.ReferenceSet + LLMClient contracts.StructuredLLMClient + LLMProfile string + Options map[string]any + Metadata map[string]any +} + +type MergeOutput struct { + LaneID string + MergerKey string + + RawContent []byte + MediaType string + + SchemaID string + SchemaName string + SchemaVersion string + + Metadata map[string]any + Warnings []contracts.Warning +} +``` + +The exact implementation shape can differ, but the key contract is that +merge consumes ordered accepted extract outputs and produces one merged raw +output for the lane. + +If no accepted extract outputs remain for a lane, the framework should not pass +rejected outputs to the merger. The lane should produce no merge output and +should be reported as rejected or omitted according to run reporting policy. + +## Normalize Stage Target Model + +The normalize stage receives accepted merge output for a lane and optionally +post-processes it into the final normalized raw payload for that lane. +Normalization may be a no-op, deterministic cleanup, schema conversion, or an +LLM-backed post-processing pass. + +The normalizer owns: + +- post-merge processing strategy; +- deterministic or LLM-backed normalization logic; +- prompt and schema usage when LLM-backed; +- the shape of normalized raw output. + +The framework owns: + +- passing accepted merge output to the normalizer; +- lane identity; +- source context; +- references; +- runtime LLM plumbing; +- validation handoff for normalized output; +- manifest provenance. + +Conceptually: + +```go +type NormalizeRequest struct { + LaneID string + Source *source.SourceDocument + MergeOutput MergeOutput + + SourceInput contracts.LLMInputMaterial + SessionID string + References contracts.ReferenceSet + LLMClient contracts.StructuredLLMClient + LLMProfile string + Options map[string]any + Metadata map[string]any +} + +type NormalizeOutput struct { + LaneID string + NormalizerKey string + + RawContent []byte + MediaType string + + SchemaID string + SchemaName string + SchemaVersion string + + Metadata map[string]any + Warnings []contracts.Warning +} +``` + +The exact implementation shape can differ, but the key contract is that +normalize consumes one accepted merged output and produces one normalized raw +output for the lane. + +## Output Stage Target Model + +The output stage receives validated normalized output bytes and writes them to +its configured destination. + +Output modules should not require normalized output to be converted into +`Artifact` values. Output modules should use the normalized output media type to +decide how to serialize or wrap the payload. A JSON output module can write raw +normalized JSON directly, while a text or Markdown output module can write text +payloads directly. Output modules may also write run manifests, warnings, +rejected outputs, and indexes. + +Output modules may still choose to provide convenience layouts, grouping, or file +naming conventions, but those should be output concerns rather than constraints +on extractor or normalizer response schemas. + +## Validation Relationship + +Validation chains should attach to returned module outputs, not to hidden +module-internal conversions. + +For chunk: + +```text +chunk(input) -> SourceChunk set -> chunk validators -> extract input +``` + +For extract: + +```text +extract(chunk) -> raw ExtractOutput -> extract validators -> merge input +``` + +For merge: + +```text +merge(extract outputs) -> raw MergeOutput -> merge validators -> normalize input +``` + +For normalize: + +```text +normalize(merge output) -> raw NormalizeOutput -> normalize validators -> output +``` + +Validators must be read-only. They inspect raw output and metadata, return +accept/reject decisions and warnings, and do not rewrite output. + +An empty validator chain approves returned output for that validation point. +Framework/runtime errors remain separate from validation rejections: if a module +or Scriptorium call fails before output is returned, the pipeline reports an +execution error rather than asking validators to evaluate nonexistent output. + +Rejected outputs do not pass to the next stage. An empty validator chain still +approves returned output for that validation point. + +## Retry Policy + +A retry means re-running the same module with the same input after that module +fails to produce valid output. Failure to produce valid output includes both: + +- framework-level execution errors, such as module errors, Scriptorium errors, + provider errors, or missing returned output; +- validator rejection of returned output. + +Retries should be configurable at the pipeline or lane level. Chunk retries are +per source input. Extract retries are per chunk. Merge and normalize retries, +when configured, are per lane. Retry attempts should preserve deterministic +reporting: the final accepted or rejected output should record attempt count and +enough diagnostics/provenance to understand prior failures without leaking +secrets or large payloads by default. + +## Relationship To Validation Roadmap + +This roadmap defines the data model that validation should evaluate. The +validation system roadmap in [`validation.md`](validation.md) defines validator +registration, mapping, execution classes, and concrete validator behavior. + +The shared boundary between the roadmaps is a returned module output: validators +inspect typed chunk output or raw module-output envelopes and decide whether +that output may continue through the pipeline. diff --git a/docs/roadmap/validation.md b/docs/roadmap/validation.md new file mode 100644 index 0000000..a27b653 --- /dev/null +++ b/docs/roadmap/validation.md @@ -0,0 +1,420 @@ +# Validation System Refactor + +This roadmap defines the target state for making validation a first-class, +composable pipeline concern. Current validation behavior is partly module-owned: +the `dnd/spells` extractor defines built-in validators inside the module package, +and the runner falls back to extractor-provided validators when a lane does not +configure validators. The desired end state is that validator implementations, +validator registration, and default module-to-validator mappings are explicit, +reviewable, and independent of concrete module packages. + +## Goals + +- Move artifact and module-output validation behavior out of `internal/modules` + and into `internal/validators`. +- Keep each validator in its own package. +- Mirror the stage and domain shape of `internal/modules` where a validator is + module-specific. +- Support deterministic and LLM-backed validators through the same framework + contract. +- Allow validators to be mapped to modules at any pipeline stage that returns + module output for validation: `chunk`, `extract`, `merge`, or `normalize`. +- Make default production module-to-validator mappings centralized and + human-readable. +- Allow pipeline configuration to override default mappings for advanced use. +- Treat an empty validator set as valid and equivalent to approval. +- Preserve the rule that module output passes forward unless a validator rejects + it. +- Make successfully returned module output the explicit validation boundary: + questions about output syntax, media type, schema conformance, and domain + acceptability should be answered by validators. + +## Non-Goals + +- Do not create a general workflow engine or arbitrary validation DAG. +- Do not enforce validator compatibility with a module or stage in this pass. +- Do not move ordinary runtime invariant checks into validator packages. +- Do not require every module to have validators. +- Do not require LLM-backed validators for modules that can be checked + deterministically. +- Do not silently reorder configured validator chains unless that behavior is + introduced deliberately and documented as part of the validator contract. + +## Validation Boundary + +Validation packages should own approve/reject/warning evaluation of successfully +returned module outputs. This means logic that decides whether a chunk result, +raw extract output, raw merge output, raw normalize output, or raw LLM response +should continue through the pipeline belongs in `internal/validators`. + +The boundary is: + +- no module output was returned: execution failed, and the pipeline should report + a module or runtime error; +- module output was returned: the validator chain decides whether that output is + acceptable, and an empty validator chain approves it. + +Scriptorium and provider errors are execution failures rather than validator +rejections. This includes provider timeouts, authentication failures, +transport/runtime failures, Scriptorium structured-output retry exhaustion, and +malformed responses that Scriptorium rejects before returning module output. + +Other validation-like checks should remain with their owning packages: + +- input parsing and source-format validation stay in input modules; +- source document and source reference invariants stay in `internal/core/source`; +- config validation stays in `internal/core/config`; +- registry, profile, and pipeline consistency checks stay in framework and CLI + code; +- response schema loading stays in module asset code; +- Scriptorium runtime errors stay in LLM runtime code. + +Domain validators may call reusable core helpers such as `source.ValidateRef`, +but the module-output approval or rejection decision should be made by a +validator. + +Validators should answer module-output questions such as: + +- is returned content syntactically valid JSON; +- does returned JSON conform to the module's declared schema; +- are required domain fields present and non-empty; +- are source references valid and appropriately grounded; +- does domain-specific output satisfy the configured policy. + +## Audita Patterns To Adapt + +The validator architecture should adapt useful patterns from +[`audita`](https://gitea.maximumdirect.net/eric/audita) without copying its +narrower transcript-correction shape directly. + +Useful patterns: + +- concrete validators live under `internal/validators`; +- shared validator runtime mechanics live under a framework package; +- built-in validator keys are stable and centrally registered; +- built-in chains are centrally reviewable; +- validators carry execution-class metadata; +- deterministic and LLM-backed validators implement one contract; +- LLM-backed validator runtime can share batching, diagnostics, structured + response handling, and malformed-response policy; +- reports and manifests can classify validator decisions by execution class. + +Important Notarius differences: + +- mappings must be keyed by stage and module key, not module key alone; +- mappings should be owned by the central production catalog, not resolved inside + concrete module constructors; +- configured mapping order should be authoritative unless the config explicitly + opts into a different ordering policy; +- validators must support chunk, extract, merge, and normalize outputs rather + than only one proposal shape. + +## Validator Package Layout + +Concrete validators should live under `internal/validators`. Module-specific +validators should mirror the module tree and use one package per validator: + +```text +internal/validators/extract/dnd/spells/shape +internal/validators/extract/dnd/spells/source_refs +internal/validators/extract/dnd/spells/source_relatedness +``` + +Generic validators may live under stage-specific generic paths when they operate +on a particular stage output shape: + +```text +internal/validators/chunk/generic/... +internal/validators/extract/generic/... +internal/validators/merge/generic/... +internal/validators/normalize/generic/... +``` + +Truly stage-independent validators may live under `internal/validators/generic` +once there is a real shared validator that justifies that location. Generic JSON +syntax and JSON schema validators are likely candidates for +`internal/validators/generic/valid_json` and +`internal/validators/generic/valid_json_schema`. + +Each validator package should expose: + +- a stable validator key; +- execution-class metadata; +- a constructor; +- a validator spec suitable for registration; +- a `Register` function; +- focused tests for decisions, warnings, errors, and diagnostics behavior. + +Reusable validator runtime mechanics should live in framework code, such as +`internal/framework/validators`, not in concrete validator packages. This package +can own shared helpers for decision cardinality, approval/rejection construction, +LLM validator batching, validator diagnostics, and Scriptorium request plumbing. + +The concrete validator packages should own policy: what they inspect, what they +approve or reject, what warning reason codes they emit, and how they interpret +domain-specific data. + +## Validator Design Policy + +Validators should follow a small-tool model: each validator should do one thing +well. If a validator both rejects output and emits unrelated warnings, split +those concerns into separate validators so production mappings can include, +exclude, and order them independently. + +Validators are read-only. A validator must not mutate pipeline state, rewrite +module output, materialize raw output into typed stage output, or enrich the +`ModuleOutput` passed to later validators. A validator returns an +accept/reject verdict for the output it evaluates, plus any warnings or +diagnostic references. Any conversion from raw module output into a downstream +representation is a separate materialization concern and must not be hidden +inside a validator. + +The initial generic validator set should include: + +- `generic/always_accept`: accepts returned module output unchanged. This is + functionally equivalent to a no-op validator and is primarily useful for tests, + demonstrations, and explicit pass-through configurations. +- `generic/always_reject`: rejects returned module output without inspecting it. + This is primarily useful for tests and for proving rejection plumbing, + manifests, and diagnostics. +- `generic/valid_json`: inspects raw returned module output and accepts only + syntactically valid JSON. +- `generic/valid_json_schema`: compares raw returned JSON with the module's + configured response schema and accepts only schema-conformant output. + +The exact keys may be adjusted during implementation to match local naming +conventions, but the validator set should preserve these four behaviors. + +For the current D&D spell behavior, the target split is: + +- `generic/valid_json`: rejects returned module output that is not syntactically + valid JSON. +- `generic/valid_json_schema`: rejects returned JSON that does not conform to + the configured response schema. +- `extract/dnd/spells/shape`: rejects malformed spell-cast payloads and missing + required spell fields. +- `extract/dnd/spells/source_refs`: rejects missing or invalid source + references. +- `extract/dnd/spells/source_relatedness`: warning-only validator that reports + when a spell name is not found in the cited source text. + +Production default mappings should generally list deterministic validators +before LLM-backed validators. This keeps cheap structural failures from consuming +model calls and keeps diagnostics easier to interpret. Pipeline-configured order +should still be authoritative; if a user explicitly lists an LLM-backed +validator before a deterministic validator, the framework should honor that +order rather than silently reshuffling it. + +## Execution Classes + +Validator specs should declare an execution class: + +```go +type ExecutionClass string + +const ( + ExecutionClassDeterministic ExecutionClass = "deterministic" + ExecutionClassLLMBacked ExecutionClass = "llm_backed" +) +``` + +Execution class should be metadata on the validator spec or registered +definition, not an ad hoc convention inferred from package paths. It should be +used for: + +- human-readable catalog and manifest reporting; +- diagnostics and timing summaries; +- operational policy such as concurrency budgeting for LLM-backed validators; +- default mapping review, where deterministic validators should usually appear + before LLM-backed validators. + +Execution class should not by itself imply compatibility with a stage or module. + +## Validator Contract + +The validator framework should support validation of outputs from `chunk`, +`extract`, `merge`, and `normalize` stages. The contract should be generalized +enough for stage-specific validators to inspect the output they care about while +ignoring irrelevant fields. + +The request should carry: + +- stage name; +- module key; +- raw module output content when available; +- response schema metadata when the module declares one; +- source document; +- source input material; +- session ID; +- references; +- LLM client and profile for LLM-backed validators; +- options and metadata; +- chunk output when validating a chunk module; +- stage-specific typed envelopes when the stage owns them, such as chunk + envelopes for chunk validation. + +A shared `ModuleOutput` envelope should represent the validation boundary. +Validators may inspect raw returned content and any already-existing typed +stage output, but they must not modify it. + +Conceptually: + +```go +type ModuleOutput struct { + Stage pipeline.Stage + ModuleKey string + + RawContent []byte + MediaType string + ResponseSchema *llm.ResponseSchemaMetadata + + Chunks []contracts.SourceChunk + Warnings []contracts.Warning +} +``` + +The final implementation does not need to use this exact shape, but it should +preserve the boundary: returned raw module output can enter validation before +any separate materialization step converts it into a stage-specific typed +representation. + +The result should continue to express validator identity, warnings, and explicit +decisions. For output collections, the implementation should define an explicit +decision shape rather than silently mutating lists. Validator decisions reject or +approve output; validators do not rewrite output. + +An empty validator list is always valid. With no validators, the framework should +pass module output forward unchanged and treat the output as approved for that +validation point. If the approved output cannot be consumed by a later stage +because its media type or envelope shape is unsuitable, that failure should be +reported at the downstream boundary that requires a different shape, not as an +implicit pre-validation rejection. + +## Module Development Workflow + +The validation system should make iterative module development easier. A module +author should be able to start with an explicit empty validator mapping and +inspect returned raw LLM output without first satisfying JSON syntax, schema, +media-type, or domain validators. + +A typical development path should be: + +1. Configure an empty validator set for the module and inspect raw returned + output. +2. Add `generic/valid_json` and adjust prompts until the model reliably returns + syntactically valid JSON. +3. Add `generic/valid_json_schema` and iterate on prompt/schema alignment. +4. Add media-type or schema validators appropriate to the module's intended + output format. +5. Add domain-specific validators one at a time until production policy is + represented explicitly in the chain. + +This workflow is a central reason for making output validation explicit and +composable rather than hiding schema, shape, or domain checks inside module +implementation code. + +## Central Production Mappings + +Production defaults should be defined in a central, human-readable location near +the production module and validator registries. The mapping should be keyed by +stage and module key, not only by module key, so future modules can share keys +only when stage context makes their ownership unambiguous. + +Conceptually: + +```go +{ + Stage: pipeline.StageExtract, + Module: "dnd/spells", + Validators: []pipeline.ModuleBinding{ + pipeline.Binding("generic/valid_json"), + pipeline.Binding("generic/valid_json_schema"), + pipeline.Binding("extract/dnd/spells/shape"), + pipeline.Binding("extract/dnd/spells/source_refs"), + pipeline.Binding("extract/dnd/spells/source_relatedness"), + }, +} +``` + +The production catalog should expose three related surfaces together: + +- available modules; +- available validators; +- default module-to-validator mappings. + +This makes production validation policy reviewable without constructing concrete +modules or searching inside module implementation packages. + +The validator registry should expose registered validator specs without building +validators, including key and execution class. Building a validator should still +be available for runtime execution. + +## Pipeline Overrides + +Pipeline configuration should be able to override the central default mapping +for a module binding. Override semantics should distinguish three states: + +- unset validators: use the central production/default mapping; +- explicit empty validators: run no validators and pass output forward; +- explicit non-empty validators: run exactly the configured validators in the + configured order. + +This keeps the happy path concise while preserving advanced control for +experimentation, debugging, and custom deployments. + +The run manifest should record the resolved validator chain for each validation +point so completed runs remain auditable after defaults or configuration change. +Each manifest entry should include at least validator key and execution class, +and should preserve the resolved order actually used for the run. + +## LLM-Backed Validator Runtime + +LLM-backed validators should use the same validator contract as deterministic +validators. Shared framework runtime should provide common support for: + +- Scriptorium request construction; +- validator prompt and schema provenance; +- diagnostics redaction; +- optional batching or context-window controls when validator inputs are large; +- mapping successful LLM validator responses into validator decisions and + warnings; +- consistent handling of Scriptorium/runtime errors. + +Scriptorium errors raised during validator execution should be treated as +validator execution errors unless a specific validator deliberately converts a +successful response into reject/warn decisions. This keeps provider/runtime +failure distinct from a validator's semantic rejection of module output. + +## Stage Coverage + +Validators should be composable across all LLM-eligible stages: + +- `chunk`: validators can evaluate chunk boundaries, coverage, overlap, metadata, + or module-specific chunk quality. +- `extract`: validators can evaluate raw extracted output, source references, + payload shape, evidence quality, media type, or domain constraints. +- `merge`: validators can evaluate merged output, cross-chunk consistency, + deduplication results, media type, or domain-specific reconciliation. +- `normalize`: validators can evaluate normalized output, final shape, + post-processing results, media type, or domain-specific policy. + +The framework should not require compatibility declarations in this pass. A +validator mapped to an unsuitable output shape should return a clear error, or +approve unchanged only when that is explicitly the validator's documented +behavior. + +## Documentation Impact + +When implemented, current-behavior docs and policy should be updated together: + +- `docs/policy/architecture.md` should describe centralized validator mappings + rather than module-owned validator chains. +- `docs/internal/modules.md` should remove claims that concrete modules own + validator defaults. +- Internal validation docs should describe validator package ownership, mapping + precedence, empty-chain approval behavior, and LLM-backed validator support. +- User/config docs should describe how pipeline validator overrides work once the + syntax is implemented. + +Roadmap docs should not remain the canonical description of implemented +validation behavior after the refactor is complete.