eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document raw pipeline completion

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-07 19:32:15 +00:00
parent 7c95791e94
commit a9d8505cdb
7 changed files with 91 additions and 822 deletions
Download Patch File Download Diff File Expand all files Collapse all files

405
docs/roadmap/pipeline.md
View File

@@ -1,403 +1,12 @@
# Raw Pipeline Data Model
This roadmap defines the target data model for the pipeline stages after input.
It is paired with the validation system roadmap in
[`validation.md`](validation.md): validators should evaluate module outputs, but
the pipeline first needs a raw-output handoff model that does not require every
module response shape to be represented by bespoke Go structs.
The raw pipeline data model has been implemented.
## Goals
Current behavior is documented in:
- Make typed chunk envelopes and raw module-output envelopes first-class
pipeline handoffs.
- Keep framework-owned provenance around raw payloads so runs remain
deterministic, ordered, and auditable.
- Avoid requiring each extractor, merger, or normalizer response schema to have
matching Go structs.
- Let extract modules produce one raw output per input chunk.
- Let merge modules receive accepted extract outputs and merge them into one raw
payload.
- Let normalize modules optionally post-process accepted merge output.
- Let output modules write normalized output bytes directly, along with
manifests, warnings, rejected outputs, and diagnostics as appropriate.
- Preserve the fixed workflow shape:
- [Pipeline Internals](../internal/pipeline.md)
- [Modules](../internal/modules.md)
- [JSON Output](../integrations/json-output.md)
```text
input -> chunk -> extract -> merge -> normalize -> output
```
## Non-Goals
- Do not turn the pipeline into an arbitrary DAG or general workflow language.
- Do not make validators mutate, materialize, or rewrite module outputs.
- Do not require a generic source-reference convention for every raw payload in
this pass.
- Do not eliminate typed framework data where the framework genuinely needs it,
such as input source documents and chunk boundaries.
## Cross-Stage Runtime Plumbing
LLM/profile/reference plumbing should be available to every stage that may need
LLM-backed or reference-aware module behavior: `chunk`, `extract`, `merge`, and
`normalize`.
For all four stages, the framework should provide the same categories of runtime
support where the concrete module contract needs them:
- configured LLM profile and profile override handling;
- Scriptorium client access through framework-owned LLM contracts;
- session ID propagation;
- declared reference slots and resolved reference bindings;
- module options and metadata;
- prompt, schema, profile, and reference provenance for manifests and
diagnostics.
`merge` must not be treated as a deterministic-only stage. A merge module may be
simple and deterministic, but it may also be LLM-backed, reference-aware, and
validator-gated in the same way as `chunk`, `extract`, and `normalize`.
## Chunk Stage Target Model
The chunk stage receives the canonical `SourceDocument` from the input stage,
plus any original source input material needed for prompt construction. The
`SourceDocument` remains the source of truth for source identity, source-unit
ordering, source-unit IDs, and source provenance. Original raw input material
may be supplied to LLM-backed chunkers, but it should not replace the
`SourceDocument` as the framework handoff.
Input adapters are responsible for assigning stable integer source-unit IDs. How
those IDs are assigned depends on the input format. A numbered JSON transcript
may map source units directly to transcript segment numbers; a PDF input adapter
may assign page or extracted-text unit numbers; an adapter for unordered source
material may assign deterministic IDs as part of input parsing. Downstream
framework code should treat those IDs as opaque integers owned by the input
adapter.
The chunk module returns one or more ordered `SourceChunk` envelopes. A single
chunk representing the whole source is valid and should be supported.
Each chunk should include framework-readable provenance and ordering metadata,
plus content suitable for extraction. The content may be JSON, plain text,
Markdown, PDF page text, or another module/input-specific representation, as
long as the framework can still associate the chunk with the source and preserve
deterministic order.
Conceptually:
```go
type SourceChunk struct {
ID string
SourceID string
Index int
StartUnitID int
EndUnitID int
Content []byte
MediaType string
Units []source.SourceUnit
Metadata map[string]any
}
```
The exact implementation shape can differ, but it should preserve:
- chunk identity;
- source identity;
- deterministic chunk order;
- source locator or range, normally integer start/end unit IDs;
- chunk content and media type;
- optional source-unit projection when useful;
- metadata needed by extractors, validators, manifests, diagnostics, and output
encoders.
The framework owns the minimal invariants required to schedule extract work:
- the chunker returns at least one chunk;
- chunk IDs and indexes are stable and non-empty;
- chunk source IDs match the source document;
- chunk ordering is deterministic;
- chunk provenance is sufficient to trace the chunk back to the input source.
Domain-specific chunk acceptability belongs in validators. For example, full
coverage, no gaps, no overlap, scene metadata quality, expected media type, and
D&D scene-boundary policy should be explicit validator concerns rather than
hidden framework rules, except where a minimal invariant is required for
extraction to run safely.
## Extract Stage Target Model
The extract stage receives one input chunk from the chunk stage and produces one
raw extract output for that chunk.
The extractor owns:
- prompt selection;
- input material assembly;
- Scriptorium request construction;
- response-schema selection;
- LLM profile and session usage;
- returned raw payload metadata.
The framework owns:
- chunk iteration;
- deterministic ordering;
- association between each extract output and its input chunk;
- execution errors when an extractor does not return output;
- handoff of returned output to validation and later stages.
An extract output should be an envelope, not just raw bytes. Conceptually:
```go
type ExtractOutput struct {
LaneID string
ExtractorKey string
ChunkID string
ChunkIndex int
SourceID string
RawContent []byte
MediaType string
SchemaID string
SchemaName string
SchemaVersion string
Metadata map[string]any
Warnings []contracts.Warning
}
```
The exact implementation shape can differ, but it should preserve:
- raw returned content;
- chunk provenance;
- chunk order;
- source identity;
- module identity;
- response schema provenance;
- metadata needed by validators, mergers, manifests, diagnostics, and output
encoders.
Extractor modules should not be required to convert raw LLM output into
`ArtifactCandidate` values. Domain-specific Go projection may still exist for
specific modules when it is useful, but it should not be the generic pipeline
contract.
Rejected extract outputs are not passed to merge. This keeps downstream
contracts simple and makes rejection behavior explicit. Support for passing
rejected outputs forward as marked data is deferred future work.
## Merge Stage Target Model
The merge stage receives the accepted extract outputs for a lane. Each extract
output corresponds to one input chunk and carries enough metadata to recover the
original chunk order.
The merger owns:
- merge/reconciliation strategy;
- deterministic or LLM-backed merge logic;
- prompt and schema usage when LLM-backed;
- the shape of merged raw output.
The framework owns:
- passing the ordered accepted extract-output set to the merger;
- lane identity;
- source context;
- references;
- runtime LLM plumbing;
- validation handoff for merged output;
- manifest provenance.
Simple merge modules may concatenate raw extract outputs in chunk order. Other
merge modules may deterministically merge JSON documents, reconcile duplicates,
or use an LLM to produce a more coherent merged result.
Conceptually:
```go
type MergeRequest struct {
LaneID string
Source *source.SourceDocument
ExtractOutputs []ExtractOutput
SourceInput contracts.LLMInputMaterial
SessionID string
References contracts.ReferenceSet
LLMClient contracts.StructuredLLMClient
LLMProfile string
Options map[string]any
Metadata map[string]any
}
type MergeOutput struct {
LaneID string
MergerKey string
RawContent []byte
MediaType string
SchemaID string
SchemaName string
SchemaVersion string
Metadata map[string]any
Warnings []contracts.Warning
}
```
The exact implementation shape can differ, but the key contract is that
merge consumes ordered accepted extract outputs and produces one merged raw
output for the lane.
If no accepted extract outputs remain for a lane, the framework should not pass
rejected outputs to the merger. The lane should produce no merge output and
should be reported as rejected or omitted according to run reporting policy.
## Normalize Stage Target Model
The normalize stage receives accepted merge output for a lane and optionally
post-processes it into the final normalized raw payload for that lane.
Normalization may be a no-op, deterministic cleanup, schema conversion, or an
LLM-backed post-processing pass.
The normalizer owns:
- post-merge processing strategy;
- deterministic or LLM-backed normalization logic;
- prompt and schema usage when LLM-backed;
- the shape of normalized raw output.
The framework owns:
- passing accepted merge output to the normalizer;
- lane identity;
- source context;
- references;
- runtime LLM plumbing;
- validation handoff for normalized output;
- manifest provenance.
Conceptually:
```go
type NormalizeRequest struct {
LaneID string
Source *source.SourceDocument
MergeOutput MergeOutput
SourceInput contracts.LLMInputMaterial
SessionID string
References contracts.ReferenceSet
LLMClient contracts.StructuredLLMClient
LLMProfile string
Options map[string]any
Metadata map[string]any
}
type NormalizeOutput struct {
LaneID string
NormalizerKey string
RawContent []byte
MediaType string
SchemaID string
SchemaName string
SchemaVersion string
Metadata map[string]any
Warnings []contracts.Warning
}
```
The exact implementation shape can differ, but the key contract is that
normalize consumes one accepted merged output and produces one normalized raw
output for the lane.
## Output Stage Target Model
The output stage receives validated normalized output bytes and writes them to
its configured destination.
Output modules should not require normalized output to be converted into
`Artifact` values. Output modules should use the normalized output media type to
decide how to serialize or wrap the payload. A JSON output module can write raw
normalized JSON directly, while a text or Markdown output module can write text
payloads directly. Output modules may also write run manifests, warnings,
rejected outputs, and indexes.
Output modules may still choose to provide convenience layouts, grouping, or file
naming conventions, but those should be output concerns rather than constraints
on extractor or normalizer response schemas.
## Validation Relationship
Validation chains should attach to returned module outputs, not to hidden
module-internal conversions.
For chunk:
```text
chunk(input) -> SourceChunk set -> chunk validators -> extract input
```
For extract:
```text
extract(chunk) -> raw ExtractOutput -> extract validators -> merge input
```
For merge:
```text
merge(extract outputs) -> raw MergeOutput -> merge validators -> normalize input
```
For normalize:
```text
normalize(merge output) -> raw NormalizeOutput -> normalize validators -> output
```
Validators must be read-only. They inspect raw output and metadata, return
accept/reject decisions and warnings, and do not rewrite output.
An empty validator chain approves returned output for that validation point.
Framework/runtime errors remain separate from validation rejections: if a module
or Scriptorium call fails before output is returned, the pipeline reports an
execution error rather than asking validators to evaluate nonexistent output.
Rejected outputs do not pass to the next stage. An empty validator chain still
approves returned output for that validation point.
## Retry Policy
A retry means re-running the same module with the same input after that module
fails to produce valid output. Failure to produce valid output includes both:
- framework-level execution errors, such as module errors, Scriptorium errors,
provider errors, or missing returned output;
- validator rejection of returned output.
Retries should be configurable at the pipeline or lane level. Chunk retries are
per source input. Extract retries are per chunk. Merge and normalize retries,
when configured, are per lane. Retry attempts should preserve deterministic
reporting: the final accepted or rejected output should record attempt count and
enough diagnostics/provenance to understand prior failures without leaking
secrets or large payloads by default.
## Relationship To Validation Roadmap
This roadmap defines the data model that validation should evaluate. The
validation system roadmap in [`validation.md`](validation.md) defines validator
registration, mapping, execution classes, and concrete validator behavior.
The shared boundary between the roadmaps is a returned module output: validators
inspect typed chunk output or raw module-output envelopes and decide whether
that output may continue through the pipeline.
Validator mapping and concrete validator behavior remain tracked separately in
[Validation Roadmap](validation.md).