404 lines
13 KiB
Markdown
404 lines
13 KiB
Markdown
# Raw Pipeline Data Model
|
|
|
|
This roadmap defines the target data model for the pipeline stages after input.
|
|
It is paired with the validation system roadmap in
|
|
[`validation.md`](validation.md): validators should evaluate module outputs, but
|
|
the pipeline first needs a raw-output handoff model that does not require every
|
|
module response shape to be represented by bespoke Go structs.
|
|
|
|
## Goals
|
|
|
|
- Make typed chunk envelopes and raw module-output envelopes first-class
|
|
pipeline handoffs.
|
|
- Keep framework-owned provenance around raw payloads so runs remain
|
|
deterministic, ordered, and auditable.
|
|
- Avoid requiring each extractor, merger, or normalizer response schema to have
|
|
matching Go structs.
|
|
- Let extract modules produce one raw output per input chunk.
|
|
- Let merge modules receive accepted extract outputs and merge them into one raw
|
|
payload.
|
|
- Let normalize modules optionally post-process accepted merge output.
|
|
- Let output modules write normalized output bytes directly, along with
|
|
manifests, warnings, rejected outputs, and diagnostics as appropriate.
|
|
- Preserve the fixed workflow shape:
|
|
|
|
```text
|
|
input -> chunk -> extract -> merge -> normalize -> output
|
|
```
|
|
|
|
## Non-Goals
|
|
|
|
- Do not turn the pipeline into an arbitrary DAG or general workflow language.
|
|
- Do not make validators mutate, materialize, or rewrite module outputs.
|
|
- Do not require a generic source-reference convention for every raw payload in
|
|
this pass.
|
|
- Do not eliminate typed framework data where the framework genuinely needs it,
|
|
such as input source documents and chunk boundaries.
|
|
|
|
## Cross-Stage Runtime Plumbing
|
|
|
|
LLM/profile/reference plumbing should be available to every stage that may need
|
|
LLM-backed or reference-aware module behavior: `chunk`, `extract`, `merge`, and
|
|
`normalize`.
|
|
|
|
For all four stages, the framework should provide the same categories of runtime
|
|
support where the concrete module contract needs them:
|
|
|
|
- configured LLM profile and profile override handling;
|
|
- Scriptorium client access through framework-owned LLM contracts;
|
|
- session ID propagation;
|
|
- declared reference slots and resolved reference bindings;
|
|
- module options and metadata;
|
|
- prompt, schema, profile, and reference provenance for manifests and
|
|
diagnostics.
|
|
|
|
`merge` must not be treated as a deterministic-only stage. A merge module may be
|
|
simple and deterministic, but it may also be LLM-backed, reference-aware, and
|
|
validator-gated in the same way as `chunk`, `extract`, and `normalize`.
|
|
|
|
## Chunk Stage Target Model
|
|
|
|
The chunk stage receives the canonical `SourceDocument` from the input stage,
|
|
plus any original source input material needed for prompt construction. The
|
|
`SourceDocument` remains the source of truth for source identity, source-unit
|
|
ordering, source-unit IDs, and source provenance. Original raw input material
|
|
may be supplied to LLM-backed chunkers, but it should not replace the
|
|
`SourceDocument` as the framework handoff.
|
|
|
|
Input adapters are responsible for assigning stable integer source-unit IDs. How
|
|
those IDs are assigned depends on the input format. A numbered JSON transcript
|
|
may map source units directly to transcript segment numbers; a PDF input adapter
|
|
may assign page or extracted-text unit numbers; an adapter for unordered source
|
|
material may assign deterministic IDs as part of input parsing. Downstream
|
|
framework code should treat those IDs as opaque integers owned by the input
|
|
adapter.
|
|
|
|
The chunk module returns one or more ordered `SourceChunk` envelopes. A single
|
|
chunk representing the whole source is valid and should be supported.
|
|
|
|
Each chunk should include framework-readable provenance and ordering metadata,
|
|
plus content suitable for extraction. The content may be JSON, plain text,
|
|
Markdown, PDF page text, or another module/input-specific representation, as
|
|
long as the framework can still associate the chunk with the source and preserve
|
|
deterministic order.
|
|
|
|
Conceptually:
|
|
|
|
```go
|
|
type SourceChunk struct {
|
|
ID string
|
|
SourceID string
|
|
Index int
|
|
|
|
StartUnitID int
|
|
EndUnitID int
|
|
|
|
Content []byte
|
|
MediaType string
|
|
|
|
Units []source.SourceUnit
|
|
Metadata map[string]any
|
|
}
|
|
```
|
|
|
|
The exact implementation shape can differ, but it should preserve:
|
|
|
|
- chunk identity;
|
|
- source identity;
|
|
- deterministic chunk order;
|
|
- source locator or range, normally integer start/end unit IDs;
|
|
- chunk content and media type;
|
|
- optional source-unit projection when useful;
|
|
- metadata needed by extractors, validators, manifests, diagnostics, and output
|
|
encoders.
|
|
|
|
The framework owns the minimal invariants required to schedule extract work:
|
|
|
|
- the chunker returns at least one chunk;
|
|
- chunk IDs and indexes are stable and non-empty;
|
|
- chunk source IDs match the source document;
|
|
- chunk ordering is deterministic;
|
|
- chunk provenance is sufficient to trace the chunk back to the input source.
|
|
|
|
Domain-specific chunk acceptability belongs in validators. For example, full
|
|
coverage, no gaps, no overlap, scene metadata quality, expected media type, and
|
|
D&D scene-boundary policy should be explicit validator concerns rather than
|
|
hidden framework rules, except where a minimal invariant is required for
|
|
extraction to run safely.
|
|
|
|
## Extract Stage Target Model
|
|
|
|
The extract stage receives one input chunk from the chunk stage and produces one
|
|
raw extract output for that chunk.
|
|
|
|
The extractor owns:
|
|
|
|
- prompt selection;
|
|
- input material assembly;
|
|
- Scriptorium request construction;
|
|
- response-schema selection;
|
|
- LLM profile and session usage;
|
|
- returned raw payload metadata.
|
|
|
|
The framework owns:
|
|
|
|
- chunk iteration;
|
|
- deterministic ordering;
|
|
- association between each extract output and its input chunk;
|
|
- execution errors when an extractor does not return output;
|
|
- handoff of returned output to validation and later stages.
|
|
|
|
An extract output should be an envelope, not just raw bytes. Conceptually:
|
|
|
|
```go
|
|
type ExtractOutput struct {
|
|
LaneID string
|
|
ExtractorKey string
|
|
|
|
ChunkID string
|
|
ChunkIndex int
|
|
SourceID string
|
|
|
|
RawContent []byte
|
|
MediaType string
|
|
|
|
SchemaID string
|
|
SchemaName string
|
|
SchemaVersion string
|
|
|
|
Metadata map[string]any
|
|
Warnings []contracts.Warning
|
|
}
|
|
```
|
|
|
|
The exact implementation shape can differ, but it should preserve:
|
|
|
|
- raw returned content;
|
|
- chunk provenance;
|
|
- chunk order;
|
|
- source identity;
|
|
- module identity;
|
|
- response schema provenance;
|
|
- metadata needed by validators, mergers, manifests, diagnostics, and output
|
|
encoders.
|
|
|
|
Extractor modules should not be required to convert raw LLM output into
|
|
`ArtifactCandidate` values. Domain-specific Go projection may still exist for
|
|
specific modules when it is useful, but it should not be the generic pipeline
|
|
contract.
|
|
|
|
Rejected extract outputs are not passed to merge. This keeps downstream
|
|
contracts simple and makes rejection behavior explicit. Support for passing
|
|
rejected outputs forward as marked data is deferred future work.
|
|
|
|
## Merge Stage Target Model
|
|
|
|
The merge stage receives the accepted extract outputs for a lane. Each extract
|
|
output corresponds to one input chunk and carries enough metadata to recover the
|
|
original chunk order.
|
|
|
|
The merger owns:
|
|
|
|
- merge/reconciliation strategy;
|
|
- deterministic or LLM-backed merge logic;
|
|
- prompt and schema usage when LLM-backed;
|
|
- the shape of merged raw output.
|
|
|
|
The framework owns:
|
|
|
|
- passing the ordered accepted extract-output set to the merger;
|
|
- lane identity;
|
|
- source context;
|
|
- references;
|
|
- runtime LLM plumbing;
|
|
- validation handoff for merged output;
|
|
- manifest provenance.
|
|
|
|
Simple merge modules may concatenate raw extract outputs in chunk order. Other
|
|
merge modules may deterministically merge JSON documents, reconcile duplicates,
|
|
or use an LLM to produce a more coherent merged result.
|
|
|
|
Conceptually:
|
|
|
|
```go
|
|
type MergeRequest struct {
|
|
LaneID string
|
|
Source *source.SourceDocument
|
|
ExtractOutputs []ExtractOutput
|
|
|
|
SourceInput contracts.LLMInputMaterial
|
|
SessionID string
|
|
References contracts.ReferenceSet
|
|
LLMClient contracts.StructuredLLMClient
|
|
LLMProfile string
|
|
Options map[string]any
|
|
Metadata map[string]any
|
|
}
|
|
|
|
type MergeOutput struct {
|
|
LaneID string
|
|
MergerKey string
|
|
|
|
RawContent []byte
|
|
MediaType string
|
|
|
|
SchemaID string
|
|
SchemaName string
|
|
SchemaVersion string
|
|
|
|
Metadata map[string]any
|
|
Warnings []contracts.Warning
|
|
}
|
|
```
|
|
|
|
The exact implementation shape can differ, but the key contract is that
|
|
merge consumes ordered accepted extract outputs and produces one merged raw
|
|
output for the lane.
|
|
|
|
If no accepted extract outputs remain for a lane, the framework should not pass
|
|
rejected outputs to the merger. The lane should produce no merge output and
|
|
should be reported as rejected or omitted according to run reporting policy.
|
|
|
|
## Normalize Stage Target Model
|
|
|
|
The normalize stage receives accepted merge output for a lane and optionally
|
|
post-processes it into the final normalized raw payload for that lane.
|
|
Normalization may be a no-op, deterministic cleanup, schema conversion, or an
|
|
LLM-backed post-processing pass.
|
|
|
|
The normalizer owns:
|
|
|
|
- post-merge processing strategy;
|
|
- deterministic or LLM-backed normalization logic;
|
|
- prompt and schema usage when LLM-backed;
|
|
- the shape of normalized raw output.
|
|
|
|
The framework owns:
|
|
|
|
- passing accepted merge output to the normalizer;
|
|
- lane identity;
|
|
- source context;
|
|
- references;
|
|
- runtime LLM plumbing;
|
|
- validation handoff for normalized output;
|
|
- manifest provenance.
|
|
|
|
Conceptually:
|
|
|
|
```go
|
|
type NormalizeRequest struct {
|
|
LaneID string
|
|
Source *source.SourceDocument
|
|
MergeOutput MergeOutput
|
|
|
|
SourceInput contracts.LLMInputMaterial
|
|
SessionID string
|
|
References contracts.ReferenceSet
|
|
LLMClient contracts.StructuredLLMClient
|
|
LLMProfile string
|
|
Options map[string]any
|
|
Metadata map[string]any
|
|
}
|
|
|
|
type NormalizeOutput struct {
|
|
LaneID string
|
|
NormalizerKey string
|
|
|
|
RawContent []byte
|
|
MediaType string
|
|
|
|
SchemaID string
|
|
SchemaName string
|
|
SchemaVersion string
|
|
|
|
Metadata map[string]any
|
|
Warnings []contracts.Warning
|
|
}
|
|
```
|
|
|
|
The exact implementation shape can differ, but the key contract is that
|
|
normalize consumes one accepted merged output and produces one normalized raw
|
|
output for the lane.
|
|
|
|
## Output Stage Target Model
|
|
|
|
The output stage receives validated normalized output bytes and writes them to
|
|
its configured destination.
|
|
|
|
Output modules should not require normalized output to be converted into
|
|
`Artifact` values. Output modules should use the normalized output media type to
|
|
decide how to serialize or wrap the payload. A JSON output module can write raw
|
|
normalized JSON directly, while a text or Markdown output module can write text
|
|
payloads directly. Output modules may also write run manifests, warnings,
|
|
rejected outputs, and indexes.
|
|
|
|
Output modules may still choose to provide convenience layouts, grouping, or file
|
|
naming conventions, but those should be output concerns rather than constraints
|
|
on extractor or normalizer response schemas.
|
|
|
|
## Validation Relationship
|
|
|
|
Validation chains should attach to returned module outputs, not to hidden
|
|
module-internal conversions.
|
|
|
|
For chunk:
|
|
|
|
```text
|
|
chunk(input) -> SourceChunk set -> chunk validators -> extract input
|
|
```
|
|
|
|
For extract:
|
|
|
|
```text
|
|
extract(chunk) -> raw ExtractOutput -> extract validators -> merge input
|
|
```
|
|
|
|
For merge:
|
|
|
|
```text
|
|
merge(extract outputs) -> raw MergeOutput -> merge validators -> normalize input
|
|
```
|
|
|
|
For normalize:
|
|
|
|
```text
|
|
normalize(merge output) -> raw NormalizeOutput -> normalize validators -> output
|
|
```
|
|
|
|
Validators must be read-only. They inspect raw output and metadata, return
|
|
accept/reject decisions and warnings, and do not rewrite output.
|
|
|
|
An empty validator chain approves returned output for that validation point.
|
|
Framework/runtime errors remain separate from validation rejections: if a module
|
|
or Scriptorium call fails before output is returned, the pipeline reports an
|
|
execution error rather than asking validators to evaluate nonexistent output.
|
|
|
|
Rejected outputs do not pass to the next stage. An empty validator chain still
|
|
approves returned output for that validation point.
|
|
|
|
## Retry Policy
|
|
|
|
A retry means re-running the same module with the same input after that module
|
|
fails to produce valid output. Failure to produce valid output includes both:
|
|
|
|
- framework-level execution errors, such as module errors, Scriptorium errors,
|
|
provider errors, or missing returned output;
|
|
- validator rejection of returned output.
|
|
|
|
Retries should be configurable at the pipeline or lane level. Chunk retries are
|
|
per source input. Extract retries are per chunk. Merge and normalize retries,
|
|
when configured, are per lane. Retry attempts should preserve deterministic
|
|
reporting: the final accepted or rejected output should record attempt count and
|
|
enough diagnostics/provenance to understand prior failures without leaking
|
|
secrets or large payloads by default.
|
|
|
|
## Relationship To Validation Roadmap
|
|
|
|
This roadmap defines the data model that validation should evaluate. The
|
|
validation system roadmap in [`validation.md`](validation.md) defines validator
|
|
registration, mapping, execution classes, and concrete validator behavior.
|
|
|
|
The shared boundary between the roadmaps is a returned module output: validators
|
|
inspect typed chunk output or raw module-output envelopes and decide whether
|
|
that output may continue through the pipeline.
|