notarius/docs/roadmap/pipeline.md

Raw Pipeline Data Model

This roadmap defines the target data model for the pipeline stages after input. It is paired with the validation system roadmap in validation.md: validators should evaluate module outputs, but the pipeline first needs a raw-output handoff model that does not require every module response shape to be represented by bespoke Go structs.

Goals

• Make typed chunk envelopes and raw module-output envelopes first-class pipeline handoffs.
• Keep framework-owned provenance around raw payloads so runs remain deterministic, ordered, and auditable.
• Avoid requiring each extractor, merger, or normalizer response schema to have matching Go structs.
• Let extract modules produce one raw output per input chunk.
• Let merge modules receive accepted extract outputs and merge them into one raw payload.
• Let normalize modules optionally post-process accepted merge output.
• Let output modules write normalized output bytes directly, along with manifests, warnings, rejected outputs, and diagnostics as appropriate.
• Preserve the fixed workflow shape:

input -> chunk -> extract -> merge -> normalize -> output

Non-Goals

• Do not turn the pipeline into an arbitrary DAG or general workflow language.
• Do not make validators mutate, materialize, or rewrite module outputs.
• Do not require a generic source-reference convention for every raw payload in this pass.
• Do not eliminate typed framework data where the framework genuinely needs it, such as input source documents and chunk boundaries.

Cross-Stage Runtime Plumbing

LLM/profile/reference plumbing should be available to every stage that may need LLM-backed or reference-aware module behavior: chunk, extract, merge, and normalize.

For all four stages, the framework should provide the same categories of runtime support where the concrete module contract needs them:

• configured LLM profile and profile override handling;
• Scriptorium client access through framework-owned LLM contracts;
• session ID propagation;
• declared reference slots and resolved reference bindings;
• module options and metadata;
• prompt, schema, profile, and reference provenance for manifests and diagnostics.

merge must not be treated as a deterministic-only stage. A merge module may be simple and deterministic, but it may also be LLM-backed, reference-aware, and validator-gated in the same way as chunk, extract, and normalize.

Chunk Stage Target Model

The chunk stage receives the canonical SourceDocument from the input stage, plus any original source input material needed for prompt construction. The SourceDocument remains the source of truth for source identity, source-unit ordering, source-unit IDs, and source provenance. Original raw input material may be supplied to LLM-backed chunkers, but it should not replace the SourceDocument as the framework handoff.

Input adapters are responsible for assigning stable integer source-unit IDs. How those IDs are assigned depends on the input format. A numbered JSON transcript may map source units directly to transcript segment numbers; a PDF input adapter may assign page or extracted-text unit numbers; an adapter for unordered source material may assign deterministic IDs as part of input parsing. Downstream framework code should treat those IDs as opaque integers owned by the input adapter.

The chunk module returns one or more ordered SourceChunk envelopes. A single chunk representing the whole source is valid and should be supported.

Each chunk should include framework-readable provenance and ordering metadata, plus content suitable for extraction. The content may be JSON, plain text, Markdown, PDF page text, or another module/input-specific representation, as long as the framework can still associate the chunk with the source and preserve deterministic order.

Conceptually:

type SourceChunk struct {
    ID       string
    SourceID string
    Index    int

    StartUnitID int
    EndUnitID   int

    Content   []byte
    MediaType string

    Units    []source.SourceUnit
    Metadata map[string]any
}

The exact implementation shape can differ, but it should preserve:

• chunk identity;
• source identity;
• deterministic chunk order;
• source locator or range, normally integer start/end unit IDs;
• chunk content and media type;
• optional source-unit projection when useful;
• metadata needed by extractors, validators, manifests, diagnostics, and output encoders.

The framework owns the minimal invariants required to schedule extract work:

• the chunker returns at least one chunk;
• chunk IDs and indexes are stable and non-empty;
• chunk source IDs match the source document;
• chunk ordering is deterministic;
• chunk provenance is sufficient to trace the chunk back to the input source.

Domain-specific chunk acceptability belongs in validators. For example, full coverage, no gaps, no overlap, scene metadata quality, expected media type, and D&D scene-boundary policy should be explicit validator concerns rather than hidden framework rules, except where a minimal invariant is required for extraction to run safely.

Extract Stage Target Model

The extract stage receives one input chunk from the chunk stage and produces one raw extract output for that chunk.

The extractor owns:

• prompt selection;
• input material assembly;
• Scriptorium request construction;
• response-schema selection;
• LLM profile and session usage;
• returned raw payload metadata.

The framework owns:

• chunk iteration;
• deterministic ordering;
• association between each extract output and its input chunk;
• execution errors when an extractor does not return output;
• handoff of returned output to validation and later stages.

An extract output should be an envelope, not just raw bytes. Conceptually:

type ExtractOutput struct {
    LaneID       string
    ExtractorKey string

    ChunkID    string
    ChunkIndex int
    SourceID   string

    RawContent []byte
    MediaType  string

    SchemaID      string
    SchemaName    string
    SchemaVersion string

    Metadata map[string]any
    Warnings []contracts.Warning
}

The exact implementation shape can differ, but it should preserve:

• raw returned content;
• chunk provenance;
• chunk order;
• source identity;
• module identity;
• response schema provenance;
• metadata needed by validators, mergers, manifests, diagnostics, and output encoders.

Extractor modules should not be required to convert raw LLM output into ArtifactCandidate values. Domain-specific Go projection may still exist for specific modules when it is useful, but it should not be the generic pipeline contract.

Rejected extract outputs are not passed to merge. This keeps downstream contracts simple and makes rejection behavior explicit. Support for passing rejected outputs forward as marked data is deferred future work.

Merge Stage Target Model

The merge stage receives the accepted extract outputs for a lane. Each extract output corresponds to one input chunk and carries enough metadata to recover the original chunk order.

The merger owns:

• merge/reconciliation strategy;
• deterministic or LLM-backed merge logic;
• prompt and schema usage when LLM-backed;
• the shape of merged raw output.

The framework owns:

• passing the ordered accepted extract-output set to the merger;
• lane identity;
• source context;
• references;
• runtime LLM plumbing;
• validation handoff for merged output;
• manifest provenance.

Simple merge modules may concatenate raw extract outputs in chunk order. Other merge modules may deterministically merge JSON documents, reconcile duplicates, or use an LLM to produce a more coherent merged result.

Conceptually:

type MergeRequest struct {
    LaneID          string
    Source          *source.SourceDocument
    ExtractOutputs  []ExtractOutput

    SourceInput contracts.LLMInputMaterial
    SessionID   string
    References  contracts.ReferenceSet
    LLMClient   contracts.StructuredLLMClient
    LLMProfile  string
    Options     map[string]any
    Metadata    map[string]any
}

type MergeOutput struct {
    LaneID    string
    MergerKey string

    RawContent []byte
    MediaType  string

    SchemaID      string
    SchemaName    string
    SchemaVersion string

    Metadata map[string]any
    Warnings []contracts.Warning
}

The exact implementation shape can differ, but the key contract is that merge consumes ordered accepted extract outputs and produces one merged raw output for the lane.

If no accepted extract outputs remain for a lane, the framework should not pass rejected outputs to the merger. The lane should produce no merge output and should be reported as rejected or omitted according to run reporting policy.

Normalize Stage Target Model

The normalize stage receives accepted merge output for a lane and optionally post-processes it into the final normalized raw payload for that lane. Normalization may be a no-op, deterministic cleanup, schema conversion, or an LLM-backed post-processing pass.

The normalizer owns:

• post-merge processing strategy;
• deterministic or LLM-backed normalization logic;
• prompt and schema usage when LLM-backed;
• the shape of normalized raw output.

The framework owns:

• passing accepted merge output to the normalizer;
• lane identity;
• source context;
• references;
• runtime LLM plumbing;
• validation handoff for normalized output;
• manifest provenance.

Conceptually:

type NormalizeRequest struct {
    LaneID      string
    Source      *source.SourceDocument
    MergeOutput MergeOutput

    SourceInput contracts.LLMInputMaterial
    SessionID   string
    References  contracts.ReferenceSet
    LLMClient   contracts.StructuredLLMClient
    LLMProfile  string
    Options     map[string]any
    Metadata    map[string]any
}

type NormalizeOutput struct {
    LaneID        string
    NormalizerKey string

    RawContent []byte
    MediaType  string

    SchemaID      string
    SchemaName    string
    SchemaVersion string

    Metadata map[string]any
    Warnings []contracts.Warning
}

The exact implementation shape can differ, but the key contract is that normalize consumes one accepted merged output and produces one normalized raw output for the lane.

Output Stage Target Model

The output stage receives validated normalized output bytes and writes them to its configured destination.

Output modules should not require normalized output to be converted into Artifact values. Output modules should use the normalized output media type to decide how to serialize or wrap the payload. A JSON output module can write raw normalized JSON directly, while a text or Markdown output module can write text payloads directly. Output modules may also write run manifests, warnings, rejected outputs, and indexes.

Output modules may still choose to provide convenience layouts, grouping, or file naming conventions, but those should be output concerns rather than constraints on extractor or normalizer response schemas.

Validation Relationship

Validation chains should attach to returned module outputs, not to hidden module-internal conversions.

For chunk:

chunk(input) -> SourceChunk set -> chunk validators -> extract input

For extract:

extract(chunk) -> raw ExtractOutput -> extract validators -> merge input

For merge:

merge(extract outputs) -> raw MergeOutput -> merge validators -> normalize input

For normalize:

normalize(merge output) -> raw NormalizeOutput -> normalize validators -> output

Validators must be read-only. They inspect raw output and metadata, return accept/reject decisions and warnings, and do not rewrite output.

An empty validator chain approves returned output for that validation point. Framework/runtime errors remain separate from validation rejections: if a module or Scriptorium call fails before output is returned, the pipeline reports an execution error rather than asking validators to evaluate nonexistent output.

Rejected outputs do not pass to the next stage. An empty validator chain still approves returned output for that validation point.

Retry Policy

A retry means re-running the same module with the same input after that module fails to produce valid output. Failure to produce valid output includes both:

• framework-level execution errors, such as module errors, Scriptorium errors, provider errors, or missing returned output;
• validator rejection of returned output.

Retries should be configurable at the pipeline or lane level. Chunk retries are per source input. Extract retries are per chunk. Merge and normalize retries, when configured, are per lane. Retry attempts should preserve deterministic reporting: the final accepted or rejected output should record attempt count and enough diagnostics/provenance to understand prior failures without leaking secrets or large payloads by default.

Relationship To Validation Roadmap

This roadmap defines the data model that validation should evaluate. The validation system roadmap in validation.md defines validator registration, mapping, execution classes, and concrete validator behavior.

The shared boundary between the roadmaps is a returned module output: validators inspect typed chunk output or raw module-output envelopes and decide whether that output may continue through the pipeline.