notarius/docs/roadmap/implementation.md

Implementation Plan: Checkpoint 3 Pipeline Stages

Status

This is a staged implementation plan for 3-pipeline-stages-chunking-merge-normalize.md. It is intended for an LLM coding agent to follow stage by stage.

This plan implements only checkpoint 3. Do not add real Seriatim parsing, real D&D extraction, production config file loading, CLI command behavior, LLM provider clients, prompt assets, response schema assets, diagnostics run directories, or durable output writing.

Policy Context

Follow:

• docs/policy/architecture.md
• docs/policy/documentation.md

Required boundaries:

• framework code must stay source-agnostic and domain-agnostic;
• concrete business logic remains under internal/modules/<stage>/..., but this checkpoint should use fake modules only;
• framework plumbing should remain consolidated under internal/framework/pipeline unless a boundary proves itself;
• capabilities are flat strings, not a type system;
• pipeline profiles are fixed-shape templates, not arbitrary step lists;
• no structural module wiring through ad hoc CLI flags.

Global Implementation Decisions

• Add no third-party dependencies.
• Keep new checkpoint 3 framework code in internal/framework/pipeline.
• Keep shared request/result contracts in internal/framework/contracts when stage modules will implement or consume them.
• Do not create new framework packages for chunking, merge, normalize, output, warnings, response schemas, or structured output.
• Use in-memory pipeline profile structs only. Do not implement YAML/TOML parsing in this checkpoint.
• Keep one shared chunker per resolved pipeline. Per-lane chunker overrides are deferred.
• Execute chunks serially in checkpoint 3. Contracts must not prevent later parallel execution.
• Sort artifact lane IDs during resolution for deterministic output and digest behavior.
• Use sha256:<hex> for resolved pipeline digests.
• Preserve existing validator decision-cardinality behavior.
• Run gofmt on all touched Go files before validation.

Stage 1: Stage Contracts And Source Chunks

Goal

Define the source chunk model and contracts for chunk, merge, normalize, and output stages.

Files To Update

• internal/framework/contracts/contracts.go
• internal/framework/contracts/contracts_test.go
• internal/framework/contracts/composition_test.go

Required API

Add:

type SourceChunk struct {
    ID       string              `json:"id"`
    SourceID string              `json:"source_id"`
    Index    int                 `json:"index"`
    Units    []source.SourceUnit `json:"units"`
    Metadata map[string]any      `json:"metadata,omitempty"`
}

type ChunkRequest struct {
    Source   *source.SourceDocument `json:"-"`
    Metadata map[string]any         `json:"metadata,omitempty"`
}

type ChunkResult struct {
    Chunks   []SourceChunk `json:"chunks"`
    Warnings []Warning     `json:"warnings,omitempty"`
}

type Chunker interface {
    Key() string
    Chunk(ctx context.Context, req ChunkRequest) (ChunkResult, error)
}

Extend ExtractionRequest:

type ExtractionRequest struct {
    Source         *source.SourceDocument `json:"-"`
    Chunk          *SourceChunk           `json:"chunk,omitempty"`
    AmbientContext map[string]any         `json:"ambient_context,omitempty"`
    LLMClient      StructuredLLMClient    `json:"-"`
    Metadata       map[string]any         `json:"metadata,omitempty"`
}

Add:

type ChunkArtifacts struct {
    Chunk      SourceChunk                   `json:"chunk"`
    Candidates []artifacts.ArtifactCandidate `json:"candidates"`
}

type MergeRequest struct {
    Source         *source.SourceDocument `json:"-"`
    LaneID         string                 `json:"lane_id"`
    ChunkArtifacts []ChunkArtifacts       `json:"chunk_artifacts"`
    Metadata       map[string]any         `json:"metadata,omitempty"`
}

type MergeResult struct {
    Candidates []artifacts.ArtifactCandidate `json:"candidates"`
    Warnings   []Warning                     `json:"warnings,omitempty"`
}

type Merger interface {
    Key() string
    Merge(ctx context.Context, req MergeRequest) (MergeResult, error)
}

type NormalizeRequest struct {
    Source     *source.SourceDocument        `json:"-"`
    LaneID     string                        `json:"lane_id"`
    Candidates []artifacts.ArtifactCandidate `json:"candidates"`
    Metadata   map[string]any                `json:"metadata,omitempty"`
}

type NormalizeResult struct {
    Candidates []artifacts.ArtifactCandidate `json:"candidates"`
    Warnings   []Warning                     `json:"warnings,omitempty"`
}

type Normalizer interface {
    Key() string
    Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error)
}

type OutputRequest struct {
    Manifest artifacts.RunManifest     `json:"manifest"`
    Approved []artifacts.Artifact      `json:"approved,omitempty"`
    Rejected []artifacts.RejectedArtifact `json:"rejected,omitempty"`
    Warnings []Warning                 `json:"warnings,omitempty"`
    Metadata map[string]any            `json:"metadata,omitempty"`
}

type OutputResult struct {
    Bytes        []byte    `json:"-"`
    ContentType  string    `json:"content_type,omitempty"`
    Warnings     []Warning `json:"warnings,omitempty"`
}

type OutputEncoder interface {
    Key() string
    Encode(ctx context.Context, req OutputRequest) (OutputResult, error)
}

If gofmt aligns fields differently, keep the formatted output.

Required Behavior

• Existing fake extractor tests must still compile with ExtractionRequest{Source: doc}.
• SourceChunk is a framework contract type, not a core source type.
• SourceChunk.Units should be copied by implementations where mutation risk exists; this stage only defines the contract.

Required Tests

Update fake implementations in existing contract tests to prove:

• fake chunker satisfies contracts.Chunker;
• fake merger satisfies contracts.Merger;
• fake normalizer satisfies contracts.Normalizer;
• fake output encoder satisfies contracts.OutputEncoder;
• fake extractor can receive a SourceChunk and ambient context.

Validation

Run:

gofmt -w internal/framework/contracts
go test ./internal/framework/contracts
go test ./...

Stage 2: Module Metadata And Stage Registries

Goal

Teach pipeline registries about module metadata and add registries for chunk, merge, normalize, configured validators, and output modules.

Files To Update

• internal/framework/pipeline/input_registry.go
• internal/framework/pipeline/input_registry_test.go
• internal/framework/pipeline/extractor_registry.go
• internal/framework/pipeline/extractor_registry_test.go

Files To Add

• internal/framework/pipeline/module.go
• internal/framework/pipeline/chunker_registry.go
• internal/framework/pipeline/chunker_registry_test.go
• internal/framework/pipeline/merger_registry.go
• internal/framework/pipeline/merger_registry_test.go
• internal/framework/pipeline/normalizer_registry.go
• internal/framework/pipeline/normalizer_registry_test.go
• internal/framework/pipeline/validator_registry.go
• internal/framework/pipeline/validator_registry_test.go
• internal/framework/pipeline/output_registry.go
• internal/framework/pipeline/output_registry_test.go

Required API

Add:

type ModuleStage string

const (
    StageInput     ModuleStage = "input"
    StageChunk     ModuleStage = "chunk"
    StageExtract   ModuleStage = "extract"
    StageMerge     ModuleStage = "merge"
    StageNormalize ModuleStage = "normalize"
    StageValidate  ModuleStage = "validate"
    StageOutput    ModuleStage = "output"
)

type ModuleSpec struct {
    Key      string
    Stage    ModuleStage
    Provides []string
    Requires []string
}

Add metadata registration methods to existing registries:

func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error
func (r *InputAdapterRegistry) Spec(key string) (ModuleSpec, bool)

func (r *ExtractorRegistry) RegisterWithSpec(spec ModuleSpec, constructor ExtractorConstructor) error
func (r *ExtractorRegistry) Spec(key string) (ModuleSpec, bool)

Keep existing Register methods. They should call RegisterWithSpec with a default spec using the provided key and correct stage.

Add typed registries for other stages:

type ChunkerConstructor func() (contracts.Chunker, error)
type ChunkerRegistry struct { /* unexported fields */ }
func NewChunkerRegistry() *ChunkerRegistry
func (r *ChunkerRegistry) Register(key string, constructor ChunkerConstructor) error
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error
func (r *ChunkerRegistry) Build(key string) (contracts.Chunker, error)
func (r *ChunkerRegistry) Spec(key string) (ModuleSpec, bool)
func (r *ChunkerRegistry) RegisteredKeys() []string

Repeat the same pattern for MergerRegistry, NormalizerRegistry, ValidatorRegistry, and OutputEncoderRegistry.

ValidatorRegistry should use:

type ValidatorConstructor func() (contracts.Validator, error)

Because contracts.Validator uses Name() rather than Key(), ValidatorRegistry.Build must require validator.Name() to match the registered key.

Required Behavior

• Registry keys are trimmed.
• Specs are normalized by trimming Key, Provides, and Requires.
• Empty capability strings are ignored.
• Provides and Requires are deduplicated and sorted for deterministic behavior.
• RegisterWithSpec rejects an empty key, wrong stage, nil constructor, and duplicate key.
• Spec returns false for unknown keys and nil registries.
• Build rejects unknown keys, constructor errors, nil modules, and key mismatches.
• Existing registry tests continue to pass.

Required Tests

For each new registry, cover the same behavior as existing input/extractor registries:

• successful registration and build;
• metadata registration and lookup;
• capability normalization;
• wrong stage rejection;
• key trimming;
• empty key rejection;
• duplicate key rejection;
• nil constructor rejection;
• unknown key build error;
• constructor error wrapping;
• nil module rejection;
• key mismatch rejection;
• sorted RegisteredKeys;
• nil registry behavior.

For ValidatorRegistry, key mismatch checks should compare the registry key to Validator.Name().

For existing input/extractor registries, add tests for metadata registration, default specs from Register, capability normalization, and wrong stage rejection.

Validation

Run:

gofmt -w internal/framework/pipeline
go test ./internal/framework/pipeline
go test ./...

Stage 3: Resolved Pipeline Model

Goal

Add fixed-shape pipeline profile structs, default resolution, lane selection, capability validation, and resolved pipeline digesting.

Files To Add

• internal/framework/pipeline/profile.go
• internal/framework/pipeline/profile_test.go

Files To Update

• internal/core/artifacts/artifacts.go
• internal/core/artifacts/artifacts_test.go

Required API

Update artifacts.RunManifest to include:

PipelineID     string `json:"pipeline_id,omitempty"`
PipelineDigest string `json:"pipeline_digest,omitempty"`

Add lane-level manifest details rather than relying on the existing shared Extractors, Merger, or Normalizer fields for multi-lane runs:

type ArtifactLaneManifest struct {
    ID         string   `json:"id"`
    Extractor  string   `json:"extractor"`
    Merger     string   `json:"merger"`
    Normalizer string   `json:"normalizer"`
    Validators []string `json:"validators,omitempty"`
}

Then add:

ArtifactLanes []ArtifactLaneManifest `json:"artifact_lanes,omitempty"`

Do not remove the existing singular fields in this checkpoint unless doing so is required to make the package compile. Treat ArtifactLanes as the authoritative manifest representation for checkpoint 3's multi-lane pipeline model.

Add pipeline model types:

type ModuleBinding struct {
    Module     string         `json:"module"`
    LLMProfile string         `json:"llm_profile,omitempty"`
    Options    map[string]any `json:"options,omitempty"`
}

type ArtifactLaneProfile struct {
    Extract    ModuleBinding   `json:"extract"`
    Merge      ModuleBinding   `json:"merge,omitempty"`
    Normalize  ModuleBinding   `json:"normalize,omitempty"`
    Validators []ModuleBinding `json:"validators,omitempty"`
}

type PipelineProfile struct {
    ID        string                         `json:"id"`
    Input     ModuleBinding                  `json:"input"`
    Chunk     ModuleBinding                  `json:"chunk,omitempty"`
    Artifacts map[string]ArtifactLaneProfile `json:"artifacts"`
    Output    ModuleBinding                  `json:"output,omitempty"`
}

type ResolveOptions struct {
    Only []string
}

type ResolvedArtifactLane struct {
    ID         string
    Extract    ModuleBinding
    Merge      ModuleBinding
    Normalize  ModuleBinding
    Validators []ModuleBinding
}

type ResolvedPipeline struct {
    ID             string
    Digest         string
    Input          ModuleBinding
    Chunk          ModuleBinding
    ArtifactLanes  []ResolvedArtifactLane
    Output         ModuleBinding
}

Add defaults:

const (
    DefaultChunkModule     = "generic"
    DefaultMergeModule     = "appendorder"
    DefaultNormalizeModule = "noop"
    DefaultOutputModule    = "json"
    DefaultLLMProfile      = "default"
)

Add:

type ModuleCatalog struct {
    Inputs      *InputAdapterRegistry
    Chunkers    *ChunkerRegistry
    Extractors  *ExtractorRegistry
    Mergers     *MergerRegistry
    Normalizers *NormalizerRegistry
    Validators  *ValidatorRegistry
    Outputs     *OutputEncoderRegistry
}

func Binding(module string) ModuleBinding
func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog ModuleCatalog) (ResolvedPipeline, error)

Required Behavior

ResolvePipeline must:

• trim PipelineProfile.ID;
• reject empty pipeline IDs;
• require non-empty input module;
• require at least one artifact lane before and after Only filtering;
• apply defaults for missing chunk, merge, normalize, output, and binding llm_profile values;
• trim all module keys;
• trim lane names in ResolveOptions.Only;
• reject empty or unknown lane names in ResolveOptions.Only;
• deduplicate repeated Only lane names;
• sort selected lane IDs alphabetically;
• reject unknown module keys using registry metadata, not constructors;
• validate flat capability requirements before execution;
• compute ResolvedPipeline.Digest from canonical JSON of the resolved pipeline excluding the digest field itself.

Capability validation algorithm:

1. Start with capabilities provided by the input module.
2. Validate chunker requirements against current capabilities, then add chunker provided capabilities.
3. For each selected lane, start with input plus chunk capabilities.
4. Validate extractor requirements, then add extractor provided capabilities.
5. Validate merger requirements, then add merger provided capabilities.
6. Validate normalizer requirements, then add normalizer provided capabilities.
7. Validate configured validator requirements in lane order, then add validator provided capabilities. If no validators are configured for a lane, do not attempt capability validation for extractor-owned default validators.
8. Validate output requirements against the union of pipeline-level capabilities and all selected lane capabilities.

Error messages must name the pipeline ID, lane ID when applicable, stage, module key, and missing capability when applicable.

Binding(module) should trim the module key and leave LLMProfile empty and Options nil before resolution. Resolution should fill empty LLMProfile values with DefaultLLMProfile on all resolved bindings.

Required Tests

Cover:

• successful resolution with explicit modules;
• default chunk, merge, normalize, output, and llm_profile;
• lane selection with Only;
• Only trimming and deduplication;
• unknown Only lane rejection;
• empty Only lane rejection;
• empty artifact set rejection;
• empty pipeline ID rejection;
• missing input rejection;
• unknown module key rejection for each stage;
• missing capability rejection for input/chunk/extract/merge/normalize/validate/output interactions;
• deterministic lane ordering;
• deterministic digest for equivalent input maps;
• digest changes when a module binding changes;
• manifest JSON still omits empty optional fields.

Use fake registries and fake specs only. Do not add config file parsing.

Validation

Run:

gofmt -w internal/core/artifacts internal/framework/pipeline
go test ./internal/core/artifacts
go test ./internal/framework/pipeline
go test ./...

Stage 4: Generic Merge And Normalize Helpers

Goal

Add generic append-in-chunk-order merge and no-op normalization behavior inside internal/framework/pipeline.

Files To Add

• internal/framework/pipeline/generic_stages.go
• internal/framework/pipeline/generic_stages_test.go

Required API

Add:

type AppendOrderMerger struct{}
func (m AppendOrderMerger) Key() string
func (m AppendOrderMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error)

type NoopNormalizer struct{}
func (n NoopNormalizer) Key() string
func (n NoopNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error)

Required Behavior

AppendOrderMerger must:

• return key appendorder;
• concatenate candidates by ChunkArtifacts slice order and candidate order;
• copy candidate slices so caller mutation does not affect the result;
• preserve source refs and metadata exactly;
• append no warnings.

NoopNormalizer must:

• return key noop;
• copy candidates so caller mutation does not affect the result;
• preserve ordering and source refs exactly;
• append no warnings.

Required Tests

Cover:

• key values;
• append order across multiple chunks;
• stable candidate order within chunks;
• no-op normalize preserves order;
• result slices are caller-mutation safe;
• nil/empty input returns empty candidates and no error.

Validation

Run:

gofmt -w internal/framework/pipeline
go test ./internal/framework/pipeline
go test ./...

Stage 5: Pipeline Runner Refactor

Goal

Refactor the existing runner from extractor-only execution to resolved pipeline execution across input, chunk, extract, merge, normalize, validate, and output.

Files To Update

• internal/framework/pipeline/runner.go
• internal/framework/pipeline/runner_test.go
• internal/framework/pipeline/registry_integration_test.go

Required API

Replace the extractor-only runner constructor with:

type Registries struct {
    Inputs      *InputAdapterRegistry
    Chunkers    *ChunkerRegistry
    Extractors  *ExtractorRegistry
    Mergers     *MergerRegistry
    Normalizers *NormalizerRegistry
    Validators  *ValidatorRegistry
    Outputs     *OutputEncoderRegistry
}

type Runner struct {
    // unexported fields
}

func New(registries Registries) *Runner

Replace RunInput with:

type RunInput struct {
    Pipeline  ResolvedPipeline
    SourceID  string
    Path      string
    RawInput  []byte
    LLMClient contracts.StructuredLLMClient
    Metadata  map[string]any
}

Replace or extend RunOutput with:

type RunOutput struct {
    Manifest      artifacts.RunManifest      `json:"manifest"`
    Approved      []artifacts.Artifact       `json:"approved,omitempty"`
    Rejected      []artifacts.RejectedArtifact `json:"rejected,omitempty"`
    Warnings      []contracts.Warning        `json:"warnings,omitempty"`
    EncodedOutput []byte                     `json:"-"`
    ContentType   string                     `json:"content_type,omitempty"`
}

Keep the existing candidate metadata normalization behavior, but it should now run per extractor and across all selected lanes.

Required Behavior

Run must:

1. reject nil runner;
2. reject missing registries for stages used by the pipeline; Validators is required only when at least one selected lane declares validator bindings;
3. reject empty resolved pipeline ID or digest;
4. build the input adapter and parse RawInput;
5. validate the resulting SourceDocument;
6. build the chunker and produce chunks;
7. reject empty chunk results;
8. for each selected artifact lane in resolved order:
   • build extractor, merger, and normalizer;
   • extract from every chunk serially;
   • pass Source, active Chunk, LLMClient, and metadata to the extractor;
   • normalize candidate metadata with global monotonically increasing indices;
   • merge per-chunk candidates;
   • normalize merged candidates;
   • run configured lane validators against normalized candidates when the lane declares validators;
   • otherwise run the extractor-owned default validator chain;
   • append approved artifacts and rejected artifacts to output;
9. build the output encoder and encode the final output bundle;
10. populate RunManifest with pipeline ID, pipeline digest, input module, chunker, lane-level extractor/merger/normalizer/validator keys, output encoder, source digest, and validation status;
11. return partial output plus wrapped errors on failure.

Validation status:

• approved when all candidates from all lanes survive validation and no lane returns rejected artifacts;
• rejected when at least one candidate is rejected and execution otherwise succeeds;
• failed when returning an error after a manifest has been initialized.

Warnings:

• collect warnings from input, chunk, extract, merge, normalize, validator, and output stages where those stages expose warnings;
• existing input adapters do not expose warnings, so no input warnings are required in this checkpoint.

Do not validate source references inside the runner in checkpoint 3. Source reference validation remains a later deterministic validator.

Required Tests

Refactor existing runner tests and add coverage for:

• nil runner error;
• missing registry errors;
• invalid resolved pipeline error;
• input adapter build and parse errors;
• invalid source document error;
• chunker build and chunk errors;
• empty chunk result error;
• extractor runs once per chunk;
• extractor receives active chunk and LLM client;
• merge receives per-chunk candidates;
• normalize receives merged candidates;
• validator approvals and rejections still work;
• configured validators are built and run in lane order;
• extractor-owned default validators run when a lane does not configure validators;
• configured validators replace extractor-owned default validators when present;
• global candidate indices across lanes and chunks;
• warnings from chunk, extract, merge, normalize, validator, and output stages;
• output encoder receives manifest and artifacts;
• manifest pipeline ID, digest, and artifact lane details are populated;
• partial output is returned when a later lane fails.

Use fake modules only.

Validation

Run:

gofmt -w internal/framework/pipeline
go test ./internal/framework/pipeline
go test ./...

Stage 6: Fixture-Driven Walking Skeleton

Goal

Add an end-to-end fixture-driven integration test that exercises the full staged path with fake modules and a fake LLM client.

Files To Add

• internal/framework/pipeline/testdata/walking_skeleton_input.json
• internal/framework/pipeline/testdata/walking_skeleton_output.json
• internal/framework/pipeline/walking_skeleton_test.go

Required Fixture Shape

Use a small input fixture with two or three source units in a simple fake JSON format. Do not use Seriatim.

Example input shape:

{
  "id": "fixture-source",
  "units": [
    {"id": "u1", "text": "First event."},
    {"id": "u2", "text": "Second event."},
    {"id": "u3", "text": "Third event."}
  ]
}

The expected output fixture should contain:

• manifest with pipeline_id and pipeline_digest;
• approved artifacts emitted from multiple chunks;
• no rejected artifacts for the happy path;
• warnings only if fake stages deliberately emit them.

Compare expected output by unmarshaled structural equality, not raw bytes, so formatting differences do not fail the test.

Required Test Setup

Register fake modules with metadata:

• fake input adapter key fake/input, provides plain_text;
• fake chunker key fake/chunk, requires plain_text, provides chunks;
• fake extractor key fake/extract, requires chunks, provides fake_artifacts;
• append-order merger key appendorder, requires fake_artifacts;
• no-op normalizer key noop;
• fake output encoder key json.

Use an in-memory PipelineProfile:

PipelineProfile{
    ID: "walking-skeleton",
    Input: Binding("fake/input"),
    Chunk: Binding("fake/chunk"),
    Artifacts: map[string]ArtifactLaneProfile{
        "events": {Extract: Binding("fake/extract")},
    },
    Output: Binding("json"),
}

Resolve the profile, run the runner with fixture bytes and fake LLM client, then assert the encoded output matches the expected fixture structurally.

Required Behavior

• The fake input adapter parses raw fixture bytes into SourceDocument.
• The fake chunker produces at least two chunks.
• The fake extractor calls the fake structured LLM client once per chunk.
• The fake extractor returns one artifact candidate per chunk.
• The append-order merger preserves chunk order.
• The no-op normalizer preserves candidates.
• The fake output encoder produces deterministic JSON bytes.
• The test fails before execution if required capabilities are missing.

Additional Tests

Add focused tests for the walking skeleton failure path:

• missing required capability fails during resolution;
• Only with an unknown lane fails during resolution;
• fake LLM call count equals chunk count in the happy path.

Validation

Run:

gofmt -w internal/framework/pipeline
go test ./internal/framework/pipeline
go test ./...

Stage 7: Final Checkpoint 3 Review Pass

Goal

Clean up naming, formatting, and accidental scope creep before checkpoint 3 is considered complete.

Required Review

Check:

• no Seriatim parser exists;
• no D&D module exists;
• no production config file loader exists;
• no CLI behavior was added;
• no LLM provider code exists;
• no prompt, response schema, diagnostics, or durable output package exists;
• no new framework package was created beyond contracts, pipeline, and validate;
• stage contracts remain source/domain agnostic;
• capabilities remain flat strings;
• resolved pipeline profiles remain fixed-shape templates;
• walking skeleton uses fake modules only.

Required Validation

Run:

gofmt -w internal/core internal/framework
go test ./...
go build ./cmd/notarius

Open Questions

None. The deferred choices identified while drafting this plan are intentionally out of scope for checkpoint 3 and should remain deferred until later roadmap items.