notarius/docs/roadmap/implementation.md

Implementation Plan: Checkpoint 5 Seriatim Input Module

Status

This is a staged implementation plan for 5-seriatim-input-module.md. It is intended for an LLM coding agent to follow stage by stage.

This plan implements only checkpoint 5. Do not add D&D extraction, real domain prompts or schemas, LLM extraction calls, a notarius run command, broad Seriatim schema support, or transcript-specific behavior in core framework packages in this checkpoint.

Policy Context

Follow:

• docs/policy/architecture.md
• docs/policy/documentation.md

Required boundaries:

• keep Seriatim JSON schema details inside internal/modules/input/seriatim;
• keep core source, runner, pipeline, extractor, validator, LLM, and config packages source-agnostic and domain-agnostic;
• do not add transcript-specific typed fields to SourceDocument, SourceUnit, runner contracts, or pipeline contracts;
• preserve transcript-specific values only as source metadata conventions;
• register the input module through the existing input adapter registry instead of adding ad hoc conditionals;
• use flat capability strings in module metadata;
• keep future or planned behavior in docs/roadmap/ until implemented.

Global Implementation Decisions

• Add no new third-party dependency. Use encoding/json with Decoder.UseNumber for Seriatim JSON parsing.
• Use seriatim as the stable input adapter key.
• Put all concrete Seriatim input code under internal/modules/input/seriatim.
• Expose a small module API:

const Key = "seriatim"

func New() *Adapter
func ModuleSpec() pipeline.ModuleSpec
func Register(registry *pipeline.InputAdapterRegistry) error

• ModuleSpec() must return stage pipeline.StageInput, no required capabilities, and these provided capabilities: source.transcript, transcript.speaker, and transcript.timestamps.
• The Seriatim adapter should satisfy contracts.InputAdapter.
• Use source.SourceDocument.Kind = "transcript".
• Use source.SourceDocument.Format = "application/vnd.seriatim.minimal+json".
• Use source.SourceUnit.Kind = "transcript_segment".
• Compute SourceDocument.Digest from the exact raw input bytes as sha256:<hex>.
• Resolve SourceDocument.ID in this order:
  1. trimmed contracts.ParseRequest.SourceID, if non-empty;
  2. trimmed string metadata.id, if present and non-empty;
  3. trimmed string metadata.source_id, if present and non-empty;
  4. deterministic fallback seriatim:<first-16-hex-chars-of-raw-sha256>.
• Segment IDs become source unit IDs exactly after validation. Reject segment IDs with leading or trailing whitespace rather than silently rewriting them.
• Copy top-level Seriatim metadata into SourceDocument.Metadata.
• Store segment speaker, start, and end in SourceUnit.Metadata under keys with those exact names.
• Store start and end as json.Number values so JSON serialization remains numeric and the original decimal representation is preserved.
• Require top-level metadata to be present and be an object, but do not require any specific metadata keys in checkpoint 5.
• Require top-level segments to be present and contain at least one segment.
• Reject unknown or extra JSON fields only if they prevent parsing the minimal shape. Otherwise ignore them so the module can tolerate compatible Seriatim additions.
• Return module-specific errors prefixed with useful Seriatim context, for example seriatim input: segment "s1" text must not be empty.
• Keep examples free of private transcript content. Use synthetic fixture text.

Stage 1: Seriatim Package Skeleton And External Model

Goal

Create the Seriatim input module package, define the module-local JSON model, and add registry-facing module metadata without changing framework contracts.

Files To Add

• internal/modules/input/seriatim/adapter.go
• internal/modules/input/seriatim/model.go
• internal/modules/input/seriatim/metadata.go
• internal/modules/input/seriatim/registry_test.go

Required API

Add:

package seriatim

const Key = "seriatim"

const (
    DocumentKind = "transcript"
    UnitKind     = "transcript_segment"
    Format       = "application/vnd.seriatim.minimal+json"
)

const (
    MetadataSpeaker = "speaker"
    MetadataStart   = "start"
    MetadataEnd     = "end"
)

type Adapter struct{}

func New() *Adapter
func (a *Adapter) Key() string
func (a *Adapter) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error)
func ModuleSpec() pipeline.ModuleSpec
func Register(registry *pipeline.InputAdapterRegistry) error

Add typed metadata helpers:

func Speaker(unit source.SourceUnit) (string, bool)
func Start(unit source.SourceUnit) (json.Number, bool)
func End(unit source.SourceUnit) (json.Number, bool)

Seriatim JSON Shape

Define module-local structs for the minimal external shape:

type transcript struct {
    Metadata map[string]any `json:"metadata"`
    Segments []segment      `json:"segments"`
}

type segment struct {
    ID      string      `json:"id"`
    Start   json.Number `json:"start"`
    End     json.Number `json:"end"`
    Speaker string      `json:"speaker"`
    Text    string      `json:"text"`
}

Use an internal decode helper based on json.NewDecoder(bytes.NewReader(raw)) and UseNumber.

Required Behavior

• New() returns a non-nil adapter.
• Adapter.Key() returns Key.
• ModuleSpec() returns defensive slices and the capability set listed in the global decisions.
• Register() calls InputAdapterRegistry.RegisterWithSpec(ModuleSpec(), ...).
• Register(nil) returns an error from the registry path rather than panicking.
• Keep external JSON structs unexported.

Required Tests

• New() returns an adapter whose key is seriatim.
• ModuleSpec() uses input stage and declares the required provided capabilities.
• Register() makes the adapter buildable from an InputAdapterRegistry.
• Registry lookup returns the Seriatim module spec.

Validation

Run:

gofmt -w internal/modules/input/seriatim
go test ./internal/modules/input/seriatim
go test ./...

Stage 2: Parse, Validate, And Map To SourceDocument

Goal

Implement the Seriatim parser and mapper from minimal Seriatim JSON into the generic source model.

Files To Add Or Update

• internal/modules/input/seriatim/adapter.go
• internal/modules/input/seriatim/model.go
• internal/modules/input/seriatim/adapter_test.go
• internal/modules/input/seriatim/testdata/valid_minimal.json
• internal/modules/input/seriatim/testdata/duplicate_segment_id.json

Required Validation

Reject:

• nil or canceled context before parsing;
• empty raw input;
• malformed JSON;
• valid JSON with trailing non-whitespace data;
• missing, null, or non-object top-level metadata;
• missing, null, empty, or non-array top-level segments;
• segment IDs that are empty after trimming;
• segment IDs with leading or trailing whitespace;
• duplicate segment IDs;
• missing or empty speaker;
• missing, empty, non-numeric, negative, or non-finite start;
• missing, empty, non-numeric, negative, or non-finite end;
• segments where end < start;
• missing or empty text.

The parser may preserve leading and trailing whitespace in segment text as long as the text is not empty after trimming.

Mapping Rules

• segment.id becomes SourceUnit.ID.
• segment.text becomes SourceUnit.Text.
• speaker, start, and end become unit metadata under the exact keys defined in stage 1.
• The document metadata is a shallow copy of top-level Seriatim metadata.
• The document digest is based on raw input bytes, not normalized JSON.
• Call source.ValidateDocument before returning the document and wrap any validation failure with Seriatim context.

Required Tests

Add tests for:

• valid minimal transcript parses to a source document with expected ID, kind, format, digest, units, and metadata;
• ParseRequest.SourceID overrides metadata-derived IDs;
• fallback document ID is deterministic and has prefix seriatim:;
• malformed JSON returns an actionable Seriatim parse error;
• missing metadata is rejected;
• missing or empty segments is rejected;
• duplicate segment IDs are rejected;
• empty segment text is rejected;
• missing speaker is rejected;
• invalid timestamp values are rejected;
• end < start is rejected;
• typed metadata helpers return the expected speaker and timestamp values;
• a source.SourceRef using the first and last generated unit IDs validates with source.ValidateRef.

Validation

Run:

gofmt -w internal/modules/input/seriatim
go test ./internal/modules/input/seriatim
go test ./...

Stage 3: Pipeline Resolution And Config Compatibility

Goal

Prove the Seriatim input module participates in pipeline-profile resolution and capability validation through existing registries and config loading.

Files To Add Or Update

• internal/modules/input/seriatim/config_test.go
• internal/modules/input/seriatim/testdata/pipeline.yml

Required Test Catalog

Build a test-only module catalog with:

• Seriatim input registered through seriatim.Register;
• a fake chunker requiring source.transcript and providing chunks;
• a fake extractor requiring chunks, transcript.speaker, and transcript.timestamps, and providing fake.artifacts;
• pipeline.AppendOrderMerger registered as appendorder, requiring fake.artifacts;
• pipeline.NoopNormalizer registered as noop;
• a fake json output encoder registered as output stage.

Do not add real extract, chunk, normalize, or output modules for this checkpoint.

YAML Fixture

Use a synthetic pipeline fixture shaped like:

version: 1
pipelines:
  seriatim-fixture:
    input: seriatim
    chunk: fake/chunk
    artifacts:
      events:
        extract: fake/extract
        merge: appendorder
        normalize: noop
    output: json

The default LLM profile supplied by config.Default() is sufficient. Do not add real provider settings to this fixture.

Required Tests

• config.ParseFileConfigYAML and Config.ApplyFileConfig load the fixture.
• Config.Resolve succeeds with pipeline ID seriatim-fixture and the test-only catalog.
• The resolved pipeline input module is seriatim.
• The resolved pipeline digest is non-empty and stable across repeated resolution.
• Removing transcript.timestamps from the Seriatim module spec in the test-only catalog causes resolution to fail with a missing capability error.
• Selecting an unknown --only lane still fails through existing resolution behavior.

Validation

Run:

gofmt -w internal/modules/input/seriatim
go test ./internal/modules/input/seriatim
go test ./internal/core/config
go test ./...

Stage 4: Runner Integration With Fake Downstream Stages

Goal

Prove real Seriatim input can flow through the existing runner into fake downstream stages while preserving source-unit IDs and metadata.

Files To Add Or Update

• internal/modules/input/seriatim/runner_test.go

Required Behavior

Use the same Seriatim fixture from stage 2 and a resolved pipeline from stage 3. Register fake downstream stages only inside the test.

The fake extractor should:

• inspect the received SourceDocument and SourceChunk;
• assert that unit IDs match Seriatim segment IDs;
• assert that speaker and timestamp metadata are present;
• return one generic artifact candidate with a source reference pointing at existing Seriatim-derived unit IDs.

The test should then assert:

• Runner.Run succeeds;
• the manifest records input module seriatim;
• the manifest source digest equals the parsed document digest;
• approved artifacts preserve valid source references;
• no transcript-specific type has been added outside the module.

Required Tests

• successful runner execution from Seriatim JSON through fake chunk, extract, merge, normalize, and output stages;
• runner failure when the Seriatim adapter returns an invalid source document, using a malformed fixture or test input;
• validation of the fake candidate's source reference with source.ValidateRef.

Validation

Run:

gofmt -w internal/modules/input/seriatim
go test ./internal/modules/input/seriatim
go test ./internal/framework/pipeline
go test ./...

Stage 5: Documentation And Final Verification

Goal

Document the implemented Seriatim integration contract once the module exists, without describing unimplemented D&D extraction or run-command behavior.

Files To Add Or Update

• docs/integrations/seriatim.md
• docs/roadmap/5-seriatim-input-module.md

Required Documentation

Create docs/integrations/seriatim.md as implemented-behavior documentation with:

• accepted minimal JSON shape;
• required fields and validation rules;
• mapping from Seriatim fields to SourceDocument and SourceUnit;
• metadata key conventions for speaker, start, and end;
• capability strings declared by the module;
• note that broader Seriatim schema variants are not yet supported.

Update docs/roadmap/5-seriatim-input-module.md only if implementation reveals a real scope or policy correction. Keep future D&D extraction behavior out of the integration doc.

Final Validation

Run:

gofmt -w internal/modules/input/seriatim
go test ./...
go build ./cmd/notarius
rm -f ./notarius

Done Criteria

• go test ./... passes.
• go build ./cmd/notarius passes.
• Seriatim minimal transcript JSON maps into SourceDocument.
• Unit IDs are stable and validate in source references.
• Transcript fields do not appear in core runner contracts.
• The input module is selectable through the input registry and pipeline-profile resolution.
• The input module declares transcript-oriented flat capabilities for pipeline validation.
• Tests prove transcript-specific assumptions are isolated to internal/modules/input/seriatim.

Open Questions

None. This plan chooses the checkpoint-5 behavior needed to implement the feature without requiring additional product decisions.