notarius/docs/roadmap/implementation.md

# Implementation Plan: Checkpoint 5 Seriatim Input Module

## Status

This is a staged implementation plan for
[`5-seriatim-input-module.md`](5-seriatim-input-module.md). It is intended for
an LLM coding agent to follow stage by stage.

This plan implements only checkpoint 5. Do not add D&D extraction, real domain
prompts or schemas, LLM extraction calls, a `notarius run` command, broad
Seriatim schema support, or transcript-specific behavior in core framework
packages in this checkpoint.

## Policy Context

Follow:

- [`docs/policy/architecture.md`](../policy/architecture.md)
- [`docs/policy/documentation.md`](../policy/documentation.md)

Required boundaries:

- keep Seriatim JSON schema details inside `internal/modules/input/seriatim`;
- keep core source, runner, pipeline, extractor, validator, LLM, and config
  packages source-agnostic and domain-agnostic;
- do not add transcript-specific typed fields to `SourceDocument`,
  `SourceUnit`, runner contracts, or pipeline contracts;
- preserve transcript-specific values only as source metadata conventions;
- register the input module through the existing input adapter registry instead
  of adding ad hoc conditionals;
- use flat capability strings in module metadata;
- keep future or planned behavior in `docs/roadmap/` until implemented.

## Global Implementation Decisions

- Add no new third-party dependency. Use `encoding/json` with
  `Decoder.UseNumber` for Seriatim JSON parsing.
- Use `seriatim` as the stable input adapter key.
- Put all concrete Seriatim input code under
  `internal/modules/input/seriatim`.
- Expose a small module API:

```go
const Key = "seriatim"

func New() *Adapter
func ModuleSpec() pipeline.ModuleSpec
func Register(registry *pipeline.InputAdapterRegistry) error
```

- `ModuleSpec()` must return stage `pipeline.StageInput`, no required
  capabilities, and these provided capabilities:
  `source.transcript`, `transcript.speaker`, and `transcript.timestamps`.
- The Seriatim adapter should satisfy `contracts.InputAdapter`.
- Use `source.SourceDocument.Kind = "transcript"`.
- Use `source.SourceDocument.Format =
  "application/vnd.seriatim.minimal+json"`.
- Use `source.SourceUnit.Kind = "transcript_segment"`.
- Compute `SourceDocument.Digest` from the exact raw input bytes as
  `sha256:<hex>`.
- Resolve `SourceDocument.ID` in this order:
  1. trimmed `contracts.ParseRequest.SourceID`, if non-empty;
  2. trimmed string `metadata.id`, if present and non-empty;
  3. trimmed string `metadata.source_id`, if present and non-empty;
  4. deterministic fallback `seriatim:<first-16-hex-chars-of-raw-sha256>`.
- Segment IDs become source unit IDs exactly after validation. Reject segment
  IDs with leading or trailing whitespace rather than silently rewriting them.
- Copy top-level Seriatim `metadata` into `SourceDocument.Metadata`.
- Store segment `speaker`, `start`, and `end` in `SourceUnit.Metadata` under
  keys with those exact names.
- Store `start` and `end` as `json.Number` values so JSON serialization remains
  numeric and the original decimal representation is preserved.
- Require top-level `metadata` to be present and be an object, but do not
  require any specific metadata keys in checkpoint 5.
- Require top-level `segments` to be present and contain at least one segment.
- Reject unknown or extra JSON fields only if they prevent parsing the minimal
  shape. Otherwise ignore them so the module can tolerate compatible Seriatim
  additions.
- Return module-specific errors prefixed with useful Seriatim context, for
  example `seriatim input: segment "s1" text must not be empty`.
- Keep examples free of private transcript content. Use synthetic fixture text.

## Stage 1: Seriatim Package Skeleton And External Model

### Goal

Create the Seriatim input module package, define the module-local JSON model,
and add registry-facing module metadata without changing framework contracts.

### Files To Add

- `internal/modules/input/seriatim/adapter.go`
- `internal/modules/input/seriatim/model.go`
- `internal/modules/input/seriatim/metadata.go`
- `internal/modules/input/seriatim/registry_test.go`

### Required API

Add:

```go
package seriatim

const Key = "seriatim"

const (
    DocumentKind = "transcript"
    UnitKind     = "transcript_segment"
    Format       = "application/vnd.seriatim.minimal+json"
)

const (
    MetadataSpeaker = "speaker"
    MetadataStart   = "start"
    MetadataEnd     = "end"
)

type Adapter struct{}

func New() *Adapter
func (a *Adapter) Key() string
func (a *Adapter) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error)
func ModuleSpec() pipeline.ModuleSpec
func Register(registry *pipeline.InputAdapterRegistry) error
```

Add typed metadata helpers:

```go
func Speaker(unit source.SourceUnit) (string, bool)
func Start(unit source.SourceUnit) (json.Number, bool)
func End(unit source.SourceUnit) (json.Number, bool)
```

### Seriatim JSON Shape

Define module-local structs for the minimal external shape:

```go
type transcript struct {
    Metadata map[string]any `json:"metadata"`
    Segments []segment      `json:"segments"`
}

type segment struct {
    ID      string      `json:"id"`
    Start   json.Number `json:"start"`
    End     json.Number `json:"end"`
    Speaker string      `json:"speaker"`
    Text    string      `json:"text"`
}
```

Use an internal decode helper based on `json.NewDecoder(bytes.NewReader(raw))`
and `UseNumber`.

### Required Behavior

- `New()` returns a non-nil adapter.
- `Adapter.Key()` returns `Key`.
- `ModuleSpec()` returns defensive slices and the capability set listed in the
  global decisions.
- `Register()` calls `InputAdapterRegistry.RegisterWithSpec(ModuleSpec(), ...)`.
- `Register(nil)` returns an error from the registry path rather than panicking.
- Keep external JSON structs unexported.

### Required Tests

- `New()` returns an adapter whose key is `seriatim`.
- `ModuleSpec()` uses input stage and declares the required provided
  capabilities.
- `Register()` makes the adapter buildable from an `InputAdapterRegistry`.
- Registry lookup returns the Seriatim module spec.

### Validation

Run:

```sh
gofmt -w internal/modules/input/seriatim
go test ./internal/modules/input/seriatim
go test ./...
```

## Stage 2: Parse, Validate, And Map To SourceDocument

### Goal

Implement the Seriatim parser and mapper from minimal Seriatim JSON into the
generic source model.

### Files To Add Or Update

- `internal/modules/input/seriatim/adapter.go`
- `internal/modules/input/seriatim/model.go`
- `internal/modules/input/seriatim/adapter_test.go`
- `internal/modules/input/seriatim/testdata/valid_minimal.json`
- `internal/modules/input/seriatim/testdata/duplicate_segment_id.json`

### Required Validation

Reject:

- nil or canceled context before parsing;
- empty raw input;
- malformed JSON;
- valid JSON with trailing non-whitespace data;
- missing, null, or non-object top-level `metadata`;
- missing, null, empty, or non-array top-level `segments`;
- segment IDs that are empty after trimming;
- segment IDs with leading or trailing whitespace;
- duplicate segment IDs;
- missing or empty `speaker`;
- missing, empty, non-numeric, negative, or non-finite `start`;
- missing, empty, non-numeric, negative, or non-finite `end`;
- segments where `end < start`;
- missing or empty `text`.

The parser may preserve leading and trailing whitespace in segment text as long
as the text is not empty after trimming.

### Mapping Rules

- `segment.id` becomes `SourceUnit.ID`.
- `segment.text` becomes `SourceUnit.Text`.
- `speaker`, `start`, and `end` become unit metadata under the exact keys
  defined in stage 1.
- The document metadata is a shallow copy of top-level Seriatim metadata.
- The document digest is based on raw input bytes, not normalized JSON.
- Call `source.ValidateDocument` before returning the document and wrap any
  validation failure with Seriatim context.

### Required Tests

Add tests for:

- valid minimal transcript parses to a source document with expected ID, kind,
  format, digest, units, and metadata;
- `ParseRequest.SourceID` overrides metadata-derived IDs;
- fallback document ID is deterministic and has prefix `seriatim:`;
- malformed JSON returns an actionable Seriatim parse error;
- missing metadata is rejected;
- missing or empty segments is rejected;
- duplicate segment IDs are rejected;
- empty segment text is rejected;
- missing speaker is rejected;
- invalid timestamp values are rejected;
- `end < start` is rejected;
- typed metadata helpers return the expected speaker and timestamp values;
- a `source.SourceRef` using the first and last generated unit IDs validates
  with `source.ValidateRef`.

### Validation

Run:

```sh
gofmt -w internal/modules/input/seriatim
go test ./internal/modules/input/seriatim
go test ./...
```

## Stage 3: Pipeline Resolution And Config Compatibility

### Goal

Prove the Seriatim input module participates in pipeline-profile resolution and
capability validation through existing registries and config loading.

### Files To Add Or Update

- `internal/modules/input/seriatim/config_test.go`
- `internal/modules/input/seriatim/testdata/pipeline.yml`

### Required Test Catalog

Build a test-only module catalog with:

- Seriatim input registered through `seriatim.Register`;
- a fake chunker requiring `source.transcript` and providing `chunks`;
- a fake extractor requiring `chunks`, `transcript.speaker`, and
  `transcript.timestamps`, and providing `fake.artifacts`;
- `pipeline.AppendOrderMerger` registered as `appendorder`, requiring
  `fake.artifacts`;
- `pipeline.NoopNormalizer` registered as `noop`;
- a fake `json` output encoder registered as output stage.

Do not add real extract, chunk, normalize, or output modules for this checkpoint.

### YAML Fixture

Use a synthetic pipeline fixture shaped like:

```yaml
version: 1
pipelines:
  seriatim-fixture:
    input: seriatim
    chunk: fake/chunk
    artifacts:
      events:
        extract: fake/extract
        merge: appendorder
        normalize: noop
    output: json
```

The default LLM profile supplied by `config.Default()` is sufficient. Do not
add real provider settings to this fixture.

### Required Tests

- `config.ParseFileConfigYAML` and `Config.ApplyFileConfig` load the fixture.
- `Config.Resolve` succeeds with pipeline ID `seriatim-fixture` and the
  test-only catalog.
- The resolved pipeline input module is `seriatim`.
- The resolved pipeline digest is non-empty and stable across repeated
  resolution.
- Removing `transcript.timestamps` from the Seriatim module spec in the
  test-only catalog causes resolution to fail with a missing capability error.
- Selecting an unknown `--only` lane still fails through existing resolution
  behavior.

### Validation

Run:

```sh
gofmt -w internal/modules/input/seriatim
go test ./internal/modules/input/seriatim
go test ./internal/core/config
go test ./...
```

## Stage 4: Runner Integration With Fake Downstream Stages

### Goal

Prove real Seriatim input can flow through the existing runner into fake
downstream stages while preserving source-unit IDs and metadata.

### Files To Add Or Update

- `internal/modules/input/seriatim/runner_test.go`

### Required Behavior

Use the same Seriatim fixture from stage 2 and a resolved pipeline from stage 3.
Register fake downstream stages only inside the test.

The fake extractor should:

- inspect the received `SourceDocument` and `SourceChunk`;
- assert that unit IDs match Seriatim segment IDs;
- assert that speaker and timestamp metadata are present;
- return one generic artifact candidate with a source reference pointing at
  existing Seriatim-derived unit IDs.

The test should then assert:

- `Runner.Run` succeeds;
- the manifest records input module `seriatim`;
- the manifest source digest equals the parsed document digest;
- approved artifacts preserve valid source references;
- no transcript-specific type has been added outside the module.

### Required Tests

- successful runner execution from Seriatim JSON through fake chunk, extract,
  merge, normalize, and output stages;
- runner failure when the Seriatim adapter returns an invalid source document,
  using a malformed fixture or test input;
- validation of the fake candidate's source reference with
  `source.ValidateRef`.

### Validation

Run:

```sh
gofmt -w internal/modules/input/seriatim
go test ./internal/modules/input/seriatim
go test ./internal/framework/pipeline
go test ./...
```

## Stage 5: Documentation And Final Verification

### Goal

Document the implemented Seriatim integration contract once the module exists,
without describing unimplemented D&D extraction or run-command behavior.

### Files To Add Or Update

- `docs/integrations/seriatim.md`
- `docs/roadmap/5-seriatim-input-module.md`

### Required Documentation

Create `docs/integrations/seriatim.md` as implemented-behavior documentation
with:

- accepted minimal JSON shape;
- required fields and validation rules;
- mapping from Seriatim fields to `SourceDocument` and `SourceUnit`;
- metadata key conventions for `speaker`, `start`, and `end`;
- capability strings declared by the module;
- note that broader Seriatim schema variants are not yet supported.

Update `docs/roadmap/5-seriatim-input-module.md` only if implementation
reveals a real scope or policy correction. Keep future D&D extraction behavior
out of the integration doc.

### Final Validation

Run:

```sh
gofmt -w internal/modules/input/seriatim
go test ./...
go build ./cmd/notarius
rm -f ./notarius
```

### Done Criteria

- `go test ./...` passes.
- `go build ./cmd/notarius` passes.
- Seriatim minimal transcript JSON maps into `SourceDocument`.
- Unit IDs are stable and validate in source references.
- Transcript fields do not appear in core runner contracts.
- The input module is selectable through the input registry and pipeline-profile
  resolution.
- The input module declares transcript-oriented flat capabilities for pipeline
  validation.
- Tests prove transcript-specific assumptions are isolated to
  `internal/modules/input/seriatim`.

## Open Questions

None. This plan chooses the checkpoint-5 behavior needed to implement the
feature without requiring additional product decisions.