notarius/docs/roadmap/implementation.md

# Raw Pipeline Data Model Implementation Plan

This plan implements the target state in
[`pipeline.md`](pipeline.md). It is intentionally staged for multiple coding
passes. Do not skip stages: later work assumes the contracts and tests from
earlier stages are already in place.

The desired end state is a fixed-shape pipeline:

```text
input -> chunk -> extract -> merge -> normalize -> output
```

`chunk` produces typed chunk envelopes. `extract`, `merge`, and `normalize`
produce raw byte payload envelopes with media type and provenance. Rejected
outputs do not pass to the next stage.

## Stage 1: Integer Source Units And Chunk Payloads

Update the core source and chunk contracts before changing downstream stages.

Implementation tasks:

- Change `internal/core/source.SourceUnit.ID` from `string` to `int`.
- Change `internal/core/source.SourceRef.StartUnitID` and `EndUnitID` from
  `string` to `int`.
- Treat source-unit IDs as positive, stable, input-adapter-owned integers.
  `0` and negative IDs should be invalid.
- Update source validation helpers, including unit lookup and source-ref
  ordering checks, to use integer IDs.
- Update `contracts.SourceChunk` to include:
  - `StartUnitID int`;
  - `EndUnitID int`;
  - `Content []byte`;
  - `MediaType string`.
- Keep `SourceChunk.Units []source.SourceUnit` for framework scheduling,
  diagnostics, and modules that need unit metadata.
- Update chunk canonicalization in `internal/framework/pipeline` to enforce
  minimal scheduling invariants:
  - at least one chunk;
  - non-empty chunk ID;
  - source ID matches the source document;
  - indexes are deterministic and contiguous from zero;
  - start/end unit IDs exist and are ordered;
  - `Units` are canonical copies from the source document;
  - content is non-empty;
  - media type is non-empty.
- Update the Seriatim input adapter to parse segment IDs as integers.
  Accept JSON numbers and numeric strings only when they are positive integers.
  Reject non-numeric IDs such as `seg-001`.
- Update Seriatim examples, fixtures, tests, and integration docs to use
  integer segment IDs.
- Update D&D source-reference helper code under
  `internal/modules/sharedassets/dnd` so source refs remain integer-valued all
  the way into `source.SourceRef`.
- Remove fallback behavior that converts integer LLM unit refs back into string
  source-unit IDs.
- Update the generic chunker and D&D scenes chunker to populate
  `StartUnitID`, `EndUnitID`, `Content`, and `MediaType`.
- For current transcript chunkers, use `application/json` content containing a
  canonical JSON encoding of the chunk's source units. Keep the original
  `Units` field populated as framework-owned typed metadata.

Tests to update or add:

- `go test ./internal/core/source`
- `go test ./internal/modules/input/seriatim`
- `go test ./internal/modules/chunk/generic`
- `go test ./internal/modules/chunk/dnd/scenes`
- `go test ./internal/framework/contracts`
- `go test ./internal/framework/pipeline`

Completion criteria:

- All source refs and source-unit IDs in core/framework types are integers.
- Chunkers return extraction-ready content bytes plus media type.
- Framework chunk validation still allows partial coverage and overlap unless a
  validator rejects them.

## Stage 2: Raw Module Output Contracts

Replace artifact-candidate stage contracts with raw payload envelopes.

Implementation tasks:

- Add contract types in `internal/framework/contracts` for raw stage payloads.
  The exact names may differ, but the contracts must preserve:
  - raw bytes;
  - media type;
  - lane ID;
  - stage module key;
  - source ID;
  - chunk ID and chunk index where applicable;
  - response schema ID, name, and version when applicable;
  - metadata;
  - warnings.
- Recommended shape:
  - `ExtractOutput`
  - `MergeOutput`
  - `NormalizeOutput`
  - a small shared payload/provenance helper if it reduces duplication.
- Update `Extractor` so `Extract` returns one raw `ExtractOutput` for one input
  chunk instead of `[]artifacts.ArtifactCandidate`.
- Remove `Extractor.ArtifactType`, `Extractor.SchemaVersion`, and
  `Extractor.Validators` from the generic extractor contract. Schema and prompt
  provenance should come from the returned output and manifest metadata, not
  artifact-specific methods.
- Update `Merger` so `Merge` receives ordered accepted `[]ExtractOutput` and
  returns one raw `MergeOutput`.
- Update `Normalizer` so `Normalize` receives one accepted `MergeOutput` and
  returns one raw `NormalizeOutput`.
- Update `OutputRequest` so output encoders receive the validated normalized
  lane outputs rather than approved `Artifact` values.
- Add a raw rejected-output record type for manifests/output files. It should
  identify stage, lane, module, chunk when relevant, validator or error reason,
  message, attempt count, and diagnostic artifact path if present. It must not
  include large raw content by default.
- Leave old artifact/candidate types in `internal/core/artifacts` only if
  temporary compatibility code still needs them during the migration. They
  should not remain in the final stage contracts.

Tests to update or add:

- Contract composition tests proving fake chunk, extract, merge, normalize, and
  output modules compose using raw envelopes.
- Defensive-copy tests for raw content and metadata maps.
- Registry integration tests proving the new interfaces build and run.

Completion criteria:

- Framework contracts no longer require extractors, mergers, normalizers, or
  output encoders to use `ArtifactCandidate` or `Artifact`.
- Stage outputs carry raw bytes and media type.

## Stage 3: Config, References, Profiles, And Retries

Make `merge` a first-class LLM/profile/reference stage and add retry policy.

Implementation tasks:

- Update `pipeline.referenceSlotStage` so `merge` modules may declare reference
  slots.
- Add `MergeReferences` to `ResolvedArtifactLane`.
- Resolve and materialize merge references using the same declared-slot model
  used by `chunk`, `extract`, and `normalize`.
- Update reference provenance so merge-stage references appear in manifests and
  resolved reference diagnostics.
- Update config parsing, config validation, redaction, cloning, and examples so
  `artifacts.<lane>.merge.references` is valid.
- Update CLI `--reference` selector parsing and ambiguity checks to support:
  - `merge.<slot>=<path>` where unambiguous;
  - `<lane>.merge.<slot>=<path>`;
  - lane-qualified shorthand only when it resolves unambiguously.
- Update `--without-reference` behavior to support merge-stage slots.
- Update `ModuleStage` LLM-capable helpers so `chunk`, `extract`, `merge`, and
  `normalize` are all considered LLM-capable.
- Update `--llm-profile` override handling so it applies to `chunk`, every lane
  `extract`, every lane `merge`, and every lane `normalize`.
- Update explicit Scriptorium profile validation so it inspects only those four
  LLM-capable stages.
- Add `retries` to `pipeline.ModuleBinding` as a non-negative integer count of
  extra attempts after the first attempt. Default is `0`.
- Apply `retries` only to `chunk`, `extract`, `merge`, and `normalize` runtime
  execution. Keep input and output retry behavior out of scope.
- Validate `retries >= 0` in config validation.
- Ensure redacted config preserves retry counts.

Tests to update or add:

- Config accepts merge references and rejects unknown/undeclared merge slots.
- CLI reference selectors work for merge and remain strict for ambiguous slots.
- `--llm-profile` overrides merge bindings as well as chunk/extract/normalize.
- Explicit profile validation includes merge and ignores non-LLM stages.
- Negative retries are rejected.

Completion criteria:

- Merge has the same runtime plumbing class as chunk/extract/normalize.
- Retry policy is representable in pipeline config and resolved bindings.

## Stage 4: Runner Orchestration And Retry Semantics

Rewrite pipeline execution around raw envelopes.

Implementation tasks:

- Introduce a small retry helper in `internal/framework/pipeline` that reruns
  the same module with the same input when:
  - the module returns a framework-level error;
  - the module returns output that is rejected by that stage's validator chain.
- Preserve context cancellation: never retry after `ctx.Err()` is non-nil.
- Record attempt count and compact attempt diagnostics for manifests/output.
- Do not log or manifest raw payload bytes by default.
- Run chunk once per source input, with retries from the chunk binding.
- After successful chunk execution, run chunk validation when validator mapping
  support exists. A rejected chunk result after final retry should stop
  downstream execution and produce a rejected run record.
- Run extract once per accepted chunk, with per-chunk retries from the extract
  binding.
- Omit rejected extract outputs from merge input.
- If a lane has no accepted extract outputs after retries, skip merge and
  normalize for that lane, record rejected output details, and continue to the
  next lane.
- Run merge once per lane with accepted extract outputs, with retries from the
  merge binding.
- If merge output is rejected after final retry, omit that lane from normalize
  and output.
- Run normalize once per accepted merge output, with retries from the normalize
  binding.
- If normalize output is rejected after final retry, omit that lane from output.
- Treat validator rejection as non-fatal run outcome. Treat unrecovered
  framework-level execution errors as run failures.
- Preserve deterministic ordering:
  - chunks by chunk index;
  - extract outputs by chunk index;
  - lane outputs by resolved lane order.
- Continue to collect warnings from every successful module attempt whose output
  is used. For failed attempts, record compact attempt diagnostics rather than
  promoting warnings as final-stage warnings unless the implementation already
  has a clear warning policy.
- Set manifest validation status to:
  - `approved` when all produced normalized lane outputs pass;
  - `rejected` when one or more module outputs are rejected;
  - failed-run status through existing failure handling for unrecovered runtime
    errors.

Validation seam scope:

- Add the runner-side raw validation seam needed by this data-model refactor.
- Represent the validation boundary as raw module output plus stage provenance,
  consistent with [`validation.md`](validation.md).
- Support empty validator chains as approval.
- Add fake validator/test-only coverage for approval, rejection, and
  retry-on-rejection behavior.
- Do not implement the full production validator package migration, default
  validator mappings, or concrete generic validators in this plan. Those remain
  owned by the validation roadmap.

Tests to update or add:

- Runner passes chunk content and media type to extractors.
- Runner passes ordered accepted extract outputs to merge.
- Runner passes accepted merge output to normalize.
- Rejected extract output is omitted from merge input.
- A lane with no accepted extracts is omitted and recorded.
- Rejected merge output prevents normalize for that lane.
- Rejected normalize output prevents output for that lane.
- Retry reruns the same module input after framework error.
- Retry reruns the same module input after validator rejection.
- Retry stops after configured attempts and records attempt count.
- Context cancellation stops retries.

Completion criteria:

- The runner no longer depends on candidate materialization between extract,
  merge, normalize, and output.
- Rejected outputs never pass to the next stage.

## Stage 5: Production Module Migration

Update concrete modules to the new contracts.

Implementation tasks:

- Update `internal/modules/extract/dnd/spells`:
  - call Scriptorium with the existing prompt/schema;
  - capture the returned raw JSON response bytes;
  - return one `ExtractOutput` with `application/json` media type and response
    schema provenance;
  - do not convert spell casts into `ArtifactCandidate`;
  - do not run shape/source-ref/source-relatedness validation inside the module.
- Keep D&D spell prompt and schema asset metadata in manifest metadata.
- Update `internal/modules/merge/appendorder` to merge raw JSON extract outputs
  deterministically:
  - preserve input order by chunk index;
  - if all extract outputs are JSON objects with one common top-level array
    field, concatenate those arrays into one JSON object under that field;
  - otherwise produce a JSON array containing each extract output's decoded JSON
    value in order;
  - reject non-JSON media types with a clear error unless a future option
    deliberately supports them.
- Update `internal/modules/normalize/noop` to pass accepted merge output through
  unchanged as normalized output.
- Update chunk modules from Stage 1 as needed after interface changes.
- Remove or quarantine old candidate validators from D&D module packages if
  they no longer compile. Their replacement belongs under the validation
  roadmap in `internal/validators`.
- Update fake/test modules throughout the repository to the raw contracts.

Tests to update or add:

- D&D spells extractor returns raw JSON with schema provenance and no candidate
  materialization.
- Append-order merger concatenates common top-level JSON arrays.
- Append-order merger falls back to ordered JSON value arrays when shapes differ.
- Append-order merger rejects invalid JSON and non-JSON media types clearly.
- No-op normalizer returns defensive copies of merge output bytes and metadata.
- Production catalog still registers all modules successfully.

Completion criteria:

- Production modules compile against raw contracts.
- D&D spell extraction no longer rejects spell candidates inside the extractor.

## Stage 6: Output, Manifest, Diagnostics, And Files

Update durable output around normalized lane outputs.

Implementation tasks:

- Update `RunOutput` to carry normalized lane outputs and rejected module-output
  records instead of approved/rejected artifacts.
- Update `artifacts.RunManifest` or introduce a more accurately named manifest
  package if the old artifact naming becomes misleading. Prefer the smallest
  rename that keeps manifest output clear and avoids a broad unrelated refactor.
- Preserve existing manifest fields that remain meaningful:
  - pipeline ID and digest;
  - module keys;
  - lane IDs;
  - source digest;
  - LLM profile provenance;
  - reference provenance;
  - module metadata;
  - validation status.
- Add manifest/reporting fields needed for raw outputs:
  - normalized lane output media type;
  - output schema provenance where present;
  - rejected stage/lane/chunk/module information;
  - retry attempt counts.
- Update `internal/modules/output/json`:
  - write `manifest.json`;
  - write `index.json`;
  - write `warnings.json`;
  - write `rejected.json`;
  - write one file per normalized lane output.
- For the JSON output encoder, accept `application/json` normalized outputs and
  write them as raw JSON files. Recommended file path:
  `lanes/<safe-lane-id>.json`.
- Reject non-JSON normalized output media types in the JSON encoder with a clear
  error. Future text/Markdown/binary encoders can support other media types.
- Keep output file name validation strict.
- Ensure diagnostics redaction still prevents raw source, prompt, reference,
  schema, and payload bytes from appearing in ordinary errors or manifests.

Tests to update or add:

- JSON output writes lane output files and index entries deterministically.
- JSON output rejects invalid JSON bytes and unsupported media types.
- Rejected output records are written without raw payload bytes.
- Manifest includes raw-output provenance and retry attempt counts.
- Diagnostics tests still prove sensitive/large content is not emitted.

Completion criteria:

- Durable output no longer assumes artifact candidates.
- JSON output remains deterministic and safe.

## Stage 7: Documentation, Examples, And Full Validation

After behavior is implemented, update canonical current-behavior docs.

Implementation tasks:

- Update `docs/internal/pipeline.md` for the raw data model.
- Update `docs/internal/modules.md` for new module contracts.
- Update `docs/internal/llm.md` for merge-stage LLM/profile plumbing.
- Update `docs/config.md` for:
  - integer source-unit expectations where config examples include source refs;
  - merge references;
  - module `retries`;
  - LLM profile override scope.
- Update `docs/cli.md` for merge reference selectors and profile behavior.
- Update `docs/integrations/json-output.md` for lane output files.
- Update `docs/integrations/dnd-spell-artifacts.md`; if the artifact-specific
  contract is no longer durable, rename or replace it with a D&D spell raw-output
  integration doc.
- Update `docs/troubleshooting.md` for common raw-output, media-type, retry, and
  validation failures.
- Update examples so Seriatim segment IDs are positive integers and expected
  output shape matches lane-normalized output.
- Replace `docs/roadmap/implementation.md` with a completed-note document only
  after this implementation is finished and reviewed.
- Leave `docs/roadmap/pipeline.md` as target-state roadmap context until the
  feature is implemented, then move implemented behavior into canonical docs and
  remove stale roadmap language.

Validation commands:

```sh
go test ./internal/core/source
go test ./internal/framework/contracts
go test ./internal/framework/pipeline
go test ./internal/core/config
go test ./internal/cli
go test ./internal/modules/input/seriatim
go test ./internal/modules/chunk/generic
go test ./internal/modules/chunk/dnd/scenes
go test ./internal/modules/extract/dnd/spells
go test ./internal/modules/merge/appendorder
go test ./internal/modules/normalize/noop
go test ./internal/modules/output/json
go test ./...
go vet ./...
go build ./cmd/notarius
```

Documentation inspection:

```sh
rg -n "ArtifactCandidate|approved artifacts|spell artifact|start_unit_id\": \"|end_unit_id\": \"" docs examples internal
rg -n "chunk`, `extract`, and `normalize|deterministic-only stage|json.RawMessage" docs/roadmap docs/internal docs/config.md docs/cli.md
```

Expected result:

- Remaining `ArtifactCandidate` mentions are only in deliberately retained
  compatibility code or historical roadmap context.
- Current-behavior docs describe raw lane outputs, integer source-unit IDs,
  merge LLM/reference support, and retry behavior.