notarius/docs/roadmap/implementation.md

Chunk Follow-Up Implementation Plan

This plan addresses review findings from the first dnd/scenes implementation. It is written for an LLM coding agent that will implement each stage in order.

Before beginning any stage, review:

• docs/policy/architecture.md
• docs/policy/development.md
• docs/policy/documentation.md
• docs/roadmap/chunk.md

Do not move planned behavior into non-roadmap docs until the corresponding code is implemented. Do not revert unrelated user changes.

Goals

• Record chunker prompt and response-schema provenance in run manifests.
• Ensure downstream extractors receive canonical source units from the source document, not chunker-mutated unit payloads.
• Prevent empty or whitespace-only scene caveats from becoming warnings.

Stage 1: Run-Manifest Module Metadata

Goal: make non-lane module provenance auditable without adding chunker-specific fields or D&D-specific framework behavior.

Design decision:

• Add a generic top-level run manifest metadata map for singleton pipeline modules:

ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"`

• Use stable stage keys:
  • input
  • chunker
  • output
• Keep existing artifact lane metadata under ArtifactLaneManifest.Metadata. Do not move extractor, merger, normalizer, or validator metadata into the top-level map in this stage.
• Record metadata only when a built module implements contracts.ManifestMetadataProvider and returns non-empty metadata.
• Continue to reject raw prompts, raw response schemas, source text, provider payloads, and secrets from manifest metadata by convention and tests.

Code changes:

• Add ModuleMetadata to internal/core/artifacts.RunManifest.
• Add a small helper in internal/framework/pipeline/runner.go to attach top-level module metadata by stage key.
• After building the input adapter, chunker, and output encoder, call that helper with keys input, chunker, and output respectively.
• Keep setLaneManifestMetadata for lane-owned modules. If practical, share metadata cloning logic with the new helper.
• Ensure failed runs that already have a manifest also retain any metadata collected before the failure.

Tests:

• Add pipeline runner tests proving top-level metadata is recorded for a fake chunker that implements ManifestMetadataProvider.
• Add a test proving the existing lane metadata behavior remains unchanged.
• Add a CLI or output integration test proving a dnd/scenes run manifest contains module_metadata.chunker with prompt/schema provenance.
• Add a negative assertion that raw prompt text, raw schema JSON, source text, provider payloads, and API key-like fields are not present in the scene chunker metadata.

Documentation:

• Update docs/internal/pipeline.md to describe top-level metadata for input, chunker, and output modules, and lane metadata for lane modules.
• Update docs/internal/modules.md to say that dnd/scenes prompt/schema provenance appears under module_metadata.chunker.
• Update docs/integrations/json-output.md and docs/operations.md if their manifest descriptions need to mention module_metadata.

Validation:

go test ./internal/core/artifacts
go test ./internal/framework/pipeline
go test ./internal/cli
go test ./internal/modules/chunk/dnd/scenes

Stage completion criteria:

• dnd/scenes prompt/schema provenance is visible in durable manifest.json and diagnostics run-manifest.json.
• Existing artifact lane metadata remains in the same JSON location as before.

Stage 2: Canonical Source Units In Chunk Results

Goal: preserve the chunker boundary contract while ensuring extractors always consume source document units, not rewritten units supplied by a chunker.

Design decision:

• Keep the chunker contract expressed in terms of contracts.SourceChunk.
• Continue validating chunk IDs, source IDs, indexes, unit membership, unit uniqueness within each chunk, and source-ordering.
• After validation, canonicalize chunk units by replacing each returned SourceUnit with a defensive copy of the matching source document unit.
• Preserve SourceChunk.Metadata as module-owned chunk metadata.
• Do not require full source coverage and do not reject overlap between chunks.
• Do not preserve chunker-mutated per-unit text, kind, or metadata. A chunker that wants to add scene-level information must use SourceChunk.Metadata.

Code changes:

• Replace or extend validateChunkResult in internal/framework/pipeline/chunk_validation.go so it returns canonical chunks, for example:

func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []contracts.SourceChunk) ([]contracts.SourceChunk, error)

• Build a source-unit lookup from the validated source document.
• For each chunk:
  • validate the existing generic invariants;
  • copy chunk ID, source ID, index, and chunk metadata;
  • replace the unit slice with cloned source units from the source document in the returned boundary/order.
• Update the runner to use canonical chunks for all downstream extraction and merge behavior.
• Ensure chunk metadata is cloned so later module or caller mutation cannot affect runner state.
• Keep the implementation source-agnostic. Do not inspect transcript-specific metadata keys.

Tests:

• Add a runner test where a fake chunker returns a valid unit ID with mutated text, kind, and unit metadata. Assert the extractor receives the original source document unit values.
• Add a runner test proving chunk metadata survives canonicalization and is not aliased to the chunker-returned map.
• Keep existing tests for invalid chunk IDs, duplicate IDs, wrong source ID, wrong index, empty units, repeated unit IDs, unknown unit IDs, out-of-order units, partial coverage, and overlap.
• Add or update tests so generic chunking still behaves unchanged.

Documentation:

• Update internal chunk contract docs to state that source units in chunks are canonicalized from the source document by ID before extractors run.
• Document that chunk metadata is the supported mechanism for passing chunker-owned context to extractors.

Validation:

go test ./internal/framework/pipeline
go test ./internal/modules/chunk/generic
go test ./internal/modules/chunk/dnd/scenes

Stage completion criteria:

• Extractors cannot observe chunker-rewritten source-unit text, kind, or unit metadata.
• Chunker-owned scene metadata still reaches extractors through SourceChunk.Metadata.

Stage 3: Scene Caveat Hygiene

Goal: ensure model caveats become useful warnings and never produce blank warnings that can confuse operators or affect diagnostics retention.

Design decision:

• Require caveat strings to be non-empty after trimming.
• Treat whitespace-only caveats as malformed structured output rather than silently dropping them. This is consistent with the dnd/scenes policy of failing explicitly for malformed model output.
• Store warning messages as trimmed caveat text.

Code changes:

• Update internal/modules/chunk/dnd/scenes/assets/schemas/dnd_scenes.v1.json so boundary_caveats.items has minLength: 1.
• Update warningsFromCaveats or response validation in internal/modules/chunk/dnd/scenes/chunker.go to trim caveats and reject empty results with a module-prefixed malformed-output error.
• Prefer validating caveats before constructing chunks so all malformed response checks happen together.

Tests:

• Add schema tests proving boundary_caveats items require non-empty strings.
• Add chunker tests proving:
  • caveat warning messages are trimmed;
  • whitespace-only caveats fail explicitly;
  • valid caveats still produce scene_boundary_caveat warnings.
• Keep existing warning tests passing.

Documentation:

• Update docs/internal/modules.md and docs/troubleshooting.md if needed to mention that malformed caveats are treated as malformed structured output.

Validation:

go test ./internal/modules/chunk/dnd/scenes
go test ./internal/cli

Stage completion criteria:

• No blank warnings can be emitted from dnd/scenes boundary caveats.
• Valid caveats remain visible as warnings.

Stage 4: Full Verification

Goal: verify the follow-up work across contracts, production wiring, documentation, and the command entry point.

Run:

go test ./...
go vet ./...
go build ./cmd/notarius

Inspect or test representative output manifests:

• manifest.json includes module_metadata.chunker for a dnd/scenes run;
• module_metadata.chunker contains prompt and response-schema provenance;
• no raw prompt, raw schema, source text, provider payload, or secret appears in module metadata;
• artifact lane metadata remains under artifact_lanes[].metadata;
• warnings.json contains trimmed scene caveats and no blank caveat warnings.

Stage completion criteria:

• Full validation commands pass.
• Current-behavior docs match implemented behavior.
• Any remaining planned or deferred behavior stays under docs/roadmap/.