# Chunk Follow-Up Implementation Plan

This plan addresses review findings from the first `dnd/scenes` implementation.
It is written for an LLM coding agent that will implement each stage in order.

Before beginning any stage, review:

- `docs/policy/architecture.md`
- `docs/policy/development.md`
- `docs/policy/documentation.md`
- `docs/roadmap/chunk.md`

Do not move planned behavior into non-roadmap docs until the corresponding code
is implemented. Do not revert unrelated user changes.

## Goals

- Record chunker prompt and response-schema provenance in run manifests.
- Ensure downstream extractors receive canonical source units from the source
  document, not chunker-mutated unit payloads.
- Prevent empty or whitespace-only scene caveats from becoming warnings.

## Stage 1: Run-Manifest Module Metadata

Goal: make non-lane module provenance auditable without adding
chunker-specific fields or D&D-specific framework behavior.

Design decision:

- Add a generic top-level run manifest metadata map for singleton pipeline
  modules:

```go
ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"`
```

- Use stable stage keys:
  - `input`
  - `chunker`
  - `output`
- Keep existing artifact lane metadata under `ArtifactLaneManifest.Metadata`.
  Do not move extractor, merger, normalizer, or validator metadata into the
  top-level map in this stage.
- Record metadata only when a built module implements
  `contracts.ManifestMetadataProvider` and returns non-empty metadata.
- Continue to reject raw prompts, raw response schemas, source text, provider
  payloads, and secrets from manifest metadata by convention and tests.

Code changes:

- Add `ModuleMetadata` to `internal/core/artifacts.RunManifest`.
- Add a small helper in `internal/framework/pipeline/runner.go` to attach
  top-level module metadata by stage key.
- After building the input adapter, chunker, and output encoder, call that
  helper with keys `input`, `chunker`, and `output` respectively.
- Keep `setLaneManifestMetadata` for lane-owned modules. If practical, share
  metadata cloning logic with the new helper.
- Ensure failed runs that already have a manifest also retain any metadata
  collected before the failure.

Tests:

- Add pipeline runner tests proving top-level metadata is recorded for a fake
  chunker that implements `ManifestMetadataProvider`.
- Add a test proving the existing lane metadata behavior remains unchanged.
- Add a CLI or output integration test proving a `dnd/scenes` run manifest
  contains `module_metadata.chunker` with prompt/schema provenance.
- Add a negative assertion that raw prompt text, raw schema JSON, source text,
  provider payloads, and API key-like fields are not present in the scene
  chunker metadata.

Documentation:

- Update `docs/internal/pipeline.md` to describe top-level metadata for input,
  chunker, and output modules, and lane metadata for lane modules.
- Update `docs/internal/modules.md` to say that `dnd/scenes` prompt/schema
  provenance appears under `module_metadata.chunker`.
- Update `docs/integrations/json-output.md` and `docs/operations.md` if their
  manifest descriptions need to mention `module_metadata`.

Validation:

```sh
go test ./internal/core/artifacts
go test ./internal/framework/pipeline
go test ./internal/cli
go test ./internal/modules/chunk/dnd/scenes
```

Stage completion criteria:

- `dnd/scenes` prompt/schema provenance is visible in durable
  `manifest.json` and diagnostics `run-manifest.json`.
- Existing artifact lane metadata remains in the same JSON location as before.

## Stage 2: Canonical Source Units In Chunk Results

Goal: preserve the chunker boundary contract while ensuring extractors always
consume source document units, not rewritten units supplied by a chunker.

Design decision:

- Keep the chunker contract expressed in terms of `contracts.SourceChunk`.
- Continue validating chunk IDs, source IDs, indexes, unit membership, unit
  uniqueness within each chunk, and source-ordering.
- After validation, canonicalize chunk units by replacing each returned
  `SourceUnit` with a defensive copy of the matching source document unit.
- Preserve `SourceChunk.Metadata` as module-owned chunk metadata.
- Do not require full source coverage and do not reject overlap between chunks.
- Do not preserve chunker-mutated per-unit text, kind, or metadata. A chunker
  that wants to add scene-level information must use `SourceChunk.Metadata`.

Code changes:

- Replace or extend `validateChunkResult` in
  `internal/framework/pipeline/chunk_validation.go` so it returns canonical
  chunks, for example:

```go
func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []contracts.SourceChunk) ([]contracts.SourceChunk, error)
```

- Build a source-unit lookup from the validated source document.
- For each chunk:
  - validate the existing generic invariants;
  - copy chunk ID, source ID, index, and chunk metadata;
  - replace the unit slice with cloned source units from the source document in
    the returned boundary/order.
- Update the runner to use canonical chunks for all downstream extraction and
  merge behavior.
- Ensure chunk metadata is cloned so later module or caller mutation cannot
  affect runner state.
- Keep the implementation source-agnostic. Do not inspect transcript-specific
  metadata keys.

Tests:

- Add a runner test where a fake chunker returns a valid unit ID with mutated
  text, kind, and unit metadata. Assert the extractor receives the original
  source document unit values.
- Add a runner test proving chunk metadata survives canonicalization and is not
  aliased to the chunker-returned map.
- Keep existing tests for invalid chunk IDs, duplicate IDs, wrong source ID,
  wrong index, empty units, repeated unit IDs, unknown unit IDs, out-of-order
  units, partial coverage, and overlap.
- Add or update tests so generic chunking still behaves unchanged.

Documentation:

- Update internal chunk contract docs to state that source units in chunks are
  canonicalized from the source document by ID before extractors run.
- Document that chunk metadata is the supported mechanism for passing
  chunker-owned context to extractors.

Validation:

```sh
go test ./internal/framework/pipeline
go test ./internal/modules/chunk/generic
go test ./internal/modules/chunk/dnd/scenes
```

Stage completion criteria:

- Extractors cannot observe chunker-rewritten source-unit text, kind, or unit
  metadata.
- Chunker-owned scene metadata still reaches extractors through
  `SourceChunk.Metadata`.

## Stage 3: Scene Caveat Hygiene

Goal: ensure model caveats become useful warnings and never produce blank
warnings that can confuse operators or affect diagnostics retention.

Design decision:

- Require caveat strings to be non-empty after trimming.
- Treat whitespace-only caveats as malformed structured output rather than
  silently dropping them. This is consistent with the `dnd/scenes` policy of
  failing explicitly for malformed model output.
- Store warning messages as trimmed caveat text.

Code changes:

- Update `internal/modules/chunk/dnd/scenes/assets/schemas/dnd_scenes.v1.json`
  so `boundary_caveats.items` has `minLength: 1`.
- Update `warningsFromCaveats` or response validation in
  `internal/modules/chunk/dnd/scenes/chunker.go` to trim caveats and reject
  empty results with a module-prefixed malformed-output error.
- Prefer validating caveats before constructing chunks so all malformed response
  checks happen together.

Tests:

- Add schema tests proving `boundary_caveats` items require non-empty strings.
- Add chunker tests proving:
  - caveat warning messages are trimmed;
  - whitespace-only caveats fail explicitly;
  - valid caveats still produce `scene_boundary_caveat` warnings.
- Keep existing warning tests passing.

Documentation:

- Update `docs/internal/modules.md` and `docs/troubleshooting.md` if needed to
  mention that malformed caveats are treated as malformed structured output.

Validation:

```sh
go test ./internal/modules/chunk/dnd/scenes
go test ./internal/cli
```

Stage completion criteria:

- No blank warnings can be emitted from `dnd/scenes` boundary caveats.
- Valid caveats remain visible as warnings.

## Stage 4: Full Verification

Goal: verify the follow-up work across contracts, production wiring,
documentation, and the command entry point.

Run:

```sh
go test ./...
go vet ./...
go build ./cmd/notarius
```

Inspect or test representative output manifests:

- `manifest.json` includes `module_metadata.chunker` for a `dnd/scenes` run;
- `module_metadata.chunker` contains prompt and response-schema provenance;
- no raw prompt, raw schema, source text, provider payload, or secret appears in
  module metadata;
- artifact lane metadata remains under `artifact_lanes[].metadata`;
- `warnings.json` contains trimmed scene caveats and no blank caveat warnings.

Stage completion criteria:

- Full validation commands pass.
- Current-behavior docs match implemented behavior.
- Any remaining planned or deferred behavior stays under `docs/roadmap/`.