Update documentation to reflect the implemented chunking module

.gitignore

@@ -1,3 +1,6 @@
+# build artifacts
+./notarius
+
 # ---> Go
 # If you prefer the allow list template instead of the deny list, see community template:
 # https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
@@ -47,7 +50,8 @@ go.work.sum
 .LSOverride
 
 # Icon must end with two \r
-Icon
+Icon
+
 
 # Thumbnails
 ._*

docs/integrations/json-output.md

@@ -57,15 +57,15 @@ approved.
   "pipeline_id": "dnd-session",
   "pipeline_digest": "sha256:...",
   "input_module": "seriatim",
-  "chunker": "generic",
+  "chunker": "dnd/scenes",
   "module_metadata": {
     "chunker": {
       "prompt_id": "dnd.scenes",
       "prompt_version": "v1",
       "prompt_sha256": "sha256:...",
       "response_schema_key": "dnd_scenes",
-      "response_schema_id": "schema-dnd-scenes",
+      "response_schema_id": "notarius.dnd.scenes",
-      "response_schema_name": "dnd_scenes",
+      "response_schema_name": "notarius_dnd_scenes_v1",
       "response_schema_version": "v1",
       "response_schema_sha256": "sha256:..."
     }
@@ -98,6 +98,8 @@ approved.
 
 Fields with empty values may be omitted by JSON encoding.
 
+`module_metadata` is omitted when no singleton module provides metadata.
+
 `validation_status` is `approved` when no candidates were rejected and
 `rejected` when one or more candidates were rejected.
 

docs/roadmap/chunk.md

@@ -1,104 +0,0 @@
-# Chunk Module Roadmap
-
-Current Notarius behavior is documented in the canonical README, CLI,
-configuration, operations, internal, and integration docs. This roadmap records
-future chunk-module behavior only.
-
-## Goal
-
-Chunk modules should be a clear module-author boundary, and LLM-backed chunking
-should be a first-class capability.
-
-The immediate target is a D&D-specific scene chunker that divides transcript
-source units into coherent scenes before extraction. The broader target is that
-any chunk module can be implemented as a black box when it satisfies the
-framework chunk contract.
-
-## Target Chunk Contract
-
-The framework chunk contract should support deterministic and LLM-backed
-chunkers through the same module interface.
-
-Chunkers should receive runtime dependencies from the runner, including the
-structured LLM client when a chunker needs model calls. Chunkers should not
-construct provider clients internally.
-
-The framework should validate these result invariants for every chunk module:
-
-- chunk IDs are non-empty and unique within a run;
-- each chunk references the input source document ID;
-- chunk indexes are deterministic and sequential in returned order;
-- each chunk contains at least one source unit;
-- each chunk source unit comes from the source document;
-- source units within each chunk appear in source-document order.
-
-The framework should not require complete source-unit coverage and should not
-forbid overlap between chunks. Individual chunk modules may enforce stricter
-policies, such as full coverage or non-overlap, when those policies are part of
-the module's own contract.
-
-Chunk metadata should remain flexible and module-owned. Framework code should
-preserve chunk metadata and pass it to downstream modules, but it should not
-adopt transcript-specific or D&D-specific metadata fields.
-
-## D&D Scene Chunker Target
-
-The D&D scene chunker should live under
-`internal/modules/chunk/dnd/scenes` and use the module key `dnd/scenes`.
-
-It should require transcript source capabilities and provide the generic
-`chunks` capability plus a scene-specific chunk capability. D&D scene-boundary
-prompt logic, response-schema interpretation, and stricter scene policies belong
-inside the module.
-
-The module should use the framework structured LLM client for scene-boundary
-detection. The model response should describe source-unit boundaries and useful
-scene metadata; the Go module should validate the response and convert it into
-`contracts.SourceChunk` values.
-
-For `dnd/scenes`, the module-owned policy should be:
-
-- cover the full source document from first source unit to last source unit;
-- return sequential, contiguous, non-overlapping scenes;
-- use exact source-unit IDs for boundaries;
-- fail with actionable errors for malformed model output rather than silently
-  falling back to a generic chunker.
-
-Scene chunk IDs and indexes should be assigned by the module, not trusted from
-model output. Useful scene information should be stored in chunk metadata, such
-as title, primary mode, participants, summary, boundary note, and boundary
-confidence. Overall boundary caveats should be surfaced as chunker warnings.
-
-## Draft Asset Target
-
-Initial D&D scene chunker prompt and schema drafts exist under
-`internal/modules/chunk/dnd/scenes/assets`. They should be revised before the
-module is implemented.
-
-The response schema should be versioned and named consistently with existing
-module-owned response schemas, such as `dnd_scenes.v1.json`, with a schema key,
-schema ID, schema version, and OpenAI-compatible response schema name.
-
-Boundary fields should use source-unit ID strings, not integer segment IDs.
-The schema should focus on boundary and metadata decisions rather than final
-framework chunk fields. Prompt terminology and schema terminology should match
-exactly, including primary mode enum values and boundary field names.
-
-The user prompt should be a Go template that includes source document ID,
-chunking scope, ordered source units, and selected metadata such as speaker and
-timestamps when available. It may contain D&D-specific scene guidance, but it
-should not imply that the framework itself is transcript-specific.
-
-## Documentation Target
-
-Current-behavior docs should be updated only after the corresponding behavior is
-implemented.
-
-Internal module-author documentation should eventually define the chunk module
-API, including `Chunker`, `ChunkRequest`, `ChunkResult`, `SourceChunk`,
-validation invariants, warning semantics, LLM-backed chunker expectations,
-module specs, capability guidance, and option parsing expectations.
-
-When `dnd/scenes` becomes production behavior, configuration, CLI, internal
-module, and troubleshooting docs should describe the implemented module and its
-failure modes.

docs/roadmap/implementation.md

@@ -1,245 +0,0 @@
-# Chunk Follow-Up Implementation Plan
-
-This plan addresses review findings from the first `dnd/scenes` implementation.
-It is written for an LLM coding agent that will implement each stage in order.
-
-Before beginning any stage, review:
-
-- `docs/policy/architecture.md`
-- `docs/policy/development.md`
-- `docs/policy/documentation.md`
-- `docs/roadmap/chunk.md`
-
-Do not move planned behavior into non-roadmap docs until the corresponding code
-is implemented. Do not revert unrelated user changes.
-
-## Goals
-
-- Record chunker prompt and response-schema provenance in run manifests.
-- Ensure downstream extractors receive canonical source units from the source
-  document, not chunker-mutated unit payloads.
-- Prevent empty or whitespace-only scene caveats from becoming warnings.
-
-## Stage 1: Run-Manifest Module Metadata
-
-Goal: make non-lane module provenance auditable without adding
-chunker-specific fields or D&D-specific framework behavior.
-
-Design decision:
-
-- Add a generic top-level run manifest metadata map for singleton pipeline
-  modules:
-
-```go
-ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"`
-```
-
-- Use stable stage keys:
-  - `input`
-  - `chunker`
-  - `output`
-- Keep existing artifact lane metadata under `ArtifactLaneManifest.Metadata`.
-  Do not move extractor, merger, normalizer, or validator metadata into the
-  top-level map in this stage.
-- Record metadata only when a built module implements
-  `contracts.ManifestMetadataProvider` and returns non-empty metadata.
-- Continue to reject raw prompts, raw response schemas, source text, provider
-  payloads, and secrets from manifest metadata by convention and tests.
-
-Code changes:
-
-- Add `ModuleMetadata` to `internal/core/artifacts.RunManifest`.
-- Add a small helper in `internal/framework/pipeline/runner.go` to attach
-  top-level module metadata by stage key.
-- After building the input adapter, chunker, and output encoder, call that
-  helper with keys `input`, `chunker`, and `output` respectively.
-- Keep `setLaneManifestMetadata` for lane-owned modules. If practical, share
-  metadata cloning logic with the new helper.
-- Ensure failed runs that already have a manifest also retain any metadata
-  collected before the failure.
-
-Tests:
-
-- Add pipeline runner tests proving top-level metadata is recorded for a fake
-  chunker that implements `ManifestMetadataProvider`.
-- Add a test proving the existing lane metadata behavior remains unchanged.
-- Add a CLI or output integration test proving a `dnd/scenes` run manifest
-  contains `module_metadata.chunker` with prompt/schema provenance.
-- Add a negative assertion that raw prompt text, raw schema JSON, source text,
-  provider payloads, and API key-like fields are not present in the scene
-  chunker metadata.
-
-Documentation:
-
-- Update `docs/internal/pipeline.md` to describe top-level metadata for input,
-  chunker, and output modules, and lane metadata for lane modules.
-- Update `docs/internal/modules.md` to say that `dnd/scenes` prompt/schema
-  provenance appears under `module_metadata.chunker`.
-- Update `docs/integrations/json-output.md` and `docs/operations.md` if their
-  manifest descriptions need to mention `module_metadata`.
-
-Validation:
-
-```sh
-go test ./internal/core/artifacts
-go test ./internal/framework/pipeline
-go test ./internal/cli
-go test ./internal/modules/chunk/dnd/scenes
-```
-
-Stage completion criteria:
-
-- `dnd/scenes` prompt/schema provenance is visible in durable
-  `manifest.json` and diagnostics `run-manifest.json`.
-- Existing artifact lane metadata remains in the same JSON location as before.
-
-## Stage 2: Canonical Source Units In Chunk Results
-
-Goal: preserve the chunker boundary contract while ensuring extractors always
-consume source document units, not rewritten units supplied by a chunker.
-
-Design decision:
-
-- Keep the chunker contract expressed in terms of `contracts.SourceChunk`.
-- Continue validating chunk IDs, source IDs, indexes, unit membership, unit
-  uniqueness within each chunk, and source-ordering.
-- After validation, canonicalize chunk units by replacing each returned
-  `SourceUnit` with a defensive copy of the matching source document unit.
-- Preserve `SourceChunk.Metadata` as module-owned chunk metadata.
-- Do not require full source coverage and do not reject overlap between chunks.
-- Do not preserve chunker-mutated per-unit text, kind, or metadata. A chunker
-  that wants to add scene-level information must use `SourceChunk.Metadata`.
-
-Code changes:
-
-- Replace or extend `validateChunkResult` in
-  `internal/framework/pipeline/chunk_validation.go` so it returns canonical
-  chunks, for example:
-
-```go
-func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []contracts.SourceChunk) ([]contracts.SourceChunk, error)
-```
-
-- Build a source-unit lookup from the validated source document.
-- For each chunk:
-  - validate the existing generic invariants;
-  - copy chunk ID, source ID, index, and chunk metadata;
-  - replace the unit slice with cloned source units from the source document in
-    the returned boundary/order.
-- Update the runner to use canonical chunks for all downstream extraction and
-  merge behavior.
-- Ensure chunk metadata is cloned so later module or caller mutation cannot
-  affect runner state.
-- Keep the implementation source-agnostic. Do not inspect transcript-specific
-  metadata keys.
-
-Tests:
-
-- Add a runner test where a fake chunker returns a valid unit ID with mutated
-  text, kind, and unit metadata. Assert the extractor receives the original
-  source document unit values.
-- Add a runner test proving chunk metadata survives canonicalization and is not
-  aliased to the chunker-returned map.
-- Keep existing tests for invalid chunk IDs, duplicate IDs, wrong source ID,
-  wrong index, empty units, repeated unit IDs, unknown unit IDs, out-of-order
-  units, partial coverage, and overlap.
-- Add or update tests so generic chunking still behaves unchanged.
-
-Documentation:
-
-- Update internal chunk contract docs to state that source units in chunks are
-  canonicalized from the source document by ID before extractors run.
-- Document that chunk metadata is the supported mechanism for passing
-  chunker-owned context to extractors.
-
-Validation:
-
-```sh
-go test ./internal/framework/pipeline
-go test ./internal/modules/chunk/generic
-go test ./internal/modules/chunk/dnd/scenes
-```
-
-Stage completion criteria:
-
-- Extractors cannot observe chunker-rewritten source-unit text, kind, or unit
-  metadata.
-- Chunker-owned scene metadata still reaches extractors through
-  `SourceChunk.Metadata`.
-
-## Stage 3: Scene Caveat Hygiene
-
-Goal: ensure model caveats become useful warnings and never produce blank
-warnings that can confuse operators or affect diagnostics retention.
-
-Design decision:
-
-- Require caveat strings to be non-empty after trimming.
-- Treat whitespace-only caveats as malformed structured output rather than
-  silently dropping them. This is consistent with the `dnd/scenes` policy of
-  failing explicitly for malformed model output.
-- Store warning messages as trimmed caveat text.
-
-Code changes:
-
-- Update `internal/modules/chunk/dnd/scenes/assets/schemas/dnd_scenes.v1.json`
-  so `boundary_caveats.items` has `minLength: 1`.
-- Update `warningsFromCaveats` or response validation in
-  `internal/modules/chunk/dnd/scenes/chunker.go` to trim caveats and reject
-  empty results with a module-prefixed malformed-output error.
-- Prefer validating caveats before constructing chunks so all malformed response
-  checks happen together.
-
-Tests:
-
-- Add schema tests proving `boundary_caveats` items require non-empty strings.
-- Add chunker tests proving:
-  - caveat warning messages are trimmed;
-  - whitespace-only caveats fail explicitly;
-  - valid caveats still produce `scene_boundary_caveat` warnings.
-- Keep existing warning tests passing.
-
-Documentation:
-
-- Update `docs/internal/modules.md` and `docs/troubleshooting.md` if needed to
-  mention that malformed caveats are treated as malformed structured output.
-
-Validation:
-
-```sh
-go test ./internal/modules/chunk/dnd/scenes
-go test ./internal/cli
-```
-
-Stage completion criteria:
-
-- No blank warnings can be emitted from `dnd/scenes` boundary caveats.
-- Valid caveats remain visible as warnings.
-
-## Stage 4: Full Verification
-
-Goal: verify the follow-up work across contracts, production wiring,
-documentation, and the command entry point.
-
-Run:
-
-```sh
-go test ./...
-go vet ./...
-go build ./cmd/notarius
-```
-
-Inspect or test representative output manifests:
-
-- `manifest.json` includes `module_metadata.chunker` for a `dnd/scenes` run;
-- `module_metadata.chunker` contains prompt and response-schema provenance;
-- no raw prompt, raw schema, source text, provider payload, or secret appears in
-  module metadata;
-- artifact lane metadata remains under `artifact_lanes[].metadata`;
-- `warnings.json` contains trimmed scene caveats and no blank caveat warnings.
-
-Stage completion criteria:
-
-- Full validation commands pass.
-- Current-behavior docs match implemented behavior.
-- Any remaining planned or deferred behavior stays under `docs/roadmap/`.