From 11d81870523b8846c00c8dc62ad9e6ec86fc6ce1 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 5 Jul 2026 08:40:24 -0500 Subject: [PATCH] Update documentation to reflect the implemented chunking module --- docs/integrations/json-output.md | 8 +- docs/roadmap/chunk.md | 104 ------------- docs/roadmap/implementation.md | 245 ------------------------------- 3 files changed, 5 insertions(+), 352 deletions(-) delete mode 100644 docs/roadmap/chunk.md delete mode 100644 docs/roadmap/implementation.md diff --git a/docs/integrations/json-output.md b/docs/integrations/json-output.md index 12f8305..7a354cf 100644 --- a/docs/integrations/json-output.md +++ b/docs/integrations/json-output.md @@ -57,15 +57,15 @@ approved. "pipeline_id": "dnd-session", "pipeline_digest": "sha256:...", "input_module": "seriatim", - "chunker": "generic", + "chunker": "dnd/scenes", "module_metadata": { "chunker": { "prompt_id": "dnd.scenes", "prompt_version": "v1", "prompt_sha256": "sha256:...", "response_schema_key": "dnd_scenes", - "response_schema_id": "schema-dnd-scenes", - "response_schema_name": "dnd_scenes", + "response_schema_id": "notarius.dnd.scenes", + "response_schema_name": "notarius_dnd_scenes_v1", "response_schema_version": "v1", "response_schema_sha256": "sha256:..." } @@ -98,6 +98,8 @@ approved. Fields with empty values may be omitted by JSON encoding. +`module_metadata` is omitted when no singleton module provides metadata. + `validation_status` is `approved` when no candidates were rejected and `rejected` when one or more candidates were rejected. diff --git a/docs/roadmap/chunk.md b/docs/roadmap/chunk.md deleted file mode 100644 index 73c7d4f..0000000 --- a/docs/roadmap/chunk.md +++ /dev/null @@ -1,104 +0,0 @@ -# Chunk Module Roadmap - -Current Notarius behavior is documented in the canonical README, CLI, -configuration, operations, internal, and integration docs. This roadmap records -future chunk-module behavior only. - -## Goal - -Chunk modules should be a clear module-author boundary, and LLM-backed chunking -should be a first-class capability. - -The immediate target is a D&D-specific scene chunker that divides transcript -source units into coherent scenes before extraction. The broader target is that -any chunk module can be implemented as a black box when it satisfies the -framework chunk contract. - -## Target Chunk Contract - -The framework chunk contract should support deterministic and LLM-backed -chunkers through the same module interface. - -Chunkers should receive runtime dependencies from the runner, including the -structured LLM client when a chunker needs model calls. Chunkers should not -construct provider clients internally. - -The framework should validate these result invariants for every chunk module: - -- chunk IDs are non-empty and unique within a run; -- each chunk references the input source document ID; -- chunk indexes are deterministic and sequential in returned order; -- each chunk contains at least one source unit; -- each chunk source unit comes from the source document; -- source units within each chunk appear in source-document order. - -The framework should not require complete source-unit coverage and should not -forbid overlap between chunks. Individual chunk modules may enforce stricter -policies, such as full coverage or non-overlap, when those policies are part of -the module's own contract. - -Chunk metadata should remain flexible and module-owned. Framework code should -preserve chunk metadata and pass it to downstream modules, but it should not -adopt transcript-specific or D&D-specific metadata fields. - -## D&D Scene Chunker Target - -The D&D scene chunker should live under -`internal/modules/chunk/dnd/scenes` and use the module key `dnd/scenes`. - -It should require transcript source capabilities and provide the generic -`chunks` capability plus a scene-specific chunk capability. D&D scene-boundary -prompt logic, response-schema interpretation, and stricter scene policies belong -inside the module. - -The module should use the framework structured LLM client for scene-boundary -detection. The model response should describe source-unit boundaries and useful -scene metadata; the Go module should validate the response and convert it into -`contracts.SourceChunk` values. - -For `dnd/scenes`, the module-owned policy should be: - -- cover the full source document from first source unit to last source unit; -- return sequential, contiguous, non-overlapping scenes; -- use exact source-unit IDs for boundaries; -- fail with actionable errors for malformed model output rather than silently - falling back to a generic chunker. - -Scene chunk IDs and indexes should be assigned by the module, not trusted from -model output. Useful scene information should be stored in chunk metadata, such -as title, primary mode, participants, summary, boundary note, and boundary -confidence. Overall boundary caveats should be surfaced as chunker warnings. - -## Draft Asset Target - -Initial D&D scene chunker prompt and schema drafts exist under -`internal/modules/chunk/dnd/scenes/assets`. They should be revised before the -module is implemented. - -The response schema should be versioned and named consistently with existing -module-owned response schemas, such as `dnd_scenes.v1.json`, with a schema key, -schema ID, schema version, and OpenAI-compatible response schema name. - -Boundary fields should use source-unit ID strings, not integer segment IDs. -The schema should focus on boundary and metadata decisions rather than final -framework chunk fields. Prompt terminology and schema terminology should match -exactly, including primary mode enum values and boundary field names. - -The user prompt should be a Go template that includes source document ID, -chunking scope, ordered source units, and selected metadata such as speaker and -timestamps when available. It may contain D&D-specific scene guidance, but it -should not imply that the framework itself is transcript-specific. - -## Documentation Target - -Current-behavior docs should be updated only after the corresponding behavior is -implemented. - -Internal module-author documentation should eventually define the chunk module -API, including `Chunker`, `ChunkRequest`, `ChunkResult`, `SourceChunk`, -validation invariants, warning semantics, LLM-backed chunker expectations, -module specs, capability guidance, and option parsing expectations. - -When `dnd/scenes` becomes production behavior, configuration, CLI, internal -module, and troubleshooting docs should describe the implemented module and its -failure modes. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 2347660..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,245 +0,0 @@ -# Chunk Follow-Up Implementation Plan - -This plan addresses review findings from the first `dnd/scenes` implementation. -It is written for an LLM coding agent that will implement each stage in order. - -Before beginning any stage, review: - -- `docs/policy/architecture.md` -- `docs/policy/development.md` -- `docs/policy/documentation.md` -- `docs/roadmap/chunk.md` - -Do not move planned behavior into non-roadmap docs until the corresponding code -is implemented. Do not revert unrelated user changes. - -## Goals - -- Record chunker prompt and response-schema provenance in run manifests. -- Ensure downstream extractors receive canonical source units from the source - document, not chunker-mutated unit payloads. -- Prevent empty or whitespace-only scene caveats from becoming warnings. - -## Stage 1: Run-Manifest Module Metadata - -Goal: make non-lane module provenance auditable without adding -chunker-specific fields or D&D-specific framework behavior. - -Design decision: - -- Add a generic top-level run manifest metadata map for singleton pipeline - modules: - -```go -ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"` -``` - -- Use stable stage keys: - - `input` - - `chunker` - - `output` -- Keep existing artifact lane metadata under `ArtifactLaneManifest.Metadata`. - Do not move extractor, merger, normalizer, or validator metadata into the - top-level map in this stage. -- Record metadata only when a built module implements - `contracts.ManifestMetadataProvider` and returns non-empty metadata. -- Continue to reject raw prompts, raw response schemas, source text, provider - payloads, and secrets from manifest metadata by convention and tests. - -Code changes: - -- Add `ModuleMetadata` to `internal/core/artifacts.RunManifest`. -- Add a small helper in `internal/framework/pipeline/runner.go` to attach - top-level module metadata by stage key. -- After building the input adapter, chunker, and output encoder, call that - helper with keys `input`, `chunker`, and `output` respectively. -- Keep `setLaneManifestMetadata` for lane-owned modules. If practical, share - metadata cloning logic with the new helper. -- Ensure failed runs that already have a manifest also retain any metadata - collected before the failure. - -Tests: - -- Add pipeline runner tests proving top-level metadata is recorded for a fake - chunker that implements `ManifestMetadataProvider`. -- Add a test proving the existing lane metadata behavior remains unchanged. -- Add a CLI or output integration test proving a `dnd/scenes` run manifest - contains `module_metadata.chunker` with prompt/schema provenance. -- Add a negative assertion that raw prompt text, raw schema JSON, source text, - provider payloads, and API key-like fields are not present in the scene - chunker metadata. - -Documentation: - -- Update `docs/internal/pipeline.md` to describe top-level metadata for input, - chunker, and output modules, and lane metadata for lane modules. -- Update `docs/internal/modules.md` to say that `dnd/scenes` prompt/schema - provenance appears under `module_metadata.chunker`. -- Update `docs/integrations/json-output.md` and `docs/operations.md` if their - manifest descriptions need to mention `module_metadata`. - -Validation: - -```sh -go test ./internal/core/artifacts -go test ./internal/framework/pipeline -go test ./internal/cli -go test ./internal/modules/chunk/dnd/scenes -``` - -Stage completion criteria: - -- `dnd/scenes` prompt/schema provenance is visible in durable - `manifest.json` and diagnostics `run-manifest.json`. -- Existing artifact lane metadata remains in the same JSON location as before. - -## Stage 2: Canonical Source Units In Chunk Results - -Goal: preserve the chunker boundary contract while ensuring extractors always -consume source document units, not rewritten units supplied by a chunker. - -Design decision: - -- Keep the chunker contract expressed in terms of `contracts.SourceChunk`. -- Continue validating chunk IDs, source IDs, indexes, unit membership, unit - uniqueness within each chunk, and source-ordering. -- After validation, canonicalize chunk units by replacing each returned - `SourceUnit` with a defensive copy of the matching source document unit. -- Preserve `SourceChunk.Metadata` as module-owned chunk metadata. -- Do not require full source coverage and do not reject overlap between chunks. -- Do not preserve chunker-mutated per-unit text, kind, or metadata. A chunker - that wants to add scene-level information must use `SourceChunk.Metadata`. - -Code changes: - -- Replace or extend `validateChunkResult` in - `internal/framework/pipeline/chunk_validation.go` so it returns canonical - chunks, for example: - -```go -func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []contracts.SourceChunk) ([]contracts.SourceChunk, error) -``` - -- Build a source-unit lookup from the validated source document. -- For each chunk: - - validate the existing generic invariants; - - copy chunk ID, source ID, index, and chunk metadata; - - replace the unit slice with cloned source units from the source document in - the returned boundary/order. -- Update the runner to use canonical chunks for all downstream extraction and - merge behavior. -- Ensure chunk metadata is cloned so later module or caller mutation cannot - affect runner state. -- Keep the implementation source-agnostic. Do not inspect transcript-specific - metadata keys. - -Tests: - -- Add a runner test where a fake chunker returns a valid unit ID with mutated - text, kind, and unit metadata. Assert the extractor receives the original - source document unit values. -- Add a runner test proving chunk metadata survives canonicalization and is not - aliased to the chunker-returned map. -- Keep existing tests for invalid chunk IDs, duplicate IDs, wrong source ID, - wrong index, empty units, repeated unit IDs, unknown unit IDs, out-of-order - units, partial coverage, and overlap. -- Add or update tests so generic chunking still behaves unchanged. - -Documentation: - -- Update internal chunk contract docs to state that source units in chunks are - canonicalized from the source document by ID before extractors run. -- Document that chunk metadata is the supported mechanism for passing - chunker-owned context to extractors. - -Validation: - -```sh -go test ./internal/framework/pipeline -go test ./internal/modules/chunk/generic -go test ./internal/modules/chunk/dnd/scenes -``` - -Stage completion criteria: - -- Extractors cannot observe chunker-rewritten source-unit text, kind, or unit - metadata. -- Chunker-owned scene metadata still reaches extractors through - `SourceChunk.Metadata`. - -## Stage 3: Scene Caveat Hygiene - -Goal: ensure model caveats become useful warnings and never produce blank -warnings that can confuse operators or affect diagnostics retention. - -Design decision: - -- Require caveat strings to be non-empty after trimming. -- Treat whitespace-only caveats as malformed structured output rather than - silently dropping them. This is consistent with the `dnd/scenes` policy of - failing explicitly for malformed model output. -- Store warning messages as trimmed caveat text. - -Code changes: - -- Update `internal/modules/chunk/dnd/scenes/assets/schemas/dnd_scenes.v1.json` - so `boundary_caveats.items` has `minLength: 1`. -- Update `warningsFromCaveats` or response validation in - `internal/modules/chunk/dnd/scenes/chunker.go` to trim caveats and reject - empty results with a module-prefixed malformed-output error. -- Prefer validating caveats before constructing chunks so all malformed response - checks happen together. - -Tests: - -- Add schema tests proving `boundary_caveats` items require non-empty strings. -- Add chunker tests proving: - - caveat warning messages are trimmed; - - whitespace-only caveats fail explicitly; - - valid caveats still produce `scene_boundary_caveat` warnings. -- Keep existing warning tests passing. - -Documentation: - -- Update `docs/internal/modules.md` and `docs/troubleshooting.md` if needed to - mention that malformed caveats are treated as malformed structured output. - -Validation: - -```sh -go test ./internal/modules/chunk/dnd/scenes -go test ./internal/cli -``` - -Stage completion criteria: - -- No blank warnings can be emitted from `dnd/scenes` boundary caveats. -- Valid caveats remain visible as warnings. - -## Stage 4: Full Verification - -Goal: verify the follow-up work across contracts, production wiring, -documentation, and the command entry point. - -Run: - -```sh -go test ./... -go vet ./... -go build ./cmd/notarius -``` - -Inspect or test representative output manifests: - -- `manifest.json` includes `module_metadata.chunker` for a `dnd/scenes` run; -- `module_metadata.chunker` contains prompt and response-schema provenance; -- no raw prompt, raw schema, source text, provider payload, or secret appears in - module metadata; -- artifact lane metadata remains under `artifact_lanes[].metadata`; -- `warnings.json` contains trimmed scene caveats and no blank caveat warnings. - -Stage completion criteria: - -- Full validation commands pass. -- Current-behavior docs match implemented behavior. -- Any remaining planned or deferred behavior stays under `docs/roadmap/`.