eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add a staged implementation plan to address gaps from the initial implementation of the scene chunking module

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-05 08:15:46 -05:00
parent 8a5419448f
commit e19cc02c4d
1 changed files with 201 additions and 268 deletions
Download Patch File Download Diff File Expand all files Collapse all files

469
docs/roadmap/implementation.md
View File

@@ -1,8 +1,7 @@
# Chunk Module Implementation Plan # Chunk Follow-Up Implementation Plan
This plan implements the accepted target state in This plan addresses review findings from the first `dnd/scenes` implementation.
[Chunk Module Roadmap](chunk.md). It is written for an LLM coding agent that It is written for an LLM coding agent that will implement each stage in order.
will implement each stage in order.
Before beginning any stage, review: Before beginning any stage, review:
@@ -14,278 +13,213 @@ Before beginning any stage, review:
Do not move planned behavior into non-roadmap docs until the corresponding code Do not move planned behavior into non-roadmap docs until the corresponding code
is implemented. Do not revert unrelated user changes. is implemented. Do not revert unrelated user changes.
## Stage 1: Framework Chunk Contract ## Goals
Goal: make LLM-backed chunking first-class and enforce generic chunk result - Record chunker prompt and response-schema provenance in run manifests.
invariants without adding any D&D-specific framework behavior. - Ensure downstream extractors receive canonical source units from the source
document, not chunker-mutated unit payloads.
- Prevent empty or whitespace-only scene caveats from becoming warnings.
## Stage 1: Run-Manifest Module Metadata
Goal: make non-lane module provenance auditable without adding
chunker-specific fields or D&D-specific framework behavior.
Design decision:
- Add a generic top-level run manifest metadata map for singleton pipeline
modules:
```go
ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"`
```
- Use stable stage keys:
- `input`
- `chunker`
- `output`
- Keep existing artifact lane metadata under `ArtifactLaneManifest.Metadata`.
Do not move extractor, merger, normalizer, or validator metadata into the
top-level map in this stage.
- Record metadata only when a built module implements
`contracts.ManifestMetadataProvider` and returns non-empty metadata.
- Continue to reject raw prompts, raw response schemas, source text, provider
payloads, and secrets from manifest metadata by convention and tests.
Code changes: Code changes:
- Add `LLMClient contracts.StructuredLLMClient` to `contracts.ChunkRequest` in - Add `ModuleMetadata` to `internal/core/artifacts.RunManifest`.
`internal/framework/contracts/contracts.go`. - Add a small helper in `internal/framework/pipeline/runner.go` to attach
- Update `internal/framework/pipeline/runner.go` so the runner passes top-level module metadata by stage key.
`input.LLMClient` to the chunker in `contracts.ChunkRequest`. - After building the input adapter, chunker, and output encoder, call that
- Add framework-level chunk result validation after `chunker.Chunk` returns and helper with keys `input`, `chunker`, and `output` respectively.
before lanes execute. - Keep `setLaneManifestMetadata` for lane-owned modules. If practical, share
- Keep validation source-generic. The validator should reject: metadata cloning logic with the new helper.
- empty chunk ID; - Ensure failed runs that already have a manifest also retain any metadata
- duplicate chunk ID; collected before the failure.
- chunk `SourceID` that does not match the source document ID;
- chunk `Index` that does not match returned order;
- empty chunk units;
- repeated source unit inside one chunk;
- source unit not found in the source document;
- chunk units that do not appear in source-document order.
- The validator must not require complete coverage and must not reject overlap
between different chunks.
- Preserve existing warning behavior: append chunker warnings before returning
chunk errors, as the runner does today.
Documentation changes:
- Update implemented internal docs under `docs/internal/` to define the chunk
module API and validation invariants once the code exists.
- Keep examples and user docs unchanged in this stage unless an existing doc
becomes inaccurate.
Tests: Tests:
- Update contract tests for the new `ChunkRequest.LLMClient` field where useful. - Add pipeline runner tests proving top-level metadata is recorded for a fake
- Add focused pipeline runner tests for each invalid chunk result case listed chunker that implements `ManifestMetadataProvider`.
above. - Add a test proving the existing lane metadata behavior remains unchanged.
- Add runner tests proving partial coverage and overlapping chunks remain - Add a CLI or output integration test proving a `dnd/scenes` run manifest
accepted. contains `module_metadata.chunker` with prompt/schema provenance.
- Run: - Add a negative assertion that raw prompt text, raw schema JSON, source text,
provider payloads, and API key-like fields are not present in the scene
chunker metadata.
Documentation:
- Update `docs/internal/pipeline.md` to describe top-level metadata for input,
chunker, and output modules, and lane metadata for lane modules.
- Update `docs/internal/modules.md` to say that `dnd/scenes` prompt/schema
provenance appears under `module_metadata.chunker`.
- Update `docs/integrations/json-output.md` and `docs/operations.md` if their
manifest descriptions need to mention `module_metadata`.
Validation:
```sh ```sh
go test ./internal/framework/contracts ./internal/framework/pipeline go test ./internal/core/artifacts
```
Stage completion criteria:
- Existing generic chunking still works.
- A fake chunker can receive the structured LLM client through `ChunkRequest`.
- Framework tests prove the accepted generic chunk invariants.
## Stage 2: D&D Scene Assets And Module Skeleton
Goal: revise the draft D&D scene prompt and schema into module-owned assets and
add load/render plumbing without registering production behavior.
Asset decisions:
- Rename `internal/modules/chunk/dnd/scenes/assets/schemas/scene_map.schema.json`
to `internal/modules/chunk/dnd/scenes/assets/schemas/dnd_scenes.v1.json`.
- Use these schema constants unless a code-local naming conflict requires a
mechanical adjustment:
- prompt ID: `dnd.scenes`
- response schema key: `dnd_scenes`
- response schema ID: `notarius.dnd.scenes`
- response schema version: `v1`
- response schema name: `notarius_dnd_scenes_v1`
- Use this structured response shape:
```json
{
"scenes": [
{
"start_unit_id": "seg-001",
"end_unit_id": "seg-010",
"short_title": "Ambush at the gate",
"primary_mode": "Combat",
"main_participants": ["Aria", "Bandit mage"],
"summary": "The party fights the bandit mage at the gate.",
"boundary_note": "The scene begins when combat starts and ends when the immediate threat is resolved.",
"boundary_confidence": "High"
}
],
"boundary_caveats": []
}
```
- Required top-level fields: `scenes`, `boundary_caveats`.
- Required scene fields: `start_unit_id`, `end_unit_id`, `short_title`,
`primary_mode`, `main_participants`, `summary`, `boundary_note`,
`boundary_confidence`.
- Boundary fields are source-unit ID strings, not integers.
- `primary_mode` enum: `Recap`, `Discussion`, `Combat`, `Narrative`.
- `boundary_confidence` enum: `High`, `Medium`, `Low`.
- Keep `additionalProperties: false` throughout the schema.
- Do not include model-authored final chunk IDs or chunk indexes in the schema.
The Go module assigns deterministic chunk IDs and indexes.
Prompt decisions:
- Keep D&D-specific scene guidance in the D&D scene module.
- Make the user prompt a Go template similar to the spell extractor prompt.
- Include source document ID and ordered source units.
- Include selected source-unit metadata when present: `speaker`, `start`, and
`end`.
- Align prompt terms exactly with schema field names and enum values.
- Keep `dnd/scenes` module policy explicit in the prompt: full coverage,
sequential scenes, no gaps, no overlap, exact source-unit IDs.
Code changes:
- Add `assets.go` with an `embed.FS` for prompts and schemas.
- Add `schema.go` with the constants and a `loadResponseSchema` function using
`llm.LoadResponseSchema`, following the pattern in
`internal/modules/extract/dnd/spells/schema.go`.
- Add prompt rendering code using `framework/prompt.Bundle`, following the
pattern in `internal/modules/extract/dnd/spells/prompt.go`.
- Add internal response structs for the schema shape.
- Do not register the module in `internal/cli/catalog.go` in this stage.
Tests:
- Add tests that the schema loads, is valid JSON, has the expected metadata, and
rejects the old integer-boundary assumption through Go-side type expectations.
- Add prompt rendering tests that source unit IDs and selected metadata appear
in the rendered user prompt.
- Run:
```sh
go test ./internal/modules/chunk/dnd/scenes
```
Stage completion criteria:
- The scene schema and prompts are loadable embedded assets.
- The prompt/schema terminology is internally consistent.
- No production catalog behavior changes yet.
## Stage 3: D&D Scene Chunker Implementation
Goal: implement `dnd/scenes` as a contract-compliant chunk module with strict
module-owned validation.
Module decisions:
- Package path: `internal/modules/chunk/dnd/scenes`.
- Package name: `scenes`.
- Module key: `dnd/scenes`.
- `ModuleSpec`:
- `Stage`: `pipeline.StageChunk`
- `Requires`: `source.transcript`
- `Provides`: `chunks`, `chunks.scenes`
- Constructor: `New() *Chunker`.
- Registration function: `Register(registry *pipeline.ChunkerRegistry) error`.
- No module options initially. Reject non-empty options with an actionable
module-prefixed error unless a clear option is implemented in the same stage.
Chunking behavior:
- Validate `context.Context`, source document, non-empty source units, and
non-nil `LLMClient`.
- Render the scene prompt over the full source document.
- Call `LLMClient.CompleteStructured` with:
- `StageName`: `dnd/scenes`
- response schema name and schema JSON from the module schema loader.
- Validate the decoded response before producing chunks:
- `scenes` must be present and non-empty;
- every boundary ID must exist in the source document;
- each scene start must be at or before its end;
- the first scene starts at the first source unit;
- the final scene ends at the final source unit;
- scenes are contiguous in source order;
- scenes do not overlap;
- required metadata fields are non-empty after trimming;
- `main_participants` entries are trimmed and empty entries rejected.
- Assign deterministic chunk fields:
- `ID`: `scene-000001`, `scene-000002`, and so on;
- `SourceID`: source document ID;
- `Index`: zero-based returned order;
- `Units`: defensive copies of the source units in the scene range.
- Store per-scene metadata on each chunk:
- `scene_title`
- `primary_mode`
- `main_participants`
- `summary`
- `boundary_note`
- `boundary_confidence`
- `start_unit_id`
- `end_unit_id`
- `unit_count`
- Convert each `boundary_caveats` entry into a `contracts.Warning` with:
- `Scope`: `dnd/scenes`
- `ReasonCode`: `scene_boundary_caveat`
- `Message`: the caveat text.
- Fail explicitly for malformed model output. Do not fall back to `generic`.
- Implement `contracts.ManifestMetadataProvider` and include prompt and
response-schema provenance without raw prompts, raw schemas, source text, or
secrets.
Tests:
- Registration and `ModuleSpec`.
- Successful chunking from a fake LLM response.
- Prompt request uses the expected schema name and schema JSON.
- Caveats become warnings.
- Defensive copy behavior for source units and metadata.
- Errors for nil context, nil source, invalid source, nil LLM client, empty
model scenes, unknown boundary ID, out-of-order boundaries, gaps, overlap,
incomplete coverage, empty metadata fields, and non-empty unsupported options.
- Manifest metadata contains prompt/schema provenance.
- Run:
```sh
go test ./internal/modules/chunk/dnd/scenes
go test ./internal/framework/pipeline go test ./internal/framework/pipeline
```
Stage completion criteria:
- `dnd/scenes` works in focused tests with fake LLM clients.
- It is still not production-registered unless Stage 4 is completed.
## Stage 4: Production Registration And Implemented Docs
Goal: make `dnd/scenes` available in production configuration and document only
the behavior that now exists.
Code changes:
- Register `dnd/scenes` in `internal/cli/catalog.go`.
- Add or update catalog/default module tests so the production catalog exposes
the new chunk module.
- Add CLI/config validation tests proving a pipeline can select
`chunk: dnd/scenes`.
- Do not change the existing maintained example config unless the related CLI
fixture tests are updated to keep it loadable and useful.
Documentation changes:
- Update `docs/config.md` implemented production module tables and chunk module
notes.
- Update `docs/cli.md` implemented production module list.
- Update `docs/internal/modules.md` with `dnd/scenes` behavior, capabilities,
metadata, and failure policy.
- Update or add internal chunk-module documentation if Stage 1 did not already
create a clear API reference.
- Update `docs/troubleshooting.md` for common scene chunker failures:
malformed model output, invalid boundaries, incomplete coverage, and provider
failures during chunking.
- Keep roadmap docs for any deferred options or future prompt tuning.
Tests:
```sh
go test ./internal/cli go test ./internal/cli
go test ./internal/core/config
go test ./internal/framework/pipeline
go test ./internal/modules/chunk/dnd/scenes go test ./internal/modules/chunk/dnd/scenes
``` ```
Stage completion criteria: Stage completion criteria:
- Config resolution can bind `dnd/scenes`. - `dnd/scenes` prompt/schema provenance is visible in durable
- User and internal docs describe the implemented module accurately. `manifest.json` and diagnostics `run-manifest.json`.
- Existing examples and CLI docs remain truthful. - Existing artifact lane metadata remains in the same JSON location as before.
## Stage 5: Full Verification ## Stage 2: Canonical Source Units In Chunk Results
Goal: verify the complete feature across contracts, production wiring, docs, and Goal: preserve the chunker boundary contract while ensuring extractors always
the command entry point. consume source document units, not rewritten units supplied by a chunker.
Design decision:
- Keep the chunker contract expressed in terms of `contracts.SourceChunk`.
- Continue validating chunk IDs, source IDs, indexes, unit membership, unit
uniqueness within each chunk, and source-ordering.
- After validation, canonicalize chunk units by replacing each returned
`SourceUnit` with a defensive copy of the matching source document unit.
- Preserve `SourceChunk.Metadata` as module-owned chunk metadata.
- Do not require full source coverage and do not reject overlap between chunks.
- Do not preserve chunker-mutated per-unit text, kind, or metadata. A chunker
that wants to add scene-level information must use `SourceChunk.Metadata`.
Code changes:
- Replace or extend `validateChunkResult` in
`internal/framework/pipeline/chunk_validation.go` so it returns canonical
chunks, for example:
```go
func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []contracts.SourceChunk) ([]contracts.SourceChunk, error)
```
- Build a source-unit lookup from the validated source document.
- For each chunk:
- validate the existing generic invariants;
- copy chunk ID, source ID, index, and chunk metadata;
- replace the unit slice with cloned source units from the source document in
the returned boundary/order.
- Update the runner to use canonical chunks for all downstream extraction and
merge behavior.
- Ensure chunk metadata is cloned so later module or caller mutation cannot
affect runner state.
- Keep the implementation source-agnostic. Do not inspect transcript-specific
metadata keys.
Tests:
- Add a runner test where a fake chunker returns a valid unit ID with mutated
text, kind, and unit metadata. Assert the extractor receives the original
source document unit values.
- Add a runner test proving chunk metadata survives canonicalization and is not
aliased to the chunker-returned map.
- Keep existing tests for invalid chunk IDs, duplicate IDs, wrong source ID,
wrong index, empty units, repeated unit IDs, unknown unit IDs, out-of-order
units, partial coverage, and overlap.
- Add or update tests so generic chunking still behaves unchanged.
Documentation:
- Update internal chunk contract docs to state that source units in chunks are
canonicalized from the source document by ID before extractors run.
- Document that chunk metadata is the supported mechanism for passing
chunker-owned context to extractors.
Validation:
```sh
go test ./internal/framework/pipeline
go test ./internal/modules/chunk/generic
go test ./internal/modules/chunk/dnd/scenes
```
Stage completion criteria:
- Extractors cannot observe chunker-rewritten source-unit text, kind, or unit
metadata.
- Chunker-owned scene metadata still reaches extractors through
`SourceChunk.Metadata`.
## Stage 3: Scene Caveat Hygiene
Goal: ensure model caveats become useful warnings and never produce blank
warnings that can confuse operators or affect diagnostics retention.
Design decision:
- Require caveat strings to be non-empty after trimming.
- Treat whitespace-only caveats as malformed structured output rather than
silently dropping them. This is consistent with the `dnd/scenes` policy of
failing explicitly for malformed model output.
- Store warning messages as trimmed caveat text.
Code changes:
- Update `internal/modules/chunk/dnd/scenes/assets/schemas/dnd_scenes.v1.json`
so `boundary_caveats.items` has `minLength: 1`.
- Update `warningsFromCaveats` or response validation in
`internal/modules/chunk/dnd/scenes/chunker.go` to trim caveats and reject
empty results with a module-prefixed malformed-output error.
- Prefer validating caveats before constructing chunks so all malformed response
checks happen together.
Tests:
- Add schema tests proving `boundary_caveats` items require non-empty strings.
- Add chunker tests proving:
- caveat warning messages are trimmed;
- whitespace-only caveats fail explicitly;
- valid caveats still produce `scene_boundary_caveat` warnings.
- Keep existing warning tests passing.
Documentation:
- Update `docs/internal/modules.md` and `docs/troubleshooting.md` if needed to
mention that malformed caveats are treated as malformed structured output.
Validation:
```sh
go test ./internal/modules/chunk/dnd/scenes
go test ./internal/cli
```
Stage completion criteria:
- No blank warnings can be emitted from `dnd/scenes` boundary caveats.
- Valid caveats remain visible as warnings.
## Stage 4: Full Verification
Goal: verify the follow-up work across contracts, production wiring,
documentation, and the command entry point.
Run: Run:
@@ -295,18 +229,17 @@ go vet ./...
go build ./cmd/notarius go build ./cmd/notarius
``` ```
Inspect diagnostics-sensitive output manually in tests or fixtures where Inspect or test representative output manifests:
relevant:
- no raw prompts, source text, provider payloads, API keys, or secrets in - `manifest.json` includes `module_metadata.chunker` for a `dnd/scenes` run;
manifest metadata; - `module_metadata.chunker` contains prompt and response-schema provenance;
- errors name the module and operation; - no raw prompt, raw schema, source text, provider payload, or secret appears in
- warnings are preserved in `RunOutput.Warnings`; module metadata;
- run manifests record the `dnd/scenes` chunker when selected. - artifact lane metadata remains under `artifact_lanes[].metadata`;
- `warnings.json` contains trimmed scene caveats and no blank caveat warnings.
Stage completion criteria: Stage completion criteria:
- Full validation commands pass. - Full validation commands pass.
- The feature is documented as implemented only where code supports it. - Current-behavior docs match implemented behavior.
- `docs/roadmap/chunk.md` retains target-state context and does not duplicate - Any remaining planned or deferred behavior stays under `docs/roadmap/`.
current-behavior reference material.