diff --git a/docs/roadmap/accepted-chunk-map-export.md b/docs/roadmap/accepted-chunk-map-export.md new file mode 100644 index 0000000..37d3b6a --- /dev/null +++ b/docs/roadmap/accepted-chunk-map-export.md @@ -0,0 +1,261 @@ +# Accepted Chunk Map Export + +Status: Accepted + +## Purpose + +Notarius already creates and validates one materialized chunk map before any +artifact lane executes. That map contains stable chunk identities, ordered +current-source ranges, and accepted namespaced annotations. It is useful +downstream data, but today it is visible only in internal execution and +explicit debug surfaces. + +Add an opt-in durable chunk-map artifact to the output bundle. Export the exact +accepted chunks used by the run rather than reconstructing them through an +extractor or exposing a chunker's unvalidated model response. + +This is a framework and output concern. Chunking remains pipeline-wide and +precedes all artifact lanes. + +## Desired End State + +The production JSON output encoder can be configured to add a canonical +`chunk-map.json` file to the logical output bundle. The file describes the +accepted materialized chunks used for lane execution without embedding source +units or transcript text. + +Use these durable identities: + +| Concern | Identity | +| --- | --- | +| Artifact kind | `source/chunk-map` | +| Logical file | `chunk-map.json` | +| Schema ID | `notarius.source.chunk_map` | +| Schema name | `notarius_source_chunk_map_v1` | +| Schema version | `v1` | +| Media type | `application/json` | + +The framework owns the artifact model, schema, validation, and canonical +encoding. Output encoders receive the serialized artifact through the generic +output request. The JSON encoder owns only the opt-in decision, logical file +placement, and output-index entry. + +## Configuration + +Add one strict option to the production `json` output module: + +```yaml +pipelines: + dnd-session: + input: seriatim + output: + module: json + options: + include_chunk_map: true +``` + +`include_chunk_map` is a boolean and defaults to `false`. Unknown options and +non-boolean values remain configuration errors. Do not add a separate CLI flag +or top-level filesystem option: this choice changes the logical files produced +by an output module, not their physical destination. + +Other output encoders may ignore the available chunk-map artifact unless and +until they define their own explicit export behavior. + +## Durable Artifact Contract + +The payload is one strict JSON object with this shape: + +```json +{ + "source_id": "session-7", + "source_digest": "sha256:0123456789abcdef...", + "plan_digest": "sha256:abcdef0123456789...", + "requested_chunker": "dnd/scenes", + "producer": { + "input_module": "seriatim", + "chunk_module": "dnd/scenes", + "llm_profile": "dnd-scenes" + }, + "plan_annotations": {}, + "chunks": [ + { + "id": "chunk-000001", + "index": 0, + "source_ref": { + "source_id": "session-7", + "start_unit_id": 1, + "end_unit_id": 18 + }, + "unit_count": 18, + "annotations": { + "dnd/scenes": { + "title": "At the city gate" + } + } + } + ] +} +``` + +Required top-level fields are: + +- `source_id`: the accepted source document identity; +- `source_digest`: the canonical digest of that document; +- `plan_digest`: the canonical digest of the accepted chunk plan; +- `requested_chunker`: the chunk module selected by the current resolved + pipeline; +- `producer`: the identity of the component that originally produced the + accepted plan; +- `plan_annotations`: the accepted plan-level annotation namespace map; and +- `chunks`: the non-empty ordered list used for lane execution. + +`producer` requires `input_module` and `chunk_module`. `llm_profile` is included +only when the producing chunker was LLM-backed. Producer references, module +metadata, warnings, creation timestamps, cache paths, and cache actions remain +in their existing provenance and diagnostic surfaces; they are not copied into +this artifact. + +Each chunk requires: + +- `id`: the stable materialized chunk ID; +- `index`: its zero-based position in execution order; +- `source_ref`: the exact inclusive current-source range; +- `unit_count`: the number of materialized source units in that range; and +- `annotations`: the accepted range-level annotation namespace map. + +Unknown fields are rejected at every fixed object level. Annotation namespaces +retain their canonical JSON values and may contain domain-specific JSON of any +type. Empty annotation maps are encoded as `{}` so the shape remains explicit. + +The durable artifact must guarantee that: + +- source and module identities are non-empty and contain no surrounding + whitespace; +- source and plan digests use the canonical `sha256:` representation; +- chunk IDs are non-empty and unique; +- chunk indexes are unique, contiguous, zero-based, and agree with array order; +- every chunk reference uses the top-level source identity; +- every range has positive endpoint IDs and was validated in source-document + order when the artifact was constructed; +- every unit count is positive; +- annotations contain valid JSON under non-empty canonical namespaces and are + canonicalized before digest calculation; and +- reconstructing the logical plan from the source digest, plan annotations, + chunk ranges, and chunk annotations reproduces `plan_digest`. + +The artifact represents chunk structure, not source content. It must not +contain materialized units, transcript bytes, source-unit metadata, chunk +content, private model responses, rejected boundary proposals, or debug +payloads. + +## Accepted-State And Failure Policy + +Construct the artifact only after plan materialization and the configured chunk +validator chain have accepted the chunks. The runner passes the same immutable +logical chunk identities, ranges, and annotations used for every lane; no +second chunking or model call occurs. + +When `include_chunk_map` is enabled: + +- emit the artifact even if one or more later artifact lanes are rejected; +- omit it when chunk validation rejects the candidate map, because no accepted + chunk map exists; +- retain the ordinary rejection record when it is omitted for that reason; and +- treat failure to construct, validate, serialize, index, or write an available + accepted chunk map as a run failure, consistent with any explicitly requested + durable output. + +A cache hit and a newly generated plan with the same accepted logical plan must +produce the same chunk ranges, annotations, plan digest, and materialized chunk +identities. The artifact records both the current `requested_chunker` and the +stored producer's `chunk_module`, since canonical plan reuse permits those +identities to differ. + +## Output Bundle Integration + +Extend `index.json` with an optional `chunk_map` descriptor: + +```json +{ + "chunk_map": { + "artifact_kind": "source/chunk-map", + "file": "chunk-map.json", + "media_type": "application/json", + "schema_id": "notarius.source.chunk_map", + "schema_name": "notarius_source_chunk_map_v1", + "schema_version": "v1" + } +} +``` + +The descriptor and file are both absent when export is disabled or no accepted +chunk map exists. The chunk map does not appear in lane-oriented +`output_files`, because it is pipeline-wide and has no lane, extractor, merger, +or normalizer identity. + +The existing manifest remains the canonical run-provenance index. Its +`chunk_plan` summary continues to own cache mode, lookup and publication +actions, producer reference provenance, module metadata, timestamps, and +validation status. The chunk-map artifact provides the accepted structure and +only the minimal producer identities needed to interpret it independently. + +## Ownership And Architecture + +Use a domain-neutral framework package for the durable chunk-map DTO, embedded +JSON Schema, invariant validation, cloning, and canonical serialization. Do not +marshal `source.Chunk` directly: its content, units, metadata, and internal +fields are intentionally broader than this external contract. + +Extend the output request with an optional cloned serialized chunk-map artifact. +The runner constructs that value from the accepted chunk-plan execution result +before invoking the output encoder. This keeps source and chunk-plan knowledge +out of the generic JSON encoder and allows future encoders to consume the same +framework-owned representation. + +The JSON encoder adds the file and index descriptor only when its +`include_chunk_map` option is true and the request contains an accepted +artifact. It applies the same logical-path validation and pretty-printed JSON +conventions as the rest of the bundle. + +## Security And Data Handling + +Chunk annotations may contain model-derived or source-derived information. +Treat `chunk-map.json` as durable user output with the same sensitivity and +retention expectations as lane artifacts. Opt-in export prevents new durable +content from appearing silently in existing pipelines. + +Do not copy annotation values, source ranges, or chunk IDs into the manifest. +Do not include external-reference content or filesystem paths in the chunk map. +Existing output-directory confinement, atomic writing, and permission policy +apply unchanged. + +## Quality And Documentation Policy + +The framework contract must remain strict, canonical, immutable across +ownership boundaries, and independent of source or D&D interpretation. The +runner must preserve accepted-state and cache-producer provenance, while the +JSON encoder must keep opt-in file selection separate from chunk semantics. + +Once implemented, the durable schema belongs in `docs/integrations/`; +configuration owns the selectable option; pipeline and state internals own the +framework handoff; and a maintained scene-chunking example should demonstrate +that namespaced annotations survive export. Tests should protect those +observable contracts without relying on exact payload lengths or private +helper structure. + +## Non-Goals + +This scope does not: + +- add a chunk-map extraction lane or artifact-lane registration; +- rerun chunking or ask an LLM to reconstruct accepted chunks; +- export transcript content, source units, source-unit metadata, or raw model + proposals; +- interpret D&D scene annotations as a generic contract; +- add scene descriptions, titles, summaries, kinds, or participant extraction; +- make chunk maps generated references or inputs to later ordered steps; +- change chunk planning, validation, cache selection, or publication behavior; +- add a chunk-map import or replay mechanism; +- enable export by default; or +- introduce a general output-plugin capability negotiation system. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 4dbd66d..38bdf7a 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -18,20 +18,10 @@ not as committed release dates. ### Export Accepted Chunk Maps -- Add an option to emit the accepted materialized chunk map as a proper, - framework-owned artifact with a documented schema identity, version, media - type, canonical encoding, and source/chunker provenance. -- Export the exact ordered chunks used for lane execution, including stable - chunk IDs and current-source ranges. Do not expose a model's raw boundary - proposal or require a second model call to reconstruct information already - owned by the framework. -- Treat chunk-map export as an output concern rather than an ordinary - extraction lane. Chunking is pipeline-wide and precedes lane extraction; a - pseudo-extractor would duplicate work and obscure that ownership boundary. -- Keep the generic contract independent of D&D interpretation. Namespaced - annotations may be preserved when they are part of the accepted chunk plan, - but downstream applications should not need domain-specific annotations to - understand chunk identity, order, or source coverage. +Add an opt-in, framework-owned durable artifact for the exact accepted +materialized chunks used by lane execution. The accepted contract, output +boundary, provenance policy, and exclusions are defined in +[Accepted Chunk Map Export](accepted-chunk-map-export.md). ### Extract D&D Scene Descriptions diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index e6d80b5..f9c89c8 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,570 +1,463 @@ -# D&D NPC Interactions Implementation Plan +# Accepted Chunk Map Export Implementation Plan Status: Ready for implementation ## Objective -Implement the accepted [D&D NPC Interactions](dnd-npc-interactions.md) -roadmap as a production D&D artifact lane. The finished lane must extract an -ordered list of minimal, evidence-grounded NPC interaction occurrences from -each accepted transcript chunk, ground every NPC against a required accepted -NPC registry, and preserve the existing fixed pipeline and generated-reference -architecture. +Implement the accepted [Accepted Chunk Map Export](accepted-chunk-map-export.md) +roadmap as an opt-in logical file in the production JSON output bundle. The +finished feature must serialize the exact accepted materialized chunks used by +the run, preserve framework ownership of chunk semantics, and keep file +selection and placement in the output encoder. -This plan is the implementation authority for sequencing and file-level work. -The feature roadmap remains authoritative for product intent, category -semantics, occurrence boundaries, evidence policy, and non-goals. Follow -[ADR-0009](../adr/0009-minimal-evidence-grounded-extraction-artifacts.md) -throughout: do not add descriptive, analytical, relationship, or state fields. +This document is the implementation authority for sequencing and concrete +engineering decisions. The feature roadmap remains authoritative for product +intent, durable field semantics, security policy, and non-goals. Implement the +stages in order and do not release or merge a partially completed sequence. ## Fixed Decisions -The implementation must use these identities: +Use these durable identities: | Concern | Identity | | --- | --- | -| Extractor key | `dnd/npc-interactions` | -| Normalizer key | `dnd/npc-interactions` | -| Artifact kind | `dnd/npc-interaction-list` | -| Durable schema ID | `notarius.dnd.npc_interactions` | -| Durable schema name | `notarius_dnd_npc_interactions_v1` | -| Durable schema version | `v1` | -| Durable media type | `application/json` | -| Private prompt ID | `dnd.npc_interactions` | -| Private response schema ID | `notarius.dnd.npc_interactions.llm` | -| Reference slot | `npcs` | +| Artifact kind | `source/chunk-map` | +| Logical file | `chunk-map.json` | +| Schema ID | `notarius.source.chunk_map` | +| Schema name | `notarius_source_chunk_map_v1` | +| Schema version | `v1` | +| Media type | `application/json` | +| JSON output option | `include_chunk_map` | -Use a top-level `interactions` array. Each durable record has exactly `name`, -`kind`, and `source_refs`. Do not add an interaction ID. The closed kind -vocabulary is `mentioned`, `noncombat_presence`, `dialogue`, `combat_ally`, -`combat_opponent`, and `other`. +Keep the schema at `v1`; Notarius is pre-release and no compatibility layer, +dual writer, or migration path is required. -The `npcs` slot is required for both extraction and normalization. It accepts -only the existing `dnd/npc-list` JSON artifact within the existing NPC registry -size limit. A bound registry with an empty `npcs` array is valid and requires an -empty interaction artifact. An absent binding, malformed registry, failed -generated handoff, or unknown interaction name is not permission to perform -ungrounded extraction. +The framework package owns the durable DTO, embedded JSON Schema, construction +from accepted source state, invariant validation, cloning, and canonical JSON +serialization. The runner owns construction and passes a serialized artifact +through the generic output request. The JSON output encoder owns the opt-in +decision, logical filename, formatting, and `index.json` descriptor. -Use the existing `internal/modules/dnd/npcs/registry.Resolver` for construction- -time external references and operation-time generated references. The -extractor presents only `Registry.PromptInput()` to the model. Name lookup uses -the registry's established comparison-key policy before normalization; the -normalizer replaces a recognized variant with the exact canonical registry -display name. Registry source references and campaign references never become -interaction evidence. +Do not register the chunk map with the artifact-lane codec registry. That +registry describes lane artifacts; the chunk map is a pipeline-wide source +artifact with no extractor, merger, or normalizer. -An occurrence is one NPC, one kind, and one coherent passage within one -accepted chunk. The model must split a category or combat-alignment transition. -No occurrence spans chunks. Merge preserves chunk order. Normalization: +The runner constructs the serialized value whenever an accepted chunk map +exists, independent of the selected output encoder and its options. Output +encoders may ignore the optional value. This keeps output-specific +configuration out of the runner and leaves future encoders free to define +their own explicit behavior. -1. replaces recognized names with canonical registry display names; -2. sorts and exact-deduplicates each record's source references; -3. orders records by earliest valid source position, then canonical name, - interaction kind, and the canonical source-reference sequence; -4. removes records only when canonical name, kind, and the complete canonical - source-reference sequence are all identical; and -5. never semantically combines nearby, overlapping, or cross-chunk records. +Use canonical `sha256:` digests throughout. Annotation values must be +semantically canonicalized before plan-digest verification; decoding must +accept insignificant JSON whitespace rather than require annotation +`json.RawMessage` bytes to already use one formatting style. -The private model response remains structural and contains name, kind, and -integer start/end unit candidates. Mapping attaches the current source ID. -Invalid strings, kinds, and ranges must survive mapping far enough for the -deterministic validator that owns the rule to reject them. - -For the final record comparator, use source-document unit position for the -earliest valid reference. Compare canonical names first by the existing NPC -identity comparison key and then by exact display string; compare kinds by -their string values; and compare canonical reference sequences -lexicographically by `source_id`, start-unit document position, and end-unit -document position. Records with valid evidence precede records without it; -normalized output validation rejects the latter before durable encoding. - -No framework change, new identity subsystem, DAG behavior, LLM-assisted -deduplication, or scene dependency is part of this plan. - -## Testing And Documentation Rules - -Apply [Testing Policy](../policy/testing.md) at every stage: - -- Protect artifact round trips, required fields, enum values, source - provenance, registry grounding, deterministic ordering, exact - deduplication, reference immutability, and generated handoff behavior. -- Use the real codec, registry, resolver, normalizer, and pipeline machinery - when they are fast and deterministic. Stub only the structured LLM boundary. -- Keep provider calls offline and deterministic. -- Test each policy at its narrowest stable owner. Do not repeat every codec, - validator, and normalizer case in an end-to-end test. -- Do not add a prompt-cache “change detector” based on exact message count, - shared-prefix length, token count, or prompt hash. Reuse the shared prompt - assets, test rendered behavior and manifest composition, and document the - ordering policy instead. -- Do not use broad golden snapshots for prompts or diagnostics. A maintained - durable JSON fixture is appropriate because the complete serialized artifact - is an intentional integration contract. - -Until the final stage, keep unimplemented behavior only in roadmap -documentation. Update current-behavior documentation and examples in the same -stage that makes the production lane selectable. - -## Stage 1 — Add The Typed Artifact And Durable Codec +## Stage 1: Add The Framework-Owned Durable Contract ### Goal -Establish the domain type and strict serialized boundary without changing the -production module catalog. +Create a source-domain chunk-map contract that can be built, validated, +encoded, and decoded without depending on an output module or D&D package. -### Changes +### Work -1. Extend `internal/modules/dnd/types.go` with: - - `NPCInteractionListKind`; - - a string-backed `NPCInteractionKind`; - - exported constants for all six accepted values; - - `NPCInteractionList` with `Interactions []NPCInteraction`; and - - `NPCInteraction` with `Name string`, `Kind NPCInteractionKind`, and - `SourceRefs []source.SourceRef`. -2. Add `internal/modules/dnd/codec/npcinteractions` following the existing NPC - and combat-turn codec boundary: - - expose the fixed schema constants and `MediaType`; - - implement the typed artifact codec for `dnd.NPCInteractionList`; - - keep `EncodeCandidate` and `DecodeCandidate` strict about one JSON value - and unknown fields while preserving typed semantic candidates; - - make approved `Encode` and `Decode` enforce non-empty names, the closed - kind enum, at least one structurally valid source reference, and the - durable schema shape; and - - report only `interaction_count` in codec metadata. -3. Add - `internal/modules/dnd/codec/npcinteractions/assets/schemas/dnd_npc_interactions.v1.json`. - It must require the top-level array and all three record fields, reject - unknown fields at every object level, encode the six-value enum, require at - least one source reference, and use the established durable source-reference - shape. -4. Add a small maintained fixture under - `internal/modules/dnd/codec/npcinteractions/testdata/` containing at least - two ordered records with distinct kinds and source ranges. +1. Add `internal/framework/chunkmap/` with: + + - `model.go` for the durable DTO and identity constants; + - `codec.go` for construction, validation, canonical encoding, and strict + decoding; + - `assets/schemas/source_chunk_map.v1.json` as the embedded external schema; + - `testdata/source_chunk_map.v1.json` as one valid contract fixture; and + - focused package tests. + +2. Define constants for the fixed artifact kind, schema identity, schema + version, and media type. Define DTOs matching the feature roadmap exactly: + + - `ChunkMap` with `source_id`, `source_digest`, `plan_digest`, + `requested_chunker`, `producer`, `plan_annotations`, and `chunks`; + - `Producer` with `input_module`, `chunk_module`, and optional + `llm_profile`; and + - `Chunk` with `id`, `index`, `source_ref`, `unit_count`, and + `annotations`. + + Do not expose source units, chunk content, source metadata, producer + references or metadata, warnings, timestamps, cache actions, or model + responses. + +3. Provide a small package API: + + ```go + type BuildRequest struct { + Source *source.SourceDocument + Plan source.ChunkPlan + Chunks []source.Chunk + RequestedChunker string + Producer Producer + } + + func Build(BuildRequest) (ChunkMap, error) + func Serialize(BuildRequest) (contracts.SerializedArtifact, error) + func New() *Codec + ``` + + `Codec` must expose the same identity and encode/decode operations used by + other durable codecs, but it is invoked directly by this package and the + JSON encoder rather than added to `pipeline.ArtifactCodecRegistry`. + +4. Make `Build` the trusted boundary between runtime source structures and the + external contract: + + - validate the source document and accepted plan with existing source + validators; + - calculate the source and plan digests with `source.DigestDocument` and + `source.DigestChunkPlan`, and require the calculated source digest to + equal both `Source.Digest` and `Plan.SourceDigest`; + - require a non-empty requested chunker and producer module identities; + - verify the supplied materialized chunks correspond exactly, in order, to + the accepted plan and source document, including IDs, indexes, source + ranges, unit membership/counts, plan annotations, and range annotations; + - validate range order against source-document position rather than numeric + unit-ID assumptions; + - deep-clone all references and annotation values; and + - encode absent annotation namespaces as non-nil empty maps so JSON contains + `{}`. + +5. Make codec decoding strict at fixed object levels: + + - accept one JSON value only and reject trailing content; + - reject unknown fields; + - validate trimmed, non-empty identities; + - require canonical lower-case `sha256:` digests; + - require a non-empty chunk array; + - require unique stable chunk IDs and contiguous zero-based indexes matching + array order; + - require every source reference to use the top-level source identity and + positive range endpoints; + - require positive unit counts; + - validate non-empty canonical annotation namespaces and valid JSON values; + - canonicalize annotation JSON before using the existing plan-digest helper + to reconstruct and verify `plan_digest`; and + - return defensively owned DTO data. + + Standalone decoding cannot prove range order against a source document it + does not contain. Construction owns that stronger invariant; decoding owns + the structural, identity, and digest invariants available from the durable + payload. + +6. Make serialization return a `contracts.SerializedArtifact` with the fixed + kind, schema, and media type, canonical content, and no duplicated metadata. + Validate the embedded schema defensively so the checked-in schema and Go + contract cannot silently diverge. + +7. Define a strict JSON Schema with: + + - `additionalProperties: false` at every fixed object level; + - all roadmap-required fields; + - `minItems: 1` for chunks; + - positive integer range endpoints and unit counts; + - zero-or-greater indexes; + - the pattern `^sha256:[0-9a-f]{64}$` for digests; + - non-empty strings for identities; + - optional, non-empty `llm_profile`; and + - annotation objects whose namespace values may be arbitrary JSON. ### Tests -At the codec package boundary, cover: +Add focused behavioral tests proving: -- fixture decode/encode round trip and canonical compact JSON; -- exact kind, schema identity, version, media type, and Go type registration; -- empty-list support and nil-versus-present-empty behavior where the existing - codecs distinguish it; -- strict rejection of malformed JSON, trailing values, and unknown fields; -- approved-boundary rejection of each meaningful required-field, enum, and - source-reference violation; -- candidate-boundary preservation of semantic values for later validators; and -- defensive copies for schema bytes and metadata. +- the valid fixture round-trips through the codec; +- building from a real accepted source, plan, and materialized chunk list + produces the fixed envelope and exact durable fields; +- output is deterministic and annotation JSON formatting does not alter the + reconstructed plan digest; +- unknown fields, trailing JSON, malformed identities/digests, invalid + indexes, duplicate IDs, mismatched source identities, invalid ranges or + counts, malformed annotation namespaces/values, and plan-digest mismatches + are rejected; +- construction rejects materialized chunks that differ from the accepted plan + or document order; and +- mutating caller-owned inputs or returned values cannot mutate another owned + representation. -Do not duplicate source-document existence checks in the codec; those belong to -the source-reference validator. +Keep these tests semantic. Do not add exact byte-length, exact schema-length, +private-helper, or redundant field-by-field “change detector” tests. ### Completion Criteria -- The new types and codec compile and pass focused tests. -- The codec can be registered into an isolated artifact codec registry with - exact type `dnd.NPCInteractionList`. -- No production registrar or configuration catalog exposes the new kind yet. +- The framework contract can represent and validate the roadmap payload + without importing output or D&D packages. +- Construction proves the artifact describes the exact accepted materialized + chunks. +- No production output behavior has changed yet. -## Stage 2 — Implement The Chunk-Scoped Extractor And Prompt +## Stage 2: Carry The Accepted Artifact Through The Runner ### Goal -Add an independently testable LLM-backed extractor that requires NPC grounding, -maps only minimal private output, and follows the established D&D prompt-cache -layout. +Construct the durable artifact at the accepted chunk-plan boundary and make it +available to output encoders without changing default logical files. -### Changes +### Work -1. Add `internal/modules/dnd/extract/npcinteractions` using the current D&D - extractor organization: - - `assets.go`; - - `canonicalize.go`; - - `extractor.go`; - - `model.go`; - - `schema.go`; - - `scriptorium_assets.go`; - - corresponding focused tests; and - - package-local embedded prompt and schema assets. -2. Give the extractor strict empty options, required capabilities `chunks` and - `source.transcript`, provided capability `dnd.npc_interactions`, and artifact - kind `dnd.NPCInteractionListKind`. -3. Build its reference slots from `shared.ReferenceSlots(...)`, then append the - `npcs` slot with: - - `Required: true`; - - accepted media type `application/json`; - - accepted artifact kind `dnd.NPCListKind`; - - the existing registry maximum byte count; and - - a description stating that the accepted registry is required identity - grounding, not evidence. - Return defensive, consistently sorted slot declarations from both - `ModuleSpec()` and the constructed extractor. -4. Construct an `npcregistry.Resolver` from the preparation-time reference set. - Resolve operation references inside `Extract`, require `Registry.Bound()`, - and fail before the LLM call if the required registry is absent or invalid. - Preserve the resolver's content-safe error behavior. -5. Use `shared.ChunkPromptMaterial` and `shared.PromptInputs`. Replace the - ordinary `npcs` input with the resolved names-only `Registry.PromptInput()`. - Do not place full registry bytes, NPC IDs, registry evidence, or provenance - in prompt inputs. -6. Add these assets: - - `assets/prompts/dnd.npc_interactions.yaml`; - - `assets/prompts/task.md`; - - `assets/prompts/instructions.md`; and - - `assets/schemas/dnd_npc_interactions_llm.v1.json`. -7. Use the existing shared prompt assets in this exact semantic order: - common system; common extraction evidence; common identity; common campaign - references; common NPC grounding; lane task; lane instructions; variable - transcript. Preserve the established cache-control placement used by the - spell and combat-turn manifests. -8. Revise the shared `common-dnd-npcs.md` wording once so it truthfully applies - to all three consumers: a normalized registry is always presented to the - prompt and may be empty. Keep its identity, context-only rule, and - prohibition on treating registry provenance as event evidence. Do not fork a - nearly identical interaction-specific grounding asset. -9. The lane prompt must state the category definitions, precedence, occurrence - splitting rules, registry-only name restriction, empty-output behavior, and - exclusions from the feature roadmap. Keep the response schema structural: - it requires the envelope and fields but leaves enum membership, non-empty - values, and source semantics to deterministic validators. -10. Map private records to `dnd.NPCInteractionList`, attach the current source - ID, sort and exact-deduplicate ranges within each candidate, and stable-sort - candidates with valid evidence by earliest source-document position. - Preserve invalid candidate fields rather than repairing or dropping them. -11. Expose prompt, response-schema, and mapping-policy identities through - manifest metadata and checkpoint fingerprints, following the current D&D - extractors. Include the NPC names-only projection digest in local checkpoint - identity. For a preparation-time external registry, expose only bounded - digest/count metadata; do not place names or source content in metadata. - Generated-reference provenance remains framework-owned. +1. Extend `contracts.OutputRequest` with: + + ```go + ChunkMap *contracts.SerializedArtifact `json:"chunk_map,omitempty"` + ``` + + Update its cloning/ownership path so the envelope, content bytes, and + metadata are defensively copied. Do not place chunk-map content in the run + manifest or generic request metadata. + +2. In `internal/framework/pipeline/runner.go`, construct the serialized chunk + map after materialization and configured chunk validation accept the plan, + and before the output encoder is invoked. Build it from: + + - the accepted source document; + - the accepted logical plan; + - the exact materialized chunks passed to lanes; + - the chunk module selected by the current resolved pipeline as + `requested_chunker`; and + - the original producer identities recorded by the accepted chunk-plan + record. + +3. Preserve cache semantics: + + - a generated or bypassed plan records the current producer; + - a cache hit keeps the stored producer's input module, chunk module, and + optional LLM profile; + - `requested_chunker` still records the current resolved chunk module; and + - producer references, metadata, warnings, and cache details remain only in + existing provenance surfaces. + +4. Apply the accepted-state policy: + + - pass no chunk map when chunk validation rejects the candidate; + - retain the artifact when a later extraction, merge, or normalization lane + is rejected; + - treat artifact construction or serialization failure after acceptance as + a framework run error; and + - do not rerun chunking or make another LLM request. + +5. If debug serialization records the output request, represent the new value + through the existing safe serialized-artifact envelope conventions. Do not + duplicate its content into the manifest or add a new default debug surface. ### Tests -At stable package boundaries, cover: +Extend `internal/framework/pipeline/runner_chunk_plan_test.go` and the nearest +existing output-request ownership tests to prove: -- nil/cancelled/invalid extraction requests and provider failures; -- required-slot declaration and rejection of an unbound or malformed registry - before any LLM call; -- operation-time generated registry resolution by an extractor prepared - without generated bytes; -- exact names-only registry prompt input and separation from transcript - evidence; -- empty registry plus empty response; -- mapping of every category, current-source attachment, evidence - canonicalization, and source-position ordering; -- preservation of invalid names, kinds, and ranges for validators; -- strict private JSON shape and unknown-field rejection; -- prompt rendering with required inputs, category policy, shared assets, and - transcript-last ordering; -- prompt/schema registration and content-safe metadata/fingerprints; and -- defensive module specifications and strict rejection of unknown options. +- accepted generated and cached plans reach a capturing output encoder as a + valid serialized chunk map; +- cache reuse distinguishes current `requested_chunker` from the stored + producer chunker; +- chunk rejection yields a nil chunk map; +- a later lane rejection still leaves the accepted chunk map available; +- captured requests do not alias runner, store, or caller-owned bytes/maps. -Run the existing spell and combat-turn prompt/asset tests after changing the -shared NPC prompt fragment. Do not assert an exact shared-prefix length. +Use existing fake modules and stores. Do not add an LLM-backed integration test +for behavior already deterministically owned by the runner, and do not add a +production injection seam solely to force an otherwise unreachable +construction failure; Stage 1 owns invalid-construction coverage. ### Completion Criteria -- The extractor is constructible and testable through the typed extractor - contract. -- A valid request makes one scheduled structured-completion call and returns a - typed candidate with current-source provenance. -- Missing required NPC grounding cannot reach the model. -- The extractor remains unregistered in the production D&D family until later - stages provide the rest of the lane. +- Every accepted chunk plan yields one validated serialized chunk map in the + output request. +- Rejected chunk plans yield none, while later lane rejection does not discard + it. +- Existing output encoders remain behaviorally unchanged. -## Stage 3 — Add Deterministic Artifact Validators +## Stage 3: Add Opt-In JSON Bundle Export ### Goal -Give each semantic invariant one clear validation owner and make invalid model -output a rejection rather than a normalization repair or framework error. +Expose the available framework artifact as `chunk-map.json` only when the JSON +output binding explicitly enables it. -### Changes +### Work -Add these typed validator packages, each with strict empty options, bounded -diagnostics, typed registration, and the fixed artifact kind: +1. Add an `Options` field: -1. `internal/modules/dnd/validate/npcinteractions/shape` - (`extract/dnd/npc-interactions/shape`): - - require a non-empty trimmed name without mutating it; - - require one of the six kinds; and - - require at least one source-reference candidate with structurally required - values. -2. `internal/modules/dnd/validate/npcinteractions/registry` - (`extract/dnd/npc-interactions/registry`): - - use `npcregistry.Resolver` at construction and operation time; - - require a bound registry; - - approve a name when `Registry.Lookup` recognizes it under the established - comparison policy; - - reject unknown names in stable record order; and - - never use registry evidence as transcript evidence. -3. `internal/modules/dnd/validate/npcinteractions/source_refs` - (`extract/dnd/npc-interactions/source_refs`): - - require the current source identity; - - require referenced units to exist; and - - require valid inclusive start/end order through the current document. - Follow the established D&D citation helpers and rejection aggregation - limits rather than introducing a generic framework dependency on the - interaction type. -4. `internal/modules/dnd/validate/npcinteractions/source_relatedness` - (`extract/dnd/npc-interactions/source_relatedness`): - - inspect only current transcript text covered by the cited ranges; - - emit at most one bounded warning per record when the NPC name cannot be - related to that evidence; - - do not claim to validate category agreement; and - - ignore campaign and registry content for relatedness. -5. `internal/modules/dnd/validate/npcinteractions/invariants` - (`normalize/dnd/npc-interactions/invariants`): - - require the exact canonical display name returned by registry lookup; - - require canonical source-reference order with no duplicate ranges; - - require the complete list order defined in Fixed Decisions; and - - reject exact duplicate normalized records. + ```go + IncludeChunkMap bool + ``` -The registry and invariants validators must support operation-time generated -references. Construction-time metadata and fingerprints follow the resolver -pattern used by registry-aware modules and must not include NPC names or -content. + Keep `New()` as the default-disabled constructor and add + `NewWithOptions(Options)` for configured construction and focused tests. + +2. Update the JSON output builder to decode only `include_chunk_map`: + + - omit or `false` means disabled; + - require a boolean when present; and + - continue rejecting unknown keys. + +3. When disabled, ignore an available `OutputRequest.ChunkMap` and preserve the + exact existing bundle shape. When enabled: + + - omit both file and descriptor if the request has no accepted chunk map; + - validate the artifact kind, schema identity/version, and media type; + - decode through `chunkmap.New()` so malformed or non-canonical payloads + fail the run; + - write valid content as pretty-printed JSON with the encoder's established + trailing-newline convention; and + - use the existing logical-path validation. + + Reuse or generalize the current serialized-artifact JSON helper rather than + adding a second subtly different validation and formatting path. + +4. Extend the output index model with an optional `chunk_map` descriptor + containing exactly: + + - `artifact_kind`; + - `file`; + - `media_type`; + - `schema_id`; + - `schema_name`; and + - `schema_version`. + + Do not add the chunk map to lane-oriented `output_files`, and do not change + the manifest. + +5. Treat a present but invalid accepted artifact as an output error. Do not + silently omit it when export was explicitly requested. ### Tests -Give each policy one primary test owner: +Extend `internal/modules/generic/output/json/encoder_test.go` and existing +output registration tests to prove: -- shape tests own empty names, enum membership, and missing evidence; -- registry tests own required binding, comparison-key recognition, unknown - names, operation-time generated overrides, empty registries, and content-safe - failures; -- source-reference tests own wrong source IDs, missing units, reversed ranges, - and valid multi-range records; -- relatedness tests own current-transcript-only warnings and bounded - diagnostics; and -- invariant tests own exact canonical names, final ordering, canonical evidence - sequences, and duplicate rejection. +- option omission and `false` retain the current default bundle even when an + artifact is available; +- `true` plus a valid artifact emits `chunk-map.json` and the exact optional + index descriptor; +- `true` plus no accepted artifact emits neither file nor descriptor; +- the chunk map never appears in lane `output_files`; +- wrong option types and unknown options are rejected; +- wrong kind, schema, media type, or invalid content returns an error; +- emitted JSON is valid, formatted consistently, and newline-terminated; and +- encoding does not mutate the request or serialized content. -Also verify that validators do not mutate artifacts, source documents, or -registries and that their specs register only for -`dnd.NPCInteractionListKind`. +Prefer testing the observable logical file map and decoded index rather than +private helper calls or exact byte lengths. ### Completion Criteria -- Valid typed candidates are approved and malformed or ungrounded candidates - are rejected at the intended boundary. -- No validator silently canonicalizes, drops, or merges records. -- References can explain identity but cannot satisfy source evidence checks. +- Existing JSON configurations remain unchanged by default. +- An explicitly enabled JSON output produces exactly one validated + `chunk-map.json` file and one pipeline-wide index descriptor when an + accepted map exists. +- No lane identity or manifest structure is fabricated for the chunk map. -## Stage 4 — Add Deterministic Merge And Normalization +## Stage 4: Prove The Assembled Behavior And Document It ### Goal -Preserve chunk-scoped occurrences through merge, then produce the canonical -ordered artifact without semantic inference. +Demonstrate the feature with the production D&D scene chunker, publish all +current-behavior documentation, and close the roadmap scope. -### Changes +### Work -1. Add `internal/modules/dnd/normalize/npcinteractions` with strict empty - options and the fixed module key and artifact kind. -2. Declare the same required `npcs` reference slot as the extractor. Construct - and resolve `npcregistry.Resolver` at the same preparation/operation - boundaries, and require a bound registry. -3. Implement the normalization algorithm in Fixed Decisions: - - clone all nested values; - - normalize display whitespace only as needed for lookup; - - replace every recognized name with the registry's exact canonical display - name; - - canonicalize each reference list without merging overlapping ranges; - - compute order from source-document positions rather than assuming numeric - unit IDs are contiguous; - - apply every specified tie-breaker; and - - collapse only exact records after canonicalization. -4. Preserve nil versus present-empty list behavior consistently with the other - D&D normalizers. -5. Emit bounded warnings for visible canonical-name changes, reference - canonicalization, record reordering, and exact duplicate removal. Use stable - reason codes and the shared D&D diagnostics helpers. Never include reference - content in a warning. -6. Add an append-only merger function for `dnd.NPCInteractionList` beside the - other D&D typed append functions in - `internal/modules/dnd/register/merge.go`. It must preserve accepted - chunk/lane order, preserve present-empty semantics, and deep-clone source - reference slices so outputs do not alias inputs. Do not register it until - Stage 5. -7. Add normalization-policy and NPC projection identities to checkpoint - fingerprints. Manifest metadata may include the policy identity and bounded - external-registry digest/count, but not names or source content. +1. Extend the representative assembled production contract in + `internal/cli/production_contract_test.go`: -### Tests + - configure `dnd/scenes` with JSON `include_chunk_map: true`; + - execute through normal production assembly with the existing deterministic + fake LLM; + - read `index.json` and `chunk-map.json`; + - decode the artifact through the framework codec; and + - assert the actual accepted scene ranges, stable IDs/order, source and plan + identity, requested/current producer provenance, and surviving namespaced + scene annotations. -Cover: + This is the single assembled proof. Do not duplicate all codec and encoder + edge cases at the CLI layer. -- canonical registry name replacement, including Unicode/case/spacing lookup; -- required and operation-time generated registry behavior; -- source-reference ordering and exact range deduplication; -- ordering by real document position with all deterministic tie-breakers; -- separation of category transitions, alignment transitions, nearby records, - overlapping-but-nonidentical evidence, and nonidentical records from - different chunks; -- collapse of exact duplicates only; -- stable, bounded warnings; -- nil/present-empty behavior; -- input ownership and nested-slice cloning; and -- append-order merger behavior independently of normalization. +2. Add a copyable maintained example such as + `examples/dnd-scene-chunk-map.config.yml` that uses `dnd/scenes`, one normal + D&D artifact lane, and the opt-in JSON output option. Add it to + `internal/cli/example_contract_test.go` so schema/config drift is caught by + the existing example contract. -Use table-driven pure normalization cases for the dense ordering and duplicate -rules, and package-level normalizer tests for resolver and warning behavior. +3. Add `docs/integrations/chunk-map.md` as the canonical durable contract + reference. Document fixed identities, every payload field, invariants, + accepted-state behavior, cache provenance semantics, sensitivity, and + excluded content. Link to the embedded schema without duplicating it in the + prose. -### Completion Criteria +4. Update current-behavior documentation: -- Normalization is deterministic and idempotent. -- Running normalization twice does not change the value or emit new - transformation warnings on the second pass. -- Semantically distinct occurrences remain distinct. -- The module and merger compile but are not yet selectable through production - composition. + - `docs/integrations/json-output.md`: optional logical file and index + descriptor; + - `docs/config.md`: strict `include_chunk_map` option and copyable binding; + - `docs/operations.md`: opt-in sensitivity and retention implications; + - `docs/internal/pipeline.md`: accepted-plan construction and output-request + handoff; + - `docs/internal/state.md`: logical output and manifest/index ownership; + - `docs/internal/modules.md`: JSON encoder option and responsibility; and + - `docs/internal/overview.md`: framework chunk-map contract and encoder + boundary. -## Stage 5 — Compose The Complete Production Lane + Keep implementation details in internal docs, user-selectable behavior in + configuration docs, and the durable schema only in integrations docs. -### Goal +5. Close planning state: -Register the complete typed lane atomically so configuration cannot select a -partial implementation. + - mark `accepted-chunk-map-export.md` as implemented; + - remove the completed “Export Accepted Chunk Maps” entry from + `docs/roadmap/future.md`; and + - mark this implementation plan completed, preserving it until the user + chooses to retire completed roadmap documents. -### Changes - -Update the D&D registrar: - -1. In `internal/modules/dnd/register/modules.go`, register: - - the NPC-interaction codec; - - extractor and prompt assets; - - the append-order merger variant for - `dnd.NPCInteractionListKind`; - - the `dnd/npc-interactions` normalizer; and - - the generic no-op normalizer variant for the new exact Go type. -2. In `internal/modules/dnd/register/validators.go`, register all five domain - validators plus generic always-accept and always-reject typed variants for - the new artifact type. -3. In `internal/modules/dnd/register/chains.go`, register: - - extraction chain: - `generic/valid_json`, - `extract/dnd/npc-interactions/shape`, - `extract/dnd/npc-interactions/registry`, - `extract/dnd/npc-interactions/source_refs`, - `generic/valid_json_schema`, - `extract/dnd/npc-interactions/source_relatedness`; - - normalization chain: - `generic/valid_json`, - `extract/dnd/npc-interactions/shape`, - `extract/dnd/npc-interactions/registry`, - `normalize/dnd/npc-interactions/invariants`, - `extract/dnd/npc-interactions/source_refs`, - `generic/valid_json_schema`, - `extract/dnd/npc-interactions/source_relatedness`. -4. Do not add a default merge validator chain. The append merger performs no - semantic mutation, and extraction plus normalized boundaries own the - consequential policies. -5. Extend registrar contract tests to verify exact typed coverage, keys, - artifact kinds, prompt/schema assets, reference-slot agreement, and default - validator order. - -### Tests - -- Extend the existing D&D family registration test rather than creating - parallel catalog snapshots. -- Verify that the extractor and normalizer both expose required compatible - `npcs` slots and that generated `dnd/npc-list` bindings resolve only from an - earlier step. -- Verify production resolution rejects a missing required binding, a later or - same-step producer, a wrong artifact kind, and incompatible media/schema - metadata through existing resolver behavior. -- Avoid retesting the framework's general ordered-step failure matrix; add only - interaction-specific composition cases not already covered generically. - -### Completion Criteria - -- Production registration exposes one complete type-consistent lane. -- A valid two-step NPC-to-interactions pipeline resolves and prepares. -- Invalid or missing NPC dependencies fail before source parsing or LLM work - whenever statically discoverable. - -## Stage 6 — Prove The Workflow And Publish Current Contracts - -### Goal - -Exercise the assembled generated-reference workflow and move all implemented -behavior into its canonical current documentation. - -### Changes - -1. Add one representative integration test under - `internal/modules/integration` with focused testdata: - - step 1 extracts and normalizes NPCs; - - step 2 consumes the generated NPC artifact in an interaction lane; - - the fake structured LLM returns multiple interaction kinds and at least - one name variant; - - the assertion proves the names-only handoff, current-transcript evidence, - canonical name, stable chronology, and final durable artifact. -2. Include one failure assertion showing that a rejected or absent normalized - NPC producer prevents interaction extraction. Reuse the existing runner - dependency behavior; do not duplicate every framework failure case. -3. Add `examples/dnd-npc-interactions.config.yml` as a complete copyable - two-step profile. The first step produces `npcs`; the second step binds that - exact step/lane artifact to its `npcs` reference and selects - `dnd/npc-interactions` for extract and normalize. -4. Add the example to the existing CLI production configuration contract test - so it is parsed and resolved offline. -5. Add the durable external contract at - `docs/integrations/dnd-npc-interaction-artifacts.md`. It owns schema - identity, JSON shape, categories, evidence semantics, ordering, - normalization, validator chains, generated-reference behavior, and manifest - metadata. -6. Update current canonical documentation: - - `docs/config.md` for module/validator catalogs, default chains, required - reference slot, and the maintained example; - - `docs/internal/modules.md` for implementation ownership, prompt inputs, - registry resolution, validation, merge, and normalization; - - `docs/internal/overview.md` for the implemented component inventory; - - `docs/internal/llm.md` only as needed to include the new lane in the - existing D&D prompt-order/cache policy without duplicating the manifest; - - `docs/integrations/dnd-npc-artifacts.md` to identify NPC interactions as a - consumer while retaining the rule that registry evidence is not event - evidence; and - - any directly affected configuration or integration links. -7. Remove the NPC-interactions entry from `docs/roadmap/future.md` once the - production behavior and current documentation are complete. Change - `docs/roadmap/dnd-npc-interactions.md` to `Status: Implemented` pending its - later retirement; do not leave it as the canonical current contract. -8. Create a small human-review worksheet or fixture set only if the repository - already has an appropriate non-test evaluation home. Otherwise record the - manual evaluation results in the implementation handoff rather than - inventing a new framework. Exercise all six categories, transitions, - repeated occurrences, mentions followed by presence, combat alignment - changes, cross-chunk repetition, and empty output on at least one intended - smaller model. Do not make model agreement a deterministic CI gate. +6. Review links, examples, terminology, and file identities across the + documentation. Do not add an ADR: this feature applies the existing fixed + pipeline, output/cache separation, and framework ownership decisions rather + than changing them. ### Validation -Run focused tests while implementing, then run: +Run focused tests after each stage. At completion, run: ```sh go test ./... go vet ./... go build ./cmd/notarius +git diff --check ``` -Also run `git diff --check`, inspect the complete documentation diff for -current-versus-future claims, and verify every new relative documentation link. +Also verify that maintained example tests pass, all new documentation links +resolve, and a default JSON output run still has no chunk-map file or index +descriptor. ### Completion Criteria -- All repository checks pass offline. -- The maintained example resolves with an explicit earlier NPC producer. -- The integration test proves the accepted generated artifact, not registry - source references, grounds the later model request. -- Durable output contains only `name`, `kind`, and current-transcript - `source_refs` per occurrence. -- Current documentation owns implemented contracts, while the feature roadmap - is marked implemented and `future.md` no longer advertises the work as - pending. +- The assembled scene-chunking run exports the exact accepted map and + namespaced annotations without a second model call. +- Current documentation and the maintained example describe only implemented + behavior. +- The future roadmap no longer lists the completed feature. +- Repository-wide tests, vet, build, formatting, and documentation checks pass. + +## Cross-Stage Guardrails + +- Do not expose internal `source.Chunk` directly as the durable DTO. +- Do not add transcript content, materialized units, source-unit metadata, + model proposals, or debug data to the artifact. +- Do not create an extractor, lane registration, normalizer, generated + reference, import path, or replay mechanism for chunk maps. +- Do not make D&D scene annotation fields part of the generic schema. +- Do not make export the default or add a CLI-global/filesystem option. +- Do not change chunk selection, validation, cache lookup/publication, or lane + execution semantics. +- Do not duplicate chunk-map payloads in the manifest or lane output indexes. +- Preserve defensive ownership at source, runner, request, codec, and encoder + boundaries. +- Keep tests offline and deterministic, and test each invariant at its stable + owning layer. ## Open Questions -None. The feature roadmap and fixed decisions above define the product and -architectural choices required for implementation. +None. The feature roadmap and the fixed decisions above resolve the +implementation-significant choices. diff --git a/internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json b/internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json new file mode 100644 index 0000000..a4ecd4b --- /dev/null +++ b/internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json @@ -0,0 +1,65 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "notarius.source.chunk_map", + "title": "notarius_source_chunk_map_v1", + "type": "object", + "additionalProperties": false, + "required": [ + "source_id", + "source_digest", + "plan_digest", + "requested_chunker", + "producer", + "plan_annotations", + "chunks" + ], + "properties": { + "source_id": {"type": "string", "minLength": 1}, + "source_digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"}, + "plan_digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"}, + "requested_chunker": {"type": "string", "minLength": 1}, + "producer": { + "type": "object", + "additionalProperties": false, + "required": ["input_module", "chunk_module"], + "properties": { + "input_module": {"type": "string", "minLength": 1}, + "chunk_module": {"type": "string", "minLength": 1}, + "llm_profile": {"type": "string", "minLength": 1} + } + }, + "plan_annotations": {"$ref": "#/$defs/annotations"}, + "chunks": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["id", "index", "source_ref", "unit_count", "annotations"], + "properties": { + "id": {"type": "string", "minLength": 1}, + "index": {"type": "integer", "minimum": 0}, + "source_ref": { + "type": "object", + "additionalProperties": false, + "required": ["source_id", "start_unit_id", "end_unit_id"], + "properties": { + "source_id": {"type": "string", "minLength": 1}, + "start_unit_id": {"type": "integer", "minimum": 1}, + "end_unit_id": {"type": "integer", "minimum": 1} + } + }, + "unit_count": {"type": "integer", "minimum": 1}, + "annotations": {"$ref": "#/$defs/annotations"} + } + } + } + }, + "$defs": { + "annotations": { + "type": "object", + "propertyNames": {"type": "string", "minLength": 1}, + "additionalProperties": true + } + } +} diff --git a/internal/framework/chunkmap/codec.go b/internal/framework/chunkmap/codec.go new file mode 100644 index 0000000..d05a0e6 --- /dev/null +++ b/internal/framework/chunkmap/codec.go @@ -0,0 +1,347 @@ +package chunkmap + +import ( + "bytes" + "embed" + "encoding/json" + "fmt" + "io" + "regexp" + "strings" + + "gitea.maximumdirect.net/eric/notarius/internal/core/source" + "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" +) + +//go:embed assets/schemas/source_chunk_map.v1.json +var schemaAssets embed.FS + +var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) + +// Codec owns strict serialization for the durable chunk-map contract. +type Codec struct{} + +func New() *Codec { return &Codec{} } + +func (c *Codec) Kind() contracts.ArtifactKind { return ArtifactKind } + +func (c *Codec) Schema() contracts.ArtifactSchema { + raw, err := c.schemaBytes() + if err != nil { + return contracts.ArtifactSchema{} + } + return contracts.ArtifactSchema{ + ID: SchemaID, + Name: SchemaName, + Version: SchemaVersion, + JSONSchema: raw, + } +} + +func (c *Codec) MediaType() string { return MediaType } + +// Build proves that a durable value describes the exact accepted source plan +// and materialized chunk list supplied by the framework. +func Build(request BuildRequest) (ChunkMap, error) { + if err := source.ValidateDocument(request.Source); err != nil { + return ChunkMap{}, fmt.Errorf("validate source document: %w", err) + } + sourceDigest, err := source.DigestDocument(request.Source) + if err != nil { + return ChunkMap{}, fmt.Errorf("digest source document: %w", err) + } + if sourceDigest != request.Source.Digest { + return ChunkMap{}, fmt.Errorf("source digest %q does not match source document digest %q", sourceDigest, request.Source.Digest) + } + if sourceDigest != request.Plan.SourceDigest { + return ChunkMap{}, fmt.Errorf("source digest %q does not match chunk plan source digest %q", sourceDigest, request.Plan.SourceDigest) + } + plan, err := source.CanonicalizeChunkPlan(request.Plan) + if err != nil { + return ChunkMap{}, fmt.Errorf("canonicalize chunk plan: %w", err) + } + if err := source.ValidateChunkPlan(request.Source, plan); err != nil { + return ChunkMap{}, fmt.Errorf("validate accepted chunk plan: %w", err) + } + planDigest, err := source.DigestChunkPlan(plan) + if err != nil { + return ChunkMap{}, fmt.Errorf("digest accepted chunk plan: %w", err) + } + expected, err := source.MaterializeChunkPlan(request.Source, plan) + if err != nil { + return ChunkMap{}, fmt.Errorf("materialize accepted chunk plan: %w", err) + } + if err := verifyMaterializedChunks(request.Chunks, expected); err != nil { + return ChunkMap{}, err + } + + value := ChunkMap{ + SourceID: request.Source.ID, + SourceDigest: sourceDigest, + PlanDigest: planDigest, + RequestedChunker: request.RequestedChunker, + Producer: request.Producer, + PlanAnnotations: source.CloneChunkAnnotations(plan.Annotations), + Chunks: make([]Chunk, len(expected)), + } + for index, chunk := range expected { + value.Chunks[index] = Chunk{ + ID: chunk.ID, + Index: chunk.Index, + SourceRef: chunk.Ref, + UnitCount: len(chunk.Units), + Annotations: source.CloneChunkAnnotations(chunk.Annotations), + } + } + canonical, err := canonicalize(value) + if err != nil { + return ChunkMap{}, fmt.Errorf("validate chunk map: %w", err) + } + return clone(canonical), nil +} + +// Serialize builds and encodes the framework-owned serialized artifact. +func Serialize(request BuildRequest) (contracts.SerializedArtifact, error) { + value, err := Build(request) + if err != nil { + return contracts.SerializedArtifact{}, err + } + codec := New() + content, err := codec.Encode(value) + if err != nil { + return contracts.SerializedArtifact{}, err + } + return contracts.SerializedArtifact{ + Kind: ArtifactKind, + Schema: codec.Schema(), + MediaType: MediaType, + Content: content, + }, nil +} + +func (c *Codec) Encode(value ChunkMap) ([]byte, error) { + if _, err := c.schemaBytes(); err != nil { + return nil, err + } + canonical, err := canonicalize(value) + if err != nil { + return nil, fmt.Errorf("encode source chunk map: %w", err) + } + content, err := json.Marshal(canonical) + if err != nil { + return nil, fmt.Errorf("encode source chunk map: %w", err) + } + return content, nil +} + +func (c *Codec) Decode(content []byte) (ChunkMap, error) { + if _, err := c.schemaBytes(); err != nil { + return ChunkMap{}, err + } + decoder := json.NewDecoder(bytes.NewReader(content)) + decoder.DisallowUnknownFields() + var value ChunkMap + if err := decoder.Decode(&value); err != nil { + return ChunkMap{}, fmt.Errorf("decode source chunk map: %w", err) + } + var trailing any + if err := decoder.Decode(&trailing); err != io.EOF { + return ChunkMap{}, fmt.Errorf("decode source chunk map: multiple JSON values") + } + canonical, err := canonicalize(value) + if err != nil { + return ChunkMap{}, fmt.Errorf("decode source chunk map: %w", err) + } + return clone(canonical), nil +} + +func (c *Codec) schemaBytes() ([]byte, error) { + raw, err := schemaAssets.ReadFile("assets/schemas/source_chunk_map.v1.json") + if err != nil { + return nil, fmt.Errorf("read source chunk map schema: %w", err) + } + var schema struct { + ID string `json:"$id"` + Title string `json:"title"` + Type string `json:"type"` + Required []string `json:"required"` + } + if err := json.Unmarshal(raw, &schema); err != nil { + return nil, fmt.Errorf("decode source chunk map schema: %w", err) + } + if schema.ID != SchemaID || schema.Title != SchemaName || schema.Type != "object" || !hasRequiredFields(schema.Required) { + return nil, fmt.Errorf("source chunk map schema identity or required fields are invalid") + } + return append([]byte(nil), raw...), nil +} + +func hasRequiredFields(required []string) bool { + want := map[string]bool{ + "source_id": true, "source_digest": true, "plan_digest": true, + "requested_chunker": true, "producer": true, "plan_annotations": true, + "chunks": true, + } + for _, field := range required { + delete(want, field) + } + return len(want) == 0 +} + +func canonicalize(value ChunkMap) (ChunkMap, error) { + if err := requireIdentity("source_id", value.SourceID); err != nil { + return ChunkMap{}, err + } + if err := requireDigest("source_digest", value.SourceDigest); err != nil { + return ChunkMap{}, err + } + if err := requireDigest("plan_digest", value.PlanDigest); err != nil { + return ChunkMap{}, err + } + if err := requireIdentity("requested_chunker", value.RequestedChunker); err != nil { + return ChunkMap{}, err + } + if err := requireIdentity("producer.input_module", value.Producer.InputModule); err != nil { + return ChunkMap{}, err + } + if err := requireIdentity("producer.chunk_module", value.Producer.ChunkModule); err != nil { + return ChunkMap{}, err + } + if value.Producer.LLMProfile != "" { + if err := requireIdentity("producer.llm_profile", value.Producer.LLMProfile); err != nil { + return ChunkMap{}, err + } + } + annotations, err := canonicalizeAnnotations("plan_annotations", value.PlanAnnotations) + if err != nil { + return ChunkMap{}, err + } + value.PlanAnnotations = annotations + if len(value.Chunks) == 0 { + return ChunkMap{}, fmt.Errorf("chunks must not be empty") + } + seenIDs := make(map[string]struct{}, len(value.Chunks)) + plan := source.ChunkPlan{SourceDigest: value.SourceDigest, Annotations: annotations, Ranges: make([]source.ChunkRange, len(value.Chunks))} + for index := range value.Chunks { + chunk := &value.Chunks[index] + if err := requireIdentity(fmt.Sprintf("chunks[%d].id", index), chunk.ID); err != nil { + return ChunkMap{}, err + } + if _, exists := seenIDs[chunk.ID]; exists { + return ChunkMap{}, fmt.Errorf("chunks[%d].id %q is duplicated", index, chunk.ID) + } + seenIDs[chunk.ID] = struct{}{} + if chunk.Index != index { + return ChunkMap{}, fmt.Errorf("chunks[%d].index = %d, want %d", index, chunk.Index, index) + } + if chunk.SourceRef.SourceID != value.SourceID { + return ChunkMap{}, fmt.Errorf("chunks[%d].source_ref.source_id %q does not match source_id %q", index, chunk.SourceRef.SourceID, value.SourceID) + } + if chunk.SourceRef.StartUnitID <= 0 || chunk.SourceRef.EndUnitID <= 0 { + return ChunkMap{}, fmt.Errorf("chunks[%d].source_ref endpoints must be positive", index) + } + if chunk.UnitCount <= 0 { + return ChunkMap{}, fmt.Errorf("chunks[%d].unit_count must be positive", index) + } + chunkAnnotations, err := canonicalizeAnnotations(fmt.Sprintf("chunks[%d].annotations", index), chunk.Annotations) + if err != nil { + return ChunkMap{}, err + } + chunk.Annotations = chunkAnnotations + plan.Ranges[index] = source.ChunkRange{ + StartUnitID: chunk.SourceRef.StartUnitID, + EndUnitID: chunk.SourceRef.EndUnitID, + Annotations: chunkAnnotations, + } + } + planDigest, err := source.DigestChunkPlan(plan) + if err != nil { + return ChunkMap{}, fmt.Errorf("reconstruct plan digest: %w", err) + } + if planDigest != value.PlanDigest { + return ChunkMap{}, fmt.Errorf("plan_digest %q does not match reconstructed plan digest %q", value.PlanDigest, planDigest) + } + return value, nil +} + +func canonicalizeAnnotations(name string, annotations source.ChunkAnnotations) (source.ChunkAnnotations, error) { + for namespace := range annotations { + if strings.TrimSpace(namespace) == "" || namespace != strings.TrimSpace(namespace) { + return nil, fmt.Errorf("%s namespace %q must be non-empty and trimmed", name, namespace) + } + } + canonical, err := source.CanonicalizeChunkAnnotations(annotations) + if err != nil { + return nil, fmt.Errorf("%s: %w", name, err) + } + if canonical == nil { + canonical = source.ChunkAnnotations{} + } + return canonical, nil +} + +func requireIdentity(name, value string) error { + if strings.TrimSpace(value) == "" || value != strings.TrimSpace(value) { + return fmt.Errorf("%s must be non-empty and trimmed", name) + } + return nil +} + +func requireDigest(name, value string) error { + if !digestPattern.MatchString(value) { + return fmt.Errorf("%s must be a canonical sha256 digest", name) + } + return nil +} + +func verifyMaterializedChunks(actual, expected []source.Chunk) error { + if len(actual) != len(expected) { + return fmt.Errorf("materialized chunks length = %d, want %d", len(actual), len(expected)) + } + for index := range expected { + got, want := actual[index], expected[index] + if got.ID != want.ID || got.SourceID != want.SourceID || got.Index != want.Index || got.Ref != want.Ref { + return fmt.Errorf("materialized chunk[%d] identity or source range differs from accepted plan", index) + } + if len(got.Units) != len(want.Units) || !sameUnits(got.Units, want.Units) { + return fmt.Errorf("materialized chunk[%d] units differ from accepted source range", index) + } + if !sameAnnotations(got.PlanAnnotations, want.PlanAnnotations) || !sameAnnotations(got.Annotations, want.Annotations) { + return fmt.Errorf("materialized chunk[%d] annotations differ from accepted plan", index) + } + } + return nil +} + +func sameUnits(left, right []source.SourceUnit) bool { + leftJSON, leftErr := json.Marshal(left) + rightJSON, rightErr := json.Marshal(right) + return leftErr == nil && rightErr == nil && bytes.Equal(leftJSON, rightJSON) +} + +func sameAnnotations(left, right source.ChunkAnnotations) bool { + leftCanonical, leftErr := source.CanonicalizeChunkAnnotations(left) + rightCanonical, rightErr := source.CanonicalizeChunkAnnotations(right) + if leftErr != nil || rightErr != nil { + return false + } + leftJSON, leftErr := json.Marshal(leftCanonical) + rightJSON, rightErr := json.Marshal(rightCanonical) + return leftErr == nil && rightErr == nil && bytes.Equal(leftJSON, rightJSON) +} + +func clone(value ChunkMap) ChunkMap { + value.PlanAnnotations = cloneAnnotations(value.PlanAnnotations) + value.Chunks = append([]Chunk(nil), value.Chunks...) + for index := range value.Chunks { + value.Chunks[index].Annotations = cloneAnnotations(value.Chunks[index].Annotations) + } + return value +} + +func cloneAnnotations(annotations source.ChunkAnnotations) source.ChunkAnnotations { + cloned := source.CloneChunkAnnotations(annotations) + if cloned == nil { + return source.ChunkAnnotations{} + } + return cloned +} diff --git a/internal/framework/chunkmap/codec_test.go b/internal/framework/chunkmap/codec_test.go new file mode 100644 index 0000000..a92a0ea --- /dev/null +++ b/internal/framework/chunkmap/codec_test.go @@ -0,0 +1,191 @@ +package chunkmap + +import ( + "bytes" + "encoding/json" + "os" + "strings" + "testing" + + "gitea.maximumdirect.net/eric/notarius/internal/core/source" +) + +func TestBuildAndSerializeAcceptedChunkMap(t *testing.T) { + request := acceptedBuildRequest(t) + value, err := Build(request) + if err != nil { + t.Fatalf("Build() error = %v", err) + } + if value.SourceID != request.Source.ID || len(value.Chunks) != 2 || value.Chunks[0].UnitCount != 2 || value.Chunks[1].SourceRef.StartUnitID != 20 { + t.Fatalf("Build() = %#v, want exact accepted chunk structure", value) + } + if value.PlanAnnotations == nil || value.Chunks[1].Annotations == nil { + t.Fatalf("Build() annotations = %#v, want explicit maps", value) + } + artifact, err := Serialize(request) + if err != nil { + t.Fatalf("Serialize() error = %v", err) + } + if artifact.Kind != ArtifactKind || artifact.Schema.ID != SchemaID || artifact.Schema.Name != SchemaName || artifact.Schema.Version != SchemaVersion || artifact.MediaType != MediaType || artifact.Metadata != nil { + t.Fatalf("Serialize() = %#v, want fixed artifact envelope without metadata", artifact) + } + decoded, err := New().Decode(artifact.Content) + if err != nil { + t.Fatalf("Decode(Serialize()) error = %v", err) + } + if decoded.PlanDigest != value.PlanDigest || decoded.Chunks[0].ID != "chunk-000001" || decoded.Chunks[1].UnitCount != 1 { + t.Fatalf("Decode(Serialize()) = %#v, want durable chunk map", decoded) + } +} + +func TestCodecRoundTripsValidFixture(t *testing.T) { + fixture, err := os.ReadFile("testdata/source_chunk_map.v1.json") + if err != nil { + t.Fatal(err) + } + codec := New() + value, err := codec.Decode(fixture) + if err != nil { + t.Fatalf("Decode(fixture) error = %v", err) + } + encoded, err := codec.Encode(value) + if err != nil { + t.Fatalf("Encode(decoded fixture) error = %v", err) + } + if !bytes.Equal(encoded, bytes.TrimSpace(fixture)) { + t.Fatalf("fixture does not use canonical encoding\nwant: %s\n got: %s", fixture, encoded) + } +} + +func TestBuildCanonicalizesAnnotationFormatting(t *testing.T) { + first := acceptedBuildRequest(t) + second := acceptedBuildRequest(t) + second.Plan.Annotations["dnd/scenes"] = json.RawMessage(" { \n \t\"title\" : \"Gate\" \n } ") + canonical, err := source.CanonicalizeChunkPlan(second.Plan) + if err != nil { + t.Fatalf("CanonicalizeChunkPlan() error = %v", err) + } + second.Chunks, err = source.MaterializeChunkPlan(second.Source, canonical) + if err != nil { + t.Fatalf("MaterializeChunkPlan() error = %v", err) + } + firstArtifact, err := Serialize(first) + if err != nil { + t.Fatalf("Serialize(first) error = %v", err) + } + secondArtifact, err := Serialize(second) + if err != nil { + t.Fatalf("Serialize(second) error = %v", err) + } + if !bytes.Equal(firstArtifact.Content, secondArtifact.Content) { + t.Fatalf("serialized content differs only because annotation whitespace changed\nfirst: %s\nsecond: %s", firstArtifact.Content, secondArtifact.Content) + } +} + +func TestBuildRejectsChunksOutsideAcceptedPlan(t *testing.T) { + request := acceptedBuildRequest(t) + request.Chunks[0].Units[0].ID = 999 + if _, err := Build(request); err == nil { + t.Fatal("Build() error = nil, want rejection for chunk units outside accepted source range") + } +} + +func TestCodecRejectsInvalidDurableBoundaries(t *testing.T) { + value, err := Build(acceptedBuildRequest(t)) + if err != nil { + t.Fatal(err) + } + for _, test := range []struct { + name string + mutate func(*ChunkMap) + }{ + {name: "blank identity", mutate: func(value *ChunkMap) { value.RequestedChunker = " " }}, + {name: "malformed digest", mutate: func(value *ChunkMap) { value.SourceDigest = "sha256:ABC" }}, + {name: "index mismatch", mutate: func(value *ChunkMap) { value.Chunks[1].Index = 4 }}, + {name: "duplicate chunk id", mutate: func(value *ChunkMap) { value.Chunks[1].ID = value.Chunks[0].ID }}, + {name: "source mismatch", mutate: func(value *ChunkMap) { value.Chunks[0].SourceRef.SourceID = "other" }}, + {name: "invalid range", mutate: func(value *ChunkMap) { value.Chunks[0].SourceRef.StartUnitID = 0 }}, + {name: "invalid count", mutate: func(value *ChunkMap) { value.Chunks[0].UnitCount = 0 }}, + {name: "invalid namespace", mutate: func(value *ChunkMap) { value.PlanAnnotations[" "] = json.RawMessage(`null`) }}, + {name: "invalid annotation", mutate: func(value *ChunkMap) { value.Chunks[0].Annotations["dnd/scenes"] = json.RawMessage(`{`) }}, + {name: "plan digest mismatch", mutate: func(value *ChunkMap) { value.PlanDigest = "sha256:" + strings.Repeat("a", 64) }}, + } { + t.Run(test.name, func(t *testing.T) { + candidate := clone(value) + test.mutate(&candidate) + if _, err := New().Encode(candidate); err == nil { + t.Fatal("Encode() error = nil, want invalid durable value rejection") + } + }) + } + content, err := New().Encode(value) + if err != nil { + t.Fatal(err) + } + for _, raw := range [][]byte{ + append(append([]byte(nil), content[:len(content)-1]...), []byte(`,"unknown":true}`)...), + append(append([]byte(nil), content...), []byte(` {}`)...), + } { + if _, err := New().Decode(raw); err == nil { + t.Fatalf("Decode(%s) error = nil, want strict JSON rejection", raw) + } + } + formatted := bytes.Replace(content, []byte(`{"title":"Gate"}`), []byte("{\n \"title\": \"Gate\"\n}"), 1) + decoded, err := New().Decode(formatted) + if err != nil { + t.Fatalf("Decode(formatted annotations) error = %v", err) + } + if string(decoded.PlanAnnotations["dnd/scenes"]) != `{"title":"Gate"}` { + t.Fatalf("decoded annotation = %s, want canonical JSON", decoded.PlanAnnotations["dnd/scenes"]) + } +} + +func TestChunkMapOwnershipIsIndependent(t *testing.T) { + request := acceptedBuildRequest(t) + first, err := Build(request) + if err != nil { + t.Fatal(err) + } + request.Plan.Annotations["dnd/scenes"][0] = '[' + first.PlanAnnotations["dnd/scenes"][0] = '[' + second, err := Build(acceptedBuildRequest(t)) + if err != nil { + t.Fatal(err) + } + if string(second.PlanAnnotations["dnd/scenes"]) != `{"title":"Gate"}` { + t.Fatalf("Build() shared mutable annotations: %s", second.PlanAnnotations["dnd/scenes"]) + } +} + +func acceptedBuildRequest(t *testing.T) BuildRequest { + t.Helper() + document := &source.SourceDocument{ + ID: "session-7", Kind: "transcript", Format: "application/json", + Units: []source.SourceUnit{ + {ID: 10, Kind: "segment", Text: "At the gate.", Ref: source.SourceRef{SourceID: "session-7", StartUnitID: 10, EndUnitID: 10}}, + {ID: 3, Kind: "segment", Text: "The guard speaks.", Ref: source.SourceRef{SourceID: "session-7", StartUnitID: 3, EndUnitID: 3}}, + {ID: 20, Kind: "segment", Text: "The party enters.", Ref: source.SourceRef{SourceID: "session-7", StartUnitID: 20, EndUnitID: 20}}, + }, + } + digest, err := source.DigestDocument(document) + if err != nil { + t.Fatal(err) + } + document.Digest = digest + plan := source.ChunkPlan{ + SourceDigest: digest, + Annotations: source.ChunkAnnotations{"dnd/scenes": json.RawMessage(`{"title":"Gate"}`)}, + Ranges: []source.ChunkRange{ + {StartUnitID: 10, EndUnitID: 3, Annotations: source.ChunkAnnotations{"dnd/scenes": json.RawMessage(`{"kind":"narrative"}`)}}, + {StartUnitID: 20, EndUnitID: 20}, + }, + } + chunks, err := source.MaterializeChunkPlan(document, plan) + if err != nil { + t.Fatal(err) + } + return BuildRequest{ + Source: document, Plan: plan, Chunks: chunks, RequestedChunker: "dnd/scenes", + Producer: Producer{InputModule: "seriatim", ChunkModule: "dnd/scenes", LLMProfile: "dnd-scenes"}, + } +} diff --git a/internal/framework/chunkmap/model.go b/internal/framework/chunkmap/model.go new file mode 100644 index 0000000..17c9036 --- /dev/null +++ b/internal/framework/chunkmap/model.go @@ -0,0 +1,51 @@ +// Package chunkmap owns the durable accepted source chunk-map contract. +package chunkmap + +import ( + "gitea.maximumdirect.net/eric/notarius/internal/core/source" + "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" +) + +const ( + ArtifactKind contracts.ArtifactKind = "source/chunk-map" + SchemaID = "notarius.source.chunk_map" + SchemaName = "notarius_source_chunk_map_v1" + SchemaVersion = "v1" + MediaType = "application/json" +) + +// ChunkMap is the durable representation of one accepted materialized chunk plan. +type ChunkMap struct { + SourceID string `json:"source_id"` + SourceDigest string `json:"source_digest"` + PlanDigest string `json:"plan_digest"` + RequestedChunker string `json:"requested_chunker"` + Producer Producer `json:"producer"` + PlanAnnotations source.ChunkAnnotations `json:"plan_annotations"` + Chunks []Chunk `json:"chunks"` +} + +// Producer identifies the component that produced the accepted logical plan. +type Producer struct { + InputModule string `json:"input_module"` + ChunkModule string `json:"chunk_module"` + LLMProfile string `json:"llm_profile,omitempty"` +} + +// Chunk describes one accepted materialized range without source content. +type Chunk struct { + ID string `json:"id"` + Index int `json:"index"` + SourceRef source.SourceRef `json:"source_ref"` + UnitCount int `json:"unit_count"` + Annotations source.ChunkAnnotations `json:"annotations"` +} + +// BuildRequest supplies the accepted runtime state used to build a chunk map. +type BuildRequest struct { + Source *source.SourceDocument + Plan source.ChunkPlan + Chunks []source.Chunk + RequestedChunker string + Producer Producer +} diff --git a/internal/framework/chunkmap/testdata/source_chunk_map.v1.json b/internal/framework/chunkmap/testdata/source_chunk_map.v1.json new file mode 100644 index 0000000..12fdefc --- /dev/null +++ b/internal/framework/chunkmap/testdata/source_chunk_map.v1.json @@ -0,0 +1 @@ +{"source_id":"session-7","source_digest":"sha256:87d04d40537217e2adcbdec841c5873dbc1034c426d83c4578a0431a5ef855f6","plan_digest":"sha256:a6bfc33d52f1c0f4eb3287dd4a00e8672b3d3d9579f9c0366f18a5ff0dba7d14","requested_chunker":"dnd/scenes","producer":{"input_module":"seriatim","chunk_module":"dnd/scenes","llm_profile":"dnd-scenes"},"plan_annotations":{"dnd/scenes":{"title":"Gate"}},"chunks":[{"id":"chunk-000001","index":0,"source_ref":{"source_id":"session-7","start_unit_id":10,"end_unit_id":3},"unit_count":2,"annotations":{"dnd/scenes":{"kind":"narrative"}}},{"id":"chunk-000002","index":1,"source_ref":{"source_id":"session-7","start_unit_id":20,"end_unit_id":20},"unit_count":1,"annotations":{}}]}