Add accepted chunk map contract
This commit is contained in:
261
docs/roadmap/accepted-chunk-map-export.md
Normal file
261
docs/roadmap/accepted-chunk-map-export.md
Normal file
@@ -0,0 +1,261 @@
|
||||
# Accepted Chunk Map Export
|
||||
|
||||
Status: Accepted
|
||||
|
||||
## Purpose
|
||||
|
||||
Notarius already creates and validates one materialized chunk map before any
|
||||
artifact lane executes. That map contains stable chunk identities, ordered
|
||||
current-source ranges, and accepted namespaced annotations. It is useful
|
||||
downstream data, but today it is visible only in internal execution and
|
||||
explicit debug surfaces.
|
||||
|
||||
Add an opt-in durable chunk-map artifact to the output bundle. Export the exact
|
||||
accepted chunks used by the run rather than reconstructing them through an
|
||||
extractor or exposing a chunker's unvalidated model response.
|
||||
|
||||
This is a framework and output concern. Chunking remains pipeline-wide and
|
||||
precedes all artifact lanes.
|
||||
|
||||
## Desired End State
|
||||
|
||||
The production JSON output encoder can be configured to add a canonical
|
||||
`chunk-map.json` file to the logical output bundle. The file describes the
|
||||
accepted materialized chunks used for lane execution without embedding source
|
||||
units or transcript text.
|
||||
|
||||
Use these durable identities:
|
||||
|
||||
| Concern | Identity |
|
||||
| --- | --- |
|
||||
| Artifact kind | `source/chunk-map` |
|
||||
| Logical file | `chunk-map.json` |
|
||||
| Schema ID | `notarius.source.chunk_map` |
|
||||
| Schema name | `notarius_source_chunk_map_v1` |
|
||||
| Schema version | `v1` |
|
||||
| Media type | `application/json` |
|
||||
|
||||
The framework owns the artifact model, schema, validation, and canonical
|
||||
encoding. Output encoders receive the serialized artifact through the generic
|
||||
output request. The JSON encoder owns only the opt-in decision, logical file
|
||||
placement, and output-index entry.
|
||||
|
||||
## Configuration
|
||||
|
||||
Add one strict option to the production `json` output module:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
dnd-session:
|
||||
input: seriatim
|
||||
output:
|
||||
module: json
|
||||
options:
|
||||
include_chunk_map: true
|
||||
```
|
||||
|
||||
`include_chunk_map` is a boolean and defaults to `false`. Unknown options and
|
||||
non-boolean values remain configuration errors. Do not add a separate CLI flag
|
||||
or top-level filesystem option: this choice changes the logical files produced
|
||||
by an output module, not their physical destination.
|
||||
|
||||
Other output encoders may ignore the available chunk-map artifact unless and
|
||||
until they define their own explicit export behavior.
|
||||
|
||||
## Durable Artifact Contract
|
||||
|
||||
The payload is one strict JSON object with this shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"source_id": "session-7",
|
||||
"source_digest": "sha256:0123456789abcdef...",
|
||||
"plan_digest": "sha256:abcdef0123456789...",
|
||||
"requested_chunker": "dnd/scenes",
|
||||
"producer": {
|
||||
"input_module": "seriatim",
|
||||
"chunk_module": "dnd/scenes",
|
||||
"llm_profile": "dnd-scenes"
|
||||
},
|
||||
"plan_annotations": {},
|
||||
"chunks": [
|
||||
{
|
||||
"id": "chunk-000001",
|
||||
"index": 0,
|
||||
"source_ref": {
|
||||
"source_id": "session-7",
|
||||
"start_unit_id": 1,
|
||||
"end_unit_id": 18
|
||||
},
|
||||
"unit_count": 18,
|
||||
"annotations": {
|
||||
"dnd/scenes": {
|
||||
"title": "At the city gate"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Required top-level fields are:
|
||||
|
||||
- `source_id`: the accepted source document identity;
|
||||
- `source_digest`: the canonical digest of that document;
|
||||
- `plan_digest`: the canonical digest of the accepted chunk plan;
|
||||
- `requested_chunker`: the chunk module selected by the current resolved
|
||||
pipeline;
|
||||
- `producer`: the identity of the component that originally produced the
|
||||
accepted plan;
|
||||
- `plan_annotations`: the accepted plan-level annotation namespace map; and
|
||||
- `chunks`: the non-empty ordered list used for lane execution.
|
||||
|
||||
`producer` requires `input_module` and `chunk_module`. `llm_profile` is included
|
||||
only when the producing chunker was LLM-backed. Producer references, module
|
||||
metadata, warnings, creation timestamps, cache paths, and cache actions remain
|
||||
in their existing provenance and diagnostic surfaces; they are not copied into
|
||||
this artifact.
|
||||
|
||||
Each chunk requires:
|
||||
|
||||
- `id`: the stable materialized chunk ID;
|
||||
- `index`: its zero-based position in execution order;
|
||||
- `source_ref`: the exact inclusive current-source range;
|
||||
- `unit_count`: the number of materialized source units in that range; and
|
||||
- `annotations`: the accepted range-level annotation namespace map.
|
||||
|
||||
Unknown fields are rejected at every fixed object level. Annotation namespaces
|
||||
retain their canonical JSON values and may contain domain-specific JSON of any
|
||||
type. Empty annotation maps are encoded as `{}` so the shape remains explicit.
|
||||
|
||||
The durable artifact must guarantee that:
|
||||
|
||||
- source and module identities are non-empty and contain no surrounding
|
||||
whitespace;
|
||||
- source and plan digests use the canonical `sha256:` representation;
|
||||
- chunk IDs are non-empty and unique;
|
||||
- chunk indexes are unique, contiguous, zero-based, and agree with array order;
|
||||
- every chunk reference uses the top-level source identity;
|
||||
- every range has positive endpoint IDs and was validated in source-document
|
||||
order when the artifact was constructed;
|
||||
- every unit count is positive;
|
||||
- annotations contain valid JSON under non-empty canonical namespaces and are
|
||||
canonicalized before digest calculation; and
|
||||
- reconstructing the logical plan from the source digest, plan annotations,
|
||||
chunk ranges, and chunk annotations reproduces `plan_digest`.
|
||||
|
||||
The artifact represents chunk structure, not source content. It must not
|
||||
contain materialized units, transcript bytes, source-unit metadata, chunk
|
||||
content, private model responses, rejected boundary proposals, or debug
|
||||
payloads.
|
||||
|
||||
## Accepted-State And Failure Policy
|
||||
|
||||
Construct the artifact only after plan materialization and the configured chunk
|
||||
validator chain have accepted the chunks. The runner passes the same immutable
|
||||
logical chunk identities, ranges, and annotations used for every lane; no
|
||||
second chunking or model call occurs.
|
||||
|
||||
When `include_chunk_map` is enabled:
|
||||
|
||||
- emit the artifact even if one or more later artifact lanes are rejected;
|
||||
- omit it when chunk validation rejects the candidate map, because no accepted
|
||||
chunk map exists;
|
||||
- retain the ordinary rejection record when it is omitted for that reason; and
|
||||
- treat failure to construct, validate, serialize, index, or write an available
|
||||
accepted chunk map as a run failure, consistent with any explicitly requested
|
||||
durable output.
|
||||
|
||||
A cache hit and a newly generated plan with the same accepted logical plan must
|
||||
produce the same chunk ranges, annotations, plan digest, and materialized chunk
|
||||
identities. The artifact records both the current `requested_chunker` and the
|
||||
stored producer's `chunk_module`, since canonical plan reuse permits those
|
||||
identities to differ.
|
||||
|
||||
## Output Bundle Integration
|
||||
|
||||
Extend `index.json` with an optional `chunk_map` descriptor:
|
||||
|
||||
```json
|
||||
{
|
||||
"chunk_map": {
|
||||
"artifact_kind": "source/chunk-map",
|
||||
"file": "chunk-map.json",
|
||||
"media_type": "application/json",
|
||||
"schema_id": "notarius.source.chunk_map",
|
||||
"schema_name": "notarius_source_chunk_map_v1",
|
||||
"schema_version": "v1"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The descriptor and file are both absent when export is disabled or no accepted
|
||||
chunk map exists. The chunk map does not appear in lane-oriented
|
||||
`output_files`, because it is pipeline-wide and has no lane, extractor, merger,
|
||||
or normalizer identity.
|
||||
|
||||
The existing manifest remains the canonical run-provenance index. Its
|
||||
`chunk_plan` summary continues to own cache mode, lookup and publication
|
||||
actions, producer reference provenance, module metadata, timestamps, and
|
||||
validation status. The chunk-map artifact provides the accepted structure and
|
||||
only the minimal producer identities needed to interpret it independently.
|
||||
|
||||
## Ownership And Architecture
|
||||
|
||||
Use a domain-neutral framework package for the durable chunk-map DTO, embedded
|
||||
JSON Schema, invariant validation, cloning, and canonical serialization. Do not
|
||||
marshal `source.Chunk` directly: its content, units, metadata, and internal
|
||||
fields are intentionally broader than this external contract.
|
||||
|
||||
Extend the output request with an optional cloned serialized chunk-map artifact.
|
||||
The runner constructs that value from the accepted chunk-plan execution result
|
||||
before invoking the output encoder. This keeps source and chunk-plan knowledge
|
||||
out of the generic JSON encoder and allows future encoders to consume the same
|
||||
framework-owned representation.
|
||||
|
||||
The JSON encoder adds the file and index descriptor only when its
|
||||
`include_chunk_map` option is true and the request contains an accepted
|
||||
artifact. It applies the same logical-path validation and pretty-printed JSON
|
||||
conventions as the rest of the bundle.
|
||||
|
||||
## Security And Data Handling
|
||||
|
||||
Chunk annotations may contain model-derived or source-derived information.
|
||||
Treat `chunk-map.json` as durable user output with the same sensitivity and
|
||||
retention expectations as lane artifacts. Opt-in export prevents new durable
|
||||
content from appearing silently in existing pipelines.
|
||||
|
||||
Do not copy annotation values, source ranges, or chunk IDs into the manifest.
|
||||
Do not include external-reference content or filesystem paths in the chunk map.
|
||||
Existing output-directory confinement, atomic writing, and permission policy
|
||||
apply unchanged.
|
||||
|
||||
## Quality And Documentation Policy
|
||||
|
||||
The framework contract must remain strict, canonical, immutable across
|
||||
ownership boundaries, and independent of source or D&D interpretation. The
|
||||
runner must preserve accepted-state and cache-producer provenance, while the
|
||||
JSON encoder must keep opt-in file selection separate from chunk semantics.
|
||||
|
||||
Once implemented, the durable schema belongs in `docs/integrations/`;
|
||||
configuration owns the selectable option; pipeline and state internals own the
|
||||
framework handoff; and a maintained scene-chunking example should demonstrate
|
||||
that namespaced annotations survive export. Tests should protect those
|
||||
observable contracts without relying on exact payload lengths or private
|
||||
helper structure.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This scope does not:
|
||||
|
||||
- add a chunk-map extraction lane or artifact-lane registration;
|
||||
- rerun chunking or ask an LLM to reconstruct accepted chunks;
|
||||
- export transcript content, source units, source-unit metadata, or raw model
|
||||
proposals;
|
||||
- interpret D&D scene annotations as a generic contract;
|
||||
- add scene descriptions, titles, summaries, kinds, or participant extraction;
|
||||
- make chunk maps generated references or inputs to later ordered steps;
|
||||
- change chunk planning, validation, cache selection, or publication behavior;
|
||||
- add a chunk-map import or replay mechanism;
|
||||
- enable export by default; or
|
||||
- introduce a general output-plugin capability negotiation system.
|
||||
@@ -18,20 +18,10 @@ not as committed release dates.
|
||||
|
||||
### Export Accepted Chunk Maps
|
||||
|
||||
- Add an option to emit the accepted materialized chunk map as a proper,
|
||||
framework-owned artifact with a documented schema identity, version, media
|
||||
type, canonical encoding, and source/chunker provenance.
|
||||
- Export the exact ordered chunks used for lane execution, including stable
|
||||
chunk IDs and current-source ranges. Do not expose a model's raw boundary
|
||||
proposal or require a second model call to reconstruct information already
|
||||
owned by the framework.
|
||||
- Treat chunk-map export as an output concern rather than an ordinary
|
||||
extraction lane. Chunking is pipeline-wide and precedes lane extraction; a
|
||||
pseudo-extractor would duplicate work and obscure that ownership boundary.
|
||||
- Keep the generic contract independent of D&D interpretation. Namespaced
|
||||
annotations may be preserved when they are part of the accepted chunk plan,
|
||||
but downstream applications should not need domain-specific annotations to
|
||||
understand chunk identity, order, or source coverage.
|
||||
Add an opt-in, framework-owned durable artifact for the exact accepted
|
||||
materialized chunks used by lane execution. The accepted contract, output
|
||||
boundary, provenance policy, and exclusions are defined in
|
||||
[Accepted Chunk Map Export](accepted-chunk-map-export.md).
|
||||
|
||||
### Extract D&D Scene Descriptions
|
||||
|
||||
|
||||
@@ -1,570 +1,463 @@
|
||||
# D&D NPC Interactions Implementation Plan
|
||||
# Accepted Chunk Map Export Implementation Plan
|
||||
|
||||
Status: Ready for implementation
|
||||
|
||||
## Objective
|
||||
|
||||
Implement the accepted [D&D NPC Interactions](dnd-npc-interactions.md)
|
||||
roadmap as a production D&D artifact lane. The finished lane must extract an
|
||||
ordered list of minimal, evidence-grounded NPC interaction occurrences from
|
||||
each accepted transcript chunk, ground every NPC against a required accepted
|
||||
NPC registry, and preserve the existing fixed pipeline and generated-reference
|
||||
architecture.
|
||||
Implement the accepted [Accepted Chunk Map Export](accepted-chunk-map-export.md)
|
||||
roadmap as an opt-in logical file in the production JSON output bundle. The
|
||||
finished feature must serialize the exact accepted materialized chunks used by
|
||||
the run, preserve framework ownership of chunk semantics, and keep file
|
||||
selection and placement in the output encoder.
|
||||
|
||||
This plan is the implementation authority for sequencing and file-level work.
|
||||
The feature roadmap remains authoritative for product intent, category
|
||||
semantics, occurrence boundaries, evidence policy, and non-goals. Follow
|
||||
[ADR-0009](../adr/0009-minimal-evidence-grounded-extraction-artifacts.md)
|
||||
throughout: do not add descriptive, analytical, relationship, or state fields.
|
||||
This document is the implementation authority for sequencing and concrete
|
||||
engineering decisions. The feature roadmap remains authoritative for product
|
||||
intent, durable field semantics, security policy, and non-goals. Implement the
|
||||
stages in order and do not release or merge a partially completed sequence.
|
||||
|
||||
## Fixed Decisions
|
||||
|
||||
The implementation must use these identities:
|
||||
Use these durable identities:
|
||||
|
||||
| Concern | Identity |
|
||||
| --- | --- |
|
||||
| Extractor key | `dnd/npc-interactions` |
|
||||
| Normalizer key | `dnd/npc-interactions` |
|
||||
| Artifact kind | `dnd/npc-interaction-list` |
|
||||
| Durable schema ID | `notarius.dnd.npc_interactions` |
|
||||
| Durable schema name | `notarius_dnd_npc_interactions_v1` |
|
||||
| Durable schema version | `v1` |
|
||||
| Durable media type | `application/json` |
|
||||
| Private prompt ID | `dnd.npc_interactions` |
|
||||
| Private response schema ID | `notarius.dnd.npc_interactions.llm` |
|
||||
| Reference slot | `npcs` |
|
||||
| Artifact kind | `source/chunk-map` |
|
||||
| Logical file | `chunk-map.json` |
|
||||
| Schema ID | `notarius.source.chunk_map` |
|
||||
| Schema name | `notarius_source_chunk_map_v1` |
|
||||
| Schema version | `v1` |
|
||||
| Media type | `application/json` |
|
||||
| JSON output option | `include_chunk_map` |
|
||||
|
||||
Use a top-level `interactions` array. Each durable record has exactly `name`,
|
||||
`kind`, and `source_refs`. Do not add an interaction ID. The closed kind
|
||||
vocabulary is `mentioned`, `noncombat_presence`, `dialogue`, `combat_ally`,
|
||||
`combat_opponent`, and `other`.
|
||||
Keep the schema at `v1`; Notarius is pre-release and no compatibility layer,
|
||||
dual writer, or migration path is required.
|
||||
|
||||
The `npcs` slot is required for both extraction and normalization. It accepts
|
||||
only the existing `dnd/npc-list` JSON artifact within the existing NPC registry
|
||||
size limit. A bound registry with an empty `npcs` array is valid and requires an
|
||||
empty interaction artifact. An absent binding, malformed registry, failed
|
||||
generated handoff, or unknown interaction name is not permission to perform
|
||||
ungrounded extraction.
|
||||
The framework package owns the durable DTO, embedded JSON Schema, construction
|
||||
from accepted source state, invariant validation, cloning, and canonical JSON
|
||||
serialization. The runner owns construction and passes a serialized artifact
|
||||
through the generic output request. The JSON output encoder owns the opt-in
|
||||
decision, logical filename, formatting, and `index.json` descriptor.
|
||||
|
||||
Use the existing `internal/modules/dnd/npcs/registry.Resolver` for construction-
|
||||
time external references and operation-time generated references. The
|
||||
extractor presents only `Registry.PromptInput()` to the model. Name lookup uses
|
||||
the registry's established comparison-key policy before normalization; the
|
||||
normalizer replaces a recognized variant with the exact canonical registry
|
||||
display name. Registry source references and campaign references never become
|
||||
interaction evidence.
|
||||
Do not register the chunk map with the artifact-lane codec registry. That
|
||||
registry describes lane artifacts; the chunk map is a pipeline-wide source
|
||||
artifact with no extractor, merger, or normalizer.
|
||||
|
||||
An occurrence is one NPC, one kind, and one coherent passage within one
|
||||
accepted chunk. The model must split a category or combat-alignment transition.
|
||||
No occurrence spans chunks. Merge preserves chunk order. Normalization:
|
||||
The runner constructs the serialized value whenever an accepted chunk map
|
||||
exists, independent of the selected output encoder and its options. Output
|
||||
encoders may ignore the optional value. This keeps output-specific
|
||||
configuration out of the runner and leaves future encoders free to define
|
||||
their own explicit behavior.
|
||||
|
||||
1. replaces recognized names with canonical registry display names;
|
||||
2. sorts and exact-deduplicates each record's source references;
|
||||
3. orders records by earliest valid source position, then canonical name,
|
||||
interaction kind, and the canonical source-reference sequence;
|
||||
4. removes records only when canonical name, kind, and the complete canonical
|
||||
source-reference sequence are all identical; and
|
||||
5. never semantically combines nearby, overlapping, or cross-chunk records.
|
||||
Use canonical `sha256:` digests throughout. Annotation values must be
|
||||
semantically canonicalized before plan-digest verification; decoding must
|
||||
accept insignificant JSON whitespace rather than require annotation
|
||||
`json.RawMessage` bytes to already use one formatting style.
|
||||
|
||||
The private model response remains structural and contains name, kind, and
|
||||
integer start/end unit candidates. Mapping attaches the current source ID.
|
||||
Invalid strings, kinds, and ranges must survive mapping far enough for the
|
||||
deterministic validator that owns the rule to reject them.
|
||||
|
||||
For the final record comparator, use source-document unit position for the
|
||||
earliest valid reference. Compare canonical names first by the existing NPC
|
||||
identity comparison key and then by exact display string; compare kinds by
|
||||
their string values; and compare canonical reference sequences
|
||||
lexicographically by `source_id`, start-unit document position, and end-unit
|
||||
document position. Records with valid evidence precede records without it;
|
||||
normalized output validation rejects the latter before durable encoding.
|
||||
|
||||
No framework change, new identity subsystem, DAG behavior, LLM-assisted
|
||||
deduplication, or scene dependency is part of this plan.
|
||||
|
||||
## Testing And Documentation Rules
|
||||
|
||||
Apply [Testing Policy](../policy/testing.md) at every stage:
|
||||
|
||||
- Protect artifact round trips, required fields, enum values, source
|
||||
provenance, registry grounding, deterministic ordering, exact
|
||||
deduplication, reference immutability, and generated handoff behavior.
|
||||
- Use the real codec, registry, resolver, normalizer, and pipeline machinery
|
||||
when they are fast and deterministic. Stub only the structured LLM boundary.
|
||||
- Keep provider calls offline and deterministic.
|
||||
- Test each policy at its narrowest stable owner. Do not repeat every codec,
|
||||
validator, and normalizer case in an end-to-end test.
|
||||
- Do not add a prompt-cache “change detector” based on exact message count,
|
||||
shared-prefix length, token count, or prompt hash. Reuse the shared prompt
|
||||
assets, test rendered behavior and manifest composition, and document the
|
||||
ordering policy instead.
|
||||
- Do not use broad golden snapshots for prompts or diagnostics. A maintained
|
||||
durable JSON fixture is appropriate because the complete serialized artifact
|
||||
is an intentional integration contract.
|
||||
|
||||
Until the final stage, keep unimplemented behavior only in roadmap
|
||||
documentation. Update current-behavior documentation and examples in the same
|
||||
stage that makes the production lane selectable.
|
||||
|
||||
## Stage 1 — Add The Typed Artifact And Durable Codec
|
||||
## Stage 1: Add The Framework-Owned Durable Contract
|
||||
|
||||
### Goal
|
||||
|
||||
Establish the domain type and strict serialized boundary without changing the
|
||||
production module catalog.
|
||||
Create a source-domain chunk-map contract that can be built, validated,
|
||||
encoded, and decoded without depending on an output module or D&D package.
|
||||
|
||||
### Changes
|
||||
### Work
|
||||
|
||||
1. Extend `internal/modules/dnd/types.go` with:
|
||||
- `NPCInteractionListKind`;
|
||||
- a string-backed `NPCInteractionKind`;
|
||||
- exported constants for all six accepted values;
|
||||
- `NPCInteractionList` with `Interactions []NPCInteraction`; and
|
||||
- `NPCInteraction` with `Name string`, `Kind NPCInteractionKind`, and
|
||||
`SourceRefs []source.SourceRef`.
|
||||
2. Add `internal/modules/dnd/codec/npcinteractions` following the existing NPC
|
||||
and combat-turn codec boundary:
|
||||
- expose the fixed schema constants and `MediaType`;
|
||||
- implement the typed artifact codec for `dnd.NPCInteractionList`;
|
||||
- keep `EncodeCandidate` and `DecodeCandidate` strict about one JSON value
|
||||
and unknown fields while preserving typed semantic candidates;
|
||||
- make approved `Encode` and `Decode` enforce non-empty names, the closed
|
||||
kind enum, at least one structurally valid source reference, and the
|
||||
durable schema shape; and
|
||||
- report only `interaction_count` in codec metadata.
|
||||
3. Add
|
||||
`internal/modules/dnd/codec/npcinteractions/assets/schemas/dnd_npc_interactions.v1.json`.
|
||||
It must require the top-level array and all three record fields, reject
|
||||
unknown fields at every object level, encode the six-value enum, require at
|
||||
least one source reference, and use the established durable source-reference
|
||||
shape.
|
||||
4. Add a small maintained fixture under
|
||||
`internal/modules/dnd/codec/npcinteractions/testdata/` containing at least
|
||||
two ordered records with distinct kinds and source ranges.
|
||||
1. Add `internal/framework/chunkmap/` with:
|
||||
|
||||
- `model.go` for the durable DTO and identity constants;
|
||||
- `codec.go` for construction, validation, canonical encoding, and strict
|
||||
decoding;
|
||||
- `assets/schemas/source_chunk_map.v1.json` as the embedded external schema;
|
||||
- `testdata/source_chunk_map.v1.json` as one valid contract fixture; and
|
||||
- focused package tests.
|
||||
|
||||
2. Define constants for the fixed artifact kind, schema identity, schema
|
||||
version, and media type. Define DTOs matching the feature roadmap exactly:
|
||||
|
||||
- `ChunkMap` with `source_id`, `source_digest`, `plan_digest`,
|
||||
`requested_chunker`, `producer`, `plan_annotations`, and `chunks`;
|
||||
- `Producer` with `input_module`, `chunk_module`, and optional
|
||||
`llm_profile`; and
|
||||
- `Chunk` with `id`, `index`, `source_ref`, `unit_count`, and
|
||||
`annotations`.
|
||||
|
||||
Do not expose source units, chunk content, source metadata, producer
|
||||
references or metadata, warnings, timestamps, cache actions, or model
|
||||
responses.
|
||||
|
||||
3. Provide a small package API:
|
||||
|
||||
```go
|
||||
type BuildRequest struct {
|
||||
Source *source.SourceDocument
|
||||
Plan source.ChunkPlan
|
||||
Chunks []source.Chunk
|
||||
RequestedChunker string
|
||||
Producer Producer
|
||||
}
|
||||
|
||||
func Build(BuildRequest) (ChunkMap, error)
|
||||
func Serialize(BuildRequest) (contracts.SerializedArtifact, error)
|
||||
func New() *Codec
|
||||
```
|
||||
|
||||
`Codec` must expose the same identity and encode/decode operations used by
|
||||
other durable codecs, but it is invoked directly by this package and the
|
||||
JSON encoder rather than added to `pipeline.ArtifactCodecRegistry`.
|
||||
|
||||
4. Make `Build` the trusted boundary between runtime source structures and the
|
||||
external contract:
|
||||
|
||||
- validate the source document and accepted plan with existing source
|
||||
validators;
|
||||
- calculate the source and plan digests with `source.DigestDocument` and
|
||||
`source.DigestChunkPlan`, and require the calculated source digest to
|
||||
equal both `Source.Digest` and `Plan.SourceDigest`;
|
||||
- require a non-empty requested chunker and producer module identities;
|
||||
- verify the supplied materialized chunks correspond exactly, in order, to
|
||||
the accepted plan and source document, including IDs, indexes, source
|
||||
ranges, unit membership/counts, plan annotations, and range annotations;
|
||||
- validate range order against source-document position rather than numeric
|
||||
unit-ID assumptions;
|
||||
- deep-clone all references and annotation values; and
|
||||
- encode absent annotation namespaces as non-nil empty maps so JSON contains
|
||||
`{}`.
|
||||
|
||||
5. Make codec decoding strict at fixed object levels:
|
||||
|
||||
- accept one JSON value only and reject trailing content;
|
||||
- reject unknown fields;
|
||||
- validate trimmed, non-empty identities;
|
||||
- require canonical lower-case `sha256:` digests;
|
||||
- require a non-empty chunk array;
|
||||
- require unique stable chunk IDs and contiguous zero-based indexes matching
|
||||
array order;
|
||||
- require every source reference to use the top-level source identity and
|
||||
positive range endpoints;
|
||||
- require positive unit counts;
|
||||
- validate non-empty canonical annotation namespaces and valid JSON values;
|
||||
- canonicalize annotation JSON before using the existing plan-digest helper
|
||||
to reconstruct and verify `plan_digest`; and
|
||||
- return defensively owned DTO data.
|
||||
|
||||
Standalone decoding cannot prove range order against a source document it
|
||||
does not contain. Construction owns that stronger invariant; decoding owns
|
||||
the structural, identity, and digest invariants available from the durable
|
||||
payload.
|
||||
|
||||
6. Make serialization return a `contracts.SerializedArtifact` with the fixed
|
||||
kind, schema, and media type, canonical content, and no duplicated metadata.
|
||||
Validate the embedded schema defensively so the checked-in schema and Go
|
||||
contract cannot silently diverge.
|
||||
|
||||
7. Define a strict JSON Schema with:
|
||||
|
||||
- `additionalProperties: false` at every fixed object level;
|
||||
- all roadmap-required fields;
|
||||
- `minItems: 1` for chunks;
|
||||
- positive integer range endpoints and unit counts;
|
||||
- zero-or-greater indexes;
|
||||
- the pattern `^sha256:[0-9a-f]{64}$` for digests;
|
||||
- non-empty strings for identities;
|
||||
- optional, non-empty `llm_profile`; and
|
||||
- annotation objects whose namespace values may be arbitrary JSON.
|
||||
|
||||
### Tests
|
||||
|
||||
At the codec package boundary, cover:
|
||||
Add focused behavioral tests proving:
|
||||
|
||||
- fixture decode/encode round trip and canonical compact JSON;
|
||||
- exact kind, schema identity, version, media type, and Go type registration;
|
||||
- empty-list support and nil-versus-present-empty behavior where the existing
|
||||
codecs distinguish it;
|
||||
- strict rejection of malformed JSON, trailing values, and unknown fields;
|
||||
- approved-boundary rejection of each meaningful required-field, enum, and
|
||||
source-reference violation;
|
||||
- candidate-boundary preservation of semantic values for later validators; and
|
||||
- defensive copies for schema bytes and metadata.
|
||||
- the valid fixture round-trips through the codec;
|
||||
- building from a real accepted source, plan, and materialized chunk list
|
||||
produces the fixed envelope and exact durable fields;
|
||||
- output is deterministic and annotation JSON formatting does not alter the
|
||||
reconstructed plan digest;
|
||||
- unknown fields, trailing JSON, malformed identities/digests, invalid
|
||||
indexes, duplicate IDs, mismatched source identities, invalid ranges or
|
||||
counts, malformed annotation namespaces/values, and plan-digest mismatches
|
||||
are rejected;
|
||||
- construction rejects materialized chunks that differ from the accepted plan
|
||||
or document order; and
|
||||
- mutating caller-owned inputs or returned values cannot mutate another owned
|
||||
representation.
|
||||
|
||||
Do not duplicate source-document existence checks in the codec; those belong to
|
||||
the source-reference validator.
|
||||
Keep these tests semantic. Do not add exact byte-length, exact schema-length,
|
||||
private-helper, or redundant field-by-field “change detector” tests.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- The new types and codec compile and pass focused tests.
|
||||
- The codec can be registered into an isolated artifact codec registry with
|
||||
exact type `dnd.NPCInteractionList`.
|
||||
- No production registrar or configuration catalog exposes the new kind yet.
|
||||
- The framework contract can represent and validate the roadmap payload
|
||||
without importing output or D&D packages.
|
||||
- Construction proves the artifact describes the exact accepted materialized
|
||||
chunks.
|
||||
- No production output behavior has changed yet.
|
||||
|
||||
## Stage 2 — Implement The Chunk-Scoped Extractor And Prompt
|
||||
## Stage 2: Carry The Accepted Artifact Through The Runner
|
||||
|
||||
### Goal
|
||||
|
||||
Add an independently testable LLM-backed extractor that requires NPC grounding,
|
||||
maps only minimal private output, and follows the established D&D prompt-cache
|
||||
layout.
|
||||
Construct the durable artifact at the accepted chunk-plan boundary and make it
|
||||
available to output encoders without changing default logical files.
|
||||
|
||||
### Changes
|
||||
### Work
|
||||
|
||||
1. Add `internal/modules/dnd/extract/npcinteractions` using the current D&D
|
||||
extractor organization:
|
||||
- `assets.go`;
|
||||
- `canonicalize.go`;
|
||||
- `extractor.go`;
|
||||
- `model.go`;
|
||||
- `schema.go`;
|
||||
- `scriptorium_assets.go`;
|
||||
- corresponding focused tests; and
|
||||
- package-local embedded prompt and schema assets.
|
||||
2. Give the extractor strict empty options, required capabilities `chunks` and
|
||||
`source.transcript`, provided capability `dnd.npc_interactions`, and artifact
|
||||
kind `dnd.NPCInteractionListKind`.
|
||||
3. Build its reference slots from `shared.ReferenceSlots(...)`, then append the
|
||||
`npcs` slot with:
|
||||
- `Required: true`;
|
||||
- accepted media type `application/json`;
|
||||
- accepted artifact kind `dnd.NPCListKind`;
|
||||
- the existing registry maximum byte count; and
|
||||
- a description stating that the accepted registry is required identity
|
||||
grounding, not evidence.
|
||||
Return defensive, consistently sorted slot declarations from both
|
||||
`ModuleSpec()` and the constructed extractor.
|
||||
4. Construct an `npcregistry.Resolver` from the preparation-time reference set.
|
||||
Resolve operation references inside `Extract`, require `Registry.Bound()`,
|
||||
and fail before the LLM call if the required registry is absent or invalid.
|
||||
Preserve the resolver's content-safe error behavior.
|
||||
5. Use `shared.ChunkPromptMaterial` and `shared.PromptInputs`. Replace the
|
||||
ordinary `npcs` input with the resolved names-only `Registry.PromptInput()`.
|
||||
Do not place full registry bytes, NPC IDs, registry evidence, or provenance
|
||||
in prompt inputs.
|
||||
6. Add these assets:
|
||||
- `assets/prompts/dnd.npc_interactions.yaml`;
|
||||
- `assets/prompts/task.md`;
|
||||
- `assets/prompts/instructions.md`; and
|
||||
- `assets/schemas/dnd_npc_interactions_llm.v1.json`.
|
||||
7. Use the existing shared prompt assets in this exact semantic order:
|
||||
common system; common extraction evidence; common identity; common campaign
|
||||
references; common NPC grounding; lane task; lane instructions; variable
|
||||
transcript. Preserve the established cache-control placement used by the
|
||||
spell and combat-turn manifests.
|
||||
8. Revise the shared `common-dnd-npcs.md` wording once so it truthfully applies
|
||||
to all three consumers: a normalized registry is always presented to the
|
||||
prompt and may be empty. Keep its identity, context-only rule, and
|
||||
prohibition on treating registry provenance as event evidence. Do not fork a
|
||||
nearly identical interaction-specific grounding asset.
|
||||
9. The lane prompt must state the category definitions, precedence, occurrence
|
||||
splitting rules, registry-only name restriction, empty-output behavior, and
|
||||
exclusions from the feature roadmap. Keep the response schema structural:
|
||||
it requires the envelope and fields but leaves enum membership, non-empty
|
||||
values, and source semantics to deterministic validators.
|
||||
10. Map private records to `dnd.NPCInteractionList`, attach the current source
|
||||
ID, sort and exact-deduplicate ranges within each candidate, and stable-sort
|
||||
candidates with valid evidence by earliest source-document position.
|
||||
Preserve invalid candidate fields rather than repairing or dropping them.
|
||||
11. Expose prompt, response-schema, and mapping-policy identities through
|
||||
manifest metadata and checkpoint fingerprints, following the current D&D
|
||||
extractors. Include the NPC names-only projection digest in local checkpoint
|
||||
identity. For a preparation-time external registry, expose only bounded
|
||||
digest/count metadata; do not place names or source content in metadata.
|
||||
Generated-reference provenance remains framework-owned.
|
||||
1. Extend `contracts.OutputRequest` with:
|
||||
|
||||
```go
|
||||
ChunkMap *contracts.SerializedArtifact `json:"chunk_map,omitempty"`
|
||||
```
|
||||
|
||||
Update its cloning/ownership path so the envelope, content bytes, and
|
||||
metadata are defensively copied. Do not place chunk-map content in the run
|
||||
manifest or generic request metadata.
|
||||
|
||||
2. In `internal/framework/pipeline/runner.go`, construct the serialized chunk
|
||||
map after materialization and configured chunk validation accept the plan,
|
||||
and before the output encoder is invoked. Build it from:
|
||||
|
||||
- the accepted source document;
|
||||
- the accepted logical plan;
|
||||
- the exact materialized chunks passed to lanes;
|
||||
- the chunk module selected by the current resolved pipeline as
|
||||
`requested_chunker`; and
|
||||
- the original producer identities recorded by the accepted chunk-plan
|
||||
record.
|
||||
|
||||
3. Preserve cache semantics:
|
||||
|
||||
- a generated or bypassed plan records the current producer;
|
||||
- a cache hit keeps the stored producer's input module, chunk module, and
|
||||
optional LLM profile;
|
||||
- `requested_chunker` still records the current resolved chunk module; and
|
||||
- producer references, metadata, warnings, and cache details remain only in
|
||||
existing provenance surfaces.
|
||||
|
||||
4. Apply the accepted-state policy:
|
||||
|
||||
- pass no chunk map when chunk validation rejects the candidate;
|
||||
- retain the artifact when a later extraction, merge, or normalization lane
|
||||
is rejected;
|
||||
- treat artifact construction or serialization failure after acceptance as
|
||||
a framework run error; and
|
||||
- do not rerun chunking or make another LLM request.
|
||||
|
||||
5. If debug serialization records the output request, represent the new value
|
||||
through the existing safe serialized-artifact envelope conventions. Do not
|
||||
duplicate its content into the manifest or add a new default debug surface.
|
||||
|
||||
### Tests
|
||||
|
||||
At stable package boundaries, cover:
|
||||
Extend `internal/framework/pipeline/runner_chunk_plan_test.go` and the nearest
|
||||
existing output-request ownership tests to prove:
|
||||
|
||||
- nil/cancelled/invalid extraction requests and provider failures;
|
||||
- required-slot declaration and rejection of an unbound or malformed registry
|
||||
before any LLM call;
|
||||
- operation-time generated registry resolution by an extractor prepared
|
||||
without generated bytes;
|
||||
- exact names-only registry prompt input and separation from transcript
|
||||
evidence;
|
||||
- empty registry plus empty response;
|
||||
- mapping of every category, current-source attachment, evidence
|
||||
canonicalization, and source-position ordering;
|
||||
- preservation of invalid names, kinds, and ranges for validators;
|
||||
- strict private JSON shape and unknown-field rejection;
|
||||
- prompt rendering with required inputs, category policy, shared assets, and
|
||||
transcript-last ordering;
|
||||
- prompt/schema registration and content-safe metadata/fingerprints; and
|
||||
- defensive module specifications and strict rejection of unknown options.
|
||||
- accepted generated and cached plans reach a capturing output encoder as a
|
||||
valid serialized chunk map;
|
||||
- cache reuse distinguishes current `requested_chunker` from the stored
|
||||
producer chunker;
|
||||
- chunk rejection yields a nil chunk map;
|
||||
- a later lane rejection still leaves the accepted chunk map available;
|
||||
- captured requests do not alias runner, store, or caller-owned bytes/maps.
|
||||
|
||||
Run the existing spell and combat-turn prompt/asset tests after changing the
|
||||
shared NPC prompt fragment. Do not assert an exact shared-prefix length.
|
||||
Use existing fake modules and stores. Do not add an LLM-backed integration test
|
||||
for behavior already deterministically owned by the runner, and do not add a
|
||||
production injection seam solely to force an otherwise unreachable
|
||||
construction failure; Stage 1 owns invalid-construction coverage.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- The extractor is constructible and testable through the typed extractor
|
||||
contract.
|
||||
- A valid request makes one scheduled structured-completion call and returns a
|
||||
typed candidate with current-source provenance.
|
||||
- Missing required NPC grounding cannot reach the model.
|
||||
- The extractor remains unregistered in the production D&D family until later
|
||||
stages provide the rest of the lane.
|
||||
- Every accepted chunk plan yields one validated serialized chunk map in the
|
||||
output request.
|
||||
- Rejected chunk plans yield none, while later lane rejection does not discard
|
||||
it.
|
||||
- Existing output encoders remain behaviorally unchanged.
|
||||
|
||||
## Stage 3 — Add Deterministic Artifact Validators
|
||||
## Stage 3: Add Opt-In JSON Bundle Export
|
||||
|
||||
### Goal
|
||||
|
||||
Give each semantic invariant one clear validation owner and make invalid model
|
||||
output a rejection rather than a normalization repair or framework error.
|
||||
Expose the available framework artifact as `chunk-map.json` only when the JSON
|
||||
output binding explicitly enables it.
|
||||
|
||||
### Changes
|
||||
### Work
|
||||
|
||||
Add these typed validator packages, each with strict empty options, bounded
|
||||
diagnostics, typed registration, and the fixed artifact kind:
|
||||
1. Add an `Options` field:
|
||||
|
||||
1. `internal/modules/dnd/validate/npcinteractions/shape`
|
||||
(`extract/dnd/npc-interactions/shape`):
|
||||
- require a non-empty trimmed name without mutating it;
|
||||
- require one of the six kinds; and
|
||||
- require at least one source-reference candidate with structurally required
|
||||
values.
|
||||
2. `internal/modules/dnd/validate/npcinteractions/registry`
|
||||
(`extract/dnd/npc-interactions/registry`):
|
||||
- use `npcregistry.Resolver` at construction and operation time;
|
||||
- require a bound registry;
|
||||
- approve a name when `Registry.Lookup` recognizes it under the established
|
||||
comparison policy;
|
||||
- reject unknown names in stable record order; and
|
||||
- never use registry evidence as transcript evidence.
|
||||
3. `internal/modules/dnd/validate/npcinteractions/source_refs`
|
||||
(`extract/dnd/npc-interactions/source_refs`):
|
||||
- require the current source identity;
|
||||
- require referenced units to exist; and
|
||||
- require valid inclusive start/end order through the current document.
|
||||
Follow the established D&D citation helpers and rejection aggregation
|
||||
limits rather than introducing a generic framework dependency on the
|
||||
interaction type.
|
||||
4. `internal/modules/dnd/validate/npcinteractions/source_relatedness`
|
||||
(`extract/dnd/npc-interactions/source_relatedness`):
|
||||
- inspect only current transcript text covered by the cited ranges;
|
||||
- emit at most one bounded warning per record when the NPC name cannot be
|
||||
related to that evidence;
|
||||
- do not claim to validate category agreement; and
|
||||
- ignore campaign and registry content for relatedness.
|
||||
5. `internal/modules/dnd/validate/npcinteractions/invariants`
|
||||
(`normalize/dnd/npc-interactions/invariants`):
|
||||
- require the exact canonical display name returned by registry lookup;
|
||||
- require canonical source-reference order with no duplicate ranges;
|
||||
- require the complete list order defined in Fixed Decisions; and
|
||||
- reject exact duplicate normalized records.
|
||||
```go
|
||||
IncludeChunkMap bool
|
||||
```
|
||||
|
||||
The registry and invariants validators must support operation-time generated
|
||||
references. Construction-time metadata and fingerprints follow the resolver
|
||||
pattern used by registry-aware modules and must not include NPC names or
|
||||
content.
|
||||
Keep `New()` as the default-disabled constructor and add
|
||||
`NewWithOptions(Options)` for configured construction and focused tests.
|
||||
|
||||
2. Update the JSON output builder to decode only `include_chunk_map`:
|
||||
|
||||
- omit or `false` means disabled;
|
||||
- require a boolean when present; and
|
||||
- continue rejecting unknown keys.
|
||||
|
||||
3. When disabled, ignore an available `OutputRequest.ChunkMap` and preserve the
|
||||
exact existing bundle shape. When enabled:
|
||||
|
||||
- omit both file and descriptor if the request has no accepted chunk map;
|
||||
- validate the artifact kind, schema identity/version, and media type;
|
||||
- decode through `chunkmap.New()` so malformed or non-canonical payloads
|
||||
fail the run;
|
||||
- write valid content as pretty-printed JSON with the encoder's established
|
||||
trailing-newline convention; and
|
||||
- use the existing logical-path validation.
|
||||
|
||||
Reuse or generalize the current serialized-artifact JSON helper rather than
|
||||
adding a second subtly different validation and formatting path.
|
||||
|
||||
4. Extend the output index model with an optional `chunk_map` descriptor
|
||||
containing exactly:
|
||||
|
||||
- `artifact_kind`;
|
||||
- `file`;
|
||||
- `media_type`;
|
||||
- `schema_id`;
|
||||
- `schema_name`; and
|
||||
- `schema_version`.
|
||||
|
||||
Do not add the chunk map to lane-oriented `output_files`, and do not change
|
||||
the manifest.
|
||||
|
||||
5. Treat a present but invalid accepted artifact as an output error. Do not
|
||||
silently omit it when export was explicitly requested.
|
||||
|
||||
### Tests
|
||||
|
||||
Give each policy one primary test owner:
|
||||
Extend `internal/modules/generic/output/json/encoder_test.go` and existing
|
||||
output registration tests to prove:
|
||||
|
||||
- shape tests own empty names, enum membership, and missing evidence;
|
||||
- registry tests own required binding, comparison-key recognition, unknown
|
||||
names, operation-time generated overrides, empty registries, and content-safe
|
||||
failures;
|
||||
- source-reference tests own wrong source IDs, missing units, reversed ranges,
|
||||
and valid multi-range records;
|
||||
- relatedness tests own current-transcript-only warnings and bounded
|
||||
diagnostics; and
|
||||
- invariant tests own exact canonical names, final ordering, canonical evidence
|
||||
sequences, and duplicate rejection.
|
||||
- option omission and `false` retain the current default bundle even when an
|
||||
artifact is available;
|
||||
- `true` plus a valid artifact emits `chunk-map.json` and the exact optional
|
||||
index descriptor;
|
||||
- `true` plus no accepted artifact emits neither file nor descriptor;
|
||||
- the chunk map never appears in lane `output_files`;
|
||||
- wrong option types and unknown options are rejected;
|
||||
- wrong kind, schema, media type, or invalid content returns an error;
|
||||
- emitted JSON is valid, formatted consistently, and newline-terminated; and
|
||||
- encoding does not mutate the request or serialized content.
|
||||
|
||||
Also verify that validators do not mutate artifacts, source documents, or
|
||||
registries and that their specs register only for
|
||||
`dnd.NPCInteractionListKind`.
|
||||
Prefer testing the observable logical file map and decoded index rather than
|
||||
private helper calls or exact byte lengths.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Valid typed candidates are approved and malformed or ungrounded candidates
|
||||
are rejected at the intended boundary.
|
||||
- No validator silently canonicalizes, drops, or merges records.
|
||||
- References can explain identity but cannot satisfy source evidence checks.
|
||||
- Existing JSON configurations remain unchanged by default.
|
||||
- An explicitly enabled JSON output produces exactly one validated
|
||||
`chunk-map.json` file and one pipeline-wide index descriptor when an
|
||||
accepted map exists.
|
||||
- No lane identity or manifest structure is fabricated for the chunk map.
|
||||
|
||||
## Stage 4 — Add Deterministic Merge And Normalization
|
||||
## Stage 4: Prove The Assembled Behavior And Document It
|
||||
|
||||
### Goal
|
||||
|
||||
Preserve chunk-scoped occurrences through merge, then produce the canonical
|
||||
ordered artifact without semantic inference.
|
||||
Demonstrate the feature with the production D&D scene chunker, publish all
|
||||
current-behavior documentation, and close the roadmap scope.
|
||||
|
||||
### Changes
|
||||
### Work
|
||||
|
||||
1. Add `internal/modules/dnd/normalize/npcinteractions` with strict empty
|
||||
options and the fixed module key and artifact kind.
|
||||
2. Declare the same required `npcs` reference slot as the extractor. Construct
|
||||
and resolve `npcregistry.Resolver` at the same preparation/operation
|
||||
boundaries, and require a bound registry.
|
||||
3. Implement the normalization algorithm in Fixed Decisions:
|
||||
- clone all nested values;
|
||||
- normalize display whitespace only as needed for lookup;
|
||||
- replace every recognized name with the registry's exact canonical display
|
||||
name;
|
||||
- canonicalize each reference list without merging overlapping ranges;
|
||||
- compute order from source-document positions rather than assuming numeric
|
||||
unit IDs are contiguous;
|
||||
- apply every specified tie-breaker; and
|
||||
- collapse only exact records after canonicalization.
|
||||
4. Preserve nil versus present-empty list behavior consistently with the other
|
||||
D&D normalizers.
|
||||
5. Emit bounded warnings for visible canonical-name changes, reference
|
||||
canonicalization, record reordering, and exact duplicate removal. Use stable
|
||||
reason codes and the shared D&D diagnostics helpers. Never include reference
|
||||
content in a warning.
|
||||
6. Add an append-only merger function for `dnd.NPCInteractionList` beside the
|
||||
other D&D typed append functions in
|
||||
`internal/modules/dnd/register/merge.go`. It must preserve accepted
|
||||
chunk/lane order, preserve present-empty semantics, and deep-clone source
|
||||
reference slices so outputs do not alias inputs. Do not register it until
|
||||
Stage 5.
|
||||
7. Add normalization-policy and NPC projection identities to checkpoint
|
||||
fingerprints. Manifest metadata may include the policy identity and bounded
|
||||
external-registry digest/count, but not names or source content.
|
||||
1. Extend the representative assembled production contract in
|
||||
`internal/cli/production_contract_test.go`:
|
||||
|
||||
### Tests
|
||||
- configure `dnd/scenes` with JSON `include_chunk_map: true`;
|
||||
- execute through normal production assembly with the existing deterministic
|
||||
fake LLM;
|
||||
- read `index.json` and `chunk-map.json`;
|
||||
- decode the artifact through the framework codec; and
|
||||
- assert the actual accepted scene ranges, stable IDs/order, source and plan
|
||||
identity, requested/current producer provenance, and surviving namespaced
|
||||
scene annotations.
|
||||
|
||||
Cover:
|
||||
This is the single assembled proof. Do not duplicate all codec and encoder
|
||||
edge cases at the CLI layer.
|
||||
|
||||
- canonical registry name replacement, including Unicode/case/spacing lookup;
|
||||
- required and operation-time generated registry behavior;
|
||||
- source-reference ordering and exact range deduplication;
|
||||
- ordering by real document position with all deterministic tie-breakers;
|
||||
- separation of category transitions, alignment transitions, nearby records,
|
||||
overlapping-but-nonidentical evidence, and nonidentical records from
|
||||
different chunks;
|
||||
- collapse of exact duplicates only;
|
||||
- stable, bounded warnings;
|
||||
- nil/present-empty behavior;
|
||||
- input ownership and nested-slice cloning; and
|
||||
- append-order merger behavior independently of normalization.
|
||||
2. Add a copyable maintained example such as
|
||||
`examples/dnd-scene-chunk-map.config.yml` that uses `dnd/scenes`, one normal
|
||||
D&D artifact lane, and the opt-in JSON output option. Add it to
|
||||
`internal/cli/example_contract_test.go` so schema/config drift is caught by
|
||||
the existing example contract.
|
||||
|
||||
Use table-driven pure normalization cases for the dense ordering and duplicate
|
||||
rules, and package-level normalizer tests for resolver and warning behavior.
|
||||
3. Add `docs/integrations/chunk-map.md` as the canonical durable contract
|
||||
reference. Document fixed identities, every payload field, invariants,
|
||||
accepted-state behavior, cache provenance semantics, sensitivity, and
|
||||
excluded content. Link to the embedded schema without duplicating it in the
|
||||
prose.
|
||||
|
||||
### Completion Criteria
|
||||
4. Update current-behavior documentation:
|
||||
|
||||
- Normalization is deterministic and idempotent.
|
||||
- Running normalization twice does not change the value or emit new
|
||||
transformation warnings on the second pass.
|
||||
- Semantically distinct occurrences remain distinct.
|
||||
- The module and merger compile but are not yet selectable through production
|
||||
composition.
|
||||
- `docs/integrations/json-output.md`: optional logical file and index
|
||||
descriptor;
|
||||
- `docs/config.md`: strict `include_chunk_map` option and copyable binding;
|
||||
- `docs/operations.md`: opt-in sensitivity and retention implications;
|
||||
- `docs/internal/pipeline.md`: accepted-plan construction and output-request
|
||||
handoff;
|
||||
- `docs/internal/state.md`: logical output and manifest/index ownership;
|
||||
- `docs/internal/modules.md`: JSON encoder option and responsibility; and
|
||||
- `docs/internal/overview.md`: framework chunk-map contract and encoder
|
||||
boundary.
|
||||
|
||||
## Stage 5 — Compose The Complete Production Lane
|
||||
Keep implementation details in internal docs, user-selectable behavior in
|
||||
configuration docs, and the durable schema only in integrations docs.
|
||||
|
||||
### Goal
|
||||
5. Close planning state:
|
||||
|
||||
Register the complete typed lane atomically so configuration cannot select a
|
||||
partial implementation.
|
||||
- mark `accepted-chunk-map-export.md` as implemented;
|
||||
- remove the completed “Export Accepted Chunk Maps” entry from
|
||||
`docs/roadmap/future.md`; and
|
||||
- mark this implementation plan completed, preserving it until the user
|
||||
chooses to retire completed roadmap documents.
|
||||
|
||||
### Changes
|
||||
|
||||
Update the D&D registrar:
|
||||
|
||||
1. In `internal/modules/dnd/register/modules.go`, register:
|
||||
- the NPC-interaction codec;
|
||||
- extractor and prompt assets;
|
||||
- the append-order merger variant for
|
||||
`dnd.NPCInteractionListKind`;
|
||||
- the `dnd/npc-interactions` normalizer; and
|
||||
- the generic no-op normalizer variant for the new exact Go type.
|
||||
2. In `internal/modules/dnd/register/validators.go`, register all five domain
|
||||
validators plus generic always-accept and always-reject typed variants for
|
||||
the new artifact type.
|
||||
3. In `internal/modules/dnd/register/chains.go`, register:
|
||||
- extraction chain:
|
||||
`generic/valid_json`,
|
||||
`extract/dnd/npc-interactions/shape`,
|
||||
`extract/dnd/npc-interactions/registry`,
|
||||
`extract/dnd/npc-interactions/source_refs`,
|
||||
`generic/valid_json_schema`,
|
||||
`extract/dnd/npc-interactions/source_relatedness`;
|
||||
- normalization chain:
|
||||
`generic/valid_json`,
|
||||
`extract/dnd/npc-interactions/shape`,
|
||||
`extract/dnd/npc-interactions/registry`,
|
||||
`normalize/dnd/npc-interactions/invariants`,
|
||||
`extract/dnd/npc-interactions/source_refs`,
|
||||
`generic/valid_json_schema`,
|
||||
`extract/dnd/npc-interactions/source_relatedness`.
|
||||
4. Do not add a default merge validator chain. The append merger performs no
|
||||
semantic mutation, and extraction plus normalized boundaries own the
|
||||
consequential policies.
|
||||
5. Extend registrar contract tests to verify exact typed coverage, keys,
|
||||
artifact kinds, prompt/schema assets, reference-slot agreement, and default
|
||||
validator order.
|
||||
|
||||
### Tests
|
||||
|
||||
- Extend the existing D&D family registration test rather than creating
|
||||
parallel catalog snapshots.
|
||||
- Verify that the extractor and normalizer both expose required compatible
|
||||
`npcs` slots and that generated `dnd/npc-list` bindings resolve only from an
|
||||
earlier step.
|
||||
- Verify production resolution rejects a missing required binding, a later or
|
||||
same-step producer, a wrong artifact kind, and incompatible media/schema
|
||||
metadata through existing resolver behavior.
|
||||
- Avoid retesting the framework's general ordered-step failure matrix; add only
|
||||
interaction-specific composition cases not already covered generically.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Production registration exposes one complete type-consistent lane.
|
||||
- A valid two-step NPC-to-interactions pipeline resolves and prepares.
|
||||
- Invalid or missing NPC dependencies fail before source parsing or LLM work
|
||||
whenever statically discoverable.
|
||||
|
||||
## Stage 6 — Prove The Workflow And Publish Current Contracts
|
||||
|
||||
### Goal
|
||||
|
||||
Exercise the assembled generated-reference workflow and move all implemented
|
||||
behavior into its canonical current documentation.
|
||||
|
||||
### Changes
|
||||
|
||||
1. Add one representative integration test under
|
||||
`internal/modules/integration` with focused testdata:
|
||||
- step 1 extracts and normalizes NPCs;
|
||||
- step 2 consumes the generated NPC artifact in an interaction lane;
|
||||
- the fake structured LLM returns multiple interaction kinds and at least
|
||||
one name variant;
|
||||
- the assertion proves the names-only handoff, current-transcript evidence,
|
||||
canonical name, stable chronology, and final durable artifact.
|
||||
2. Include one failure assertion showing that a rejected or absent normalized
|
||||
NPC producer prevents interaction extraction. Reuse the existing runner
|
||||
dependency behavior; do not duplicate every framework failure case.
|
||||
3. Add `examples/dnd-npc-interactions.config.yml` as a complete copyable
|
||||
two-step profile. The first step produces `npcs`; the second step binds that
|
||||
exact step/lane artifact to its `npcs` reference and selects
|
||||
`dnd/npc-interactions` for extract and normalize.
|
||||
4. Add the example to the existing CLI production configuration contract test
|
||||
so it is parsed and resolved offline.
|
||||
5. Add the durable external contract at
|
||||
`docs/integrations/dnd-npc-interaction-artifacts.md`. It owns schema
|
||||
identity, JSON shape, categories, evidence semantics, ordering,
|
||||
normalization, validator chains, generated-reference behavior, and manifest
|
||||
metadata.
|
||||
6. Update current canonical documentation:
|
||||
- `docs/config.md` for module/validator catalogs, default chains, required
|
||||
reference slot, and the maintained example;
|
||||
- `docs/internal/modules.md` for implementation ownership, prompt inputs,
|
||||
registry resolution, validation, merge, and normalization;
|
||||
- `docs/internal/overview.md` for the implemented component inventory;
|
||||
- `docs/internal/llm.md` only as needed to include the new lane in the
|
||||
existing D&D prompt-order/cache policy without duplicating the manifest;
|
||||
- `docs/integrations/dnd-npc-artifacts.md` to identify NPC interactions as a
|
||||
consumer while retaining the rule that registry evidence is not event
|
||||
evidence; and
|
||||
- any directly affected configuration or integration links.
|
||||
7. Remove the NPC-interactions entry from `docs/roadmap/future.md` once the
|
||||
production behavior and current documentation are complete. Change
|
||||
`docs/roadmap/dnd-npc-interactions.md` to `Status: Implemented` pending its
|
||||
later retirement; do not leave it as the canonical current contract.
|
||||
8. Create a small human-review worksheet or fixture set only if the repository
|
||||
already has an appropriate non-test evaluation home. Otherwise record the
|
||||
manual evaluation results in the implementation handoff rather than
|
||||
inventing a new framework. Exercise all six categories, transitions,
|
||||
repeated occurrences, mentions followed by presence, combat alignment
|
||||
changes, cross-chunk repetition, and empty output on at least one intended
|
||||
smaller model. Do not make model agreement a deterministic CI gate.
|
||||
6. Review links, examples, terminology, and file identities across the
|
||||
documentation. Do not add an ADR: this feature applies the existing fixed
|
||||
pipeline, output/cache separation, and framework ownership decisions rather
|
||||
than changing them.
|
||||
|
||||
### Validation
|
||||
|
||||
Run focused tests while implementing, then run:
|
||||
Run focused tests after each stage. At completion, run:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Also run `git diff --check`, inspect the complete documentation diff for
|
||||
current-versus-future claims, and verify every new relative documentation link.
|
||||
Also verify that maintained example tests pass, all new documentation links
|
||||
resolve, and a default JSON output run still has no chunk-map file or index
|
||||
descriptor.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- All repository checks pass offline.
|
||||
- The maintained example resolves with an explicit earlier NPC producer.
|
||||
- The integration test proves the accepted generated artifact, not registry
|
||||
source references, grounds the later model request.
|
||||
- Durable output contains only `name`, `kind`, and current-transcript
|
||||
`source_refs` per occurrence.
|
||||
- Current documentation owns implemented contracts, while the feature roadmap
|
||||
is marked implemented and `future.md` no longer advertises the work as
|
||||
pending.
|
||||
- The assembled scene-chunking run exports the exact accepted map and
|
||||
namespaced annotations without a second model call.
|
||||
- Current documentation and the maintained example describe only implemented
|
||||
behavior.
|
||||
- The future roadmap no longer lists the completed feature.
|
||||
- Repository-wide tests, vet, build, formatting, and documentation checks pass.
|
||||
|
||||
## Cross-Stage Guardrails
|
||||
|
||||
- Do not expose internal `source.Chunk` directly as the durable DTO.
|
||||
- Do not add transcript content, materialized units, source-unit metadata,
|
||||
model proposals, or debug data to the artifact.
|
||||
- Do not create an extractor, lane registration, normalizer, generated
|
||||
reference, import path, or replay mechanism for chunk maps.
|
||||
- Do not make D&D scene annotation fields part of the generic schema.
|
||||
- Do not make export the default or add a CLI-global/filesystem option.
|
||||
- Do not change chunk selection, validation, cache lookup/publication, or lane
|
||||
execution semantics.
|
||||
- Do not duplicate chunk-map payloads in the manifest or lane output indexes.
|
||||
- Preserve defensive ownership at source, runner, request, codec, and encoder
|
||||
boundaries.
|
||||
- Keep tests offline and deterministic, and test each invariant at its stable
|
||||
owning layer.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The feature roadmap and fixed decisions above define the product and
|
||||
architectural choices required for implementation.
|
||||
None. The feature roadmap and the fixed decisions above resolve the
|
||||
implementation-significant choices.
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "notarius.source.chunk_map",
|
||||
"title": "notarius_source_chunk_map_v1",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"source_id",
|
||||
"source_digest",
|
||||
"plan_digest",
|
||||
"requested_chunker",
|
||||
"producer",
|
||||
"plan_annotations",
|
||||
"chunks"
|
||||
],
|
||||
"properties": {
|
||||
"source_id": {"type": "string", "minLength": 1},
|
||||
"source_digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
||||
"plan_digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
||||
"requested_chunker": {"type": "string", "minLength": 1},
|
||||
"producer": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["input_module", "chunk_module"],
|
||||
"properties": {
|
||||
"input_module": {"type": "string", "minLength": 1},
|
||||
"chunk_module": {"type": "string", "minLength": 1},
|
||||
"llm_profile": {"type": "string", "minLength": 1}
|
||||
}
|
||||
},
|
||||
"plan_annotations": {"$ref": "#/$defs/annotations"},
|
||||
"chunks": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["id", "index", "source_ref", "unit_count", "annotations"],
|
||||
"properties": {
|
||||
"id": {"type": "string", "minLength": 1},
|
||||
"index": {"type": "integer", "minimum": 0},
|
||||
"source_ref": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["source_id", "start_unit_id", "end_unit_id"],
|
||||
"properties": {
|
||||
"source_id": {"type": "string", "minLength": 1},
|
||||
"start_unit_id": {"type": "integer", "minimum": 1},
|
||||
"end_unit_id": {"type": "integer", "minimum": 1}
|
||||
}
|
||||
},
|
||||
"unit_count": {"type": "integer", "minimum": 1},
|
||||
"annotations": {"$ref": "#/$defs/annotations"}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"annotations": {
|
||||
"type": "object",
|
||||
"propertyNames": {"type": "string", "minLength": 1},
|
||||
"additionalProperties": true
|
||||
}
|
||||
}
|
||||
}
|
||||
347
internal/framework/chunkmap/codec.go
Normal file
347
internal/framework/chunkmap/codec.go
Normal file
@@ -0,0 +1,347 @@
|
||||
package chunkmap
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"embed"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
//go:embed assets/schemas/source_chunk_map.v1.json
|
||||
var schemaAssets embed.FS
|
||||
|
||||
var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
|
||||
|
||||
// Codec owns strict serialization for the durable chunk-map contract.
|
||||
type Codec struct{}
|
||||
|
||||
func New() *Codec { return &Codec{} }
|
||||
|
||||
func (c *Codec) Kind() contracts.ArtifactKind { return ArtifactKind }
|
||||
|
||||
func (c *Codec) Schema() contracts.ArtifactSchema {
|
||||
raw, err := c.schemaBytes()
|
||||
if err != nil {
|
||||
return contracts.ArtifactSchema{}
|
||||
}
|
||||
return contracts.ArtifactSchema{
|
||||
ID: SchemaID,
|
||||
Name: SchemaName,
|
||||
Version: SchemaVersion,
|
||||
JSONSchema: raw,
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Codec) MediaType() string { return MediaType }
|
||||
|
||||
// Build proves that a durable value describes the exact accepted source plan
|
||||
// and materialized chunk list supplied by the framework.
|
||||
func Build(request BuildRequest) (ChunkMap, error) {
|
||||
if err := source.ValidateDocument(request.Source); err != nil {
|
||||
return ChunkMap{}, fmt.Errorf("validate source document: %w", err)
|
||||
}
|
||||
sourceDigest, err := source.DigestDocument(request.Source)
|
||||
if err != nil {
|
||||
return ChunkMap{}, fmt.Errorf("digest source document: %w", err)
|
||||
}
|
||||
if sourceDigest != request.Source.Digest {
|
||||
return ChunkMap{}, fmt.Errorf("source digest %q does not match source document digest %q", sourceDigest, request.Source.Digest)
|
||||
}
|
||||
if sourceDigest != request.Plan.SourceDigest {
|
||||
return ChunkMap{}, fmt.Errorf("source digest %q does not match chunk plan source digest %q", sourceDigest, request.Plan.SourceDigest)
|
||||
}
|
||||
plan, err := source.CanonicalizeChunkPlan(request.Plan)
|
||||
if err != nil {
|
||||
return ChunkMap{}, fmt.Errorf("canonicalize chunk plan: %w", err)
|
||||
}
|
||||
if err := source.ValidateChunkPlan(request.Source, plan); err != nil {
|
||||
return ChunkMap{}, fmt.Errorf("validate accepted chunk plan: %w", err)
|
||||
}
|
||||
planDigest, err := source.DigestChunkPlan(plan)
|
||||
if err != nil {
|
||||
return ChunkMap{}, fmt.Errorf("digest accepted chunk plan: %w", err)
|
||||
}
|
||||
expected, err := source.MaterializeChunkPlan(request.Source, plan)
|
||||
if err != nil {
|
||||
return ChunkMap{}, fmt.Errorf("materialize accepted chunk plan: %w", err)
|
||||
}
|
||||
if err := verifyMaterializedChunks(request.Chunks, expected); err != nil {
|
||||
return ChunkMap{}, err
|
||||
}
|
||||
|
||||
value := ChunkMap{
|
||||
SourceID: request.Source.ID,
|
||||
SourceDigest: sourceDigest,
|
||||
PlanDigest: planDigest,
|
||||
RequestedChunker: request.RequestedChunker,
|
||||
Producer: request.Producer,
|
||||
PlanAnnotations: source.CloneChunkAnnotations(plan.Annotations),
|
||||
Chunks: make([]Chunk, len(expected)),
|
||||
}
|
||||
for index, chunk := range expected {
|
||||
value.Chunks[index] = Chunk{
|
||||
ID: chunk.ID,
|
||||
Index: chunk.Index,
|
||||
SourceRef: chunk.Ref,
|
||||
UnitCount: len(chunk.Units),
|
||||
Annotations: source.CloneChunkAnnotations(chunk.Annotations),
|
||||
}
|
||||
}
|
||||
canonical, err := canonicalize(value)
|
||||
if err != nil {
|
||||
return ChunkMap{}, fmt.Errorf("validate chunk map: %w", err)
|
||||
}
|
||||
return clone(canonical), nil
|
||||
}
|
||||
|
||||
// Serialize builds and encodes the framework-owned serialized artifact.
|
||||
func Serialize(request BuildRequest) (contracts.SerializedArtifact, error) {
|
||||
value, err := Build(request)
|
||||
if err != nil {
|
||||
return contracts.SerializedArtifact{}, err
|
||||
}
|
||||
codec := New()
|
||||
content, err := codec.Encode(value)
|
||||
if err != nil {
|
||||
return contracts.SerializedArtifact{}, err
|
||||
}
|
||||
return contracts.SerializedArtifact{
|
||||
Kind: ArtifactKind,
|
||||
Schema: codec.Schema(),
|
||||
MediaType: MediaType,
|
||||
Content: content,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (c *Codec) Encode(value ChunkMap) ([]byte, error) {
|
||||
if _, err := c.schemaBytes(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
canonical, err := canonicalize(value)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("encode source chunk map: %w", err)
|
||||
}
|
||||
content, err := json.Marshal(canonical)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("encode source chunk map: %w", err)
|
||||
}
|
||||
return content, nil
|
||||
}
|
||||
|
||||
func (c *Codec) Decode(content []byte) (ChunkMap, error) {
|
||||
if _, err := c.schemaBytes(); err != nil {
|
||||
return ChunkMap{}, err
|
||||
}
|
||||
decoder := json.NewDecoder(bytes.NewReader(content))
|
||||
decoder.DisallowUnknownFields()
|
||||
var value ChunkMap
|
||||
if err := decoder.Decode(&value); err != nil {
|
||||
return ChunkMap{}, fmt.Errorf("decode source chunk map: %w", err)
|
||||
}
|
||||
var trailing any
|
||||
if err := decoder.Decode(&trailing); err != io.EOF {
|
||||
return ChunkMap{}, fmt.Errorf("decode source chunk map: multiple JSON values")
|
||||
}
|
||||
canonical, err := canonicalize(value)
|
||||
if err != nil {
|
||||
return ChunkMap{}, fmt.Errorf("decode source chunk map: %w", err)
|
||||
}
|
||||
return clone(canonical), nil
|
||||
}
|
||||
|
||||
func (c *Codec) schemaBytes() ([]byte, error) {
|
||||
raw, err := schemaAssets.ReadFile("assets/schemas/source_chunk_map.v1.json")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read source chunk map schema: %w", err)
|
||||
}
|
||||
var schema struct {
|
||||
ID string `json:"$id"`
|
||||
Title string `json:"title"`
|
||||
Type string `json:"type"`
|
||||
Required []string `json:"required"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &schema); err != nil {
|
||||
return nil, fmt.Errorf("decode source chunk map schema: %w", err)
|
||||
}
|
||||
if schema.ID != SchemaID || schema.Title != SchemaName || schema.Type != "object" || !hasRequiredFields(schema.Required) {
|
||||
return nil, fmt.Errorf("source chunk map schema identity or required fields are invalid")
|
||||
}
|
||||
return append([]byte(nil), raw...), nil
|
||||
}
|
||||
|
||||
func hasRequiredFields(required []string) bool {
|
||||
want := map[string]bool{
|
||||
"source_id": true, "source_digest": true, "plan_digest": true,
|
||||
"requested_chunker": true, "producer": true, "plan_annotations": true,
|
||||
"chunks": true,
|
||||
}
|
||||
for _, field := range required {
|
||||
delete(want, field)
|
||||
}
|
||||
return len(want) == 0
|
||||
}
|
||||
|
||||
func canonicalize(value ChunkMap) (ChunkMap, error) {
|
||||
if err := requireIdentity("source_id", value.SourceID); err != nil {
|
||||
return ChunkMap{}, err
|
||||
}
|
||||
if err := requireDigest("source_digest", value.SourceDigest); err != nil {
|
||||
return ChunkMap{}, err
|
||||
}
|
||||
if err := requireDigest("plan_digest", value.PlanDigest); err != nil {
|
||||
return ChunkMap{}, err
|
||||
}
|
||||
if err := requireIdentity("requested_chunker", value.RequestedChunker); err != nil {
|
||||
return ChunkMap{}, err
|
||||
}
|
||||
if err := requireIdentity("producer.input_module", value.Producer.InputModule); err != nil {
|
||||
return ChunkMap{}, err
|
||||
}
|
||||
if err := requireIdentity("producer.chunk_module", value.Producer.ChunkModule); err != nil {
|
||||
return ChunkMap{}, err
|
||||
}
|
||||
if value.Producer.LLMProfile != "" {
|
||||
if err := requireIdentity("producer.llm_profile", value.Producer.LLMProfile); err != nil {
|
||||
return ChunkMap{}, err
|
||||
}
|
||||
}
|
||||
annotations, err := canonicalizeAnnotations("plan_annotations", value.PlanAnnotations)
|
||||
if err != nil {
|
||||
return ChunkMap{}, err
|
||||
}
|
||||
value.PlanAnnotations = annotations
|
||||
if len(value.Chunks) == 0 {
|
||||
return ChunkMap{}, fmt.Errorf("chunks must not be empty")
|
||||
}
|
||||
seenIDs := make(map[string]struct{}, len(value.Chunks))
|
||||
plan := source.ChunkPlan{SourceDigest: value.SourceDigest, Annotations: annotations, Ranges: make([]source.ChunkRange, len(value.Chunks))}
|
||||
for index := range value.Chunks {
|
||||
chunk := &value.Chunks[index]
|
||||
if err := requireIdentity(fmt.Sprintf("chunks[%d].id", index), chunk.ID); err != nil {
|
||||
return ChunkMap{}, err
|
||||
}
|
||||
if _, exists := seenIDs[chunk.ID]; exists {
|
||||
return ChunkMap{}, fmt.Errorf("chunks[%d].id %q is duplicated", index, chunk.ID)
|
||||
}
|
||||
seenIDs[chunk.ID] = struct{}{}
|
||||
if chunk.Index != index {
|
||||
return ChunkMap{}, fmt.Errorf("chunks[%d].index = %d, want %d", index, chunk.Index, index)
|
||||
}
|
||||
if chunk.SourceRef.SourceID != value.SourceID {
|
||||
return ChunkMap{}, fmt.Errorf("chunks[%d].source_ref.source_id %q does not match source_id %q", index, chunk.SourceRef.SourceID, value.SourceID)
|
||||
}
|
||||
if chunk.SourceRef.StartUnitID <= 0 || chunk.SourceRef.EndUnitID <= 0 {
|
||||
return ChunkMap{}, fmt.Errorf("chunks[%d].source_ref endpoints must be positive", index)
|
||||
}
|
||||
if chunk.UnitCount <= 0 {
|
||||
return ChunkMap{}, fmt.Errorf("chunks[%d].unit_count must be positive", index)
|
||||
}
|
||||
chunkAnnotations, err := canonicalizeAnnotations(fmt.Sprintf("chunks[%d].annotations", index), chunk.Annotations)
|
||||
if err != nil {
|
||||
return ChunkMap{}, err
|
||||
}
|
||||
chunk.Annotations = chunkAnnotations
|
||||
plan.Ranges[index] = source.ChunkRange{
|
||||
StartUnitID: chunk.SourceRef.StartUnitID,
|
||||
EndUnitID: chunk.SourceRef.EndUnitID,
|
||||
Annotations: chunkAnnotations,
|
||||
}
|
||||
}
|
||||
planDigest, err := source.DigestChunkPlan(plan)
|
||||
if err != nil {
|
||||
return ChunkMap{}, fmt.Errorf("reconstruct plan digest: %w", err)
|
||||
}
|
||||
if planDigest != value.PlanDigest {
|
||||
return ChunkMap{}, fmt.Errorf("plan_digest %q does not match reconstructed plan digest %q", value.PlanDigest, planDigest)
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
|
||||
func canonicalizeAnnotations(name string, annotations source.ChunkAnnotations) (source.ChunkAnnotations, error) {
|
||||
for namespace := range annotations {
|
||||
if strings.TrimSpace(namespace) == "" || namespace != strings.TrimSpace(namespace) {
|
||||
return nil, fmt.Errorf("%s namespace %q must be non-empty and trimmed", name, namespace)
|
||||
}
|
||||
}
|
||||
canonical, err := source.CanonicalizeChunkAnnotations(annotations)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", name, err)
|
||||
}
|
||||
if canonical == nil {
|
||||
canonical = source.ChunkAnnotations{}
|
||||
}
|
||||
return canonical, nil
|
||||
}
|
||||
|
||||
func requireIdentity(name, value string) error {
|
||||
if strings.TrimSpace(value) == "" || value != strings.TrimSpace(value) {
|
||||
return fmt.Errorf("%s must be non-empty and trimmed", name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func requireDigest(name, value string) error {
|
||||
if !digestPattern.MatchString(value) {
|
||||
return fmt.Errorf("%s must be a canonical sha256 digest", name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func verifyMaterializedChunks(actual, expected []source.Chunk) error {
|
||||
if len(actual) != len(expected) {
|
||||
return fmt.Errorf("materialized chunks length = %d, want %d", len(actual), len(expected))
|
||||
}
|
||||
for index := range expected {
|
||||
got, want := actual[index], expected[index]
|
||||
if got.ID != want.ID || got.SourceID != want.SourceID || got.Index != want.Index || got.Ref != want.Ref {
|
||||
return fmt.Errorf("materialized chunk[%d] identity or source range differs from accepted plan", index)
|
||||
}
|
||||
if len(got.Units) != len(want.Units) || !sameUnits(got.Units, want.Units) {
|
||||
return fmt.Errorf("materialized chunk[%d] units differ from accepted source range", index)
|
||||
}
|
||||
if !sameAnnotations(got.PlanAnnotations, want.PlanAnnotations) || !sameAnnotations(got.Annotations, want.Annotations) {
|
||||
return fmt.Errorf("materialized chunk[%d] annotations differ from accepted plan", index)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func sameUnits(left, right []source.SourceUnit) bool {
|
||||
leftJSON, leftErr := json.Marshal(left)
|
||||
rightJSON, rightErr := json.Marshal(right)
|
||||
return leftErr == nil && rightErr == nil && bytes.Equal(leftJSON, rightJSON)
|
||||
}
|
||||
|
||||
func sameAnnotations(left, right source.ChunkAnnotations) bool {
|
||||
leftCanonical, leftErr := source.CanonicalizeChunkAnnotations(left)
|
||||
rightCanonical, rightErr := source.CanonicalizeChunkAnnotations(right)
|
||||
if leftErr != nil || rightErr != nil {
|
||||
return false
|
||||
}
|
||||
leftJSON, leftErr := json.Marshal(leftCanonical)
|
||||
rightJSON, rightErr := json.Marshal(rightCanonical)
|
||||
return leftErr == nil && rightErr == nil && bytes.Equal(leftJSON, rightJSON)
|
||||
}
|
||||
|
||||
func clone(value ChunkMap) ChunkMap {
|
||||
value.PlanAnnotations = cloneAnnotations(value.PlanAnnotations)
|
||||
value.Chunks = append([]Chunk(nil), value.Chunks...)
|
||||
for index := range value.Chunks {
|
||||
value.Chunks[index].Annotations = cloneAnnotations(value.Chunks[index].Annotations)
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func cloneAnnotations(annotations source.ChunkAnnotations) source.ChunkAnnotations {
|
||||
cloned := source.CloneChunkAnnotations(annotations)
|
||||
if cloned == nil {
|
||||
return source.ChunkAnnotations{}
|
||||
}
|
||||
return cloned
|
||||
}
|
||||
191
internal/framework/chunkmap/codec_test.go
Normal file
191
internal/framework/chunkmap/codec_test.go
Normal file
@@ -0,0 +1,191 @@
|
||||
package chunkmap
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
)
|
||||
|
||||
func TestBuildAndSerializeAcceptedChunkMap(t *testing.T) {
|
||||
request := acceptedBuildRequest(t)
|
||||
value, err := Build(request)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
if value.SourceID != request.Source.ID || len(value.Chunks) != 2 || value.Chunks[0].UnitCount != 2 || value.Chunks[1].SourceRef.StartUnitID != 20 {
|
||||
t.Fatalf("Build() = %#v, want exact accepted chunk structure", value)
|
||||
}
|
||||
if value.PlanAnnotations == nil || value.Chunks[1].Annotations == nil {
|
||||
t.Fatalf("Build() annotations = %#v, want explicit maps", value)
|
||||
}
|
||||
artifact, err := Serialize(request)
|
||||
if err != nil {
|
||||
t.Fatalf("Serialize() error = %v", err)
|
||||
}
|
||||
if artifact.Kind != ArtifactKind || artifact.Schema.ID != SchemaID || artifact.Schema.Name != SchemaName || artifact.Schema.Version != SchemaVersion || artifact.MediaType != MediaType || artifact.Metadata != nil {
|
||||
t.Fatalf("Serialize() = %#v, want fixed artifact envelope without metadata", artifact)
|
||||
}
|
||||
decoded, err := New().Decode(artifact.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("Decode(Serialize()) error = %v", err)
|
||||
}
|
||||
if decoded.PlanDigest != value.PlanDigest || decoded.Chunks[0].ID != "chunk-000001" || decoded.Chunks[1].UnitCount != 1 {
|
||||
t.Fatalf("Decode(Serialize()) = %#v, want durable chunk map", decoded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodecRoundTripsValidFixture(t *testing.T) {
|
||||
fixture, err := os.ReadFile("testdata/source_chunk_map.v1.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
codec := New()
|
||||
value, err := codec.Decode(fixture)
|
||||
if err != nil {
|
||||
t.Fatalf("Decode(fixture) error = %v", err)
|
||||
}
|
||||
encoded, err := codec.Encode(value)
|
||||
if err != nil {
|
||||
t.Fatalf("Encode(decoded fixture) error = %v", err)
|
||||
}
|
||||
if !bytes.Equal(encoded, bytes.TrimSpace(fixture)) {
|
||||
t.Fatalf("fixture does not use canonical encoding\nwant: %s\n got: %s", fixture, encoded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildCanonicalizesAnnotationFormatting(t *testing.T) {
|
||||
first := acceptedBuildRequest(t)
|
||||
second := acceptedBuildRequest(t)
|
||||
second.Plan.Annotations["dnd/scenes"] = json.RawMessage(" { \n \t\"title\" : \"Gate\" \n } ")
|
||||
canonical, err := source.CanonicalizeChunkPlan(second.Plan)
|
||||
if err != nil {
|
||||
t.Fatalf("CanonicalizeChunkPlan() error = %v", err)
|
||||
}
|
||||
second.Chunks, err = source.MaterializeChunkPlan(second.Source, canonical)
|
||||
if err != nil {
|
||||
t.Fatalf("MaterializeChunkPlan() error = %v", err)
|
||||
}
|
||||
firstArtifact, err := Serialize(first)
|
||||
if err != nil {
|
||||
t.Fatalf("Serialize(first) error = %v", err)
|
||||
}
|
||||
secondArtifact, err := Serialize(second)
|
||||
if err != nil {
|
||||
t.Fatalf("Serialize(second) error = %v", err)
|
||||
}
|
||||
if !bytes.Equal(firstArtifact.Content, secondArtifact.Content) {
|
||||
t.Fatalf("serialized content differs only because annotation whitespace changed\nfirst: %s\nsecond: %s", firstArtifact.Content, secondArtifact.Content)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildRejectsChunksOutsideAcceptedPlan(t *testing.T) {
|
||||
request := acceptedBuildRequest(t)
|
||||
request.Chunks[0].Units[0].ID = 999
|
||||
if _, err := Build(request); err == nil {
|
||||
t.Fatal("Build() error = nil, want rejection for chunk units outside accepted source range")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodecRejectsInvalidDurableBoundaries(t *testing.T) {
|
||||
value, err := Build(acceptedBuildRequest(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
mutate func(*ChunkMap)
|
||||
}{
|
||||
{name: "blank identity", mutate: func(value *ChunkMap) { value.RequestedChunker = " " }},
|
||||
{name: "malformed digest", mutate: func(value *ChunkMap) { value.SourceDigest = "sha256:ABC" }},
|
||||
{name: "index mismatch", mutate: func(value *ChunkMap) { value.Chunks[1].Index = 4 }},
|
||||
{name: "duplicate chunk id", mutate: func(value *ChunkMap) { value.Chunks[1].ID = value.Chunks[0].ID }},
|
||||
{name: "source mismatch", mutate: func(value *ChunkMap) { value.Chunks[0].SourceRef.SourceID = "other" }},
|
||||
{name: "invalid range", mutate: func(value *ChunkMap) { value.Chunks[0].SourceRef.StartUnitID = 0 }},
|
||||
{name: "invalid count", mutate: func(value *ChunkMap) { value.Chunks[0].UnitCount = 0 }},
|
||||
{name: "invalid namespace", mutate: func(value *ChunkMap) { value.PlanAnnotations[" "] = json.RawMessage(`null`) }},
|
||||
{name: "invalid annotation", mutate: func(value *ChunkMap) { value.Chunks[0].Annotations["dnd/scenes"] = json.RawMessage(`{`) }},
|
||||
{name: "plan digest mismatch", mutate: func(value *ChunkMap) { value.PlanDigest = "sha256:" + strings.Repeat("a", 64) }},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
candidate := clone(value)
|
||||
test.mutate(&candidate)
|
||||
if _, err := New().Encode(candidate); err == nil {
|
||||
t.Fatal("Encode() error = nil, want invalid durable value rejection")
|
||||
}
|
||||
})
|
||||
}
|
||||
content, err := New().Encode(value)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, raw := range [][]byte{
|
||||
append(append([]byte(nil), content[:len(content)-1]...), []byte(`,"unknown":true}`)...),
|
||||
append(append([]byte(nil), content...), []byte(` {}`)...),
|
||||
} {
|
||||
if _, err := New().Decode(raw); err == nil {
|
||||
t.Fatalf("Decode(%s) error = nil, want strict JSON rejection", raw)
|
||||
}
|
||||
}
|
||||
formatted := bytes.Replace(content, []byte(`{"title":"Gate"}`), []byte("{\n \"title\": \"Gate\"\n}"), 1)
|
||||
decoded, err := New().Decode(formatted)
|
||||
if err != nil {
|
||||
t.Fatalf("Decode(formatted annotations) error = %v", err)
|
||||
}
|
||||
if string(decoded.PlanAnnotations["dnd/scenes"]) != `{"title":"Gate"}` {
|
||||
t.Fatalf("decoded annotation = %s, want canonical JSON", decoded.PlanAnnotations["dnd/scenes"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestChunkMapOwnershipIsIndependent(t *testing.T) {
|
||||
request := acceptedBuildRequest(t)
|
||||
first, err := Build(request)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
request.Plan.Annotations["dnd/scenes"][0] = '['
|
||||
first.PlanAnnotations["dnd/scenes"][0] = '['
|
||||
second, err := Build(acceptedBuildRequest(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(second.PlanAnnotations["dnd/scenes"]) != `{"title":"Gate"}` {
|
||||
t.Fatalf("Build() shared mutable annotations: %s", second.PlanAnnotations["dnd/scenes"])
|
||||
}
|
||||
}
|
||||
|
||||
func acceptedBuildRequest(t *testing.T) BuildRequest {
|
||||
t.Helper()
|
||||
document := &source.SourceDocument{
|
||||
ID: "session-7", Kind: "transcript", Format: "application/json",
|
||||
Units: []source.SourceUnit{
|
||||
{ID: 10, Kind: "segment", Text: "At the gate.", Ref: source.SourceRef{SourceID: "session-7", StartUnitID: 10, EndUnitID: 10}},
|
||||
{ID: 3, Kind: "segment", Text: "The guard speaks.", Ref: source.SourceRef{SourceID: "session-7", StartUnitID: 3, EndUnitID: 3}},
|
||||
{ID: 20, Kind: "segment", Text: "The party enters.", Ref: source.SourceRef{SourceID: "session-7", StartUnitID: 20, EndUnitID: 20}},
|
||||
},
|
||||
}
|
||||
digest, err := source.DigestDocument(document)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
document.Digest = digest
|
||||
plan := source.ChunkPlan{
|
||||
SourceDigest: digest,
|
||||
Annotations: source.ChunkAnnotations{"dnd/scenes": json.RawMessage(`{"title":"Gate"}`)},
|
||||
Ranges: []source.ChunkRange{
|
||||
{StartUnitID: 10, EndUnitID: 3, Annotations: source.ChunkAnnotations{"dnd/scenes": json.RawMessage(`{"kind":"narrative"}`)}},
|
||||
{StartUnitID: 20, EndUnitID: 20},
|
||||
},
|
||||
}
|
||||
chunks, err := source.MaterializeChunkPlan(document, plan)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return BuildRequest{
|
||||
Source: document, Plan: plan, Chunks: chunks, RequestedChunker: "dnd/scenes",
|
||||
Producer: Producer{InputModule: "seriatim", ChunkModule: "dnd/scenes", LLMProfile: "dnd-scenes"},
|
||||
}
|
||||
}
|
||||
51
internal/framework/chunkmap/model.go
Normal file
51
internal/framework/chunkmap/model.go
Normal file
@@ -0,0 +1,51 @@
|
||||
// Package chunkmap owns the durable accepted source chunk-map contract.
|
||||
package chunkmap
|
||||
|
||||
import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
const (
|
||||
ArtifactKind contracts.ArtifactKind = "source/chunk-map"
|
||||
SchemaID = "notarius.source.chunk_map"
|
||||
SchemaName = "notarius_source_chunk_map_v1"
|
||||
SchemaVersion = "v1"
|
||||
MediaType = "application/json"
|
||||
)
|
||||
|
||||
// ChunkMap is the durable representation of one accepted materialized chunk plan.
|
||||
type ChunkMap struct {
|
||||
SourceID string `json:"source_id"`
|
||||
SourceDigest string `json:"source_digest"`
|
||||
PlanDigest string `json:"plan_digest"`
|
||||
RequestedChunker string `json:"requested_chunker"`
|
||||
Producer Producer `json:"producer"`
|
||||
PlanAnnotations source.ChunkAnnotations `json:"plan_annotations"`
|
||||
Chunks []Chunk `json:"chunks"`
|
||||
}
|
||||
|
||||
// Producer identifies the component that produced the accepted logical plan.
|
||||
type Producer struct {
|
||||
InputModule string `json:"input_module"`
|
||||
ChunkModule string `json:"chunk_module"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
}
|
||||
|
||||
// Chunk describes one accepted materialized range without source content.
|
||||
type Chunk struct {
|
||||
ID string `json:"id"`
|
||||
Index int `json:"index"`
|
||||
SourceRef source.SourceRef `json:"source_ref"`
|
||||
UnitCount int `json:"unit_count"`
|
||||
Annotations source.ChunkAnnotations `json:"annotations"`
|
||||
}
|
||||
|
||||
// BuildRequest supplies the accepted runtime state used to build a chunk map.
|
||||
type BuildRequest struct {
|
||||
Source *source.SourceDocument
|
||||
Plan source.ChunkPlan
|
||||
Chunks []source.Chunk
|
||||
RequestedChunker string
|
||||
Producer Producer
|
||||
}
|
||||
1
internal/framework/chunkmap/testdata/source_chunk_map.v1.json
vendored
Normal file
1
internal/framework/chunkmap/testdata/source_chunk_map.v1.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{"source_id":"session-7","source_digest":"sha256:87d04d40537217e2adcbdec841c5873dbc1034c426d83c4578a0431a5ef855f6","plan_digest":"sha256:a6bfc33d52f1c0f4eb3287dd4a00e8672b3d3d9579f9c0366f18a5ff0dba7d14","requested_chunker":"dnd/scenes","producer":{"input_module":"seriatim","chunk_module":"dnd/scenes","llm_profile":"dnd-scenes"},"plan_annotations":{"dnd/scenes":{"title":"Gate"}},"chunks":[{"id":"chunk-000001","index":0,"source_ref":{"source_id":"session-7","start_unit_id":10,"end_unit_id":3},"unit_count":2,"annotations":{"dnd/scenes":{"kind":"narrative"}}},{"id":"chunk-000002","index":1,"source_ref":{"source_id":"session-7","start_unit_id":20,"end_unit_id":20},"unit_count":1,"annotations":{}}]}
|
||||
Reference in New Issue
Block a user