notarius/docs/roadmap/implementation.md

Accepted Chunk Map Export Implementation Plan

Status: Completed

Objective

Implement the accepted Accepted Chunk Map Export roadmap as an opt-in logical file in the production JSON output bundle. The finished feature must serialize the exact accepted materialized chunks used by the run, preserve framework ownership of chunk semantics, and keep file selection and placement in the output encoder.

This document is the implementation authority for sequencing and concrete engineering decisions. The feature roadmap remains authoritative for product intent, durable field semantics, security policy, and non-goals. Implement the stages in order and do not release or merge a partially completed sequence.

Fixed Decisions

Use these durable identities:

Concern	Identity
Artifact kind	source/chunk-map
Logical file	chunk-map.json
Schema ID	notarius.source.chunk_map
Schema name	notarius_source_chunk_map_v1
Schema version	v1
Media type	application/json
JSON output option	include_chunk_map

Keep the schema at v1; Notarius is pre-release and no compatibility layer, dual writer, or migration path is required.

The framework package owns the durable DTO, embedded JSON Schema, construction from accepted source state, invariant validation, cloning, and canonical JSON serialization. The runner owns construction and passes a serialized artifact through the generic output request. The JSON output encoder owns the opt-in decision, logical filename, formatting, and index.json descriptor.

Do not register the chunk map with the artifact-lane codec registry. That registry describes lane artifacts; the chunk map is a pipeline-wide source artifact with no extractor, merger, or normalizer.

The runner constructs the serialized value whenever an accepted chunk map exists, independent of the selected output encoder and its options. Output encoders may ignore the optional value. This keeps output-specific configuration out of the runner and leaves future encoders free to define their own explicit behavior.

Use canonical sha256: digests throughout. Annotation values must be semantically canonicalized before plan-digest verification; decoding must accept insignificant JSON whitespace rather than require annotation json.RawMessage bytes to already use one formatting style.

Stage 1: Add The Framework-Owned Durable Contract

Goal

Create a source-domain chunk-map contract that can be built, validated, encoded, and decoded without depending on an output module or D&D package.

Work

1. Add internal/framework/chunkmap/ with:

   • model.go for the durable DTO and identity constants;
   • codec.go for construction, validation, canonical encoding, and strict decoding;
   • assets/schemas/source_chunk_map.v1.json as the embedded external schema;
   • testdata/source_chunk_map.v1.json as one valid contract fixture; and
   • focused package tests.

2. Define constants for the fixed artifact kind, schema identity, schema version, and media type. Define DTOs matching the feature roadmap exactly:

   • ChunkMap with source_id, source_digest, plan_digest, requested_chunker, producer, plan_annotations, and chunks;
   • Producer with input_module, chunk_module, and optional llm_profile; and
   • Chunk with id, index, source_ref, unit_count, and annotations.

   Do not expose source units, chunk content, source metadata, producer references or metadata, warnings, timestamps, cache actions, or model responses.

3. Provide a small package API:

   type BuildRequest struct {
       Source             *source.SourceDocument
       Plan               source.ChunkPlan
       Chunks             []source.Chunk
       RequestedChunker   string
       Producer           Producer
   }
   
   func Build(BuildRequest) (ChunkMap, error)
   func Serialize(BuildRequest) (contracts.SerializedArtifact, error)
   func New() *Codec
   

   Codec must expose the same identity and encode/decode operations used by other durable codecs, but it is invoked directly by this package and the JSON encoder rather than added to pipeline.ArtifactCodecRegistry.

4. Make Build the trusted boundary between runtime source structures and the external contract:

   • validate the source document and accepted plan with existing source validators;
   • calculate the source and plan digests with source.DigestDocument and source.DigestChunkPlan, and require the calculated source digest to equal both Source.Digest and Plan.SourceDigest;
   • require a non-empty requested chunker and producer module identities;
   • verify the supplied materialized chunks correspond exactly, in order, to the accepted plan and source document, including IDs, indexes, source ranges, unit membership/counts, plan annotations, and range annotations;
   • validate range order against source-document position rather than numeric unit-ID assumptions;
   • deep-clone all references and annotation values; and
   • encode absent annotation namespaces as non-nil empty maps so JSON contains {}.

5. Make codec decoding strict at fixed object levels:

   • accept one JSON value only and reject trailing content;
   • reject unknown fields;
   • validate trimmed, non-empty identities;
   • require canonical lower-case sha256: digests;
   • require a non-empty chunk array;
   • require unique stable chunk IDs and contiguous zero-based indexes matching array order;
   • require every source reference to use the top-level source identity and positive range endpoints;
   • require positive unit counts;
   • validate non-empty canonical annotation namespaces and valid JSON values;
   • canonicalize annotation JSON before using the existing plan-digest helper to reconstruct and verify plan_digest; and
   • return defensively owned DTO data.

   Standalone decoding cannot prove range order against a source document it does not contain. Construction owns that stronger invariant; decoding owns the structural, identity, and digest invariants available from the durable payload.

6. Make serialization return a contracts.SerializedArtifact with the fixed kind, schema, and media type, canonical content, and no duplicated metadata. Validate the embedded schema defensively so the checked-in schema and Go contract cannot silently diverge.

7. Define a strict JSON Schema with:

   • additionalProperties: false at every fixed object level;
   • all roadmap-required fields;
   • minItems: 1 for chunks;
   • positive integer range endpoints and unit counts;
   • zero-or-greater indexes;
   • the pattern ^sha256:[0-9a-f]{64}$ for digests;
   • non-empty strings for identities;
   • optional, non-empty llm_profile; and
   • annotation objects whose namespace values may be arbitrary JSON.

Tests

Add focused behavioral tests proving:

• the valid fixture round-trips through the codec;
• building from a real accepted source, plan, and materialized chunk list produces the fixed envelope and exact durable fields;
• output is deterministic and annotation JSON formatting does not alter the reconstructed plan digest;
• unknown fields, trailing JSON, malformed identities/digests, invalid indexes, duplicate IDs, mismatched source identities, invalid ranges or counts, malformed annotation namespaces/values, and plan-digest mismatches are rejected;
• construction rejects materialized chunks that differ from the accepted plan or document order; and
• mutating caller-owned inputs or returned values cannot mutate another owned representation.

Keep these tests semantic. Do not add exact byte-length, exact schema-length, private-helper, or redundant field-by-field “change detector” tests.

Completion Criteria

• The framework contract can represent and validate the roadmap payload without importing output or D&D packages.
• Construction proves the artifact describes the exact accepted materialized chunks.
• No production output behavior has changed yet.

Stage 2: Carry The Accepted Artifact Through The Runner

Goal

Construct the durable artifact at the accepted chunk-plan boundary and make it available to output encoders without changing default logical files.

Work

1. Extend contracts.OutputRequest with:

   ChunkMap *contracts.SerializedArtifact `json:"chunk_map,omitempty"`
   

   Update its cloning/ownership path so the envelope, content bytes, and metadata are defensively copied. Do not place chunk-map content in the run manifest or generic request metadata.

2. In internal/framework/pipeline/runner.go, construct the serialized chunk map after materialization and configured chunk validation accept the plan, and before the output encoder is invoked. Build it from:

   • the accepted source document;
   • the accepted logical plan;
   • the exact materialized chunks passed to lanes;
   • the chunk module selected by the current resolved pipeline as requested_chunker; and
   • the original producer identities recorded by the accepted chunk-plan record.

3. Preserve cache semantics:

   • a generated or bypassed plan records the current producer;
   • a cache hit keeps the stored producer's input module, chunk module, and optional LLM profile;
   • requested_chunker still records the current resolved chunk module; and
   • producer references, metadata, warnings, and cache details remain only in existing provenance surfaces.

4. Apply the accepted-state policy:

   • pass no chunk map when chunk validation rejects the candidate;
   • retain the artifact when a later extraction, merge, or normalization lane is rejected;
   • treat artifact construction or serialization failure after acceptance as a framework run error; and
   • do not rerun chunking or make another LLM request.

5. If debug serialization records the output request, represent the new value through the existing safe serialized-artifact envelope conventions. Do not duplicate its content into the manifest or add a new default debug surface.

Tests

Extend internal/framework/pipeline/runner_chunk_plan_test.go and the nearest existing output-request ownership tests to prove:

• accepted generated and cached plans reach a capturing output encoder as a valid serialized chunk map;
• cache reuse distinguishes current requested_chunker from the stored producer chunker;
• chunk rejection yields a nil chunk map;
• a later lane rejection still leaves the accepted chunk map available;
• captured requests do not alias runner, store, or caller-owned bytes/maps.

Use existing fake modules and stores. Do not add an LLM-backed integration test for behavior already deterministically owned by the runner, and do not add a production injection seam solely to force an otherwise unreachable construction failure; Stage 1 owns invalid-construction coverage.

Completion Criteria

• Every accepted chunk plan yields one validated serialized chunk map in the output request.
• Rejected chunk plans yield none, while later lane rejection does not discard it.
• Existing output encoders remain behaviorally unchanged.

Stage 3: Add Opt-In JSON Bundle Export

Goal

Expose the available framework artifact as chunk-map.json only when the JSON output binding explicitly enables it.

Work

1. Add an Options field:

   IncludeChunkMap bool
   

   Keep New() as the default-disabled constructor and add NewWithOptions(Options) for configured construction and focused tests.

2. Update the JSON output builder to decode only include_chunk_map:

   • omit or false means disabled;
   • require a boolean when present; and
   • continue rejecting unknown keys.

3. When disabled, ignore an available OutputRequest.ChunkMap and preserve the exact existing bundle shape. When enabled:

   • omit both file and descriptor if the request has no accepted chunk map;
   • validate the artifact kind, schema identity/version, and media type;
   • decode through chunkmap.New() so malformed or non-canonical payloads fail the run;
   • write valid content as pretty-printed JSON with the encoder's established trailing-newline convention; and
   • use the existing logical-path validation.

   Reuse or generalize the current serialized-artifact JSON helper rather than adding a second subtly different validation and formatting path.

4. Extend the output index model with an optional chunk_map descriptor containing exactly:

   • artifact_kind;
   • file;
   • media_type;
   • schema_id;
   • schema_name; and
   • schema_version.

   Do not add the chunk map to lane-oriented output_files, and do not change the manifest.

5. Treat a present but invalid accepted artifact as an output error. Do not silently omit it when export was explicitly requested.

Tests

Extend internal/modules/generic/output/json/encoder_test.go and existing output registration tests to prove:

• option omission and false retain the current default bundle even when an artifact is available;
• true plus a valid artifact emits chunk-map.json and the exact optional index descriptor;
• true plus no accepted artifact emits neither file nor descriptor;
• the chunk map never appears in lane output_files;
• wrong option types and unknown options are rejected;
• wrong kind, schema, media type, or invalid content returns an error;
• emitted JSON is valid, formatted consistently, and newline-terminated; and
• encoding does not mutate the request or serialized content.

Prefer testing the observable logical file map and decoded index rather than private helper calls or exact byte lengths.

Completion Criteria

• Existing JSON configurations remain unchanged by default.
• An explicitly enabled JSON output produces exactly one validated chunk-map.json file and one pipeline-wide index descriptor when an accepted map exists.
• No lane identity or manifest structure is fabricated for the chunk map.

Stage 4: Prove The Assembled Behavior And Document It

Goal

Demonstrate the feature with the production D&D scene chunker, publish all current-behavior documentation, and close the roadmap scope.

Work

1. Extend the representative assembled production contract in internal/cli/production_contract_test.go:

   • configure dnd/scenes with JSON include_chunk_map: true;
   • execute through normal production assembly with the existing deterministic fake LLM;
   • read index.json and chunk-map.json;
   • decode the artifact through the framework codec; and
   • assert the actual accepted scene ranges, stable IDs/order, source and plan identity, requested/current producer provenance, and surviving namespaced scene annotations.

   This is the single assembled proof. Do not duplicate all codec and encoder edge cases at the CLI layer.

2. Add a copyable maintained example such as examples/dnd-scene-chunk-map.config.yml that uses dnd/scenes, one normal D&D artifact lane, and the opt-in JSON output option. Add it to internal/cli/example_contract_test.go so schema/config drift is caught by the existing example contract.

3. Add docs/integrations/chunk-map.md as the canonical durable contract reference. Document fixed identities, every payload field, invariants, accepted-state behavior, cache provenance semantics, sensitivity, and excluded content. Link to the embedded schema without duplicating it in the prose.

4. Update current-behavior documentation:

   • docs/integrations/json-output.md: optional logical file and index descriptor;
   • docs/config.md: strict include_chunk_map option and copyable binding;
   • docs/operations.md: opt-in sensitivity and retention implications;
   • docs/internal/pipeline.md: accepted-plan construction and output-request handoff;
   • docs/internal/state.md: logical output and manifest/index ownership;
   • docs/internal/modules.md: JSON encoder option and responsibility; and
   • docs/internal/overview.md: framework chunk-map contract and encoder boundary.

   Keep implementation details in internal docs, user-selectable behavior in configuration docs, and the durable schema only in integrations docs.

5. Close planning state:

   • mark accepted-chunk-map-export.md as implemented;
   • remove the completed “Export Accepted Chunk Maps” entry from docs/roadmap/future.md; and
   • mark this implementation plan completed, preserving it until the user chooses to retire completed roadmap documents.

6. Review links, examples, terminology, and file identities across the documentation. Do not add an ADR: this feature applies the existing fixed pipeline, output/cache separation, and framework ownership decisions rather than changing them.

Validation

Run focused tests after each stage. At completion, run:

go test ./...
go vet ./...
go build ./cmd/notarius
git diff --check

Also verify that maintained example tests pass, all new documentation links resolve, and a default JSON output run still has no chunk-map file or index descriptor.

Completion Criteria

• The assembled scene-chunking run exports the exact accepted map and namespaced annotations without a second model call.
• Current documentation and the maintained example describe only implemented behavior.
• The future roadmap no longer lists the completed feature.
• Repository-wide tests, vet, build, formatting, and documentation checks pass.

Cross-Stage Guardrails

• Do not expose internal source.Chunk directly as the durable DTO.
• Do not add transcript content, materialized units, source-unit metadata, model proposals, or debug data to the artifact.
• Do not create an extractor, lane registration, normalizer, generated reference, import path, or replay mechanism for chunk maps.
• Do not make D&D scene annotation fields part of the generic schema.
• Do not make export the default or add a CLI-global/filesystem option.
• Do not change chunk selection, validation, cache lookup/publication, or lane execution semantics.
• Do not duplicate chunk-map payloads in the manifest or lane output indexes.
• Preserve defensive ownership at source, runner, request, codec, and encoder boundaries.
• Keep tests offline and deterministic, and test each invariant at its stable owning layer.

Open Questions

None. The feature roadmap and the fixed decisions above resolve the implementation-significant choices.