notarius/docs/roadmap/implementation.md

# Accepted Chunk Map Export Implementation Plan

Status: Completed

## Objective

Implement the accepted [Accepted Chunk Map Export](accepted-chunk-map-export.md)
roadmap as an opt-in logical file in the production JSON output bundle. The
finished feature must serialize the exact accepted materialized chunks used by
the run, preserve framework ownership of chunk semantics, and keep file
selection and placement in the output encoder.

This document is the implementation authority for sequencing and concrete
engineering decisions. The feature roadmap remains authoritative for product
intent, durable field semantics, security policy, and non-goals. Implement the
stages in order and do not release or merge a partially completed sequence.

## Fixed Decisions

Use these durable identities:

| Concern | Identity |
| --- | --- |
| Artifact kind | `source/chunk-map` |
| Logical file | `chunk-map.json` |
| Schema ID | `notarius.source.chunk_map` |
| Schema name | `notarius_source_chunk_map_v1` |
| Schema version | `v1` |
| Media type | `application/json` |
| JSON output option | `include_chunk_map` |

Keep the schema at `v1`; Notarius is pre-release and no compatibility layer,
dual writer, or migration path is required.

The framework package owns the durable DTO, embedded JSON Schema, construction
from accepted source state, invariant validation, cloning, and canonical JSON
serialization. The runner owns construction and passes a serialized artifact
through the generic output request. The JSON output encoder owns the opt-in
decision, logical filename, formatting, and `index.json` descriptor.

Do not register the chunk map with the artifact-lane codec registry. That
registry describes lane artifacts; the chunk map is a pipeline-wide source
artifact with no extractor, merger, or normalizer.

The runner constructs the serialized value whenever an accepted chunk map
exists, independent of the selected output encoder and its options. Output
encoders may ignore the optional value. This keeps output-specific
configuration out of the runner and leaves future encoders free to define
their own explicit behavior.

Use canonical `sha256:` digests throughout. Annotation values must be
semantically canonicalized before plan-digest verification; decoding must
accept insignificant JSON whitespace rather than require annotation
`json.RawMessage` bytes to already use one formatting style.

## Stage 1: Add The Framework-Owned Durable Contract

### Goal

Create a source-domain chunk-map contract that can be built, validated,
encoded, and decoded without depending on an output module or D&D package.

### Work

1. Add `internal/framework/chunkmap/` with:

   - `model.go` for the durable DTO and identity constants;
   - `codec.go` for construction, validation, canonical encoding, and strict
     decoding;
   - `assets/schemas/source_chunk_map.v1.json` as the embedded external schema;
   - `testdata/source_chunk_map.v1.json` as one valid contract fixture; and
   - focused package tests.

2. Define constants for the fixed artifact kind, schema identity, schema
   version, and media type. Define DTOs matching the feature roadmap exactly:

   - `ChunkMap` with `source_id`, `source_digest`, `plan_digest`,
     `requested_chunker`, `producer`, `plan_annotations`, and `chunks`;
   - `Producer` with `input_module`, `chunk_module`, and optional
     `llm_profile`; and
   - `Chunk` with `id`, `index`, `source_ref`, `unit_count`, and
     `annotations`.

   Do not expose source units, chunk content, source metadata, producer
   references or metadata, warnings, timestamps, cache actions, or model
   responses.

3. Provide a small package API:

   ```go
   type BuildRequest struct {
       Source             *source.SourceDocument
       Plan               source.ChunkPlan
       Chunks             []source.Chunk
       RequestedChunker   string
       Producer           Producer
   }

   func Build(BuildRequest) (ChunkMap, error)
   func Serialize(BuildRequest) (contracts.SerializedArtifact, error)
   func New() *Codec
   ```

   `Codec` must expose the same identity and encode/decode operations used by
   other durable codecs, but it is invoked directly by this package and the
   JSON encoder rather than added to `pipeline.ArtifactCodecRegistry`.

4. Make `Build` the trusted boundary between runtime source structures and the
   external contract:

   - validate the source document and accepted plan with existing source
     validators;
   - calculate the source and plan digests with `source.DigestDocument` and
     `source.DigestChunkPlan`, and require the calculated source digest to
     equal both `Source.Digest` and `Plan.SourceDigest`;
   - require a non-empty requested chunker and producer module identities;
   - verify the supplied materialized chunks correspond exactly, in order, to
     the accepted plan and source document, including IDs, indexes, source
     ranges, unit membership/counts, plan annotations, and range annotations;
   - validate range order against source-document position rather than numeric
     unit-ID assumptions;
   - deep-clone all references and annotation values; and
   - encode absent annotation namespaces as non-nil empty maps so JSON contains
     `{}`.

5. Make codec decoding strict at fixed object levels:

   - accept one JSON value only and reject trailing content;
   - reject unknown fields;
   - validate trimmed, non-empty identities;
   - require canonical lower-case `sha256:` digests;
   - require a non-empty chunk array;
   - require unique stable chunk IDs and contiguous zero-based indexes matching
     array order;
   - require every source reference to use the top-level source identity and
     positive range endpoints;
   - require positive unit counts;
   - validate non-empty canonical annotation namespaces and valid JSON values;
   - canonicalize annotation JSON before using the existing plan-digest helper
     to reconstruct and verify `plan_digest`; and
   - return defensively owned DTO data.

   Standalone decoding cannot prove range order against a source document it
   does not contain. Construction owns that stronger invariant; decoding owns
   the structural, identity, and digest invariants available from the durable
   payload.

6. Make serialization return a `contracts.SerializedArtifact` with the fixed
   kind, schema, and media type, canonical content, and no duplicated metadata.
   Validate the embedded schema defensively so the checked-in schema and Go
   contract cannot silently diverge.

7. Define a strict JSON Schema with:

   - `additionalProperties: false` at every fixed object level;
   - all roadmap-required fields;
   - `minItems: 1` for chunks;
   - positive integer range endpoints and unit counts;
   - zero-or-greater indexes;
   - the pattern `^sha256:[0-9a-f]{64}$` for digests;
   - non-empty strings for identities;
   - optional, non-empty `llm_profile`; and
   - annotation objects whose namespace values may be arbitrary JSON.

### Tests

Add focused behavioral tests proving:

- the valid fixture round-trips through the codec;
- building from a real accepted source, plan, and materialized chunk list
  produces the fixed envelope and exact durable fields;
- output is deterministic and annotation JSON formatting does not alter the
  reconstructed plan digest;
- unknown fields, trailing JSON, malformed identities/digests, invalid
  indexes, duplicate IDs, mismatched source identities, invalid ranges or
  counts, malformed annotation namespaces/values, and plan-digest mismatches
  are rejected;
- construction rejects materialized chunks that differ from the accepted plan
  or document order; and
- mutating caller-owned inputs or returned values cannot mutate another owned
  representation.

Keep these tests semantic. Do not add exact byte-length, exact schema-length,
private-helper, or redundant field-by-field “change detector” tests.

### Completion Criteria

- The framework contract can represent and validate the roadmap payload
  without importing output or D&D packages.
- Construction proves the artifact describes the exact accepted materialized
  chunks.
- No production output behavior has changed yet.

## Stage 2: Carry The Accepted Artifact Through The Runner

### Goal

Construct the durable artifact at the accepted chunk-plan boundary and make it
available to output encoders without changing default logical files.

### Work

1. Extend `contracts.OutputRequest` with:

   ```go
   ChunkMap *contracts.SerializedArtifact `json:"chunk_map,omitempty"`
   ```

   Update its cloning/ownership path so the envelope, content bytes, and
   metadata are defensively copied. Do not place chunk-map content in the run
   manifest or generic request metadata.

2. In `internal/framework/pipeline/runner.go`, construct the serialized chunk
   map after materialization and configured chunk validation accept the plan,
   and before the output encoder is invoked. Build it from:

   - the accepted source document;
   - the accepted logical plan;
   - the exact materialized chunks passed to lanes;
   - the chunk module selected by the current resolved pipeline as
     `requested_chunker`; and
   - the original producer identities recorded by the accepted chunk-plan
     record.

3. Preserve cache semantics:

   - a generated or bypassed plan records the current producer;
   - a cache hit keeps the stored producer's input module, chunk module, and
     optional LLM profile;
   - `requested_chunker` still records the current resolved chunk module; and
   - producer references, metadata, warnings, and cache details remain only in
     existing provenance surfaces.

4. Apply the accepted-state policy:

   - pass no chunk map when chunk validation rejects the candidate;
   - retain the artifact when a later extraction, merge, or normalization lane
     is rejected;
   - treat artifact construction or serialization failure after acceptance as
     a framework run error; and
   - do not rerun chunking or make another LLM request.

5. If debug serialization records the output request, represent the new value
   through the existing safe serialized-artifact envelope conventions. Do not
   duplicate its content into the manifest or add a new default debug surface.

### Tests

Extend `internal/framework/pipeline/runner_chunk_plan_test.go` and the nearest
existing output-request ownership tests to prove:

- accepted generated and cached plans reach a capturing output encoder as a
  valid serialized chunk map;
- cache reuse distinguishes current `requested_chunker` from the stored
  producer chunker;
- chunk rejection yields a nil chunk map;
- a later lane rejection still leaves the accepted chunk map available;
- captured requests do not alias runner, store, or caller-owned bytes/maps.

Use existing fake modules and stores. Do not add an LLM-backed integration test
for behavior already deterministically owned by the runner, and do not add a
production injection seam solely to force an otherwise unreachable
construction failure; Stage 1 owns invalid-construction coverage.

### Completion Criteria

- Every accepted chunk plan yields one validated serialized chunk map in the
  output request.
- Rejected chunk plans yield none, while later lane rejection does not discard
  it.
- Existing output encoders remain behaviorally unchanged.

## Stage 3: Add Opt-In JSON Bundle Export

### Goal

Expose the available framework artifact as `chunk-map.json` only when the JSON
output binding explicitly enables it.

### Work

1. Add an `Options` field:

   ```go
   IncludeChunkMap bool
   ```

   Keep `New()` as the default-disabled constructor and add
   `NewWithOptions(Options)` for configured construction and focused tests.

2. Update the JSON output builder to decode only `include_chunk_map`:

   - omit or `false` means disabled;
   - require a boolean when present; and
   - continue rejecting unknown keys.

3. When disabled, ignore an available `OutputRequest.ChunkMap` and preserve the
   exact existing bundle shape. When enabled:

   - omit both file and descriptor if the request has no accepted chunk map;
   - validate the artifact kind, schema identity/version, and media type;
   - decode through `chunkmap.New()` so malformed or non-canonical payloads
     fail the run;
   - write valid content as pretty-printed JSON with the encoder's established
     trailing-newline convention; and
   - use the existing logical-path validation.

   Reuse or generalize the current serialized-artifact JSON helper rather than
   adding a second subtly different validation and formatting path.

4. Extend the output index model with an optional `chunk_map` descriptor
   containing exactly:

   - `artifact_kind`;
   - `file`;
   - `media_type`;
   - `schema_id`;
   - `schema_name`; and
   - `schema_version`.

   Do not add the chunk map to lane-oriented `output_files`, and do not change
   the manifest.

5. Treat a present but invalid accepted artifact as an output error. Do not
   silently omit it when export was explicitly requested.

### Tests

Extend `internal/modules/generic/output/json/encoder_test.go` and existing
output registration tests to prove:

- option omission and `false` retain the current default bundle even when an
  artifact is available;
- `true` plus a valid artifact emits `chunk-map.json` and the exact optional
  index descriptor;
- `true` plus no accepted artifact emits neither file nor descriptor;
- the chunk map never appears in lane `output_files`;
- wrong option types and unknown options are rejected;
- wrong kind, schema, media type, or invalid content returns an error;
- emitted JSON is valid, formatted consistently, and newline-terminated; and
- encoding does not mutate the request or serialized content.

Prefer testing the observable logical file map and decoded index rather than
private helper calls or exact byte lengths.

### Completion Criteria

- Existing JSON configurations remain unchanged by default.
- An explicitly enabled JSON output produces exactly one validated
  `chunk-map.json` file and one pipeline-wide index descriptor when an
  accepted map exists.
- No lane identity or manifest structure is fabricated for the chunk map.

## Stage 4: Prove The Assembled Behavior And Document It

### Goal

Demonstrate the feature with the production D&D scene chunker, publish all
current-behavior documentation, and close the roadmap scope.

### Work

1. Extend the representative assembled production contract in
   `internal/cli/production_contract_test.go`:

   - configure `dnd/scenes` with JSON `include_chunk_map: true`;
   - execute through normal production assembly with the existing deterministic
     fake LLM;
   - read `index.json` and `chunk-map.json`;
   - decode the artifact through the framework codec; and
   - assert the actual accepted scene ranges, stable IDs/order, source and plan
     identity, requested/current producer provenance, and surviving namespaced
     scene annotations.

   This is the single assembled proof. Do not duplicate all codec and encoder
   edge cases at the CLI layer.

2. Add a copyable maintained example such as
   `examples/dnd-scene-chunk-map.config.yml` that uses `dnd/scenes`, one normal
   D&D artifact lane, and the opt-in JSON output option. Add it to
   `internal/cli/example_contract_test.go` so schema/config drift is caught by
   the existing example contract.

3. Add `docs/integrations/chunk-map.md` as the canonical durable contract
   reference. Document fixed identities, every payload field, invariants,
   accepted-state behavior, cache provenance semantics, sensitivity, and
   excluded content. Link to the embedded schema without duplicating it in the
   prose.

4. Update current-behavior documentation:

   - `docs/integrations/json-output.md`: optional logical file and index
     descriptor;
   - `docs/config.md`: strict `include_chunk_map` option and copyable binding;
   - `docs/operations.md`: opt-in sensitivity and retention implications;
   - `docs/internal/pipeline.md`: accepted-plan construction and output-request
     handoff;
   - `docs/internal/state.md`: logical output and manifest/index ownership;
   - `docs/internal/modules.md`: JSON encoder option and responsibility; and
   - `docs/internal/overview.md`: framework chunk-map contract and encoder
     boundary.

   Keep implementation details in internal docs, user-selectable behavior in
   configuration docs, and the durable schema only in integrations docs.

5. Close planning state:

   - mark `accepted-chunk-map-export.md` as implemented;
   - remove the completed “Export Accepted Chunk Maps” entry from
     `docs/roadmap/future.md`; and
   - mark this implementation plan completed, preserving it until the user
     chooses to retire completed roadmap documents.

6. Review links, examples, terminology, and file identities across the
   documentation. Do not add an ADR: this feature applies the existing fixed
   pipeline, output/cache separation, and framework ownership decisions rather
   than changing them.

### Validation

Run focused tests after each stage. At completion, run:

```sh
go test ./...
go vet ./...
go build ./cmd/notarius
git diff --check
```

Also verify that maintained example tests pass, all new documentation links
resolve, and a default JSON output run still has no chunk-map file or index
descriptor.

### Completion Criteria

- The assembled scene-chunking run exports the exact accepted map and
  namespaced annotations without a second model call.
- Current documentation and the maintained example describe only implemented
  behavior.
- The future roadmap no longer lists the completed feature.
- Repository-wide tests, vet, build, formatting, and documentation checks pass.

## Cross-Stage Guardrails

- Do not expose internal `source.Chunk` directly as the durable DTO.
- Do not add transcript content, materialized units, source-unit metadata,
  model proposals, or debug data to the artifact.
- Do not create an extractor, lane registration, normalizer, generated
  reference, import path, or replay mechanism for chunk maps.
- Do not make D&D scene annotation fields part of the generic schema.
- Do not make export the default or add a CLI-global/filesystem option.
- Do not change chunk selection, validation, cache lookup/publication, or lane
  execution semantics.
- Do not duplicate chunk-map payloads in the manifest or lane output indexes.
- Preserve defensive ownership at source, runner, request, codec, and encoder
  boundaries.
- Keep tests offline and deterministic, and test each invariant at its stable
  owning layer.

## Open Questions

None. The feature roadmap and the fixed decisions above resolve the
implementation-significant choices.