Document accepted chunk map export

docs/config.md

@@ -25,6 +25,7 @@ The explicit-path option is defined in the [CLI reference](cli.md).
 - [D&D combat-turn configuration](../examples/dnd-combat-turns.config.yml)
 - [D&D NPC-grounded spell and combat configuration](../examples/dnd-npc-grounded.config.yml)
 - [D&D NPC interaction configuration](../examples/dnd-npc-interactions.config.yml)
+- [D&D scene chunk-map configuration](../examples/dnd-scene-chunk-map.config.yml)
 
 All are complete version 3 files. The fragments below illustrate individual
 fields and are not alternate complete configurations.
@@ -331,6 +332,25 @@ During resolution, each selected module's registered option validator runs.
 Production input, chunk, and output bindings reject unknown or invalid options
 with the affected binding context.
 
+### JSON Output Options
+
+The `json` output module accepts only `include_chunk_map`, a boolean that
+defaults to `false`. When `true`, it adds the accepted pipeline-wide chunk map
+to the logical output bundle when one exists. It does not create a CLI flag or
+change output placement.
+
+```yaml
+output:
+  module: json
+  options:
+    include_chunk_map: true
+```
+
+The payload and index descriptor are defined by the
+[Accepted Chunk Map contract](integrations/chunk-map.md). See the complete
+[D&D scene chunk-map configuration](../examples/dnd-scene-chunk-map.config.yml)
+for a copyable pipeline.
+
 Validator bindings use the same shorthand or object module-binding form, but
 only these fields are supported:
 
@@ -363,7 +383,7 @@ production validators do not call the LLM and must not set `llm_profile`.
 | normalize | `dnd/npcs` | Deterministically consolidates typed D&D NPC-list artifacts by canonical name and unions exact evidence. |
 | normalize | `dnd/combat-turns` | Deterministically canonicalizes, orders, and de-duplicates typed D&D combat-turn artifacts. |
 | normalize | `dnd/npc-interactions` | Canonicalizes registry NPC names, orders interaction occurrences, and removes only exact duplicates. |
-| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
+| output | `json` | Produces JSON output files for normalized `application/json` lanes and can opt in to an accepted chunk map. |
 
 ## Implemented Production Validators
 

docs/integrations/chunk-map.md

@@ -0,0 +1,83 @@
+# Accepted Chunk Map
+
+This document defines the durable accepted chunk-map artifact that the JSON
+output encoder can write as `chunk-map.json`. It describes the exact accepted,
+materialized chunks used by a run; it is not a lane artifact and is never an
+input to later pipeline steps. Enable it with the JSON output option described
+in [Configuration](../config.md#json-output-options).
+
+## Identity
+
+- Artifact kind: `source/chunk-map`
+- Logical file: `chunk-map.json`
+- Schema ID: `notarius.source.chunk_map`
+- Schema name: `notarius_source_chunk_map_v1`
+- Schema version: `v1`
+- Media type: `application/json`
+
+The checked-in [JSON Schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json)
+defines the strict wire shape. `chunk-map.json` is listed by the optional
+`chunk_map` descriptor in [the JSON output index](json-output.md#indexjson),
+not by the lane-oriented `output_files` collection.
+
+## Payload
+
+The payload has these required fields:
+
+- `source_id`: accepted source-document identity.
+- `source_digest`: canonical lower-case `sha256:` digest of that document.
+- `plan_digest`: canonical lower-case `sha256:` digest of the accepted logical
+  plan.
+- `requested_chunker`: chunk module selected by the current resolved pipeline.
+- `producer`: the original accepted-plan producer, with required
+  `input_module` and `chunk_module`; `llm_profile` is present only for an
+  LLM-backed producer.
+- `plan_annotations`: accepted plan-level annotation namespace map. It is
+  `{}` when no namespaces are present.
+- `chunks`: non-empty execution-order collection of accepted chunks.
+
+Each chunk has `id`, zero-based `index`, `source_ref`, positive `unit_count`,
+and an explicit `annotations` namespace map. A source reference has the source
+ID and inclusive positive `start_unit_id` and `end_unit_id` endpoints.
+Annotation values are arbitrary valid JSON under non-empty namespaces. They
+are preserved as canonical JSON without interpreting any module-specific
+namespace.
+
+## Invariants
+
+The framework constructs this artifact only after materializing the selected
+logical plan and accepting it through the configured chunk validator chain.
+Construction proves the source and plan digests, source-document range order,
+materialized chunk IDs and indexes, source references, unit membership and
+counts, and plan/range annotations agree exactly. Chunk IDs are unique and
+indexes are contiguous and agree with array order.
+
+The codec rejects unknown fixed-object fields, malformed identities or
+digests, invalid annotation JSON, trailing JSON content, and any payload whose
+reconstructed logical plan does not reproduce `plan_digest`. It makes
+defensive copies at serialization and decoding boundaries.
+
+## Acceptance And Provenance
+
+The artifact is available only when the chunk plan was accepted. It remains
+available when a later extraction, merge, or normalization result is rejected;
+it is absent when chunk validation rejects the candidate plan.
+
+`requested_chunker` describes the current pipeline selection. `producer`
+describes who originally produced the accepted plan. On a cache hit these can
+differ: the accepted ranges, annotations, digests, and stable materialized IDs
+are reused, while the producer remains the stored producer. Cache paths,
+actions, references, metadata, warnings, timestamps, and detailed provenance
+remain in the run manifest rather than this payload.
+
+## Data Handling
+
+The map contains structure, not source content. It excludes transcript bytes,
+materialized units, source-unit metadata, chunk content, private model
+responses, rejected proposals, debug data, external-reference content, and
+filesystem paths.
+
+Annotations can nevertheless be source- or model-derived. Treat an enabled
+`chunk-map.json` with the same sensitivity and retention expectations as lane
+output. Physical placement, confined atomic writing, and permissions follow
+the ordinary [output operation](../operations.md#output).

docs/integrations/json-output.md

@@ -18,6 +18,8 @@ The encoder writes:
 - `lanes/<lane-id>.json`, one file per normalized serialized artifact
 - `rejected.json`
 - `warnings.json`
+- `chunk-map.json`, only when the JSON output binding enables
+  `include_chunk_map` and the run has an accepted chunk map
 
 Files are pretty-printed JSON with a trailing newline when the payload is JSON.
 Logical file paths are relative, slash-separated, and may not contain `..`.
@@ -60,6 +62,13 @@ contains the normalized payload `media_type`, normalizer `module_key`, and
 response `schema_id`, `schema_name`, and `schema_version` when those values are
 available.
 
+When present, the top-level optional `chunk_map` descriptor contains exactly
+`artifact_kind`, `file`, `media_type`, `schema_id`, `schema_name`, and
+`schema_version`. It identifies the pipeline-wide `chunk-map.json`; it is not
+a lane output and never appears in `output_files`. The descriptor and file are
+both absent when export is disabled or no chunk plan was accepted. Its payload
+contract is defined by [Accepted Chunk Map](chunk-map.md).
+
 ## `manifest.json`
 
 `manifest.json` contains a run manifest. This abridged example shows its core

docs/internal/modules.md

@@ -391,6 +391,12 @@ safe logical names, pretty-prints JSON payloads, and assembles the logical index
 manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
 media types, unsafe names, and sanitized-name collisions are errors.
 
+Its strict `include_chunk_map` option is disabled by default. When enabled, it
+validates the framework-supplied accepted chunk map through its codec and adds
+the pipeline-wide `chunk-map.json` plus its index descriptor; it does not treat
+the map as a lane payload. The external shape is owned by the
+[Accepted Chunk Map contract](../integrations/chunk-map.md).
+
 The encoder returns logical files only. The CLI places them on disk, and the
 [JSON output contract](../integrations/json-output.md) defines their external
 paths and schemas.

docs/internal/overview.md

@@ -50,6 +50,7 @@ required normalized artifacts have crossed the typed handoff.
 | `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
 | `internal/framework/checkpoint` | Root-based checkpoint loading, recording, identity, and payload serialization. |
 | `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
+| `internal/framework/chunkmap` | Strict durable accepted chunk-map construction, schema, validation, cloning, and serialization. |
 | `internal/framework/debug` | Root-based framework and LLM debug recording. |
 
 Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
@@ -109,7 +110,7 @@ Configuration. The implemented module packages are:
 | `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
 | `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
 | `internal/modules/dnd/normalize/npcs` | Consolidates NPC records deterministically by canonical name, unions exact evidence, and reports bounded warnings. |
-| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
+| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, rejections, and an explicitly enabled accepted chunk map as logical JSON files. |
 
 `internal/modules/dnd/shared` owns reusable D&D prompt fragments,
 reference declarations, prompt input assembly, source-unit reference helpers,

docs/internal/pipeline.md

@@ -208,15 +208,18 @@ The runner:
 3. selects a stored plan or executes the configured chunker's `Plan` operation;
 4. canonicalizes and materializes the plan, then validates the resulting
    chunks;
-5. executes each resolved step in configuration order. For one step, it
+5. builds the framework-owned accepted chunk map from the accepted source,
+   logical plan, and exact materialized chunks, then supplies it to the output
+   request independently of output-module options;
+6. executes each resolved step in configuration order. For one step, it
    dispatches extract jobs in source-chunk then resolved-lane order, starts a
    bounded lane continuation when all extracts for that lane are terminal, and
    waits for every lane to become terminal;
-6. encodes and validates each accepted normalized producer artifact, then
+7. encodes and validates each accepted normalized producer artifact, then
    builds the immutable generated reference sets for the next step;
-7. invokes the prepared output encoder only after every step succeeds and
+8. invokes the prepared output encoder only after every step succeeds and
    validates its logical file results;
-8. returns the assembled manifest, outcomes, warnings, and files.
+9. returns the assembled manifest, outcomes, warnings, and files.
 
 Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
 and validators while performing these transitions:
@@ -265,6 +268,14 @@ currently requested chunker and the effective plan producer. Cache state and
 paths are configured and operated outside the runner; see
 [Configuration](../config.md#state-surfaces) and [Operations](../operations.md).
 
+For an accepted plan, the runner also constructs the strict framework-owned
+[Accepted Chunk Map](../integrations/chunk-map.md) before lane execution. It
+uses the current resolved chunker as `requested_chunker` and the stored or
+generated record as `producer`, preserving that distinction on reuse. Chunk
+rejection supplies no map; later lane rejection does not discard it. Output
+encoders receive a defensively owned serialized value and may explicitly
+ignore it.
+
 The extract job channel has the same capacity as the effective extract worker
 count, so dispatch applies backpressure. A fixed continuation executor prevents
 ready or checkpoint-reused lanes from creating one goroutine each. Workers and

docs/internal/state.md

@@ -17,6 +17,11 @@ debug roots.
 The pipeline runner returns logical output files. After validating every
 logical name, the CLI exclusively creates the run directory beneath the
 selected output root and performs confined, atomic file writes within it.
+The runner supplies an accepted chunk map as an optional, defensively owned
+output-request artifact. The JSON encoder alone decides whether its explicit
+option writes the map and optional index descriptor; neither the map payload
+nor its annotations are copied into the run manifest. The durable fields are
+owned by the [Accepted Chunk Map contract](../integrations/chunk-map.md).
 
 `internal/framework/chunkplan` owns source-addressed plan storage, validation,
 and atomic publication. Its store is constructed only when the selected mode is

docs/operations.md

@@ -37,6 +37,12 @@ automatically removes output. The
 [JSON output contract](integrations/json-output.md) owns the logical file
 names, schemas, and media types inside a run directory.
 
+An enabled JSON `include_chunk_map` option adds an accepted chunk map to durable
+output. Its annotations may contain source- or model-derived data, so retain
+and protect it like lane output. The map is opt-in and does not alter existing
+bundles; its payload exclusions are defined in the
+[Accepted Chunk Map contract](integrations/chunk-map.md).
+
 Remove an output run directory only after its consumer data is no longer
 needed. This is data deletion, not cache cleanup.
 

docs/roadmap/accepted-chunk-map-export.md

@@ -1,6 +1,6 @@
 # Accepted Chunk Map Export
 
-Status: Accepted
+Status: Implemented
 
 ## Purpose
 

docs/roadmap/future.md

@@ -16,13 +16,6 @@ not as committed release dates.
   validator, and normalizer development. Treat model-quality review as an
   iterative human evaluation aid, not a deterministic correctness gate.
 
-### Export Accepted Chunk Maps
-
-Add an opt-in, framework-owned durable artifact for the exact accepted
-materialized chunks used by lane execution. The accepted contract, output
-boundary, provenance policy, and exclusions are defined in
-[Accepted Chunk Map Export](accepted-chunk-map-export.md).
-
 ### Extract D&D Scene Descriptions
 
 - Add a `dnd/scene-descriptions` extractor that runs once for each accepted

docs/roadmap/implementation.md

@@ -1,6 +1,6 @@
 # Accepted Chunk Map Export Implementation Plan
 
-Status: Ready for implementation
+Status: Completed
 
 ## Objective
 

examples/dnd-scene-chunk-map.config.yml

@@ -0,0 +1,23 @@
+version: 3
+output:
+  directory: ./notarius-output
+cache:
+  chunk_plans:
+    mode: bypass
+  checkpoints:
+    enabled: false
+    directory: ""
+debug:
+  directory: ./notarius-debug
+pipelines:
+  dnd-scene-chunk-map:
+    input: seriatim
+    chunk: dnd/scenes
+    output:
+      module: json
+      options:
+        include_chunk_map: true
+    artifacts:
+      spells:
+        extract: dnd/spells
+        normalize: dnd/spells

internal/cli/production_contract_test.go

@@ -18,6 +18,7 @@ import (
 
 	"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
 	"gitea.maximumdirect.net/eric/notarius/internal/core/config"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkmap"
 	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
 	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
 	"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
@@ -481,6 +482,42 @@ func TestProductionSceneRunRecordsChunkerWarningsAndProvenance(t *testing.T) {
 	if got := manifest.ChunkPlan.ProducerMetadata["response_schema_id"]; got != scenes.ResponseSchemaID {
 		t.Fatalf("chunk producer schema metadata = %#v, want %q", got, scenes.ResponseSchemaID)
 	}
+	index := readProductionJSON[productionChunkMapIndex](t, filepath.Join(outputRoot, productionRunID, "index.json"))
+	if index.ChunkMap == nil || index.ChunkMap.ArtifactKind != chunkmap.ArtifactKind || index.ChunkMap.File != "chunk-map.json" || index.ChunkMap.MediaType != chunkmap.MediaType || index.ChunkMap.SchemaID != chunkmap.SchemaID || index.ChunkMap.SchemaName != chunkmap.SchemaName || index.ChunkMap.SchemaVersion != chunkmap.SchemaVersion {
+		t.Fatalf("chunk map index = %#v, want fixed chunk map descriptor", index.ChunkMap)
+	}
+	for _, output := range index.OutputFiles {
+		if output.File == index.ChunkMap.File {
+			t.Fatalf("lane output files = %#v, want no chunk map", index.OutputFiles)
+		}
+	}
+	content, err := os.ReadFile(filepath.Join(outputRoot, productionRunID, index.ChunkMap.File))
+	if err != nil {
+		t.Fatal(err)
+	}
+	chunkMap, err := chunkmap.New().Decode(content)
+	if err != nil {
+		t.Fatalf("Decode(chunk map) error = %v", err)
+	}
+	if chunkMap.SourceID != "session-alpha" || chunkMap.SourceDigest != manifest.ChunkPlan.SourceDigest || chunkMap.PlanDigest != manifest.ChunkPlan.PlanDigest || chunkMap.RequestedChunker != scenes.Key || chunkMap.Producer.InputModule != "seriatim" || chunkMap.Producer.ChunkModule != scenes.Key || chunkMap.Producer.LLMProfile != manifest.ChunkPlan.ProducerLLMProfile {
+		t.Fatalf("chunk map identity and producer = %#v, want accepted scene plan provenance", chunkMap)
+	}
+	if len(chunkMap.Chunks) != 1 || chunkMap.Chunks[0].ID != "chunk-000001" || chunkMap.Chunks[0].Index != 0 || chunkMap.Chunks[0].SourceRef.SourceID != "session-alpha" || chunkMap.Chunks[0].SourceRef.StartUnitID != 1 || chunkMap.Chunks[0].SourceRef.EndUnitID != 2 || chunkMap.Chunks[0].UnitCount != 2 {
+		t.Fatalf("chunk map chunks = %#v, want one stable accepted scene range", chunkMap.Chunks)
+	}
+	var planAnnotation struct {
+		BoundaryCaveats []string `json:"boundary_caveats"`
+	}
+	if err := json.Unmarshal(chunkMap.PlanAnnotations["dnd/scenes"], &planAnnotation); err != nil || len(planAnnotation.BoundaryCaveats) != 1 {
+		t.Fatalf("chunk map plan annotation = %s, %v; want scene caveat", chunkMap.PlanAnnotations["dnd/scenes"], err)
+	}
+	var rangeAnnotation struct {
+		ShortTitle  string `json:"short_title"`
+		PrimaryMode string `json:"primary_mode"`
+	}
+	if err := json.Unmarshal(chunkMap.Chunks[0].Annotations["dnd/scenes"], &rangeAnnotation); err != nil || rangeAnnotation.ShortTitle != "Opening scene" || rangeAnnotation.PrimaryMode != "Narrative" {
+		t.Fatalf("chunk map range annotation = %s, %v; want surviving scene annotation", chunkMap.Chunks[0].Annotations["dnd/scenes"], err)
+	}
 	warnings := readProductionJSON[struct {
 		Warnings []contracts.Warning `json:"warnings"`
 	}](t, filepath.Join(outputRoot, productionRunID, "warnings.json"))
@@ -507,6 +544,7 @@ func maintainedExampleFiles(t *testing.T) []maintainedExample {
 		{name: "combat", path: repositoryPath("examples", "dnd-combat-turns.config.yml"), pipelineIDs: []string{"dnd-combat"}},
 		{name: "npc-grounded", path: repositoryPath("examples", "dnd-npc-grounded.config.yml"), pipelineIDs: []string{"dnd-npc-grounded"}},
 		{name: "npc-interactions", path: repositoryPath("examples", "dnd-npc-interactions.config.yml"), pipelineIDs: []string{"dnd-npc-interactions"}},
+		{name: "scene-chunk-map", path: repositoryPath("examples", "dnd-scene-chunk-map.config.yml"), pipelineIDs: []string{"dnd-scene-chunk-map"}},
 	}
 }
 
@@ -588,12 +626,30 @@ pipelines:
   dnd-session:
     input: seriatim
     chunk: %s
+    output:
+      module: json
+      options:
+        include_chunk_map: true
     artifacts:
       spells:
         extract: dnd/spells
 `, outputRoot, filepath.Join(filepath.Dir(outputRoot), "debug"), chunkModule)
 }
 
+type productionChunkMapIndex struct {
+	OutputFiles []struct {
+		File string `json:"file"`
+	} `json:"output_files"`
+	ChunkMap *struct {
+		ArtifactKind  contracts.ArtifactKind `json:"artifact_kind"`
+		File          string                 `json:"file"`
+		MediaType     string                 `json:"media_type"`
+		SchemaID      string                 `json:"schema_id"`
+		SchemaName    string                 `json:"schema_name"`
+		SchemaVersion string                 `json:"schema_version"`
+	} `json:"chunk_map"`
+}
+
 func writeProductionContractConfig(t *testing.T, content string) string {
 	t.Helper()
 	path := filepath.Join(t.TempDir(), "config.yml")