notarius/docs/roadmap/accepted-chunk-map-export.md

# Accepted Chunk Map Export

Status: Implemented

## Purpose

Notarius already creates and validates one materialized chunk map before any
artifact lane executes. That map contains stable chunk identities, ordered
current-source ranges, and accepted namespaced annotations. It is useful
downstream data, but today it is visible only in internal execution and
explicit debug surfaces.

Add an opt-in durable chunk-map artifact to the output bundle. Export the exact
accepted chunks used by the run rather than reconstructing them through an
extractor or exposing a chunker's unvalidated model response.

This is a framework and output concern. Chunking remains pipeline-wide and
precedes all artifact lanes.

## Desired End State

The production JSON output encoder can be configured to add a canonical
`chunk-map.json` file to the logical output bundle. The file describes the
accepted materialized chunks used for lane execution without embedding source
units or transcript text.

Use these durable identities:

| Concern | Identity |
| --- | --- |
| Artifact kind | `source/chunk-map` |
| Logical file | `chunk-map.json` |
| Schema ID | `notarius.source.chunk_map` |
| Schema name | `notarius_source_chunk_map_v1` |
| Schema version | `v1` |
| Media type | `application/json` |

The framework owns the artifact model, schema, validation, and canonical
encoding. Output encoders receive the serialized artifact through the generic
output request. The JSON encoder owns only the opt-in decision, logical file
placement, and output-index entry.

## Configuration

Add one strict option to the production `json` output module:

```yaml
pipelines:
  dnd-session:
    input: seriatim
    output:
      module: json
      options:
        include_chunk_map: true
```

`include_chunk_map` is a boolean and defaults to `false`. Unknown options and
non-boolean values remain configuration errors. Do not add a separate CLI flag
or top-level filesystem option: this choice changes the logical files produced
by an output module, not their physical destination.

Other output encoders may ignore the available chunk-map artifact unless and
until they define their own explicit export behavior.

## Durable Artifact Contract

The payload is one strict JSON object with this shape:

```json
{
  "source_id": "session-7",
  "source_digest": "sha256:0123456789abcdef...",
  "plan_digest": "sha256:abcdef0123456789...",
  "requested_chunker": "dnd/scenes",
  "producer": {
    "input_module": "seriatim",
    "chunk_module": "dnd/scenes",
    "llm_profile": "dnd-scenes"
  },
  "plan_annotations": {},
  "chunks": [
    {
      "id": "chunk-000001",
      "index": 0,
      "source_ref": {
        "source_id": "session-7",
        "start_unit_id": 1,
        "end_unit_id": 18
      },
      "unit_count": 18,
      "annotations": {
        "dnd/scenes": {
          "title": "At the city gate"
        }
      }
    }
  ]
}
```

Required top-level fields are:

- `source_id`: the accepted source document identity;
- `source_digest`: the canonical digest of that document;
- `plan_digest`: the canonical digest of the accepted chunk plan;
- `requested_chunker`: the chunk module selected by the current resolved
  pipeline;
- `producer`: the identity of the component that originally produced the
  accepted plan;
- `plan_annotations`: the accepted plan-level annotation namespace map; and
- `chunks`: the non-empty ordered list used for lane execution.

`producer` requires `input_module` and `chunk_module`. `llm_profile` is included
only when the producing chunker was LLM-backed. Producer references, module
metadata, warnings, creation timestamps, cache paths, and cache actions remain
in their existing provenance and diagnostic surfaces; they are not copied into
this artifact.

Each chunk requires:

- `id`: the stable materialized chunk ID;
- `index`: its zero-based position in execution order;
- `source_ref`: the exact inclusive current-source range;
- `unit_count`: the number of materialized source units in that range; and
- `annotations`: the accepted range-level annotation namespace map.

Unknown fields are rejected at every fixed object level. Annotation namespaces
retain their canonical JSON values and may contain domain-specific JSON of any
type. Empty annotation maps are encoded as `{}` so the shape remains explicit.

The durable artifact must guarantee that:

- source and module identities are non-empty and contain no surrounding
  whitespace;
- source and plan digests use the canonical `sha256:` representation;
- chunk IDs are non-empty and unique;
- chunk indexes are unique, contiguous, zero-based, and agree with array order;
- every chunk reference uses the top-level source identity;
- every range has positive endpoint IDs and was validated in source-document
  order when the artifact was constructed;
- every unit count is positive;
- annotations contain valid JSON under non-empty canonical namespaces and are
  canonicalized before digest calculation; and
- reconstructing the logical plan from the source digest, plan annotations,
  chunk ranges, and chunk annotations reproduces `plan_digest`.

The artifact represents chunk structure, not source content. It must not
contain materialized units, transcript bytes, source-unit metadata, chunk
content, private model responses, rejected boundary proposals, or debug
payloads.

## Accepted-State And Failure Policy

Construct the artifact only after plan materialization and the configured chunk
validator chain have accepted the chunks. The runner passes the same immutable
logical chunk identities, ranges, and annotations used for every lane; no
second chunking or model call occurs.

When `include_chunk_map` is enabled:

- emit the artifact even if one or more later artifact lanes are rejected;
- omit it when chunk validation rejects the candidate map, because no accepted
  chunk map exists;
- retain the ordinary rejection record when it is omitted for that reason; and
- treat failure to construct, validate, serialize, index, or write an available
  accepted chunk map as a run failure, consistent with any explicitly requested
  durable output.

A cache hit and a newly generated plan with the same accepted logical plan must
produce the same chunk ranges, annotations, plan digest, and materialized chunk
identities. The artifact records both the current `requested_chunker` and the
stored producer's `chunk_module`, since canonical plan reuse permits those
identities to differ.

## Output Bundle Integration

Extend `index.json` with an optional `chunk_map` descriptor:

```json
{
  "chunk_map": {
    "artifact_kind": "source/chunk-map",
    "file": "chunk-map.json",
    "media_type": "application/json",
    "schema_id": "notarius.source.chunk_map",
    "schema_name": "notarius_source_chunk_map_v1",
    "schema_version": "v1"
  }
}
```

The descriptor and file are both absent when export is disabled or no accepted
chunk map exists. The chunk map does not appear in lane-oriented
`output_files`, because it is pipeline-wide and has no lane, extractor, merger,
or normalizer identity.

The existing manifest remains the canonical run-provenance index. Its
`chunk_plan` summary continues to own cache mode, lookup and publication
actions, producer reference provenance, module metadata, timestamps, and
validation status. The chunk-map artifact provides the accepted structure and
only the minimal producer identities needed to interpret it independently.

## Ownership And Architecture

Use a domain-neutral framework package for the durable chunk-map DTO, embedded
JSON Schema, invariant validation, cloning, and canonical serialization. Do not
marshal `source.Chunk` directly: its content, units, metadata, and internal
fields are intentionally broader than this external contract.

Extend the output request with an optional cloned serialized chunk-map artifact.
The runner constructs that value from the accepted chunk-plan execution result
before invoking the output encoder. This keeps source and chunk-plan knowledge
out of the generic JSON encoder and allows future encoders to consume the same
framework-owned representation.

The JSON encoder adds the file and index descriptor only when its
`include_chunk_map` option is true and the request contains an accepted
artifact. It applies the same logical-path validation and pretty-printed JSON
conventions as the rest of the bundle.

## Security And Data Handling

Chunk annotations may contain model-derived or source-derived information.
Treat `chunk-map.json` as durable user output with the same sensitivity and
retention expectations as lane artifacts. Opt-in export prevents new durable
content from appearing silently in existing pipelines.

Do not copy annotation values, source ranges, or chunk IDs into the manifest.
Do not include external-reference content or filesystem paths in the chunk map.
Existing output-directory confinement, atomic writing, and permission policy
apply unchanged.

## Quality And Documentation Policy

The framework contract must remain strict, canonical, immutable across
ownership boundaries, and independent of source or D&D interpretation. The
runner must preserve accepted-state and cache-producer provenance, while the
JSON encoder must keep opt-in file selection separate from chunk semantics.

Once implemented, the durable schema belongs in `docs/integrations/`;
configuration owns the selectable option; pipeline and state internals own the
framework handoff; and a maintained scene-chunking example should demonstrate
that namespaced annotations survive export. Tests should protect those
observable contracts without relying on exact payload lengths or private
helper structure.

## Non-Goals

This scope does not:

- add a chunk-map extraction lane or artifact-lane registration;
- rerun chunking or ask an LLM to reconstruct accepted chunks;
- export transcript content, source units, source-unit metadata, or raw model
  proposals;
- interpret D&D scene annotations as a generic contract;
- add scene descriptions, titles, summaries, kinds, or participant extraction;
- make chunk maps generated references or inputs to later ordered steps;
- change chunk planning, validation, cache selection, or publication behavior;
- add a chunk-map import or replay mechanism;
- enable export by default; or
- introduce a general output-plugin capability negotiation system.