notarius/docs/roadmap/accepted-chunk-map-export.md

Accepted Chunk Map Export

Status: Implemented

Purpose

Notarius already creates and validates one materialized chunk map before any artifact lane executes. That map contains stable chunk identities, ordered current-source ranges, and accepted namespaced annotations. It is useful downstream data, but today it is visible only in internal execution and explicit debug surfaces.

Add an opt-in durable chunk-map artifact to the output bundle. Export the exact accepted chunks used by the run rather than reconstructing them through an extractor or exposing a chunker's unvalidated model response.

This is a framework and output concern. Chunking remains pipeline-wide and precedes all artifact lanes.

Desired End State

The production JSON output encoder can be configured to add a canonical chunk-map.json file to the logical output bundle. The file describes the accepted materialized chunks used for lane execution without embedding source units or transcript text.

Use these durable identities:

Concern	Identity
Artifact kind	source/chunk-map
Logical file	chunk-map.json
Schema ID	notarius.source.chunk_map
Schema name	notarius_source_chunk_map_v1
Schema version	v1
Media type	application/json

The framework owns the artifact model, schema, validation, and canonical encoding. Output encoders receive the serialized artifact through the generic output request. The JSON encoder owns only the opt-in decision, logical file placement, and output-index entry.

Configuration

Add one strict option to the production json output module:

pipelines:
  dnd-session:
    input: seriatim
    output:
      module: json
      options:
        include_chunk_map: true

include_chunk_map is a boolean and defaults to false. Unknown options and non-boolean values remain configuration errors. Do not add a separate CLI flag or top-level filesystem option: this choice changes the logical files produced by an output module, not their physical destination.

Other output encoders may ignore the available chunk-map artifact unless and until they define their own explicit export behavior.

Durable Artifact Contract

The payload is one strict JSON object with this shape:

{
  "source_id": "session-7",
  "source_digest": "sha256:0123456789abcdef...",
  "plan_digest": "sha256:abcdef0123456789...",
  "requested_chunker": "dnd/scenes",
  "producer": {
    "input_module": "seriatim",
    "chunk_module": "dnd/scenes",
    "llm_profile": "dnd-scenes"
  },
  "plan_annotations": {},
  "chunks": [
    {
      "id": "chunk-000001",
      "index": 0,
      "source_ref": {
        "source_id": "session-7",
        "start_unit_id": 1,
        "end_unit_id": 18
      },
      "unit_count": 18,
      "annotations": {
        "dnd/scenes": {
          "title": "At the city gate"
        }
      }
    }
  ]
}

Required top-level fields are:

• source_id: the accepted source document identity;
• source_digest: the canonical digest of that document;
• plan_digest: the canonical digest of the accepted chunk plan;
• requested_chunker: the chunk module selected by the current resolved pipeline;
• producer: the identity of the component that originally produced the accepted plan;
• plan_annotations: the accepted plan-level annotation namespace map; and
• chunks: the non-empty ordered list used for lane execution.

producer requires input_module and chunk_module. llm_profile is included only when the producing chunker was LLM-backed. Producer references, module metadata, warnings, creation timestamps, cache paths, and cache actions remain in their existing provenance and diagnostic surfaces; they are not copied into this artifact.

Each chunk requires:

• id: the stable materialized chunk ID;
• index: its zero-based position in execution order;
• source_ref: the exact inclusive current-source range;
• unit_count: the number of materialized source units in that range; and
• annotations: the accepted range-level annotation namespace map.

Unknown fields are rejected at every fixed object level. Annotation namespaces retain their canonical JSON values and may contain domain-specific JSON of any type. Empty annotation maps are encoded as {} so the shape remains explicit.

The durable artifact must guarantee that:

• source and module identities are non-empty and contain no surrounding whitespace;
• source and plan digests use the canonical sha256: representation;
• chunk IDs are non-empty and unique;
• chunk indexes are unique, contiguous, zero-based, and agree with array order;
• every chunk reference uses the top-level source identity;
• every range has positive endpoint IDs and was validated in source-document order when the artifact was constructed;
• every unit count is positive;
• annotations contain valid JSON under non-empty canonical namespaces and are canonicalized before digest calculation; and
• reconstructing the logical plan from the source digest, plan annotations, chunk ranges, and chunk annotations reproduces plan_digest.

The artifact represents chunk structure, not source content. It must not contain materialized units, transcript bytes, source-unit metadata, chunk content, private model responses, rejected boundary proposals, or debug payloads.

Accepted-State And Failure Policy

Construct the artifact only after plan materialization and the configured chunk validator chain have accepted the chunks. The runner passes the same immutable logical chunk identities, ranges, and annotations used for every lane; no second chunking or model call occurs.

When include_chunk_map is enabled:

• emit the artifact even if one or more later artifact lanes are rejected;
• omit it when chunk validation rejects the candidate map, because no accepted chunk map exists;
• retain the ordinary rejection record when it is omitted for that reason; and
• treat failure to construct, validate, serialize, index, or write an available accepted chunk map as a run failure, consistent with any explicitly requested durable output.

A cache hit and a newly generated plan with the same accepted logical plan must produce the same chunk ranges, annotations, plan digest, and materialized chunk identities. The artifact records both the current requested_chunker and the stored producer's chunk_module, since canonical plan reuse permits those identities to differ.

Output Bundle Integration

Extend index.json with an optional chunk_map descriptor:

{
  "chunk_map": {
    "artifact_kind": "source/chunk-map",
    "file": "chunk-map.json",
    "media_type": "application/json",
    "schema_id": "notarius.source.chunk_map",
    "schema_name": "notarius_source_chunk_map_v1",
    "schema_version": "v1"
  }
}

The descriptor and file are both absent when export is disabled or no accepted chunk map exists. The chunk map does not appear in lane-oriented output_files, because it is pipeline-wide and has no lane, extractor, merger, or normalizer identity.

The existing manifest remains the canonical run-provenance index. Its chunk_plan summary continues to own cache mode, lookup and publication actions, producer reference provenance, module metadata, timestamps, and validation status. The chunk-map artifact provides the accepted structure and only the minimal producer identities needed to interpret it independently.

Ownership And Architecture

Use a domain-neutral framework package for the durable chunk-map DTO, embedded JSON Schema, invariant validation, cloning, and canonical serialization. Do not marshal source.Chunk directly: its content, units, metadata, and internal fields are intentionally broader than this external contract.

Extend the output request with an optional cloned serialized chunk-map artifact. The runner constructs that value from the accepted chunk-plan execution result before invoking the output encoder. This keeps source and chunk-plan knowledge out of the generic JSON encoder and allows future encoders to consume the same framework-owned representation.

The JSON encoder adds the file and index descriptor only when its include_chunk_map option is true and the request contains an accepted artifact. It applies the same logical-path validation and pretty-printed JSON conventions as the rest of the bundle.

Security And Data Handling

Chunk annotations may contain model-derived or source-derived information. Treat chunk-map.json as durable user output with the same sensitivity and retention expectations as lane artifacts. Opt-in export prevents new durable content from appearing silently in existing pipelines.

Do not copy annotation values, source ranges, or chunk IDs into the manifest. Do not include external-reference content or filesystem paths in the chunk map. Existing output-directory confinement, atomic writing, and permission policy apply unchanged.

Quality And Documentation Policy

The framework contract must remain strict, canonical, immutable across ownership boundaries, and independent of source or D&D interpretation. The runner must preserve accepted-state and cache-producer provenance, while the JSON encoder must keep opt-in file selection separate from chunk semantics.

Once implemented, the durable schema belongs in docs/integrations/; configuration owns the selectable option; pipeline and state internals own the framework handoff; and a maintained scene-chunking example should demonstrate that namespaced annotations survive export. Tests should protect those observable contracts without relying on exact payload lengths or private helper structure.

Non-Goals

This scope does not:

• add a chunk-map extraction lane or artifact-lane registration;
• rerun chunking or ask an LLM to reconstruct accepted chunks;
• export transcript content, source units, source-unit metadata, or raw model proposals;
• interpret D&D scene annotations as a generic contract;
• add scene descriptions, titles, summaries, kinds, or participant extraction;
• make chunk maps generated references or inputs to later ordered steps;
• change chunk planning, validation, cache selection, or publication behavior;
• add a chunk-map import or replay mechanism;
• enable export by default; or
• introduce a general output-plugin capability negotiation system.