Document accepted chunk map export

This commit is contained in:
2026-07-23 15:06:33 +00:00
parent 16a998055c
commit 06148074a2
13 changed files with 228 additions and 15 deletions

View File

@@ -25,6 +25,7 @@ The explicit-path option is defined in the [CLI reference](cli.md).
- [D&D combat-turn configuration](../examples/dnd-combat-turns.config.yml)
- [D&D NPC-grounded spell and combat configuration](../examples/dnd-npc-grounded.config.yml)
- [D&D NPC interaction configuration](../examples/dnd-npc-interactions.config.yml)
- [D&D scene chunk-map configuration](../examples/dnd-scene-chunk-map.config.yml)
All are complete version 3 files. The fragments below illustrate individual
fields and are not alternate complete configurations.
@@ -331,6 +332,25 @@ During resolution, each selected module's registered option validator runs.
Production input, chunk, and output bindings reject unknown or invalid options
with the affected binding context.
### JSON Output Options
The `json` output module accepts only `include_chunk_map`, a boolean that
defaults to `false`. When `true`, it adds the accepted pipeline-wide chunk map
to the logical output bundle when one exists. It does not create a CLI flag or
change output placement.
```yaml
output:
module: json
options:
include_chunk_map: true
```
The payload and index descriptor are defined by the
[Accepted Chunk Map contract](integrations/chunk-map.md). See the complete
[D&D scene chunk-map configuration](../examples/dnd-scene-chunk-map.config.yml)
for a copyable pipeline.
Validator bindings use the same shorthand or object module-binding form, but
only these fields are supported:
@@ -363,7 +383,7 @@ production validators do not call the LLM and must not set `llm_profile`.
| normalize | `dnd/npcs` | Deterministically consolidates typed D&D NPC-list artifacts by canonical name and unions exact evidence. |
| normalize | `dnd/combat-turns` | Deterministically canonicalizes, orders, and de-duplicates typed D&D combat-turn artifacts. |
| normalize | `dnd/npc-interactions` | Canonicalizes registry NPC names, orders interaction occurrences, and removes only exact duplicates. |
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
| output | `json` | Produces JSON output files for normalized `application/json` lanes and can opt in to an accepted chunk map. |
## Implemented Production Validators

View File

@@ -0,0 +1,83 @@
# Accepted Chunk Map
This document defines the durable accepted chunk-map artifact that the JSON
output encoder can write as `chunk-map.json`. It describes the exact accepted,
materialized chunks used by a run; it is not a lane artifact and is never an
input to later pipeline steps. Enable it with the JSON output option described
in [Configuration](../config.md#json-output-options).
## Identity
- Artifact kind: `source/chunk-map`
- Logical file: `chunk-map.json`
- Schema ID: `notarius.source.chunk_map`
- Schema name: `notarius_source_chunk_map_v1`
- Schema version: `v1`
- Media type: `application/json`
The checked-in [JSON Schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json)
defines the strict wire shape. `chunk-map.json` is listed by the optional
`chunk_map` descriptor in [the JSON output index](json-output.md#indexjson),
not by the lane-oriented `output_files` collection.
## Payload
The payload has these required fields:
- `source_id`: accepted source-document identity.
- `source_digest`: canonical lower-case `sha256:` digest of that document.
- `plan_digest`: canonical lower-case `sha256:` digest of the accepted logical
plan.
- `requested_chunker`: chunk module selected by the current resolved pipeline.
- `producer`: the original accepted-plan producer, with required
`input_module` and `chunk_module`; `llm_profile` is present only for an
LLM-backed producer.
- `plan_annotations`: accepted plan-level annotation namespace map. It is
`{}` when no namespaces are present.
- `chunks`: non-empty execution-order collection of accepted chunks.
Each chunk has `id`, zero-based `index`, `source_ref`, positive `unit_count`,
and an explicit `annotations` namespace map. A source reference has the source
ID and inclusive positive `start_unit_id` and `end_unit_id` endpoints.
Annotation values are arbitrary valid JSON under non-empty namespaces. They
are preserved as canonical JSON without interpreting any module-specific
namespace.
## Invariants
The framework constructs this artifact only after materializing the selected
logical plan and accepting it through the configured chunk validator chain.
Construction proves the source and plan digests, source-document range order,
materialized chunk IDs and indexes, source references, unit membership and
counts, and plan/range annotations agree exactly. Chunk IDs are unique and
indexes are contiguous and agree with array order.
The codec rejects unknown fixed-object fields, malformed identities or
digests, invalid annotation JSON, trailing JSON content, and any payload whose
reconstructed logical plan does not reproduce `plan_digest`. It makes
defensive copies at serialization and decoding boundaries.
## Acceptance And Provenance
The artifact is available only when the chunk plan was accepted. It remains
available when a later extraction, merge, or normalization result is rejected;
it is absent when chunk validation rejects the candidate plan.
`requested_chunker` describes the current pipeline selection. `producer`
describes who originally produced the accepted plan. On a cache hit these can
differ: the accepted ranges, annotations, digests, and stable materialized IDs
are reused, while the producer remains the stored producer. Cache paths,
actions, references, metadata, warnings, timestamps, and detailed provenance
remain in the run manifest rather than this payload.
## Data Handling
The map contains structure, not source content. It excludes transcript bytes,
materialized units, source-unit metadata, chunk content, private model
responses, rejected proposals, debug data, external-reference content, and
filesystem paths.
Annotations can nevertheless be source- or model-derived. Treat an enabled
`chunk-map.json` with the same sensitivity and retention expectations as lane
output. Physical placement, confined atomic writing, and permissions follow
the ordinary [output operation](../operations.md#output).

View File

@@ -18,6 +18,8 @@ The encoder writes:
- `lanes/<lane-id>.json`, one file per normalized serialized artifact
- `rejected.json`
- `warnings.json`
- `chunk-map.json`, only when the JSON output binding enables
`include_chunk_map` and the run has an accepted chunk map
Files are pretty-printed JSON with a trailing newline when the payload is JSON.
Logical file paths are relative, slash-separated, and may not contain `..`.
@@ -60,6 +62,13 @@ contains the normalized payload `media_type`, normalizer `module_key`, and
response `schema_id`, `schema_name`, and `schema_version` when those values are
available.
When present, the top-level optional `chunk_map` descriptor contains exactly
`artifact_kind`, `file`, `media_type`, `schema_id`, `schema_name`, and
`schema_version`. It identifies the pipeline-wide `chunk-map.json`; it is not
a lane output and never appears in `output_files`. The descriptor and file are
both absent when export is disabled or no chunk plan was accepted. Its payload
contract is defined by [Accepted Chunk Map](chunk-map.md).
## `manifest.json`
`manifest.json` contains a run manifest. This abridged example shows its core

View File

@@ -391,6 +391,12 @@ safe logical names, pretty-prints JSON payloads, and assembles the logical index
manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
media types, unsafe names, and sanitized-name collisions are errors.
Its strict `include_chunk_map` option is disabled by default. When enabled, it
validates the framework-supplied accepted chunk map through its codec and adds
the pipeline-wide `chunk-map.json` plus its index descriptor; it does not treat
the map as a lane payload. The external shape is owned by the
[Accepted Chunk Map contract](../integrations/chunk-map.md).
The encoder returns logical files only. The CLI places them on disk, and the
[JSON output contract](../integrations/json-output.md) defines their external
paths and schemas.

View File

@@ -50,6 +50,7 @@ required normalized artifacts have crossed the typed handoff.
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
| `internal/framework/checkpoint` | Root-based checkpoint loading, recording, identity, and payload serialization. |
| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
| `internal/framework/chunkmap` | Strict durable accepted chunk-map construction, schema, validation, cloning, and serialization. |
| `internal/framework/debug` | Root-based framework and LLM debug recording. |
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
@@ -109,7 +110,7 @@ Configuration. The implemented module packages are:
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
| `internal/modules/dnd/normalize/npcs` | Consolidates NPC records deterministically by canonical name, unions exact evidence, and reports bounded warnings. |
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, rejections, and an explicitly enabled accepted chunk map as logical JSON files. |
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
reference declarations, prompt input assembly, source-unit reference helpers,

View File

@@ -208,15 +208,18 @@ The runner:
3. selects a stored plan or executes the configured chunker's `Plan` operation;
4. canonicalizes and materializes the plan, then validates the resulting
chunks;
5. executes each resolved step in configuration order. For one step, it
5. builds the framework-owned accepted chunk map from the accepted source,
logical plan, and exact materialized chunks, then supplies it to the output
request independently of output-module options;
6. executes each resolved step in configuration order. For one step, it
dispatches extract jobs in source-chunk then resolved-lane order, starts a
bounded lane continuation when all extracts for that lane are terminal, and
waits for every lane to become terminal;
6. encodes and validates each accepted normalized producer artifact, then
7. encodes and validates each accepted normalized producer artifact, then
builds the immutable generated reference sets for the next step;
7. invokes the prepared output encoder only after every step succeeds and
8. invokes the prepared output encoder only after every step succeeds and
validates its logical file results;
8. returns the assembled manifest, outcomes, warnings, and files.
9. returns the assembled manifest, outcomes, warnings, and files.
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
and validators while performing these transitions:
@@ -265,6 +268,14 @@ currently requested chunker and the effective plan producer. Cache state and
paths are configured and operated outside the runner; see
[Configuration](../config.md#state-surfaces) and [Operations](../operations.md).
For an accepted plan, the runner also constructs the strict framework-owned
[Accepted Chunk Map](../integrations/chunk-map.md) before lane execution. It
uses the current resolved chunker as `requested_chunker` and the stored or
generated record as `producer`, preserving that distinction on reuse. Chunk
rejection supplies no map; later lane rejection does not discard it. Output
encoders receive a defensively owned serialized value and may explicitly
ignore it.
The extract job channel has the same capacity as the effective extract worker
count, so dispatch applies backpressure. A fixed continuation executor prevents
ready or checkpoint-reused lanes from creating one goroutine each. Workers and

View File

@@ -17,6 +17,11 @@ debug roots.
The pipeline runner returns logical output files. After validating every
logical name, the CLI exclusively creates the run directory beneath the
selected output root and performs confined, atomic file writes within it.
The runner supplies an accepted chunk map as an optional, defensively owned
output-request artifact. The JSON encoder alone decides whether its explicit
option writes the map and optional index descriptor; neither the map payload
nor its annotations are copied into the run manifest. The durable fields are
owned by the [Accepted Chunk Map contract](../integrations/chunk-map.md).
`internal/framework/chunkplan` owns source-addressed plan storage, validation,
and atomic publication. Its store is constructed only when the selected mode is

View File

@@ -37,6 +37,12 @@ automatically removes output. The
[JSON output contract](integrations/json-output.md) owns the logical file
names, schemas, and media types inside a run directory.
An enabled JSON `include_chunk_map` option adds an accepted chunk map to durable
output. Its annotations may contain source- or model-derived data, so retain
and protect it like lane output. The map is opt-in and does not alter existing
bundles; its payload exclusions are defined in the
[Accepted Chunk Map contract](integrations/chunk-map.md).
Remove an output run directory only after its consumer data is no longer
needed. This is data deletion, not cache cleanup.

View File

@@ -1,6 +1,6 @@
# Accepted Chunk Map Export
Status: Accepted
Status: Implemented
## Purpose

View File

@@ -16,13 +16,6 @@ not as committed release dates.
validator, and normalizer development. Treat model-quality review as an
iterative human evaluation aid, not a deterministic correctness gate.
### Export Accepted Chunk Maps
Add an opt-in, framework-owned durable artifact for the exact accepted
materialized chunks used by lane execution. The accepted contract, output
boundary, provenance policy, and exclusions are defined in
[Accepted Chunk Map Export](accepted-chunk-map-export.md).
### Extract D&D Scene Descriptions
- Add a `dnd/scene-descriptions` extractor that runs once for each accepted

View File

@@ -1,6 +1,6 @@
# Accepted Chunk Map Export Implementation Plan
Status: Ready for implementation
Status: Completed
## Objective