Document accepted chunk map export
This commit is contained in:
@@ -25,6 +25,7 @@ The explicit-path option is defined in the [CLI reference](cli.md).
|
||||
- [D&D combat-turn configuration](../examples/dnd-combat-turns.config.yml)
|
||||
- [D&D NPC-grounded spell and combat configuration](../examples/dnd-npc-grounded.config.yml)
|
||||
- [D&D NPC interaction configuration](../examples/dnd-npc-interactions.config.yml)
|
||||
- [D&D scene chunk-map configuration](../examples/dnd-scene-chunk-map.config.yml)
|
||||
|
||||
All are complete version 3 files. The fragments below illustrate individual
|
||||
fields and are not alternate complete configurations.
|
||||
@@ -331,6 +332,25 @@ During resolution, each selected module's registered option validator runs.
|
||||
Production input, chunk, and output bindings reject unknown or invalid options
|
||||
with the affected binding context.
|
||||
|
||||
### JSON Output Options
|
||||
|
||||
The `json` output module accepts only `include_chunk_map`, a boolean that
|
||||
defaults to `false`. When `true`, it adds the accepted pipeline-wide chunk map
|
||||
to the logical output bundle when one exists. It does not create a CLI flag or
|
||||
change output placement.
|
||||
|
||||
```yaml
|
||||
output:
|
||||
module: json
|
||||
options:
|
||||
include_chunk_map: true
|
||||
```
|
||||
|
||||
The payload and index descriptor are defined by the
|
||||
[Accepted Chunk Map contract](integrations/chunk-map.md). See the complete
|
||||
[D&D scene chunk-map configuration](../examples/dnd-scene-chunk-map.config.yml)
|
||||
for a copyable pipeline.
|
||||
|
||||
Validator bindings use the same shorthand or object module-binding form, but
|
||||
only these fields are supported:
|
||||
|
||||
@@ -363,7 +383,7 @@ production validators do not call the LLM and must not set `llm_profile`.
|
||||
| normalize | `dnd/npcs` | Deterministically consolidates typed D&D NPC-list artifacts by canonical name and unions exact evidence. |
|
||||
| normalize | `dnd/combat-turns` | Deterministically canonicalizes, orders, and de-duplicates typed D&D combat-turn artifacts. |
|
||||
| normalize | `dnd/npc-interactions` | Canonicalizes registry NPC names, orders interaction occurrences, and removes only exact duplicates. |
|
||||
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
|
||||
| output | `json` | Produces JSON output files for normalized `application/json` lanes and can opt in to an accepted chunk map. |
|
||||
|
||||
## Implemented Production Validators
|
||||
|
||||
|
||||
83
docs/integrations/chunk-map.md
Normal file
83
docs/integrations/chunk-map.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# Accepted Chunk Map
|
||||
|
||||
This document defines the durable accepted chunk-map artifact that the JSON
|
||||
output encoder can write as `chunk-map.json`. It describes the exact accepted,
|
||||
materialized chunks used by a run; it is not a lane artifact and is never an
|
||||
input to later pipeline steps. Enable it with the JSON output option described
|
||||
in [Configuration](../config.md#json-output-options).
|
||||
|
||||
## Identity
|
||||
|
||||
- Artifact kind: `source/chunk-map`
|
||||
- Logical file: `chunk-map.json`
|
||||
- Schema ID: `notarius.source.chunk_map`
|
||||
- Schema name: `notarius_source_chunk_map_v1`
|
||||
- Schema version: `v1`
|
||||
- Media type: `application/json`
|
||||
|
||||
The checked-in [JSON Schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json)
|
||||
defines the strict wire shape. `chunk-map.json` is listed by the optional
|
||||
`chunk_map` descriptor in [the JSON output index](json-output.md#indexjson),
|
||||
not by the lane-oriented `output_files` collection.
|
||||
|
||||
## Payload
|
||||
|
||||
The payload has these required fields:
|
||||
|
||||
- `source_id`: accepted source-document identity.
|
||||
- `source_digest`: canonical lower-case `sha256:` digest of that document.
|
||||
- `plan_digest`: canonical lower-case `sha256:` digest of the accepted logical
|
||||
plan.
|
||||
- `requested_chunker`: chunk module selected by the current resolved pipeline.
|
||||
- `producer`: the original accepted-plan producer, with required
|
||||
`input_module` and `chunk_module`; `llm_profile` is present only for an
|
||||
LLM-backed producer.
|
||||
- `plan_annotations`: accepted plan-level annotation namespace map. It is
|
||||
`{}` when no namespaces are present.
|
||||
- `chunks`: non-empty execution-order collection of accepted chunks.
|
||||
|
||||
Each chunk has `id`, zero-based `index`, `source_ref`, positive `unit_count`,
|
||||
and an explicit `annotations` namespace map. A source reference has the source
|
||||
ID and inclusive positive `start_unit_id` and `end_unit_id` endpoints.
|
||||
Annotation values are arbitrary valid JSON under non-empty namespaces. They
|
||||
are preserved as canonical JSON without interpreting any module-specific
|
||||
namespace.
|
||||
|
||||
## Invariants
|
||||
|
||||
The framework constructs this artifact only after materializing the selected
|
||||
logical plan and accepting it through the configured chunk validator chain.
|
||||
Construction proves the source and plan digests, source-document range order,
|
||||
materialized chunk IDs and indexes, source references, unit membership and
|
||||
counts, and plan/range annotations agree exactly. Chunk IDs are unique and
|
||||
indexes are contiguous and agree with array order.
|
||||
|
||||
The codec rejects unknown fixed-object fields, malformed identities or
|
||||
digests, invalid annotation JSON, trailing JSON content, and any payload whose
|
||||
reconstructed logical plan does not reproduce `plan_digest`. It makes
|
||||
defensive copies at serialization and decoding boundaries.
|
||||
|
||||
## Acceptance And Provenance
|
||||
|
||||
The artifact is available only when the chunk plan was accepted. It remains
|
||||
available when a later extraction, merge, or normalization result is rejected;
|
||||
it is absent when chunk validation rejects the candidate plan.
|
||||
|
||||
`requested_chunker` describes the current pipeline selection. `producer`
|
||||
describes who originally produced the accepted plan. On a cache hit these can
|
||||
differ: the accepted ranges, annotations, digests, and stable materialized IDs
|
||||
are reused, while the producer remains the stored producer. Cache paths,
|
||||
actions, references, metadata, warnings, timestamps, and detailed provenance
|
||||
remain in the run manifest rather than this payload.
|
||||
|
||||
## Data Handling
|
||||
|
||||
The map contains structure, not source content. It excludes transcript bytes,
|
||||
materialized units, source-unit metadata, chunk content, private model
|
||||
responses, rejected proposals, debug data, external-reference content, and
|
||||
filesystem paths.
|
||||
|
||||
Annotations can nevertheless be source- or model-derived. Treat an enabled
|
||||
`chunk-map.json` with the same sensitivity and retention expectations as lane
|
||||
output. Physical placement, confined atomic writing, and permissions follow
|
||||
the ordinary [output operation](../operations.md#output).
|
||||
@@ -18,6 +18,8 @@ The encoder writes:
|
||||
- `lanes/<lane-id>.json`, one file per normalized serialized artifact
|
||||
- `rejected.json`
|
||||
- `warnings.json`
|
||||
- `chunk-map.json`, only when the JSON output binding enables
|
||||
`include_chunk_map` and the run has an accepted chunk map
|
||||
|
||||
Files are pretty-printed JSON with a trailing newline when the payload is JSON.
|
||||
Logical file paths are relative, slash-separated, and may not contain `..`.
|
||||
@@ -60,6 +62,13 @@ contains the normalized payload `media_type`, normalizer `module_key`, and
|
||||
response `schema_id`, `schema_name`, and `schema_version` when those values are
|
||||
available.
|
||||
|
||||
When present, the top-level optional `chunk_map` descriptor contains exactly
|
||||
`artifact_kind`, `file`, `media_type`, `schema_id`, `schema_name`, and
|
||||
`schema_version`. It identifies the pipeline-wide `chunk-map.json`; it is not
|
||||
a lane output and never appears in `output_files`. The descriptor and file are
|
||||
both absent when export is disabled or no chunk plan was accepted. Its payload
|
||||
contract is defined by [Accepted Chunk Map](chunk-map.md).
|
||||
|
||||
## `manifest.json`
|
||||
|
||||
`manifest.json` contains a run manifest. This abridged example shows its core
|
||||
|
||||
@@ -391,6 +391,12 @@ safe logical names, pretty-prints JSON payloads, and assembles the logical index
|
||||
manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
|
||||
media types, unsafe names, and sanitized-name collisions are errors.
|
||||
|
||||
Its strict `include_chunk_map` option is disabled by default. When enabled, it
|
||||
validates the framework-supplied accepted chunk map through its codec and adds
|
||||
the pipeline-wide `chunk-map.json` plus its index descriptor; it does not treat
|
||||
the map as a lane payload. The external shape is owned by the
|
||||
[Accepted Chunk Map contract](../integrations/chunk-map.md).
|
||||
|
||||
The encoder returns logical files only. The CLI places them on disk, and the
|
||||
[JSON output contract](../integrations/json-output.md) defines their external
|
||||
paths and schemas.
|
||||
|
||||
@@ -50,6 +50,7 @@ required normalized artifacts have crossed the typed handoff.
|
||||
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
|
||||
| `internal/framework/checkpoint` | Root-based checkpoint loading, recording, identity, and payload serialization. |
|
||||
| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
|
||||
| `internal/framework/chunkmap` | Strict durable accepted chunk-map construction, schema, validation, cloning, and serialization. |
|
||||
| `internal/framework/debug` | Root-based framework and LLM debug recording. |
|
||||
|
||||
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
|
||||
@@ -109,7 +110,7 @@ Configuration. The implemented module packages are:
|
||||
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
|
||||
| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
|
||||
| `internal/modules/dnd/normalize/npcs` | Consolidates NPC records deterministically by canonical name, unions exact evidence, and reports bounded warnings. |
|
||||
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
|
||||
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, rejections, and an explicitly enabled accepted chunk map as logical JSON files. |
|
||||
|
||||
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
|
||||
reference declarations, prompt input assembly, source-unit reference helpers,
|
||||
|
||||
@@ -208,15 +208,18 @@ The runner:
|
||||
3. selects a stored plan or executes the configured chunker's `Plan` operation;
|
||||
4. canonicalizes and materializes the plan, then validates the resulting
|
||||
chunks;
|
||||
5. executes each resolved step in configuration order. For one step, it
|
||||
5. builds the framework-owned accepted chunk map from the accepted source,
|
||||
logical plan, and exact materialized chunks, then supplies it to the output
|
||||
request independently of output-module options;
|
||||
6. executes each resolved step in configuration order. For one step, it
|
||||
dispatches extract jobs in source-chunk then resolved-lane order, starts a
|
||||
bounded lane continuation when all extracts for that lane are terminal, and
|
||||
waits for every lane to become terminal;
|
||||
6. encodes and validates each accepted normalized producer artifact, then
|
||||
7. encodes and validates each accepted normalized producer artifact, then
|
||||
builds the immutable generated reference sets for the next step;
|
||||
7. invokes the prepared output encoder only after every step succeeds and
|
||||
8. invokes the prepared output encoder only after every step succeeds and
|
||||
validates its logical file results;
|
||||
8. returns the assembled manifest, outcomes, warnings, and files.
|
||||
9. returns the assembled manifest, outcomes, warnings, and files.
|
||||
|
||||
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
|
||||
and validators while performing these transitions:
|
||||
@@ -265,6 +268,14 @@ currently requested chunker and the effective plan producer. Cache state and
|
||||
paths are configured and operated outside the runner; see
|
||||
[Configuration](../config.md#state-surfaces) and [Operations](../operations.md).
|
||||
|
||||
For an accepted plan, the runner also constructs the strict framework-owned
|
||||
[Accepted Chunk Map](../integrations/chunk-map.md) before lane execution. It
|
||||
uses the current resolved chunker as `requested_chunker` and the stored or
|
||||
generated record as `producer`, preserving that distinction on reuse. Chunk
|
||||
rejection supplies no map; later lane rejection does not discard it. Output
|
||||
encoders receive a defensively owned serialized value and may explicitly
|
||||
ignore it.
|
||||
|
||||
The extract job channel has the same capacity as the effective extract worker
|
||||
count, so dispatch applies backpressure. A fixed continuation executor prevents
|
||||
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
|
||||
|
||||
@@ -17,6 +17,11 @@ debug roots.
|
||||
The pipeline runner returns logical output files. After validating every
|
||||
logical name, the CLI exclusively creates the run directory beneath the
|
||||
selected output root and performs confined, atomic file writes within it.
|
||||
The runner supplies an accepted chunk map as an optional, defensively owned
|
||||
output-request artifact. The JSON encoder alone decides whether its explicit
|
||||
option writes the map and optional index descriptor; neither the map payload
|
||||
nor its annotations are copied into the run manifest. The durable fields are
|
||||
owned by the [Accepted Chunk Map contract](../integrations/chunk-map.md).
|
||||
|
||||
`internal/framework/chunkplan` owns source-addressed plan storage, validation,
|
||||
and atomic publication. Its store is constructed only when the selected mode is
|
||||
|
||||
@@ -37,6 +37,12 @@ automatically removes output. The
|
||||
[JSON output contract](integrations/json-output.md) owns the logical file
|
||||
names, schemas, and media types inside a run directory.
|
||||
|
||||
An enabled JSON `include_chunk_map` option adds an accepted chunk map to durable
|
||||
output. Its annotations may contain source- or model-derived data, so retain
|
||||
and protect it like lane output. The map is opt-in and does not alter existing
|
||||
bundles; its payload exclusions are defined in the
|
||||
[Accepted Chunk Map contract](integrations/chunk-map.md).
|
||||
|
||||
Remove an output run directory only after its consumer data is no longer
|
||||
needed. This is data deletion, not cache cleanup.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Accepted Chunk Map Export
|
||||
|
||||
Status: Accepted
|
||||
Status: Implemented
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
@@ -16,13 +16,6 @@ not as committed release dates.
|
||||
validator, and normalizer development. Treat model-quality review as an
|
||||
iterative human evaluation aid, not a deterministic correctness gate.
|
||||
|
||||
### Export Accepted Chunk Maps
|
||||
|
||||
Add an opt-in, framework-owned durable artifact for the exact accepted
|
||||
materialized chunks used by lane execution. The accepted contract, output
|
||||
boundary, provenance policy, and exclusions are defined in
|
||||
[Accepted Chunk Map Export](accepted-chunk-map-export.md).
|
||||
|
||||
### Extract D&D Scene Descriptions
|
||||
|
||||
- Add a `dnd/scene-descriptions` extractor that runs once for each accepted
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Accepted Chunk Map Export Implementation Plan
|
||||
|
||||
Status: Ready for implementation
|
||||
Status: Completed
|
||||
|
||||
## Objective
|
||||
|
||||
|
||||
Reference in New Issue
Block a user