Document accepted chunk map export
This commit is contained in:
@@ -391,6 +391,12 @@ safe logical names, pretty-prints JSON payloads, and assembles the logical index
|
||||
manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
|
||||
media types, unsafe names, and sanitized-name collisions are errors.
|
||||
|
||||
Its strict `include_chunk_map` option is disabled by default. When enabled, it
|
||||
validates the framework-supplied accepted chunk map through its codec and adds
|
||||
the pipeline-wide `chunk-map.json` plus its index descriptor; it does not treat
|
||||
the map as a lane payload. The external shape is owned by the
|
||||
[Accepted Chunk Map contract](../integrations/chunk-map.md).
|
||||
|
||||
The encoder returns logical files only. The CLI places them on disk, and the
|
||||
[JSON output contract](../integrations/json-output.md) defines their external
|
||||
paths and schemas.
|
||||
|
||||
@@ -50,6 +50,7 @@ required normalized artifacts have crossed the typed handoff.
|
||||
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
|
||||
| `internal/framework/checkpoint` | Root-based checkpoint loading, recording, identity, and payload serialization. |
|
||||
| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
|
||||
| `internal/framework/chunkmap` | Strict durable accepted chunk-map construction, schema, validation, cloning, and serialization. |
|
||||
| `internal/framework/debug` | Root-based framework and LLM debug recording. |
|
||||
|
||||
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
|
||||
@@ -109,7 +110,7 @@ Configuration. The implemented module packages are:
|
||||
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
|
||||
| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
|
||||
| `internal/modules/dnd/normalize/npcs` | Consolidates NPC records deterministically by canonical name, unions exact evidence, and reports bounded warnings. |
|
||||
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
|
||||
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, rejections, and an explicitly enabled accepted chunk map as logical JSON files. |
|
||||
|
||||
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
|
||||
reference declarations, prompt input assembly, source-unit reference helpers,
|
||||
|
||||
@@ -208,15 +208,18 @@ The runner:
|
||||
3. selects a stored plan or executes the configured chunker's `Plan` operation;
|
||||
4. canonicalizes and materializes the plan, then validates the resulting
|
||||
chunks;
|
||||
5. executes each resolved step in configuration order. For one step, it
|
||||
5. builds the framework-owned accepted chunk map from the accepted source,
|
||||
logical plan, and exact materialized chunks, then supplies it to the output
|
||||
request independently of output-module options;
|
||||
6. executes each resolved step in configuration order. For one step, it
|
||||
dispatches extract jobs in source-chunk then resolved-lane order, starts a
|
||||
bounded lane continuation when all extracts for that lane are terminal, and
|
||||
waits for every lane to become terminal;
|
||||
6. encodes and validates each accepted normalized producer artifact, then
|
||||
7. encodes and validates each accepted normalized producer artifact, then
|
||||
builds the immutable generated reference sets for the next step;
|
||||
7. invokes the prepared output encoder only after every step succeeds and
|
||||
8. invokes the prepared output encoder only after every step succeeds and
|
||||
validates its logical file results;
|
||||
8. returns the assembled manifest, outcomes, warnings, and files.
|
||||
9. returns the assembled manifest, outcomes, warnings, and files.
|
||||
|
||||
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
|
||||
and validators while performing these transitions:
|
||||
@@ -265,6 +268,14 @@ currently requested chunker and the effective plan producer. Cache state and
|
||||
paths are configured and operated outside the runner; see
|
||||
[Configuration](../config.md#state-surfaces) and [Operations](../operations.md).
|
||||
|
||||
For an accepted plan, the runner also constructs the strict framework-owned
|
||||
[Accepted Chunk Map](../integrations/chunk-map.md) before lane execution. It
|
||||
uses the current resolved chunker as `requested_chunker` and the stored or
|
||||
generated record as `producer`, preserving that distinction on reuse. Chunk
|
||||
rejection supplies no map; later lane rejection does not discard it. Output
|
||||
encoders receive a defensively owned serialized value and may explicitly
|
||||
ignore it.
|
||||
|
||||
The extract job channel has the same capacity as the effective extract worker
|
||||
count, so dispatch applies backpressure. A fixed continuation executor prevents
|
||||
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
|
||||
|
||||
@@ -17,6 +17,11 @@ debug roots.
|
||||
The pipeline runner returns logical output files. After validating every
|
||||
logical name, the CLI exclusively creates the run directory beneath the
|
||||
selected output root and performs confined, atomic file writes within it.
|
||||
The runner supplies an accepted chunk map as an optional, defensively owned
|
||||
output-request artifact. The JSON encoder alone decides whether its explicit
|
||||
option writes the map and optional index descriptor; neither the map payload
|
||||
nor its annotations are copied into the run manifest. The durable fields are
|
||||
owned by the [Accepted Chunk Map contract](../integrations/chunk-map.md).
|
||||
|
||||
`internal/framework/chunkplan` owns source-addressed plan storage, validation,
|
||||
and atomic publication. Its store is constructed only when the selected mode is
|
||||
|
||||
Reference in New Issue
Block a user