Harmonize foundational integration contracts

This commit is contained in:
2026-07-26 13:38:55 +00:00
parent fdf8c4afd4
commit 2f506f4985
4 changed files with 252 additions and 338 deletions

View File

@@ -1,83 +1,81 @@
# Accepted Chunk Map
This document defines the durable accepted chunk-map artifact that the JSON
output encoder can write as `chunk-map.json`. It describes the exact accepted,
materialized chunks used by a run; it is not a lane artifact and is never an
input to later pipeline steps. Enable it with the JSON output option described
in [Configuration](../config.md#json-output-options).
This document defines the optional durable `chunk-map.json` artifact in a
[published JSON bundle](json-output.md). It describes the accepted,
materialized chunk plan used by one run. It is not a lane payload and is never
an input to a later pipeline step.
## Identity
## Contract Identity
- Artifact kind: `source/chunk-map`
- Logical file: `chunk-map.json`
- Schema ID: `notarius.source.chunk_map`
- Schema name: `notarius_source_chunk_map_v1`
- Schema version: `v1`
- Media type: `application/json`
| Property | Value |
| --- | --- |
| Artifact kind | `source/chunk-map` |
| Logical file | `chunk-map.json` |
| Media type | `application/json` |
| Schema ID | `notarius.source.chunk_map` |
| Schema name | `notarius_source_chunk_map_v1` |
| Schema version | `v1` |
The checked-in [JSON Schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json)
defines the strict wire shape. `chunk-map.json` is listed by the optional
`chunk_map` descriptor in [the JSON output index](json-output.md#indexjson),
not by the lane-oriented `output_files` collection.
The optional `chunk_map` descriptor in `index.json` identifies this artifact.
Export is controlled by the JSON output binding described in
[Configuration](../config.md#module-bindings-and-validators).
## Payload
## Wire Shape
The payload has these required fields:
Every payload has these required fields:
- `source_id`: accepted source-document identity.
- `source_digest`: canonical lower-case `sha256:` digest of that document.
- `plan_digest`: canonical lower-case `sha256:` digest of the accepted logical
plan.
- `requested_chunker`: chunk module selected by the current resolved pipeline.
- `producer`: the original accepted-plan producer, with required
`input_module` and `chunk_module`; `llm_profile` is present only for an
LLM-backed producer.
- `plan_annotations`: accepted plan-level annotation namespace map. It is
`{}` when no namespaces are present.
- `chunks`: non-empty execution-order collection of accepted chunks.
| Field | Meaning |
| --- | --- |
| `source_id` | Accepted source-document identity. |
| `source_digest` | Lower-case `sha256:` digest of that source document. |
| `plan_digest` | Lower-case `sha256:` digest of the logical chunk plan. |
| `requested_chunker` | Chunk module selected by the resolved pipeline. |
| `producer` | Original accepted-plan producer. `input_module` and `chunk_module` are required; `llm_profile` is optional. |
| `plan_annotations` | Plan-level annotation namespace map; `{}` when none are present. |
| `chunks` | Non-empty execution-order chunk collection. |
Each chunk has `id`, zero-based `index`, `source_ref`, positive `unit_count`,
and an explicit `annotations` namespace map. A source reference has the source
ID and inclusive positive `start_unit_id` and `end_unit_id` endpoints.
Annotation values are arbitrary valid JSON under non-empty namespaces. They
are preserved as canonical JSON without interpreting any module-specific
namespace.
Each `chunks` entry contains non-empty `id`, zero-based `index`, `source_ref`,
positive `unit_count`, and an explicit `annotations` map. `source_ref` contains
the same `source_id` as the top-level value plus positive inclusive
`start_unit_id` and `end_unit_id` values. Endpoints identify source units; their
numeric values do not by themselves establish source-document order.
## Invariants
Annotation namespaces are non-empty trimmed strings. Their values are arbitrary
valid JSON and are retained without interpreting a module-specific namespace.
The framework constructs this artifact only after materializing the selected
logical plan and accepting it through the configured chunk validator chain.
Construction proves the source and plan digests, source-document range order,
materialized chunk IDs and indexes, source references, unit membership and
counts, and plan/range annotations agree exactly. Chunk IDs are unique and
indexes are contiguous and agree with array order.
## Ordering And Validation
The codec rejects unknown fixed-object fields, malformed identities or
digests, invalid annotation JSON, trailing JSON content, and any payload whose
reconstructed logical plan does not reproduce `plan_digest`. It makes
defensive copies at serialization and decoding boundaries.
`chunks` are in execution order. Their indexes are contiguous, start at zero,
and equal their array positions; chunk IDs are unique. The emitted map is built
only after the selected plan has been accepted and materialized against the
source document, so its ranges, unit counts, annotations, and digests describe
that exact plan.
## Acceptance And Provenance
The codec rejects malformed JSON, trailing content, unknown fixed-object
fields, invalid identities or digests, invalid annotations, duplicate chunk
IDs, non-contiguous indexes, and a `plan_digest` that does not match the
reconstructed logical plan. The checked-in
[schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json)
defines the strict JSON shape.
The artifact is available only when the chunk plan was accepted. It remains
available when a later extraction, merge, or normalization result is rejected;
it is absent when chunk validation rejects the candidate plan.
## Valid Example
`requested_chunker` describes the current pipeline selection. `producer`
describes who originally produced the accepted plan. On a cache hit these can
differ: the accepted ranges, annotations, digests, and stable materialized IDs
are reused, while the producer remains the stored producer. Cache paths,
actions, references, metadata, warnings, timestamps, and detailed provenance
remain in the run manifest rather than this payload.
The compact
[source chunk-map fixture](../../internal/framework/chunkmap/testdata/source_chunk_map.v1.json)
is decoded by the production codec and demonstrates an accepted map with
annotations, producer identity, and ordered chunks.
## Data Handling
## Publication And Compatibility
The map contains structure, not source content. It excludes transcript bytes,
materialized units, source-unit metadata, chunk content, private model
responses, rejected proposals, debug data, external-reference content, and
filesystem paths.
The map is present only when a chunk plan was accepted and its export is
enabled. It remains publishable if a later lane is rejected, but is absent when
chunk-plan validation rejects the plan. `requested_chunker` identifies the
current pipeline selection, while `producer` identifies the component that
originally produced the accepted plan; they may differ when an accepted plan is
reused.
Annotations can nevertheless be source- or model-derived. Treat an enabled
`chunk-map.json` with the same sensitivity and retention expectations as lane
output. Physical placement, confined atomic writing, and permissions follow
the ordinary [output operation](../operations.md#output).
The map contains structure rather than source content: it excludes transcript
bytes, source-unit metadata, chunk text, private model output, reference
content, debug data, and filesystem paths. Treat the exported map with the
same care as other published output. Publication location and retention are
defined in [Operations](../operations.md#output-bundles).