Harmonize foundational integration contracts
This commit is contained in:
@@ -1,83 +1,81 @@
|
||||
# Accepted Chunk Map
|
||||
|
||||
This document defines the durable accepted chunk-map artifact that the JSON
|
||||
output encoder can write as `chunk-map.json`. It describes the exact accepted,
|
||||
materialized chunks used by a run; it is not a lane artifact and is never an
|
||||
input to later pipeline steps. Enable it with the JSON output option described
|
||||
in [Configuration](../config.md#json-output-options).
|
||||
This document defines the optional durable `chunk-map.json` artifact in a
|
||||
[published JSON bundle](json-output.md). It describes the accepted,
|
||||
materialized chunk plan used by one run. It is not a lane payload and is never
|
||||
an input to a later pipeline step.
|
||||
|
||||
## Identity
|
||||
## Contract Identity
|
||||
|
||||
- Artifact kind: `source/chunk-map`
|
||||
- Logical file: `chunk-map.json`
|
||||
- Schema ID: `notarius.source.chunk_map`
|
||||
- Schema name: `notarius_source_chunk_map_v1`
|
||||
- Schema version: `v1`
|
||||
- Media type: `application/json`
|
||||
| Property | Value |
|
||||
| --- | --- |
|
||||
| Artifact kind | `source/chunk-map` |
|
||||
| Logical file | `chunk-map.json` |
|
||||
| Media type | `application/json` |
|
||||
| Schema ID | `notarius.source.chunk_map` |
|
||||
| Schema name | `notarius_source_chunk_map_v1` |
|
||||
| Schema version | `v1` |
|
||||
|
||||
The checked-in [JSON Schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json)
|
||||
defines the strict wire shape. `chunk-map.json` is listed by the optional
|
||||
`chunk_map` descriptor in [the JSON output index](json-output.md#indexjson),
|
||||
not by the lane-oriented `output_files` collection.
|
||||
The optional `chunk_map` descriptor in `index.json` identifies this artifact.
|
||||
Export is controlled by the JSON output binding described in
|
||||
[Configuration](../config.md#module-bindings-and-validators).
|
||||
|
||||
## Payload
|
||||
## Wire Shape
|
||||
|
||||
The payload has these required fields:
|
||||
Every payload has these required fields:
|
||||
|
||||
- `source_id`: accepted source-document identity.
|
||||
- `source_digest`: canonical lower-case `sha256:` digest of that document.
|
||||
- `plan_digest`: canonical lower-case `sha256:` digest of the accepted logical
|
||||
plan.
|
||||
- `requested_chunker`: chunk module selected by the current resolved pipeline.
|
||||
- `producer`: the original accepted-plan producer, with required
|
||||
`input_module` and `chunk_module`; `llm_profile` is present only for an
|
||||
LLM-backed producer.
|
||||
- `plan_annotations`: accepted plan-level annotation namespace map. It is
|
||||
`{}` when no namespaces are present.
|
||||
- `chunks`: non-empty execution-order collection of accepted chunks.
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `source_id` | Accepted source-document identity. |
|
||||
| `source_digest` | Lower-case `sha256:` digest of that source document. |
|
||||
| `plan_digest` | Lower-case `sha256:` digest of the logical chunk plan. |
|
||||
| `requested_chunker` | Chunk module selected by the resolved pipeline. |
|
||||
| `producer` | Original accepted-plan producer. `input_module` and `chunk_module` are required; `llm_profile` is optional. |
|
||||
| `plan_annotations` | Plan-level annotation namespace map; `{}` when none are present. |
|
||||
| `chunks` | Non-empty execution-order chunk collection. |
|
||||
|
||||
Each chunk has `id`, zero-based `index`, `source_ref`, positive `unit_count`,
|
||||
and an explicit `annotations` namespace map. A source reference has the source
|
||||
ID and inclusive positive `start_unit_id` and `end_unit_id` endpoints.
|
||||
Annotation values are arbitrary valid JSON under non-empty namespaces. They
|
||||
are preserved as canonical JSON without interpreting any module-specific
|
||||
namespace.
|
||||
Each `chunks` entry contains non-empty `id`, zero-based `index`, `source_ref`,
|
||||
positive `unit_count`, and an explicit `annotations` map. `source_ref` contains
|
||||
the same `source_id` as the top-level value plus positive inclusive
|
||||
`start_unit_id` and `end_unit_id` values. Endpoints identify source units; their
|
||||
numeric values do not by themselves establish source-document order.
|
||||
|
||||
## Invariants
|
||||
Annotation namespaces are non-empty trimmed strings. Their values are arbitrary
|
||||
valid JSON and are retained without interpreting a module-specific namespace.
|
||||
|
||||
The framework constructs this artifact only after materializing the selected
|
||||
logical plan and accepting it through the configured chunk validator chain.
|
||||
Construction proves the source and plan digests, source-document range order,
|
||||
materialized chunk IDs and indexes, source references, unit membership and
|
||||
counts, and plan/range annotations agree exactly. Chunk IDs are unique and
|
||||
indexes are contiguous and agree with array order.
|
||||
## Ordering And Validation
|
||||
|
||||
The codec rejects unknown fixed-object fields, malformed identities or
|
||||
digests, invalid annotation JSON, trailing JSON content, and any payload whose
|
||||
reconstructed logical plan does not reproduce `plan_digest`. It makes
|
||||
defensive copies at serialization and decoding boundaries.
|
||||
`chunks` are in execution order. Their indexes are contiguous, start at zero,
|
||||
and equal their array positions; chunk IDs are unique. The emitted map is built
|
||||
only after the selected plan has been accepted and materialized against the
|
||||
source document, so its ranges, unit counts, annotations, and digests describe
|
||||
that exact plan.
|
||||
|
||||
## Acceptance And Provenance
|
||||
The codec rejects malformed JSON, trailing content, unknown fixed-object
|
||||
fields, invalid identities or digests, invalid annotations, duplicate chunk
|
||||
IDs, non-contiguous indexes, and a `plan_digest` that does not match the
|
||||
reconstructed logical plan. The checked-in
|
||||
[schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json)
|
||||
defines the strict JSON shape.
|
||||
|
||||
The artifact is available only when the chunk plan was accepted. It remains
|
||||
available when a later extraction, merge, or normalization result is rejected;
|
||||
it is absent when chunk validation rejects the candidate plan.
|
||||
## Valid Example
|
||||
|
||||
`requested_chunker` describes the current pipeline selection. `producer`
|
||||
describes who originally produced the accepted plan. On a cache hit these can
|
||||
differ: the accepted ranges, annotations, digests, and stable materialized IDs
|
||||
are reused, while the producer remains the stored producer. Cache paths,
|
||||
actions, references, metadata, warnings, timestamps, and detailed provenance
|
||||
remain in the run manifest rather than this payload.
|
||||
The compact
|
||||
[source chunk-map fixture](../../internal/framework/chunkmap/testdata/source_chunk_map.v1.json)
|
||||
is decoded by the production codec and demonstrates an accepted map with
|
||||
annotations, producer identity, and ordered chunks.
|
||||
|
||||
## Data Handling
|
||||
## Publication And Compatibility
|
||||
|
||||
The map contains structure, not source content. It excludes transcript bytes,
|
||||
materialized units, source-unit metadata, chunk content, private model
|
||||
responses, rejected proposals, debug data, external-reference content, and
|
||||
filesystem paths.
|
||||
The map is present only when a chunk plan was accepted and its export is
|
||||
enabled. It remains publishable if a later lane is rejected, but is absent when
|
||||
chunk-plan validation rejects the plan. `requested_chunker` identifies the
|
||||
current pipeline selection, while `producer` identifies the component that
|
||||
originally produced the accepted plan; they may differ when an accepted plan is
|
||||
reused.
|
||||
|
||||
Annotations can nevertheless be source- or model-derived. Treat an enabled
|
||||
`chunk-map.json` with the same sensitivity and retention expectations as lane
|
||||
output. Physical placement, confined atomic writing, and permissions follow
|
||||
the ordinary [output operation](../operations.md#output).
|
||||
The map contains structure rather than source content: it excludes transcript
|
||||
bytes, source-unit metadata, chunk text, private model output, reference
|
||||
content, debug data, and filesystem paths. Treat the exported map with the
|
||||
same care as other published output. Publication location and retention are
|
||||
defined in [Operations](../operations.md#output-bundles).
|
||||
|
||||
Reference in New Issue
Block a user