diff --git a/docs/config.md b/docs/config.md index 9c2f027..1b804d4 100644 --- a/docs/config.md +++ b/docs/config.md @@ -211,19 +211,41 @@ Validator bindings accept only **module**, **llm_profile**, and **options**. They reject **references**, **retries**, and nested **validators**. Deterministic validators reject an explicit **llm_profile**. -The **json** output module accepts one option: +The **json** output module accepts optional **include_chunk_map** and +**evidence_context** settings: ~~~yaml output: module: json options: include_chunk_map: true + evidence_context: + enabled: true + window_units: 3 + lanes: + - npcs + - spells ~~~ **include_chunk_map** is a boolean and defaults to false. It adds the accepted chunk map when one exists; its wire format is defined in the [chunk-map contract](integrations/chunk-map.md). +Omitting **evidence_context** disables evidence publication. When present, it +is an object with these strict fields: + +| Field | Type | Rules | +| --- | --- | --- | +| **enabled** | boolean | Required. `false` permits no other evidence fields. | +| **lanes** | array of strings | Required and non-empty when enabled. Each value is trimmed and must be unique; every value must name a configured pipeline lane. | +| **window_units** | non-negative integer | Optional when enabled; defaults to 3. Zero retains only directly cited units. | + +Unknown outer or nested option fields are rejected, as are incompatible YAML +types. The allowlist remains valid when a run uses lane filtering: a configured +lane that is not active for that invocation simply contributes no evidence. +Evidence publication is opt-in because it can persist source text and metadata. +Its payload contract is [Published Evidence Context](integrations/evidence-context.md). + ## References And Ordered Handoffs Reference maps bind named slots that the selected target declares. A scalar is diff --git a/docs/consumers/subprocess.md b/docs/consumers/subprocess.md index 7d17827..a828639 100644 --- a/docs/consumers/subprocess.md +++ b/docs/consumers/subprocess.md @@ -49,6 +49,14 @@ guessed filename. Before decoding a selected payload, verify its descriptor's media type and schema identity against the relevant published artifact contract. The JSON bundle contract links to the available lane contracts. +If `index.json` has an `evidence_context` descriptor, treat it as a +pipeline-wide artifact rather than a lane entry. Verify its six descriptor +fields before decoding the linked file according to the [Published Evidence +Context contract](../integrations/evidence-context.md). Use each +`evidence_refs` entry as the citation to source material. Its surrounding +context range and included units explain the citation, but do not widen or +replace the cited source reference. + A zero exit status may still report rejected outputs, warnings, or absent lanes. The caller decides which lane IDs are required for its own work and which are optional; it should make that decision explicitly rather than infer @@ -61,4 +69,6 @@ Keep the receipt with the published `manifest.json`, and retain them. Treat the input, output bundle, cache, debug bundle, and captured process logs as potentially sensitive data. Apply the caller's access controls and retention policy, and avoid copying secrets into arguments, logs, or -provenance records. +provenance records. An evidence-context artifact contains source-unit text and +metadata, and selected lanes can cover most of an input; preserve and share it +only when that source content is authorized for the recipient. diff --git a/docs/integrations/evidence-context.md b/docs/integrations/evidence-context.md new file mode 100644 index 0000000..82d4e12 --- /dev/null +++ b/docs/integrations/evidence-context.md @@ -0,0 +1,116 @@ +# Published Evidence Context + +This contract defines the optional `source/evidence-context` artifact emitted +by the production JSON output. Its configuration is owned by +[Configuration](../config.md#module-bindings-and-validators); its logical-file +discovery is owned by [Published JSON Output](json-output.md). + +## Identity And Discovery + +When enabled, the JSON bundle contains `evidence-context.json` and an +`index.json` `evidence_context` descriptor with the same six fields as other +pipeline-wide artifact descriptors. + +| Property | Value | +| --- | --- | +| Artifact kind | `source/evidence-context` | +| Media type | `application/json` | +| Schema ID | `notarius.source.evidence_context` | +| Schema name | `notarius_source_evidence_context_v1` | +| Schema version | `v1` | +| Logical file | `evidence-context.json` | + +Consumers must discover the file from the descriptor, verify all six descriptor +fields, and decode only a supported schema version. The descriptor is optional: +its absence means evidence publication was not enabled for that bundle. + +## Payload + +The v1 payload is a JSON object with required `source_id`, `source_digest`, +`window_units`, `selected_lanes`, and `contexts` fields. `selected_lanes` and +`contexts` are always arrays; an enabled configuration with no accepted direct +evidence publishes `contexts: []`. + +```json +{ + "source_id": "session-alpha", + "source_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "window_units": 1, + "selected_lanes": ["npcs", "spells"], + "contexts": [ + { + "context_ref": { + "source_id": "session-alpha", + "start_unit_id": 10, + "end_unit_id": 20 + }, + "evidence_refs": [ + { + "lane_id": "spells", + "source_ref": { + "source_id": "session-alpha", + "start_unit_id": 10, + "end_unit_id": 10 + } + } + ], + "units": [ + { + "id": 10, + "kind": "transcript_segment", + "text": "Aria casts Cure Wounds.", + "ref": { + "source_id": "session-alpha", + "start_unit_id": 10, + "end_unit_id": 10 + } + }, + { + "id": 20, + "kind": "transcript_segment", + "text": "The party regroups.", + "ref": { + "source_id": "session-alpha", + "start_unit_id": 20, + "end_unit_id": 20 + } + } + ] + } + ] +} +``` + +Each context requires `context_ref`, `evidence_refs`, and `units` arrays. +`context_ref` identifies the first and last included unit. Each evidence entry +contains a selected `lane_id` and an original `source_ref`. A unit uses the +existing source-unit shape: required `id`, `kind`, `text`, and self `ref`, plus +optional JSON-object `metadata`. Fixed payload objects reject unknown fields; +unit metadata may contain application-defined JSON values. + +## Citations And Context + +`evidence_refs` are the authoritative citations. They identify the direct +references emitted by accepted normalized artifacts. `context_ref` and the +units collection include those cited units plus nearby source units selected by +the configured window. They are explanatory context, not widened citations. + +Only accepted outputs from the configured lane allowlist contribute. Rejected, +failed, absent, and lane-filtered outputs do not contribute. The artifact never +contains raw input bytes, prompts, model responses, auxiliary reference +content, credentials, or filesystem paths. + +## Ordering And Compatibility + +The selected lane allowlist is lexical. Contexts and units are in source +document position order, not numeric unit-ID order. Direct evidence entries +are deterministically ordered by lane and source reference. Overlapping or +contiguous windows merge, and each source unit appears at most once in the +resulting contexts. + +The artifact is additive to the JSON bundle and is not a lane payload, +normalized-output count, checkpoint, or generated reference. Consumers that +do not need it must tolerate the absent optional descriptor. Consumers that do +use it should preserve the artifact and its schema identity with the run +provenance, and should treat its source text and metadata as sensitive durable +content. diff --git a/docs/integrations/json-output.md b/docs/integrations/json-output.md index fd187e0..b5eea11 100644 --- a/docs/integrations/json-output.md +++ b/docs/integrations/json-output.md @@ -3,14 +3,14 @@ This document defines the logical JSON bundle emitted by the production JSON output encoder. The bundle’s physical destination, atomic publication, and retention are operational concerns; see [Operations](../operations.md#output-bundles). -Output configuration, including chunk-map export, belongs in +Output configuration, including chunk-map and evidence-context publication, belongs in [Configuration](../config.md#module-bindings-and-validators). ## Bundle Layout All paths below are logical, relative, slash-separated bundle paths. The -encoder always emits the first four JSON files below and adds lane or chunk-map -files when their corresponding artifacts are available: +encoder always emits the first four JSON files below and adds lane or +pipeline-wide artifact files when their corresponding artifacts are available: A subprocess caller first obtains the physical bundle root from the [run-result receipt](run-result.md), then resolves `index.json` beneath that @@ -24,6 +24,7 @@ root for the logical discovery described here. | `warnings.json` | Accepted-output and run warnings. | | `lanes/.json` | One normalized artifact payload for each lane. | | `chunk-map.json` | Optional accepted chunk map, when its export is enabled and available. | +| `evidence-context.json` | Optional source-context artifact, when evidence publication is enabled. | JSON files are pretty-printed with a trailing newline. Lane payloads are accepted only when their media type is `application/json`. @@ -49,13 +50,15 @@ normalized lanes has this valid minimal index: | `rejected_file` | Yes | Always `rejected.json`. | | `warnings_file` | Yes | Always `warnings.json`. | | `chunk_map` | No | Descriptor for the pipeline-wide `chunk-map.json`; never a lane descriptor. | +| `evidence_context` | No | Descriptor for the pipeline-wide `evidence-context.json`; never a lane descriptor. | Each lane descriptor has required `lane_id` and `file`. It may also include `media_type`, `module_key`, `schema_id`, `schema_name`, and `schema_version` -when supplied by the normalized artifact. A `chunk_map` descriptor contains -`artifact_kind`, `file`, `media_type`, `schema_id`, `schema_name`, and -`schema_version`; its payload is defined by the -[Accepted Chunk Map contract](chunk-map.md). +when supplied by the normalized artifact. Each pipeline-wide artifact +descriptor (`chunk_map` or `evidence_context`) contains `artifact_kind`, +`file`, `media_type`, `schema_id`, `schema_name`, and `schema_version`. Their +payloads are defined by the [Accepted Chunk Map contract](chunk-map.md) and +[Published Evidence Context](evidence-context.md), respectively. The lane path is derived from its lane ID. Characters outside letters, digits, periods, underscores, and hyphens become underscores; `..` sequences are diff --git a/docs/internal/modules.md b/docs/internal/modules.md index 77ef39b..9586e88 100644 --- a/docs/internal/modules.md +++ b/docs/internal/modules.md @@ -30,6 +30,14 @@ they need, register each leaf implementation, and add any family-owned assets or default validator chains. They return contextual errors so production composition fails at startup rather than at the first run. +An artifact family can register an optional typed evidence projector alongside +its codec. The projector returns defensive copies of the artifact's direct +generic source references and must use the codec's exact Go type. It does not +interpret surrounding context or publish files; the pipeline validates the +capability during preparation and the output boundary owns publication. See +the [Published Evidence Context contract](../integrations/evidence-context.md) +for the durable result. + ## Production Composition Production composition is intentionally split by family: diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md index 1cf487c..049609c 100644 --- a/docs/internal/pipeline.md +++ b/docs/internal/pipeline.md @@ -52,6 +52,13 @@ checkpoint fingerprints. Missing registrations, incompatible typed entries, nil implementations, and constructor failures are reported before source parsing or any stage operation begins. +An output encoder can opt into source-evidence publication through its output +policy. Preparation keeps the configured lane allowlist and active lanes +separate, then verifies an exact typed evidence projector and registered codec +for each active lane. The resulting private plan is immutable; lanes excluded +by invocation filtering remain configured but do not acquire a projector for +that run. + ## Typed Lanes And References Each resolved lane has one artifact kind, codec, and exact Go type. The @@ -110,11 +117,16 @@ directives consume this same budget and validate any final safe fallback through the normalizer chain. After terminal lane work, the runner assembles manifest provenance, normalized -artifacts, rejections, warnings, and an optional accepted chunk map. The output -encoder returns logical files; it does not choose a physical directory. The CLI -publishes those files only after the runner returns without a framework error. -Logical file names and schemas are defined by the -[output integration contracts](../integrations/). +artifacts, rejections, warnings, and an optional accepted chunk map. When an +output policy selected evidence lanes, it decodes accepted serialized normalize +outputs through their registered codecs and invokes the prepared typed +projectors. Rejected or absent lanes contribute nothing. This reconstruction is +also used after normalized-checkpoint reuse, so no second typed output channel +is retained. The runner passes the resulting owned artifact to the output +encoder, which returns logical files and does not choose a physical directory. +The CLI publishes those files only after the runner returns without a framework +error. Logical file names and schemas are defined by the [output integration +contracts](../integrations/). ## Checkpoint And Debug Hooks diff --git a/docs/operations.md b/docs/operations.md index 7b501e8..eede0c7 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -36,7 +36,11 @@ On supported Unix systems, output directories and files are created with requested modes **0755** and **0644**. Chunk-plan, checkpoint, and debug directories and files use **0700** and **0600**. The operating system's umask may impose stricter output modes. Cache and debug roots may contain sensitive -source-derived data, so provision them for one trusted account or service. +source-derived data, so provision them for one trusted account or service. An +output bundle can also contain source content when its JSON output enables +evidence publication. Apply an appropriate umask and output-root access policy +before enabling that option; the requested output modes alone may not be +suitable for transcript-bearing bundles. ## Run Lifecycle @@ -66,7 +70,13 @@ run directory remains for inspection and is never removed automatically. Treat an output bundle as durable user data. Do not use cache-cleanup policy to remove it. An optional accepted chunk map is also durable output and can carry source- or model-derived annotations; its content and compatibility contract -are defined in [Accepted Chunk Map](integrations/chunk-map.md). +are defined in [Accepted Chunk Map](integrations/chunk-map.md). An optional +[evidence context](integrations/evidence-context.md) contains source-unit text +and metadata. It is not a cache or debug artifact: retain it with the output +bundle only for as long as consumers need it, and apply source-content access +controls to the entire bundle. Selected lanes may collectively cite most of a +transcript, so a broad allowlist can make the evidence artifact nearly as +sensitive and large as the source itself. ## Chunk-Plan Cache diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 0e68699..d7f177d 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -79,6 +79,14 @@ Pipeline resolution requires a compatible codec and matching kind-specific variants before a typed lane can be accepted. Framework-owned erasure remains private and must report type incompatibility as an error rather than a panic. +An artifact kind may additionally provide a typed evidence projection that +copies its direct generic source references. Preparation proves that projection +matches the artifact codec's exact Go type before retaining it for an output +policy. The runner reconstructs evidence only from accepted serialized +normalized artifacts, and the output boundary owns any resulting publication. +Generic framework code never infers evidence by inspecting domain JSON or +depends on domain artifact types. + Auxiliary references provide context or disambiguation. They are not source evidence and must not be converted into source references. diff --git a/docs/roadmap/evidence.md b/docs/roadmap/evidence.md index cd86f02..fc2c714 100644 --- a/docs/roadmap/evidence.md +++ b/docs/roadmap/evidence.md @@ -2,7 +2,7 @@ ## Status -Accepted for implementation. +Implemented. ## Purpose diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 05f35d9..d86e8ea 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -2,7 +2,7 @@ ## Status -Ready for implementation. +Completed. ## Objective