227 lines
9.4 KiB
Markdown
227 lines
9.4 KiB
Markdown
# Published Evidence Context
|
|
|
|
## Status
|
|
|
|
Implemented.
|
|
|
|
## Purpose
|
|
|
|
Let downstream consumers build narrative reports from normalized artifacts
|
|
without separately parsing the original transcript or resolving source-unit
|
|
references themselves.
|
|
|
|
The production JSON output optionally publishes one deterministic, deduplicated
|
|
evidence-context artifact containing the transcript units relevant to explicitly
|
|
selected normalized lanes. Existing lane payloads remain the canonical semantic
|
|
results and retain their precise source references.
|
|
|
|
## Desired End State
|
|
|
|
When evidence-context publication is enabled, a consumer can:
|
|
|
|
1. discover one versioned evidence-context document through `index.json`;
|
|
2. obtain the union of source units needed to understand evidence cited by the
|
|
selected normalized lanes;
|
|
3. distinguish each artifact's direct evidence references from surrounding
|
|
units included only for narrative context;
|
|
4. retain speaker, timestamp, and other accepted source-unit metadata needed to
|
|
interpret the transcript; and
|
|
5. produce a narrative report without receiving duplicated transcript text in
|
|
every lane payload.
|
|
|
|
This is deterministic output projection. It does not invoke an LLM, change
|
|
normalization, or make surrounding context part of an artifact's evidence.
|
|
|
|
## Configuration Policy
|
|
|
|
Evidence publication is configured on the production JSON output module. The
|
|
intended configuration shape is:
|
|
|
|
```yaml
|
|
output:
|
|
module: json
|
|
options:
|
|
evidence_context:
|
|
enabled: true
|
|
window_units: 3
|
|
lanes:
|
|
- combat-turns
|
|
- item-events
|
|
- npc-interactions
|
|
- npcs
|
|
- spells
|
|
```
|
|
|
|
- Omitting `evidence_context` disables publication. When the object is present,
|
|
`enabled` is required.
|
|
- `enabled: false` accepts no `lanes` or `window_units` fields, preventing
|
|
silently ignored configuration.
|
|
- `lanes` is a required, non-empty allowlist of configured final lane IDs when
|
|
evidence publication is enabled. Values are trimmed, unique, and normalized
|
|
to lexical order.
|
|
- `window_units` is a non-negative integer and defaults to `3`. Zero publishes
|
|
only directly referenced units.
|
|
- Unknown lanes, duplicate lane IDs, and selected lanes whose artifact kind
|
|
cannot expose source evidence fail configuration resolution or pipeline
|
|
preparation.
|
|
- A selected lane that completes without a normalized output contributes no
|
|
evidence and does not make an otherwise successful run fail.
|
|
- Invocation-level lane filtering does not invalidate the configured allowlist.
|
|
Allowlisted lanes excluded from the effective run contribute nothing, while
|
|
the evidence document still records the configured allowlist.
|
|
|
|
The allowlist is intentional safety and stability policy. Scene descriptions
|
|
and other broad-range lanes are excluded unless named expressly. Adding a new
|
|
pipeline lane never silently increases output size or publishes more transcript
|
|
content.
|
|
|
|
## Evidence Collection Boundary
|
|
|
|
Evidence collection applies to accepted final normalized artifacts from the
|
|
selected lanes. It must not inspect arbitrary serialized JSON for fields named
|
|
`source_ref` or `source_refs`, and the generic JSON output module must not
|
|
depend on D&D artifact types.
|
|
|
|
Artifact-kind registrations expose their source references through an explicit
|
|
typed projection contract. The framework uses that contract to assemble a
|
|
domain-neutral evidence request containing:
|
|
|
|
- the accepted generic source document;
|
|
- the selected lane and artifact identities; and
|
|
- defensive copies of their direct source references.
|
|
|
|
The output stage owns publication of the resulting logical artifact. Generic
|
|
framework code owns range validation, position-based expansion, and union
|
|
logic. Domain-specific adapters own only the extraction of evidence references
|
|
from their typed artifacts.
|
|
|
|
Both plural-reference artifacts and singular-reference artifacts, such as
|
|
scene descriptions, can participate through the same projection contract.
|
|
They do so only when their configured lane is allowlisted.
|
|
|
|
## Range Expansion And Deduplication
|
|
|
|
For every valid direct source reference:
|
|
|
|
1. resolve its endpoints through source-document positions, not numeric
|
|
unit-ID arithmetic;
|
|
2. expand the range by `window_units` positions on each side;
|
|
3. clip the expanded range at document boundaries; and
|
|
4. union overlapping or contiguous expanded ranges.
|
|
|
|
Published contexts and units remain in source-document order. Each source unit
|
|
appears at most once in a merged context. Original direct references remain
|
|
unchanged and are associated with their contributing lane IDs so consumers can
|
|
tell why a context was included.
|
|
|
|
The projector must not silently omit or repair an invalid reference that
|
|
reaches this boundary. Such a value violates the accepted normalized-artifact
|
|
contract and causes output projection to fail with a content-safe error.
|
|
|
|
No implicit coverage limit truncates selected evidence. If the allowlisted
|
|
lanes collectively cite most or all of a transcript, the evidence document may
|
|
contain most or all of it. The explicit lane allowlist is the control that
|
|
prevents a broad lane such as scene descriptions from doing so accidentally.
|
|
|
|
## Durable Evidence Artifact
|
|
|
|
The JSON bundle gains one optional, non-lane artifact with these durable
|
|
identities:
|
|
|
|
| Property | Value |
|
|
| --- | --- |
|
|
| Logical file | `evidence-context.json` |
|
|
| Index descriptor | `evidence_context` |
|
|
| Artifact kind | `source/evidence-context` |
|
|
| Media type | `application/json` |
|
|
| Schema ID | `notarius.source.evidence_context` |
|
|
| Schema name | `notarius_source_evidence_context_v1` |
|
|
| Schema version | `v1` |
|
|
|
|
The descriptor in `index.json` carries the artifact and schema identities,
|
|
analogous to the existing chunk-map descriptor. The artifact is present
|
|
whenever evidence publication is enabled, including when its context collection
|
|
is empty.
|
|
|
|
The document contains:
|
|
|
|
- the source document ID and semantic digest;
|
|
- the effective window size;
|
|
- the sorted configured lane allowlist;
|
|
- an ordered context collection;
|
|
- each context's expanded start and end unit IDs;
|
|
- the original direct references and contributing lane IDs covered by that
|
|
context; and
|
|
- the ordered accepted source units, including unit ID, kind, text,
|
|
self-reference, and metadata.
|
|
|
|
Expanded context bounds are navigation aids, not citations. The original
|
|
references embedded in each context remain the authoritative direct evidence.
|
|
The evidence artifact is discovered separately from lane payloads and does not
|
|
increase the normalized-lane count reported by the runner or subprocess
|
|
receipt.
|
|
|
|
## Failure And Publication Semantics
|
|
|
|
- Evidence projection occurs only after selected normalized outputs are known
|
|
and before the output encoder returns its logical files.
|
|
- Projection or encoding failure is an output-stage framework error; the CLI
|
|
does not publish a partially assembled output bundle.
|
|
- Rejected or absent lane outputs contribute nothing. Their attempted values
|
|
and source references must not be published through this artifact.
|
|
- Context generation is deterministic for the same source document, selected
|
|
normalized outputs, lane allowlist, and window size.
|
|
- Existing output, checkpoint, warning, rejection, debug, and subprocess
|
|
success semantics remain unchanged.
|
|
|
|
## Sensitivity And Size
|
|
|
|
Unlike the current chunk map, the evidence artifact contains transcript text
|
|
and source-unit metadata. Enabling it therefore creates additional durable
|
|
sensitive data and may materially increase bundle size.
|
|
|
|
The implemented configuration, operations, integration, and consumer documents
|
|
state that:
|
|
|
|
- evidence publication is opt-in;
|
|
- output permissions and retention must be appropriate for source content;
|
|
- selecting broad or numerous lanes can publish most of the transcript; and
|
|
- the artifact must not contain raw input bytes, LLM prompts or responses,
|
|
auxiliary reference content, credentials, debug-only data, or filesystem
|
|
paths.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- Evidence publication is disabled by default and leaves existing bundles
|
|
unchanged.
|
|
- Enabling it requires an explicit non-empty lane allowlist.
|
|
- References from all selected successful lanes contribute to one deduplicated
|
|
document.
|
|
- Non-monotonic unit IDs are expanded and ordered correctly by document
|
|
position.
|
|
- Overlapping windows share one ordered copy of each included source unit.
|
|
- Direct references remain distinguishable from added context.
|
|
- Scene descriptions cannot contribute unless their lane is explicitly
|
|
allowlisted.
|
|
- Invalid selected lanes and unsupported artifact kinds fail before execution;
|
|
invalid accepted references fail output projection rather than being ignored.
|
|
- Empty selected-lane results produce a valid empty evidence artifact.
|
|
- Existing D&D lane schemas, normalized-output counts, and source-reference
|
|
semantics do not change.
|
|
- Generic framework and output packages do not depend on D&D types or parse
|
|
artifact JSON heuristically.
|
|
- The published contract and operational documentation clearly describe source
|
|
sensitivity, discovery, compatibility, and retention.
|
|
|
|
## Out Of Scope
|
|
|
|
- Embedding transcript units directly into each D&D record or lane payload.
|
|
- Replacing precise source references with expanded context ranges.
|
|
- Automatically including every configured lane.
|
|
- An explicit full-transcript publication mode.
|
|
- LLM summarization, retrieval, ranking, or narrative generation.
|
|
- Per-record window sizes or lane-specific window sizes.
|
|
- CLI overrides for evidence configuration.
|
|
- Reading rejected attempts, debug artifacts, auxiliary references, or prior
|
|
output bundles as evidence sources.
|