175 lines
8.4 KiB
Markdown
175 lines
8.4 KiB
Markdown
# Minimal Evidence Context
|
|
|
|
## Purpose
|
|
|
|
Redefine the optional `evidence-context.json` output as a compact reading
|
|
excerpt rather than a second provenance model. The artifact should contain the
|
|
smallest useful source projection: the ordered union of source units cited by
|
|
selected accepted lane artifacts, expanded by the configured surrounding
|
|
window.
|
|
|
|
The normalized lane artifacts remain authoritative for which lane cited which
|
|
source ranges. The original source remains authoritative for the complete
|
|
input. Evidence context is an optional convenience artifact for consumers that
|
|
need the relevant source material alongside the extracted results.
|
|
|
|
## Motivation
|
|
|
|
The current artifact repeats source units inside context objects and adds
|
|
context ranges, lane attribution, direct evidence references, source identity,
|
|
source digest, selected lanes, and window configuration. With several lanes
|
|
and a nonzero window, this representation can be substantially larger than the
|
|
complete input by both bytes and tokens. That defeats its intended value as a
|
|
convenient evidence excerpt.
|
|
|
|
The application already publishes authoritative source references in each
|
|
normalized artifact. Repeating their provenance in `evidence-context.json`
|
|
does not justify the additional size or contract complexity.
|
|
|
|
## Target Contract
|
|
|
|
`evidence-context.json` remains an optional pipeline-wide JSON artifact with
|
|
the existing logical filename, artifact kind, media type, schema identity, and
|
|
schema version. Because Notarius is pre-release, the v1 schema may be replaced
|
|
in place and no reader compatibility or migration path is required.
|
|
|
|
The payload is a top-level JSON array of generic source units. It has no
|
|
evidence-context-specific wrapper or metadata. Every element uses the existing
|
|
`source.SourceUnit` representation and therefore preserves its required
|
|
`id`, `kind`, `text`, and self-reference together with any metadata already
|
|
owned by the source unit. “No additional metadata” means that evidence-context
|
|
construction does not annotate, reshape, or enrich a source unit; it does not
|
|
mean stripping metadata supplied by the input adapter.
|
|
|
|
An enabled output with no accepted cited evidence emits `[]`, not `null` and
|
|
not an absent artifact. The artifact remains absent when evidence-context
|
|
publication is not enabled.
|
|
|
|
## Selection Semantics
|
|
|
|
Construction must:
|
|
|
|
1. obtain source references only through the typed evidence projections of
|
|
accepted normalized outputs from the configured lane allowlist;
|
|
2. validate every projected reference against the current source document;
|
|
3. expand each valid referenced range by `window_units` positions on both
|
|
sides, clamping at the source boundaries;
|
|
4. take the union of all expanded ranges;
|
|
5. emit every selected source unit exactly once and in source-document
|
|
position order; and
|
|
6. return owned copies so later mutation cannot alias the source document or
|
|
output artifact.
|
|
|
|
Overlapping and adjacent ranges may be coalesced internally, but range groups
|
|
are not represented in the payload. Repeated citations, citations from
|
|
multiple lanes, and overlapping windows never duplicate a source unit.
|
|
Rejected, failed, absent, inactive, and unselected lanes contribute nothing.
|
|
|
|
The selected set is therefore bounded by the source document: it can contain
|
|
at most every source unit once. With broad evidence coverage or a sufficiently
|
|
large window, it may legitimately equal the complete generic source document.
|
|
No byte- or token-size guarantee is made because generic source-unit
|
|
serialization can differ from the external input format and output formatting
|
|
has its own overhead. The application must not truncate a complete excerpt to
|
|
meet an arbitrary size limit.
|
|
|
|
## Configuration And Publication
|
|
|
|
Retain the current `output.options.evidence_context` configuration:
|
|
|
|
- `enabled` continues to control publication;
|
|
- `lanes` remains the non-empty allowlist of configured artifact lanes whose
|
|
accepted normalized evidence contributes to the excerpt; and
|
|
- `window_units` remains a non-negative optional value with the existing
|
|
default of three.
|
|
|
|
Retain `evidence-context.json` and the pipeline-wide `index.json`
|
|
`evidence_context` descriptor. The descriptor remains the canonical place for
|
|
artifact identity and discovery. It must not be copied into the payload.
|
|
|
|
## Internal Design
|
|
|
|
Keep evidence preparation at the current framework and output boundary. The
|
|
generic framework should continue to consume typed evidence projections rather
|
|
than inspect domain JSON or import domain artifact types.
|
|
|
|
Simplify `internal/framework/evidencecontext` around the new contract:
|
|
|
|
- represent the durable document as a collection of source units rather than
|
|
contexts and evidence contributions;
|
|
- reduce the build request to the source document, window size, and projected
|
|
source references needed to select units;
|
|
- remove durable and internal types used only for lane attribution, context
|
|
ranges, and evidence-reference publication;
|
|
- retain strict schema validation, deterministic serialization, source and
|
|
reference validation, ownership, and stable artifact identity; and
|
|
- preserve the existing pipeline preparation logic that validates configured
|
|
lanes and typed evidence projections.
|
|
|
|
The pipeline may continue to record a content-free debug summary containing
|
|
artifact identity, selected lane configuration, window size, source digest,
|
|
and emitted unit count. Remove the obsolete context count. Debug information
|
|
is operational state outside `evidence-context.json` and must not contain the
|
|
selected source text or source-unit metadata.
|
|
|
|
## Documentation Impact
|
|
|
|
When the implementation lands, rewrite the durable contract in
|
|
`docs/integrations/evidence-context.md`. Update the configuration, JSON-output,
|
|
subprocess-consumer, operations, and internal-module documentation wherever it
|
|
describes the old context or provenance structure. Preserve canonical
|
|
ownership: the integration contract defines the payload; other documents give
|
|
only the information appropriate to their audience and link to that contract.
|
|
|
|
The maintained complete D&D configuration should retain its existing evidence
|
|
publication settings. No new example is needed because the configuration
|
|
surface is unchanged.
|
|
|
|
## Testing Expectations
|
|
|
|
Tests should protect the revised behavior rather than the former internal
|
|
shape. At the narrowest stable boundaries, verify:
|
|
|
|
- window expansion and source-boundary clamping;
|
|
- union, deduplication, and source-position ordering, including non-monotonic
|
|
unit IDs;
|
|
- unchanged source-unit values and owned metadata;
|
|
- `[]` for enabled publication with no accepted cited evidence;
|
|
- rejection of invalid sources, references, and durable payloads;
|
|
- omission when publication is disabled;
|
|
- contribution only from selected accepted normalized outputs, including
|
|
checkpoint reconstruction; and
|
|
- correct JSON bundle publication and index discovery.
|
|
|
|
Do not add an exact byte-length, token-count, or ratio test. Such a test would
|
|
be format-sensitive and would not protect the durable contract. The meaningful
|
|
boundedness invariant is structural: every emitted element corresponds to one
|
|
distinct unit from the source document, and no source unit is emitted more
|
|
than once.
|
|
|
|
## Documentation Decision
|
|
|
|
No new ADR is required. This work simplifies a pre-release output contract but
|
|
does not change the architectural ownership of source evidence, typed
|
|
projections, pipeline stages, or output publication. The durable contract and
|
|
current implementation documentation should be updated when the behavior is
|
|
implemented; this roadmap owns the proposed behavior until then.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- `evidence-context.json` is a top-level array containing only selected generic
|
|
source units.
|
|
- The payload contains no context groups, lane IDs, evidence references,
|
|
source-level wrapper fields, or evidence-context-specific annotations.
|
|
- Units are copied unchanged from the source document, appear once, and retain
|
|
source-document order.
|
|
- Existing lane selection and `window_units` behavior remain configurable and
|
|
retain their current validation and defaults.
|
|
- The artifact remains optional, keeps its filename and identity, and remains
|
|
discoverable through `index.json`.
|
|
- The implementation retains strict validation, deterministic output,
|
|
ownership, failure propagation, checkpoint behavior, and content-free debug
|
|
reporting.
|
|
- Current documentation and maintained examples accurately describe the new
|
|
contract, and the repository-wide tests, vet, and build pass.
|