diff --git a/docs/integrations/evidence-context.md b/docs/integrations/evidence-context.md index 886e9a1..03b2c94 100644 --- a/docs/integrations/evidence-context.md +++ b/docs/integrations/evidence-context.md @@ -63,7 +63,8 @@ Each source unit has required `id`, `kind`, `text`, and self `ref` fields. endpoints identify that unit's `id`. A unit may also contain source-owned `metadata`, an open-ended JSON object. Fixed unit and reference fields are strict: consumers must reject unknown fixed fields, malformed units, invalid -self-references, and a payload that is not the array described here. +self-references, units whose `source_id` differs from other units in the same +excerpt, and a payload that is not the array described here. The excerpt preserves each selected unit exactly as represented by the validated generic source document. It does not add evidence-context-specific diff --git a/docs/roadmap/evidence-context.md b/docs/roadmap/evidence-context.md deleted file mode 100644 index 6b60330..0000000 --- a/docs/roadmap/evidence-context.md +++ /dev/null @@ -1,174 +0,0 @@ -# Minimal Evidence Context - -## Purpose - -Redefine the optional `evidence-context.json` output as a compact reading -excerpt rather than a second provenance model. The artifact should contain the -smallest useful source projection: the ordered union of source units cited by -selected accepted lane artifacts, expanded by the configured surrounding -window. - -The normalized lane artifacts remain authoritative for which lane cited which -source ranges. The original source remains authoritative for the complete -input. Evidence context is an optional convenience artifact for consumers that -need the relevant source material alongside the extracted results. - -## Motivation - -The current artifact repeats source units inside context objects and adds -context ranges, lane attribution, direct evidence references, source identity, -source digest, selected lanes, and window configuration. With several lanes -and a nonzero window, this representation can be substantially larger than the -complete input by both bytes and tokens. That defeats its intended value as a -convenient evidence excerpt. - -The application already publishes authoritative source references in each -normalized artifact. Repeating their provenance in `evidence-context.json` -does not justify the additional size or contract complexity. - -## Target Contract - -`evidence-context.json` remains an optional pipeline-wide JSON artifact with -the existing logical filename, artifact kind, media type, schema identity, and -schema version. Because Notarius is pre-release, the v1 schema may be replaced -in place and no reader compatibility or migration path is required. - -The payload is a top-level JSON array of generic source units. It has no -evidence-context-specific wrapper or metadata. Every element uses the existing -`source.SourceUnit` representation and therefore preserves its required -`id`, `kind`, `text`, and self-reference together with any metadata already -owned by the source unit. “No additional metadata” means that evidence-context -construction does not annotate, reshape, or enrich a source unit; it does not -mean stripping metadata supplied by the input adapter. - -An enabled output with no accepted cited evidence emits `[]`, not `null` and -not an absent artifact. The artifact remains absent when evidence-context -publication is not enabled. - -## Selection Semantics - -Construction must: - -1. obtain source references only through the typed evidence projections of - accepted normalized outputs from the configured lane allowlist; -2. validate every projected reference against the current source document; -3. expand each valid referenced range by `window_units` positions on both - sides, clamping at the source boundaries; -4. take the union of all expanded ranges; -5. emit every selected source unit exactly once and in source-document - position order; and -6. return owned copies so later mutation cannot alias the source document or - output artifact. - -Overlapping and adjacent ranges may be coalesced internally, but range groups -are not represented in the payload. Repeated citations, citations from -multiple lanes, and overlapping windows never duplicate a source unit. -Rejected, failed, absent, inactive, and unselected lanes contribute nothing. - -The selected set is therefore bounded by the source document: it can contain -at most every source unit once. With broad evidence coverage or a sufficiently -large window, it may legitimately equal the complete generic source document. -No byte- or token-size guarantee is made because generic source-unit -serialization can differ from the external input format and output formatting -has its own overhead. The application must not truncate a complete excerpt to -meet an arbitrary size limit. - -## Configuration And Publication - -Retain the current `output.options.evidence_context` configuration: - -- `enabled` continues to control publication; -- `lanes` remains the non-empty allowlist of configured artifact lanes whose - accepted normalized evidence contributes to the excerpt; and -- `window_units` remains a non-negative optional value with the existing - default of three. - -Retain `evidence-context.json` and the pipeline-wide `index.json` -`evidence_context` descriptor. The descriptor remains the canonical place for -artifact identity and discovery. It must not be copied into the payload. - -## Internal Design - -Keep evidence preparation at the current framework and output boundary. The -generic framework should continue to consume typed evidence projections rather -than inspect domain JSON or import domain artifact types. - -Simplify `internal/framework/evidencecontext` around the new contract: - -- represent the durable document as a collection of source units rather than - contexts and evidence contributions; -- reduce the build request to the source document, window size, and projected - source references needed to select units; -- remove durable and internal types used only for lane attribution, context - ranges, and evidence-reference publication; -- retain strict schema validation, deterministic serialization, source and - reference validation, ownership, and stable artifact identity; and -- preserve the existing pipeline preparation logic that validates configured - lanes and typed evidence projections. - -The pipeline may continue to record a content-free debug summary containing -artifact identity, selected lane configuration, window size, source digest, -and emitted unit count. Remove the obsolete context count. Debug information -is operational state outside `evidence-context.json` and must not contain the -selected source text or source-unit metadata. - -## Documentation Impact - -When the implementation lands, rewrite the durable contract in -`docs/integrations/evidence-context.md`. Update the configuration, JSON-output, -subprocess-consumer, operations, and internal-module documentation wherever it -describes the old context or provenance structure. Preserve canonical -ownership: the integration contract defines the payload; other documents give -only the information appropriate to their audience and link to that contract. - -The maintained complete D&D configuration should retain its existing evidence -publication settings. No new example is needed because the configuration -surface is unchanged. - -## Testing Expectations - -Tests should protect the revised behavior rather than the former internal -shape. At the narrowest stable boundaries, verify: - -- window expansion and source-boundary clamping; -- union, deduplication, and source-position ordering, including non-monotonic - unit IDs; -- unchanged source-unit values and owned metadata; -- `[]` for enabled publication with no accepted cited evidence; -- rejection of invalid sources, references, and durable payloads; -- omission when publication is disabled; -- contribution only from selected accepted normalized outputs, including - checkpoint reconstruction; and -- correct JSON bundle publication and index discovery. - -Do not add an exact byte-length, token-count, or ratio test. Such a test would -be format-sensitive and would not protect the durable contract. The meaningful -boundedness invariant is structural: every emitted element corresponds to one -distinct unit from the source document, and no source unit is emitted more -than once. - -## Documentation Decision - -No new ADR is required. This work simplifies a pre-release output contract but -does not change the architectural ownership of source evidence, typed -projections, pipeline stages, or output publication. The durable contract and -current implementation documentation should be updated when the behavior is -implemented; this roadmap owns the proposed behavior until then. - -## Acceptance Criteria - -- `evidence-context.json` is a top-level array containing only selected generic - source units. -- The payload contains no context groups, lane IDs, evidence references, - source-level wrapper fields, or evidence-context-specific annotations. -- Units are copied unchanged from the source document, appear once, and retain - source-document order. -- Existing lane selection and `window_units` behavior remain configurable and - retain their current validation and defaults. -- The artifact remains optional, keeps its filename and identity, and remains - discoverable through `index.json`. -- The implementation retains strict validation, deterministic output, - ownership, failure propagation, checkpoint behavior, and content-free debug - reporting. -- Current documentation and maintained examples accurately describe the new - contract, and the repository-wide tests, vet, and build pass. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 758bea2..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,315 +0,0 @@ -# Minimal Evidence Context Implementation Plan - -## Objective - -Implement the target state defined in -[Minimal Evidence Context](evidence-context.md): retain the optional -`evidence-context.json` output and its existing configuration while replacing -its provenance-heavy payload with the ordered, deduplicated union of selected -generic source units. - -This plan intentionally changes the existing v1 contract in place. Notarius is -pre-release, so do not add legacy decoding, schema-version branching, -migration, dual publication, or other backward-compatibility machinery. - -## Global Implementation Rules - -Apply these rules in every stage: - -- Follow `docs/policy/architecture.md`, `docs/policy/documentation.md`, and - `docs/policy/testing.md`. -- Treat the normalized lane artifacts as authoritative for citations and the - source document as authoritative for source-unit content. -- Preserve the typed evidence-projection boundary. Generic code must not - inspect domain JSON or depend on D&D artifact types. -- Preserve the logical filename `evidence-context.json`, artifact kind - `source/evidence-context`, media type `application/json`, schema ID - `notarius.source.evidence_context`, schema name - `notarius_source_evidence_context_v1`, and schema version `v1`. -- Preserve the current configuration keys, validation, lane allowlist - semantics, and default `window_units` value. -- Do not introduce a payload wrapper. The durable JSON value is a top-level - array of generic source units; the empty value is `[]`. -- Preserve source-unit fields and metadata exactly as represented in the - validated generic source document. Add no evidence-context-specific fields. -- Do not add byte-count, token-count, or compression-ratio tests. Test the - structural boundedness and behavioral contract instead. -- Use focused tests while iterating and leave the repository compiling and its - relevant test suites passing at the end of every stage. - -## Stage 1: Replace The Core Evidence-Context Contract - -### Goal - -Replace the context/provenance model with a strict top-level source-unit array -and simplify construction to select the union of expanded source ranges. - -### Required Changes - -1. Update `internal/framework/evidencecontext/model.go`: - - represent `Document` as the durable collection of `source.SourceUnit` - values that serializes directly as a JSON array; - - retain the existing artifact identity constants; - - reduce `BuildRequest` to `Source`, `WindowUnits`, and a flat collection of - projected `SourceRefs`; - - remove `Context`, `EvidenceRef`, `LaneEvidence`, and any other types whose - only purpose was durable lane attribution or context grouping. -2. Update `internal/framework/evidencecontext/build.go`: - - validate the window and complete source document, including its digest; - - validate every projected source reference against the source index; - - convert referenced endpoints to source positions, expand and clamp them, - and merge their union; - - append owned copies of the selected source units once in document-position - order; - - return an initialized empty document when there are no references; - - remove sorting keys and helpers that exist only to order lane attribution - or evidence records. -3. Update `internal/framework/evidencecontext/codec.go` and - `internal/framework/evidencecontext/assets/schemas/source_evidence_context.v1.json`: - - define the root schema as an array of strict source-unit objects; - - preserve the existing source-unit contract: required `id`, `kind`, `text`, - and `ref`, with optional open-ended source-owned `metadata`; - - ensure decode rejects `null`, wrapper objects, malformed units, unknown - fixed unit/reference fields, invalid self-references, and noncanonical - documents; - - ensure encode and decode own all slice and nested metadata values and - serialize an empty document as `[]`; - - remove canonicalization logic used only by the deleted wrapper and context - structures. -4. Replace the fixture in - `internal/framework/evidencecontext/testdata/source_evidence_context.v1.json` - with a compact top-level unit array. -5. Rewrite `internal/framework/evidencecontext/evidencecontext_test.go` around - durable behavior: - - cover expanded overlapping and adjacent ranges, boundary clamping, - repeated references, and non-monotonic unit IDs; - - prove that the output is source-ordered, contains each selected source - unit once, and never invents a unit; - - prove preservation and ownership of source-unit metadata; - - cover empty-array encoding and strict invalid-input rejection; - - retain artifact identity and deterministic round-trip coverage without - testing deleted internal structures. -6. Adapt `internal/framework/pipeline/evidence_output.go` so it flattens the - typed references from prepared selected lanes into the new build request. - Preserve all existing checks for duplicate accepted lane outputs, source and - artifact-kind compatibility, codec decoding, and evidence projection - failures. -7. Adapt the pipeline debug summary and its use in `runner.go` as necessary: - - calculate selected lanes, window size, and source digest from the prepared - plan and source document rather than the payload; - - report the emitted unit count directly; - - remove `context_count`; - - keep source text and unit metadata out of debug summaries. -8. Rewrite the focused tests in - `internal/framework/pipeline/evidence_output_test.go` to assert the new - payload while preserving coverage of selected accepted outputs, absent and - rejected lanes, incompatible artifacts, failure propagation, ownership, - checkpoint reconstruction, opt-in behavior, and redacted debug reporting. - -### Acceptance Criteria - -- The evidence-context package has no durable or internal model for context - groups, lane attribution, or published evidence references. -- Its codec accepts and emits only the new top-level array contract. -- Its builder returns the exact ordered union of source units selected by the - expanded validated ranges. -- Pipeline construction still obtains references exclusively through prepared - typed evidence projections. -- The payload remains absent when publication is not configured and is `[]` - when enabled with no contributing evidence. -- Focused framework tests pass. - -### Validation - -```sh -gofmt -w internal/framework/evidencecontext internal/framework/pipeline/evidence_output.go internal/framework/pipeline/evidence_output_test.go -go test ./internal/framework/evidencecontext ./internal/framework/pipeline -``` - -### Prompt Scope - -This is one coherent but substantial gpt-5.6-terra prompt. Keep the work -limited to the core contract, its pipeline construction boundary, and their -focused tests; do not update generic JSON output tests, production integration -tests, or documentation in this stage. - -## Stage 2: Update Output And Production Integration Coverage - -### Goal - -Carry the revised artifact through the generic JSON output and representative -assembled D&D workflows without changing publication or discovery semantics. - -### Required Changes - -1. Review `internal/modules/generic/output/json/encoder.go` and retain the - existing option decoding, default window, lane validation, logical filename, - artifact identity checks, pretty JSON publication, and `index.json` - descriptor. Change production code only where it assumes the old payload. -2. Rewrite evidence-context fixtures and assertions in - `internal/modules/generic/output/json/encoder_test.go`: - - validate and publish a top-level unit array; - - retain descriptor, omission, strict artifact validation, and - content-nondisclosure failure tests; - - remove expectations for wrapper fields or contexts. -3. Update evidence assertions in the representative integration and CLI - contract tests, including: - - `internal/modules/integration/dnd_npc_grounded_test.go`; - - `internal/modules/integration/dnd_location_registry_runner_test.go`; - - `internal/cli/dnd_enemy_events_contract_test.go`; and - - any additional test located by searching for `evidence-context`, - `EvidenceContext`, `Contexts`, `context_count`, or old fixture fields. -4. Ensure the integration assertions protect the behavior that matters: - - only selected accepted lanes contribute references; - - multiple lanes and overlapping windows produce one ordered unit union; - - non-monotonic unit IDs do not change source-position order; - - scene-description selection remains explicit; - - the output descriptor remains pipeline-wide rather than a lane artifact. -5. Remove obsolete helpers and fixtures that only inspect old context groups or - per-lane evidence records. Do not preserve compatibility helpers. - -### Acceptance Criteria - -- The generic JSON encoder publishes the new payload under the existing - filename and descriptor. -- Maintained integration and CLI workflows observe a deduplicated array of - unchanged source units. -- No production code or test outside the archived roadmap refers to deleted - payload fields or types. -- Focused output, integration, and CLI tests pass. - -### Validation - -```sh -gofmt -w internal/modules/generic/output/json internal/modules/integration internal/cli -go test ./internal/modules/generic/output/json ./internal/modules/integration ./internal/cli -rg -n 'context_ref|evidence_refs|selected_lanes|source_digest|context_count|\.Contexts\b' internal --glob '*.go' --glob '*.json' -``` - -Review every remaining search result rather than requiring the search to be -empty: `source_digest` and similar terms are valid in unrelated contracts and -content-free debug state. - -### Prompt Scope - -This stage is appropriately sized for one gpt-5.6-terra prompt. It owns output -and assembled-workflow adaptation only; do not revise public documentation in -this stage. - -## Stage 3: Rewrite The Canonical Documentation - -### Goal - -Make all current documentation describe the implemented minimal evidence -excerpt, with the durable contract defined in one canonical location. - -### Required Changes - -1. Rewrite `docs/integrations/evidence-context.md` as the canonical durable - contract: - - retain identity, discovery, filename, and optionality; - - define the top-level source-unit array and empty-array behavior; - - define typed-evidence selection, window expansion, union, deduplication, - source-position ordering, and unchanged-unit semantics; - - state that the excerpt can include at most every generic source unit once - but has no byte/token-size guarantee; - - state that lane artifacts remain authoritative for citations and the - excerpt contains no lane attribution; - - preserve sensitivity and consumer validation guidance. -2. Update `docs/config.md` only where it describes the result of - `evidence_context`. Preserve the existing keys, validation rules, and default - window. Link to the integration contract for payload details. -3. Update `docs/integrations/json-output.md` to describe the logical file as the - optional selected source-unit excerpt without duplicating its schema. -4. Update `docs/consumers/subprocess.md` so consumers decode the array and use - lane artifacts—not the excerpt—for authoritative citations and lane - provenance. -5. Update `docs/operations.md`, `docs/internal/modules.md`, and any other current - documentation that describes context groups, provenance records, or the old - wrapper. Keep operational sensitivity guidance and internal ownership in - their canonical documents. -6. Review `examples/dnd-complete.config.yml`. Retain the current - `evidence_context` configuration and lane list unless repository inspection - reveals an independently stale module key. Do not add a new example solely - for this unchanged configuration surface. -7. Do not modify ADRs or add a new ADR. Do not rewrite archived audit history - to match current behavior. - -### Acceptance Criteria - -- `docs/integrations/evidence-context.md` is the sole canonical definition of - the payload. -- Other current documents accurately summarize their audience-specific aspect - and link to the contract instead of duplicating it. -- No current documentation claims that `evidence-context.json` contains - context groups, evidence-reference records, lane attribution, source-level - wrapper metadata, or a guaranteed byte/token reduction. -- The maintained complete example remains valid and reflects implemented - configuration. - -### Validation - -```sh -rg -n 'context_ref|evidence_refs|selected_lanes|context_count' README.md docs examples --glob '!docs/roadmap/**' --glob '!docs/adr/**' -go test ./internal/config ./internal/modules/generic/output/json ./internal/cli -``` - -Inspect any search matches for legitimate historical or unrelated use. If the -repository provides no automated Markdown link checker, manually verify every -link changed in this stage. - -### Prompt Scope - -This documentation migration is small enough for one gpt-5.6-terra prompt. - -## Stage 4: Final Verification And Cleanup - -### Goal - -Verify the complete feature across package, integration, documentation, and -repository boundaries, then remove only obsolete implementation debris found -by that verification. - -### Required Changes - -1. Review the diff against `docs/roadmap/evidence-context.md` and confirm every - acceptance criterion is implemented. -2. Search the non-archived repository for deleted Go types, old JSON fields, - obsolete fixture shapes, and statements that describe the former contract. -3. Confirm the schema, codec, builder, pipeline request, debug summary, output - encoder, index descriptor, example configuration, and integration contract - agree on the final shape and identities. -4. Run formatting and the complete offline validation suite. -5. Make only narrowly scoped corrections required by failed checks or direct - roadmap divergence. Do not introduce unrelated refactors or new features. -6. Leave the active roadmap documents in place for post-implementation review; - retirement occurs only when separately requested after the implementation - is accepted. - -### Acceptance Criteria - -- All roadmap acceptance criteria are satisfied with no legacy compatibility - path. -- No current code, schema, fixture, example, or documentation depends on the - former wrapper/context/provenance payload. -- The structural bound is protected: the artifact contains only distinct, - unchanged units from the source document in source order. -- Repository formatting, tests, vet, and build all pass. -- The worktree contains only intentional feature changes. - -### Validation - -```sh -gofmt -w internal/framework/evidencecontext internal/framework/pipeline internal/modules/generic/output/json internal/modules/integration internal/cli -go test ./... -go vet ./... -go build ./cmd/notarius -git diff --check -git status --short -``` - -### Prompt Scope - -This verification pass is small enough for one gpt-5.6-terra prompt. If it -uncovers a material unrelated defect, report it separately instead of widening -this feature. diff --git a/internal/framework/evidencecontext/build.go b/internal/framework/evidencecontext/build.go index 26b908a..70234b3 100644 --- a/internal/framework/evidencecontext/build.go +++ b/internal/framework/evidencecontext/build.go @@ -44,7 +44,11 @@ func Build(request BuildRequest) (Document, error) { } merged := mergeRanges(ranges) - document := make(Document, 0) + unitCount := 0 + for _, value := range merged { + unitCount += value.endPos - value.startPos + 1 + } + document := make(Document, 0, unitCount) for _, value := range merged { for position := value.startPos; position <= value.endPos; position++ { unit, err := cloneSourceUnit(request.Source.Units[position]) diff --git a/internal/framework/evidencecontext/codec.go b/internal/framework/evidencecontext/codec.go index 1f4bbff..5b9259e 100644 --- a/internal/framework/evidencecontext/codec.go +++ b/internal/framework/evidencecontext/codec.go @@ -167,6 +167,7 @@ func canonicalizeOwned(value Document) (Document, error) { return nil, fmt.Errorf("document must be a JSON array") } seenUnitIDs := make(map[int]struct{}, len(value)) + sourceID := "" for unitIndex := range value { unit := value[unitIndex] if unit.ID <= 0 || strings.TrimSpace(unit.Kind) == "" || strings.TrimSpace(unit.Text) == "" { @@ -175,6 +176,11 @@ func canonicalizeOwned(value Document) (Document, error) { if err := validateUnitRef(unit, unitIndex); err != nil { return nil, err } + if sourceID == "" { + sourceID = unit.Ref.SourceID + } else if unit.Ref.SourceID != sourceID { + return nil, fmt.Errorf("units[%d].ref.source_id must match units[0].ref.source_id", unitIndex) + } if _, exists := seenUnitIDs[unit.ID]; exists { return nil, fmt.Errorf("units contains duplicate unit id %d", unit.ID) } diff --git a/internal/framework/evidencecontext/evidencecontext_test.go b/internal/framework/evidencecontext/evidencecontext_test.go index 66318b4..34c0224 100644 --- a/internal/framework/evidencecontext/evidencecontext_test.go +++ b/internal/framework/evidencecontext/evidencecontext_test.go @@ -19,6 +19,7 @@ func TestBuildSelectsExpandedSourceUnitUnion(t *testing.T) { wantIDs []int }{ {name: "zero window", refs: []source.SourceRef{ref(3, 3)}, wantIDs: []int{3}}, + {name: "multi unit citation includes complete range", refs: []source.SourceRef{ref(3, 7)}, wantIDs: []int{3, 30, 7}}, {name: "non monotonic IDs use document positions", window: 1, refs: []source.SourceRef{ref(3, 3)}, wantIDs: []int{10, 3, 30}}, {name: "boundary clamping", window: 1, refs: []source.SourceRef{ref(10, 10), ref(50, 50)}, wantIDs: []int{10, 3, 7, 50}}, {name: "overlapping and adjacent windows merge", window: 1, refs: []source.SourceRef{ref(3, 3), ref(30, 30), ref(30, 30)}, wantIDs: []int{10, 3, 30, 7}}, @@ -143,6 +144,7 @@ func TestCodecRejectsInvalidDurablePayloads(t *testing.T) { {name: "unknown unit field", content: `[{"id":1,"kind":"segment","text":"text","ref":{"source_id":"session","start_unit_id":1,"end_unit_id":1},"unknown":true}]`}, {name: "unknown reference field", content: `[{"id":1,"kind":"segment","text":"text","ref":{"source_id":"session","start_unit_id":1,"end_unit_id":1,"unknown":true}}]`}, {name: "invalid self reference", content: `[{"id":1,"kind":"segment","text":"text","ref":{"source_id":"session","start_unit_id":1,"end_unit_id":2}}]`}, + {name: "mixed source documents", content: `[{"id":1,"kind":"segment","text":"one","ref":{"source_id":"session-one","start_unit_id":1,"end_unit_id":1}},{"id":2,"kind":"segment","text":"two","ref":{"source_id":"session-two","start_unit_id":2,"end_unit_id":2}}]`}, {name: "duplicate units", content: `[{"id":1,"kind":"segment","text":"one","ref":{"source_id":"session","start_unit_id":1,"end_unit_id":1}},{"id":1,"kind":"segment","text":"two","ref":{"source_id":"session","start_unit_id":1,"end_unit_id":1}}]`}, {name: "multiple JSON values", content: `[] []`}, } {