notarius/docs/roadmap/implementation.md

Published Evidence Context Implementation Plan

Status

Completed.

Objective

Implement the accepted Published Evidence Context roadmap as an optional, deterministic extension of the production JSON output. The completed work must publish one deduplicated source-context artifact for an explicit set of successful normalized lanes while leaving existing lane payloads, normalization, checkpointing, subprocess results, and disabled output bundles unchanged.

Complete the stages below in order. Each stage must leave its affected packages passing before the next begins. Do not implement per-record hydration, implicit all-lane collection, a full-transcript mode, LLM processing, or any other roadmap item marked out of scope.

Decisions And Invariants

• Evidence publication is output policy. Normalizers continue to return semantic artifacts with precise source references and do not receive hydration responsibilities.
• The framework operates on the accepted generic source.SourceDocument and typed artifact projections. It must not inspect serialized JSON for source_ref or source_refs, and generic packages must not depend on D&D types.
• The selected lane allowlist is explicit, non-empty, and globally addressed by resolved lane ID. Pipeline resolution already guarantees lane IDs are unique across ordered steps.
• The framework decodes accepted serialized normalize outputs through their registered artifact codecs before invoking typed evidence projectors. This supports both fresh and checkpoint-reused normalize outputs without retaining a second typed result channel.
• Expansion uses source-document positions. Numeric unit IDs are identities, not sequence numbers.
• Direct source references are never widened or rewritten. Expanded ranges are context bounds only.
• Rejected, failed, and absent normalized lane outputs contribute no evidence.
• Evidence output is sensitive durable source content, not cache or debug state. It contains accepted source units only and never raw input bytes, prompts, model responses, auxiliary references, paths, or credentials.
• The optional evidence_context index field is an additive v1 JSON-bundle change. Existing D&D artifact schemas and the subprocess receipt do not change.

Stage 1: Add Typed Evidence Capability And Resolve Output Policy

Add a dedicated artifact-evidence registry under the pipeline framework:

• pipeline.ArtifactEvidenceRegistry stores one typed projector per artifact kind.
• pipeline.ArtifactEvidenceProjector[T] is func(T) []source.SourceRef.
• pipeline.RegisterArtifactEvidence[T](registry, kind, projector) accepts a non-empty kind and non-nil projector, records the exact Go type for T, and rejects duplicate kinds.
• The erased projection boundary checks the exact registered type, invokes the projector, and returns a defensive copy of its references.
• The registry exposes only the discovery and projection operations required by resolution, preparation, and execution; do not expose its mutable entries.

Add the registry to pipeline.Registries and pipeline.ModuleCatalog, including CLI catalog conversion, production construction, empty-set detection, and test registry helpers. A nil evidence registry remains valid when evidence publication is disabled. Production construction and the D&D registrar require and populate it.

Define these framework-level output-policy contracts:

type EvidenceContextPolicy struct {
    Enabled     bool
    WindowUnits int
    LaneIDs     []string
}

type EvidenceContextPolicyProvider interface {
    EvidenceContextPolicy() EvidenceContextPolicy
}

Provider and prepared-pipeline boundaries defensively copy LaneIDs.

Add an optional output-profile option-validation callback to OutputEncoderRegistry:

type OutputProfileOptionContext struct {
    LaneIDs []string
}

type OutputProfileOptionValidator func(
    OutputProfileOptionContext,
    map[string]any,
) error

Add RegisterBuilderWithProfileValidation(spec, validateOptions, validateProfile, builder) and make existing output registration methods delegate to it with no profile callback. The registry passes defensive copies to validation. The callback receives the complete configured lane-ID set before invocation-level --only filtering. The production JSON output uses it only to prove that every configured evidence lane exists. Keep the extension generic: the pipeline supplies lane identities, while the output module interprets its own options. Build that lane set from the normalized legacy-or-steps profile before selection and reject duplicate configured lane IDs through the existing pipeline identity rules.

Extend the production JSON output options with the nested evidence_context object:

• omission disables the feature;
• enabled is required when the object is present;
• enabled: false permits no lanes or window_units fields;
• enabled: true requires a non-empty lanes array;
• lane values are strings, trimmed, non-empty, unique after trimming, and normalized to lexical order;
• window_units is an optional non-negative integer with default 3; and
• outer and nested unknown fields and incompatible YAML value types remain strict configuration errors.

The JSON encoder implements the policy provider from its decoded immutable options. Pipeline resolution invokes its profile validator against all configured steps, so an unknown evidence lane fails even when another lane is selected with --only.

During pipeline.Prepare, after constructing the output encoder:

1. obtain and defensively normalize an enabled policy;
2. intersect its configured IDs with the effective prepared lanes, treating allowlisted lanes removed by invocation-level filtering as inactive;
3. require an artifact-evidence registration for each active lane kind;
4. prove that its projector Go type exactly matches the active lane's registered artifact codec type; and
5. retain an immutable private evidence plan on PreparedPipeline.

Duplicate or empty provider values, a missing evidence registry for an active lane, unsupported active artifact kinds, and type mismatches fail preparation with pipeline/output/lane context. A disabled or non-participating output encoder creates no evidence plan and preserves existing preparation behavior. The private plan retains both the full configured allowlist for publication and the active lane/projector intersection for execution.

Register D&D evidence projectors for all six current artifact kinds. Each projector returns copies of the artifact's direct references in record order: spells, NPCs, combat turns, item events, NPC interactions, and the singular reference from each scene description. Scene descriptions gain capability but remain excluded unless their configured lane ID is allowlisted.

Stage tests:

• Registry tests cover nil, blank, duplicate, exact-type, defensive-copy, and deterministic discovery behavior.
• JSON option tests cover disabled, enabled/default-window, explicit zero window, normalization, duplicates, unknown fields, and invalid types.
• Preparation tests cover selected lanes across steps, unknown lanes, unsupported kinds, projector/codec type mismatch, disabled behavior, and defensive policy ownership.
• Resolution/preparation tests prove a valid allowlist survives --only, an excluded lane contributes no active projector, and a genuinely unknown configured lane still fails profile resolution.
• D&D registration tests prove every production D&D artifact kind has the expected evidence capability without testing individual field loops redundantly.
• One table-driven D&D projector test supplies representative values for all six artifact kinds and proves plural and singular references are copied without aliasing or semantic rewriting.

Stage completion:

• go test ./internal/framework/pipeline
• go test ./internal/modules/generic/output/json
• go test ./internal/modules/dnd/register
• go test ./internal/cli

Stage 2: Define And Build The Evidence-Context Artifact

Add a domain-neutral internal/framework/evidencecontext package that owns the durable model, JSON Schema, strict codec, projection algorithm, and these exact identities:

• artifact kind source/evidence-context;
• media type application/json;
• schema ID notarius.source.evidence_context;
• schema name notarius_source_evidence_context_v1; and
• schema version v1.

Use these package-level model and build contracts:

type Document struct {
    SourceID      string
    SourceDigest  string
    WindowUnits   int
    SelectedLanes []string
    Contexts      []Context
}

type Context struct {
    ContextRef  source.SourceRef
    EvidenceRefs []EvidenceRef
    Units       []source.SourceUnit
}

type EvidenceRef struct {
    LaneID   string
    SourceRef source.SourceRef
}

type LaneEvidence struct {
    LaneID     string
    SourceRefs []source.SourceRef
}

type BuildRequest struct {
    Source        *source.SourceDocument
    WindowUnits   int
    SelectedLanes []string
    LaneEvidence  []LaneEvidence
}

Apply the JSON field names shown below. Provide Build(BuildRequest), Serialize(BuildRequest), and a Codec with the same identity/encode/decode responsibilities as the chunk-map codec.

The v1 payload has this exact shape:

{
  "source_id": "session-alpha",
  "source_digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
  "window_units": 0,
  "selected_lanes": ["npcs"],
  "contexts": [
    {
      "context_ref": {
        "source_id": "session-alpha",
        "start_unit_id": 13,
        "end_unit_id": 13
      },
      "evidence_refs": [
        {
          "lane_id": "npcs",
          "source_ref": {
            "source_id": "session-alpha",
            "start_unit_id": 13,
            "end_unit_id": 13
          }
        }
      ],
      "units": [
        {
          "id": 13,
          "kind": "transcript_segment",
          "text": "The party meets Rowan.",
          "ref": {
            "source_id": "session-alpha",
            "start_unit_id": 13,
            "end_unit_id": 13
          }
        }
      ]
    }
  ]
}

All displayed fields are required. selected_lanes, contexts, evidence_refs, and units encode as arrays rather than null; contexts may be empty. Each unit uses the existing source.SourceUnit JSON shape with required id, kind, text, and ref, plus optional JSON-shaped metadata. Fixed objects reject unknown fields; metadata remains an arbitrary JSON object.

The builder accepts the validated source document, selected lane IDs, effective window, and lane-attributed direct references, then:

1. requires a non-negative window and a non-empty, trimmed, unique selected lane set, then stores that set in lexical order;
2. requires every LaneEvidence.LaneID to belong to the selected set;
3. validates the source document, recomputes its semantic digest, and requires it to equal SourceDocument.Digest;
4. validates every direct reference against one source.DocumentIndex;
5. deduplicates exact (lane_id, source_ref) contributions;
6. resolves endpoints to document positions;
7. expands each side without integer overflow and clips at document bounds;
8. sorts by expanded document position with deterministic lane/reference tie-breakers;
9. merges overlapping or position-contiguous expanded intervals;
10. unions and deterministically sorts each merged context's direct contributions; and
11. deep-clones the corresponding source units and JSON-shaped metadata.

Contexts are disjoint and ordered by document position, so a source unit occurs at most once in the document. context_ref identifies the first and last included units; evidence_refs retains only original citations. An empty reference collection produces the same source identity, window, sorted allowlist, and an explicit empty contexts array.

Projection failures identify only structural scope such as lane and reference position. They must not include source text, metadata values, raw serialized artifacts, or unrelated paths.

The codec must validate its model before encoding, produce deterministic JSON, strictly decode the checked-in schema, and return independently owned values. The JSON output encoder remains responsible for pretty-printing the logical file with its standard trailing newline. Follow the existing chunk-map package's separation between model, builder, codec, schema asset, and contract tests where useful, without coupling the two artifact formats.

Stage tests:

• A table-driven builder suite covers zero and nonzero windows, boundary clipping, non-monotonic unit IDs, separate gaps, overlapping and contiguous windows, duplicate contributions, multiple lanes, stable ordering, empty contexts, invalid selected/contributing lanes, source-digest mismatch, and invalid references.
• Ownership tests prove output mutation cannot affect the source document or projector inputs, including nested metadata.
• Codec tests cover round trip, required arrays, schema identity, malformed and trailing JSON, unknown fixed fields, invalid ordering/ranges, mismatched source identities, and independently owned decoded metadata.
• Use structured assertions and a compact valid fixture; do not add a large transcript golden file.

Stage completion:

• go test ./internal/framework/evidencecontext

Stage 3: Integrate Projection With Runner Output

Extend contracts.OutputRequest with an optional EvidenceContext *SerializedArtifact field and clone it at every ownership handoff, following the existing chunk-map pointer pattern.

After lane execution and final manifest population, but before invoking the output encoder, the runner must:

1. skip all work when the prepared evidence plan is absent;
2. index accepted NormalizeOutputs by their globally unique lane IDs and fail on an internal duplicate rather than silently overwrite it;
3. for each selected lane with an output, verify its source and artifact kind, decode it through the prepared artifact codec registry, and invoke the prepared typed projector;
4. build and serialize the evidence document through the evidence-context package; and
5. pass a defensive serialized-artifact copy to the output encoder.

Selected lanes without normalized output contribute nothing. Normalize rejections remain successful pipeline outcomes; evidence projection does not inspect rejected candidates. An invalid accepted reference, incompatible serialized artifact, projection type failure, or evidence serialization failure is an output-stage framework error before logical files are returned or physically published.

At this external-content consumption boundary, do not propagate artifact-codec or metadata-cloning errors with %w when their text could contain artifact fields or source metadata. Return fixed, actionable categories scoped by lane and operation; detailed codec errors remain available to direct trusted callers and their focused tests.

Add an allowlisted debug summary containing only evidence artifact identity, selected lanes, window, context count, unit count, and source digest. Do not duplicate transcript text or source-unit metadata into a new evidence-specific debug envelope. Existing normalized-output debug behavior remains unchanged.

The output artifact is not a normalized lane, generated reference, checkpoint, or manifest normalized-output entry. It does not alter normalized-output, rejection, or warning counts. Resume continues to reuse normalized checkpoints; evidence is deterministically rebuilt during the always-executed output stage.

Stage tests:

• Runner tests use a real codec, evidence projector, and small capturing output encoder to prove selected-lane union, absent/rejected lane omission, invalid accepted-reference failure, output-request defensive ownership, and no work when disabled.
• Include one checkpoint-reused normalized-output case to prove evidence is reconstructed identically without retaining typed normalize values.
• Confirm projection failures prevent output encoding and return a failed manifest without changing rejection semantics.

Stage completion:

• go test ./internal/framework/pipeline

Stage 4: Publish Through The JSON Bundle

Teach the production JSON encoder to recognize the optional evidence-context artifact, verify its exact kind, media type, schema identity, schema digest, and payload validity through the evidence-context codec, and emit:

• logical file evidence-context.json; and
• optional index.json descriptor field evidence_context.

The descriptor uses the same six fields as chunk_map: artifact_kind, file, media_type, schema_id, schema_name, and schema_version. Refactor the encoder's private descriptor representation only as needed to share that shape; do not change the existing chunk_map wire contract. Evidence output is ordered with the encoder's other fixed logical files, remains a non-lane artifact, and is present with an empty contexts array when enabled but no selected lane produces references.

The JSON encoder's validation boundary returns a fixed content-safe evidence artifact error rather than propagating decoder or schema diagnostics that could echo transcript text or metadata. Direct evidence-context codec tests retain detailed structural errors.

Update the maintained complete D&D configuration to enable evidence context with window 3 for item-events, npcs, spells, combat-turns, and npc-interactions. Deliberately omit scene-descriptions. Keep the minimal configuration disabled by omission.

Stage tests:

• JSON encoder tests own descriptor shape, exact logical filename, identity checking, empty evidence publication, and disabled bundle stability.
• One assembled production D&D test uses multiple selected lanes with overlapping references and non-monotonic unit IDs, decodes the published artifact through its production codec, and proves union/deduplication and scene-description exclusion.
• A second narrow case explicitly allowlists a scene-description lane to prove capability is opt-in rather than hard-coded exclusion.
• Existing index, chunk-map, lane, manifest, warning, and rejection tests remain the owners of their current formats; do not repeat their full matrices.

Stage completion:

• go test ./internal/modules/generic/output/json
• go test ./internal/modules/dnd/...
• go test ./internal/modules/integration
• go test ./internal/cli

Stage 5: Publish Current-Behavior Documentation

After implementation and behavioral tests pass, update canonical documentation:

• docs/config.md owns the nested JSON output options, defaults, strict validation, required lane allowlist, and a small configuration snippet.
• A new docs/integrations/evidence-context.md owns the complete v1 payload, identities, direct-evidence versus context semantics, ordering, compatibility, and a compact valid example.
• docs/integrations/json-output.md owns the optional logical file and index.json descriptor; link to the evidence contract rather than repeating its payload.
• docs/operations.md owns durable source-content sensitivity, permissions, retention, and the possibility that selected lanes cover most of a transcript.
• docs/consumers/subprocess.md explains discovery through the optional index descriptor and requires consumers to treat evidence_refs, not expanded context bounds, as citations.
• docs/policy/architecture.md records the generic typed evidence-projection boundary and output ownership without adding D&D or wire-format detail.
• docs/internal/pipeline.md and docs/internal/modules.md describe the typed evidence registry, preparation checks, reconstruction from serialized normalize outputs, and output-stage ownership without restating public wire fields.

Update only the smallest orientation links needed for discoverability. Do not add a CLI flag, configuration environment override, or duplicate the complete configuration outside examples/.

After all current-behavior documentation is accurate:

• set the feature roadmap status to Implemented;
• set this plan's status to Completed; and
• leave the integration and configuration documents, not either roadmap, as the canonical implemented contract.

Final verification:

• git diff --check
• go test ./...
• go vet ./...
• go build ./cmd/notarius
• go test -race ./internal/framework/evidencecontext ./internal/framework/pipeline ./internal/modules/generic/output/json ./internal/modules/dnd/... ./internal/modules/integration ./internal/cli

Open Questions

None. The plan fixes the configuration shape and defaults, typed projection boundary, preparation timing, durable schema and identities, range-union algorithm, failure semantics, JSON discovery, D&D coverage, documentation ownership, and test boundaries.