notarius/docs/roadmap/implementation.md

# Published Evidence Context Implementation Plan

## Status

Ready for implementation.

## Objective

Implement the accepted [Published Evidence Context](evidence.md) roadmap as an
optional, deterministic extension of the production JSON output. The completed
work must publish one deduplicated source-context artifact for an explicit set
of successful normalized lanes while leaving existing lane payloads,
normalization, checkpointing, subprocess results, and disabled output bundles
unchanged.

Complete the stages below in order. Each stage must leave its affected packages
passing before the next begins. Do not implement per-record hydration, implicit
all-lane collection, a full-transcript mode, LLM processing, or any other
roadmap item marked out of scope.

## Decisions And Invariants

- Evidence publication is output policy. Normalizers continue to return
  semantic artifacts with precise source references and do not receive
  hydration responsibilities.
- The framework operates on the accepted generic `source.SourceDocument` and
  typed artifact projections. It must not inspect serialized JSON for
  `source_ref` or `source_refs`, and generic packages must not depend on D&D
  types.
- The selected lane allowlist is explicit, non-empty, and globally addressed by
  resolved lane ID. Pipeline resolution already guarantees lane IDs are unique
  across ordered steps.
- The framework decodes accepted serialized normalize outputs through their
  registered artifact codecs before invoking typed evidence projectors. This
  supports both fresh and checkpoint-reused normalize outputs without retaining
  a second typed result channel.
- Expansion uses source-document positions. Numeric unit IDs are identities,
  not sequence numbers.
- Direct source references are never widened or rewritten. Expanded ranges are
  context bounds only.
- Rejected, failed, and absent normalized lane outputs contribute no evidence.
- Evidence output is sensitive durable source content, not cache or debug
  state. It contains accepted source units only and never raw input bytes,
  prompts, model responses, auxiliary references, paths, or credentials.
- The optional `evidence_context` index field is an additive v1 JSON-bundle
  change. Existing D&D artifact schemas and the subprocess receipt do not
  change.

## Stage 1: Add Typed Evidence Capability And Resolve Output Policy

Add a dedicated artifact-evidence registry under the pipeline framework:

- `pipeline.ArtifactEvidenceRegistry` stores one typed projector per artifact
  kind.
- `pipeline.ArtifactEvidenceProjector[T]` is
  `func(T) []source.SourceRef`.
- `pipeline.RegisterArtifactEvidence[T](registry, kind, projector)` accepts a
  non-empty kind and non-nil projector, records the exact Go type for `T`, and
  rejects duplicate kinds.
- The erased projection boundary checks the exact registered type, invokes the
  projector, and returns a defensive copy of its references.
- The registry exposes only the discovery and projection operations required by
  resolution, preparation, and execution; do not expose its mutable entries.

Add the registry to `pipeline.Registries` and `pipeline.ModuleCatalog`, including
CLI catalog conversion, production construction, empty-set detection, and
test registry helpers. A nil evidence registry remains valid when evidence
publication is disabled. Production construction and the D&D registrar require
and populate it.

Define these framework-level output-policy contracts:

```go
type EvidenceContextPolicy struct {
    Enabled     bool
    WindowUnits int
    LaneIDs     []string
}

type EvidenceContextPolicyProvider interface {
    EvidenceContextPolicy() EvidenceContextPolicy
}
```

Provider and prepared-pipeline boundaries defensively copy `LaneIDs`.

Add an optional output-profile option-validation callback to
`OutputEncoderRegistry`:

```go
type OutputProfileOptionContext struct {
    LaneIDs []string
}

type OutputProfileOptionValidator func(
    OutputProfileOptionContext,
    map[string]any,
) error
```

Add `RegisterBuilderWithProfileValidation(spec, validateOptions,
validateProfile, builder)` and make existing output registration methods
delegate to it with no profile callback. The registry passes defensive copies
to validation. The callback receives the complete configured lane-ID set before
invocation-level `--only` filtering. The production JSON output uses it only to
prove that every configured evidence lane exists. Keep the extension generic:
the pipeline supplies lane identities, while the output module interprets its
own options. Build that lane set from the normalized legacy-or-steps profile
before selection and reject duplicate configured lane IDs through the existing
pipeline identity rules.

Extend the production JSON output options with the nested
`evidence_context` object:

- omission disables the feature;
- `enabled` is required when the object is present;
- `enabled: false` permits no `lanes` or `window_units` fields;
- `enabled: true` requires a non-empty `lanes` array;
- lane values are strings, trimmed, non-empty, unique after trimming, and
  normalized to lexical order;
- `window_units` is an optional non-negative integer with default `3`; and
- outer and nested unknown fields and incompatible YAML value types remain
  strict configuration errors.

The JSON encoder implements the policy provider from its decoded immutable
options. Pipeline resolution invokes its profile validator against all
configured steps, so an unknown evidence lane fails even when another lane is
selected with `--only`.

During `pipeline.Prepare`, after constructing the output encoder:

1. obtain and defensively normalize an enabled policy;
2. intersect its configured IDs with the effective prepared lanes, treating
   allowlisted lanes removed by invocation-level filtering as inactive;
3. require an artifact-evidence registration for each active lane kind;
4. prove that its projector Go type exactly matches the active lane's registered
   artifact codec type; and
5. retain an immutable private evidence plan on `PreparedPipeline`.

Duplicate or empty provider values, a missing evidence registry for an active
lane, unsupported active artifact kinds, and type mismatches fail preparation
with pipeline/output/lane context. A disabled or non-participating output
encoder creates no evidence plan and preserves existing preparation behavior.
The private plan retains both the full configured allowlist for publication and
the active lane/projector intersection for execution.

Register D&D evidence projectors for all six current artifact kinds. Each
projector returns copies of the artifact's direct references in record order:
spells, NPCs, combat turns, item events, NPC interactions, and the singular
reference from each scene description. Scene descriptions gain capability but
remain excluded unless their configured lane ID is allowlisted.

Stage tests:

- Registry tests cover nil, blank, duplicate, exact-type, defensive-copy, and
  deterministic discovery behavior.
- JSON option tests cover disabled, enabled/default-window, explicit zero
  window, normalization, duplicates, unknown fields, and invalid types.
- Preparation tests cover selected lanes across steps, unknown lanes,
  unsupported kinds, projector/codec type mismatch, disabled behavior, and
  defensive policy ownership.
- Resolution/preparation tests prove a valid allowlist survives `--only`, an
  excluded lane contributes no active projector, and a genuinely unknown
  configured lane still fails profile resolution.
- D&D registration tests prove every production D&D artifact kind has the
  expected evidence capability without testing individual field loops
  redundantly.
- One table-driven D&D projector test supplies representative values for all
  six artifact kinds and proves plural and singular references are copied
  without aliasing or semantic rewriting.

Stage completion:

- `go test ./internal/framework/pipeline`
- `go test ./internal/modules/generic/output/json`
- `go test ./internal/modules/dnd/register`
- `go test ./internal/cli`

## Stage 2: Define And Build The Evidence-Context Artifact

Add a domain-neutral `internal/framework/evidencecontext` package that owns the
durable model, JSON Schema, strict codec, projection algorithm, and these exact
identities:

- artifact kind `source/evidence-context`;
- media type `application/json`;
- schema ID `notarius.source.evidence_context`;
- schema name `notarius_source_evidence_context_v1`; and
- schema version `v1`.

Use these package-level model and build contracts:

```go
type Document struct {
    SourceID      string
    SourceDigest  string
    WindowUnits   int
    SelectedLanes []string
    Contexts      []Context
}

type Context struct {
    ContextRef  source.SourceRef
    EvidenceRefs []EvidenceRef
    Units       []source.SourceUnit
}

type EvidenceRef struct {
    LaneID   string
    SourceRef source.SourceRef
}

type LaneEvidence struct {
    LaneID     string
    SourceRefs []source.SourceRef
}

type BuildRequest struct {
    Source        *source.SourceDocument
    WindowUnits   int
    SelectedLanes []string
    LaneEvidence  []LaneEvidence
}
```

Apply the JSON field names shown below. Provide `Build(BuildRequest)`,
`Serialize(BuildRequest)`, and a `Codec` with the same identity/encode/decode
responsibilities as the chunk-map codec.

The v1 payload has this exact shape:

```json
{
  "source_id": "session-alpha",
  "source_digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
  "window_units": 0,
  "selected_lanes": ["npcs"],
  "contexts": [
    {
      "context_ref": {
        "source_id": "session-alpha",
        "start_unit_id": 13,
        "end_unit_id": 13
      },
      "evidence_refs": [
        {
          "lane_id": "npcs",
          "source_ref": {
            "source_id": "session-alpha",
            "start_unit_id": 13,
            "end_unit_id": 13
          }
        }
      ],
      "units": [
        {
          "id": 13,
          "kind": "transcript_segment",
          "text": "The party meets Rowan.",
          "ref": {
            "source_id": "session-alpha",
            "start_unit_id": 13,
            "end_unit_id": 13
          }
        }
      ]
    }
  ]
}
```

All displayed fields are required. `selected_lanes`, `contexts`,
`evidence_refs`, and `units` encode as arrays rather than `null`; `contexts`
may be empty. Each unit uses the existing `source.SourceUnit` JSON shape with
required `id`, `kind`, `text`, and `ref`, plus optional JSON-shaped `metadata`.
Fixed objects reject unknown fields; metadata remains an arbitrary JSON object.

The builder accepts the validated source document, selected lane IDs, effective
window, and lane-attributed direct references, then:

1. requires a non-negative window and a non-empty, trimmed, unique selected
   lane set, then stores that set in lexical order;
2. requires every `LaneEvidence.LaneID` to belong to the selected set;
3. validates the source document, recomputes its semantic digest, and requires
   it to equal `SourceDocument.Digest`;
4. validates every direct reference against one `source.DocumentIndex`;
5. deduplicates exact `(lane_id, source_ref)` contributions;
6. resolves endpoints to document positions;
7. expands each side without integer overflow and clips at document bounds;
8. sorts by expanded document position with deterministic lane/reference
   tie-breakers;
9. merges overlapping or position-contiguous expanded intervals;
10. unions and deterministically sorts each merged context's direct
   contributions; and
11. deep-clones the corresponding source units and JSON-shaped metadata.

Contexts are disjoint and ordered by document position, so a source unit occurs
at most once in the document. `context_ref` identifies the first and last
included units; `evidence_refs` retains only original citations. An empty
reference collection produces the same source identity, window, sorted
allowlist, and an explicit empty contexts array.

Projection failures identify only structural scope such as lane and reference
position. They must not include source text, metadata values, raw serialized
artifacts, or unrelated paths.

The codec must validate its model before encoding, produce deterministic JSON,
strictly decode the checked-in schema, and return independently owned values.
The JSON output encoder remains responsible for pretty-printing the logical
file with its standard trailing newline. Follow the existing chunk-map
package's separation between model, builder, codec, schema asset, and contract
tests where useful, without coupling the two artifact formats.

Stage tests:

- A table-driven builder suite covers zero and nonzero windows, boundary
  clipping, non-monotonic unit IDs, separate gaps, overlapping and contiguous
  windows, duplicate contributions, multiple lanes, stable ordering, empty
  contexts, invalid selected/contributing lanes, source-digest mismatch, and
  invalid references.
- Ownership tests prove output mutation cannot affect the source document or
  projector inputs, including nested metadata.
- Codec tests cover round trip, required arrays, schema identity, malformed and
  trailing JSON, unknown fixed fields, invalid ordering/ranges, mismatched
  source identities, and independently owned decoded metadata.
- Use structured assertions and a compact valid fixture; do not add a large
  transcript golden file.

Stage completion:

- `go test ./internal/framework/evidencecontext`

## Stage 3: Integrate Projection With Runner Output

Extend `contracts.OutputRequest` with an optional
`EvidenceContext *SerializedArtifact` field and clone it at every ownership
handoff, following the existing chunk-map pointer pattern.

After lane execution and final manifest population, but before invoking the
output encoder, the runner must:

1. skip all work when the prepared evidence plan is absent;
2. index accepted `NormalizeOutputs` by their globally unique lane IDs and fail
   on an internal duplicate rather than silently overwrite it;
3. for each selected lane with an output, verify its source and artifact kind,
   decode it through the prepared artifact codec registry, and invoke the
   prepared typed projector;
4. build and serialize the evidence document through the evidence-context
   package; and
5. pass a defensive serialized-artifact copy to the output encoder.

Selected lanes without normalized output contribute nothing. Normalize
rejections remain successful pipeline outcomes; evidence projection does not
inspect rejected candidates. An invalid accepted reference, incompatible
serialized artifact, projection type failure, or evidence serialization failure
is an output-stage framework error before logical files are returned or
physically published.

At this external-content consumption boundary, do not propagate artifact-codec
or metadata-cloning errors with `%w` when their text could contain artifact
fields or source metadata. Return fixed, actionable categories scoped by lane
and operation; detailed codec errors remain available to direct trusted
callers and their focused tests.

Add an allowlisted debug summary containing only evidence artifact identity,
selected lanes, window, context count, unit count, and source digest. Do not
duplicate transcript text or source-unit metadata into a new evidence-specific
debug envelope. Existing normalized-output debug behavior remains unchanged.

The output artifact is not a normalized lane, generated reference, checkpoint,
or manifest normalized-output entry. It does not alter normalized-output,
rejection, or warning counts. Resume continues to reuse normalized checkpoints;
evidence is deterministically rebuilt during the always-executed output stage.

Stage tests:

- Runner tests use a real codec, evidence projector, and small capturing output
  encoder to prove selected-lane union, absent/rejected lane omission, invalid
  accepted-reference failure, output-request defensive ownership, and no work
  when disabled.
- Include one checkpoint-reused normalized-output case to prove evidence is
  reconstructed identically without retaining typed normalize values.
- Confirm projection failures prevent output encoding and return a failed
  manifest without changing rejection semantics.

Stage completion:

- `go test ./internal/framework/pipeline`

## Stage 4: Publish Through The JSON Bundle

Teach the production JSON encoder to recognize the optional evidence-context
artifact, verify its exact kind, media type, schema identity, schema digest, and
payload validity through the evidence-context codec, and emit:

- logical file `evidence-context.json`; and
- optional `index.json` descriptor field `evidence_context`.

The descriptor uses the same six fields as `chunk_map`:
`artifact_kind`, `file`, `media_type`, `schema_id`, `schema_name`, and
`schema_version`. Refactor the encoder's private descriptor representation only
as needed to share that shape; do not change the existing `chunk_map` wire
contract. Evidence output is ordered with the encoder's other fixed logical
files, remains a non-lane artifact, and is present with an empty contexts array
when enabled but no selected lane produces references.

The JSON encoder's validation boundary returns a fixed content-safe evidence
artifact error rather than propagating decoder or schema diagnostics that could
echo transcript text or metadata. Direct evidence-context codec tests retain
detailed structural errors.

Update the maintained complete D&D configuration to enable evidence context
with window `3` for `item-events`, `npcs`, `spells`, `combat-turns`, and
`npc-interactions`. Deliberately omit `scene-descriptions`. Keep the minimal
configuration disabled by omission.

Stage tests:

- JSON encoder tests own descriptor shape, exact logical filename, identity
  checking, empty evidence publication, and disabled bundle stability.
- One assembled production D&D test uses multiple selected lanes with
  overlapping references and non-monotonic unit IDs, decodes the published
  artifact through its production codec, and proves union/deduplication and
  scene-description exclusion.
- A second narrow case explicitly allowlists a scene-description lane to prove
  capability is opt-in rather than hard-coded exclusion.
- Existing index, chunk-map, lane, manifest, warning, and rejection tests remain
  the owners of their current formats; do not repeat their full matrices.

Stage completion:

- `go test ./internal/modules/generic/output/json`
- `go test ./internal/modules/dnd/...`
- `go test ./internal/modules/integration`
- `go test ./internal/cli`

## Stage 5: Publish Current-Behavior Documentation

After implementation and behavioral tests pass, update canonical documentation:

- `docs/config.md` owns the nested JSON output options, defaults, strict
  validation, required lane allowlist, and a small configuration snippet.
- A new `docs/integrations/evidence-context.md` owns the complete v1 payload,
  identities, direct-evidence versus context semantics, ordering,
  compatibility, and a compact valid example.
- `docs/integrations/json-output.md` owns the optional logical file and
  `index.json` descriptor; link to the evidence contract rather than repeating
  its payload.
- `docs/operations.md` owns durable source-content sensitivity, permissions,
  retention, and the possibility that selected lanes cover most of a
  transcript.
- `docs/consumers/subprocess.md` explains discovery through the optional index
  descriptor and requires consumers to treat `evidence_refs`, not expanded
  context bounds, as citations.
- `docs/policy/architecture.md` records the generic typed evidence-projection
  boundary and output ownership without adding D&D or wire-format detail.
- `docs/internal/pipeline.md` and `docs/internal/modules.md` describe the typed
  evidence registry, preparation checks, reconstruction from serialized
  normalize outputs, and output-stage ownership without restating public wire
  fields.

Update only the smallest orientation links needed for discoverability. Do not
add a CLI flag, configuration environment override, or duplicate the complete
configuration outside `examples/`.

After all current-behavior documentation is accurate:

- set [the feature roadmap](evidence.md) status to `Implemented`;
- set this plan's status to `Completed`; and
- leave the integration and configuration documents, not either roadmap, as
  the canonical implemented contract.

Final verification:

- `git diff --check`
- `go test ./...`
- `go vet ./...`
- `go build ./cmd/notarius`
- `go test -race ./internal/framework/evidencecontext ./internal/framework/pipeline ./internal/modules/generic/output/json ./internal/modules/dnd/... ./internal/modules/integration ./internal/cli`

## Open Questions

None. The plan fixes the configuration shape and defaults, typed projection
boundary, preparation timing, durable schema and identities, range-union
algorithm, failure semantics, JSON discovery, D&D coverage, documentation
ownership, and test boundaries.