ConfigKit/docs/roadmap/diff.md

# Effective Configuration Diff Roadmap

Status: implemented. This document preserves the selected contract and scope;
the [configuration contract](../config.md#effective-configuration-diffs) owns
current behavior, and the [implementation record](implementation.md) preserves
the completed sequence and validation evidence.

## Targeted end state

ConfigKit compares the canonical effective inspection values retained by two
successful loads and returns structured differences. Consumers decide how to
render those differences: a changes table, a tree, or a changed-region diff are
possible presentations. ConfigKit owns comparison and safe value disclosure,
while applications own presentation and labels.

Both configurations have completed the normal loading lifecycle, including
typed decoding, resolution, secret lookup, and validation. Invalid input on
either side produces the existing load failure and diagnostics, with no usable
diff. Comparing successful results does not rerun callbacks, reopen sources, or
resolve credentials again. Later mutation of caller-owned `Result.Config` does
not change the frozen values being compared.

The comparison is over retained inspection values, not every value accessible
through the returned Go configuration. In particular, typed secret references
contribute their configured environment names, not resolved credential bytes.

The motivating workflows are:

- Load `app.conf.bak` and `app.conf`, then inspect effective differences.
- Load a base configuration and separately load it with a selected overlay,
  then inspect the overlay's effective impact, including subsequent resolution.

An overlay alone is not the right side of the second workflow: omitted overlay
keys retain base values. Both workflows use successful, independent loads;
applications control their definitions, options, and external inputs. Using the
same definition and controlled inputs makes comparisons easier to interpret.
Sharing a Go type does not prove equal defaults or callbacks, and comparison
does not require callback-function identity checks.

## Selected public contract

### Entry point and records

The target entry point is:

```go
Diff[T any](left, right *Result[T]) ([]Difference, error)
```

It compares results of the same configuration type. Nil or uninitialized results
produce an error rather than an empty successful diff. Valid results with equal
effective values produce no difference records. Loading remains a separate
operation; this API does not aggregate load failures or orchestrate callbacks.

The root-owned `Difference` record contains:

- `Path Path`: the semantic path of the changed region;
- `Left ValueSnapshot`: the safe value before the change;
- `Right ValueSnapshot`: the safe value after the change; and
- `HiddenChanged bool`: whether at least one actual change in the region is
  withheld by diff redaction, as defined below.

Both sides reuse the existing `ValueSnapshot` representation: `Present`,
`Redacted`, and deterministic `YAML`. Absence is `Present=false` with empty
YAML; explicit null is present and renders as `null`. Presence identifies
additions and removals. A record with both sides present describes a replacement
at its reported path. Record membership establishes a change even when both
safe snapshots render identically.

Paths and records are detached. Consumer mutation does not affect subsequent
comparisons or either result's existing inspection state. The public facade
exposes no internal document or parser types.

### Equality, granularity, and ordering

Compare the underlying canonical effective inspection values before redaction.
Equal values produce no differences even if source locations, explicit-input
status, or loading histories differ. An explicit zero and an omitted field
defaulting to zero therefore compare equally. Presence and provenance remain
separately inspectable through each result's existing APIs. Original comments,
formatting, and mapping declaration order do not participate in equality.

Retain the existing comparator's granularity:

- Mappings compare recursively. Additions and removals report the affected key;
  changes beneath mappings report the changed descendant region.
- Sequences compare atomically: any change reports the sequence path.
- Scalar, cross-kind, and null/container replacements report the affected path.

Sequence comparison introduces no element alignment or public correspondence
between old and new indices. Records are ordered by canonical pointer spelling,
matching the existing comparator. Paths retain semantic key/index intent and
proper escaping, including numeric mapping keys.

### Cross-configuration redaction

Apply the applicable sensitivity from both loads before returning either side's
value. A classification on either side protects both snapshots. Include
historical and dependency-derived classification, classified ancestors and
descendants, and paths absent from one effective tree.

Where the two shapes permit corresponding sensitive regions to be identified,
mask those regions while retaining public sibling values. Do not treat matching
pointer text alone as proof of matching mapping-key versus sequence-index shape.

When a structural replacement prevents safe correspondence for a classified
region, mask the complete reported replacement region on both present sides.
For example, replacing a mapping containing a classified descendant with a
scalar masks the whole mapping/scalar replacement, rather than trying to apply
the old descendant path to the scalar. This fallback also covers incompatible
mapping/sequence shapes within an atomic replacement. An absent side remains
absent; redaction does not fabricate a value for it.

`ValueSnapshot.Redacted` describes masking, not change. `HiddenChanged` is true
when underlying value or presence differences occur in a region withheld by the
combined redaction policy. It is false when only public content changes and all
masked regions retain equal values and presence. This includes atomic sequence
records: an unchanged masked member does not alone set the flag. The complete
replacement fallback sets the flag because the changed region itself is
withheld. The flag does not identify individual changed children, counts, or
sequence correspondence, and does not assert that every redacted child changed.

For a wholly masked changed value, record membership and `HiddenChanged` let a
consumer show `[REDACTED]` on the left and `[CHANGED]` on the right. These
strings are presentation choices, not configuration values or the encoding of
change state. Literal marker-looking input remains ordinary input; applications
use record metadata rather than interpreting YAML marker text.

Errors and returned records disclose neither classified values nor credential
bytes.

### Credential boundary and renderer context

Compare classified values and configured secret reference names retained in the
effective inspection snapshots. Rotation of a resolved credential behind an
unchanged reference is outside this comparison. Diff does not read mutable
`Result.Config`, reread the environment, or introduce credential retention or
credential-derived comparison tokens.

Return changed regions only. The records do not include paired full-document
context or a structured dynamic value tree, and do not alone supply unchanged
context for unified hunks. Calling `WriteEffective` independently on each side
is not a substitute for cross-protected context because each call applies only
that result's sensitivity. Consumers can render the returned safe regions
without reconstructing unredacted context.

## Scope and architectural fit

This extends ConfigKit's existing snapshot-based inspection responsibilities.
Applications can compare completed configurations without duplicating structural
comparison, presence handling, or safe value disclosure. Loading and composition
rules remain unchanged.

The feature does not include input-text comparison, invalid-document inspection,
patch application, generating a reconstructing overlay, fingerprints, automatic
reload, resolved-credential comparison, cross-type comparison, or a built-in
renderer. A diff may describe removals that existing overlay semantics cannot
express. These boundaries define this feature's selected scope, not intrinsic
correctness restrictions on every possible future extension.

## Implementation references

- [Deferred scope](deferred.md) continues to exclude fingerprints and unrelated
  items; effective configuration diffs are implemented.
- The [configuration contract](../config.md#typed-projection-and-ownership)
  describes canonical typed projection and private structural comparison.
  [The existing comparator](../../internal/engine/document_compare.go) reports
  mapping additions, removals, and replacements, ignores mapping order, treats
  sequences atomically, and sorts changes by pointer spelling. It ignores
  provenance and sensitivity metadata.
- Results retain private frozen inspection state. Existing
  [inspection types](../../inspection.go) include `ValueSnapshot`, with
  presence, redaction, and deterministic YAML. Explanation records describe
  one load's history rather than a comparison between loads.
- [Secret classification](../config.md#secret-classification-and-effective-rendering)
  includes historical classification and dependency propagation. Existing
  snapshots apply one load's sensitivity; diff requires protection from both.
- [Typed secret references](../config.md#typed-secret-references) retain reference
  names in inspection, not resolved credential bytes.
- The [public facade](../internal/overview.md) is root-owned and delegates to
  the engine. The new operation and records follow that same boundary.

## Implementation outcome

The accepted inspection boundary is recorded in
[ADR 0010](../adr/0010-effective-configuration-diff.md), and the deferred
inventory retains fingerprints while recognizing implemented diffs. The engine
owns canonical comparison and paired disclosure in
[internal/engine/diff.go](../../internal/engine/diff.go); the root facade exposes
detached records in [diff.go](../../diff.go). The configuration
contract, consumer guidance, component overview, architecture policy, and
development guide now link to their respective behavior and implementation
owners.

Validation and coverage are recorded in the completed
[implementation plan](implementation.md). The implementation compares frozen
effective values, protects both sides with combined sensitivity, retains the
selected credential boundary, and provides executable file and overlay
workflows. Publication and release preparation remain separate.