notarius/docs/roadmap/implementation.md

# ADR-0005 Implementation Record

This document records the implementation of the target state in
[ADR-0005 Feature Roadmap](adr0005.md), governed by
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md).

Stages 1 through 8 summarize the completed initial implementation. Stages 9
through 12 track pending remediation identified by the post-implementation
review. Current behavior is documented in the canonical references linked from
[Development](../development.md); those references must not describe the Stages
9 through 12 target state until the corresponding stage is complete.

## Execution Rules

Before every stage:

1. read [Development](../development.md), both documents under `docs/policy/`,
   ADR-0005, this plan, and every stage-specific document named below;
2. inspect `git status` and preserve user changes;
3. inspect the focused package contracts and tests before editing; and
4. confirm that all earlier stages are complete.

During every stage:

- implement only that stage and prerequisites discovered to be inseparable;
- use existing package boundaries and helpers before introducing abstractions;
- keep framework and core packages independent of concrete D&D modules;
- preserve cancellation, deterministic ordering, redaction, path confinement,
  and atomic-write invariants;
- add or update focused tests with every behavior change;
- do not update current-behavior documentation before the final documentation
  stage, except for ADR acceptance in Stage 1; and
- do not begin the next stage after meeting the current exit criteria.

At the end of every stage:

1. run the focused tests listed for that stage;
2. run `go test ./...`;
3. run `go vet ./...`;
4. run `go build ./cmd/notarius`;
5. run `git diff --check`; and
6. mark the stage complete in this document only after all checks pass.

If repository reality conflicts with this plan, stop and update the plan in the
same change before implementing a materially different design. Do not silently
invent a new contract.

## Fixed Decisions

These choices are settled for this implementation:

- `source.SourceDocument.Digest` is the cache lookup identity.
- The chunk-plan root is independent of `workspace.directory` and every other
  state root.
- Its default joins the directory returned by `os.UserCacheDir` with
  `notarius/chunk-plans`.
- `workspace.chunk_cache.directory` overrides that default and names the
  chunk-plan root itself. The recommended value for a system-wide Linux service
  is `/var/cache/notarius/chunk-plans`.
- The plan file is
  `<chunk-plan-root>/<64-character-lowercase-source-sha256-hex>/plan.json`. The
  validated `sha256:` prefix remains part of the logical digest but is removed
  from the path segment.
- The plan envelope schema is `notarius.chunk-plan.v1`.
- There is one mutable plan file per source, atomically replaced. There are no
  immutable plan objects, pointers, history, locks, or rollback.
- Concurrent writers use last-successful-atomic-write semantics; they must
  never expose a partial file.
- Chunk modules retain the `contracts.Chunker` name but replace `Chunk` with a
  `Plan` operation. No legacy interface remains after Stage 2.
- Plans and ranges use `map[string]json.RawMessage` annotations. Annotation
  values are canonical valid JSON and namespaces are non-empty trimmed strings.
- Materialized chunks expose range annotations through
  `source.Chunk.Annotations` and plan annotations through
  `source.Chunk.PlanAnnotations`, never through `Metadata`.
- Framework materialization owns chunk IDs, indexes, references, content,
  media type, units, and generic metadata.
- Only the generic `chunks` capability is hard; `chunks.scenes` is removed.
- Cached-plan hits still construct the configured chunker during preparation
  but never call its operation or the chunk-stage LLM.
- The current pipeline's configured chunk validators run against materialized
  chunks on every path.
- Canonical plan storage is the only chunk-reuse mechanism; chunk checkpoint
  APIs and payloads are removed.
- The mode values are `auto`, `bypass`, and `refresh`. They are valid in config,
  environment, and CLI, including persistent `refresh` configuration.
- The YAML field is `workspace.chunk_cache.mode`.
- The YAML directory field is `workspace.chunk_cache.directory`.
- The environment variable is `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`.
- The root environment variable is `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`.
- The CLI flag is `--chunk_cache <auto|bypass|refresh>`.
- Precedence is CLI, environment, file, then default `auto`.
- Chunk-plan-root precedence is environment, file, then the per-user default;
  there is no CLI root override.
- Every supplied file, environment, and CLI value is parsed strictly even when
  a higher-precedence source would override it; precedence selects among valid
  values and does not mask malformed configuration.
- Existing chunk checkpoint files are ignored and never migrated.
- A structurally invalid stored plan is an `auto` miss. A configured-validator
  rejection of a structurally valid hit does not trigger regeneration.

## Planned Contracts

Use these names unless an existing collision requires the smallest obvious
adjustment.

### Source-zone types

Add to `internal/core/source`:

```go
type ChunkAnnotations map[string]json.RawMessage

type ChunkPlan struct {
    SourceDigest string           `json:"source_digest"`
    Ranges       []ChunkRange     `json:"ranges"`
    Annotations  ChunkAnnotations `json:"annotations,omitempty"`
}

type ChunkRange struct {
    StartUnitID int              `json:"start_unit_id"`
    EndUnitID   int              `json:"end_unit_id"`
    Annotations ChunkAnnotations `json:"annotations,omitempty"`
}
```

Add `Annotations ChunkAnnotations` and
`PlanAnnotations ChunkAnnotations` to `source.Chunk`, with JSON names
`annotations` and `plan_annotations` and `omitempty` on both.

The plan digest covers `SourceDigest`, ordered ranges, and canonical annotation
bytes. It excludes producer provenance, warnings, timestamps, and storage
schema. Materialization produces:

- IDs `chunk-%06d`, numbered from one;
- zero-based indexes in plan order;
- references spanning the first and last units in each range;
- cloned contiguous source units;
- media type `application/json`;
- canonical JSON content shaped as `{"units":[...]}`;
- generic metadata containing `start_unit_id`, `end_unit_id`, and `unit_count`;
- cloned range annotations in `Chunk.Annotations`; and
- cloned plan annotations in `Chunk.PlanAnnotations`.

The plan and range scopes remain distinct even when both use the same namespace.
Copying plan annotations into each runtime chunk is intentional: it makes them
available to downstream chunk-scoped operations without duplicating source
content in durable storage.

Framework validation permits gaps and overlap between ranges, but requires
strictly increasing start positions and rejects duplicate or backward ranges.
Producing modules may impose stricter policy before returning a plan.

### Module contract

Replace the current chunk result and operation with:

```go
type ChunkPlanResult struct {
    Plan     source.ChunkPlan
    Warnings []Warning
}

type Chunker interface {
    Key() string
    ReferenceSlots() []ReferenceSlot
    Plan(context.Context, ChunkRequest) (ChunkPlanResult, error)
}
```

Keep `ChunkRequest`, chunk module keys, stage name `chunk`, configuration
bindings, registries, and preparation ownership unchanged.

### Durable plan record

Define framework-facing record and store contracts beside other pipeline
collaborator contracts:

```go
type ChunkPlanProducer struct {
    InputModule  string
    ChunkModule  string
    LLMProfile   string
    References   []artifacts.ReferenceProvenance
    Metadata     map[string]any
}

type ChunkPlanRecord struct {
    SchemaVersion string
    SourceDigest  string
    PlanDigest    string
    Plan          source.ChunkPlan
    Producer      ChunkPlanProducer
    Warnings      []contracts.Warning
    CreatedAt     time.Time
}

type ChunkPlanStore interface {
    Load(sourceDigest string) (ChunkPlanRecord, ChunkPlanDecision, error)
    Save(ChunkPlanRecord) error
}

type ChunkPlanStoreFactory func(root string) (ChunkPlanStore, error)

type ChunkPlanDecision struct {
    Status ChunkPlanStatus
    Reason string
}

type ChunkPlanStatus string

const (
    ChunkPlanHit     ChunkPlanStatus = "hit"
    ChunkPlanMissing ChunkPlanStatus = "missing"
    ChunkPlanInvalid ChunkPlanStatus = "invalid"
)
```

`ChunkPlanDecision` reports `hit`, `missing`, or `invalid` plus a redacted
reason. A miss or invalid result returns a zero record and nil error. Missing
and structurally invalid records are recoverable in `auto`. Filesystem access
failures other than absence return an error and are framework failures.

The concrete constructor is:

```go
// Package chunkplan
func NewFilesystemStore(root string) (pipeline.ChunkPlanStore, error)
```

It treats `root` as the exact chunk-plan root. CLI composition injects this
constructor through `cli.Options.ChunkPlanStoreFactory`; tests may inject a
recording factory. `cli.Options.UserCacheDir` has signature
`func() (string, error)` and defaults to `os.UserCacheDir`.

The JSON envelope uses snake-case equivalents of the record fields. It stores
references as provenance only, never reference content. Before returning a hit,
the loader validates schema version, source digest, plan digest, plan structure,
annotation JSON, and materializability against the current source in the runner.

### Cache policy and provenance

Add a `ChunkCacheMode` enum with `auto`, `bypass`, and `refresh`, plus strict
parsing and validation.

Add to `RunInput`:

```go
ChunkCacheMode ChunkCacheMode
ChunkPlans     ChunkPlanStore
```

An empty mode used by direct framework callers means `bypass`; the production
CLI always supplies the effective non-empty mode.

Add a run-manifest summary:

```go
type ChunkPlanManifest struct {
    Mode                string                `json:"mode"`
    Action              string                `json:"action,omitempty"`
    SourceDigest        string                `json:"source_digest,omitempty"`
    PlanDigest          string                `json:"plan_digest,omitempty"`
    PlanSchemaVersion   string                `json:"plan_schema_version,omitempty"`
    RequestedModule     string                `json:"requested_module"`
    ProducerInputModule string                `json:"producer_input_module,omitempty"`
    ProducerModule      string                `json:"producer_module,omitempty"`
    ProducerLLMProfile  string                `json:"producer_llm_profile,omitempty"`
    ProducerReferences  []ReferenceProvenance `json:"producer_references,omitempty"`
    ProducerMetadata    map[string]any        `json:"producer_metadata,omitempty"`
    CreatedAt           *time.Time            `json:"created_at,omitempty"`
}
```

`Action` is one of `reused`, `generated`, `refreshed`, or `bypassed`. Retain
the existing top-level `input_module` and `chunker` fields as the requested
pipeline modules. Add this field to `artifacts.RunManifest`:

```go
ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"`
```

The runner initializes it with the effective mode and requested chunk module,
then fills candidate and producer fields when a plan reaches materialization.

## Completed Stage Summaries

### Stage 1: Finalize ADR and add the source-zone plan model

**Status:** Complete

Accepted ADR-0005 and added the domain-neutral chunk-plan, range, annotation,
validation, digest, cloning, and deterministic materialization primitives in
`internal/core/source`.

### Stage 2: Replace the chunk operation with plan generation

**Status:** Complete

Replaced chunk-returning module operations with `Chunker.Plan`, converted the
generic and D&D scene chunkers to produce ranges and optional annotations, and
made the runner materialize and validate generated plans.

### Stage 3: Add dedicated cache configuration and the plan store

**Status:** Complete

Added strict cache modes, independent per-user or explicitly configured
chunk-plan roots, the source-addressed filesystem store, restrictive
permissions, envelope validation, and atomic replacement.

### Stage 4: Integrate cache policy and remove chunk checkpoints

**Status:** Complete

Implemented runner semantics for `auto`, `bypass`, and `refresh`; retained
current chunk-validator behavior; made accepted canonical plans the sole chunk
reuse mechanism; and removed chunk checkpoint loading and recording.

### Stage 5: Wire persistent policy into the CLI

**Status:** Complete

Added CLI, environment, and file configuration precedence; per-user root
resolution; persistent store construction; bypass isolation; and cross-run
reuse through normal CLI execution.

### Stage 6: Expose producer provenance and safe diagnostics

**Status:** Complete

Added requested-versus-producer chunk-plan provenance to manifests and a
payload-free chunk-plan decision summary to normal diagnostics, with complete
plan material confined to opt-in debug artifacts.

### Stage 7: Add cross-run, corruption, and concurrency hardening

**Status:** Complete

Added integration, corruption, interruption, concurrency, compatibility,
cross-domain annotation, import-boundary, and race coverage for the implemented
cache design.

### Stage 8: Publish current-behavior documentation

**Status:** Complete

Updated architecture, pipeline, module, configuration, CLI, operations,
diagnostics, integration, and overview documentation for the implemented
feature and its per-user and system-wide deployment guidance.

## Stage 9: Redact invalid cache-record diagnostics

**Status:** Pending

### Objective

Ensure malformed or incompatible cache contents cannot enter normal errors,
warnings, manifests, or diagnostics while retaining useful closed lookup
decisions.

### Read first

- [Architecture Policy](../policy/architecture.md), especially secret handling
- [Diagnostics Internals](../internal/diagnostics.md)
- `internal/framework/chunkplan/store.go` and its focused tests
- `internal/framework/pipeline/runner_chunk_plan.go` and its focused tests
- `internal/core/artifacts/artifacts.go`
- the CLI path that writes `chunk-plan.json`

### Implement

1. Stop interpolating decoder errors, unknown field names, schema values,
   annotation namespaces, timestamps, paths, or any other stored content into
   recoverable `ChunkPlanDecision.Reason` values.
2. At the normal-diagnostic boundary, ignore the reason supplied by a
   `ChunkPlanStore` and derive `ChunkPlanSummary.LookupReason` solely from
   status using exactly:
   - `hit`: `stored chunk plan is valid`;
   - `missing`: `chunk plan not found`;
   - `invalid`: `stored chunk plan is invalid`; and
   - `skipped`: `chunk plan lookup skipped`.
3. Keep operational read failures as framework errors rather than recoverable
   invalid-record decisions. Their normal error text must identify the failed
   operation without including cache-file content.
4. Do not add raw invalid-record details to another default surface. Direct
   operator inspection or a future explicitly sensitive debug facility may
   expose them, but this stage does not need to add such a facility.

### Tests

Add focused tests proving that sentinel secrets placed in unknown field names,
schema values, annotation namespaces, timestamps, and malformed trailing JSON
appear in neither the store decision nor `chunk-plan.json`. Add a runner test
with a custom store whose reason contains a sentinel and prove that the summary
uses the fixed status-derived reason. Preserve useful missing, hit, invalid, and
operational-error classification.

Run focused tests for `internal/framework/chunkplan`,
`internal/framework/pipeline`, and `internal/cli`, then perform the
repository-wide validation in the execution rules.

### Exit criteria

Every normal lookup reason belongs to the fixed safe vocabulary, hostile
invalid-file content cannot reach default diagnostics, operational failures
remain correctly classified, and all focused and repository-wide checks pass.

## Stage 10: Enforce rooted chunk-plan filesystem access

**Status:** Pending

### Objective

Make the cache's path-confinement guarantee resistant to symlinked cache entries
while preserving atomicity, permissions, and concurrent access semantics.

### Read first

- [Architecture Policy](../policy/architecture.md), especially state and path
  invariants
- [Operations](../operations.md)
- `internal/framework/chunkplan/store.go` and all store tests
- Go's `os.Root` API for the repository's declared Go version

### Implement

1. Treat the configured cache root as the filesystem trust boundary. A symlink
   in the operator-supplied root path itself may be followed when the root is
   opened, but every store-owned operation after that point must remain beneath
   the opened root.
2. Perform digest-directory, plan-file, temporary-file, and rename operations
   through `os.Root` or an equivalently race-resistant directory-relative
   mechanism. Do not rely on `filepath.Join` plus a preflight `Lstat` as the
   confinement boundary.
3. Reject symlinks and unexpected file types at the store-owned digest-directory
   and `plan.json` positions. A missing cache root on `Load` remains a normal
   miss and is not created; `Save` creates the root and digest directory as
   needed.
4. Preserve the exact
   `<root>/<full-source-sha256-hex>/plan.json` layout, `0700` directories,
   `0600` plan files, strict digest validation, and no implicit path suffix.
5. Publish using an exclusively created random temporary file inside the digest
   directory, sync and close it, then atomically rename it through the same
   rooted handle. Remove temporary files on every pre-rename failure. Preserve
   last-successful-write and complete-reader semantics.
6. Keep store access serial at the runner boundary and safe for concurrent
   independent store callers. Do not introduce a lock, history, or rollback
   protocol.

### Tests

Add load and save cases for digest-directory and plan-file symlinks that resolve
outside the configured root. Prove that they are rejected and that no outside
file is read, created, chmodded, or replaced. Include unexpected file types and
failed-publication preservation.

Retain and run tests for absent-root behavior, exact paths, permissions, round
trip, malformed digests, atomic replacement, interrupted writes, temporary-file
cleanup, concurrent readers, and concurrent writers. Run the race detector for
`internal/framework/chunkplan`, then perform the repository-wide validation in
the execution rules.

### Exit criteria

No store-owned operation can escape the opened root through a cache entry,
normal layout and permissions remain unchanged, publication remains atomic,
concurrency tests and the race detector pass, and the repository is green.

## Stage 11: Deep-clone source metadata during materialization

**Status:** Pending

### Objective

Ensure source documents, materialized chunks, and separate materializations
never share mutable metadata storage, regardless of the concrete JSON-shaped
map, slice, array, interface, raw-message, or byte-slice types used by an input
adapter.

### Read first

- `internal/core/source/source.go`
- `internal/core/source/chunk_plan.go` and its focused tests
- `internal/core/source/digest.go` and validation behavior
- every metadata clone helper and ownership boundary found with `rg`
- focused runner tests that hand chunks to validators and extractors

### Implement

1. Replace the concrete-type allowlist used by plan materialization with one
   reusable source-owned deep-clone implementation for supported
   JSON-serializable metadata.
2. Recursively clone string-keyed maps, slices, arrays, interfaces,
   `json.RawMessage`, and byte slices without sharing mutable backing storage.
   Preserve scalar values and the logical JSON representation. Typed composite
   containers must not fall through as shared values.
3. Detect cyclic or unsupported values and return a contextual error from
   materialization rather than retaining an alias, panicking, or recursing
   indefinitely.
4. Use the source-owned helper at pipeline ownership boundaries where doing so
   removes a duplicate incomplete implementation. Do not introduce a dependency
   from `internal/core/source` to a framework package and do not expand this
   stage into unrelated metadata-schema redesign.
5. Preserve deterministic materialized content, chunk digests, annotations, IDs,
   references, and metadata fields for all currently supported production
   inputs.

### Tests

Cover typed nested maps, slices of typed maps, arrays, interfaces,
`json.RawMessage`, and byte slices. Mutate each materialized value and prove
that neither the source document nor a separately materialized chunk changes.
Add cyclic and unsupported-value cases that fail deterministically with
contextual errors. Retain the existing exact-materialization and digest tests.

Run focused tests for `internal/core/source`,
`internal/framework/pipeline`, and affected module packages, then perform the
repository-wide validation in the execution rules.

### Exit criteria

Every supported mutable metadata value has independent ownership across source,
chunk, validation, and extraction boundaries; invalid metadata fails safely;
current materialized bytes remain stable; and all checks pass.

## Stage 12: Finalize remediation documentation and roadmap status

**Status:** Pending

### Objective

Make the roadmap and canonical documentation accurately describe the completed
feature after Stages 9 through 11 land.

### Read first

- [Documentation Policy](../policy/documentation.md)
- [ADR-0005 Feature Roadmap](adr0005.md)
- [Architecture Policy](../policy/architecture.md)
- [Diagnostics Internals](../internal/diagnostics.md)
- [Operations](../operations.md)
- the final code and tests from Stages 9 through 11

### Implement

1. Remove the feature roadmap's statement that ADR-0005 is unimplemented and
   describe it as the implemented target-state record. Mark its completion
   outcomes as achieved without adding staged implementation detail.
2. Recheck architecture, diagnostics, operations, pipeline, and module
   documentation against the remediated behavior. Update a canonical document
   only where Stages 9 through 11 changed a current operator or developer
   contract; do not duplicate implementation detail across documents.
3. Mark Stages 9 through 12 complete only after their respective code,
   documentation, focused tests, and repository-wide validation have passed.
   Update this document's introduction to state that all stages are complete and
   that it remains as a concise historical implementation record.
4. Validate every changed link, heading, symbol, diagnostic field, permission,
   and filesystem claim against the implementation.

### Tests and validation

Run:

```sh
go test ./...
go vet ./...
go build ./cmd/notarius
git diff --check
```

Run any repository documentation or link checker if present, and manually
inspect the reading map in [Development](../development.md) and component links
in [Internal Overview](../internal/overview.md).

### Exit criteria

The feature roadmap and canonical references describe implemented behavior,
this plan accurately records completion, documentation ownership remains clear,
all links and names are current, and the repository-wide validation is green.