notarius/docs/roadmap/implementation.md

# ADR-0005 Staged Implementation Plan

This document is the executable implementation plan for the target state in
[ADR-0005 Feature Roadmap](adr0005.md), governed by
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The feature is
not implemented.

The audience is an LLM coding agent. Implement the stages in order. Each stage
is deliberately scoped to finish with a compiling, tested repository and may be
assigned as one implementation prompt.

## Execution Rules

Before every stage:

1. read [Development](../development.md), both documents under `docs/policy/`,
   ADR-0005, this plan, and every stage-specific document named below;
2. inspect `git status` and preserve user changes;
3. inspect the focused package contracts and tests before editing; and
4. confirm that all earlier stages are complete.

During every stage:

- implement only that stage and prerequisites discovered to be inseparable;
- use existing package boundaries and helpers before introducing abstractions;
- keep framework and core packages independent of concrete D&D modules;
- preserve cancellation, deterministic ordering, redaction, path confinement,
  and atomic-write invariants;
- add or update focused tests with every behavior change;
- do not update current-behavior documentation before the final documentation
  stage, except for ADR acceptance in Stage 1; and
- do not begin the next stage after meeting the current exit criteria.

At the end of every stage:

1. run the focused tests listed for that stage;
2. run `go test ./...`;
3. run `go vet ./...`;
4. run `go build ./cmd/notarius`;
5. run `git diff --check`; and
6. mark the stage complete in this document only after all checks pass.

If repository reality conflicts with this plan, stop and update the plan in the
same change before implementing a materially different design. Do not silently
invent a new contract.

## Fixed Decisions

These choices are settled for this implementation:

- `source.SourceDocument.Digest` is the cache lookup identity.
- The chunk-plan root is independent of `workspace.directory` and every other
  state root.
- Its default joins the directory returned by `os.UserCacheDir` with
  `notarius/chunk-plans`.
- `workspace.chunk_cache.directory` overrides that default and names the
  chunk-plan root itself. The recommended value for a system-wide Linux service
  is `/var/cache/notarius/chunk-plans`.
- The plan file is
  `<chunk-plan-root>/<64-character-lowercase-source-sha256-hex>/plan.json`. The
  validated `sha256:` prefix remains part of the logical digest but is removed
  from the path segment.
- The plan envelope schema is `notarius.chunk-plan.v1`.
- There is one mutable plan file per source, atomically replaced. There are no
  immutable plan objects, pointers, history, locks, or rollback.
- Concurrent writers use last-successful-atomic-write semantics; they must
  never expose a partial file.
- Chunk modules retain the `contracts.Chunker` name but replace `Chunk` with a
  `Plan` operation. No legacy interface remains after Stage 2.
- Plans and ranges use `map[string]json.RawMessage` annotations. Annotation
  values are canonical valid JSON and namespaces are non-empty trimmed strings.
- Materialized chunks expose range annotations through
  `source.Chunk.Annotations` and plan annotations through
  `source.Chunk.PlanAnnotations`, never through `Metadata`.
- Framework materialization owns chunk IDs, indexes, references, content,
  media type, units, and generic metadata.
- Only the generic `chunks` capability is hard; `chunks.scenes` is removed.
- Cached-plan hits still construct the configured chunker during preparation
  but never call its operation or the chunk-stage LLM.
- The current pipeline's configured chunk validators run against materialized
  chunks on every path.
- Canonical plan storage is the only chunk-reuse mechanism; chunk checkpoint
  APIs and payloads are removed.
- The mode values are `auto`, `bypass`, and `refresh`. They are valid in config,
  environment, and CLI, including persistent `refresh` configuration.
- The YAML field is `workspace.chunk_cache.mode`.
- The YAML directory field is `workspace.chunk_cache.directory`.
- The environment variable is `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`.
- The root environment variable is `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`.
- The CLI flag is `--chunk_cache <auto|bypass|refresh>`.
- Precedence is CLI, environment, file, then default `auto`.
- Chunk-plan-root precedence is environment, file, then the per-user default;
  there is no CLI root override.
- Every supplied file, environment, and CLI value is parsed strictly even when
  a higher-precedence source would override it; precedence selects among valid
  values and does not mask malformed configuration.
- Existing chunk checkpoint files are ignored and never migrated.
- A structurally invalid stored plan is an `auto` miss. A configured-validator
  rejection of a structurally valid hit does not trigger regeneration.

## Planned Contracts

Use these names unless an existing collision requires the smallest obvious
adjustment.

### Source-zone types

Add to `internal/core/source`:

```go
type ChunkAnnotations map[string]json.RawMessage

type ChunkPlan struct {
    SourceDigest string           `json:"source_digest"`
    Ranges       []ChunkRange     `json:"ranges"`
    Annotations  ChunkAnnotations `json:"annotations,omitempty"`
}

type ChunkRange struct {
    StartUnitID int              `json:"start_unit_id"`
    EndUnitID   int              `json:"end_unit_id"`
    Annotations ChunkAnnotations `json:"annotations,omitempty"`
}
```

Add `Annotations ChunkAnnotations` and
`PlanAnnotations ChunkAnnotations` to `source.Chunk`, with JSON names
`annotations` and `plan_annotations` and `omitempty` on both.

The plan digest covers `SourceDigest`, ordered ranges, and canonical annotation
bytes. It excludes producer provenance, warnings, timestamps, and storage
schema. Materialization produces:

- IDs `chunk-%06d`, numbered from one;
- zero-based indexes in plan order;
- references spanning the first and last units in each range;
- cloned contiguous source units;
- media type `application/json`;
- canonical JSON content shaped as `{"units":[...]}`;
- generic metadata containing `start_unit_id`, `end_unit_id`, and `unit_count`;
- cloned range annotations in `Chunk.Annotations`; and
- cloned plan annotations in `Chunk.PlanAnnotations`.

The plan and range scopes remain distinct even when both use the same namespace.
Copying plan annotations into each runtime chunk is intentional: it makes them
available to downstream chunk-scoped operations without duplicating source
content in durable storage.

Framework validation permits gaps and overlap between ranges, but requires
strictly increasing start positions and rejects duplicate or backward ranges.
Producing modules may impose stricter policy before returning a plan.

### Module contract

Replace the current chunk result and operation with:

```go
type ChunkPlanResult struct {
    Plan     source.ChunkPlan
    Warnings []Warning
}

type Chunker interface {
    Key() string
    ReferenceSlots() []ReferenceSlot
    Plan(context.Context, ChunkRequest) (ChunkPlanResult, error)
}
```

Keep `ChunkRequest`, chunk module keys, stage name `chunk`, configuration
bindings, registries, and preparation ownership unchanged.

### Durable plan record

Define framework-facing record and store contracts beside other pipeline
collaborator contracts:

```go
type ChunkPlanProducer struct {
    InputModule  string
    ChunkModule  string
    LLMProfile   string
    References   []artifacts.ReferenceProvenance
    Metadata     map[string]any
}

type ChunkPlanRecord struct {
    SchemaVersion string
    SourceDigest  string
    PlanDigest    string
    Plan          source.ChunkPlan
    Producer      ChunkPlanProducer
    Warnings      []contracts.Warning
    CreatedAt     time.Time
}

type ChunkPlanStore interface {
    Load(sourceDigest string) (ChunkPlanRecord, ChunkPlanDecision, error)
    Save(ChunkPlanRecord) error
}

type ChunkPlanStoreFactory func(root string) (ChunkPlanStore, error)

type ChunkPlanDecision struct {
    Status ChunkPlanStatus
    Reason string
}

type ChunkPlanStatus string

const (
    ChunkPlanHit     ChunkPlanStatus = "hit"
    ChunkPlanMissing ChunkPlanStatus = "missing"
    ChunkPlanInvalid ChunkPlanStatus = "invalid"
)
```

`ChunkPlanDecision` reports `hit`, `missing`, or `invalid` plus a redacted
reason. A miss or invalid result returns a zero record and nil error. Missing
and structurally invalid records are recoverable in `auto`. Filesystem access
failures other than absence return an error and are framework failures.

The concrete constructor is:

```go
// Package chunkplan
func NewFilesystemStore(root string) (pipeline.ChunkPlanStore, error)
```

It treats `root` as the exact chunk-plan root. CLI composition injects this
constructor through `cli.Options.ChunkPlanStoreFactory`; tests may inject a
recording factory. `cli.Options.UserCacheDir` has signature
`func() (string, error)` and defaults to `os.UserCacheDir`.

The JSON envelope uses snake-case equivalents of the record fields. It stores
references as provenance only, never reference content. Before returning a hit,
the loader validates schema version, source digest, plan digest, plan structure,
annotation JSON, and materializability against the current source in the runner.

### Cache policy and provenance

Add a `ChunkCacheMode` enum with `auto`, `bypass`, and `refresh`, plus strict
parsing and validation.

Add to `RunInput`:

```go
ChunkCacheMode ChunkCacheMode
ChunkPlans     ChunkPlanStore
```

An empty mode used by direct framework callers means `bypass`; the production
CLI always supplies the effective non-empty mode.

Add a run-manifest summary:

```go
type ChunkPlanManifest struct {
    Mode                string                `json:"mode"`
    Action              string                `json:"action,omitempty"`
    SourceDigest        string                `json:"source_digest,omitempty"`
    PlanDigest          string                `json:"plan_digest,omitempty"`
    PlanSchemaVersion   string                `json:"plan_schema_version,omitempty"`
    RequestedModule     string                `json:"requested_module"`
    ProducerInputModule string                `json:"producer_input_module,omitempty"`
    ProducerModule      string                `json:"producer_module,omitempty"`
    ProducerLLMProfile  string                `json:"producer_llm_profile,omitempty"`
    ProducerReferences  []ReferenceProvenance `json:"producer_references,omitempty"`
    ProducerMetadata    map[string]any        `json:"producer_metadata,omitempty"`
    CreatedAt           *time.Time            `json:"created_at,omitempty"`
}
```

`Action` is one of `reused`, `generated`, `refreshed`, or `bypassed`. Retain
the existing top-level `input_module` and `chunker` fields as the requested
pipeline modules. Add this field to `artifacts.RunManifest`:

```go
ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"`
```

The runner initializes it with the effective mode and requested chunk module,
then fills candidate and producer fields when a plan reaches materialization.

## Stage 1: Finalize ADR and add the source-zone plan model

**Status:** Complete

### Objective

Accept the architectural decision and introduce domain-neutral plan,
annotation, validation, digest, clone, and materialization primitives without
changing chunk-module execution yet.

### Read first

- `internal/core/source/source.go`
- `internal/core/source/digest.go`
- `internal/core/source/validation.go`
- `internal/core/source/source_test.go`
- every current `source.Chunk` clone or serialization helper found with `rg`

### Implement

1. Revise ADR-0005 before acceptance:
   - confirm it retains one source-addressed mutable plan and atomic refresh;
   - preserve the independent cache-surface decision;
   - keep filesystem and CLI details out of the ADR; and
   - change status from `Proposed` to `Accepted`.
2. Add `ChunkAnnotations`, `ChunkPlan`, and `ChunkRange` in
   `internal/core/source`.
3. Add canonical annotation validation and cloning. Canonicalize each raw JSON
   value by decoding with `json.Decoder.UseNumber` and re-encoding; reject
   blank namespaces, invalid JSON, trailing JSON values, and non-finite or
   unsupported values.
4. Add plan validation against a `SourceDocument` using the rules in Planned
   Contracts.
5. Add `DigestChunkPlan` and deterministic `MaterializeChunkPlan`.
6. Add `Annotations` and `PlanAnnotations` to `source.Chunk`, `DigestChunk`,
   every clone helper, debug envelope, and serialized chunk-validation
   representation. Ensure all ownership boundaries clone raw bytes and retain
   the two scopes separately.
7. Do not change the chunker interface or runner flow in this stage.

### Tests

Add focused tests for:

- annotation canonicalization, invalid JSON, blank namespaces, and mutation
  safety;
- plan source mismatch, missing ranges, missing units, backward ranges,
  duplicate starts, gaps, and overlap;
- stable plan digests and digest changes for boundary or annotation changes;
- exact materialized IDs, references, content bytes, metadata, annotations,
  gaps, and overlap;
- repeated byte-identical materialization; and
- plan- and range-annotation preservation through existing chunk clones, debug
  conversion, serialized validation, and downstream extract requests.

Run at minimum:

```sh
go test ./internal/core/source
go test ./internal/framework/contracts ./internal/framework/pipeline
```

### Exit criteria

The repository is green; ADR-0005 is accepted; plan primitives are complete and
tested; current modules and the runner still behave as before.

## Stage 2: Replace the chunk operation with plan generation

**Status:** Not started

### Objective

Make all chunk modules produce plans and make the runner materialize those plans
without adding durable reuse yet.

### Read first

- `docs/internal/modules.md`
- `docs/internal/pipeline.md`
- `internal/framework/contracts/contracts.go`
- `internal/framework/pipeline/chunker_registry.go`
- `internal/framework/pipeline/runner.go`
- `internal/framework/pipeline/chunk_validation.go`
- both production chunk-module packages and all chunk fakes in tests

### Implement

1. Replace `ChunkResult` with `ChunkPlanResult` and `Chunker.Chunk` with
   `Chunker.Plan`; keep the request, registry, module keys, and stage name.
2. Update all fakes, registry tests, preparation tests, integration helpers, and
   compile-time interface assertions in the same stage. Do not retain adapters
   or a legacy chunk-returning interface.
3. Convert `generic/chunk/units` to emit ranges only. Preserve `max_units` and
   `overlap_units` semantics.
4. Convert `dnd/chunk/scenes` to emit namespace `dnd/scenes` at both scopes:
   - each range value is an object with exact keys `short_title`,
     `primary_mode`, `main_participants`, `summary`, `boundary_note`, and
     `boundary_confidence`, preserving the current normalized scene values;
     and
   - the plan value is an object with exact key `boundary_caveats`, containing
     the normalized caveat array.
5. Preserve accepted scene boundary caveats as `contracts.Warning` values as
   well as in the plan annotation so warning behavior remains visible.
6. Remove `chunks.scenes` from the D&D scene module's provided capabilities.
   Keep only `chunks`.
7. Change the runner’s non-cached chunk path to:
   - call `Plan` under the existing retry and debug attempt boundary;
   - validate the plan structurally;
   - materialize chunks;
   - run the configured chunk-validator chain on materialized chunks;
   - preserve retry and rejection semantics; and
   - pass materialized chunks to lanes exactly as before.
8. Update chunk debug payloads to distinguish the generated plan from
   materialized chunks. Plan annotations may appear only in opt-in debug data.
9. Leave current chunk checkpoint loading and recording temporarily in place,
   storing materialized chunks, so existing resume tests remain green until
   Stage 4 removes that path.

### Tests

Update focused module and framework tests to cover:

- exact generic ranges and overlap;
- exact D&D scene annotations and warning conversion;
- annotations absent from generic plans;
- framework materialization rather than module materialization;
- chunk validators receiving materialized annotated chunks;
- retries around generation plus validation;
- stable chunk IDs independent of producing module; and
- no `chunks.scenes` resolution dependency.

Run at minimum:

```sh
go test ./internal/modules/generic/chunk/units
go test ./internal/modules/dnd/chunk/scenes
go test ./internal/framework/contracts ./internal/framework/pipeline
go test ./internal/modules/integration ./internal/cli
```

### Exit criteria

Every chunk module uses the single plan contract, the runner always
materializes, production behavior remains functionally equivalent apart from
framework-owned stable chunk IDs and namespaced annotations, and the repository
is green.

## Stage 3: Add dedicated cache configuration and the plan store

**Status:** Not started

### Objective

Implement the durable source-addressed store and its independent cache-root
configuration without changing the runner to consume the store.

### Read first

- `docs/config.md`
- `docs/operations.md`
- `internal/core/config/*`
- `internal/core/workspace/*`
- `internal/framework/checkpoint/` as the local pattern for a workspace-backed
  implementation
- `internal/cli/run.go` configuration loading, without wiring the store yet

### Implement

1. Add `ChunkCacheMode` and strict parsing in the pipeline package.
2. Add `WorkspaceChunkCacheConfig` with `Mode` and `Directory` to config and
   file config. The YAML fields are `workspace.chunk_cache.mode` and
   `workspace.chunk_cache.directory`.
3. Set default mode to `auto`; support
   `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`; preserve precedence file then
   environment. CLI precedence arrives in Stage 5.
4. Support `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR` with environment-over-file
   precedence. Trim the value; an unset or empty effective value selects the
   per-user default. Do not add a CLI directory override.
5. Add `workspace.DefaultChunkPlanRoot`, accepting an injectable
   user-cache-directory resolver and returning
   `filepath.Join(userCacheDir, "notarius", "chunk-plans")`. Production later
   supplies `os.UserCacheDir`; tests supply a stub. Reject resolver errors and
   empty returned directories. Use this exact signature:

   ```go
   func DefaultChunkPlanRoot(userCacheDir func() (string, error)) (string, error)
   ```
6. Treat a non-empty `workspace.chunk_cache.directory` as the plan root itself,
   cleaned with the same path rules as other workspace roots. It does not gain
   an implicit `chunk-plans` suffix. `workspace.directory` must not affect this
   root, and the chunk-cache directory must not affect checkpoint, diagnostics,
   debug, or output roots. Do not change `workspace.FromConfig` or
   `workspace.Settings` for this feature.
7. Add the plan record, producer, decision, and store interfaces described
   above in the pipeline package.
8. Add `internal/framework/chunkplan` with:
   - a filesystem implementation;
   - a constructor that accepts the already resolved chunk-plan root;
   - path `<chunk-plan-root>/<64-character-source-sha256-hex>/plan.json`, after
     strictly validating and stripping the logical `sha256:` prefix;
   - schema `notarius.chunk-plan.v1`;
   - directories mode `0700` and file mode `0600`;
   - atomic same-directory temp-file rename;
   - strict envelope decoding and unknown-field rejection;
   - complete digest and annotation validation; and
   - no raw reference content.
9. `Load` distinguishes missing, invalid, and operational error. `Save` always
   atomically replaces and implements last-successful-write semantics.
10. Keep store fakes local to their tests; do not add a production in-memory or
    no-op store because `bypass` and a nil collaborator already express
    disabled persistence.
11. Do not wire the store into `Runner` or CLI execution in this stage.

### Tests

Cover:

- config defaults, YAML decoding, environment override, invalid values,
  cloning, redaction, and validation;
- directory environment-over-file precedence, empty-value fallback, cloning,
  redaction, cleaning, and validation;
- exact injected per-user default and resolver failure or empty-result errors;
- independence in both directions between `workspace.directory` and
  `workspace.chunk_cache.directory`;
- exact acceptance of `/var/cache/notarius/chunk-plans` as a configured root
  without writing to that real path;
- safe full-digest paths and rejection of malformed digests;
- `0700`/`0600` permissions where supported;
- envelope round trip, strict decoding, source and plan digest mismatch,
  invalid annotations, schema mismatch, missing files, operational read errors,
  atomic replacement, and concurrent writers; and
- absence of raw reference content in persisted JSON.

Run at minimum:

```sh
go test ./internal/core/config ./internal/core/workspace
go test ./internal/framework/chunkplan ./internal/framework/pipeline
go test ./internal/cli
```

### Exit criteria

Configuration and storage are complete and independently tested, but normal
runs still use Stage 2’s generate-and-materialize behavior.

## Stage 4: Integrate cache policy and remove chunk checkpoints

**Status:** Not started

### Objective

Make the runner’s chunk path use the canonical plan store under all three modes
and establish it as the only chunk-reuse mechanism.

### Read first

- `internal/framework/pipeline/runner.go`
- `internal/framework/pipeline/checkpoint.go`
- `internal/framework/pipeline/runner_concurrent.go`
- `internal/framework/checkpoint/loader.go`
- `internal/framework/checkpoint/recorder.go`
- focused runner retry, rejection, debug, and checkpoint tests

### Implement

1. Add `ChunkCacheMode` and `ChunkPlans` to `RunInput`. Empty mode means
   `bypass` for direct framework callers.
2. Add synchronized store wrapping only if runner access can be concurrent;
   otherwise document and test the intentionally serial chunk-plan access.
3. Implement mode behavior:
   - `auto` hit: load, structurally validate against the current source,
     materialize, then run configured chunk validators once;
   - `auto` miss or invalid: generate through existing retries, validate,
     materialize, validate chunks, then save;
   - `bypass`: generate and validate normally without calling `Load` or `Save`;
   - `refresh`: generate and validate normally, then replace through `Save`.
4. A valid-hit validator rejection is recorded without retry, store mutation,
   or implicit regeneration. A store I/O error is a framework error.
5. A failed or rejected generated plan is never saved. An invalid old file is
   overwritten only after a replacement passes module execution, structural
   validation, materialization, and configured validators.
6. Build producer provenance from the prepared input module key, prepared chunk
   module key, resolved chunk-stage LLM profile, chunk-target auxiliary
   reference provenance, and chunk-module manifest metadata. Clone and redact
   according to existing boundaries.
7. Replay stored producer warnings on a hit, then append current validation
   warnings. Do not duplicate warnings from discarded generation attempts.
8. Remove all chunk methods and types from `CheckpointRecorder`,
   `CheckpointLoader`, synchronized wrappers, workspace recorder/loader, and
   checkpoint manifests and envelopes.
9. Remove runner chunk-checkpoint decisions and events. Retain source, extract,
   merge, and normalize checkpoints. Keep downstream fingerprints based on the
   digest of materialized chunks.
10. Ignore existing on-disk `chunk/` checkpoint files; do not read, delete, or
    migrate them.
11. Update debug behavior:
    - hit records lookup and materialization but no chunk module attempt or LLM
      call;
    - generated, bypassed, and refreshed paths retain attempt debug; and
    - default debug envelopes remain redacted.

### Tests

Add table-driven runner tests for every mode and outcome:

- hit, missing, invalid, and I/O error;
- generation success, error, rejection, retry, and cancellation;
- validator warning, rejection, and error on hits and generated plans;
- bypass performs zero store calls;
- refresh performs no load and exactly one successful save;
- invalid record remains untouched when replacement fails;
- stored warnings replay once;
- no module or LLM call on hit;
- downstream chunk digest changes on refresh;
- legacy chunk checkpoint files are ignored; and
- source and lane checkpoint behavior remains intact.

Run at minimum:

```sh
go test ./internal/framework/pipeline
go test ./internal/framework/checkpoint
go test ./internal/modules/integration
```

### Exit criteria

The runner implements complete cache semantics, chunk checkpoints no longer
exist as a code path, all direct framework tests are green, and production CLI
runs are not yet wired to persistent plan storage.

## Stage 5: Wire persistent policy into the CLI

**Status:** Not started

### Objective

Make normal CLI runs resolve the effective mode and dedicated chunk-plan root,
construct the durable plan store when appropriate, and pass it into the runner.

### Read first

- `docs/cli.md`
- `docs/config.md`
- `cmd/notarius/main.go` and `internal/cli/run.go`
- CLI flag, environment, invocation-metadata, and exit-code tests
- the workspace and configuration code changed in Stage 3

### Implement

1. Add the exact flag `--chunk_cache <auto|bypass|refresh>`. Represent an
   omitted CLI value separately from an explicit value so precedence is not
   inferred from the default printed by the flag package.
2. Resolve the effective mode in this order: explicit CLI flag, environment,
   configuration file, default `auto`. An invalid CLI flag is a usage error
   with exit code 2; invalid file or environment configuration follows the
   existing configuration-load failure path and exit code 1.
3. Include an explicitly requested CLI value in invocation metadata and the
   resolved mode in effective-configuration metadata without adding field-level
   origin tracking or plan and annotation payloads. Name the optional
   invocation JSON field `chunk_cache_override`; the resolved mode already
   appears at `workspace.chunk_cache.mode` in the redacted effective config.
4. Add `UserCacheDir func() (string, error)` and
   `ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory` to `cli.Options`,
   defaulting them to `os.UserCacheDir` and
   `chunkplan.NewFilesystemStore`. For `auto` and `refresh`, choose the non-empty
   effective `workspace.chunk_cache.directory`; otherwise resolve the per-user
   default with `workspace.DefaultChunkPlanRoot`. Invoke the store factory with
   that exact root and pass the result and effective mode through `RunInput`.
   Never append `chunk-plans` to an explicit directory. The injectable factory
   lets CLI tests assert root selection without writing to production paths.
5. For `bypass`, do not resolve, inspect, or create the chunk-plan root and pass
   no store. Existing general-workspace setup remains unchanged and independent:
   `workspace.directory`, resume, diagnostics, and debug continue to use their
   current paths and semantics.
6. Keep resume policy independent of plan reuse. Enabling, disabling, or
   selecting a resume run must not change the chunk-cache mode or path.
7. Configuration validation and pipeline-listing commands validate the mode but
   do not create workspace directories or plan files.
8. Update CLI usage text only as needed to expose the new flag. Defer all
   narrative documentation to Stage 8.

### Tests

Add CLI and integration tests for:

- a default `auto` run writing a plan and a second independent invocation
  reusing it without a chunk operation or chunk LLM call;
- file, environment, and CLI precedence, including persistent `refresh` in
  file and environment configuration;
- `bypass` performing no plan-store access and resolving or creating no default
  chunk-plan root;
- invalid flag, environment, and file values with their established error
  classifications;
- an explicit chunk-plan root from file and environment, including exact use of
  `/var/cache/notarius/chunk-plans` without an appended suffix, using a
  recording store factory and never writing to that real path;
- an explicit root succeeding without calling a failing per-user cache resolver;
- an injected per-user default resolving exactly to
  `<user-cache-dir>/notarius/chunk-plans`;
- `workspace.directory` changes having no effect on chunk-plan placement and a
  chunk-cache directory having no effect on other workspace state;
- per-user cache-root discovery failure under `auto`, `refresh`, and `bypass`,
  including the allowed no-state `bypass` case, plus unwritable-store failures;
- resume plus each cache mode, showing independent behavior;
- validation and listing commands having no filesystem side effects; and
- reuse across changed pipeline, chunk module/options, references, validators,
  and LLM settings when the source digest is unchanged.

Run at minimum:

```sh
go test ./internal/cli
go test ./internal/framework/pipeline ./internal/framework/chunkplan
go test ./internal/modules/integration
```

### Exit criteria

The production CLI implements the complete mode and precedence contract,
independent invocations reuse source-addressed plans by default, bypass and
refresh have exact state semantics, and the repository is green.

## Stage 6: Expose producer provenance and safe diagnostics

**Status:** Not started

### Objective

Make every output and diagnostic record distinguish the requested chunk module
from the producer of the effective plan without exposing sensitive payloads by
default.

### Read first

- `docs/integrations/json-output.md`
- `docs/internal/diagnostics.md`
- manifest assembly in `internal/framework/pipeline/runner.go`
- `internal/core/artifacts/artifacts.go`
- current diagnostics and debug-envelope implementations and tests

### Implement

1. Add `ChunkPlanManifest` to the run manifest and all required clone,
   conversion, and serialization paths.
2. Populate `mode` and exactly one action after a candidate plan has passed
   structural validation and materialization:
   - `reused` for an `auto` hit;
   - `generated` for an `auto` miss or invalid record;
   - `refreshed` for generation under `refresh`; and
   - `bypassed` for generation under `bypass`.
   Retain that action if the current chunk-validator chain rejects the
   candidate; publication still follows Stage 4's accept-only rule. If no
   candidate reaches materialization, leave action and candidate plan fields
   empty in the partial failure manifest and report the failure in diagnostics.
3. Preserve the existing top-level `chunker` and its module metadata as the
   requested module. Never overwrite those fields with cached producer data.
4. Populate producer module, LLM profile, references, metadata, creation time,
   source digest, plan digest, and schema from the effective stored or newly
   generated record. A reused record reports its original producer, even when
   it differs from the requested module.
5. Define producer LLM profile as the resolved chunk-stage profile used during
   generation. Leave it empty for producers that did not use an LLM; do not
   substitute the current run's requested profile on reuse.
6. Keep producer references as existing redacted provenance records. Deep-clone
   metadata, references, warnings, annotations, and raw JSON at every ownership
   boundary.
7. Add a redacted `chunk-plan.json` diagnostic artifact, backed by a summary on
   `RunOutput` so the CLI can write it on success, rejection, or framework
   failure. It contains mode, source and candidate plan digests, requested
   module, lookup status and redacted reason, action, validation status, and
   publication status. Use these closed values:
   - lookup: `hit`, `missing`, `invalid`, or `skipped`;
   - validation: `not_run`, `approved`, `approved_with_warnings`, `rejected`,
     or `error`; and
   - publication: `not_requested`, `published`, or `not_published`.
   It contains no source units, materialized chunk content, annotation bytes,
   reference content, prompts, model responses, or raw invalid-file content.
8. Keep complete plan, annotations, attempts, and materialized chunks confined
   to existing opt-in debug artifacts and apply their normal restrictive file
   handling.
9. Preserve JSON-output compatibility by adding fields rather than renaming
   unrelated existing fields. Omit producer-only optional values when absent in
   the same style as the surrounding manifest schema.

### Tests

Cover:

- all four actions and their exact manifest fields;
- requested and producer modules differing on a hit;
- producer metadata remaining stable when current configuration changes;
- LLM-backed and non-LLM producers;
- deep-clone and mutation safety for raw annotations and metadata;
- default diagnostic redaction, including invalid stored files;
- complete data appearing only under explicit debug configuration; and
- JSON serialization and compatibility fixtures.

Run at minimum:

```sh
go test ./internal/core/artifacts
go test ./internal/framework/pipeline
go test ./internal/cli
```

### Exit criteria

Manifests and diagnostics explain which plan was requested, selected, and
produced; default output remains redacted; debug behavior is explicit; and all
compatibility tests pass.

## Stage 7: Add cross-run, corruption, and concurrency hardening

**Status:** Not started

### Objective

Exercise the completed design across package boundaries and adversarial state,
then close any correctness gaps without changing the settled contract.

### Read first

- all tests added in Stages 1–6
- end-to-end CLI and module integration test harnesses
- atomic-write, cancellation, retry, and race-sensitive framework code
- repository import-boundary tests

### Implement

1. Add black-box tests using separate CLI application instances and a shared
   temporary chunk-plan root. Prove reuse across different pipelines,
   configured chunk modules, options, references, lanes, validators, and LLM
   profiles.
2. Prove that a changed canonical source digest selects a different path, while
   byte-identical canonical sources reuse the same plan.
3. Exercise generic plans through D&D pipelines and D&D-annotated plans through
   generic pipelines. Optional annotations must survive, while no downstream
   component may require `dnd/scenes` merely because the requested chunker is
   the D&D scene module.
4. Add corruption cases for truncated JSON, unknown fields, schema mismatch,
   source mismatch, plan-digest mismatch, bad boundaries, invalid annotations,
   and materialization failure. Verify `auto` regeneration and non-destructive
   failure behavior exactly.
5. Add concurrent reader/writer and writer/writer tests. Readers observe either
   the old complete envelope or the new complete envelope; after successful
   writers, the final file is one complete valid record. Do not assert which
   writer wins.
6. Exercise interruption before save, during temp-file creation, and around
   rename where the implementation permits deterministic fault injection.
   Prior valid state must remain readable unless a complete replacement was
   published.
7. Verify cancellation, retry limits, warning order, error classification, and
   debug ordering on misses and hits. No cache hit may consume chunk-stage retry
   budget or invoke the module.
8. Verify legacy chunk checkpoint files cannot influence reuse and that lane
   resume still keys from effective materialized chunk digests after refresh.
9. Extend import-boundary tests so `internal/core/source` owns no framework or
   module dependencies and the generic store owns no concrete module imports.
10. Ensure examples and all tests isolate workspace state in temporary
    directories and isolate chunk-plan state with a temporary configured root
    or explicitly select `bypass`; tests must never write to the real per-user
    default or `/var/cache/notarius/chunk-plans`.
11. Run the race detector over the packages that access the plan store and
    runner concurrently, and fix any race within feature scope.

### Tests

In addition to the new focused tests, run:

```sh
go test ./...
go test -race ./internal/framework/chunkplan ./internal/framework/pipeline
go vet ./...
go build ./cmd/notarius
```

### Exit criteria

The feature is demonstrated across independent runs, corruption and concurrent
publication cannot expose partial state, cross-domain annotations remain
optional, cancellation and retry behavior is deterministic, and all repository
and race checks pass.

## Stage 8: Publish current-behavior documentation

**Status:** Not started

### Objective

After implementation is complete, make the supported behavior discoverable
without duplicating canonical ownership across documents.

### Read first

- `docs/policy/architecture.md`
- `docs/policy/documentation.md`
- every current-behavior document linked below
- the final code, flags, schemas, examples, and tests from Stages 1–7

### Implement

1. Update `docs/policy/architecture.md` with the
   plan-generation/materialization
   boundary, source-addressed reuse, and the independent cache-state surface at
   the level appropriate for the canonical system overview.
2. Update `docs/internal/pipeline.md` with mode behavior, hit/miss control flow,
   validator placement, retry behavior, and removal of chunk checkpoints.
3. Update `docs/internal/modules.md` with the `Chunker.Plan` contract, generic
   boundaries, optional annotation namespaces, and framework materialization.
4. Update `docs/internal/overview.md` with the new plan-store package and
   current state ownership, and remove chunk checkpoint ownership from its
   checkpoint description. Keep detailed checkpoint and plan execution flow in
   `docs/internal/pipeline.md` rather than creating a parallel internal state
   reference.
5. Update `docs/config.md` with `workspace.chunk_cache.mode`,
   `workspace.chunk_cache.directory`, both environment variables, strict
   values, precedence, and the per-user cache-root default. State explicitly
   that `workspace.directory` does not affect plan placement and that the
   configured directory is the root itself. Keep field-level details owned
   here. Document that Unix `os.UserCacheDir` rejects a relative
   `XDG_CACHE_HOME` and that this is a configuration error rather than a fallback.
6. Update `docs/cli.md` with `--chunk_cache` semantics and examples. Link to
   configuration rather than restating its entire precedence model.
7. Update `docs/operations.md` with plan location, permissions, invalid-state
   recovery, refresh and bypass operations, concurrency semantics, sensitive
   debug data, deletion as a recoverable but potentially expensive
   regeneration event, and the no-history/no-rollback limitation. Document both:
   - the ordinary Linux per-user default,
     `$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is a valid
     absolute path, or `$HOME/.cache/notarius/chunk-plans` when it is unset; and
   - `/var/cache/notarius/chunk-plans` as the recommended configured root for a
     system-wide Linux deployment running under a dedicated service account.
   Explain that the system path is not the unprivileged default, show the
   following minimal configuration, require the operator or package installer
   to provision it with restrictive service-account ownership and permissions,
   and warn against sharing the cache across mutually untrusted users:

   ```yaml
   workspace:
     chunk_cache:
       directory: /var/cache/notarius/chunk-plans
   ```
8. Update `docs/integrations/json-output.md` and
   `docs/internal/diagnostics.md` with requested versus producer provenance,
   actions, warnings, and redaction boundaries.
9. Keep package-level plan/range/annotation and materialization invariants in
   `docs/internal/pipeline.md`; no separate source document is needed unless
   the implemented subsystem becomes too large for that canonical owner.
10. Update examples only when they materially clarify operation; examples must
    use temporary or explicitly configured workspace and chunk-cache paths.
11. Remove future-tense disclaimers from the feature roadmap, mark its
    completion outcomes as achieved, and mark all completed stages in this
    document. Keep the roadmap as historical target-state context rather than
    copying implementation details into it.
12. Validate every changed link, heading, code symbol, flag, environment name,
    path, and JSON field against the implementation. Search for stale
    `Chunker.Chunk`, `ChunkResult`, `chunks.scenes`, and chunk-checkpoint claims.

### Tests and validation

Run:

```sh
go test ./...
go vet ./...
go build ./cmd/notarius
git diff --check
```

Also run repository link or documentation checks if present, and manually
inspect the reading map in `docs/development.md` and component links in
`docs/internal/overview.md` for canonical ownership and navigation consistency.

### Exit criteria

All canonical documents describe implemented behavior, operational guidance is
complete and non-duplicative, no stale chunk API or checkpoint documentation
remains, roadmap status is accurate, and the repository is green.