From b3328b93e52776831d8c6f4d7c6f05ecc47f0ef0 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 17 Jul 2026 20:55:39 -0500 Subject: [PATCH] Update implementation plan to incorporate follow-up fixes --- docs/roadmap/implementation.md | 896 ++++++++++----------------------- 1 file changed, 256 insertions(+), 640 deletions(-) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index f4b817c..49baff8 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,12 +1,14 @@ # ADR-0005 Implementation Record -This document records the completed implementation of the target state in +This document records the implementation of the target state in [ADR-0005 Feature Roadmap](adr0005.md), governed by [ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). -All stages below are complete. The plan remains as historical target-state -context; current behavior is documented in the canonical references linked from -[Development](../development.md). +Stages 1 through 8 summarize the completed initial implementation. Stages 9 +through 12 track pending remediation identified by the post-implementation +review. Current behavior is documented in the canonical references linked from +[Development](../development.md); those references must not describe the Stages +9 through 12 target state until the corresponding stage is complete. ## Execution Rules @@ -281,666 +283,280 @@ ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"` The runner initializes it with the effective mode and requested chunk module, then fills candidate and producer fields when a plan reaches materialization. -## Stage 1: Finalize ADR and add the source-zone plan model +## Completed Stage Summaries + +### Stage 1: Finalize ADR and add the source-zone plan model **Status:** Complete +Accepted ADR-0005 and added the domain-neutral chunk-plan, range, annotation, +validation, digest, cloning, and deterministic materialization primitives in +`internal/core/source`. + +### Stage 2: Replace the chunk operation with plan generation + +**Status:** Complete + +Replaced chunk-returning module operations with `Chunker.Plan`, converted the +generic and D&D scene chunkers to produce ranges and optional annotations, and +made the runner materialize and validate generated plans. + +### Stage 3: Add dedicated cache configuration and the plan store + +**Status:** Complete + +Added strict cache modes, independent per-user or explicitly configured +chunk-plan roots, the source-addressed filesystem store, restrictive +permissions, envelope validation, and atomic replacement. + +### Stage 4: Integrate cache policy and remove chunk checkpoints + +**Status:** Complete + +Implemented runner semantics for `auto`, `bypass`, and `refresh`; retained +current chunk-validator behavior; made accepted canonical plans the sole chunk +reuse mechanism; and removed chunk checkpoint loading and recording. + +### Stage 5: Wire persistent policy into the CLI + +**Status:** Complete + +Added CLI, environment, and file configuration precedence; per-user root +resolution; persistent store construction; bypass isolation; and cross-run +reuse through normal CLI execution. + +### Stage 6: Expose producer provenance and safe diagnostics + +**Status:** Complete + +Added requested-versus-producer chunk-plan provenance to manifests and a +payload-free chunk-plan decision summary to normal diagnostics, with complete +plan material confined to opt-in debug artifacts. + +### Stage 7: Add cross-run, corruption, and concurrency hardening + +**Status:** Complete + +Added integration, corruption, interruption, concurrency, compatibility, +cross-domain annotation, import-boundary, and race coverage for the implemented +cache design. + +### Stage 8: Publish current-behavior documentation + +**Status:** Complete + +Updated architecture, pipeline, module, configuration, CLI, operations, +diagnostics, integration, and overview documentation for the implemented +feature and its per-user and system-wide deployment guidance. + +## Stage 9: Redact invalid cache-record diagnostics + +**Status:** Pending + ### Objective -Accept the architectural decision and introduce domain-neutral plan, -annotation, validation, digest, clone, and materialization primitives without -changing chunk-module execution yet. +Ensure malformed or incompatible cache contents cannot enter normal errors, +warnings, manifests, or diagnostics while retaining useful closed lookup +decisions. + +### Read first + +- [Architecture Policy](../policy/architecture.md), especially secret handling +- [Diagnostics Internals](../internal/diagnostics.md) +- `internal/framework/chunkplan/store.go` and its focused tests +- `internal/framework/pipeline/runner_chunk_plan.go` and its focused tests +- `internal/core/artifacts/artifacts.go` +- the CLI path that writes `chunk-plan.json` + +### Implement + +1. Stop interpolating decoder errors, unknown field names, schema values, + annotation namespaces, timestamps, paths, or any other stored content into + recoverable `ChunkPlanDecision.Reason` values. +2. At the normal-diagnostic boundary, ignore the reason supplied by a + `ChunkPlanStore` and derive `ChunkPlanSummary.LookupReason` solely from + status using exactly: + - `hit`: `stored chunk plan is valid`; + - `missing`: `chunk plan not found`; + - `invalid`: `stored chunk plan is invalid`; and + - `skipped`: `chunk plan lookup skipped`. +3. Keep operational read failures as framework errors rather than recoverable + invalid-record decisions. Their normal error text must identify the failed + operation without including cache-file content. +4. Do not add raw invalid-record details to another default surface. Direct + operator inspection or a future explicitly sensitive debug facility may + expose them, but this stage does not need to add such a facility. + +### Tests + +Add focused tests proving that sentinel secrets placed in unknown field names, +schema values, annotation namespaces, timestamps, and malformed trailing JSON +appear in neither the store decision nor `chunk-plan.json`. Add a runner test +with a custom store whose reason contains a sentinel and prove that the summary +uses the fixed status-derived reason. Preserve useful missing, hit, invalid, and +operational-error classification. + +Run focused tests for `internal/framework/chunkplan`, +`internal/framework/pipeline`, and `internal/cli`, then perform the +repository-wide validation in the execution rules. + +### Exit criteria + +Every normal lookup reason belongs to the fixed safe vocabulary, hostile +invalid-file content cannot reach default diagnostics, operational failures +remain correctly classified, and all focused and repository-wide checks pass. + +## Stage 10: Enforce rooted chunk-plan filesystem access + +**Status:** Pending + +### Objective + +Make the cache's path-confinement guarantee resistant to symlinked cache entries +while preserving atomicity, permissions, and concurrent access semantics. + +### Read first + +- [Architecture Policy](../policy/architecture.md), especially state and path + invariants +- [Operations](../operations.md) +- `internal/framework/chunkplan/store.go` and all store tests +- Go's `os.Root` API for the repository's declared Go version + +### Implement + +1. Treat the configured cache root as the filesystem trust boundary. A symlink + in the operator-supplied root path itself may be followed when the root is + opened, but every store-owned operation after that point must remain beneath + the opened root. +2. Perform digest-directory, plan-file, temporary-file, and rename operations + through `os.Root` or an equivalently race-resistant directory-relative + mechanism. Do not rely on `filepath.Join` plus a preflight `Lstat` as the + confinement boundary. +3. Reject symlinks and unexpected file types at the store-owned digest-directory + and `plan.json` positions. A missing cache root on `Load` remains a normal + miss and is not created; `Save` creates the root and digest directory as + needed. +4. Preserve the exact + `//plan.json` layout, `0700` directories, + `0600` plan files, strict digest validation, and no implicit path suffix. +5. Publish using an exclusively created random temporary file inside the digest + directory, sync and close it, then atomically rename it through the same + rooted handle. Remove temporary files on every pre-rename failure. Preserve + last-successful-write and complete-reader semantics. +6. Keep store access serial at the runner boundary and safe for concurrent + independent store callers. Do not introduce a lock, history, or rollback + protocol. + +### Tests + +Add load and save cases for digest-directory and plan-file symlinks that resolve +outside the configured root. Prove that they are rejected and that no outside +file is read, created, chmodded, or replaced. Include unexpected file types and +failed-publication preservation. + +Retain and run tests for absent-root behavior, exact paths, permissions, round +trip, malformed digests, atomic replacement, interrupted writes, temporary-file +cleanup, concurrent readers, and concurrent writers. Run the race detector for +`internal/framework/chunkplan`, then perform the repository-wide validation in +the execution rules. + +### Exit criteria + +No store-owned operation can escape the opened root through a cache entry, +normal layout and permissions remain unchanged, publication remains atomic, +concurrency tests and the race detector pass, and the repository is green. + +## Stage 11: Deep-clone source metadata during materialization + +**Status:** Pending + +### Objective + +Ensure source documents, materialized chunks, and separate materializations +never share mutable metadata storage, regardless of the concrete JSON-shaped +map, slice, array, interface, raw-message, or byte-slice types used by an input +adapter. ### Read first - `internal/core/source/source.go` -- `internal/core/source/digest.go` -- `internal/core/source/validation.go` -- `internal/core/source/source_test.go` -- every current `source.Chunk` clone or serialization helper found with `rg` +- `internal/core/source/chunk_plan.go` and its focused tests +- `internal/core/source/digest.go` and validation behavior +- every metadata clone helper and ownership boundary found with `rg` +- focused runner tests that hand chunks to validators and extractors ### Implement -1. Revise ADR-0005 before acceptance: - - confirm it retains one source-addressed mutable plan and atomic refresh; - - preserve the independent cache-surface decision; - - keep filesystem and CLI details out of the ADR; and - - change status from `Proposed` to `Accepted`. -2. Add `ChunkAnnotations`, `ChunkPlan`, and `ChunkRange` in - `internal/core/source`. -3. Add canonical annotation validation and cloning. Canonicalize each raw JSON - value by decoding with `json.Decoder.UseNumber` and re-encoding; reject - blank namespaces, invalid JSON, trailing JSON values, and non-finite or - unsupported values. -4. Add plan validation against a `SourceDocument` using the rules in Planned - Contracts. -5. Add `DigestChunkPlan` and deterministic `MaterializeChunkPlan`. -6. Add `Annotations` and `PlanAnnotations` to `source.Chunk`, `DigestChunk`, - every clone helper, debug envelope, and serialized chunk-validation - representation. Ensure all ownership boundaries clone raw bytes and retain - the two scopes separately. -7. Do not change the chunker interface or runner flow in this stage. +1. Replace the concrete-type allowlist used by plan materialization with one + reusable source-owned deep-clone implementation for supported + JSON-serializable metadata. +2. Recursively clone string-keyed maps, slices, arrays, interfaces, + `json.RawMessage`, and byte slices without sharing mutable backing storage. + Preserve scalar values and the logical JSON representation. Typed composite + containers must not fall through as shared values. +3. Detect cyclic or unsupported values and return a contextual error from + materialization rather than retaining an alias, panicking, or recursing + indefinitely. +4. Use the source-owned helper at pipeline ownership boundaries where doing so + removes a duplicate incomplete implementation. Do not introduce a dependency + from `internal/core/source` to a framework package and do not expand this + stage into unrelated metadata-schema redesign. +5. Preserve deterministic materialized content, chunk digests, annotations, IDs, + references, and metadata fields for all currently supported production + inputs. ### Tests -Add focused tests for: +Cover typed nested maps, slices of typed maps, arrays, interfaces, +`json.RawMessage`, and byte slices. Mutate each materialized value and prove +that neither the source document nor a separately materialized chunk changes. +Add cyclic and unsupported-value cases that fail deterministically with +contextual errors. Retain the existing exact-materialization and digest tests. -- annotation canonicalization, invalid JSON, blank namespaces, and mutation - safety; -- plan source mismatch, missing ranges, missing units, backward ranges, - duplicate starts, gaps, and overlap; -- stable plan digests and digest changes for boundary or annotation changes; -- exact materialized IDs, references, content bytes, metadata, annotations, - gaps, and overlap; -- repeated byte-identical materialization; and -- plan- and range-annotation preservation through existing chunk clones, debug - conversion, serialized validation, and downstream extract requests. - -Run at minimum: - -```sh -go test ./internal/core/source -go test ./internal/framework/contracts ./internal/framework/pipeline -``` +Run focused tests for `internal/core/source`, +`internal/framework/pipeline`, and affected module packages, then perform the +repository-wide validation in the execution rules. ### Exit criteria -The repository is green; ADR-0005 is accepted; plan primitives are complete and -tested; current modules and the runner still behave as before. +Every supported mutable metadata value has independent ownership across source, +chunk, validation, and extraction boundaries; invalid metadata fails safely; +current materialized bytes remain stable; and all checks pass. -## Stage 2: Replace the chunk operation with plan generation +## Stage 12: Finalize remediation documentation and roadmap status -**Status:** Complete +**Status:** Pending ### Objective -Make all chunk modules produce plans and make the runner materialize those plans -without adding durable reuse yet. +Make the roadmap and canonical documentation accurately describe the completed +feature after Stages 9 through 11 land. ### Read first -- `docs/internal/modules.md` -- `docs/internal/pipeline.md` -- `internal/framework/contracts/contracts.go` -- `internal/framework/pipeline/chunker_registry.go` -- `internal/framework/pipeline/runner.go` -- `internal/framework/pipeline/chunk_validation.go` -- both production chunk-module packages and all chunk fakes in tests +- [Documentation Policy](../policy/documentation.md) +- [ADR-0005 Feature Roadmap](adr0005.md) +- [Architecture Policy](../policy/architecture.md) +- [Diagnostics Internals](../internal/diagnostics.md) +- [Operations](../operations.md) +- the final code and tests from Stages 9 through 11 ### Implement -1. Replace `ChunkResult` with `ChunkPlanResult` and `Chunker.Chunk` with - `Chunker.Plan`; keep the request, registry, module keys, and stage name. -2. Update all fakes, registry tests, preparation tests, integration helpers, and - compile-time interface assertions in the same stage. Do not retain adapters - or a legacy chunk-returning interface. -3. Convert `generic/chunk/units` to emit ranges only. Preserve `max_units` and - `overlap_units` semantics. -4. Convert `dnd/chunk/scenes` to emit namespace `dnd/scenes` at both scopes: - - each range value is an object with exact keys `short_title`, - `primary_mode`, `main_participants`, `summary`, `boundary_note`, and - `boundary_confidence`, preserving the current normalized scene values; - and - - the plan value is an object with exact key `boundary_caveats`, containing - the normalized caveat array. -5. Preserve accepted scene boundary caveats as `contracts.Warning` values as - well as in the plan annotation so warning behavior remains visible. -6. Remove `chunks.scenes` from the D&D scene module's provided capabilities. - Keep only `chunks`. -7. Change the runner’s non-cached chunk path to: - - call `Plan` under the existing retry and debug attempt boundary; - - validate the plan structurally; - - materialize chunks; - - run the configured chunk-validator chain on materialized chunks; - - preserve retry and rejection semantics; and - - pass materialized chunks to lanes exactly as before. -8. Update chunk debug payloads to distinguish the generated plan from - materialized chunks. Plan annotations may appear only in opt-in debug data. -9. Leave current chunk checkpoint loading and recording temporarily in place, - storing materialized chunks, so existing resume tests remain green until - Stage 4 removes that path. - -### Tests - -Update focused module and framework tests to cover: - -- exact generic ranges and overlap; -- exact D&D scene annotations and warning conversion; -- annotations absent from generic plans; -- framework materialization rather than module materialization; -- chunk validators receiving materialized annotated chunks; -- retries around generation plus validation; -- stable chunk IDs independent of producing module; and -- no `chunks.scenes` resolution dependency. - -Run at minimum: - -```sh -go test ./internal/modules/generic/chunk/units -go test ./internal/modules/dnd/chunk/scenes -go test ./internal/framework/contracts ./internal/framework/pipeline -go test ./internal/modules/integration ./internal/cli -``` - -### Exit criteria - -Every chunk module uses the single plan contract, the runner always -materializes, production behavior remains functionally equivalent apart from -framework-owned stable chunk IDs and namespaced annotations, and the repository -is green. - -## Stage 3: Add dedicated cache configuration and the plan store - -**Status:** Complete - -### Objective - -Implement the durable source-addressed store and its independent cache-root -configuration without changing the runner to consume the store. - -### Read first - -- `docs/config.md` -- `docs/operations.md` -- `internal/core/config/*` -- `internal/core/workspace/*` -- `internal/framework/checkpoint/` as the local pattern for a workspace-backed - implementation -- `internal/cli/run.go` configuration loading, without wiring the store yet - -### Implement - -1. Add `ChunkCacheMode` and strict parsing in the pipeline package. -2. Add `WorkspaceChunkCacheConfig` with `Mode` and `Directory` to config and - file config. The YAML fields are `workspace.chunk_cache.mode` and - `workspace.chunk_cache.directory`. -3. Set default mode to `auto`; support - `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`; preserve precedence file then - environment. CLI precedence arrives in Stage 5. -4. Support `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR` with environment-over-file - precedence. Trim the value; an unset or empty effective value selects the - per-user default. Do not add a CLI directory override. -5. Add `workspace.DefaultChunkPlanRoot`, accepting an injectable - user-cache-directory resolver and returning - `filepath.Join(userCacheDir, "notarius", "chunk-plans")`. Production later - supplies `os.UserCacheDir`; tests supply a stub. Reject resolver errors and - empty returned directories. Use this exact signature: - - ```go - func DefaultChunkPlanRoot(userCacheDir func() (string, error)) (string, error) - ``` -6. Treat a non-empty `workspace.chunk_cache.directory` as the plan root itself, - cleaned with the same path rules as other workspace roots. It does not gain - an implicit `chunk-plans` suffix. `workspace.directory` must not affect this - root, and the chunk-cache directory must not affect checkpoint, diagnostics, - debug, or output roots. Do not change `workspace.FromConfig` or - `workspace.Settings` for this feature. -7. Add the plan record, producer, decision, and store interfaces described - above in the pipeline package. -8. Add `internal/framework/chunkplan` with: - - a filesystem implementation; - - a constructor that accepts the already resolved chunk-plan root; - - path `/<64-character-source-sha256-hex>/plan.json`, after - strictly validating and stripping the logical `sha256:` prefix; - - schema `notarius.chunk-plan.v1`; - - directories mode `0700` and file mode `0600`; - - atomic same-directory temp-file rename; - - strict envelope decoding and unknown-field rejection; - - complete digest and annotation validation; and - - no raw reference content. -9. `Load` distinguishes missing, invalid, and operational error. `Save` always - atomically replaces and implements last-successful-write semantics. -10. Keep store fakes local to their tests; do not add a production in-memory or - no-op store because `bypass` and a nil collaborator already express - disabled persistence. -11. Do not wire the store into `Runner` or CLI execution in this stage. - -### Tests - -Cover: - -- config defaults, YAML decoding, environment override, invalid values, - cloning, redaction, and validation; -- directory environment-over-file precedence, empty-value fallback, cloning, - redaction, cleaning, and validation; -- exact injected per-user default and resolver failure or empty-result errors; -- independence in both directions between `workspace.directory` and - `workspace.chunk_cache.directory`; -- exact acceptance of `/var/cache/notarius/chunk-plans` as a configured root - without writing to that real path; -- safe full-digest paths and rejection of malformed digests; -- `0700`/`0600` permissions where supported; -- envelope round trip, strict decoding, source and plan digest mismatch, - invalid annotations, schema mismatch, missing files, operational read errors, - atomic replacement, and concurrent writers; and -- absence of raw reference content in persisted JSON. - -Run at minimum: - -```sh -go test ./internal/core/config ./internal/core/workspace -go test ./internal/framework/chunkplan ./internal/framework/pipeline -go test ./internal/cli -``` - -### Exit criteria - -Configuration and storage are complete and independently tested, but normal -runs still use Stage 2’s generate-and-materialize behavior. - -## Stage 4: Integrate cache policy and remove chunk checkpoints - -**Status:** Complete - -### Objective - -Make the runner’s chunk path use the canonical plan store under all three modes -and establish it as the only chunk-reuse mechanism. - -### Read first - -- `internal/framework/pipeline/runner.go` -- `internal/framework/pipeline/checkpoint.go` -- `internal/framework/pipeline/runner_concurrent.go` -- `internal/framework/checkpoint/loader.go` -- `internal/framework/checkpoint/recorder.go` -- focused runner retry, rejection, debug, and checkpoint tests - -### Implement - -1. Add `ChunkCacheMode` and `ChunkPlans` to `RunInput`. Empty mode means - `bypass` for direct framework callers. -2. Add synchronized store wrapping only if runner access can be concurrent; - otherwise document and test the intentionally serial chunk-plan access. -3. Implement mode behavior: - - `auto` hit: load, structurally validate against the current source, - materialize, then run configured chunk validators once; - - `auto` miss or invalid: generate through existing retries, validate, - materialize, validate chunks, then save; - - `bypass`: generate and validate normally without calling `Load` or `Save`; - - `refresh`: generate and validate normally, then replace through `Save`. -4. A valid-hit validator rejection is recorded without retry, store mutation, - or implicit regeneration. A store I/O error is a framework error. -5. A failed or rejected generated plan is never saved. An invalid old file is - overwritten only after a replacement passes module execution, structural - validation, materialization, and configured validators. -6. Build producer provenance from the prepared input module key, prepared chunk - module key, resolved chunk-stage LLM profile, chunk-target auxiliary - reference provenance, and chunk-module manifest metadata. Clone and redact - according to existing boundaries. -7. Replay stored producer warnings on a hit, then append current validation - warnings. Do not duplicate warnings from discarded generation attempts. -8. Remove all chunk methods and types from `CheckpointRecorder`, - `CheckpointLoader`, synchronized wrappers, workspace recorder/loader, and - checkpoint manifests and envelopes. -9. Remove runner chunk-checkpoint decisions and events. Retain source, extract, - merge, and normalize checkpoints. Keep downstream fingerprints based on the - digest of materialized chunks. -10. Ignore existing on-disk `chunk/` checkpoint files; do not read, delete, or - migrate them. -11. Update debug behavior: - - hit records lookup and materialization but no chunk module attempt or LLM - call; - - generated, bypassed, and refreshed paths retain attempt debug; and - - default debug envelopes remain redacted. - -### Tests - -Add table-driven runner tests for every mode and outcome: - -- hit, missing, invalid, and I/O error; -- generation success, error, rejection, retry, and cancellation; -- validator warning, rejection, and error on hits and generated plans; -- bypass performs zero store calls; -- refresh performs no load and exactly one successful save; -- invalid record remains untouched when replacement fails; -- stored warnings replay once; -- no module or LLM call on hit; -- downstream chunk digest changes on refresh; -- legacy chunk checkpoint files are ignored; and -- source and lane checkpoint behavior remains intact. - -Run at minimum: - -```sh -go test ./internal/framework/pipeline -go test ./internal/framework/checkpoint -go test ./internal/modules/integration -``` - -### Exit criteria - -The runner implements complete cache semantics, chunk checkpoints no longer -exist as a code path, all direct framework tests are green, and production CLI -runs are not yet wired to persistent plan storage. - -## Stage 5: Wire persistent policy into the CLI - -**Status:** Complete - -### Objective - -Make normal CLI runs resolve the effective mode and dedicated chunk-plan root, -construct the durable plan store when appropriate, and pass it into the runner. - -### Read first - -- `docs/cli.md` -- `docs/config.md` -- `cmd/notarius/main.go` and `internal/cli/run.go` -- CLI flag, environment, invocation-metadata, and exit-code tests -- the workspace and configuration code changed in Stage 3 - -### Implement - -1. Add the exact flag `--chunk_cache `. Represent an - omitted CLI value separately from an explicit value so precedence is not - inferred from the default printed by the flag package. -2. Resolve the effective mode in this order: explicit CLI flag, environment, - configuration file, default `auto`. An invalid CLI flag is a usage error - with exit code 2; invalid file or environment configuration follows the - existing configuration-load failure path and exit code 1. -3. Include an explicitly requested CLI value in invocation metadata and the - resolved mode in effective-configuration metadata without adding field-level - origin tracking or plan and annotation payloads. Name the optional - invocation JSON field `chunk_cache_override`; the resolved mode already - appears at `workspace.chunk_cache.mode` in the redacted effective config. -4. Add `UserCacheDir func() (string, error)` and - `ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory` to `cli.Options`, - defaulting them to `os.UserCacheDir` and - `chunkplan.NewFilesystemStore`. For `auto` and `refresh`, choose the non-empty - effective `workspace.chunk_cache.directory`; otherwise resolve the per-user - default with `workspace.DefaultChunkPlanRoot`. Invoke the store factory with - that exact root and pass the result and effective mode through `RunInput`. - Never append `chunk-plans` to an explicit directory. The injectable factory - lets CLI tests assert root selection without writing to production paths. -5. For `bypass`, do not resolve, inspect, or create the chunk-plan root and pass - no store. Existing general-workspace setup remains unchanged and independent: - `workspace.directory`, resume, diagnostics, and debug continue to use their - current paths and semantics. -6. Keep resume policy independent of plan reuse. Enabling, disabling, or - selecting a resume run must not change the chunk-cache mode or path. -7. Configuration validation and pipeline-listing commands validate the mode but - do not create workspace directories or plan files. -8. Update CLI usage text only as needed to expose the new flag. Defer all - narrative documentation to Stage 8. - -### Tests - -Add CLI and integration tests for: - -- a default `auto` run writing a plan and a second independent invocation - reusing it without a chunk operation or chunk LLM call; -- file, environment, and CLI precedence, including persistent `refresh` in - file and environment configuration; -- `bypass` performing no plan-store access and resolving or creating no default - chunk-plan root; -- invalid flag, environment, and file values with their established error - classifications; -- an explicit chunk-plan root from file and environment, including exact use of - `/var/cache/notarius/chunk-plans` without an appended suffix, using a - recording store factory and never writing to that real path; -- an explicit root succeeding without calling a failing per-user cache resolver; -- an injected per-user default resolving exactly to - `/notarius/chunk-plans`; -- `workspace.directory` changes having no effect on chunk-plan placement and a - chunk-cache directory having no effect on other workspace state; -- per-user cache-root discovery failure under `auto`, `refresh`, and `bypass`, - including the allowed no-state `bypass` case, plus unwritable-store failures; -- resume plus each cache mode, showing independent behavior; -- validation and listing commands having no filesystem side effects; and -- reuse across changed pipeline, chunk module/options, references, validators, - and LLM settings when the source digest is unchanged. - -Run at minimum: - -```sh -go test ./internal/cli -go test ./internal/framework/pipeline ./internal/framework/chunkplan -go test ./internal/modules/integration -``` - -### Exit criteria - -The production CLI implements the complete mode and precedence contract, -independent invocations reuse source-addressed plans by default, bypass and -refresh have exact state semantics, and the repository is green. - -## Stage 6: Expose producer provenance and safe diagnostics - -**Status:** Complete - -### Objective - -Make every output and diagnostic record distinguish the requested chunk module -from the producer of the effective plan without exposing sensitive payloads by -default. - -### Read first - -- `docs/integrations/json-output.md` -- `docs/internal/diagnostics.md` -- manifest assembly in `internal/framework/pipeline/runner.go` -- `internal/core/artifacts/artifacts.go` -- current diagnostics and debug-envelope implementations and tests - -### Implement - -1. Add `ChunkPlanManifest` to the run manifest and all required clone, - conversion, and serialization paths. -2. Populate `mode` and exactly one action after a candidate plan has passed - structural validation and materialization: - - `reused` for an `auto` hit; - - `generated` for an `auto` miss or invalid record; - - `refreshed` for generation under `refresh`; and - - `bypassed` for generation under `bypass`. - Retain that action if the current chunk-validator chain rejects the - candidate; publication still follows Stage 4's accept-only rule. If no - candidate reaches materialization, leave action and candidate plan fields - empty in the partial failure manifest and report the failure in diagnostics. -3. Preserve the existing top-level `chunker` and its module metadata as the - requested module. Never overwrite those fields with cached producer data. -4. Populate producer module, LLM profile, references, metadata, creation time, - source digest, plan digest, and schema from the effective stored or newly - generated record. A reused record reports its original producer, even when - it differs from the requested module. -5. Define producer LLM profile as the resolved chunk-stage profile used during - generation. Leave it empty for producers that did not use an LLM; do not - substitute the current run's requested profile on reuse. -6. Keep producer references as existing redacted provenance records. Deep-clone - metadata, references, warnings, annotations, and raw JSON at every ownership - boundary. -7. Add a redacted `chunk-plan.json` diagnostic artifact, backed by a summary on - `RunOutput` so the CLI can write it on success, rejection, or framework - failure. It contains mode, source and candidate plan digests, requested - module, lookup status and redacted reason, action, validation status, and - publication status. Use these closed values: - - lookup: `hit`, `missing`, `invalid`, or `skipped`; - - validation: `not_run`, `approved`, `approved_with_warnings`, `rejected`, - or `error`; and - - publication: `not_requested`, `published`, or `not_published`. - It contains no source units, materialized chunk content, annotation bytes, - reference content, prompts, model responses, or raw invalid-file content. -8. Keep complete plan, annotations, attempts, and materialized chunks confined - to existing opt-in debug artifacts and apply their normal restrictive file - handling. -9. Preserve JSON-output compatibility by adding fields rather than renaming - unrelated existing fields. Omit producer-only optional values when absent in - the same style as the surrounding manifest schema. - -### Tests - -Cover: - -- all four actions and their exact manifest fields; -- requested and producer modules differing on a hit; -- producer metadata remaining stable when current configuration changes; -- LLM-backed and non-LLM producers; -- deep-clone and mutation safety for raw annotations and metadata; -- default diagnostic redaction, including invalid stored files; -- complete data appearing only under explicit debug configuration; and -- JSON serialization and compatibility fixtures. - -Run at minimum: - -```sh -go test ./internal/core/artifacts -go test ./internal/framework/pipeline -go test ./internal/cli -``` - -### Exit criteria - -Manifests and diagnostics explain which plan was requested, selected, and -produced; default output remains redacted; debug behavior is explicit; and all -compatibility tests pass. - -## Stage 7: Add cross-run, corruption, and concurrency hardening - -**Status:** Complete - -### Objective - -Exercise the completed design across package boundaries and adversarial state, -then close any correctness gaps without changing the settled contract. - -### Read first - -- all tests added in Stages 1–6 -- end-to-end CLI and module integration test harnesses -- atomic-write, cancellation, retry, and race-sensitive framework code -- repository import-boundary tests - -### Implement - -1. Add black-box tests using separate CLI application instances and a shared - temporary chunk-plan root. Prove reuse across different pipelines, - configured chunk modules, options, references, lanes, validators, and LLM - profiles. -2. Prove that a changed canonical source digest selects a different path, while - byte-identical canonical sources reuse the same plan. -3. Exercise generic plans through D&D pipelines and D&D-annotated plans through - generic pipelines. Optional annotations must survive, while no downstream - component may require `dnd/scenes` merely because the requested chunker is - the D&D scene module. -4. Add corruption cases for truncated JSON, unknown fields, schema mismatch, - source mismatch, plan-digest mismatch, bad boundaries, invalid annotations, - and materialization failure. Verify `auto` regeneration and non-destructive - failure behavior exactly. -5. Add concurrent reader/writer and writer/writer tests. Readers observe either - the old complete envelope or the new complete envelope; after successful - writers, the final file is one complete valid record. Do not assert which - writer wins. -6. Exercise interruption before save, during temp-file creation, and around - rename where the implementation permits deterministic fault injection. - Prior valid state must remain readable unless a complete replacement was - published. -7. Verify cancellation, retry limits, warning order, error classification, and - debug ordering on misses and hits. No cache hit may consume chunk-stage retry - budget or invoke the module. -8. Verify legacy chunk checkpoint files cannot influence reuse and that lane - resume still keys from effective materialized chunk digests after refresh. -9. Extend import-boundary tests so `internal/core/source` owns no framework or - module dependencies and the generic store owns no concrete module imports. -10. Ensure examples and all tests isolate workspace state in temporary - directories and isolate chunk-plan state with a temporary configured root - or explicitly select `bypass`; tests must never write to the real per-user - default or `/var/cache/notarius/chunk-plans`. -11. Run the race detector over the packages that access the plan store and - runner concurrently, and fix any race within feature scope. - -### Tests - -In addition to the new focused tests, run: - -```sh -go test ./... -go test -race ./internal/framework/chunkplan ./internal/framework/pipeline -go vet ./... -go build ./cmd/notarius -``` - -### Exit criteria - -The feature is demonstrated across independent runs, corruption and concurrent -publication cannot expose partial state, cross-domain annotations remain -optional, cancellation and retry behavior is deterministic, and all repository -and race checks pass. - -## Stage 8: Publish current-behavior documentation - -**Status:** Complete - -### Objective - -After implementation is complete, make the supported behavior discoverable -without duplicating canonical ownership across documents. - -### Read first - -- `docs/policy/architecture.md` -- `docs/policy/documentation.md` -- every current-behavior document linked below -- the final code, flags, schemas, examples, and tests from Stages 1–7 - -### Implement - -1. Update `docs/policy/architecture.md` with the - plan-generation/materialization - boundary, source-addressed reuse, and the independent cache-state surface at - the level appropriate for the canonical system overview. -2. Update `docs/internal/pipeline.md` with mode behavior, hit/miss control flow, - validator placement, retry behavior, and removal of chunk checkpoints. -3. Update `docs/internal/modules.md` with the `Chunker.Plan` contract, generic - boundaries, optional annotation namespaces, and framework materialization. -4. Update `docs/internal/overview.md` with the new plan-store package and - current state ownership, and remove chunk checkpoint ownership from its - checkpoint description. Keep detailed checkpoint and plan execution flow in - `docs/internal/pipeline.md` rather than creating a parallel internal state - reference. -5. Update `docs/config.md` with `workspace.chunk_cache.mode`, - `workspace.chunk_cache.directory`, both environment variables, strict - values, precedence, and the per-user cache-root default. State explicitly - that `workspace.directory` does not affect plan placement and that the - configured directory is the root itself. Keep field-level details owned - here. Document that Unix `os.UserCacheDir` rejects a relative - `XDG_CACHE_HOME` and that this is a configuration error rather than a fallback. -6. Update `docs/cli.md` with `--chunk_cache` semantics and examples. Link to - configuration rather than restating its entire precedence model. -7. Update `docs/operations.md` with plan location, permissions, invalid-state - recovery, refresh and bypass operations, concurrency semantics, sensitive - debug data, deletion as a recoverable but potentially expensive - regeneration event, and the no-history/no-rollback limitation. Document both: - - the ordinary Linux per-user default, - `$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is a valid - absolute path, or `$HOME/.cache/notarius/chunk-plans` when it is unset; and - - `/var/cache/notarius/chunk-plans` as the recommended configured root for a - system-wide Linux deployment running under a dedicated service account. - Explain that the system path is not the unprivileged default, show the - following minimal configuration, require the operator or package installer - to provision it with restrictive service-account ownership and permissions, - and warn against sharing the cache across mutually untrusted users: - - ```yaml - workspace: - chunk_cache: - directory: /var/cache/notarius/chunk-plans - ``` -8. Update `docs/integrations/json-output.md` and - `docs/internal/diagnostics.md` with requested versus producer provenance, - actions, warnings, and redaction boundaries. -9. Keep package-level plan/range/annotation and materialization invariants in - `docs/internal/pipeline.md`; no separate source document is needed unless - the implemented subsystem becomes too large for that canonical owner. -10. Update examples only when they materially clarify operation; examples must - use temporary or explicitly configured workspace and chunk-cache paths. -11. Remove future-tense disclaimers from the feature roadmap, mark its - completion outcomes as achieved, and mark all completed stages in this - document. Keep the roadmap as historical target-state context rather than - copying implementation details into it. -12. Validate every changed link, heading, code symbol, flag, environment name, - path, and JSON field against the implementation. Search for stale - `Chunker.Chunk`, `ChunkResult`, `chunks.scenes`, and chunk-checkpoint claims. +1. Remove the feature roadmap's statement that ADR-0005 is unimplemented and + describe it as the implemented target-state record. Mark its completion + outcomes as achieved without adding staged implementation detail. +2. Recheck architecture, diagnostics, operations, pipeline, and module + documentation against the remediated behavior. Update a canonical document + only where Stages 9 through 11 changed a current operator or developer + contract; do not duplicate implementation detail across documents. +3. Mark Stages 9 through 12 complete only after their respective code, + documentation, focused tests, and repository-wide validation have passed. + Update this document's introduction to state that all stages are complete and + that it remains as a concise historical implementation record. +4. Validate every changed link, heading, symbol, diagnostic field, permission, + and filesystem claim against the implementation. ### Tests and validation @@ -953,12 +569,12 @@ go build ./cmd/notarius git diff --check ``` -Also run repository link or documentation checks if present, and manually -inspect the reading map in `docs/development.md` and component links in -`docs/internal/overview.md` for canonical ownership and navigation consistency. +Run any repository documentation or link checker if present, and manually +inspect the reading map in [Development](../development.md) and component links +in [Internal Overview](../internal/overview.md). ### Exit criteria -All canonical documents describe implemented behavior, operational guidance is -complete and non-duplicative, no stale chunk API or checkpoint documentation -remains, roadmap status is accurate, and the repository is green. +The feature roadmap and canonical references describe implemented behavior, +this plan accurately records completion, documentation ownership remains clear, +all links and names are current, and the repository-wide validation is green.