eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update implementation plan to incorporate follow-up fixes

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-17 20:55:39 -05:00
parent 96a49bb7cd
commit b3328b93e5
1 changed files with 256 additions and 640 deletions
Download Patch File Download Diff File Expand all files Collapse all files

896
docs/roadmap/implementation.md
View File

@@ -1,12 +1,14 @@
# ADR-0005 Implementation Record # ADR-0005 Implementation Record
This document records the completed implementation of the target state in This document records the implementation of the target state in
[ADR-0005 Feature Roadmap](adr0005.md), governed by [ADR-0005 Feature Roadmap](adr0005.md), governed by
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). [ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md).
All stages below are complete. The plan remains as historical target-state Stages 1 through 8 summarize the completed initial implementation. Stages 9
context; current behavior is documented in the canonical references linked from through 12 track pending remediation identified by the post-implementation
[Development](../development.md). review. Current behavior is documented in the canonical references linked from
[Development](../development.md); those references must not describe the Stages
9 through 12 target state until the corresponding stage is complete.
## Execution Rules ## Execution Rules
@@ -281,666 +283,280 @@ ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"`
The runner initializes it with the effective mode and requested chunk module, The runner initializes it with the effective mode and requested chunk module,
then fills candidate and producer fields when a plan reaches materialization. then fills candidate and producer fields when a plan reaches materialization.
## Stage 1: Finalize ADR and add the source-zone plan model ## Completed Stage Summaries
### Stage 1: Finalize ADR and add the source-zone plan model
**Status:** Complete **Status:** Complete
Accepted ADR-0005 and added the domain-neutral chunk-plan, range, annotation,
validation, digest, cloning, and deterministic materialization primitives in
`internal/core/source`.
### Stage 2: Replace the chunk operation with plan generation
**Status:** Complete
Replaced chunk-returning module operations with `Chunker.Plan`, converted the
generic and D&D scene chunkers to produce ranges and optional annotations, and
made the runner materialize and validate generated plans.
### Stage 3: Add dedicated cache configuration and the plan store
**Status:** Complete
Added strict cache modes, independent per-user or explicitly configured
chunk-plan roots, the source-addressed filesystem store, restrictive
permissions, envelope validation, and atomic replacement.
### Stage 4: Integrate cache policy and remove chunk checkpoints
**Status:** Complete
Implemented runner semantics for `auto`, `bypass`, and `refresh`; retained
current chunk-validator behavior; made accepted canonical plans the sole chunk
reuse mechanism; and removed chunk checkpoint loading and recording.
### Stage 5: Wire persistent policy into the CLI
**Status:** Complete
Added CLI, environment, and file configuration precedence; per-user root
resolution; persistent store construction; bypass isolation; and cross-run
reuse through normal CLI execution.
### Stage 6: Expose producer provenance and safe diagnostics
**Status:** Complete
Added requested-versus-producer chunk-plan provenance to manifests and a
payload-free chunk-plan decision summary to normal diagnostics, with complete
plan material confined to opt-in debug artifacts.
### Stage 7: Add cross-run, corruption, and concurrency hardening
**Status:** Complete
Added integration, corruption, interruption, concurrency, compatibility,
cross-domain annotation, import-boundary, and race coverage for the implemented
cache design.
### Stage 8: Publish current-behavior documentation
**Status:** Complete
Updated architecture, pipeline, module, configuration, CLI, operations,
diagnostics, integration, and overview documentation for the implemented
feature and its per-user and system-wide deployment guidance.
## Stage 9: Redact invalid cache-record diagnostics
**Status:** Pending
### Objective ### Objective
Accept the architectural decision and introduce domain-neutral plan, Ensure malformed or incompatible cache contents cannot enter normal errors,
annotation, validation, digest, clone, and materialization primitives without warnings, manifests, or diagnostics while retaining useful closed lookup
changing chunk-module execution yet. decisions.
### Read first
- [Architecture Policy](../policy/architecture.md), especially secret handling
- [Diagnostics Internals](../internal/diagnostics.md)
- `internal/framework/chunkplan/store.go` and its focused tests
- `internal/framework/pipeline/runner_chunk_plan.go` and its focused tests
- `internal/core/artifacts/artifacts.go`
- the CLI path that writes `chunk-plan.json`
### Implement
1. Stop interpolating decoder errors, unknown field names, schema values,
annotation namespaces, timestamps, paths, or any other stored content into
recoverable `ChunkPlanDecision.Reason` values.
2. At the normal-diagnostic boundary, ignore the reason supplied by a
`ChunkPlanStore` and derive `ChunkPlanSummary.LookupReason` solely from
status using exactly:
- `hit`: `stored chunk plan is valid`;
- `missing`: `chunk plan not found`;
- `invalid`: `stored chunk plan is invalid`; and
- `skipped`: `chunk plan lookup skipped`.
3. Keep operational read failures as framework errors rather than recoverable
invalid-record decisions. Their normal error text must identify the failed
operation without including cache-file content.
4. Do not add raw invalid-record details to another default surface. Direct
operator inspection or a future explicitly sensitive debug facility may
expose them, but this stage does not need to add such a facility.
### Tests
Add focused tests proving that sentinel secrets placed in unknown field names,
schema values, annotation namespaces, timestamps, and malformed trailing JSON
appear in neither the store decision nor `chunk-plan.json`. Add a runner test
with a custom store whose reason contains a sentinel and prove that the summary
uses the fixed status-derived reason. Preserve useful missing, hit, invalid, and
operational-error classification.
Run focused tests for `internal/framework/chunkplan`,
`internal/framework/pipeline`, and `internal/cli`, then perform the
repository-wide validation in the execution rules.
### Exit criteria
Every normal lookup reason belongs to the fixed safe vocabulary, hostile
invalid-file content cannot reach default diagnostics, operational failures
remain correctly classified, and all focused and repository-wide checks pass.
## Stage 10: Enforce rooted chunk-plan filesystem access
**Status:** Pending
### Objective
Make the cache's path-confinement guarantee resistant to symlinked cache entries
while preserving atomicity, permissions, and concurrent access semantics.
### Read first
- [Architecture Policy](../policy/architecture.md), especially state and path
invariants
- [Operations](../operations.md)
- `internal/framework/chunkplan/store.go` and all store tests
- Go's `os.Root` API for the repository's declared Go version
### Implement
1. Treat the configured cache root as the filesystem trust boundary. A symlink
in the operator-supplied root path itself may be followed when the root is
opened, but every store-owned operation after that point must remain beneath
the opened root.
2. Perform digest-directory, plan-file, temporary-file, and rename operations
through `os.Root` or an equivalently race-resistant directory-relative
mechanism. Do not rely on `filepath.Join` plus a preflight `Lstat` as the
confinement boundary.
3. Reject symlinks and unexpected file types at the store-owned digest-directory
and `plan.json` positions. A missing cache root on `Load` remains a normal
miss and is not created; `Save` creates the root and digest directory as
needed.
4. Preserve the exact
`<root>/<full-source-sha256-hex>/plan.json` layout, `0700` directories,
`0600` plan files, strict digest validation, and no implicit path suffix.
5. Publish using an exclusively created random temporary file inside the digest
directory, sync and close it, then atomically rename it through the same
rooted handle. Remove temporary files on every pre-rename failure. Preserve
last-successful-write and complete-reader semantics.
6. Keep store access serial at the runner boundary and safe for concurrent
independent store callers. Do not introduce a lock, history, or rollback
protocol.
### Tests
Add load and save cases for digest-directory and plan-file symlinks that resolve
outside the configured root. Prove that they are rejected and that no outside
file is read, created, chmodded, or replaced. Include unexpected file types and
failed-publication preservation.
Retain and run tests for absent-root behavior, exact paths, permissions, round
trip, malformed digests, atomic replacement, interrupted writes, temporary-file
cleanup, concurrent readers, and concurrent writers. Run the race detector for
`internal/framework/chunkplan`, then perform the repository-wide validation in
the execution rules.
### Exit criteria
No store-owned operation can escape the opened root through a cache entry,
normal layout and permissions remain unchanged, publication remains atomic,
concurrency tests and the race detector pass, and the repository is green.
## Stage 11: Deep-clone source metadata during materialization
**Status:** Pending
### Objective
Ensure source documents, materialized chunks, and separate materializations
never share mutable metadata storage, regardless of the concrete JSON-shaped
map, slice, array, interface, raw-message, or byte-slice types used by an input
adapter.
### Read first ### Read first
- `internal/core/source/source.go` - `internal/core/source/source.go`
- `internal/core/source/digest.go` - `internal/core/source/chunk_plan.go` and its focused tests
- `internal/core/source/validation.go` - `internal/core/source/digest.go` and validation behavior
- `internal/core/source/source_test.go` - every metadata clone helper and ownership boundary found with `rg`
- every current `source.Chunk` clone or serialization helper found with `rg` - focused runner tests that hand chunks to validators and extractors
### Implement ### Implement
1. Revise ADR-0005 before acceptance: 1. Replace the concrete-type allowlist used by plan materialization with one
- confirm it retains one source-addressed mutable plan and atomic refresh; reusable source-owned deep-clone implementation for supported
- preserve the independent cache-surface decision; JSON-serializable metadata.
- keep filesystem and CLI details out of the ADR; and 2. Recursively clone string-keyed maps, slices, arrays, interfaces,
- change status from `Proposed` to `Accepted`. `json.RawMessage`, and byte slices without sharing mutable backing storage.
2. Add `ChunkAnnotations`, `ChunkPlan`, and `ChunkRange` in Preserve scalar values and the logical JSON representation. Typed composite
`internal/core/source`. containers must not fall through as shared values.
3. Add canonical annotation validation and cloning. Canonicalize each raw JSON 3. Detect cyclic or unsupported values and return a contextual error from
value by decoding with `json.Decoder.UseNumber` and re-encoding; reject materialization rather than retaining an alias, panicking, or recursing
blank namespaces, invalid JSON, trailing JSON values, and non-finite or indefinitely.
unsupported values. 4. Use the source-owned helper at pipeline ownership boundaries where doing so
4. Add plan validation against a `SourceDocument` using the rules in Planned removes a duplicate incomplete implementation. Do not introduce a dependency
Contracts. from `internal/core/source` to a framework package and do not expand this
5. Add `DigestChunkPlan` and deterministic `MaterializeChunkPlan`. stage into unrelated metadata-schema redesign.
6. Add `Annotations` and `PlanAnnotations` to `source.Chunk`, `DigestChunk`, 5. Preserve deterministic materialized content, chunk digests, annotations, IDs,
every clone helper, debug envelope, and serialized chunk-validation references, and metadata fields for all currently supported production
representation. Ensure all ownership boundaries clone raw bytes and retain inputs.
the two scopes separately.
7. Do not change the chunker interface or runner flow in this stage.
### Tests ### Tests
Add focused tests for: Cover typed nested maps, slices of typed maps, arrays, interfaces,
`json.RawMessage`, and byte slices. Mutate each materialized value and prove
that neither the source document nor a separately materialized chunk changes.
Add cyclic and unsupported-value cases that fail deterministically with
contextual errors. Retain the existing exact-materialization and digest tests.
- annotation canonicalization, invalid JSON, blank namespaces, and mutation Run focused tests for `internal/core/source`,
safety; `internal/framework/pipeline`, and affected module packages, then perform the
- plan source mismatch, missing ranges, missing units, backward ranges, repository-wide validation in the execution rules.
duplicate starts, gaps, and overlap;
- stable plan digests and digest changes for boundary or annotation changes;
- exact materialized IDs, references, content bytes, metadata, annotations,
gaps, and overlap;
- repeated byte-identical materialization; and
- plan- and range-annotation preservation through existing chunk clones, debug
conversion, serialized validation, and downstream extract requests.
Run at minimum:
```sh
go test ./internal/core/source
go test ./internal/framework/contracts ./internal/framework/pipeline
```
### Exit criteria ### Exit criteria
The repository is green; ADR-0005 is accepted; plan primitives are complete and Every supported mutable metadata value has independent ownership across source,
tested; current modules and the runner still behave as before. chunk, validation, and extraction boundaries; invalid metadata fails safely;
current materialized bytes remain stable; and all checks pass.
## Stage 2: Replace the chunk operation with plan generation ## Stage 12: Finalize remediation documentation and roadmap status
**Status:** Complete **Status:** Pending
### Objective ### Objective
Make all chunk modules produce plans and make the runner materialize those plans Make the roadmap and canonical documentation accurately describe the completed
without adding durable reuse yet. feature after Stages 9 through 11 land.
### Read first ### Read first
- `docs/internal/modules.md` - [Documentation Policy](../policy/documentation.md)
- `docs/internal/pipeline.md` - [ADR-0005 Feature Roadmap](adr0005.md)
- `internal/framework/contracts/contracts.go` - [Architecture Policy](../policy/architecture.md)
- `internal/framework/pipeline/chunker_registry.go` - [Diagnostics Internals](../internal/diagnostics.md)
- `internal/framework/pipeline/runner.go` - [Operations](../operations.md)
- `internal/framework/pipeline/chunk_validation.go` - the final code and tests from Stages 9 through 11
- both production chunk-module packages and all chunk fakes in tests
### Implement ### Implement
1. Replace `ChunkResult` with `ChunkPlanResult` and `Chunker.Chunk` with 1. Remove the feature roadmap's statement that ADR-0005 is unimplemented and
`Chunker.Plan`; keep the request, registry, module keys, and stage name. describe it as the implemented target-state record. Mark its completion
2. Update all fakes, registry tests, preparation tests, integration helpers, and outcomes as achieved without adding staged implementation detail.
compile-time interface assertions in the same stage. Do not retain adapters 2. Recheck architecture, diagnostics, operations, pipeline, and module
or a legacy chunk-returning interface. documentation against the remediated behavior. Update a canonical document
3. Convert `generic/chunk/units` to emit ranges only. Preserve `max_units` and only where Stages 9 through 11 changed a current operator or developer
`overlap_units` semantics. contract; do not duplicate implementation detail across documents.
4. Convert `dnd/chunk/scenes` to emit namespace `dnd/scenes` at both scopes: 3. Mark Stages 9 through 12 complete only after their respective code,
- each range value is an object with exact keys `short_title`, documentation, focused tests, and repository-wide validation have passed.
`primary_mode`, `main_participants`, `summary`, `boundary_note`, and Update this document's introduction to state that all stages are complete and
`boundary_confidence`, preserving the current normalized scene values; that it remains as a concise historical implementation record.
and 4. Validate every changed link, heading, symbol, diagnostic field, permission,
- the plan value is an object with exact key `boundary_caveats`, containing and filesystem claim against the implementation.
the normalized caveat array.
5. Preserve accepted scene boundary caveats as `contracts.Warning` values as
well as in the plan annotation so warning behavior remains visible.
6. Remove `chunks.scenes` from the D&D scene module's provided capabilities.
Keep only `chunks`.
7. Change the runner’s non-cached chunk path to:
- call `Plan` under the existing retry and debug attempt boundary;
- validate the plan structurally;
- materialize chunks;
- run the configured chunk-validator chain on materialized chunks;
- preserve retry and rejection semantics; and
- pass materialized chunks to lanes exactly as before.
8. Update chunk debug payloads to distinguish the generated plan from
materialized chunks. Plan annotations may appear only in opt-in debug data.
9. Leave current chunk checkpoint loading and recording temporarily in place,
storing materialized chunks, so existing resume tests remain green until
Stage 4 removes that path.
### Tests
Update focused module and framework tests to cover:
- exact generic ranges and overlap;
- exact D&D scene annotations and warning conversion;
- annotations absent from generic plans;
- framework materialization rather than module materialization;
- chunk validators receiving materialized annotated chunks;
- retries around generation plus validation;
- stable chunk IDs independent of producing module; and
- no `chunks.scenes` resolution dependency.
Run at minimum:
```sh
go test ./internal/modules/generic/chunk/units
go test ./internal/modules/dnd/chunk/scenes
go test ./internal/framework/contracts ./internal/framework/pipeline
go test ./internal/modules/integration ./internal/cli
```
### Exit criteria
Every chunk module uses the single plan contract, the runner always
materializes, production behavior remains functionally equivalent apart from
framework-owned stable chunk IDs and namespaced annotations, and the repository
is green.
## Stage 3: Add dedicated cache configuration and the plan store
**Status:** Complete
### Objective
Implement the durable source-addressed store and its independent cache-root
configuration without changing the runner to consume the store.
### Read first
- `docs/config.md`
- `docs/operations.md`
- `internal/core/config/*`
- `internal/core/workspace/*`
- `internal/framework/checkpoint/` as the local pattern for a workspace-backed
implementation
- `internal/cli/run.go` configuration loading, without wiring the store yet
### Implement
1. Add `ChunkCacheMode` and strict parsing in the pipeline package.
2. Add `WorkspaceChunkCacheConfig` with `Mode` and `Directory` to config and
file config. The YAML fields are `workspace.chunk_cache.mode` and
`workspace.chunk_cache.directory`.
3. Set default mode to `auto`; support
`NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`; preserve precedence file then
environment. CLI precedence arrives in Stage 5.
4. Support `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR` with environment-over-file
precedence. Trim the value; an unset or empty effective value selects the
per-user default. Do not add a CLI directory override.
5. Add `workspace.DefaultChunkPlanRoot`, accepting an injectable
user-cache-directory resolver and returning
`filepath.Join(userCacheDir, "notarius", "chunk-plans")`. Production later
supplies `os.UserCacheDir`; tests supply a stub. Reject resolver errors and
empty returned directories. Use this exact signature:
```go
func DefaultChunkPlanRoot(userCacheDir func() (string, error)) (string, error)
```
6. Treat a non-empty `workspace.chunk_cache.directory` as the plan root itself,
cleaned with the same path rules as other workspace roots. It does not gain
an implicit `chunk-plans` suffix. `workspace.directory` must not affect this
root, and the chunk-cache directory must not affect checkpoint, diagnostics,
debug, or output roots. Do not change `workspace.FromConfig` or
`workspace.Settings` for this feature.
7. Add the plan record, producer, decision, and store interfaces described
above in the pipeline package.
8. Add `internal/framework/chunkplan` with:
- a filesystem implementation;
- a constructor that accepts the already resolved chunk-plan root;
- path `<chunk-plan-root>/<64-character-source-sha256-hex>/plan.json`, after
strictly validating and stripping the logical `sha256:` prefix;
- schema `notarius.chunk-plan.v1`;
- directories mode `0700` and file mode `0600`;
- atomic same-directory temp-file rename;
- strict envelope decoding and unknown-field rejection;
- complete digest and annotation validation; and
- no raw reference content.
9. `Load` distinguishes missing, invalid, and operational error. `Save` always
atomically replaces and implements last-successful-write semantics.
10. Keep store fakes local to their tests; do not add a production in-memory or
no-op store because `bypass` and a nil collaborator already express
disabled persistence.
11. Do not wire the store into `Runner` or CLI execution in this stage.
### Tests
Cover:
- config defaults, YAML decoding, environment override, invalid values,
cloning, redaction, and validation;
- directory environment-over-file precedence, empty-value fallback, cloning,
redaction, cleaning, and validation;
- exact injected per-user default and resolver failure or empty-result errors;
- independence in both directions between `workspace.directory` and
`workspace.chunk_cache.directory`;
- exact acceptance of `/var/cache/notarius/chunk-plans` as a configured root
without writing to that real path;
- safe full-digest paths and rejection of malformed digests;
- `0700`/`0600` permissions where supported;
- envelope round trip, strict decoding, source and plan digest mismatch,
invalid annotations, schema mismatch, missing files, operational read errors,
atomic replacement, and concurrent writers; and
- absence of raw reference content in persisted JSON.
Run at minimum:
```sh
go test ./internal/core/config ./internal/core/workspace
go test ./internal/framework/chunkplan ./internal/framework/pipeline
go test ./internal/cli
```
### Exit criteria
Configuration and storage are complete and independently tested, but normal
runs still use Stage 2’s generate-and-materialize behavior.
## Stage 4: Integrate cache policy and remove chunk checkpoints
**Status:** Complete
### Objective
Make the runner’s chunk path use the canonical plan store under all three modes
and establish it as the only chunk-reuse mechanism.
### Read first
- `internal/framework/pipeline/runner.go`
- `internal/framework/pipeline/checkpoint.go`
- `internal/framework/pipeline/runner_concurrent.go`
- `internal/framework/checkpoint/loader.go`
- `internal/framework/checkpoint/recorder.go`
- focused runner retry, rejection, debug, and checkpoint tests
### Implement
1. Add `ChunkCacheMode` and `ChunkPlans` to `RunInput`. Empty mode means
`bypass` for direct framework callers.
2. Add synchronized store wrapping only if runner access can be concurrent;
otherwise document and test the intentionally serial chunk-plan access.
3. Implement mode behavior:
- `auto` hit: load, structurally validate against the current source,
materialize, then run configured chunk validators once;
- `auto` miss or invalid: generate through existing retries, validate,
materialize, validate chunks, then save;
- `bypass`: generate and validate normally without calling `Load` or `Save`;
- `refresh`: generate and validate normally, then replace through `Save`.
4. A valid-hit validator rejection is recorded without retry, store mutation,
or implicit regeneration. A store I/O error is a framework error.
5. A failed or rejected generated plan is never saved. An invalid old file is
overwritten only after a replacement passes module execution, structural
validation, materialization, and configured validators.
6. Build producer provenance from the prepared input module key, prepared chunk
module key, resolved chunk-stage LLM profile, chunk-target auxiliary
reference provenance, and chunk-module manifest metadata. Clone and redact
according to existing boundaries.
7. Replay stored producer warnings on a hit, then append current validation
warnings. Do not duplicate warnings from discarded generation attempts.
8. Remove all chunk methods and types from `CheckpointRecorder`,
`CheckpointLoader`, synchronized wrappers, workspace recorder/loader, and
checkpoint manifests and envelopes.
9. Remove runner chunk-checkpoint decisions and events. Retain source, extract,
merge, and normalize checkpoints. Keep downstream fingerprints based on the
digest of materialized chunks.
10. Ignore existing on-disk `chunk/` checkpoint files; do not read, delete, or
migrate them.
11. Update debug behavior:
- hit records lookup and materialization but no chunk module attempt or LLM
call;
- generated, bypassed, and refreshed paths retain attempt debug; and
- default debug envelopes remain redacted.
### Tests
Add table-driven runner tests for every mode and outcome:
- hit, missing, invalid, and I/O error;
- generation success, error, rejection, retry, and cancellation;
- validator warning, rejection, and error on hits and generated plans;
- bypass performs zero store calls;
- refresh performs no load and exactly one successful save;
- invalid record remains untouched when replacement fails;
- stored warnings replay once;
- no module or LLM call on hit;
- downstream chunk digest changes on refresh;
- legacy chunk checkpoint files are ignored; and
- source and lane checkpoint behavior remains intact.
Run at minimum:
```sh
go test ./internal/framework/pipeline
go test ./internal/framework/checkpoint
go test ./internal/modules/integration
```
### Exit criteria
The runner implements complete cache semantics, chunk checkpoints no longer
exist as a code path, all direct framework tests are green, and production CLI
runs are not yet wired to persistent plan storage.
## Stage 5: Wire persistent policy into the CLI
**Status:** Complete
### Objective
Make normal CLI runs resolve the effective mode and dedicated chunk-plan root,
construct the durable plan store when appropriate, and pass it into the runner.
### Read first
- `docs/cli.md`
- `docs/config.md`
- `cmd/notarius/main.go` and `internal/cli/run.go`
- CLI flag, environment, invocation-metadata, and exit-code tests
- the workspace and configuration code changed in Stage 3
### Implement
1. Add the exact flag `--chunk_cache <auto|bypass|refresh>`. Represent an
omitted CLI value separately from an explicit value so precedence is not
inferred from the default printed by the flag package.
2. Resolve the effective mode in this order: explicit CLI flag, environment,
configuration file, default `auto`. An invalid CLI flag is a usage error
with exit code 2; invalid file or environment configuration follows the
existing configuration-load failure path and exit code 1.
3. Include an explicitly requested CLI value in invocation metadata and the
resolved mode in effective-configuration metadata without adding field-level
origin tracking or plan and annotation payloads. Name the optional
invocation JSON field `chunk_cache_override`; the resolved mode already
appears at `workspace.chunk_cache.mode` in the redacted effective config.
4. Add `UserCacheDir func() (string, error)` and
`ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory` to `cli.Options`,
defaulting them to `os.UserCacheDir` and
`chunkplan.NewFilesystemStore`. For `auto` and `refresh`, choose the non-empty
effective `workspace.chunk_cache.directory`; otherwise resolve the per-user
default with `workspace.DefaultChunkPlanRoot`. Invoke the store factory with
that exact root and pass the result and effective mode through `RunInput`.
Never append `chunk-plans` to an explicit directory. The injectable factory
lets CLI tests assert root selection without writing to production paths.
5. For `bypass`, do not resolve, inspect, or create the chunk-plan root and pass
no store. Existing general-workspace setup remains unchanged and independent:
`workspace.directory`, resume, diagnostics, and debug continue to use their
current paths and semantics.
6. Keep resume policy independent of plan reuse. Enabling, disabling, or
selecting a resume run must not change the chunk-cache mode or path.
7. Configuration validation and pipeline-listing commands validate the mode but
do not create workspace directories or plan files.
8. Update CLI usage text only as needed to expose the new flag. Defer all
narrative documentation to Stage 8.
### Tests
Add CLI and integration tests for:
- a default `auto` run writing a plan and a second independent invocation
reusing it without a chunk operation or chunk LLM call;
- file, environment, and CLI precedence, including persistent `refresh` in
file and environment configuration;
- `bypass` performing no plan-store access and resolving or creating no default
chunk-plan root;
- invalid flag, environment, and file values with their established error
classifications;
- an explicit chunk-plan root from file and environment, including exact use of
`/var/cache/notarius/chunk-plans` without an appended suffix, using a
recording store factory and never writing to that real path;
- an explicit root succeeding without calling a failing per-user cache resolver;
- an injected per-user default resolving exactly to
`<user-cache-dir>/notarius/chunk-plans`;
- `workspace.directory` changes having no effect on chunk-plan placement and a
chunk-cache directory having no effect on other workspace state;
- per-user cache-root discovery failure under `auto`, `refresh`, and `bypass`,
including the allowed no-state `bypass` case, plus unwritable-store failures;
- resume plus each cache mode, showing independent behavior;
- validation and listing commands having no filesystem side effects; and
- reuse across changed pipeline, chunk module/options, references, validators,
and LLM settings when the source digest is unchanged.
Run at minimum:
```sh
go test ./internal/cli
go test ./internal/framework/pipeline ./internal/framework/chunkplan
go test ./internal/modules/integration
```
### Exit criteria
The production CLI implements the complete mode and precedence contract,
independent invocations reuse source-addressed plans by default, bypass and
refresh have exact state semantics, and the repository is green.
## Stage 6: Expose producer provenance and safe diagnostics
**Status:** Complete
### Objective
Make every output and diagnostic record distinguish the requested chunk module
from the producer of the effective plan without exposing sensitive payloads by
default.
### Read first
- `docs/integrations/json-output.md`
- `docs/internal/diagnostics.md`
- manifest assembly in `internal/framework/pipeline/runner.go`
- `internal/core/artifacts/artifacts.go`
- current diagnostics and debug-envelope implementations and tests
### Implement
1. Add `ChunkPlanManifest` to the run manifest and all required clone,
conversion, and serialization paths.
2. Populate `mode` and exactly one action after a candidate plan has passed
structural validation and materialization:
- `reused` for an `auto` hit;
- `generated` for an `auto` miss or invalid record;
- `refreshed` for generation under `refresh`; and
- `bypassed` for generation under `bypass`.
Retain that action if the current chunk-validator chain rejects the
candidate; publication still follows Stage 4's accept-only rule. If no
candidate reaches materialization, leave action and candidate plan fields
empty in the partial failure manifest and report the failure in diagnostics.
3. Preserve the existing top-level `chunker` and its module metadata as the
requested module. Never overwrite those fields with cached producer data.
4. Populate producer module, LLM profile, references, metadata, creation time,
source digest, plan digest, and schema from the effective stored or newly
generated record. A reused record reports its original producer, even when
it differs from the requested module.
5. Define producer LLM profile as the resolved chunk-stage profile used during
generation. Leave it empty for producers that did not use an LLM; do not
substitute the current run's requested profile on reuse.
6. Keep producer references as existing redacted provenance records. Deep-clone
metadata, references, warnings, annotations, and raw JSON at every ownership
boundary.
7. Add a redacted `chunk-plan.json` diagnostic artifact, backed by a summary on
`RunOutput` so the CLI can write it on success, rejection, or framework
failure. It contains mode, source and candidate plan digests, requested
module, lookup status and redacted reason, action, validation status, and
publication status. Use these closed values:
- lookup: `hit`, `missing`, `invalid`, or `skipped`;
- validation: `not_run`, `approved`, `approved_with_warnings`, `rejected`,
or `error`; and
- publication: `not_requested`, `published`, or `not_published`.
It contains no source units, materialized chunk content, annotation bytes,
reference content, prompts, model responses, or raw invalid-file content.
8. Keep complete plan, annotations, attempts, and materialized chunks confined
to existing opt-in debug artifacts and apply their normal restrictive file
handling.
9. Preserve JSON-output compatibility by adding fields rather than renaming
unrelated existing fields. Omit producer-only optional values when absent in
the same style as the surrounding manifest schema.
### Tests
Cover:
- all four actions and their exact manifest fields;
- requested and producer modules differing on a hit;
- producer metadata remaining stable when current configuration changes;
- LLM-backed and non-LLM producers;
- deep-clone and mutation safety for raw annotations and metadata;
- default diagnostic redaction, including invalid stored files;
- complete data appearing only under explicit debug configuration; and
- JSON serialization and compatibility fixtures.
Run at minimum:
```sh
go test ./internal/core/artifacts
go test ./internal/framework/pipeline
go test ./internal/cli
```
### Exit criteria
Manifests and diagnostics explain which plan was requested, selected, and
produced; default output remains redacted; debug behavior is explicit; and all
compatibility tests pass.
## Stage 7: Add cross-run, corruption, and concurrency hardening
**Status:** Complete
### Objective
Exercise the completed design across package boundaries and adversarial state,
then close any correctness gaps without changing the settled contract.
### Read first
- all tests added in Stages 1–6
- end-to-end CLI and module integration test harnesses
- atomic-write, cancellation, retry, and race-sensitive framework code
- repository import-boundary tests
### Implement
1. Add black-box tests using separate CLI application instances and a shared
temporary chunk-plan root. Prove reuse across different pipelines,
configured chunk modules, options, references, lanes, validators, and LLM
profiles.
2. Prove that a changed canonical source digest selects a different path, while
byte-identical canonical sources reuse the same plan.
3. Exercise generic plans through D&D pipelines and D&D-annotated plans through
generic pipelines. Optional annotations must survive, while no downstream
component may require `dnd/scenes` merely because the requested chunker is
the D&D scene module.
4. Add corruption cases for truncated JSON, unknown fields, schema mismatch,
source mismatch, plan-digest mismatch, bad boundaries, invalid annotations,
and materialization failure. Verify `auto` regeneration and non-destructive
failure behavior exactly.
5. Add concurrent reader/writer and writer/writer tests. Readers observe either
the old complete envelope or the new complete envelope; after successful
writers, the final file is one complete valid record. Do not assert which
writer wins.
6. Exercise interruption before save, during temp-file creation, and around
rename where the implementation permits deterministic fault injection.
Prior valid state must remain readable unless a complete replacement was
published.
7. Verify cancellation, retry limits, warning order, error classification, and
debug ordering on misses and hits. No cache hit may consume chunk-stage retry
budget or invoke the module.
8. Verify legacy chunk checkpoint files cannot influence reuse and that lane
resume still keys from effective materialized chunk digests after refresh.
9. Extend import-boundary tests so `internal/core/source` owns no framework or
module dependencies and the generic store owns no concrete module imports.
10. Ensure examples and all tests isolate workspace state in temporary
directories and isolate chunk-plan state with a temporary configured root
or explicitly select `bypass`; tests must never write to the real per-user
default or `/var/cache/notarius/chunk-plans`.
11. Run the race detector over the packages that access the plan store and
runner concurrently, and fix any race within feature scope.
### Tests
In addition to the new focused tests, run:
```sh
go test ./...
go test -race ./internal/framework/chunkplan ./internal/framework/pipeline
go vet ./...
go build ./cmd/notarius
```
### Exit criteria
The feature is demonstrated across independent runs, corruption and concurrent
publication cannot expose partial state, cross-domain annotations remain
optional, cancellation and retry behavior is deterministic, and all repository
and race checks pass.
## Stage 8: Publish current-behavior documentation
**Status:** Complete
### Objective
After implementation is complete, make the supported behavior discoverable
without duplicating canonical ownership across documents.
### Read first
- `docs/policy/architecture.md`
- `docs/policy/documentation.md`
- every current-behavior document linked below
- the final code, flags, schemas, examples, and tests from Stages 1–7
### Implement
1. Update `docs/policy/architecture.md` with the
plan-generation/materialization
boundary, source-addressed reuse, and the independent cache-state surface at
the level appropriate for the canonical system overview.
2. Update `docs/internal/pipeline.md` with mode behavior, hit/miss control flow,
validator placement, retry behavior, and removal of chunk checkpoints.
3. Update `docs/internal/modules.md` with the `Chunker.Plan` contract, generic
boundaries, optional annotation namespaces, and framework materialization.
4. Update `docs/internal/overview.md` with the new plan-store package and
current state ownership, and remove chunk checkpoint ownership from its
checkpoint description. Keep detailed checkpoint and plan execution flow in
`docs/internal/pipeline.md` rather than creating a parallel internal state
reference.
5. Update `docs/config.md` with `workspace.chunk_cache.mode`,
`workspace.chunk_cache.directory`, both environment variables, strict
values, precedence, and the per-user cache-root default. State explicitly
that `workspace.directory` does not affect plan placement and that the
configured directory is the root itself. Keep field-level details owned
here. Document that Unix `os.UserCacheDir` rejects a relative
`XDG_CACHE_HOME` and that this is a configuration error rather than a fallback.
6. Update `docs/cli.md` with `--chunk_cache` semantics and examples. Link to
configuration rather than restating its entire precedence model.
7. Update `docs/operations.md` with plan location, permissions, invalid-state
recovery, refresh and bypass operations, concurrency semantics, sensitive
debug data, deletion as a recoverable but potentially expensive
regeneration event, and the no-history/no-rollback limitation. Document both:
- the ordinary Linux per-user default,
`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is a valid
absolute path, or `$HOME/.cache/notarius/chunk-plans` when it is unset; and
- `/var/cache/notarius/chunk-plans` as the recommended configured root for a
system-wide Linux deployment running under a dedicated service account.
Explain that the system path is not the unprivileged default, show the
following minimal configuration, require the operator or package installer
to provision it with restrictive service-account ownership and permissions,
and warn against sharing the cache across mutually untrusted users:
```yaml
workspace:
chunk_cache:
directory: /var/cache/notarius/chunk-plans
```
8. Update `docs/integrations/json-output.md` and
`docs/internal/diagnostics.md` with requested versus producer provenance,
actions, warnings, and redaction boundaries.
9. Keep package-level plan/range/annotation and materialization invariants in
`docs/internal/pipeline.md`; no separate source document is needed unless
the implemented subsystem becomes too large for that canonical owner.
10. Update examples only when they materially clarify operation; examples must
use temporary or explicitly configured workspace and chunk-cache paths.
11. Remove future-tense disclaimers from the feature roadmap, mark its
completion outcomes as achieved, and mark all completed stages in this
document. Keep the roadmap as historical target-state context rather than
copying implementation details into it.
12. Validate every changed link, heading, code symbol, flag, environment name,
path, and JSON field against the implementation. Search for stale
`Chunker.Chunk`, `ChunkResult`, `chunks.scenes`, and chunk-checkpoint claims.
### Tests and validation ### Tests and validation
@@ -953,12 +569,12 @@ go build ./cmd/notarius
git diff --check git diff --check
``` ```
Also run repository link or documentation checks if present, and manually Run any repository documentation or link checker if present, and manually
inspect the reading map in `docs/development.md` and component links in inspect the reading map in [Development](../development.md) and component links
`docs/internal/overview.md` for canonical ownership and navigation consistency. in [Internal Overview](../internal/overview.md).
### Exit criteria ### Exit criteria
All canonical documents describe implemented behavior, operational guidance is The feature roadmap and canonical references describe implemented behavior,
complete and non-duplicative, no stale chunk API or checkpoint documentation this plan accurately records completion, documentation ownership remains clear,
remains, roadmap status is accurate, and the repository is green. all links and names are current, and the repository-wide validation is green.