Update implementation plan to incorporate follow-up fixes
This commit is contained in:
@@ -1,12 +1,14 @@
|
||||
# ADR-0005 Implementation Record
|
||||
|
||||
This document records the completed implementation of the target state in
|
||||
This document records the implementation of the target state in
|
||||
[ADR-0005 Feature Roadmap](adr0005.md), governed by
|
||||
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md).
|
||||
|
||||
All stages below are complete. The plan remains as historical target-state
|
||||
context; current behavior is documented in the canonical references linked from
|
||||
[Development](../development.md).
|
||||
Stages 1 through 8 summarize the completed initial implementation. Stages 9
|
||||
through 12 track pending remediation identified by the post-implementation
|
||||
review. Current behavior is documented in the canonical references linked from
|
||||
[Development](../development.md); those references must not describe the Stages
|
||||
9 through 12 target state until the corresponding stage is complete.
|
||||
|
||||
## Execution Rules
|
||||
|
||||
@@ -281,666 +283,280 @@ ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"`
|
||||
The runner initializes it with the effective mode and requested chunk module,
|
||||
then fills candidate and producer fields when a plan reaches materialization.
|
||||
|
||||
## Stage 1: Finalize ADR and add the source-zone plan model
|
||||
## Completed Stage Summaries
|
||||
|
||||
### Stage 1: Finalize ADR and add the source-zone plan model
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
Accepted ADR-0005 and added the domain-neutral chunk-plan, range, annotation,
|
||||
validation, digest, cloning, and deterministic materialization primitives in
|
||||
`internal/core/source`.
|
||||
|
||||
### Stage 2: Replace the chunk operation with plan generation
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
Replaced chunk-returning module operations with `Chunker.Plan`, converted the
|
||||
generic and D&D scene chunkers to produce ranges and optional annotations, and
|
||||
made the runner materialize and validate generated plans.
|
||||
|
||||
### Stage 3: Add dedicated cache configuration and the plan store
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
Added strict cache modes, independent per-user or explicitly configured
|
||||
chunk-plan roots, the source-addressed filesystem store, restrictive
|
||||
permissions, envelope validation, and atomic replacement.
|
||||
|
||||
### Stage 4: Integrate cache policy and remove chunk checkpoints
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
Implemented runner semantics for `auto`, `bypass`, and `refresh`; retained
|
||||
current chunk-validator behavior; made accepted canonical plans the sole chunk
|
||||
reuse mechanism; and removed chunk checkpoint loading and recording.
|
||||
|
||||
### Stage 5: Wire persistent policy into the CLI
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
Added CLI, environment, and file configuration precedence; per-user root
|
||||
resolution; persistent store construction; bypass isolation; and cross-run
|
||||
reuse through normal CLI execution.
|
||||
|
||||
### Stage 6: Expose producer provenance and safe diagnostics
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
Added requested-versus-producer chunk-plan provenance to manifests and a
|
||||
payload-free chunk-plan decision summary to normal diagnostics, with complete
|
||||
plan material confined to opt-in debug artifacts.
|
||||
|
||||
### Stage 7: Add cross-run, corruption, and concurrency hardening
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
Added integration, corruption, interruption, concurrency, compatibility,
|
||||
cross-domain annotation, import-boundary, and race coverage for the implemented
|
||||
cache design.
|
||||
|
||||
### Stage 8: Publish current-behavior documentation
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
Updated architecture, pipeline, module, configuration, CLI, operations,
|
||||
diagnostics, integration, and overview documentation for the implemented
|
||||
feature and its per-user and system-wide deployment guidance.
|
||||
|
||||
## Stage 9: Redact invalid cache-record diagnostics
|
||||
|
||||
**Status:** Pending
|
||||
|
||||
### Objective
|
||||
|
||||
Accept the architectural decision and introduce domain-neutral plan,
|
||||
annotation, validation, digest, clone, and materialization primitives without
|
||||
changing chunk-module execution yet.
|
||||
Ensure malformed or incompatible cache contents cannot enter normal errors,
|
||||
warnings, manifests, or diagnostics while retaining useful closed lookup
|
||||
decisions.
|
||||
|
||||
### Read first
|
||||
|
||||
- [Architecture Policy](../policy/architecture.md), especially secret handling
|
||||
- [Diagnostics Internals](../internal/diagnostics.md)
|
||||
- `internal/framework/chunkplan/store.go` and its focused tests
|
||||
- `internal/framework/pipeline/runner_chunk_plan.go` and its focused tests
|
||||
- `internal/core/artifacts/artifacts.go`
|
||||
- the CLI path that writes `chunk-plan.json`
|
||||
|
||||
### Implement
|
||||
|
||||
1. Stop interpolating decoder errors, unknown field names, schema values,
|
||||
annotation namespaces, timestamps, paths, or any other stored content into
|
||||
recoverable `ChunkPlanDecision.Reason` values.
|
||||
2. At the normal-diagnostic boundary, ignore the reason supplied by a
|
||||
`ChunkPlanStore` and derive `ChunkPlanSummary.LookupReason` solely from
|
||||
status using exactly:
|
||||
- `hit`: `stored chunk plan is valid`;
|
||||
- `missing`: `chunk plan not found`;
|
||||
- `invalid`: `stored chunk plan is invalid`; and
|
||||
- `skipped`: `chunk plan lookup skipped`.
|
||||
3. Keep operational read failures as framework errors rather than recoverable
|
||||
invalid-record decisions. Their normal error text must identify the failed
|
||||
operation without including cache-file content.
|
||||
4. Do not add raw invalid-record details to another default surface. Direct
|
||||
operator inspection or a future explicitly sensitive debug facility may
|
||||
expose them, but this stage does not need to add such a facility.
|
||||
|
||||
### Tests
|
||||
|
||||
Add focused tests proving that sentinel secrets placed in unknown field names,
|
||||
schema values, annotation namespaces, timestamps, and malformed trailing JSON
|
||||
appear in neither the store decision nor `chunk-plan.json`. Add a runner test
|
||||
with a custom store whose reason contains a sentinel and prove that the summary
|
||||
uses the fixed status-derived reason. Preserve useful missing, hit, invalid, and
|
||||
operational-error classification.
|
||||
|
||||
Run focused tests for `internal/framework/chunkplan`,
|
||||
`internal/framework/pipeline`, and `internal/cli`, then perform the
|
||||
repository-wide validation in the execution rules.
|
||||
|
||||
### Exit criteria
|
||||
|
||||
Every normal lookup reason belongs to the fixed safe vocabulary, hostile
|
||||
invalid-file content cannot reach default diagnostics, operational failures
|
||||
remain correctly classified, and all focused and repository-wide checks pass.
|
||||
|
||||
## Stage 10: Enforce rooted chunk-plan filesystem access
|
||||
|
||||
**Status:** Pending
|
||||
|
||||
### Objective
|
||||
|
||||
Make the cache's path-confinement guarantee resistant to symlinked cache entries
|
||||
while preserving atomicity, permissions, and concurrent access semantics.
|
||||
|
||||
### Read first
|
||||
|
||||
- [Architecture Policy](../policy/architecture.md), especially state and path
|
||||
invariants
|
||||
- [Operations](../operations.md)
|
||||
- `internal/framework/chunkplan/store.go` and all store tests
|
||||
- Go's `os.Root` API for the repository's declared Go version
|
||||
|
||||
### Implement
|
||||
|
||||
1. Treat the configured cache root as the filesystem trust boundary. A symlink
|
||||
in the operator-supplied root path itself may be followed when the root is
|
||||
opened, but every store-owned operation after that point must remain beneath
|
||||
the opened root.
|
||||
2. Perform digest-directory, plan-file, temporary-file, and rename operations
|
||||
through `os.Root` or an equivalently race-resistant directory-relative
|
||||
mechanism. Do not rely on `filepath.Join` plus a preflight `Lstat` as the
|
||||
confinement boundary.
|
||||
3. Reject symlinks and unexpected file types at the store-owned digest-directory
|
||||
and `plan.json` positions. A missing cache root on `Load` remains a normal
|
||||
miss and is not created; `Save` creates the root and digest directory as
|
||||
needed.
|
||||
4. Preserve the exact
|
||||
`<root>/<full-source-sha256-hex>/plan.json` layout, `0700` directories,
|
||||
`0600` plan files, strict digest validation, and no implicit path suffix.
|
||||
5. Publish using an exclusively created random temporary file inside the digest
|
||||
directory, sync and close it, then atomically rename it through the same
|
||||
rooted handle. Remove temporary files on every pre-rename failure. Preserve
|
||||
last-successful-write and complete-reader semantics.
|
||||
6. Keep store access serial at the runner boundary and safe for concurrent
|
||||
independent store callers. Do not introduce a lock, history, or rollback
|
||||
protocol.
|
||||
|
||||
### Tests
|
||||
|
||||
Add load and save cases for digest-directory and plan-file symlinks that resolve
|
||||
outside the configured root. Prove that they are rejected and that no outside
|
||||
file is read, created, chmodded, or replaced. Include unexpected file types and
|
||||
failed-publication preservation.
|
||||
|
||||
Retain and run tests for absent-root behavior, exact paths, permissions, round
|
||||
trip, malformed digests, atomic replacement, interrupted writes, temporary-file
|
||||
cleanup, concurrent readers, and concurrent writers. Run the race detector for
|
||||
`internal/framework/chunkplan`, then perform the repository-wide validation in
|
||||
the execution rules.
|
||||
|
||||
### Exit criteria
|
||||
|
||||
No store-owned operation can escape the opened root through a cache entry,
|
||||
normal layout and permissions remain unchanged, publication remains atomic,
|
||||
concurrency tests and the race detector pass, and the repository is green.
|
||||
|
||||
## Stage 11: Deep-clone source metadata during materialization
|
||||
|
||||
**Status:** Pending
|
||||
|
||||
### Objective
|
||||
|
||||
Ensure source documents, materialized chunks, and separate materializations
|
||||
never share mutable metadata storage, regardless of the concrete JSON-shaped
|
||||
map, slice, array, interface, raw-message, or byte-slice types used by an input
|
||||
adapter.
|
||||
|
||||
### Read first
|
||||
|
||||
- `internal/core/source/source.go`
|
||||
- `internal/core/source/digest.go`
|
||||
- `internal/core/source/validation.go`
|
||||
- `internal/core/source/source_test.go`
|
||||
- every current `source.Chunk` clone or serialization helper found with `rg`
|
||||
- `internal/core/source/chunk_plan.go` and its focused tests
|
||||
- `internal/core/source/digest.go` and validation behavior
|
||||
- every metadata clone helper and ownership boundary found with `rg`
|
||||
- focused runner tests that hand chunks to validators and extractors
|
||||
|
||||
### Implement
|
||||
|
||||
1. Revise ADR-0005 before acceptance:
|
||||
- confirm it retains one source-addressed mutable plan and atomic refresh;
|
||||
- preserve the independent cache-surface decision;
|
||||
- keep filesystem and CLI details out of the ADR; and
|
||||
- change status from `Proposed` to `Accepted`.
|
||||
2. Add `ChunkAnnotations`, `ChunkPlan`, and `ChunkRange` in
|
||||
`internal/core/source`.
|
||||
3. Add canonical annotation validation and cloning. Canonicalize each raw JSON
|
||||
value by decoding with `json.Decoder.UseNumber` and re-encoding; reject
|
||||
blank namespaces, invalid JSON, trailing JSON values, and non-finite or
|
||||
unsupported values.
|
||||
4. Add plan validation against a `SourceDocument` using the rules in Planned
|
||||
Contracts.
|
||||
5. Add `DigestChunkPlan` and deterministic `MaterializeChunkPlan`.
|
||||
6. Add `Annotations` and `PlanAnnotations` to `source.Chunk`, `DigestChunk`,
|
||||
every clone helper, debug envelope, and serialized chunk-validation
|
||||
representation. Ensure all ownership boundaries clone raw bytes and retain
|
||||
the two scopes separately.
|
||||
7. Do not change the chunker interface or runner flow in this stage.
|
||||
1. Replace the concrete-type allowlist used by plan materialization with one
|
||||
reusable source-owned deep-clone implementation for supported
|
||||
JSON-serializable metadata.
|
||||
2. Recursively clone string-keyed maps, slices, arrays, interfaces,
|
||||
`json.RawMessage`, and byte slices without sharing mutable backing storage.
|
||||
Preserve scalar values and the logical JSON representation. Typed composite
|
||||
containers must not fall through as shared values.
|
||||
3. Detect cyclic or unsupported values and return a contextual error from
|
||||
materialization rather than retaining an alias, panicking, or recursing
|
||||
indefinitely.
|
||||
4. Use the source-owned helper at pipeline ownership boundaries where doing so
|
||||
removes a duplicate incomplete implementation. Do not introduce a dependency
|
||||
from `internal/core/source` to a framework package and do not expand this
|
||||
stage into unrelated metadata-schema redesign.
|
||||
5. Preserve deterministic materialized content, chunk digests, annotations, IDs,
|
||||
references, and metadata fields for all currently supported production
|
||||
inputs.
|
||||
|
||||
### Tests
|
||||
|
||||
Add focused tests for:
|
||||
Cover typed nested maps, slices of typed maps, arrays, interfaces,
|
||||
`json.RawMessage`, and byte slices. Mutate each materialized value and prove
|
||||
that neither the source document nor a separately materialized chunk changes.
|
||||
Add cyclic and unsupported-value cases that fail deterministically with
|
||||
contextual errors. Retain the existing exact-materialization and digest tests.
|
||||
|
||||
- annotation canonicalization, invalid JSON, blank namespaces, and mutation
|
||||
safety;
|
||||
- plan source mismatch, missing ranges, missing units, backward ranges,
|
||||
duplicate starts, gaps, and overlap;
|
||||
- stable plan digests and digest changes for boundary or annotation changes;
|
||||
- exact materialized IDs, references, content bytes, metadata, annotations,
|
||||
gaps, and overlap;
|
||||
- repeated byte-identical materialization; and
|
||||
- plan- and range-annotation preservation through existing chunk clones, debug
|
||||
conversion, serialized validation, and downstream extract requests.
|
||||
|
||||
Run at minimum:
|
||||
|
||||
```sh
|
||||
go test ./internal/core/source
|
||||
go test ./internal/framework/contracts ./internal/framework/pipeline
|
||||
```
|
||||
Run focused tests for `internal/core/source`,
|
||||
`internal/framework/pipeline`, and affected module packages, then perform the
|
||||
repository-wide validation in the execution rules.
|
||||
|
||||
### Exit criteria
|
||||
|
||||
The repository is green; ADR-0005 is accepted; plan primitives are complete and
|
||||
tested; current modules and the runner still behave as before.
|
||||
Every supported mutable metadata value has independent ownership across source,
|
||||
chunk, validation, and extraction boundaries; invalid metadata fails safely;
|
||||
current materialized bytes remain stable; and all checks pass.
|
||||
|
||||
## Stage 2: Replace the chunk operation with plan generation
|
||||
## Stage 12: Finalize remediation documentation and roadmap status
|
||||
|
||||
**Status:** Complete
|
||||
**Status:** Pending
|
||||
|
||||
### Objective
|
||||
|
||||
Make all chunk modules produce plans and make the runner materialize those plans
|
||||
without adding durable reuse yet.
|
||||
Make the roadmap and canonical documentation accurately describe the completed
|
||||
feature after Stages 9 through 11 land.
|
||||
|
||||
### Read first
|
||||
|
||||
- `docs/internal/modules.md`
|
||||
- `docs/internal/pipeline.md`
|
||||
- `internal/framework/contracts/contracts.go`
|
||||
- `internal/framework/pipeline/chunker_registry.go`
|
||||
- `internal/framework/pipeline/runner.go`
|
||||
- `internal/framework/pipeline/chunk_validation.go`
|
||||
- both production chunk-module packages and all chunk fakes in tests
|
||||
- [Documentation Policy](../policy/documentation.md)
|
||||
- [ADR-0005 Feature Roadmap](adr0005.md)
|
||||
- [Architecture Policy](../policy/architecture.md)
|
||||
- [Diagnostics Internals](../internal/diagnostics.md)
|
||||
- [Operations](../operations.md)
|
||||
- the final code and tests from Stages 9 through 11
|
||||
|
||||
### Implement
|
||||
|
||||
1. Replace `ChunkResult` with `ChunkPlanResult` and `Chunker.Chunk` with
|
||||
`Chunker.Plan`; keep the request, registry, module keys, and stage name.
|
||||
2. Update all fakes, registry tests, preparation tests, integration helpers, and
|
||||
compile-time interface assertions in the same stage. Do not retain adapters
|
||||
or a legacy chunk-returning interface.
|
||||
3. Convert `generic/chunk/units` to emit ranges only. Preserve `max_units` and
|
||||
`overlap_units` semantics.
|
||||
4. Convert `dnd/chunk/scenes` to emit namespace `dnd/scenes` at both scopes:
|
||||
- each range value is an object with exact keys `short_title`,
|
||||
`primary_mode`, `main_participants`, `summary`, `boundary_note`, and
|
||||
`boundary_confidence`, preserving the current normalized scene values;
|
||||
and
|
||||
- the plan value is an object with exact key `boundary_caveats`, containing
|
||||
the normalized caveat array.
|
||||
5. Preserve accepted scene boundary caveats as `contracts.Warning` values as
|
||||
well as in the plan annotation so warning behavior remains visible.
|
||||
6. Remove `chunks.scenes` from the D&D scene module's provided capabilities.
|
||||
Keep only `chunks`.
|
||||
7. Change the runner’s non-cached chunk path to:
|
||||
- call `Plan` under the existing retry and debug attempt boundary;
|
||||
- validate the plan structurally;
|
||||
- materialize chunks;
|
||||
- run the configured chunk-validator chain on materialized chunks;
|
||||
- preserve retry and rejection semantics; and
|
||||
- pass materialized chunks to lanes exactly as before.
|
||||
8. Update chunk debug payloads to distinguish the generated plan from
|
||||
materialized chunks. Plan annotations may appear only in opt-in debug data.
|
||||
9. Leave current chunk checkpoint loading and recording temporarily in place,
|
||||
storing materialized chunks, so existing resume tests remain green until
|
||||
Stage 4 removes that path.
|
||||
|
||||
### Tests
|
||||
|
||||
Update focused module and framework tests to cover:
|
||||
|
||||
- exact generic ranges and overlap;
|
||||
- exact D&D scene annotations and warning conversion;
|
||||
- annotations absent from generic plans;
|
||||
- framework materialization rather than module materialization;
|
||||
- chunk validators receiving materialized annotated chunks;
|
||||
- retries around generation plus validation;
|
||||
- stable chunk IDs independent of producing module; and
|
||||
- no `chunks.scenes` resolution dependency.
|
||||
|
||||
Run at minimum:
|
||||
|
||||
```sh
|
||||
go test ./internal/modules/generic/chunk/units
|
||||
go test ./internal/modules/dnd/chunk/scenes
|
||||
go test ./internal/framework/contracts ./internal/framework/pipeline
|
||||
go test ./internal/modules/integration ./internal/cli
|
||||
```
|
||||
|
||||
### Exit criteria
|
||||
|
||||
Every chunk module uses the single plan contract, the runner always
|
||||
materializes, production behavior remains functionally equivalent apart from
|
||||
framework-owned stable chunk IDs and namespaced annotations, and the repository
|
||||
is green.
|
||||
|
||||
## Stage 3: Add dedicated cache configuration and the plan store
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
### Objective
|
||||
|
||||
Implement the durable source-addressed store and its independent cache-root
|
||||
configuration without changing the runner to consume the store.
|
||||
|
||||
### Read first
|
||||
|
||||
- `docs/config.md`
|
||||
- `docs/operations.md`
|
||||
- `internal/core/config/*`
|
||||
- `internal/core/workspace/*`
|
||||
- `internal/framework/checkpoint/` as the local pattern for a workspace-backed
|
||||
implementation
|
||||
- `internal/cli/run.go` configuration loading, without wiring the store yet
|
||||
|
||||
### Implement
|
||||
|
||||
1. Add `ChunkCacheMode` and strict parsing in the pipeline package.
|
||||
2. Add `WorkspaceChunkCacheConfig` with `Mode` and `Directory` to config and
|
||||
file config. The YAML fields are `workspace.chunk_cache.mode` and
|
||||
`workspace.chunk_cache.directory`.
|
||||
3. Set default mode to `auto`; support
|
||||
`NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`; preserve precedence file then
|
||||
environment. CLI precedence arrives in Stage 5.
|
||||
4. Support `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR` with environment-over-file
|
||||
precedence. Trim the value; an unset or empty effective value selects the
|
||||
per-user default. Do not add a CLI directory override.
|
||||
5. Add `workspace.DefaultChunkPlanRoot`, accepting an injectable
|
||||
user-cache-directory resolver and returning
|
||||
`filepath.Join(userCacheDir, "notarius", "chunk-plans")`. Production later
|
||||
supplies `os.UserCacheDir`; tests supply a stub. Reject resolver errors and
|
||||
empty returned directories. Use this exact signature:
|
||||
|
||||
```go
|
||||
func DefaultChunkPlanRoot(userCacheDir func() (string, error)) (string, error)
|
||||
```
|
||||
6. Treat a non-empty `workspace.chunk_cache.directory` as the plan root itself,
|
||||
cleaned with the same path rules as other workspace roots. It does not gain
|
||||
an implicit `chunk-plans` suffix. `workspace.directory` must not affect this
|
||||
root, and the chunk-cache directory must not affect checkpoint, diagnostics,
|
||||
debug, or output roots. Do not change `workspace.FromConfig` or
|
||||
`workspace.Settings` for this feature.
|
||||
7. Add the plan record, producer, decision, and store interfaces described
|
||||
above in the pipeline package.
|
||||
8. Add `internal/framework/chunkplan` with:
|
||||
- a filesystem implementation;
|
||||
- a constructor that accepts the already resolved chunk-plan root;
|
||||
- path `<chunk-plan-root>/<64-character-source-sha256-hex>/plan.json`, after
|
||||
strictly validating and stripping the logical `sha256:` prefix;
|
||||
- schema `notarius.chunk-plan.v1`;
|
||||
- directories mode `0700` and file mode `0600`;
|
||||
- atomic same-directory temp-file rename;
|
||||
- strict envelope decoding and unknown-field rejection;
|
||||
- complete digest and annotation validation; and
|
||||
- no raw reference content.
|
||||
9. `Load` distinguishes missing, invalid, and operational error. `Save` always
|
||||
atomically replaces and implements last-successful-write semantics.
|
||||
10. Keep store fakes local to their tests; do not add a production in-memory or
|
||||
no-op store because `bypass` and a nil collaborator already express
|
||||
disabled persistence.
|
||||
11. Do not wire the store into `Runner` or CLI execution in this stage.
|
||||
|
||||
### Tests
|
||||
|
||||
Cover:
|
||||
|
||||
- config defaults, YAML decoding, environment override, invalid values,
|
||||
cloning, redaction, and validation;
|
||||
- directory environment-over-file precedence, empty-value fallback, cloning,
|
||||
redaction, cleaning, and validation;
|
||||
- exact injected per-user default and resolver failure or empty-result errors;
|
||||
- independence in both directions between `workspace.directory` and
|
||||
`workspace.chunk_cache.directory`;
|
||||
- exact acceptance of `/var/cache/notarius/chunk-plans` as a configured root
|
||||
without writing to that real path;
|
||||
- safe full-digest paths and rejection of malformed digests;
|
||||
- `0700`/`0600` permissions where supported;
|
||||
- envelope round trip, strict decoding, source and plan digest mismatch,
|
||||
invalid annotations, schema mismatch, missing files, operational read errors,
|
||||
atomic replacement, and concurrent writers; and
|
||||
- absence of raw reference content in persisted JSON.
|
||||
|
||||
Run at minimum:
|
||||
|
||||
```sh
|
||||
go test ./internal/core/config ./internal/core/workspace
|
||||
go test ./internal/framework/chunkplan ./internal/framework/pipeline
|
||||
go test ./internal/cli
|
||||
```
|
||||
|
||||
### Exit criteria
|
||||
|
||||
Configuration and storage are complete and independently tested, but normal
|
||||
runs still use Stage 2’s generate-and-materialize behavior.
|
||||
|
||||
## Stage 4: Integrate cache policy and remove chunk checkpoints
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
### Objective
|
||||
|
||||
Make the runner’s chunk path use the canonical plan store under all three modes
|
||||
and establish it as the only chunk-reuse mechanism.
|
||||
|
||||
### Read first
|
||||
|
||||
- `internal/framework/pipeline/runner.go`
|
||||
- `internal/framework/pipeline/checkpoint.go`
|
||||
- `internal/framework/pipeline/runner_concurrent.go`
|
||||
- `internal/framework/checkpoint/loader.go`
|
||||
- `internal/framework/checkpoint/recorder.go`
|
||||
- focused runner retry, rejection, debug, and checkpoint tests
|
||||
|
||||
### Implement
|
||||
|
||||
1. Add `ChunkCacheMode` and `ChunkPlans` to `RunInput`. Empty mode means
|
||||
`bypass` for direct framework callers.
|
||||
2. Add synchronized store wrapping only if runner access can be concurrent;
|
||||
otherwise document and test the intentionally serial chunk-plan access.
|
||||
3. Implement mode behavior:
|
||||
- `auto` hit: load, structurally validate against the current source,
|
||||
materialize, then run configured chunk validators once;
|
||||
- `auto` miss or invalid: generate through existing retries, validate,
|
||||
materialize, validate chunks, then save;
|
||||
- `bypass`: generate and validate normally without calling `Load` or `Save`;
|
||||
- `refresh`: generate and validate normally, then replace through `Save`.
|
||||
4. A valid-hit validator rejection is recorded without retry, store mutation,
|
||||
or implicit regeneration. A store I/O error is a framework error.
|
||||
5. A failed or rejected generated plan is never saved. An invalid old file is
|
||||
overwritten only after a replacement passes module execution, structural
|
||||
validation, materialization, and configured validators.
|
||||
6. Build producer provenance from the prepared input module key, prepared chunk
|
||||
module key, resolved chunk-stage LLM profile, chunk-target auxiliary
|
||||
reference provenance, and chunk-module manifest metadata. Clone and redact
|
||||
according to existing boundaries.
|
||||
7. Replay stored producer warnings on a hit, then append current validation
|
||||
warnings. Do not duplicate warnings from discarded generation attempts.
|
||||
8. Remove all chunk methods and types from `CheckpointRecorder`,
|
||||
`CheckpointLoader`, synchronized wrappers, workspace recorder/loader, and
|
||||
checkpoint manifests and envelopes.
|
||||
9. Remove runner chunk-checkpoint decisions and events. Retain source, extract,
|
||||
merge, and normalize checkpoints. Keep downstream fingerprints based on the
|
||||
digest of materialized chunks.
|
||||
10. Ignore existing on-disk `chunk/` checkpoint files; do not read, delete, or
|
||||
migrate them.
|
||||
11. Update debug behavior:
|
||||
- hit records lookup and materialization but no chunk module attempt or LLM
|
||||
call;
|
||||
- generated, bypassed, and refreshed paths retain attempt debug; and
|
||||
- default debug envelopes remain redacted.
|
||||
|
||||
### Tests
|
||||
|
||||
Add table-driven runner tests for every mode and outcome:
|
||||
|
||||
- hit, missing, invalid, and I/O error;
|
||||
- generation success, error, rejection, retry, and cancellation;
|
||||
- validator warning, rejection, and error on hits and generated plans;
|
||||
- bypass performs zero store calls;
|
||||
- refresh performs no load and exactly one successful save;
|
||||
- invalid record remains untouched when replacement fails;
|
||||
- stored warnings replay once;
|
||||
- no module or LLM call on hit;
|
||||
- downstream chunk digest changes on refresh;
|
||||
- legacy chunk checkpoint files are ignored; and
|
||||
- source and lane checkpoint behavior remains intact.
|
||||
|
||||
Run at minimum:
|
||||
|
||||
```sh
|
||||
go test ./internal/framework/pipeline
|
||||
go test ./internal/framework/checkpoint
|
||||
go test ./internal/modules/integration
|
||||
```
|
||||
|
||||
### Exit criteria
|
||||
|
||||
The runner implements complete cache semantics, chunk checkpoints no longer
|
||||
exist as a code path, all direct framework tests are green, and production CLI
|
||||
runs are not yet wired to persistent plan storage.
|
||||
|
||||
## Stage 5: Wire persistent policy into the CLI
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
### Objective
|
||||
|
||||
Make normal CLI runs resolve the effective mode and dedicated chunk-plan root,
|
||||
construct the durable plan store when appropriate, and pass it into the runner.
|
||||
|
||||
### Read first
|
||||
|
||||
- `docs/cli.md`
|
||||
- `docs/config.md`
|
||||
- `cmd/notarius/main.go` and `internal/cli/run.go`
|
||||
- CLI flag, environment, invocation-metadata, and exit-code tests
|
||||
- the workspace and configuration code changed in Stage 3
|
||||
|
||||
### Implement
|
||||
|
||||
1. Add the exact flag `--chunk_cache <auto|bypass|refresh>`. Represent an
|
||||
omitted CLI value separately from an explicit value so precedence is not
|
||||
inferred from the default printed by the flag package.
|
||||
2. Resolve the effective mode in this order: explicit CLI flag, environment,
|
||||
configuration file, default `auto`. An invalid CLI flag is a usage error
|
||||
with exit code 2; invalid file or environment configuration follows the
|
||||
existing configuration-load failure path and exit code 1.
|
||||
3. Include an explicitly requested CLI value in invocation metadata and the
|
||||
resolved mode in effective-configuration metadata without adding field-level
|
||||
origin tracking or plan and annotation payloads. Name the optional
|
||||
invocation JSON field `chunk_cache_override`; the resolved mode already
|
||||
appears at `workspace.chunk_cache.mode` in the redacted effective config.
|
||||
4. Add `UserCacheDir func() (string, error)` and
|
||||
`ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory` to `cli.Options`,
|
||||
defaulting them to `os.UserCacheDir` and
|
||||
`chunkplan.NewFilesystemStore`. For `auto` and `refresh`, choose the non-empty
|
||||
effective `workspace.chunk_cache.directory`; otherwise resolve the per-user
|
||||
default with `workspace.DefaultChunkPlanRoot`. Invoke the store factory with
|
||||
that exact root and pass the result and effective mode through `RunInput`.
|
||||
Never append `chunk-plans` to an explicit directory. The injectable factory
|
||||
lets CLI tests assert root selection without writing to production paths.
|
||||
5. For `bypass`, do not resolve, inspect, or create the chunk-plan root and pass
|
||||
no store. Existing general-workspace setup remains unchanged and independent:
|
||||
`workspace.directory`, resume, diagnostics, and debug continue to use their
|
||||
current paths and semantics.
|
||||
6. Keep resume policy independent of plan reuse. Enabling, disabling, or
|
||||
selecting a resume run must not change the chunk-cache mode or path.
|
||||
7. Configuration validation and pipeline-listing commands validate the mode but
|
||||
do not create workspace directories or plan files.
|
||||
8. Update CLI usage text only as needed to expose the new flag. Defer all
|
||||
narrative documentation to Stage 8.
|
||||
|
||||
### Tests
|
||||
|
||||
Add CLI and integration tests for:
|
||||
|
||||
- a default `auto` run writing a plan and a second independent invocation
|
||||
reusing it without a chunk operation or chunk LLM call;
|
||||
- file, environment, and CLI precedence, including persistent `refresh` in
|
||||
file and environment configuration;
|
||||
- `bypass` performing no plan-store access and resolving or creating no default
|
||||
chunk-plan root;
|
||||
- invalid flag, environment, and file values with their established error
|
||||
classifications;
|
||||
- an explicit chunk-plan root from file and environment, including exact use of
|
||||
`/var/cache/notarius/chunk-plans` without an appended suffix, using a
|
||||
recording store factory and never writing to that real path;
|
||||
- an explicit root succeeding without calling a failing per-user cache resolver;
|
||||
- an injected per-user default resolving exactly to
|
||||
`<user-cache-dir>/notarius/chunk-plans`;
|
||||
- `workspace.directory` changes having no effect on chunk-plan placement and a
|
||||
chunk-cache directory having no effect on other workspace state;
|
||||
- per-user cache-root discovery failure under `auto`, `refresh`, and `bypass`,
|
||||
including the allowed no-state `bypass` case, plus unwritable-store failures;
|
||||
- resume plus each cache mode, showing independent behavior;
|
||||
- validation and listing commands having no filesystem side effects; and
|
||||
- reuse across changed pipeline, chunk module/options, references, validators,
|
||||
and LLM settings when the source digest is unchanged.
|
||||
|
||||
Run at minimum:
|
||||
|
||||
```sh
|
||||
go test ./internal/cli
|
||||
go test ./internal/framework/pipeline ./internal/framework/chunkplan
|
||||
go test ./internal/modules/integration
|
||||
```
|
||||
|
||||
### Exit criteria
|
||||
|
||||
The production CLI implements the complete mode and precedence contract,
|
||||
independent invocations reuse source-addressed plans by default, bypass and
|
||||
refresh have exact state semantics, and the repository is green.
|
||||
|
||||
## Stage 6: Expose producer provenance and safe diagnostics
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
### Objective
|
||||
|
||||
Make every output and diagnostic record distinguish the requested chunk module
|
||||
from the producer of the effective plan without exposing sensitive payloads by
|
||||
default.
|
||||
|
||||
### Read first
|
||||
|
||||
- `docs/integrations/json-output.md`
|
||||
- `docs/internal/diagnostics.md`
|
||||
- manifest assembly in `internal/framework/pipeline/runner.go`
|
||||
- `internal/core/artifacts/artifacts.go`
|
||||
- current diagnostics and debug-envelope implementations and tests
|
||||
|
||||
### Implement
|
||||
|
||||
1. Add `ChunkPlanManifest` to the run manifest and all required clone,
|
||||
conversion, and serialization paths.
|
||||
2. Populate `mode` and exactly one action after a candidate plan has passed
|
||||
structural validation and materialization:
|
||||
- `reused` for an `auto` hit;
|
||||
- `generated` for an `auto` miss or invalid record;
|
||||
- `refreshed` for generation under `refresh`; and
|
||||
- `bypassed` for generation under `bypass`.
|
||||
Retain that action if the current chunk-validator chain rejects the
|
||||
candidate; publication still follows Stage 4's accept-only rule. If no
|
||||
candidate reaches materialization, leave action and candidate plan fields
|
||||
empty in the partial failure manifest and report the failure in diagnostics.
|
||||
3. Preserve the existing top-level `chunker` and its module metadata as the
|
||||
requested module. Never overwrite those fields with cached producer data.
|
||||
4. Populate producer module, LLM profile, references, metadata, creation time,
|
||||
source digest, plan digest, and schema from the effective stored or newly
|
||||
generated record. A reused record reports its original producer, even when
|
||||
it differs from the requested module.
|
||||
5. Define producer LLM profile as the resolved chunk-stage profile used during
|
||||
generation. Leave it empty for producers that did not use an LLM; do not
|
||||
substitute the current run's requested profile on reuse.
|
||||
6. Keep producer references as existing redacted provenance records. Deep-clone
|
||||
metadata, references, warnings, annotations, and raw JSON at every ownership
|
||||
boundary.
|
||||
7. Add a redacted `chunk-plan.json` diagnostic artifact, backed by a summary on
|
||||
`RunOutput` so the CLI can write it on success, rejection, or framework
|
||||
failure. It contains mode, source and candidate plan digests, requested
|
||||
module, lookup status and redacted reason, action, validation status, and
|
||||
publication status. Use these closed values:
|
||||
- lookup: `hit`, `missing`, `invalid`, or `skipped`;
|
||||
- validation: `not_run`, `approved`, `approved_with_warnings`, `rejected`,
|
||||
or `error`; and
|
||||
- publication: `not_requested`, `published`, or `not_published`.
|
||||
It contains no source units, materialized chunk content, annotation bytes,
|
||||
reference content, prompts, model responses, or raw invalid-file content.
|
||||
8. Keep complete plan, annotations, attempts, and materialized chunks confined
|
||||
to existing opt-in debug artifacts and apply their normal restrictive file
|
||||
handling.
|
||||
9. Preserve JSON-output compatibility by adding fields rather than renaming
|
||||
unrelated existing fields. Omit producer-only optional values when absent in
|
||||
the same style as the surrounding manifest schema.
|
||||
|
||||
### Tests
|
||||
|
||||
Cover:
|
||||
|
||||
- all four actions and their exact manifest fields;
|
||||
- requested and producer modules differing on a hit;
|
||||
- producer metadata remaining stable when current configuration changes;
|
||||
- LLM-backed and non-LLM producers;
|
||||
- deep-clone and mutation safety for raw annotations and metadata;
|
||||
- default diagnostic redaction, including invalid stored files;
|
||||
- complete data appearing only under explicit debug configuration; and
|
||||
- JSON serialization and compatibility fixtures.
|
||||
|
||||
Run at minimum:
|
||||
|
||||
```sh
|
||||
go test ./internal/core/artifacts
|
||||
go test ./internal/framework/pipeline
|
||||
go test ./internal/cli
|
||||
```
|
||||
|
||||
### Exit criteria
|
||||
|
||||
Manifests and diagnostics explain which plan was requested, selected, and
|
||||
produced; default output remains redacted; debug behavior is explicit; and all
|
||||
compatibility tests pass.
|
||||
|
||||
## Stage 7: Add cross-run, corruption, and concurrency hardening
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
### Objective
|
||||
|
||||
Exercise the completed design across package boundaries and adversarial state,
|
||||
then close any correctness gaps without changing the settled contract.
|
||||
|
||||
### Read first
|
||||
|
||||
- all tests added in Stages 1–6
|
||||
- end-to-end CLI and module integration test harnesses
|
||||
- atomic-write, cancellation, retry, and race-sensitive framework code
|
||||
- repository import-boundary tests
|
||||
|
||||
### Implement
|
||||
|
||||
1. Add black-box tests using separate CLI application instances and a shared
|
||||
temporary chunk-plan root. Prove reuse across different pipelines,
|
||||
configured chunk modules, options, references, lanes, validators, and LLM
|
||||
profiles.
|
||||
2. Prove that a changed canonical source digest selects a different path, while
|
||||
byte-identical canonical sources reuse the same plan.
|
||||
3. Exercise generic plans through D&D pipelines and D&D-annotated plans through
|
||||
generic pipelines. Optional annotations must survive, while no downstream
|
||||
component may require `dnd/scenes` merely because the requested chunker is
|
||||
the D&D scene module.
|
||||
4. Add corruption cases for truncated JSON, unknown fields, schema mismatch,
|
||||
source mismatch, plan-digest mismatch, bad boundaries, invalid annotations,
|
||||
and materialization failure. Verify `auto` regeneration and non-destructive
|
||||
failure behavior exactly.
|
||||
5. Add concurrent reader/writer and writer/writer tests. Readers observe either
|
||||
the old complete envelope or the new complete envelope; after successful
|
||||
writers, the final file is one complete valid record. Do not assert which
|
||||
writer wins.
|
||||
6. Exercise interruption before save, during temp-file creation, and around
|
||||
rename where the implementation permits deterministic fault injection.
|
||||
Prior valid state must remain readable unless a complete replacement was
|
||||
published.
|
||||
7. Verify cancellation, retry limits, warning order, error classification, and
|
||||
debug ordering on misses and hits. No cache hit may consume chunk-stage retry
|
||||
budget or invoke the module.
|
||||
8. Verify legacy chunk checkpoint files cannot influence reuse and that lane
|
||||
resume still keys from effective materialized chunk digests after refresh.
|
||||
9. Extend import-boundary tests so `internal/core/source` owns no framework or
|
||||
module dependencies and the generic store owns no concrete module imports.
|
||||
10. Ensure examples and all tests isolate workspace state in temporary
|
||||
directories and isolate chunk-plan state with a temporary configured root
|
||||
or explicitly select `bypass`; tests must never write to the real per-user
|
||||
default or `/var/cache/notarius/chunk-plans`.
|
||||
11. Run the race detector over the packages that access the plan store and
|
||||
runner concurrently, and fix any race within feature scope.
|
||||
|
||||
### Tests
|
||||
|
||||
In addition to the new focused tests, run:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./internal/framework/chunkplan ./internal/framework/pipeline
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
### Exit criteria
|
||||
|
||||
The feature is demonstrated across independent runs, corruption and concurrent
|
||||
publication cannot expose partial state, cross-domain annotations remain
|
||||
optional, cancellation and retry behavior is deterministic, and all repository
|
||||
and race checks pass.
|
||||
|
||||
## Stage 8: Publish current-behavior documentation
|
||||
|
||||
**Status:** Complete
|
||||
|
||||
### Objective
|
||||
|
||||
After implementation is complete, make the supported behavior discoverable
|
||||
without duplicating canonical ownership across documents.
|
||||
|
||||
### Read first
|
||||
|
||||
- `docs/policy/architecture.md`
|
||||
- `docs/policy/documentation.md`
|
||||
- every current-behavior document linked below
|
||||
- the final code, flags, schemas, examples, and tests from Stages 1–7
|
||||
|
||||
### Implement
|
||||
|
||||
1. Update `docs/policy/architecture.md` with the
|
||||
plan-generation/materialization
|
||||
boundary, source-addressed reuse, and the independent cache-state surface at
|
||||
the level appropriate for the canonical system overview.
|
||||
2. Update `docs/internal/pipeline.md` with mode behavior, hit/miss control flow,
|
||||
validator placement, retry behavior, and removal of chunk checkpoints.
|
||||
3. Update `docs/internal/modules.md` with the `Chunker.Plan` contract, generic
|
||||
boundaries, optional annotation namespaces, and framework materialization.
|
||||
4. Update `docs/internal/overview.md` with the new plan-store package and
|
||||
current state ownership, and remove chunk checkpoint ownership from its
|
||||
checkpoint description. Keep detailed checkpoint and plan execution flow in
|
||||
`docs/internal/pipeline.md` rather than creating a parallel internal state
|
||||
reference.
|
||||
5. Update `docs/config.md` with `workspace.chunk_cache.mode`,
|
||||
`workspace.chunk_cache.directory`, both environment variables, strict
|
||||
values, precedence, and the per-user cache-root default. State explicitly
|
||||
that `workspace.directory` does not affect plan placement and that the
|
||||
configured directory is the root itself. Keep field-level details owned
|
||||
here. Document that Unix `os.UserCacheDir` rejects a relative
|
||||
`XDG_CACHE_HOME` and that this is a configuration error rather than a fallback.
|
||||
6. Update `docs/cli.md` with `--chunk_cache` semantics and examples. Link to
|
||||
configuration rather than restating its entire precedence model.
|
||||
7. Update `docs/operations.md` with plan location, permissions, invalid-state
|
||||
recovery, refresh and bypass operations, concurrency semantics, sensitive
|
||||
debug data, deletion as a recoverable but potentially expensive
|
||||
regeneration event, and the no-history/no-rollback limitation. Document both:
|
||||
- the ordinary Linux per-user default,
|
||||
`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is a valid
|
||||
absolute path, or `$HOME/.cache/notarius/chunk-plans` when it is unset; and
|
||||
- `/var/cache/notarius/chunk-plans` as the recommended configured root for a
|
||||
system-wide Linux deployment running under a dedicated service account.
|
||||
Explain that the system path is not the unprivileged default, show the
|
||||
following minimal configuration, require the operator or package installer
|
||||
to provision it with restrictive service-account ownership and permissions,
|
||||
and warn against sharing the cache across mutually untrusted users:
|
||||
|
||||
```yaml
|
||||
workspace:
|
||||
chunk_cache:
|
||||
directory: /var/cache/notarius/chunk-plans
|
||||
```
|
||||
8. Update `docs/integrations/json-output.md` and
|
||||
`docs/internal/diagnostics.md` with requested versus producer provenance,
|
||||
actions, warnings, and redaction boundaries.
|
||||
9. Keep package-level plan/range/annotation and materialization invariants in
|
||||
`docs/internal/pipeline.md`; no separate source document is needed unless
|
||||
the implemented subsystem becomes too large for that canonical owner.
|
||||
10. Update examples only when they materially clarify operation; examples must
|
||||
use temporary or explicitly configured workspace and chunk-cache paths.
|
||||
11. Remove future-tense disclaimers from the feature roadmap, mark its
|
||||
completion outcomes as achieved, and mark all completed stages in this
|
||||
document. Keep the roadmap as historical target-state context rather than
|
||||
copying implementation details into it.
|
||||
12. Validate every changed link, heading, code symbol, flag, environment name,
|
||||
path, and JSON field against the implementation. Search for stale
|
||||
`Chunker.Chunk`, `ChunkResult`, `chunks.scenes`, and chunk-checkpoint claims.
|
||||
1. Remove the feature roadmap's statement that ADR-0005 is unimplemented and
|
||||
describe it as the implemented target-state record. Mark its completion
|
||||
outcomes as achieved without adding staged implementation detail.
|
||||
2. Recheck architecture, diagnostics, operations, pipeline, and module
|
||||
documentation against the remediated behavior. Update a canonical document
|
||||
only where Stages 9 through 11 changed a current operator or developer
|
||||
contract; do not duplicate implementation detail across documents.
|
||||
3. Mark Stages 9 through 12 complete only after their respective code,
|
||||
documentation, focused tests, and repository-wide validation have passed.
|
||||
Update this document's introduction to state that all stages are complete and
|
||||
that it remains as a concise historical implementation record.
|
||||
4. Validate every changed link, heading, symbol, diagnostic field, permission,
|
||||
and filesystem claim against the implementation.
|
||||
|
||||
### Tests and validation
|
||||
|
||||
@@ -953,12 +569,12 @@ go build ./cmd/notarius
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Also run repository link or documentation checks if present, and manually
|
||||
inspect the reading map in `docs/development.md` and component links in
|
||||
`docs/internal/overview.md` for canonical ownership and navigation consistency.
|
||||
Run any repository documentation or link checker if present, and manually
|
||||
inspect the reading map in [Development](../development.md) and component links
|
||||
in [Internal Overview](../internal/overview.md).
|
||||
|
||||
### Exit criteria
|
||||
|
||||
All canonical documents describe implemented behavior, operational guidance is
|
||||
complete and non-duplicative, no stale chunk API or checkpoint documentation
|
||||
remains, roadmap status is accurate, and the repository is green.
|
||||
The feature roadmap and canonical references describe implemented behavior,
|
||||
this plan accurately records completion, documentation ownership remains clear,
|
||||
all links and names are current, and the repository-wide validation is green.
|
||||
|
||||
Reference in New Issue
Block a user