eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add ADR, roadmap, and staged implementation plan for input chunk caching

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-17 18:38:08 -05:00
parent 60b86dc40c
commit 1c13e1d64a
3 changed files with 1269 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

138
docs/adr/0005-cache-canonical-chunk-plans-by-source.md Normal file
View File

@@ -0,0 +1,138 @@
# ADR-0005: Cache one canonical chunk plan per source
**Status:** Proposed
**Date:** 2026-07-17
## Context
Notarius may run several extraction passes over the same source. A D&D
transcript, for example, may first produce NPC artifacts and later produce
spell or combat artifacts, with output from an earlier pass supplied as a
reference to a later pass.
An LLM-backed chunker may process an entire, potentially large source in one
expensive request. Recomputing boundaries for every pipeline or pass repeats
that cost and can make otherwise comparable extraction runs use different
source partitions. Stable chunk material also gives later extraction requests
a better opportunity to benefit from provider-side prompt caching.
Chunk boundaries can affect extraction quality. Evidence may span a boundary,
overlap may produce duplicates, and different partitions may change the context
available to a model. Merge and normalization should remove structural signs
of chunking from durable output, but they cannot guarantee recovery of evidence
that an extractor did not receive.
Notarius therefore needs an explicit policy for choosing between automatically
applying the latest chunking configuration and preserving one stable partition
for repeated work on the same source.
## Decision
Notarius assigns one active canonical chunk plan to a source and reuses that
plan by default across pipelines and invocations.
The canonical source identity is derived from the validated generic source
document and covers the source-unit identity, order, and content needed to
interpret plan boundaries. Input-adapter and chunk-producer identities are
recorded as provenance, but the active-plan lookup does not vary with:
- pipeline identity or selected artifact lanes;
- the configured chunk module or its options;
- references;
- LLM provider, model, profile, prompt, or response schema; or
- configuration for later pipeline stages.
When an active plan exists, Notarius uses it even if the current pipeline
configures a different chunk module or different chunk-module settings. The
configured chunk module generates a plan only when none exists or when the
operator explicitly requests recomputation.
The framework-owned minimum plan contract is an ordered, non-empty set of
source-unit ranges. Each range identifies the inclusive start and end unit for
one chunk. A chunk module may also provide namespaced, domain-specific
annotations at plan or range scope. Those annotations are stored with the plan
and passed through the pipeline when present, but they remain optional.
Downstream stages must not assume that annotations associated with the
currently configured chunk module are present on a reused plan produced by a
different module.
The cache stores the plan rather than fully materialized chunks. The framework
validates a reused plan against the current source and deterministically
materializes its ranges into chunks. The same source and plan must produce
byte-stable chunk input for later stages.
Canonical plan storage is a distinct cache surface with an independently
configurable location. It is not coupled to the roots or lifecycles of
invocation checkpoints, diagnostics, debug artifacts, or durable output. This
allows per-user and system-service deployments to apply cache-specific
ownership, permissions, placement, and cleanup policy without relocating other
Notarius state.
One mutable active plan is stored under the canonical source identity and
retains provenance for the module and relevant runtime inputs that produced it.
The effective plan producer is reported separately from the chunk module
requested by the current pipeline; reuse must not attribute cached boundaries
or annotations to a module that did not produce them.
Reuse is enabled by default. Operators can explicitly:
- bypass cached plans for an invocation without changing the active plan; or
- recompute a plan with the configured chunk module and make it active for
later work.
Exact storage layout, configuration fields, CLI syntax, publication mechanics,
recovery behavior, and diagnostics are implementation and operational
contracts rather than part of this decision.
## Alternatives considered
- Recompute chunks on every invocation. This always applies the current
chunking configuration, but repeats the most expensive stage and weakens
provider-side caching and cross-pass comparability.
- Cache every distinct chunking request by including module options,
references, prompts, profiles, and other runtime inputs in its identity. This
closely associates a cached result with its producing request, but reduces
reuse and permits boundary drift across operationally different passes.
- Key plans by source plus chunk module and options. This shares plans across
pipelines using the same strategy, but changing the configured strategy
silently selects a different partition rather than preserving one canonical
partition for the source.
- Require operators to name or supply a plan for every run. Explicit selection
is reproducible and may be useful as an advanced operation, but adds friction
to the default workflow and does not provide automatic reuse.
- Store fully materialized chunks. This simplifies loading, but duplicates
source content and couples durable state to the current chunk representation
rather than the stable boundary decision.
- Store canonical plans beneath the general workspace root. This would reuse an
existing location setting, but it couples a reusable application cache to
checkpoint, diagnostic, and debug state that have different ownership,
sensitivity, retention, and deployment requirements.
## Consequences
Independent pipelines and passes over the same source use stable boundaries by
default. This reduces repeated LLM work, improves cross-pass comparability, and
increases the opportunity for cached provider reads.
The configured chunk module may not execute, and its settings may have no
effect, when an active plan already exists. Domain-specific annotations reflect
the plan's original producer and may be absent or differ from those the current
module would produce. User-visible provenance must make the effective plan
clear.
A poor or outdated partition remains active until an operator replaces it.
This can preserve suboptimal context boundaries and affect extraction recall or
duplication even when merge and normalization hide the partition structure in
durable output. Stable reuse is an intentional priority over automatically
incorporating later chunk-strategy changes.
The framework gains a durable minimal chunk-plan contract and deterministic
materialization responsibility. Chunk modules must separate required boundary
output from optional annotations, and downstream modules may rely only on the
minimal boundary contract unless a future decision introduces an explicit plan
compatibility mechanism.
Operators must configure and secure canonical plan storage independently from
other workspace state when the per-user default is not appropriate. Removing
that cache remains recoverable because Notarius can regenerate it from the
source, but doing so may repeat an expensive LLM operation.

166
docs/roadmap/adr0005.md Normal file
View File

@@ -0,0 +1,166 @@
# ADR-0005 Feature Roadmap
This roadmap defines the intended end state for
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The feature
is not implemented. The ordered coding work belongs in the
[implementation plan](implementation.md).
## Intent
Notarius may run multiple extraction passes over the same source. Chunking,
especially LLM-backed chunking over a large transcript, can be substantially
more expensive than later per-chunk operations. Stable chunk material also
improves the opportunity for provider-side prompt-cache reads across passes.
Notarius therefore prioritizes stable, aggressively reused source partitions
over automatically applying later chunk-module, reference, prompt, model, or
pipeline changes. An operator must explicitly request repartitioning.
## Target State
### Canonical source plan
Each validated generic source document has at most one stored chunk plan. The
source document's canonical digest selects that plan. Pipeline identity,
selected lanes, configured chunk module and options, references, validators,
prompts, schemas, and LLM settings do not participate in lookup.
In the default mode, the first accepted plan for a source is reused by later
pipelines and invocations. A configured chunk module runs only when no valid
stored plan exists, when reuse is bypassed for one invocation, or when the
operator requests refresh.
### Plan and annotation contract
A plan contains an ordered, non-empty collection of inclusive source-unit
ranges. The framework validates these boundaries and deterministically
materializes runtime chunks from the current source document.
Chunk modules may attach optional domain annotations at plan or range scope.
Annotations are namespaced, validated JSON values. They are preserved through
storage and materialization but are never a hard cross-domain capability.
Downstream modules may rely on the generic chunk contract only.
The store contains boundaries, annotations, warnings, and provenance. It does
not contain fully materialized chunks or duplicated source-unit content.
### Persistence
Chunk plans use a dedicated cache root that is independent of
`workspace.directory` and the locations of checkpoints, diagnostics, debug
artifacts, and durable output. The default is the platform-appropriate per-user
cache directory: on Linux, `$XDG_CACHE_HOME/notarius/chunk-plans` when
`XDG_CACHE_HOME` is set to a valid absolute path, otherwise
`$HOME/.cache/notarius/chunk-plans` when it is unset, through the platform
cache-directory resolver. An invalid relative `XDG_CACHE_HOME` is an error, not
a fallback.
Operators may set `workspace.chunk_cache.directory` to replace that root. The
recommended system-wide setting for a dedicated Linux service account is
`/var/cache/notarius/chunk-plans`. This is an operational recommendation, not
the application default; the service account must own the directory and it must
not be shared across mutually untrusted users.
One mutable, versioned plan file is stored beneath the chunk-plan root under the
full canonical source digest. Normal publication and refresh replace that file
atomically; history, rollback, multiple variants, and content deduplication are
outside this feature.
Plan state is potentially sensitive. Paths are confined, directories and files
use restrictive permissions, default diagnostics omit annotation and source
payloads, and debug output remains explicitly opt-in.
### Runtime policy
The effective chunk-cache mode is one of:
- `auto`: load a valid stored plan; otherwise generate, validate, and publish
one;
- `bypass`: do not read or write plan state for this invocation; or
- `refresh`: generate and validate a plan, then atomically replace stored state.
`auto` is the default. Invalid or incompatible stored state is a reported miss
in `auto`; it is replaced only after a newly generated plan is accepted. A
failed generation never overwrites prior state. Concurrent writes must never
expose partial data and use last-successful-atomic-write semantics.
The public CLI override is
`--chunk_cache <auto|bypass|refresh>`. Configuration and environment values use
the same three modes, with CLI taking highest precedence.
### Validation and execution
Framework plan validation and deterministic materialization run for generated
and reused plans. The current pipeline's configured chunk-validator chain then
validates the materialized chunks.
On a cache hit, the configured chunk module is constructed during normal
pipeline preparation but its operation is not invoked and it makes no LLM call.
A validator rejection of a structurally valid reused plan is a run outcome; it
does not implicitly authorize rechunking.
Only the generic `chunks` capability is hard. Domain annotations such as D&D
scene information are opportunistic and cannot be required solely because the
current pipeline selected the module that normally produces them.
Canonical plans are the sole owner of chunk reuse. Invocation-scoped
checkpoints no longer load or record chunk outputs. Downstream checkpoint
identity continues to depend on the digest of the effective materialized
chunks, so refresh invalidates affected downstream work.
### Provenance
Every run distinguishes the chunk module requested by the resolved pipeline
from the producer of the effective stored plan. Durable provenance records the
source and plan digests, plan schema version, effective cache mode and action,
producer module, producer references and LLM profile where applicable, and
non-sensitive producer metadata.
Default diagnostics record lookup, validation, generation, and publication
decisions without source or annotation payloads. Opt-in debug artifacts may
contain complete plan and annotation material and are treated as sensitive.
### Deployment documentation
The implemented configuration reference identifies the dedicated directory
field, its environment override, and the per-user default. The operations guide
documents the resulting filesystem layout and permissions, and recommends
`/var/cache/notarius/chunk-plans` for a system-wide Linux deployment running as
a dedicated service account. It also explains that cache deletion is
recoverable but may repeat expensive chunk generation. Neither document
presents the system-wide path as the default for an ordinary unprivileged
invocation.
## Compatibility Policy
Existing fully materialized chunk checkpoint files are not migrated or promoted
to canonical plans. They are ignored for chunk reuse after this feature lands.
The first `auto` run generates the source's plan through the configured chunk
module.
The existing source digest is the canonical lookup identity. Any source change
that alters that digest creates a separate plan. Earlier plan schema versions,
malformed files, and digest mismatches are incompatible and follow the invalid
`auto`-miss behavior.
## Non-Goals
This feature does not provide:
- plan editing, comparison, history, rollback, or garbage collection;
- remote or shared plan storage;
- multiple active or automatically selected plan variants for one source;
- mandatory domain annotation contracts;
- automatic rechunking because configuration or model inputs changed; or
- a guarantee that an LLM provider will report cache hits.
## Completion Outcomes
The feature is complete when independent runs over the same canonical source
reuse byte-stable materialized chunks across pipeline, lane, reference,
chunk-module, and LLM configuration changes; bypass and refresh obey their
documented state semantics; provenance identifies the effective producer;
legacy chunk checkpoints cannot compete with plan reuse; and all focused,
integration, compatibility, CLI, and repository-wide validation passes. The
configuration and operations references must also document both the per-user
default and the recommended system-wide Linux setting.

965
docs/roadmap/implementation.md Normal file
View File

@@ -0,0 +1,965 @@
# ADR-0005 Staged Implementation Plan
This document is the executable implementation plan for the target state in
[ADR-0005 Feature Roadmap](adr0005.md), governed by
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The feature is
not implemented.
The audience is an LLM coding agent. Implement the stages in order. Each stage
is deliberately scoped to finish with a compiling, tested repository and may be
assigned as one implementation prompt.
## Execution Rules
Before every stage:
1. read [Development](../development.md), both documents under `docs/policy/`,
ADR-0005, this plan, and every stage-specific document named below;
2. inspect `git status` and preserve user changes;
3. inspect the focused package contracts and tests before editing; and
4. confirm that all earlier stages are complete.
During every stage:
- implement only that stage and prerequisites discovered to be inseparable;
- use existing package boundaries and helpers before introducing abstractions;
- keep framework and core packages independent of concrete D&D modules;
- preserve cancellation, deterministic ordering, redaction, path confinement,
and atomic-write invariants;
- add or update focused tests with every behavior change;
- do not update current-behavior documentation before the final documentation
stage, except for ADR acceptance in Stage 1; and
- do not begin the next stage after meeting the current exit criteria.
At the end of every stage:
1. run the focused tests listed for that stage;
2. run `go test ./...`;
3. run `go vet ./...`;
4. run `go build ./cmd/notarius`;
5. run `git diff --check`; and
6. mark the stage complete in this document only after all checks pass.
If repository reality conflicts with this plan, stop and update the plan in the
same change before implementing a materially different design. Do not silently
invent a new contract.
## Fixed Decisions
These choices are settled for this implementation:
- `source.SourceDocument.Digest` is the cache lookup identity.
- The chunk-plan root is independent of `workspace.directory` and every other
state root.
- Its default joins the directory returned by `os.UserCacheDir` with
`notarius/chunk-plans`.
- `workspace.chunk_cache.directory` overrides that default and names the
chunk-plan root itself. The recommended value for a system-wide Linux service
is `/var/cache/notarius/chunk-plans`.
- The plan file is
`<chunk-plan-root>/<64-character-lowercase-source-sha256-hex>/plan.json`. The
validated `sha256:` prefix remains part of the logical digest but is removed
from the path segment.
- The plan envelope schema is `notarius.chunk-plan.v1`.
- There is one mutable plan file per source, atomically replaced. There are no
immutable plan objects, pointers, history, locks, or rollback.
- Concurrent writers use last-successful-atomic-write semantics; they must
never expose a partial file.
- Chunk modules retain the `contracts.Chunker` name but replace `Chunk` with a
`Plan` operation. No legacy interface remains after Stage 2.
- Plans and ranges use `map[string]json.RawMessage` annotations. Annotation
values are canonical valid JSON and namespaces are non-empty trimmed strings.
- Materialized chunks expose range annotations through
`source.Chunk.Annotations` and plan annotations through
`source.Chunk.PlanAnnotations`, never through `Metadata`.
- Framework materialization owns chunk IDs, indexes, references, content,
media type, units, and generic metadata.
- Only the generic `chunks` capability is hard; `chunks.scenes` is removed.
- Cached-plan hits still construct the configured chunker during preparation
but never call its operation or the chunk-stage LLM.
- The current pipeline's configured chunk validators run against materialized
chunks on every path.
- Canonical plan storage is the only chunk-reuse mechanism; chunk checkpoint
APIs and payloads are removed.
- The mode values are `auto`, `bypass`, and `refresh`. They are valid in config,
environment, and CLI, including persistent `refresh` configuration.
- The YAML field is `workspace.chunk_cache.mode`.
- The YAML directory field is `workspace.chunk_cache.directory`.
- The environment variable is `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`.
- The root environment variable is `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`.
- The CLI flag is `--chunk_cache <auto|bypass|refresh>`.
- Precedence is CLI, environment, file, then default `auto`.
- Chunk-plan-root precedence is environment, file, then the per-user default;
there is no CLI root override.
- Every supplied file, environment, and CLI value is parsed strictly even when
a higher-precedence source would override it; precedence selects among valid
values and does not mask malformed configuration.
- Existing chunk checkpoint files are ignored and never migrated.
- A structurally invalid stored plan is an `auto` miss. A configured-validator
rejection of a structurally valid hit does not trigger regeneration.
## Planned Contracts
Use these names unless an existing collision requires the smallest obvious
adjustment.
### Source-zone types
Add to `internal/core/source`:
```go
type ChunkAnnotations map[string]json.RawMessage
type ChunkPlan struct {
SourceDigest string `json:"source_digest"`
Ranges []ChunkRange `json:"ranges"`
Annotations ChunkAnnotations `json:"annotations,omitempty"`
}
type ChunkRange struct {
StartUnitID int `json:"start_unit_id"`
EndUnitID int `json:"end_unit_id"`
Annotations ChunkAnnotations `json:"annotations,omitempty"`
}
```
Add `Annotations ChunkAnnotations` and
`PlanAnnotations ChunkAnnotations` to `source.Chunk`, with JSON names
`annotations` and `plan_annotations` and `omitempty` on both.
The plan digest covers `SourceDigest`, ordered ranges, and canonical annotation
bytes. It excludes producer provenance, warnings, timestamps, and storage
schema. Materialization produces:
- IDs `chunk-%06d`, numbered from one;
- zero-based indexes in plan order;
- references spanning the first and last units in each range;
- cloned contiguous source units;
- media type `application/json`;
- canonical JSON content shaped as `{"units":[...]}`;
- generic metadata containing `start_unit_id`, `end_unit_id`, and `unit_count`;
- cloned range annotations in `Chunk.Annotations`; and
- cloned plan annotations in `Chunk.PlanAnnotations`.
The plan and range scopes remain distinct even when both use the same namespace.
Copying plan annotations into each runtime chunk is intentional: it makes them
available to downstream chunk-scoped operations without duplicating source
content in durable storage.
Framework validation permits gaps and overlap between ranges, but requires
strictly increasing start positions and rejects duplicate or backward ranges.
Producing modules may impose stricter policy before returning a plan.
### Module contract
Replace the current chunk result and operation with:
```go
type ChunkPlanResult struct {
Plan source.ChunkPlan
Warnings []Warning
}
type Chunker interface {
Key() string
ReferenceSlots() []ReferenceSlot
Plan(context.Context, ChunkRequest) (ChunkPlanResult, error)
}
```
Keep `ChunkRequest`, chunk module keys, stage name `chunk`, configuration
bindings, registries, and preparation ownership unchanged.
### Durable plan record
Define framework-facing record and store contracts beside other pipeline
collaborator contracts:
```go
type ChunkPlanProducer struct {
InputModule string
ChunkModule string
LLMProfile string
References []artifacts.ReferenceProvenance
Metadata map[string]any
}
type ChunkPlanRecord struct {
SchemaVersion string
SourceDigest string
PlanDigest string
Plan source.ChunkPlan
Producer ChunkPlanProducer
Warnings []contracts.Warning
CreatedAt time.Time
}
type ChunkPlanStore interface {
Load(sourceDigest string) (ChunkPlanRecord, ChunkPlanDecision, error)
Save(ChunkPlanRecord) error
}
type ChunkPlanStoreFactory func(root string) (ChunkPlanStore, error)
type ChunkPlanDecision struct {
Status ChunkPlanStatus
Reason string
}
type ChunkPlanStatus string
const (
ChunkPlanHit ChunkPlanStatus = "hit"
ChunkPlanMissing ChunkPlanStatus = "missing"
ChunkPlanInvalid ChunkPlanStatus = "invalid"
)
```
`ChunkPlanDecision` reports `hit`, `missing`, or `invalid` plus a redacted
reason. A miss or invalid result returns a zero record and nil error. Missing
and structurally invalid records are recoverable in `auto`. Filesystem access
failures other than absence return an error and are framework failures.
The concrete constructor is:
```go
// Package chunkplan
func NewFilesystemStore(root string) (pipeline.ChunkPlanStore, error)
```
It treats `root` as the exact chunk-plan root. CLI composition injects this
constructor through `cli.Options.ChunkPlanStoreFactory`; tests may inject a
recording factory. `cli.Options.UserCacheDir` has signature
`func() (string, error)` and defaults to `os.UserCacheDir`.
The JSON envelope uses snake-case equivalents of the record fields. It stores
references as provenance only, never reference content. Before returning a hit,
the loader validates schema version, source digest, plan digest, plan structure,
annotation JSON, and materializability against the current source in the runner.
### Cache policy and provenance
Add a `ChunkCacheMode` enum with `auto`, `bypass`, and `refresh`, plus strict
parsing and validation.
Add to `RunInput`:
```go
ChunkCacheMode ChunkCacheMode
ChunkPlans ChunkPlanStore
```
An empty mode used by direct framework callers means `bypass`; the production
CLI always supplies the effective non-empty mode.
Add a run-manifest summary:
```go
type ChunkPlanManifest struct {
Mode string `json:"mode"`
Action string `json:"action,omitempty"`
SourceDigest string `json:"source_digest,omitempty"`
PlanDigest string `json:"plan_digest,omitempty"`
PlanSchemaVersion string `json:"plan_schema_version,omitempty"`
RequestedModule string `json:"requested_module"`
ProducerInputModule string `json:"producer_input_module,omitempty"`
ProducerModule string `json:"producer_module,omitempty"`
ProducerLLMProfile string `json:"producer_llm_profile,omitempty"`
ProducerReferences []ReferenceProvenance `json:"producer_references,omitempty"`
ProducerMetadata map[string]any `json:"producer_metadata,omitempty"`
CreatedAt *time.Time `json:"created_at,omitempty"`
}
```
`Action` is one of `reused`, `generated`, `refreshed`, or `bypassed`. Retain
the existing top-level `input_module` and `chunker` fields as the requested
pipeline modules. Add this field to `artifacts.RunManifest`:
```go
ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"`
```
The runner initializes it with the effective mode and requested chunk module,
then fills candidate and producer fields when a plan reaches materialization.
## Stage 1: Finalize ADR and add the source-zone plan model
**Status:** Not started
### Objective
Accept the architectural decision and introduce domain-neutral plan,
annotation, validation, digest, clone, and materialization primitives without
changing chunk-module execution yet.
### Read first
- `internal/core/source/source.go`
- `internal/core/source/digest.go`
- `internal/core/source/validation.go`
- `internal/core/source/source_test.go`
- every current `source.Chunk` clone or serialization helper found with `rg`
### Implement
1. Revise ADR-0005 before acceptance:
- confirm it retains one source-addressed mutable plan and atomic refresh;
- preserve the independent cache-surface decision;
- keep filesystem and CLI details out of the ADR; and
- change status from `Proposed` to `Accepted`.
2. Add `ChunkAnnotations`, `ChunkPlan`, and `ChunkRange` in
`internal/core/source`.
3. Add canonical annotation validation and cloning. Canonicalize each raw JSON
value by decoding with `json.Decoder.UseNumber` and re-encoding; reject
blank namespaces, invalid JSON, trailing JSON values, and non-finite or
unsupported values.
4. Add plan validation against a `SourceDocument` using the rules in Planned
Contracts.
5. Add `DigestChunkPlan` and deterministic `MaterializeChunkPlan`.
6. Add `Annotations` and `PlanAnnotations` to `source.Chunk`, `DigestChunk`,
every clone helper, debug envelope, and serialized chunk-validation
representation. Ensure all ownership boundaries clone raw bytes and retain
the two scopes separately.
7. Do not change the chunker interface or runner flow in this stage.
### Tests
Add focused tests for:
- annotation canonicalization, invalid JSON, blank namespaces, and mutation
safety;
- plan source mismatch, missing ranges, missing units, backward ranges,
duplicate starts, gaps, and overlap;
- stable plan digests and digest changes for boundary or annotation changes;
- exact materialized IDs, references, content bytes, metadata, annotations,
gaps, and overlap;
- repeated byte-identical materialization; and
- plan- and range-annotation preservation through existing chunk clones, debug
conversion, serialized validation, and downstream extract requests.
Run at minimum:
```sh
go test ./internal/core/source
go test ./internal/framework/contracts ./internal/framework/pipeline
```
### Exit criteria
The repository is green; ADR-0005 is accepted; plan primitives are complete and
tested; current modules and the runner still behave as before.
## Stage 2: Replace the chunk operation with plan generation
**Status:** Not started
### Objective
Make all chunk modules produce plans and make the runner materialize those plans
without adding durable reuse yet.
### Read first
- `docs/internal/modules.md`
- `docs/internal/pipeline.md`
- `internal/framework/contracts/contracts.go`
- `internal/framework/pipeline/chunker_registry.go`
- `internal/framework/pipeline/runner.go`
- `internal/framework/pipeline/chunk_validation.go`
- both production chunk-module packages and all chunk fakes in tests
### Implement
1. Replace `ChunkResult` with `ChunkPlanResult` and `Chunker.Chunk` with
`Chunker.Plan`; keep the request, registry, module keys, and stage name.
2. Update all fakes, registry tests, preparation tests, integration helpers, and
compile-time interface assertions in the same stage. Do not retain adapters
or a legacy chunk-returning interface.
3. Convert `generic/chunk/units` to emit ranges only. Preserve `max_units` and
`overlap_units` semantics.
4. Convert `dnd/chunk/scenes` to emit namespace `dnd/scenes` at both scopes:
- each range value is an object with exact keys `short_title`,
`primary_mode`, `main_participants`, `summary`, `boundary_note`, and
`boundary_confidence`, preserving the current normalized scene values;
and
- the plan value is an object with exact key `boundary_caveats`, containing
the normalized caveat array.
5. Preserve accepted scene boundary caveats as `contracts.Warning` values as
well as in the plan annotation so warning behavior remains visible.
6. Remove `chunks.scenes` from the D&D scene module's provided capabilities.
Keep only `chunks`.
7. Change the runner’s non-cached chunk path to:
- call `Plan` under the existing retry and debug attempt boundary;
- validate the plan structurally;
- materialize chunks;
- run the configured chunk-validator chain on materialized chunks;
- preserve retry and rejection semantics; and
- pass materialized chunks to lanes exactly as before.
8. Update chunk debug payloads to distinguish the generated plan from
materialized chunks. Plan annotations may appear only in opt-in debug data.
9. Leave current chunk checkpoint loading and recording temporarily in place,
storing materialized chunks, so existing resume tests remain green until
Stage 4 removes that path.
### Tests
Update focused module and framework tests to cover:
- exact generic ranges and overlap;
- exact D&D scene annotations and warning conversion;
- annotations absent from generic plans;
- framework materialization rather than module materialization;
- chunk validators receiving materialized annotated chunks;
- retries around generation plus validation;
- stable chunk IDs independent of producing module; and
- no `chunks.scenes` resolution dependency.
Run at minimum:
```sh
go test ./internal/modules/generic/chunk/units
go test ./internal/modules/dnd/chunk/scenes
go test ./internal/framework/contracts ./internal/framework/pipeline
go test ./internal/modules/integration ./internal/cli
```
### Exit criteria
Every chunk module uses the single plan contract, the runner always
materializes, production behavior remains functionally equivalent apart from
framework-owned stable chunk IDs and namespaced annotations, and the repository
is green.
## Stage 3: Add dedicated cache configuration and the plan store
**Status:** Not started
### Objective
Implement the durable source-addressed store and its independent cache-root
configuration without changing the runner to consume the store.
### Read first
- `docs/config.md`
- `docs/operations.md`
- `internal/core/config/*`
- `internal/core/workspace/*`
- `internal/framework/checkpoint/` as the local pattern for a workspace-backed
implementation
- `internal/cli/run.go` configuration loading, without wiring the store yet
### Implement
1. Add `ChunkCacheMode` and strict parsing in the pipeline package.
2. Add `WorkspaceChunkCacheConfig` with `Mode` and `Directory` to config and
file config. The YAML fields are `workspace.chunk_cache.mode` and
`workspace.chunk_cache.directory`.
3. Set default mode to `auto`; support
`NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`; preserve precedence file then
environment. CLI precedence arrives in Stage 5.
4. Support `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR` with environment-over-file
precedence. Trim the value; an unset or empty effective value selects the
per-user default. Do not add a CLI directory override.
5. Add `workspace.DefaultChunkPlanRoot`, accepting an injectable
user-cache-directory resolver and returning
`filepath.Join(userCacheDir, "notarius", "chunk-plans")`. Production later
supplies `os.UserCacheDir`; tests supply a stub. Reject resolver errors and
empty returned directories. Use this exact signature:
```go
func DefaultChunkPlanRoot(userCacheDir func() (string, error)) (string, error)
```
6. Treat a non-empty `workspace.chunk_cache.directory` as the plan root itself,
cleaned with the same path rules as other workspace roots. It does not gain
an implicit `chunk-plans` suffix. `workspace.directory` must not affect this
root, and the chunk-cache directory must not affect checkpoint, diagnostics,
debug, or output roots. Do not change `workspace.FromConfig` or
`workspace.Settings` for this feature.
7. Add the plan record, producer, decision, and store interfaces described
above in the pipeline package.
8. Add `internal/framework/chunkplan` with:
- a filesystem implementation;
- a constructor that accepts the already resolved chunk-plan root;
- path `<chunk-plan-root>/<64-character-source-sha256-hex>/plan.json`, after
strictly validating and stripping the logical `sha256:` prefix;
- schema `notarius.chunk-plan.v1`;
- directories mode `0700` and file mode `0600`;
- atomic same-directory temp-file rename;
- strict envelope decoding and unknown-field rejection;
- complete digest and annotation validation; and
- no raw reference content.
9. `Load` distinguishes missing, invalid, and operational error. `Save` always
atomically replaces and implements last-successful-write semantics.
10. Keep store fakes local to their tests; do not add a production in-memory or
no-op store because `bypass` and a nil collaborator already express
disabled persistence.
11. Do not wire the store into `Runner` or CLI execution in this stage.
### Tests
Cover:
- config defaults, YAML decoding, environment override, invalid values,
cloning, redaction, and validation;
- directory environment-over-file precedence, empty-value fallback, cloning,
redaction, cleaning, and validation;
- exact injected per-user default and resolver failure or empty-result errors;
- independence in both directions between `workspace.directory` and
`workspace.chunk_cache.directory`;
- exact acceptance of `/var/cache/notarius/chunk-plans` as a configured root
without writing to that real path;
- safe full-digest paths and rejection of malformed digests;
- `0700`/`0600` permissions where supported;
- envelope round trip, strict decoding, source and plan digest mismatch,
invalid annotations, schema mismatch, missing files, operational read errors,
atomic replacement, and concurrent writers; and
- absence of raw reference content in persisted JSON.
Run at minimum:
```sh
go test ./internal/core/config ./internal/core/workspace
go test ./internal/framework/chunkplan ./internal/framework/pipeline
go test ./internal/cli
```
### Exit criteria
Configuration and storage are complete and independently tested, but normal
runs still use Stage 2’s generate-and-materialize behavior.
## Stage 4: Integrate cache policy and remove chunk checkpoints
**Status:** Not started
### Objective
Make the runner’s chunk path use the canonical plan store under all three modes
and establish it as the only chunk-reuse mechanism.
### Read first
- `internal/framework/pipeline/runner.go`
- `internal/framework/pipeline/checkpoint.go`
- `internal/framework/pipeline/runner_concurrent.go`
- `internal/framework/checkpoint/loader.go`
- `internal/framework/checkpoint/recorder.go`
- focused runner retry, rejection, debug, and checkpoint tests
### Implement
1. Add `ChunkCacheMode` and `ChunkPlans` to `RunInput`. Empty mode means
`bypass` for direct framework callers.
2. Add synchronized store wrapping only if runner access can be concurrent;
otherwise document and test the intentionally serial chunk-plan access.
3. Implement mode behavior:
- `auto` hit: load, structurally validate against the current source,
materialize, then run configured chunk validators once;
- `auto` miss or invalid: generate through existing retries, validate,
materialize, validate chunks, then save;
- `bypass`: generate and validate normally without calling `Load` or `Save`;
- `refresh`: generate and validate normally, then replace through `Save`.
4. A valid-hit validator rejection is recorded without retry, store mutation,
or implicit regeneration. A store I/O error is a framework error.
5. A failed or rejected generated plan is never saved. An invalid old file is
overwritten only after a replacement passes module execution, structural
validation, materialization, and configured validators.
6. Build producer provenance from the prepared input module key, prepared chunk
module key, resolved chunk-stage LLM profile, chunk-target auxiliary
reference provenance, and chunk-module manifest metadata. Clone and redact
according to existing boundaries.
7. Replay stored producer warnings on a hit, then append current validation
warnings. Do not duplicate warnings from discarded generation attempts.
8. Remove all chunk methods and types from `CheckpointRecorder`,
`CheckpointLoader`, synchronized wrappers, workspace recorder/loader, and
checkpoint manifests and envelopes.
9. Remove runner chunk-checkpoint decisions and events. Retain source, extract,
merge, and normalize checkpoints. Keep downstream fingerprints based on the
digest of materialized chunks.
10. Ignore existing on-disk `chunk/` checkpoint files; do not read, delete, or
migrate them.
11. Update debug behavior:
- hit records lookup and materialization but no chunk module attempt or LLM
call;
- generated, bypassed, and refreshed paths retain attempt debug; and
- default debug envelopes remain redacted.
### Tests
Add table-driven runner tests for every mode and outcome:
- hit, missing, invalid, and I/O error;
- generation success, error, rejection, retry, and cancellation;
- validator warning, rejection, and error on hits and generated plans;
- bypass performs zero store calls;
- refresh performs no load and exactly one successful save;
- invalid record remains untouched when replacement fails;
- stored warnings replay once;
- no module or LLM call on hit;
- downstream chunk digest changes on refresh;
- legacy chunk checkpoint files are ignored; and
- source and lane checkpoint behavior remains intact.
Run at minimum:
```sh
go test ./internal/framework/pipeline
go test ./internal/framework/checkpoint
go test ./internal/modules/integration
```
### Exit criteria
The runner implements complete cache semantics, chunk checkpoints no longer
exist as a code path, all direct framework tests are green, and production CLI
runs are not yet wired to persistent plan storage.
## Stage 5: Wire persistent policy into the CLI
**Status:** Not started
### Objective
Make normal CLI runs resolve the effective mode and dedicated chunk-plan root,
construct the durable plan store when appropriate, and pass it into the runner.
### Read first
- `docs/cli.md`
- `docs/config.md`
- `cmd/notarius/main.go` and `internal/cli/run.go`
- CLI flag, environment, invocation-metadata, and exit-code tests
- the workspace and configuration code changed in Stage 3
### Implement
1. Add the exact flag `--chunk_cache <auto|bypass|refresh>`. Represent an
omitted CLI value separately from an explicit value so precedence is not
inferred from the default printed by the flag package.
2. Resolve the effective mode in this order: explicit CLI flag, environment,
configuration file, default `auto`. An invalid CLI flag is a usage error
with exit code 2; invalid file or environment configuration follows the
existing configuration-load failure path and exit code 1.
3. Include an explicitly requested CLI value in invocation metadata and the
resolved mode in effective-configuration metadata without adding field-level
origin tracking or plan and annotation payloads. Name the optional
invocation JSON field `chunk_cache_override`; the resolved mode already
appears at `workspace.chunk_cache.mode` in the redacted effective config.
4. Add `UserCacheDir func() (string, error)` and
`ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory` to `cli.Options`,
defaulting them to `os.UserCacheDir` and
`chunkplan.NewFilesystemStore`. For `auto` and `refresh`, choose the non-empty
effective `workspace.chunk_cache.directory`; otherwise resolve the per-user
default with `workspace.DefaultChunkPlanRoot`. Invoke the store factory with
that exact root and pass the result and effective mode through `RunInput`.
Never append `chunk-plans` to an explicit directory. The injectable factory
lets CLI tests assert root selection without writing to production paths.
5. For `bypass`, do not resolve, inspect, or create the chunk-plan root and pass
no store. Existing general-workspace setup remains unchanged and independent:
`workspace.directory`, resume, diagnostics, and debug continue to use their
current paths and semantics.
6. Keep resume policy independent of plan reuse. Enabling, disabling, or
selecting a resume run must not change the chunk-cache mode or path.
7. Configuration validation and pipeline-listing commands validate the mode but
do not create workspace directories or plan files.
8. Update CLI usage text only as needed to expose the new flag. Defer all
narrative documentation to Stage 8.
### Tests
Add CLI and integration tests for:
- a default `auto` run writing a plan and a second independent invocation
reusing it without a chunk operation or chunk LLM call;
- file, environment, and CLI precedence, including persistent `refresh` in
file and environment configuration;
- `bypass` performing no plan-store access and resolving or creating no default
chunk-plan root;
- invalid flag, environment, and file values with their established error
classifications;
- an explicit chunk-plan root from file and environment, including exact use of
`/var/cache/notarius/chunk-plans` without an appended suffix, using a
recording store factory and never writing to that real path;
- an explicit root succeeding without calling a failing per-user cache resolver;
- an injected per-user default resolving exactly to
`<user-cache-dir>/notarius/chunk-plans`;
- `workspace.directory` changes having no effect on chunk-plan placement and a
chunk-cache directory having no effect on other workspace state;
- per-user cache-root discovery failure under `auto`, `refresh`, and `bypass`,
including the allowed no-state `bypass` case, plus unwritable-store failures;
- resume plus each cache mode, showing independent behavior;
- validation and listing commands having no filesystem side effects; and
- reuse across changed pipeline, chunk module/options, references, validators,
and LLM settings when the source digest is unchanged.
Run at minimum:
```sh
go test ./internal/cli
go test ./internal/framework/pipeline ./internal/framework/chunkplan
go test ./internal/modules/integration
```
### Exit criteria
The production CLI implements the complete mode and precedence contract,
independent invocations reuse source-addressed plans by default, bypass and
refresh have exact state semantics, and the repository is green.
## Stage 6: Expose producer provenance and safe diagnostics
**Status:** Not started
### Objective
Make every output and diagnostic record distinguish the requested chunk module
from the producer of the effective plan without exposing sensitive payloads by
default.
### Read first
- `docs/integrations/json-output.md`
- `docs/internal/diagnostics.md`
- manifest assembly in `internal/framework/pipeline/runner.go`
- `internal/core/artifacts/artifacts.go`
- current diagnostics and debug-envelope implementations and tests
### Implement
1. Add `ChunkPlanManifest` to the run manifest and all required clone,
conversion, and serialization paths.
2. Populate `mode` and exactly one action after a candidate plan has passed
structural validation and materialization:
- `reused` for an `auto` hit;
- `generated` for an `auto` miss or invalid record;
- `refreshed` for generation under `refresh`; and
- `bypassed` for generation under `bypass`.
Retain that action if the current chunk-validator chain rejects the
candidate; publication still follows Stage 4's accept-only rule. If no
candidate reaches materialization, leave action and candidate plan fields
empty in the partial failure manifest and report the failure in diagnostics.
3. Preserve the existing top-level `chunker` and its module metadata as the
requested module. Never overwrite those fields with cached producer data.
4. Populate producer module, LLM profile, references, metadata, creation time,
source digest, plan digest, and schema from the effective stored or newly
generated record. A reused record reports its original producer, even when
it differs from the requested module.
5. Define producer LLM profile as the resolved chunk-stage profile used during
generation. Leave it empty for producers that did not use an LLM; do not
substitute the current run's requested profile on reuse.
6. Keep producer references as existing redacted provenance records. Deep-clone
metadata, references, warnings, annotations, and raw JSON at every ownership
boundary.
7. Add a redacted `chunk-plan.json` diagnostic artifact, backed by a summary on
`RunOutput` so the CLI can write it on success, rejection, or framework
failure. It contains mode, source and candidate plan digests, requested
module, lookup status and redacted reason, action, validation status, and
publication status. Use these closed values:
- lookup: `hit`, `missing`, `invalid`, or `skipped`;
- validation: `not_run`, `approved`, `approved_with_warnings`, `rejected`,
or `error`; and
- publication: `not_requested`, `published`, or `not_published`.
It contains no source units, materialized chunk content, annotation bytes,
reference content, prompts, model responses, or raw invalid-file content.
8. Keep complete plan, annotations, attempts, and materialized chunks confined
to existing opt-in debug artifacts and apply their normal restrictive file
handling.
9. Preserve JSON-output compatibility by adding fields rather than renaming
unrelated existing fields. Omit producer-only optional values when absent in
the same style as the surrounding manifest schema.
### Tests
Cover:
- all four actions and their exact manifest fields;
- requested and producer modules differing on a hit;
- producer metadata remaining stable when current configuration changes;
- LLM-backed and non-LLM producers;
- deep-clone and mutation safety for raw annotations and metadata;
- default diagnostic redaction, including invalid stored files;
- complete data appearing only under explicit debug configuration; and
- JSON serialization and compatibility fixtures.
Run at minimum:
```sh
go test ./internal/core/artifacts
go test ./internal/framework/pipeline
go test ./internal/cli
```
### Exit criteria
Manifests and diagnostics explain which plan was requested, selected, and
produced; default output remains redacted; debug behavior is explicit; and all
compatibility tests pass.
## Stage 7: Add cross-run, corruption, and concurrency hardening
**Status:** Not started
### Objective
Exercise the completed design across package boundaries and adversarial state,
then close any correctness gaps without changing the settled contract.
### Read first
- all tests added in Stages 1–6
- end-to-end CLI and module integration test harnesses
- atomic-write, cancellation, retry, and race-sensitive framework code
- repository import-boundary tests
### Implement
1. Add black-box tests using separate CLI application instances and a shared
temporary chunk-plan root. Prove reuse across different pipelines,
configured chunk modules, options, references, lanes, validators, and LLM
profiles.
2. Prove that a changed canonical source digest selects a different path, while
byte-identical canonical sources reuse the same plan.
3. Exercise generic plans through D&D pipelines and D&D-annotated plans through
generic pipelines. Optional annotations must survive, while no downstream
component may require `dnd/scenes` merely because the requested chunker is
the D&D scene module.
4. Add corruption cases for truncated JSON, unknown fields, schema mismatch,
source mismatch, plan-digest mismatch, bad boundaries, invalid annotations,
and materialization failure. Verify `auto` regeneration and non-destructive
failure behavior exactly.
5. Add concurrent reader/writer and writer/writer tests. Readers observe either
the old complete envelope or the new complete envelope; after successful
writers, the final file is one complete valid record. Do not assert which
writer wins.
6. Exercise interruption before save, during temp-file creation, and around
rename where the implementation permits deterministic fault injection.
Prior valid state must remain readable unless a complete replacement was
published.
7. Verify cancellation, retry limits, warning order, error classification, and
debug ordering on misses and hits. No cache hit may consume chunk-stage retry
budget or invoke the module.
8. Verify legacy chunk checkpoint files cannot influence reuse and that lane
resume still keys from effective materialized chunk digests after refresh.
9. Extend import-boundary tests so `internal/core/source` owns no framework or
module dependencies and the generic store owns no concrete module imports.
10. Ensure examples and all tests isolate workspace state in temporary
directories and isolate chunk-plan state with a temporary configured root
or explicitly select `bypass`; tests must never write to the real per-user
default or `/var/cache/notarius/chunk-plans`.
11. Run the race detector over the packages that access the plan store and
runner concurrently, and fix any race within feature scope.
### Tests
In addition to the new focused tests, run:
```sh
go test ./...
go test -race ./internal/framework/chunkplan ./internal/framework/pipeline
go vet ./...
go build ./cmd/notarius
```
### Exit criteria
The feature is demonstrated across independent runs, corruption and concurrent
publication cannot expose partial state, cross-domain annotations remain
optional, cancellation and retry behavior is deterministic, and all repository
and race checks pass.
## Stage 8: Publish current-behavior documentation
**Status:** Not started
### Objective
After implementation is complete, make the supported behavior discoverable
without duplicating canonical ownership across documents.
### Read first
- `docs/policy/architecture.md`
- `docs/policy/documentation.md`
- every current-behavior document linked below
- the final code, flags, schemas, examples, and tests from Stages 1–7
### Implement
1. Update `docs/policy/architecture.md` with the
plan-generation/materialization
boundary, source-addressed reuse, and the independent cache-state surface at
the level appropriate for the canonical system overview.
2. Update `docs/internal/pipeline.md` with mode behavior, hit/miss control flow,
validator placement, retry behavior, and removal of chunk checkpoints.
3. Update `docs/internal/modules.md` with the `Chunker.Plan` contract, generic
boundaries, optional annotation namespaces, and framework materialization.
4. Update `docs/internal/overview.md` with the new plan-store package and
current state ownership, and remove chunk checkpoint ownership from its
checkpoint description. Keep detailed checkpoint and plan execution flow in
`docs/internal/pipeline.md` rather than creating a parallel internal state
reference.
5. Update `docs/config.md` with `workspace.chunk_cache.mode`,
`workspace.chunk_cache.directory`, both environment variables, strict
values, precedence, and the per-user cache-root default. State explicitly
that `workspace.directory` does not affect plan placement and that the
configured directory is the root itself. Keep field-level details owned
here. Document that Unix `os.UserCacheDir` rejects a relative
`XDG_CACHE_HOME` and that this is a configuration error rather than a fallback.
6. Update `docs/cli.md` with `--chunk_cache` semantics and examples. Link to
configuration rather than restating its entire precedence model.
7. Update `docs/operations.md` with plan location, permissions, invalid-state
recovery, refresh and bypass operations, concurrency semantics, sensitive
debug data, deletion as a recoverable but potentially expensive
regeneration event, and the no-history/no-rollback limitation. Document both:
- the ordinary Linux per-user default,
`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is a valid
absolute path, or `$HOME/.cache/notarius/chunk-plans` when it is unset; and
- `/var/cache/notarius/chunk-plans` as the recommended configured root for a
system-wide Linux deployment running under a dedicated service account.
Explain that the system path is not the unprivileged default, show the
following minimal configuration, require the operator or package installer
to provision it with restrictive service-account ownership and permissions,
and warn against sharing the cache across mutually untrusted users:
```yaml
workspace:
chunk_cache:
directory: /var/cache/notarius/chunk-plans
```
8. Update `docs/integrations/json-output.md` and
`docs/internal/diagnostics.md` with requested versus producer provenance,
actions, warnings, and redaction boundaries.
9. Keep package-level plan/range/annotation and materialization invariants in
`docs/internal/pipeline.md`; no separate source document is needed unless
the implemented subsystem becomes too large for that canonical owner.
10. Update examples only when they materially clarify operation; examples must
use temporary or explicitly configured workspace and chunk-cache paths.
11. Remove future-tense disclaimers from the feature roadmap, mark its
completion outcomes as achieved, and mark all completed stages in this
document. Keep the roadmap as historical target-state context rather than
copying implementation details into it.
12. Validate every changed link, heading, code symbol, flag, environment name,
path, and JSON field against the implementation. Search for stale
`Chunker.Chunk`, `ChunkResult`, `chunks.scenes`, and chunk-checkpoint claims.
### Tests and validation
Run:
```sh
go test ./...
go vet ./...
go build ./cmd/notarius
git diff --check
```
Also run repository link or documentation checks if present, and manually
inspect the reading map in `docs/development.md` and component links in
`docs/internal/overview.md` for canonical ownership and navigation consistency.
### Exit criteria
All canonical documents describe implemented behavior, operational guidance is
complete and non-duplicative, no stale chunk API or checkpoint documentation
remains, roadmap status is accurate, and the repository is green.