Document chunk plan caching and provenance

This commit is contained in:
2026-07-18 00:57:51 +00:00
parent 6d0a19c94c
commit 96a49bb7cd
12 changed files with 235 additions and 44 deletions

View File

@@ -9,7 +9,7 @@ For the minimal end-to-end invocation, see the [README](../README.md).
```text
notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--output-dir path] [--diagnostics-dir path] [--llm-profile id] [--resume] [--session-id id] [--reference selector=path] [--without-reference selector]
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--chunk_cache auto|bypass|refresh] [--output-dir path] [--diagnostics-dir path] [--llm-profile id] [--resume] [--session-id id] [--reference selector=path] [--without-reference selector]
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list [--config path/to/config.yml] [--json]
```
@@ -31,6 +31,12 @@ Flags:
comma-separated and must be non-empty.
- `--resume`: request checkpoint reuse for this invocation. See
[Operations](operations.md#checkpoints) for prerequisites and reuse behavior.
- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
invocation. `auto` reuses a valid plan by canonical source digest, `bypass`
performs no plan-cache I/O, and `refresh` regenerates and replaces a valid
plan only after chunk validation succeeds. See
[Configuration](config.md#workspace) for the persistent setting, precedence,
and cache-root selection.
- `--output-dir path`: output root. Defaults to `./notarius-output`.
- `--diagnostics-dir path`: diagnostics work directory override for this
invocation. It does not change the workspace directory.
@@ -125,6 +131,24 @@ go run ./cmd/notarius run dnd-session \
--resume
```
Use `refresh` when intentionally replacing the cached plan for the same source:
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-spells.config.yml \
--input examples/seriatim-minimal-transcript.json \
--chunk_cache refresh
```
Use `bypass` for a one-off run that must not inspect or create plan-cache state:
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-spells.config.yml \
--input examples/seriatim-minimal-transcript.json \
--chunk_cache bypass
```
For checkpoint behavior, durable output, diagnostics, retention, and failure
inspection, see [Operations](operations.md).

View File

@@ -48,6 +48,8 @@ Built-in defaults:
- `workspace.diagnostics.enabled`: `true`
- `workspace.resume.enabled`: `false`
- `workspace.debug.enabled`: `false`
- `workspace.chunk_cache.mode`: `auto`
- `workspace.chunk_cache.directory`: unset
No pipelines are built in. A run requires a configured pipeline.
@@ -97,6 +99,8 @@ These environment variables are applied after the config file:
- `NOTARIUS_WORKSPACE_RESUME_ENABLED`: boolean resume checkpointing
enablement.
- `NOTARIUS_WORKSPACE_DEBUG_ENABLED`: boolean debug artifact enablement.
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`: chunk-plan cache mode.
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`: chunk-plan cache root.
- `NOTARIUS_WORK_DIR`: deprecated diagnostics work directory compatibility
override.
- `NOTARIUS_DIAGNOSTICS_RETENTION`: deprecated diagnostics retention
@@ -344,11 +348,35 @@ casts still must be present in the source transcript.
- `directory`: optional workspace root for Notarius-owned local state.
- `resume.enabled`: boolean resume checkpointing setting.
- `debug.enabled`: boolean debug artifact setting.
- `chunk_cache.mode`: persistent chunk-plan cache mode: `auto`, `bypass`, or
`refresh`. The default is `auto`.
- `chunk_cache.directory`: optional chunk-plan cache root. This value is the
root itself; Notarius does not append `chunk-plans` to it.
- `diagnostics`: optional diagnostics settings defined below.
`workspace.resume.enabled` and `workspace.debug.enabled` are independent.
Enabling one does not enable the other. For directory layout, state lifecycle,
permissions, and sensitive content, see [Operations](operations.md).
`workspace.resume.enabled`, `workspace.debug.enabled`, and
`workspace.chunk_cache` are independent. `workspace.directory` does not affect
chunk-plan placement. For directory layout, state lifecycle, permissions, and
sensitive content, see [Operations](operations.md).
`chunk_cache.mode` accepts only `auto`, `bypass`, and `refresh`. In `auto`, a
valid source-addressed plan is reused and a missing or invalid record is
regenerated and published after chunk validation. `bypass` neither reads nor
writes plan-cache state. `refresh` always generates a new plan and publishes it
only after validation succeeds.
Configuration values are applied in file then environment order; an explicit
`--chunk_cache` CLI value has highest precedence for the mode. The cache root
is selected from environment, file, then the per-user default; there is no CLI
root override. Every supplied value is parsed strictly even when a higher
precedence value wins, so malformed configuration is still an error.
When `chunk_cache.directory` is unset, the root is
`<os.UserCacheDir>/notarius/chunk-plans`. On Unix this is ordinarily
`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is an absolute
path, or `$HOME/.cache/notarius/chunk-plans` when it is unset. A relative
`XDG_CACHE_HOME` is rejected by `os.UserCacheDir`; Notarius reports that as a
configuration error and does not fall back to another directory.
## Diagnostics
@@ -390,6 +418,7 @@ Configuration validation checks:
- supported stage-worker keys and an effective extract worker count in the
inclusive range `1..concurrency.total_llm`;
- supported diagnostics retention and non-empty work directory;
- a supported chunk-cache mode and a chunk-cache directory without NUL bytes;
- stale removed fields such as `llm_profiles`.
Pipeline resolution additionally checks:

View File

@@ -91,6 +91,16 @@ The manifest fields are:
identity;
- `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, and
`output_encoder`: resolved module keys;
- `chunk_plan`: payload-free provenance for the effective chunk plan. `mode`
is the effective cache mode; `action` is `reused`, `generated`,
`refreshed`, or `bypassed` when a plan was materialized. `requested_module`
is the current pipeline chunker, while `producer_input_module`,
`producer_module`, `producer_llm_profile`, `producer_references`,
`producer_metadata`, `source_digest`, `plan_digest`, `plan_schema_version`,
and `created_at` describe the stored or generated producer when available.
A cached plan can therefore identify a producer different from the requested
module. This object never embeds ranges, units, annotations, prompts,
responses, or reference content;
- `module_metadata` and `artifact_lanes`: module and per-lane provenance,
including prompt and response-schema provenance when provided;
- `validator_chains`: resolved validation points and validators;
@@ -115,6 +125,10 @@ references.
`validation_status` is `approved` when no outputs were rejected and `rejected`
when one or more outputs were rejected.
Producer warnings and the current run's chunk-validation warnings remain in
`warnings.json`. The manifest records only provenance and decision summaries;
empty producer-only values are omitted for compatibility with existing readers.
`validator_chains` records the resolved validator chain for each validation
point. Entries include stage, lane ID when applicable, module key, and validators
with key and execution class. Empty chains are recorded with an empty

View File

@@ -22,10 +22,18 @@ constructor.
Typed methods on `RunDirectory` write invocation metadata, redacted effective
configuration, resolved pipeline/reference data, checkpoint events, source data
when explicitly requested, manifests, reports, warnings, and error text. The
when explicitly requested, manifests, reports, warnings, redacted chunk-plan
summaries, and error text. The
current filenames and their operator-facing contents are listed in
[Operations](../operations.md#diagnostics-directory).
The chunk-plan summary records the effective mode, source and candidate
digests, requested module, lookup decision, materialization action, validation
decision, and publication decision. Its closed decision values make failures
and recoverable invalid records inspectable without serializing plan ranges,
annotations, source content, reference content, prompts, model responses, or
raw invalid-file bytes.
JSON methods indent their payload and append a newline. All artifact writes use
a temporary file in the target directory, apply the requested permissions, and
rename it into place. Artifact resolution accepts only a single relative base
@@ -60,7 +68,8 @@ pipeline results, and the final report. This ordering permits later failures to
retain the context already established.
Failures before construction have no `RunDirectory`. Later failures write an
error log, preserve any available partial manifest, and apply a failed-run
error log, preserve any available partial manifest and chunk-plan summary, and
apply a failed-run
retention decision. A diagnostics write failure is itself a command failure so
the CLI does not report success after losing requested inspection data.

View File

@@ -67,34 +67,41 @@ rules are defined in the
## Chunkers
Chunkers implement `contracts.Chunker.Plan`. A plan identifies ordered source
unit ranges and may carry optional namespaced JSON annotations; it does not
contain materialized chunk content. The framework canonicalizes annotations,
validates ranges against the current source, and materializes chunk IDs,
indexes, references, content, units, and generic metadata. Annotation
namespaces remain optional data: generic framework code and downstream modules
must not require D&D scene annotations or import `dnd/scenes`.
### `internal/modules/generic/chunk/units`
The generic chunker validates the source document, walks units in configured
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
Overlap changes the next window start but never reorders units. It records the
first and last unit and unit count in chunk metadata, and derives the chunk's
canonical source reference from those unit references.
The generic chunker validates the source document and returns ranges over units
in configured windows. Overlap changes the next window start but never reorders
units. Framework materialization derives the resulting chunk identity and
generic metadata from those ranges.
The accepted options and defaults are defined in
[Configuration](../config.md#implemented-production-modules). Generic
framework validation canonicalizes the returned unit slices before extraction.
The chunker decodes its options during construction and retains only the typed
window settings used by `Chunk`.
window settings used by `Plan`.
### `internal/modules/dnd/chunk/scenes`
The scene chunker prepares a structured Scriptorium request from the full
transcript, session, and optional D&D reference inputs. It validates the model's
scene boundaries against source-unit IDs and converts them into deterministic
chunks with canonical source references spanning each scene's units.
Preparation injects the shared structured LLM client into the chunker; `Chunk`
plan ranges with optional scene annotations. Preparation injects the shared
structured LLM client into the chunker; `Plan`
supplies only the run-specific profile, session, source, references, and
metadata.
Scene validation requires sequential, contiguous, non-overlapping coverage from
the first source unit through the last. Each chunk contains JSON scene content
and module-owned metadata for the scene description, boundaries, confidence,
participants, and unit count. Boundary caveats become warnings. Malformed
the first source unit through the last. Scene descriptions, boundaries,
confidence, and participants are module-owned annotations. Boundary caveats
become warnings. Malformed
structured output is returned as an error; there is no fallback chunker.
The package embeds its prompt and response schema and reports their non-secret

View File

@@ -35,7 +35,7 @@ normalize continuations that may overlap across lanes.
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
| `internal/core/source` | Generic source documents, units, chunks, canonical references, lookup, validation, and deterministic source digests. |
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, and checkpoint identity and manifest models. |
## Framework Packages
@@ -47,6 +47,7 @@ normalize continuations that may overlap across lanes.
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
@@ -122,7 +123,8 @@ Implementation details for all production extensions are in
| --- | --- | --- |
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
| Diagnostics | `internal/core/diagnostics` and `internal/cli` | Record redacted invocation, resolution, result, and failure inspection data. |
| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable stage outcomes. |
| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable extract, merge, and normalize outcomes. |
| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. |
| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Capture sensitive framework-boundary and LLM-call material. |
Physical layout, retention, recovery, and sensitive-data handling are defined

View File

@@ -8,7 +8,8 @@ defaults, and selectable keys are defined in
Resolution fixes the selected lanes and all stage bindings; preparation
constructs every selected implementation before the runner begins source work.
After serial input parsing and chunking, the runner dispatches extract work to
After serial input parsing and plan selection or generation, the runner
materializes chunks and dispatches extract work to
one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
merge and normalize operations remain serial and may overlap other lanes once
all extracts for that lane are terminal.
@@ -118,7 +119,8 @@ validator context as applicable. It never invokes an operation method.
`PreparedPipeline` keeps private constructed executors and exposes cloned
resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
that prepared pipeline, raw source input, run identity and timing, optional
session and profile metadata, and checkpoint/debug collaborators. The runner
session and profile metadata, a chunk-plan store and mode, and checkpoint/debug
collaborators. The runner
parses source bytes through the already constructed input adapter. Later stage
requests receive the generic source model; extract requests receive
chunk-scoped input material, while chunk, merge, and normalize requests retain
@@ -151,8 +153,9 @@ The runner:
1. validates its prepared input;
2. parses the raw input with the prepared adapter and validates the generic
source document;
3. obtains or executes the chunk result;
4. validates and canonicalizes chunks;
3. selects a stored plan or executes the configured chunker's `Plan` operation;
4. canonicalizes and materializes the plan, then validates the resulting
chunks;
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
bounded lane continuation when all extracts for that lane are terminal;
6. invokes the prepared output encoder and validates its logical file results;
@@ -173,6 +176,31 @@ and validators while performing these transitions:
Module-provided warnings and payload warnings are promoted only from attempts
whose results are accepted and used.
## Chunk Plans And Reuse
`Chunker.Plan` returns a `source.ChunkPlan`: the canonical source digest,
ordered unit-ID ranges, and optional plan or range annotations. The framework
owns plan canonicalization and materialization. It creates the generic chunks
and therefore owns their IDs, indexes, source references, JSON content, units,
media type, and generic metadata. Plan and range annotations are independently
owned raw JSON and become `Chunk.PlanAnnotations` and `Chunk.Annotations`.
In `auto`, the runner looks up the source digest before invoking the chunker. A
valid hit is materialized and sent through the current run's configured chunk
validators; it does not invoke the chunk module, consume its retry budget, or
make a chunk-stage LLM call. A missing, invalid, or unmaterializable record
generates a candidate. `refresh` generates without lookup; `bypass` generates
without cache access. Generated plans are published only after the full chunk
validator chain approves them. A validator rejection is a regular rejected
pipeline outcome and never replaces a cached plan.
The store is source-addressed, not pipeline-addressed. Changes to pipeline
configuration, requested chunker, options, references, lanes, validators, or
LLM profile do not prevent a source-digest hit. The manifest records both the
currently requested chunker and the effective plan producer. Cache state and
paths are configured and operated outside the runner; see
[Configuration](../config.md#workspace) and [Operations](../operations.md).
The extract job channel has the same capacity as the effective extract worker
count, so dispatch applies backpressure. A fixed continuation executor prevents
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
@@ -180,19 +208,19 @@ continuations publish lane-local results; the coordinator is the only writer of
aggregate output and merges those results in resolved lane and source-chunk
order.
## Chunk Canonicalization
## Plan Canonicalization And Chunk Materialization
Before lane execution, generic validation requires unique chunk IDs, matching
source identity, indexes matching returned order, a valid canonical reference,
non-empty content and media type, and at least one valid source unit per chunk.
Units may not repeat inside a chunk and must form a contiguous range in
source-document order. The chunk reference must exactly match the source and
the first and last unit references.
Plan canonicalization requires canonical JSON annotations, a matching source
digest, at least one range, existing ordered boundaries, and increasing range
starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy.
Materialization deterministically reconstructs each range from the current
source document and copies annotations without interpreting their namespaces.
The runner then rebuilds each chunk's unit slice from the source document by
unit ID. It preserves the canonical reference, content, media type, and cloned
metadata. The framework permits gaps and overlap between separate
chunks; stricter coverage policy belongs to the chunk implementation.
Before lane execution, generic chunk validation checks the materialized chunks'
identities, order, source references, content, media type, units, and metadata.
No chunk checkpoint participates in plan selection: plan storage is the only
chunk-reuse mechanism. Extract, merge, and normalize checkpoints continue to
use materialized chunk digests as their dependencies.
## Validation And Retries

View File

@@ -57,6 +57,10 @@ Implemented diagnostics artifacts:
binding source, without reference content.
- `checkpoint-events.json`: checkpoint steps that were reused or executed
during an explicit resume invocation.
- `chunk-plan.json`: redacted plan-cache lookup, validation, and publication
summary. It contains identifiers and decisions, never source units, plan
annotations, reference content, prompts, model responses, or invalid-file
bytes.
- `run-manifest.json`: the same run manifest written to durable output when it
is available, including top-level module metadata when present.
- `warnings.json`: warning list.
@@ -64,6 +68,54 @@ Implemented diagnostics artifacts:
- `error.log`: failure message, written after diagnostics directory creation
when a run fails.
## Chunk-Plan Cache
The chunk-plan cache is independent of the workspace and checkpoints. Its
configuration and selection precedence are defined in
[Configuration](config.md#workspace); the invocation override is documented in
the [CLI reference](cli.md#run).
When no root is configured, a normal Linux user uses
`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is a valid absolute
path, or `$HOME/.cache/notarius/chunk-plans` when it is unset. A relative
`XDG_CACHE_HOME` is a configuration error. A configured
`workspace.chunk_cache.directory` is the root itself, not a parent to which
Notarius adds a suffix.
Each source digest has one file:
```text
<chunk-plan-root>/<source-sha256-hex>/plan.json
```
Directories are created with `0700` permissions and plan files with `0600`.
`auto` reuses a complete valid plan or regenerates an absent or invalid one;
`refresh` deliberately regenerates; `bypass` performs no cache I/O. A stored
plan is still validated and materialized against the current source before use,
and the current run's chunk validators always run. Invalid state is recoverable:
an `auto` run regenerates and atomically replaces it only after validation
succeeds. Delete an exact cache root or digest directory only when regeneration
cost is acceptable.
Publication uses atomic replacement. Concurrent readers observe a complete old
or new plan, and concurrent writers leave one complete valid winner; there is
no history, lock protocol, or rollback facility. Do not share a cache root
between mutually untrusted users because plans can contain source-derived
structure and annotations.
For a system-wide Linux deployment under a dedicated service account, configure
and provision a separate restrictive root such as:
```yaml
workspace:
chunk_cache:
directory: /var/cache/notarius/chunk-plans
```
`/var/cache/notarius/chunk-plans` is a recommended configured service root, not
the unprivileged default. The operator or package installer must create it with
restrictive service-account ownership and permissions before use.
## Checkpoints
When checkpoint writing is enabled for a configured workspace, runs write
@@ -158,8 +210,12 @@ failure before a candidate exists has no candidate payload. If the envelope
cannot be persisted, the run does not retry that module attempt and reports the
debug failure together with any primary attempt error.
Checkpoint-reused chunk, extract, merge, and normalize work retains the
stage-level input and output artifacts but has no retry-attempt artifacts
Chunk-plan candidates, materialized chunks, annotations, and chunk-attempt
details appear only in these opt-in debug artifacts. They are intentionally not
included in normal manifests or the `chunk-plan.json` diagnostics summary.
Checkpoint-reused extract, merge, and normalize work retains the stage-level
input and output artifacts but has no retry-attempt artifacts
because no module attempt executed. Debug artifacts may contain source
material, reference material, prompt inputs, model outputs, and other sensitive
data. Typed artifact
@@ -218,6 +274,10 @@ rm -rf /var/lib/notarius/checkpoints/dnd-session/seriatim-abcdef123456/7890abcd1
rm -rf /var/lib/notarius/debug/run-1234567890
```
Chunk-plan cache entries can likewise be removed by exact digest directory or
configured root. Removal is recoverable, but the next non-bypass run may need
to regenerate plans and repeat any chunk-stage LLM work.
Use exact run-directory paths. Avoid broad cleanup commands against parent
directories unless they are part of your own operational policy.

View File

@@ -105,6 +105,14 @@ Stage ownership is explicit:
- normalize modules reconcile merged output;
- output modules encode accepted results and run outcomes into logical files.
Chunk modules produce source-addressed chunk plans rather than materialized
chunks. The framework validates and materializes those plans into the generic
chunk representation before chunk validation and lane execution. Plan reuse is
therefore independent of the configured pipeline, module options, references,
lanes, validators, and LLM profile: the canonical source digest selects the
plan, while the current run still applies its configured chunk validators to
the materialized chunks.
The framework owns orchestration and handoff provenance. Modules return logical
results and warnings; they do not own CLI reporting, workspace paths, durable
file placement, checkpoints, or diagnostics.
@@ -191,6 +199,12 @@ surfaces with separate ownership:
- debug artifacts are opt-in inspection data and may contain sensitive source,
prompt, reference, and model-output content.
Chunk-plan cache state is an additional independent surface. It is keyed only
by canonical source digest, is not rooted under `workspace.directory`, and is
not a checkpoint or a diagnostic. A cache record is atomically replaced as one
complete plan envelope; it has no history, locking, or rollback interface.
Invalid records are recoverable cache misses rather than pipeline state.
Writes of durable state are atomic where practical. Paths for writes, moves,
overwrites, and deletion must be narrow and explicit. Cleanup that can lose data
is opt-in.

View File

@@ -1,13 +1,12 @@
# ADR-0005 Staged Implementation Plan
# ADR-0005 Implementation Record
This document is the executable implementation plan for the target state in
This document records the completed implementation of the target state in
[ADR-0005 Feature Roadmap](adr0005.md), governed by
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The feature is
not implemented.
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md).
The audience is an LLM coding agent. Implement the stages in order. Each stage
is deliberately scoped to finish with a compiling, tested repository and may be
assigned as one implementation prompt.
All stages below are complete. The plan remains as historical target-state
context; current behavior is documented in the canonical references linked from
[Development](../development.md).
## Execution Rules
@@ -870,7 +869,7 @@ and race checks pass.
## Stage 8: Publish current-behavior documentation
**Status:** Not started
**Status:** Complete
### Objective

View File

@@ -12,6 +12,8 @@ workspace:
enabled: false
debug:
enabled: false
chunk_cache:
directory: /var/cache/notarius/chunk-plans
pipelines:
dnd-session:
input: seriatim

View File

@@ -1,4 +1,7 @@
version: 2
workspace:
chunk_cache:
mode: bypass
pipelines:
dnd-session:
input: seriatim