Document chunk plan caching and provenance
This commit is contained in:
26
docs/cli.md
26
docs/cli.md
@@ -9,7 +9,7 @@ For the minimal end-to-end invocation, see the [README](../README.md).
|
||||
|
||||
```text
|
||||
notarius help
|
||||
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--output-dir path] [--diagnostics-dir path] [--llm-profile id] [--resume] [--session-id id] [--reference selector=path] [--without-reference selector]
|
||||
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--chunk_cache auto|bypass|refresh] [--output-dir path] [--diagnostics-dir path] [--llm-profile id] [--resume] [--session-id id] [--reference selector=path] [--without-reference selector]
|
||||
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
|
||||
notarius pipelines list [--config path/to/config.yml] [--json]
|
||||
```
|
||||
@@ -31,6 +31,12 @@ Flags:
|
||||
comma-separated and must be non-empty.
|
||||
- `--resume`: request checkpoint reuse for this invocation. See
|
||||
[Operations](operations.md#checkpoints) for prerequisites and reuse behavior.
|
||||
- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
|
||||
invocation. `auto` reuses a valid plan by canonical source digest, `bypass`
|
||||
performs no plan-cache I/O, and `refresh` regenerates and replaces a valid
|
||||
plan only after chunk validation succeeds. See
|
||||
[Configuration](config.md#workspace) for the persistent setting, precedence,
|
||||
and cache-root selection.
|
||||
- `--output-dir path`: output root. Defaults to `./notarius-output`.
|
||||
- `--diagnostics-dir path`: diagnostics work directory override for this
|
||||
invocation. It does not change the workspace directory.
|
||||
@@ -125,6 +131,24 @@ go run ./cmd/notarius run dnd-session \
|
||||
--resume
|
||||
```
|
||||
|
||||
Use `refresh` when intentionally replacing the cached plan for the same source:
|
||||
|
||||
```sh
|
||||
go run ./cmd/notarius run dnd-session \
|
||||
--config examples/dnd-spells.config.yml \
|
||||
--input examples/seriatim-minimal-transcript.json \
|
||||
--chunk_cache refresh
|
||||
```
|
||||
|
||||
Use `bypass` for a one-off run that must not inspect or create plan-cache state:
|
||||
|
||||
```sh
|
||||
go run ./cmd/notarius run dnd-session \
|
||||
--config examples/dnd-spells.config.yml \
|
||||
--input examples/seriatim-minimal-transcript.json \
|
||||
--chunk_cache bypass
|
||||
```
|
||||
|
||||
For checkpoint behavior, durable output, diagnostics, retention, and failure
|
||||
inspection, see [Operations](operations.md).
|
||||
|
||||
|
||||
@@ -48,6 +48,8 @@ Built-in defaults:
|
||||
- `workspace.diagnostics.enabled`: `true`
|
||||
- `workspace.resume.enabled`: `false`
|
||||
- `workspace.debug.enabled`: `false`
|
||||
- `workspace.chunk_cache.mode`: `auto`
|
||||
- `workspace.chunk_cache.directory`: unset
|
||||
|
||||
No pipelines are built in. A run requires a configured pipeline.
|
||||
|
||||
@@ -97,6 +99,8 @@ These environment variables are applied after the config file:
|
||||
- `NOTARIUS_WORKSPACE_RESUME_ENABLED`: boolean resume checkpointing
|
||||
enablement.
|
||||
- `NOTARIUS_WORKSPACE_DEBUG_ENABLED`: boolean debug artifact enablement.
|
||||
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`: chunk-plan cache mode.
|
||||
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`: chunk-plan cache root.
|
||||
- `NOTARIUS_WORK_DIR`: deprecated diagnostics work directory compatibility
|
||||
override.
|
||||
- `NOTARIUS_DIAGNOSTICS_RETENTION`: deprecated diagnostics retention
|
||||
@@ -344,11 +348,35 @@ casts still must be present in the source transcript.
|
||||
- `directory`: optional workspace root for Notarius-owned local state.
|
||||
- `resume.enabled`: boolean resume checkpointing setting.
|
||||
- `debug.enabled`: boolean debug artifact setting.
|
||||
- `chunk_cache.mode`: persistent chunk-plan cache mode: `auto`, `bypass`, or
|
||||
`refresh`. The default is `auto`.
|
||||
- `chunk_cache.directory`: optional chunk-plan cache root. This value is the
|
||||
root itself; Notarius does not append `chunk-plans` to it.
|
||||
- `diagnostics`: optional diagnostics settings defined below.
|
||||
|
||||
`workspace.resume.enabled` and `workspace.debug.enabled` are independent.
|
||||
Enabling one does not enable the other. For directory layout, state lifecycle,
|
||||
permissions, and sensitive content, see [Operations](operations.md).
|
||||
`workspace.resume.enabled`, `workspace.debug.enabled`, and
|
||||
`workspace.chunk_cache` are independent. `workspace.directory` does not affect
|
||||
chunk-plan placement. For directory layout, state lifecycle, permissions, and
|
||||
sensitive content, see [Operations](operations.md).
|
||||
|
||||
`chunk_cache.mode` accepts only `auto`, `bypass`, and `refresh`. In `auto`, a
|
||||
valid source-addressed plan is reused and a missing or invalid record is
|
||||
regenerated and published after chunk validation. `bypass` neither reads nor
|
||||
writes plan-cache state. `refresh` always generates a new plan and publishes it
|
||||
only after validation succeeds.
|
||||
|
||||
Configuration values are applied in file then environment order; an explicit
|
||||
`--chunk_cache` CLI value has highest precedence for the mode. The cache root
|
||||
is selected from environment, file, then the per-user default; there is no CLI
|
||||
root override. Every supplied value is parsed strictly even when a higher
|
||||
precedence value wins, so malformed configuration is still an error.
|
||||
|
||||
When `chunk_cache.directory` is unset, the root is
|
||||
`<os.UserCacheDir>/notarius/chunk-plans`. On Unix this is ordinarily
|
||||
`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is an absolute
|
||||
path, or `$HOME/.cache/notarius/chunk-plans` when it is unset. A relative
|
||||
`XDG_CACHE_HOME` is rejected by `os.UserCacheDir`; Notarius reports that as a
|
||||
configuration error and does not fall back to another directory.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
@@ -390,6 +418,7 @@ Configuration validation checks:
|
||||
- supported stage-worker keys and an effective extract worker count in the
|
||||
inclusive range `1..concurrency.total_llm`;
|
||||
- supported diagnostics retention and non-empty work directory;
|
||||
- a supported chunk-cache mode and a chunk-cache directory without NUL bytes;
|
||||
- stale removed fields such as `llm_profiles`.
|
||||
|
||||
Pipeline resolution additionally checks:
|
||||
|
||||
@@ -91,6 +91,16 @@ The manifest fields are:
|
||||
identity;
|
||||
- `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, and
|
||||
`output_encoder`: resolved module keys;
|
||||
- `chunk_plan`: payload-free provenance for the effective chunk plan. `mode`
|
||||
is the effective cache mode; `action` is `reused`, `generated`,
|
||||
`refreshed`, or `bypassed` when a plan was materialized. `requested_module`
|
||||
is the current pipeline chunker, while `producer_input_module`,
|
||||
`producer_module`, `producer_llm_profile`, `producer_references`,
|
||||
`producer_metadata`, `source_digest`, `plan_digest`, `plan_schema_version`,
|
||||
and `created_at` describe the stored or generated producer when available.
|
||||
A cached plan can therefore identify a producer different from the requested
|
||||
module. This object never embeds ranges, units, annotations, prompts,
|
||||
responses, or reference content;
|
||||
- `module_metadata` and `artifact_lanes`: module and per-lane provenance,
|
||||
including prompt and response-schema provenance when provided;
|
||||
- `validator_chains`: resolved validation points and validators;
|
||||
@@ -115,6 +125,10 @@ references.
|
||||
`validation_status` is `approved` when no outputs were rejected and `rejected`
|
||||
when one or more outputs were rejected.
|
||||
|
||||
Producer warnings and the current run's chunk-validation warnings remain in
|
||||
`warnings.json`. The manifest records only provenance and decision summaries;
|
||||
empty producer-only values are omitted for compatibility with existing readers.
|
||||
|
||||
`validator_chains` records the resolved validator chain for each validation
|
||||
point. Entries include stage, lane ID when applicable, module key, and validators
|
||||
with key and execution class. Empty chains are recorded with an empty
|
||||
|
||||
@@ -22,10 +22,18 @@ constructor.
|
||||
|
||||
Typed methods on `RunDirectory` write invocation metadata, redacted effective
|
||||
configuration, resolved pipeline/reference data, checkpoint events, source data
|
||||
when explicitly requested, manifests, reports, warnings, and error text. The
|
||||
when explicitly requested, manifests, reports, warnings, redacted chunk-plan
|
||||
summaries, and error text. The
|
||||
current filenames and their operator-facing contents are listed in
|
||||
[Operations](../operations.md#diagnostics-directory).
|
||||
|
||||
The chunk-plan summary records the effective mode, source and candidate
|
||||
digests, requested module, lookup decision, materialization action, validation
|
||||
decision, and publication decision. Its closed decision values make failures
|
||||
and recoverable invalid records inspectable without serializing plan ranges,
|
||||
annotations, source content, reference content, prompts, model responses, or
|
||||
raw invalid-file bytes.
|
||||
|
||||
JSON methods indent their payload and append a newline. All artifact writes use
|
||||
a temporary file in the target directory, apply the requested permissions, and
|
||||
rename it into place. Artifact resolution accepts only a single relative base
|
||||
@@ -60,7 +68,8 @@ pipeline results, and the final report. This ordering permits later failures to
|
||||
retain the context already established.
|
||||
|
||||
Failures before construction have no `RunDirectory`. Later failures write an
|
||||
error log, preserve any available partial manifest, and apply a failed-run
|
||||
error log, preserve any available partial manifest and chunk-plan summary, and
|
||||
apply a failed-run
|
||||
retention decision. A diagnostics write failure is itself a command failure so
|
||||
the CLI does not report success after losing requested inspection data.
|
||||
|
||||
|
||||
@@ -67,34 +67,41 @@ rules are defined in the
|
||||
|
||||
## Chunkers
|
||||
|
||||
Chunkers implement `contracts.Chunker.Plan`. A plan identifies ordered source
|
||||
unit ranges and may carry optional namespaced JSON annotations; it does not
|
||||
contain materialized chunk content. The framework canonicalizes annotations,
|
||||
validates ranges against the current source, and materializes chunk IDs,
|
||||
indexes, references, content, units, and generic metadata. Annotation
|
||||
namespaces remain optional data: generic framework code and downstream modules
|
||||
must not require D&D scene annotations or import `dnd/scenes`.
|
||||
|
||||
### `internal/modules/generic/chunk/units`
|
||||
|
||||
The generic chunker validates the source document, walks units in configured
|
||||
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
|
||||
Overlap changes the next window start but never reorders units. It records the
|
||||
first and last unit and unit count in chunk metadata, and derives the chunk's
|
||||
canonical source reference from those unit references.
|
||||
The generic chunker validates the source document and returns ranges over units
|
||||
in configured windows. Overlap changes the next window start but never reorders
|
||||
units. Framework materialization derives the resulting chunk identity and
|
||||
generic metadata from those ranges.
|
||||
|
||||
The accepted options and defaults are defined in
|
||||
[Configuration](../config.md#implemented-production-modules). Generic
|
||||
framework validation canonicalizes the returned unit slices before extraction.
|
||||
The chunker decodes its options during construction and retains only the typed
|
||||
window settings used by `Chunk`.
|
||||
window settings used by `Plan`.
|
||||
|
||||
### `internal/modules/dnd/chunk/scenes`
|
||||
|
||||
The scene chunker prepares a structured Scriptorium request from the full
|
||||
transcript, session, and optional D&D reference inputs. It validates the model's
|
||||
scene boundaries against source-unit IDs and converts them into deterministic
|
||||
chunks with canonical source references spanning each scene's units.
|
||||
Preparation injects the shared structured LLM client into the chunker; `Chunk`
|
||||
plan ranges with optional scene annotations. Preparation injects the shared
|
||||
structured LLM client into the chunker; `Plan`
|
||||
supplies only the run-specific profile, session, source, references, and
|
||||
metadata.
|
||||
|
||||
Scene validation requires sequential, contiguous, non-overlapping coverage from
|
||||
the first source unit through the last. Each chunk contains JSON scene content
|
||||
and module-owned metadata for the scene description, boundaries, confidence,
|
||||
participants, and unit count. Boundary caveats become warnings. Malformed
|
||||
the first source unit through the last. Scene descriptions, boundaries,
|
||||
confidence, and participants are module-owned annotations. Boundary caveats
|
||||
become warnings. Malformed
|
||||
structured output is returned as an error; there is no fallback chunker.
|
||||
|
||||
The package embeds its prompt and response schema and reports their non-secret
|
||||
|
||||
@@ -35,7 +35,7 @@ normalize continuations that may overlap across lanes.
|
||||
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
||||
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
|
||||
| `internal/core/source` | Generic source documents, units, chunks, canonical references, lookup, validation, and deterministic source digests. |
|
||||
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
|
||||
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, and checkpoint identity and manifest models. |
|
||||
|
||||
## Framework Packages
|
||||
|
||||
@@ -47,6 +47,7 @@ normalize continuations that may overlap across lanes.
|
||||
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
|
||||
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
|
||||
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
|
||||
| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
|
||||
| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
|
||||
|
||||
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
|
||||
@@ -122,7 +123,8 @@ Implementation details for all production extensions are in
|
||||
| --- | --- | --- |
|
||||
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
|
||||
| Diagnostics | `internal/core/diagnostics` and `internal/cli` | Record redacted invocation, resolution, result, and failure inspection data. |
|
||||
| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable stage outcomes. |
|
||||
| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable extract, merge, and normalize outcomes. |
|
||||
| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. |
|
||||
| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Capture sensitive framework-boundary and LLM-call material. |
|
||||
|
||||
Physical layout, retention, recovery, and sensitive-data handling are defined
|
||||
|
||||
@@ -8,7 +8,8 @@ defaults, and selectable keys are defined in
|
||||
|
||||
Resolution fixes the selected lanes and all stage bindings; preparation
|
||||
constructs every selected implementation before the runner begins source work.
|
||||
After serial input parsing and chunking, the runner dispatches extract work to
|
||||
After serial input parsing and plan selection or generation, the runner
|
||||
materializes chunks and dispatches extract work to
|
||||
one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
|
||||
merge and normalize operations remain serial and may overlap other lanes once
|
||||
all extracts for that lane are terminal.
|
||||
@@ -118,7 +119,8 @@ validator context as applicable. It never invokes an operation method.
|
||||
`PreparedPipeline` keeps private constructed executors and exposes cloned
|
||||
resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
|
||||
that prepared pipeline, raw source input, run identity and timing, optional
|
||||
session and profile metadata, and checkpoint/debug collaborators. The runner
|
||||
session and profile metadata, a chunk-plan store and mode, and checkpoint/debug
|
||||
collaborators. The runner
|
||||
parses source bytes through the already constructed input adapter. Later stage
|
||||
requests receive the generic source model; extract requests receive
|
||||
chunk-scoped input material, while chunk, merge, and normalize requests retain
|
||||
@@ -151,8 +153,9 @@ The runner:
|
||||
1. validates its prepared input;
|
||||
2. parses the raw input with the prepared adapter and validates the generic
|
||||
source document;
|
||||
3. obtains or executes the chunk result;
|
||||
4. validates and canonicalizes chunks;
|
||||
3. selects a stored plan or executes the configured chunker's `Plan` operation;
|
||||
4. canonicalizes and materializes the plan, then validates the resulting
|
||||
chunks;
|
||||
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
|
||||
bounded lane continuation when all extracts for that lane are terminal;
|
||||
6. invokes the prepared output encoder and validates its logical file results;
|
||||
@@ -173,6 +176,31 @@ and validators while performing these transitions:
|
||||
Module-provided warnings and payload warnings are promoted only from attempts
|
||||
whose results are accepted and used.
|
||||
|
||||
## Chunk Plans And Reuse
|
||||
|
||||
`Chunker.Plan` returns a `source.ChunkPlan`: the canonical source digest,
|
||||
ordered unit-ID ranges, and optional plan or range annotations. The framework
|
||||
owns plan canonicalization and materialization. It creates the generic chunks
|
||||
and therefore owns their IDs, indexes, source references, JSON content, units,
|
||||
media type, and generic metadata. Plan and range annotations are independently
|
||||
owned raw JSON and become `Chunk.PlanAnnotations` and `Chunk.Annotations`.
|
||||
|
||||
In `auto`, the runner looks up the source digest before invoking the chunker. A
|
||||
valid hit is materialized and sent through the current run's configured chunk
|
||||
validators; it does not invoke the chunk module, consume its retry budget, or
|
||||
make a chunk-stage LLM call. A missing, invalid, or unmaterializable record
|
||||
generates a candidate. `refresh` generates without lookup; `bypass` generates
|
||||
without cache access. Generated plans are published only after the full chunk
|
||||
validator chain approves them. A validator rejection is a regular rejected
|
||||
pipeline outcome and never replaces a cached plan.
|
||||
|
||||
The store is source-addressed, not pipeline-addressed. Changes to pipeline
|
||||
configuration, requested chunker, options, references, lanes, validators, or
|
||||
LLM profile do not prevent a source-digest hit. The manifest records both the
|
||||
currently requested chunker and the effective plan producer. Cache state and
|
||||
paths are configured and operated outside the runner; see
|
||||
[Configuration](../config.md#workspace) and [Operations](../operations.md).
|
||||
|
||||
The extract job channel has the same capacity as the effective extract worker
|
||||
count, so dispatch applies backpressure. A fixed continuation executor prevents
|
||||
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
|
||||
@@ -180,19 +208,19 @@ continuations publish lane-local results; the coordinator is the only writer of
|
||||
aggregate output and merges those results in resolved lane and source-chunk
|
||||
order.
|
||||
|
||||
## Chunk Canonicalization
|
||||
## Plan Canonicalization And Chunk Materialization
|
||||
|
||||
Before lane execution, generic validation requires unique chunk IDs, matching
|
||||
source identity, indexes matching returned order, a valid canonical reference,
|
||||
non-empty content and media type, and at least one valid source unit per chunk.
|
||||
Units may not repeat inside a chunk and must form a contiguous range in
|
||||
source-document order. The chunk reference must exactly match the source and
|
||||
the first and last unit references.
|
||||
Plan canonicalization requires canonical JSON annotations, a matching source
|
||||
digest, at least one range, existing ordered boundaries, and increasing range
|
||||
starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy.
|
||||
Materialization deterministically reconstructs each range from the current
|
||||
source document and copies annotations without interpreting their namespaces.
|
||||
|
||||
The runner then rebuilds each chunk's unit slice from the source document by
|
||||
unit ID. It preserves the canonical reference, content, media type, and cloned
|
||||
metadata. The framework permits gaps and overlap between separate
|
||||
chunks; stricter coverage policy belongs to the chunk implementation.
|
||||
Before lane execution, generic chunk validation checks the materialized chunks'
|
||||
identities, order, source references, content, media type, units, and metadata.
|
||||
No chunk checkpoint participates in plan selection: plan storage is the only
|
||||
chunk-reuse mechanism. Extract, merge, and normalize checkpoints continue to
|
||||
use materialized chunk digests as their dependencies.
|
||||
|
||||
## Validation And Retries
|
||||
|
||||
|
||||
@@ -57,6 +57,10 @@ Implemented diagnostics artifacts:
|
||||
binding source, without reference content.
|
||||
- `checkpoint-events.json`: checkpoint steps that were reused or executed
|
||||
during an explicit resume invocation.
|
||||
- `chunk-plan.json`: redacted plan-cache lookup, validation, and publication
|
||||
summary. It contains identifiers and decisions, never source units, plan
|
||||
annotations, reference content, prompts, model responses, or invalid-file
|
||||
bytes.
|
||||
- `run-manifest.json`: the same run manifest written to durable output when it
|
||||
is available, including top-level module metadata when present.
|
||||
- `warnings.json`: warning list.
|
||||
@@ -64,6 +68,54 @@ Implemented diagnostics artifacts:
|
||||
- `error.log`: failure message, written after diagnostics directory creation
|
||||
when a run fails.
|
||||
|
||||
## Chunk-Plan Cache
|
||||
|
||||
The chunk-plan cache is independent of the workspace and checkpoints. Its
|
||||
configuration and selection precedence are defined in
|
||||
[Configuration](config.md#workspace); the invocation override is documented in
|
||||
the [CLI reference](cli.md#run).
|
||||
|
||||
When no root is configured, a normal Linux user uses
|
||||
`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is a valid absolute
|
||||
path, or `$HOME/.cache/notarius/chunk-plans` when it is unset. A relative
|
||||
`XDG_CACHE_HOME` is a configuration error. A configured
|
||||
`workspace.chunk_cache.directory` is the root itself, not a parent to which
|
||||
Notarius adds a suffix.
|
||||
|
||||
Each source digest has one file:
|
||||
|
||||
```text
|
||||
<chunk-plan-root>/<source-sha256-hex>/plan.json
|
||||
```
|
||||
|
||||
Directories are created with `0700` permissions and plan files with `0600`.
|
||||
`auto` reuses a complete valid plan or regenerates an absent or invalid one;
|
||||
`refresh` deliberately regenerates; `bypass` performs no cache I/O. A stored
|
||||
plan is still validated and materialized against the current source before use,
|
||||
and the current run's chunk validators always run. Invalid state is recoverable:
|
||||
an `auto` run regenerates and atomically replaces it only after validation
|
||||
succeeds. Delete an exact cache root or digest directory only when regeneration
|
||||
cost is acceptable.
|
||||
|
||||
Publication uses atomic replacement. Concurrent readers observe a complete old
|
||||
or new plan, and concurrent writers leave one complete valid winner; there is
|
||||
no history, lock protocol, or rollback facility. Do not share a cache root
|
||||
between mutually untrusted users because plans can contain source-derived
|
||||
structure and annotations.
|
||||
|
||||
For a system-wide Linux deployment under a dedicated service account, configure
|
||||
and provision a separate restrictive root such as:
|
||||
|
||||
```yaml
|
||||
workspace:
|
||||
chunk_cache:
|
||||
directory: /var/cache/notarius/chunk-plans
|
||||
```
|
||||
|
||||
`/var/cache/notarius/chunk-plans` is a recommended configured service root, not
|
||||
the unprivileged default. The operator or package installer must create it with
|
||||
restrictive service-account ownership and permissions before use.
|
||||
|
||||
## Checkpoints
|
||||
|
||||
When checkpoint writing is enabled for a configured workspace, runs write
|
||||
@@ -158,8 +210,12 @@ failure before a candidate exists has no candidate payload. If the envelope
|
||||
cannot be persisted, the run does not retry that module attempt and reports the
|
||||
debug failure together with any primary attempt error.
|
||||
|
||||
Checkpoint-reused chunk, extract, merge, and normalize work retains the
|
||||
stage-level input and output artifacts but has no retry-attempt artifacts
|
||||
Chunk-plan candidates, materialized chunks, annotations, and chunk-attempt
|
||||
details appear only in these opt-in debug artifacts. They are intentionally not
|
||||
included in normal manifests or the `chunk-plan.json` diagnostics summary.
|
||||
|
||||
Checkpoint-reused extract, merge, and normalize work retains the stage-level
|
||||
input and output artifacts but has no retry-attempt artifacts
|
||||
because no module attempt executed. Debug artifacts may contain source
|
||||
material, reference material, prompt inputs, model outputs, and other sensitive
|
||||
data. Typed artifact
|
||||
@@ -218,6 +274,10 @@ rm -rf /var/lib/notarius/checkpoints/dnd-session/seriatim-abcdef123456/7890abcd1
|
||||
rm -rf /var/lib/notarius/debug/run-1234567890
|
||||
```
|
||||
|
||||
Chunk-plan cache entries can likewise be removed by exact digest directory or
|
||||
configured root. Removal is recoverable, but the next non-bypass run may need
|
||||
to regenerate plans and repeat any chunk-stage LLM work.
|
||||
|
||||
Use exact run-directory paths. Avoid broad cleanup commands against parent
|
||||
directories unless they are part of your own operational policy.
|
||||
|
||||
|
||||
@@ -105,6 +105,14 @@ Stage ownership is explicit:
|
||||
- normalize modules reconcile merged output;
|
||||
- output modules encode accepted results and run outcomes into logical files.
|
||||
|
||||
Chunk modules produce source-addressed chunk plans rather than materialized
|
||||
chunks. The framework validates and materializes those plans into the generic
|
||||
chunk representation before chunk validation and lane execution. Plan reuse is
|
||||
therefore independent of the configured pipeline, module options, references,
|
||||
lanes, validators, and LLM profile: the canonical source digest selects the
|
||||
plan, while the current run still applies its configured chunk validators to
|
||||
the materialized chunks.
|
||||
|
||||
The framework owns orchestration and handoff provenance. Modules return logical
|
||||
results and warnings; they do not own CLI reporting, workspace paths, durable
|
||||
file placement, checkpoints, or diagnostics.
|
||||
@@ -191,6 +199,12 @@ surfaces with separate ownership:
|
||||
- debug artifacts are opt-in inspection data and may contain sensitive source,
|
||||
prompt, reference, and model-output content.
|
||||
|
||||
Chunk-plan cache state is an additional independent surface. It is keyed only
|
||||
by canonical source digest, is not rooted under `workspace.directory`, and is
|
||||
not a checkpoint or a diagnostic. A cache record is atomically replaced as one
|
||||
complete plan envelope; it has no history, locking, or rollback interface.
|
||||
Invalid records are recoverable cache misses rather than pipeline state.
|
||||
|
||||
Writes of durable state are atomic where practical. Paths for writes, moves,
|
||||
overwrites, and deletion must be narrow and explicit. Cleanup that can lose data
|
||||
is opt-in.
|
||||
|
||||
@@ -1,13 +1,12 @@
|
||||
# ADR-0005 Staged Implementation Plan
|
||||
# ADR-0005 Implementation Record
|
||||
|
||||
This document is the executable implementation plan for the target state in
|
||||
This document records the completed implementation of the target state in
|
||||
[ADR-0005 Feature Roadmap](adr0005.md), governed by
|
||||
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The feature is
|
||||
not implemented.
|
||||
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md).
|
||||
|
||||
The audience is an LLM coding agent. Implement the stages in order. Each stage
|
||||
is deliberately scoped to finish with a compiling, tested repository and may be
|
||||
assigned as one implementation prompt.
|
||||
All stages below are complete. The plan remains as historical target-state
|
||||
context; current behavior is documented in the canonical references linked from
|
||||
[Development](../development.md).
|
||||
|
||||
## Execution Rules
|
||||
|
||||
@@ -870,7 +869,7 @@ and race checks pass.
|
||||
|
||||
## Stage 8: Publish current-behavior documentation
|
||||
|
||||
**Status:** Not started
|
||||
**Status:** Complete
|
||||
|
||||
### Objective
|
||||
|
||||
|
||||
@@ -12,6 +12,8 @@ workspace:
|
||||
enabled: false
|
||||
debug:
|
||||
enabled: false
|
||||
chunk_cache:
|
||||
directory: /var/cache/notarius/chunk-plans
|
||||
pipelines:
|
||||
dnd-session:
|
||||
input: seriatim
|
||||
|
||||
@@ -1,4 +1,7 @@
|
||||
version: 2
|
||||
workspace:
|
||||
chunk_cache:
|
||||
mode: bypass
|
||||
pipelines:
|
||||
dnd-session:
|
||||
input: seriatim
|
||||
|
||||
Reference in New Issue
Block a user