Document chunk plan caching and provenance
This commit is contained in:
26
docs/cli.md
26
docs/cli.md
@@ -9,7 +9,7 @@ For the minimal end-to-end invocation, see the [README](../README.md).
|
|||||||
|
|
||||||
```text
|
```text
|
||||||
notarius help
|
notarius help
|
||||||
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--output-dir path] [--diagnostics-dir path] [--llm-profile id] [--resume] [--session-id id] [--reference selector=path] [--without-reference selector]
|
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--chunk_cache auto|bypass|refresh] [--output-dir path] [--diagnostics-dir path] [--llm-profile id] [--resume] [--session-id id] [--reference selector=path] [--without-reference selector]
|
||||||
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
|
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
|
||||||
notarius pipelines list [--config path/to/config.yml] [--json]
|
notarius pipelines list [--config path/to/config.yml] [--json]
|
||||||
```
|
```
|
||||||
@@ -31,6 +31,12 @@ Flags:
|
|||||||
comma-separated and must be non-empty.
|
comma-separated and must be non-empty.
|
||||||
- `--resume`: request checkpoint reuse for this invocation. See
|
- `--resume`: request checkpoint reuse for this invocation. See
|
||||||
[Operations](operations.md#checkpoints) for prerequisites and reuse behavior.
|
[Operations](operations.md#checkpoints) for prerequisites and reuse behavior.
|
||||||
|
- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
|
||||||
|
invocation. `auto` reuses a valid plan by canonical source digest, `bypass`
|
||||||
|
performs no plan-cache I/O, and `refresh` regenerates and replaces a valid
|
||||||
|
plan only after chunk validation succeeds. See
|
||||||
|
[Configuration](config.md#workspace) for the persistent setting, precedence,
|
||||||
|
and cache-root selection.
|
||||||
- `--output-dir path`: output root. Defaults to `./notarius-output`.
|
- `--output-dir path`: output root. Defaults to `./notarius-output`.
|
||||||
- `--diagnostics-dir path`: diagnostics work directory override for this
|
- `--diagnostics-dir path`: diagnostics work directory override for this
|
||||||
invocation. It does not change the workspace directory.
|
invocation. It does not change the workspace directory.
|
||||||
@@ -125,6 +131,24 @@ go run ./cmd/notarius run dnd-session \
|
|||||||
--resume
|
--resume
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Use `refresh` when intentionally replacing the cached plan for the same source:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go run ./cmd/notarius run dnd-session \
|
||||||
|
--config examples/dnd-spells.config.yml \
|
||||||
|
--input examples/seriatim-minimal-transcript.json \
|
||||||
|
--chunk_cache refresh
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `bypass` for a one-off run that must not inspect or create plan-cache state:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go run ./cmd/notarius run dnd-session \
|
||||||
|
--config examples/dnd-spells.config.yml \
|
||||||
|
--input examples/seriatim-minimal-transcript.json \
|
||||||
|
--chunk_cache bypass
|
||||||
|
```
|
||||||
|
|
||||||
For checkpoint behavior, durable output, diagnostics, retention, and failure
|
For checkpoint behavior, durable output, diagnostics, retention, and failure
|
||||||
inspection, see [Operations](operations.md).
|
inspection, see [Operations](operations.md).
|
||||||
|
|
||||||
|
|||||||
@@ -48,6 +48,8 @@ Built-in defaults:
|
|||||||
- `workspace.diagnostics.enabled`: `true`
|
- `workspace.diagnostics.enabled`: `true`
|
||||||
- `workspace.resume.enabled`: `false`
|
- `workspace.resume.enabled`: `false`
|
||||||
- `workspace.debug.enabled`: `false`
|
- `workspace.debug.enabled`: `false`
|
||||||
|
- `workspace.chunk_cache.mode`: `auto`
|
||||||
|
- `workspace.chunk_cache.directory`: unset
|
||||||
|
|
||||||
No pipelines are built in. A run requires a configured pipeline.
|
No pipelines are built in. A run requires a configured pipeline.
|
||||||
|
|
||||||
@@ -97,6 +99,8 @@ These environment variables are applied after the config file:
|
|||||||
- `NOTARIUS_WORKSPACE_RESUME_ENABLED`: boolean resume checkpointing
|
- `NOTARIUS_WORKSPACE_RESUME_ENABLED`: boolean resume checkpointing
|
||||||
enablement.
|
enablement.
|
||||||
- `NOTARIUS_WORKSPACE_DEBUG_ENABLED`: boolean debug artifact enablement.
|
- `NOTARIUS_WORKSPACE_DEBUG_ENABLED`: boolean debug artifact enablement.
|
||||||
|
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`: chunk-plan cache mode.
|
||||||
|
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`: chunk-plan cache root.
|
||||||
- `NOTARIUS_WORK_DIR`: deprecated diagnostics work directory compatibility
|
- `NOTARIUS_WORK_DIR`: deprecated diagnostics work directory compatibility
|
||||||
override.
|
override.
|
||||||
- `NOTARIUS_DIAGNOSTICS_RETENTION`: deprecated diagnostics retention
|
- `NOTARIUS_DIAGNOSTICS_RETENTION`: deprecated diagnostics retention
|
||||||
@@ -344,11 +348,35 @@ casts still must be present in the source transcript.
|
|||||||
- `directory`: optional workspace root for Notarius-owned local state.
|
- `directory`: optional workspace root for Notarius-owned local state.
|
||||||
- `resume.enabled`: boolean resume checkpointing setting.
|
- `resume.enabled`: boolean resume checkpointing setting.
|
||||||
- `debug.enabled`: boolean debug artifact setting.
|
- `debug.enabled`: boolean debug artifact setting.
|
||||||
|
- `chunk_cache.mode`: persistent chunk-plan cache mode: `auto`, `bypass`, or
|
||||||
|
`refresh`. The default is `auto`.
|
||||||
|
- `chunk_cache.directory`: optional chunk-plan cache root. This value is the
|
||||||
|
root itself; Notarius does not append `chunk-plans` to it.
|
||||||
- `diagnostics`: optional diagnostics settings defined below.
|
- `diagnostics`: optional diagnostics settings defined below.
|
||||||
|
|
||||||
`workspace.resume.enabled` and `workspace.debug.enabled` are independent.
|
`workspace.resume.enabled`, `workspace.debug.enabled`, and
|
||||||
Enabling one does not enable the other. For directory layout, state lifecycle,
|
`workspace.chunk_cache` are independent. `workspace.directory` does not affect
|
||||||
permissions, and sensitive content, see [Operations](operations.md).
|
chunk-plan placement. For directory layout, state lifecycle, permissions, and
|
||||||
|
sensitive content, see [Operations](operations.md).
|
||||||
|
|
||||||
|
`chunk_cache.mode` accepts only `auto`, `bypass`, and `refresh`. In `auto`, a
|
||||||
|
valid source-addressed plan is reused and a missing or invalid record is
|
||||||
|
regenerated and published after chunk validation. `bypass` neither reads nor
|
||||||
|
writes plan-cache state. `refresh` always generates a new plan and publishes it
|
||||||
|
only after validation succeeds.
|
||||||
|
|
||||||
|
Configuration values are applied in file then environment order; an explicit
|
||||||
|
`--chunk_cache` CLI value has highest precedence for the mode. The cache root
|
||||||
|
is selected from environment, file, then the per-user default; there is no CLI
|
||||||
|
root override. Every supplied value is parsed strictly even when a higher
|
||||||
|
precedence value wins, so malformed configuration is still an error.
|
||||||
|
|
||||||
|
When `chunk_cache.directory` is unset, the root is
|
||||||
|
`<os.UserCacheDir>/notarius/chunk-plans`. On Unix this is ordinarily
|
||||||
|
`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is an absolute
|
||||||
|
path, or `$HOME/.cache/notarius/chunk-plans` when it is unset. A relative
|
||||||
|
`XDG_CACHE_HOME` is rejected by `os.UserCacheDir`; Notarius reports that as a
|
||||||
|
configuration error and does not fall back to another directory.
|
||||||
|
|
||||||
## Diagnostics
|
## Diagnostics
|
||||||
|
|
||||||
@@ -390,6 +418,7 @@ Configuration validation checks:
|
|||||||
- supported stage-worker keys and an effective extract worker count in the
|
- supported stage-worker keys and an effective extract worker count in the
|
||||||
inclusive range `1..concurrency.total_llm`;
|
inclusive range `1..concurrency.total_llm`;
|
||||||
- supported diagnostics retention and non-empty work directory;
|
- supported diagnostics retention and non-empty work directory;
|
||||||
|
- a supported chunk-cache mode and a chunk-cache directory without NUL bytes;
|
||||||
- stale removed fields such as `llm_profiles`.
|
- stale removed fields such as `llm_profiles`.
|
||||||
|
|
||||||
Pipeline resolution additionally checks:
|
Pipeline resolution additionally checks:
|
||||||
|
|||||||
@@ -91,6 +91,16 @@ The manifest fields are:
|
|||||||
identity;
|
identity;
|
||||||
- `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, and
|
- `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, and
|
||||||
`output_encoder`: resolved module keys;
|
`output_encoder`: resolved module keys;
|
||||||
|
- `chunk_plan`: payload-free provenance for the effective chunk plan. `mode`
|
||||||
|
is the effective cache mode; `action` is `reused`, `generated`,
|
||||||
|
`refreshed`, or `bypassed` when a plan was materialized. `requested_module`
|
||||||
|
is the current pipeline chunker, while `producer_input_module`,
|
||||||
|
`producer_module`, `producer_llm_profile`, `producer_references`,
|
||||||
|
`producer_metadata`, `source_digest`, `plan_digest`, `plan_schema_version`,
|
||||||
|
and `created_at` describe the stored or generated producer when available.
|
||||||
|
A cached plan can therefore identify a producer different from the requested
|
||||||
|
module. This object never embeds ranges, units, annotations, prompts,
|
||||||
|
responses, or reference content;
|
||||||
- `module_metadata` and `artifact_lanes`: module and per-lane provenance,
|
- `module_metadata` and `artifact_lanes`: module and per-lane provenance,
|
||||||
including prompt and response-schema provenance when provided;
|
including prompt and response-schema provenance when provided;
|
||||||
- `validator_chains`: resolved validation points and validators;
|
- `validator_chains`: resolved validation points and validators;
|
||||||
@@ -115,6 +125,10 @@ references.
|
|||||||
`validation_status` is `approved` when no outputs were rejected and `rejected`
|
`validation_status` is `approved` when no outputs were rejected and `rejected`
|
||||||
when one or more outputs were rejected.
|
when one or more outputs were rejected.
|
||||||
|
|
||||||
|
Producer warnings and the current run's chunk-validation warnings remain in
|
||||||
|
`warnings.json`. The manifest records only provenance and decision summaries;
|
||||||
|
empty producer-only values are omitted for compatibility with existing readers.
|
||||||
|
|
||||||
`validator_chains` records the resolved validator chain for each validation
|
`validator_chains` records the resolved validator chain for each validation
|
||||||
point. Entries include stage, lane ID when applicable, module key, and validators
|
point. Entries include stage, lane ID when applicable, module key, and validators
|
||||||
with key and execution class. Empty chains are recorded with an empty
|
with key and execution class. Empty chains are recorded with an empty
|
||||||
|
|||||||
@@ -22,10 +22,18 @@ constructor.
|
|||||||
|
|
||||||
Typed methods on `RunDirectory` write invocation metadata, redacted effective
|
Typed methods on `RunDirectory` write invocation metadata, redacted effective
|
||||||
configuration, resolved pipeline/reference data, checkpoint events, source data
|
configuration, resolved pipeline/reference data, checkpoint events, source data
|
||||||
when explicitly requested, manifests, reports, warnings, and error text. The
|
when explicitly requested, manifests, reports, warnings, redacted chunk-plan
|
||||||
|
summaries, and error text. The
|
||||||
current filenames and their operator-facing contents are listed in
|
current filenames and their operator-facing contents are listed in
|
||||||
[Operations](../operations.md#diagnostics-directory).
|
[Operations](../operations.md#diagnostics-directory).
|
||||||
|
|
||||||
|
The chunk-plan summary records the effective mode, source and candidate
|
||||||
|
digests, requested module, lookup decision, materialization action, validation
|
||||||
|
decision, and publication decision. Its closed decision values make failures
|
||||||
|
and recoverable invalid records inspectable without serializing plan ranges,
|
||||||
|
annotations, source content, reference content, prompts, model responses, or
|
||||||
|
raw invalid-file bytes.
|
||||||
|
|
||||||
JSON methods indent their payload and append a newline. All artifact writes use
|
JSON methods indent their payload and append a newline. All artifact writes use
|
||||||
a temporary file in the target directory, apply the requested permissions, and
|
a temporary file in the target directory, apply the requested permissions, and
|
||||||
rename it into place. Artifact resolution accepts only a single relative base
|
rename it into place. Artifact resolution accepts only a single relative base
|
||||||
@@ -60,7 +68,8 @@ pipeline results, and the final report. This ordering permits later failures to
|
|||||||
retain the context already established.
|
retain the context already established.
|
||||||
|
|
||||||
Failures before construction have no `RunDirectory`. Later failures write an
|
Failures before construction have no `RunDirectory`. Later failures write an
|
||||||
error log, preserve any available partial manifest, and apply a failed-run
|
error log, preserve any available partial manifest and chunk-plan summary, and
|
||||||
|
apply a failed-run
|
||||||
retention decision. A diagnostics write failure is itself a command failure so
|
retention decision. A diagnostics write failure is itself a command failure so
|
||||||
the CLI does not report success after losing requested inspection data.
|
the CLI does not report success after losing requested inspection data.
|
||||||
|
|
||||||
|
|||||||
@@ -67,34 +67,41 @@ rules are defined in the
|
|||||||
|
|
||||||
## Chunkers
|
## Chunkers
|
||||||
|
|
||||||
|
Chunkers implement `contracts.Chunker.Plan`. A plan identifies ordered source
|
||||||
|
unit ranges and may carry optional namespaced JSON annotations; it does not
|
||||||
|
contain materialized chunk content. The framework canonicalizes annotations,
|
||||||
|
validates ranges against the current source, and materializes chunk IDs,
|
||||||
|
indexes, references, content, units, and generic metadata. Annotation
|
||||||
|
namespaces remain optional data: generic framework code and downstream modules
|
||||||
|
must not require D&D scene annotations or import `dnd/scenes`.
|
||||||
|
|
||||||
### `internal/modules/generic/chunk/units`
|
### `internal/modules/generic/chunk/units`
|
||||||
|
|
||||||
The generic chunker validates the source document, walks units in configured
|
The generic chunker validates the source document and returns ranges over units
|
||||||
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
|
in configured windows. Overlap changes the next window start but never reorders
|
||||||
Overlap changes the next window start but never reorders units. It records the
|
units. Framework materialization derives the resulting chunk identity and
|
||||||
first and last unit and unit count in chunk metadata, and derives the chunk's
|
generic metadata from those ranges.
|
||||||
canonical source reference from those unit references.
|
|
||||||
|
|
||||||
The accepted options and defaults are defined in
|
The accepted options and defaults are defined in
|
||||||
[Configuration](../config.md#implemented-production-modules). Generic
|
[Configuration](../config.md#implemented-production-modules). Generic
|
||||||
framework validation canonicalizes the returned unit slices before extraction.
|
framework validation canonicalizes the returned unit slices before extraction.
|
||||||
The chunker decodes its options during construction and retains only the typed
|
The chunker decodes its options during construction and retains only the typed
|
||||||
window settings used by `Chunk`.
|
window settings used by `Plan`.
|
||||||
|
|
||||||
### `internal/modules/dnd/chunk/scenes`
|
### `internal/modules/dnd/chunk/scenes`
|
||||||
|
|
||||||
The scene chunker prepares a structured Scriptorium request from the full
|
The scene chunker prepares a structured Scriptorium request from the full
|
||||||
transcript, session, and optional D&D reference inputs. It validates the model's
|
transcript, session, and optional D&D reference inputs. It validates the model's
|
||||||
scene boundaries against source-unit IDs and converts them into deterministic
|
scene boundaries against source-unit IDs and converts them into deterministic
|
||||||
chunks with canonical source references spanning each scene's units.
|
plan ranges with optional scene annotations. Preparation injects the shared
|
||||||
Preparation injects the shared structured LLM client into the chunker; `Chunk`
|
structured LLM client into the chunker; `Plan`
|
||||||
supplies only the run-specific profile, session, source, references, and
|
supplies only the run-specific profile, session, source, references, and
|
||||||
metadata.
|
metadata.
|
||||||
|
|
||||||
Scene validation requires sequential, contiguous, non-overlapping coverage from
|
Scene validation requires sequential, contiguous, non-overlapping coverage from
|
||||||
the first source unit through the last. Each chunk contains JSON scene content
|
the first source unit through the last. Scene descriptions, boundaries,
|
||||||
and module-owned metadata for the scene description, boundaries, confidence,
|
confidence, and participants are module-owned annotations. Boundary caveats
|
||||||
participants, and unit count. Boundary caveats become warnings. Malformed
|
become warnings. Malformed
|
||||||
structured output is returned as an error; there is no fallback chunker.
|
structured output is returned as an error; there is no fallback chunker.
|
||||||
|
|
||||||
The package embeds its prompt and response schema and reports their non-secret
|
The package embeds its prompt and response schema and reports their non-secret
|
||||||
|
|||||||
@@ -35,7 +35,7 @@ normalize continuations that may overlap across lanes.
|
|||||||
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
||||||
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
|
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
|
||||||
| `internal/core/source` | Generic source documents, units, chunks, canonical references, lookup, validation, and deterministic source digests. |
|
| `internal/core/source` | Generic source documents, units, chunks, canonical references, lookup, validation, and deterministic source digests. |
|
||||||
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
|
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, and checkpoint identity and manifest models. |
|
||||||
|
|
||||||
## Framework Packages
|
## Framework Packages
|
||||||
|
|
||||||
@@ -47,6 +47,7 @@ normalize continuations that may overlap across lanes.
|
|||||||
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
|
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
|
||||||
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
|
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
|
||||||
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
|
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
|
||||||
|
| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
|
||||||
| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
|
| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
|
||||||
|
|
||||||
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
|
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
|
||||||
@@ -122,7 +123,8 @@ Implementation details for all production extensions are in
|
|||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
|
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
|
||||||
| Diagnostics | `internal/core/diagnostics` and `internal/cli` | Record redacted invocation, resolution, result, and failure inspection data. |
|
| Diagnostics | `internal/core/diagnostics` and `internal/cli` | Record redacted invocation, resolution, result, and failure inspection data. |
|
||||||
| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable stage outcomes. |
|
| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable extract, merge, and normalize outcomes. |
|
||||||
|
| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. |
|
||||||
| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Capture sensitive framework-boundary and LLM-call material. |
|
| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Capture sensitive framework-boundary and LLM-call material. |
|
||||||
|
|
||||||
Physical layout, retention, recovery, and sensitive-data handling are defined
|
Physical layout, retention, recovery, and sensitive-data handling are defined
|
||||||
|
|||||||
@@ -8,7 +8,8 @@ defaults, and selectable keys are defined in
|
|||||||
|
|
||||||
Resolution fixes the selected lanes and all stage bindings; preparation
|
Resolution fixes the selected lanes and all stage bindings; preparation
|
||||||
constructs every selected implementation before the runner begins source work.
|
constructs every selected implementation before the runner begins source work.
|
||||||
After serial input parsing and chunking, the runner dispatches extract work to
|
After serial input parsing and plan selection or generation, the runner
|
||||||
|
materializes chunks and dispatches extract work to
|
||||||
one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
|
one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
|
||||||
merge and normalize operations remain serial and may overlap other lanes once
|
merge and normalize operations remain serial and may overlap other lanes once
|
||||||
all extracts for that lane are terminal.
|
all extracts for that lane are terminal.
|
||||||
@@ -118,7 +119,8 @@ validator context as applicable. It never invokes an operation method.
|
|||||||
`PreparedPipeline` keeps private constructed executors and exposes cloned
|
`PreparedPipeline` keeps private constructed executors and exposes cloned
|
||||||
resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
|
resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
|
||||||
that prepared pipeline, raw source input, run identity and timing, optional
|
that prepared pipeline, raw source input, run identity and timing, optional
|
||||||
session and profile metadata, and checkpoint/debug collaborators. The runner
|
session and profile metadata, a chunk-plan store and mode, and checkpoint/debug
|
||||||
|
collaborators. The runner
|
||||||
parses source bytes through the already constructed input adapter. Later stage
|
parses source bytes through the already constructed input adapter. Later stage
|
||||||
requests receive the generic source model; extract requests receive
|
requests receive the generic source model; extract requests receive
|
||||||
chunk-scoped input material, while chunk, merge, and normalize requests retain
|
chunk-scoped input material, while chunk, merge, and normalize requests retain
|
||||||
@@ -151,8 +153,9 @@ The runner:
|
|||||||
1. validates its prepared input;
|
1. validates its prepared input;
|
||||||
2. parses the raw input with the prepared adapter and validates the generic
|
2. parses the raw input with the prepared adapter and validates the generic
|
||||||
source document;
|
source document;
|
||||||
3. obtains or executes the chunk result;
|
3. selects a stored plan or executes the configured chunker's `Plan` operation;
|
||||||
4. validates and canonicalizes chunks;
|
4. canonicalizes and materializes the plan, then validates the resulting
|
||||||
|
chunks;
|
||||||
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
|
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
|
||||||
bounded lane continuation when all extracts for that lane are terminal;
|
bounded lane continuation when all extracts for that lane are terminal;
|
||||||
6. invokes the prepared output encoder and validates its logical file results;
|
6. invokes the prepared output encoder and validates its logical file results;
|
||||||
@@ -173,6 +176,31 @@ and validators while performing these transitions:
|
|||||||
Module-provided warnings and payload warnings are promoted only from attempts
|
Module-provided warnings and payload warnings are promoted only from attempts
|
||||||
whose results are accepted and used.
|
whose results are accepted and used.
|
||||||
|
|
||||||
|
## Chunk Plans And Reuse
|
||||||
|
|
||||||
|
`Chunker.Plan` returns a `source.ChunkPlan`: the canonical source digest,
|
||||||
|
ordered unit-ID ranges, and optional plan or range annotations. The framework
|
||||||
|
owns plan canonicalization and materialization. It creates the generic chunks
|
||||||
|
and therefore owns their IDs, indexes, source references, JSON content, units,
|
||||||
|
media type, and generic metadata. Plan and range annotations are independently
|
||||||
|
owned raw JSON and become `Chunk.PlanAnnotations` and `Chunk.Annotations`.
|
||||||
|
|
||||||
|
In `auto`, the runner looks up the source digest before invoking the chunker. A
|
||||||
|
valid hit is materialized and sent through the current run's configured chunk
|
||||||
|
validators; it does not invoke the chunk module, consume its retry budget, or
|
||||||
|
make a chunk-stage LLM call. A missing, invalid, or unmaterializable record
|
||||||
|
generates a candidate. `refresh` generates without lookup; `bypass` generates
|
||||||
|
without cache access. Generated plans are published only after the full chunk
|
||||||
|
validator chain approves them. A validator rejection is a regular rejected
|
||||||
|
pipeline outcome and never replaces a cached plan.
|
||||||
|
|
||||||
|
The store is source-addressed, not pipeline-addressed. Changes to pipeline
|
||||||
|
configuration, requested chunker, options, references, lanes, validators, or
|
||||||
|
LLM profile do not prevent a source-digest hit. The manifest records both the
|
||||||
|
currently requested chunker and the effective plan producer. Cache state and
|
||||||
|
paths are configured and operated outside the runner; see
|
||||||
|
[Configuration](../config.md#workspace) and [Operations](../operations.md).
|
||||||
|
|
||||||
The extract job channel has the same capacity as the effective extract worker
|
The extract job channel has the same capacity as the effective extract worker
|
||||||
count, so dispatch applies backpressure. A fixed continuation executor prevents
|
count, so dispatch applies backpressure. A fixed continuation executor prevents
|
||||||
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
|
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
|
||||||
@@ -180,19 +208,19 @@ continuations publish lane-local results; the coordinator is the only writer of
|
|||||||
aggregate output and merges those results in resolved lane and source-chunk
|
aggregate output and merges those results in resolved lane and source-chunk
|
||||||
order.
|
order.
|
||||||
|
|
||||||
## Chunk Canonicalization
|
## Plan Canonicalization And Chunk Materialization
|
||||||
|
|
||||||
Before lane execution, generic validation requires unique chunk IDs, matching
|
Plan canonicalization requires canonical JSON annotations, a matching source
|
||||||
source identity, indexes matching returned order, a valid canonical reference,
|
digest, at least one range, existing ordered boundaries, and increasing range
|
||||||
non-empty content and media type, and at least one valid source unit per chunk.
|
starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy.
|
||||||
Units may not repeat inside a chunk and must form a contiguous range in
|
Materialization deterministically reconstructs each range from the current
|
||||||
source-document order. The chunk reference must exactly match the source and
|
source document and copies annotations without interpreting their namespaces.
|
||||||
the first and last unit references.
|
|
||||||
|
|
||||||
The runner then rebuilds each chunk's unit slice from the source document by
|
Before lane execution, generic chunk validation checks the materialized chunks'
|
||||||
unit ID. It preserves the canonical reference, content, media type, and cloned
|
identities, order, source references, content, media type, units, and metadata.
|
||||||
metadata. The framework permits gaps and overlap between separate
|
No chunk checkpoint participates in plan selection: plan storage is the only
|
||||||
chunks; stricter coverage policy belongs to the chunk implementation.
|
chunk-reuse mechanism. Extract, merge, and normalize checkpoints continue to
|
||||||
|
use materialized chunk digests as their dependencies.
|
||||||
|
|
||||||
## Validation And Retries
|
## Validation And Retries
|
||||||
|
|
||||||
|
|||||||
@@ -57,6 +57,10 @@ Implemented diagnostics artifacts:
|
|||||||
binding source, without reference content.
|
binding source, without reference content.
|
||||||
- `checkpoint-events.json`: checkpoint steps that were reused or executed
|
- `checkpoint-events.json`: checkpoint steps that were reused or executed
|
||||||
during an explicit resume invocation.
|
during an explicit resume invocation.
|
||||||
|
- `chunk-plan.json`: redacted plan-cache lookup, validation, and publication
|
||||||
|
summary. It contains identifiers and decisions, never source units, plan
|
||||||
|
annotations, reference content, prompts, model responses, or invalid-file
|
||||||
|
bytes.
|
||||||
- `run-manifest.json`: the same run manifest written to durable output when it
|
- `run-manifest.json`: the same run manifest written to durable output when it
|
||||||
is available, including top-level module metadata when present.
|
is available, including top-level module metadata when present.
|
||||||
- `warnings.json`: warning list.
|
- `warnings.json`: warning list.
|
||||||
@@ -64,6 +68,54 @@ Implemented diagnostics artifacts:
|
|||||||
- `error.log`: failure message, written after diagnostics directory creation
|
- `error.log`: failure message, written after diagnostics directory creation
|
||||||
when a run fails.
|
when a run fails.
|
||||||
|
|
||||||
|
## Chunk-Plan Cache
|
||||||
|
|
||||||
|
The chunk-plan cache is independent of the workspace and checkpoints. Its
|
||||||
|
configuration and selection precedence are defined in
|
||||||
|
[Configuration](config.md#workspace); the invocation override is documented in
|
||||||
|
the [CLI reference](cli.md#run).
|
||||||
|
|
||||||
|
When no root is configured, a normal Linux user uses
|
||||||
|
`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is a valid absolute
|
||||||
|
path, or `$HOME/.cache/notarius/chunk-plans` when it is unset. A relative
|
||||||
|
`XDG_CACHE_HOME` is a configuration error. A configured
|
||||||
|
`workspace.chunk_cache.directory` is the root itself, not a parent to which
|
||||||
|
Notarius adds a suffix.
|
||||||
|
|
||||||
|
Each source digest has one file:
|
||||||
|
|
||||||
|
```text
|
||||||
|
<chunk-plan-root>/<source-sha256-hex>/plan.json
|
||||||
|
```
|
||||||
|
|
||||||
|
Directories are created with `0700` permissions and plan files with `0600`.
|
||||||
|
`auto` reuses a complete valid plan or regenerates an absent or invalid one;
|
||||||
|
`refresh` deliberately regenerates; `bypass` performs no cache I/O. A stored
|
||||||
|
plan is still validated and materialized against the current source before use,
|
||||||
|
and the current run's chunk validators always run. Invalid state is recoverable:
|
||||||
|
an `auto` run regenerates and atomically replaces it only after validation
|
||||||
|
succeeds. Delete an exact cache root or digest directory only when regeneration
|
||||||
|
cost is acceptable.
|
||||||
|
|
||||||
|
Publication uses atomic replacement. Concurrent readers observe a complete old
|
||||||
|
or new plan, and concurrent writers leave one complete valid winner; there is
|
||||||
|
no history, lock protocol, or rollback facility. Do not share a cache root
|
||||||
|
between mutually untrusted users because plans can contain source-derived
|
||||||
|
structure and annotations.
|
||||||
|
|
||||||
|
For a system-wide Linux deployment under a dedicated service account, configure
|
||||||
|
and provision a separate restrictive root such as:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
workspace:
|
||||||
|
chunk_cache:
|
||||||
|
directory: /var/cache/notarius/chunk-plans
|
||||||
|
```
|
||||||
|
|
||||||
|
`/var/cache/notarius/chunk-plans` is a recommended configured service root, not
|
||||||
|
the unprivileged default. The operator or package installer must create it with
|
||||||
|
restrictive service-account ownership and permissions before use.
|
||||||
|
|
||||||
## Checkpoints
|
## Checkpoints
|
||||||
|
|
||||||
When checkpoint writing is enabled for a configured workspace, runs write
|
When checkpoint writing is enabled for a configured workspace, runs write
|
||||||
@@ -158,8 +210,12 @@ failure before a candidate exists has no candidate payload. If the envelope
|
|||||||
cannot be persisted, the run does not retry that module attempt and reports the
|
cannot be persisted, the run does not retry that module attempt and reports the
|
||||||
debug failure together with any primary attempt error.
|
debug failure together with any primary attempt error.
|
||||||
|
|
||||||
Checkpoint-reused chunk, extract, merge, and normalize work retains the
|
Chunk-plan candidates, materialized chunks, annotations, and chunk-attempt
|
||||||
stage-level input and output artifacts but has no retry-attempt artifacts
|
details appear only in these opt-in debug artifacts. They are intentionally not
|
||||||
|
included in normal manifests or the `chunk-plan.json` diagnostics summary.
|
||||||
|
|
||||||
|
Checkpoint-reused extract, merge, and normalize work retains the stage-level
|
||||||
|
input and output artifacts but has no retry-attempt artifacts
|
||||||
because no module attempt executed. Debug artifacts may contain source
|
because no module attempt executed. Debug artifacts may contain source
|
||||||
material, reference material, prompt inputs, model outputs, and other sensitive
|
material, reference material, prompt inputs, model outputs, and other sensitive
|
||||||
data. Typed artifact
|
data. Typed artifact
|
||||||
@@ -218,6 +274,10 @@ rm -rf /var/lib/notarius/checkpoints/dnd-session/seriatim-abcdef123456/7890abcd1
|
|||||||
rm -rf /var/lib/notarius/debug/run-1234567890
|
rm -rf /var/lib/notarius/debug/run-1234567890
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Chunk-plan cache entries can likewise be removed by exact digest directory or
|
||||||
|
configured root. Removal is recoverable, but the next non-bypass run may need
|
||||||
|
to regenerate plans and repeat any chunk-stage LLM work.
|
||||||
|
|
||||||
Use exact run-directory paths. Avoid broad cleanup commands against parent
|
Use exact run-directory paths. Avoid broad cleanup commands against parent
|
||||||
directories unless they are part of your own operational policy.
|
directories unless they are part of your own operational policy.
|
||||||
|
|
||||||
|
|||||||
@@ -105,6 +105,14 @@ Stage ownership is explicit:
|
|||||||
- normalize modules reconcile merged output;
|
- normalize modules reconcile merged output;
|
||||||
- output modules encode accepted results and run outcomes into logical files.
|
- output modules encode accepted results and run outcomes into logical files.
|
||||||
|
|
||||||
|
Chunk modules produce source-addressed chunk plans rather than materialized
|
||||||
|
chunks. The framework validates and materializes those plans into the generic
|
||||||
|
chunk representation before chunk validation and lane execution. Plan reuse is
|
||||||
|
therefore independent of the configured pipeline, module options, references,
|
||||||
|
lanes, validators, and LLM profile: the canonical source digest selects the
|
||||||
|
plan, while the current run still applies its configured chunk validators to
|
||||||
|
the materialized chunks.
|
||||||
|
|
||||||
The framework owns orchestration and handoff provenance. Modules return logical
|
The framework owns orchestration and handoff provenance. Modules return logical
|
||||||
results and warnings; they do not own CLI reporting, workspace paths, durable
|
results and warnings; they do not own CLI reporting, workspace paths, durable
|
||||||
file placement, checkpoints, or diagnostics.
|
file placement, checkpoints, or diagnostics.
|
||||||
@@ -191,6 +199,12 @@ surfaces with separate ownership:
|
|||||||
- debug artifacts are opt-in inspection data and may contain sensitive source,
|
- debug artifacts are opt-in inspection data and may contain sensitive source,
|
||||||
prompt, reference, and model-output content.
|
prompt, reference, and model-output content.
|
||||||
|
|
||||||
|
Chunk-plan cache state is an additional independent surface. It is keyed only
|
||||||
|
by canonical source digest, is not rooted under `workspace.directory`, and is
|
||||||
|
not a checkpoint or a diagnostic. A cache record is atomically replaced as one
|
||||||
|
complete plan envelope; it has no history, locking, or rollback interface.
|
||||||
|
Invalid records are recoverable cache misses rather than pipeline state.
|
||||||
|
|
||||||
Writes of durable state are atomic where practical. Paths for writes, moves,
|
Writes of durable state are atomic where practical. Paths for writes, moves,
|
||||||
overwrites, and deletion must be narrow and explicit. Cleanup that can lose data
|
overwrites, and deletion must be narrow and explicit. Cleanup that can lose data
|
||||||
is opt-in.
|
is opt-in.
|
||||||
|
|||||||
@@ -1,13 +1,12 @@
|
|||||||
# ADR-0005 Staged Implementation Plan
|
# ADR-0005 Implementation Record
|
||||||
|
|
||||||
This document is the executable implementation plan for the target state in
|
This document records the completed implementation of the target state in
|
||||||
[ADR-0005 Feature Roadmap](adr0005.md), governed by
|
[ADR-0005 Feature Roadmap](adr0005.md), governed by
|
||||||
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The feature is
|
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md).
|
||||||
not implemented.
|
|
||||||
|
|
||||||
The audience is an LLM coding agent. Implement the stages in order. Each stage
|
All stages below are complete. The plan remains as historical target-state
|
||||||
is deliberately scoped to finish with a compiling, tested repository and may be
|
context; current behavior is documented in the canonical references linked from
|
||||||
assigned as one implementation prompt.
|
[Development](../development.md).
|
||||||
|
|
||||||
## Execution Rules
|
## Execution Rules
|
||||||
|
|
||||||
@@ -870,7 +869,7 @@ and race checks pass.
|
|||||||
|
|
||||||
## Stage 8: Publish current-behavior documentation
|
## Stage 8: Publish current-behavior documentation
|
||||||
|
|
||||||
**Status:** Not started
|
**Status:** Complete
|
||||||
|
|
||||||
### Objective
|
### Objective
|
||||||
|
|
||||||
|
|||||||
@@ -12,6 +12,8 @@ workspace:
|
|||||||
enabled: false
|
enabled: false
|
||||||
debug:
|
debug:
|
||||||
enabled: false
|
enabled: false
|
||||||
|
chunk_cache:
|
||||||
|
directory: /var/cache/notarius/chunk-plans
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
|
|||||||
@@ -1,4 +1,7 @@
|
|||||||
version: 2
|
version: 2
|
||||||
|
workspace:
|
||||||
|
chunk_cache:
|
||||||
|
mode: bypass
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
|
|||||||
Reference in New Issue
Block a user