eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement ADR-0007

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-19 10:15:28 -05:00
parent f64bb7c883
commit 385e4593f4
15 changed files with 165 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/adr/0006-separate-output-cache-and-debug-state.md
View File

@@ -1,6 +1,6 @@
# ADR-0006: Separate output, cache, and debug state
**Status:** Accepted
**Status:** Superseded by [ADR-0007](0007-separate-checkpoint-recording-from-reuse.md)
**Date:** 2026-07-17
## Context

50
docs/adr/0007-separate-checkpoint-recording-from-reuse.md Normal file
View File

@@ -0,0 +1,50 @@
# ADR-0007: Separate checkpoint recording from reuse
**Status:** Accepted
**Date:** 2026-07-19
## Context
ADR-0006 made checkpoint I/O conditional on an explicit `--resume` invocation.
That policy requires an operator to anticipate the need for recovery before a
run begins. A failed ordinary run cannot reuse completed work because it did not
record checkpoints.
Recording reconstructible state and authorizing reuse are separate operational
decisions. Recording consumes storage and retains sensitive derived application
data, while reuse may change which module operations execute during a run.
## Decision
ADR-0006's separation of output, cache, and debug surfaces remains in effect;
this decision supersedes only its checkpoint invocation policy.
Checkpoint recording is controlled by an explicit persistent Boolean
configuration setting and remains disabled by default. When recording is
enabled, every run records checkpoint transitions and reusable approved stage
results.
Checkpoint loading remains an invocation policy. Only a run with `--resume`
loads and reuses compatible completed work. A recording-enabled run without
`--resume` executes every stage normally and never loads checkpoints. A resume
request while recording is disabled is rejected.
The existing checkpoint identities, compatibility rules, payload format,
filesystem root behavior, and pipeline collaborator contracts remain unchanged.
## Alternatives considered
- Continue coupling reads and writes to `--resume`. This is safe by default but
prevents recovery unless resume was anticipated on the earlier run.
- Always record checkpoints. This maximizes recovery but creates potentially
sensitive state without explicit operator consent.
- Add a multi-value recording policy. This preserves the old behavior as an
option but adds configuration complexity without a current need.
## Consequences
Operators can opt into recovery-ready runs while keeping checkpoint reuse
explicit. Enabled successful, rejected, and failed runs may all leave sensitive
checkpoint state, so operators remain responsible for access and retention.
Disabled configurations perform no checkpoint I/O, and `--resume` requires the
operator to enable recording first.

9
docs/cli.md
View File

@@ -29,7 +29,8 @@ Flags:
rules in [Configuration](config.md#discovery).
- `--only lane-a,lane-b`: run only the named artifact lanes. Values are
comma-separated and must be non-empty.
- `--resume`: request checkpoint reuse for this invocation. See
- `--resume`: request checkpoint reuse for this invocation. Checkpoint recording
must be enabled in configuration. See
[Operations](operations.md#checkpoint-cache) for prerequisites and reuse
behavior.
- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
@@ -124,9 +125,9 @@ go run ./cmd/notarius run dnd-session \
--session-id campaign-17-session-04
```
The resume flag can be added to an otherwise identical run invocation. It both
loads compatible checkpoints and records replacements for work executed by that
invocation; without it, the checkpoint root is not used:
When `cache.checkpoints.enabled` is `true`, runs record checkpoints whether or
not `--resume` is present. Add the resume flag to load and reuse compatible
recorded work; using it while checkpoint recording is disabled is an error:
```sh
go run ./cmd/notarius run dnd-session \

19
docs/config.md
View File

@@ -48,6 +48,7 @@ Built-in defaults:
- `cache.chunk_plans.mode`: `auto`
- `cache.chunk_plans.directory`: unset, selecting
`<os.UserCacheDir>/notarius/chunk-plans`
- `cache.checkpoints.enabled`: `false`
- `cache.checkpoints.directory`: unset, selecting
`<os.UserCacheDir>/notarius/checkpoints`
- `debug.directory`: `./notarius-debug`
@@ -349,6 +350,7 @@ cache:
directory: ""
mode: auto
checkpoints:
enabled: false
directory: ""
debug:
directory: ./notarius-debug
@@ -372,9 +374,11 @@ no CLI cache-root override. The defaults are
uses an absolute `$XDG_CACHE_HOME` or falls back to `$HOME/.cache`. A relative
`XDG_CACHE_HOME` is an error.
Checkpoint I/O occurs only for `notarius run --resume`. That invocation loads
compatible checkpoints and records work it executes. Without `--resume`,
Notarius does not resolve, create, load, or record the checkpoint root.
`cache.checkpoints.enabled` defaults to `false`. When `true`, every run records
checkpoint transitions and reusable approved results. When `false`, Notarius
does not resolve or create the checkpoint root, and `--resume` is rejected.
The `--resume` flag authorizes loading compatible checkpoints; it does not
control recording.
`debug.directory` chooses a root but never enables debug capture. Its precedence
is `--debug-dir`, `NOTARIUS_DEBUG_DIR`, the file value, then the default.
@@ -418,13 +422,15 @@ cache:
directory: /srv/notarius/chunk-plans
mode: auto
checkpoints:
enabled: true
directory: /srv/notarius/state/checkpoints
debug:
directory: /srv/notarius/debug
```
Run the migrated configuration with `--resume` when checkpoint reuse or
recording is wanted, and with `--debug` when a debug bundle is wanted.
Run the migrated configuration with `--resume` when checkpoint reuse is wanted,
and with `--debug` when a debug bundle is wanted. Enabled checkpoint recording
occurs with or without `--resume`.
The removed fields are `workspace.directory`, `workspace.resume.enabled`,
`workspace.debug.enabled`, `workspace.chunk_cache.mode`,
@@ -452,7 +458,8 @@ Configuration validation checks:
- supported stage-worker keys and an effective extract worker count in the
inclusive range `1..concurrency.total_llm`;
- non-empty output and debug directories;
- a supported chunk-cache mode and state-surface directories without NUL bytes;
- a supported chunk-cache mode, Boolean checkpoint enablement, and state-surface
directories without NUL bytes;
- stale removed fields such as `llm_profiles`.
Pipeline resolution additionally checks:

5
docs/internal/state.md
View File

@@ -23,8 +23,9 @@ and atomic publication. Its store is constructed only when the selected mode is
not `bypass`.
`internal/framework/checkpoint` owns checkpoint identity, manifests, payload
codecs, loader, and recorder. The CLI constructs both loader and recorder only
for a `--resume` invocation. The serialized
codecs, loader, and recorder. The CLI constructs a recorder whenever checkpoint
recording is enabled and constructs a loader only for a `--resume` invocation.
The serialized
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
they do not describe a current public state surface.

18
docs/operations.md
View File

@@ -13,8 +13,9 @@ Notarius uses three independent filesystem surfaces:
- debug is explicitly requested inspection data.
Choose separate roots and access controls for each surface. A normal run writes
durable output and may use the chunk-plan cache. It does not create checkpoint
or debug state unless its invocation includes `--resume` or `--debug`.
durable output, may use the chunk-plan cache, and records checkpoints when
`cache.checkpoints.enabled` is true. It does not create debug state unless its
invocation includes `--debug`.
## Output
@@ -73,10 +74,14 @@ cache:
## Checkpoint Cache
Checkpoint state is used only by an invocation with `--resume`. That invocation
loads compatible completed work and records checkpoints for work it executes.
Without `--resume`, Notarius neither resolves nor creates the checkpoint root,
and neither loads nor records checkpoints.
Checkpoint recording is controlled by `cache.checkpoints.enabled`, which
defaults to `false`. When enabled, every run records running, succeeded, and
failed transitions and reusable validator-approved results. Successful,
rejected, and failed runs may therefore all leave checkpoint state. The
`--resume` flag additionally loads compatible completed work before executing
missing or incompatible stages. Without `--resume`, a recording-enabled run
never loads checkpoints. Using `--resume` while recording is disabled is an
error.
Checkpoints use the selected root and the existing identity hierarchy:
@@ -102,6 +107,7 @@ For a Linux service account, independently provision:
```yaml
cache:
checkpoints:
enabled: true
directory: /var/cache/notarius/checkpoints
```

7
docs/policy/architecture.md
View File

@@ -198,9 +198,10 @@ lifecycle:
- debug is explicitly requested inspection data, combining a redacted summary
with a detailed trace.
Chunk plans are keyed only by canonical source digest. Checkpoints are used only
for an invocation that explicitly requests resume. Debug is never a cache input
and is never created without an explicit request. Pipeline modules receive
Chunk plans are keyed only by canonical source digest. Configured checkpoint
recording is independent of checkpoint reuse; checkpoints are loaded only for
an invocation that explicitly requests resume. Debug is never a cache input and
is never created without an explicit request. Pipeline modules receive
collaborator interfaces and never physical roots.
Writes are atomic where practical. Paths for writes, moves, overwrites, and

1
examples/dnd-spells-production.config.yml
View File

@@ -10,6 +10,7 @@ cache:
directory: /var/cache/notarius/chunk-plans
mode: auto
checkpoints:
enabled: false
directory: /var/cache/notarius/checkpoints
debug:
directory: ./notarius-debug

1
examples/dnd-spells.config.yml
View File

@@ -5,6 +5,7 @@ cache:
chunk_plans:
mode: bypass
checkpoints:
enabled: false
directory: ""
debug:
directory: ./notarius-debug

13
internal/cli/cache_contract_test.go
View File

@@ -296,8 +296,9 @@ func TestRunResumeSelectsConfiguredOrPerUserCheckpointRoot(t *testing.T) {
})
}
t.Run("without resume avoids checkpoint root resolution", func(t *testing.T) {
t.Run("disabled avoids checkpoint root resolution", func(t *testing.T) {
roots := newStateTestRoots(t)
replaceStateTestConfigLine(t, roots.config, " enabled: true\n", " enabled: false\n")
removeStateTestConfigLine(t, roots.config, fmt.Sprintf(" directory: %q\n", roots.checkpoints))
opts := newStateTestHarness().options()
opts.UserCacheDir = func() (string, error) { return "", errors.New("checkpoint cache must not be resolved") }
@@ -308,6 +309,16 @@ func TestRunResumeSelectsConfiguredOrPerUserCheckpointRoot(t *testing.T) {
assertStateTestOutput(t, roots.output)
assertAbsent(t, roots.checkpoints)
})
t.Run("resume requires enabled checkpoint recording", func(t *testing.T) {
roots := newStateTestRoots(t)
replaceStateTestConfigLine(t, roots.config, " enabled: true\n", " enabled: false\n")
result := runStateTest(t, roots, newStateTestHarness().options(), true, true, "bypass")
if result.code != 1 || !strings.Contains(result.stderr, "--resume requires cache.checkpoints.enabled: true") {
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
}
assertNoRunState(t, roots)
})
}
func TestConfigCommandsDoNotResolveRunState(t *testing.T) {

20
internal/cli/run.go
View File

@@ -135,7 +135,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
debug := fs.Bool("debug", false, "write a debug bundle")
debugDir := fs.String("debug-dir", "", "debug bundle directory")
llmProfile := fs.String("llm-profile", "", "LLM profile override")
resume := fs.Bool("resume", false, "reuse and record compatible checkpoints")
resume := fs.Bool("resume", false, "reuse compatible recorded checkpoints")
chunkCache := chunkCacheFlag{}
sessionID := sessionIDFlag{}
referenceFlags := stringListFlag{}
@@ -219,6 +219,10 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
if *resume && !cfg.Cache.Checkpoints.Enabled {
fmt.Fprintln(stderr, "notarius: --resume requires cache.checkpoints.enabled: true")
return 1
}
startedAt := opts.Now().UTC()
runID, err := opts.RunIDGenerator(startedAt)
@@ -437,7 +441,10 @@ func checkpointHandlersForRun(
sessionID string,
resume bool,
) (pipeline.CheckpointRecorder, pipeline.CheckpointLoader, error) {
if !resume {
if !settings.Enabled {
if resume {
return nil, nil, fmt.Errorf("--resume requires cache.checkpoints.enabled: true")
}
return pipeline.NoopCheckpointRecorder(), pipeline.NoopCheckpointLoader(), nil
}
identity, err := checkpoint.NewIdentity(checkpoint.IdentityInput{
@@ -463,9 +470,12 @@ func checkpointHandlersForRun(
if err != nil {
return nil, nil, fmt.Errorf("create checkpoint recorder: %w", err)
}
loader, err := checkpoint.NewFilesystemLoader(checkpointRoot, identity)
if err != nil {
return nil, nil, fmt.Errorf("create checkpoint loader: %w", err)
loader := pipeline.NoopCheckpointLoader()
if resume {
loader, err = checkpoint.NewFilesystemLoader(checkpointRoot, identity)
if err != nil {
return nil, nil, fmt.Errorf("create checkpoint loader: %w", err)
}
}
return recorder, loader, nil
}

17
internal/cli/state_hardening_test.go
View File

@@ -56,12 +56,8 @@ func TestRunStateSurfaceMatrix(t *testing.T) {
t.Fatalf("chunk plan store roots = %v, want [%q]", storeRoots, roots.plans)
}
}
if resume {
assertAnyFile(t, roots.checkpoints)
assertRestrictedTree(t, roots.checkpoints)
} else {
assertAbsent(t, roots.checkpoints)
}
assertAnyFile(t, roots.checkpoints)
assertRestrictedTree(t, roots.checkpoints)
if debug {
bundle := onlyChildDir(t, roots.debug)
assertFile(t, filepath.Join(bundle, "summary", "invocation.json"))
@@ -97,6 +93,9 @@ func TestRunKeepsStateRootsIndependentAndReusesSelectedCheckpointRoot(t *testing
if harness.chunkCalls != 1 {
t.Fatalf("chunk calls after debug toggle = %d, want 1", harness.chunkCalls)
}
if harness.extractCalls != 2 {
t.Fatalf("extract calls after two recording-only runs = %d, want 2", harness.extractCalls)
}
if got, err := os.ReadFile(planPath); err != nil || !bytes.Equal(got, initialPlan) {
t.Fatalf("chunk plan changed after debug toggle: %v", err)
}
@@ -105,10 +104,14 @@ func TestRunKeepsStateRootsIndependentAndReusesSelectedCheckpointRoot(t *testing
}
checkpointRoot := roots.checkpoints
extractCallsBeforeResume := harness.extractCalls
seed := runStateTest(t, roots, harness.options(), false, true, "auto")
if seed.code != 0 {
t.Fatalf("checkpoint seed code=%d stderr=%q", seed.code, seed.stderr)
}
if harness.extractCalls != extractCallsBeforeResume {
t.Fatalf("extract calls after reusing recording-only checkpoint = %d, want %d", harness.extractCalls, extractCallsBeforeResume)
}
extractCalls := harness.extractCalls
checkpointFiles := readTree(t, checkpointRoot)
reused := runStateTest(t, roots, harness.options(), false, true, "auto")
@@ -632,7 +635,7 @@ func newStateTestRoots(t *testing.T) stateTestRoots {
t.Fatal(err)
}
roots.config = filepath.Join(base, "config.yml")
config := fmt.Sprintf("version: 3\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input: test/input\n chunk: test/chunk\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n output: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
config := fmt.Sprintf("version: 3\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n enabled: true\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input: test/input\n chunk: test/chunk\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n output: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
t.Fatal(err)
}

1
internal/core/config/config.go
View File

@@ -43,6 +43,7 @@ type ChunkPlanCacheConfig struct {
}
type CheckpointCacheConfig struct {
Enabled bool `json:"enabled"`
Directory string `json:"directory,omitempty"`
}
type DebugConfig struct {

14
internal/core/config/file_config.go
View File

@@ -60,6 +60,7 @@ type FileChunkPlanCacheConfig struct {
Mode *string `yaml:"mode,omitempty"`
}
type FileCheckpointCacheConfig struct {
Enabled *bool `yaml:"enabled,omitempty"`
Directory *string `yaml:"directory,omitempty"`
}
type FileDebugConfig struct {
@@ -358,10 +359,15 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
}
}
}
if fileCfg.Cache.Checkpoints != nil && fileCfg.Cache.Checkpoints.Directory != nil {
c.Cache.Checkpoints.Directory = cleanOptionalPath(*fileCfg.Cache.Checkpoints.Directory)
if strings.ContainsRune(c.Cache.Checkpoints.Directory, '\x00') {
return fmt.Errorf("cache.checkpoints.directory must not contain NUL")
if fileCfg.Cache.Checkpoints != nil {
if fileCfg.Cache.Checkpoints.Enabled != nil {
c.Cache.Checkpoints.Enabled = *fileCfg.Cache.Checkpoints.Enabled
}
if fileCfg.Cache.Checkpoints.Directory != nil {
c.Cache.Checkpoints.Directory = cleanOptionalPath(*fileCfg.Cache.Checkpoints.Directory)
if strings.ContainsRune(c.Cache.Checkpoints.Directory, '\x00') {
return fmt.Errorf("cache.checkpoints.directory must not contain NUL")
}
}
}
}

29
internal/core/config/file_config_contract_test.go
View File

@@ -18,7 +18,7 @@ func TestDefaultReturnsDocumentedValuesAndIndependentMaps(t *testing.T) {
if first.Output.Directory != "./notarius-output" || first.Debug.Directory != "./notarius-debug" {
t.Fatalf("output/debug defaults = %#v, %#v", first.Output, first.Debug)
}
if first.Cache.ChunkPlans.Mode != pipeline.ChunkCacheAuto || first.Cache.ChunkPlans.Directory != "" || first.Cache.Checkpoints.Directory != "" {
if first.Cache.ChunkPlans.Mode != pipeline.ChunkCacheAuto || first.Cache.ChunkPlans.Directory != "" || first.Cache.Checkpoints.Enabled || first.Cache.Checkpoints.Directory != "" {
t.Fatalf("cache defaults = %#v", first.Cache)
}
if len(first.Pipelines) != 0 {
@@ -91,6 +91,16 @@ func TestFileConfigRejectsUnknownCurrentAndRemovedFields(t *testing.T) {
yaml: "version: 3\npipelines:\n main:\n input:\n module: seriatim\n unknown: true\n",
want: "field unknown not found in module binding",
},
{
name: "checkpoint field",
yaml: "version: 3\ncache:\n checkpoints:\n unknown: true\n",
want: "field unknown not found",
},
{
name: "checkpoint enabled type",
yaml: "version: 3\ncache:\n checkpoints:\n enabled: definitely\n",
want: "cannot unmarshal",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
@@ -279,6 +289,7 @@ cache:
directory: ./plans
mode: bypass
checkpoints:
enabled: true
directory: ./checkpoints
debug:
directory: ./debug
@@ -290,7 +301,7 @@ debug:
t.Fatalf("concurrency = %#v", cfg.Concurrency)
}
if cfg.Output.Directory != "./output" || cfg.Cache.ChunkPlans.Directory != "plans" || cfg.Cache.ChunkPlans.Mode != pipeline.ChunkCacheBypass ||
cfg.Cache.Checkpoints.Directory != "checkpoints" || cfg.Debug.Directory != "./debug" {
!cfg.Cache.Checkpoints.Enabled || cfg.Cache.Checkpoints.Directory != "checkpoints" || cfg.Debug.Directory != "./debug" {
t.Fatalf("state sections = %#v, %#v, %#v, %#v", cfg.Output, cfg.Cache, cfg.Debug, cfg.Scriptorium)
}
if cfg.Output.Directory == cfg.Cache.ChunkPlans.Directory || cfg.Cache.ChunkPlans.Directory == cfg.Cache.Checkpoints.Directory || cfg.Cache.Checkpoints.Directory == cfg.Debug.Directory {
@@ -298,6 +309,20 @@ debug:
}
}
func TestFileConfigCheckpointEnabledCanBeExplicitlyDisabled(t *testing.T) {
cfg := applyFileConfig(t, "version: 3\ncache:\n checkpoints:\n enabled: true\n")
if !cfg.Cache.Checkpoints.Enabled || !cloneConfig(cfg).Cache.Checkpoints.Enabled {
t.Fatalf("enabled checkpoint config was not retained: %#v", cfg.Cache.Checkpoints)
}
file := parseFileConfig(t, "version: 3\ncache:\n checkpoints:\n enabled: false\n")
if err := cfg.ApplyFileConfig(file); err != nil {
t.Fatal(err)
}
if cfg.Cache.Checkpoints.Enabled {
t.Fatalf("explicit false checkpoint config was not applied: %#v", cfg.Cache.Checkpoints)
}
}
func TestFileConfigRejectsTrimmedKeyCollisions(t *testing.T) {
tests := []struct {
name string