narratio/docs/roadmap/notarius-cache-reuse.md

# Notarius Cache Reuse and Artifact Development

Status: Accepted direction — implementation pending

## Goal and User Intent

Allow Narratio to reuse Notarius's cached chunk plans and accepted intermediate
artifacts when extraction actually executes, particularly when retrying failed
work. Preserve an explicit force path that regenerates the selected extraction
pipeline's work. Keep transcript generation separate from iterative artifact
development, and prominently explain the existing analysis-only command.

This roadmap defines the desired end state, not an implementation sequence.
The [implementation plan](implementation.md) owns staged execution and checks.
It builds on the local Notarius v0.7.0 CLI and cache contracts reviewed for this
proposal. No upstream changes are required by the proposed design.

## Current Behavior and Underlying Issue

Narratio's manifest-based resume can skip an already-completed extract stage.
When that stage must execute, however, the Notarius adapter never passes
`--resume`, so Notarius cannot load compatible execution checkpoints.

Notarius has two distinct mechanisms:

- Chunk-plan caching defaults to `auto`. Valid source-addressed plans can
  already be reused through Notarius configuration. `--chunk_cache` overrides
  the configured mode with `auto`, `refresh`, or `bypass`.
- Checkpoint recording is disabled by default and is enabled in Notarius with
  `cache.checkpoints.enabled: true`. Recording-enabled runs save accepted work
  even without `--resume`; that flag separately permits checkpoint loading.
  Asking for resume while recording is disabled fails.

There is also a reference-identity obstacle. Narratio supplies references via
invocation-local snapshots, whose paths change with each run ID. Notarius's
checkpoint identity includes reference origin URIs as well as content digests.
Simply adding `--resume` therefore does not establish reuse across otherwise
equivalent invocations that supply those references.

Relevant upstream owners are Notarius's `docs/cli.md`, `docs/config.md`, and
`docs/operations.md`; reference fingerprint construction is in
`internal/framework/checkpoint/identity.go`. Narratio's existing contracts are
documented in [Extract Internals](../internal/stage-extract.md) and the
[Notarius Integration](../integrations/notarius.md).

## Configuration and Invocation Contract

Add two optional fields to Narratio's Notarius section:

```yaml
notarius:
  resume: true
  chunk_cache: auto
```

| Field | Default | Meaning |
| --- | --- | --- |
| `notarius.resume` | `false` | Permit compatible Notarius checkpoint loading when extraction executes without force. |
| `notarius.chunk_cache` | unset | Optional chunk-plan mode override: `auto`, `refresh`, or `bypass`; unset preserves Notarius configuration. |

These are typed, validated configuration fields, not arbitrary CLI arguments.
They participate in existing profile composition, safe inspection, and source
reporting. Do not introduce another secret, configuration, or cache resolver.

Notarius continues to own checkpoint recording, cache locations, compatibility
checks, and cache formats. Operators enable recording and choose persistent
cache directories in Notarius configuration. Narratio neither reads that
configuration to infer settings nor writes a replacement configuration. A
misconfigured resume request remains an actionable downstream error; do not
silently retry without reuse, which could unexpectedly incur provider costs.

### Force Takes Precedence

| Extraction invocation | Checkpoint loading | Chunk-plan policy |
| --- | --- | --- |
| Ordinary execution, `resume: false` | No `--resume` | Configured Narratio override, or no flag |
| Ordinary execution, `resume: true` | Pass `--resume` | Configured Narratio override, or no flag |
| Forced extraction | Never pass `--resume` | Always pass `--chunk_cache refresh` |

Force regenerates the selected pipeline's chunk plan and objects rather than
hydrating accepted checkpoint results. Checkpoint recording may still save the
new work if enabled by Notarius. Force overrides even an explicit Narratio
`chunk_cache: bypass` selection; document this precedence clearly.

“Clear the cache” means bypass the current input's cached chunk plan and replace
it with a newly accepted plan, not recursively delete the Notarius cache root.
Notarius's refresh operation publishes only an accepted replacement. A failed
refresh need not erase the old plan. Other inputs' plans and unrelated checkpoint
identities are left alone. No global purge, transactional regeneration, provider-
side cache bypass, or guarantee of different generated text is implied.

Disabled extraction retains its ordinary explicit self-skip. The existing
force scope and failure propagation remain unchanged: a failed extraction does
not proceed to analysis.

## Stable Reference Snapshots

Use session-scoped, content-addressed reference snapshots at stable locations
outside invocation-local run directories. Equivalent references in the same
session must receive identical absolute snapshot paths across invocations;
changed bytes must select different snapshot identities. Central path helpers
own the layout, not adapter-local string construction.

Retain the useful protections of the existing snapshot mechanism:

- resolve inputs from manifest-backed prepared reference evidence;
- validate content against its recorded checksum and size;
- safely materialize missing snapshots and verify existing ones before reuse;
- retain confinement and post-invocation verification rather than silently
  trusting the cache or substituting mutable prepared files;
- use the existing session lock and narrow file-operation helpers, not a new
  general-purpose cache framework.

Snapshots are reconstructible local cache state, not canonical artifacts,
published outputs, or credentials. Apply Narratio's ordinary group-shareable
workspace policy. Preserve them across ordinary run cleanup so later extraction
can reuse their locations; do not add garbage collection in this feature.
Document their location, retention, and safe removal while no session operation
is active. Missing snapshots can be reconstructed from valid prepared inputs.

Old invocation-local snapshots and checkpoints require no migration. A first
execution under the stable layout may miss old checkpoint identities. Moving a
workspace may likewise cause cold misses; cross-host or relocated-path cache
identity is not promised.

## Two Distinct Resume Layers

Narratio remains responsible for deciding whether an existing complete extract
result is current. Notarius remains responsible for reusing compatible internal
work when an extraction process actually runs. Cached work must still produce
an ordinary successful bundle that passes Narratio's existing validation and
immutable promotion boundary.

The two new controls are operational settings, not blanket invalidation keys.
Changing them alone does not stale a valid completed extraction. Explicit force
is required to execute otherwise-current extraction with a fresh chunk plan.
Preserve existing reference-content and transcript invalidation and the rule
that changes hidden inside Notarius configuration require operator action.

Persistent upstream cache roots must not live inside disposable run output.
Recommend explicitly configured Notarius cache directories for service
deployments rather than assuming interactive-shell cache discovery is inherited
through Narratio's minimal subprocess environment. Do not broaden environment
inheritance or impose a new private-data policy on ordinary D&D material.

## Development-Focused Operator Guidance

Make the following distinction prominent at the beginning of the artifact
development workflow in [Operations](../operations.md), with links to the
[CLI Reference](../cli.md) for exact syntax:

| Operator intent | Existing command | Result |
| --- | --- | --- |
| Iterate on analysis prompts or artifacts using existing extracted data | `narratio analyze SESSION` | Force analysis only; preserve extract results and transcripts. |
| Rebuild extraction and analysis | `narratio regenerate-artifacts SESSION` | Force extract through analyze; refresh chunking and avoid checkpoint loading. |
| Continue incomplete artifact work with configured reuse | `narratio run SESSION --from extract --through analyze` | Reuse valid Narratio results; permit configured Notarius reuse when extraction executes. |

Explain `--artifacts` alongside these choices: it selects analysis targets and
their required analysis prerequisites, not a subset of the Notarius bundle.
The bounded commands require valid existing prefix-stage state; they do not
repair or rerun transcript generation automatically. Publishing and retention
remain separate actions.

`regenerate-artifacts` remains exactly a convenience expansion to
`run SESSION --force --from extract --through analyze`. Do not give it private
cache semantics. `analyze` remains the existing forced analysis-only command;
no new alias is needed.

## Required Changes and Acceptance

- Extend the configuration owner and safe projections for the two fields.
- Carry effective cache intent through extract's typed adapter request, using
  the existing force state; construct CLI flags only in the Notarius adapter.
- Replace invocation-dependent reference delivery paths with the stable snapshot
  design without weakening prepared-input or bundle validation.
- Update canonical configuration, integration, internal, example, and recovery
  documentation where owned. Put the workflow comparison front and center in
  Operations, not only in developer internals or release notes.
- Cover default/explicit mode mapping, invalid modes, force precedence, and
  disabled extraction with focused behavioral tests.
- Prove cross-invocation reuse with the supported Notarius behavior, including
  unchanged reference bytes at stable paths, changed reference content causing
  a miss, and retry after partial failure reusing accepted work.
- Prove forced extraction regenerates rather than reuses internal work, while
  analysis-only invocation never launches Notarius or changes its results.
- Verify cache loss is recoverable, run cleanup preserves reusable snapshots,
  and receipt validation, promotion, and failure handling remain intact.

Follow the [Testing Policy](../policy/testing.md): default tests remain offline
and credential-free. An argv-only fake cannot prove upstream checkpoint reuse;
use controlled upstream integration evidence with deterministic/local test
modules. Do not make the default suite require a developer's installed Notarius
or a paid provider, and do not duplicate Notarius's internal cache test matrix.

## Non-goals

- Notarius `--recompute-step` or lane-selective extraction.
- A new Narratio cache-management command, cache format, or cache garbage collector.
- Parsing downstream configuration or automatically enabling checkpoint recording.
- Global cache deletion or bypassing provider-side caches.
- A new release, upstream change, or migration of old cache contents in this task.
- New cross-process locking for independent Notarius users or distributed cache
  coordination; upstream cache concurrency limitations remain applicable.

## Target End State

Operators can cheaply continue incomplete extraction using compatible Notarius
work, deliberately regenerate the full extraction pipeline with force, or
iterate only on analysis using the existing `analyze` command. Stable reference
delivery makes reuse useful across Narratio invocations. Notarius owns its
caches; Narratio owns orchestration, reference snapshots, and validated durable
results. The operator guide makes the everyday development choices obvious.