narratio/docs/roadmap/notarius-cache-reuse.md

Notarius Cache Reuse and Artifact Development

Status: Accepted direction — implementation pending

Goal and User Intent

Allow Narratio to reuse Notarius's cached chunk plans and accepted intermediate artifacts when extraction actually executes, particularly when retrying failed work. Preserve an explicit force path that regenerates the selected extraction pipeline's work. Keep transcript generation separate from iterative artifact development, and prominently explain the existing analysis-only command.

This roadmap defines the desired end state, not an implementation sequence. The implementation plan owns staged execution and checks. It builds on the local Notarius v0.7.0 CLI and cache contracts reviewed for this proposal. No upstream changes are required by the proposed design.

Current Behavior and Underlying Issue

Narratio's manifest-based resume can skip an already-completed extract stage. When that stage must execute, however, the Notarius adapter never passes --resume, so Notarius cannot load compatible execution checkpoints.

Notarius has two distinct mechanisms:

• Chunk-plan caching defaults to auto. Valid source-addressed plans can already be reused through Notarius configuration. --chunk_cache overrides the configured mode with auto, refresh, or bypass.
• Checkpoint recording is disabled by default and is enabled in Notarius with cache.checkpoints.enabled: true. Recording-enabled runs save accepted work even without --resume; that flag separately permits checkpoint loading. Asking for resume while recording is disabled fails.

There is also a reference-identity obstacle. Narratio supplies references via invocation-local snapshots, whose paths change with each run ID. Notarius's checkpoint identity includes reference origin URIs as well as content digests. Simply adding --resume therefore does not establish reuse across otherwise equivalent invocations that supply those references.

Relevant upstream owners are Notarius's docs/cli.md, docs/config.md, and docs/operations.md; reference fingerprint construction is in internal/framework/checkpoint/identity.go. Narratio's existing contracts are documented in Extract Internals and the Notarius Integration.

Configuration and Invocation Contract

Add two optional fields to Narratio's Notarius section:

notarius:
  resume: true
  chunk_cache: auto

Field	Default	Meaning
notarius.resume	false	Permit compatible Notarius checkpoint loading when extraction executes without force.
notarius.chunk_cache	unset	Optional chunk-plan mode override: auto, refresh, or bypass; unset preserves Notarius configuration.

These are typed, validated configuration fields, not arbitrary CLI arguments. They participate in existing profile composition, safe inspection, and source reporting. Do not introduce another secret, configuration, or cache resolver.

Notarius continues to own checkpoint recording, cache locations, compatibility checks, and cache formats. Operators enable recording and choose persistent cache directories in Notarius configuration. Narratio neither reads that configuration to infer settings nor writes a replacement configuration. A misconfigured resume request remains an actionable downstream error; do not silently retry without reuse, which could unexpectedly incur provider costs.

Force Takes Precedence

Extraction invocation	Checkpoint loading	Chunk-plan policy
Ordinary execution, resume: false	No --resume	Configured Narratio override, or no flag
Ordinary execution, resume: true	Pass --resume	Configured Narratio override, or no flag
Forced extraction	Never pass --resume	Always pass --chunk_cache refresh

Force regenerates the selected pipeline's chunk plan and objects rather than hydrating accepted checkpoint results. Checkpoint recording may still save the new work if enabled by Notarius. Force overrides even an explicit Narratio chunk_cache: bypass selection; document this precedence clearly.

“Clear the cache” means bypass the current input's cached chunk plan and replace it with a newly accepted plan, not recursively delete the Notarius cache root. Notarius's refresh operation publishes only an accepted replacement. A failed refresh need not erase the old plan. Other inputs' plans and unrelated checkpoint identities are left alone. No global purge, transactional regeneration, provider- side cache bypass, or guarantee of different generated text is implied.

Disabled extraction retains its ordinary explicit self-skip. The existing force scope and failure propagation remain unchanged: a failed extraction does not proceed to analysis.

Stable Reference Snapshots

Use session-scoped, content-addressed reference snapshots at stable locations outside invocation-local run directories. Equivalent references in the same session must receive identical absolute snapshot paths across invocations; changed bytes must select different snapshot identities. Central path helpers own the layout, not adapter-local string construction.

Retain the useful protections of the existing snapshot mechanism:

• resolve inputs from manifest-backed prepared reference evidence;
• validate content against its recorded checksum and size;
• safely materialize missing snapshots and verify existing ones before reuse;
• retain confinement and post-invocation verification rather than silently trusting the cache or substituting mutable prepared files;
• use the existing session lock and narrow file-operation helpers, not a new general-purpose cache framework.

Snapshots are reconstructible local cache state, not canonical artifacts, published outputs, or credentials. Apply Narratio's ordinary group-shareable workspace policy. Preserve them across ordinary run cleanup so later extraction can reuse their locations; do not add garbage collection in this feature. Document their location, retention, and safe removal while no session operation is active. Missing snapshots can be reconstructed from valid prepared inputs.

Old invocation-local snapshots and checkpoints require no migration. A first execution under the stable layout may miss old checkpoint identities. Moving a workspace may likewise cause cold misses; cross-host or relocated-path cache identity is not promised.

Two Distinct Resume Layers

Narratio remains responsible for deciding whether an existing complete extract result is current. Notarius remains responsible for reusing compatible internal work when an extraction process actually runs. Cached work must still produce an ordinary successful bundle that passes Narratio's existing validation and immutable promotion boundary.

The two new controls are operational settings, not blanket invalidation keys. Changing them alone does not stale a valid completed extraction. Explicit force is required to execute otherwise-current extraction with a fresh chunk plan. Preserve existing reference-content and transcript invalidation and the rule that changes hidden inside Notarius configuration require operator action.

Persistent upstream cache roots must not live inside disposable run output. Recommend explicitly configured Notarius cache directories for service deployments rather than assuming interactive-shell cache discovery is inherited through Narratio's minimal subprocess environment. Do not broaden environment inheritance or impose a new private-data policy on ordinary D&D material.

Development-Focused Operator Guidance

Make the following distinction prominent at the beginning of the artifact development workflow in Operations, with links to the CLI Reference for exact syntax:

Operator intent	Existing command	Result
Iterate on analysis prompts or artifacts using existing extracted data	narratio analyze SESSION	Force analysis only; preserve extract results and transcripts.
Rebuild extraction and analysis	narratio regenerate-artifacts SESSION	Force extract through analyze; refresh chunking and avoid checkpoint loading.
Continue incomplete artifact work with configured reuse	narratio run SESSION --from extract --through analyze	Reuse valid Narratio results; permit configured Notarius reuse when extraction executes.

Explain --artifacts alongside these choices: it selects analysis targets and their required analysis prerequisites, not a subset of the Notarius bundle. The bounded commands require valid existing prefix-stage state; they do not repair or rerun transcript generation automatically. Publishing and retention remain separate actions.

regenerate-artifacts remains exactly a convenience expansion to run SESSION --force --from extract --through analyze. Do not give it private cache semantics. analyze remains the existing forced analysis-only command; no new alias is needed.

Required Changes and Acceptance

• Extend the configuration owner and safe projections for the two fields.
• Carry effective cache intent through extract's typed adapter request, using the existing force state; construct CLI flags only in the Notarius adapter.
• Replace invocation-dependent reference delivery paths with the stable snapshot design without weakening prepared-input or bundle validation.
• Update canonical configuration, integration, internal, example, and recovery documentation where owned. Put the workflow comparison front and center in Operations, not only in developer internals or release notes.
• Cover default/explicit mode mapping, invalid modes, force precedence, and disabled extraction with focused behavioral tests.
• Prove cross-invocation reuse with the supported Notarius behavior, including unchanged reference bytes at stable paths, changed reference content causing a miss, and retry after partial failure reusing accepted work.
• Prove forced extraction regenerates rather than reuses internal work, while analysis-only invocation never launches Notarius or changes its results.
• Verify cache loss is recoverable, run cleanup preserves reusable snapshots, and receipt validation, promotion, and failure handling remain intact.

Follow the Testing Policy: default tests remain offline and credential-free. An argv-only fake cannot prove upstream checkpoint reuse; use controlled upstream integration evidence with deterministic/local test modules. Do not make the default suite require a developer's installed Notarius or a paid provider, and do not duplicate Notarius's internal cache test matrix.

Non-goals

• Notarius --recompute-step or lane-selective extraction.
• A new Narratio cache-management command, cache format, or cache garbage collector.
• Parsing downstream configuration or automatically enabling checkpoint recording.
• Global cache deletion or bypassing provider-side caches.
• A new release, upstream change, or migration of old cache contents in this task.
• New cross-process locking for independent Notarius users or distributed cache coordination; upstream cache concurrency limitations remain applicable.

Target End State

Operators can cheaply continue incomplete extraction using compatible Notarius work, deliberately regenerate the full extraction pipeline with force, or iterate only on analysis using the existing analyze command. Stable reference delivery makes reuse useful across Narratio invocations. Notarius owns its caches; Narratio owns orchestration, reference snapshots, and validated durable results. The operator guide makes the everyday development choices obvious.