Plan Notarius cache reuse

docs/roadmap/notarius-cache-reuse.md

@@ -0,0 +1,212 @@
+# Notarius Cache Reuse and Artifact Development
+
+Status: Accepted direction — implementation pending
+
+## Goal and User Intent
+
+Allow Narratio to reuse Notarius's cached chunk plans and accepted intermediate
+artifacts when extraction actually executes, particularly when retrying failed
+work. Preserve an explicit force path that regenerates the selected extraction
+pipeline's work. Keep transcript generation separate from iterative artifact
+development, and prominently explain the existing analysis-only command.
+
+This roadmap defines the desired end state, not an implementation sequence.
+The [implementation plan](implementation.md) owns staged execution and checks.
+It builds on the local Notarius v0.7.0 CLI and cache contracts reviewed for this
+proposal. No upstream changes are required by the proposed design.
+
+## Current Behavior and Underlying Issue
+
+Narratio's manifest-based resume can skip an already-completed extract stage.
+When that stage must execute, however, the Notarius adapter never passes
+`--resume`, so Notarius cannot load compatible execution checkpoints.
+
+Notarius has two distinct mechanisms:
+
+- Chunk-plan caching defaults to `auto`. Valid source-addressed plans can
+  already be reused through Notarius configuration. `--chunk_cache` overrides
+  the configured mode with `auto`, `refresh`, or `bypass`.
+- Checkpoint recording is disabled by default and is enabled in Notarius with
+  `cache.checkpoints.enabled: true`. Recording-enabled runs save accepted work
+  even without `--resume`; that flag separately permits checkpoint loading.
+  Asking for resume while recording is disabled fails.
+
+There is also a reference-identity obstacle. Narratio supplies references via
+invocation-local snapshots, whose paths change with each run ID. Notarius's
+checkpoint identity includes reference origin URIs as well as content digests.
+Simply adding `--resume` therefore does not establish reuse across otherwise
+equivalent invocations that supply those references.
+
+Relevant upstream owners are Notarius's `docs/cli.md`, `docs/config.md`, and
+`docs/operations.md`; reference fingerprint construction is in
+`internal/framework/checkpoint/identity.go`. Narratio's existing contracts are
+documented in [Extract Internals](../internal/stage-extract.md) and the
+[Notarius Integration](../integrations/notarius.md).
+
+## Configuration and Invocation Contract
+
+Add two optional fields to Narratio's Notarius section:
+
+```yaml
+notarius:
+  resume: true
+  chunk_cache: auto
+```
+
+| Field | Default | Meaning |
+| --- | --- | --- |
+| `notarius.resume` | `false` | Permit compatible Notarius checkpoint loading when extraction executes without force. |
+| `notarius.chunk_cache` | unset | Optional chunk-plan mode override: `auto`, `refresh`, or `bypass`; unset preserves Notarius configuration. |
+
+These are typed, validated configuration fields, not arbitrary CLI arguments.
+They participate in existing profile composition, safe inspection, and source
+reporting. Do not introduce another secret, configuration, or cache resolver.
+
+Notarius continues to own checkpoint recording, cache locations, compatibility
+checks, and cache formats. Operators enable recording and choose persistent
+cache directories in Notarius configuration. Narratio neither reads that
+configuration to infer settings nor writes a replacement configuration. A
+misconfigured resume request remains an actionable downstream error; do not
+silently retry without reuse, which could unexpectedly incur provider costs.
+
+### Force Takes Precedence
+
+| Extraction invocation | Checkpoint loading | Chunk-plan policy |
+| --- | --- | --- |
+| Ordinary execution, `resume: false` | No `--resume` | Configured Narratio override, or no flag |
+| Ordinary execution, `resume: true` | Pass `--resume` | Configured Narratio override, or no flag |
+| Forced extraction | Never pass `--resume` | Always pass `--chunk_cache refresh` |
+
+Force regenerates the selected pipeline's chunk plan and objects rather than
+hydrating accepted checkpoint results. Checkpoint recording may still save the
+new work if enabled by Notarius. Force overrides even an explicit Narratio
+`chunk_cache: bypass` selection; document this precedence clearly.
+
+“Clear the cache” means bypass the current input's cached chunk plan and replace
+it with a newly accepted plan, not recursively delete the Notarius cache root.
+Notarius's refresh operation publishes only an accepted replacement. A failed
+refresh need not erase the old plan. Other inputs' plans and unrelated checkpoint
+identities are left alone. No global purge, transactional regeneration, provider-
+side cache bypass, or guarantee of different generated text is implied.
+
+Disabled extraction retains its ordinary explicit self-skip. The existing
+force scope and failure propagation remain unchanged: a failed extraction does
+not proceed to analysis.
+
+## Stable Reference Snapshots
+
+Use session-scoped, content-addressed reference snapshots at stable locations
+outside invocation-local run directories. Equivalent references in the same
+session must receive identical absolute snapshot paths across invocations;
+changed bytes must select different snapshot identities. Central path helpers
+own the layout, not adapter-local string construction.
+
+Retain the useful protections of the existing snapshot mechanism:
+
+- resolve inputs from manifest-backed prepared reference evidence;
+- validate content against its recorded checksum and size;
+- safely materialize missing snapshots and verify existing ones before reuse;
+- retain confinement and post-invocation verification rather than silently
+  trusting the cache or substituting mutable prepared files;
+- use the existing session lock and narrow file-operation helpers, not a new
+  general-purpose cache framework.
+
+Snapshots are reconstructible local cache state, not canonical artifacts,
+published outputs, or credentials. Apply Narratio's ordinary group-shareable
+workspace policy. Preserve them across ordinary run cleanup so later extraction
+can reuse their locations; do not add garbage collection in this feature.
+Document their location, retention, and safe removal while no session operation
+is active. Missing snapshots can be reconstructed from valid prepared inputs.
+
+Old invocation-local snapshots and checkpoints require no migration. A first
+execution under the stable layout may miss old checkpoint identities. Moving a
+workspace may likewise cause cold misses; cross-host or relocated-path cache
+identity is not promised.
+
+## Two Distinct Resume Layers
+
+Narratio remains responsible for deciding whether an existing complete extract
+result is current. Notarius remains responsible for reusing compatible internal
+work when an extraction process actually runs. Cached work must still produce
+an ordinary successful bundle that passes Narratio's existing validation and
+immutable promotion boundary.
+
+The two new controls are operational settings, not blanket invalidation keys.
+Changing them alone does not stale a valid completed extraction. Explicit force
+is required to execute otherwise-current extraction with a fresh chunk plan.
+Preserve existing reference-content and transcript invalidation and the rule
+that changes hidden inside Notarius configuration require operator action.
+
+Persistent upstream cache roots must not live inside disposable run output.
+Recommend explicitly configured Notarius cache directories for service
+deployments rather than assuming interactive-shell cache discovery is inherited
+through Narratio's minimal subprocess environment. Do not broaden environment
+inheritance or impose a new private-data policy on ordinary D&D material.
+
+## Development-Focused Operator Guidance
+
+Make the following distinction prominent at the beginning of the artifact
+development workflow in [Operations](../operations.md), with links to the
+[CLI Reference](../cli.md) for exact syntax:
+
+| Operator intent | Existing command | Result |
+| --- | --- | --- |
+| Iterate on analysis prompts or artifacts using existing extracted data | `narratio analyze SESSION` | Force analysis only; preserve extract results and transcripts. |
+| Rebuild extraction and analysis | `narratio regenerate-artifacts SESSION` | Force extract through analyze; refresh chunking and avoid checkpoint loading. |
+| Continue incomplete artifact work with configured reuse | `narratio run SESSION --from extract --through analyze` | Reuse valid Narratio results; permit configured Notarius reuse when extraction executes. |
+
+Explain `--artifacts` alongside these choices: it selects analysis targets and
+their required analysis prerequisites, not a subset of the Notarius bundle.
+The bounded commands require valid existing prefix-stage state; they do not
+repair or rerun transcript generation automatically. Publishing and retention
+remain separate actions.
+
+`regenerate-artifacts` remains exactly a convenience expansion to
+`run SESSION --force --from extract --through analyze`. Do not give it private
+cache semantics. `analyze` remains the existing forced analysis-only command;
+no new alias is needed.
+
+## Required Changes and Acceptance
+
+- Extend the configuration owner and safe projections for the two fields.
+- Carry effective cache intent through extract's typed adapter request, using
+  the existing force state; construct CLI flags only in the Notarius adapter.
+- Replace invocation-dependent reference delivery paths with the stable snapshot
+  design without weakening prepared-input or bundle validation.
+- Update canonical configuration, integration, internal, example, and recovery
+  documentation where owned. Put the workflow comparison front and center in
+  Operations, not only in developer internals or release notes.
+- Cover default/explicit mode mapping, invalid modes, force precedence, and
+  disabled extraction with focused behavioral tests.
+- Prove cross-invocation reuse with the supported Notarius behavior, including
+  unchanged reference bytes at stable paths, changed reference content causing
+  a miss, and retry after partial failure reusing accepted work.
+- Prove forced extraction regenerates rather than reuses internal work, while
+  analysis-only invocation never launches Notarius or changes its results.
+- Verify cache loss is recoverable, run cleanup preserves reusable snapshots,
+  and receipt validation, promotion, and failure handling remain intact.
+
+Follow the [Testing Policy](../policy/testing.md): default tests remain offline
+and credential-free. An argv-only fake cannot prove upstream checkpoint reuse;
+use controlled upstream integration evidence with deterministic/local test
+modules. Do not make the default suite require a developer's installed Notarius
+or a paid provider, and do not duplicate Notarius's internal cache test matrix.
+
+## Non-goals
+
+- Notarius `--recompute-step` or lane-selective extraction.
+- A new Narratio cache-management command, cache format, or cache garbage collector.
+- Parsing downstream configuration or automatically enabling checkpoint recording.
+- Global cache deletion or bypassing provider-side caches.
+- A new release, upstream change, or migration of old cache contents in this task.
+- New cross-process locking for independent Notarius users or distributed cache
+  coordination; upstream cache concurrency limitations remain applicable.
+
+## Target End State
+
+Operators can cheaply continue incomplete extraction using compatible Notarius
+work, deliberately regenerate the full extraction pipeline with force, or
+iterate only on analysis using the existing `analyze` command. Stable reference
+delivery makes reuse useful across Narratio invocations. Notarius owns its
+caches; Narratio owns orchestration, reference snapshots, and validated durable
+results. The operator guide makes the everyday development choices obvious.