narratio/docs/roadmap/implementation.md

# Notarius Cache Reuse Implementation Plan

Status: Not started

Implement the [feature roadmap](notarius-cache-reuse.md) in the numbered order
below. Each stage is a bounded assignment for a gpt-5.6-terra coding agent.
Record completion and checks beside each stage; intermediate stages are not a
claim that the whole feature is ready. No release, tag, push, upstream edit, or
dependency upgrade is authorized by this plan.

## Instructions and Settled Decisions

Read `docs/development.md`, every document in `docs/policy/`, the feature
roadmap, and the sources listed for the current stage. Inspect callers and
existing tests before editing. File lists identify owners, not edit allowlists.
Use existing path, file-operation, configuration, and subprocess boundaries.
Do not create a general cache framework or change unrelated safety policies.

- Add `Resume bool` and `ChunkCache string` to `config.NotariusConfig`, using
  YAML/JSON names `resume` and `chunk_cache` with `omitempty`. False and the empty
  string are the unset defaults. Null resets to the zero value through existing
  ConfigKit behavior. For enabled Notarius configuration, trim surrounding
  whitespace and accept only empty, `auto`, `bypass`, or `refresh`; do not add
  case-insensitive aliases. Preserve the existing disabled-section validation
  policy rather than introducing new mandatory configuration checks there.
- Ordinary extraction passes the configured resume boolean and chunk mode.
  Forced extraction always passes resume false and chunk mode `refresh`, even
  if configuration requests bypass. The stage resolves force precedence; the
  adapter translates effective intent into flags. No private alias behavior.
- Use stable snapshot paths under
  `<session-root>/cache/notarius/references/<sha256>/<prepared-basename>`.
  SHA-256 is the full lowercase hexadecimal digest of the reference bytes.
  The basename preserves the prepared file extension and is validated as one
  path component. Identity contains neither run ID nor timestamps. Different
  sources with identical bytes and basename can share a verified snapshot.
  Same basename with different bytes cannot collide. This is an implementation
  layout, not a new external artifact schema.
- Keep prepared-input identities, metadata, and extract resume fingerprints
  unchanged. Snapshot paths are delivery details. Neither new setting belongs
  in semantic resume evidence; safe effective configuration can expose them.
- A missing snapshot is reconstructed. A corrupt regular snapshot is atomically
  replaced from a currently verified prepared source. Unsafe paths, symlinks,
  directories, or other non-regular conflicts fail with the affected path and
  recovery guidance, never recursive deletion. A prepared-source mismatch
  remains an error, not a cache miss. No eviction or cache ledger is needed.
- Snapshots are ordinary group-shareable workspace files, not API-key material.
  They survive ordinary run cleanup, are not published/restored as artifacts,
  and do not become prerequisites for reuse of a complete durable extract bundle.
- Notarius owns checkpoint enablement, roots, recording, and validity. Do not
  read its config to infer enablement, silently retry without resume, or widen
  inherited environment. Recommend explicit persistent upstream cache roots.
- Cache refresh is not deletion: regenerate the current input's plan and let
  Notarius publish an accepted replacement. Leave other plans/checkpoints alone.
  No guarantee of different text, provider-side cache bypass, or transactional
  regeneration is introduced.
- The real integration acceptance baseline is Notarius v0.7.0. Do not add a
  runtime version probe or reject older compatible executables just by version.
  Unsupported flags fail through the existing subprocess error path.

## Stage 1 — Add Typed Cache Controls

Status: Complete (2026-09-16)

Completion: Added typed, zero-default `resume` and `chunk_cache` controls with
enabled-section normalization and validation, safe YAML/JSON projection, and
ConfigKit profile/provenance coverage. Verified with
`GOWORK=off go test ./internal/config` and `GOWORK=off go test ./...`.

Read: `internal/config/config.go`, `credential_projection.go`,
`pipeline_normalize.go`, `validate_notarius.go`, `notarius_test.go`,
`pipeline_configkit_test.go`, `effective_projection.go`; configuration sections
of `docs/config.md` and `docs/internal/configuration.md`.

1. Add the two fields and implement the zero-value, normalization, and validation
   decisions above. Keep defaults centralized; no stage-specific environment
   overrides or raw extra-argument map.
2. Extend `NotariusConfig.MarshalYAML` so direct serialization retains both
   fields and still projects only the credential reference name. Confirm that
   effective projection, source reporting, copying, and profile overrides carry
   the fields without adding another projection layer.
3. Extend focused configuration tests for absence, explicit modes, invalid
   enabled modes, null reset, profile replacement, and safe YAML/JSON inspection.
   Protect the existing optional credential behavior. Do not snapshot complete
   error wording or repeat every case at the CLI layer.

Acceptance: `GOWORK=off go test ./internal/config` and
`GOWORK=off go test ./...` pass. Existing configurations require no edits.
This stage alone does not enable delivery and is not a finished feature.

## Stage 2 — Forward Cache Intent and Apply Force Precedence

Status: Complete (2026-09-16)

Completion: Carried typed cache intent through extract requests, applied force
precedence in stage orchestration, mapped validated effective values to exact
Notarius flags, and confirmed both controls remain outside semantic resume
evidence. Verified normal and race tests for `./internal/stage` and
`./internal/adapters/notarius`.

Read: `internal/stage/extract.go`, `stage.go`, `extract_test.go`,
`semantic_contracts_delivery.go`, `semantic_contracts_delivery_test.go`;
`internal/adapters/notarius/runner.go`, `subprocess.go`, `subprocess_test.go`,
`fake.go`; `docs/internal/adapters.md`, `docs/integrations/notarius.md`.

1. Add `Resume bool` and `ChunkCache string` to the typed Notarius request.
   Populate them in extract, applying `env.Force` precedence in one place.
2. Append `--resume` only for true and `--chunk_cache <mode>` only for nonempty
   mode. Preserve existing arguments, references, credentials, diagnostics,
   receipt parsing, and bundle discovery. Validate programmatic nonempty modes
   before starting a subprocess; do not import application config into the
   adapter merely to share a trivial enum check.
3. Test the effective request matrix with ordinary and forced execution,
   including force overriding explicit bypass and resume. Adapter tests own the
   exact flag mapping and rejection of invalid direct requests. Keep a test for
   absent settings producing the previous ordinary argv. Disabled extraction
   still never invokes the adapter.
4. Add a focused semantic-fingerprint exclusion test for the two controls; do
   not bump the fingerprint version or change manifest schema or invalidation.

Acceptance: focused normal and race tests for `./internal/stage` and
`./internal/adapters/notarius` pass. No cache deletion or upstream config parsing
has been introduced. Existing credential-redaction tests remain green.

## Stage 3 — Replace Run-Local Reference Snapshots with Stable Verified Snapshots

Status: Complete (2026-09-16)

Completion: Replaced run-ID reference copies with confined, content-addressed
session snapshots; added canonical cache path ownership, verified reuse,
reconstruction and regular-file repair, unsafe-conflict failures, narrow cleanup
of invalid new writes, and shared post-run verification. Removed the obsolete
run-local reference helper. Verified the required artifacts/stage/fileops/
pathsafe tests and race tests for artifacts/stage.

Read: `internal/artifacts/paths.go` and its tests;
`internal/stage/extract_references.go`, `extract.go`, `extract_resume.go`,
`extract_test.go` and reference-focused tests; existing root-confined helpers in
`internal/pathsafe/` and atomic/checksum helpers in `internal/fileops/`;
`docs/internal/artifacts.md`, `docs/internal/workspace.md`.

1. Add canonical artifact/path-owner helpers for the cache root and confined
   snapshot destination. Reuse the package's existing path-segment conventions
   and central constants; validate digest syntax before using it as a segment.
   Do not embed new layout construction independently in materialization and
   post-run verification.
2. Refactor `materializeExtractReferenceSnapshots` to resolve stable destinations.
   Verify an existing regular file's size and SHA-256 before reuse. For missing
   or corrupt regular entries, copy from the manifest-verified source using the
   existing atomic/checksum primitive, checking the result before invocation.
   Deduplicate equivalent destinations within the request. Check path components
   before directory creation/opening using existing confinement facilities.
3. If a newly written entry fails verification because its source changed, do
   not invoke Notarius. Remove only that invalid, newly produced cache file
   through a confined operation if safe; do not remove the root or other entries.
   Preserve an actionable prepared-source error.
4. Make post-run verification use the same destination resolver and reject
   mutated, missing, symlinked, or non-regular snapshots before bundle promotion.
   Continue checking actual content, not just names or modification times.
5. Remove obsolete runtime use of the run-ID snapshot path helper. Remove the
   helper if no callers remain; no compatibility shim or migration is needed.
   Leave old run files alone. Do not change current prepared-reference metadata,
   recorded source paths, semantic evidence, or durable bundle locations.
6. Adapt existing reference tests rather than keeping parallel old/new suites.
   Cover stable paths across run IDs, changed content, same basename/different
   content, verified reuse, reconstruction after deletion, corrupt regular-file
   repair, unsafe conflicts, and post-invocation mutation rejection. Assert
   behavior and confinement, not incidental helper choreography.

Acceptance: `GOWORK=off go test ./internal/artifacts ./internal/stage
./internal/fileops ./internal/pathsafe` (on one shell line) and race tests for
artifacts/stage pass. No new general cache abstraction, lock service, retention
policy, or permission regime was added.

## Stage 4 — Verify Session Lifecycle and Command Boundaries

Status: Not started

Read: `internal/app/extract_lifecycle_test.go`, `regenerate_artifacts.go`,
`regenerate_artifacts_test.go`, `semantic_resume_test.go`,
`cleanup_orchestration_test.go`; `internal/stage/cleanup_plan.go`,
`cleanup_test.go`; publish/restore enumeration owners identified through their
callers; `docs/operations.md` and `docs/cli.md`.

1. Extend the existing lifecycle fixture to prove a failed extraction followed
   by a new invocation delivers identical snapshot paths for unchanged prepared
   references. A changed valid reference selects a new path. Keep old snapshots
   available; they are not current artifact evidence.
2. Verify a successful complete extract remains reusable after cache-control
   changes or deletion of its local snapshots. No snapshot directory should be
   created merely to plan or skip extraction.
3. Exercise ordinary successful cleanup with snapshots present: run-local
   output is removed as before while snapshots survive. Confirm publish and
   restore selection do not include the new cache subtree; add a narrow
   regression where existing enumeration could accidentally include it, not a
   second publish/restore test framework.
4. Extend existing command/lifecycle tests to show `analyze SESSION` never
   invokes Notarius and preserves extract outputs, whereas
   `regenerate-artifacts SESSION` uses the ordinary forced extract/analyze path.
   `--artifacts` continues to narrow analysis, not the extraction bundle.
5. Confirm errors still stop before analysis and leave no partially promoted
   current bundle. Reuse existing failure/promotion coverage where sufficient.

Acceptance: normal and race tests for `./internal/app ./internal/stage` pass.
No new alias semantics, CLI options, manifest schema, or unrelated lifecycle
changes. This stage proves Narratio orchestration, not upstream cache hits.

## Stage 5 — Prove Reuse with Real Notarius and a Local Test Provider

Status: Not started

Read Notarius v0.7.0 `docs/cli.md`, `docs/config.md`, `docs/operations.md`,
`examples/dnd-minimal.config.yml`, `examples/seriatim-minimal-transcript.json`,
`internal/cli/production_contract_test.go` (local-backend example), and
`internal/framework/checkpoint/identity.go`. Inspect the selected built-in
module's schema and reference slots rather than inventing output payloads.
Read Narratio's existing extraction lifecycle and adapter test fixtures.

1. Add an explicitly opt-in Go integration test, invoked with an absolute
   `NARRATIO_TEST_NOTARIUS_BINARY` path. If unset, skip with a clear reason; if
   set but unusable, fail. Document the focused `go test -run` invocation.
   The normal suite must not discover Notarius on PATH, build/download upstream,
   or depend on the sibling checkout. Do not add Notarius to `go.mod`.
2. Run the real executable with a small built-in D&D pipeline and its ordinary
   JSON output module. Use a local `httptest` OpenAI-compatible chat-completions
   endpoint via Notarius's `promptkit.local_backend`, a `backend: local` profile,
   and schema-valid deterministic responses. No real provider, credentials, or
   upstream test-only module registration is necessary. Use explicit temporary
   config, profile, cache, and output roots; keep endpoint/profile content stable
   across the compared invocations because they affect checkpoint identity.
3. Drive production Narratio snapshot delivery and the production adapter from
   the fixture, not a hand-assembled imitation of their argv. Use different run
   IDs/output roots and at least one declared prepared reference. Count local
   model requests and inspect upstream checkpoint decision codes: first execution
   populates state; second ordinary resume reuses accepted work; changing valid
   reference bytes causes execution rather than a false hit.
4. Add a deterministic partial-failure scenario with accepted predecessor work:
   configure ordered work and return a controlled failure for a later producer,
   including any bounded retries. Keep configuration and endpoint unchanged on
   retry; restore only server behavior. Verify accepted earlier work is reused
   and the failed work is executed. Do not infer reuse from equal output bytes.
5. Exercise forced extraction against populated state. Verify checkpoint loading
   is disabled and work executes again. Demonstrate actual chunk-plan refresh
   through Notarius's chunk provenance/cache behavior in addition to the adapter
   flag test; deterministic plans need not differ in bytes. Check that an
   unrelated cached input remains untouched.
6. Run against a locally built v0.7.0 executable (or a documented compatible
   newer version) without modifying or tagging the upstream repository. Record
   version, focused command, and results in this stage's completion note. Keep
   test fixtures small and scoped; do not reproduce upstream's whole cache suite.

Acceptance: the opt-in test passes without internet/provider calls, ordinary
`go test ./...` remains independent of upstream installation, and there is real
evidence of cache reuse across Narratio invocations. A skipped integration test
does not complete this stage. If upstream behavior blocks the stated contract,
record the evidence and stop for direction rather than weakening the test or
silently modifying upstream.

## Stage 6 — Document the Workflow and Complete Validation

Status: Not started

Read: `docs/operations.md`, `docs/cli.md`, `docs/config.md`,
`docs/integrations/notarius.md`, `docs/internal/stage-extract.md`,
`docs/internal/adapters.md`, `docs/internal/workspace.md`,
`docs/troubleshooting.md`, `examples/README.md`, `docs/release.md`.

1. Put the roadmap's three-way development workflow comparison prominently at
   the start of Operations' artifact-development section: analysis only,
   extraction plus analysis regeneration, and ordinary bounded continuation.
   Explain preservation of transcripts/extraction, analysis selection, and
   separate publishing. Link to CLI-owned exact syntax and avoid redefining it.
2. Update field/default/force precedence documentation and a maintained example.
   Clearly explain upstream checkpoint enablement, explicit persistent roots,
   cold misses from changed references/profile/paths, and the distinction between
   Narratio skipping a stage and Notarius reusing work inside an executed stage.
3. Document stable snapshot location/retention, regular corruption recovery,
   unsafe-path errors, and manual removal only while the session is inactive.
   Do not add a cache purge command, auto-GC, mandatory private permissions, or
   promise that all changed configuration is visible to Narratio resume.
4. Record the tested Notarius integration baseline and opt-in test instructions.
   Preserve already-published release notes; leave version choice and release
   publication to a separately authorized task.
5. Run the checks below. Mark the roadmap and stages complete only when all
   required evidence, including Stage 5, is present. Record any environmental
   limitations accurately; do not wait for CI or assert native macOS validation
   based on cross-compilation.

```sh
GOWORK=off go mod tidy -diff
GOWORK=off go mod verify
GOWORK=off go test -count=1 ./...
GOWORK=off go test -race -count=1 ./...
GOWORK=off go vet ./...
GOWORK=off go build ./...
GOWORK=off go test ./internal/doccheck
GOWORK=off go test ./internal/config -run '^TestExamplesLoadAndValidate$'
git diff --check
```

Check changed Go files with `gofmt -l`. Run the repository's
`scripts/build-release-assets.sh v0.0.0-test OUTPUT_DIR` into an empty absolute
directory created with `mktemp -d` to check the three maintained targets, without
publishing assets or tags. Keep any temporary binaries outside the worktree.

## Open Questions

None requiring an operator decision. Defaults, force precedence, cache ownership,
snapshot lifetime, and the command distinction are settled. The exact test
payloads are implementation details governed by the real upstream schemas, not
new product policy. Stage 5 must establish actual reuse before completion.