diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 95f6aed..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,337 +0,0 @@ -# Notarius Cache Reuse Implementation Plan - -Status: Complete (2026-09-16) - -Implement the [feature roadmap](notarius-cache-reuse.md) in the numbered order -below. Each stage is a bounded assignment for a gpt-5.6-terra coding agent. -Record completion and checks beside each stage; intermediate stages are not a -claim that the whole feature is ready. No release, tag, push, upstream edit, or -dependency upgrade is authorized by this plan. - -## Instructions and Settled Decisions - -Read `docs/development.md`, every document in `docs/policy/`, the feature -roadmap, and the sources listed for the current stage. Inspect callers and -existing tests before editing. File lists identify owners, not edit allowlists. -Use existing path, file-operation, configuration, and subprocess boundaries. -Do not create a general cache framework or change unrelated safety policies. - -- Add `Resume bool` and `ChunkCache string` to `config.NotariusConfig`, using - YAML/JSON names `resume` and `chunk_cache` with `omitempty`. False and the empty - string are the unset defaults. Null resets to the zero value through existing - ConfigKit behavior. For enabled Notarius configuration, trim surrounding - whitespace and accept only empty, `auto`, `bypass`, or `refresh`; do not add - case-insensitive aliases. Preserve the existing disabled-section validation - policy rather than introducing new mandatory configuration checks there. -- Ordinary extraction passes the configured resume boolean and chunk mode. - Forced extraction always passes resume false and chunk mode `refresh`, even - if configuration requests bypass. The stage resolves force precedence; the - adapter translates effective intent into flags. No private alias behavior. -- Use stable snapshot paths under - `/cache/notarius/references//`. - SHA-256 is the full lowercase hexadecimal digest of the reference bytes. - The basename preserves the prepared file extension and is validated as one - path component. Identity contains neither run ID nor timestamps. Different - sources with identical bytes and basename can share a verified snapshot. - Same basename with different bytes cannot collide. This is an implementation - layout, not a new external artifact schema. -- Keep prepared-input identities, metadata, and extract resume fingerprints - unchanged. Snapshot paths are delivery details. Neither new setting belongs - in semantic resume evidence; safe effective configuration can expose them. -- A missing snapshot is reconstructed. A corrupt regular snapshot is atomically - replaced from a currently verified prepared source. Unsafe paths, symlinks, - directories, or other non-regular conflicts fail with the affected path and - recovery guidance, never recursive deletion. A prepared-source mismatch - remains an error, not a cache miss. No eviction or cache ledger is needed. -- Snapshots are ordinary group-shareable workspace files, not API-key material. - They survive ordinary run cleanup, are not published/restored as artifacts, - and do not become prerequisites for reuse of a complete durable extract bundle. -- Notarius owns checkpoint enablement, roots, recording, and validity. Do not - read its config to infer enablement, silently retry without resume, or widen - inherited environment. Recommend explicit persistent upstream cache roots. -- Cache refresh is not deletion: regenerate the current input's plan and let - Notarius publish an accepted replacement. Leave other plans/checkpoints alone. - No guarantee of different text, provider-side cache bypass, or transactional - regeneration is introduced. -- The real integration acceptance baseline is Notarius v0.7.0. Do not add a - runtime version probe or reject older compatible executables just by version. - Unsupported flags fail through the existing subprocess error path. - -## Stage 1 — Add Typed Cache Controls - -Status: Complete (2026-09-16) - -Completion: Added typed, zero-default `resume` and `chunk_cache` controls with -enabled-section normalization and validation, safe YAML/JSON projection, and -ConfigKit profile/provenance coverage. Verified with -`GOWORK=off go test ./internal/config` and `GOWORK=off go test ./...`. - -Read: `internal/config/config.go`, `credential_projection.go`, -`pipeline_normalize.go`, `validate_notarius.go`, `notarius_test.go`, -`pipeline_configkit_test.go`, `effective_projection.go`; configuration sections -of `docs/config.md` and `docs/internal/configuration.md`. - -1. Add the two fields and implement the zero-value, normalization, and validation - decisions above. Keep defaults centralized; no stage-specific environment - overrides or raw extra-argument map. -2. Extend `NotariusConfig.MarshalYAML` so direct serialization retains both - fields and still projects only the credential reference name. Confirm that - effective projection, source reporting, copying, and profile overrides carry - the fields without adding another projection layer. -3. Extend focused configuration tests for absence, explicit modes, invalid - enabled modes, null reset, profile replacement, and safe YAML/JSON inspection. - Protect the existing optional credential behavior. Do not snapshot complete - error wording or repeat every case at the CLI layer. - -Acceptance: `GOWORK=off go test ./internal/config` and -`GOWORK=off go test ./...` pass. Existing configurations require no edits. -This stage alone does not enable delivery and is not a finished feature. - -## Stage 2 — Forward Cache Intent and Apply Force Precedence - -Status: Complete (2026-09-16) - -Completion: Carried typed cache intent through extract requests, applied force -precedence in stage orchestration, mapped validated effective values to exact -Notarius flags, and confirmed both controls remain outside semantic resume -evidence. Verified normal and race tests for `./internal/stage` and -`./internal/adapters/notarius`. - -Read: `internal/stage/extract.go`, `stage.go`, `extract_test.go`, -`semantic_contracts_delivery.go`, `semantic_contracts_delivery_test.go`; -`internal/adapters/notarius/runner.go`, `subprocess.go`, `subprocess_test.go`, -`fake.go`; `docs/internal/adapters.md`, `docs/integrations/notarius.md`. - -1. Add `Resume bool` and `ChunkCache string` to the typed Notarius request. - Populate them in extract, applying `env.Force` precedence in one place. -2. Append `--resume` only for true and `--chunk_cache ` only for nonempty - mode. Preserve existing arguments, references, credentials, diagnostics, - receipt parsing, and bundle discovery. Validate programmatic nonempty modes - before starting a subprocess; do not import application config into the - adapter merely to share a trivial enum check. -3. Test the effective request matrix with ordinary and forced execution, - including force overriding explicit bypass and resume. Adapter tests own the - exact flag mapping and rejection of invalid direct requests. Keep a test for - absent settings producing the previous ordinary argv. Disabled extraction - still never invokes the adapter. -4. Add a focused semantic-fingerprint exclusion test for the two controls; do - not bump the fingerprint version or change manifest schema or invalidation. - -Acceptance: focused normal and race tests for `./internal/stage` and -`./internal/adapters/notarius` pass. No cache deletion or upstream config parsing -has been introduced. Existing credential-redaction tests remain green. - -## Stage 3 — Replace Run-Local Reference Snapshots with Stable Verified Snapshots - -Status: Complete (2026-09-16) - -Completion: Replaced run-ID reference copies with confined, content-addressed -session snapshots; added canonical cache path ownership, verified reuse, -reconstruction and regular-file repair, unsafe-conflict failures, narrow cleanup -of invalid new writes, and shared post-run verification. Removed the obsolete -run-local reference helper. Verified the required artifacts/stage/fileops/ -pathsafe tests and race tests for artifacts/stage. - -Read: `internal/artifacts/paths.go` and its tests; -`internal/stage/extract_references.go`, `extract.go`, `extract_resume.go`, -`extract_test.go` and reference-focused tests; existing root-confined helpers in -`internal/pathsafe/` and atomic/checksum helpers in `internal/fileops/`; -`docs/internal/artifacts.md`, `docs/internal/workspace.md`. - -1. Add canonical artifact/path-owner helpers for the cache root and confined - snapshot destination. Reuse the package's existing path-segment conventions - and central constants; validate digest syntax before using it as a segment. - Do not embed new layout construction independently in materialization and - post-run verification. -2. Refactor `materializeExtractReferenceSnapshots` to resolve stable destinations. - Verify an existing regular file's size and SHA-256 before reuse. For missing - or corrupt regular entries, copy from the manifest-verified source using the - existing atomic/checksum primitive, checking the result before invocation. - Deduplicate equivalent destinations within the request. Check path components - before directory creation/opening using existing confinement facilities. -3. If a newly written entry fails verification because its source changed, do - not invoke Notarius. Remove only that invalid, newly produced cache file - through a confined operation if safe; do not remove the root or other entries. - Preserve an actionable prepared-source error. -4. Make post-run verification use the same destination resolver and reject - mutated, missing, symlinked, or non-regular snapshots before bundle promotion. - Continue checking actual content, not just names or modification times. -5. Remove obsolete runtime use of the run-ID snapshot path helper. Remove the - helper if no callers remain; no compatibility shim or migration is needed. - Leave old run files alone. Do not change current prepared-reference metadata, - recorded source paths, semantic evidence, or durable bundle locations. -6. Adapt existing reference tests rather than keeping parallel old/new suites. - Cover stable paths across run IDs, changed content, same basename/different - content, verified reuse, reconstruction after deletion, corrupt regular-file - repair, unsafe conflicts, and post-invocation mutation rejection. Assert - behavior and confinement, not incidental helper choreography. - -Acceptance: `GOWORK=off go test ./internal/artifacts ./internal/stage -./internal/fileops ./internal/pathsafe` (on one shell line) and race tests for -artifacts/stage pass. No new general cache abstraction, lock service, retention -policy, or permission regime was added. - -## Stage 4 — Verify Session Lifecycle and Command Boundaries - -Status: Complete (2026-09-16) - -Completion: Added lifecycle coverage for stable retry snapshots, changed -references, durable extract reuse after cache-control changes or snapshot loss, -analysis-only preservation, and forced extraction/analysis regeneration. Added -narrow cleanup, publish, and restore regressions proving the snapshot cache is -retained and excluded from artifact transfer. Verified normal and race tests -for `./internal/app ./internal/stage`. - -Read: `internal/app/extract_lifecycle_test.go`, `regenerate_artifacts.go`, -`regenerate_artifacts_test.go`, `semantic_resume_test.go`, -`cleanup_orchestration_test.go`; `internal/stage/cleanup_plan.go`, -`cleanup_test.go`; publish/restore enumeration owners identified through their -callers; `docs/operations.md` and `docs/cli.md`. - -1. Extend the existing lifecycle fixture to prove a failed extraction followed - by a new invocation delivers identical snapshot paths for unchanged prepared - references. A changed valid reference selects a new path. Keep old snapshots - available; they are not current artifact evidence. -2. Verify a successful complete extract remains reusable after cache-control - changes or deletion of its local snapshots. No snapshot directory should be - created merely to plan or skip extraction. -3. Exercise ordinary successful cleanup with snapshots present: run-local - output is removed as before while snapshots survive. Confirm publish and - restore selection do not include the new cache subtree; add a narrow - regression where existing enumeration could accidentally include it, not a - second publish/restore test framework. -4. Extend existing command/lifecycle tests to show `analyze SESSION` never - invokes Notarius and preserves extract outputs, whereas - `regenerate-artifacts SESSION` uses the ordinary forced extract/analyze path. - `--artifacts` continues to narrow analysis, not the extraction bundle. -5. Confirm errors still stop before analysis and leave no partially promoted - current bundle. Reuse existing failure/promotion coverage where sufficient. - -Acceptance: normal and race tests for `./internal/app ./internal/stage` pass. -No new alias semantics, CLI options, manifest schema, or unrelated lifecycle -changes. This stage proves Narratio orchestration, not upstream cache hits. - -## Stage 5 — Prove Reuse with Real Notarius and a Local Test Provider - -Status: Complete (2026-09-16) - -Completion: Added an opt-in production-boundary integration test using a local -OpenAI-compatible provider and an absolute `NARRATIO_TEST_NOTARIUS_BINARY`. -It proves checkpoint reuse across distinct Narratio runs, cold execution after -reference changes, partial-failure recovery, forced checkpoint bypass, chunk -plan refresh, and preservation of an unrelated cached plan. It also aligned -receipt validation with Notarius's valid zero-attempt summaries for reused -producers. Tested against locally built `notarius v0.7.0` with -`NARRATIO_TEST_NOTARIUS_BINARY=/tmp/narratio-notarius-v070.g6gID8/notarius -GOWORK=off go test ./internal/stage -run -'^TestNotariusCacheReuseIntegration$' -count=1 -v`; the focused race test, -default `GOWORK=off go test -count=1 ./...`, and adapter/stage race tests pass. - -Read Notarius v0.7.0 `docs/cli.md`, `docs/config.md`, `docs/operations.md`, -`examples/dnd-minimal.config.yml`, `examples/seriatim-minimal-transcript.json`, -`internal/cli/production_contract_test.go` (local-backend example), and -`internal/framework/checkpoint/identity.go`. Inspect the selected built-in -module's schema and reference slots rather than inventing output payloads. -Read Narratio's existing extraction lifecycle and adapter test fixtures. - -1. Add an explicitly opt-in Go integration test, invoked with an absolute - `NARRATIO_TEST_NOTARIUS_BINARY` path. If unset, skip with a clear reason; if - set but unusable, fail. Document the focused `go test -run` invocation. - The normal suite must not discover Notarius on PATH, build/download upstream, - or depend on the sibling checkout. Do not add Notarius to `go.mod`. -2. Run the real executable with a small built-in D&D pipeline and its ordinary - JSON output module. Use a local `httptest` OpenAI-compatible chat-completions - endpoint via Notarius's `promptkit.local_backend`, a `backend: local` profile, - and schema-valid deterministic responses. No real provider, credentials, or - upstream test-only module registration is necessary. Use explicit temporary - config, profile, cache, and output roots; keep endpoint/profile content stable - across the compared invocations because they affect checkpoint identity. -3. Drive production Narratio snapshot delivery and the production adapter from - the fixture, not a hand-assembled imitation of their argv. Use different run - IDs/output roots and at least one declared prepared reference. Count local - model requests and inspect upstream checkpoint decision codes: first execution - populates state; second ordinary resume reuses accepted work; changing valid - reference bytes causes execution rather than a false hit. -4. Add a deterministic partial-failure scenario with accepted predecessor work: - configure ordered work and return a controlled failure for a later producer, - including any bounded retries. Keep configuration and endpoint unchanged on - retry; restore only server behavior. Verify accepted earlier work is reused - and the failed work is executed. Do not infer reuse from equal output bytes. -5. Exercise forced extraction against populated state. Verify checkpoint loading - is disabled and work executes again. Demonstrate actual chunk-plan refresh - through Notarius's chunk provenance/cache behavior in addition to the adapter - flag test; deterministic plans need not differ in bytes. Check that an - unrelated cached input remains untouched. -6. Run against a locally built v0.7.0 executable (or a documented compatible - newer version) without modifying or tagging the upstream repository. Record - version, focused command, and results in this stage's completion note. Keep - test fixtures small and scoped; do not reproduce upstream's whole cache suite. - -Acceptance: the opt-in test passes without internet/provider calls, ordinary -`go test ./...` remains independent of upstream installation, and there is real -evidence of cache reuse across Narratio invocations. A skipped integration test -does not complete this stage. If upstream behavior blocks the stated contract, -record the evidence and stop for direction rather than weakening the test or -silently modifying upstream. - -## Stage 6 — Document the Workflow and Complete Validation - -Status: Complete (2026-09-16) - -Completion: Documented the analysis-only, extraction-plus-analysis, and -ordinary continuation workflows; cache fields and force precedence; Notarius -checkpoint ownership; stable snapshot retention and recovery; troubleshooting; -and the tested v0.7.0 integration baseline. Updated the maintained annotated -example. Module tidy/verification, uncached normal and race tests, vet, build, -doccheck, example validation, Go formatting, whitespace checks, and all three -`v0.0.0-test` release-asset builds pass. - -Read: `docs/operations.md`, `docs/cli.md`, `docs/config.md`, -`docs/integrations/notarius.md`, `docs/internal/stage-extract.md`, -`docs/internal/adapters.md`, `docs/internal/workspace.md`, -`docs/troubleshooting.md`, `examples/README.md`, `docs/release.md`. - -1. Put the roadmap's three-way development workflow comparison prominently at - the start of Operations' artifact-development section: analysis only, - extraction plus analysis regeneration, and ordinary bounded continuation. - Explain preservation of transcripts/extraction, analysis selection, and - separate publishing. Link to CLI-owned exact syntax and avoid redefining it. -2. Update field/default/force precedence documentation and a maintained example. - Clearly explain upstream checkpoint enablement, explicit persistent roots, - cold misses from changed references/profile/paths, and the distinction between - Narratio skipping a stage and Notarius reusing work inside an executed stage. -3. Document stable snapshot location/retention, regular corruption recovery, - unsafe-path errors, and manual removal only while the session is inactive. - Do not add a cache purge command, auto-GC, mandatory private permissions, or - promise that all changed configuration is visible to Narratio resume. -4. Record the tested Notarius integration baseline and opt-in test instructions. - Preserve already-published release notes; leave version choice and release - publication to a separately authorized task. -5. Run the checks below. Mark the roadmap and stages complete only when all - required evidence, including Stage 5, is present. Record any environmental - limitations accurately; do not wait for CI or assert native macOS validation - based on cross-compilation. - -```sh -GOWORK=off go mod tidy -diff -GOWORK=off go mod verify -GOWORK=off go test -count=1 ./... -GOWORK=off go test -race -count=1 ./... -GOWORK=off go vet ./... -GOWORK=off go build ./... -GOWORK=off go test ./internal/doccheck -GOWORK=off go test ./internal/config -run '^TestExamplesLoadAndValidate$' -git diff --check -``` - -Check changed Go files with `gofmt -l`. Run the repository's -`scripts/build-release-assets.sh v0.0.0-test OUTPUT_DIR` into an empty absolute -directory created with `mktemp -d` to check the three maintained targets, without -publishing assets or tags. Keep any temporary binaries outside the worktree. - -## Open Questions - -None requiring an operator decision. Defaults, force precedence, cache ownership, -snapshot lifetime, and the command distinction are settled. The exact test -payloads are implementation details governed by the real upstream schemas, not -new product policy. Stage 5 must establish actual reuse before completion. diff --git a/docs/roadmap/notarius-cache-reuse.md b/docs/roadmap/notarius-cache-reuse.md deleted file mode 100644 index 798dc7d..0000000 --- a/docs/roadmap/notarius-cache-reuse.md +++ /dev/null @@ -1,212 +0,0 @@ -# Notarius Cache Reuse and Artifact Development - -Status: Accepted direction — implementation pending - -## Goal and User Intent - -Allow Narratio to reuse Notarius's cached chunk plans and accepted intermediate -artifacts when extraction actually executes, particularly when retrying failed -work. Preserve an explicit force path that regenerates the selected extraction -pipeline's work. Keep transcript generation separate from iterative artifact -development, and prominently explain the existing analysis-only command. - -This roadmap defines the desired end state, not an implementation sequence. -The [implementation plan](implementation.md) owns staged execution and checks. -It builds on the local Notarius v0.7.0 CLI and cache contracts reviewed for this -proposal. No upstream changes are required by the proposed design. - -## Current Behavior and Underlying Issue - -Narratio's manifest-based resume can skip an already-completed extract stage. -When that stage must execute, however, the Notarius adapter never passes -`--resume`, so Notarius cannot load compatible execution checkpoints. - -Notarius has two distinct mechanisms: - -- Chunk-plan caching defaults to `auto`. Valid source-addressed plans can - already be reused through Notarius configuration. `--chunk_cache` overrides - the configured mode with `auto`, `refresh`, or `bypass`. -- Checkpoint recording is disabled by default and is enabled in Notarius with - `cache.checkpoints.enabled: true`. Recording-enabled runs save accepted work - even without `--resume`; that flag separately permits checkpoint loading. - Asking for resume while recording is disabled fails. - -There is also a reference-identity obstacle. Narratio supplies references via -invocation-local snapshots, whose paths change with each run ID. Notarius's -checkpoint identity includes reference origin URIs as well as content digests. -Simply adding `--resume` therefore does not establish reuse across otherwise -equivalent invocations that supply those references. - -Relevant upstream owners are Notarius's `docs/cli.md`, `docs/config.md`, and -`docs/operations.md`; reference fingerprint construction is in -`internal/framework/checkpoint/identity.go`. Narratio's existing contracts are -documented in [Extract Internals](../internal/stage-extract.md) and the -[Notarius Integration](../integrations/notarius.md). - -## Configuration and Invocation Contract - -Add two optional fields to Narratio's Notarius section: - -```yaml -notarius: - resume: true - chunk_cache: auto -``` - -| Field | Default | Meaning | -| --- | --- | --- | -| `notarius.resume` | `false` | Permit compatible Notarius checkpoint loading when extraction executes without force. | -| `notarius.chunk_cache` | unset | Optional chunk-plan mode override: `auto`, `refresh`, or `bypass`; unset preserves Notarius configuration. | - -These are typed, validated configuration fields, not arbitrary CLI arguments. -They participate in existing profile composition, safe inspection, and source -reporting. Do not introduce another secret, configuration, or cache resolver. - -Notarius continues to own checkpoint recording, cache locations, compatibility -checks, and cache formats. Operators enable recording and choose persistent -cache directories in Notarius configuration. Narratio neither reads that -configuration to infer settings nor writes a replacement configuration. A -misconfigured resume request remains an actionable downstream error; do not -silently retry without reuse, which could unexpectedly incur provider costs. - -### Force Takes Precedence - -| Extraction invocation | Checkpoint loading | Chunk-plan policy | -| --- | --- | --- | -| Ordinary execution, `resume: false` | No `--resume` | Configured Narratio override, or no flag | -| Ordinary execution, `resume: true` | Pass `--resume` | Configured Narratio override, or no flag | -| Forced extraction | Never pass `--resume` | Always pass `--chunk_cache refresh` | - -Force regenerates the selected pipeline's chunk plan and objects rather than -hydrating accepted checkpoint results. Checkpoint recording may still save the -new work if enabled by Notarius. Force overrides even an explicit Narratio -`chunk_cache: bypass` selection; document this precedence clearly. - -“Clear the cache” means bypass the current input's cached chunk plan and replace -it with a newly accepted plan, not recursively delete the Notarius cache root. -Notarius's refresh operation publishes only an accepted replacement. A failed -refresh need not erase the old plan. Other inputs' plans and unrelated checkpoint -identities are left alone. No global purge, transactional regeneration, provider- -side cache bypass, or guarantee of different generated text is implied. - -Disabled extraction retains its ordinary explicit self-skip. The existing -force scope and failure propagation remain unchanged: a failed extraction does -not proceed to analysis. - -## Stable Reference Snapshots - -Use session-scoped, content-addressed reference snapshots at stable locations -outside invocation-local run directories. Equivalent references in the same -session must receive identical absolute snapshot paths across invocations; -changed bytes must select different snapshot identities. Central path helpers -own the layout, not adapter-local string construction. - -Retain the useful protections of the existing snapshot mechanism: - -- resolve inputs from manifest-backed prepared reference evidence; -- validate content against its recorded checksum and size; -- safely materialize missing snapshots and verify existing ones before reuse; -- retain confinement and post-invocation verification rather than silently - trusting the cache or substituting mutable prepared files; -- use the existing session lock and narrow file-operation helpers, not a new - general-purpose cache framework. - -Snapshots are reconstructible local cache state, not canonical artifacts, -published outputs, or credentials. Apply Narratio's ordinary group-shareable -workspace policy. Preserve them across ordinary run cleanup so later extraction -can reuse their locations; do not add garbage collection in this feature. -Document their location, retention, and safe removal while no session operation -is active. Missing snapshots can be reconstructed from valid prepared inputs. - -Old invocation-local snapshots and checkpoints require no migration. A first -execution under the stable layout may miss old checkpoint identities. Moving a -workspace may likewise cause cold misses; cross-host or relocated-path cache -identity is not promised. - -## Two Distinct Resume Layers - -Narratio remains responsible for deciding whether an existing complete extract -result is current. Notarius remains responsible for reusing compatible internal -work when an extraction process actually runs. Cached work must still produce -an ordinary successful bundle that passes Narratio's existing validation and -immutable promotion boundary. - -The two new controls are operational settings, not blanket invalidation keys. -Changing them alone does not stale a valid completed extraction. Explicit force -is required to execute otherwise-current extraction with a fresh chunk plan. -Preserve existing reference-content and transcript invalidation and the rule -that changes hidden inside Notarius configuration require operator action. - -Persistent upstream cache roots must not live inside disposable run output. -Recommend explicitly configured Notarius cache directories for service -deployments rather than assuming interactive-shell cache discovery is inherited -through Narratio's minimal subprocess environment. Do not broaden environment -inheritance or impose a new private-data policy on ordinary D&D material. - -## Development-Focused Operator Guidance - -Make the following distinction prominent at the beginning of the artifact -development workflow in [Operations](../operations.md), with links to the -[CLI Reference](../cli.md) for exact syntax: - -| Operator intent | Existing command | Result | -| --- | --- | --- | -| Iterate on analysis prompts or artifacts using existing extracted data | `narratio analyze SESSION` | Force analysis only; preserve extract results and transcripts. | -| Rebuild extraction and analysis | `narratio regenerate-artifacts SESSION` | Force extract through analyze; refresh chunking and avoid checkpoint loading. | -| Continue incomplete artifact work with configured reuse | `narratio run SESSION --from extract --through analyze` | Reuse valid Narratio results; permit configured Notarius reuse when extraction executes. | - -Explain `--artifacts` alongside these choices: it selects analysis targets and -their required analysis prerequisites, not a subset of the Notarius bundle. -The bounded commands require valid existing prefix-stage state; they do not -repair or rerun transcript generation automatically. Publishing and retention -remain separate actions. - -`regenerate-artifacts` remains exactly a convenience expansion to -`run SESSION --force --from extract --through analyze`. Do not give it private -cache semantics. `analyze` remains the existing forced analysis-only command; -no new alias is needed. - -## Required Changes and Acceptance - -- Extend the configuration owner and safe projections for the two fields. -- Carry effective cache intent through extract's typed adapter request, using - the existing force state; construct CLI flags only in the Notarius adapter. -- Replace invocation-dependent reference delivery paths with the stable snapshot - design without weakening prepared-input or bundle validation. -- Update canonical configuration, integration, internal, example, and recovery - documentation where owned. Put the workflow comparison front and center in - Operations, not only in developer internals or release notes. -- Cover default/explicit mode mapping, invalid modes, force precedence, and - disabled extraction with focused behavioral tests. -- Prove cross-invocation reuse with the supported Notarius behavior, including - unchanged reference bytes at stable paths, changed reference content causing - a miss, and retry after partial failure reusing accepted work. -- Prove forced extraction regenerates rather than reuses internal work, while - analysis-only invocation never launches Notarius or changes its results. -- Verify cache loss is recoverable, run cleanup preserves reusable snapshots, - and receipt validation, promotion, and failure handling remain intact. - -Follow the [Testing Policy](../policy/testing.md): default tests remain offline -and credential-free. An argv-only fake cannot prove upstream checkpoint reuse; -use controlled upstream integration evidence with deterministic/local test -modules. Do not make the default suite require a developer's installed Notarius -or a paid provider, and do not duplicate Notarius's internal cache test matrix. - -## Non-goals - -- Notarius `--recompute-step` or lane-selective extraction. -- A new Narratio cache-management command, cache format, or cache garbage collector. -- Parsing downstream configuration or automatically enabling checkpoint recording. -- Global cache deletion or bypassing provider-side caches. -- A new release, upstream change, or migration of old cache contents in this task. -- New cross-process locking for independent Notarius users or distributed cache - coordination; upstream cache concurrency limitations remain applicable. - -## Target End State - -Operators can cheaply continue incomplete extraction using compatible Notarius -work, deliberately regenerate the full extraction pipeline with force, or -iterate only on analysis using the existing `analyze` command. Stable reference -delivery makes reuse useful across Narratio invocations. Notarius owns its -caches; Narratio owns orchestration, reference snapshots, and validated durable -results. The operator guide makes the everyday development choices obvious. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md deleted file mode 100644 index fe85443..0000000 --- a/docs/roadmap/remediation.md +++ /dev/null @@ -1,85 +0,0 @@ -# Notarius Cache Reuse Review Remediation - -Status: Complete - -Review scope: commits `5086596` through `a0312ed`, compared with the accepted -cache roadmap and implementation plan. The core cache integration works, -including the real Notarius v0.7.0 integration test. Address the following -bounded findings; no broader cache redesign is required. - -## 1. P2 — Preserve Relative Workspace Support at the Snapshot Boundary - -Status: Complete. Snapshot materialization and post-run verification normalize -their local session/cache roots to absolute paths. Regression coverage proves -stable absolute adapter bindings, successful production-adapter launch, absence -of the duplicated directory tree, and rejection of a symlink on the actual -cache path for a relative workspace. - -Location: `internal/stage/extract_references.go`, especially -`extractReferenceSnapshotDestination` and `checkExtractSnapshotDirectory`; -`internal/artifacts/paths.go` snapshot path construction. - -The new snapshot destination inherits `SessionPaths.CacheDir` verbatim. A -relative workspace root therefore produces a relative reference binding, -whereas the previous extract implementation explicitly converted its snapshot -root to an absolute path. The Notarius adapter requires absolute reference -paths and rejects the new binding before launch. Configuration does not require -an absolute workspace root. - -There is a related path-composition defect: `checkExtractSnapshotDirectory` -passes an already session-prefixed relative destination to -`pathsafe.SlashRelativeFromRoot`, which interprets relative targets as relative -to the supplied root. The resulting directory walk duplicates the session -prefix instead of checking the actual destination ancestry. Subsequent copying -can create the intended directories through a different path, leaving the -wrong extra directory tree and bypassing the intended component checks there. - -Evidence: a temporary focused test created a valid prepared party reference, -expressed the session/cache roots relative to the working directory, and called -the production snapshot materializer. It returned a binding beginning -`work/campaign-a/.../cache/notarius/references/.../party.yml`, not an absolute -path. The diagnostic test was removed after the review. - -Required change: - -- Normalize the session root and snapshot destination consistently to absolute - paths before filesystem checks and adapter delivery, preserving the existing - meaning of relative workspace configuration. -- Ensure the component walk checks the same path subsequently read/written. -- Do not fix this by requiring operators to configure absolute workspace roots - or by changing global path semantics unnecessarily. -- Add a regression using a relative workspace, real snapshot materialization, - and the adapter validation boundary. Verify absolute stable bindings across - invocations, no duplicated session-prefix directory, and rejection of an - unsafe component on the actual cache path. - -## 2. P3 — Remove the Accidental Reference-Version Requirement - -Status: Complete. The integration contract now distinguishes the v0.6.0 -reference capability from selected cache flags and the tested v0.7.0 baseline, -without introducing a runtime gate. - -Location: `docs/integrations/notarius.md`, Invocation section, line 28. - -The documentation now says Narratio requires the Notarius v0.7.0 CLI contract -when references are configured. References were already supported under the -documented v0.6.0 contract. The implementation plan selected v0.7.0 as the -tested cache integration baseline and explicitly did not require rejecting -older compatible executables. The revised sentence unnecessarily suggests an -upgrade is mandatory for existing reference-only use. - -Required change: distinguish the tested v0.7.0 cache integration baseline from -the reference CLI contract. Document required capabilities for the selected -flags without claiming a new unconditional minimum version. Keep runtime -behavior unchanged; do not add a version probe or gate. - -## Validation Recorded - -- `GOWORK=off go test ./...` passed. -- Race tests for stage, app, config, artifacts, and Notarius adapter passed. -- `GOWORK=off go vet ./...` passed. -- The opt-in `TestNotariusCacheReuseIntegration` passed against the previously - built local v0.7.0 binary with its local test provider, covering actual reuse, - reference-change misses, partial recovery, and forced chunk refresh. -- The additional relative-path diagnostic failed as described above; this case - was not covered by the existing absolute-path fixtures.