Remove completed Notarius cache roadmaps
Some checks failed
ci/woodpecker/push/verify Pipeline was canceled
Some checks failed
ci/woodpecker/push/verify Pipeline was canceled
This commit is contained in:
@@ -1,337 +0,0 @@
|
||||
# Notarius Cache Reuse Implementation Plan
|
||||
|
||||
Status: Complete (2026-09-16)
|
||||
|
||||
Implement the [feature roadmap](notarius-cache-reuse.md) in the numbered order
|
||||
below. Each stage is a bounded assignment for a gpt-5.6-terra coding agent.
|
||||
Record completion and checks beside each stage; intermediate stages are not a
|
||||
claim that the whole feature is ready. No release, tag, push, upstream edit, or
|
||||
dependency upgrade is authorized by this plan.
|
||||
|
||||
## Instructions and Settled Decisions
|
||||
|
||||
Read `docs/development.md`, every document in `docs/policy/`, the feature
|
||||
roadmap, and the sources listed for the current stage. Inspect callers and
|
||||
existing tests before editing. File lists identify owners, not edit allowlists.
|
||||
Use existing path, file-operation, configuration, and subprocess boundaries.
|
||||
Do not create a general cache framework or change unrelated safety policies.
|
||||
|
||||
- Add `Resume bool` and `ChunkCache string` to `config.NotariusConfig`, using
|
||||
YAML/JSON names `resume` and `chunk_cache` with `omitempty`. False and the empty
|
||||
string are the unset defaults. Null resets to the zero value through existing
|
||||
ConfigKit behavior. For enabled Notarius configuration, trim surrounding
|
||||
whitespace and accept only empty, `auto`, `bypass`, or `refresh`; do not add
|
||||
case-insensitive aliases. Preserve the existing disabled-section validation
|
||||
policy rather than introducing new mandatory configuration checks there.
|
||||
- Ordinary extraction passes the configured resume boolean and chunk mode.
|
||||
Forced extraction always passes resume false and chunk mode `refresh`, even
|
||||
if configuration requests bypass. The stage resolves force precedence; the
|
||||
adapter translates effective intent into flags. No private alias behavior.
|
||||
- Use stable snapshot paths under
|
||||
`<session-root>/cache/notarius/references/<sha256>/<prepared-basename>`.
|
||||
SHA-256 is the full lowercase hexadecimal digest of the reference bytes.
|
||||
The basename preserves the prepared file extension and is validated as one
|
||||
path component. Identity contains neither run ID nor timestamps. Different
|
||||
sources with identical bytes and basename can share a verified snapshot.
|
||||
Same basename with different bytes cannot collide. This is an implementation
|
||||
layout, not a new external artifact schema.
|
||||
- Keep prepared-input identities, metadata, and extract resume fingerprints
|
||||
unchanged. Snapshot paths are delivery details. Neither new setting belongs
|
||||
in semantic resume evidence; safe effective configuration can expose them.
|
||||
- A missing snapshot is reconstructed. A corrupt regular snapshot is atomically
|
||||
replaced from a currently verified prepared source. Unsafe paths, symlinks,
|
||||
directories, or other non-regular conflicts fail with the affected path and
|
||||
recovery guidance, never recursive deletion. A prepared-source mismatch
|
||||
remains an error, not a cache miss. No eviction or cache ledger is needed.
|
||||
- Snapshots are ordinary group-shareable workspace files, not API-key material.
|
||||
They survive ordinary run cleanup, are not published/restored as artifacts,
|
||||
and do not become prerequisites for reuse of a complete durable extract bundle.
|
||||
- Notarius owns checkpoint enablement, roots, recording, and validity. Do not
|
||||
read its config to infer enablement, silently retry without resume, or widen
|
||||
inherited environment. Recommend explicit persistent upstream cache roots.
|
||||
- Cache refresh is not deletion: regenerate the current input's plan and let
|
||||
Notarius publish an accepted replacement. Leave other plans/checkpoints alone.
|
||||
No guarantee of different text, provider-side cache bypass, or transactional
|
||||
regeneration is introduced.
|
||||
- The real integration acceptance baseline is Notarius v0.7.0. Do not add a
|
||||
runtime version probe or reject older compatible executables just by version.
|
||||
Unsupported flags fail through the existing subprocess error path.
|
||||
|
||||
## Stage 1 — Add Typed Cache Controls
|
||||
|
||||
Status: Complete (2026-09-16)
|
||||
|
||||
Completion: Added typed, zero-default `resume` and `chunk_cache` controls with
|
||||
enabled-section normalization and validation, safe YAML/JSON projection, and
|
||||
ConfigKit profile/provenance coverage. Verified with
|
||||
`GOWORK=off go test ./internal/config` and `GOWORK=off go test ./...`.
|
||||
|
||||
Read: `internal/config/config.go`, `credential_projection.go`,
|
||||
`pipeline_normalize.go`, `validate_notarius.go`, `notarius_test.go`,
|
||||
`pipeline_configkit_test.go`, `effective_projection.go`; configuration sections
|
||||
of `docs/config.md` and `docs/internal/configuration.md`.
|
||||
|
||||
1. Add the two fields and implement the zero-value, normalization, and validation
|
||||
decisions above. Keep defaults centralized; no stage-specific environment
|
||||
overrides or raw extra-argument map.
|
||||
2. Extend `NotariusConfig.MarshalYAML` so direct serialization retains both
|
||||
fields and still projects only the credential reference name. Confirm that
|
||||
effective projection, source reporting, copying, and profile overrides carry
|
||||
the fields without adding another projection layer.
|
||||
3. Extend focused configuration tests for absence, explicit modes, invalid
|
||||
enabled modes, null reset, profile replacement, and safe YAML/JSON inspection.
|
||||
Protect the existing optional credential behavior. Do not snapshot complete
|
||||
error wording or repeat every case at the CLI layer.
|
||||
|
||||
Acceptance: `GOWORK=off go test ./internal/config` and
|
||||
`GOWORK=off go test ./...` pass. Existing configurations require no edits.
|
||||
This stage alone does not enable delivery and is not a finished feature.
|
||||
|
||||
## Stage 2 — Forward Cache Intent and Apply Force Precedence
|
||||
|
||||
Status: Complete (2026-09-16)
|
||||
|
||||
Completion: Carried typed cache intent through extract requests, applied force
|
||||
precedence in stage orchestration, mapped validated effective values to exact
|
||||
Notarius flags, and confirmed both controls remain outside semantic resume
|
||||
evidence. Verified normal and race tests for `./internal/stage` and
|
||||
`./internal/adapters/notarius`.
|
||||
|
||||
Read: `internal/stage/extract.go`, `stage.go`, `extract_test.go`,
|
||||
`semantic_contracts_delivery.go`, `semantic_contracts_delivery_test.go`;
|
||||
`internal/adapters/notarius/runner.go`, `subprocess.go`, `subprocess_test.go`,
|
||||
`fake.go`; `docs/internal/adapters.md`, `docs/integrations/notarius.md`.
|
||||
|
||||
1. Add `Resume bool` and `ChunkCache string` to the typed Notarius request.
|
||||
Populate them in extract, applying `env.Force` precedence in one place.
|
||||
2. Append `--resume` only for true and `--chunk_cache <mode>` only for nonempty
|
||||
mode. Preserve existing arguments, references, credentials, diagnostics,
|
||||
receipt parsing, and bundle discovery. Validate programmatic nonempty modes
|
||||
before starting a subprocess; do not import application config into the
|
||||
adapter merely to share a trivial enum check.
|
||||
3. Test the effective request matrix with ordinary and forced execution,
|
||||
including force overriding explicit bypass and resume. Adapter tests own the
|
||||
exact flag mapping and rejection of invalid direct requests. Keep a test for
|
||||
absent settings producing the previous ordinary argv. Disabled extraction
|
||||
still never invokes the adapter.
|
||||
4. Add a focused semantic-fingerprint exclusion test for the two controls; do
|
||||
not bump the fingerprint version or change manifest schema or invalidation.
|
||||
|
||||
Acceptance: focused normal and race tests for `./internal/stage` and
|
||||
`./internal/adapters/notarius` pass. No cache deletion or upstream config parsing
|
||||
has been introduced. Existing credential-redaction tests remain green.
|
||||
|
||||
## Stage 3 — Replace Run-Local Reference Snapshots with Stable Verified Snapshots
|
||||
|
||||
Status: Complete (2026-09-16)
|
||||
|
||||
Completion: Replaced run-ID reference copies with confined, content-addressed
|
||||
session snapshots; added canonical cache path ownership, verified reuse,
|
||||
reconstruction and regular-file repair, unsafe-conflict failures, narrow cleanup
|
||||
of invalid new writes, and shared post-run verification. Removed the obsolete
|
||||
run-local reference helper. Verified the required artifacts/stage/fileops/
|
||||
pathsafe tests and race tests for artifacts/stage.
|
||||
|
||||
Read: `internal/artifacts/paths.go` and its tests;
|
||||
`internal/stage/extract_references.go`, `extract.go`, `extract_resume.go`,
|
||||
`extract_test.go` and reference-focused tests; existing root-confined helpers in
|
||||
`internal/pathsafe/` and atomic/checksum helpers in `internal/fileops/`;
|
||||
`docs/internal/artifacts.md`, `docs/internal/workspace.md`.
|
||||
|
||||
1. Add canonical artifact/path-owner helpers for the cache root and confined
|
||||
snapshot destination. Reuse the package's existing path-segment conventions
|
||||
and central constants; validate digest syntax before using it as a segment.
|
||||
Do not embed new layout construction independently in materialization and
|
||||
post-run verification.
|
||||
2. Refactor `materializeExtractReferenceSnapshots` to resolve stable destinations.
|
||||
Verify an existing regular file's size and SHA-256 before reuse. For missing
|
||||
or corrupt regular entries, copy from the manifest-verified source using the
|
||||
existing atomic/checksum primitive, checking the result before invocation.
|
||||
Deduplicate equivalent destinations within the request. Check path components
|
||||
before directory creation/opening using existing confinement facilities.
|
||||
3. If a newly written entry fails verification because its source changed, do
|
||||
not invoke Notarius. Remove only that invalid, newly produced cache file
|
||||
through a confined operation if safe; do not remove the root or other entries.
|
||||
Preserve an actionable prepared-source error.
|
||||
4. Make post-run verification use the same destination resolver and reject
|
||||
mutated, missing, symlinked, or non-regular snapshots before bundle promotion.
|
||||
Continue checking actual content, not just names or modification times.
|
||||
5. Remove obsolete runtime use of the run-ID snapshot path helper. Remove the
|
||||
helper if no callers remain; no compatibility shim or migration is needed.
|
||||
Leave old run files alone. Do not change current prepared-reference metadata,
|
||||
recorded source paths, semantic evidence, or durable bundle locations.
|
||||
6. Adapt existing reference tests rather than keeping parallel old/new suites.
|
||||
Cover stable paths across run IDs, changed content, same basename/different
|
||||
content, verified reuse, reconstruction after deletion, corrupt regular-file
|
||||
repair, unsafe conflicts, and post-invocation mutation rejection. Assert
|
||||
behavior and confinement, not incidental helper choreography.
|
||||
|
||||
Acceptance: `GOWORK=off go test ./internal/artifacts ./internal/stage
|
||||
./internal/fileops ./internal/pathsafe` (on one shell line) and race tests for
|
||||
artifacts/stage pass. No new general cache abstraction, lock service, retention
|
||||
policy, or permission regime was added.
|
||||
|
||||
## Stage 4 — Verify Session Lifecycle and Command Boundaries
|
||||
|
||||
Status: Complete (2026-09-16)
|
||||
|
||||
Completion: Added lifecycle coverage for stable retry snapshots, changed
|
||||
references, durable extract reuse after cache-control changes or snapshot loss,
|
||||
analysis-only preservation, and forced extraction/analysis regeneration. Added
|
||||
narrow cleanup, publish, and restore regressions proving the snapshot cache is
|
||||
retained and excluded from artifact transfer. Verified normal and race tests
|
||||
for `./internal/app ./internal/stage`.
|
||||
|
||||
Read: `internal/app/extract_lifecycle_test.go`, `regenerate_artifacts.go`,
|
||||
`regenerate_artifacts_test.go`, `semantic_resume_test.go`,
|
||||
`cleanup_orchestration_test.go`; `internal/stage/cleanup_plan.go`,
|
||||
`cleanup_test.go`; publish/restore enumeration owners identified through their
|
||||
callers; `docs/operations.md` and `docs/cli.md`.
|
||||
|
||||
1. Extend the existing lifecycle fixture to prove a failed extraction followed
|
||||
by a new invocation delivers identical snapshot paths for unchanged prepared
|
||||
references. A changed valid reference selects a new path. Keep old snapshots
|
||||
available; they are not current artifact evidence.
|
||||
2. Verify a successful complete extract remains reusable after cache-control
|
||||
changes or deletion of its local snapshots. No snapshot directory should be
|
||||
created merely to plan or skip extraction.
|
||||
3. Exercise ordinary successful cleanup with snapshots present: run-local
|
||||
output is removed as before while snapshots survive. Confirm publish and
|
||||
restore selection do not include the new cache subtree; add a narrow
|
||||
regression where existing enumeration could accidentally include it, not a
|
||||
second publish/restore test framework.
|
||||
4. Extend existing command/lifecycle tests to show `analyze SESSION` never
|
||||
invokes Notarius and preserves extract outputs, whereas
|
||||
`regenerate-artifacts SESSION` uses the ordinary forced extract/analyze path.
|
||||
`--artifacts` continues to narrow analysis, not the extraction bundle.
|
||||
5. Confirm errors still stop before analysis and leave no partially promoted
|
||||
current bundle. Reuse existing failure/promotion coverage where sufficient.
|
||||
|
||||
Acceptance: normal and race tests for `./internal/app ./internal/stage` pass.
|
||||
No new alias semantics, CLI options, manifest schema, or unrelated lifecycle
|
||||
changes. This stage proves Narratio orchestration, not upstream cache hits.
|
||||
|
||||
## Stage 5 — Prove Reuse with Real Notarius and a Local Test Provider
|
||||
|
||||
Status: Complete (2026-09-16)
|
||||
|
||||
Completion: Added an opt-in production-boundary integration test using a local
|
||||
OpenAI-compatible provider and an absolute `NARRATIO_TEST_NOTARIUS_BINARY`.
|
||||
It proves checkpoint reuse across distinct Narratio runs, cold execution after
|
||||
reference changes, partial-failure recovery, forced checkpoint bypass, chunk
|
||||
plan refresh, and preservation of an unrelated cached plan. It also aligned
|
||||
receipt validation with Notarius's valid zero-attempt summaries for reused
|
||||
producers. Tested against locally built `notarius v0.7.0` with
|
||||
`NARRATIO_TEST_NOTARIUS_BINARY=/tmp/narratio-notarius-v070.g6gID8/notarius
|
||||
GOWORK=off go test ./internal/stage -run
|
||||
'^TestNotariusCacheReuseIntegration$' -count=1 -v`; the focused race test,
|
||||
default `GOWORK=off go test -count=1 ./...`, and adapter/stage race tests pass.
|
||||
|
||||
Read Notarius v0.7.0 `docs/cli.md`, `docs/config.md`, `docs/operations.md`,
|
||||
`examples/dnd-minimal.config.yml`, `examples/seriatim-minimal-transcript.json`,
|
||||
`internal/cli/production_contract_test.go` (local-backend example), and
|
||||
`internal/framework/checkpoint/identity.go`. Inspect the selected built-in
|
||||
module's schema and reference slots rather than inventing output payloads.
|
||||
Read Narratio's existing extraction lifecycle and adapter test fixtures.
|
||||
|
||||
1. Add an explicitly opt-in Go integration test, invoked with an absolute
|
||||
`NARRATIO_TEST_NOTARIUS_BINARY` path. If unset, skip with a clear reason; if
|
||||
set but unusable, fail. Document the focused `go test -run` invocation.
|
||||
The normal suite must not discover Notarius on PATH, build/download upstream,
|
||||
or depend on the sibling checkout. Do not add Notarius to `go.mod`.
|
||||
2. Run the real executable with a small built-in D&D pipeline and its ordinary
|
||||
JSON output module. Use a local `httptest` OpenAI-compatible chat-completions
|
||||
endpoint via Notarius's `promptkit.local_backend`, a `backend: local` profile,
|
||||
and schema-valid deterministic responses. No real provider, credentials, or
|
||||
upstream test-only module registration is necessary. Use explicit temporary
|
||||
config, profile, cache, and output roots; keep endpoint/profile content stable
|
||||
across the compared invocations because they affect checkpoint identity.
|
||||
3. Drive production Narratio snapshot delivery and the production adapter from
|
||||
the fixture, not a hand-assembled imitation of their argv. Use different run
|
||||
IDs/output roots and at least one declared prepared reference. Count local
|
||||
model requests and inspect upstream checkpoint decision codes: first execution
|
||||
populates state; second ordinary resume reuses accepted work; changing valid
|
||||
reference bytes causes execution rather than a false hit.
|
||||
4. Add a deterministic partial-failure scenario with accepted predecessor work:
|
||||
configure ordered work and return a controlled failure for a later producer,
|
||||
including any bounded retries. Keep configuration and endpoint unchanged on
|
||||
retry; restore only server behavior. Verify accepted earlier work is reused
|
||||
and the failed work is executed. Do not infer reuse from equal output bytes.
|
||||
5. Exercise forced extraction against populated state. Verify checkpoint loading
|
||||
is disabled and work executes again. Demonstrate actual chunk-plan refresh
|
||||
through Notarius's chunk provenance/cache behavior in addition to the adapter
|
||||
flag test; deterministic plans need not differ in bytes. Check that an
|
||||
unrelated cached input remains untouched.
|
||||
6. Run against a locally built v0.7.0 executable (or a documented compatible
|
||||
newer version) without modifying or tagging the upstream repository. Record
|
||||
version, focused command, and results in this stage's completion note. Keep
|
||||
test fixtures small and scoped; do not reproduce upstream's whole cache suite.
|
||||
|
||||
Acceptance: the opt-in test passes without internet/provider calls, ordinary
|
||||
`go test ./...` remains independent of upstream installation, and there is real
|
||||
evidence of cache reuse across Narratio invocations. A skipped integration test
|
||||
does not complete this stage. If upstream behavior blocks the stated contract,
|
||||
record the evidence and stop for direction rather than weakening the test or
|
||||
silently modifying upstream.
|
||||
|
||||
## Stage 6 — Document the Workflow and Complete Validation
|
||||
|
||||
Status: Complete (2026-09-16)
|
||||
|
||||
Completion: Documented the analysis-only, extraction-plus-analysis, and
|
||||
ordinary continuation workflows; cache fields and force precedence; Notarius
|
||||
checkpoint ownership; stable snapshot retention and recovery; troubleshooting;
|
||||
and the tested v0.7.0 integration baseline. Updated the maintained annotated
|
||||
example. Module tidy/verification, uncached normal and race tests, vet, build,
|
||||
doccheck, example validation, Go formatting, whitespace checks, and all three
|
||||
`v0.0.0-test` release-asset builds pass.
|
||||
|
||||
Read: `docs/operations.md`, `docs/cli.md`, `docs/config.md`,
|
||||
`docs/integrations/notarius.md`, `docs/internal/stage-extract.md`,
|
||||
`docs/internal/adapters.md`, `docs/internal/workspace.md`,
|
||||
`docs/troubleshooting.md`, `examples/README.md`, `docs/release.md`.
|
||||
|
||||
1. Put the roadmap's three-way development workflow comparison prominently at
|
||||
the start of Operations' artifact-development section: analysis only,
|
||||
extraction plus analysis regeneration, and ordinary bounded continuation.
|
||||
Explain preservation of transcripts/extraction, analysis selection, and
|
||||
separate publishing. Link to CLI-owned exact syntax and avoid redefining it.
|
||||
2. Update field/default/force precedence documentation and a maintained example.
|
||||
Clearly explain upstream checkpoint enablement, explicit persistent roots,
|
||||
cold misses from changed references/profile/paths, and the distinction between
|
||||
Narratio skipping a stage and Notarius reusing work inside an executed stage.
|
||||
3. Document stable snapshot location/retention, regular corruption recovery,
|
||||
unsafe-path errors, and manual removal only while the session is inactive.
|
||||
Do not add a cache purge command, auto-GC, mandatory private permissions, or
|
||||
promise that all changed configuration is visible to Narratio resume.
|
||||
4. Record the tested Notarius integration baseline and opt-in test instructions.
|
||||
Preserve already-published release notes; leave version choice and release
|
||||
publication to a separately authorized task.
|
||||
5. Run the checks below. Mark the roadmap and stages complete only when all
|
||||
required evidence, including Stage 5, is present. Record any environmental
|
||||
limitations accurately; do not wait for CI or assert native macOS validation
|
||||
based on cross-compilation.
|
||||
|
||||
```sh
|
||||
GOWORK=off go mod tidy -diff
|
||||
GOWORK=off go mod verify
|
||||
GOWORK=off go test -count=1 ./...
|
||||
GOWORK=off go test -race -count=1 ./...
|
||||
GOWORK=off go vet ./...
|
||||
GOWORK=off go build ./...
|
||||
GOWORK=off go test ./internal/doccheck
|
||||
GOWORK=off go test ./internal/config -run '^TestExamplesLoadAndValidate$'
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Check changed Go files with `gofmt -l`. Run the repository's
|
||||
`scripts/build-release-assets.sh v0.0.0-test OUTPUT_DIR` into an empty absolute
|
||||
directory created with `mktemp -d` to check the three maintained targets, without
|
||||
publishing assets or tags. Keep any temporary binaries outside the worktree.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None requiring an operator decision. Defaults, force precedence, cache ownership,
|
||||
snapshot lifetime, and the command distinction are settled. The exact test
|
||||
payloads are implementation details governed by the real upstream schemas, not
|
||||
new product policy. Stage 5 must establish actual reuse before completion.
|
||||
@@ -1,212 +0,0 @@
|
||||
# Notarius Cache Reuse and Artifact Development
|
||||
|
||||
Status: Accepted direction — implementation pending
|
||||
|
||||
## Goal and User Intent
|
||||
|
||||
Allow Narratio to reuse Notarius's cached chunk plans and accepted intermediate
|
||||
artifacts when extraction actually executes, particularly when retrying failed
|
||||
work. Preserve an explicit force path that regenerates the selected extraction
|
||||
pipeline's work. Keep transcript generation separate from iterative artifact
|
||||
development, and prominently explain the existing analysis-only command.
|
||||
|
||||
This roadmap defines the desired end state, not an implementation sequence.
|
||||
The [implementation plan](implementation.md) owns staged execution and checks.
|
||||
It builds on the local Notarius v0.7.0 CLI and cache contracts reviewed for this
|
||||
proposal. No upstream changes are required by the proposed design.
|
||||
|
||||
## Current Behavior and Underlying Issue
|
||||
|
||||
Narratio's manifest-based resume can skip an already-completed extract stage.
|
||||
When that stage must execute, however, the Notarius adapter never passes
|
||||
`--resume`, so Notarius cannot load compatible execution checkpoints.
|
||||
|
||||
Notarius has two distinct mechanisms:
|
||||
|
||||
- Chunk-plan caching defaults to `auto`. Valid source-addressed plans can
|
||||
already be reused through Notarius configuration. `--chunk_cache` overrides
|
||||
the configured mode with `auto`, `refresh`, or `bypass`.
|
||||
- Checkpoint recording is disabled by default and is enabled in Notarius with
|
||||
`cache.checkpoints.enabled: true`. Recording-enabled runs save accepted work
|
||||
even without `--resume`; that flag separately permits checkpoint loading.
|
||||
Asking for resume while recording is disabled fails.
|
||||
|
||||
There is also a reference-identity obstacle. Narratio supplies references via
|
||||
invocation-local snapshots, whose paths change with each run ID. Notarius's
|
||||
checkpoint identity includes reference origin URIs as well as content digests.
|
||||
Simply adding `--resume` therefore does not establish reuse across otherwise
|
||||
equivalent invocations that supply those references.
|
||||
|
||||
Relevant upstream owners are Notarius's `docs/cli.md`, `docs/config.md`, and
|
||||
`docs/operations.md`; reference fingerprint construction is in
|
||||
`internal/framework/checkpoint/identity.go`. Narratio's existing contracts are
|
||||
documented in [Extract Internals](../internal/stage-extract.md) and the
|
||||
[Notarius Integration](../integrations/notarius.md).
|
||||
|
||||
## Configuration and Invocation Contract
|
||||
|
||||
Add two optional fields to Narratio's Notarius section:
|
||||
|
||||
```yaml
|
||||
notarius:
|
||||
resume: true
|
||||
chunk_cache: auto
|
||||
```
|
||||
|
||||
| Field | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `notarius.resume` | `false` | Permit compatible Notarius checkpoint loading when extraction executes without force. |
|
||||
| `notarius.chunk_cache` | unset | Optional chunk-plan mode override: `auto`, `refresh`, or `bypass`; unset preserves Notarius configuration. |
|
||||
|
||||
These are typed, validated configuration fields, not arbitrary CLI arguments.
|
||||
They participate in existing profile composition, safe inspection, and source
|
||||
reporting. Do not introduce another secret, configuration, or cache resolver.
|
||||
|
||||
Notarius continues to own checkpoint recording, cache locations, compatibility
|
||||
checks, and cache formats. Operators enable recording and choose persistent
|
||||
cache directories in Notarius configuration. Narratio neither reads that
|
||||
configuration to infer settings nor writes a replacement configuration. A
|
||||
misconfigured resume request remains an actionable downstream error; do not
|
||||
silently retry without reuse, which could unexpectedly incur provider costs.
|
||||
|
||||
### Force Takes Precedence
|
||||
|
||||
| Extraction invocation | Checkpoint loading | Chunk-plan policy |
|
||||
| --- | --- | --- |
|
||||
| Ordinary execution, `resume: false` | No `--resume` | Configured Narratio override, or no flag |
|
||||
| Ordinary execution, `resume: true` | Pass `--resume` | Configured Narratio override, or no flag |
|
||||
| Forced extraction | Never pass `--resume` | Always pass `--chunk_cache refresh` |
|
||||
|
||||
Force regenerates the selected pipeline's chunk plan and objects rather than
|
||||
hydrating accepted checkpoint results. Checkpoint recording may still save the
|
||||
new work if enabled by Notarius. Force overrides even an explicit Narratio
|
||||
`chunk_cache: bypass` selection; document this precedence clearly.
|
||||
|
||||
“Clear the cache” means bypass the current input's cached chunk plan and replace
|
||||
it with a newly accepted plan, not recursively delete the Notarius cache root.
|
||||
Notarius's refresh operation publishes only an accepted replacement. A failed
|
||||
refresh need not erase the old plan. Other inputs' plans and unrelated checkpoint
|
||||
identities are left alone. No global purge, transactional regeneration, provider-
|
||||
side cache bypass, or guarantee of different generated text is implied.
|
||||
|
||||
Disabled extraction retains its ordinary explicit self-skip. The existing
|
||||
force scope and failure propagation remain unchanged: a failed extraction does
|
||||
not proceed to analysis.
|
||||
|
||||
## Stable Reference Snapshots
|
||||
|
||||
Use session-scoped, content-addressed reference snapshots at stable locations
|
||||
outside invocation-local run directories. Equivalent references in the same
|
||||
session must receive identical absolute snapshot paths across invocations;
|
||||
changed bytes must select different snapshot identities. Central path helpers
|
||||
own the layout, not adapter-local string construction.
|
||||
|
||||
Retain the useful protections of the existing snapshot mechanism:
|
||||
|
||||
- resolve inputs from manifest-backed prepared reference evidence;
|
||||
- validate content against its recorded checksum and size;
|
||||
- safely materialize missing snapshots and verify existing ones before reuse;
|
||||
- retain confinement and post-invocation verification rather than silently
|
||||
trusting the cache or substituting mutable prepared files;
|
||||
- use the existing session lock and narrow file-operation helpers, not a new
|
||||
general-purpose cache framework.
|
||||
|
||||
Snapshots are reconstructible local cache state, not canonical artifacts,
|
||||
published outputs, or credentials. Apply Narratio's ordinary group-shareable
|
||||
workspace policy. Preserve them across ordinary run cleanup so later extraction
|
||||
can reuse their locations; do not add garbage collection in this feature.
|
||||
Document their location, retention, and safe removal while no session operation
|
||||
is active. Missing snapshots can be reconstructed from valid prepared inputs.
|
||||
|
||||
Old invocation-local snapshots and checkpoints require no migration. A first
|
||||
execution under the stable layout may miss old checkpoint identities. Moving a
|
||||
workspace may likewise cause cold misses; cross-host or relocated-path cache
|
||||
identity is not promised.
|
||||
|
||||
## Two Distinct Resume Layers
|
||||
|
||||
Narratio remains responsible for deciding whether an existing complete extract
|
||||
result is current. Notarius remains responsible for reusing compatible internal
|
||||
work when an extraction process actually runs. Cached work must still produce
|
||||
an ordinary successful bundle that passes Narratio's existing validation and
|
||||
immutable promotion boundary.
|
||||
|
||||
The two new controls are operational settings, not blanket invalidation keys.
|
||||
Changing them alone does not stale a valid completed extraction. Explicit force
|
||||
is required to execute otherwise-current extraction with a fresh chunk plan.
|
||||
Preserve existing reference-content and transcript invalidation and the rule
|
||||
that changes hidden inside Notarius configuration require operator action.
|
||||
|
||||
Persistent upstream cache roots must not live inside disposable run output.
|
||||
Recommend explicitly configured Notarius cache directories for service
|
||||
deployments rather than assuming interactive-shell cache discovery is inherited
|
||||
through Narratio's minimal subprocess environment. Do not broaden environment
|
||||
inheritance or impose a new private-data policy on ordinary D&D material.
|
||||
|
||||
## Development-Focused Operator Guidance
|
||||
|
||||
Make the following distinction prominent at the beginning of the artifact
|
||||
development workflow in [Operations](../operations.md), with links to the
|
||||
[CLI Reference](../cli.md) for exact syntax:
|
||||
|
||||
| Operator intent | Existing command | Result |
|
||||
| --- | --- | --- |
|
||||
| Iterate on analysis prompts or artifacts using existing extracted data | `narratio analyze SESSION` | Force analysis only; preserve extract results and transcripts. |
|
||||
| Rebuild extraction and analysis | `narratio regenerate-artifacts SESSION` | Force extract through analyze; refresh chunking and avoid checkpoint loading. |
|
||||
| Continue incomplete artifact work with configured reuse | `narratio run SESSION --from extract --through analyze` | Reuse valid Narratio results; permit configured Notarius reuse when extraction executes. |
|
||||
|
||||
Explain `--artifacts` alongside these choices: it selects analysis targets and
|
||||
their required analysis prerequisites, not a subset of the Notarius bundle.
|
||||
The bounded commands require valid existing prefix-stage state; they do not
|
||||
repair or rerun transcript generation automatically. Publishing and retention
|
||||
remain separate actions.
|
||||
|
||||
`regenerate-artifacts` remains exactly a convenience expansion to
|
||||
`run SESSION --force --from extract --through analyze`. Do not give it private
|
||||
cache semantics. `analyze` remains the existing forced analysis-only command;
|
||||
no new alias is needed.
|
||||
|
||||
## Required Changes and Acceptance
|
||||
|
||||
- Extend the configuration owner and safe projections for the two fields.
|
||||
- Carry effective cache intent through extract's typed adapter request, using
|
||||
the existing force state; construct CLI flags only in the Notarius adapter.
|
||||
- Replace invocation-dependent reference delivery paths with the stable snapshot
|
||||
design without weakening prepared-input or bundle validation.
|
||||
- Update canonical configuration, integration, internal, example, and recovery
|
||||
documentation where owned. Put the workflow comparison front and center in
|
||||
Operations, not only in developer internals or release notes.
|
||||
- Cover default/explicit mode mapping, invalid modes, force precedence, and
|
||||
disabled extraction with focused behavioral tests.
|
||||
- Prove cross-invocation reuse with the supported Notarius behavior, including
|
||||
unchanged reference bytes at stable paths, changed reference content causing
|
||||
a miss, and retry after partial failure reusing accepted work.
|
||||
- Prove forced extraction regenerates rather than reuses internal work, while
|
||||
analysis-only invocation never launches Notarius or changes its results.
|
||||
- Verify cache loss is recoverable, run cleanup preserves reusable snapshots,
|
||||
and receipt validation, promotion, and failure handling remain intact.
|
||||
|
||||
Follow the [Testing Policy](../policy/testing.md): default tests remain offline
|
||||
and credential-free. An argv-only fake cannot prove upstream checkpoint reuse;
|
||||
use controlled upstream integration evidence with deterministic/local test
|
||||
modules. Do not make the default suite require a developer's installed Notarius
|
||||
or a paid provider, and do not duplicate Notarius's internal cache test matrix.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Notarius `--recompute-step` or lane-selective extraction.
|
||||
- A new Narratio cache-management command, cache format, or cache garbage collector.
|
||||
- Parsing downstream configuration or automatically enabling checkpoint recording.
|
||||
- Global cache deletion or bypassing provider-side caches.
|
||||
- A new release, upstream change, or migration of old cache contents in this task.
|
||||
- New cross-process locking for independent Notarius users or distributed cache
|
||||
coordination; upstream cache concurrency limitations remain applicable.
|
||||
|
||||
## Target End State
|
||||
|
||||
Operators can cheaply continue incomplete extraction using compatible Notarius
|
||||
work, deliberately regenerate the full extraction pipeline with force, or
|
||||
iterate only on analysis using the existing `analyze` command. Stable reference
|
||||
delivery makes reuse useful across Narratio invocations. Notarius owns its
|
||||
caches; Narratio owns orchestration, reference snapshots, and validated durable
|
||||
results. The operator guide makes the everyday development choices obvious.
|
||||
@@ -1,85 +0,0 @@
|
||||
# Notarius Cache Reuse Review Remediation
|
||||
|
||||
Status: Complete
|
||||
|
||||
Review scope: commits `5086596` through `a0312ed`, compared with the accepted
|
||||
cache roadmap and implementation plan. The core cache integration works,
|
||||
including the real Notarius v0.7.0 integration test. Address the following
|
||||
bounded findings; no broader cache redesign is required.
|
||||
|
||||
## 1. P2 — Preserve Relative Workspace Support at the Snapshot Boundary
|
||||
|
||||
Status: Complete. Snapshot materialization and post-run verification normalize
|
||||
their local session/cache roots to absolute paths. Regression coverage proves
|
||||
stable absolute adapter bindings, successful production-adapter launch, absence
|
||||
of the duplicated directory tree, and rejection of a symlink on the actual
|
||||
cache path for a relative workspace.
|
||||
|
||||
Location: `internal/stage/extract_references.go`, especially
|
||||
`extractReferenceSnapshotDestination` and `checkExtractSnapshotDirectory`;
|
||||
`internal/artifacts/paths.go` snapshot path construction.
|
||||
|
||||
The new snapshot destination inherits `SessionPaths.CacheDir` verbatim. A
|
||||
relative workspace root therefore produces a relative reference binding,
|
||||
whereas the previous extract implementation explicitly converted its snapshot
|
||||
root to an absolute path. The Notarius adapter requires absolute reference
|
||||
paths and rejects the new binding before launch. Configuration does not require
|
||||
an absolute workspace root.
|
||||
|
||||
There is a related path-composition defect: `checkExtractSnapshotDirectory`
|
||||
passes an already session-prefixed relative destination to
|
||||
`pathsafe.SlashRelativeFromRoot`, which interprets relative targets as relative
|
||||
to the supplied root. The resulting directory walk duplicates the session
|
||||
prefix instead of checking the actual destination ancestry. Subsequent copying
|
||||
can create the intended directories through a different path, leaving the
|
||||
wrong extra directory tree and bypassing the intended component checks there.
|
||||
|
||||
Evidence: a temporary focused test created a valid prepared party reference,
|
||||
expressed the session/cache roots relative to the working directory, and called
|
||||
the production snapshot materializer. It returned a binding beginning
|
||||
`work/campaign-a/.../cache/notarius/references/.../party.yml`, not an absolute
|
||||
path. The diagnostic test was removed after the review.
|
||||
|
||||
Required change:
|
||||
|
||||
- Normalize the session root and snapshot destination consistently to absolute
|
||||
paths before filesystem checks and adapter delivery, preserving the existing
|
||||
meaning of relative workspace configuration.
|
||||
- Ensure the component walk checks the same path subsequently read/written.
|
||||
- Do not fix this by requiring operators to configure absolute workspace roots
|
||||
or by changing global path semantics unnecessarily.
|
||||
- Add a regression using a relative workspace, real snapshot materialization,
|
||||
and the adapter validation boundary. Verify absolute stable bindings across
|
||||
invocations, no duplicated session-prefix directory, and rejection of an
|
||||
unsafe component on the actual cache path.
|
||||
|
||||
## 2. P3 — Remove the Accidental Reference-Version Requirement
|
||||
|
||||
Status: Complete. The integration contract now distinguishes the v0.6.0
|
||||
reference capability from selected cache flags and the tested v0.7.0 baseline,
|
||||
without introducing a runtime gate.
|
||||
|
||||
Location: `docs/integrations/notarius.md`, Invocation section, line 28.
|
||||
|
||||
The documentation now says Narratio requires the Notarius v0.7.0 CLI contract
|
||||
when references are configured. References were already supported under the
|
||||
documented v0.6.0 contract. The implementation plan selected v0.7.0 as the
|
||||
tested cache integration baseline and explicitly did not require rejecting
|
||||
older compatible executables. The revised sentence unnecessarily suggests an
|
||||
upgrade is mandatory for existing reference-only use.
|
||||
|
||||
Required change: distinguish the tested v0.7.0 cache integration baseline from
|
||||
the reference CLI contract. Document required capabilities for the selected
|
||||
flags without claiming a new unconditional minimum version. Keep runtime
|
||||
behavior unchanged; do not add a version probe or gate.
|
||||
|
||||
## Validation Recorded
|
||||
|
||||
- `GOWORK=off go test ./...` passed.
|
||||
- Race tests for stage, app, config, artifacts, and Notarius adapter passed.
|
||||
- `GOWORK=off go vet ./...` passed.
|
||||
- The opt-in `TestNotariusCacheReuseIntegration` passed against the previously
|
||||
built local v0.7.0 binary with its local test provider, covering actual reuse,
|
||||
reference-change misses, partial recovery, and forced chunk refresh.
|
||||
- The additional relative-path diagnostic failed as described above; this case
|
||||
was not covered by the existing absolute-path fixtures.
|
||||
Reference in New Issue
Block a user