Plan Notarius cache reuse
This commit is contained in:
292
docs/roadmap/implementation.md
Normal file
292
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,292 @@
|
|||||||
|
# Notarius Cache Reuse Implementation Plan
|
||||||
|
|
||||||
|
Status: Not started
|
||||||
|
|
||||||
|
Implement the [feature roadmap](notarius-cache-reuse.md) in the numbered order
|
||||||
|
below. Each stage is a bounded assignment for a gpt-5.6-terra coding agent.
|
||||||
|
Record completion and checks beside each stage; intermediate stages are not a
|
||||||
|
claim that the whole feature is ready. No release, tag, push, upstream edit, or
|
||||||
|
dependency upgrade is authorized by this plan.
|
||||||
|
|
||||||
|
## Instructions and Settled Decisions
|
||||||
|
|
||||||
|
Read `docs/development.md`, every document in `docs/policy/`, the feature
|
||||||
|
roadmap, and the sources listed for the current stage. Inspect callers and
|
||||||
|
existing tests before editing. File lists identify owners, not edit allowlists.
|
||||||
|
Use existing path, file-operation, configuration, and subprocess boundaries.
|
||||||
|
Do not create a general cache framework or change unrelated safety policies.
|
||||||
|
|
||||||
|
- Add `Resume bool` and `ChunkCache string` to `config.NotariusConfig`, using
|
||||||
|
YAML/JSON names `resume` and `chunk_cache` with `omitempty`. False and the empty
|
||||||
|
string are the unset defaults. Null resets to the zero value through existing
|
||||||
|
ConfigKit behavior. For enabled Notarius configuration, trim surrounding
|
||||||
|
whitespace and accept only empty, `auto`, `bypass`, or `refresh`; do not add
|
||||||
|
case-insensitive aliases. Preserve the existing disabled-section validation
|
||||||
|
policy rather than introducing new mandatory configuration checks there.
|
||||||
|
- Ordinary extraction passes the configured resume boolean and chunk mode.
|
||||||
|
Forced extraction always passes resume false and chunk mode `refresh`, even
|
||||||
|
if configuration requests bypass. The stage resolves force precedence; the
|
||||||
|
adapter translates effective intent into flags. No private alias behavior.
|
||||||
|
- Use stable snapshot paths under
|
||||||
|
`<session-root>/cache/notarius/references/<sha256>/<prepared-basename>`.
|
||||||
|
SHA-256 is the full lowercase hexadecimal digest of the reference bytes.
|
||||||
|
The basename preserves the prepared file extension and is validated as one
|
||||||
|
path component. Identity contains neither run ID nor timestamps. Different
|
||||||
|
sources with identical bytes and basename can share a verified snapshot.
|
||||||
|
Same basename with different bytes cannot collide. This is an implementation
|
||||||
|
layout, not a new external artifact schema.
|
||||||
|
- Keep prepared-input identities, metadata, and extract resume fingerprints
|
||||||
|
unchanged. Snapshot paths are delivery details. Neither new setting belongs
|
||||||
|
in semantic resume evidence; safe effective configuration can expose them.
|
||||||
|
- A missing snapshot is reconstructed. A corrupt regular snapshot is atomically
|
||||||
|
replaced from a currently verified prepared source. Unsafe paths, symlinks,
|
||||||
|
directories, or other non-regular conflicts fail with the affected path and
|
||||||
|
recovery guidance, never recursive deletion. A prepared-source mismatch
|
||||||
|
remains an error, not a cache miss. No eviction or cache ledger is needed.
|
||||||
|
- Snapshots are ordinary group-shareable workspace files, not API-key material.
|
||||||
|
They survive ordinary run cleanup, are not published/restored as artifacts,
|
||||||
|
and do not become prerequisites for reuse of a complete durable extract bundle.
|
||||||
|
- Notarius owns checkpoint enablement, roots, recording, and validity. Do not
|
||||||
|
read its config to infer enablement, silently retry without resume, or widen
|
||||||
|
inherited environment. Recommend explicit persistent upstream cache roots.
|
||||||
|
- Cache refresh is not deletion: regenerate the current input's plan and let
|
||||||
|
Notarius publish an accepted replacement. Leave other plans/checkpoints alone.
|
||||||
|
No guarantee of different text, provider-side cache bypass, or transactional
|
||||||
|
regeneration is introduced.
|
||||||
|
- The real integration acceptance baseline is Notarius v0.7.0. Do not add a
|
||||||
|
runtime version probe or reject older compatible executables just by version.
|
||||||
|
Unsupported flags fail through the existing subprocess error path.
|
||||||
|
|
||||||
|
## Stage 1 — Add Typed Cache Controls
|
||||||
|
|
||||||
|
Status: Not started
|
||||||
|
|
||||||
|
Read: `internal/config/config.go`, `credential_projection.go`,
|
||||||
|
`pipeline_normalize.go`, `validate_notarius.go`, `notarius_test.go`,
|
||||||
|
`pipeline_configkit_test.go`, `effective_projection.go`; configuration sections
|
||||||
|
of `docs/config.md` and `docs/internal/configuration.md`.
|
||||||
|
|
||||||
|
1. Add the two fields and implement the zero-value, normalization, and validation
|
||||||
|
decisions above. Keep defaults centralized; no stage-specific environment
|
||||||
|
overrides or raw extra-argument map.
|
||||||
|
2. Extend `NotariusConfig.MarshalYAML` so direct serialization retains both
|
||||||
|
fields and still projects only the credential reference name. Confirm that
|
||||||
|
effective projection, source reporting, copying, and profile overrides carry
|
||||||
|
the fields without adding another projection layer.
|
||||||
|
3. Extend focused configuration tests for absence, explicit modes, invalid
|
||||||
|
enabled modes, null reset, profile replacement, and safe YAML/JSON inspection.
|
||||||
|
Protect the existing optional credential behavior. Do not snapshot complete
|
||||||
|
error wording or repeat every case at the CLI layer.
|
||||||
|
|
||||||
|
Acceptance: `GOWORK=off go test ./internal/config` and
|
||||||
|
`GOWORK=off go test ./...` pass. Existing configurations require no edits.
|
||||||
|
This stage alone does not enable delivery and is not a finished feature.
|
||||||
|
|
||||||
|
## Stage 2 — Forward Cache Intent and Apply Force Precedence
|
||||||
|
|
||||||
|
Status: Not started
|
||||||
|
|
||||||
|
Read: `internal/stage/extract.go`, `stage.go`, `extract_test.go`,
|
||||||
|
`semantic_contracts_delivery.go`, `semantic_contracts_delivery_test.go`;
|
||||||
|
`internal/adapters/notarius/runner.go`, `subprocess.go`, `subprocess_test.go`,
|
||||||
|
`fake.go`; `docs/internal/adapters.md`, `docs/integrations/notarius.md`.
|
||||||
|
|
||||||
|
1. Add `Resume bool` and `ChunkCache string` to the typed Notarius request.
|
||||||
|
Populate them in extract, applying `env.Force` precedence in one place.
|
||||||
|
2. Append `--resume` only for true and `--chunk_cache <mode>` only for nonempty
|
||||||
|
mode. Preserve existing arguments, references, credentials, diagnostics,
|
||||||
|
receipt parsing, and bundle discovery. Validate programmatic nonempty modes
|
||||||
|
before starting a subprocess; do not import application config into the
|
||||||
|
adapter merely to share a trivial enum check.
|
||||||
|
3. Test the effective request matrix with ordinary and forced execution,
|
||||||
|
including force overriding explicit bypass and resume. Adapter tests own the
|
||||||
|
exact flag mapping and rejection of invalid direct requests. Keep a test for
|
||||||
|
absent settings producing the previous ordinary argv. Disabled extraction
|
||||||
|
still never invokes the adapter.
|
||||||
|
4. Add a focused semantic-fingerprint exclusion test for the two controls; do
|
||||||
|
not bump the fingerprint version or change manifest schema or invalidation.
|
||||||
|
|
||||||
|
Acceptance: focused normal and race tests for `./internal/stage` and
|
||||||
|
`./internal/adapters/notarius` pass. No cache deletion or upstream config parsing
|
||||||
|
has been introduced. Existing credential-redaction tests remain green.
|
||||||
|
|
||||||
|
## Stage 3 — Replace Run-Local Reference Snapshots with Stable Verified Snapshots
|
||||||
|
|
||||||
|
Status: Not started
|
||||||
|
|
||||||
|
Read: `internal/artifacts/paths.go` and its tests;
|
||||||
|
`internal/stage/extract_references.go`, `extract.go`, `extract_resume.go`,
|
||||||
|
`extract_test.go` and reference-focused tests; existing root-confined helpers in
|
||||||
|
`internal/pathsafe/` and atomic/checksum helpers in `internal/fileops/`;
|
||||||
|
`docs/internal/artifacts.md`, `docs/internal/workspace.md`.
|
||||||
|
|
||||||
|
1. Add canonical artifact/path-owner helpers for the cache root and confined
|
||||||
|
snapshot destination. Reuse the package's existing path-segment conventions
|
||||||
|
and central constants; validate digest syntax before using it as a segment.
|
||||||
|
Do not embed new layout construction independently in materialization and
|
||||||
|
post-run verification.
|
||||||
|
2. Refactor `materializeExtractReferenceSnapshots` to resolve stable destinations.
|
||||||
|
Verify an existing regular file's size and SHA-256 before reuse. For missing
|
||||||
|
or corrupt regular entries, copy from the manifest-verified source using the
|
||||||
|
existing atomic/checksum primitive, checking the result before invocation.
|
||||||
|
Deduplicate equivalent destinations within the request. Check path components
|
||||||
|
before directory creation/opening using existing confinement facilities.
|
||||||
|
3. If a newly written entry fails verification because its source changed, do
|
||||||
|
not invoke Notarius. Remove only that invalid, newly produced cache file
|
||||||
|
through a confined operation if safe; do not remove the root or other entries.
|
||||||
|
Preserve an actionable prepared-source error.
|
||||||
|
4. Make post-run verification use the same destination resolver and reject
|
||||||
|
mutated, missing, symlinked, or non-regular snapshots before bundle promotion.
|
||||||
|
Continue checking actual content, not just names or modification times.
|
||||||
|
5. Remove obsolete runtime use of the run-ID snapshot path helper. Remove the
|
||||||
|
helper if no callers remain; no compatibility shim or migration is needed.
|
||||||
|
Leave old run files alone. Do not change current prepared-reference metadata,
|
||||||
|
recorded source paths, semantic evidence, or durable bundle locations.
|
||||||
|
6. Adapt existing reference tests rather than keeping parallel old/new suites.
|
||||||
|
Cover stable paths across run IDs, changed content, same basename/different
|
||||||
|
content, verified reuse, reconstruction after deletion, corrupt regular-file
|
||||||
|
repair, unsafe conflicts, and post-invocation mutation rejection. Assert
|
||||||
|
behavior and confinement, not incidental helper choreography.
|
||||||
|
|
||||||
|
Acceptance: `GOWORK=off go test ./internal/artifacts ./internal/stage
|
||||||
|
./internal/fileops ./internal/pathsafe` (on one shell line) and race tests for
|
||||||
|
artifacts/stage pass. No new general cache abstraction, lock service, retention
|
||||||
|
policy, or permission regime was added.
|
||||||
|
|
||||||
|
## Stage 4 — Verify Session Lifecycle and Command Boundaries
|
||||||
|
|
||||||
|
Status: Not started
|
||||||
|
|
||||||
|
Read: `internal/app/extract_lifecycle_test.go`, `regenerate_artifacts.go`,
|
||||||
|
`regenerate_artifacts_test.go`, `semantic_resume_test.go`,
|
||||||
|
`cleanup_orchestration_test.go`; `internal/stage/cleanup_plan.go`,
|
||||||
|
`cleanup_test.go`; publish/restore enumeration owners identified through their
|
||||||
|
callers; `docs/operations.md` and `docs/cli.md`.
|
||||||
|
|
||||||
|
1. Extend the existing lifecycle fixture to prove a failed extraction followed
|
||||||
|
by a new invocation delivers identical snapshot paths for unchanged prepared
|
||||||
|
references. A changed valid reference selects a new path. Keep old snapshots
|
||||||
|
available; they are not current artifact evidence.
|
||||||
|
2. Verify a successful complete extract remains reusable after cache-control
|
||||||
|
changes or deletion of its local snapshots. No snapshot directory should be
|
||||||
|
created merely to plan or skip extraction.
|
||||||
|
3. Exercise ordinary successful cleanup with snapshots present: run-local
|
||||||
|
output is removed as before while snapshots survive. Confirm publish and
|
||||||
|
restore selection do not include the new cache subtree; add a narrow
|
||||||
|
regression where existing enumeration could accidentally include it, not a
|
||||||
|
second publish/restore test framework.
|
||||||
|
4. Extend existing command/lifecycle tests to show `analyze SESSION` never
|
||||||
|
invokes Notarius and preserves extract outputs, whereas
|
||||||
|
`regenerate-artifacts SESSION` uses the ordinary forced extract/analyze path.
|
||||||
|
`--artifacts` continues to narrow analysis, not the extraction bundle.
|
||||||
|
5. Confirm errors still stop before analysis and leave no partially promoted
|
||||||
|
current bundle. Reuse existing failure/promotion coverage where sufficient.
|
||||||
|
|
||||||
|
Acceptance: normal and race tests for `./internal/app ./internal/stage` pass.
|
||||||
|
No new alias semantics, CLI options, manifest schema, or unrelated lifecycle
|
||||||
|
changes. This stage proves Narratio orchestration, not upstream cache hits.
|
||||||
|
|
||||||
|
## Stage 5 — Prove Reuse with Real Notarius and a Local Test Provider
|
||||||
|
|
||||||
|
Status: Not started
|
||||||
|
|
||||||
|
Read Notarius v0.7.0 `docs/cli.md`, `docs/config.md`, `docs/operations.md`,
|
||||||
|
`examples/dnd-minimal.config.yml`, `examples/seriatim-minimal-transcript.json`,
|
||||||
|
`internal/cli/production_contract_test.go` (local-backend example), and
|
||||||
|
`internal/framework/checkpoint/identity.go`. Inspect the selected built-in
|
||||||
|
module's schema and reference slots rather than inventing output payloads.
|
||||||
|
Read Narratio's existing extraction lifecycle and adapter test fixtures.
|
||||||
|
|
||||||
|
1. Add an explicitly opt-in Go integration test, invoked with an absolute
|
||||||
|
`NARRATIO_TEST_NOTARIUS_BINARY` path. If unset, skip with a clear reason; if
|
||||||
|
set but unusable, fail. Document the focused `go test -run` invocation.
|
||||||
|
The normal suite must not discover Notarius on PATH, build/download upstream,
|
||||||
|
or depend on the sibling checkout. Do not add Notarius to `go.mod`.
|
||||||
|
2. Run the real executable with a small built-in D&D pipeline and its ordinary
|
||||||
|
JSON output module. Use a local `httptest` OpenAI-compatible chat-completions
|
||||||
|
endpoint via Notarius's `promptkit.local_backend`, a `backend: local` profile,
|
||||||
|
and schema-valid deterministic responses. No real provider, credentials, or
|
||||||
|
upstream test-only module registration is necessary. Use explicit temporary
|
||||||
|
config, profile, cache, and output roots; keep endpoint/profile content stable
|
||||||
|
across the compared invocations because they affect checkpoint identity.
|
||||||
|
3. Drive production Narratio snapshot delivery and the production adapter from
|
||||||
|
the fixture, not a hand-assembled imitation of their argv. Use different run
|
||||||
|
IDs/output roots and at least one declared prepared reference. Count local
|
||||||
|
model requests and inspect upstream checkpoint decision codes: first execution
|
||||||
|
populates state; second ordinary resume reuses accepted work; changing valid
|
||||||
|
reference bytes causes execution rather than a false hit.
|
||||||
|
4. Add a deterministic partial-failure scenario with accepted predecessor work:
|
||||||
|
configure ordered work and return a controlled failure for a later producer,
|
||||||
|
including any bounded retries. Keep configuration and endpoint unchanged on
|
||||||
|
retry; restore only server behavior. Verify accepted earlier work is reused
|
||||||
|
and the failed work is executed. Do not infer reuse from equal output bytes.
|
||||||
|
5. Exercise forced extraction against populated state. Verify checkpoint loading
|
||||||
|
is disabled and work executes again. Demonstrate actual chunk-plan refresh
|
||||||
|
through Notarius's chunk provenance/cache behavior in addition to the adapter
|
||||||
|
flag test; deterministic plans need not differ in bytes. Check that an
|
||||||
|
unrelated cached input remains untouched.
|
||||||
|
6. Run against a locally built v0.7.0 executable (or a documented compatible
|
||||||
|
newer version) without modifying or tagging the upstream repository. Record
|
||||||
|
version, focused command, and results in this stage's completion note. Keep
|
||||||
|
test fixtures small and scoped; do not reproduce upstream's whole cache suite.
|
||||||
|
|
||||||
|
Acceptance: the opt-in test passes without internet/provider calls, ordinary
|
||||||
|
`go test ./...` remains independent of upstream installation, and there is real
|
||||||
|
evidence of cache reuse across Narratio invocations. A skipped integration test
|
||||||
|
does not complete this stage. If upstream behavior blocks the stated contract,
|
||||||
|
record the evidence and stop for direction rather than weakening the test or
|
||||||
|
silently modifying upstream.
|
||||||
|
|
||||||
|
## Stage 6 — Document the Workflow and Complete Validation
|
||||||
|
|
||||||
|
Status: Not started
|
||||||
|
|
||||||
|
Read: `docs/operations.md`, `docs/cli.md`, `docs/config.md`,
|
||||||
|
`docs/integrations/notarius.md`, `docs/internal/stage-extract.md`,
|
||||||
|
`docs/internal/adapters.md`, `docs/internal/workspace.md`,
|
||||||
|
`docs/troubleshooting.md`, `examples/README.md`, `docs/release.md`.
|
||||||
|
|
||||||
|
1. Put the roadmap's three-way development workflow comparison prominently at
|
||||||
|
the start of Operations' artifact-development section: analysis only,
|
||||||
|
extraction plus analysis regeneration, and ordinary bounded continuation.
|
||||||
|
Explain preservation of transcripts/extraction, analysis selection, and
|
||||||
|
separate publishing. Link to CLI-owned exact syntax and avoid redefining it.
|
||||||
|
2. Update field/default/force precedence documentation and a maintained example.
|
||||||
|
Clearly explain upstream checkpoint enablement, explicit persistent roots,
|
||||||
|
cold misses from changed references/profile/paths, and the distinction between
|
||||||
|
Narratio skipping a stage and Notarius reusing work inside an executed stage.
|
||||||
|
3. Document stable snapshot location/retention, regular corruption recovery,
|
||||||
|
unsafe-path errors, and manual removal only while the session is inactive.
|
||||||
|
Do not add a cache purge command, auto-GC, mandatory private permissions, or
|
||||||
|
promise that all changed configuration is visible to Narratio resume.
|
||||||
|
4. Record the tested Notarius integration baseline and opt-in test instructions.
|
||||||
|
Preserve already-published release notes; leave version choice and release
|
||||||
|
publication to a separately authorized task.
|
||||||
|
5. Run the checks below. Mark the roadmap and stages complete only when all
|
||||||
|
required evidence, including Stage 5, is present. Record any environmental
|
||||||
|
limitations accurately; do not wait for CI or assert native macOS validation
|
||||||
|
based on cross-compilation.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
GOWORK=off go mod tidy -diff
|
||||||
|
GOWORK=off go mod verify
|
||||||
|
GOWORK=off go test -count=1 ./...
|
||||||
|
GOWORK=off go test -race -count=1 ./...
|
||||||
|
GOWORK=off go vet ./...
|
||||||
|
GOWORK=off go build ./...
|
||||||
|
GOWORK=off go test ./internal/doccheck
|
||||||
|
GOWORK=off go test ./internal/config -run '^TestExamplesLoadAndValidate$'
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
Check changed Go files with `gofmt -l`. Run the repository's
|
||||||
|
`scripts/build-release-assets.sh v0.0.0-test OUTPUT_DIR` into an empty absolute
|
||||||
|
directory created with `mktemp -d` to check the three maintained targets, without
|
||||||
|
publishing assets or tags. Keep any temporary binaries outside the worktree.
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
None requiring an operator decision. Defaults, force precedence, cache ownership,
|
||||||
|
snapshot lifetime, and the command distinction are settled. The exact test
|
||||||
|
payloads are implementation details governed by the real upstream schemas, not
|
||||||
|
new product policy. Stage 5 must establish actual reuse before completion.
|
||||||
212
docs/roadmap/notarius-cache-reuse.md
Normal file
212
docs/roadmap/notarius-cache-reuse.md
Normal file
@@ -0,0 +1,212 @@
|
|||||||
|
# Notarius Cache Reuse and Artifact Development
|
||||||
|
|
||||||
|
Status: Accepted direction — implementation pending
|
||||||
|
|
||||||
|
## Goal and User Intent
|
||||||
|
|
||||||
|
Allow Narratio to reuse Notarius's cached chunk plans and accepted intermediate
|
||||||
|
artifacts when extraction actually executes, particularly when retrying failed
|
||||||
|
work. Preserve an explicit force path that regenerates the selected extraction
|
||||||
|
pipeline's work. Keep transcript generation separate from iterative artifact
|
||||||
|
development, and prominently explain the existing analysis-only command.
|
||||||
|
|
||||||
|
This roadmap defines the desired end state, not an implementation sequence.
|
||||||
|
The [implementation plan](implementation.md) owns staged execution and checks.
|
||||||
|
It builds on the local Notarius v0.7.0 CLI and cache contracts reviewed for this
|
||||||
|
proposal. No upstream changes are required by the proposed design.
|
||||||
|
|
||||||
|
## Current Behavior and Underlying Issue
|
||||||
|
|
||||||
|
Narratio's manifest-based resume can skip an already-completed extract stage.
|
||||||
|
When that stage must execute, however, the Notarius adapter never passes
|
||||||
|
`--resume`, so Notarius cannot load compatible execution checkpoints.
|
||||||
|
|
||||||
|
Notarius has two distinct mechanisms:
|
||||||
|
|
||||||
|
- Chunk-plan caching defaults to `auto`. Valid source-addressed plans can
|
||||||
|
already be reused through Notarius configuration. `--chunk_cache` overrides
|
||||||
|
the configured mode with `auto`, `refresh`, or `bypass`.
|
||||||
|
- Checkpoint recording is disabled by default and is enabled in Notarius with
|
||||||
|
`cache.checkpoints.enabled: true`. Recording-enabled runs save accepted work
|
||||||
|
even without `--resume`; that flag separately permits checkpoint loading.
|
||||||
|
Asking for resume while recording is disabled fails.
|
||||||
|
|
||||||
|
There is also a reference-identity obstacle. Narratio supplies references via
|
||||||
|
invocation-local snapshots, whose paths change with each run ID. Notarius's
|
||||||
|
checkpoint identity includes reference origin URIs as well as content digests.
|
||||||
|
Simply adding `--resume` therefore does not establish reuse across otherwise
|
||||||
|
equivalent invocations that supply those references.
|
||||||
|
|
||||||
|
Relevant upstream owners are Notarius's `docs/cli.md`, `docs/config.md`, and
|
||||||
|
`docs/operations.md`; reference fingerprint construction is in
|
||||||
|
`internal/framework/checkpoint/identity.go`. Narratio's existing contracts are
|
||||||
|
documented in [Extract Internals](../internal/stage-extract.md) and the
|
||||||
|
[Notarius Integration](../integrations/notarius.md).
|
||||||
|
|
||||||
|
## Configuration and Invocation Contract
|
||||||
|
|
||||||
|
Add two optional fields to Narratio's Notarius section:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
notarius:
|
||||||
|
resume: true
|
||||||
|
chunk_cache: auto
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Default | Meaning |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `notarius.resume` | `false` | Permit compatible Notarius checkpoint loading when extraction executes without force. |
|
||||||
|
| `notarius.chunk_cache` | unset | Optional chunk-plan mode override: `auto`, `refresh`, or `bypass`; unset preserves Notarius configuration. |
|
||||||
|
|
||||||
|
These are typed, validated configuration fields, not arbitrary CLI arguments.
|
||||||
|
They participate in existing profile composition, safe inspection, and source
|
||||||
|
reporting. Do not introduce another secret, configuration, or cache resolver.
|
||||||
|
|
||||||
|
Notarius continues to own checkpoint recording, cache locations, compatibility
|
||||||
|
checks, and cache formats. Operators enable recording and choose persistent
|
||||||
|
cache directories in Notarius configuration. Narratio neither reads that
|
||||||
|
configuration to infer settings nor writes a replacement configuration. A
|
||||||
|
misconfigured resume request remains an actionable downstream error; do not
|
||||||
|
silently retry without reuse, which could unexpectedly incur provider costs.
|
||||||
|
|
||||||
|
### Force Takes Precedence
|
||||||
|
|
||||||
|
| Extraction invocation | Checkpoint loading | Chunk-plan policy |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Ordinary execution, `resume: false` | No `--resume` | Configured Narratio override, or no flag |
|
||||||
|
| Ordinary execution, `resume: true` | Pass `--resume` | Configured Narratio override, or no flag |
|
||||||
|
| Forced extraction | Never pass `--resume` | Always pass `--chunk_cache refresh` |
|
||||||
|
|
||||||
|
Force regenerates the selected pipeline's chunk plan and objects rather than
|
||||||
|
hydrating accepted checkpoint results. Checkpoint recording may still save the
|
||||||
|
new work if enabled by Notarius. Force overrides even an explicit Narratio
|
||||||
|
`chunk_cache: bypass` selection; document this precedence clearly.
|
||||||
|
|
||||||
|
“Clear the cache” means bypass the current input's cached chunk plan and replace
|
||||||
|
it with a newly accepted plan, not recursively delete the Notarius cache root.
|
||||||
|
Notarius's refresh operation publishes only an accepted replacement. A failed
|
||||||
|
refresh need not erase the old plan. Other inputs' plans and unrelated checkpoint
|
||||||
|
identities are left alone. No global purge, transactional regeneration, provider-
|
||||||
|
side cache bypass, or guarantee of different generated text is implied.
|
||||||
|
|
||||||
|
Disabled extraction retains its ordinary explicit self-skip. The existing
|
||||||
|
force scope and failure propagation remain unchanged: a failed extraction does
|
||||||
|
not proceed to analysis.
|
||||||
|
|
||||||
|
## Stable Reference Snapshots
|
||||||
|
|
||||||
|
Use session-scoped, content-addressed reference snapshots at stable locations
|
||||||
|
outside invocation-local run directories. Equivalent references in the same
|
||||||
|
session must receive identical absolute snapshot paths across invocations;
|
||||||
|
changed bytes must select different snapshot identities. Central path helpers
|
||||||
|
own the layout, not adapter-local string construction.
|
||||||
|
|
||||||
|
Retain the useful protections of the existing snapshot mechanism:
|
||||||
|
|
||||||
|
- resolve inputs from manifest-backed prepared reference evidence;
|
||||||
|
- validate content against its recorded checksum and size;
|
||||||
|
- safely materialize missing snapshots and verify existing ones before reuse;
|
||||||
|
- retain confinement and post-invocation verification rather than silently
|
||||||
|
trusting the cache or substituting mutable prepared files;
|
||||||
|
- use the existing session lock and narrow file-operation helpers, not a new
|
||||||
|
general-purpose cache framework.
|
||||||
|
|
||||||
|
Snapshots are reconstructible local cache state, not canonical artifacts,
|
||||||
|
published outputs, or credentials. Apply Narratio's ordinary group-shareable
|
||||||
|
workspace policy. Preserve them across ordinary run cleanup so later extraction
|
||||||
|
can reuse their locations; do not add garbage collection in this feature.
|
||||||
|
Document their location, retention, and safe removal while no session operation
|
||||||
|
is active. Missing snapshots can be reconstructed from valid prepared inputs.
|
||||||
|
|
||||||
|
Old invocation-local snapshots and checkpoints require no migration. A first
|
||||||
|
execution under the stable layout may miss old checkpoint identities. Moving a
|
||||||
|
workspace may likewise cause cold misses; cross-host or relocated-path cache
|
||||||
|
identity is not promised.
|
||||||
|
|
||||||
|
## Two Distinct Resume Layers
|
||||||
|
|
||||||
|
Narratio remains responsible for deciding whether an existing complete extract
|
||||||
|
result is current. Notarius remains responsible for reusing compatible internal
|
||||||
|
work when an extraction process actually runs. Cached work must still produce
|
||||||
|
an ordinary successful bundle that passes Narratio's existing validation and
|
||||||
|
immutable promotion boundary.
|
||||||
|
|
||||||
|
The two new controls are operational settings, not blanket invalidation keys.
|
||||||
|
Changing them alone does not stale a valid completed extraction. Explicit force
|
||||||
|
is required to execute otherwise-current extraction with a fresh chunk plan.
|
||||||
|
Preserve existing reference-content and transcript invalidation and the rule
|
||||||
|
that changes hidden inside Notarius configuration require operator action.
|
||||||
|
|
||||||
|
Persistent upstream cache roots must not live inside disposable run output.
|
||||||
|
Recommend explicitly configured Notarius cache directories for service
|
||||||
|
deployments rather than assuming interactive-shell cache discovery is inherited
|
||||||
|
through Narratio's minimal subprocess environment. Do not broaden environment
|
||||||
|
inheritance or impose a new private-data policy on ordinary D&D material.
|
||||||
|
|
||||||
|
## Development-Focused Operator Guidance
|
||||||
|
|
||||||
|
Make the following distinction prominent at the beginning of the artifact
|
||||||
|
development workflow in [Operations](../operations.md), with links to the
|
||||||
|
[CLI Reference](../cli.md) for exact syntax:
|
||||||
|
|
||||||
|
| Operator intent | Existing command | Result |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Iterate on analysis prompts or artifacts using existing extracted data | `narratio analyze SESSION` | Force analysis only; preserve extract results and transcripts. |
|
||||||
|
| Rebuild extraction and analysis | `narratio regenerate-artifacts SESSION` | Force extract through analyze; refresh chunking and avoid checkpoint loading. |
|
||||||
|
| Continue incomplete artifact work with configured reuse | `narratio run SESSION --from extract --through analyze` | Reuse valid Narratio results; permit configured Notarius reuse when extraction executes. |
|
||||||
|
|
||||||
|
Explain `--artifacts` alongside these choices: it selects analysis targets and
|
||||||
|
their required analysis prerequisites, not a subset of the Notarius bundle.
|
||||||
|
The bounded commands require valid existing prefix-stage state; they do not
|
||||||
|
repair or rerun transcript generation automatically. Publishing and retention
|
||||||
|
remain separate actions.
|
||||||
|
|
||||||
|
`regenerate-artifacts` remains exactly a convenience expansion to
|
||||||
|
`run SESSION --force --from extract --through analyze`. Do not give it private
|
||||||
|
cache semantics. `analyze` remains the existing forced analysis-only command;
|
||||||
|
no new alias is needed.
|
||||||
|
|
||||||
|
## Required Changes and Acceptance
|
||||||
|
|
||||||
|
- Extend the configuration owner and safe projections for the two fields.
|
||||||
|
- Carry effective cache intent through extract's typed adapter request, using
|
||||||
|
the existing force state; construct CLI flags only in the Notarius adapter.
|
||||||
|
- Replace invocation-dependent reference delivery paths with the stable snapshot
|
||||||
|
design without weakening prepared-input or bundle validation.
|
||||||
|
- Update canonical configuration, integration, internal, example, and recovery
|
||||||
|
documentation where owned. Put the workflow comparison front and center in
|
||||||
|
Operations, not only in developer internals or release notes.
|
||||||
|
- Cover default/explicit mode mapping, invalid modes, force precedence, and
|
||||||
|
disabled extraction with focused behavioral tests.
|
||||||
|
- Prove cross-invocation reuse with the supported Notarius behavior, including
|
||||||
|
unchanged reference bytes at stable paths, changed reference content causing
|
||||||
|
a miss, and retry after partial failure reusing accepted work.
|
||||||
|
- Prove forced extraction regenerates rather than reuses internal work, while
|
||||||
|
analysis-only invocation never launches Notarius or changes its results.
|
||||||
|
- Verify cache loss is recoverable, run cleanup preserves reusable snapshots,
|
||||||
|
and receipt validation, promotion, and failure handling remain intact.
|
||||||
|
|
||||||
|
Follow the [Testing Policy](../policy/testing.md): default tests remain offline
|
||||||
|
and credential-free. An argv-only fake cannot prove upstream checkpoint reuse;
|
||||||
|
use controlled upstream integration evidence with deterministic/local test
|
||||||
|
modules. Do not make the default suite require a developer's installed Notarius
|
||||||
|
or a paid provider, and do not duplicate Notarius's internal cache test matrix.
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- Notarius `--recompute-step` or lane-selective extraction.
|
||||||
|
- A new Narratio cache-management command, cache format, or cache garbage collector.
|
||||||
|
- Parsing downstream configuration or automatically enabling checkpoint recording.
|
||||||
|
- Global cache deletion or bypassing provider-side caches.
|
||||||
|
- A new release, upstream change, or migration of old cache contents in this task.
|
||||||
|
- New cross-process locking for independent Notarius users or distributed cache
|
||||||
|
coordination; upstream cache concurrency limitations remain applicable.
|
||||||
|
|
||||||
|
## Target End State
|
||||||
|
|
||||||
|
Operators can cheaply continue incomplete extraction using compatible Notarius
|
||||||
|
work, deliberately regenerate the full extraction pipeline with force, or
|
||||||
|
iterate only on analysis using the existing `analyze` command. Stable reference
|
||||||
|
delivery makes reuse useful across Narratio invocations. Notarius owns its
|
||||||
|
caches; Narratio owns orchestration, reference snapshots, and validated durable
|
||||||
|
results. The operator guide makes the everyday development choices obvious.
|
||||||
Reference in New Issue
Block a user