Plan Notarius cache reuse
This commit is contained in:
292
docs/roadmap/implementation.md
Normal file
292
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,292 @@
|
||||
# Notarius Cache Reuse Implementation Plan
|
||||
|
||||
Status: Not started
|
||||
|
||||
Implement the [feature roadmap](notarius-cache-reuse.md) in the numbered order
|
||||
below. Each stage is a bounded assignment for a gpt-5.6-terra coding agent.
|
||||
Record completion and checks beside each stage; intermediate stages are not a
|
||||
claim that the whole feature is ready. No release, tag, push, upstream edit, or
|
||||
dependency upgrade is authorized by this plan.
|
||||
|
||||
## Instructions and Settled Decisions
|
||||
|
||||
Read `docs/development.md`, every document in `docs/policy/`, the feature
|
||||
roadmap, and the sources listed for the current stage. Inspect callers and
|
||||
existing tests before editing. File lists identify owners, not edit allowlists.
|
||||
Use existing path, file-operation, configuration, and subprocess boundaries.
|
||||
Do not create a general cache framework or change unrelated safety policies.
|
||||
|
||||
- Add `Resume bool` and `ChunkCache string` to `config.NotariusConfig`, using
|
||||
YAML/JSON names `resume` and `chunk_cache` with `omitempty`. False and the empty
|
||||
string are the unset defaults. Null resets to the zero value through existing
|
||||
ConfigKit behavior. For enabled Notarius configuration, trim surrounding
|
||||
whitespace and accept only empty, `auto`, `bypass`, or `refresh`; do not add
|
||||
case-insensitive aliases. Preserve the existing disabled-section validation
|
||||
policy rather than introducing new mandatory configuration checks there.
|
||||
- Ordinary extraction passes the configured resume boolean and chunk mode.
|
||||
Forced extraction always passes resume false and chunk mode `refresh`, even
|
||||
if configuration requests bypass. The stage resolves force precedence; the
|
||||
adapter translates effective intent into flags. No private alias behavior.
|
||||
- Use stable snapshot paths under
|
||||
`<session-root>/cache/notarius/references/<sha256>/<prepared-basename>`.
|
||||
SHA-256 is the full lowercase hexadecimal digest of the reference bytes.
|
||||
The basename preserves the prepared file extension and is validated as one
|
||||
path component. Identity contains neither run ID nor timestamps. Different
|
||||
sources with identical bytes and basename can share a verified snapshot.
|
||||
Same basename with different bytes cannot collide. This is an implementation
|
||||
layout, not a new external artifact schema.
|
||||
- Keep prepared-input identities, metadata, and extract resume fingerprints
|
||||
unchanged. Snapshot paths are delivery details. Neither new setting belongs
|
||||
in semantic resume evidence; safe effective configuration can expose them.
|
||||
- A missing snapshot is reconstructed. A corrupt regular snapshot is atomically
|
||||
replaced from a currently verified prepared source. Unsafe paths, symlinks,
|
||||
directories, or other non-regular conflicts fail with the affected path and
|
||||
recovery guidance, never recursive deletion. A prepared-source mismatch
|
||||
remains an error, not a cache miss. No eviction or cache ledger is needed.
|
||||
- Snapshots are ordinary group-shareable workspace files, not API-key material.
|
||||
They survive ordinary run cleanup, are not published/restored as artifacts,
|
||||
and do not become prerequisites for reuse of a complete durable extract bundle.
|
||||
- Notarius owns checkpoint enablement, roots, recording, and validity. Do not
|
||||
read its config to infer enablement, silently retry without resume, or widen
|
||||
inherited environment. Recommend explicit persistent upstream cache roots.
|
||||
- Cache refresh is not deletion: regenerate the current input's plan and let
|
||||
Notarius publish an accepted replacement. Leave other plans/checkpoints alone.
|
||||
No guarantee of different text, provider-side cache bypass, or transactional
|
||||
regeneration is introduced.
|
||||
- The real integration acceptance baseline is Notarius v0.7.0. Do not add a
|
||||
runtime version probe or reject older compatible executables just by version.
|
||||
Unsupported flags fail through the existing subprocess error path.
|
||||
|
||||
## Stage 1 — Add Typed Cache Controls
|
||||
|
||||
Status: Not started
|
||||
|
||||
Read: `internal/config/config.go`, `credential_projection.go`,
|
||||
`pipeline_normalize.go`, `validate_notarius.go`, `notarius_test.go`,
|
||||
`pipeline_configkit_test.go`, `effective_projection.go`; configuration sections
|
||||
of `docs/config.md` and `docs/internal/configuration.md`.
|
||||
|
||||
1. Add the two fields and implement the zero-value, normalization, and validation
|
||||
decisions above. Keep defaults centralized; no stage-specific environment
|
||||
overrides or raw extra-argument map.
|
||||
2. Extend `NotariusConfig.MarshalYAML` so direct serialization retains both
|
||||
fields and still projects only the credential reference name. Confirm that
|
||||
effective projection, source reporting, copying, and profile overrides carry
|
||||
the fields without adding another projection layer.
|
||||
3. Extend focused configuration tests for absence, explicit modes, invalid
|
||||
enabled modes, null reset, profile replacement, and safe YAML/JSON inspection.
|
||||
Protect the existing optional credential behavior. Do not snapshot complete
|
||||
error wording or repeat every case at the CLI layer.
|
||||
|
||||
Acceptance: `GOWORK=off go test ./internal/config` and
|
||||
`GOWORK=off go test ./...` pass. Existing configurations require no edits.
|
||||
This stage alone does not enable delivery and is not a finished feature.
|
||||
|
||||
## Stage 2 — Forward Cache Intent and Apply Force Precedence
|
||||
|
||||
Status: Not started
|
||||
|
||||
Read: `internal/stage/extract.go`, `stage.go`, `extract_test.go`,
|
||||
`semantic_contracts_delivery.go`, `semantic_contracts_delivery_test.go`;
|
||||
`internal/adapters/notarius/runner.go`, `subprocess.go`, `subprocess_test.go`,
|
||||
`fake.go`; `docs/internal/adapters.md`, `docs/integrations/notarius.md`.
|
||||
|
||||
1. Add `Resume bool` and `ChunkCache string` to the typed Notarius request.
|
||||
Populate them in extract, applying `env.Force` precedence in one place.
|
||||
2. Append `--resume` only for true and `--chunk_cache <mode>` only for nonempty
|
||||
mode. Preserve existing arguments, references, credentials, diagnostics,
|
||||
receipt parsing, and bundle discovery. Validate programmatic nonempty modes
|
||||
before starting a subprocess; do not import application config into the
|
||||
adapter merely to share a trivial enum check.
|
||||
3. Test the effective request matrix with ordinary and forced execution,
|
||||
including force overriding explicit bypass and resume. Adapter tests own the
|
||||
exact flag mapping and rejection of invalid direct requests. Keep a test for
|
||||
absent settings producing the previous ordinary argv. Disabled extraction
|
||||
still never invokes the adapter.
|
||||
4. Add a focused semantic-fingerprint exclusion test for the two controls; do
|
||||
not bump the fingerprint version or change manifest schema or invalidation.
|
||||
|
||||
Acceptance: focused normal and race tests for `./internal/stage` and
|
||||
`./internal/adapters/notarius` pass. No cache deletion or upstream config parsing
|
||||
has been introduced. Existing credential-redaction tests remain green.
|
||||
|
||||
## Stage 3 — Replace Run-Local Reference Snapshots with Stable Verified Snapshots
|
||||
|
||||
Status: Not started
|
||||
|
||||
Read: `internal/artifacts/paths.go` and its tests;
|
||||
`internal/stage/extract_references.go`, `extract.go`, `extract_resume.go`,
|
||||
`extract_test.go` and reference-focused tests; existing root-confined helpers in
|
||||
`internal/pathsafe/` and atomic/checksum helpers in `internal/fileops/`;
|
||||
`docs/internal/artifacts.md`, `docs/internal/workspace.md`.
|
||||
|
||||
1. Add canonical artifact/path-owner helpers for the cache root and confined
|
||||
snapshot destination. Reuse the package's existing path-segment conventions
|
||||
and central constants; validate digest syntax before using it as a segment.
|
||||
Do not embed new layout construction independently in materialization and
|
||||
post-run verification.
|
||||
2. Refactor `materializeExtractReferenceSnapshots` to resolve stable destinations.
|
||||
Verify an existing regular file's size and SHA-256 before reuse. For missing
|
||||
or corrupt regular entries, copy from the manifest-verified source using the
|
||||
existing atomic/checksum primitive, checking the result before invocation.
|
||||
Deduplicate equivalent destinations within the request. Check path components
|
||||
before directory creation/opening using existing confinement facilities.
|
||||
3. If a newly written entry fails verification because its source changed, do
|
||||
not invoke Notarius. Remove only that invalid, newly produced cache file
|
||||
through a confined operation if safe; do not remove the root or other entries.
|
||||
Preserve an actionable prepared-source error.
|
||||
4. Make post-run verification use the same destination resolver and reject
|
||||
mutated, missing, symlinked, or non-regular snapshots before bundle promotion.
|
||||
Continue checking actual content, not just names or modification times.
|
||||
5. Remove obsolete runtime use of the run-ID snapshot path helper. Remove the
|
||||
helper if no callers remain; no compatibility shim or migration is needed.
|
||||
Leave old run files alone. Do not change current prepared-reference metadata,
|
||||
recorded source paths, semantic evidence, or durable bundle locations.
|
||||
6. Adapt existing reference tests rather than keeping parallel old/new suites.
|
||||
Cover stable paths across run IDs, changed content, same basename/different
|
||||
content, verified reuse, reconstruction after deletion, corrupt regular-file
|
||||
repair, unsafe conflicts, and post-invocation mutation rejection. Assert
|
||||
behavior and confinement, not incidental helper choreography.
|
||||
|
||||
Acceptance: `GOWORK=off go test ./internal/artifacts ./internal/stage
|
||||
./internal/fileops ./internal/pathsafe` (on one shell line) and race tests for
|
||||
artifacts/stage pass. No new general cache abstraction, lock service, retention
|
||||
policy, or permission regime was added.
|
||||
|
||||
## Stage 4 — Verify Session Lifecycle and Command Boundaries
|
||||
|
||||
Status: Not started
|
||||
|
||||
Read: `internal/app/extract_lifecycle_test.go`, `regenerate_artifacts.go`,
|
||||
`regenerate_artifacts_test.go`, `semantic_resume_test.go`,
|
||||
`cleanup_orchestration_test.go`; `internal/stage/cleanup_plan.go`,
|
||||
`cleanup_test.go`; publish/restore enumeration owners identified through their
|
||||
callers; `docs/operations.md` and `docs/cli.md`.
|
||||
|
||||
1. Extend the existing lifecycle fixture to prove a failed extraction followed
|
||||
by a new invocation delivers identical snapshot paths for unchanged prepared
|
||||
references. A changed valid reference selects a new path. Keep old snapshots
|
||||
available; they are not current artifact evidence.
|
||||
2. Verify a successful complete extract remains reusable after cache-control
|
||||
changes or deletion of its local snapshots. No snapshot directory should be
|
||||
created merely to plan or skip extraction.
|
||||
3. Exercise ordinary successful cleanup with snapshots present: run-local
|
||||
output is removed as before while snapshots survive. Confirm publish and
|
||||
restore selection do not include the new cache subtree; add a narrow
|
||||
regression where existing enumeration could accidentally include it, not a
|
||||
second publish/restore test framework.
|
||||
4. Extend existing command/lifecycle tests to show `analyze SESSION` never
|
||||
invokes Notarius and preserves extract outputs, whereas
|
||||
`regenerate-artifacts SESSION` uses the ordinary forced extract/analyze path.
|
||||
`--artifacts` continues to narrow analysis, not the extraction bundle.
|
||||
5. Confirm errors still stop before analysis and leave no partially promoted
|
||||
current bundle. Reuse existing failure/promotion coverage where sufficient.
|
||||
|
||||
Acceptance: normal and race tests for `./internal/app ./internal/stage` pass.
|
||||
No new alias semantics, CLI options, manifest schema, or unrelated lifecycle
|
||||
changes. This stage proves Narratio orchestration, not upstream cache hits.
|
||||
|
||||
## Stage 5 — Prove Reuse with Real Notarius and a Local Test Provider
|
||||
|
||||
Status: Not started
|
||||
|
||||
Read Notarius v0.7.0 `docs/cli.md`, `docs/config.md`, `docs/operations.md`,
|
||||
`examples/dnd-minimal.config.yml`, `examples/seriatim-minimal-transcript.json`,
|
||||
`internal/cli/production_contract_test.go` (local-backend example), and
|
||||
`internal/framework/checkpoint/identity.go`. Inspect the selected built-in
|
||||
module's schema and reference slots rather than inventing output payloads.
|
||||
Read Narratio's existing extraction lifecycle and adapter test fixtures.
|
||||
|
||||
1. Add an explicitly opt-in Go integration test, invoked with an absolute
|
||||
`NARRATIO_TEST_NOTARIUS_BINARY` path. If unset, skip with a clear reason; if
|
||||
set but unusable, fail. Document the focused `go test -run` invocation.
|
||||
The normal suite must not discover Notarius on PATH, build/download upstream,
|
||||
or depend on the sibling checkout. Do not add Notarius to `go.mod`.
|
||||
2. Run the real executable with a small built-in D&D pipeline and its ordinary
|
||||
JSON output module. Use a local `httptest` OpenAI-compatible chat-completions
|
||||
endpoint via Notarius's `promptkit.local_backend`, a `backend: local` profile,
|
||||
and schema-valid deterministic responses. No real provider, credentials, or
|
||||
upstream test-only module registration is necessary. Use explicit temporary
|
||||
config, profile, cache, and output roots; keep endpoint/profile content stable
|
||||
across the compared invocations because they affect checkpoint identity.
|
||||
3. Drive production Narratio snapshot delivery and the production adapter from
|
||||
the fixture, not a hand-assembled imitation of their argv. Use different run
|
||||
IDs/output roots and at least one declared prepared reference. Count local
|
||||
model requests and inspect upstream checkpoint decision codes: first execution
|
||||
populates state; second ordinary resume reuses accepted work; changing valid
|
||||
reference bytes causes execution rather than a false hit.
|
||||
4. Add a deterministic partial-failure scenario with accepted predecessor work:
|
||||
configure ordered work and return a controlled failure for a later producer,
|
||||
including any bounded retries. Keep configuration and endpoint unchanged on
|
||||
retry; restore only server behavior. Verify accepted earlier work is reused
|
||||
and the failed work is executed. Do not infer reuse from equal output bytes.
|
||||
5. Exercise forced extraction against populated state. Verify checkpoint loading
|
||||
is disabled and work executes again. Demonstrate actual chunk-plan refresh
|
||||
through Notarius's chunk provenance/cache behavior in addition to the adapter
|
||||
flag test; deterministic plans need not differ in bytes. Check that an
|
||||
unrelated cached input remains untouched.
|
||||
6. Run against a locally built v0.7.0 executable (or a documented compatible
|
||||
newer version) without modifying or tagging the upstream repository. Record
|
||||
version, focused command, and results in this stage's completion note. Keep
|
||||
test fixtures small and scoped; do not reproduce upstream's whole cache suite.
|
||||
|
||||
Acceptance: the opt-in test passes without internet/provider calls, ordinary
|
||||
`go test ./...` remains independent of upstream installation, and there is real
|
||||
evidence of cache reuse across Narratio invocations. A skipped integration test
|
||||
does not complete this stage. If upstream behavior blocks the stated contract,
|
||||
record the evidence and stop for direction rather than weakening the test or
|
||||
silently modifying upstream.
|
||||
|
||||
## Stage 6 — Document the Workflow and Complete Validation
|
||||
|
||||
Status: Not started
|
||||
|
||||
Read: `docs/operations.md`, `docs/cli.md`, `docs/config.md`,
|
||||
`docs/integrations/notarius.md`, `docs/internal/stage-extract.md`,
|
||||
`docs/internal/adapters.md`, `docs/internal/workspace.md`,
|
||||
`docs/troubleshooting.md`, `examples/README.md`, `docs/release.md`.
|
||||
|
||||
1. Put the roadmap's three-way development workflow comparison prominently at
|
||||
the start of Operations' artifact-development section: analysis only,
|
||||
extraction plus analysis regeneration, and ordinary bounded continuation.
|
||||
Explain preservation of transcripts/extraction, analysis selection, and
|
||||
separate publishing. Link to CLI-owned exact syntax and avoid redefining it.
|
||||
2. Update field/default/force precedence documentation and a maintained example.
|
||||
Clearly explain upstream checkpoint enablement, explicit persistent roots,
|
||||
cold misses from changed references/profile/paths, and the distinction between
|
||||
Narratio skipping a stage and Notarius reusing work inside an executed stage.
|
||||
3. Document stable snapshot location/retention, regular corruption recovery,
|
||||
unsafe-path errors, and manual removal only while the session is inactive.
|
||||
Do not add a cache purge command, auto-GC, mandatory private permissions, or
|
||||
promise that all changed configuration is visible to Narratio resume.
|
||||
4. Record the tested Notarius integration baseline and opt-in test instructions.
|
||||
Preserve already-published release notes; leave version choice and release
|
||||
publication to a separately authorized task.
|
||||
5. Run the checks below. Mark the roadmap and stages complete only when all
|
||||
required evidence, including Stage 5, is present. Record any environmental
|
||||
limitations accurately; do not wait for CI or assert native macOS validation
|
||||
based on cross-compilation.
|
||||
|
||||
```sh
|
||||
GOWORK=off go mod tidy -diff
|
||||
GOWORK=off go mod verify
|
||||
GOWORK=off go test -count=1 ./...
|
||||
GOWORK=off go test -race -count=1 ./...
|
||||
GOWORK=off go vet ./...
|
||||
GOWORK=off go build ./...
|
||||
GOWORK=off go test ./internal/doccheck
|
||||
GOWORK=off go test ./internal/config -run '^TestExamplesLoadAndValidate$'
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Check changed Go files with `gofmt -l`. Run the repository's
|
||||
`scripts/build-release-assets.sh v0.0.0-test OUTPUT_DIR` into an empty absolute
|
||||
directory created with `mktemp -d` to check the three maintained targets, without
|
||||
publishing assets or tags. Keep any temporary binaries outside the worktree.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None requiring an operator decision. Defaults, force precedence, cache ownership,
|
||||
snapshot lifetime, and the command distinction are settled. The exact test
|
||||
payloads are implementation details governed by the real upstream schemas, not
|
||||
new product policy. Stage 5 must establish actual reuse before completion.
|
||||
212
docs/roadmap/notarius-cache-reuse.md
Normal file
212
docs/roadmap/notarius-cache-reuse.md
Normal file
@@ -0,0 +1,212 @@
|
||||
# Notarius Cache Reuse and Artifact Development
|
||||
|
||||
Status: Accepted direction — implementation pending
|
||||
|
||||
## Goal and User Intent
|
||||
|
||||
Allow Narratio to reuse Notarius's cached chunk plans and accepted intermediate
|
||||
artifacts when extraction actually executes, particularly when retrying failed
|
||||
work. Preserve an explicit force path that regenerates the selected extraction
|
||||
pipeline's work. Keep transcript generation separate from iterative artifact
|
||||
development, and prominently explain the existing analysis-only command.
|
||||
|
||||
This roadmap defines the desired end state, not an implementation sequence.
|
||||
The [implementation plan](implementation.md) owns staged execution and checks.
|
||||
It builds on the local Notarius v0.7.0 CLI and cache contracts reviewed for this
|
||||
proposal. No upstream changes are required by the proposed design.
|
||||
|
||||
## Current Behavior and Underlying Issue
|
||||
|
||||
Narratio's manifest-based resume can skip an already-completed extract stage.
|
||||
When that stage must execute, however, the Notarius adapter never passes
|
||||
`--resume`, so Notarius cannot load compatible execution checkpoints.
|
||||
|
||||
Notarius has two distinct mechanisms:
|
||||
|
||||
- Chunk-plan caching defaults to `auto`. Valid source-addressed plans can
|
||||
already be reused through Notarius configuration. `--chunk_cache` overrides
|
||||
the configured mode with `auto`, `refresh`, or `bypass`.
|
||||
- Checkpoint recording is disabled by default and is enabled in Notarius with
|
||||
`cache.checkpoints.enabled: true`. Recording-enabled runs save accepted work
|
||||
even without `--resume`; that flag separately permits checkpoint loading.
|
||||
Asking for resume while recording is disabled fails.
|
||||
|
||||
There is also a reference-identity obstacle. Narratio supplies references via
|
||||
invocation-local snapshots, whose paths change with each run ID. Notarius's
|
||||
checkpoint identity includes reference origin URIs as well as content digests.
|
||||
Simply adding `--resume` therefore does not establish reuse across otherwise
|
||||
equivalent invocations that supply those references.
|
||||
|
||||
Relevant upstream owners are Notarius's `docs/cli.md`, `docs/config.md`, and
|
||||
`docs/operations.md`; reference fingerprint construction is in
|
||||
`internal/framework/checkpoint/identity.go`. Narratio's existing contracts are
|
||||
documented in [Extract Internals](../internal/stage-extract.md) and the
|
||||
[Notarius Integration](../integrations/notarius.md).
|
||||
|
||||
## Configuration and Invocation Contract
|
||||
|
||||
Add two optional fields to Narratio's Notarius section:
|
||||
|
||||
```yaml
|
||||
notarius:
|
||||
resume: true
|
||||
chunk_cache: auto
|
||||
```
|
||||
|
||||
| Field | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `notarius.resume` | `false` | Permit compatible Notarius checkpoint loading when extraction executes without force. |
|
||||
| `notarius.chunk_cache` | unset | Optional chunk-plan mode override: `auto`, `refresh`, or `bypass`; unset preserves Notarius configuration. |
|
||||
|
||||
These are typed, validated configuration fields, not arbitrary CLI arguments.
|
||||
They participate in existing profile composition, safe inspection, and source
|
||||
reporting. Do not introduce another secret, configuration, or cache resolver.
|
||||
|
||||
Notarius continues to own checkpoint recording, cache locations, compatibility
|
||||
checks, and cache formats. Operators enable recording and choose persistent
|
||||
cache directories in Notarius configuration. Narratio neither reads that
|
||||
configuration to infer settings nor writes a replacement configuration. A
|
||||
misconfigured resume request remains an actionable downstream error; do not
|
||||
silently retry without reuse, which could unexpectedly incur provider costs.
|
||||
|
||||
### Force Takes Precedence
|
||||
|
||||
| Extraction invocation | Checkpoint loading | Chunk-plan policy |
|
||||
| --- | --- | --- |
|
||||
| Ordinary execution, `resume: false` | No `--resume` | Configured Narratio override, or no flag |
|
||||
| Ordinary execution, `resume: true` | Pass `--resume` | Configured Narratio override, or no flag |
|
||||
| Forced extraction | Never pass `--resume` | Always pass `--chunk_cache refresh` |
|
||||
|
||||
Force regenerates the selected pipeline's chunk plan and objects rather than
|
||||
hydrating accepted checkpoint results. Checkpoint recording may still save the
|
||||
new work if enabled by Notarius. Force overrides even an explicit Narratio
|
||||
`chunk_cache: bypass` selection; document this precedence clearly.
|
||||
|
||||
“Clear the cache” means bypass the current input's cached chunk plan and replace
|
||||
it with a newly accepted plan, not recursively delete the Notarius cache root.
|
||||
Notarius's refresh operation publishes only an accepted replacement. A failed
|
||||
refresh need not erase the old plan. Other inputs' plans and unrelated checkpoint
|
||||
identities are left alone. No global purge, transactional regeneration, provider-
|
||||
side cache bypass, or guarantee of different generated text is implied.
|
||||
|
||||
Disabled extraction retains its ordinary explicit self-skip. The existing
|
||||
force scope and failure propagation remain unchanged: a failed extraction does
|
||||
not proceed to analysis.
|
||||
|
||||
## Stable Reference Snapshots
|
||||
|
||||
Use session-scoped, content-addressed reference snapshots at stable locations
|
||||
outside invocation-local run directories. Equivalent references in the same
|
||||
session must receive identical absolute snapshot paths across invocations;
|
||||
changed bytes must select different snapshot identities. Central path helpers
|
||||
own the layout, not adapter-local string construction.
|
||||
|
||||
Retain the useful protections of the existing snapshot mechanism:
|
||||
|
||||
- resolve inputs from manifest-backed prepared reference evidence;
|
||||
- validate content against its recorded checksum and size;
|
||||
- safely materialize missing snapshots and verify existing ones before reuse;
|
||||
- retain confinement and post-invocation verification rather than silently
|
||||
trusting the cache or substituting mutable prepared files;
|
||||
- use the existing session lock and narrow file-operation helpers, not a new
|
||||
general-purpose cache framework.
|
||||
|
||||
Snapshots are reconstructible local cache state, not canonical artifacts,
|
||||
published outputs, or credentials. Apply Narratio's ordinary group-shareable
|
||||
workspace policy. Preserve them across ordinary run cleanup so later extraction
|
||||
can reuse their locations; do not add garbage collection in this feature.
|
||||
Document their location, retention, and safe removal while no session operation
|
||||
is active. Missing snapshots can be reconstructed from valid prepared inputs.
|
||||
|
||||
Old invocation-local snapshots and checkpoints require no migration. A first
|
||||
execution under the stable layout may miss old checkpoint identities. Moving a
|
||||
workspace may likewise cause cold misses; cross-host or relocated-path cache
|
||||
identity is not promised.
|
||||
|
||||
## Two Distinct Resume Layers
|
||||
|
||||
Narratio remains responsible for deciding whether an existing complete extract
|
||||
result is current. Notarius remains responsible for reusing compatible internal
|
||||
work when an extraction process actually runs. Cached work must still produce
|
||||
an ordinary successful bundle that passes Narratio's existing validation and
|
||||
immutable promotion boundary.
|
||||
|
||||
The two new controls are operational settings, not blanket invalidation keys.
|
||||
Changing them alone does not stale a valid completed extraction. Explicit force
|
||||
is required to execute otherwise-current extraction with a fresh chunk plan.
|
||||
Preserve existing reference-content and transcript invalidation and the rule
|
||||
that changes hidden inside Notarius configuration require operator action.
|
||||
|
||||
Persistent upstream cache roots must not live inside disposable run output.
|
||||
Recommend explicitly configured Notarius cache directories for service
|
||||
deployments rather than assuming interactive-shell cache discovery is inherited
|
||||
through Narratio's minimal subprocess environment. Do not broaden environment
|
||||
inheritance or impose a new private-data policy on ordinary D&D material.
|
||||
|
||||
## Development-Focused Operator Guidance
|
||||
|
||||
Make the following distinction prominent at the beginning of the artifact
|
||||
development workflow in [Operations](../operations.md), with links to the
|
||||
[CLI Reference](../cli.md) for exact syntax:
|
||||
|
||||
| Operator intent | Existing command | Result |
|
||||
| --- | --- | --- |
|
||||
| Iterate on analysis prompts or artifacts using existing extracted data | `narratio analyze SESSION` | Force analysis only; preserve extract results and transcripts. |
|
||||
| Rebuild extraction and analysis | `narratio regenerate-artifacts SESSION` | Force extract through analyze; refresh chunking and avoid checkpoint loading. |
|
||||
| Continue incomplete artifact work with configured reuse | `narratio run SESSION --from extract --through analyze` | Reuse valid Narratio results; permit configured Notarius reuse when extraction executes. |
|
||||
|
||||
Explain `--artifacts` alongside these choices: it selects analysis targets and
|
||||
their required analysis prerequisites, not a subset of the Notarius bundle.
|
||||
The bounded commands require valid existing prefix-stage state; they do not
|
||||
repair or rerun transcript generation automatically. Publishing and retention
|
||||
remain separate actions.
|
||||
|
||||
`regenerate-artifacts` remains exactly a convenience expansion to
|
||||
`run SESSION --force --from extract --through analyze`. Do not give it private
|
||||
cache semantics. `analyze` remains the existing forced analysis-only command;
|
||||
no new alias is needed.
|
||||
|
||||
## Required Changes and Acceptance
|
||||
|
||||
- Extend the configuration owner and safe projections for the two fields.
|
||||
- Carry effective cache intent through extract's typed adapter request, using
|
||||
the existing force state; construct CLI flags only in the Notarius adapter.
|
||||
- Replace invocation-dependent reference delivery paths with the stable snapshot
|
||||
design without weakening prepared-input or bundle validation.
|
||||
- Update canonical configuration, integration, internal, example, and recovery
|
||||
documentation where owned. Put the workflow comparison front and center in
|
||||
Operations, not only in developer internals or release notes.
|
||||
- Cover default/explicit mode mapping, invalid modes, force precedence, and
|
||||
disabled extraction with focused behavioral tests.
|
||||
- Prove cross-invocation reuse with the supported Notarius behavior, including
|
||||
unchanged reference bytes at stable paths, changed reference content causing
|
||||
a miss, and retry after partial failure reusing accepted work.
|
||||
- Prove forced extraction regenerates rather than reuses internal work, while
|
||||
analysis-only invocation never launches Notarius or changes its results.
|
||||
- Verify cache loss is recoverable, run cleanup preserves reusable snapshots,
|
||||
and receipt validation, promotion, and failure handling remain intact.
|
||||
|
||||
Follow the [Testing Policy](../policy/testing.md): default tests remain offline
|
||||
and credential-free. An argv-only fake cannot prove upstream checkpoint reuse;
|
||||
use controlled upstream integration evidence with deterministic/local test
|
||||
modules. Do not make the default suite require a developer's installed Notarius
|
||||
or a paid provider, and do not duplicate Notarius's internal cache test matrix.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Notarius `--recompute-step` or lane-selective extraction.
|
||||
- A new Narratio cache-management command, cache format, or cache garbage collector.
|
||||
- Parsing downstream configuration or automatically enabling checkpoint recording.
|
||||
- Global cache deletion or bypassing provider-side caches.
|
||||
- A new release, upstream change, or migration of old cache contents in this task.
|
||||
- New cross-process locking for independent Notarius users or distributed cache
|
||||
coordination; upstream cache concurrency limitations remain applicable.
|
||||
|
||||
## Target End State
|
||||
|
||||
Operators can cheaply continue incomplete extraction using compatible Notarius
|
||||
work, deliberately regenerate the full extraction pipeline with force, or
|
||||
iterate only on analysis using the existing `analyze` command. Stable reference
|
||||
delivery makes reuse useful across Narratio invocations. Notarius owns its
|
||||
caches; Narratio owns orchestration, reference snapshots, and validated durable
|
||||
results. The operator guide makes the everyday development choices obvious.
|
||||
Reference in New Issue
Block a user