Document Notarius cache reuse workflow
This commit is contained in:
@@ -212,6 +212,10 @@ artifacts and any prerequisites required by those targets; without
|
|||||||
never run. Common session/configuration options and repeatable artifact values
|
never run. Common session/configuration options and repeatable artifact values
|
||||||
pass through unchanged.
|
pass through unchanged.
|
||||||
|
|
||||||
|
The forced extraction rebuilds the complete configured Notarius bundle,
|
||||||
|
disables Notarius checkpoint loading, and requests chunk-plan refresh.
|
||||||
|
`--artifacts` narrows only the analysis work; it never selects Notarius lanes.
|
||||||
|
|
||||||
Because the expansion owns `--force`, `--from`, and `--through`, callers cannot
|
Because the expansion owns `--force`, `--from`, and `--through`, callers cannot
|
||||||
supply those options. The shared `run` parser reports them as duplicate
|
supply those options. The shared `run` parser reports them as duplicate
|
||||||
singleton flags. The alias has no private execution options or behavior, and
|
singleton flags. The alias has no private execution options or behavior, and
|
||||||
@@ -254,6 +258,10 @@ Equivalent to:
|
|||||||
narratio run-stage analyze <session_id> --force [...common config flags]
|
narratio run-stage analyze <session_id> --force [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
This selects only `analyze`: it does not invoke Notarius, replace extraction,
|
||||||
|
or remove existing extract outputs. `--artifacts` narrows the forced analysis
|
||||||
|
targets and their required analysis prerequisites.
|
||||||
|
|
||||||
### `publish`
|
### `publish`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -362,6 +362,8 @@ Rules:
|
|||||||
| `pipeline.notarius.pipeline_id` | string | Conditional | required when enabled |
|
| `pipeline.notarius.pipeline_id` | string | Conditional | required when enabled |
|
||||||
| `pipeline.notarius.timeout` | duration | No | `3h`; must be positive |
|
| `pipeline.notarius.timeout` | duration | No | `3h`; must be positive |
|
||||||
| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from the pipeline file directory |
|
| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from the pipeline file directory |
|
||||||
|
| `pipeline.notarius.resume` | bool | No | `false`; request compatible Notarius checkpoint reuse during ordinary executed extraction |
|
||||||
|
| `pipeline.notarius.chunk_cache` | string | No | empty; when set, exactly `auto`, `bypass`, or `refresh` |
|
||||||
| `pipeline.notarius.references` | map[string]string | No | empty; maps normalized Notarius selectors to supported prepared Narratio source IDs; maximum 256 entries |
|
| `pipeline.notarius.references` | map[string]string | No | empty; maps normalized Notarius selectors to supported prepared Narratio source IDs; maximum 256 entries |
|
||||||
| `pipeline.notarius.outputs` | map | Conditional | at least one entry when enabled |
|
| `pipeline.notarius.outputs` | map | Conditional | at least one entry when enabled |
|
||||||
| `pipeline.render.enabled` | bool | No | `true` |
|
| `pipeline.render.enabled` | bool | No | `true` |
|
||||||
@@ -406,6 +408,20 @@ configuration digest; rotating only its value does neither.
|
|||||||
See the [Notarius integration contract](./integrations/notarius.md) for the
|
See the [Notarius integration contract](./integrations/notarius.md) for the
|
||||||
delivery and redaction boundary.
|
delivery and redaction boundary.
|
||||||
|
|
||||||
|
### Notarius Cache Controls
|
||||||
|
|
||||||
|
`pipeline.notarius.resume` defaults to `false`. When true, an executed ordinary
|
||||||
|
extraction passes `--resume`; the deployed Notarius configuration must already
|
||||||
|
enable checkpoint recording and should select an explicit persistent checkpoint
|
||||||
|
root. Narratio does not inspect or generate that configuration.
|
||||||
|
|
||||||
|
`pipeline.notarius.chunk_cache` is empty by default, leaving Notarius's
|
||||||
|
configured mode in control. A nonempty value must be `auto`, `bypass`, or
|
||||||
|
`refresh` and is passed as `--chunk_cache`. Forced extraction takes precedence
|
||||||
|
over both fields: it omits `--resume` and passes `--chunk_cache refresh`.
|
||||||
|
Neither field changes Narratio's semantic extraction fingerprint, so changing
|
||||||
|
them alone does not rerun an otherwise complete durable extraction.
|
||||||
|
|
||||||
### Notarius Reference Bindings
|
### Notarius Reference Bindings
|
||||||
|
|
||||||
`pipeline.notarius.references` maps a Notarius CLI selector to a prepared
|
`pipeline.notarius.references` maps a Notarius CLI selector to a prepared
|
||||||
@@ -435,8 +451,8 @@ It validates only selector structure and the prepared source vocabulary;
|
|||||||
Notarius owns target-slot declarations and media compatibility.
|
Notarius owns target-slot declarations and media compatibility.
|
||||||
|
|
||||||
Before extraction, Narratio resolves every binding from the current prepared
|
Before extraction, Narratio resolves every binding from the current prepared
|
||||||
session manifest and streams it into a verified invocation-local snapshot whose
|
session manifest and streams it into a verified, content-addressed session
|
||||||
absolute path is passed to Notarius. Missing, unsafe, empty,
|
snapshot whose absolute path is passed to Notarius. Missing, unsafe, empty,
|
||||||
changed-during-copy, or checksum-inconsistent prepared evidence fails with
|
changed-during-copy, or checksum-inconsistent prepared evidence fails with
|
||||||
guidance to force `prepare`. Bindings are sorted by normalized selector and are
|
guidance to force `prepare`. Bindings are sorted by normalized selector and are
|
||||||
part of extraction fingerprint and resume identity. See the
|
part of extraction fingerprint and resume identity. See the
|
||||||
|
|||||||
@@ -7,14 +7,14 @@ lanes from the final trimmed Seriatim transcript. Narratio owns invocation,
|
|||||||
safe bundle discovery, lane selection, and its own artifact metadata. Notarius
|
safe bundle discovery, lane selection, and its own artifact metadata. Notarius
|
||||||
owns pipeline definitions, lane schemas, the receipt, and bundle formats.
|
owns pipeline definitions, lane schemas, the receipt, and bundle formats.
|
||||||
|
|
||||||
Canonical Notarius v0.6.0 references:
|
Canonical Notarius v0.7.0 references:
|
||||||
|
|
||||||
- [CLI reference](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.6.0/docs/cli.md)
|
- [CLI reference](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.7.0/docs/cli.md)
|
||||||
- [Subprocess consumer contract](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.6.0/docs/consumers/subprocess.md)
|
- [Subprocess consumer contract](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.7.0/docs/consumers/subprocess.md)
|
||||||
- [D&D pipeline and lane contracts](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.6.0/docs/consumers/dnd-pipeline.md)
|
- [D&D pipeline and lane contracts](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.7.0/docs/consumers/dnd-pipeline.md)
|
||||||
- [Run-result receipt](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.6.0/docs/integrations/run-result.md)
|
- [Run-result receipt](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.7.0/docs/integrations/run-result.md)
|
||||||
- [JSON output bundle](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.6.0/docs/integrations/json-output.md)
|
- [JSON output bundle](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.7.0/docs/integrations/json-output.md)
|
||||||
- [D&D spell-catalog overlay](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.6.0/docs/integrations/dnd-spell-catalog-overlays.md)
|
- [D&D spell-catalog overlay](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.7.0/docs/integrations/dnd-spell-catalog-overlays.md)
|
||||||
|
|
||||||
The [complete Narratio example](../../examples/pipeline.full.annotated.yml)
|
The [complete Narratio example](../../examples/pipeline.full.annotated.yml)
|
||||||
records the exact current constraints for all ten D&D lanes. Treat the linked
|
records the exact current constraints for all ten D&D lanes. Treat the linked
|
||||||
@@ -25,19 +25,23 @@ duplicate the complete schemas.
|
|||||||
|
|
||||||
When `pipeline.notarius.enabled` is true, Narratio resolves the executable,
|
When `pipeline.notarius.enabled` is true, Narratio resolves the executable,
|
||||||
configuration path, input path, output directory, and working directory to
|
configuration path, input path, output directory, and working directory to
|
||||||
absolute paths. Narratio requires the Notarius v0.6.0 CLI contract when
|
absolute paths. Narratio requires the Notarius v0.7.0 CLI contract when
|
||||||
references are configured and invokes each binding as a separate argument
|
references are configured and invokes each binding as a separate argument
|
||||||
before `--json`:
|
before `--json`:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
notarius run <pipeline_id> --config <config_path> --input <trimmed_json> --output-dir <staging_dir> [--reference <selector>=<verified_snapshot_path>]... --json
|
notarius run <pipeline_id> --config <config_path> --input <trimmed_json> --output-dir <staging_dir> [--resume] [--chunk_cache <mode>] [--reference <selector>=<verified_snapshot_path>]... --json
|
||||||
```
|
```
|
||||||
|
|
||||||
Reference paths are absolute invocation-local snapshots streamed from the
|
Reference paths are absolute stable snapshots streamed from manifest-verified
|
||||||
manifest-verified canonical files prepared inside the current Narratio session
|
canonical files prepared inside the current Narratio session workspace. They
|
||||||
workspace. Narratio verifies snapshot checksum and size before and after the
|
live at `cache/notarius/references/<sha256>/<prepared-basename>` and retain the
|
||||||
subprocess, and passes only configured bindings, ordered lexically by normalized
|
full lowercase content digest. Narratio verifies snapshot checksum and size
|
||||||
selector, as direct argument-vector entries without shell interpretation. A CLI
|
before and after the subprocess, reconstructs missing or corrupt regular files
|
||||||
|
from a still-valid prepared source, and passes only configured bindings,
|
||||||
|
ordered lexically by normalized selector, as direct argument-vector entries
|
||||||
|
without shell interpretation. Symlinks, directories, other non-regular
|
||||||
|
conflicts, and prepared-source mismatches fail rather than being deleted. A CLI
|
||||||
binding takes precedence over a matching external path in Notarius
|
binding takes precedence over a matching external path in Notarius
|
||||||
configuration. Narratio never emits `--without-reference`.
|
configuration. Narratio never emits `--without-reference`.
|
||||||
|
|
||||||
@@ -75,6 +79,17 @@ It does not pass a Narratio session ID or run `notarius config validate`
|
|||||||
automatically; the configured working directory and Narratio's minimal child
|
automatically; the configured working directory and Narratio's minimal child
|
||||||
environment apply to the subprocess.
|
environment apply to the subprocess.
|
||||||
|
|
||||||
|
Ordinary extraction maps `pipeline.notarius.resume: true` to `--resume` and a
|
||||||
|
nonempty `pipeline.notarius.chunk_cache` to `--chunk_cache <mode>`. Forced
|
||||||
|
extraction always omits `--resume` and supplies `--chunk_cache refresh`.
|
||||||
|
Notarius owns checkpoint enablement, recording, identity, and both cache roots;
|
||||||
|
Narratio neither reads nor synthesizes its configuration. Use explicit
|
||||||
|
persistent roots in Notarius when reuse must span subprocess invocations.
|
||||||
|
Changing a reference path/digest or other upstream identity input correctly
|
||||||
|
causes a cold miss. Chunk refresh bypasses lookup and publishes the accepted
|
||||||
|
replacement for the active source; it does not delete unrelated plans or
|
||||||
|
checkpoint state.
|
||||||
|
|
||||||
When `pipeline.notarius.llm_api_key` is absent or null, Narratio adds no
|
When `pipeline.notarius.llm_api_key` is absent or null, Narratio adds no
|
||||||
credential to that environment. When configured, the field names an environment
|
credential to that environment. When configured, the field names an environment
|
||||||
variable whose already-loaded nonempty value is forwarded under that exact
|
variable whose already-loaded nonempty value is forwarded under that exact
|
||||||
@@ -98,7 +113,7 @@ encoded credentials.
|
|||||||
|
|
||||||
## Accepted Result
|
## Accepted Result
|
||||||
|
|
||||||
Narratio's supported invocation baseline is Notarius v0.6.0. The accepted
|
Narratio's tested invocation baseline is Notarius v0.7.0. The accepted
|
||||||
receipt remains `notarius.run-result.v2`; reference flags do not change the
|
receipt remains `notarius.run-result.v2`; reference flags do not change the
|
||||||
receipt or ten-lane output contract. The receipt
|
receipt or ten-lane output contract. The receipt
|
||||||
must identify the configured pipeline, and its `index_file` must be exactly
|
must identify the configured pipeline, and its `index_file` must be exactly
|
||||||
@@ -123,6 +138,8 @@ version, and, when configured, module key. Missing, duplicate, rejected, or
|
|||||||
incompatible required lanes fail extraction even if Notarius exited zero. A
|
incompatible required lanes fail extraction even if Notarius exited zero. A
|
||||||
configured lane whose v2 validation summary is `rejected` or `incomplete` also
|
configured lane whose v2 validation summary is `rejected` or `incomplete` also
|
||||||
fails extraction.
|
fails extraction.
|
||||||
|
An accepted reused producer may report `producer_attempt_count: 0`; Narratio
|
||||||
|
preserves that valid summary rather than treating it as a missing field.
|
||||||
Unconfigured lanes may remain in the preserved bundle but do not become
|
Unconfigured lanes may remain in the preserved bundle but do not become
|
||||||
selectable Narratio sources.
|
selectable Narratio sources.
|
||||||
|
|
||||||
@@ -151,3 +168,18 @@ Configuration fields and defaults are in [Configuration](../config.md).
|
|||||||
Operator paths, rerun procedures, and bundle retention are in
|
Operator paths, rerun procedures, and bundle retention are in
|
||||||
[Operations](../operations.md). See [Troubleshooting](../troubleshooting.md)
|
[Operations](../operations.md). See [Troubleshooting](../troubleshooting.md)
|
||||||
for failure recovery.
|
for failure recovery.
|
||||||
|
|
||||||
|
## Opt-in Integration Test
|
||||||
|
|
||||||
|
The production-boundary cache test requires an absolute path to a locally
|
||||||
|
built Notarius v0.7.0-compatible executable. It uses only a local
|
||||||
|
OpenAI-compatible `httptest` provider and temporary cache/config/output roots:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
NARRATIO_TEST_NOTARIUS_BINARY=/absolute/path/to/notarius \
|
||||||
|
GOWORK=off go test ./internal/stage \
|
||||||
|
-run '^TestNotariusCacheReuseIntegration$' -count=1 -v
|
||||||
|
```
|
||||||
|
|
||||||
|
When the variable is unset, the normal test suite skips this test and never
|
||||||
|
searches `PATH`, a sibling checkout, the network, or a real provider.
|
||||||
|
|||||||
@@ -65,6 +65,9 @@ Notarius is composed only when extraction is enabled; the extract stage owns
|
|||||||
prepared reference resolution, receipt, bundle, and configured-lane policy.
|
prepared reference resolution, receipt, bundle, and configured-lane policy.
|
||||||
The adapter validates the ordered selector/absolute-path pairs and is the sole
|
The adapter validates the ordered selector/absolute-path pairs and is the sole
|
||||||
owner of serializing them as repeated `--reference` arguments before `--json`.
|
owner of serializing them as repeated `--reference` arguments before `--json`.
|
||||||
|
It also serializes an effective resume request as `--resume` and a nonempty
|
||||||
|
validated chunk mode as `--chunk_cache <mode>`. The extract stage, not the
|
||||||
|
adapter, resolves force precedence before constructing the request.
|
||||||
Its request may also carry one already-resolved opaque ConfigKit secret
|
Its request may also carry one already-resolved opaque ConfigKit secret
|
||||||
reference. Only the Notarius adapter reads that value, rejects an unresolved or
|
reference. Only the Notarius adapter reads that value, rejects an unresolved or
|
||||||
empty programmatic reference, and maps the exact configured name/value into the
|
empty programmatic reference, and maps the exact configured name/value into the
|
||||||
|
|||||||
@@ -19,13 +19,15 @@ procedures belong in [Operations](../operations.md).
|
|||||||
1. resolves the final trimmed transcript from the shared artifact catalog;
|
1. resolves the final trimmed transcript from the shared artifact catalog;
|
||||||
2. resolves every configured prepared reference through the shared
|
2. resolves every configured prepared reference through the shared
|
||||||
manifest-authoritative identity resolver before creating run-local output;
|
manifest-authoritative identity resolver before creating run-local output;
|
||||||
3. streams each verified reference into an invocation-local snapshot and
|
3. resolves each reference to a stable content-addressed session snapshot,
|
||||||
rejects any source change observed while copying;
|
reusing verified bytes or reconstructing a missing/corrupt regular entry,
|
||||||
|
and rejects any source change observed while copying;
|
||||||
4. fingerprints the byte- and provenance-bearing Notarius invocation evidence,
|
4. fingerprints the byte- and provenance-bearing Notarius invocation evidence,
|
||||||
including sorted reference identities;
|
including sorted reference identities;
|
||||||
5. creates a run-local staging directory and invokes the injected
|
5. creates a run-local staging directory and invokes the injected
|
||||||
`notarius.Runner`, carrying the optional already-resolved credential
|
`notarius.Runner`, carrying the optional already-resolved credential
|
||||||
reference without exposing its value;
|
reference without exposing its value, plus effective resume and chunk-cache
|
||||||
|
controls after force precedence;
|
||||||
6. revalidates the reference snapshots, then validates the v2 successful
|
6. revalidates the reference snapshots, then validates the v2 successful
|
||||||
receipt, confined index, management documents, configured required lane
|
receipt, confined index, management documents, configured required lane
|
||||||
descriptors, validation summaries, and regular payload files;
|
descriptors, validation summaries, and regular payload files;
|
||||||
@@ -42,14 +44,22 @@ Narratio run ID, the resolved trimmed-input identity, and invocation
|
|||||||
fingerprint. The input identity binds the exact transcript bytes, canonical
|
fingerprint. The input identity binds the exact transcript bytes, canonical
|
||||||
source ID, producer stage/output/run identity, and resolution provenance.
|
source ID, producer stage/output/run identity, and resolution provenance.
|
||||||
Reference metadata contains only selector, source ID, canonical session-relative
|
Reference metadata contains only selector, source ID, canonical session-relative
|
||||||
path, checksum, and size; adapter requests receive selector and absolute
|
path, checksum, and size; adapter requests receive selector and absolute stable
|
||||||
invocation-local snapshot path, never payload contents. Snapshot bytes must
|
snapshot path, never payload contents. Snapshot bytes must
|
||||||
match the prepared identity both before and after Notarius runs, so a concurrent
|
match the prepared identity both before and after Notarius runs, so a concurrent
|
||||||
prepared-file replacement cannot make recorded provenance describe different
|
prepared-file replacement cannot make recorded provenance describe different
|
||||||
bytes from those supplied to Notarius.
|
bytes from those supplied to Notarius.
|
||||||
Validation completes before the single durable rename, so a rejected result
|
Validation completes before the single durable rename, so a rejected result
|
||||||
cannot expose a partial durable bundle.
|
cannot expose a partial durable bundle.
|
||||||
|
|
||||||
|
Snapshot destinations are owned by `internal/artifacts` under
|
||||||
|
`cache/notarius/references/<sha256>/<prepared-basename>`. They contain no run ID
|
||||||
|
or timestamp and can be reused across invocations. Identical bytes and basename
|
||||||
|
share a verified file; changed bytes select a different digest directory.
|
||||||
|
Unsafe path components and non-regular conflicts fail without recursive
|
||||||
|
deletion. Snapshot paths are delivery details and never become manifest output
|
||||||
|
evidence or prerequisites for reuse of an already complete extract bundle.
|
||||||
|
|
||||||
Any executed extraction outcome that replaces a different effective outcome
|
Any executed extraction outcome that replaces a different effective outcome
|
||||||
marks succeeded analysis and delivery dependents stale. Render is an independent
|
marks succeeded analysis and delivery dependents stale. Render is an independent
|
||||||
sibling and remains current. Repeating the same disabled self-skip with no
|
sibling and remains current. Repeating the same disabled self-skip with no
|
||||||
@@ -74,7 +84,10 @@ sorted prepared-reference identities, pipeline identity, and configured output
|
|||||||
contracts. The same reference helper and transcript identity are resolved again
|
contracts. The same reference helper and transcript identity are resolved again
|
||||||
for artifact evidence, so changing current transcript bytes, reference bytes,
|
for artifact evidence, so changing current transcript bytes, reference bytes,
|
||||||
or producer identity makes the prior extraction obsolete. Operational runner
|
or producer identity makes the prior extraction obsolete. Operational runner
|
||||||
settings do not invalidate otherwise current durable evidence.
|
settings do not invalidate otherwise current durable evidence. This includes
|
||||||
|
`resume` and `chunk_cache`; changing them alone cannot make a complete
|
||||||
|
extraction non-resumable. If extraction executes, ordinary runs forward the
|
||||||
|
configured values, while force disables resume and selects `refresh`.
|
||||||
|
|
||||||
A valid prepared-reference change makes extraction non-resumable. Missing,
|
A valid prepared-reference change makes extraction non-resumable. Missing,
|
||||||
unsafe, or checksum-inconsistent prepared evidence is a hard validation error
|
unsafe, or checksum-inconsistent prepared evidence is a hard validation error
|
||||||
@@ -115,9 +128,13 @@ recovery.
|
|||||||
`internal/stage/extract_resume.go`,
|
`internal/stage/extract_resume.go`,
|
||||||
`internal/stage/extract_test.go`,
|
`internal/stage/extract_test.go`,
|
||||||
`internal/stage/semantic_contracts_delivery.go`
|
`internal/stage/semantic_contracts_delivery.go`
|
||||||
|
- Stable snapshot materialization and verification:
|
||||||
|
`internal/stage/extract_references.go`
|
||||||
- Subprocess boundary: `internal/adapters/notarius/subprocess.go`,
|
- Subprocess boundary: `internal/adapters/notarius/subprocess.go`,
|
||||||
`internal/adapters/notarius/subprocess_test.go`
|
`internal/adapters/notarius/subprocess_test.go`
|
||||||
- Catalog hydration: `internal/artifacts/extraction_catalog.go`,
|
- Catalog hydration: `internal/artifacts/extraction_catalog.go`,
|
||||||
`internal/artifacts/extraction_catalog_test.go`
|
`internal/artifacts/extraction_catalog_test.go`
|
||||||
- Composition and downstream behavior: `internal/app/runner_test.go`,
|
- Composition and downstream behavior: `internal/app/runner_test.go`,
|
||||||
`internal/stage/analyze_test.go`, `internal/stage/publish_test.go`
|
`internal/stage/analyze_test.go`, `internal/stage/publish_test.go`
|
||||||
|
- Opt-in Notarius v0.7.0 production-boundary coverage:
|
||||||
|
`internal/stage/notarius_cache_integration_test.go`
|
||||||
|
|||||||
@@ -37,6 +37,12 @@ validates and renames the external bundle to the unique immutable Notarius
|
|||||||
bundle path supplied by `internal/artifacts`. Exact physical paths belong in
|
bundle path supplied by `internal/artifacts`. Exact physical paths belong in
|
||||||
[Operations](../operations.md#extraction-workflow).
|
[Operations](../operations.md#extraction-workflow).
|
||||||
|
|
||||||
|
Prepared references delivered to Notarius use session-durable paths owned by
|
||||||
|
`internal/artifacts` beneath
|
||||||
|
`cache/notarius/references/<sha256>/<prepared-basename>`. These verified
|
||||||
|
content-addressed snapshots are outside run-local layout, survive ordinary
|
||||||
|
published-run cleanup, and are excluded from publish and restore enumeration.
|
||||||
|
|
||||||
## Locking
|
## Locking
|
||||||
|
|
||||||
`artifacts.LocalStore` enforces the single-writer session lock via an
|
`artifacts.LocalStore` enforces the single-writer session lock via an
|
||||||
|
|||||||
@@ -187,7 +187,20 @@ shared session lifecycle. In particular, render does not require Notarius or
|
|||||||
Scriptorium, extract does not require Scriptorium, and analyze does not require
|
Scriptorium, extract does not require Scriptorium, and analyze does not require
|
||||||
the transcription, Seriatim, Audita, or Notarius adapters.
|
the transcription, Seriatim, Audita, or Notarius adapters.
|
||||||
|
|
||||||
For the common post-transcript development loop, use:
|
## Artifact Development Workflows
|
||||||
|
|
||||||
|
Choose the narrowest workflow that matches the intended replacement:
|
||||||
|
|
||||||
|
| Intent | Command | Preserved and rebuilt state |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Rebuild analysis only | [`narratio analyze 2026-04-04`](./cli.md#analyze) | Preserves transcripts and the current extraction bundle; force-runs selected analysis artifacts and their required analysis prerequisites. |
|
||||||
|
| Rebuild extraction and analysis | [`narratio regenerate-artifacts 2026-04-04`](./cli.md#regenerate-artifacts) | Preserves transcripts; force-runs the complete configured extraction bundle, then selected analysis. `--artifacts` narrows analysis only. |
|
||||||
|
| Continue bounded work normally | [`narratio run 2026-04-04 --from extract --through analyze`](./cli.md#run) | Uses ordinary resume decisions: current extraction and analysis can be skipped, while incomplete or obsolete work runs. |
|
||||||
|
|
||||||
|
All three stop before publication. Inspect the results and run
|
||||||
|
[`narratio publish`](./cli.md#publish) separately when delivery is intended.
|
||||||
|
|
||||||
|
For the extraction-plus-analysis development loop, use:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio regenerate-artifacts 2026-04-04
|
narratio regenerate-artifacts 2026-04-04
|
||||||
@@ -260,9 +273,14 @@ are not selectable or published implicitly.
|
|||||||
Configured Notarius references resolve only from the current manifest-backed
|
Configured Notarius references resolve only from the current manifest-backed
|
||||||
prepared inputs. Their canonical locations are `inputs/party.yml`,
|
prepared inputs. Their canonical locations are `inputs/party.yml`,
|
||||||
`inputs/players.yml`, `inputs/glossary.yml`, and, when configured,
|
`inputs/players.yml`, `inputs/glossary.yml`, and, when configured,
|
||||||
`inputs/spell_catalog.json`. Extraction supplies Notarius with verified copies
|
`inputs/spell_catalog.json`. Extraction supplies Notarius with verified,
|
||||||
under `runs/<run_id>/extract/references/` so a concurrent refresh of canonical
|
content-addressed copies under
|
||||||
prepared files cannot change the bytes consumed by an in-flight invocation.
|
`cache/notarius/references/<sha256>/<prepared-basename>`. The full lowercase
|
||||||
|
SHA-256 digest identifies the reference bytes, so retries and later invocations
|
||||||
|
reuse the same path while changed bytes select a new path. Older snapshots are
|
||||||
|
retained and are not current artifact evidence. A concurrent refresh of
|
||||||
|
canonical prepared files therefore cannot change the bytes consumed by an
|
||||||
|
in-flight invocation.
|
||||||
For a canonical party, preparation retains the validated authored party bytes
|
For a canonical party, preparation retains the validated authored party bytes
|
||||||
at `inputs/party.yml` and generates `inputs/players.yml` from that roster.
|
at `inputs/party.yml` and generates `inputs/players.yml` from that roster.
|
||||||
The manifest records their checksums separately, with the players input marked
|
The manifest records their checksums separately, with the players input marked
|
||||||
@@ -297,6 +315,28 @@ Notarius extraction validates its run-local bundle and moves it once to the
|
|||||||
unique durable bundle path. A pre-existing durable destination is a collision;
|
unique durable bundle path. A pre-existing durable destination is a collision;
|
||||||
no duplicate promotion tree is created.
|
no duplicate promotion tree is created.
|
||||||
|
|
||||||
|
`pipeline.notarius.resume` and `pipeline.notarius.chunk_cache` control work
|
||||||
|
inside an executed Notarius invocation. Ordinary extraction forwards their
|
||||||
|
configured values. Forced extraction always disables checkpoint loading and
|
||||||
|
requests chunk-plan `refresh`, even when the configured mode is `bypass`.
|
||||||
|
Refresh regenerates and replaces only the active input's accepted chunk plan;
|
||||||
|
it is not cache deletion.
|
||||||
|
|
||||||
|
Notarius checkpoint recording must be enabled in the deployed Notarius
|
||||||
|
configuration. Narratio neither creates a Notarius pipeline/configuration nor
|
||||||
|
reads it to infer checkpoint availability. Configure explicit persistent
|
||||||
|
Notarius chunk-plan and checkpoint roots when reuse must survive separate
|
||||||
|
Narratio subprocesses. A changed reference, transcript, selected profile,
|
||||||
|
profile content, endpoint, or other Notarius checkpoint identity input can
|
||||||
|
produce a legitimate cold miss.
|
||||||
|
|
||||||
|
There are two independent resume boundaries: Narratio may skip a completed
|
||||||
|
`extract` stage without invoking Notarius, while `--resume` allows Notarius to
|
||||||
|
reuse compatible accepted work only after Narratio has decided extraction must
|
||||||
|
execute. Deleting Narratio's local reference snapshots does not invalidate a
|
||||||
|
complete durable extraction; missing snapshots are reconstructed only when a
|
||||||
|
later extraction actually runs.
|
||||||
|
|
||||||
## External Command Lifecycle
|
## External Command Lifecycle
|
||||||
|
|
||||||
When an external command is cancelled or times out, Narratio terminates its
|
When an external command is cancelled or times out, Narratio terminates its
|
||||||
@@ -509,6 +549,7 @@ Durable session paths:
|
|||||||
- `logs/**`
|
- `logs/**`
|
||||||
- `config/**`
|
- `config/**`
|
||||||
- `runs/**`
|
- `runs/**`
|
||||||
|
- `cache/notarius/references/<sha256>/<prepared-basename>`
|
||||||
|
|
||||||
`manifest.json` is the versioned session-progress document. Narratio accepts
|
`manifest.json` is the versioned session-progress document. Narratio accepts
|
||||||
only `narratio.session-manifest.v1`. Do not edit unsupported session state to
|
only `narratio.session-manifest.v1`. Do not edit unsupported session state to
|
||||||
@@ -519,6 +560,13 @@ relevant workflow requires.
|
|||||||
Validated Notarius bundles live below `artifacts/notarius/{run_id}/`; receipt,
|
Validated Notarius bundles live below `artifacts/notarius/{run_id}/`; receipt,
|
||||||
stderr, and pre-promotion output remain in the producing run's `extract`
|
stderr, and pre-promotion output remain in the producing run's `extract`
|
||||||
directory as described in [Extraction Workflow](#extraction-workflow).
|
directory as described in [Extraction Workflow](#extraction-workflow).
|
||||||
|
Stable reference snapshots live in the session cache path above. Ordinary
|
||||||
|
published-run cleanup removes its selected run-local target but retains this
|
||||||
|
cache, and publish/restore never enumerate it as an artifact. Remove a snapshot
|
||||||
|
or digest directory manually only while no command is active for the session;
|
||||||
|
regular missing or corrupt content is reconstructed on the next executed
|
||||||
|
extraction, while symlinks, directories, and other unsafe conflicts must be
|
||||||
|
investigated and removed deliberately.
|
||||||
|
|
||||||
Run-local layout:
|
Run-local layout:
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Notarius Cache Reuse Implementation Plan
|
# Notarius Cache Reuse Implementation Plan
|
||||||
|
|
||||||
Status: Not started
|
Status: Complete (2026-09-16)
|
||||||
|
|
||||||
Implement the [feature roadmap](notarius-cache-reuse.md) in the numbered order
|
Implement the [feature roadmap](notarius-cache-reuse.md) in the numbered order
|
||||||
below. Each stage is a bounded assignment for a gpt-5.6-terra coding agent.
|
below. Each stage is a bounded assignment for a gpt-5.6-terra coding agent.
|
||||||
@@ -276,7 +276,15 @@ silently modifying upstream.
|
|||||||
|
|
||||||
## Stage 6 — Document the Workflow and Complete Validation
|
## Stage 6 — Document the Workflow and Complete Validation
|
||||||
|
|
||||||
Status: Not started
|
Status: Complete (2026-09-16)
|
||||||
|
|
||||||
|
Completion: Documented the analysis-only, extraction-plus-analysis, and
|
||||||
|
ordinary continuation workflows; cache fields and force precedence; Notarius
|
||||||
|
checkpoint ownership; stable snapshot retention and recovery; troubleshooting;
|
||||||
|
and the tested v0.7.0 integration baseline. Updated the maintained annotated
|
||||||
|
example. Module tidy/verification, uncached normal and race tests, vet, build,
|
||||||
|
doccheck, example validation, Go formatting, whitespace checks, and all three
|
||||||
|
`v0.0.0-test` release-asset builds pass.
|
||||||
|
|
||||||
Read: `docs/operations.md`, `docs/cli.md`, `docs/config.md`,
|
Read: `docs/operations.md`, `docs/cli.md`, `docs/config.md`,
|
||||||
`docs/integrations/notarius.md`, `docs/internal/stage-extract.md`,
|
`docs/integrations/notarius.md`, `docs/internal/stage-extract.md`,
|
||||||
|
|||||||
@@ -323,6 +323,59 @@ Do not point Notarius directly at the original source path or edit the manifest
|
|||||||
checksum. Relevant references: [Notarius reference configuration](./config.md#notarius-reference-bindings)
|
checksum. Relevant references: [Notarius reference configuration](./config.md#notarius-reference-bindings)
|
||||||
and [Operations: Extraction Workflow](./operations.md#extraction-workflow).
|
and [Operations: Extraction Workflow](./operations.md#extraction-workflow).
|
||||||
|
|
||||||
|
## Notarius reference snapshot cache conflict
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- extraction reports that a path beneath
|
||||||
|
`cache/notarius/references/<sha256>/` is a symlink, directory, or another
|
||||||
|
non-regular or unsafe entry.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- stop commands for the affected session and inspect the exact reported path;
|
||||||
|
- investigate unexpected links or special files as possible tampering; and
|
||||||
|
- remove only the reported conflicting leaf or digest directory after
|
||||||
|
confirming it belongs to the reconstructible reference cache.
|
||||||
|
|
||||||
|
A missing snapshot or corrupt regular snapshot is reconstructed automatically
|
||||||
|
from currently verified prepared input. Do not recursively delete the session
|
||||||
|
root, edit the manifest checksum, or remove cache entries while extraction is
|
||||||
|
active.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Extraction Workflow](./operations.md#extraction-workflow).
|
||||||
|
|
||||||
|
## Notarius resume executes work instead of reusing it
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- Narratio executes `extract`, and Notarius makes provider calls despite
|
||||||
|
`pipeline.notarius.resume: true`.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- checkpoint recording is disabled in the Notarius configuration;
|
||||||
|
- Notarius is using a nonpersistent or different checkpoint root;
|
||||||
|
- transcript/reference bytes, reference paths, pipeline/profile content,
|
||||||
|
local-backend endpoint, selected lanes, or another checkpoint identity input
|
||||||
|
changed; or
|
||||||
|
- Narratio extraction was forced, which deliberately omits `--resume`.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- validate the deployed Notarius configuration and configure explicit
|
||||||
|
persistent chunk-plan and checkpoint roots;
|
||||||
|
- inspect Notarius checkpoint decision reason codes for a legitimate cold miss;
|
||||||
|
and
|
||||||
|
- retry without force only when reuse is intended.
|
||||||
|
|
||||||
|
Remember that Narratio skipping a completed `extract` stage is separate from
|
||||||
|
Notarius reusing work inside an extraction that does execute. Narratio does not
|
||||||
|
read the Notarius configuration to infer checkpoint enablement.
|
||||||
|
|
||||||
|
Relevant references: [Notarius cache controls](./config.md#notarius-cache-controls)
|
||||||
|
and [Notarius integration](./integrations/notarius.md).
|
||||||
|
|
||||||
## Notarius reference selector or generated-handoff collision
|
## Notarius reference selector or generated-handoff collision
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
|
|||||||
@@ -60,7 +60,7 @@ with the sample campaign and a compatible local- or S3-audio session.
|
|||||||
[glossary](campaigns/sample-campaign/glossary.yml), and canonical
|
[glossary](campaigns/sample-campaign/glossary.yml), and canonical
|
||||||
[party](campaigns/sample-campaign/party.yml) fixture, plus an optional
|
[party](campaigns/sample-campaign/party.yml) fixture, plus an optional
|
||||||
[spell-catalog overlay](campaigns/sample-campaign/spell_catalog.json) that
|
[spell-catalog overlay](campaigns/sample-campaign/spell_catalog.json) that
|
||||||
follows the Notarius v0.6 contract. Narratio derives the players projection
|
follows the Notarius v0.7 contract. Narratio derives the players projection
|
||||||
from this party source.
|
from this party source.
|
||||||
- [Sample speaker audio](audio/sample-speaker.flac) is a text placeholder that
|
- [Sample speaker audio](audio/sample-speaker.flac) is a text placeholder that
|
||||||
reserves the expected filename and directory shape. Replace it with a real
|
reserves the expected filename and directory shape. Replace it with a real
|
||||||
|
|||||||
@@ -128,7 +128,7 @@ trim:
|
|||||||
report: false
|
report: false
|
||||||
|
|
||||||
notarius:
|
notarius:
|
||||||
# Optional structured extraction between trim and render.
|
# Optional structured extraction after render and before analyze.
|
||||||
enabled: true
|
enabled: true
|
||||||
binary: notarius
|
binary: notarius
|
||||||
# Optional environment-variable reference. Match this name to api_key_env in
|
# Optional environment-variable reference. Match this name to api_key_env in
|
||||||
@@ -138,6 +138,12 @@ notarius:
|
|||||||
pipeline_id: dnd-session
|
pipeline_id: dnd-session
|
||||||
timeout: 3h
|
timeout: 3h
|
||||||
working_directory: /usr/local/etc/notarius
|
working_directory: /usr/local/etc/notarius
|
||||||
|
# Reuse compatible accepted Notarius checkpoints during executed extraction.
|
||||||
|
# The Notarius config must enable recording and provide persistent cache roots.
|
||||||
|
resume: true
|
||||||
|
# auto uses or creates a source-addressed chunk plan. Forced Narratio
|
||||||
|
# extraction overrides this with refresh and disables checkpoint loading.
|
||||||
|
chunk_cache: auto
|
||||||
# External campaign references use prepared Narratio source IDs. Omit an
|
# External campaign references use prepared Narratio source IDs. Omit an
|
||||||
# optional binding when the selected Notarius pipeline does not need it.
|
# optional binding when the selected Notarius pipeline does not need it.
|
||||||
references:
|
references:
|
||||||
|
|||||||
Reference in New Issue
Block a user