From 701b6726d7cabdff246934d3a7c70789fe46b595 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 10 Aug 2026 02:10:37 +0000 Subject: [PATCH] Reconcile Notarius extraction documentation --- docs/cli.md | 2 +- docs/integrations/notarius.md | 8 +++-- docs/internal/stage-extract.md | 14 ++++---- docs/internal/stage-publish.md | 11 ++++-- docs/operations.md | 27 +++++++++++--- docs/roadmap/implementation.md | 6 ++-- docs/roadmap/notarius-extract-stage.md | 49 +++++++++++++++++++------- docs/troubleshooting.md | 29 +++++++++++++-- 8 files changed, 113 insertions(+), 33 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 5fc5ade..706aaae 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -76,7 +76,7 @@ Behavior: - evaluates full stage order; - runs `extract` between `trim` and `render`; an omitted or disabled Notarius - configuration makes extraction a no-op; + configuration records an explicit `notarius_disabled` self-skip; - skips already-succeeded stages unless `--force` is set or a stage-specific resume check finds its durable result obsolete; - continues interrupted or partially completed sessions by running non-succeeded stages; diff --git a/docs/integrations/notarius.md b/docs/integrations/notarius.md index fe99db2..37dc69c 100644 --- a/docs/integrations/notarius.md +++ b/docs/integrations/notarius.md @@ -39,9 +39,11 @@ apply to the subprocess. ## Accepted Result Narratio currently accepts receipt schema `notarius.run-result.v1`. The receipt -must identify the configured pipeline and a bundle root containing a confined -`index.json`. All receipt, index, and lane paths must stay inside that bundle; -symlinks and non-regular lane payloads are rejected. +must identify the configured pipeline, and its `index_file` must be exactly +`index.json` beneath the reported bundle root. The production index must name +the management files exactly as `manifest.json`, `rejected.json`, and +`warnings.json`. All receipt, index, and lane paths must stay inside that +bundle; symlinks and non-regular lane payloads are rejected. Supported receipt and index shapes tolerate unknown fields for forward compatibility, while required identity, validation, count, manifest, diff --git a/docs/internal/stage-extract.md b/docs/internal/stage-extract.md index 312ec29..d29a43f 100644 --- a/docs/internal/stage-extract.md +++ b/docs/internal/stage-extract.md @@ -4,8 +4,8 @@ `extract` runs after `trim` and before `render`. It converts the canonical `narratio.transcript.final_trimmed` JSON into configured Notarius lane artifacts. -An omitted or disabled Notarius section makes the stage succeed as a no-op and -does not require a Notarius runner. +An omitted or disabled Notarius section makes the stage explicitly self-skip +with reason `notarius_disabled`, no outputs, and no Notarius runner. The external protocol is documented in the [Notarius integration contract](../integrations/notarius.md). Configuration @@ -34,6 +34,10 @@ root, receipt, diagnostic paths, rejection/warning summaries, producing Narratio run ID, and invocation fingerprint. Validation completes before promotion, so a rejected result cannot expose a partial durable bundle. +Any executed extraction outcome that replaces a different effective outcome +marks succeeded downstream stages stale. Repeating the same disabled self-skip +with no outputs is stable and does not repeatedly invalidate downstream stages. + ## Resume Validation `internal/stage/extract_resume.go` permits a skip only when the existing stage @@ -63,14 +67,12 @@ has passed validation and promotion. ## Implementation And Focused Tests -- Stage execution and selection: `internal/stage/extract.go`, +- Stage execution, selection, and resume validation: `internal/stage/extract.go`, + `internal/stage/extract_resume.go`, `internal/stage/extract_test.go` -- Resume validation: `internal/stage/extract_resume.go`, - `internal/stage/extract_resume_test.go` - Subprocess boundary: `internal/adapters/notarius/subprocess.go`, `internal/adapters/notarius/subprocess_test.go` - Catalog hydration: `internal/artifacts/extraction_catalog.go`, `internal/artifacts/extraction_catalog_test.go` - Composition and downstream behavior: `internal/app/runner_test.go`, `internal/stage/analyze_test.go`, `internal/stage/publish_test.go` - diff --git a/docs/internal/stage-publish.md b/docs/internal/stage-publish.md index ceeaeea..9293e08 100644 --- a/docs/internal/stage-publish.md +++ b/docs/internal/stage-publish.md @@ -26,8 +26,13 @@ Exact remote placement and the operator workflow belong in - stage can self-skip when publish disabled or run upload disabled. - validates prerequisite stage success and object-store availability. -- collects deterministic run file list plus run `manifest.json`. +- collects a deterministic run file list plus run `manifest.json`, excluding + `audio/**` and the run-local `extract/notarius-output/**` staging bundle. +- keeps run-local Notarius receipt and stderr diagnostics eligible for the run + archive. - resolves publish output sources through runtime artifact catalog and manifest-aware resolution. +- publishes extraction lanes only through explicit configured output rules; + neither run-local nor durable Notarius bundles are scanned or uploaded wholesale. - selected artifact filter applies to configured artifact sources only. - locked outputs are skipped intentionally (including required ones). - optional missing outputs are skipped; required missing unlocked outputs fail. @@ -48,7 +53,9 @@ Includes counts/lists for: ## Invariants - `current/run_id.txt` is the remote commit marker and is written last. -- run upload excludes `audio/**`. +- run upload excludes `audio/**` and `extract/notarius-output/**`. +- `extract/notarius.receipt.json` and `extract/notarius.stderr.log` remain + eligible run-record diagnostics. - publish locks are not overridden by `--force`. The commit boundary and cleanup gate are normative architecture invariants; see diff --git a/docs/operations.md b/docs/operations.md index 65e8118..0516721 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -85,7 +85,12 @@ Execution rules: - succeeded stages are skipped unless `--force` is set; - `run` continues interrupted or partially completed sessions by running non-succeeded stages; -- force rerunning a succeeded upstream stage marks succeeded downstream stages as `stale`. +- forcing an upstream stage marks succeeded downstream stages as `stale` before + the replacement runs; and +- an executed failure, changed self-skip, or success that replaces a different + effective upstream outcome also marks succeeded downstream stages stale. A + repeated self-skip with the same reason and no outputs is stable and does not + perpetually rerun downstream work. Single-stage execution: @@ -108,6 +113,10 @@ Selection behavior: ## Extraction Workflow +When Notarius is omitted or disabled, `extract` records an explicit skipped +outcome with reason `notarius_disabled` and no outputs. A later invocation +reconsiders the skipped stage, so enabling Notarius does not require force. + When Notarius extraction is enabled, the stage consumes the final trimmed JSON and preserves the complete validated Notarius bundle at: @@ -130,6 +139,12 @@ Run-local diagnostics are: - `runs/{run_id}/extract/notarius.stderr.log` - `runs/{run_id}/extract/notarius-output/` before durable promotion +The run-record upload excludes the complete +`extract/notarius-output/**` subtree. The receipt and stderr files remain +eligible run-record diagnostics. The durable bundle is never scanned for +implicit publication; only lanes named by explicit `pipeline.publish.outputs` +rules are uploaded. + To intentionally replace the current extraction result, run: ```bash @@ -141,7 +156,9 @@ or durable output validation changes. It cannot fingerprint configuration files, profiles, prompts, modules, or references loaded transitively by Notarius. Force extraction after changing any of those inputs, even when the top-level Narratio and Notarius config paths remain the same. A forced extract -marks successful downstream stages stale through normal continuation behavior. +marks successful downstream stages stale. Ordinary extraction failures or +outcome changes also stale affected downstream stages, while an identical +repeated `notarius_disabled` self-skip does not repeatedly invalidate them. ## Publish Workflow @@ -159,8 +176,10 @@ narratio run-stage publish 2026-04-04 --force Publish commit model: -- uploads run files under `{session_prefix}/runs/{run_id}/`; -- uploads configured published outputs; +- uploads eligible run files under `{session_prefix}/runs/{run_id}/`, excluding + audio and the run-local Notarius staging bundle; +- uploads configured published outputs, including only explicitly configured + extraction lanes; - uploads `previous/**` cache files when present; - writes `current/manifest.json`; - writes `current/run_id.txt` last. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 237c0ae..1f8f0f0 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -2,8 +2,8 @@ ## Status And Audience -Remediation planned. Stages 1 through 9 are complete; Stages 10 through 14 -address gaps found during the post-implementation architecture review. +Complete. Stages 1 through 9 delivered the feature, and Stages 10 through 14 +closed the gaps found during the post-implementation architecture review. This plan is written for a GPT-5.6 Terra coding agent. Implement incomplete stages in strict numerical order. Do not skip ahead, combine stages merely to @@ -31,7 +31,7 @@ implementation sequence. | Stage 11 | Complete | | Stage 12 | Complete | | Stage 13 | Complete | -| Stage 14 | Pending | +| Stage 14 | Complete | After completing and validating a stage, update only that stage's row to `Complete` and record any material deviation in the relevant stage section. diff --git a/docs/roadmap/notarius-extract-stage.md b/docs/roadmap/notarius-extract-stage.md index fcd89d7..dd68b68 100644 --- a/docs/roadmap/notarius-extract-stage.md +++ b/docs/roadmap/notarius-extract-stage.md @@ -68,9 +68,10 @@ prepare -> transcribe -> merge -> polish -> normalize -> trim `extract` consumes `narratio.transcript.final_trimmed`, produced by `trim`. It does not consume rendered Markdown. -Narratio currently invalidates succeeded stages by canonical downstream order. -Placing `extract` before `render` means forcing extraction may rerun the less -expensive deterministic render stage, while forcing render does not rerun the +Narratio invalidates succeeded stages by canonical downstream order whenever a +forced run or a different executed upstream outcome replaces current state. +Placing `extract` before `render` means replacing extraction may rerun the less +expensive deterministic render stage, while replacing render does not rerun the more expensive Notarius pipeline. This is preferable to placing extraction after render and does not require dependency-aware scheduling or a DAG. @@ -127,8 +128,8 @@ The adapter owns: - validation of required receipt fields; - confinement of the receipt's absolute `output_directory` beneath the absolute output root Narratio supplied for this invocation; -- confinement of `index_file` beneath the receipt's absolute - `output_directory`; +- requiring receipt `index_file` to be exactly `index.json` beneath the + receipt's absolute `output_directory`; - tolerant decoding of the supported `index.json` contract; - confinement of every index descriptor path beneath the bundle root; and - bounded tolerant decoding of `rejected.json` and `warnings.json` into @@ -163,8 +164,9 @@ The initial compatibility baseline is: - `notarius run --config ... --input ... --output-dir ... --json`; - successful receipt schema `notarius.run-result.v1`; - an absolute receipt `output_directory`; -- logical `index_file` discovery beneath that directory; -- the production JSON `index.json` descriptor model; and +- receipt `index_file` exactly `index.json` beneath that directory; +- the production JSON `index.json` descriptor model with management files + exactly `manifest.json`, `rejected.json`, and `warnings.json`; and - the exact media type and schema identity configured for each required lane. Compatibility is decided from these published contracts, not by parsing @@ -287,8 +289,15 @@ transitive content of files owned by Notarius. - Forcing `trim` or an earlier stage stales succeeded `extract` and all later stages. - Forcing `extract` stales succeeded `render`, `analyze`, `publish`, and - `notify` under the existing canonical-order rule. + `notify` under the canonical-order rule, even if extraction later skips or + fails. - Forcing `render` does not stale `extract` because extraction precedes it. +- A non-forced extraction failure, changed self-skip, or success replacing a + different effective outcome stales succeeded downstream stages. Repeating + the same `notarius_disabled` self-skip with no outputs does not stale them + again. +- Reusing a resumable succeeded result does not invalidate downstream stages; + a resume-validation error stops without mutating either result. - A failed, skipped, stale, or interrupted extract stage never supplies current extraction sources. @@ -314,6 +323,11 @@ temporary directory on the same filesystem, recursively copies only regular files and directories, rejects symlinks and special files, preserves relative layout, and renames the completed temporary tree into place. +Atomic no-replace promotion is implemented on Linux, macOS, and Windows. On +other operating systems extraction reports an unsupported-capability error +before creating the sibling temporary tree. This limitation is confined to +extraction bundle promotion and does not define a broader platform guarantee. + The promoted tree preserves `index.json`, `manifest.json`, `rejected.json`, `warnings.json`, `lanes/`, and any emitted pipeline-wide artifacts such as `chunk-map.json` and `evidence-context.json`. Unknown regular files may be @@ -434,10 +448,13 @@ Publish source validation and resolution accept configured catalog. Operators may publish individual structured lanes without manually copying files. -The Notarius bundle is not automatically published wholesale. Bundle files or -lanes are published only through explicit configured publish rules. Existing -publish locking, destination safety, commit ordering, and required/unselected -artifact behavior remain unchanged. +The run-record upload excludes the run-local +`extract/notarius-output/**` staging bundle while retaining its receipt and +stderr diagnostics as eligible run files. Durable Notarius bundles are not +scanned or automatically published wholesale. Only lanes named by explicit +configured publish rules are uploaded. Existing publish locking, destination +safety, commit ordering, and required/unselected artifact behavior remain +unchanged. ## Failure And Diagnostic Semantics @@ -568,14 +585,22 @@ required after focused tests pass. - Extraction consumes the manifest-resolved final trimmed Seriatim JSON. - Disabled extraction is recorded as skipped and runs normally if later enabled. +- Different executed extraction outcomes stale succeeded downstream stages, + while an identical repeated disabled self-skip remains stable. - Successful resume requires valid manifest-recorded immutable outputs rather than filesystem presence alone. - The complete bundle is promoted to a unique immutable directory without following symlinks or exposing a partial destination. +- Atomic no-replace bundle promotion is supported on Linux, macOS, and Windows; + unsupported operating systems fail before a temporary promotion tree is + created. - Every configured lane is discovered by lane ID, contract-checked, checksummed, and recorded with explicit source identity and external provenance. - Analysis and publish resolve extraction sources only from a current successful extract manifest record. +- Run-record uploads exclude the staged Notarius bundle, and only explicitly + configured extraction lanes are published; receipt and stderr diagnostics + remain eligible run files. - Failed, skipped, stale, partial, rejected, unsafe, incompatible, or tampered output never becomes a current input. - The complete D&D example maps all ten current lanes and demonstrates curated diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index d58c089..134b71e 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -154,6 +154,29 @@ Safe fix: Relevant reference: [Operations: Extraction Workflow](./operations.md#extraction-workflow). +## Atomic Notarius promotion unsupported + +Symptom: + +- extraction fails with `atomic no-replace directory promotion is unsupported` + before a durable bundle or temporary promotion tree is created. + +Likely cause: + +- Narratio is running on an operating system other than Linux, macOS, or + Windows, where the required atomic no-replace directory primitive has not + been implemented and verified. + +Safe fix: + +- run extraction on Linux, macOS, or Windows. Do not replace the atomic commit + with a manual copy or move; the session manifest must never observe a partial + or overwritten bundle. + +This is an extraction-specific platform boundary, not a support statement for +unrelated Narratio workflows. See +[Operations: Extraction Workflow](./operations.md#extraction-workflow). + ## Notarius receipt or index incompatible Symptom: @@ -170,8 +193,10 @@ Likely causes: Safe fix: - compare installed Notarius output with the canonical Notarius contracts, - align `pipeline.notarius` constraints, and rerun. Do not bypass confinement or - schema checks. + including receipt `index_file: index.json` and index management names + `manifest.json`, `rejected.json`, and `warnings.json`; align + `pipeline.notarius` constraints and rerun. Do not bypass confinement or schema + checks. Relevant reference: [Notarius integration](./integrations/notarius.md).