Reconcile Notarius extraction documentation

This commit is contained in:
2026-08-10 02:10:37 +00:00
parent 665039f4dc
commit 701b6726d7
8 changed files with 113 additions and 33 deletions

View File

@@ -76,7 +76,7 @@ Behavior:
- evaluates full stage order; - evaluates full stage order;
- runs `extract` between `trim` and `render`; an omitted or disabled Notarius - runs `extract` between `trim` and `render`; an omitted or disabled Notarius
configuration makes extraction a no-op; configuration records an explicit `notarius_disabled` self-skip;
- skips already-succeeded stages unless `--force` is set or a stage-specific - skips already-succeeded stages unless `--force` is set or a stage-specific
resume check finds its durable result obsolete; resume check finds its durable result obsolete;
- continues interrupted or partially completed sessions by running non-succeeded stages; - continues interrupted or partially completed sessions by running non-succeeded stages;

View File

@@ -39,9 +39,11 @@ apply to the subprocess.
## Accepted Result ## Accepted Result
Narratio currently accepts receipt schema `notarius.run-result.v1`. The receipt Narratio currently accepts receipt schema `notarius.run-result.v1`. The receipt
must identify the configured pipeline and a bundle root containing a confined must identify the configured pipeline, and its `index_file` must be exactly
`index.json`. All receipt, index, and lane paths must stay inside that bundle; `index.json` beneath the reported bundle root. The production index must name
symlinks and non-regular lane payloads are rejected. the management files exactly as `manifest.json`, `rejected.json`, and
`warnings.json`. All receipt, index, and lane paths must stay inside that
bundle; symlinks and non-regular lane payloads are rejected.
Supported receipt and index shapes tolerate unknown fields for forward Supported receipt and index shapes tolerate unknown fields for forward
compatibility, while required identity, validation, count, manifest, compatibility, while required identity, validation, count, manifest,

View File

@@ -4,8 +4,8 @@
`extract` runs after `trim` and before `render`. It converts the canonical `extract` runs after `trim` and before `render`. It converts the canonical
`narratio.transcript.final_trimmed` JSON into configured Notarius lane artifacts. `narratio.transcript.final_trimmed` JSON into configured Notarius lane artifacts.
An omitted or disabled Notarius section makes the stage succeed as a no-op and An omitted or disabled Notarius section makes the stage explicitly self-skip
does not require a Notarius runner. with reason `notarius_disabled`, no outputs, and no Notarius runner.
The external protocol is documented in the The external protocol is documented in the
[Notarius integration contract](../integrations/notarius.md). Configuration [Notarius integration contract](../integrations/notarius.md). Configuration
@@ -34,6 +34,10 @@ root, receipt, diagnostic paths, rejection/warning summaries, producing
Narratio run ID, and invocation fingerprint. Validation completes before Narratio run ID, and invocation fingerprint. Validation completes before
promotion, so a rejected result cannot expose a partial durable bundle. promotion, so a rejected result cannot expose a partial durable bundle.
Any executed extraction outcome that replaces a different effective outcome
marks succeeded downstream stages stale. Repeating the same disabled self-skip
with no outputs is stable and does not repeatedly invalidate downstream stages.
## Resume Validation ## Resume Validation
`internal/stage/extract_resume.go` permits a skip only when the existing stage `internal/stage/extract_resume.go` permits a skip only when the existing stage
@@ -63,14 +67,12 @@ has passed validation and promotion.
## Implementation And Focused Tests ## Implementation And Focused Tests
- Stage execution and selection: `internal/stage/extract.go`, - Stage execution, selection, and resume validation: `internal/stage/extract.go`,
`internal/stage/extract_resume.go`,
`internal/stage/extract_test.go` `internal/stage/extract_test.go`
- Resume validation: `internal/stage/extract_resume.go`,
`internal/stage/extract_resume_test.go`
- Subprocess boundary: `internal/adapters/notarius/subprocess.go`, - Subprocess boundary: `internal/adapters/notarius/subprocess.go`,
`internal/adapters/notarius/subprocess_test.go` `internal/adapters/notarius/subprocess_test.go`
- Catalog hydration: `internal/artifacts/extraction_catalog.go`, - Catalog hydration: `internal/artifacts/extraction_catalog.go`,
`internal/artifacts/extraction_catalog_test.go` `internal/artifacts/extraction_catalog_test.go`
- Composition and downstream behavior: `internal/app/runner_test.go`, - Composition and downstream behavior: `internal/app/runner_test.go`,
`internal/stage/analyze_test.go`, `internal/stage/publish_test.go` `internal/stage/analyze_test.go`, `internal/stage/publish_test.go`

View File

@@ -26,8 +26,13 @@ Exact remote placement and the operator workflow belong in
- stage can self-skip when publish disabled or run upload disabled. - stage can self-skip when publish disabled or run upload disabled.
- validates prerequisite stage success and object-store availability. - validates prerequisite stage success and object-store availability.
- collects deterministic run file list plus run `manifest.json`. - collects a deterministic run file list plus run `manifest.json`, excluding
`audio/**` and the run-local `extract/notarius-output/**` staging bundle.
- keeps run-local Notarius receipt and stderr diagnostics eligible for the run
archive.
- resolves publish output sources through runtime artifact catalog and manifest-aware resolution. - resolves publish output sources through runtime artifact catalog and manifest-aware resolution.
- publishes extraction lanes only through explicit configured output rules;
neither run-local nor durable Notarius bundles are scanned or uploaded wholesale.
- selected artifact filter applies to configured artifact sources only. - selected artifact filter applies to configured artifact sources only.
- locked outputs are skipped intentionally (including required ones). - locked outputs are skipped intentionally (including required ones).
- optional missing outputs are skipped; required missing unlocked outputs fail. - optional missing outputs are skipped; required missing unlocked outputs fail.
@@ -48,7 +53,9 @@ Includes counts/lists for:
## Invariants ## Invariants
- `current/run_id.txt` is the remote commit marker and is written last. - `current/run_id.txt` is the remote commit marker and is written last.
- run upload excludes `audio/**`. - run upload excludes `audio/**` and `extract/notarius-output/**`.
- `extract/notarius.receipt.json` and `extract/notarius.stderr.log` remain
eligible run-record diagnostics.
- publish locks are not overridden by `--force`. - publish locks are not overridden by `--force`.
The commit boundary and cleanup gate are normative architecture invariants; see The commit boundary and cleanup gate are normative architecture invariants; see

View File

@@ -85,7 +85,12 @@ Execution rules:
- succeeded stages are skipped unless `--force` is set; - succeeded stages are skipped unless `--force` is set;
- `run` continues interrupted or partially completed sessions by running non-succeeded stages; - `run` continues interrupted or partially completed sessions by running non-succeeded stages;
- force rerunning a succeeded upstream stage marks succeeded downstream stages as `stale`. - forcing an upstream stage marks succeeded downstream stages as `stale` before
the replacement runs; and
- an executed failure, changed self-skip, or success that replaces a different
effective upstream outcome also marks succeeded downstream stages stale. A
repeated self-skip with the same reason and no outputs is stable and does not
perpetually rerun downstream work.
Single-stage execution: Single-stage execution:
@@ -108,6 +113,10 @@ Selection behavior:
## Extraction Workflow ## Extraction Workflow
When Notarius is omitted or disabled, `extract` records an explicit skipped
outcome with reason `notarius_disabled` and no outputs. A later invocation
reconsiders the skipped stage, so enabling Notarius does not require force.
When Notarius extraction is enabled, the stage consumes the final trimmed JSON When Notarius extraction is enabled, the stage consumes the final trimmed JSON
and preserves the complete validated Notarius bundle at: and preserves the complete validated Notarius bundle at:
@@ -130,6 +139,12 @@ Run-local diagnostics are:
- `runs/{run_id}/extract/notarius.stderr.log` - `runs/{run_id}/extract/notarius.stderr.log`
- `runs/{run_id}/extract/notarius-output/` before durable promotion - `runs/{run_id}/extract/notarius-output/` before durable promotion
The run-record upload excludes the complete
`extract/notarius-output/**` subtree. The receipt and stderr files remain
eligible run-record diagnostics. The durable bundle is never scanned for
implicit publication; only lanes named by explicit `pipeline.publish.outputs`
rules are uploaded.
To intentionally replace the current extraction result, run: To intentionally replace the current extraction result, run:
```bash ```bash
@@ -141,7 +156,9 @@ or durable output validation changes. It cannot fingerprint configuration
files, profiles, prompts, modules, or references loaded transitively by files, profiles, prompts, modules, or references loaded transitively by
Notarius. Force extraction after changing any of those inputs, even when the Notarius. Force extraction after changing any of those inputs, even when the
top-level Narratio and Notarius config paths remain the same. A forced extract top-level Narratio and Notarius config paths remain the same. A forced extract
marks successful downstream stages stale through normal continuation behavior. marks successful downstream stages stale. Ordinary extraction failures or
outcome changes also stale affected downstream stages, while an identical
repeated `notarius_disabled` self-skip does not repeatedly invalidate them.
## Publish Workflow ## Publish Workflow
@@ -159,8 +176,10 @@ narratio run-stage publish 2026-04-04 --force
Publish commit model: Publish commit model:
- uploads run files under `{session_prefix}/runs/{run_id}/`; - uploads eligible run files under `{session_prefix}/runs/{run_id}/`, excluding
- uploads configured published outputs; audio and the run-local Notarius staging bundle;
- uploads configured published outputs, including only explicitly configured
extraction lanes;
- uploads `previous/**` cache files when present; - uploads `previous/**` cache files when present;
- writes `current/manifest.json`; - writes `current/manifest.json`;
- writes `current/run_id.txt` last. - writes `current/run_id.txt` last.

View File

@@ -2,8 +2,8 @@
## Status And Audience ## Status And Audience
Remediation planned. Stages 1 through 9 are complete; Stages 10 through 14 Complete. Stages 1 through 9 delivered the feature, and Stages 10 through 14
address gaps found during the post-implementation architecture review. closed the gaps found during the post-implementation architecture review.
This plan is written for a GPT-5.6 Terra coding agent. Implement incomplete This plan is written for a GPT-5.6 Terra coding agent. Implement incomplete
stages in strict numerical order. Do not skip ahead, combine stages merely to stages in strict numerical order. Do not skip ahead, combine stages merely to
@@ -31,7 +31,7 @@ implementation sequence.
| Stage 11 | Complete | | Stage 11 | Complete |
| Stage 12 | Complete | | Stage 12 | Complete |
| Stage 13 | Complete | | Stage 13 | Complete |
| Stage 14 | Pending | | Stage 14 | Complete |
After completing and validating a stage, update only that stage's row to After completing and validating a stage, update only that stage's row to
`Complete` and record any material deviation in the relevant stage section. `Complete` and record any material deviation in the relevant stage section.

View File

@@ -68,9 +68,10 @@ prepare -> transcribe -> merge -> polish -> normalize -> trim
`extract` consumes `narratio.transcript.final_trimmed`, produced by `trim`. It `extract` consumes `narratio.transcript.final_trimmed`, produced by `trim`. It
does not consume rendered Markdown. does not consume rendered Markdown.
Narratio currently invalidates succeeded stages by canonical downstream order. Narratio invalidates succeeded stages by canonical downstream order whenever a
Placing `extract` before `render` means forcing extraction may rerun the less forced run or a different executed upstream outcome replaces current state.
expensive deterministic render stage, while forcing render does not rerun the Placing `extract` before `render` means replacing extraction may rerun the less
expensive deterministic render stage, while replacing render does not rerun the
more expensive Notarius pipeline. This is preferable to placing extraction more expensive Notarius pipeline. This is preferable to placing extraction
after render and does not require dependency-aware scheduling or a DAG. after render and does not require dependency-aware scheduling or a DAG.
@@ -127,8 +128,8 @@ The adapter owns:
- validation of required receipt fields; - validation of required receipt fields;
- confinement of the receipt's absolute `output_directory` beneath the - confinement of the receipt's absolute `output_directory` beneath the
absolute output root Narratio supplied for this invocation; absolute output root Narratio supplied for this invocation;
- confinement of `index_file` beneath the receipt's absolute - requiring receipt `index_file` to be exactly `index.json` beneath the
`output_directory`; receipt's absolute `output_directory`;
- tolerant decoding of the supported `index.json` contract; - tolerant decoding of the supported `index.json` contract;
- confinement of every index descriptor path beneath the bundle root; and - confinement of every index descriptor path beneath the bundle root; and
- bounded tolerant decoding of `rejected.json` and `warnings.json` into - bounded tolerant decoding of `rejected.json` and `warnings.json` into
@@ -163,8 +164,9 @@ The initial compatibility baseline is:
- `notarius run <pipeline-id> --config ... --input ... --output-dir ... --json`; - `notarius run <pipeline-id> --config ... --input ... --output-dir ... --json`;
- successful receipt schema `notarius.run-result.v1`; - successful receipt schema `notarius.run-result.v1`;
- an absolute receipt `output_directory`; - an absolute receipt `output_directory`;
- logical `index_file` discovery beneath that directory; - receipt `index_file` exactly `index.json` beneath that directory;
- the production JSON `index.json` descriptor model; and - the production JSON `index.json` descriptor model with management files
exactly `manifest.json`, `rejected.json`, and `warnings.json`; and
- the exact media type and schema identity configured for each required lane. - the exact media type and schema identity configured for each required lane.
Compatibility is decided from these published contracts, not by parsing Compatibility is decided from these published contracts, not by parsing
@@ -287,8 +289,15 @@ transitive content of files owned by Notarius.
- Forcing `trim` or an earlier stage stales succeeded `extract` and all later - Forcing `trim` or an earlier stage stales succeeded `extract` and all later
stages. stages.
- Forcing `extract` stales succeeded `render`, `analyze`, `publish`, and - Forcing `extract` stales succeeded `render`, `analyze`, `publish`, and
`notify` under the existing canonical-order rule. `notify` under the canonical-order rule, even if extraction later skips or
fails.
- Forcing `render` does not stale `extract` because extraction precedes it. - Forcing `render` does not stale `extract` because extraction precedes it.
- A non-forced extraction failure, changed self-skip, or success replacing a
different effective outcome stales succeeded downstream stages. Repeating
the same `notarius_disabled` self-skip with no outputs does not stale them
again.
- Reusing a resumable succeeded result does not invalidate downstream stages;
a resume-validation error stops without mutating either result.
- A failed, skipped, stale, or interrupted extract stage never supplies current - A failed, skipped, stale, or interrupted extract stage never supplies current
extraction sources. extraction sources.
@@ -314,6 +323,11 @@ temporary directory on the same filesystem, recursively copies only regular
files and directories, rejects symlinks and special files, preserves relative files and directories, rejects symlinks and special files, preserves relative
layout, and renames the completed temporary tree into place. layout, and renames the completed temporary tree into place.
Atomic no-replace promotion is implemented on Linux, macOS, and Windows. On
other operating systems extraction reports an unsupported-capability error
before creating the sibling temporary tree. This limitation is confined to
extraction bundle promotion and does not define a broader platform guarantee.
The promoted tree preserves `index.json`, `manifest.json`, `rejected.json`, The promoted tree preserves `index.json`, `manifest.json`, `rejected.json`,
`warnings.json`, `lanes/`, and any emitted pipeline-wide artifacts such as `warnings.json`, `lanes/`, and any emitted pipeline-wide artifacts such as
`chunk-map.json` and `evidence-context.json`. Unknown regular files may be `chunk-map.json` and `evidence-context.json`. Unknown regular files may be
@@ -434,10 +448,13 @@ Publish source validation and resolution accept configured
catalog. Operators may publish individual structured lanes without manually catalog. Operators may publish individual structured lanes without manually
copying files. copying files.
The Notarius bundle is not automatically published wholesale. Bundle files or The run-record upload excludes the run-local
lanes are published only through explicit configured publish rules. Existing `extract/notarius-output/**` staging bundle while retaining its receipt and
publish locking, destination safety, commit ordering, and required/unselected stderr diagnostics as eligible run files. Durable Notarius bundles are not
artifact behavior remain unchanged. scanned or automatically published wholesale. Only lanes named by explicit
configured publish rules are uploaded. Existing publish locking, destination
safety, commit ordering, and required/unselected artifact behavior remain
unchanged.
## Failure And Diagnostic Semantics ## Failure And Diagnostic Semantics
@@ -568,14 +585,22 @@ required after focused tests pass.
- Extraction consumes the manifest-resolved final trimmed Seriatim JSON. - Extraction consumes the manifest-resolved final trimmed Seriatim JSON.
- Disabled extraction is recorded as skipped and runs normally if later - Disabled extraction is recorded as skipped and runs normally if later
enabled. enabled.
- Different executed extraction outcomes stale succeeded downstream stages,
while an identical repeated disabled self-skip remains stable.
- Successful resume requires valid manifest-recorded immutable outputs rather - Successful resume requires valid manifest-recorded immutable outputs rather
than filesystem presence alone. than filesystem presence alone.
- The complete bundle is promoted to a unique immutable directory without - The complete bundle is promoted to a unique immutable directory without
following symlinks or exposing a partial destination. following symlinks or exposing a partial destination.
- Atomic no-replace bundle promotion is supported on Linux, macOS, and Windows;
unsupported operating systems fail before a temporary promotion tree is
created.
- Every configured lane is discovered by lane ID, contract-checked, checksummed, - Every configured lane is discovered by lane ID, contract-checked, checksummed,
and recorded with explicit source identity and external provenance. and recorded with explicit source identity and external provenance.
- Analysis and publish resolve extraction sources only from a current - Analysis and publish resolve extraction sources only from a current
successful extract manifest record. successful extract manifest record.
- Run-record uploads exclude the staged Notarius bundle, and only explicitly
configured extraction lanes are published; receipt and stderr diagnostics
remain eligible run files.
- Failed, skipped, stale, partial, rejected, unsafe, incompatible, or tampered - Failed, skipped, stale, partial, rejected, unsafe, incompatible, or tampered
output never becomes a current input. output never becomes a current input.
- The complete D&D example maps all ten current lanes and demonstrates curated - The complete D&D example maps all ten current lanes and demonstrates curated

View File

@@ -154,6 +154,29 @@ Safe fix:
Relevant reference: [Operations: Extraction Workflow](./operations.md#extraction-workflow). Relevant reference: [Operations: Extraction Workflow](./operations.md#extraction-workflow).
## Atomic Notarius promotion unsupported
Symptom:
- extraction fails with `atomic no-replace directory promotion is unsupported`
before a durable bundle or temporary promotion tree is created.
Likely cause:
- Narratio is running on an operating system other than Linux, macOS, or
Windows, where the required atomic no-replace directory primitive has not
been implemented and verified.
Safe fix:
- run extraction on Linux, macOS, or Windows. Do not replace the atomic commit
with a manual copy or move; the session manifest must never observe a partial
or overwritten bundle.
This is an extraction-specific platform boundary, not a support statement for
unrelated Narratio workflows. See
[Operations: Extraction Workflow](./operations.md#extraction-workflow).
## Notarius receipt or index incompatible ## Notarius receipt or index incompatible
Symptom: Symptom:
@@ -170,8 +193,10 @@ Likely causes:
Safe fix: Safe fix:
- compare installed Notarius output with the canonical Notarius contracts, - compare installed Notarius output with the canonical Notarius contracts,
align `pipeline.notarius` constraints, and rerun. Do not bypass confinement or including receipt `index_file: index.json` and index management names
schema checks. `manifest.json`, `rejected.json`, and `warnings.json`; align
`pipeline.notarius` constraints and rerun. Do not bypass confinement or schema
checks.
Relevant reference: [Notarius integration](./integrations/notarius.md). Relevant reference: [Notarius integration](./integrations/notarius.md).