Reconcile Notarius extraction documentation
This commit is contained in:
@@ -76,7 +76,7 @@ Behavior:
|
||||
|
||||
- evaluates full stage order;
|
||||
- runs `extract` between `trim` and `render`; an omitted or disabled Notarius
|
||||
configuration makes extraction a no-op;
|
||||
configuration records an explicit `notarius_disabled` self-skip;
|
||||
- skips already-succeeded stages unless `--force` is set or a stage-specific
|
||||
resume check finds its durable result obsolete;
|
||||
- continues interrupted or partially completed sessions by running non-succeeded stages;
|
||||
|
||||
@@ -39,9 +39,11 @@ apply to the subprocess.
|
||||
## Accepted Result
|
||||
|
||||
Narratio currently accepts receipt schema `notarius.run-result.v1`. The receipt
|
||||
must identify the configured pipeline and a bundle root containing a confined
|
||||
`index.json`. All receipt, index, and lane paths must stay inside that bundle;
|
||||
symlinks and non-regular lane payloads are rejected.
|
||||
must identify the configured pipeline, and its `index_file` must be exactly
|
||||
`index.json` beneath the reported bundle root. The production index must name
|
||||
the management files exactly as `manifest.json`, `rejected.json`, and
|
||||
`warnings.json`. All receipt, index, and lane paths must stay inside that
|
||||
bundle; symlinks and non-regular lane payloads are rejected.
|
||||
|
||||
Supported receipt and index shapes tolerate unknown fields for forward
|
||||
compatibility, while required identity, validation, count, manifest,
|
||||
|
||||
@@ -4,8 +4,8 @@
|
||||
|
||||
`extract` runs after `trim` and before `render`. It converts the canonical
|
||||
`narratio.transcript.final_trimmed` JSON into configured Notarius lane artifacts.
|
||||
An omitted or disabled Notarius section makes the stage succeed as a no-op and
|
||||
does not require a Notarius runner.
|
||||
An omitted or disabled Notarius section makes the stage explicitly self-skip
|
||||
with reason `notarius_disabled`, no outputs, and no Notarius runner.
|
||||
|
||||
The external protocol is documented in the
|
||||
[Notarius integration contract](../integrations/notarius.md). Configuration
|
||||
@@ -34,6 +34,10 @@ root, receipt, diagnostic paths, rejection/warning summaries, producing
|
||||
Narratio run ID, and invocation fingerprint. Validation completes before
|
||||
promotion, so a rejected result cannot expose a partial durable bundle.
|
||||
|
||||
Any executed extraction outcome that replaces a different effective outcome
|
||||
marks succeeded downstream stages stale. Repeating the same disabled self-skip
|
||||
with no outputs is stable and does not repeatedly invalidate downstream stages.
|
||||
|
||||
## Resume Validation
|
||||
|
||||
`internal/stage/extract_resume.go` permits a skip only when the existing stage
|
||||
@@ -63,14 +67,12 @@ has passed validation and promotion.
|
||||
|
||||
## Implementation And Focused Tests
|
||||
|
||||
- Stage execution and selection: `internal/stage/extract.go`,
|
||||
- Stage execution, selection, and resume validation: `internal/stage/extract.go`,
|
||||
`internal/stage/extract_resume.go`,
|
||||
`internal/stage/extract_test.go`
|
||||
- Resume validation: `internal/stage/extract_resume.go`,
|
||||
`internal/stage/extract_resume_test.go`
|
||||
- Subprocess boundary: `internal/adapters/notarius/subprocess.go`,
|
||||
`internal/adapters/notarius/subprocess_test.go`
|
||||
- Catalog hydration: `internal/artifacts/extraction_catalog.go`,
|
||||
`internal/artifacts/extraction_catalog_test.go`
|
||||
- Composition and downstream behavior: `internal/app/runner_test.go`,
|
||||
`internal/stage/analyze_test.go`, `internal/stage/publish_test.go`
|
||||
|
||||
|
||||
@@ -26,8 +26,13 @@ Exact remote placement and the operator workflow belong in
|
||||
|
||||
- stage can self-skip when publish disabled or run upload disabled.
|
||||
- validates prerequisite stage success and object-store availability.
|
||||
- collects deterministic run file list plus run `manifest.json`.
|
||||
- collects a deterministic run file list plus run `manifest.json`, excluding
|
||||
`audio/**` and the run-local `extract/notarius-output/**` staging bundle.
|
||||
- keeps run-local Notarius receipt and stderr diagnostics eligible for the run
|
||||
archive.
|
||||
- resolves publish output sources through runtime artifact catalog and manifest-aware resolution.
|
||||
- publishes extraction lanes only through explicit configured output rules;
|
||||
neither run-local nor durable Notarius bundles are scanned or uploaded wholesale.
|
||||
- selected artifact filter applies to configured artifact sources only.
|
||||
- locked outputs are skipped intentionally (including required ones).
|
||||
- optional missing outputs are skipped; required missing unlocked outputs fail.
|
||||
@@ -48,7 +53,9 @@ Includes counts/lists for:
|
||||
## Invariants
|
||||
|
||||
- `current/run_id.txt` is the remote commit marker and is written last.
|
||||
- run upload excludes `audio/**`.
|
||||
- run upload excludes `audio/**` and `extract/notarius-output/**`.
|
||||
- `extract/notarius.receipt.json` and `extract/notarius.stderr.log` remain
|
||||
eligible run-record diagnostics.
|
||||
- publish locks are not overridden by `--force`.
|
||||
|
||||
The commit boundary and cleanup gate are normative architecture invariants; see
|
||||
|
||||
@@ -85,7 +85,12 @@ Execution rules:
|
||||
|
||||
- succeeded stages are skipped unless `--force` is set;
|
||||
- `run` continues interrupted or partially completed sessions by running non-succeeded stages;
|
||||
- force rerunning a succeeded upstream stage marks succeeded downstream stages as `stale`.
|
||||
- forcing an upstream stage marks succeeded downstream stages as `stale` before
|
||||
the replacement runs; and
|
||||
- an executed failure, changed self-skip, or success that replaces a different
|
||||
effective upstream outcome also marks succeeded downstream stages stale. A
|
||||
repeated self-skip with the same reason and no outputs is stable and does not
|
||||
perpetually rerun downstream work.
|
||||
|
||||
Single-stage execution:
|
||||
|
||||
@@ -108,6 +113,10 @@ Selection behavior:
|
||||
|
||||
## Extraction Workflow
|
||||
|
||||
When Notarius is omitted or disabled, `extract` records an explicit skipped
|
||||
outcome with reason `notarius_disabled` and no outputs. A later invocation
|
||||
reconsiders the skipped stage, so enabling Notarius does not require force.
|
||||
|
||||
When Notarius extraction is enabled, the stage consumes the final trimmed JSON
|
||||
and preserves the complete validated Notarius bundle at:
|
||||
|
||||
@@ -130,6 +139,12 @@ Run-local diagnostics are:
|
||||
- `runs/{run_id}/extract/notarius.stderr.log`
|
||||
- `runs/{run_id}/extract/notarius-output/` before durable promotion
|
||||
|
||||
The run-record upload excludes the complete
|
||||
`extract/notarius-output/**` subtree. The receipt and stderr files remain
|
||||
eligible run-record diagnostics. The durable bundle is never scanned for
|
||||
implicit publication; only lanes named by explicit `pipeline.publish.outputs`
|
||||
rules are uploaded.
|
||||
|
||||
To intentionally replace the current extraction result, run:
|
||||
|
||||
```bash
|
||||
@@ -141,7 +156,9 @@ or durable output validation changes. It cannot fingerprint configuration
|
||||
files, profiles, prompts, modules, or references loaded transitively by
|
||||
Notarius. Force extraction after changing any of those inputs, even when the
|
||||
top-level Narratio and Notarius config paths remain the same. A forced extract
|
||||
marks successful downstream stages stale through normal continuation behavior.
|
||||
marks successful downstream stages stale. Ordinary extraction failures or
|
||||
outcome changes also stale affected downstream stages, while an identical
|
||||
repeated `notarius_disabled` self-skip does not repeatedly invalidate them.
|
||||
|
||||
## Publish Workflow
|
||||
|
||||
@@ -159,8 +176,10 @@ narratio run-stage publish 2026-04-04 --force
|
||||
|
||||
Publish commit model:
|
||||
|
||||
- uploads run files under `{session_prefix}/runs/{run_id}/`;
|
||||
- uploads configured published outputs;
|
||||
- uploads eligible run files under `{session_prefix}/runs/{run_id}/`, excluding
|
||||
audio and the run-local Notarius staging bundle;
|
||||
- uploads configured published outputs, including only explicitly configured
|
||||
extraction lanes;
|
||||
- uploads `previous/**` cache files when present;
|
||||
- writes `current/manifest.json`;
|
||||
- writes `current/run_id.txt` last.
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
## Status And Audience
|
||||
|
||||
Remediation planned. Stages 1 through 9 are complete; Stages 10 through 14
|
||||
address gaps found during the post-implementation architecture review.
|
||||
Complete. Stages 1 through 9 delivered the feature, and Stages 10 through 14
|
||||
closed the gaps found during the post-implementation architecture review.
|
||||
|
||||
This plan is written for a GPT-5.6 Terra coding agent. Implement incomplete
|
||||
stages in strict numerical order. Do not skip ahead, combine stages merely to
|
||||
@@ -31,7 +31,7 @@ implementation sequence.
|
||||
| Stage 11 | Complete |
|
||||
| Stage 12 | Complete |
|
||||
| Stage 13 | Complete |
|
||||
| Stage 14 | Pending |
|
||||
| Stage 14 | Complete |
|
||||
|
||||
After completing and validating a stage, update only that stage's row to
|
||||
`Complete` and record any material deviation in the relevant stage section.
|
||||
|
||||
@@ -68,9 +68,10 @@ prepare -> transcribe -> merge -> polish -> normalize -> trim
|
||||
`extract` consumes `narratio.transcript.final_trimmed`, produced by `trim`. It
|
||||
does not consume rendered Markdown.
|
||||
|
||||
Narratio currently invalidates succeeded stages by canonical downstream order.
|
||||
Placing `extract` before `render` means forcing extraction may rerun the less
|
||||
expensive deterministic render stage, while forcing render does not rerun the
|
||||
Narratio invalidates succeeded stages by canonical downstream order whenever a
|
||||
forced run or a different executed upstream outcome replaces current state.
|
||||
Placing `extract` before `render` means replacing extraction may rerun the less
|
||||
expensive deterministic render stage, while replacing render does not rerun the
|
||||
more expensive Notarius pipeline. This is preferable to placing extraction
|
||||
after render and does not require dependency-aware scheduling or a DAG.
|
||||
|
||||
@@ -127,8 +128,8 @@ The adapter owns:
|
||||
- validation of required receipt fields;
|
||||
- confinement of the receipt's absolute `output_directory` beneath the
|
||||
absolute output root Narratio supplied for this invocation;
|
||||
- confinement of `index_file` beneath the receipt's absolute
|
||||
`output_directory`;
|
||||
- requiring receipt `index_file` to be exactly `index.json` beneath the
|
||||
receipt's absolute `output_directory`;
|
||||
- tolerant decoding of the supported `index.json` contract;
|
||||
- confinement of every index descriptor path beneath the bundle root; and
|
||||
- bounded tolerant decoding of `rejected.json` and `warnings.json` into
|
||||
@@ -163,8 +164,9 @@ The initial compatibility baseline is:
|
||||
- `notarius run <pipeline-id> --config ... --input ... --output-dir ... --json`;
|
||||
- successful receipt schema `notarius.run-result.v1`;
|
||||
- an absolute receipt `output_directory`;
|
||||
- logical `index_file` discovery beneath that directory;
|
||||
- the production JSON `index.json` descriptor model; and
|
||||
- receipt `index_file` exactly `index.json` beneath that directory;
|
||||
- the production JSON `index.json` descriptor model with management files
|
||||
exactly `manifest.json`, `rejected.json`, and `warnings.json`; and
|
||||
- the exact media type and schema identity configured for each required lane.
|
||||
|
||||
Compatibility is decided from these published contracts, not by parsing
|
||||
@@ -287,8 +289,15 @@ transitive content of files owned by Notarius.
|
||||
- Forcing `trim` or an earlier stage stales succeeded `extract` and all later
|
||||
stages.
|
||||
- Forcing `extract` stales succeeded `render`, `analyze`, `publish`, and
|
||||
`notify` under the existing canonical-order rule.
|
||||
`notify` under the canonical-order rule, even if extraction later skips or
|
||||
fails.
|
||||
- Forcing `render` does not stale `extract` because extraction precedes it.
|
||||
- A non-forced extraction failure, changed self-skip, or success replacing a
|
||||
different effective outcome stales succeeded downstream stages. Repeating
|
||||
the same `notarius_disabled` self-skip with no outputs does not stale them
|
||||
again.
|
||||
- Reusing a resumable succeeded result does not invalidate downstream stages;
|
||||
a resume-validation error stops without mutating either result.
|
||||
- A failed, skipped, stale, or interrupted extract stage never supplies current
|
||||
extraction sources.
|
||||
|
||||
@@ -314,6 +323,11 @@ temporary directory on the same filesystem, recursively copies only regular
|
||||
files and directories, rejects symlinks and special files, preserves relative
|
||||
layout, and renames the completed temporary tree into place.
|
||||
|
||||
Atomic no-replace promotion is implemented on Linux, macOS, and Windows. On
|
||||
other operating systems extraction reports an unsupported-capability error
|
||||
before creating the sibling temporary tree. This limitation is confined to
|
||||
extraction bundle promotion and does not define a broader platform guarantee.
|
||||
|
||||
The promoted tree preserves `index.json`, `manifest.json`, `rejected.json`,
|
||||
`warnings.json`, `lanes/`, and any emitted pipeline-wide artifacts such as
|
||||
`chunk-map.json` and `evidence-context.json`. Unknown regular files may be
|
||||
@@ -434,10 +448,13 @@ Publish source validation and resolution accept configured
|
||||
catalog. Operators may publish individual structured lanes without manually
|
||||
copying files.
|
||||
|
||||
The Notarius bundle is not automatically published wholesale. Bundle files or
|
||||
lanes are published only through explicit configured publish rules. Existing
|
||||
publish locking, destination safety, commit ordering, and required/unselected
|
||||
artifact behavior remain unchanged.
|
||||
The run-record upload excludes the run-local
|
||||
`extract/notarius-output/**` staging bundle while retaining its receipt and
|
||||
stderr diagnostics as eligible run files. Durable Notarius bundles are not
|
||||
scanned or automatically published wholesale. Only lanes named by explicit
|
||||
configured publish rules are uploaded. Existing publish locking, destination
|
||||
safety, commit ordering, and required/unselected artifact behavior remain
|
||||
unchanged.
|
||||
|
||||
## Failure And Diagnostic Semantics
|
||||
|
||||
@@ -568,14 +585,22 @@ required after focused tests pass.
|
||||
- Extraction consumes the manifest-resolved final trimmed Seriatim JSON.
|
||||
- Disabled extraction is recorded as skipped and runs normally if later
|
||||
enabled.
|
||||
- Different executed extraction outcomes stale succeeded downstream stages,
|
||||
while an identical repeated disabled self-skip remains stable.
|
||||
- Successful resume requires valid manifest-recorded immutable outputs rather
|
||||
than filesystem presence alone.
|
||||
- The complete bundle is promoted to a unique immutable directory without
|
||||
following symlinks or exposing a partial destination.
|
||||
- Atomic no-replace bundle promotion is supported on Linux, macOS, and Windows;
|
||||
unsupported operating systems fail before a temporary promotion tree is
|
||||
created.
|
||||
- Every configured lane is discovered by lane ID, contract-checked, checksummed,
|
||||
and recorded with explicit source identity and external provenance.
|
||||
- Analysis and publish resolve extraction sources only from a current
|
||||
successful extract manifest record.
|
||||
- Run-record uploads exclude the staged Notarius bundle, and only explicitly
|
||||
configured extraction lanes are published; receipt and stderr diagnostics
|
||||
remain eligible run files.
|
||||
- Failed, skipped, stale, partial, rejected, unsafe, incompatible, or tampered
|
||||
output never becomes a current input.
|
||||
- The complete D&D example maps all ten current lanes and demonstrates curated
|
||||
|
||||
@@ -154,6 +154,29 @@ Safe fix:
|
||||
|
||||
Relevant reference: [Operations: Extraction Workflow](./operations.md#extraction-workflow).
|
||||
|
||||
## Atomic Notarius promotion unsupported
|
||||
|
||||
Symptom:
|
||||
|
||||
- extraction fails with `atomic no-replace directory promotion is unsupported`
|
||||
before a durable bundle or temporary promotion tree is created.
|
||||
|
||||
Likely cause:
|
||||
|
||||
- Narratio is running on an operating system other than Linux, macOS, or
|
||||
Windows, where the required atomic no-replace directory primitive has not
|
||||
been implemented and verified.
|
||||
|
||||
Safe fix:
|
||||
|
||||
- run extraction on Linux, macOS, or Windows. Do not replace the atomic commit
|
||||
with a manual copy or move; the session manifest must never observe a partial
|
||||
or overwritten bundle.
|
||||
|
||||
This is an extraction-specific platform boundary, not a support statement for
|
||||
unrelated Narratio workflows. See
|
||||
[Operations: Extraction Workflow](./operations.md#extraction-workflow).
|
||||
|
||||
## Notarius receipt or index incompatible
|
||||
|
||||
Symptom:
|
||||
@@ -170,8 +193,10 @@ Likely causes:
|
||||
Safe fix:
|
||||
|
||||
- compare installed Notarius output with the canonical Notarius contracts,
|
||||
align `pipeline.notarius` constraints, and rerun. Do not bypass confinement or
|
||||
schema checks.
|
||||
including receipt `index_file: index.json` and index management names
|
||||
`manifest.json`, `rejected.json`, and `warnings.json`; align
|
||||
`pipeline.notarius` constraints and rerun. Do not bypass confinement or schema
|
||||
checks.
|
||||
|
||||
Relevant reference: [Notarius integration](./integrations/notarius.md).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user