Internal: Extract Stage

Responsibility

extract runs after render and before analyze. It converts the canonical narratio.transcript.final_trimmed JSON into configured Notarius lane artifacts. An omitted or disabled Notarius section makes the stage explicitly self-skip with reason notarius_disabled, no outputs, and no Notarius runner.

The external protocol is documented in the Notarius integration contract. Configuration fields belong in Configuration, and physical paths and force procedures belong in Operations.

Lifecycle

internal/stage/extract.go:

resolves the final trimmed transcript from the shared artifact catalog;
resolves every configured prepared reference through the shared manifest-authoritative identity resolver before creating run-local output;
streams each verified reference into an invocation-local snapshot and rejects any source change observed while copying;
fingerprints the byte- and provenance-bearing Notarius invocation evidence, including sorted reference identities;
creates a run-local staging directory and invokes the injected notarius.Runner;
revalidates the reference snapshots, then validates the v2 successful receipt, confined index, management documents, configured required lane descriptors, validation summaries, and regular payload files;
atomically promotes the complete bundle to its immutable durable location;
records one non-selectable notarius_index output and one selectable notarius_lane output per configured lane; and
registers each lane as narratio.extraction.<output_key> for downstream Scriptorium and publish resolution.

Lane records retain checksum, contract, producer run ID, and Notarius system, run, pipeline, and lane provenance. Stage metadata retains the durable bundle root, receipt, diagnostic paths, rejection/warning summaries, producing Narratio run ID, the resolved trimmed-input identity, and invocation fingerprint. The input identity binds the exact transcript bytes, canonical source ID, producer stage/output/run identity, and resolution provenance. Reference metadata contains only selector, source ID, canonical session-relative path, checksum, and size; adapter requests receive selector and absolute invocation-local snapshot path, never payload contents. Snapshot bytes must match the prepared identity both before and after Notarius runs, so a concurrent prepared-file replacement cannot make recorded provenance describe different bytes from those supplied to Notarius. Validation completes before promotion, so a rejected result cannot expose a partial durable bundle.

Any executed extraction outcome that replaces a different effective outcome marks succeeded analysis and delivery dependents stale. Render is an independent sibling and remains current. Repeating the same disabled self-skip with no outputs is stable and does not repeatedly invalidate dependent stages.

Resume Validation

Before the focused validator runs, the application compares extract's versioned semantic fingerprint. It covers enablement, Notarius pipeline identity, sorted reference selector/source mappings, sorted declared output contracts, and each canonical narratio.extraction.<key> output identity. It excludes executable, timeout, working directory, config path, and private Notarius config contents.

internal/stage/extract_resume.go then permits a skip only when the existing stage record still matches the current byte- and provenance-bearing invocation evidence. That evidence covers the current direct trimmed-transcript identity, sorted prepared-reference identities, pipeline identity, and configured output contracts. The same reference helper and transcript identity are resolved again for artifact evidence, so changing current transcript bytes, reference bytes, or producer identity makes the prior extraction obsolete. Operational runner settings do not invalidate otherwise current durable evidence.

A valid prepared-reference change makes extraction non-resumable. Missing, unsafe, or checksum-inconsistent prepared evidence is a hard validation error with prepare-force guidance because an immediate extract rerun cannot succeed.

The validator then checks the producing run identity, canonical immutable bundle root, path confinement and absence of symlink components, receipt identity, exactly one canonical index, the exact configured source set, contracts and provenance, regular-file status, and stored checksums. Missing or obsolete results are non-resumable and run again; unsafe filesystem conditions return an error rather than silently accepting or replacing data.

Neither contract can observe files imported by Notarius configuration, profile contents, prompt/module definitions, or other transitive inputs. Operators must force extraction after changing any such private input behind a stable identifier.

Failure Behavior

Adapter startup, timeout, nonzero exit, receipt decoding, path confinement, index compatibility, inconsistent warning or diagnostic envelopes, required-lane rejection or incomplete validation, payload inspection, checksum, or promotion errors fail the stage through ordinary manifest transition handling. Stdout receipt and stderr diagnostics remain separate. Downstream stages are not given selectable extraction sources unless the complete configured result has passed validation and promotion.

When a replacement attempt begins, the current session-stage record no longer advertises payload from the previous success. A failed replacement therefore has no current outputs, logs, generated configuration references, or metadata, while the earlier invocation manifest and immutable promoted bundle remain available for audit and recovery.

Implementation And Focused Tests

Stage execution, selection, and resume validation: internal/stage/extract.go, internal/stage/extract_resume.go, internal/stage/extract_test.go, internal/stage/semantic_contracts_delivery.go
Subprocess boundary: internal/adapters/notarius/subprocess.go, internal/adapters/notarius/subprocess_test.go
Catalog hydration: internal/artifacts/extraction_catalog.go, internal/artifacts/extraction_catalog_test.go
Composition and downstream behavior: internal/app/runner_test.go, internal/stage/analyze_test.go, internal/stage/publish_test.go

6.3 KiB Raw Blame History