narratio/docs/roadmap/notarius-extract-stage.md

Notarius Extraction Stage

Status

Proposed. Implementation has not started.

The ordered implementation plan is maintained in implementation.md. Until that plan is complete, this document is the only Narratio documentation that describes the proposed extract stage, Notarius configuration, or narratio.extraction.* sources.

Purpose

Add a first-class Narratio extract stage that runs Notarius against the session's final trimmed transcript and makes validated structured artifacts available to later analysis and publish work.

Notarius remains responsible for its D&D extraction pipeline, prompts, references, LLM profiles, retries, validation, normalization, and published schemas. Narratio owns invocation, required-output policy, safe bundle ingestion, artifact identity, manifest state, resume, and downstream availability.

The integration must preserve Narratio's explicit stage model. It must not turn Narratio into a generic workflow engine or reproduce Notarius's configuration language.

User Outcome

An operator can enable one Notarius pipeline for a Narratio campaign. During a normal run, Narratio will:

1. produce the final trimmed Seriatim JSON transcript;
2. invoke Notarius once with that transcript;
3. discover the exact run bundle through Notarius's machine-readable receipt;
4. validate every output contract Narratio is configured to require;
5. promote the complete validated bundle into immutable session storage;
6. record exact artifact paths, checksums, contracts, and external provenance;
7. make configured lanes available as narratio.extraction.<key> sources; and
8. allow individual Scriptorium artifacts and publish rules to select those sources explicitly.

The maintained complete D&D example will require all ten lanes published by Notarius's dnd-session pipeline. Ordinary deployments may configure a narrower required set.

Chosen Architecture

First-Class Stage

Extraction is an independently observable and resumable pipeline stage. It is not part of analyze, a Scriptorium artifact producer, or an implicit external preprocessing requirement.

This boundary is required because Notarius work is expensive, produces multiple durable outputs, has its own compatibility and diagnostic contracts, and may be consumed by both analyze and publish.

Canonical Order

The canonical stage order becomes:

prepare -> transcribe -> merge -> polish -> normalize -> trim
        -> extract -> render -> analyze -> publish -> notify

extract consumes narratio.transcript.final_trimmed, produced by trim. It does not consume rendered Markdown.

Narratio currently invalidates succeeded stages by canonical downstream order. Placing extract before render means forcing extraction may rerun the less expensive deterministic render stage, while forcing render does not rerun the more expensive Notarius pipeline. This is preferable to placing extraction after render and does not require dependency-aware scheduling or a DAG.

Adding the stage must update every canonical-stage inventory, full-plan and single-stage selection, prerequisite validation, downstream invalidation, resume behavior, run manifests, CLI validation and help, and focused tests.

Stage Boundary

The stage owns Narratio policy:

• resolve the final trimmed transcript through the manifest-aware artifact resolver;
• allocate run-local receipt, log, and output-root paths;
• build a transport-neutral Notarius request from resolved configuration;
• call the configured Notarius adapter once;
• enforce Narratio's configured required-output policy;
• validate selected payloads as regular, non-empty, syntactically valid JSON;
• promote the validated bundle to immutable session artifact storage;
• return explicit source IDs, checksums, contracts, and external provenance; and
• report an explicit skipped outcome when Notarius is disabled.

The stage must not construct subprocess arguments, guess Notarius filenames, parse interactive output, decode D&D payload structures, or reproduce Notarius pipeline configuration.

Adapter Boundary

Add internal/adapters/notarius with a narrow runner interface, production subprocess implementation, and small fake.

The request contains only:

• resolved executable path or name;
• absolute Notarius configuration path;
• pipeline ID;
• absolute transcript path;
• absolute output root;
• working directory;
• timeout; and
• stdout receipt and stderr log destinations.

The adapter owns:

• exact notarius run ... --json argument construction;
• stdout and stderr separation;
• context cancellation and timeout through Narratio's shared subprocess boundary;
• environment inheritance;
• exit-status handling;
• bounded receipt loading after exit status zero;
• tolerant decoding of supported notarius.run-result.v1 documents;
• validation of required receipt fields;
• confinement of the receipt's absolute output_directory beneath the absolute output root Narratio supplied for this invocation;
• confinement of index_file beneath the receipt's absolute output_directory;
• tolerant decoding of the supported index.json contract;
• confinement of every index descriptor path beneath the bundle root; and
• bounded tolerant decoding of rejected.json and warnings.json into transport-neutral summaries without reading lane payload bodies; summaries persisted by Narratio contain structured scope, lane, and reason fields but not unbounded free-form external messages; and
• returning transport-neutral receipt, descriptor, diagnostic, and bundle information.

Only exit status zero permits receipt decoding. Unsupported schema versions, malformed documents, missing required fields, absolute logical paths, path escapes, symlinks at consumed paths, and incompatible structural metadata are integration failures.

The production runner will not execute notarius config validate before every session. Notarius run-time validation remains authoritative, while operators may use the separate validation command as deployment preflight. A second automatic subprocess can be added later only if operational evidence warrants it.

The adapter does not write Narratio manifests, decide required lanes, choose analysis inputs, or interpret D&D payloads.

External Contract Baseline

The integration consumes the contracts documented by Notarius in ../notarius/docs/consumers/dnd-pipeline.md and its linked canonical integration documents.

The initial compatibility baseline is:

• notarius run <pipeline-id> --config ... --input ... --output-dir ... --json;
• successful receipt schema notarius.run-result.v1;
• an absolute receipt output_directory;
• logical index_file discovery beneath that directory;
• the production JSON index.json descriptor model; and
• the exact media type and schema identity configured for each required lane.

Compatibility is decided from these published contracts, not by parsing notarius --version. Unknown fields in supported receipt and index versions are tolerated. Unsupported versions or incompatible required descriptors fail before any artifact becomes current in Narratio.

Configuration Contract

Add a strict optional pipeline.notarius section:

notarius:
  enabled: true
  binary: notarius
  config_path: /absolute/path/to/notarius.yml
  pipeline_id: dnd-session
  timeout: 3h
  working_directory: /absolute/path/to/deployment
  outputs:
    npc_registry:
      lane_id: npc-registry
      media_type: application/json
      schema_id: <published-schema-id>
      schema_version: <published-schema-version>

Fields:

• enabled is an explicit opt-in and defaults to false;
• binary defaults to notarius;
• config_path is required when enabled;
• pipeline_id is required when enabled;
• timeout defaults to 3h and must be positive;
• working_directory is optional and defaults to the directory containing config_path; and
• outputs maps stable Narratio extraction keys to required Notarius lane contracts.

All configured paths become absolute during configuration resolution. Notarius reference paths continue to follow Notarius's configuration-relative rules, while PromptKit profile paths remain relative to the chosen process working directory where Notarius permits that behavior.

Each output entry contains:

• exact lane_id;
• exact media_type;
• exact schema_id;
• exact schema_version; and
• optional module_key.

The map key produces narratio.extraction.<key>. Keys use Narratio's existing path-safe configured-artifact key grammar. Keys, normalized source IDs, and lane IDs must be non-empty and unique. Extraction source IDs must not collide with built-ins or configured Scriptorium sources.

Every configured output is required for extraction-stage success. Operators who need only a subset configure only that subset. Narratio does not add a second required_lanes list or a runtime lane-selection flag.

Narratio does not configure Notarius lane topology, references, prompts, LLM profiles, model settings, concurrency, retry behavior, or session IDs. One stage execution runs the configured Notarius pipeline as a unit. Narratio does not pass Notarius's --session-id override.

Stage Outcomes And Resume

Explicit Self-Skip

Extend the stage-result contract with an explicit disposition whose zero value remains successful for backward compatibility. A stage may return:

• succeeded; or
• skipped with a stable reason.

When Notarius is absent or disabled, extract returns skipped with reason notarius_disabled, no outputs, and bounded metadata. The runner records a skipped session-stage and run-stage outcome rather than a successful empty stage. Skipped stages are reconsidered on later invocations, so subsequently enabling Notarius causes extraction to run without requiring force.

A genuine self-skip clears any older outputs for that stage before persisting the new skipped state. Downstream consumers cannot resolve artifacts retained from an earlier extraction after the stage is disabled.

Resume Validation

Add a small optional resume-validation interface implemented by extract. Before skipping an already-succeeded extract stage, the runner asks it whether the recorded result remains resumable.

The resume validator confirms:

• Notarius is still enabled;
• recorded configuration identity matches the current config path, pipeline ID, and output-contract map;
• the immutable bundle and canonical index still exist;
• every configured source is present in the succeeded extract record;
• lane files remain confined regular files;
• stored checksums still match; and
• stored descriptor contracts and external provenance remain compatible.

An ordinary contract mismatch returns a non-resumable decision with a bounded reason. The runner marks the stage stale and executes it. An environmental error that prevents making a safe decision returns an error and stops the run.

Narratio does not recursively interpret Notarius configuration, PromptKit profiles, or reference files. Changes to those external inputs are therefore not automatically detectable. Operator documentation must require --force extract after changing them.

The configuration fingerprint covers the resolved binary, config path, pipeline ID, timeout, working directory, and the deterministically sorted output-contract map. It identifies Narratio's invocation contract, not the transitive content of files owned by Notarius.

Force And Invalidation

• Forcing trim or an earlier stage stales succeeded extract and all later stages.
• Forcing extract stales succeeded render, analyze, publish, and notify under the existing canonical-order rule.
• Forcing render does not stale extract because extraction precedes it.
• A failed, skipped, stale, or interrupted extract stage never supplies current extraction sources.

Bundle Storage And Commit

Run-Local Execution

Notarius runs against a run-local output root beneath the Narratio run's extract directory. Receipt bytes and stderr remain run-local diagnostics. Failed and malformed bundles remain outside durable artifact storage for inspection and never become current merely because files exist.

Immutable Durable Bundles

After complete validation, promote the exact Notarius bundle into:

artifacts/notarius/<narratio-run-id>/

The destination is unique and must not already exist. Promotion uses a sibling temporary directory on the same filesystem, recursively copies only regular files and directories, rejects symlinks and special files, preserves relative layout, and renames the completed temporary tree into place.

The promoted tree preserves index.json, manifest.json, rejected.json, warnings.json, lanes/, and any emitted pipeline-wide artifacts such as chunk-map.json and evidence-context.json. Unknown regular files may be preserved because the complete bundle is provenance, but no unknown file is registered as a stable Narratio source.

There is no mutable filesystem current directory or symlink. The atomically saved Narratio session manifest selects the current successful bundle. Older successful bundles remain immutable until explicit cleanup policy removes them.

The raw Notarius receipt is retained without rewriting its original output_directory. Narratio's manifest records promoted durable paths.

Artifact And Manifest Model

Explicit Source Identity

Extend artifacts.Ref with an optional explicit SourceID. The application runner prefers that value and retains existing stage-specific inference only as a backward-compatible fallback. extract must not require another stage-name special case in output mapping.

Add neutral optional artifact metadata models under internal/artifactmodel:

• contract metadata: media type, schema ID, schema version, and optional module key; and
• external provenance: system, external run ID, pipeline ID, and external artifact ID.

Both artifacts.Ref and manifest.ArtifactRecord carry these nested models. Existing manifests remain readable because the fields are optional and use omitempty encoding.

For a Notarius lane, external provenance uses:

• system notarius;
• receipt run ID;
• receipt pipeline ID; and
• lane ID as the external artifact ID.

Registered Sources

For each configured output, extract records one artifact with:

• source ID narratio.extraction.<key>;
• durable lane path discovered through the promoted canonical index;
• producer stage and Narratio run ID;
• SHA-256 checksum;
• configured and observed contract metadata; and
• Notarius external provenance.

The canonical promoted index.json is also a stage output with kind notarius_index, but it is not a selectable extraction source. Extract-stage metadata records the durable bundle root, receipt path, Notarius validation status, counts, warnings/rejections paths and summaries, the producing Narratio run ID, and the normalized configured-contract fingerprint used by resume validation.

The producing Narratio run ID belongs to the successful extract result. It is not compared with the session manifest's top-level run_id, which advances on later invocations even when extraction is validly resumed. Resume and catalog checks instead require the extract outputs, immutable bundle path, and stored extract-stage producer identity to agree with one another.

Artifact Catalog And Resolution

Add extraction as a first-class artifact-policy and runtime-catalog family:

• source kind extraction;
• canonical prefix narratio.extraction.;
• registration from pipeline.notarius.outputs; and
• manifest-backed availability from the current successful extract record.

Extraction availability is never inferred by scanning artifacts/notarius/. A source is available only when:

• it is declared in current configuration;
• the session manifest records extract as succeeded and not stale;
• the exact matching source output is present;
• its durable path is confined and valid;
• its checksum matches; and
• its recorded contract and external provenance are compatible.

The catalog should expose one shared registration and hydration path used by both analyze and publish. Avoid parallel extract-specific resolution logic inside each stage.

Analyze Integration

An enabled Scriptorium artifact may select an extraction source through the existing input contract:

inputs:
  npcs:
    source: narratio.extraction.npc_registry
    required: true

Required missing extraction inputs fail with guidance to enable/configure or rerun extract. Optional missing inputs follow the existing Scriptorium input contract.

Narratio never injects every extraction output into every analysis. Each Scriptorium artifact chooses the smallest useful set. This limits context size, cost, and the risk of treating derived claims as transcript authority. Analysis prompts should continue to treat the transcript as authoritative and Notarius artifacts as structured, cited, derived evidence.

The existing --artifacts selection remains scoped to Scriptorium artifacts. It does not select Notarius lanes or partially run the Notarius pipeline.

Publish Integration

Publish source validation and resolution accept configured narratio.extraction.<key> sources through the shared runtime artifact catalog. Operators may publish individual structured lanes without manually copying files.

The Notarius bundle is not automatically published wholesale. Bundle files or lanes are published only through explicit configured publish rules. Existing publish locking, destination safety, commit ordering, and required/unselected artifact behavior remain unchanged.

Failure And Diagnostic Semantics

The stage fails before invoking Notarius when enabled configuration or the final trimmed transcript is invalid.

The stage fails after invocation for:

• cancellation or timeout;
• nonzero process exit;
• malformed, oversized, or unsupported receipt data;
• malformed or unsupported index data;
• unsafe receipt, descriptor, or filesystem paths;
• symlinks or special files in the promoted bundle;
• mismatched receipt pipeline identity;
• missing configured lanes;
• duplicate lane descriptors;
• rejected required lanes;
• incompatible media type, schema identity/version, or module key;
• empty or syntactically invalid required JSON payloads;
• checksum or promotion failure; or
• manifest persistence failure.

Process success alone does not establish consumer success. Notarius may exit zero while omitting or rejecting a lane, and Narratio's configured required set remains authoritative.

Failure retains bounded receipt and stderr diagnostics plus the run-local bundle where available. Durable artifact storage and extraction source records are updated only after complete validation and promotion. A promoted unique bundle whose later manifest save fails is unreferenced and may be reclaimed by explicit cleanup; it is never inferred as current.

Security And Privacy

Transcripts, Notarius lanes, evidence context, manifests, receipts, warnings, rejections, debug data, and logs are private campaign material.

• Secrets do not appear in command arguments, generated configuration, artifact metadata, logs, examples, or documentation.
• Credentials continue to enter through Notarius and PromptKit's documented environment or secret mechanisms.
• Receipt and index paths are untrusted external input until confined.
• Recursive promotion never follows symlinks or copies special files.
• Diagnostics remain bounded and do not echo payload bodies.
• Automatic cleanup follows Narratio's existing post-publish gates and path safety rules; it does not silently remove immutable extraction bundles outside an explicit covered policy.

Maintained Complete D&D Example

Add a maintained example that enables Notarius's complete dnd-session pipeline and maps these required lanes:

Extraction key	Notarius lane ID
item_registry	item-registry
npc_registry	npc-registry
location_registry	location-registry
scene_descriptions	scene-descriptions
item_occurrences	item-occurrences
spells	spells
combat_turns	combat-turns
npc_occurrences	npc-occurrences
location_occurrences	location-occurrences
enemy_events	enemy-events

The example obtains exact media types and schema identities from Notarius's published contracts at implementation time. It demonstrates at least one Scriptorium artifact consuming a small, purpose-specific subset of extraction sources. It uses only placeholders and repository-relative example paths, contains no credentials, and passes maintained example validation.

Documentation Deliverables

When implementation lands, update current-behavior documentation in the same change:

• add docs/integrations/notarius.md for the consumed CLI, receipt, index, and compatibility contract, linking to Notarius's canonical documentation;
• add docs/internal/stage-extract.md for stage flow, collaborators, state, failures, resume validation, and focused tests;
• update docs/internal/overview.md, docs/internal/adapters.md, docs/internal/artifacts.md, docs/internal/manifest.md, and docs/internal/workspace.md within their canonical scopes;
• update docs/policy/architecture.md for the Notarius boundary and explicit skipped/resume-validation contracts;
• update docs/config.md, docs/cli.md, docs/operations.md, docs/troubleshooting.md, README.md, and maintained examples only within their canonical scopes; and
• update docs/development.md only if its contributor routing changes.

Outside this roadmap, do not describe the feature as implemented until its code and documentation are complete.

Testing Expectations

Tests should protect contracts and meaningful risks rather than private helper structure. The implementation plan assigns detailed ownership, with coverage for:

• strict configuration, defaults, normalization, cross-source validation, and maintained examples;
• stage result dispositions and clearing of self-skipped outputs;
• backward-compatible artifact metadata serialization;
• exact adapter arguments, streams, cancellation, timeout, exit behavior, receipt/index compatibility, and every path-confinement boundary;
• recursive promotion safety, atomic visibility, cleanup on failure, symlink rejection, and immutable destination behavior;
• required-lane policy, descriptor compatibility, JSON syntax, checksums, and provenance;
• canonical order, single-stage selection, force, staleness, resume validation, and enable-after-skip behavior;
• manifest-backed extraction catalog resolution for required, optional, missing, stale, skipped, incompatible, and tampered artifacts;
• analyze and publish integration without automatic lane injection; and
• representative assembled execution with a fake Notarius runner and no live LLM, credentials, or external subprocess in the ordinary test suite.

Repository-wide tests, vet, build, and maintained-example validation are required after focused tests pass.

Acceptance Criteria

• extract is a first-class stage between trim and render everywhere Narratio models stage order and lifecycle.
• Narratio invokes Notarius only through a narrow tested adapter.
• Extraction consumes the manifest-resolved final trimmed Seriatim JSON.
• Disabled extraction is recorded as skipped and runs normally if later enabled.
• Successful resume requires valid manifest-recorded immutable outputs rather than filesystem presence alone.
• The complete bundle is promoted to a unique immutable directory without following symlinks or exposing a partial destination.
• Every configured lane is discovered by lane ID, contract-checked, checksummed, and recorded with explicit source identity and external provenance.
• Analysis and publish resolve extraction sources only from a current successful extract manifest record.
• Failed, skipped, stale, partial, rejected, unsafe, incompatible, or tampered output never becomes a current input.
• The complete D&D example maps all ten current lanes and demonstrates curated analysis inputs.
• Tests remain deterministic, offline, and independent of real Notarius, PromptKit, LLM providers, and credentials.
• Current-behavior documentation is updated only as implementation becomes complete.

Non-Goals

• Reimplementing Notarius extraction, configuration, schemas, prompts, references, retries, profiles, validation, or lane orchestration.
• Supporting arbitrary extractor programs or multiple Notarius pipelines in one Narratio run.
• Turning canonical stage execution into a DAG or generic workflow engine.
• Folding extraction into analyze or Scriptorium.
• Automatically injecting all structured outputs into every prompt.
• Selecting Notarius lanes through Narratio's --artifacts flag.
• Decoding D&D payload bodies in the generic adapter or stage.
• Automatically detecting changes throughout Notarius's referenced config, PromptKit profile, and campaign-reference closure.
• Supporting previous-session extraction sources in the initial feature.
• Automatically publishing the complete Notarius bundle.
• Requiring live Notarius, PromptKit, an LLM provider, or external services in the ordinary test suite.