narratio/docs/roadmap/notarius-extract-stage.md

Notarius Extraction Stage

Status

Proposed.

Purpose

Add a first-class Narratio extract stage that runs Notarius against the session's final trimmed transcript, validates and collects the resulting structured D&D artifacts, and registers those artifacts for later use by the analyze and publish stages.

This feature should integrate Notarius through Narratio's existing stage, adapter, manifest, workspace, and artifact-catalog boundaries. It must not turn Narratio into a generic workflow engine or a second configuration language for Notarius pipelines.

User Outcome

An operator can enable one configured Notarius pipeline for a Narratio campaign. During a normal run, Narratio will:

1. finish producing the session transcript tiers;
2. invoke Notarius once with the final trimmed Seriatim JSON transcript;
3. collect and validate the configured structured artifact lanes;
4. record their exact files and provenance in the Narratio manifest; and
5. make those artifacts selectable as inputs to Scriptorium artifacts in the later analyze stage.

The maintained D&D example should demonstrate all ten lanes emitted by Notarius's complete dnd-session pipeline.

Target Stage Architecture

Canonical Order

The canonical stage order becomes:

prepare -> transcribe -> merge -> polish -> normalize -> trim -> render
        -> extract -> analyze -> publish -> notify

extract is deliberately after all transcript-producing stages and before analysis. Its source document is the manifest-resolved narratio.transcript.final_trimmed artifact, normally transcripts/final.trimmed.json. It does not consume rendered Markdown.

Adding the stage must update full-plan construction, explicit stage selection, downstream invalidation, prerequisite checks, resume behavior, run manifests, CLI stage validation and help, and every canonical-stage inventory. Forcing an upstream transcript stage must stale a previously successful extract stage and its downstream stages. Forcing extract must stale analyze, publish, and notify according to existing rules.

Stage Boundary

The stage owns Narratio policy and state transitions:

• resolve the final trimmed transcript through the runtime artifact catalog;
• build a Narratio-level Notarius request from validated configuration and run-local paths;
• call a narrow Notarius adapter;
• apply the configured required-output policy;
• materialize the validated bundle into its canonical session location;
• return manifest-ready artifact references and bounded metadata; and
• fail without marking the stage successful when any required contract or materialization step fails.

The stage must not construct subprocess arguments, infer Notarius output filenames, parse provider logs, or decode individual D&D payload bodies.

Adapter Boundary

Add a dedicated Notarius adapter package with a small interface, production subprocess implementation, and test fake. Its request should contain only the resolved Notarius binary, configuration path, pipeline ID, transcript path, output root, working directory, timeout, and process-log destinations needed for one run.

The adapter owns:

• optional notarius config validate preflight for the configured pipeline;
• exact notarius run ... --json argument construction;
• stdout and stderr separation;
• context cancellation and timeout propagation through Narratio's shared subprocess boundary;
• exit-status handling;
• decoding the notarius.run-result.v1 success receipt;
• receipt and index path-confinement checks;
• decoding index.json and resolving descriptor paths safely beneath the reported output directory; and
• returning a transport-neutral result containing the bundle location, receipt summary, lane descriptors, pipeline-wide descriptors, warnings and rejection locations, and diagnostic log paths.

Only exit status zero permits receipt decoding. Receipt, index, or descriptor paths that are absolute where a logical relative path is required, or that escape their owning root, are integration failures. Unknown fields in a supported receipt or index schema should be tolerated. Unsupported schema versions and incompatible descriptor metadata should fail clearly.

The adapter must not write Narratio manifests, choose required lanes, decide analysis inputs, or contain D&D domain logic.

Configuration Contract

Add a strict optional pipeline.notarius configuration section. Omission or enabled: false keeps the current workflow usable and causes extract to self-skip without outputs.

The section should provide:

• enabled: explicit opt-in;
• binary: Notarius executable, defaulting to notarius;
• config_path: required when enabled;
• pipeline_id: required when enabled;
• timeout: a positive stage timeout with a documented default;
• working_directory: optional explicit subprocess working directory, defaulting to the directory containing config_path; and
• an outputs map defining the Notarius lane artifacts Narratio promises to collect.

Each output-map key is a stable Narratio extraction key. Each value must define:

• the exact Notarius lane_id;
• the expected media_type;
• the expected schema_id;
• the expected schema_version; and
• optionally an expected module_key when the operator needs to constrain the producing module as part of compatibility.

Narratio derives the downstream source ID narratio.extraction.<output-key> from the map key. Keys and lane IDs must be non-empty, unique after normalization, path-safe under the existing artifact policy, and collision-free with built-in and configured artifact identities. Every configured output is required: a successful Notarius process that omits one, rejects it, or reports incompatible descriptor metadata fails the extract stage.

This explicit map keeps Narratio's consumer contract stable when a Notarius lane ID or schema changes and avoids hard-coding the current D&D family into a generic adapter. It also replaces a separate required_lanes list, which would duplicate configuration.

Narratio should not reproduce Notarius lane selection, references, LLM profiles, model settings, retries, concurrency, or prompt configuration. Those remain in the referenced Notarius configuration. Narratio should not expose a runtime lane-selection flag for extract; one stage invocation runs the configured Notarius pipeline as a unit.

All configured paths should become absolute during Narratio configuration resolution. The deterministic default working directory allows a Notarius profile path relative to that directory, but operator documentation should still recommend absolute deployment paths where practical. Notarius reference paths continue to follow Notarius's own configuration-relative rules.

Output And Artifact Model

Canonical Bundle

Run Notarius against a run-local output root. After all configured descriptors are validated, materialize the contents of the exact run-specific Notarius bundle into a fixed canonical session directory:

artifacts/notarius/

Preserve its relative layout, including index.json, manifest.json, rejected.json, warnings.json, lanes/, and any indexed chunk-map.json or evidence-context.json. Materialize the complete directory as one narrow, transactional replacement so a failed or interrupted rerun cannot mix files from different Notarius runs.

The raw subprocess receipt and stderr log belong in the run-local extract report and log directories. The raw receipt identifies the original run-local Notarius bundle and must not be rewritten to pretend that the canonical copy was its original output_directory. Narratio's manifest is the durable ledger for the canonical materialized paths.

Registered Artifact Sources

For each configured output, locate the lane through the canonical copy of index.json and record a manifest artifact with:

• source ID narratio.extraction.<output-key>;
• canonical lane-file path discovered from the index;
• producer stage and Narratio run ID;
• checksum;
• Notarius lane ID; and
• descriptor media type, schema identity/version, and module key when present.

If the current manifest model cannot carry descriptor compatibility metadata, extend its artifact metadata in a backward-tolerant way rather than encoding that information in filenames or source IDs.

Also record the canonical Notarius index as a stage output or stage metadata so operators can discover the complete bundle, including non-lane artifacts. The configured lane sources are the stable interface for analysis; the index and bundle remain the provenance and inspection interface.

Analysis And Publish Integration

Extend the runtime artifact catalog and configured Scriptorium input validation so an enabled analysis artifact can declare, for example:

inputs:
  npc_registry:
    source: narratio.extraction.npc_registry

Resolution must remain manifest-first and verify that the recorded artifact was produced by a successful current extract stage. A required extraction source that is unavailable must fail analysis with guidance to configure or rerun extract; an optional source may be omitted according to the existing Scriptorium input contract.

Publish source resolution should accept configured narratio.extraction.<output-key> sources through the same artifact catalog so operators may publish selected structured artifacts without manually copying paths. The existing --artifacts flag remains scoped to Scriptorium artifact selection and must not partially execute the Notarius pipeline.

No current-session analysis artifact should consume an incidental file from a failed, stale, skipped, or superseded extraction run.

Failure, Skip, Resume, And Diagnostics

• Missing or invalid enabled Notarius configuration fails configuration validation before stage execution where statically discoverable.
• A disabled or absent Notarius configuration makes extract skip with clear stage metadata and no new outputs.
• A missing or invalid final trimmed transcript fails extract before starting Notarius.
• Preflight failure, nonzero Notarius exit, cancellation, timeout, malformed or unsupported receipt/index data, unsafe paths, incompatible descriptors, rejected required outputs, or missing configured lanes fails the entire stage.
• Process success does not override Narratio's required-output policy.
• A failed run retains bounded run-local receipt bytes, stderr, and the unpublished Notarius bundle for diagnosis, subject to Narratio's existing sensitive-data and cleanup policies.
• The canonical bundle and manifest artifacts are updated only after complete validation and materialization.
• Resume skips a succeeded, non-stale extract stage only when its manifest-recorded canonical index and configured lane outputs still validate.
• Force and staleness behavior follows the ordinary stage contract; it must not depend on merely finding artifacts/notarius/ on disk.

Transcripts, Notarius outputs, evidence context, manifests, receipts, and logs are private campaign material. Subprocess arguments and manifest metadata must not contain secrets. Credentials remain in the environment or in mechanisms owned by Notarius and PromptKit.

Maintained D&D Example

Add or update a Narratio example that enables Notarius's complete dnd-session pipeline and maps these ten required lanes to stable extraction keys:

Output key	Notarius lane ID
item_registry	item-registry
npc_registry	npc-registry
location_registry	location-registry
scene_descriptions	scene-descriptions
item_occurrences	item-occurrences
spells	spells
combat_turns	combat-turns
npc_occurrences	npc-occurrences
location_occurrences	location-occurrences
enemy_events	enemy-events

The example must include each lane's current media type and schema identity from Notarius's published contracts. It should also demonstrate at least one Scriptorium analysis artifact consuming one or more narratio.extraction.* sources. The example must use placeholders and relative paths suitable for the example tree, contain no credentials, and pass the repository's configuration validation tests.

Compatibility Policy

The initial integration baseline is the public subprocess contract available in Notarius v0.3.0:

• successful JSON receipt schema notarius.run-result.v1;
• production JSON bundle discovery through index.json; and
• the schema IDs and versions explicitly configured for required lanes.

Runtime compatibility should be decided from those published contracts, not from textual parsing of notarius --version. New optional receipt or index fields must not break Narratio. An unsupported receipt version or lane schema must fail before the artifact is registered for analysis.

Documentation Deliverables When Implemented

Update current-behavior documentation in the same change that implements the feature:

• add docs/integrations/notarius.md for the external CLI, receipt, bundle, and adapter contract, linking to Notarius's canonical documentation;
• add docs/internal/stage-extract.md for stage inputs, outputs, collaborators, state transitions, failures, and focused tests;
• update docs/internal/adapters.md, docs/internal/artifacts.md, docs/internal/manifest.md, and the internal stage inventory;
• update docs/policy/architecture.md to list Notarius among isolated external systems and preserve the adapter/stage boundary;
• update docs/config.md, docs/cli.md, docs/operations.md, docs/troubleshooting.md, README.md, and maintained examples only to the extent their canonical scopes require; and
• update docs/development.md only to the extent its canonical contributor routing scope requires.

Outside this roadmap, do not describe extract, Notarius configuration, or narratio.extraction.* sources as implemented until the code exists.

Testing And Validation Expectations

Implementation should provide focused tests for:

• strict configuration decoding, defaults, required fields, path resolution, output-map validation, normalized-key collisions, and example loading;
• exact stage order, selection, downstream staleness, resume, force, and prerequisite behavior;
• adapter command construction, deterministic working directory, environment inheritance, stdout/stderr separation, cancellation, timeout, and nonzero exits;
• supported and unsupported receipt versions, unknown optional fields, malformed receipts, index decoding, and path escapes at every boundary;
• descriptor lookup by lane ID rather than filename, expected metadata checks, missing/rejected configured lanes, and tolerated unconfigured lanes;
• run-local execution, transactional canonical-bundle replacement, checksums, failed-run preservation, and manifest recording;
• artifact-catalog resolution from narratio.extraction.* into analysis and publish, including required, optional, missing, stale, and skipped cases; and
• end-to-end stage execution with a fake Notarius adapter, without live LLM or external subprocess requirements in the ordinary test suite.

Run the repository-wide Go tests, vet, build, and maintained example validation after focused tests pass.

Acceptance Criteria

• extract is a first-class transactional stage between render and analyze everywhere Narratio models stage order or state.
• Narratio invokes Notarius only through a narrow, tested adapter.
• The stage consumes the manifest-resolved final trimmed Seriatim transcript.
• The Notarius configuration remains owned by Notarius; Narratio configures only invocation and its downstream consumer contract.
• Every configured output is discovered through the receipt and index.json, contract-checked, materialized transactionally, and recorded with a stable narratio.extraction.* source ID.
• The complete D&D example maps all ten current lanes and passes strict config validation.
• Analysis can consume extraction sources through the existing artifact input model, and publish can select them through the artifact catalog.
• Failed, partial, rejected, unsafe, stale, or incompatible output never becomes a current analysis input.
• Resume and force behavior remains manifest-driven.
• Documentation accurately describes the implemented stage, adapter, configuration, operations, and compatibility boundary without duplicating Notarius's canonical schemas.

Non-Goals

• Reimplementing Notarius extraction, prompts, schemas, references, retries, profiles, or lane orchestration in Narratio.
• Allowing one Narratio run to invoke arbitrary extractor programs or multiple Notarius pipelines.
• Making extract a configurable DAG or folding it into the Scriptorium analyze stage.
• Partially selecting Notarius lanes through Narratio's --artifacts flag.
• Decoding D&D payload bodies in the generic Notarius adapter.
• Supporting previous-session extraction artifacts in the initial feature.
• Requiring live Notarius, PromptKit, an LLM provider, or external services in the ordinary unit test suite.