narratio/docs/roadmap/implementation.md

# Notarius Extraction Implementation Plan

## Status And Audience

Proposed and not started.

This plan is written for a GPT-5.6 Terra coding agent. Implement the stages in
strict numerical order. Do not skip ahead, combine stages merely to reduce the
number of prompts, or describe unfinished behavior as current functionality.

The target state and acceptance criteria are owned by
[notarius-extract-stage.md](notarius-extract-stage.md). This document owns the
implementation sequence.

## Progress

| Stage | Status |
| --- | --- |
| Stage 1 | Complete |
| Stage 2 | Complete |
| Stage 3 | Complete |
| Stage 4 | Not started |
| Stage 5 | Not started |
| Stage 6 | Not started |
| Stage 7 | Not started |
| Stage 8 | Not started |
| Stage 9 | Not started |

After completing and validating a stage, update only that stage's row to
`Complete` and record any material deviation in the relevant stage section.
Do not mark a stage complete while required tests or exit criteria remain.

## Working Rules

Before Stage 1, read:

- `docs/development.md` and its task-specific references;
- `docs/roadmap/notarius-extract-stage.md` completely;
- `../notarius/docs/consumers/dnd-pipeline.md` and its linked subprocess,
  receipt, JSON-output, and lane-contract documentation; and
- the focused Narratio documents and tests named by the current stage.

For every stage:

1. Inspect the current implementation before editing. Prefer the codebase
   knowledge graph for code discovery as required by `AGENTS.md`.
2. Preserve unrelated worktree changes.
3. Implement the complete stage scope, including focused tests. Do not leave
   TODO implementations, compatibility shims without owners, or knowingly
   dead code for a later stage.
4. Use real internal collaborators in tests where fast and deterministic; fake
   only subprocess, remote, clock, randomness, or other external boundaries.
5. Keep tests offline and independent of real Notarius, PromptKit, LLM
   providers, credentials, and mutable services.
6. Run the focused package tests listed for the stage. Fix failures before
   proceeding.
7. Run `go test ./...` after every stage. Also run `go vet ./...` and
   `go build ./cmd/narratio` when the stage changes shared contracts,
   composition, CLI behavior, or documentation examples.
8. Keep proposed behavior in `docs/roadmap/` until Stage 9. Intermediate code
   is development work and must not cause current-behavior documentation to
   claim that the end-to-end feature is complete.

Intermediate stages must compile and pass the repository test suite. The work
is not release-ready until Stage 9 is complete.

## Stage 1: Establish Shared Stage-Outcome And Artifact-Provenance Contracts

Introduce the reusable internal contracts needed by extraction without adding
Notarius configuration or a registered stage.

Implementation:

1. Under `internal/artifactmodel`, add optional neutral models for:
   - contract metadata: `media_type`, `schema_id`, `schema_version`, and
     optional `module_key`;
   - external provenance: `system`, `run_id`, `pipeline_id`, and
     `artifact_id`.
2. Extend `artifacts.Ref` with:
   - optional explicit `SourceID`;
   - optional contract metadata; and
   - optional external provenance.
3. Extend `manifest.ArtifactRecord` with the same nested optional metadata.
   Use backward-compatible `omitempty` JSON fields. Old manifests must load
   and round-trip without fabricated metadata.
4. Refactor `internal/app.mapResultOutputs` to prefer `Ref.SourceID` when set,
   copy both metadata structures, and retain existing output-kind and analyze
   inference only as fallback behavior.
5. Add an explicit `StageResult` disposition with succeeded and skipped
   values. Preserve zero-value success so existing stages do not need
   mechanical edits.
6. Add a stable skip-reason field. Reject or fail safely if a skipped result
   contains outputs.
7. Teach `executeStages` to persist a self-skipped stage as skipped in both
   session and run manifests, apply bounded result metadata and diagnostics,
   clear any older session-stage outputs, and continue to later stages.
   Distinguish this from the runner's existing decision to skip an already
   succeeded stage.
8. Keep successful and failed behavior unchanged for existing stages.

Tests:

- `internal/artifactmodel`: JSON behavior for complete and omitted metadata.
- `internal/manifest`: old manifest compatibility and self-skip clearing of
  earlier outputs.
- `internal/app`: explicit source ID precedence, fallback inference,
  provenance copying, self-skip persistence, continuation after self-skip,
  and rejection of skipped results containing outputs.

Exit criteria:

- Existing stage outputs and manifests remain compatible.
- A synthetic stage can self-skip without being recorded as succeeded.
- A later invocation will reconsider a self-skipped stage.
- `go test ./...`, `go vet ./...`, and `go build ./cmd/narratio` pass.

## Stage 2: Add Notarius Configuration And Extraction Source Policy

Add strict configuration and stable source identity, but do not invoke
Notarius or register `extract` yet.

Implementation:

1. Add an optional `Notarius *NotariusConfig` field to `PipelineConfig`.
2. Define:
   - `enabled` with false default;
   - `binary` with `notarius` default;
   - `config_path`;
   - `pipeline_id`;
   - `timeout` with `3h` default;
   - optional `working_directory`; and
   - `outputs map[string]NotariusOutputConfig`.
3. Define each output with required `lane_id`, `media_type`, `schema_id`, and
   `schema_version`, plus optional `module_key`.
4. Apply defaults centrally. When enabled, resolve `config_path` to an
   absolute path using the same configuration-source semantics as comparable
   adapter paths. Default `working_directory` to the resolved config file's
   directory, and resolve an explicitly configured working directory to an
   absolute path.
5. Validate enabled configuration before stage execution:
   - non-empty binary, config path, pipeline ID, and output map;
   - positive duration;
   - non-empty output contract fields;
   - output keys accepted by the existing configured-artifact key policy;
   - unique normalized extraction source IDs;
   - unique lane IDs;
   - no collision with built-in, configured Scriptorium, or previous-session
     source identities.
6. Add `SourceKindExtraction` and helpers for the exact
   `narratio.extraction.<key>` family in `internal/artifactpolicy` and
   `internal/artifacts`. Do not accept arbitrary unconfigured extraction
   sources at configuration validation boundaries.
7. Extend Scriptorium input and publish source validation to accept only
   extraction keys declared by the same effective pipeline configuration.
8. Do not expose a Notarius session-ID setting, lane-selection flag, prompt,
   profile, model, retry, or reference configuration.

Tests:

- strict YAML decoding and unknown-field rejection;
- omitted and disabled behavior;
- defaults and path resolution from pipeline-file location;
- enabled required fields and positive timeout;
- invalid/safe keys, normalized collisions, duplicate lanes, incomplete
  contracts, and source-family collisions;
- valid and unknown extraction sources in Scriptorium inputs and publish
  rules; and
- existing configurations and maintained examples continue to load.

Exit criteria:

- A complete Notarius configuration loads and normalizes deterministically.
- Invalid downstream extraction references fail during config validation.
- No runtime code invokes Notarius yet.
- `go test ./internal/config ./internal/artifactpolicy ./internal/artifacts`
  and `go test ./...` pass.

## Stage 3: Add Safe Immutable Directory Promotion

Implement the filesystem primitive used to promote a validated external bundle
without exposing a partial or mixed durable directory.

Implementation:

1. Add a narrowly named directory-promotion operation to `internal/fileops`.
   It accepts an existing source directory and a new destination directory.
2. Require a non-empty source and destination. The destination must not exist.
3. Create a temporary sibling of the destination so final installation uses a
   same-filesystem rename.
4. Recursively walk with `Lstat` semantics:
   - permit directories and regular files only;
   - reject symlinks, devices, sockets, named pipes, and other special files;
   - never follow a symlink;
   - use deterministic lexical traversal;
   - preserve relative layout; and
   - use bounded safe permissions rather than copying unsafe mode bits.
5. Copy and sync each regular file, sync the completed temporary tree where
   practical, and rename the temporary directory to the unique destination.
6. On any failure, remove only the operation's validated temporary sibling.
   Never remove or overwrite the source, destination, workspace root, or a
   broad caller-provided directory.
7. Return an error if a concurrent actor creates the destination before final
   installation.

Tests using `t.TempDir()`:

- nested regular-tree success and byte-for-byte preservation;
- deterministic layout;
- destination already exists;
- source is not a directory;
- symlink to a file, symlink to a directory, and escaping symlink rejection;
- representative special-file rejection where portable;
- copy/install failure cleanup with no partial destination; and
- source preservation after success and failure.

Exit criteria:

- A promoted directory appears only as a complete unique tree.
- The operation cannot follow symlinks or replace an existing destination.
- `go test ./internal/fileops` and `go test ./...` pass.

## Stage 4: Implement The Notarius Adapter

Create the external subprocess boundary independently of stage policy.

Implementation:

1. Add `internal/adapters/notarius` with:
   - a `Runner` interface;
   - transport-neutral request and result models;
   - receipt, index, lane-descriptor, and pipeline-wide-descriptor models;
   - a production subprocess runner; and
   - a small configurable fake for stage tests.
2. Build exactly:

   ```text
   notarius run <pipeline-id> --config <path> --input <path>
       --output-dir <path> --json
   ```

   Use separate argument elements, absolute supplied paths, the configured
   working directory, inherited environment, and Narratio's shared subprocess
   timeout/cancellation behavior. Do not pass `--session-id`.
3. Write stdout directly to the configured receipt path and stderr to the
   configured log path. Do not combine streams.
4. On nonzero exit, cancellation, or timeout, return the subprocess error and
   do not parse stdout.
5. After exit zero, read receipt bytes with an explicit upper bound, decode
   `notarius.run-result.v1`, tolerate unknown fields, and validate required
   fields and matching pipeline ID.
6. Require an absolute `output_directory` and confine it beneath the absolute
   output root from the request. Confine the relative `index_file` beneath the
   returned directory using Narratio's root-scoped path helpers. Reject
   absolute logical paths, traversal, symlinks at consumed paths, and
   non-regular files.
7. Decode the supported production JSON index tolerantly. Validate required
   management paths and lane descriptors, reject duplicate lane IDs, and
   confine every lane and pipeline-wide descriptor file beneath the bundle
   root.
8. Load `rejected.json` and `warnings.json` with explicit size bounds and
   tolerant decoding. Return only their generic summaries, including lane ID
   and reason code where present; do not read lane payload bodies or interpret
   D&D fields. Keep free-form external messages bounded for immediate errors
   and out of manifest metadata.
9. Return descriptors and paths without deciding which lanes are required or
   whether their configured contracts match.
10. Do not implement automatic `notarius config validate` execution.

Tests:

- exact argument order and omission of `--session-id`;
- working directory, environment inheritance, stream separation, cancellation,
  timeout, and nonzero exit;
- stdout ignored after failure;
- receipt size limit, malformed JSON, unsupported version, missing fields,
  pipeline mismatch, and tolerated unknown fields;
- output-directory escape from the requested output root;
- index malformed/unsupported shapes, duplicate lanes, missing management
  paths, and tolerated optional fields;
- malformed or oversized rejection/warning documents and tolerated unknown
  summary fields;
- absolute logical paths, lexical traversal, root-prefix confusion, symlink
  paths, and descriptor escape attempts; and
- valid lane plus chunk-map/evidence-context discovery.

Use a test helper subprocess or injected shared runner. Do not require a real
Notarius binary.

Exit criteria:

- The adapter fully enforces the generic Notarius subprocess and discovery
  boundary without D&D policy.
- `go test ./internal/adapters/notarius ./internal/adapters/subprocess` and
  `go test ./...` pass.

## Stage 5: Implement Extract-Stage Execution And Bundle Materialization

Implement the stage behind direct package tests, but do not yet add it to the
canonical `stage.All()` registry.

Implementation:

1. Add `Notarius notarius.Runner` to `stage.Env` and default production
   composition in `internal/app/runner.go` when enabled extraction is selected.
2. Add canonical path helpers for:
   - the run-local extract directory;
   - receipt and stderr paths;
   - the run-local Notarius output root; and
   - immutable `artifacts/notarius/<narratio-run-id>` destinations.
3. Implement `extractStage` with name `extract` and declarations matching its
   final-trimmed input and structured outputs.
4. When configuration is absent or disabled, return the explicit skipped
   result with `notarius_disabled` and no outputs.
5. Resolve `narratio.transcript.final_trimmed` through the manifest-aware
   resolver. Do not fall back to guessed filenames outside that resolver.
6. Invoke the adapter once with resolved absolute paths.
7. Apply required-output policy:
   - find each configured output by exact lane ID;
   - require exactly one descriptor;
   - compare media type, schema ID/version, and optional module key;
   - treat rejected or absent configured lanes as failure;
   - tolerate unconfigured lanes;
   - require each selected file to be regular, non-empty, and syntactically
     valid JSON; and
   - compute a staging checksum for later copy verification.
8. Validate the complete bundle tree for promotion, then use Stage 3's helper
   to copy it to the unique immutable destination.
9. Re-resolve the promoted index and configured descriptor paths rather than
   retaining staging paths. Compute authoritative SHA-256 checksums from the
   promoted files and fail if promoted bytes differ from the validated source.
10. Return one `artifacts.Ref` per configured lane with explicit extraction
    source ID, checksum, contract metadata, and Notarius external provenance.
11. Return the promoted `index.json` as a `notarius_index` output without a
    selectable source ID.
12. Return bounded stage metadata containing durable bundle root, receipt and
    diagnostic paths, receipt counts/status, rejection/warning summaries, the
    producing Narratio run ID, and a deterministic fingerprint of the resolved
    binary, config path, pipeline ID, timeout, working directory, and sorted
    output-contract map.
13. Do not delete run-local diagnostics or failed bundles.

Tests:

- disabled self-skip;
- missing/invalid final-trimmed input before adapter invocation;
- exact adapter request construction;
- required, missing, rejected, duplicate, incompatible, and unconfigured
  lanes;
- non-empty and JSON-syntax validation without D&D decoding;
- immutable promotion and promoted-path re-resolution;
- checksums, explicit source IDs, nested metadata, and external provenance;
- adapter and promotion failures do not return successful outputs; and
- complete bundle files are preserved while unknown files are not registered
  as sources.

Exit criteria:

- Direct extract-stage execution produces a complete manifest-ready result
  using only a fake Notarius runner.
- The stage is not yet in normal CLI plans.
- `go test ./internal/stage ./internal/artifacts ./internal/fileops` and
  `go test ./...` pass.

## Stage 6: Integrate Canonical Ordering, CLI Lifecycle, Force, And Resume

Make `extract` a real pipeline stage and complete its orchestration semantics.

Implementation:

1. Register `extractStage` in `stage.All()` immediately after `trim` and before
   `render`.
2. Update all hard-coded or user-visible stage inventories, stage-name
   validation, CLI help, planning, prerequisite checks, and test fixtures.
3. Confirm full and single-stage commands can select `extract`. Do not add a
   lane-selection flag and do not broaden `--artifacts` beyond Scriptorium
   artifact selection.
4. Add an optional stage resume-validation contract. Use a result containing:
   - resumable boolean; and
   - bounded non-resumable reason.
   Reserve an ordinary error for cases where a safe decision cannot be made.
5. Before skipping a succeeded stage, `executeStages` invokes the optional
   validator. A non-resumable result marks the session stage stale with its
   reason and schedules it to run. A validation error stops execution without
   mutating the succeeded record.
6. Implement extraction resume validation against:
   - enabled state;
   - stored invocation/config-contract fingerprint;
   - succeeded extract record;
   - immutable bundle and index paths;
   - exact configured source set;
   - regular confined lane files;
   - checksums;
   - contracts; and
   - Notarius external provenance.
   Require the producing Narratio run ID recorded in extract metadata, output
   records, and the immutable bundle path to agree. Do not compare it with the
   session manifest's top-level `run_id`, which identifies the latest
   invocation and may legitimately differ after resume.
7. Do not inspect or hash Notarius's transitive reference/profile closure.
8. Verify canonical downstream invalidation:
   - force through `trim` stales extract and later succeeded stages;
   - force `extract` stales render and later succeeded stages;
   - force `render` does not stale extract.
9. Verify existing session manifests without an extract record normalize and
   execute the new stage rather than failing migration.

Tests:

- exact canonical order and CLI stage inventory;
- full-plan and `run-stage extract` behavior;
- disabled skip followed by later enablement;
- successful resume skip;
- missing, tampered, incompatible, or config-mismatched outputs trigger stale
  and rerun;
- unsafe validation errors stop without corrupting prior success;
- force and downstream invalidation in each direction named above; and
- old manifest compatibility.

Exit criteria:

- Normal execution includes `extract` in the agreed position.
- Disabled and resume behavior cannot make obsolete outputs current.
- `go test ./internal/stage ./internal/app ./internal/manifest`,
  `go test ./...`, `go vet ./...`, and `go build ./cmd/narratio` pass.

## Stage 7: Integrate Extraction Sources With The Runtime Catalog And Analyze

Make succeeded extraction lanes selectable by configured Scriptorium
artifacts.

Implementation:

1. Extend `ArtifactCatalog` with extraction definitions registered from the
   effective Notarius output map.
2. Add one shared manifest-backed hydration operation for extraction sources.
   It accepts session paths, manifest, and configured extraction definitions
   and marks entries available only after verifying the roadmap's full
   successful-stage, source, path, checksum, contract, and provenance rules.
3. Add a distinct extraction provenance value such as
   `manifest.current_extract_run`.
4. Extend `ResolveSessionArtifactWithCatalog` to resolve configured extraction
   sources through catalog availability. It must never scan the Notarius
   artifact directory or accept an incidental file.
5. Refactor catalog construction enough that extraction registration and
   hydration are reusable by analyze and publish. Do not create a generic
   workflow abstraction or move Scriptorium execution policy into artifacts.
6. Update analyze catalog creation to include effective extraction
   definitions and current manifest state before planning Scriptorium inputs.
7. Preserve required/optional input behavior. Required unavailable extraction
   inputs fail with actionable extract configuration/rerun guidance; optional
   inputs are omitted normally.
8. Do not automatically add extraction inputs to any configured artifact.

Tests:

- registration, lookup, deterministic ordering, and source-family collision;
- successful current-manifest hydration;
- missing stage, skipped, failed, stale, interrupted, missing-source,
  internally inconsistent producer identity, incompatible-contract,
  missing-file, tampered-checksum, and unsafe-path rejection;
- required and optional analyze input behavior;
- selected Scriptorium artifact execution receives only explicitly configured
  extraction inputs; and
- no object-store calls or directory scanning occur during analyze
  extraction resolution.

Exit criteria:

- A fake successful extract run can feed selected JSON files into a fake
  Scriptorium analysis run.
- No failed or incidental bundle can become available through the catalog.
- `go test ./internal/artifacts ./internal/stage ./internal/config` and
  `go test ./...` pass.

## Stage 8: Integrate Extraction Sources With Publish And Operator Inspection

Complete downstream artifact handling without automatically publishing the
bundle.

Implementation:

1. Use Stage 7's shared extraction catalog registration and hydration in
   publish planning and source resolution.
2. Permit explicit publish rules whose source is a configured extraction
   source. Preserve existing destination policy, lock behavior, required
   output policy, selected-Scriptorium behavior, and remote commit order.
3. Do not publish the complete Notarius bundle unless individual explicit
   rules name selectable sources. `notarius_index` remains non-selectable in
   the initial feature.
4. Ensure the published session manifest retains extraction artifact contract
   and external provenance metadata through ordinary serialization.
5. Extend existing artifact/status/inspection output only where necessary so
   configured extraction sources report planned, available, unavailable, and
   published states consistently. Do not expose payload contents.
6. Confirm restore safely round-trips explicitly published extraction files
   through existing manifest and confined-path behavior; add code only if a
   real incompatibility is found.

Tests:

- successful explicit lane publication;
- required missing or invalid extraction source failure;
- optional/unselected behavior remains consistent;
- publish lock and commit-marker order remain unchanged;
- `--artifacts` does not partially select Notarius lanes;
- manifest metadata survives publish serialization and restore loading; and
- operator inspection never reads or prints extraction payload bodies.

Exit criteria:

- Explicit extraction publish rules work through the shared catalog.
- No implicit whole-bundle publication is introduced.
- `go test ./internal/stage ./internal/app ./internal/artifacts` and
  `go test ./...` pass.

## Stage 9: Add Maintained Examples, Current Documentation, And Final Validation

Finish the public and maintainer contract only after the implementation is
complete.

Implementation:

1. Read Notarius's current published lane contracts and add exact media type,
   schema ID, schema version, and module-key constraints for all ten lanes to
   a maintained Narratio complete example. Do not guess or copy stale roadmap
   placeholders.
2. Add a small maintained Scriptorium example that consumes a purpose-specific
   subset of extraction sources rather than all ten.
3. Add `docs/integrations/notarius.md` as Narratio's external-consumer
   contract. Link to Notarius's canonical documents instead of duplicating
   complete schemas.
4. Add `docs/internal/stage-extract.md` for implemented stage mechanics,
   lifecycle, resume validation, failure behavior, and focused tests.
5. Update the canonical owners listed in the roadmap's Documentation
   Deliverables section. Keep configuration fields/defaults only in
   `docs/config.md`, commands only in `docs/cli.md`, physical layout and force
   procedures only in `docs/operations.md`, and implementation mechanics only
   in internal documents.
6. Update every implemented stage inventory to show `extract` between `trim`
   and `render`.
7. Add troubleshooting guidance for missing Notarius, nonzero exit,
   receipt/index incompatibility, required-lane rejection, resume
   invalidation, and the requirement to force extraction after changing
   Notarius's transitive configuration inputs.
8. Review the roadmap status. If every acceptance criterion is implemented,
   mark it complete or move completed planning material according to repository
   convention without deleting useful decision context prematurely.
9. Check all changed links, examples, commands, field names, defaults, schemas,
   and paths against the implementation.

Validation:

- focused example/configuration validation tests;
- `go test ./...`;
- `go vet ./...`;
- `go build ./cmd/narratio`;
- inspect `git diff --check`;
- confirm ordinary tests use no live Notarius, PromptKit, LLM provider,
  credentials, network service, or mutable external state; and
- confirm no secrets or private campaign content were added.

Exit criteria:

- Every feature-roadmap acceptance criterion is satisfied.
- Maintained examples load under strict configuration validation.
- Current-behavior documentation accurately describes the shipped feature
  without duplicating Notarius-owned schema definitions.
- The repository-wide validation suite passes.

## Open Questions

None. The target architecture, defaults, lifecycle behavior, compatibility
boundary, storage layout, source identity, downstream selection model, and
implementation order are decision-complete for implementation. If repository
inspection exposes a contradiction with an implemented invariant, stop at the
affected stage and document the concrete conflict rather than silently
changing this plan's target behavior.