narratio/docs/roadmap/implementation.md

# Notarius Extraction Implementation Plan

## Status And Audience

Remediation planned. Stages 1 through 9 are complete; Stages 10 through 14
address gaps found during the post-implementation architecture review.

This plan is written for a GPT-5.6 Terra coding agent. Implement incomplete
stages in strict numerical order. Do not skip ahead, combine stages merely to
reduce the number of prompts, or describe unfinished behavior as current
functionality.

The target state and acceptance criteria are owned by
[notarius-extract-stage.md](notarius-extract-stage.md). This document owns the
implementation sequence.

## Progress

| Stage | Status |
| --- | --- |
| Stage 1 | Complete |
| Stage 2 | Complete |
| Stage 3 | Complete |
| Stage 4 | Complete |
| Stage 5 | Complete |
| Stage 6 | Complete |
| Stage 7 | Complete |
| Stage 8 | Complete |
| Stage 9 | Complete |
| Stage 10 | Complete |
| Stage 11 | Complete |
| Stage 12 | Complete |
| Stage 13 | Pending |
| Stage 14 | Pending |

After completing and validating a stage, update only that stage's row to
`Complete` and record any material deviation in the relevant stage section.
Do not mark a stage complete while required tests or exit criteria remain.

## Working Rules

Before Stage 10, read:

- `docs/development.md` and its task-specific references;
- `docs/roadmap/notarius-extract-stage.md` completely;
- `../notarius/docs/consumers/dnd-pipeline.md` and its linked subprocess,
  receipt, JSON-output, and lane-contract documentation; and
- the focused Narratio documents and tests named by the current stage.

For every incomplete stage:

1. Inspect the current implementation before editing. Prefer the codebase
   knowledge graph for code discovery as required by `AGENTS.md`.
2. Preserve unrelated worktree changes.
3. Implement the complete stage scope, including focused regression tests. Do
   not leave TODO implementations or defer a known part of the stage.
4. Keep tests offline and independent of real Notarius, PromptKit, LLM
   providers, credentials, network services, and mutable external state.
5. Run the focused package tests listed for the stage, then `go test ./...`.
6. Run `go vet ./...` and `go build ./cmd/narratio` whenever shared contracts,
   composition, CLI behavior, portability, or documentation examples change.
7. Do not broaden the feature beyond the target architecture while fixing a
   remediation. Prefer a general runner or filesystem invariant when the defect
   is general, but avoid unrelated cleanup.

## Completed Stages 1-9 Summary

1. **Stage 1 — Shared contracts:** Added explicit stage self-skip semantics,
   artifact source identity, contract metadata, external provenance, and
   backward-compatible manifest serialization.
2. **Stage 2 — Configuration and source policy:** Added strict optional
   Notarius configuration, deterministic defaults/path resolution, output-lane
   contracts, and the `narratio.extraction.<key>` source family.
3. **Stage 3 — Immutable promotion:** Added regular-file-only bundle copying,
   symlink and special-file rejection, bounded permissions, syncing, cleanup,
   and atomic no-replace installation on Linux.
4. **Stage 4 — Notarius adapter:** Added the subprocess boundary, exact CLI
   invocation, separate receipt/diagnostic streams, bounded tolerant decoding,
   path confinement, and generic receipt/index discovery.
5. **Stage 5 — Extract execution:** Added final-trimmed transcript consumption,
   configured required-lane validation, immutable bundle promotion, checksums,
   manifest-ready outputs, provenance, diagnostics, and invocation
   fingerprinting.
6. **Stage 6 — Lifecycle and resume:** Registered `extract` between `trim` and
   `render`, added full and single-stage CLI support, resume validation, force
   ordering, and old-manifest compatibility.
7. **Stage 7 — Analyze integration:** Added shared manifest-backed extraction
   catalog hydration and explicit required/optional Scriptorium input
   resolution without directory scanning or implicit inputs.
8. **Stage 8 — Publish and inspection integration:** Added explicit extraction
   publish rules, metadata round-tripping, restore compatibility, and operator
   availability reporting.
9. **Stage 9 — Documentation and examples:** Added maintained ten-lane and
   subset examples, integration and maintainer documentation, operational
   guidance, and repository-wide validation.

## Stage 10: Correct Downstream Invalidation For Every Replaced Stage Result

Fix runner semantics so a newly produced extraction result cannot coexist with
downstream stages that are still considered current from an older result. Make
the correction at the general stage-runner boundary rather than adding an
`extract`-specific exception.

Implementation:

1. In `internal/app`, capture the prior session-stage outcome before changing a
   selected stage to running. Preserve enough prior state to distinguish a
   reused result, a repeated identical self-skip, and a replaced result.
2. Define replacement and invalidation behavior as follows:
   - an already-succeeded stage skipped without execution does not invalidate
     anything;
   - a resume-validation error still stops without mutating the succeeded stage
     or downstream stages;
   - a non-resumable succeeded result is marked stale and invalidates succeeded
     downstream stages before rerun, preserving the current behavior;
   - any forced execution invalidates succeeded downstream stages even if the
     execution later self-skips or fails;
   - a non-forced execution that changes a skipped, failed, stale, pending, or
     absent stage to succeeded invalidates succeeded downstream stages;
   - an executed failure invalidates succeeded downstream stages before the
     failed manifest state is committed; and
   - a self-skip invalidates succeeded downstream stages only when it replaces
     a different effective outcome. Repeating the same stable skip reason with
     no outputs must not make every disabled full-pipeline invocation rerun all
     downstream stages.
3. Persist invalidation with the same session-manifest transition that makes the
   upstream replacement observable. Do not leave a committed failed or skipped
   upstream result with succeeded downstream records that depend on its former
   output.
4. Continue to use canonical stage order for invalidation. For `extract`, the
   downstream set remains `render`, `analyze`, `publish`, and `notify`.
5. Recheck each precomputed stage decision immediately before acting on it.
   When an earlier stage invalidates a later succeeded stage during the same
   full run, the later stage must execute instead of honoring its original skip
   decision.
6. Use stable bounded stale reasons that distinguish forced replacement,
   changed upstream result, upstream failure, and upstream self-skip where that
   distinction is operationally useful.
7. Remove or consolidate the old success-only force invalidation path after the
   new invariant owns all replacement cases. Do not invalidate stages earlier
   than the replaced stage.

Tests:

- extend `internal/app/extract_lifecycle_test.go` with a full-plan or focused
  multi-stage fixture in which disabled `extract` and downstream `analyze` first
  complete, Notarius is enabled, and the next ordinary run executes both
  `extract` and the previously succeeded downstream stages;
- a failed extract followed by a successful ordinary retry invalidates and
  reruns previously succeeded downstream stages;
- `run-stage extract --force` followed by `notarius_disabled` leaves downstream
  succeeded stages stale;
- forced extraction failure leaves downstream succeeded stages stale;
- repeated identical disabled self-skip does not repeatedly stale downstream
  stages;
- successful resume reuse does not invalidate downstream stages;
- resume-validation error preserves both the prior extract success and
  downstream state; and
- forcing `render` still does not stale `extract`.

Exit criteria:

- Enabling, retrying, forcing, failing, or disabling extraction cannot leave an
  incompatible succeeded analyze/publish result current.
- Repeated unchanged disabled extraction remains inexpensive and stable.
- `go test ./internal/app ./internal/stage ./internal/manifest`,
  `go test ./...`, `go vet ./...`, and `go build ./cmd/narratio` pass.

## Stage 11: Exclude Run-Local Notarius Bundles From Implicit Publication

Enforce the existing explicit-only publication boundary while preserving
ordinary run-record diagnostics.

Implementation:

1. Update publish run-file collection so the canonical run-local
   `extract/notarius-output/` subtree is never included in the run-record upload.
   Match the exact slash-normalized relative subtree; do not use a broad
   substring rule that could suppress unrelated files.
2. Keep the Notarius receipt and stderr files eligible for the existing run
   archive. They are diagnostics, not the published bundle. Preserve current
   audio exclusion and all unrelated run-file behavior.
3. Do not scan or upload the durable `artifacts/notarius/<run-id>/` directory.
   A configured lane may still be uploaded only through an explicit
   `pipeline.publish.outputs` rule resolved through the artifact catalog.
4. Preserve locks, required/optional output handling, upload ordering, current
   manifest publication, and the final current-run pointer commit.
5. Update publish metadata so excluded staging-bundle files are not counted in
   `run_files_uploaded` or listed in `run_uploaded_paths`.

Tests:

- materialize a realistic current-run `extract/notarius-output/<external-run>/`
  tree containing index, management, lane, pipeline-wide, and unknown regular
  files, then prove none are uploaded under the run prefix;
- prove receipt and stderr diagnostics in the same extract directory remain
  ordinary run uploads;
- prove no Notarius bundle member is uploaded when there is no explicit
  extraction publish rule;
- prove one explicit extraction rule uploads only its durable selected lane to
  the configured session destination; and
- retain assertions for commit-marker order, locks, previous-cache publication,
  and unrelated run files.

Exit criteria:

- No run-local or durable Notarius bundle is published wholesale.
- Explicit configured lane publication remains functional and auditable.
- `go test ./internal/stage ./internal/app ./internal/artifacts` and
  `go test ./...` pass.

## Stage 12: Enforce Canonical Notarius Management-File Semantics

Make the adapter's accepted production contract agree with extract, resume, and
catalog invariants.

Implementation:

1. Re-read Notarius's current `run-result.md` and `json-output.md` before editing
   and keep tolerant decoding of unknown fields within the supported schema.
2. For `notarius.run-result.v1` using the production JSON output, require the
   receipt's logical `index_file` to be exactly `index.json`. A different safe
   relative path is semantically incompatible and must fail adapter discovery,
   not produce a successful result that later becomes non-resumable.
3. Require the production index management fields to be exactly:
   - `manifest_file: manifest.json`;
   - `rejected_file: rejected.json`; and
   - `warnings_file: warnings.json`.
4. Retain all existing confinement, regular-file, symlink, size-limit, and
   tolerant unknown-field checks after the exact semantic checks. Do not tighten
   optional lane descriptor fields beyond Notarius's published contract.
5. Keep the canonical index invariant shared by stage output, resume validation,
   and catalog hydration. Avoid introducing a second configurable or inferred
   management-path model.
6. Return errors that identify the incompatible field and observed value without
   including transcript-derived payload content.

Tests:

- reject alternate but confined values such as `nested/index.json`,
  `./index.json`, and renamed management files;
- continue rejecting absolute paths, traversal, symlinks, and missing files;
- continue accepting canonical documents with unknown optional fields;
- prove a successful adapter result promotes to canonical `index.json`; and
- prove a successful extract record is immediately accepted by resume validation
  and catalog hydration without normalization or path rewriting.

Exit criteria:

- Every adapter success satisfies the canonical management-file assumptions of
  extraction, resume, and catalog code.
- `go test ./internal/adapters/notarius ./internal/stage ./internal/artifacts`
  and `go test ./...` pass.

## Stage 13: Complete And Document The Atomic-Promotion Platform Contract

Replace the accidental Linux-only behavior with an explicit, tested support
boundary. Support Linux, macOS, and Windows; fail early and clearly on other
platforms until they gain an atomic no-replace primitive.

Implementation:

1. Keep Linux installation based on `renameat2(RENAME_NOREPLACE)`.
2. Add a Darwin implementation using `renamex_np(RENAME_EXCL)` from
   `golang.org/x/sys/unix`.
3. Add a Windows implementation using the no-replace form of
   `golang.org/x/sys/windows.MoveFileEx`. The sibling temporary directory keeps
   the move on one volume; do not set the replace-existing flag.
4. Split directory-sync behavior by platform where necessary. Unsupported
   directory syncing may be treated as best effort only for documented
   platform-specific unsupported-operation errors; do not suppress ordinary I/O
   or permission failures.
5. Add a small build-specific capability check so an unsupported platform fails
   before copying the bundle into a temporary tree. Retain safe cleanup if any
   later platform operation fails.
6. Preserve the core invariant on every supported platform: a concurrent actor
   that creates the destination wins or causes a clean error; Narratio never
   replaces that destination.
7. Document Linux, macOS, and Windows as the supported atomic-promotion
   platforms and the explicit extraction limitation on other operating systems.
   Do not imply that all of Narratio has a broader support guarantee than its
   existing documentation establishes.

Tests and validation:

- retain the portable promotion conformance tests for successful nested copy,
  source preservation, existing destination, cleanup, symlink rejection, and
  install collision;
- add platform-specific no-replace tests that run on their native CI platform;
- cross-compile the fileops tests for Linux, Darwin, and Windows;
- confirm the unsupported-platform implementation returns its capability error
  before creating a temporary sibling; and
- run `go test ./internal/fileops`, `go test ./...`, `go vet ./...`, and
  `go build ./cmd/narratio` on the development platform.

Exit criteria:

- Directory promotion is functional and no-replace on Linux, macOS, and
  Windows rather than merely compiling there.
- Unsupported platforms fail before expensive copying and have an explicit
  documented boundary.
- All available native and cross-compilation checks pass.

## Stage 14: Reconcile Documentation And Perform Final Remediation Validation

Make current-behavior documentation match the corrected implementation and
close the remediation effort only after end-to-end verification.

Implementation:

1. Correct `docs/internal/stage-extract.md` to say that omitted or disabled
   Notarius explicitly self-skips with `notarius_disabled`; do not describe the
   manifest result as succeeded.
2. Replace the nonexistent `internal/stage/extract_resume_test.go` reference with
   the actual focused test owner, or create that file only if tests were
   intentionally reorganized during remediation.
3. Update operations, publish, and extraction documentation to state that:
   - changed extraction outcomes stale affected downstream stages;
   - repeated identical disabled self-skip does not cause perpetual reruns;
   - the run-local Notarius bundle is excluded from run-record upload; and
   - only explicit configured extraction lanes are published.
4. Update platform-support documentation from Stage 13 and troubleshooting for
   unsupported atomic promotion.
5. Reconcile `docs/roadmap/notarius-extract-stage.md` with the remediated current
   state. Mark the feature complete only if every original and remediation
   acceptance criterion is satisfied.
6. Recheck maintained examples, internal links, stage inventories, command
   examples, field names, schema identities, defaults, and paths against code.
7. Keep roadmap history concise; do not restore the former detailed completed
   Stage 1-9 instructions to this file.

Validation:

- run focused lifecycle, adapter, fileops, catalog, analyze, publish, restore,
  and operator tests without live external services;
- run focused race tests covering the changed extraction, artifact, publish,
  fileops, and runner paths;
- run `go test -count=1 ./...`;
- run `go vet ./...`;
- run `go build ./cmd/narratio`;
- run the Stage 13 cross-compilation checks;
- run `git diff --check`; and
- confirm no secrets, private campaign content, generated binaries, or test
  artifacts were added to the repository.

Exit criteria:

- All post-implementation review findings are corrected or explicitly bounded
  by the documented platform contract.
- Current documentation accurately describes stage outcomes, invalidation,
  publication, adapter compatibility, and supported platforms.
- Every progress row is `Complete`, the repository-wide validation suite passes,
  and the target feature roadmap can truthfully remain complete.

## Open Questions

None. The remediation stages adopt the long-term-maintainable defaults: general
runner invalidation semantics, explicit-only bundle publication, exact current
Notarius management-file semantics, and atomic promotion support on Linux,
macOS, and Windows with an explicit early failure elsewhere.