narratio/docs/roadmap/implementation.md

Notarius Extraction Implementation Plan

Status And Audience

Remediation planned. Stages 1 through 9 are complete; Stages 10 through 14 address gaps found during the post-implementation architecture review.

This plan is written for a GPT-5.6 Terra coding agent. Implement incomplete stages in strict numerical order. Do not skip ahead, combine stages merely to reduce the number of prompts, or describe unfinished behavior as current functionality.

The target state and acceptance criteria are owned by notarius-extract-stage.md. This document owns the implementation sequence.

Progress

Stage	Status
Stage 1	Complete
Stage 2	Complete
Stage 3	Complete
Stage 4	Complete
Stage 5	Complete
Stage 6	Complete
Stage 7	Complete
Stage 8	Complete
Stage 9	Complete
Stage 10	Complete
Stage 11	Pending
Stage 12	Pending
Stage 13	Pending
Stage 14	Pending

After completing and validating a stage, update only that stage's row to Complete and record any material deviation in the relevant stage section. Do not mark a stage complete while required tests or exit criteria remain.

Working Rules

Before Stage 10, read:

• docs/development.md and its task-specific references;
• docs/roadmap/notarius-extract-stage.md completely;
• ../notarius/docs/consumers/dnd-pipeline.md and its linked subprocess, receipt, JSON-output, and lane-contract documentation; and
• the focused Narratio documents and tests named by the current stage.

For every incomplete stage:

1. Inspect the current implementation before editing. Prefer the codebase knowledge graph for code discovery as required by AGENTS.md.
2. Preserve unrelated worktree changes.
3. Implement the complete stage scope, including focused regression tests. Do not leave TODO implementations or defer a known part of the stage.
4. Keep tests offline and independent of real Notarius, PromptKit, LLM providers, credentials, network services, and mutable external state.
5. Run the focused package tests listed for the stage, then go test ./....
6. Run go vet ./... and go build ./cmd/narratio whenever shared contracts, composition, CLI behavior, portability, or documentation examples change.
7. Do not broaden the feature beyond the target architecture while fixing a remediation. Prefer a general runner or filesystem invariant when the defect is general, but avoid unrelated cleanup.

Completed Stages 1-9 Summary

1. Stage 1 — Shared contracts: Added explicit stage self-skip semantics, artifact source identity, contract metadata, external provenance, and backward-compatible manifest serialization.
2. Stage 2 — Configuration and source policy: Added strict optional Notarius configuration, deterministic defaults/path resolution, output-lane contracts, and the narratio.extraction.<key> source family.
3. Stage 3 — Immutable promotion: Added regular-file-only bundle copying, symlink and special-file rejection, bounded permissions, syncing, cleanup, and atomic no-replace installation on Linux.
4. Stage 4 — Notarius adapter: Added the subprocess boundary, exact CLI invocation, separate receipt/diagnostic streams, bounded tolerant decoding, path confinement, and generic receipt/index discovery.
5. Stage 5 — Extract execution: Added final-trimmed transcript consumption, configured required-lane validation, immutable bundle promotion, checksums, manifest-ready outputs, provenance, diagnostics, and invocation fingerprinting.
6. Stage 6 — Lifecycle and resume: Registered extract between trim and render, added full and single-stage CLI support, resume validation, force ordering, and old-manifest compatibility.
7. Stage 7 — Analyze integration: Added shared manifest-backed extraction catalog hydration and explicit required/optional Scriptorium input resolution without directory scanning or implicit inputs.
8. Stage 8 — Publish and inspection integration: Added explicit extraction publish rules, metadata round-tripping, restore compatibility, and operator availability reporting.
9. Stage 9 — Documentation and examples: Added maintained ten-lane and subset examples, integration and maintainer documentation, operational guidance, and repository-wide validation.

Stage 10: Correct Downstream Invalidation For Every Replaced Stage Result

Fix runner semantics so a newly produced extraction result cannot coexist with downstream stages that are still considered current from an older result. Make the correction at the general stage-runner boundary rather than adding an extract-specific exception.

Implementation:

1. In internal/app, capture the prior session-stage outcome before changing a selected stage to running. Preserve enough prior state to distinguish a reused result, a repeated identical self-skip, and a replaced result.
2. Define replacement and invalidation behavior as follows:
   • an already-succeeded stage skipped without execution does not invalidate anything;
   • a resume-validation error still stops without mutating the succeeded stage or downstream stages;
   • a non-resumable succeeded result is marked stale and invalidates succeeded downstream stages before rerun, preserving the current behavior;
   • any forced execution invalidates succeeded downstream stages even if the execution later self-skips or fails;
   • a non-forced execution that changes a skipped, failed, stale, pending, or absent stage to succeeded invalidates succeeded downstream stages;
   • an executed failure invalidates succeeded downstream stages before the failed manifest state is committed; and
   • a self-skip invalidates succeeded downstream stages only when it replaces a different effective outcome. Repeating the same stable skip reason with no outputs must not make every disabled full-pipeline invocation rerun all downstream stages.
3. Persist invalidation with the same session-manifest transition that makes the upstream replacement observable. Do not leave a committed failed or skipped upstream result with succeeded downstream records that depend on its former output.
4. Continue to use canonical stage order for invalidation. For extract, the downstream set remains render, analyze, publish, and notify.
5. Recheck each precomputed stage decision immediately before acting on it. When an earlier stage invalidates a later succeeded stage during the same full run, the later stage must execute instead of honoring its original skip decision.
6. Use stable bounded stale reasons that distinguish forced replacement, changed upstream result, upstream failure, and upstream self-skip where that distinction is operationally useful.
7. Remove or consolidate the old success-only force invalidation path after the new invariant owns all replacement cases. Do not invalidate stages earlier than the replaced stage.

Tests:

• extend internal/app/extract_lifecycle_test.go with a full-plan or focused multi-stage fixture in which disabled extract and downstream analyze first complete, Notarius is enabled, and the next ordinary run executes both extract and the previously succeeded downstream stages;
• a failed extract followed by a successful ordinary retry invalidates and reruns previously succeeded downstream stages;
• run-stage extract --force followed by notarius_disabled leaves downstream succeeded stages stale;
• forced extraction failure leaves downstream succeeded stages stale;
• repeated identical disabled self-skip does not repeatedly stale downstream stages;
• successful resume reuse does not invalidate downstream stages;
• resume-validation error preserves both the prior extract success and downstream state; and
• forcing render still does not stale extract.

Exit criteria:

• Enabling, retrying, forcing, failing, or disabling extraction cannot leave an incompatible succeeded analyze/publish result current.
• Repeated unchanged disabled extraction remains inexpensive and stable.
• go test ./internal/app ./internal/stage ./internal/manifest, go test ./..., go vet ./..., and go build ./cmd/narratio pass.

Stage 11: Exclude Run-Local Notarius Bundles From Implicit Publication

Enforce the existing explicit-only publication boundary while preserving ordinary run-record diagnostics.

Implementation:

1. Update publish run-file collection so the canonical run-local extract/notarius-output/ subtree is never included in the run-record upload. Match the exact slash-normalized relative subtree; do not use a broad substring rule that could suppress unrelated files.
2. Keep the Notarius receipt and stderr files eligible for the existing run archive. They are diagnostics, not the published bundle. Preserve current audio exclusion and all unrelated run-file behavior.
3. Do not scan or upload the durable artifacts/notarius/<run-id>/ directory. A configured lane may still be uploaded only through an explicit pipeline.publish.outputs rule resolved through the artifact catalog.
4. Preserve locks, required/optional output handling, upload ordering, current manifest publication, and the final current-run pointer commit.
5. Update publish metadata so excluded staging-bundle files are not counted in run_files_uploaded or listed in run_uploaded_paths.

Tests:

• materialize a realistic current-run extract/notarius-output/<external-run>/ tree containing index, management, lane, pipeline-wide, and unknown regular files, then prove none are uploaded under the run prefix;
• prove receipt and stderr diagnostics in the same extract directory remain ordinary run uploads;
• prove no Notarius bundle member is uploaded when there is no explicit extraction publish rule;
• prove one explicit extraction rule uploads only its durable selected lane to the configured session destination; and
• retain assertions for commit-marker order, locks, previous-cache publication, and unrelated run files.

Exit criteria:

• No run-local or durable Notarius bundle is published wholesale.
• Explicit configured lane publication remains functional and auditable.
• go test ./internal/stage ./internal/app ./internal/artifacts and go test ./... pass.

Stage 12: Enforce Canonical Notarius Management-File Semantics

Make the adapter's accepted production contract agree with extract, resume, and catalog invariants.

Implementation:

1. Re-read Notarius's current run-result.md and json-output.md before editing and keep tolerant decoding of unknown fields within the supported schema.
2. For notarius.run-result.v1 using the production JSON output, require the receipt's logical index_file to be exactly index.json. A different safe relative path is semantically incompatible and must fail adapter discovery, not produce a successful result that later becomes non-resumable.
3. Require the production index management fields to be exactly:
   • manifest_file: manifest.json;
   • rejected_file: rejected.json; and
   • warnings_file: warnings.json.
4. Retain all existing confinement, regular-file, symlink, size-limit, and tolerant unknown-field checks after the exact semantic checks. Do not tighten optional lane descriptor fields beyond Notarius's published contract.
5. Keep the canonical index invariant shared by stage output, resume validation, and catalog hydration. Avoid introducing a second configurable or inferred management-path model.
6. Return errors that identify the incompatible field and observed value without including transcript-derived payload content.

Tests:

• reject alternate but confined values such as nested/index.json, ./index.json, and renamed management files;
• continue rejecting absolute paths, traversal, symlinks, and missing files;
• continue accepting canonical documents with unknown optional fields;
• prove a successful adapter result promotes to canonical index.json; and
• prove a successful extract record is immediately accepted by resume validation and catalog hydration without normalization or path rewriting.

Exit criteria:

• Every adapter success satisfies the canonical management-file assumptions of extraction, resume, and catalog code.
• go test ./internal/adapters/notarius ./internal/stage ./internal/artifacts and go test ./... pass.

Stage 13: Complete And Document The Atomic-Promotion Platform Contract

Replace the accidental Linux-only behavior with an explicit, tested support boundary. Support Linux, macOS, and Windows; fail early and clearly on other platforms until they gain an atomic no-replace primitive.

Implementation:

1. Keep Linux installation based on renameat2(RENAME_NOREPLACE).
2. Add a Darwin implementation using renamex_np(RENAME_EXCL) from golang.org/x/sys/unix.
3. Add a Windows implementation using the no-replace form of golang.org/x/sys/windows.MoveFileEx. The sibling temporary directory keeps the move on one volume; do not set the replace-existing flag.
4. Split directory-sync behavior by platform where necessary. Unsupported directory syncing may be treated as best effort only for documented platform-specific unsupported-operation errors; do not suppress ordinary I/O or permission failures.
5. Add a small build-specific capability check so an unsupported platform fails before copying the bundle into a temporary tree. Retain safe cleanup if any later platform operation fails.
6. Preserve the core invariant on every supported platform: a concurrent actor that creates the destination wins or causes a clean error; Narratio never replaces that destination.
7. Document Linux, macOS, and Windows as the supported atomic-promotion platforms and the explicit extraction limitation on other operating systems. Do not imply that all of Narratio has a broader support guarantee than its existing documentation establishes.

Tests and validation:

• retain the portable promotion conformance tests for successful nested copy, source preservation, existing destination, cleanup, symlink rejection, and install collision;
• add platform-specific no-replace tests that run on their native CI platform;
• cross-compile the fileops tests for Linux, Darwin, and Windows;
• confirm the unsupported-platform implementation returns its capability error before creating a temporary sibling; and
• run go test ./internal/fileops, go test ./..., go vet ./..., and go build ./cmd/narratio on the development platform.

Exit criteria:

• Directory promotion is functional and no-replace on Linux, macOS, and Windows rather than merely compiling there.
• Unsupported platforms fail before expensive copying and have an explicit documented boundary.
• All available native and cross-compilation checks pass.

Stage 14: Reconcile Documentation And Perform Final Remediation Validation

Make current-behavior documentation match the corrected implementation and close the remediation effort only after end-to-end verification.

Implementation:

1. Correct docs/internal/stage-extract.md to say that omitted or disabled Notarius explicitly self-skips with notarius_disabled; do not describe the manifest result as succeeded.
2. Replace the nonexistent internal/stage/extract_resume_test.go reference with the actual focused test owner, or create that file only if tests were intentionally reorganized during remediation.
3. Update operations, publish, and extraction documentation to state that:
   • changed extraction outcomes stale affected downstream stages;
   • repeated identical disabled self-skip does not cause perpetual reruns;
   • the run-local Notarius bundle is excluded from run-record upload; and
   • only explicit configured extraction lanes are published.
4. Update platform-support documentation from Stage 13 and troubleshooting for unsupported atomic promotion.
5. Reconcile docs/roadmap/notarius-extract-stage.md with the remediated current state. Mark the feature complete only if every original and remediation acceptance criterion is satisfied.
6. Recheck maintained examples, internal links, stage inventories, command examples, field names, schema identities, defaults, and paths against code.
7. Keep roadmap history concise; do not restore the former detailed completed Stage 1-9 instructions to this file.

Validation:

• run focused lifecycle, adapter, fileops, catalog, analyze, publish, restore, and operator tests without live external services;
• run focused race tests covering the changed extraction, artifact, publish, fileops, and runner paths;
• run go test -count=1 ./...;
• run go vet ./...;
• run go build ./cmd/narratio;
• run the Stage 13 cross-compilation checks;
• run git diff --check; and
• confirm no secrets, private campaign content, generated binaries, or test artifacts were added to the repository.

Exit criteria:

• All post-implementation review findings are corrected or explicitly bounded by the documented platform contract.
• Current documentation accurately describes stage outcomes, invalidation, publication, adapter compatibility, and supported platforms.
• Every progress row is Complete, the repository-wide validation suite passes, and the target feature roadmap can truthfully remain complete.

Open Questions

None. The remediation stages adopt the long-term-maintainable defaults: general runner invalidation semantics, explicit-only bundle publication, exact current Notarius management-file semantics, and atomic promotion support on Linux, macOS, and Windows with an explicit early failure elsewhere.