narratio/docs/roadmap/implementation.md

Notarius Extraction Implementation Plan

Status And Audience

Remediation required. Stages 1 through 14 are complete. Stages 15 through 18 address the remaining security, lifecycle-state, and native-platform validation gaps identified by the second post-implementation review.

This plan is written for a GPT-5.6 Terra coding agent. Implement incomplete stages in strict numerical order. Do not skip ahead, combine stages merely to reduce the number of prompts, or describe unfinished behavior as current functionality.

The target state and acceptance criteria are owned by notarius-extract-stage.md. This document owns the implementation sequence.

Progress

Stage	Status
Stage 1	Complete
Stage 2	Complete
Stage 3	Complete
Stage 4	Complete
Stage 5	Complete
Stage 6	Complete
Stage 7	Complete
Stage 8	Complete
Stage 9	Complete
Stage 10	Complete
Stage 11	Complete
Stage 12	Complete
Stage 13	Complete
Stage 14	Complete
Stage 15	Complete
Stage 16	Complete
Stage 17	Pending
Stage 18	Pending

After completing and validating a stage, update only that stage's row to Complete and record any material deviation in the relevant stage section. Do not mark a stage complete while required tests or exit criteria remain.

Working Rules

Before Stage 15, read:

• docs/development.md and its task-specific references;
• docs/roadmap/notarius-extract-stage.md completely; and
• the focused Narratio documents, production code, and tests named by the current stage.

For every incomplete stage:

1. Inspect the current implementation before editing. Prefer the codebase knowledge graph for code discovery as required by AGENTS.md.
2. Preserve unrelated worktree changes.
3. Implement the complete stage scope, including focused regression tests. Do not leave TODO implementations or defer a known part of the stage.
4. Keep tests offline and independent of real Notarius, PromptKit, LLM providers, credentials, network services, and mutable external state.
5. Run the focused package tests listed for the stage, then go test ./....
6. Run go vet ./... and build cmd/narratio whenever shared contracts, composition, portability, or documentation examples change. Direct build output to a temporary directory so validation does not leave an ignored binary in the repository root.
7. Do not broaden the feature beyond the target architecture while fixing a remediation. Prefer a general manifest or filesystem invariant when the defect is general, but avoid unrelated cleanup.

Completed Stages 1-14 Summary

1. Stage 1 — Shared contracts: Added explicit stage self-skip semantics, artifact identities, contract metadata, provenance, and compatible manifest serialization.
2. Stage 2 — Configuration and source policy: Added strict optional Notarius configuration, deterministic path resolution, lane contracts, and the narratio.extraction.<key> source family.
3. Stage 3 — Immutable promotion: Added regular-file-only bundle copying, static symlink and special-file rejection, syncing, cleanup, and atomic no-replace installation on Linux.
4. Stage 4 — Notarius adapter: Added exact subprocess invocation, separate receipt and diagnostic streams, bounded tolerant decoding, confinement, and receipt/index discovery.
5. Stage 5 — Extract execution: Added transcript consumption, required-lane validation, immutable promotion, checksums, outputs, provenance, diagnostics, and invocation fingerprinting.
6. Stage 6 — Lifecycle and resume: Registered extract between trim and render, added CLI selection, resume validation, force behavior, and old-manifest compatibility.
7. Stage 7 — Analyze integration: Added shared manifest-backed extraction catalog hydration and explicit required/optional Scriptorium input resolution.
8. Stage 8 — Publish and inspection integration: Added explicit extraction publish rules, metadata round-tripping, restore compatibility, and operator availability reporting.
9. Stage 9 — Documentation and examples: Added maintained complete and subset examples, integration and maintainer documentation, operational guidance, and repository-wide validation.
10. Stage 10 — Replacement invalidation: Generalized runner semantics so forced, failed, skipped, changed, and non-resumable upstream executions invalidate succeeded downstream results without causing perpetual reruns for identical self-skips.
11. Stage 11 — Explicit-only publication: Excluded the run-local Notarius bundle from run-record uploads while retaining diagnostics and explicit configured-lane publication.
12. Stage 12 — Canonical management files: Required the production receipt and index to use the canonical index, manifest, rejection, and warning filenames assumed by extraction, resume, and catalog code.
13. Stage 13 — Platform promotion contract: Added atomic no-replace implementations for Linux, macOS, and Windows, explicit early failure on unsupported systems, and platform-specific directory syncing.
14. Stage 14 — Documentation reconciliation: Aligned current-behavior, operations, troubleshooting, integration, internal, and roadmap documentation with the implemented lifecycle and publication semantics.

Stage 15: Make Bundle Promotion Race-Safe Against Symlink Replacement

Close the gap between the documented no-symlink promotion invariant and the current path-based recursive copy. A source directory that is replaced after inspection must never cause Narratio to traverse a symlink or copy data from outside the originally opened source tree.

Implementation:

1. Refactor internal/fileops source traversal to anchor the copy to an opened source root for the entire operation. Use Go's os.Root APIs, available at the module's declared Go version, instead of recursively reopening source directories through unconstrained absolute paths.
2. Reject a source root that is itself a symlink. After opening the root, compare the opened root identity with the identity inspected before opening; fail if the source changed during that transition.
3. For every source entry:
   • inspect it relative to the already opened parent root;
   • reject symlinks, link-like Windows reparse points exposed as links, and every non-regular, non-directory file type;
   • open regular files relative to that root, compare the opened file identity with the inspected identity, and copy only from the verified handle; and
   • open child directories relative to the parent root, verify the opened directory identity against the inspected identity, and recurse through the opened child root rather than its pathname.
4. Treat any identity mismatch, disappearing entry, replacement, or unsupported file type as a clean promotion failure. Preserve the source, remove the temporary destination tree, and never install a partial destination.
5. Keep the destination-side behavior unchanged: deterministic traversal, normalized permissions, file and directory syncing, sibling temporary tree, and atomic no-replace installation.
6. Keep support bounded to Linux, macOS, and Windows. Do not weaken the strict no-symlink rule to permit links that happen to remain inside the source root.
7. Keep test-only orchestration narrow. If deterministic replacement testing requires an internal seam between inspection and opening, inject only that boundary; do not expose it publicly or turn file copying into a generic filesystem framework.

Tests:

• retain the existing nested-copy, source-preservation, static symlink, special-file, cleanup, existing-destination, and install-collision tests;
• deterministically replace an inspected child directory with a symlink before it is opened, and prove promotion fails without copying an outside sentinel;
• cover source-root replacement between initial inspection and root opening;
• cover regular-file and directory identity mismatches without timing-based sleeps or probabilistic race loops;
• prove every failure leaves the source and any concurrent destination intact and removes the temporary sibling; and
• run the focused tests with the race detector.

Exit criteria:

• Recursive promotion cannot escape or switch away from the originally opened source tree through a symlink replacement race.
• Static and concurrent symlink replacement are both protected by durable, deterministic tests.
• go test -count=1 ./internal/fileops, go test -race -count=1 ./internal/fileops, go test ./..., and go vet ./... pass on the development platform.

Stage 16: Clear Superseded Session-Stage Result Payloads

Make the session manifest describe only the current stage outcome. Historical successful result details belong to their immutable run manifests and must not remain attached to a current stage record that is running, failed, skipped, or newly succeeded without corresponding details.

Implementation:

1. Centralize clearing of result-bearing manifest.StageRecord fields: Outputs, Logs, GeneratedConfigs, and Metadata.
2. Clear those fields when MarkStageRunning begins a replacement attempt. The runner already captures the prior outcome before this transition; do not change self-skip comparison or downstream invalidation decisions.
3. Also make MarkStageFailed and MarkStageSkipped enforce the empty-result invariant themselves so callers cannot construct a failed or skipped record with inherited successful output details. Avoid runner-only cleanup that leaves direct manifest transitions inconsistent.
4. Do not clear result details merely when MarkStageStale is called. Resume validation must still be able to inspect the prior result before deciding to rerun it, and stale records remain useful diagnostic state until execution actually begins.
5. Ensure MarkStageSucceeded followed by result application cannot inherit old logs, generated configurations, or metadata when the new successful result omits those fields. Clearing at the running transition should own this invariant; do not add scattered empty-map special cases.
6. Preserve immutable prior run manifests and durable Notarius bundles. This stage changes the current session-stage record, not historical files or retention policy.
7. Consolidate the existing runner self-skip cleanup helper if it becomes redundant after manifest transitions own the invariant.

Tests:

• add manifest transition tests proving running, failed, and skipped records cannot retain prior outputs, logs, generated configurations, or metadata;
• prove MarkStageStale retains the prior details required for resume validation;
• change the forced-extraction-failure lifecycle fixture to first complete a successful extraction, then force a failing replacement, and assert the current extract record is failed with no inherited result payload;
• retain the assertion that succeeded downstream stages become stale;
• prove a successful replacement whose new result omits optional details does not inherit details from its predecessor; and
• retain resume-error, repeated-self-skip, and ordinary-retry coverage.

Exit criteria:

• A current running, failed, or skipped session-stage record never advertises result payload belonging to an earlier success.
• Historical run manifests and immutable extraction bundles remain available for audit and recovery.
• go test -count=1 ./internal/manifest ./internal/app ./internal/stage, focused lifecycle race tests, go test ./..., and go vet ./... pass.

Stage 17: Establish Native Promotion Validation On Every Supported Platform

Turn the Linux, macOS, and Windows promotion claim into a continuously verified contract. Cross-compilation remains useful but cannot substitute for executing filesystem operations on each native platform.

Implementation:

1. Resolve the CI substrate and native runner labels in the open question below before editing workflow files. Follow the repository host's established workflow location and syntax; do not invent runner labels that cannot run.
2. Add checked-in CI jobs that execute on native Linux, macOS, and Windows runners. Each native job must run at least:
   • go test -count=1 ./internal/fileops;
   • the platform-specific no-replace test; and
   • the portable promotion conformance tests, including successful promotion, collision preservation, cleanup, and the Stage 15 race-safe traversal cases that apply on that platform.
3. Ensure the jobs exercise the real platform implementation rather than a mocked rename function. A destination created before installation must win, and the source and destination contents must prove that no replacement occurred.
4. On Windows, ensure successful promotion exercises both MoveFileEx and the directory-sync path. Do not treat permission or ordinary I/O failures as unsupported-operation success.
5. Retain deterministic cross-compilation of the fileops test binary for Linux, Darwin, and Windows. Write compiled test binaries to a temporary or CI artifact directory, never the repository root.
6. Keep unsupported-platform compilation coverage, but do not misrepresent an unexecuted cross-compiled test as native validation.
7. Record any required runner-specific limitation in this stage and in the canonical contributor/CI documentation. Do not weaken production semantics merely to accommodate an inadequately provisioned runner.

Validation:

• obtain a successful native Linux job;
• obtain a successful native macOS job;
• obtain a successful native Windows job;
• retain successful Linux, Darwin, and Windows cross-compilation; and
• confirm CI does not leave generated test binaries or application binaries in the worktree or commit them as artifacts of the source tree.

Exit criteria:

• Atomic no-replace promotion and its directory-sync behavior execute successfully on native Linux, macOS, and Windows.
• The checked-in CI configuration will rerun those tests on future changes.
• Stage 17 is not marked complete based solely on Linux execution and cross-compilation.

Stage 18: Reconcile Security Documentation And Perform Final Validation

Close the remediation only after the corrected filesystem and manifest invariants are implemented and native platform evidence exists.

Implementation:

1. Reconcile docs/roadmap/notarius-extract-stage.md with the Stage 15 source traversal mechanism and Stage 16 current-result semantics. Keep normative security and manifest guarantees in their existing canonical owners.
2. Update focused internal or contributor documentation only where the new mechanism or CI workflow changes maintained implementation guidance. Do not duplicate volatile platform commands across multiple documents.
3. Recheck the extraction, workspace, manifest, operations, troubleshooting, integration, and testing documentation for contradictions introduced by the remediation.
4. Keep completed roadmap history concise. Mark the target feature roadmap and this implementation plan complete only after all Stage 15 through 17 exit criteria are satisfied.
5. Confirm that no secrets, private campaign material, generated binaries, platform test binaries, or temporary bundle trees were added to the repository.

Validation:

• run focused fileops, manifest, runner lifecycle, extraction, adapter, artifact-catalog, analyze, publish, restore, and operator tests;
• run focused race tests covering fileops, manifest transitions, runner lifecycle, extraction, artifacts, and publication;
• run go test -count=1 ./...;
• run go vet ./...;
• build cmd/narratio with -o targeting a temporary directory;
• rerun Linux, Darwin, and Windows fileops test cross-compilation;
• confirm the native CI jobs from Stage 17 are successful;
• run git diff --check; and
• verify the worktree contains no generated validation artifacts.

Exit criteria:

• The source traversal and current-manifest replacement gaps are closed by focused regression tests.
• Native filesystem behavior substantiates the documented Linux, macOS, and Windows support boundary.
• Current documentation matches the remediated implementation.
• Every progress row is Complete, the repository-wide validation suite passes, and the target feature roadmap can truthfully remain complete.

Open Questions

Which CI system and native runner labels are authoritative for this repository?

Recommended approach: Use Gitea Actions with a checked-in workflow under .gitea/workflows/, provided the repository owner supplies or confirms native Linux, macOS, and Windows runner labels. The repository is hosted on Gitea, so keeping the workflow with the authoritative repository minimizes mirrored configuration and makes the platform contract visible beside the code. Stage 17 must remain pending until all three native jobs have actually run.

Viable alternative: Use an existing external CI service with real native capacity, such as a maintained GitHub Actions mirror or Buildkite installation, and check its workflow or pipeline definition into the conventional repository location. This is appropriate when that service already owns release gating or when native macOS and Windows runners are unavailable in Gitea. The alternative must still produce repeatable native results for all three platforms; cross-compilation or an undocumented one-time manual run is not sufficient.