narratio/docs/roadmap/implementation.md

# Notarius Extraction Implementation Plan

## Status And Audience

Remediation required. Stages 1 through 14 are complete. Stages 15 through 18
address the remaining security, lifecycle-state, and native-platform validation
gaps identified by the second post-implementation review.

This plan is written for a GPT-5.6 Terra coding agent. Implement incomplete
stages in strict numerical order. Do not skip ahead, combine stages merely to
reduce the number of prompts, or describe unfinished behavior as current
functionality.

The target state and acceptance criteria are owned by
[notarius-extract-stage.md](notarius-extract-stage.md). This document owns the
implementation sequence.

## Progress

| Stage | Status |
| --- | --- |
| Stage 1 | Complete |
| Stage 2 | Complete |
| Stage 3 | Complete |
| Stage 4 | Complete |
| Stage 5 | Complete |
| Stage 6 | Complete |
| Stage 7 | Complete |
| Stage 8 | Complete |
| Stage 9 | Complete |
| Stage 10 | Complete |
| Stage 11 | Complete |
| Stage 12 | Complete |
| Stage 13 | Complete |
| Stage 14 | Complete |
| Stage 15 | Complete |
| Stage 16 | Pending |
| Stage 17 | Pending |
| Stage 18 | Pending |

After completing and validating a stage, update only that stage's row to
`Complete` and record any material deviation in the relevant stage section.
Do not mark a stage complete while required tests or exit criteria remain.

## Working Rules

Before Stage 15, read:

- `docs/development.md` and its task-specific references;
- `docs/roadmap/notarius-extract-stage.md` completely; and
- the focused Narratio documents, production code, and tests named by the
  current stage.

For every incomplete stage:

1. Inspect the current implementation before editing. Prefer the codebase
   knowledge graph for code discovery as required by `AGENTS.md`.
2. Preserve unrelated worktree changes.
3. Implement the complete stage scope, including focused regression tests. Do
   not leave TODO implementations or defer a known part of the stage.
4. Keep tests offline and independent of real Notarius, PromptKit, LLM
   providers, credentials, network services, and mutable external state.
5. Run the focused package tests listed for the stage, then `go test ./...`.
6. Run `go vet ./...` and build `cmd/narratio` whenever shared contracts,
   composition, portability, or documentation examples change. Direct build
   output to a temporary directory so validation does not leave an ignored
   binary in the repository root.
7. Do not broaden the feature beyond the target architecture while fixing a
   remediation. Prefer a general manifest or filesystem invariant when the
   defect is general, but avoid unrelated cleanup.

## Completed Stages 1-14 Summary

1. **Stage 1 — Shared contracts:** Added explicit stage self-skip semantics,
   artifact identities, contract metadata, provenance, and compatible manifest
   serialization.
2. **Stage 2 — Configuration and source policy:** Added strict optional
   Notarius configuration, deterministic path resolution, lane contracts, and
   the `narratio.extraction.<key>` source family.
3. **Stage 3 — Immutable promotion:** Added regular-file-only bundle copying,
   static symlink and special-file rejection, syncing, cleanup, and atomic
   no-replace installation on Linux.
4. **Stage 4 — Notarius adapter:** Added exact subprocess invocation, separate
   receipt and diagnostic streams, bounded tolerant decoding, confinement, and
   receipt/index discovery.
5. **Stage 5 — Extract execution:** Added transcript consumption, required-lane
   validation, immutable promotion, checksums, outputs, provenance,
   diagnostics, and invocation fingerprinting.
6. **Stage 6 — Lifecycle and resume:** Registered `extract` between `trim` and
   `render`, added CLI selection, resume validation, force behavior, and
   old-manifest compatibility.
7. **Stage 7 — Analyze integration:** Added shared manifest-backed extraction
   catalog hydration and explicit required/optional Scriptorium input
   resolution.
8. **Stage 8 — Publish and inspection integration:** Added explicit extraction
   publish rules, metadata round-tripping, restore compatibility, and operator
   availability reporting.
9. **Stage 9 — Documentation and examples:** Added maintained complete and
   subset examples, integration and maintainer documentation, operational
   guidance, and repository-wide validation.
10. **Stage 10 — Replacement invalidation:** Generalized runner semantics so
    forced, failed, skipped, changed, and non-resumable upstream executions
    invalidate succeeded downstream results without causing perpetual reruns
    for identical self-skips.
11. **Stage 11 — Explicit-only publication:** Excluded the run-local Notarius
    bundle from run-record uploads while retaining diagnostics and explicit
    configured-lane publication.
12. **Stage 12 — Canonical management files:** Required the production receipt
    and index to use the canonical index, manifest, rejection, and warning
    filenames assumed by extraction, resume, and catalog code.
13. **Stage 13 — Platform promotion contract:** Added atomic no-replace
    implementations for Linux, macOS, and Windows, explicit early failure on
    unsupported systems, and platform-specific directory syncing.
14. **Stage 14 — Documentation reconciliation:** Aligned current-behavior,
    operations, troubleshooting, integration, internal, and roadmap
    documentation with the implemented lifecycle and publication semantics.

## Stage 15: Make Bundle Promotion Race-Safe Against Symlink Replacement

Close the gap between the documented no-symlink promotion invariant and the
current path-based recursive copy. A source directory that is replaced after
inspection must never cause Narratio to traverse a symlink or copy data from
outside the originally opened source tree.

Implementation:

1. Refactor `internal/fileops` source traversal to anchor the copy to an opened
   source root for the entire operation. Use Go's `os.Root` APIs, available at
   the module's declared Go version, instead of recursively reopening source
   directories through unconstrained absolute paths.
2. Reject a source root that is itself a symlink. After opening the root,
   compare the opened root identity with the identity inspected before opening;
   fail if the source changed during that transition.
3. For every source entry:
   - inspect it relative to the already opened parent root;
   - reject symlinks, link-like Windows reparse points exposed as links, and
     every non-regular, non-directory file type;
   - open regular files relative to that root, compare the opened file identity
     with the inspected identity, and copy only from the verified handle; and
   - open child directories relative to the parent root, verify the opened
     directory identity against the inspected identity, and recurse through the
     opened child root rather than its pathname.
4. Treat any identity mismatch, disappearing entry, replacement, or unsupported
   file type as a clean promotion failure. Preserve the source, remove the
   temporary destination tree, and never install a partial destination.
5. Keep the destination-side behavior unchanged: deterministic traversal,
   normalized permissions, file and directory syncing, sibling temporary tree,
   and atomic no-replace installation.
6. Keep support bounded to Linux, macOS, and Windows. Do not weaken the strict
   no-symlink rule to permit links that happen to remain inside the source root.
7. Keep test-only orchestration narrow. If deterministic replacement testing
   requires an internal seam between inspection and opening, inject only that
   boundary; do not expose it publicly or turn file copying into a generic
   filesystem framework.

Tests:

- retain the existing nested-copy, source-preservation, static symlink,
  special-file, cleanup, existing-destination, and install-collision tests;
- deterministically replace an inspected child directory with a symlink before
  it is opened, and prove promotion fails without copying an outside sentinel;
- cover source-root replacement between initial inspection and root opening;
- cover regular-file and directory identity mismatches without timing-based
  sleeps or probabilistic race loops;
- prove every failure leaves the source and any concurrent destination intact
  and removes the temporary sibling; and
- run the focused tests with the race detector.

Exit criteria:

- Recursive promotion cannot escape or switch away from the originally opened
  source tree through a symlink replacement race.
- Static and concurrent symlink replacement are both protected by durable,
  deterministic tests.
- `go test -count=1 ./internal/fileops`,
  `go test -race -count=1 ./internal/fileops`, `go test ./...`, and
  `go vet ./...` pass on the development platform.

## Stage 16: Clear Superseded Session-Stage Result Payloads

Make the session manifest describe only the current stage outcome. Historical
successful result details belong to their immutable run manifests and must not
remain attached to a current stage record that is running, failed, skipped, or
newly succeeded without corresponding details.

Implementation:

1. Centralize clearing of result-bearing `manifest.StageRecord` fields:
   `Outputs`, `Logs`, `GeneratedConfigs`, and `Metadata`.
2. Clear those fields when `MarkStageRunning` begins a replacement attempt.
   The runner already captures the prior outcome before this transition; do not
   change self-skip comparison or downstream invalidation decisions.
3. Also make `MarkStageFailed` and `MarkStageSkipped` enforce the empty-result
   invariant themselves so callers cannot construct a failed or skipped record
   with inherited successful output details. Avoid runner-only cleanup that
   leaves direct manifest transitions inconsistent.
4. Do not clear result details merely when `MarkStageStale` is called. Resume
   validation must still be able to inspect the prior result before deciding to
   rerun it, and stale records remain useful diagnostic state until execution
   actually begins.
5. Ensure `MarkStageSucceeded` followed by result application cannot inherit
   old logs, generated configurations, or metadata when the new successful
   result omits those fields. Clearing at the running transition should own this
   invariant; do not add scattered empty-map special cases.
6. Preserve immutable prior run manifests and durable Notarius bundles. This
   stage changes the current session-stage record, not historical files or
   retention policy.
7. Consolidate the existing runner self-skip cleanup helper if it becomes
   redundant after manifest transitions own the invariant.

Tests:

- add manifest transition tests proving running, failed, and skipped records
  cannot retain prior outputs, logs, generated configurations, or metadata;
- prove `MarkStageStale` retains the prior details required for resume
  validation;
- change the forced-extraction-failure lifecycle fixture to first complete a
  successful extraction, then force a failing replacement, and assert the
  current extract record is failed with no inherited result payload;
- retain the assertion that succeeded downstream stages become stale;
- prove a successful replacement whose new result omits optional details does
  not inherit details from its predecessor; and
- retain resume-error, repeated-self-skip, and ordinary-retry coverage.

Exit criteria:

- A current running, failed, or skipped session-stage record never advertises
  result payload belonging to an earlier success.
- Historical run manifests and immutable extraction bundles remain available
  for audit and recovery.
- `go test -count=1 ./internal/manifest ./internal/app ./internal/stage`,
  focused lifecycle race tests, `go test ./...`, and `go vet ./...` pass.

## Stage 17: Establish Native Promotion Validation On Every Supported Platform

Turn the Linux, macOS, and Windows promotion claim into a continuously verified
contract. Cross-compilation remains useful but cannot substitute for executing
filesystem operations on each native platform.

Implementation:

1. Resolve the CI substrate and native runner labels in the open question below
   before editing workflow files. Follow the repository host's established
   workflow location and syntax; do not invent runner labels that cannot run.
2. Add checked-in CI jobs that execute on native Linux, macOS, and Windows
   runners. Each native job must run at least:
   - `go test -count=1 ./internal/fileops`;
   - the platform-specific no-replace test; and
   - the portable promotion conformance tests, including successful promotion,
     collision preservation, cleanup, and the Stage 15 race-safe traversal
     cases that apply on that platform.
3. Ensure the jobs exercise the real platform implementation rather than a
   mocked rename function. A destination created before installation must win,
   and the source and destination contents must prove that no replacement
   occurred.
4. On Windows, ensure successful promotion exercises both `MoveFileEx` and the
   directory-sync path. Do not treat permission or ordinary I/O failures as
   unsupported-operation success.
5. Retain deterministic cross-compilation of the fileops test binary for Linux,
   Darwin, and Windows. Write compiled test binaries to a temporary or CI
   artifact directory, never the repository root.
6. Keep unsupported-platform compilation coverage, but do not misrepresent an
   unexecuted cross-compiled test as native validation.
7. Record any required runner-specific limitation in this stage and in the
   canonical contributor/CI documentation. Do not weaken production semantics
   merely to accommodate an inadequately provisioned runner.

Validation:

- obtain a successful native Linux job;
- obtain a successful native macOS job;
- obtain a successful native Windows job;
- retain successful Linux, Darwin, and Windows cross-compilation; and
- confirm CI does not leave generated test binaries or application binaries in
  the worktree or commit them as artifacts of the source tree.

Exit criteria:

- Atomic no-replace promotion and its directory-sync behavior execute
  successfully on native Linux, macOS, and Windows.
- The checked-in CI configuration will rerun those tests on future changes.
- Stage 17 is not marked complete based solely on Linux execution and
  cross-compilation.

## Stage 18: Reconcile Security Documentation And Perform Final Validation

Close the remediation only after the corrected filesystem and manifest
invariants are implemented and native platform evidence exists.

Implementation:

1. Reconcile `docs/roadmap/notarius-extract-stage.md` with the Stage 15 source
   traversal mechanism and Stage 16 current-result semantics. Keep normative
   security and manifest guarantees in their existing canonical owners.
2. Update focused internal or contributor documentation only where the new
   mechanism or CI workflow changes maintained implementation guidance. Do not
   duplicate volatile platform commands across multiple documents.
3. Recheck the extraction, workspace, manifest, operations, troubleshooting,
   integration, and testing documentation for contradictions introduced by the
   remediation.
4. Keep completed roadmap history concise. Mark the target feature roadmap and
   this implementation plan complete only after all Stage 15 through 17 exit
   criteria are satisfied.
5. Confirm that no secrets, private campaign material, generated binaries,
   platform test binaries, or temporary bundle trees were added to the
   repository.

Validation:

- run focused fileops, manifest, runner lifecycle, extraction, adapter,
  artifact-catalog, analyze, publish, restore, and operator tests;
- run focused race tests covering fileops, manifest transitions, runner
  lifecycle, extraction, artifacts, and publication;
- run `go test -count=1 ./...`;
- run `go vet ./...`;
- build `cmd/narratio` with `-o` targeting a temporary directory;
- rerun Linux, Darwin, and Windows fileops test cross-compilation;
- confirm the native CI jobs from Stage 17 are successful;
- run `git diff --check`; and
- verify the worktree contains no generated validation artifacts.

Exit criteria:

- The source traversal and current-manifest replacement gaps are closed by
  focused regression tests.
- Native filesystem behavior substantiates the documented Linux, macOS, and
  Windows support boundary.
- Current documentation matches the remediated implementation.
- Every progress row is `Complete`, the repository-wide validation suite
  passes, and the target feature roadmap can truthfully remain complete.

## Open Questions

### Which CI system and native runner labels are authoritative for this repository?

**Recommended approach:** Use Gitea Actions with a checked-in workflow under
`.gitea/workflows/`, provided the repository owner supplies or confirms native
Linux, macOS, and Windows runner labels. The repository is hosted on Gitea, so
keeping the workflow with the authoritative repository minimizes mirrored
configuration and makes the platform contract visible beside the code. Stage
17 must remain pending until all three native jobs have actually run.

**Viable alternative:** Use an existing external CI service with real native
capacity, such as a maintained GitHub Actions mirror or Buildkite installation,
and check its workflow or pipeline definition into the conventional repository
location. This is appropriate when that service already owns release gating or
when native macOS and Windows runners are unavailable in Gitea. The alternative
must still produce repeatable native results for all three platforms;
cross-compilation or an undocumented one-time manual run is not sufficient.