eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

Close the completed roadmap documents

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-10 03:24:49 +00:00
parent 7cb18a1a40
commit 74e2d21de5
3 changed files with 98 additions and 994 deletions
Download Patch File Download Diff File Expand all files Collapse all files

349
docs/roadmap/implementation.md
View File

@@ -1,349 +0,0 @@
# Notarius Extraction Implementation Plan
## Status And Audience
Remediation required. Stages 1 through 14 are complete. Stages 15 through 18
address the remaining security, lifecycle-state, and native-platform validation
gaps identified by the second post-implementation review.
This plan is written for a GPT-5.6 Terra coding agent. Implement incomplete
stages in strict numerical order. Do not skip ahead, combine stages merely to
reduce the number of prompts, or describe unfinished behavior as current
functionality.
The target state and acceptance criteria are owned by
[notarius-extract-stage.md](notarius-extract-stage.md). This document owns the
implementation sequence.
## Progress
| Stage | Status |
| --- | --- |
| Stage 1 | Complete |
| Stage 2 | Complete |
| Stage 3 | Complete |
| Stage 4 | Complete |
| Stage 5 | Complete |
| Stage 6 | Complete |
| Stage 7 | Complete |
| Stage 8 | Complete |
| Stage 9 | Complete |
| Stage 10 | Complete |
| Stage 11 | Complete |
| Stage 12 | Complete |
| Stage 13 | Complete |
| Stage 14 | Complete |
| Stage 15 | Complete |
| Stage 16 | Complete |
| Stage 17 | Pending |
| Stage 18 | Pending |
After completing and validating a stage, update only that stage's row to
`Complete` and record any material deviation in the relevant stage section.
Do not mark a stage complete while required tests or exit criteria remain.
## Working Rules
Before Stage 15, read:
- `docs/development.md` and its task-specific references;
- `docs/roadmap/notarius-extract-stage.md` completely; and
- the focused Narratio documents, production code, and tests named by the
current stage.
For every incomplete stage:
1. Inspect the current implementation before editing. Prefer the codebase
knowledge graph for code discovery as required by `AGENTS.md`.
2. Preserve unrelated worktree changes.
3. Implement the complete stage scope, including focused regression tests. Do
not leave TODO implementations or defer a known part of the stage.
4. Keep tests offline and independent of real Notarius, PromptKit, LLM
providers, credentials, network services, and mutable external state.
5. Run the focused package tests listed for the stage, then `go test ./...`.
6. Run `go vet ./...` and build `cmd/narratio` whenever shared contracts,
composition, portability, or documentation examples change. Direct build
output to a temporary directory so validation does not leave an ignored
binary in the repository root.
7. Do not broaden the feature beyond the target architecture while fixing a
remediation. Prefer a general manifest or filesystem invariant when the
defect is general, but avoid unrelated cleanup.
## Completed Stages 1-14 Summary
1. **Stage 1 — Shared contracts:** Added explicit stage self-skip semantics,
artifact identities, contract metadata, provenance, and compatible manifest
serialization.
2. **Stage 2 — Configuration and source policy:** Added strict optional
Notarius configuration, deterministic path resolution, lane contracts, and
the `narratio.extraction.<key>` source family.
3. **Stage 3 — Immutable promotion:** Added regular-file-only bundle copying,
static symlink and special-file rejection, syncing, cleanup, and atomic
no-replace installation on Linux.
4. **Stage 4 — Notarius adapter:** Added exact subprocess invocation, separate
receipt and diagnostic streams, bounded tolerant decoding, confinement, and
receipt/index discovery.
5. **Stage 5 — Extract execution:** Added transcript consumption, required-lane
validation, immutable promotion, checksums, outputs, provenance,
diagnostics, and invocation fingerprinting.
6. **Stage 6 — Lifecycle and resume:** Registered `extract` between `trim` and
`render`, added CLI selection, resume validation, force behavior, and
old-manifest compatibility.
7. **Stage 7 — Analyze integration:** Added shared manifest-backed extraction
catalog hydration and explicit required/optional Scriptorium input
resolution.
8. **Stage 8 — Publish and inspection integration:** Added explicit extraction
publish rules, metadata round-tripping, restore compatibility, and operator
availability reporting.
9. **Stage 9 — Documentation and examples:** Added maintained complete and
subset examples, integration and maintainer documentation, operational
guidance, and repository-wide validation.
10. **Stage 10 — Replacement invalidation:** Generalized runner semantics so
forced, failed, skipped, changed, and non-resumable upstream executions
invalidate succeeded downstream results without causing perpetual reruns
for identical self-skips.
11. **Stage 11 — Explicit-only publication:** Excluded the run-local Notarius
bundle from run-record uploads while retaining diagnostics and explicit
configured-lane publication.
12. **Stage 12 — Canonical management files:** Required the production receipt
and index to use the canonical index, manifest, rejection, and warning
filenames assumed by extraction, resume, and catalog code.
13. **Stage 13 — Platform promotion contract:** Added atomic no-replace
implementations for Linux, macOS, and Windows, explicit early failure on
unsupported systems, and platform-specific directory syncing.
14. **Stage 14 — Documentation reconciliation:** Aligned current-behavior,
operations, troubleshooting, integration, internal, and roadmap
documentation with the implemented lifecycle and publication semantics.
## Stage 15: Make Bundle Promotion Race-Safe Against Symlink Replacement
Close the gap between the documented no-symlink promotion invariant and the
current path-based recursive copy. A source directory that is replaced after
inspection must never cause Narratio to traverse a symlink or copy data from
outside the originally opened source tree.
Implementation:
1. Refactor `internal/fileops` source traversal to anchor the copy to an opened
source root for the entire operation. Use Go's `os.Root` APIs, available at
the module's declared Go version, instead of recursively reopening source
directories through unconstrained absolute paths.
2. Reject a source root that is itself a symlink. After opening the root,
compare the opened root identity with the identity inspected before opening;
fail if the source changed during that transition.
3. For every source entry:
- inspect it relative to the already opened parent root;
- reject symlinks, link-like Windows reparse points exposed as links, and
every non-regular, non-directory file type;
- open regular files relative to that root, compare the opened file identity
with the inspected identity, and copy only from the verified handle; and
- open child directories relative to the parent root, verify the opened
directory identity against the inspected identity, and recurse through the
opened child root rather than its pathname.
4. Treat any identity mismatch, disappearing entry, replacement, or unsupported
file type as a clean promotion failure. Preserve the source, remove the
temporary destination tree, and never install a partial destination.
5. Keep the destination-side behavior unchanged: deterministic traversal,
normalized permissions, file and directory syncing, sibling temporary tree,
and atomic no-replace installation.
6. Keep support bounded to Linux, macOS, and Windows. Do not weaken the strict
no-symlink rule to permit links that happen to remain inside the source root.
7. Keep test-only orchestration narrow. If deterministic replacement testing
requires an internal seam between inspection and opening, inject only that
boundary; do not expose it publicly or turn file copying into a generic
filesystem framework.
Tests:
- retain the existing nested-copy, source-preservation, static symlink,
special-file, cleanup, existing-destination, and install-collision tests;
- deterministically replace an inspected child directory with a symlink before
it is opened, and prove promotion fails without copying an outside sentinel;
- cover source-root replacement between initial inspection and root opening;
- cover regular-file and directory identity mismatches without timing-based
sleeps or probabilistic race loops;
- prove every failure leaves the source and any concurrent destination intact
and removes the temporary sibling; and
- run the focused tests with the race detector.
Exit criteria:
- Recursive promotion cannot escape or switch away from the originally opened
source tree through a symlink replacement race.
- Static and concurrent symlink replacement are both protected by durable,
deterministic tests.
- `go test -count=1 ./internal/fileops`,
`go test -race -count=1 ./internal/fileops`, `go test ./...`, and
`go vet ./...` pass on the development platform.
## Stage 16: Clear Superseded Session-Stage Result Payloads
Make the session manifest describe only the current stage outcome. Historical
successful result details belong to their immutable run manifests and must not
remain attached to a current stage record that is running, failed, skipped, or
newly succeeded without corresponding details.
Implementation:
1. Centralize clearing of result-bearing `manifest.StageRecord` fields:
`Outputs`, `Logs`, `GeneratedConfigs`, and `Metadata`.
2. Clear those fields when `MarkStageRunning` begins a replacement attempt.
The runner already captures the prior outcome before this transition; do not
change self-skip comparison or downstream invalidation decisions.
3. Also make `MarkStageFailed` and `MarkStageSkipped` enforce the empty-result
invariant themselves so callers cannot construct a failed or skipped record
with inherited successful output details. Avoid runner-only cleanup that
leaves direct manifest transitions inconsistent.
4. Do not clear result details merely when `MarkStageStale` is called. Resume
validation must still be able to inspect the prior result before deciding to
rerun it, and stale records remain useful diagnostic state until execution
actually begins.
5. Ensure `MarkStageSucceeded` followed by result application cannot inherit
old logs, generated configurations, or metadata when the new successful
result omits those fields. Clearing at the running transition should own this
invariant; do not add scattered empty-map special cases.
6. Preserve immutable prior run manifests and durable Notarius bundles. This
stage changes the current session-stage record, not historical files or
retention policy.
7. Consolidate the existing runner self-skip cleanup helper if it becomes
redundant after manifest transitions own the invariant.
Tests:
- add manifest transition tests proving running, failed, and skipped records
cannot retain prior outputs, logs, generated configurations, or metadata;
- prove `MarkStageStale` retains the prior details required for resume
validation;
- change the forced-extraction-failure lifecycle fixture to first complete a
successful extraction, then force a failing replacement, and assert the
current extract record is failed with no inherited result payload;
- retain the assertion that succeeded downstream stages become stale;
- prove a successful replacement whose new result omits optional details does
not inherit details from its predecessor; and
- retain resume-error, repeated-self-skip, and ordinary-retry coverage.
Exit criteria:
- A current running, failed, or skipped session-stage record never advertises
result payload belonging to an earlier success.
- Historical run manifests and immutable extraction bundles remain available
for audit and recovery.
- `go test -count=1 ./internal/manifest ./internal/app ./internal/stage`,
focused lifecycle race tests, `go test ./...`, and `go vet ./...` pass.
## Stage 17: Establish Native Promotion Validation On Every Supported Platform
Turn the Linux, macOS, and Windows promotion claim into a continuously verified
contract. Cross-compilation remains useful but cannot substitute for executing
filesystem operations on each native platform.
Implementation:
1. Resolve the CI substrate and native runner labels in the open question below
before editing workflow files. Follow the repository host's established
workflow location and syntax; do not invent runner labels that cannot run.
2. Add checked-in CI jobs that execute on native Linux, macOS, and Windows
runners. Each native job must run at least:
- `go test -count=1 ./internal/fileops`;
- the platform-specific no-replace test; and
- the portable promotion conformance tests, including successful promotion,
collision preservation, cleanup, and the Stage 15 race-safe traversal
cases that apply on that platform.
3. Ensure the jobs exercise the real platform implementation rather than a
mocked rename function. A destination created before installation must win,
and the source and destination contents must prove that no replacement
occurred.
4. On Windows, ensure successful promotion exercises both `MoveFileEx` and the
directory-sync path. Do not treat permission or ordinary I/O failures as
unsupported-operation success.
5. Retain deterministic cross-compilation of the fileops test binary for Linux,
Darwin, and Windows. Write compiled test binaries to a temporary or CI
artifact directory, never the repository root.
6. Keep unsupported-platform compilation coverage, but do not misrepresent an
unexecuted cross-compiled test as native validation.
7. Record any required runner-specific limitation in this stage and in the
canonical contributor/CI documentation. Do not weaken production semantics
merely to accommodate an inadequately provisioned runner.
Validation:
- obtain a successful native Linux job;
- obtain a successful native macOS job;
- obtain a successful native Windows job;
- retain successful Linux, Darwin, and Windows cross-compilation; and
- confirm CI does not leave generated test binaries or application binaries in
the worktree or commit them as artifacts of the source tree.
Exit criteria:
- Atomic no-replace promotion and its directory-sync behavior execute
successfully on native Linux, macOS, and Windows.
- The checked-in CI configuration will rerun those tests on future changes.
- Stage 17 is not marked complete based solely on Linux execution and
cross-compilation.
## Stage 18: Reconcile Security Documentation And Perform Final Validation
Close the remediation only after the corrected filesystem and manifest
invariants are implemented and native platform evidence exists.
Implementation:
1. Reconcile `docs/roadmap/notarius-extract-stage.md` with the Stage 15 source
traversal mechanism and Stage 16 current-result semantics. Keep normative
security and manifest guarantees in their existing canonical owners.
2. Update focused internal or contributor documentation only where the new
mechanism or CI workflow changes maintained implementation guidance. Do not
duplicate volatile platform commands across multiple documents.
3. Recheck the extraction, workspace, manifest, operations, troubleshooting,
integration, and testing documentation for contradictions introduced by the
remediation.
4. Keep completed roadmap history concise. Mark the target feature roadmap and
this implementation plan complete only after all Stage 15 through 17 exit
criteria are satisfied.
5. Confirm that no secrets, private campaign material, generated binaries,
platform test binaries, or temporary bundle trees were added to the
repository.
Validation:
- run focused fileops, manifest, runner lifecycle, extraction, adapter,
artifact-catalog, analyze, publish, restore, and operator tests;
- run focused race tests covering fileops, manifest transitions, runner
lifecycle, extraction, artifacts, and publication;
- run `go test -count=1 ./...`;
- run `go vet ./...`;
- build `cmd/narratio` with `-o` targeting a temporary directory;
- rerun Linux, Darwin, and Windows fileops test cross-compilation;
- confirm the native CI jobs from Stage 17 are successful;
- run `git diff --check`; and
- verify the worktree contains no generated validation artifacts.
Exit criteria:
- The source traversal and current-manifest replacement gaps are closed by
focused regression tests.
- Native filesystem behavior substantiates the documented Linux, macOS, and
Windows support boundary.
- Current documentation matches the remediated implementation.
- Every progress row is `Complete`, the repository-wide validation suite
passes, and the target feature roadmap can truthfully remain complete.
## Open Questions
### Which CI system and native runner labels are authoritative for this repository?
**Recommended approach:** Use Gitea Actions with a checked-in workflow under
`.gitea/workflows/`, provided the repository owner supplies or confirms native
Linux, macOS, and Windows runner labels. The repository is hosted on Gitea, so
keeping the workflow with the authoritative repository minimizes mirrored
configuration and makes the platform contract visible beside the code. Stage
17 must remain pending until all three native jobs have actually run.
**Viable alternative:** Use an existing external CI service with real native
capacity, such as a maintained GitHub Actions mirror or Buildkite installation,
and check its workflow or pipeline definition into the conventional repository
location. This is appropriate when that service already owns release gating or
when native macOS and Windows runners are unavailable in Gitea. The alternative
must still produce repeatable native results for all three platforms;
cross-compilation or an undocumented one-time manual run is not sufficient.