Plan the computational efficiency implementation

docs/roadmap/implementation.md

@@ -0,0 +1,381 @@
+# Computational Efficiency Implementation Plan
+
+**Status:** Ready for implementation; all stages pending.
+
+## Purpose And Baseline
+
+This plan turns the [performance roadmap](performance.md) into ordered work for
+a Luna coding agent. The inspected baseline is commit
+`ff90510c380b47a23d0ac1cd8c314b1c4366a15b`; this planning change updates only
+the two roadmap documents. Recheck the worktree and relevant code before each
+stage because subsequent stages build on earlier changes.
+
+Execute Stage 1 through Stage 11 in order. Each stage is one focused
+implementation prompt. Complete its changes, applicable validation, canonical
+documentation updates, and local commit under the
+[agent policy](../policy/agents.md), then proceed. The sequence ends with a
+results report for review. Consumer pilots, hosted operations, and intermediate
+releases are outside this implementation sequence.
+
+Read [development.md](../development.md) and its applicable policy guide.
+Apply the [architecture policy](../policy/architecture.md),
+[testing policy](../policy/testing.md), and
+[documentation policy](../policy/documentation.md). Use graph tools for code
+discovery as instructed by `AGENTS.md`.
+
+## Selected Design And Scope
+
+| Area | Selected implementation |
+| --- | --- |
+| Child process | A small piped-process helper in `internal/process`, sharing existing command setup and error wrapping. |
+| Git blobs | One lazy `git cat-file --batch` session per candidate tree, with one request/response exchange at a time. |
+| Raw file bytes | Caller-owned bytes for regular-file reads; immutable symlink strings cached by object ID. |
+| Repeated checks | Compact formatting outcomes, parsed Markdown documents, and replacement summaries cached by object ID within their check. |
+| Artifact hashing | Data-only declarations followed by one existing local preflight verification; retain build hashing and missing-upload verification. |
+| Go output | Separate status and captured-output methods; transfer completed buffers without copying. |
+| Runtime version | A standard-library `sync.OnceValue` cache for the fallback, with an immediate injected-version return. |
+
+These are internal implementation choices. Preserve public facade signatures,
+configuration, CLI semantics, selected check ordering, callback sequencing,
+findings, failure causes, publication guards, and delivery recovery. Existing
+public contracts remain authoritative. New helpers expose only the internal
+operations needed here. File-size selection and existing supported inputs keep
+their current semantics.
+
+The Git session and checker caches belong to one validation operation. Existing
+sequential check execution remains sufficient for this design. The runtime
+fallback is immutable process data and uses the standard library's concurrent
+initialization mechanism.
+
+## Stage 1: Measure Candidate Content And Validation
+
+**Owners:** Benchmark files in `internal/validation` and root facade fixtures.
+
+1. Add benchmarks named with the `BenchmarkPerformance` prefix. Reuse local Git
+   fixture patterns; adapt shared helper parameters to `testing.TB` only where
+   a benchmark needs them.
+2. Measure content access and formatting for 1,000 unique regular Go blobs.
+   Include loose and packed objects and a repeated-object-ID case. Measure
+   Markdown checking with a 1 MiB document, repeated links to it, and identical
+   contents under different directories.
+3. Measure a complete facade `Check` using a temporary committed Go module with
+   a valid release note, Go formatting, local replacements, and Markdown selected.
+   Keep its Go suite real and offline. Prepare fixtures outside timed work.
+   Report allocations and bytes processed, separating content access, tree
+   metadata, and Go test/compile work.
+4. Add one diagnostic retained-memory observation after file processing, with
+   the candidate and document cache kept alive, to expose raw-content retention.
+   Keep GC/profile collection outside timing loops.
+5. Create a measurement record in this document for baseline and final results.
+   Include commit, Go/Git versions, fixture sizes, packed/loose state, and cache
+   state. Keep raw benchmark output or profiles in temporary storage.
+
+**Validation and completion:** Run these benchmarks with `-run '^$'`,
+`-bench '^BenchmarkPerformance'`, `-benchmem`, and `-count=3`. Verify the
+fixtures inspect the intended bytes. Synthetic modules use test-owned module
+caches under the developer guide. Timing ratios remain reported measurements;
+behavioral tests protect content, failures, and ownership. This stage changes
+measurement support only.
+
+## Stage 2: Measure Delivery, Output, And Runtime Lookup
+
+**Owners:** Benchmark files in `internal/delivery`, `internal/gotool`,
+`internal/runtimeinfo`, and root delivery fixtures.
+
+1. Extend the same benchmark prefix to declaration/preflight processing of two
+   deterministic 16 MiB artifact files. Measure prebuilt facade `Deliver`
+   against a local HTTP fixture for fresh uploads and matching retries.
+   Prepare files outside timed work and reset hosted state for each fresh run.
+   Stream binary bodies through the fixture's digest/size checks and serve
+   matching files incrementally so fixture buffering does not obscure allocation
+   improvements. Describe fixture processing costs in the workflow results.
+2. Benchmark completed Go-output capture with modest output and 8 MiB diagnostics,
+   including forwarding to `io.Discard`. Reuse the combined writer for component
+   measurements and existing child fixtures for any command measurements.
+3. Measure injected version access and repeated real metadata-fallback access.
+   Preserve benchmark coverage of initialization and repeated access when the
+   implementation later adds caching.
+4. Record time, allocations, bytes processed, and workload/cache settings in the
+   measurement record. Run the benchmark set with the same flags and repetitions
+   as Stage 1. Verify delivered bytes and matching-retry behavior.
+
+**Validation and completion:** Measurements are repeatable offline using local
+collaborators and temporary files. Fixture state changes occur outside timed
+work where they are setup. These two measurement stages establish the baseline
+before production refactoring begins. Default tests remain independent of real
+credentials and mutable external services.
+
+## Stage 3: Add The Piped Child-Process Boundary
+
+**Owner:** [internal/process](../../internal/process/process.go) and its tests.
+
+1. Factor existing command construction and command-error wrapping into small
+   private helpers shared by `Run` and a new internal `StartPiped` operation.
+   Preserve arguments, directory, environment overlay/removal, stderr routing,
+   context handling, and wrapped exit errors.
+2. `StartPiped` accepts an `Invocation` for command settings and stderr; stdin
+   and stdout are supplied by pipes. Return a small process handle containing
+   the input writer and output reader, plus `Cancel` and `Wait` operations.
+   Keep the underlying `exec.Cmd` private. Use a derived cancellable context so
+   the owner can abort a blocked exchange without cancelling its caller.
+3. Close acquired pipes on setup/start failure. The caller owns input closure,
+   stdout consumption, and one wait/reap after a successful start. `Cancel`
+   interrupts the child; `Wait` retains the current context/exit error semantics.
+   Read stdout to completion before a normal wait. Release the derived context
+   when the process is reaped; abort paths close active pipes before waiting.
+4. Extend the existing test executable helper for an exchange that returns
+   output before stdin EOF. Test real argument/environment propagation, normal
+   EOF completion, cancellation after a readiness signal, start failure, and
+   a nonzero child exit. Synchronize cancellation with the child instead of
+   depending on sleep duration.
+
+**Validation and completion:** Run `internal/process` tests. Existing `Run`
+behavior still passes, and the piped helper supports repeated exchanges followed
+by deterministic cleanup. Update the process inventory description when needed.
+
+## Stage 4: Implement The Git Batch Reader
+
+**Owners:** A focused reader file and tests under `internal/git`, reusing
+[source.go](../../internal/git/source.go).
+
+1. Add an internal `BlobReader` that stores a repository directory and parent
+   context. Create its process lazily on the first content request through
+   `process.StartPiped` with `git cat-file --batch`.
+2. Use the same child-only `GIT_NO_LAZY_FETCH=1` and `GIT_ALLOW_PROTOCOL=`
+   settings as `readLocalObjects`. Keep existing standalone `ReadBlob`,
+   `ReadCommitFile`, and batched metadata functions for their current callers.
+3. Implement `Read(ctx, objectID) ([]byte, error)` as one serialized exchange:
+   validate the ID using the existing helper, write its newline-terminated
+   request, read the response header, validate object/type/size, read exactly
+   the declared bytes, and consume the framing newline. Compare hexadecimal
+   IDs without narrowing the existing accepted case conventions.
+4. Allocate the response body once and transfer it to the caller. Use buffered
+   header reading and `io.ReadFull`; blob contents can contain arbitrary bytes
+   and newlines. Read only requested bodies and avoid whole-session buffering.
+   Size conversion errors reflect representability on the host.
+5. Treat Git's missing-object response as a contextual inspection error. Preserve
+   stderr and process causes when the child fails. A broken protocol or I/O
+   exchange aborts that session and reaps the child; return the original cause.
+6. Honor the read context during blocked I/O, for example by registering
+   `context.AfterFunc` to cancel the child and stopping that registration after
+   the exchange. Coordinate callback completion when the stop function reports
+   it has begun; stopping alone does not wait for that callback. If cancellation
+   races completion, preserve its cause and abort the session. Read captured
+   stderr after reaping to avoid writer races.
+7. Add idempotent `Close`: an unused reader has nothing to close; a healthy
+   reader closes stdin, finishes stdout consumption, and waits; an aborted
+   reader closes pipes, cancels, and waits. Preserve unexpected exit failures;
+   deliberate abort cleanup retains the original read failure.
+
+**Validation and completion:** Test repeated reads from loose and packed Git
+objects, empty/binary bodies, and missing objects. Reuse the partial-clone test
+approach to verify absent objects cause errors with no transport or object-store
+population. Use a small child fixture for truncated framing and cancellation
+mid-response. Assert content and cleanup outcomes; benchmark process reuse
+without making a subprocess count a public contract. Run Git and process tests.
+
+## Stage 5: Give Candidate Validation A Scoped Reader
+
+**Owners:** [tree.go](../../internal/validation/tree.go),
+[validation.go](../../internal/validation/validation.go), and tree fixtures.
+
+1. Give `CandidateTree` one lazy `BlobReader` and an internal `Close` method.
+   `ReadCandidateTree` still reads metadata only. Route blob misses through the
+   reader while temporarily retaining the existing byte-cache behavior.
+2. The shared validation executor owns closure on every return after loading
+   a tree. Complete cleanup before product checks or build callbacks run.
+   Metadata-only selections leave the content reader unstarted.
+3. Preserve an earlier check's `Failure` and cancellation cause during cleanup.
+   If cleanup reveals an unexpected child failure after successful checks,
+   return a contextual `Failure` attributed to the last content-reading check.
+   Track that check in the executor; normal intentional cancellation during
+   cleanup adds no independent failure classification.
+4. Update direct tree users in tests and benchmarks to register cleanup or close
+   explicitly. Update synthetic tree fixtures to support lazy reader creation.
+   Keep tree paths, source capture, symlink resolution, and check order intact.
+
+**Validation and completion:** Run validation, candidate, publication, and facade
+candidate tests. Cover early check failure and cancellation with cleanup, and
+preserve workspace/file-size metadata behavior. Re-run content benchmarks with
+reader startup and closure included in the content-operation timing. Update
+[validation mechanics](../internal/validation.md) for the landed lifecycle.
+
+## Stage 6: Release One-Use Source Bytes And Reuse Formatting Results
+
+**Owners:** `internal/validation/tree.go`, `validation.go`, and destination tests.
+
+1. Remove the tree's general raw-blob cache. A regular-file `ReadFile` returns
+   its newly read, caller-owned body without a cache copy. Its GoDoc describes
+   that ownership and the scoped reader; independently repeated raw reads can
+   request the same object again.
+2. Cache committed symlink targets as owned immutable strings keyed by object
+   ID. Add a private target-reading helper used by destination resolution and
+   module-symlink handling. Preserve original target bytes and traversal order.
+   Explicit `ReadFile` of a symlink still returns independently owned bytes.
+3. In `checkGoFormat`, cache the formatted/unformatted result for successfully
+   inspected objects by object ID. Release source and formatted bytes from
+   retained references after comparing them. Emit a finding at every selected
+   unformatted path, including duplicate-content paths. Parse/read errors still
+   stop with the current path and cause; check cancellation for each entry.
+4. Adapt tests that construct the old cache directly. Preserve observable
+   isolation: mutating returned bytes cannot alter later validation or reads.
+   Reuse destination-cycle and repeated-symlink tests for shared target strings.
+
+**Validation and completion:** Run validation tests and formatting/destination
+benchmarks. Confirm many unique Go files no longer remain as retained source
+buffers after formatting, and duplicate-content files retain per-path findings.
+Record the tradeoff for explicit repeated raw reads. Update validation mechanics
+with the new ownership and reuse choices.
+
+## Stage 7: Reuse Owned Markdown And Module Results
+
+**Owners:** [markdown.go](../../internal/validation/markdown.go),
+[markdown_check.go](../../internal/validation/markdown_check.go), and
+[local_replacements.go](../../internal/validation/local_replacements.go).
+
+1. Key parsed Markdown reuse by resolved regular blob object ID. Keep a path
+   lookup when useful for callers, backed by the shared document records.
+   Relative link resolution and findings use the current source path; a shared
+   parsed document stores content-relative locations, links, and anchors only.
+2. Give retained link destinations and titles their own string storage, using
+   `strings.Clone` at extraction where needed, so parser/source views cannot
+   keep a large raw body alive. Cache parsed records, release the AST and source
+   from retained references, and preserve heading collision rules, image behavior,
+   and use-site locations.
+3. Refactor module-declaration selection to identify the resolved regular entry
+   before reading its body. After successful parsing, cache a compact list of
+   filesystem replacements by resolved object ID: target string and source line.
+   Own retained target strings and release raw bytes and the parsed syntax tree
+   from retained references.
+4. Apply each cached replacement summary to the current declaration path when
+   constructing findings. Resolve each symlink declaration in its own path
+   context. On read or parse failure, preserve the current declaration's path
+   and underlying cause; preserve findings order and versioned-replacement rules.
+
+**Validation and completion:** Run Markdown/module and facade validation tests.
+Add behavioral cases for identical Markdown in different directories and two
+module declarations resolving to one object, including multiple findings with
+correct paths/lines. Reuse parser and boundary/cycle coverage. Re-run
+large-document and repeated-content benchmarks and retained-memory observations.
+Update the validation document's cache description to match implemented ownership.
+
+## Stage 8: Consolidate Delivery Preflight Hashing
+
+**Owners:** [delivery declarations](../../internal/delivery/assets.go),
+[delivery orchestration](../../internal/delivery/delivery.go), and their tests.
+
+1. Make `declaredBuildAssets` perform structural declaration checks, stat files,
+   reuse the recorded artifact SHA-256, and construct expected assets and the
+   manifest. Remove its file-opening/hash pass. Keep target, filename, regular
+   file, declared digest format, and manifest validation at the appropriate
+   existing owner.
+2. Keep one `gitea.VerifyLocalAssets` call in `Deliver` after declaration and
+   build-identity checks and before hosted requests. This verifies actual bytes
+   and sizes against recorded identities for both managed and prebuilt assets.
+3. Keep builder hashing, CLI receipt verification, and missing-upload preflight
+   in their current phases. Managed delivery reuses its fresh build metadata for
+   declarations; prebuilt delivery reuses its caller's recorded digests only as
+   expected identities until current bytes have been verified.
+4. Move the declaration helper's tampering assertion to delivery preflight's
+   behavioral owner. Verify same-size changed contents fail before any hosted
+   request. Reuse source-only, retained-prebuilt, managed cleanup, conflict,
+   partial-upload, and retry coverage.
+
+**Validation and completion:** Run delivery, Gitea, facade delivery, and CLI
+receipt/delivery tests. Re-run artifact and complete delivery benchmarks. A fresh
+missing binary has three hash passes including a managed build, or two within
+prebuilt delivery; these are explanatory measurements of the chosen flow.
+Streaming upload, exact remote comparison, and later upload verification remain
+intact. Update the inventory's delivery mechanics where needed.
+
+## Stage 9: Transfer Completed Go Output Without Copies
+
+**Owners:** [gotool.go](../../internal/gotool/gotool.go), validation/build callers,
+and Go tool fixture tests.
+
+1. Give `Tool` a status-returning `Run(...args) error` and an output-returning
+   `Output(...args) ([]byte, error)`, with one private execution path. Keep their
+   existing directory, mode, target, writer, and argument inputs.
+2. On completion, a status-only success releases its capture. An output-consuming
+   call receives the completed buffer's backing bytes with ownership transferred
+   from the private writer. Replace `combinedWriter.Bytes` copying with a small
+   synchronized ownership-transfer operation used after child output has finished.
+3. Preserve complete failure diagnostics, original causes, combined-output
+   synchronization, and forwarding errors. If `Output` fails, return its captured
+   bytes and diagnostic error as before. Status failures format the same useful
+   diagnostics without creating an unnecessary intermediate snapshot.
+4. Convert validation test/vet/compile and artifact-build calls to status-only
+   `Run`. Use `Output` for compiler identification. Adapt preparation failure
+   formatting to the same completed-buffer ownership discipline, preserving its
+   network-capable environment. Leave configuration discovery's separate stdout
+   JSON and stderr handling intact.
+5. Keep full capture in memory. Measure remaining large-log retention and record
+   the disk-backed alternative as follow-up evidence, as scoped by the roadmap.
+
+**Validation and completion:** Extend existing process/Go fixtures for successful
+capture, large failure output, cancellation, and a failing output writer. Verify
+forwarded bytes and meaningful error content/cause. Run Go tool, validation,
+build, preparation, and affected CLI tests. Re-run output benchmarks and update
+[Go execution mechanics](../internal/gotool.md).
+
+## Stage 10: Cache The Runtime Metadata Fallback
+
+**Owners:** [runtimeinfo/version.go](../../internal/runtimeinfo/version.go) and
+runtime/buildinfo tests.
+
+1. Preserve `injectedVersion` and its linker identity. `Version` checks and
+   returns injection first, then calls a fallback accessor created with
+   `sync.OnceValue`.
+2. Keep the existing pure `version(injected, readBuildInfo)` selection helper.
+   Add a small private fallback factory accepting the metadata reader and
+   returning the cached accessor. Production supplies `debug.ReadBuildInfo`;
+   tests create independent cache instances with controlled metadata.
+3. Reuse selection tests. Add repeated/concurrent access coverage with a
+   counting reader to verify reuse and stable results. Test unavailable and
+   development metadata fallback through isolated instances. Keep shared
+   production state immutable during tests.
+
+**Validation and completion:** Run runtime/buildinfo tests, the existing consumer
+build-and-run injection test, and focused race validation of the fallback cache.
+Run facade/dependency checks. Re-run version benchmarks after initializing the
+cache and report initialization separately from repeated access. The runtime
+facade and its dependency boundary retain their existing public shape.
+
+## Stage 11: Verify The Complete Implementation And Record Results
+
+**Owners:** Affected tests and canonical internal documentation; roadmap status.
+
+1. Run the complete offline test suite and compiled facade/dependency checks
+   using the [developer guide](../development.md). Resolve remaining failures
+   within this performance scope. Reuse targeted race results where code is
+   unchanged; broaden them only for a concrete remaining concurrency risk.
+2. Run the Stage 1 and Stage 2 benchmark set again with comparable toolchain,
+   fixtures, cache state, repetitions, and measurement boundaries. Compare
+   component and complete `Check`/`Deliver` results. Record memory and explicit
+   reread tradeoffs as well as time and allocation improvements.
+3. Record final results alongside the baseline in this document. Explain the
+   remaining Go compilation/test cost, remote-comparison work, and diagnostic
+   retention where measurements show they matter. Separate measured effects
+   from expectations and include machine/workload limitations.
+4. Check canonical internal docs describe landed reader lifetime, source/result
+   ownership, verification flow, output ownership, and runtime reuse. Preserve
+   existing public contracts; update a public document only if its explanation
+   became inaccurate. Check local links and prose whitespace.
+5. Mark each stage's actual completion, update the feature roadmap status, and
+   commit the completed work under repository policy. End with a concise report
+   of changes, tests, performance results, and remaining follow-up opportunities.
+   Retain these planning documents with their results for the final review.
+
+**Completion evidence:** All five target improvements are implemented, relevant
+existing behavior passes, the public facade checks pass, comparable measurements
+are recorded, and the worktree is clean. Completion depends on observed behavior
+and reviewed diffs; benchmarks establish improvements within their measured
+scope without adding numerical runtime guarantees.
+
+## Open Questions
+
+None required for this implementation. The plan selects scoped batch reading,
+result-based reuse, preserved verification boundaries, in-memory diagnostics,
+and cached runtime fallback. Disk-backed diagnostics remain a follow-up option
+whose benefit and lifecycle costs can be assessed from the recorded output
+measurements.