eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Record unnecessary restrictions found in the resilience review

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-25 00:30:23 +00:00
parent ec78b5a5db
commit efaefdbf3d
1 changed files with 116 additions and 207 deletions
Download Patch File Download Diff File Expand all files Collapse all files

323
docs/roadmap/remediation.md
View File

@@ -1,241 +1,150 @@
# Resilience Review: Required Remediation # Resilience Review: Required Remediation
**Reviewed:** 2026-09-24, source commit `6523d7a`. **Reviewed:** 2026-09-25, source commit `ec78b5a`.
**Scope:** The fourteen implementation commits after `8ffbf41`, compared with **Scope:** Review of the resilience implementation for unintentionally strengthened
[implementation.md](implementation.md), [resilience.md](resilience.md), and the invariants, contracts, guarantees, and test requirements, compared with
current architecture and output contracts. [resilience.md](resilience.md), [implementation.md](implementation.md), and the
**Status:** R6–R10 addressed and verified. The findings below describe the reviewed [agent policy](../policy/agents.md).
source; their line references are historical. See the resolution record below. **Status:** R11–R14 remain open. This document records required follow-up work;
no application behavior has changed as part of this review.
The earlier library-adoption findings R1–R5 were resolved before this work. The earlier findings R1–R10 have been addressed. Their findings and resolution
Their findings and resolution evidence remain in the records remain in Git history; R6–R10 were resolved in `ec78b5a`.
[previous remediation document](https://gitea.maximumdirect.net/eric/audita/src/commit/6523d7a/docs/roadmap/remediation.md).
Numbering continues here to distinguish new work from those resolved findings. Numbering continues here to distinguish new work from those resolved findings.
## Architecture Assessment ## Architecture Assessment
The intended architecture is substantially implemented: The intended resilience architecture remains substantially implemented:
- ConfigKit owns configuration loading, overlays, explicit overrides, and resolved - ConfigKit owns configuration loading and credentials; PromptKit owns transport,
credentials. Reporting and debug are independent configuration selections. bounded retries/repairs, and backend admission.
- One shared PromptKit engine serves proposal and validation roles, owns bounded - Audita normalizes and chunks once, preserves section identity and stage barriers,
retries/repairs and backend admission, and supplies execution facts. Audita's and validates corrections before applying them. Recoverable model failures stay
adapter distinguishes recoverable model failures from fatal setup/internal local; fatal failures cancel peers and join workers.
errors and caller cancellation. - Processing outcomes feed separate report mapping and presentation. Ordinary
- Audita normalizes and chunks once, retains section identity as text evolves, YAML reports and Markdown matrices share counts with the CLI summary; debug
preserves stage barriers, and validates proposals through the required chains. persistence is independently selectable.
Failed validation batches exclude their candidates while successful survivors - The output session prepares destinations, initializes reports, and publishes
continue. Model errors do not cancel unrelated work; fatal errors cancel peers required outputs. Optional debug failures remain nonfatal and visible.
and join workers.
- Processing-owned outcomes feed a separate report mapper and presentation DTOs.
The ordinary bundle contains versioned error/recovery YAML and fixed Markdown
matrices. The CLI uses the same counts; detailed persisted evidence is opt-in.
- The filesystem output session checks destinations, initializes reports, and
publishes transcript and reports through per-file replacement. CLI composition
owns streams, lifecycle, fatal finalization, and optional debug handling.
The ordinary resilience path is supported by the retained integration tests, The findings below concern unnecessary restrictions and incidental test contracts,
including total model failure, partial validation success, stable chunks, not missing resilience machinery. Some restrictions were introduced by the
recovery counting, and reporting/debug combinations. The findings below concern implementation plan itself, rather than by a coding agent deviating from it.
specific boundary defects and incomplete evidence handling, not a need to Preserve validation-before-application, fatal cancellation, required-report
redesign the library ownership or introduce compatibility machinery. initialization and finalization checks, ordinary file permissions, configured-
secret redaction, and independent debug selection while addressing these findings.
## R6 — P1: Honor Restrictive Creation Permissions ## R11 — P2: Allow Intentional In-Place Transcript Output
**Category:** New filesystem/security regression in Stage 10. **Category:** Overly broad filesystem collision restriction.
**Evidence:** [`atomicReplace`](../../internal/core/diagnostics/output_session.go#L249) **Evidence:** [`PrepareOutput`](../../internal/core/diagnostics/output_session.go)
creates a temporary file and then unconditionally changes its mode to `0644` for compares every active destination against every protected input path and rejects
an absent destination (lines 261–265). `Chmod` does not apply the process umask. all overlaps, including aliases. The CLI includes the transcript input among
The previous transcript writer used ordinary file creation, which did apply it. those protected paths. Collision tests in
This affects both the published output and the temporary file while populated. [`output_session_test.go`](../../internal/core/diagnostics/output_session_test.go)
enforce rejection for direct, symbolic-link, and hard-link aliases.
An actual CLI run under umask `0077`, with no LLM modules and a valid transcript, The required protection was to avoid destroying inputs during preflight or early
returned success but created both `out.json` and `errors.yml` with mode `0644`, report initialization. It did not require forbidding intentional replacement of
rather than `0600`. In a traversable output directory, other users can therefore the transcript after it has been read and successfully processed. The broad
read transcript content despite the operator's restrictive creation policy. prohibition appeared in the implementation plan and should not become a lasting
product restriction solely on that basis.
**Required change:** Keep standard per-file replacement, but honor ordinary **Required change:** Distinguish transcript publication from early report writes.
creation permissions for previously absent destinations. Preserve intended Allow intentional in-place transcript output after successful processing, while
existing-file permission behavior without blindly widening new files. Do not retaining checks for destructive report/input collisions and incompatible output
read/change a process-global umask around concurrent application operations or destinations. Keep the change proportional; no filesystem transaction or
add a transaction/durability framework. concurrent-host-mutation protection framework is needed. Update affected contract
documentation and tests when the behavior changes.
**Acceptance:** A subprocess under a restrictive umask produces new transcript **Acceptance:** A representative CLI workflow can read and successfully replace
and report files with appropriately restricted permissions. Existing restrictive its input transcript. Preparation does not truncate the source, and failed
output permissions remain preserved after replacement. Keep the current cleanup processing does not publish a replacement. Alias checks do not impose a blanket
and late-write failure behavior. Test umask behavior in an isolated process. ban on otherwise valid in-place output. Required report destinations still cannot
destroy inputs during initialization or conflict with transcript publication.
## R7 — P2: Do Not Report Success Without Publishing a Transcript ## R12 — P2: Protect Only Profiles That Were Actually Loaded
**Category:** New destination-selection inconsistency in Stages 10/12. **Category:** Speculative protected-path restriction.
**Evidence:** [`PrepareOutput`](../../internal/core/diagnostics/output_session.go#L53) **Evidence:** [`selectedProfilePaths`](../../internal/cli/process.go) constructs
uses a trimmed path to decide whether a file destination exists, as does both `.yaml` and `.yml` paths for each selected profile, regardless of which file
[`selectedReportDir`](../../internal/cli/config_input.go#L215). However, was loaded or whether the alternate exists. These paths enter the output-session
[`finish`](../../internal/cli/process.go#L377) selects stdout only when the original protected-input set alongside actual configuration sources.
path equals the empty string. Parsing does not reject a whitespace-only output
path. Consequently, neither branch publishes the transcript.
A real invocation with `--output ' ' --reporting=false` returned exit 0 and the As a result, an unused alternate profile filename is reserved as though it were
successful completion message, wrote zero bytes to stdout, and created no output an input. This exceeds the intended protection of known loaded sources and can
file. This violates the central contract that successful processing produces a reject an otherwise valid output selection.
transcript, independently of reporting selection.
**Required change:** Resolve file versus stdout once and use that decision **Required change:** Derive protected profile paths from actual loaded sources,
consistently throughout placement, preflight, publication, and reporting. Prefer using the existing profile-loading boundary rather than reserving hypothetical
preserving a supplied nonempty filename literally, including legal whitespace, alternates. Retain protection appropriate to real inputs and the publication
with only the empty value selecting stdout. Do not silently turn a file request phase, consistent with R11. Do not introduce a second profile-discovery policy.
into discarded output.
**Acceptance:** Exercise a whitespace-only output filename through the actual CLI: **Acceptance:** An unused alternate profile filename is not rejected solely
success must mean the requested file exists and contains the encoded transcript. because its stem matches a selected profile. The actual loaded profile remains
Omitted output still writes the transcript to stdout, and reports follow the same protected from destructive early report writes. Configuration-source protection
resolved destination decision. continues to use actual loaded sources.
## R8 — P2: Redact Paths Printed to Stderr ## R13 — P2: Do Not Limit Transcript Destinations to Regular Files
**Category:** New report-location bypass plus an unclosed debug-location gap in **Category:** New output-interface restriction introduced by replacement mechanics.
the Stage 11/12 redaction boundary.
**Evidence:** [`finish`](../../internal/cli/process.go#L398) prints report paths and **Evidence:** [`preflightTranscript`](../../internal/core/diagnostics/output_session.go)
the debug directory without redaction. The debug locations printed on failure rejects existing destinations that are not regular files. The previous ordinary
at lines 365 and 394 do the same. Ordinary report renderers and error diagnostics transcript writer did not impose that restriction, allowing destinations such as
already have configured-secret redaction; these announcements bypass it. `/dev/null` and named pipes. Per-file atomic replacement is useful for regular
files, but does not justify narrowing every transcript destination to that kind
of file.
A CLI reproduction used a synthetic configured credential also present in the **Required change:** Separate regular-file replacement from writing to supported
selected output-directory name. `report.md` correctly redacted the value, but stream destinations. Preserve normal write-error handling and directory rejection;
the successful run printed it verbatim in all three `Report:` lines on stderr. do not attempt to rename over a device or pipe. Apply preparation checks relevant
Configured credentials may be short or coincide with ordinary metadata; their to the selected destination, without requiring regular-file replacement machinery
redaction cannot depend on the value not appearing in a path. for stream output. Keep ordinary report-file handling unchanged.
**Required change:** Apply the existing configured-secret string redactor to **Acceptance:** Transcript output to `/dev/null` succeeds without replacing the
path announcements on both success and failure, including debug locations and device. A representative named-pipe workflow can deliver the transcript to its
unselected backend credentials. Leave actual filesystem paths and processing data reader without replacing the pipe. Preparation does not open a pipe merely to
unchanged. No new redaction implementation is needed. probe it and thereby unnecessarily wait for a reader. Regular-file publication,
permission handling, directory rejection, and real write-failure reporting remain
covered by focused tests.
**Acceptance:** A controlled CLI workflow with synthetic credential-bearing report ## R14 — P2: Test Markdown Semantics Without Freezing Incidental Prose
and debug paths exposes no credential value in stderr, while files are created at
the real selected locations. Preserve typed/numeric report fields and transcript
output behavior.
## R9 — P2: Retain Completed Validation Decisions on Fatal Return **Category:** Over-specified presentation tests and redundant assertions.
**Category:** Incomplete integration of partial validator results in Stage 6. **Evidence:** [`markdown_test.go`](../../internal/core/reporting/markdown_test.go)
compares the complete rendered report byte-for-byte with a golden file, then
checks overlapping exact fragments. Additional assertions depend on exact prose.
This treats incidental wording and whitespace as durable contracts beyond the
selected requirements for complete fixed matrices, ordering, counts, states,
escaping, and readability.
**Evidence:** The LLM validator deliberately returns prior batch decisions with **Required change:** Prefer semantic assertions for the intended report contract,
its fatal error in or limit a retained layout golden to the table structure whose layout matters.
[`Validate`](../../internal/framework/validators/llm_validators.go#L182). Remove redundant assertions and exact-prose requirements that do not protect
[`validateSectionCandidates`](../../internal/framework/runner/runner.go#L640) meaningful behavior. Do not weaken the selected fixed-layout reporting contract
appends the operation outcomes, but returns on error before incorporating or introduce adaptive formatting as part of this cleanup.
`vResult.Decisions`. The retained decisions only describe earlier validators,
not completed batches of the validator that was interrupted.
A focused runner probe supplied one completed negative decision and one successful **Acceptance:** Tests still detect missing or misordered stage/chunk rows and
operation alongside a later fatal error. The runner retained one operation and columns, incorrect error/recovery counts or states, and broken table escaping.
zero decisions, while correctly returning the fatal cause and no approved Harmless prose or whitespace edits outside the required layout no longer require
candidates. The retained validator test already proves that successful earlier updating a whole-document fixture and overlapping assertions. Retain meaningful
batch decisions reach this boundary; they are lost at the next layer. agreement checks between YAML, Markdown, and CLI totals.
As a result, debug ledgers can label a proposal as merely interrupted even when a
validator already rejected it, and omit the completed decision and its rationale.
This conflicts with the selected partial-evidence/debug contract. It does not
currently cause unsafe corrections to be applied.
**Required change:** Preserve trustworthy completed partial decisions as diagnostic
evidence when a validator returns a fatal error. Keep identity/duplicate checks
appropriate to a partial result, without requiring decisions for unattempted
candidates. Do not turn partial approval into permission to apply a candidate or
weaken fatal cancellation and worker joining.
**Acceptance:** A runner-level multi-batch workflow with completed decisions
followed by fatal failure/cancellation retains those decisions in the final debug
ledger, retains operation facts and the originating error, and publishes no
completed transcript or unchecked correction.
## R10 — P2: Surface Optional Interaction-Artifact Write Failures
**Category:** Pre-existing ignored errors left inconsistent with the selected
Stage 11 debug contract.
**Evidence:** Both
[`GenerateCandidates`](../../internal/framework/proposal_generation/generate.go#L148)
and [`Validate`](../../internal/framework/validators/llm_validators.go#L154)
discard the error from `WriteInteraction`. These errors never reach the CLI's
nonfatal debug diagnostics. The CLI also silently skips configuration-origin and
selected-profile serialization errors in `startDebug`.
A controlled local-provider reproduction enabled debug, then placed a regular file
at the module interaction-directory path before returning a valid empty proposal
response. Interaction writes failed, but the process printed only its success
summary and debug location, with no indication that requested evidence was missing.
Exit 0 and unchanged processing are correct; silent evidence loss is not.
**Required change:** Carry optional debug failures to the CLI through a small,
explicit mechanism and emit redacted stderr diagnostics. Preserve nonfatal
behavior, keep failures out of model error/recovery counts, and retain paths only
for artifacts actually written. Handle the analogous serialization failures in
`startDebug`; do not introduce a logging framework or make debug a success gate.
**Acceptance:** A representative failed interaction write produces a safe stderr
diagnostic, while transcript, counts, and exit status match a run without that
failure. Cover both proposal and validator plumbing at their narrow boundaries;
one assembled CLI case is sufficient. Debug-off must still create no artifacts.
## Verification and Review Limits ## Verification and Review Limits
- The full uncached offline suite passed: `go test -count=1 ./...`. - This was a source and contract review; no application changes or new runtime
- `go vet ./...` passed. reproductions were made during this pass.
- Focused race checks passed for runner, LLM, validators, process-report assembly, - The preceding remediation passed the full Go test suite, vet, focused race
diagnostics/output, reporting, and CLI subprocess packages. checks, and Linux amd64/arm64 builds. Those checks were not repeated for this
- A temporary CLI binary and controlled local HTTP provider reproduced R6, R7, documentation-only review and do not establish that R11–R14 are resolved.
R8, and R10. No live provider or paid service was used. - Several current tests deliberately enforce the restrictions identified above;
- A temporary runner probe reproduced R9; its expected evidence-preservation passing them does not establish that those restrictions were intended.
assertion failed. The probe and all temporary artifacts were removed. - The prior race checks and target-platform builds were proportionate to the
resilience refactor. They should not be promoted into mandatory gates for every
At review time, the retained tests did not cover the five failures above. subsequent edit solely because they were used there.
The initial review changed only this document; - Application fixes, upstream changes, deferred feature work, and release
application fixes, upstream changes, deferred feature work, and release publication publication remain outside this document update.
remain outside scope. Target-platform release builds were not repeated for this
review; the native temporary CLI build succeeded.
## Resolution Record
All five findings are addressed without changing retry budgets, library ownership,
correction acceptance, or the independent reporting/debug selections.
- **R6:** Per-file replacement uses ordinary file creation inside a private
temporary directory in the destination parent. Creation applies the umask;
existing permission bits are restored only for existing destinations. Temporary
files/directories are removed on success and failure. The subprocess permission
test covers both restrictive and ordinary masks and subsequent replacement of
restrictive files; existing output-session failure/cleanup tests still pass.
- **R7:** File selection uses the literal nonempty output path throughout report
placement and output preparation. Final CLI stream selection uses the prepared
output session. `TestSubprocessWhitespaceOutputIsALiteralFile` proves the output
and reports are produced; the existing stdout integration test remains passing.
- **R8:** Every CLI report/debug location announcement uses the shared configured-
secret string redactor. `TestSubprocessPathAnnouncementsAreRedacted` covers
success and failure with selected and unselected credentials in real paths,
while confirming that the real destinations are used.
- **R9:** Fatal validator returns use partial identity/cardinality validation,
retain completed decisions and rejection details, and never return application
approvals. `TestFatalBatchRetainsEarlierDecisionsInLedger` exercises the real
runner and batching validator with two completed batches and a third fatal
failure, retaining their decisions and all operation facts in the debug ledger.
A separate runner regression rejects unknown/duplicate partial identities while
retaining the originating failure. Existing cancellation/joining tests pass.
- **R10:** An explicit diagnostic-error callback connects proposal and validator
writers to the CLI's synchronized, redacted nonfatal diagnostics. Origin/profile
serialization failures use the same path. Narrow proposal/validator tests and
`TestSubprocessInteractionWriteFailureIsVisibleAndNonfatal` verify visible
failures, unchanged transcript/counts/exit status, and no fabricated artifact
paths. Existing reporting/debug-selection tests verify debug-off persistence
remains disabled.
Validation completed: full `go test ./...`, `go vet ./...`, focused race tests for
runner, validators, proposal generation, report assembly, diagnostics/output, CLI,
and subprocess integration; required Linux amd64/arm64 builds in a temporary
output directory; and documentation link/whitespace checks. No live provider,
upstream change, tag, or publication was involved.