eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Record resilience implementation review findings

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-24 19:30:22 +00:00
parent 6523d7ac74
commit 9fc4de22b7
1 changed files with 159 additions and 249 deletions
Download Patch File Download Diff File Expand all files Collapse all files

408
docs/roadmap/remediation.md
View File

@@ -1,289 +1,199 @@
# Library Adoption Review: Required Remediation
# Resilience Review: Required Remediation
**Reviewed:** 2026-09-24, source commit `7cefb3f`.
**Scope:** The twelve implementation commits after `1ad2119`, compared with the
[implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md), [combined target](library-adoption.md),
and [configuration target](configkit.md).
**Status:** R1–R5 addressed and verified on 2026-09-24. The evidence in the
original findings below describes source commit `7cefb3f`; line references are
historical. See the resolution record at the end for the implemented fixes.
**Reviewed:** 2026-09-24, source commit `6523d7a`.
**Scope:** The fourteen implementation commits after `8ffbf41`, compared with
[implementation.md](implementation.md), [resilience.md](resilience.md), and the
current architecture and output contracts.
**Status:** Five open findings, R6–R10. This review implements no fixes.
The earlier library-adoption findings R1–R5 were resolved before this work.
Their findings and resolution evidence remain in the
[previous remediation document](https://gitea.maximumdirect.net/eric/audita/src/commit/6523d7a/docs/roadmap/remediation.md).
Numbering continues here to distinguish new work from those resolved findings.
## Architecture Assessment
The intended ownership split is substantially implemented. ConfigKit v0.4.0 owns
schema-2 loading, overlays, presence/default resolution, explicit overrides,
credentials, and configuration inspection. One shared PromptKit v0.11.0 engine
owns both roles' profiles, embedded prompt rendering, prepared execution, provider
transport, retry/repair behavior, and per-backend admission. Audita retains domain
configuration, CLI composition, ordered modules, deterministic validators,
proposal/application checks, validation batching/cardinality, reporting, and
artifact lifecycle. The root module requires Go 1.25.5 and has no local replacement.
The intended architecture is substantially implemented:
Legacy loaders, provider transport, template rendering, global/role schedulers,
and the isolated prototype were removed. All nine prompt definitions preserve
the original instruction text after accounting for the new hardening variable
and whitespace. There is no need to revisit the library-adoption design or add
compatibility machinery. The findings below are concrete defects or unfulfilled
parts of the selected final contract.
- ConfigKit owns configuration loading, overlays, explicit overrides, and resolved
credentials. Reporting and debug are independent configuration selections.
- One shared PromptKit engine serves proposal and validation roles, owns bounded
retries/repairs and backend admission, and supplies execution facts. Audita's
adapter distinguishes recoverable model failures from fatal setup/internal
errors and caller cancellation.
- Audita normalizes and chunks once, retains section identity as text evolves,
preserves stage barriers, and validates proposals through the required chains.
Failed validation batches exclude their candidates while successful survivors
continue. Model errors do not cancel unrelated work; fatal errors cancel peers
and join workers.
- Processing-owned outcomes feed a separate report mapper and presentation DTOs.
The ordinary bundle contains versioned error/recovery YAML and fixed Markdown
matrices. The CLI uses the same counts; detailed persisted evidence is opt-in.
- The filesystem output session checks destinations, initializes reports, and
publishes transcript and reports through per-file replacement. CLI composition
owns streams, lifecycle, fatal finalization, and optional debug handling.
## R1 — P1: Redact Model-Derived Content in Every Retained Report Artifact
The ordinary resilience path is supported by the retained integration tests,
including total model failure, partial validation success, stable chunks,
recovery counting, and reporting/debug combinations. The findings below concern
specific boundary defects and incomplete evidence handling, not a need to
redesign the library ownership or introduce compatibility machinery.
**Category:** Incomplete implementation of Stage 10's redaction boundary.
## R6 — P1: Honor Restrictive Creation Permissions
**Evidence:** Both success and failure paths in
[`runProcess`](../../internal/cli/run.go#L311) write
`BuildCorrectionLedger(...)` directly through `RunDirectory.WriteJSONArtifact`.
The success-path write is at line 348. That writer serializes its input without
redaction. Validator decision/rejection messages contain provider-supplied reason
text. Redacting the separate process report does not protect the ledger.
**Category:** New filesystem/security regression in Stage 10.
A controlled HTTP subprocess reproduction returned a schema-valid negative
assessment whose reason contained the synthetic configured backend credential.
Processing succeeded and the process report was redacted, but
`correction-ledger.json` retained the credential verbatim. The existing provider-
error test does not exercise this successful, schema-valid response path.
**Evidence:** [`atomicReplace`](../../internal/core/diagnostics/output_session.go#L249)
creates a temporary file and then unconditionally changes its mode to `0644` for
an absent destination (lines 261–265). `Chmod` does not apply the process umask.
The previous transcript writer used ordinary file creation, which did apply it.
This affects both the published output and the temporary file while populated.
**Required change:** Apply the configured-secret redaction boundary to the ledger
and other model-derived diagnostic/report outputs before persistence. Audit the
new execution/utilization and selected-profile artifact paths for the same bypass;
metadata-only output is not a substitute for enforcing the selected secret-value
contract. Use all resolved backend credentials, including unselected backends.
Keep processing/domain data separate from its sanitized diagnostic representation.
Coordinate the shared redaction implementation with R2.
An actual CLI run under umask `0077`, with no LLM modules and a valid transcript,
returned success but created both `out.json` and `errors.yml` with mode `0644`,
rather than `0600`. In a traversable output directory, other users can therefore
read transcript content despite the operator's restrictive creation policy.
**Acceptance:** A controlled provider echoes synthetic credentials in a valid
assessment reason. Neither success nor partial-failure reports, ledgers, interaction
artifacts, or other affected diagnostic outputs contain those values. Include an
unselected backend credential. Preserve valid serialization and domain outcomes;
do not expand tests into exhaustive permutations of every artifact.
**Required change:** Keep standard per-file replacement, but honor ordinary
creation permissions for previously absent destinations. Preserve intended
existing-file permission behavior without blindly widening new files. Do not
read/change a process-global umask around concurrent application operations or
add a transaction/durability framework.
## R2 — P1: Remove the Report Redactor's Unredacted Fallback
**Acceptance:** A subprocess under a restrictive umask produces new transcript
and report files with appropriately restricted permissions. Existing restrictive
output permissions remain preserved after replacement. Keep the current cleanup
and late-write failure behavior. Test umask behavior in an isolated process.
**Category:** New code-quality/security defect in Stage 10.
## R7 — P2: Do Not Report Success Without Publishing a Transcript
**Evidence:** [`redactProcessReport`](../../internal/cli/run.go#L392) replaces secret
substrings in the serialized JSON document, then decodes it back into the report.
If decoding fails, it returns the original report. Replacements can corrupt JSON
syntax or typed values rather than just redact strings.
**Category:** New destination-selection inconsistency in Stages 10/12.
A focused reproduction loaded an otherwise valid config with the synthetic
credential `2`, then redacted a report containing `config_version: 2` and the string
`echo:2`. Replacing the numeric token made the JSON invalid; the helper returned
the original `echo:2`. Short numeric credentials are accepted by the selected
secret-reference contract, so the implementation cannot silently assume otherwise.
Quotes, escapes, overlapping credentials, and replacement order also need handling
at the serialization boundary.
**Evidence:** [`PrepareOutput`](../../internal/core/diagnostics/output_session.go#L53)
uses a trimmed path to decide whether a file destination exists, as does
[`selectedReportDir`](../../internal/cli/config_input.go#L215). However,
[`finish`](../../internal/cli/process.go#L377) selects stdout only when the original
path equals the empty string. Parsing does not reject a whitespace-only output
path. Consequently, neither branch publishes the transcript.
**Required change:** Redact string content without changing JSON structure or
numeric/boolean types. Never return or write the original sensitive object when
redaction/serialization fails. Share the appropriate string-redaction logic across
report and artifact writers rather than maintaining incompatible versions. Handle
escaped content and overlapping values deliberately; retain typed error causes
without exposing provider error text.
A real invocation with `--output ' ' --reporting=false` returned exit 0 and the
successful completion message, wrote zero bytes to stdout, and created no output
file. This violates the central contract that successful processing produces a
transcript, independently of reporting selection.
**Acceptance:** Focused tests cover a numeric credential coinciding with report
numbers, quoted/backslash-containing credentials, and overlapping values. Output
remains valid with the original numeric/boolean meanings, secrets are removed,
and a redaction failure cannot emit the original object. Exercise the actual
external and run-local report writing paths with representative coverage.
**Required change:** Resolve file versus stdout once and use that decision
consistently throughout placement, preflight, publication, and reporting. Prefer
preserving a supplied nonempty filename literally, including legal whitespace,
with only the empty value selecting stdout. Do not silently turn a file request
into discarded output.
## R3 — P1: Cancel Remaining Work When a Validator Fails
**Acceptance:** Exercise a whitespace-only output filename through the actual CLI:
success must mean the requested file exists and contains the encoded transcript.
Omitted output still writes the transcript to stdout, and reports follow the same
resolved destination decision.
**Category:** Pre-existing runner behavior left inconsistent with the selected
Stage 8 final state and architecture policy; not newly introduced by PromptKit.
## R8 — P2: Redact Paths Printed to Stderr
**Evidence:** [`runModulePipeline`](../../internal/framework/runner/runner.go#L383)
puts validation errors into a buffered channel, but reads that channel only after
`vwg.Wait()` (lines 414–421). `setErr` therefore cannot cancel the shared context
in response to a validator failure until all validation workers have finished.
Proposal work can continue launching validation in the meantime.
**Category:** New report-location bypass plus an unclosed debug-location gap in
the Stage 11/12 redaction boundary.
A controlled two-section reproduction made one validator fail while its peer
waited for context cancellation or an explicit test release. The peer was not
canceled; the pipeline returned its error only after the peer was manually released.
In production, a failed validation operation can thus leave other queued or active
provider calls consuming capacity and money until they finish or reach their own
deadlines. Adapter-level cancellation tests do not cover this runner behavior.
**Evidence:** [`finish`](../../internal/cli/process.go#L398) prints report paths and
the debug directory without redaction. The debug locations printed on failure
at lines 365 and 394 do the same. Ordinary report renderers and error diagnostics
already have configured-secret redaction; these announcements bypass it.
**Required change:** Propagate validation failures to shared cancellation as soon
as they occur, while retaining the originating error, joining active workers, and
collecting available results deterministically. Use a coordinator or properly
synchronized failure state; simply calling the current `setErr` from workers would
also require protecting concurrent reads of `firstErr`. Do not add a scheduler,
outer retry loop, or compatibility layer.
A CLI reproduction used a synthetic configured credential also present in the
selected output-directory name. `report.md` correctly redacted the value, but
the successful run printed it verbatim in all three `Report:` lines on stderr.
Configured credentials may be short or coincide with ordinary metadata; their
redaction cannot depend on the value not appearing in a path.
**Acceptance:** A runner integration test with a failing validator and a blocked
peer proves cancellation arrives without releasing the peer manually. Cover queued
provider work and preservation of the originating failure/partial observations.
The affected runner/runtime tests pass under the race detector, and failure emits
no purportedly complete corrected transcript.
**Required change:** Apply the existing configured-secret string redactor to
path announcements on both success and failure, including debug locations and
unselected backend credentials. Leave actual filesystem paths and processing data
unchanged. No new redaction implementation is needed.
## R4 — P2: Separate Module Identity from Section-Specific Diagnostic Names
**Acceptance:** A controlled CLI workflow with synthetic credential-bearing report
and debug paths exposes no credential value in stderr, while files are created at
the real selected locations. Preserve typed/numeric report fields and transcript
output behavior.
**Category:** New observation/report attribution defect in Stages 9–10.
## R9 — P2: Retain Completed Validation Decisions on Fatal Return
**Evidence:** [`validateSectionCandidates`](../../internal/framework/runner/runner.go#L525)
passes `ModuleInstanceForStages` as the validation request's module identity. This
value includes `:section-NNNN`. The
[validator call](../../internal/framework/validators/llm_validators.go#L110) copies
it into `DiagnosticContext.ModuleInstance` and leaves `SectionIndex` unset.
[`BuildUtilization`](../../internal/framework/processreport/llm.go#L25) groups
these records by that field, so proposal and validation calls for one real module
are reported under different module identities.
**Category:** Incomplete integration of partial validator results in Stage 6.
The subprocess reproduction with one grammar module and one section produced:
**Evidence:** The LLM validator deliberately returns prior batch decisions with
its fatal error in
[`Validate`](../../internal/framework/validators/llm_validators.go#L182).
[`validateSectionCandidates`](../../internal/framework/runner/runner.go#L640)
appends the operation outcomes, but returns on error before incorporating
`vResult.Decisions`. The retained decisions only describe earlier validators,
not completed batches of the validator that was interrupted.
| Reported module instance | Proposal calls | Validation calls |
| --- | ---: | ---: |
| `grammar` | 1 | 0 |
| `grammar:section-0000` | 0 | 1 |
A focused runner probe supplied one completed negative decision and one successful
operation alongside a later fatal error. The runner retained one operation and
zero decisions, while correctly returning the fatal cause and no approved
candidates. The retained validator test already proves that successful earlier
batch decisions reach this boundary; they are lost at the next layer.
The latter is not a configured module instance. Its timing cannot join the real
module result, and the actual module incorrectly reports zero validation calls.
Multi-section runs multiply these spurious entries. Explicit section attribution
is also lost from validation call/event records.
As a result, debug ledgers can label a proposal as merely interrupted even when a
validator already rejected it, and omit the completed decision and its rationale.
This conflicts with the selected partial-evidence/debug contract. It does not
currently cause unsafe corrections to be applied.
**Required change:** Carry actual module instance and section index separately
through the validator request and structured-call context. Keep section-qualified
artifact names as a separate naming concern so files remain distinct. Group counts
and timing by the actual module instance. Update tests that currently use batch
index as a substitute for section identity.
**Required change:** Preserve trustworthy completed partial decisions as diagnostic
evidence when a validator returns a fatal error. Keep identity/duplicate checks
appropriate to a partial result, without requiring decisions for unattempted
candidates. Do not turn partial approval into permission to apply a candidate or
weaken fatal cancellation and worker joining.
**Acceptance:** A multi-section workflow with repeated module instances yields one
utilization entry per actual instance, correct proposal/validation totals, and
explicit section/batch identity in validation observations. Artifact names remain
unique and correction-ledger ordering is unchanged.
**Acceptance:** A runner-level multi-batch workflow with completed decisions
followed by fatal failure/cancellation retains those decisions in the final debug
ledger, retains operation facts and the originating error, and publishes no
completed transcript or unchecked correction.
## R5 — P2: Index Execution Events Instead of Repeatedly Scanning Them
## R10 — P2: Surface Optional Interaction-Artifact Write Failures
**Category:** New report-building performance/maintainability defect in Stage 10.
**Category:** Pre-existing ignored errors left inconsistent with the selected
Stage 11 debug contract.
**Evidence:** [`BuildLLMExecution`](../../internal/framework/processreport/llm.go#L151)
scans every event for every call. `BuildUtilization` additionally tests every event
against a linear `hasCall` search for each backend's calls. With C calls and E
events, these paths perform O(C × E) work even for one backend. Retries and repairs
increase E. This work occurs after processing, before the CLI completes.
**Evidence:** Both
[`GenerateCandidates`](../../internal/framework/proposal_generation/generate.go#L148)
and [`Validate`](../../internal/framework/validators/llm_validators.go#L154)
discard the error from `WriteInteraction`. These errors never reach the CLI's
nonfatal debug diagnostics. The CLI also silently skips configuration-origin and
selected-profile serialization errors in `startDebug`.
A synthetic review probe using one terminal event per call measured:
A controlled local-provider reproduction enabled debug, then placed a regular file
at the module interaction-directory path before returning a valid empty proposal
response. Interaction writes failed, but the process printed only its success
summary and debug location, with no indication that requested evidence was missing.
Exit 0 and unchanged processing are correct; silent evidence loss is not.
| Calls/events | Execution mapping | Utilization mapping |
| ---: | ---: | ---: |
| 1,000 | 14 ms | 6 ms |
| 10,000 | 1.61 s | 0.52 s |
| 20,000 | 8.08 s | 2.42 s |
**Required change:** Carry optional debug failures to the CLI through a small,
explicit mechanism and emit redacted stderr diagnostics. Preserve nonfatal
behavior, keep failures out of model error/recovery counts, and retain paths only
for artifacts actually written. Handle the analogous serialization failures in
`startDebug`; do not introduce a logging framework or make debug a success gate.
These are local diagnostic measurements, not performance guarantees; the nested
scans establish the underlying growth independently of machine timing. Actual
snapshots usually contain more than one event per call.
**Acceptance:** A representative failed interaction write produces a safe stderr
diagnostic, while transcript, counts, and exit status match a run without that
failure. Cover both proposal and validator plumbing at their narrow boundaries;
one assembled CLI case is sufficient. Debug-off must still create no artifacts.
**Required change:** Index calls/events by call ID once and aggregate by backend
and actual module identity using direct lookups. Reuse terminal records for totals
without rescanning all events for each call. Preserve event/call ordering and
knownness/completeness semantics. Straightforward maps are sufficient; no reporting
framework or new persistent cache is needed.
## Verification and Review Limits
**Acceptance:** Existing report semantics remain intact, including preparation
failures, undispatched calls, and incomplete usage. A local benchmark or profiling
comparison demonstrates removal of the quadratic scans. Do not add fragile
wall-clock timing assertions to the default test suite.
- The full uncached offline suite passed: `go test -count=1 ./...`.
- `go vet ./...` passed.
- Focused race checks passed for runner, LLM, validators, process-report assembly,
diagnostics/output, reporting, and CLI subprocess packages.
- A temporary CLI binary and controlled local HTTP provider reproduced R6, R7,
R8, and R10. No live provider or paid service was used.
- A temporary runner probe reproduced R9; its expected evidence-preservation
assertion failed. The probe and all temporary artifacts were removed.
## Review Verification and Limits
The existing root `go test ./...`, focused race tests for LLM/runner/subprocess
packages, `go vet ./...`, and Linux amd64/arm64 builds passed. Temporary review
probes reproduced R1–R4 and measured R5; those probe files were removed afterward.
At review time, the retained suite passed despite these defects. Add durable regression
coverage with the fixes, using the narrowest useful production boundaries.
The initial review changed only this remediation document. No application fixes,
upstream changes, live provider calls, release publication, or compatibility work
were performed. Release-distribution compliance remains the explicitly deferred
release task and is not counted as an implementation defect here.
## Resolution Record
All five findings are resolved without compatibility layers, new scheduling,
or changes to library ownership.
### R1 and R2 — Shared Serialization Boundary
`core/redaction` now sanitizes decoded JSON strings and YAML string nodes,
retaining numeric/boolean types and avoiding mutation of domain objects.
Longest matching credentials take precedence in a single replacement pass;
JSON-escaped credential forms are handled for embedded response content.
Serialization or key-collision failures return no payload. The unredacted report
fallback has been removed.
The CLI supplies all resolved backend credentials to the external report writer
and run directory. Report copies, ledgers, execution/utilization artifacts,
interaction diagnostics, selected profiles, and error text share the redactor.
ConfigKit still owns credential-reference redaction of effective configuration.
Evidence:
- `TestSubprocessRedactsValidModelReasonsOnSuccessAndPartialFailure` exercises
schema-valid negative assessments containing selected and unselected secrets,
including quotes/backslashes. Both reports and retained artifacts are checked;
partial failure preserves earlier decisions and writes no corrected output.
- `TestReportWritersRedactStringsAndFailClosed` exercises both actual report
writers with credential `2`, preserving numeric configuration version 2 while
redacting string content, and verifies failed serialization writes no artifact.
- `TestStringRedactionPreservesSerializationAndDomainData` covers overlapping
secrets, escaped nested content, large numbers, booleans, YAML scalar types,
unchanged domain data, and failures without an original-payload fallback.
### R3 — Immediate Shared Cancellation
Validation workers record their error and cancel shared work immediately.
All accesses to the originating-error state use a mutex. Proposal collection
stops launching validation after a recorded failure; active workers are joined
and available results retain deterministic collection order.
`TestValidatorFailureCancelsActiveAndPendingProviderWork` uses the real PromptKit
runtime with backend concurrency one and queue capacity one. A failing validator
cancels two peer executions, one active in a controlled transport and one pending,
without manually releasing either. The originating error survives, only one
provider attempt occurs, and both logical calls retain finished observations.
The runner and runtime pass the race detector.
### R4 — Independent Module and Section Identity
Validation requests carry the configured module instance and an explicit section
index. The structured call forwards both; only interaction filenames combine
module, section, validator, and batch into a diagnostic name. The runner test
helper now uses section index rather than batch index to identify sections.
`TestSubprocessRepeatedModulesKeepSectionAttribution` runs two grammar instances
across two sections through the actual CLI/runtime. It verifies exactly two
utilization entries with two proposal and two validation calls each, explicit
section/batch identity, four distinct validation response artifacts, and unchanged
proposal ordering in rejected decisions.
### R5 — Indexed Report Aggregation
Execution mapping indexes events by call ID once. Utilization directly groups
calls/events by backend and indexes modules by real instance, seeded in module
result order. Neither builder scans all events for each logical call.
Existing summary tests and `TestIndexedReportsPreserveOrderAndPartialUsage`
verify pre-dispatch failures, original call/event order, backend grouping, and
known-but-incomplete usage. `BenchmarkExecutionReports` retains a reproducible
1,000/10,000/20,000-call benchmark without timing assertions.
A local three-iteration sample measured 20,000 calls at approximately **43 ms**
for execution mapping and **84 ms** for utilization, compared with the review
probe's 8.08 s and 2.42 s respectively. Timings are illustrative; the direct
indexes establish removal of the quadratic scans independently of machine load.
### Completion Validation
- Full offline suite: `go test ./...`.
- Race checks: runner, LLM runtime, process reporting, redaction, and CLI subprocess
packages.
- Static analysis: `go vet ./...`.
- Both required Linux architectures: `scripts/build-release.sh` in a temporary
output directory, without tagging or publishing.
- Documentation/diff whitespace checks.
Release publication and deferred release-distribution work remain outside this
remediation unit.
Passing retained tests do not cover the five failures above. Add proportionate
regression coverage with their fixes. This review changes only this document;
application fixes, upstream changes, deferred feature work, and release publication
remain outside scope. Target-platform release builds were not repeated for this
review; the native temporary CLI build succeeded.