Record resilience implementation review findings
This commit is contained in:
@@ -1,289 +1,199 @@
|
|||||||
# Library Adoption Review: Required Remediation
|
# Resilience Review: Required Remediation
|
||||||
|
|
||||||
**Reviewed:** 2026-09-24, source commit `7cefb3f`.
|
**Reviewed:** 2026-09-24, source commit `6523d7a`.
|
||||||
**Scope:** The twelve implementation commits after `1ad2119`, compared with the
|
**Scope:** The fourteen implementation commits after `8ffbf41`, compared with
|
||||||
[implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md), [combined target](library-adoption.md),
|
[implementation.md](implementation.md), [resilience.md](resilience.md), and the
|
||||||
and [configuration target](configkit.md).
|
current architecture and output contracts.
|
||||||
**Status:** R1–R5 addressed and verified on 2026-09-24. The evidence in the
|
**Status:** Five open findings, R6–R10. This review implements no fixes.
|
||||||
original findings below describes source commit `7cefb3f`; line references are
|
|
||||||
historical. See the resolution record at the end for the implemented fixes.
|
The earlier library-adoption findings R1–R5 were resolved before this work.
|
||||||
|
Their findings and resolution evidence remain in the
|
||||||
|
[previous remediation document](https://gitea.maximumdirect.net/eric/audita/src/commit/6523d7a/docs/roadmap/remediation.md).
|
||||||
|
Numbering continues here to distinguish new work from those resolved findings.
|
||||||
|
|
||||||
## Architecture Assessment
|
## Architecture Assessment
|
||||||
|
|
||||||
The intended ownership split is substantially implemented. ConfigKit v0.4.0 owns
|
The intended architecture is substantially implemented:
|
||||||
schema-2 loading, overlays, presence/default resolution, explicit overrides,
|
|
||||||
credentials, and configuration inspection. One shared PromptKit v0.11.0 engine
|
|
||||||
owns both roles' profiles, embedded prompt rendering, prepared execution, provider
|
|
||||||
transport, retry/repair behavior, and per-backend admission. Audita retains domain
|
|
||||||
configuration, CLI composition, ordered modules, deterministic validators,
|
|
||||||
proposal/application checks, validation batching/cardinality, reporting, and
|
|
||||||
artifact lifecycle. The root module requires Go 1.25.5 and has no local replacement.
|
|
||||||
|
|
||||||
Legacy loaders, provider transport, template rendering, global/role schedulers,
|
- ConfigKit owns configuration loading, overlays, explicit overrides, and resolved
|
||||||
and the isolated prototype were removed. All nine prompt definitions preserve
|
credentials. Reporting and debug are independent configuration selections.
|
||||||
the original instruction text after accounting for the new hardening variable
|
- One shared PromptKit engine serves proposal and validation roles, owns bounded
|
||||||
and whitespace. There is no need to revisit the library-adoption design or add
|
retries/repairs and backend admission, and supplies execution facts. Audita's
|
||||||
compatibility machinery. The findings below are concrete defects or unfulfilled
|
adapter distinguishes recoverable model failures from fatal setup/internal
|
||||||
parts of the selected final contract.
|
errors and caller cancellation.
|
||||||
|
- Audita normalizes and chunks once, retains section identity as text evolves,
|
||||||
|
preserves stage barriers, and validates proposals through the required chains.
|
||||||
|
Failed validation batches exclude their candidates while successful survivors
|
||||||
|
continue. Model errors do not cancel unrelated work; fatal errors cancel peers
|
||||||
|
and join workers.
|
||||||
|
- Processing-owned outcomes feed a separate report mapper and presentation DTOs.
|
||||||
|
The ordinary bundle contains versioned error/recovery YAML and fixed Markdown
|
||||||
|
matrices. The CLI uses the same counts; detailed persisted evidence is opt-in.
|
||||||
|
- The filesystem output session checks destinations, initializes reports, and
|
||||||
|
publishes transcript and reports through per-file replacement. CLI composition
|
||||||
|
owns streams, lifecycle, fatal finalization, and optional debug handling.
|
||||||
|
|
||||||
## R1 — P1: Redact Model-Derived Content in Every Retained Report Artifact
|
The ordinary resilience path is supported by the retained integration tests,
|
||||||
|
including total model failure, partial validation success, stable chunks,
|
||||||
|
recovery counting, and reporting/debug combinations. The findings below concern
|
||||||
|
specific boundary defects and incomplete evidence handling, not a need to
|
||||||
|
redesign the library ownership or introduce compatibility machinery.
|
||||||
|
|
||||||
**Category:** Incomplete implementation of Stage 10's redaction boundary.
|
## R6 — P1: Honor Restrictive Creation Permissions
|
||||||
|
|
||||||
**Evidence:** Both success and failure paths in
|
**Category:** New filesystem/security regression in Stage 10.
|
||||||
[`runProcess`](../../internal/cli/run.go#L311) write
|
|
||||||
`BuildCorrectionLedger(...)` directly through `RunDirectory.WriteJSONArtifact`.
|
|
||||||
The success-path write is at line 348. That writer serializes its input without
|
|
||||||
redaction. Validator decision/rejection messages contain provider-supplied reason
|
|
||||||
text. Redacting the separate process report does not protect the ledger.
|
|
||||||
|
|
||||||
A controlled HTTP subprocess reproduction returned a schema-valid negative
|
**Evidence:** [`atomicReplace`](../../internal/core/diagnostics/output_session.go#L249)
|
||||||
assessment whose reason contained the synthetic configured backend credential.
|
creates a temporary file and then unconditionally changes its mode to `0644` for
|
||||||
Processing succeeded and the process report was redacted, but
|
an absent destination (lines 261–265). `Chmod` does not apply the process umask.
|
||||||
`correction-ledger.json` retained the credential verbatim. The existing provider-
|
The previous transcript writer used ordinary file creation, which did apply it.
|
||||||
error test does not exercise this successful, schema-valid response path.
|
This affects both the published output and the temporary file while populated.
|
||||||
|
|
||||||
**Required change:** Apply the configured-secret redaction boundary to the ledger
|
An actual CLI run under umask `0077`, with no LLM modules and a valid transcript,
|
||||||
and other model-derived diagnostic/report outputs before persistence. Audit the
|
returned success but created both `out.json` and `errors.yml` with mode `0644`,
|
||||||
new execution/utilization and selected-profile artifact paths for the same bypass;
|
rather than `0600`. In a traversable output directory, other users can therefore
|
||||||
metadata-only output is not a substitute for enforcing the selected secret-value
|
read transcript content despite the operator's restrictive creation policy.
|
||||||
contract. Use all resolved backend credentials, including unselected backends.
|
|
||||||
Keep processing/domain data separate from its sanitized diagnostic representation.
|
|
||||||
Coordinate the shared redaction implementation with R2.
|
|
||||||
|
|
||||||
**Acceptance:** A controlled provider echoes synthetic credentials in a valid
|
**Required change:** Keep standard per-file replacement, but honor ordinary
|
||||||
assessment reason. Neither success nor partial-failure reports, ledgers, interaction
|
creation permissions for previously absent destinations. Preserve intended
|
||||||
artifacts, or other affected diagnostic outputs contain those values. Include an
|
existing-file permission behavior without blindly widening new files. Do not
|
||||||
unselected backend credential. Preserve valid serialization and domain outcomes;
|
read/change a process-global umask around concurrent application operations or
|
||||||
do not expand tests into exhaustive permutations of every artifact.
|
add a transaction/durability framework.
|
||||||
|
|
||||||
## R2 — P1: Remove the Report Redactor's Unredacted Fallback
|
**Acceptance:** A subprocess under a restrictive umask produces new transcript
|
||||||
|
and report files with appropriately restricted permissions. Existing restrictive
|
||||||
|
output permissions remain preserved after replacement. Keep the current cleanup
|
||||||
|
and late-write failure behavior. Test umask behavior in an isolated process.
|
||||||
|
|
||||||
**Category:** New code-quality/security defect in Stage 10.
|
## R7 — P2: Do Not Report Success Without Publishing a Transcript
|
||||||
|
|
||||||
**Evidence:** [`redactProcessReport`](../../internal/cli/run.go#L392) replaces secret
|
**Category:** New destination-selection inconsistency in Stages 10/12.
|
||||||
substrings in the serialized JSON document, then decodes it back into the report.
|
|
||||||
If decoding fails, it returns the original report. Replacements can corrupt JSON
|
|
||||||
syntax or typed values rather than just redact strings.
|
|
||||||
|
|
||||||
A focused reproduction loaded an otherwise valid config with the synthetic
|
**Evidence:** [`PrepareOutput`](../../internal/core/diagnostics/output_session.go#L53)
|
||||||
credential `2`, then redacted a report containing `config_version: 2` and the string
|
uses a trimmed path to decide whether a file destination exists, as does
|
||||||
`echo:2`. Replacing the numeric token made the JSON invalid; the helper returned
|
[`selectedReportDir`](../../internal/cli/config_input.go#L215). However,
|
||||||
the original `echo:2`. Short numeric credentials are accepted by the selected
|
[`finish`](../../internal/cli/process.go#L377) selects stdout only when the original
|
||||||
secret-reference contract, so the implementation cannot silently assume otherwise.
|
path equals the empty string. Parsing does not reject a whitespace-only output
|
||||||
Quotes, escapes, overlapping credentials, and replacement order also need handling
|
path. Consequently, neither branch publishes the transcript.
|
||||||
at the serialization boundary.
|
|
||||||
|
|
||||||
**Required change:** Redact string content without changing JSON structure or
|
A real invocation with `--output ' ' --reporting=false` returned exit 0 and the
|
||||||
numeric/boolean types. Never return or write the original sensitive object when
|
successful completion message, wrote zero bytes to stdout, and created no output
|
||||||
redaction/serialization fails. Share the appropriate string-redaction logic across
|
file. This violates the central contract that successful processing produces a
|
||||||
report and artifact writers rather than maintaining incompatible versions. Handle
|
transcript, independently of reporting selection.
|
||||||
escaped content and overlapping values deliberately; retain typed error causes
|
|
||||||
without exposing provider error text.
|
|
||||||
|
|
||||||
**Acceptance:** Focused tests cover a numeric credential coinciding with report
|
**Required change:** Resolve file versus stdout once and use that decision
|
||||||
numbers, quoted/backslash-containing credentials, and overlapping values. Output
|
consistently throughout placement, preflight, publication, and reporting. Prefer
|
||||||
remains valid with the original numeric/boolean meanings, secrets are removed,
|
preserving a supplied nonempty filename literally, including legal whitespace,
|
||||||
and a redaction failure cannot emit the original object. Exercise the actual
|
with only the empty value selecting stdout. Do not silently turn a file request
|
||||||
external and run-local report writing paths with representative coverage.
|
into discarded output.
|
||||||
|
|
||||||
## R3 — P1: Cancel Remaining Work When a Validator Fails
|
**Acceptance:** Exercise a whitespace-only output filename through the actual CLI:
|
||||||
|
success must mean the requested file exists and contains the encoded transcript.
|
||||||
|
Omitted output still writes the transcript to stdout, and reports follow the same
|
||||||
|
resolved destination decision.
|
||||||
|
|
||||||
**Category:** Pre-existing runner behavior left inconsistent with the selected
|
## R8 — P2: Redact Paths Printed to Stderr
|
||||||
Stage 8 final state and architecture policy; not newly introduced by PromptKit.
|
|
||||||
|
|
||||||
**Evidence:** [`runModulePipeline`](../../internal/framework/runner/runner.go#L383)
|
**Category:** New report-location bypass plus an unclosed debug-location gap in
|
||||||
puts validation errors into a buffered channel, but reads that channel only after
|
the Stage 11/12 redaction boundary.
|
||||||
`vwg.Wait()` (lines 414–421). `setErr` therefore cannot cancel the shared context
|
|
||||||
in response to a validator failure until all validation workers have finished.
|
|
||||||
Proposal work can continue launching validation in the meantime.
|
|
||||||
|
|
||||||
A controlled two-section reproduction made one validator fail while its peer
|
**Evidence:** [`finish`](../../internal/cli/process.go#L398) prints report paths and
|
||||||
waited for context cancellation or an explicit test release. The peer was not
|
the debug directory without redaction. The debug locations printed on failure
|
||||||
canceled; the pipeline returned its error only after the peer was manually released.
|
at lines 365 and 394 do the same. Ordinary report renderers and error diagnostics
|
||||||
In production, a failed validation operation can thus leave other queued or active
|
already have configured-secret redaction; these announcements bypass it.
|
||||||
provider calls consuming capacity and money until they finish or reach their own
|
|
||||||
deadlines. Adapter-level cancellation tests do not cover this runner behavior.
|
|
||||||
|
|
||||||
**Required change:** Propagate validation failures to shared cancellation as soon
|
A CLI reproduction used a synthetic configured credential also present in the
|
||||||
as they occur, while retaining the originating error, joining active workers, and
|
selected output-directory name. `report.md` correctly redacted the value, but
|
||||||
collecting available results deterministically. Use a coordinator or properly
|
the successful run printed it verbatim in all three `Report:` lines on stderr.
|
||||||
synchronized failure state; simply calling the current `setErr` from workers would
|
Configured credentials may be short or coincide with ordinary metadata; their
|
||||||
also require protecting concurrent reads of `firstErr`. Do not add a scheduler,
|
redaction cannot depend on the value not appearing in a path.
|
||||||
outer retry loop, or compatibility layer.
|
|
||||||
|
|
||||||
**Acceptance:** A runner integration test with a failing validator and a blocked
|
**Required change:** Apply the existing configured-secret string redactor to
|
||||||
peer proves cancellation arrives without releasing the peer manually. Cover queued
|
path announcements on both success and failure, including debug locations and
|
||||||
provider work and preservation of the originating failure/partial observations.
|
unselected backend credentials. Leave actual filesystem paths and processing data
|
||||||
The affected runner/runtime tests pass under the race detector, and failure emits
|
unchanged. No new redaction implementation is needed.
|
||||||
no purportedly complete corrected transcript.
|
|
||||||
|
|
||||||
## R4 — P2: Separate Module Identity from Section-Specific Diagnostic Names
|
**Acceptance:** A controlled CLI workflow with synthetic credential-bearing report
|
||||||
|
and debug paths exposes no credential value in stderr, while files are created at
|
||||||
|
the real selected locations. Preserve typed/numeric report fields and transcript
|
||||||
|
output behavior.
|
||||||
|
|
||||||
**Category:** New observation/report attribution defect in Stages 9–10.
|
## R9 — P2: Retain Completed Validation Decisions on Fatal Return
|
||||||
|
|
||||||
**Evidence:** [`validateSectionCandidates`](../../internal/framework/runner/runner.go#L525)
|
**Category:** Incomplete integration of partial validator results in Stage 6.
|
||||||
passes `ModuleInstanceForStages` as the validation request's module identity. This
|
|
||||||
value includes `:section-NNNN`. The
|
|
||||||
[validator call](../../internal/framework/validators/llm_validators.go#L110) copies
|
|
||||||
it into `DiagnosticContext.ModuleInstance` and leaves `SectionIndex` unset.
|
|
||||||
[`BuildUtilization`](../../internal/framework/processreport/llm.go#L25) groups
|
|
||||||
these records by that field, so proposal and validation calls for one real module
|
|
||||||
are reported under different module identities.
|
|
||||||
|
|
||||||
The subprocess reproduction with one grammar module and one section produced:
|
**Evidence:** The LLM validator deliberately returns prior batch decisions with
|
||||||
|
its fatal error in
|
||||||
|
[`Validate`](../../internal/framework/validators/llm_validators.go#L182).
|
||||||
|
[`validateSectionCandidates`](../../internal/framework/runner/runner.go#L640)
|
||||||
|
appends the operation outcomes, but returns on error before incorporating
|
||||||
|
`vResult.Decisions`. The retained decisions only describe earlier validators,
|
||||||
|
not completed batches of the validator that was interrupted.
|
||||||
|
|
||||||
| Reported module instance | Proposal calls | Validation calls |
|
A focused runner probe supplied one completed negative decision and one successful
|
||||||
| --- | ---: | ---: |
|
operation alongside a later fatal error. The runner retained one operation and
|
||||||
| `grammar` | 1 | 0 |
|
zero decisions, while correctly returning the fatal cause and no approved
|
||||||
| `grammar:section-0000` | 0 | 1 |
|
candidates. The retained validator test already proves that successful earlier
|
||||||
|
batch decisions reach this boundary; they are lost at the next layer.
|
||||||
|
|
||||||
The latter is not a configured module instance. Its timing cannot join the real
|
As a result, debug ledgers can label a proposal as merely interrupted even when a
|
||||||
module result, and the actual module incorrectly reports zero validation calls.
|
validator already rejected it, and omit the completed decision and its rationale.
|
||||||
Multi-section runs multiply these spurious entries. Explicit section attribution
|
This conflicts with the selected partial-evidence/debug contract. It does not
|
||||||
is also lost from validation call/event records.
|
currently cause unsafe corrections to be applied.
|
||||||
|
|
||||||
**Required change:** Carry actual module instance and section index separately
|
**Required change:** Preserve trustworthy completed partial decisions as diagnostic
|
||||||
through the validator request and structured-call context. Keep section-qualified
|
evidence when a validator returns a fatal error. Keep identity/duplicate checks
|
||||||
artifact names as a separate naming concern so files remain distinct. Group counts
|
appropriate to a partial result, without requiring decisions for unattempted
|
||||||
and timing by the actual module instance. Update tests that currently use batch
|
candidates. Do not turn partial approval into permission to apply a candidate or
|
||||||
index as a substitute for section identity.
|
weaken fatal cancellation and worker joining.
|
||||||
|
|
||||||
**Acceptance:** A multi-section workflow with repeated module instances yields one
|
**Acceptance:** A runner-level multi-batch workflow with completed decisions
|
||||||
utilization entry per actual instance, correct proposal/validation totals, and
|
followed by fatal failure/cancellation retains those decisions in the final debug
|
||||||
explicit section/batch identity in validation observations. Artifact names remain
|
ledger, retains operation facts and the originating error, and publishes no
|
||||||
unique and correction-ledger ordering is unchanged.
|
completed transcript or unchecked correction.
|
||||||
|
|
||||||
## R5 — P2: Index Execution Events Instead of Repeatedly Scanning Them
|
## R10 — P2: Surface Optional Interaction-Artifact Write Failures
|
||||||
|
|
||||||
**Category:** New report-building performance/maintainability defect in Stage 10.
|
**Category:** Pre-existing ignored errors left inconsistent with the selected
|
||||||
|
Stage 11 debug contract.
|
||||||
|
|
||||||
**Evidence:** [`BuildLLMExecution`](../../internal/framework/processreport/llm.go#L151)
|
**Evidence:** Both
|
||||||
scans every event for every call. `BuildUtilization` additionally tests every event
|
[`GenerateCandidates`](../../internal/framework/proposal_generation/generate.go#L148)
|
||||||
against a linear `hasCall` search for each backend's calls. With C calls and E
|
and [`Validate`](../../internal/framework/validators/llm_validators.go#L154)
|
||||||
events, these paths perform O(C × E) work even for one backend. Retries and repairs
|
discard the error from `WriteInteraction`. These errors never reach the CLI's
|
||||||
increase E. This work occurs after processing, before the CLI completes.
|
nonfatal debug diagnostics. The CLI also silently skips configuration-origin and
|
||||||
|
selected-profile serialization errors in `startDebug`.
|
||||||
|
|
||||||
A synthetic review probe using one terminal event per call measured:
|
A controlled local-provider reproduction enabled debug, then placed a regular file
|
||||||
|
at the module interaction-directory path before returning a valid empty proposal
|
||||||
|
response. Interaction writes failed, but the process printed only its success
|
||||||
|
summary and debug location, with no indication that requested evidence was missing.
|
||||||
|
Exit 0 and unchanged processing are correct; silent evidence loss is not.
|
||||||
|
|
||||||
| Calls/events | Execution mapping | Utilization mapping |
|
**Required change:** Carry optional debug failures to the CLI through a small,
|
||||||
| ---: | ---: | ---: |
|
explicit mechanism and emit redacted stderr diagnostics. Preserve nonfatal
|
||||||
| 1,000 | 14 ms | 6 ms |
|
behavior, keep failures out of model error/recovery counts, and retain paths only
|
||||||
| 10,000 | 1.61 s | 0.52 s |
|
for artifacts actually written. Handle the analogous serialization failures in
|
||||||
| 20,000 | 8.08 s | 2.42 s |
|
`startDebug`; do not introduce a logging framework or make debug a success gate.
|
||||||
|
|
||||||
These are local diagnostic measurements, not performance guarantees; the nested
|
**Acceptance:** A representative failed interaction write produces a safe stderr
|
||||||
scans establish the underlying growth independently of machine timing. Actual
|
diagnostic, while transcript, counts, and exit status match a run without that
|
||||||
snapshots usually contain more than one event per call.
|
failure. Cover both proposal and validator plumbing at their narrow boundaries;
|
||||||
|
one assembled CLI case is sufficient. Debug-off must still create no artifacts.
|
||||||
|
|
||||||
**Required change:** Index calls/events by call ID once and aggregate by backend
|
## Verification and Review Limits
|
||||||
and actual module identity using direct lookups. Reuse terminal records for totals
|
|
||||||
without rescanning all events for each call. Preserve event/call ordering and
|
|
||||||
knownness/completeness semantics. Straightforward maps are sufficient; no reporting
|
|
||||||
framework or new persistent cache is needed.
|
|
||||||
|
|
||||||
**Acceptance:** Existing report semantics remain intact, including preparation
|
- The full uncached offline suite passed: `go test -count=1 ./...`.
|
||||||
failures, undispatched calls, and incomplete usage. A local benchmark or profiling
|
- `go vet ./...` passed.
|
||||||
comparison demonstrates removal of the quadratic scans. Do not add fragile
|
- Focused race checks passed for runner, LLM, validators, process-report assembly,
|
||||||
wall-clock timing assertions to the default test suite.
|
diagnostics/output, reporting, and CLI subprocess packages.
|
||||||
|
- A temporary CLI binary and controlled local HTTP provider reproduced R6, R7,
|
||||||
|
R8, and R10. No live provider or paid service was used.
|
||||||
|
- A temporary runner probe reproduced R9; its expected evidence-preservation
|
||||||
|
assertion failed. The probe and all temporary artifacts were removed.
|
||||||
|
|
||||||
## Review Verification and Limits
|
Passing retained tests do not cover the five failures above. Add proportionate
|
||||||
|
regression coverage with their fixes. This review changes only this document;
|
||||||
The existing root `go test ./...`, focused race tests for LLM/runner/subprocess
|
application fixes, upstream changes, deferred feature work, and release publication
|
||||||
packages, `go vet ./...`, and Linux amd64/arm64 builds passed. Temporary review
|
remain outside scope. Target-platform release builds were not repeated for this
|
||||||
probes reproduced R1–R4 and measured R5; those probe files were removed afterward.
|
review; the native temporary CLI build succeeded.
|
||||||
At review time, the retained suite passed despite these defects. Add durable regression
|
|
||||||
coverage with the fixes, using the narrowest useful production boundaries.
|
|
||||||
|
|
||||||
The initial review changed only this remediation document. No application fixes,
|
|
||||||
upstream changes, live provider calls, release publication, or compatibility work
|
|
||||||
were performed. Release-distribution compliance remains the explicitly deferred
|
|
||||||
release task and is not counted as an implementation defect here.
|
|
||||||
|
|
||||||
## Resolution Record
|
|
||||||
|
|
||||||
All five findings are resolved without compatibility layers, new scheduling,
|
|
||||||
or changes to library ownership.
|
|
||||||
|
|
||||||
### R1 and R2 — Shared Serialization Boundary
|
|
||||||
|
|
||||||
`core/redaction` now sanitizes decoded JSON strings and YAML string nodes,
|
|
||||||
retaining numeric/boolean types and avoiding mutation of domain objects.
|
|
||||||
Longest matching credentials take precedence in a single replacement pass;
|
|
||||||
JSON-escaped credential forms are handled for embedded response content.
|
|
||||||
Serialization or key-collision failures return no payload. The unredacted report
|
|
||||||
fallback has been removed.
|
|
||||||
|
|
||||||
The CLI supplies all resolved backend credentials to the external report writer
|
|
||||||
and run directory. Report copies, ledgers, execution/utilization artifacts,
|
|
||||||
interaction diagnostics, selected profiles, and error text share the redactor.
|
|
||||||
ConfigKit still owns credential-reference redaction of effective configuration.
|
|
||||||
|
|
||||||
Evidence:
|
|
||||||
|
|
||||||
- `TestSubprocessRedactsValidModelReasonsOnSuccessAndPartialFailure` exercises
|
|
||||||
schema-valid negative assessments containing selected and unselected secrets,
|
|
||||||
including quotes/backslashes. Both reports and retained artifacts are checked;
|
|
||||||
partial failure preserves earlier decisions and writes no corrected output.
|
|
||||||
- `TestReportWritersRedactStringsAndFailClosed` exercises both actual report
|
|
||||||
writers with credential `2`, preserving numeric configuration version 2 while
|
|
||||||
redacting string content, and verifies failed serialization writes no artifact.
|
|
||||||
- `TestStringRedactionPreservesSerializationAndDomainData` covers overlapping
|
|
||||||
secrets, escaped nested content, large numbers, booleans, YAML scalar types,
|
|
||||||
unchanged domain data, and failures without an original-payload fallback.
|
|
||||||
|
|
||||||
### R3 — Immediate Shared Cancellation
|
|
||||||
|
|
||||||
Validation workers record their error and cancel shared work immediately.
|
|
||||||
All accesses to the originating-error state use a mutex. Proposal collection
|
|
||||||
stops launching validation after a recorded failure; active workers are joined
|
|
||||||
and available results retain deterministic collection order.
|
|
||||||
|
|
||||||
`TestValidatorFailureCancelsActiveAndPendingProviderWork` uses the real PromptKit
|
|
||||||
runtime with backend concurrency one and queue capacity one. A failing validator
|
|
||||||
cancels two peer executions, one active in a controlled transport and one pending,
|
|
||||||
without manually releasing either. The originating error survives, only one
|
|
||||||
provider attempt occurs, and both logical calls retain finished observations.
|
|
||||||
The runner and runtime pass the race detector.
|
|
||||||
|
|
||||||
### R4 — Independent Module and Section Identity
|
|
||||||
|
|
||||||
Validation requests carry the configured module instance and an explicit section
|
|
||||||
index. The structured call forwards both; only interaction filenames combine
|
|
||||||
module, section, validator, and batch into a diagnostic name. The runner test
|
|
||||||
helper now uses section index rather than batch index to identify sections.
|
|
||||||
|
|
||||||
`TestSubprocessRepeatedModulesKeepSectionAttribution` runs two grammar instances
|
|
||||||
across two sections through the actual CLI/runtime. It verifies exactly two
|
|
||||||
utilization entries with two proposal and two validation calls each, explicit
|
|
||||||
section/batch identity, four distinct validation response artifacts, and unchanged
|
|
||||||
proposal ordering in rejected decisions.
|
|
||||||
|
|
||||||
### R5 — Indexed Report Aggregation
|
|
||||||
|
|
||||||
Execution mapping indexes events by call ID once. Utilization directly groups
|
|
||||||
calls/events by backend and indexes modules by real instance, seeded in module
|
|
||||||
result order. Neither builder scans all events for each logical call.
|
|
||||||
|
|
||||||
Existing summary tests and `TestIndexedReportsPreserveOrderAndPartialUsage`
|
|
||||||
verify pre-dispatch failures, original call/event order, backend grouping, and
|
|
||||||
known-but-incomplete usage. `BenchmarkExecutionReports` retains a reproducible
|
|
||||||
1,000/10,000/20,000-call benchmark without timing assertions.
|
|
||||||
|
|
||||||
A local three-iteration sample measured 20,000 calls at approximately **43 ms**
|
|
||||||
for execution mapping and **84 ms** for utilization, compared with the review
|
|
||||||
probe's 8.08 s and 2.42 s respectively. Timings are illustrative; the direct
|
|
||||||
indexes establish removal of the quadratic scans independently of machine load.
|
|
||||||
|
|
||||||
### Completion Validation
|
|
||||||
|
|
||||||
- Full offline suite: `go test ./...`.
|
|
||||||
- Race checks: runner, LLM runtime, process reporting, redaction, and CLI subprocess
|
|
||||||
packages.
|
|
||||||
- Static analysis: `go vet ./...`.
|
|
||||||
- Both required Linux architectures: `scripts/build-release.sh` in a temporary
|
|
||||||
output directory, without tagging or publishing.
|
|
||||||
- Documentation/diff whitespace checks.
|
|
||||||
|
|
||||||
Release publication and deferred release-distribution work remain outside this
|
|
||||||
remediation unit.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user