eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Document resilient reports and remove obsolete warning material

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-24 17:55:37 +00:00
parent 2f036b0356
commit a770c77a23
26 changed files with 346 additions and 536 deletions
Download Patch File Download Diff File Expand all files Collapse all files

119
docs/internal/diagnostics-reporting.md
View File

@@ -1,91 +1,42 @@
# Diagnostics and Report Assembly
# Diagnostics and Reporting Internals
## Scope
The [output-file contract](../integrations/output-files.md) owns public
schemas and the [operations guide](../operations.md) owns physical placement.
This document owns implementation flow and collaborators. Physical artifacts and
retention belong to [operations](../operations.md). Serialized reports, ledger
records, and utilization shapes belong to the
[output-file contract](../integrations/output-files.md).
`internal/framework/outcomes` contains processing-owned operation references,
counts, failure reasons, filtering details, and work states. It imports no
reporting, CLI, or provider package. The runner records those facts per initial
section and module instance. `internal/framework/processreport` maps them once
into `internal/core/reporting.Snapshot`, including nullable counts, grouped
work-unit errors, separate run errors, and one recovery per successful logical
operation. Reports order stages, sections, validators, and batches by semantic
identity, never concurrent completion time. The core reporting package owns
Audita's versioned YAML DTOs, fixed Markdown rendering, and brief CLI
completion formatter. YAML redaction walks serialized string nodes so numeric
and boolean values retain their types. Ordinary reports omit raw causes,
transcript text, complete decisions, and token/timing payloads.
## Run Directory and Artifact Writers
`internal/core/diagnostics.OutputSession` owns active destination checks,
transcript preflight, report initialization, and per-file final replacement.
CLI composes it after configuration/profile inspection and before reading
inputs or dispatching model work. It does not create ordinary report parents.
On a final write error it attempts other enabled reports and one failed-status
pass; a published transcript is never erased by a later report failure.
`internal/core/diagnostics` creates run directories, writes snapshots and metadata,
and evaluates retention. CLI orchestration supplies the selected configuration,
paths, run outcome, and correction-presence information. Artifact names are
centralized in diagnostics; their operator-facing inventory is in
[operations](../operations.md#diagnostics-directory).
Snapshot writes are attempted at the relevant processing phases. Several diagnostic
writes are best-effort, so report metadata can identify a location without proving
that a file was written. The CLI distinguishes primary processing errors from
secondary diagnostic/report write errors.
## Process Report Mapping
`internal/framework/processreport.Build` maps invocation facts, normalization and
chunking summaries, and runner output into `reporting.ProcessReport`. Core reporting
owns the Go data structures and JSON writer. The builder attaches diagnostic
metadata when a run-directory path is available.
Module mapping copies outcomes from runner results, collects summary counts, and
preserves available partial module results on failure. The CLI failure path does
not pass normalization or chunking summaries to report construction even when
some earlier processing completed. Consumer interpretation belongs to the
[process-report contract](../integrations/output-files.md#process-report).
## Correction Ledger and Utilization
`BuildCorrectionLedger` flattens module changes, rejections, and failures into
ledger entries. It associates validator records by proposal index within each
module, separates execution classes through validator metadata, and sorts entries
by module instance, proposal index, and disposition.
Runner instrumentation and PromptKit observations collect utilization; CLI
orchestration writes the backend summary and safe call/event artifact when run
output is available. Report assembly maps the same observations to `llm_summary`
without independently measuring calls. A selected backend remains attributable
when preparation fails before dispatch. Domain module/validator timing stays
separate from provider timing, and those durations can overlap.
For the serialized forms see [ledger](../integrations/output-files.md#correction-ledger)
and [utilization](../integrations/output-files.md#utilization-diagnostics).
## CLI Coordination and Retention
CLI orchestration writes caller-selected reports and run-local artifacts before
asking diagnostics to apply retention. `HasSkippedCorrections` derives the
retention input from runner outcomes. Keep the exact lifecycle definition in
[operations](../operations.md#retention-behavior), not in another internal table.
On a processing failure, secondary report-write errors do not replace the original
error. On the success path, an external report-write failure returns before the
run-local report write; retention errors occur later. This ordering explains why
artifacts and process exit status must be assessed together.
## Redaction
Effective configuration uses ConfigKit redaction. The CLI supplies all resolved
backend credentials, including unselected backends, to report and diagnostic
writers. `core/redaction` sanitizes the serialized representation rather than
mutating processing results. JSON string values and keys are redacted after
parsing with number preservation; YAML redaction preserves scalar types.
Serialization failure returns an error without writing the original object.
Interaction artifacts, correction ledgers, execution/utilization diagnostics,
selected profiles, both report copies, and surfaced LLM errors share this
string-redaction implementation. Longer overlapping values take precedence,
and replacement text is not processed again within a redaction pass.
Execution reports index events by call ID. Utilization groups calls directly by
backend and actual module instance, keeping module result order and retaining
usage knownness/completeness. Section and batch indices remain separate from
module identity; section-qualified interaction filenames avoid collisions.
These boundaries implement the [architecture requirement](../policy/architecture.md#errors-and-secret-handling).
Operational handling of transcript-bearing artifacts belongs to
[operations](../operations.md#artifact-handling).
Debug is independent. Only `diagnostics.debug` creates a unique retained child
under `diagnostics.work_dir`. CLI passes empty diagnostics paths and nil writers
when debug is off. With debug on, source/normalized snapshots, invocation,
effective settings, origins, profiles, initial chunk summary, correction ledger,
indexed execution/utilization, and model interactions are written. Runner
candidate capture includes proposals later rejected or interrupted. Debug
write failures produce safe stderr messages and do not abort processing.
Every debug writer uses all configured backend secrets for redaction, including
unselected backend credentials and source snapshots. Debug can still contain
sensitive transcript and prompt content.
## Key Tests
- `internal/core/redaction/redaction_test.go`
- `internal/core/diagnostics/*_test.go`
- `internal/framework/processreport/*_test.go`
- `internal/core/reporting/report_test.go`
- `cmd/audita/main_integration_test.go`
Useful tests: `internal/core/reporting/*_test.go`,
`internal/core/diagnostics/*_test.go`,
`internal/framework/processreport/*_test.go`, and
`cmd/audita/main_integration_test.go`.