92 lines
4.7 KiB
Markdown
92 lines
4.7 KiB
Markdown
# Diagnostics and Report Assembly
|
|
|
|
## Scope
|
|
|
|
This document owns implementation flow and collaborators. Physical artifacts and
|
|
retention belong to [operations](../operations.md). Serialized reports, ledger
|
|
records, and utilization shapes belong to the
|
|
[output-file contract](../integrations/output-files.md).
|
|
|
|
## Run Directory and Artifact Writers
|
|
|
|
`internal/core/diagnostics` creates run directories, writes snapshots and metadata,
|
|
and evaluates retention. CLI orchestration supplies the selected configuration,
|
|
paths, run outcome, and correction-presence information. Artifact names are
|
|
centralized in diagnostics; their operator-facing inventory is in
|
|
[operations](../operations.md#diagnostics-directory).
|
|
|
|
Snapshot writes are attempted at the relevant processing phases. Several diagnostic
|
|
writes are best-effort, so report metadata can identify a location without proving
|
|
that a file was written. The CLI distinguishes primary processing errors from
|
|
secondary diagnostic/report write errors.
|
|
|
|
## Process Report Mapping
|
|
|
|
`internal/framework/processreport.Build` maps invocation facts, normalization and
|
|
chunking summaries, and runner output into `reporting.ProcessReport`. Core reporting
|
|
owns the Go data structures and JSON writer. The builder attaches diagnostic
|
|
metadata when a run-directory path is available.
|
|
|
|
Module mapping copies outcomes from runner results, collects summary counts, and
|
|
preserves available partial module results on failure. The CLI failure path does
|
|
not pass normalization or chunking summaries to report construction even when
|
|
some earlier processing completed. Consumer interpretation belongs to the
|
|
[process-report contract](../integrations/output-files.md#process-report).
|
|
|
|
## Correction Ledger and Utilization
|
|
|
|
`BuildCorrectionLedger` flattens module changes, rejections, and failures into
|
|
ledger entries. It associates validator records by proposal index within each
|
|
module, separates execution classes through validator metadata, and sorts entries
|
|
by module instance, proposal index, and disposition.
|
|
|
|
Runner instrumentation and PromptKit observations collect utilization; CLI
|
|
orchestration writes the backend summary and safe call/event artifact when run
|
|
output is available. Report assembly maps the same observations to `llm_summary`
|
|
without independently measuring calls. A selected backend remains attributable
|
|
when preparation fails before dispatch. Domain module/validator timing stays
|
|
separate from provider timing, and those durations can overlap.
|
|
For the serialized forms see [ledger](../integrations/output-files.md#correction-ledger)
|
|
and [utilization](../integrations/output-files.md#utilization-diagnostics).
|
|
|
|
## CLI Coordination and Retention
|
|
|
|
CLI orchestration writes caller-selected reports and run-local artifacts before
|
|
asking diagnostics to apply retention. `HasSkippedCorrections` derives the
|
|
retention input from runner outcomes. Keep the exact lifecycle definition in
|
|
[operations](../operations.md#retention-behavior), not in another internal table.
|
|
|
|
On a processing failure, secondary report-write errors do not replace the original
|
|
error. On the success path, an external report-write failure returns before the
|
|
run-local report write; retention errors occur later. This ordering explains why
|
|
artifacts and process exit status must be assessed together.
|
|
|
|
## Redaction
|
|
|
|
Effective configuration uses ConfigKit redaction. The CLI supplies all resolved
|
|
backend credentials, including unselected backends, to report and diagnostic
|
|
writers. `core/redaction` sanitizes the serialized representation rather than
|
|
mutating processing results. JSON string values and keys are redacted after
|
|
parsing with number preservation; YAML redaction preserves scalar types.
|
|
Serialization failure returns an error without writing the original object.
|
|
Interaction artifacts, correction ledgers, execution/utilization diagnostics,
|
|
selected profiles, both report copies, and surfaced LLM errors share this
|
|
string-redaction implementation. Longer overlapping values take precedence,
|
|
and replacement text is not processed again within a redaction pass.
|
|
|
|
Execution reports index events by call ID. Utilization groups calls directly by
|
|
backend and actual module instance, keeping module result order and retaining
|
|
usage knownness/completeness. Section and batch indices remain separate from
|
|
module identity; section-qualified interaction filenames avoid collisions.
|
|
These boundaries implement the [architecture requirement](../policy/architecture.md#errors-and-secret-handling).
|
|
Operational handling of transcript-bearing artifacts belongs to
|
|
[operations](../operations.md#artifact-handling).
|
|
|
|
## Key Tests
|
|
|
|
- `internal/core/redaction/redaction_test.go`
|
|
- `internal/core/diagnostics/*_test.go`
|
|
- `internal/framework/processreport/*_test.go`
|
|
- `internal/core/reporting/report_test.go`
|
|
- `cmd/audita/main_integration_test.go`
|