Add version two LLM reports and safe execution diagnostics

This commit is contained in:
2026-09-24 02:31:37 +00:00
parent 4bfc5f3456
commit 9561e80efb
16 changed files with 428 additions and 40 deletions

View File

@@ -39,7 +39,7 @@ from the transcript output envelope:
| Metadata field | Type / current value |
| --- | --- |
| `report_schema_name` | String: `audita-process-report` |
| `report_schema_version` | String: `v1` |
| `report_schema_version` | String: `v2` |
| `output_schema` | String identifying the selected transcript encoder |
| `config_version` | Optional integer, present when file configuration was loaded |
@@ -60,6 +60,7 @@ The remaining top-level fields are:
| `normalization_skipped` | Object of optional integer counters: `different_speakers`, `gap_too_large`, `duration_exceeded`, `token_limit_exceeded`; can be empty |
| `chunking` | Optional section summary described below |
| `diagnostics` | Optional artifact-path metadata described below |
| `llm_summary` | Optional logical LLM execution summary, present when observations were initialized |
| `modules_summary` | Optional aggregate module outcome |
| `module_results` | Optional array of module outcomes |
@@ -88,7 +89,7 @@ human-readable, not structured fields to parse for control flow.
- `directory_path`
- `source_transcript_path`, `parsed_source_transcript_path`, `normalized_transcript_path`
- `normalization_summary_path`, `chunking_summary_path`
- `utilization_summary_path`, `correction_ledger_path`
- `utilization_summary_path`, `llm_execution_path`, `selected_profiles_path`, `correction_ledger_path`
- `invocation_metadata_path`, `redacted_effective_config_path`, `error_log_path`
The [operations guide](../operations.md#diagnostics-directory) defines the physical
@@ -163,24 +164,26 @@ or resume protocol.
## Utilization Diagnostics
This unversioned JSON object describes observations from the runtime, not service
limits or performance guarantees:
`llm_summary` contains integer `logical_calls`, `provider_attempts`,
`transient_retries`, and `structural_repairs`. Its `usage` contains `prompt`,
`completion`, `total`, `cached`, and `cache_write`, each with integer `value`
and boolean `known` and `complete`. A call counts once even if preparation fails.
If no call dispatches, usage is known-zero and complete. Missing `llm_summary`
means observations were unavailable.
- `effective_concurrency`: integer `total_llm`, `proposal_llm`, `validation_llm`.
- `run_timing`: millisecond integers `run_wall_time_ms`, `scheduler_queue_wait_ms`,
`llm_execution_time_ms`, `deterministic_validation_time_ms`; integer
`max_in_flight_llm_calls` and `average_in_flight_llm_calls`.
- `llm_calls`: integer `total_proposal_calls` and `total_validation_calls`.
- Optional `modules`: records with `module_key`, `module_instance`, integer
`proposal_llm_calls` and `validation_llm_calls`, and millisecond integer
`scheduler_queue_wait_ms`, `llm_execution_time_ms`,
`deterministic_validation_time_ms`, and `module_wall_time_ms`.
- Optional `validators`: records with `module_key`, `module_instance`,
`validator_key`, boolean `llm_backed`, integer `calls`, and millisecond integer
`elapsed_ms`.
The unversioned utilization JSON contains `backends`, keyed by backend ID. Each
entry has configured `concurrency` and `queue_capacity`, the same execution
summary fields, and integer `capacity_wait_ms`, `generation_ms`,
`retry_delay_ms`, and `logical_wall_ms`. It also has `run_wall_time_ms` and
optional `modules` and `validators` arrays. Module records contain the module
identity, logical proposal/validation call counts, and `module_wall_time_ms`.
Validator records contain module/validator identity, `llm_backed`, `calls`, and
`elapsed_ms`. Timings can overlap; they do not decompose total wall time.
Aggregated execution/wait durations can overlap in concurrent work; they need
not sum to wall time. The observed average concurrency is serialized as an integer.
Other source snapshots, prompt payloads, and diagnostic files are debugging
material; their placement is documented in operations without defining a separate
public schema for every file.
`llm-execution.json` has `calls` and `events` arrays joined by `call_id`. Calls
include logical attribution, final classification, summary, elapsed milliseconds,
and available prompt/profile/model metadata. Events contain safe execution
metadata, including attempt and terminal records, with millisecond durations.
Preparation failures have a call and no fabricated events. This artifact has no
provider messages, endpoints, credentials, or transcript content. Other request
and response artifacts can contain redacted prompt and transcript content.