Add version two LLM reports and safe execution diagnostics
This commit is contained in:
@@ -39,7 +39,7 @@ from the transcript output envelope:
|
||||
| Metadata field | Type / current value |
|
||||
| --- | --- |
|
||||
| `report_schema_name` | String: `audita-process-report` |
|
||||
| `report_schema_version` | String: `v1` |
|
||||
| `report_schema_version` | String: `v2` |
|
||||
| `output_schema` | String identifying the selected transcript encoder |
|
||||
| `config_version` | Optional integer, present when file configuration was loaded |
|
||||
|
||||
@@ -60,6 +60,7 @@ The remaining top-level fields are:
|
||||
| `normalization_skipped` | Object of optional integer counters: `different_speakers`, `gap_too_large`, `duration_exceeded`, `token_limit_exceeded`; can be empty |
|
||||
| `chunking` | Optional section summary described below |
|
||||
| `diagnostics` | Optional artifact-path metadata described below |
|
||||
| `llm_summary` | Optional logical LLM execution summary, present when observations were initialized |
|
||||
| `modules_summary` | Optional aggregate module outcome |
|
||||
| `module_results` | Optional array of module outcomes |
|
||||
|
||||
@@ -88,7 +89,7 @@ human-readable, not structured fields to parse for control flow.
|
||||
- `directory_path`
|
||||
- `source_transcript_path`, `parsed_source_transcript_path`, `normalized_transcript_path`
|
||||
- `normalization_summary_path`, `chunking_summary_path`
|
||||
- `utilization_summary_path`, `correction_ledger_path`
|
||||
- `utilization_summary_path`, `llm_execution_path`, `selected_profiles_path`, `correction_ledger_path`
|
||||
- `invocation_metadata_path`, `redacted_effective_config_path`, `error_log_path`
|
||||
|
||||
The [operations guide](../operations.md#diagnostics-directory) defines the physical
|
||||
@@ -163,24 +164,26 @@ or resume protocol.
|
||||
|
||||
## Utilization Diagnostics
|
||||
|
||||
This unversioned JSON object describes observations from the runtime, not service
|
||||
limits or performance guarantees:
|
||||
`llm_summary` contains integer `logical_calls`, `provider_attempts`,
|
||||
`transient_retries`, and `structural_repairs`. Its `usage` contains `prompt`,
|
||||
`completion`, `total`, `cached`, and `cache_write`, each with integer `value`
|
||||
and boolean `known` and `complete`. A call counts once even if preparation fails.
|
||||
If no call dispatches, usage is known-zero and complete. Missing `llm_summary`
|
||||
means observations were unavailable.
|
||||
|
||||
- `effective_concurrency`: integer `total_llm`, `proposal_llm`, `validation_llm`.
|
||||
- `run_timing`: millisecond integers `run_wall_time_ms`, `scheduler_queue_wait_ms`,
|
||||
`llm_execution_time_ms`, `deterministic_validation_time_ms`; integer
|
||||
`max_in_flight_llm_calls` and `average_in_flight_llm_calls`.
|
||||
- `llm_calls`: integer `total_proposal_calls` and `total_validation_calls`.
|
||||
- Optional `modules`: records with `module_key`, `module_instance`, integer
|
||||
`proposal_llm_calls` and `validation_llm_calls`, and millisecond integer
|
||||
`scheduler_queue_wait_ms`, `llm_execution_time_ms`,
|
||||
`deterministic_validation_time_ms`, and `module_wall_time_ms`.
|
||||
- Optional `validators`: records with `module_key`, `module_instance`,
|
||||
`validator_key`, boolean `llm_backed`, integer `calls`, and millisecond integer
|
||||
`elapsed_ms`.
|
||||
The unversioned utilization JSON contains `backends`, keyed by backend ID. Each
|
||||
entry has configured `concurrency` and `queue_capacity`, the same execution
|
||||
summary fields, and integer `capacity_wait_ms`, `generation_ms`,
|
||||
`retry_delay_ms`, and `logical_wall_ms`. It also has `run_wall_time_ms` and
|
||||
optional `modules` and `validators` arrays. Module records contain the module
|
||||
identity, logical proposal/validation call counts, and `module_wall_time_ms`.
|
||||
Validator records contain module/validator identity, `llm_backed`, `calls`, and
|
||||
`elapsed_ms`. Timings can overlap; they do not decompose total wall time.
|
||||
|
||||
Aggregated execution/wait durations can overlap in concurrent work; they need
|
||||
not sum to wall time. The observed average concurrency is serialized as an integer.
|
||||
Other source snapshots, prompt payloads, and diagnostic files are debugging
|
||||
material; their placement is documented in operations without defining a separate
|
||||
public schema for every file.
|
||||
`llm-execution.json` has `calls` and `events` arrays joined by `call_id`. Calls
|
||||
include logical attribution, final classification, summary, elapsed milliseconds,
|
||||
and available prompt/profile/model metadata. Events contain safe execution
|
||||
metadata, including attempt and terminal records, with millisecond durations.
|
||||
Preparation failures have a call and no fabricated events. This artifact has no
|
||||
provider messages, endpoints, credentials, or transcript content. Other request
|
||||
and response artifacts can contain redacted prompt and transcript content.
|
||||
|
||||
Reference in New Issue
Block a user