audita/docs/internal/llm-runtime.md

Audita LLM Runtime

Scope

This document describes the structured LLM runtime and scheduler behavior.

Client Boundary

All runtime LLM calls go through contracts.StructuredLLMClient.

Primary adapter:

• internal/framework/llm/OpenAICompatibleClient

Request/Response Behavior

The OpenAI-compatible adapter sends chat completions requests with:

• model;
• messages;
• response_format.type = json_schema;
• strict schema envelope (name, schema, strict=true).

The response is decoded into the requested structured output target.

Response Schema Registry

Structured response schemas are registered in internal/framework/responseschema:

• correction_set
• validator_decision_set

Each schema includes stable diagnostics metadata (id, version, name, sha256).

Retries and Error Handling

Adapter retries apply to retryable conditions (for example transport/decoding/retryable status classes) up to configured max_retries.

Errors are sanitized to redact configured API-key values before surfacing.

Malformed structured output detection is shared through internal/framework/structuredoutput and is used by:

• proposal generation;
• LLM-backed validators.

Scheduling and Concurrency

internal/framework/llm/Scheduler provides FIFO, context-aware permit gating.

Runner composes scheduler limits across:

• total LLM concurrency;
• proposal LLM concurrency;
• validation LLM concurrency.

Scheduler release is guarded to avoid permit leaks on cancellation/error.

Diagnostics and Redaction

internal/framework/llm/DiagnosticsWriter writes request/response/error artifacts.

Configured secrets are derived from llm.ConfiguredSecrets(cfg) and redacted from:

• diagnostics payloads;
• surfaced runtime/adapter errors.

Key Tests

• internal/framework/llm/openai_compatible_client_test.go
• internal/framework/llm/scheduler_test.go
• internal/framework/llm/diagnostics_test.go
• internal/framework/responseschema/registry_test.go
• internal/framework/structuredoutput/malformed_test.go