68 lines
2.0 KiB
Markdown
68 lines
2.0 KiB
Markdown
# Audita LLM Runtime
|
|
|
|
## Scope
|
|
|
|
This document describes the structured LLM runtime and scheduler behavior.
|
|
|
|
## Client Boundary
|
|
|
|
All runtime LLM calls go through `contracts.StructuredLLMClient`.
|
|
|
|
Primary adapter:
|
|
- `internal/framework/llm/OpenAICompatibleClient`
|
|
|
|
## Request/Response Behavior
|
|
|
|
The OpenAI-compatible adapter sends chat completions requests with:
|
|
- model;
|
|
- messages;
|
|
- `response_format.type = json_schema`;
|
|
- strict schema envelope (`name`, `schema`, `strict=true`).
|
|
|
|
The response is decoded into the requested structured output target.
|
|
|
|
## Response Schema Registry
|
|
|
|
Structured response schemas are registered in `internal/framework/responseschema`:
|
|
- `correction_set`
|
|
- `validator_decision_set`
|
|
|
|
Each schema includes stable diagnostics metadata (`id`, `version`, `name`, `sha256`).
|
|
|
|
## Retries and Error Handling
|
|
|
|
Adapter retries apply to retryable conditions (for example transport/decoding/retryable status classes) up to configured `max_retries`.
|
|
|
|
Errors are sanitized to redact configured API-key values before surfacing.
|
|
|
|
Malformed structured output detection is shared through `internal/framework/structuredoutput` and is used by:
|
|
- proposal generation;
|
|
- LLM-backed validators.
|
|
|
|
## Scheduling and Concurrency
|
|
|
|
`internal/framework/llm/Scheduler` provides FIFO, context-aware permit gating.
|
|
|
|
Runner composes scheduler limits across:
|
|
- total LLM concurrency;
|
|
- proposal LLM concurrency;
|
|
- validation LLM concurrency.
|
|
|
|
Scheduler release is guarded to avoid permit leaks on cancellation/error.
|
|
|
|
## Diagnostics and Redaction
|
|
|
|
`internal/framework/llm/DiagnosticsWriter` writes request/response/error artifacts.
|
|
|
|
Configured secrets are derived from `llm.ConfiguredSecrets(cfg)` and redacted from:
|
|
- diagnostics payloads;
|
|
- surfaced runtime/adapter errors.
|
|
|
|
## Key Tests
|
|
|
|
- `internal/framework/llm/openai_compatible_client_test.go`
|
|
- `internal/framework/llm/scheduler_test.go`
|
|
- `internal/framework/llm/diagnostics_test.go`
|
|
- `internal/framework/responseschema/registry_test.go`
|
|
- `internal/framework/structuredoutput/malformed_test.go`
|