69 lines
2.8 KiB
Markdown
69 lines
2.8 KiB
Markdown
# Audita LLM Runtime
|
|
|
|
## Scope
|
|
|
|
This document describes the structured LLM runtime and scheduler behavior.
|
|
|
|
## Client Boundary
|
|
|
|
All runtime LLM calls go through `contracts.StructuredLLMClient`.
|
|
|
|
Primary adapter:
|
|
- `internal/framework/llm/OpenAICompatibleClient`
|
|
|
|
## Adapter and Response Schema Registry
|
|
|
|
The adapter implements the [LLM wire contract](../integrations/openai-compatible-llm.md).
|
|
It maps internal structured requests to HTTP messages and decodes provider content
|
|
into the supplied Go output target. Endpoint, authentication, response shapes, and
|
|
retry conditions are defined in that contract rather than repeated here.
|
|
|
|
`internal/framework/responseschema` stores the per-call response schema definitions
|
|
and diagnostic metadata. Proposal generation and validation choose their schema
|
|
before invoking the client. Provider strict mode does not replace local checks.
|
|
|
|
## Error Classification
|
|
|
|
The adapter marks retryable errors internally and uses context state and attempt
|
|
count to decide whether to retry. It sanitizes errors before returning them.
|
|
`internal/framework/structuredoutput` classifies malformed output for proposal
|
|
and validator call sites, which can turn those errors into warnings and rejected
|
|
section/batch work. Other errors propagate through the runner.
|
|
|
|
## Scheduling and Concurrency
|
|
|
|
`internal/framework/llm/Scheduler` provides FIFO, context-aware permit gating.
|
|
|
|
CLI orchestration composes global and stage schedulers; runner instrumentation
|
|
measures them. Limit definitions and inheritance belong to
|
|
[configuration](../config.md#concurrency).
|
|
|
|
Scheduler release is guarded to avoid permit leaks on cancellation/error. Runner
|
|
collects utilization observations; their serialized shape is defined by the
|
|
[output-file contract](../integrations/output-files.md#utilization-diagnostics).
|
|
|
|
## Diagnostics and Redaction
|
|
|
|
`internal/framework/llm/DiagnosticsWriter` writes request/response/error artifacts.
|
|
|
|
Configured secrets are derived from `llm.ConfiguredSecrets(cfg)` and redacted from:
|
|
- diagnostics payloads;
|
|
- surfaced runtime/adapter errors.
|
|
|
|
## Changing LLM Integration or Scheduling
|
|
|
|
Review client, scheduler, redaction, and structured-output malformed-response
|
|
coverage alongside runner orchestration and utilization tests. Use the
|
|
[testing policy](../policy/testing.md) to select sufficient checks; preserve the
|
|
[LLM integration contract](../integrations/openai-compatible-llm.md) and
|
|
[architecture invariants](../policy/architecture.md#correction-and-concurrency-invariants).
|
|
|
|
## Key Tests
|
|
|
|
- `internal/framework/llm/openai_compatible_client_test.go`
|
|
- `internal/framework/llm/scheduler_test.go`
|
|
- `internal/framework/llm/diagnostics_test.go`
|
|
- `internal/framework/responseschema/registry_test.go`
|
|
- `internal/framework/structuredoutput/malformed_test.go`
|
|
- `internal/framework/runner/*_test.go` (orchestration and utilization)
|