Use PromptKit prepared execution for structured model calls
This commit is contained in:
@@ -1,76 +1,31 @@
|
||||
# OpenAI-Compatible LLM Integration
|
||||
|
||||
This document owns the external protocol used by Audita's structured LLM adapter.
|
||||
Settings, inheritance, defaults, and credential supply belong to
|
||||
[configuration](../config.md#llm-settings); implementation and scheduling belong to
|
||||
[LLM runtime internals](../internal/llm-runtime.md).
|
||||
Audita uses the pinned [PromptKit](https://gitea.maximumdirect.net/eric/promptkit)
|
||||
built-in OpenAI-compatible transport. PromptKit owns chat-completions wire framing,
|
||||
provider envelope parsing, transient retries, per-generation timeouts, and JSON
|
||||
Schema repair. A backend's configured base endpoint must support the selected
|
||||
profile's model and structured JSON-schema responses; local configuration
|
||||
inspection does not contact the provider.
|
||||
|
||||
## Endpoint and Authentication
|
||||
Audita selects a role profile, supplies its registered backend credential, and
|
||||
passes the embedded response schema as a complete per-call output contract. It
|
||||
prepares each call once, checks that the prepared target still matches the
|
||||
inspected backend and endpoint, then executes that same prepared request. A
|
||||
profile cannot introduce a separate credential environment lookup. An absent
|
||||
backend credential permits an unauthenticated local service.
|
||||
|
||||
Audita sends an HTTP `POST` to the configured base URL with trailing slashes
|
||||
removed and `/chat/completions` appended. Supply an HTTP or HTTPS base URL that
|
||||
makes this the intended endpoint; the adapter does not require HTTPS in code.
|
||||
Requests have `Content-Type: application/json`. A non-empty configured API key
|
||||
adds `Authorization: Bearer <api_key>`; otherwise that header is omitted.
|
||||
`llm.transient_retries` counts additional attempts for eligible transient
|
||||
failures; explicit zero makes one attempt. `llm.repair_attempts` counts
|
||||
additional generations for schema-invalid model output. The operation timeout
|
||||
covers preparation, capacity waiting, retries, generation, and repair. Caller
|
||||
cancellation or an earlier deadline also applies. See [configuration](../config.md#llm-settings)
|
||||
for defaults and [PromptKit's consumer contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/consumers/pkg-promptkit.md)
|
||||
for the pinned protocol and retry details.
|
||||
|
||||
The endpoint must support the configured model and structured JSON-schema
|
||||
responses. Local configuration validation does not probe that support.
|
||||
|
||||
## Request Shape
|
||||
|
||||
The payload contains `model`, an array of `messages` with string `role`/`content`,
|
||||
and `response_format`. The structured-output portion has this illustrative shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "json_schema",
|
||||
"json_schema": {
|
||||
"name": "correction_set",
|
||||
"strict": true,
|
||||
"schema": {"type": "object"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The actual schema is supplied for the call; the fragment above is not a complete
|
||||
request schema. Runtime calls use `correction_set` and `validator_decision_set`
|
||||
identities. The adapter requires a non-empty resolved model, non-empty message
|
||||
roles/content, a schema name, and valid schema JSON. It sends strict mode to the
|
||||
provider; local typed decoding and proposal/decision checks still apply afterward.
|
||||
|
||||
## Response Handling Contract
|
||||
|
||||
A successful response must be a JSON envelope with at least one `choices` item.
|
||||
Audita reads the first choice's `message.content`, accepting a non-empty string
|
||||
containing JSON or a non-null raw JSON value that can decode into the expected
|
||||
output type. Empty content, missing choices, and incompatible payloads fail.
|
||||
The adapter also reads optional `model` and token-usage metadata.
|
||||
|
||||
There is no streaming protocol or provider SDK dependency in this integration.
|
||||
Audita uses this chat-completions contract, not other provider API shapes.
|
||||
|
||||
## Retries and Timeouts
|
||||
|
||||
The effective adapter retry count is additional attempts after the initial request;
|
||||
see [configuration](../config.md#llm-settings) for the current zero-value behavior.
|
||||
Transport/read errors, malformed response envelopes/content, typed decode failures,
|
||||
HTTP `429`, and server-error responses are retryable. Other non-success HTTP
|
||||
responses and a response missing choices are not retried by the adapter.
|
||||
Cancellation or an expired context stops retrying.
|
||||
|
||||
The configured HTTP timeout applies per request; it is not a whole-process time
|
||||
budget. A parent process can impose its own overall timeout. See
|
||||
[configuration](../config.md#llm-settings) for timeout and retry values and
|
||||
[subprocess guidance](subprocess.md) for parent responsibilities.
|
||||
|
||||
## Error Behavior
|
||||
|
||||
For non-success responses, error detail comes from provider `error.message`, then
|
||||
a top-level `message`, otherwise the HTTP status and response body. Configured
|
||||
keys are redacted from surfaced errors and LLM diagnostic payloads.
|
||||
|
||||
Malformed structured responses can become section/batch warnings and rejected
|
||||
proposals rather than fatal run errors, depending on the call site. Other adapter
|
||||
failures can terminate processing with partial results. Consult the
|
||||
[report contract](output-files.md#module-outcomes) for those outcomes and
|
||||
[operations](../operations.md#artifact-handling) for handling diagnostic contents.
|
||||
A failed JSON Schema result after repair is classified as malformed generated
|
||||
output; proposal and validator callers can turn it into a warning. Provider,
|
||||
transport, malformed HTTP envelope, retry, admission, and deadline failures stay
|
||||
runtime failures. Audita decodes into domain types only after PromptKit validates
|
||||
the JSON. Errors exposed to users carry safe messages while retaining typed
|
||||
causes for internal classification. Diagnostic request/response content is
|
||||
sensitive and follows [operations guidance](../operations.md#artifact-handling).
|
||||
|
||||
@@ -1,68 +1,34 @@
|
||||
# Audita LLM Runtime
|
||||
|
||||
## Scope
|
||||
|
||||
This document describes the structured LLM runtime and scheduler behavior.
|
||||
|
||||
## Client Boundary
|
||||
|
||||
All runtime LLM calls go through `contracts.StructuredLLMClient`.
|
||||
`llm.NewRuntime` assembles one PromptKit engine for both roles. It loads embedded
|
||||
prompt and schema assets, registers application backends, and inspects the
|
||||
selected proposal and validation profiles offline. A role-bound `llm.RoleClient`
|
||||
implements `structuredoutput.Client`; proposal and validator contracts alias that
|
||||
interface during migration. Both roles share engine-local backend capacity.
|
||||
|
||||
Primary adapter:
|
||||
- `internal/framework/llm/OpenAICompatibleClient`
|
||||
For each logical call, the adapter starts an operation context and a diagnostic
|
||||
record before preparation. It supplies the reserved hardening text, the selected
|
||||
profile and resolved backend credential, and the complete embedded schema/repair
|
||||
contract. `PrepareExecution` produces the exact messages and hashes retained in
|
||||
an Audita snapshot. The adapter checks the prepared backend and endpoint before
|
||||
`RunPrepared`. It records available final content, usage, model, and profile
|
||||
metadata and decodes only a validated result. Call hooks expose logical
|
||||
start/finish records and PromptKit execution events for the report collector.
|
||||
|
||||
## Adapter and Response Schema Registry
|
||||
The [integration contract](../integrations/openai-compatible-llm.md) describes
|
||||
wire ownership and failure behavior. `structuredoutput.Failure` has a safe
|
||||
message and a typed cause; only exhausted structural repair is a malformed
|
||||
output failure. Secrets from all configured backends, including unselected
|
||||
ones, are available to the diagnostics redaction boundary.
|
||||
|
||||
The adapter implements the [LLM wire contract](../integrations/openai-compatible-llm.md).
|
||||
It maps internal structured requests to HTTP messages and decodes provider content
|
||||
into the supplied Go output target. Endpoint, authentication, response shapes, and
|
||||
retry conditions are defined in that contract rather than repeated here.
|
||||
## Current Orchestration
|
||||
|
||||
`internal/framework/responseschema` stores the per-call response schema definitions
|
||||
and diagnostic metadata. Proposal generation and validation choose their schema
|
||||
before invoking the client. Provider strict mode does not replace local checks.
|
||||
The old `llm.Scheduler` and runner instrumentation still exist pending the
|
||||
orchestration migration. The final runtime uses PromptKit backend admission,
|
||||
retry, and repair rather than that scheduler. `llm.DiagnosticsWriter` remains
|
||||
the current protected artifact writer.
|
||||
|
||||
## Error Classification
|
||||
|
||||
The adapter marks retryable errors internally and uses context state and attempt
|
||||
count to decide whether to retry. It sanitizes errors before returning them.
|
||||
`internal/framework/structuredoutput` classifies malformed output for proposal
|
||||
and validator call sites, which can turn those errors into warnings and rejected
|
||||
section/batch work. Other errors propagate through the runner.
|
||||
|
||||
## Scheduling and Concurrency
|
||||
|
||||
`internal/framework/llm/Scheduler` provides FIFO, context-aware permit gating.
|
||||
|
||||
CLI orchestration composes global and stage schedulers; runner instrumentation
|
||||
measures them. Limit definitions and inheritance belong to
|
||||
[configuration](../config.md#concurrency).
|
||||
|
||||
Scheduler release is guarded to avoid permit leaks on cancellation/error. Runner
|
||||
collects utilization observations; their serialized shape is defined by the
|
||||
[output-file contract](../integrations/output-files.md#utilization-diagnostics).
|
||||
|
||||
## Diagnostics and Redaction
|
||||
|
||||
`internal/framework/llm/DiagnosticsWriter` writes request/response/error artifacts.
|
||||
|
||||
Configured secrets are derived from `llm.ConfiguredSecrets(cfg)` and redacted from:
|
||||
- diagnostics payloads;
|
||||
- surfaced runtime/adapter errors.
|
||||
|
||||
## Changing LLM Integration or Scheduling
|
||||
|
||||
Review client, scheduler, redaction, and structured-output malformed-response
|
||||
coverage alongside runner orchestration and utilization tests. Use the
|
||||
[testing policy](../policy/testing.md) to select sufficient checks; preserve the
|
||||
[LLM integration contract](../integrations/openai-compatible-llm.md) and
|
||||
[architecture invariants](../policy/architecture.md#correction-and-concurrency-invariants).
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/framework/llm/openai_compatible_client_test.go`
|
||||
- `internal/framework/llm/scheduler_test.go`
|
||||
- `internal/framework/llm/diagnostics_test.go`
|
||||
- `internal/framework/responseschema/registry_test.go`
|
||||
- `internal/framework/structuredoutput/malformed_test.go`
|
||||
- `internal/framework/runner/*_test.go` (orchestration and utilization)
|
||||
Focused adapter tests use controlled local HTTP responses. Complete runner and
|
||||
CLI integration checks follow the caller and orchestration migration.
|
||||
|
||||
@@ -264,7 +264,15 @@ secret absence and overlay/CLI provenance, including inherited validation select
|
||||
|
||||
## Stage 5: Implement the Final PromptKit Structured-Call Adapter
|
||||
|
||||
**Status:** Pending. **Depends on:** Stages 3–4.
|
||||
**Status:** Complete (2026-09-24). Added role-bound prepared execution with
|
||||
target verification, validated-only decoding, typed safe failures, protected
|
||||
prepared/final-content snapshots, and logical-call and execution-event hooks.
|
||||
Removed the old direct HTTP client and role resolver; updated LLM integration
|
||||
documentation. `go test ./internal/framework/llm` and
|
||||
`go test ./internal/framework/structuredoutput` passed, including controlled
|
||||
HTTP repair/retry, target-drift, transport/protocol, and cancellation cases;
|
||||
`git diff --check` passed. Caller integration remains for Stages 6–8.
|
||||
**Depends on:** Stages 3–4.
|
||||
**Owns:** `internal/framework/llm`, the typed failure boundary in
|
||||
`internal/framework/structuredoutput`, and adapter tests.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user