audita/docs/architecture/structured-llm.md

Structured LLM Architecture

Purpose

This document describes Audita's structured LLM runtime boundary and adapter behavior.

Why Audita owns the adapter

Audita owns a small structured LLM adapter so that core runtime behavior is controlled inside the repository:

• request construction and schema handling are explicit and testable;
• retries, timeouts, cancellation, and error redaction are consistent across modules and validators;
• provider SDK types are not exposed outside the adapter boundary;
• dependency weight and transitive provider-specific behavior are reduced.

At runtime, the rest of Audita depends only on the internal contract:

• StructuredLLMClient
• CompleteStructured(ctx, req, out)

OpenAI-compatible request shape

At a conceptual level, Audita sends chat completion requests with:

• model
• messages (role/content pairs)
• response_format:
  • type = "json_schema"
  • json_schema.name (stable schema name)
  • json_schema.strict = true
  • json_schema.schema (registered JSON Schema payload)

The adapter uses OpenAI-compatible POST {base_url}/chat/completions over net/http.

Structured response schema registry

Structured response schemas are registered in internal/framework/responseschema with stable metadata:

• schema key
• schema ID
• schema version
• schema name (OpenAI-compatible response_format name)
• raw JSON Schema payload
• SHA-256 hash

Current schemas:

• correction_set:
  • id audita.correction_set
  • version v1
  • name audita_correction_set_v1
• validator_decision_set:
  • id audita.validator_decision_set
  • version v1
  • name audita_validator_decision_set_v1

Provider compatibility assumptions

Audita assumes an OpenAI-compatible chat-completions endpoint that:

• accepts message arrays with model selection;
• accepts response_format.type = json_schema;
• returns a completion with assistant message content and optional usage metadata.

Provider-specific differences are expected in strictness and error payload shapes, so the adapter treats provider output as untrusted until locally decoded.

Local decode and validation remain mandatory

Provider-level structured output is a transport guardrail, not final validation.

After receiving a response, Audita still:

• decodes assistant content into typed request-specific structs;
• validates proposal and validator payload invariants locally;
• enforces deterministic validator/cardinality rules before any transcript application.

This protects runtime correctness even when provider responses are malformed, partial, or semantically inconsistent.

Diagnostics and redaction

When structured schemas are used, diagnostics metadata records:

• schema ID
• schema version
• schema name
• schema hash

Diagnostics and surfaced errors preserve secret redaction:

• API keys and bearer tokens are redacted from request/response/error artifacts;
• redaction is applied before diagnostic files are written.

Runtime behavior guarantees

The structured LLM path preserves existing runtime guarantees:

• bounded LLM call execution through schedulers;
• context-aware cancellation and timeout propagation;
• retry behavior for transient failures and retryable malformed structured responses;
• deterministic module/chunk/proposal/validator behavior outside provider nondeterminism.