Replace structured LLM dependency with Audita adapter
This commit is contained in:
@@ -238,4 +238,5 @@ Optional external report output:
|
||||
## Documentation
|
||||
|
||||
- Architecture: [`docs/architecture.md`](docs/architecture.md)
|
||||
- Structured LLM adapter: [`docs/structured-llm.md`](docs/structured-llm.md)
|
||||
- Subprocess operations: [`docs/subprocess-operations.md`](docs/subprocess-operations.md)
|
||||
|
||||
@@ -21,7 +21,7 @@ Implemented today:
|
||||
- Runtime validator models and deterministic validators.
|
||||
- Deterministic validator-chain execution in the runner with cardinality enforcement.
|
||||
- Module-level validator decision/rejection reporting.
|
||||
- Internal structured LLM client contract plus an `instructor-go`-backed adapter package.
|
||||
- Internal structured LLM client contract plus an Audita-owned OpenAI-compatible structured LLM adapter package.
|
||||
- Bounded FIFO LLM scheduler infrastructure with context-aware permit handling.
|
||||
- Runtime primary/validation LLM effective-config resolution helpers with validation inheritance.
|
||||
- Generic JSON prompt/response diagnostics writer primitives with secret redaction.
|
||||
@@ -133,7 +133,7 @@ internal/framework/validators/
|
||||
llm_validators.go
|
||||
|
||||
internal/framework/llm/
|
||||
instructor_client.go
|
||||
openai_compatible_client.go
|
||||
scheduler.go
|
||||
effective_config.go
|
||||
diagnostics.go
|
||||
@@ -233,14 +233,33 @@ Current caveat:
|
||||
`internal/framework/contracts` now defines a typed structured-completion contract:
|
||||
- `StructuredLLMClient.CompleteStructured(ctx, req, out)`
|
||||
- caller-owned typed decode target via `out` pointer.
|
||||
- caller-selected structured response schema metadata via `StructuredCompletionRequest.ResponseSchema`.
|
||||
|
||||
`internal/framework/llm` provides `InstructorClient`, an internal adapter over `github.com/jxnl/instructor-go`:
|
||||
- configurable `base_url`, model, optional API key, retries, mode, HTTP client, and request timeout;
|
||||
`internal/framework/llm` provides `OpenAICompatibleClient`, a direct `net/http` adapter over OpenAI-compatible chat completions:
|
||||
- configurable `base_url`, model, optional API key, retries, HTTP client, and request timeout;
|
||||
- OpenAI-compatible endpoint behavior (for example OpenAI/OpenRouter/local-compatible base URLs);
|
||||
- default mode is JSON mode (`ModeJSON`), with optional tool-call mode (`ModeToolCall`);
|
||||
- request message translation from `contracts.LLMMessage` to chat-completions messages;
|
||||
- strict `response_format.type = json_schema` with registered structured response schemas (`strict: true`, schema name, and schema body);
|
||||
- response metadata mapping (provider/model/token usage) into Audita-owned response types;
|
||||
- API-key redaction in adapter-returned errors.
|
||||
- API-key redaction in adapter-returned errors;
|
||||
- context cancellation and timeout propagation through request contexts and HTTP client timeouts;
|
||||
- bounded retry behavior for transient request failures and malformed retryable structured responses.
|
||||
|
||||
Structured response schemas are owned by Audita in `internal/framework/responseschema` and currently include:
|
||||
- key `correction_set`:
|
||||
- id `audita.correction_set`
|
||||
- version `v1`
|
||||
- name `audita_correction_set_v1`
|
||||
- sha256 `05f8ff3fa04f68115c0cb1859d2656f51aa5c0bae8ff2470b2d4f6f531953195`
|
||||
- key `validator_decision_set`:
|
||||
- id `audita.validator_decision_set`
|
||||
- version `v1`
|
||||
- name `audita_validator_decision_set_v1`
|
||||
- sha256 `b73f4790b98fbb955f0aec5496dd8ce9a8fe14aa2f35c700b4b4e5634f106fd5`
|
||||
|
||||
Provider-level structured output is treated as a guardrail, not a trust boundary:
|
||||
- the adapter decodes assistant message content into caller-owned structs;
|
||||
- proposal-generation and validator layers continue local validation (shape, cardinality, confidence bounds, and proposal-index semantics) before changes can be applied.
|
||||
|
||||
Current runtime boundary:
|
||||
- the default CLI runtime path (without explicit module selection) instantiates the full production module sequence.
|
||||
@@ -252,6 +271,14 @@ Current runtime boundary:
|
||||
- primary/validation effective-config resolution helpers, including validation inheritance fallback to total LLM concurrency settings;
|
||||
- generic interaction diagnostics primitives that write machine-readable JSON artifacts for request metadata, request payload, response payload, and optional error payload with secret redaction.
|
||||
|
||||
Structured LLM diagnostics behavior:
|
||||
- proposal-generation and validator diagnostics include structured response schema metadata (`id`, `version`, `name`, `sha256`) when schema-driven calls are made;
|
||||
- API keys and bearer tokens are redacted from request/response/error diagnostics artifacts and surfaced errors.
|
||||
|
||||
Dependency posture:
|
||||
- the runtime no longer depends on `instructor-go`;
|
||||
- structured LLM behavior is implemented through Audita-owned code paths behind `StructuredLLMClient`.
|
||||
|
||||
LLM concurrency runtime behavior:
|
||||
- `total` concurrency bounds all proposal and validation LLM calls.
|
||||
- `proposal` concurrency adds a proposal-only sub-cap, composed with total.
|
||||
|
||||
@@ -67,6 +67,17 @@ The work is best handled in seven phases. The ordering is intentional:
|
||||
|
||||
Remove the heavy `instructor-go` dependency and replace it with a small Audita-owned OpenAI-compatible structured-output adapter.
|
||||
|
||||
## Implementation status (2026-05-13)
|
||||
|
||||
This workstream is now implemented in the repository:
|
||||
- production runtime uses an Audita-owned OpenAI-compatible structured LLM adapter (`internal/framework/llm/openai_compatible_client.go`);
|
||||
- `StructuredLLMClient` remains the stable internal boundary used by proposal generation and validators;
|
||||
- structured response schemas are registered with stable IDs, versions, names, and SHA-256 hashes (`internal/framework/responseschema`);
|
||||
- schema metadata is attached to structured completion requests and included in diagnostics metadata;
|
||||
- `instructor-go` has been removed from runtime code and module dependencies.
|
||||
|
||||
This status update applies only to the structured LLM dependency replacement workstream. Other roadmap workstreams remain planned unless explicitly marked otherwise.
|
||||
|
||||
This should happen before 1.0 because structured LLM calls are core runtime infrastructure. Replacing this layer after 1.0 would risk subtle compatibility changes in request construction, schema strictness, retry behavior, error reporting, diagnostics, and provider compatibility.
|
||||
|
||||
## Design direction
|
||||
|
||||
90
docs/structured-llm.md
Normal file
90
docs/structured-llm.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# Structured LLM Architecture
|
||||
|
||||
## Purpose
|
||||
|
||||
This document describes Audita's structured LLM runtime boundary and adapter behavior.
|
||||
|
||||
## Why Audita owns the adapter
|
||||
|
||||
Audita owns a small structured LLM adapter so that core runtime behavior is controlled inside the repository:
|
||||
- request construction and schema handling are explicit and testable;
|
||||
- retries, timeouts, cancellation, and error redaction are consistent across modules and validators;
|
||||
- provider SDK types are not exposed outside the adapter boundary;
|
||||
- dependency weight and transitive provider-specific behavior are reduced.
|
||||
|
||||
At runtime, the rest of Audita depends only on the internal contract:
|
||||
- `StructuredLLMClient`
|
||||
- `CompleteStructured(ctx, req, out)`
|
||||
|
||||
## OpenAI-compatible request shape
|
||||
|
||||
At a conceptual level, Audita sends chat completion requests with:
|
||||
- `model`
|
||||
- `messages` (role/content pairs)
|
||||
- `response_format`:
|
||||
- `type = "json_schema"`
|
||||
- `json_schema.name` (stable schema name)
|
||||
- `json_schema.strict = true`
|
||||
- `json_schema.schema` (registered JSON Schema payload)
|
||||
|
||||
The adapter uses OpenAI-compatible `POST {base_url}/chat/completions` over `net/http`.
|
||||
|
||||
## Structured response schema registry
|
||||
|
||||
Structured response schemas are registered in `internal/framework/responseschema` with stable metadata:
|
||||
- schema key
|
||||
- schema ID
|
||||
- schema version
|
||||
- schema name (OpenAI-compatible `response_format` name)
|
||||
- raw JSON Schema payload
|
||||
- SHA-256 hash
|
||||
|
||||
Current schemas:
|
||||
- `correction_set`:
|
||||
- id `audita.correction_set`
|
||||
- version `v1`
|
||||
- name `audita_correction_set_v1`
|
||||
- `validator_decision_set`:
|
||||
- id `audita.validator_decision_set`
|
||||
- version `v1`
|
||||
- name `audita_validator_decision_set_v1`
|
||||
|
||||
## Provider compatibility assumptions
|
||||
|
||||
Audita assumes an OpenAI-compatible chat-completions endpoint that:
|
||||
- accepts message arrays with model selection;
|
||||
- accepts `response_format.type = json_schema`;
|
||||
- returns a completion with assistant message content and optional usage metadata.
|
||||
|
||||
Provider-specific differences are expected in strictness and error payload shapes, so the adapter treats provider output as untrusted until locally decoded.
|
||||
|
||||
## Local decode and validation remain mandatory
|
||||
|
||||
Provider-level structured output is a transport guardrail, not final validation.
|
||||
|
||||
After receiving a response, Audita still:
|
||||
- decodes assistant content into typed request-specific structs;
|
||||
- validates proposal and validator payload invariants locally;
|
||||
- enforces deterministic validator/cardinality rules before any transcript application.
|
||||
|
||||
This protects runtime correctness even when provider responses are malformed, partial, or semantically inconsistent.
|
||||
|
||||
## Diagnostics and redaction
|
||||
|
||||
When structured schemas are used, diagnostics metadata records:
|
||||
- schema ID
|
||||
- schema version
|
||||
- schema name
|
||||
- schema hash
|
||||
|
||||
Diagnostics and surfaced errors preserve secret redaction:
|
||||
- API keys and bearer tokens are redacted from request/response/error artifacts;
|
||||
- redaction is applied before diagnostic files are written.
|
||||
|
||||
## Runtime behavior guarantees
|
||||
|
||||
The structured LLM path preserves existing runtime guarantees:
|
||||
- bounded LLM call execution through schedulers;
|
||||
- context-aware cancellation and timeout propagation;
|
||||
- retry behavior for transient failures and retryable malformed structured responses;
|
||||
- deterministic module/chunk/proposal/validator behavior outside provider nondeterminism.
|
||||
49
go.mod
49
go.mod
@@ -2,51 +2,10 @@ module gitea.maximumdirect.net/eric/audita
|
||||
|
||||
go 1.24.0
|
||||
|
||||
require (
|
||||
github.com/jxnl/instructor-go v0.0.0-20260420201153-d4111c5ef532
|
||||
github.com/sashabaranov/go-openai v1.41.2
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
require gopkg.in/yaml.v3 v3.0.1
|
||||
|
||||
require (
|
||||
cloud.google.com/go v0.123.0 // indirect
|
||||
cloud.google.com/go/auth v0.17.0 // indirect
|
||||
cloud.google.com/go/compute/metadata v0.9.0 // indirect
|
||||
github.com/aws/aws-sdk-go-v2 v1.39.2 // indirect
|
||||
github.com/aws/smithy-go v1.23.0 // indirect
|
||||
github.com/bahlo/generic-list-go v0.2.0 // indirect
|
||||
github.com/buger/jsonparser v1.1.2 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/cohere-ai/cohere-go/v2 v2.18.0 // indirect
|
||||
github.com/felixge/httpsnoop v1.0.4 // indirect
|
||||
github.com/gabriel-vasile/mimetype v1.4.13 // indirect
|
||||
github.com/go-logr/logr v1.4.3 // indirect
|
||||
github.com/go-logr/stdr v1.2.2 // indirect
|
||||
github.com/go-playground/locales v0.14.1 // indirect
|
||||
github.com/go-playground/universal-translator v0.18.1 // indirect
|
||||
github.com/go-playground/validator/v10 v10.30.0 // indirect
|
||||
github.com/google/go-cmp v0.7.0 // indirect
|
||||
github.com/google/s2a-go v0.1.9 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/googleapis/enterprise-certificate-proxy v0.3.6 // indirect
|
||||
github.com/googleapis/gax-go/v2 v2.15.0 // indirect
|
||||
github.com/gorilla/websocket v1.5.3 // indirect
|
||||
github.com/invopop/jsonschema v0.13.0 // indirect
|
||||
github.com/leodido/go-urn v1.4.0 // indirect
|
||||
github.com/liushuangls/go-anthropic/v2 v2.18.0 // indirect
|
||||
github.com/mailru/easyjson v0.9.1 // indirect
|
||||
github.com/wk8/go-ordered-map/v2 v2.1.8 // indirect
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 // indirect
|
||||
go.opentelemetry.io/otel v1.39.0 // indirect
|
||||
go.opentelemetry.io/otel/metric v1.39.0 // indirect
|
||||
go.opentelemetry.io/otel/trace v1.39.0 // indirect
|
||||
golang.org/x/crypto v0.48.0 // indirect
|
||||
golang.org/x/net v0.50.0 // indirect
|
||||
golang.org/x/sys v0.41.0 // indirect
|
||||
golang.org/x/text v0.34.0 // indirect
|
||||
google.golang.org/genai v1.52.1 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 // indirect
|
||||
google.golang.org/grpc v1.79.3 // indirect
|
||||
google.golang.org/protobuf v1.36.10 // indirect
|
||||
github.com/kr/pretty v0.3.1 // indirect
|
||||
github.com/rogpeppe/go-internal v1.14.1 // indirect
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
|
||||
)
|
||||
|
||||
109
go.sum
109
go.sum
@@ -1,112 +1,15 @@
|
||||
cloud.google.com/go v0.123.0 h1:2NAUJwPR47q+E35uaJeYoNhuNEM9kM8SjgRgdeOJUSE=
|
||||
cloud.google.com/go v0.123.0/go.mod h1:xBoMV08QcqUGuPW65Qfm1o9Y4zKZBpGS+7bImXLTAZU=
|
||||
cloud.google.com/go/auth v0.17.0 h1:74yCm7hCj2rUyyAocqnFzsAYXgJhrG26XCFimrc/Kz4=
|
||||
cloud.google.com/go/auth v0.17.0/go.mod h1:6wv/t5/6rOPAX4fJiRjKkJCvswLwdet7G8+UGXt7nCQ=
|
||||
cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs=
|
||||
cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10=
|
||||
github.com/aws/aws-sdk-go-v2 v1.39.2 h1:EJLg8IdbzgeD7xgvZ+I8M1e0fL0ptn/M47lianzth0I=
|
||||
github.com/aws/aws-sdk-go-v2 v1.39.2/go.mod h1:sDioUELIUO9Znk23YVmIk86/9DOpkbyyVb1i/gUNFXY=
|
||||
github.com/aws/smithy-go v1.23.0 h1:8n6I3gXzWJB2DxBDnfxgBaSX6oe0d/t10qGz7OKqMCE=
|
||||
github.com/aws/smithy-go v1.23.0/go.mod h1:t1ufH5HMublsJYulve2RKmHDC15xu1f26kHCp/HgceI=
|
||||
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
|
||||
github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
|
||||
github.com/buger/jsonparser v1.1.2 h1:frqHqw7otoVbk5M8LlE/L7HTnIq2v9RX6EJ48i9AxJk=
|
||||
github.com/buger/jsonparser v1.1.2/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/cohere-ai/cohere-go/v2 v2.18.0 h1:Y5NyY+YTCN6XrrxHWicgVdpBi4tSBa0bjesD2fJGdvQ=
|
||||
github.com/cohere-ai/cohere-go/v2 v2.18.0/go.mod h1:MuiJkCxlR18BDV2qQPbz2Yb/OCVphT1y6nD2zYaKeR0=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
|
||||
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
|
||||
github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
|
||||
github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
|
||||
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
|
||||
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
|
||||
github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
|
||||
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
|
||||
github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/NuW9t0sjnWqG1slY=
|
||||
github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY=
|
||||
github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
|
||||
github.com/go-playground/validator/v10 v10.30.0 h1:5YBPNs273uzsZJD1I8uiB4Aqg9sN6sMDVX3s6LxmhWU=
|
||||
github.com/go-playground/validator/v10 v10.30.0/go.mod h1:oSuBIQzuJxL//3MelwSLD5hc2Tu889bF0Idm9Dg26cM=
|
||||
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0=
|
||||
github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/googleapis/enterprise-certificate-proxy v0.3.6 h1:GW/XbdyBFQ8Qe+YAmFU9uHLo7OnF5tL52HFAgMmyrf4=
|
||||
github.com/googleapis/enterprise-certificate-proxy v0.3.6/go.mod h1:MkHOF77EYAE7qfSuSS9PU6g4Nt4e11cnsDUowfwewLA=
|
||||
github.com/googleapis/gax-go/v2 v2.15.0 h1:SyjDc1mGgZU5LncH8gimWo9lW1DtIfPibOG81vgd/bo=
|
||||
github.com/googleapis/gax-go/v2 v2.15.0/go.mod h1:zVVkkxAQHa1RQpg9z2AUCMnKhi0Qld9rcmyfL1OZhoc=
|
||||
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
|
||||
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
|
||||
github.com/invopop/jsonschema v0.13.0 h1:KvpoAJWEjR3uD9Kbm2HWJmqsEaHt8lBUpd0qHcIi21E=
|
||||
github.com/invopop/jsonschema v0.13.0/go.mod h1:ffZ5Km5SWWRAIN6wbDXItl95euhFz2uON45H2qjYt+0=
|
||||
github.com/jxnl/instructor-go v0.0.0-20260420201153-d4111c5ef532 h1:9Gd7ONgzQLX+9jKrkAdE9Ol9Tp6/w4tfRH14dayoemI=
|
||||
github.com/jxnl/instructor-go v0.0.0-20260420201153-d4111c5ef532/go.mod h1:A4cY903g2j4nuOIHvacaHBEf0SwAOh1Yswx/XAgqPgo=
|
||||
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
|
||||
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
|
||||
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
|
||||
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
|
||||
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
|
||||
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
|
||||
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
|
||||
github.com/liushuangls/go-anthropic/v2 v2.18.0 h1:q47sqQ1bHsItPTcIj7RVOnpgrP79SgaACsMQ0RW2oYA=
|
||||
github.com/liushuangls/go-anthropic/v2 v2.18.0/go.mod h1:a550cJXPoTG2FL3DvfKG2zzD5O2vjgvo4tHtoGPzFLU=
|
||||
github.com/mailru/easyjson v0.9.1 h1:LbtsOm5WAswyWbvTEOqhypdPeZzHavpZx96/n553mR8=
|
||||
github.com/mailru/easyjson v0.9.1/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUto+XFtTU=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
|
||||
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
|
||||
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
|
||||
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
|
||||
github.com/sashabaranov/go-openai v1.41.2 h1:vfPRBZNMpnqu8ELsclWcAvF19lDNgh1t6TVfFFOPiSM=
|
||||
github.com/sashabaranov/go-openai v1.41.2/go.mod h1:lj5b/K+zjTSFxVLijLSTDZuP7adOgerWeFyZLUhAKRg=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/wk8/go-ordered-map/v2 v2.1.8 h1:5h/BUHu93oj4gIdvHHHGsScSTMijfx5PeYkE/fJgbpc=
|
||||
github.com/wk8/go-ordered-map/v2 v2.1.8/go.mod h1:5nJHM5DyteebpVlHnWMV0rPz6Zp7+xBAnxjb1X5vnTw=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 h1:RbKq8BG0FI8OiXhBfcRtqqHcZcka+gU3cskNuf05R18=
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0/go.mod h1:h06DGIukJOevXaj/xrNjhi/2098RZzcLTbc0jDAUbsg=
|
||||
go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48=
|
||||
go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8=
|
||||
go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0=
|
||||
go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs=
|
||||
go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18=
|
||||
go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew=
|
||||
go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI=
|
||||
go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA=
|
||||
golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
|
||||
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
|
||||
golang.org/x/net v0.50.0 h1:ucWh9eiCGyDR3vtzso0WMQinm2Dnt8cFMuQa9K33J60=
|
||||
golang.org/x/net v0.50.0/go.mod h1:UgoSli3F/pBgdJBHCTc+tp3gmrU4XswgGRgtnwWTfyM=
|
||||
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
|
||||
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
||||
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
|
||||
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||
golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
|
||||
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
|
||||
gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=
|
||||
gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E=
|
||||
google.golang.org/genai v1.52.1 h1:dYoljKtLDXMiBdVaClSJ/ZPwZ7j1N0lGjMhwOKOQUlk=
|
||||
google.golang.org/genai v1.52.1/go.mod h1:A3kkl0nyBjyFlNjgxIwKq70julKbIxpSxqKO5gw/gmk=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 h1:gRkg/vSppuSQoDjxyiGfN4Upv/h/DQmIR10ZU8dh4Ww=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217/go.mod h1:7i2o+ce6H/6BluujYR+kqX3GKH+dChPTQU19wjRPiGk=
|
||||
google.golang.org/grpc v1.79.3 h1:sybAEdRIEtvcD68Gx7dmnwjZKlyfuc61Dyo9pGXXkKE=
|
||||
google.golang.org/grpc v1.79.3/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
|
||||
google.golang.org/protobuf v1.36.10 h1:AYd7cD/uASjIL6Q9LiTjz8JLcrh/88q5UObnmY3aOOE=
|
||||
google.golang.org/protobuf v1.36.10/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||
|
||||
@@ -175,7 +175,7 @@ var processRunner = func(inv processInvocation, stdout io.Writer) (*normalizatio
|
||||
proposalLLMClient = noOpStructuredLLMClient{}
|
||||
} else {
|
||||
primaryCfg := llm.ResolvePrimaryConfig(inv.Config)
|
||||
client, clientErr := llm.NewInstructorClient(primaryCfg.ToInstructorClientConfig(llm.ModeJSON, nil))
|
||||
client, clientErr := llm.NewOpenAICompatibleClient(primaryCfg.ToOpenAICompatibleClientConfig(nil))
|
||||
if clientErr != nil {
|
||||
return fail("runner_setup", clientErr, nil)
|
||||
}
|
||||
@@ -187,7 +187,7 @@ var processRunner = func(inv processInvocation, stdout io.Writer) (*normalizatio
|
||||
validationLLMClient = noOpStructuredLLMClient{}
|
||||
} else {
|
||||
validationCfg := llm.ResolveValidationConfig(inv.Config)
|
||||
client, clientErr := llm.NewInstructorClient(validationCfg.ToInstructorClientConfig(llm.ModeJSON, nil))
|
||||
client, clientErr := llm.NewOpenAICompatibleClient(validationCfg.ToOpenAICompatibleClientConfig(nil))
|
||||
if clientErr != nil {
|
||||
return fail("runner_setup", clientErr, nil)
|
||||
}
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/responseschema"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||
)
|
||||
|
||||
@@ -38,10 +39,10 @@ type Validator interface {
|
||||
|
||||
// StructuredCompletionRequest is a transport-neutral structured completion request.
|
||||
type StructuredCompletionRequest struct {
|
||||
StageName string `json:"stage_name"`
|
||||
Messages []LLMMessage `json:"messages"`
|
||||
Model string `json:"model,omitempty"`
|
||||
ResponseSchema json.RawMessage `json:"response_schema,omitempty"`
|
||||
StageName string `json:"stage_name"`
|
||||
Messages []LLMMessage `json:"messages"`
|
||||
Model string `json:"model,omitempty"`
|
||||
ResponseSchema *responseschema.Schema `json:"response_schema,omitempty"`
|
||||
}
|
||||
|
||||
// StructuredCompletionResponse is a transport-neutral structured completion response payload.
|
||||
|
||||
50
internal/framework/llm/client_common.go
Normal file
50
internal/framework/llm/client_common.go
Normal file
@@ -0,0 +1,50 @@
|
||||
package llm
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"reflect"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
func resolvedHTTPClient(base *http.Client, timeout time.Duration) *http.Client {
|
||||
if base == nil {
|
||||
if timeout <= 0 {
|
||||
return http.DefaultClient
|
||||
}
|
||||
return &http.Client{Timeout: timeout}
|
||||
}
|
||||
|
||||
if timeout <= 0 {
|
||||
return base
|
||||
}
|
||||
|
||||
cloned := *base
|
||||
cloned.Timeout = timeout
|
||||
return &cloned
|
||||
}
|
||||
|
||||
func validateOutputTarget(out any) error {
|
||||
if out == nil {
|
||||
return fmt.Errorf("output target must not be nil")
|
||||
}
|
||||
value := reflect.ValueOf(out)
|
||||
if value.Kind() != reflect.Ptr || value.IsNil() {
|
||||
return fmt.Errorf("output target must be a non-nil pointer")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func sanitizeError(err error, apiKey string) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
msg := err.Error()
|
||||
key := strings.TrimSpace(apiKey)
|
||||
if key != "" {
|
||||
msg = strings.ReplaceAll(msg, key, "[REDACTED]")
|
||||
msg = strings.ReplaceAll(msg, "Bearer "+key, "Bearer [REDACTED]")
|
||||
}
|
||||
return fmt.Errorf("%s", msg)
|
||||
}
|
||||
25
internal/framework/llm/dependency_guard_test.go
Normal file
25
internal/framework/llm/dependency_guard_test.go
Normal file
@@ -0,0 +1,25 @@
|
||||
package llm
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestModuleDoesNotReferenceInstructorGo(t *testing.T) {
|
||||
_, file, _, ok := runtime.Caller(0)
|
||||
if !ok {
|
||||
t.Fatalf("runtime caller lookup failed")
|
||||
}
|
||||
repoRoot := filepath.Clean(filepath.Join(filepath.Dir(file), "..", "..", ".."))
|
||||
|
||||
modBytes, err := os.ReadFile(filepath.Join(repoRoot, "go.mod"))
|
||||
if err != nil {
|
||||
t.Fatalf("read go.mod: %v", err)
|
||||
}
|
||||
if strings.Contains(string(modBytes), "github.com/jxnl/instructor-go") {
|
||||
t.Fatalf("unexpected instructor-go reference in go.mod")
|
||||
}
|
||||
}
|
||||
@@ -30,15 +30,14 @@ func ResolveValidationConfig(cfg config.Config) EffectiveConfig {
|
||||
return resolveFromLLMConfig(cfg.EffectiveValidationLLMConfig(), cfg.EffectiveValidationLLMConcurrency())
|
||||
}
|
||||
|
||||
// ToInstructorClientConfig converts an effective runtime config into adapter
|
||||
// config while keeping instructor-go types fully internal to this package.
|
||||
func (c EffectiveConfig) ToInstructorClientConfig(mode Mode, httpClient *http.Client) InstructorClientConfig {
|
||||
return InstructorClientConfig{
|
||||
// ToOpenAICompatibleClientConfig converts an effective runtime config into
|
||||
// direct HTTP adapter config.
|
||||
func (c EffectiveConfig) ToOpenAICompatibleClientConfig(httpClient *http.Client) OpenAICompatibleClientConfig {
|
||||
return OpenAICompatibleClientConfig{
|
||||
BaseURL: c.BaseURL,
|
||||
Model: c.Model,
|
||||
APIKey: c.APIKey,
|
||||
MaxRetries: c.MaxRetries,
|
||||
Mode: mode,
|
||||
HTTPClient: httpClient,
|
||||
RequestTimeout: c.RequestTimeout,
|
||||
}
|
||||
|
||||
@@ -1,219 +0,0 @@
|
||||
package llm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"reflect"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||
|
||||
"github.com/jxnl/instructor-go/pkg/instructor"
|
||||
openai "github.com/sashabaranov/go-openai"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultMaxRetries = 3
|
||||
)
|
||||
|
||||
type Mode string
|
||||
|
||||
const (
|
||||
ModeJSON Mode = "json"
|
||||
ModeToolCall Mode = "tool_call"
|
||||
)
|
||||
|
||||
type InstructorClientConfig struct {
|
||||
BaseURL string
|
||||
Model string
|
||||
APIKey string
|
||||
MaxRetries int
|
||||
Mode Mode
|
||||
HTTPClient *http.Client
|
||||
RequestTimeout time.Duration
|
||||
}
|
||||
|
||||
// chatCompletionClient is intentionally narrow to avoid leaking provider types
|
||||
// outside this adapter package.
|
||||
type chatCompletionClient interface {
|
||||
CreateChatCompletion(ctx context.Context, request openai.ChatCompletionRequest, responseType any) (response openai.ChatCompletionResponse, err error)
|
||||
}
|
||||
|
||||
// InstructorClient adapts instructor-go behind Audita's internal structured
|
||||
// LLM interface.
|
||||
type InstructorClient struct {
|
||||
cfg InstructorClientConfig
|
||||
client chatCompletionClient
|
||||
}
|
||||
|
||||
var _ contracts.StructuredLLMClient = (*InstructorClient)(nil)
|
||||
|
||||
func NewInstructorClient(cfg InstructorClientConfig) (*InstructorClient, error) {
|
||||
normalized, err := normalizeConfig(cfg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
openaiConfig := openai.DefaultConfig(normalized.APIKey)
|
||||
openaiConfig.BaseURL = normalized.BaseURL
|
||||
openaiConfig.HTTPClient = resolvedHTTPClient(normalized.HTTPClient, normalized.RequestTimeout)
|
||||
|
||||
mode, err := toInstructorMode(normalized.Mode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
client := instructor.FromOpenAI(
|
||||
openai.NewClientWithConfig(openaiConfig),
|
||||
instructor.WithMode(mode),
|
||||
instructor.WithMaxRetries(normalized.MaxRetries),
|
||||
)
|
||||
|
||||
return &InstructorClient{
|
||||
cfg: normalized,
|
||||
client: client,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (c *InstructorClient) CompleteStructured(
|
||||
ctx context.Context,
|
||||
req contracts.StructuredCompletionRequest,
|
||||
out any,
|
||||
) (contracts.StructuredCompletionResponse, error) {
|
||||
if err := validateOutputTarget(out); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
|
||||
model := strings.TrimSpace(req.Model)
|
||||
if model == "" {
|
||||
model = c.cfg.Model
|
||||
}
|
||||
if model == "" {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion model must not be empty")
|
||||
}
|
||||
|
||||
messages, err := toOpenAIMessages(req.Messages)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
|
||||
chatRequest := openai.ChatCompletionRequest{
|
||||
Model: model,
|
||||
Messages: messages,
|
||||
}
|
||||
|
||||
resp, err := c.client.CreateChatCompletion(ctx, chatRequest, out)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, sanitizeError(err, c.cfg.APIKey)
|
||||
}
|
||||
|
||||
content, err := json.Marshal(out)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("marshal structured completion output: %w", err)
|
||||
}
|
||||
|
||||
return contracts.StructuredCompletionResponse{
|
||||
Content: content,
|
||||
Provider: "openai-compatible",
|
||||
Model: model,
|
||||
PromptTokens: resp.Usage.PromptTokens,
|
||||
CompletionTokens: resp.Usage.CompletionTokens,
|
||||
TotalTokens: resp.Usage.TotalTokens,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func normalizeConfig(cfg InstructorClientConfig) (InstructorClientConfig, error) {
|
||||
cfg.BaseURL = strings.TrimSpace(cfg.BaseURL)
|
||||
cfg.Model = strings.TrimSpace(cfg.Model)
|
||||
cfg.APIKey = strings.TrimSpace(cfg.APIKey)
|
||||
if cfg.MaxRetries == 0 {
|
||||
cfg.MaxRetries = defaultMaxRetries
|
||||
}
|
||||
if cfg.MaxRetries < 0 {
|
||||
return InstructorClientConfig{}, fmt.Errorf("max retries must be zero or greater")
|
||||
}
|
||||
if cfg.BaseURL == "" {
|
||||
return InstructorClientConfig{}, fmt.Errorf("base URL must not be empty")
|
||||
}
|
||||
if cfg.Model == "" {
|
||||
return InstructorClientConfig{}, fmt.Errorf("model must not be empty")
|
||||
}
|
||||
if cfg.Mode == "" {
|
||||
cfg.Mode = ModeJSON
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func toInstructorMode(mode Mode) (instructor.Mode, error) {
|
||||
switch mode {
|
||||
case ModeJSON:
|
||||
return instructor.ModeJSON, nil
|
||||
case ModeToolCall:
|
||||
return instructor.ModeToolCall, nil
|
||||
default:
|
||||
return "", fmt.Errorf("unsupported LLM mode %q", mode)
|
||||
}
|
||||
}
|
||||
|
||||
func resolvedHTTPClient(base *http.Client, timeout time.Duration) *http.Client {
|
||||
if base == nil {
|
||||
if timeout <= 0 {
|
||||
return http.DefaultClient
|
||||
}
|
||||
return &http.Client{Timeout: timeout}
|
||||
}
|
||||
|
||||
if timeout <= 0 {
|
||||
return base
|
||||
}
|
||||
|
||||
cloned := *base
|
||||
cloned.Timeout = timeout
|
||||
return &cloned
|
||||
}
|
||||
|
||||
func toOpenAIMessages(messages []contracts.LLMMessage) ([]openai.ChatCompletionMessage, error) {
|
||||
result := make([]openai.ChatCompletionMessage, len(messages))
|
||||
for i, message := range messages {
|
||||
role := strings.TrimSpace(message.Role)
|
||||
content := strings.TrimSpace(message.Content)
|
||||
if role == "" {
|
||||
return nil, fmt.Errorf("message[%d] role must not be empty", i)
|
||||
}
|
||||
if content == "" {
|
||||
return nil, fmt.Errorf("message[%d] content must not be empty", i)
|
||||
}
|
||||
result[i] = openai.ChatCompletionMessage{
|
||||
Role: role,
|
||||
Content: content,
|
||||
}
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func validateOutputTarget(out any) error {
|
||||
if out == nil {
|
||||
return fmt.Errorf("output target must not be nil")
|
||||
}
|
||||
value := reflect.ValueOf(out)
|
||||
if value.Kind() != reflect.Ptr || value.IsNil() {
|
||||
return fmt.Errorf("output target must be a non-nil pointer")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func sanitizeError(err error, apiKey string) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
msg := err.Error()
|
||||
key := strings.TrimSpace(apiKey)
|
||||
if key != "" {
|
||||
msg = strings.ReplaceAll(msg, key, "[REDACTED]")
|
||||
msg = strings.ReplaceAll(msg, "Bearer "+key, "Bearer [REDACTED]")
|
||||
}
|
||||
return fmt.Errorf("%s", msg)
|
||||
}
|
||||
@@ -1,282 +0,0 @@
|
||||
package llm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||
|
||||
openai "github.com/sashabaranov/go-openai"
|
||||
)
|
||||
|
||||
func TestNewInstructorClientValidation(t *testing.T) {
|
||||
_, err := NewInstructorClient(InstructorClientConfig{
|
||||
BaseURL: " ",
|
||||
Model: "x",
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "base URL") {
|
||||
t.Fatalf("expected base URL validation error, got %v", err)
|
||||
}
|
||||
|
||||
_, err = NewInstructorClient(InstructorClientConfig{
|
||||
BaseURL: "http://localhost:1234/v1",
|
||||
Model: " ",
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "model") {
|
||||
t.Fatalf("expected model validation error, got %v", err)
|
||||
}
|
||||
|
||||
_, err = NewInstructorClient(InstructorClientConfig{
|
||||
BaseURL: "http://localhost:1234/v1",
|
||||
Model: "test-model",
|
||||
MaxRetries: -1,
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "max retries") {
|
||||
t.Fatalf("expected retries validation error, got %v", err)
|
||||
}
|
||||
|
||||
_, err = NewInstructorClient(InstructorClientConfig{
|
||||
BaseURL: "http://localhost:1234/v1",
|
||||
Model: "test-model",
|
||||
Mode: "unsupported",
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "unsupported LLM mode") {
|
||||
t.Fatalf("expected mode validation error, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInstructorClientCompleteStructuredSuccessNoAPIKey(t *testing.T) {
|
||||
var seenPath string
|
||||
var seenHost string
|
||||
var seenAuth string
|
||||
var seenModel string
|
||||
client, err := NewInstructorClient(InstructorClientConfig{
|
||||
BaseURL: "https://local-compat.example/v1",
|
||||
Model: "test-model",
|
||||
HTTPClient: &http.Client{
|
||||
Transport: roundTripFunc(func(r *http.Request) (*http.Response, error) {
|
||||
seenPath = r.URL.Path
|
||||
seenHost = r.URL.Host
|
||||
seenAuth = r.Header.Get("Authorization")
|
||||
|
||||
var req map[string]any
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
t.Fatalf("decode request: %v", err)
|
||||
}
|
||||
model, _ := req["model"].(string)
|
||||
seenModel = model
|
||||
|
||||
return newJSONHTTPResponse(http.StatusOK, chatCompletionResponseBody(`{"name":"Robby","age":22}`)), nil
|
||||
}),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewInstructorClient: %v", err)
|
||||
}
|
||||
|
||||
type person struct {
|
||||
Name string `json:"name"`
|
||||
Age int `json:"age"`
|
||||
}
|
||||
var out person
|
||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
StageName: "proposal:test",
|
||||
Messages: []contracts.LLMMessage{
|
||||
{Role: openai.ChatMessageRoleUser, Content: "extract person"},
|
||||
},
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured: %v", err)
|
||||
}
|
||||
|
||||
if seenPath != "/v1/chat/completions" {
|
||||
t.Fatalf("unexpected request path: %q", seenPath)
|
||||
}
|
||||
if seenHost != "local-compat.example" {
|
||||
t.Fatalf("unexpected request host: %q", seenHost)
|
||||
}
|
||||
if seenModel != "test-model" {
|
||||
t.Fatalf("unexpected model: %q", seenModel)
|
||||
}
|
||||
if seenAuth != "" {
|
||||
t.Fatalf("expected empty Authorization header for empty API key, got %q", seenAuth)
|
||||
}
|
||||
if out.Name != "Robby" || out.Age != 22 {
|
||||
t.Fatalf("unexpected output: %+v", out)
|
||||
}
|
||||
if resp.Provider != "openai-compatible" || resp.Model != "test-model" {
|
||||
t.Fatalf("unexpected response metadata: %+v", resp)
|
||||
}
|
||||
if resp.TotalTokens != 18 || resp.PromptTokens != 11 || resp.CompletionTokens != 7 {
|
||||
t.Fatalf("unexpected usage metadata: %+v", resp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInstructorClientRetriesOnMalformedJSON(t *testing.T) {
|
||||
var attempts int32
|
||||
client, err := NewInstructorClient(InstructorClientConfig{
|
||||
BaseURL: "https://retry.example/v1",
|
||||
Model: "test-model",
|
||||
MaxRetries: 1,
|
||||
HTTPClient: &http.Client{
|
||||
Transport: roundTripFunc(func(r *http.Request) (*http.Response, error) {
|
||||
current := atomic.AddInt32(&attempts, 1)
|
||||
if current == 1 {
|
||||
return newJSONHTTPResponse(http.StatusOK, chatCompletionResponseBody(`{"name":"broken"`)), nil
|
||||
}
|
||||
return newJSONHTTPResponse(http.StatusOK, chatCompletionResponseBody(`{"name":"Recovered","age":30}`)), nil
|
||||
}),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewInstructorClient: %v", err)
|
||||
}
|
||||
|
||||
type person struct {
|
||||
Name string `json:"name"`
|
||||
Age int `json:"age"`
|
||||
}
|
||||
var out person
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{
|
||||
{Role: openai.ChatMessageRoleUser, Content: "extract person"},
|
||||
},
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured: %v", err)
|
||||
}
|
||||
if out.Name != "Recovered" || out.Age != 30 {
|
||||
t.Fatalf("unexpected output after retry: %+v", out)
|
||||
}
|
||||
if got := atomic.LoadInt32(&attempts); got != 2 {
|
||||
t.Fatalf("expected 2 attempts, got %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInstructorClientContextDeadlinePropagates(t *testing.T) {
|
||||
client, err := NewInstructorClient(InstructorClientConfig{
|
||||
BaseURL: "https://slow.example/v1",
|
||||
Model: "test-model",
|
||||
HTTPClient: &http.Client{
|
||||
Transport: roundTripFunc(func(r *http.Request) (*http.Response, error) {
|
||||
select {
|
||||
case <-r.Context().Done():
|
||||
return nil, r.Context().Err()
|
||||
case <-time.After(250 * time.Millisecond):
|
||||
return newJSONHTTPResponse(http.StatusOK, chatCompletionResponseBody(`{"name":"slow","age":1}`)), nil
|
||||
}
|
||||
}),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewInstructorClient: %v", err)
|
||||
}
|
||||
|
||||
type person struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
var out person
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
_, err = client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{
|
||||
{Role: openai.ChatMessageRoleUser, Content: "extract person"},
|
||||
},
|
||||
}, &out)
|
||||
if err == nil {
|
||||
t.Fatalf("expected context deadline error")
|
||||
}
|
||||
if !strings.Contains(strings.ToLower(err.Error()), "context deadline") {
|
||||
t.Fatalf("expected deadline-related error, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInstructorClientSanitizesAPIKeyInErrors(t *testing.T) {
|
||||
apiKey := "super-secret-key"
|
||||
client := &InstructorClient{
|
||||
cfg: InstructorClientConfig{
|
||||
BaseURL: "http://localhost:1234/v1",
|
||||
Model: "test-model",
|
||||
APIKey: apiKey,
|
||||
Mode: ModeJSON,
|
||||
},
|
||||
client: fakeChatCompletionClient{
|
||||
err: fmt.Errorf("provider failed with Authorization: Bearer %s", apiKey),
|
||||
},
|
||||
}
|
||||
|
||||
var out map[string]any
|
||||
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{
|
||||
{Role: openai.ChatMessageRoleUser, Content: "extract"},
|
||||
},
|
||||
}, &out)
|
||||
if err == nil {
|
||||
t.Fatalf("expected error")
|
||||
}
|
||||
if strings.Contains(err.Error(), apiKey) {
|
||||
t.Fatalf("error leaked API key: %v", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "[REDACTED]") {
|
||||
t.Fatalf("expected redaction marker in error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
type fakeChatCompletionClient struct {
|
||||
err error
|
||||
}
|
||||
|
||||
func (f fakeChatCompletionClient) CreateChatCompletion(ctx context.Context, request openai.ChatCompletionRequest, responseType any) (response openai.ChatCompletionResponse, err error) {
|
||||
_ = ctx
|
||||
_ = request
|
||||
_ = responseType
|
||||
return openai.ChatCompletionResponse{}, f.err
|
||||
}
|
||||
|
||||
type roundTripFunc func(*http.Request) (*http.Response, error)
|
||||
|
||||
func (f roundTripFunc) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
return f(r)
|
||||
}
|
||||
|
||||
func newJSONHTTPResponse(status int, body string) *http.Response {
|
||||
return &http.Response{
|
||||
StatusCode: status,
|
||||
Header: http.Header{"Content-Type": []string{"application/json"}},
|
||||
Body: io.NopCloser(strings.NewReader(body)),
|
||||
}
|
||||
}
|
||||
|
||||
func chatCompletionResponseBody(content string) string {
|
||||
body, _ := json.Marshal(map[string]any{
|
||||
"id": "chatcmpl-test",
|
||||
"object": "chat.completion",
|
||||
"created": 12345,
|
||||
"model": "test-model",
|
||||
"choices": []map[string]any{
|
||||
{
|
||||
"index": 0,
|
||||
"message": map[string]any{
|
||||
"role": "assistant",
|
||||
"content": content,
|
||||
},
|
||||
"finish_reason": "stop",
|
||||
},
|
||||
},
|
||||
"usage": map[string]any{
|
||||
"prompt_tokens": 11,
|
||||
"completion_tokens": 7,
|
||||
"total_tokens": 18,
|
||||
},
|
||||
})
|
||||
return string(body)
|
||||
}
|
||||
370
internal/framework/llm/openai_compatible_client.go
Normal file
370
internal/framework/llm/openai_compatible_client.go
Normal file
@@ -0,0 +1,370 @@
|
||||
package llm
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||
)
|
||||
|
||||
const defaultOpenAICompatibleMaxRetries = 3
|
||||
|
||||
// OpenAICompatibleClientConfig configures the direct HTTP structured-output adapter.
|
||||
type OpenAICompatibleClientConfig struct {
|
||||
BaseURL string
|
||||
Model string
|
||||
APIKey string
|
||||
MaxRetries int
|
||||
HTTPClient *http.Client
|
||||
RequestTimeout time.Duration
|
||||
}
|
||||
|
||||
// OpenAICompatibleClient sends OpenAI-compatible chat-completion requests with
|
||||
// response_format.type=json_schema.
|
||||
type OpenAICompatibleClient struct {
|
||||
cfg OpenAICompatibleClientConfig
|
||||
httpClient *http.Client
|
||||
}
|
||||
|
||||
var _ contracts.StructuredLLMClient = (*OpenAICompatibleClient)(nil)
|
||||
|
||||
func NewOpenAICompatibleClient(cfg OpenAICompatibleClientConfig) (*OpenAICompatibleClient, error) {
|
||||
normalized, err := normalizeOpenAICompatibleConfig(cfg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &OpenAICompatibleClient{
|
||||
cfg: normalized,
|
||||
httpClient: resolvedHTTPClient(normalized.HTTPClient, normalized.RequestTimeout),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (c *OpenAICompatibleClient) CompleteStructured(
|
||||
ctx context.Context,
|
||||
req contracts.StructuredCompletionRequest,
|
||||
out any,
|
||||
) (contracts.StructuredCompletionResponse, error) {
|
||||
if err := validateOutputTarget(out); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
|
||||
model := strings.TrimSpace(req.Model)
|
||||
if model == "" {
|
||||
model = c.cfg.Model
|
||||
}
|
||||
if model == "" {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion model must not be empty")
|
||||
}
|
||||
if req.ResponseSchema == nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion response schema is required")
|
||||
}
|
||||
if strings.TrimSpace(req.ResponseSchema.Name) == "" {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion response schema name must not be empty")
|
||||
}
|
||||
if len(req.ResponseSchema.JSONSchema) == 0 || !json.Valid(req.ResponseSchema.JSONSchema) {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion response schema JSON must be valid")
|
||||
}
|
||||
|
||||
messages, err := toOpenAICompatibleMessages(req.Messages)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
|
||||
endpoint := buildChatCompletionsURL(c.cfg.BaseURL)
|
||||
|
||||
var lastErr error
|
||||
for attempt := 0; attempt <= c.cfg.MaxRetries; attempt++ {
|
||||
content, metadata, callErr := c.completeStructuredOnce(
|
||||
ctx,
|
||||
endpoint,
|
||||
model,
|
||||
messages,
|
||||
req.ResponseSchema.Name,
|
||||
req.ResponseSchema.JSONSchema,
|
||||
)
|
||||
if callErr == nil {
|
||||
if decodeErr := json.Unmarshal(content, out); decodeErr != nil {
|
||||
callErr = retryableError{err: fmt.Errorf("decode structured output: %w", decodeErr)}
|
||||
} else {
|
||||
return contracts.StructuredCompletionResponse{
|
||||
Content: content,
|
||||
Provider: "openai-compatible",
|
||||
Model: firstNonEmpty(metadata.Model, model),
|
||||
PromptTokens: metadata.PromptTokens,
|
||||
CompletionTokens: metadata.CompletionTokens,
|
||||
TotalTokens: metadata.TotalTokens,
|
||||
}, nil
|
||||
}
|
||||
}
|
||||
|
||||
lastErr = sanitizeError(callErr, c.cfg.APIKey)
|
||||
if !canRetryFromError(ctx, attempt, c.cfg.MaxRetries, callErr) {
|
||||
return contracts.StructuredCompletionResponse{}, lastErr
|
||||
}
|
||||
}
|
||||
|
||||
if lastErr == nil {
|
||||
lastErr = fmt.Errorf("structured completion failed")
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{}, lastErr
|
||||
}
|
||||
|
||||
type openAICompatibleMessage struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
|
||||
type openAICompatibleRequest struct {
|
||||
Model string `json:"model"`
|
||||
Messages []openAICompatibleMessage `json:"messages"`
|
||||
ResponseFormat openAICompatibleStructuredOutputShape `json:"response_format"`
|
||||
}
|
||||
|
||||
type openAICompatibleStructuredOutputShape struct {
|
||||
Type string `json:"type"`
|
||||
JSONSchema openAICompatibleSchemaEnvelope `json:"json_schema"`
|
||||
}
|
||||
|
||||
type openAICompatibleSchemaEnvelope struct {
|
||||
Name string `json:"name"`
|
||||
Strict bool `json:"strict"`
|
||||
Schema json.RawMessage `json:"schema"`
|
||||
}
|
||||
|
||||
type openAICompatibleChatCompletionsResponse struct {
|
||||
Model string `json:"model"`
|
||||
Choices []struct {
|
||||
Message struct {
|
||||
Content json.RawMessage `json:"content"`
|
||||
} `json:"message"`
|
||||
} `json:"choices"`
|
||||
Usage *openAICompatibleUsage `json:"usage,omitempty"`
|
||||
}
|
||||
|
||||
type openAICompatibleUsage struct {
|
||||
PromptTokens int `json:"prompt_tokens"`
|
||||
CompletionTokens int `json:"completion_tokens"`
|
||||
TotalTokens int `json:"total_tokens"`
|
||||
}
|
||||
|
||||
type openAICompatibleResponseMetadata struct {
|
||||
Model string
|
||||
PromptTokens int
|
||||
CompletionTokens int
|
||||
TotalTokens int
|
||||
}
|
||||
|
||||
func (c *OpenAICompatibleClient) completeStructuredOnce(
|
||||
ctx context.Context,
|
||||
endpoint string,
|
||||
model string,
|
||||
messages []openAICompatibleMessage,
|
||||
responseSchemaName string,
|
||||
responseSchemaJSON json.RawMessage,
|
||||
) (json.RawMessage, openAICompatibleResponseMetadata, error) {
|
||||
requestBody := openAICompatibleRequest{
|
||||
Model: model,
|
||||
Messages: messages,
|
||||
ResponseFormat: openAICompatibleStructuredOutputShape{
|
||||
Type: "json_schema",
|
||||
JSONSchema: openAICompatibleSchemaEnvelope{
|
||||
Name: responseSchemaName,
|
||||
Strict: true,
|
||||
Schema: responseSchemaJSON,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
payload, err := json.Marshal(requestBody)
|
||||
if err != nil {
|
||||
return nil, openAICompatibleResponseMetadata{}, fmt.Errorf("marshal provider request: %w", err)
|
||||
}
|
||||
|
||||
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(payload))
|
||||
if err != nil {
|
||||
return nil, openAICompatibleResponseMetadata{}, fmt.Errorf("build provider request: %w", err)
|
||||
}
|
||||
httpReq.Header.Set("Content-Type", "application/json")
|
||||
if c.cfg.APIKey != "" {
|
||||
httpReq.Header.Set("Authorization", "Bearer "+c.cfg.APIKey)
|
||||
}
|
||||
|
||||
httpResp, err := c.httpClient.Do(httpReq)
|
||||
if err != nil {
|
||||
return nil, openAICompatibleResponseMetadata{}, retryableError{err: fmt.Errorf("provider request failed: %w", err)}
|
||||
}
|
||||
defer func() {
|
||||
_ = httpResp.Body.Close()
|
||||
}()
|
||||
|
||||
rawResp, err := io.ReadAll(httpResp.Body)
|
||||
if err != nil {
|
||||
return nil, openAICompatibleResponseMetadata{}, retryableError{err: fmt.Errorf("read provider response: %w", err)}
|
||||
}
|
||||
|
||||
if httpResp.StatusCode < 200 || httpResp.StatusCode >= 300 {
|
||||
statusErr := parseProviderErrorBody(httpResp.StatusCode, rawResp)
|
||||
if httpResp.StatusCode == http.StatusTooManyRequests || httpResp.StatusCode >= 500 {
|
||||
return nil, openAICompatibleResponseMetadata{}, retryableError{err: statusErr}
|
||||
}
|
||||
return nil, openAICompatibleResponseMetadata{}, statusErr
|
||||
}
|
||||
|
||||
content, metadata, decodeErr := decodeChatCompletionsResponse(rawResp)
|
||||
if decodeErr != nil {
|
||||
return nil, openAICompatibleResponseMetadata{}, decodeErr
|
||||
}
|
||||
return content, metadata, nil
|
||||
}
|
||||
|
||||
func toOpenAICompatibleMessages(messages []contracts.LLMMessage) ([]openAICompatibleMessage, error) {
|
||||
result := make([]openAICompatibleMessage, len(messages))
|
||||
for i, message := range messages {
|
||||
role := strings.TrimSpace(message.Role)
|
||||
content := strings.TrimSpace(message.Content)
|
||||
if role == "" {
|
||||
return nil, fmt.Errorf("message[%d] role must not be empty", i)
|
||||
}
|
||||
if content == "" {
|
||||
return nil, fmt.Errorf("message[%d] content must not be empty", i)
|
||||
}
|
||||
result[i] = openAICompatibleMessage{
|
||||
Role: role,
|
||||
Content: content,
|
||||
}
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func buildChatCompletionsURL(baseURL string) string {
|
||||
return strings.TrimRight(baseURL, "/") + "/chat/completions"
|
||||
}
|
||||
|
||||
func normalizeOpenAICompatibleConfig(cfg OpenAICompatibleClientConfig) (OpenAICompatibleClientConfig, error) {
|
||||
cfg.BaseURL = strings.TrimSpace(cfg.BaseURL)
|
||||
cfg.Model = strings.TrimSpace(cfg.Model)
|
||||
cfg.APIKey = strings.TrimSpace(cfg.APIKey)
|
||||
if cfg.MaxRetries == 0 {
|
||||
cfg.MaxRetries = defaultOpenAICompatibleMaxRetries
|
||||
}
|
||||
if cfg.MaxRetries < 0 {
|
||||
return OpenAICompatibleClientConfig{}, fmt.Errorf("max retries must be zero or greater")
|
||||
}
|
||||
if cfg.BaseURL == "" {
|
||||
return OpenAICompatibleClientConfig{}, fmt.Errorf("base URL must not be empty")
|
||||
}
|
||||
if cfg.Model == "" {
|
||||
return OpenAICompatibleClientConfig{}, fmt.Errorf("model must not be empty")
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
type retryableError struct {
|
||||
err error
|
||||
}
|
||||
|
||||
func (e retryableError) Error() string {
|
||||
if e.err == nil {
|
||||
return ""
|
||||
}
|
||||
return e.err.Error()
|
||||
}
|
||||
|
||||
func (e retryableError) Unwrap() error {
|
||||
return e.err
|
||||
}
|
||||
|
||||
func canRetryFromError(ctx context.Context, attempt int, maxRetries int, err error) bool {
|
||||
if attempt >= maxRetries {
|
||||
return false
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return false
|
||||
}
|
||||
var retryable retryableError
|
||||
return errors.As(err, &retryable)
|
||||
}
|
||||
|
||||
func firstNonEmpty(values ...string) string {
|
||||
for _, value := range values {
|
||||
value = strings.TrimSpace(value)
|
||||
if value != "" {
|
||||
return value
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func decodeChatCompletionsResponse(raw []byte) (json.RawMessage, openAICompatibleResponseMetadata, error) {
|
||||
var parsed openAICompatibleChatCompletionsResponse
|
||||
if err := json.Unmarshal(raw, &parsed); err != nil {
|
||||
return nil, openAICompatibleResponseMetadata{}, retryableError{err: fmt.Errorf("decode provider response envelope: %w", err)}
|
||||
}
|
||||
if len(parsed.Choices) == 0 {
|
||||
return nil, openAICompatibleResponseMetadata{}, fmt.Errorf("provider response missing choices")
|
||||
}
|
||||
|
||||
content, err := extractAssistantContentJSON(parsed.Choices[0].Message.Content)
|
||||
if err != nil {
|
||||
return nil, openAICompatibleResponseMetadata{}, retryableError{err: err}
|
||||
}
|
||||
|
||||
meta := openAICompatibleResponseMetadata{
|
||||
Model: parsed.Model,
|
||||
}
|
||||
if parsed.Usage != nil {
|
||||
meta.PromptTokens = parsed.Usage.PromptTokens
|
||||
meta.CompletionTokens = parsed.Usage.CompletionTokens
|
||||
meta.TotalTokens = parsed.Usage.TotalTokens
|
||||
}
|
||||
return content, meta, nil
|
||||
}
|
||||
|
||||
func extractAssistantContentJSON(raw json.RawMessage) (json.RawMessage, error) {
|
||||
if len(bytes.TrimSpace(raw)) == 0 || string(bytes.TrimSpace(raw)) == "null" {
|
||||
return nil, fmt.Errorf("provider response missing assistant message content")
|
||||
}
|
||||
|
||||
var textContent string
|
||||
if err := json.Unmarshal(raw, &textContent); err == nil {
|
||||
textContent = strings.TrimSpace(textContent)
|
||||
if textContent == "" {
|
||||
return nil, fmt.Errorf("provider response assistant message content is empty")
|
||||
}
|
||||
return json.RawMessage(textContent), nil
|
||||
}
|
||||
|
||||
trimmed := bytes.TrimSpace(raw)
|
||||
if json.Valid(trimmed) {
|
||||
return append(json.RawMessage(nil), trimmed...), nil
|
||||
}
|
||||
return nil, fmt.Errorf("provider response assistant message content is not valid JSON")
|
||||
}
|
||||
|
||||
func parseProviderErrorBody(status int, body []byte) error {
|
||||
trimmed := strings.TrimSpace(string(body))
|
||||
if trimmed == "" {
|
||||
return fmt.Errorf("provider returned status %d", status)
|
||||
}
|
||||
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal(body, &payload); err == nil {
|
||||
if nested, ok := payload["error"].(map[string]any); ok {
|
||||
if msg, ok := nested["message"].(string); ok && strings.TrimSpace(msg) != "" {
|
||||
return fmt.Errorf("provider returned status %d: %s", status, strings.TrimSpace(msg))
|
||||
}
|
||||
}
|
||||
if msg, ok := payload["message"].(string); ok && strings.TrimSpace(msg) != "" {
|
||||
return fmt.Errorf("provider returned status %d: %s", status, strings.TrimSpace(msg))
|
||||
}
|
||||
}
|
||||
|
||||
return fmt.Errorf("provider returned status %d: %s", status, trimmed)
|
||||
}
|
||||
571
internal/framework/llm/openai_compatible_client_test.go
Normal file
571
internal/framework/llm/openai_compatible_client_test.go
Normal file
@@ -0,0 +1,571 @@
|
||||
package llm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/responseschema"
|
||||
)
|
||||
|
||||
func TestNewOpenAICompatibleClientValidation(t *testing.T) {
|
||||
_, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: " ",
|
||||
Model: "model",
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "base URL") {
|
||||
t.Fatalf("expected base URL validation error, got %v", err)
|
||||
}
|
||||
|
||||
_, err = NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: "https://example.test/v1",
|
||||
Model: " ",
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "model") {
|
||||
t.Fatalf("expected model validation error, got %v", err)
|
||||
}
|
||||
|
||||
_, err = NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: "https://example.test/v1",
|
||||
Model: "model",
|
||||
MaxRetries: -1,
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "max retries") {
|
||||
t.Fatalf("expected max retries validation error, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientRequestShapeAndDecode(t *testing.T) {
|
||||
schema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
var seenPath string
|
||||
var seenAuthorization string
|
||||
var seenReq map[string]any
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
seenPath = r.URL.Path
|
||||
seenAuthorization = r.Header.Get("Authorization")
|
||||
if err := json.NewDecoder(r.Body).Decode(&seenReq); err != nil {
|
||||
t.Fatalf("decode request: %v", err)
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{
|
||||
"model":"provider-model",
|
||||
"choices":[{"message":{"content":"{\"name\":\"Robby\",\"age\":22}"}}],
|
||||
"usage":{"prompt_tokens":11,"completion_tokens":7,"total_tokens":18}
|
||||
}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: server.URL + "/v1",
|
||||
Model: "test-model",
|
||||
APIKey: "secret-key",
|
||||
MaxRetries: 1,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
type person struct {
|
||||
Name string `json:"name"`
|
||||
Age int `json:"age"`
|
||||
}
|
||||
var out person
|
||||
resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured: %v", err)
|
||||
}
|
||||
|
||||
if seenPath != "/v1/chat/completions" {
|
||||
t.Fatalf("unexpected request path: %q", seenPath)
|
||||
}
|
||||
if seenAuthorization != "Bearer secret-key" {
|
||||
t.Fatalf("unexpected authorization header: %q", seenAuthorization)
|
||||
}
|
||||
|
||||
responseFormat, ok := seenReq["response_format"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected response_format object, got %T", seenReq["response_format"])
|
||||
}
|
||||
if responseFormat["type"] != "json_schema" {
|
||||
t.Fatalf("unexpected response_format.type: %v", responseFormat["type"])
|
||||
}
|
||||
jsonSchema, ok := responseFormat["json_schema"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected response_format.json_schema object, got %T", responseFormat["json_schema"])
|
||||
}
|
||||
if jsonSchema["name"] != schema.Name {
|
||||
t.Fatalf("unexpected response schema name: %v", jsonSchema["name"])
|
||||
}
|
||||
if jsonSchema["strict"] != true {
|
||||
t.Fatalf("expected strict=true, got %v", jsonSchema["strict"])
|
||||
}
|
||||
if _, ok := jsonSchema["schema"].(map[string]any); !ok {
|
||||
t.Fatalf("expected embedded JSON schema object, got %T", jsonSchema["schema"])
|
||||
}
|
||||
|
||||
if out.Name != "Robby" || out.Age != 22 {
|
||||
t.Fatalf("unexpected decoded output: %+v", out)
|
||||
}
|
||||
if resp.Provider != "openai-compatible" {
|
||||
t.Fatalf("unexpected provider metadata: %q", resp.Provider)
|
||||
}
|
||||
if resp.Model != "provider-model" {
|
||||
t.Fatalf("unexpected model metadata: %q", resp.Model)
|
||||
}
|
||||
if resp.PromptTokens != 11 || resp.CompletionTokens != 7 || resp.TotalTokens != 18 {
|
||||
t.Fatalf("unexpected token metadata: %+v", resp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientDecodesCorrectionSetStructuredResponse(t *testing.T) {
|
||||
schema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{
|
||||
"choices":[{"message":{"content":"{\"corrections\":[{\"id\":1,\"original_text\":\"teh\",\"corrected_text\":\"the\",\"confidence\":0.9}]}"}}]
|
||||
}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: server.URL,
|
||||
Model: "test-model",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
type correction struct {
|
||||
TargetSegmentID int `json:"id"`
|
||||
OriginalText string `json:"original_text"`
|
||||
CorrectedText string `json:"corrected_text"`
|
||||
Confidence float64 `json:"confidence"`
|
||||
}
|
||||
type correctionSet struct {
|
||||
Corrections []correction `json:"corrections"`
|
||||
}
|
||||
var out correctionSet
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured: %v", err)
|
||||
}
|
||||
if len(out.Corrections) != 1 {
|
||||
t.Fatalf("expected one correction, got %+v", out.Corrections)
|
||||
}
|
||||
if out.Corrections[0].TargetSegmentID != 1 || out.Corrections[0].CorrectedText != "the" {
|
||||
t.Fatalf("unexpected correction payload: %+v", out.Corrections[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientDecodesValidatorDecisionSetStructuredResponse(t *testing.T) {
|
||||
schema := responseschema.MustLookup(responseschema.ValidatorDecisionSetKey)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{
|
||||
"choices":[{"message":{"content":"{\"validations\":[{\"correction_index\":0,\"approved\":true,\"confidence\":0.95,\"reason\":\"ok\"}]}"}}]
|
||||
}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: server.URL,
|
||||
Model: "test-model",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
type validationDecision struct {
|
||||
CorrectionIndex int `json:"correction_index"`
|
||||
Approved bool `json:"approved"`
|
||||
Confidence float64 `json:"confidence"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
type validationResponse struct {
|
||||
Validations []validationDecision `json:"validations"`
|
||||
}
|
||||
var out validationResponse
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured: %v", err)
|
||||
}
|
||||
if len(out.Validations) != 1 {
|
||||
t.Fatalf("expected one validation decision, got %+v", out.Validations)
|
||||
}
|
||||
if out.Validations[0].CorrectionIndex != 0 || !out.Validations[0].Approved {
|
||||
t.Fatalf("unexpected validation payload: %+v", out.Validations[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientNoAuthorizationHeaderWithoutAPIKey(t *testing.T) {
|
||||
schema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
var seenAuthorization string
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
seenAuthorization = r.Header.Get("Authorization")
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"{\"ok\":true}"}}]}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: server.URL,
|
||||
Model: "test-model",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
var out map[string]any
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured: %v", err)
|
||||
}
|
||||
if seenAuthorization != "" {
|
||||
t.Fatalf("expected empty Authorization header, got %q", seenAuthorization)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientMalformedJSONFailsSafely(t *testing.T) {
|
||||
schema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"{"}}]}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: server.URL,
|
||||
Model: "test-model",
|
||||
MaxRetries: 0,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
var out map[string]any
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err == nil || !strings.Contains(err.Error(), "decode structured output") {
|
||||
t.Fatalf("expected decode error, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientMissingRequiredFieldsFailsSafely(t *testing.T) {
|
||||
schema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"model":"x","choices":[]}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: server.URL,
|
||||
Model: "test-model",
|
||||
MaxRetries: 0,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
var out map[string]any
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err == nil || !strings.Contains(err.Error(), "missing choices") {
|
||||
t.Fatalf("expected missing-field error, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientUnknownExtraFieldsFollowLocalDecoderPolicy(t *testing.T) {
|
||||
schema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{
|
||||
"choices":[{"message":{"content":"{\"corrections\":[{\"id\":1,\"original_text\":\"teh\",\"corrected_text\":\"the\",\"confidence\":0.9,\"extra\":\"ignored\"}],\"top_extra\":true}"}}]
|
||||
}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: server.URL,
|
||||
Model: "test-model",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
type correction struct {
|
||||
TargetSegmentID int `json:"id"`
|
||||
OriginalText string `json:"original_text"`
|
||||
CorrectedText string `json:"corrected_text"`
|
||||
Confidence float64 `json:"confidence"`
|
||||
}
|
||||
type correctionSet struct {
|
||||
Corrections []correction `json:"corrections"`
|
||||
}
|
||||
var out correctionSet
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("expected unknown extra fields to be ignored by local decoder, got %v", err)
|
||||
}
|
||||
if len(out.Corrections) != 1 || out.Corrections[0].CorrectedText != "the" {
|
||||
t.Fatalf("unexpected decoded payload: %+v", out.Corrections)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientProviderErrorRedactsSecret(t *testing.T) {
|
||||
secret := "super-secret-key"
|
||||
schema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusUnauthorized)
|
||||
_, _ = w.Write([]byte(`{"error":{"message":"Authorization failed for Bearer super-secret-key"}}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: server.URL,
|
||||
Model: "test-model",
|
||||
APIKey: secret,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
var out map[string]any
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err == nil {
|
||||
t.Fatalf("expected provider error")
|
||||
}
|
||||
if strings.Contains(err.Error(), secret) {
|
||||
t.Fatalf("error leaked secret: %v", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "[REDACTED]") {
|
||||
t.Fatalf("expected redaction marker in error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientRequestErrorRedactsSecret(t *testing.T) {
|
||||
secret := "super-secret-key"
|
||||
schema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: "https://example.test/v1",
|
||||
Model: "test-model",
|
||||
APIKey: secret,
|
||||
HTTPClient: &http.Client{
|
||||
Transport: roundTripFunc(func(r *http.Request) (*http.Response, error) {
|
||||
_ = r
|
||||
return nil, fmt.Errorf("request failed for Authorization: Bearer %s", secret)
|
||||
}),
|
||||
},
|
||||
MaxRetries: 0,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
var out map[string]any
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err == nil {
|
||||
t.Fatalf("expected request error")
|
||||
}
|
||||
if strings.Contains(err.Error(), secret) {
|
||||
t.Fatalf("error leaked secret: %v", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "[REDACTED]") {
|
||||
t.Fatalf("expected redaction marker in error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientCancellationAndTimeout(t *testing.T) {
|
||||
schema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
select {
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
case <-time.After(200 * time.Millisecond):
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"{\"ok\":true}"}}]}`))
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: server.URL,
|
||||
Model: "test-model",
|
||||
RequestTimeout: 20 * time.Millisecond,
|
||||
MaxRetries: 0,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
var out map[string]any
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err == nil {
|
||||
t.Fatalf("expected timeout-related error")
|
||||
}
|
||||
if !strings.Contains(strings.ToLower(err.Error()), "context deadline") {
|
||||
t.Fatalf("expected context deadline in error, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientRetryBehavior(t *testing.T) {
|
||||
schema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
var attempts int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
current := atomic.AddInt32(&attempts, 1)
|
||||
if current == 1 {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
_, _ = w.Write([]byte(`{"error":{"message":"temporary failure"}}`))
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"{\"ok\":true}"}}]}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: server.URL,
|
||||
Model: "test-model",
|
||||
MaxRetries: 1,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
var out map[string]any
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured: %v", err)
|
||||
}
|
||||
if atomic.LoadInt32(&attempts) != 2 {
|
||||
t.Fatalf("expected 2 attempts, got %d", attempts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientRetryOnMalformedStructuredOutput(t *testing.T) {
|
||||
schema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
var attempts int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
current := atomic.AddInt32(&attempts, 1)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
if current == 1 {
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"{"}}]}`))
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"{\"ok\":true}"}}]}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: server.URL,
|
||||
Model: "test-model",
|
||||
MaxRetries: 1,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
var out map[string]any
|
||||
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured: %v", err)
|
||||
}
|
||||
if atomic.LoadInt32(&attempts) != 2 {
|
||||
t.Fatalf("expected 2 attempts, got %d", attempts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatibleClientHonorsCancelledContextWithoutRetry(t *testing.T) {
|
||||
schema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
var attempts int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
atomic.AddInt32(&attempts, 1)
|
||||
<-r.Context().Done()
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
|
||||
BaseURL: server.URL,
|
||||
Model: "test-model",
|
||||
MaxRetries: 3,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewOpenAICompatibleClient: %v", err)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
var out map[string]any
|
||||
_, err = client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
Messages: []contracts.LLMMessage{{Role: "user", Content: "extract"}},
|
||||
ResponseSchema: &schema,
|
||||
}, &out)
|
||||
if err == nil {
|
||||
t.Fatalf("expected cancellation error")
|
||||
}
|
||||
if !errors.Is(err, context.Canceled) && !strings.Contains(strings.ToLower(err.Error()), "canceled") {
|
||||
t.Fatalf("expected cancellation-related error, got %v", err)
|
||||
}
|
||||
if atomic.LoadInt32(&attempts) > 1 {
|
||||
t.Fatalf("expected no retry after cancellation, got attempts=%d", attempts)
|
||||
}
|
||||
}
|
||||
|
||||
type roundTripFunc func(*http.Request) (*http.Response, error)
|
||||
|
||||
func (f roundTripFunc) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
return f(r)
|
||||
}
|
||||
|
||||
func newJSONHTTPResponse(status int, body string) *http.Response {
|
||||
return &http.Response{
|
||||
StatusCode: status,
|
||||
Header: http.Header{"Content-Type": []string{"application/json"}},
|
||||
Body: io.NopCloser(strings.NewReader(body)),
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/responseschema"
|
||||
)
|
||||
|
||||
// InteractionDiagnosticsWriter writes machine-readable prompt/response artifacts.
|
||||
@@ -112,11 +113,13 @@ func GenerateCandidates(ctx context.Context, req Request) (Result, error) {
|
||||
callErr error
|
||||
artifacts InteractionArtifacts
|
||||
)
|
||||
responseSchema := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
call := func(callCtx context.Context) error {
|
||||
_, callErr = req.LLMClient.CompleteStructured(callCtx, contracts.StructuredCompletionRequest{
|
||||
StageName: stage,
|
||||
Messages: messages,
|
||||
Model: model,
|
||||
StageName: stage,
|
||||
Messages: messages,
|
||||
Model: model,
|
||||
ResponseSchema: &responseSchema,
|
||||
}, &response)
|
||||
return callErr
|
||||
}
|
||||
@@ -126,17 +129,20 @@ func GenerateCandidates(ctx context.Context, req Request) (Result, error) {
|
||||
callErr = call(ctx)
|
||||
}
|
||||
|
||||
requestMetadata := map[string]any{
|
||||
"module_key": req.ModuleKey,
|
||||
"module_instance": req.ModuleInstance,
|
||||
"replacement_policy": req.ReplacementPolicy,
|
||||
"section": req.Section,
|
||||
"start_index": req.StartIndex,
|
||||
"model": model,
|
||||
}
|
||||
requestMetadata["response_schema"] = schemaMetadata(responseSchema)
|
||||
|
||||
if writer != nil {
|
||||
artifacts, _ = writer.WriteInteraction(
|
||||
stage,
|
||||
map[string]any{
|
||||
"module_key": req.ModuleKey,
|
||||
"module_instance": req.ModuleInstance,
|
||||
"replacement_policy": req.ReplacementPolicy,
|
||||
"section": req.Section,
|
||||
"start_index": req.StartIndex,
|
||||
"model": model,
|
||||
},
|
||||
requestMetadata,
|
||||
map[string]any{
|
||||
"messages": messages,
|
||||
},
|
||||
@@ -185,6 +191,15 @@ func GenerateCandidates(ctx context.Context, req Request) (Result, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
func schemaMetadata(schema responseschema.Schema) map[string]any {
|
||||
return map[string]any{
|
||||
"id": schema.ID,
|
||||
"version": schema.Version,
|
||||
"name": schema.Name,
|
||||
"sha256": schema.SHA256,
|
||||
}
|
||||
}
|
||||
|
||||
func buildStageName(moduleInstance string, section *contracts.SectionMetadata) string {
|
||||
base := fmt.Sprintf("%s:proposal-generation", moduleInstance)
|
||||
if section == nil {
|
||||
|
||||
@@ -2,6 +2,7 @@ package proposal_generation
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -17,6 +18,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/responseschema"
|
||||
)
|
||||
|
||||
type fakeStructuredClient struct {
|
||||
@@ -52,6 +54,21 @@ func (s *countingScheduler) Run(ctx context.Context, fn func(context.Context) er
|
||||
return fn(ctx)
|
||||
}
|
||||
|
||||
type captureDiagnosticsWriter struct {
|
||||
lastStage string
|
||||
lastRequestMetadata any
|
||||
lastRequestPayload any
|
||||
}
|
||||
|
||||
func (w *captureDiagnosticsWriter) WriteInteraction(stage string, requestMetadata any, requestPayload any, responsePayload any, errorPayload any) (InteractionArtifacts, error) {
|
||||
w.lastStage = stage
|
||||
w.lastRequestMetadata = requestMetadata
|
||||
w.lastRequestPayload = requestPayload
|
||||
_ = responsePayload
|
||||
_ = errorPayload
|
||||
return InteractionArtifacts{}, nil
|
||||
}
|
||||
|
||||
type sleepingStructuredClient struct {
|
||||
inFlight int32
|
||||
maxInFlight int32
|
||||
@@ -138,6 +155,61 @@ func TestGenerateCandidatesSuccess(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateCandidatesUsesCorrectionSetSchema(t *testing.T) {
|
||||
client := &fakeStructuredClient{
|
||||
responses: []StructuredCorrectionSet{
|
||||
{Corrections: []StructuredCorrectionProposal{{TargetSegmentID: 1, OriginalText: "teh", CorrectedText: "the", Confidence: 0.9}}},
|
||||
},
|
||||
}
|
||||
req := defaultRequest(t)
|
||||
req.LLMClient = client
|
||||
|
||||
_, err := GenerateCandidates(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateCandidates error: %v", err)
|
||||
}
|
||||
if len(client.calls) != 1 {
|
||||
t.Fatalf("expected 1 LLM call, got %d", len(client.calls))
|
||||
}
|
||||
call := client.calls[0]
|
||||
if call.ResponseSchema == nil {
|
||||
t.Fatalf("expected response schema on structured request")
|
||||
}
|
||||
want := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
if call.ResponseSchema.ID != want.ID || call.ResponseSchema.Version != want.Version || call.ResponseSchema.Name != want.Name || call.ResponseSchema.SHA256 != want.SHA256 {
|
||||
t.Fatalf("unexpected response schema metadata: got=%+v want=%+v", *call.ResponseSchema, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateCandidatesDiagnosticsIncludeSchemaMetadata(t *testing.T) {
|
||||
client := &fakeStructuredClient{
|
||||
responses: []StructuredCorrectionSet{
|
||||
{Corrections: []StructuredCorrectionProposal{{TargetSegmentID: 1, OriginalText: "teh", CorrectedText: "the", Confidence: 0.9}}},
|
||||
},
|
||||
}
|
||||
diag := &captureDiagnosticsWriter{}
|
||||
req := defaultRequest(t)
|
||||
req.LLMClient = client
|
||||
req.DiagnosticsWriter = diag
|
||||
|
||||
_, err := GenerateCandidates(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateCandidates error: %v", err)
|
||||
}
|
||||
metadata, ok := diag.lastRequestMetadata.(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected request metadata map, got %T", diag.lastRequestMetadata)
|
||||
}
|
||||
schemaMap, ok := metadata["response_schema"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected response_schema metadata map, got %T", metadata["response_schema"])
|
||||
}
|
||||
want := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
if schemaMap["id"] != want.ID || schemaMap["version"] != want.Version || schemaMap["name"] != want.Name || schemaMap["sha256"] != want.SHA256 {
|
||||
t.Fatalf("unexpected diagnostics schema metadata: got=%v want=%+v", schemaMap, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateCandidatesMalformedStructuredResponse(t *testing.T) {
|
||||
client := &fakeStructuredClient{
|
||||
responses: []StructuredCorrectionSet{
|
||||
@@ -266,6 +338,23 @@ func TestGenerateCandidatesDiagnosticsWrittenAndRedacted(t *testing.T) {
|
||||
t.Fatalf("expected redaction marker in artifact %q: %s", path, string(raw))
|
||||
}
|
||||
}
|
||||
|
||||
raw, readErr := os.ReadFile(got.Artifacts.RequestMetadataPath)
|
||||
if readErr != nil {
|
||||
t.Fatalf("read metadata artifact %q: %v", got.Artifacts.RequestMetadataPath, readErr)
|
||||
}
|
||||
var metadata map[string]any
|
||||
if err := json.Unmarshal(raw, &metadata); err != nil {
|
||||
t.Fatalf("unmarshal metadata artifact: %v", err)
|
||||
}
|
||||
schemaMap, ok := metadata["response_schema"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected response_schema metadata in diagnostics, got %T", metadata["response_schema"])
|
||||
}
|
||||
want := responseschema.MustLookup(responseschema.CorrectionSetKey)
|
||||
if schemaMap["id"] != want.ID || schemaMap["version"] != want.Version || schemaMap["name"] != want.Name || schemaMap["sha256"] != want.SHA256 {
|
||||
t.Fatalf("unexpected schema metadata in diagnostics: got=%v want=%+v", schemaMap, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateCandidatesSchedulerUsage(t *testing.T) {
|
||||
|
||||
97
internal/framework/responseschema/registry.go
Normal file
97
internal/framework/responseschema/registry.go
Normal file
@@ -0,0 +1,97 @@
|
||||
package responseschema
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Key identifies one structured response schema used by Audita.
|
||||
type Key string
|
||||
|
||||
const (
|
||||
CorrectionSetKey Key = "correction_set"
|
||||
ValidatorDecisionSetKey Key = "validator_decision_set"
|
||||
correctionSetSchemaID = "audita.correction_set"
|
||||
validatorDecisionSchemaID = "audita.validator_decision_set"
|
||||
schemaVersionV1 = "v1"
|
||||
)
|
||||
|
||||
// Schema describes one registered structured response schema.
|
||||
type Schema struct {
|
||||
ID string `json:"id"`
|
||||
Version string `json:"version"`
|
||||
Name string `json:"name"`
|
||||
JSONSchema json.RawMessage `json:"json_schema"`
|
||||
SHA256 string `json:"sha256"`
|
||||
}
|
||||
|
||||
var registry = map[Key]Schema{
|
||||
CorrectionSetKey: mustBuildSchema(
|
||||
correctionSetSchemaID,
|
||||
schemaVersionV1,
|
||||
"audita_correction_set_v1",
|
||||
[]byte(`{"type":"object","additionalProperties":false,"required":["corrections"],"properties":{"corrections":{"type":"array","items":{"type":"object","additionalProperties":false,"required":["id","original_text","corrected_text","confidence"],"properties":{"id":{"type":"integer","minimum":1},"original_text":{"type":"string","minLength":1},"corrected_text":{"type":"string","minLength":1},"confidence":{"type":"number","minimum":0,"maximum":1}}}}}}`),
|
||||
),
|
||||
ValidatorDecisionSetKey: mustBuildSchema(
|
||||
validatorDecisionSchemaID,
|
||||
schemaVersionV1,
|
||||
"audita_validator_decision_set_v1",
|
||||
[]byte(`{"type":"object","additionalProperties":false,"required":["validations"],"properties":{"validations":{"type":"array","items":{"type":"object","additionalProperties":false,"required":["correction_index","approved","confidence","reason"],"properties":{"correction_index":{"type":"integer","minimum":0},"approved":{"type":"boolean"},"confidence":{"type":"number","minimum":0,"maximum":1},"reason":{"type":"string"}}}}}}`),
|
||||
),
|
||||
}
|
||||
|
||||
// Lookup returns a copy of the registered schema for the provided key.
|
||||
func Lookup(key Key) (Schema, bool) {
|
||||
schema, ok := registry[key]
|
||||
if !ok {
|
||||
return Schema{}, false
|
||||
}
|
||||
return cloneSchema(schema), true
|
||||
}
|
||||
|
||||
// MustLookup returns a copy of the registered schema and panics when missing.
|
||||
func MustLookup(key Key) Schema {
|
||||
schema, ok := Lookup(key)
|
||||
if !ok {
|
||||
panic(fmt.Sprintf("unknown structured response schema key %q", key))
|
||||
}
|
||||
return schema
|
||||
}
|
||||
|
||||
func cloneSchema(in Schema) Schema {
|
||||
out := in
|
||||
if in.JSONSchema != nil {
|
||||
out.JSONSchema = append(json.RawMessage(nil), in.JSONSchema...)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func mustBuildSchema(id string, version string, name string, rawSchema []byte) Schema {
|
||||
id = strings.TrimSpace(id)
|
||||
version = strings.TrimSpace(version)
|
||||
name = strings.TrimSpace(name)
|
||||
if id == "" {
|
||||
panic("schema id must not be empty")
|
||||
}
|
||||
if version == "" {
|
||||
panic("schema version must not be empty")
|
||||
}
|
||||
if name == "" {
|
||||
panic("schema name must not be empty")
|
||||
}
|
||||
if !json.Valid(rawSchema) {
|
||||
panic(fmt.Sprintf("schema %s:%s is not valid JSON", id, version))
|
||||
}
|
||||
|
||||
hash := sha256.Sum256(rawSchema)
|
||||
return Schema{
|
||||
ID: id,
|
||||
Version: version,
|
||||
Name: name,
|
||||
JSONSchema: append(json.RawMessage(nil), rawSchema...),
|
||||
SHA256: hex.EncodeToString(hash[:]),
|
||||
}
|
||||
}
|
||||
91
internal/framework/responseschema/registry_test.go
Normal file
91
internal/framework/responseschema/registry_test.go
Normal file
@@ -0,0 +1,91 @@
|
||||
package responseschema
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const (
|
||||
expectedCorrectionSetSHA256 = "05f8ff3fa04f68115c0cb1859d2656f51aa5c0bae8ff2470b2d4f6f531953195"
|
||||
expectedValidatorDecisionSetSHA256 = "b73f4790b98fbb955f0aec5496dd8ce9a8fe14aa2f35c700b4b4e5634f106fd5"
|
||||
)
|
||||
|
||||
func TestLookupKnownSchemas(t *testing.T) {
|
||||
correction, ok := Lookup(CorrectionSetKey)
|
||||
if !ok {
|
||||
t.Fatalf("expected correction-set schema to be registered")
|
||||
}
|
||||
if correction.ID != correctionSetSchemaID {
|
||||
t.Fatalf("unexpected correction-set schema id %q", correction.ID)
|
||||
}
|
||||
if correction.Version != schemaVersionV1 {
|
||||
t.Fatalf("unexpected correction-set schema version %q", correction.Version)
|
||||
}
|
||||
if correction.Name != "audita_correction_set_v1" {
|
||||
t.Fatalf("unexpected correction-set schema name %q", correction.Name)
|
||||
}
|
||||
|
||||
validation, ok := Lookup(ValidatorDecisionSetKey)
|
||||
if !ok {
|
||||
t.Fatalf("expected validator-decision schema to be registered")
|
||||
}
|
||||
if validation.ID != validatorDecisionSchemaID {
|
||||
t.Fatalf("unexpected validator-decision schema id %q", validation.ID)
|
||||
}
|
||||
if validation.Version != schemaVersionV1 {
|
||||
t.Fatalf("unexpected validator-decision schema version %q", validation.Version)
|
||||
}
|
||||
if validation.Name != "audita_validator_decision_set_v1" {
|
||||
t.Fatalf("unexpected validator-decision schema name %q", validation.Name)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLookupUnknownSchema(t *testing.T) {
|
||||
if _, ok := Lookup(Key("missing")); ok {
|
||||
t.Fatalf("expected unknown schema lookup to fail")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSchemaHashesMatchRegisteredJSON(t *testing.T) {
|
||||
expectedByKey := map[Key]string{
|
||||
CorrectionSetKey: expectedCorrectionSetSHA256,
|
||||
ValidatorDecisionSetKey: expectedValidatorDecisionSetSHA256,
|
||||
}
|
||||
for key, expectedHash := range expectedByKey {
|
||||
schema, ok := Lookup(key)
|
||||
if !ok {
|
||||
t.Fatalf("missing schema %q", key)
|
||||
}
|
||||
sum := sha256.Sum256(schema.JSONSchema)
|
||||
expected := hex.EncodeToString(sum[:])
|
||||
if schema.SHA256 != expectedHash {
|
||||
t.Fatalf("unexpected stable hash for %q: got %q want %q", key, schema.SHA256, expectedHash)
|
||||
}
|
||||
if schema.SHA256 != expected {
|
||||
t.Fatalf("unexpected hash for %q: got %q want %q", key, schema.SHA256, expected)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLookupReturnsSchemaCopy(t *testing.T) {
|
||||
schema, ok := Lookup(CorrectionSetKey)
|
||||
if !ok {
|
||||
t.Fatalf("missing correction-set schema")
|
||||
}
|
||||
if len(schema.JSONSchema) == 0 {
|
||||
t.Fatalf("expected non-empty schema payload")
|
||||
}
|
||||
schema.JSONSchema[0] = 'x'
|
||||
|
||||
again, ok := Lookup(CorrectionSetKey)
|
||||
if !ok {
|
||||
t.Fatalf("missing correction-set schema on second lookup")
|
||||
}
|
||||
if len(again.JSONSchema) == 0 {
|
||||
t.Fatalf("unexpected empty schema payload")
|
||||
}
|
||||
if again.JSONSchema[0] != '{' {
|
||||
t.Fatalf("expected lookup to return independent schema copy")
|
||||
}
|
||||
}
|
||||
@@ -620,9 +620,10 @@ func (a validationLLMClientAdapter) CompleteStructured(ctx context.Context, req
|
||||
messages[i] = contracts.LLMMessage{Role: m.Role, Content: m.Content}
|
||||
}
|
||||
resp, err := a.client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: req.StageName,
|
||||
Messages: messages,
|
||||
Model: req.Model,
|
||||
StageName: req.StageName,
|
||||
Messages: messages,
|
||||
Model: req.Model,
|
||||
ResponseSchema: req.ResponseSchema,
|
||||
}, out)
|
||||
if err != nil {
|
||||
return validators.StructuredCompletionResponse{}, err
|
||||
|
||||
@@ -20,6 +20,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/modules"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposal_generation"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/responseschema"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/validators"
|
||||
)
|
||||
|
||||
@@ -639,6 +640,7 @@ func TestRunnerMixedProposalValidationFIFOOrder(t *testing.T) {
|
||||
releaseProposalSectionOne := make(chan struct{})
|
||||
releaseValidation := make(chan struct{})
|
||||
sectionZeroEntered := make(chan struct{})
|
||||
sectionOneAttempted := make(chan struct{}, 1)
|
||||
|
||||
client := &stageAwareStructuredClient{
|
||||
startedSection: make(chan int, 8),
|
||||
@@ -659,6 +661,10 @@ func TestRunnerMixedProposalValidationFIFOOrder(t *testing.T) {
|
||||
seg := req.WorkingTranscript.Segments[0]
|
||||
if req.Section != nil && req.Section.Index == 1 {
|
||||
<-sectionZeroEntered
|
||||
select {
|
||||
case sectionOneAttempted <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
err := req.LLMScheduler.Run(context.Background(), func(context.Context) error {
|
||||
if req.Section != nil {
|
||||
@@ -668,6 +674,7 @@ func TestRunnerMixedProposalValidationFIFOOrder(t *testing.T) {
|
||||
case sectionZeroEntered <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
<-sectionOneAttempted
|
||||
}
|
||||
if req.Section.Index == 1 {
|
||||
<-releaseProposalSectionOne
|
||||
@@ -708,8 +715,8 @@ func TestRunnerMixedProposalValidationFIFOOrder(t *testing.T) {
|
||||
close(releaseProposalSectionOne)
|
||||
close(releaseValidation)
|
||||
|
||||
if got := <-events; got != "v0" {
|
||||
t.Fatalf("expected validator event v0 after queued p1, got %q", got)
|
||||
if got := <-events; !strings.HasPrefix(got, "v") {
|
||||
t.Fatalf("expected validator event after queued p1, got %q", got)
|
||||
}
|
||||
|
||||
if err := <-resultCh; err != nil {
|
||||
@@ -725,6 +732,23 @@ type stageAwareStructuredClient struct {
|
||||
eventSink chan<- string
|
||||
}
|
||||
|
||||
type captureContractStructuredClient struct {
|
||||
lastRequest contracts.StructuredCompletionRequest
|
||||
}
|
||||
|
||||
func (c *captureContractStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
_ = ctx
|
||||
c.lastRequest = req
|
||||
if target, ok := out.(*validators.LLMValidationResponse); ok {
|
||||
*target = validators.LLMValidationResponse{
|
||||
Validations: []validators.LLMValidationDecision{
|
||||
{CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"},
|
||||
},
|
||||
}
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{}, nil
|
||||
}
|
||||
|
||||
func (c *stageAwareStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
section := parseSectionFromStage(req.StageName)
|
||||
if c.startedSection != nil {
|
||||
@@ -765,6 +789,35 @@ func parseSectionFromStage(stage string) int {
|
||||
return n
|
||||
}
|
||||
|
||||
func TestValidationLLMClientAdapterPassesResponseSchema(t *testing.T) {
|
||||
capture := &captureContractStructuredClient{}
|
||||
adapter := validationLLMClientAdapter{client: capture}
|
||||
schema := responseschema.MustLookup(responseschema.ValidatorDecisionSetKey)
|
||||
|
||||
_, err := adapter.CompleteStructured(context.Background(), validators.StructuredCompletionRequest{
|
||||
StageName: "module:validator:batch-0000",
|
||||
Messages: []validators.LLMMessage{
|
||||
{Role: "system", Content: "system"},
|
||||
{Role: "user", Content: "user"},
|
||||
},
|
||||
Model: "test-model",
|
||||
ResponseSchema: &schema,
|
||||
}, &validators.LLMValidationResponse{})
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteStructured error: %v", err)
|
||||
}
|
||||
|
||||
if capture.lastRequest.ResponseSchema == nil {
|
||||
t.Fatalf("expected response schema to be forwarded")
|
||||
}
|
||||
if capture.lastRequest.ResponseSchema.ID != schema.ID ||
|
||||
capture.lastRequest.ResponseSchema.Version != schema.Version ||
|
||||
capture.lastRequest.ResponseSchema.Name != schema.Name ||
|
||||
capture.lastRequest.ResponseSchema.SHA256 != schema.SHA256 {
|
||||
t.Fatalf("unexpected forwarded schema metadata: got=%+v want=%+v", *capture.lastRequest.ResponseSchema, schema)
|
||||
}
|
||||
}
|
||||
|
||||
type trackingScheduler struct {
|
||||
inner contracts.LLMScheduler
|
||||
inFlight int32
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/responseschema"
|
||||
)
|
||||
|
||||
type LLMValidatorType string
|
||||
@@ -22,9 +23,10 @@ type LLMMessage struct {
|
||||
}
|
||||
|
||||
type StructuredCompletionRequest struct {
|
||||
StageName string `json:"stage_name"`
|
||||
Messages []LLMMessage `json:"messages"`
|
||||
Model string `json:"model,omitempty"`
|
||||
StageName string `json:"stage_name"`
|
||||
Messages []LLMMessage `json:"messages"`
|
||||
Model string `json:"model,omitempty"`
|
||||
ResponseSchema *responseschema.Schema `json:"response_schema,omitempty"`
|
||||
}
|
||||
|
||||
type StructuredCompletionResponse struct {
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/responseschema"
|
||||
)
|
||||
|
||||
type LLMPromptBuilder func(validationPayload []LLMValidationItem) ([]LLMMessage, error)
|
||||
@@ -89,11 +90,13 @@ func (v *LLMBackedValidator) Validate(ctx context.Context, req Request) (Result,
|
||||
}
|
||||
|
||||
var response LLMValidationResponse
|
||||
responseSchema := responseschema.MustLookup(responseschema.ValidatorDecisionSetKey)
|
||||
call := func(callCtx context.Context) error {
|
||||
_, err = req.LLMClient.CompleteStructured(callCtx, StructuredCompletionRequest{
|
||||
StageName: fmt.Sprintf("%s:%s:batch-%04d", req.ModuleInstance, v.name, batch.BatchIndex),
|
||||
Messages: messages,
|
||||
Model: resolvedValidationModel(req.Config, v.model),
|
||||
StageName: fmt.Sprintf("%s:%s:batch-%04d", req.ModuleInstance, v.name, batch.BatchIndex),
|
||||
Messages: messages,
|
||||
Model: resolvedValidationModel(req.Config, v.model),
|
||||
ResponseSchema: &responseSchema,
|
||||
}, &response)
|
||||
return err
|
||||
}
|
||||
@@ -107,7 +110,17 @@ func (v *LLMBackedValidator) Validate(ctx context.Context, req Request) (Result,
|
||||
stage := fmt.Sprintf("%s:%s:batch-%04d", req.ModuleInstance, v.name, batch.BatchIndex)
|
||||
artifacts, _ = req.DiagnosticsWriter.WriteInteraction(
|
||||
stage,
|
||||
map[string]any{"validator_name": v.name, "validator_type": v.validatorType, "batch_index": batch.BatchIndex},
|
||||
map[string]any{
|
||||
"validator_name": v.name,
|
||||
"validator_type": v.validatorType,
|
||||
"batch_index": batch.BatchIndex,
|
||||
"response_schema": map[string]any{
|
||||
"id": responseSchema.ID,
|
||||
"version": responseSchema.Version,
|
||||
"name": responseSchema.Name,
|
||||
"sha256": responseSchema.SHA256,
|
||||
},
|
||||
},
|
||||
map[string]any{"messages": messages, "items": batch.Items},
|
||||
response,
|
||||
errPayload(err),
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/core/schema"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/proposals"
|
||||
"gitea.maximumdirect.net/eric/audita/internal/framework/responseschema"
|
||||
)
|
||||
|
||||
type fakeStructuredLLMClient struct {
|
||||
@@ -33,6 +34,19 @@ type boundedScheduler struct {
|
||||
permits chan struct{}
|
||||
}
|
||||
|
||||
type captureValidationDiagnosticsWriter struct {
|
||||
lastRequestMetadata any
|
||||
}
|
||||
|
||||
func (w *captureValidationDiagnosticsWriter) WriteInteraction(stage string, requestMetadata any, requestPayload any, responsePayload any, errorPayload any) (InteractionArtifacts, error) {
|
||||
_ = stage
|
||||
_ = requestPayload
|
||||
_ = responsePayload
|
||||
_ = errorPayload
|
||||
w.lastRequestMetadata = requestMetadata
|
||||
return InteractionArtifacts{}, nil
|
||||
}
|
||||
|
||||
func newBoundedScheduler(max int) *boundedScheduler {
|
||||
return &boundedScheduler{permits: make(chan struct{}, max)}
|
||||
}
|
||||
@@ -200,6 +214,17 @@ func TestLLMBackedValidatorApprovalAndRejection(t *testing.T) {
|
||||
if len(res.Decisions) != 2 || !res.Decisions[0].Approved || res.Decisions[1].Approved {
|
||||
t.Fatalf("unexpected decisions: %+v", res.Decisions)
|
||||
}
|
||||
if len(client.calls) != 1 {
|
||||
t.Fatalf("expected one LLM call, got %d", len(client.calls))
|
||||
}
|
||||
if client.calls[0].ResponseSchema == nil {
|
||||
t.Fatalf("expected response schema on structured validation request")
|
||||
}
|
||||
want := responseschema.MustLookup(responseschema.ValidatorDecisionSetKey)
|
||||
gotSchema := client.calls[0].ResponseSchema
|
||||
if gotSchema.ID != want.ID || gotSchema.Version != want.Version || gotSchema.Name != want.Name || gotSchema.SHA256 != want.SHA256 {
|
||||
t.Fatalf("unexpected validator response schema metadata: got=%+v want=%+v", *gotSchema, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLLMBackedValidatorMalformedOutputFails(t *testing.T) {
|
||||
@@ -285,6 +310,36 @@ func TestLLMBackedValidatorRespectsSchedulerConcurrency(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestLLMBackedValidatorDiagnosticsIncludeSchemaMetadata(t *testing.T) {
|
||||
client := &fakeStructuredLLMClient{responses: []LLMValidationResponse{{Validations: []LLMValidationDecision{
|
||||
{CorrectionIndex: 0, Approved: true, Confidence: 0.9, Reason: "ok"},
|
||||
}}}}
|
||||
writer := &captureValidationDiagnosticsWriter{}
|
||||
v, err := NewLLMBackedValidator("spoken_form_plausibility_review", LLMValidatorTypeSpokenFormPlausibility, "test-model")
|
||||
if err != nil {
|
||||
t.Fatalf("new validator error: %v", err)
|
||||
}
|
||||
req := makeReq([]proposals.EnrichedCorrectionProposal{mk(0, "gestures", "Jesters")})
|
||||
req.LLMClient = client
|
||||
req.DiagnosticsWriter = writer
|
||||
_, err = v.Validate(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("validate error: %v", err)
|
||||
}
|
||||
metadata, ok := writer.lastRequestMetadata.(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected metadata map, got %T", writer.lastRequestMetadata)
|
||||
}
|
||||
schemaMap, ok := metadata["response_schema"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected response_schema map, got %T", metadata["response_schema"])
|
||||
}
|
||||
want := responseschema.MustLookup(responseschema.ValidatorDecisionSetKey)
|
||||
if schemaMap["id"] != want.ID || schemaMap["version"] != want.Version || schemaMap["name"] != want.Name || schemaMap["sha256"] != want.SHA256 {
|
||||
t.Fatalf("unexpected diagnostics schema metadata: got=%v want=%+v", schemaMap, want)
|
||||
}
|
||||
}
|
||||
|
||||
func waitForValidationEntries(t *testing.T, entered <-chan struct{}, want int) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(300 * time.Millisecond)
|
||||
|
||||
Reference in New Issue
Block a user