Implement Phase 9 structured LLM adapter spike
This commit is contained in:
@@ -22,6 +22,7 @@ Implemented today:
|
||||
- Runtime validator models and deterministic validators.
|
||||
- Deterministic validator-chain execution in the runner with cardinality enforcement.
|
||||
- Module-level validator decision/rejection reporting.
|
||||
- Internal structured LLM client contract plus an `instructor-go`-backed adapter package.
|
||||
|
||||
Not implemented in CLI runtime path today:
|
||||
- Real module execution pipeline (`glossary`, `homophones`, `spoken_word`, `grammar`).
|
||||
@@ -31,7 +32,7 @@ Not implemented in CLI runtime path today:
|
||||
- End-to-end transcript polishing with real module behavior.
|
||||
|
||||
Phase sequencing note:
|
||||
- structured LLM client and scheduler infrastructure remain Phase 9 work;
|
||||
- structured LLM client infrastructure is implemented, but scheduler and runtime wiring remain Phase 9 follow-up work;
|
||||
- LLM-backed validators remain Phase 10 work.
|
||||
|
||||
## Actual Go package layout
|
||||
@@ -88,6 +89,9 @@ internal/framework/runner/
|
||||
internal/framework/validators/
|
||||
models.go
|
||||
deterministic.go
|
||||
|
||||
internal/framework/llm/
|
||||
instructor_client.go
|
||||
```
|
||||
|
||||
## Current CLI behavior
|
||||
@@ -168,6 +172,23 @@ Implemented config surfaces include:
|
||||
Current caveat:
|
||||
- LLM/module-related settings are mostly infrastructure-only today; runtime path does not execute LLM or modules.
|
||||
|
||||
## Implemented structured LLM infrastructure
|
||||
`internal/framework/contracts` now defines a typed structured-completion contract:
|
||||
- `StructuredLLMClient.CompleteStructured(ctx, req, out)`
|
||||
- caller-owned typed decode target via `out` pointer.
|
||||
|
||||
`internal/framework/llm` provides `InstructorClient`, an internal adapter over `github.com/jxnl/instructor-go`:
|
||||
- configurable `base_url`, model, optional API key, retries, mode, HTTP client, and request timeout;
|
||||
- OpenAI-compatible endpoint behavior (for example OpenAI/OpenRouter/local-compatible base URLs);
|
||||
- default mode is JSON mode (`ModeJSON`), with optional tool-call mode (`ModeToolCall`);
|
||||
- request message translation from `contracts.LLMMessage` to chat-completions messages;
|
||||
- response metadata mapping (provider/model/token usage) into Audita-owned response types;
|
||||
- API-key redaction in adapter-returned errors.
|
||||
|
||||
Current runtime boundary:
|
||||
- the CLI/runner runtime path does not instantiate this adapter yet;
|
||||
- no production LLM requests are performed by `audita process`.
|
||||
|
||||
## Implemented normalization behavior
|
||||
Normalization (`internal/core/normalization`) currently:
|
||||
- sorts by segment start time;
|
||||
|
||||
@@ -52,10 +52,17 @@ Implemented:
|
||||
- Deterministic validator-chain execution in the production runner.
|
||||
- Module reports including validator decisions and validator rejections.
|
||||
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
|
||||
- Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
|
||||
- `internal/framework/llm` instructor-go-backed adapter with:
|
||||
- configurable base URL/model/retries/mode/timeout
|
||||
- optional API key support for local-compatible endpoints
|
||||
- API-key redaction in returned errors
|
||||
- typed structured decode into caller-provided outputs.
|
||||
|
||||
Not yet implemented in runtime pipeline:
|
||||
- Real correction modules.
|
||||
- Structured LLM client integration.
|
||||
- Structured LLM scheduler/concurrency orchestration.
|
||||
- Runtime wiring from production runner/modules into the structured LLM adapter.
|
||||
- LLM-backed validators.
|
||||
- Prompt/response diagnostics for LLM calls.
|
||||
- End-to-end transcript polishing behavior.
|
||||
@@ -189,37 +196,44 @@ Implemented:
|
||||
|
||||
Not implemented in Phase 8 (by design):
|
||||
- LLM-backed validators (Phase 10).
|
||||
- Structured LLM client implementation or scheduler behavior (Phase 9).
|
||||
- Structured LLM scheduler behavior or runtime wiring (Phase 9 follow-up).
|
||||
- Real correction modules.
|
||||
- Prompt/response diagnostics.
|
||||
- End-to-end transcript polishing.
|
||||
|
||||
## Remaining work plan
|
||||
|
||||
Next recommended phase: **Phase 9 (structured LLM client and scheduler infrastructure)**.
|
||||
Next recommended phase: **Phase 9 follow-up (scheduler + runtime LLM wiring, still no real modules)**.
|
||||
|
||||
## Phase 9: Structured LLM client and scheduler infrastructure
|
||||
|
||||
### Status
|
||||
|
||||
Partially completed.
|
||||
|
||||
Implemented in this phase so far:
|
||||
- Added internal structured LLM contract support for caller-provided typed outputs.
|
||||
- Added `internal/framework/llm` adapter backed by `github.com/jxnl/instructor-go`.
|
||||
- Confirmed OpenAI-compatible base URL support through the adapter path.
|
||||
- Added adapter unit tests for model/base URL handling, retries, context cancellation, optional API key behavior, and error redaction.
|
||||
|
||||
Still pending in Phase 9:
|
||||
- Scheduler/semaphore behavior for bounded concurrency.
|
||||
- Runtime wiring in runner/module infrastructure (without introducing real modules yet).
|
||||
- Prompt/response diagnostics writer primitives for LLM call artifacts.
|
||||
- Full primary vs validation LLM config-resolution plumbing into runtime LLM call sites.
|
||||
|
||||
### Purpose
|
||||
|
||||
Implement the provider-neutral LLM infrastructure needed by both proposal generation and LLM-backed validators, without yet implementing real modules.
|
||||
|
||||
### Scope
|
||||
|
||||
Implement:
|
||||
- `StructuredLLMClient` interface refinement if needed.
|
||||
- OpenAI-compatible chat completions client.
|
||||
- Structured JSON response support.
|
||||
- Request/response types for structured calls.
|
||||
- Retry handling for malformed structured output.
|
||||
- Per-request timeout behavior.
|
||||
- Context cancellation.
|
||||
Implement (remaining):
|
||||
- Primary LLM config resolution.
|
||||
- Validation LLM config resolution and inheritance from primary settings.
|
||||
- Optional API key behavior for self-hosted endpoints.
|
||||
- Redaction of credentials in all diagnostics and reports.
|
||||
- LLM scheduler/semaphore for bounded backend concurrency.
|
||||
- Unit tests using fake HTTP servers or fake client implementations.
|
||||
- Prompt/response diagnostics writer primitives that can later be used by modules and validators.
|
||||
|
||||
Do not implement:
|
||||
@@ -230,15 +244,13 @@ Do not implement:
|
||||
|
||||
### Expected behavior at end of phase
|
||||
|
||||
The codebase has a tested OpenAI-compatible structured-output client and scheduler, but the CLI still does not perform real LLM polishing unless later phases wire modules into the runner.
|
||||
The codebase has a tested OpenAI-compatible structured-output client adapter, but scheduler and runtime wiring remain before this phase is fully complete. The CLI still does not perform real LLM polishing.
|
||||
|
||||
### Definition of done
|
||||
|
||||
- Structured LLM client is implemented and tested.
|
||||
Remaining checklist to close Phase 9:
|
||||
- Scheduler enforces configured concurrency.
|
||||
- Primary and validation LLM settings resolve correctly.
|
||||
- Timeout and retry behavior are tested.
|
||||
- Malformed structured responses fail cleanly or retry according to config.
|
||||
- Primary and validation LLM settings resolve correctly in runtime wiring.
|
||||
- Prompt/response diagnostic primitives exist.
|
||||
- API keys are not leaked.
|
||||
- No real module behavior is introduced.
|
||||
|
||||
Reference in New Issue
Block a user