From a48f6da1f4f13a362f3742a8e5e603a2ace794a0 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 12 May 2026 16:00:26 -0500 Subject: [PATCH] Audit LLM concurrency flow --- docs/llm-concurrency-audit.md | 138 ++++++++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 docs/llm-concurrency-audit.md diff --git a/docs/llm-concurrency-audit.md b/docs/llm-concurrency-audit.md new file mode 100644 index 0000000..bc96dc5 --- /dev/null +++ b/docs/llm-concurrency-audit.md @@ -0,0 +1,138 @@ +# LLM Concurrency Audit + +Date: 2026-05-12 + +## Summary + +This document audits the current Audita execution flow for module sequencing, chunk proposal execution, validator execution, and LLM concurrency control. + +Current implementation already provides: +- serial module execution over a mutable working transcript, +- per-module chunk recomputation from the current working transcript, +- concurrent per-section proposal execution, +- scheduler wiring for both proposal-generation and LLM-backed validator calls, +- deterministic proposal aggregation and deterministic per-module apply ordering, +- subprocess-safe behavior and deterministic test hooks without requiring live LLM credentials. + +Main gap versus the requested target architecture: +- there is no separate `--proposal-llm-concurrency` flag, +- there is no separate `--total-llm-concurrency` flag, +- global concurrency is currently represented by existing `--llm-concurrency`, +- scheduler implementation is semaphore-based and does not explicitly guarantee FIFO ordering. + +## Audit Findings (Questions 1-14) + +1. **Does the current runner execute modules serially?** +- Yes. `Runner.Run` loops through `input.ModuleSpecs` sequentially and updates `working` per module. + +2. **Are chunks within a module processed serially or concurrently?** +- Concurrently. `collectSectionProposals` starts one goroutine per section, bounded by `maxWorkers`. + +3. **Are chunks recomputed per module from the current working transcript, or computed once before all modules?** +- Recomputed per module from current `working` transcript via `chunkWorkingTranscript` inside the module loop. + +4. **Are proposal-generation LLM calls routed through the existing scheduler consistently?** +- Yes for production module paths. Each module `Propose` calls `proposal_generation.GenerateCandidates`, which wraps LLM calls in `req.Scheduler.Run(...)` when a scheduler is present. + +5. **Are LLM-backed validator calls routed through the existing scheduler consistently?** +- Yes. `LLMBackedValidator.Validate` wraps each batch call in `req.Scheduler.Run(...)` when provided. + +6. **Are proposal-generation and validation LLM calls using the same scheduler, separate schedulers, or no scheduler?** +- By default, same global scheduler instance. +- If validation concurrency is explicitly lower than primary, validation uses a composed scheduler (`global AND validation-subcap`). +- Proposal uses global scheduler only. + +7. **Is there currently a single global LLM concurrency limit?** +- Yes. `--llm-concurrency` (primary concurrency) drives the global scheduler capacity. + +8. **Is there currently a proposal-specific concurrency limit?** +- No dedicated config/flag. Proposal concurrency is bounded by: + - global scheduler permits (`--llm-concurrency`), and + - runner worker fan-out (`maxWorkers`, currently tied to `PrimaryLLM.Concurrency`). + +9. **Is there currently a validation-specific concurrency limit?** +- Yes, optional. `--validation-llm-concurrency` can create an additional validation sub-cap (composed with global). If unset, validation inherits primary/global. + +10. **Is the existing scheduler FIFO, semaphore-only, or otherwise unspecified?** +- Semaphore-only (`chan struct{}` permit pool). FIFO queueing is not explicitly implemented/guaranteed. + +11. **Are proposals applied per chunk as chunks complete, or collected and applied once per module?** +- Collected across all sections first, validated through the module’s validator chain, then applied once per module. + +12. **Are proposal indexes assigned deterministically by module/chunk/segment order, or can completion order affect indexes?** +- Deterministic. Section results are stored by section position and flattened in section order; completion order does not drive index assignment. + +13. **Are report entries deterministic if chunks or validators complete out of order?** +- For chunk proposal execution: yes, deterministic output ordering is preserved. +- Validator decisions are recorded in validator-return order; current validator implementations are deterministic for the same inputs, and LLM-backed validator decisions are sorted by proposal index before return. +- Module-level reporting remains deterministic under normal deterministic/fake-client test conditions. + +14. **Does the current implementation already have test hooks suitable for deterministic concurrency tests?** +- Yes. + - CLI test hooks: injectable LLM clients/schedulers and subprocess test hooks (`AUDITA_SUBPROCESS_TEST_LLM_MODE`, timeout hook). + - Runner tests already verify concurrent section proposals, deterministic index/order under out-of-order completion, and scheduler invocation behavior. + - LLM validator and proposal-generation packages include scheduler-focused tests. + +## Current Flow (Concise) + +1. CLI validates inputs, normalizes transcript, and writes diagnostics artifacts. +2. Runner starts with normalized transcript and executes modules sequentially. +3. For each module: +- chunk current `working` transcript, +- run section proposals concurrently (bounded), +- aggregate proposals deterministically, +- execute validator chain, +- apply approved proposals once, +- emit deterministic module result data. +4. Process report and diagnostics are written; subprocess contracts remain stable. + +## Gaps vs Desired Target Architecture + +Matches target: +- Modules are serial. +- Each module sees previous module’s output transcript. +- Chunks can run concurrently for proposal generation. +- Proposal and validator LLM calls both use scheduler paths. +- Approved proposals are collected then applied once per module. +- Completion order does not affect final proposal index ordering. +- `go test ./...` does not require real LLM credentials. +- Subprocess-oriented behavior remains intact. + +Gaps: +- Missing dedicated `--proposal-llm-concurrency` surface. +- Missing dedicated `--total-llm-concurrency` surface (today this role is played by `--llm-concurrency`). +- Scheduler does not currently provide explicit FIFO semantics/policy abstraction. +- Validation is internally batched and called sequentially within a validator; only scheduler-level sharing enforces global contention, not explicit per-validator parallel fan-out. + +## Minimum Implementation Plan + +1. **Config/CLI surface** +- Add explicit `total llm concurrency` setting and CLI/env wiring. +- Add explicit `proposal llm concurrency` setting and CLI/env wiring. +- Keep existing validation concurrency as the validator sub-cap. +- Validate: `proposal <= total`, `validation <= total`. + +2. **Scheduler construction in CLI runtime** +- Build one global scheduler from total concurrency. +- Build proposal scheduler as composed limiter: `global AND proposal-subcap` when proposal sub-cap is lower than total; otherwise global. +- Keep validation scheduler composition pattern: `global AND validation-subcap` when explicit lower cap exists. + +3. **Runner proposal worker limit alignment** +- Decouple section worker fan-out from `PrimaryLLM.Concurrency` and align with effective proposal concurrency cap to avoid worker oversubscription beyond intended proposal limit. + +4. **FIFO scheduling policy support** +- Introduce scheduler policy abstraction (default FIFO) in `internal/framework/llm` while preserving `contracts.LLMScheduler` interface shape used by runner/modules/validators. +- Keep current behavior-compatible defaults where policy config is not yet exposed. + +5. **Tests** +- Add config/CLI tests for new concurrency constraints and inheritance behavior. +- Add runner/LLM tests proving: + - total cap across proposal + validation combined, + - proposal sub-cap enforcement, + - validation sub-cap enforcement, + - deterministic outputs unchanged under out-of-order completion. + +## Notes + +- No runtime behavior was changed as part of this audit. +- No prompt/module/validator/report schema changes are proposed in this audit.