150 lines
8.4 KiB
Markdown
150 lines
8.4 KiB
Markdown
# LLM Concurrency Audit
|
||
|
||
Date: 2026-05-12
|
||
|
||
## Summary
|
||
|
||
This document audits the current Audita execution flow for module sequencing, chunk proposal execution, validator execution, and LLM concurrency control.
|
||
|
||
Current implementation already provides:
|
||
- serial module execution over a mutable working transcript,
|
||
- per-module chunk recomputation from the current working transcript,
|
||
- concurrent per-section proposal execution,
|
||
- scheduler wiring for both proposal-generation and LLM-backed validator calls,
|
||
- deterministic proposal aggregation and deterministic per-module apply ordering,
|
||
- subprocess-safe behavior and deterministic test hooks without requiring live LLM credentials.
|
||
|
||
At audit time, the main gaps versus the requested target architecture were:
|
||
- there was no separate `--proposal-llm-concurrency` flag,
|
||
- there was no separate `--total-llm-concurrency` flag,
|
||
- global concurrency was represented by existing `--llm-concurrency`,
|
||
- scheduler implementation was semaphore-based and did not explicitly guarantee FIFO ordering.
|
||
|
||
## Implementation Status (2026-05-12 Update)
|
||
|
||
The targeted concurrency gaps identified in this audit have now been addressed:
|
||
- explicit total/proposal/validation LLM concurrency controls are implemented in config/env/CLI,
|
||
- legacy `llm-concurrency` settings are preserved as compatibility aliases to total concurrency,
|
||
- proposal and validation schedulers are composed with a global total-cap scheduler,
|
||
- scheduler default behavior is FIFO with context-aware queued cancellation and reliable permit release,
|
||
- runner proposal worker fan-out is aligned with effective proposal concurrency.
|
||
|
||
## Audit Findings (Questions 1-14)
|
||
|
||
The findings in this section reflect repository state at audit time (before the refactor). See the implementation-status section for current-state behavior.
|
||
|
||
1. **Does the current runner execute modules serially?**
|
||
- Yes. `Runner.Run` loops through `input.ModuleSpecs` sequentially and updates `working` per module.
|
||
|
||
2. **Are chunks within a module processed serially or concurrently?**
|
||
- Concurrently. `collectSectionProposals` starts one goroutine per section, bounded by `maxWorkers`.
|
||
|
||
3. **Are chunks recomputed per module from the current working transcript, or computed once before all modules?**
|
||
- Recomputed per module from current `working` transcript via `chunkWorkingTranscript` inside the module loop.
|
||
|
||
4. **Are proposal-generation LLM calls routed through the existing scheduler consistently?**
|
||
- Yes for production module paths. Each module `Propose` calls `proposal_generation.GenerateCandidates`, which wraps LLM calls in `req.Scheduler.Run(...)` when a scheduler is present.
|
||
|
||
5. **Are LLM-backed validator calls routed through the existing scheduler consistently?**
|
||
- Yes. `LLMBackedValidator.Validate` wraps each batch call in `req.Scheduler.Run(...)` when provided.
|
||
|
||
6. **Are proposal-generation and validation LLM calls using the same scheduler, separate schedulers, or no scheduler?**
|
||
- By default, same global scheduler instance.
|
||
- If validation concurrency is explicitly lower than primary, validation uses a composed scheduler (`global AND validation-subcap`).
|
||
- Proposal uses global scheduler only.
|
||
|
||
7. **Is there currently a single global LLM concurrency limit?**
|
||
- Yes. `--llm-concurrency` (primary concurrency) drives the global scheduler capacity.
|
||
|
||
8. **Is there currently a proposal-specific concurrency limit?**
|
||
- No dedicated config/flag. Proposal concurrency is bounded by:
|
||
- global scheduler permits (`--llm-concurrency`), and
|
||
- runner worker fan-out (`maxWorkers`, currently tied to `PrimaryLLM.Concurrency`).
|
||
|
||
9. **Is there currently a validation-specific concurrency limit?**
|
||
- Yes, optional. `--validation-llm-concurrency` can create an additional validation sub-cap (composed with global). If unset, validation inherits primary/global.
|
||
|
||
10. **Is the existing scheduler FIFO, semaphore-only, or otherwise unspecified?**
|
||
- Semaphore-only (`chan struct{}` permit pool). FIFO queueing is not explicitly implemented/guaranteed.
|
||
|
||
11. **Are proposals applied per chunk as chunks complete, or collected and applied once per module?**
|
||
- Collected across all sections first, validated through the module’s validator chain, then applied once per module.
|
||
|
||
12. **Are proposal indexes assigned deterministically by module/chunk/segment order, or can completion order affect indexes?**
|
||
- Deterministic. Section results are stored by section position and flattened in section order; completion order does not drive index assignment.
|
||
|
||
13. **Are report entries deterministic if chunks or validators complete out of order?**
|
||
- For chunk proposal execution: yes, deterministic output ordering is preserved.
|
||
- Validator decisions are recorded in validator-return order; current validator implementations are deterministic for the same inputs, and LLM-backed validator decisions are sorted by proposal index before return.
|
||
- Module-level reporting remains deterministic under normal deterministic/fake-client test conditions.
|
||
|
||
14. **Does the current implementation already have test hooks suitable for deterministic concurrency tests?**
|
||
- Yes.
|
||
- CLI test hooks: injectable LLM clients/schedulers and subprocess test hooks (`AUDITA_SUBPROCESS_TEST_LLM_MODE`, timeout hook).
|
||
- Runner tests already verify concurrent section proposals, deterministic index/order under out-of-order completion, and scheduler invocation behavior.
|
||
- LLM validator and proposal-generation packages include scheduler-focused tests.
|
||
|
||
## Current Flow (Concise)
|
||
|
||
1. CLI validates inputs, normalizes transcript, and writes diagnostics artifacts.
|
||
2. Runner starts with normalized transcript and executes modules sequentially.
|
||
3. For each module:
|
||
- chunk current `working` transcript,
|
||
- run section proposals concurrently (bounded),
|
||
- aggregate proposals deterministically,
|
||
- execute validator chain,
|
||
- apply approved proposals once,
|
||
- emit deterministic module result data.
|
||
4. Process report and diagnostics are written; subprocess contracts remain stable.
|
||
|
||
## Gaps vs Desired Target Architecture (Audit-Time Snapshot)
|
||
|
||
Matches target:
|
||
- Modules are serial.
|
||
- Each module sees previous module’s output transcript.
|
||
- Chunks can run concurrently for proposal generation.
|
||
- Proposal and validator LLM calls both use scheduler paths.
|
||
- Approved proposals are collected then applied once per module.
|
||
- Completion order does not affect final proposal index ordering.
|
||
- `go test ./...` does not require real LLM credentials.
|
||
- Subprocess-oriented behavior remains intact.
|
||
|
||
Gaps at audit time:
|
||
- missing dedicated `--proposal-llm-concurrency` surface,
|
||
- missing dedicated `--total-llm-concurrency` surface (at the time this role was played by `--llm-concurrency`),
|
||
- scheduler did not provide explicit FIFO semantics,
|
||
- validation remained internally batched and sequential within one validator invocation.
|
||
|
||
## Minimum Implementation Plan (Completed)
|
||
|
||
1. **Config/CLI surface**
|
||
- Add explicit `total llm concurrency` setting and CLI/env wiring.
|
||
- Add explicit `proposal llm concurrency` setting and CLI/env wiring.
|
||
- Keep existing validation concurrency as the validator sub-cap.
|
||
- Validate: `proposal <= total`, `validation <= total`.
|
||
|
||
2. **Scheduler construction in CLI runtime**
|
||
- Build one global scheduler from total concurrency.
|
||
- Build proposal scheduler as composed limiter: `global AND proposal-subcap` when proposal sub-cap is lower than total; otherwise global.
|
||
- Keep validation scheduler composition pattern: `global AND validation-subcap` when explicit lower cap exists.
|
||
|
||
3. **Runner proposal worker limit alignment**
|
||
- Decouple section worker fan-out from `PrimaryLLM.Concurrency` and align with effective proposal concurrency cap to avoid worker oversubscription beyond intended proposal limit.
|
||
|
||
4. **FIFO scheduling policy support**
|
||
- Introduce scheduler policy abstraction (default FIFO) in `internal/framework/llm` while preserving `contracts.LLMScheduler` interface shape used by runner/modules/validators.
|
||
- Keep current behavior-compatible defaults where policy config is not yet exposed.
|
||
|
||
5. **Tests**
|
||
- Add config/CLI tests for new concurrency constraints and inheritance behavior.
|
||
- Add runner/LLM tests proving:
|
||
- total cap across proposal + validation combined,
|
||
- proposal sub-cap enforcement,
|
||
- validation sub-cap enforcement,
|
||
- deterministic outputs unchanged under out-of-order completion.
|
||
|
||
## Notes
|
||
|
||
- This document is retained as an audit record; see the implementation-status section for current behavior.
|
||
- No prompt/module/validator/report schema changes were required to close the identified concurrency gaps.
|