Audit LLM concurrency flow

docs/llm-concurrency-audit.md

@@ -0,0 +1,138 @@
+# LLM Concurrency Audit
+
+Date: 2026-05-12
+
+## Summary
+
+This document audits the current Audita execution flow for module sequencing, chunk proposal execution, validator execution, and LLM concurrency control.
+
+Current implementation already provides:
+- serial module execution over a mutable working transcript,
+- per-module chunk recomputation from the current working transcript,
+- concurrent per-section proposal execution,
+- scheduler wiring for both proposal-generation and LLM-backed validator calls,
+- deterministic proposal aggregation and deterministic per-module apply ordering,
+- subprocess-safe behavior and deterministic test hooks without requiring live LLM credentials.
+
+Main gap versus the requested target architecture:
+- there is no separate `--proposal-llm-concurrency` flag,
+- there is no separate `--total-llm-concurrency` flag,
+- global concurrency is currently represented by existing `--llm-concurrency`,
+- scheduler implementation is semaphore-based and does not explicitly guarantee FIFO ordering.
+
+## Audit Findings (Questions 1-14)
+
+1. **Does the current runner execute modules serially?**
+- Yes. `Runner.Run` loops through `input.ModuleSpecs` sequentially and updates `working` per module.
+
+2. **Are chunks within a module processed serially or concurrently?**
+- Concurrently. `collectSectionProposals` starts one goroutine per section, bounded by `maxWorkers`.
+
+3. **Are chunks recomputed per module from the current working transcript, or computed once before all modules?**
+- Recomputed per module from current `working` transcript via `chunkWorkingTranscript` inside the module loop.
+
+4. **Are proposal-generation LLM calls routed through the existing scheduler consistently?**
+- Yes for production module paths. Each module `Propose` calls `proposal_generation.GenerateCandidates`, which wraps LLM calls in `req.Scheduler.Run(...)` when a scheduler is present.
+
+5. **Are LLM-backed validator calls routed through the existing scheduler consistently?**
+- Yes. `LLMBackedValidator.Validate` wraps each batch call in `req.Scheduler.Run(...)` when provided.
+
+6. **Are proposal-generation and validation LLM calls using the same scheduler, separate schedulers, or no scheduler?**
+- By default, same global scheduler instance.
+- If validation concurrency is explicitly lower than primary, validation uses a composed scheduler (`global AND validation-subcap`).
+- Proposal uses global scheduler only.
+
+7. **Is there currently a single global LLM concurrency limit?**
+- Yes. `--llm-concurrency` (primary concurrency) drives the global scheduler capacity.
+
+8. **Is there currently a proposal-specific concurrency limit?**
+- No dedicated config/flag. Proposal concurrency is bounded by:
+  - global scheduler permits (`--llm-concurrency`), and
+  - runner worker fan-out (`maxWorkers`, currently tied to `PrimaryLLM.Concurrency`).
+
+9. **Is there currently a validation-specific concurrency limit?**
+- Yes, optional. `--validation-llm-concurrency` can create an additional validation sub-cap (composed with global). If unset, validation inherits primary/global.
+
+10. **Is the existing scheduler FIFO, semaphore-only, or otherwise unspecified?**
+- Semaphore-only (`chan struct{}` permit pool). FIFO queueing is not explicitly implemented/guaranteed.
+
+11. **Are proposals applied per chunk as chunks complete, or collected and applied once per module?**
+- Collected across all sections first, validated through the module’s validator chain, then applied once per module.
+
+12. **Are proposal indexes assigned deterministically by module/chunk/segment order, or can completion order affect indexes?**
+- Deterministic. Section results are stored by section position and flattened in section order; completion order does not drive index assignment.
+
+13. **Are report entries deterministic if chunks or validators complete out of order?**
+- For chunk proposal execution: yes, deterministic output ordering is preserved.
+- Validator decisions are recorded in validator-return order; current validator implementations are deterministic for the same inputs, and LLM-backed validator decisions are sorted by proposal index before return.
+- Module-level reporting remains deterministic under normal deterministic/fake-client test conditions.
+
+14. **Does the current implementation already have test hooks suitable for deterministic concurrency tests?**
+- Yes.
+  - CLI test hooks: injectable LLM clients/schedulers and subprocess test hooks (`AUDITA_SUBPROCESS_TEST_LLM_MODE`, timeout hook).
+  - Runner tests already verify concurrent section proposals, deterministic index/order under out-of-order completion, and scheduler invocation behavior.
+  - LLM validator and proposal-generation packages include scheduler-focused tests.
+
+## Current Flow (Concise)
+
+1. CLI validates inputs, normalizes transcript, and writes diagnostics artifacts.
+2. Runner starts with normalized transcript and executes modules sequentially.
+3. For each module:
+- chunk current `working` transcript,
+- run section proposals concurrently (bounded),
+- aggregate proposals deterministically,
+- execute validator chain,
+- apply approved proposals once,
+- emit deterministic module result data.
+4. Process report and diagnostics are written; subprocess contracts remain stable.
+
+## Gaps vs Desired Target Architecture
+
+Matches target:
+- Modules are serial.
+- Each module sees previous module’s output transcript.
+- Chunks can run concurrently for proposal generation.
+- Proposal and validator LLM calls both use scheduler paths.
+- Approved proposals are collected then applied once per module.
+- Completion order does not affect final proposal index ordering.
+- `go test ./...` does not require real LLM credentials.
+- Subprocess-oriented behavior remains intact.
+
+Gaps:
+- Missing dedicated `--proposal-llm-concurrency` surface.
+- Missing dedicated `--total-llm-concurrency` surface (today this role is played by `--llm-concurrency`).
+- Scheduler does not currently provide explicit FIFO semantics/policy abstraction.
+- Validation is internally batched and called sequentially within a validator; only scheduler-level sharing enforces global contention, not explicit per-validator parallel fan-out.
+
+## Minimum Implementation Plan
+
+1. **Config/CLI surface**
+- Add explicit `total llm concurrency` setting and CLI/env wiring.
+- Add explicit `proposal llm concurrency` setting and CLI/env wiring.
+- Keep existing validation concurrency as the validator sub-cap.
+- Validate: `proposal <= total`, `validation <= total`.
+
+2. **Scheduler construction in CLI runtime**
+- Build one global scheduler from total concurrency.
+- Build proposal scheduler as composed limiter: `global AND proposal-subcap` when proposal sub-cap is lower than total; otherwise global.
+- Keep validation scheduler composition pattern: `global AND validation-subcap` when explicit lower cap exists.
+
+3. **Runner proposal worker limit alignment**
+- Decouple section worker fan-out from `PrimaryLLM.Concurrency` and align with effective proposal concurrency cap to avoid worker oversubscription beyond intended proposal limit.
+
+4. **FIFO scheduling policy support**
+- Introduce scheduler policy abstraction (default FIFO) in `internal/framework/llm` while preserving `contracts.LLMScheduler` interface shape used by runner/modules/validators.
+- Keep current behavior-compatible defaults where policy config is not yet exposed.
+
+5. **Tests**
+- Add config/CLI tests for new concurrency constraints and inheritance behavior.
+- Add runner/LLM tests proving:
+  - total cap across proposal + validation combined,
+  - proposal sub-cap enforcement,
+  - validation sub-cap enforcement,
+  - deterministic outputs unchanged under out-of-order completion.
+
+## Notes
+
+- No runtime behavior was changed as part of this audit.
+- No prompt/module/validator/report schema changes are proposed in this audit.