# LLM Concurrency Audit

Date: 2026-05-12

## Summary

This document audits the current Audita execution flow for module sequencing, chunk proposal execution, validator execution, and LLM concurrency control.

Current implementation already provides:
- serial module execution over a mutable working transcript,
- per-module chunk recomputation from the current working transcript,
- concurrent per-section proposal execution,
- scheduler wiring for both proposal-generation and LLM-backed validator calls,
- deterministic proposal aggregation and deterministic per-module apply ordering,
- subprocess-safe behavior and deterministic test hooks without requiring live LLM credentials.

At audit time, the main gaps versus the requested target architecture were:
- there was no separate `--proposal-llm-concurrency` flag,
- there was no separate `--total-llm-concurrency` flag,
- global concurrency was represented by existing `--llm-concurrency`,
- scheduler implementation was semaphore-based and did not explicitly guarantee FIFO ordering.

## Implementation Status (2026-05-12 Update)

The targeted concurrency gaps identified in this audit have now been addressed:
- explicit total/proposal/validation LLM concurrency controls are implemented in config/env/CLI,
- legacy `llm-concurrency` settings are preserved as compatibility aliases to total concurrency,
- proposal and validation schedulers are composed with a global total-cap scheduler,
- scheduler default behavior is FIFO with context-aware queued cancellation and reliable permit release,
- runner proposal worker fan-out is aligned with effective proposal concurrency,
- intra-module execution now pipelines section validation so proposal and validation LLM work can overlap within a module while retaining deterministic module-level apply ordering.

## Audit Findings (Questions 1-14)

The findings in this section reflect repository state at audit time (before the refactor). See the implementation-status section for current-state behavior.

1. **Does the current runner execute modules serially?**
- Yes. `Runner.Run` loops through `input.ModuleSpecs` sequentially and updates `working` per module.

2. **Are chunks within a module processed serially or concurrently?**
- Concurrently. `collectSectionProposals` starts one goroutine per section, bounded by `maxWorkers`.

3. **Are chunks recomputed per module from the current working transcript, or computed once before all modules?**
- Recomputed per module from current `working` transcript via `chunkWorkingTranscript` inside the module loop.

4. **Are proposal-generation LLM calls routed through the existing scheduler consistently?**
- Yes for production module paths. Each module `Propose` calls `proposal_generation.GenerateCandidates`, which wraps LLM calls in `req.Scheduler.Run(...)` when a scheduler is present.

5. **Are LLM-backed validator calls routed through the existing scheduler consistently?**
- Yes. `LLMBackedValidator.Validate` wraps each batch call in `req.Scheduler.Run(...)` when provided.

6. **Are proposal-generation and validation LLM calls using the same scheduler, separate schedulers, or no scheduler?**
- By default, same global scheduler instance.
- If validation concurrency is explicitly lower than primary, validation uses a composed scheduler (`global AND validation-subcap`).
- Proposal uses global scheduler only.

7. **Is there currently a single global LLM concurrency limit?**
- Yes. `--llm-concurrency` (primary concurrency) drives the global scheduler capacity.

8. **Is there currently a proposal-specific concurrency limit?**
- No dedicated config/flag. Proposal concurrency is bounded by:
  - global scheduler permits (`--llm-concurrency`), and
  - runner worker fan-out (`maxWorkers`, currently tied to `PrimaryLLM.Concurrency`).

9. **Is there currently a validation-specific concurrency limit?**
- Yes, optional. `--validation-llm-concurrency` can create an additional validation sub-cap (composed with global). If unset, validation inherits primary/global.

10. **Is the existing scheduler FIFO, semaphore-only, or otherwise unspecified?**
- Semaphore-only (`chan struct{}` permit pool). FIFO queueing is not explicitly implemented/guaranteed.

11. **Are proposals applied per chunk as chunks complete, or collected and applied once per module?**
- Collected across all sections first, validated through the module’s validator chain, then applied once per module.

12. **Are proposal indexes assigned deterministically by module/chunk/segment order, or can completion order affect indexes?**
- Deterministic. Section results are stored by section position and flattened in section order; completion order does not drive index assignment.

13. **Are report entries deterministic if chunks or validators complete out of order?**
- For chunk proposal execution: yes, deterministic output ordering is preserved.
- Validator decisions are recorded in validator-return order; current validator implementations are deterministic for the same inputs, and LLM-backed validator decisions are sorted by proposal index before return.
- Module-level reporting remains deterministic under normal deterministic/fake-client test conditions.

14. **Does the current implementation already have test hooks suitable for deterministic concurrency tests?**
- Yes.
  - CLI test hooks: injectable LLM clients/schedulers and subprocess test hooks (`AUDITA_SUBPROCESS_TEST_LLM_MODE`, timeout hook).
  - Runner tests already verify concurrent section proposals, deterministic index/order under out-of-order completion, and scheduler invocation behavior.
  - LLM validator and proposal-generation packages include scheduler-focused tests.

## Current Flow (Concise)

1. CLI validates inputs, normalizes transcript, and writes diagnostics artifacts.
2. Runner starts with normalized transcript and executes modules sequentially.
3. For each module:
- chunk current `working` transcript,
- run section proposals concurrently (bounded),
- aggregate proposals deterministically,
- execute validator chain,
- apply approved proposals once,
- emit deterministic module result data.
4. Process report and diagnostics are written; subprocess contracts remain stable.

## Gaps vs Desired Target Architecture (Audit-Time Snapshot)

Matches target:
- Modules are serial.
- Each module sees previous module’s output transcript.
- Chunks can run concurrently for proposal generation.
- Proposal and validator LLM calls both use scheduler paths.
- Approved proposals are collected then applied once per module.
- Completion order does not affect final proposal index ordering.
- `go test ./...` does not require real LLM credentials.
- Subprocess-oriented behavior remains intact.

Gaps at audit time:
- missing dedicated `--proposal-llm-concurrency` surface,
- missing dedicated `--total-llm-concurrency` surface (at the time this role was played by `--llm-concurrency`),
- scheduler did not provide explicit FIFO semantics,
- validation remained internally batched and sequential within one validator invocation.

## Minimum Implementation Plan (Completed)

1. **Config/CLI surface**
- Add explicit `total llm concurrency` setting and CLI/env wiring.
- Add explicit `proposal llm concurrency` setting and CLI/env wiring.
- Keep existing validation concurrency as the validator sub-cap.
- Validate: `proposal <= total`, `validation <= total`.

2. **Scheduler construction in CLI runtime**
- Build one global scheduler from total concurrency.
- Build proposal scheduler as composed limiter: `global AND proposal-subcap` when proposal sub-cap is lower than total; otherwise global.
- Keep validation scheduler composition pattern: `global AND validation-subcap` when explicit lower cap exists.

3. **Runner proposal worker limit alignment**
- Decouple section worker fan-out from `PrimaryLLM.Concurrency` and align with effective proposal concurrency cap to avoid worker oversubscription beyond intended proposal limit.

4. **FIFO scheduling policy support**
- Introduce scheduler policy abstraction (default FIFO) in `internal/framework/llm` while preserving `contracts.LLMScheduler` interface shape used by runner/modules/validators.
- Keep current behavior-compatible defaults where policy config is not yet exposed.

5. **Tests**
- Add config/CLI tests for new concurrency constraints and inheritance behavior.
- Add runner/LLM tests proving:
  - total cap across proposal + validation combined,
  - proposal sub-cap enforcement,
  - validation sub-cap enforcement,
  - deterministic outputs unchanged under out-of-order completion.

## Notes

- This document is retained as an audit record; see the implementation-status section for current behavior.
- No prompt/module/validator/report schema changes were required to close the identified concurrency gaps.