audita/docs/llm-concurrency-audit.md

LLM Concurrency Audit

Date: 2026-05-12

Summary

This document audits the current Audita execution flow for module sequencing, chunk proposal execution, validator execution, and LLM concurrency control.

Current implementation already provides:

• serial module execution over a mutable working transcript,
• per-module chunk recomputation from the current working transcript,
• concurrent per-section proposal execution,
• scheduler wiring for both proposal-generation and LLM-backed validator calls,
• deterministic proposal aggregation and deterministic per-module apply ordering,
• subprocess-safe behavior and deterministic test hooks without requiring live LLM credentials.

Main gap versus the requested target architecture:

• there is no separate --proposal-llm-concurrency flag,
• there is no separate --total-llm-concurrency flag,
• global concurrency is currently represented by existing --llm-concurrency,
• scheduler implementation is semaphore-based and does not explicitly guarantee FIFO ordering.

Audit Findings (Questions 1-14)

1. Does the current runner execute modules serially?

• Yes. Runner.Run loops through input.ModuleSpecs sequentially and updates working per module.

2. Are chunks within a module processed serially or concurrently?

• Concurrently. collectSectionProposals starts one goroutine per section, bounded by maxWorkers.

3. Are chunks recomputed per module from the current working transcript, or computed once before all modules?

• Recomputed per module from current working transcript via chunkWorkingTranscript inside the module loop.

4. Are proposal-generation LLM calls routed through the existing scheduler consistently?

• Yes for production module paths. Each module Propose calls proposal_generation.GenerateCandidates, which wraps LLM calls in req.Scheduler.Run(...) when a scheduler is present.

5. Are LLM-backed validator calls routed through the existing scheduler consistently?

• Yes. LLMBackedValidator.Validate wraps each batch call in req.Scheduler.Run(...) when provided.

6. Are proposal-generation and validation LLM calls using the same scheduler, separate schedulers, or no scheduler?

• By default, same global scheduler instance.
• If validation concurrency is explicitly lower than primary, validation uses a composed scheduler (global AND validation-subcap).
• Proposal uses global scheduler only.

7. Is there currently a single global LLM concurrency limit?

• Yes. --llm-concurrency (primary concurrency) drives the global scheduler capacity.

8. Is there currently a proposal-specific concurrency limit?

• No dedicated config/flag. Proposal concurrency is bounded by:
  • global scheduler permits (--llm-concurrency), and
  • runner worker fan-out (maxWorkers, currently tied to PrimaryLLM.Concurrency).

9. Is there currently a validation-specific concurrency limit?

• Yes, optional. --validation-llm-concurrency can create an additional validation sub-cap (composed with global). If unset, validation inherits primary/global.

10. Is the existing scheduler FIFO, semaphore-only, or otherwise unspecified?

• Semaphore-only (chan struct{} permit pool). FIFO queueing is not explicitly implemented/guaranteed.

11. Are proposals applied per chunk as chunks complete, or collected and applied once per module?

• Collected across all sections first, validated through the module’s validator chain, then applied once per module.

12. Are proposal indexes assigned deterministically by module/chunk/segment order, or can completion order affect indexes?

• Deterministic. Section results are stored by section position and flattened in section order; completion order does not drive index assignment.

13. Are report entries deterministic if chunks or validators complete out of order?

• For chunk proposal execution: yes, deterministic output ordering is preserved.
• Validator decisions are recorded in validator-return order; current validator implementations are deterministic for the same inputs, and LLM-backed validator decisions are sorted by proposal index before return.
• Module-level reporting remains deterministic under normal deterministic/fake-client test conditions.

14. Does the current implementation already have test hooks suitable for deterministic concurrency tests?

• Yes.
  • CLI test hooks: injectable LLM clients/schedulers and subprocess test hooks (AUDITA_SUBPROCESS_TEST_LLM_MODE, timeout hook).
  • Runner tests already verify concurrent section proposals, deterministic index/order under out-of-order completion, and scheduler invocation behavior.
  • LLM validator and proposal-generation packages include scheduler-focused tests.

Current Flow (Concise)

1. CLI validates inputs, normalizes transcript, and writes diagnostics artifacts.
2. Runner starts with normalized transcript and executes modules sequentially.
3. For each module:

• chunk current working transcript,
• run section proposals concurrently (bounded),
• aggregate proposals deterministically,
• execute validator chain,
• apply approved proposals once,
• emit deterministic module result data.

4. Process report and diagnostics are written; subprocess contracts remain stable.

Gaps vs Desired Target Architecture

Matches target:

• Modules are serial.
• Each module sees previous module’s output transcript.
• Chunks can run concurrently for proposal generation.
• Proposal and validator LLM calls both use scheduler paths.
• Approved proposals are collected then applied once per module.
• Completion order does not affect final proposal index ordering.
• go test ./... does not require real LLM credentials.
• Subprocess-oriented behavior remains intact.

Gaps:

• Missing dedicated --proposal-llm-concurrency surface.
• Missing dedicated --total-llm-concurrency surface (today this role is played by --llm-concurrency).
• Scheduler does not currently provide explicit FIFO semantics/policy abstraction.
• Validation is internally batched and called sequentially within a validator; only scheduler-level sharing enforces global contention, not explicit per-validator parallel fan-out.

Minimum Implementation Plan

1. Config/CLI surface

• Add explicit total llm concurrency setting and CLI/env wiring.
• Add explicit proposal llm concurrency setting and CLI/env wiring.
• Keep existing validation concurrency as the validator sub-cap.
• Validate: proposal <= total, validation <= total.

2. Scheduler construction in CLI runtime

• Build one global scheduler from total concurrency.
• Build proposal scheduler as composed limiter: global AND proposal-subcap when proposal sub-cap is lower than total; otherwise global.
• Keep validation scheduler composition pattern: global AND validation-subcap when explicit lower cap exists.

3. Runner proposal worker limit alignment

• Decouple section worker fan-out from PrimaryLLM.Concurrency and align with effective proposal concurrency cap to avoid worker oversubscription beyond intended proposal limit.

4. FIFO scheduling policy support

• Introduce scheduler policy abstraction (default FIFO) in internal/framework/llm while preserving contracts.LLMScheduler interface shape used by runner/modules/validators.
• Keep current behavior-compatible defaults where policy config is not yet exposed.

5. Tests

• Add config/CLI tests for new concurrency constraints and inheritance behavior.
• Add runner/LLM tests proving:
  • total cap across proposal + validation combined,
  • proposal sub-cap enforcement,
  • validation sub-cap enforcement,
  • deterministic outputs unchanged under out-of-order completion.

Notes

• No runtime behavior was changed as part of this audit.
• No prompt/module/validator/report schema changes are proposed in this audit.