audita/docs/llm-concurrency-audit.md

LLM Concurrency Audit

Date: 2026-05-12

Summary

This document audits the current Audita execution flow for module sequencing, chunk proposal execution, validator execution, and LLM concurrency control.

Current implementation already provides:

• serial module execution over a mutable working transcript,
• per-module chunk recomputation from the current working transcript,
• concurrent per-section proposal execution,
• scheduler wiring for both proposal-generation and LLM-backed validator calls,
• deterministic proposal aggregation and deterministic per-module apply ordering,
• subprocess-safe behavior and deterministic test hooks without requiring live LLM credentials.

At audit time, the main gaps versus the requested target architecture were:

• there was no separate --proposal-llm-concurrency flag,
• there was no separate --total-llm-concurrency flag,
• global concurrency was represented by existing --llm-concurrency,
• scheduler implementation was semaphore-based and did not explicitly guarantee FIFO ordering.

Implementation Status (2026-05-12 Update)

The targeted concurrency gaps identified in this audit have now been addressed:

• explicit total/proposal/validation LLM concurrency controls are implemented in config/env/CLI,
• legacy llm-concurrency settings are preserved as compatibility aliases to total concurrency,
• proposal and validation schedulers are composed with a global total-cap scheduler,
• scheduler default behavior is FIFO with context-aware queued cancellation and reliable permit release,
• runner proposal worker fan-out is aligned with effective proposal concurrency.

Audit Findings (Questions 1-14)

The findings in this section reflect repository state at audit time (before the refactor). See the implementation-status section for current-state behavior.

1. Does the current runner execute modules serially?

• Yes. Runner.Run loops through input.ModuleSpecs sequentially and updates working per module.

2. Are chunks within a module processed serially or concurrently?

• Concurrently. collectSectionProposals starts one goroutine per section, bounded by maxWorkers.

3. Are chunks recomputed per module from the current working transcript, or computed once before all modules?

• Recomputed per module from current working transcript via chunkWorkingTranscript inside the module loop.

4. Are proposal-generation LLM calls routed through the existing scheduler consistently?

• Yes for production module paths. Each module Propose calls proposal_generation.GenerateCandidates, which wraps LLM calls in req.Scheduler.Run(...) when a scheduler is present.

5. Are LLM-backed validator calls routed through the existing scheduler consistently?

• Yes. LLMBackedValidator.Validate wraps each batch call in req.Scheduler.Run(...) when provided.

6. Are proposal-generation and validation LLM calls using the same scheduler, separate schedulers, or no scheduler?

• By default, same global scheduler instance.
• If validation concurrency is explicitly lower than primary, validation uses a composed scheduler (global AND validation-subcap).
• Proposal uses global scheduler only.

7. Is there currently a single global LLM concurrency limit?

• Yes. --llm-concurrency (primary concurrency) drives the global scheduler capacity.

8. Is there currently a proposal-specific concurrency limit?

• No dedicated config/flag. Proposal concurrency is bounded by:
  • global scheduler permits (--llm-concurrency), and
  • runner worker fan-out (maxWorkers, currently tied to PrimaryLLM.Concurrency).

9. Is there currently a validation-specific concurrency limit?

• Yes, optional. --validation-llm-concurrency can create an additional validation sub-cap (composed with global). If unset, validation inherits primary/global.

10. Is the existing scheduler FIFO, semaphore-only, or otherwise unspecified?

• Semaphore-only (chan struct{} permit pool). FIFO queueing is not explicitly implemented/guaranteed.

11. Are proposals applied per chunk as chunks complete, or collected and applied once per module?

• Collected across all sections first, validated through the module’s validator chain, then applied once per module.

12. Are proposal indexes assigned deterministically by module/chunk/segment order, or can completion order affect indexes?

• Deterministic. Section results are stored by section position and flattened in section order; completion order does not drive index assignment.

13. Are report entries deterministic if chunks or validators complete out of order?

• For chunk proposal execution: yes, deterministic output ordering is preserved.
• Validator decisions are recorded in validator-return order; current validator implementations are deterministic for the same inputs, and LLM-backed validator decisions are sorted by proposal index before return.
• Module-level reporting remains deterministic under normal deterministic/fake-client test conditions.

14. Does the current implementation already have test hooks suitable for deterministic concurrency tests?

• Yes.
  • CLI test hooks: injectable LLM clients/schedulers and subprocess test hooks (AUDITA_SUBPROCESS_TEST_LLM_MODE, timeout hook).
  • Runner tests already verify concurrent section proposals, deterministic index/order under out-of-order completion, and scheduler invocation behavior.
  • LLM validator and proposal-generation packages include scheduler-focused tests.

Current Flow (Concise)

1. CLI validates inputs, normalizes transcript, and writes diagnostics artifacts.
2. Runner starts with normalized transcript and executes modules sequentially.
3. For each module:

• chunk current working transcript,
• run section proposals concurrently (bounded),
• aggregate proposals deterministically,
• execute validator chain,
• apply approved proposals once,
• emit deterministic module result data.

4. Process report and diagnostics are written; subprocess contracts remain stable.

Gaps vs Desired Target Architecture (Audit-Time Snapshot)

Matches target:

• Modules are serial.
• Each module sees previous module’s output transcript.
• Chunks can run concurrently for proposal generation.
• Proposal and validator LLM calls both use scheduler paths.
• Approved proposals are collected then applied once per module.
• Completion order does not affect final proposal index ordering.
• go test ./... does not require real LLM credentials.
• Subprocess-oriented behavior remains intact.

Gaps at audit time:

• missing dedicated --proposal-llm-concurrency surface,
• missing dedicated --total-llm-concurrency surface (at the time this role was played by --llm-concurrency),
• scheduler did not provide explicit FIFO semantics,
• validation remained internally batched and sequential within one validator invocation.

Minimum Implementation Plan (Completed)

1. Config/CLI surface

• Add explicit total llm concurrency setting and CLI/env wiring.
• Add explicit proposal llm concurrency setting and CLI/env wiring.
• Keep existing validation concurrency as the validator sub-cap.
• Validate: proposal <= total, validation <= total.

2. Scheduler construction in CLI runtime

• Build one global scheduler from total concurrency.
• Build proposal scheduler as composed limiter: global AND proposal-subcap when proposal sub-cap is lower than total; otherwise global.
• Keep validation scheduler composition pattern: global AND validation-subcap when explicit lower cap exists.

3. Runner proposal worker limit alignment

• Decouple section worker fan-out from PrimaryLLM.Concurrency and align with effective proposal concurrency cap to avoid worker oversubscription beyond intended proposal limit.

4. FIFO scheduling policy support

• Introduce scheduler policy abstraction (default FIFO) in internal/framework/llm while preserving contracts.LLMScheduler interface shape used by runner/modules/validators.
• Keep current behavior-compatible defaults where policy config is not yet exposed.

5. Tests

• Add config/CLI tests for new concurrency constraints and inheritance behavior.
• Add runner/LLM tests proving:
  • total cap across proposal + validation combined,
  • proposal sub-cap enforcement,
  • validation sub-cap enforcement,
  • deterministic outputs unchanged under out-of-order completion.

Notes

• This document is retained as an audit record; see the implementation-status section for current behavior.
• No prompt/module/validator/report schema changes were required to close the identified concurrency gaps.