# Intra-Module LLM Pipeline Audit

Date: 2026-05-12

## Implementation Status (2026-05-12 Update)

The intra-module pipelining gap identified in this audit has now been addressed:
- section proposal jobs are launched promptly for each module;
- as section proposals become available in deterministic section order, section-local validator work starts without waiting for all section proposals to finish;
- deterministic validators run before LLM-backed validators for each section;
- proposal and validation LLM calls can overlap through the existing composed scheduler path;
- module application remains a single deterministic apply barrier.

## Summary

This audit checks whether Audita currently maximizes available LLM concurrency within each module by overlapping proposal and validation work.

Current state:
- Modules are serial and deterministic.
- Section proposal generation is concurrent and scheduler-limited.
- Validation (deterministic + LLM-backed) starts only after all section proposals complete.
- Proposal and validation LLM work therefore does **not** overlap in time within a module today.

Conclusion:
- Current behavior is correct and deterministic, but it does **not** fully match the target pipelined intra-module behavior.
- A narrow runner refactor is needed to pipeline section-level validation after section proposal completion while preserving apply-once-per-module semantics.

## Current Execution Flow (Per Module)

Implementation anchor: `internal/framework/runner/runner.go`.

1. `chunkWorkingTranscript` is called for the module.
2. `collectSectionProposals(...)` runs section `Module.Propose(...)` calls concurrently.
3. Runner waits for **all** proposal goroutines to finish (`wg.Wait()`).
4. Proposals are flattened deterministically by section order and assigned proposal indexes.
5. Validator chain runs over the full module proposal set (`eligible := enriched`; loop over `module.Validators()`).
6. Approved proposals are applied once via `proposals.ApplyProposals(...)`.

## Audit Answers

1. **At module start, are all section proposal jobs launched/queued promptly, or in smaller batches?**
- Jobs are launched promptly (goroutine per section), but section entry is throttled by worker semaphore in `collectSectionProposals`.

2. **Does worker fan-out submit all sections and let scheduler enforce concurrency, or does fan-out itself block scheduler entry?**
- Fan-out itself blocks scheduler entry: runner’s `sem` gate limits how many section jobs can even call `Module.Propose` (and thus reach scheduler).

3. **When a section proposal returns, do deterministic validators run immediately for that section?**
- No. Deterministic validators run only after all section proposals finish and aggregation completes.

4. **When deterministic validators pass, are LLM validator jobs submitted immediately?**
- No. LLM validator calls occur only during module-level validator pass after full proposal collection.

5. **Does validator chain architecture allow section-level validation independently?**
- Not as currently orchestrated by runner; validators receive module-wide `CandidateProposal` slices.

6. **Does LLM validator batching depend on all module proposals being present?**
- Current invocation pattern does. `LLMBackedValidator.Validate` batches over the full provided candidate set; runner currently supplies full-module candidates.

7. **Would immediate per-section validator submission reduce batching efficiency or change semantics?**
- Likely yes for efficiency: smaller per-section batches can increase LLM calls.
- Semantics can remain equivalent if ordering/cardinality/report mapping is preserved, but batching shape and diagnostic timing will differ.

8. **When are proposal indexes assigned?**
- After concurrent proposal responses return, during deterministic aggregation in `collectSectionProposals`.

9. **Can immediate validator submission preserve deterministic proposal indexes/report order?**
- Yes, if indexes are preallocated/stable by section-order offsets (or equivalent deterministic mapping) before emitting validator work.

10. **Are diagnostics paths stable/deterministic enough if validator calls interleave with proposal calls?**
- Mostly yes: stage names are deterministic (`module:proposal:section-*`, `module:validator:batch-*`).
- But batch indices and artifact emission timing could change unless explicitly stabilized by per-section deterministic indexing.

11. **Do schedulers enforce total/proposal/validation limits correctly when proposal and validation overlap?**
- Scheduler composition supports this (`global` + proposal/validation subcaps), but runner currently does not create overlap, so end-to-end overlap behavior is not exercised by runtime path.

12. **Do existing tests prove proposal and validation jobs can overlap while respecting total concurrency?**
- No direct end-to-end runner test proving overlap of proposal and validation jobs in one module.

13. **Are there tests proving FIFO queue contains both proposal and validation jobs in submission order?**
- No cross-type FIFO test. FIFO is tested at scheduler unit level (`internal/framework/llm/scheduler_test.go`), and composed-cap tests exist in CLI tests, but not mixed proposal+validation submission order in runner.

14. **Smallest safe implementation change to reach target behavior?**
- Refactor runner module execution into a two-lane pipeline:
  - keep current concurrent section proposal launch,
  - on each section completion, run deterministic validators for that section’s proposals,
  - immediately enqueue section LLM-validator work for survivors,
  - collect all approved proposals in deterministic global proposal-index order,
  - apply once per module exactly as today.

## Minimal Change Plan (No Redesign)

Primary change area:
- `internal/framework/runner/runner.go`

Narrow implementation approach:
1. Split current `collectSectionProposals` into pipeline stages that emit per-section proposal results plus deterministic section ordering metadata.
2. Introduce per-section validator execution helper that preserves existing validator semantics but operates on section-local candidate subsets where safe.
3. Preserve current final aggregation shape (`ModuleResult`, applied/skipped/rejected records) and apply-once-per-module behavior.
4. Preserve deterministic proposal index assignment by precomputing stable per-section index ranges or equivalent deterministic indexing strategy.
5. Keep scheduler interfaces unchanged; continue using composed proposal/validation schedulers from CLI wiring.

Tests to add/update (minimum):
- Runner test proving proposal and validation LLM calls overlap in time while total cap is respected.
- Runner test proving deterministic final transcript/proposal indexes/report ordering under out-of-order section completion.
- Runner/CLI test proving mixed proposal+validation jobs still honor global + subcap limits.
- Optional scheduler integration test proving mixed proposal/validation FIFO submission behavior at runtime boundary.

## Notes

- No runtime behavior was changed in this audit.
- Existing docs contain prior-audit historical notes (`docs/llm-concurrency-audit.md`) that may now be stale relative to current repository state; this file focuses only on current intra-module pipeline behavior.