audita/docs/intra-module-pipeline-audit.md

Intra-Module LLM Pipeline Audit

Date: 2026-05-12

Implementation Status (2026-05-12 Update)

The intra-module pipelining gap identified in this audit has now been addressed:

• section proposal jobs are launched promptly for each module;
• as section proposals become available in deterministic section order, section-local validator work starts without waiting for all section proposals to finish;
• deterministic validators run before LLM-backed validators for each section;
• proposal and validation LLM calls can overlap through the existing composed scheduler path;
• module application remains a single deterministic apply barrier.

Summary

This audit checks whether Audita currently maximizes available LLM concurrency within each module by overlapping proposal and validation work.

Current state:

• Modules are serial and deterministic.
• Section proposal generation is concurrent and scheduler-limited.
• Validation (deterministic + LLM-backed) starts only after all section proposals complete.
• Proposal and validation LLM work therefore does not overlap in time within a module today.

Conclusion:

• Current behavior is correct and deterministic, but it does not fully match the target pipelined intra-module behavior.
• A narrow runner refactor is needed to pipeline section-level validation after section proposal completion while preserving apply-once-per-module semantics.

Current Execution Flow (Per Module)

Implementation anchor: internal/framework/runner/runner.go.

1. chunkWorkingTranscript is called for the module.
2. collectSectionProposals(...) runs section Module.Propose(...) calls concurrently.
3. Runner waits for all proposal goroutines to finish (wg.Wait()).
4. Proposals are flattened deterministically by section order and assigned proposal indexes.
5. Validator chain runs over the full module proposal set (eligible := enriched; loop over module.Validators()).
6. Approved proposals are applied once via proposals.ApplyProposals(...).

Audit Answers

1. At module start, are all section proposal jobs launched/queued promptly, or in smaller batches?

• Jobs are launched promptly (goroutine per section), but section entry is throttled by worker semaphore in collectSectionProposals.

2. Does worker fan-out submit all sections and let scheduler enforce concurrency, or does fan-out itself block scheduler entry?

• Fan-out itself blocks scheduler entry: runner’s sem gate limits how many section jobs can even call Module.Propose (and thus reach scheduler).

3. When a section proposal returns, do deterministic validators run immediately for that section?

• No. Deterministic validators run only after all section proposals finish and aggregation completes.

4. When deterministic validators pass, are LLM validator jobs submitted immediately?

• No. LLM validator calls occur only during module-level validator pass after full proposal collection.

5. Does validator chain architecture allow section-level validation independently?

• Not as currently orchestrated by runner; validators receive module-wide CandidateProposal slices.

6. Does LLM validator batching depend on all module proposals being present?

• Current invocation pattern does. LLMBackedValidator.Validate batches over the full provided candidate set; runner currently supplies full-module candidates.

7. Would immediate per-section validator submission reduce batching efficiency or change semantics?

• Likely yes for efficiency: smaller per-section batches can increase LLM calls.
• Semantics can remain equivalent if ordering/cardinality/report mapping is preserved, but batching shape and diagnostic timing will differ.

8. When are proposal indexes assigned?

• After concurrent proposal responses return, during deterministic aggregation in collectSectionProposals.

9. Can immediate validator submission preserve deterministic proposal indexes/report order?

• Yes, if indexes are preallocated/stable by section-order offsets (or equivalent deterministic mapping) before emitting validator work.

10. Are diagnostics paths stable/deterministic enough if validator calls interleave with proposal calls?

• Mostly yes: stage names are deterministic (module:proposal:section-*, module:validator:batch-*).
• But batch indices and artifact emission timing could change unless explicitly stabilized by per-section deterministic indexing.

11. Do schedulers enforce total/proposal/validation limits correctly when proposal and validation overlap?

• Scheduler composition supports this (global + proposal/validation subcaps), but runner currently does not create overlap, so end-to-end overlap behavior is not exercised by runtime path.

12. Do existing tests prove proposal and validation jobs can overlap while respecting total concurrency?

• No direct end-to-end runner test proving overlap of proposal and validation jobs in one module.

13. Are there tests proving FIFO queue contains both proposal and validation jobs in submission order?

• No cross-type FIFO test. FIFO is tested at scheduler unit level (internal/framework/llm/scheduler_test.go), and composed-cap tests exist in CLI tests, but not mixed proposal+validation submission order in runner.

14. Smallest safe implementation change to reach target behavior?

• Refactor runner module execution into a two-lane pipeline:
  • keep current concurrent section proposal launch,
  • on each section completion, run deterministic validators for that section’s proposals,
  • immediately enqueue section LLM-validator work for survivors,
  • collect all approved proposals in deterministic global proposal-index order,
  • apply once per module exactly as today.

Minimal Change Plan (No Redesign)

Primary change area:

• internal/framework/runner/runner.go

Narrow implementation approach:

1. Split current collectSectionProposals into pipeline stages that emit per-section proposal results plus deterministic section ordering metadata.
2. Introduce per-section validator execution helper that preserves existing validator semantics but operates on section-local candidate subsets where safe.
3. Preserve current final aggregation shape (ModuleResult, applied/skipped/rejected records) and apply-once-per-module behavior.
4. Preserve deterministic proposal index assignment by precomputing stable per-section index ranges or equivalent deterministic indexing strategy.
5. Keep scheduler interfaces unchanged; continue using composed proposal/validation schedulers from CLI wiring.

Tests to add/update (minimum):

• Runner test proving proposal and validation LLM calls overlap in time while total cap is respected.
• Runner test proving deterministic final transcript/proposal indexes/report ordering under out-of-order section completion.
• Runner/CLI test proving mixed proposal+validation jobs still honor global + subcap limits.
• Optional scheduler integration test proving mixed proposal/validation FIFO submission behavior at runtime boundary.

Notes

• No runtime behavior was changed in this audit.
• Existing docs contain prior-audit historical notes (docs/llm-concurrency-audit.md) that may now be stale relative to current repository state; this file focuses only on current intra-module pipeline behavior.