eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Audit intra-module LLM pipeline

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-12 18:24:15 -05:00
parent a85a7e204e
commit 1afd753fad
1 changed files with 102 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

102
docs/intra-module-pipeline-audit.md Normal file
View File

@@ -0,0 +1,102 @@
# Intra-Module LLM Pipeline Audit
Date: 2026-05-12
## Summary
This audit checks whether Audita currently maximizes available LLM concurrency within each module by overlapping proposal and validation work.
Current state:
- Modules are serial and deterministic.
- Section proposal generation is concurrent and scheduler-limited.
- Validation (deterministic + LLM-backed) starts only after all section proposals complete.
- Proposal and validation LLM work therefore does **not** overlap in time within a module today.
Conclusion:
- Current behavior is correct and deterministic, but it does **not** fully match the target pipelined intra-module behavior.
- A narrow runner refactor is needed to pipeline section-level validation after section proposal completion while preserving apply-once-per-module semantics.
## Current Execution Flow (Per Module)
Implementation anchor: `internal/framework/runner/runner.go`.
1. `chunkWorkingTranscript` is called for the module.
2. `collectSectionProposals(...)` runs section `Module.Propose(...)` calls concurrently.
3. Runner waits for **all** proposal goroutines to finish (`wg.Wait()`).
4. Proposals are flattened deterministically by section order and assigned proposal indexes.
5. Validator chain runs over the full module proposal set (`eligible := enriched`; loop over `module.Validators()`).
6. Approved proposals are applied once via `proposals.ApplyProposals(...)`.
## Audit Answers
1. **At module start, are all section proposal jobs launched/queued promptly, or in smaller batches?**
- Jobs are launched promptly (goroutine per section), but section entry is throttled by worker semaphore in `collectSectionProposals`.
2. **Does worker fan-out submit all sections and let scheduler enforce concurrency, or does fan-out itself block scheduler entry?**
- Fan-out itself blocks scheduler entry: runner’s `sem` gate limits how many section jobs can even call `Module.Propose` (and thus reach scheduler).
3. **When a section proposal returns, do deterministic validators run immediately for that section?**
- No. Deterministic validators run only after all section proposals finish and aggregation completes.
4. **When deterministic validators pass, are LLM validator jobs submitted immediately?**
- No. LLM validator calls occur only during module-level validator pass after full proposal collection.
5. **Does validator chain architecture allow section-level validation independently?**
- Not as currently orchestrated by runner; validators receive module-wide `CandidateProposal` slices.
6. **Does LLM validator batching depend on all module proposals being present?**
- Current invocation pattern does. `LLMBackedValidator.Validate` batches over the full provided candidate set; runner currently supplies full-module candidates.
7. **Would immediate per-section validator submission reduce batching efficiency or change semantics?**
- Likely yes for efficiency: smaller per-section batches can increase LLM calls.
- Semantics can remain equivalent if ordering/cardinality/report mapping is preserved, but batching shape and diagnostic timing will differ.
8. **When are proposal indexes assigned?**
- After concurrent proposal responses return, during deterministic aggregation in `collectSectionProposals`.
9. **Can immediate validator submission preserve deterministic proposal indexes/report order?**
- Yes, if indexes are preallocated/stable by section-order offsets (or equivalent deterministic mapping) before emitting validator work.
10. **Are diagnostics paths stable/deterministic enough if validator calls interleave with proposal calls?**
- Mostly yes: stage names are deterministic (`module:proposal:section-*`, `module:validator:batch-*`).
- But batch indices and artifact emission timing could change unless explicitly stabilized by per-section deterministic indexing.
11. **Do schedulers enforce total/proposal/validation limits correctly when proposal and validation overlap?**
- Scheduler composition supports this (`global` + proposal/validation subcaps), but runner currently does not create overlap, so end-to-end overlap behavior is not exercised by runtime path.
12. **Do existing tests prove proposal and validation jobs can overlap while respecting total concurrency?**
- No direct end-to-end runner test proving overlap of proposal and validation jobs in one module.
13. **Are there tests proving FIFO queue contains both proposal and validation jobs in submission order?**
- No cross-type FIFO test. FIFO is tested at scheduler unit level (`internal/framework/llm/scheduler_test.go`), and composed-cap tests exist in CLI tests, but not mixed proposal+validation submission order in runner.
14. **Smallest safe implementation change to reach target behavior?**
- Refactor runner module execution into a two-lane pipeline:
- keep current concurrent section proposal launch,
- on each section completion, run deterministic validators for that section’s proposals,
- immediately enqueue section LLM-validator work for survivors,
- collect all approved proposals in deterministic global proposal-index order,
- apply once per module exactly as today.
## Minimal Change Plan (No Redesign)
Primary change area:
- `internal/framework/runner/runner.go`
Narrow implementation approach:
1. Split current `collectSectionProposals` into pipeline stages that emit per-section proposal results plus deterministic section ordering metadata.
2. Introduce per-section validator execution helper that preserves existing validator semantics but operates on section-local candidate subsets where safe.
3. Preserve current final aggregation shape (`ModuleResult`, applied/skipped/rejected records) and apply-once-per-module behavior.
4. Preserve deterministic proposal index assignment by precomputing stable per-section index ranges or equivalent deterministic indexing strategy.
5. Keep scheduler interfaces unchanged; continue using composed proposal/validation schedulers from CLI wiring.
Tests to add/update (minimum):
- Runner test proving proposal and validation LLM calls overlap in time while total cap is respected.
- Runner test proving deterministic final transcript/proposal indexes/report ordering under out-of-order section completion.
- Runner/CLI test proving mixed proposal+validation jobs still honor global + subcap limits.
- Optional scheduler integration test proving mixed proposal/validation FIFO submission behavior at runtime boundary.
## Notes
- No runtime behavior was changed in this audit.
- Existing docs contain prior-audit historical notes (`docs/llm-concurrency-audit.md`) that may now be stale relative to current repository state; this file focuses only on current intra-module pipeline behavior.