112 lines
7.1 KiB
Markdown
112 lines
7.1 KiB
Markdown
# Intra-Module LLM Pipeline Audit
|
||
|
||
Date: 2026-05-12
|
||
|
||
## Implementation Status (2026-05-12 Update)
|
||
|
||
The intra-module pipelining gap identified in this audit has now been addressed:
|
||
- section proposal jobs are launched promptly for each module;
|
||
- as section proposals become available in deterministic section order, section-local validator work starts without waiting for all section proposals to finish;
|
||
- deterministic validators run before LLM-backed validators for each section;
|
||
- proposal and validation LLM calls can overlap through the existing composed scheduler path;
|
||
- module application remains a single deterministic apply barrier.
|
||
|
||
## Summary
|
||
|
||
This audit checks whether Audita currently maximizes available LLM concurrency within each module by overlapping proposal and validation work.
|
||
|
||
Current state:
|
||
- Modules are serial and deterministic.
|
||
- Section proposal generation is concurrent and scheduler-limited.
|
||
- Validation (deterministic + LLM-backed) starts only after all section proposals complete.
|
||
- Proposal and validation LLM work therefore does **not** overlap in time within a module today.
|
||
|
||
Conclusion:
|
||
- Current behavior is correct and deterministic, but it does **not** fully match the target pipelined intra-module behavior.
|
||
- A narrow runner refactor is needed to pipeline section-level validation after section proposal completion while preserving apply-once-per-module semantics.
|
||
|
||
## Current Execution Flow (Per Module)
|
||
|
||
Implementation anchor: `internal/framework/runner/runner.go`.
|
||
|
||
1. `chunkWorkingTranscript` is called for the module.
|
||
2. `collectSectionProposals(...)` runs section `Module.Propose(...)` calls concurrently.
|
||
3. Runner waits for **all** proposal goroutines to finish (`wg.Wait()`).
|
||
4. Proposals are flattened deterministically by section order and assigned proposal indexes.
|
||
5. Validator chain runs over the full module proposal set (`eligible := enriched`; loop over `module.Validators()`).
|
||
6. Approved proposals are applied once via `proposals.ApplyProposals(...)`.
|
||
|
||
## Audit Answers
|
||
|
||
1. **At module start, are all section proposal jobs launched/queued promptly, or in smaller batches?**
|
||
- Jobs are launched promptly (goroutine per section), but section entry is throttled by worker semaphore in `collectSectionProposals`.
|
||
|
||
2. **Does worker fan-out submit all sections and let scheduler enforce concurrency, or does fan-out itself block scheduler entry?**
|
||
- Fan-out itself blocks scheduler entry: runner’s `sem` gate limits how many section jobs can even call `Module.Propose` (and thus reach scheduler).
|
||
|
||
3. **When a section proposal returns, do deterministic validators run immediately for that section?**
|
||
- No. Deterministic validators run only after all section proposals finish and aggregation completes.
|
||
|
||
4. **When deterministic validators pass, are LLM validator jobs submitted immediately?**
|
||
- No. LLM validator calls occur only during module-level validator pass after full proposal collection.
|
||
|
||
5. **Does validator chain architecture allow section-level validation independently?**
|
||
- Not as currently orchestrated by runner; validators receive module-wide `CandidateProposal` slices.
|
||
|
||
6. **Does LLM validator batching depend on all module proposals being present?**
|
||
- Current invocation pattern does. `LLMBackedValidator.Validate` batches over the full provided candidate set; runner currently supplies full-module candidates.
|
||
|
||
7. **Would immediate per-section validator submission reduce batching efficiency or change semantics?**
|
||
- Likely yes for efficiency: smaller per-section batches can increase LLM calls.
|
||
- Semantics can remain equivalent if ordering/cardinality/report mapping is preserved, but batching shape and diagnostic timing will differ.
|
||
|
||
8. **When are proposal indexes assigned?**
|
||
- After concurrent proposal responses return, during deterministic aggregation in `collectSectionProposals`.
|
||
|
||
9. **Can immediate validator submission preserve deterministic proposal indexes/report order?**
|
||
- Yes, if indexes are preallocated/stable by section-order offsets (or equivalent deterministic mapping) before emitting validator work.
|
||
|
||
10. **Are diagnostics paths stable/deterministic enough if validator calls interleave with proposal calls?**
|
||
- Mostly yes: stage names are deterministic (`module:proposal:section-*`, `module:validator:batch-*`).
|
||
- But batch indices and artifact emission timing could change unless explicitly stabilized by per-section deterministic indexing.
|
||
|
||
11. **Do schedulers enforce total/proposal/validation limits correctly when proposal and validation overlap?**
|
||
- Scheduler composition supports this (`global` + proposal/validation subcaps), but runner currently does not create overlap, so end-to-end overlap behavior is not exercised by runtime path.
|
||
|
||
12. **Do existing tests prove proposal and validation jobs can overlap while respecting total concurrency?**
|
||
- No direct end-to-end runner test proving overlap of proposal and validation jobs in one module.
|
||
|
||
13. **Are there tests proving FIFO queue contains both proposal and validation jobs in submission order?**
|
||
- No cross-type FIFO test. FIFO is tested at scheduler unit level (`internal/framework/llm/scheduler_test.go`), and composed-cap tests exist in CLI tests, but not mixed proposal+validation submission order in runner.
|
||
|
||
14. **Smallest safe implementation change to reach target behavior?**
|
||
- Refactor runner module execution into a two-lane pipeline:
|
||
- keep current concurrent section proposal launch,
|
||
- on each section completion, run deterministic validators for that section’s proposals,
|
||
- immediately enqueue section LLM-validator work for survivors,
|
||
- collect all approved proposals in deterministic global proposal-index order,
|
||
- apply once per module exactly as today.
|
||
|
||
## Minimal Change Plan (No Redesign)
|
||
|
||
Primary change area:
|
||
- `internal/framework/runner/runner.go`
|
||
|
||
Narrow implementation approach:
|
||
1. Split current `collectSectionProposals` into pipeline stages that emit per-section proposal results plus deterministic section ordering metadata.
|
||
2. Introduce per-section validator execution helper that preserves existing validator semantics but operates on section-local candidate subsets where safe.
|
||
3. Preserve current final aggregation shape (`ModuleResult`, applied/skipped/rejected records) and apply-once-per-module behavior.
|
||
4. Preserve deterministic proposal index assignment by precomputing stable per-section index ranges or equivalent deterministic indexing strategy.
|
||
5. Keep scheduler interfaces unchanged; continue using composed proposal/validation schedulers from CLI wiring.
|
||
|
||
Tests to add/update (minimum):
|
||
- Runner test proving proposal and validation LLM calls overlap in time while total cap is respected.
|
||
- Runner test proving deterministic final transcript/proposal indexes/report ordering under out-of-order section completion.
|
||
- Runner/CLI test proving mixed proposal+validation jobs still honor global + subcap limits.
|
||
- Optional scheduler integration test proving mixed proposal/validation FIFO submission behavior at runtime boundary.
|
||
|
||
## Notes
|
||
|
||
- No runtime behavior was changed in this audit.
|
||
- Existing docs contain prior-audit historical notes (`docs/llm-concurrency-audit.md`) that may now be stale relative to current repository state; this file focuses only on current intra-module pipeline behavior.
|