diff --git a/docs/intra-module-pipeline-audit.md b/docs/intra-module-pipeline-audit.md new file mode 100644 index 0000000..3f09868 --- /dev/null +++ b/docs/intra-module-pipeline-audit.md @@ -0,0 +1,102 @@ +# Intra-Module LLM Pipeline Audit + +Date: 2026-05-12 + +## Summary + +This audit checks whether Audita currently maximizes available LLM concurrency within each module by overlapping proposal and validation work. + +Current state: +- Modules are serial and deterministic. +- Section proposal generation is concurrent and scheduler-limited. +- Validation (deterministic + LLM-backed) starts only after all section proposals complete. +- Proposal and validation LLM work therefore does **not** overlap in time within a module today. + +Conclusion: +- Current behavior is correct and deterministic, but it does **not** fully match the target pipelined intra-module behavior. +- A narrow runner refactor is needed to pipeline section-level validation after section proposal completion while preserving apply-once-per-module semantics. + +## Current Execution Flow (Per Module) + +Implementation anchor: `internal/framework/runner/runner.go`. + +1. `chunkWorkingTranscript` is called for the module. +2. `collectSectionProposals(...)` runs section `Module.Propose(...)` calls concurrently. +3. Runner waits for **all** proposal goroutines to finish (`wg.Wait()`). +4. Proposals are flattened deterministically by section order and assigned proposal indexes. +5. Validator chain runs over the full module proposal set (`eligible := enriched`; loop over `module.Validators()`). +6. Approved proposals are applied once via `proposals.ApplyProposals(...)`. + +## Audit Answers + +1. **At module start, are all section proposal jobs launched/queued promptly, or in smaller batches?** +- Jobs are launched promptly (goroutine per section), but section entry is throttled by worker semaphore in `collectSectionProposals`. + +2. **Does worker fan-out submit all sections and let scheduler enforce concurrency, or does fan-out itself block scheduler entry?** +- Fan-out itself blocks scheduler entry: runner’s `sem` gate limits how many section jobs can even call `Module.Propose` (and thus reach scheduler). + +3. **When a section proposal returns, do deterministic validators run immediately for that section?** +- No. Deterministic validators run only after all section proposals finish and aggregation completes. + +4. **When deterministic validators pass, are LLM validator jobs submitted immediately?** +- No. LLM validator calls occur only during module-level validator pass after full proposal collection. + +5. **Does validator chain architecture allow section-level validation independently?** +- Not as currently orchestrated by runner; validators receive module-wide `CandidateProposal` slices. + +6. **Does LLM validator batching depend on all module proposals being present?** +- Current invocation pattern does. `LLMBackedValidator.Validate` batches over the full provided candidate set; runner currently supplies full-module candidates. + +7. **Would immediate per-section validator submission reduce batching efficiency or change semantics?** +- Likely yes for efficiency: smaller per-section batches can increase LLM calls. +- Semantics can remain equivalent if ordering/cardinality/report mapping is preserved, but batching shape and diagnostic timing will differ. + +8. **When are proposal indexes assigned?** +- After concurrent proposal responses return, during deterministic aggregation in `collectSectionProposals`. + +9. **Can immediate validator submission preserve deterministic proposal indexes/report order?** +- Yes, if indexes are preallocated/stable by section-order offsets (or equivalent deterministic mapping) before emitting validator work. + +10. **Are diagnostics paths stable/deterministic enough if validator calls interleave with proposal calls?** +- Mostly yes: stage names are deterministic (`module:proposal:section-*`, `module:validator:batch-*`). +- But batch indices and artifact emission timing could change unless explicitly stabilized by per-section deterministic indexing. + +11. **Do schedulers enforce total/proposal/validation limits correctly when proposal and validation overlap?** +- Scheduler composition supports this (`global` + proposal/validation subcaps), but runner currently does not create overlap, so end-to-end overlap behavior is not exercised by runtime path. + +12. **Do existing tests prove proposal and validation jobs can overlap while respecting total concurrency?** +- No direct end-to-end runner test proving overlap of proposal and validation jobs in one module. + +13. **Are there tests proving FIFO queue contains both proposal and validation jobs in submission order?** +- No cross-type FIFO test. FIFO is tested at scheduler unit level (`internal/framework/llm/scheduler_test.go`), and composed-cap tests exist in CLI tests, but not mixed proposal+validation submission order in runner. + +14. **Smallest safe implementation change to reach target behavior?** +- Refactor runner module execution into a two-lane pipeline: + - keep current concurrent section proposal launch, + - on each section completion, run deterministic validators for that section’s proposals, + - immediately enqueue section LLM-validator work for survivors, + - collect all approved proposals in deterministic global proposal-index order, + - apply once per module exactly as today. + +## Minimal Change Plan (No Redesign) + +Primary change area: +- `internal/framework/runner/runner.go` + +Narrow implementation approach: +1. Split current `collectSectionProposals` into pipeline stages that emit per-section proposal results plus deterministic section ordering metadata. +2. Introduce per-section validator execution helper that preserves existing validator semantics but operates on section-local candidate subsets where safe. +3. Preserve current final aggregation shape (`ModuleResult`, applied/skipped/rejected records) and apply-once-per-module behavior. +4. Preserve deterministic proposal index assignment by precomputing stable per-section index ranges or equivalent deterministic indexing strategy. +5. Keep scheduler interfaces unchanged; continue using composed proposal/validation schedulers from CLI wiring. + +Tests to add/update (minimum): +- Runner test proving proposal and validation LLM calls overlap in time while total cap is respected. +- Runner test proving deterministic final transcript/proposal indexes/report ordering under out-of-order section completion. +- Runner/CLI test proving mixed proposal+validation jobs still honor global + subcap limits. +- Optional scheduler integration test proving mixed proposal/validation FIFO submission behavior at runtime boundary. + +## Notes + +- No runtime behavior was changed in this audit. +- Existing docs contain prior-audit historical notes (`docs/llm-concurrency-audit.md`) that may now be stale relative to current repository state; this file focuses only on current intra-module pipeline behavior.