eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Rewrite and simplify the section chunking algorithm

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-12 15:42:32 -05:00
parent 390daa8b84
commit df96f9fdf6
3 changed files with 530 additions and 657 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docs/architecture.md
View File

@@ -268,12 +268,14 @@ Chunking (`internal/core/chunking`) currently provides:
- deterministic heuristic token estimation;
- contiguous sectioning with section metadata;
- max/min section token validation;
- optional `target_sections` handling with target-aware merge/split logic;
- optional `target_sections` override for section-count planning;
- summary and detailed summary generation.
Current behavior details:
- if a single segment exceeds max tokens, it is emitted as its own section (not hard-failed);
- section balancing is deterministic but heuristic.
- default section count is planned from `ceil(total_tokens / max_section_tokens)`;
- section sizing targets `ceil(total_tokens / section_count)` with a deterministic forward pass;
- sections remain contiguous and ordered, and segments are never split.
## Implemented proposal/replacement infrastructure
`internal/framework/proposals` provides deterministic proposal composition logic: