Record assembled workflow benchmark results

This commit is contained in:
2026-10-09 00:50:41 +00:00
parent 743e78d514
commit f46d5a4591

View File

@@ -3,7 +3,7 @@
Implement the remaining local work in the [performance roadmap](performance.md)
in order. Each sequential stage is scoped for one Luna coding-agent prompt.
Update status and evidence as each work unit lands.
Stages 1–10 are complete; Stages 11–14 remain pending. The completed
Stages 1–11 are complete; Stages 12–14 remain pending. The completed
inspection plan is preserved at
`481e2d4:docs/roadmap/implementation.md` in Git history.
@@ -421,6 +421,11 @@ prompt metadata remain identical for the same captured context.
## Stage 11 — Establish Assembled Workflow Benchmarks
**Status: Complete (2026-10-09).** The benchmark-only baseline is committed at
`743e78d`; it covers public preparation, prepared-execution creation, `Run`,
and repeated `Details()` with small, large-message, and mutable-payload
fixtures. Fixture checks and the repeated measurements are recorded below.
**Outcome:** Reproducible end-to-end local benchmarks, including the baseline
for conditional snapshot optimization. No production change.
@@ -713,6 +718,73 @@ GOTOOLCHAIN=local GOWORK=off GOPROXY=off GONOPROXY=none GOSUMDB=off \
Verification passed on `c5f733a`:
`go test -race -mod=readonly ./internal/configuration ./internal/usecase .`.
### Assembled Workflow Benchmark Baseline
The benchmark-only fixture is committed at `743e78d` (`Add assembled workflow
benchmark baseline`). The same committed source produced the measurements
below. Environment: Linux/amd64, Intel Core i5-7260U @ 2.20 GHz, benchmark
parallelism 4, Go 1.26.8, ConfigKit `v0.5.0`, JSON Schema `v6.0.2`, and backend
modules `v3.20260930.0`.
Command (five 500 ms samples per case, without the race detector):
```sh
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GONOPROXY=none GOSUMDB=off \
go test -mod=readonly -run '^$' \
-bench '^(BenchmarkEnginePreparation|BenchmarkEngineRun|BenchmarkPreparedExecutionDetails)$' \
-benchmem -benchtime=500ms -count=5 .
```
All cases use one assembled engine constructed before timing and a deterministic
injected client; there is no live HTTP or credential lookup. The small prompt
renders `Hello Ada`. The large-message prompt has a 256 KiB literal prefix and
renders that prefix plus a short interpolated suffix. The mutable-payload
prompt uses 64 nested provider-option entries, each with labels, limits, and
route maps/slices, plus a local JSON Schema with 256 optional string
properties and `additionalProperties: false`. Preflight confirms that the
injected `{}` response passes the real schema validator; the `Run` benchmark
also measures that validation. The schema, configuration, and request fixtures
are created before each timer starts.
`BenchmarkEnginePreparation` measures ordinary `Prepare` separately from
`PrepareExecution` creation. The prepared-execution case discards each unused
handle inside its timed iteration and retains only the most recent discarded
handle pointer. `BenchmarkPreparedExecutionDetails` creates and validates one
ready handle outside timing, measures repeated public `Details()` calls on it,
retains only the latest caller-owned result, then discards the handle. Ordinary
prepare and run sinks are overwritten per iteration and cleared after timing.
The large-message result shares immutable string content, so its allocation
figures are not treated as deep-copy bytes. No benchmark retains every
iteration or includes engine construction/provider latency.
Times list the five `ns/op` samples in command order, followed by median and
range. Allocation figures are `B/op; allocs/op` (ranges where they varied).
| Benchmark case | `ns/op` samples (median; range) | Allocations |
| --- | --- | --- |
| Prepare, small | 10935, 10121, 13018, 10288, 8485 (10288; 8485–13018) | 4513; 50 |
| Prepare, large message | 2064223, 1627980, 1588515, 1008441, 911123 (1588515; 911123–2064223) | 872964–872988; 50 |
| Prepare, mutable payload | 6430306, 6519806, 5970344, 5854621, 6456738 (6430306; 5854621–6519806) | 2914750–2914824; 41134–41135 |
| Prepared execution creation + discard, small | 8194, 8016, 8095, 8073, 7939 (8073; 7939–8194) | 5210; 56 |
| Prepared execution creation + discard, large message | 937857, 955672, 919810, 1058503, 1094114 (955672; 919810–1094114) | 873679–873681; 56 |
| Prepared execution creation + discard, mutable payload | 8114785, 8394320, 10387083, 8418604, 8528766 (8418604; 8114785–10387083) | 3711282–3711329; 60727 |
| Run, small | 12157, 11921, 12128, 13762, 12491 (12157; 11921–13762) | 6091; 66 |
| Run, large message | 1082064, 1058363, 1040185, 996377, 951818 (1040185; 951818–1082064) | 874562–874577; 66–67 |
| Run, mutable payload | 6072294, 6338179, 6386632, 7167500, 6805715 (6386632; 6072294–7167500) | 2993599–2993661; 41808–41809 |
| Details, small | 1104, 1039, 1010, 1045, 1058 (1045; 1010–1104) | 1056; 6 |
| Details, large message | 975.1, 2480, 4734, 3402, 1008 (2480; 975.1–4734) | 1056; 6 |
| Details, mutable payload | 2275096, 3281027, 1504724, 1365349, 1329800 (1504724; 1329800–3281027) | 670315–670339; 11548 |
The 256 KiB message does not make `Details()` allocate 256 KiB per call;
current public conversion shares immutable message strings while copying the
message slice and other metadata. By contrast, `Details()` over the mutable
payload costs about 670 KB and 11.5 thousand allocations per call, providing a
specific snapshot-copy baseline for later ownership analysis. Large-message
timings, especially `Details()`, are noisy and are retained as ranges rather
than point claims. Focused verification passed: `go test -mod=readonly .`;
representative rendered messages, prepared details, generated output, and
structured validation were checked outside benchmark timing.
### Validation And Compiled-Rendering Baseline
The benchmark-only baseline is commit