Record assembled workflow benchmark results
This commit is contained in:
@@ -3,7 +3,7 @@
|
|||||||
Implement the remaining local work in the [performance roadmap](performance.md)
|
Implement the remaining local work in the [performance roadmap](performance.md)
|
||||||
in order. Each sequential stage is scoped for one Luna coding-agent prompt.
|
in order. Each sequential stage is scoped for one Luna coding-agent prompt.
|
||||||
Update status and evidence as each work unit lands.
|
Update status and evidence as each work unit lands.
|
||||||
Stages 1–10 are complete; Stages 11–14 remain pending. The completed
|
Stages 1–11 are complete; Stages 12–14 remain pending. The completed
|
||||||
inspection plan is preserved at
|
inspection plan is preserved at
|
||||||
`481e2d4:docs/roadmap/implementation.md` in Git history.
|
`481e2d4:docs/roadmap/implementation.md` in Git history.
|
||||||
|
|
||||||
@@ -421,6 +421,11 @@ prompt metadata remain identical for the same captured context.
|
|||||||
|
|
||||||
## Stage 11 — Establish Assembled Workflow Benchmarks
|
## Stage 11 — Establish Assembled Workflow Benchmarks
|
||||||
|
|
||||||
|
**Status: Complete (2026-10-09).** The benchmark-only baseline is committed at
|
||||||
|
`743e78d`; it covers public preparation, prepared-execution creation, `Run`,
|
||||||
|
and repeated `Details()` with small, large-message, and mutable-payload
|
||||||
|
fixtures. Fixture checks and the repeated measurements are recorded below.
|
||||||
|
|
||||||
**Outcome:** Reproducible end-to-end local benchmarks, including the baseline
|
**Outcome:** Reproducible end-to-end local benchmarks, including the baseline
|
||||||
for conditional snapshot optimization. No production change.
|
for conditional snapshot optimization. No production change.
|
||||||
|
|
||||||
@@ -713,6 +718,73 @@ GOTOOLCHAIN=local GOWORK=off GOPROXY=off GONOPROXY=none GOSUMDB=off \
|
|||||||
Verification passed on `c5f733a`:
|
Verification passed on `c5f733a`:
|
||||||
`go test -race -mod=readonly ./internal/configuration ./internal/usecase .`.
|
`go test -race -mod=readonly ./internal/configuration ./internal/usecase .`.
|
||||||
|
|
||||||
|
### Assembled Workflow Benchmark Baseline
|
||||||
|
|
||||||
|
The benchmark-only fixture is committed at `743e78d` (`Add assembled workflow
|
||||||
|
benchmark baseline`). The same committed source produced the measurements
|
||||||
|
below. Environment: Linux/amd64, Intel Core i5-7260U @ 2.20 GHz, benchmark
|
||||||
|
parallelism 4, Go 1.26.8, ConfigKit `v0.5.0`, JSON Schema `v6.0.2`, and backend
|
||||||
|
modules `v3.20260930.0`.
|
||||||
|
|
||||||
|
Command (five 500 ms samples per case, without the race detector):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GONOPROXY=none GOSUMDB=off \
|
||||||
|
go test -mod=readonly -run '^$' \
|
||||||
|
-bench '^(BenchmarkEnginePreparation|BenchmarkEngineRun|BenchmarkPreparedExecutionDetails)$' \
|
||||||
|
-benchmem -benchtime=500ms -count=5 .
|
||||||
|
```
|
||||||
|
|
||||||
|
All cases use one assembled engine constructed before timing and a deterministic
|
||||||
|
injected client; there is no live HTTP or credential lookup. The small prompt
|
||||||
|
renders `Hello Ada`. The large-message prompt has a 256 KiB literal prefix and
|
||||||
|
renders that prefix plus a short interpolated suffix. The mutable-payload
|
||||||
|
prompt uses 64 nested provider-option entries, each with labels, limits, and
|
||||||
|
route maps/slices, plus a local JSON Schema with 256 optional string
|
||||||
|
properties and `additionalProperties: false`. Preflight confirms that the
|
||||||
|
injected `{}` response passes the real schema validator; the `Run` benchmark
|
||||||
|
also measures that validation. The schema, configuration, and request fixtures
|
||||||
|
are created before each timer starts.
|
||||||
|
|
||||||
|
`BenchmarkEnginePreparation` measures ordinary `Prepare` separately from
|
||||||
|
`PrepareExecution` creation. The prepared-execution case discards each unused
|
||||||
|
handle inside its timed iteration and retains only the most recent discarded
|
||||||
|
handle pointer. `BenchmarkPreparedExecutionDetails` creates and validates one
|
||||||
|
ready handle outside timing, measures repeated public `Details()` calls on it,
|
||||||
|
retains only the latest caller-owned result, then discards the handle. Ordinary
|
||||||
|
prepare and run sinks are overwritten per iteration and cleared after timing.
|
||||||
|
The large-message result shares immutable string content, so its allocation
|
||||||
|
figures are not treated as deep-copy bytes. No benchmark retains every
|
||||||
|
iteration or includes engine construction/provider latency.
|
||||||
|
|
||||||
|
Times list the five `ns/op` samples in command order, followed by median and
|
||||||
|
range. Allocation figures are `B/op; allocs/op` (ranges where they varied).
|
||||||
|
|
||||||
|
| Benchmark case | `ns/op` samples (median; range) | Allocations |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Prepare, small | 10935, 10121, 13018, 10288, 8485 (10288; 8485–13018) | 4513; 50 |
|
||||||
|
| Prepare, large message | 2064223, 1627980, 1588515, 1008441, 911123 (1588515; 911123–2064223) | 872964–872988; 50 |
|
||||||
|
| Prepare, mutable payload | 6430306, 6519806, 5970344, 5854621, 6456738 (6430306; 5854621–6519806) | 2914750–2914824; 41134–41135 |
|
||||||
|
| Prepared execution creation + discard, small | 8194, 8016, 8095, 8073, 7939 (8073; 7939–8194) | 5210; 56 |
|
||||||
|
| Prepared execution creation + discard, large message | 937857, 955672, 919810, 1058503, 1094114 (955672; 919810–1094114) | 873679–873681; 56 |
|
||||||
|
| Prepared execution creation + discard, mutable payload | 8114785, 8394320, 10387083, 8418604, 8528766 (8418604; 8114785–10387083) | 3711282–3711329; 60727 |
|
||||||
|
| Run, small | 12157, 11921, 12128, 13762, 12491 (12157; 11921–13762) | 6091; 66 |
|
||||||
|
| Run, large message | 1082064, 1058363, 1040185, 996377, 951818 (1040185; 951818–1082064) | 874562–874577; 66–67 |
|
||||||
|
| Run, mutable payload | 6072294, 6338179, 6386632, 7167500, 6805715 (6386632; 6072294–7167500) | 2993599–2993661; 41808–41809 |
|
||||||
|
| Details, small | 1104, 1039, 1010, 1045, 1058 (1045; 1010–1104) | 1056; 6 |
|
||||||
|
| Details, large message | 975.1, 2480, 4734, 3402, 1008 (2480; 975.1–4734) | 1056; 6 |
|
||||||
|
| Details, mutable payload | 2275096, 3281027, 1504724, 1365349, 1329800 (1504724; 1329800–3281027) | 670315–670339; 11548 |
|
||||||
|
|
||||||
|
The 256 KiB message does not make `Details()` allocate 256 KiB per call;
|
||||||
|
current public conversion shares immutable message strings while copying the
|
||||||
|
message slice and other metadata. By contrast, `Details()` over the mutable
|
||||||
|
payload costs about 670 KB and 11.5 thousand allocations per call, providing a
|
||||||
|
specific snapshot-copy baseline for later ownership analysis. Large-message
|
||||||
|
timings, especially `Details()`, are noisy and are retained as ranges rather
|
||||||
|
than point claims. Focused verification passed: `go test -mod=readonly .`;
|
||||||
|
representative rendered messages, prepared details, generated output, and
|
||||||
|
structured validation were checked outside benchmark timing.
|
||||||
|
|
||||||
### Validation And Compiled-Rendering Baseline
|
### Validation And Compiled-Rendering Baseline
|
||||||
|
|
||||||
The benchmark-only baseline is commit
|
The benchmark-only baseline is commit
|
||||||
|
|||||||
Reference in New Issue
Block a user