base-layer/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Plan Go-based testing for maintained examples

Browse Source
This commit is contained in:
Eric Rakestraw
2026-11-03 01:00:45 +00:00
parent 995c309560
commit 79dfd97afa
1 changed files with 119 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

119
docs/roadmap/example-testing.md Normal file
View File

@@ -0,0 +1,119 @@
# Go-Based Testing For Maintained Examples
## Purpose
Move the maintained consumer-example checks into Promptkit's ordinary Go test
suite. Developers and ReleaseKit should exercise these workflows through the
same testing entry point, without a separate Python example runner or custom
release-validation command.
Keep the examples useful as readable, copyable consumer guidance. The objective
is simpler ownership and execution of tests, not a language-only translation or
a particular reduction in line count. This is a tooling change, not a new public
library API or a new release gate.
## Current Baseline
The [preparation](../../examples/go-library/prepare/main.go),
[execution](../../examples/go-library/run/main.go), and
[observation](../../examples/go-library/observe/main.go) examples are Go programs.
Their deterministic clients and transport keep them independent of live
providers. Preparation and execution use repository-relative configuration
paths; observation embeds its assets and already has a workflow helper returning
a summary and error.
[scripts/check-examples.py](../../scripts/check-examples.py) launches all three
programs, decodes their JSON summaries, and checks meaningful results. Ordinary
`go test ./...` compiles the example packages but currently has no example tests
to execute. The [ReleaseKit profile](../../releasekit.json) runs the Python
script as its one custom check after shared validation.
The separate [release-wiring test](../../scripts/test-release.py) is also Python,
but is a development-only diagnostic, not a release gate. Its disposable fixture
currently assumes the example script is a configured product check.
## Target State
Ordinary Go tests alongside the examples exercise the actual example workflows.
They are discovered by `go test ./...` and can be selected with
`go test ./examples/go-library/...`. ReleaseKit's existing standard suite runs
them with its selected race and vet settings; no second example-validation pass
is needed.
Preserve the useful coverage currently supplied by the Python runner:
- Preparation: collection composition and overrides, scoped selection,
inspection/preparation, and the resulting identities and inventories.
- Execution: injected-client execution, expected output and structural validity,
model selection, and usage reporting.
- Observation: bounded transient retries, attempt metadata, incomplete usage
reporting, and application-owned interpretation of proposal/assessment results.
Assertions should remain tied to the synthetic examples and meaningful results.
Their output formatting and fixture values do not become new public API
guarantees. Preserve relevant output coverage without freezing complete JSON
documents or introducing broad golden snapshots.
## Example And Test Structure
Prefer small, unexported workflow helpers that return errors or results and,
where useful, accept an output writer. Keep `main` responsible for process-level
output and exit handling. Reuse the observation example's existing separation
where it already meets the need; uniform helper signatures are not an objective.
Tests should normally call the workflow directly, using local output buffers
and the existing deterministic collaborators. Handle resource paths explicitly
so package-directory test execution and the documented repository-root
`go run` commands both work. A small resource-base or filesystem input is
preferable to changing process-wide working directories or redirecting global
standard output merely to reach the workflow.
Use a Go subprocess test only where launching the executable protects a distinct
risk or avoids making an example materially harder to understand. Do not add
one for every example automatically. Subprocesses need their own consideration
of toolchain, offline execution, and instrumentation; parent race or coverage
settings do not automatically instrument a separately launched program.
Keep tests deterministic, offline, and independent of real credentials under the
[testing policy](../policy/testing.md). Avoid a new public package, shared
example framework, or expanded failure-case inventory solely for this migration.
Examples remain consumer programs rather than release artifacts or library
implementation dependencies, consistent with the
[architecture policy](../policy/architecture.md).
## Integration And Documentation Work
Once the Go tests cover the retained behavior, remove `scripts/check-examples.py`
and its custom command and environment overlay from `releasekit.json`. Keep the
existing built-in validation selections and source/hosted publication workflow.
There is no need for new ReleaseKit functionality or a dependency upgrade.
Adapt the development-only release-wiring fixture so it no longer expects the
removed script or product-check marker. Retain checks relevant to the actual
launcher and profile; do not manufacture an unused custom hook merely to retain
old fixture assertions. The Python release-wiring diagnostic can remain Python.
Removing Python from every repository tool is outside this scope.
Update the [development guide](../development.md),
[testing policy](../policy/testing.md), [release policy](../policy/release.md),
and [release procedure](../release.md) when implementation lands. Replace the
standalone example command with Go test guidance, describe example coverage as
part of the standard suite, and remove Python as a normal release prerequisite.
Continue documenting Python where the optional release-wiring test needs it.
Check other incoming references to the removed runner and retain the existing
consumer quickstart commands.
## Verification
Verify the example packages through ordinary Go tests and the selected
race-enabled suite and vet. Confirm the documented `go run` entry points still
work from the repository root, and exercise the adapted release-wiring test in
disposable repositories. These are migration checks, not new recurring release
steps. Check that a meaningful example assertion failure fails the Go suite
rather than relying only on successful compilation.
Completion means useful example behavior remains covered, normal test execution
includes that coverage, and the obsolete runner, profile hook, and documentation
references are removed. No performance thresholds, coverage percentage targets,
new platform commitments, hosted CI requirements, or changes to Promptkit's
consumer contracts are introduced by this work.