promptkit/docs/policy/testing.md

# Testing Policy

**Base Policy Version:** 1.3

**Last Updated**: 2026-09-27

**Upstream Source:** [testing.md](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/testing.md)

## Purpose

Our tests exist to make **incorrect changes expensive and correct changes cheap**.

We optimize for sufficient confidence in important behavior while minimizing friction on future development.

## Every test has a cost

Every test has an immediate and continuing cost: writing, review, execution,
diagnosis, fixture maintenance, and updates when legitimate behavior changes.
Tests can also constrain refactoring, duplicate policy, and add failure noise.

A test is warranted only when the confidence it provides justifies these costs.
Evaluate that tradeoff at two levels:

1. **Per test:** What realistic defect does this test detect, how consequential would that defect be, and is that protection worth the test's lifetime cost?
2. **Across the suite:** Does this collection provide materially more confidence than a smaller, simpler suite would?

Some friction is intentional. Tests should make dangerous changes—such as breaking compatibility, corrupting data, violating security boundaries, or reintroducing subtle bugs—require deliberate review. Keep ordinary internal changes inexpensive.

Evaluate maintenance cost against the specific failure being protected. When omitting a plausible test, be able to state why the protected failure is low-risk, already covered, obvious, reversible, or cheaper to detect elsewhere. For consequential, subtle, or difficult-to-observe behavior, the presumption should favor testing.

## Default testing style

Use a **classical/Detroit-style** approach:

- Test observable behavior, resulting state, contracts, and invariants.
- Use real internal collaborators when they are fast and deterministic.
- Use fakes, stubs, or mocks primarily at expensive, nondeterministic, destructive, or external boundaries.
- Prefer package-level behavioral tests over tests coupled to private helpers or internal call sequences.
- Treat exact collaborator interactions as testable behavior only when the interaction itself is a requirement.

Examples of appropriate seams include clocks, randomness, subprocesses, remote APIs, object storage, email, and paid LLM calls.

## Promptkit Test Boundaries

Promptkit is a library without a CLI or hosted CI. The execution requirements,
facade test ownership, public error guidance, and offline consumer examples
below tailor the standard to that boundary.

## Test execution requirements

Promptkit currently uses maintainer-run validation rather than hosted CI.
Ordinary work requires validation proportionate to its scope and risk. Do not
repeat successful checks for unchanged content merely to create a commit.
Release candidates require the complete local workflow in the
[development guide](../development.md#maintainer-validation). That guide routes
contributors to the executable checks for formatting, documentation links,
and repository hygiene.
Introducing hosted CI later would supplement, not silently redefine, this
documented validation model.

Full release-candidate validation includes race-enabled package tests, explicit
static analysis, and execution with automated semantic checks of the three maintained
offline consumer examples. The test run builds the packages; the examples
also exercise ordinary compilation. Separate ordinary tests and a standalone
build are not additional release gates. Ordinary tests remain useful during
development. The preparation example protects assembled preparation and
inspection behavior. The execution example separately protects assembled
`Run`, injected-client, validation, usage, and result behavior. The observation
example protects assembled built-in retries, structured proposal and assessment
runs, and application-owned interpretation of metadata.

Tests in the default suite must be deterministic, offline, and independent of
real credentials. They must not invoke paid APIs, use live network
dependencies, or depend on mutable external services. Tests that require live
infrastructure must be explicitly opt-in and clearly separated from the
default suite.

Control clocks, randomness, environment variables, and other process-global or
machine-specific state when they affect behavior. Tests must be parallel-safe:
they should run repeatedly and alongside other tests without depending on
execution order, shared mutable state, fixed ports, or state left by an earlier
test.

## Test types and assets

Use each test type where it protects a distinct risk:

- Unit and package tests protect focused behavior and invariants through the
  narrowest stable boundary.
- Contract tests protect exported behavior, compatibility, and error identity
  relied upon by consumers.
- Integration tests use real collaborators when correctness depends on their
  interaction, while replacing live or nondeterministic external boundaries.
- External-package root tests exercise the public facade as a Go consumer,
  while internal package tests own focused implementation behavior.
- The maintained offline preparation, execution, and observation examples protect distinct
  representative assembled consumer workflows without contacting a model
  provider.
- Fixtures should be minimal, synthetic, versioned with the behavior they
  exercise, and free of credentials or private data.
- Golden files are appropriate only when the complete output is intentionally
  stable and semantic review of updates is practical.
- Failure-path tests should cover consequential malformed input, dependency
  failure, cancellation, partial results, and recovery behavior.

## What deserves tests

To the extent the following features are present in the application, prioritize tests for:

1. Public and package-level contracts.
2. Domain rules and important invariants.
3. Boundary conditions and malformed input.
4. Failure handling, cancellation, retries, recovery, and partial success.
5. Serialization, schemas, compatibility, and round trips.
6. Previously observed or plausible regressions.
7. Representative integration and end-to-end workflows.

A package-level contract comprises behavior relied upon by another package or major collaborator.

Choose tests for requested behavior and established contracts. Tests must not
introduce new restrictions, guarantees, or threat scenarios merely because they
could be asserted. Follow the [agent policy](agents.md) when deciding whether a
behavior belongs to the contract; current implementation behavior alone is not
sufficient authority.

For established behavior involving **data integrity, destructive operations,
compatibility, security, concurrency, idempotency, or recovery**, favor durable
tests for realistic failures whose consequences warrant protection. Identify
the failure and the existing contract it would violate. These categories call
for attention, not an automatic test inventory or permission to strengthen the
application's guarantees. Account for protection already provided at other layers.

Do not add tests merely because a function, branch, or line exists. Do not add a test when the same meaningful risk is already adequately protected elsewhere.

## Choose the right test boundary

Choose the narrowest stable boundary that expresses the behavior clearly and
provides durable confidence with minimal incidental coupling.

This is often the package API, but it may instead be:

- a smaller pure function when dense domain logic is most clearly isolated there;
- a package-level operation when several internal collaborators jointly produce the behavior; or
- a larger integration boundary when correctness emerges from interaction with a real dependency.

## Test behavior, not implementation

A test should protect a selected decision, contract, or invariant.

A test is suspect when it primarily detects changes to private constants or
helpers, equivalent operation ordering, incidental formatting, equivalent
algorithms, or internal structure that preserves behavior.

Behavioral tests should normally remain stable through internal refactoring that preserves the contract.

## Expected effects of different changes

Use the following expectations when evaluating test failures and test maintenance:

| Change | Expected effect on tests |
|---|---|
| Internal refactor that preserves behavior | Existing tests should normally remain unchanged and continue to pass. |
| Change to an internal default with no contractual significance | Behavioral tests should normally remain unchanged; tests should derive expectations from configuration or relationships rather than duplicate the old value. |
| Intentional change to public behavior, policy, schema, or compatibility guarantees | The relevant tests should be reviewed and changed deliberately. |
| Accidental violation of a contract or invariant | Tests should fail; fix the production code rather than rewriting the tests to accept the defect. |

A test failing is not the same as a test needing to be edited. Many tests may correctly fail because of one production defect. The maintenance smell is a correct internal change that requires unrelated expectation updates throughout the suite.

## Separate mechanism from policy

Derive expectations from test-controlled configuration and relationships.

For this example, assume the selected contract rejects a request to start more
processes than a configured limit. Do not encode its production default indirectly:

```go
// Production default:
const maxConcurrency = 4

// Brittle test:
err := startProcesses(5)
require.Error(t, err)
```

Instead, test the mechanism relationally:

```go
const limit = 2
runner := NewRunner(limit)

require.NoError(t, runner.Start(limit))
require.ErrorIs(t, runner.Start(limit+1), ErrTooMuchConcurrency)
```

For this selected contract, the test should prove:

- the configured limit is accepted; and
- one beyond the configured limit is rejected.

Rejection is not a general requirement for concurrency control. If the selected
contract queues excess work or limits active workers while accepting larger
batches, test that behavior instead. Do not add rejection or a configurable
limit solely to reproduce this example.

The production default should be tested exactly only when its literal value is itself a public, operational, safety, protocol, or compatibility requirement.

Apply the same rule to limits, timeouts, capacities, retry counts, and ranges: test relationships and behavior.

For established concurrency limits, distinguish these behaviors where the contract requires them:

1. **Configuration handling:** values are accepted or rejected according to the selected contract.
2. **Runtime enforcement:** observed peak concurrency never exceeds the configured limit.

Use a test-controlled limit and measure the behavior relative to that limit.

## Assign each behavior a test owner

Where these layers exist, assign ownership as follows:

- Parser tests own parsing cases.
- Validator tests own validation rules.
- Domain tests own transformations and invariants.
- Adapter tests own external integration behavior.
- Orchestrator tests own coordination and failure propagation.
- External-package root tests own the public Go facade contract.
- Consumer-workflow tests prove that representative assembled library use works.

Give each layer its distinct responsibility and keep expectation updates localized to the contract being changed.

## Use test doubles deliberately

Choose the least elaborate test double that provides the required control or observation.

As a default:

1. Prefer real collaborators when they are fast and deterministic.
2. Use small in-memory fakes when realistic stateful behavior is helpful.
3. Use stubs when a dependency only needs to provide controlled responses.
4. Use mocks when the interaction itself is contractual.

Mocks are appropriate when the contract includes facts such as:

- a notification is sent exactly once;
- a transaction is committed only after successful writes;
- cancellation reaches a subprocess;
- an expensive API is called no more than once; or
- a security audit event is emitted.

## Go-specific guidance

Select techniques by behavior and risk:

- table-driven tests for meaningful behavioral categories and boundaries;
- `t.TempDir()` for real filesystem behavior;
- `httptest.Server` for realistic HTTP interactions;
- fuzz tests when exploring broad input spaces, such as parsers or path handling, adds useful protection beyond representative cases;
- golden files only when the complete output is intentionally stable;
- integration tests where correctness depends on component interaction; and
- a small number of representative end-to-end tests when assembled workflows expose meaningful risks not adequately covered at narrower boundaries.

Prefer `errors.Is`, `errors.As`, typed errors, or structured error fields. Use exact error-wording assertions only when the wording is contractual.

At public API boundaries, prefer stable error identity, structured values, and
the smallest semantic fragment needed to identify the failure. Use complete
diagnostic snapshots only when the wording is contractual.

Golden-file updates must require an explicit local flag. Ordinary validation runs must never update golden files automatically, and reviewers must inspect the semantic diff before accepting an update.

Keep tests readable and direct. Size test helpers and fixture frameworks to the maintenance benefit they provide.

## Coverage

Use coverage to locate untested critical branches and unexpectedly weak packages. Evaluate test quality through meaningful behavioral protection and choose coverage according to risk.

Pure domain logic will often warrant higher coverage than facade wiring or external adapters. Uneven coverage is acceptable when it reflects risk.

## Regression tests

A bug fix should normally include a regression test that fails before the fix and passes afterward.

Retain the test when the defect could realistically recur and its consequences justify the ongoing cost. Prefer the narrowest durable test of the violated contract or invariant.

Remove or consolidate a test when the underlying design has made recurrence impossible, its protection is redundant, or a stronger invariant test subsumes it.

## Deleting or rewriting tests

Delete or rewrite tests when maintenance cost exceeds their continuing
protective value.

Strong candidates include tests that:

- require updates after harmless internal changes;
- assert incidental private constants or snapshot large amounts of incidental output;
- verify mock choreography rather than outcomes;
- duplicate policy across layers, retest trivial helpers covered by package
  behavior, or protect risks covered more effectively elsewhere;
- are flaky, misleading, obsolete, or disproportionately expensive to diagnose; or
- no longer correspond to a plausible failure mode.

Several brittle tests may encode one genuine requirement. Consolidate them into one durable behavior-level or invariant test.

## Reviewing a proposed test

Use the following questions when a proposed test needs closer evaluation. Significant test additions should be reviewable against them; written answers are optional for routine tests.

1. Which requested behavior or established contract does it protect, and what realistic defect would violate it?
2. How likely and consequential is that defect, and is it already protected elsewhere?
3. What is the narrowest stable boundary that can test it clearly?
4. Would a correct internal refactor or a change to an incidental default require test edits?
5. Does its confidence justify its maintenance, execution, and diagnostic cost, or would a smaller test suffice?

## Definition of sufficient

A test suite is sufficient when:

- important requested behavior and established contracts are protected against realistic, consequential failures;
- boundaries and component interactions have coverage where they introduce meaningful risks not protected elsewhere;
- failures provide useful, distinct signals;
- tests usually remain stable through legitimate internal changes; and
- additional tests would mostly repeat existing protection or preserve inconsequential implementation details.

Reassess sufficiency as the application, its users, and the consequences of
failure evolve.