|
|
|
|
@@ -1,14 +1,16 @@
|
|
|
|
|
# Testing Policy
|
|
|
|
|
|
|
|
|
|
**Base Policy Version:** 1.0
|
|
|
|
|
**Base Policy Version:** 1.3
|
|
|
|
|
|
|
|
|
|
**Last Updated:** 2026-09-06
|
|
|
|
|
**Last Updated**: 2026-09-28
|
|
|
|
|
|
|
|
|
|
**Upstream Source:** [Standard Repository Policies](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/testing.md)
|
|
|
|
|
|
|
|
|
|
## Purpose
|
|
|
|
|
|
|
|
|
|
Our tests exist to make **incorrect changes expensive and correct changes cheap**.
|
|
|
|
|
|
|
|
|
|
We do not optimize for test count, line coverage, exhaustive isolation, or the fewest possible tests. We optimize for sufficient confidence in important behavior while imposing as little unnecessary friction as possible on future development.
|
|
|
|
|
We optimize for sufficient confidence in important behavior while minimizing friction on future development.
|
|
|
|
|
|
|
|
|
|
## ConfigKit-specific priorities
|
|
|
|
|
|
|
|
|
|
@@ -44,32 +46,19 @@ policy does not imply that those tools or tests already exist.
|
|
|
|
|
|
|
|
|
|
## Every test has a cost
|
|
|
|
|
|
|
|
|
|
Testing is not an unqualified good. Every test imposes both an immediate cost and a continuing lifetime cost.
|
|
|
|
|
|
|
|
|
|
A test must be:
|
|
|
|
|
|
|
|
|
|
- written and reviewed;
|
|
|
|
|
- understood by future maintainers and coding agents;
|
|
|
|
|
- executed in local and CI workflows;
|
|
|
|
|
- diagnosed when it fails;
|
|
|
|
|
- updated when legitimate behavior changes;
|
|
|
|
|
- maintained as fixtures, APIs, and dependencies evolve; and
|
|
|
|
|
- removed or rewritten when it becomes redundant, brittle, misleading, or obsolete.
|
|
|
|
|
|
|
|
|
|
Tests also create cognitive and architectural friction. They can constrain refactoring, duplicate policy, slow feedback loops, add noise to failures, and cause harmless implementation changes to require unrelated edits across the suite.
|
|
|
|
|
Every test has an immediate and continuing cost: writing, review, execution,
|
|
|
|
|
diagnosis, fixture maintenance, and updates when legitimate behavior changes.
|
|
|
|
|
Tests can also constrain refactoring, duplicate policy, and add failure noise.
|
|
|
|
|
|
|
|
|
|
A test is warranted only when the confidence it provides justifies these costs.
|
|
|
|
|
|
|
|
|
|
Apply this cost-benefit analysis at two levels:
|
|
|
|
|
Evaluate that tradeoff at two levels:
|
|
|
|
|
|
|
|
|
|
1. **Per test:** What realistic defect does this test detect, how consequential would that defect be, and is that protection worth the test's lifetime cost?
|
|
|
|
|
2. **Across the suite:** Does this collection provide materially more confidence than a smaller, simpler suite would?
|
|
|
|
|
|
|
|
|
|
The preferred test suite is a **lean suite that provides sufficient confidence in the risks that matter, without redundant or low-value tests**. We seek sufficient confidence with the least unnecessary testing friction, not the fewest possible tests.
|
|
|
|
|
Some friction is intentional. Tests should make dangerous changes—such as breaking compatibility, corrupting data, violating security boundaries, or reintroducing subtle bugs—require deliberate review. Keep ordinary internal changes inexpensive.
|
|
|
|
|
|
|
|
|
|
Some friction is intentional. Tests should make dangerous changes—such as breaking compatibility, corrupting data, violating security boundaries, or reintroducing subtle bugs—require deliberate review. They should not make ordinary internal changes needlessly expensive.
|
|
|
|
|
|
|
|
|
|
The cost of a test is not a reason to omit testing by default. Do not cite maintenance cost abstractly. When omitting a plausible test, be able to state why the protected failure is low-risk, already covered, obvious, reversible, or cheaper to detect elsewhere. For consequential, subtle, or difficult-to-observe behavior, the presumption should favor testing.
|
|
|
|
|
Evaluate maintenance cost against the specific failure being protected. When omitting a plausible test, be able to state why the protected failure is low-risk, already covered, obvious, reversible, or cheaper to detect elsewhere. For consequential, subtle, or difficult-to-observe behavior, the presumption should favor testing.
|
|
|
|
|
|
|
|
|
|
## Default testing style
|
|
|
|
|
|
|
|
|
|
@@ -81,6 +70,8 @@ Use a **classical/Detroit-style** approach:
|
|
|
|
|
- Prefer package-level behavioral tests over tests coupled to private helpers or internal call sequences.
|
|
|
|
|
- Treat exact collaborator interactions as testable behavior only when the interaction itself is a requirement.
|
|
|
|
|
|
|
|
|
|
Examples of appropriate seams include clocks, randomness, subprocesses, remote APIs, object storage, email, and paid LLM calls.
|
|
|
|
|
|
|
|
|
|
For ConfigKit, controlled input readers and filesystem failures are useful seams.
|
|
|
|
|
Prefer actual parsing, composition, and decoding over mocks of their call order.
|
|
|
|
|
|
|
|
|
|
@@ -92,7 +83,7 @@ Control clocks, randomness, environment variables, and other process-global or m
|
|
|
|
|
|
|
|
|
|
## What deserves tests
|
|
|
|
|
|
|
|
|
|
Prioritize tests for:
|
|
|
|
|
To the extent the following features are present in the application, prioritize tests for:
|
|
|
|
|
|
|
|
|
|
1. Public and package-level contracts.
|
|
|
|
|
2. Domain rules and important invariants.
|
|
|
|
|
@@ -102,15 +93,27 @@ Prioritize tests for:
|
|
|
|
|
6. Previously observed or plausible regressions.
|
|
|
|
|
7. Representative integration and end-to-end workflows.
|
|
|
|
|
|
|
|
|
|
A package-level contract is behavior relied upon by another package or major collaborator, not every observable detail of a package implementation.
|
|
|
|
|
A package-level contract comprises behavior relied upon by another package or major collaborator.
|
|
|
|
|
|
|
|
|
|
For behavior involving **data integrity, destructive operations, compatibility, security, concurrency, idempotency, or recovery**, presume that durable tests are required unless the behavior is already credibly protected at another layer.
|
|
|
|
|
Choose tests for requested behavior and established contracts. Tests must not
|
|
|
|
|
introduce new restrictions, guarantees, or threat scenarios merely because they
|
|
|
|
|
could be asserted. Follow the [agent policy](agents.md) when deciding whether a
|
|
|
|
|
behavior belongs to the contract; current implementation behavior alone is not
|
|
|
|
|
sufficient authority.
|
|
|
|
|
|
|
|
|
|
For established behavior involving **data integrity, destructive operations,
|
|
|
|
|
compatibility, security, concurrency, idempotency, or recovery**, favor durable
|
|
|
|
|
tests for realistic failures whose consequences warrant protection. Identify
|
|
|
|
|
the failure and the existing contract it would violate. These categories call
|
|
|
|
|
for attention, not an automatic test inventory or permission to strengthen the
|
|
|
|
|
application's guarantees. Account for protection already provided at other layers.
|
|
|
|
|
|
|
|
|
|
Do not add tests merely because a function, branch, or line exists. Do not add a test when the same meaningful risk is already adequately protected elsewhere.
|
|
|
|
|
|
|
|
|
|
## Choose the right test boundary
|
|
|
|
|
|
|
|
|
|
Test through the narrowest stable boundary that expresses the behavior clearly.
|
|
|
|
|
Choose the narrowest stable boundary that expresses the behavior clearly and
|
|
|
|
|
provides durable confidence with minimal incidental coupling.
|
|
|
|
|
|
|
|
|
|
This is often the package API, but it may instead be:
|
|
|
|
|
|
|
|
|
|
@@ -118,28 +121,15 @@ This is often the package API, but it may instead be:
|
|
|
|
|
- a package-level operation when several internal collaborators jointly produce the behavior; or
|
|
|
|
|
- a larger integration boundary when correctness emerges from interaction with a real dependency.
|
|
|
|
|
|
|
|
|
|
Do not force all behavior through oversized end-to-end tests. Do not test every private helper merely because it exists. Choose the boundary that gives durable confidence with the least incidental coupling.
|
|
|
|
|
|
|
|
|
|
## Test behavior, not implementation
|
|
|
|
|
|
|
|
|
|
A test should protect a decision, contract, or invariant—not memorialize the current implementation.
|
|
|
|
|
A test should protect a selected decision, contract, or invariant.
|
|
|
|
|
|
|
|
|
|
Before adding or retaining a test, ask:
|
|
|
|
|
A test is suspect when it primarily detects changes to private constants or
|
|
|
|
|
helpers, equivalent operation ordering, incidental formatting, equivalent
|
|
|
|
|
algorithms, or internal structure that preserves behavior.
|
|
|
|
|
|
|
|
|
|
> What realistic defect would this test catch?
|
|
|
|
|
|
|
|
|
|
A test is suspect when its main purpose is to detect that someone:
|
|
|
|
|
|
|
|
|
|
- changed an internal constant;
|
|
|
|
|
- renamed or split a private helper;
|
|
|
|
|
- reordered equivalent internal operations;
|
|
|
|
|
- changed incidental formatting;
|
|
|
|
|
- replaced one correct algorithm with another; or
|
|
|
|
|
- refactored internal object structure without changing behavior.
|
|
|
|
|
|
|
|
|
|
Refactoring should normally require no test edits unless the refactored structure is itself part of the contract.
|
|
|
|
|
|
|
|
|
|
A test can be factually correct and still have negative value. Accurately describing current behavior is not enough; the protected behavior must be important enough to justify the future friction.
|
|
|
|
|
Behavioral tests should normally remain stable through internal refactoring that preserves the contract.
|
|
|
|
|
|
|
|
|
|
## Expected effects of different changes
|
|
|
|
|
|
|
|
|
|
@@ -156,12 +146,13 @@ A test failing is not the same as a test needing to be edited. Many tests may co
|
|
|
|
|
|
|
|
|
|
## Separate mechanism from policy
|
|
|
|
|
|
|
|
|
|
Configurable thresholds and defaults must not be duplicated throughout the test suite.
|
|
|
|
|
Derive expectations from test-controlled configuration and relationships.
|
|
|
|
|
|
|
|
|
|
For example, do not encode an internal concurrency limit indirectly:
|
|
|
|
|
For this example, assume the selected contract rejects a request to start more
|
|
|
|
|
processes than a configured limit. Do not encode its production default indirectly:
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
// Production policy:
|
|
|
|
|
// Production default:
|
|
|
|
|
const maxConcurrency = 4
|
|
|
|
|
|
|
|
|
|
// Brittle test:
|
|
|
|
|
@@ -179,25 +170,30 @@ require.NoError(t, runner.Start(limit))
|
|
|
|
|
require.ErrorIs(t, runner.Start(limit+1), ErrTooMuchConcurrency)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The test should prove:
|
|
|
|
|
For this selected contract, the test should prove:
|
|
|
|
|
|
|
|
|
|
- the configured limit is accepted; and
|
|
|
|
|
- one beyond the configured limit is rejected.
|
|
|
|
|
|
|
|
|
|
Rejection is not a general requirement for concurrency control. If the selected
|
|
|
|
|
contract queues excess work or limits active workers while accepting larger
|
|
|
|
|
batches, test that behavior instead. Do not add rejection or a configurable
|
|
|
|
|
limit solely to reproduce this example.
|
|
|
|
|
|
|
|
|
|
The production default should be tested exactly only when its literal value is itself a public, operational, safety, protocol, or compatibility requirement.
|
|
|
|
|
|
|
|
|
|
Apply the same rule to limits, timeouts, capacities, retry counts, and ranges: test relationships and behavior, not duplicated literals.
|
|
|
|
|
Apply the same rule to limits, timeouts, capacities, retry counts, and ranges: test relationships and behavior.
|
|
|
|
|
|
|
|
|
|
For concurrency limits, test both kinds of behavior when relevant:
|
|
|
|
|
For established concurrency limits, distinguish these behaviors where the contract requires them:
|
|
|
|
|
|
|
|
|
|
1. **Configuration enforcement:** invalid or excessive requested values are handled correctly.
|
|
|
|
|
1. **Configuration handling:** values are accepted or rejected according to the selected contract.
|
|
|
|
|
2. **Runtime enforcement:** observed peak concurrency never exceeds the configured limit.
|
|
|
|
|
|
|
|
|
|
Use a test-controlled limit and measure the behavior relative to that limit. Do not merely assert today's default value.
|
|
|
|
|
Use a test-controlled limit and measure the behavior relative to that limit.
|
|
|
|
|
|
|
|
|
|
## Avoid semantic duplication across layers
|
|
|
|
|
## Assign each behavior a test owner
|
|
|
|
|
|
|
|
|
|
Each behavior should have a clear test owner.
|
|
|
|
|
For ConfigKit, assign ownership as follows:
|
|
|
|
|
|
|
|
|
|
- YAML boundary tests own syntax and policy enforcement.
|
|
|
|
|
- Composition and overlay tests own their operation semantics.
|
|
|
|
|
@@ -209,9 +205,7 @@ Each behavior should have a clear test owner.
|
|
|
|
|
|
|
|
|
|
These are responsibility boundaries, not a required package layout.
|
|
|
|
|
|
|
|
|
|
Higher-level tests should not repeat every lower-level case. A single intentional policy change should not require unrelated edits across many test files.
|
|
|
|
|
|
|
|
|
|
Tests that are individually reasonable may still be collectively redundant. Evaluate the marginal value of each additional test in light of the protection already provided by the rest of the suite.
|
|
|
|
|
Give each layer its distinct responsibility and keep expectation updates localized to the contract being changed.
|
|
|
|
|
|
|
|
|
|
## Use test doubles deliberately
|
|
|
|
|
|
|
|
|
|
@@ -232,104 +226,77 @@ Mocks are appropriate when the contract includes facts such as:
|
|
|
|
|
- an expensive API is called no more than once; or
|
|
|
|
|
- a security audit event is emitted.
|
|
|
|
|
|
|
|
|
|
Do not use mocks merely to isolate every object or reproduce the implementation's call graph.
|
|
|
|
|
|
|
|
|
|
## Go-specific guidance
|
|
|
|
|
|
|
|
|
|
Use:
|
|
|
|
|
Select techniques by behavior and risk:
|
|
|
|
|
|
|
|
|
|
- table-driven tests for meaningful behavioral categories and boundaries;
|
|
|
|
|
- `t.TempDir()` for real filesystem behavior;
|
|
|
|
|
- fuzz tests for parsers, normalization, path handling, and broad input spaces;
|
|
|
|
|
- `httptest.Server` for realistic HTTP interactions;
|
|
|
|
|
- fuzz tests when exploring broad input spaces, such as parsers or path handling, adds useful protection beyond representative cases;
|
|
|
|
|
- golden files only when the complete output is intentionally stable;
|
|
|
|
|
- integration tests where correctness depends on component interaction; and
|
|
|
|
|
- a small number of representative end-to-end tests.
|
|
|
|
|
- a small number of representative end-to-end tests when assembled workflows expose meaningful risks not adequately covered at narrower boundaries.
|
|
|
|
|
|
|
|
|
|
Avoid exact error-string assertions unless the wording is itself contractual. Prefer `errors.Is`, `errors.As`, typed errors, or structured error fields.
|
|
|
|
|
Prefer `errors.Is`, `errors.As`, typed errors, or structured error fields. Use exact error-wording assertions only when the wording is contractual.
|
|
|
|
|
|
|
|
|
|
At the public library boundary, prefer diagnostic codes, paths, and source
|
|
|
|
|
locations over complete message snapshots. Assert wording only when contractual.
|
|
|
|
|
|
|
|
|
|
Golden-file updates must require an explicit local flag. CI must not update golden files automatically, and reviewers must inspect the semantic diff before accepting an update.
|
|
|
|
|
|
|
|
|
|
Keep tests readable and direct. Test helpers and fixture frameworks must earn their own maintenance cost; do not build elaborate test infrastructure for small or isolated needs.
|
|
|
|
|
Keep tests readable and direct. Size test helpers and fixture frameworks to the maintenance benefit they provide.
|
|
|
|
|
|
|
|
|
|
## Coverage
|
|
|
|
|
|
|
|
|
|
Coverage is a diagnostic, not a target.
|
|
|
|
|
Use coverage to locate untested critical branches and unexpectedly weak packages. Evaluate test quality through meaningful behavioral protection and choose coverage according to risk.
|
|
|
|
|
|
|
|
|
|
Use it to find untested critical branches and unexpectedly weak packages. Do not write low-value tests solely to increase a percentage, and do not infer test quality from coverage alone.
|
|
|
|
|
|
|
|
|
|
Dense configuration semantics will often warrant higher coverage than simple input wiring. Uneven coverage is acceptable when it reflects risk.
|
|
|
|
|
|
|
|
|
|
Increasing coverage is valuable only when the newly covered behavior protects a meaningful risk at an acceptable cost.
|
|
|
|
|
Pure domain logic will often warrant higher coverage than CLI wiring or external adapters. Uneven coverage is acceptable when it reflects risk.
|
|
|
|
|
|
|
|
|
|
## Regression tests
|
|
|
|
|
|
|
|
|
|
A bug fix should normally include a regression test that fails before the fix and passes afterward.
|
|
|
|
|
|
|
|
|
|
Retain the test when the defect could realistically recur and its consequences justify the ongoing cost. Prefer the narrowest durable test of the violated contract or invariant; do not preserve accidental implementation details from the original bug.
|
|
|
|
|
Retain the test when the defect could realistically recur and its consequences justify the ongoing cost. Prefer the narrowest durable test of the violated contract or invariant.
|
|
|
|
|
|
|
|
|
|
Not every historical bug requires a permanent test. If the underlying design has made recurrence impossible, the test has become redundant, or a stronger invariant test now subsumes it, remove or consolidate it.
|
|
|
|
|
Remove or consolidate a test when the underlying design has made recurrence impossible, its protection is redundant, or a stronger invariant test subsumes it.
|
|
|
|
|
|
|
|
|
|
## Deleting or rewriting tests
|
|
|
|
|
|
|
|
|
|
Tests are maintained code, not permanent historical artifacts.
|
|
|
|
|
|
|
|
|
|
Delete or rewrite a test when its maintenance cost exceeds the confidence it provides.
|
|
|
|
|
Delete or rewrite tests when maintenance cost exceeds their continuing
|
|
|
|
|
protective value.
|
|
|
|
|
|
|
|
|
|
Strong candidates include tests that:
|
|
|
|
|
|
|
|
|
|
- require updates after harmless internal changes;
|
|
|
|
|
- directly assert private constants without protecting a real contract;
|
|
|
|
|
- duplicate the same policy across several layers;
|
|
|
|
|
- assert incidental private constants or snapshot large amounts of incidental output;
|
|
|
|
|
- verify mock choreography rather than outcomes;
|
|
|
|
|
- snapshot large amounts of incidental output;
|
|
|
|
|
- test trivial private helpers already exercised through stable package behavior;
|
|
|
|
|
- protect risks already covered more effectively elsewhere;
|
|
|
|
|
- duplicate policy across layers, retest trivial helpers covered by package
|
|
|
|
|
behavior, or protect risks covered more effectively elsewhere;
|
|
|
|
|
- are flaky, misleading, obsolete, or disproportionately expensive to diagnose; or
|
|
|
|
|
- no longer correspond to a plausible failure mode.
|
|
|
|
|
|
|
|
|
|
Several brittle tests may encode one genuine requirement. Replace them with one durable behavior-level or invariant test rather than preserving all of them.
|
|
|
|
|
|
|
|
|
|
Deleting a low-value test can improve the quality of the suite by reducing noise, maintenance burden, and friction around legitimate change.
|
|
|
|
|
Several brittle tests may encode one genuine requirement. Consolidate them into one durable behavior-level or invariant test.
|
|
|
|
|
|
|
|
|
|
## Reviewing a proposed test
|
|
|
|
|
|
|
|
|
|
Use the following questions when the value, boundary, or durability of a proposed test is not self-evident. Significant test additions should be reviewable against them, but written answers are not required for every routine test.
|
|
|
|
|
Use the following questions when a proposed test needs closer evaluation. Significant test additions should be reviewable against them; written answers are optional for routine tests.
|
|
|
|
|
|
|
|
|
|
1. What realistic defect would it catch?
|
|
|
|
|
2. How likely is that defect?
|
|
|
|
|
3. How consequential would it be?
|
|
|
|
|
4. Is the behavior already protected elsewhere?
|
|
|
|
|
5. At which layer should this behavior be owned?
|
|
|
|
|
6. Does the test assert a durable contract or an incidental implementation detail?
|
|
|
|
|
7. Could the implementation be refactored without changing the behavior and without editing this test?
|
|
|
|
|
8. What should cause this test to fail?
|
|
|
|
|
9. What legitimate changes should not cause this test to fail?
|
|
|
|
|
10. What ongoing maintenance, execution, and diagnostic cost will the test impose?
|
|
|
|
|
11. Is there a smaller or more direct test that protects the same risk?
|
|
|
|
|
|
|
|
|
|
Do not add the test when its expected lifetime cost exceeds its expected protective value.
|
|
|
|
|
|
|
|
|
|
When deciding not to test plausible behavior, record or be able to explain why the risk is low, already protected, obvious, reversible, or cheaper to detect elsewhere.
|
|
|
|
|
1. Which requested behavior or established contract does it protect, and what realistic defect would violate it?
|
|
|
|
|
2. How likely and consequential is that defect, and is it already protected elsewhere?
|
|
|
|
|
3. What is the narrowest stable boundary that can test it clearly?
|
|
|
|
|
4. Would a correct internal refactor or a change to an incidental default require test edits?
|
|
|
|
|
5. Does its confidence justify its maintenance, execution, and diagnostic cost, or would a smaller test suffice?
|
|
|
|
|
|
|
|
|
|
## Definition of sufficient
|
|
|
|
|
|
|
|
|
|
A test suite is sufficient when:
|
|
|
|
|
|
|
|
|
|
- important contracts and invariants are protected;
|
|
|
|
|
- meaningful boundaries and failure modes are exercised;
|
|
|
|
|
- realistic and consequential regressions are credibly protected against silent recurrence;
|
|
|
|
|
- behavior involving data integrity, destructive operations, compatibility, security, concurrency, idempotency, and recovery is credibly protected;
|
|
|
|
|
- important external boundaries have realistic integration coverage;
|
|
|
|
|
- representative complete workflows are tested;
|
|
|
|
|
- failures provide useful signal rather than redundant noise;
|
|
|
|
|
- legitimate internal changes usually do not require test edits; and
|
|
|
|
|
- important requested behavior and established contracts are protected against realistic, consequential failures;
|
|
|
|
|
- boundaries and component interactions have coverage where they introduce meaningful risks not protected elsewhere;
|
|
|
|
|
- failures provide useful, distinct signals;
|
|
|
|
|
- tests usually remain stable through legitimate internal changes; and
|
|
|
|
|
- additional tests would mostly repeat existing protection or preserve inconsequential implementation details.
|
|
|
|
|
|
|
|
|
|
Sufficiency is a risk judgment, not a coverage percentage or test count. Reassess it as the library, its consumers, and the consequences of failure evolve.
|
|
|
|
|
|
|
|
|
|
The governing rule is:
|
|
|
|
|
|
|
|
|
|
> Test heavily where failure is consequential, subtle, or difficult to detect after the fact. Test lightly where failure is obvious, reversible, and inexpensive—and retain no test whose lifetime cost exceeds the confidence it provides.
|
|
|
|
|
Reassess sufficiency as the application, its users, and the consequences of
|
|
|
|
|
failure evolve.
|
|
|
|
|
|
