eric/scriptorium
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 9 Wiki Activity

Adopt updated policies and simplify release publication

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-26 14:25:32 +00:00
parent f33af20524
commit 537bddf40c
11 changed files with 1301 additions and 440 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
docs/policy/agents.md Normal file
View File

@@ -0,0 +1,77 @@
# Agent Policy
**Base Policy Version:** 1.1
**Last Updated:** 2026-09-26
## Purpose
Agents working in this repository must read and adhere to the following house rules.
## Implementation Discipline
**KEY PRINCIPLES:** Write the simplest idiomatic code that satisfies the user's
request. Do not implicitly strengthen requirements, restrictions, or guarantees
beyond what the user actually requested.
**Keep implementation machinery proportionate.** Use straightforward,
idiomatic code and introduce only the machinery needed to support the requested
behavior and flexibility. Optimize for conceptual simplicity and
maintainability, not for the fewest lines, fewest options, or narrowest product
surface.
**Do not invent restrictions.** Do not reject, prohibit, constrain, cap,
harden, or otherwise narrow behavior unless required by the user, an explicit
repository contract, or an unavoidable correctness constraint. Preserve
operator and caller choice where policy has not been established. If an
additional restriction appears advisable, present it as a decision rather than
silently implementing it.
**Do not invent guarantees.** Do not strengthen public contracts, guarantees,
or invariants beyond those expressly requested by the user.
**Use the standard library.** For example: in almost every case, the best
approach to copying files is to use the standard library. Writing custom code to
implement power-loss protection at the application level, or to guard against
bad actors with privileged and concurrent access to the host's filesystem,
is wasteful, unnecessary, and not idiomatic. Stick to the simple, standard, and
idiomatic approach unless another path is expressly requested by the user.
**An implementation detail is not repository policy.** Tests and documentation
must not elevate incidental behavior into a durable guarantee or restriction
unless that behavior was expressly selected as part of the contract.
## Git Commit Policy
By default, create a commit after completing each coherent unit of work, using a
concise, plain-English message describing what changed. This policy is standing
authorization to create those local commits; a separate user request is not
required. An explicit user instruction to leave work uncommitted takes precedence.
A unit of work is a focused, reviewable outcome, including its relevant tests and
documentation. Do not commit after every intermediate edit or create empty commits.
Complete the appropriate validation before committing. If work is incomplete or
validation is blocked, report that state rather than presenting it as completed.
Before committing:
1. Run `git status --short` and review the relevant diff, including any staged
changes. Identify changes that predate the current unit of work.
2. Exclude secrets, local environment files, generated junk, and unrelated changes.
3. Stage only the files or hunks belonging to the completed unit. Use `git add .`
only when every included change belongs to that unit. Do not disturb unrelated
staged or unstaged work.
4. Review `git diff --staged` and run appropriate checks for the change. Do not
repeat checks already passed for unchanged content merely to create a commit.
5. Review `git log --oneline -n 10` and use
`git commit -m "<concise plain-English commit message>"`.
6. Report the commit identifier and any remaining uncommitted work.
If the unit cannot be safely separated from pre-existing work, leave it uncommitted
and explain the overlap rather than silently including that work. If a commit
fails, preserve the changes and report the failure; do not bypass hooks.
Default commit authorization does not authorize pushing, publishing, amending,
resetting, reverting, or rewriting history. Those actions require a separate user
instruction. Do not use `git commit -a` or `git commit --no-verify` to bypass the
staging and validation discipline above.

51
docs/policy/architecture.md
View File

@@ -1,7 +1,21 @@
# Architecture
# Architecture Policy
This document defines Scriptorium's current application architecture and
durable development boundaries.
**Base Policy Version:** 1.1
**Last Updated:** 2026-09-26
## Purpose And Ownership
This policy defines Scriptorium's purpose, responsibility and dependency
boundaries, established invariants, and non-goals. Concrete component inventory
and implementation mechanics belong in [internal documentation](../internal/overview.md);
public behavior belongs in its canonical contracts.
Adopted constraints govern future development without implying that every
capability they could govern exists or is planned. Keep proposed structure and
implementation status in roadmaps, and decision rationale in ADRs. Acceptance
of a decision is not proof of implementation. Do not turn incidental choices
into architectural invariants.
## System Shape
@@ -21,6 +35,16 @@ deployment policy.
The concrete package inventory is maintained in the
[internal overview](../internal/overview.md).
As established by [ADR 0004](../adr/0004-definition-boundary.md), Scriptorium
aims to support the complete set of features in valid Promptkit prompt and
profile definitions for its selected upstream version. Promptkit remains the
canonical parser and validator. Library-only features are considered
individually against the simple predefined-prompt execution model.
Maintain adapter boundaries with proportionate, idiomatic code. This approach
does not require an interface for every type, additional wrappers, or a fixed
directory tree for integrations that do not exist.
## Dependency Direction
```text
@@ -46,6 +70,10 @@ gitea.maximumdirect.net/eric/promptkit
The repository architecture guard enforces these import and removal
invariants.
Cross-project changes land and release in Promptkit before Scriptorium adopts
the tagged dependency. Do not commit Go workspaces, local replacements,
vendored Promptkit implementation, or unpublished dependency assumptions.
## Retained Boundaries
- `internal/adapter/cli` owns commands, flags, configuration precedence,
@@ -71,6 +99,13 @@ That reader denies file references without an artifact root, enforces the
configured byte limit, and applies Scriptorium's lexical root-containment rule.
The operating system still follows symlinks after the lexical check.
This boundary handles untrusted request references under the documented
deployment model; it is not a filesystem sandbox. Defending against a
privileged actor concurrently modifying the host filesystem is outside the
adopted scope, not a claim that such activity cannot occur. Do not infer
additional defenses or guarantees from hypothetical threats; raise unsettled
scope decisions under the [agent policy](agents.md).
The [HTTP API](../api.md) owns observable request outcomes, and
[operations](../operations.md) owns deployment permissions and root selection.
@@ -80,6 +115,14 @@ Scriptorium has no durable run-state store, checkpoint, cache, or resume
mechanism. Recovery is a new request after correcting inputs, configuration, or
environment.
Each request is self-contained from the caller's perspective, without
conversation or workflow state retained from completed requests. This does
not mean the process is internally stateless. Shared engines, backend capacity
and queue accounting, in-flight coordination, cancellation, and server
lifecycle resources are permitted and must be shared where necessary to
enforce configured backend limits across concurrent requests. Such state must
be concurrency-safe and does not create a durable user-workflow contract.
Adapters map Promptkit public error identities into CLI exits or HTTP statuses
without classifying by message text. Raw API keys are not accepted in
Scriptorium configuration, CLI arguments, or HTTP payloads, and resolved
@@ -102,5 +145,7 @@ secrets must not be emitted.
into Scriptorium.
- Do not move CLI, inbound HTTP, process, or deployment policy into Promptkit.
- Do not add durable workflow, archive, or resume behavior.
- Do not add application-managed prompt modification, follow-up message flows,
conversation management, or multi-step workflow orchestration.
Work that is not implemented belongs in `docs/roadmap/`.

36
docs/policy/documentation.md
View File

@@ -1,5 +1,9 @@
# Documentation Policy
**Base Policy Version:** 1.3
**Last Updated:** 2026-09-26
## Purpose
This policy assigns each documentation topic to one canonical owner. Its goal is
@@ -30,10 +34,14 @@ semantics.
Outside `docs/roadmap/`, documentation describes implemented behavior only.
Partial features may be described only to their implemented boundary.
ADRs are the narrow exception: an ADR may record an accepted architectural
decision before implementation, but acceptance must not be presented as proof
that the behavior exists. The roadmap owns implementation status and sequencing
until the decision is implemented. Current architecture, user, operator,
Architectural constraints may govern future changes without implying that the
components or capabilities they would govern already exist. This does not
permit describing a proposed system structure as implemented.
ADRs may record an accepted architectural decision before implementation, but
acceptance must not be presented as proof that the behavior exists. The roadmap
owns implementation status and sequencing until the decision is implemented.
Current architecture, user, operator,
integration, and internal documentation are updated when the behavior lands.
### Audience And Detail
@@ -65,13 +73,15 @@ secret values.
| --- | --- | --- | --- |
| Product orientation and minimal end-to-end quickstart | `README.md` | What this project is, why it is useful, one shortest successful invocation, and links onward. | Complete command reference, configuration reference, operational procedures, implementation detail. |
| Contributor entry point | `docs/development.md` | Task-oriented reading guide, minimal contributor orientation, baseline validation commands, and links to canonical docs. | Package inventory, architecture rules, subsystem behavior, and detailed change recipes, which belong in the relevant internal component document. |
| Current application architecture | `docs/policy/architecture.md` | System shape, normative ownership, dependency direction, architectural boundaries, invariants, safety properties, and non-goals. | Concrete package inventory, implementation mechanics, contributor procedures, decision history, future work. |
| Application architecture policy | `docs/policy/architecture.md` | System shape, normative ownership, dependency direction, architectural boundaries, invariants, safety properties, and non-goals. | Concrete package inventory, implementation mechanics, contributor procedures, decision history, future work. |
| Agent policy | `docs/policy/agents.md` | Agent implementation discipline, proportionality, idiomatic simplicity, local commit rules, and rules against inventing requirements, restrictions, guarantees, or repository policy. | Application contracts, architecture, testing policy, and contributor procedures. |
| Documentation organization | `docs/policy/documentation.md` | Documentation ownership, audience boundaries, maintenance rules, and ADR/document lifecycle. | Application architecture or product behavior. |
| Testing policy | `docs/policy/testing.md` | Test philosophy, risk-based sufficiency, test boundaries, doubles, coverage guidance, regression-test policy, and criteria for adding, rewriting, or deleting tests. | Subsystem behavior, application contracts, subsystem-specific test inventories, and implementation plans. |
| CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, and exit codes. | End-to-end operating procedures, configuration field definitions, runtime filesystem layout, module implementation details. |
| Configuration contract | `docs/config.md` | Application discovery and precedence, source locations, server fields, render default, HTTP limits, and credential mapping. | Promptkit framework formats and defaults, complete example files, CLI syntax, runtime lifecycle, and implementation detail. |
| Operations | `docs/operations.md` | Runtime workflows, physical filesystem and state layout, output, cache, and debug handling, resume, cleanup, permissions, recovery, and operational limits. | CLI flag syntax, configuration field definitions, logical output schemas, implementation mechanics. |
| Release procedure | `docs/release.md` | Candidate validation, version and tag operations, hosted-workflow observation, and published-artifact verification. | Runtime operations, version-specific announcements, and complete application-interface contracts. |
| Release policy | `docs/policy/release.md` | Versioning and compatibility, release integrity and immutability, documentation requirements, proportionate gates, publication controls, source completion, and optional verification boundaries. | Repository-specific commands, installation guidance, historical summaries, and CI implementation details. |
| Release procedure | `docs/release.md` | Maintainer prerequisites, candidate preparation, publication command and guards, asynchronous delivery, optional diagnostics, and failure recovery. | General release requirements, runtime operations, historical announcements, and current application contracts. |
| Version-specific release notes | `docs/releases/` | Immutable release summaries, compatibility notices, and migration announcements for one published version. | Complete CLI, HTTP, configuration, operations, or dependency contracts. |
| Public HTTP contract | `docs/api.md` | Routes, authentication, media types, request and response schemas, status codes, pagination, caching, idempotency, rate limits, and HTTP retry semantics. | Client walkthroughs, upstream or downstream integration internals, implementation detail. |
| Consumer guidance | `docs/consumers/` | Choosing between Scriptorium's executable interfaces and understanding consumer responsibilities. | HTTP wire semantics, CLI syntax, Promptkit's Go package, and internal implementation detail. |
@@ -103,6 +113,15 @@ runtime state and how to operate or recover the application. When a workflow
crosses these topics, choose the document that owns the task and link to the
other contracts.
### Releases
The release policy owns release requirements; the release procedure supplies
the commands that satisfy them. Release notes are historical summaries, not
current-state contract owners. The checked-in note at an immutable tag records
that release; current documentation changes with the behavior it describes.
Neither a procedure nor a note may weaken policy or defer required updates to
canonical contracts.
### Contracts And Implementation
Integration and API documents define externally observable shapes and
@@ -150,6 +169,11 @@ ADR; rejected product ideas belong in the roadmap.
## Maintenance
Every policy document records its underlying template version and local
last-updated date immediately below the title. Update the date when revising
local policy; change the base version only when adopting a newer template.
Do not retain template adoption instructions in the local policy.
When behavior changes, update its canonical owner in the same change. If
ownership moves, remove the old definition and replace it with a link where
navigation remains useful.

207
docs/policy/release.md Normal file
View File

@@ -0,0 +1,207 @@
# Release Policy
**Base Policy Version:** 1.1
**Last Updated:** 2026-09-26
## Purpose
This policy defines a proportionate workflow for validating and publishing an
intended source revision under a fresh version. Releases should require little
routine intervention, avoid repeated work, and preserve published contents.
The repository-specific commands belong in `docs/release.md`. Historical release
summaries belong in `docs/releases/`. The procedure must agree with the adopted
policy and link to it rather than introduce additional release requirements.
## Release Model And Completion
A source release identifies one exact committed source revision through an
immutable version tag and a version-matched, checked-in release note. The
primary release branch is `main`; the authoritative publication remote is
`origin`.
A successful push of the intended tag, with its source commit available on the
authoritative primary branch, completes the release. Publish `main` before
invoking the publication command; the command requires
that the remote branch exactly identifies the validated candidate and pushes
only the selected annotated tag.
Release commands must finish at successful publication. They must not wait for
or poll CI, require successful CI status, inspect downloadable artifacts, or
check consumer installation or Go module availability as completion gates.
A failed or ambiguous push requires resolving the authoritative remote state
before declaring success or retrying; successful pushes do not require a
separate remote read-back gate.
Scriptorium publishes Linux binaries and checksums asynchronously from the
tagged source. The procedure must identify those deliverables and the
responsibility of their automation. Published deliverables must be traceable
to that source and identify the same version.
Artifact delivery is a separate status. A successful tag push is not evidence
that artifacts are available. Delivery failures do not reverse source-release
success, and optional investigation must not delay release completion.
## Versioning And Compatibility
Stable releases use Semantic Versioning in the form `vMAJOR.MINOR.PATCH`, with
no leading zeroes. Prereleases are not part of the adopted release channel.
The selected version must be unused locally and on the authoritative remote.
Do not require queries to every configured remote or publication service.
A version known to have already been published must not be reused, even if the
normal guards do not inspect that channel. Adding a distribution channel
requires a deliberate update to the policy and procedure where needed.
For releases at or after `v1.0.0`:
- a major release may make intentional incompatible changes;
- a minor release may add backward-compatible behavior; and
- a patch release may make backward-compatible fixes.
Before `v1.0.0`, a minor release may change a documented public, operational,
configuration, data, integration, or compatibility contract when the release
note explains the impact and required consumer action. A patch release must
not intentionally break those contracts within its minor line.
Additional compatibility promises and deliberate exceptions belong in the
adopted policy or canonical contract documentation. Release notes explain
changes; they do not redefine current compatibility contracts.
## Release Documentation And Preparation
Every new release must include `docs/releases/<version>.md` in the tagged
source tree. Use the complete version tag as the filename and begin with a
title containing the product name and that exact version.
Include a short summary and any relevant compatibility impact or upgrade
instructions. Keep the note proportionate to the change. Fixed sections and
explicit no-action boilerplate are not required. Existing historical notes
need not be rewritten to adopt this format.
Keep exact current contracts in their canonical documents and update affected
documentation with the behavior change. Link to those documents from the note
where useful. A release note must not substitute for current documentation.
Use annotated tags with their message derived from the checked-in note rather
than requiring a separately authored account of the same release.
Version selection, release-note content, and retirement of completed roadmaps
are preparation responsibilities, not separate human approval gates. Preserve
planned and deferred work and repair affected links when retiring roadmaps.
Ordinary change review covers unintended files, credentials, and private data;
do not repeat it as a release-specific sign-off. Ignored local files are not
prohibited merely because they exist: they are not included in the tagged source.
## Candidate Validation
Provide one publication command that validates the final committed candidate,
checks publication state, and publishes the intended refs. A validation-only
mode or command may be provided for diagnosis, but must not be a required step
before publication repeats the same checks.
Run the complete deterministic, offline default test suite once for the final
candidate, following the [testing policy](testing.md), and verify the required
build targets. Select those targets for the application's actual support
commitments; native testing on each platform and access to particular hardware
are not default release prerequisites.
The selected test invocation includes the race detector because the HTTP
service shares engines and backend capacity state across concurrent requests.
This single invocation covers both ordinary assertions and race detection.
Build both declared Linux architectures with the hosted release flags to catch
cross-compilation and packaging failures; native test hardware for each target
is not required. Existing architecture tests enforce dependency boundaries.
Retain narrow source guards against committed workspaces, vendor trees, and
module replacements: these can otherwise make a passing local build depend on
unpublished or substituted framework code. Disable workspace discovery and
vendor mode during validation. These guards preserve the established
cross-project boundary; they do not require exact dependency-version audits or
module-metadata rewrites at release time.
Additional gates must address a concrete risk that the existing checks do not
adequately cover. Do not automatically require linting, race testing, module
hygiene audits, documentation audits, credential scans, example validation, or
expanded platform matrices solely because they are possible. Retain such
checks when their distinct protection justifies their cost. A one-time migration
check must not silently become a permanent release requirement.
Release tests must not require real credentials, mutable application services,
paid APIs, provider probes, or inference calls. The procedure describes
dependency preparation separately from offline validation.
Avoid duplicate validation across release phases. Successful checks for the
same final candidate need not be repeated solely to satisfy another phase;
any reuse must clearly apply to that exact candidate. Persistent validation
caches are not required. If the candidate changes, validate the new candidate.
Cheap publication-state checks may be repeated immediately before pushing
because local or remote state may have changed during validation.
Automate mechanical guards. Human review gates are strongly disfavored: add one
only through an explicit project-specific policy decision explaining the
concrete risk and why ordinary change review or automation is insufficient.
Do not add routine confirmations for checks the command can perform itself.
## Publication Guards
Automate the checks needed to publish the intended source under a fresh version:
- the version has the adopted format and is unused locally and on the
authoritative remote;
- the worktree and index are clean on the primary release branch;
- the source being published is the exact candidate that passed validation,
including its matching release note;
- the remote primary branch already identifies that exact candidate; and
- the tag identifies the validated candidate and only the intended refs are
published.
Fetch the authoritative branch and tag state as needed for these checks.
Do not force branch or tag updates to bypass a failed guard. Publish explicit
refs rather than all local tags. Do not require CI success, publication-service
availability checks, or a separate manual confirmation for each guard.
## Immutability And Failure Recovery
Published version tags and published contents are immutable. Do not move,
overwrite, recreate, or delete a published tag, or replace a published artifact,
checksum, signature, provenance statement, or release note under that version.
The checked-in note at the immutable tag is that release's historical record.
If validation fails before publication, correct and commit the candidate and
rerun the publication command. If a push fails or its outcome is ambiguous,
inspect the authoritative remote before retrying. Remove an unpublished local
tag only after confirming it was never published or used by a release channel.
An unavailable remote leaves publication status unknown; do not overwrite a tag
to resolve uncertainty.
Transient network, authentication, CI, or consumer-availability failures do not
by themselves require a new version. Asynchronous delivery may be retried from
the same immutable tag. Missing artifacts may be published later, provided
existing published contents are not replaced and the additions remain consistent
with the tagged source and any already-published checksum manifest. If that
cannot be achieved, publish a new version.
A defect requiring changes to published source or contents requires a corrected
commit, a new semantically appropriate version, a matching release note, and
the release procedure again. Leave the original release intact. A channel's
yank, deprecation, or advisory mechanism may warn consumers away from a defective
release without replacing its contents or reusing its version.
## Repository Release Procedure
Maintain `docs/release.md` as a short, executable maintainer runbook. It must
identify:
1. prerequisites, primary branch, and authoritative remote;
2. version and release-note preparation;
3. one publication command with its selected local checks and automated guards;
4. declared deliverables, required build targets, and automation responsibilities;
5. the successful-push completion point and separately reported delivery status;
and
6. failure handling, ambiguous-push recovery, and asynchronous delivery retries.
Use copyable commands where practical. Keep optional diagnostics and consumer
verification clearly optional. The procedure must not turn preparation work,
ordinary change review, or post-publication investigation into additional
release gates.

125
docs/policy/testing.md
View File

@@ -1,5 +1,9 @@
# Testing Policy
**Base Policy Version:** 1.1
**Last Updated:** 2026-09-26
## Purpose
Our tests exist to make **incorrect changes expensive and correct changes cheap**.
@@ -8,29 +12,16 @@ We do not optimize for test count, line coverage, exhaustive isolation, or the f
## Every test has a cost
Testing is not an unqualified good. Every test imposes both an immediate cost and a continuing lifetime cost.
A test must be:
- written and reviewed;
- understood by future maintainers and coding agents;
- executed in local and CI workflows;
- diagnosed when it fails;
- updated when legitimate behavior changes;
- maintained as fixtures, APIs, and dependencies evolve; and
- removed or rewritten when it becomes redundant, brittle, misleading, or obsolete.
Tests also create cognitive and architectural friction. They can constrain refactoring, duplicate policy, slow feedback loops, add noise to failures, and cause harmless implementation changes to require unrelated edits across the suite.
Every test has an immediate and continuing cost: writing, review, execution,
diagnosis, fixture maintenance, and updates when legitimate behavior changes.
Tests can also constrain refactoring, duplicate policy, and add failure noise.
A test is warranted only when the confidence it provides justifies these costs.
Apply this cost-benefit analysis at two levels:
Evaluate that tradeoff at two levels:
1. **Per test:** What realistic defect does this test detect, how consequential would that defect be, and is that protection worth the test's lifetime cost?
2. **Across the suite:** Does this collection provide materially more confidence than a smaller, simpler suite would?
The preferred test suite is a **lean suite that provides sufficient confidence in the risks that matter, without redundant or low-value tests**. We seek sufficient confidence with the least unnecessary testing friction, not the fewest possible tests.
Some friction is intentional. Tests should make dangerous changes—such as breaking compatibility, corrupting data, violating security boundaries, or reintroducing subtle bugs—require deliberate review. They should not make ordinary internal changes needlessly expensive.
The cost of a test is not a reason to omit testing by default. Do not cite maintenance cost abstractly. When omitting a plausible test, be able to state why the protected failure is low-risk, already covered, obvious, reversible, or cheaper to detect elsewhere. For consequential, subtle, or difficult-to-observe behavior, the presumption should favor testing.
@@ -55,7 +46,7 @@ Control clocks, randomness, environment variables, and other process-global or m
## What deserves tests
Prioritize tests for:
To the extent the following features are present in the application, prioritize tests for:
1. Public and package-level contracts.
2. Domain rules and important invariants.
@@ -67,7 +58,18 @@ Prioritize tests for:
A package-level contract is behavior relied upon by another package or major collaborator, not every observable detail of a package implementation.
For behavior involving **data integrity, destructive operations, compatibility, security, concurrency, idempotency, or recovery**, presume that durable tests are required unless the behavior is already credibly protected at another layer.
Choose tests for requested behavior and established contracts. Tests must not
introduce new restrictions, guarantees, or threat scenarios merely because they
could be asserted. Follow the [agent policy](agents.md) when deciding whether a
behavior belongs to the contract; current implementation behavior alone is not
sufficient authority.
For established behavior involving **data integrity, destructive operations,
compatibility, security, concurrency, idempotency, or recovery**, favor durable
tests for realistic failures whose consequences warrant protection. Identify
the failure and the existing contract it would violate. These categories call
for attention, not an automatic test inventory or permission to strengthen the
application's guarantees. Account for protection already provided at other layers.
Do not add tests merely because a function, branch, or line exists. Do not add a test when the same meaningful risk is already adequately protected elsewhere.
@@ -77,7 +79,7 @@ Test through the narrowest stable boundary that expresses the behavior clearly.
This is often the package API, but it may instead be:
- a smaller pure function when dense application logic is most clearly isolated there;
- a smaller pure function when dense domain logic is most clearly isolated there;
- a package-level operation when several internal collaborators jointly produce the behavior; or
- a larger integration boundary when correctness emerges from interaction with a real dependency.
@@ -85,11 +87,7 @@ Do not force all behavior through oversized end-to-end tests. Do not test every
## Test behavior, not implementation
A test should protect a decision, contract, or invariant—not memorialize the current implementation.
Before adding or retaining a test, ask:
> What realistic defect would this test catch?
A test should protect a decision, contract, or invariant. A test must not memorialize the current implementation.
A test is suspect when its main purpose is to detect that someone:
@@ -102,8 +100,6 @@ A test is suspect when its main purpose is to detect that someone:
Refactoring should normally require no test edits unless the refactored structure is itself part of the contract.
A test can be factually correct and still have negative value. Accurately describing current behavior is not enough; the protected behavior must be important enough to justify the future friction.
## Expected effects of different changes
Use the following expectations when evaluating test failures and test maintenance:
@@ -121,10 +117,11 @@ A test failing is not the same as a test needing to be edited. Many tests may co
Configurable thresholds and defaults must not be duplicated throughout the test suite.
For example, do not encode an internal concurrency limit indirectly:
For this example, assume the selected contract rejects a request to start more
processes than a configured limit. Do not encode its production default indirectly:
```go
// Production policy:
// Production default:
const maxConcurrency = 4
// Brittle test:
@@ -142,18 +139,23 @@ require.NoError(t, runner.Start(limit))
require.ErrorIs(t, runner.Start(limit+1), ErrTooMuchConcurrency)
```
The test should prove:
For this selected contract, the test should prove:
- the configured limit is accepted; and
- one beyond the configured limit is rejected.
Rejection is not a general requirement for concurrency control. If the selected
contract queues excess work or limits active workers while accepting larger
batches, test that behavior instead. Do not add rejection or a configurable
limit solely to reproduce this example.
The production default should be tested exactly only when its literal value is itself a public, operational, safety, protocol, or compatibility requirement.
Apply the same rule to limits, timeouts, capacities, retry counts, and ranges: test relationships and behavior, not duplicated literals.
For concurrency limits, test both kinds of behavior when relevant:
For established concurrency limits, distinguish these behaviors where the contract requires them:
1. **Configuration enforcement:** invalid or excessive requested values are handled correctly.
1. **Configuration handling:** values are accepted or rejected according to the selected contract.
2. **Runtime enforcement:** observed peak concurrency never exceeds the configured limit.
Use a test-controlled limit and measure the behavior relative to that limit. Do not merely assert today's default value.
@@ -162,21 +164,21 @@ Use a test-controlled limit and measure the behavior relative to that limit. Do
Each behavior should have a clear test owner.
- Configuration tests own application YAML, discovery, precedence, and
application defaults.
- Configuration tests own application YAML, strict decoding, discovery,
precedence, and application defaults.
- CLI tests own argument mapping, streams, summaries, exit behavior, and
representative command workflows.
- HTTP tests own DTOs, strict decoding, limits, status mapping, and restricted
artifact policy.
- Formatter tests own prepared-run text and JSON presentation.
- Formatter tests own prepared-run and definition-inspection text and JSON
presentation.
- Architecture tests own dependency direction and removal invariants.
- Promptkit owns framework parsing, orchestration, validation, profiles, and
model-client behavior.
model-client behavior. Scriptorium's definition compatibility fixtures protect
its adapter plumbing without duplicating the upstream format contract.
Higher-level tests should not repeat every lower-level case. A single intentional policy change should not require unrelated edits across many test files.
Tests that are individually reasonable may still be collectively redundant. Evaluate the marginal value of each additional test in light of the protection already provided by the rest of the suite.
## Use test doubles deliberately
Choose the least elaborate test double that provides the required control or observation.
@@ -200,15 +202,16 @@ Do not use mocks merely to isolate every object or reproduce the implementation'
## Go-specific guidance
Use:
Select techniques according to the behavior and risk being tested. The following
are options, not a checklist that every package or application must satisfy:
- table-driven tests for meaningful behavioral categories and boundaries;
- `t.TempDir()` for real filesystem behavior;
- `httptest.Server` for realistic HTTP interactions;
- fuzz tests for parsers, normalization, path handling, and broad input spaces;
- fuzz tests when exploring broad input spaces, such as parsers or path handling, adds useful protection beyond representative cases;
- golden files only when the complete output is intentionally stable;
- integration tests where correctness depends on component interaction; and
- a small number of representative end-to-end tests.
- a small number of representative end-to-end tests when assembled workflows expose meaningful risks not adequately covered at narrower boundaries.
Avoid exact error-string assertions unless the wording is itself contractual. Prefer `errors.Is`, `errors.As`, typed errors, or structured error fields.
@@ -228,8 +231,6 @@ Security-sensitive HTTP containment and external mappings may warrant denser
coverage than straightforward process wiring. Uneven coverage is acceptable
when it reflects risk.
Increasing coverage is valuable only when the newly covered behavior protects a meaningful risk at an acceptable cost.
## Regression tests
A bug fix should normally include a regression test that fails before the fix and passes afterward.
@@ -258,44 +259,26 @@ Strong candidates include tests that:
Several brittle tests may encode one genuine requirement. Replace them with one durable behavior-level or invariant test rather than preserving all of them.
Deleting a low-value test can improve the quality of the suite by reducing noise, maintenance burden, and friction around legitimate change.
## Reviewing a proposed test
Use the following questions when the value, boundary, or durability of a proposed test is not self-evident. Significant test additions should be reviewable against them, but written answers are not required for every routine test.
1. What realistic defect would it catch?
2. How likely is that defect?
3. How consequential would it be?
4. Is the behavior already protected elsewhere?
5. At which layer should this behavior be owned?
6. Does the test assert a durable contract or an incidental implementation detail?
7. Could the implementation be refactored without changing the behavior and without editing this test?
8. What should cause this test to fail?
9. What legitimate changes should not cause this test to fail?
10. What ongoing maintenance, execution, and diagnostic cost will the test impose?
11. Is there a smaller or more direct test that protects the same risk?
Do not add the test when its expected lifetime cost exceeds its expected protective value.
When deciding not to test plausible behavior, record or be able to explain why the risk is low, already protected, obvious, reversible, or cheaper to detect elsewhere.
1. Which requested behavior or established contract does it protect, and what realistic defect would violate it?
2. How likely and consequential is that defect, and is it already protected elsewhere?
3. What is the narrowest stable boundary that can test it clearly?
4. Would a correct internal refactor or a change to an incidental default require test edits?
5. Does its confidence justify its maintenance, execution, and diagnostic cost, or would a smaller test suffice?
## Definition of sufficient
A test suite is sufficient when:
- important contracts and invariants are protected;
- meaningful boundaries and failure modes are exercised;
- realistic and consequential regressions are credibly protected against silent recurrence;
- behavior involving data integrity, destructive operations, compatibility, security, concurrency, idempotency, and recovery is credibly protected;
- important external boundaries have realistic integration coverage;
- representative complete workflows are tested;
- important requested behavior and established contracts are protected against realistic, consequential failures;
- boundaries and component interactions have coverage where they introduce meaningful risks not protected elsewhere;
- failures provide useful signal rather than redundant noise;
- legitimate internal changes usually do not require test edits; and
- additional tests would mostly repeat existing protection or preserve inconsequential implementation details.
Sufficiency is a risk judgment, not a coverage percentage or test count. Reassess it as the application, its users, and the consequences of failure evolve.
The governing rule is:
> Test heavily where failure is consequential, subtle, or difficult to detect after the fact. Test lightly where failure is obvious, reversible, and inexpensive—and retain no test whose lifetime cost exceeds the confidence it provides.
Sufficiency is a risk judgment, not a coverage percentage, test count, or required
mix of testing techniques. Reassess it as the application, its users, and the
consequences of failure evolve.