eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Align architecture documentation and testing policies with current standards

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-25 20:26:34 +00:00
parent cc51b8470a
commit cee254651f
3 changed files with 92 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
docs/policy/architecture.md
View File

@@ -1,11 +1,25 @@
# Architecture
**Base Policy Version:** 1.1
**Last Updated**: 2026-09-25
**Upstream Source:** `policy` commit `45591a79004c6811950f14dca554f6ff69b7f58e`.
Notarius adapts the base template to its fixed pipeline, artifact families,
external adapters, state boundaries, and source-only distribution.
This document defines the intended high-level architecture of Notarius and the
invariants that changes must preserve. Implemented component details belong in
[Internal Overview](../internal/overview.md) and its linked documents. The
reasoning behind significant architectural choices belongs in
[ADRs](../adr/).
An adopted constraint can govern future changes without implying that every
component it could govern exists. Descriptions of implemented structure must
remain accurate; proposals and transition status belong in roadmaps and ADRs.
Constraints reflect selected requirements, not incidental implementation choices.
## System Shape
Notarius is a small, dependency-light Go application for extracting structured
@@ -281,6 +295,13 @@ model, correction, or large payload content.
## State, Output, And Safety
Source material and model responses cross application validation boundaries.
Host access policy belongs to the deployment; hypothetical privileged actors
concurrently changing the filesystem do not justify new application defenses
without an explicit decision. This scope statement does not remove the adopted
correctness boundaries below or claim excluded scenarios cannot occur. Follow
[Agent Policy](agents.md) when a new threat or guarantee needs a decision.
Notarius exposes three filesystem surfaces with independent roots and
lifecycle:

22
docs/policy/documentation.md
View File

@@ -1,5 +1,14 @@
# Documentation Policy
**Base Policy Version:** 1.3
**Last Updated**: 2026-09-25
**Upstream Source:** `policy` commit `45591a79004c6811950f14dca554f6ff69b7f58e`.
Notarius retains subprocess-consumer ownership, the distinction between
descriptive behavior and policy, and no placeholders for absent interfaces.
## Purpose
This policy assigns each documentation topic to one canonical owner. Its goal is
@@ -24,8 +33,12 @@ contract, maintain it only in the owning document.
Outside `docs/roadmap/`, documentation describes implemented behavior only.
Partial features may be described only to their implemented boundary.
ADRs are the narrow exception: an ADR may record an accepted architectural
decision before implementation, but acceptance must not be presented as proof
Architectural constraints may govern future changes without implying that the
components or capabilities they would govern already exist. This does not permit
describing a proposed system structure as implemented.
ADRs may record an accepted architectural decision before implementation, but
acceptance must not be presented as proof
that the behavior exists. The roadmap owns implementation status and sequencing
until the decision is implemented. Current architecture, user, operator,
integration, and internal documentation are updated when the behavior lands.
@@ -71,7 +84,8 @@ secret values.
| --- | --- | --- | --- |
| Product orientation and minimal end-to-end quickstart | `README.md` | What Notarius is, why it is useful, one shortest successful invocation, and links onward. | Complete command reference, configuration reference, operational procedures, implementation detail. |
| Contributor entry point | `docs/development.md` | Task-oriented reading guide, minimal contributor orientation, baseline validation commands, and links to canonical docs. | Package inventory, architecture rules, subsystem behavior, detailed change recipes. |
| Current application architecture | `docs/policy/architecture.md` | System shape, normative ownership, dependency direction, architectural boundaries, invariants, safety properties, and non-goals. | Concrete package inventory, implementation mechanics, contributor procedures, decision history, future work. |
| Application architecture policy | `docs/policy/architecture.md` | System shape, normative ownership, dependency direction, architectural boundaries, invariants, safety properties, and non-goals. | Concrete package inventory, implementation mechanics, contributor procedures, decision history, future work. |
| Agent policy | `docs/policy/agents.md` | Implementation discipline, rules against invented restrictions or guarantees, and local commit authorization. | Application contracts, architecture, testing policy, and contributor procedures. |
| Documentation organization | `docs/policy/documentation.md` | Documentation ownership, audience boundaries, maintenance rules, and ADR/document lifecycle. | Application architecture or product behavior. |
| Testing policy | `docs/policy/testing.md` | Test philosophy, risk-based sufficiency, test boundaries, doubles, coverage guidance, regression-test policy, and criteria for adding, rewriting, or deleting tests. | Subsystem behavior, application contracts, subsystem-specific test inventories, and implementation plans. |
| CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, and exit codes. | End-to-end operating procedures, configuration field definitions, runtime filesystem layout, module implementation details. |
@@ -80,7 +94,7 @@ secret values.
| Source release procedure | `docs/release.md` | Maintainer release selection, candidate validation, tagging, publication guards, verification, and immutable-tag recovery. | Product installation summary, CLI version semantics, historical release summaries, CI implementation detail. |
| Release-note history | `docs/releases/` | One checked-in historical summary for each source release made under the procedure. The note at the immutable tag is that release's record. | Current commands, behavior, contracts, and compatibility definitions. |
| Public HTTP contract, if introduced | `docs/api.md` | Routes, authentication, media types, request and response schemas, status codes, pagination, caching, idempotency, rate limits, and HTTP retry semantics. | Client walkthroughs, upstream or downstream integration internals, implementation detail. |
| Consumer guidance, if a public package or API is introduced | `docs/consumers/` | Task-oriented use of the public interface, minimal client examples, and consumer responsibilities. | HTTP wire semantics, external protocol contracts, internal implementation detail. |
| Consumer guidance | `docs/consumers/` | Task-oriented subprocess use, minimal client examples, consumer responsibilities, and guidance for any future public package or API. | Wire semantics, external protocol contracts, internal implementation detail. |
| External and durable integration contracts | `docs/integrations/` | External file formats and protocols, upstream and downstream contracts, logical output bundle paths and schemas, media types, and compatibility behavior. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, configuration defaults. |
| Implemented component inventory | `docs/internal/overview.md` | Current packages and components, their implemented responsibilities, and links to focused internal docs. | Normative architecture, contributor reading policy, external contracts. |
| Internal component behavior | Other files under `docs/internal/` | Implementation flow, internal collaborators and state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, configuration definitions and defaults, external schemas, operator procedures. |

127
docs/policy/testing.md
View File

@@ -1,5 +1,14 @@
# Testing Policy
**Base Policy Version:** 1.1
**Last Updated**: 2026-09-25
**Upstream Source:** `policy` commit `45591a79004c6811950f14dca554f6ff69b7f58e`.
Adopted without substantive deviations; Notarius's established contracts remain
subject to [Agent Policy](agents.md).
## Purpose
Our tests exist to make **incorrect changes expensive and correct changes cheap**.
@@ -8,29 +17,16 @@ We do not optimize for test count, line coverage, exhaustive isolation, or the f
## Every test has a cost
Testing is not an unqualified good. Every test imposes both an immediate cost and a continuing lifetime cost.
A test must be:
- written and reviewed;
- understood by future maintainers and coding agents;
- executed in local and CI workflows;
- diagnosed when it fails;
- updated when legitimate behavior changes;
- maintained as fixtures, APIs, and dependencies evolve; and
- removed or rewritten when it becomes redundant, brittle, misleading, or obsolete.
Tests also create cognitive and architectural friction. They can constrain refactoring, duplicate policy, slow feedback loops, add noise to failures, and cause harmless implementation changes to require unrelated edits across the suite.
Every test has an immediate and continuing cost: writing, review, execution,
diagnosis, fixture maintenance, and updates when legitimate behavior changes.
Tests can also constrain refactoring, duplicate policy, and add failure noise.
A test is warranted only when the confidence it provides justifies these costs.
Apply this cost-benefit analysis at two levels:
Evaluate that tradeoff at two levels:
1. **Per test:** What realistic defect does this test detect, how consequential would that defect be, and is that protection worth the test's lifetime cost?
2. **Across the suite:** Does this collection provide materially more confidence than a smaller, simpler suite would?
The preferred test suite is a **lean suite that provides sufficient confidence in the risks that matter, without redundant or low-value tests**. We seek sufficient confidence with the least unnecessary testing friction, not the fewest possible tests.
Some friction is intentional. Tests should make dangerous changes—such as breaking compatibility, corrupting data, violating security boundaries, or reintroducing subtle bugs—require deliberate review. They should not make ordinary internal changes needlessly expensive.
The cost of a test is not a reason to omit testing by default. Do not cite maintenance cost abstractly. When omitting a plausible test, be able to state why the protected failure is low-risk, already covered, obvious, reversible, or cheaper to detect elsewhere. For consequential, subtle, or difficult-to-observe behavior, the presumption should favor testing.
@@ -55,7 +51,7 @@ Control clocks, randomness, environment variables, and other process-global or m
## What deserves tests
Prioritize tests for:
To the extent the following features are present in the application, prioritize tests for:
1. Public and package-level contracts.
2. Domain rules and important invariants.
@@ -67,7 +63,18 @@ Prioritize tests for:
A package-level contract is behavior relied upon by another package or major collaborator, not every observable detail of a package implementation.
For behavior involving **data integrity, destructive operations, compatibility, security, concurrency, idempotency, or recovery**, presume that durable tests are required unless the behavior is already credibly protected at another layer.
Choose tests for requested behavior and established contracts. Tests must not
introduce new restrictions, guarantees, or threat scenarios merely because they
could be asserted. Follow the [agent policy](agents.md) when deciding whether a
behavior belongs to the contract; current implementation behavior alone is not
sufficient authority.
For established behavior involving **data integrity, destructive operations,
compatibility, security, concurrency, idempotency, or recovery**, favor durable
tests for realistic failures whose consequences warrant protection. Identify
the failure and the existing contract it would violate. These categories call
for attention, not an automatic test inventory or permission to strengthen the
application's guarantees. Account for protection already provided at other layers.
Do not add tests merely because a function, branch, or line exists. Do not add a test when the same meaningful risk is already adequately protected elsewhere.
@@ -85,18 +92,7 @@ Do not force all behavior through oversized end-to-end tests. Do not test every
## Test behavior, not implementation
A test should protect a decision, contract, or invariant—not memorialize the current implementation.
Tests must not create a stronger product contract than the requirement they
protect. In particular, do not turn an incidental limit, restriction,
permission choice, validation rule, or implementation default into policy by
asserting it without an expressly selected contract. Follow the
[implementation-discipline rules](agents.md#implementation-discipline)
when deciding what behavior deserves durable protection.
Before adding or retaining a test, ask:
> What realistic defect would this test catch?
A test should protect a decision, contract, or invariant. A test must not memorialize the current implementation.
A test is suspect when its main purpose is to detect that someone:
@@ -109,8 +105,6 @@ A test is suspect when its main purpose is to detect that someone:
Refactoring should normally require no test edits unless the refactored structure is itself part of the contract.
A test can be factually correct and still have negative value. Accurately describing current behavior is not enough; the protected behavior must be important enough to justify the future friction.
## Expected effects of different changes
Use the following expectations when evaluating test failures and test maintenance:
@@ -128,10 +122,11 @@ A test failing is not the same as a test needing to be edited. Many tests may co
Configurable thresholds and defaults must not be duplicated throughout the test suite.
For example, do not encode an internal concurrency limit indirectly:
For this example, assume the selected contract rejects a request to start more
processes than a configured limit. Do not encode its production default indirectly:
```go
// Production policy:
// Production default:
const maxConcurrency = 4
// Brittle test:
@@ -149,25 +144,30 @@ require.NoError(t, runner.Start(limit))
require.ErrorIs(t, runner.Start(limit+1), ErrTooMuchConcurrency)
```
The test should prove:
For this selected contract, the test should prove:
- the configured limit is accepted; and
- one beyond the configured limit is rejected.
Rejection is not a general requirement for concurrency control. If the selected
contract queues excess work or limits active workers while accepting larger
batches, test that behavior instead. Do not add rejection or a configurable
limit solely to reproduce this example.
The production default should be tested exactly only when its literal value is itself a public, operational, safety, protocol, or compatibility requirement.
Apply the same rule to limits, timeouts, capacities, retry counts, and ranges: test relationships and behavior, not duplicated literals.
For concurrency limits, test both kinds of behavior when relevant:
For established concurrency limits, distinguish these behaviors where the contract requires them:
1. **Configuration enforcement:** invalid or excessive requested values are handled correctly.
1. **Configuration handling:** values are accepted or rejected according to the selected contract.
2. **Runtime enforcement:** observed peak concurrency never exceeds the configured limit.
Use a test-controlled limit and measure the behavior relative to that limit. Do not merely assert today's default value.
## Avoid semantic duplication across layers
Each behavior should have a clear test owner.
Each behavior should have a clear test owner. Where these layers exist:
- Parser tests own parsing cases.
- Validator tests own validation rules.
@@ -179,8 +179,6 @@ Each behavior should have a clear test owner.
Higher-level tests should not repeat every lower-level case. A single intentional policy change should not require unrelated edits across many test files.
Tests that are individually reasonable may still be collectively redundant. Evaluate the marginal value of each additional test in light of the protection already provided by the rest of the suite.
## Use test doubles deliberately
Choose the least elaborate test double that provides the required control or observation.
@@ -204,15 +202,16 @@ Do not use mocks merely to isolate every object or reproduce the implementation'
## Go-specific guidance
Use:
Select techniques according to the behavior and risk being tested. The following
are options, not a checklist that every package or application must satisfy:
- table-driven tests for meaningful behavioral categories and boundaries;
- `t.TempDir()` for real filesystem behavior;
- `httptest.Server` for realistic HTTP interactions;
- fuzz tests for parsers, normalization, path handling, and broad input spaces;
- fuzz tests when exploring broad input spaces, such as parsers or path handling, adds useful protection beyond representative cases;
- golden files only when the complete output is intentionally stable;
- integration tests where correctness depends on component interaction; and
- a small number of representative end-to-end tests.
- a small number of representative end-to-end tests when assembled workflows expose meaningful risks not adequately covered at narrower boundaries.
Avoid exact error-string assertions unless the wording is itself contractual. Prefer `errors.Is`, `errors.As`, typed errors, or structured error fields.
@@ -230,8 +229,6 @@ Use it to find untested critical branches and unexpectedly weak packages. Do not
Pure domain logic will often warrant higher coverage than CLI wiring or external adapters. Uneven coverage is acceptable when it reflects risk.
Increasing coverage is valuable only when the newly covered behavior protects a meaningful risk at an acceptable cost.
## Regression tests
A bug fix should normally include a regression test that fails before the fix and passes afterward.
@@ -260,44 +257,26 @@ Strong candidates include tests that:
Several brittle tests may encode one genuine requirement. Replace them with one durable behavior-level or invariant test rather than preserving all of them.
Deleting a low-value test can improve the quality of the suite by reducing noise, maintenance burden, and friction around legitimate change.
## Reviewing a proposed test
Use the following questions when the value, boundary, or durability of a proposed test is not self-evident. Significant test additions should be reviewable against them, but written answers are not required for every routine test.
1. What realistic defect would it catch?
2. How likely is that defect?
3. How consequential would it be?
4. Is the behavior already protected elsewhere?
5. At which layer should this behavior be owned?
6. Does the test assert a durable contract or an incidental implementation detail?
7. Could the implementation be refactored without changing the behavior and without editing this test?
8. What should cause this test to fail?
9. What legitimate changes should not cause this test to fail?
10. What ongoing maintenance, execution, and diagnostic cost will the test impose?
11. Is there a smaller or more direct test that protects the same risk?
Do not add the test when its expected lifetime cost exceeds its expected protective value.
When deciding not to test plausible behavior, record or be able to explain why the risk is low, already protected, obvious, reversible, or cheaper to detect elsewhere.
1. Which requested behavior or established contract does it protect, and what realistic defect would violate it?
2. How likely and consequential is that defect, and is it already protected elsewhere?
3. What is the narrowest stable boundary that can test it clearly?
4. Would a correct internal refactor or a change to an incidental default require test edits?
5. Does its confidence justify its maintenance, execution, and diagnostic cost, or would a smaller test suffice?
## Definition of sufficient
A test suite is sufficient when:
- important contracts and invariants are protected;
- meaningful boundaries and failure modes are exercised;
- realistic and consequential regressions are credibly protected against silent recurrence;
- behavior involving data integrity, destructive operations, compatibility, security, concurrency, idempotency, and recovery is credibly protected;
- important external boundaries have realistic integration coverage;
- representative complete workflows are tested;
- important requested behavior and established contracts are protected against realistic, consequential failures;
- boundaries and component interactions have coverage where they introduce meaningful risks not protected elsewhere;
- failures provide useful signal rather than redundant noise;
- legitimate internal changes usually do not require test edits; and
- additional tests would mostly repeat existing protection or preserve inconsequential implementation details.
Sufficiency is a risk judgment, not a coverage percentage or test count. Reassess it as the application, its users, and the consequences of failure evolve.
The governing rule is:
> Test heavily where failure is consequential, subtle, or difficult to detect after the fact. Test lightly where failure is obvious, reversible, and inexpensive—and retain no test whose lifetime cost exceeds the confidence it provides.
Sufficiency is a risk judgment, not a coverage percentage, test count, or required
mix of testing techniques. Reassess it as the application, its users, and the
consequences of failure evolve.