eric/ansible-workspace-dnd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan classical test suite remediation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-09 04:19:51 +00:00
parent 4177b4de81
commit 482e8473c2
5 changed files with 1008 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

192
docs/roadmap/test-suite-audit.md Normal file
View File

@@ -0,0 +1,192 @@
# Test Suite Audit Plan
Status: complete
## Objective
Audit the implemented integration tests for the three completed standalone
roles against the workspace's classical, state-based testing policy. The audit
will identify tests that protect stable role-owned behavior, tests that should
be simplified or replaced, and tests whose maintenance cost exceeds their
value.
This is a review plan, not authorization to change role code or test suites.
Any proposed remediation will be reviewed before a separate implementation
plan is prepared.
## Scope
The audit covers the test harnesses, playbooks, fixtures, inventories, and
test-related documentation in:
- `ansible-role-seriatim`
- `ansible-role-audita`
- `ansible-role-notarius`
Production role tasks, defaults, templates, and metadata may be inspected when
needed to determine whether a test protects public role behavior or merely
repeats an implementation detail. They are not remediation targets during the
audit.
The audit does not cover the proof-of-concept `rakestraw-ansible` role, the
suite-level role, or the collection. Those projects do not form part of the
three completed standalone-role test suites.
## Governing Standard
Apply the normative testing house style in `docs/conventions.md`. In
particular, the audit will favor tests that exercise the role through its
public variables and inspect meaningful final application behavior. It will
disfavor tests that:
- freeze current defaults or other intentionally changeable values;
- assert task names, order, individual task status, or other execution traces;
- independently re-prove the documented postconditions of trusted Ansible
modules;
- exhaustively exercise self-evident assertions or equivalent low-value
combinations; or
- duplicate another test without protecting a distinct requirement or
plausible regression.
An immediate second convergence with no reported changes remains required for
each role. An explicitly selected version may be verified through the
application's native interface; the role's current default version may not be
treated as a permanent expected value.
## Classification Rubric
Classify every behavioral test case, including each materially distinct case
embedded in a shared playbook or shell harness, as one of the following:
- **Retain:** protects stable, role-owned behavior through an externally
meaningful outcome.
- **Simplify:** protects useful behavior but includes redundant assertions,
duplicated setup, or implementation coupling that can be removed.
- **Replace:** addresses a valid requirement through an unstable or indirect
assertion and should be expressed as a state-based outcome instead.
- **Remove:** tests a mutable default, trusted module behavior, an
implementation trace, or behavior without a stated requirement or plausible
regression.
- **Add:** identifies a material role-owned behavior that currently lacks
representative coverage.
Each classification must state the behavior at issue and the reason for the
recommended action. File length, test count, and aesthetic preference are not
sufficient reasons by themselves.
## Audit Record
Write the completed review to
`docs/roadmap/test-suite-audit-findings.md`. Record each finding in a table with
these fields:
| Repository | Test or case | Behavior protected | Classification | Recommended action | Rationale |
| --- | --- | --- | --- | --- | --- |
The report must also provide a proposed lean target suite for each role and
identify any shared harness behavior that should be made consistent across the
three repositories. Recommendations to remove coverage must state whether the
behavior remains covered elsewhere or why it does not warrant coverage.
## Stage 1: Establish the Baseline
1. Confirm that the workspace and all three role repositories begin clean.
2. Inventory every test entry point, playbook, fixture, inventory, helper, and
test-related README statement in scope.
3. Run each existing integration entry point without modification:
```text
(cd ansible-role-seriatim && tests/integration)
(cd ansible-role-audita && tests/integration)
(cd ansible-role-notarius && tests/integration)
```
4. Record success or failure, the scenarios exercised, and any environmental
assumptions needed to reproduce the result. A pre-existing failure is audit
evidence, not permission to repair it during this stage.
Completion criterion: the report contains a reproducible baseline and a
complete test inventory for all three roles.
## Stage 2: Audit Seriatim
1. Map every Seriatim test case to a documented public behavior or classify it
as lacking one.
2. Review version selection, caller-supplied build tooling, rebuild behavior,
metadata handling, platform validation, and repeat convergence using the
common rubric.
3. Identify assertions that inspect Ansible execution details or separately
verify state already owned by trusted modules.
4. Propose the smallest representative Seriatim suite that preserves valuable
regression protection.
Completion criterion: every Seriatim case is classified and the report states
the proposed retained, revised, removed, and missing coverage.
## Stage 3: Audit Audita
1. Map every Audita test case to a documented public behavior or classify it
as lacking one.
2. Review explicit version selection, native application verification,
configuration rendering, omission of null settings, preservation of falsey
values, platform validation, and repeat convergence.
3. Distinguish tests of the role's configuration decisions from tests that
merely repeat `template`, `file`, `git`, or package-module guarantees.
4. Propose the smallest representative Audita suite that preserves valuable
regression protection.
Completion criterion: every Audita case is classified and the report states
the proposed retained, revised, removed, and missing coverage.
## Stage 4: Audit Notarius
1. Map every Notarius test case to a documented public behavior or classify it
as lacking one.
2. Review explicit version selection, native application verification,
optional configuration rendering, falsey values, caller-composed pipeline
mappings, platform validation, and repeat convergence.
3. Distinguish tests of role-authored mapping and rendering behavior from
assertions about trusted module mechanics or the current implementation
layout.
4. Propose the smallest representative Notarius suite that preserves valuable
regression protection.
Completion criterion: every Notarius case is classified and the report states
the proposed retained, revised, removed, and missing coverage.
## Stage 5: Reconcile the Three Suites
1. Compare classifications across roles so equivalent patterns receive
equivalent treatment, while preserving genuine application differences.
2. Identify duplicated harness machinery, inconsistent terminology, and
documentation that describes implementation details or mutable defaults as
promised test behavior.
3. Separate static validation such as syntax checking and linting from
behavioral integration coverage.
4. Produce a repository-by-repository remediation proposal ordered by value
and dependency. Do not propose a shared abstraction unless it removes
demonstrated duplication without obscuring the individual role suites.
5. Confirm that the proposed target for every role still includes a default or
representative convergence, an immediate clean second convergence, and
focused coverage of each retained role-owned behavior.
Completion criterion: the findings report is internally consistent, gives a
lean target suite for each repository, and contains enough detail to support a
later implementation plan without leaving test-policy decisions to the
executor.
## Audit Completion Criteria
The audit is complete when:
- every implemented behavioral case across all three repositories has an
explicit classification and rationale;
- baseline results and reproduction commands are recorded;
- proposed removals do not silently discard coverage of a required public
behavior;
- proposed additions are limited to material role-owned behavior;
- README and harness-documentation corrections are identified alongside test
changes;
- no production or test implementation has been changed during the audit; and
- the findings have been reviewed before any remediation plan is written to
`docs/roadmap/implementation.md`.