ansible-workspace-dnd/docs/roadmap/test-suite-audit.md

# Test Suite Audit Plan

Status: complete

## Objective

Audit the implemented integration tests for the three completed standalone
roles against the workspace's classical, state-based testing policy. The audit
will identify tests that protect stable role-owned behavior, tests that should
be simplified or replaced, and tests whose maintenance cost exceeds their
value.

This is a review plan, not authorization to change role code or test suites.
Any proposed remediation will be reviewed before a separate implementation
plan is prepared.

## Scope

The audit covers the test harnesses, playbooks, fixtures, inventories, and
test-related documentation in:

- `ansible-role-seriatim`
- `ansible-role-audita`
- `ansible-role-notarius`

Production role tasks, defaults, templates, and metadata may be inspected when
needed to determine whether a test protects public role behavior or merely
repeats an implementation detail. They are not remediation targets during the
audit.

The audit does not cover the proof-of-concept `rakestraw-ansible` role, the
suite-level role, or the collection. Those projects do not form part of the
three completed standalone-role test suites.

## Governing Standard

Apply the normative testing house style in `docs/testing.md`. In
particular, the audit will favor tests that exercise the role through its
public variables and inspect meaningful final application behavior. It will
disfavor tests that:

- freeze current defaults or other intentionally changeable values;
- assert task names, order, individual task status, or other execution traces;
- independently re-prove the documented postconditions of trusted Ansible
  modules;
- exhaustively exercise self-evident assertions or equivalent low-value
  combinations; or
- duplicate another test without protecting a distinct requirement or
  plausible regression.

An immediate second convergence with no reported changes remains required for
each role. An explicitly selected version may be verified through the
application's native interface; the role's current default version may not be
treated as a permanent expected value.

## Classification Rubric

Classify every behavioral test case, including each materially distinct case
embedded in a shared playbook or shell harness, as one of the following:

- **Retain:** protects stable, role-owned behavior through an externally
  meaningful outcome.
- **Simplify:** protects useful behavior but includes redundant assertions,
  duplicated setup, or implementation coupling that can be removed.
- **Replace:** addresses a valid requirement through an unstable or indirect
  assertion and should be expressed as a state-based outcome instead.
- **Remove:** tests a mutable default, trusted module behavior, an
  implementation trace, or behavior without a stated requirement or plausible
  regression.
- **Add:** identifies a material role-owned behavior that currently lacks
  representative coverage.

Each classification must state the behavior at issue and the reason for the
recommended action. File length, test count, and aesthetic preference are not
sufficient reasons by themselves.

## Audit Record

Write the completed review to
`docs/roadmap/test-suite-audit-findings.md`. Record each finding in a table with
these fields:

| Repository | Test or case | Behavior protected | Classification | Recommended action | Rationale |
| --- | --- | --- | --- | --- | --- |

The report must also provide a proposed lean target suite for each role and
identify any shared harness behavior that should be made consistent across the
three repositories. Recommendations to remove coverage must state whether the
behavior remains covered elsewhere or why it does not warrant coverage.

## Stage 1: Establish the Baseline

1. Confirm that the workspace and all three role repositories begin clean.
2. Inventory every test entry point, playbook, fixture, inventory, helper, and
   test-related README statement in scope.
3. Run each existing integration entry point without modification:

   ```text
   (cd ansible-role-seriatim && tests/integration)
   (cd ansible-role-audita && tests/integration)
   (cd ansible-role-notarius && tests/integration)
   ```

4. Record success or failure, the scenarios exercised, and any environmental
   assumptions needed to reproduce the result. A pre-existing failure is audit
   evidence, not permission to repair it during this stage.

Completion criterion: the report contains a reproducible baseline and a
complete test inventory for all three roles.

## Stage 2: Audit Seriatim

1. Map every Seriatim test case to a documented public behavior or classify it
   as lacking one.
2. Review version selection, caller-supplied build tooling, rebuild behavior,
   metadata handling, platform validation, and repeat convergence using the
   common rubric.
3. Identify assertions that inspect Ansible execution details or separately
   verify state already owned by trusted modules.
4. Propose the smallest representative Seriatim suite that preserves valuable
   regression protection.

Completion criterion: every Seriatim case is classified and the report states
the proposed retained, revised, removed, and missing coverage.

## Stage 3: Audit Audita

1. Map every Audita test case to a documented public behavior or classify it
   as lacking one.
2. Review explicit version selection, native application verification,
   configuration rendering, omission of null settings, preservation of falsey
   values, platform validation, and repeat convergence.
3. Distinguish tests of the role's configuration decisions from tests that
   merely repeat `template`, `file`, `git`, or package-module guarantees.
4. Propose the smallest representative Audita suite that preserves valuable
   regression protection.

Completion criterion: every Audita case is classified and the report states
the proposed retained, revised, removed, and missing coverage.

## Stage 4: Audit Notarius

1. Map every Notarius test case to a documented public behavior or classify it
   as lacking one.
2. Review explicit version selection, native application verification,
   optional configuration rendering, falsey values, caller-composed pipeline
   mappings, platform validation, and repeat convergence.
3. Distinguish tests of role-authored mapping and rendering behavior from
   assertions about trusted module mechanics or the current implementation
   layout.
4. Propose the smallest representative Notarius suite that preserves valuable
   regression protection.

Completion criterion: every Notarius case is classified and the report states
the proposed retained, revised, removed, and missing coverage.

## Stage 5: Reconcile the Three Suites

1. Compare classifications across roles so equivalent patterns receive
   equivalent treatment, while preserving genuine application differences.
2. Identify duplicated harness machinery, inconsistent terminology, and
   documentation that describes implementation details or mutable defaults as
   promised test behavior.
3. Separate static validation such as syntax checking and linting from
   behavioral integration coverage.
4. Produce a repository-by-repository remediation proposal ordered by value
   and dependency. Do not propose a shared abstraction unless it removes
   demonstrated duplication without obscuring the individual role suites.
5. Confirm that the proposed target for every role still includes a default or
   representative convergence, an immediate clean second convergence, and
   focused coverage of each retained role-owned behavior.

Completion criterion: the findings report is internally consistent, gives a
lean target suite for each repository, and contains enough detail to support a
later implementation plan without leaving test-policy decisions to the
executor.

## Audit Completion Criteria

The audit is complete when:

- every implemented behavioral case across all three repositories has an
  explicit classification and rationale;
- baseline results and reproduction commands are recorded;
- proposed removals do not silently discard coverage of a required public
  behavior;
- proposed additions are limited to material role-owned behavior;
- README and harness-documentation corrections are identified alongside test
  changes;
- no production or test implementation has been changed during the audit; and
- the findings have been reviewed before any remediation plan is written to
  `docs/roadmap/implementation.md`.