ansible-workspace-dnd/docs/roadmap/test-suite-audit.md

Test Suite Audit Plan

Status: complete

Objective

Audit the implemented integration tests for the three completed standalone roles against the workspace's classical, state-based testing policy. The audit will identify tests that protect stable role-owned behavior, tests that should be simplified or replaced, and tests whose maintenance cost exceeds their value.

This is a review plan, not authorization to change role code or test suites. Any proposed remediation will be reviewed before a separate implementation plan is prepared.

Scope

The audit covers the test harnesses, playbooks, fixtures, inventories, and test-related documentation in:

• ansible-role-seriatim
• ansible-role-audita
• ansible-role-notarius

Production role tasks, defaults, templates, and metadata may be inspected when needed to determine whether a test protects public role behavior or merely repeats an implementation detail. They are not remediation targets during the audit.

The audit does not cover the proof-of-concept rakestraw-ansible role, the suite-level role, or the collection. Those projects do not form part of the three completed standalone-role test suites.

Governing Standard

Apply the normative testing house style in docs/testing.md. In particular, the audit will favor tests that exercise the role through its public variables and inspect meaningful final application behavior. It will disfavor tests that:

• freeze current defaults or other intentionally changeable values;
• assert task names, order, individual task status, or other execution traces;
• independently re-prove the documented postconditions of trusted Ansible modules;
• exhaustively exercise self-evident assertions or equivalent low-value combinations; or
• duplicate another test without protecting a distinct requirement or plausible regression.

An immediate second convergence with no reported changes remains required for each role. An explicitly selected version may be verified through the application's native interface; the role's current default version may not be treated as a permanent expected value.

Classification Rubric

Classify every behavioral test case, including each materially distinct case embedded in a shared playbook or shell harness, as one of the following:

• Retain: protects stable, role-owned behavior through an externally meaningful outcome.
• Simplify: protects useful behavior but includes redundant assertions, duplicated setup, or implementation coupling that can be removed.
• Replace: addresses a valid requirement through an unstable or indirect assertion and should be expressed as a state-based outcome instead.
• Remove: tests a mutable default, trusted module behavior, an implementation trace, or behavior without a stated requirement or plausible regression.
• Add: identifies a material role-owned behavior that currently lacks representative coverage.

Each classification must state the behavior at issue and the reason for the recommended action. File length, test count, and aesthetic preference are not sufficient reasons by themselves.

Audit Record

Write the completed review to docs/roadmap/test-suite-audit-findings.md. Record each finding in a table with these fields:

Repository	Test or case	Behavior protected	Classification	Recommended action	Rationale

The report must also provide a proposed lean target suite for each role and identify any shared harness behavior that should be made consistent across the three repositories. Recommendations to remove coverage must state whether the behavior remains covered elsewhere or why it does not warrant coverage.

Stage 1: Establish the Baseline

1. Confirm that the workspace and all three role repositories begin clean.

2. Inventory every test entry point, playbook, fixture, inventory, helper, and test-related README statement in scope.

3. Run each existing integration entry point without modification:

   (cd ansible-role-seriatim && tests/integration)
   (cd ansible-role-audita && tests/integration)
   (cd ansible-role-notarius && tests/integration)
   

4. Record success or failure, the scenarios exercised, and any environmental assumptions needed to reproduce the result. A pre-existing failure is audit evidence, not permission to repair it during this stage.

Completion criterion: the report contains a reproducible baseline and a complete test inventory for all three roles.

Stage 2: Audit Seriatim

1. Map every Seriatim test case to a documented public behavior or classify it as lacking one.
2. Review version selection, caller-supplied build tooling, rebuild behavior, metadata handling, platform validation, and repeat convergence using the common rubric.
3. Identify assertions that inspect Ansible execution details or separately verify state already owned by trusted modules.
4. Propose the smallest representative Seriatim suite that preserves valuable regression protection.

Completion criterion: every Seriatim case is classified and the report states the proposed retained, revised, removed, and missing coverage.

Stage 3: Audit Audita

1. Map every Audita test case to a documented public behavior or classify it as lacking one.
2. Review explicit version selection, native application verification, configuration rendering, omission of null settings, preservation of falsey values, platform validation, and repeat convergence.
3. Distinguish tests of the role's configuration decisions from tests that merely repeat template, file, git, or package-module guarantees.
4. Propose the smallest representative Audita suite that preserves valuable regression protection.

Completion criterion: every Audita case is classified and the report states the proposed retained, revised, removed, and missing coverage.

Stage 4: Audit Notarius

1. Map every Notarius test case to a documented public behavior or classify it as lacking one.
2. Review explicit version selection, native application verification, optional configuration rendering, falsey values, caller-composed pipeline mappings, platform validation, and repeat convergence.
3. Distinguish tests of role-authored mapping and rendering behavior from assertions about trusted module mechanics or the current implementation layout.
4. Propose the smallest representative Notarius suite that preserves valuable regression protection.

Completion criterion: every Notarius case is classified and the report states the proposed retained, revised, removed, and missing coverage.

Stage 5: Reconcile the Three Suites

1. Compare classifications across roles so equivalent patterns receive equivalent treatment, while preserving genuine application differences.
2. Identify duplicated harness machinery, inconsistent terminology, and documentation that describes implementation details or mutable defaults as promised test behavior.
3. Separate static validation such as syntax checking and linting from behavioral integration coverage.
4. Produce a repository-by-repository remediation proposal ordered by value and dependency. Do not propose a shared abstraction unless it removes demonstrated duplication without obscuring the individual role suites.
5. Confirm that the proposed target for every role still includes a default or representative convergence, an immediate clean second convergence, and focused coverage of each retained role-owned behavior.

Completion criterion: the findings report is internally consistent, gives a lean target suite for each repository, and contains enough detail to support a later implementation plan without leaving test-policy decisions to the executor.

Audit Completion Criteria

The audit is complete when:

• every implemented behavioral case across all three repositories has an explicit classification and rationale;
• baseline results and reproduction commands are recorded;
• proposed removals do not silently discard coverage of a required public behavior;
• proposed additions are limited to material role-owned behavior;
• README and harness-documentation corrections are identified alongside test changes;
• no production or test implementation has been changed during the audit; and
• the findings have been reviewed before any remediation plan is written to docs/roadmap/implementation.md.