ansible-workspace-dnd/docs/roadmap/test-suite-audit-findings.md

Test Suite Audit Findings

Status: resolved

Executive Summary

All three integration suites pass, and each includes the most important common behavioral check: an immediate repeated role application reports no changes. The suites also contain useful coverage of explicitly selected releases, caller-supplied Go executables, missing-binary recovery, configuration rendering, native configuration validation, and Notarius pipeline composition.

The suites are nevertheless substantially larger and more implementation-bound than the workspace testing policy permits. The recurring problems are:

• exact assertions about the current default release and its commit;
• independent verification of ownership, permissions, checkout state, and template output already managed by trusted Ansible modules;
• parsing Ansible output for specific task names and individual changed/skipped results;
• deliberate metadata drift followed by tests of ansible.builtin.file;
• three-case negative tests of self-evident platform assertions;
• repeated clean-run assertions after nearly every scenario; and
• duplicated final shell assertions after equivalent Ansible verifiers.

The appropriate remediation is subtraction and consolidation. The retained suites should exercise stable role-owned behavior through final application outcomes, with one representative case for each meaningful behavior.

Baseline

The audit ran on Ubuntu 26.04 x86_64 with Ansible Core 2.21.3 and ansible-lint 26.8.0. The role revisions were:

Repository	Revision	Integration result
ansible-role-seriatim	fb8dc41	Pass
ansible-role-audita	847c4c3	Pass
ansible-role-notarius	eb4f511	Pass

The following commands all exited successfully:

(cd ansible-role-seriatim && tests/integration)
(cd ansible-role-audita && tests/integration)
(cd ansible-role-notarius && tests/integration)

Each driver installed its pinned collection requirements into a private temporary directory, ran a syntax check and production-profile ansible-lint, executed its behavioral scenarios, and restored its default deployment during cleanup. The runs emitted non-fatal local-tooling warnings about Python interpreter discovery or an already installed duplicate collection; these did not affect the results or any classification below.

All three role repositories were clean before and after the audit. The control repository began dirty solely because the new testing convention, audit plan, and status update were awaiting commit.

Test Inventory

Repository	Entry point and supporting files
ansible-role-seriatim	tests/integration, tests/inventory, tests/converge.yml, tests/verify.yml, tests/unsupported.yml, collections/requirements.yml, and the README validation section
ansible-role-audita	tests/integration, tests/inventory, tests/converge.yml, tests/verify.yml, tests/verify-config.yml, tests/unsupported.yml, collections/requirements.yml, and the README validation section
ansible-role-notarius	tests/integration, tests/inventory, tests/converge.yml, tests/verify.yml, tests/verify-config.yml, tests/unsupported.yml, collections/requirements.yml, and the README validation section

Cross-Repository Findings

Repository	Test or case	Behavior protected	Classification	Recommended action	Rationale
All three	Pinned collection installation, syntax check, and ansible-lint	Static validity and reproducible dependencies	Retain	Keep as static validation before behavioral scenarios.	These checks complement rather than duplicate behavioral testing.
All three	require_clean_recap after an immediate repeat	Role-level repeated convergence	Retain	Keep at least one immediate clean repeat per role and repeats specifically required after missing-binary recovery.	Overall changed=0 is a first-class public role outcome.
All three	require_changed_recap after each mutating setup	Ansible reports some change on the first run	Remove	Stop asserting a nonzero aggregate change count except where a future requirement specifically makes reporting itself significant.	Final application state is the behavior of interest; these checks add little beyond successful convergence and make scenario sequencing brittle.
All three	Greps for named build, metadata, and configuration tasks	Current task names and individual changed or skipped results	Remove	Delete all task-output parsing.	These are direct implementation traces and fail under behavior-preserving task refactors.
All three	Binary/source/configuration ownership and mode assertions	Postconditions declared to file, git, and template	Remove	Delete independent metadata and recursive-ownership verification.	These tests re-prove trusted module behavior rather than role-authored decisions.
All three	Binary mode mutation followed by convergence	ansible.builtin.file repairs mode drift without building	Remove	Delete the scenario.	It is a focused regression test for the built-in file module and current task arrangement.
All three	Debian, old Ubuntu, and unsupported-architecture matrix	Straightforward platform assertion branches	Remove	Delete tests/unsupported.yml and the three-case driver loop.	The declarative assertion is self-evident and does not warrant a negative matrix.
All three	Restoration of the current default followed by verification and another repeat	Test cleanup and the current default identity	Simplify	Keep one best-effort cleanup convergence, but do not treat it as another test or assert the restored release value.	Cleanup is useful; testing a mutable default and repeating already established idempotence are not.
All three	Final shell assertions duplicating earlier verifiers	Current commit, metadata, version, or config state	Remove	Delete duplicate end-of-driver assertions.	They add maintenance cost without protecting a distinct behavior.

Seriatim Classification

Repository	Test or case	Behavior protected	Classification	Recommended action	Rationale
Seriatim	Default convergence	The role can be applied with its default interface	Retain	Keep as a smoke test without asserting the exact default tag, commit, or filesystem metadata.	A default invocation is useful even though the present default value is not a stable test expectation.
Seriatim	Default v1.5.0 and commit verification	Present default release identity	Remove	Remove the fixed default_commit and exact default-version assertions.	This is a change detector; advancing the supported default should not break unrelated tests.
Seriatim	Baseline immediate repeat	Clean convergence	Retain	Keep the overall changed=0, failed=0 check.	This directly expresses the agreed idempotence requirement.
Seriatim	Missing binary	The imperative build runs when its output is absent and then converges	Simplify	Delete the binary, apply the role, verify the executable through --version, and require the next run to be clean; remove task-name greps.	The binary-existence trigger is role-authored behavior, while the current task name and status are not.
Seriatim	Dependency opt-out and custom Go executable	The caller-supplied build executable is honored	Simplify	Retain the wrapper invocation evidence, preferably combined with the missing-binary scenario; remove build-task greps and fixed-default assertions.	The selected build executable is a public role choice that cannot be observed reliably from the final binary alone.
Seriatim	Binary metadata repair	Built-in file convergence	Remove	Delete the complete scenario.	It tests ansible.builtin.file, not Seriatim-specific behavior.
Seriatim	Explicit v1.4.0 selection	A caller-selected immutable release is installed	Simplify	Retain explicit seriatim_version: v1.4.0 and verify seriatim version v1.4.0; remove the expected commit and task-state assertions.	This is the stable version-selection test described by the testing convention.
Seriatim	Restore v1.5.0 default	Cleanup plus present default identity	Simplify	Reduce to cleanup only.	Exact current-default verification duplicates the harmful baseline snapshot.
Seriatim	Platform failure matrix	Distribution, release, and architecture assertions	Remove	Delete the matrix and tests/unsupported.yml.	No plausible role regression justifies separately testing each simple assertion.
Seriatim	tests/verify.yml installation contract	Commit, source metadata, binary metadata, and version	Replace	Replace the broad verifier with a narrow native version check used for an explicitly selected tag.	Only the application outcome is valuable; the remaining assertions repeat trusted modules.
Seriatim	README validation claims	Description of the current large matrix	Replace	Describe the lean retained behaviors after remediation.	Documentation should not promise deleted implementation-trace and module tests.

Audita Classification

Repository	Test or case	Behavior protected	Classification	Recommended action	Rationale
Audita	Default convergence	The role installs and configures successfully through its default interface	Retain	Keep as a smoke test and validate the resulting configuration with Audita; do not inspect exact default identity or metadata.	Successful native validation is an application outcome and does not freeze the default document.
Audita	Default commit assertion	Present default checkout identity	Remove	Delete default_commit and exact commit verification.	It is both a mutable-default snapshot and a re-test of ansible.builtin.git.
Audita	Source, binary, and configuration metadata	Trusted module postconditions	Remove	Delete the ownership, permission, regular-file, and executable assertions.	The role directly declares these states with trusted modules.
Audita	Exact minimal default document {"version": 1}	Current default rendering	Remove	Do not snapshot the complete default document. Test null omission with explicitly supplied inputs in a representative configuration case instead.	A future intentional concrete default would otherwise cause an unrelated failure.
Audita	Native validation of the managed configuration	The role produces configuration Audita accepts	Retain	Keep one native validation for default smoke coverage and one for the representative explicit configuration.	This observes application behavior rather than reconstructing its schema.
Audita	Baseline immediate repeat	Clean convergence	Retain	Keep one role-level clean repeat.	This is required public behavior.
Audita	Missing binary	Absent build output is recreated and convergence resumes	Simplify	Retain recovery and the following clean repeat; prove usability with the Audita CLI and remove task-output greps.	The imperative build trigger is role logic, but its task name is not.
Audita	Dependency opt-out and custom Go executable	Caller-selected compiler path is used	Simplify	Retain wrapper invocation evidence and combine it with missing-binary recovery; remove task greps and redundant repeat runs.	This covers a documented public choice with one representative case.
Audita	Binary metadata repair	Built-in file convergence	Remove	Delete the complete scenario.	It tests Ansible's file module.
Audita	Representative configuration containing nearly every field	Field-to-YAML mapping, nested sections, and native validity	Simplify	Use a smaller explicit sample spanning representative scalar, list, nested, and duration shapes. Assert only selected paths and specifically selected omissions, then run native validation.	Full-document equality across nearly every field unnecessarily ossifies the template and public interface.
Audita	Explicit falsey configuration	Empty lists and strings are rendered rather than omitted	Retain	Keep representative falsey values, preferably in the same explicit configuration scenario, and assert only the affected paths.	The null-versus-falsey distinction is role-authored and easy to regress.
Audita	Configuration task changed while build task skipped	Configuration changes independently of installation	Remove	Delete task-name and status inspection. Confirm the rendered application behavior and successful convergence instead.	Whether named tasks ran is an implementation detail; the ordinary when expressions are suitable for review and lint.
Audita	Restore the minimal default configuration	Cleanup and exact current default	Simplify	Keep restoration only as cleanup and remove fixed-document, fixed-commit, task-state, and extra repeat assertions.	It does not add a distinct stable behavior.
Audita	Alternate raw commit 46b7356...	A non-default Git revision can be selected	Replace	Remove the raw-commit case. Add an explicit immutable-release test only when Audita exposes a stable native way to identify the installed application release.	A raw source commit checked with git rev-parse tests checkout mechanics, not the installed application's release. Audita currently has no --version or version command, and tags v0.9.0 and v0.9.1 both resolve to a84941d..., so neither tag provides a distinguishable installed outcome today.
Audita	Platform failure matrix	Straightforward platform assertions	Remove	Delete the matrix and tests/unsupported.yml.	This is expressly outside the testing house style.
Audita	Final commit, metadata, and config checks	Duplicate checkout/module/validation assertions	Remove	Delete them.	Equivalent checks already ran, and most should themselves be removed.
Audita	tests/verify.yml broad installation contract	Commit, metadata, default document, and native config validity	Replace	Reduce it to native configuration validation; move explicit field assertions to the focused representative verifier.	Only native validity is a stable application outcome in this verifier.
Audita	README validation claims	Description of default, metadata, rebuild, and platform tests	Replace	Describe only the retained static checks and behavioral scenarios.	The present prose promises low-value coverage that should be removed.

Audita Version-Coverage Gap

The current suite does not—and with the present application interface cannot— perform the useful test “select immutable release X and verify that the installed Audita reports X.” Running either audita --version or audita version fails as an unknown command. The recommended role-test remediation is to remove the misleading raw-commit scenario rather than replace it with another Git-module assertion. A native Audita release-identity command would enable the desired explicit-version test later, but adding that application feature is outside this audit and should be selected separately.

Notarius Classification

Repository	Test or case	Behavior protected	Classification	Recommended action	Rationale
Notarius	Default convergence	The default public interface installs and configures successfully	Retain	Keep as a smoke test, run notarius --version without fixing its output to the current default, and validate the configuration natively.	This proves the default path remains usable without snapshotting its values.
Notarius	Default v0.6.0 and commit verification	Present default release identity	Remove	Delete fixed default_commit and exact default-version assertions.	These fail solely when the supported default advances.
Notarius	Source, binary, and configuration metadata	Trusted module postconditions	Remove	Delete ownership, mode, and regular-file assertions.	They re-test git, file, and template.
Notarius	Exact minimal default document {"version": 4}	Current default rendering	Remove	Do not snapshot the default document. Supply notarius_config_version explicitly in the representative case and make any omission assertions about explicitly null inputs.	This preserves schema-bound behavior without turning future defaults into failures.
Notarius	Native version command and configuration validation	Installed CLI is usable and accepts the managed configuration	Retain	Keep both native interfaces, but assert an exact version only in the explicitly selected release scenario.	These are application-level outcomes.
Notarius	Baseline immediate repeat	Clean convergence	Retain	Keep one overall clean repeat.	This directly enforces the idempotence requirement.
Notarius	Missing binary	The required output is rebuilt and then converges	Simplify	Retain artifact removal, native CLI verification, and the following clean repeat; remove task greps.	Missing-output recovery is role logic needed around the imperative build.
Notarius	Dependency opt-out and custom Go executable	The caller-selected build executable is honored	Simplify	Keep wrapper invocation evidence and combine with missing-binary recovery; remove task greps and redundant repetitions.	This is meaningful configurable behavior with no better final-state discriminator.
Notarius	Binary metadata repair	Built-in file convergence	Remove	Delete the complete scenario.	It tests trusted module behavior.
Notarius	Explicit v0.5.0 selection	A caller-selected immutable tag becomes the installed application	Simplify	Retain notarius_version: v0.5.0 and exact native output notarius v0.5.0; remove expected commit and task-state assertions.	This remains useful after the role's default changes.
Notarius	Restore v0.6.0 default	Cleanup and present default identity	Simplify	Reduce to cleanup only.	It is not a distinct stable behavioral test.
Notarius	Representative configuration	Field mapping, nested sections, pipelines, and validity	Simplify	Use a smaller set of explicit values and assert only those paths plus specifically selected null omissions; retain native validation.	Exact equality for a large complete document is broader and more brittle than the behavior requires.
Notarius	Separate profile_dir case	One simple optional leaf mapping	Remove	Do not maintain a standalone scenario; a future representative case may select one profile source if it adds value.	The direct conditional and serialization are self-evident and already represented by other optional fields.
Notarius	Separate profile_file case	A second simple optional leaf mapping	Remove	Delete the standalone scenario.	It duplicates the shape and risk of the profile-directory case without protecting a distinct likely regression.
Notarius	Explicit falsey values	Zero, false, empty mapping, and empty string survive rendering	Retain	Fold representative falsey values into the focused configuration scenario and assert only those paths.	Null-versus-falsey behavior is a deliberate role invariant.
Notarius	Caller-side pipeline composition	Independently defined mappings can be combined and passed through intact	Retain	Keep composition through ordinary combine, assert the two resulting pipeline entries, and run native validation; fold it into the representative configuration scenario if clarity permits.	This is an expressly selected public boundary and a plausible integration regression.
Notarius	Configuration task changed while build task skipped	Configuration changes without rebuilding	Remove	Delete require_configuration_change_without_build and all task-trace greps. Verify final configuration, CLI usability, and clean repeat instead.	Task execution is not a stable public interface.
Notarius	Restore the minimal default configuration	Cleanup plus mutable default assertions	Simplify	Keep cleanup only; remove exact default document, version, commit, task-state, and extra repeat checks.	The behavioral assertions duplicate change detectors elsewhere.
Notarius	Platform failure matrix	Straightforward platform assertions	Remove	Delete the matrix and tests/unsupported.yml.	It provides no proportionate role-specific value.
Notarius	Final commit, metadata, version, and config checks	Duplicate assertions	Remove	Delete them.	They repeat earlier verification and freeze the current default.
Notarius	tests/verify.yml broad installation contract	Commit, metadata, default document, version, and validity	Replace	Reduce to native CLI smoke and configuration validation, with exact version verification only for an explicit tag.	This retains application outcomes while removing module tests and mutable snapshots.
Notarius	README validation claims	Description of the current broad suite	Replace	Rewrite after simplification to describe the retained behavior.	Documentation should match the lean suite rather than preserve obsolete coverage promises.

Proposed Lean Target Suites

Seriatim

1. Install pinned collections, syntax-check the convergence play, and lint the role and retained test files.
2. Apply the role with defaults as a smoke test and immediately require a clean repeat, without asserting the current default tag or commit.
3. Select v1.4.0 explicitly and verify native output seriatim version v1.4.0.
4. Delete the binary, supply a wrapper through seriatim_go_binary with dependency management disabled, converge, verify that the wrapper was used and the installed CLI works, then require a clean repeat.
5. Restore defaults as cleanup without treating the restored value as a test expectation.

Audita

1. Install pinned collections, syntax-check the convergence play, and lint the role and retained test files.
2. Apply the role with defaults, validate the managed configuration with Audita, and immediately require a clean repeat. Do not assert the complete default document or current source identity.
3. Apply one representative explicit configuration spanning nested, list, scalar, duration, falsey, and explicit-null behavior. Assert only the selected paths and omissions, validate it with Audita, and require a clean repeat.
4. Delete the binary, supply a logging wrapper through audita_go_binary with dependency management disabled, converge, verify wrapper use and CLI usability, then require a clean repeat.
5. Omit version-selection coverage until Audita exposes a native release identity; do not substitute a raw-commit checkout test.
6. Restore defaults as cleanup only.

Notarius

1. Install pinned collections, syntax-check the convergence play, and lint the role and retained test files.
2. Apply the role with defaults, exercise notarius --version, validate the managed configuration, and immediately require a clean repeat without asserting the current default values.
3. Select v0.5.0 explicitly and verify native output notarius v0.5.0.
4. Delete the binary, supply a logging wrapper through notarius_go_binary with dependency management disabled, converge, verify wrapper use and CLI usability, then require a clean repeat.
5. Apply one representative explicit configuration that includes selected nested values, explicit falsey and null values, and two caller-composed pipeline mappings. Assert only those behaviors, validate the file with Notarius, and require a clean repeat.
6. Restore defaults as cleanup only.

Documentation Follow-up

The README validation sections in all three role repositories should be rewritten with their test harnesses. They currently advertise current-default, metadata, task-arrangement, and platform-guard coverage recommended for removal.

The durable feature roadmaps may continue to record the selected default tags, paths, ownership, permissions, and platform support as feature requirements. Those requirements need not each have a dedicated integration assertion. Test-specific roadmap language should, however, be corrected as follows:

• Feature 0001 and Feature 0003 should describe exact native version checks as checks of an explicitly selected immutable tag, not the current default.
• Feature 0002 currently says repository-local tests execute Audita to verify its installed version. That is not implementable with Audita's current CLI and should be replaced with the limitation and follow-up described above.
• Platform support remains part of each role contract, but the roadmaps should not imply that each straightforward assertion requires a negative matrix.

Recommended Remediation Order

1. Simplify the Seriatim driver and verifier, remove its unsupported-platform playbook, and update its README.
2. Simplify Audita's driver and verifiers, consolidate its configuration case, remove its unsupported-platform playbook, and update its README.
3. Simplify Notarius's driver and verifiers, consolidate configuration and pipeline coverage, remove its unsupported-platform playbook, and update its README.
4. Correct the test-specific language in the three feature roadmaps.
5. Run all three complete integration entry points and confirm every role repository remains clean after its cleanup path.

No shared test abstraction is recommended. The three shell drivers have similar mechanics, but keeping each suite local and readable is worth the small amount of duplication, and a shared helper would create cross-repository coupling for little present benefit.

Audit Completion

Every materially distinct implemented scenario has been classified above. The recommended removals either duplicate retained application-level coverage or exercise behavior owned by trusted Ansible modules or straightforward assertions. The lean targets retain default smoke coverage, immediate clean convergence, imperative missing-output recovery, caller-supplied build tooling, explicit release selection where natively observable, and representative configuration behavior.

The audit made no changes to production roles or test implementations.

Remediation Status

The recommended remediation was implemented and independently reviewed. No production role behavior changed. The role commits are:

Repository	Remediation commit	Final integration result
ansible-role-seriatim	e08c63c	Pass
ansible-role-audita	d2d0b9d	Pass
ansible-role-notarius	cfecf37	Pass

Final sequential validation on the approved Ubuntu 26.04 x86_64 proxy ran each repository's tests/integration entry point successfully. Each cleanup path restored the default deployment, and the role repositories were clean after their remediation commits. The existing classifications remain the authoritative rationale for this implemented suite shape.

Independent review found no remaining code-quality issues or implementation gaps. The three complete integration entry points passed again sequentially; Bash syntax validation and git diff --check passed; native application checks confirmed that cleanup restored usable default deployments; and every repository in the workspace remained clean. No additional remediation record was required.