Plan classical test suite remediation
This commit is contained in:
@@ -1,12 +1,505 @@
|
||||
# Implementation Plan
|
||||
# Implementation Plan R0001: Classical Test Suite Remediation
|
||||
|
||||
Status: pending
|
||||
Status: ready
|
||||
|
||||
No implementation plan is currently active. Feature 0003 was completed in
|
||||
`ansible-role-notarius` commits `287fc4a`, `733a194`, and `eb4f511`. Its
|
||||
durable scope remains in
|
||||
[the Feature 0003 roadmap](features/0003-notarius-source-installation-and-configuration.md),
|
||||
and its review history remains in [the remediation record](remediation.md).
|
||||
Target executor: GPT-5.6-Terra
|
||||
|
||||
Select and approve the next feature roadmap before replacing this notice with
|
||||
a staged implementation plan.
|
||||
Authoritative audit:
|
||||
`docs/roadmap/test-suite-audit-findings.md`
|
||||
|
||||
Active plan location: `docs/roadmap/implementation.md`
|
||||
|
||||
## Objective
|
||||
|
||||
Replace the oversized, implementation-bound integration suites in the three
|
||||
completed standalone role repositories with lean classical, state-based
|
||||
suites. Retain tests of stable role-owned behavior and application outcomes;
|
||||
remove mutable-default snapshots, task-trace assertions, tests of trusted
|
||||
Ansible modules, redundant scenarios, and negative platform matrices.
|
||||
|
||||
Production role behavior is not changing. This plan changes only test
|
||||
implementations, test-related role documentation, and the workspace planning
|
||||
records identified by the audit.
|
||||
|
||||
## Execution Rules
|
||||
|
||||
- Execute stages in numeric order.
|
||||
- Before each stage, confirm its owning repository is clean and record its
|
||||
starting revision.
|
||||
- Preserve unrelated and pre-existing changes.
|
||||
- Do not modify production tasks, defaults, templates, argument
|
||||
specifications, dependency pins, or role metadata. If a retained test
|
||||
exposes a production defect, stop and report it instead of expanding this
|
||||
remediation.
|
||||
- Follow the normative testing house style in `docs/conventions.md` and the
|
||||
classifications in the authoritative audit.
|
||||
- Exercise roles through their public variables and inspect final application
|
||||
behavior. Do not parse task names, task order, or individual task status.
|
||||
- Retain only aggregate `changed=0` checks on specified immediate repeat runs.
|
||||
Do not require a nonzero change count on preceding runs.
|
||||
- Do not verify ownership, permissions, path types, checkout commits, or other
|
||||
postconditions already declared through trusted modules.
|
||||
- Do not add a test merely to replace each removed test one-for-one. The target
|
||||
is the smallest suite that protects the retained behavior.
|
||||
- Keep each repository's shell driver self-contained. Do not create a shared
|
||||
cross-repository test helper or abstraction.
|
||||
- Run each stage's full repository integration entry point before committing.
|
||||
- Commit each completed stage separately with the plain-English intent given
|
||||
in that stage.
|
||||
- Do not begin a dependent stage after an earlier stage fails validation.
|
||||
|
||||
## Starting State and Preconditions
|
||||
|
||||
- The workspace conventions, audit plan, audit findings, status update, and
|
||||
this implementation plan must be reviewed and committed in the workspace
|
||||
repository before Stage 1 begins.
|
||||
- `ansible-role-seriatim` starts clean at `fb8dc41`.
|
||||
- `ansible-role-audita` starts clean at `847c4c3`.
|
||||
- `ansible-role-notarius` starts clean at `eb4f511`.
|
||||
- The workspace `.venv` provides Ansible Core and ansible-lint, and the local
|
||||
Ubuntu 26.04 `x86_64` host remains the approved MVP integration proxy.
|
||||
- The executor can use passwordless privilege escalation for the existing
|
||||
`/usr/local` integration-test paths and can reach Ansible Galaxy, the three
|
||||
application Git repositories, Snap infrastructure, and Go module sources.
|
||||
- Existing installed application state may be present. Each driver must remain
|
||||
repeatable from that state and restore its default role deployment through a
|
||||
best-effort cleanup trap.
|
||||
- Audita exposes no native application-release identity command. Its tags
|
||||
`v0.9.0` and `v0.9.1` currently resolve to the same commit. This plan
|
||||
deliberately omits an Audita version-selection test rather than substituting
|
||||
a raw-commit or Git-module assertion.
|
||||
|
||||
## Repository Scope
|
||||
|
||||
Repositories that change:
|
||||
|
||||
- `ansible-role-seriatim`: lean integration driver and verifier, removal of
|
||||
the unsupported-platform playbook, and accurate validation documentation.
|
||||
- `ansible-role-audita`: lean integration driver and verifiers, removal of the
|
||||
unsupported-platform playbook, and accurate validation documentation.
|
||||
- `ansible-role-notarius`: lean integration driver and verifiers, removal of
|
||||
the unsupported-platform playbook, and accurate validation documentation.
|
||||
- Workspace control repository: test-specific corrections to Features 0001,
|
||||
0002, and 0003, remediation status, and cross-repository validation evidence.
|
||||
|
||||
Repositories that may be inspected but must remain unchanged:
|
||||
|
||||
- `../seriatim`, `../audita`, and `../notarius` for native CLI behavior.
|
||||
- Every other application, role, collection, and proof-of-concept repository.
|
||||
|
||||
## Stage 1: Simplify the Seriatim Test Suite
|
||||
|
||||
### Objective
|
||||
|
||||
Reduce Seriatim's integration suite to static validation and representative
|
||||
tests of default usability, explicit version selection, missing-binary
|
||||
recovery, caller-supplied Go, and clean repeated convergence.
|
||||
|
||||
### Preconditions
|
||||
|
||||
- The workspace planning commit described above exists.
|
||||
- `ansible-role-seriatim` is clean at `fb8dc41`.
|
||||
- The existing `tests/integration` baseline passes.
|
||||
|
||||
### Owning Repository
|
||||
|
||||
`ansible-role-seriatim`
|
||||
|
||||
### Required Changes
|
||||
|
||||
1. Retain `tests/integration` as the single entry point, its private temporary
|
||||
directory, pinned collection installation, syntax check, production-profile
|
||||
ansible-lint, and best-effort cleanup convergence.
|
||||
2. Retain a default role convergence as a smoke test. Exercise
|
||||
`/usr/local/bin/seriatim --version` successfully without comparing it to the
|
||||
current default version, then immediately apply the default role again and
|
||||
require only the aggregate recap `changed=0` and `failed=0`.
|
||||
3. Add or retain one explicit version-selection scenario using
|
||||
`seriatim_version: v1.4.0`. Verify through the native CLI that the result is
|
||||
exactly `seriatim version v1.4.0`. Do not resolve or assert the corresponding
|
||||
Git commit.
|
||||
4. Combine missing-binary recovery and caller-supplied build tooling into one
|
||||
representative scenario:
|
||||
- create an executable temporary wrapper that records invocation and then
|
||||
executes `/snap/bin/go` with the original arguments;
|
||||
- remove `/usr/local/bin/seriatim`;
|
||||
- apply the role with `seriatim_version: v1.4.0`,
|
||||
`seriatim_manage_build_dependencies: false`, and
|
||||
`seriatim_go_binary` set to the wrapper;
|
||||
- require the wrapper log to show that the supplied executable was used;
|
||||
- verify native output `seriatim version v1.4.0`; and
|
||||
- immediately repeat with the same variables and require aggregate
|
||||
`changed=0` and `failed=0`.
|
||||
5. Delete `tests/verify.yml` and perform the narrow native version assertions
|
||||
directly in the shell driver. Do not retain source, commit, ownership,
|
||||
permission, file-type, or checksum inspection.
|
||||
6. Remove the binary-mode mutation and repair scenario, every
|
||||
`require_changed_recap` use, every task-name or individual-status grep, the
|
||||
default and previous commit constants, redundant default-restoration
|
||||
assertions, and final duplicate shell assertions.
|
||||
7. Delete `tests/unsupported.yml` and remove it from lint and execution. Do not
|
||||
replace its distribution, release, or architecture cases.
|
||||
8. Keep a cleanup trap that reapplies the default role and removes only the
|
||||
private temporary directory. Cleanup output may remain suppressed and must
|
||||
not become another behavioral assertion.
|
||||
9. Rewrite the README validation section to describe only the retained static
|
||||
checks and behavioral scenarios. Do not advertise exact-default, metadata,
|
||||
task-arrangement, or platform-matrix coverage.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```text
|
||||
tests/integration
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
Expected result: static validation and every retained Seriatim scenario pass;
|
||||
the specified repeat runs report no changes; cleanup completes; and the only
|
||||
working-tree changes are the planned test and README edits.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- No test asserts Seriatim's current default version or a Git commit.
|
||||
- No test inspects task names, individual task status, ownership, permissions,
|
||||
or the platform assertion branches.
|
||||
- Explicit `v1.4.0` selection is verified through Seriatim itself.
|
||||
- One scenario proves both missing-binary recovery and use of the configured Go
|
||||
executable, followed by a clean repeat.
|
||||
- The complete integration entry point passes.
|
||||
|
||||
### Commit
|
||||
|
||||
Commit intent: `Simplify the Seriatim integration tests`
|
||||
|
||||
## Stage 2: Simplify the Audita Test Suite
|
||||
|
||||
### Objective
|
||||
|
||||
Reduce Audita's integration suite to static validation and representative
|
||||
tests of default usability, native configuration validity, explicit
|
||||
configuration rendering, missing-binary recovery, caller-supplied Go, and clean
|
||||
repeated convergence.
|
||||
|
||||
### Preconditions
|
||||
|
||||
- Stage 1 is complete and committed.
|
||||
- `ansible-role-audita` is clean at `847c4c3`.
|
||||
- The existing `tests/integration` baseline passes.
|
||||
|
||||
### Owning Repository
|
||||
|
||||
`ansible-role-audita`
|
||||
|
||||
### Required Changes
|
||||
|
||||
1. Retain `tests/integration` as the single entry point, its private temporary
|
||||
directory, pinned collection installation, syntax check, production-profile
|
||||
ansible-lint, and best-effort cleanup convergence.
|
||||
2. Retain a default role convergence as a smoke test. Validate the managed
|
||||
configuration with
|
||||
`/usr/local/bin/audita config validate --config /usr/local/etc/audita/config.yml`,
|
||||
then immediately repeat the default role and require only aggregate
|
||||
`changed=0` and `failed=0`.
|
||||
3. Replace the exhaustive configuration document and separate falsey scenario
|
||||
with one representative explicit configuration. Supply these values:
|
||||
- `audita_config_version: 1`;
|
||||
- `audita_pipeline_modules: ["glossary"]`;
|
||||
- `audita_llm_proposal_base_url: "https://openrouter.ai/api/v1"`;
|
||||
- `audita_llm_proposal_model: "openrouter/google/gemma-4-31b-it"`;
|
||||
- `audita_llm_proposal_timeout: "10s"`;
|
||||
- `audita_llm_proposal_max_retries: 0`;
|
||||
- `audita_context_description: ""`;
|
||||
- `audita_diagnostics_retention: "never"`;
|
||||
- `audita_llm_validation_model: null`;
|
||||
- `audita_output_schema: null`; and
|
||||
- `audita_diagnostics_work_dir: null`.
|
||||
4. In `tests/verify-config.yml`, parse the rendered YAML and assert only that:
|
||||
- `version` is `1`;
|
||||
- `pipeline.modules` is `["glossary"]`;
|
||||
- the selected proposal URL, model, timeout, and zero retry count are
|
||||
present;
|
||||
- the empty context description and diagnostics retention value are
|
||||
present;
|
||||
- `llm.validation`, `output`, and `diagnostics.work_dir` are absent; and
|
||||
- Audita's native configuration validation command succeeds.
|
||||
Do not compare the complete parsed document to an expected mapping.
|
||||
5. Immediately repeat the representative configuration convergence with the
|
||||
same explicit variables and require aggregate `changed=0` and `failed=0`.
|
||||
6. Combine missing-binary recovery and caller-supplied build tooling into one
|
||||
representative scenario. Create a logging wrapper around `/snap/bin/go`,
|
||||
remove `/usr/local/bin/audita`, converge with
|
||||
`audita_manage_build_dependencies: false` and `audita_go_binary` set to the
|
||||
wrapper, verify wrapper use, validate the managed configuration through
|
||||
Audita, and immediately repeat the same convergence with aggregate
|
||||
`changed=0` and `failed=0`.
|
||||
7. Delete `tests/verify.yml` and invoke Audita's native configuration validator
|
||||
directly from the shell driver for the smoke and recovery scenarios.
|
||||
Successful command completion is sufficient; do not separately assert the
|
||||
validator's fixed success text.
|
||||
8. Remove the exact default configuration comparison, source-commit checks,
|
||||
source/binary/configuration metadata checks, binary-mode repair scenario,
|
||||
all `require_changed_recap` uses, and every task-name or individual-status
|
||||
grep.
|
||||
9. Remove the raw alternate-commit scenario entirely. Do not replace it with a
|
||||
`git rev-parse`, tag-resolution, embedded-string, or Go build-metadata test.
|
||||
Do not add an Audita version-selection test until the application provides a
|
||||
stable native release-identity interface.
|
||||
10. Delete `tests/unsupported.yml` and remove it from lint and execution. Do not
|
||||
replace its platform cases.
|
||||
11. Retain default restoration only in the cleanup trap. Remove its fixed
|
||||
commit, exact document, task-status, and extra idempotence assertions.
|
||||
12. Rewrite the README validation section to describe only the retained static
|
||||
checks and behavioral scenarios and to avoid claiming application-version,
|
||||
metadata, task-arrangement, or platform-matrix coverage.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```text
|
||||
tests/integration
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
Expected result: static validation and every retained Audita scenario pass;
|
||||
both default and representative configurations validate natively; the
|
||||
specified repeats report no changes; cleanup completes; and the only
|
||||
working-tree changes are the planned test and README edits.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- No test asserts Audita's current default, a source commit, or a complete
|
||||
rendered configuration document.
|
||||
- No test inspects task traces, trusted-module postconditions, or platform
|
||||
assertion branches.
|
||||
- One focused explicit case proves representative nesting, null omission, and
|
||||
falsey preservation and passes Audita's validator.
|
||||
- One scenario proves both missing-binary recovery and use of the configured Go
|
||||
executable, followed by a clean repeat.
|
||||
- The absence of native Audita version coverage is documented accurately and
|
||||
is not replaced by an implementation proxy.
|
||||
- The complete integration entry point passes.
|
||||
|
||||
### Commit
|
||||
|
||||
Commit intent: `Simplify the Audita integration tests`
|
||||
|
||||
## Stage 3: Simplify the Notarius Test Suite
|
||||
|
||||
### Objective
|
||||
|
||||
Reduce Notarius's integration suite to static validation and representative
|
||||
tests of default usability, explicit version selection, missing-binary
|
||||
recovery, caller-supplied Go, configuration behavior, caller-side pipeline
|
||||
composition, native validation, and clean repeated convergence.
|
||||
|
||||
### Preconditions
|
||||
|
||||
- Stages 1 and 2 are complete and committed.
|
||||
- `ansible-role-notarius` is clean at `eb4f511`.
|
||||
- The existing `tests/integration` baseline passes.
|
||||
|
||||
### Owning Repository
|
||||
|
||||
`ansible-role-notarius`
|
||||
|
||||
### Required Changes
|
||||
|
||||
1. Retain `tests/integration` as the single entry point, its private temporary
|
||||
directory, pinned collection installation, syntax check, production-profile
|
||||
ansible-lint, and best-effort cleanup convergence.
|
||||
2. Retain a default convergence as a smoke test. Require
|
||||
`/usr/local/bin/notarius --version` and
|
||||
`/usr/local/bin/notarius config validate --config /usr/local/etc/notarius/config.yml`
|
||||
to succeed without comparing the version output or complete configuration
|
||||
to current defaults. Immediately repeat the default role and require only
|
||||
aggregate `changed=0` and `failed=0`.
|
||||
3. Retain one explicit version-selection scenario with
|
||||
`notarius_version: v0.5.0` and verify exact native output
|
||||
`notarius v0.5.0`. Do not resolve or assert the corresponding Git commit.
|
||||
4. Combine missing-binary recovery and caller-supplied build tooling into one
|
||||
scenario. Create a logging wrapper around `/snap/bin/go`, remove
|
||||
`/usr/local/bin/notarius`, converge with `notarius_version: v0.5.0`,
|
||||
`notarius_manage_build_dependencies: false`, and `notarius_go_binary` set to
|
||||
the wrapper, verify wrapper use and native output `notarius v0.5.0`, then
|
||||
immediately repeat with aggregate `changed=0` and `failed=0`.
|
||||
5. Replace the exhaustive configuration, separate profile-source cases,
|
||||
separate falsey case, and separate pipeline-composition case with one
|
||||
representative explicit configuration. Define two caller-side pipeline
|
||||
mappings and set `notarius_pipelines` to their ordinary `combine` result.
|
||||
Supply these additional values:
|
||||
- `notarius_config_version: 4`;
|
||||
- `notarius_promptkit_local_backend_endpoint: "http://localhost:8000/v1"`;
|
||||
- `notarius_promptkit_local_backend_concurrency_limit: 0`;
|
||||
- `notarius_cache_checkpoints_enabled: false`;
|
||||
- `notarius_cache_chunk_plans_directory: ""`;
|
||||
- `notarius_output_directory: "/tmp/notarius-output"`;
|
||||
- `notarius_promptkit_profile_dir: null`;
|
||||
- `notarius_promptkit_profile_file: null`; and
|
||||
- `notarius_debug_directory: null`.
|
||||
Use the two valid `dnd-session` and `dnd-archive` pipeline definitions from
|
||||
the existing composition scenario.
|
||||
6. In `tests/verify-config.yml`, parse the rendered YAML and assert only that:
|
||||
- `version` is `4`;
|
||||
- local-backend endpoint and zero concurrency are present;
|
||||
- checkpoints are explicitly disabled and the empty chunk-plan directory is
|
||||
present;
|
||||
- the selected output directory is present;
|
||||
- both composed pipeline IDs and their supplied definitions are present
|
||||
without rewriting;
|
||||
- `promptkit.profile_dir`, `promptkit.profile_file`, and the top-level
|
||||
`debug` section are absent; and
|
||||
- Notarius's native configuration validation succeeds.
|
||||
Do not compare the complete document to an expected mapping.
|
||||
7. Immediately repeat the representative configuration convergence with the
|
||||
same explicit variables and require aggregate `changed=0` and `failed=0`.
|
||||
8. Delete `tests/verify.yml`. Perform native CLI smoke, configuration
|
||||
validation, and the explicit `v0.5.0` comparison directly in the shell
|
||||
driver.
|
||||
9. Remove current-default version and commit assertions, complete default
|
||||
configuration comparisons, all source/binary/configuration metadata checks,
|
||||
the binary-mode repair scenario, every `require_changed_recap` use,
|
||||
`require_configuration_change_without_build`, and every task-name or
|
||||
individual-status grep.
|
||||
10. Delete the standalone `profile_dir` and `profile_file` scenarios. Their
|
||||
simple leaf mappings do not require dedicated replacements.
|
||||
11. Delete `tests/unsupported.yml` and remove it from lint and execution. Do not
|
||||
replace its platform cases.
|
||||
12. Retain default restoration only in the cleanup trap and remove duplicate
|
||||
final assertions.
|
||||
13. Rewrite the README validation section to describe only the retained static
|
||||
checks and behavioral scenarios. Do not advertise current-default,
|
||||
metadata, task-arrangement, profile-branch, or platform-matrix coverage.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```text
|
||||
tests/integration
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
Expected result: static validation and every retained Notarius scenario pass;
|
||||
the explicit tag is verified through Notarius; representative falsey, null,
|
||||
and composed-pipeline behavior is verified and validates natively; the
|
||||
specified repeats report no changes; cleanup completes; and only the planned
|
||||
test and README changes remain.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- No test snapshots Notarius's current default release, commit, or complete
|
||||
default configuration.
|
||||
- No test inspects task traces, trusted-module postconditions, or platform
|
||||
assertion branches.
|
||||
- Explicit `v0.5.0` selection is verified through Notarius itself.
|
||||
- One scenario proves both missing-binary recovery and use of the configured Go
|
||||
executable, followed by a clean repeat.
|
||||
- One representative configuration case proves null omission, falsey
|
||||
preservation, caller-side pipeline composition, and native validity.
|
||||
- The complete integration entry point passes.
|
||||
|
||||
### Commit
|
||||
|
||||
Commit intent: `Simplify the Notarius integration tests`
|
||||
|
||||
## Stage 4: Reconcile Workspace Documentation and Validate the Suite
|
||||
|
||||
### Objective
|
||||
|
||||
Align durable test-specific roadmap language with the canonical testing policy,
|
||||
record the remediation implementation for review, and validate all three lean
|
||||
suites together.
|
||||
|
||||
### Preconditions
|
||||
|
||||
- Stages 1 through 3 are complete and committed in their owning repositories.
|
||||
- The workspace repository is clean at the planning commit created before
|
||||
Stage 1.
|
||||
- The three role repositories contain only their committed remediation changes.
|
||||
|
||||
### Owning Repository
|
||||
|
||||
Workspace control repository (`ansible-workspace-dnd`)
|
||||
|
||||
### Required Changes
|
||||
|
||||
1. Revise only test-specific language in Features 0001, 0002, and 0003. Retain
|
||||
their durable production requirements, including current defaults, paths,
|
||||
ownership, permissions, platform support, and convergence behavior.
|
||||
2. In Feature 0001, describe the exact native version assertion as coverage of
|
||||
an explicitly selected immutable tag rather than an assertion about the
|
||||
role's current default.
|
||||
3. In Feature 0002, remove the incorrect claim that tests execute Audita to
|
||||
verify its installed application version. State that the current CLI has no
|
||||
native release-identity command and that the suite deliberately avoids a
|
||||
raw-commit proxy. Retain native configuration validation.
|
||||
4. In Feature 0003, describe exact version coverage using explicitly selected
|
||||
`v0.5.0`, not the current default `v0.6.0`. Keep the current default as a
|
||||
production requirement rather than a frozen test expectation.
|
||||
5. Remove any implication in the three roadmaps that each platform assertion
|
||||
must have a negative integration-test matrix. Do not remove platform support
|
||||
itself from scope or acceptance criteria.
|
||||
6. Update `docs/roadmap/test-suite-audit-findings.md` to say remediation was
|
||||
implemented and is pending review. Add the three role commit hashes and the
|
||||
final validation results; do not erase the audit classifications.
|
||||
7. Update `docs/status.md` so `R0001` remains `in_progress` pending review, its
|
||||
evidence names the three role commits, and the current focus is review of
|
||||
the implemented test-suite remediation. Do not mark remediation `done`.
|
||||
8. Add concise progress notes to this plan with the four stage commits and any
|
||||
deviations. Do not replace the active plan with a pending notice before
|
||||
review.
|
||||
|
||||
### Validation
|
||||
|
||||
Run sequentially from the workspace root:
|
||||
|
||||
```text
|
||||
(cd ansible-role-seriatim && tests/integration)
|
||||
(cd ansible-role-audita && tests/integration)
|
||||
(cd ansible-role-notarius && tests/integration)
|
||||
git diff --check
|
||||
scripts/status
|
||||
```
|
||||
|
||||
Expected result: all static and behavioral checks pass; every role cleanup path
|
||||
completes; the three role repositories are clean and contain their remediation
|
||||
commits; no unrelated repository changed; and the workspace has only the
|
||||
planned documentation changes before its stage commit.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- All three full integration suites pass sequentially from the workspace.
|
||||
- Feature roadmaps distinguish production defaults from stable test inputs.
|
||||
- Audita's native version limitation is stated accurately.
|
||||
- Audit findings contain implementation and validation evidence without being
|
||||
rewritten as an implementation log.
|
||||
- Workspace status leaves remediation pending review.
|
||||
- Every repository changed by the implementation is clean after its owning
|
||||
commit.
|
||||
|
||||
### Commit
|
||||
|
||||
Commit intent: `Record the test suite remediation results`
|
||||
|
||||
## Final Validation and Handoff
|
||||
|
||||
- Report the Stage 1 through Stage 4 commit hashes and confirm the working-tree
|
||||
state of every repository in `workspace.yml`.
|
||||
- Summarize the retained scenarios for each role and identify any deviation
|
||||
from this plan.
|
||||
- Report the Audita version-identity limitation; do not silently fill it with a
|
||||
source-control assertion.
|
||||
- Report all validation commands and results.
|
||||
- Do not push commits unless the user separately authorizes a push.
|
||||
- Do not mark `R0001` done or replace this plan with a pending notice. Final
|
||||
completion requires review of the implemented suites.
|
||||
|
||||
## Progress Notes
|
||||
|
||||
- Planning complete; implementation has not started.
|
||||
|
||||
260
docs/roadmap/test-suite-audit-findings.md
Normal file
260
docs/roadmap/test-suite-audit-findings.md
Normal file
@@ -0,0 +1,260 @@
|
||||
# Test Suite Audit Findings
|
||||
|
||||
Status: complete; remediation pending review
|
||||
|
||||
## Executive Summary
|
||||
|
||||
All three integration suites pass, and each includes the most important common
|
||||
behavioral check: an immediate repeated role application reports no changes.
|
||||
The suites also contain useful coverage of explicitly selected releases,
|
||||
caller-supplied Go executables, missing-binary recovery, configuration
|
||||
rendering, native configuration validation, and Notarius pipeline composition.
|
||||
|
||||
The suites are nevertheless substantially larger and more implementation-bound
|
||||
than the workspace testing policy permits. The recurring problems are:
|
||||
|
||||
- exact assertions about the current default release and its commit;
|
||||
- independent verification of ownership, permissions, checkout state, and
|
||||
template output already managed by trusted Ansible modules;
|
||||
- parsing Ansible output for specific task names and individual
|
||||
`changed`/`skipped` results;
|
||||
- deliberate metadata drift followed by tests of `ansible.builtin.file`;
|
||||
- three-case negative tests of self-evident platform assertions;
|
||||
- repeated clean-run assertions after nearly every scenario; and
|
||||
- duplicated final shell assertions after equivalent Ansible verifiers.
|
||||
|
||||
The appropriate remediation is subtraction and consolidation. The retained
|
||||
suites should exercise stable role-owned behavior through final application
|
||||
outcomes, with one representative case for each meaningful behavior.
|
||||
|
||||
## Baseline
|
||||
|
||||
The audit ran on Ubuntu 26.04 `x86_64` with Ansible Core 2.21.3 and
|
||||
ansible-lint 26.8.0. The role revisions were:
|
||||
|
||||
| Repository | Revision | Integration result |
|
||||
| --- | --- | --- |
|
||||
| `ansible-role-seriatim` | `fb8dc41` | Pass |
|
||||
| `ansible-role-audita` | `847c4c3` | Pass |
|
||||
| `ansible-role-notarius` | `eb4f511` | Pass |
|
||||
|
||||
The following commands all exited successfully:
|
||||
|
||||
```text
|
||||
(cd ansible-role-seriatim && tests/integration)
|
||||
(cd ansible-role-audita && tests/integration)
|
||||
(cd ansible-role-notarius && tests/integration)
|
||||
```
|
||||
|
||||
Each driver installed its pinned collection requirements into a private
|
||||
temporary directory, ran a syntax check and production-profile ansible-lint,
|
||||
executed its behavioral scenarios, and restored its default deployment during
|
||||
cleanup. The runs emitted non-fatal local-tooling warnings about Python
|
||||
interpreter discovery or an already installed duplicate collection; these did
|
||||
not affect the results or any classification below.
|
||||
|
||||
All three role repositories were clean before and after the audit. The control
|
||||
repository began dirty solely because the new testing convention, audit plan,
|
||||
and status update were awaiting commit.
|
||||
|
||||
## Test Inventory
|
||||
|
||||
| Repository | Entry point and supporting files |
|
||||
| --- | --- |
|
||||
| `ansible-role-seriatim` | `tests/integration`, `tests/inventory`, `tests/converge.yml`, `tests/verify.yml`, `tests/unsupported.yml`, `collections/requirements.yml`, and the README validation section |
|
||||
| `ansible-role-audita` | `tests/integration`, `tests/inventory`, `tests/converge.yml`, `tests/verify.yml`, `tests/verify-config.yml`, `tests/unsupported.yml`, `collections/requirements.yml`, and the README validation section |
|
||||
| `ansible-role-notarius` | `tests/integration`, `tests/inventory`, `tests/converge.yml`, `tests/verify.yml`, `tests/verify-config.yml`, `tests/unsupported.yml`, `collections/requirements.yml`, and the README validation section |
|
||||
|
||||
## Cross-Repository Findings
|
||||
|
||||
| Repository | Test or case | Behavior protected | Classification | Recommended action | Rationale |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| All three | Pinned collection installation, syntax check, and ansible-lint | Static validity and reproducible dependencies | Retain | Keep as static validation before behavioral scenarios. | These checks complement rather than duplicate behavioral testing. |
|
||||
| All three | `require_clean_recap` after an immediate repeat | Role-level repeated convergence | Retain | Keep at least one immediate clean repeat per role and repeats specifically required after missing-binary recovery. | Overall `changed=0` is a first-class public role outcome. |
|
||||
| All three | `require_changed_recap` after each mutating setup | Ansible reports some change on the first run | Remove | Stop asserting a nonzero aggregate change count except where a future requirement specifically makes reporting itself significant. | Final application state is the behavior of interest; these checks add little beyond successful convergence and make scenario sequencing brittle. |
|
||||
| All three | Greps for named build, metadata, and configuration tasks | Current task names and individual `changed` or `skipped` results | Remove | Delete all task-output parsing. | These are direct implementation traces and fail under behavior-preserving task refactors. |
|
||||
| All three | Binary/source/configuration ownership and mode assertions | Postconditions declared to `file`, `git`, and `template` | Remove | Delete independent metadata and recursive-ownership verification. | These tests re-prove trusted module behavior rather than role-authored decisions. |
|
||||
| All three | Binary mode mutation followed by convergence | `ansible.builtin.file` repairs mode drift without building | Remove | Delete the scenario. | It is a focused regression test for the built-in file module and current task arrangement. |
|
||||
| All three | Debian, old Ubuntu, and unsupported-architecture matrix | Straightforward platform assertion branches | Remove | Delete `tests/unsupported.yml` and the three-case driver loop. | The declarative assertion is self-evident and does not warrant a negative matrix. |
|
||||
| All three | Restoration of the current default followed by verification and another repeat | Test cleanup and the current default identity | Simplify | Keep one best-effort cleanup convergence, but do not treat it as another test or assert the restored release value. | Cleanup is useful; testing a mutable default and repeating already established idempotence are not. |
|
||||
| All three | Final shell assertions duplicating earlier verifiers | Current commit, metadata, version, or config state | Remove | Delete duplicate end-of-driver assertions. | They add maintenance cost without protecting a distinct behavior. |
|
||||
|
||||
## Seriatim Classification
|
||||
|
||||
| Repository | Test or case | Behavior protected | Classification | Recommended action | Rationale |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| Seriatim | Default convergence | The role can be applied with its default interface | Retain | Keep as a smoke test without asserting the exact default tag, commit, or filesystem metadata. | A default invocation is useful even though the present default value is not a stable test expectation. |
|
||||
| Seriatim | Default `v1.5.0` and commit verification | Present default release identity | Remove | Remove the fixed `default_commit` and exact default-version assertions. | This is a change detector; advancing the supported default should not break unrelated tests. |
|
||||
| Seriatim | Baseline immediate repeat | Clean convergence | Retain | Keep the overall `changed=0`, `failed=0` check. | This directly expresses the agreed idempotence requirement. |
|
||||
| Seriatim | Missing binary | The imperative build runs when its output is absent and then converges | Simplify | Delete the binary, apply the role, verify the executable through `--version`, and require the next run to be clean; remove task-name greps. | The binary-existence trigger is role-authored behavior, while the current task name and status are not. |
|
||||
| Seriatim | Dependency opt-out and custom Go executable | The caller-supplied build executable is honored | Simplify | Retain the wrapper invocation evidence, preferably combined with the missing-binary scenario; remove build-task greps and fixed-default assertions. | The selected build executable is a public role choice that cannot be observed reliably from the final binary alone. |
|
||||
| Seriatim | Binary metadata repair | Built-in file convergence | Remove | Delete the complete scenario. | It tests `ansible.builtin.file`, not Seriatim-specific behavior. |
|
||||
| Seriatim | Explicit `v1.4.0` selection | A caller-selected immutable release is installed | Simplify | Retain explicit `seriatim_version: v1.4.0` and verify `seriatim version v1.4.0`; remove the expected commit and task-state assertions. | This is the stable version-selection test described by the testing convention. |
|
||||
| Seriatim | Restore `v1.5.0` default | Cleanup plus present default identity | Simplify | Reduce to cleanup only. | Exact current-default verification duplicates the harmful baseline snapshot. |
|
||||
| Seriatim | Platform failure matrix | Distribution, release, and architecture assertions | Remove | Delete the matrix and `tests/unsupported.yml`. | No plausible role regression justifies separately testing each simple assertion. |
|
||||
| Seriatim | `tests/verify.yml` installation contract | Commit, source metadata, binary metadata, and version | Replace | Replace the broad verifier with a narrow native version check used for an explicitly selected tag. | Only the application outcome is valuable; the remaining assertions repeat trusted modules. |
|
||||
| Seriatim | README validation claims | Description of the current large matrix | Replace | Describe the lean retained behaviors after remediation. | Documentation should not promise deleted implementation-trace and module tests. |
|
||||
|
||||
## Audita Classification
|
||||
|
||||
| Repository | Test or case | Behavior protected | Classification | Recommended action | Rationale |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| Audita | Default convergence | The role installs and configures successfully through its default interface | Retain | Keep as a smoke test and validate the resulting configuration with Audita; do not inspect exact default identity or metadata. | Successful native validation is an application outcome and does not freeze the default document. |
|
||||
| Audita | Default commit assertion | Present default checkout identity | Remove | Delete `default_commit` and exact commit verification. | It is both a mutable-default snapshot and a re-test of `ansible.builtin.git`. |
|
||||
| Audita | Source, binary, and configuration metadata | Trusted module postconditions | Remove | Delete the ownership, permission, regular-file, and executable assertions. | The role directly declares these states with trusted modules. |
|
||||
| Audita | Exact minimal default document `{"version": 1}` | Current default rendering | Remove | Do not snapshot the complete default document. Test null omission with explicitly supplied inputs in a representative configuration case instead. | A future intentional concrete default would otherwise cause an unrelated failure. |
|
||||
| Audita | Native validation of the managed configuration | The role produces configuration Audita accepts | Retain | Keep one native validation for default smoke coverage and one for the representative explicit configuration. | This observes application behavior rather than reconstructing its schema. |
|
||||
| Audita | Baseline immediate repeat | Clean convergence | Retain | Keep one role-level clean repeat. | This is required public behavior. |
|
||||
| Audita | Missing binary | Absent build output is recreated and convergence resumes | Simplify | Retain recovery and the following clean repeat; prove usability with the Audita CLI and remove task-output greps. | The imperative build trigger is role logic, but its task name is not. |
|
||||
| Audita | Dependency opt-out and custom Go executable | Caller-selected compiler path is used | Simplify | Retain wrapper invocation evidence and combine it with missing-binary recovery; remove task greps and redundant repeat runs. | This covers a documented public choice with one representative case. |
|
||||
| Audita | Binary metadata repair | Built-in file convergence | Remove | Delete the complete scenario. | It tests Ansible's file module. |
|
||||
| Audita | Representative configuration containing nearly every field | Field-to-YAML mapping, nested sections, and native validity | Simplify | Use a smaller explicit sample spanning representative scalar, list, nested, and duration shapes. Assert only selected paths and specifically selected omissions, then run native validation. | Full-document equality across nearly every field unnecessarily ossifies the template and public interface. |
|
||||
| Audita | Explicit falsey configuration | Empty lists and strings are rendered rather than omitted | Retain | Keep representative falsey values, preferably in the same explicit configuration scenario, and assert only the affected paths. | The null-versus-falsey distinction is role-authored and easy to regress. |
|
||||
| Audita | Configuration task changed while build task skipped | Configuration changes independently of installation | Remove | Delete task-name and status inspection. Confirm the rendered application behavior and successful convergence instead. | Whether named tasks ran is an implementation detail; the ordinary `when` expressions are suitable for review and lint. |
|
||||
| Audita | Restore the minimal default configuration | Cleanup and exact current default | Simplify | Keep restoration only as cleanup and remove fixed-document, fixed-commit, task-state, and extra repeat assertions. | It does not add a distinct stable behavior. |
|
||||
| Audita | Alternate raw commit `46b7356...` | A non-default Git revision can be selected | Replace | Remove the raw-commit case. Add an explicit immutable-release test only when Audita exposes a stable native way to identify the installed application release. | A raw source commit checked with `git rev-parse` tests checkout mechanics, not the installed application's release. Audita currently has no `--version` or `version` command, and tags `v0.9.0` and `v0.9.1` both resolve to `a84941d...`, so neither tag provides a distinguishable installed outcome today. |
|
||||
| Audita | Platform failure matrix | Straightforward platform assertions | Remove | Delete the matrix and `tests/unsupported.yml`. | This is expressly outside the testing house style. |
|
||||
| Audita | Final commit, metadata, and config checks | Duplicate checkout/module/validation assertions | Remove | Delete them. | Equivalent checks already ran, and most should themselves be removed. |
|
||||
| Audita | `tests/verify.yml` broad installation contract | Commit, metadata, default document, and native config validity | Replace | Reduce it to native configuration validation; move explicit field assertions to the focused representative verifier. | Only native validity is a stable application outcome in this verifier. |
|
||||
| Audita | README validation claims | Description of default, metadata, rebuild, and platform tests | Replace | Describe only the retained static checks and behavioral scenarios. | The present prose promises low-value coverage that should be removed. |
|
||||
|
||||
### Audita Version-Coverage Gap
|
||||
|
||||
The current suite does not—and with the present application interface cannot—
|
||||
perform the useful test “select immutable release X and verify that the
|
||||
installed Audita reports X.” Running either `audita --version` or
|
||||
`audita version` fails as an unknown command. The recommended role-test
|
||||
remediation is to remove the misleading raw-commit scenario rather than
|
||||
replace it with another Git-module assertion. A native Audita release-identity
|
||||
command would enable the desired explicit-version test later, but adding that
|
||||
application feature is outside this audit and should be selected separately.
|
||||
|
||||
## Notarius Classification
|
||||
|
||||
| Repository | Test or case | Behavior protected | Classification | Recommended action | Rationale |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| Notarius | Default convergence | The default public interface installs and configures successfully | Retain | Keep as a smoke test, run `notarius --version` without fixing its output to the current default, and validate the configuration natively. | This proves the default path remains usable without snapshotting its values. |
|
||||
| Notarius | Default `v0.6.0` and commit verification | Present default release identity | Remove | Delete fixed `default_commit` and exact default-version assertions. | These fail solely when the supported default advances. |
|
||||
| Notarius | Source, binary, and configuration metadata | Trusted module postconditions | Remove | Delete ownership, mode, and regular-file assertions. | They re-test `git`, `file`, and `template`. |
|
||||
| Notarius | Exact minimal default document `{"version": 4}` | Current default rendering | Remove | Do not snapshot the default document. Supply `notarius_config_version` explicitly in the representative case and make any omission assertions about explicitly null inputs. | This preserves schema-bound behavior without turning future defaults into failures. |
|
||||
| Notarius | Native version command and configuration validation | Installed CLI is usable and accepts the managed configuration | Retain | Keep both native interfaces, but assert an exact version only in the explicitly selected release scenario. | These are application-level outcomes. |
|
||||
| Notarius | Baseline immediate repeat | Clean convergence | Retain | Keep one overall clean repeat. | This directly enforces the idempotence requirement. |
|
||||
| Notarius | Missing binary | The required output is rebuilt and then converges | Simplify | Retain artifact removal, native CLI verification, and the following clean repeat; remove task greps. | Missing-output recovery is role logic needed around the imperative build. |
|
||||
| Notarius | Dependency opt-out and custom Go executable | The caller-selected build executable is honored | Simplify | Keep wrapper invocation evidence and combine with missing-binary recovery; remove task greps and redundant repetitions. | This is meaningful configurable behavior with no better final-state discriminator. |
|
||||
| Notarius | Binary metadata repair | Built-in file convergence | Remove | Delete the complete scenario. | It tests trusted module behavior. |
|
||||
| Notarius | Explicit `v0.5.0` selection | A caller-selected immutable tag becomes the installed application | Simplify | Retain `notarius_version: v0.5.0` and exact native output `notarius v0.5.0`; remove expected commit and task-state assertions. | This remains useful after the role's default changes. |
|
||||
| Notarius | Restore `v0.6.0` default | Cleanup and present default identity | Simplify | Reduce to cleanup only. | It is not a distinct stable behavioral test. |
|
||||
| Notarius | Representative configuration | Field mapping, nested sections, pipelines, and validity | Simplify | Use a smaller set of explicit values and assert only those paths plus specifically selected null omissions; retain native validation. | Exact equality for a large complete document is broader and more brittle than the behavior requires. |
|
||||
| Notarius | Separate `profile_dir` case | One simple optional leaf mapping | Remove | Do not maintain a standalone scenario; a future representative case may select one profile source if it adds value. | The direct conditional and serialization are self-evident and already represented by other optional fields. |
|
||||
| Notarius | Separate `profile_file` case | A second simple optional leaf mapping | Remove | Delete the standalone scenario. | It duplicates the shape and risk of the profile-directory case without protecting a distinct likely regression. |
|
||||
| Notarius | Explicit falsey values | Zero, false, empty mapping, and empty string survive rendering | Retain | Fold representative falsey values into the focused configuration scenario and assert only those paths. | Null-versus-falsey behavior is a deliberate role invariant. |
|
||||
| Notarius | Caller-side pipeline composition | Independently defined mappings can be combined and passed through intact | Retain | Keep composition through ordinary `combine`, assert the two resulting pipeline entries, and run native validation; fold it into the representative configuration scenario if clarity permits. | This is an expressly selected public boundary and a plausible integration regression. |
|
||||
| Notarius | Configuration task changed while build task skipped | Configuration changes without rebuilding | Remove | Delete `require_configuration_change_without_build` and all task-trace greps. Verify final configuration, CLI usability, and clean repeat instead. | Task execution is not a stable public interface. |
|
||||
| Notarius | Restore the minimal default configuration | Cleanup plus mutable default assertions | Simplify | Keep cleanup only; remove exact default document, version, commit, task-state, and extra repeat checks. | The behavioral assertions duplicate change detectors elsewhere. |
|
||||
| Notarius | Platform failure matrix | Straightforward platform assertions | Remove | Delete the matrix and `tests/unsupported.yml`. | It provides no proportionate role-specific value. |
|
||||
| Notarius | Final commit, metadata, version, and config checks | Duplicate assertions | Remove | Delete them. | They repeat earlier verification and freeze the current default. |
|
||||
| Notarius | `tests/verify.yml` broad installation contract | Commit, metadata, default document, version, and validity | Replace | Reduce to native CLI smoke and configuration validation, with exact version verification only for an explicit tag. | This retains application outcomes while removing module tests and mutable snapshots. |
|
||||
| Notarius | README validation claims | Description of the current broad suite | Replace | Rewrite after simplification to describe the retained behavior. | Documentation should match the lean suite rather than preserve obsolete coverage promises. |
|
||||
|
||||
## Proposed Lean Target Suites
|
||||
|
||||
### Seriatim
|
||||
|
||||
1. Install pinned collections, syntax-check the convergence play, and lint the
|
||||
role and retained test files.
|
||||
2. Apply the role with defaults as a smoke test and immediately require a clean
|
||||
repeat, without asserting the current default tag or commit.
|
||||
3. Select `v1.4.0` explicitly and verify native output
|
||||
`seriatim version v1.4.0`.
|
||||
4. Delete the binary, supply a wrapper through `seriatim_go_binary` with
|
||||
dependency management disabled, converge, verify that the wrapper was used
|
||||
and the installed CLI works, then require a clean repeat.
|
||||
5. Restore defaults as cleanup without treating the restored value as a test
|
||||
expectation.
|
||||
|
||||
### Audita
|
||||
|
||||
1. Install pinned collections, syntax-check the convergence play, and lint the
|
||||
role and retained test files.
|
||||
2. Apply the role with defaults, validate the managed configuration with
|
||||
Audita, and immediately require a clean repeat. Do not assert the complete
|
||||
default document or current source identity.
|
||||
3. Apply one representative explicit configuration spanning nested, list,
|
||||
scalar, duration, falsey, and explicit-null behavior. Assert only the
|
||||
selected paths and omissions, validate it with Audita, and require a clean
|
||||
repeat.
|
||||
4. Delete the binary, supply a logging wrapper through `audita_go_binary` with
|
||||
dependency management disabled, converge, verify wrapper use and CLI
|
||||
usability, then require a clean repeat.
|
||||
5. Omit version-selection coverage until Audita exposes a native release
|
||||
identity; do not substitute a raw-commit checkout test.
|
||||
6. Restore defaults as cleanup only.
|
||||
|
||||
### Notarius
|
||||
|
||||
1. Install pinned collections, syntax-check the convergence play, and lint the
|
||||
role and retained test files.
|
||||
2. Apply the role with defaults, exercise `notarius --version`, validate the
|
||||
managed configuration, and immediately require a clean repeat without
|
||||
asserting the current default values.
|
||||
3. Select `v0.5.0` explicitly and verify native output `notarius v0.5.0`.
|
||||
4. Delete the binary, supply a logging wrapper through `notarius_go_binary`
|
||||
with dependency management disabled, converge, verify wrapper use and CLI
|
||||
usability, then require a clean repeat.
|
||||
5. Apply one representative explicit configuration that includes selected
|
||||
nested values, explicit falsey and null values, and two caller-composed
|
||||
pipeline mappings. Assert only those behaviors, validate the file with
|
||||
Notarius, and require a clean repeat.
|
||||
6. Restore defaults as cleanup only.
|
||||
|
||||
## Documentation Follow-up
|
||||
|
||||
The README validation sections in all three role repositories should be
|
||||
rewritten with their test harnesses. They currently advertise current-default,
|
||||
metadata, task-arrangement, and platform-guard coverage recommended for
|
||||
removal.
|
||||
|
||||
The durable feature roadmaps may continue to record the selected default tags,
|
||||
paths, ownership, permissions, and platform support as feature requirements.
|
||||
Those requirements need not each have a dedicated integration assertion.
|
||||
Test-specific roadmap language should, however, be corrected as follows:
|
||||
|
||||
- Feature 0001 and Feature 0003 should describe exact native version checks as
|
||||
checks of an explicitly selected immutable tag, not the current default.
|
||||
- Feature 0002 currently says repository-local tests execute Audita to verify
|
||||
its installed version. That is not implementable with Audita's current CLI
|
||||
and should be replaced with the limitation and follow-up described above.
|
||||
- Platform support remains part of each role contract, but the roadmaps should
|
||||
not imply that each straightforward assertion requires a negative matrix.
|
||||
|
||||
## Recommended Remediation Order
|
||||
|
||||
1. Simplify the Seriatim driver and verifier, remove its unsupported-platform
|
||||
playbook, and update its README.
|
||||
2. Simplify Audita's driver and verifiers, consolidate its configuration case,
|
||||
remove its unsupported-platform playbook, and update its README.
|
||||
3. Simplify Notarius's driver and verifiers, consolidate configuration and
|
||||
pipeline coverage, remove its unsupported-platform playbook, and update its
|
||||
README.
|
||||
4. Correct the test-specific language in the three feature roadmaps.
|
||||
5. Run all three complete integration entry points and confirm every role
|
||||
repository remains clean after its cleanup path.
|
||||
|
||||
No shared test abstraction is recommended. The three shell drivers have
|
||||
similar mechanics, but keeping each suite local and readable is worth the
|
||||
small amount of duplication, and a shared helper would create cross-repository
|
||||
coupling for little present benefit.
|
||||
|
||||
## Audit Completion
|
||||
|
||||
Every materially distinct implemented scenario has been classified above. The
|
||||
recommended removals either duplicate retained application-level coverage or
|
||||
exercise behavior owned by trusted Ansible modules or straightforward
|
||||
assertions. The lean targets retain default smoke coverage, immediate clean
|
||||
convergence, imperative missing-output recovery, caller-supplied build tooling,
|
||||
explicit release selection where natively observable, and representative
|
||||
configuration behavior.
|
||||
|
||||
The audit made no changes to production roles or test implementations.
|
||||
Remediation should begin only after these findings and the Audita
|
||||
release-identity recommendation have been reviewed.
|
||||
192
docs/roadmap/test-suite-audit.md
Normal file
192
docs/roadmap/test-suite-audit.md
Normal file
@@ -0,0 +1,192 @@
|
||||
# Test Suite Audit Plan
|
||||
|
||||
Status: complete
|
||||
|
||||
## Objective
|
||||
|
||||
Audit the implemented integration tests for the three completed standalone
|
||||
roles against the workspace's classical, state-based testing policy. The audit
|
||||
will identify tests that protect stable role-owned behavior, tests that should
|
||||
be simplified or replaced, and tests whose maintenance cost exceeds their
|
||||
value.
|
||||
|
||||
This is a review plan, not authorization to change role code or test suites.
|
||||
Any proposed remediation will be reviewed before a separate implementation
|
||||
plan is prepared.
|
||||
|
||||
## Scope
|
||||
|
||||
The audit covers the test harnesses, playbooks, fixtures, inventories, and
|
||||
test-related documentation in:
|
||||
|
||||
- `ansible-role-seriatim`
|
||||
- `ansible-role-audita`
|
||||
- `ansible-role-notarius`
|
||||
|
||||
Production role tasks, defaults, templates, and metadata may be inspected when
|
||||
needed to determine whether a test protects public role behavior or merely
|
||||
repeats an implementation detail. They are not remediation targets during the
|
||||
audit.
|
||||
|
||||
The audit does not cover the proof-of-concept `rakestraw-ansible` role, the
|
||||
suite-level role, or the collection. Those projects do not form part of the
|
||||
three completed standalone-role test suites.
|
||||
|
||||
## Governing Standard
|
||||
|
||||
Apply the normative testing house style in `docs/conventions.md`. In
|
||||
particular, the audit will favor tests that exercise the role through its
|
||||
public variables and inspect meaningful final application behavior. It will
|
||||
disfavor tests that:
|
||||
|
||||
- freeze current defaults or other intentionally changeable values;
|
||||
- assert task names, order, individual task status, or other execution traces;
|
||||
- independently re-prove the documented postconditions of trusted Ansible
|
||||
modules;
|
||||
- exhaustively exercise self-evident assertions or equivalent low-value
|
||||
combinations; or
|
||||
- duplicate another test without protecting a distinct requirement or
|
||||
plausible regression.
|
||||
|
||||
An immediate second convergence with no reported changes remains required for
|
||||
each role. An explicitly selected version may be verified through the
|
||||
application's native interface; the role's current default version may not be
|
||||
treated as a permanent expected value.
|
||||
|
||||
## Classification Rubric
|
||||
|
||||
Classify every behavioral test case, including each materially distinct case
|
||||
embedded in a shared playbook or shell harness, as one of the following:
|
||||
|
||||
- **Retain:** protects stable, role-owned behavior through an externally
|
||||
meaningful outcome.
|
||||
- **Simplify:** protects useful behavior but includes redundant assertions,
|
||||
duplicated setup, or implementation coupling that can be removed.
|
||||
- **Replace:** addresses a valid requirement through an unstable or indirect
|
||||
assertion and should be expressed as a state-based outcome instead.
|
||||
- **Remove:** tests a mutable default, trusted module behavior, an
|
||||
implementation trace, or behavior without a stated requirement or plausible
|
||||
regression.
|
||||
- **Add:** identifies a material role-owned behavior that currently lacks
|
||||
representative coverage.
|
||||
|
||||
Each classification must state the behavior at issue and the reason for the
|
||||
recommended action. File length, test count, and aesthetic preference are not
|
||||
sufficient reasons by themselves.
|
||||
|
||||
## Audit Record
|
||||
|
||||
Write the completed review to
|
||||
`docs/roadmap/test-suite-audit-findings.md`. Record each finding in a table with
|
||||
these fields:
|
||||
|
||||
| Repository | Test or case | Behavior protected | Classification | Recommended action | Rationale |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
|
||||
The report must also provide a proposed lean target suite for each role and
|
||||
identify any shared harness behavior that should be made consistent across the
|
||||
three repositories. Recommendations to remove coverage must state whether the
|
||||
behavior remains covered elsewhere or why it does not warrant coverage.
|
||||
|
||||
## Stage 1: Establish the Baseline
|
||||
|
||||
1. Confirm that the workspace and all three role repositories begin clean.
|
||||
2. Inventory every test entry point, playbook, fixture, inventory, helper, and
|
||||
test-related README statement in scope.
|
||||
3. Run each existing integration entry point without modification:
|
||||
|
||||
```text
|
||||
(cd ansible-role-seriatim && tests/integration)
|
||||
(cd ansible-role-audita && tests/integration)
|
||||
(cd ansible-role-notarius && tests/integration)
|
||||
```
|
||||
|
||||
4. Record success or failure, the scenarios exercised, and any environmental
|
||||
assumptions needed to reproduce the result. A pre-existing failure is audit
|
||||
evidence, not permission to repair it during this stage.
|
||||
|
||||
Completion criterion: the report contains a reproducible baseline and a
|
||||
complete test inventory for all three roles.
|
||||
|
||||
## Stage 2: Audit Seriatim
|
||||
|
||||
1. Map every Seriatim test case to a documented public behavior or classify it
|
||||
as lacking one.
|
||||
2. Review version selection, caller-supplied build tooling, rebuild behavior,
|
||||
metadata handling, platform validation, and repeat convergence using the
|
||||
common rubric.
|
||||
3. Identify assertions that inspect Ansible execution details or separately
|
||||
verify state already owned by trusted modules.
|
||||
4. Propose the smallest representative Seriatim suite that preserves valuable
|
||||
regression protection.
|
||||
|
||||
Completion criterion: every Seriatim case is classified and the report states
|
||||
the proposed retained, revised, removed, and missing coverage.
|
||||
|
||||
## Stage 3: Audit Audita
|
||||
|
||||
1. Map every Audita test case to a documented public behavior or classify it
|
||||
as lacking one.
|
||||
2. Review explicit version selection, native application verification,
|
||||
configuration rendering, omission of null settings, preservation of falsey
|
||||
values, platform validation, and repeat convergence.
|
||||
3. Distinguish tests of the role's configuration decisions from tests that
|
||||
merely repeat `template`, `file`, `git`, or package-module guarantees.
|
||||
4. Propose the smallest representative Audita suite that preserves valuable
|
||||
regression protection.
|
||||
|
||||
Completion criterion: every Audita case is classified and the report states
|
||||
the proposed retained, revised, removed, and missing coverage.
|
||||
|
||||
## Stage 4: Audit Notarius
|
||||
|
||||
1. Map every Notarius test case to a documented public behavior or classify it
|
||||
as lacking one.
|
||||
2. Review explicit version selection, native application verification,
|
||||
optional configuration rendering, falsey values, caller-composed pipeline
|
||||
mappings, platform validation, and repeat convergence.
|
||||
3. Distinguish tests of role-authored mapping and rendering behavior from
|
||||
assertions about trusted module mechanics or the current implementation
|
||||
layout.
|
||||
4. Propose the smallest representative Notarius suite that preserves valuable
|
||||
regression protection.
|
||||
|
||||
Completion criterion: every Notarius case is classified and the report states
|
||||
the proposed retained, revised, removed, and missing coverage.
|
||||
|
||||
## Stage 5: Reconcile the Three Suites
|
||||
|
||||
1. Compare classifications across roles so equivalent patterns receive
|
||||
equivalent treatment, while preserving genuine application differences.
|
||||
2. Identify duplicated harness machinery, inconsistent terminology, and
|
||||
documentation that describes implementation details or mutable defaults as
|
||||
promised test behavior.
|
||||
3. Separate static validation such as syntax checking and linting from
|
||||
behavioral integration coverage.
|
||||
4. Produce a repository-by-repository remediation proposal ordered by value
|
||||
and dependency. Do not propose a shared abstraction unless it removes
|
||||
demonstrated duplication without obscuring the individual role suites.
|
||||
5. Confirm that the proposed target for every role still includes a default or
|
||||
representative convergence, an immediate clean second convergence, and
|
||||
focused coverage of each retained role-owned behavior.
|
||||
|
||||
Completion criterion: the findings report is internally consistent, gives a
|
||||
lean target suite for each repository, and contains enough detail to support a
|
||||
later implementation plan without leaving test-policy decisions to the
|
||||
executor.
|
||||
|
||||
## Audit Completion Criteria
|
||||
|
||||
The audit is complete when:
|
||||
|
||||
- every implemented behavioral case across all three repositories has an
|
||||
explicit classification and rationale;
|
||||
- baseline results and reproduction commands are recorded;
|
||||
- proposed removals do not silently discard coverage of a required public
|
||||
behavior;
|
||||
- proposed additions are limited to material role-owned behavior;
|
||||
- README and harness-documentation corrections are identified alongside test
|
||||
changes;
|
||||
- no production or test implementation has been changed during the audit; and
|
||||
- the findings have been reviewed before any remediation plan is written to
|
||||
`docs/roadmap/implementation.md`.
|
||||
Reference in New Issue
Block a user