diff --git a/docs/conventions.md b/docs/conventions.md index 2faf7b6..809e7a7 100644 --- a/docs/conventions.md +++ b/docs/conventions.md @@ -71,6 +71,51 @@ control flow, and concise variable use, while retaining current practices such as fully qualified collection names. +## Testing House Style (Normative) + +- Use a classical state-based approach, also commonly described as Chicago or + Detroit style. Exercise the real role through its public interface and + observe the resulting deployed system. +- Test behavior owned by the role, not the current arrangement of tasks that + implements it. A useful test should normally remain valid after an internal + refactor that preserves the same public behavior. +- Require an immediate second role application to report no changes. Clean + repeated convergence is a first-class role outcome, not a test of any one + module. +- Distinguish configurable behavior from current defaults. To prove version + selection, explicitly request a version and verify that the application + reports that version. Do not assert that the same version remains the role's + default. For example, a useful test sets `audita_version: v0.9.0` and verifies + that the installed Audita reports `v0.9.0`; a harmful test asserts that + `v0.9.0` is the role's current default. The former continues to prove version + selection after the default advances to `v0.10.0`, while the latter fails + merely because of the intended update. +- Use a default invocation as a smoke test when useful, but do not snapshot + exact default values, selected commits, complete default configuration + documents, or counts of default variables. Those assertions are change + detectors that turn intentional updates into unrelated test failures. +- Do not assert task names, task order, individual `changed` or `skipped` + results, or other implementation traces. Inspect the final system and the + overall clean-repeat result instead. +- Do not test documented behavior of `ansible.builtin`, widely used + `community.general`, or other trusted modules. In particular, do not + independently verify file existence, ownership, permissions, package state, + checkout state, or template deployment merely to repeat module arguments. +- Test role-authored decisions only when they express meaningful public + behavior. Examples include an explicitly selected application version, a + caller-supplied build executable, configuration rendering with representative + non-null and falsey values, and composition of caller-supplied mappings. +- Verify application outcomes with the application's native interface when + practical. Prefer executing the installed application or its configuration + validator over reconstructing its rules in Ansible assertions. +- Self-evident declarative guards, including straightforward platform + assertions, do not require a negative test matrix. +- Prefer one representative case for a behavior over exhaustive combinations. + Every additional test must protect a requirement or a plausible regression, + not merely increase coverage. +- Keep syntax checking and linting as static validation. They complement the + behavioral suite but do not justify tests that freeze implementation details. + ## Application Configuration Interfaces - Expose application configuration through documented, field-specific role @@ -104,9 +149,11 @@ - Manage the resulting file with one ordinary `ansible.builtin.template` task. Do not add configuration assembly facts, preliminary file inspection, or production validation unless independently required. -- Tests should prove the minimal default document, representative rendering - across all sections, omission of null fields, preservation of intentional - falsey values, native configuration validity, and repeated convergence. +- Configuration tests should use explicitly supplied representative values to + exercise conditional rendering, null omission, intentional falsey values, + native configuration validity, and caller-facing composition boundaries. + They should follow the testing house style above rather than snapshotting the + current default document. Role defaults define what Ansible intends to manage, not a mirror of every application default. A concrete default means the role deliberately owns that diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 6bba461..f9f1c14 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,12 +1,505 @@ -# Implementation Plan +# Implementation Plan R0001: Classical Test Suite Remediation -Status: pending +Status: ready -No implementation plan is currently active. Feature 0003 was completed in -`ansible-role-notarius` commits `287fc4a`, `733a194`, and `eb4f511`. Its -durable scope remains in -[the Feature 0003 roadmap](features/0003-notarius-source-installation-and-configuration.md), -and its review history remains in [the remediation record](remediation.md). +Target executor: GPT-5.6-Terra -Select and approve the next feature roadmap before replacing this notice with -a staged implementation plan. +Authoritative audit: +`docs/roadmap/test-suite-audit-findings.md` + +Active plan location: `docs/roadmap/implementation.md` + +## Objective + +Replace the oversized, implementation-bound integration suites in the three +completed standalone role repositories with lean classical, state-based +suites. Retain tests of stable role-owned behavior and application outcomes; +remove mutable-default snapshots, task-trace assertions, tests of trusted +Ansible modules, redundant scenarios, and negative platform matrices. + +Production role behavior is not changing. This plan changes only test +implementations, test-related role documentation, and the workspace planning +records identified by the audit. + +## Execution Rules + +- Execute stages in numeric order. +- Before each stage, confirm its owning repository is clean and record its + starting revision. +- Preserve unrelated and pre-existing changes. +- Do not modify production tasks, defaults, templates, argument + specifications, dependency pins, or role metadata. If a retained test + exposes a production defect, stop and report it instead of expanding this + remediation. +- Follow the normative testing house style in `docs/conventions.md` and the + classifications in the authoritative audit. +- Exercise roles through their public variables and inspect final application + behavior. Do not parse task names, task order, or individual task status. +- Retain only aggregate `changed=0` checks on specified immediate repeat runs. + Do not require a nonzero change count on preceding runs. +- Do not verify ownership, permissions, path types, checkout commits, or other + postconditions already declared through trusted modules. +- Do not add a test merely to replace each removed test one-for-one. The target + is the smallest suite that protects the retained behavior. +- Keep each repository's shell driver self-contained. Do not create a shared + cross-repository test helper or abstraction. +- Run each stage's full repository integration entry point before committing. +- Commit each completed stage separately with the plain-English intent given + in that stage. +- Do not begin a dependent stage after an earlier stage fails validation. + +## Starting State and Preconditions + +- The workspace conventions, audit plan, audit findings, status update, and + this implementation plan must be reviewed and committed in the workspace + repository before Stage 1 begins. +- `ansible-role-seriatim` starts clean at `fb8dc41`. +- `ansible-role-audita` starts clean at `847c4c3`. +- `ansible-role-notarius` starts clean at `eb4f511`. +- The workspace `.venv` provides Ansible Core and ansible-lint, and the local + Ubuntu 26.04 `x86_64` host remains the approved MVP integration proxy. +- The executor can use passwordless privilege escalation for the existing + `/usr/local` integration-test paths and can reach Ansible Galaxy, the three + application Git repositories, Snap infrastructure, and Go module sources. +- Existing installed application state may be present. Each driver must remain + repeatable from that state and restore its default role deployment through a + best-effort cleanup trap. +- Audita exposes no native application-release identity command. Its tags + `v0.9.0` and `v0.9.1` currently resolve to the same commit. This plan + deliberately omits an Audita version-selection test rather than substituting + a raw-commit or Git-module assertion. + +## Repository Scope + +Repositories that change: + +- `ansible-role-seriatim`: lean integration driver and verifier, removal of + the unsupported-platform playbook, and accurate validation documentation. +- `ansible-role-audita`: lean integration driver and verifiers, removal of the + unsupported-platform playbook, and accurate validation documentation. +- `ansible-role-notarius`: lean integration driver and verifiers, removal of + the unsupported-platform playbook, and accurate validation documentation. +- Workspace control repository: test-specific corrections to Features 0001, + 0002, and 0003, remediation status, and cross-repository validation evidence. + +Repositories that may be inspected but must remain unchanged: + +- `../seriatim`, `../audita`, and `../notarius` for native CLI behavior. +- Every other application, role, collection, and proof-of-concept repository. + +## Stage 1: Simplify the Seriatim Test Suite + +### Objective + +Reduce Seriatim's integration suite to static validation and representative +tests of default usability, explicit version selection, missing-binary +recovery, caller-supplied Go, and clean repeated convergence. + +### Preconditions + +- The workspace planning commit described above exists. +- `ansible-role-seriatim` is clean at `fb8dc41`. +- The existing `tests/integration` baseline passes. + +### Owning Repository + +`ansible-role-seriatim` + +### Required Changes + +1. Retain `tests/integration` as the single entry point, its private temporary + directory, pinned collection installation, syntax check, production-profile + ansible-lint, and best-effort cleanup convergence. +2. Retain a default role convergence as a smoke test. Exercise + `/usr/local/bin/seriatim --version` successfully without comparing it to the + current default version, then immediately apply the default role again and + require only the aggregate recap `changed=0` and `failed=0`. +3. Add or retain one explicit version-selection scenario using + `seriatim_version: v1.4.0`. Verify through the native CLI that the result is + exactly `seriatim version v1.4.0`. Do not resolve or assert the corresponding + Git commit. +4. Combine missing-binary recovery and caller-supplied build tooling into one + representative scenario: + - create an executable temporary wrapper that records invocation and then + executes `/snap/bin/go` with the original arguments; + - remove `/usr/local/bin/seriatim`; + - apply the role with `seriatim_version: v1.4.0`, + `seriatim_manage_build_dependencies: false`, and + `seriatim_go_binary` set to the wrapper; + - require the wrapper log to show that the supplied executable was used; + - verify native output `seriatim version v1.4.0`; and + - immediately repeat with the same variables and require aggregate + `changed=0` and `failed=0`. +5. Delete `tests/verify.yml` and perform the narrow native version assertions + directly in the shell driver. Do not retain source, commit, ownership, + permission, file-type, or checksum inspection. +6. Remove the binary-mode mutation and repair scenario, every + `require_changed_recap` use, every task-name or individual-status grep, the + default and previous commit constants, redundant default-restoration + assertions, and final duplicate shell assertions. +7. Delete `tests/unsupported.yml` and remove it from lint and execution. Do not + replace its distribution, release, or architecture cases. +8. Keep a cleanup trap that reapplies the default role and removes only the + private temporary directory. Cleanup output may remain suppressed and must + not become another behavioral assertion. +9. Rewrite the README validation section to describe only the retained static + checks and behavioral scenarios. Do not advertise exact-default, metadata, + task-arrangement, or platform-matrix coverage. + +### Validation + +Run: + +```text +tests/integration +git diff --check +git status --short +``` + +Expected result: static validation and every retained Seriatim scenario pass; +the specified repeat runs report no changes; cleanup completes; and the only +working-tree changes are the planned test and README edits. + +### Completion Criteria + +- No test asserts Seriatim's current default version or a Git commit. +- No test inspects task names, individual task status, ownership, permissions, + or the platform assertion branches. +- Explicit `v1.4.0` selection is verified through Seriatim itself. +- One scenario proves both missing-binary recovery and use of the configured Go + executable, followed by a clean repeat. +- The complete integration entry point passes. + +### Commit + +Commit intent: `Simplify the Seriatim integration tests` + +## Stage 2: Simplify the Audita Test Suite + +### Objective + +Reduce Audita's integration suite to static validation and representative +tests of default usability, native configuration validity, explicit +configuration rendering, missing-binary recovery, caller-supplied Go, and clean +repeated convergence. + +### Preconditions + +- Stage 1 is complete and committed. +- `ansible-role-audita` is clean at `847c4c3`. +- The existing `tests/integration` baseline passes. + +### Owning Repository + +`ansible-role-audita` + +### Required Changes + +1. Retain `tests/integration` as the single entry point, its private temporary + directory, pinned collection installation, syntax check, production-profile + ansible-lint, and best-effort cleanup convergence. +2. Retain a default role convergence as a smoke test. Validate the managed + configuration with + `/usr/local/bin/audita config validate --config /usr/local/etc/audita/config.yml`, + then immediately repeat the default role and require only aggregate + `changed=0` and `failed=0`. +3. Replace the exhaustive configuration document and separate falsey scenario + with one representative explicit configuration. Supply these values: + - `audita_config_version: 1`; + - `audita_pipeline_modules: ["glossary"]`; + - `audita_llm_proposal_base_url: "https://openrouter.ai/api/v1"`; + - `audita_llm_proposal_model: "openrouter/google/gemma-4-31b-it"`; + - `audita_llm_proposal_timeout: "10s"`; + - `audita_llm_proposal_max_retries: 0`; + - `audita_context_description: ""`; + - `audita_diagnostics_retention: "never"`; + - `audita_llm_validation_model: null`; + - `audita_output_schema: null`; and + - `audita_diagnostics_work_dir: null`. +4. In `tests/verify-config.yml`, parse the rendered YAML and assert only that: + - `version` is `1`; + - `pipeline.modules` is `["glossary"]`; + - the selected proposal URL, model, timeout, and zero retry count are + present; + - the empty context description and diagnostics retention value are + present; + - `llm.validation`, `output`, and `diagnostics.work_dir` are absent; and + - Audita's native configuration validation command succeeds. + Do not compare the complete parsed document to an expected mapping. +5. Immediately repeat the representative configuration convergence with the + same explicit variables and require aggregate `changed=0` and `failed=0`. +6. Combine missing-binary recovery and caller-supplied build tooling into one + representative scenario. Create a logging wrapper around `/snap/bin/go`, + remove `/usr/local/bin/audita`, converge with + `audita_manage_build_dependencies: false` and `audita_go_binary` set to the + wrapper, verify wrapper use, validate the managed configuration through + Audita, and immediately repeat the same convergence with aggregate + `changed=0` and `failed=0`. +7. Delete `tests/verify.yml` and invoke Audita's native configuration validator + directly from the shell driver for the smoke and recovery scenarios. + Successful command completion is sufficient; do not separately assert the + validator's fixed success text. +8. Remove the exact default configuration comparison, source-commit checks, + source/binary/configuration metadata checks, binary-mode repair scenario, + all `require_changed_recap` uses, and every task-name or individual-status + grep. +9. Remove the raw alternate-commit scenario entirely. Do not replace it with a + `git rev-parse`, tag-resolution, embedded-string, or Go build-metadata test. + Do not add an Audita version-selection test until the application provides a + stable native release-identity interface. +10. Delete `tests/unsupported.yml` and remove it from lint and execution. Do not + replace its platform cases. +11. Retain default restoration only in the cleanup trap. Remove its fixed + commit, exact document, task-status, and extra idempotence assertions. +12. Rewrite the README validation section to describe only the retained static + checks and behavioral scenarios and to avoid claiming application-version, + metadata, task-arrangement, or platform-matrix coverage. + +### Validation + +Run: + +```text +tests/integration +git diff --check +git status --short +``` + +Expected result: static validation and every retained Audita scenario pass; +both default and representative configurations validate natively; the +specified repeats report no changes; cleanup completes; and the only +working-tree changes are the planned test and README edits. + +### Completion Criteria + +- No test asserts Audita's current default, a source commit, or a complete + rendered configuration document. +- No test inspects task traces, trusted-module postconditions, or platform + assertion branches. +- One focused explicit case proves representative nesting, null omission, and + falsey preservation and passes Audita's validator. +- One scenario proves both missing-binary recovery and use of the configured Go + executable, followed by a clean repeat. +- The absence of native Audita version coverage is documented accurately and + is not replaced by an implementation proxy. +- The complete integration entry point passes. + +### Commit + +Commit intent: `Simplify the Audita integration tests` + +## Stage 3: Simplify the Notarius Test Suite + +### Objective + +Reduce Notarius's integration suite to static validation and representative +tests of default usability, explicit version selection, missing-binary +recovery, caller-supplied Go, configuration behavior, caller-side pipeline +composition, native validation, and clean repeated convergence. + +### Preconditions + +- Stages 1 and 2 are complete and committed. +- `ansible-role-notarius` is clean at `eb4f511`. +- The existing `tests/integration` baseline passes. + +### Owning Repository + +`ansible-role-notarius` + +### Required Changes + +1. Retain `tests/integration` as the single entry point, its private temporary + directory, pinned collection installation, syntax check, production-profile + ansible-lint, and best-effort cleanup convergence. +2. Retain a default convergence as a smoke test. Require + `/usr/local/bin/notarius --version` and + `/usr/local/bin/notarius config validate --config /usr/local/etc/notarius/config.yml` + to succeed without comparing the version output or complete configuration + to current defaults. Immediately repeat the default role and require only + aggregate `changed=0` and `failed=0`. +3. Retain one explicit version-selection scenario with + `notarius_version: v0.5.0` and verify exact native output + `notarius v0.5.0`. Do not resolve or assert the corresponding Git commit. +4. Combine missing-binary recovery and caller-supplied build tooling into one + scenario. Create a logging wrapper around `/snap/bin/go`, remove + `/usr/local/bin/notarius`, converge with `notarius_version: v0.5.0`, + `notarius_manage_build_dependencies: false`, and `notarius_go_binary` set to + the wrapper, verify wrapper use and native output `notarius v0.5.0`, then + immediately repeat with aggregate `changed=0` and `failed=0`. +5. Replace the exhaustive configuration, separate profile-source cases, + separate falsey case, and separate pipeline-composition case with one + representative explicit configuration. Define two caller-side pipeline + mappings and set `notarius_pipelines` to their ordinary `combine` result. + Supply these additional values: + - `notarius_config_version: 4`; + - `notarius_promptkit_local_backend_endpoint: "http://localhost:8000/v1"`; + - `notarius_promptkit_local_backend_concurrency_limit: 0`; + - `notarius_cache_checkpoints_enabled: false`; + - `notarius_cache_chunk_plans_directory: ""`; + - `notarius_output_directory: "/tmp/notarius-output"`; + - `notarius_promptkit_profile_dir: null`; + - `notarius_promptkit_profile_file: null`; and + - `notarius_debug_directory: null`. + Use the two valid `dnd-session` and `dnd-archive` pipeline definitions from + the existing composition scenario. +6. In `tests/verify-config.yml`, parse the rendered YAML and assert only that: + - `version` is `4`; + - local-backend endpoint and zero concurrency are present; + - checkpoints are explicitly disabled and the empty chunk-plan directory is + present; + - the selected output directory is present; + - both composed pipeline IDs and their supplied definitions are present + without rewriting; + - `promptkit.profile_dir`, `promptkit.profile_file`, and the top-level + `debug` section are absent; and + - Notarius's native configuration validation succeeds. + Do not compare the complete document to an expected mapping. +7. Immediately repeat the representative configuration convergence with the + same explicit variables and require aggregate `changed=0` and `failed=0`. +8. Delete `tests/verify.yml`. Perform native CLI smoke, configuration + validation, and the explicit `v0.5.0` comparison directly in the shell + driver. +9. Remove current-default version and commit assertions, complete default + configuration comparisons, all source/binary/configuration metadata checks, + the binary-mode repair scenario, every `require_changed_recap` use, + `require_configuration_change_without_build`, and every task-name or + individual-status grep. +10. Delete the standalone `profile_dir` and `profile_file` scenarios. Their + simple leaf mappings do not require dedicated replacements. +11. Delete `tests/unsupported.yml` and remove it from lint and execution. Do not + replace its platform cases. +12. Retain default restoration only in the cleanup trap and remove duplicate + final assertions. +13. Rewrite the README validation section to describe only the retained static + checks and behavioral scenarios. Do not advertise current-default, + metadata, task-arrangement, profile-branch, or platform-matrix coverage. + +### Validation + +Run: + +```text +tests/integration +git diff --check +git status --short +``` + +Expected result: static validation and every retained Notarius scenario pass; +the explicit tag is verified through Notarius; representative falsey, null, +and composed-pipeline behavior is verified and validates natively; the +specified repeats report no changes; cleanup completes; and only the planned +test and README changes remain. + +### Completion Criteria + +- No test snapshots Notarius's current default release, commit, or complete + default configuration. +- No test inspects task traces, trusted-module postconditions, or platform + assertion branches. +- Explicit `v0.5.0` selection is verified through Notarius itself. +- One scenario proves both missing-binary recovery and use of the configured Go + executable, followed by a clean repeat. +- One representative configuration case proves null omission, falsey + preservation, caller-side pipeline composition, and native validity. +- The complete integration entry point passes. + +### Commit + +Commit intent: `Simplify the Notarius integration tests` + +## Stage 4: Reconcile Workspace Documentation and Validate the Suite + +### Objective + +Align durable test-specific roadmap language with the canonical testing policy, +record the remediation implementation for review, and validate all three lean +suites together. + +### Preconditions + +- Stages 1 through 3 are complete and committed in their owning repositories. +- The workspace repository is clean at the planning commit created before + Stage 1. +- The three role repositories contain only their committed remediation changes. + +### Owning Repository + +Workspace control repository (`ansible-workspace-dnd`) + +### Required Changes + +1. Revise only test-specific language in Features 0001, 0002, and 0003. Retain + their durable production requirements, including current defaults, paths, + ownership, permissions, platform support, and convergence behavior. +2. In Feature 0001, describe the exact native version assertion as coverage of + an explicitly selected immutable tag rather than an assertion about the + role's current default. +3. In Feature 0002, remove the incorrect claim that tests execute Audita to + verify its installed application version. State that the current CLI has no + native release-identity command and that the suite deliberately avoids a + raw-commit proxy. Retain native configuration validation. +4. In Feature 0003, describe exact version coverage using explicitly selected + `v0.5.0`, not the current default `v0.6.0`. Keep the current default as a + production requirement rather than a frozen test expectation. +5. Remove any implication in the three roadmaps that each platform assertion + must have a negative integration-test matrix. Do not remove platform support + itself from scope or acceptance criteria. +6. Update `docs/roadmap/test-suite-audit-findings.md` to say remediation was + implemented and is pending review. Add the three role commit hashes and the + final validation results; do not erase the audit classifications. +7. Update `docs/status.md` so `R0001` remains `in_progress` pending review, its + evidence names the three role commits, and the current focus is review of + the implemented test-suite remediation. Do not mark remediation `done`. +8. Add concise progress notes to this plan with the four stage commits and any + deviations. Do not replace the active plan with a pending notice before + review. + +### Validation + +Run sequentially from the workspace root: + +```text +(cd ansible-role-seriatim && tests/integration) +(cd ansible-role-audita && tests/integration) +(cd ansible-role-notarius && tests/integration) +git diff --check +scripts/status +``` + +Expected result: all static and behavioral checks pass; every role cleanup path +completes; the three role repositories are clean and contain their remediation +commits; no unrelated repository changed; and the workspace has only the +planned documentation changes before its stage commit. + +### Completion Criteria + +- All three full integration suites pass sequentially from the workspace. +- Feature roadmaps distinguish production defaults from stable test inputs. +- Audita's native version limitation is stated accurately. +- Audit findings contain implementation and validation evidence without being + rewritten as an implementation log. +- Workspace status leaves remediation pending review. +- Every repository changed by the implementation is clean after its owning + commit. + +### Commit + +Commit intent: `Record the test suite remediation results` + +## Final Validation and Handoff + +- Report the Stage 1 through Stage 4 commit hashes and confirm the working-tree + state of every repository in `workspace.yml`. +- Summarize the retained scenarios for each role and identify any deviation + from this plan. +- Report the Audita version-identity limitation; do not silently fill it with a + source-control assertion. +- Report all validation commands and results. +- Do not push commits unless the user separately authorizes a push. +- Do not mark `R0001` done or replace this plan with a pending notice. Final + completion requires review of the implemented suites. + +## Progress Notes + +- Planning complete; implementation has not started. diff --git a/docs/roadmap/test-suite-audit-findings.md b/docs/roadmap/test-suite-audit-findings.md new file mode 100644 index 0000000..7215c7c --- /dev/null +++ b/docs/roadmap/test-suite-audit-findings.md @@ -0,0 +1,260 @@ +# Test Suite Audit Findings + +Status: complete; remediation pending review + +## Executive Summary + +All three integration suites pass, and each includes the most important common +behavioral check: an immediate repeated role application reports no changes. +The suites also contain useful coverage of explicitly selected releases, +caller-supplied Go executables, missing-binary recovery, configuration +rendering, native configuration validation, and Notarius pipeline composition. + +The suites are nevertheless substantially larger and more implementation-bound +than the workspace testing policy permits. The recurring problems are: + +- exact assertions about the current default release and its commit; +- independent verification of ownership, permissions, checkout state, and + template output already managed by trusted Ansible modules; +- parsing Ansible output for specific task names and individual + `changed`/`skipped` results; +- deliberate metadata drift followed by tests of `ansible.builtin.file`; +- three-case negative tests of self-evident platform assertions; +- repeated clean-run assertions after nearly every scenario; and +- duplicated final shell assertions after equivalent Ansible verifiers. + +The appropriate remediation is subtraction and consolidation. The retained +suites should exercise stable role-owned behavior through final application +outcomes, with one representative case for each meaningful behavior. + +## Baseline + +The audit ran on Ubuntu 26.04 `x86_64` with Ansible Core 2.21.3 and +ansible-lint 26.8.0. The role revisions were: + +| Repository | Revision | Integration result | +| --- | --- | --- | +| `ansible-role-seriatim` | `fb8dc41` | Pass | +| `ansible-role-audita` | `847c4c3` | Pass | +| `ansible-role-notarius` | `eb4f511` | Pass | + +The following commands all exited successfully: + +```text +(cd ansible-role-seriatim && tests/integration) +(cd ansible-role-audita && tests/integration) +(cd ansible-role-notarius && tests/integration) +``` + +Each driver installed its pinned collection requirements into a private +temporary directory, ran a syntax check and production-profile ansible-lint, +executed its behavioral scenarios, and restored its default deployment during +cleanup. The runs emitted non-fatal local-tooling warnings about Python +interpreter discovery or an already installed duplicate collection; these did +not affect the results or any classification below. + +All three role repositories were clean before and after the audit. The control +repository began dirty solely because the new testing convention, audit plan, +and status update were awaiting commit. + +## Test Inventory + +| Repository | Entry point and supporting files | +| --- | --- | +| `ansible-role-seriatim` | `tests/integration`, `tests/inventory`, `tests/converge.yml`, `tests/verify.yml`, `tests/unsupported.yml`, `collections/requirements.yml`, and the README validation section | +| `ansible-role-audita` | `tests/integration`, `tests/inventory`, `tests/converge.yml`, `tests/verify.yml`, `tests/verify-config.yml`, `tests/unsupported.yml`, `collections/requirements.yml`, and the README validation section | +| `ansible-role-notarius` | `tests/integration`, `tests/inventory`, `tests/converge.yml`, `tests/verify.yml`, `tests/verify-config.yml`, `tests/unsupported.yml`, `collections/requirements.yml`, and the README validation section | + +## Cross-Repository Findings + +| Repository | Test or case | Behavior protected | Classification | Recommended action | Rationale | +| --- | --- | --- | --- | --- | --- | +| All three | Pinned collection installation, syntax check, and ansible-lint | Static validity and reproducible dependencies | Retain | Keep as static validation before behavioral scenarios. | These checks complement rather than duplicate behavioral testing. | +| All three | `require_clean_recap` after an immediate repeat | Role-level repeated convergence | Retain | Keep at least one immediate clean repeat per role and repeats specifically required after missing-binary recovery. | Overall `changed=0` is a first-class public role outcome. | +| All three | `require_changed_recap` after each mutating setup | Ansible reports some change on the first run | Remove | Stop asserting a nonzero aggregate change count except where a future requirement specifically makes reporting itself significant. | Final application state is the behavior of interest; these checks add little beyond successful convergence and make scenario sequencing brittle. | +| All three | Greps for named build, metadata, and configuration tasks | Current task names and individual `changed` or `skipped` results | Remove | Delete all task-output parsing. | These are direct implementation traces and fail under behavior-preserving task refactors. | +| All three | Binary/source/configuration ownership and mode assertions | Postconditions declared to `file`, `git`, and `template` | Remove | Delete independent metadata and recursive-ownership verification. | These tests re-prove trusted module behavior rather than role-authored decisions. | +| All three | Binary mode mutation followed by convergence | `ansible.builtin.file` repairs mode drift without building | Remove | Delete the scenario. | It is a focused regression test for the built-in file module and current task arrangement. | +| All three | Debian, old Ubuntu, and unsupported-architecture matrix | Straightforward platform assertion branches | Remove | Delete `tests/unsupported.yml` and the three-case driver loop. | The declarative assertion is self-evident and does not warrant a negative matrix. | +| All three | Restoration of the current default followed by verification and another repeat | Test cleanup and the current default identity | Simplify | Keep one best-effort cleanup convergence, but do not treat it as another test or assert the restored release value. | Cleanup is useful; testing a mutable default and repeating already established idempotence are not. | +| All three | Final shell assertions duplicating earlier verifiers | Current commit, metadata, version, or config state | Remove | Delete duplicate end-of-driver assertions. | They add maintenance cost without protecting a distinct behavior. | + +## Seriatim Classification + +| Repository | Test or case | Behavior protected | Classification | Recommended action | Rationale | +| --- | --- | --- | --- | --- | --- | +| Seriatim | Default convergence | The role can be applied with its default interface | Retain | Keep as a smoke test without asserting the exact default tag, commit, or filesystem metadata. | A default invocation is useful even though the present default value is not a stable test expectation. | +| Seriatim | Default `v1.5.0` and commit verification | Present default release identity | Remove | Remove the fixed `default_commit` and exact default-version assertions. | This is a change detector; advancing the supported default should not break unrelated tests. | +| Seriatim | Baseline immediate repeat | Clean convergence | Retain | Keep the overall `changed=0`, `failed=0` check. | This directly expresses the agreed idempotence requirement. | +| Seriatim | Missing binary | The imperative build runs when its output is absent and then converges | Simplify | Delete the binary, apply the role, verify the executable through `--version`, and require the next run to be clean; remove task-name greps. | The binary-existence trigger is role-authored behavior, while the current task name and status are not. | +| Seriatim | Dependency opt-out and custom Go executable | The caller-supplied build executable is honored | Simplify | Retain the wrapper invocation evidence, preferably combined with the missing-binary scenario; remove build-task greps and fixed-default assertions. | The selected build executable is a public role choice that cannot be observed reliably from the final binary alone. | +| Seriatim | Binary metadata repair | Built-in file convergence | Remove | Delete the complete scenario. | It tests `ansible.builtin.file`, not Seriatim-specific behavior. | +| Seriatim | Explicit `v1.4.0` selection | A caller-selected immutable release is installed | Simplify | Retain explicit `seriatim_version: v1.4.0` and verify `seriatim version v1.4.0`; remove the expected commit and task-state assertions. | This is the stable version-selection test described by the testing convention. | +| Seriatim | Restore `v1.5.0` default | Cleanup plus present default identity | Simplify | Reduce to cleanup only. | Exact current-default verification duplicates the harmful baseline snapshot. | +| Seriatim | Platform failure matrix | Distribution, release, and architecture assertions | Remove | Delete the matrix and `tests/unsupported.yml`. | No plausible role regression justifies separately testing each simple assertion. | +| Seriatim | `tests/verify.yml` installation contract | Commit, source metadata, binary metadata, and version | Replace | Replace the broad verifier with a narrow native version check used for an explicitly selected tag. | Only the application outcome is valuable; the remaining assertions repeat trusted modules. | +| Seriatim | README validation claims | Description of the current large matrix | Replace | Describe the lean retained behaviors after remediation. | Documentation should not promise deleted implementation-trace and module tests. | + +## Audita Classification + +| Repository | Test or case | Behavior protected | Classification | Recommended action | Rationale | +| --- | --- | --- | --- | --- | --- | +| Audita | Default convergence | The role installs and configures successfully through its default interface | Retain | Keep as a smoke test and validate the resulting configuration with Audita; do not inspect exact default identity or metadata. | Successful native validation is an application outcome and does not freeze the default document. | +| Audita | Default commit assertion | Present default checkout identity | Remove | Delete `default_commit` and exact commit verification. | It is both a mutable-default snapshot and a re-test of `ansible.builtin.git`. | +| Audita | Source, binary, and configuration metadata | Trusted module postconditions | Remove | Delete the ownership, permission, regular-file, and executable assertions. | The role directly declares these states with trusted modules. | +| Audita | Exact minimal default document `{"version": 1}` | Current default rendering | Remove | Do not snapshot the complete default document. Test null omission with explicitly supplied inputs in a representative configuration case instead. | A future intentional concrete default would otherwise cause an unrelated failure. | +| Audita | Native validation of the managed configuration | The role produces configuration Audita accepts | Retain | Keep one native validation for default smoke coverage and one for the representative explicit configuration. | This observes application behavior rather than reconstructing its schema. | +| Audita | Baseline immediate repeat | Clean convergence | Retain | Keep one role-level clean repeat. | This is required public behavior. | +| Audita | Missing binary | Absent build output is recreated and convergence resumes | Simplify | Retain recovery and the following clean repeat; prove usability with the Audita CLI and remove task-output greps. | The imperative build trigger is role logic, but its task name is not. | +| Audita | Dependency opt-out and custom Go executable | Caller-selected compiler path is used | Simplify | Retain wrapper invocation evidence and combine it with missing-binary recovery; remove task greps and redundant repeat runs. | This covers a documented public choice with one representative case. | +| Audita | Binary metadata repair | Built-in file convergence | Remove | Delete the complete scenario. | It tests Ansible's file module. | +| Audita | Representative configuration containing nearly every field | Field-to-YAML mapping, nested sections, and native validity | Simplify | Use a smaller explicit sample spanning representative scalar, list, nested, and duration shapes. Assert only selected paths and specifically selected omissions, then run native validation. | Full-document equality across nearly every field unnecessarily ossifies the template and public interface. | +| Audita | Explicit falsey configuration | Empty lists and strings are rendered rather than omitted | Retain | Keep representative falsey values, preferably in the same explicit configuration scenario, and assert only the affected paths. | The null-versus-falsey distinction is role-authored and easy to regress. | +| Audita | Configuration task changed while build task skipped | Configuration changes independently of installation | Remove | Delete task-name and status inspection. Confirm the rendered application behavior and successful convergence instead. | Whether named tasks ran is an implementation detail; the ordinary `when` expressions are suitable for review and lint. | +| Audita | Restore the minimal default configuration | Cleanup and exact current default | Simplify | Keep restoration only as cleanup and remove fixed-document, fixed-commit, task-state, and extra repeat assertions. | It does not add a distinct stable behavior. | +| Audita | Alternate raw commit `46b7356...` | A non-default Git revision can be selected | Replace | Remove the raw-commit case. Add an explicit immutable-release test only when Audita exposes a stable native way to identify the installed application release. | A raw source commit checked with `git rev-parse` tests checkout mechanics, not the installed application's release. Audita currently has no `--version` or `version` command, and tags `v0.9.0` and `v0.9.1` both resolve to `a84941d...`, so neither tag provides a distinguishable installed outcome today. | +| Audita | Platform failure matrix | Straightforward platform assertions | Remove | Delete the matrix and `tests/unsupported.yml`. | This is expressly outside the testing house style. | +| Audita | Final commit, metadata, and config checks | Duplicate checkout/module/validation assertions | Remove | Delete them. | Equivalent checks already ran, and most should themselves be removed. | +| Audita | `tests/verify.yml` broad installation contract | Commit, metadata, default document, and native config validity | Replace | Reduce it to native configuration validation; move explicit field assertions to the focused representative verifier. | Only native validity is a stable application outcome in this verifier. | +| Audita | README validation claims | Description of default, metadata, rebuild, and platform tests | Replace | Describe only the retained static checks and behavioral scenarios. | The present prose promises low-value coverage that should be removed. | + +### Audita Version-Coverage Gap + +The current suite does not—and with the present application interface cannot— +perform the useful test “select immutable release X and verify that the +installed Audita reports X.” Running either `audita --version` or +`audita version` fails as an unknown command. The recommended role-test +remediation is to remove the misleading raw-commit scenario rather than +replace it with another Git-module assertion. A native Audita release-identity +command would enable the desired explicit-version test later, but adding that +application feature is outside this audit and should be selected separately. + +## Notarius Classification + +| Repository | Test or case | Behavior protected | Classification | Recommended action | Rationale | +| --- | --- | --- | --- | --- | --- | +| Notarius | Default convergence | The default public interface installs and configures successfully | Retain | Keep as a smoke test, run `notarius --version` without fixing its output to the current default, and validate the configuration natively. | This proves the default path remains usable without snapshotting its values. | +| Notarius | Default `v0.6.0` and commit verification | Present default release identity | Remove | Delete fixed `default_commit` and exact default-version assertions. | These fail solely when the supported default advances. | +| Notarius | Source, binary, and configuration metadata | Trusted module postconditions | Remove | Delete ownership, mode, and regular-file assertions. | They re-test `git`, `file`, and `template`. | +| Notarius | Exact minimal default document `{"version": 4}` | Current default rendering | Remove | Do not snapshot the default document. Supply `notarius_config_version` explicitly in the representative case and make any omission assertions about explicitly null inputs. | This preserves schema-bound behavior without turning future defaults into failures. | +| Notarius | Native version command and configuration validation | Installed CLI is usable and accepts the managed configuration | Retain | Keep both native interfaces, but assert an exact version only in the explicitly selected release scenario. | These are application-level outcomes. | +| Notarius | Baseline immediate repeat | Clean convergence | Retain | Keep one overall clean repeat. | This directly enforces the idempotence requirement. | +| Notarius | Missing binary | The required output is rebuilt and then converges | Simplify | Retain artifact removal, native CLI verification, and the following clean repeat; remove task greps. | Missing-output recovery is role logic needed around the imperative build. | +| Notarius | Dependency opt-out and custom Go executable | The caller-selected build executable is honored | Simplify | Keep wrapper invocation evidence and combine with missing-binary recovery; remove task greps and redundant repetitions. | This is meaningful configurable behavior with no better final-state discriminator. | +| Notarius | Binary metadata repair | Built-in file convergence | Remove | Delete the complete scenario. | It tests trusted module behavior. | +| Notarius | Explicit `v0.5.0` selection | A caller-selected immutable tag becomes the installed application | Simplify | Retain `notarius_version: v0.5.0` and exact native output `notarius v0.5.0`; remove expected commit and task-state assertions. | This remains useful after the role's default changes. | +| Notarius | Restore `v0.6.0` default | Cleanup and present default identity | Simplify | Reduce to cleanup only. | It is not a distinct stable behavioral test. | +| Notarius | Representative configuration | Field mapping, nested sections, pipelines, and validity | Simplify | Use a smaller set of explicit values and assert only those paths plus specifically selected null omissions; retain native validation. | Exact equality for a large complete document is broader and more brittle than the behavior requires. | +| Notarius | Separate `profile_dir` case | One simple optional leaf mapping | Remove | Do not maintain a standalone scenario; a future representative case may select one profile source if it adds value. | The direct conditional and serialization are self-evident and already represented by other optional fields. | +| Notarius | Separate `profile_file` case | A second simple optional leaf mapping | Remove | Delete the standalone scenario. | It duplicates the shape and risk of the profile-directory case without protecting a distinct likely regression. | +| Notarius | Explicit falsey values | Zero, false, empty mapping, and empty string survive rendering | Retain | Fold representative falsey values into the focused configuration scenario and assert only those paths. | Null-versus-falsey behavior is a deliberate role invariant. | +| Notarius | Caller-side pipeline composition | Independently defined mappings can be combined and passed through intact | Retain | Keep composition through ordinary `combine`, assert the two resulting pipeline entries, and run native validation; fold it into the representative configuration scenario if clarity permits. | This is an expressly selected public boundary and a plausible integration regression. | +| Notarius | Configuration task changed while build task skipped | Configuration changes without rebuilding | Remove | Delete `require_configuration_change_without_build` and all task-trace greps. Verify final configuration, CLI usability, and clean repeat instead. | Task execution is not a stable public interface. | +| Notarius | Restore the minimal default configuration | Cleanup plus mutable default assertions | Simplify | Keep cleanup only; remove exact default document, version, commit, task-state, and extra repeat checks. | The behavioral assertions duplicate change detectors elsewhere. | +| Notarius | Platform failure matrix | Straightforward platform assertions | Remove | Delete the matrix and `tests/unsupported.yml`. | It provides no proportionate role-specific value. | +| Notarius | Final commit, metadata, version, and config checks | Duplicate assertions | Remove | Delete them. | They repeat earlier verification and freeze the current default. | +| Notarius | `tests/verify.yml` broad installation contract | Commit, metadata, default document, version, and validity | Replace | Reduce to native CLI smoke and configuration validation, with exact version verification only for an explicit tag. | This retains application outcomes while removing module tests and mutable snapshots. | +| Notarius | README validation claims | Description of the current broad suite | Replace | Rewrite after simplification to describe the retained behavior. | Documentation should match the lean suite rather than preserve obsolete coverage promises. | + +## Proposed Lean Target Suites + +### Seriatim + +1. Install pinned collections, syntax-check the convergence play, and lint the + role and retained test files. +2. Apply the role with defaults as a smoke test and immediately require a clean + repeat, without asserting the current default tag or commit. +3. Select `v1.4.0` explicitly and verify native output + `seriatim version v1.4.0`. +4. Delete the binary, supply a wrapper through `seriatim_go_binary` with + dependency management disabled, converge, verify that the wrapper was used + and the installed CLI works, then require a clean repeat. +5. Restore defaults as cleanup without treating the restored value as a test + expectation. + +### Audita + +1. Install pinned collections, syntax-check the convergence play, and lint the + role and retained test files. +2. Apply the role with defaults, validate the managed configuration with + Audita, and immediately require a clean repeat. Do not assert the complete + default document or current source identity. +3. Apply one representative explicit configuration spanning nested, list, + scalar, duration, falsey, and explicit-null behavior. Assert only the + selected paths and omissions, validate it with Audita, and require a clean + repeat. +4. Delete the binary, supply a logging wrapper through `audita_go_binary` with + dependency management disabled, converge, verify wrapper use and CLI + usability, then require a clean repeat. +5. Omit version-selection coverage until Audita exposes a native release + identity; do not substitute a raw-commit checkout test. +6. Restore defaults as cleanup only. + +### Notarius + +1. Install pinned collections, syntax-check the convergence play, and lint the + role and retained test files. +2. Apply the role with defaults, exercise `notarius --version`, validate the + managed configuration, and immediately require a clean repeat without + asserting the current default values. +3. Select `v0.5.0` explicitly and verify native output `notarius v0.5.0`. +4. Delete the binary, supply a logging wrapper through `notarius_go_binary` + with dependency management disabled, converge, verify wrapper use and CLI + usability, then require a clean repeat. +5. Apply one representative explicit configuration that includes selected + nested values, explicit falsey and null values, and two caller-composed + pipeline mappings. Assert only those behaviors, validate the file with + Notarius, and require a clean repeat. +6. Restore defaults as cleanup only. + +## Documentation Follow-up + +The README validation sections in all three role repositories should be +rewritten with their test harnesses. They currently advertise current-default, +metadata, task-arrangement, and platform-guard coverage recommended for +removal. + +The durable feature roadmaps may continue to record the selected default tags, +paths, ownership, permissions, and platform support as feature requirements. +Those requirements need not each have a dedicated integration assertion. +Test-specific roadmap language should, however, be corrected as follows: + +- Feature 0001 and Feature 0003 should describe exact native version checks as + checks of an explicitly selected immutable tag, not the current default. +- Feature 0002 currently says repository-local tests execute Audita to verify + its installed version. That is not implementable with Audita's current CLI + and should be replaced with the limitation and follow-up described above. +- Platform support remains part of each role contract, but the roadmaps should + not imply that each straightforward assertion requires a negative matrix. + +## Recommended Remediation Order + +1. Simplify the Seriatim driver and verifier, remove its unsupported-platform + playbook, and update its README. +2. Simplify Audita's driver and verifiers, consolidate its configuration case, + remove its unsupported-platform playbook, and update its README. +3. Simplify Notarius's driver and verifiers, consolidate configuration and + pipeline coverage, remove its unsupported-platform playbook, and update its + README. +4. Correct the test-specific language in the three feature roadmaps. +5. Run all three complete integration entry points and confirm every role + repository remains clean after its cleanup path. + +No shared test abstraction is recommended. The three shell drivers have +similar mechanics, but keeping each suite local and readable is worth the +small amount of duplication, and a shared helper would create cross-repository +coupling for little present benefit. + +## Audit Completion + +Every materially distinct implemented scenario has been classified above. The +recommended removals either duplicate retained application-level coverage or +exercise behavior owned by trusted Ansible modules or straightforward +assertions. The lean targets retain default smoke coverage, immediate clean +convergence, imperative missing-output recovery, caller-supplied build tooling, +explicit release selection where natively observable, and representative +configuration behavior. + +The audit made no changes to production roles or test implementations. +Remediation should begin only after these findings and the Audita +release-identity recommendation have been reviewed. diff --git a/docs/roadmap/test-suite-audit.md b/docs/roadmap/test-suite-audit.md new file mode 100644 index 0000000..b6fce14 --- /dev/null +++ b/docs/roadmap/test-suite-audit.md @@ -0,0 +1,192 @@ +# Test Suite Audit Plan + +Status: complete + +## Objective + +Audit the implemented integration tests for the three completed standalone +roles against the workspace's classical, state-based testing policy. The audit +will identify tests that protect stable role-owned behavior, tests that should +be simplified or replaced, and tests whose maintenance cost exceeds their +value. + +This is a review plan, not authorization to change role code or test suites. +Any proposed remediation will be reviewed before a separate implementation +plan is prepared. + +## Scope + +The audit covers the test harnesses, playbooks, fixtures, inventories, and +test-related documentation in: + +- `ansible-role-seriatim` +- `ansible-role-audita` +- `ansible-role-notarius` + +Production role tasks, defaults, templates, and metadata may be inspected when +needed to determine whether a test protects public role behavior or merely +repeats an implementation detail. They are not remediation targets during the +audit. + +The audit does not cover the proof-of-concept `rakestraw-ansible` role, the +suite-level role, or the collection. Those projects do not form part of the +three completed standalone-role test suites. + +## Governing Standard + +Apply the normative testing house style in `docs/conventions.md`. In +particular, the audit will favor tests that exercise the role through its +public variables and inspect meaningful final application behavior. It will +disfavor tests that: + +- freeze current defaults or other intentionally changeable values; +- assert task names, order, individual task status, or other execution traces; +- independently re-prove the documented postconditions of trusted Ansible + modules; +- exhaustively exercise self-evident assertions or equivalent low-value + combinations; or +- duplicate another test without protecting a distinct requirement or + plausible regression. + +An immediate second convergence with no reported changes remains required for +each role. An explicitly selected version may be verified through the +application's native interface; the role's current default version may not be +treated as a permanent expected value. + +## Classification Rubric + +Classify every behavioral test case, including each materially distinct case +embedded in a shared playbook or shell harness, as one of the following: + +- **Retain:** protects stable, role-owned behavior through an externally + meaningful outcome. +- **Simplify:** protects useful behavior but includes redundant assertions, + duplicated setup, or implementation coupling that can be removed. +- **Replace:** addresses a valid requirement through an unstable or indirect + assertion and should be expressed as a state-based outcome instead. +- **Remove:** tests a mutable default, trusted module behavior, an + implementation trace, or behavior without a stated requirement or plausible + regression. +- **Add:** identifies a material role-owned behavior that currently lacks + representative coverage. + +Each classification must state the behavior at issue and the reason for the +recommended action. File length, test count, and aesthetic preference are not +sufficient reasons by themselves. + +## Audit Record + +Write the completed review to +`docs/roadmap/test-suite-audit-findings.md`. Record each finding in a table with +these fields: + +| Repository | Test or case | Behavior protected | Classification | Recommended action | Rationale | +| --- | --- | --- | --- | --- | --- | + +The report must also provide a proposed lean target suite for each role and +identify any shared harness behavior that should be made consistent across the +three repositories. Recommendations to remove coverage must state whether the +behavior remains covered elsewhere or why it does not warrant coverage. + +## Stage 1: Establish the Baseline + +1. Confirm that the workspace and all three role repositories begin clean. +2. Inventory every test entry point, playbook, fixture, inventory, helper, and + test-related README statement in scope. +3. Run each existing integration entry point without modification: + + ```text + (cd ansible-role-seriatim && tests/integration) + (cd ansible-role-audita && tests/integration) + (cd ansible-role-notarius && tests/integration) + ``` + +4. Record success or failure, the scenarios exercised, and any environmental + assumptions needed to reproduce the result. A pre-existing failure is audit + evidence, not permission to repair it during this stage. + +Completion criterion: the report contains a reproducible baseline and a +complete test inventory for all three roles. + +## Stage 2: Audit Seriatim + +1. Map every Seriatim test case to a documented public behavior or classify it + as lacking one. +2. Review version selection, caller-supplied build tooling, rebuild behavior, + metadata handling, platform validation, and repeat convergence using the + common rubric. +3. Identify assertions that inspect Ansible execution details or separately + verify state already owned by trusted modules. +4. Propose the smallest representative Seriatim suite that preserves valuable + regression protection. + +Completion criterion: every Seriatim case is classified and the report states +the proposed retained, revised, removed, and missing coverage. + +## Stage 3: Audit Audita + +1. Map every Audita test case to a documented public behavior or classify it + as lacking one. +2. Review explicit version selection, native application verification, + configuration rendering, omission of null settings, preservation of falsey + values, platform validation, and repeat convergence. +3. Distinguish tests of the role's configuration decisions from tests that + merely repeat `template`, `file`, `git`, or package-module guarantees. +4. Propose the smallest representative Audita suite that preserves valuable + regression protection. + +Completion criterion: every Audita case is classified and the report states +the proposed retained, revised, removed, and missing coverage. + +## Stage 4: Audit Notarius + +1. Map every Notarius test case to a documented public behavior or classify it + as lacking one. +2. Review explicit version selection, native application verification, + optional configuration rendering, falsey values, caller-composed pipeline + mappings, platform validation, and repeat convergence. +3. Distinguish tests of role-authored mapping and rendering behavior from + assertions about trusted module mechanics or the current implementation + layout. +4. Propose the smallest representative Notarius suite that preserves valuable + regression protection. + +Completion criterion: every Notarius case is classified and the report states +the proposed retained, revised, removed, and missing coverage. + +## Stage 5: Reconcile the Three Suites + +1. Compare classifications across roles so equivalent patterns receive + equivalent treatment, while preserving genuine application differences. +2. Identify duplicated harness machinery, inconsistent terminology, and + documentation that describes implementation details or mutable defaults as + promised test behavior. +3. Separate static validation such as syntax checking and linting from + behavioral integration coverage. +4. Produce a repository-by-repository remediation proposal ordered by value + and dependency. Do not propose a shared abstraction unless it removes + demonstrated duplication without obscuring the individual role suites. +5. Confirm that the proposed target for every role still includes a default or + representative convergence, an immediate clean second convergence, and + focused coverage of each retained role-owned behavior. + +Completion criterion: the findings report is internally consistent, gives a +lean target suite for each repository, and contains enough detail to support a +later implementation plan without leaving test-policy decisions to the +executor. + +## Audit Completion Criteria + +The audit is complete when: + +- every implemented behavioral case across all three repositories has an + explicit classification and rationale; +- baseline results and reproduction commands are recorded; +- proposed removals do not silently discard coverage of a required public + behavior; +- proposed additions are limited to material role-owned behavior; +- README and harness-documentation corrections are identified alongside test + changes; +- no production or test implementation has been changed during the audit; and +- the findings have been reviewed before any remediation plan is written to + `docs/roadmap/implementation.md`. diff --git a/docs/status.md b/docs/status.md index 8f2bec8..2fc094c 100644 --- a/docs/status.md +++ b/docs/status.md @@ -1,10 +1,10 @@ # Workspace Status -Last updated: 2026-09-08 +Last updated: 2026-09-09 Current milestone: MVP standalone application roles -Current focus: Select the next work stream. +Current focus: Review and commit the staged test-suite remediation plan. | ID | Deliverable | Owner | State | Evidence or next step | | --- | --- | --- | --- | --- | @@ -20,6 +20,8 @@ Current focus: Select the next work stream. | F0001 | Install Seriatim from pinned source | ansible-role-seriatim | done | Implementation, integration coverage, and final remediation completed through `fb8dc41` | | F0002 | Install and configure Audita | ansible-role-audita | done | Field-specific configuration completed in `e61f02f` and `847c4c3`; full suite passed | | F0003 | Install and configure Notarius | ansible-role-notarius | done | Implemented and validated through remediation commit `eb4f511` | +| A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` | +| R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | ready | Execute `docs/roadmap/implementation.md` in four stages | ## Updating This File