From 332bc4742b74b6df7448a9c664aca48ef403aabd Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 9 Sep 2026 13:03:24 +0000 Subject: [PATCH] Complete the test suite remediation --- docs/roadmap/implementation.md | 519 +--------------------- docs/roadmap/test-suite-audit-findings.md | 11 +- docs/status.md | 4 +- 3 files changed, 19 insertions(+), 515 deletions(-) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 7dc8a06..b66b2eb 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,514 +1,11 @@ -# Implementation Plan R0001: Classical Test Suite Remediation +# Implementation Plan -Status: ready +Status: pending -Target executor: GPT-5.6-Terra +No implementation plan is currently active. Test-suite remediation R0001 was +implemented in role commits `e08c63c`, `d2d0b9d`, and `cfecf37`, then reviewed +successfully. Its rationale and validation record remain in +[the test-suite audit findings](test-suite-audit-findings.md). -Authoritative audit: -`docs/roadmap/test-suite-audit-findings.md` - -Active plan location: `docs/roadmap/implementation.md` - -## Objective - -Replace the oversized, implementation-bound integration suites in the three -completed standalone role repositories with lean classical, state-based -suites. Retain tests of stable role-owned behavior and application outcomes; -remove mutable-default snapshots, task-trace assertions, tests of trusted -Ansible modules, redundant scenarios, and negative platform matrices. - -Production role behavior is not changing. This plan changes only test -implementations, test-related role documentation, and the workspace planning -records identified by the audit. - -## Execution Rules - -- Execute stages in numeric order. -- Before each stage, confirm its owning repository is clean and record its - starting revision. -- Preserve unrelated and pre-existing changes. -- Do not modify production tasks, defaults, templates, argument - specifications, dependency pins, or role metadata. If a retained test - exposes a production defect, stop and report it instead of expanding this - remediation. -- Follow the normative testing house style in `docs/conventions.md` and the - classifications in the authoritative audit. -- Exercise roles through their public variables and inspect final application - behavior. Do not parse task names, task order, or individual task status. -- Retain only aggregate `changed=0` checks on specified immediate repeat runs. - Do not require a nonzero change count on preceding runs. -- Do not verify ownership, permissions, path types, checkout commits, or other - postconditions already declared through trusted modules. -- Do not add a test merely to replace each removed test one-for-one. The target - is the smallest suite that protects the retained behavior. -- Keep each repository's shell driver self-contained. Do not create a shared - cross-repository test helper or abstraction. -- Run each stage's full repository integration entry point before committing. -- Commit each completed stage separately with the plain-English intent given - in that stage. -- Do not begin a dependent stage after an earlier stage fails validation. - -## Starting State and Preconditions - -- The workspace conventions, audit plan, audit findings, status update, and - this implementation plan must be reviewed and committed in the workspace - repository before Stage 1 begins. -- `ansible-role-seriatim` starts clean at `fb8dc41`. -- `ansible-role-audita` starts clean at `847c4c3`. -- `ansible-role-notarius` starts clean at `eb4f511`. -- The workspace `.venv` provides Ansible Core and ansible-lint, and the local - Ubuntu 26.04 `x86_64` host remains the approved MVP integration proxy. -- The executor can use passwordless privilege escalation for the existing - `/usr/local` integration-test paths and can reach Ansible Galaxy, the three - application Git repositories, Snap infrastructure, and Go module sources. -- Existing installed application state may be present. Each driver must remain - repeatable from that state and restore its default role deployment through a - best-effort cleanup trap. -- Audita exposes no native application-release identity command. Its tags - `v0.9.0` and `v0.9.1` currently resolve to the same commit. This plan - deliberately omits an Audita version-selection test rather than substituting - a raw-commit or Git-module assertion. - -## Repository Scope - -Repositories that change: - -- `ansible-role-seriatim`: lean integration driver and verifier, removal of - the unsupported-platform playbook, and accurate validation documentation. -- `ansible-role-audita`: lean integration driver and verifiers, removal of the - unsupported-platform playbook, and accurate validation documentation. -- `ansible-role-notarius`: lean integration driver and verifiers, removal of - the unsupported-platform playbook, and accurate validation documentation. -- Workspace control repository: test-specific corrections to Features 0001, - 0002, and 0003, remediation status, and cross-repository validation evidence. - -Repositories that may be inspected but must remain unchanged: - -- `../seriatim`, `../audita`, and `../notarius` for native CLI behavior. -- Every other application, role, collection, and proof-of-concept repository. - -## Stage 1: Simplify the Seriatim Test Suite - -### Objective - -Reduce Seriatim's integration suite to static validation and representative -tests of default usability, explicit version selection, missing-binary -recovery, caller-supplied Go, and clean repeated convergence. - -### Preconditions - -- The workspace planning commit described above exists. -- `ansible-role-seriatim` is clean at `fb8dc41`. -- The existing `tests/integration` baseline passes. - -### Owning Repository - -`ansible-role-seriatim` - -### Required Changes - -1. Retain `tests/integration` as the single entry point, its private temporary - directory, pinned collection installation, syntax check, production-profile - ansible-lint, and best-effort cleanup convergence. -2. Retain a default role convergence as a smoke test. Exercise - `/usr/local/bin/seriatim --version` successfully without comparing it to the - current default version, then immediately apply the default role again and - require only the aggregate recap `changed=0` and `failed=0`. -3. Add or retain one explicit version-selection scenario using - `seriatim_version: v1.4.0`. Verify through the native CLI that the result is - exactly `seriatim version v1.4.0`. Do not resolve or assert the corresponding - Git commit. -4. Combine missing-binary recovery and caller-supplied build tooling into one - representative scenario: - - create an executable temporary wrapper that records invocation and then - executes `/snap/bin/go` with the original arguments; - - remove `/usr/local/bin/seriatim`; - - apply the role with `seriatim_version: v1.4.0`, - `seriatim_manage_build_dependencies: false`, and - `seriatim_go_binary` set to the wrapper; - - require the wrapper log to show that the supplied executable was used; - - verify native output `seriatim version v1.4.0`; and - - immediately repeat with the same variables and require aggregate - `changed=0` and `failed=0`. -5. Delete `tests/verify.yml` and perform the narrow native version assertions - directly in the shell driver. Do not retain source, commit, ownership, - permission, file-type, or checksum inspection. -6. Remove the binary-mode mutation and repair scenario, every - `require_changed_recap` use, every task-name or individual-status grep, the - default and previous commit constants, redundant default-restoration - assertions, and final duplicate shell assertions. -7. Delete `tests/unsupported.yml` and remove it from lint and execution. Do not - replace its distribution, release, or architecture cases. -8. Keep a cleanup trap that reapplies the default role and removes only the - private temporary directory. Cleanup output may remain suppressed and must - not become another behavioral assertion. -9. Rewrite the README validation section to describe only the retained static - checks and behavioral scenarios. Do not advertise exact-default, metadata, - task-arrangement, or platform-matrix coverage. - -### Validation - -Run: - -```text -tests/integration -git diff --check -git status --short -``` - -Expected result: static validation and every retained Seriatim scenario pass; -the specified repeat runs report no changes; cleanup completes; and the only -working-tree changes are the planned test and README edits. - -### Completion Criteria - -- No test asserts Seriatim's current default version or a Git commit. -- No test inspects task names, individual task status, ownership, permissions, - or the platform assertion branches. -- Explicit `v1.4.0` selection is verified through Seriatim itself. -- One scenario proves both missing-binary recovery and use of the configured Go - executable, followed by a clean repeat. -- The complete integration entry point passes. - -### Commit - -Commit intent: `Simplify the Seriatim integration tests` - -## Stage 2: Simplify the Audita Test Suite - -### Objective - -Reduce Audita's integration suite to static validation and representative -tests of default usability, native configuration validity, explicit -configuration rendering, missing-binary recovery, caller-supplied Go, and clean -repeated convergence. - -### Preconditions - -- Stage 1 is complete and committed. -- `ansible-role-audita` is clean at `847c4c3`. -- The existing `tests/integration` baseline passes. - -### Owning Repository - -`ansible-role-audita` - -### Required Changes - -1. Retain `tests/integration` as the single entry point, its private temporary - directory, pinned collection installation, syntax check, production-profile - ansible-lint, and best-effort cleanup convergence. -2. Retain a default role convergence as a smoke test. Validate the managed - configuration with - `/usr/local/bin/audita config validate --config /usr/local/etc/audita/config.yml`, - then immediately repeat the default role and require only aggregate - `changed=0` and `failed=0`. -3. Replace the exhaustive configuration document and separate falsey scenario - with one representative explicit configuration. Supply these values: - - `audita_config_version: 1`; - - `audita_pipeline_modules: ["glossary"]`; - - `audita_llm_proposal_base_url: "https://openrouter.ai/api/v1"`; - - `audita_llm_proposal_model: "openrouter/google/gemma-4-31b-it"`; - - `audita_llm_proposal_timeout: "10s"`; - - `audita_llm_proposal_max_retries: 0`; - - `audita_context_description: ""`; - - `audita_diagnostics_retention: "never"`; - - `audita_llm_validation_model: null`; - - `audita_output_schema: null`; and - - `audita_diagnostics_work_dir: null`. -4. In `tests/verify-config.yml`, parse the rendered YAML and assert only that: - - `version` is `1`; - - `pipeline.modules` is `["glossary"]`; - - the selected proposal URL, model, timeout, and zero retry count are - present; - - the empty context description and diagnostics retention value are - present; - - `llm.validation`, `output`, and `diagnostics.work_dir` are absent; and - - Audita's native configuration validation command succeeds. - Do not compare the complete parsed document to an expected mapping. -5. Immediately repeat the representative configuration convergence with the - same explicit variables and require aggregate `changed=0` and `failed=0`. -6. Combine missing-binary recovery and caller-supplied build tooling into one - representative scenario. Create a logging wrapper around `/snap/bin/go`, - remove `/usr/local/bin/audita`, converge with - `audita_manage_build_dependencies: false` and `audita_go_binary` set to the - wrapper, verify wrapper use, validate the managed configuration through - Audita, and immediately repeat the same convergence with aggregate - `changed=0` and `failed=0`. -7. Delete `tests/verify.yml` and invoke Audita's native configuration validator - directly from the shell driver for the smoke and recovery scenarios. - Successful command completion is sufficient; do not separately assert the - validator's fixed success text. -8. Remove the exact default configuration comparison, source-commit checks, - source/binary/configuration metadata checks, binary-mode repair scenario, - all `require_changed_recap` uses, and every task-name or individual-status - grep. -9. Remove the raw alternate-commit scenario entirely. Do not replace it with a - `git rev-parse`, tag-resolution, embedded-string, or Go build-metadata test. - Do not add an Audita version-selection test until the application provides a - stable native release-identity interface. -10. Delete `tests/unsupported.yml` and remove it from lint and execution. Do not - replace its platform cases. -11. Retain default restoration only in the cleanup trap. Remove its fixed - commit, exact document, task-status, and extra idempotence assertions. -12. Rewrite the README validation section to describe only the retained static - checks and behavioral scenarios and to avoid claiming application-version, - metadata, task-arrangement, or platform-matrix coverage. - -### Validation - -Run: - -```text -tests/integration -git diff --check -git status --short -``` - -Expected result: static validation and every retained Audita scenario pass; -both default and representative configurations validate natively; the -specified repeats report no changes; cleanup completes; and the only -working-tree changes are the planned test and README edits. - -### Completion Criteria - -- No test asserts Audita's current default, a source commit, or a complete - rendered configuration document. -- No test inspects task traces, trusted-module postconditions, or platform - assertion branches. -- One focused explicit case proves representative nesting, null omission, and - falsey preservation and passes Audita's validator. -- One scenario proves both missing-binary recovery and use of the configured Go - executable, followed by a clean repeat. -- The absence of native Audita version coverage is documented accurately and - is not replaced by an implementation proxy. -- The complete integration entry point passes. - -### Commit - -Commit intent: `Simplify the Audita integration tests` - -## Stage 3: Simplify the Notarius Test Suite - -### Objective - -Reduce Notarius's integration suite to static validation and representative -tests of default usability, explicit version selection, missing-binary -recovery, caller-supplied Go, configuration behavior, caller-side pipeline -composition, native validation, and clean repeated convergence. - -### Preconditions - -- Stages 1 and 2 are complete and committed. -- `ansible-role-notarius` is clean at `eb4f511`. -- The existing `tests/integration` baseline passes. - -### Owning Repository - -`ansible-role-notarius` - -### Required Changes - -1. Retain `tests/integration` as the single entry point, its private temporary - directory, pinned collection installation, syntax check, production-profile - ansible-lint, and best-effort cleanup convergence. -2. Retain a default convergence as a smoke test. Require - `/usr/local/bin/notarius --version` and - `/usr/local/bin/notarius config validate --config /usr/local/etc/notarius/config.yml` - to succeed without comparing the version output or complete configuration - to current defaults. Immediately repeat the default role and require only - aggregate `changed=0` and `failed=0`. -3. Retain one explicit version-selection scenario with - `notarius_version: v0.5.0` and verify exact native output - `notarius v0.5.0`. Do not resolve or assert the corresponding Git commit. -4. Combine missing-binary recovery and caller-supplied build tooling into one - scenario. Create a logging wrapper around `/snap/bin/go`, remove - `/usr/local/bin/notarius`, converge with `notarius_version: v0.5.0`, - `notarius_manage_build_dependencies: false`, and `notarius_go_binary` set to - the wrapper, verify wrapper use and native output `notarius v0.5.0`, then - immediately repeat with aggregate `changed=0` and `failed=0`. -5. Replace the exhaustive configuration, separate profile-source cases, - separate falsey case, and separate pipeline-composition case with one - representative explicit configuration. Define two caller-side pipeline - mappings and set `notarius_pipelines` to their ordinary `combine` result. - Supply these additional values: - - `notarius_config_version: 4`; - - `notarius_promptkit_local_backend_endpoint: "http://localhost:8000/v1"`; - - `notarius_promptkit_local_backend_concurrency_limit: 0`; - - `notarius_cache_checkpoints_enabled: false`; - - `notarius_cache_chunk_plans_directory: ""`; - - `notarius_output_directory: "/tmp/notarius-output"`; - - `notarius_promptkit_profile_dir: null`; - - `notarius_promptkit_profile_file: null`; and - - `notarius_debug_directory: null`. - Use the two valid `dnd-session` and `dnd-archive` pipeline definitions from - the existing composition scenario. -6. In `tests/verify-config.yml`, parse the rendered YAML and assert only that: - - `version` is `4`; - - local-backend endpoint and zero concurrency are present; - - checkpoints are explicitly disabled and the empty chunk-plan directory is - present; - - the selected output directory is present; - - both composed pipeline IDs and their supplied definitions are present - without rewriting; - - `promptkit.profile_dir`, `promptkit.profile_file`, and the top-level - `debug` section are absent; and - - Notarius's native configuration validation succeeds. - Do not compare the complete document to an expected mapping. -7. Immediately repeat the representative configuration convergence with the - same explicit variables and require aggregate `changed=0` and `failed=0`. -8. Delete `tests/verify.yml`. Perform native CLI smoke, configuration - validation, and the explicit `v0.5.0` comparison directly in the shell - driver. -9. Remove current-default version and commit assertions, complete default - configuration comparisons, all source/binary/configuration metadata checks, - the binary-mode repair scenario, every `require_changed_recap` use, - `require_configuration_change_without_build`, and every task-name or - individual-status grep. -10. Delete the standalone `profile_dir` and `profile_file` scenarios. Their - simple leaf mappings do not require dedicated replacements. -11. Delete `tests/unsupported.yml` and remove it from lint and execution. Do not - replace its platform cases. -12. Retain default restoration only in the cleanup trap and remove duplicate - final assertions. -13. Rewrite the README validation section to describe only the retained static - checks and behavioral scenarios. Do not advertise current-default, - metadata, task-arrangement, profile-branch, or platform-matrix coverage. - -### Validation - -Run: - -```text -tests/integration -git diff --check -git status --short -``` - -Expected result: static validation and every retained Notarius scenario pass; -the explicit tag is verified through Notarius; representative falsey, null, -and composed-pipeline behavior is verified and validates natively; the -specified repeats report no changes; cleanup completes; and only the planned -test and README changes remain. - -### Completion Criteria - -- No test snapshots Notarius's current default release, commit, or complete - default configuration. -- No test inspects task traces, trusted-module postconditions, or platform - assertion branches. -- Explicit `v0.5.0` selection is verified through Notarius itself. -- One scenario proves both missing-binary recovery and use of the configured Go - executable, followed by a clean repeat. -- One representative configuration case proves null omission, falsey - preservation, caller-side pipeline composition, and native validity. -- The complete integration entry point passes. - -### Commit - -Commit intent: `Simplify the Notarius integration tests` - -## Stage 4: Reconcile Workspace Documentation and Validate the Suite - -### Objective - -Align durable test-specific roadmap language with the canonical testing policy, -record the remediation implementation for review, and validate all three lean -suites together. - -### Preconditions - -- Stages 1 through 3 are complete and committed in their owning repositories. -- The workspace repository is clean at the planning commit created before - Stage 1. -- The three role repositories contain only their committed remediation changes. - -### Owning Repository - -Workspace control repository (`ansible-workspace-dnd`) - -### Required Changes - -1. Revise only test-specific language in Features 0001, 0002, and 0003. Retain - their durable production requirements, including current defaults, paths, - ownership, permissions, platform support, and convergence behavior. -2. In Feature 0001, describe the exact native version assertion as coverage of - an explicitly selected immutable tag rather than an assertion about the - role's current default. -3. In Feature 0002, remove the incorrect claim that tests execute Audita to - verify its installed application version. State that the current CLI has no - native release-identity command and that the suite deliberately avoids a - raw-commit proxy. Retain native configuration validation. -4. In Feature 0003, describe exact version coverage using explicitly selected - `v0.5.0`, not the current default `v0.6.0`. Keep the current default as a - production requirement rather than a frozen test expectation. -5. Remove any implication in the three roadmaps that each platform assertion - must have a negative integration-test matrix. Do not remove platform support - itself from scope or acceptance criteria. -6. Update `docs/roadmap/test-suite-audit-findings.md` to say remediation was - implemented and is pending review. Add the three role commit hashes and the - final validation results; do not erase the audit classifications. -7. Update `docs/status.md` so `R0001` remains `in_progress` pending review, its - evidence names the three role commits, and the current focus is review of - the implemented test-suite remediation. Do not mark remediation `done`. -8. Add concise progress notes to this plan with the four stage commits and any - deviations. Do not replace the active plan with a pending notice before - review. - -### Validation - -Run sequentially from the workspace root: - -```text -(cd ansible-role-seriatim && tests/integration) -(cd ansible-role-audita && tests/integration) -(cd ansible-role-notarius && tests/integration) -git diff --check -scripts/status -``` - -Expected result: all static and behavioral checks pass; every role cleanup path -completes; the three role repositories are clean and contain their remediation -commits; no unrelated repository changed; and the workspace has only the -planned documentation changes before its stage commit. - -### Completion Criteria - -- All three full integration suites pass sequentially from the workspace. -- Feature roadmaps distinguish production defaults from stable test inputs. -- Audita's native version limitation is stated accurately. -- Audit findings contain implementation and validation evidence without being - rewritten as an implementation log. -- Workspace status leaves remediation pending review. -- Every repository changed by the implementation is clean after its owning - commit. - -### Commit - -Commit intent: `Record the test suite remediation results` - -## Final Validation and Handoff - -- Report the Stage 1 through Stage 4 commit hashes and confirm the working-tree - state of every repository in `workspace.yml`. -- Summarize the retained scenarios for each role and identify any deviation - from this plan. -- Report the Audita version-identity limitation; do not silently fill it with a - source-control assertion. -- Report all validation commands and results. -- Do not push commits unless the user separately authorizes a push. -- Do not mark `R0001` done or replace this plan with a pending notice. Final - completion requires review of the implemented suites. - -## Progress Notes - -- Planning complete; implementation has not started. -- [x] Stage 1 complete — `ansible-role-seriatim` commit `e08c63c`; full - `tests/integration` passed. No deviations. -- [x] Stage 2 complete — `ansible-role-audita` commit `d2d0b9d`; full - `tests/integration` passed. No deviations. -- [x] Stage 3 complete — `ansible-role-notarius` commit `cfecf37`; full - `tests/integration` passed. No deviations. -- [x] Stage 4 complete — workspace commit `ca82dd7`; sequential full - integration validation passed for Seriatim, Audita, and Notarius. No - deviations. +Select and approve the next unit of work before replacing this notice with a +new staged implementation plan. diff --git a/docs/roadmap/test-suite-audit-findings.md b/docs/roadmap/test-suite-audit-findings.md index 66c0237..375b347 100644 --- a/docs/roadmap/test-suite-audit-findings.md +++ b/docs/roadmap/test-suite-audit-findings.md @@ -1,6 +1,6 @@ # Test Suite Audit Findings -Status: remediation implemented; pending review +Status: resolved ## Executive Summary @@ -259,7 +259,7 @@ The audit made no changes to production roles or test implementations. ## Remediation Status -The recommended remediation was implemented and is pending review. No +The recommended remediation was implemented and independently reviewed. No production role behavior changed. The role commits are: | Repository | Remediation commit | Final integration result | @@ -273,3 +273,10 @@ each repository's `tests/integration` entry point successfully. Each cleanup path restored the default deployment, and the role repositories were clean after their remediation commits. The existing classifications remain the authoritative rationale for this implemented suite shape. + +Independent review found no remaining code-quality issues or implementation +gaps. The three complete integration entry points passed again sequentially; +Bash syntax validation and `git diff --check` passed; native application checks +confirmed that cleanup restored usable default deployments; and every +repository in the workspace remained clean. No additional remediation record +was required. diff --git a/docs/status.md b/docs/status.md index 8780f42..a437488 100644 --- a/docs/status.md +++ b/docs/status.md @@ -4,7 +4,7 @@ Last updated: 2026-09-09 Current milestone: MVP standalone application roles -Current focus: Review the implemented test-suite remediation. +Current focus: Select the next unit of work. | ID | Deliverable | Owner | State | Evidence or next step | | --- | --- | --- | --- | --- | @@ -21,7 +21,7 @@ Current focus: Review the implemented test-suite remediation. | F0002 | Install and configure Audita | ansible-role-audita | done | Field-specific configuration completed in `e61f02f` and `847c4c3`; full suite passed | | F0003 | Install and configure Notarius | ansible-role-notarius | done | Implemented and validated through remediation commit `eb4f511` | | A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` | -| R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | in_progress | Pending review; role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed their full integration suites | +| R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review | ## Updating This File