ansible-workspace-dnd/docs/roadmap/implementation.md

Implementation Plan R0001: Classical Test Suite Remediation

Status: ready

Target executor: GPT-5.6-Terra

Authoritative audit: docs/roadmap/test-suite-audit-findings.md

Active plan location: docs/roadmap/implementation.md

Objective

Replace the oversized, implementation-bound integration suites in the three completed standalone role repositories with lean classical, state-based suites. Retain tests of stable role-owned behavior and application outcomes; remove mutable-default snapshots, task-trace assertions, tests of trusted Ansible modules, redundant scenarios, and negative platform matrices.

Production role behavior is not changing. This plan changes only test implementations, test-related role documentation, and the workspace planning records identified by the audit.

Execution Rules

• Execute stages in numeric order.
• Before each stage, confirm its owning repository is clean and record its starting revision.
• Preserve unrelated and pre-existing changes.
• Do not modify production tasks, defaults, templates, argument specifications, dependency pins, or role metadata. If a retained test exposes a production defect, stop and report it instead of expanding this remediation.
• Follow the normative testing house style in docs/conventions.md and the classifications in the authoritative audit.
• Exercise roles through their public variables and inspect final application behavior. Do not parse task names, task order, or individual task status.
• Retain only aggregate changed=0 checks on specified immediate repeat runs. Do not require a nonzero change count on preceding runs.
• Do not verify ownership, permissions, path types, checkout commits, or other postconditions already declared through trusted modules.
• Do not add a test merely to replace each removed test one-for-one. The target is the smallest suite that protects the retained behavior.
• Keep each repository's shell driver self-contained. Do not create a shared cross-repository test helper or abstraction.
• Run each stage's full repository integration entry point before committing.
• Commit each completed stage separately with the plain-English intent given in that stage.
• Do not begin a dependent stage after an earlier stage fails validation.

Starting State and Preconditions

• The workspace conventions, audit plan, audit findings, status update, and this implementation plan must be reviewed and committed in the workspace repository before Stage 1 begins.
• ansible-role-seriatim starts clean at fb8dc41.
• ansible-role-audita starts clean at 847c4c3.
• ansible-role-notarius starts clean at eb4f511.
• The workspace .venv provides Ansible Core and ansible-lint, and the local Ubuntu 26.04 x86_64 host remains the approved MVP integration proxy.
• The executor can use passwordless privilege escalation for the existing /usr/local integration-test paths and can reach Ansible Galaxy, the three application Git repositories, Snap infrastructure, and Go module sources.
• Existing installed application state may be present. Each driver must remain repeatable from that state and restore its default role deployment through a best-effort cleanup trap.
• Audita exposes no native application-release identity command. Its tags v0.9.0 and v0.9.1 currently resolve to the same commit. This plan deliberately omits an Audita version-selection test rather than substituting a raw-commit or Git-module assertion.

Repository Scope

Repositories that change:

• ansible-role-seriatim: lean integration driver and verifier, removal of the unsupported-platform playbook, and accurate validation documentation.
• ansible-role-audita: lean integration driver and verifiers, removal of the unsupported-platform playbook, and accurate validation documentation.
• ansible-role-notarius: lean integration driver and verifiers, removal of the unsupported-platform playbook, and accurate validation documentation.
• Workspace control repository: test-specific corrections to Features 0001, 0002, and 0003, remediation status, and cross-repository validation evidence.

Repositories that may be inspected but must remain unchanged:

• ../seriatim, ../audita, and ../notarius for native CLI behavior.
• Every other application, role, collection, and proof-of-concept repository.

Stage 1: Simplify the Seriatim Test Suite

Objective

Reduce Seriatim's integration suite to static validation and representative tests of default usability, explicit version selection, missing-binary recovery, caller-supplied Go, and clean repeated convergence.

Preconditions

• The workspace planning commit described above exists.
• ansible-role-seriatim is clean at fb8dc41.
• The existing tests/integration baseline passes.

Owning Repository

ansible-role-seriatim

Required Changes

1. Retain tests/integration as the single entry point, its private temporary directory, pinned collection installation, syntax check, production-profile ansible-lint, and best-effort cleanup convergence.
2. Retain a default role convergence as a smoke test. Exercise /usr/local/bin/seriatim --version successfully without comparing it to the current default version, then immediately apply the default role again and require only the aggregate recap changed=0 and failed=0.
3. Add or retain one explicit version-selection scenario using seriatim_version: v1.4.0. Verify through the native CLI that the result is exactly seriatim version v1.4.0. Do not resolve or assert the corresponding Git commit.
4. Combine missing-binary recovery and caller-supplied build tooling into one representative scenario:
   • create an executable temporary wrapper that records invocation and then executes /snap/bin/go with the original arguments;
   • remove /usr/local/bin/seriatim;
   • apply the role with seriatim_version: v1.4.0, seriatim_manage_build_dependencies: false, and seriatim_go_binary set to the wrapper;
   • require the wrapper log to show that the supplied executable was used;
   • verify native output seriatim version v1.4.0; and
   • immediately repeat with the same variables and require aggregate changed=0 and failed=0.
5. Delete tests/verify.yml and perform the narrow native version assertions directly in the shell driver. Do not retain source, commit, ownership, permission, file-type, or checksum inspection.
6. Remove the binary-mode mutation and repair scenario, every require_changed_recap use, every task-name or individual-status grep, the default and previous commit constants, redundant default-restoration assertions, and final duplicate shell assertions.
7. Delete tests/unsupported.yml and remove it from lint and execution. Do not replace its distribution, release, or architecture cases.
8. Keep a cleanup trap that reapplies the default role and removes only the private temporary directory. Cleanup output may remain suppressed and must not become another behavioral assertion.
9. Rewrite the README validation section to describe only the retained static checks and behavioral scenarios. Do not advertise exact-default, metadata, task-arrangement, or platform-matrix coverage.

Validation

Run:

tests/integration
git diff --check
git status --short

Expected result: static validation and every retained Seriatim scenario pass; the specified repeat runs report no changes; cleanup completes; and the only working-tree changes are the planned test and README edits.

Completion Criteria

• No test asserts Seriatim's current default version or a Git commit.
• No test inspects task names, individual task status, ownership, permissions, or the platform assertion branches.
• Explicit v1.4.0 selection is verified through Seriatim itself.
• One scenario proves both missing-binary recovery and use of the configured Go executable, followed by a clean repeat.
• The complete integration entry point passes.

Commit

Commit intent: Simplify the Seriatim integration tests

Stage 2: Simplify the Audita Test Suite

Objective

Reduce Audita's integration suite to static validation and representative tests of default usability, native configuration validity, explicit configuration rendering, missing-binary recovery, caller-supplied Go, and clean repeated convergence.

Preconditions

• Stage 1 is complete and committed.
• ansible-role-audita is clean at 847c4c3.
• The existing tests/integration baseline passes.

Owning Repository

ansible-role-audita

Required Changes

1. Retain tests/integration as the single entry point, its private temporary directory, pinned collection installation, syntax check, production-profile ansible-lint, and best-effort cleanup convergence.
2. Retain a default role convergence as a smoke test. Validate the managed configuration with /usr/local/bin/audita config validate --config /usr/local/etc/audita/config.yml, then immediately repeat the default role and require only aggregate changed=0 and failed=0.
3. Replace the exhaustive configuration document and separate falsey scenario with one representative explicit configuration. Supply these values:
   • audita_config_version: 1;
   • audita_pipeline_modules: ["glossary"];
   • audita_llm_proposal_base_url: "https://openrouter.ai/api/v1";
   • audita_llm_proposal_model: "openrouter/google/gemma-4-31b-it";
   • audita_llm_proposal_timeout: "10s";
   • audita_llm_proposal_max_retries: 0;
   • audita_context_description: "";
   • audita_diagnostics_retention: "never";
   • audita_llm_validation_model: null;
   • audita_output_schema: null; and
   • audita_diagnostics_work_dir: null.
4. In tests/verify-config.yml, parse the rendered YAML and assert only that:
   • version is 1;
   • pipeline.modules is ["glossary"];
   • the selected proposal URL, model, timeout, and zero retry count are present;
   • the empty context description and diagnostics retention value are present;
   • llm.validation, output, and diagnostics.work_dir are absent; and
   • Audita's native configuration validation command succeeds. Do not compare the complete parsed document to an expected mapping.
5. Immediately repeat the representative configuration convergence with the same explicit variables and require aggregate changed=0 and failed=0.
6. Combine missing-binary recovery and caller-supplied build tooling into one representative scenario. Create a logging wrapper around /snap/bin/go, remove /usr/local/bin/audita, converge with audita_manage_build_dependencies: false and audita_go_binary set to the wrapper, verify wrapper use, validate the managed configuration through Audita, and immediately repeat the same convergence with aggregate changed=0 and failed=0.
7. Delete tests/verify.yml and invoke Audita's native configuration validator directly from the shell driver for the smoke and recovery scenarios. Successful command completion is sufficient; do not separately assert the validator's fixed success text.
8. Remove the exact default configuration comparison, source-commit checks, source/binary/configuration metadata checks, binary-mode repair scenario, all require_changed_recap uses, and every task-name or individual-status grep.
9. Remove the raw alternate-commit scenario entirely. Do not replace it with a git rev-parse, tag-resolution, embedded-string, or Go build-metadata test. Do not add an Audita version-selection test until the application provides a stable native release-identity interface.
10. Delete tests/unsupported.yml and remove it from lint and execution. Do not replace its platform cases.
11. Retain default restoration only in the cleanup trap. Remove its fixed commit, exact document, task-status, and extra idempotence assertions.
12. Rewrite the README validation section to describe only the retained static checks and behavioral scenarios and to avoid claiming application-version, metadata, task-arrangement, or platform-matrix coverage.

Validation

Run:

tests/integration
git diff --check
git status --short

Expected result: static validation and every retained Audita scenario pass; both default and representative configurations validate natively; the specified repeats report no changes; cleanup completes; and the only working-tree changes are the planned test and README edits.

Completion Criteria

• No test asserts Audita's current default, a source commit, or a complete rendered configuration document.
• No test inspects task traces, trusted-module postconditions, or platform assertion branches.
• One focused explicit case proves representative nesting, null omission, and falsey preservation and passes Audita's validator.
• One scenario proves both missing-binary recovery and use of the configured Go executable, followed by a clean repeat.
• The absence of native Audita version coverage is documented accurately and is not replaced by an implementation proxy.
• The complete integration entry point passes.

Commit

Commit intent: Simplify the Audita integration tests

Stage 3: Simplify the Notarius Test Suite

Objective

Reduce Notarius's integration suite to static validation and representative tests of default usability, explicit version selection, missing-binary recovery, caller-supplied Go, configuration behavior, caller-side pipeline composition, native validation, and clean repeated convergence.

Preconditions

• Stages 1 and 2 are complete and committed.
• ansible-role-notarius is clean at eb4f511.
• The existing tests/integration baseline passes.

Owning Repository

ansible-role-notarius

Required Changes

1. Retain tests/integration as the single entry point, its private temporary directory, pinned collection installation, syntax check, production-profile ansible-lint, and best-effort cleanup convergence.
2. Retain a default convergence as a smoke test. Require /usr/local/bin/notarius --version and /usr/local/bin/notarius config validate --config /usr/local/etc/notarius/config.yml to succeed without comparing the version output or complete configuration to current defaults. Immediately repeat the default role and require only aggregate changed=0 and failed=0.
3. Retain one explicit version-selection scenario with notarius_version: v0.5.0 and verify exact native output notarius v0.5.0. Do not resolve or assert the corresponding Git commit.
4. Combine missing-binary recovery and caller-supplied build tooling into one scenario. Create a logging wrapper around /snap/bin/go, remove /usr/local/bin/notarius, converge with notarius_version: v0.5.0, notarius_manage_build_dependencies: false, and notarius_go_binary set to the wrapper, verify wrapper use and native output notarius v0.5.0, then immediately repeat with aggregate changed=0 and failed=0.
5. Replace the exhaustive configuration, separate profile-source cases, separate falsey case, and separate pipeline-composition case with one representative explicit configuration. Define two caller-side pipeline mappings and set notarius_pipelines to their ordinary combine result. Supply these additional values:
   • notarius_config_version: 4;
   • notarius_promptkit_local_backend_endpoint: "http://localhost:8000/v1";
   • notarius_promptkit_local_backend_concurrency_limit: 0;
   • notarius_cache_checkpoints_enabled: false;
   • notarius_cache_chunk_plans_directory: "";
   • notarius_output_directory: "/tmp/notarius-output";
   • notarius_promptkit_profile_dir: null;
   • notarius_promptkit_profile_file: null; and
   • notarius_debug_directory: null. Use the two valid dnd-session and dnd-archive pipeline definitions from the existing composition scenario.
6. In tests/verify-config.yml, parse the rendered YAML and assert only that:
   • version is 4;
   • local-backend endpoint and zero concurrency are present;
   • checkpoints are explicitly disabled and the empty chunk-plan directory is present;
   • the selected output directory is present;
   • both composed pipeline IDs and their supplied definitions are present without rewriting;
   • promptkit.profile_dir, promptkit.profile_file, and the top-level debug section are absent; and
   • Notarius's native configuration validation succeeds. Do not compare the complete document to an expected mapping.
7. Immediately repeat the representative configuration convergence with the same explicit variables and require aggregate changed=0 and failed=0.
8. Delete tests/verify.yml. Perform native CLI smoke, configuration validation, and the explicit v0.5.0 comparison directly in the shell driver.
9. Remove current-default version and commit assertions, complete default configuration comparisons, all source/binary/configuration metadata checks, the binary-mode repair scenario, every require_changed_recap use, require_configuration_change_without_build, and every task-name or individual-status grep.
10. Delete the standalone profile_dir and profile_file scenarios. Their simple leaf mappings do not require dedicated replacements.
11. Delete tests/unsupported.yml and remove it from lint and execution. Do not replace its platform cases.
12. Retain default restoration only in the cleanup trap and remove duplicate final assertions.
13. Rewrite the README validation section to describe only the retained static checks and behavioral scenarios. Do not advertise current-default, metadata, task-arrangement, profile-branch, or platform-matrix coverage.

Validation

Run:

tests/integration
git diff --check
git status --short

Expected result: static validation and every retained Notarius scenario pass; the explicit tag is verified through Notarius; representative falsey, null, and composed-pipeline behavior is verified and validates natively; the specified repeats report no changes; cleanup completes; and only the planned test and README changes remain.

Completion Criteria

• No test snapshots Notarius's current default release, commit, or complete default configuration.
• No test inspects task traces, trusted-module postconditions, or platform assertion branches.
• Explicit v0.5.0 selection is verified through Notarius itself.
• One scenario proves both missing-binary recovery and use of the configured Go executable, followed by a clean repeat.
• One representative configuration case proves null omission, falsey preservation, caller-side pipeline composition, and native validity.
• The complete integration entry point passes.

Commit

Commit intent: Simplify the Notarius integration tests

Stage 4: Reconcile Workspace Documentation and Validate the Suite

Objective

Align durable test-specific roadmap language with the canonical testing policy, record the remediation implementation for review, and validate all three lean suites together.

Preconditions

• Stages 1 through 3 are complete and committed in their owning repositories.
• The workspace repository is clean at the planning commit created before Stage 1.
• The three role repositories contain only their committed remediation changes.

Owning Repository

Workspace control repository (ansible-workspace-dnd)

Required Changes

1. Revise only test-specific language in Features 0001, 0002, and 0003. Retain their durable production requirements, including current defaults, paths, ownership, permissions, platform support, and convergence behavior.
2. In Feature 0001, describe the exact native version assertion as coverage of an explicitly selected immutable tag rather than an assertion about the role's current default.
3. In Feature 0002, remove the incorrect claim that tests execute Audita to verify its installed application version. State that the current CLI has no native release-identity command and that the suite deliberately avoids a raw-commit proxy. Retain native configuration validation.
4. In Feature 0003, describe exact version coverage using explicitly selected v0.5.0, not the current default v0.6.0. Keep the current default as a production requirement rather than a frozen test expectation.
5. Remove any implication in the three roadmaps that each platform assertion must have a negative integration-test matrix. Do not remove platform support itself from scope or acceptance criteria.
6. Update docs/roadmap/test-suite-audit-findings.md to say remediation was implemented and is pending review. Add the three role commit hashes and the final validation results; do not erase the audit classifications.
7. Update docs/status.md so R0001 remains in_progress pending review, its evidence names the three role commits, and the current focus is review of the implemented test-suite remediation. Do not mark remediation done.
8. Add concise progress notes to this plan with the four stage commits and any deviations. Do not replace the active plan with a pending notice before review.

Validation

Run sequentially from the workspace root:

(cd ansible-role-seriatim && tests/integration)
(cd ansible-role-audita && tests/integration)
(cd ansible-role-notarius && tests/integration)
git diff --check
scripts/status

Expected result: all static and behavioral checks pass; every role cleanup path completes; the three role repositories are clean and contain their remediation commits; no unrelated repository changed; and the workspace has only the planned documentation changes before its stage commit.

Completion Criteria

• All three full integration suites pass sequentially from the workspace.
• Feature roadmaps distinguish production defaults from stable test inputs.
• Audita's native version limitation is stated accurately.
• Audit findings contain implementation and validation evidence without being rewritten as an implementation log.
• Workspace status leaves remediation pending review.
• Every repository changed by the implementation is clean after its owning commit.

Commit

Commit intent: Record the test suite remediation results

Final Validation and Handoff

• Report the Stage 1 through Stage 4 commit hashes and confirm the working-tree state of every repository in workspace.yml.
• Summarize the retained scenarios for each role and identify any deviation from this plan.
• Report the Audita version-identity limitation; do not silently fill it with a source-control assertion.
• Report all validation commands and results.
• Do not push commits unless the user separately authorizes a push.
• Do not mark R0001 done or replace this plan with a pending notice. Final completion requires review of the implemented suites.

Progress Notes

• Planning complete; implementation has not started.
• Stage 1 complete — ansible-role-seriatim commit e08c63c; full tests/integration passed. No deviations.
• Stage 2 complete — ansible-role-audita commit d2d0b9d; full tests/integration passed. No deviations.
• Stage 3 complete — ansible-role-notarius commit cfecf37; full tests/integration passed. No deviations.