eric/ansible-workspace-dnd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Complete the test suite remediation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-09 13:03:24 +00:00
parent 6d6b2307ff
commit 332bc4742b
3 changed files with 19 additions and 515 deletions
Download Patch File Download Diff File Expand all files Collapse all files

519
docs/roadmap/implementation.md
View File

@@ -1,514 +1,11 @@
# Implementation Plan R0001: Classical Test Suite Remediation
# Implementation Plan
Status: ready
Status: pending
Target executor: GPT-5.6-Terra
No implementation plan is currently active. Test-suite remediation R0001 was
implemented in role commits `e08c63c`, `d2d0b9d`, and `cfecf37`, then reviewed
successfully. Its rationale and validation record remain in
[the test-suite audit findings](test-suite-audit-findings.md).
Authoritative audit:
`docs/roadmap/test-suite-audit-findings.md`
Active plan location: `docs/roadmap/implementation.md`
## Objective
Replace the oversized, implementation-bound integration suites in the three
completed standalone role repositories with lean classical, state-based
suites. Retain tests of stable role-owned behavior and application outcomes;
remove mutable-default snapshots, task-trace assertions, tests of trusted
Ansible modules, redundant scenarios, and negative platform matrices.
Production role behavior is not changing. This plan changes only test
implementations, test-related role documentation, and the workspace planning
records identified by the audit.
## Execution Rules
- Execute stages in numeric order.
- Before each stage, confirm its owning repository is clean and record its
starting revision.
- Preserve unrelated and pre-existing changes.
- Do not modify production tasks, defaults, templates, argument
specifications, dependency pins, or role metadata. If a retained test
exposes a production defect, stop and report it instead of expanding this
remediation.
- Follow the normative testing house style in `docs/conventions.md` and the
classifications in the authoritative audit.
- Exercise roles through their public variables and inspect final application
behavior. Do not parse task names, task order, or individual task status.
- Retain only aggregate `changed=0` checks on specified immediate repeat runs.
Do not require a nonzero change count on preceding runs.
- Do not verify ownership, permissions, path types, checkout commits, or other
postconditions already declared through trusted modules.
- Do not add a test merely to replace each removed test one-for-one. The target
is the smallest suite that protects the retained behavior.
- Keep each repository's shell driver self-contained. Do not create a shared
cross-repository test helper or abstraction.
- Run each stage's full repository integration entry point before committing.
- Commit each completed stage separately with the plain-English intent given
in that stage.
- Do not begin a dependent stage after an earlier stage fails validation.
## Starting State and Preconditions
- The workspace conventions, audit plan, audit findings, status update, and
this implementation plan must be reviewed and committed in the workspace
repository before Stage 1 begins.
- `ansible-role-seriatim` starts clean at `fb8dc41`.
- `ansible-role-audita` starts clean at `847c4c3`.
- `ansible-role-notarius` starts clean at `eb4f511`.
- The workspace `.venv` provides Ansible Core and ansible-lint, and the local
Ubuntu 26.04 `x86_64` host remains the approved MVP integration proxy.
- The executor can use passwordless privilege escalation for the existing
`/usr/local` integration-test paths and can reach Ansible Galaxy, the three
application Git repositories, Snap infrastructure, and Go module sources.
- Existing installed application state may be present. Each driver must remain
repeatable from that state and restore its default role deployment through a
best-effort cleanup trap.
- Audita exposes no native application-release identity command. Its tags
`v0.9.0` and `v0.9.1` currently resolve to the same commit. This plan
deliberately omits an Audita version-selection test rather than substituting
a raw-commit or Git-module assertion.
## Repository Scope
Repositories that change:
- `ansible-role-seriatim`: lean integration driver and verifier, removal of
the unsupported-platform playbook, and accurate validation documentation.
- `ansible-role-audita`: lean integration driver and verifiers, removal of the
unsupported-platform playbook, and accurate validation documentation.
- `ansible-role-notarius`: lean integration driver and verifiers, removal of
the unsupported-platform playbook, and accurate validation documentation.
- Workspace control repository: test-specific corrections to Features 0001,
0002, and 0003, remediation status, and cross-repository validation evidence.
Repositories that may be inspected but must remain unchanged:
- `../seriatim`, `../audita`, and `../notarius` for native CLI behavior.
- Every other application, role, collection, and proof-of-concept repository.
## Stage 1: Simplify the Seriatim Test Suite
### Objective
Reduce Seriatim's integration suite to static validation and representative
tests of default usability, explicit version selection, missing-binary
recovery, caller-supplied Go, and clean repeated convergence.
### Preconditions
- The workspace planning commit described above exists.
- `ansible-role-seriatim` is clean at `fb8dc41`.
- The existing `tests/integration` baseline passes.
### Owning Repository
`ansible-role-seriatim`
### Required Changes
1. Retain `tests/integration` as the single entry point, its private temporary
directory, pinned collection installation, syntax check, production-profile
ansible-lint, and best-effort cleanup convergence.
2. Retain a default role convergence as a smoke test. Exercise
`/usr/local/bin/seriatim --version` successfully without comparing it to the
current default version, then immediately apply the default role again and
require only the aggregate recap `changed=0` and `failed=0`.
3. Add or retain one explicit version-selection scenario using
`seriatim_version: v1.4.0`. Verify through the native CLI that the result is
exactly `seriatim version v1.4.0`. Do not resolve or assert the corresponding
Git commit.
4. Combine missing-binary recovery and caller-supplied build tooling into one
representative scenario:
- create an executable temporary wrapper that records invocation and then
executes `/snap/bin/go` with the original arguments;
- remove `/usr/local/bin/seriatim`;
- apply the role with `seriatim_version: v1.4.0`,
`seriatim_manage_build_dependencies: false`, and
`seriatim_go_binary` set to the wrapper;
- require the wrapper log to show that the supplied executable was used;
- verify native output `seriatim version v1.4.0`; and
- immediately repeat with the same variables and require aggregate
`changed=0` and `failed=0`.
5. Delete `tests/verify.yml` and perform the narrow native version assertions
directly in the shell driver. Do not retain source, commit, ownership,
permission, file-type, or checksum inspection.
6. Remove the binary-mode mutation and repair scenario, every
`require_changed_recap` use, every task-name or individual-status grep, the
default and previous commit constants, redundant default-restoration
assertions, and final duplicate shell assertions.
7. Delete `tests/unsupported.yml` and remove it from lint and execution. Do not
replace its distribution, release, or architecture cases.
8. Keep a cleanup trap that reapplies the default role and removes only the
private temporary directory. Cleanup output may remain suppressed and must
not become another behavioral assertion.
9. Rewrite the README validation section to describe only the retained static
checks and behavioral scenarios. Do not advertise exact-default, metadata,
task-arrangement, or platform-matrix coverage.
### Validation
Run:
```text
tests/integration
git diff --check
git status --short
```
Expected result: static validation and every retained Seriatim scenario pass;
the specified repeat runs report no changes; cleanup completes; and the only
working-tree changes are the planned test and README edits.
### Completion Criteria
- No test asserts Seriatim's current default version or a Git commit.
- No test inspects task names, individual task status, ownership, permissions,
or the platform assertion branches.
- Explicit `v1.4.0` selection is verified through Seriatim itself.
- One scenario proves both missing-binary recovery and use of the configured Go
executable, followed by a clean repeat.
- The complete integration entry point passes.
### Commit
Commit intent: `Simplify the Seriatim integration tests`
## Stage 2: Simplify the Audita Test Suite
### Objective
Reduce Audita's integration suite to static validation and representative
tests of default usability, native configuration validity, explicit
configuration rendering, missing-binary recovery, caller-supplied Go, and clean
repeated convergence.
### Preconditions
- Stage 1 is complete and committed.
- `ansible-role-audita` is clean at `847c4c3`.
- The existing `tests/integration` baseline passes.
### Owning Repository
`ansible-role-audita`
### Required Changes
1. Retain `tests/integration` as the single entry point, its private temporary
directory, pinned collection installation, syntax check, production-profile
ansible-lint, and best-effort cleanup convergence.
2. Retain a default role convergence as a smoke test. Validate the managed
configuration with
`/usr/local/bin/audita config validate --config /usr/local/etc/audita/config.yml`,
then immediately repeat the default role and require only aggregate
`changed=0` and `failed=0`.
3. Replace the exhaustive configuration document and separate falsey scenario
with one representative explicit configuration. Supply these values:
- `audita_config_version: 1`;
- `audita_pipeline_modules: ["glossary"]`;
- `audita_llm_proposal_base_url: "https://openrouter.ai/api/v1"`;
- `audita_llm_proposal_model: "openrouter/google/gemma-4-31b-it"`;
- `audita_llm_proposal_timeout: "10s"`;
- `audita_llm_proposal_max_retries: 0`;
- `audita_context_description: ""`;
- `audita_diagnostics_retention: "never"`;
- `audita_llm_validation_model: null`;
- `audita_output_schema: null`; and
- `audita_diagnostics_work_dir: null`.
4. In `tests/verify-config.yml`, parse the rendered YAML and assert only that:
- `version` is `1`;
- `pipeline.modules` is `["glossary"]`;
- the selected proposal URL, model, timeout, and zero retry count are
present;
- the empty context description and diagnostics retention value are
present;
- `llm.validation`, `output`, and `diagnostics.work_dir` are absent; and
- Audita's native configuration validation command succeeds.
Do not compare the complete parsed document to an expected mapping.
5. Immediately repeat the representative configuration convergence with the
same explicit variables and require aggregate `changed=0` and `failed=0`.
6. Combine missing-binary recovery and caller-supplied build tooling into one
representative scenario. Create a logging wrapper around `/snap/bin/go`,
remove `/usr/local/bin/audita`, converge with
`audita_manage_build_dependencies: false` and `audita_go_binary` set to the
wrapper, verify wrapper use, validate the managed configuration through
Audita, and immediately repeat the same convergence with aggregate
`changed=0` and `failed=0`.
7. Delete `tests/verify.yml` and invoke Audita's native configuration validator
directly from the shell driver for the smoke and recovery scenarios.
Successful command completion is sufficient; do not separately assert the
validator's fixed success text.
8. Remove the exact default configuration comparison, source-commit checks,
source/binary/configuration metadata checks, binary-mode repair scenario,
all `require_changed_recap` uses, and every task-name or individual-status
grep.
9. Remove the raw alternate-commit scenario entirely. Do not replace it with a
`git rev-parse`, tag-resolution, embedded-string, or Go build-metadata test.
Do not add an Audita version-selection test until the application provides a
stable native release-identity interface.
10. Delete `tests/unsupported.yml` and remove it from lint and execution. Do not
replace its platform cases.
11. Retain default restoration only in the cleanup trap. Remove its fixed
commit, exact document, task-status, and extra idempotence assertions.
12. Rewrite the README validation section to describe only the retained static
checks and behavioral scenarios and to avoid claiming application-version,
metadata, task-arrangement, or platform-matrix coverage.
### Validation
Run:
```text
tests/integration
git diff --check
git status --short
```
Expected result: static validation and every retained Audita scenario pass;
both default and representative configurations validate natively; the
specified repeats report no changes; cleanup completes; and the only
working-tree changes are the planned test and README edits.
### Completion Criteria
- No test asserts Audita's current default, a source commit, or a complete
rendered configuration document.
- No test inspects task traces, trusted-module postconditions, or platform
assertion branches.
- One focused explicit case proves representative nesting, null omission, and
falsey preservation and passes Audita's validator.
- One scenario proves both missing-binary recovery and use of the configured Go
executable, followed by a clean repeat.
- The absence of native Audita version coverage is documented accurately and
is not replaced by an implementation proxy.
- The complete integration entry point passes.
### Commit
Commit intent: `Simplify the Audita integration tests`
## Stage 3: Simplify the Notarius Test Suite
### Objective
Reduce Notarius's integration suite to static validation and representative
tests of default usability, explicit version selection, missing-binary
recovery, caller-supplied Go, configuration behavior, caller-side pipeline
composition, native validation, and clean repeated convergence.
### Preconditions
- Stages 1 and 2 are complete and committed.
- `ansible-role-notarius` is clean at `eb4f511`.
- The existing `tests/integration` baseline passes.
### Owning Repository
`ansible-role-notarius`
### Required Changes
1. Retain `tests/integration` as the single entry point, its private temporary
directory, pinned collection installation, syntax check, production-profile
ansible-lint, and best-effort cleanup convergence.
2. Retain a default convergence as a smoke test. Require
`/usr/local/bin/notarius --version` and
`/usr/local/bin/notarius config validate --config /usr/local/etc/notarius/config.yml`
to succeed without comparing the version output or complete configuration
to current defaults. Immediately repeat the default role and require only
aggregate `changed=0` and `failed=0`.
3. Retain one explicit version-selection scenario with
`notarius_version: v0.5.0` and verify exact native output
`notarius v0.5.0`. Do not resolve or assert the corresponding Git commit.
4. Combine missing-binary recovery and caller-supplied build tooling into one
scenario. Create a logging wrapper around `/snap/bin/go`, remove
`/usr/local/bin/notarius`, converge with `notarius_version: v0.5.0`,
`notarius_manage_build_dependencies: false`, and `notarius_go_binary` set to
the wrapper, verify wrapper use and native output `notarius v0.5.0`, then
immediately repeat with aggregate `changed=0` and `failed=0`.
5. Replace the exhaustive configuration, separate profile-source cases,
separate falsey case, and separate pipeline-composition case with one
representative explicit configuration. Define two caller-side pipeline
mappings and set `notarius_pipelines` to their ordinary `combine` result.
Supply these additional values:
- `notarius_config_version: 4`;
- `notarius_promptkit_local_backend_endpoint: "http://localhost:8000/v1"`;
- `notarius_promptkit_local_backend_concurrency_limit: 0`;
- `notarius_cache_checkpoints_enabled: false`;
- `notarius_cache_chunk_plans_directory: ""`;
- `notarius_output_directory: "/tmp/notarius-output"`;
- `notarius_promptkit_profile_dir: null`;
- `notarius_promptkit_profile_file: null`; and
- `notarius_debug_directory: null`.
Use the two valid `dnd-session` and `dnd-archive` pipeline definitions from
the existing composition scenario.
6. In `tests/verify-config.yml`, parse the rendered YAML and assert only that:
- `version` is `4`;
- local-backend endpoint and zero concurrency are present;
- checkpoints are explicitly disabled and the empty chunk-plan directory is
present;
- the selected output directory is present;
- both composed pipeline IDs and their supplied definitions are present
without rewriting;
- `promptkit.profile_dir`, `promptkit.profile_file`, and the top-level
`debug` section are absent; and
- Notarius's native configuration validation succeeds.
Do not compare the complete document to an expected mapping.
7. Immediately repeat the representative configuration convergence with the
same explicit variables and require aggregate `changed=0` and `failed=0`.
8. Delete `tests/verify.yml`. Perform native CLI smoke, configuration
validation, and the explicit `v0.5.0` comparison directly in the shell
driver.
9. Remove current-default version and commit assertions, complete default
configuration comparisons, all source/binary/configuration metadata checks,
the binary-mode repair scenario, every `require_changed_recap` use,
`require_configuration_change_without_build`, and every task-name or
individual-status grep.
10. Delete the standalone `profile_dir` and `profile_file` scenarios. Their
simple leaf mappings do not require dedicated replacements.
11. Delete `tests/unsupported.yml` and remove it from lint and execution. Do not
replace its platform cases.
12. Retain default restoration only in the cleanup trap and remove duplicate
final assertions.
13. Rewrite the README validation section to describe only the retained static
checks and behavioral scenarios. Do not advertise current-default,
metadata, task-arrangement, profile-branch, or platform-matrix coverage.
### Validation
Run:
```text
tests/integration
git diff --check
git status --short
```
Expected result: static validation and every retained Notarius scenario pass;
the explicit tag is verified through Notarius; representative falsey, null,
and composed-pipeline behavior is verified and validates natively; the
specified repeats report no changes; cleanup completes; and only the planned
test and README changes remain.
### Completion Criteria
- No test snapshots Notarius's current default release, commit, or complete
default configuration.
- No test inspects task traces, trusted-module postconditions, or platform
assertion branches.
- Explicit `v0.5.0` selection is verified through Notarius itself.
- One scenario proves both missing-binary recovery and use of the configured Go
executable, followed by a clean repeat.
- One representative configuration case proves null omission, falsey
preservation, caller-side pipeline composition, and native validity.
- The complete integration entry point passes.
### Commit
Commit intent: `Simplify the Notarius integration tests`
## Stage 4: Reconcile Workspace Documentation and Validate the Suite
### Objective
Align durable test-specific roadmap language with the canonical testing policy,
record the remediation implementation for review, and validate all three lean
suites together.
### Preconditions
- Stages 1 through 3 are complete and committed in their owning repositories.
- The workspace repository is clean at the planning commit created before
Stage 1.
- The three role repositories contain only their committed remediation changes.
### Owning Repository
Workspace control repository (`ansible-workspace-dnd`)
### Required Changes
1. Revise only test-specific language in Features 0001, 0002, and 0003. Retain
their durable production requirements, including current defaults, paths,
ownership, permissions, platform support, and convergence behavior.
2. In Feature 0001, describe the exact native version assertion as coverage of
an explicitly selected immutable tag rather than an assertion about the
role's current default.
3. In Feature 0002, remove the incorrect claim that tests execute Audita to
verify its installed application version. State that the current CLI has no
native release-identity command and that the suite deliberately avoids a
raw-commit proxy. Retain native configuration validation.
4. In Feature 0003, describe exact version coverage using explicitly selected
`v0.5.0`, not the current default `v0.6.0`. Keep the current default as a
production requirement rather than a frozen test expectation.
5. Remove any implication in the three roadmaps that each platform assertion
must have a negative integration-test matrix. Do not remove platform support
itself from scope or acceptance criteria.
6. Update `docs/roadmap/test-suite-audit-findings.md` to say remediation was
implemented and is pending review. Add the three role commit hashes and the
final validation results; do not erase the audit classifications.
7. Update `docs/status.md` so `R0001` remains `in_progress` pending review, its
evidence names the three role commits, and the current focus is review of
the implemented test-suite remediation. Do not mark remediation `done`.
8. Add concise progress notes to this plan with the four stage commits and any
deviations. Do not replace the active plan with a pending notice before
review.
### Validation
Run sequentially from the workspace root:
```text
(cd ansible-role-seriatim && tests/integration)
(cd ansible-role-audita && tests/integration)
(cd ansible-role-notarius && tests/integration)
git diff --check
scripts/status
```
Expected result: all static and behavioral checks pass; every role cleanup path
completes; the three role repositories are clean and contain their remediation
commits; no unrelated repository changed; and the workspace has only the
planned documentation changes before its stage commit.
### Completion Criteria
- All three full integration suites pass sequentially from the workspace.
- Feature roadmaps distinguish production defaults from stable test inputs.
- Audita's native version limitation is stated accurately.
- Audit findings contain implementation and validation evidence without being
rewritten as an implementation log.
- Workspace status leaves remediation pending review.
- Every repository changed by the implementation is clean after its owning
commit.
### Commit
Commit intent: `Record the test suite remediation results`
## Final Validation and Handoff
- Report the Stage 1 through Stage 4 commit hashes and confirm the working-tree
state of every repository in `workspace.yml`.
- Summarize the retained scenarios for each role and identify any deviation
from this plan.
- Report the Audita version-identity limitation; do not silently fill it with a
source-control assertion.
- Report all validation commands and results.
- Do not push commits unless the user separately authorizes a push.
- Do not mark `R0001` done or replace this plan with a pending notice. Final
completion requires review of the implemented suites.
## Progress Notes
- Planning complete; implementation has not started.
- [x] Stage 1 complete — `ansible-role-seriatim` commit `e08c63c`; full
`tests/integration` passed. No deviations.
- [x] Stage 2 complete — `ansible-role-audita` commit `d2d0b9d`; full
`tests/integration` passed. No deviations.
- [x] Stage 3 complete — `ansible-role-notarius` commit `cfecf37`; full
`tests/integration` passed. No deviations.
- [x] Stage 4 complete — workspace commit `ca82dd7`; sequential full
integration validation passed for Seriatim, Audita, and Notarius. No
deviations.
Select and approve the next unit of work before replacing this notice with a
new staged implementation plan.

11
docs/roadmap/test-suite-audit-findings.md
View File

@@ -1,6 +1,6 @@
# Test Suite Audit Findings
Status: remediation implemented; pending review
Status: resolved
## Executive Summary
@@ -259,7 +259,7 @@ The audit made no changes to production roles or test implementations.
## Remediation Status
The recommended remediation was implemented and is pending review. No
The recommended remediation was implemented and independently reviewed. No
production role behavior changed. The role commits are:
| Repository | Remediation commit | Final integration result |
@@ -273,3 +273,10 @@ each repository's `tests/integration` entry point successfully. Each cleanup
path restored the default deployment, and the role repositories were clean
after their remediation commits. The existing classifications remain the
authoritative rationale for this implemented suite shape.
Independent review found no remaining code-quality issues or implementation
gaps. The three complete integration entry points passed again sequentially;
Bash syntax validation and `git diff --check` passed; native application checks
confirmed that cleanup restored usable default deployments; and every
repository in the workspace remained clean. No additional remediation record
was required.

4
docs/status.md
View File

@@ -4,7 +4,7 @@ Last updated: 2026-09-09
Current milestone: MVP standalone application roles
Current focus: Review the implemented test-suite remediation.
Current focus: Select the next unit of work.
| ID | Deliverable | Owner | State | Evidence or next step |
| --- | --- | --- | --- | --- |
@@ -21,7 +21,7 @@ Current focus: Review the implemented test-suite remediation.
| F0002 | Install and configure Audita | ansible-role-audita | done | Field-specific configuration completed in `e61f02f` and `847c4c3`; full suite passed |
| F0003 | Install and configure Notarius | ansible-role-notarius | done | Implemented and validated through remediation commit `eb4f511` |
| A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` |
| R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | in_progress | Pending review; role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed their full integration suites |
| R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review |
## Updating This File