ansible-workspace-dnd/docs/roadmap/implementation.md

Implementation Plan 0022: Coordinated Suite Upgrade

Status: in_progress

Target executor: GPT-5.6-Terra

Feature roadmap: Feature 0022

Objective

Integrate the reviewed application upgrades into the pipeline role, assembled collection, and consumer repository. The approved roadmap owns scope, target state, and policy. No operational deployment or tagged release is included.

Execution Rules and Starting State

• Execute Stage 1 through Stage 6 in order. Before each stage, reread its listed policies, check applicable repository instructions, and confirm its owning repository's working-tree state. Preserve unrelated changes.
• Commit valid completed stages with the stated plain English intent. Do not advance past failed validation. Distinguish actual blockers from unavailable optional tools or operational credentials that this plan does not require.
• Use workspace .venv/bin Ansible tooling. Missing PATH entries are not a missing controller. Existing integration scripts choose local tooling and install dependencies into temporary paths.
• Pipeline/collection integration runs use local sudo and mutate the local development installation, including source/binaries, configuration, fixture secrets, and test users. Read harness cleanup and verify these locations are disposable first. Do not run operational playbooks or contact live inference, WhisperX, or S3 services. Public Git/dependency downloads are expected.
• Use ordinary declarative modules; no production validation, preliminary state probes, compatibility aliases, manifest conversion, pruning, or new credential-discovery mechanism. New tasks are limited to Audita profile directory creation and looped profile templating.
• Publish only the required reviewed role revisions and feature commits to existing origins as specified below. Check remote URLs and outgoing commits before each push. Never force-push or move tags. If unrelated commits would be published or credentials prevent publication, report that boundary; do not substitute local paths in consumer manifests or subtree imports.
• Do not edit standalone application-role implementations, upstream apps, campaign/prompt assets, unrelated consumer inventory, Vault content, or vendored consumer roles. Required code owners are pipeline, collection, and consumer; workspace owns progress. Keep Seriatim and native collection pins.
• The new retry/repair/deadline pass-throughs default to null. Preserve the existing Audita 900-second per-attempt profile policy. Do not copy native defaults into suite variables.

Stage 1: Make Reviewed Standalone Dependencies Public

Completion: complete (2026-09-29).

Required Policy Review

Read AGENTS.md, docs/orientation.md, docs/conventions.md, and docs/planning.md before starting.

Owning Repositories and Preconditions

The four standalone role repositories and workspace. Features 0018–0021 are complete. Confirm these exact revisions and their existing Gitea origins:

Role	Revision
Scriptorium	b174c8c2990bb79f8149d8f288c5e06ccb42bd73
Audita	c3dd59f5ac9b942306ffe540ae9e71d493998137
Notarius	6aef48b7e12e3651971444b8c224d96ff1879e6c
Narratio	3c0700b5323293d44e3b1e67a50074d82d7a3bff

Work and Validation

Mark F0022 in_progress. For each repository, fetch origin, inspect its status and outgoing history, and publish the reviewed revision to origin/main if not already reachable there. Use a normal non-force push of the exact revision to refs/heads/main where a fast-forward is appropriate. A remote already containing it needs no push. Verify reachability after fetch; do not replace a selected pin merely because a newer remote commit exists.

Commands, substituting the role and exact revision from the table:

git -C ROLE remote -v
git -C ROLE fetch origin
git -C ROLE log --oneline origin/main..REVISION
git -C ROLE push origin REVISION:refs/heads/main
git -C ROLE fetch origin
git -C ROLE merge-base --is-ancestor REVISION origin/main

Completion and Commit

All four pinned revisions are publicly available. No standalone code change or new role commit is needed. Record evidence and commit workspace progress as Record the coordinated suite upgrade in progress.

Stage 2: Migrate Suite Configuration and Existing Integration Cases

Completion: complete (2026-09-29).

Required Policy Review

Read AGENTS.md, docs/orientation.md, docs/conventions.md, docs/ansible.md, and docs/testing.md before starting.

Owning Repository and Preconditions

ansible-role-dnd-pipeline. Stage 1 is complete. Inspect current defaults, specs, dependency vars, tasks, templates, and tests; read tagged Audita v2.0.1 migration/configuration and Narratio v3.0.0 contracts linked by the roadmap.

Required Changes

1. Pin the four standalone roles in requirements.yml to Stage 1 revisions. Set application defaults to Scriptorium v0.15.0, Audita v2.0.1, Notarius v0.8.1, Narratio v3.0.0. Preserve Seriatim v1.5.0 and its role pin.
2. Add dnd_pipeline_audita_backends (dict), default dnd with endpoint https://inference.ai.rakestrawhome.com/v1, credential AUDITA_LLM_API_KEY, concurrency 4; omit queue capacity. Add dnd_pipeline_audita_profiles (dict) with dnd-proposal and dnd-validation keys. Each value is a complete native mapping containing matching id, backend dnd, model google/gemma-4-31b-it, and timeout_seconds: 900.
3. Add string defaults dnd_pipeline_audita_llm_proposal_profile: dnd-proposal and dnd_pipeline_audita_llm_validation_profile: dnd-validation. Map these and backends into the standalone role. Set audita_profiles_dir to /usr/local/etc/audita/profiles.
4. Add a file directory task and one looped template task over dnd_pipeline_audita_profiles | dict2items. Write profiles/<key>.yml using an audita-profile.yml.j2 template containing item.value | to_nice_yaml. Use runtime owner/group, suite configuration modes, and task-level become. Document that keys are filename-safe IDs matching each mapping's id. Do not merge/rewrite profile values or delete removed entries.
5. Remove old dnd_pipeline_audita_llm_{proposal,validation}_{base_url,model, api_key_env,timeout,max_retries}, dnd_pipeline_audita_concurrency_*, and diagnostics retention from defaults, specs, dependency wiring, and active examples. Preserve diagnostics work directory and unchanged domain tuning. Convert normalization max gap/ellipsis gap/max duration to numeric 3.0/3.0/240.0 with float specs. Leave debug/reporting at native defaults.
6. Add these optional nullable suite variables, forwarding to the corresponding standalone variable by removing dnd_pipeline_:
   • dnd_pipeline_audita_llm_transient_retries: int.
   • dnd_pipeline_audita_llm_repair_attempts: int.
   • dnd_pipeline_audita_llm_operation_timeout: str.
   • dnd_pipeline_notarius_promptkit_transient_retries: int.
   • dnd_pipeline_scriptorium_transient_retries: int. Keep nulls and explicit zero; do not supply a fallback native default.
7. Add list-of-string credential defaults dnd_pipeline_narratio_audita_credentials: [AUDITA_LLM_API_KEY] and dnd_pipeline_narratio_notarius_credentials: [OPENROUTER_API_KEY]; wire to Narratio's credential lists. Remove the old suite Notarius singular-key variable and obsolete standalone-role arguments. Preserve secrets delivery.
8. Change Narratio/Notarius directory creation and testing-file destinations from profiles to overlays. Change Narratio testing to notarius.config_overlay: testing. Leave old ignored files alone. Preserve Audita's LLM profiles/, artifact fragments, model selections, caches, paths, publication, ACLs, and operator behavior.
9. Migrate existing tests/integration selectors and parsed assertions: Narratio --overlay, Notarius --config-overlay, config_overlay, and credential lists replace retired fields. Direct Audita validation receives the harness's synthetic AUDITA_LLM_API_KEY explicitly in its environment; do not assume Narratio's bootstrap applies to that process. Add Scriptorium native config validation. Keep old meaningful scenarios, clean repeats, and fixture cleanup; do not test upstream forwarding internals.
10. Update README public variables and migration mapping in the same commit so the role's current interface is documented. Explain defaults versus inventory overrides and retain existing native configuration ownership.

Validation

From workspace root:

bash -n ansible-role-dnd-pipeline/tests/integration
ansible-role-dnd-pipeline/tests/integration
git -C ansible-role-dnd-pipeline diff --check

The harness installs public pinned dependencies and includes syntax/lint. Require native acceptance of base/testing configuration and existing meaningful behavior with clean repeats, without live service calls. If an unrelated asset contract incompatibility appears, report it rather than rewriting assets.

Completion and Commit

Suite configuration, specs, docs, and existing tests are compatible together. Commit as Integrate the reviewed application upgrades into the suite.

Stage 3: Verify Inventory Overrides and Finish Suite Migration Guidance

Required Policy Review

Read AGENTS.md, docs/orientation.md, docs/conventions.md, docs/ansible.md, and docs/testing.md before starting.

Owning Repository and Preconditions

ansible-role-dnd-pipeline. Stage 2 passes. Extend its existing harness rather than introducing a test framework or changing production convergence.

Required Changes

1. Add one representative override case with a fixture backend endpoint, caller-composed native profile map, selected proposal/validation IDs, and an explicit custom credential-name list. Use synthetic values only; provide required names both through managed fixture secrets for Narratio and direct test environment for Audita. Use simple offline-valid profile values.
2. In that case supply zero for the four retry/repair integers and a valid explicit Audita operation timeout, such as 20m. Inspect native effective settings/profile views to verify supplied values and selected models. Inspect Narratio configuration to verify the credential names. Do not assert current application default retry counts or model catalogs.
3. Explicitly null the five new pass-throughs in a baseline case and inspect rendered application YAML for omission (effective views may include native defaults). Require an immediate unchanged repeat for the override case. Retain existing pipeline-map composition, testing selection, shared-user, cache, storage, and artifact tests; do not add module metadata checks.
4. Document exact old-to-new inventory choices, shared backend concurrency, filename/profile ID contract, credential list ownership, and CLI changes. Explain the retained 900-second attempt setting and independent native operation deadline. Link upstream manifest migration; do not automate it. Explain ignored legacy testing files and normal named-selector invalidation.

Validation

bash -n ansible-role-dnd-pipeline/tests/integration
ansible-role-dnd-pipeline/tests/integration
git -C ansible-role-dnd-pipeline diff --check

All native checks and immediate repeats pass. Review the final pipeline diff against roadmap scope, including no live-provider test or new production probe.

Completion and Commit

Commit as Verify suite configuration overrides and document migration. Record the final pipeline commit as the assembly revision. Inspect outgoing commits and publish that revision to the pipeline's existing origin/main with a normal push; verify remote reachability before Stage 4.

Stage 4: Assemble and Validate the Collection

Required Policy Review

Read AGENTS.md, docs/orientation.md, docs/conventions.md, docs/ansible.md, and docs/testing.md before starting.

Owning Repository and Preconditions

ansible-collection-dnd. The validated Stage 3 pipeline commit is public. Read the collection README's subtree and dependency workflow.

Required Changes

1. Import that exact revision using the authoritative public Gitea origin:

   git -C ansible-collection-dnd subtree pull --prefix=roles/pipeline \
     https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \
     PIPELINE_REVISION --squash
   

   Substitute the Stage 3 hash; never a local sibling path. Subtree-generated commits are expected. Do not edit imported role files directly.

2. Match the standalone revisions in root requirements.yml to Stage 1. Keep collection metadata version 0.1.0 and published release instructions intact; describe the new development snapshot separately.

3. Migrate collection tests/integration CLI/field assertions and direct Audita synthetic environment handling as in Stage 2. Validate Audita profiles and Scriptorium configuration natively. Keep collection tests focused on the installed artifact and dependency boundary, base/testing selection, and clean repeat; do not duplicate Stage 3's complete override matrix.

4. Update compatibility guidance and reference the pipeline migration notes. A fresh public dependency install must work without sibling role repos. Existing local campaign/prompt fixture overrides in the test harness remain acceptable; production manifests must not acquire those local paths.

Validation

bash -n ansible-collection-dnd/tests/integration
ansible-collection-dnd/tests/integration
git -C ansible-collection-dnd diff --check

Require successful build, disposable artifact install, syntax/lint, native configuration checks, and immediate clean repeat using public role dependencies.

Completion and Commit

Commit remaining changes as Assemble the upgraded D&D suite after the subtree import commits. Inspect and publish feature commits to origin/main; record and verify the final public collection hash for Stage 5. No tags or release assets are created or replaced.

Stage 5: Update the Consumer Without Deploying

Required Policy Review

Read AGENTS.md, docs/orientation.md, docs/conventions.md, docs/ansible.md, and docs/testing.md before starting.

Owning Repository and Preconditions

rakestraw-ansible. Stage 4's collection and all referenced dependencies are public. Read applicable consumer instructions and docs/pipeline-dnd.md.

Required Changes

1. Update only the relevant role pins in requirements.yml to Stage 1 and its D&D collection pin to Stage 4. Preserve unrelated requirements.
2. Keep playbooks/pipeline/dnd.yml minimal. Existing host-vars credential names remain valid; preserve encrypted Vault content and all unrelated values. Do not add copied suite defaults to inventory. If an obsolete feature-owned setting is present, replace only that setting and document it.
3. Update docs/pipeline-dnd.md to the coordinated versions, exact collection refresh hash, new testing invocation, and migration prerequisites. Preserve the explicit refresh guidance for an equal-version development collection. Remove misleading claims that the v3 upgrade needs no configuration or manifest migration. Distinguish future operator commands from checks run.

Validation

Use the workspace tools and disposable role/collection paths. From consumer root, set ANSIBLE_ROLES_PATH and ANSIBLE_COLLECTIONS_PATH to directories under a new mktemp -d path, then run:

/home/eric/Workspace/ansible-dnd/.venv/bin/ansible-galaxy install -r requirements.yml
/home/eric/Workspace/ansible-dnd/.venv/bin/ansible-playbook -i inventory playbooks/pipeline/dnd.yml --list-hosts --limit pipeline-dnd
/home/eric/Workspace/ansible-dnd/.venv/bin/ansible-playbook -i inventory playbooks/pipeline/dnd.yml --syntax-check --limit pipeline-dnd
git diff --check

These commands must not connect to the target or execute role tasks. If loading encrypted inventory requires unavailable Vault credentials, validate syntax with a temporary inventory containing only pipeline-dnd, a synthetic WhisperX URL, and synthetic required inputs instead. Inspect the real playbook target and inventory entry read-only; report this limited check accurately. Do not decrypt Vault, require real keys, add ansible-doc gates, or execute deployment.

Completion and Commit

Public dependencies resolve and consumer syntax/target checks pass (or the documented synthetic-inventory fallback is used). Commit as Prepare the consumer for the coordinated suite upgrade. Leave this consumer commit local unless separately asked to push; it is not a downstream dependency.

Stage 6: Record Evidence and Hand Off for Independent Review

Required Policy Review

Read AGENTS.md, docs/orientation.md, docs/conventions.md, docs/ansible.md, docs/testing.md, and docs/planning.md before starting.

Owning Repository and Preconditions

Workspace. Stages 1–5 are complete and committed. No new code or deployment work belongs in this stage.

Required Changes and Validation

Record final hashes, public availability, commands/results, environment limits, and any plan deviations in the feature roadmap and concise progress notes. Update docs/status.md and the upgrade sequence to implementation complete, review pending, using in_progress until independent review. Review that pins across pipeline, collection, and consumer agree, and no standalone role or application/asset code changed outside scope.

git diff --check
scripts/status

Reuse earlier integration evidence for documentation-only changes. Report which commits were pushed and which remain local; a clean working tree does not imply a synchronized remote. Keep deployment, session migration, and formal release work explicitly deferred.

Completion and Commit

Commit as Record coordinated suite upgrade validation. All changed working trees are clean, and implementation is ready for independent review. Do not mark F0022 done, begin another feature, or push unrelated workspace history.

Progress Notes

Plan approved. Stage 1 publication is blocked. On 2026-09-29, all four exact reviewed revisions were present locally; clean worktrees and expected Gitea origins were confirmed. After fetching, each origin/main was three reviewed feature commits behind the selected revision and could fast-forward. The first normal push, for Scriptorium, failed because HTTPS Git could not read a username. At that time no role revision was published and no downstream stage had started.

• Stage 1 resumed on 2026-09-29: authenticated pushes of the exact reviewed Scriptorium b174c8c, Audita c3dd59f, Notarius 6aef48b, and Narratio 3c0700b revisions to their existing Gitea origin/main branches succeeded. A fresh fetch and ancestor check verified all four public references. No standalone code or dependency pin changed in this stage.

Open Questions

None. The user approved application-owned retry/repair/deadline defaults with nullable inventory overrides and the retained Audita per-attempt policy.