ansible-workspace-dnd/docs/roadmap/implementation.md

Implementation Plan 0016: Deploy the D&D Pipeline to pipeline-dnd

Status: in_progress

Target executor: GPT-5.6-Terra

Feature roadmap: docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md

Active plan location: docs/roadmap/implementation.md

Objective

Deploy the reviewed collection through a minimal consumer playbook, host inventory, and Ansible Vault. Demonstrate clean repeated convergence and one operator-selected session through real publication. The feature roadmap owns scope, rationale, target state, and acceptance criteria.

Execution Rules

• Execute stages in order. Before each stage, reread its Required Policy Review and check the owning repository's working tree. Preserve unrelated changes. Validate and commit each completed stage with its stated intent.
• Keep the playbook limited to hosts and roles. Use existing interfaces; add no wrapper, preflight framework, custom module, or test harness.
• Inherit suite storage, publication, caching, cleanup, version, and asset defaults. Set only the roadmap-selected host choices.
• Never print decrypted inventory, credential values, or Vault passwords in commands, logs, documentation, or Git. Do not use decrypted inventory dumps as validation evidence. The operator supplies secrets directly through Ansible Vault's encrypted-file workflow, using an established secure editor. Do not solicit secrets in chat or create placeholder credentials.
• Stop dependent work when operational inputs are missing or validation fails. Record the concrete blocker; local checks do not prove a live deployment.
• Scope blockers to the operation that requires the missing input. Complete Stages 1–2 locally without remote SSH, production Vault values, or an acceptance session. Controller/SSH/Vault inputs gate Stage 3; session and audio selection gate Stage 4. Continue independent authorized preparation.
• A missing executable on PATH is not proof that tooling is uninstalled. Check the workspace .venv/bin and repository test drivers before reporting a missing dependency. Use the verified executable path in diagnostics.
• An Ansible controller is the environment running Ansible; it requires no controller daemon or /etc/ansible directory. Use explicit configuration, inventory, role, and collection paths for local checks. Do not recreate the operational controller layout or install system-wide tooling for that purpose.
• Retain the historical playbook and proof-of-concept role. Role, collection, application, and asset repositories remain read-only during this feature.
• Deployment and paid provider execution occur only during the approved sprint with the selected controller, target, credentials, and session. Do not substitute an arbitrary session or broaden the host limit.

Starting State and Preconditions

• Feature 0015 passed independent review without findings. Its pipeline commit is 51566afcf69083cdbb5b5718f2544c86b079001c; the reviewed public collection commit is 46913446765e86ecfb29ebad9955d579ef6a4c7e.
• rakestraw-ansible starts clean on main at d86ec29a9cbd821fead248736171b5cdd8440ad6. Inspect and preserve any subsequent changes before adapting this plan.
• Root requirements.yml already has an unpinned Gitea rakestraw.dnd entry alongside unrelated collections. The historical playbook is playbooks/apps/dnd.yml; it invokes pipeline_dnd on automation-dnd.
• Host pipeline-dnd already connects as eric to pipeline-dnd.rakestrawhome.com. Preserve its connection settings.
• inventory/host_vars/pipeline-dnd/narratio.yml contains the legacy campaign and S3 variables to migrate.
• The selected collection still declares version 0.1.0. An installed release with that number is not sufficient evidence of the selected content.
• This plan is approved, with this revision permitting local preparation before operational inputs are resolved. Do not invent those inputs; resolve them at the stage that needs them.
• Local tooling is already installed and verified: Ansible Core 2.21.3 in /home/eric/Workspace/ansible-dnd/.venv. Use its bin/ansible, bin/ansible-playbook, bin/ansible-galaxy, and bin/ansible-doc, or prepend that bin directory to PATH. No activation is required for absolute paths.

Repository Scope

• rakestraw-ansible: requirements, playbook, host configuration, encrypted credentials, and docs/pipeline-dnd.md runbook linked from README.
• Workspace: feature status, implementation progress, and review handoff.
• Other repositories: read-only compatibility and application references.

Stage 1: Establish Local Preparation and Record Deployment Inputs

Completion: complete (consumer 716a80ef68208537dacfa5f0f22259bf42ae6279)

Objective and Owning Repository

Record known and pending deployment inputs in rakestraw-ansible and close the reviewed Feature 0015 in the workspace.

Required Policy Review

Read AGENTS.md, docs/orientation.md, docs/conventions.md, docs/ansible.md, docs/testing.md, docs/planning.md, the Feature 0016 roadmap, and rakestraw-ansible/README.md before this stage.

Preconditions

The plan is approved and repositories have been checked. Local tooling is available; no remote access or credential/session input is required.

Required Changes

1. Create rakestraw-ansible/docs/pipeline-dnd.md and link it from README. Record the local validation setup and known operational details. List the operational controller/checkout, effective paths, Vault selection, and acceptance session as pending where unknown, with their consuming stage. Do not fabricate completed checks or turn missing values into blockers here.
2. Confirm local Ansible Core is at least 2.18 and the public suite repositories and Galaxy are reachable. Inspect effective configuration without dumping decrypted host variables. Existing repository defaults are /etc/ansible/inventory, /etc/ansible/roles, and /etc/ansible/collections; document these as operational defaults, not required local directories.
3. Record the reported target SSH rejection as a Stage 3 prerequisite. Defer target Ubuntu and sudo checks until authorized access is available. Repeated attempts with the same rejected identity do not advance local preparation.
4. Mark Feature 0015 done in its roadmap and workspace status, citing its reviewed commits and the successful independent standalone/collection integration runs. Leave historical remediation records unchanged. Mark Feature 0016 in_progress when execution begins.

Validation and Completion Criteria

Local tooling works and the runbook distinguishes known facts from pending operational inputs, assigning each to Stage 3 or Stage 4. Run git diff --check in both changed repositories. No deployment or provider request is necessary for this stage.

Commit

Consumer: Document the D&D deployment context. Workspace: Close storage review and begin host deployment.

Stage 2: Prepare the Consumer Playbook and Inventory

Objective and Owning Repository

Make rakestraw-ansible a complete consumer of the reviewed suite snapshot.

Required Policy Review

Read AGENTS.md, docs/orientation.md, docs/conventions.md, docs/ansible.md, docs/testing.md, docs/planning.md, the Feature 0016 roadmap, ansible-collection-dnd/README.md, its requirements.yml, and the Stage 1 runbook before this stage.

Preconditions

Stage 1 is complete. Production credentials and remote access are not required.

Required Changes

1. Add playbooks/pipeline/dnd.yml in the roadmap's exact minimal shape: one play targeting pipeline-dnd with only rakestraw.dnd.pipeline in its roles list. No play variables, tasks, or play-level privilege escalation.
2. Replace the legacy contents of inventory/host_vars/pipeline-dnd/narratio.yml with the roadmap's public inventory example: operator eric, the confirmed WhisperX URL, narratio_campaigns_default_campaign_id: dilfs, and the four Vault-backed dnd_pipeline_secrets mappings. Remove obsolete narratio_s3_* and narratio_default_campaign_id settings. Preserve unrelated intervening edits and the separate connection file. Do not copy suite defaults into inventory.
3. Document inventory/host_vars/pipeline-dnd/vault.yml and its four required vault_dnd_* names. Keep the inventory references unresolved until the operator provisions that encrypted file in Stage 3. Do not create an empty Vault file or substitute credentials to make local checks appear complete.
4. Pin the existing rakestraw.dnd Git entry in root requirements.yml to 46913446765e86ecfb29ebad9955d579ef6a4c7e with its existing public URL.
5. Copy the five roles entries from that collection's compatibility manifest, retaining role names, public Gitea URLs, scm: git, and v0.1.0 tags. Reconcile native collection entries to ansible.posix: 2.2.0 and community.general: 13.3.0, without duplicates. Preserve unrelated dependencies; report demonstrated conflicts rather than silently changing the reviewed suite pins.
6. Extend the runbook with the concrete commands for dependency installation, collection refresh, Vault selection, deployment, fresh login, and native validation used below. Link upstream Narratio v2.1.1 operator documentation.

Validation

Parse edited plaintext YAML and run git diff --check. In isolated controller dependency paths, install from public sources with ansible-galaxy install -r requirements.yml. Configure role and collection destinations through Ansible configuration/environment; avoid a role-only custom path that would skip the collection section of the mixed manifest. Confirm resolution with ansible-doc -t role rakestraw.dnd.pipeline and the five rakestraw.<app> role names.

For local checks, use the workspace executables and the checked-out inventory. Configure isolated role and collection paths explicitly. From the consumer checkout, run:

ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --list-hosts
ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --syntax-check

Host listing and syntax checking do not deploy or establish that credentials resolve. Run them without evaluating or printing secret values; do not supply fake Vault variables. If an existing unrelated encrypted inventory file needs an unavailable password, preserve it and use a disposable local inventory with only the target alias and its non-secret host variables for syntax checking. Report that narrower scope explicitly; full encrypted-inventory validation belongs to Stage 3. Do not change operational inventory to bypass encryption. Do not impose whole-repository lint changes on the established minimal-playbook style.

Completion Criteria

Only the intended host is selected. Public dependencies resolve without local sibling checkouts, local syntax checking succeeds within its recorded scope, and the diff contains no plaintext secrets.

Commit

Add the D&D pipeline deployment playbook.

Stage 3: Deploy and Verify Clean Convergence

Objective and Owning Repository

Deploy from the operational controller; record results in the consumer runbook.

Required Policy Review

Read AGENTS.md, docs/orientation.md, docs/conventions.md, docs/ansible.md, docs/testing.md, docs/planning.md, the Feature 0016 roadmap, and rakestraw-ansible/docs/pipeline-dnd.md before this stage.

Preconditions

Stage 2 passed. The operational controller/checkout and delivery workflow are identified, and target SSH access and the Vault mechanism are available. Deliver committed changes through that workflow without overwriting a dirty checkout. Acceptance-session selection is not a prerequisite for this stage.

Required Changes and Validation

Before installation, confirm the operational controller's Ansible version and effective inventory, role, and collection paths. Confirm Ubuntu 26.04 and the existing eric account's required sudo access through read-only target checks. Have the operator create inventory/host_vars/pipeline-dnd/vault.yml encrypted in full, defining the four roadmap credential variables. Commit ciphertext with intent Add encrypted D&D host credentials after checking the diff exposes no plaintext. Rerun host listing and syntax validation with the real encrypted inventory and recorded Vault options before convergence.

1. Install root requirements at the confirmed controller paths. Explicitly refresh only the D&D collection from its selected commit, even when 0.1.0 is already installed:

   ansible-galaxy collection install git+https://gitea.maximumdirect.net/eric/ansible-collection-dnd.git,46913446765e86ecfb29ebad9955d579ef6a4c7e --force --no-deps
   

   Use the configured collection path; dependencies come from the manifest. Do not force-update unrelated unpinned collections. Confirm role resolution and playbook syntax on this operational controller as in Stage 2.

2. Run ansible-playbook playbooks/pipeline/dnd.yml --limit pipeline-dnd using the recorded inventory, Vault, and any required sudo-password options. Immediately repeat the identical command after success. Require changed=0, failed=0, and no unreachable hosts on the repeat.

3. On the target, execute sudo -u narratio /usr/local/bin/narratio config validate, then validate with --profile testing. The managed default campaign should make an explicit campaign selector unnecessary.

4. Start a fresh login as eric and execute the same default native validation without switching user, demonstrating usable supplementary group access.

5. Record consumer commit, target, date, repeat recap, and native validation results. Diagnose failures at the owning boundary; report required changes to roles/applications separately rather than expanding this feature.

Completion Criteria

The actual host converges cleanly and both runtime identities validate the configuration. Run git diff --check on the evidence/runbook changes.

Commit

Record successful D&D host deployment.

Stage 4: Exercise One Real Session

Objective and Owning Repository

Demonstrate production processing and publication; record acceptance in rakestraw-ansible/docs/pipeline-dnd.md.

Required Policy Review

Read AGENTS.md, docs/orientation.md, docs/conventions.md, docs/testing.md, docs/planning.md, the Feature 0016 roadmap, the runbook, and Narratio v2.1.1's docs/config.md, docs/cli.md, and docs/operations.md before this stage. Follow the sibling application's AGENTS.md and development reading guide when consulting its repository.

Preconditions

Stage 3 passed. Actual credentials, the operator-selected campaign/session, session file, and audio are accessible. The exercise must demonstrate real transcription and publication, not merely resume an already complete run. Do not erase existing state to create a fixture.

Required Changes and Validation

1. As eric in a fresh authorized login, run:

   narratio session validate SESSION_ID --campaign CAMPAIGN --session SESSION_FILE
   narratio run SESSION_ID --campaign CAMPAIGN --session SESSION_FILE
   narratio session artifacts SESSION_ID --remote --campaign CAMPAIGN --session SESSION_FILE
   

   Substitute the operator-recorded values and use the production profile. The session document remains operational input; no session-management task belongs in the playbook. Observe transcription, processing, and successful publication. Do not force reprocessing or reduce the selected artifact set merely to bypass a failure.

2. Observe a representative published transcript or recap at the configured destination through the existing storage/operator interface. Successful cleanup may remove local workspace content; local survival is not required.

3. Record identity, input location, runtime user, completion outcome, and non-secret publication evidence. Avoid committing raw provider logs or credential-bearing URLs. Do not repeat paid work just to duplicate evidence or expand acceptance into a restore/locking test matrix.

Completion Criteria

A real session completes and its published output is observable. Record any provider/application blocker and leave acceptance incomplete if it cannot finish. Run git diff --check on the runbook changes.

Commit

Record the first published D&D session.

Stage 5: Prepare the Independent Review Handoff

Objective and Owning Repository

Record completed implementation evidence in the workspace.

Required Policy Review

Read AGENTS.md, docs/orientation.md, docs/conventions.md, docs/testing.md, docs/planning.md, and the Feature 0016 roadmap before this stage.

Preconditions

Stages 1–4 completed with committed changes and actual deployment/session evidence. A blocked live exercise does not satisfy this stage.

Required Changes and Validation

Update docs/status.md and this plan's progress notes with owning commits, runbook link, validation outcomes, and deviations. Keep Feature 0016 in_progress pending independent review. Report any additional complexity and its authorizing requirement. Run git diff --check and confirm changed repositories are clean after committing. Do not rerun role integration matrices or paid sessions without a new change or failure requiring it.

Completion Criteria

Evidence is recorded and implementation is ready for independent review.

Commit

Record the D&D deployment implementation.

Final Validation and Handoff

Report dependency revision, consumer commits, controller/target, public dependency and syntax results, immediate clean convergence, native validation for both identities, and session publication evidence. Distinguish completed work from operator-dependent blockers. Do not mark Feature 0016 done before independent review.

Progress Notes

• Stage 1 is complete in consumer commit 716a80ef68208537dacfa5f0f22259bf42ae6279. The linked deployment runbook records the Ansible Core 2.21.3 local toolchain, the effective repository defaults, successful public-source reachability, and the inputs pending for live deployment and session acceptance.
• Feature 0015 is closed after independent review found no remediation issue. Its reviewed pipeline commit is 51566afcf69083cdbb5b5718f2544c86b079001c; collection fixture, squash, and merge commits are cf3de016ee09edf91afc3bc1addfd926d4a172ad, 77f67adbc0f700e88bc15554e02052eaa22f1033, and 46913446765e86ecfb29ebad9955d579ef6a4c7e.
• Stage 2 local consumer preparation is next. The rejected target SSH identity, operational controller details, and Vault workflow/values remain Stage 3 prerequisites. The acceptance session/audio selection remains a Stage 4 prerequisite. No live deployment has begun.

Open Questions

Questions 1–2 gate Stage 3 only. Question 3 gates Stage 4 only. None prevents local preparation in Stages 1–2.

1. Which controller/checkout executes deployment and how do committed changes reach it? Recommended: use the existing operational controller and its established /etc/ansible/ layout, keeping one operational inventory. Alternative: a dedicated checkout with explicit inventory and dependency paths, if the operator prefers isolation and records those paths consistently.
2. Which Vault identity/password mechanism is available, and can the operator populate all four values? Recommended: reuse the controller's established Vault workflow and populate the encrypted file directly. Alternative: an interactive Vault password prompt for this file if no identity is established. Both keep passwords outside Git; neither requires secrets in chat.
3. Which campaign/session/audio supplies acceptance? Recommended: an operator-selected unprocessed dilfs session with accessible audio, matching the retained default campaign and avoiding forced reprocessing. Alternative: a selected session from another deployed campaign with an explicit selector. Record the session file and audio location before execution; do not invent sample production input.