ansible-workspace-dnd/docs/roadmap/implementation.md

# Implementation Plan 0016: Deploy the D&D Pipeline to `pipeline-dnd`

Status: in_progress

Target executor: GPT-5.6-Terra

Feature roadmap: `docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md`

Active plan location: `docs/roadmap/implementation.md`

## Objective

Deploy the reviewed collection through a minimal consumer playbook, host
inventory, and Ansible Vault. Demonstrate clean repeated convergence and one
operator-selected session through real publication. The feature roadmap owns
scope, rationale, target state, and acceptance criteria.

## Execution Rules

- Execute stages in order. Before each stage, reread its Required Policy
  Review and check the owning repository's working tree. Preserve unrelated
  changes. Validate and commit each completed stage with its stated intent.
- Keep the playbook limited to hosts and roles. Use existing interfaces;
  add no wrapper, preflight framework, custom module, or test harness.
- Inherit suite storage, publication, caching, cleanup, version, and asset
  defaults. Set only the roadmap-selected host choices.
- Never print decrypted inventory, credential values, or Vault passwords in
  commands, logs, documentation, or Git. Do not use decrypted inventory dumps
  as validation evidence. The operator supplies secrets directly through
  Ansible Vault's encrypted-file workflow, using an established secure editor.
  Do not solicit secrets in chat or create placeholder credentials.
- Stop dependent work when operational inputs are missing or validation fails.
  Record the concrete blocker; local checks do not prove a live deployment.
- Scope blockers to the operation that requires the missing input. Complete
  Stages 1–2 locally without remote SSH, production Vault values, or an
  acceptance session. Controller/SSH/Vault inputs gate Stage 3; session and
  audio selection gate Stage 4. Continue independent authorized preparation.
- A missing executable on `PATH` is not proof that tooling is uninstalled.
  Check the workspace `.venv/bin` and repository test drivers before reporting
  a missing dependency. Use the verified executable path in diagnostics.
- An Ansible controller is the environment running Ansible; it requires no
  controller daemon or `/etc/ansible` directory. Use explicit configuration,
  inventory, role, and collection paths for local checks. Do not recreate the
  operational controller layout or install system-wide tooling for that purpose.
- Retain the historical playbook and proof-of-concept role. Role, collection,
  application, and asset repositories remain read-only during this feature.
- Deployment and paid provider execution occur only during the approved sprint
  with the selected controller, target, credentials, and session. Do not
  substitute an arbitrary session or broaden the host limit.

## Starting State and Preconditions

- Feature 0015 passed independent review without findings. Its pipeline commit
  is `51566afcf69083cdbb5b5718f2544c86b079001c`; the reviewed public collection
  commit is `46913446765e86ecfb29ebad9955d579ef6a4c7e`.
- `rakestraw-ansible` starts clean on `main` at
  `d86ec29a9cbd821fead248736171b5cdd8440ad6`. Inspect and preserve any subsequent
  changes before adapting this plan.
- Root `requirements.yml` already has an unpinned Gitea `rakestraw.dnd` entry
  alongside unrelated collections. The historical playbook is
  `playbooks/apps/dnd.yml`; it invokes `pipeline_dnd` on `automation-dnd`.
- Host `pipeline-dnd` already connects as `eric` to
  `pipeline-dnd.rakestrawhome.com`. Preserve its connection settings.
- `inventory/host_vars/pipeline-dnd/narratio.yml` contains the legacy campaign
  and S3 variables to migrate.
- The selected collection still declares version `0.1.0`. An installed
  release with that number is not sufficient evidence of the selected content.
- This plan is approved, with this revision permitting local preparation before
  operational inputs are resolved. Do not invent those inputs; resolve them
  at the stage that needs them.
- Local tooling is already installed and verified: Ansible Core 2.21.3 in
  `/home/eric/Workspace/ansible-dnd/.venv`. Use its `bin/ansible`,
  `bin/ansible-playbook`, `bin/ansible-galaxy`, and `bin/ansible-doc`, or prepend
  that `bin` directory to `PATH`. No activation is required for absolute paths.

## Repository Scope

- `rakestraw-ansible`: requirements, playbook, host configuration, encrypted
  credentials, and `docs/pipeline-dnd.md` runbook linked from README.
- Workspace: feature status, implementation progress, and review handoff.
- Other repositories: read-only compatibility and application references.

## Stage 1: Establish Local Preparation and Record Deployment Inputs

Completion: complete (consumer `716a80ef68208537dacfa5f0f22259bf42ae6279`)

### Objective and Owning Repository

Record known and pending deployment inputs in `rakestraw-ansible` and close the reviewed
Feature 0015 in the workspace.

### Required Policy Review

Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, `docs/planning.md`, the Feature 0016
roadmap, and `rakestraw-ansible/README.md` before this stage.

### Preconditions

The plan is approved and repositories have been checked. Local tooling is
available; no remote access or credential/session input is required.

### Required Changes

1. Create `rakestraw-ansible/docs/pipeline-dnd.md` and link it from README.
   Record the local validation setup and known operational details. List the
   operational controller/checkout, effective paths, Vault selection, and
   acceptance session as pending where unknown, with their consuming stage.
   Do not fabricate completed checks or turn missing values into blockers here.
2. Confirm local Ansible Core is at least 2.18 and the public suite
   repositories and Galaxy are reachable. Inspect effective configuration
   without dumping decrypted host variables. Existing repository defaults are
   `/etc/ansible/inventory`, `/etc/ansible/roles`, and `/etc/ansible/collections`;
   document these as operational defaults, not required local directories.
3. Record the reported target SSH rejection as a Stage 3 prerequisite. Defer
   target Ubuntu and sudo checks until authorized access is available. Repeated
   attempts with the same rejected identity do not advance local preparation.
4. Mark Feature 0015 `done` in its roadmap and workspace status, citing its
   reviewed commits and the successful independent standalone/collection
   integration runs. Leave historical remediation records unchanged. Mark
   Feature 0016 `in_progress` when execution begins.

### Validation and Completion Criteria

Local tooling works and the runbook distinguishes known facts from pending
operational inputs, assigning each to Stage 3 or Stage 4.
Run `git diff --check` in both changed repositories. No deployment or provider
request is necessary for this stage.

### Commit

Consumer: `Document the D&D deployment context`.
Workspace: `Close storage review and begin host deployment`.

## Stage 2: Prepare the Consumer Playbook and Inventory

### Objective and Owning Repository

Make `rakestraw-ansible` a complete consumer of the reviewed suite snapshot.

### Required Policy Review

Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, `docs/planning.md`, the Feature 0016
roadmap, `ansible-collection-dnd/README.md`, its `requirements.yml`, and the
Stage 1 runbook before this stage.

### Preconditions

Stage 1 is complete. Production credentials and remote access are not required.

### Required Changes

1. Add `playbooks/pipeline/dnd.yml` in the roadmap's exact minimal shape:
   one play targeting `pipeline-dnd` with only `rakestraw.dnd.pipeline` in its
   roles list. No play variables, tasks, or play-level privilege escalation.
2. Replace the legacy contents of
   `inventory/host_vars/pipeline-dnd/narratio.yml` with the roadmap's public
   inventory example: operator `eric`, the confirmed WhisperX URL,
   `narratio_campaigns_default_campaign_id: dilfs`, and the four Vault-backed
   `dnd_pipeline_secrets` mappings. Remove obsolete `narratio_s3_*` and
   `narratio_default_campaign_id` settings. Preserve unrelated intervening edits
   and the separate connection file. Do not copy suite defaults into inventory.
3. Document `inventory/host_vars/pipeline-dnd/vault.yml` and its four required
   `vault_dnd_*` names. Keep the inventory references unresolved until the
   operator provisions that encrypted file in Stage 3. Do not create an empty
   Vault file or substitute credentials to make local checks appear complete.
4. Pin the existing `rakestraw.dnd` Git entry in root `requirements.yml` to
   `46913446765e86ecfb29ebad9955d579ef6a4c7e` with its existing public URL.
5. Copy the five `roles` entries from that collection's compatibility manifest,
   retaining role names, public Gitea URLs, `scm: git`, and `v0.1.0` tags.
   Reconcile native collection entries to `ansible.posix: 2.2.0` and
   `community.general: 13.3.0`, without duplicates. Preserve unrelated
   dependencies; report demonstrated conflicts rather than silently changing
   the reviewed suite pins.
6. Extend the runbook with the concrete commands for dependency installation,
   collection refresh, Vault selection, deployment, fresh login, and native
   validation used below. Link upstream Narratio v2.1.1 operator documentation.

### Validation

Parse edited plaintext YAML and run `git diff --check`. In isolated controller
dependency paths, install from public sources with
`ansible-galaxy install -r requirements.yml`. Configure role and collection
destinations through Ansible configuration/environment; avoid a role-only
custom path that would skip the collection section of the mixed manifest.
Confirm resolution with `ansible-doc -t role rakestraw.dnd.pipeline` and the
five `rakestraw.<app>` role names.

For local checks, use the workspace executables and the checked-out inventory.
Configure isolated role and collection paths explicitly. From the consumer
checkout, run:

```text
ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --list-hosts
ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --syntax-check
```

Host listing and syntax checking do not deploy or establish that credentials
resolve. Run them without evaluating or printing secret values; do not supply
fake Vault variables. If an existing unrelated encrypted inventory file needs
an unavailable password, preserve it and use a disposable local inventory with
only the target alias and its non-secret host variables for syntax checking.
Report that narrower scope explicitly; full encrypted-inventory validation
belongs to Stage 3. Do not change operational inventory to bypass encryption.
Do not impose
whole-repository lint changes on the established minimal-playbook style.

### Completion Criteria

Only the intended host is selected. Public dependencies resolve without local
sibling checkouts, local syntax checking succeeds within its recorded scope, and the
diff contains no plaintext secrets.

### Commit

`Add the D&D pipeline deployment playbook`.

## Stage 3: Deploy and Verify Clean Convergence

### Objective and Owning Repository

Deploy from the operational controller; record results in the consumer runbook.

### Required Policy Review

Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, `docs/planning.md`, the Feature 0016
roadmap, and `rakestraw-ansible/docs/pipeline-dnd.md` before this stage.

### Preconditions

Stage 2 passed. The operational controller/checkout and delivery workflow are
identified, and target SSH access and the Vault mechanism are available.
Deliver committed changes through that workflow without overwriting a dirty
checkout. Acceptance-session selection is not a prerequisite for this stage.

### Required Changes and Validation

Before installation, confirm the operational controller's Ansible version and
effective inventory, role, and collection paths. Confirm Ubuntu 26.04 and the
existing `eric` account's required sudo access through read-only target checks.
Have the operator create `inventory/host_vars/pipeline-dnd/vault.yml` encrypted
in full, defining the four roadmap credential variables. Commit ciphertext
with intent `Add encrypted D&D host credentials` after checking the diff
exposes no plaintext. Rerun host listing and syntax validation with the real
encrypted inventory and recorded Vault options before convergence.

1. Install root requirements at the confirmed controller paths. Explicitly
   refresh only the D&D collection from its selected commit, even when `0.1.0`
   is already installed:

   ```text
   ansible-galaxy collection install git+https://gitea.maximumdirect.net/eric/ansible-collection-dnd.git,46913446765e86ecfb29ebad9955d579ef6a4c7e --force --no-deps
   ```

   Use the configured collection path; dependencies come from the manifest.
   Do not force-update unrelated unpinned collections. Confirm role resolution
   and playbook syntax on this operational controller as in Stage 2.
2. Run `ansible-playbook playbooks/pipeline/dnd.yml --limit pipeline-dnd`
   using the recorded inventory, Vault, and any required sudo-password options.
   Immediately repeat the identical command after success. Require
   `changed=0`, `failed=0`, and no unreachable hosts on the repeat.
3. On the target, execute
   `sudo -u narratio /usr/local/bin/narratio config validate`, then validate
   with `--profile testing`. The managed default campaign should make an
   explicit campaign selector unnecessary.
4. Start a fresh login as `eric` and execute the same default native validation
   without switching user, demonstrating usable supplementary group access.
5. Record consumer commit, target, date, repeat recap, and native validation
   results. Diagnose failures at the owning boundary; report required changes
   to roles/applications separately rather than expanding this feature.

### Completion Criteria

The actual host converges cleanly and both runtime identities validate the
configuration. Run `git diff --check` on the evidence/runbook changes.

### Commit

`Record successful D&D host deployment`.

## Stage 4: Exercise One Real Session

### Objective and Owning Repository

Demonstrate production processing and publication; record acceptance in
`rakestraw-ansible/docs/pipeline-dnd.md`.

### Required Policy Review

Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/testing.md`, `docs/planning.md`, the Feature 0016 roadmap, the runbook,
and Narratio v2.1.1's `docs/config.md`, `docs/cli.md`, and
`docs/operations.md` before this stage. Follow the sibling application's
`AGENTS.md` and development reading guide when consulting its repository.

### Preconditions

Stage 3 passed. Actual credentials, the operator-selected campaign/session,
session file, and audio are accessible. The exercise must demonstrate real
transcription and publication, not merely resume an already complete run.
Do not erase existing state to create a fixture.

### Required Changes and Validation

1. As `eric` in a fresh authorized login, run:

   ```text
   narratio session validate SESSION_ID --campaign CAMPAIGN --session SESSION_FILE
   narratio run SESSION_ID --campaign CAMPAIGN --session SESSION_FILE
   narratio session artifacts SESSION_ID --remote --campaign CAMPAIGN --session SESSION_FILE
   ```

   Substitute the operator-recorded values and use the production profile.
   The session document remains operational input; no session-management task
   belongs in the playbook. Observe transcription, processing, and successful
   publication. Do not force reprocessing or reduce the selected artifact set
   merely to bypass a failure.
2. Observe a representative published transcript or recap at the configured
   destination through the existing storage/operator interface. Successful
   cleanup may remove local workspace content; local survival is not required.
3. Record identity, input location, runtime user, completion outcome, and
   non-secret publication evidence. Avoid committing raw provider logs or
   credential-bearing URLs. Do not repeat paid work just to duplicate evidence
   or expand acceptance into a restore/locking test matrix.

### Completion Criteria

A real session completes and its published output is observable. Record any
provider/application blocker and leave acceptance incomplete if it cannot
finish. Run `git diff --check` on the runbook changes.

### Commit

`Record the first published D&D session`.

## Stage 5: Prepare the Independent Review Handoff

### Objective and Owning Repository

Record completed implementation evidence in the workspace.

### Required Policy Review

Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/testing.md`, `docs/planning.md`, and the Feature 0016 roadmap before
this stage.

### Preconditions

Stages 1–4 completed with committed changes and actual deployment/session
evidence. A blocked live exercise does not satisfy this stage.

### Required Changes and Validation

Update `docs/status.md` and this plan's progress notes with owning commits,
runbook link, validation outcomes, and deviations. Keep Feature 0016
`in_progress` pending independent review. Report any additional complexity
and its authorizing requirement. Run `git diff --check` and confirm changed
repositories are clean after committing. Do not rerun role integration
matrices or paid sessions without a new change or failure requiring it.

### Completion Criteria

Evidence is recorded and implementation is ready for independent review.

### Commit

`Record the D&D deployment implementation`.

## Final Validation and Handoff

Report dependency revision, consumer commits, controller/target, public
dependency and syntax results, immediate clean convergence, native validation
for both identities, and session publication evidence. Distinguish completed
work from operator-dependent blockers. Do not mark Feature 0016 `done` before
independent review.

## Progress Notes

- Stage 1 is complete in consumer commit
  `716a80ef68208537dacfa5f0f22259bf42ae6279`. The linked deployment runbook
  records the Ansible Core 2.21.3 local toolchain, the effective repository
  defaults, successful public-source reachability, and the inputs pending for
  live deployment and session acceptance.
- Feature 0015 is closed after independent review found no remediation issue.
  Its reviewed pipeline commit is
  `51566afcf69083cdbb5b5718f2544c86b079001c`; collection fixture, squash, and
  merge commits are `cf3de016ee09edf91afc3bc1addfd926d4a172ad`,
  `77f67adbc0f700e88bc15554e02052eaa22f1033`, and
  `46913446765e86ecfb29ebad9955d579ef6a4c7e`.
- Stage 2 local consumer preparation is next. The rejected target SSH identity,
  operational controller details, and Vault workflow/values remain Stage 3
  prerequisites. The acceptance session/audio selection remains a Stage 4
  prerequisite. No live deployment has begun.

## Open Questions

Questions 1–2 gate Stage 3 only. Question 3 gates Stage 4 only. None prevents
local preparation in Stages 1–2.

1. Which controller/checkout executes deployment and how do committed changes
   reach it? Recommended: use the existing operational controller and its
   established `/etc/ansible/` layout, keeping one operational inventory.
   Alternative: a dedicated checkout with explicit inventory and dependency
   paths, if the operator prefers isolation and records those paths consistently.
2. Which Vault identity/password mechanism is available, and can the operator
   populate all four values? Recommended: reuse the controller's established
   Vault workflow and populate the encrypted file directly. Alternative: an
   interactive Vault password prompt for this file if no identity is established.
   Both keep passwords outside Git; neither requires secrets in chat.
3. Which campaign/session/audio supplies acceptance? Recommended: an
   operator-selected unprocessed `dilfs` session with accessible audio, matching
   the retained default campaign and avoiding forced reprocessing. Alternative:
   a selected session from another deployed campaign with an explicit selector.
   Record the session file and audio location before execution; do not invent
   sample production input.