ansible-workspace-dnd/docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md

# Feature 0016: Deploy the D&D Pipeline to `pipeline-dnd`

Status: in_progress

## Summary

Add a minimal playbook and host inventory to `rakestraw-ansible` that apply
`rakestraw.dnd.pipeline` to the existing `pipeline-dnd` host. Configure the
host through `host_vars`, enroll `eric` as the initial human operator, keep
credential values in Ansible Vault, and prove the complete pipeline through a
real repeatable deployment and representative D&D session.

[Feature 0015](0015-dnd-s3-storage-and-publishing.md) has implemented and passed
independent review of the storage and publishing behavior needed for an
operational end-to-end pipeline. Feature 0016 consumes that reviewed interface
and its suite defaults.

## Problem or Opportunity

The collection and its application roles are implemented and installable, but
the operational Ansible repository does not yet have a playbook for deploying
the new collection architecture. Its existing D&D playbook targets a retired
inventory name and invokes the proof-of-concept local `pipeline_dnd` role.

The current `pipeline-dnd` host inventory already records the connection
identity and useful legacy Narratio storage values. A production-shaped first
deployment now needs to express the new role's public interface, deliver
secrets safely, reconcile the controller's dependency manifest, and validate
the actual host rather than extending the role repositories with another
synthetic scenario.

## Scope

- Add `rakestraw-ansible/playbooks/pipeline/dnd.yml`, targeting the existing
  `pipeline-dnd` inventory host and applying only
  `rakestraw.dnd.pipeline`.
- Keep the playbook limited to its host selection and role list. Put all
  deployment choices in `inventory/host_vars/pipeline-dnd/`.
- Configure the external WhisperX transcription endpoint as
  `https://transcription.ai.rakestrawhome.com/transcribe`.
- Enroll the existing `eric` account through
  `dnd_pipeline_operator_users`.
- Map the role's `dnd_pipeline_secrets` interface to host-specific Vault
  variables. Store credential values only in an Ansible Vault-encrypted host
  variables file.
- Inherit Feature 0015's storage and publishing defaults, which already match
  the intended Backblaze B2 deployment. Host inventory contains only actual
  host choices and deliberate overrides, without copying suite defaults.
- Preserve the existing default campaign choice, `dilfs`, through the
  standalone role's `narratio_campaigns_default_campaign_id` variable.
- Reconcile existing legacy Narratio host variables so they cannot silently
  override the suite policy selected by the new pipeline role.
- Update `rakestraw-ansible/requirements.yml` so a clean controller can install
  the reviewed collection snapshot, all five standalone role dependencies,
  and required native collections from one manifest.
- Validate controller dependency resolution, playbook syntax, initial
  convergence, immediate clean convergence, operator access, native Narratio
  configuration, and one representative end-to-end pipeline run on the real
  host.
- Document the operational invocation, Vault prerequisite, initial operator,
  and expected validation path in `rakestraw-ansible`.

## Non-goals

- Implementing storage or publishing behavior. Feature 0015 owns that
  prerequisite.
- Changing application or role behavior merely to accommodate this
  deployment.
- Replacing the existing inventory alias with `pipeline.dnd`, changing DNS,
  or moving connection variables out of the current host inventory.
- Adding a wrapper role, pre-task layer, service, timer, scheduler, web UI, or
  unattended session-ingestion mechanism.
- Creating or managing the `eric` account, its login policy, SSH access, or
  sudo policy.
- Adding another production host, high availability, backup policy, or
  disaster-recovery automation.
- Removing the proof-of-concept local `pipeline_dnd` role or its historical
  playbook before the new deployment has demonstrated operational parity.
- Publishing a new coordinated suite release solely as part of this
  deployment feature.

## Target End State

`rakestraw-ansible/playbooks/pipeline/dnd.yml` is an intentionally small
consumer playbook equivalent to:

```yaml
---
- hosts:
    - pipeline-dnd
  roles:
    - rakestraw.dnd.pipeline
...
```

The existing inventory selects the host's connection identity. A dedicated
host-variable file selects the WhisperX endpoint, the initial operator list,
and the default campaign. Storage, caching, publication, and cleanup inherit
the reviewed suite defaults. The file maps runtime credential names to
Vault-backed variables without embedding credential values in the playbook or
ordinary plaintext inventory, for example:

```yaml
dnd_pipeline_operator_users:
  - eric

dnd_pipeline_whisperx_transcribe_url: >-
  https://transcription.ai.rakestrawhome.com/transcribe

narratio_campaigns_default_campaign_id: dilfs

dnd_pipeline_secrets:
  AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
  OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"
  OBJECT_STORAGE_KEY_ID: "{{ vault_dnd_object_storage_key_id }}"
  OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}"
```

The corresponding credential variables reside in a tracked file encrypted in
full with Ansible Vault. The Vault password or identity mechanism remains
outside Git and outside this feature's managed content.

The controller dependency manifest selects public collection commit
`46913446765e86ecfb29ebad9955d579ef6a4c7e`, containing the reviewed Features
0014 and 0015 and pipeline commit
`51566afcf69083cdbb5b5718f2544c86b079001c`. Its compatibility manifest selects
the five standalone roles at `v0.1.0`, `ansible.posix` at `2.2.0`, and
`community.general` at `13.3.0`. A clean controller therefore does not depend
on sibling development checkouts or previously installed roles.

The published collection `v0.1.0` predates these features. The selected
development snapshot still declares collection version `0.1.0`, so the
deployment instructions explicitly refresh an already-installed collection
from the pinned commit instead of allowing an equal-version installation to
be skipped. Unrelated controller dependencies remain preserved.

On `pipeline-dnd`, the role converges the full application suite and its D&D
configuration. The `narratio` service identity can run the pipeline, and
`eric` can run it after starting a login session that observes the granted
supplementary group membership. A representative session can be transcribed,
processed, stored, and published through the configured production path. An
immediate identical Ansible run reports no changes.

## Repository Ownership

- `rakestraw-ansible` owns the consumer playbook, host inventory, encrypted
  host credentials, controller dependency manifest, and deployment runbook.
- `ansible-role-dnd-pipeline` owns the suite's public variables and the
  opinionated cross-application behavior consumed by the deployment.
- `ansible-collection-dnd` owns the assembled `rakestraw.dnd.pipeline` role
  snapshot and consumer-facing compatibility manifest.
- The standalone application roles continue to own installation and native
  application configuration. Feature 0016 consumes their reviewed releases or
  commits without reimplementing their behavior.
- `ansible-workspace-dnd` owns this roadmap, the later implementation plan,
  progress tracking, and independent review record.
- Application and asset repositories are read-only sources of runtime
  contracts and deployed content for this feature.

## Constraints and Settled Decisions

- Feature 0015's implementation and independent review satisfy the storage
  and publishing prerequisite. Its workspace completion record should be
  closed out before the Feature 0016 sprint.
- The target is the inventory alias `pipeline-dnd` as currently defined in
  `rakestraw-ansible`, not a new `pipeline.dnd` alias.
- The playbook follows the repository's established style: hosts and roles
  only. Facts remain at Ansible's normal default, privilege escalation remains
  task-owned by the roles, and deployment variables belong in host inventory.
- `eric` is the initial human operator and must already exist on the managed
  host. The pipeline role grants only additive runtime-group membership.
- API credentials use the existing typed `dnd_pipeline_secrets` interface and
  Ansible Vault. Plaintext secret values must not enter Git, command lines,
  logs, examples, or ordinary variable files.
- Public Gitea repositories supply the suite's Git dependencies; native
  collections come from Ansible Galaxy. The deployment must not reference
  this workspace's sibling checkout paths.
- The collection is pinned to the reviewed public commit identified above.
  Feature 0015 did not create a release; this deployment does not track an
  unbounded branch or require a new coordinated release.
- The collection artifact does not itself install arbitrary Git-hosted role
  dependencies. The operational manifest must therefore include the five
  standalone roles as well as the collection and its native collection
  dependencies.
- Remove the legacy `narratio_s3_*` host variables because their intended
  values are supplied by the suite defaults. Replace
  `narratio_default_campaign_id` with
  `narratio_campaigns_default_campaign_id: dilfs`. The standalone role already
  exposes this host-specific setting; no new pipeline variable is required.
- The initial real-host run is an operational acceptance exercise. It should
  test observable pipeline behavior and convergence, not module task traces or
  the current text of defaults.

## Operational Prerequisites

Local repository preparation and syntax/dependency checks can proceed with the
workspace's existing Ansible virtual environment. Before live deployment,
establish the controller, target access, and Vault details below. Select the
session/audio before the end-to-end exercise, not before local implementation.
Record non-secret choices in the implementation plan or deployment runbook:

- Identify the operational Ansible controller and its deployment checkout.
  The repository configuration currently selects `/etc/ansible/inventory`,
  `/etc/ansible/roles`, and `/etc/ansible/collections`. Confirm the effective
  paths so dependency installation and playbook execution use the same
  inventory and installed content. Distinguish those paths from this
  development workspace.
- The target must satisfy the roles' Ubuntu 26.04 and access requirements.
  Normal connection, role guards, and privileged tasks can establish these;
  separate platform or sudo probes are not required acceptance tests.
- Establish the controller's existing Vault password or identity mechanism
  and operator availability to populate the four credential values in the
  encrypted host file. Record how Vault is selected, never its password or
  decrypted contents.
- Have the operator select the acceptance campaign, session identity, and
  audio input, with the corresponding session file and accessible input
  location. This is a real processing and publication exercise using the
  selected providers; session selection remains an operator decision.

These are deployment inputs and operational checks, not additional production
tasks or a new automated preflight framework.

## Simplest Viable Implementation

Use the reviewed collection revision and associated compatibility manifest to
update the controller's single root requirements file. Add the minimal playbook and one ordinary
host-variable file for the endpoint, operator, campaign, and credential
references, inheriting the suite's storage and publication defaults. Add one
fully encrypted host Vault file for the four credential values, populated by
the operator without exposing its plaintext during implementation or review.

Install or update dependencies from the root manifest at the confirmed
controller paths, explicitly refreshing the collection from its pinned commit
even when `0.1.0` is already installed. Confirm that Ansible
resolves `rakestraw.dnd.pipeline`, syntax-check the playbook, and converge only
`pipeline-dnd`. Start a fresh operator login after enrollment, validate the
native Narratio configuration under the intended runtime identity, rerun the
same play immediately to confirm clean convergence, and process one selected
session through the complete Feature 0015 publishing path. Record concise
operational instructions and results; do not add a second test harness around
the real deployment.

## Rationale and Alternatives

Keeping the playbook nearly declarative makes inventory the durable record of
what differs about this host and allows ordinary Ansible precedence to remain
the customization mechanism. Embedding endpoint, operator, storage, or secret
mapping variables in the playbook would make a one-host play harder to reuse
and would diverge from the established `rakestraw-ansible` style.

A local wrapper role could hide the collection interface, but it would create
another layer with no independent behavior. Applying the collection role by
FQCN proves the same consumer path that future hosts should use.

Tracking collection `main` would be convenient during development, but it
would make a later controller bootstrap select an unreviewed compatibility
set. An exact public commit retains development flexibility until the next
release while still making this deployment reproducible.

Feature 0015 now supplies the reviewed storage and publishing path needed for
meaningful end-to-end acceptance. Inheriting that policy keeps host inventory
focused on deployment choices and avoids maintaining a second copy of the
suite defaults.

## Acceptance Criteria

- Feature 0015 is complete and independently reviewed before Feature 0016
  implementation begins.
- `playbooks/pipeline/dnd.yml` targets only `pipeline-dnd`, applies only
  `rakestraw.dnd.pipeline`, and contains no deployment variables or procedural
  setup.
- Host inventory sets the confirmed WhisperX endpoint, enrolls only `eric`,
  preserves `dilfs` as the default campaign through the existing standalone
  role variable, and inherits Feature 0015's storage and publishing defaults.
- `dnd_pipeline_secrets` maps the required typed runtime keys to host Vault
  variables. Credential values in the consumer repository are Vault-encrypted;
  runtime delivery uses the suite's existing restricted secrets files.
- This feature introduces no plaintext credential values into commits, diffs,
  command examples, validation output, or documentation. A repository-wide
  historical secret audit is not an acceptance requirement.
- Legacy Narratio storage values are either represented through the current
  pipeline interface or deliberately retired; no stale high-precedence host
  variable can override the selected suite configuration accidentally.
- The root requirements manifest preserves unrelated existing dependencies
  and installs the exact reviewed public collection snapshot, its five
  standalone role dependencies, and its required native collections without
  using local repository paths.
- The controller uses the selected collection commit even if collection
  version `0.1.0` was already installed. The documented inventory, role, and
  collection paths correspond to the operational controller.
- Public dependency installation succeeds, and the new playbook passes syntax
  checking with the installed collection and roles. Documentation rendering
  with `ansible-doc` is not required for deployment acceptance.
- The initial limited playbook run against `pipeline-dnd` completes
  successfully using the encrypted inventory and ordinary sudo-capable remote
  account.
- The immediate identical second run reports `changed=0` and `failed=0`.
- Narratio's native configuration validation succeeds under the intended
  runtime identity, and a fresh session for `eric` can access and invoke the
  shared pipeline through runtime-group membership.
- One operator-selected D&D session completes transcription, processing,
  storage, and publishing, and its expected published output is observable at
  the configured destination. Either supported runtime identity may execute
  it. Production is the default profile; the operator may select the testing
  profile while retaining the same processing and publication acceptance scope.
- The operational documentation explains dependency installation, Vault use,
  deployment invocation, the login-session membership caveat, convergence
  verification, and representative pipeline invocation without disclosing
  credentials.
- The historical proof-of-concept role and playbook remain available during
  the initial rollout; their later retirement requires a separate explicit
  decision.

## Dependencies and Follow-up Work

[Feature 0015](0015-dnd-s3-storage-and-publishing.md) supplies the implemented
and independently reviewed pipeline interface, credentials, and publishing
boundary consumed here. Its administrative closeout remains before the
Feature 0016 sprint; its runtime implementation needs no further remediation.

Features 0013 and 0014 provide the public collection installation path and
existing-operator enrollment required here. Features 0009 and 0011 provide
the shared filesystem and managed credential-delivery policies.

After the real deployment demonstrates parity, a later cleanup feature may
retire the proof-of-concept `pipeline_dnd` role, its old playbook, and any
obsolete inventory values. Service scheduling, unattended ingestion,
additional human operators, backup policy, and additional hosts also remain
separate work.