ansible-workspace-dnd/docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md

# Feature 0016: Deploy the D&D Pipeline to `pipeline-dnd`

Status: proposed

## Summary

Add a minimal playbook and host inventory to `rakestraw-ansible` that apply
`rakestraw.dnd.pipeline` to the existing `pipeline-dnd` host. Configure the
host through `host_vars`, enroll `eric` as the initial human operator, keep
credential values in Ansible Vault, and prove the complete pipeline through a
real repeatable deployment and representative D&D session.

[Feature 0015](0015-dnd-s3-storage-and-publishing.md) must first provide the
storage and publishing behavior needed for an operational end-to-end pipeline.
Feature 0016 consumes that reviewed interface; it does not invent a temporary
storage contract.

## Problem or Opportunity

The collection and its application roles are implemented and installable, but
the operational Ansible repository does not yet have a playbook for deploying
the new collection architecture. Its existing D&D playbook targets a retired
inventory name and invokes the proof-of-concept local `pipeline_dnd` role.

The current `pipeline-dnd` host inventory already records the connection
identity and useful legacy Narratio storage values. A production-shaped first
deployment now needs to express the new role's public interface, deliver
secrets safely, reconcile the controller's dependency manifest, and validate
the actual host rather than extending the role repositories with another
synthetic scenario.

## Scope

- Add `rakestraw-ansible/playbooks/pipeline/dnd.yml`, targeting the existing
  `pipeline-dnd` inventory host and applying only
  `rakestraw.dnd.pipeline`.
- Keep the playbook limited to its host selection and role list. Put all
  deployment choices in `inventory/host_vars/pipeline-dnd/`.
- Configure the external WhisperX transcription endpoint as
  `https://transcription.ai.rakestrawhome.com/transcribe`.
- Enroll the existing `eric` account through
  `dnd_pipeline_operator_users`.
- Map the role's `dnd_pipeline_secrets` interface to host-specific Vault
  variables. Store credential values only in an Ansible Vault-encrypted host
  variables file.
- Carry the storage and publishing settings selected by Feature 0015 through
  its final public pipeline interface. Preserve the operational intent of the
  existing Backblaze B2 endpoint, bucket, prefix, region, and path-style
  settings where they remain applicable.
- Reconcile existing legacy Narratio host variables so they cannot silently
  override the suite policy selected by the new pipeline role.
- Update `rakestraw-ansible/requirements.yml` so a clean controller can install
  the reviewed collection snapshot, all five standalone role dependencies,
  and required native collections from one manifest.
- Validate controller dependency resolution, playbook syntax, initial
  convergence, immediate clean convergence, operator access, native Narratio
  configuration, and one representative end-to-end pipeline run on the real
  host.
- Document the operational invocation, Vault prerequisite, initial operator,
  and expected validation path in `rakestraw-ansible`.

## Non-goals

- Implementing storage or publishing behavior. Feature 0015 owns that
  prerequisite.
- Changing application or role behavior merely to accommodate this
  deployment.
- Replacing the existing inventory alias with `pipeline.dnd`, changing DNS,
  or moving connection variables out of the current host inventory.
- Adding a wrapper role, pre-task layer, service, timer, scheduler, web UI, or
  unattended session-ingestion mechanism.
- Creating or managing the `eric` account, its login policy, SSH access, or
  sudo policy.
- Adding another production host, high availability, backup policy, or
  disaster-recovery automation.
- Removing the proof-of-concept local `pipeline_dnd` role or its historical
  playbook before the new deployment has demonstrated operational parity.
- Publishing a new coordinated suite release solely as part of this
  deployment feature.

## Target End State

`rakestraw-ansible/playbooks/pipeline/dnd.yml` is an intentionally small
consumer playbook equivalent to:

```yaml
---
- hosts:
    - pipeline-dnd
  roles:
    - rakestraw.dnd.pipeline
...
```

The existing inventory selects the host's connection identity. A dedicated
host-variable file selects the pipeline's non-secret public inputs, including
the WhisperX endpoint, the initial operator list, and the storage and
publishing values exposed by Feature 0015. It maps runtime credential names to
Vault-backed variables without embedding credential values in the playbook or
ordinary plaintext inventory, for example:

```yaml
dnd_pipeline_operator_users:
  - eric

dnd_pipeline_whisperx_transcribe_url: >-
  https://transcription.ai.rakestrawhome.com/transcribe

dnd_pipeline_secrets:
  AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
  OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"
  OBJECT_STORAGE_KEY_ID: "{{ vault_dnd_object_storage_key_id }}"
  OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}"
```

The corresponding credential variables reside in a tracked file encrypted in
full with Ansible Vault. The Vault password or identity mechanism remains
outside Git and outside this feature's managed content.

The controller dependency manifest selects one reviewed, publicly reachable
suite snapshot containing Features 0014 and 0015. It also installs that
snapshot's exact standalone role and native collection dependencies. A clean
controller therefore does not depend on sibling development checkouts or
previously installed roles.

On `pipeline-dnd`, the role converges the full application suite and its D&D
configuration. The `narratio` service identity can run the pipeline, and
`eric` can run it after starting a login session that observes the granted
supplementary group membership. A representative session can be transcribed,
processed, stored, and published through the configured production path. An
immediate identical Ansible run reports no changes.

## Repository Ownership

- `rakestraw-ansible` owns the consumer playbook, host inventory, encrypted
  host credentials, controller dependency manifest, and deployment runbook.
- `ansible-role-dnd-pipeline` owns the suite's public variables and the
  opinionated cross-application behavior consumed by the deployment.
- `ansible-collection-dnd` owns the assembled `rakestraw.dnd.pipeline` role
  snapshot and consumer-facing compatibility manifest.
- The standalone application roles continue to own installation and native
  application configuration. Feature 0016 consumes their reviewed releases or
  commits without reimplementing their behavior.
- `ansible-workspace-dnd` owns this roadmap, the later implementation plan,
  progress tracking, and independent review record.
- Application and asset repositories are read-only sources of runtime
  contracts and deployed content for this feature.

## Constraints and Settled Decisions

- Feature 0015 is a prerequisite and must settle, implement, and review the
  storage and publishing interface before Feature 0016 becomes ready for an
  implementation sprint.
- The target is the inventory alias `pipeline-dnd` as currently defined in
  `rakestraw-ansible`, not a new `pipeline.dnd` alias.
- The playbook follows the repository's established style: hosts and roles
  only. Facts remain at Ansible's normal default, privilege escalation remains
  task-owned by the roles, and deployment variables belong in host inventory.
- `eric` is the initial human operator and must already exist on the managed
  host. The pipeline role grants only additive runtime-group membership.
- API credentials use the existing typed `dnd_pipeline_secrets` interface and
  Ansible Vault. Plaintext secret values must not enter Git, command lines,
  logs, examples, or ordinary variable files.
- The public Gitea repositories are the sole dependency sources. The
  deployment must not reference this workspace's sibling checkout paths.
- The dependency manifest must select an immutable release when Feature 0015
  produces one. Otherwise, it must pin the exact reviewed public Git commit;
  it must not track an unbounded branch for the production deployment.
- The collection artifact does not itself install arbitrary Git-hosted role
  dependencies. The operational manifest must therefore include the five
  standalone roles as well as the collection and its native collection
  dependencies.
- The legacy host variables must be compared with the final Feature 0015
  interface. Applicable values move to the new suite configuration, while
  obsolete or conflicting variables are removed rather than left to win by
  inventory precedence.
- The initial real-host run is an operational acceptance exercise. It should
  test observable pipeline behavior and convergence, not module task traces or
  the current text of defaults.

## Simplest Viable Implementation

After Feature 0015 is reviewed, select its exact public collection revision
and use the associated compatibility manifest to update the controller's
single root requirements file. Add the minimal playbook and one ordinary
host-variable file for public inputs. Add one fully encrypted host Vault file
for the two credential values, populated by the operator without exposing its
plaintext during implementation or review.

Install or update dependencies from the root manifest, confirm that Ansible
resolves `rakestraw.dnd.pipeline`, syntax-check the playbook, and converge only
`pipeline-dnd`. Start a fresh operator login after enrollment, validate the
native Narratio configuration under the intended runtime identity, rerun the
same play immediately to confirm clean convergence, and process one selected
session through the complete Feature 0015 publishing path. Record concise
operational instructions and results; do not add a second test harness around
the real deployment.

## Rationale and Alternatives

Keeping the playbook nearly declarative makes inventory the durable record of
what differs about this host and allows ordinary Ansible precedence to remain
the customization mechanism. Embedding endpoint, operator, storage, or secret
mapping variables in the playbook would make a one-host play harder to reuse
and would diverge from the established `rakestraw-ansible` style.

A local wrapper role could hide the collection interface, but it would create
another layer with no independent behavior. Applying the collection role by
FQCN proves the same consumer path that future hosts should use.

Tracking collection `main` would be convenient during development, but it
would make a later controller bootstrap select an unreviewed compatibility
set. An exact public commit retains development flexibility until the next
release while still making this deployment reproducible.

Running the real-host deployment before Feature 0015 would establish only an
installation demonstration: the pipeline still could not complete its
required storage and publishing path. Making Feature 0015 an explicit
prerequisite avoids temporary inventory variables and gives the acceptance
run a meaningful end-to-end boundary.

## Acceptance Criteria

- Feature 0015 is complete and independently reviewed before Feature 0016
  implementation begins.
- `playbooks/pipeline/dnd.yml` targets only `pipeline-dnd`, applies only
  `rakestraw.dnd.pipeline`, and contains no deployment variables or procedural
  setup.
- Host inventory sets the confirmed WhisperX endpoint, enrolls only `eric`,
  and supplies the final Feature 0015 storage and publishing inputs.
- `dnd_pipeline_secrets` maps the required typed runtime keys to host Vault
  variables, and the actual values exist only in a valid Ansible
  Vault-encrypted file.
- No plaintext credential value is present in any tracked revision, diff,
  command example, validation output, or deployment documentation.
- Legacy Narratio storage values are either represented through the current
  pipeline interface or deliberately retired; no stale high-precedence host
  variable can override the selected suite configuration accidentally.
- The root requirements manifest preserves unrelated existing dependencies
  and installs the exact reviewed public collection snapshot, its five
  standalone role dependencies, and its required native collections without
  using local repository paths.
- On a controller bootstrapped from that manifest, `ansible-doc` resolves
  `rakestraw.dnd.pipeline` and each role dependency, and the new playbook
  passes syntax checking.
- The initial limited playbook run against `pipeline-dnd` completes
  successfully using the encrypted inventory and ordinary sudo-capable remote
  account.
- The immediate identical second run reports `changed=0` and `failed=0`.
- Narratio's native configuration validation succeeds under the intended
  runtime identity, and a fresh session for `eric` can access and invoke the
  shared pipeline through runtime-group membership.
- One operator-selected D&D session completes transcription, processing,
  storage, and publishing, and its expected published output is observable at
  the configured destination.
- The operational documentation explains dependency installation, Vault use,
  deployment invocation, the login-session membership caveat, convergence
  verification, and representative pipeline invocation without disclosing
  credentials.
- The historical proof-of-concept role and playbook remain available during
  the initial rollout; their later retirement requires a separate explicit
  decision.

## Dependencies and Follow-up Work

[Feature 0015](0015-dnd-s3-storage-and-publishing.md) is the immediate next unit
of work and a hard prerequisite. It selects the pipeline interface,
credentials, and observable publishing boundary that Feature 0016 will deploy.

Features 0013 and 0014 provide the public collection installation path and
existing-operator enrollment required here. Features 0009 and 0011 provide
the shared filesystem and managed credential-delivery policies.

After the real deployment demonstrates parity, a later cleanup feature may
retire the proof-of-concept `pipeline_dnd` role, its old playbook, and any
obsolete inventory values. Service scheduling, unattended ingestion,
additional human operators, backup policy, and additional hosts also remain
separate work.