ansible-workspace-dnd/docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md

Feature 0016: Deploy the D&D Pipeline to pipeline-dnd

Status: proposed

Summary

Add a minimal playbook and host inventory to rakestraw-ansible that apply rakestraw.dnd.pipeline to the existing pipeline-dnd host. Configure the host through host_vars, enroll eric as the initial human operator, keep credential values in Ansible Vault, and prove the complete pipeline through a real repeatable deployment and representative D&D session.

Feature 0015 must first provide the storage and publishing behavior needed for an operational end-to-end pipeline. Feature 0016 consumes that reviewed interface; it does not invent a temporary storage contract.

Problem or Opportunity

The collection and its application roles are implemented and installable, but the operational Ansible repository does not yet have a playbook for deploying the new collection architecture. Its existing D&D playbook targets a retired inventory name and invokes the proof-of-concept local pipeline_dnd role.

The current pipeline-dnd host inventory already records the connection identity and useful legacy Narratio storage values. A production-shaped first deployment now needs to express the new role's public interface, deliver secrets safely, reconcile the controller's dependency manifest, and validate the actual host rather than extending the role repositories with another synthetic scenario.

Scope

• Add rakestraw-ansible/playbooks/pipeline/dnd.yml, targeting the existing pipeline-dnd inventory host and applying only rakestraw.dnd.pipeline.
• Keep the playbook limited to its host selection and role list. Put all deployment choices in inventory/host_vars/pipeline-dnd/.
• Configure the external WhisperX transcription endpoint as https://transcription.ai.rakestrawhome.com/transcribe.
• Enroll the existing eric account through dnd_pipeline_operator_users.
• Map the role's dnd_pipeline_secrets interface to host-specific Vault variables. Store credential values only in an Ansible Vault-encrypted host variables file.
• Carry the storage and publishing settings selected by Feature 0015 through its final public pipeline interface. Preserve the operational intent of the existing Backblaze B2 endpoint, bucket, prefix, region, and path-style settings where they remain applicable.
• Reconcile existing legacy Narratio host variables so they cannot silently override the suite policy selected by the new pipeline role.
• Update rakestraw-ansible/requirements.yml so a clean controller can install the reviewed collection snapshot, all five standalone role dependencies, and required native collections from one manifest.
• Validate controller dependency resolution, playbook syntax, initial convergence, immediate clean convergence, operator access, native Narratio configuration, and one representative end-to-end pipeline run on the real host.
• Document the operational invocation, Vault prerequisite, initial operator, and expected validation path in rakestraw-ansible.

Non-goals

• Implementing storage or publishing behavior. Feature 0015 owns that prerequisite.
• Changing application or role behavior merely to accommodate this deployment.
• Replacing the existing inventory alias with pipeline.dnd, changing DNS, or moving connection variables out of the current host inventory.
• Adding a wrapper role, pre-task layer, service, timer, scheduler, web UI, or unattended session-ingestion mechanism.
• Creating or managing the eric account, its login policy, SSH access, or sudo policy.
• Adding another production host, high availability, backup policy, or disaster-recovery automation.
• Removing the proof-of-concept local pipeline_dnd role or its historical playbook before the new deployment has demonstrated operational parity.
• Publishing a new coordinated suite release solely as part of this deployment feature.

Target End State

rakestraw-ansible/playbooks/pipeline/dnd.yml is an intentionally small consumer playbook equivalent to:

---
- hosts:
    - pipeline-dnd
  roles:
    - rakestraw.dnd.pipeline
...

The existing inventory selects the host's connection identity. A dedicated host-variable file selects the pipeline's non-secret public inputs, including the WhisperX endpoint, the initial operator list, and the storage and publishing values exposed by Feature 0015. It maps runtime credential names to Vault-backed variables without embedding credential values in the playbook or ordinary plaintext inventory, for example:

dnd_pipeline_operator_users:
  - eric

dnd_pipeline_whisperx_transcribe_url: >-
  https://transcription.ai.rakestrawhome.com/transcribe

dnd_pipeline_secrets:
  AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
  OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"
  OBJECT_STORAGE_KEY_ID: "{{ vault_dnd_object_storage_key_id }}"
  OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}"

The corresponding credential variables reside in a tracked file encrypted in full with Ansible Vault. The Vault password or identity mechanism remains outside Git and outside this feature's managed content.

The controller dependency manifest selects one reviewed, publicly reachable suite snapshot containing Features 0014 and 0015. It also installs that snapshot's exact standalone role and native collection dependencies. A clean controller therefore does not depend on sibling development checkouts or previously installed roles.

On pipeline-dnd, the role converges the full application suite and its D&D configuration. The narratio service identity can run the pipeline, and eric can run it after starting a login session that observes the granted supplementary group membership. A representative session can be transcribed, processed, stored, and published through the configured production path. An immediate identical Ansible run reports no changes.

Repository Ownership

• rakestraw-ansible owns the consumer playbook, host inventory, encrypted host credentials, controller dependency manifest, and deployment runbook.
• ansible-role-dnd-pipeline owns the suite's public variables and the opinionated cross-application behavior consumed by the deployment.
• ansible-collection-dnd owns the assembled rakestraw.dnd.pipeline role snapshot and consumer-facing compatibility manifest.
• The standalone application roles continue to own installation and native application configuration. Feature 0016 consumes their reviewed releases or commits without reimplementing their behavior.
• ansible-workspace-dnd owns this roadmap, the later implementation plan, progress tracking, and independent review record.
• Application and asset repositories are read-only sources of runtime contracts and deployed content for this feature.

Constraints and Settled Decisions

• Feature 0015 is a prerequisite and must settle, implement, and review the storage and publishing interface before Feature 0016 becomes ready for an implementation sprint.
• The target is the inventory alias pipeline-dnd as currently defined in rakestraw-ansible, not a new pipeline.dnd alias.
• The playbook follows the repository's established style: hosts and roles only. Facts remain at Ansible's normal default, privilege escalation remains task-owned by the roles, and deployment variables belong in host inventory.
• eric is the initial human operator and must already exist on the managed host. The pipeline role grants only additive runtime-group membership.
• API credentials use the existing typed dnd_pipeline_secrets interface and Ansible Vault. Plaintext secret values must not enter Git, command lines, logs, examples, or ordinary variable files.
• The public Gitea repositories are the sole dependency sources. The deployment must not reference this workspace's sibling checkout paths.
• The dependency manifest must select an immutable release when Feature 0015 produces one. Otherwise, it must pin the exact reviewed public Git commit; it must not track an unbounded branch for the production deployment.
• The collection artifact does not itself install arbitrary Git-hosted role dependencies. The operational manifest must therefore include the five standalone roles as well as the collection and its native collection dependencies.
• The legacy host variables must be compared with the final Feature 0015 interface. Applicable values move to the new suite configuration, while obsolete or conflicting variables are removed rather than left to win by inventory precedence.
• The initial real-host run is an operational acceptance exercise. It should test observable pipeline behavior and convergence, not module task traces or the current text of defaults.

Simplest Viable Implementation

After Feature 0015 is reviewed, select its exact public collection revision and use the associated compatibility manifest to update the controller's single root requirements file. Add the minimal playbook and one ordinary host-variable file for public inputs. Add one fully encrypted host Vault file for the two credential values, populated by the operator without exposing its plaintext during implementation or review.

Install or update dependencies from the root manifest, confirm that Ansible resolves rakestraw.dnd.pipeline, syntax-check the playbook, and converge only pipeline-dnd. Start a fresh operator login after enrollment, validate the native Narratio configuration under the intended runtime identity, rerun the same play immediately to confirm clean convergence, and process one selected session through the complete Feature 0015 publishing path. Record concise operational instructions and results; do not add a second test harness around the real deployment.

Rationale and Alternatives

Keeping the playbook nearly declarative makes inventory the durable record of what differs about this host and allows ordinary Ansible precedence to remain the customization mechanism. Embedding endpoint, operator, storage, or secret mapping variables in the playbook would make a one-host play harder to reuse and would diverge from the established rakestraw-ansible style.

A local wrapper role could hide the collection interface, but it would create another layer with no independent behavior. Applying the collection role by FQCN proves the same consumer path that future hosts should use.

Tracking collection main would be convenient during development, but it would make a later controller bootstrap select an unreviewed compatibility set. An exact public commit retains development flexibility until the next release while still making this deployment reproducible.

Running the real-host deployment before Feature 0015 would establish only an installation demonstration: the pipeline still could not complete its required storage and publishing path. Making Feature 0015 an explicit prerequisite avoids temporary inventory variables and gives the acceptance run a meaningful end-to-end boundary.

Acceptance Criteria

• Feature 0015 is complete and independently reviewed before Feature 0016 implementation begins.
• playbooks/pipeline/dnd.yml targets only pipeline-dnd, applies only rakestraw.dnd.pipeline, and contains no deployment variables or procedural setup.
• Host inventory sets the confirmed WhisperX endpoint, enrolls only eric, and supplies the final Feature 0015 storage and publishing inputs.
• dnd_pipeline_secrets maps the required typed runtime keys to host Vault variables, and the actual values exist only in a valid Ansible Vault-encrypted file.
• No plaintext credential value is present in any tracked revision, diff, command example, validation output, or deployment documentation.
• Legacy Narratio storage values are either represented through the current pipeline interface or deliberately retired; no stale high-precedence host variable can override the selected suite configuration accidentally.
• The root requirements manifest preserves unrelated existing dependencies and installs the exact reviewed public collection snapshot, its five standalone role dependencies, and its required native collections without using local repository paths.
• On a controller bootstrapped from that manifest, ansible-doc resolves rakestraw.dnd.pipeline and each role dependency, and the new playbook passes syntax checking.
• The initial limited playbook run against pipeline-dnd completes successfully using the encrypted inventory and ordinary sudo-capable remote account.
• The immediate identical second run reports changed=0 and failed=0.
• Narratio's native configuration validation succeeds under the intended runtime identity, and a fresh session for eric can access and invoke the shared pipeline through runtime-group membership.
• One operator-selected D&D session completes transcription, processing, storage, and publishing, and its expected published output is observable at the configured destination.
• The operational documentation explains dependency installation, Vault use, deployment invocation, the login-session membership caveat, convergence verification, and representative pipeline invocation without disclosing credentials.
• The historical proof-of-concept role and playbook remain available during the initial rollout; their later retirement requires a separate explicit decision.

Dependencies and Follow-up Work

Feature 0015 is the immediate next unit of work and a hard prerequisite. It selects the pipeline interface, credentials, and observable publishing boundary that Feature 0016 will deploy.

Features 0013 and 0014 provide the public collection installation path and existing-operator enrollment required here. Features 0009 and 0011 provide the shared filesystem and managed credential-delivery policies.

After the real deployment demonstrates parity, a later cleanup feature may retire the proof-of-concept pipeline_dnd role, its old playbook, and any obsolete inventory values. Service scheduling, unattended ingestion, additional human operators, backup policy, and additional hosts also remain separate work.