diff --git a/docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md b/docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md index e2599c0..36296ef 100644 --- a/docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md +++ b/docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md @@ -1,88 +1,41 @@ -# Feature 0016: Deploy the D&D Pipeline to `pipeline-dnd` +# Feature 0016: Prepare the D&D Deployment Playbook for `pipeline-dnd` Status: in_progress ## Summary -Add a minimal playbook and host inventory to `rakestraw-ansible` that apply -`rakestraw.dnd.pipeline` to the existing `pipeline-dnd` host. Configure the -host through `host_vars`, enroll `eric` as the initial human operator, keep -credential values in Ansible Vault, and prove the complete pipeline through a -real repeatable deployment and representative D&D session. - -[Feature 0015](0015-dnd-s3-storage-and-publishing.md) has implemented and passed -independent review of the storage and publishing behavior needed for an -operational end-to-end pipeline. Feature 0016 consumes that reviewed interface -and its suite defaults. - -## Problem or Opportunity - -The collection and its application roles are implemented and installable, but -the operational Ansible repository does not yet have a playbook for deploying -the new collection architecture. Its existing D&D playbook targets a retired -inventory name and invokes the proof-of-concept local `pipeline_dnd` role. - -The current `pipeline-dnd` host inventory already records the connection -identity and useful legacy Narratio storage values. A production-shaped first -deployment now needs to express the new role's public interface, deliver -secrets safely, reconcile the controller's dependency manifest, and validate -the actual host rather than extending the role repositories with another -synthetic scenario. +Write and commit the consumer playbook, necessary host variables, dependency +manifest, and concise usage documentation in `rakestraw-ansible`. This feature +ends with repository preparation. Deployment to a controller or target host +and execution of the pipeline are separate operator work. ## Scope -- Add `rakestraw-ansible/playbooks/pipeline/dnd.yml`, targeting the existing - `pipeline-dnd` inventory host and applying only - `rakestraw.dnd.pipeline`. -- Keep the playbook limited to its host selection and role list. Put all - deployment choices in `inventory/host_vars/pipeline-dnd/`. -- Configure the external WhisperX transcription endpoint as - `https://transcription.ai.rakestrawhome.com/transcribe`. -- Enroll the existing `eric` account through - `dnd_pipeline_operator_users`. -- Map the role's `dnd_pipeline_secrets` interface to host-specific Vault - variables. Store credential values only in an Ansible Vault-encrypted host - variables file. -- Inherit Feature 0015's storage and publishing defaults, which already match - the intended Backblaze B2 deployment. Host inventory contains only actual - host choices and deliberate overrides, without copying suite defaults. -- Preserve the existing default campaign choice, `dilfs`, through the - standalone role's `narratio_campaigns_default_campaign_id` variable. -- Reconcile existing legacy Narratio host variables so they cannot silently - override the suite policy selected by the new pipeline role. -- Update `rakestraw-ansible/requirements.yml` so a clean controller can install - the reviewed collection snapshot, all five standalone role dependencies, - and required native collections from one manifest. -- Validate controller dependency resolution, playbook syntax, initial - convergence, immediate clean convergence, operator access, native Narratio - configuration, and one representative end-to-end pipeline run on the real - host. -- Document the operational invocation, Vault prerequisite, initial operator, - and expected validation path in `rakestraw-ansible`. +- `playbooks/pipeline/dnd.yml` targets the existing `pipeline-dnd` alias and + applies `rakestraw.dnd.pipeline`, using only hosts and roles. +- Host choices belong in `inventory/host_vars/pipeline-dnd/`: the WhisperX + endpoint, initial operator `eric`, default campaign `dilfs`, and credential + references. Unknown required values are explained in adjacent comments. +- Inherit the reviewed suite's storage, publication, caching, cleanup, and + application defaults. Retire obsolete legacy host variables. +- Root `requirements.yml` selects the reviewed public collection snapshot and + its role/native collection dependencies, preserving unrelated entries. +- Document local validation and eventual operator use. Commit the completed + repository changes without requiring remote access or real credentials. ## Non-goals -- Implementing storage or publishing behavior. Feature 0015 owns that - prerequisite. -- Changing application or role behavior merely to accommodate this - deployment. -- Replacing the existing inventory alias with `pipeline.dnd`, changing DNS, - or moving connection variables out of the current host inventory. -- Adding a wrapper role, pre-task layer, service, timer, scheduler, web UI, or - unattended session-ingestion mechanism. -- Creating or managing the `eric` account, its login policy, SSH access, or - sudo policy. -- Adding another production host, high availability, backup policy, or - disaster-recovery automation. -- Removing the proof-of-concept local `pipeline_dnd` role or its historical - playbook before the new deployment has demonstrated operational parity. -- Publishing a new coordinated suite release solely as part of this - deployment feature. +- Installing or updating anything on the operational controller or target. +- Connecting to `pipeline-dnd`, validating its platform or sudo access, + converging the host, checking runtime identities, or executing a real session. +- Obtaining API keys, choosing a Vault password, populating real credentials, + or selecting acceptance audio/session data. +- Changing application/role behavior, publishing a new suite release, adding + orchestration layers, or retiring the proof-of-concept role/playbook. ## Target End State -`rakestraw-ansible/playbooks/pipeline/dnd.yml` is an intentionally small -consumer playbook equivalent to: +The committed playbook is: ```yaml --- @@ -93,12 +46,8 @@ consumer playbook equivalent to: ... ``` -The existing inventory selects the host's connection identity. A dedicated -host-variable file selects the WhisperX endpoint, the initial operator list, -and the default campaign. Storage, caching, publication, and cleanup inherit -the reviewed suite defaults. The file maps runtime credential names to -Vault-backed variables without embedding credential values in the playbook or -ordinary plaintext inventory, for example: +Existing connection variables remain in the host's `main.yml`. Its +`narratio.yml` contains the following public configuration and comments: ```yaml dnd_pipeline_operator_users: @@ -109,6 +58,10 @@ dnd_pipeline_whisperx_transcribe_url: >- narratio_campaigns_default_campaign_id: dilfs +# Before deployment, define the four vault_dnd_* variables below in an +# Ansible Vault-encrypted host_vars/pipeline-dnd/vault.yml file. +# Values are not known here; leave these references unresolved until supplied. +# Do not put API keys or object-storage credentials in this plaintext file. dnd_pipeline_secrets: AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}" OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}" @@ -116,214 +69,77 @@ dnd_pipeline_secrets: OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}" ``` -The corresponding credential variables reside in a tracked file encrypted in -full with Ansible Vault. The Vault password or identity mechanism remains -outside Git and outside this feature's managed content. +The references and comments are a complete deliverable for unknown secrets. +No encrypted file, placeholder key, or real credential is required for feature +completion. Future operator-supplied values should be Vault-encrypted in Git; +the suite's existing restricted runtime secrets files remain the delivery path. -The controller dependency manifest selects public collection commit -`46913446765e86ecfb29ebad9955d579ef6a4c7e`, containing the reviewed Features -0014 and 0015 and pipeline commit -`51566afcf69083cdbb5b5718f2544c86b079001c`. Its compatibility manifest selects -the five standalone roles at `v0.1.0`, `ansible.posix` at `2.2.0`, and -`community.general` at `13.3.0`. A clean controller therefore does not depend -on sibling development checkouts or previously installed roles. +The manifest pins the collection to +`46913446765e86ecfb29ebad9955d579ef6a4c7e`, containing reviewed pipeline commit +`51566afcf69083cdbb5b5718f2544c86b079001c`. Its five standalone role dependencies +use their reviewed `v0.1.0` tags, with `ansible.posix` `2.2.0` and +`community.general` `13.3.0`. Suite Git dependencies use public Gitea URLs; +native collections come from Galaxy. Local sibling checkouts are not dependencies. -The published collection `v0.1.0` predates these features. The selected -development snapshot still declares collection version `0.1.0`, so the -deployment instructions explicitly refresh an already-installed collection -from the pinned commit instead of allowing an equal-version installation to -be skipped. Unrelated controller dependencies remain preserved. - -On `pipeline-dnd`, the role converges the full application suite and its D&D -configuration. The `narratio` service identity can run the pipeline, and -`eric` can run it after starting a login session that observes the granted -supplementary group membership. A representative session can be transcribed, -processed, stored, and published through the configured production path. An -immediate identical Ansible run reports no changes. +The runbook explains that the selected collection snapshot still reports +`0.1.0`, so eventual operator installation must explicitly refresh it from +the pinned commit if an older `0.1.0` is installed. This is an instruction +for future use, not an operation required by this feature. ## Repository Ownership -- `rakestraw-ansible` owns the consumer playbook, host inventory, encrypted - host credentials, controller dependency manifest, and deployment runbook. -- `ansible-role-dnd-pipeline` owns the suite's public variables and the - opinionated cross-application behavior consumed by the deployment. -- `ansible-collection-dnd` owns the assembled `rakestraw.dnd.pipeline` role - snapshot and consumer-facing compatibility manifest. -- The standalone application roles continue to own installation and native - application configuration. Feature 0016 consumes their reviewed releases or - commits without reimplementing their behavior. -- `ansible-workspace-dnd` owns this roadmap, the later implementation plan, - progress tracking, and independent review record. -- Application and asset repositories are read-only sources of runtime - contracts and deployed content for this feature. +- `rakestraw-ansible`: consumer files, comments, dependency manifest, and runbook. +- Workspace: roadmap, staged plan, progress, and independent review record. +- Roles, collection, applications, and assets: read-only contract references. ## Constraints and Settled Decisions -- Feature 0015's implementation and independent review satisfy the storage - and publishing prerequisite. Its workspace completion record should be - closed out before the Feature 0016 sprint. -- The target is the inventory alias `pipeline-dnd` as currently defined in - `rakestraw-ansible`, not a new `pipeline.dnd` alias. -- The playbook follows the repository's established style: hosts and roles - only. Facts remain at Ansible's normal default, privilege escalation remains - task-owned by the roles, and deployment variables belong in host inventory. -- `eric` is the initial human operator and must already exist on the managed - host. The pipeline role grants only additive runtime-group membership. -- API credentials use the existing typed `dnd_pipeline_secrets` interface and - Ansible Vault. Plaintext secret values must not enter Git, command lines, - logs, examples, or ordinary variable files. -- Public Gitea repositories supply the suite's Git dependencies; native - collections come from Ansible Galaxy. The deployment must not reference - this workspace's sibling checkout paths. -- The collection is pinned to the reviewed public commit identified above. - Feature 0015 did not create a release; this deployment does not track an - unbounded branch or require a new coordinated release. -- The collection artifact does not itself install arbitrary Git-hosted role - dependencies. The operational manifest must therefore include the five - standalone roles as well as the collection and its native collection - dependencies. -- Remove the legacy `narratio_s3_*` host variables because their intended - values are supplied by the suite defaults. Replace - `narratio_default_campaign_id` with - `narratio_campaigns_default_campaign_id: dilfs`. The standalone role already - exposes this host-specific setting; no new pipeline variable is required. -- The initial real-host run is an operational acceptance exercise. It should - test observable pipeline behavior and convergence, not module task traces or - the current text of defaults. +The workspace [Ansible](../../ansible.md), [testing](../../testing.md), and +[planning](../../planning.md) policies apply. Feature 0015 has passed review +and supplies the required suite interface. -## Operational Prerequisites - -Local repository preparation and syntax/dependency checks can proceed with the -workspace's existing Ansible virtual environment. Before live deployment, -establish the controller, target access, and Vault details below. Select the -session/audio before the end-to-end exercise, not before local implementation. -Record non-secret choices in the implementation plan or deployment runbook: - -- Identify the operational Ansible controller and its deployment checkout. - The repository configuration currently selects `/etc/ansible/inventory`, - `/etc/ansible/roles`, and `/etc/ansible/collections`. Confirm the effective - paths so dependency installation and playbook execution use the same - inventory and installed content. Distinguish those paths from this - development workspace. -- The target must satisfy the roles' Ubuntu 26.04 and access requirements. - Normal connection, role guards, and privileged tasks can establish these; - separate platform or sudo probes are not required acceptance tests. -- Establish the controller's existing Vault password or identity mechanism - and operator availability to populate the four credential values in the - encrypted host file. Record how Vault is selected, never its password or - decrypted contents. -- Have the operator select the acceptance campaign, session identity, and - audio input, with the corresponding session file and accessible input - location. This is a real processing and publication exercise using the - selected providers; session selection remains an operator decision. - -These are deployment inputs and operational checks, not additional production -tasks or a new automated preflight framework. +- Keep playbooks minimal and settings in inventory. Preserve `pipeline-dnd` + and its connection identity; do not rename it to `pipeline.dnd`. +- Preserve `dilfs` using the existing standalone Narratio variable. Remove + legacy `narratio_s3_*` and `narratio_default_campaign_id` host settings. +- Missing operational inputs are documented, not treated as implementation + blockers. No remote controller, SSH identity, Vault password, or session is + required for repository preparation. +- Local dependency installation and syntax/host-selection checks are permitted. + Existing passing evidence may be reused when relevant inputs are unchanged. +- `ansible-doc` rendering is not an acceptance gate. Missing role argument + descriptions are a separate documentation improvement. ## Simplest Viable Implementation -Use the reviewed collection revision and associated compatibility manifest to -update the controller's single root requirements file. Add the minimal playbook and one ordinary -host-variable file for the endpoint, operator, campaign, and credential -references, inheriting the suite's storage and publication defaults. Add one -fully encrypted host Vault file for the four credential values, populated by -the operator without exposing its plaintext during implementation or review. - -Install or update dependencies from the root manifest at the confirmed -controller paths, explicitly refreshing the collection from its pinned commit -even when `0.1.0` is already installed. Confirm that Ansible -resolves `rakestraw.dnd.pipeline`, syntax-check the playbook, and converge only -`pipeline-dnd`. Start a fresh operator login after enrollment, validate the -native Narratio configuration under the intended runtime identity, rerun the -same play immediately to confirm clean convergence, and process one selected -session through the complete Feature 0015 publishing path. Record concise -operational instructions and results; do not add a second test harness around -the real deployment. - -## Rationale and Alternatives - -Keeping the playbook nearly declarative makes inventory the durable record of -what differs about this host and allows ordinary Ansible precedence to remain -the customization mechanism. Embedding endpoint, operator, storage, or secret -mapping variables in the playbook would make a one-host play harder to reuse -and would diverge from the established `rakestraw-ansible` style. - -A local wrapper role could hide the collection interface, but it would create -another layer with no independent behavior. Applying the collection role by -FQCN proves the same consumer path that future hosts should use. - -Tracking collection `main` would be convenient during development, but it -would make a later controller bootstrap select an unreviewed compatibility -set. An exact public commit retains development flexibility until the next -release while still making this deployment reproducible. - -Feature 0015 now supplies the reviewed storage and publishing path needed for -meaningful end-to-end acceptance. Inheriting that policy keeps host inventory -focused on deployment choices and avoids maintaining a second copy of the -suite defaults. +Retain the already-committed minimal playbook, manifest, and public inventory. +Add adjacent comments for unresolved credentials, align the runbook with the +repository-only scope, perform appropriate local checks, and commit. No new +Ansible task, runtime test harness, or deployment automation is needed. ## Acceptance Criteria -- Feature 0015 is complete and independently reviewed before Feature 0016 - implementation begins. -- `playbooks/pipeline/dnd.yml` targets only `pipeline-dnd`, applies only - `rakestraw.dnd.pipeline`, and contains no deployment variables or procedural - setup. -- Host inventory sets the confirmed WhisperX endpoint, enrolls only `eric`, - preserves `dilfs` as the default campaign through the existing standalone - role variable, and inherits Feature 0015's storage and publishing defaults. -- `dnd_pipeline_secrets` maps the required typed runtime keys to host Vault - variables. Credential values in the consumer repository are Vault-encrypted; - runtime delivery uses the suite's existing restricted secrets files. -- This feature introduces no plaintext credential values into commits, diffs, - command examples, validation output, or documentation. A repository-wide - historical secret audit is not an acceptance requirement. -- Legacy Narratio storage values are either represented through the current - pipeline interface or deliberately retired; no stale high-precedence host - variable can override the selected suite configuration accidentally. -- The root requirements manifest preserves unrelated existing dependencies - and installs the exact reviewed public collection snapshot, its five - standalone role dependencies, and its required native collections without - using local repository paths. -- The controller uses the selected collection commit even if collection - version `0.1.0` was already installed. The documented inventory, role, and - collection paths correspond to the operational controller. -- Public dependency installation succeeds, and the new playbook passes syntax - checking with the installed collection and roles. Documentation rendering - with `ansible-doc` is not required for deployment acceptance. -- The initial limited playbook run against `pipeline-dnd` completes - successfully using the encrypted inventory and ordinary sudo-capable remote - account. -- The immediate identical second run reports `changed=0` and `failed=0`. -- Narratio's native configuration validation succeeds under the intended - runtime identity, and a fresh session for `eric` can access and invoke the - shared pipeline through runtime-group membership. -- One operator-selected D&D session completes transcription, processing, - storage, and publishing, and its expected published output is observable at - the configured destination. Either supported runtime identity may execute - it. Production is the default profile; the operator may select the testing - profile while retaining the same processing and publication acceptance scope. -- The operational documentation explains dependency installation, Vault use, - deployment invocation, the login-session membership caveat, convergence - verification, and representative pipeline invocation without disclosing - credentials. -- The historical proof-of-concept role and playbook remain available during - the initial rollout; their later retirement requires a separate explicit - decision. +- The playbook targets only `pipeline-dnd` and applies only the collection role. +- Host inventory expresses the known choices above and documents unknown + credentials next to their references without real or fake secret values. +- The dependency manifest contains the reviewed public pins and preserves + unrelated dependencies. Public installation and local syntax/host-selection + evidence are available; no documentation-rendering check is required. +- The runbook distinguishes repository readiness from future deployment and + explains how the operator supplies missing values and eventually invokes it. +- Changes are committed in the owning repositories, with clean working trees + for the feature's changes. No controller deployment, target convergence, + clean-repeat run, or real pipeline execution is required or claimed. +- Historical role/playbook files remain available. ## Dependencies and Follow-up Work -[Feature 0015](0015-dnd-s3-storage-and-publishing.md) supplies the implemented -and independently reviewed pipeline interface, credentials, and publishing -boundary consumed here. Its administrative closeout remains before the -Feature 0016 sprint; its runtime implementation needs no further remediation. +Features 0013–0015 supply public dependencies, operator enrollment, and storage +policy. Feature 0016's consumer preparation already exists in `12b2462`, with +initial documentation in `716a80e`; remaining work aligns comments and docs +with the user's clarified scope. -Features 0013 and 0014 provide the public collection installation path and -existing-operator enrollment required here. Features 0009 and 0011 provide -the shared filesystem and managed credential-delivery policies. - -After the real deployment demonstrates parity, a later cleanup feature may -retire the proof-of-concept `pipeline_dnd` role, its old playbook, and any -obsolete inventory values. Service scheduling, unattended ingestion, -additional human operators, backup policy, and additional hosts also remain -separate work. +The operator may later provision Vault values, install dependencies on their +controller, deploy to the host, and validate real processing/publication. +Those actions require a separate request and are not unfinished Feature 0016 +acceptance requirements. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index d21c641..8a608bd 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,6 +1,6 @@ -# Implementation Plan 0016: Deploy the D&D Pipeline to `pipeline-dnd` +# Implementation Plan 0016: Prepare the D&D Deployment Playbook -Status: in_progress +Status: ready Target executor: GPT-5.6-Terra @@ -10,440 +10,169 @@ Active plan location: `docs/roadmap/implementation.md` ## Objective -Deploy the reviewed collection through a minimal consumer playbook, host -inventory, and Ansible Vault. Demonstrate clean repeated convergence and one -operator-selected session through real publication. The feature roadmap owns -scope, rationale, target state, and acceptance criteria. +Finish and commit repository preparation in `rakestraw-ansible`: minimal +playbook, host variables with comments for unknown values, reviewed dependency +pins, and concise operator instructions. This revision supersedes all previous +deployment and live-acceptance stages. Do not execute them. ## Execution Rules -- Execute stages in order. Before each stage, reread its Required Policy - Review and check the owning repository's working tree. Preserve unrelated - changes. Validate and commit each completed stage with its stated intent. -- Keep the playbook limited to hosts and roles. Use existing interfaces; - add no wrapper, preflight framework, custom module, or test harness. -- Inherit suite storage, publication, caching, cleanup, version, and asset - defaults. Set only the roadmap-selected host choices. -- Never print decrypted inventory, credential values, or Vault passwords in - commands, logs, documentation, or Git. Do not use decrypted inventory dumps - as validation evidence. The operator supplies secrets directly through - Ansible Vault's encrypted-file workflow, using an established secure editor. - Do not solicit secrets in chat or create placeholder credentials. -- Stop dependent work when operational inputs are missing or validation fails. - Record the concrete blocker; local checks do not prove a live deployment. -- Evaluate failures against the required outcome. Documentation rendering, - incidental warnings, and optional diagnostics are not deployment acceptance - gates. Report them separately unless they demonstrate a failure of required - installation, parsing, convergence, or application behavior. Do not add new - checks as mandatory conditions during execution. -- Reuse successful validation evidence while its inputs remain unchanged. - Rerun affected checks after changes; do not repeat entire stages solely to - produce another record or because a non-blocking diagnostic failed. -- Scope blockers to the operation that requires the missing input. Complete - Stages 1–2 locally without remote SSH, production Vault values, or an - acceptance session. Controller/SSH/Vault inputs gate Stage 3; session and - audio selection gate Stage 4. Continue independent authorized preparation. -- A missing executable on `PATH` is not proof that tooling is uninstalled. - Check the workspace `.venv/bin` and repository test drivers before reporting - a missing dependency. Use the verified executable path in diagnostics. -- An Ansible controller is the environment running Ansible; it requires no - controller daemon or `/etc/ansible` directory. Use explicit configuration, - inventory, role, and collection paths for local checks. Do not recreate the - operational controller layout or install system-wide tooling for that purpose. -- Retain the historical playbook and proof-of-concept role. Role, collection, - application, and asset repositories remain read-only during this feature. -- Deployment and paid provider execution occur only during the approved sprint - with the selected controller, target, credentials, and session. Do not - substitute an arbitrary session or broaden the host limit. +- Execute the stages below in order, rereading each stage's listed policies. + Check working trees first, preserve unrelated work, validate, and commit each + completed unit with its stated plain-English intent. +- Do not deploy to an operational controller or target, connect to the target, + populate actual credentials, or run a real pipeline session. +- Missing controller, SSH, Vault, and session inputs are not blockers. Express + unknown required values through commented instructions and variable references. +- Keep playbooks limited to hosts and roles. No new tasks, wrapper, probes, + runtime assertions, or automated test harness is required. +- Use existing workspace `.venv/bin` executables for local Ansible checks. + Core 2.21.3 is installed; a missing `PATH` entry or `/etc/ansible` directory + does not mean tooling is missing. Explicit local paths suffice. +- Reuse passing validation whose inputs are unchanged. Optional diagnostics + and `ansible-doc` documentation rendering are not acceptance gates. +- Keep credential values out of plaintext files, Git, commands, and logs. + Do not create fake keys or an empty Vault file to satisfy inventory references. -## Starting State and Preconditions +## Starting State and Repository Scope -- Feature 0015 passed independent review without findings. Its pipeline commit - is `51566afcf69083cdbb5b5718f2544c86b079001c`; the reviewed public collection - commit is `46913446765e86ecfb29ebad9955d579ef6a4c7e`. -- `rakestraw-ansible` starts clean on `main` at - `d86ec29a9cbd821fead248736171b5cdd8440ad6`. Inspect and preserve any subsequent - changes before adapting this plan. -- Root `requirements.yml` already has an unpinned Gitea `rakestraw.dnd` entry - alongside unrelated collections. The historical playbook is - `playbooks/apps/dnd.yml`; it invokes `pipeline_dnd` on `automation-dnd`. -- Host `pipeline-dnd` already connects as `eric` to - `pipeline-dnd.rakestrawhome.com`. Preserve its connection settings. -- `inventory/host_vars/pipeline-dnd/narratio.yml` contains the legacy campaign - and S3 variables to migrate. -- The selected collection still declares version `0.1.0`. An installed - release with that number is not sufficient evidence of the selected content. -- This plan is approved, with this revision permitting local preparation before - operational inputs are resolved. Do not invent those inputs; resolve them - at the stage that needs them. -- Local tooling is already installed and verified: Ansible Core 2.21.3 in - `/home/eric/Workspace/ansible-dnd/.venv`. Use its `bin/ansible`, - `bin/ansible-playbook` and `bin/ansible-galaxy`, or prepend - that `bin` directory to `PATH`. No activation is required for absolute paths. +- Feature 0015 is complete and independently reviewed. +- Consumer `716a80e` added the initial runbook; `12b2462` committed the new + playbook, host configuration, and dependencies. Inspect intervening commits; + do not repeat or discard completed work. +- Public dependency installation, target-only host listing, and syntax checking + passed for the prepared consumer configuration. The missing option descriptions + in three roles affect `ansible-doc` only and are not part of this feature. +- `rakestraw-ansible` owns remaining comments/runbook changes. The workspace + owns status and review handoff. All other repositories remain read-only. +- No credentials, controller access, target access, or session selection is + needed to complete this plan. -## Repository Scope - -- `rakestraw-ansible`: requirements, playbook, host configuration, encrypted - credentials, and `docs/pipeline-dnd.md` runbook linked from README. -- Workspace: feature status, implementation progress, and review handoff. -- Other repositories: read-only compatibility and application references. - -## Stage 1: Establish Local Preparation and Record Deployment Inputs - -Completion: complete (consumer `716a80ef68208537dacfa5f0f22259bf42ae6279`) +## Stage 1: Finish Consumer Comments and Usage Documentation ### Objective and Owning Repository -Record known and pending deployment inputs in `rakestraw-ansible` and close the reviewed -Feature 0015 in the workspace. +Make the existing committed consumer configuration self-explanatory in +`rakestraw-ansible` and remove live execution as a completion requirement. ### Required Policy Review -Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, -`docs/ansible.md`, `docs/testing.md`, `docs/planning.md`, the Feature 0016 -roadmap, and `rakestraw-ansible/README.md` before this stage. +Before this stage, read `AGENTS.md`, `docs/orientation.md`, +`docs/conventions.md`, `docs/ansible.md`, `docs/testing.md`, +`docs/planning.md`, the Feature 0016 roadmap, and the consumer README/runbook. ### Preconditions -The plan is approved and repositories have been checked. Local tooling is -available; no remote access or credential/session input is required. +Inspect the current consumer changes and the two completed commits above. +Local Ansible tools are available. There are no remote prerequisites. ### Required Changes -1. Create `rakestraw-ansible/docs/pipeline-dnd.md` and link it from README. - Record the local validation setup and known operational details. List the - operational controller/checkout, effective paths, Vault selection, and - acceptance session as pending where unknown, with their consuming stage. - Do not fabricate completed checks or turn missing values into blockers here. -2. Confirm local Ansible Core is at least 2.18 and the public suite - repositories and Galaxy are reachable. Inspect effective configuration - without dumping decrypted host variables. Existing repository defaults are - `/etc/ansible/inventory`, `/etc/ansible/roles`, and `/etc/ansible/collections`; - document these as operational defaults, not required local directories. -3. Record the reported target SSH rejection as a Stage 3 prerequisite. Defer - target Ubuntu and sudo checks until authorized access is available. Repeated - attempts with the same rejected identity do not advance local preparation. -4. Mark Feature 0015 `done` in its roadmap and workspace status, citing its - reviewed commits and the successful independent standalone/collection - integration runs. Leave historical remediation records unchanged. Mark - Feature 0016 `in_progress` when execution begins. - -### Validation and Completion Criteria - -Local tooling works and the runbook distinguishes known facts from pending -operational inputs, assigning each to Stage 3 or Stage 4. -Run `git diff --check` in both changed repositories. No deployment or provider -request is necessary for this stage. - -### Commit - -Consumer: `Document the D&D deployment context`. -Workspace: `Close storage review and begin host deployment`. - -## Stage 2: Prepare the Consumer Playbook and Inventory - -Completion: complete (consumer `12b2462`) - -### Objective and Owning Repository - -Make `rakestraw-ansible` a complete consumer of the reviewed suite snapshot. - -### Required Policy Review - -Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, -`docs/ansible.md`, `docs/testing.md`, `docs/planning.md`, the Feature 0016 -roadmap, `ansible-collection-dnd/README.md`, its `requirements.yml`, and the -Stage 1 runbook before this stage. - -### Preconditions - -Stage 1 is complete. Production credentials and remote access are not required. - -### Required Changes - -1. Add `playbooks/pipeline/dnd.yml` in the roadmap's exact minimal shape: - one play targeting `pipeline-dnd` with only `rakestraw.dnd.pipeline` in its - roles list. No play variables, tasks, or play-level privilege escalation. -2. Replace the legacy contents of - `inventory/host_vars/pipeline-dnd/narratio.yml` with the roadmap's public - inventory example: operator `eric`, the confirmed WhisperX URL, - `narratio_campaigns_default_campaign_id: dilfs`, and the four Vault-backed - `dnd_pipeline_secrets` mappings. Remove obsolete `narratio_s3_*` and - `narratio_default_campaign_id` settings. Preserve unrelated intervening edits - and the separate connection file. Do not copy suite defaults into inventory. -3. Document `inventory/host_vars/pipeline-dnd/vault.yml` and its four required - `vault_dnd_*` names. Keep the inventory references unresolved until the - operator provisions that encrypted file in Stage 3. Do not create an empty - Vault file or substitute credentials to make local checks appear complete. -4. Pin the existing `rakestraw.dnd` Git entry in root `requirements.yml` to - `46913446765e86ecfb29ebad9955d579ef6a4c7e` with its existing public URL. -5. Copy the five `roles` entries from that collection's compatibility manifest, - retaining role names, public Gitea URLs, `scm: git`, and `v0.1.0` tags. - Reconcile native collection entries to `ansible.posix: 2.2.0` and - `community.general: 13.3.0`, without duplicates. Preserve unrelated - dependencies; report demonstrated conflicts rather than silently changing - the reviewed suite pins. -6. Extend the runbook with the concrete commands for dependency installation, - collection refresh, Vault selection, deployment, fresh login, and native - validation used below. Link upstream Narratio v2.1.1 operator documentation. +1. Retain `playbooks/pipeline/dnd.yml`, which contains only the `pipeline-dnd` + host and `rakestraw.dnd.pipeline` role. Retain reviewed dependency pins and + known host values. Compare against the revised roadmap; correct only actual + discrepancies, without adding suite-default copies or extra play content. +2. In `inventory/host_vars/pipeline-dnd/narratio.yml`, add the roadmap's comments + beside `dnd_pipeline_secrets`: define the four referenced `vault_dnd_*` + variables in an encrypted `vault.yml` before eventual deployment. Retain + unresolved references. Do not obtain secret values or create a Vault file. +3. Revise `docs/pipeline-dnd.md` to describe repository preparation as the + completed deliverable. Clearly label installation, credential provisioning, + and deployment commands as future operator instructions, not steps the + implementation agent must execute. Remove operational prerequisite checklists + and session-acceptance requirements from this feature's completion path. +4. Retain useful concise future-use instructions: the pinned collection refresh, + Vault variable names and selection options, and minimal playbook invocation. + Link application documentation for later pipeline operation rather than + maintaining a live-session test script or demanding session selection now. +5. Preserve connection inventory, unrelated dependencies, and the historical + proof-of-concept playbook/role. ### Validation -Parse edited plaintext YAML and run `git diff --check`. In isolated controller -dependency paths, install from public sources with -`ansible-galaxy install -r requirements.yml`. Configure role and collection -destinations through Ansible configuration/environment; avoid a role-only -custom path that would skip the collection section of the mixed manifest. -Successful installation and playbook syntax checking are the dependency -validation boundary. `ansible-doc` renders documentation and is not required. +Run `git diff --check` in `rakestraw-ansible` and review the changed documentation +and comments. Reuse the recorded successful public installation, syntax, and +host-listing results if only comments/documentation change. No runtime tests +or repeated dependency installation are necessary for those edits. -For local checks, use the workspace executables and the checked-out inventory. -Configure isolated role and collection paths explicitly. From the consumer -checkout, run: +If executable configuration changes, rerun only affected local checks using +the workspace tools and explicit local inventory/dependency paths: ```text ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --list-hosts ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --syntax-check ``` -Host listing and syntax checking do not deploy or establish that credentials -resolve. Run them without evaluating or printing secret values; do not supply -fake Vault variables. If an existing unrelated encrypted inventory file needs -an unavailable password, preserve it and use a disposable local inventory with -only the target alias and its non-secret host variables for syntax checking. -Report that narrower scope explicitly; full encrypted-inventory validation -belongs to Stage 3. Do not change operational inventory to bypass encryption. -Do not impose -whole-repository lint changes on the established minimal-playbook style. +Use previously installed isolated public dependencies when available. If needed, +install from `requirements.yml` into disposable local role/collection paths. +Do not use operational `/etc/ansible` destinations or local sibling role +sources. Keep unresolved credentials untouched; local syntax checking does +not claim they resolve. If unrelated encrypted inventory prevents a local +check, use a disposable inventory containing the target alias and non-secret +variables, and report that narrower validation scope. ### Completion Criteria -Only the intended host is selected. Public dependencies resolve without local -sibling checkouts, local syntax checking succeeds within its recorded scope, and the -diff contains no plaintext secrets. +The required public files exist, unknown values have explanatory comments, +local validation evidence is sufficient, and the consumer changes are committed. +No deployment or real credential provisioning is required. ### Commit -`Add the D&D pipeline deployment playbook`. +`Document the remaining D&D deployment inputs`. -## Stage 3: Deploy and Verify Clean Convergence +## Stage 2: Record the Repository Preparation Handoff ### Objective and Owning Repository -Deploy from the operational controller; record results in the consumer runbook. +Record Feature 0016 implementation readiness for independent review in the +workspace, scoped entirely to committed repository content. ### Required Policy Review -Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, -`docs/ansible.md`, `docs/testing.md`, `docs/planning.md`, the Feature 0016 -roadmap, and `rakestraw-ansible/docs/pipeline-dnd.md` before this stage. +Before this stage, read `AGENTS.md`, `docs/orientation.md`, +`docs/conventions.md`, `docs/testing.md`, `docs/planning.md`, and the revised +Feature 0016 roadmap. ### Preconditions -Stage 2 passed. The operational controller/checkout and delivery workflow are -identified, and target SSH access and the Vault mechanism are available. -Deliver committed changes through that workflow without overwriting a dirty -checkout. Acceptance-session selection is not a prerequisite for this stage. +Stage 1 has passed local validation and is committed. No remote prerequisites. -### Required Changes and Validation +### Required Changes -Before installation, confirm the operational controller's Ansible version and -effective inventory, role, and collection paths. The target must support the -roles' Ubuntu 26.04 contract and required sudo operations; ordinary connection, -fact gathering, role platform guards, and privileged tasks establish these -conditions. Separate preflight probes are optional troubleshooting, not gates. -Have the operator create `inventory/host_vars/pipeline-dnd/vault.yml` encrypted -in full, defining the four roadmap credential variables. Commit ciphertext -with intent `Add encrypted D&D host credentials` after checking the diff -exposes no plaintext. Rerun host listing and syntax validation with the real -encrypted inventory and recorded Vault options before convergence. +1. Update `docs/status.md` and this plan's progress notes with the consumer + commit and local validation evidence. Describe real deployment as separately + deferred operator work, not a blocker or incomplete acceptance criterion. +2. Keep Feature 0016 `in_progress` pending independent review. Report any actual + repository issues; do not mark it `done` solely because implementation ended. +3. Confirm the feature's owning working trees are clean after committing. + Preserve any unrelated changes and identify them instead of discarding them. -1. Install root requirements at the confirmed controller paths. Explicitly - refresh only the D&D collection from its selected commit, even when `0.1.0` - is already installed: +### Validation and Completion Criteria - ```text - ansible-galaxy collection install git+https://gitea.maximumdirect.net/eric/ansible-collection-dnd.git,46913446765e86ecfb29ebad9955d579ef6a4c7e --force --no-deps - ``` - - Use the configured collection path; dependencies come from the manifest. - Do not force-update unrelated unpinned collections. Confirm role resolution - and playbook syntax on this operational controller as in Stage 2. -2. Run `ansible-playbook playbooks/pipeline/dnd.yml --limit pipeline-dnd` - using the recorded inventory, Vault, and any required sudo-password options. - Immediately repeat the identical command after success. Require - `changed=0`, `failed=0`, and no unreachable hosts on the repeat. -3. On the target, execute - `sudo -u narratio /usr/local/bin/narratio config validate`. - Testing-overlay validation is optional; it has already passed suite review - and is not changed by this deployment. The managed default campaign should make an - explicit campaign selector unnecessary. -4. Start a fresh login as `eric` and execute the same default native validation - without switching user, demonstrating usable supplementary group access. -5. Record consumer commit, target, date, repeat recap, and native validation - results. Diagnose failures at the owning boundary; report required changes - to roles/applications separately rather than expanding this feature. - -### Completion Criteria - -The actual host converges cleanly and both runtime identities validate the -configuration. Run `git diff --check` on the evidence/runbook changes. +Run `git diff --check`. Evidence points to committed consumer content and +appropriate local checks. The handoff makes no claims of live deployment or +runtime acceptance. No paid request or host access is necessary. ### Commit -`Record successful D&D host deployment`. - -## Stage 4: Exercise One Real Session - -### Objective and Owning Repository - -Demonstrate production processing and publication; record acceptance in -`rakestraw-ansible/docs/pipeline-dnd.md`. - -### Required Policy Review - -Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, -`docs/testing.md`, `docs/planning.md`, the Feature 0016 roadmap, the runbook, -and Narratio v2.1.1's `docs/config.md`, `docs/cli.md`, and -`docs/operations.md` before this stage. Follow the sibling application's -`AGENTS.md` and development reading guide when consulting its repository. - -### Preconditions - -Stage 3 passed. Actual credentials, the operator-selected campaign/session, -session file, and audio are accessible. The exercise must demonstrate real -transcription and publication, not merely resume an already complete run. -Do not erase existing state to create a fixture. - -### Required Changes and Validation - -1. Under either supported runtime identity (`narratio` or `eric` with effective - runtime-group membership), run: - - ```text - narratio session validate SESSION_ID --campaign CAMPAIGN --session SESSION_FILE - narratio run SESSION_ID --campaign CAMPAIGN --session SESSION_FILE - narratio session artifacts SESSION_ID --remote --campaign CAMPAIGN --session SESSION_FILE - ``` - - Substitute the operator-recorded values. Production is the default; an - operator-selected testing profile is also valid if the exercise covers the - agreed processing and publication path. Record the selected profile. - The session document remains operational input; no session-management task - belongs in the playbook. Observe transcription, processing, and successful - publication. Do not force reprocessing or reduce the selected artifact set - merely to bypass a failure. -2. Observe a representative published transcript or recap at the configured - destination through the existing storage/operator interface. Successful - cleanup may remove local workspace content; local survival is not required. -3. Record identity, input location, runtime user, completion outcome, and - non-secret publication evidence. Avoid committing raw provider logs or - credential-bearing URLs. Do not repeat paid work just to duplicate evidence - or expand acceptance into a restore/locking test matrix. - -### Completion Criteria - -A real session completes and its published output is observable. Record any -provider/application blocker and leave acceptance incomplete if it cannot -finish. Run `git diff --check` on the runbook changes. - -### Commit - -`Record the first published D&D session`. - -## Stage 5: Prepare the Independent Review Handoff - -### Objective and Owning Repository - -Record completed implementation evidence in the workspace. - -### Required Policy Review - -Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, -`docs/testing.md`, `docs/planning.md`, and the Feature 0016 roadmap before -this stage. - -### Preconditions - -Record progress at any stopping point, including a live-operation blocker. -Full implementation handoff requires Stages 1–4 and actual deployment/session -evidence; a partial handoff must explicitly identify uncompleted stages. - -### Required Changes and Validation - -Update `docs/status.md` and this plan's progress notes with owning commits, -runbook link, validation outcomes, and deviations. Keep Feature 0016 -`in_progress` pending independent review. Report any additional complexity -and its authorizing requirement. Run `git diff --check` and confirm changed -repositories are clean after committing. Do not rerun role integration -matrices or paid sessions without a new change or failure requiring it. - -### Completion Criteria - -Evidence is recorded and implementation is ready for independent review. - -### Commit - -`Record the D&D deployment implementation`. +`Record D&D playbook preparation for review`. ## Final Validation and Handoff -Report dependency revision, consumer commits, controller/target, public -dependency and syntax results, immediate clean convergence, native validation -for both identities, and session publication evidence. Distinguish completed -work from operator-dependent blockers. Do not mark Feature 0016 `done` before -independent review. +Report owning commits, changed consumer files, comments explaining missing +values, and local validation performed or reused. Feature review evaluates the +committed playbook/configuration, not operational deployment. No push is +required unless separately requested. ## Progress Notes -- Stage 1 is complete in consumer commit - `716a80ef68208537dacfa5f0f22259bf42ae6279`. The linked deployment runbook - records the Ansible Core 2.21.3 local toolchain, the effective repository - defaults, successful public-source reachability, and the inputs pending for - live deployment and session acceptance. -- Feature 0015 is closed after independent review found no remediation issue. - Its reviewed pipeline commit is - `51566afcf69083cdbb5b5718f2544c86b079001c`; collection fixture, squash, and - merge commits are `cf3de016ee09edf91afc3bc1addfd926d4a172ad`, - `77f67adbc0f700e88bc15554e02052eaa22f1033`, and - `46913446765e86ecfb29ebad9955d579ef6a4c7e`. -- Stage 2 local consumer preparation is complete. The rejected target SSH identity, - operational controller details, and Vault workflow/values remain Stage 3 - prerequisites. The acceptance session/audio selection remains a Stage 4 - prerequisite. No live deployment has begun. -- Stage 2 consumer changes are committed in `12b2462`. Installation from the - public manifest, host selection, and syntax checks passed. The former - `ansible-doc` gate was unnecessary and has been removed: missing argument - descriptions prevent documentation rendering, not the demonstrated role - loading. The prepared consumer diff was reviewed and whitespace checks passed - before committing. Resume at Stage 3 when its operational inputs are - available. No dependency pin change is needed. - -## Non-blocking Follow-up - -The pinned Notarius, Scriptorium, and Narratio `v0.1.0` roles lack option -descriptions required by `ansible-doc` documentation rendering. Consider adding -descriptions in a separate role documentation change and normal later release. -This is not a Feature 0016 acceptance criterion or a reason to change its pins. +The earlier consumer preparation is committed in `716a80e` and `12b2462`. +This revised two-stage plan finishes comments and documentation, then records +handoff. Earlier live-deployment stages and their open questions are withdrawn +from Feature 0016. No live deployment has been performed. ## Open Questions -Questions 1–2 gate Stage 3 only. Question 3 gates Stage 4 only. None prevents -local preparation in Stages 1–2. - -1. Which controller/checkout executes deployment and how do committed changes - reach it? Recommended: use the existing operational controller and its - established `/etc/ansible/` layout, keeping one operational inventory. - Alternative: a dedicated checkout with explicit inventory and dependency - paths, if the operator prefers isolation and records those paths consistently. -2. Which Vault identity/password mechanism is available, and can the operator - populate all four values? Recommended: reuse the controller's established - Vault workflow and populate the encrypted file directly. Alternative: an - interactive Vault password prompt for this file if no identity is established. - Both keep passwords outside Git; neither requires secrets in chat. -3. Which campaign/session/audio supplies acceptance? Recommended: an - operator-selected unprocessed `dilfs` session with accessible audio, matching - the retained default campaign and avoiding forced reprocessing. Alternative: - a selected session from another deployed campaign with an explicit selector. - Record the session file and audio location before execution; do not invent - sample production input. +None required for this repository-only implementation. Unknown API keys and +future operational inputs are documented for the operator, not requested now. diff --git a/docs/status.md b/docs/status.md index 595acc3..ffe6d6b 100644 --- a/docs/status.md +++ b/docs/status.md @@ -4,10 +4,10 @@ Last updated: 2026-09-21 Current milestone: Operational D&D pipeline deployment -Current focus: Resume Feature 0016 at Stage 3 once controller, SSH, and Vault -inputs are available. Stage 2 is committed in consumer `12b2462`. The -unnecessary `ansible-doc` gate has been removed; role documentation omissions -are a separate non-blocking follow-up. +Current focus: Finish Feature 0016 repository preparation under the revised +two-stage plan: comments for unknown host values and concise operator docs, +then commit and hand off for review. Controller deployment, target convergence, +and session execution are outside this feature; no operational input blocks it. Feature 0015 is complete after independent review found no remediation issue. | ID | Deliverable | Owner | State | Evidence or next step | @@ -36,7 +36,7 @@ Feature 0015 is complete after independent review found no remediation issue. | F0013 | Publish the first coordinated suite release | all Ansible repositories and workspace | done | Public [`rakestraw.dnd` 0.1.0 release](https://gitea.maximumdirect.net/eric/ansible-collection-dnd/releases/tag/v0.1.0); documentation remediation completed in pipeline `78afffe` and collection `5e695f3` | | F0014 | Enroll existing pipeline operators | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `c13f726` and collection import `c0b0ebd` passed their complete integration suites and independent review with no remediation findings | | F0015 | Enable D&D S3 storage and publishing | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `51566afcf69083cdbb5b5718f2544c86b079001c`, collection fixture `cf3de016ee09edf91afc3bc1addfd926d4a172ad`, collection squash `77f67adbc0f700e88bc15554e02052eaa22f1033`, and collection merge `46913446765e86ecfb29ebad9955d579ef6a4c7e` passed the complete repository suites and independent review with no remediation findings | -| F0016 | Deploy the D&D pipeline to `pipeline-dnd` | rakestraw-ansible and workspace | in_progress | Stage 1: consumer `716a80e`; Stage 2: consumer `12b2462`, with public installation, host selection, and syntax checks passed; resume Stage 3 when operational inputs are available | +| F0016 | Prepare the D&D deployment playbook for `pipeline-dnd` | rakestraw-ansible and workspace | in_progress | Consumer preparation is in `716a80e` and `12b2462`; revised [plan](roadmap/implementation.md) finishes comments/docs and repository handoff only; previous live-deployment stages withdrawn | | A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` | | R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review |