diff --git a/docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md b/docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md index 2fdc858..b3589ba 100644 --- a/docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md +++ b/docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md @@ -1,6 +1,6 @@ # Feature 0014: Enroll Existing Pipeline Operators -Status: in_progress +Status: done ## Summary diff --git a/docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md b/docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md new file mode 100644 index 0000000..10cb18e --- /dev/null +++ b/docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md @@ -0,0 +1,392 @@ +# Feature 0015: Enable D&D S3 Storage and Publishing + +Status: ready + +## Summary + +Make the D&D pipeline operational with Narratio's current S3 storage, +publication, caching, cleanup, and restore capabilities. The pipeline role +will select the established Backblaze B2 destination, reference two typed +object-storage credentials delivered through its existing secrets directory, +publish the useful transcript and D&D artifact outputs, and retain maximum S3 +audio caching. + +The standalone Narratio role already exposes Narratio v2.1.0's complete +configuration surface for this behavior. This feature supplies D&D suite +policy through that existing interface; it does not add another storage +implementation or credential mechanism. + +## Problem or Opportunity + +The pipeline role currently installs Narratio v2.1.0 and configures the full +processing graph, but deliberately leaves Narratio storage and publishing at +their application-level defaults. Its README consequently describes storage, +publishing, and end-to-end external execution as deferred. The resulting suite +can validate configuration and process local state, but it does not select the +remote archive or publish the outputs needed for an operational deployment. + +The prior test-machine configuration records the intended Backblaze B2 +endpoint, bucket, prefix, credential names, publication set, and post-publish +workspace cleanup. Narratio's current contract has since replaced the legacy +credential field names and secrets configuration with ConfigKit typed +references and a conventional sibling `secrets/` directory. The Ansible suite +already implements that new credential-delivery boundary for Audita and +Notarius, so S3 needs only to join the same path. + +## Governing Policies + +- [The Ansible house style](../../ansible.md) requires suite policy to flow + through the standalone role's field-specific interface and optional fields + to remain application-owned unless the suite deliberately selects them. +- [The testing house style](../../testing.md) requires representative + public-interface and native-application validation without duplicating AWS + SDK, ConfigKit, or Ansible module behavior. +- [The MVP roadmap](../mvp.md) makes the pipeline role responsible for + cross-application configuration and end-to-end validation. +- [Feature 0011](0011-managed-pipeline-credential-delivery.md) establishes the + managed Narratio secrets directory and typed credential-forwarding policy. +- [Feature 0012](0012-collection-assembly.md) makes the standalone pipeline + repository authoritative and imports reviewed changes through public + Gitea. +- [Feature 0016](0016-deploy-dnd-pipeline-to-pipeline-dnd.md) consumes this + feature's reviewed interface for the first real-host deployment. + +## Scope + +- Add suite-owned pipeline variables for the selected Narratio workspace, + S3, cache, and publish settings. +- Select the S3 backend with these D&D deployment defaults: + + | Setting | Suite default | + | --- | --- | + | Endpoint | `https://s3.us-east-005.backblazeb2.com` | + | Bucket | `rakestraw-assets` | + | Root prefix | `dnd` | + | Region | `us-east-1` | + | Force path style | `true` | + | Access-key reference | `OBJECT_STORAGE_KEY_ID` | + | Secret-key reference | `OBJECT_STORAGE_KEY` | + +- Pass the two credential reference names to the standalone Narratio role. + Their values use the existing `dnd_pipeline_secrets` map and managed + `/usr/local/etc/narratio/secrets/` directory. +- Enable S3 audio caching explicitly so the suite retains Narratio's maximum + available audio reuse. +- Enable run publication and publish these complete native output rules: + + | Source | Required | + | --- | --- | + | `narratio.transcript.final` | `true` | + | `narratio.transcript.final_trimmed` | `true` | + | `narratio.transcript.final_markdown` | `true` | + | `narratio.transcript.final_trimmed_markdown` | `true` | + | `narratio.artifact.session_recap` | `true` | + | `narratio.artifact.hostile_rumors` | `true` | + | `narratio.artifact.deranged_rumors` | `false` | + +- Let Narratio derive each destination from the built-in transcript path or + configured artifact output path rather than duplicating those paths in the + publish list. +- Enable workspace cleanup only after Narratio records successful run + publication. Leave spool-audio deletion under Narratio's existing default + because the suite has selected no policy requiring source-audio removal. +- Keep the production and `testing` profile storage and publishing behavior + identical. The existing overlay remains limited to model-profile selection. +- Document the four credentials normally required by the operational suite: + `AUDITA_LLM_API_KEY`, `OPENROUTER_API_KEY`, + `OBJECT_STORAGE_KEY_ID`, and `OBJECT_STORAGE_KEY`. +- Document the operator-visible publication, restore, S3-session-audio, lock, + caching, and cleanup capabilities enabled by the selected configuration, + while linking Narratio's canonical CLI and operations documentation for + command details. +- Preserve every accepted Notarius lane as its existing + `narratio.extraction.` source. Future Narratio artifact + definitions can consume those sources during analyze without first making + the extraction lanes separate published outputs. +- Adapt the existing pipeline integration scenario to resolve synthetic S3 + credentials and validate one explicitly supplied representative storage and + publishing configuration through Narratio's native interface without + contacting object storage. +- Import the reviewed public pipeline commit into the collection through the + established squashed subtree workflow, adapt the collection's artifact-level + fixture to supply the added credential names, and validate the assembled + artifact. + +## Non-goals + +- Implementing an S3 client, publication engine, remote commit format, + restore mechanism, publish-lock mechanism, or audio cache in Ansible. + Narratio owns all of those behaviors. +- Placing credential values in role defaults, templates, documentation, + tests, ordinary inventory, or the generated pipeline configuration. +- Adding a second secret-delivery mechanism, environment wrapper, service + unit, credential synchronizer, or AWS configuration-file manager. +- Managing Backblaze buckets, object lifecycle policy, application keys, + retention policy, versioning, encryption, quotas, or network access. +- Managing per-session `session.yml` files or selecting local versus S3 audio + input for an individual session. +- Running a paid or networked publication test from a repository-local test + suite. Feature 0016 owns the representative live end-to-end exercise. +- Publishing every Notarius extraction lane as a named current output. The + lanes remain available as analyze inputs independently of publication; stable + remote destinations require an explicit operator need. +- Setting Narratio's optional `notarius.chunk_cache` override. The managed + Notarius configuration remains the single owner of its ordinary `auto` + cache policy, and Narratio v2.1 already requests refresh for forced + extraction. +- Adding notification providers. Narratio v2.1 supports only its existing + `noop` notification mode. +- Releasing new role or collection versions as part of this implementation. + +## Target End State + +The standalone pipeline role exposes these suite variables with the selected +defaults: + +```yaml +dnd_pipeline_narratio_workspace_cleanup_after_publish: true +dnd_pipeline_narratio_storage_backend: s3 +dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets +dnd_pipeline_narratio_storage_s3_root_prefix: dnd +dnd_pipeline_narratio_storage_s3_region: us-east-1 +dnd_pipeline_narratio_storage_s3_endpoint: >- + https://s3.us-east-005.backblazeb2.com +dnd_pipeline_narratio_storage_s3_force_path_style: true +dnd_pipeline_narratio_storage_s3_access_key_id: OBJECT_STORAGE_KEY_ID +dnd_pipeline_narratio_storage_s3_secret_access_key: OBJECT_STORAGE_KEY +dnd_pipeline_narratio_cache_s3_audio: true +dnd_pipeline_narratio_publish_enabled: true +dnd_pipeline_narratio_publish_upload_run: true +dnd_pipeline_narratio_publish_outputs: + - source: narratio.transcript.final + required: true + - source: narratio.transcript.final_trimmed + required: true + - source: narratio.transcript.final_markdown + required: true + - source: narratio.transcript.final_trimmed_markdown + required: true + - source: narratio.artifact.session_recap + required: true + - source: narratio.artifact.hostile_rumors + required: true + - source: narratio.artifact.deranged_rumors + required: false +``` + +Each value is declared in the pipeline argument specification and passed +directly to its corresponding `narratio_*` variable in the pipeline role's +Narratio dependency. Narratio's existing root template renders the native +configuration. No new pipeline fragment or production task is required. + +The ordinary generated configuration contains only the credential names: + +```yaml +storage: + backend: s3 + s3: + access_key_id: OBJECT_STORAGE_KEY_ID + secret_access_key: OBJECT_STORAGE_KEY +``` + +Consumers supply the two values beside the existing model-provider values +through encrypted inventory: + +```yaml +dnd_pipeline_secrets: + AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}" + OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}" + OBJECT_STORAGE_KEY_ID: "{{ vault_dnd_object_storage_key_id }}" + OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}" +``` + +The example defines references only; no roadmap, role, or test contains real +credential values. The existing credential task writes the selected entries +with the suite's established group-authorized secret modes. Omitted map keys +remain unmanaged, as defined by Feature 0011. + +With valid credentials, Narratio can accept either local or S3 audio selected +by each session file, cache S3 audio beneath `/var/cache/narratio`, publish an +immutable run and the selected stable outputs, use remote publication locks, +restore committed state, and clean the published workspace. These are native +Narratio commands and lifecycle rules rather than additional Ansible tasks. + +The ten accepted Notarius outputs continue to enter Narratio's artifact +catalog as `narratio.extraction.*` sources. Analyze-stage Scriptorium artifacts +may reference those sources directly in their input mappings. Feature 0015 +does not yet add such artifact definitions, but its publication choices do not +remove, rename, or constrain this future composition path. + +## Repository Ownership + +- `ansible-role-dnd-pipeline` owns the D&D suite variables, opinionated + defaults, dependency mappings, consumer documentation, and representative + integration scenario. +- `ansible-role-narratio` remains the generic owner of installation and native + pipeline rendering. Its reviewed v2.1.0 interface already covers every + configuration field required by this feature and should remain unchanged. +- `ansible-collection-dnd` owns the faithful pipeline subtree import, the + unified consumer dependency manifest, and artifact-boundary validation. +- `ansible-workspace-dnd` owns this roadmap, the later implementation plan, + status, and independent review record. +- `narratio` is a read-only source of storage, publication, credential, + caching, cleanup, restore, and CLI contracts. +- `rakestraw-ansible` remains unchanged until Feature 0016 consumes this + interface on `pipeline-dnd`. + +## Constraints and Settled Decisions + +- The S3 endpoint, bucket, prefix, region, and path-style values are deliberate + D&D suite defaults. They are ordinary variables and remain overrideable by + inventory. +- Credential references are also policy defaults, but credential values have + no defaults. Operators must supply matching entries through + `dnd_pipeline_secrets` or another environment source supported by Narratio. +- The two credential-reference variables are nullable as a pair. Setting both + to `null` omits the references and preserves Narratio's AWS SDK provider-chain + path; configuring only one remains invalid under Narratio's native contract. +- The role does not assert that the secrets map contains particular keys. + Narratio's native loader owns typed-reference resolution, and external + environment values remain a supported higher-precedence source. +- `dnd_pipeline_narratio_publish_outputs` is one complete native list. A caller + replacing it assumes responsibility for the entire selected publication + set; the role does not merge rules. +- Publish destinations remain omitted where Narratio can derive them. The + configured source identity and artifact output path remain the single source + of truth. +- Workspace cleanup is gated by Narratio's own successful-publication + evidence. Ansible does not independently inspect remote state or remove + runtime content. +- S3 audio caching is enabled, and ordinary cleanup preserves the cache. + Explicit operator cache clearing remains a Narratio CLI operation. +- The testing profile does not silently disable Notarius, storage, + publication, cleanup, or credentials. It continues to differ only in the + already selected inexpensive Scriptorium model profile. +- S3 storage also enables Narratio's native restore and publication-lock + commands without another configuration surface. Ansible documentation may + describe availability but must not duplicate Narratio's detailed operating + manual. +- Notarius extraction-source availability and remote publication are separate + concerns. Analyze consumes `narratio.extraction.*` sources from the accepted + extraction catalog; `publish.outputs` selects only which sources also receive + stable published destinations. + +## Simplest Viable Implementation + +Add the selected variables to the pipeline defaults and argument specification, +then map them into the existing `rakestraw.narratio` dependency. Update the +pipeline README to document the policy, credential names, complete-list +replacement semantics, and native capabilities now available. Do not add a +template, task, handler, fact, probe, or storage-specific validation layer. + +In the existing pipeline integration driver, extend the synthetic secrets map +with object-storage values and explicitly supply representative S3 and publish +variables. Use `narratio config validate` and its secret-safe effective +configuration output to prove that the public inputs compose into a valid +configuration. Retain the immediate clean repeat and secret non-disclosure +checks. Do not call a real S3 service or assert that the representative values +equal current role defaults. + +Push the reviewed pipeline commit, import that exact public revision into the +collection through `git subtree pull --squash`, and extend only the +collection-level fixture inputs needed for native validation. The collection +suite should continue to prove artifact installation and one representative +assembled convergence rather than duplicate the detailed pipeline matrix. + +## Rationale and Alternatives + +Implementing storage behavior in Ansible would duplicate a mature native +application boundary. Narratio already owns credential resolution, AWS SDK +provider construction, immutable commits, verification, locks, restore, +caching, and cleanup. Passing declarative configuration through the generic +role is both smaller and more reliable. + +Using the AWS SDK's ambient provider chain would avoid named secret files, but +the initial deployment has explicit Backblaze application credentials and the +suite already has a suitable managed typed-secret path. Keeping the two +credential-reference variables nullable preserves the ambient-provider +alternative: consumers can set both suite variables to `null` when their +deployment uses another provider. + +Copying destinations into every publish rule would make the output layout more +visible locally, but it would duplicate paths already owned by Narratio's +built-in sources and the configured artifact definitions. Native destination +derivation keeps path coordination automatic. + +Deleting spool audio after publication would minimize transient disk usage, +but no such retention policy has been selected, and local-audio sessions may +depend on that content. Leaving the field omitted preserves Narratio's default +while enabling the explicitly requested S3 audio cache. + +Publishing all ten Notarius extraction lanes would create stable remote output +contracts that have not been requested. It is also unnecessary for the stated +future use: analyze-stage artifacts can consume `narratio.extraction.*` +directly. The immutable published run retains the processing result, while the +selected transcript and human-facing artifact outputs match the known +operational workflow. + +## Acceptance Criteria + +- The pipeline role declares and documents every suite variable shown in the + target state, with matching argument-spec entries and direct Narratio + dependency mappings. +- The two credential-reference variables accept paired `null` overrides so a + caller can deliberately select Narratio's ambient AWS SDK provider chain. +- The default effective Narratio configuration selects the specified + Backblaze-compatible S3 destination, typed credential names, S3 audio cache, + successful-publication workspace cleanup, run upload, and seven output + rules. +- The publish list relies on Narratio's native destination derivation and + retains `deranged_rumors` as optional while the other six outputs are + required. +- Spool-audio deletion remains unset by the pipeline role, and the testing + profile introduces no storage or publication override. +- `dnd_pipeline_secrets` remains a generic empty-by-default mapping; the role + documents all four expected operational entries without embedding any + credential value. +- No new production task, template, handler, fact, state file, wrapper, or + storage-validation command is introduced. +- The standalone Narratio role remains unchanged because its v2.1.0 interface + already renders the required native fields and preserves null omission. +- The existing pipeline integration scenario explicitly supplies + representative S3 and publish inputs, resolves synthetic values for all four + typed credentials, passes Narratio's native configuration validation, and + exposes no synthetic value in captured output. +- The immediate identical second pipeline application reports `changed=0` + and `failed=0`; syntax checking, production-profile lint, the complete + standalone integration suite, and `git diff --check` pass. +- The exact reviewed pipeline commit is pushed to public Gitea before the + collection imports it through the documented remote-only squashed subtree + workflow. +- The collection fixture supplies the added synthetic credential names, and + its existing build, isolated installation, FQCN resolution, native + validation, and clean-repeat suite passes without a duplicate S3 behavior + test. +- No real object-storage request, paid model request, application repository + change, asset change, release tag, or published artifact is required. +- Pipeline documentation links operators to Narratio's canonical procedures + for S3 session audio, publish, restore, locks, status, and cleanup and no + longer describes storage and publishing as deferred. +- Documentation distinguishes accepted Notarius extraction sources from + published outputs and confirms that future analyze-stage artifacts may use + `narratio.extraction.` inputs without publishing those lanes. + +## Dependencies and Follow-up Work + +Features 0009, 0011, 0012, 0013, and 0014 are prerequisites. In particular, +Feature 0011 supplies the credential directory and Feature 0014 supplies the +human-operator access needed for the eventual live deployment. + +[Feature 0016](0016-deploy-dnd-pipeline-to-pipeline-dnd.md) follows this work. +It will put the four actual values in encrypted host inventory, pin the exact +reviewed suite revision in `rakestraw-ansible`, converge `pipeline-dnd`, and +exercise one real session through remote publication. Bucket lifecycle policy, +backup policy, credential rotation, additional publish outputs, automated +scheduling, and cleanup of the proof-of-concept deployment remain separate +future work. + +A later artifact-development feature may add D&D-specific Narratio artifact +definitions whose Scriptorium input mappings consume selected +`narratio.extraction.*` sources. That feature should define each artifact's +purpose, inputs, prompt, output contract, and publication choice; it does not +require a storage-interface change. diff --git a/docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md b/docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md new file mode 100644 index 0000000..2bc6cff --- /dev/null +++ b/docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md @@ -0,0 +1,278 @@ +# Feature 0016: Deploy the D&D Pipeline to `pipeline-dnd` + +Status: proposed + +## Summary + +Add a minimal playbook and host inventory to `rakestraw-ansible` that apply +`rakestraw.dnd.pipeline` to the existing `pipeline-dnd` host. Configure the +host through `host_vars`, enroll `eric` as the initial human operator, keep +credential values in Ansible Vault, and prove the complete pipeline through a +real repeatable deployment and representative D&D session. + +[Feature 0015](0015-dnd-s3-storage-and-publishing.md) must first provide the +storage and publishing behavior needed for an operational end-to-end pipeline. +Feature 0016 consumes that reviewed interface; it does not invent a temporary +storage contract. + +## Problem or Opportunity + +The collection and its application roles are implemented and installable, but +the operational Ansible repository does not yet have a playbook for deploying +the new collection architecture. Its existing D&D playbook targets a retired +inventory name and invokes the proof-of-concept local `pipeline_dnd` role. + +The current `pipeline-dnd` host inventory already records the connection +identity and useful legacy Narratio storage values. A production-shaped first +deployment now needs to express the new role's public interface, deliver +secrets safely, reconcile the controller's dependency manifest, and validate +the actual host rather than extending the role repositories with another +synthetic scenario. + +## Scope + +- Add `rakestraw-ansible/playbooks/pipeline/dnd.yml`, targeting the existing + `pipeline-dnd` inventory host and applying only + `rakestraw.dnd.pipeline`. +- Keep the playbook limited to its host selection and role list. Put all + deployment choices in `inventory/host_vars/pipeline-dnd/`. +- Configure the external WhisperX transcription endpoint as + `https://transcription.ai.rakestrawhome.com/transcribe`. +- Enroll the existing `eric` account through + `dnd_pipeline_operator_users`. +- Map the role's `dnd_pipeline_secrets` interface to host-specific Vault + variables. Store credential values only in an Ansible Vault-encrypted host + variables file. +- Carry the storage and publishing settings selected by Feature 0015 through + its final public pipeline interface. Preserve the operational intent of the + existing Backblaze B2 endpoint, bucket, prefix, region, and path-style + settings where they remain applicable. +- Reconcile existing legacy Narratio host variables so they cannot silently + override the suite policy selected by the new pipeline role. +- Update `rakestraw-ansible/requirements.yml` so a clean controller can install + the reviewed collection snapshot, all five standalone role dependencies, + and required native collections from one manifest. +- Validate controller dependency resolution, playbook syntax, initial + convergence, immediate clean convergence, operator access, native Narratio + configuration, and one representative end-to-end pipeline run on the real + host. +- Document the operational invocation, Vault prerequisite, initial operator, + and expected validation path in `rakestraw-ansible`. + +## Non-goals + +- Implementing storage or publishing behavior. Feature 0015 owns that + prerequisite. +- Changing application or role behavior merely to accommodate this + deployment. +- Replacing the existing inventory alias with `pipeline.dnd`, changing DNS, + or moving connection variables out of the current host inventory. +- Adding a wrapper role, pre-task layer, service, timer, scheduler, web UI, or + unattended session-ingestion mechanism. +- Creating or managing the `eric` account, its login policy, SSH access, or + sudo policy. +- Adding another production host, high availability, backup policy, or + disaster-recovery automation. +- Removing the proof-of-concept local `pipeline_dnd` role or its historical + playbook before the new deployment has demonstrated operational parity. +- Publishing a new coordinated suite release solely as part of this + deployment feature. + +## Target End State + +`rakestraw-ansible/playbooks/pipeline/dnd.yml` is an intentionally small +consumer playbook equivalent to: + +```yaml +--- +- hosts: + - pipeline-dnd + roles: + - rakestraw.dnd.pipeline +... +``` + +The existing inventory selects the host's connection identity. A dedicated +host-variable file selects the pipeline's non-secret public inputs, including +the WhisperX endpoint, the initial operator list, and the storage and +publishing values exposed by Feature 0015. It maps runtime credential names to +Vault-backed variables without embedding credential values in the playbook or +ordinary plaintext inventory, for example: + +```yaml +dnd_pipeline_operator_users: + - eric + +dnd_pipeline_whisperx_transcribe_url: >- + https://transcription.ai.rakestrawhome.com/transcribe + +dnd_pipeline_secrets: + AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}" + OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}" + OBJECT_STORAGE_KEY_ID: "{{ vault_dnd_object_storage_key_id }}" + OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}" +``` + +The corresponding credential variables reside in a tracked file encrypted in +full with Ansible Vault. The Vault password or identity mechanism remains +outside Git and outside this feature's managed content. + +The controller dependency manifest selects one reviewed, publicly reachable +suite snapshot containing Features 0014 and 0015. It also installs that +snapshot's exact standalone role and native collection dependencies. A clean +controller therefore does not depend on sibling development checkouts or +previously installed roles. + +On `pipeline-dnd`, the role converges the full application suite and its D&D +configuration. The `narratio` service identity can run the pipeline, and +`eric` can run it after starting a login session that observes the granted +supplementary group membership. A representative session can be transcribed, +processed, stored, and published through the configured production path. An +immediate identical Ansible run reports no changes. + +## Repository Ownership + +- `rakestraw-ansible` owns the consumer playbook, host inventory, encrypted + host credentials, controller dependency manifest, and deployment runbook. +- `ansible-role-dnd-pipeline` owns the suite's public variables and the + opinionated cross-application behavior consumed by the deployment. +- `ansible-collection-dnd` owns the assembled `rakestraw.dnd.pipeline` role + snapshot and consumer-facing compatibility manifest. +- The standalone application roles continue to own installation and native + application configuration. Feature 0016 consumes their reviewed releases or + commits without reimplementing their behavior. +- `ansible-workspace-dnd` owns this roadmap, the later implementation plan, + progress tracking, and independent review record. +- Application and asset repositories are read-only sources of runtime + contracts and deployed content for this feature. + +## Constraints and Settled Decisions + +- Feature 0015 is a prerequisite and must settle, implement, and review the + storage and publishing interface before Feature 0016 becomes ready for an + implementation sprint. +- The target is the inventory alias `pipeline-dnd` as currently defined in + `rakestraw-ansible`, not a new `pipeline.dnd` alias. +- The playbook follows the repository's established style: hosts and roles + only. Facts remain at Ansible's normal default, privilege escalation remains + task-owned by the roles, and deployment variables belong in host inventory. +- `eric` is the initial human operator and must already exist on the managed + host. The pipeline role grants only additive runtime-group membership. +- API credentials use the existing typed `dnd_pipeline_secrets` interface and + Ansible Vault. Plaintext secret values must not enter Git, command lines, + logs, examples, or ordinary variable files. +- The public Gitea repositories are the sole dependency sources. The + deployment must not reference this workspace's sibling checkout paths. +- The dependency manifest must select an immutable release when Feature 0015 + produces one. Otherwise, it must pin the exact reviewed public Git commit; + it must not track an unbounded branch for the production deployment. +- The collection artifact does not itself install arbitrary Git-hosted role + dependencies. The operational manifest must therefore include the five + standalone roles as well as the collection and its native collection + dependencies. +- The legacy host variables must be compared with the final Feature 0015 + interface. Applicable values move to the new suite configuration, while + obsolete or conflicting variables are removed rather than left to win by + inventory precedence. +- The initial real-host run is an operational acceptance exercise. It should + test observable pipeline behavior and convergence, not module task traces or + the current text of defaults. + +## Simplest Viable Implementation + +After Feature 0015 is reviewed, select its exact public collection revision +and use the associated compatibility manifest to update the controller's +single root requirements file. Add the minimal playbook and one ordinary +host-variable file for public inputs. Add one fully encrypted host Vault file +for the two credential values, populated by the operator without exposing its +plaintext during implementation or review. + +Install or update dependencies from the root manifest, confirm that Ansible +resolves `rakestraw.dnd.pipeline`, syntax-check the playbook, and converge only +`pipeline-dnd`. Start a fresh operator login after enrollment, validate the +native Narratio configuration under the intended runtime identity, rerun the +same play immediately to confirm clean convergence, and process one selected +session through the complete Feature 0015 publishing path. Record concise +operational instructions and results; do not add a second test harness around +the real deployment. + +## Rationale and Alternatives + +Keeping the playbook nearly declarative makes inventory the durable record of +what differs about this host and allows ordinary Ansible precedence to remain +the customization mechanism. Embedding endpoint, operator, storage, or secret +mapping variables in the playbook would make a one-host play harder to reuse +and would diverge from the established `rakestraw-ansible` style. + +A local wrapper role could hide the collection interface, but it would create +another layer with no independent behavior. Applying the collection role by +FQCN proves the same consumer path that future hosts should use. + +Tracking collection `main` would be convenient during development, but it +would make a later controller bootstrap select an unreviewed compatibility +set. An exact public commit retains development flexibility until the next +release while still making this deployment reproducible. + +Running the real-host deployment before Feature 0015 would establish only an +installation demonstration: the pipeline still could not complete its +required storage and publishing path. Making Feature 0015 an explicit +prerequisite avoids temporary inventory variables and gives the acceptance +run a meaningful end-to-end boundary. + +## Acceptance Criteria + +- Feature 0015 is complete and independently reviewed before Feature 0016 + implementation begins. +- `playbooks/pipeline/dnd.yml` targets only `pipeline-dnd`, applies only + `rakestraw.dnd.pipeline`, and contains no deployment variables or procedural + setup. +- Host inventory sets the confirmed WhisperX endpoint, enrolls only `eric`, + and supplies the final Feature 0015 storage and publishing inputs. +- `dnd_pipeline_secrets` maps the required typed runtime keys to host Vault + variables, and the actual values exist only in a valid Ansible + Vault-encrypted file. +- No plaintext credential value is present in any tracked revision, diff, + command example, validation output, or deployment documentation. +- Legacy Narratio storage values are either represented through the current + pipeline interface or deliberately retired; no stale high-precedence host + variable can override the selected suite configuration accidentally. +- The root requirements manifest preserves unrelated existing dependencies + and installs the exact reviewed public collection snapshot, its five + standalone role dependencies, and its required native collections without + using local repository paths. +- On a controller bootstrapped from that manifest, `ansible-doc` resolves + `rakestraw.dnd.pipeline` and each role dependency, and the new playbook + passes syntax checking. +- The initial limited playbook run against `pipeline-dnd` completes + successfully using the encrypted inventory and ordinary sudo-capable remote + account. +- The immediate identical second run reports `changed=0` and `failed=0`. +- Narratio's native configuration validation succeeds under the intended + runtime identity, and a fresh session for `eric` can access and invoke the + shared pipeline through runtime-group membership. +- One operator-selected D&D session completes transcription, processing, + storage, and publishing, and its expected published output is observable at + the configured destination. +- The operational documentation explains dependency installation, Vault use, + deployment invocation, the login-session membership caveat, convergence + verification, and representative pipeline invocation without disclosing + credentials. +- The historical proof-of-concept role and playbook remain available during + the initial rollout; their later retirement requires a separate explicit + decision. + +## Dependencies and Follow-up Work + +[Feature 0015](0015-dnd-s3-storage-and-publishing.md) is the immediate next unit +of work and a hard prerequisite. It selects the pipeline interface, +credentials, and observable publishing boundary that Feature 0016 will deploy. + +Features 0013 and 0014 provide the public collection installation path and +existing-operator enrollment required here. Features 0009 and 0011 provide +the shared filesystem and managed credential-delivery policies. + +After the real deployment demonstrates parity, a later cleanup feature may +retire the proof-of-concept `pipeline_dnd` role, its old playbook, and any +obsolete inventory values. Service scheduling, unattended ingestion, +additional human operators, backup policy, and additional hosts also remain +separate work. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 4c3dc69..efe5c9d 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,26 +1,29 @@ -# Implementation Plan 0014: Enroll Existing Pipeline Operators +# Implementation Plan 0015: Enable D&D S3 Storage and Publishing Status: ready Target executor: GPT-5.6-Terra Feature roadmap: -`docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md` +`docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md` Active plan location: `docs/roadmap/implementation.md` ## Objective -Add one empty-by-default pipeline interface that enrolls explicitly selected -existing human accounts in the suite's runtime group. Implement the behavior -with one account-existence lookup and one additive membership task, adapt the -existing operator integration scenario, and import the reviewed result into -the collection from public Gitea. +Configure the authoritative D&D pipeline role to use the selected Backblaze +S3-compatible archive, typed object-storage credential references, maximum S3 +audio caching, immutable run publication, selected transcript and D&D artifact +outputs, and cleanup of successfully published workspace state. Validate the +configuration through Narratio's native interface, import the exact reviewed +pipeline commit into the collection from public Gitea, and leave the result +ready for the real-host deployment in Feature 0016. -The Feature 0014 roadmap remains authoritative for scope, rationale, target -state, and acceptance criteria. This plan does not authorize human-account -creation or removal, exact-list reconciliation, filesystem-policy changes, or -a new release. +The Feature 0015 roadmap remains authoritative for scope, rationale, target +state, and acceptance criteria. This plan does not authorize a second storage +implementation, real object-storage calls from repository tests, per-session +configuration management, stable publication of every Notarius extraction +lane, or changes to the generic Narratio role. ## Execution Rules @@ -33,37 +36,49 @@ a new release. worktree changes or branch divergence rather than overwriting them. - Do not make unresolved product or architecture decisions during execution; stop and report the decision required. -- Keep the production behavior to the roadmap's exact direct task flow: one - `ansible.builtin.getent` loop followed by one `ansible.builtin.user` loop. -- Do not add an assertion, registered fact, handler, command, shell operation, - explicit empty-list condition, membership comparison, post-change query, - sidecar state, or account abstraction. -- Trust `getent` to fail for a missing key and `user` to converge additive - membership. Do not reproduce either module's checks. -- Test the public behavior by adapting the existing temporary-operator - scenario. Do not add a missing-user test, inspect `/etc/passwd` or - `/etc/group`, assert task results, or duplicate the detailed test in the - collection. -- Run each stage's validation before committing it. Do not proceed after a - validation failure. +- Implement storage and publication entirely through the standalone Narratio + role's existing field-specific variables. Do not add a production task, + template, handler, fact, probe, command, wrapper, state file, or validation + layer to the pipeline role. +- Keep `dnd_pipeline_secrets` as the existing generic empty-by-default map. + Add no real, generated, placeholder, or fallback credential value to role + defaults, templates, documentation, tests, logs, or ordinary inventory. +- Keep the S3 access-key and secret-key reference variables nullable as a + pair. Do not add an Ansible assertion for pairing or secret presence; + Narratio owns native configuration validation and AWS provider selection. +- Do not configure `narratio_spool_delete_audio_after_publish` or + `narratio_notarius_chunk_cache`. Preserve the roadmap's single-owner + decisions for spool retention and Notarius cache policy. +- Treat accepted Notarius outputs and published outputs as separate + interfaces. Preserve `narratio.extraction.` sources for future + analyze inputs without adding publication rules or new artifacts for them. +- Exercise explicitly supplied representative values in the detailed pipeline + integration scenario. Do not assert that those values are current defaults + or duplicate Narratio's own storage and publication test matrix. +- Do not make a real S3 request or paid model request during repository-local + validation. +- Run every stage's validation before completing the stage. Do not proceed + after a validation failure. - Commit each completed stage with the stated plain-English intent. Push the authoritative pipeline commit before importing it into the collection. -- Use only the public Gitea pipeline URL for the subtree update; do not import +- Use only the public Gitea pipeline URL for the subtree update. Do not import from the local sibling repository. ## Starting State and Preconditions -- Features 0009, 0011, 0012, and 0013 are complete. -- This roadmap and implementation plan have been reviewed and committed in the - workspace repository before execution begins. -- The completed Feature 0013 closeout commits `eda361f` in the workspace and - `5e695f3` in the collection are pushed to their respective `origin/main` - branches before implementation begins. Preserve the Git-generated subtree - squash commit that accompanies `5e695f3`. -- `ansible-role-dnd-pipeline` starts clean on `main` at `78afffe`, equal to - `origin/main`. -- `ansible-collection-dnd` starts clean on `main` at `5e695f3`, equal to - `origin/main` after the prerequisite push. +- Features 0009, 0011, 0012, 0013, and 0014 are complete. +- The approved Feature 0015 roadmap and this implementation plan are committed + in the workspace repository before the sprint begins. +- `ansible-role-dnd-pipeline` starts clean on `main` at + `c13f726a0a846180a1aa95d110b8bbc801edaae5`, equal to `origin/main`. +- `ansible-collection-dnd` starts clean on `main` at + `c0b0ebd13e92199f3d94e138c6cc50d1ece5dadd`, equal to `origin/main`. +- `ansible-role-narratio` is clean at + `292d8669e7aad5cfe80b0c469d22b6177e8cf0f1`, equal to `origin/main`, and its + `v0.1.0` role release contains the Narratio v2.1.0 field-specific interface + required by this feature. +- The sibling Narratio application repository is clean at public release + `v2.1.0`. It is a read-only contract source during this sprint. - The workspace `.venv` supplies the tested `ansible-playbook`, `ansible-galaxy`, and `ansible-lint` commands used by the repository test drivers. @@ -71,27 +86,31 @@ a new release. test-time sudo is available for the existing integration fixtures. - Git credentials can push the pipeline and collection commits to their public Gitea origins. +- No actual Backblaze or model-provider credential is required or available to + the implementation. Tests use conspicuously synthetic values only. ## Repository Scope -- `ansible-role-dnd-pipeline` owns the variable, argument specification, - account lookup, additive enrollment task, README, and detailed integration - behavior. -- `ansible-collection-dnd` receives the reviewed pipeline commit only through - its established squashed subtree and validates the assembled artifact. -- `ansible-workspace-dnd` records implementation progress and the review - handoff after the two implementation repositories are complete. -- All standalone application-role repositories, application source - repositories, asset repositories, and `rakestraw-ansible` are out of scope - and must remain unchanged. +- `ansible-role-dnd-pipeline` owns the suite defaults, argument specification, + Narratio dependency mappings, consumer documentation, and detailed + representative integration behavior. +- `ansible-collection-dnd` owns the artifact-level fixture update, faithful + squashed subtree import, assembled-artifact validation, and resulting public + collection snapshot. +- `ansible-workspace-dnd` records the implementation handoff after the two + implementation repositories are complete. +- `ansible-role-narratio`, all other standalone roles, application sources, + asset repositories, and `rakestraw-ansible` may be inspected but must remain + unchanged. -## Stage 1: Implement Existing-Operator Enrollment +## Stage 1: Implement D&D Storage and Publication Policy ### Objective -Give the standalone pipeline role its minimal public interface for enrolling -existing human accounts and prove the behavior through the existing operator -integration scenario. +Expose the approved suite-level S3 and publishing inputs through the +authoritative pipeline role, map them directly into Narratio, document the +operator contract, and prove representative composition without networked +storage access. ### Required Policy Review @@ -103,17 +122,22 @@ Before beginning this stage, read: - `docs/ansible.md` - `docs/testing.md` - `docs/planning.md` -- `docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md` +- `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md` +- `ansible-role-dnd-pipeline/README.md` +- `../narratio/AGENTS.md` +- `../narratio/docs/development.md` +- `../narratio/docs/config.md` +- `../narratio/docs/operations.md` ### Preconditions -- The workspace planning commit is present. -- `ansible-role-dnd-pipeline` is clean at `78afffe` on `main`, is not behind +- The workspace planning commit containing this plan is present. +- `ansible-role-dnd-pipeline` is clean at the starting revision, is not behind or diverged from `origin/main`, and contains no unrelated changes. -- Its Narratio dependency still creates the selected runtime group before the - pipeline role's ordinary tasks execute. -- The existing integration driver still contains the temporary operator and - representative group-access scenario described by the roadmap. +- The current `rakestraw.narratio` dependency mapping and existing integration + driver still match the structures reviewed while preparing this plan. +- Narratio v2.1.0 still accepts the roadmap-selected storage, publish, cleanup, + cache, and typed credential-reference fields. ### Owning Repository @@ -121,38 +145,102 @@ Before beginning this stage, read: ### Required Changes -1. Add `dnd_pipeline_operator_users: []` beside the existing runtime identity - defaults in `defaults/main.yml`. -2. Add the matching `meta/argument_specs.yml` entry with `type: list`, - `elements: str`, and the same required/default-backed convention used by - the role's other public variables. -3. Add one unprivileged `ansible.builtin.getent` task near the beginning of - `tasks/main.yml`. Loop directly over `dnd_pipeline_operator_users`, use the - `passwd` database and each username as `key`, and rely on the module's - default missing-key failure. Do not register its result or add a separate - assertion. -4. Immediately follow it with one `become: true` `ansible.builtin.user` task - over the same list. Set `name` to the selected account, - `groups` to `dnd_pipeline_runtime_group`, `append: true`, and - `state: present`. Do not set or manage any other account property. -5. Revise the README variable reference and shared-execution section. Replace - the consumer-owned membership example and deferred wording with the new - interface, existing-account prerequisite, additive non-removal behavior, - authorization consequence, and new-login-session caveat. -6. Adapt `tests/integration` rather than adding another scenario. Create its - temporary operator before the initial role application without membership - in the runtime group, pass that username through - `dnd_pipeline_operator_users` for the baseline and repeated applications, - and retain the existing native configuration, secret-boundary, and shared - cache exercises under that identity. Reorder cleanup as necessary so any - final baseline convergence occurs while the selected fixture account still - exists, then remove the fixture account. -7. Retain the existing clean-repeat assertion. Do not add a negative - missing-user case or assertions about module internals, task traces, the - current default value, or raw account database entries. -8. Run the complete existing repository validation, commit the result, push - `main` to the public pipeline origin, and record the full pushed commit for - Stage 2. +1. Add the following exact suite defaults to `defaults/main.yml` near the + existing Narratio policy variables: + + ```yaml + dnd_pipeline_narratio_workspace_cleanup_after_publish: true + dnd_pipeline_narratio_storage_backend: s3 + dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets + dnd_pipeline_narratio_storage_s3_root_prefix: dnd + dnd_pipeline_narratio_storage_s3_region: us-east-1 + dnd_pipeline_narratio_storage_s3_endpoint: https://s3.us-east-005.backblazeb2.com + dnd_pipeline_narratio_storage_s3_force_path_style: true + dnd_pipeline_narratio_storage_s3_access_key_id: OBJECT_STORAGE_KEY_ID + dnd_pipeline_narratio_storage_s3_secret_access_key: OBJECT_STORAGE_KEY + dnd_pipeline_narratio_cache_s3_audio: true + dnd_pipeline_narratio_publish_enabled: true + dnd_pipeline_narratio_publish_upload_run: true + dnd_pipeline_narratio_publish_outputs: + - source: narratio.transcript.final + required: true + - source: narratio.transcript.final_trimmed + required: true + - source: narratio.transcript.final_markdown + required: true + - source: narratio.transcript.final_trimmed_markdown + required: true + - source: narratio.artifact.session_recap + required: true + - source: narratio.artifact.hostile_rumors + required: true + - source: narratio.artifact.deranged_rumors + required: false + ``` + + Keep every `dest` omitted. Narratio derives transcript destinations from + built-in sources and artifact destinations from the already managed + artifact output paths. +2. Add matching public entries to `meta/argument_specs.yml`. Use `type: bool` + for the cleanup, path-style, cache, enabled, and upload settings; `type: str` + for the backend and S3 string fields; and `type: list` for the complete + publish output list. The S3 child fields, including both credential + references, must be non-required so callers may set every S3 child to + `null` when deliberately selecting the local backend or set the credential + pair to `null` when selecting the AWS SDK provider chain. The remaining + concrete suite-policy fields follow the role's existing required/defaulted + argument-spec convention. +3. In the `rakestraw.narratio` dependency entry in `meta/main.yml`, map every + new suite variable directly to its same-purpose standalone variable: + `narratio_workspace_cleanup_after_publish`, the eight + `narratio_storage_*` fields, `narratio_cache_s3_audio`, + `narratio_publish_enabled`, `narratio_publish_upload_run`, and + `narratio_publish_outputs`. +4. Do not map `narratio_spool_delete_audio_after_publish`. Do not map + `narratio_notarius_chunk_cache`. Do not add a new import, fragment, + template, task, handler, assertion, or configuration-validation command. +5. Revise the README's variable reference and processing-policy sections to + document the selected S3 destination, paired nullable credential + references, complete-list publish semantics, native destination derivation, + maximum S3 audio caching, successful-publication workspace cleanup, and + deliberate omission of spool deletion. +6. Extend the README's managed-credential example to show + `OBJECT_STORAGE_KEY_ID` and `OBJECT_STORAGE_KEY` mapped from encrypted + inventory beside the existing Audita and OpenRouter entries. Explain that + values remain outside ordinary configuration and that setting both S3 + credential references to `null` selects Narratio's ambient AWS SDK provider + chain. +7. Replace the README's statement that storage and publishing are deferred + with concise operator-facing descriptions and public upstream links for S3 + session audio, publish, restore, publication locks, status, and cleanup. + Explain that accepted Notarius lanes remain + `narratio.extraction.` analyze inputs independently of whether + they have stable publish rules. +8. Adapt the existing `tests/integration` baseline rather than adding a new + scenario. Add two synthetic object-storage values to + `dnd_pipeline_secrets`; explicitly unset `OBJECT_STORAGE_KEY_ID` and + `OBJECT_STORAGE_KEY` alongside the existing provider variables when + invoking Ansible; and explicitly pass representative non-default S3, + cleanup, cache, and publish values through the new public variables. +9. Use representative values that exercise the interface without freezing the + defaults: for example an `.invalid` HTTPS endpoint and fixture bucket, + prefix, and region; intentional `false` values for cleanup, path style, and + S3 audio caching; the two configured credential reference names; enabled + run publication; and a short complete publish list containing one built-in + transcript source and `narratio.artifact.session_recap` with omitted + destinations. +10. Extend the existing secret-safe effective-configuration inspection only + enough to demonstrate the explicitly supplied representative storage + fields, typed credential names, falsey cache/cleanup choices, enabled + publication, and Narratio-derived destinations. Continue to use + `narratio config validate` as the native validity boundary. Do not inspect + role task results, call S3, or reproduce Narratio's validation rules. +11. Extend the current non-disclosure check to cover both synthetic + object-storage values. Preserve the complete existing integration matrix, + cleanup behavior, and immediate clean-repeat assertions. +12. Run the complete repository validation. Commit the valid result with the + stated plain-English message, push `main` to public Gitea, and record the + full pushed commit hash for Stage 3. ### Validation @@ -162,34 +250,36 @@ Run from the workspace root: (cd ansible-role-dnd-pipeline && tests/integration && git diff --check) ``` -Expected result: syntax and production-profile lint pass; the temporary -existing account is enrolled through the public role variable and completes -the existing representative operator-access scenario; native application -checks pass; the immediate identical role application reports `changed=0` -and `failed=0`; and no credential value appears in test output. +Expected result: syntax checking and production-profile lint pass; all five +applications remain valid; the explicitly supplied S3 and publish values +resolve through Narratio's native configuration interface; only reference +names appear in effective output; no synthetic credential value appears in +captured output; no external S3 or model request occurs; and the immediate +identical repeat reports `changed=0` and `failed=0`. ### Completion Criteria -- The public interface, argument specification, two direct tasks, and README - implement exactly the roadmap contract. -- The existing operator scenario uses the role interface rather than fixture - group enrollment. +- Defaults, argument specifications, dependency mappings, and documentation + implement exactly the Feature 0015 contract. +- No production task, template, fragment, validation command, or second secret + path was added. +- The representative integration scenario exercises public inputs rather than + snapshotting defaults and retains the existing clean-repeat boundary. - The full standalone integration suite and `git diff --check` pass. -- The role repository is clean after a plain-English commit. +- The pipeline repository is clean after the plain-English commit. - The exact commit is present on public `origin/main` and its full hash is available for the collection import. ### Commit -Commit intent: `Add optional pipeline operator enrollment` +Commit intent: `Enable D&D storage and publishing` -## Stage 2: Import the Reviewed Pipeline Change +## Stage 2: Prepare the Collection Credential Fixture ### Objective -Expose the reviewed operator interface through `rakestraw.dnd.pipeline` while -preserving the authoritative-standalone-repository and collection-artifact -boundaries. +Make the collection's artifact-level fixture compatible with the forthcoming +pipeline snapshot while its currently bundled role remains valid. ### Required Policy Review @@ -201,17 +291,93 @@ Before beginning this stage, read: - `docs/ansible.md` - `docs/testing.md` - `docs/planning.md` -- `docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md` +- `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md` - `ansible-collection-dnd/README.md` ### Preconditions -- Stage 1 is complete and its exact commit is publicly reachable from - `https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git`. -- `ansible-collection-dnd` is clean at `5e695f3` on `main`, is not behind or +- Stage 1 is complete, validated, committed, and pushed. +- `ansible-collection-dnd` is clean at the starting revision, is not behind or diverged from `origin/main`, and contains no unrelated changes. -- The collection's current `roles/pipeline` subtree still corresponds to the - authoritative pipeline history through `78afffe`. +- Its top-level `tests/integration` fixture still supplies the two existing + synthetic provider credentials through `dnd_pipeline_secrets`. + +### Owning Repository + +`ansible-collection-dnd` + +### Required Changes + +1. Update only the collection-level `tests/integration` fixture outside + `roles/pipeline`. Add conspicuously synthetic values for + `OBJECT_STORAGE_KEY_ID` and `OBJECT_STORAGE_KEY` to its existing + `dnd_pipeline_secrets` input. +2. Explicitly unset those two environment variables beside the existing + Audita and OpenRouter variables when the fixture runs Ansible so ambient + controller state cannot replace the managed synthetic files. +3. Do not add storage assertions or duplicate the standalone pipeline's + detailed representative test. The collection fixture owns only enough + input to validate the assembled artifact after Stage 3. +4. Do not edit anything under `roles/pipeline`, alter dependency manifests, + or change collection release metadata. +5. Check the fixture's shell syntax and whitespace, then commit the compatible + preparation with the stated message. Do not push separately unless needed; + Stage 3 will push both collection commits after full artifact validation. + +### Validation + +Run from the workspace root: + +```text +(cd ansible-collection-dnd && bash -n tests/integration && git diff --check) +``` + +Expected result: the collection integration driver remains valid shell; the +diff changes only its generic managed-secret input and environment scrub; and +no subtree, dependency, or release file changed. Full semantic validation +occurs once against the imported snapshot in Stage 3. + +### Completion Criteria + +- The artifact-level fixture provides all four synthetic operational + credential names without exposing real values. +- No detailed storage behavior was duplicated in the collection. +- Only the collection-owned fixture changed. +- Shell syntax and `git diff --check` pass. +- The collection repository is clean after the plain-English commit. + +### Commit + +Commit intent: `Prepare collection storage credentials` + +## Stage 3: Import and Validate the Reviewed Pipeline Snapshot + +### Objective + +Expose the reviewed Feature 0015 behavior through `rakestraw.dnd.pipeline` +while preserving the public-remote subtree and collection-artifact boundaries. + +### Required Policy Review + +Before beginning this stage, read: + +- `AGENTS.md` +- `docs/orientation.md` +- `docs/conventions.md` +- `docs/ansible.md` +- `docs/testing.md` +- `docs/planning.md` +- `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md` +- `ansible-collection-dnd/README.md` + +### Preconditions + +- Stage 1's exact pipeline commit is publicly reachable from + `https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git`. +- Stage 2 is complete and committed in `ansible-collection-dnd`. +- The collection is clean at the Stage 2 commit and its bundled pipeline + subtree still corresponds to the authoritative pipeline history through + `c13f726a0a846180a1aa95d110b8bbc801edaae5`. ### Owning Repository @@ -220,27 +386,28 @@ Before beginning this stage, read: ### Required Changes 1. Set a shell variable to the full pushed Stage 1 commit hash and run the - documented remote-only update from the collection repository: + documented remote-only subtree update from the collection repository: ```text pipeline_revision= git subtree pull --prefix=roles/pipeline \ https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \ - "$pipeline_revision" --squash -m "Import pipeline operator enrollment" + "$pipeline_revision" --squash -m "Import D&D storage and publishing" ``` -2. Do not edit role behavior directly beneath `roles/pipeline`. The subtree - import should carry the new default, argument specification, tasks, README, - and standalone tests faithfully. -3. Do not change `galaxy.yml`, dependency manifests, collection-level tests, - or release documentation. No dependency or published version changes in - this feature, and the detailed behavior remains covered by Stage 1. -4. Run the existing collection integration suite against the built and - installed artifact. Commit no additional files after the subtree command - unless validation exposes a genuine collection-boundary issue within this - feature's scope. -5. Push the resulting collection history to `origin/main` and record the - resulting merge commit and Git-generated squashed subtree commit. +2. Do not edit role behavior directly beneath `roles/pipeline`. The import + must faithfully carry the reviewed defaults, argument specification, + dependency mappings, README, and detailed integration fixture. +3. Do not modify `galaxy.yml`, `requirements.yml`, release documentation, or + the published `v0.1.0` artifacts. Feature 0015 changes no standalone role or + native collection dependency and publishes no release. +4. Run the complete collection integration suite against the built and + installed artifact. Do not add collection-level assertions for the detailed + S3 configuration; native validation and clean convergence remain the + collection boundary. +5. If validation succeeds, push the Stage 2 fixture commit, the subtree merge + commit, and its Git-generated squash commit to `origin/main`. Record the + resulting commit hashes for Stage 4. ### Validation @@ -250,33 +417,34 @@ Run from the workspace root: (cd ansible-collection-dnd && tests/integration && git diff --check) ``` -Expected result: the collection builds; the archive contains the updated -pipeline role while retaining its development-file exclusions; isolated -dependency and artifact installation succeeds; the FQCN resolves; native -representative application checks pass; and the immediate repeat reports -`changed=0` and `failed=0`. +Expected result: the collection builds; its archive contains the updated +pipeline role while retaining development-file exclusions; isolated +dependency and artifact installation succeeds; the FQCN resolves; Narratio +loads its typed storage references from the synthetic credential files without +contacting S3; native representative application checks pass; and the +immediate repeat reports `changed=0` and `failed=0`. ### Completion Criteria -- Git history records a squashed subtree import from the exact public Stage 1 - commit with the stated plain-English merge message. +- Git history records the exact public Stage 1 revision through the documented + squashed subtree workflow and stated merge message. - No pipeline behavior was authored directly in the collection. -- The existing collection suite and `git diff --check` pass without an - unnecessary duplicate operator test. -- The collection is clean and its new history is present on `origin/main`. +- The complete collection suite and `git diff --check` pass. +- The collection repository is clean and its Stage 2 and Stage 3 history is + present on public `origin/main`. ### Commit -Commit intent: `Import pipeline operator enrollment` +Commit intent: `Import D&D storage and publishing` -The subtree command creates the stage's merge commit and its generated squash +The subtree command creates this stage's merge commit and its generated squash commit; do not add an empty or redundant manual commit. -## Stage 3: Record the Implementation Handoff +## Stage 4: Record the Implementation Handoff ### Objective -Record the completed implementation evidence without marking Feature 0014 +Record the completed implementation evidence without marking Feature 0015 done before independent review. ### Required Policy Review @@ -287,14 +455,15 @@ Before beginning this stage, read: - `docs/orientation.md` - `docs/conventions.md` - `docs/planning.md` -- `docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md` +- `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md` ### Preconditions -- Stages 1 and 2 are complete, committed, pushed, and validated. +- Stages 1 through 3 are complete, committed, pushed where required, and + validated. - Their full commit hashes and validation results are available. -- The workspace repository is clean apart from the expected progress-record - edits for this stage and is not behind or diverged from `origin/main`. +- The workspace repository is clean at the planning commit, is not behind or + diverged from `origin/main`, and contains no unrelated changes. ### Owning Repository @@ -302,15 +471,17 @@ Before beginning this stage, read: ### Required Changes -1. Change Feature 0014's roadmap status and status-table state to +1. Change Feature 0015's roadmap status and status-table state to `in_progress`, indicating implementation is complete but independent review remains outstanding. -2. Update the workspace current focus and Feature 0014 evidence with the exact - pipeline and collection commit hashes and the validation completed. -3. Replace this plan's Progress Notes with a concise record of the three stage +2. Update the workspace current focus and Feature 0015 evidence with the exact + pipeline commit, collection fixture commit, collection subtree merge and + squash commits, and validation completed. +3. Replace this plan's Progress Notes with a concise record of the four stage states and their commits. Do not copy command logs into the repository. -4. Do not mark the feature `done`, remove the active plan, or create - `docs/roadmap/remediation.md`; those are review and closeout decisions. +4. Do not mark the feature `done`, remove this active plan, change Feature + 0016's state, or create `docs/roadmap/remediation.md`; those are later review + and closeout decisions. 5. Commit the workspace handoff record with the stated plain-English message. ### Validation @@ -324,49 +495,39 @@ git diff --check Expected result: every repository is clean; the pipeline and collection are not ahead of or behind their origins; only the workspace handoff commit may be -ahead of `origin/main`; and the dashboard points to the exact implementation +ahead of `origin/main`; and the dashboard points to exact implementation evidence while retaining independent review as the next step. ### Completion Criteria -- Workspace status accurately identifies Feature 0014 as implemented and - awaiting review. -- The roadmap remains the durable target-state record and is not polluted with - execution logs. +- Workspace status accurately identifies Feature 0015 as implemented and + awaiting independent review. +- The roadmap remains the durable target-state record and contains no command + log or transient execution notes. - The active plan records concise implementation evidence and contains no false completion claim. - The workspace repository is clean after the handoff commit. ### Commit -Commit intent: `Record pipeline operator enrollment implementation` +Commit intent: `Record D&D storage and publishing implementation` ## Final Validation and Handoff -- Confirm Stage 1's complete standalone integration result and Stage 2's - complete collection integration result. +- Confirm Stage 1's complete standalone pipeline integration result and Stage + 3's complete collection integration result. - Run `./scripts/status` and confirm every repository is clean, with no unexpected branch divergence or untracked artifact. -- Report the full pipeline, collection merge, collection subtree-squash, and - workspace handoff commit hashes. -- Summarize any deviation from this plan, unresolved finding, or added - implementation complexity and identify the acceptance criterion that - required it. -- Do not mark Feature 0014 `done`; implementation results require independent +- Report the full pipeline commit, collection fixture commit, collection + subtree merge and squash commits, and workspace handoff commit. +- Confirm that `ansible-role-narratio`, all other standalone roles, + applications, assets, and `rakestraw-ansible` remained unchanged. +- Summarize any deviation from this plan, unresolved finding, or complexity + added beyond direct variable mapping and identify the acceptance criterion + that required it. +- Do not mark Feature 0015 `done`; implementation results require independent review first. ## Progress Notes -- Stage 1 complete: `ansible-role-dnd-pipeline` - `c13f726a0a846180a1aa95d110b8bbc801edaae5` implements optional enrollment - of existing operators; the complete standalone integration suite passed and - the commit is public on `origin/main`. -- Stage 2 complete: `ansible-collection-dnd` merge - `c0b0ebd13e92199f3d94e138c6cc50d1ece5dadd` imports the exact public pipeline - change through squash `d292297b6ad179e4664c01ad07cddaefff12f769`; - the complete collection integration suite passed and the history is public - on `origin/main`. -- Stage 3 complete: `ansible-workspace-dnd` - `bf81461de60001a0d8050e560383c6ddf533209f` records the implementation - evidence while Feature 0014 remains `in_progress` for independent review. -- Deviations: none. +- Planning complete; implementation has not begun. diff --git a/docs/roadmap/mvp.md b/docs/roadmap/mvp.md index 87d49be..3f8107c 100644 --- a/docs/roadmap/mvp.md +++ b/docs/roadmap/mvp.md @@ -370,6 +370,15 @@ application-role releases in one source repository. - The Narratio `testing` profile keeps the Notarius stage enabled and inherits the same D&D pipeline and extraction profile as production until a distinct testing extraction policy is selected. +- The D&D suite selects its Backblaze-compatible S3 archive, typed + `OBJECT_STORAGE_KEY_ID` and `OBJECT_STORAGE_KEY` references, maximum S3 audio + caching, run publication, human-facing transcript and artifact outputs, and + cleanup of successfully published workspace state. Feature 0015 owns the + exact overrideable defaults and publication set. +- Accepted Notarius lane outputs remain addressable inside Narratio as + `narratio.extraction.` sources. Future analyze-stage Scriptorium + artifacts may consume those sources directly; publishing an extraction lane + to a stable remote destination is an independent policy choice. ### Execution model @@ -389,7 +398,7 @@ later design decision changes them: - `/usr/local/etc/notarius/pipelines` for suite-owned imported Notarius pipeline definitions - `/usr/local/etc/scriptorium/config.yml` for Scriptorium configuration -- `/usr/local/etc/narratio/secrets` for future restricted credential material +- `/usr/local/etc/narratio/secrets` for suite-managed restricted credentials - `/usr/local/share/narratio/campaigns` for version-controlled campaign data - `/usr/local/share/narratio/prompt-library` for version-controlled prompts and schemas diff --git a/docs/status.md b/docs/status.md index 2347a71..e68c2ca 100644 --- a/docs/status.md +++ b/docs/status.md @@ -1,10 +1,10 @@ # Workspace Status -Last updated: 2026-09-18 +Last updated: 2026-09-21 -Current milestone: Existing pipeline operator enrollment +Current milestone: Operational D&D pipeline deployment -Current focus: Independently review the Feature 0014 implementation. +Current focus: Review the Feature 0015 staged implementation plan. | ID | Deliverable | Owner | State | Evidence or next step | | --- | --- | --- | --- | --- | @@ -30,7 +30,9 @@ Current focus: Independently review the Feature 0014 implementation. | F0011 | Deliver pipeline credentials and Notarius reuse | ansible-role-narratio and ansible-role-dnd-pipeline | done | Implemented and independently reviewed through remediation commit `de1b191` | | F0012 | Assemble the D&D collection | ansible-collection-dnd and ansible-role-dnd-pipeline | done | Pipeline `600eb21` assembled and validated through collection remediation commit `da67b34` | | F0013 | Publish the first coordinated suite release | all Ansible repositories and workspace | done | Public [`rakestraw.dnd` 0.1.0 release](https://gitea.maximumdirect.net/eric/ansible-collection-dnd/releases/tag/v0.1.0); documentation remediation completed in pipeline `78afffe` and collection `5e695f3` | -| F0014 | Enroll existing pipeline operators | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | in_progress | Review pipeline `c13f726a0a846180a1aa95d110b8bbc801edaae5` and collection `c0b0ebd13e92199f3d94e138c6cc50d1ece5dadd`; both complete integration suites passed | +| F0014 | Enroll existing pipeline operators | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `c13f726` and collection import `c0b0ebd` passed their complete integration suites and independent review with no remediation findings | +| F0015 | Enable D&D S3 storage and publishing | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | ready | [Roadmap](roadmap/features/0015-dnd-s3-storage-and-publishing.md) approved; prepare the staged implementation plan | +| F0016 | Deploy the D&D pipeline to `pipeline-dnd` | rakestraw-ansible and workspace | proposed | [Roadmap](roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md) drafted; Feature 0015 is a prerequisite | | A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` | | R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review |