ansible-workspace-dnd/docs/roadmap/implementation.md

Implementation Plan 0015: Enable D&D S3 Storage and Publishing

Status: ready

Target executor: GPT-5.6-Terra

Feature roadmap: docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md

Active plan location: docs/roadmap/implementation.md

Objective

Configure the authoritative D&D pipeline role to use the selected Backblaze S3-compatible archive, typed object-storage credential references, maximum S3 audio caching, immutable run publication, selected transcript and D&D artifact outputs, and cleanup of successfully published workspace state. Advance the pipeline to Narratio v2.1.1 so explicit publish requiredness composes with native destination derivation. Validate the configuration through Narratio's native interface, import the exact reviewed pipeline commit into the collection from public Gitea, and leave the result ready for the real-host deployment in Feature 0016.

The Feature 0015 roadmap remains authoritative for scope, rationale, target state, and acceptance criteria. This plan does not authorize a second storage implementation, real object-storage calls from repository tests, per-session configuration management, stable publication of every Notarius extraction lane, or changes to the generic Narratio role.

Execution Rules

• Execute stages in numeric order.
• Before each stage, confirm its preconditions and the working-tree state of its owning repository.
• Before each stage, reread every document listed in that stage's Required Policy Review. Do not rely on policy recalled from an earlier stage.
• Preserve unrelated and pre-existing changes. Stop and report unexpected worktree changes or branch divergence rather than overwriting them.
• Do not make unresolved product or architecture decisions during execution; stop and report the decision required.
• Implement storage and publication entirely through the standalone Narratio role's existing field-specific variables. Do not add a production task, template, handler, fact, probe, command, wrapper, state file, or validation layer to the pipeline role.
• Keep dnd_pipeline_secrets as the existing generic empty-by-default map. Add no real, generated, placeholder, or fallback credential value to role defaults, templates, documentation, tests, logs, or ordinary inventory.
• Keep the S3 access-key and secret-key reference variables nullable as a pair. Do not add an Ansible assertion for pairing or secret presence; Narratio owns native configuration validation and AWS provider selection.
• Do not configure narratio_spool_delete_audio_after_publish or narratio_notarius_chunk_cache. Preserve the roadmap's single-owner decisions for spool retention and Notarius cache policy.
• Treat accepted Notarius outputs and published outputs as separate interfaces. Preserve narratio.extraction.<output_key> sources for future analyze inputs without adding publication rules or new artifacts for them.
• Exercise explicitly supplied representative values in the detailed pipeline integration scenario. Do not assert that those values are current defaults or duplicate Narratio's own storage and publication test matrix.
• Do not make a real S3 request or paid model request during repository-local validation.
• Run every stage's validation before completing the stage. Do not proceed after a validation failure.
• Commit each completed stage with the stated plain-English intent. Push the authoritative pipeline commit before importing it into the collection.
• Use only the public Gitea pipeline URL for the subtree update. Do not import from the local sibling repository.

Starting State and Preconditions

• Features 0009, 0011, 0012, 0013, and 0014 are complete.
• The approved Feature 0015 roadmap and this implementation plan are committed in the workspace repository before the sprint begins.
• ansible-role-dnd-pipeline remains based on c13f726a0a846180a1aa95d110b8bbc801edaae5, equal to origin/main, with the interrupted Stage 1 edits preserved and uncommitted in README.md, defaults/main.yml, meta/argument_specs.yml, meta/main.yml, and tests/integration. Review and complete those edits; do not discard them.
• ansible-collection-dnd starts clean on main at c0b0ebd13e92199f3d94e138c6cc50d1ece5dadd, equal to origin/main.
• ansible-role-narratio is clean at 292d8669e7aad5cfe80b0c469d22b6177e8cf0f1, equal to origin/main, and its v0.1.0 role release contains the field-specific interface required by this feature and can install Narratio v2.1.1 through narratio_version.
• The sibling Narratio application repository is clean at public release v2.1.1, commit 442469b9f57e65ce17f09059609dd1a37cf8c3b6. That backward-compatible release corrects destination derivation with explicit publish requiredness and is a read-only contract source during the resumed sprint.
• The workspace .venv supplies the tested ansible-playbook, ansible-galaxy, and ansible-lint commands used by the repository test drivers.
• The local host remains the approved Ubuntu 26.04 proxy, and passwordless test-time sudo is available for the existing integration fixtures.
• Git credentials can push the pipeline and collection commits to their public Gitea origins.
• No actual Backblaze or model-provider credential is required or available to the implementation. Tests use conspicuously synthetic values only.

Repository Scope

• ansible-role-dnd-pipeline owns the suite defaults, argument specification, Narratio dependency mappings, consumer documentation, and detailed representative integration behavior.
• ansible-collection-dnd owns the artifact-level fixture update, faithful squashed subtree import, assembled-artifact validation, and resulting public collection snapshot.
• ansible-workspace-dnd records the implementation handoff after the two implementation repositories are complete.
• ansible-role-narratio, all standalone roles, application sources, asset repositories, and rakestraw-ansible may be inspected but must remain unchanged.

Stage 1: Implement D&D Storage and Publication Policy

Completion: complete (51566afcf69083cdbb5b5718f2544c86b079001c)

Objective

Expose the approved suite-level S3 and publishing inputs through the authoritative pipeline role, map them directly into Narratio, document the operator contract, and prove representative composition without networked storage access.

Required Policy Review

Before beginning this stage, read:

• AGENTS.md
• docs/orientation.md
• docs/conventions.md
• docs/ansible.md
• docs/testing.md
• docs/planning.md
• docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md
• ansible-role-dnd-pipeline/README.md
• ../narratio/AGENTS.md
• ../narratio/docs/development.md
• ../narratio/docs/config.md
• ../narratio/docs/operations.md

Preconditions

• The workspace planning commit containing the original approved plan and this resumed-plan revision are present.
• ansible-role-dnd-pipeline is based on the starting revision, is not behind or diverged from origin/main, and its only working-tree changes are the preserved interrupted Stage 1 edits identified in Starting State and Preconditions.
• The current rakestraw.narratio dependency mapping and existing integration driver still match the structures reviewed while preparing this plan.
• Narratio v2.1.1 remains publicly available and accepts the roadmap-selected storage, publish, cleanup, cache, and typed credential-reference fields, including rules with explicit required and omitted derivable dest.

Owning Repository

ansible-role-dnd-pipeline

Required Changes

1. Advance the existing application default to dnd_pipeline_narratio_version: v2.1.1. Do not change the standalone Narratio role release pin: its existing public interface can install this compatible application patch release.

2. Add the following exact suite defaults to defaults/main.yml near the existing Narratio policy variables:

   dnd_pipeline_narratio_workspace_cleanup_after_publish: true
   dnd_pipeline_narratio_storage_backend: s3
   dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets
   dnd_pipeline_narratio_storage_s3_root_prefix: dnd
   dnd_pipeline_narratio_storage_s3_region: us-east-1
   dnd_pipeline_narratio_storage_s3_endpoint: https://s3.us-east-005.backblazeb2.com
   dnd_pipeline_narratio_storage_s3_force_path_style: true
   dnd_pipeline_narratio_storage_s3_access_key_id: OBJECT_STORAGE_KEY_ID
   dnd_pipeline_narratio_storage_s3_secret_access_key: OBJECT_STORAGE_KEY
   dnd_pipeline_narratio_cache_s3_audio: true
   dnd_pipeline_narratio_publish_enabled: true
   dnd_pipeline_narratio_publish_upload_run: true
   dnd_pipeline_narratio_publish_outputs:
     - source: narratio.transcript.final
       required: true
     - source: narratio.transcript.final_trimmed
       required: true
     - source: narratio.transcript.final_markdown
       required: true
     - source: narratio.transcript.final_trimmed_markdown
       required: true
     - source: narratio.artifact.session_recap
       required: true
     - source: narratio.artifact.hostile_rumors
       required: true
     - source: narratio.artifact.deranged_rumors
       required: false
   

   Keep every dest omitted. Narratio derives transcript destinations from built-in sources and artifact destinations from the already managed artifact output paths.

3. Add matching public entries to meta/argument_specs.yml. Use type: bool for the cleanup, path-style, cache, enabled, and upload settings; type: str for the backend and S3 string fields; and type: list for the complete publish output list. The S3 child fields, including both credential references, must be non-required so callers may set every S3 child to null when deliberately selecting the local backend or set the credential pair to null when selecting the AWS SDK provider chain. The remaining concrete suite-policy fields follow the role's existing required/defaulted argument-spec convention.

4. In the rakestraw.narratio dependency entry in meta/main.yml, map every new suite variable directly to its same-purpose standalone variable: narratio_workspace_cleanup_after_publish, the eight narratio_storage_* fields, narratio_cache_s3_audio, narratio_publish_enabled, narratio_publish_upload_run, and narratio_publish_outputs.

5. Do not map narratio_spool_delete_audio_after_publish. Do not map narratio_notarius_chunk_cache. Do not add a new import, fragment, template, task, handler, assertion, or configuration-validation command.

6. Revise the README's application-version reference, upstream links, variable reference, and processing-policy sections for Narratio v2.1.1 and document the selected S3 destination, paired nullable credential references, complete-list publish semantics, native destination derivation, maximum S3 audio caching, successful-publication workspace cleanup, and deliberate omission of spool deletion.

7. Extend the README's managed-credential example to show OBJECT_STORAGE_KEY_ID and OBJECT_STORAGE_KEY mapped from encrypted inventory beside the existing Audita and OpenRouter entries. Explain that values remain outside ordinary configuration and that setting both S3 credential references to null selects Narratio's ambient AWS SDK provider chain.

8. Replace the README's statement that storage and publishing are deferred with concise operator-facing descriptions and public upstream links for S3 session audio, publish, restore, publication locks, status, and cleanup. Explain that accepted Notarius lanes remain narratio.extraction.<output_key> analyze inputs independently of whether they have stable publish rules.

9. Adapt the existing tests/integration baseline rather than adding a new scenario. Add two synthetic object-storage values to dnd_pipeline_secrets; explicitly unset OBJECT_STORAGE_KEY_ID and OBJECT_STORAGE_KEY alongside the existing provider variables when invoking Ansible; and explicitly pass representative non-default S3, cleanup, cache, and publish values through the new public variables.

10. Use representative values that exercise the interface without freezing the defaults: for example an .invalid HTTPS endpoint and fixture bucket, prefix, and region; intentional false values for cleanup, path style, and S3 audio caching; the two configured credential reference names; enabled run publication; and a short complete publish list containing one built-in transcript source and narratio.artifact.session_recap with omitted destinations.

11. Extend the existing secret-safe effective-configuration inspection only enough to demonstrate the explicitly supplied representative storage fields, typed credential names, falsey cache/cleanup choices, enabled publication, and Narratio-derived destinations. Continue to use narratio config validate as the native validity boundary. Do not inspect role task results, call S3, or reproduce Narratio's validation rules.

12. Extend the current non-disclosure check to cover both synthetic object-storage values. Preserve the complete existing integration matrix, cleanup behavior, and immediate clean-repeat assertions.

13. Run the complete repository validation. Commit the valid result with the stated plain-English message, push main to public Gitea, and record the full pushed commit hash for Stage 3.

Validation

Run from the workspace root:

(cd ansible-role-dnd-pipeline && tests/integration && git diff --check)

Expected result: syntax checking and production-profile lint pass; all five applications remain valid; the explicitly supplied S3 and publish values resolve through Narratio v2.1.1's native configuration interface; explicit publish requiredness and omitted destinations produce the derived destinations; only reference names appear in effective output; no synthetic credential value appears in captured output; no external S3 or model request occurs; and the immediate identical repeat reports changed=0 and failed=0.

Completion Criteria

• The pipeline selects Narratio v2.1.1, and defaults, argument specifications, dependency mappings, and documentation implement exactly the Feature 0015 contract.
• No production task, template, fragment, validation command, or second secret path was added.
• The representative integration scenario exercises public inputs rather than snapshotting defaults and retains the existing clean-repeat boundary.
• The full standalone integration suite and git diff --check pass.
• The pipeline repository is clean after the plain-English commit.
• The exact commit is present on public origin/main and its full hash is available for the collection import.

Commit

Commit intent: Enable D&D storage and publishing

Stage 2: Prepare the Collection Credential Fixture

Completion: complete (cf3de016ee09edf91afc3bc1addfd926d4a172ad)

Objective

Make the collection's artifact-level fixture compatible with the forthcoming pipeline snapshot while its currently bundled role remains valid.

Required Policy Review

Before beginning this stage, read:

• AGENTS.md
• docs/orientation.md
• docs/conventions.md
• docs/ansible.md
• docs/testing.md
• docs/planning.md
• docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md
• ansible-collection-dnd/README.md

Preconditions

• Stage 1 is complete, validated, committed, and pushed.
• ansible-collection-dnd is clean at the starting revision, is not behind or diverged from origin/main, and contains no unrelated changes.
• Its top-level tests/integration fixture still supplies the two existing synthetic provider credentials through dnd_pipeline_secrets.

Owning Repository

ansible-collection-dnd

Required Changes

1. Update only the collection-level tests/integration fixture outside roles/pipeline. Add conspicuously synthetic values for OBJECT_STORAGE_KEY_ID and OBJECT_STORAGE_KEY to its existing dnd_pipeline_secrets input.
2. Explicitly unset those two environment variables beside the existing Audita and OpenRouter variables when the fixture runs Ansible so ambient controller state cannot replace the managed synthetic files.
3. Do not add storage assertions or duplicate the standalone pipeline's detailed representative test. The collection fixture owns only enough input to validate the assembled artifact after Stage 3.
4. Do not edit anything under roles/pipeline, alter dependency manifests, or change collection release metadata.
5. Check the fixture's shell syntax and whitespace, then commit the compatible preparation with the stated message. Do not push separately unless needed; Stage 3 will push both collection commits after full artifact validation.

Validation

Run from the workspace root:

(cd ansible-collection-dnd && bash -n tests/integration && git diff --check)

Expected result: the collection integration driver remains valid shell; the diff changes only its generic managed-secret input and environment scrub; and no subtree, dependency, or release file changed. Full semantic validation occurs once against the imported snapshot in Stage 3.

Completion Criteria

• The artifact-level fixture provides all four synthetic operational credential names without exposing real values.
• No detailed storage behavior was duplicated in the collection.
• Only the collection-owned fixture changed.
• Shell syntax and git diff --check pass.
• The collection repository is clean after the plain-English commit.

Commit

Commit intent: Prepare collection storage credentials

Stage 3: Import and Validate the Reviewed Pipeline Snapshot

Completion: complete (merge 46913446765e86ecfb29ebad9955d579ef6a4c7e; squash 77f67adbc0f700e88bc15554e02052eaa22f1033)

Objective

Expose the reviewed Feature 0015 behavior through rakestraw.dnd.pipeline while preserving the public-remote subtree and collection-artifact boundaries.

Required Policy Review

Before beginning this stage, read:

• AGENTS.md
• docs/orientation.md
• docs/conventions.md
• docs/ansible.md
• docs/testing.md
• docs/planning.md
• docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md
• ansible-collection-dnd/README.md

Preconditions

• Stage 1's exact pipeline commit is publicly reachable from https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git.
• Stage 2 is complete and committed in ansible-collection-dnd.
• The collection is clean at the Stage 2 commit and its bundled pipeline subtree still corresponds to the authoritative pipeline history through c13f726a0a846180a1aa95d110b8bbc801edaae5.

Owning Repository

ansible-collection-dnd

Required Changes

1. Set a shell variable to the full pushed Stage 1 commit hash and run the documented remote-only subtree update from the collection repository:

   pipeline_revision=<FULL_PUSHED_STAGE_1_COMMIT>
   git subtree pull --prefix=roles/pipeline \
     https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \
     "$pipeline_revision" --squash -m "Import D&D storage and publishing"
   

2. Do not edit role behavior directly beneath roles/pipeline. The import must faithfully carry the reviewed defaults, argument specification, dependency mappings, README, and detailed integration fixture.

3. Do not modify galaxy.yml, requirements.yml, release documentation, or the published v0.1.0 artifacts. Feature 0015 changes no standalone role or native collection dependency and publishes no new release; the pipeline's application-version variable carries the Narratio v2.1.1 selection.

4. Run the complete collection integration suite against the built and installed artifact. Do not add collection-level assertions for the detailed S3 configuration; native validation and clean convergence remain the collection boundary.

5. If validation succeeds, push the Stage 2 fixture commit, the subtree merge commit, and its Git-generated squash commit to origin/main. Record the resulting commit hashes for Stage 4.

Validation

Run from the workspace root:

(cd ansible-collection-dnd && tests/integration && git diff --check)

Expected result: the collection builds; its archive contains the updated pipeline role while retaining development-file exclusions; isolated dependency and artifact installation succeeds; the FQCN resolves; Narratio loads its typed storage references from the synthetic credential files without contacting S3; native representative application checks pass; and the immediate repeat reports changed=0 and failed=0.

Completion Criteria

• Git history records the exact public Stage 1 revision through the documented squashed subtree workflow and stated merge message.
• No pipeline behavior was authored directly in the collection.
• The complete collection suite and git diff --check pass.
• The collection repository is clean and its Stage 2 and Stage 3 history is present on public origin/main.

Commit

Commit intent: Import D&D storage and publishing

The subtree command creates this stage's merge commit and its generated squash commit; do not add an empty or redundant manual commit.

Stage 4: Record the Implementation Handoff

Objective

Record the completed implementation evidence without marking Feature 0015 done before independent review.

Required Policy Review

Before beginning this stage, read:

• AGENTS.md
• docs/orientation.md
• docs/conventions.md
• docs/planning.md
• docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md

Preconditions

• Stages 1 through 3 are complete, committed, pushed where required, and validated.
• Their full commit hashes and validation results are available.
• The workspace repository is clean at the planning commit, is not behind or diverged from origin/main, and contains no unrelated changes.

Owning Repository

ansible-workspace-dnd

Required Changes

1. Change Feature 0015's roadmap status and status-table state to in_progress, indicating implementation is complete but independent review remains outstanding.
2. Update the workspace current focus and Feature 0015 evidence with the exact pipeline commit, collection fixture commit, collection subtree merge and squash commits, and validation completed.
3. Replace this plan's Progress Notes with a concise record of the four stage states and their commits. Do not copy command logs into the repository.
4. Do not mark the feature done, remove this active plan, change Feature 0016's state, or create docs/roadmap/remediation.md; those are later review and closeout decisions.
5. Commit the workspace handoff record with the stated plain-English message.

Validation

Run from the workspace root:

./scripts/status
git diff --check

Expected result: every repository is clean; the pipeline and collection are not ahead of or behind their origins; only the workspace handoff commit may be ahead of origin/main; and the dashboard points to exact implementation evidence while retaining independent review as the next step.

Completion Criteria

• Workspace status accurately identifies Feature 0015 as implemented and awaiting independent review.
• The roadmap remains the durable target-state record and contains no command log or transient execution notes.
• The active plan records concise implementation evidence and contains no false completion claim.
• The workspace repository is clean after the handoff commit.

Commit

Commit intent: Record D&D storage and publishing implementation

Final Validation and Handoff

• Confirm Stage 1's complete standalone pipeline integration result and Stage 3's complete collection integration result.
• Run ./scripts/status and confirm every repository is clean, with no unexpected branch divergence or untracked artifact.
• Report the full pipeline commit, collection fixture commit, collection subtree merge and squash commits, and workspace handoff commit.
• Confirm that ansible-role-narratio, all standalone roles, applications, assets, and rakestraw-ansible remained unchanged during the resumed sprint; Narratio v2.1.1 was an already released prerequisite.
• Summarize any deviation from this plan, unresolved finding, or complexity added beyond direct variable mapping and identify the acceptance criterion that required it.
• Do not mark Feature 0015 done; implementation results require independent review first.

Progress Notes

• The original plan was approved and Stage 1 began against Narratio v2.1.0.
• Native validation exposed Narratio's publish-output atomic-sequence bookkeeping defect; the implementation agent correctly stopped with its partial pipeline edits uncommitted.
• Narratio v2.1.1 at 442469b9f57e65ce17f09059609dd1a37cf8c3b6 corrected the defect and added focused regression coverage without changing the public configuration contract.
• Resume Stage 1 from the preserved pipeline edits, advance the application pin to v2.1.1, rerun the complete validation, and then continue in stage order.