ansible-workspace-dnd/docs/roadmap/implementation.md

# Implementation Plan 0015: Enable D&D S3 Storage and Publishing

Status: ready

Target executor: GPT-5.6-Terra

Feature roadmap:
`docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md`

Active plan location: `docs/roadmap/implementation.md`

## Objective

Configure the authoritative D&D pipeline role to use the selected Backblaze
S3-compatible archive, typed object-storage credential references, maximum S3
audio caching, immutable run publication, selected transcript and D&D artifact
outputs, and cleanup of successfully published workspace state. Advance the
pipeline to Narratio v2.1.1 so explicit publish requiredness composes with
native destination derivation. Validate the configuration through Narratio's
native interface, import the exact reviewed pipeline commit into the collection
from public Gitea, and leave the result ready for the real-host deployment in
Feature 0016.

The Feature 0015 roadmap remains authoritative for scope, rationale, target
state, and acceptance criteria. This plan does not authorize a second storage
implementation, real object-storage calls from repository tests, per-session
configuration management, stable publication of every Notarius extraction
lane, or changes to the generic Narratio role.

## Execution Rules

- Execute stages in numeric order.
- Before each stage, confirm its preconditions and the working-tree state of
  its owning repository.
- Before each stage, reread every document listed in that stage's Required
  Policy Review. Do not rely on policy recalled from an earlier stage.
- Preserve unrelated and pre-existing changes. Stop and report unexpected
  worktree changes or branch divergence rather than overwriting them.
- Do not make unresolved product or architecture decisions during execution;
  stop and report the decision required.
- Implement storage and publication entirely through the standalone Narratio
  role's existing field-specific variables. Do not add a production task,
  template, handler, fact, probe, command, wrapper, state file, or validation
  layer to the pipeline role.
- Keep `dnd_pipeline_secrets` as the existing generic empty-by-default map.
  Add no real, generated, placeholder, or fallback credential value to role
  defaults, templates, documentation, tests, logs, or ordinary inventory.
- Keep the S3 access-key and secret-key reference variables nullable as a
  pair. Do not add an Ansible assertion for pairing or secret presence;
  Narratio owns native configuration validation and AWS provider selection.
- Do not configure `narratio_spool_delete_audio_after_publish` or
  `narratio_notarius_chunk_cache`. Preserve the roadmap's single-owner
  decisions for spool retention and Notarius cache policy.
- Treat accepted Notarius outputs and published outputs as separate
  interfaces. Preserve `narratio.extraction.<output_key>` sources for future
  analyze inputs without adding publication rules or new artifacts for them.
- Exercise explicitly supplied representative values in the detailed pipeline
  integration scenario. Do not assert that those values are current defaults
  or duplicate Narratio's own storage and publication test matrix.
- Do not make a real S3 request or paid model request during repository-local
  validation.
- Run every stage's validation before completing the stage. Do not proceed
  after a validation failure.
- Commit each completed stage with the stated plain-English intent. Push the
  authoritative pipeline commit before importing it into the collection.
- Use only the public Gitea pipeline URL for the subtree update. Do not import
  from the local sibling repository.

## Starting State and Preconditions

- Features 0009, 0011, 0012, 0013, and 0014 are complete.
- The approved Feature 0015 roadmap and this implementation plan are committed
  in the workspace repository before the sprint begins.
- `ansible-role-dnd-pipeline` remains based on
  `c13f726a0a846180a1aa95d110b8bbc801edaae5`, equal to `origin/main`, with the
  interrupted Stage 1 edits preserved and uncommitted in `README.md`,
  `defaults/main.yml`, `meta/argument_specs.yml`, `meta/main.yml`, and
  `tests/integration`. Review and complete those edits; do not discard them.
- `ansible-collection-dnd` starts clean on `main` at
  `c0b0ebd13e92199f3d94e138c6cc50d1ece5dadd`, equal to `origin/main`.
- `ansible-role-narratio` is clean at
  `292d8669e7aad5cfe80b0c469d22b6177e8cf0f1`, equal to `origin/main`, and its
  `v0.1.0` role release contains the field-specific interface required by this
  feature and can install Narratio v2.1.1 through `narratio_version`.
- The sibling Narratio application repository is clean at public release
  `v2.1.1`, commit `442469b9f57e65ce17f09059609dd1a37cf8c3b6`. That
  backward-compatible release corrects destination derivation with explicit
  publish requiredness and is a read-only contract source during the resumed
  sprint.
- The workspace `.venv` supplies the tested `ansible-playbook`,
  `ansible-galaxy`, and `ansible-lint` commands used by the repository test
  drivers.
- The local host remains the approved Ubuntu 26.04 proxy, and passwordless
  test-time sudo is available for the existing integration fixtures.
- Git credentials can push the pipeline and collection commits to their public
  Gitea origins.
- No actual Backblaze or model-provider credential is required or available to
  the implementation. Tests use conspicuously synthetic values only.

## Repository Scope

- `ansible-role-dnd-pipeline` owns the suite defaults, argument specification,
  Narratio dependency mappings, consumer documentation, and detailed
  representative integration behavior.
- `ansible-collection-dnd` owns the artifact-level fixture update, faithful
  squashed subtree import, assembled-artifact validation, and resulting public
  collection snapshot.
- `ansible-workspace-dnd` records the implementation handoff after the two
  implementation repositories are complete.
- `ansible-role-narratio`, all standalone roles, application sources,
  asset repositories, and `rakestraw-ansible` may be inspected but must remain
  unchanged.

## Stage 1: Implement D&D Storage and Publication Policy

Completion: complete (`51566afcf69083cdbb5b5718f2544c86b079001c`)

### Objective

Expose the approved suite-level S3 and publishing inputs through the
authoritative pipeline role, map them directly into Narratio, document the
operator contract, and prove representative composition without networked
storage access.

### Required Policy Review

Before beginning this stage, read:

- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/ansible.md`
- `docs/testing.md`
- `docs/planning.md`
- `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md`
- `ansible-role-dnd-pipeline/README.md`
- `../narratio/AGENTS.md`
- `../narratio/docs/development.md`
- `../narratio/docs/config.md`
- `../narratio/docs/operations.md`

### Preconditions

- The workspace planning commit containing the original approved plan and this
  resumed-plan revision are present.
- `ansible-role-dnd-pipeline` is based on the starting revision, is not behind
  or diverged from `origin/main`, and its only working-tree changes are the
  preserved interrupted Stage 1 edits identified in Starting State and
  Preconditions.
- The current `rakestraw.narratio` dependency mapping and existing integration
  driver still match the structures reviewed while preparing this plan.
- Narratio v2.1.1 remains publicly available and accepts the roadmap-selected
  storage, publish, cleanup, cache, and typed credential-reference fields,
  including rules with explicit `required` and omitted derivable `dest`.

### Owning Repository

`ansible-role-dnd-pipeline`

### Required Changes

1. Advance the existing application default to
   `dnd_pipeline_narratio_version: v2.1.1`. Do not change the standalone
   Narratio role release pin: its existing public interface can install this
   compatible application patch release.
2. Add the following exact suite defaults to `defaults/main.yml` near the
   existing Narratio policy variables:

   ```yaml
   dnd_pipeline_narratio_workspace_cleanup_after_publish: true
   dnd_pipeline_narratio_storage_backend: s3
   dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets
   dnd_pipeline_narratio_storage_s3_root_prefix: dnd
   dnd_pipeline_narratio_storage_s3_region: us-east-1
   dnd_pipeline_narratio_storage_s3_endpoint: https://s3.us-east-005.backblazeb2.com
   dnd_pipeline_narratio_storage_s3_force_path_style: true
   dnd_pipeline_narratio_storage_s3_access_key_id: OBJECT_STORAGE_KEY_ID
   dnd_pipeline_narratio_storage_s3_secret_access_key: OBJECT_STORAGE_KEY
   dnd_pipeline_narratio_cache_s3_audio: true
   dnd_pipeline_narratio_publish_enabled: true
   dnd_pipeline_narratio_publish_upload_run: true
   dnd_pipeline_narratio_publish_outputs:
     - source: narratio.transcript.final
       required: true
     - source: narratio.transcript.final_trimmed
       required: true
     - source: narratio.transcript.final_markdown
       required: true
     - source: narratio.transcript.final_trimmed_markdown
       required: true
     - source: narratio.artifact.session_recap
       required: true
     - source: narratio.artifact.hostile_rumors
       required: true
     - source: narratio.artifact.deranged_rumors
       required: false
   ```

   Keep every `dest` omitted. Narratio derives transcript destinations from
   built-in sources and artifact destinations from the already managed
   artifact output paths.
3. Add matching public entries to `meta/argument_specs.yml`. Use `type: bool`
   for the cleanup, path-style, cache, enabled, and upload settings; `type: str`
   for the backend and S3 string fields; and `type: list` for the complete
   publish output list. The S3 child fields, including both credential
   references, must be non-required so callers may set every S3 child to
   `null` when deliberately selecting the local backend or set the credential
   pair to `null` when selecting the AWS SDK provider chain. The remaining
   concrete suite-policy fields follow the role's existing required/defaulted
   argument-spec convention.
4. In the `rakestraw.narratio` dependency entry in `meta/main.yml`, map every
   new suite variable directly to its same-purpose standalone variable:
   `narratio_workspace_cleanup_after_publish`, the eight
   `narratio_storage_*` fields, `narratio_cache_s3_audio`,
   `narratio_publish_enabled`, `narratio_publish_upload_run`, and
   `narratio_publish_outputs`.
5. Do not map `narratio_spool_delete_audio_after_publish`. Do not map
   `narratio_notarius_chunk_cache`. Do not add a new import, fragment,
   template, task, handler, assertion, or configuration-validation command.
6. Revise the README's application-version reference, upstream links, variable
   reference, and processing-policy sections for Narratio v2.1.1 and
   document the selected S3 destination, paired nullable credential
   references, complete-list publish semantics, native destination derivation,
   maximum S3 audio caching, successful-publication workspace cleanup, and
   deliberate omission of spool deletion.
7. Extend the README's managed-credential example to show
   `OBJECT_STORAGE_KEY_ID` and `OBJECT_STORAGE_KEY` mapped from encrypted
   inventory beside the existing Audita and OpenRouter entries. Explain that
   values remain outside ordinary configuration and that setting both S3
   credential references to `null` selects Narratio's ambient AWS SDK provider
   chain.
8. Replace the README's statement that storage and publishing are deferred
   with concise operator-facing descriptions and public upstream links for S3
   session audio, publish, restore, publication locks, status, and cleanup.
   Explain that accepted Notarius lanes remain
   `narratio.extraction.<output_key>` analyze inputs independently of whether
   they have stable publish rules.
9. Adapt the existing `tests/integration` baseline rather than adding a new
   scenario. Add two synthetic object-storage values to
   `dnd_pipeline_secrets`; explicitly unset `OBJECT_STORAGE_KEY_ID` and
   `OBJECT_STORAGE_KEY` alongside the existing provider variables when
   invoking Ansible; and explicitly pass representative non-default S3,
   cleanup, cache, and publish values through the new public variables.
10. Use representative values that exercise the interface without freezing the
   defaults: for example an `.invalid` HTTPS endpoint and fixture bucket,
   prefix, and region; intentional `false` values for cleanup, path style, and
   S3 audio caching; the two configured credential reference names; enabled
   run publication; and a short complete publish list containing one built-in
   transcript source and `narratio.artifact.session_recap` with omitted
   destinations.
11. Extend the existing secret-safe effective-configuration inspection only
    enough to demonstrate the explicitly supplied representative storage
    fields, typed credential names, falsey cache/cleanup choices, enabled
    publication, and Narratio-derived destinations. Continue to use
    `narratio config validate` as the native validity boundary. Do not inspect
    role task results, call S3, or reproduce Narratio's validation rules.
12. Extend the current non-disclosure check to cover both synthetic
    object-storage values. Preserve the complete existing integration matrix,
    cleanup behavior, and immediate clean-repeat assertions.
13. Run the complete repository validation. Commit the valid result with the
    stated plain-English message, push `main` to public Gitea, and record the
    full pushed commit hash for Stage 3.

### Validation

Run from the workspace root:

```text
(cd ansible-role-dnd-pipeline && tests/integration && git diff --check)
```

Expected result: syntax checking and production-profile lint pass; all five
applications remain valid; the explicitly supplied S3 and publish values
resolve through Narratio v2.1.1's native configuration interface; explicit
publish requiredness and omitted destinations produce the derived destinations;
only reference names appear in effective output; no synthetic credential value
appears in captured output; no external S3 or model request occurs; and the
immediate identical repeat reports `changed=0` and `failed=0`.

### Completion Criteria

- The pipeline selects Narratio v2.1.1, and defaults, argument specifications,
  dependency mappings, and documentation implement exactly the Feature 0015
  contract.
- No production task, template, fragment, validation command, or second secret
  path was added.
- The representative integration scenario exercises public inputs rather than
  snapshotting defaults and retains the existing clean-repeat boundary.
- The full standalone integration suite and `git diff --check` pass.
- The pipeline repository is clean after the plain-English commit.
- The exact commit is present on public `origin/main` and its full hash is
  available for the collection import.

### Commit

Commit intent: `Enable D&D storage and publishing`

## Stage 2: Prepare the Collection Credential Fixture

### Objective

Make the collection's artifact-level fixture compatible with the forthcoming
pipeline snapshot while its currently bundled role remains valid.

### Required Policy Review

Before beginning this stage, read:

- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/ansible.md`
- `docs/testing.md`
- `docs/planning.md`
- `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md`
- `ansible-collection-dnd/README.md`

### Preconditions

- Stage 1 is complete, validated, committed, and pushed.
- `ansible-collection-dnd` is clean at the starting revision, is not behind or
  diverged from `origin/main`, and contains no unrelated changes.
- Its top-level `tests/integration` fixture still supplies the two existing
  synthetic provider credentials through `dnd_pipeline_secrets`.

### Owning Repository

`ansible-collection-dnd`

### Required Changes

1. Update only the collection-level `tests/integration` fixture outside
   `roles/pipeline`. Add conspicuously synthetic values for
   `OBJECT_STORAGE_KEY_ID` and `OBJECT_STORAGE_KEY` to its existing
   `dnd_pipeline_secrets` input.
2. Explicitly unset those two environment variables beside the existing
   Audita and OpenRouter variables when the fixture runs Ansible so ambient
   controller state cannot replace the managed synthetic files.
3. Do not add storage assertions or duplicate the standalone pipeline's
   detailed representative test. The collection fixture owns only enough
   input to validate the assembled artifact after Stage 3.
4. Do not edit anything under `roles/pipeline`, alter dependency manifests,
   or change collection release metadata.
5. Check the fixture's shell syntax and whitespace, then commit the compatible
   preparation with the stated message. Do not push separately unless needed;
   Stage 3 will push both collection commits after full artifact validation.

### Validation

Run from the workspace root:

```text
(cd ansible-collection-dnd && bash -n tests/integration && git diff --check)
```

Expected result: the collection integration driver remains valid shell; the
diff changes only its generic managed-secret input and environment scrub; and
no subtree, dependency, or release file changed. Full semantic validation
occurs once against the imported snapshot in Stage 3.

### Completion Criteria

- The artifact-level fixture provides all four synthetic operational
  credential names without exposing real values.
- No detailed storage behavior was duplicated in the collection.
- Only the collection-owned fixture changed.
- Shell syntax and `git diff --check` pass.
- The collection repository is clean after the plain-English commit.

### Commit

Commit intent: `Prepare collection storage credentials`

## Stage 3: Import and Validate the Reviewed Pipeline Snapshot

### Objective

Expose the reviewed Feature 0015 behavior through `rakestraw.dnd.pipeline`
while preserving the public-remote subtree and collection-artifact boundaries.

### Required Policy Review

Before beginning this stage, read:

- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/ansible.md`
- `docs/testing.md`
- `docs/planning.md`
- `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md`
- `ansible-collection-dnd/README.md`

### Preconditions

- Stage 1's exact pipeline commit is publicly reachable from
  `https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git`.
- Stage 2 is complete and committed in `ansible-collection-dnd`.
- The collection is clean at the Stage 2 commit and its bundled pipeline
  subtree still corresponds to the authoritative pipeline history through
  `c13f726a0a846180a1aa95d110b8bbc801edaae5`.

### Owning Repository

`ansible-collection-dnd`

### Required Changes

1. Set a shell variable to the full pushed Stage 1 commit hash and run the
   documented remote-only subtree update from the collection repository:

   ```text
   pipeline_revision=<FULL_PUSHED_STAGE_1_COMMIT>
   git subtree pull --prefix=roles/pipeline \
     https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \
     "$pipeline_revision" --squash -m "Import D&D storage and publishing"
   ```

2. Do not edit role behavior directly beneath `roles/pipeline`. The import
   must faithfully carry the reviewed defaults, argument specification,
   dependency mappings, README, and detailed integration fixture.
3. Do not modify `galaxy.yml`, `requirements.yml`, release documentation, or
   the published `v0.1.0` artifacts. Feature 0015 changes no standalone role or
   native collection dependency and publishes no new release; the pipeline's
   application-version variable carries the Narratio v2.1.1 selection.
4. Run the complete collection integration suite against the built and
   installed artifact. Do not add collection-level assertions for the detailed
   S3 configuration; native validation and clean convergence remain the
   collection boundary.
5. If validation succeeds, push the Stage 2 fixture commit, the subtree merge
   commit, and its Git-generated squash commit to `origin/main`. Record the
   resulting commit hashes for Stage 4.

### Validation

Run from the workspace root:

```text
(cd ansible-collection-dnd && tests/integration && git diff --check)
```

Expected result: the collection builds; its archive contains the updated
pipeline role while retaining development-file exclusions; isolated
dependency and artifact installation succeeds; the FQCN resolves; Narratio
loads its typed storage references from the synthetic credential files without
contacting S3; native representative application checks pass; and the
immediate repeat reports `changed=0` and `failed=0`.

### Completion Criteria

- Git history records the exact public Stage 1 revision through the documented
  squashed subtree workflow and stated merge message.
- No pipeline behavior was authored directly in the collection.
- The complete collection suite and `git diff --check` pass.
- The collection repository is clean and its Stage 2 and Stage 3 history is
  present on public `origin/main`.

### Commit

Commit intent: `Import D&D storage and publishing`

The subtree command creates this stage's merge commit and its generated squash
commit; do not add an empty or redundant manual commit.

## Stage 4: Record the Implementation Handoff

### Objective

Record the completed implementation evidence without marking Feature 0015
done before independent review.

### Required Policy Review

Before beginning this stage, read:

- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/planning.md`
- `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md`

### Preconditions

- Stages 1 through 3 are complete, committed, pushed where required, and
  validated.
- Their full commit hashes and validation results are available.
- The workspace repository is clean at the planning commit, is not behind or
  diverged from `origin/main`, and contains no unrelated changes.

### Owning Repository

`ansible-workspace-dnd`

### Required Changes

1. Change Feature 0015's roadmap status and status-table state to
   `in_progress`, indicating implementation is complete but independent review
   remains outstanding.
2. Update the workspace current focus and Feature 0015 evidence with the exact
   pipeline commit, collection fixture commit, collection subtree merge and
   squash commits, and validation completed.
3. Replace this plan's Progress Notes with a concise record of the four stage
   states and their commits. Do not copy command logs into the repository.
4. Do not mark the feature `done`, remove this active plan, change Feature
   0016's state, or create `docs/roadmap/remediation.md`; those are later review
   and closeout decisions.
5. Commit the workspace handoff record with the stated plain-English message.

### Validation

Run from the workspace root:

```text
./scripts/status
git diff --check
```

Expected result: every repository is clean; the pipeline and collection are
not ahead of or behind their origins; only the workspace handoff commit may be
ahead of `origin/main`; and the dashboard points to exact implementation
evidence while retaining independent review as the next step.

### Completion Criteria

- Workspace status accurately identifies Feature 0015 as implemented and
  awaiting independent review.
- The roadmap remains the durable target-state record and contains no command
  log or transient execution notes.
- The active plan records concise implementation evidence and contains no
  false completion claim.
- The workspace repository is clean after the handoff commit.

### Commit

Commit intent: `Record D&D storage and publishing implementation`

## Final Validation and Handoff

- Confirm Stage 1's complete standalone pipeline integration result and Stage
  3's complete collection integration result.
- Run `./scripts/status` and confirm every repository is clean, with no
  unexpected branch divergence or untracked artifact.
- Report the full pipeline commit, collection fixture commit, collection
  subtree merge and squash commits, and workspace handoff commit.
- Confirm that `ansible-role-narratio`, all standalone roles, applications,
  assets, and `rakestraw-ansible` remained unchanged during the resumed sprint;
  Narratio v2.1.1 was an already released prerequisite.
- Summarize any deviation from this plan, unresolved finding, or complexity
  added beyond direct variable mapping and identify the acceptance criterion
  that required it.
- Do not mark Feature 0015 `done`; implementation results require independent
  review first.

## Progress Notes

- The original plan was approved and Stage 1 began against Narratio v2.1.0.
- Native validation exposed Narratio's publish-output atomic-sequence
  bookkeeping defect; the implementation agent correctly stopped with its
  partial pipeline edits uncommitted.
- Narratio v2.1.1 at `442469b9f57e65ce17f09059609dd1a37cf8c3b6`
  corrected the defect and added focused regression coverage without changing
  the public configuration contract.
- Resume Stage 1 from the preserved pipeline edits, advance the application pin
  to v2.1.1, rerun the complete validation, and then continue in stage order.