ansible-workspace-dnd/docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md

# Feature 0015: Enable D&D S3 Storage and Publishing

Status: done

## Summary

Make the D&D pipeline operational with Narratio's current S3 storage,
publication, caching, cleanup, and restore capabilities. The pipeline role
will select the established Backblaze B2 destination, reference two typed
object-storage credentials delivered through its existing secrets directory,
publish the useful transcript and D&D artifact outputs, and retain maximum S3
audio caching.

The standalone Narratio role already exposes Narratio v2.1.1's complete
configuration surface for this behavior. This feature supplies D&D suite
policy through that existing interface; it does not add another storage
implementation or credential mechanism.

## Problem or Opportunity

The pipeline role currently installs Narratio v2.1.0 and configures the full
processing graph, but deliberately leaves Narratio storage and publishing at
their application-level defaults. Its README consequently describes storage,
publishing, and end-to-end external execution as deferred. The resulting suite
can validate configuration and process local state, but it does not select the
remote archive or publish the outputs needed for an operational deployment.

Narratio v2.1.0 also rejected a valid publish list when a rule explicitly set
`required` while omitting a derivable `dest`. Narratio v2.1.1 corrects that
resolution bookkeeping defect without changing the publish schema or
destination rules. This feature therefore advances the pipeline application
pin to v2.1.1 and retains explicit suite-owned requiredness together with
native destination derivation.

The prior test-machine configuration records the intended Backblaze B2
endpoint, bucket, prefix, credential names, publication set, and post-publish
workspace cleanup. Narratio's current contract has since replaced the legacy
credential field names and secrets configuration with ConfigKit typed
references and a conventional sibling `secrets/` directory. The Ansible suite
already implements that new credential-delivery boundary for Audita and
Notarius, so S3 needs only to join the same path.

## Governing Policies

- [The Ansible house style](../../ansible.md) requires suite policy to flow
  through the standalone role's field-specific interface and optional fields
  to remain application-owned unless the suite deliberately selects them.
- [The testing house style](../../testing.md) requires representative
  public-interface and native-application validation without duplicating AWS
  SDK, ConfigKit, or Ansible module behavior.
- [The MVP roadmap](../mvp.md) makes the pipeline role responsible for
  cross-application configuration and end-to-end validation.
- [Feature 0011](0011-managed-pipeline-credential-delivery.md) establishes the
  managed Narratio secrets directory and typed credential-forwarding policy.
- [Feature 0012](0012-collection-assembly.md) makes the standalone pipeline
  repository authoritative and imports reviewed changes through public
  Gitea.
- [Feature 0016](0016-deploy-dnd-pipeline-to-pipeline-dnd.md) consumes this
  feature's reviewed interface for the first real-host deployment.

## Scope

- Advance the suite's Narratio application version from `v2.1.0` to the
  backward-compatible `v2.1.1` bugfix release.
- Add suite-owned pipeline variables for the selected Narratio workspace,
  S3, cache, and publish settings.
- Select the S3 backend with these D&D deployment defaults:

  | Setting | Suite default |
  | --- | --- |
  | Endpoint | `https://s3.us-east-005.backblazeb2.com` |
  | Bucket | `rakestraw-assets` |
  | Root prefix | `dnd` |
  | Region | `us-east-1` |
  | Force path style | `true` |
  | Access-key reference | `OBJECT_STORAGE_KEY_ID` |
  | Secret-key reference | `OBJECT_STORAGE_KEY` |

- Pass the two credential reference names to the standalone Narratio role.
  Their values use the existing `dnd_pipeline_secrets` map and managed
  `/usr/local/etc/narratio/secrets/` directory.
- Enable S3 audio caching explicitly so the suite retains Narratio's maximum
  available audio reuse.
- Enable run publication and publish these complete native output rules:

  | Source | Required |
  | --- | --- |
  | `narratio.transcript.final` | `true` |
  | `narratio.transcript.final_trimmed` | `true` |
  | `narratio.transcript.final_markdown` | `true` |
  | `narratio.transcript.final_trimmed_markdown` | `true` |
  | `narratio.artifact.session_recap` | `true` |
  | `narratio.artifact.hostile_rumors` | `true` |
  | `narratio.artifact.deranged_rumors` | `false` |

- Let Narratio derive each destination from the built-in transcript path or
  configured artifact output path rather than duplicating those paths in the
  publish list.
- Enable workspace cleanup only after Narratio records successful run
  publication. Leave spool-audio deletion under Narratio's existing default
  because the suite has selected no policy requiring source-audio removal.
- Keep the production and `testing` profile storage and publishing behavior
  identical. The existing overlay remains limited to model-profile selection.
- Document the four credentials normally required by the operational suite:
  `AUDITA_LLM_API_KEY`, `OPENROUTER_API_KEY`,
  `OBJECT_STORAGE_KEY_ID`, and `OBJECT_STORAGE_KEY`.
- Document the operator-visible publication, restore, S3-session-audio, lock,
  caching, and cleanup capabilities enabled by the selected configuration,
  while linking Narratio's canonical CLI and operations documentation for
  command details.
- Preserve every accepted Notarius lane as its existing
  `narratio.extraction.<output_key>` source. Future Narratio artifact
  definitions can consume those sources during analyze without first making
  the extraction lanes separate published outputs.
- Adapt the existing pipeline integration scenario to resolve synthetic S3
  credentials and validate one explicitly supplied representative storage and
  publishing configuration through Narratio's native interface without
  contacting object storage.
- Import the reviewed public pipeline commit into the collection through the
  established squashed subtree workflow, adapt the collection's artifact-level
  fixture to supply the added credential names, and validate the assembled
  artifact.

## Non-goals

- Implementing an S3 client, publication engine, remote commit format,
  restore mechanism, publish-lock mechanism, or audio cache in Ansible.
  Narratio owns all of those behaviors.
- Placing credential values in role defaults, templates, documentation,
  tests, ordinary inventory, or the generated pipeline configuration.
- Adding a second secret-delivery mechanism, environment wrapper, service
  unit, credential synchronizer, or AWS configuration-file manager.
- Managing Backblaze buckets, object lifecycle policy, application keys,
  retention policy, versioning, encryption, quotas, or network access.
- Managing per-session `session.yml` files or selecting local versus S3 audio
  input for an individual session.
- Running a paid or networked publication test from a repository-local test
  suite. Feature 0016 owns the representative live end-to-end exercise.
- Publishing every Notarius extraction lane as a named current output. The
  lanes remain available as analyze inputs independently of publication; stable
  remote destinations require an explicit operator need.
- Setting Narratio's optional `notarius.chunk_cache` override. The managed
  Notarius configuration remains the single owner of its ordinary `auto`
  cache policy, and Narratio v2.1 already requests refresh for forced
  extraction.
- Adding notification providers. Narratio v2.1 supports only its existing
  `noop` notification mode.
- Releasing new role or collection versions as part of this implementation.
- Modifying Narratio, ConfigKit, or the standalone Narratio role. Narratio
  v2.1.1 is an already published prerequisite, and the existing generic role
  can install the selected application tag through `narratio_version`.

## Target End State

The standalone pipeline role exposes these suite variables with the selected
defaults:

```yaml
dnd_pipeline_narratio_version: v2.1.1
dnd_pipeline_narratio_workspace_cleanup_after_publish: true
dnd_pipeline_narratio_storage_backend: s3
dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets
dnd_pipeline_narratio_storage_s3_root_prefix: dnd
dnd_pipeline_narratio_storage_s3_region: us-east-1
dnd_pipeline_narratio_storage_s3_endpoint: >-
  https://s3.us-east-005.backblazeb2.com
dnd_pipeline_narratio_storage_s3_force_path_style: true
dnd_pipeline_narratio_storage_s3_access_key_id: OBJECT_STORAGE_KEY_ID
dnd_pipeline_narratio_storage_s3_secret_access_key: OBJECT_STORAGE_KEY
dnd_pipeline_narratio_cache_s3_audio: true
dnd_pipeline_narratio_publish_enabled: true
dnd_pipeline_narratio_publish_upload_run: true
dnd_pipeline_narratio_publish_outputs:
  - source: narratio.transcript.final
    required: true
  - source: narratio.transcript.final_trimmed
    required: true
  - source: narratio.transcript.final_markdown
    required: true
  - source: narratio.transcript.final_trimmed_markdown
    required: true
  - source: narratio.artifact.session_recap
    required: true
  - source: narratio.artifact.hostile_rumors
    required: true
  - source: narratio.artifact.deranged_rumors
    required: false
```

Each value is declared in the pipeline argument specification and passed
directly to its corresponding `narratio_*` variable in the pipeline role's
Narratio dependency. Narratio's existing root template renders the native
configuration. No new pipeline fragment or production task is required.

The ordinary generated configuration contains only the credential names:

```yaml
storage:
  backend: s3
  s3:
    access_key_id: OBJECT_STORAGE_KEY_ID
    secret_access_key: OBJECT_STORAGE_KEY
```

Consumers supply the two values beside the existing model-provider values
through encrypted inventory:

```yaml
dnd_pipeline_secrets:
  AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
  OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"
  OBJECT_STORAGE_KEY_ID: "{{ vault_dnd_object_storage_key_id }}"
  OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}"
```

The example defines references only; no roadmap, role, or test contains real
credential values. The existing credential task writes the selected entries
with the suite's established group-authorized secret modes. Omitted map keys
remain unmanaged, as defined by Feature 0011.

With valid credentials, Narratio can accept either local or S3 audio selected
by each session file, cache S3 audio beneath `/var/cache/narratio`, publish an
immutable run and the selected stable outputs, use remote publication locks,
restore committed state, and clean the published workspace. These are native
Narratio commands and lifecycle rules rather than additional Ansible tasks.

The ten accepted Notarius outputs continue to enter Narratio's artifact
catalog as `narratio.extraction.*` sources. Analyze-stage Scriptorium artifacts
may reference those sources directly in their input mappings. Feature 0015
does not yet add such artifact definitions, but its publication choices do not
remove, rename, or constrain this future composition path.

## Repository Ownership

- `ansible-role-dnd-pipeline` owns the D&D suite variables, opinionated
  defaults, dependency mappings, consumer documentation, and representative
  integration scenario.
- `ansible-role-narratio` remains the generic owner of installation and native
  pipeline rendering. Its reviewed `v0.1.0` role interface is compatible with
  Narratio v2.1.1 and already covers every
  configuration field required by this feature and should remain unchanged.
- `ansible-collection-dnd` owns the faithful pipeline subtree import, the
  unified consumer dependency manifest, and artifact-boundary validation.
- `ansible-workspace-dnd` owns this roadmap, the later implementation plan,
  status, and independent review record.
- `narratio` v2.1.1 is the released compatibility prerequisite and a read-only
  source of storage, publication, credential, caching, cleanup, restore, and
  CLI contracts during this implementation.
- `rakestraw-ansible` remains unchanged until Feature 0016 consumes this
  interface on `pipeline-dnd`.

## Constraints and Settled Decisions

- The S3 endpoint, bucket, prefix, region, and path-style values are deliberate
  D&D suite defaults. They are ordinary variables and remain overrideable by
  inventory.
- Credential references are also policy defaults, but credential values have
  no defaults. Operators must supply matching entries through
  `dnd_pipeline_secrets` or another environment source supported by Narratio.
- The two credential-reference variables are nullable as a pair. Setting both
  to `null` omits the references and preserves Narratio's AWS SDK provider-chain
  path; configuring only one remains invalid under Narratio's native contract.
- The role does not assert that the secrets map contains particular keys.
  Narratio's native loader owns typed-reference resolution, and external
  environment values remain a supported higher-precedence source.
- `dnd_pipeline_narratio_publish_outputs` is one complete native list. A caller
  replacing it assumes responsibility for the entire selected publication
  set; the role does not merge rules.
- Publish destinations remain omitted where Narratio can derive them. The
  configured source identity and artifact output path remain the single source
  of truth.
- Narratio v2.1.1 is the minimum application version for this feature because
  it preserves destination derivation when `required` is explicit. Do not work
  around older behavior by duplicating destinations or omitting deliberate
  requiredness.
- Workspace cleanup is gated by Narratio's own successful-publication
  evidence. Ansible does not independently inspect remote state or remove
  runtime content.
- S3 audio caching is enabled, and ordinary cleanup preserves the cache.
  Explicit operator cache clearing remains a Narratio CLI operation.
- The testing profile does not silently disable Notarius, storage,
  publication, cleanup, or credentials. It continues to differ only in the
  already selected inexpensive Scriptorium model profile.
- S3 storage also enables Narratio's native restore and publication-lock
  commands without another configuration surface. Ansible documentation may
  describe availability but must not duplicate Narratio's detailed operating
  manual.
- Notarius extraction-source availability and remote publication are separate
  concerns. Analyze consumes `narratio.extraction.*` sources from the accepted
  extraction catalog; `publish.outputs` selects only which sources also receive
  stable published destinations.

## Simplest Viable Implementation

Advance `dnd_pipeline_narratio_version` to v2.1.1. Add the selected variables to
the pipeline defaults and argument specification, then map them into the
existing `rakestraw.narratio` dependency. Update the pipeline README to document
the policy, credential names, complete-list replacement semantics, and native
capabilities now available. Do not add a template, task, handler, fact, probe,
or storage-specific validation layer.

In the existing pipeline integration driver, extend the synthetic secrets map
with object-storage values and explicitly supply representative S3 and publish
variables. Use `narratio config validate` and its secret-safe effective
configuration output to prove that the public inputs compose into a valid
configuration. Retain the immediate clean repeat and secret non-disclosure
checks. Do not call a real S3 service or assert that the representative values
equal current role defaults.

Push the reviewed pipeline commit, import that exact public revision into the
collection through `git subtree pull --squash`, and extend only the
collection-level fixture inputs needed for native validation. The collection
suite should continue to prove artifact installation and one representative
assembled convergence rather than duplicate the detailed pipeline matrix.

## Rationale and Alternatives

Implementing storage behavior in Ansible would duplicate a mature native
application boundary. Narratio already owns credential resolution, AWS SDK
provider construction, immutable commits, verification, locks, restore,
caching, and cleanup. Passing declarative configuration through the generic
role is both smaller and more reliable.

Using the AWS SDK's ambient provider chain would avoid named secret files, but
the initial deployment has explicit Backblaze application credentials and the
suite already has a suitable managed typed-secret path. Keeping the two
credential-reference variables nullable preserves the ambient-provider
alternative: consumers can set both suite variables to `null` when their
deployment uses another provider.

Copying destinations into every publish rule would make the output layout more
visible locally, but it would duplicate paths already owned by Narratio's
built-in sources and the configured artifact definitions. Native destination
derivation keeps path coordination automatic.

Omitting explicit `required: true` values would avoid the v2.1.0 failure only
by relying on an unrelated defaulting path. Changing the generic role's YAML
format would not repair the underlying sequence-resolution mismatch. Narratio
v2.1.1 fixes the owning application boundary, allowing the suite to state its
requiredness policy directly without duplicating destinations.

Deleting spool audio after publication would minimize transient disk usage,
but no such retention policy has been selected, and local-audio sessions may
depend on that content. Leaving the field omitted preserves Narratio's default
while enabling the explicitly requested S3 audio cache.

Publishing all ten Notarius extraction lanes would create stable remote output
contracts that have not been requested. It is also unnecessary for the stated
future use: analyze-stage artifacts can consume `narratio.extraction.*`
directly. The immutable published run retains the processing result, while the
selected transcript and human-facing artifact outputs match the known
operational workflow.

## Acceptance Criteria

- The pipeline role declares and documents every suite variable shown in the
  target state, with matching argument-spec entries and direct Narratio
  dependency mappings.
- The pipeline installs Narratio v2.1.1, whose released configuration loader
  supports explicit requiredness together with derived publish destinations.
- The two credential-reference variables accept paired `null` overrides so a
  caller can deliberately select Narratio's ambient AWS SDK provider chain.
- The default effective Narratio configuration selects the specified
  Backblaze-compatible S3 destination, typed credential names, S3 audio cache,
  successful-publication workspace cleanup, run upload, and seven output
  rules.
- The publish list relies on Narratio's native destination derivation and
  retains `deranged_rumors` as optional while the other six outputs are
  required.
- Spool-audio deletion remains unset by the pipeline role, and the testing
  profile introduces no storage or publication override.
- `dnd_pipeline_secrets` remains a generic empty-by-default mapping; the role
  documents all four expected operational entries without embedding any
  credential value.
- No new production task, template, handler, fact, state file, wrapper, or
  storage-validation command is introduced.
- The standalone Narratio role remains unchanged because its `v0.1.0` role
  interface can install Narratio v2.1.1, renders the required native fields,
  and preserves null omission.
- The existing pipeline integration scenario explicitly supplies
  representative S3 and publish inputs, resolves synthetic values for all four
  typed credentials, passes Narratio's native configuration validation, and
  exposes no synthetic value in captured output.
- The immediate identical second pipeline application reports `changed=0`
  and `failed=0`; syntax checking, production-profile lint, the complete
  standalone integration suite, and `git diff --check` pass.
- The exact reviewed pipeline commit is pushed to public Gitea before the
  collection imports it through the documented remote-only squashed subtree
  workflow.
- The collection fixture supplies the added synthetic credential names, and
  its existing build, isolated installation, FQCN resolution, native
  validation, and clean-repeat suite passes without a duplicate S3 behavior
  test.
- No real object-storage request, paid model request, application repository
  change during the remaining implementation, asset change, new release tag,
  or published artifact is required. The existing Narratio v2.1.1 release is a
  prerequisite rather than an implementation deliverable.
- Pipeline documentation links operators to Narratio's canonical procedures
  for S3 session audio, publish, restore, locks, status, and cleanup and no
  longer describes storage and publishing as deferred.
- Documentation distinguishes accepted Notarius extraction sources from
  published outputs and confirms that future analyze-stage artifacts may use
  `narratio.extraction.<output_key>` inputs without publishing those lanes.

## Dependencies and Follow-up Work

Features 0009, 0011, 0012, 0013, and 0014 and the public Narratio v2.1.1 bugfix
release are prerequisites. In particular, Feature 0011 supplies the credential
directory and Feature 0014 supplies the human-operator access needed for the
eventual live deployment.

[Feature 0016](0016-deploy-dnd-pipeline-to-pipeline-dnd.md) follows this work.
It will put the four actual values in encrypted host inventory, pin the exact
reviewed suite revision in `rakestraw-ansible`, converge `pipeline-dnd`, and
exercise one real session through remote publication. Bucket lifecycle policy,
backup policy, credential rotation, additional publish outputs, automated
scheduling, and cleanup of the proof-of-concept deployment remain separate
future work.

A later artifact-development feature may add D&D-specific Narratio artifact
definitions whose Scriptorium input mappings consume selected
`narratio.extraction.*` sources. That feature should define each artifact's
purpose, inputs, prompt, output contract, and publication choice; it does not
require a storage-interface change.