ansible-workspace-dnd/docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md

Feature 0015: Enable D&D S3 Storage and Publishing

Status: done

Summary

Make the D&D pipeline operational with Narratio's current S3 storage, publication, caching, cleanup, and restore capabilities. The pipeline role will select the established Backblaze B2 destination, reference two typed object-storage credentials delivered through its existing secrets directory, publish the useful transcript and D&D artifact outputs, and retain maximum S3 audio caching.

The standalone Narratio role already exposes Narratio v2.1.1's complete configuration surface for this behavior. This feature supplies D&D suite policy through that existing interface; it does not add another storage implementation or credential mechanism.

Problem or Opportunity

The pipeline role currently installs Narratio v2.1.0 and configures the full processing graph, but deliberately leaves Narratio storage and publishing at their application-level defaults. Its README consequently describes storage, publishing, and end-to-end external execution as deferred. The resulting suite can validate configuration and process local state, but it does not select the remote archive or publish the outputs needed for an operational deployment.

Narratio v2.1.0 also rejected a valid publish list when a rule explicitly set required while omitting a derivable dest. Narratio v2.1.1 corrects that resolution bookkeeping defect without changing the publish schema or destination rules. This feature therefore advances the pipeline application pin to v2.1.1 and retains explicit suite-owned requiredness together with native destination derivation.

The prior test-machine configuration records the intended Backblaze B2 endpoint, bucket, prefix, credential names, publication set, and post-publish workspace cleanup. Narratio's current contract has since replaced the legacy credential field names and secrets configuration with ConfigKit typed references and a conventional sibling secrets/ directory. The Ansible suite already implements that new credential-delivery boundary for Audita and Notarius, so S3 needs only to join the same path.

Governing Policies

• The Ansible house style requires suite policy to flow through the standalone role's field-specific interface and optional fields to remain application-owned unless the suite deliberately selects them.
• The testing house style requires representative public-interface and native-application validation without duplicating AWS SDK, ConfigKit, or Ansible module behavior.
• The MVP roadmap makes the pipeline role responsible for cross-application configuration and end-to-end validation.
• Feature 0011 establishes the managed Narratio secrets directory and typed credential-forwarding policy.
• Feature 0012 makes the standalone pipeline repository authoritative and imports reviewed changes through public Gitea.
• Feature 0016 consumes this feature's reviewed interface for the first real-host deployment.

Scope

• Advance the suite's Narratio application version from v2.1.0 to the backward-compatible v2.1.1 bugfix release.

• Add suite-owned pipeline variables for the selected Narratio workspace, S3, cache, and publish settings.

• Select the S3 backend with these D&D deployment defaults:

  Setting	Suite default
  Endpoint	https://s3.us-east-005.backblazeb2.com
  Bucket	rakestraw-assets
  Root prefix	dnd
  Region	us-east-1
  Force path style	true
  Access-key reference	OBJECT_STORAGE_KEY_ID
  Secret-key reference	OBJECT_STORAGE_KEY

• Pass the two credential reference names to the standalone Narratio role. Their values use the existing dnd_pipeline_secrets map and managed /usr/local/etc/narratio/secrets/ directory.

• Enable S3 audio caching explicitly so the suite retains Narratio's maximum available audio reuse.

• Enable run publication and publish these complete native output rules:

  Source	Required
  narratio.transcript.final	true
  narratio.transcript.final_trimmed	true
  narratio.transcript.final_markdown	true
  narratio.transcript.final_trimmed_markdown	true
  narratio.artifact.session_recap	true
  narratio.artifact.hostile_rumors	true
  narratio.artifact.deranged_rumors	false

• Let Narratio derive each destination from the built-in transcript path or configured artifact output path rather than duplicating those paths in the publish list.

• Enable workspace cleanup only after Narratio records successful run publication. Leave spool-audio deletion under Narratio's existing default because the suite has selected no policy requiring source-audio removal.

• Keep the production and testing profile storage and publishing behavior identical. The existing overlay remains limited to model-profile selection.

• Document the four credentials normally required by the operational suite: AUDITA_LLM_API_KEY, OPENROUTER_API_KEY, OBJECT_STORAGE_KEY_ID, and OBJECT_STORAGE_KEY.

• Document the operator-visible publication, restore, S3-session-audio, lock, caching, and cleanup capabilities enabled by the selected configuration, while linking Narratio's canonical CLI and operations documentation for command details.

• Preserve every accepted Notarius lane as its existing narratio.extraction.<output_key> source. Future Narratio artifact definitions can consume those sources during analyze without first making the extraction lanes separate published outputs.

• Adapt the existing pipeline integration scenario to resolve synthetic S3 credentials and validate one explicitly supplied representative storage and publishing configuration through Narratio's native interface without contacting object storage.

• Import the reviewed public pipeline commit into the collection through the established squashed subtree workflow, adapt the collection's artifact-level fixture to supply the added credential names, and validate the assembled artifact.

Non-goals

• Implementing an S3 client, publication engine, remote commit format, restore mechanism, publish-lock mechanism, or audio cache in Ansible. Narratio owns all of those behaviors.
• Placing credential values in role defaults, templates, documentation, tests, ordinary inventory, or the generated pipeline configuration.
• Adding a second secret-delivery mechanism, environment wrapper, service unit, credential synchronizer, or AWS configuration-file manager.
• Managing Backblaze buckets, object lifecycle policy, application keys, retention policy, versioning, encryption, quotas, or network access.
• Managing per-session session.yml files or selecting local versus S3 audio input for an individual session.
• Running a paid or networked publication test from a repository-local test suite. Feature 0016 owns the representative live end-to-end exercise.
• Publishing every Notarius extraction lane as a named current output. The lanes remain available as analyze inputs independently of publication; stable remote destinations require an explicit operator need.
• Setting Narratio's optional notarius.chunk_cache override. The managed Notarius configuration remains the single owner of its ordinary auto cache policy, and Narratio v2.1 already requests refresh for forced extraction.
• Adding notification providers. Narratio v2.1 supports only its existing noop notification mode.
• Releasing new role or collection versions as part of this implementation.
• Modifying Narratio, ConfigKit, or the standalone Narratio role. Narratio v2.1.1 is an already published prerequisite, and the existing generic role can install the selected application tag through narratio_version.

Target End State

The standalone pipeline role exposes these suite variables with the selected defaults:

dnd_pipeline_narratio_version: v2.1.1
dnd_pipeline_narratio_workspace_cleanup_after_publish: true
dnd_pipeline_narratio_storage_backend: s3
dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets
dnd_pipeline_narratio_storage_s3_root_prefix: dnd
dnd_pipeline_narratio_storage_s3_region: us-east-1
dnd_pipeline_narratio_storage_s3_endpoint: >-
  https://s3.us-east-005.backblazeb2.com
dnd_pipeline_narratio_storage_s3_force_path_style: true
dnd_pipeline_narratio_storage_s3_access_key_id: OBJECT_STORAGE_KEY_ID
dnd_pipeline_narratio_storage_s3_secret_access_key: OBJECT_STORAGE_KEY
dnd_pipeline_narratio_cache_s3_audio: true
dnd_pipeline_narratio_publish_enabled: true
dnd_pipeline_narratio_publish_upload_run: true
dnd_pipeline_narratio_publish_outputs:
  - source: narratio.transcript.final
    required: true
  - source: narratio.transcript.final_trimmed
    required: true
  - source: narratio.transcript.final_markdown
    required: true
  - source: narratio.transcript.final_trimmed_markdown
    required: true
  - source: narratio.artifact.session_recap
    required: true
  - source: narratio.artifact.hostile_rumors
    required: true
  - source: narratio.artifact.deranged_rumors
    required: false

Each value is declared in the pipeline argument specification and passed directly to its corresponding narratio_* variable in the pipeline role's Narratio dependency. Narratio's existing root template renders the native configuration. No new pipeline fragment or production task is required.

The ordinary generated configuration contains only the credential names:

storage:
  backend: s3
  s3:
    access_key_id: OBJECT_STORAGE_KEY_ID
    secret_access_key: OBJECT_STORAGE_KEY

Consumers supply the two values beside the existing model-provider values through encrypted inventory:

dnd_pipeline_secrets:
  AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
  OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"
  OBJECT_STORAGE_KEY_ID: "{{ vault_dnd_object_storage_key_id }}"
  OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}"

The example defines references only; no roadmap, role, or test contains real credential values. The existing credential task writes the selected entries with the suite's established group-authorized secret modes. Omitted map keys remain unmanaged, as defined by Feature 0011.

With valid credentials, Narratio can accept either local or S3 audio selected by each session file, cache S3 audio beneath /var/cache/narratio, publish an immutable run and the selected stable outputs, use remote publication locks, restore committed state, and clean the published workspace. These are native Narratio commands and lifecycle rules rather than additional Ansible tasks.

The ten accepted Notarius outputs continue to enter Narratio's artifact catalog as narratio.extraction.* sources. Analyze-stage Scriptorium artifacts may reference those sources directly in their input mappings. Feature 0015 does not yet add such artifact definitions, but its publication choices do not remove, rename, or constrain this future composition path.

Repository Ownership

• ansible-role-dnd-pipeline owns the D&D suite variables, opinionated defaults, dependency mappings, consumer documentation, and representative integration scenario.
• ansible-role-narratio remains the generic owner of installation and native pipeline rendering. Its reviewed v0.1.0 role interface is compatible with Narratio v2.1.1 and already covers every configuration field required by this feature and should remain unchanged.
• ansible-collection-dnd owns the faithful pipeline subtree import, the unified consumer dependency manifest, and artifact-boundary validation.
• ansible-workspace-dnd owns this roadmap, the later implementation plan, status, and independent review record.
• narratio v2.1.1 is the released compatibility prerequisite and a read-only source of storage, publication, credential, caching, cleanup, restore, and CLI contracts during this implementation.
• rakestraw-ansible remains unchanged until Feature 0016 consumes this interface on pipeline-dnd.

Constraints and Settled Decisions

• The S3 endpoint, bucket, prefix, region, and path-style values are deliberate D&D suite defaults. They are ordinary variables and remain overrideable by inventory.
• Credential references are also policy defaults, but credential values have no defaults. Operators must supply matching entries through dnd_pipeline_secrets or another environment source supported by Narratio.
• The two credential-reference variables are nullable as a pair. Setting both to null omits the references and preserves Narratio's AWS SDK provider-chain path; configuring only one remains invalid under Narratio's native contract.
• The role does not assert that the secrets map contains particular keys. Narratio's native loader owns typed-reference resolution, and external environment values remain a supported higher-precedence source.
• dnd_pipeline_narratio_publish_outputs is one complete native list. A caller replacing it assumes responsibility for the entire selected publication set; the role does not merge rules.
• Publish destinations remain omitted where Narratio can derive them. The configured source identity and artifact output path remain the single source of truth.
• Narratio v2.1.1 is the minimum application version for this feature because it preserves destination derivation when required is explicit. Do not work around older behavior by duplicating destinations or omitting deliberate requiredness.
• Workspace cleanup is gated by Narratio's own successful-publication evidence. Ansible does not independently inspect remote state or remove runtime content.
• S3 audio caching is enabled, and ordinary cleanup preserves the cache. Explicit operator cache clearing remains a Narratio CLI operation.
• The testing profile does not silently disable Notarius, storage, publication, cleanup, or credentials. It continues to differ only in the already selected inexpensive Scriptorium model profile.
• S3 storage also enables Narratio's native restore and publication-lock commands without another configuration surface. Ansible documentation may describe availability but must not duplicate Narratio's detailed operating manual.
• Notarius extraction-source availability and remote publication are separate concerns. Analyze consumes narratio.extraction.* sources from the accepted extraction catalog; publish.outputs selects only which sources also receive stable published destinations.

Simplest Viable Implementation

Advance dnd_pipeline_narratio_version to v2.1.1. Add the selected variables to the pipeline defaults and argument specification, then map them into the existing rakestraw.narratio dependency. Update the pipeline README to document the policy, credential names, complete-list replacement semantics, and native capabilities now available. Do not add a template, task, handler, fact, probe, or storage-specific validation layer.

In the existing pipeline integration driver, extend the synthetic secrets map with object-storage values and explicitly supply representative S3 and publish variables. Use narratio config validate and its secret-safe effective configuration output to prove that the public inputs compose into a valid configuration. Retain the immediate clean repeat and secret non-disclosure checks. Do not call a real S3 service or assert that the representative values equal current role defaults.

Push the reviewed pipeline commit, import that exact public revision into the collection through git subtree pull --squash, and extend only the collection-level fixture inputs needed for native validation. The collection suite should continue to prove artifact installation and one representative assembled convergence rather than duplicate the detailed pipeline matrix.

Rationale and Alternatives

Implementing storage behavior in Ansible would duplicate a mature native application boundary. Narratio already owns credential resolution, AWS SDK provider construction, immutable commits, verification, locks, restore, caching, and cleanup. Passing declarative configuration through the generic role is both smaller and more reliable.

Using the AWS SDK's ambient provider chain would avoid named secret files, but the initial deployment has explicit Backblaze application credentials and the suite already has a suitable managed typed-secret path. Keeping the two credential-reference variables nullable preserves the ambient-provider alternative: consumers can set both suite variables to null when their deployment uses another provider.

Copying destinations into every publish rule would make the output layout more visible locally, but it would duplicate paths already owned by Narratio's built-in sources and the configured artifact definitions. Native destination derivation keeps path coordination automatic.

Omitting explicit required: true values would avoid the v2.1.0 failure only by relying on an unrelated defaulting path. Changing the generic role's YAML format would not repair the underlying sequence-resolution mismatch. Narratio v2.1.1 fixes the owning application boundary, allowing the suite to state its requiredness policy directly without duplicating destinations.

Deleting spool audio after publication would minimize transient disk usage, but no such retention policy has been selected, and local-audio sessions may depend on that content. Leaving the field omitted preserves Narratio's default while enabling the explicitly requested S3 audio cache.

Publishing all ten Notarius extraction lanes would create stable remote output contracts that have not been requested. It is also unnecessary for the stated future use: analyze-stage artifacts can consume narratio.extraction.* directly. The immutable published run retains the processing result, while the selected transcript and human-facing artifact outputs match the known operational workflow.

Acceptance Criteria

• The pipeline role declares and documents every suite variable shown in the target state, with matching argument-spec entries and direct Narratio dependency mappings.
• The pipeline installs Narratio v2.1.1, whose released configuration loader supports explicit requiredness together with derived publish destinations.
• The two credential-reference variables accept paired null overrides so a caller can deliberately select Narratio's ambient AWS SDK provider chain.
• The default effective Narratio configuration selects the specified Backblaze-compatible S3 destination, typed credential names, S3 audio cache, successful-publication workspace cleanup, run upload, and seven output rules.
• The publish list relies on Narratio's native destination derivation and retains deranged_rumors as optional while the other six outputs are required.
• Spool-audio deletion remains unset by the pipeline role, and the testing profile introduces no storage or publication override.
• dnd_pipeline_secrets remains a generic empty-by-default mapping; the role documents all four expected operational entries without embedding any credential value.
• No new production task, template, handler, fact, state file, wrapper, or storage-validation command is introduced.
• The standalone Narratio role remains unchanged because its v0.1.0 role interface can install Narratio v2.1.1, renders the required native fields, and preserves null omission.
• The existing pipeline integration scenario explicitly supplies representative S3 and publish inputs, resolves synthetic values for all four typed credentials, passes Narratio's native configuration validation, and exposes no synthetic value in captured output.
• The immediate identical second pipeline application reports changed=0 and failed=0; syntax checking, production-profile lint, the complete standalone integration suite, and git diff --check pass.
• The exact reviewed pipeline commit is pushed to public Gitea before the collection imports it through the documented remote-only squashed subtree workflow.
• The collection fixture supplies the added synthetic credential names, and its existing build, isolated installation, FQCN resolution, native validation, and clean-repeat suite passes without a duplicate S3 behavior test.
• No real object-storage request, paid model request, application repository change during the remaining implementation, asset change, new release tag, or published artifact is required. The existing Narratio v2.1.1 release is a prerequisite rather than an implementation deliverable.
• Pipeline documentation links operators to Narratio's canonical procedures for S3 session audio, publish, restore, locks, status, and cleanup and no longer describes storage and publishing as deferred.
• Documentation distinguishes accepted Notarius extraction sources from published outputs and confirms that future analyze-stage artifacts may use narratio.extraction.<output_key> inputs without publishing those lanes.

Dependencies and Follow-up Work

Features 0009, 0011, 0012, 0013, and 0014 and the public Narratio v2.1.1 bugfix release are prerequisites. In particular, Feature 0011 supplies the credential directory and Feature 0014 supplies the human-operator access needed for the eventual live deployment.

Feature 0016 follows this work. It will put the four actual values in encrypted host inventory, pin the exact reviewed suite revision in rakestraw-ansible, converge pipeline-dnd, and exercise one real session through remote publication. Bucket lifecycle policy, backup policy, credential rotation, additional publish outputs, automated scheduling, and cleanup of the proof-of-concept deployment remain separate future work.

A later artifact-development feature may add D&D-specific Narratio artifact definitions whose Scriptorium input mappings consume selected narratio.extraction.* sources. That feature should define each artifact's purpose, inputs, prompt, output contract, and publication choice; it does not require a storage-interface change.