Plan D&D S3 storage and publishing
This commit is contained in:
392
docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md
Normal file
392
docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md
Normal file
@@ -0,0 +1,392 @@
|
||||
# Feature 0015: Enable D&D S3 Storage and Publishing
|
||||
|
||||
Status: ready
|
||||
|
||||
## Summary
|
||||
|
||||
Make the D&D pipeline operational with Narratio's current S3 storage,
|
||||
publication, caching, cleanup, and restore capabilities. The pipeline role
|
||||
will select the established Backblaze B2 destination, reference two typed
|
||||
object-storage credentials delivered through its existing secrets directory,
|
||||
publish the useful transcript and D&D artifact outputs, and retain maximum S3
|
||||
audio caching.
|
||||
|
||||
The standalone Narratio role already exposes Narratio v2.1.0's complete
|
||||
configuration surface for this behavior. This feature supplies D&D suite
|
||||
policy through that existing interface; it does not add another storage
|
||||
implementation or credential mechanism.
|
||||
|
||||
## Problem or Opportunity
|
||||
|
||||
The pipeline role currently installs Narratio v2.1.0 and configures the full
|
||||
processing graph, but deliberately leaves Narratio storage and publishing at
|
||||
their application-level defaults. Its README consequently describes storage,
|
||||
publishing, and end-to-end external execution as deferred. The resulting suite
|
||||
can validate configuration and process local state, but it does not select the
|
||||
remote archive or publish the outputs needed for an operational deployment.
|
||||
|
||||
The prior test-machine configuration records the intended Backblaze B2
|
||||
endpoint, bucket, prefix, credential names, publication set, and post-publish
|
||||
workspace cleanup. Narratio's current contract has since replaced the legacy
|
||||
credential field names and secrets configuration with ConfigKit typed
|
||||
references and a conventional sibling `secrets/` directory. The Ansible suite
|
||||
already implements that new credential-delivery boundary for Audita and
|
||||
Notarius, so S3 needs only to join the same path.
|
||||
|
||||
## Governing Policies
|
||||
|
||||
- [The Ansible house style](../../ansible.md) requires suite policy to flow
|
||||
through the standalone role's field-specific interface and optional fields
|
||||
to remain application-owned unless the suite deliberately selects them.
|
||||
- [The testing house style](../../testing.md) requires representative
|
||||
public-interface and native-application validation without duplicating AWS
|
||||
SDK, ConfigKit, or Ansible module behavior.
|
||||
- [The MVP roadmap](../mvp.md) makes the pipeline role responsible for
|
||||
cross-application configuration and end-to-end validation.
|
||||
- [Feature 0011](0011-managed-pipeline-credential-delivery.md) establishes the
|
||||
managed Narratio secrets directory and typed credential-forwarding policy.
|
||||
- [Feature 0012](0012-collection-assembly.md) makes the standalone pipeline
|
||||
repository authoritative and imports reviewed changes through public
|
||||
Gitea.
|
||||
- [Feature 0016](0016-deploy-dnd-pipeline-to-pipeline-dnd.md) consumes this
|
||||
feature's reviewed interface for the first real-host deployment.
|
||||
|
||||
## Scope
|
||||
|
||||
- Add suite-owned pipeline variables for the selected Narratio workspace,
|
||||
S3, cache, and publish settings.
|
||||
- Select the S3 backend with these D&D deployment defaults:
|
||||
|
||||
| Setting | Suite default |
|
||||
| --- | --- |
|
||||
| Endpoint | `https://s3.us-east-005.backblazeb2.com` |
|
||||
| Bucket | `rakestraw-assets` |
|
||||
| Root prefix | `dnd` |
|
||||
| Region | `us-east-1` |
|
||||
| Force path style | `true` |
|
||||
| Access-key reference | `OBJECT_STORAGE_KEY_ID` |
|
||||
| Secret-key reference | `OBJECT_STORAGE_KEY` |
|
||||
|
||||
- Pass the two credential reference names to the standalone Narratio role.
|
||||
Their values use the existing `dnd_pipeline_secrets` map and managed
|
||||
`/usr/local/etc/narratio/secrets/` directory.
|
||||
- Enable S3 audio caching explicitly so the suite retains Narratio's maximum
|
||||
available audio reuse.
|
||||
- Enable run publication and publish these complete native output rules:
|
||||
|
||||
| Source | Required |
|
||||
| --- | --- |
|
||||
| `narratio.transcript.final` | `true` |
|
||||
| `narratio.transcript.final_trimmed` | `true` |
|
||||
| `narratio.transcript.final_markdown` | `true` |
|
||||
| `narratio.transcript.final_trimmed_markdown` | `true` |
|
||||
| `narratio.artifact.session_recap` | `true` |
|
||||
| `narratio.artifact.hostile_rumors` | `true` |
|
||||
| `narratio.artifact.deranged_rumors` | `false` |
|
||||
|
||||
- Let Narratio derive each destination from the built-in transcript path or
|
||||
configured artifact output path rather than duplicating those paths in the
|
||||
publish list.
|
||||
- Enable workspace cleanup only after Narratio records successful run
|
||||
publication. Leave spool-audio deletion under Narratio's existing default
|
||||
because the suite has selected no policy requiring source-audio removal.
|
||||
- Keep the production and `testing` profile storage and publishing behavior
|
||||
identical. The existing overlay remains limited to model-profile selection.
|
||||
- Document the four credentials normally required by the operational suite:
|
||||
`AUDITA_LLM_API_KEY`, `OPENROUTER_API_KEY`,
|
||||
`OBJECT_STORAGE_KEY_ID`, and `OBJECT_STORAGE_KEY`.
|
||||
- Document the operator-visible publication, restore, S3-session-audio, lock,
|
||||
caching, and cleanup capabilities enabled by the selected configuration,
|
||||
while linking Narratio's canonical CLI and operations documentation for
|
||||
command details.
|
||||
- Preserve every accepted Notarius lane as its existing
|
||||
`narratio.extraction.<output_key>` source. Future Narratio artifact
|
||||
definitions can consume those sources during analyze without first making
|
||||
the extraction lanes separate published outputs.
|
||||
- Adapt the existing pipeline integration scenario to resolve synthetic S3
|
||||
credentials and validate one explicitly supplied representative storage and
|
||||
publishing configuration through Narratio's native interface without
|
||||
contacting object storage.
|
||||
- Import the reviewed public pipeline commit into the collection through the
|
||||
established squashed subtree workflow, adapt the collection's artifact-level
|
||||
fixture to supply the added credential names, and validate the assembled
|
||||
artifact.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Implementing an S3 client, publication engine, remote commit format,
|
||||
restore mechanism, publish-lock mechanism, or audio cache in Ansible.
|
||||
Narratio owns all of those behaviors.
|
||||
- Placing credential values in role defaults, templates, documentation,
|
||||
tests, ordinary inventory, or the generated pipeline configuration.
|
||||
- Adding a second secret-delivery mechanism, environment wrapper, service
|
||||
unit, credential synchronizer, or AWS configuration-file manager.
|
||||
- Managing Backblaze buckets, object lifecycle policy, application keys,
|
||||
retention policy, versioning, encryption, quotas, or network access.
|
||||
- Managing per-session `session.yml` files or selecting local versus S3 audio
|
||||
input for an individual session.
|
||||
- Running a paid or networked publication test from a repository-local test
|
||||
suite. Feature 0016 owns the representative live end-to-end exercise.
|
||||
- Publishing every Notarius extraction lane as a named current output. The
|
||||
lanes remain available as analyze inputs independently of publication; stable
|
||||
remote destinations require an explicit operator need.
|
||||
- Setting Narratio's optional `notarius.chunk_cache` override. The managed
|
||||
Notarius configuration remains the single owner of its ordinary `auto`
|
||||
cache policy, and Narratio v2.1 already requests refresh for forced
|
||||
extraction.
|
||||
- Adding notification providers. Narratio v2.1 supports only its existing
|
||||
`noop` notification mode.
|
||||
- Releasing new role or collection versions as part of this implementation.
|
||||
|
||||
## Target End State
|
||||
|
||||
The standalone pipeline role exposes these suite variables with the selected
|
||||
defaults:
|
||||
|
||||
```yaml
|
||||
dnd_pipeline_narratio_workspace_cleanup_after_publish: true
|
||||
dnd_pipeline_narratio_storage_backend: s3
|
||||
dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets
|
||||
dnd_pipeline_narratio_storage_s3_root_prefix: dnd
|
||||
dnd_pipeline_narratio_storage_s3_region: us-east-1
|
||||
dnd_pipeline_narratio_storage_s3_endpoint: >-
|
||||
https://s3.us-east-005.backblazeb2.com
|
||||
dnd_pipeline_narratio_storage_s3_force_path_style: true
|
||||
dnd_pipeline_narratio_storage_s3_access_key_id: OBJECT_STORAGE_KEY_ID
|
||||
dnd_pipeline_narratio_storage_s3_secret_access_key: OBJECT_STORAGE_KEY
|
||||
dnd_pipeline_narratio_cache_s3_audio: true
|
||||
dnd_pipeline_narratio_publish_enabled: true
|
||||
dnd_pipeline_narratio_publish_upload_run: true
|
||||
dnd_pipeline_narratio_publish_outputs:
|
||||
- source: narratio.transcript.final
|
||||
required: true
|
||||
- source: narratio.transcript.final_trimmed
|
||||
required: true
|
||||
- source: narratio.transcript.final_markdown
|
||||
required: true
|
||||
- source: narratio.transcript.final_trimmed_markdown
|
||||
required: true
|
||||
- source: narratio.artifact.session_recap
|
||||
required: true
|
||||
- source: narratio.artifact.hostile_rumors
|
||||
required: true
|
||||
- source: narratio.artifact.deranged_rumors
|
||||
required: false
|
||||
```
|
||||
|
||||
Each value is declared in the pipeline argument specification and passed
|
||||
directly to its corresponding `narratio_*` variable in the pipeline role's
|
||||
Narratio dependency. Narratio's existing root template renders the native
|
||||
configuration. No new pipeline fragment or production task is required.
|
||||
|
||||
The ordinary generated configuration contains only the credential names:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
access_key_id: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key: OBJECT_STORAGE_KEY
|
||||
```
|
||||
|
||||
Consumers supply the two values beside the existing model-provider values
|
||||
through encrypted inventory:
|
||||
|
||||
```yaml
|
||||
dnd_pipeline_secrets:
|
||||
AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
|
||||
OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"
|
||||
OBJECT_STORAGE_KEY_ID: "{{ vault_dnd_object_storage_key_id }}"
|
||||
OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}"
|
||||
```
|
||||
|
||||
The example defines references only; no roadmap, role, or test contains real
|
||||
credential values. The existing credential task writes the selected entries
|
||||
with the suite's established group-authorized secret modes. Omitted map keys
|
||||
remain unmanaged, as defined by Feature 0011.
|
||||
|
||||
With valid credentials, Narratio can accept either local or S3 audio selected
|
||||
by each session file, cache S3 audio beneath `/var/cache/narratio`, publish an
|
||||
immutable run and the selected stable outputs, use remote publication locks,
|
||||
restore committed state, and clean the published workspace. These are native
|
||||
Narratio commands and lifecycle rules rather than additional Ansible tasks.
|
||||
|
||||
The ten accepted Notarius outputs continue to enter Narratio's artifact
|
||||
catalog as `narratio.extraction.*` sources. Analyze-stage Scriptorium artifacts
|
||||
may reference those sources directly in their input mappings. Feature 0015
|
||||
does not yet add such artifact definitions, but its publication choices do not
|
||||
remove, rename, or constrain this future composition path.
|
||||
|
||||
## Repository Ownership
|
||||
|
||||
- `ansible-role-dnd-pipeline` owns the D&D suite variables, opinionated
|
||||
defaults, dependency mappings, consumer documentation, and representative
|
||||
integration scenario.
|
||||
- `ansible-role-narratio` remains the generic owner of installation and native
|
||||
pipeline rendering. Its reviewed v2.1.0 interface already covers every
|
||||
configuration field required by this feature and should remain unchanged.
|
||||
- `ansible-collection-dnd` owns the faithful pipeline subtree import, the
|
||||
unified consumer dependency manifest, and artifact-boundary validation.
|
||||
- `ansible-workspace-dnd` owns this roadmap, the later implementation plan,
|
||||
status, and independent review record.
|
||||
- `narratio` is a read-only source of storage, publication, credential,
|
||||
caching, cleanup, restore, and CLI contracts.
|
||||
- `rakestraw-ansible` remains unchanged until Feature 0016 consumes this
|
||||
interface on `pipeline-dnd`.
|
||||
|
||||
## Constraints and Settled Decisions
|
||||
|
||||
- The S3 endpoint, bucket, prefix, region, and path-style values are deliberate
|
||||
D&D suite defaults. They are ordinary variables and remain overrideable by
|
||||
inventory.
|
||||
- Credential references are also policy defaults, but credential values have
|
||||
no defaults. Operators must supply matching entries through
|
||||
`dnd_pipeline_secrets` or another environment source supported by Narratio.
|
||||
- The two credential-reference variables are nullable as a pair. Setting both
|
||||
to `null` omits the references and preserves Narratio's AWS SDK provider-chain
|
||||
path; configuring only one remains invalid under Narratio's native contract.
|
||||
- The role does not assert that the secrets map contains particular keys.
|
||||
Narratio's native loader owns typed-reference resolution, and external
|
||||
environment values remain a supported higher-precedence source.
|
||||
- `dnd_pipeline_narratio_publish_outputs` is one complete native list. A caller
|
||||
replacing it assumes responsibility for the entire selected publication
|
||||
set; the role does not merge rules.
|
||||
- Publish destinations remain omitted where Narratio can derive them. The
|
||||
configured source identity and artifact output path remain the single source
|
||||
of truth.
|
||||
- Workspace cleanup is gated by Narratio's own successful-publication
|
||||
evidence. Ansible does not independently inspect remote state or remove
|
||||
runtime content.
|
||||
- S3 audio caching is enabled, and ordinary cleanup preserves the cache.
|
||||
Explicit operator cache clearing remains a Narratio CLI operation.
|
||||
- The testing profile does not silently disable Notarius, storage,
|
||||
publication, cleanup, or credentials. It continues to differ only in the
|
||||
already selected inexpensive Scriptorium model profile.
|
||||
- S3 storage also enables Narratio's native restore and publication-lock
|
||||
commands without another configuration surface. Ansible documentation may
|
||||
describe availability but must not duplicate Narratio's detailed operating
|
||||
manual.
|
||||
- Notarius extraction-source availability and remote publication are separate
|
||||
concerns. Analyze consumes `narratio.extraction.*` sources from the accepted
|
||||
extraction catalog; `publish.outputs` selects only which sources also receive
|
||||
stable published destinations.
|
||||
|
||||
## Simplest Viable Implementation
|
||||
|
||||
Add the selected variables to the pipeline defaults and argument specification,
|
||||
then map them into the existing `rakestraw.narratio` dependency. Update the
|
||||
pipeline README to document the policy, credential names, complete-list
|
||||
replacement semantics, and native capabilities now available. Do not add a
|
||||
template, task, handler, fact, probe, or storage-specific validation layer.
|
||||
|
||||
In the existing pipeline integration driver, extend the synthetic secrets map
|
||||
with object-storage values and explicitly supply representative S3 and publish
|
||||
variables. Use `narratio config validate` and its secret-safe effective
|
||||
configuration output to prove that the public inputs compose into a valid
|
||||
configuration. Retain the immediate clean repeat and secret non-disclosure
|
||||
checks. Do not call a real S3 service or assert that the representative values
|
||||
equal current role defaults.
|
||||
|
||||
Push the reviewed pipeline commit, import that exact public revision into the
|
||||
collection through `git subtree pull --squash`, and extend only the
|
||||
collection-level fixture inputs needed for native validation. The collection
|
||||
suite should continue to prove artifact installation and one representative
|
||||
assembled convergence rather than duplicate the detailed pipeline matrix.
|
||||
|
||||
## Rationale and Alternatives
|
||||
|
||||
Implementing storage behavior in Ansible would duplicate a mature native
|
||||
application boundary. Narratio already owns credential resolution, AWS SDK
|
||||
provider construction, immutable commits, verification, locks, restore,
|
||||
caching, and cleanup. Passing declarative configuration through the generic
|
||||
role is both smaller and more reliable.
|
||||
|
||||
Using the AWS SDK's ambient provider chain would avoid named secret files, but
|
||||
the initial deployment has explicit Backblaze application credentials and the
|
||||
suite already has a suitable managed typed-secret path. Keeping the two
|
||||
credential-reference variables nullable preserves the ambient-provider
|
||||
alternative: consumers can set both suite variables to `null` when their
|
||||
deployment uses another provider.
|
||||
|
||||
Copying destinations into every publish rule would make the output layout more
|
||||
visible locally, but it would duplicate paths already owned by Narratio's
|
||||
built-in sources and the configured artifact definitions. Native destination
|
||||
derivation keeps path coordination automatic.
|
||||
|
||||
Deleting spool audio after publication would minimize transient disk usage,
|
||||
but no such retention policy has been selected, and local-audio sessions may
|
||||
depend on that content. Leaving the field omitted preserves Narratio's default
|
||||
while enabling the explicitly requested S3 audio cache.
|
||||
|
||||
Publishing all ten Notarius extraction lanes would create stable remote output
|
||||
contracts that have not been requested. It is also unnecessary for the stated
|
||||
future use: analyze-stage artifacts can consume `narratio.extraction.*`
|
||||
directly. The immutable published run retains the processing result, while the
|
||||
selected transcript and human-facing artifact outputs match the known
|
||||
operational workflow.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- The pipeline role declares and documents every suite variable shown in the
|
||||
target state, with matching argument-spec entries and direct Narratio
|
||||
dependency mappings.
|
||||
- The two credential-reference variables accept paired `null` overrides so a
|
||||
caller can deliberately select Narratio's ambient AWS SDK provider chain.
|
||||
- The default effective Narratio configuration selects the specified
|
||||
Backblaze-compatible S3 destination, typed credential names, S3 audio cache,
|
||||
successful-publication workspace cleanup, run upload, and seven output
|
||||
rules.
|
||||
- The publish list relies on Narratio's native destination derivation and
|
||||
retains `deranged_rumors` as optional while the other six outputs are
|
||||
required.
|
||||
- Spool-audio deletion remains unset by the pipeline role, and the testing
|
||||
profile introduces no storage or publication override.
|
||||
- `dnd_pipeline_secrets` remains a generic empty-by-default mapping; the role
|
||||
documents all four expected operational entries without embedding any
|
||||
credential value.
|
||||
- No new production task, template, handler, fact, state file, wrapper, or
|
||||
storage-validation command is introduced.
|
||||
- The standalone Narratio role remains unchanged because its v2.1.0 interface
|
||||
already renders the required native fields and preserves null omission.
|
||||
- The existing pipeline integration scenario explicitly supplies
|
||||
representative S3 and publish inputs, resolves synthetic values for all four
|
||||
typed credentials, passes Narratio's native configuration validation, and
|
||||
exposes no synthetic value in captured output.
|
||||
- The immediate identical second pipeline application reports `changed=0`
|
||||
and `failed=0`; syntax checking, production-profile lint, the complete
|
||||
standalone integration suite, and `git diff --check` pass.
|
||||
- The exact reviewed pipeline commit is pushed to public Gitea before the
|
||||
collection imports it through the documented remote-only squashed subtree
|
||||
workflow.
|
||||
- The collection fixture supplies the added synthetic credential names, and
|
||||
its existing build, isolated installation, FQCN resolution, native
|
||||
validation, and clean-repeat suite passes without a duplicate S3 behavior
|
||||
test.
|
||||
- No real object-storage request, paid model request, application repository
|
||||
change, asset change, release tag, or published artifact is required.
|
||||
- Pipeline documentation links operators to Narratio's canonical procedures
|
||||
for S3 session audio, publish, restore, locks, status, and cleanup and no
|
||||
longer describes storage and publishing as deferred.
|
||||
- Documentation distinguishes accepted Notarius extraction sources from
|
||||
published outputs and confirms that future analyze-stage artifacts may use
|
||||
`narratio.extraction.<output_key>` inputs without publishing those lanes.
|
||||
|
||||
## Dependencies and Follow-up Work
|
||||
|
||||
Features 0009, 0011, 0012, 0013, and 0014 are prerequisites. In particular,
|
||||
Feature 0011 supplies the credential directory and Feature 0014 supplies the
|
||||
human-operator access needed for the eventual live deployment.
|
||||
|
||||
[Feature 0016](0016-deploy-dnd-pipeline-to-pipeline-dnd.md) follows this work.
|
||||
It will put the four actual values in encrypted host inventory, pin the exact
|
||||
reviewed suite revision in `rakestraw-ansible`, converge `pipeline-dnd`, and
|
||||
exercise one real session through remote publication. Bucket lifecycle policy,
|
||||
backup policy, credential rotation, additional publish outputs, automated
|
||||
scheduling, and cleanup of the proof-of-concept deployment remain separate
|
||||
future work.
|
||||
|
||||
A later artifact-development feature may add D&D-specific Narratio artifact
|
||||
definitions whose Scriptorium input mappings consume selected
|
||||
`narratio.extraction.*` sources. That feature should define each artifact's
|
||||
purpose, inputs, prompt, output contract, and publication choice; it does not
|
||||
require a storage-interface change.
|
||||
Reference in New Issue
Block a user