eric/ansible-workspace-dnd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan D&D S3 storage and publishing

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-21 02:20:57 +00:00
parent ec69f5a03b
commit e262b61292
6 changed files with 1023 additions and 181 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md
View File

@@ -1,6 +1,6 @@
# Feature 0014: Enroll Existing Pipeline Operators # Feature 0014: Enroll Existing Pipeline Operators
Status: in_progress Status: done
## Summary ## Summary

392
docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md Normal file
View File

@@ -0,0 +1,392 @@
# Feature 0015: Enable D&D S3 Storage and Publishing
Status: ready
## Summary
Make the D&D pipeline operational with Narratio's current S3 storage,
publication, caching, cleanup, and restore capabilities. The pipeline role
will select the established Backblaze B2 destination, reference two typed
object-storage credentials delivered through its existing secrets directory,
publish the useful transcript and D&D artifact outputs, and retain maximum S3
audio caching.
The standalone Narratio role already exposes Narratio v2.1.0's complete
configuration surface for this behavior. This feature supplies D&D suite
policy through that existing interface; it does not add another storage
implementation or credential mechanism.
## Problem or Opportunity
The pipeline role currently installs Narratio v2.1.0 and configures the full
processing graph, but deliberately leaves Narratio storage and publishing at
their application-level defaults. Its README consequently describes storage,
publishing, and end-to-end external execution as deferred. The resulting suite
can validate configuration and process local state, but it does not select the
remote archive or publish the outputs needed for an operational deployment.
The prior test-machine configuration records the intended Backblaze B2
endpoint, bucket, prefix, credential names, publication set, and post-publish
workspace cleanup. Narratio's current contract has since replaced the legacy
credential field names and secrets configuration with ConfigKit typed
references and a conventional sibling `secrets/` directory. The Ansible suite
already implements that new credential-delivery boundary for Audita and
Notarius, so S3 needs only to join the same path.
## Governing Policies
- [The Ansible house style](../../ansible.md) requires suite policy to flow
through the standalone role's field-specific interface and optional fields
to remain application-owned unless the suite deliberately selects them.
- [The testing house style](../../testing.md) requires representative
public-interface and native-application validation without duplicating AWS
SDK, ConfigKit, or Ansible module behavior.
- [The MVP roadmap](../mvp.md) makes the pipeline role responsible for
cross-application configuration and end-to-end validation.
- [Feature 0011](0011-managed-pipeline-credential-delivery.md) establishes the
managed Narratio secrets directory and typed credential-forwarding policy.
- [Feature 0012](0012-collection-assembly.md) makes the standalone pipeline
repository authoritative and imports reviewed changes through public
Gitea.
- [Feature 0016](0016-deploy-dnd-pipeline-to-pipeline-dnd.md) consumes this
feature's reviewed interface for the first real-host deployment.
## Scope
- Add suite-owned pipeline variables for the selected Narratio workspace,
S3, cache, and publish settings.
- Select the S3 backend with these D&D deployment defaults:
| Setting | Suite default |
| --- | --- |
| Endpoint | `https://s3.us-east-005.backblazeb2.com` |
| Bucket | `rakestraw-assets` |
| Root prefix | `dnd` |
| Region | `us-east-1` |
| Force path style | `true` |
| Access-key reference | `OBJECT_STORAGE_KEY_ID` |
| Secret-key reference | `OBJECT_STORAGE_KEY` |
- Pass the two credential reference names to the standalone Narratio role.
Their values use the existing `dnd_pipeline_secrets` map and managed
`/usr/local/etc/narratio/secrets/` directory.
- Enable S3 audio caching explicitly so the suite retains Narratio's maximum
available audio reuse.
- Enable run publication and publish these complete native output rules:
| Source | Required |
| --- | --- |
| `narratio.transcript.final` | `true` |
| `narratio.transcript.final_trimmed` | `true` |
| `narratio.transcript.final_markdown` | `true` |
| `narratio.transcript.final_trimmed_markdown` | `true` |
| `narratio.artifact.session_recap` | `true` |
| `narratio.artifact.hostile_rumors` | `true` |
| `narratio.artifact.deranged_rumors` | `false` |
- Let Narratio derive each destination from the built-in transcript path or
configured artifact output path rather than duplicating those paths in the
publish list.
- Enable workspace cleanup only after Narratio records successful run
publication. Leave spool-audio deletion under Narratio's existing default
because the suite has selected no policy requiring source-audio removal.
- Keep the production and `testing` profile storage and publishing behavior
identical. The existing overlay remains limited to model-profile selection.
- Document the four credentials normally required by the operational suite:
`AUDITA_LLM_API_KEY`, `OPENROUTER_API_KEY`,
`OBJECT_STORAGE_KEY_ID`, and `OBJECT_STORAGE_KEY`.
- Document the operator-visible publication, restore, S3-session-audio, lock,
caching, and cleanup capabilities enabled by the selected configuration,
while linking Narratio's canonical CLI and operations documentation for
command details.
- Preserve every accepted Notarius lane as its existing
`narratio.extraction.<output_key>` source. Future Narratio artifact
definitions can consume those sources during analyze without first making
the extraction lanes separate published outputs.
- Adapt the existing pipeline integration scenario to resolve synthetic S3
credentials and validate one explicitly supplied representative storage and
publishing configuration through Narratio's native interface without
contacting object storage.
- Import the reviewed public pipeline commit into the collection through the
established squashed subtree workflow, adapt the collection's artifact-level
fixture to supply the added credential names, and validate the assembled
artifact.
## Non-goals
- Implementing an S3 client, publication engine, remote commit format,
restore mechanism, publish-lock mechanism, or audio cache in Ansible.
Narratio owns all of those behaviors.
- Placing credential values in role defaults, templates, documentation,
tests, ordinary inventory, or the generated pipeline configuration.
- Adding a second secret-delivery mechanism, environment wrapper, service
unit, credential synchronizer, or AWS configuration-file manager.
- Managing Backblaze buckets, object lifecycle policy, application keys,
retention policy, versioning, encryption, quotas, or network access.
- Managing per-session `session.yml` files or selecting local versus S3 audio
input for an individual session.
- Running a paid or networked publication test from a repository-local test
suite. Feature 0016 owns the representative live end-to-end exercise.
- Publishing every Notarius extraction lane as a named current output. The
lanes remain available as analyze inputs independently of publication; stable
remote destinations require an explicit operator need.
- Setting Narratio's optional `notarius.chunk_cache` override. The managed
Notarius configuration remains the single owner of its ordinary `auto`
cache policy, and Narratio v2.1 already requests refresh for forced
extraction.
- Adding notification providers. Narratio v2.1 supports only its existing
`noop` notification mode.
- Releasing new role or collection versions as part of this implementation.
## Target End State
The standalone pipeline role exposes these suite variables with the selected
defaults:
```yaml
dnd_pipeline_narratio_workspace_cleanup_after_publish: true
dnd_pipeline_narratio_storage_backend: s3
dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets
dnd_pipeline_narratio_storage_s3_root_prefix: dnd
dnd_pipeline_narratio_storage_s3_region: us-east-1
dnd_pipeline_narratio_storage_s3_endpoint: >-
https://s3.us-east-005.backblazeb2.com
dnd_pipeline_narratio_storage_s3_force_path_style: true
dnd_pipeline_narratio_storage_s3_access_key_id: OBJECT_STORAGE_KEY_ID
dnd_pipeline_narratio_storage_s3_secret_access_key: OBJECT_STORAGE_KEY
dnd_pipeline_narratio_cache_s3_audio: true
dnd_pipeline_narratio_publish_enabled: true
dnd_pipeline_narratio_publish_upload_run: true
dnd_pipeline_narratio_publish_outputs:
- source: narratio.transcript.final
required: true
- source: narratio.transcript.final_trimmed
required: true
- source: narratio.transcript.final_markdown
required: true
- source: narratio.transcript.final_trimmed_markdown
required: true
- source: narratio.artifact.session_recap
required: true
- source: narratio.artifact.hostile_rumors
required: true
- source: narratio.artifact.deranged_rumors
required: false
```
Each value is declared in the pipeline argument specification and passed
directly to its corresponding `narratio_*` variable in the pipeline role's
Narratio dependency. Narratio's existing root template renders the native
configuration. No new pipeline fragment or production task is required.
The ordinary generated configuration contains only the credential names:
```yaml
storage:
backend: s3
s3:
access_key_id: OBJECT_STORAGE_KEY_ID
secret_access_key: OBJECT_STORAGE_KEY
```
Consumers supply the two values beside the existing model-provider values
through encrypted inventory:
```yaml
dnd_pipeline_secrets:
AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"
OBJECT_STORAGE_KEY_ID: "{{ vault_dnd_object_storage_key_id }}"
OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}"
```
The example defines references only; no roadmap, role, or test contains real
credential values. The existing credential task writes the selected entries
with the suite's established group-authorized secret modes. Omitted map keys
remain unmanaged, as defined by Feature 0011.
With valid credentials, Narratio can accept either local or S3 audio selected
by each session file, cache S3 audio beneath `/var/cache/narratio`, publish an
immutable run and the selected stable outputs, use remote publication locks,
restore committed state, and clean the published workspace. These are native
Narratio commands and lifecycle rules rather than additional Ansible tasks.
The ten accepted Notarius outputs continue to enter Narratio's artifact
catalog as `narratio.extraction.*` sources. Analyze-stage Scriptorium artifacts
may reference those sources directly in their input mappings. Feature 0015
does not yet add such artifact definitions, but its publication choices do not
remove, rename, or constrain this future composition path.
## Repository Ownership
- `ansible-role-dnd-pipeline` owns the D&D suite variables, opinionated
defaults, dependency mappings, consumer documentation, and representative
integration scenario.
- `ansible-role-narratio` remains the generic owner of installation and native
pipeline rendering. Its reviewed v2.1.0 interface already covers every
configuration field required by this feature and should remain unchanged.
- `ansible-collection-dnd` owns the faithful pipeline subtree import, the
unified consumer dependency manifest, and artifact-boundary validation.
- `ansible-workspace-dnd` owns this roadmap, the later implementation plan,
status, and independent review record.
- `narratio` is a read-only source of storage, publication, credential,
caching, cleanup, restore, and CLI contracts.
- `rakestraw-ansible` remains unchanged until Feature 0016 consumes this
interface on `pipeline-dnd`.
## Constraints and Settled Decisions
- The S3 endpoint, bucket, prefix, region, and path-style values are deliberate
D&D suite defaults. They are ordinary variables and remain overrideable by
inventory.
- Credential references are also policy defaults, but credential values have
no defaults. Operators must supply matching entries through
`dnd_pipeline_secrets` or another environment source supported by Narratio.
- The two credential-reference variables are nullable as a pair. Setting both
to `null` omits the references and preserves Narratio's AWS SDK provider-chain
path; configuring only one remains invalid under Narratio's native contract.
- The role does not assert that the secrets map contains particular keys.
Narratio's native loader owns typed-reference resolution, and external
environment values remain a supported higher-precedence source.
- `dnd_pipeline_narratio_publish_outputs` is one complete native list. A caller
replacing it assumes responsibility for the entire selected publication
set; the role does not merge rules.
- Publish destinations remain omitted where Narratio can derive them. The
configured source identity and artifact output path remain the single source
of truth.
- Workspace cleanup is gated by Narratio's own successful-publication
evidence. Ansible does not independently inspect remote state or remove
runtime content.
- S3 audio caching is enabled, and ordinary cleanup preserves the cache.
Explicit operator cache clearing remains a Narratio CLI operation.
- The testing profile does not silently disable Notarius, storage,
publication, cleanup, or credentials. It continues to differ only in the
already selected inexpensive Scriptorium model profile.
- S3 storage also enables Narratio's native restore and publication-lock
commands without another configuration surface. Ansible documentation may
describe availability but must not duplicate Narratio's detailed operating
manual.
- Notarius extraction-source availability and remote publication are separate
concerns. Analyze consumes `narratio.extraction.*` sources from the accepted
extraction catalog; `publish.outputs` selects only which sources also receive
stable published destinations.
## Simplest Viable Implementation
Add the selected variables to the pipeline defaults and argument specification,
then map them into the existing `rakestraw.narratio` dependency. Update the
pipeline README to document the policy, credential names, complete-list
replacement semantics, and native capabilities now available. Do not add a
template, task, handler, fact, probe, or storage-specific validation layer.
In the existing pipeline integration driver, extend the synthetic secrets map
with object-storage values and explicitly supply representative S3 and publish
variables. Use `narratio config validate` and its secret-safe effective
configuration output to prove that the public inputs compose into a valid
configuration. Retain the immediate clean repeat and secret non-disclosure
checks. Do not call a real S3 service or assert that the representative values
equal current role defaults.
Push the reviewed pipeline commit, import that exact public revision into the
collection through `git subtree pull --squash`, and extend only the
collection-level fixture inputs needed for native validation. The collection
suite should continue to prove artifact installation and one representative
assembled convergence rather than duplicate the detailed pipeline matrix.
## Rationale and Alternatives
Implementing storage behavior in Ansible would duplicate a mature native
application boundary. Narratio already owns credential resolution, AWS SDK
provider construction, immutable commits, verification, locks, restore,
caching, and cleanup. Passing declarative configuration through the generic
role is both smaller and more reliable.
Using the AWS SDK's ambient provider chain would avoid named secret files, but
the initial deployment has explicit Backblaze application credentials and the
suite already has a suitable managed typed-secret path. Keeping the two
credential-reference variables nullable preserves the ambient-provider
alternative: consumers can set both suite variables to `null` when their
deployment uses another provider.
Copying destinations into every publish rule would make the output layout more
visible locally, but it would duplicate paths already owned by Narratio's
built-in sources and the configured artifact definitions. Native destination
derivation keeps path coordination automatic.
Deleting spool audio after publication would minimize transient disk usage,
but no such retention policy has been selected, and local-audio sessions may
depend on that content. Leaving the field omitted preserves Narratio's default
while enabling the explicitly requested S3 audio cache.
Publishing all ten Notarius extraction lanes would create stable remote output
contracts that have not been requested. It is also unnecessary for the stated
future use: analyze-stage artifacts can consume `narratio.extraction.*`
directly. The immutable published run retains the processing result, while the
selected transcript and human-facing artifact outputs match the known
operational workflow.
## Acceptance Criteria
- The pipeline role declares and documents every suite variable shown in the
target state, with matching argument-spec entries and direct Narratio
dependency mappings.
- The two credential-reference variables accept paired `null` overrides so a
caller can deliberately select Narratio's ambient AWS SDK provider chain.
- The default effective Narratio configuration selects the specified
Backblaze-compatible S3 destination, typed credential names, S3 audio cache,
successful-publication workspace cleanup, run upload, and seven output
rules.
- The publish list relies on Narratio's native destination derivation and
retains `deranged_rumors` as optional while the other six outputs are
required.
- Spool-audio deletion remains unset by the pipeline role, and the testing
profile introduces no storage or publication override.
- `dnd_pipeline_secrets` remains a generic empty-by-default mapping; the role
documents all four expected operational entries without embedding any
credential value.
- No new production task, template, handler, fact, state file, wrapper, or
storage-validation command is introduced.
- The standalone Narratio role remains unchanged because its v2.1.0 interface
already renders the required native fields and preserves null omission.
- The existing pipeline integration scenario explicitly supplies
representative S3 and publish inputs, resolves synthetic values for all four
typed credentials, passes Narratio's native configuration validation, and
exposes no synthetic value in captured output.
- The immediate identical second pipeline application reports `changed=0`
and `failed=0`; syntax checking, production-profile lint, the complete
standalone integration suite, and `git diff --check` pass.
- The exact reviewed pipeline commit is pushed to public Gitea before the
collection imports it through the documented remote-only squashed subtree
workflow.
- The collection fixture supplies the added synthetic credential names, and
its existing build, isolated installation, FQCN resolution, native
validation, and clean-repeat suite passes without a duplicate S3 behavior
test.
- No real object-storage request, paid model request, application repository
change, asset change, release tag, or published artifact is required.
- Pipeline documentation links operators to Narratio's canonical procedures
for S3 session audio, publish, restore, locks, status, and cleanup and no
longer describes storage and publishing as deferred.
- Documentation distinguishes accepted Notarius extraction sources from
published outputs and confirms that future analyze-stage artifacts may use
`narratio.extraction.<output_key>` inputs without publishing those lanes.
## Dependencies and Follow-up Work
Features 0009, 0011, 0012, 0013, and 0014 are prerequisites. In particular,
Feature 0011 supplies the credential directory and Feature 0014 supplies the
human-operator access needed for the eventual live deployment.
[Feature 0016](0016-deploy-dnd-pipeline-to-pipeline-dnd.md) follows this work.
It will put the four actual values in encrypted host inventory, pin the exact
reviewed suite revision in `rakestraw-ansible`, converge `pipeline-dnd`, and
exercise one real session through remote publication. Bucket lifecycle policy,
backup policy, credential rotation, additional publish outputs, automated
scheduling, and cleanup of the proof-of-concept deployment remain separate
future work.
A later artifact-development feature may add D&D-specific Narratio artifact
definitions whose Scriptorium input mappings consume selected
`narratio.extraction.*` sources. That feature should define each artifact's
purpose, inputs, prompt, output contract, and publication choice; it does not
require a storage-interface change.

278
docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md Normal file
View File

@@ -0,0 +1,278 @@
# Feature 0016: Deploy the D&D Pipeline to `pipeline-dnd`
Status: proposed
## Summary
Add a minimal playbook and host inventory to `rakestraw-ansible` that apply
`rakestraw.dnd.pipeline` to the existing `pipeline-dnd` host. Configure the
host through `host_vars`, enroll `eric` as the initial human operator, keep
credential values in Ansible Vault, and prove the complete pipeline through a
real repeatable deployment and representative D&D session.
[Feature 0015](0015-dnd-s3-storage-and-publishing.md) must first provide the
storage and publishing behavior needed for an operational end-to-end pipeline.
Feature 0016 consumes that reviewed interface; it does not invent a temporary
storage contract.
## Problem or Opportunity
The collection and its application roles are implemented and installable, but
the operational Ansible repository does not yet have a playbook for deploying
the new collection architecture. Its existing D&D playbook targets a retired
inventory name and invokes the proof-of-concept local `pipeline_dnd` role.
The current `pipeline-dnd` host inventory already records the connection
identity and useful legacy Narratio storage values. A production-shaped first
deployment now needs to express the new role's public interface, deliver
secrets safely, reconcile the controller's dependency manifest, and validate
the actual host rather than extending the role repositories with another
synthetic scenario.
## Scope
- Add `rakestraw-ansible/playbooks/pipeline/dnd.yml`, targeting the existing
`pipeline-dnd` inventory host and applying only
`rakestraw.dnd.pipeline`.
- Keep the playbook limited to its host selection and role list. Put all
deployment choices in `inventory/host_vars/pipeline-dnd/`.
- Configure the external WhisperX transcription endpoint as
`https://transcription.ai.rakestrawhome.com/transcribe`.
- Enroll the existing `eric` account through
`dnd_pipeline_operator_users`.
- Map the role's `dnd_pipeline_secrets` interface to host-specific Vault
variables. Store credential values only in an Ansible Vault-encrypted host
variables file.
- Carry the storage and publishing settings selected by Feature 0015 through
its final public pipeline interface. Preserve the operational intent of the
existing Backblaze B2 endpoint, bucket, prefix, region, and path-style
settings where they remain applicable.
- Reconcile existing legacy Narratio host variables so they cannot silently
override the suite policy selected by the new pipeline role.
- Update `rakestraw-ansible/requirements.yml` so a clean controller can install
the reviewed collection snapshot, all five standalone role dependencies,
and required native collections from one manifest.
- Validate controller dependency resolution, playbook syntax, initial
convergence, immediate clean convergence, operator access, native Narratio
configuration, and one representative end-to-end pipeline run on the real
host.
- Document the operational invocation, Vault prerequisite, initial operator,
and expected validation path in `rakestraw-ansible`.
## Non-goals
- Implementing storage or publishing behavior. Feature 0015 owns that
prerequisite.
- Changing application or role behavior merely to accommodate this
deployment.
- Replacing the existing inventory alias with `pipeline.dnd`, changing DNS,
or moving connection variables out of the current host inventory.
- Adding a wrapper role, pre-task layer, service, timer, scheduler, web UI, or
unattended session-ingestion mechanism.
- Creating or managing the `eric` account, its login policy, SSH access, or
sudo policy.
- Adding another production host, high availability, backup policy, or
disaster-recovery automation.
- Removing the proof-of-concept local `pipeline_dnd` role or its historical
playbook before the new deployment has demonstrated operational parity.
- Publishing a new coordinated suite release solely as part of this
deployment feature.
## Target End State
`rakestraw-ansible/playbooks/pipeline/dnd.yml` is an intentionally small
consumer playbook equivalent to:
```yaml
---
- hosts:
- pipeline-dnd
roles:
- rakestraw.dnd.pipeline
...
```
The existing inventory selects the host's connection identity. A dedicated
host-variable file selects the pipeline's non-secret public inputs, including
the WhisperX endpoint, the initial operator list, and the storage and
publishing values exposed by Feature 0015. It maps runtime credential names to
Vault-backed variables without embedding credential values in the playbook or
ordinary plaintext inventory, for example:
```yaml
dnd_pipeline_operator_users:
- eric
dnd_pipeline_whisperx_transcribe_url: >-
https://transcription.ai.rakestrawhome.com/transcribe
dnd_pipeline_secrets:
AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"
OBJECT_STORAGE_KEY_ID: "{{ vault_dnd_object_storage_key_id }}"
OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}"
```
The corresponding credential variables reside in a tracked file encrypted in
full with Ansible Vault. The Vault password or identity mechanism remains
outside Git and outside this feature's managed content.
The controller dependency manifest selects one reviewed, publicly reachable
suite snapshot containing Features 0014 and 0015. It also installs that
snapshot's exact standalone role and native collection dependencies. A clean
controller therefore does not depend on sibling development checkouts or
previously installed roles.
On `pipeline-dnd`, the role converges the full application suite and its D&D
configuration. The `narratio` service identity can run the pipeline, and
`eric` can run it after starting a login session that observes the granted
supplementary group membership. A representative session can be transcribed,
processed, stored, and published through the configured production path. An
immediate identical Ansible run reports no changes.
## Repository Ownership
- `rakestraw-ansible` owns the consumer playbook, host inventory, encrypted
host credentials, controller dependency manifest, and deployment runbook.
- `ansible-role-dnd-pipeline` owns the suite's public variables and the
opinionated cross-application behavior consumed by the deployment.
- `ansible-collection-dnd` owns the assembled `rakestraw.dnd.pipeline` role
snapshot and consumer-facing compatibility manifest.
- The standalone application roles continue to own installation and native
application configuration. Feature 0016 consumes their reviewed releases or
commits without reimplementing their behavior.
- `ansible-workspace-dnd` owns this roadmap, the later implementation plan,
progress tracking, and independent review record.
- Application and asset repositories are read-only sources of runtime
contracts and deployed content for this feature.
## Constraints and Settled Decisions
- Feature 0015 is a prerequisite and must settle, implement, and review the
storage and publishing interface before Feature 0016 becomes ready for an
implementation sprint.
- The target is the inventory alias `pipeline-dnd` as currently defined in
`rakestraw-ansible`, not a new `pipeline.dnd` alias.
- The playbook follows the repository's established style: hosts and roles
only. Facts remain at Ansible's normal default, privilege escalation remains
task-owned by the roles, and deployment variables belong in host inventory.
- `eric` is the initial human operator and must already exist on the managed
host. The pipeline role grants only additive runtime-group membership.
- API credentials use the existing typed `dnd_pipeline_secrets` interface and
Ansible Vault. Plaintext secret values must not enter Git, command lines,
logs, examples, or ordinary variable files.
- The public Gitea repositories are the sole dependency sources. The
deployment must not reference this workspace's sibling checkout paths.
- The dependency manifest must select an immutable release when Feature 0015
produces one. Otherwise, it must pin the exact reviewed public Git commit;
it must not track an unbounded branch for the production deployment.
- The collection artifact does not itself install arbitrary Git-hosted role
dependencies. The operational manifest must therefore include the five
standalone roles as well as the collection and its native collection
dependencies.
- The legacy host variables must be compared with the final Feature 0015
interface. Applicable values move to the new suite configuration, while
obsolete or conflicting variables are removed rather than left to win by
inventory precedence.
- The initial real-host run is an operational acceptance exercise. It should
test observable pipeline behavior and convergence, not module task traces or
the current text of defaults.
## Simplest Viable Implementation
After Feature 0015 is reviewed, select its exact public collection revision
and use the associated compatibility manifest to update the controller's
single root requirements file. Add the minimal playbook and one ordinary
host-variable file for public inputs. Add one fully encrypted host Vault file
for the two credential values, populated by the operator without exposing its
plaintext during implementation or review.
Install or update dependencies from the root manifest, confirm that Ansible
resolves `rakestraw.dnd.pipeline`, syntax-check the playbook, and converge only
`pipeline-dnd`. Start a fresh operator login after enrollment, validate the
native Narratio configuration under the intended runtime identity, rerun the
same play immediately to confirm clean convergence, and process one selected
session through the complete Feature 0015 publishing path. Record concise
operational instructions and results; do not add a second test harness around
the real deployment.
## Rationale and Alternatives
Keeping the playbook nearly declarative makes inventory the durable record of
what differs about this host and allows ordinary Ansible precedence to remain
the customization mechanism. Embedding endpoint, operator, storage, or secret
mapping variables in the playbook would make a one-host play harder to reuse
and would diverge from the established `rakestraw-ansible` style.
A local wrapper role could hide the collection interface, but it would create
another layer with no independent behavior. Applying the collection role by
FQCN proves the same consumer path that future hosts should use.
Tracking collection `main` would be convenient during development, but it
would make a later controller bootstrap select an unreviewed compatibility
set. An exact public commit retains development flexibility until the next
release while still making this deployment reproducible.
Running the real-host deployment before Feature 0015 would establish only an
installation demonstration: the pipeline still could not complete its
required storage and publishing path. Making Feature 0015 an explicit
prerequisite avoids temporary inventory variables and gives the acceptance
run a meaningful end-to-end boundary.
## Acceptance Criteria
- Feature 0015 is complete and independently reviewed before Feature 0016
implementation begins.
- `playbooks/pipeline/dnd.yml` targets only `pipeline-dnd`, applies only
`rakestraw.dnd.pipeline`, and contains no deployment variables or procedural
setup.
- Host inventory sets the confirmed WhisperX endpoint, enrolls only `eric`,
and supplies the final Feature 0015 storage and publishing inputs.
- `dnd_pipeline_secrets` maps the required typed runtime keys to host Vault
variables, and the actual values exist only in a valid Ansible
Vault-encrypted file.
- No plaintext credential value is present in any tracked revision, diff,
command example, validation output, or deployment documentation.
- Legacy Narratio storage values are either represented through the current
pipeline interface or deliberately retired; no stale high-precedence host
variable can override the selected suite configuration accidentally.
- The root requirements manifest preserves unrelated existing dependencies
and installs the exact reviewed public collection snapshot, its five
standalone role dependencies, and its required native collections without
using local repository paths.
- On a controller bootstrapped from that manifest, `ansible-doc` resolves
`rakestraw.dnd.pipeline` and each role dependency, and the new playbook
passes syntax checking.
- The initial limited playbook run against `pipeline-dnd` completes
successfully using the encrypted inventory and ordinary sudo-capable remote
account.
- The immediate identical second run reports `changed=0` and `failed=0`.
- Narratio's native configuration validation succeeds under the intended
runtime identity, and a fresh session for `eric` can access and invoke the
shared pipeline through runtime-group membership.
- One operator-selected D&D session completes transcription, processing,
storage, and publishing, and its expected published output is observable at
the configured destination.
- The operational documentation explains dependency installation, Vault use,
deployment invocation, the login-session membership caveat, convergence
verification, and representative pipeline invocation without disclosing
credentials.
- The historical proof-of-concept role and playbook remain available during
the initial rollout; their later retirement requires a separate explicit
decision.
## Dependencies and Follow-up Work
[Feature 0015](0015-dnd-s3-storage-and-publishing.md) is the immediate next unit
of work and a hard prerequisite. It selects the pipeline interface,
credentials, and observable publishing boundary that Feature 0016 will deploy.
Features 0013 and 0014 provide the public collection installation path and
existing-operator enrollment required here. Features 0009 and 0011 provide
the shared filesystem and managed credential-delivery policies.
After the real deployment demonstrates parity, a later cleanup feature may
retire the proof-of-concept `pipeline_dnd` role, its old playbook, and any
obsolete inventory values. Service scheduling, unattended ingestion,
additional human operators, backup policy, and additional hosts also remain
separate work.

511
docs/roadmap/implementation.md
View File

@@ -1,26 +1,29 @@
# Implementation Plan 0014: Enroll Existing Pipeline Operators # Implementation Plan 0015: Enable D&D S3 Storage and Publishing
Status: ready Status: ready
Target executor: GPT-5.6-Terra Target executor: GPT-5.6-Terra
Feature roadmap: Feature roadmap:
`docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md` `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md`
Active plan location: `docs/roadmap/implementation.md` Active plan location: `docs/roadmap/implementation.md`
## Objective ## Objective
Add one empty-by-default pipeline interface that enrolls explicitly selected Configure the authoritative D&D pipeline role to use the selected Backblaze
existing human accounts in the suite's runtime group. Implement the behavior S3-compatible archive, typed object-storage credential references, maximum S3
with one account-existence lookup and one additive membership task, adapt the audio caching, immutable run publication, selected transcript and D&D artifact
existing operator integration scenario, and import the reviewed result into outputs, and cleanup of successfully published workspace state. Validate the
the collection from public Gitea. configuration through Narratio's native interface, import the exact reviewed
pipeline commit into the collection from public Gitea, and leave the result
ready for the real-host deployment in Feature 0016.
The Feature 0014 roadmap remains authoritative for scope, rationale, target The Feature 0015 roadmap remains authoritative for scope, rationale, target
state, and acceptance criteria. This plan does not authorize human-account state, and acceptance criteria. This plan does not authorize a second storage
creation or removal, exact-list reconciliation, filesystem-policy changes, or implementation, real object-storage calls from repository tests, per-session
a new release. configuration management, stable publication of every Notarius extraction
lane, or changes to the generic Narratio role.
## Execution Rules ## Execution Rules
@@ -33,37 +36,49 @@ a new release.
worktree changes or branch divergence rather than overwriting them. worktree changes or branch divergence rather than overwriting them.
- Do not make unresolved product or architecture decisions during execution; - Do not make unresolved product or architecture decisions during execution;
stop and report the decision required. stop and report the decision required.
- Keep the production behavior to the roadmap's exact direct task flow: one - Implement storage and publication entirely through the standalone Narratio
`ansible.builtin.getent` loop followed by one `ansible.builtin.user` loop. role's existing field-specific variables. Do not add a production task,
- Do not add an assertion, registered fact, handler, command, shell operation, template, handler, fact, probe, command, wrapper, state file, or validation
explicit empty-list condition, membership comparison, post-change query, layer to the pipeline role.
sidecar state, or account abstraction. - Keep `dnd_pipeline_secrets` as the existing generic empty-by-default map.
- Trust `getent` to fail for a missing key and `user` to converge additive Add no real, generated, placeholder, or fallback credential value to role
membership. Do not reproduce either module's checks. defaults, templates, documentation, tests, logs, or ordinary inventory.
- Test the public behavior by adapting the existing temporary-operator - Keep the S3 access-key and secret-key reference variables nullable as a
scenario. Do not add a missing-user test, inspect `/etc/passwd` or pair. Do not add an Ansible assertion for pairing or secret presence;
`/etc/group`, assert task results, or duplicate the detailed test in the Narratio owns native configuration validation and AWS provider selection.
collection. - Do not configure `narratio_spool_delete_audio_after_publish` or
- Run each stage's validation before committing it. Do not proceed after a `narratio_notarius_chunk_cache`. Preserve the roadmap's single-owner
validation failure. decisions for spool retention and Notarius cache policy.
- Treat accepted Notarius outputs and published outputs as separate
interfaces. Preserve `narratio.extraction.<output_key>` sources for future
analyze inputs without adding publication rules or new artifacts for them.
- Exercise explicitly supplied representative values in the detailed pipeline
integration scenario. Do not assert that those values are current defaults
or duplicate Narratio's own storage and publication test matrix.
- Do not make a real S3 request or paid model request during repository-local
validation.
- Run every stage's validation before completing the stage. Do not proceed
after a validation failure.
- Commit each completed stage with the stated plain-English intent. Push the - Commit each completed stage with the stated plain-English intent. Push the
authoritative pipeline commit before importing it into the collection. authoritative pipeline commit before importing it into the collection.
- Use only the public Gitea pipeline URL for the subtree update; do not import - Use only the public Gitea pipeline URL for the subtree update. Do not import
from the local sibling repository. from the local sibling repository.
## Starting State and Preconditions ## Starting State and Preconditions
- Features 0009, 0011, 0012, and 0013 are complete. - Features 0009, 0011, 0012, 0013, and 0014 are complete.
- This roadmap and implementation plan have been reviewed and committed in the - The approved Feature 0015 roadmap and this implementation plan are committed
workspace repository before execution begins. in the workspace repository before the sprint begins.
- The completed Feature 0013 closeout commits `eda361f` in the workspace and - `ansible-role-dnd-pipeline` starts clean on `main` at
`5e695f3` in the collection are pushed to their respective `origin/main` `c13f726a0a846180a1aa95d110b8bbc801edaae5`, equal to `origin/main`.
branches before implementation begins. Preserve the Git-generated subtree - `ansible-collection-dnd` starts clean on `main` at
squash commit that accompanies `5e695f3`. `c0b0ebd13e92199f3d94e138c6cc50d1ece5dadd`, equal to `origin/main`.
- `ansible-role-dnd-pipeline` starts clean on `main` at `78afffe`, equal to - `ansible-role-narratio` is clean at
`origin/main`. `292d8669e7aad5cfe80b0c469d22b6177e8cf0f1`, equal to `origin/main`, and its
- `ansible-collection-dnd` starts clean on `main` at `5e695f3`, equal to `v0.1.0` role release contains the Narratio v2.1.0 field-specific interface
`origin/main` after the prerequisite push. required by this feature.
- The sibling Narratio application repository is clean at public release
`v2.1.0`. It is a read-only contract source during this sprint.
- The workspace `.venv` supplies the tested `ansible-playbook`, - The workspace `.venv` supplies the tested `ansible-playbook`,
`ansible-galaxy`, and `ansible-lint` commands used by the repository test `ansible-galaxy`, and `ansible-lint` commands used by the repository test
drivers. drivers.
@@ -71,27 +86,31 @@ a new release.
test-time sudo is available for the existing integration fixtures. test-time sudo is available for the existing integration fixtures.
- Git credentials can push the pipeline and collection commits to their public - Git credentials can push the pipeline and collection commits to their public
Gitea origins. Gitea origins.
- No actual Backblaze or model-provider credential is required or available to
the implementation. Tests use conspicuously synthetic values only.
## Repository Scope ## Repository Scope
- `ansible-role-dnd-pipeline` owns the variable, argument specification, - `ansible-role-dnd-pipeline` owns the suite defaults, argument specification,
account lookup, additive enrollment task, README, and detailed integration Narratio dependency mappings, consumer documentation, and detailed
behavior. representative integration behavior.
- `ansible-collection-dnd` receives the reviewed pipeline commit only through - `ansible-collection-dnd` owns the artifact-level fixture update, faithful
its established squashed subtree and validates the assembled artifact. squashed subtree import, assembled-artifact validation, and resulting public
- `ansible-workspace-dnd` records implementation progress and the review collection snapshot.
handoff after the two implementation repositories are complete. - `ansible-workspace-dnd` records the implementation handoff after the two
- All standalone application-role repositories, application source implementation repositories are complete.
repositories, asset repositories, and `rakestraw-ansible` are out of scope - `ansible-role-narratio`, all other standalone roles, application sources,
and must remain unchanged. asset repositories, and `rakestraw-ansible` may be inspected but must remain
unchanged.
## Stage 1: Implement Existing-Operator Enrollment ## Stage 1: Implement D&D Storage and Publication Policy
### Objective ### Objective
Give the standalone pipeline role its minimal public interface for enrolling Expose the approved suite-level S3 and publishing inputs through the
existing human accounts and prove the behavior through the existing operator authoritative pipeline role, map them directly into Narratio, document the
integration scenario. operator contract, and prove representative composition without networked
storage access.
### Required Policy Review ### Required Policy Review
@@ -103,17 +122,22 @@ Before beginning this stage, read:
- `docs/ansible.md` - `docs/ansible.md`
- `docs/testing.md` - `docs/testing.md`
- `docs/planning.md` - `docs/planning.md`
- `docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md` - `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md`
- `ansible-role-dnd-pipeline/README.md`
- `../narratio/AGENTS.md`
- `../narratio/docs/development.md`
- `../narratio/docs/config.md`
- `../narratio/docs/operations.md`
### Preconditions ### Preconditions
- The workspace planning commit is present. - The workspace planning commit containing this plan is present.
- `ansible-role-dnd-pipeline` is clean at `78afffe` on `main`, is not behind - `ansible-role-dnd-pipeline` is clean at the starting revision, is not behind
or diverged from `origin/main`, and contains no unrelated changes. or diverged from `origin/main`, and contains no unrelated changes.
- Its Narratio dependency still creates the selected runtime group before the - The current `rakestraw.narratio` dependency mapping and existing integration
pipeline role's ordinary tasks execute. driver still match the structures reviewed while preparing this plan.
- The existing integration driver still contains the temporary operator and - Narratio v2.1.0 still accepts the roadmap-selected storage, publish, cleanup,
representative group-access scenario described by the roadmap. cache, and typed credential-reference fields.
### Owning Repository ### Owning Repository
@@ -121,38 +145,102 @@ Before beginning this stage, read:
### Required Changes ### Required Changes
1. Add `dnd_pipeline_operator_users: []` beside the existing runtime identity 1. Add the following exact suite defaults to `defaults/main.yml` near the
defaults in `defaults/main.yml`. existing Narratio policy variables:
2. Add the matching `meta/argument_specs.yml` entry with `type: list`,
`elements: str`, and the same required/default-backed convention used by ```yaml
the role's other public variables. dnd_pipeline_narratio_workspace_cleanup_after_publish: true
3. Add one unprivileged `ansible.builtin.getent` task near the beginning of dnd_pipeline_narratio_storage_backend: s3
`tasks/main.yml`. Loop directly over `dnd_pipeline_operator_users`, use the dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets
`passwd` database and each username as `key`, and rely on the module's dnd_pipeline_narratio_storage_s3_root_prefix: dnd
default missing-key failure. Do not register its result or add a separate dnd_pipeline_narratio_storage_s3_region: us-east-1
assertion. dnd_pipeline_narratio_storage_s3_endpoint: https://s3.us-east-005.backblazeb2.com
4. Immediately follow it with one `become: true` `ansible.builtin.user` task dnd_pipeline_narratio_storage_s3_force_path_style: true
over the same list. Set `name` to the selected account, dnd_pipeline_narratio_storage_s3_access_key_id: OBJECT_STORAGE_KEY_ID
`groups` to `dnd_pipeline_runtime_group`, `append: true`, and dnd_pipeline_narratio_storage_s3_secret_access_key: OBJECT_STORAGE_KEY
`state: present`. Do not set or manage any other account property. dnd_pipeline_narratio_cache_s3_audio: true
5. Revise the README variable reference and shared-execution section. Replace dnd_pipeline_narratio_publish_enabled: true
the consumer-owned membership example and deferred wording with the new dnd_pipeline_narratio_publish_upload_run: true
interface, existing-account prerequisite, additive non-removal behavior, dnd_pipeline_narratio_publish_outputs:
authorization consequence, and new-login-session caveat. - source: narratio.transcript.final
6. Adapt `tests/integration` rather than adding another scenario. Create its required: true
temporary operator before the initial role application without membership - source: narratio.transcript.final_trimmed
in the runtime group, pass that username through required: true
`dnd_pipeline_operator_users` for the baseline and repeated applications, - source: narratio.transcript.final_markdown
and retain the existing native configuration, secret-boundary, and shared required: true
cache exercises under that identity. Reorder cleanup as necessary so any - source: narratio.transcript.final_trimmed_markdown
final baseline convergence occurs while the selected fixture account still required: true
exists, then remove the fixture account. - source: narratio.artifact.session_recap
7. Retain the existing clean-repeat assertion. Do not add a negative required: true
missing-user case or assertions about module internals, task traces, the - source: narratio.artifact.hostile_rumors
current default value, or raw account database entries. required: true
8. Run the complete existing repository validation, commit the result, push - source: narratio.artifact.deranged_rumors
`main` to the public pipeline origin, and record the full pushed commit for required: false
Stage 2. ```
Keep every `dest` omitted. Narratio derives transcript destinations from
built-in sources and artifact destinations from the already managed
artifact output paths.
2. Add matching public entries to `meta/argument_specs.yml`. Use `type: bool`
for the cleanup, path-style, cache, enabled, and upload settings; `type: str`
for the backend and S3 string fields; and `type: list` for the complete
publish output list. The S3 child fields, including both credential
references, must be non-required so callers may set every S3 child to
`null` when deliberately selecting the local backend or set the credential
pair to `null` when selecting the AWS SDK provider chain. The remaining
concrete suite-policy fields follow the role's existing required/defaulted
argument-spec convention.
3. In the `rakestraw.narratio` dependency entry in `meta/main.yml`, map every
new suite variable directly to its same-purpose standalone variable:
`narratio_workspace_cleanup_after_publish`, the eight
`narratio_storage_*` fields, `narratio_cache_s3_audio`,
`narratio_publish_enabled`, `narratio_publish_upload_run`, and
`narratio_publish_outputs`.
4. Do not map `narratio_spool_delete_audio_after_publish`. Do not map
`narratio_notarius_chunk_cache`. Do not add a new import, fragment,
template, task, handler, assertion, or configuration-validation command.
5. Revise the README's variable reference and processing-policy sections to
document the selected S3 destination, paired nullable credential
references, complete-list publish semantics, native destination derivation,
maximum S3 audio caching, successful-publication workspace cleanup, and
deliberate omission of spool deletion.
6. Extend the README's managed-credential example to show
`OBJECT_STORAGE_KEY_ID` and `OBJECT_STORAGE_KEY` mapped from encrypted
inventory beside the existing Audita and OpenRouter entries. Explain that
values remain outside ordinary configuration and that setting both S3
credential references to `null` selects Narratio's ambient AWS SDK provider
chain.
7. Replace the README's statement that storage and publishing are deferred
with concise operator-facing descriptions and public upstream links for S3
session audio, publish, restore, publication locks, status, and cleanup.
Explain that accepted Notarius lanes remain
`narratio.extraction.<output_key>` analyze inputs independently of whether
they have stable publish rules.
8. Adapt the existing `tests/integration` baseline rather than adding a new
scenario. Add two synthetic object-storage values to
`dnd_pipeline_secrets`; explicitly unset `OBJECT_STORAGE_KEY_ID` and
`OBJECT_STORAGE_KEY` alongside the existing provider variables when
invoking Ansible; and explicitly pass representative non-default S3,
cleanup, cache, and publish values through the new public variables.
9. Use representative values that exercise the interface without freezing the
defaults: for example an `.invalid` HTTPS endpoint and fixture bucket,
prefix, and region; intentional `false` values for cleanup, path style, and
S3 audio caching; the two configured credential reference names; enabled
run publication; and a short complete publish list containing one built-in
transcript source and `narratio.artifact.session_recap` with omitted
destinations.
10. Extend the existing secret-safe effective-configuration inspection only
enough to demonstrate the explicitly supplied representative storage
fields, typed credential names, falsey cache/cleanup choices, enabled
publication, and Narratio-derived destinations. Continue to use
`narratio config validate` as the native validity boundary. Do not inspect
role task results, call S3, or reproduce Narratio's validation rules.
11. Extend the current non-disclosure check to cover both synthetic
object-storage values. Preserve the complete existing integration matrix,
cleanup behavior, and immediate clean-repeat assertions.
12. Run the complete repository validation. Commit the valid result with the
stated plain-English message, push `main` to public Gitea, and record the
full pushed commit hash for Stage 3.
### Validation ### Validation
@@ -162,34 +250,36 @@ Run from the workspace root:
(cd ansible-role-dnd-pipeline && tests/integration && git diff --check) (cd ansible-role-dnd-pipeline && tests/integration && git diff --check)
``` ```
Expected result: syntax and production-profile lint pass; the temporary Expected result: syntax checking and production-profile lint pass; all five
existing account is enrolled through the public role variable and completes applications remain valid; the explicitly supplied S3 and publish values
the existing representative operator-access scenario; native application resolve through Narratio's native configuration interface; only reference
checks pass; the immediate identical role application reports `changed=0` names appear in effective output; no synthetic credential value appears in
and `failed=0`; and no credential value appears in test output. captured output; no external S3 or model request occurs; and the immediate
identical repeat reports `changed=0` and `failed=0`.
### Completion Criteria ### Completion Criteria
- The public interface, argument specification, two direct tasks, and README - Defaults, argument specifications, dependency mappings, and documentation
implement exactly the roadmap contract. implement exactly the Feature 0015 contract.
- The existing operator scenario uses the role interface rather than fixture - No production task, template, fragment, validation command, or second secret
group enrollment. path was added.
- The representative integration scenario exercises public inputs rather than
snapshotting defaults and retains the existing clean-repeat boundary.
- The full standalone integration suite and `git diff --check` pass. - The full standalone integration suite and `git diff --check` pass.
- The role repository is clean after a plain-English commit. - The pipeline repository is clean after the plain-English commit.
- The exact commit is present on public `origin/main` and its full hash is - The exact commit is present on public `origin/main` and its full hash is
available for the collection import. available for the collection import.
### Commit ### Commit
Commit intent: `Add optional pipeline operator enrollment` Commit intent: `Enable D&D storage and publishing`
## Stage 2: Import the Reviewed Pipeline Change ## Stage 2: Prepare the Collection Credential Fixture
### Objective ### Objective
Expose the reviewed operator interface through `rakestraw.dnd.pipeline` while Make the collection's artifact-level fixture compatible with the forthcoming
preserving the authoritative-standalone-repository and collection-artifact pipeline snapshot while its currently bundled role remains valid.
boundaries.
### Required Policy Review ### Required Policy Review
@@ -201,17 +291,93 @@ Before beginning this stage, read:
- `docs/ansible.md` - `docs/ansible.md`
- `docs/testing.md` - `docs/testing.md`
- `docs/planning.md` - `docs/planning.md`
- `docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md` - `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md`
- `ansible-collection-dnd/README.md` - `ansible-collection-dnd/README.md`
### Preconditions ### Preconditions
- Stage 1 is complete and its exact commit is publicly reachable from - Stage 1 is complete, validated, committed, and pushed.
`https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git`. - `ansible-collection-dnd` is clean at the starting revision, is not behind or
- `ansible-collection-dnd` is clean at `5e695f3` on `main`, is not behind or
diverged from `origin/main`, and contains no unrelated changes. diverged from `origin/main`, and contains no unrelated changes.
- The collection's current `roles/pipeline` subtree still corresponds to the - Its top-level `tests/integration` fixture still supplies the two existing
authoritative pipeline history through `78afffe`. synthetic provider credentials through `dnd_pipeline_secrets`.
### Owning Repository
`ansible-collection-dnd`
### Required Changes
1. Update only the collection-level `tests/integration` fixture outside
`roles/pipeline`. Add conspicuously synthetic values for
`OBJECT_STORAGE_KEY_ID` and `OBJECT_STORAGE_KEY` to its existing
`dnd_pipeline_secrets` input.
2. Explicitly unset those two environment variables beside the existing
Audita and OpenRouter variables when the fixture runs Ansible so ambient
controller state cannot replace the managed synthetic files.
3. Do not add storage assertions or duplicate the standalone pipeline's
detailed representative test. The collection fixture owns only enough
input to validate the assembled artifact after Stage 3.
4. Do not edit anything under `roles/pipeline`, alter dependency manifests,
or change collection release metadata.
5. Check the fixture's shell syntax and whitespace, then commit the compatible
preparation with the stated message. Do not push separately unless needed;
Stage 3 will push both collection commits after full artifact validation.
### Validation
Run from the workspace root:
```text
(cd ansible-collection-dnd && bash -n tests/integration && git diff --check)
```
Expected result: the collection integration driver remains valid shell; the
diff changes only its generic managed-secret input and environment scrub; and
no subtree, dependency, or release file changed. Full semantic validation
occurs once against the imported snapshot in Stage 3.
### Completion Criteria
- The artifact-level fixture provides all four synthetic operational
credential names without exposing real values.
- No detailed storage behavior was duplicated in the collection.
- Only the collection-owned fixture changed.
- Shell syntax and `git diff --check` pass.
- The collection repository is clean after the plain-English commit.
### Commit
Commit intent: `Prepare collection storage credentials`
## Stage 3: Import and Validate the Reviewed Pipeline Snapshot
### Objective
Expose the reviewed Feature 0015 behavior through `rakestraw.dnd.pipeline`
while preserving the public-remote subtree and collection-artifact boundaries.
### Required Policy Review
Before beginning this stage, read:
- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/ansible.md`
- `docs/testing.md`
- `docs/planning.md`
- `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md`
- `ansible-collection-dnd/README.md`
### Preconditions
- Stage 1's exact pipeline commit is publicly reachable from
`https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git`.
- Stage 2 is complete and committed in `ansible-collection-dnd`.
- The collection is clean at the Stage 2 commit and its bundled pipeline
subtree still corresponds to the authoritative pipeline history through
`c13f726a0a846180a1aa95d110b8bbc801edaae5`.
### Owning Repository ### Owning Repository
@@ -220,27 +386,28 @@ Before beginning this stage, read:
### Required Changes ### Required Changes
1. Set a shell variable to the full pushed Stage 1 commit hash and run the 1. Set a shell variable to the full pushed Stage 1 commit hash and run the
documented remote-only update from the collection repository: documented remote-only subtree update from the collection repository:
```text ```text
pipeline_revision=<FULL_PUSHED_STAGE_1_COMMIT> pipeline_revision=<FULL_PUSHED_STAGE_1_COMMIT>
git subtree pull --prefix=roles/pipeline \ git subtree pull --prefix=roles/pipeline \
https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \ https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \
"$pipeline_revision" --squash -m "Import pipeline operator enrollment" "$pipeline_revision" --squash -m "Import D&D storage and publishing"
``` ```
2. Do not edit role behavior directly beneath `roles/pipeline`. The subtree 2. Do not edit role behavior directly beneath `roles/pipeline`. The import
import should carry the new default, argument specification, tasks, README, must faithfully carry the reviewed defaults, argument specification,
and standalone tests faithfully. dependency mappings, README, and detailed integration fixture.
3. Do not change `galaxy.yml`, dependency manifests, collection-level tests, 3. Do not modify `galaxy.yml`, `requirements.yml`, release documentation, or
or release documentation. No dependency or published version changes in the published `v0.1.0` artifacts. Feature 0015 changes no standalone role or
this feature, and the detailed behavior remains covered by Stage 1. native collection dependency and publishes no release.
4. Run the existing collection integration suite against the built and 4. Run the complete collection integration suite against the built and
installed artifact. Commit no additional files after the subtree command installed artifact. Do not add collection-level assertions for the detailed
unless validation exposes a genuine collection-boundary issue within this S3 configuration; native validation and clean convergence remain the
feature's scope. collection boundary.
5. Push the resulting collection history to `origin/main` and record the 5. If validation succeeds, push the Stage 2 fixture commit, the subtree merge
resulting merge commit and Git-generated squashed subtree commit. commit, and its Git-generated squash commit to `origin/main`. Record the
resulting commit hashes for Stage 4.
### Validation ### Validation
@@ -250,33 +417,34 @@ Run from the workspace root:
(cd ansible-collection-dnd && tests/integration && git diff --check) (cd ansible-collection-dnd && tests/integration && git diff --check)
``` ```
Expected result: the collection builds; the archive contains the updated Expected result: the collection builds; its archive contains the updated
pipeline role while retaining its development-file exclusions; isolated pipeline role while retaining development-file exclusions; isolated
dependency and artifact installation succeeds; the FQCN resolves; native dependency and artifact installation succeeds; the FQCN resolves; Narratio
representative application checks pass; and the immediate repeat reports loads its typed storage references from the synthetic credential files without
`changed=0` and `failed=0`. contacting S3; native representative application checks pass; and the
immediate repeat reports `changed=0` and `failed=0`.
### Completion Criteria ### Completion Criteria
- Git history records a squashed subtree import from the exact public Stage 1 - Git history records the exact public Stage 1 revision through the documented
commit with the stated plain-English merge message. squashed subtree workflow and stated merge message.
- No pipeline behavior was authored directly in the collection. - No pipeline behavior was authored directly in the collection.
- The existing collection suite and `git diff --check` pass without an - The complete collection suite and `git diff --check` pass.
unnecessary duplicate operator test. - The collection repository is clean and its Stage 2 and Stage 3 history is
- The collection is clean and its new history is present on `origin/main`. present on public `origin/main`.
### Commit ### Commit
Commit intent: `Import pipeline operator enrollment` Commit intent: `Import D&D storage and publishing`
The subtree command creates the stage's merge commit and its generated squash The subtree command creates this stage's merge commit and its generated squash
commit; do not add an empty or redundant manual commit. commit; do not add an empty or redundant manual commit.
## Stage 3: Record the Implementation Handoff ## Stage 4: Record the Implementation Handoff
### Objective ### Objective
Record the completed implementation evidence without marking Feature 0014 Record the completed implementation evidence without marking Feature 0015
done before independent review. done before independent review.
### Required Policy Review ### Required Policy Review
@@ -287,14 +455,15 @@ Before beginning this stage, read:
- `docs/orientation.md` - `docs/orientation.md`
- `docs/conventions.md` - `docs/conventions.md`
- `docs/planning.md` - `docs/planning.md`
- `docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md` - `docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md`
### Preconditions ### Preconditions
- Stages 1 and 2 are complete, committed, pushed, and validated. - Stages 1 through 3 are complete, committed, pushed where required, and
validated.
- Their full commit hashes and validation results are available. - Their full commit hashes and validation results are available.
- The workspace repository is clean apart from the expected progress-record - The workspace repository is clean at the planning commit, is not behind or
edits for this stage and is not behind or diverged from `origin/main`. diverged from `origin/main`, and contains no unrelated changes.
### Owning Repository ### Owning Repository
@@ -302,15 +471,17 @@ Before beginning this stage, read:
### Required Changes ### Required Changes
1. Change Feature 0014's roadmap status and status-table state to 1. Change Feature 0015's roadmap status and status-table state to
`in_progress`, indicating implementation is complete but independent review `in_progress`, indicating implementation is complete but independent review
remains outstanding. remains outstanding.
2. Update the workspace current focus and Feature 0014 evidence with the exact 2. Update the workspace current focus and Feature 0015 evidence with the exact
pipeline and collection commit hashes and the validation completed. pipeline commit, collection fixture commit, collection subtree merge and
3. Replace this plan's Progress Notes with a concise record of the three stage squash commits, and validation completed.
3. Replace this plan's Progress Notes with a concise record of the four stage
states and their commits. Do not copy command logs into the repository. states and their commits. Do not copy command logs into the repository.
4. Do not mark the feature `done`, remove the active plan, or create 4. Do not mark the feature `done`, remove this active plan, change Feature
`docs/roadmap/remediation.md`; those are review and closeout decisions. 0016's state, or create `docs/roadmap/remediation.md`; those are later review
and closeout decisions.
5. Commit the workspace handoff record with the stated plain-English message. 5. Commit the workspace handoff record with the stated plain-English message.
### Validation ### Validation
@@ -324,49 +495,39 @@ git diff --check
Expected result: every repository is clean; the pipeline and collection are Expected result: every repository is clean; the pipeline and collection are
not ahead of or behind their origins; only the workspace handoff commit may be not ahead of or behind their origins; only the workspace handoff commit may be
ahead of `origin/main`; and the dashboard points to the exact implementation ahead of `origin/main`; and the dashboard points to exact implementation
evidence while retaining independent review as the next step. evidence while retaining independent review as the next step.
### Completion Criteria ### Completion Criteria
- Workspace status accurately identifies Feature 0014 as implemented and - Workspace status accurately identifies Feature 0015 as implemented and
awaiting review. awaiting independent review.
- The roadmap remains the durable target-state record and is not polluted with - The roadmap remains the durable target-state record and contains no command
execution logs. log or transient execution notes.
- The active plan records concise implementation evidence and contains no - The active plan records concise implementation evidence and contains no
false completion claim. false completion claim.
- The workspace repository is clean after the handoff commit. - The workspace repository is clean after the handoff commit.
### Commit ### Commit
Commit intent: `Record pipeline operator enrollment implementation` Commit intent: `Record D&D storage and publishing implementation`
## Final Validation and Handoff ## Final Validation and Handoff
- Confirm Stage 1's complete standalone integration result and Stage 2's - Confirm Stage 1's complete standalone pipeline integration result and Stage
complete collection integration result. 3's complete collection integration result.
- Run `./scripts/status` and confirm every repository is clean, with no - Run `./scripts/status` and confirm every repository is clean, with no
unexpected branch divergence or untracked artifact. unexpected branch divergence or untracked artifact.
- Report the full pipeline, collection merge, collection subtree-squash, and - Report the full pipeline commit, collection fixture commit, collection
workspace handoff commit hashes. subtree merge and squash commits, and workspace handoff commit.
- Summarize any deviation from this plan, unresolved finding, or added - Confirm that `ansible-role-narratio`, all other standalone roles,
implementation complexity and identify the acceptance criterion that applications, assets, and `rakestraw-ansible` remained unchanged.
required it. - Summarize any deviation from this plan, unresolved finding, or complexity
- Do not mark Feature 0014 `done`; implementation results require independent added beyond direct variable mapping and identify the acceptance criterion
that required it.
- Do not mark Feature 0015 `done`; implementation results require independent
review first. review first.
## Progress Notes ## Progress Notes
- Stage 1 complete: `ansible-role-dnd-pipeline` - Planning complete; implementation has not begun.
`c13f726a0a846180a1aa95d110b8bbc801edaae5` implements optional enrollment
of existing operators; the complete standalone integration suite passed and
the commit is public on `origin/main`.
- Stage 2 complete: `ansible-collection-dnd` merge
`c0b0ebd13e92199f3d94e138c6cc50d1ece5dadd` imports the exact public pipeline
change through squash `d292297b6ad179e4664c01ad07cddaefff12f769`;
the complete collection integration suite passed and the history is public
on `origin/main`.
- Stage 3 complete: `ansible-workspace-dnd`
`bf81461de60001a0d8050e560383c6ddf533209f` records the implementation
evidence while Feature 0014 remains `in_progress` for independent review.
- Deviations: none.

11
docs/roadmap/mvp.md
View File

@@ -370,6 +370,15 @@ application-role releases in one source repository.
- The Narratio `testing` profile keeps the Notarius stage enabled and inherits - The Narratio `testing` profile keeps the Notarius stage enabled and inherits
the same D&D pipeline and extraction profile as production until a distinct the same D&D pipeline and extraction profile as production until a distinct
testing extraction policy is selected. testing extraction policy is selected.
- The D&D suite selects its Backblaze-compatible S3 archive, typed
`OBJECT_STORAGE_KEY_ID` and `OBJECT_STORAGE_KEY` references, maximum S3 audio
caching, run publication, human-facing transcript and artifact outputs, and
cleanup of successfully published workspace state. Feature 0015 owns the
exact overrideable defaults and publication set.
- Accepted Notarius lane outputs remain addressable inside Narratio as
`narratio.extraction.<output_key>` sources. Future analyze-stage Scriptorium
artifacts may consume those sources directly; publishing an extraction lane
to a stable remote destination is an independent policy choice.
### Execution model ### Execution model
@@ -389,7 +398,7 @@ later design decision changes them:
- `/usr/local/etc/notarius/pipelines` for suite-owned imported Notarius - `/usr/local/etc/notarius/pipelines` for suite-owned imported Notarius
pipeline definitions pipeline definitions
- `/usr/local/etc/scriptorium/config.yml` for Scriptorium configuration - `/usr/local/etc/scriptorium/config.yml` for Scriptorium configuration
- `/usr/local/etc/narratio/secrets` for future restricted credential material - `/usr/local/etc/narratio/secrets` for suite-managed restricted credentials
- `/usr/local/share/narratio/campaigns` for version-controlled campaign data - `/usr/local/share/narratio/campaigns` for version-controlled campaign data
- `/usr/local/share/narratio/prompt-library` for version-controlled prompts - `/usr/local/share/narratio/prompt-library` for version-controlled prompts
and schemas and schemas

10
docs/status.md
View File

@@ -1,10 +1,10 @@
# Workspace Status # Workspace Status
Last updated: 2026-09-18 Last updated: 2026-09-21
Current milestone: Existing pipeline operator enrollment Current milestone: Operational D&D pipeline deployment
Current focus: Independently review the Feature 0014 implementation. Current focus: Review the Feature 0015 staged implementation plan.
| ID | Deliverable | Owner | State | Evidence or next step | | ID | Deliverable | Owner | State | Evidence or next step |
| --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- |
@@ -30,7 +30,9 @@ Current focus: Independently review the Feature 0014 implementation.
| F0011 | Deliver pipeline credentials and Notarius reuse | ansible-role-narratio and ansible-role-dnd-pipeline | done | Implemented and independently reviewed through remediation commit `de1b191` | | F0011 | Deliver pipeline credentials and Notarius reuse | ansible-role-narratio and ansible-role-dnd-pipeline | done | Implemented and independently reviewed through remediation commit `de1b191` |
| F0012 | Assemble the D&D collection | ansible-collection-dnd and ansible-role-dnd-pipeline | done | Pipeline `600eb21` assembled and validated through collection remediation commit `da67b34` | | F0012 | Assemble the D&D collection | ansible-collection-dnd and ansible-role-dnd-pipeline | done | Pipeline `600eb21` assembled and validated through collection remediation commit `da67b34` |
| F0013 | Publish the first coordinated suite release | all Ansible repositories and workspace | done | Public [`rakestraw.dnd` 0.1.0 release](https://gitea.maximumdirect.net/eric/ansible-collection-dnd/releases/tag/v0.1.0); documentation remediation completed in pipeline `78afffe` and collection `5e695f3` | | F0013 | Publish the first coordinated suite release | all Ansible repositories and workspace | done | Public [`rakestraw.dnd` 0.1.0 release](https://gitea.maximumdirect.net/eric/ansible-collection-dnd/releases/tag/v0.1.0); documentation remediation completed in pipeline `78afffe` and collection `5e695f3` |
| F0014 | Enroll existing pipeline operators | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | in_progress | Review pipeline `c13f726a0a846180a1aa95d110b8bbc801edaae5` and collection `c0b0ebd13e92199f3d94e138c6cc50d1ece5dadd`; both complete integration suites passed | | F0014 | Enroll existing pipeline operators | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `c13f726` and collection import `c0b0ebd` passed their complete integration suites and independent review with no remediation findings |
| F0015 | Enable D&D S3 storage and publishing | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | ready | [Roadmap](roadmap/features/0015-dnd-s3-storage-and-publishing.md) approved; prepare the staged implementation plan |
| F0016 | Deploy the D&D pipeline to `pipeline-dnd` | rakestraw-ansible and workspace | proposed | [Roadmap](roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md) drafted; Feature 0015 is a prerequisite |
| A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` | | A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` |
| R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review | | R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review |