Compare commits

...

15 Commits

6 changed files with 719 additions and 315 deletions

View File

@@ -1,6 +1,6 @@
# Feature 0014: Enroll Existing Pipeline Operators
Status: in_progress
Status: done
## Summary

View File

@@ -0,0 +1,424 @@
# Feature 0015: Enable D&D S3 Storage and Publishing
Status: done
## Summary
Make the D&D pipeline operational with Narratio's current S3 storage,
publication, caching, cleanup, and restore capabilities. The pipeline role
will select the established Backblaze B2 destination, reference two typed
object-storage credentials delivered through its existing secrets directory,
publish the useful transcript and D&D artifact outputs, and retain maximum S3
audio caching.
The standalone Narratio role already exposes Narratio v2.1.1's complete
configuration surface for this behavior. This feature supplies D&D suite
policy through that existing interface; it does not add another storage
implementation or credential mechanism.
## Problem or Opportunity
The pipeline role currently installs Narratio v2.1.0 and configures the full
processing graph, but deliberately leaves Narratio storage and publishing at
their application-level defaults. Its README consequently describes storage,
publishing, and end-to-end external execution as deferred. The resulting suite
can validate configuration and process local state, but it does not select the
remote archive or publish the outputs needed for an operational deployment.
Narratio v2.1.0 also rejected a valid publish list when a rule explicitly set
`required` while omitting a derivable `dest`. Narratio v2.1.1 corrects that
resolution bookkeeping defect without changing the publish schema or
destination rules. This feature therefore advances the pipeline application
pin to v2.1.1 and retains explicit suite-owned requiredness together with
native destination derivation.
The prior test-machine configuration records the intended Backblaze B2
endpoint, bucket, prefix, credential names, publication set, and post-publish
workspace cleanup. Narratio's current contract has since replaced the legacy
credential field names and secrets configuration with ConfigKit typed
references and a conventional sibling `secrets/` directory. The Ansible suite
already implements that new credential-delivery boundary for Audita and
Notarius, so S3 needs only to join the same path.
## Governing Policies
- [The Ansible house style](../../ansible.md) requires suite policy to flow
through the standalone role's field-specific interface and optional fields
to remain application-owned unless the suite deliberately selects them.
- [The testing house style](../../testing.md) requires representative
public-interface and native-application validation without duplicating AWS
SDK, ConfigKit, or Ansible module behavior.
- [The MVP roadmap](../mvp.md) makes the pipeline role responsible for
cross-application configuration and end-to-end validation.
- [Feature 0011](0011-managed-pipeline-credential-delivery.md) establishes the
managed Narratio secrets directory and typed credential-forwarding policy.
- [Feature 0012](0012-collection-assembly.md) makes the standalone pipeline
repository authoritative and imports reviewed changes through public
Gitea.
- [Feature 0016](0016-deploy-dnd-pipeline-to-pipeline-dnd.md) consumes this
feature's reviewed interface for the first real-host deployment.
## Scope
- Advance the suite's Narratio application version from `v2.1.0` to the
backward-compatible `v2.1.1` bugfix release.
- Add suite-owned pipeline variables for the selected Narratio workspace,
S3, cache, and publish settings.
- Select the S3 backend with these D&D deployment defaults:
| Setting | Suite default |
| --- | --- |
| Endpoint | `https://s3.us-east-005.backblazeb2.com` |
| Bucket | `rakestraw-assets` |
| Root prefix | `dnd` |
| Region | `us-east-1` |
| Force path style | `true` |
| Access-key reference | `OBJECT_STORAGE_KEY_ID` |
| Secret-key reference | `OBJECT_STORAGE_KEY` |
- Pass the two credential reference names to the standalone Narratio role.
Their values use the existing `dnd_pipeline_secrets` map and managed
`/usr/local/etc/narratio/secrets/` directory.
- Enable S3 audio caching explicitly so the suite retains Narratio's maximum
available audio reuse.
- Enable run publication and publish these complete native output rules:
| Source | Required |
| --- | --- |
| `narratio.transcript.final` | `true` |
| `narratio.transcript.final_trimmed` | `true` |
| `narratio.transcript.final_markdown` | `true` |
| `narratio.transcript.final_trimmed_markdown` | `true` |
| `narratio.artifact.session_recap` | `true` |
| `narratio.artifact.hostile_rumors` | `true` |
| `narratio.artifact.deranged_rumors` | `false` |
- Let Narratio derive each destination from the built-in transcript path or
configured artifact output path rather than duplicating those paths in the
publish list.
- Enable workspace cleanup only after Narratio records successful run
publication. Leave spool-audio deletion under Narratio's existing default
because the suite has selected no policy requiring source-audio removal.
- Keep the production and `testing` profile storage and publishing behavior
identical. The existing overlay remains limited to model-profile selection.
- Document the four credentials normally required by the operational suite:
`AUDITA_LLM_API_KEY`, `OPENROUTER_API_KEY`,
`OBJECT_STORAGE_KEY_ID`, and `OBJECT_STORAGE_KEY`.
- Document the operator-visible publication, restore, S3-session-audio, lock,
caching, and cleanup capabilities enabled by the selected configuration,
while linking Narratio's canonical CLI and operations documentation for
command details.
- Preserve every accepted Notarius lane as its existing
`narratio.extraction.<output_key>` source. Future Narratio artifact
definitions can consume those sources during analyze without first making
the extraction lanes separate published outputs.
- Adapt the existing pipeline integration scenario to resolve synthetic S3
credentials and validate one explicitly supplied representative storage and
publishing configuration through Narratio's native interface without
contacting object storage.
- Import the reviewed public pipeline commit into the collection through the
established squashed subtree workflow, adapt the collection's artifact-level
fixture to supply the added credential names, and validate the assembled
artifact.
## Non-goals
- Implementing an S3 client, publication engine, remote commit format,
restore mechanism, publish-lock mechanism, or audio cache in Ansible.
Narratio owns all of those behaviors.
- Placing credential values in role defaults, templates, documentation,
tests, ordinary inventory, or the generated pipeline configuration.
- Adding a second secret-delivery mechanism, environment wrapper, service
unit, credential synchronizer, or AWS configuration-file manager.
- Managing Backblaze buckets, object lifecycle policy, application keys,
retention policy, versioning, encryption, quotas, or network access.
- Managing per-session `session.yml` files or selecting local versus S3 audio
input for an individual session.
- Running a paid or networked publication test from a repository-local test
suite. Feature 0016 owns the representative live end-to-end exercise.
- Publishing every Notarius extraction lane as a named current output. The
lanes remain available as analyze inputs independently of publication; stable
remote destinations require an explicit operator need.
- Setting Narratio's optional `notarius.chunk_cache` override. The managed
Notarius configuration remains the single owner of its ordinary `auto`
cache policy, and Narratio v2.1 already requests refresh for forced
extraction.
- Adding notification providers. Narratio v2.1 supports only its existing
`noop` notification mode.
- Releasing new role or collection versions as part of this implementation.
- Modifying Narratio, ConfigKit, or the standalone Narratio role. Narratio
v2.1.1 is an already published prerequisite, and the existing generic role
can install the selected application tag through `narratio_version`.
## Target End State
The standalone pipeline role exposes these suite variables with the selected
defaults:
```yaml
dnd_pipeline_narratio_version: v2.1.1
dnd_pipeline_narratio_workspace_cleanup_after_publish: true
dnd_pipeline_narratio_storage_backend: s3
dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets
dnd_pipeline_narratio_storage_s3_root_prefix: dnd
dnd_pipeline_narratio_storage_s3_region: us-east-1
dnd_pipeline_narratio_storage_s3_endpoint: >-
https://s3.us-east-005.backblazeb2.com
dnd_pipeline_narratio_storage_s3_force_path_style: true
dnd_pipeline_narratio_storage_s3_access_key_id: OBJECT_STORAGE_KEY_ID
dnd_pipeline_narratio_storage_s3_secret_access_key: OBJECT_STORAGE_KEY
dnd_pipeline_narratio_cache_s3_audio: true
dnd_pipeline_narratio_publish_enabled: true
dnd_pipeline_narratio_publish_upload_run: true
dnd_pipeline_narratio_publish_outputs:
- source: narratio.transcript.final
required: true
- source: narratio.transcript.final_trimmed
required: true
- source: narratio.transcript.final_markdown
required: true
- source: narratio.transcript.final_trimmed_markdown
required: true
- source: narratio.artifact.session_recap
required: true
- source: narratio.artifact.hostile_rumors
required: true
- source: narratio.artifact.deranged_rumors
required: false
```
Each value is declared in the pipeline argument specification and passed
directly to its corresponding `narratio_*` variable in the pipeline role's
Narratio dependency. Narratio's existing root template renders the native
configuration. No new pipeline fragment or production task is required.
The ordinary generated configuration contains only the credential names:
```yaml
storage:
backend: s3
s3:
access_key_id: OBJECT_STORAGE_KEY_ID
secret_access_key: OBJECT_STORAGE_KEY
```
Consumers supply the two values beside the existing model-provider values
through encrypted inventory:
```yaml
dnd_pipeline_secrets:
AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"
OBJECT_STORAGE_KEY_ID: "{{ vault_dnd_object_storage_key_id }}"
OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}"
```
The example defines references only; no roadmap, role, or test contains real
credential values. The existing credential task writes the selected entries
with the suite's established group-authorized secret modes. Omitted map keys
remain unmanaged, as defined by Feature 0011.
With valid credentials, Narratio can accept either local or S3 audio selected
by each session file, cache S3 audio beneath `/var/cache/narratio`, publish an
immutable run and the selected stable outputs, use remote publication locks,
restore committed state, and clean the published workspace. These are native
Narratio commands and lifecycle rules rather than additional Ansible tasks.
The ten accepted Notarius outputs continue to enter Narratio's artifact
catalog as `narratio.extraction.*` sources. Analyze-stage Scriptorium artifacts
may reference those sources directly in their input mappings. Feature 0015
does not yet add such artifact definitions, but its publication choices do not
remove, rename, or constrain this future composition path.
## Repository Ownership
- `ansible-role-dnd-pipeline` owns the D&D suite variables, opinionated
defaults, dependency mappings, consumer documentation, and representative
integration scenario.
- `ansible-role-narratio` remains the generic owner of installation and native
pipeline rendering. Its reviewed `v0.1.0` role interface is compatible with
Narratio v2.1.1 and already covers every
configuration field required by this feature and should remain unchanged.
- `ansible-collection-dnd` owns the faithful pipeline subtree import, the
unified consumer dependency manifest, and artifact-boundary validation.
- `ansible-workspace-dnd` owns this roadmap, the later implementation plan,
status, and independent review record.
- `narratio` v2.1.1 is the released compatibility prerequisite and a read-only
source of storage, publication, credential, caching, cleanup, restore, and
CLI contracts during this implementation.
- `rakestraw-ansible` remains unchanged until Feature 0016 consumes this
interface on `pipeline-dnd`.
## Constraints and Settled Decisions
- The S3 endpoint, bucket, prefix, region, and path-style values are deliberate
D&D suite defaults. They are ordinary variables and remain overrideable by
inventory.
- Credential references are also policy defaults, but credential values have
no defaults. Operators must supply matching entries through
`dnd_pipeline_secrets` or another environment source supported by Narratio.
- The two credential-reference variables are nullable as a pair. Setting both
to `null` omits the references and preserves Narratio's AWS SDK provider-chain
path; configuring only one remains invalid under Narratio's native contract.
- The role does not assert that the secrets map contains particular keys.
Narratio's native loader owns typed-reference resolution, and external
environment values remain a supported higher-precedence source.
- `dnd_pipeline_narratio_publish_outputs` is one complete native list. A caller
replacing it assumes responsibility for the entire selected publication
set; the role does not merge rules.
- Publish destinations remain omitted where Narratio can derive them. The
configured source identity and artifact output path remain the single source
of truth.
- Narratio v2.1.1 is the minimum application version for this feature because
it preserves destination derivation when `required` is explicit. Do not work
around older behavior by duplicating destinations or omitting deliberate
requiredness.
- Workspace cleanup is gated by Narratio's own successful-publication
evidence. Ansible does not independently inspect remote state or remove
runtime content.
- S3 audio caching is enabled, and ordinary cleanup preserves the cache.
Explicit operator cache clearing remains a Narratio CLI operation.
- The testing profile does not silently disable Notarius, storage,
publication, cleanup, or credentials. It continues to differ only in the
already selected inexpensive Scriptorium model profile.
- S3 storage also enables Narratio's native restore and publication-lock
commands without another configuration surface. Ansible documentation may
describe availability but must not duplicate Narratio's detailed operating
manual.
- Notarius extraction-source availability and remote publication are separate
concerns. Analyze consumes `narratio.extraction.*` sources from the accepted
extraction catalog; `publish.outputs` selects only which sources also receive
stable published destinations.
## Simplest Viable Implementation
Advance `dnd_pipeline_narratio_version` to v2.1.1. Add the selected variables to
the pipeline defaults and argument specification, then map them into the
existing `rakestraw.narratio` dependency. Update the pipeline README to document
the policy, credential names, complete-list replacement semantics, and native
capabilities now available. Do not add a template, task, handler, fact, probe,
or storage-specific validation layer.
In the existing pipeline integration driver, extend the synthetic secrets map
with object-storage values and explicitly supply representative S3 and publish
variables. Use `narratio config validate` and its secret-safe effective
configuration output to prove that the public inputs compose into a valid
configuration. Retain the immediate clean repeat and secret non-disclosure
checks. Do not call a real S3 service or assert that the representative values
equal current role defaults.
Push the reviewed pipeline commit, import that exact public revision into the
collection through `git subtree pull --squash`, and extend only the
collection-level fixture inputs needed for native validation. The collection
suite should continue to prove artifact installation and one representative
assembled convergence rather than duplicate the detailed pipeline matrix.
## Rationale and Alternatives
Implementing storage behavior in Ansible would duplicate a mature native
application boundary. Narratio already owns credential resolution, AWS SDK
provider construction, immutable commits, verification, locks, restore,
caching, and cleanup. Passing declarative configuration through the generic
role is both smaller and more reliable.
Using the AWS SDK's ambient provider chain would avoid named secret files, but
the initial deployment has explicit Backblaze application credentials and the
suite already has a suitable managed typed-secret path. Keeping the two
credential-reference variables nullable preserves the ambient-provider
alternative: consumers can set both suite variables to `null` when their
deployment uses another provider.
Copying destinations into every publish rule would make the output layout more
visible locally, but it would duplicate paths already owned by Narratio's
built-in sources and the configured artifact definitions. Native destination
derivation keeps path coordination automatic.
Omitting explicit `required: true` values would avoid the v2.1.0 failure only
by relying on an unrelated defaulting path. Changing the generic role's YAML
format would not repair the underlying sequence-resolution mismatch. Narratio
v2.1.1 fixes the owning application boundary, allowing the suite to state its
requiredness policy directly without duplicating destinations.
Deleting spool audio after publication would minimize transient disk usage,
but no such retention policy has been selected, and local-audio sessions may
depend on that content. Leaving the field omitted preserves Narratio's default
while enabling the explicitly requested S3 audio cache.
Publishing all ten Notarius extraction lanes would create stable remote output
contracts that have not been requested. It is also unnecessary for the stated
future use: analyze-stage artifacts can consume `narratio.extraction.*`
directly. The immutable published run retains the processing result, while the
selected transcript and human-facing artifact outputs match the known
operational workflow.
## Acceptance Criteria
- The pipeline role declares and documents every suite variable shown in the
target state, with matching argument-spec entries and direct Narratio
dependency mappings.
- The pipeline installs Narratio v2.1.1, whose released configuration loader
supports explicit requiredness together with derived publish destinations.
- The two credential-reference variables accept paired `null` overrides so a
caller can deliberately select Narratio's ambient AWS SDK provider chain.
- The default effective Narratio configuration selects the specified
Backblaze-compatible S3 destination, typed credential names, S3 audio cache,
successful-publication workspace cleanup, run upload, and seven output
rules.
- The publish list relies on Narratio's native destination derivation and
retains `deranged_rumors` as optional while the other six outputs are
required.
- Spool-audio deletion remains unset by the pipeline role, and the testing
profile introduces no storage or publication override.
- `dnd_pipeline_secrets` remains a generic empty-by-default mapping; the role
documents all four expected operational entries without embedding any
credential value.
- No new production task, template, handler, fact, state file, wrapper, or
storage-validation command is introduced.
- The standalone Narratio role remains unchanged because its `v0.1.0` role
interface can install Narratio v2.1.1, renders the required native fields,
and preserves null omission.
- The existing pipeline integration scenario explicitly supplies
representative S3 and publish inputs, resolves synthetic values for all four
typed credentials, passes Narratio's native configuration validation, and
exposes no synthetic value in captured output.
- The immediate identical second pipeline application reports `changed=0`
and `failed=0`; syntax checking, production-profile lint, the complete
standalone integration suite, and `git diff --check` pass.
- The exact reviewed pipeline commit is pushed to public Gitea before the
collection imports it through the documented remote-only squashed subtree
workflow.
- The collection fixture supplies the added synthetic credential names, and
its existing build, isolated installation, FQCN resolution, native
validation, and clean-repeat suite passes without a duplicate S3 behavior
test.
- No real object-storage request, paid model request, application repository
change during the remaining implementation, asset change, new release tag,
or published artifact is required. The existing Narratio v2.1.1 release is a
prerequisite rather than an implementation deliverable.
- Pipeline documentation links operators to Narratio's canonical procedures
for S3 session audio, publish, restore, locks, status, and cleanup and no
longer describes storage and publishing as deferred.
- Documentation distinguishes accepted Notarius extraction sources from
published outputs and confirms that future analyze-stage artifacts may use
`narratio.extraction.<output_key>` inputs without publishing those lanes.
## Dependencies and Follow-up Work
Features 0009, 0011, 0012, 0013, and 0014 and the public Narratio v2.1.1 bugfix
release are prerequisites. In particular, Feature 0011 supplies the credential
directory and Feature 0014 supplies the human-operator access needed for the
eventual live deployment.
[Feature 0016](0016-deploy-dnd-pipeline-to-pipeline-dnd.md) follows this work.
It will put the four actual values in encrypted host inventory, pin the exact
reviewed suite revision in `rakestraw-ansible`, converge `pipeline-dnd`, and
exercise one real session through remote publication. Bucket lifecycle policy,
backup policy, credential rotation, additional publish outputs, automated
scheduling, and cleanup of the proof-of-concept deployment remain separate
future work.
A later artifact-development feature may add D&D-specific Narratio artifact
definitions whose Scriptorium input mappings consume selected
`narratio.extraction.*` sources. That feature should define each artifact's
purpose, inputs, prompt, output contract, and publication choice; it does not
require a storage-interface change.

View File

@@ -0,0 +1,145 @@
# Feature 0016: Prepare the D&D Deployment Playbook for `pipeline-dnd`
Status: in_progress
## Summary
Write and commit the consumer playbook, necessary host variables, dependency
manifest, and concise usage documentation in `rakestraw-ansible`. This feature
ends with repository preparation. Deployment to a controller or target host
and execution of the pipeline are separate operator work.
## Scope
- `playbooks/pipeline/dnd.yml` targets the existing `pipeline-dnd` alias and
applies `rakestraw.dnd.pipeline`, using only hosts and roles.
- Host choices belong in `inventory/host_vars/pipeline-dnd/`: the WhisperX
endpoint, initial operator `eric`, default campaign `dilfs`, and credential
references. Unknown required values are explained in adjacent comments.
- Inherit the reviewed suite's storage, publication, caching, cleanup, and
application defaults. Retire obsolete legacy host variables.
- Root `requirements.yml` selects the reviewed public collection snapshot and
its role/native collection dependencies, preserving unrelated entries.
- Document local validation and eventual operator use. Commit the completed
repository changes without requiring remote access or real credentials.
## Non-goals
- Installing or updating anything on the operational controller or target.
- Connecting to `pipeline-dnd`, validating its platform or sudo access,
converging the host, checking runtime identities, or executing a real session.
- Obtaining API keys, choosing a Vault password, populating real credentials,
or selecting acceptance audio/session data.
- Changing application/role behavior, publishing a new suite release, adding
orchestration layers, or retiring the proof-of-concept role/playbook.
## Target End State
The committed playbook is:
```yaml
---
- hosts:
- pipeline-dnd
roles:
- rakestraw.dnd.pipeline
...
```
Existing connection variables remain in the host's `main.yml`. Its
`narratio.yml` contains the following public configuration and comments:
```yaml
dnd_pipeline_operator_users:
- eric
dnd_pipeline_whisperx_transcribe_url: >-
https://transcription.ai.rakestrawhome.com/transcribe
narratio_campaigns_default_campaign_id: dilfs
# Before deployment, define the four vault_dnd_* variables below in an
# Ansible Vault-encrypted host_vars/pipeline-dnd/vault.yml file.
# Values are not known here; leave these references unresolved until supplied.
# Do not put API keys or object-storage credentials in this plaintext file.
dnd_pipeline_secrets:
AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"
OBJECT_STORAGE_KEY_ID: "{{ vault_dnd_object_storage_key_id }}"
OBJECT_STORAGE_KEY: "{{ vault_dnd_object_storage_key }}"
```
The references and comments are a complete deliverable for unknown secrets.
No encrypted file, placeholder key, or real credential is required for feature
completion. Future operator-supplied values should be Vault-encrypted in Git;
the suite's existing restricted runtime secrets files remain the delivery path.
The manifest pins the collection to
`46913446765e86ecfb29ebad9955d579ef6a4c7e`, containing reviewed pipeline commit
`51566afcf69083cdbb5b5718f2544c86b079001c`. Its five standalone role dependencies
use their reviewed `v0.1.0` tags, with `ansible.posix` `2.2.0` and
`community.general` `13.3.0`. Suite Git dependencies use public Gitea URLs;
native collections come from Galaxy. Local sibling checkouts are not dependencies.
The runbook explains that the selected collection snapshot still reports
`0.1.0`, so eventual operator installation must explicitly refresh it from
the pinned commit if an older `0.1.0` is installed. This is an instruction
for future use, not an operation required by this feature.
## Repository Ownership
- `rakestraw-ansible`: consumer files, comments, dependency manifest, and runbook.
- Workspace: roadmap, staged plan, progress, and independent review record.
- Roles, collection, applications, and assets: read-only contract references.
## Constraints and Settled Decisions
The workspace [Ansible](../../ansible.md), [testing](../../testing.md), and
[planning](../../planning.md) policies apply. Feature 0015 has passed review
and supplies the required suite interface.
- Keep playbooks minimal and settings in inventory. Preserve `pipeline-dnd`
and its connection identity; do not rename it to `pipeline.dnd`.
- Preserve `dilfs` using the existing standalone Narratio variable. Remove
legacy `narratio_s3_*` and `narratio_default_campaign_id` host settings.
- Missing operational inputs are documented, not treated as implementation
blockers. No remote controller, SSH identity, Vault password, or session is
required for repository preparation.
- Local dependency installation and syntax/host-selection checks are permitted.
Existing passing evidence may be reused when relevant inputs are unchanged.
- `ansible-doc` rendering is not an acceptance gate. Missing role argument
descriptions are a separate documentation improvement.
## Simplest Viable Implementation
Retain the already-committed minimal playbook, manifest, and public inventory.
Add adjacent comments for unresolved credentials, align the runbook with the
repository-only scope, perform appropriate local checks, and commit. No new
Ansible task, runtime test harness, or deployment automation is needed.
## Acceptance Criteria
- The playbook targets only `pipeline-dnd` and applies only the collection role.
- Host inventory expresses the known choices above and documents unknown
credentials next to their references without real or fake secret values.
- The dependency manifest contains the reviewed public pins and preserves
unrelated dependencies. Public installation and local syntax/host-selection
evidence are available; no documentation-rendering check is required.
- The runbook distinguishes repository readiness from future deployment and
explains how the operator supplies missing values and eventually invokes it.
- Changes are committed in the owning repositories, with clean working trees
for the feature's changes. No controller deployment, target convergence,
clean-repeat run, or real pipeline execution is required or claimed.
- Historical role/playbook files remain available.
## Dependencies and Follow-up Work
Features 0013–0015 supply public dependencies, operator enrollment, and storage
policy. Feature 0016's consumer preparation already exists in `12b2462`, with
initial documentation in `716a80e`; remaining work aligns comments and docs
with the user's clarified scope.
The operator may later provision Vault values, install dependencies on their
controller, deploy to the host, and validate real processing/publication.
Those actions require a separate request and are not unfinished Feature 0016
acceptance requirements.

View File

@@ -1,372 +1,193 @@
# Implementation Plan 0014: Enroll Existing Pipeline Operators
# Implementation Plan 0016: Prepare the D&D Deployment Playbook
Status: ready
Status: in_progress
Target executor: GPT-5.6-Terra
Feature roadmap:
`docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md`
Feature roadmap: `docs/roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md`
Active plan location: `docs/roadmap/implementation.md`
## Objective
Add one empty-by-default pipeline interface that enrolls explicitly selected
existing human accounts in the suite's runtime group. Implement the behavior
with one account-existence lookup and one additive membership task, adapt the
existing operator integration scenario, and import the reviewed result into
the collection from public Gitea.
The Feature 0014 roadmap remains authoritative for scope, rationale, target
state, and acceptance criteria. This plan does not authorize human-account
creation or removal, exact-list reconciliation, filesystem-policy changes, or
a new release.
Finish and commit repository preparation in `rakestraw-ansible`: minimal
playbook, host variables with comments for unknown values, reviewed dependency
pins, and concise operator instructions. This revision supersedes all previous
deployment and live-acceptance stages. Do not execute them.
## Execution Rules
- Execute stages in numeric order.
- Before each stage, confirm its preconditions and the working-tree state of
its owning repository.
- Before each stage, reread every document listed in that stage's Required
Policy Review. Do not rely on policy recalled from an earlier stage.
- Preserve unrelated and pre-existing changes. Stop and report unexpected
worktree changes or branch divergence rather than overwriting them.
- Do not make unresolved product or architecture decisions during execution;
stop and report the decision required.
- Keep the production behavior to the roadmap's exact direct task flow: one
`ansible.builtin.getent` loop followed by one `ansible.builtin.user` loop.
- Do not add an assertion, registered fact, handler, command, shell operation,
explicit empty-list condition, membership comparison, post-change query,
sidecar state, or account abstraction.
- Trust `getent` to fail for a missing key and `user` to converge additive
membership. Do not reproduce either module's checks.
- Test the public behavior by adapting the existing temporary-operator
scenario. Do not add a missing-user test, inspect `/etc/passwd` or
`/etc/group`, assert task results, or duplicate the detailed test in the
collection.
- Run each stage's validation before committing it. Do not proceed after a
validation failure.
- Commit each completed stage with the stated plain-English intent. Push the
authoritative pipeline commit before importing it into the collection.
- Use only the public Gitea pipeline URL for the subtree update; do not import
from the local sibling repository.
- Execute the stages below in order, rereading each stage's listed policies.
Check working trees first, preserve unrelated work, validate, and commit each
completed unit with its stated plain-English intent.
- Do not deploy to an operational controller or target, connect to the target,
populate actual credentials, or run a real pipeline session.
- Missing controller, SSH, Vault, and session inputs are not blockers. Express
unknown required values through commented instructions and variable references.
- Keep playbooks limited to hosts and roles. No new tasks, wrapper, probes,
runtime assertions, or automated test harness is required.
- Use existing workspace `.venv/bin` executables for local Ansible checks.
Core 2.21.3 is installed; a missing `PATH` entry or `/etc/ansible` directory
does not mean tooling is missing. Explicit local paths suffice.
- Reuse passing validation whose inputs are unchanged. Optional diagnostics
and `ansible-doc` documentation rendering are not acceptance gates.
- Keep credential values out of plaintext files, Git, commands, and logs.
Do not create fake keys or an empty Vault file to satisfy inventory references.
## Starting State and Preconditions
## Starting State and Repository Scope
- Features 0009, 0011, 0012, and 0013 are complete.
- This roadmap and implementation plan have been reviewed and committed in the
workspace repository before execution begins.
- The completed Feature 0013 closeout commits `eda361f` in the workspace and
`5e695f3` in the collection are pushed to their respective `origin/main`
branches before implementation begins. Preserve the Git-generated subtree
squash commit that accompanies `5e695f3`.
- `ansible-role-dnd-pipeline` starts clean on `main` at `78afffe`, equal to
`origin/main`.
- `ansible-collection-dnd` starts clean on `main` at `5e695f3`, equal to
`origin/main` after the prerequisite push.
- The workspace `.venv` supplies the tested `ansible-playbook`,
`ansible-galaxy`, and `ansible-lint` commands used by the repository test
drivers.
- The local host remains the approved Ubuntu 26.04 proxy, and passwordless
test-time sudo is available for the existing integration fixtures.
- Git credentials can push the pipeline and collection commits to their public
Gitea origins.
- Feature 0015 is complete and independently reviewed.
- Consumer `716a80e` added the initial runbook; `12b2462` committed the new
playbook, host configuration, and dependencies. Inspect intervening commits;
do not repeat or discard completed work.
- Public dependency installation, target-only host listing, and syntax checking
passed for the prepared consumer configuration. The missing option descriptions
in three roles affect `ansible-doc` only and are not part of this feature.
- `rakestraw-ansible` owns remaining comments/runbook changes. The workspace
owns status and review handoff. All other repositories remain read-only.
- No credentials, controller access, target access, or session selection is
needed to complete this plan.
## Repository Scope
## Stage 1: Finish Consumer Comments and Usage Documentation
- `ansible-role-dnd-pipeline` owns the variable, argument specification,
account lookup, additive enrollment task, README, and detailed integration
behavior.
- `ansible-collection-dnd` receives the reviewed pipeline commit only through
its established squashed subtree and validates the assembled artifact.
- `ansible-workspace-dnd` records implementation progress and the review
handoff after the two implementation repositories are complete.
- All standalone application-role repositories, application source
repositories, asset repositories, and `rakestraw-ansible` are out of scope
and must remain unchanged.
Completion: complete (consumer `7f8353d74efabb5ae408e41be9837df53010dd95`)
## Stage 1: Implement Existing-Operator Enrollment
### Objective and Owning Repository
### Objective
Give the standalone pipeline role its minimal public interface for enrolling
existing human accounts and prove the behavior through the existing operator
integration scenario.
Make the existing committed consumer configuration self-explanatory in
`rakestraw-ansible` and remove live execution as a completion requirement.
### Required Policy Review
Before beginning this stage, read:
- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/ansible.md`
- `docs/testing.md`
- `docs/planning.md`
- `docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md`
Before this stage, read `AGENTS.md`, `docs/orientation.md`,
`docs/conventions.md`, `docs/ansible.md`, `docs/testing.md`,
`docs/planning.md`, the Feature 0016 roadmap, and the consumer README/runbook.
### Preconditions
- The workspace planning commit is present.
- `ansible-role-dnd-pipeline` is clean at `78afffe` on `main`, is not behind
or diverged from `origin/main`, and contains no unrelated changes.
- Its Narratio dependency still creates the selected runtime group before the
pipeline role's ordinary tasks execute.
- The existing integration driver still contains the temporary operator and
representative group-access scenario described by the roadmap.
### Owning Repository
`ansible-role-dnd-pipeline`
Inspect the current consumer changes and the two completed commits above.
Local Ansible tools are available. There are no remote prerequisites.
### Required Changes
1. Add `dnd_pipeline_operator_users: []` beside the existing runtime identity
defaults in `defaults/main.yml`.
2. Add the matching `meta/argument_specs.yml` entry with `type: list`,
`elements: str`, and the same required/default-backed convention used by
the role's other public variables.
3. Add one unprivileged `ansible.builtin.getent` task near the beginning of
`tasks/main.yml`. Loop directly over `dnd_pipeline_operator_users`, use the
`passwd` database and each username as `key`, and rely on the module's
default missing-key failure. Do not register its result or add a separate
assertion.
4. Immediately follow it with one `become: true` `ansible.builtin.user` task
over the same list. Set `name` to the selected account,
`groups` to `dnd_pipeline_runtime_group`, `append: true`, and
`state: present`. Do not set or manage any other account property.
5. Revise the README variable reference and shared-execution section. Replace
the consumer-owned membership example and deferred wording with the new
interface, existing-account prerequisite, additive non-removal behavior,
authorization consequence, and new-login-session caveat.
6. Adapt `tests/integration` rather than adding another scenario. Create its
temporary operator before the initial role application without membership
in the runtime group, pass that username through
`dnd_pipeline_operator_users` for the baseline and repeated applications,
and retain the existing native configuration, secret-boundary, and shared
cache exercises under that identity. Reorder cleanup as necessary so any
final baseline convergence occurs while the selected fixture account still
exists, then remove the fixture account.
7. Retain the existing clean-repeat assertion. Do not add a negative
missing-user case or assertions about module internals, task traces, the
current default value, or raw account database entries.
8. Run the complete existing repository validation, commit the result, push
`main` to the public pipeline origin, and record the full pushed commit for
Stage 2.
1. Retain `playbooks/pipeline/dnd.yml`, which contains only the `pipeline-dnd`
host and `rakestraw.dnd.pipeline` role. Retain reviewed dependency pins and
known host values. Compare against the revised roadmap; correct only actual
discrepancies, without adding suite-default copies or extra play content.
2. In `inventory/host_vars/pipeline-dnd/narratio.yml`, add the roadmap's comments
beside `dnd_pipeline_secrets`: define the four referenced `vault_dnd_*`
variables in an encrypted `vault.yml` before eventual deployment. Retain
unresolved references. Do not obtain secret values or create a Vault file.
3. Revise `docs/pipeline-dnd.md` to describe repository preparation as the
completed deliverable. Clearly label installation, credential provisioning,
and deployment commands as future operator instructions, not steps the
implementation agent must execute. Remove operational prerequisite checklists
and session-acceptance requirements from this feature's completion path.
4. Retain useful concise future-use instructions: the pinned collection refresh,
Vault variable names and selection options, and minimal playbook invocation.
Link application documentation for later pipeline operation rather than
maintaining a live-session test script or demanding session selection now.
5. Preserve connection inventory, unrelated dependencies, and the historical
proof-of-concept playbook/role.
### Validation
Run from the workspace root:
Run `git diff --check` in `rakestraw-ansible` and review the changed documentation
and comments. Reuse the recorded successful public installation, syntax, and
host-listing results if only comments/documentation change. No runtime tests
or repeated dependency installation are necessary for those edits.
If executable configuration changes, rerun only affected local checks using
the workspace tools and explicit local inventory/dependency paths:
```text
(cd ansible-role-dnd-pipeline && tests/integration && git diff --check)
ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --list-hosts
ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --syntax-check
```
Expected result: syntax and production-profile lint pass; the temporary
existing account is enrolled through the public role variable and completes
the existing representative operator-access scenario; native application
checks pass; the immediate identical role application reports `changed=0`
and `failed=0`; and no credential value appears in test output.
Use previously installed isolated public dependencies when available. If needed,
install from `requirements.yml` into disposable local role/collection paths.
Do not use operational `/etc/ansible` destinations or local sibling role
sources. Keep unresolved credentials untouched; local syntax checking does
not claim they resolve. If unrelated encrypted inventory prevents a local
check, use a disposable inventory containing the target alias and non-secret
variables, and report that narrower validation scope.
### Completion Criteria
- The public interface, argument specification, two direct tasks, and README
implement exactly the roadmap contract.
- The existing operator scenario uses the role interface rather than fixture
group enrollment.
- The full standalone integration suite and `git diff --check` pass.
- The role repository is clean after a plain-English commit.
- The exact commit is present on public `origin/main` and its full hash is
available for the collection import.
The required public files exist, unknown values have explanatory comments,
local validation evidence is sufficient, and the consumer changes are committed.
No deployment or real credential provisioning is required.
### Commit
Commit intent: `Add optional pipeline operator enrollment`
`Document the remaining D&D deployment inputs`.
## Stage 2: Import the Reviewed Pipeline Change
## Stage 2: Record the Repository Preparation Handoff
### Objective
Completion: complete (handoff recorded by this change)
Expose the reviewed operator interface through `rakestraw.dnd.pipeline` while
preserving the authoritative-standalone-repository and collection-artifact
boundaries.
### Objective and Owning Repository
Record Feature 0016 implementation readiness for independent review in the
workspace, scoped entirely to committed repository content.
### Required Policy Review
Before beginning this stage, read:
- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/ansible.md`
- `docs/testing.md`
- `docs/planning.md`
- `docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md`
- `ansible-collection-dnd/README.md`
Before this stage, read `AGENTS.md`, `docs/orientation.md`,
`docs/conventions.md`, `docs/testing.md`, `docs/planning.md`, and the revised
Feature 0016 roadmap.
### Preconditions
- Stage 1 is complete and its exact commit is publicly reachable from
`https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git`.
- `ansible-collection-dnd` is clean at `5e695f3` on `main`, is not behind or
diverged from `origin/main`, and contains no unrelated changes.
- The collection's current `roles/pipeline` subtree still corresponds to the
authoritative pipeline history through `78afffe`.
### Owning Repository
`ansible-collection-dnd`
Stage 1 has passed local validation and is committed. No remote prerequisites.
### Required Changes
1. Set a shell variable to the full pushed Stage 1 commit hash and run the
documented remote-only update from the collection repository:
1. Update `docs/status.md` and this plan's progress notes with the consumer
commit and local validation evidence. Describe real deployment as separately
deferred operator work, not a blocker or incomplete acceptance criterion.
2. Keep Feature 0016 `in_progress` pending independent review. Report any actual
repository issues; do not mark it `done` solely because implementation ended.
3. Confirm the feature's owning working trees are clean after committing.
Preserve any unrelated changes and identify them instead of discarding them.
```text
pipeline_revision=<FULL_PUSHED_STAGE_1_COMMIT>
git subtree pull --prefix=roles/pipeline \
https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \
"$pipeline_revision" --squash -m "Import pipeline operator enrollment"
```
### Validation and Completion Criteria
2. Do not edit role behavior directly beneath `roles/pipeline`. The subtree
import should carry the new default, argument specification, tasks, README,
and standalone tests faithfully.
3. Do not change `galaxy.yml`, dependency manifests, collection-level tests,
or release documentation. No dependency or published version changes in
this feature, and the detailed behavior remains covered by Stage 1.
4. Run the existing collection integration suite against the built and
installed artifact. Commit no additional files after the subtree command
unless validation exposes a genuine collection-boundary issue within this
feature's scope.
5. Push the resulting collection history to `origin/main` and record the
resulting merge commit and Git-generated squashed subtree commit.
### Validation
Run from the workspace root:
```text
(cd ansible-collection-dnd && tests/integration && git diff --check)
```
Expected result: the collection builds; the archive contains the updated
pipeline role while retaining its development-file exclusions; isolated
dependency and artifact installation succeeds; the FQCN resolves; native
representative application checks pass; and the immediate repeat reports
`changed=0` and `failed=0`.
### Completion Criteria
- Git history records a squashed subtree import from the exact public Stage 1
commit with the stated plain-English merge message.
- No pipeline behavior was authored directly in the collection.
- The existing collection suite and `git diff --check` pass without an
unnecessary duplicate operator test.
- The collection is clean and its new history is present on `origin/main`.
Run `git diff --check`. Evidence points to committed consumer content and
appropriate local checks. The handoff makes no claims of live deployment or
runtime acceptance. No paid request or host access is necessary.
### Commit
Commit intent: `Import pipeline operator enrollment`
The subtree command creates the stage's merge commit and its generated squash
commit; do not add an empty or redundant manual commit.
## Stage 3: Record the Implementation Handoff
### Objective
Record the completed implementation evidence without marking Feature 0014
done before independent review.
### Required Policy Review
Before beginning this stage, read:
- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/planning.md`
- `docs/roadmap/features/0014-existing-pipeline-operator-enrollment.md`
### Preconditions
- Stages 1 and 2 are complete, committed, pushed, and validated.
- Their full commit hashes and validation results are available.
- The workspace repository is clean apart from the expected progress-record
edits for this stage and is not behind or diverged from `origin/main`.
### Owning Repository
`ansible-workspace-dnd`
### Required Changes
1. Change Feature 0014's roadmap status and status-table state to
`in_progress`, indicating implementation is complete but independent review
remains outstanding.
2. Update the workspace current focus and Feature 0014 evidence with the exact
pipeline and collection commit hashes and the validation completed.
3. Replace this plan's Progress Notes with a concise record of the three stage
states and their commits. Do not copy command logs into the repository.
4. Do not mark the feature `done`, remove the active plan, or create
`docs/roadmap/remediation.md`; those are review and closeout decisions.
5. Commit the workspace handoff record with the stated plain-English message.
### Validation
Run from the workspace root:
```text
./scripts/status
git diff --check
```
Expected result: every repository is clean; the pipeline and collection are
not ahead of or behind their origins; only the workspace handoff commit may be
ahead of `origin/main`; and the dashboard points to the exact implementation
evidence while retaining independent review as the next step.
### Completion Criteria
- Workspace status accurately identifies Feature 0014 as implemented and
awaiting review.
- The roadmap remains the durable target-state record and is not polluted with
execution logs.
- The active plan records concise implementation evidence and contains no
false completion claim.
- The workspace repository is clean after the handoff commit.
### Commit
Commit intent: `Record pipeline operator enrollment implementation`
`Record D&D playbook preparation for review`.
## Final Validation and Handoff
- Confirm Stage 1's complete standalone integration result and Stage 2's
complete collection integration result.
- Run `./scripts/status` and confirm every repository is clean, with no
unexpected branch divergence or untracked artifact.
- Report the full pipeline, collection merge, collection subtree-squash, and
workspace handoff commit hashes.
- Summarize any deviation from this plan, unresolved finding, or added
implementation complexity and identify the acceptance criterion that
required it.
- Do not mark Feature 0014 `done`; implementation results require independent
review first.
Report owning commits, changed consumer files, comments explaining missing
values, and local validation performed or reused. Feature review evaluates the
committed playbook/configuration, not operational deployment. No push is
required unless separately requested.
## Progress Notes
- Stage 1 complete: `ansible-role-dnd-pipeline`
`c13f726a0a846180a1aa95d110b8bbc801edaae5` implements optional enrollment
of existing operators; the complete standalone integration suite passed and
the commit is public on `origin/main`.
- Stage 2 complete: `ansible-collection-dnd` merge
`c0b0ebd13e92199f3d94e138c6cc50d1ece5dadd` imports the exact public pipeline
change through squash `d292297b6ad179e4664c01ad07cddaefff12f769`;
the complete collection integration suite passed and the history is public
on `origin/main`.
- Stage 3 complete: `ansible-workspace-dnd`
`bf81461de60001a0d8050e560383c6ddf533209f` records the implementation
evidence while Feature 0014 remains `in_progress` for independent review.
- Deviations: none.
- Consumer `716a80e` added the initial runbook, and `12b2462` committed the
minimal playbook, public host choices, and reviewed dependency manifest.
Public dependency installation from the mixed manifest succeeded, host
listing selected only `pipeline-dnd`, and syntax checking passed.
- Stage 1 is complete in consumer commit
`7f8353d74efabb5ae408e41be9837df53010dd95`. The plaintext inventory now
explains its unresolved Vault references, and the runbook distinguishes the
completed repository preparation from future operator installation,
credential provisioning, deployment, and application operation.
- Stage 1 changed only comments and documentation, so the prior installation,
host-listing, and syntax evidence was reused as directed. `git diff --check`
passed, and the consumer working tree is clean.
- Stage 2 records the implementation handoff for independent review. No live
deployment, target validation, credential provisioning, or pipeline session
was performed or is claimed by Feature 0016.
## Open Questions
None required for this repository-only implementation. Unknown API keys and
future operational inputs are documented for the operator, not requested now.

View File

@@ -370,6 +370,15 @@ application-role releases in one source repository.
- The Narratio `testing` profile keeps the Notarius stage enabled and inherits
the same D&D pipeline and extraction profile as production until a distinct
testing extraction policy is selected.
- The D&D suite selects its Backblaze-compatible S3 archive, typed
`OBJECT_STORAGE_KEY_ID` and `OBJECT_STORAGE_KEY` references, maximum S3 audio
caching, run publication, human-facing transcript and artifact outputs, and
cleanup of successfully published workspace state. Feature 0015 owns the
exact overrideable defaults and publication set.
- Accepted Notarius lane outputs remain addressable inside Narratio as
`narratio.extraction.<output_key>` sources. Future analyze-stage Scriptorium
artifacts may consume those sources directly; publishing an extraction lane
to a stable remote destination is an independent policy choice.
### Execution model
@@ -389,7 +398,7 @@ later design decision changes them:
- `/usr/local/etc/notarius/pipelines` for suite-owned imported Notarius
pipeline definitions
- `/usr/local/etc/scriptorium/config.yml` for Scriptorium configuration
- `/usr/local/etc/narratio/secrets` for future restricted credential material
- `/usr/local/etc/narratio/secrets` for suite-managed restricted credentials
- `/usr/local/share/narratio/campaigns` for version-controlled campaign data
- `/usr/local/share/narratio/prompt-library` for version-controlled prompts
and schemas

View File

@@ -1,10 +1,13 @@
# Workspace Status
Last updated: 2026-09-18
Last updated: 2026-09-21
Current milestone: Existing pipeline operator enrollment
Current milestone: Operational D&D pipeline deployment
Current focus: Independently review the Feature 0014 implementation.
Current focus: Independently review the completed Feature 0016 repository
preparation. Controller deployment, target convergence, credential provisioning,
and session execution are separate operator work and are not claimed here.
Feature 0015 is complete after independent review found no remediation issue.
| ID | Deliverable | Owner | State | Evidence or next step |
| --- | --- | --- | --- | --- |
@@ -30,7 +33,9 @@ Current focus: Independently review the Feature 0014 implementation.
| F0011 | Deliver pipeline credentials and Notarius reuse | ansible-role-narratio and ansible-role-dnd-pipeline | done | Implemented and independently reviewed through remediation commit `de1b191` |
| F0012 | Assemble the D&D collection | ansible-collection-dnd and ansible-role-dnd-pipeline | done | Pipeline `600eb21` assembled and validated through collection remediation commit `da67b34` |
| F0013 | Publish the first coordinated suite release | all Ansible repositories and workspace | done | Public [`rakestraw.dnd` 0.1.0 release](https://gitea.maximumdirect.net/eric/ansible-collection-dnd/releases/tag/v0.1.0); documentation remediation completed in pipeline `78afffe` and collection `5e695f3` |
| F0014 | Enroll existing pipeline operators | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | in_progress | Review pipeline `c13f726a0a846180a1aa95d110b8bbc801edaae5` and collection `c0b0ebd13e92199f3d94e138c6cc50d1ece5dadd`; both complete integration suites passed |
| F0014 | Enroll existing pipeline operators | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `c13f726` and collection import `c0b0ebd` passed their complete integration suites and independent review with no remediation findings |
| F0015 | Enable D&D S3 storage and publishing | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `51566afcf69083cdbb5b5718f2544c86b079001c`, collection fixture `cf3de016ee09edf91afc3bc1addfd926d4a172ad`, collection squash `77f67adbc0f700e88bc15554e02052eaa22f1033`, and collection merge `46913446765e86ecfb29ebad9955d579ef6a4c7e` passed the complete repository suites and independent review with no remediation findings |
| F0016 | Prepare the D&D deployment playbook for `pipeline-dnd` | rakestraw-ansible and workspace | in_progress | Consumer preparation is in `716a80e`, `12b2462`, and `7f8353d`; public dependency installation, target-only host listing, syntax checking, and whitespace validation passed; independent review is next |
| A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` |
| R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review |