ansible-workspace-dnd/docs/roadmap/implementation.md

# Implementation Plan 0022: Coordinated Suite Upgrade

Status: in_progress

Target executor: GPT-5.6-Terra

Feature roadmap: [Feature 0022](features/0022-coordinated-suite-upgrade.md)

## Objective

Integrate the reviewed application upgrades into the pipeline role, assembled
collection, and consumer repository. The approved roadmap owns scope, target
state, and policy. No operational deployment or tagged release is included.

## Execution Rules and Starting State

- Execute Stage 1 through Stage 6 in order. Before each stage, reread its
  listed policies, check applicable repository instructions, and confirm its
  owning repository's working-tree state. Preserve unrelated changes.
- Commit valid completed stages with the stated plain English intent. Do not
  advance past failed validation. Distinguish actual blockers from unavailable
  optional tools or operational credentials that this plan does not require.
- Use workspace `.venv/bin` Ansible tooling. Missing PATH entries are not a
  missing controller. Existing integration scripts choose local tooling and
  install dependencies into temporary paths.
- Pipeline/collection integration runs use local sudo and mutate the local
  development installation, including source/binaries, configuration, fixture
  secrets, and test users. Read harness cleanup and verify these locations are
  disposable first. Do not run operational playbooks or contact live inference,
  WhisperX, or S3 services. Public Git/dependency downloads are expected.
- Use ordinary declarative modules; no production validation, preliminary
  state probes, compatibility aliases, manifest conversion, pruning, or new
  credential-discovery mechanism. New tasks are limited to Audita profile
  directory creation and looped profile templating.
- Publish only the required reviewed role revisions and feature commits to
  existing origins as specified below. Check remote URLs and outgoing commits
  before each push. Never force-push or move tags. If unrelated commits would
  be published or credentials prevent publication, report that boundary; do
  not substitute local paths in consumer manifests or subtree imports.
- Do not edit standalone application-role implementations, upstream apps,
  campaign/prompt assets, unrelated consumer inventory, Vault content, or
  vendored consumer roles. Required code owners are pipeline, collection, and
  consumer; workspace owns progress. Keep Seriatim and native collection pins.
- The new retry/repair/deadline pass-throughs default to null. Preserve the
  existing Audita 900-second per-attempt profile policy. Do not copy native
  defaults into suite variables.

## Stage 1: Make Reviewed Standalone Dependencies Public

Completion: complete (2026-09-29).

### Required Policy Review

Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, and
`docs/planning.md` before starting.

### Owning Repositories and Preconditions

The four standalone role repositories and workspace. Features 0018–0021 are
complete. Confirm these exact revisions and their existing Gitea origins:

| Role | Revision |
| --- | --- |
| Scriptorium | `b174c8c2990bb79f8149d8f288c5e06ccb42bd73` |
| Audita | `c3dd59f5ac9b942306ffe540ae9e71d493998137` |
| Notarius | `6aef48b7e12e3651971444b8c224d96ff1879e6c` |
| Narratio | `3c0700b5323293d44e3b1e67a50074d82d7a3bff` |

### Work and Validation

Mark F0022 `in_progress`. For each repository, fetch origin, inspect its status
and outgoing history, and publish the reviewed revision to `origin/main` if
not already reachable there. Use a normal non-force push of the exact revision
to `refs/heads/main` where a fast-forward is appropriate. A remote already
containing it needs no push. Verify reachability after fetch; do not replace a
selected pin merely because a newer remote commit exists.

Commands, substituting the role and exact revision from the table:

```sh
git -C ROLE remote -v
git -C ROLE fetch origin
git -C ROLE log --oneline origin/main..REVISION
git -C ROLE push origin REVISION:refs/heads/main
git -C ROLE fetch origin
git -C ROLE merge-base --is-ancestor REVISION origin/main
```

### Completion and Commit

All four pinned revisions are publicly available. No standalone code change or
new role commit is needed. Record evidence and commit workspace progress as
`Record the coordinated suite upgrade in progress`.

## Stage 2: Migrate Suite Configuration and Existing Integration Cases

Completion: complete (2026-09-29).

### Required Policy Review

Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, and `docs/testing.md` before starting.

### Owning Repository and Preconditions

`ansible-role-dnd-pipeline`. Stage 1 is complete. Inspect current defaults,
specs, dependency vars, tasks, templates, and tests; read tagged Audita v2.0.1
migration/configuration and Narratio v3.0.0 contracts linked by the roadmap.

### Required Changes

1. Pin the four standalone roles in `requirements.yml` to Stage 1 revisions.
   Set application defaults to Scriptorium v0.15.0, Audita v2.0.1, Notarius
   v0.8.1, Narratio v3.0.0. Preserve Seriatim v1.5.0 and its role pin.
2. Add `dnd_pipeline_audita_backends` (dict), default `dnd` with endpoint
   `https://inference.ai.rakestrawhome.com/v1`, credential `AUDITA_LLM_API_KEY`,
   concurrency `4`; omit queue capacity. Add `dnd_pipeline_audita_profiles`
   (dict) with `dnd-proposal` and `dnd-validation` keys. Each value is a complete
   native mapping containing matching `id`, backend `dnd`, model
   `google/gemma-4-31b-it`, and `timeout_seconds: 900`.
3. Add string defaults `dnd_pipeline_audita_llm_proposal_profile: dnd-proposal`
   and `dnd_pipeline_audita_llm_validation_profile: dnd-validation`. Map these
   and backends into the standalone role. Set `audita_profiles_dir` to
   `/usr/local/etc/audita/profiles`.
4. Add a `file` directory task and one looped `template` task over
   `dnd_pipeline_audita_profiles | dict2items`. Write `profiles/<key>.yml` using
   an `audita-profile.yml.j2` template containing `item.value | to_nice_yaml`.
   Use runtime owner/group, suite configuration modes, and task-level become.
   Document that keys are filename-safe IDs matching each mapping's `id`.
   Do not merge/rewrite profile values or delete removed entries.
5. Remove old `dnd_pipeline_audita_llm_{proposal,validation}_{base_url,model,
   api_key_env,timeout,max_retries}`, `dnd_pipeline_audita_concurrency_*`, and
   diagnostics retention from defaults, specs, dependency wiring, and active
   examples. Preserve diagnostics work directory and unchanged domain tuning.
   Convert normalization max gap/ellipsis gap/max duration to numeric
   `3.0`/`3.0`/`240.0` with float specs. Leave debug/reporting at native defaults.
6. Add these optional nullable suite variables, forwarding to the corresponding
   standalone variable by removing `dnd_pipeline_`:
   - `dnd_pipeline_audita_llm_transient_retries`: int.
   - `dnd_pipeline_audita_llm_repair_attempts`: int.
   - `dnd_pipeline_audita_llm_operation_timeout`: str.
   - `dnd_pipeline_notarius_promptkit_transient_retries`: int.
   - `dnd_pipeline_scriptorium_transient_retries`: int.
   Keep nulls and explicit zero; do not supply a fallback native default.
7. Add list-of-string credential defaults
   `dnd_pipeline_narratio_audita_credentials: [AUDITA_LLM_API_KEY]` and
   `dnd_pipeline_narratio_notarius_credentials: [OPENROUTER_API_KEY]`; wire to
   Narratio's credential lists. Remove the old suite Notarius singular-key
   variable and obsolete standalone-role arguments. Preserve secrets delivery.
8. Change Narratio/Notarius directory creation and testing-file destinations
   from `profiles` to `overlays`. Change Narratio testing to
   `notarius.config_overlay: testing`. Leave old ignored files alone. Preserve
   Audita's LLM `profiles/`, artifact fragments, model selections, caches,
   paths, publication, ACLs, and operator behavior.
9. Migrate existing `tests/integration` selectors and parsed assertions:
   Narratio `--overlay`, Notarius `--config-overlay`, `config_overlay`, and
   credential lists replace retired fields. Direct Audita validation receives
   the harness's synthetic `AUDITA_LLM_API_KEY` explicitly in its environment;
   do not assume Narratio's bootstrap applies to that process. Add Scriptorium
   native config validation. Keep old meaningful scenarios, clean repeats,
   and fixture cleanup; do not test upstream forwarding internals.
10. Update README public variables and migration mapping in the same commit
    so the role's current interface is documented. Explain defaults versus
    inventory overrides and retain existing native configuration ownership.

### Validation

From workspace root:

```sh
bash -n ansible-role-dnd-pipeline/tests/integration
ansible-role-dnd-pipeline/tests/integration
git -C ansible-role-dnd-pipeline diff --check
```

The harness installs public pinned dependencies and includes syntax/lint.
Require native acceptance of base/testing configuration and existing meaningful
behavior with clean repeats, without live service calls. If an unrelated asset
contract incompatibility appears, report it rather than rewriting assets.

### Completion and Commit

Suite configuration, specs, docs, and existing tests are compatible together.
Commit as `Integrate the reviewed application upgrades into the suite`.

## Stage 3: Verify Inventory Overrides and Finish Suite Migration Guidance

Completion: complete (2026-09-29). Pipeline assembly revision: `ea8b09e190419586ede6df3815d1ea999a0f4391`.

### Required Policy Review

Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, and `docs/testing.md` before starting.

### Owning Repository and Preconditions

`ansible-role-dnd-pipeline`. Stage 2 passes. Extend its existing harness rather
than introducing a test framework or changing production convergence.

### Required Changes

1. Add one representative override case with a fixture backend endpoint,
   caller-composed native profile map, selected proposal/validation IDs, and
   an explicit custom credential-name list. Use synthetic values only; provide
   required names both through managed fixture secrets for Narratio and direct
   test environment for Audita. Use simple offline-valid profile values.
2. In that case supply zero for the four retry/repair integers and a valid
   explicit Audita operation timeout, such as `20m`. Inspect native effective
   settings/profile views to verify supplied values and selected models. Inspect
   Narratio configuration to verify the credential names. Do not assert current
   application default retry counts or model catalogs.
3. Explicitly null the five new pass-throughs in a baseline case and inspect
   rendered application YAML for omission (effective views may include native
   defaults). Require an immediate unchanged repeat for the override case.
   Retain existing pipeline-map composition, testing selection, shared-user,
   cache, storage, and artifact tests; do not add module metadata checks.
4. Document exact old-to-new inventory choices, shared backend concurrency,
   filename/profile ID contract, credential list ownership, and CLI changes.
   Explain the retained 900-second attempt setting and independent native
   operation deadline. Link upstream manifest migration; do not automate it.
   Explain ignored legacy testing files and normal named-selector invalidation.

### Validation

```sh
bash -n ansible-role-dnd-pipeline/tests/integration
ansible-role-dnd-pipeline/tests/integration
git -C ansible-role-dnd-pipeline diff --check
```

All native checks and immediate repeats pass. Review the final pipeline diff
against roadmap scope, including no live-provider test or new production probe.

### Completion and Commit

Commit as `Verify suite configuration overrides and document migration`.
Record the final pipeline commit as the assembly revision. Inspect outgoing
commits and publish that revision to the pipeline's existing `origin/main`
with a normal push; verify remote reachability before Stage 4.

## Stage 4: Assemble and Validate the Collection

Completion: complete (2026-09-29). Public collection revision: `3d77eb382bcb4c1742015d3c32e23630bb8c73bf`.

### Required Policy Review

Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, and `docs/testing.md` before starting.

### Owning Repository and Preconditions

`ansible-collection-dnd`. The validated Stage 3 pipeline commit is public.
Read the collection README's subtree and dependency workflow.

### Required Changes

1. Import that exact revision using the authoritative public Gitea origin:

   ```sh
   git -C ansible-collection-dnd subtree pull --prefix=roles/pipeline \
     https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \
     PIPELINE_REVISION --squash
   ```

   Substitute the Stage 3 hash; never a local sibling path. Subtree-generated
   commits are expected. Do not edit imported role files directly.
2. Match the standalone revisions in root `requirements.yml` to Stage 1.
   Keep collection metadata version 0.1.0 and published release instructions
   intact; describe the new development snapshot separately.
3. Migrate collection `tests/integration` CLI/field assertions and direct Audita
   synthetic environment handling as in Stage 2. Validate Audita profiles and
   Scriptorium configuration natively. Keep collection tests focused on the
   installed artifact and dependency boundary, base/testing selection, and
   clean repeat; do not duplicate Stage 3's complete override matrix.
4. Update compatibility guidance and reference the pipeline migration notes.
   A fresh public dependency install must work without sibling role repos.
   Existing local campaign/prompt fixture overrides in the test harness remain
   acceptable; production manifests must not acquire those local paths.

### Validation

```sh
bash -n ansible-collection-dnd/tests/integration
ansible-collection-dnd/tests/integration
git -C ansible-collection-dnd diff --check
```

Require successful build, disposable artifact install, syntax/lint, native
configuration checks, and immediate clean repeat using public role dependencies.

### Completion and Commit

Commit remaining changes as `Assemble the upgraded D&D suite` after the subtree
import commits. Inspect and publish feature commits to `origin/main`; record
and verify the final public collection hash for Stage 5. No tags or release
assets are created or replaced.

## Stage 5: Update the Consumer Without Deploying

Completion: complete (2026-09-29). Local consumer revision: `f694dad9426ee21138b175e44a3855d42be99d4b`.

### Required Policy Review

Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, and `docs/testing.md` before starting.

### Owning Repository and Preconditions

`rakestraw-ansible`. Stage 4's collection and all referenced dependencies are
public. Read applicable consumer instructions and `docs/pipeline-dnd.md`.

### Required Changes

1. Update only the relevant role pins in `requirements.yml` to Stage 1 and its
   D&D collection pin to Stage 4. Preserve unrelated requirements.
2. Keep `playbooks/pipeline/dnd.yml` minimal. Existing host-vars credential
   names remain valid; preserve encrypted Vault content and all unrelated
   values. Do not add copied suite defaults to inventory. If an obsolete
   feature-owned setting is present, replace only that setting and document it.
3. Update `docs/pipeline-dnd.md` to the coordinated versions, exact collection
   refresh hash, new testing invocation, and migration prerequisites. Preserve
   the explicit refresh guidance for an equal-version development collection.
   Remove misleading claims that the v3 upgrade needs no configuration or
   manifest migration. Distinguish future operator commands from checks run.

### Validation

Use the workspace tools and disposable role/collection paths. From consumer
root, set `ANSIBLE_ROLES_PATH` and `ANSIBLE_COLLECTIONS_PATH` to directories
under a new `mktemp -d` path, then run:

```sh
/home/eric/Workspace/ansible-dnd/.venv/bin/ansible-galaxy install -r requirements.yml
/home/eric/Workspace/ansible-dnd/.venv/bin/ansible-playbook -i inventory playbooks/pipeline/dnd.yml --list-hosts --limit pipeline-dnd
/home/eric/Workspace/ansible-dnd/.venv/bin/ansible-playbook -i inventory playbooks/pipeline/dnd.yml --syntax-check --limit pipeline-dnd
git diff --check
```

These commands must not connect to the target or execute role tasks. If loading
encrypted inventory requires unavailable Vault credentials, validate syntax
with a temporary inventory containing only `pipeline-dnd`, a synthetic WhisperX
URL, and synthetic required inputs instead. Inspect the real playbook target
and inventory entry read-only; report this limited check accurately. Do not
decrypt Vault, require real keys, add ansible-doc gates, or execute deployment.

### Completion and Commit

Public dependencies resolve and consumer syntax/target checks pass (or the
documented synthetic-inventory fallback is used). Commit as
`Prepare the consumer for the coordinated suite upgrade`. Leave this consumer
commit local unless separately asked to push; it is not a downstream dependency.

## Stage 6: Record Evidence and Hand Off for Independent Review

Completion: complete (2026-09-29).

### Required Policy Review

Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, and `docs/planning.md` before starting.

### Owning Repository and Preconditions

Workspace. Stages 1–5 are complete and committed. No new code or deployment
work belongs in this stage.

### Required Changes and Validation

Record final hashes, public availability, commands/results, environment limits,
and any plan deviations in the feature roadmap and concise progress notes.
Update `docs/status.md` and the upgrade sequence to implementation complete,
review pending, using `in_progress` until independent review. Review that pins
across pipeline, collection, and consumer agree, and no standalone role or
application/asset code changed outside scope.

```sh
git diff --check
scripts/status
```

Reuse earlier integration evidence for documentation-only changes. Report
which commits were pushed and which remain local; a clean working tree does
not imply a synchronized remote. Keep deployment, session migration, and formal
release work explicitly deferred.

### Completion and Commit

Commit as `Record coordinated suite upgrade validation`. All changed working
trees are clean, and implementation is ready for independent review. Do not
mark F0022 done, begin another feature, or push unrelated workspace history.

## Progress Notes

Plan approved. Stage 1 publication was initially blocked. On 2026-09-29, all four exact
reviewed revisions were present locally; clean worktrees and expected Gitea
origins were confirmed. After fetching, each `origin/main` was three reviewed
feature commits behind the selected revision and could fast-forward. The first
normal push, for Scriptorium, failed because HTTPS Git could not read a
username. At that time no role revision was published and no downstream stage
had started.

- Stage 1 resumed on 2026-09-29: authenticated pushes of the exact reviewed
  Scriptorium `b174c8c`, Audita `c3dd59f`, Notarius `6aef48b`, and Narratio
  `3c0700b` revisions to their existing Gitea `origin/main` branches succeeded.
  A fresh fetch and ancestor check verified all four public references. No
  standalone code or dependency pin changed in this stage.
- Stage 2: pipeline migration and existing integration cases passed `bash -n`,
  the full role harness, and `git diff --check`; role commit `8ea8621`.
- Stage 3: representative override, nullable omission, native effective views,
  and clean repeats passed the full role harness; public assembly revision
  `ea8b09e190419586ede6df3815d1ea999a0f4391`. A subsequent fixture-JSON
  line formatting change passed shell syntax, JSON parsing, and whitespace
  checks without another full harness run.
- Stage 4: the exact public pipeline revision was subtree-imported. The
  collection artifact harness passed with public dependencies, native base and
  testing validation, and a clean repeat; public collection revision
  `3d77eb382bcb4c1742015d3c32e23630bb8c73bf`.
- Stage 5: public consumer dependencies installed into disposable paths;
  real-inventory `--list-hosts` selected `pipeline-dnd`, and `--syntax-check`
  and `git diff --check` passed without Vault decryption or deployment. Consumer
  revision `f694dad9426ee21138b175e44a3855d42be99d4b` remains local.
- Stage 6: cross-repository pins and owning repository worktrees were checked;
  implementation is complete, independent review pending. Operational
  deployment, live processing, session migration, and tagged release remain
  deferred.

## Open Questions

None. The user approved application-owned retry/repair/deadline defaults with
nullable inventory overrides and the retained Audita per-attempt policy.