Resume D&D storage and publishing plan

docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md

@@ -11,7 +11,7 @@ object-storage credentials delivered through its existing secrets directory,
 publish the useful transcript and D&D artifact outputs, and retain maximum S3
 audio caching.
 
-The standalone Narratio role already exposes Narratio v2.1.0's complete
+The standalone Narratio role already exposes Narratio v2.1.1's complete
 configuration surface for this behavior. This feature supplies D&D suite
 policy through that existing interface; it does not add another storage
 implementation or credential mechanism.
@@ -25,6 +25,13 @@ publishing, and end-to-end external execution as deferred. The resulting suite
 can validate configuration and process local state, but it does not select the
 remote archive or publish the outputs needed for an operational deployment.
 
+Narratio v2.1.0 also rejected a valid publish list when a rule explicitly set
+`required` while omitting a derivable `dest`. Narratio v2.1.1 corrects that
+resolution bookkeeping defect without changing the publish schema or
+destination rules. This feature therefore advances the pipeline application
+pin to v2.1.1 and retains explicit suite-owned requiredness together with
+native destination derivation.
+
 The prior test-machine configuration records the intended Backblaze B2
 endpoint, bucket, prefix, credential names, publication set, and post-publish
 workspace cleanup. Narratio's current contract has since replaced the legacy
@@ -53,6 +60,8 @@ Notarius, so S3 needs only to join the same path.
 
 ## Scope
 
+- Advance the suite's Narratio application version from `v2.1.0` to the
+  backward-compatible `v2.1.1` bugfix release.
 - Add suite-owned pipeline variables for the selected Narratio workspace,
   S3, cache, and publish settings.
 - Select the S3 backend with these D&D deployment defaults:
@@ -137,6 +146,9 @@ Notarius, so S3 needs only to join the same path.
 - Adding notification providers. Narratio v2.1 supports only its existing
   `noop` notification mode.
 - Releasing new role or collection versions as part of this implementation.
+- Modifying Narratio, ConfigKit, or the standalone Narratio role. Narratio
+  v2.1.1 is an already published prerequisite, and the existing generic role
+  can install the selected application tag through `narratio_version`.
 
 ## Target End State
 
@@ -144,6 +156,7 @@ The standalone pipeline role exposes these suite variables with the selected
 defaults:
 
 ```yaml
+dnd_pipeline_narratio_version: v2.1.1
 dnd_pipeline_narratio_workspace_cleanup_after_publish: true
 dnd_pipeline_narratio_storage_backend: s3
 dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets
@@ -223,14 +236,16 @@ remove, rename, or constrain this future composition path.
   defaults, dependency mappings, consumer documentation, and representative
   integration scenario.
 - `ansible-role-narratio` remains the generic owner of installation and native
-  pipeline rendering. Its reviewed v2.1.0 interface already covers every
+  pipeline rendering. Its reviewed `v0.1.0` role interface is compatible with
+  Narratio v2.1.1 and already covers every
   configuration field required by this feature and should remain unchanged.
 - `ansible-collection-dnd` owns the faithful pipeline subtree import, the
   unified consumer dependency manifest, and artifact-boundary validation.
 - `ansible-workspace-dnd` owns this roadmap, the later implementation plan,
   status, and independent review record.
-- `narratio` is a read-only source of storage, publication, credential,
+- `narratio` v2.1.1 is the released compatibility prerequisite and a read-only
-  caching, cleanup, restore, and CLI contracts.
+  source of storage, publication, credential, caching, cleanup, restore, and
+  CLI contracts during this implementation.
 - `rakestraw-ansible` remains unchanged until Feature 0016 consumes this
   interface on `pipeline-dnd`.
 
@@ -254,6 +269,10 @@ remove, rename, or constrain this future composition path.
 - Publish destinations remain omitted where Narratio can derive them. The
   configured source identity and artifact output path remain the single source
   of truth.
+- Narratio v2.1.1 is the minimum application version for this feature because
+  it preserves destination derivation when `required` is explicit. Do not work
+  around older behavior by duplicating destinations or omitting deliberate
+  requiredness.
 - Workspace cleanup is gated by Narratio's own successful-publication
   evidence. Ansible does not independently inspect remote state or remove
   runtime content.
@@ -273,11 +292,12 @@ remove, rename, or constrain this future composition path.
 
 ## Simplest Viable Implementation
 
-Add the selected variables to the pipeline defaults and argument specification,
+Advance `dnd_pipeline_narratio_version` to v2.1.1. Add the selected variables to
-then map them into the existing `rakestraw.narratio` dependency. Update the
+the pipeline defaults and argument specification, then map them into the
-pipeline README to document the policy, credential names, complete-list
+existing `rakestraw.narratio` dependency. Update the pipeline README to document
-replacement semantics, and native capabilities now available. Do not add a
+the policy, credential names, complete-list replacement semantics, and native
-template, task, handler, fact, probe, or storage-specific validation layer.
+capabilities now available. Do not add a template, task, handler, fact, probe,
+or storage-specific validation layer.
 
 In the existing pipeline integration driver, extend the synthetic secrets map
 with object-storage values and explicitly supply representative S3 and publish
@@ -313,6 +333,12 @@ visible locally, but it would duplicate paths already owned by Narratio's
 built-in sources and the configured artifact definitions. Native destination
 derivation keeps path coordination automatic.
 
+Omitting explicit `required: true` values would avoid the v2.1.0 failure only
+by relying on an unrelated defaulting path. Changing the generic role's YAML
+format would not repair the underlying sequence-resolution mismatch. Narratio
+v2.1.1 fixes the owning application boundary, allowing the suite to state its
+requiredness policy directly without duplicating destinations.
+
 Deleting spool audio after publication would minimize transient disk usage,
 but no such retention policy has been selected, and local-audio sessions may
 depend on that content. Leaving the field omitted preserves Narratio's default
@@ -330,6 +356,8 @@ operational workflow.
 - The pipeline role declares and documents every suite variable shown in the
   target state, with matching argument-spec entries and direct Narratio
   dependency mappings.
+- The pipeline installs Narratio v2.1.1, whose released configuration loader
+  supports explicit requiredness together with derived publish destinations.
 - The two credential-reference variables accept paired `null` overrides so a
   caller can deliberately select Narratio's ambient AWS SDK provider chain.
 - The default effective Narratio configuration selects the specified
@@ -346,8 +374,9 @@ operational workflow.
   credential value.
 - No new production task, template, handler, fact, state file, wrapper, or
   storage-validation command is introduced.
-- The standalone Narratio role remains unchanged because its v2.1.0 interface
+- The standalone Narratio role remains unchanged because its `v0.1.0` role
-  already renders the required native fields and preserves null omission.
+  interface can install Narratio v2.1.1, renders the required native fields,
+  and preserves null omission.
 - The existing pipeline integration scenario explicitly supplies
   representative S3 and publish inputs, resolves synthetic values for all four
   typed credentials, passes Narratio's native configuration validation, and
@@ -363,7 +392,9 @@ operational workflow.
   validation, and clean-repeat suite passes without a duplicate S3 behavior
   test.
 - No real object-storage request, paid model request, application repository
-  change, asset change, release tag, or published artifact is required.
+  change during the remaining implementation, asset change, new release tag,
+  or published artifact is required. The existing Narratio v2.1.1 release is a
+  prerequisite rather than an implementation deliverable.
 - Pipeline documentation links operators to Narratio's canonical procedures
   for S3 session audio, publish, restore, locks, status, and cleanup and no
   longer describes storage and publishing as deferred.
@@ -373,9 +404,10 @@ operational workflow.
 
 ## Dependencies and Follow-up Work
 
-Features 0009, 0011, 0012, 0013, and 0014 are prerequisites. In particular,
+Features 0009, 0011, 0012, 0013, and 0014 and the public Narratio v2.1.1 bugfix
-Feature 0011 supplies the credential directory and Feature 0014 supplies the
+release are prerequisites. In particular, Feature 0011 supplies the credential
-human-operator access needed for the eventual live deployment.
+directory and Feature 0014 supplies the human-operator access needed for the
+eventual live deployment.
 
 [Feature 0016](0016-deploy-dnd-pipeline-to-pipeline-dnd.md) follows this work.
 It will put the four actual values in encrypted host inventory, pin the exact

docs/roadmap/implementation.md

@@ -14,10 +14,12 @@ Active plan location: `docs/roadmap/implementation.md`
 Configure the authoritative D&D pipeline role to use the selected Backblaze
 S3-compatible archive, typed object-storage credential references, maximum S3
 audio caching, immutable run publication, selected transcript and D&D artifact
-outputs, and cleanup of successfully published workspace state. Validate the
+outputs, and cleanup of successfully published workspace state. Advance the
-configuration through Narratio's native interface, import the exact reviewed
+pipeline to Narratio v2.1.1 so explicit publish requiredness composes with
-pipeline commit into the collection from public Gitea, and leave the result
+native destination derivation. Validate the configuration through Narratio's
-ready for the real-host deployment in Feature 0016.
+native interface, import the exact reviewed pipeline commit into the collection
+from public Gitea, and leave the result ready for the real-host deployment in
+Feature 0016.
 
 The Feature 0015 roadmap remains authoritative for scope, rationale, target
 state, and acceptance criteria. This plan does not authorize a second storage
@@ -69,16 +71,22 @@ lane, or changes to the generic Narratio role.
 - Features 0009, 0011, 0012, 0013, and 0014 are complete.
 - The approved Feature 0015 roadmap and this implementation plan are committed
   in the workspace repository before the sprint begins.
-- `ansible-role-dnd-pipeline` starts clean on `main` at
+- `ansible-role-dnd-pipeline` remains based on
-  `c13f726a0a846180a1aa95d110b8bbc801edaae5`, equal to `origin/main`.
+  `c13f726a0a846180a1aa95d110b8bbc801edaae5`, equal to `origin/main`, with the
+  interrupted Stage 1 edits preserved and uncommitted in `README.md`,
+  `defaults/main.yml`, `meta/argument_specs.yml`, `meta/main.yml`, and
+  `tests/integration`. Review and complete those edits; do not discard them.
 - `ansible-collection-dnd` starts clean on `main` at
   `c0b0ebd13e92199f3d94e138c6cc50d1ece5dadd`, equal to `origin/main`.
 - `ansible-role-narratio` is clean at
   `292d8669e7aad5cfe80b0c469d22b6177e8cf0f1`, equal to `origin/main`, and its
-  `v0.1.0` role release contains the Narratio v2.1.0 field-specific interface
+  `v0.1.0` role release contains the field-specific interface required by this
-  required by this feature.
+  feature and can install Narratio v2.1.1 through `narratio_version`.
 - The sibling Narratio application repository is clean at public release
-  `v2.1.0`. It is a read-only contract source during this sprint.
+  `v2.1.1`, commit `442469b9f57e65ce17f09059609dd1a37cf8c3b6`. That
+  backward-compatible release corrects destination derivation with explicit
+  publish requiredness and is a read-only contract source during the resumed
+  sprint.
 - The workspace `.venv` supplies the tested `ansible-playbook`,
   `ansible-galaxy`, and `ansible-lint` commands used by the repository test
   drivers.
@@ -99,7 +107,7 @@ lane, or changes to the generic Narratio role.
   collection snapshot.
 - `ansible-workspace-dnd` records the implementation handoff after the two
   implementation repositories are complete.
-- `ansible-role-narratio`, all other standalone roles, application sources,
+- `ansible-role-narratio`, all standalone roles, application sources,
   asset repositories, and `rakestraw-ansible` may be inspected but must remain
   unchanged.
 
@@ -131,13 +139,17 @@ Before beginning this stage, read:
 
 ### Preconditions
 
-- The workspace planning commit containing this plan is present.
+- The workspace planning commit containing the original approved plan and this
-- `ansible-role-dnd-pipeline` is clean at the starting revision, is not behind
+  resumed-plan revision are present.
-  or diverged from `origin/main`, and contains no unrelated changes.
+- `ansible-role-dnd-pipeline` is based on the starting revision, is not behind
+  or diverged from `origin/main`, and its only working-tree changes are the
+  preserved interrupted Stage 1 edits identified in Starting State and
+  Preconditions.
 - The current `rakestraw.narratio` dependency mapping and existing integration
   driver still match the structures reviewed while preparing this plan.
-- Narratio v2.1.0 still accepts the roadmap-selected storage, publish, cleanup,
+- Narratio v2.1.1 remains publicly available and accepts the roadmap-selected
-  cache, and typed credential-reference fields.
+  storage, publish, cleanup, cache, and typed credential-reference fields,
+  including rules with explicit `required` and omitted derivable `dest`.
 
 ### Owning Repository
 
@@ -145,7 +157,11 @@ Before beginning this stage, read:
 
 ### Required Changes
 
-1. Add the following exact suite defaults to `defaults/main.yml` near the
+1. Advance the existing application default to
+   `dnd_pipeline_narratio_version: v2.1.1`. Do not change the standalone
+   Narratio role release pin: its existing public interface can install this
+   compatible application patch release.
+2. Add the following exact suite defaults to `defaults/main.yml` near the
    existing Narratio policy variables:
 
    ```yaml
@@ -181,7 +197,7 @@ Before beginning this stage, read:
    Keep every `dest` omitted. Narratio derives transcript destinations from
    built-in sources and artifact destinations from the already managed
    artifact output paths.
-2. Add matching public entries to `meta/argument_specs.yml`. Use `type: bool`
+3. Add matching public entries to `meta/argument_specs.yml`. Use `type: bool`
    for the cleanup, path-style, cache, enabled, and upload settings; `type: str`
    for the backend and S3 string fields; and `type: list` for the complete
    publish output list. The S3 child fields, including both credential
@@ -190,55 +206,56 @@ Before beginning this stage, read:
    pair to `null` when selecting the AWS SDK provider chain. The remaining
    concrete suite-policy fields follow the role's existing required/defaulted
    argument-spec convention.
-3. In the `rakestraw.narratio` dependency entry in `meta/main.yml`, map every
+4. In the `rakestraw.narratio` dependency entry in `meta/main.yml`, map every
    new suite variable directly to its same-purpose standalone variable:
    `narratio_workspace_cleanup_after_publish`, the eight
    `narratio_storage_*` fields, `narratio_cache_s3_audio`,
    `narratio_publish_enabled`, `narratio_publish_upload_run`, and
    `narratio_publish_outputs`.
-4. Do not map `narratio_spool_delete_audio_after_publish`. Do not map
+5. Do not map `narratio_spool_delete_audio_after_publish`. Do not map
    `narratio_notarius_chunk_cache`. Do not add a new import, fragment,
    template, task, handler, assertion, or configuration-validation command.
-5. Revise the README's variable reference and processing-policy sections to
+6. Revise the README's application-version reference, upstream links, variable
+   reference, and processing-policy sections for Narratio v2.1.1 and
    document the selected S3 destination, paired nullable credential
    references, complete-list publish semantics, native destination derivation,
    maximum S3 audio caching, successful-publication workspace cleanup, and
    deliberate omission of spool deletion.
-6. Extend the README's managed-credential example to show
+7. Extend the README's managed-credential example to show
    `OBJECT_STORAGE_KEY_ID` and `OBJECT_STORAGE_KEY` mapped from encrypted
    inventory beside the existing Audita and OpenRouter entries. Explain that
    values remain outside ordinary configuration and that setting both S3
    credential references to `null` selects Narratio's ambient AWS SDK provider
    chain.
-7. Replace the README's statement that storage and publishing are deferred
+8. Replace the README's statement that storage and publishing are deferred
    with concise operator-facing descriptions and public upstream links for S3
    session audio, publish, restore, publication locks, status, and cleanup.
    Explain that accepted Notarius lanes remain
    `narratio.extraction.<output_key>` analyze inputs independently of whether
    they have stable publish rules.
-8. Adapt the existing `tests/integration` baseline rather than adding a new
+9. Adapt the existing `tests/integration` baseline rather than adding a new
    scenario. Add two synthetic object-storage values to
    `dnd_pipeline_secrets`; explicitly unset `OBJECT_STORAGE_KEY_ID` and
    `OBJECT_STORAGE_KEY` alongside the existing provider variables when
    invoking Ansible; and explicitly pass representative non-default S3,
    cleanup, cache, and publish values through the new public variables.
-9. Use representative values that exercise the interface without freezing the
+10. Use representative values that exercise the interface without freezing the
    defaults: for example an `.invalid` HTTPS endpoint and fixture bucket,
    prefix, and region; intentional `false` values for cleanup, path style, and
    S3 audio caching; the two configured credential reference names; enabled
    run publication; and a short complete publish list containing one built-in
    transcript source and `narratio.artifact.session_recap` with omitted
    destinations.
-10. Extend the existing secret-safe effective-configuration inspection only
+11. Extend the existing secret-safe effective-configuration inspection only
     enough to demonstrate the explicitly supplied representative storage
     fields, typed credential names, falsey cache/cleanup choices, enabled
     publication, and Narratio-derived destinations. Continue to use
     `narratio config validate` as the native validity boundary. Do not inspect
     role task results, call S3, or reproduce Narratio's validation rules.
-11. Extend the current non-disclosure check to cover both synthetic
+12. Extend the current non-disclosure check to cover both synthetic
     object-storage values. Preserve the complete existing integration matrix,
     cleanup behavior, and immediate clean-repeat assertions.
-12. Run the complete repository validation. Commit the valid result with the
+13. Run the complete repository validation. Commit the valid result with the
     stated plain-English message, push `main` to public Gitea, and record the
     full pushed commit hash for Stage 3.
 
@@ -252,15 +269,17 @@ Run from the workspace root:
 
 Expected result: syntax checking and production-profile lint pass; all five
 applications remain valid; the explicitly supplied S3 and publish values
-resolve through Narratio's native configuration interface; only reference
+resolve through Narratio v2.1.1's native configuration interface; explicit
-names appear in effective output; no synthetic credential value appears in
+publish requiredness and omitted destinations produce the derived destinations;
-captured output; no external S3 or model request occurs; and the immediate
+only reference names appear in effective output; no synthetic credential value
-identical repeat reports `changed=0` and `failed=0`.
+appears in captured output; no external S3 or model request occurs; and the
+immediate identical repeat reports `changed=0` and `failed=0`.
 
 ### Completion Criteria
 
-- Defaults, argument specifications, dependency mappings, and documentation
+- The pipeline selects Narratio v2.1.1, and defaults, argument specifications,
-  implement exactly the Feature 0015 contract.
+  dependency mappings, and documentation implement exactly the Feature 0015
+  contract.
 - No production task, template, fragment, validation command, or second secret
   path was added.
 - The representative integration scenario exercises public inputs rather than
@@ -400,7 +419,8 @@ Before beginning this stage, read:
    dependency mappings, README, and detailed integration fixture.
 3. Do not modify `galaxy.yml`, `requirements.yml`, release documentation, or
    the published `v0.1.0` artifacts. Feature 0015 changes no standalone role or
-   native collection dependency and publishes no release.
+   native collection dependency and publishes no new release; the pipeline's
+   application-version variable carries the Narratio v2.1.1 selection.
 4. Run the complete collection integration suite against the built and
    installed artifact. Do not add collection-level assertions for the detailed
    S3 configuration; native validation and clean convergence remain the
@@ -520,8 +540,9 @@ Commit intent: `Record D&D storage and publishing implementation`
   unexpected branch divergence or untracked artifact.
 - Report the full pipeline commit, collection fixture commit, collection
   subtree merge and squash commits, and workspace handoff commit.
-- Confirm that `ansible-role-narratio`, all other standalone roles,
+- Confirm that `ansible-role-narratio`, all standalone roles, applications,
-  applications, assets, and `rakestraw-ansible` remained unchanged.
+  assets, and `rakestraw-ansible` remained unchanged during the resumed sprint;
+  Narratio v2.1.1 was an already released prerequisite.
 - Summarize any deviation from this plan, unresolved finding, or complexity
   added beyond direct variable mapping and identify the acceptance criterion
   that required it.
@@ -530,4 +551,12 @@ Commit intent: `Record D&D storage and publishing implementation`
 
 ## Progress Notes
 
-- Planning complete; implementation has not begun.
+- The original plan was approved and Stage 1 began against Narratio v2.1.0.
+- Native validation exposed Narratio's publish-output atomic-sequence
+  bookkeeping defect; the implementation agent correctly stopped with its
+  partial pipeline edits uncommitted.
+- Narratio v2.1.1 at `442469b9f57e65ce17f09059609dd1a37cf8c3b6`
+  corrected the defect and added focused regression coverage without changing
+  the public configuration contract.
+- Resume Stage 1 from the preserved pipeline edits, advance the application pin
+  to v2.1.1, rerun the complete validation, and then continue in stage order.

docs/status.md

@@ -4,7 +4,7 @@ Last updated: 2026-09-21
 
 Current milestone: Operational D&D pipeline deployment
 
-Current focus: Review the Feature 0015 staged implementation plan.
+Current focus: Resume Feature 0015 implementation against Narratio v2.1.1.
 
 | ID | Deliverable | Owner | State | Evidence or next step |
 | --- | --- | --- | --- | --- |
@@ -31,7 +31,7 @@ Current focus: Review the Feature 0015 staged implementation plan.
 | F0012 | Assemble the D&D collection | ansible-collection-dnd and ansible-role-dnd-pipeline | done | Pipeline `600eb21` assembled and validated through collection remediation commit `da67b34` |
 | F0013 | Publish the first coordinated suite release | all Ansible repositories and workspace | done | Public [`rakestraw.dnd` 0.1.0 release](https://gitea.maximumdirect.net/eric/ansible-collection-dnd/releases/tag/v0.1.0); documentation remediation completed in pipeline `78afffe` and collection `5e695f3` |
 | F0014 | Enroll existing pipeline operators | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `c13f726` and collection import `c0b0ebd` passed their complete integration suites and independent review with no remediation findings |
-| F0015 | Enable D&D S3 storage and publishing | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | ready | [Roadmap](roadmap/features/0015-dnd-s3-storage-and-publishing.md) approved; prepare the staged implementation plan |
+| F0015 | Enable D&D S3 storage and publishing | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | ready | [Roadmap](roadmap/features/0015-dnd-s3-storage-and-publishing.md) and staged plan updated for released [Narratio v2.1.1](https://gitea.maximumdirect.net/eric/narratio/releases/tag/v2.1.1); resume Stage 1 from preserved pipeline edits |
 | F0016 | Deploy the D&D pipeline to `pipeline-dnd` | rakestraw-ansible and workspace | proposed | [Roadmap](roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md) drafted; Feature 0015 is a prerequisite |
 | A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` |
 | R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review |