From a95a0ea2801aac92be0823d4f550dea40735dc0f Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 21 Sep 2026 03:20:03 +0000 Subject: [PATCH] Resume D&D storage and publishing plan --- .../0015-dnd-s3-storage-and-publishing.md | 62 ++++++++--- docs/roadmap/implementation.md | 103 +++++++++++------- docs/status.md | 4 +- 3 files changed, 115 insertions(+), 54 deletions(-) diff --git a/docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md b/docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md index 10cb18e..a349764 100644 --- a/docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md +++ b/docs/roadmap/features/0015-dnd-s3-storage-and-publishing.md @@ -11,7 +11,7 @@ object-storage credentials delivered through its existing secrets directory, publish the useful transcript and D&D artifact outputs, and retain maximum S3 audio caching. -The standalone Narratio role already exposes Narratio v2.1.0's complete +The standalone Narratio role already exposes Narratio v2.1.1's complete configuration surface for this behavior. This feature supplies D&D suite policy through that existing interface; it does not add another storage implementation or credential mechanism. @@ -25,6 +25,13 @@ publishing, and end-to-end external execution as deferred. The resulting suite can validate configuration and process local state, but it does not select the remote archive or publish the outputs needed for an operational deployment. +Narratio v2.1.0 also rejected a valid publish list when a rule explicitly set +`required` while omitting a derivable `dest`. Narratio v2.1.1 corrects that +resolution bookkeeping defect without changing the publish schema or +destination rules. This feature therefore advances the pipeline application +pin to v2.1.1 and retains explicit suite-owned requiredness together with +native destination derivation. + The prior test-machine configuration records the intended Backblaze B2 endpoint, bucket, prefix, credential names, publication set, and post-publish workspace cleanup. Narratio's current contract has since replaced the legacy @@ -53,6 +60,8 @@ Notarius, so S3 needs only to join the same path. ## Scope +- Advance the suite's Narratio application version from `v2.1.0` to the + backward-compatible `v2.1.1` bugfix release. - Add suite-owned pipeline variables for the selected Narratio workspace, S3, cache, and publish settings. - Select the S3 backend with these D&D deployment defaults: @@ -137,6 +146,9 @@ Notarius, so S3 needs only to join the same path. - Adding notification providers. Narratio v2.1 supports only its existing `noop` notification mode. - Releasing new role or collection versions as part of this implementation. +- Modifying Narratio, ConfigKit, or the standalone Narratio role. Narratio + v2.1.1 is an already published prerequisite, and the existing generic role + can install the selected application tag through `narratio_version`. ## Target End State @@ -144,6 +156,7 @@ The standalone pipeline role exposes these suite variables with the selected defaults: ```yaml +dnd_pipeline_narratio_version: v2.1.1 dnd_pipeline_narratio_workspace_cleanup_after_publish: true dnd_pipeline_narratio_storage_backend: s3 dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets @@ -223,14 +236,16 @@ remove, rename, or constrain this future composition path. defaults, dependency mappings, consumer documentation, and representative integration scenario. - `ansible-role-narratio` remains the generic owner of installation and native - pipeline rendering. Its reviewed v2.1.0 interface already covers every + pipeline rendering. Its reviewed `v0.1.0` role interface is compatible with + Narratio v2.1.1 and already covers every configuration field required by this feature and should remain unchanged. - `ansible-collection-dnd` owns the faithful pipeline subtree import, the unified consumer dependency manifest, and artifact-boundary validation. - `ansible-workspace-dnd` owns this roadmap, the later implementation plan, status, and independent review record. -- `narratio` is a read-only source of storage, publication, credential, - caching, cleanup, restore, and CLI contracts. +- `narratio` v2.1.1 is the released compatibility prerequisite and a read-only + source of storage, publication, credential, caching, cleanup, restore, and + CLI contracts during this implementation. - `rakestraw-ansible` remains unchanged until Feature 0016 consumes this interface on `pipeline-dnd`. @@ -254,6 +269,10 @@ remove, rename, or constrain this future composition path. - Publish destinations remain omitted where Narratio can derive them. The configured source identity and artifact output path remain the single source of truth. +- Narratio v2.1.1 is the minimum application version for this feature because + it preserves destination derivation when `required` is explicit. Do not work + around older behavior by duplicating destinations or omitting deliberate + requiredness. - Workspace cleanup is gated by Narratio's own successful-publication evidence. Ansible does not independently inspect remote state or remove runtime content. @@ -273,11 +292,12 @@ remove, rename, or constrain this future composition path. ## Simplest Viable Implementation -Add the selected variables to the pipeline defaults and argument specification, -then map them into the existing `rakestraw.narratio` dependency. Update the -pipeline README to document the policy, credential names, complete-list -replacement semantics, and native capabilities now available. Do not add a -template, task, handler, fact, probe, or storage-specific validation layer. +Advance `dnd_pipeline_narratio_version` to v2.1.1. Add the selected variables to +the pipeline defaults and argument specification, then map them into the +existing `rakestraw.narratio` dependency. Update the pipeline README to document +the policy, credential names, complete-list replacement semantics, and native +capabilities now available. Do not add a template, task, handler, fact, probe, +or storage-specific validation layer. In the existing pipeline integration driver, extend the synthetic secrets map with object-storage values and explicitly supply representative S3 and publish @@ -313,6 +333,12 @@ visible locally, but it would duplicate paths already owned by Narratio's built-in sources and the configured artifact definitions. Native destination derivation keeps path coordination automatic. +Omitting explicit `required: true` values would avoid the v2.1.0 failure only +by relying on an unrelated defaulting path. Changing the generic role's YAML +format would not repair the underlying sequence-resolution mismatch. Narratio +v2.1.1 fixes the owning application boundary, allowing the suite to state its +requiredness policy directly without duplicating destinations. + Deleting spool audio after publication would minimize transient disk usage, but no such retention policy has been selected, and local-audio sessions may depend on that content. Leaving the field omitted preserves Narratio's default @@ -330,6 +356,8 @@ operational workflow. - The pipeline role declares and documents every suite variable shown in the target state, with matching argument-spec entries and direct Narratio dependency mappings. +- The pipeline installs Narratio v2.1.1, whose released configuration loader + supports explicit requiredness together with derived publish destinations. - The two credential-reference variables accept paired `null` overrides so a caller can deliberately select Narratio's ambient AWS SDK provider chain. - The default effective Narratio configuration selects the specified @@ -346,8 +374,9 @@ operational workflow. credential value. - No new production task, template, handler, fact, state file, wrapper, or storage-validation command is introduced. -- The standalone Narratio role remains unchanged because its v2.1.0 interface - already renders the required native fields and preserves null omission. +- The standalone Narratio role remains unchanged because its `v0.1.0` role + interface can install Narratio v2.1.1, renders the required native fields, + and preserves null omission. - The existing pipeline integration scenario explicitly supplies representative S3 and publish inputs, resolves synthetic values for all four typed credentials, passes Narratio's native configuration validation, and @@ -363,7 +392,9 @@ operational workflow. validation, and clean-repeat suite passes without a duplicate S3 behavior test. - No real object-storage request, paid model request, application repository - change, asset change, release tag, or published artifact is required. + change during the remaining implementation, asset change, new release tag, + or published artifact is required. The existing Narratio v2.1.1 release is a + prerequisite rather than an implementation deliverable. - Pipeline documentation links operators to Narratio's canonical procedures for S3 session audio, publish, restore, locks, status, and cleanup and no longer describes storage and publishing as deferred. @@ -373,9 +404,10 @@ operational workflow. ## Dependencies and Follow-up Work -Features 0009, 0011, 0012, 0013, and 0014 are prerequisites. In particular, -Feature 0011 supplies the credential directory and Feature 0014 supplies the -human-operator access needed for the eventual live deployment. +Features 0009, 0011, 0012, 0013, and 0014 and the public Narratio v2.1.1 bugfix +release are prerequisites. In particular, Feature 0011 supplies the credential +directory and Feature 0014 supplies the human-operator access needed for the +eventual live deployment. [Feature 0016](0016-deploy-dnd-pipeline-to-pipeline-dnd.md) follows this work. It will put the four actual values in encrypted host inventory, pin the exact diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index efe5c9d..4d17042 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -14,10 +14,12 @@ Active plan location: `docs/roadmap/implementation.md` Configure the authoritative D&D pipeline role to use the selected Backblaze S3-compatible archive, typed object-storage credential references, maximum S3 audio caching, immutable run publication, selected transcript and D&D artifact -outputs, and cleanup of successfully published workspace state. Validate the -configuration through Narratio's native interface, import the exact reviewed -pipeline commit into the collection from public Gitea, and leave the result -ready for the real-host deployment in Feature 0016. +outputs, and cleanup of successfully published workspace state. Advance the +pipeline to Narratio v2.1.1 so explicit publish requiredness composes with +native destination derivation. Validate the configuration through Narratio's +native interface, import the exact reviewed pipeline commit into the collection +from public Gitea, and leave the result ready for the real-host deployment in +Feature 0016. The Feature 0015 roadmap remains authoritative for scope, rationale, target state, and acceptance criteria. This plan does not authorize a second storage @@ -69,16 +71,22 @@ lane, or changes to the generic Narratio role. - Features 0009, 0011, 0012, 0013, and 0014 are complete. - The approved Feature 0015 roadmap and this implementation plan are committed in the workspace repository before the sprint begins. -- `ansible-role-dnd-pipeline` starts clean on `main` at - `c13f726a0a846180a1aa95d110b8bbc801edaae5`, equal to `origin/main`. +- `ansible-role-dnd-pipeline` remains based on + `c13f726a0a846180a1aa95d110b8bbc801edaae5`, equal to `origin/main`, with the + interrupted Stage 1 edits preserved and uncommitted in `README.md`, + `defaults/main.yml`, `meta/argument_specs.yml`, `meta/main.yml`, and + `tests/integration`. Review and complete those edits; do not discard them. - `ansible-collection-dnd` starts clean on `main` at `c0b0ebd13e92199f3d94e138c6cc50d1ece5dadd`, equal to `origin/main`. - `ansible-role-narratio` is clean at `292d8669e7aad5cfe80b0c469d22b6177e8cf0f1`, equal to `origin/main`, and its - `v0.1.0` role release contains the Narratio v2.1.0 field-specific interface - required by this feature. + `v0.1.0` role release contains the field-specific interface required by this + feature and can install Narratio v2.1.1 through `narratio_version`. - The sibling Narratio application repository is clean at public release - `v2.1.0`. It is a read-only contract source during this sprint. + `v2.1.1`, commit `442469b9f57e65ce17f09059609dd1a37cf8c3b6`. That + backward-compatible release corrects destination derivation with explicit + publish requiredness and is a read-only contract source during the resumed + sprint. - The workspace `.venv` supplies the tested `ansible-playbook`, `ansible-galaxy`, and `ansible-lint` commands used by the repository test drivers. @@ -99,7 +107,7 @@ lane, or changes to the generic Narratio role. collection snapshot. - `ansible-workspace-dnd` records the implementation handoff after the two implementation repositories are complete. -- `ansible-role-narratio`, all other standalone roles, application sources, +- `ansible-role-narratio`, all standalone roles, application sources, asset repositories, and `rakestraw-ansible` may be inspected but must remain unchanged. @@ -131,13 +139,17 @@ Before beginning this stage, read: ### Preconditions -- The workspace planning commit containing this plan is present. -- `ansible-role-dnd-pipeline` is clean at the starting revision, is not behind - or diverged from `origin/main`, and contains no unrelated changes. +- The workspace planning commit containing the original approved plan and this + resumed-plan revision are present. +- `ansible-role-dnd-pipeline` is based on the starting revision, is not behind + or diverged from `origin/main`, and its only working-tree changes are the + preserved interrupted Stage 1 edits identified in Starting State and + Preconditions. - The current `rakestraw.narratio` dependency mapping and existing integration driver still match the structures reviewed while preparing this plan. -- Narratio v2.1.0 still accepts the roadmap-selected storage, publish, cleanup, - cache, and typed credential-reference fields. +- Narratio v2.1.1 remains publicly available and accepts the roadmap-selected + storage, publish, cleanup, cache, and typed credential-reference fields, + including rules with explicit `required` and omitted derivable `dest`. ### Owning Repository @@ -145,7 +157,11 @@ Before beginning this stage, read: ### Required Changes -1. Add the following exact suite defaults to `defaults/main.yml` near the +1. Advance the existing application default to + `dnd_pipeline_narratio_version: v2.1.1`. Do not change the standalone + Narratio role release pin: its existing public interface can install this + compatible application patch release. +2. Add the following exact suite defaults to `defaults/main.yml` near the existing Narratio policy variables: ```yaml @@ -181,7 +197,7 @@ Before beginning this stage, read: Keep every `dest` omitted. Narratio derives transcript destinations from built-in sources and artifact destinations from the already managed artifact output paths. -2. Add matching public entries to `meta/argument_specs.yml`. Use `type: bool` +3. Add matching public entries to `meta/argument_specs.yml`. Use `type: bool` for the cleanup, path-style, cache, enabled, and upload settings; `type: str` for the backend and S3 string fields; and `type: list` for the complete publish output list. The S3 child fields, including both credential @@ -190,55 +206,56 @@ Before beginning this stage, read: pair to `null` when selecting the AWS SDK provider chain. The remaining concrete suite-policy fields follow the role's existing required/defaulted argument-spec convention. -3. In the `rakestraw.narratio` dependency entry in `meta/main.yml`, map every +4. In the `rakestraw.narratio` dependency entry in `meta/main.yml`, map every new suite variable directly to its same-purpose standalone variable: `narratio_workspace_cleanup_after_publish`, the eight `narratio_storage_*` fields, `narratio_cache_s3_audio`, `narratio_publish_enabled`, `narratio_publish_upload_run`, and `narratio_publish_outputs`. -4. Do not map `narratio_spool_delete_audio_after_publish`. Do not map +5. Do not map `narratio_spool_delete_audio_after_publish`. Do not map `narratio_notarius_chunk_cache`. Do not add a new import, fragment, template, task, handler, assertion, or configuration-validation command. -5. Revise the README's variable reference and processing-policy sections to +6. Revise the README's application-version reference, upstream links, variable + reference, and processing-policy sections for Narratio v2.1.1 and document the selected S3 destination, paired nullable credential references, complete-list publish semantics, native destination derivation, maximum S3 audio caching, successful-publication workspace cleanup, and deliberate omission of spool deletion. -6. Extend the README's managed-credential example to show +7. Extend the README's managed-credential example to show `OBJECT_STORAGE_KEY_ID` and `OBJECT_STORAGE_KEY` mapped from encrypted inventory beside the existing Audita and OpenRouter entries. Explain that values remain outside ordinary configuration and that setting both S3 credential references to `null` selects Narratio's ambient AWS SDK provider chain. -7. Replace the README's statement that storage and publishing are deferred +8. Replace the README's statement that storage and publishing are deferred with concise operator-facing descriptions and public upstream links for S3 session audio, publish, restore, publication locks, status, and cleanup. Explain that accepted Notarius lanes remain `narratio.extraction.` analyze inputs independently of whether they have stable publish rules. -8. Adapt the existing `tests/integration` baseline rather than adding a new +9. Adapt the existing `tests/integration` baseline rather than adding a new scenario. Add two synthetic object-storage values to `dnd_pipeline_secrets`; explicitly unset `OBJECT_STORAGE_KEY_ID` and `OBJECT_STORAGE_KEY` alongside the existing provider variables when invoking Ansible; and explicitly pass representative non-default S3, cleanup, cache, and publish values through the new public variables. -9. Use representative values that exercise the interface without freezing the +10. Use representative values that exercise the interface without freezing the defaults: for example an `.invalid` HTTPS endpoint and fixture bucket, prefix, and region; intentional `false` values for cleanup, path style, and S3 audio caching; the two configured credential reference names; enabled run publication; and a short complete publish list containing one built-in transcript source and `narratio.artifact.session_recap` with omitted destinations. -10. Extend the existing secret-safe effective-configuration inspection only +11. Extend the existing secret-safe effective-configuration inspection only enough to demonstrate the explicitly supplied representative storage fields, typed credential names, falsey cache/cleanup choices, enabled publication, and Narratio-derived destinations. Continue to use `narratio config validate` as the native validity boundary. Do not inspect role task results, call S3, or reproduce Narratio's validation rules. -11. Extend the current non-disclosure check to cover both synthetic +12. Extend the current non-disclosure check to cover both synthetic object-storage values. Preserve the complete existing integration matrix, cleanup behavior, and immediate clean-repeat assertions. -12. Run the complete repository validation. Commit the valid result with the +13. Run the complete repository validation. Commit the valid result with the stated plain-English message, push `main` to public Gitea, and record the full pushed commit hash for Stage 3. @@ -252,15 +269,17 @@ Run from the workspace root: Expected result: syntax checking and production-profile lint pass; all five applications remain valid; the explicitly supplied S3 and publish values -resolve through Narratio's native configuration interface; only reference -names appear in effective output; no synthetic credential value appears in -captured output; no external S3 or model request occurs; and the immediate -identical repeat reports `changed=0` and `failed=0`. +resolve through Narratio v2.1.1's native configuration interface; explicit +publish requiredness and omitted destinations produce the derived destinations; +only reference names appear in effective output; no synthetic credential value +appears in captured output; no external S3 or model request occurs; and the +immediate identical repeat reports `changed=0` and `failed=0`. ### Completion Criteria -- Defaults, argument specifications, dependency mappings, and documentation - implement exactly the Feature 0015 contract. +- The pipeline selects Narratio v2.1.1, and defaults, argument specifications, + dependency mappings, and documentation implement exactly the Feature 0015 + contract. - No production task, template, fragment, validation command, or second secret path was added. - The representative integration scenario exercises public inputs rather than @@ -400,7 +419,8 @@ Before beginning this stage, read: dependency mappings, README, and detailed integration fixture. 3. Do not modify `galaxy.yml`, `requirements.yml`, release documentation, or the published `v0.1.0` artifacts. Feature 0015 changes no standalone role or - native collection dependency and publishes no release. + native collection dependency and publishes no new release; the pipeline's + application-version variable carries the Narratio v2.1.1 selection. 4. Run the complete collection integration suite against the built and installed artifact. Do not add collection-level assertions for the detailed S3 configuration; native validation and clean convergence remain the @@ -520,8 +540,9 @@ Commit intent: `Record D&D storage and publishing implementation` unexpected branch divergence or untracked artifact. - Report the full pipeline commit, collection fixture commit, collection subtree merge and squash commits, and workspace handoff commit. -- Confirm that `ansible-role-narratio`, all other standalone roles, - applications, assets, and `rakestraw-ansible` remained unchanged. +- Confirm that `ansible-role-narratio`, all standalone roles, applications, + assets, and `rakestraw-ansible` remained unchanged during the resumed sprint; + Narratio v2.1.1 was an already released prerequisite. - Summarize any deviation from this plan, unresolved finding, or complexity added beyond direct variable mapping and identify the acceptance criterion that required it. @@ -530,4 +551,12 @@ Commit intent: `Record D&D storage and publishing implementation` ## Progress Notes -- Planning complete; implementation has not begun. +- The original plan was approved and Stage 1 began against Narratio v2.1.0. +- Native validation exposed Narratio's publish-output atomic-sequence + bookkeeping defect; the implementation agent correctly stopped with its + partial pipeline edits uncommitted. +- Narratio v2.1.1 at `442469b9f57e65ce17f09059609dd1a37cf8c3b6` + corrected the defect and added focused regression coverage without changing + the public configuration contract. +- Resume Stage 1 from the preserved pipeline edits, advance the application pin + to v2.1.1, rerun the complete validation, and then continue in stage order. diff --git a/docs/status.md b/docs/status.md index e68c2ca..1c1f92b 100644 --- a/docs/status.md +++ b/docs/status.md @@ -4,7 +4,7 @@ Last updated: 2026-09-21 Current milestone: Operational D&D pipeline deployment -Current focus: Review the Feature 0015 staged implementation plan. +Current focus: Resume Feature 0015 implementation against Narratio v2.1.1. | ID | Deliverable | Owner | State | Evidence or next step | | --- | --- | --- | --- | --- | @@ -31,7 +31,7 @@ Current focus: Review the Feature 0015 staged implementation plan. | F0012 | Assemble the D&D collection | ansible-collection-dnd and ansible-role-dnd-pipeline | done | Pipeline `600eb21` assembled and validated through collection remediation commit `da67b34` | | F0013 | Publish the first coordinated suite release | all Ansible repositories and workspace | done | Public [`rakestraw.dnd` 0.1.0 release](https://gitea.maximumdirect.net/eric/ansible-collection-dnd/releases/tag/v0.1.0); documentation remediation completed in pipeline `78afffe` and collection `5e695f3` | | F0014 | Enroll existing pipeline operators | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `c13f726` and collection import `c0b0ebd` passed their complete integration suites and independent review with no remediation findings | -| F0015 | Enable D&D S3 storage and publishing | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | ready | [Roadmap](roadmap/features/0015-dnd-s3-storage-and-publishing.md) approved; prepare the staged implementation plan | +| F0015 | Enable D&D S3 storage and publishing | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | ready | [Roadmap](roadmap/features/0015-dnd-s3-storage-and-publishing.md) and staged plan updated for released [Narratio v2.1.1](https://gitea.maximumdirect.net/eric/narratio/releases/tag/v2.1.1); resume Stage 1 from preserved pipeline edits | | F0016 | Deploy the D&D pipeline to `pipeline-dnd` | rakestraw-ansible and workspace | proposed | [Roadmap](roadmap/features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md) drafted; Feature 0015 is a prerequisite | | A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` | | R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review |