diff --git a/docs/roadmap/features/0017-notarius-testing-overlay.md b/docs/roadmap/features/0017-notarius-testing-overlay.md new file mode 100644 index 0000000..f44ee23 --- /dev/null +++ b/docs/roadmap/features/0017-notarius-testing-overlay.md @@ -0,0 +1,183 @@ +# Feature 0017: Notarius Testing Overlay + +Status: ready + +## Summary + +Selecting Narratio's `testing` profile also selects a separately managed +Notarius `testing` configuration overlay. Operators can develop D&D assets +with economical extraction settings without modifying production configuration +or invoking Notarius separately. + +## Problem or Opportunity + +The suite's existing testing overlay changes Scriptorium selections but leaves +Notarius on its production configuration. Narratio v2.2.0 now provides the +explicit `pipeline.notarius.config_profile` interface requested by this consumer. +Notarius v0.7.1 retains the corresponding `--config-profile` option and updates +PromptKit and its provider catalogs. This feature adopts v0.7.1 as its baseline. + +## Scope + +- Expose the optional native selector in the standalone Narratio role. +- Update standalone Notarius and suite version defaults to v0.7.1, with + matching current-version documentation and collection distribution. +- Deploy a Notarius ConfigKit testing overlay through the suite role. +- Explicitly select that overlay from Narratio's existing testing overlay. +- Select `gpt-terra-latest` for production and `gpt-luna-latest` for testing, + while retaining inventory overrides for both choices. +- Document and validate the composed configuration, then distribute the + updated suite through the collection and refresh the consumer dependency pin. + +## Non-goals + +- Further upstream application changes or a Narratio-side LLM-profile override. +- Automatically propagating profile names between applications. +- Disabling Notarius, caching, publishing, or other pipeline stages for testing. +- New credentials, cache layouts, other production tuning, or deployment automation. +- Live controller/target deployment or paid model calls as completion gates. + +## Target End State + +### Standalone Narratio Interface + +`narratio_notarius_config_profile` defaults to `null`. The role omits +`notarius.config_profile` when null and serializes explicitly supplied values, +including an empty string. Documentation and argument specifications expose +the field without duplicating or narrowing Narratio's selector semantics. + +### Explicit Suite Composition + +The suite leaves the base selector unset. Its existing Narratio +`profiles/testing.yml` includes: + +```yaml +notarius: + config_profile: testing +``` + +The suite deploys `/usr/local/etc/notarius/profiles/testing.yml`. That file +overlays the selected suite pipeline's `llm_profile`, using +`dnd_pipeline_narratio_notarius_pipeline_id` as the pipeline key and a new +`dnd_pipeline_notarius_testing_llm_profile` variable as the value. It does not +copy the full pipeline definition. ConfigKit performs the ordinary merge. + +The default suite pipeline currently inherits its LLM selection at pipeline +level. Caller-supplied binding-level selections retain Notarius's native +precedence; this feature does not rewrite arbitrary caller pipelines or promise +that a pipeline-level default overrides explicit binding choices. + +### Production and Testing LLM Defaults + +The suite's base `dnd-session` pipeline selects `llm_profile: gpt-terra-latest` +instead of `dnd-extraction`. Callers can continue overriding that choice through +the existing `dnd_pipeline_notarius_pipelines` mapping. The new +`dnd_pipeline_notarius_testing_llm_profile` defaults to `gpt-luna-latest`. + +Both IDs are supplied by the OpenRouter catalog `v2.20260922.1` selected by +Notarius v0.7.1. Use those native catalog profiles directly; no operator profile +asset or profile-source setting is needed for these defaults. Retain the +existing `OPENROUTER_API_KEY` credential reference. These are deliberately +moving `latest` selections, not promises of immutable underlying model versions. +This production-model policy belongs to the suite, not the standalone role. + +Notarius stays enabled and retains its configured caching, output contracts, +references, and production defaults outside this overlay. Existing Scriptorium +testing overrides remain intact. Credentials continue through the existing +typed reference and suite secrets interfaces; no credential values enter YAML. + +Configuration files and directories use the suite's existing owner, group, +and mode variables. The selector and overlay add no new runtime identity or +filesystem-permission policy. + +## Repository Ownership + +- `ansible-role-narratio`: generic optional selector, documentation, and tests. +- `ansible-role-notarius`: v0.7.1 installation default and corresponding + documentation; existing profile-source settings remain sufficient. +- `ansible-role-dnd-pipeline`: testing policy, overlay assets, and composition. +- `ansible-collection-dnd`: imported pipeline snapshot and compatible dependency + manifest. Its current Narratio-role `v0.1.0` pin must advance to include the + new public variable; changing only the sibling role is insufficient. +- `rakestraw-ansible`: matching dependency pins and concise operator guidance. +- Workspace: roadmap, implementation plan, progress, and review. +- Narratio and Notarius applications: read-only authoritative contracts. + The standalone Notarius role already exposes profile-source settings and + does not need D&D-specific overlay policy. + +## Constraints and Settled Decisions + +Follow the [MVP roadmap](../mvp.md), [Ansible house style](../../ansible.md), +and [testing house style](../../testing.md). Narratio v2.2.0 is already selected +by the suite; Notarius v0.7.1 is the selected target. Consult these tags, not +only newer application checkouts. + +### Notarius v0.7.1 Compatibility + +The [upstream release notes](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.7.1/docs/releases/v0.7.1.md) +describe unchanged CLI, configuration, artifact, and subprocess contracts. +The application-owned `dnd-extraction` fallback retains its settings and no +configuration migration is required. Independently of that upstream behavior, +the user has selected `gpt-terra-latest` as this suite's new production default. + +Profile selection must use the catalogs selected by v0.7.1 (PromptKit v0.10.0). +The removed `deepseek-3-2`, `deepseek-4-flash`, and `deepseek-4-pro` catalog IDs +must not be assumed available or silently replaced by similarly named profiles. +This is deployment guidance, not a request for an Ansible catalog validator. + +Existing checkpoints with the previous profile-source identity are not reused; +source-keyed chunk plans and existing output artifacts are unaffected. Document +this upstream upgrade behavior without deleting caches or forcing Narratio +extraction automatically. + +Each application owns its configuration. Narratio selects the downstream +overlay; Notarius resolves and validates it. Existing application behavior owns +extraction reuse when selections change and explicit-force handling for changes +behind a stable profile name. Do not add Ansible state tracking for either. + +## Simplest Viable Implementation + +Add one nullable role field and its conditional template output. Extend the +suite's existing directory/template management to deploy the Notarius overlay +and extend the existing Narratio testing template with the explicit selector. +Use native serialization and ordinary file/template modules. Select the two +existing catalog IDs rather than deploying duplicate profile definitions. No assembly facts, +preflight state probes, or generic overlay-management framework is required. + +## Rationale and Alternatives + +A downstream ConfigKit overlay can own model choices and later Notarius tuning +without expanding Narratio's configuration surface for each setting. Direct +LLM-profile forwarding would solve only the model-selection case. Automatic +profile propagation would couple unrelated profile names and is not selected. + +## Acceptance Criteria + +- The standalone role omits an unset selector and preserves explicitly supplied + representative values, including the empty-string base selector. +- Native configuration inspection shows Narratio testing selecting Notarius + testing, and Notarius testing selecting the requested LLM profile. Base + configuration retains its production selection. +- A representative inventory-supplied testing selection is supported. Tests + exercise behavior rather than asserting that today's default model stays fixed. +- Notarius's deployed-version validation accepts the composed pipeline and + selected profile. No paid inference is required. +- Existing integration coverage remains valid, and immediate repeated role + convergence reports no changes. Do not independently retest file-module + ownership, modes, or template-file existence. +- The built collection works with its declared public role dependencies; no + local sibling checkout is needed by consumers. Subtree imports follow the + documented public-Gitea workflow, and published tags remain immutable. + +## Dependencies and Follow-up Work + +The upstream selector shipped in Narratio v2.2.0; the downstream baseline is +now Notarius v0.7.1. The staged implementation plan is maintained separately +at [implementation.md](../implementation.md). +Future testing-specific Notarius settings belong in its overlay but are not +implicitly included in this feature. + +## Open Questions + +None. The user selected `gpt-luna-latest` for testing and `gpt-terra-latest` +for production; both are available in the catalog selected by Notarius v0.7.1. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index c8b2983..b3089ba 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,12 +1,292 @@ -# Implementation Plan +# Implementation Plan 0017: Notarius Testing Overlay -No implementation plan is currently active. Select and agree on the next unit -of work before replacing this notice with its staged plan. +Status: ready -Feature 0016 is complete. Its durable scope and completion evidence are in -[the feature roadmap](features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md) -and [workspace status](../status.md). Git history preserves the completed plan. +Target executor: GPT-5.6-Terra -Completion covers the committed deployment playbook, host configuration, -dependency manifest, and operator documentation—not deployment or runtime -acceptance on the operational host. +Feature roadmap: [0017-notarius-testing-overlay.md](features/0017-notarius-testing-overlay.md) + +## Objective + +Implement the approved roadmap: Narratio testing explicitly selects Notarius +testing, with `gpt-luna-latest` for testing and `gpt-terra-latest` for production. +Keep the standalone roles generic and distribute a coherent public dependency +set. The roadmap owns scope and rationale; this document owns execution order. + +## Execution Rules and Starting State + +- Execute stages in order. Before each stage reread its Required Policy Review, + inspect owning working trees, and preserve unrelated changes. Paths to policy + below are relative to the workspace, not the nested repository. +- Commit validated units with plain-English messages. Record resulting full + commit IDs in Progress Notes; never invent future hashes or move release tags. +- Use the existing `.venv/bin` Ansible tools. Missing PATH entries, a local + `/etc/ansible`, or controller access are not blockers. `ansible-doc` rendering + is not an acceptance gate. +- Repository integration scripts converge this development host and use sudo; + use this established local test environment, not the operational controller + or `pipeline-dnd`. Do not provision real credentials or run paid inference. +- Read the scripts before running them. Preserve existing fixture and cleanup + conventions; report an actual missing prerequisite rather than expanding scope. +- Trust Ansible modules. Add no runtime validation commands, state fingerprints, + cache deletion, force-extraction automation, or profile discovery framework. +- Local validation can use local role sources where the existing suite does; + distribution validation must use the declared public revisions. +- Publishing changed role commits to their existing Gitea main branches is + required before public dependency installation and subtree import. Publish + only completed, validated work; if publication is unavailable, preserve the + commits and report that specific distribution blocker. +- No new release/tag or replacement of the published 0.1.0 artifact is required. + Use full pushed commit pins for this development compatibility set. + +Current versions: Narratio v2.2.0; Notarius v0.7.0, to become v0.7.1. +The latter supplies PromptKit v0.10.0 and OpenRouter catalog v2.20260922.1, +which contains both selected profile IDs. Existing OPENROUTER_API_KEY delivery +is sufficient. The standalone role tests deliberately select older application +versions in some cases; do not update those merely to match a new default. + +Repository scope: change the Narratio and Notarius roles, pipeline role, +collection, consumer `rakestraw-ansible`, and workspace planning records. +Applications, other standalone roles, and asset repositories are read-only. + +## Stage 1: Expose Narratio's Native Profile Selector + +### Required Policy Review + +Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, +`docs/ansible.md`, `docs/testing.md`, and the Feature 0017 roadmap. + +### Owning Repository and Preconditions + +`ansible-role-narratio`. Its default already selects Narratio v2.2.0. +Read that tag's configuration contract for `notarius.config_profile`. + +### Required Changes + +1. Add `narratio_notarius_config_profile: null` to defaults and an optional + string entry to `meta/argument_specs.yml`; document the public variable. +2. Include the variable in the enclosing Notarius-section condition in + `templates/pipeline.yml.j2`. Render `config_profile` with `to_json` only + when the variable is not none. Preserve explicit empty strings and exact + values. Add no name validation or file discovery. +3. Extend the existing representative configuration tests with a supplied + nonempty selector, null omission, and intentional empty-string rendering. + Keep fixtures small; no downstream inference or overlay file is necessary + merely to validate Narratio's native selector field. Existing tests against + older application versions must leave the new field unset. + +### Validation and Completion + +From this repository run `tests/integration` and `git diff --check`. +The public variable renders correctly and native configuration validation and +clean repeat pass. Extend existing tests rather than creating a parallel harness. + +### Commit + +`Expose the Notarius configuration profile in the Narratio role`. +Push this validated commit to origin/main and record its full ID as NARRATIO_ROLE. + +## Stage 2: Adopt Notarius v0.7.1 + +### Required Policy Review + +Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, +`docs/ansible.md`, `docs/testing.md`, and the Feature 0017 roadmap. + +### Owning Repository and Preconditions + +`ansible-role-notarius`. Stage 1 is complete. Inspect v0.7.1 release notes and +the role's existing integration entry point. + +### Required Changes + +Set `notarius_version` to v0.7.1 and update the README's current default. +No role configuration or task change is needed for this dependency upgrade. +Retain explicitly selected historical-version tests. Do not copy new provider +catalogs or model policy into this generic role. + +### Validation and Completion + +Run `tests/integration` and `git diff --check` in this repository. The normal +default convergence uses v0.7.1, native checks pass, and repeat convergence is +clean. Preserve tests of caller version selection instead of asserting the default. + +### Commit + +`Install Notarius v0.7.1 by default`. +Push the validated commit and record its full ID as NOTARIUS_ROLE. + +## Stage 3: Compose and Validate the Suite Testing Overlay + +### Required Policy Review + +Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, +`docs/ansible.md`, `docs/testing.md`, and the Feature 0017 roadmap. + +### Owning Repository and Preconditions + +`ansible-role-dnd-pipeline`. Stages 1-2 are validated and publicly available. +Review `tests/integration`, especially its profile diff, inventory override, +and alternate pipeline composition cases. + +### Required Changes + +1. Pin the Narratio and Notarius roles in `requirements.yml` to NARRATIO_ROLE + and NOTARIUS_ROLE. Leave unrelated dependencies unchanged. +2. Set `dnd_pipeline_notarius_version: v0.7.1`. In the existing default + `dnd_pipeline_notarius_pipelines` mapping, change the base dnd-session + `llm_profile` to `gpt-terra-latest`. Do not introduce a second production + profile variable: the mapping remains its inventory override interface. +3. Add `dnd_pipeline_notarius_testing_llm_profile: gpt-luna-latest`, its string + argument specification, and README documentation. +4. Extend ordinary directory management to create + `/usr/local/etc/notarius/profiles`. Add one template for + `/usr/local/etc/notarius/profiles/testing.yml` containing only: + + ```yaml + pipelines: + : + llm_profile: + ``` + + Serialize both dynamic key and value safely. Use the existing + `dnd_pipeline_narratio_notarius_pipeline_id` for the key. Manage the file + and directory with existing runtime owner/group and configuration modes. + Do not hard-code dnd-session or copy the full pipeline into the overlay. +5. Add `notarius: {config_profile: testing}` to Narratio's existing testing + template without changing its Scriptorium settings. Leave the base Narratio + selector unset. No extra suite-level selector variable or PromptKit asset + is needed. ConfigKit finds the overlay normally; do not import it into base. +6. Document base/testing choices, mapping overrides, native binding precedence, + moving catalog IDs, unchanged credential reference, and v0.7.1 checkpoint + compatibility. Keep Notarius enabled and caching unchanged. +7. Extend the existing integration script to inspect base and testing effective + configurations through the applications' native commands. Use explicit + caller-supplied values for assertions; for example, explicitly select + `gpt-luna-latest` in the production fixture and `gpt-terra-latest` in the + testing fixture to demonstrate independence from defaults. Validate the + selected pipeline with Notarius `--config-profile testing`, without inference. + Exercise the existing alternate pipeline ID case with the testing overlay + to prove the key follows inventory. Do not add a binding-precedence matrix. +8. Replace the old test restriction that Narratio testing differences may only + end in `.profile_id`: it excludes the new `.notarius.config_profile` setting. + Prefer assertions of the selected effective fields over an exhaustive diff + allowlist. Retain useful Scriptorium override coverage and clean-repeat checks. + +### Validation and Completion + +Run `tests/integration` and `git diff --check`. Required outcomes: native +validation passes for base and testing; selected inventory values are visible; +base remains independent of testing; alternate pipeline composition works; +immediate repeat is clean. No real keys or provider requests are required. + +### Commit + +`Add the Notarius testing overlay and production model defaults`. +Push the validated commit and record its full ID as PIPELINE_ROLE. + +## Stage 4: Assemble the Public Collection Snapshot + +### Required Policy Review + +Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, +`docs/ansible.md`, `docs/testing.md`, and the collection README's subtree policy. + +### Owning Repository and Preconditions + +`ansible-collection-dnd`. Stage 3 and its dependency commits are pushed. + +### Required Changes + +1. Import PIPELINE_ROLE from Gitea using the documented workflow: + + ```sh + git subtree pull --prefix=roles/pipeline \ + https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \ + FULL_PIPELINE_ROLE_COMMIT --squash \ + -m 'Bundle the Notarius testing overlay' + ``` + + Substitute the actual full commit ID. Do not edit the subtree directly. +2. Update collection-root `requirements.yml` to the same Narratio/Notarius + role commits used by the pipeline manifest. Keep unrelated pins unchanged. +3. Extend the collection's existing installed-artifact smoke coverage to validate + Notarius testing and inspect Narratio's effective downstream selector. Reuse + Stage 3's detailed behavior evidence rather than copying its whole test suite. +4. Keep version 0.1.0 and published tags/artifacts unchanged. Document the current + development snapshot where necessary without rewriting historical release claims. + +### Validation and Completion + +Run `tests/integration` and `git diff --check`. This must build/install the +collection with its public declared roles and pass native base/testing checks +and clean repeat. Compare `git rev-parse HEAD:roles/pipeline` with the source +pipeline commit's root tree: they must match. No sibling-role substitution. + +### Commit + +The subtree command creates its own commit. Commit remaining changes as +`Validate and pin the testing overlay collection dependencies`. +Push the completed collection and record its full final ID as COLLECTION. + +## Stage 5: Update Consumer Pins and Record Review Handoff + +### Required Policy Review + +Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, +`docs/ansible.md`, `docs/testing.md`, `docs/planning.md`, and Feature 0017. + +### Owning Repositories and Preconditions + +`rakestraw-ansible` and workspace. Stage 4 is validated and pushed. + +### Required Changes + +1. In the consumer manifest pin the collection to COLLECTION, and Narratio and + Notarius roles to their corresponding public commits. Preserve unrelated pins. +2. Update the collection refresh command in `docs/pipeline-dnd.md` to COLLECTION. + Explain the testing overlay behavior and v0.7.1 cache compatibility concisely. + Preserve the minimal hosts/roles playbook, host inventory, and secret references. + No host-variable copies of the new suite defaults are needed. +3. Update workspace status and this plan's Progress Notes with commit IDs and + validation evidence. Mark Feature 0017 in_progress pending independent review, + not done. Preserve the durable roadmap; no new feature scope is introduced. + +### Validation and Completion + +Run `git diff --check` in both repositories. Reuse Stage 4's successful public +dependency installation for the same dependency set. With those isolated role +and collection paths and workspace `.venv/bin` tools, run from the consumer: + +```text +ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --list-hosts +ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --syntax-check +``` + +Supply explicit ANSIBLE_ROLES_PATH and ANSIBLE_COLLECTIONS_PATH for the isolated +installation. Do not install to the controller's /etc/ansible or connect to the +target. If unrelated encrypted inventory prevents local checks, use a disposable +inventory with the target alias and non-secret settings and report that scope. +Missing real Vault values are not a feature blocker; do not populate them. + +### Commits + +Consumer: `Select the collection with Notarius testing profiles`. +Workspace: `Record the Notarius testing overlay implementation for review`. +Push the consumer update so the controller can retrieve the published pins. + +## Final Validation and Handoff + +Report all resulting commits, validation performed, and any real blockers or +deviations. Confirm changed working trees are clean. Distinguish repository +readiness from an operational deployment; none is required or claimed. Note +any complexity added beyond the planned declarative tasks and its requirement. + +## Progress Notes + +Implementation plan approved by the user; implementation has not started. + +## Open Questions + +None. Production and testing selections, upstream versions, public commit-pin +distribution, and ownership boundaries are settled for this plan. diff --git a/docs/status.md b/docs/status.md index cc8b447..a59db26 100644 --- a/docs/status.md +++ b/docs/status.md @@ -1,13 +1,15 @@ # Workspace Status -Last updated: 2026-09-21 +Last updated: 2026-09-22 Current milestone: Operational D&D pipeline deployment -Current focus: Select the next unit of work. Feature 0016 repository preparation -is complete and independently reviewed with no remediation findings. The -subsequent Narratio v2.1.2 update is published and pinned by the consumer. -Deployment and session execution remain separate operator work. +Current focus: Feature 0017 is approved and ready for implementation. Its finalized +roadmap and five-stage plan select `gpt-terra-latest` for production and +`gpt-luna-latest` for testing. The roadmap targets +Notarius v0.7.1; that role upgrade is not yet implemented. Narratio v2.2.0 is +published in the suite snapshot and consumer pin (`54889e6`, `1b13e86`). +Feature 0016 is complete; deployment remains separate operator work. | ID | Deliverable | Owner | State | Evidence or next step | | --- | --- | --- | --- | --- | @@ -36,6 +38,7 @@ Deployment and session execution remain separate operator work. | F0014 | Enroll existing pipeline operators | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `c13f726` and collection import `c0b0ebd` passed their complete integration suites and independent review with no remediation findings | | F0015 | Enable D&D S3 storage and publishing | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `51566afcf69083cdbb5b5718f2544c86b079001c`, collection fixture `cf3de016ee09edf91afc3bc1addfd926d4a172ad`, collection squash `77f67adbc0f700e88bc15554e02052eaa22f1033`, and collection merge `46913446765e86ecfb29ebad9955d579ef6a4c7e` passed the complete repository suites and independent review with no remediation findings | | F0016 | Prepare the D&D deployment playbook for `pipeline-dnd` | rakestraw-ansible and workspace | done | Consumer `716a80e`, `12b2462`, and `7f8353d` passed independent review with no findings; public dependency installation, target-only host listing, syntax and whitespace checks passed. Follow-up `7f4c90a` pins collection `2a53979` for Narratio v2.1.2; collection build and syntax checks passed | +| F0017 | Compose the Notarius testing overlay | Narratio role, pipeline role, collection, consumer, and workspace | ready | [Finalized roadmap](roadmap/features/0017-notarius-testing-overlay.md) and [five-stage implementation plan](roadmap/implementation.md); implementation not started | | A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` | | R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review |