eric/ansible-workspace-dnd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan the Notarius testing overlay implementation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-22 18:23:20 +00:00
parent 58738e4efc
commit 38dfb0422c
3 changed files with 480 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

183
docs/roadmap/features/0017-notarius-testing-overlay.md Normal file
View File

@@ -0,0 +1,183 @@
# Feature 0017: Notarius Testing Overlay
Status: ready
## Summary
Selecting Narratio's `testing` profile also selects a separately managed
Notarius `testing` configuration overlay. Operators can develop D&D assets
with economical extraction settings without modifying production configuration
or invoking Notarius separately.
## Problem or Opportunity
The suite's existing testing overlay changes Scriptorium selections but leaves
Notarius on its production configuration. Narratio v2.2.0 now provides the
explicit `pipeline.notarius.config_profile` interface requested by this consumer.
Notarius v0.7.1 retains the corresponding `--config-profile` option and updates
PromptKit and its provider catalogs. This feature adopts v0.7.1 as its baseline.
## Scope
- Expose the optional native selector in the standalone Narratio role.
- Update standalone Notarius and suite version defaults to v0.7.1, with
matching current-version documentation and collection distribution.
- Deploy a Notarius ConfigKit testing overlay through the suite role.
- Explicitly select that overlay from Narratio's existing testing overlay.
- Select `gpt-terra-latest` for production and `gpt-luna-latest` for testing,
while retaining inventory overrides for both choices.
- Document and validate the composed configuration, then distribute the
updated suite through the collection and refresh the consumer dependency pin.
## Non-goals
- Further upstream application changes or a Narratio-side LLM-profile override.
- Automatically propagating profile names between applications.
- Disabling Notarius, caching, publishing, or other pipeline stages for testing.
- New credentials, cache layouts, other production tuning, or deployment automation.
- Live controller/target deployment or paid model calls as completion gates.
## Target End State
### Standalone Narratio Interface
`narratio_notarius_config_profile` defaults to `null`. The role omits
`notarius.config_profile` when null and serializes explicitly supplied values,
including an empty string. Documentation and argument specifications expose
the field without duplicating or narrowing Narratio's selector semantics.
### Explicit Suite Composition
The suite leaves the base selector unset. Its existing Narratio
`profiles/testing.yml` includes:
```yaml
notarius:
config_profile: testing
```
The suite deploys `/usr/local/etc/notarius/profiles/testing.yml`. That file
overlays the selected suite pipeline's `llm_profile`, using
`dnd_pipeline_narratio_notarius_pipeline_id` as the pipeline key and a new
`dnd_pipeline_notarius_testing_llm_profile` variable as the value. It does not
copy the full pipeline definition. ConfigKit performs the ordinary merge.
The default suite pipeline currently inherits its LLM selection at pipeline
level. Caller-supplied binding-level selections retain Notarius's native
precedence; this feature does not rewrite arbitrary caller pipelines or promise
that a pipeline-level default overrides explicit binding choices.
### Production and Testing LLM Defaults
The suite's base `dnd-session` pipeline selects `llm_profile: gpt-terra-latest`
instead of `dnd-extraction`. Callers can continue overriding that choice through
the existing `dnd_pipeline_notarius_pipelines` mapping. The new
`dnd_pipeline_notarius_testing_llm_profile` defaults to `gpt-luna-latest`.
Both IDs are supplied by the OpenRouter catalog `v2.20260922.1` selected by
Notarius v0.7.1. Use those native catalog profiles directly; no operator profile
asset or profile-source setting is needed for these defaults. Retain the
existing `OPENROUTER_API_KEY` credential reference. These are deliberately
moving `latest` selections, not promises of immutable underlying model versions.
This production-model policy belongs to the suite, not the standalone role.
Notarius stays enabled and retains its configured caching, output contracts,
references, and production defaults outside this overlay. Existing Scriptorium
testing overrides remain intact. Credentials continue through the existing
typed reference and suite secrets interfaces; no credential values enter YAML.
Configuration files and directories use the suite's existing owner, group,
and mode variables. The selector and overlay add no new runtime identity or
filesystem-permission policy.
## Repository Ownership
- `ansible-role-narratio`: generic optional selector, documentation, and tests.
- `ansible-role-notarius`: v0.7.1 installation default and corresponding
documentation; existing profile-source settings remain sufficient.
- `ansible-role-dnd-pipeline`: testing policy, overlay assets, and composition.
- `ansible-collection-dnd`: imported pipeline snapshot and compatible dependency
manifest. Its current Narratio-role `v0.1.0` pin must advance to include the
new public variable; changing only the sibling role is insufficient.
- `rakestraw-ansible`: matching dependency pins and concise operator guidance.
- Workspace: roadmap, implementation plan, progress, and review.
- Narratio and Notarius applications: read-only authoritative contracts.
The standalone Notarius role already exposes profile-source settings and
does not need D&D-specific overlay policy.
## Constraints and Settled Decisions
Follow the [MVP roadmap](../mvp.md), [Ansible house style](../../ansible.md),
and [testing house style](../../testing.md). Narratio v2.2.0 is already selected
by the suite; Notarius v0.7.1 is the selected target. Consult these tags, not
only newer application checkouts.
### Notarius v0.7.1 Compatibility
The [upstream release notes](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.7.1/docs/releases/v0.7.1.md)
describe unchanged CLI, configuration, artifact, and subprocess contracts.
The application-owned `dnd-extraction` fallback retains its settings and no
configuration migration is required. Independently of that upstream behavior,
the user has selected `gpt-terra-latest` as this suite's new production default.
Profile selection must use the catalogs selected by v0.7.1 (PromptKit v0.10.0).
The removed `deepseek-3-2`, `deepseek-4-flash`, and `deepseek-4-pro` catalog IDs
must not be assumed available or silently replaced by similarly named profiles.
This is deployment guidance, not a request for an Ansible catalog validator.
Existing checkpoints with the previous profile-source identity are not reused;
source-keyed chunk plans and existing output artifacts are unaffected. Document
this upstream upgrade behavior without deleting caches or forcing Narratio
extraction automatically.
Each application owns its configuration. Narratio selects the downstream
overlay; Notarius resolves and validates it. Existing application behavior owns
extraction reuse when selections change and explicit-force handling for changes
behind a stable profile name. Do not add Ansible state tracking for either.
## Simplest Viable Implementation
Add one nullable role field and its conditional template output. Extend the
suite's existing directory/template management to deploy the Notarius overlay
and extend the existing Narratio testing template with the explicit selector.
Use native serialization and ordinary file/template modules. Select the two
existing catalog IDs rather than deploying duplicate profile definitions. No assembly facts,
preflight state probes, or generic overlay-management framework is required.
## Rationale and Alternatives
A downstream ConfigKit overlay can own model choices and later Notarius tuning
without expanding Narratio's configuration surface for each setting. Direct
LLM-profile forwarding would solve only the model-selection case. Automatic
profile propagation would couple unrelated profile names and is not selected.
## Acceptance Criteria
- The standalone role omits an unset selector and preserves explicitly supplied
representative values, including the empty-string base selector.
- Native configuration inspection shows Narratio testing selecting Notarius
testing, and Notarius testing selecting the requested LLM profile. Base
configuration retains its production selection.
- A representative inventory-supplied testing selection is supported. Tests
exercise behavior rather than asserting that today's default model stays fixed.
- Notarius's deployed-version validation accepts the composed pipeline and
selected profile. No paid inference is required.
- Existing integration coverage remains valid, and immediate repeated role
convergence reports no changes. Do not independently retest file-module
ownership, modes, or template-file existence.
- The built collection works with its declared public role dependencies; no
local sibling checkout is needed by consumers. Subtree imports follow the
documented public-Gitea workflow, and published tags remain immutable.
## Dependencies and Follow-up Work
The upstream selector shipped in Narratio v2.2.0; the downstream baseline is
now Notarius v0.7.1. The staged implementation plan is maintained separately
at [implementation.md](../implementation.md).
Future testing-specific Notarius settings belong in its overlay but are not
implicitly included in this feature.
## Open Questions
None. The user selected `gpt-luna-latest` for testing and `gpt-terra-latest`
for production; both are available in the catalog selected by Notarius v0.7.1.

298
docs/roadmap/implementation.md
View File

@@ -1,12 +1,292 @@
# Implementation Plan
# Implementation Plan 0017: Notarius Testing Overlay
No implementation plan is currently active. Select and agree on the next unit
of work before replacing this notice with its staged plan.
Status: ready
Feature 0016 is complete. Its durable scope and completion evidence are in
[the feature roadmap](features/0016-deploy-dnd-pipeline-to-pipeline-dnd.md)
and [workspace status](../status.md). Git history preserves the completed plan.
Target executor: GPT-5.6-Terra
Completion covers the committed deployment playbook, host configuration,
dependency manifest, and operator documentation—not deployment or runtime
acceptance on the operational host.
Feature roadmap: [0017-notarius-testing-overlay.md](features/0017-notarius-testing-overlay.md)
## Objective
Implement the approved roadmap: Narratio testing explicitly selects Notarius
testing, with `gpt-luna-latest` for testing and `gpt-terra-latest` for production.
Keep the standalone roles generic and distribute a coherent public dependency
set. The roadmap owns scope and rationale; this document owns execution order.
## Execution Rules and Starting State
- Execute stages in order. Before each stage reread its Required Policy Review,
inspect owning working trees, and preserve unrelated changes. Paths to policy
below are relative to the workspace, not the nested repository.
- Commit validated units with plain-English messages. Record resulting full
commit IDs in Progress Notes; never invent future hashes or move release tags.
- Use the existing `.venv/bin` Ansible tools. Missing PATH entries, a local
`/etc/ansible`, or controller access are not blockers. `ansible-doc` rendering
is not an acceptance gate.
- Repository integration scripts converge this development host and use sudo;
use this established local test environment, not the operational controller
or `pipeline-dnd`. Do not provision real credentials or run paid inference.
- Read the scripts before running them. Preserve existing fixture and cleanup
conventions; report an actual missing prerequisite rather than expanding scope.
- Trust Ansible modules. Add no runtime validation commands, state fingerprints,
cache deletion, force-extraction automation, or profile discovery framework.
- Local validation can use local role sources where the existing suite does;
distribution validation must use the declared public revisions.
- Publishing changed role commits to their existing Gitea main branches is
required before public dependency installation and subtree import. Publish
only completed, validated work; if publication is unavailable, preserve the
commits and report that specific distribution blocker.
- No new release/tag or replacement of the published 0.1.0 artifact is required.
Use full pushed commit pins for this development compatibility set.
Current versions: Narratio v2.2.0; Notarius v0.7.0, to become v0.7.1.
The latter supplies PromptKit v0.10.0 and OpenRouter catalog v2.20260922.1,
which contains both selected profile IDs. Existing OPENROUTER_API_KEY delivery
is sufficient. The standalone role tests deliberately select older application
versions in some cases; do not update those merely to match a new default.
Repository scope: change the Narratio and Notarius roles, pipeline role,
collection, consumer `rakestraw-ansible`, and workspace planning records.
Applications, other standalone roles, and asset repositories are read-only.
## Stage 1: Expose Narratio's Native Profile Selector
### Required Policy Review
Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, and the Feature 0017 roadmap.
### Owning Repository and Preconditions
`ansible-role-narratio`. Its default already selects Narratio v2.2.0.
Read that tag's configuration contract for `notarius.config_profile`.
### Required Changes
1. Add `narratio_notarius_config_profile: null` to defaults and an optional
string entry to `meta/argument_specs.yml`; document the public variable.
2. Include the variable in the enclosing Notarius-section condition in
`templates/pipeline.yml.j2`. Render `config_profile` with `to_json` only
when the variable is not none. Preserve explicit empty strings and exact
values. Add no name validation or file discovery.
3. Extend the existing representative configuration tests with a supplied
nonempty selector, null omission, and intentional empty-string rendering.
Keep fixtures small; no downstream inference or overlay file is necessary
merely to validate Narratio's native selector field. Existing tests against
older application versions must leave the new field unset.
### Validation and Completion
From this repository run `tests/integration` and `git diff --check`.
The public variable renders correctly and native configuration validation and
clean repeat pass. Extend existing tests rather than creating a parallel harness.
### Commit
`Expose the Notarius configuration profile in the Narratio role`.
Push this validated commit to origin/main and record its full ID as NARRATIO_ROLE.
## Stage 2: Adopt Notarius v0.7.1
### Required Policy Review
Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, and the Feature 0017 roadmap.
### Owning Repository and Preconditions
`ansible-role-notarius`. Stage 1 is complete. Inspect v0.7.1 release notes and
the role's existing integration entry point.
### Required Changes
Set `notarius_version` to v0.7.1 and update the README's current default.
No role configuration or task change is needed for this dependency upgrade.
Retain explicitly selected historical-version tests. Do not copy new provider
catalogs or model policy into this generic role.
### Validation and Completion
Run `tests/integration` and `git diff --check` in this repository. The normal
default convergence uses v0.7.1, native checks pass, and repeat convergence is
clean. Preserve tests of caller version selection instead of asserting the default.
### Commit
`Install Notarius v0.7.1 by default`.
Push the validated commit and record its full ID as NOTARIUS_ROLE.
## Stage 3: Compose and Validate the Suite Testing Overlay
### Required Policy Review
Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, and the Feature 0017 roadmap.
### Owning Repository and Preconditions
`ansible-role-dnd-pipeline`. Stages 1-2 are validated and publicly available.
Review `tests/integration`, especially its profile diff, inventory override,
and alternate pipeline composition cases.
### Required Changes
1. Pin the Narratio and Notarius roles in `requirements.yml` to NARRATIO_ROLE
and NOTARIUS_ROLE. Leave unrelated dependencies unchanged.
2. Set `dnd_pipeline_notarius_version: v0.7.1`. In the existing default
`dnd_pipeline_notarius_pipelines` mapping, change the base dnd-session
`llm_profile` to `gpt-terra-latest`. Do not introduce a second production
profile variable: the mapping remains its inventory override interface.
3. Add `dnd_pipeline_notarius_testing_llm_profile: gpt-luna-latest`, its string
argument specification, and README documentation.
4. Extend ordinary directory management to create
`/usr/local/etc/notarius/profiles`. Add one template for
`/usr/local/etc/notarius/profiles/testing.yml` containing only:
```yaml
pipelines:
<selected suite pipeline ID>:
llm_profile: <testing LLM profile variable>
```
Serialize both dynamic key and value safely. Use the existing
`dnd_pipeline_narratio_notarius_pipeline_id` for the key. Manage the file
and directory with existing runtime owner/group and configuration modes.
Do not hard-code dnd-session or copy the full pipeline into the overlay.
5. Add `notarius: {config_profile: testing}` to Narratio's existing testing
template without changing its Scriptorium settings. Leave the base Narratio
selector unset. No extra suite-level selector variable or PromptKit asset
is needed. ConfigKit finds the overlay normally; do not import it into base.
6. Document base/testing choices, mapping overrides, native binding precedence,
moving catalog IDs, unchanged credential reference, and v0.7.1 checkpoint
compatibility. Keep Notarius enabled and caching unchanged.
7. Extend the existing integration script to inspect base and testing effective
configurations through the applications' native commands. Use explicit
caller-supplied values for assertions; for example, explicitly select
`gpt-luna-latest` in the production fixture and `gpt-terra-latest` in the
testing fixture to demonstrate independence from defaults. Validate the
selected pipeline with Notarius `--config-profile testing`, without inference.
Exercise the existing alternate pipeline ID case with the testing overlay
to prove the key follows inventory. Do not add a binding-precedence matrix.
8. Replace the old test restriction that Narratio testing differences may only
end in `.profile_id`: it excludes the new `.notarius.config_profile` setting.
Prefer assertions of the selected effective fields over an exhaustive diff
allowlist. Retain useful Scriptorium override coverage and clean-repeat checks.
### Validation and Completion
Run `tests/integration` and `git diff --check`. Required outcomes: native
validation passes for base and testing; selected inventory values are visible;
base remains independent of testing; alternate pipeline composition works;
immediate repeat is clean. No real keys or provider requests are required.
### Commit
`Add the Notarius testing overlay and production model defaults`.
Push the validated commit and record its full ID as PIPELINE_ROLE.
## Stage 4: Assemble the Public Collection Snapshot
### Required Policy Review
Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, and the collection README's subtree policy.
### Owning Repository and Preconditions
`ansible-collection-dnd`. Stage 3 and its dependency commits are pushed.
### Required Changes
1. Import PIPELINE_ROLE from Gitea using the documented workflow:
```sh
git subtree pull --prefix=roles/pipeline \
https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \
FULL_PIPELINE_ROLE_COMMIT --squash \
-m 'Bundle the Notarius testing overlay'
```
Substitute the actual full commit ID. Do not edit the subtree directly.
2. Update collection-root `requirements.yml` to the same Narratio/Notarius
role commits used by the pipeline manifest. Keep unrelated pins unchanged.
3. Extend the collection's existing installed-artifact smoke coverage to validate
Notarius testing and inspect Narratio's effective downstream selector. Reuse
Stage 3's detailed behavior evidence rather than copying its whole test suite.
4. Keep version 0.1.0 and published tags/artifacts unchanged. Document the current
development snapshot where necessary without rewriting historical release claims.
### Validation and Completion
Run `tests/integration` and `git diff --check`. This must build/install the
collection with its public declared roles and pass native base/testing checks
and clean repeat. Compare `git rev-parse HEAD:roles/pipeline` with the source
pipeline commit's root tree: they must match. No sibling-role substitution.
### Commit
The subtree command creates its own commit. Commit remaining changes as
`Validate and pin the testing overlay collection dependencies`.
Push the completed collection and record its full final ID as COLLECTION.
## Stage 5: Update Consumer Pins and Record Review Handoff
### Required Policy Review
Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, `docs/planning.md`, and Feature 0017.
### Owning Repositories and Preconditions
`rakestraw-ansible` and workspace. Stage 4 is validated and pushed.
### Required Changes
1. In the consumer manifest pin the collection to COLLECTION, and Narratio and
Notarius roles to their corresponding public commits. Preserve unrelated pins.
2. Update the collection refresh command in `docs/pipeline-dnd.md` to COLLECTION.
Explain the testing overlay behavior and v0.7.1 cache compatibility concisely.
Preserve the minimal hosts/roles playbook, host inventory, and secret references.
No host-variable copies of the new suite defaults are needed.
3. Update workspace status and this plan's Progress Notes with commit IDs and
validation evidence. Mark Feature 0017 in_progress pending independent review,
not done. Preserve the durable roadmap; no new feature scope is introduced.
### Validation and Completion
Run `git diff --check` in both repositories. Reuse Stage 4's successful public
dependency installation for the same dependency set. With those isolated role
and collection paths and workspace `.venv/bin` tools, run from the consumer:
```text
ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --list-hosts
ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --syntax-check
```
Supply explicit ANSIBLE_ROLES_PATH and ANSIBLE_COLLECTIONS_PATH for the isolated
installation. Do not install to the controller's /etc/ansible or connect to the
target. If unrelated encrypted inventory prevents local checks, use a disposable
inventory with the target alias and non-secret settings and report that scope.
Missing real Vault values are not a feature blocker; do not populate them.
### Commits
Consumer: `Select the collection with Notarius testing profiles`.
Workspace: `Record the Notarius testing overlay implementation for review`.
Push the consumer update so the controller can retrieve the published pins.
## Final Validation and Handoff
Report all resulting commits, validation performed, and any real blockers or
deviations. Confirm changed working trees are clean. Distinguish repository
readiness from an operational deployment; none is required or claimed. Note
any complexity added beyond the planned declarative tasks and its requirement.
## Progress Notes
Implementation plan approved by the user; implementation has not started.
## Open Questions
None. Production and testing selections, upstream versions, public commit-pin
distribution, and ownership boundaries are settled for this plan.

13
docs/status.md
View File

@@ -1,13 +1,15 @@
# Workspace Status
Last updated: 2026-09-21
Last updated: 2026-09-22
Current milestone: Operational D&D pipeline deployment
Current focus: Select the next unit of work. Feature 0016 repository preparation
is complete and independently reviewed with no remediation findings. The
subsequent Narratio v2.1.2 update is published and pinned by the consumer.
Deployment and session execution remain separate operator work.
Current focus: Feature 0017 is approved and ready for implementation. Its finalized
roadmap and five-stage plan select `gpt-terra-latest` for production and
`gpt-luna-latest` for testing. The roadmap targets
Notarius v0.7.1; that role upgrade is not yet implemented. Narratio v2.2.0 is
published in the suite snapshot and consumer pin (`54889e6`, `1b13e86`).
Feature 0016 is complete; deployment remains separate operator work.
| ID | Deliverable | Owner | State | Evidence or next step |
| --- | --- | --- | --- | --- |
@@ -36,6 +38,7 @@ Deployment and session execution remain separate operator work.
| F0014 | Enroll existing pipeline operators | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `c13f726` and collection import `c0b0ebd` passed their complete integration suites and independent review with no remediation findings |
| F0015 | Enable D&D S3 storage and publishing | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `51566afcf69083cdbb5b5718f2544c86b079001c`, collection fixture `cf3de016ee09edf91afc3bc1addfd926d4a172ad`, collection squash `77f67adbc0f700e88bc15554e02052eaa22f1033`, and collection merge `46913446765e86ecfb29ebad9955d579ef6a4c7e` passed the complete repository suites and independent review with no remediation findings |
| F0016 | Prepare the D&D deployment playbook for `pipeline-dnd` | rakestraw-ansible and workspace | done | Consumer `716a80e`, `12b2462`, and `7f8353d` passed independent review with no findings; public dependency installation, target-only host listing, syntax and whitespace checks passed. Follow-up `7f4c90a` pins collection `2a53979` for Narratio v2.1.2; collection build and syntax checks passed |
| F0017 | Compose the Notarius testing overlay | Narratio role, pipeline role, collection, consumer, and workspace | ready | [Finalized roadmap](roadmap/features/0017-notarius-testing-overlay.md) and [five-stage implementation plan](roadmap/implementation.md); implementation not started |
| A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` |
| R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review |