eric/ansible-workspace-dnd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Close out the Notarius testing overlay feature

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-22 19:25:35 +00:00
parent 47129b213d
commit 3bc042ea22
3 changed files with 29 additions and 330 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
docs/roadmap/features/0017-notarius-testing-overlay.md
View File

@@ -1,6 +1,6 @@
# Feature 0017: Notarius Testing Overlay
Status: in_progress
Status: done
## Summary
@@ -172,11 +172,27 @@ profile propagation would couple unrelated profile names and is not selected.
## Dependencies and Follow-up Work
The upstream selector shipped in Narratio v2.2.0; the downstream baseline is
now Notarius v0.7.1. The staged implementation plan is maintained separately
at [implementation.md](../implementation.md).
now Notarius v0.7.1. Git history preserves the completed implementation plan.
Future testing-specific Notarius settings belong in its overlay but are not
implicitly included in this feature.
## Completion and Review Evidence
Completed and published in Narratio role `330afe7`, Notarius role `a82a83d`,
pipeline role `c56a942`, collection `ce5b838`, and consumer `0ced48c`.
Implementation reported passing full integration suites for both changed
application roles, the pipeline, and the collection, including clean repeated
convergence. Independent review found no code-quality issues, architecture gaps,
or unnecessarily strengthened requirements requiring remediation.
The review independently verified collection build and public dependency
installation, syntax and shell checks, native configuration validation,
consistent dependency pins, and exact subtree/source agreement. It reviewed
the recorded integration results without rerunning host-mutating suites.
Operational deployment and paid inference were neither required nor performed
as part of this feature review.
## Open Questions
None. The user selected `gpt-luna-latest` for testing and `gpt-terra-latest`

331
docs/roadmap/implementation.md
View File

@@ -1,326 +1,9 @@
# Implementation Plan 0017: Notarius Testing Overlay
# Implementation Plan
Status: in_progress
No implementation plan is currently active. Select and agree on the next unit
of work before replacing this notice with its staged plan.
Target executor: GPT-5.6-Terra
Feature roadmap: [0017-notarius-testing-overlay.md](features/0017-notarius-testing-overlay.md)
## Objective
Implement the approved roadmap: Narratio testing explicitly selects Notarius
testing, with `gpt-luna-latest` for testing and `gpt-terra-latest` for production.
Keep the standalone roles generic and distribute a coherent public dependency
set. The roadmap owns scope and rationale; this document owns execution order.
## Execution Rules and Starting State
- Execute stages in order. Before each stage reread its Required Policy Review,
inspect owning working trees, and preserve unrelated changes. Paths to policy
below are relative to the workspace, not the nested repository.
- Commit validated units with plain-English messages. Record resulting full
commit IDs in Progress Notes; never invent future hashes or move release tags.
- Use the existing `.venv/bin` Ansible tools. Missing PATH entries, a local
`/etc/ansible`, or controller access are not blockers. `ansible-doc` rendering
is not an acceptance gate.
- Repository integration scripts converge this development host and use sudo;
use this established local test environment, not the operational controller
or `pipeline-dnd`. Do not provision real credentials or run paid inference.
- Read the scripts before running them. Preserve existing fixture and cleanup
conventions; report an actual missing prerequisite rather than expanding scope.
- Trust Ansible modules. Add no runtime validation commands, state fingerprints,
cache deletion, force-extraction automation, or profile discovery framework.
- Local validation can use local role sources where the existing suite does;
distribution validation must use the declared public revisions.
- Publishing changed role commits to their existing Gitea main branches is
required before public dependency installation and subtree import. Publish
only completed, validated work; if publication is unavailable, preserve the
commits and report that specific distribution blocker.
- No new release/tag or replacement of the published 0.1.0 artifact is required.
Use full pushed commit pins for this development compatibility set.
Current versions: Narratio v2.2.0; Notarius v0.7.0, to become v0.7.1.
The latter supplies PromptKit v0.10.0 and OpenRouter catalog v2.20260922.1,
which contains both selected profile IDs. Existing OPENROUTER_API_KEY delivery
is sufficient. The standalone role tests deliberately select older application
versions in some cases; do not update those merely to match a new default.
Repository scope: change the Narratio and Notarius roles, pipeline role,
collection, consumer `rakestraw-ansible`, and workspace planning records.
Applications, other standalone roles, and asset repositories are read-only.
## Stage 1: Expose Narratio's Native Profile Selector
Completion: complete — Narratio role commit
`330afe7856da94b2402a75447f4b60aba767843f` is published on `origin/main`;
`tests/integration` and `git diff --check` passed.
### Required Policy Review
Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, and the Feature 0017 roadmap.
### Owning Repository and Preconditions
`ansible-role-narratio`. Its default already selects Narratio v2.2.0.
Read that tag's configuration contract for `notarius.config_profile`.
### Required Changes
1. Add `narratio_notarius_config_profile: null` to defaults and an optional
string entry to `meta/argument_specs.yml`; document the public variable.
2. Include the variable in the enclosing Notarius-section condition in
`templates/pipeline.yml.j2`. Render `config_profile` with `to_json` only
when the variable is not none. Preserve explicit empty strings and exact
values. Add no name validation or file discovery.
3. Extend the existing representative configuration tests with a supplied
nonempty selector, null omission, and intentional empty-string rendering.
Keep fixtures small; no downstream inference or overlay file is necessary
merely to validate Narratio's native selector field. Existing tests against
older application versions must leave the new field unset.
### Validation and Completion
From this repository run `tests/integration` and `git diff --check`.
The public variable renders correctly and native configuration validation and
clean repeat pass. Extend existing tests rather than creating a parallel harness.
### Commit
`Expose the Notarius configuration profile in the Narratio role`.
Push this validated commit to origin/main and record its full ID as NARRATIO_ROLE.
## Stage 2: Adopt Notarius v0.7.1
Completion: complete — Notarius role commit
`a82a83dfc23bb80a59f45f94f78d6754c47285e7` is published on `origin/main`;
`tests/integration` and `git diff --check` passed.
### Required Policy Review
Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, and the Feature 0017 roadmap.
### Owning Repository and Preconditions
`ansible-role-notarius`. Stage 1 is complete. Inspect v0.7.1 release notes and
the role's existing integration entry point.
### Required Changes
Set `notarius_version` to v0.7.1 and update the README's current default.
No role configuration or task change is needed for this dependency upgrade.
Retain explicitly selected historical-version tests. Do not copy new provider
catalogs or model policy into this generic role.
### Validation and Completion
Run `tests/integration` and `git diff --check` in this repository. The normal
default convergence uses v0.7.1, native checks pass, and repeat convergence is
clean. Preserve tests of caller version selection instead of asserting the default.
### Commit
`Install Notarius v0.7.1 by default`.
Push the validated commit and record its full ID as NOTARIUS_ROLE.
## Stage 3: Compose and Validate the Suite Testing Overlay
Completion: complete — pipeline role commit
`c56a9425d30ef714166f1ff7675b202590f86425` is published on `origin/main`;
`tests/integration`, `bash -n tests/integration`, and `git diff --check` passed.
### Required Policy Review
Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, and the Feature 0017 roadmap.
### Owning Repository and Preconditions
`ansible-role-dnd-pipeline`. Stages 1-2 are validated and publicly available.
Review `tests/integration`, especially its profile diff, inventory override,
and alternate pipeline composition cases.
### Required Changes
1. Pin the Narratio and Notarius roles in `requirements.yml` to NARRATIO_ROLE
and NOTARIUS_ROLE. Leave unrelated dependencies unchanged.
2. Set `dnd_pipeline_notarius_version: v0.7.1`. In the existing default
`dnd_pipeline_notarius_pipelines` mapping, change the base dnd-session
`llm_profile` to `gpt-terra-latest`. Do not introduce a second production
profile variable: the mapping remains its inventory override interface.
3. Add `dnd_pipeline_notarius_testing_llm_profile: gpt-luna-latest`, its string
argument specification, and README documentation.
4. Extend ordinary directory management to create
`/usr/local/etc/notarius/profiles`. Add one template for
`/usr/local/etc/notarius/profiles/testing.yml` containing only:
```yaml
pipelines:
<selected suite pipeline ID>:
llm_profile: <testing LLM profile variable>
```
Serialize both dynamic key and value safely. Use the existing
`dnd_pipeline_narratio_notarius_pipeline_id` for the key. Manage the file
and directory with existing runtime owner/group and configuration modes.
Do not hard-code dnd-session or copy the full pipeline into the overlay.
5. Add `notarius: {config_profile: testing}` to Narratio's existing testing
template without changing its Scriptorium settings. Leave the base Narratio
selector unset. No extra suite-level selector variable or PromptKit asset
is needed. ConfigKit finds the overlay normally; do not import it into base.
6. Document base/testing choices, mapping overrides, native binding precedence,
moving catalog IDs, unchanged credential reference, and v0.7.1 checkpoint
compatibility. Keep Notarius enabled and caching unchanged.
7. Extend the existing integration script to inspect base and testing effective
configurations through the applications' native commands. Use explicit
caller-supplied values for assertions; for example, explicitly select
`gpt-luna-latest` in the production fixture and `gpt-terra-latest` in the
testing fixture to demonstrate independence from defaults. Validate the
selected pipeline with Notarius `--config-profile testing`, without inference.
Exercise the existing alternate pipeline ID case with the testing overlay
to prove the key follows inventory. Do not add a binding-precedence matrix.
8. Replace the old test restriction that Narratio testing differences may only
end in `.profile_id`: it excludes the new `.notarius.config_profile` setting.
Prefer assertions of the selected effective fields over an exhaustive diff
allowlist. Retain useful Scriptorium override coverage and clean-repeat checks.
### Validation and Completion
Run `tests/integration` and `git diff --check`. Required outcomes: native
validation passes for base and testing; selected inventory values are visible;
base remains independent of testing; alternate pipeline composition works;
immediate repeat is clean. No real keys or provider requests are required.
### Commit
`Add the Notarius testing overlay and production model defaults`.
Push the validated commit and record its full ID as PIPELINE_ROLE.
## Stage 4: Assemble the Public Collection Snapshot
Completion: complete — subtree import `b6d7130` and final collection commit
`ce5b838d95050f7e6c0fde4d483d269daf5d6e0a` are published on `origin/main`;
`tests/integration`, subtree tree-identity comparison, `bash -n tests/integration`,
and `git diff --check` passed.
### Required Policy Review
Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, and the collection README's subtree policy.
### Owning Repository and Preconditions
`ansible-collection-dnd`. Stage 3 and its dependency commits are pushed.
### Required Changes
1. Import PIPELINE_ROLE from Gitea using the documented workflow:
```sh
git subtree pull --prefix=roles/pipeline \
https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \
FULL_PIPELINE_ROLE_COMMIT --squash \
-m 'Bundle the Notarius testing overlay'
```
Substitute the actual full commit ID. Do not edit the subtree directly.
2. Update collection-root `requirements.yml` to the same Narratio/Notarius
role commits used by the pipeline manifest. Keep unrelated pins unchanged.
3. Extend the collection's existing installed-artifact smoke coverage to validate
Notarius testing and inspect Narratio's effective downstream selector. Reuse
Stage 3's detailed behavior evidence rather than copying its whole test suite.
4. Keep version 0.1.0 and published tags/artifacts unchanged. Document the current
development snapshot where necessary without rewriting historical release claims.
### Validation and Completion
Run `tests/integration` and `git diff --check`. This must build/install the
collection with its public declared roles and pass native base/testing checks
and clean repeat. Compare `git rev-parse HEAD:roles/pipeline` with the source
pipeline commit's root tree: they must match. No sibling-role substitution.
### Commit
The subtree command creates its own commit. Commit remaining changes as
`Validate and pin the testing overlay collection dependencies`.
Push the completed collection and record its full final ID as COLLECTION.
## Stage 5: Update Consumer Pins and Record Review Handoff
Completion: complete — consumer commit
`0ced48c99010772b2cbf9a2df5eb694af6ae77f6` is published on `origin/main`;
public dependency installation, target-only host listing, syntax checking, and
`git diff --check` passed. The workspace review handoff is recorded by this change.
### Required Policy Review
Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
`docs/ansible.md`, `docs/testing.md`, `docs/planning.md`, and Feature 0017.
### Owning Repositories and Preconditions
`rakestraw-ansible` and workspace. Stage 4 is validated and pushed.
### Required Changes
1. In the consumer manifest pin the collection to COLLECTION, and Narratio and
Notarius roles to their corresponding public commits. Preserve unrelated pins.
2. Update the collection refresh command in `docs/pipeline-dnd.md` to COLLECTION.
Explain the testing overlay behavior and v0.7.1 cache compatibility concisely.
Preserve the minimal hosts/roles playbook, host inventory, and secret references.
No host-variable copies of the new suite defaults are needed.
3. Update workspace status and this plan's Progress Notes with commit IDs and
validation evidence. Mark Feature 0017 in_progress pending independent review,
not done. Preserve the durable roadmap; no new feature scope is introduced.
### Validation and Completion
Run `git diff --check` in both repositories. Reuse Stage 4's successful public
dependency installation for the same dependency set. With those isolated role
and collection paths and workspace `.venv/bin` tools, run from the consumer:
```text
ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --list-hosts
ansible-playbook -i inventory playbooks/pipeline/dnd.yml --limit pipeline-dnd --syntax-check
```
Supply explicit ANSIBLE_ROLES_PATH and ANSIBLE_COLLECTIONS_PATH for the isolated
installation. Do not install to the controller's /etc/ansible or connect to the
target. If unrelated encrypted inventory prevents local checks, use a disposable
inventory with the target alias and non-secret settings and report that scope.
Missing real Vault values are not a feature blocker; do not populate them.
### Commits
Consumer: `Select the collection with Notarius testing profiles`.
Workspace: `Record the Notarius testing overlay implementation for review`.
Push the consumer update so the controller can retrieve the published pins.
## Final Validation and Handoff
Report all resulting commits, validation performed, and any real blockers or
deviations. Confirm changed working trees are clean. Distinguish repository
readiness from an operational deployment; none is required or claimed. Note
any complexity added beyond the planned declarative tasks and its requirement.
## Progress Notes
- NARRATIO_ROLE: `330afe7856da94b2402a75447f4b60aba767843f`
- NOTARIUS_ROLE: `a82a83dfc23bb80a59f45f94f78d6754c47285e7`
- PIPELINE_ROLE: `c56a9425d30ef714166f1ff7675b202590f86425`
- Collection subtree merge: `b6d7130046e25c7551e1412e805a28460f841900`
- COLLECTION: `ce5b838d95050f7e6c0fde4d483d269daf5d6e0a`
- Consumer: `0ced48c99010772b2cbf9a2df5eb694af6ae77f6`
- Full integration suites passed in the Narratio, Notarius, pipeline, and
collection repositories. The collection subtree tree matched the source
pipeline tree. The consumer installed the public manifest into isolated
paths and passed target-only host listing and syntax checking.
- Feature 0017 remains `in_progress` pending independent review. No operational
controller deployment, target connection, credential provisioning, or paid
inference was performed or claimed.
## Open Questions
None. Production and testing selections, upstream versions, public commit-pin
distribution, and ownership boundaries are settled for this plan.
Feature 0017 is complete and independently reviewed with no remediation
findings. Its scope and completion evidence are recorded in
[the feature roadmap](features/0017-notarius-testing-overlay.md) and
[workspace status](../status.md). Git history preserves the completed plan.

6
docs/status.md
View File

@@ -4,8 +4,8 @@ Last updated: 2026-09-22
Current milestone: Operational D&D pipeline deployment
Current focus: Feature 0017 implementation is complete and awaiting independent
review. The published development set selects `gpt-terra-latest` for production,
Current focus: Select the next unit of work. Feature 0017 passed independent
review with no remediation findings. The published development set selects `gpt-terra-latest` for production,
`gpt-luna-latest` for testing, and Notarius v0.7.1. Consumer commit `0ced48c`
pins collection commit `ce5b838` and the compatible Narratio and Notarius role
commits. Feature 0016 is complete; deployment remains separate operator work.
@@ -37,7 +37,7 @@ commits. Feature 0016 is complete; deployment remains separate operator work.
| F0014 | Enroll existing pipeline operators | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `c13f726` and collection import `c0b0ebd` passed their complete integration suites and independent review with no remediation findings |
| F0015 | Enable D&D S3 storage and publishing | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `51566afcf69083cdbb5b5718f2544c86b079001c`, collection fixture `cf3de016ee09edf91afc3bc1addfd926d4a172ad`, collection squash `77f67adbc0f700e88bc15554e02052eaa22f1033`, and collection merge `46913446765e86ecfb29ebad9955d579ef6a4c7e` passed the complete repository suites and independent review with no remediation findings |
| F0016 | Prepare the D&D deployment playbook for `pipeline-dnd` | rakestraw-ansible and workspace | done | Consumer `716a80e`, `12b2462`, and `7f8353d` passed independent review with no findings; public dependency installation, target-only host listing, syntax and whitespace checks passed. Follow-up `7f4c90a` pins collection `2a53979` for Narratio v2.1.2; collection build and syntax checks passed |
| F0017 | Compose the Notarius testing overlay | Narratio role, pipeline role, collection, consumer, and workspace | in_progress | Implementation is published through Narratio `330afe7`, Notarius `a82a83d`, pipeline `c56a942`, collection `ce5b838`, and consumer `0ced48c`; independent review is next |
| F0017 | Compose the Notarius testing overlay | Narratio role, pipeline role, collection, consumer, and workspace | done | Published through Narratio `330afe7`, Notarius `a82a83d`, pipeline `c56a942`, collection `ce5b838`, and consumer `0ced48c`; independent review found no remediation issues; validation evidence is in the feature roadmap |
| A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` |
| R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review |