Plan the Notarius testing overlay implementation
This commit is contained in:
183
docs/roadmap/features/0017-notarius-testing-overlay.md
Normal file
183
docs/roadmap/features/0017-notarius-testing-overlay.md
Normal file
@@ -0,0 +1,183 @@
|
||||
# Feature 0017: Notarius Testing Overlay
|
||||
|
||||
Status: ready
|
||||
|
||||
## Summary
|
||||
|
||||
Selecting Narratio's `testing` profile also selects a separately managed
|
||||
Notarius `testing` configuration overlay. Operators can develop D&D assets
|
||||
with economical extraction settings without modifying production configuration
|
||||
or invoking Notarius separately.
|
||||
|
||||
## Problem or Opportunity
|
||||
|
||||
The suite's existing testing overlay changes Scriptorium selections but leaves
|
||||
Notarius on its production configuration. Narratio v2.2.0 now provides the
|
||||
explicit `pipeline.notarius.config_profile` interface requested by this consumer.
|
||||
Notarius v0.7.1 retains the corresponding `--config-profile` option and updates
|
||||
PromptKit and its provider catalogs. This feature adopts v0.7.1 as its baseline.
|
||||
|
||||
## Scope
|
||||
|
||||
- Expose the optional native selector in the standalone Narratio role.
|
||||
- Update standalone Notarius and suite version defaults to v0.7.1, with
|
||||
matching current-version documentation and collection distribution.
|
||||
- Deploy a Notarius ConfigKit testing overlay through the suite role.
|
||||
- Explicitly select that overlay from Narratio's existing testing overlay.
|
||||
- Select `gpt-terra-latest` for production and `gpt-luna-latest` for testing,
|
||||
while retaining inventory overrides for both choices.
|
||||
- Document and validate the composed configuration, then distribute the
|
||||
updated suite through the collection and refresh the consumer dependency pin.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Further upstream application changes or a Narratio-side LLM-profile override.
|
||||
- Automatically propagating profile names between applications.
|
||||
- Disabling Notarius, caching, publishing, or other pipeline stages for testing.
|
||||
- New credentials, cache layouts, other production tuning, or deployment automation.
|
||||
- Live controller/target deployment or paid model calls as completion gates.
|
||||
|
||||
## Target End State
|
||||
|
||||
### Standalone Narratio Interface
|
||||
|
||||
`narratio_notarius_config_profile` defaults to `null`. The role omits
|
||||
`notarius.config_profile` when null and serializes explicitly supplied values,
|
||||
including an empty string. Documentation and argument specifications expose
|
||||
the field without duplicating or narrowing Narratio's selector semantics.
|
||||
|
||||
### Explicit Suite Composition
|
||||
|
||||
The suite leaves the base selector unset. Its existing Narratio
|
||||
`profiles/testing.yml` includes:
|
||||
|
||||
```yaml
|
||||
notarius:
|
||||
config_profile: testing
|
||||
```
|
||||
|
||||
The suite deploys `/usr/local/etc/notarius/profiles/testing.yml`. That file
|
||||
overlays the selected suite pipeline's `llm_profile`, using
|
||||
`dnd_pipeline_narratio_notarius_pipeline_id` as the pipeline key and a new
|
||||
`dnd_pipeline_notarius_testing_llm_profile` variable as the value. It does not
|
||||
copy the full pipeline definition. ConfigKit performs the ordinary merge.
|
||||
|
||||
The default suite pipeline currently inherits its LLM selection at pipeline
|
||||
level. Caller-supplied binding-level selections retain Notarius's native
|
||||
precedence; this feature does not rewrite arbitrary caller pipelines or promise
|
||||
that a pipeline-level default overrides explicit binding choices.
|
||||
|
||||
### Production and Testing LLM Defaults
|
||||
|
||||
The suite's base `dnd-session` pipeline selects `llm_profile: gpt-terra-latest`
|
||||
instead of `dnd-extraction`. Callers can continue overriding that choice through
|
||||
the existing `dnd_pipeline_notarius_pipelines` mapping. The new
|
||||
`dnd_pipeline_notarius_testing_llm_profile` defaults to `gpt-luna-latest`.
|
||||
|
||||
Both IDs are supplied by the OpenRouter catalog `v2.20260922.1` selected by
|
||||
Notarius v0.7.1. Use those native catalog profiles directly; no operator profile
|
||||
asset or profile-source setting is needed for these defaults. Retain the
|
||||
existing `OPENROUTER_API_KEY` credential reference. These are deliberately
|
||||
moving `latest` selections, not promises of immutable underlying model versions.
|
||||
This production-model policy belongs to the suite, not the standalone role.
|
||||
|
||||
Notarius stays enabled and retains its configured caching, output contracts,
|
||||
references, and production defaults outside this overlay. Existing Scriptorium
|
||||
testing overrides remain intact. Credentials continue through the existing
|
||||
typed reference and suite secrets interfaces; no credential values enter YAML.
|
||||
|
||||
Configuration files and directories use the suite's existing owner, group,
|
||||
and mode variables. The selector and overlay add no new runtime identity or
|
||||
filesystem-permission policy.
|
||||
|
||||
## Repository Ownership
|
||||
|
||||
- `ansible-role-narratio`: generic optional selector, documentation, and tests.
|
||||
- `ansible-role-notarius`: v0.7.1 installation default and corresponding
|
||||
documentation; existing profile-source settings remain sufficient.
|
||||
- `ansible-role-dnd-pipeline`: testing policy, overlay assets, and composition.
|
||||
- `ansible-collection-dnd`: imported pipeline snapshot and compatible dependency
|
||||
manifest. Its current Narratio-role `v0.1.0` pin must advance to include the
|
||||
new public variable; changing only the sibling role is insufficient.
|
||||
- `rakestraw-ansible`: matching dependency pins and concise operator guidance.
|
||||
- Workspace: roadmap, implementation plan, progress, and review.
|
||||
- Narratio and Notarius applications: read-only authoritative contracts.
|
||||
The standalone Notarius role already exposes profile-source settings and
|
||||
does not need D&D-specific overlay policy.
|
||||
|
||||
## Constraints and Settled Decisions
|
||||
|
||||
Follow the [MVP roadmap](../mvp.md), [Ansible house style](../../ansible.md),
|
||||
and [testing house style](../../testing.md). Narratio v2.2.0 is already selected
|
||||
by the suite; Notarius v0.7.1 is the selected target. Consult these tags, not
|
||||
only newer application checkouts.
|
||||
|
||||
### Notarius v0.7.1 Compatibility
|
||||
|
||||
The [upstream release notes](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.7.1/docs/releases/v0.7.1.md)
|
||||
describe unchanged CLI, configuration, artifact, and subprocess contracts.
|
||||
The application-owned `dnd-extraction` fallback retains its settings and no
|
||||
configuration migration is required. Independently of that upstream behavior,
|
||||
the user has selected `gpt-terra-latest` as this suite's new production default.
|
||||
|
||||
Profile selection must use the catalogs selected by v0.7.1 (PromptKit v0.10.0).
|
||||
The removed `deepseek-3-2`, `deepseek-4-flash`, and `deepseek-4-pro` catalog IDs
|
||||
must not be assumed available or silently replaced by similarly named profiles.
|
||||
This is deployment guidance, not a request for an Ansible catalog validator.
|
||||
|
||||
Existing checkpoints with the previous profile-source identity are not reused;
|
||||
source-keyed chunk plans and existing output artifacts are unaffected. Document
|
||||
this upstream upgrade behavior without deleting caches or forcing Narratio
|
||||
extraction automatically.
|
||||
|
||||
Each application owns its configuration. Narratio selects the downstream
|
||||
overlay; Notarius resolves and validates it. Existing application behavior owns
|
||||
extraction reuse when selections change and explicit-force handling for changes
|
||||
behind a stable profile name. Do not add Ansible state tracking for either.
|
||||
|
||||
## Simplest Viable Implementation
|
||||
|
||||
Add one nullable role field and its conditional template output. Extend the
|
||||
suite's existing directory/template management to deploy the Notarius overlay
|
||||
and extend the existing Narratio testing template with the explicit selector.
|
||||
Use native serialization and ordinary file/template modules. Select the two
|
||||
existing catalog IDs rather than deploying duplicate profile definitions. No assembly facts,
|
||||
preflight state probes, or generic overlay-management framework is required.
|
||||
|
||||
## Rationale and Alternatives
|
||||
|
||||
A downstream ConfigKit overlay can own model choices and later Notarius tuning
|
||||
without expanding Narratio's configuration surface for each setting. Direct
|
||||
LLM-profile forwarding would solve only the model-selection case. Automatic
|
||||
profile propagation would couple unrelated profile names and is not selected.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- The standalone role omits an unset selector and preserves explicitly supplied
|
||||
representative values, including the empty-string base selector.
|
||||
- Native configuration inspection shows Narratio testing selecting Notarius
|
||||
testing, and Notarius testing selecting the requested LLM profile. Base
|
||||
configuration retains its production selection.
|
||||
- A representative inventory-supplied testing selection is supported. Tests
|
||||
exercise behavior rather than asserting that today's default model stays fixed.
|
||||
- Notarius's deployed-version validation accepts the composed pipeline and
|
||||
selected profile. No paid inference is required.
|
||||
- Existing integration coverage remains valid, and immediate repeated role
|
||||
convergence reports no changes. Do not independently retest file-module
|
||||
ownership, modes, or template-file existence.
|
||||
- The built collection works with its declared public role dependencies; no
|
||||
local sibling checkout is needed by consumers. Subtree imports follow the
|
||||
documented public-Gitea workflow, and published tags remain immutable.
|
||||
|
||||
## Dependencies and Follow-up Work
|
||||
|
||||
The upstream selector shipped in Narratio v2.2.0; the downstream baseline is
|
||||
now Notarius v0.7.1. The staged implementation plan is maintained separately
|
||||
at [implementation.md](../implementation.md).
|
||||
Future testing-specific Notarius settings belong in its overlay but are not
|
||||
implicitly included in this feature.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The user selected `gpt-luna-latest` for testing and `gpt-terra-latest`
|
||||
for production; both are available in the catalog selected by Notarius v0.7.1.
|
||||
Reference in New Issue
Block a user