Plan the standalone Scriptorium upgrade

2026-09-28 03:40:04 +00:00
parent 3bc042ea22
commit 5a8e3fccc9
4 changed files with 456 additions and 13 deletions
--- a/docs/roadmap/application-upgrade-sequence.md
+++ b/docs/roadmap/application-upgrade-sequence.md
@@ -0,0 +1,87 @@
+# Application Upgrade Sequence
+
+Decision agreed: 2026-09-28
+
+## Purpose and Order
+
+Adopt the newly released applications through focused, individually reviewed
+features in this order:
+
+**Scriptorium → Audita → Notarius → Narratio → suite integration.**
+
+This document tracks the cross-feature direction, not implementation steps.
+Each feature roadmap owns its scope and decisions. Only one staged plan is
+active at a time, at [implementation.md](implementation.md).
+
+## Version Baseline
+
+These are the suite pins at the start of this upgrade, not assertions about
+what is installed on an operational host. Targets are the releases reviewed
+on the decision date; later releases require deliberate review.
+
+| Application | Starting suite version | Adoption target |
+| --- | --- | --- |
+| Scriptorium | v0.13.0 | v0.15.0 |
+| Audita | v0.9.1 | v2.0.1 |
+| Notarius | v0.7.1 | v0.8.1 |
+| Narratio | v2.2.0 | v3.0.0 |
+
+Seriatim is unchanged by this sequence.
+
+## Progress and Feature Links
+
+Release-summary and migration-guidance triage is complete. Detailed adoption
+reviews and implementation have not started. Add the numbered feature link
+beside each item when its roadmap is created; check it off after independent
+review and closeout.
+
+- [ ] [Feature 0018: Scriptorium adoption](features/0018-scriptorium-v0-15-adoption.md) — roadmap and staged plan approved; ready for implementation.
+- [ ] Audita adoption — feature roadmap not yet created.
+- [ ] Notarius adoption — feature roadmap not yet created.
+- [ ] Narratio adoption — feature roadmap not yet created.
+- [ ] Suite integration — feature roadmap not yet created.
+
+## Adoption Boundaries
+
+Completing a standalone-role update does **not** advance the suite's application
+or role dependency pins, collection snapshot, or deployment-repository pins.
+Keep that existing compatibility set until the final integration feature
+assembles and validates the coordinated configuration.
+
+Standalone features own generic application interfaces. The integration feature
+owns D&D defaults, cross-application configuration, testing overlays, credential
+references, compatible dependency pins, and consumer upgrade guidance. Each
+feature follows roadmap → staged plan → implementation → review → closeout.
+No operational deployment is authorized merely by this sequencing decision.
+
+## Compatibility Constraints to Carry Forward
+
+- **Scriptorium is not a documentation-only upgrade from our baseline.** The
+  intervening v0.14.0 introduces ConfigKit and changes configuration acceptance,
+  selection, and retry behavior. Review those changes as well as v0.15.0's
+  licensing and dependency updates.
+- **Audita and Narratio need a coordinated suite cutover.** Audita v2 requires
+  migrated configuration; Narratio v3 expects that interface and no longer owns
+  the retired downstream settings or generated Audita configuration.
+- **Overlay terminology changes on both sides of the Notarius boundary.**
+  Narratio's selector and the suite-managed downstream files must migrate
+  together. Preserve explicit selection and independent application ownership.
+- **Existing Narratio manifests need operator migration guidance.** Carry the
+  upstream provenance-field migration into final integration planning. Preserve
+  session progress; do not infer a requirement to delete checkpoints, force all
+  stages, edit immutable remote snapshots, or automate the migration.
+- **Review retry and timeout semantics explicitly.** New transport retry and
+  repair behavior is not necessarily equivalent to the previous settings.
+  Preserve user intent without copying old values mechanically or inventing
+  new suite policy.
+
+The upstream documentation remains authoritative for exact fields and migration
+procedures:
+
+- [Scriptorium v0.14.0](https://gitea.maximumdirect.net/eric/scriptorium/src/tag/v0.14.0/docs/releases/v0.14.0.md)
+  and [v0.15.0](https://gitea.maximumdirect.net/eric/scriptorium/src/tag/v0.15.0/docs/releases/v0.15.0.md).
+- [Audita v2 migration](https://gitea.maximumdirect.net/eric/audita/src/tag/v2.0.1/docs/migrations/v2.md).
+- [Notarius v0.8.0](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.8.0/docs/releases/v0.8.0.md)
+  and [v0.8.1](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.8.1/docs/releases/v0.8.1.md).
+- [Narratio v3.0.0](https://gitea.maximumdirect.net/eric/narratio/src/tag/v3.0.0/docs/releases/v3.0.0.md)
+  and [migration procedures](https://gitea.maximumdirect.net/eric/narratio/src/tag/v3.0.0/docs/operations.md#migrating-manifest-overlay-provenance).
--- a/docs/roadmap/features/0018-scriptorium-v0-15-adoption.md
+++ b/docs/roadmap/features/0018-scriptorium-v0-15-adoption.md
@@ -0,0 +1,149 @@
+# Feature 0018: Adopt Scriptorium v0.15.0 in the Standalone Role
+
+Status: ready
+
+## Summary
+
+Update the standalone Scriptorium role from v0.13.0 to v0.15.0 and expose its
+new application-configuration settings using our existing nullable-variable
+conventions. This is the first feature in the agreed
+[application upgrade sequence](../application-upgrade-sequence.md).
+
+## Problem or Opportunity
+
+Although v0.15.0 primarily changes licensing and dependency versions, the
+intervening v0.14.0 adopted ConfigKit and added application imports, overlays,
+inspection, and transient retry controls. A version bump alone would leave
+new configuration settings unavailable through the role's public interface.
+
+The current role already emits `{}` for an unconfigured installation, omits
+null optional fields, serializes mappings, and retains explicit falsey values.
+Those practices remain compatible with the stricter upstream configuration
+contract; no new production validation or migration mechanism is needed.
+
+## Scope
+
+- Adopt v0.15.0 as the standalone role's default source version.
+- Expose ConfigKit imports and the application transient-retry setting.
+- Confirm existing configuration variables and representative backend mappings
+  remain usable with the new application version.
+- Update role documentation and focused offline integration coverage.
+- Record completion and the role revision for later coordinated integration.
+
+## Non-goals
+
+- Changing pipeline, collection, consumer, or other application-role pins.
+- D&D model choices, production/testing policy, or new provider credentials.
+- Deploying prompt/profile/schema libraries or managing arbitrary overlay files.
+- Selecting a runtime overlay for callers, managing a service, or adding HTTP
+  request/CLI-only controls such as execution reports and run deadlines to YAML.
+- Upstream application changes, paid inference, or operational host deployment.
+- Relicensing this Ansible role or copying upstream source/license material
+  into the role repository merely because the installed application changed.
+
+## Target End State
+
+### Installation and Existing Configuration
+
+`scriptorium_version` defaults to `v0.15.0`. Retain the existing source checkout,
+Go build, ownership, and permission behavior and the caller-selected version
+interface. No build-toolchain change is required: the current Go 1.26 selection
+satisfies the release's Go requirement.
+
+Existing application fields keep their public variable names. Optional variables
+remain null by default and are omitted; explicit supported zero/empty values
+are serialized without truthiness checks. An installation with no application
+settings still receives `{}`, not an empty file.
+
+### New Public Variables
+
+| Variable | Default | Rendered field |
+| --- | --- | --- |
+| `scriptorium_configkit_imports` | `null` | `configkit.imports` |
+| `scriptorium_transient_retries` | `null` | `transient_retries` |
+
+Imports are a native list of mapping entries, consistent with the other roles'
+ConfigKit interfaces. Serialize the supplied list without reinterpreting path,
+glob, or optional-entry semantics. Emit the parent section only when selected.
+The caller owns referenced fragment and overlay files; ordinary inventory can
+compose the list without a second raw-configuration interface.
+
+Transient retries is an optional integer. Null means omit and inherit the
+application/PromptKit default; zero explicitly disables additional transient
+attempts. The standalone role does not freeze the upstream default or add a
+retry budget of its own. Argument specifications and documentation describe
+both variables consistently.
+
+### Compatibility Guidance
+
+Document the distinction between ConfigKit `overlays/` selected by
+`--config-overlay` and PromptKit LLM profiles. Imports are configuration-root
+relative, while application path values retain their process-working-directory
+semantics. Link upstream for detailed validation and precedence rules.
+
+Call out stricter YAML acceptance and changed retry behavior, plus the removal
+of legacy DeepSeek catalog IDs, without installing compatibility aliases or
+inventing role-side configuration guards. New fields should only be supplied
+when selecting an application version that supports them.
+
+Mention that Scriptorium v0.15.0's application grant is GPL-3.0-or-later and link
+its tagged licensing notices. Do not conflate the application's grant with the
+role's license or introduce legal approval as a deployment gate.
+
+## Repository Ownership
+
+- `ansible-role-scriptorium`: default version, variables, argument specification,
+  template, README, and repository-local tests.
+- Workspace: feature roadmap, staged plan, progress, and review evidence.
+- Scriptorium application: read-only tagged configuration/release contracts.
+- Pipeline, collection, consumer, and asset repositories remain unchanged.
+
+## Constraints and Settled Decisions
+
+Follow the [Ansible house style](../../ansible.md),
+[testing house style](../../testing.md), [MVP roadmap](../mvp.md), and
+[upgrade sequence](../application-upgrade-sequence.md). Review the tagged
+v0.15.0 contract rather than assuming later application changes apply.
+
+The existing suite continues installing its current Scriptorium role/application
+pins until final integration. Standalone adoption is not suite publication.
+
+## Simplest Viable Implementation
+
+Update the version default and extend the existing defaults, argument
+specification, and single configuration template with the two optional fields.
+Include both in the empty-document condition. Reuse the existing source/build
+and file/template tasks unchanged. No additional runtime tasks are needed.
+
+## Acceptance Criteria
+
+- The default role installs v0.15.0 and native offline configuration validation
+  accepts the default and representative managed configurations.
+- Representative non-null and falsey values remain correctly rendered; the new
+  retry field preserves explicit zero and is omitted when unset.
+- A caller-supplied import is composed successfully by native configuration
+  inspection using a small externally supplied test fragment. An optional test
+  overlay can demonstrate native composition without adding role-owned overlay
+  management or retesting ConfigKit's complete precedence/error matrix.
+- Existing offline profile/backend inspection remains useful and passes.
+  Application validation alone does not prove PromptKit source validity.
+- Immediate repeated role application reports no changes. Tests assert supplied
+  configuration outcomes, not today's upstream/default version, catalog contents,
+  task arrangement, or file-module postconditions.
+- Syntax/lint and repository integration checks pass without provider calls,
+  real secrets, or operational controller access.
+- Suite and consumer pins remain unchanged, and completion identifies the role
+  commit available for the final integration feature.
+
+## Dependencies and Follow-up Work
+
+The approved roadmap's staged plan is at
+[implementation.md](../implementation.md). Audita adoption follows independent
+review and closeout. The eventual suite integration feature owns propagation of
+the new role revision, deployment settings, and coordinated upgrade guidance.
+
+## Upstream References
+
+- [v0.15.0 configuration](https://gitea.maximumdirect.net/eric/scriptorium/src/tag/v0.15.0/docs/config.md)
+- [v0.14.0 migration notes](https://gitea.maximumdirect.net/eric/scriptorium/src/tag/v0.14.0/docs/releases/v0.14.0.md)
+- [v0.15.0 release and licensing notes](https://gitea.maximumdirect.net/eric/scriptorium/src/tag/v0.15.0/docs/releases/v0.15.0.md)
--- a/docs/roadmap/implementation.md
+++ b/docs/roadmap/implementation.md
@@ -1,9 +1,214 @@
-# Implementation Plan
+# Implementation Plan 0018: Standalone Scriptorium v0.15.0 Adoption

-No implementation plan is currently active. Select and agree on the next unit
-of work before replacing this notice with its staged plan.
+Status: ready

-Feature 0017 is complete and independently reviewed with no remediation
-findings. Its scope and completion evidence are recorded in
-[the feature roadmap](features/0017-notarius-testing-overlay.md) and
-[workspace status](../status.md). Git history preserves the completed plan.
+Target executor: GPT-5.6-Terra
+
+Feature roadmap: [0018-scriptorium-v0-15-adoption.md](features/0018-scriptorium-v0-15-adoption.md)
+
+## Objective and Boundaries
+
+Adopt Scriptorium v0.15.0 in its standalone role, expose ConfigKit imports and
+transient retries, and validate the resulting configuration offline. The approved
+feature roadmap owns scope, rationale, and acceptance criteria. This plan does
+not advance the [coordinated upgrade sequence](application-upgrade-sequence.md)
+beyond its Scriptorium feature.
+
+Only `ansible-role-scriptorium` and workspace planning records may change.
+Upstream applications and all other roles, collection, consumer, and asset
+repositories are read-only. In particular, do not update suite application
+defaults, role dependency manifests, collection subtrees, or deployment pins.
+
+## Execution Rules and Starting State
+
+- Execute the consecutive stages below in order. Before each stage reread its
+  listed policy documents, inspect the owning working trees, and preserve
+  unrelated changes. All policy paths below are workspace-relative.
+- The role currently defaults to v0.13.0; the selected target is v0.15.0, with
+  ConfigKit v0.4.2 and PromptKit v0.11.2. Consult tagged application documentation
+  rather than assuming newer checkout behavior belongs to this feature.
+- Use workspace `.venv/bin` Ansible tools and the role's existing integration
+  entry point. Missing PATH entries or an operational controller are not blockers.
+  Existing Go 1.26 satisfies upstream's Go requirement.
+- The integration script performs local development-host convergence with sudo.
+  Read its fixture and cleanup behavior before execution. Do not connect to the
+  operational controller or pipeline host, supply real keys, or call providers.
+- Trust module state management. No new production tasks, runtime validation,
+  probes, state files, custom retry implementation, or migration automation are
+  required. Tests may inspect rendered configuration to verify role-owned output.
+- Keep null optional fields omitted and serialize native values safely. Do not
+  freeze upstream defaults or enforce additional selector/import restrictions.
+- Commit validated stages with the plain-English intent given below. No push,
+  release tag, collection assembly, or deployment is required by this plan.
+- If validation reveals a real incompatibility, report its evidence and resolve
+  only what falls within the roadmap. Do not silently weaken useful tests or
+  broaden scope. Record validation and commit IDs in Progress Notes.
+
+## Stage 1: Adopt the New Application Version
+
+### Required Policy Review
+
+Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
+`docs/ansible.md`, `docs/testing.md`, and the Feature 0018 roadmap.
+
+### Owning Repository and Preconditions
+
+`ansible-role-scriptorium`. Inspect its current defaults, tasks, template,
+argument specifications, README, and `tests/integration`. Read the upstream
+v0.14.0 migration notes and v0.15.0 configuration/release references linked by
+the roadmap. Confirm the target tag without modifying upstream worktrees.
+
+### Required Changes
+
+1. Set `scriptorium_version: v0.15.0` and update its README default example.
+   Keep the source/build/dependency and filesystem tasks unchanged.
+2. Add native `config validate --config <managed-file>` checks to the existing
+   integration script for default and representative configurations. Retain
+   existing offline profile/backend inspection, because application validation
+   alone does not construct a PromptKit engine.
+3. Preserve existing caller-supplied build-executable and missing-binary coverage.
+   Do not add an assertion that v0.15.0 must always be the default or require a
+   new historical-version matrix. Limit fixture corrections to actual changed
+   application contracts.
+
+### Validation and Completion Criteria
+
+From `ansible-role-scriptorium`, run:
+
+```sh
+tests/integration
+git diff --check
+```
+
+The new version builds; default and representative configurations validate;
+offline profile inspection succeeds; immediate repeat convergence is clean.
+The existing script already includes syntax checking and linting.
+
+### Commit
+
+`Adopt Scriptorium v0.15.0 in the standalone role`.
+
+## Stage 2: Expose Imports and Transient Retries
+
+### Required Policy Review
+
+Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
+`docs/ansible.md`, `docs/testing.md`, and the Feature 0018 roadmap.
+
+### Owning Repository and Preconditions
+
+`ansible-role-scriptorium`. Stage 1 is committed and validated. Review the tagged
+configuration reference's import resolution, null, and retry semantics.
+
+### Required Changes
+
+1. Add defaults `scriptorium_configkit_imports: null` and
+   `scriptorium_transient_retries: null`. Add optional argument specifications:
+   imports is a list of dictionaries, retries is an integer. Use native mappings
+   for import entries; do not recreate ConfigKit's nested schema in the role.
+2. Extend `templates/config.yml.j2` to render `configkit.imports` when imports
+   is not none, and top-level `transient_retries` when retries is not none.
+   Use existing serialization conventions. Include both variables in the
+   empty-document condition so either field alone prevents output of `{}`.
+   With all application variables null, retain the minimal `{}` document.
+3. Extend the existing integration fixture with a caller-owned configuration
+   fragment under a temporary configuration root. Select that root using
+   `scriptorium_config_path`. Supply imports through the role's public variable;
+   use a relative `path` entry naming the fragment.
+4. In that focused case, render only imports and `scriptorium_transient_retries: 0`
+   through application variables. Put one representative setting, such as
+   `server.addr: 127.0.0.1:18081`, in the external fragment. Do not also define
+   that setting in the main file: ordinary imports reject field conflicts.
+   Native `config show` should expose the imported setting and retry zero.
+   Inspect the rendered root to verify that the supplied import was serialized.
+5. Reapply the same role inputs immediately and require a clean recap. For the
+   default case, verify omission of the two new fields in the rendered YAML,
+   not exact native defaults in resolved output. Reuse existing test structure;
+   no additional overlay scenario is required for this role's import interface.
+6. Ensure the temporary configuration and fragment live under the script's
+   existing disposable fixture directory and cleanup restores the default role
+   configuration. No production task should create the test fragment or manage
+   arbitrary import/overlay content.
+7. Document the new public variables and one concise inventory import example.
+   Explain null omission, zero retry behavior, caller-owned files, and that newer
+   fields must be left unset when selecting older unsupported binaries.
+
+### Validation and Completion Criteria
+
+From the role repository run:
+
+```sh
+tests/integration
+git diff --check
+```
+
+Native validation accepts the new configuration. Native inspection resolves
+the explicitly supplied fragment and retry setting without provider calls.
+Default omission and repeated convergence pass. No tests assert exact task
+layout, default version/model, or file-module ownership/existence postconditions.
+
+### Commit
+
+`Expose Scriptorium imports and transient retry settings`.
+
+## Stage 3: Document Compatibility and Record the Review Handoff
+
+### Required Policy Review
+
+Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`,
+`docs/ansible.md`, `docs/testing.md`, `docs/planning.md`, the Feature 0018 roadmap,
+and `docs/roadmap/application-upgrade-sequence.md`.
+
+### Owning Repositories and Preconditions
+
+`ansible-role-scriptorium` and workspace. Stages 1-2 are validated and committed.
+
+### Required Changes
+
+1. Finish the role README's upgrade guidance: strict YAML/empty-document behavior,
+   ConfigKit imports and `--config-overlay` versus PromptKit LLM profiles,
+   configuration-source versus application-value path semantics, and changed
+   transport retry behavior. Link tagged upstream contracts for full detail.
+2. Mention removed legacy DeepSeek catalog IDs without adding compatibility aliases.
+   Note Scriptorium v0.15.0's GPL-3.0-or-later application grant and link its
+   licensing notices; leave the role's own license untouched. Do not introduce
+   a legal approval gate or copy license files into this installer role.
+3. Update the README validation summary to match actual coverage. Do not claim
+   provider connectivity, live service operation, or suite compatibility testing.
+4. Record role commits and validation evidence in this plan and workspace status.
+   Mark Feature 0018 `in_progress` pending independent review. In the sequence
+   document, note implementation is awaiting review, leaving its checkbox open.
+   Do not begin Audita work or mark this feature done automatically.
+5. Confirm this feature changed no pipeline/collection/consumer pins. Inspect
+   existing changes without reverting unrelated work by the user or other tasks.
+
+### Validation and Completion Criteria
+
+Run `git diff --check` in the role and workspace. Check new local Markdown links
+and examples against the selected application contract. Reuse Stage 2 integration
+results for documentation-only changes; no second full suite or operational
+deployment is needed. The handoff names the role revision for later integration.
+
+### Commits
+
+Role, if documentation remains uncommitted:
+`Document Scriptorium v0.15 compatibility`.
+
+Workspace: `Record the standalone Scriptorium upgrade for review`.
+
+## Final Validation and Handoff
+
+Report owning commits, tests performed, and any deviations or blockers. Confirm
+the feature's working trees are clean. State explicitly that the existing suite
+compatibility set remains unchanged and that no operational deployment occurred.
+Independent review and closeout precede the next application adoption feature.
+
+## Progress Notes
+
+Roadmap and implementation plan approved by the user. Implementation has not
+started.
+
+## Open Questions
+
+None required for the approved standalone scope. Suite retry policy and later
+deployment choices belong to the final integration feature, not this role update.
--- a/docs/status.md
+++ b/docs/status.md
@@ -1,14 +1,15 @@
 # Workspace Status

-Last updated: 2026-09-22
+Last updated: 2026-09-28

 Current milestone: Operational D&D pipeline deployment

-Current focus: Select the next unit of work. Feature 0017 passed independent
-review with no remediation findings. The published development set selects `gpt-terra-latest` for production,
-`gpt-luna-latest` for testing, and Notarius v0.7.1. Consumer commit `0ced48c`
-pins collection commit `ce5b838` and the compatible Narratio and Notarius role
-commits. Feature 0016 is complete; deployment remains separate operator work.
+Current focus: Feature 0018 is approved and ready for its implementation sprint.
+Scriptorium adoption is first in the agreed
+[application upgrade sequence](roadmap/application-upgrade-sequence.md):
+Scriptorium → Audita → Notarius → Narratio → suite integration. Release triage
+is complete; detailed adoption work has not started. Retain the existing suite
+and consumer pins until coordinated integration. Feature 0017 is complete.

 | ID | Deliverable | Owner | State | Evidence or next step |
 | --- | --- | --- | --- | --- |
@@ -38,6 +39,7 @@ commits. Feature 0016 is complete; deployment remains separate operator work.
 | F0015 | Enable D&D S3 storage and publishing | ansible-role-dnd-pipeline, ansible-collection-dnd, and workspace | done | Pipeline `51566afcf69083cdbb5b5718f2544c86b079001c`, collection fixture `cf3de016ee09edf91afc3bc1addfd926d4a172ad`, collection squash `77f67adbc0f700e88bc15554e02052eaa22f1033`, and collection merge `46913446765e86ecfb29ebad9955d579ef6a4c7e` passed the complete repository suites and independent review with no remediation findings |
 | F0016 | Prepare the D&D deployment playbook for `pipeline-dnd` | rakestraw-ansible and workspace | done | Consumer `716a80e`, `12b2462`, and `7f8353d` passed independent review with no findings; public dependency installation, target-only host listing, syntax and whitespace checks passed. Follow-up `7f4c90a` pins collection `2a53979` for Narratio v2.1.2; collection build and syntax checks passed |
 | F0017 | Compose the Notarius testing overlay | Narratio role, pipeline role, collection, consumer, and workspace | done | Published through Narratio `330afe7`, Notarius `a82a83d`, pipeline `c56a942`, collection `ce5b838`, and consumer `0ced48c`; independent review found no remediation issues; validation evidence is in the feature roadmap |
+| F0018 | Adopt Scriptorium v0.15.0 | Scriptorium role and workspace | ready | [Approved roadmap](roadmap/features/0018-scriptorium-v0-15-adoption.md) and [staged plan](roadmap/implementation.md); standalone adoption only, suite pins unchanged |
 | A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` |
 | R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review |