diff --git a/docs/roadmap/upgrade-kit.md b/docs/roadmap/upgrade-kit.md new file mode 100644 index 0000000..5cc29c1 --- /dev/null +++ b/docs/roadmap/upgrade-kit.md @@ -0,0 +1,308 @@ +# Promptkit v0.11 Adoption + +## Status And Intent + +Accepted feature direction, recorded on 2026-09-26; implementation is pending. +Upgrade directly from Promptkit v0.9.0 to v0.11.0, including the intervening +v0.10.0 catalog transition. This document defines the desired end state and +the changes from the current baseline, not a staged coding-agent plan. + +The target is complete definition compatibility, controllable transient +retries, an optional whole-run deadline, and modest opt-in execution diagnostics. +This is a focused adapter and configuration update, not another architecture +migration. Optional diagnostics are included in the desired feature set but +need not block the core compatibility upgrade; they remain separately tracked +until delivered. “Optional” means callers opt in, not that they appear in +default output. + +[ADR 0004](../adr/0004-definition-boundary.md) and the +[architecture policy](../policy/architecture.md) govern scope. Promptkit remains +the parser, validator, execution engine, and owner of framework defaults. +Scriptorium remains a simple executable application for independent prompt +executions, with shared operational state where required for backend limits. + +## Baseline And Upstream Changes + +Scriptorium currently selects Promptkit v0.9.0 and the v1.0.0 backend catalog +modules. It already supports CLI execution, rendering, prompt/profile +inspection, HTTP execution, configured backends, and presence-aware execution +overrides. It does not expose transient retry settings, a whole-run deadline, +or execution observers. CLI execution currently supplies a background context. + +The upstream review covered the local tagged release v0.11.0 at commit +`db05b56166ca2d8650dfaf660e1cf0eed3932e1d`. Its canonical references are: + +- [v0.10.0 migration guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.10.0/docs/releases/v0.10.0.md); +- [v0.11.0 release notes](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/releases/v0.11.0.md); +- [definition formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/formats.md); +- [execution and retry integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/integrations/openai-compatible-chat.md); +- [consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/consumers/pkg-promptkit.md); and +- [observation declarations](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/observations.go). + +A diagnostic run using a temporary module file compiled Scriptorium against +v0.11.0. The existing test suite reported one failing profile-inspection case +that selects the removed `deepseek-4-flash` profile; other packages passed. +This establishes an initial migration issue, not proof of complete behavioral +compatibility or coverage of the new capabilities. The diagnostic did not +change the repository dependency pin. + +Neither intervening release adds declarative prompt/profile fields requiring +new adapter plumbing. The definition-facing changes concern catalog contents +and resolution precedence. Retry and observer APIs are execution controls; +their adoption is a deliberate case-by-case choice under ADR 0004. + +## Dependency And Definition Compatibility + +### Target End State + +Scriptorium uses tagged Promptkit v0.11.0 through its supported root package, +with the catalog versions selected by that release: + +- `promptkit-backend-openrouter/v2` v2.20260922.1; +- `promptkit-backend-rakestrawhome/v2` v2.20260922.0. + +Valid definitions for the selected Promptkit version remain usable through +Scriptorium's existing sources and adapters, subject to its established HTTP +artifact and deployment boundaries. Rendering and inspection remain offline +operations that do not generate model output. + +The maintained names `promptkit-openrouter-standard` and +`promptkit-openrouter-auto` work wherever a profile ID is accepted, including +prompt defaults, explicit selection, inheritance, and profile inspection. +They are opt-in choices, not new Scriptorium defaults. Ordinary custom-source +precedence remains intact. + +A stable maintained name is not a frozen model/settings selection. External +catalog entries can change or disappear across upgrades; inheriting from one +does not create a frozen copy. Complete application-owned profiles remain the +appropriate choice when an application owns its model and settings. + +### Required Changes + +- Update `go.mod` and `go.sum`, retaining tagged upstream dependencies and no + committed workspace, replacement, vendor tree, or direct catalog imports. +- Replace the obsolete built-in inspection test and add focused coverage for + the maintained names, prompt/default selection, inheritance, and overrides. + Preserve existing definition conformance coverage rather than rebuilding + Promptkit's parser tests in Scriptorium. +- Review maintained prompts, profile references, examples, and fixtures for + removed catalog IDs. The selected upstream catalog removes `deepseek-3-2`, + `deepseek-4-flash`, and `deepseek-4-pro`. Newer DeepSeek names are not automatic + compatibility aliases, and Scriptorium must not silently invent aliases. +- Explain migration options: choose an available catalog profile, choose a + maintained Promptkit name, or supply a complete application-owned definition. + Keep the existing application-owned example profiles where fixed settings + are the instructional intent. +- Audit dependency-boundary tests for the `/v2` module paths without allowing + direct catalog imports or duplicating catalog contents. + +## Controllable Transient Retries + +### Target End State + +Expose `transient_retries` in application configuration, with a CLI execution +override and an HTTP request override. Preserve value presence throughout: + +| Value | Meaning | +| --- | --- | +| Omitted | Inherit the engine setting, ultimately Promptkit's default. | +| Zero | Disable additional transient attempts. | +| Positive integer | Permit that many additional transient attempts. | +| Negative integer | Invalid configuration or request. | + +Adopt Promptkit's default of two additional attempts rather than silently +pinning the previous single-attempt behavior. Users must be able to select zero +explicitly. Request overrides take precedence over application configuration; +omission must not accidentally become an explicit zero. + +Promptkit alone owns eligible statuses, attempt execution, backoff, +`Retry-After`, and capacity coordination. Scriptorium adds no outer model retry +loop. Transient retries and structural repairs have distinct budgets: each +generation, including each repair generation, receives its transient budget. +Retries do not imply exactly-once generation or complete billing knowledge. + +### Required Changes + +- Extend strict application configuration and engine settings, mapping the + optional value to `promptkit.Config.TransientRetries` in shared engine + construction, including the server engine. +- Add presence-aware CLI and HTTP request mapping to + `promptkit.RunRequest.TransientRetries`. Keep this an execution setting, not + a copied prompt/profile schema field or provider model parameter. +- Validate malformed and negative values at the appropriate configuration or + request boundary, using Promptkit for framework semantics. +- Update CLI, HTTP, configuration, operations, and migration guidance to make + the default change, cost/latency implications, zero override, and independent + repair budget explicit. +- Protect inheritance and explicit-zero mapping with focused tests; include a + representative built-in HTTP retry path and verify that cancellation and + shared backend limits remain effective. Do not duplicate the entire upstream + status/backoff test matrix. + +## Optional Whole-Run Deadline + +### Target End State + +Callers can opt into a total execution deadline independently of the existing +generation timeout. No new default deadline is imposed. Existing timeout +controls must not silently change meaning: in v0.11.0 they apply per generation +attempt, whereas a whole-run deadline bounds the overall operation, including +preparation, capacity waiting, retry delays, attempts, and repairs as supported +by the upstream context contract. + +Expose this choice through CLI execution and HTTP execution. Derive any timeout +context from the caller's existing context, so a caller cancellation or earlier +deadline still wins. This is a cancellation boundary, not a guarantee of +instantaneous interruption of arbitrary callback or filesystem code. + +### Required Changes + +- Replace the CLI's unconditional background-context execution with a + context carrying the selected whole-run deadline when requested. +- Apply an HTTP execution deadline as a child of the incoming request context, + retaining disconnect cancellation and existing server lifecycle behavior. +- Keep total-run duration handling separate from per-attempt timeout conversion; + do not reuse names or fields that would reinterpret existing requests. +- Document that provider retry timing can exceed local backoff limits and that + per-attempt timeouts do not cap the complete run. +- Test deadline/cancellation propagation through a representative retry wait + and capacity wait, without adding another admission queue or retry engine. + +Exact option names, units, and zero-value semantics belong in the subsequent +implementation plan and must be settled before coding this interface. + +## Opt-In Execution Reports + +### Target End State + +Provide a modest terminal execution report for CLI and HTTP callers who opt +in. Capture selected metadata through `RunRequest.Observer`, including attempt, +retry, repair, timing, outcome, and presence-aware usage information. Reports +should be available for completed successful and failed execution invocations +when Promptkit supplies terminal observations. Do not invent execution data for +failures that occur before invocation. + +Keep generated CLI content on stdout or the existing output path. Execution +reports use a separate destination. Ordinary CLI output and HTTP response +shapes remain unchanged when reporting is not requested. + +The report has a Scriptorium-owned schema rather than serializing upstream +events directly: `ExecutionEvent` has no stable JSON contract. Preserve unknown +usage versus reported zero, and incomplete versus complete observed totals. +Existing successful-result usage semantics remain unchanged. A terminal total +is an aggregate, not another attempt to add to the bill; no report is a billing +ledger. + +### Required Changes + +- Add an opt-in reporting interface and explicit application-owned report DTOs. +- Collect per-request observations without shared mutable report state, durable + storage, or blocking work inside synchronous callbacks. Shared operational + infrastructure, if used, must remain concurrency-safe. +- Format a concise terminal report rather than introducing event streaming, + an event archive, or a general telemetry framework. +- Define success and failure response/report handling without manufacturing a + partial `RunResult` when Promptkit returns an operational error. +- Exclude prompt and response bodies, credentials, headers, endpoint URLs, + provider diagnostic text, and raw errors. Treat selected identifiers as + potentially sensitive even when the upstream event is content-free. +- Test omission by default, success/failure reporting, unknown versus known-zero + usage, incomplete totals, no aggregate double counting, and concurrent-request + isolation at the adapter/report boundary. + +The report destination, wire fields, report-write failure handling, and precise +scope of reported metadata remain implementation-plan decisions. Full event +streaming and durable history are explicitly outside this feature. + +## Errors And Compatibility + +Preserve existing CLI exit classifications and HTTP `502 / llm_failed` +classification for generation failures. Keep safe public messages rather than +exposing provider error bodies or parsing diagnostic strings. + +Promptkit adds `ErrTransport`, `ErrMalformedResponse`, `ErrRetriesExhausted`, and +`RetryExhaustedError`. Use their public identities to support optional structured +failure details in execution diagnostics, including actual attempts where +available. Retry exhaustion and the final cause are distinct facts; do not lose +the underlying failure category by flattening them into one label. + +Review mapping precedence and add representative wrapped-error tests while +retaining existing capacity and validation behavior. Specify total-deadline +outcomes in the implementation plan rather than accidentally treating every +context error as an unclassified failure. No breaking replacement of existing +error codes is required by this upgrade. + +## Licensing And Distribution Review + +Promptkit v0.11.0 changes from GPLv3 to LGPL-3.0-or-later. Scriptorium's current +binary workflow produces statically linked Go executables, and upstream's +[license summary](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/README.md#license) +explicitly calls out distribution considerations for that model. Catalog and +other dependency licenses remain separate. + +Before distributing the upgraded binaries, resolve the applicable notices, +license texts, source availability, and any required materials or permissions +for recombination with a modified library. Record the chosen distribution +approach and make any corresponding packaging/documentation changes. This +roadmap is not a legal determination or a complete compliance checklist. + +Do not automatically relicense Scriptorium, assume that library source alone +settles every distribution requirement, or alter historical release contents. +Licensing review is a concrete adoption prerequisite, not a new blanket manual +approval gate for every future release. + +## Documentation And Delivery + +Update current documentation to link to the selected v0.11.0 contracts when +the implementation lands. CLI, HTTP, configuration, and operations documents +own their respective new behavior; internal docs explain adapter mapping; +release notes summarize compatibility impact. Preserve historical release notes +and ADR decision content. Revise current examples and migration guidance where +needed rather than globally replacing version strings in historical material. + +Deliver the upgrade as a minor Scriptorium release, not a compatibility patch. +Prominently explain removed catalog profiles, changed retry behavior, the +explicit single-attempt option, and timeout layering. Optional diagnostics may +follow in a separate feature release without representing unfinished reports +as already available. + +Use the existing [release policy](../policy/release.md) and +[runbook](../release.md). This roadmap authorizes no publication. No new ADR is +needed unless a later decision changes the established product or compatibility +boundaries. + +## Completion Criteria + +- The tagged dependency and selected catalog modules are adopted without + unsupported imports or local dependency substitutions. +- Existing definition conformance tests and focused maintained-profile tests + pass; removed profile IDs have explicit migration guidance, not silent aliases. +- Retry settings preserve omission and zero across configuration, CLI, and HTTP; + Promptkit remains the sole retry/capacity implementation. +- Optional total deadlines bound the caller operation without redefining + per-attempt timeouts or disabling parent cancellation. +- Opt-in terminal reports provide safe, isolated, presence-aware execution + metadata without changing default output or creating durable workflow state. + If deferred to a follow-up release, this item remains visibly incomplete. +- Existing error classifications remain compatible, with deliberate handling + of the new public identities and context outcomes. +- The licensing/distribution approach is resolved before upgraded binary + distribution, and any required materials are included in the chosen workflow. +- Affected contracts, examples, migration guidance, and release notes match + implemented behavior. Relevant tests remain offline and credential-free, + and normal development and release validation pass. + +## Decisions To Resolve In The Implementation Plan + +- Exact placement/naming of retry overrides and which preparation-only CLI + options should accept them for consistency without implying execution. +- Whole-run deadline option names, units, zero handling, and external deadline + error presentation; omission must impose no new deadline. +- Report destinations and opt-in request shape, report DTO fields, failure + reporting, and report-output error behavior; defaults must remain unchanged. +- Whether reports ship with the core upgrade or in a clearly tracked follow-up. +- The concrete licensing/distribution approach and any resulting packaging work. + +These are bounded interface and delivery decisions, not permission to expand +into conversations, follow-up messages, application-owned semantic retry loops, +prepared-handle coordination, or multi-step workflows.