From 635674c066d0b432187f96e72b9c03d085443ac5 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 12 Oct 2026 20:17:52 +0000 Subject: [PATCH] Prepare v0.15.0 release documentation --- README.md | 3 + docs/releases/v0.15.0.md | 34 ++++++ docs/roadmap/implementation.md | 195 --------------------------------- docs/roadmap/remediation.md | 50 --------- docs/roadmap/validate.md | 168 ---------------------------- 5 files changed, 37 insertions(+), 413 deletions(-) create mode 100644 docs/releases/v0.15.0.md delete mode 100644 docs/roadmap/implementation.md delete mode 100644 docs/roadmap/remediation.md delete mode 100644 docs/roadmap/validate.md diff --git a/README.md b/README.md index 44496f7..8516215 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,9 @@ boundary and constraints that framework work must preserve. ## Release Guidance +The [v0.15.0 release notes](docs/releases/v0.15.0.md) cover error-only +validation of concrete requests with `Engine.ValidateRequest`. + Consumers upgrading to `v0.12.0` should read the [v0.12.0 release notes](docs/releases/v0.12.0.md) for the intentional ConfigKit configuration and public construction API migration. diff --git a/docs/releases/v0.15.0.md b/docs/releases/v0.15.0.md new file mode 100644 index 0000000..c01bd2b --- /dev/null +++ b/docs/releases/v0.15.0.md @@ -0,0 +1,34 @@ +# Promptkit v0.15.0 + +This release adds error-only validation of a concrete prompt request without +calling a model. The minimum supported Go version remains 1.26.0. + +## Validate A Concrete Request + +`Engine.ValidateRequest` accepts an ordinary `RunRequest` and returns nil when +it passes preparation, or the existing preparation error. Consumers can use it +for fail-fast checks, including downstream CLI validation commands, without +receiving a prepared snapshot. + +The method shares `Prepare`'s request conversion and preparation path. Unlike +requirements inspection, it checks the actual supplied values and bindings. +Artifact and schema resource I/O still occurs; success does not certify provider +acceptance or later execution success. A subsequent run prepares independently. +Validation is optional, not a required step before preparation or execution. + +See the [request validation guide](../consumers/pkg-promptkit.md#validate-a-concrete-request) +and [GoDoc](../../engine.go) for usage and the exact contract. Consumers that +need execution tied to a checked snapshot can use the existing +[prepared execution workflow](../consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later). + +## Compatibility And Upgrade + +No application-code, configuration, or Go-toolchain migration is required from +v0.14.0. This is an optional API addition; existing inspection, preparation, and +execution workflows remain available. + +```sh +go get gitea.maximumdirect.net/eric/promptkit@v0.15.0 +``` + +[COPYRIGHT](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.15.0/COPYRIGHT) | [LICENSE](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.15.0/LICENSE) | [ATTRIBUTION](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.15.0/docs/oss/attribution.md) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 5e05e95..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,195 +0,0 @@ -# Request Validation Implementation Plan - -Implement [the finalized feature roadmap](validate.md) in the numbered order -below. Each stage is scoped for one Luna coding-agent prompt. The feature -roadmap owns the target behavior and downstream intent; this document owns the -implementation sequence. - -## Baseline And Implementation Decisions - -The baseline is commit `0c3651f`. `Engine.Prepare` in `engine.go` clears execution -observation state, checks the engine, converts `RunRequest`, calls -`internal/engine.Prepare`, maps preparation errors, and converts the returned -internal value into a caller-owned public snapshot. The internal engine already -delegates to `internal/usecase.Runner.Prepare`, which runs both preparation -phases without admission or generation. - -Add exactly this public operation: - -```go -func (e *Engine) ValidateRequest(ctx context.Context, req RunRequest) error -``` - -Extract the existing facade preparation plumbing into an unexported -`(*Engine).prepareRequest` helper returning `(*domain.PreparedRun, error)`. -`Prepare` will call that helper and retain its existing public result conversion; -`ValidateRequest` will call it and discard the internal result. This shares -request conversion, context handling, nil-engine checks, and error mapping while -avoiding public snapshot copies for validation. The private helper is an -implementation choice, not a consumer contract. - -No new internal-engine or runner operation is needed. Internal prepared values, -hashing, schema preparation, and rendering may still occur; eliminating them -would change the selected preparation-parity behavior or require work outside -this small addition. - -## Working Instructions - -- Start with [the development guide](../development.md) and read the applicable - [agent](../policy/agents.md), [architecture](../policy/architecture.md), - [testing](../policy/testing.md), and [documentation](../policy/documentation.md) - policies. Use the existing facade GoDoc and - [runner description](../internal/runner.md#shared-preparation-pipeline) as - current-behavior references. -- Preserve `Prepare`'s behavior and error wrapping, including its existing - context classification. Do not add a general `errors.Is(err, ctx.Err())` - guarantee, new error sentinels, aggregated diagnostics, stricter binding - rules, provider checks, or capacity checks. -- Use the full `RunRequest` conversion path, not a separate subset of its fields - or a dependency inventory from `InspectRequirements`. Leave credential, - resource, template, and request-format rules unchanged. -- Keep tests offline, deterministic, and external-package at the public boundary. - Use real preparation collaborators and small injected doubles for external - effects. Test representative forwarding risks rather than reproducing the - complete preparation suite. -- Commit each completed stage after its checks and relevant diff review, with a - plain-English message. Do not push, tag, or run a publication command as part - of this plan. - -## Stage 1: Share Facade Preparation And Add ValidateRequest - -Scope: the small facade change, GoDoc, and basic contract tests. - -1. Review `Engine.Prepare`, `toDomainRunRequest`, `mapPublicError`, and - `fromDomainPreparedRun`. Extract only the existing preparation plumbing into - the private helper described above. Keep the existing operation order and - wrapping at each error boundary; do not change error semantics as incidental - cleanup. -2. Make `Prepare` call the helper and perform the existing - `fromDomainPreparedRun` conversion on success. Its signature, snapshot - ownership, and failure behavior remain unchanged. -3. Add `ValidateRequest` in `engine.go`. Return the helper's error, discarding its - internal prepared value. Do not call the public `Prepare` method, construct a - public snapshot, or introduce an internal validation pipeline. -4. Add GoDoc explaining that nil means the concrete request passed preparation - now, no LLM call occurs, resource I/O still occurs, and no executable snapshot - is retained. State parity with preparation's errors and context handling, - including the nil-engine `ErrInvalidConfig` convention. Link conceptually to - `Prepare` and `PrepareExecution`; do not promise subsequent provider success. -5. Add `validate_request_contract_test.go` in package `promptkit_test`. Reuse - compatible helpers from `requirements_inspection_contract_test.go` and - `consumer_test_helpers_test.go` where helpful. Construct a synthetic prompt - and explicit offline profile rather than depending on catalog model choices. - Cover a successful request, a nil engine, a conversion-level invalid request - such as an invalid appended-message role, and one missing required input. -6. In those tests, invoke both `Prepare` and `ValidateRequest` for the same - test-controlled request and compare success or the relevant public sentinel - through `errors.Is`. Use one intentional failure per case rather than - expecting a particular error from several independently failing inputs. - Inject a counting client and verify neither operation generates a model call. -7. Format changed Go files and run - `go test . ./internal/usecase ./internal/prompt`, followed by - `git diff --check`. Existing `Prepare` tests must still pass. - -Completion: external consumers can call the new error-only method, all facade -preparation plumbing is shared, and the public snapshot conversion remains only -on `Prepare`'s success path. - -**Completed: 2026-10-12.** - -## Stage 2: Verify Concrete-Request Parity And Resource Boundaries - -Scope: focused tests of the finalized roadmap's clarifications. - -1. Extend the root contract tests with a prompt using - `{{if eq .mode "detailed"}}{{input "details"}}{{end}}`. For an input not - declared required, verify `mode=brief` succeeds without it and - `mode=detailed` fails without it but succeeds with an inline binding. Use a - second definition with the input declared required to verify absence fails - even on the brief branch. Compare each case with `Prepare`. -2. Cover an evaluated direct variable lookup with an absent key and with an - explicitly supplied empty string. Include a missing `index` lookup to show - its existing optional semantics are retained. These assertions concern - preparation only, not nonempty model-output rules. An extra unrelated caller - variable must remain acceptable where `Prepare` accepts it. -3. Add a case with an unreadable supplied input that is not referenced on the - selected branch. A synthetic missing file under `t.TempDir()` is sufficient. - Both operations must report the applicable artifact-load error, proving - validation uses actual preparation instead of a names-only or lazy subset. -4. Cover representative selected-profile and output-schema preparation failures - with existing valid fixtures: a missing profile should retain - `ErrProfileNotFound`, and an unavailable selected schema should retain the - category already returned by `Prepare`. Avoid adding source-level or - intermediate-configuration validation. -5. Add one successful request with a profile default and a runtime override that - completes a partial profile, plus a valid direct session ID and appended - message where useful. Inspect `Prepare`'s result only as needed to establish - that the fixture exercised those settings; `ValidateRequest` must accept the - same request. Existing tests own the exhaustive override and message matrix. -6. Cover a controlled canceled-context or canceled-loader failure through the - public facade, asserting the existing preparation category for both calls. - Do not assert a stronger universal relationship to `context.Canceled` or - introduce timing-based tests. Keep the model-call count at zero across the - validation cases. -7. Run `go test -race . ./internal/usecase ./internal/prompt` and - `git diff --check`, formatting changed Go files. Review tests for redundant - preparation coverage and assertions on incidental error text or private call - order. - -Completion: success/failure parity includes concrete branch choices, declared -required inputs, variable presence, all supplied artifact bindings, representative -resource/selection errors, and ordinary request forwarding. No provider or -capacity-readiness interpretation has been introduced. - -**Completed: 2026-10-12.** - -## Stage 3: Document Consumer Use And Verify The Integrated Feature - -Scope: canonical documentation, the maintained preparation example, and final -validation. - -1. Add a small `ValidateRequest` example to - `docs/consumers/pkg-promptkit.md`, next to requirements inspection and offline - preparation. Show a full `RunRequest` and nil/error handling, and link to - GoDoc for exact behavior. Distinguish concrete-request validation from - dependency discovery and from obtaining a prepared execution handle. -2. Explain that Scriptorium can map the error to its own diagnostics and exit - status, and should build equivalent requests for its `validate` and `run` - commands. Keep this as consumer guidance, not CLI code or a new PromptKit - transport contract. Explicitly note that separate subprocess calls prepare - independently and cannot transfer the opaque handle. -3. Update `examples/go-library/prepare/main.go` to demonstrate the method using - the same request value as its existing `Prepare` demonstration. Keep its - semantic output unchanged. Explain in the consumer guide that showing both - operations is illustrative, not a requirement to validate before `Prepare` - or `Run`; it performs preparation twice. -4. Update `docs/internal/runner.md` to describe the error-only facade's reuse of - the existing preparation path, and update `docs/internal/overview.md` to - include the implemented public operation. Keep business logic under the - existing internal ownership and avoid describing a new validator component. -5. Ensure GoDoc and guidance state the same boundaries: resource I/O and cost - remain, all supplied inputs follow existing loading behavior, only the - concrete invocation is checked, errors are not aggregated, and no future - provider or execution success is certified. Do not change format references - to create new requirements. -6. `v0.14.0` is already tagged at planning time; leave that historical note - unchanged. Record this optional API addition in the next release notes when - that release is prepared. Release publication and new version selection are - separate from this implementation plan. -7. Run `go test -race ./...`, `go vet ./...`, - `python3 scripts/check-examples.py`, and `git diff --check`. Check affected - documentation links and fragments. Review exported declarations to confirm - that only `Engine.ValidateRequest` was added, and review the complete diff - against [the feature roadmap](validate.md). - -Completion: API, focused tests, canonical documentation, and the maintained -example are coherent; existing preparation and execution workflows remain -available; and the integrated checks pass. Report actual check results and stage -commits. The completed roadmap can be retired during later release preparation. - -**Completed: 2026-10-12.** - -## Open Questions - -None. The finalized roadmap resolves the public name, request shape, result, -preparation scope, and downstream boundary required for these stages. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md deleted file mode 100644 index 1a532f2..0000000 --- a/docs/roadmap/remediation.md +++ /dev/null @@ -1,50 +0,0 @@ -# Request Validation Review Findings - -Reviewed commits `f315620`, `f3148bf`, and `60784e7` against the finalized -[feature roadmap](validate.md) and [implementation plan](implementation.md). - -## 1. Reuse The Existing Preparation-Parity Test Helper - -Priority: P3 (test maintainability). Status: complete. - -At reviewed commit `60784e7`, in -[validate_request_contract_test.go](../../validate_request_contract_test.go), -`TestValidateRequestMatchesPrepareContract` defines a local `testParity` closure -at lines 25–45 that duplicates the package-level -`assertValidateRequestParity` helper at lines 87–108. Both invoke `Prepare` and -`ValidateRequest` and assert the same success/result and public-error behavior; -the package-level helper additionally returns the successful prepared value. -The nil-engine subtest repeats the same failure assertions again. - -Retaining separate copies makes future parity-assertion improvements require -multiple edits and increases the risk that related tests diverge unnecessarily. -This is a small cleanup, not a demonstrated functional or release-blocking bug. - -### Requested Remediation - -- Replace calls to the local closure with `assertValidateRequestParity` and - remove the redundant closure. -- Reuse that helper for the nil-engine subtest as well; retain its - `ErrInvalidConfig` expectation. -- Preserve all existing cases, public-error assertions, and no-model-call - checks. No production or public-contract changes are needed. - -Verify with `go test -race . -run '^TestValidateRequest'` and -`git diff --check`, formatting the changed Go file. - -### Completion - -Removed the local closure and reused `assertValidateRequestParity` for all four -cases, including the nil-engine `ErrInvalidConfig` expectation. Existing cases, -public-error assertions, and no-model-call checks are preserved; production code -and public contracts are unchanged. - -Formatted the Go file. The uncached, race-enabled `TestValidateRequest` tests -and `git diff --check` passed. - -## Review Outcome - -The public method and shared facade helper match the intended architecture. -No implementation gaps or unintended strengthening of contracts were found. -The uncached full race-enabled suite, vet, maintained consumer examples, and -implementation diff whitespace checks passed during review. diff --git a/docs/roadmap/validate.md b/docs/roadmap/validate.md deleted file mode 100644 index e9d579e..0000000 --- a/docs/roadmap/validate.md +++ /dev/null @@ -1,168 +0,0 @@ -# Request Preflight Validation - -Status: accepted; implementation pending. - -## Purpose - -Provide an explicit success/error operation for consumers that want to check an -actual prompt request without generating a model response or receiving a -prepared snapshot. - -The check applies to concrete variable values and input bindings for one -invocation. It does not certify that an application's configured bindings cover -every possible request or template branch. - -Scriptorium motivates this addition: applications invoke it as a subprocess and -could use a `validate` command to fail early when their supplied variables, -inputs, or other request settings cannot pass preparation. PromptKit owns the -reusable request operation; Scriptorium owns command syntax, diagnostics, -process exit statuses, and any application-specific interpretation. - -## Current Baseline - -`Engine.InspectRequirements` discovers recognizable input and caller-variable -dependencies without loading artifacts or rendering messages. Its `Complete` -flag describes discovery coverage, not whether supplied dependencies satisfy a -request. It is useful for authoring and planning, but is not actual preflight. - -`Engine.Prepare` already performs preflight through request conversion and the -shared internal preparation pipeline. It resolves the prompt, profile, effective -settings, and output contract; loads input artifacts and applicable schema -resources; and renders messages and the session ID without calling an LLM. -The facade then converts the internal prepared value into a caller-owned -`PreparedRun`. Consumers can discard that result today, but the API does not -directly express a success/error-only operation. - -See the [consumer inspection and preparation guide](../consumers/pkg-promptkit.md#inspect-requirements-before-preparation), -[existing preparation GoDoc](../../engine.go), and -[internal preparation flow](../internal/runner.md#shared-preparation-pipeline). - -## Target Public API - -Add a separate method on the existing engine: - -```go -func (e *Engine) ValidateRequest(ctx context.Context, req RunRequest) error -``` - -`ValidateRequest` names the object being checked and distinguishes this operation -from generated-output validation. `validate` remains the recommended downstream -CLI command name; preflight describes the workflow, without implying provider -or capacity readiness checks. - -Use the complete existing `RunRequest`, including input bindings, caller -variables, profile selection, execution overrides, session settings, output -contract overrides, and appended messages. Apply the same request conversion -and preparation rules as `Prepare`; validation is not a name-only check or an -analysis of declared requirements. - -A nil error means **the supplied request passed preparation at the time of the -check**. A failure returns the error encountered through the existing -preparation path, with the same public error categories and applicable error -identities. A nil engine follows `Prepare`'s `ErrInvalidConfig` convention. -Context handling follows preparation rather than introducing a stronger -cancellation guarantee. The result does not aggregate all missing requirements -or return a prepared value, execution handle, or model output. - -Keep `Prepare` unchanged. A separate method avoids a mode flag that changes the -meaning of its returned snapshot. The intended distinction is: - -- `InspectRequirements`: discover what a prompt may require for a supplied - context, including partial discovery where appropriate. -- `ValidateRequest`: determine whether the actual supplied request passes - preparation. -- `Prepare`: perform preparation and return its snapshot. -- `PrepareExecution`: prepare an owned snapshot that can later be executed. -- `Run`: prepare and execute the request. - -## Meaning And Limits - -Validation exercises actual preparation, including loading inputs and rendering -the branches selected by supplied values. It applies existing rules to missing -bindings and variables rather than adding stricter required-field rules, -rejecting additional caller keys, or restricting supported templates. - -For example, consider a message containing -`{{if eq .mode "detailed"}}{{input "details"}}{{end}}`, where `details` is not -declared required. A request with `mode=brief` can prepare without that input; -one with `mode=detailed` needs it. Declaring the input required makes it -necessary regardless of the selected branch. Similarly, an evaluated -`{{.audience}}` fails when the key is absent, but supplying an empty string does -not itself fail preparation. Optional lookups such as `{{index . "audience"}}` -retain their existing semantics. These examples illustrate preparation parity, -not new required-field rules; see the [input and template formats](../formats.md#inputs) -and [requirement classifications](../../requirements.go). - -No LLM request is made. This is not a promise of zero I/O: artifact and schema -loaders still perform their existing resource work, including caller-supplied -loader behavior. The current preparation path loads every supplied input, even -if the selected template branch does not use it. Avoiding inference makes the -operation useful for fail-fast workflows, but large inputs or resource -processing can still cost time and memory. No fixed latency or allocation -guarantee is proposed. - -Success does not establish provider acceptance, context-window fit, generated -output validity, or application-domain correctness. It also does not reserve -execution capacity or retain a snapshot for a subsequent invocation. Later -requests can differ, and referenced resources can change. Consumers that need -execution tied to the checked snapshot within one process and engine should use -`PrepareExecution`. Its [opaque handle](../../prepared_execution.go) cannot -connect separate subprocess invocations. - -This is request-scoped validation using an already constructed engine. It does -not expand configuration ownership beyond the final handoff described by -[ADR 0001](../adr/0001-configuration-boundary.md), or introduce eager validation -of all configured prompts and resources. - -## Downstream CLI Guidance - -Scriptorium's proposed `validate` and existing `run` commands should share -request construction: argument interpretation, defaults, profile selection, -and overrides must describe the same request for equivalent caller inputs. -This allows PromptKit's preparation parity to carry through the CLI boundary. -Scriptorium owns exit statuses and diagnostic presentation. - -A `scriptorium validate` call followed by `scriptorium run` performs two -independent preparations; the first does not retain an executable snapshot for -the second. When assessing CLI latency, measure the complete subprocess call, -including process startup and engine construction, as well as preparation. -Avoiding the public prepared snapshot does not eliminate the other preparation -work. These are downstream integration recommendations, not Scriptorium changes -included in this unit of work. - -## Work Needed - -- Add the public error-only method through the existing facade and internal - preparation path. Reuse request conversion, preparation, and error mapping - rather than implementing a second validation engine or an inspection-based - approximation. -- Discard the internal prepared result without converting it into a public - snapshot. The existing internal prepared value may still be constructed; - removing all preparation allocations is not required for this addition. - Share small facade plumbing where helpful without a broad refactor. -- Preserve preparation's existing context, credential, resource, and execution - separation behavior. No model generation, output-repair loop, or capacity - admission is needed for the new operation. -- Add focused external-package tests for successful preparation and - representative missing-variable/input, resource, selection, and invalid-request - failures. Use existing fixtures where suitable to verify parity with - `Prepare`, including a branch-sensitive example and the nil-engine boundary. - Check public error identities rather than exact messages, and confirm that - the injected model client is not called. Avoid duplicating the full - preparation test suite. -- Document the method in GoDoc and add a short consumer example distinguishing - validation from requirement discovery and prepared execution. Update the - internal runner description where needed when implementation lands, and - record the optional API addition in its eventual release notes. - -## Completion And Scope - -The feature is complete when consumers can submit an ordinary `RunRequest` to -`Engine.ValidateRequest` and obtain preparation success or its existing failure -without a model call or public prepared snapshot, with focused tests and -canonical documentation establishing that behavior. - -This unit of work does not include changes to Scriptorium, CLI commands or exit -codes, aggregated diagnostics, presence-only validation modes, provider probes, -domain-aware validation, or new execution guarantees. It introduces no changes -to prompt/profile formats or existing preparation and execution semantics.