diff --git a/docs/roadmap/validate.md b/docs/roadmap/validate.md index 18431f1..e9d579e 100644 --- a/docs/roadmap/validate.md +++ b/docs/roadmap/validate.md @@ -8,6 +8,10 @@ Provide an explicit success/error operation for consumers that want to check an actual prompt request without generating a model response or receiving a prepared snapshot. +The check applies to concrete variable values and input bindings for one +invocation. It does not certify that an application's configured bindings cover +every possible request or template branch. + Scriptorium motivates this addition: applications invoke it as a subprocess and could use a `validate` command to fail early when their supplied variables, inputs, or other request settings cannot pass preparation. PromptKit owns the @@ -38,9 +42,14 @@ See the [consumer inspection and preparation guide](../consumers/pkg-promptkit.m Add a separate method on the existing engine: ```go -func (e *Engine) Validate(ctx context.Context, req RunRequest) error +func (e *Engine) ValidateRequest(ctx context.Context, req RunRequest) error ``` +`ValidateRequest` names the object being checked and distinguishes this operation +from generated-output validation. `validate` remains the recommended downstream +CLI command name; preflight describes the workflow, without implying provider +or capacity readiness checks. + Use the complete existing `RunRequest`, including input bindings, caller variables, profile selection, execution overrides, session settings, output contract overrides, and appended messages. Apply the same request conversion @@ -60,7 +69,8 @@ meaning of its returned snapshot. The intended distinction is: - `InspectRequirements`: discover what a prompt may require for a supplied context, including partial discovery where appropriate. -- `Validate`: determine whether the actual supplied request passes preparation. +- `ValidateRequest`: determine whether the actual supplied request passes + preparation. - `Prepare`: perform preparation and return its snapshot. - `PrepareExecution`: prepare an owned snapshot that can later be executed. - `Run`: prepare and execute the request. @@ -72,23 +82,54 @@ the branches selected by supplied values. It applies existing rules to missing bindings and variables rather than adding stricter required-field rules, rejecting additional caller keys, or restricting supported templates. +For example, consider a message containing +`{{if eq .mode "detailed"}}{{input "details"}}{{end}}`, where `details` is not +declared required. A request with `mode=brief` can prepare without that input; +one with `mode=detailed` needs it. Declaring the input required makes it +necessary regardless of the selected branch. Similarly, an evaluated +`{{.audience}}` fails when the key is absent, but supplying an empty string does +not itself fail preparation. Optional lookups such as `{{index . "audience"}}` +retain their existing semantics. These examples illustrate preparation parity, +not new required-field rules; see the [input and template formats](../formats.md#inputs) +and [requirement classifications](../../requirements.go). + No LLM request is made. This is not a promise of zero I/O: artifact and schema loaders still perform their existing resource work, including caller-supplied -loader behavior. Avoiding inference makes the operation useful for fail-fast -workflows, but large inputs or resource processing can still cost time and -memory. No fixed latency or allocation guarantee is proposed. +loader behavior. The current preparation path loads every supplied input, even +if the selected template branch does not use it. Avoiding inference makes the +operation useful for fail-fast workflows, but large inputs or resource +processing can still cost time and memory. No fixed latency or allocation +guarantee is proposed. Success does not establish provider acceptance, context-window fit, generated output validity, or application-domain correctness. It also does not reserve execution capacity or retain a snapshot for a subsequent invocation. Later requests can differ, and referenced resources can change. Consumers that need -execution tied to the checked snapshot should use `PrepareExecution`. +execution tied to the checked snapshot within one process and engine should use +`PrepareExecution`. Its [opaque handle](../../prepared_execution.go) cannot +connect separate subprocess invocations. This is request-scoped validation using an already constructed engine. It does not expand configuration ownership beyond the final handoff described by [ADR 0001](../adr/0001-configuration-boundary.md), or introduce eager validation of all configured prompts and resources. +## Downstream CLI Guidance + +Scriptorium's proposed `validate` and existing `run` commands should share +request construction: argument interpretation, defaults, profile selection, +and overrides must describe the same request for equivalent caller inputs. +This allows PromptKit's preparation parity to carry through the CLI boundary. +Scriptorium owns exit statuses and diagnostic presentation. + +A `scriptorium validate` call followed by `scriptorium run` performs two +independent preparations; the first does not retain an executable snapshot for +the second. When assessing CLI latency, measure the complete subprocess call, +including process startup and engine construction, as well as preparation. +Avoiding the public prepared snapshot does not eliminate the other preparation +work. These are downstream integration recommendations, not Scriptorium changes +included in this unit of work. + ## Work Needed - Add the public error-only method through the existing facade and internal @@ -117,9 +158,9 @@ of all configured prompts and resources. ## Completion And Scope The feature is complete when consumers can submit an ordinary `RunRequest` to -`Engine.Validate` and obtain preparation success or its existing failure without -a model call or public prepared snapshot, with focused tests and canonical -documentation establishing that behavior. +`Engine.ValidateRequest` and obtain preparation success or its existing failure +without a model call or public prepared snapshot, with focused tests and +canonical documentation establishing that behavior. This unit of work does not include changes to Scriptorium, CLI commands or exit codes, aggregated diagnostics, presence-only validation modes, provider probes,