Clarify request validation roadmap and CLI guidance

This commit is contained in:
2026-10-12 17:26:50 +00:00
parent 5a5d47d36e
commit 0c3651fceb

View File

@@ -8,6 +8,10 @@ Provide an explicit success/error operation for consumers that want to check an
actual prompt request without generating a model response or receiving a
prepared snapshot.
The check applies to concrete variable values and input bindings for one
invocation. It does not certify that an application's configured bindings cover
every possible request or template branch.
Scriptorium motivates this addition: applications invoke it as a subprocess and
could use a `validate` command to fail early when their supplied variables,
inputs, or other request settings cannot pass preparation. PromptKit owns the
@@ -38,9 +42,14 @@ See the [consumer inspection and preparation guide](../consumers/pkg-promptkit.m
Add a separate method on the existing engine:
```go
func (e *Engine) Validate(ctx context.Context, req RunRequest) error
func (e *Engine) ValidateRequest(ctx context.Context, req RunRequest) error
```
`ValidateRequest` names the object being checked and distinguishes this operation
from generated-output validation. `validate` remains the recommended downstream
CLI command name; preflight describes the workflow, without implying provider
or capacity readiness checks.
Use the complete existing `RunRequest`, including input bindings, caller
variables, profile selection, execution overrides, session settings, output
contract overrides, and appended messages. Apply the same request conversion
@@ -60,7 +69,8 @@ meaning of its returned snapshot. The intended distinction is:
- `InspectRequirements`: discover what a prompt may require for a supplied
context, including partial discovery where appropriate.
- `Validate`: determine whether the actual supplied request passes preparation.
- `ValidateRequest`: determine whether the actual supplied request passes
preparation.
- `Prepare`: perform preparation and return its snapshot.
- `PrepareExecution`: prepare an owned snapshot that can later be executed.
- `Run`: prepare and execute the request.
@@ -72,23 +82,54 @@ the branches selected by supplied values. It applies existing rules to missing
bindings and variables rather than adding stricter required-field rules,
rejecting additional caller keys, or restricting supported templates.
For example, consider a message containing
`{{if eq .mode "detailed"}}{{input "details"}}{{end}}`, where `details` is not
declared required. A request with `mode=brief` can prepare without that input;
one with `mode=detailed` needs it. Declaring the input required makes it
necessary regardless of the selected branch. Similarly, an evaluated
`{{.audience}}` fails when the key is absent, but supplying an empty string does
not itself fail preparation. Optional lookups such as `{{index . "audience"}}`
retain their existing semantics. These examples illustrate preparation parity,
not new required-field rules; see the [input and template formats](../formats.md#inputs)
and [requirement classifications](../../requirements.go).
No LLM request is made. This is not a promise of zero I/O: artifact and schema
loaders still perform their existing resource work, including caller-supplied
loader behavior. Avoiding inference makes the operation useful for fail-fast
workflows, but large inputs or resource processing can still cost time and
memory. No fixed latency or allocation guarantee is proposed.
loader behavior. The current preparation path loads every supplied input, even
if the selected template branch does not use it. Avoiding inference makes the
operation useful for fail-fast workflows, but large inputs or resource
processing can still cost time and memory. No fixed latency or allocation
guarantee is proposed.
Success does not establish provider acceptance, context-window fit, generated
output validity, or application-domain correctness. It also does not reserve
execution capacity or retain a snapshot for a subsequent invocation. Later
requests can differ, and referenced resources can change. Consumers that need
execution tied to the checked snapshot should use `PrepareExecution`.
execution tied to the checked snapshot within one process and engine should use
`PrepareExecution`. Its [opaque handle](../../prepared_execution.go) cannot
connect separate subprocess invocations.
This is request-scoped validation using an already constructed engine. It does
not expand configuration ownership beyond the final handoff described by
[ADR 0001](../adr/0001-configuration-boundary.md), or introduce eager validation
of all configured prompts and resources.
## Downstream CLI Guidance
Scriptorium's proposed `validate` and existing `run` commands should share
request construction: argument interpretation, defaults, profile selection,
and overrides must describe the same request for equivalent caller inputs.
This allows PromptKit's preparation parity to carry through the CLI boundary.
Scriptorium owns exit statuses and diagnostic presentation.
A `scriptorium validate` call followed by `scriptorium run` performs two
independent preparations; the first does not retain an executable snapshot for
the second. When assessing CLI latency, measure the complete subprocess call,
including process startup and engine construction, as well as preparation.
Avoiding the public prepared snapshot does not eliminate the other preparation
work. These are downstream integration recommendations, not Scriptorium changes
included in this unit of work.
## Work Needed
- Add the public error-only method through the existing facade and internal
@@ -117,9 +158,9 @@ of all configured prompts and resources.
## Completion And Scope
The feature is complete when consumers can submit an ordinary `RunRequest` to
`Engine.Validate` and obtain preparation success or its existing failure without
a model call or public prepared snapshot, with focused tests and canonical
documentation establishing that behavior.
`Engine.ValidateRequest` and obtain preparation success or its existing failure
without a model call or public prepared snapshot, with focused tests and
canonical documentation establishing that behavior.
This unit of work does not include changes to Scriptorium, CLI commands or exit
codes, aggregated diagnostics, presence-only validation modes, provider probes,