Clarify request validation roadmap and CLI guidance
This commit is contained in:
@@ -8,6 +8,10 @@ Provide an explicit success/error operation for consumers that want to check an
|
||||
actual prompt request without generating a model response or receiving a
|
||||
prepared snapshot.
|
||||
|
||||
The check applies to concrete variable values and input bindings for one
|
||||
invocation. It does not certify that an application's configured bindings cover
|
||||
every possible request or template branch.
|
||||
|
||||
Scriptorium motivates this addition: applications invoke it as a subprocess and
|
||||
could use a `validate` command to fail early when their supplied variables,
|
||||
inputs, or other request settings cannot pass preparation. PromptKit owns the
|
||||
@@ -38,9 +42,14 @@ See the [consumer inspection and preparation guide](../consumers/pkg-promptkit.m
|
||||
Add a separate method on the existing engine:
|
||||
|
||||
```go
|
||||
func (e *Engine) Validate(ctx context.Context, req RunRequest) error
|
||||
func (e *Engine) ValidateRequest(ctx context.Context, req RunRequest) error
|
||||
```
|
||||
|
||||
`ValidateRequest` names the object being checked and distinguishes this operation
|
||||
from generated-output validation. `validate` remains the recommended downstream
|
||||
CLI command name; preflight describes the workflow, without implying provider
|
||||
or capacity readiness checks.
|
||||
|
||||
Use the complete existing `RunRequest`, including input bindings, caller
|
||||
variables, profile selection, execution overrides, session settings, output
|
||||
contract overrides, and appended messages. Apply the same request conversion
|
||||
@@ -60,7 +69,8 @@ meaning of its returned snapshot. The intended distinction is:
|
||||
|
||||
- `InspectRequirements`: discover what a prompt may require for a supplied
|
||||
context, including partial discovery where appropriate.
|
||||
- `Validate`: determine whether the actual supplied request passes preparation.
|
||||
- `ValidateRequest`: determine whether the actual supplied request passes
|
||||
preparation.
|
||||
- `Prepare`: perform preparation and return its snapshot.
|
||||
- `PrepareExecution`: prepare an owned snapshot that can later be executed.
|
||||
- `Run`: prepare and execute the request.
|
||||
@@ -72,23 +82,54 @@ the branches selected by supplied values. It applies existing rules to missing
|
||||
bindings and variables rather than adding stricter required-field rules,
|
||||
rejecting additional caller keys, or restricting supported templates.
|
||||
|
||||
For example, consider a message containing
|
||||
`{{if eq .mode "detailed"}}{{input "details"}}{{end}}`, where `details` is not
|
||||
declared required. A request with `mode=brief` can prepare without that input;
|
||||
one with `mode=detailed` needs it. Declaring the input required makes it
|
||||
necessary regardless of the selected branch. Similarly, an evaluated
|
||||
`{{.audience}}` fails when the key is absent, but supplying an empty string does
|
||||
not itself fail preparation. Optional lookups such as `{{index . "audience"}}`
|
||||
retain their existing semantics. These examples illustrate preparation parity,
|
||||
not new required-field rules; see the [input and template formats](../formats.md#inputs)
|
||||
and [requirement classifications](../../requirements.go).
|
||||
|
||||
No LLM request is made. This is not a promise of zero I/O: artifact and schema
|
||||
loaders still perform their existing resource work, including caller-supplied
|
||||
loader behavior. Avoiding inference makes the operation useful for fail-fast
|
||||
workflows, but large inputs or resource processing can still cost time and
|
||||
memory. No fixed latency or allocation guarantee is proposed.
|
||||
loader behavior. The current preparation path loads every supplied input, even
|
||||
if the selected template branch does not use it. Avoiding inference makes the
|
||||
operation useful for fail-fast workflows, but large inputs or resource
|
||||
processing can still cost time and memory. No fixed latency or allocation
|
||||
guarantee is proposed.
|
||||
|
||||
Success does not establish provider acceptance, context-window fit, generated
|
||||
output validity, or application-domain correctness. It also does not reserve
|
||||
execution capacity or retain a snapshot for a subsequent invocation. Later
|
||||
requests can differ, and referenced resources can change. Consumers that need
|
||||
execution tied to the checked snapshot should use `PrepareExecution`.
|
||||
execution tied to the checked snapshot within one process and engine should use
|
||||
`PrepareExecution`. Its [opaque handle](../../prepared_execution.go) cannot
|
||||
connect separate subprocess invocations.
|
||||
|
||||
This is request-scoped validation using an already constructed engine. It does
|
||||
not expand configuration ownership beyond the final handoff described by
|
||||
[ADR 0001](../adr/0001-configuration-boundary.md), or introduce eager validation
|
||||
of all configured prompts and resources.
|
||||
|
||||
## Downstream CLI Guidance
|
||||
|
||||
Scriptorium's proposed `validate` and existing `run` commands should share
|
||||
request construction: argument interpretation, defaults, profile selection,
|
||||
and overrides must describe the same request for equivalent caller inputs.
|
||||
This allows PromptKit's preparation parity to carry through the CLI boundary.
|
||||
Scriptorium owns exit statuses and diagnostic presentation.
|
||||
|
||||
A `scriptorium validate` call followed by `scriptorium run` performs two
|
||||
independent preparations; the first does not retain an executable snapshot for
|
||||
the second. When assessing CLI latency, measure the complete subprocess call,
|
||||
including process startup and engine construction, as well as preparation.
|
||||
Avoiding the public prepared snapshot does not eliminate the other preparation
|
||||
work. These are downstream integration recommendations, not Scriptorium changes
|
||||
included in this unit of work.
|
||||
|
||||
## Work Needed
|
||||
|
||||
- Add the public error-only method through the existing facade and internal
|
||||
@@ -117,9 +158,9 @@ of all configured prompts and resources.
|
||||
## Completion And Scope
|
||||
|
||||
The feature is complete when consumers can submit an ordinary `RunRequest` to
|
||||
`Engine.Validate` and obtain preparation success or its existing failure without
|
||||
a model call or public prepared snapshot, with focused tests and canonical
|
||||
documentation establishing that behavior.
|
||||
`Engine.ValidateRequest` and obtain preparation success or its existing failure
|
||||
without a model call or public prepared snapshot, with focused tests and
|
||||
canonical documentation establishing that behavior.
|
||||
|
||||
This unit of work does not include changes to Scriptorium, CLI commands or exit
|
||||
codes, aggregated diagnostics, presence-only validation modes, provider probes,
|
||||
|
||||
Reference in New Issue
Block a user