Clarify request validation roadmap and CLI guidance
This commit is contained in:
@@ -8,6 +8,10 @@ Provide an explicit success/error operation for consumers that want to check an
|
|||||||
actual prompt request without generating a model response or receiving a
|
actual prompt request without generating a model response or receiving a
|
||||||
prepared snapshot.
|
prepared snapshot.
|
||||||
|
|
||||||
|
The check applies to concrete variable values and input bindings for one
|
||||||
|
invocation. It does not certify that an application's configured bindings cover
|
||||||
|
every possible request or template branch.
|
||||||
|
|
||||||
Scriptorium motivates this addition: applications invoke it as a subprocess and
|
Scriptorium motivates this addition: applications invoke it as a subprocess and
|
||||||
could use a `validate` command to fail early when their supplied variables,
|
could use a `validate` command to fail early when their supplied variables,
|
||||||
inputs, or other request settings cannot pass preparation. PromptKit owns the
|
inputs, or other request settings cannot pass preparation. PromptKit owns the
|
||||||
@@ -38,9 +42,14 @@ See the [consumer inspection and preparation guide](../consumers/pkg-promptkit.m
|
|||||||
Add a separate method on the existing engine:
|
Add a separate method on the existing engine:
|
||||||
|
|
||||||
```go
|
```go
|
||||||
func (e *Engine) Validate(ctx context.Context, req RunRequest) error
|
func (e *Engine) ValidateRequest(ctx context.Context, req RunRequest) error
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`ValidateRequest` names the object being checked and distinguishes this operation
|
||||||
|
from generated-output validation. `validate` remains the recommended downstream
|
||||||
|
CLI command name; preflight describes the workflow, without implying provider
|
||||||
|
or capacity readiness checks.
|
||||||
|
|
||||||
Use the complete existing `RunRequest`, including input bindings, caller
|
Use the complete existing `RunRequest`, including input bindings, caller
|
||||||
variables, profile selection, execution overrides, session settings, output
|
variables, profile selection, execution overrides, session settings, output
|
||||||
contract overrides, and appended messages. Apply the same request conversion
|
contract overrides, and appended messages. Apply the same request conversion
|
||||||
@@ -60,7 +69,8 @@ meaning of its returned snapshot. The intended distinction is:
|
|||||||
|
|
||||||
- `InspectRequirements`: discover what a prompt may require for a supplied
|
- `InspectRequirements`: discover what a prompt may require for a supplied
|
||||||
context, including partial discovery where appropriate.
|
context, including partial discovery where appropriate.
|
||||||
- `Validate`: determine whether the actual supplied request passes preparation.
|
- `ValidateRequest`: determine whether the actual supplied request passes
|
||||||
|
preparation.
|
||||||
- `Prepare`: perform preparation and return its snapshot.
|
- `Prepare`: perform preparation and return its snapshot.
|
||||||
- `PrepareExecution`: prepare an owned snapshot that can later be executed.
|
- `PrepareExecution`: prepare an owned snapshot that can later be executed.
|
||||||
- `Run`: prepare and execute the request.
|
- `Run`: prepare and execute the request.
|
||||||
@@ -72,23 +82,54 @@ the branches selected by supplied values. It applies existing rules to missing
|
|||||||
bindings and variables rather than adding stricter required-field rules,
|
bindings and variables rather than adding stricter required-field rules,
|
||||||
rejecting additional caller keys, or restricting supported templates.
|
rejecting additional caller keys, or restricting supported templates.
|
||||||
|
|
||||||
|
For example, consider a message containing
|
||||||
|
`{{if eq .mode "detailed"}}{{input "details"}}{{end}}`, where `details` is not
|
||||||
|
declared required. A request with `mode=brief` can prepare without that input;
|
||||||
|
one with `mode=detailed` needs it. Declaring the input required makes it
|
||||||
|
necessary regardless of the selected branch. Similarly, an evaluated
|
||||||
|
`{{.audience}}` fails when the key is absent, but supplying an empty string does
|
||||||
|
not itself fail preparation. Optional lookups such as `{{index . "audience"}}`
|
||||||
|
retain their existing semantics. These examples illustrate preparation parity,
|
||||||
|
not new required-field rules; see the [input and template formats](../formats.md#inputs)
|
||||||
|
and [requirement classifications](../../requirements.go).
|
||||||
|
|
||||||
No LLM request is made. This is not a promise of zero I/O: artifact and schema
|
No LLM request is made. This is not a promise of zero I/O: artifact and schema
|
||||||
loaders still perform their existing resource work, including caller-supplied
|
loaders still perform their existing resource work, including caller-supplied
|
||||||
loader behavior. Avoiding inference makes the operation useful for fail-fast
|
loader behavior. The current preparation path loads every supplied input, even
|
||||||
workflows, but large inputs or resource processing can still cost time and
|
if the selected template branch does not use it. Avoiding inference makes the
|
||||||
memory. No fixed latency or allocation guarantee is proposed.
|
operation useful for fail-fast workflows, but large inputs or resource
|
||||||
|
processing can still cost time and memory. No fixed latency or allocation
|
||||||
|
guarantee is proposed.
|
||||||
|
|
||||||
Success does not establish provider acceptance, context-window fit, generated
|
Success does not establish provider acceptance, context-window fit, generated
|
||||||
output validity, or application-domain correctness. It also does not reserve
|
output validity, or application-domain correctness. It also does not reserve
|
||||||
execution capacity or retain a snapshot for a subsequent invocation. Later
|
execution capacity or retain a snapshot for a subsequent invocation. Later
|
||||||
requests can differ, and referenced resources can change. Consumers that need
|
requests can differ, and referenced resources can change. Consumers that need
|
||||||
execution tied to the checked snapshot should use `PrepareExecution`.
|
execution tied to the checked snapshot within one process and engine should use
|
||||||
|
`PrepareExecution`. Its [opaque handle](../../prepared_execution.go) cannot
|
||||||
|
connect separate subprocess invocations.
|
||||||
|
|
||||||
This is request-scoped validation using an already constructed engine. It does
|
This is request-scoped validation using an already constructed engine. It does
|
||||||
not expand configuration ownership beyond the final handoff described by
|
not expand configuration ownership beyond the final handoff described by
|
||||||
[ADR 0001](../adr/0001-configuration-boundary.md), or introduce eager validation
|
[ADR 0001](../adr/0001-configuration-boundary.md), or introduce eager validation
|
||||||
of all configured prompts and resources.
|
of all configured prompts and resources.
|
||||||
|
|
||||||
|
## Downstream CLI Guidance
|
||||||
|
|
||||||
|
Scriptorium's proposed `validate` and existing `run` commands should share
|
||||||
|
request construction: argument interpretation, defaults, profile selection,
|
||||||
|
and overrides must describe the same request for equivalent caller inputs.
|
||||||
|
This allows PromptKit's preparation parity to carry through the CLI boundary.
|
||||||
|
Scriptorium owns exit statuses and diagnostic presentation.
|
||||||
|
|
||||||
|
A `scriptorium validate` call followed by `scriptorium run` performs two
|
||||||
|
independent preparations; the first does not retain an executable snapshot for
|
||||||
|
the second. When assessing CLI latency, measure the complete subprocess call,
|
||||||
|
including process startup and engine construction, as well as preparation.
|
||||||
|
Avoiding the public prepared snapshot does not eliminate the other preparation
|
||||||
|
work. These are downstream integration recommendations, not Scriptorium changes
|
||||||
|
included in this unit of work.
|
||||||
|
|
||||||
## Work Needed
|
## Work Needed
|
||||||
|
|
||||||
- Add the public error-only method through the existing facade and internal
|
- Add the public error-only method through the existing facade and internal
|
||||||
@@ -117,9 +158,9 @@ of all configured prompts and resources.
|
|||||||
## Completion And Scope
|
## Completion And Scope
|
||||||
|
|
||||||
The feature is complete when consumers can submit an ordinary `RunRequest` to
|
The feature is complete when consumers can submit an ordinary `RunRequest` to
|
||||||
`Engine.Validate` and obtain preparation success or its existing failure without
|
`Engine.ValidateRequest` and obtain preparation success or its existing failure
|
||||||
a model call or public prepared snapshot, with focused tests and canonical
|
without a model call or public prepared snapshot, with focused tests and
|
||||||
documentation establishing that behavior.
|
canonical documentation establishing that behavior.
|
||||||
|
|
||||||
This unit of work does not include changes to Scriptorium, CLI commands or exit
|
This unit of work does not include changes to Scriptorium, CLI commands or exit
|
||||||
codes, aggregated diagnostics, presence-only validation modes, provider probes,
|
codes, aggregated diagnostics, presence-only validation modes, provider probes,
|
||||||
|
|||||||
Reference in New Issue
Block a user