From 5a5d47d36eb6b03c50d1844cfc67ed4fd11ccf89 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 12 Oct 2026 16:56:33 +0000 Subject: [PATCH] Define request preflight validation roadmap --- docs/roadmap/validate.md | 127 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 docs/roadmap/validate.md diff --git a/docs/roadmap/validate.md b/docs/roadmap/validate.md new file mode 100644 index 0000000..18431f1 --- /dev/null +++ b/docs/roadmap/validate.md @@ -0,0 +1,127 @@ +# Request Preflight Validation + +Status: accepted; implementation pending. + +## Purpose + +Provide an explicit success/error operation for consumers that want to check an +actual prompt request without generating a model response or receiving a +prepared snapshot. + +Scriptorium motivates this addition: applications invoke it as a subprocess and +could use a `validate` command to fail early when their supplied variables, +inputs, or other request settings cannot pass preparation. PromptKit owns the +reusable request operation; Scriptorium owns command syntax, diagnostics, +process exit statuses, and any application-specific interpretation. + +## Current Baseline + +`Engine.InspectRequirements` discovers recognizable input and caller-variable +dependencies without loading artifacts or rendering messages. Its `Complete` +flag describes discovery coverage, not whether supplied dependencies satisfy a +request. It is useful for authoring and planning, but is not actual preflight. + +`Engine.Prepare` already performs preflight through request conversion and the +shared internal preparation pipeline. It resolves the prompt, profile, effective +settings, and output contract; loads input artifacts and applicable schema +resources; and renders messages and the session ID without calling an LLM. +The facade then converts the internal prepared value into a caller-owned +`PreparedRun`. Consumers can discard that result today, but the API does not +directly express a success/error-only operation. + +See the [consumer inspection and preparation guide](../consumers/pkg-promptkit.md#inspect-requirements-before-preparation), +[existing preparation GoDoc](../../engine.go), and +[internal preparation flow](../internal/runner.md#shared-preparation-pipeline). + +## Target Public API + +Add a separate method on the existing engine: + +```go +func (e *Engine) Validate(ctx context.Context, req RunRequest) error +``` + +Use the complete existing `RunRequest`, including input bindings, caller +variables, profile selection, execution overrides, session settings, output +contract overrides, and appended messages. Apply the same request conversion +and preparation rules as `Prepare`; validation is not a name-only check or an +analysis of declared requirements. + +A nil error means **the supplied request passed preparation at the time of the +check**. A failure returns the error encountered through the existing +preparation path, with the same public error categories and applicable error +identities. A nil engine follows `Prepare`'s `ErrInvalidConfig` convention. +Context handling follows preparation rather than introducing a stronger +cancellation guarantee. The result does not aggregate all missing requirements +or return a prepared value, execution handle, or model output. + +Keep `Prepare` unchanged. A separate method avoids a mode flag that changes the +meaning of its returned snapshot. The intended distinction is: + +- `InspectRequirements`: discover what a prompt may require for a supplied + context, including partial discovery where appropriate. +- `Validate`: determine whether the actual supplied request passes preparation. +- `Prepare`: perform preparation and return its snapshot. +- `PrepareExecution`: prepare an owned snapshot that can later be executed. +- `Run`: prepare and execute the request. + +## Meaning And Limits + +Validation exercises actual preparation, including loading inputs and rendering +the branches selected by supplied values. It applies existing rules to missing +bindings and variables rather than adding stricter required-field rules, +rejecting additional caller keys, or restricting supported templates. + +No LLM request is made. This is not a promise of zero I/O: artifact and schema +loaders still perform their existing resource work, including caller-supplied +loader behavior. Avoiding inference makes the operation useful for fail-fast +workflows, but large inputs or resource processing can still cost time and +memory. No fixed latency or allocation guarantee is proposed. + +Success does not establish provider acceptance, context-window fit, generated +output validity, or application-domain correctness. It also does not reserve +execution capacity or retain a snapshot for a subsequent invocation. Later +requests can differ, and referenced resources can change. Consumers that need +execution tied to the checked snapshot should use `PrepareExecution`. + +This is request-scoped validation using an already constructed engine. It does +not expand configuration ownership beyond the final handoff described by +[ADR 0001](../adr/0001-configuration-boundary.md), or introduce eager validation +of all configured prompts and resources. + +## Work Needed + +- Add the public error-only method through the existing facade and internal + preparation path. Reuse request conversion, preparation, and error mapping + rather than implementing a second validation engine or an inspection-based + approximation. +- Discard the internal prepared result without converting it into a public + snapshot. The existing internal prepared value may still be constructed; + removing all preparation allocations is not required for this addition. + Share small facade plumbing where helpful without a broad refactor. +- Preserve preparation's existing context, credential, resource, and execution + separation behavior. No model generation, output-repair loop, or capacity + admission is needed for the new operation. +- Add focused external-package tests for successful preparation and + representative missing-variable/input, resource, selection, and invalid-request + failures. Use existing fixtures where suitable to verify parity with + `Prepare`, including a branch-sensitive example and the nil-engine boundary. + Check public error identities rather than exact messages, and confirm that + the injected model client is not called. Avoid duplicating the full + preparation test suite. +- Document the method in GoDoc and add a short consumer example distinguishing + validation from requirement discovery and prepared execution. Update the + internal runner description where needed when implementation lands, and + record the optional API addition in its eventual release notes. + +## Completion And Scope + +The feature is complete when consumers can submit an ordinary `RunRequest` to +`Engine.Validate` and obtain preparation success or its existing failure without +a model call or public prepared snapshot, with focused tests and canonical +documentation establishing that behavior. + +This unit of work does not include changes to Scriptorium, CLI commands or exit +codes, aggregated diagnostics, presence-only validation modes, provider probes, +domain-aware validation, or new execution guarantees. It introduces no changes +to prompt/profile formats or existing preparation and execution semantics.