Plan effective requirement inspection implementation
This commit is contained in:
176
docs/roadmap/scriptorium-feature-request.md
Normal file
176
docs/roadmap/scriptorium-feature-request.md
Normal file
@@ -0,0 +1,176 @@
|
||||
# Feature Request: Inspect Effective Prompt Requirements
|
||||
|
||||
Status: proposed consumer request from Scriptorium, recorded 2026-10-08.
|
||||
This document specifies requested public API behavior. API names and types are
|
||||
open for agreement; implementation structure is left to Promptkit.
|
||||
|
||||
## Consumer Need
|
||||
|
||||
Scriptorium exposes Promptkit through CLI and HTTP interfaces for executing
|
||||
predefined prompts. A subprocess consumer, Narratio, uses Scriptorium to produce
|
||||
D&D session analysis from transcripts and other pipeline products.
|
||||
|
||||
Narratio needs to check its input bindings before starting expensive upstream
|
||||
work. At that point, the eventual input files often do not exist. A pipeline
|
||||
source ID and a prompt input name are separate identities: a transcript product
|
||||
may need to be bound under the exact prompt input name `transcript`.
|
||||
|
||||
Scriptorium needs Promptkit to describe the selected execution context's input
|
||||
and variable requirements through a supported public API. Scriptorium can then
|
||||
present those requirements to operators and executable consumers without
|
||||
interpreting prompt definitions or duplicating framework semantics.
|
||||
|
||||
## Public API Gap
|
||||
|
||||
The [consumer guide](../consumers/pkg-promptkit.md#inspect-a-prompt-before-preparation)
|
||||
describes `InspectPrompt` as declared-input inspection without profile
|
||||
resolution. That is useful, but declared inputs alone do not describe all
|
||||
requirements introduced by templates. `InspectProfile` addresses profile
|
||||
inspection separately, and `Prepare` loads inputs and renders messages.
|
||||
|
||||
The requested operation serves a different purpose: discover effective caller
|
||||
requirements before input artifacts are available. Existing inspection and
|
||||
execution operations should retain their established contracts.
|
||||
|
||||
## Requested Operation And Selection
|
||||
|
||||
Provide a read-only public operation on a configured engine. Scriptorium needs
|
||||
to supply:
|
||||
|
||||
- A prompt ID and optional version selector.
|
||||
- An optional execution profile selector.
|
||||
- Supplied template variables.
|
||||
- An optional direct session ID, because it can bypass a session-ID template.
|
||||
|
||||
Selection and interpretation should follow the same public semantics as an
|
||||
equivalent execution request, including omitted selectors, exact identities,
|
||||
variable values, and session-ID override behavior. The result should identify
|
||||
the selected prompt ID/version and effective profile ID, including selection
|
||||
through the prompt's default profile.
|
||||
|
||||
Inspection uses the engine's configured sources. Scriptorium continues to own
|
||||
application configuration discovery, CLI flags, and source selection. The API
|
||||
should require neither input references or contents nor an output path.
|
||||
|
||||
## Required Result Information
|
||||
|
||||
Expose public values with documented meanings that Scriptorium can project into
|
||||
its own versioned JSON response and readable text output:
|
||||
|
||||
- Selected prompt and profile identities.
|
||||
- Discoverable caller-supplied input names, including declared inputs and
|
||||
references in message and applicable session-ID templates.
|
||||
- Whether each input is required, optional, or conditional for the supplied
|
||||
context, with an explanation where available. Preserve exact execution names.
|
||||
- Declared input descriptions and content-type metadata, distinguishing absent
|
||||
metadata from a known declaration.
|
||||
- Caller-supplied variable requirements separately from artifact inputs,
|
||||
including whether supplied values satisfy the known requirements.
|
||||
- Whether discovery is complete for the supplied context, together with
|
||||
machine-readable reasons and safe explanations for unresolved requirements.
|
||||
|
||||
Provide predictable ordering for repeatable presentation and enough public
|
||||
classification information that callers do not need to parse diagnostic prose.
|
||||
Exact type names, fields, and ordering conventions remain API design choices.
|
||||
Scriptorium owns its subprocess JSON schema; a Promptkit-specific JSON wire
|
||||
format is not required by this request.
|
||||
|
||||
The [format reference](../formats.md#inputs) currently provides input
|
||||
content-type metadata, rather than an input-schema declaration. Report the
|
||||
meaning of existing metadata accurately. This request does not introduce input
|
||||
format enforcement, input schemas, or variable defaults. Where a supported
|
||||
format provides defaults or constraints, inspection should reflect their actual
|
||||
execution semantics rather than infer them from names.
|
||||
|
||||
## Context-Sensitive And Incomplete Discovery
|
||||
|
||||
A declared required input remains required independently of whether a template
|
||||
branch references it. An input whose only requirement comes from an inactive
|
||||
branch should not be classified as required for that context.
|
||||
|
||||
Use supplied variables when they determine active branches or dynamically
|
||||
selected names. Variable interpretation must agree with execution, including
|
||||
template truthiness and presence semantics. Include requirements from an active
|
||||
session-ID template; account for its bypass by a direct session ID.
|
||||
|
||||
Where unavailable input contents, missing variables, dynamic names, or other
|
||||
analysis limitations prevent a complete determination, return the known
|
||||
requirements with explicit uncertainty. Explain conditions through public
|
||||
information where possible, without revealing template text or supplied values.
|
||||
A valid prompt remains valid when its requirements cannot be fully discovered.
|
||||
An undeclared template input reference is not, by itself, a new definition error.
|
||||
|
||||
An empty complete result establishes that no caller-supplied inputs are needed
|
||||
for the inspected context. An empty incomplete result does not. Completeness
|
||||
describes discovery, not satisfaction: a known required variable may still be
|
||||
missing, even when its requirement has been fully identified. If its absence
|
||||
also leaves input dependencies unknown, report that uncertainty separately.
|
||||
|
||||
Incomplete discovery should be a successful result with explicit status, rather
|
||||
than a selection error. Document the discovery boundary so Scriptorium can
|
||||
distinguish a fully determined requirement set from a useful partial result.
|
||||
|
||||
## Effects, Errors, And Safety
|
||||
|
||||
Inspection should work without future input artifacts, generation credentials,
|
||||
model access, capacity reservation, or creation of execution artifacts. Reading
|
||||
configuration and prompt resources needed for inspection is expected.
|
||||
Profile selection establishes identity; this operation is not a generation
|
||||
readiness or credential check.
|
||||
|
||||
Invalid selectors, unknown or ambiguous selections, unreadable necessary
|
||||
resources, and invalid definitions that prevent inspection should produce
|
||||
publicly classifiable errors consistent with existing selection semantics.
|
||||
Scriptorium needs stable categories or error identities and safe descriptions
|
||||
to produce structured CLI failures. It should not need to match error-message
|
||||
text or access implementation packages.
|
||||
|
||||
Results and consumer-facing diagnostics should exclude credential values,
|
||||
input payloads, supplied variable values, and rendered or raw prompt contents.
|
||||
Identifiers and declared descriptions remain useful operator-facing metadata.
|
||||
|
||||
For a complete result, the reported requirements should agree with execution
|
||||
for the same effective configuration and supplied context. An incomplete
|
||||
result should accurately describe what is known and what remains uncertain.
|
||||
Neither result guarantees generation success or validates future input bytes.
|
||||
Later configuration or prompt-resource changes may change a later execution's
|
||||
requirements.
|
||||
|
||||
## Consumer Acceptance Scenarios
|
||||
|
||||
1. A required `transcript` input is reported with its exact name, allowing
|
||||
Scriptorium's caller to detect a binding named `session_transcript` instead.
|
||||
2. A literal template reference to an undeclared input is discoverable without
|
||||
requiring that input's contents or rejecting the reference as undeclared.
|
||||
3. A declared optional glossary remains optional when no active template use
|
||||
makes it required. A declared required input remains required in an inactive
|
||||
template branch.
|
||||
4. Supplied variables determine conditional requirements and dynamic names
|
||||
where discoverable. Content-dependent or otherwise unresolved cases return
|
||||
known entries with explicit incompleteness.
|
||||
5. Changing the prompt version or profile reports the corresponding selected
|
||||
identities. Omitted selections follow execution's defaults and ambiguity
|
||||
rules.
|
||||
6. Session-template dependencies are reported when applicable; a direct session
|
||||
ID removes requirements introduced solely by the bypassed template.
|
||||
7. Missing caller variables are reported separately from missing artifact
|
||||
bindings, without returning supplied values or prompt contents.
|
||||
8. Inspection succeeds before artifacts exist and without credentials or
|
||||
provider access, assuming necessary configuration and resources are valid
|
||||
and readable.
|
||||
9. Unknown prompts, ambiguous versions, and invalid profile selections yield
|
||||
classifiable failures; analysis uncertainty yields an incomplete result.
|
||||
10. A genuinely input-free context is distinguishable from an incomplete
|
||||
discovery that happened to find no input names.
|
||||
|
||||
## Scope And Ownership
|
||||
|
||||
Promptkit supplies the public requirement-discovery semantics. Scriptorium owns
|
||||
CLI exposure, text and JSON presentation, stream handling, and exit statuses.
|
||||
Narratio owns its source catalog, bindings, pipeline ordering, dependency graph,
|
||||
and session availability checks.
|
||||
|
||||
No Narratio-specific identifiers or pipeline concepts are needed in Promptkit's
|
||||
API. A graph renderer, HTTP endpoint, batch interface, changes to generation
|
||||
behavior, and execution-resource preflight are outside this request. This is a
|
||||
consumer capability request, not an implementation plan.
|
||||
Reference in New Issue
Block a user