221 lines
12 KiB
Markdown
221 lines
12 KiB
Markdown
# Effective Prompt Requirement Inspection
|
|
|
|
Status: Implemented (2026-10-08). The
|
|
[implementation plan](implementation.md) records the completed work units and
|
|
final verification. Current public API details live in GoDoc; the consumer
|
|
guide, format reference, and internal documents link to those canonical
|
|
contracts.
|
|
|
|
## Purpose
|
|
|
|
Let consumers enumerate a selected prompt's inputs and caller variables before
|
|
its input artifacts exist, with each name classified as required, conditional,
|
|
or optional for the supplied context. Scriptorium can present that inventory
|
|
to operators and Narratio can compare planned bindings with exact input names
|
|
before undertaking expensive pipeline work.
|
|
|
|
The originating request is [Scriptorium's feature request](scriptorium-feature-request.md).
|
|
Promptkit supplies application-neutral inspection; consumers own presentation,
|
|
binding checks, and their own workflows.
|
|
|
|
## Original Review Baseline
|
|
|
|
At review baseline `b595dca`:
|
|
|
|
- [InspectPrompt](../consumers/pkg-promptkit.md#inspect-a-prompt-before-preparation)
|
|
selects a captured prompt and reports declared inputs and output metadata,
|
|
without analyzing templates or resolving an execution profile.
|
|
- [InspectProfile](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work)
|
|
reports a captured profile's effective target separately.
|
|
- [Prepare](../consumers/pkg-promptkit.md#prepare-without-model-execution)
|
|
selects an execution context, loads artifacts and applicable schemas, and
|
|
renders messages. It needs actual input data.
|
|
- Engine construction captures final definitions and compiled message/session
|
|
templates. Rendering uses Go templates, string-valued caller variables, and
|
|
the `input` helper. Required input declarations are checked independently of
|
|
branches; an executed helper also needs its named input.
|
|
- Declared content types and descriptions currently use string fields. The
|
|
new inspection needs declaration information and effective metadata presence,
|
|
including the difference between an absent field and an explicitly empty one.
|
|
|
|
The [format reference](../formats.md#messages-and-templates), existing GoDoc,
|
|
and [internal source document](../internal/sources.md) own current behavior.
|
|
|
|
## Target Public Operation
|
|
|
|
Add `Engine.InspectRequirements` with dedicated public request/result values.
|
|
Preserve `InspectPrompt`, `InspectProfile`, preparation, and execution.
|
|
|
|
The request supplies a prompt ID, optional version and profile selectors,
|
|
caller variables, and an optional direct session ID. It needs no artifact
|
|
references or contents, output path, or credentials.
|
|
|
|
Use execution's selection semantics: exact identities, omitted-version
|
|
ambiguity, request profile or prompt default, and normalized direct session-ID
|
|
bypass. Report the selected prompt ID/version and effective profile ID.
|
|
Selecting a profile establishes identity, not generation readiness; partial
|
|
profiles remain inspectable.
|
|
|
|
Use the configured engine's captured definitions. Source selection remains a
|
|
consumer choice. Results describe that captured context, not a later engine or
|
|
execution with different variables.
|
|
|
|
## Result And Classification
|
|
|
|
Return an inventory of inputs and a separate inventory of caller variables.
|
|
Each entry has one of three requirement classifications:
|
|
|
|
- **Required:** A declaration or recognized definite access establishes that
|
|
the caller must supply it.
|
|
- **Conditional:** Its necessity remains unresolved, including references on
|
|
paths whose applicability the supported analysis cannot determine.
|
|
- **Optional:** It is known not to be required, including recognizable references
|
|
used only on known inactive paths and nonrequired declarations or lookups.
|
|
|
|
A required declaration stays required even in an inactive branch. A definite
|
|
helper use promotes an optional input declaration to required. Aggregate repeated
|
|
occurrences by exact name: required takes precedence over conditional, which
|
|
takes precedence over optional. Conservative conditional classification is
|
|
acceptable where analysis cannot establish necessity; inspection need not prove
|
|
equivalence between alternative execution paths.
|
|
|
|
Input entries distinguish declarations from discovered undeclared references
|
|
and preserve available descriptions/content types. They do not infer input
|
|
schemas or introduce format enforcement. Variables identify caller-map keys,
|
|
not template-local symbol names, and report whether each key was supplied.
|
|
An empty supplied string is present; absence is separate from classification.
|
|
|
|
Include recognizable references on known inactive paths in both inventories
|
|
as optional entries with an inactive-use explanation. Keep explanations separate
|
|
from declared input descriptions. An applicable required or conditional use
|
|
elsewhere overrides the optional classification. Any applicable occurrence,
|
|
including an optional lookup, removes an inactive-only explanation. Known
|
|
inactivity introduces neither an execution requirement nor discovery uncertainty.
|
|
|
|
Deduplicate names and provide predictable ordering. Results also identify the
|
|
selected context and provide discovery completeness with machine-readable
|
|
limitation categories and safe explanations. Exact public spellings belong in
|
|
the implementation plan and, once implemented, GoDoc. No Promptkit JSON wire
|
|
schema is part of this feature.
|
|
|
|
## Simple Discovery Boundary
|
|
|
|
Use parsed template syntax, not raw-text matching or rendering with placeholder
|
|
artifacts. The intended algorithm is straightforward:
|
|
|
|
1. Seed the inventory from input declarations and their effective metadata.
|
|
2. Visit the selected message templates and applicable session template,
|
|
collecting recognizable input and caller-variable references.
|
|
3. Classify occurrences using their surrounding path and a small amount of
|
|
supplied-context evaluation.
|
|
4. Aggregate entries and report any unresolved discovery.
|
|
|
|
The source inventory is finite and ordinarily small. Direct traversal and
|
|
name-indexed aggregation are sufficient; no size caps, analysis budgets,
|
|
branch-combination enumeration, or global analysis cache are needed.
|
|
|
|
Recognize literal input names, direct caller-variable access, literal-key root
|
|
lookups, direct variable-derived input names, and ordinary `if`/`with` branches.
|
|
Use simple truthiness, negation, and equality/inequality where straightforward.
|
|
Empty strings and the nonempty string `"false"` follow actual Go truthiness.
|
|
The implementation plan defines this small initial coverage precisely.
|
|
|
|
Retain enough lexical/dot scope awareness to distinguish caller variables from
|
|
locals. Complicated assignments, pipelines, computed names, ranges, or invoked
|
|
named templates can remain outside complete analysis. Preserve identifiable
|
|
references and report limitations rather than implementing a general template
|
|
interpreter, joining assignment environments, or propagating possible-name sets.
|
|
No cross-branch proof of a common required dependency is necessary.
|
|
|
|
On a known inactive path, collect names visible in syntax without evaluating
|
|
its expressions. Literal input names and recognizable caller keys remain in
|
|
the inventory; skipped dynamic input names need not be resolved. Unsupported
|
|
skipped expressions do not themselves make discovery incomplete.
|
|
|
|
A direct nonblank session ID excludes the bypassed session template from both
|
|
inventories. Never-invoked template definitions do not create requirements.
|
|
Declarations and references in selected messages remain relevant.
|
|
|
|
Complex computed names may be missed by the inventory. That is an accepted
|
|
analysis limitation, not a reason to reject an otherwise valid prompt.
|
|
Inspection must expose potential incompleteness rather than imply that unknown
|
|
names are absent. Template authors benefit from keeping dependency expressions
|
|
simple, but this feature adds no template-authoring prohibition.
|
|
|
|
## Completeness And Limitations
|
|
|
|
Completeness describes discovery, not satisfaction or readiness. A missing
|
|
variable used only for direct text substitution can be a known required entry
|
|
with complete discovery. A missing value used to select a branch or name can
|
|
leave dependencies unresolved.
|
|
|
|
When unavailable input content, unresolved names, missing decision values, or
|
|
unsupported analysis affect discovery, return known entries with successful
|
|
incomplete status and structured reasons. Conservative incompleteness is
|
|
acceptable; inspection need not prove an exact requirement set through complex
|
|
expressions. Unsupported pure text with no effect on dependencies need not
|
|
introduce uncertainty.
|
|
|
|
Unresolved names can potentially refer to existing optional entries. Account
|
|
for that possibility in classification, while leaving established required
|
|
entries required. Do not label a potentially needed entry definitively optional.
|
|
|
|
An empty complete input inventory, or a complete input inventory containing
|
|
only optional entries, establishes that no artifact bindings are needed.
|
|
An empty incomplete inventory does not. Consumers must use classifications,
|
|
not inventory length, to identify bindings that need preparation.
|
|
|
|
## Effects, Errors, And Disclosure
|
|
|
|
Inspection uses captured resources without future artifact reads, schema
|
|
preparation, credential lookup, model calls, capacity admission, observations,
|
|
run IDs, or execution artifact creation. Necessary configuration/resource
|
|
failures can still prevent engine construction or inspection.
|
|
|
|
Preserve publicly classifiable selection errors and ordinary context
|
|
cancellation. Discovery uncertainty is a successful result, not an operational
|
|
error or a new definition-validity rule. A complete result does not promise
|
|
successful rendering, generation, future artifact validity, or provider
|
|
availability.
|
|
|
|
Results and diagnostics exclude credential values, artifact payloads,
|
|
raw/rendered prompt content, and unrelated supplied values. Selected identities,
|
|
input/variable names, and declared descriptions are intentional operator
|
|
metadata. A directly resolved input identifier may equal a supplied variable
|
|
value; reporting that binding name is necessary, without echoing its originating
|
|
expression or variable map.
|
|
|
|
## Implemented Outcome
|
|
|
|
The root API selects a captured prompt and profile identity, then returns
|
|
caller-owned input and variable inventories with conservative classifications
|
|
and structured discovery limitations. The internal prompt walker uses captured
|
|
template syntax and effective metadata; inspection does not enter artifact,
|
|
schema, credential, rendering, provider, or execution preparation.
|
|
|
|
The final verification and the commit sequence are recorded in the
|
|
[implementation plan](implementation.md). The exact public contract is owned by
|
|
[GoDoc](../../requirements.go); consumer usage, framework-format cross-reference,
|
|
and internal selection/source descriptions are maintained in their respective
|
|
documents.
|
|
|
|
## Acceptance Outcomes
|
|
|
|
- Required `transcript` is reported exactly before its artifact exists.
|
|
- Literal undeclared helper names are discoverable without definition rejection.
|
|
- Input and variable references are classified as required, conditional, or
|
|
optional, including recognizable inactive references in both inventories.
|
|
- Required declarations stay required; definite uses promote optional entries;
|
|
other occurrences never erase an established required use.
|
|
- Simple supplied-context branches and direct dynamic names follow execution;
|
|
complicated or content-dependent cases preserve known entries and limitations.
|
|
- Caller keys remain distinct from template locals, and supplied empty values
|
|
remain distinct from absent keys.
|
|
- Selection and session bypass follow execution without readiness checks.
|
|
- Complete artifact-free inventories remain distinguishable from empty partial
|
|
discovery; completeness remains separate from supplied-variable presence.
|
|
- Results/errors preserve the disclosure and public classification boundaries.
|
|
- Existing inspection/execution contracts remain intact. Application APIs,
|
|
pipeline planning, binding validation, presentation, provider preflight, input
|
|
schemas, and a general-purpose template evaluator remain outside this scope.
|