promptkit/docs/roadmap/inspection.md

Effective Prompt Requirement Inspection

Status: Implementation planned; the feature is not implemented. The implementation plan owns the ordered migration steps and implements the agreed analysis scope and inventory below. This document owns the target end state and migration scope.

Purpose

Let consumers discover a selected prompt's caller requirements before its input artifacts exist. Scriptorium can expose that information to operators and to Narratio, which can compare its planned bindings with Promptkit's exact input names before undertaking expensive pipeline work.

The originating request is docs/roadmap/scriptorium-feature-request.md. The capability is application-neutral: Promptkit describes prompt requirements; consumers interpret them for their own configuration, presentation, and workflow.

Current Baseline

At review baseline b595dca:

• InspectPrompt selects a captured prompt and reports declared inputs and output metadata. It does not analyze templates or resolve the execution profile.
• InspectProfile reports one captured profile's effective target separately.
• Prepare selects the execution context, checks target requirements, loads artifacts and applicable schemas, and renders messages. It needs actual input data.
• The configuration adapter captures final effective definitions, message contents, and compiled message/session templates during engine construction. Runtime rendering uses Go templates, string-valued caller variables, and the input helper. Declared required inputs are checked independently of template branches; active helper calls also require their named inputs.
• Declared input metadata currently uses string fields for content type and description. The new result needs to identify a declaration and distinguish available metadata from absent metadata, including effective presence where necessary; current empty strings alone do not establish source-field presence.

The format reference, internal source document, and existing GoDoc own the implemented semantics. No existing operation provides artifact-free, context-sensitive requirement discovery.

Target Public Operation

Add a separate read-only engine operation, planned as Engine.InspectRequirements, with a dedicated request and result. Preserve InspectPrompt, InspectProfile, preparation, and execution contracts.

The request supplies a prompt ID, optional version, optional profile selector, supplied template variables, and an optional direct session ID. It needs no artifact references or contents, output path, API key, or execution settings unrelated to discovery.

Selection follows equivalent execution semantics: exact identities, omitted version ambiguity, request profile or prompt default, and direct session-ID normalization and bypass. Return selected prompt ID/version and effective profile ID. Profile selection establishes identity, not generation readiness; profiles that depend on later execution overrides need not be completed here.

Use the configured engine's captured definitions. Source discovery and layering remain consumer choices. Results describe that captured context, not a promise about a later engine or later execution with changed variables.

Result And Requirement Meanings

Expose public Go values with documented classifications and predictable ordering. Exact symbol and field spelling belongs in the implementation plan and GoDoc. No Promptkit JSON wire schema is part of this feature.

The result contains:

• Selected prompt and profile identities.
• Known input names, preserving execution identity, with required, optional, or conditional classification for the supplied context.
• Declaration information, descriptions, and content-type metadata where available. A template-discovered undeclared input must not look like a declared input with known metadata. Report existing media-type metadata, not an invented input schema or inferred format constraint.
• An inactive-use explanation for statically identifiable undeclared inputs referenced only on known inactive paths. These are optional inventory entries, not bindings the current context needs. Keep this explanation separate from declared descriptions.
• Variable requirements separately from artifact inputs, with requirement classification and whether supplied values satisfy the known requirement. Report presence and actual supported semantics, not the supplied values.
• A discovery-completeness indicator and structured uncertainty reasons, with safe explanations and relevant identifiers where available.

Required means the caller must supply the input for this context. A declared required input stays required even if no active template references it. A declared optional input becomes required when an active template use needs it. Optional means it is known not to be required for this context; conditional means its requirement depends on unresolved context. Do not use optional to hide uncertainty. Known required declarations retain that classification even when other requirements are conditional or undiscoverable.

Include statically identifiable undeclared references on known inactive paths as optional entries with an inactive-use explanation. A required or conditional use elsewhere takes precedence over that inactive-only classification and explanation. Inactivity itself is not discovery uncertainty.

An unresolved dynamic name can refer to an otherwise optional input entry. Its absence from recognized active uses alone does not establish that it is definitely optional; classification and uncertainty must account for that possibility.

Known input names should be deduplicated across declarations, messages, and session templates. Aggregate uses without discarding an established required use. Conditions need not expose raw expressions; machine-readable reason categories and safe summaries are sufficient for basic consumer handling.

Completeness describes discovery, not readiness or satisfaction. For example, a missing variable used only to insert text can be a known unsatisfied requirement while the complete input set is still discoverable. If that variable selects an input name or branch, its absence can also make discovery incomplete. An empty complete input set means no artifact bindings are needed; a complete inventory containing only optional entries also needs no artifact bindings. An empty incomplete set does not establish that fact. Consumers must distinguish the inventory from applicable requirements, rather than treating every reported name as a needed binding.

Context-Sensitive Discovery Boundary

Analyze parsed templates rather than searching raw text or rendering with placeholder artifacts. Use supplied variable values where they determine branches and input names, following execution's presence, truthiness, scope, and evaluation semantics. An empty string is not an absent variable, and a nonempty string must not be reinterpreted as a Boolean merely from its spelling.

Include dependencies from messages and the applicable session-ID template. A direct nonblank session ID bypasses that template, removing only requirements introduced solely by it; declarations and message uses remain relevant. The bypassed session template is also excluded from the inactive inventory.

Literal input references are discoverable without reading bodies, including undeclared references. Artifact content being unavailable does not by itself make discovery incomplete: it does so when the content affects which further requirements apply. Variable-derived names are discoverable when supported analysis can determine the exact name from the supplied context.

When missing variables, content-dependent decisions, unresolved names, or analysis limitations prevent determination, preserve known requirements and return successful incomplete discovery. Structured reasons should distinguish these situations without requiring diagnostic-text matching. Unsupported analysis is not a new prompt-validity rule. Completeness is asserted only when the supported analysis establishes the requirement set; it does not promise universal analysis of every valid Go template.

The selected practical analysis core and inactive inventory are described below. They do not change execution or expand the template language.

Effects, Errors, And Disclosure

Inspection uses captured prompt resources and needs no future artifact files, credential lookup, model access, capacity admission, or execution artifact creation. It does not prepare an output-validation schema or validate future input bytes. Necessary configuration/resource failures can still prevent engine construction or inspection; this is not a general resource preflight.

Use existing public selection/error identities wherever applicable, including invalid requests, missing prompt/profile selections, and ambiguous versions. Preserve ordinary context cancellation behavior. Discovery uncertainty is a result state, distinct from an operational or selection error. Expose any additional necessary classifications through the public facade, not diagnostic text parsing or internal types.

Results and safe diagnostics exclude credential values, artifact payloads, raw/rendered prompt content, and unrelated supplied variable values. Selected identities, input/variable names, and declared descriptions are intentional operator metadata. A resolved dynamic input identifier may equal a supplied variable value; reporting that identifier is necessary to describe the binding. Keep the disclosure exception limited to its role as an identifier, without echoing the variable map or the expression that produced it. Scriptorium owns the presentation and disclosure of this metadata to its callers.

A complete result describes requirements, not successful rendering or generation, future artifact validity, or provider availability. Incomplete results remain useful without authorizing consumers to treat unknown dependencies as absent.

Work Needed From The Baseline

Add public request/result values and the engine method through the existing facade. Put selection coordination and analysis in internal components with cohesive ownership. Reuse existing prompt/profile selection, session override, and template parsing semantics rather than introducing a second preparation path or a general-purpose template interpreter.

Make the captured templates and effective declaration metadata available to internal inspection. Retain any required metadata presence at the ConfigKit handoff. Apply ADR 0001: inspection interprets final effective definitions, not raw sources, configuration history, or intermediate artifacts. Operator provenance remains a separate explicit API.

Implement requirement aggregation, context-sensitive analysis, and safe public uncertainty/error conversion. Keep request-specific analysis separate from shared captured state. Prefer direct traversals and indexed aggregation; avoid enumerating every possible branch combination or repeated full-template scans. Predictable result ordering does not require a stronger complexity guarantee or a new global cache.

Add focused analyzer tests and public-facade contract coverage under the testing policy. Use real captured definitions and templates, synthetic variables, and explicit unavailable artifacts. Compare complete discovery with representative execution semantics. Test incomplete results separately from errors and verify disclosure boundaries and absence of artifact reads/provider calls. Do not retest ConfigKit's loading matrix.

When implementation lands, update GoDoc, the consumer guide, relevant format cross-references, and the internal component/source documents. Describe the analysis boundary and examples of complete, unsatisfied, and incomplete results in their canonical owners. Release notes explain the new capability; they do not own its contract. The implementation plan owns implementation sequencing.

Acceptance Outcomes

• A required transcript is reported exactly, allowing a consumer to detect a planned binding under the wrong name before the transcript exists.
• Literal undeclared helper references are discoverable without making their absence from declarations a new definition error.
• Declared required inputs remain required in inactive branches; optional declarations remain optional where no applicable use makes them required.
• Statically identifiable inactive undeclared references are optional entries with an inactive-use explanation; active and unresolved uses of the same name take precedence. These entries do not introduce variable requirements or incompleteness solely from skipped expressions.
• Supported variable-dependent branches and names agree with execution; unresolved content/name/analysis cases retain known entries and uncertainty.
• Version/profile/default selection and session-template bypass agree with execution without credential or generation-readiness checks.
• Variables are reported separately, with absent and empty supplied values interpreted according to the existing template semantics.
• Artifact-free complete results, including all-optional inventories, are distinguishable from empty partial discovery.
• Errors remain publicly classifiable; results and safe diagnostics disclose identifiers and declared metadata without unrelated content or secret values.
• Existing inspection and execution APIs retain their contracts. Pipeline planning, binding validation, CLI/HTTP exposure, presentation, JSON schemas, graph rendering, batch interfaces, and provider preflight remain downstream.

Selected Analysis Scope And Inventory

Initial Template Analysis Coverage

Implement a documented practical core: literal input calls, direct caller-variable access, ordinary if/with control flow, and simple variable-derived names and comparisons where their values are determinable. Account for template variable scope and short-circuit behavior for supported expressions. Additional constructs can be supported where straightforward; otherwise report incompleteness when they affect requirement discovery. The implementation plan specifies the exact expression families for this baseline. This covers the representative consumer needs while keeping completeness honest and avoiding a second general-purpose interpreter.

Inactive Undeclared Reference Inventory

Always include declarations and discovered active or unresolved dependencies. Also include statically identifiable undeclared references on known inactive paths as optional entries with an inactive-use explanation. This broader inventory helps operators explore prompt behavior while preserving the distinction between inventory and applicable dependencies.

Collect inactive names from syntax without evaluating skipped expressions. The initial scope covers literal helper names and equivalent parenthesized or piped literal forms. Skipped dynamic names need not be resolved, and skipped variable accesses do not become caller requirements or uncertainty reasons. Include known skipped branches and short-circuit operands, without expanding analysis to never-invoked template definitions or a bypassed session template. Preserve declared optional metadata and required declarations; when another use makes a name required or conditional, its inactive occurrence does not override that classification or imply that the name is inactive-only.