promptkit/docs/roadmap/inspection.md

Effective Prompt Requirement Inspection

Status: Implementation planned; the feature is not implemented. The implementation plan owns the ordered migration steps and uses the recommended scope below. Open Decisions still records the alternative scope choices for confirmation. This document owns the target end state and migration scope.

Purpose

Let consumers discover a selected prompt's caller requirements before its input artifacts exist. Scriptorium can expose that information to operators and to Narratio, which can compare its planned bindings with Promptkit's exact input names before undertaking expensive pipeline work.

The originating request is docs/roadmap/scriptorium-feature-request.md. The capability is application-neutral: Promptkit describes prompt requirements; consumers interpret them for their own configuration, presentation, and workflow.

Current Baseline

At review baseline b595dca:

• InspectPrompt selects a captured prompt and reports declared inputs and output metadata. It does not analyze templates or resolve the execution profile.
• InspectProfile reports one captured profile's effective target separately.
• Prepare selects the execution context, checks target requirements, loads artifacts and applicable schemas, and renders messages. It needs actual input data.
• The configuration adapter captures final effective definitions, message contents, and compiled message/session templates during engine construction. Runtime rendering uses Go templates, string-valued caller variables, and the input helper. Declared required inputs are checked independently of template branches; active helper calls also require their named inputs.
• Declared input metadata currently uses string fields for content type and description. The new result needs to identify a declaration and distinguish available metadata from absent metadata, including effective presence where necessary; current empty strings alone do not establish source-field presence.

The format reference, internal source document, and existing GoDoc own the implemented semantics. No existing operation provides artifact-free, context-sensitive requirement discovery.

Target Public Operation

Add a separate read-only engine operation, planned as Engine.InspectRequirements, with a dedicated request and result. Preserve InspectPrompt, InspectProfile, preparation, and execution contracts.

The request supplies a prompt ID, optional version, optional profile selector, supplied template variables, and an optional direct session ID. It needs no artifact references or contents, output path, API key, or execution settings unrelated to discovery.

Selection follows equivalent execution semantics: exact identities, omitted version ambiguity, request profile or prompt default, and direct session-ID normalization and bypass. Return selected prompt ID/version and effective profile ID. Profile selection establishes identity, not generation readiness; profiles that depend on later execution overrides need not be completed here.

Use the configured engine's captured definitions. Source discovery and layering remain consumer choices. Results describe that captured context, not a promise about a later engine or later execution with changed variables.

Result And Requirement Meanings

Expose public Go values with documented classifications and predictable ordering. Exact symbol and field spelling belongs in the implementation plan and GoDoc. No Promptkit JSON wire schema is part of this feature.

The result contains:

• Selected prompt and profile identities.
• Known input names, preserving execution identity, with required, optional, or conditional classification for the supplied context.
• Declaration information, descriptions, and content-type metadata where available. A template-discovered undeclared input must not look like a declared input with known metadata. Report existing media-type metadata, not an invented input schema or inferred format constraint.
• Variable requirements separately from artifact inputs, with requirement classification and whether supplied values satisfy the known requirement. Report presence and actual supported semantics, not the supplied values.
• A discovery-completeness indicator and structured uncertainty reasons, with safe explanations and relevant identifiers where available.

Required means the caller must supply the input for this context. A declared required input stays required even if no active template references it. A declared optional input becomes required when an active template use needs it. Optional means it is known not to be required for this context; conditional means its requirement depends on unresolved context. Do not use optional to hide uncertainty. Known required declarations retain that classification even when other requirements are conditional or undiscoverable.

An unresolved dynamic name can refer to a declared optional input. Its absence from recognized active uses alone does not establish that it is definitely optional; classification and uncertainty must account for that possibility.

Known input names should be deduplicated across declarations, messages, and session templates. Aggregate uses without discarding an established required use. Conditions need not expose raw expressions; machine-readable reason categories and safe summaries are sufficient for basic consumer handling.

Completeness describes discovery, not readiness or satisfaction. For example, a missing variable used only to insert text can be a known unsatisfied requirement while the complete input set is still discoverable. If that variable selects an input name or branch, its absence can also make discovery incomplete. An empty complete input set means no artifact bindings are needed; an empty incomplete set does not establish that fact.

Context-Sensitive Discovery Boundary

Analyze parsed templates rather than searching raw text or rendering with placeholder artifacts. Use supplied variable values where they determine branches and input names, following execution's presence, truthiness, scope, and evaluation semantics. An empty string is not an absent variable, and a nonempty string must not be reinterpreted as a Boolean merely from its spelling.

Include dependencies from messages and the applicable session-ID template. A direct nonblank session ID bypasses that template, removing only requirements introduced solely by it; declarations and message uses remain relevant.

Literal input references are discoverable without reading bodies, including undeclared references. Artifact content being unavailable does not by itself make discovery incomplete: it does so when the content affects which further requirements apply. Variable-derived names are discoverable when supported analysis can determine the exact name from the supplied context.

When missing variables, content-dependent decisions, unresolved names, or analysis limitations prevent determination, preserve known requirements and return successful incomplete discovery. Structured reasons should distinguish these situations without requiring diagnostic-text matching. Unsupported analysis is not a new prompt-validity rule. Completeness is asserted only when the supported analysis establishes the requirement set; it does not promise universal analysis of every valid Go template.

The initial analysis coverage and inactive-reference inventory are the Open Decisions below. They do not change execution or expand the template language.

Effects, Errors, And Disclosure

Inspection uses captured prompt resources and needs no future artifact files, credential lookup, model access, capacity admission, or execution artifact creation. It does not prepare an output-validation schema or validate future input bytes. Necessary configuration/resource failures can still prevent engine construction or inspection; this is not a general resource preflight.

Use existing public selection/error identities wherever applicable, including invalid requests, missing prompt/profile selections, and ambiguous versions. Preserve ordinary context cancellation behavior. Discovery uncertainty is a result state, distinct from an operational or selection error. Expose any additional necessary classifications through the public facade, not diagnostic text parsing or internal types.

Results and safe diagnostics exclude credential values, artifact payloads, raw/rendered prompt content, and unrelated supplied variable values. Selected identities, input/variable names, and declared descriptions are intentional operator metadata. A resolved dynamic input identifier may equal a supplied variable value; reporting that identifier is necessary to describe the binding. Keep the disclosure exception limited to its role as an identifier, without echoing the variable map or the expression that produced it. Scriptorium owns the presentation and disclosure of this metadata to its callers.

A complete result describes requirements, not successful rendering or generation, future artifact validity, or provider availability. Incomplete results remain useful without authorizing consumers to treat unknown dependencies as absent.

Work Needed From The Baseline

Add public request/result values and the engine method through the existing facade. Put selection coordination and analysis in internal components with cohesive ownership. Reuse existing prompt/profile selection, session override, and template parsing semantics rather than introducing a second preparation path or a general-purpose template interpreter.

Make the captured templates and effective declaration metadata available to internal inspection. Retain any required metadata presence at the ConfigKit handoff. Apply ADR 0001: inspection interprets final effective definitions, not raw sources, configuration history, or intermediate artifacts. Operator provenance remains a separate explicit API.

Implement requirement aggregation, context-sensitive analysis, and safe public uncertainty/error conversion. Keep request-specific analysis separate from shared captured state. Prefer direct traversals and indexed aggregation; avoid enumerating every possible branch combination or repeated full-template scans. Predictable result ordering does not require a stronger complexity guarantee or a new global cache.

Add focused analyzer tests and public-facade contract coverage under the testing policy. Use real captured definitions and templates, synthetic variables, and explicit unavailable artifacts. Compare complete discovery with representative execution semantics. Test incomplete results separately from errors and verify disclosure boundaries and absence of artifact reads/provider calls. Do not retest ConfigKit's loading matrix.

When implementation lands, update GoDoc, the consumer guide, relevant format cross-references, and the internal component/source documents. Describe the analysis boundary and examples of complete, unsatisfied, and incomplete results in their canonical owners. Release notes explain the new capability; they do not own its contract. The implementation plan owns implementation sequencing.

Acceptance Outcomes

• A required transcript is reported exactly, allowing a consumer to detect a planned binding under the wrong name before the transcript exists.
• Literal undeclared helper references are discoverable without making their absence from declarations a new definition error.
• Declared required inputs remain required in inactive branches; optional declarations remain optional where no applicable use makes them required.
• Supported variable-dependent branches and names agree with execution; unresolved content/name/analysis cases retain known entries and uncertainty.
• Version/profile/default selection and session-template bypass agree with execution without credential or generation-readiness checks.
• Variables are reported separately, with absent and empty supplied values interpreted according to the existing template semantics.
• No-input completeness is distinguishable from empty partial discovery.
• Errors remain publicly classifiable; results and safe diagnostics disclose identifiers and declared metadata without unrelated content or secret values.
• Existing inspection and execution APIs retain their contracts. Pipeline planning, binding validation, CLI/HTTP exposure, presentation, JSON schemas, graph rendering, batch interfaces, and provider preflight remain downstream.

Open Decisions

Initial Template Analysis Coverage

Recommendation: Implement a documented practical core: literal input calls, direct caller-variable access, ordinary if/with control flow, and simple variable-derived names and comparisons where their values are determinable. Account for template variable scope and short-circuit behavior for supported expressions. Additional constructs can be supported where straightforward; otherwise report incompleteness when they affect requirement discovery. The implementation plan specifies the exact expression families for this baseline. This covers the representative consumer needs while keeping completeness honest and avoiding a second general-purpose interpreter.

Alternative: Support substantially broader Go-template evaluation now, including loops, named templates, and more built-in pipelines. It would produce complete results for more existing prompts, but increases semantic machinery, maintenance, and parity-testing costs. It still cannot determine decisions based on unavailable artifact contents. Prefer it if concrete consumer examples show that the practical core would frequently return unhelpful partial results.

Inactive Undeclared Reference Inventory

Recommendation: Always include declarations, plus template-discovered names needed on active or unresolved paths. Omit undeclared names used solely in a branch known to be inactive. Conditional names on unresolved paths remain explicitly conditional. This keeps the result focused on the effective caller requirements and avoids presenting irrelevant bindings as work to prepare.

Alternative: Also list statically identifiable undeclared references on known inactive paths, marking them optional with an inactive-use explanation. This gives operators a broader inventory for exploring prompt behavior, but adds irrelevant names and requires consumers to distinguish that inventory from applicable dependencies. Either choice must preserve declared optional metadata and must not classify a known inactive use as required.