diff --git a/docs/roadmap/inspection.md b/docs/roadmap/inspection.md new file mode 100644 index 0000000..dca144a --- /dev/null +++ b/docs/roadmap/inspection.md @@ -0,0 +1,258 @@ +# Effective Prompt Requirement Inspection + +Status: Proposed feature scope; implementation has not started. The +recommendations under Open Decisions need agreement before implementation +planning. This document owns the target end state and migration scope, not a +staged implementation sequence. + +## Purpose + +Let consumers discover a selected prompt's caller requirements before its input +artifacts exist. Scriptorium can expose that information to operators and to +Narratio, which can compare its planned bindings with Promptkit's exact input +names before undertaking expensive pipeline work. + +The originating request is `docs/roadmap/scriptorium-feature-request.md`. +The capability is application-neutral: Promptkit describes prompt requirements; +consumers interpret them for their own configuration, presentation, and workflow. + +## Current Baseline + +At review baseline `b595dca`: + +- [InspectPrompt](../consumers/pkg-promptkit.md#inspect-a-prompt-before-preparation) + selects a captured prompt and reports declared inputs and output metadata. + It does not analyze templates or resolve the execution profile. +- [InspectProfile](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work) + reports one captured profile's effective target separately. +- [Prepare](../consumers/pkg-promptkit.md#prepare-without-model-execution) + selects the execution context, checks target requirements, loads artifacts + and applicable schemas, and renders messages. It needs actual input data. +- The configuration adapter captures final effective definitions, message + contents, and compiled message/session templates during engine construction. + Runtime rendering uses Go templates, string-valued caller variables, and the + `input` helper. Declared required inputs are checked independently of template + branches; active helper calls also require their named inputs. +- Declared input metadata currently uses string fields for content type and + description. The new result needs to identify a declaration and distinguish + available metadata from absent metadata, including effective presence where + necessary; current empty strings alone do not establish source-field presence. + +The [format reference](../formats.md#messages-and-templates), +[internal source document](../internal/sources.md), and existing GoDoc own the +implemented semantics. No existing operation provides artifact-free, +context-sensitive requirement discovery. + +## Target Public Operation + +Add a separate read-only engine operation, provisionally +`Engine.InspectRequirements`, with a dedicated request and result. Preserve +`InspectPrompt`, `InspectProfile`, preparation, and execution contracts. + +The request supplies a prompt ID, optional version, optional profile selector, +supplied template variables, and an optional direct session ID. It needs no +artifact references or contents, output path, API key, or execution settings +unrelated to discovery. + +Selection follows equivalent execution semantics: exact identities, omitted +version ambiguity, request profile or prompt default, and direct session-ID +normalization and bypass. Return selected prompt ID/version and effective +profile ID. Profile selection establishes identity, not generation readiness; +profiles that depend on later execution overrides need not be completed here. + +Use the configured engine's captured definitions. Source discovery and layering +remain consumer choices. Results describe that captured context, not a promise +about a later engine or later execution with changed variables. + +## Result And Requirement Meanings + +Expose public Go values with documented classifications and predictable ordering. +Exact symbol and field spelling belongs in the subsequent implementation plan +and GoDoc. No Promptkit JSON wire schema is part of this feature. + +The result contains: + +- Selected prompt and profile identities. +- Known input names, preserving execution identity, with required, optional, + or conditional classification for the supplied context. +- Declaration information, descriptions, and content-type metadata where + available. A template-discovered undeclared input must not look like a + declared input with known metadata. Report existing media-type metadata, + not an invented input schema or inferred format constraint. +- Variable requirements separately from artifact inputs, with requirement + classification and whether supplied values satisfy the known requirement. + Report presence and actual supported semantics, not the supplied values. +- A discovery-completeness indicator and structured uncertainty reasons, with + safe explanations and relevant identifiers where available. + +Required means the caller must supply the input for this context. A declared +required input stays required even if no active template references it. A +declared optional input becomes required when an active template use needs it. +Optional means it is known not to be required for this context; conditional +means its requirement depends on unresolved context. Do not use optional to +hide uncertainty. Known required declarations retain that classification even +when other requirements are conditional or undiscoverable. + +An unresolved dynamic name can refer to a declared optional input. Its absence +from recognized active uses alone does not establish that it is definitely +optional; classification and uncertainty must account for that possibility. + +Known input names should be deduplicated across declarations, messages, and +session templates. Aggregate uses without discarding an established required +use. Conditions need not expose raw expressions; machine-readable reason +categories and safe summaries are sufficient for basic consumer handling. + +Completeness describes discovery, not readiness or satisfaction. For example, +a missing variable used only to insert text can be a known unsatisfied +requirement while the complete input set is still discoverable. If that +variable selects an input name or branch, its absence can also make discovery +incomplete. An empty complete input set means no artifact bindings are needed; +an empty incomplete set does not establish that fact. + +## Context-Sensitive Discovery Boundary + +Analyze parsed templates rather than searching raw text or rendering with +placeholder artifacts. Use supplied variable values where they determine +branches and input names, following execution's presence, truthiness, scope, +and evaluation semantics. An empty string is not an absent variable, and a +nonempty string must not be reinterpreted as a Boolean merely from its spelling. + +Include dependencies from messages and the applicable session-ID template. +A direct nonblank session ID bypasses that template, removing only requirements +introduced solely by it; declarations and message uses remain relevant. + +Literal `input` references are discoverable without reading bodies, including +undeclared references. Artifact content being unavailable does not by itself +make discovery incomplete: it does so when the content affects which further +requirements apply. Variable-derived names are discoverable when supported +analysis can determine the exact name from the supplied context. + +When missing variables, content-dependent decisions, unresolved names, or +analysis limitations prevent determination, preserve known requirements and +return successful incomplete discovery. Structured reasons should distinguish +these situations without requiring diagnostic-text matching. Unsupported +analysis is not a new prompt-validity rule. Completeness is asserted only when +the supported analysis establishes the requirement set; it does not promise +universal analysis of every valid Go template. + +The initial analysis coverage and inactive-reference inventory are the Open +Decisions below. They do not change execution or expand the template language. + +## Effects, Errors, And Disclosure + +Inspection uses captured prompt resources and needs no future artifact files, +credential lookup, model access, capacity admission, or execution artifact +creation. It does not prepare an output-validation schema or validate future +input bytes. Necessary configuration/resource failures can still prevent engine +construction or inspection; this is not a general resource preflight. + +Use existing public selection/error identities wherever applicable, including +invalid requests, missing prompt/profile selections, and ambiguous versions. +Preserve ordinary context cancellation behavior. Discovery uncertainty is a +result state, distinct from an operational or selection error. Expose any +additional necessary classifications through the public facade, not diagnostic +text parsing or internal types. + +Results and safe diagnostics exclude credential values, artifact payloads, +raw/rendered prompt content, and unrelated supplied variable values. Selected +identities, input/variable names, and declared descriptions are intentional +operator metadata. A resolved dynamic input identifier may equal a supplied +variable value; reporting that identifier is necessary to describe the binding. +Keep the disclosure exception limited to its role as an identifier, without +echoing the variable map or the expression that produced it. Scriptorium owns +the presentation and disclosure of this metadata to its callers. + +A complete result describes requirements, not successful rendering or generation, +future artifact validity, or provider availability. Incomplete results remain +useful without authorizing consumers to treat unknown dependencies as absent. + +## Work Needed From The Baseline + +Add public request/result values and the engine method through the existing +facade. Put selection coordination and analysis in internal components with +cohesive ownership. Reuse existing prompt/profile selection, session override, +and template parsing semantics rather than introducing a second preparation +path or a general-purpose template interpreter. + +Make the captured templates and effective declaration metadata available to +internal inspection. Retain any required metadata presence at the ConfigKit +handoff. Apply [ADR 0001](../adr/0001-configuration-boundary.md): inspection +interprets final effective definitions, not raw sources, configuration history, +or intermediate artifacts. Operator provenance remains a separate explicit API. + +Implement requirement aggregation, context-sensitive analysis, and safe public +uncertainty/error conversion. Keep request-specific analysis separate from +shared captured state. Prefer direct traversals and indexed aggregation; avoid +enumerating every possible branch combination or repeated full-template scans. +Predictable result ordering does not require a stronger complexity guarantee +or a new global cache. + +Add focused analyzer tests and public-facade contract coverage under the +[testing policy](../policy/testing.md). Use real captured definitions and +templates, synthetic variables, and explicit unavailable artifacts. Compare +complete discovery with representative execution semantics. Test incomplete +results separately from errors and verify disclosure boundaries and absence +of artifact reads/provider calls. Do not retest ConfigKit's loading matrix. + +When implementation lands, update GoDoc, the consumer guide, relevant format +cross-references, and the internal component/source documents. Describe the +analysis boundary and examples of complete, unsatisfied, and incomplete results +in their canonical owners. Release notes explain the new capability; they do +not own its contract. A separate staged plan will own implementation sequencing. + +## Acceptance Outcomes + +- A required `transcript` is reported exactly, allowing a consumer to detect a + planned binding under the wrong name before the transcript exists. +- Literal undeclared helper references are discoverable without making their + absence from declarations a new definition error. +- Declared required inputs remain required in inactive branches; optional + declarations remain optional where no applicable use makes them required. +- Supported variable-dependent branches and names agree with execution; + unresolved content/name/analysis cases retain known entries and uncertainty. +- Version/profile/default selection and session-template bypass agree with + execution without credential or generation-readiness checks. +- Variables are reported separately, with absent and empty supplied values + interpreted according to the existing template semantics. +- No-input completeness is distinguishable from empty partial discovery. +- Errors remain publicly classifiable; results and safe diagnostics disclose + identifiers and declared metadata without unrelated content or secret values. +- Existing inspection and execution APIs retain their contracts. Pipeline + planning, binding validation, CLI/HTTP exposure, presentation, JSON schemas, + graph rendering, batch interfaces, and provider preflight remain downstream. + +## Open Decisions + +### Initial Template Analysis Coverage + +**Recommendation:** Implement a documented practical core: literal input calls, +direct caller-variable access, ordinary `if`/`with` control flow, and simple +variable-derived names and comparisons where their values are determinable. +Account for template variable scope and short-circuit behavior for supported +expressions. Additional constructs can be supported where straightforward; +otherwise report incompleteness when they affect requirement discovery. Specify +the exact supported expression families before writing the implementation plan. +This covers the representative consumer needs while keeping completeness +honest and avoiding a second general-purpose interpreter. + +**Alternative:** Support substantially broader Go-template evaluation now, +including loops, named templates, and more built-in pipelines. It would produce +complete results for more existing prompts, but increases semantic machinery, +maintenance, and parity-testing costs. It still cannot determine decisions based +on unavailable artifact contents. Prefer it if concrete consumer examples show +that the practical core would frequently return unhelpful partial results. + +### Inactive Undeclared Reference Inventory + +**Recommendation:** Always include declarations, plus template-discovered names +needed on active or unresolved paths. Omit undeclared names used solely in a +branch known to be inactive. Conditional names on unresolved paths remain +explicitly conditional. This keeps the result focused on the effective caller +requirements and avoids presenting irrelevant bindings as work to prepare. + +**Alternative:** Also list statically identifiable undeclared references on +known inactive paths, marking them optional with an inactive-use explanation. +This gives operators a broader inventory for exploring prompt behavior, but +adds irrelevant names and requires consumers to distinguish that inventory +from applicable dependencies. Either choice must preserve declared optional +metadata and must not classify a known inactive use as required.