Document prompt inspection API
This commit is contained in:
@@ -40,6 +40,30 @@ validation, and default transport behavior. Source discovery, format
|
||||
validation, and profile precedence are defined by the
|
||||
[framework format reference](../formats.md).
|
||||
|
||||
## Inspect A Prompt Before Preparation
|
||||
|
||||
Use [`Engine.InspectPrompt`](../../engine.go) to check one configured prompt's
|
||||
declared inputs and output workflow without creating placeholder inputs or
|
||||
resolving a profile:
|
||||
|
||||
```go
|
||||
inspection, err := engine.InspectPrompt(ctx, "meeting.summary", "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, input := range inspection.Inputs {
|
||||
// Compare the declared input with application configuration.
|
||||
}
|
||||
```
|
||||
|
||||
Use this configuration-time boundary when the application needs only the
|
||||
declared prompt interface. Use `InspectProfile` separately when it must also
|
||||
check a configured profile. Use `Prepare` when it needs inputs, schemas, or
|
||||
rendered messages, and use prepared execution when that work must remain tied
|
||||
to later execution. The method's [GoDoc](../../engine.go) owns exact fields,
|
||||
hash, ownership, and error semantics.
|
||||
|
||||
## Prepare Without Model Execution
|
||||
|
||||
[`Engine.Prepare`](../../engine.go) resolves the selected prompt and profile,
|
||||
|
||||
@@ -58,6 +58,11 @@ When a request omits a version, the selected prompt ID must identify exactly
|
||||
one definition. When it supplies a version, the ID and version pair must be
|
||||
unique.
|
||||
|
||||
Exact prompt inspection uses this same configured source, strict decoding,
|
||||
referenced content-file resolution, and ID/version selection. It reports the
|
||||
selected definition's declared metadata without changing the prompt format or
|
||||
executing the definition.
|
||||
|
||||
### Inputs
|
||||
|
||||
Each `inputs` item has these fields:
|
||||
|
||||
@@ -11,7 +11,7 @@ contributor workflow and validation.
|
||||
|
||||
| Component | Implemented responsibility | References |
|
||||
| --- | --- | --- |
|
||||
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, public error mapping, and engine-local assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
|
||||
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, public error mapping, and engine-local assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
|
||||
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
|
||||
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
|
||||
| `internal/backend` | Constructs each engine's immutable registry from the built-in OpenRouter definition and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
|
||||
@@ -27,7 +27,7 @@ contributor workflow and validation.
|
||||
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
|
||||
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates frozen validation plans for prepared execution. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
|
||||
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
|
||||
| `internal/usecase` | Resolves profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, generation, validation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
||||
| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, generation, validation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
||||
|
||||
The root package assembles these internal components without exposing their
|
||||
representations. Consumers depend only on the root facade.
|
||||
|
||||
@@ -28,6 +28,19 @@ constructor does not enable one.
|
||||
Each invocation carries its state in request, prepared-run, and result values.
|
||||
The runner has no durable run or session store.
|
||||
|
||||
## Shared Prompt Selection
|
||||
|
||||
The runner uses one prompt-selection and hashing boundary for ordinary
|
||||
preparation and exact prompt inspection. Preparation retains its early
|
||||
request-ID check before direct-session normalization; both operations then use
|
||||
the configured prompt repository to select one definition, load referenced
|
||||
message content, and calculate the same prompt hash.
|
||||
|
||||
Inspection stops after that structural lookup. It does not parse templates or
|
||||
touch profile, artifact, schema, renderer, validator, admission, or model
|
||||
collaborators. The root [`Engine.InspectPrompt`](../../engine.go) GoDoc owns
|
||||
the public operation's exact contract.
|
||||
|
||||
## Shared Profile Selection
|
||||
|
||||
The runner uses one profile-selection and target-resolution boundary for
|
||||
|
||||
@@ -16,6 +16,11 @@ validation modes, built-in catalog, and source precedence.
|
||||
definitions, selects an ID and optional version, and resolves file-backed
|
||||
message content within the selected operating-system or `fs.FS` source.
|
||||
|
||||
Exact prompt inspection performs one point-in-time lookup through that same
|
||||
repository and validates referenced message content before returning declared
|
||||
metadata. It does not parse templates or read profile, input, or schema
|
||||
sources, and it does not retain the definition for a later execution.
|
||||
|
||||
Its package tests own prompt selection, strict decoding, definition validation,
|
||||
duplicate detection, and source containment:
|
||||
[prompt-definition repository tests](../../internal/promptdef/repository_test.go).
|
||||
|
||||
@@ -33,10 +33,6 @@ consumers.
|
||||
|
||||
## Ideas
|
||||
|
||||
Prompt-definition inspection has been selected for active planning in the
|
||||
[focused feature roadmap](prompt-inspection.md). The remaining idea is still
|
||||
available for future selection.
|
||||
|
||||
### Structured capacity errors
|
||||
|
||||
Add safe structured context to backend admission rejection, as requested by
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Prompt-Definition Inspection Implementation Plan
|
||||
|
||||
**Status:** Accepted.
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Prompt-Definition Inspection
|
||||
|
||||
**Status:** Accepted.
|
||||
**Status:** Complete.
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
@@ -106,8 +106,9 @@ describes the actual execution.
|
||||
|
||||
## Priority 2: Prompt-Definition Inspection
|
||||
|
||||
**Disposition:** Accepted into the
|
||||
[prompt-definition inspection](prompt-inspection.md) feature roadmap.
|
||||
**Disposition:** Implemented as
|
||||
[`Engine.InspectPrompt`](../../engine.go). See the
|
||||
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-prompt-before-preparation).
|
||||
|
||||
### Downstream need
|
||||
|
||||
@@ -123,65 +124,22 @@ response schemas that implement those reports. It needs to validate that the
|
||||
report registry and embedded prompt corpus agree before weather collection or
|
||||
provider execution.
|
||||
|
||||
### Current integration option
|
||||
### Previous integration option
|
||||
|
||||
Weatherreporter can maintain synthetic data-package fixtures and call
|
||||
`Engine.Prepare` for every report prompt during tests. Runtime validation can
|
||||
also occur through the ordinary per-report preparation stage.
|
||||
Before prompt inspection was available, Weatherreporter could maintain
|
||||
synthetic data-package fixtures and call `Engine.Prepare` for every report
|
||||
prompt during tests. Runtime validation could also occur through the ordinary
|
||||
per-report preparation stage.
|
||||
|
||||
This works, but it requires complete placeholder inputs and profile resolution
|
||||
when the application primarily wants to inspect prompt identity and declared
|
||||
contracts.
|
||||
|
||||
### Requested capability
|
||||
|
||||
Add exact prompt-definition lookup without rendering or generation:
|
||||
|
||||
```go
|
||||
type PromptInfo struct {
|
||||
PromptID string
|
||||
PromptVersion string
|
||||
PromptHash string
|
||||
DefaultProfileID string
|
||||
Inputs []InputDefinition
|
||||
OutputContract OutputContract
|
||||
}
|
||||
|
||||
func (e *Engine) ResolvePrompt(
|
||||
ctx context.Context,
|
||||
promptID string,
|
||||
promptVersion string,
|
||||
) (PromptInfo, error)
|
||||
```
|
||||
|
||||
The exact returned shape may differ. Weatherreporter needs enough information
|
||||
to verify prompt existence, version selection, declared inputs, default
|
||||
profile identity, output format, validation mode, and schema selection without
|
||||
supplying synthetic prompt input.
|
||||
|
||||
### Design considerations
|
||||
|
||||
- Use ordinary PromptKit prompt-source precedence and exact ID/version
|
||||
selection.
|
||||
- Fully load and structurally validate the selected prompt definition.
|
||||
- Validate referenced prompt content files without rendering their templates.
|
||||
- Resolve and validate the selected output contract and schema reference where
|
||||
practical.
|
||||
- Return an opaque prompt-definition equality value rather than raw source
|
||||
bytes.
|
||||
- Do not return rendered messages, schema bodies, profile credentials, or
|
||||
another source of sensitive content.
|
||||
- Preserve typed or sentinel errors for missing and invalid prompts.
|
||||
- Return caller-owned values.
|
||||
- Enumeration of all known prompts is not required for Weatherreporter; exact
|
||||
lookup is sufficient.
|
||||
This required complete placeholder inputs and profile resolution when the
|
||||
application primarily wanted to inspect prompt identity and declared contracts.
|
||||
|
||||
### Value to Weatherreporter
|
||||
|
||||
This would let Weatherreporter directly verify that every report prompt
|
||||
exists, requires the curated `data_package` input, and declares the expected
|
||||
Markdown or JSON Schema output contract. It would reduce synthetic test setup
|
||||
and move failures ahead of weather collection.
|
||||
The implemented interface lets Weatherreporter directly verify that every
|
||||
report prompt exists, requires the curated `data_package` input, and declares
|
||||
the expected Markdown or JSON Schema output contract. It reduces synthetic
|
||||
test setup and moves failures ahead of weather collection.
|
||||
|
||||
## Priority 3: Prompt-Independent Profile Inspection
|
||||
|
||||
@@ -403,11 +361,10 @@ Weatherreporter would be:
|
||||
|
||||
1. Add executable preparation handles, ideally sharing implementation with an
|
||||
atomic detailed-run API.
|
||||
2. Add prompt-definition inspection.
|
||||
3. Consider eager source validation after evaluating whether the two exact
|
||||
2. Consider eager source validation after evaluating whether the two exact
|
||||
inspection APIs are sufficient.
|
||||
4. Add structured generation errors.
|
||||
5. Add structured capacity errors and semantic execution-target fingerprints
|
||||
3. Add structured generation errors.
|
||||
4. Add structured capacity errors and semantic execution-target fingerprints
|
||||
as lower-priority operational improvements.
|
||||
|
||||
The first item removes the only material integration workaround. Prompt and
|
||||
|
||||
Reference in New Issue
Block a user