Document prompt inspection API
This commit is contained in:
@@ -40,6 +40,30 @@ validation, and default transport behavior. Source discovery, format
|
|||||||
validation, and profile precedence are defined by the
|
validation, and profile precedence are defined by the
|
||||||
[framework format reference](../formats.md).
|
[framework format reference](../formats.md).
|
||||||
|
|
||||||
|
## Inspect A Prompt Before Preparation
|
||||||
|
|
||||||
|
Use [`Engine.InspectPrompt`](../../engine.go) to check one configured prompt's
|
||||||
|
declared inputs and output workflow without creating placeholder inputs or
|
||||||
|
resolving a profile:
|
||||||
|
|
||||||
|
```go
|
||||||
|
inspection, err := engine.InspectPrompt(ctx, "meeting.summary", "")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, input := range inspection.Inputs {
|
||||||
|
// Compare the declared input with application configuration.
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Use this configuration-time boundary when the application needs only the
|
||||||
|
declared prompt interface. Use `InspectProfile` separately when it must also
|
||||||
|
check a configured profile. Use `Prepare` when it needs inputs, schemas, or
|
||||||
|
rendered messages, and use prepared execution when that work must remain tied
|
||||||
|
to later execution. The method's [GoDoc](../../engine.go) owns exact fields,
|
||||||
|
hash, ownership, and error semantics.
|
||||||
|
|
||||||
## Prepare Without Model Execution
|
## Prepare Without Model Execution
|
||||||
|
|
||||||
[`Engine.Prepare`](../../engine.go) resolves the selected prompt and profile,
|
[`Engine.Prepare`](../../engine.go) resolves the selected prompt and profile,
|
||||||
|
|||||||
@@ -58,6 +58,11 @@ When a request omits a version, the selected prompt ID must identify exactly
|
|||||||
one definition. When it supplies a version, the ID and version pair must be
|
one definition. When it supplies a version, the ID and version pair must be
|
||||||
unique.
|
unique.
|
||||||
|
|
||||||
|
Exact prompt inspection uses this same configured source, strict decoding,
|
||||||
|
referenced content-file resolution, and ID/version selection. It reports the
|
||||||
|
selected definition's declared metadata without changing the prompt format or
|
||||||
|
executing the definition.
|
||||||
|
|
||||||
### Inputs
|
### Inputs
|
||||||
|
|
||||||
Each `inputs` item has these fields:
|
Each `inputs` item has these fields:
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ contributor workflow and validation.
|
|||||||
|
|
||||||
| Component | Implemented responsibility | References |
|
| Component | Implemented responsibility | References |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, public error mapping, and engine-local assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
|
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, public error mapping, and engine-local assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
|
||||||
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
|
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
|
||||||
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
|
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
|
||||||
| `internal/backend` | Constructs each engine's immutable registry from the built-in OpenRouter definition and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
|
| `internal/backend` | Constructs each engine's immutable registry from the built-in OpenRouter definition and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
|
||||||
@@ -27,7 +27,7 @@ contributor workflow and validation.
|
|||||||
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
|
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
|
||||||
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates frozen validation plans for prepared execution. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
|
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates frozen validation plans for prepared execution. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
|
||||||
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
|
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
|
||||||
| `internal/usecase` | Resolves profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, generation, validation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, generation, validation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
||||||
|
|
||||||
The root package assembles these internal components without exposing their
|
The root package assembles these internal components without exposing their
|
||||||
representations. Consumers depend only on the root facade.
|
representations. Consumers depend only on the root facade.
|
||||||
|
|||||||
@@ -28,6 +28,19 @@ constructor does not enable one.
|
|||||||
Each invocation carries its state in request, prepared-run, and result values.
|
Each invocation carries its state in request, prepared-run, and result values.
|
||||||
The runner has no durable run or session store.
|
The runner has no durable run or session store.
|
||||||
|
|
||||||
|
## Shared Prompt Selection
|
||||||
|
|
||||||
|
The runner uses one prompt-selection and hashing boundary for ordinary
|
||||||
|
preparation and exact prompt inspection. Preparation retains its early
|
||||||
|
request-ID check before direct-session normalization; both operations then use
|
||||||
|
the configured prompt repository to select one definition, load referenced
|
||||||
|
message content, and calculate the same prompt hash.
|
||||||
|
|
||||||
|
Inspection stops after that structural lookup. It does not parse templates or
|
||||||
|
touch profile, artifact, schema, renderer, validator, admission, or model
|
||||||
|
collaborators. The root [`Engine.InspectPrompt`](../../engine.go) GoDoc owns
|
||||||
|
the public operation's exact contract.
|
||||||
|
|
||||||
## Shared Profile Selection
|
## Shared Profile Selection
|
||||||
|
|
||||||
The runner uses one profile-selection and target-resolution boundary for
|
The runner uses one profile-selection and target-resolution boundary for
|
||||||
|
|||||||
@@ -16,6 +16,11 @@ validation modes, built-in catalog, and source precedence.
|
|||||||
definitions, selects an ID and optional version, and resolves file-backed
|
definitions, selects an ID and optional version, and resolves file-backed
|
||||||
message content within the selected operating-system or `fs.FS` source.
|
message content within the selected operating-system or `fs.FS` source.
|
||||||
|
|
||||||
|
Exact prompt inspection performs one point-in-time lookup through that same
|
||||||
|
repository and validates referenced message content before returning declared
|
||||||
|
metadata. It does not parse templates or read profile, input, or schema
|
||||||
|
sources, and it does not retain the definition for a later execution.
|
||||||
|
|
||||||
Its package tests own prompt selection, strict decoding, definition validation,
|
Its package tests own prompt selection, strict decoding, definition validation,
|
||||||
duplicate detection, and source containment:
|
duplicate detection, and source containment:
|
||||||
[prompt-definition repository tests](../../internal/promptdef/repository_test.go).
|
[prompt-definition repository tests](../../internal/promptdef/repository_test.go).
|
||||||
|
|||||||
@@ -33,10 +33,6 @@ consumers.
|
|||||||
|
|
||||||
## Ideas
|
## Ideas
|
||||||
|
|
||||||
Prompt-definition inspection has been selected for active planning in the
|
|
||||||
[focused feature roadmap](prompt-inspection.md). The remaining idea is still
|
|
||||||
available for future selection.
|
|
||||||
|
|
||||||
### Structured capacity errors
|
### Structured capacity errors
|
||||||
|
|
||||||
Add safe structured context to backend admission rejection, as requested by
|
Add safe structured context to backend admission rejection, as requested by
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Prompt-Definition Inspection Implementation Plan
|
# Prompt-Definition Inspection Implementation Plan
|
||||||
|
|
||||||
**Status:** Accepted.
|
**Status:** Complete.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Prompt-Definition Inspection
|
# Prompt-Definition Inspection
|
||||||
|
|
||||||
**Status:** Accepted.
|
**Status:** Complete.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
|
|||||||
@@ -106,8 +106,9 @@ describes the actual execution.
|
|||||||
|
|
||||||
## Priority 2: Prompt-Definition Inspection
|
## Priority 2: Prompt-Definition Inspection
|
||||||
|
|
||||||
**Disposition:** Accepted into the
|
**Disposition:** Implemented as
|
||||||
[prompt-definition inspection](prompt-inspection.md) feature roadmap.
|
[`Engine.InspectPrompt`](../../engine.go). See the
|
||||||
|
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-prompt-before-preparation).
|
||||||
|
|
||||||
### Downstream need
|
### Downstream need
|
||||||
|
|
||||||
@@ -123,65 +124,22 @@ response schemas that implement those reports. It needs to validate that the
|
|||||||
report registry and embedded prompt corpus agree before weather collection or
|
report registry and embedded prompt corpus agree before weather collection or
|
||||||
provider execution.
|
provider execution.
|
||||||
|
|
||||||
### Current integration option
|
### Previous integration option
|
||||||
|
|
||||||
Weatherreporter can maintain synthetic data-package fixtures and call
|
Before prompt inspection was available, Weatherreporter could maintain
|
||||||
`Engine.Prepare` for every report prompt during tests. Runtime validation can
|
synthetic data-package fixtures and call `Engine.Prepare` for every report
|
||||||
also occur through the ordinary per-report preparation stage.
|
prompt during tests. Runtime validation could also occur through the ordinary
|
||||||
|
per-report preparation stage.
|
||||||
|
|
||||||
This works, but it requires complete placeholder inputs and profile resolution
|
This required complete placeholder inputs and profile resolution when the
|
||||||
when the application primarily wants to inspect prompt identity and declared
|
application primarily wanted to inspect prompt identity and declared contracts.
|
||||||
contracts.
|
|
||||||
|
|
||||||
### Requested capability
|
|
||||||
|
|
||||||
Add exact prompt-definition lookup without rendering or generation:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type PromptInfo struct {
|
|
||||||
PromptID string
|
|
||||||
PromptVersion string
|
|
||||||
PromptHash string
|
|
||||||
DefaultProfileID string
|
|
||||||
Inputs []InputDefinition
|
|
||||||
OutputContract OutputContract
|
|
||||||
}
|
|
||||||
|
|
||||||
func (e *Engine) ResolvePrompt(
|
|
||||||
ctx context.Context,
|
|
||||||
promptID string,
|
|
||||||
promptVersion string,
|
|
||||||
) (PromptInfo, error)
|
|
||||||
```
|
|
||||||
|
|
||||||
The exact returned shape may differ. Weatherreporter needs enough information
|
|
||||||
to verify prompt existence, version selection, declared inputs, default
|
|
||||||
profile identity, output format, validation mode, and schema selection without
|
|
||||||
supplying synthetic prompt input.
|
|
||||||
|
|
||||||
### Design considerations
|
|
||||||
|
|
||||||
- Use ordinary PromptKit prompt-source precedence and exact ID/version
|
|
||||||
selection.
|
|
||||||
- Fully load and structurally validate the selected prompt definition.
|
|
||||||
- Validate referenced prompt content files without rendering their templates.
|
|
||||||
- Resolve and validate the selected output contract and schema reference where
|
|
||||||
practical.
|
|
||||||
- Return an opaque prompt-definition equality value rather than raw source
|
|
||||||
bytes.
|
|
||||||
- Do not return rendered messages, schema bodies, profile credentials, or
|
|
||||||
another source of sensitive content.
|
|
||||||
- Preserve typed or sentinel errors for missing and invalid prompts.
|
|
||||||
- Return caller-owned values.
|
|
||||||
- Enumeration of all known prompts is not required for Weatherreporter; exact
|
|
||||||
lookup is sufficient.
|
|
||||||
|
|
||||||
### Value to Weatherreporter
|
### Value to Weatherreporter
|
||||||
|
|
||||||
This would let Weatherreporter directly verify that every report prompt
|
The implemented interface lets Weatherreporter directly verify that every
|
||||||
exists, requires the curated `data_package` input, and declares the expected
|
report prompt exists, requires the curated `data_package` input, and declares
|
||||||
Markdown or JSON Schema output contract. It would reduce synthetic test setup
|
the expected Markdown or JSON Schema output contract. It reduces synthetic
|
||||||
and move failures ahead of weather collection.
|
test setup and moves failures ahead of weather collection.
|
||||||
|
|
||||||
## Priority 3: Prompt-Independent Profile Inspection
|
## Priority 3: Prompt-Independent Profile Inspection
|
||||||
|
|
||||||
@@ -403,11 +361,10 @@ Weatherreporter would be:
|
|||||||
|
|
||||||
1. Add executable preparation handles, ideally sharing implementation with an
|
1. Add executable preparation handles, ideally sharing implementation with an
|
||||||
atomic detailed-run API.
|
atomic detailed-run API.
|
||||||
2. Add prompt-definition inspection.
|
2. Consider eager source validation after evaluating whether the two exact
|
||||||
3. Consider eager source validation after evaluating whether the two exact
|
|
||||||
inspection APIs are sufficient.
|
inspection APIs are sufficient.
|
||||||
4. Add structured generation errors.
|
3. Add structured generation errors.
|
||||||
5. Add structured capacity errors and semantic execution-target fingerprints
|
4. Add structured capacity errors and semantic execution-target fingerprints
|
||||||
as lower-priority operational improvements.
|
as lower-priority operational improvements.
|
||||||
|
|
||||||
The first item removes the only material integration workaround. Prompt and
|
The first item removes the only material integration workaround. Prompt and
|
||||||
|
|||||||
Reference in New Issue
Block a user