diff --git a/docs/consumers/pkg-promptkit.md b/docs/consumers/pkg-promptkit.md index a28b487..d8b5ced 100644 --- a/docs/consumers/pkg-promptkit.md +++ b/docs/consumers/pkg-promptkit.md @@ -135,6 +135,34 @@ For programmatic profiles, [`OpenAICompatibleProfile`](../../profiles.go) converts ordinary OpenAI-compatible settings into a value accepted by `WithProfiles`. +### Inspect A Profile Before Prompt Work + +Use [`Engine.InspectProfile`](../../engine.go) to validate one configured +profile without constructing a synthetic prompt or placeholder inputs. It +resolves the profile's effective target but does not prepare or execute a +prompt: + +```go +inspection, err := engine.InspectProfile(ctx, profileID) +if err != nil { + return err +} + +target := inspection.EffectiveModelParams +if target.APIKeyEnv != "" { + // Apply application policy for the named environment variable. +} else if inspection.APIKeyRequired { + // Arrange a direct credential before later execution. +} +``` + +Use this configuration-time boundary when only the profile and its target need +checking. Use `Prepare` when the application also needs prompt, input, schema, +or rendering work; use prepared execution when that work must remain tied to a +later execution. Inspection reports credential requirements but leaves the +timing of credential enforcement to the application. The method's +[GoDoc](../../engine.go) owns its exact result and error contract. + ### Set A Per-Run Session And Reasoning Supply a direct session ID when one prompt should be correlated with a diff --git a/docs/formats.md b/docs/formats.md index 3ab14ed..01fd4d9 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -152,7 +152,7 @@ extra_params: | Field | Required | Meaning | | --- | --- | --- | | `id` | yes | Non-empty profile identifier. IDs must be unique within one source. | -| `backend` | unless `endpoint` is present | Backend registry ID. It is trimmed and registry membership is checked when the profile is prepared. | +| `backend` | unless `endpoint` is present | Backend registry ID. It is trimmed and registry membership is checked when the profile is prepared or inspected. | | `endpoint` | unless `backend` is present | Non-empty OpenAI-compatible base URL, including an API version path when required. When both connection fields are present, this overrides the backend endpoint without changing backend identity. | | `model` | yes | Non-empty provider model name. | | `temperature` | no | Number from 0 through 2. | @@ -219,7 +219,9 @@ defines how the effective settings are serialized. ### Source And Profile Precedence An explicit request profile ID takes precedence over the prompt's -`default_profile`. If neither is present, preparation fails. +`default_profile`. If neither is present, preparation fails. Exact profile +inspection instead takes one explicit profile ID and does not use a prompt +default. Profile sources resolve matching IDs in this order: @@ -231,6 +233,7 @@ A higher-precedence source falls back only when the profile is absent. An invalid matching profile is an error and does not fall back. In-memory `Profile` values follow the same ranges as YAML profiles. They use `APIKeyRequired` for request-scoped credentials instead of `api_key_env`. +Preparation and exact profile inspection use this same source precedence. ## Built-In Profile Catalog diff --git a/docs/internal/overview.md b/docs/internal/overview.md index 83ae8d5..77e9ffa 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -11,7 +11,7 @@ contributor workflow and validation. | Component | Implemented responsibility | References | | --- | --- | --- | -| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request and result values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, public error mapping, and engine-local assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) | +| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, public error mapping, and engine-local assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) | | `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) | | `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) | | `internal/backend` | Constructs each engine's immutable registry from the built-in OpenRouter definition and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) | @@ -27,7 +27,7 @@ contributor workflow and validation. | `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) | | `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates frozen validation plans for prepared execution. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) | | `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) | -| `internal/usecase` | Resolves backend, profile, and request settings and coordinates preparation, ordinary execution, and one-attempt prepared execution across internal sources, rendering, artifact loading, generation, validation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) | +| `internal/usecase` | Resolves profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, generation, validation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) | The root package assembles these internal components without exposing their representations. Consumers depend only on the root facade. diff --git a/docs/internal/runner.md b/docs/internal/runner.md index 7c7019b..04d55a2 100644 --- a/docs/internal/runner.md +++ b/docs/internal/runner.md @@ -28,6 +28,21 @@ constructor does not enable one. Each invocation carries its state in request, prepared-run, and result values. The runner has no durable run or session store. +## Shared Profile Selection + +The runner uses one profile-selection and target-resolution boundary for +ordinary preparation and exact profile inspection. Preparation first selects a +request profile or a prompt default; inspection begins with its required +explicit profile ID. Both then apply the ordinary source precedence, resolve a +named backend, and construct the effective target from framework, backend, and +profile values. + +Inspection stops after the resulting endpoint and model are structurally +validated. It does not check credential availability or perform prompt, +artifact, schema, rendering, admission, or model-client work. The root +[`Engine.InspectProfile`](../../engine.go) GoDoc owns the public operation's +exact contract. + ## Shared Preparation Pipeline `Prepare` and `Run` share one private preparation pipeline split at the point diff --git a/docs/internal/sources.md b/docs/internal/sources.md index f579494..c8fa62c 100644 --- a/docs/internal/sources.md +++ b/docs/internal/sources.md @@ -28,7 +28,12 @@ with fallback only when the primary reports that a profile is absent. Strict YAML decoding recognizes the optional `backend` field, trims its value, and requires a model plus at least one non-blank backend or endpoint. Loading does not check registry membership because the available registry belongs to the -assembled engine; the runner checks membership during preparation. +assembled engine; the runner checks membership during preparation and exact +profile inspection. + +Exact profile inspection performs one point-in-time lookup through those +profile sources and checks the resolved target without reading prompt, input, +or schema sources. It does not retain that lookup for a later execution. `internal/profile/builtin` embeds the maintained built-in profile catalog and can place a caller-selected repository ahead of that catalog. Every embedded @@ -79,7 +84,7 @@ captured root document. loading and hashing, session and message rendering, and target resolution. `RunPrepared` uses the retained source-derived state and validation plan; it does not reopen prompt, profile, input, or schema sources and does not rerender -the request. By contrast, ordinary `Prepare` produces an inspection value only: +the request. By contrast, ordinary `Prepare` produces a preparation value only: a later `Run` performs its own source resolution and preparation. The [validator tests](../../internal/validate/standard_validator_test.go) own diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 4b70f69..6e9dd64 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -33,10 +33,6 @@ consumers. ## Ideas -Prompt-independent profile inspection has been selected for active planning in -the [focused feature roadmap](profile-inspection.md). The remaining ideas are -still available for future selection. - ### Prompt-definition inspection Provide exact prompt-definition lookup without rendering, placeholder inputs, diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 3b9e6c6..a3f9e5b 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,6 +1,6 @@ # Prompt-Independent Profile Inspection Implementation Plan -**Status:** Ready for implementation. +**Status:** Complete. ## Purpose diff --git a/docs/roadmap/notarius-promptkit-wishlist.md b/docs/roadmap/notarius-promptkit-wishlist.md index beee476..e38e2cb 100644 --- a/docs/roadmap/notarius-promptkit-wishlist.md +++ b/docs/roadmap/notarius-promptkit-wishlist.md @@ -88,8 +88,9 @@ corresponds atomically to the actual execution. ## Priority 2: Prompt-Independent Profile Inspection -**Disposition:** Covered by the accepted -[prompt-independent profile inspection](profile-inspection.md) roadmap. +**Disposition:** Implemented as +[`Engine.InspectProfile`](../../engine.go). See the +[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work). ### Downstream need @@ -104,62 +105,22 @@ needs to determine whether: This validation should not require model generation. -### Current integration +### Previous integration -Notarius constructs a synthetic prompt using `testing/fstest.MapFS`, supplies a -dummy transcript, and calls `Engine.Prepare` solely to exercise profile and -backend resolution. This works, but prompt preparation is serving as a -substitute for a profile-inspection interface. - -### Requested capability - -Add a prompt-independent profile-resolution API, for example: - -```go -type ResolvedProfile struct { - ProfileID string - BackendID string - EffectiveTarget ExecutionTarget - APIKeyEnv string -} - -func (e *Engine) ResolveProfile( - ctx context.Context, - profileID string, -) (ResolvedProfile, error) -``` - -The returned shape may differ, but it should provide enough information for a -consumer to validate an explicit profile selection without inventing a prompt -or supplying placeholder inputs. - -### Design considerations - -- Resolve built-in, file-backed, and programmatic profiles using normal - PromptKit precedence. -- Validate that a referenced backend registration exists. -- Do not resolve, retain, or expose credential values. -- Report credential requirements, such as an environment-variable name, so - the consuming application can decide whether availability is required at - configuration-validation time or only at execution time. -- Return caller-owned values. -- Preserve typed or sentinel error classification for missing and invalid - profiles. -- Consider accepting an `ExecutionTargetOverride` if consumers need to inspect - the same effective target that a run-level override would produce. -- Enumeration of all profiles is not required for the Notarius use case; exact - lookup by ID is sufficient. +Before profile inspection was available, Notarius constructed a synthetic +prompt using `testing/fstest.MapFS`, supplied a dummy transcript, and called +`Engine.Prepare` solely to exercise profile and backend resolution. ### Value to Notarius -This would eliminate a synthetic production-only prompt fixture and establish -a direct, supported contract for configuration-time profile and backend -validation. +The implemented interface eliminates a synthetic production-only prompt +fixture and establishes a direct, supported contract for configuration-time +profile and backend validation. ## Priority 3: Semantic Execution-Target Fingerprints -**Disposition:** Deferred until prompt-independent profile inspection defines -the resolved target whose configuration identity would be fingerprinted. +**Disposition:** Deferred pending a separate semantic-equality design for +resolved execution targets. ### Downstream need @@ -348,11 +309,9 @@ If the PromptKit team chooses to pursue these ideas, the most useful order for Notarius would be: 1. Add atomic execution that returns prepared details and the completed result. -2. Add prompt-independent profile inspection. -3. Add a semantic execution-target digest, preferably as part of profile +2. Add a semantic execution-target digest, preferably alongside profile inspection. -4. Add a typed capacity error carrying backend identity. +3. Add a typed capacity error carrying backend identity. -The first two address concrete workarounds in current Notarius code. The third -would improve checkpoint correctness and reduce coupling. The fourth is -operational polish. +The first addresses a concrete execution workaround. The second would improve +checkpoint correctness and reduce coupling. The third is operational polish. diff --git a/docs/roadmap/profile-inspection.md b/docs/roadmap/profile-inspection.md index 8e55d70..09827be 100644 --- a/docs/roadmap/profile-inspection.md +++ b/docs/roadmap/profile-inspection.md @@ -1,6 +1,6 @@ # Prompt-Independent Profile Inspection -**Status:** Accepted. +**Status:** Complete. ## Purpose diff --git a/docs/roadmap/weatherreporter-promptkit-wishlist.md b/docs/roadmap/weatherreporter-promptkit-wishlist.md index fa4fc65..f7c4cf9 100644 --- a/docs/roadmap/weatherreporter-promptkit-wishlist.md +++ b/docs/roadmap/weatherreporter-promptkit-wishlist.md @@ -185,51 +185,23 @@ and move failures ahead of weather collection. ## Priority 3: Prompt-Independent Profile Inspection -**Disposition:** Covered by the accepted -[prompt-independent profile inspection](profile-inspection.md) roadmap. +**Disposition:** Implemented as +[`Engine.InspectProfile`](../../engine.go). See the +[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work). ### Downstream need Weatherreporter will allow operators to select an external PromptKit profile -source and may allow an explicit profile override. It should reject a missing -profile, unknown backend, malformed execution target, or unsatisfied credential -requirement before collecting weather data or writing report artifacts. - -### Current integration option - -Weatherreporter can validate an explicit profile by preparing one embedded -prompt with fixture input. Prompts that use their own default profiles can be -validated during their normal preparation stage. - -This couples configuration validation to one prompt and requires placeholder -input even when only profile and backend resolution are relevant. - -### Requested capability - -The prompt-independent `ResolveProfile` API proposed by the -[Notarius wishlist](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection) -would satisfy this need. It should resolve built-in, file-backed, and -programmatic profiles, validate backend membership, report credential -requirements without resolving credential values, and preserve typed error -classification. - -### Additional Weatherreporter considerations - -- An explicit application profile override should be inspectable without - selecting a report prompt. -- A prompt-definition inspection result may expose its default profile ID so - Weatherreporter can inspect that profile separately. -- Inspection should distinguish structural profile validity from current - credential availability so configuration validation can apply explicit - application policy. -- An optional execution-target override should be considered only if it - describes the same target that a later run will use. +source and may allow an explicit profile override. It needs to reject a missing +profile, unknown backend, or malformed execution target before collecting +weather data or writing report artifacts, and to apply its own policy to a +reported credential requirement. ### Value to Weatherreporter -This would improve fail-fast configuration validation and give operator-facing -errors direct profile and backend context. It is valuable but not required for -the initial migration. +The implemented interface improves fail-fast configuration validation and gives +operator-facing errors direct profile and backend context. It remains optional +for the initial migration. ## Priority 4: Eager Source Validation @@ -368,8 +340,8 @@ It should not block adoption. ### Semantic Execution-Target Fingerprints -**Disposition:** Deferred until prompt-independent profile inspection defines -the resolved target whose configuration identity would be fingerprinted. +**Disposition:** Deferred pending a separate semantic-equality design for +resolved execution targets. The semantic target digest proposed by the [Notarius wishlist](notarius-promptkit-wishlist.md#priority-3-semantic-execution-target-fingerprints) @@ -432,11 +404,10 @@ Weatherreporter would be: 1. Add executable preparation handles, ideally sharing implementation with an atomic detailed-run API. 2. Add prompt-definition inspection. -3. Add prompt-independent profile inspection. -4. Consider eager source validation after evaluating whether the two exact +3. Consider eager source validation after evaluating whether the two exact inspection APIs are sufficient. -5. Add structured generation errors. -6. Add structured capacity errors and semantic execution-target fingerprints +4. Add structured generation errors. +5. Add structured capacity errors and semantic execution-target fingerprints as lower-priority operational improvements. The first item removes the only material integration workaround. Prompt and