From 369ab5392de49b5b340b1ac52724878ddb45fd17 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 30 Jul 2026 19:42:51 +0000 Subject: [PATCH] Add feature roadmap and implementation plan for profile inspection API --- docs/roadmap/future.md | 20 +- docs/roadmap/implementation.md | 1106 ++++++++--------- docs/roadmap/notarius-promptkit-wishlist.md | 4 +- docs/roadmap/profile-inspection.md | 246 ++++ .../weatherreporter-promptkit-wishlist.md | 4 +- 5 files changed, 779 insertions(+), 601 deletions(-) create mode 100644 docs/roadmap/profile-inspection.md diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 555a907..4b70f69 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -33,26 +33,10 @@ consumers. ## Ideas -Executable preparation handles have been selected for active planning in the -[focused feature roadmap](prepared-execution.md). The remaining ideas are +Prompt-independent profile inspection has been selected for active planning in +the [focused feature roadmap](profile-inspection.md). The remaining ideas are still available for future selection. -### Prompt-independent profile inspection - -Provide exact profile lookup and structural resolution without requiring a -synthetic prompt, placeholder inputs, or model generation. This shared need is -described by -[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection) -and -[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection). - -- Apply ordinary built-in, file-backed, and programmatic profile precedence. -- Validate referenced backend membership and the structurally resolved - execution target. -- Report credential requirements and environment-variable names without - exposing credential values or requiring current credential availability. -- Support exact lookup by profile ID; enumeration is not required initially. - ### Prompt-definition inspection Provide exact prompt-definition lookup without rendering, placeholder inputs, diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 50e70ca..3b9e6c6 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,12 +1,12 @@ -# Executable Preparation Handles Implementation Plan +# Prompt-Independent Profile Inspection Implementation Plan -**Status:** Complete. +**Status:** Ready for implementation. ## Purpose This document is the decision-complete implementation plan for -[executable preparation handles](prepared-execution.md). It is written for a -coding agent that will implement each stage in order. +[prompt-independent profile inspection](profile-inspection.md). It is written +for a coding agent that will implement each stage in order. The feature roadmap owns the motivation, consumer workflow, policy choices, compatibility requirements, non-goals, and target end state. This document @@ -18,705 +18,653 @@ ownership, documentation work, validation commands, and completion gates. - Complete the stages in order. Keep the repository compiling and the focused tests passing at every stage boundary. - Preserve unrelated working-tree changes. In particular, retain the accepted - roadmap and the downstream wishlist and future-roadmap edits that may - already be present. + feature roadmap and the corresponding future-roadmap and downstream-wishlist + edits that may already be uncommitted. - Follow every policy under `docs/policy/`, the task-specific reading guide in `docs/development.md`, and the accepted behavior in - `prepared-execution.md`. -- Keep the supported API in the root `promptkit` package. Internal lifecycle, - validation, and execution state must remain below Go's `internal/` + `profile-inspection.md`. +- Keep the supported API in the root `promptkit` package. Profile repositories, + backend resolution, and target assembly remain below Go's `internal/` boundary. -- Preserve the observable behavior and ordering of existing `Engine.Prepare` - and `Engine.Run`. Do not implement the new workflow by redefining either - existing method in terms of the new public methods. -- Reuse preparation and post-preparation execution mechanics where their - ordering is genuinely shared. Preserve `Run`'s existing early admission - boundary while ensuring `PrepareExecution` performs complete preparation - without admission. -- Do not add durable serialization, retries, provider failover, background - workers, a task queue, capacity reservation during preparation, or - consumer-specific persistence and redaction policy. -- Keep direct API-key values in private execution state only. Every public - details value, formatted handle, JSON value, result, hash, and error must - remain credential-free. -- Keep tests lean but give durable coverage to the public compatibility, - security, concurrency, one-attempt, source-snapshot, cancellation, and - capacity invariants. Use real internal collaborators where they are fast and - deterministic and fakes at model-generation and synchronization boundaries. -- Update exact GoDoc with exported declarations. Update current-state - consumer and internal documentation only after the corresponding behavior - exists. -- Do not add release notes, change a module version, create a release, or tag - a commit as part of this work. +- Reuse the exact profile-source, backend-membership, and execution-target + precedence used by `Prepare`, `PrepareExecution`, and `Run`. Do not create a + second profile loader, backend registry, or target-merging implementation. +- Preserve the observable behavior and error ordering of existing preparation + and execution methods. The new inspection operation must not require a + prompt, prompt default profile, request override, credential value, capacity + admission, model client, renderer, artifact reader, schema source, or + validator. +- Treat structural validation as the existing profile, backend, and effective + target invariants. Do not add new endpoint URL policy, provider + connectivity checks, reserved-extra-parameter policy, credential syntax, or + file-format validation rules as part of this feature. +- Never read, retain, return, format, or log an environment credential value. + The result may contain only the effective environment-variable name and the + direct-key-required boolean. +- Return caller-owned public values. In particular, nested + `ExecutionTarget.ExtraParams` maps, slices, and objects must not alias + engine-owned state or another inspection result. +- Keep tests lean and behavior-focused. Reuse existing profile repository, + backend registry, target precedence, and preparation tests rather than + duplicating their complete matrices. +- Update exact contracts in GoDoc with the exported declarations. Update + current-state consumer, format, and internal documentation only after the + corresponding code exists. +- Do not add release notes, change a module version, create a release, or tag a + commit as part of this work. ## Fixed Design ### Public API -Add an opaque root-package type and these methods: +Add this root-package value immediately after `ExecutionTarget` in `types.go`: ```go -type PreparedExecution struct { - // Unexported Promptkit-owned state only. -} - -func (e *Engine) PrepareExecution( - ctx context.Context, - req RunRequest, -) (*PreparedExecution, error) - -func (p *PreparedExecution) Details() PreparedRun - -func (p *PreparedExecution) Discard() - -func (e *Engine) RunPrepared( - ctx context.Context, - prepared *PreparedExecution, -) (*RunResult, error) -``` - -Place the public handle and its handle-local methods in a new root file named -`prepared_execution.go`. Keep the two engine operations in `engine.go` beside -`Prepare` and `Run` so all engine workflows remain discoverable together. - -`PreparedExecution` must contain no exported fields. It wraps one pointer to -Promptkit-owned internal state; copying the public struct therefore preserves -one shared lifecycle rather than creating another execution opportunity. - -Implement both: - -```go -func (p *PreparedExecution) String() string -func (p *PreparedExecution) GoString() string -``` - -with a constant credential- and content-free representation such as -`promptkit.PreparedExecution{opaque}`. These methods must not branch on, -inspect, or format retained execution state. Do not add a custom JSON -representation: with no exported fields, ordinary JSON encoding exposes no -state, and JSON is explicitly outside the handle contract. - -The exact GoDoc must own these behaviors: - -- `PrepareExecution` performs complete preparation but no model call and no - backend-capacity admission; -- the handle is bound to the creating engine and permits one - `RunPrepared` invocation; -- `Details` returns a fresh caller-owned, credential-redacted copy on every - call and remains usable after discard or execution; -- `Discard` is nil-safe and idempotent, invalidates an unclaimed handle, and - is not an execution-cancellation mechanism; -- `RunPrepared` atomically consumes the one attempt before credential - revalidation, admission, generation, or validation; -- lifecycle misuse matches `ErrInvalidRequest`, while a nil engine matches - `ErrInvalidConfig`; -- an operational execution error returns no partial `RunResult`; -- the `RunPrepared` context does not inherit from the preparation context and - governs the execution attempt; and -- handles are opaque in-process values, not serializable or restartable jobs. - -For a nil or zero-value handle, `Details` returns a zero `PreparedRun` and -`Discard` does nothing. A nil, zero-value, foreign-engine, discarded, -claimed, or used handle passed to `RunPrepared` returns an error matching -`ErrInvalidRequest`. - -### Internal Validation Snapshot - -Extend `internal/validate/validator.go` with two internal interfaces: - -```go -// PreparedValidation validates artifacts against one frozen output contract. -type PreparedValidation interface { - Validate( - ctx context.Context, - artifact *domain.Artifact, - ) (domain.ValidationResult, error) - - // SchemaDocument returns the root JSON Schema document used for provider - // structured output, or nil for non-schema modes. The returned value is - // internal immutable state and must not be mutated by callers. - SchemaDocument() any -} - -// ValidationPreparer freezes the validation resources for one output -// contract. -type ValidationPreparer interface { - PrepareValidation( - ctx context.Context, - contract domain.OutputContract, - ) (PreparedValidation, error) +// ProfileInspection is the caller-owned result of Engine.InspectProfile. +// The exact GoDoc is specified below. +type ProfileInspection struct { + ProfileID string + EffectiveModelParams ExecutionTarget + APIKeyRequired bool } ``` -The exact names may be adjusted only to avoid an actual Go naming conflict; -the separation of responsibilities and method shapes are fixed. -`validate.Validator` and its existing `Validate` method remain unchanged so -the existing `Run` behavior and internal test fakes do not acquire a new -mandatory method. - -`StandardValidator` and `FSValidator` implement `ValidationPreparer`. Their -prepared-validation implementation: - -- captures a value copy of the complete `OutputContract`; -- handles none, basic, and JSON validation without any source access; -- for JSON Schema, loads the root document, validates its dialect, registers - it with a new compiler, compiles it during `PrepareValidation`, and retains - the compiled schema; -- allows the compiler to resolve every transitive reference during that - compilation using the existing contained filesystem or `fs.FS` loader; -- returns the same loaded root document through `SchemaDocument`, so the - provider-facing structured-output constraint and later validation derive - from one root snapshot; and -- validates generated instances only against retained in-memory state. Its - `Validate` method must not reopen a path, call `fs.ReadFile`, resolve a - symlink, or invoke a schema loader after preparation succeeds. - -Reuse the existing JSON parsing, validation-result construction, schema -dialect checks, contained-path rules, compiler configuration, and validation -error wording. A prepared JSON Schema compilation or reference-loading failure -is a preparation-time validation error. Existing direct calls to -`Validator.Validate`, including the existing `Run` path, retain their current -lazy compilation and source-loading behavior. - -Do not expose the JSON Schema library's compiled type outside -`internal/validate`. Do not place compiled schemas or validation interfaces in -`internal/domain`. - -### Internal Prepared-Execution State - -Add `internal/usecase/prepared_execution.go`. Define an internal -`PreparedExecution` owned by one `Runner`. It contains: - -- the immutable owner `*Runner`; -- a mutex; -- a small private lifecycle state with at least ready, claimed, and discarded - states; -- a credential-redacted prepared-details snapshot that survives every - lifecycle transition; and -- while ready, a pointer to a private payload containing the full - `domain.PreparedRun` and its `validate.PreparedValidation`. - -The payload is the only retained object containing `RunRequest.APIKey`. -Construct the public-details snapshot as a deep copy of the prepared run and -clear its `EffectiveModelParams.APIKey`. Deep-copy every mutable nested value, -including messages, maps, target extra parameters, and the structured-output -schema. Reuse the repository's JSON-compatible copy helpers rather than -maintaining an ad hoc reflection copier. - -Add these internal operations: +Add this method to `Engine` in `engine.go`, immediately before `Prepare`: ```go -func (r *Runner) PrepareExecution( +func (e *Engine) InspectProfile( ctx context.Context, - req domain.RunRequest, -) (*PreparedExecution, error) - -func (p *PreparedExecution) Details() *domain.PreparedRun - -func (p *PreparedExecution) Discard() - -func (r *Runner) RunPrepared( - ctx context.Context, - prepared *PreparedExecution, -) (*domain.RunResult, error) + profileID string, +) (*ProfileInspection, error) ``` -Internal exported names are acceptable here because the root facade must call -them, but they remain inaccessible to downstream consumers through Go's -`internal` rule. +Do not accept `RunRequest`, `ExecutionTargetOverride`, an API key, an +environment override, or options on this method. Exact inspection of one +explicit profile ID is the complete operation. -`PrepareExecution` performs these operations in order: +`ProfileInspection` has no stable JSON contract. Do not add JSON tags, +`MarshalJSON`, `UnmarshalJSON`, or a custom string representation. Ordinary Go +encoding of the exported fields is not prohibited, but consumers must not be +promised compatibility for that encoding. -1. call the existing request-copy boundary in the root facade before entering - the runner; -2. run `resolvePreparation`, including the existing environment credential - availability check; -3. prepare a frozen validation plan through - `validate.ValidationPreparer`; -4. construct provider structured-output metadata from the plan's root schema - document for JSON Schema mode; -5. load and hash artifacts, render the session and messages, and assemble the - complete `domain.PreparedRun`; and -6. construct the ready internal handle with independent full and redacted - snapshots. +The exact type and method GoDoc must establish: -Extract the artifact-loading, rendering, hashing, timing, and -`domain.PreparedRun` assembly portion of `completePreparation` into a helper -that accepts an already-resolved `*domain.StructuredOutputSpec`. -`completePreparation` must continue to call the existing -`resolveStructuredOutput` first and then use that helper. This preserves -existing `Prepare` and `Run` schema behavior. `PrepareExecution` uses the -frozen validation plan's schema document to build the same structured-output -shape without calling `resolveStructuredOutput`. +- surrounding whitespace is trimmed from `profileID`; the resulting nonblank + ID is looked up exactly and case-sensitively; +- the result applies the engine's ordinary in-memory, configured-source, and + built-in profile precedence; +- `EffectiveModelParams` contains framework defaults overlaid by the selected + backend and then the selected profile, with no request override; +- `EffectiveModelParams.BackendID` is empty for endpoint-only profiles; +- `EffectiveModelParams.APIKeyEnv` is an environment-variable name and never + its value; +- `APIKeyRequired` means a direct request credential is required and is + mutually exclusive with a nonblank effective `APIKeyEnv`; +- the method never derives an ID from a prompt's `default_profile`; +- the method performs no prompt lookup, rendering, artifact or schema work, + backend admission, provider connectivity check, or model generation; +- credential availability is not checked, so an absent or blank named + environment variable is not an error; +- the returned value and all nested mutable values are caller-owned; +- filesystem-backed inspection is a point-in-time lookup and does not freeze + a profile for a later execution; +- a nil engine matches `ErrInvalidConfig`; +- a blank ID matches `ErrInvalidRequest`; +- an absent exact ID matches `ErrProfileNotFound` and not `ErrProfileLoad`; +- malformed or unreadable profile data, an unknown backend, or an invalid + resolved target matches `ErrProfileLoad`; +- cancellation observed during profile loading matches `ErrProfileLoad` while + preserving the context error through `errors.Is`; and +- the method returns no partial result on error. -When the runner has no validator, preserve the existing runner semantics: -validation is skipped and a no-op prepared plan is sufficient. When it has a -non-nil validator that does not implement `ValidationPreparer`, -`PrepareExecution` fails with `ErrValidation`; do not fall back to a plan that -would reopen sources later. The root engine's standard and `fs.FS` validators -always implement the new interface. +Update the `Engine` type GoDoc to include `InspectProfile` among the operations +safe for concurrent calls. Do not imply that injected collaborators become +safe when their existing contracts do not provide that guarantee. -### Claim, Cleanup, And Engine Binding +Update `doc.go` in the same stage: -`RunPrepared` checks engine ownership before attempting a claim. A foreign -runner returns `ErrInvalidRequest` without changing the handle. The owning -runner then locks the state and atomically: +- include `Engine.InspectProfile` in the package's operation list; +- include inspection in the concurrency and caller-ownership summary; and +- list `ProfileInspection` among construction and inspection values without a + stable JSON representation. -1. accepts only the ready state; -2. changes it permanently to claimed; -3. detaches the private payload from the handle; and -4. unlocks before doing any blocking or collaborator work. +Do not add `ProfileInspection` to the stable JSON list. -Every valid owning-engine invocation consumes the attempt at this point. Run -ID creation, credential failure, pre-canceled context, capacity rejection, -generation failure, operational validation failure, and success all leave the -handle claimed and unusable. +### Internal Result -The execution call holds the detached payload locally and defers cleanup on -every exit. Cleanup must at minimum: +Add this internal value to `internal/domain/domain.go` near +`ExecutionProfile` and `ExecutionTarget`: -- overwrite the direct `EffectiveModelParams.APIKey` field with an empty - string; -- drop the local prepared-run pointer; -- drop the frozen-validation reference; and -- leave the separate redacted details snapshot intact. +```go +// ProfileInspection is the resolved result of exact profile inspection. +type ProfileInspection struct { + ProfileID string + EffectiveModelParams ExecutionTarget + APIKeyRequired bool +} +``` -This is reference cleanup, not a promise of secure Go string-memory erasure. +Do not add JSON or YAML tags. The internal value is a use-case result, not a +file format or persistence contract. -`Discard` locks the same state. If ready, it changes the state to discarded, -detaches the payload, unlocks, and performs the same cleanup. If claimed or -already discarded, it is a no-op. Therefore a `RunPrepared`/`Discard` race has -one winner: discard prevents a claim only if it acquires the ready state -first; otherwise it does not cancel the running attempt. +Add a root conversion in `convert.go`: -`Details` locks only long enough to read the retained redacted snapshot and -returns a fresh deep copy. It must not return a pointer, slice, map, schema, or -extra-parameter tree shared with either the handle or another `Details` call. +```go +func fromDomainProfileInspection( + inspection *domain.ProfileInspection, +) *ProfileInspection +``` -### Prepared Execution Mechanics +The conversion must: -Refactor the post-preparation portion of `Runner.Run` into a private execution -helper that accepts: +- return `nil` for a nil internal value; +- copy the scalar fields; +- convert the target through the existing `fromDomainExecutionTarget`; and +- therefore use the existing recursive `copyAnyMap` boundary for nested extra + parameters. -- a fully prepared `domain.PreparedRun`; -- a caller-supplied run ID and execution start time; and -- a validation function that accepts the artifact and repair-attempt count. +Do not add `APIKeyRequired` to the public `ExecutionTarget`. Keeping the +requirement on `ProfileInspection` preserves the existing stable JSON and +general execution-target contract. -The helper owns model generation, artifact construction, content validation, -internal repair, final result assembly, and result end timing. The live -`Runner.Run` path supplies its existing `validateOutput` function. The -prepared path supplies a wrapper around the retained -`validate.PreparedValidation`. Both wrappers set the actual repair-attempt -count on the returned validation result. +### Shared Profile Selection -Keep admission outside this shared helper: +Add `internal/usecase/profile_inspection.go`. Define one private selection +value: -- `Runner.Run` retains its current sequence of run-ID creation, start time, - target resolution, early admission, complete preparation, execution, and - release. -- `Runner.RunPrepared` claims the handle, creates the run ID and execution - start time, revalidates credential availability, admits the frozen backend, - executes the shared helper, and releases admission. +```go +type resolvedProfileSelection struct { + id string + profile *domain.ExecutionProfile + backend *domain.Backend +} +``` -Extract a small private admission helper only if it preserves the exact -existing capacity wrapping and release behavior. The limited-backend lease -must cover generation, validation, and all internal repair calls and must be -released on every return. +Add a private runner helper: -Before admission, `RunPrepared` calls the existing `validateAPIKey` with the -frozen environment-variable name, retained direct key, and frozen requirement -flag. A missing environment credential returns an error wrapping -`ErrInvalidRequest` and `ErrAPIKeyEnvMissing`. Do not read and retain the -environment value; the model client continues to resolve it when generation -begins. +```go +func (r *Runner) resolveProfileSelection( + ctx context.Context, + profileID string, +) (*resolvedProfileSelection, error) +``` -The result produced by the shared helper uses the frozen preparation values -for all provenance. For `RunPrepared`, `StartTime`, `EndTime`, and `Duration` -begin after the successful claim and exclude preparation and consumer-held -delay. For ordinary `Run`, retain the current timing boundary beginning before -preparation. +The helper must perform these operations in order: -### Error Mapping And Compatibility +1. trim surrounding whitespace from `profileID`; +2. reject a blank result with `ErrInvalidRequest`; +3. reject a nil runner profile repository with `ErrProfileLoad` rather than + panicking; +4. call the existing `profile.Repository.GetProfile` exactly once; +5. wrap every repository error with `ErrProfileLoad` while preserving the + underlying error with `%w`; +6. reject a nil profile returned without an error as `ErrProfileLoad`; +7. make a value copy of the selected profile before normalization so the + runner does not mutate repository-owned state; +8. trim the copied profile's backend ID; +9. when that ID is nonblank, resolve it exactly once through the existing + `BackendResolver`, preserving the current unknown-backend + `ErrProfileLoad` wrapping and backend ID context; and +10. return the normalized exact ID, copied profile, and optional defensive + backend value. -Do not add a new public sentinel. Internal lifecycle errors wrap -`usecase.ErrInvalidRequest`, which the existing root mapping exposes as -`promptkit.ErrInvalidRequest`. +Do not inspect all sources, enumerate profiles, expose a source path, fall back +after a malformed higher-precedence match, or infer a backend from an endpoint +or model. -`PrepareExecution` uses the same public categories as `Prepare`. -`RunPrepared` may return: +Refactor the profile-loading and backend-resolution block in +`Runner.resolvePreparation` to call `resolveProfileSelection`. Leave prompt +selection, prompt default-profile selection, prompt hashing, target overrides, +request credentials, output-contract resolution, artifact work, and all later +ordering where they currently occur. -- `ErrInvalidConfig` for a nil engine; -- `ErrInvalidRequest` for handle misuse and invalid credentials; -- `ErrAPIKeyEnvMissing` together with `ErrInvalidRequest` when the frozen - environment name is no longer set and no direct key is retained; -- `ErrCapacityExceeded` for admission rejection; -- `ErrLLMGenerate` for model-generation failures; and -- `ErrValidation` for inability to validate or repair. +This refactor must preserve: -Preserve wrapped collaborator and context errors according to the existing -mapping rules. A content validation rejection remains a successful -`RunResult` with `ValidationFailed`, not an operational error. +- explicit request profile selection over prompt `default_profile`; +- the existing `ErrProfileRequired` result when neither exists; +- exact repository and backend lookup behavior; +- existing public not-found versus profile-load classification; +- request override precedence and target-presence tracking; +- credential checking during ordinary preparation; and +- current `Prepare`, `PrepareExecution`, `Run`, and `RunPrepared` behavior. -Do not change stable JSON for `PreparedRun` or `RunResult`, public request and -result ownership, provider request bodies or headers, backend selection, -capacity policies, `LLMClient`, `ArtifactReader`, `Prepare`, or `Run`. +### Shared Target Resolution And Structural Validation + +Continue using the existing `resolveExecutionTarget` for framework, backend, +profile, and optional request precedence. Do not duplicate or move the +individual merge rules into the inspection code. + +Extract the two existing effective-target requiredness checks from +`resolvePreparation` into a private pure helper: + +```go +func validateResolvedExecutionTarget( + target domain.ExecutionTarget, +) error +``` + +It checks only that the trimmed endpoint and model are nonblank and returns a +plain descriptive error. It does not validate URL syntax, contact the +endpoint, validate credentials, or introduce new parameter rules. + +`resolvePreparation` calls this helper at the same current point: after +request override resolution and direct API-key assignment, and before +`validateAPIKey`. It wraps a failure with `ErrInvalidRequest`, preserving +ordinary preparation behavior. + +Add the internal inspection method: + +```go +func (r *Runner) InspectProfile( + ctx context.Context, + profileID string, +) (*domain.ProfileInspection, error) +``` + +It performs these operations: + +1. trim and reject a blank ID as `ErrInvalidRequest`; +2. if the supplied context is already canceled, return an error wrapping both + `ErrProfileLoad` and `ctx.Err()` without touching the repository; +3. call `resolveProfileSelection`; +4. call `resolveExecutionTarget` with the selected backend, selected profile, + and a nil request override; +5. wrap any target-resolution error with `ErrProfileLoad`; +6. call `validateResolvedExecutionTarget` and wrap a failure with + `ErrProfileLoad`; +7. defensively clear `target.APIKey`; +8. copy `target.APIKeyRequired` into the result's `APIKeyRequired`; and +9. return the normalized profile ID and effective target. + +Do not call `validateAPIKey`, `os.Getenv`, prompt repositories, artifact +readers, renderers, validators, capacity admission, output repair, or the +model client. Do not populate request target-presence state. + +The internal `ExecutionTarget` may retain its private `APIKeyRequired` field; +the root conversion intentionally omits that private field from the public +target and publishes the separate inspection boolean. + +### Root Facade And Error Mapping + +`Engine.InspectProfile` follows the existing facade pattern: + +1. reject a nil engine or nil runner with `ErrInvalidConfig`; +2. pass the context and string ID directly to `Runner.InspectProfile`; +3. map internal failures through the existing `mapPublicError`; and +4. convert a successful result with `fromDomainProfileInspection`. + +No request conversion is needed. Do not add a public sentinel or typed error. + +The existing error mapping already has the required ordering: + +- an underlying `profile.ErrProfileNotFound` maps to + `ErrProfileNotFound` before the enclosing use-case `ErrProfileLoad` is + considered; +- other use-case `ErrProfileLoad` failures map to `ErrProfileLoad`; and +- `usecase.ErrInvalidRequest` maps to `ErrInvalidRequest`. + +Do not reorder or otherwise change `errors.go` unless a focused test proves +that the existing mapping does not meet this plan. Preserve underlying +repository, backend, and context errors through `errors.Is`. + +### Credentials, Ownership, And Concurrency + +Inspection has no input through which a direct credential value can enter. +Backend and file-profile `APIKeyEnv` values remain names only. An in-memory +profile with `APIKeyRequired` clears an inherited backend environment name +through the existing target merge behavior. + +The implementation must succeed for: + +- a backend or profile naming an unset environment variable; +- a backend or profile naming an environment variable whose value is blank; +- an in-memory profile requiring a direct key when no key is supplied; and +- a profile requiring no credential. + +The result expresses these effective states: + +| `EffectiveModelParams.APIKeyEnv` | `APIKeyRequired` | Meaning | +| --- | --- | --- | +| nonblank | `false` | The profile resolves to the named environment source. | +| blank | `true` | A later execution must supply a direct key or explicit request environment override. | +| blank | `false` | The resolved target declares no credential requirement. | + +The implementation must never produce a successful public result with both a +nonblank `APIKeyEnv` and `APIKeyRequired == true`. + +Public ownership is enforced at the root conversion boundary. A consumer may +mutate the returned target and arbitrarily nested JSON-compatible extra +parameters without affecting: + +- the engine registry or profile repository; +- a later `InspectProfile` call; +- `Prepare`, prepared execution, or `Run`; or +- another result already returned to a caller. + +No new mutable engine state, cache, global registry, lock, or goroutine is +needed. Concurrency safety follows from the existing immutable registry and +repository contracts plus per-call values. ### Test Ownership -Tests must be split by the narrowest stable owner. +Add focused internal tests in +`internal/usecase/profile_inspection_test.go`. Use small repository and backend +fakes already present in the package where practical; do not build a parallel +fixture framework. -`internal/validate/standard_validator_test.go` owns frozen schema mechanics. -Add focused tests for both configured schema-source forms: +The internal tests own: -- a directory-backed `StandardValidator` test whose root schema references a - second file; after `PrepareValidation`, replace or remove both files and - prove that valid and invalid artifacts are judged by the original compiled - schema; and -- an `FSValidator` test using a mutable in-memory `fs.FS`; mutate its root and - referenced entries after `PrepareValidation` and prove the same invariant. +- blank-ID rejection; +- pre-canceled context classification without repository access; +- one exact profile and backend lookup; +- framework-default, backend, and profile target precedence through the + existing resolver; +- endpoint-only behavior; +- effective environment-name reporting without availability checks; +- direct-key-required behavior clearing an inherited backend environment + name; +- missing-profile and unknown-backend wrapping; and +- nil repository, nil returned profile, and invalid resolved target defenses + only if these cases are not already cheaply covered through existing runner + fakes. -Each test must also confirm that `SchemaDocument` is the original root -document used at preparation. Existing path-containment and dialect matrices -remain the owners of those rules and must not be duplicated. +Keep the internal matrix compact. Profile parser tests continue to own YAML, +duplicates, raw-key rejection, ranges, and source discovery. Backend tests +continue to own registration validation. Existing target tests continue to +own every merge field and numeric override boundary. -`internal/usecase` tests own orchestration details that cannot be observed -cleanly at the facade: +Add external-package public contract tests in `public_contract_test.go`. They +own: -- `PrepareExecution` completes source loading, artifact reading, rendering, - and validation-plan creation without calling admission or generation; -- `RunPrepared` rechecks environment credential availability before admission; -- the prepared path supplies the frozen validation plan to initial validation - and every repair attempt; -- admission is released on success and each error exit; and -- the ordinary `Run` admission-before-complete-preparation ordering remains - unchanged after refactoring. +- the exported method and result shape through normal Go use; +- a nil engine returning `ErrInvalidConfig`; +- blank, missing, malformed, and unknown-backend public error identities, + including that not-found does not also match `ErrProfileLoad`; +- a pre-canceled inspection preserving both `ErrProfileLoad` and the context + error without consulting the profile repository; +- successful inspection with an absent credential environment value; +- no prompt dependency, demonstrated with an empty configured prompt + `fs.FS`; +- no model invocation, using the existing deterministic fake client; +- the three credential-requirement result states; +- an endpoint-only profile's empty backend ID; +- deep caller ownership of nested extra parameters across repeated + inspections; and +- representative equivalence between `InspectProfile.EffectiveModelParams` + and `Prepare.EffectiveModelParams` for the same engine, profile, and source + state with no request execution override. -Use small test-controlled collaborators. Do not assert private enum values, -mutex layout, helper call graphs, or cleanup implementation details. +Combine closely related assertions into a few readable behavioral tests. +Do not add a JSON golden test because `ProfileInspection` deliberately has no +stable JSON contract. Do not duplicate the complete repository-precedence, +target-field, parser-error, or credential-execution suites at the root layer. -Add `prepared_execution_contract_test.go` in external package -`promptkit_test` for the public contract. Cover these distinct risks with the -smallest coherent set of tests: +Existing tests that must continue passing without semantic edits include: -1. **Frozen execution and independent details.** Prepare a request, mutate - caller-owned request values, mutable prompt/profile/input/schema sources, - and one returned `PreparedRun`, then run the handle. Assert that the - captured `GenerateRequest`, validation, result provenance, and a second - `Details` call all retain the original prepared state. Assert that source - collaborators are not reopened during execution. -2. **Lifecycle and engine binding.** Prove that a foreign engine is rejected - without consuming the handle, the owner can then run it once, a second - call through either the original or a copied public handle is rejected, and - details remain available. -3. **Concurrent claim.** Race two owning-engine `RunPrepared` calls against a - blocking fake client. Exactly one reaches generation and the other matches - `ErrInvalidRequest`. Run this test under the race detector. -4. **Discard and formatting security.** Retain a distinctive direct API key, - verify that `Details`, ordinary JSON encoding, `%v`, `%+v`, `%#v`, and - lifecycle errors do not contain it or rendered content, discard the handle, - and verify idempotence, execution rejection, and retained details. -5. **Credential and capacity timing.** Prove that preparation does not occupy - admission; removing a required environment credential makes execution fail - before generation; a capacity rejection consumes the handle; and the - execution result's timing excludes a test-controlled delay between - preparation and execution. - -Reuse existing root contract fixtures and capacity fakes where practical. -Never put a real-looking credential in a fixture. Do not test JSON equality -for the opaque handle, exact lifecycle error strings, private state, run-ID -format, or clock-duration precision. +- profile repository and built-in fallback tests; +- backend registry defensive-copy and lookup tests; +- execution-target precedence tests; +- `Prepare` and `Run` profile/backend equivalence tests; +- prepared-execution snapshot and credential tests; and +- public error mapping tests. ### Documentation Ownership -After implementation: +After the implementation and public tests pass, update current-state +documentation: -- exported declarations and GoDoc in `prepared_execution.go`, `engine.go`, and - `types.go` own the exact public lifecycle, ownership, security, timing, - cancellation, and error contracts; -- `docs/consumers/pkg-promptkit.md` explains when to choose `Prepare`, `Run`, - or `PrepareExecution` plus `RunPrepared`, and shows one concise two-phase - workflow using `defer prepared.Discard()`; -- `docs/internal/sources.md` explains that prepared execution freezes all - source-derived state, including transitive schema references, while - ordinary `Prepare` remains inspection-only; -- `docs/internal/llm.md` explains that prepared generation uses the retained - target and messages, rechecks environment credential availability, and - does not reopen sources; -- `docs/internal/capacity.md` explains that `PrepareExecution` performs no - admission and that `RunPrepared` acquires and holds the normal run lease - across generation, validation, and internal repair; and -- `docs/internal/overview.md` updates the validator and use-case inventory to - mention frozen validation plans and one-attempt prepared execution. +- `docs/consumers/pkg-promptkit.md`: add a concise task-oriented section near + profile selection showing `Engine.InspectProfile`, explaining when to use it + instead of a synthetic `Prepare`, how to interpret `APIKeyEnv` and + `APIKeyRequired`, and that credential enforcement timing remains + application policy. Link to the exact GoDoc rather than restating every + error and field contract. +- `docs/formats.md`: update profile selection and backend-membership wording + so exact inspection is recognized as another consumer of the existing + source and profile precedence. Do not redefine the exported method here. +- `docs/internal/runner.md`: describe the shared profile-selection boundary + and explain that inspection stops after structural target resolution, + before credential availability and all prompt-dependent work. +- `docs/internal/sources.md`: record that exact profile inspection performs + one point-in-time profile-source lookup without reading prompt, input, or + schema sources, and replace any ambiguous use of “inspection value” for + `PreparedRun` with “preparation value.” +- `docs/internal/overview.md`: add profile inspection to the existing root + facade and `internal/usecase` responsibility descriptions. Do not add a new + component or package row. +- `docs/roadmap/future.md`: remove the statement that profile inspection is in + active planning and leave the remaining unselected ideas intact. +- `docs/roadmap/notarius-promptkit-wishlist.md` and + `docs/roadmap/weatherreporter-promptkit-wishlist.md`: change the profile + inspection disposition from accepted planning to implemented behavior and + link to the durable consumer guidance or GoDoc rather than treating the + roadmap as current-state documentation. +- `docs/roadmap/profile-inspection.md`: change its status to `Complete` only + after the code, tests, current-state documentation, and full validation are + complete. +- `docs/roadmap/implementation.md`: change its status to `Complete` only after + every completion gate in this plan is satisfied. -Keep these documents at their established abstraction levels. Link to exact -Go declarations rather than reproducing field-by-field contracts. No README, -format reference, integration contract, maintained example, architecture -policy, release note, or release-procedure update is required because the -feature changes none of their owned concerns. +Do not update release guidance in this feature implementation. A later release +pass decides whether the change warrants a supplemental release document. -## Stage 1: Add Frozen Validation Plans - -**Status:** Complete. +## Stage 1: Implement Shared Internal Profile Resolution ### Objective -Create an internal validation artifact that eagerly captures every schema -resource required by later validation, without changing existing validation -or engine behavior. +Add the internal inspection result and operation, share profile/backend +selection and structural target validation with ordinary preparation, and +prove the internal behavior without publishing the root API yet. ### Implementation Prompt -1. Re-read `docs/development.md`, all files under `docs/policy/`, - `prepared-execution.md`, `internal/validate/validator.go`, - `internal/validate/standard_validator.go`, and the focused validator tests. -2. Confirm the working tree and preserve all existing roadmap and wishlist - changes. -3. Add `PreparedValidation` and `ValidationPreparer` to - `internal/validate/validator.go` with the exact responsibilities in the - fixed design. Leave `Validator` unchanged. -4. Implement `PrepareValidation` for `StandardValidator` and `FSValidator`. - Reuse the existing root resolution, resource identifiers, contained - loaders, dialect checks, compiler configuration, JSON parser, and - validation-result builder. -5. Ensure the root document is registered from the already-loaded value and - every transitive reference is loaded during compilation. Retain only the - compiled schema, root document, and copied contract required for later - validation. -6. Ensure the prepared validator checks its validation-time context and does - not touch either source after successful preparation. -7. Add the two focused frozen-reference tests described under Test Ownership. - Keep existing tests unchanged except for shared test setup that materially - reduces duplication. +Implement only Stage 1 of +`docs/roadmap/implementation.md`. Read the complete feature roadmap and the +implementation rules and fixed design above before editing. + +1. Add `domain.ProfileInspection` to `internal/domain/domain.go` without + serialization tags. +2. Add `internal/usecase/profile_inspection.go` with + `resolvedProfileSelection`, `resolveProfileSelection`, + `validateResolvedExecutionTarget`, and `Runner.InspectProfile` exactly as + specified. +3. Refactor `Runner.resolvePreparation` in `internal/usecase/runner.go` to use + the shared selection and target-validation helpers while preserving its + current ordering, error classification, target overrides, credentials, and + results. +4. Add lean behavioral tests in + `internal/usecase/profile_inspection_test.go`. +5. Run the focused validation below and repair regressions before ending the + stage. + +Do not add the root public type or method, update current-state documentation, +or alter profile formats, backend registration, credential availability, +capacity, model-client, or provider behavior in this stage. ### Focused Validation Run: ```sh -gofmt -w internal/validate/validator.go \ - internal/validate/standard_validator.go \ - internal/validate/standard_validator_test.go -go test ./internal/validate -go test -race ./internal/validate -go vet ./internal/validate -git diff --check +gofmt -w internal/domain/domain.go \ + internal/usecase/profile_inspection.go \ + internal/usecase/profile_inspection_test.go \ + internal/usecase/runner.go +go test ./internal/usecase ./internal/profile ./internal/profile/builtin \ + ./internal/backend +go test ./internal/usecase -run \ + 'TestRunner(InspectProfile|Prepare|Run|PrepareExecution|RunPrepared)' +go vet ./internal/usecase ./internal/profile ./internal/backend ``` ### Completion Gate -Stage 1 is complete when: +Stage 1 is complete only when: -- both built-in validator forms produce prepared plans; -- prepared schema validation works after root and referenced sources change or - disappear; -- provider schema metadata can use the exact root document captured by the - plan; -- existing `Validator.Validate` behavior remains unchanged; and -- focused tests, race tests, vet, formatting, and whitespace checks pass. +- inspection reaches profile and backend resolution without any prompt or + execution collaborator; +- profile/backend selection is shared with ordinary preparation; +- target merging and requiredness checks are shared rather than duplicated; +- inspection does not check credential values or capacity; +- internal errors preserve the required identities; +- the ordinary preparation and execution tests still pass without weakened + assertions; and +- no root public API or current-state documentation claims the feature yet. -## Stage 2: Implement Internal Handle Lifecycle And Prepared Execution - -**Status:** Complete. +## Stage 2: Publish The Root Facade And Public Contract ### Objective -Add complete preparation, one-attempt lifecycle management, cleanup, and -source-free execution to the use-case layer while preserving existing -`Prepare` and `Run` behavior. +Expose the minimal caller-owned inspection API through `Engine`, preserve +public error and JSON compatibility, and protect the consumer-visible +contract. ### Implementation Prompt -1. Re-read the completed Stage 1 declarations, `internal/usecase/runner.go`, - its focused tests, `internal/capacity`, and the execution flow described in - `prepared-execution.md`. -2. Add `internal/usecase/prepared_execution.go` with the internal handle, - payload, lifecycle transitions, deep-copying, redaction, cleanup, and - runner ownership described in the fixed design. -3. Extract the common final preparation helper that accepts resolved - structured-output metadata. Keep `completePreparation` and existing - `resolveStructuredOutput` behavior intact for `Prepare` and `Run`. -4. Implement `Runner.PrepareExecution` using - `validate.ValidationPreparer`. Build JSON Schema structured-output metadata - from the frozen plan's root document. -5. Refactor post-preparation generation, validation, repair, and result - assembly into the shared private helper. Preserve ordinary `Run` timing, - early admission, error ordering, and live validator behavior. -6. Implement `Runner.RunPrepared`: verify owner, claim once, create run ID and - execution start time, recheck credentials, acquire admission, execute - against the retained plan, release admission, and clean the detached - payload on every exit. -7. Implement nil-safe, idempotent discard and fresh redacted details copying. - Confirm the `RunPrepared`/`Discard` race is resolved solely by the shared - mutex transition. -8. Add the internal orchestration tests described under Test Ownership, - including regression coverage for ordinary `Run`'s admission ordering. +Implement only Stage 2 of +`docs/roadmap/implementation.md` after Stage 1 satisfies its completion gate. +Re-read the fixed public API, error, credential, ownership, and test sections +before editing. + +1. Add `ProfileInspection` and its exact GoDoc to `types.go` immediately after + `ExecutionTarget`. Do not add JSON tags or change `ExecutionTarget`. +2. Add `fromDomainProfileInspection` to `convert.go` using + `fromDomainExecutionTarget`. +3. Add `Engine.InspectProfile` and its exact GoDoc to `engine.go` immediately + before `Prepare`. +4. Update `Engine` GoDoc to include concurrent inspection. +5. Update the package GoDoc in `doc.go` for operation discovery, concurrency, + ownership, and the non-stable JSON classification. +6. Add compact external-package contract coverage in + `public_contract_test.go`, using existing fixtures and fakes where they + remain clear. +7. Confirm that the existing `errors.go` mapping meets the plan; do not change + it unless a required public identity test fails for a genuine mapping + reason. +8. Run the focused validation below and repair regressions before ending the + stage. + +Do not add enumeration, request overrides, profile fingerprints, JSON +stability, prompt inspection, environment credential checks, caching, or +current-state prose documentation in this stage. ### Focused Validation Run: ```sh -gofmt -w internal/usecase/runner.go \ - internal/usecase/prepared_execution.go \ - internal/usecase/*_test.go -go test ./internal/usecase -go test -race ./internal/usecase -go test ./internal/validate -go vet ./internal/usecase ./internal/validate -git diff --check -``` - -### Completion Gate - -Stage 2 is complete when: - -- a runner can prepare and later execute without reopening any request source; -- one valid invocation consumes the handle on every outcome; -- foreign-runner rejection does not consume the handle; -- discard and claim are race-safe and cleanup drops secret-bearing state; -- prepared execution rechecks credentials and admits only at execution time; -- frozen validation is reused across initial output and repair attempts; -- existing `Prepare` and `Run` tests retain their observable ordering and - behavior; and -- focused tests, race tests, vet, formatting, and whitespace checks pass. - -## Stage 3: Publish The Root Facade And Public Contract - -**Status:** Complete. - -### Objective - -Expose the accepted opaque handle workflow through the root package and -protect its compatibility, security, and concurrency guarantees at the -consumer boundary. - -### Implementation Prompt - -1. Re-read the accepted public workflow, root conversion and error-mapping - helpers, existing `Prepare` and `Run` GoDoc, and the external-package - contract-test conventions. -2. Add root `prepared_execution.go` with the opaque wrapper, `Details`, - `Discard`, `String`, and `GoString`. -3. Add `Engine.PrepareExecution` and `Engine.RunPrepared` to `engine.go`. - Reuse `toDomainRunRequest`, `fromDomainPreparedRun`, - `fromDomainRunResult`, and `mapPublicError`. -4. Add or reuse one public prepared-run deep-copy helper so every `Details` - call returns fresh nested state. Do not expose the internal handle or - retain a public direct credential. -5. Write complete GoDoc for the type and methods. Update `RunRequest`, - `PreparedRun`, and `RunResult` GoDoc only where cross-references are needed - to distinguish the new workflow; do not change their stable JSON or - ownership contracts. -6. Add `prepared_execution_contract_test.go` and cover the five risk groups - under Test Ownership. Reuse existing fixtures and fakes rather than - duplicating backend, profile, schema, and capacity matrices. -7. Explicitly inspect formatted and JSON-encoded handles, lifecycle errors, - details, captured model requests, and results for the test sentinel direct - key. - -### Focused Validation - -Run: - -```sh -gofmt -w prepared_execution.go engine.go types.go convert.go \ - prepared_execution_contract_test.go -go test . -run 'PreparedExecution|RunPrepared|PrepareExecution' -go test -race . -run 'PreparedExecution|RunPrepared|PrepareExecution' +gofmt -w doc.go types.go convert.go engine.go public_contract_test.go go test . +go test ./internal/usecase +go test . -run 'Test(InspectProfile|.*Profile.*PublicError|.*Profile.*Contract)' go vet . go build . -git diff --check ``` +If the repository's actual focused test names differ, use the implemented test +names rather than weakening or skipping the intended assertions. + ### Completion Gate -Stage 3 is complete when: +Stage 2 is complete only when: -- the public API matches the accepted shape; -- details are fresh, stable, credential-redacted copies before and after every - lifecycle outcome; -- default formatting and JSON expose no retained request or credential data; -- foreign, copied, concurrent, discarded, and reused handles obey the - one-attempt contract; -- execution uses exactly the frozen request and validation state; -- preparation performs no generation or admission; -- result provenance matches details and execution timing excludes preparation - and retention delay; -- public errors preserve the required `errors.Is` identities; and -- focused public, race, package, vet, build, formatting, and whitespace checks - pass. +- a consumer can call `Engine.InspectProfile` through the root package; +- the result contains only the normalized ID, caller-owned effective target, + and direct-key-required signal; +- no credential value can enter the result; +- environment, direct, and no-credential states are unambiguous; +- nil, blank, missing, malformed, unknown-backend, and cancellation errors + have the required public identities; +- not-found remains distinct from profile-load failure; +- the effective target matches ordinary preparation absent request overrides; +- repeated calls and caller mutation cannot alter engine-owned state; +- no stable JSON or new error contract was introduced; and +- existing public preparation, execution, and stable JSON tests remain + unchanged and passing. -## Stage 4: Update Documentation And Validate The Repository - -**Status:** Complete. +## Stage 3: Update Documentation And Validate The Repository ### Objective -Publish current-state guidance at the correct documentation owners, validate -the complete repository, and mark the temporary roadmaps complete. +Make implemented profile inspection discoverable in its canonical +documentation, reconcile temporary roadmap state, and complete full repository +validation. ### Implementation Prompt -1. Re-read the implemented declarations and tests before documenting them. -2. Update `docs/consumers/pkg-promptkit.md`, - `docs/internal/sources.md`, `docs/internal/llm.md`, - `docs/internal/capacity.md`, and `docs/internal/overview.md` according to - Documentation Ownership. -3. Keep the consumer example concise, copyable, secret-free, and explicit - about `defer prepared.Discard()`. Link to declarations for exact lifecycle - and error semantics. -4. Follow every added or changed Markdown link and confirm its target exists. - Confirm that repository-relative links in both roadmap documents and all - changed current-state documents resolve. -5. Run the complete maintainer validation sequence below. -6. Inspect the final diff for accidental provider-wire changes, source - reloading during prepared execution, stable-JSON changes, credentials, - generated artifacts, local workspace files, module replacements, or - unrelated edits. -7. After every completion gate passes, set every stage status, this plan's - status, and the status in `prepared-execution.md` to `Complete`. Do not - retire or remove the roadmaps in the implementation change; roadmap - retirement follows implementation review. +Implement only Stage 3 of +`docs/roadmap/implementation.md` after Stages 1 and 2 satisfy their completion +gates. + +1. Update `docs/consumers/pkg-promptkit.md`, `docs/formats.md`, + `docs/internal/runner.md`, `docs/internal/sources.md`, and + `docs/internal/overview.md` according to the documentation ownership + section above. +2. Update the future catalog and both downstream wishlist dispositions so they + no longer describe profile inspection as merely accepted work. +3. Check every changed Markdown link and confirm its file and heading target. +4. Run the full validation sequence below. +5. Only after every check passes, set the feature roadmap and this + implementation plan to `**Status:** Complete.` +6. Re-run `git diff --check` after the status edits. + +Do not add release notes, an example program, a new public package, or a +duplicate API reference. Keep detailed contracts in GoDoc and task-oriented +usage in the consumer guide. ### Full Validation -Run the complete sequence from `docs/development.md`: +Run from the repository root: ```sh +gofmt -w internal/domain/domain.go \ + internal/usecase/profile_inspection.go \ + internal/usecase/profile_inspection_test.go \ + internal/usecase/runner.go \ + doc.go types.go convert.go engine.go public_contract_test.go +gofmt -l $(git ls-files '*.go') go test ./... go test -race ./... -go run ./examples/go-library/prepare go vet ./... go build ./... -gofmt -l $(git ls-files '*.go') -``` - -Then run the documented Markdown-link and repository-hygiene checks from -`docs/development.md`, followed by: - -```sh +go run ./examples/go-library/prepare git diff --check git status --short ``` +The `gofmt -l` command must print no paths. The maintained example must remain +offline and must not require a real credential or provider. + +Inspect the final diff and confirm: + +- only files required by this feature and pre-existing user changes are + present; +- no credential values, private infrastructure details, workspace files, + local replacements, generated binaries, or unrelated formatting changes + were added; +- the public declarations and GoDoc own exact API behavior; +- current-state documentation describes only implemented behavior and links + to canonical owners; +- roadmap documents contain future scope or completion status rather than a + duplicate current API reference; and +- no release, commit, or tag was created. + ### Completion Gate -Stage 4 and the feature are complete when: +The implementation is complete only when: -- exact public contracts and task-oriented guidance are documented at their - canonical owners; -- internal source, validation, execution, credential, and capacity boundaries - are accurately described without duplicating public GoDoc; -- all changed links resolve and no documentation claims unimplemented - behavior; -- the complete ordinary and race test suites pass offline; -- the maintained preparation example, vet, build, formatting, link, hygiene, - and whitespace checks pass; -- the final diff is limited to the feature, its tests, documentation, and - already-present roadmap changes; and -- both roadmap documents accurately report completion. +- every Stage 1 and Stage 2 gate remains satisfied; +- the complete ordinary and race-enabled suites pass; +- vet, build, formatting, the maintained offline example, Markdown links, and + whitespace checks pass; +- consumer, format, internal, future, and wishlist documentation are + consistent with the implemented boundary; +- both roadmap statuses are `Complete`; +- the working tree contains no unintended files or changes; and +- the repository is ready for maintainer review without a commit or release + having been created by this plan. ## Open Questions -None. +None. The accepted feature roadmap and the fixed decisions above fully specify +the implementation boundary. diff --git a/docs/roadmap/notarius-promptkit-wishlist.md b/docs/roadmap/notarius-promptkit-wishlist.md index e8b2871..beee476 100644 --- a/docs/roadmap/notarius-promptkit-wishlist.md +++ b/docs/roadmap/notarius-promptkit-wishlist.md @@ -88,8 +88,8 @@ corresponds atomically to the actual execution. ## Priority 2: Prompt-Independent Profile Inspection -**Disposition:** Accepted into the -[future catalog](future.md#prompt-independent-profile-inspection). +**Disposition:** Covered by the accepted +[prompt-independent profile inspection](profile-inspection.md) roadmap. ### Downstream need diff --git a/docs/roadmap/profile-inspection.md b/docs/roadmap/profile-inspection.md new file mode 100644 index 0000000..8e55d70 --- /dev/null +++ b/docs/roadmap/profile-inspection.md @@ -0,0 +1,246 @@ +# Prompt-Independent Profile Inspection + +**Status:** Accepted. + +## Purpose + +Allow consumers to look up one execution profile by ID and inspect its +structurally resolved model target without selecting a prompt, supplying +placeholder inputs, checking credential availability, or invoking a model. + +This provides a direct configuration-validation boundary for +[Notarius](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection) +and +[Weatherreporter](weatherreporter-promptkit-wishlist.md#priority-3-prompt-independent-profile-inspection). + +## Motivation + +Both downstream consumers need to reject invalid configured profile IDs before +starting application work. They currently have to construct a synthetic prompt +and call `Engine.Prepare` merely to exercise profile loading, backend lookup, +and execution-target resolution. + +That workaround couples profile validation to unrelated prompt definitions, +fixture inputs, rendering, schema behavior, and current credential +availability. Promptkit already owns profile precedence, backend membership, +and target resolution, so it should expose that cohesive capability directly. + +## Consumer Workflow + +The target public workflow is: + +```go +inspection, err := engine.InspectProfile(ctx, profileID) +if err != nil { + // Reject or report the configured profile. + return +} + +target := inspection.EffectiveModelParams +if target.APIKeyEnv != "" { + // Apply application policy for the named environment variable. +} +``` + +The target public surface is: + +```go +type ProfileInspection struct { + ProfileID string + EffectiveModelParams ExecutionTarget + APIKeyRequired bool +} + +func (e *Engine) InspectProfile( + ctx context.Context, + profileID string, +) (*ProfileInspection, error) +``` + +The declarations and GoDoc will own the exact implemented contract. The +important public shape is exact lookup through the existing engine, one +caller-owned inspection value, the resolved `ExecutionTarget`, and an explicit +signal for a direct API-key requirement. + +`EffectiveModelParams.BackendID` identifies the selected registered backend +and remains empty for endpoint-only profiles. +`EffectiveModelParams.APIKeyEnv` reports the effective credential environment +variable name. `APIKeyRequired` reports that the profile requires a direct +request credential instead. These states are mutually exclusive after normal +profile and backend precedence is applied. + +`ProfileInspection` does not need a stable JSON representation. Consumers that +persist application configuration or diagnostics can select the fields their +own format requires. + +## Lookup And Precedence + +`InspectProfile` requires a non-blank explicit profile ID. It trims surrounding +whitespace and otherwise performs the same case-sensitive exact lookup used by +ordinary execution. + +Lookup applies the engine's normal profile-source precedence: + +- programmatic profiles supplied through `WithProfiles`; +- the configured file, directory, or `fs.FS` profile source; and +- the built-in profile catalog. + +A valid higher-precedence match shadows a lower-precedence profile with the +same ID. A malformed or unreadable higher-precedence match fails rather than +silently falling back. The +[profile format reference](../formats.md#profile-definitions) remains the +canonical owner of profile-source and file-format behavior. + +Inspection never derives a profile ID from a prompt's `default_profile`; the +caller is inspecting one explicitly named profile. + +## Structural Resolution + +Inspection loads and validates the selected profile, verifies that a named +backend exists in the engine's immutable backend registry, and applies normal +framework-default, backend, and profile precedence to produce the effective +target. + +For the same engine state and profile ID, with no per-run execution override, +the inspected target must match the target that ordinary preparation would +resolve before applying request credentials and checking their availability. +This equivalence must use one shared resolution path rather than a second set +of precedence rules. + +Structural resolution includes: + +- backend routing identity; +- endpoint and model; +- sampling, token, timeout, service-tier, and reasoning settings; +- the effective credential environment-variable name or direct-key + requirement; and +- deeply copied provider-specific extra parameters. + +Inspection returns the effective target rather than a raw profile definition. +This keeps framework and backend defaults visible to consumers without +creating a second public profile-loading interface. + +The result does not include request-override presence because no +`ExecutionTargetOverride` participates in inspection. + +## Credentials And Sensitive Data + +Inspection reports credential requirements but never resolves, retains, or +returns a credential value. + +The operation does not read the named environment variable and succeeds when +that variable is absent or blank. It accepts neither a direct API key nor an +API-key environment override. Consumers decide whether credential availability +must be enforced during application configuration, while `Prepare`, +`PrepareExecution`, `Run`, and `RunPrepared` retain their execution-time +credential contracts. + +Error messages, formatting, and returned values must not expose environment +values or other resolved secrets. Existing restrictions against raw API keys +in profile sources remain unchanged. + +## Ownership, Consistency, And Concurrency + +Each successful call returns a caller-owned snapshot. Mutating the returned +target or any nested extra-parameter map or slice cannot affect the engine, +later inspection, or later execution. + +Inspection is safe to call concurrently under the engine's existing immutable +registry and repository contracts. It does not mutate profile sources or +cache a result globally. + +For filesystem-backed sources, an inspection describes the state observed by +that call. It does not freeze the profile for a later `Run`; a source may +change between operations. Consumers requiring an exact preflight-to-execution +snapshot should use the existing prepared-execution workflow. + +## Errors And Cancellation + +The operation uses existing public error categories: + +- a blank profile ID matches `ErrInvalidRequest`; +- an absent exact ID matches `ErrProfileNotFound` and not `ErrProfileLoad`; +- read, decode, validation, and source-selection failures match + `ErrProfileLoad`; and +- an unknown referenced backend or an invalid structurally resolved target + matches `ErrProfileLoad`. + +Errors should preserve useful underlying collaborator and context identities +through `errors.Is` where the existing facade does so, without exposing +internal package types. Context cancellation governs inspection and no partial +inspection result is returned on failure. + +Missing credential values are not inspection errors. The operation cannot +return capacity or model-generation failures because it performs neither +backend admission nor generation. + +## Compatibility And Boundaries + +This feature is additive. Existing profile formats, source precedence, +backend registration, `Prepare`, prepared execution, and `Run` behavior remain +unchanged. + +The method belongs on the root `Engine` facade. Profile repositories and the +backend registry remain internal implementation details, and no new public +repository interface is introduced. + +Inspection does not require prompt lookup, rendering, artifact loading, schema +loading, validation, backend-capacity admission, or model-client access. +Engine construction retains its ordinary configuration requirements; this +feature does not introduce a separate profile-only engine. + +## Documentation + +The completed documentation set has these ownership boundaries: + +- exported declarations and GoDoc own the exact method, result, ownership, + credential, error, and cancellation contracts; +- the promptkit consumer guide explains configuration-time profile inspection + and distinguishes it from `Prepare` and prepared execution; +- the profile format reference continues to own profile fields and source + precedence; and +- internal documentation describes shared profile and target resolution + without duplicating public contracts. + +## Non-Goals + +This work does not include: + +- enumerating or searching profiles; +- returning raw profile definitions or profile source paths; +- accepting per-run execution overrides, direct API keys, or API-key + environment overrides; +- checking environment-variable contents or other credential availability; +- semantic execution-target fingerprints or profile hashes; +- prompt-definition inspection or prompt default-profile resolution; +- full-corpus validation across every profile source; +- freezing a filesystem-backed profile for later execution; +- exposing backend concurrency limits, queue state, or capacity policy; +- model generation, provider health checks, or endpoint connectivity tests; +- dynamic backend or profile registration after engine construction; or +- changing current profile, backend, prepared-execution, or stable JSON + contracts. + +## Target End State + +After this work: + +- consumers can validate one configured profile without inventing a prompt or + placeholder inputs; +- lookup observes ordinary programmatic, configured-source, and built-in + precedence; +- a successful result proves that the profile exists, is valid, references a + registered backend when applicable, and resolves to a structurally valid + effective target; +- the inspected target matches ordinary preparation for the same profile and + engine state before per-run overrides and credential availability checks; +- credential requirements are visible without reading or exposing credential + values; +- returned targets and nested data are caller-owned; +- inspection performs no rendering, source loading unrelated to the profile, + capacity admission, or model work; +- existing execution workflows and compatibility contracts remain unchanged; + and +- Promptkit owns reusable profile validation while downstream applications + retain configuration policy, persistence, logging, and credential-timing + decisions. diff --git a/docs/roadmap/weatherreporter-promptkit-wishlist.md b/docs/roadmap/weatherreporter-promptkit-wishlist.md index 7b69ffd..fa4fc65 100644 --- a/docs/roadmap/weatherreporter-promptkit-wishlist.md +++ b/docs/roadmap/weatherreporter-promptkit-wishlist.md @@ -185,8 +185,8 @@ and move failures ahead of weather collection. ## Priority 3: Prompt-Independent Profile Inspection -**Disposition:** Accepted into the -[future catalog](future.md#prompt-independent-profile-inspection). +**Disposition:** Covered by the accepted +[prompt-independent profile inspection](profile-inspection.md) roadmap. ### Downstream need