promptkit/docs/roadmap/weatherreporter-promptkit-wishlist.md

Weatherreporter PromptKit Wishlist

Purpose

This document records features and interface changes that would be useful additions to PromptKit from the perspective of the maintainers of Weatherreporter, a downstream application planning to replace its Scriptorium CLI integration with PromptKit.

PromptKit v0.3.0 provides the capabilities Weatherreporter needs for the migration. None of the ideas below is a hard adoption requirement. They are opportunities to avoid duplicate preparation, validate configuration earlier, improve durable failure diagnostics, and make the integration more direct.

The examples are API sketches intended to communicate the desired capability, not prescriptive names or finalized Go contracts. The related Notarius PromptKit wishlist proposes several overlapping features from another downstream consumer's perspective.

Priority 1: Executable Preparation Handles

Disposition: Accepted into the future catalog.

Downstream need

Weatherreporter treats prompt preparation as a durable preflight boundary. It needs to:

1. prepare the exact request that will be executed;
2. persist a safe preparation record before starting the provider call; and
3. execute without reloading or rerendering prompt, profile, schema, or input sources.

Persisting preflight before generation leaves useful evidence when a provider call fails or the process is interrupted during generation.

Current integration option

With PromptKit v0.3.0, Weatherreporter can call Engine.Prepare, save selected fields from the returned PreparedRun, and then call Engine.Run with the same request. Because Run performs preparation internally, the work is repeated.

Weatherreporter plans to use embedded prompt and schema files plus immutable inline input bytes, which removes most of the consistency risk. An external profile file or directory can still change between the two calls, and the second preparation remains unnecessary work.

The atomic RunDetailed operation proposed by the Notarius wishlist would guarantee that returned preparation details describe the completed execution. However, returning those details only after generation would not preserve Weatherreporter's preflight-before-generation persistence boundary.

Requested capability

Add an opt-in two-phase API that returns a prepared execution handle:

prepared, err := engine.PrepareExecution(ctx, request)
if err != nil {
	// Handle preparation failure.
}

details := prepared.Details()
// Persist a consumer-selected safe preparation record.

result, err := engine.RunPrepared(ctx, prepared)

The exact names and shapes are flexible. The important contract is that RunPrepared executes the already prepared prompt and does not reload or rerender its prompt, profile, schema, or input sources.

Details should return the same caller-owned public preparation information currently represented by PreparedRun. The execution handle may retain opaque engine-owned state needed to invoke the model and validate the response.

Design considerations

• Keep Prepare and Run available for consumers that do not need a two-phase execution boundary.
• Bind a prepared handle to the engine that constructed it.
• Define whether a handle is one-shot, reusable, or safe for concurrent use. A one-shot contract may be the safest initial design.
• Do not give the opaque handle a stable JSON representation.
• Do not expose or serialize resolved credential values through Details.
• Define how a direct request API key is retained and released when an opaque handle must carry it until execution.
• Preserve caller-owned copies for all public details.
• Make context cancellation and backend admission timing explicit.
• Document whether profile credential environment values are resolved during preparation or execution.
• Ensure an execution error does not invalidate the public details already returned to the consumer.
• Consider whether an atomic RunDetailed can share the same internal prepared-execution implementation.

Value to Weatherreporter

This is the highest-value upstream addition. It would preserve Weatherreporter's durable preflight behavior, remove duplicate work, eliminate the remaining source-consistency window, and ensure that persisted provenance describes the actual execution.

Priority 2: Prompt-Definition Inspection

Disposition: Accepted into the future catalog.

Downstream need

Weatherreporter has a fixed registry of seven report definitions. Each report selects a prompt ID and one of two output workflows:

• direct Markdown; or
• structured generated text followed by application-owned domain validation and Markdown template rendering.

Weatherreporter will embed the PromptKit prompt definitions and private response schemas that implement those reports. It needs to validate that the report registry and embedded prompt corpus agree before weather collection or provider execution.

Current integration option

Weatherreporter can maintain synthetic data-package fixtures and call Engine.Prepare for every report prompt during tests. Runtime validation can also occur through the ordinary per-report preparation stage.

This works, but it requires complete placeholder inputs and profile resolution when the application primarily wants to inspect prompt identity and declared contracts.

Requested capability

Add exact prompt-definition lookup without rendering or generation:

type PromptInfo struct {
	PromptID         string
	PromptVersion    string
	PromptHash       string
	DefaultProfileID string
	Inputs           []InputDefinition
	OutputContract   OutputContract
}

func (e *Engine) ResolvePrompt(
	ctx context.Context,
	promptID string,
	promptVersion string,
) (PromptInfo, error)

The exact returned shape may differ. Weatherreporter needs enough information to verify prompt existence, version selection, declared inputs, default profile identity, output format, validation mode, and schema selection without supplying synthetic prompt input.

Design considerations

• Use ordinary PromptKit prompt-source precedence and exact ID/version selection.
• Fully load and structurally validate the selected prompt definition.
• Validate referenced prompt content files without rendering their templates.
• Resolve and validate the selected output contract and schema reference where practical.
• Return an opaque prompt-definition equality value rather than raw source bytes.
• Do not return rendered messages, schema bodies, profile credentials, or another source of sensitive content.
• Preserve typed or sentinel errors for missing and invalid prompts.
• Return caller-owned values.
• Enumeration of all known prompts is not required for Weatherreporter; exact lookup is sufficient.

Value to Weatherreporter

This would let Weatherreporter directly verify that every report prompt exists, requires the curated data_package input, and declares the expected Markdown or JSON Schema output contract. It would reduce synthetic test setup and move failures ahead of weather collection.

Priority 3: Prompt-Independent Profile Inspection

Disposition: Accepted into the future catalog.

Downstream need

Weatherreporter will allow operators to select an external PromptKit profile source and may allow an explicit profile override. It should reject a missing profile, unknown backend, malformed execution target, or unsatisfied credential requirement before collecting weather data or writing report artifacts.

Current integration option

Weatherreporter can validate an explicit profile by preparing one embedded prompt with fixture input. Prompts that use their own default profiles can be validated during their normal preparation stage.

This couples configuration validation to one prompt and requires placeholder input even when only profile and backend resolution are relevant.

Requested capability

The prompt-independent ResolveProfile API proposed by the Notarius wishlist would satisfy this need. It should resolve built-in, file-backed, and programmatic profiles, validate backend membership, report credential requirements without resolving credential values, and preserve typed error classification.

Additional Weatherreporter considerations

• An explicit application profile override should be inspectable without selecting a report prompt.
• A prompt-definition inspection result may expose its default profile ID so Weatherreporter can inspect that profile separately.
• Inspection should distinguish structural profile validity from current credential availability so configuration validation can apply explicit application policy.
• An optional execution-target override should be considered only if it describes the same target that a later run will use.

Value to Weatherreporter

This would improve fail-fast configuration validation and give operator-facing errors direct profile and backend context. It is valuable but not required for the initial migration.

Priority 4: Eager Source Validation

Disposition: Deferred until prompt and profile inspection have been used to determine whether a broader engine-wide validation operation is still needed.

Downstream need

PromptKit deliberately defers reading and validating filesystem and fs.FS prompt, profile, and schema content until a request needs it. Weatherreporter has a small fixed embedded prompt corpus and one optional external profile source. It would benefit from an explicit offline validation operation for tests, startup diagnostics, and configuration checks.

Current integration option

Weatherreporter can prepare every report prompt with fixture inputs and inspect any explicit profiles individually. That provides strong coverage but requires consumer-maintained traversal and synthetic material.

Requested capability

Consider an opt-in source-validation operation:

type SourceValidationOptions struct {
	RequireCredentials bool
}

func (e *Engine) ValidateSources(
	ctx context.Context,
	opts SourceValidationOptions,
) error

The operation should eagerly discover and structurally validate the configured prompt, profile, and schema sources without model generation.

Design considerations

• Keep deferred validation as the normal NewEngine behavior.
• Make eager validation an explicit consumer choice.
• Validate duplicate IDs and versions, strict YAML decoding, referenced content files, profile/backend membership, schema syntax, and schema references.
• Distinguish structural credential declarations from current environment availability.
• Do not read or expose credential values when credential availability is not requested.
• Preserve source-specific public error identities and useful path context.
• Respect context cancellation during filesystem discovery and schema work.
• Consider whether exact prompt and profile inspection APIs already provide a smaller sufficient surface before adding an engine-wide operation.

Value to Weatherreporter

This would simplify offline corpus checks and catch malformed operator profile sources before report work begins. It is helpful but lower priority than exact prompt and profile inspection.

Priority 5: Structured Generation Errors

Disposition: Deferred pending stronger downstream demand and a narrower design that does not duplicate prepared provenance or impose HTTP-specific fields on injected model clients.

Downstream need

Weatherreporter preserves redacted, inspectable failure receipts for report runs. When model generation fails operationally, it needs to classify the failure and retain safe execution context without parsing error prose.

Prompt preparation already supplies selected profile, backend, and model identity. Provider status classification would add useful operator context, especially when the built-in OpenAI-compatible client receives a non-success HTTP status.

Current integration option

PromptKit exposes ErrLLMGenerate and preserves injected client errors through errors.Is. Weatherreporter can reliably classify generation failure and use its preparation record for profile, backend, and model provenance. Any further diagnostic detail remains a redacted error string.

Requested capability

Consider a typed generation error that continues to match ErrLLMGenerate:

type GenerationError struct {
	BackendID string
	Model     string
	StatusCode int
}

The exact fields may differ. The useful contract is safe structured context available through errors.As, while errors.Is(err, ErrLLMGenerate) remains compatible.

Design considerations

• Include only fields that PromptKit knows reliably and can expose safely.
• Treat an HTTP status as optional because injected model clients may not use HTTP.
• Do not expose provider response bodies, endpoints, credential environment names, credential values, request content, or generated content.
• Do not make a structured error a second source of prompt/profile provenance already present in a prepared execution.
• Preserve injected client error identity.
• Keep retry and backoff policy with the consuming application.

Value to Weatherreporter

This would improve durable failure receipts and troubleshooting, particularly for built-in transport failures. It is not required if preparation details and the existing sentinel remain available.

Lower-Priority Shared Wishlist Items

Structured Capacity Errors

Disposition: Accepted into the future catalog.

The typed capacity error proposed by the Notarius wishlist would improve Weatherreporter diagnostics by exposing the stable backend ID without parsing error text.

Weatherreporter currently generates batch reports sequentially and constructs one engine per invocation, so engine-local capacity exhaustion is unlikely in the initial design. The feature would become more valuable if report generation later becomes concurrent or PromptKit engines become longer-lived. It should not block adoption.

Semantic Execution-Target Fingerprints

Disposition: Deferred until prompt-independent profile inspection defines the resolved target whose configuration identity would be fingerprinted.

The semantic target digest proposed by the Notarius wishlist would provide a compact equality signal for audit metadata.

Weatherreporter does not currently reuse LLM-dependent checkpoints. Its Recent Changes behavior compares deterministic module snapshots rather than generated reports, so the digest has no immediate cache-correctness role. Existing PromptKit result metadata is sufficient for the initial integration. A digest would still be useful provenance and future-proofing, but it is not a migration priority.

Capabilities PromptKit Already Provides Well

PromptKit v0.3.0 already provides the essential Weatherreporter integration surface:

• importable in-process engine construction;
• filesystem, fs.FS, and programmatic prompt, profile, and schema sources;
• offline preparation without model execution;
• versioned prompt selection;
• text, Markdown, JSON, and JSON Schema output contracts;
• single-pass output validation with raw output retained after completed validation failure;
• inline input artifacts with provenance URIs and input hashes;
• selected profile, backend, model, effective target, prompt hashes, timing, and token-usage provenance;
• endpoint-only profiles and engine-scoped backend registration;
• injected model-client and artifact-reader interfaces;
• caller cancellation, generation timeout, and transport timeout behavior; and
• public error sentinels for configuration, prompt, profile, artifact, validation, capacity, and generation failures.

These capabilities are sufficient for Weatherreporter to adopt PromptKit without waiting for new upstream work.

Responsibilities That Should Remain In Weatherreporter

The following concerns belong to Weatherreporter and should not move into PromptKit:

• report definitions, valid periods, batches, and output naming;
• application prompt content and private report response schemas;
• deterministic weather facts, modules, and Recent Changes;
• curated data_package construction and persistence;
• generated-text domain validation and Markdown template rendering;
• managed artifact paths, atomic writes, metadata, and inspection commands;
• preparation, execution, raw-output, and failure-receipt schemas;
• CLI configuration loading and precedence;
• debug enablement, redaction, placement, sensitivity, and retention;
• distributor notification;
• batch continuation and any future retry policy; and
• application-level compatibility and migration policy.

Suggested Upstream Sequence

If the PromptKit team chooses to pursue these ideas, the most useful order for Weatherreporter would be:

1. Add executable preparation handles, ideally sharing implementation with an atomic detailed-run API.
2. Add prompt-definition inspection.
3. Add prompt-independent profile inspection.
4. Consider eager source validation after evaluating whether the two exact inspection APIs are sufficient.
5. Add structured generation errors.
6. Add structured capacity errors and semantic execution-target fingerprints as lower-priority operational improvements.

The first item removes the only material integration workaround. Prompt and profile inspection improve fail-fast validation. The remaining items improve ergonomics and diagnostics.

Adoption Sequencing

Weatherreporter should not wait for the complete wishlist. PromptKit v0.3.0 is already sufficient when Weatherreporter:

• embeds immutable prompt and schema assets;
• supplies immutable inline data-package bytes;
• constructs one engine per CLI invocation;
• calls Prepare and Run with the same request; and
• keeps PromptKit behind a weatherreporter-owned adapter contract.

If executable preparation handles are scheduled for a near-term PromptKit release, Weatherreporter may defer only its final adapter implementation to avoid implementing and then removing duplicate preparation. Prompt corpus retrieval, application-contract design, configuration work, embedded assets, state contracts, and offline fixtures can proceed independently.

If the feature is not scheduled, Weatherreporter can adopt v0.3.0 and keep the duplicate Prepare and Run sequence inside its adapter. A later PromptKit upgrade would remain localized behind that neutral boundary.

Prompt inspection, profile inspection, source validation, structured errors, capacity details, and semantic fingerprints should not gate adoption.