promptkit/docs/roadmap/weatherreporter-promptkit-wishlist.md

Weatherreporter PromptKit Wishlist

Purpose

This document records features and interface changes that would be useful additions to PromptKit from the perspective of the maintainers of Weatherreporter, a downstream application planning to replace its Scriptorium CLI integration with PromptKit.

PromptKit now provides the capabilities Weatherreporter needs for the migration. The remaining deferred ideas are optional opportunities to validate configuration earlier and improve durable failure diagnostics.

The examples are API sketches intended to communicate the desired capability, not prescriptive names or finalized Go contracts. The related Notarius PromptKit wishlist proposes several overlapping features from another downstream consumer's perspective.

Priority 1: Executable Preparation Handles

Disposition: Implemented as Engine.PrepareExecution and Engine.RunPrepared. See the consumer guidance.

Downstream need

Weatherreporter treats prompt preparation as a durable preflight boundary. It needs to:

1. prepare the exact request that will be executed;
2. persist a safe preparation record before starting the provider call; and
3. execute without reloading or rerendering prompt, profile, schema, or input sources.

Persisting preflight before generation leaves useful evidence when a provider call fails or the process is interrupted during generation.

Implemented behavior

Weatherreporter can prepare one frozen execution snapshot, persist a caller-owned and credential-redacted Details value, and execute that same snapshot through RunPrepared. The opaque handle is engine-bound and single-use; an unused handle can be released with Discard. The consumer guide and exported GoDoc own the exact lifecycle, credential, cancellation, and capacity contracts.

Value to Weatherreporter

This preserves Weatherreporter's durable preflight behavior, removes duplicate work, eliminates the source-consistency window, and ensures that persisted provenance describes the actual execution.

Priority 2: Prompt-Definition Inspection

Disposition: Implemented as Engine.InspectPrompt. See the consumer guidance.

Downstream need

Weatherreporter has a fixed registry of seven report definitions. Each report selects a prompt ID and one of two output workflows:

• direct Markdown; or
• structured generated text followed by application-owned domain validation and Markdown template rendering.

Weatherreporter will embed the PromptKit prompt definitions and private response schemas that implement those reports. It needs to validate that the report registry and embedded prompt corpus agree before weather collection or provider execution.

Previous integration option

Before prompt inspection was available, Weatherreporter could maintain synthetic data-package fixtures and call Engine.Prepare for every report prompt during tests. Runtime validation could also occur through the ordinary per-report preparation stage.

This required complete placeholder inputs and profile resolution when the application primarily wanted to inspect prompt identity and declared contracts.

Value to Weatherreporter

The implemented interface lets Weatherreporter directly verify that every report prompt exists, requires the curated data_package input, and declares the expected Markdown or JSON Schema output contract. It reduces synthetic test setup and moves failures ahead of weather collection.

Priority 3: Prompt-Independent Profile Inspection

Disposition: Implemented as Engine.InspectProfile. See the consumer guidance.

Downstream need

Weatherreporter will allow operators to select an external PromptKit profile source and may allow an explicit profile override. It needs to reject a missing profile, unknown backend, or malformed execution target before collecting weather data or writing report artifacts, and to apply its own policy to a reported credential requirement.

Value to Weatherreporter

The implemented interface improves fail-fast configuration validation and gives operator-facing errors direct profile and backend context. It remains optional for the initial migration.

Priority 4: Eager Source Validation

Disposition: Deferred until prompt and profile inspection have been used to determine whether a broader engine-wide validation operation is still needed.

Downstream need

PromptKit deliberately defers reading and validating filesystem and fs.FS prompt, profile, and schema content until a request needs it. Weatherreporter has a small fixed embedded prompt corpus and one optional external profile source. It would benefit from an explicit offline validation operation for tests, startup diagnostics, and configuration checks.

Current integration option

Weatherreporter can prepare every report prompt with fixture inputs and inspect any explicit profiles individually. That provides strong coverage but requires consumer-maintained traversal and synthetic material.

Requested capability

Consider an opt-in source-validation operation:

type SourceValidationOptions struct {
	RequireCredentials bool
}

func (e *Engine) ValidateSources(
	ctx context.Context,
	opts SourceValidationOptions,
) error

The operation should eagerly discover and structurally validate the configured prompt, profile, and schema sources without model generation.

Design considerations

• Keep deferred validation as the normal NewEngine behavior.
• Make eager validation an explicit consumer choice.
• Validate duplicate IDs and versions, strict YAML decoding, referenced content files, profile/backend membership, schema syntax, and schema references.
• Distinguish structural credential declarations from current environment availability.
• Do not read or expose credential values when credential availability is not requested.
• Preserve source-specific public error identities and useful path context.
• Respect context cancellation during filesystem discovery and schema work.
• Consider whether exact prompt and profile inspection APIs already provide a smaller sufficient surface before adding an engine-wide operation.

Value to Weatherreporter

This would simplify offline corpus checks and catch malformed operator profile sources before report work begins. It is helpful but lower priority than exact prompt and profile inspection.

Priority 5: Structured Generation Errors

Disposition: Deferred pending stronger downstream demand and a narrower design that does not duplicate prepared provenance or impose HTTP-specific fields on injected model clients.

Downstream need

Weatherreporter preserves redacted, inspectable failure receipts for report runs. When model generation fails operationally, it needs to classify the failure and retain safe execution context without parsing error prose.

Prompt preparation already supplies selected profile, backend, and model identity. Provider status classification would add useful operator context, especially when the built-in OpenAI-compatible client receives a non-success HTTP status.

Current integration option

PromptKit exposes ErrLLMGenerate and preserves injected client errors through errors.Is. Weatherreporter can reliably classify generation failure and use its preparation record for profile, backend, and model provenance. Any further diagnostic detail remains a redacted error string.

Requested capability

Consider a typed generation error that continues to match ErrLLMGenerate:

type GenerationError struct {
	BackendID string
	Model     string
	StatusCode int
}

The exact fields may differ. The useful contract is safe structured context available through errors.As, while errors.Is(err, ErrLLMGenerate) remains compatible.

Design considerations

• Include only fields that PromptKit knows reliably and can expose safely.
• Treat an HTTP status as optional because injected model clients may not use HTTP.
• Do not expose provider response bodies, endpoints, credential environment names, credential values, request content, or generated content.
• Do not make a structured error a second source of prompt/profile provenance already present in a prepared execution.
• Preserve injected client error identity.
• Keep retry and backoff policy with the consuming application.

Value to Weatherreporter

This would improve durable failure receipts and troubleshooting, particularly for built-in transport failures. It is not required if preparation details and the existing sentinel remain available.

Lower-Priority Shared Wishlist Items

Structured Capacity Errors

Disposition: Implemented behavior. See the consumer guide's Handle Errors section.

PromptKit now exposes the stable backend ID for capacity rejection without requiring Weatherreporter to parse error text.

Weatherreporter currently generates batch reports sequentially and constructs one engine per invocation, so engine-local capacity exhaustion is unlikely in the initial design. The typed error becomes more valuable if report generation later becomes concurrent or PromptKit engines become longer-lived. It should not block adoption.

Semantic Execution-Target Fingerprints

Disposition: Deferred pending a separate semantic-equality design for resolved execution targets.

The semantic target digest proposed by the Notarius wishlist would provide a compact equality signal for audit metadata.

Weatherreporter does not currently reuse LLM-dependent checkpoints. Its Recent Changes behavior compares deterministic module snapshots rather than generated reports, so the digest has no immediate cache-correctness role. Existing PromptKit result metadata is sufficient for the initial integration. A digest would still be useful provenance and future-proofing, but it is not a migration priority.

Capabilities PromptKit Already Provides Well

PromptKit already provides the essential Weatherreporter integration surface:

• importable in-process engine construction;
• filesystem, fs.FS, and programmatic prompt, profile, and schema sources;
• offline preparation without model execution;
• prepared execution handles for a durable preflight boundary;
• exact prompt and profile inspection;
• versioned prompt selection;
• text, Markdown, JSON, and JSON Schema output contracts;
• single-pass output validation with raw output retained after completed validation failure;
• inline input artifacts with provenance URIs and input hashes;
• selected profile, backend, model, effective target, prompt hashes, timing, and token-usage provenance;
• endpoint-only profiles and engine-scoped backend registration;
• injected model-client and artifact-reader interfaces;
• caller cancellation, generation timeout, and transport timeout behavior; and
• public error sentinels for configuration, prompt, profile, artifact, validation, capacity, and generation failures; and
• structured backend identity for capacity rejection.

These capabilities are sufficient for Weatherreporter to adopt PromptKit without waiting for new upstream work.

Responsibilities That Should Remain In Weatherreporter

The following concerns belong to Weatherreporter and should not move into PromptKit:

• report definitions, valid periods, batches, and output naming;
• application prompt content and private report response schemas;
• deterministic weather facts, modules, and Recent Changes;
• curated data_package construction and persistence;
• generated-text domain validation and Markdown template rendering;
• managed artifact paths, atomic writes, metadata, and inspection commands;
• preparation, execution, raw-output, and failure-receipt schemas;
• CLI configuration loading and precedence;
• debug enablement, redaction, placement, sensitivity, and retention;
• distributor notification;
• batch continuation and any future retry policy; and
• application-level compatibility and migration policy.

Suggested Upstream Sequence

For downstream adoption and any remaining upstream work, the useful order is:

1. Adopt the implemented executable preparation handles.
2. Consider eager source validation after evaluating whether the two exact inspection APIs are sufficient.
3. Add structured generation errors.
4. Use structured capacity errors and consider semantic execution-target fingerprints as lower-priority operational improvements.

The first item removes the material integration workaround. Prompt and profile inspection improve fail-fast validation. The remaining items are optional ergonomic and diagnostic improvements.

Adoption Sequencing

Weatherreporter should not wait for the deferred wishlist items. The current PromptKit interface is sufficient when Weatherreporter:

• embeds immutable prompt and schema assets;
• supplies immutable inline data-package bytes;
• constructs one engine per CLI invocation;
• prepares an execution, persists selected Details, and calls RunPrepared; and
• keeps PromptKit behind a weatherreporter-owned adapter contract.

Prompt inspection, profile inspection, source validation, structured errors, capacity details, and semantic fingerprints should not gate adoption.