promptkit/docs/roadmap/weatherreporter-promptkit-wishlist.md

# Weatherreporter PromptKit Wishlist

## Purpose

This document records features and interface changes that would be useful
additions to PromptKit from the perspective of the maintainers of
Weatherreporter, a downstream application planning to replace its Scriptorium
CLI integration with PromptKit.

PromptKit v0.3.0 provides the capabilities Weatherreporter needs for the
migration. None of the ideas below is a hard adoption requirement. They are
opportunities to avoid duplicate preparation, validate configuration earlier,
improve durable failure diagnostics, and make the integration more direct.

The examples are API sketches intended to communicate the desired capability,
not prescriptive names or finalized Go contracts. The related
[Notarius PromptKit wishlist](notarius-promptkit-wishlist.md) proposes several
overlapping features from another downstream consumer's perspective.

## Priority 1: Executable Preparation Handles

**Disposition:** Accepted into the
[executable preparation handles](prepared-execution.md) feature roadmap.

### Downstream need

Weatherreporter treats prompt preparation as a durable preflight boundary. It
needs to:

1. prepare the exact request that will be executed;
2. persist a safe preparation record before starting the provider call; and
3. execute without reloading or rerendering prompt, profile, schema, or input
   sources.

Persisting preflight before generation leaves useful evidence when a provider
call fails or the process is interrupted during generation.

### Current integration option

With PromptKit v0.3.0, Weatherreporter can call `Engine.Prepare`, save selected
fields from the returned `PreparedRun`, and then call `Engine.Run` with the
same request. Because `Run` performs preparation internally, the work is
repeated.

Weatherreporter plans to use embedded prompt and schema files plus immutable
inline input bytes, which removes most of the consistency risk. An external
profile file or directory can still change between the two calls, and the
second preparation remains unnecessary work.

The atomic `RunDetailed` operation proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-1-atomic-execution-with-prepared-details)
would guarantee that returned preparation details describe the completed
execution. However, returning those details only after generation would not
preserve Weatherreporter's preflight-before-generation persistence boundary.

### Requested capability

Add an opt-in two-phase API that returns a prepared execution handle:

```go
prepared, err := engine.PrepareExecution(ctx, request)
if err != nil {
	// Handle preparation failure.
}

details := prepared.Details()
// Persist a consumer-selected safe preparation record.

result, err := engine.RunPrepared(ctx, prepared)
```

The exact names and shapes are flexible. The important contract is that
`RunPrepared` executes the already prepared prompt and does not reload or
rerender its prompt, profile, schema, or input sources.

`Details` should return the same caller-owned public preparation information
currently represented by `PreparedRun`. The execution handle may retain opaque
engine-owned state needed to invoke the model and validate the response.

### Design considerations

- Keep `Prepare` and `Run` available for consumers that do not need a
  two-phase execution boundary.
- Bind a prepared handle to the engine that constructed it.
- Define whether a handle is one-shot, reusable, or safe for concurrent use.
  A one-shot contract may be the safest initial design.
- Do not give the opaque handle a stable JSON representation.
- Do not expose or serialize resolved credential values through `Details`.
- Define how a direct request API key is retained and released when an opaque
  handle must carry it until execution.
- Preserve caller-owned copies for all public details.
- Make context cancellation and backend admission timing explicit.
- Document whether profile credential environment values are resolved during
  preparation or execution.
- Ensure an execution error does not invalidate the public details already
  returned to the consumer.
- Consider whether an atomic `RunDetailed` can share the same internal
  prepared-execution implementation.

### Value to Weatherreporter

This is the highest-value upstream addition. It would preserve
Weatherreporter's durable preflight behavior, remove duplicate work, eliminate
the remaining source-consistency window, and ensure that persisted provenance
describes the actual execution.

## Priority 2: Prompt-Definition Inspection

**Disposition:** Implemented as
[`Engine.InspectPrompt`](../../engine.go). See the
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-prompt-before-preparation).

### Downstream need

Weatherreporter has a fixed registry of seven report definitions. Each report
selects a prompt ID and one of two output workflows:

- direct Markdown; or
- structured generated text followed by application-owned domain validation
  and Markdown template rendering.

Weatherreporter will embed the PromptKit prompt definitions and private
response schemas that implement those reports. It needs to validate that the
report registry and embedded prompt corpus agree before weather collection or
provider execution.

### Previous integration option

Before prompt inspection was available, Weatherreporter could maintain
synthetic data-package fixtures and call `Engine.Prepare` for every report
prompt during tests. Runtime validation could also occur through the ordinary
per-report preparation stage.

This required complete placeholder inputs and profile resolution when the
application primarily wanted to inspect prompt identity and declared contracts.

### Value to Weatherreporter

The implemented interface lets Weatherreporter directly verify that every
report prompt exists, requires the curated `data_package` input, and declares
the expected Markdown or JSON Schema output contract. It reduces synthetic
test setup and moves failures ahead of weather collection.

## Priority 3: Prompt-Independent Profile Inspection

**Disposition:** Implemented as
[`Engine.InspectProfile`](../../engine.go). See the
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work).

### Downstream need

Weatherreporter will allow operators to select an external PromptKit profile
source and may allow an explicit profile override. It needs to reject a missing
profile, unknown backend, or malformed execution target before collecting
weather data or writing report artifacts, and to apply its own policy to a
reported credential requirement.

### Value to Weatherreporter

The implemented interface improves fail-fast configuration validation and gives
operator-facing errors direct profile and backend context. It remains optional
for the initial migration.

## Priority 4: Eager Source Validation

**Disposition:** Deferred until prompt and profile inspection have been used
to determine whether a broader engine-wide validation operation is still
needed.

### Downstream need

PromptKit deliberately defers reading and validating filesystem and `fs.FS`
prompt, profile, and schema content until a request needs it. Weatherreporter
has a small fixed embedded prompt corpus and one optional external profile
source. It would benefit from an explicit offline validation operation for
tests, startup diagnostics, and configuration checks.

### Current integration option

Weatherreporter can prepare every report prompt with fixture inputs and inspect
any explicit profiles individually. That provides strong coverage but requires
consumer-maintained traversal and synthetic material.

### Requested capability

Consider an opt-in source-validation operation:

```go
type SourceValidationOptions struct {
	RequireCredentials bool
}

func (e *Engine) ValidateSources(
	ctx context.Context,
	opts SourceValidationOptions,
) error
```

The operation should eagerly discover and structurally validate the configured
prompt, profile, and schema sources without model generation.

### Design considerations

- Keep deferred validation as the normal `NewEngine` behavior.
- Make eager validation an explicit consumer choice.
- Validate duplicate IDs and versions, strict YAML decoding, referenced content
  files, profile/backend membership, schema syntax, and schema references.
- Distinguish structural credential declarations from current environment
  availability.
- Do not read or expose credential values when credential availability is not
  requested.
- Preserve source-specific public error identities and useful path context.
- Respect context cancellation during filesystem discovery and schema work.
- Consider whether exact prompt and profile inspection APIs already provide a
  smaller sufficient surface before adding an engine-wide operation.

### Value to Weatherreporter

This would simplify offline corpus checks and catch malformed operator profile
sources before report work begins. It is helpful but lower priority than exact
prompt and profile inspection.

## Priority 5: Structured Generation Errors

**Disposition:** Deferred pending stronger downstream demand and a narrower
design that does not duplicate prepared provenance or impose HTTP-specific
fields on injected model clients.

### Downstream need

Weatherreporter preserves redacted, inspectable failure receipts for report
runs. When model generation fails operationally, it needs to classify the
failure and retain safe execution context without parsing error prose.

Prompt preparation already supplies selected profile, backend, and model
identity. Provider status classification would add useful operator context,
especially when the built-in OpenAI-compatible client receives a non-success
HTTP status.

### Current integration option

PromptKit exposes `ErrLLMGenerate` and preserves injected client errors through
`errors.Is`. Weatherreporter can reliably classify generation failure and use
its preparation record for profile, backend, and model provenance. Any further
diagnostic detail remains a redacted error string.

### Requested capability

Consider a typed generation error that continues to match `ErrLLMGenerate`:

```go
type GenerationError struct {
	BackendID string
	Model     string
	StatusCode int
}
```

The exact fields may differ. The useful contract is safe structured context
available through `errors.As`, while `errors.Is(err, ErrLLMGenerate)` remains
compatible.

### Design considerations

- Include only fields that PromptKit knows reliably and can expose safely.
- Treat an HTTP status as optional because injected model clients may not use
  HTTP.
- Do not expose provider response bodies, endpoints, credential environment
  names, credential values, request content, or generated content.
- Do not make a structured error a second source of prompt/profile provenance
  already present in a prepared execution.
- Preserve injected client error identity.
- Keep retry and backoff policy with the consuming application.

### Value to Weatherreporter

This would improve durable failure receipts and troubleshooting, particularly
for built-in transport failures. It is not required if preparation details and
the existing sentinel remain available.

## Lower-Priority Shared Wishlist Items

### Structured Capacity Errors

**Disposition:** Implemented behavior. See the consumer guide's
[Handle Errors](../consumers/pkg-promptkit.md#handle-errors) section.

PromptKit now exposes the stable backend ID for capacity rejection without
requiring Weatherreporter to parse error text.

Weatherreporter currently generates batch reports sequentially and constructs
one engine per invocation, so engine-local capacity exhaustion is unlikely in
the initial design. The typed error becomes more valuable if report generation
later becomes concurrent or PromptKit engines become longer-lived.
It should not block adoption.

### Semantic Execution-Target Fingerprints

**Disposition:** Deferred pending a separate semantic-equality design for
resolved execution targets.

The semantic target digest proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-3-semantic-execution-target-fingerprints)
would provide a compact equality signal for audit metadata.

Weatherreporter does not currently reuse LLM-dependent checkpoints. Its Recent
Changes behavior compares deterministic module snapshots rather than generated
reports, so the digest has no immediate cache-correctness role. Existing
PromptKit result metadata is sufficient for the initial integration. A digest
would still be useful provenance and future-proofing, but it is not a
migration priority.

## Capabilities PromptKit Already Provides Well

PromptKit v0.3.0 already provides the essential Weatherreporter integration
surface:

- importable in-process engine construction;
- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
- offline preparation without model execution;
- versioned prompt selection;
- text, Markdown, JSON, and JSON Schema output contracts;
- single-pass output validation with raw output retained after completed
  validation failure;
- inline input artifacts with provenance URIs and input hashes;
- selected profile, backend, model, effective target, prompt hashes, timing,
  and token-usage provenance;
- endpoint-only profiles and engine-scoped backend registration;
- injected model-client and artifact-reader interfaces;
- caller cancellation, generation timeout, and transport timeout behavior; and
- public error sentinels for configuration, prompt, profile, artifact,
  validation, capacity, and generation failures.

These capabilities are sufficient for Weatherreporter to adopt PromptKit
without waiting for new upstream work.

## Responsibilities That Should Remain In Weatherreporter

The following concerns belong to Weatherreporter and should not move into
PromptKit:

- report definitions, valid periods, batches, and output naming;
- application prompt content and private report response schemas;
- deterministic weather facts, modules, and Recent Changes;
- curated `data_package` construction and persistence;
- generated-text domain validation and Markdown template rendering;
- managed artifact paths, atomic writes, metadata, and inspection commands;
- preparation, execution, raw-output, and failure-receipt schemas;
- CLI configuration loading and precedence;
- debug enablement, redaction, placement, sensitivity, and retention;
- distributor notification;
- batch continuation and any future retry policy; and
- application-level compatibility and migration policy.

## Suggested Upstream Sequence

If the PromptKit team chooses to pursue these ideas, the most useful order for
Weatherreporter would be:

1. Add executable preparation handles, ideally sharing implementation with an
   atomic detailed-run API.
2. Consider eager source validation after evaluating whether the two exact
   inspection APIs are sufficient.
3. Add structured generation errors.
4. Use structured capacity errors and consider semantic execution-target
   fingerprints as lower-priority operational improvements.

The first item removes the only material integration workaround. Prompt and
profile inspection improve fail-fast validation. The remaining items improve
ergonomics and diagnostics.

## Adoption Sequencing

Weatherreporter should not wait for the complete wishlist. PromptKit v0.3.0 is
already sufficient when Weatherreporter:

- embeds immutable prompt and schema assets;
- supplies immutable inline data-package bytes;
- constructs one engine per CLI invocation;
- calls `Prepare` and `Run` with the same request; and
- keeps PromptKit behind a weatherreporter-owned adapter contract.

If executable preparation handles are scheduled for a near-term PromptKit
release, Weatherreporter may defer only its final adapter implementation to
avoid implementing and then removing duplicate preparation. Prompt corpus
retrieval, application-contract design, configuration work, embedded assets,
state contracts, and offline fixtures can proceed independently.

If the feature is not scheduled, Weatherreporter can adopt v0.3.0 and keep the
duplicate `Prepare` and `Run` sequence inside its adapter. A later PromptKit
upgrade would remain localized behind that neutral boundary.

Prompt inspection, profile inspection, source validation, structured errors,
capacity details, and semantic fingerprints should not gate adoption.