promptkit/docs/roadmap/weatherreporter-promptkit-wishlist.md

# Weatherreporter PromptKit Wishlist

## Purpose

This document records features and interface changes that would be useful
additions to PromptKit from the perspective of the maintainers of
Weatherreporter, a downstream application planning to replace its Scriptorium
CLI integration with PromptKit.

PromptKit v0.3.0 provides the capabilities Weatherreporter needs for the
migration. None of the ideas below is a hard adoption requirement. They are
opportunities to avoid duplicate preparation, validate configuration earlier,
improve durable failure diagnostics, and make the integration more direct.

The examples are API sketches intended to communicate the desired capability,
not prescriptive names or finalized Go contracts. The related
[Notarius PromptKit wishlist](notarius-promptkit-wishlist.md) proposes several
overlapping features from another downstream consumer's perspective.

## Priority 1: Executable Preparation Handles

**Disposition:** Accepted into the
[future catalog](future.md#executable-preparation-handles).

### Downstream need

Weatherreporter treats prompt preparation as a durable preflight boundary. It
needs to:

1. prepare the exact request that will be executed;
2. persist a safe preparation record before starting the provider call; and
3. execute without reloading or rerendering prompt, profile, schema, or input
   sources.

Persisting preflight before generation leaves useful evidence when a provider
call fails or the process is interrupted during generation.

### Current integration option

With PromptKit v0.3.0, Weatherreporter can call `Engine.Prepare`, save selected
fields from the returned `PreparedRun`, and then call `Engine.Run` with the
same request. Because `Run` performs preparation internally, the work is
repeated.

Weatherreporter plans to use embedded prompt and schema files plus immutable
inline input bytes, which removes most of the consistency risk. An external
profile file or directory can still change between the two calls, and the
second preparation remains unnecessary work.

The atomic `RunDetailed` operation proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-1-atomic-execution-with-prepared-details)
would guarantee that returned preparation details describe the completed
execution. However, returning those details only after generation would not
preserve Weatherreporter's preflight-before-generation persistence boundary.

### Requested capability

Add an opt-in two-phase API that returns a prepared execution handle:

```go
prepared, err := engine.PrepareExecution(ctx, request)
if err != nil {
	// Handle preparation failure.
}

details := prepared.Details()
// Persist a consumer-selected safe preparation record.

result, err := engine.RunPrepared(ctx, prepared)
```

The exact names and shapes are flexible. The important contract is that
`RunPrepared` executes the already prepared prompt and does not reload or
rerender its prompt, profile, schema, or input sources.

`Details` should return the same caller-owned public preparation information
currently represented by `PreparedRun`. The execution handle may retain opaque
engine-owned state needed to invoke the model and validate the response.

### Design considerations

- Keep `Prepare` and `Run` available for consumers that do not need a
  two-phase execution boundary.
- Bind a prepared handle to the engine that constructed it.
- Define whether a handle is one-shot, reusable, or safe for concurrent use.
  A one-shot contract may be the safest initial design.
- Do not give the opaque handle a stable JSON representation.
- Do not expose or serialize resolved credential values through `Details`.
- Define how a direct request API key is retained and released when an opaque
  handle must carry it until execution.
- Preserve caller-owned copies for all public details.
- Make context cancellation and backend admission timing explicit.
- Document whether profile credential environment values are resolved during
  preparation or execution.
- Ensure an execution error does not invalidate the public details already
  returned to the consumer.
- Consider whether an atomic `RunDetailed` can share the same internal
  prepared-execution implementation.

### Value to Weatherreporter

This is the highest-value upstream addition. It would preserve
Weatherreporter's durable preflight behavior, remove duplicate work, eliminate
the remaining source-consistency window, and ensure that persisted provenance
describes the actual execution.

## Priority 2: Prompt-Definition Inspection

**Disposition:** Accepted into the
[future catalog](future.md#prompt-definition-inspection).

### Downstream need

Weatherreporter has a fixed registry of seven report definitions. Each report
selects a prompt ID and one of two output workflows:

- direct Markdown; or
- structured generated text followed by application-owned domain validation
  and Markdown template rendering.

Weatherreporter will embed the PromptKit prompt definitions and private
response schemas that implement those reports. It needs to validate that the
report registry and embedded prompt corpus agree before weather collection or
provider execution.

### Current integration option

Weatherreporter can maintain synthetic data-package fixtures and call
`Engine.Prepare` for every report prompt during tests. Runtime validation can
also occur through the ordinary per-report preparation stage.

This works, but it requires complete placeholder inputs and profile resolution
when the application primarily wants to inspect prompt identity and declared
contracts.

### Requested capability

Add exact prompt-definition lookup without rendering or generation:

```go
type PromptInfo struct {
	PromptID         string
	PromptVersion    string
	PromptHash       string
	DefaultProfileID string
	Inputs           []InputDefinition
	OutputContract   OutputContract
}

func (e *Engine) ResolvePrompt(
	ctx context.Context,
	promptID string,
	promptVersion string,
) (PromptInfo, error)
```

The exact returned shape may differ. Weatherreporter needs enough information
to verify prompt existence, version selection, declared inputs, default
profile identity, output format, validation mode, and schema selection without
supplying synthetic prompt input.

### Design considerations

- Use ordinary PromptKit prompt-source precedence and exact ID/version
  selection.
- Fully load and structurally validate the selected prompt definition.
- Validate referenced prompt content files without rendering their templates.
- Resolve and validate the selected output contract and schema reference where
  practical.
- Return an opaque prompt-definition equality value rather than raw source
  bytes.
- Do not return rendered messages, schema bodies, profile credentials, or
  another source of sensitive content.
- Preserve typed or sentinel errors for missing and invalid prompts.
- Return caller-owned values.
- Enumeration of all known prompts is not required for Weatherreporter; exact
  lookup is sufficient.

### Value to Weatherreporter

This would let Weatherreporter directly verify that every report prompt
exists, requires the curated `data_package` input, and declares the expected
Markdown or JSON Schema output contract. It would reduce synthetic test setup
and move failures ahead of weather collection.

## Priority 3: Prompt-Independent Profile Inspection

**Disposition:** Accepted into the
[future catalog](future.md#prompt-independent-profile-inspection).

### Downstream need

Weatherreporter will allow operators to select an external PromptKit profile
source and may allow an explicit profile override. It should reject a missing
profile, unknown backend, malformed execution target, or unsatisfied credential
requirement before collecting weather data or writing report artifacts.

### Current integration option

Weatherreporter can validate an explicit profile by preparing one embedded
prompt with fixture input. Prompts that use their own default profiles can be
validated during their normal preparation stage.

This couples configuration validation to one prompt and requires placeholder
input even when only profile and backend resolution are relevant.

### Requested capability

The prompt-independent `ResolveProfile` API proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-2-prompt-independent-profile-inspection)
would satisfy this need. It should resolve built-in, file-backed, and
programmatic profiles, validate backend membership, report credential
requirements without resolving credential values, and preserve typed error
classification.

### Additional Weatherreporter considerations

- An explicit application profile override should be inspectable without
  selecting a report prompt.
- A prompt-definition inspection result may expose its default profile ID so
  Weatherreporter can inspect that profile separately.
- Inspection should distinguish structural profile validity from current
  credential availability so configuration validation can apply explicit
  application policy.
- An optional execution-target override should be considered only if it
  describes the same target that a later run will use.

### Value to Weatherreporter

This would improve fail-fast configuration validation and give operator-facing
errors direct profile and backend context. It is valuable but not required for
the initial migration.

## Priority 4: Eager Source Validation

**Disposition:** Deferred until prompt and profile inspection have been used
to determine whether a broader engine-wide validation operation is still
needed.

### Downstream need

PromptKit deliberately defers reading and validating filesystem and `fs.FS`
prompt, profile, and schema content until a request needs it. Weatherreporter
has a small fixed embedded prompt corpus and one optional external profile
source. It would benefit from an explicit offline validation operation for
tests, startup diagnostics, and configuration checks.

### Current integration option

Weatherreporter can prepare every report prompt with fixture inputs and inspect
any explicit profiles individually. That provides strong coverage but requires
consumer-maintained traversal and synthetic material.

### Requested capability

Consider an opt-in source-validation operation:

```go
type SourceValidationOptions struct {
	RequireCredentials bool
}

func (e *Engine) ValidateSources(
	ctx context.Context,
	opts SourceValidationOptions,
) error
```

The operation should eagerly discover and structurally validate the configured
prompt, profile, and schema sources without model generation.

### Design considerations

- Keep deferred validation as the normal `NewEngine` behavior.
- Make eager validation an explicit consumer choice.
- Validate duplicate IDs and versions, strict YAML decoding, referenced content
  files, profile/backend membership, schema syntax, and schema references.
- Distinguish structural credential declarations from current environment
  availability.
- Do not read or expose credential values when credential availability is not
  requested.
- Preserve source-specific public error identities and useful path context.
- Respect context cancellation during filesystem discovery and schema work.
- Consider whether exact prompt and profile inspection APIs already provide a
  smaller sufficient surface before adding an engine-wide operation.

### Value to Weatherreporter

This would simplify offline corpus checks and catch malformed operator profile
sources before report work begins. It is helpful but lower priority than exact
prompt and profile inspection.

## Priority 5: Structured Generation Errors

**Disposition:** Deferred pending stronger downstream demand and a narrower
design that does not duplicate prepared provenance or impose HTTP-specific
fields on injected model clients.

### Downstream need

Weatherreporter preserves redacted, inspectable failure receipts for report
runs. When model generation fails operationally, it needs to classify the
failure and retain safe execution context without parsing error prose.

Prompt preparation already supplies selected profile, backend, and model
identity. Provider status classification would add useful operator context,
especially when the built-in OpenAI-compatible client receives a non-success
HTTP status.

### Current integration option

PromptKit exposes `ErrLLMGenerate` and preserves injected client errors through
`errors.Is`. Weatherreporter can reliably classify generation failure and use
its preparation record for profile, backend, and model provenance. Any further
diagnostic detail remains a redacted error string.

### Requested capability

Consider a typed generation error that continues to match `ErrLLMGenerate`:

```go
type GenerationError struct {
	BackendID string
	Model     string
	StatusCode int
}
```

The exact fields may differ. The useful contract is safe structured context
available through `errors.As`, while `errors.Is(err, ErrLLMGenerate)` remains
compatible.

### Design considerations

- Include only fields that PromptKit knows reliably and can expose safely.
- Treat an HTTP status as optional because injected model clients may not use
  HTTP.
- Do not expose provider response bodies, endpoints, credential environment
  names, credential values, request content, or generated content.
- Do not make a structured error a second source of prompt/profile provenance
  already present in a prepared execution.
- Preserve injected client error identity.
- Keep retry and backoff policy with the consuming application.

### Value to Weatherreporter

This would improve durable failure receipts and troubleshooting, particularly
for built-in transport failures. It is not required if preparation details and
the existing sentinel remain available.

## Lower-Priority Shared Wishlist Items

### Structured Capacity Errors

**Disposition:** Accepted into the
[future catalog](future.md#structured-capacity-errors).

The typed capacity error proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-4-structured-capacity-errors)
would improve Weatherreporter diagnostics by exposing the stable backend ID
without parsing error text.

Weatherreporter currently generates batch reports sequentially and constructs
one engine per invocation, so engine-local capacity exhaustion is unlikely in
the initial design. The feature would become more valuable if report
generation later becomes concurrent or PromptKit engines become longer-lived.
It should not block adoption.

### Semantic Execution-Target Fingerprints

**Disposition:** Deferred until prompt-independent profile inspection defines
the resolved target whose configuration identity would be fingerprinted.

The semantic target digest proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-3-semantic-execution-target-fingerprints)
would provide a compact equality signal for audit metadata.

Weatherreporter does not currently reuse LLM-dependent checkpoints. Its Recent
Changes behavior compares deterministic module snapshots rather than generated
reports, so the digest has no immediate cache-correctness role. Existing
PromptKit result metadata is sufficient for the initial integration. A digest
would still be useful provenance and future-proofing, but it is not a
migration priority.

## Capabilities PromptKit Already Provides Well

PromptKit v0.3.0 already provides the essential Weatherreporter integration
surface:

- importable in-process engine construction;
- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
- offline preparation without model execution;
- versioned prompt selection;
- text, Markdown, JSON, and JSON Schema output contracts;
- single-pass output validation with raw output retained after completed
  validation failure;
- inline input artifacts with provenance URIs and input hashes;
- selected profile, backend, model, effective target, prompt hashes, timing,
  and token-usage provenance;
- endpoint-only profiles and engine-scoped backend registration;
- injected model-client and artifact-reader interfaces;
- caller cancellation, generation timeout, and transport timeout behavior; and
- public error sentinels for configuration, prompt, profile, artifact,
  validation, capacity, and generation failures.

These capabilities are sufficient for Weatherreporter to adopt PromptKit
without waiting for new upstream work.

## Responsibilities That Should Remain In Weatherreporter

The following concerns belong to Weatherreporter and should not move into
PromptKit:

- report definitions, valid periods, batches, and output naming;
- application prompt content and private report response schemas;
- deterministic weather facts, modules, and Recent Changes;
- curated `data_package` construction and persistence;
- generated-text domain validation and Markdown template rendering;
- managed artifact paths, atomic writes, metadata, and inspection commands;
- preparation, execution, raw-output, and failure-receipt schemas;
- CLI configuration loading and precedence;
- debug enablement, redaction, placement, sensitivity, and retention;
- distributor notification;
- batch continuation and any future retry policy; and
- application-level compatibility and migration policy.

## Suggested Upstream Sequence

If the PromptKit team chooses to pursue these ideas, the most useful order for
Weatherreporter would be:

1. Add executable preparation handles, ideally sharing implementation with an
   atomic detailed-run API.
2. Add prompt-definition inspection.
3. Add prompt-independent profile inspection.
4. Consider eager source validation after evaluating whether the two exact
   inspection APIs are sufficient.
5. Add structured generation errors.
6. Add structured capacity errors and semantic execution-target fingerprints
   as lower-priority operational improvements.

The first item removes the only material integration workaround. Prompt and
profile inspection improve fail-fast validation. The remaining items improve
ergonomics and diagnostics.

## Adoption Sequencing

Weatherreporter should not wait for the complete wishlist. PromptKit v0.3.0 is
already sufficient when Weatherreporter:

- embeds immutable prompt and schema assets;
- supplies immutable inline data-package bytes;
- constructs one engine per CLI invocation;
- calls `Prepare` and `Run` with the same request; and
- keeps PromptKit behind a weatherreporter-owned adapter contract.

If executable preparation handles are scheduled for a near-term PromptKit
release, Weatherreporter may defer only its final adapter implementation to
avoid implementing and then removing duplicate preparation. Prompt corpus
retrieval, application-contract design, configuration work, embedded assets,
state contracts, and offline fixtures can proceed independently.

If the feature is not scheduled, Weatherreporter can adopt v0.3.0 and keep the
duplicate `Prepare` and `Run` sequence inside its adapter. A later PromptKit
upgrade would remain localized behind that neutral boundary.

Prompt inspection, profile inspection, source validation, structured errors,
capacity details, and semantic fingerprints should not gate adoption.