promptkit/docs/roadmap/weatherreporter-promptkit-wishlist.md

# Weatherreporter PromptKit Wishlist

## Purpose

This document records features and interface changes that would be useful
additions to PromptKit from the perspective of the maintainers of
Weatherreporter, a downstream application planning to replace its Scriptorium
CLI integration with PromptKit.

PromptKit now provides the capabilities Weatherreporter needs for the
migration. The remaining deferred ideas are optional opportunities to validate
configuration earlier and improve durable failure diagnostics.

The examples are API sketches intended to communicate the desired capability,
not prescriptive names or finalized Go contracts. The related
[Notarius PromptKit wishlist](notarius-promptkit-wishlist.md) proposes several
overlapping features from another downstream consumer's perspective.

## Priority 1: Executable Preparation Handles

**Disposition:** Implemented as [`Engine.PrepareExecution` and
`Engine.RunPrepared`](../../engine.go). See the
[consumer guidance](../consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later).

### Downstream need

Weatherreporter treats prompt preparation as a durable preflight boundary. It
needs to:

1. prepare the exact request that will be executed;
2. persist a safe preparation record before starting the provider call; and
3. execute without reloading or rerendering prompt, profile, schema, or input
   sources.

Persisting preflight before generation leaves useful evidence when a provider
call fails or the process is interrupted during generation.

### Implemented behavior

Weatherreporter can prepare one frozen execution snapshot, persist a
caller-owned and credential-redacted `Details` value, and execute that same
snapshot through `RunPrepared`. The opaque handle is engine-bound and
single-use; an unused handle can be released with `Discard`. The consumer
guide and exported GoDoc own the exact lifecycle, credential, cancellation,
and capacity contracts.

### Value to Weatherreporter

This preserves Weatherreporter's durable preflight behavior, removes duplicate
work, eliminates the source-consistency window, and ensures that persisted
provenance describes the actual execution.

## Priority 2: Prompt-Definition Inspection

**Disposition:** Implemented as
[`Engine.InspectPrompt`](../../engine.go). See the
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-prompt-before-preparation).

### Downstream need

Weatherreporter has a fixed registry of seven report definitions. Each report
selects a prompt ID and one of two output workflows:

- direct Markdown; or
- structured generated text followed by application-owned domain validation
  and Markdown template rendering.

Weatherreporter will embed the PromptKit prompt definitions and private
response schemas that implement those reports. It needs to validate that the
report registry and embedded prompt corpus agree before weather collection or
provider execution.

### Previous integration option

Before prompt inspection was available, Weatherreporter could maintain
synthetic data-package fixtures and call `Engine.Prepare` for every report
prompt during tests. Runtime validation could also occur through the ordinary
per-report preparation stage.

This required complete placeholder inputs and profile resolution when the
application primarily wanted to inspect prompt identity and declared contracts.

### Value to Weatherreporter

The implemented interface lets Weatherreporter directly verify that every
report prompt exists, requires the curated `data_package` input, and declares
the expected Markdown or JSON Schema output contract. It reduces synthetic
test setup and moves failures ahead of weather collection.

## Priority 3: Prompt-Independent Profile Inspection

**Disposition:** Implemented as
[`Engine.InspectProfile`](../../engine.go). See the
[consumer guidance](../consumers/pkg-promptkit.md#inspect-a-profile-before-prompt-work).

### Downstream need

Weatherreporter will allow operators to select an external PromptKit profile
source and may allow an explicit profile override. It needs to reject a missing
profile, unknown backend, or malformed execution target before collecting
weather data or writing report artifacts, and to apply its own policy to a
reported credential requirement.

### Value to Weatherreporter

The implemented interface improves fail-fast configuration validation and gives
operator-facing errors direct profile and backend context. It remains optional
for the initial migration.

## Priority 4: Eager Source Validation

**Disposition:** Deferred until prompt and profile inspection have been used
to determine whether a broader engine-wide validation operation is still
needed.

### Downstream need

PromptKit deliberately defers reading and validating filesystem and `fs.FS`
prompt, profile, and schema content until a request needs it. Weatherreporter
has a small fixed embedded prompt corpus and one optional external profile
source. It would benefit from an explicit offline validation operation for
tests, startup diagnostics, and configuration checks.

### Current integration option

Weatherreporter can prepare every report prompt with fixture inputs and inspect
any explicit profiles individually. That provides strong coverage but requires
consumer-maintained traversal and synthetic material.

### Requested capability

Consider an opt-in source-validation operation:

```go
type SourceValidationOptions struct {
	RequireCredentials bool
}

func (e *Engine) ValidateSources(
	ctx context.Context,
	opts SourceValidationOptions,
) error
```

The operation should eagerly discover and structurally validate the configured
prompt, profile, and schema sources without model generation.

### Design considerations

- Keep deferred validation as the normal `NewEngine` behavior.
- Make eager validation an explicit consumer choice.
- Validate duplicate IDs and versions, strict YAML decoding, referenced content
  files, profile/backend membership, schema syntax, and schema references.
- Distinguish structural credential declarations from current environment
  availability.
- Do not read or expose credential values when credential availability is not
  requested.
- Preserve source-specific public error identities and useful path context.
- Respect context cancellation during filesystem discovery and schema work.
- Consider whether exact prompt and profile inspection APIs already provide a
  smaller sufficient surface before adding an engine-wide operation.

### Value to Weatherreporter

This would simplify offline corpus checks and catch malformed operator profile
sources before report work begins. It is helpful but lower priority than exact
prompt and profile inspection.

## Priority 5: Structured Generation Errors

**Disposition:** Deferred pending stronger downstream demand and a narrower
design that does not duplicate prepared provenance or impose HTTP-specific
fields on injected model clients.

### Downstream need

Weatherreporter preserves redacted, inspectable failure receipts for report
runs. When model generation fails operationally, it needs to classify the
failure and retain safe execution context without parsing error prose.

Prompt preparation already supplies selected profile, backend, and model
identity. Provider status classification would add useful operator context,
especially when the built-in OpenAI-compatible client receives a non-success
HTTP status.

### Current integration option

PromptKit exposes `ErrLLMGenerate` and preserves injected client errors through
`errors.Is`. Weatherreporter can reliably classify generation failure and use
its preparation record for profile, backend, and model provenance. Any further
diagnostic detail remains a redacted error string.

### Requested capability

Consider a typed generation error that continues to match `ErrLLMGenerate`:

```go
type GenerationError struct {
	BackendID string
	Model     string
	StatusCode int
}
```

The exact fields may differ. The useful contract is safe structured context
available through `errors.As`, while `errors.Is(err, ErrLLMGenerate)` remains
compatible.

### Design considerations

- Include only fields that PromptKit knows reliably and can expose safely.
- Treat an HTTP status as optional because injected model clients may not use
  HTTP.
- Do not expose provider response bodies, endpoints, credential environment
  names, credential values, request content, or generated content.
- Do not make a structured error a second source of prompt/profile provenance
  already present in a prepared execution.
- Preserve injected client error identity.
- Keep retry and backoff policy with the consuming application.

### Value to Weatherreporter

This would improve durable failure receipts and troubleshooting, particularly
for built-in transport failures. It is not required if preparation details and
the existing sentinel remain available.

## Lower-Priority Shared Wishlist Items

### Structured Capacity Errors

**Disposition:** Implemented behavior. See the consumer guide's
[Handle Errors](../consumers/pkg-promptkit.md#handle-errors) section.

PromptKit now exposes the stable backend ID for capacity rejection without
requiring Weatherreporter to parse error text.

Weatherreporter currently generates batch reports sequentially and constructs
one engine per invocation, so engine-local capacity exhaustion is unlikely in
the initial design. The typed error becomes more valuable if report generation
later becomes concurrent or PromptKit engines become longer-lived.
It should not block adoption.

### Semantic Execution-Target Fingerprints

**Disposition:** Deferred pending a separate semantic-equality design for
resolved execution targets.

The semantic target digest proposed by the
[Notarius wishlist](notarius-promptkit-wishlist.md#priority-3-semantic-execution-target-fingerprints)
would provide a compact equality signal for audit metadata.

Weatherreporter does not currently reuse LLM-dependent checkpoints. Its Recent
Changes behavior compares deterministic module snapshots rather than generated
reports, so the digest has no immediate cache-correctness role. Existing
PromptKit result metadata is sufficient for the initial integration. A digest
would still be useful provenance and future-proofing, but it is not a
migration priority.

## Capabilities PromptKit Already Provides Well

PromptKit already provides the essential Weatherreporter integration surface:

- importable in-process engine construction;
- filesystem, `fs.FS`, and programmatic prompt, profile, and schema sources;
- offline preparation without model execution;
- prepared execution handles for a durable preflight boundary;
- exact prompt and profile inspection;
- versioned prompt selection;
- text, Markdown, JSON, and JSON Schema output contracts;
- single-pass output validation with raw output retained after completed
  validation failure;
- inline input artifacts with provenance URIs and input hashes;
- selected profile, backend, model, effective target, prompt hashes, timing,
  and token-usage provenance;
- endpoint-only profiles and engine-scoped backend registration;
- injected model-client and artifact-reader interfaces;
- caller cancellation, generation timeout, and transport timeout behavior; and
- public error sentinels for configuration, prompt, profile, artifact,
  validation, capacity, and generation failures; and
- structured backend identity for capacity rejection.

These capabilities are sufficient for Weatherreporter to adopt PromptKit
without waiting for new upstream work.

## Responsibilities That Should Remain In Weatherreporter

The following concerns belong to Weatherreporter and should not move into
PromptKit:

- report definitions, valid periods, batches, and output naming;
- application prompt content and private report response schemas;
- deterministic weather facts, modules, and Recent Changes;
- curated `data_package` construction and persistence;
- generated-text domain validation and Markdown template rendering;
- managed artifact paths, atomic writes, metadata, and inspection commands;
- preparation, execution, raw-output, and failure-receipt schemas;
- CLI configuration loading and precedence;
- debug enablement, redaction, placement, sensitivity, and retention;
- distributor notification;
- batch continuation and any future retry policy; and
- application-level compatibility and migration policy.

## Suggested Upstream Sequence

For downstream adoption and any remaining upstream work, the useful order is:

1. Adopt the implemented executable preparation handles.
2. Consider eager source validation after evaluating whether the two exact
   inspection APIs are sufficient.
3. Add structured generation errors.
4. Use structured capacity errors and consider semantic execution-target
   fingerprints as lower-priority operational improvements.

The first item removes the material integration workaround. Prompt and profile
inspection improve fail-fast validation. The remaining items are optional
ergonomic and diagnostic improvements.

## Adoption Sequencing

Weatherreporter should not wait for the deferred wishlist items. The current
PromptKit interface is sufficient when Weatherreporter:

- embeds immutable prompt and schema assets;
- supplies immutable inline data-package bytes;
- constructs one engine per CLI invocation;
- prepares an execution, persists selected `Details`, and calls
  `RunPrepared`; and
- keeps PromptKit behind a weatherreporter-owned adapter contract.

Prompt inspection, profile inspection, source validation, structured errors,
capacity details, and semantic fingerprints should not gate adoption.