Plan domain-specific prompt profiles

This commit is contained in:
2026-08-01 02:15:31 +00:00
parent 8c19ad763b
commit f014a078ee
2 changed files with 458 additions and 0 deletions

View File

@@ -0,0 +1,294 @@
# Domain-Specific Prompt Profiles Roadmap
Status: Accepted feature direction; implementation has not started.
## Purpose
Weatherreporter should provide stable, domain-specific Promptkit profile IDs
that express the relative resource and analysis needs of its report products.
These logical profiles should give each report an appropriate default while
allowing operators to replace any definition through the existing configured
profile source.
This roadmap defines the scope, policy, and intended end state. A later staged
implementation plan will translate it into implementation work. This document
does not prescribe implementation stages.
## User Intent
The feature is intended to provide three related benefits:
- frequent reports can use a cost-effective model by default;
- reports needing broader synthesis can select a stronger default without
forcing the same cost on every invocation; and
- an installation can map a stable Weatherreporter profile ID to a model on a
local network endpoint without modifying embedded prompts or application
code.
`weather-light` describes the profile's intended resource tier, not a latency
guarantee. A locally hosted lightweight model may still generate slowly on the
available hardware.
## Current State
Daily, Today, Tomorrow, and Hourly each declare Promptkit's
`gemini-flash-latest` profile as their prompt default. The optional
`promptkit.profile` setting overrides that default for every selected report in
an invocation.
Weatherreporter accepts either `promptkit.profile_file` or
`promptkit.profile_dir` and passes that source to Promptkit. A matching external
profile can override a Promptkit built-in profile, and the configured local
backend can support profiles that select `backend: local`. Endpoint-only
OpenAI-compatible profiles can also provide their own endpoint.
Weatherreporter does not currently own or embed execution profiles. Promptkit's
current public source precedence also has no layer where application-owned
fallback profiles can remain overridable by the ordinary configured source.
## Prerequisite
Promptkit must first provide the application fallback profile capability
defined in the companion
[upstream feature request](promptkit-fallback-profiles-feature-request.md).
Weatherreporter should adopt a tagged Promptkit release containing that public
capability. It should not depend on Promptkit internal packages, reproduce
Promptkit's profile repository behavior, or commit a local module replacement
as the production dependency.
## Desired End State
Weatherreporter embeds usable definitions for these exact logical profile IDs:
- `weather-light`
- `weather-balanced`
- `weather-deep`
The profiles are Weatherreporter-owned assets and remain behind the existing
Promptkit adapter boundary. Prompt definitions select the logical IDs, while
Promptkit resolves the effective backend, endpoint, model, and generation
settings.
An operator can place a profile with the same ID in `profile_file` or
`profile_dir`. The operator definition completely replaces the embedded
Weatherreporter definition for that ID. If the external source does not contain
the selected ID, lookup falls through to Weatherreporter's embedded profile and
then to Promptkit's built-in catalog.
The existing global `promptkit.profile` setting remains available as an
explicit all-report override. No new configuration field is required for the
initial feature.
## Profile Catalog And Report Assignment
| Profile | Meaning | Initial default reports |
| --- | --- | --- |
| `weather-light` | Lowest-cost supported tier for frequent, bounded synthesis. It makes no latency promise. | Hourly |
| `weather-balanced` | General-purpose tier for broader day-scale synthesis and forecast discussion. | Daily, Today, Tomorrow |
| `weather-deep` | Highest-capability tier for explicit operator use and future products whose measured quality benefit warrants the cost. | None initially |
The initial assignment recognizes that Weatherreporter's deterministic modules
already perform most weather selection and calculation. A higher-capability
model should not become a default merely because it is available. Moving an
existing report to `weather-deep` requires evidence that the stronger tier
materially improves supported reasoning or output quality.
The three profile IDs are capability policies, not permanent aliases for one
provider or model family. Their embedded definitions may change in a future
Weatherreporter release, with the change disclosed through normal release and
compatibility documentation.
## Selection And Definition Precedence
Profile ID selection and profile definition lookup are separate decisions.
Weatherreporter selects the profile ID in this order:
1. nonblank `promptkit.profile`; or
2. the exact prompt version's `default_profile`.
Promptkit then resolves the selected profile definition in this order:
1. explicit in-memory profiles, when used by an embedding consumer or test;
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
3. Weatherreporter's embedded fallback profiles; and
4. Promptkit's embedded built-in profiles.
A higher-precedence source falls through only when the selected ID is absent.
A matching but malformed operator profile fails before weather collection and
must not silently use the embedded definition.
## Local Endpoint Experience
An operator should be able to override `weather-light` with an endpoint-only
profile whose model name is understood by the local OpenAI-compatible server.
This path does not require a separate Weatherreporter local-backend setting.
Alternatively, an override may select `backend: local`; in that case the
existing `promptkit.local.endpoint` and concurrency settings continue to own
the shared local backend definition.
The selected local profile is deterministic configuration, not a preference
hint. Weatherreporter will not probe for a local model and will not
automatically fall back to a remote or paid profile when the endpoint is
unavailable. The failure remains visible and attributable to the selected
profile.
## Embedded Profile Policy
Each embedded profile must be a complete, valid Promptkit profile and must be
usable in a default installation with the documented credential mechanism. The
initial embedded profiles are expected to use Promptkit's `openrouter` backend,
allowing them to inherit its endpoint and `OPENROUTER_API_KEY` environment
variable without embedding credentials.
Embedded definitions should include only settings that are intentional for the
selected model and supported by its backend. Avoid incidental generation
parameters that reduce portability or trigger provider-specific request
failures without a demonstrated quality benefit.
The initial model candidates are:
- `weather-light`: the maintained OpenRouter identifier for the Gemma 4 31B
instruction model;
- `weather-balanced`: Promptkit's maintained Gemini Flash family target; and
- `weather-deep`: Promptkit's maintained Gemini Pro family target.
The exact identifiers and generation settings must be finalized against the
available tagged Promptkit catalog and a representative Weatherreporter
evaluation corpus before the staged implementation plan is approved. The
logical profile IDs and report assignments do not depend on those concrete
choices.
## Prompt And Provenance Contract
Changing a prompt's `default_profile` is a material prompt-definition change.
The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new
defaults are introduced. Prompt content and generated-text schemas need not
change solely for this feature.
Prompt inspection must continue to occur before weather collection. It should
report the selected logical profile ID and the resolved backend and model
without exposing endpoints or credentials.
Preparation, execution, and failure artifacts should retain both the logical
profile identity and effective model provenance already available through the
project-owned execution contract. This feature should not replace a logical
profile ID with only the concrete model name in durable records.
## Evaluation Policy
Concrete model assignments should be evaluated with representative,
secret-free Daily, Today, Tomorrow, and Hourly data packages. Evaluation should
consider:
- strict-schema success rate;
- unsupported or invented weather claims;
- precipitation-timing accuracy and empty-string behavior;
- correct use of deterministic hazards, periods, and uncertainty;
- summary and forecast-discussion usefulness;
- generation latency;
- token use and provider cost; and
- behavior through a representative local OpenAI-compatible endpoint.
The purpose is to choose an appropriate default for each tier, not to add a
permanent benchmark framework or live-provider requirement to the ordinary
test suite. Repository tests remain offline and deterministic.
## Scope
The completed feature includes:
- Weatherreporter-owned embedded profile assets for all three logical IDs;
- Promptkit adapter wiring that supplies those assets as the application
fallback profile source;
- per-prompt default-profile assignments matching the catalog above;
- an exact prompt-version update for the changed definitions;
- preservation of the global profile override;
- same-ID override behavior through both supported external profile-source
forms;
- local-backend and endpoint-only override coverage;
- fail-fast inspection of missing, malformed, or unusable selected profiles;
- offline tests for selection, source precedence, effective model inspection,
batch reuse, and durable provenance;
- maintained operator examples for overriding `weather-light` locally; and
- updates to the canonical configuration, Promptkit integration, report
registry, operations, troubleshooting, internal adapter, and release
documentation as applicable when implementation lands.
## Non-Goals
The feature does not include:
- automatic discovery, health checking, or benchmarking of local endpoints;
- implicit failover between local and remote profiles;
- retries with a more expensive tier after provider or validation failure;
- per-report profile configuration fields outside prompt defaults;
- profile inheritance, aliases, or field-level merging;
- runtime model selection based on weather severity, token count, or report
content;
- moving Weatherreporter profile policy into Promptkit's built-in catalog;
- exposing Promptkit types outside the adapter boundary; or
- making live provider calls part of the default repository test suite.
## Compatibility And Operational Policy
Existing configurations with a nonblank `promptkit.profile` retain their
all-report behavior. Existing `profile_file`, `profile_dir`, local-backend, and
credential configuration fields retain their meanings.
Configurations that rely on the omitted profile setting will intentionally
observe new per-report defaults. This is a user-visible model-selection and
cost change and must be called out in release notes. Operators who require the
old all-report model can preserve it by setting an explicit global profile.
An external same-ID override is an operator-owned compatibility commitment.
Weatherreporter may evolve its embedded definitions, but it must not rewrite or
silently merge an operator file.
## Completion Criteria
The roadmap is complete when:
- a tagged Promptkit dependency supports the required fallback layer;
- every operational prompt selects its assigned logical profile at exact
version `1.1.0`;
- all three embedded profiles inspect successfully without an external profile
source;
- configured same-ID definitions override embedded definitions through both
`profile_file` and `profile_dir`;
- an invalid matching external definition fails without fallback;
- `weather-light` can resolve through an endpoint-only or configured-local
override without requiring code or prompt changes;
- global `promptkit.profile` still overrides every report in an invocation;
- prompt preparation and execution artifacts record the selected logical
profile and effective model;
- morning and evening batch preflight deduplicates inspection of shared
effective profile IDs as it does today;
- the default test suite remains offline and deterministic; and
- implemented behavior is documented by its canonical current-state owners.
## Open Questions
### Exact initial model identifiers and settings
Recommendation: evaluate the current Gemma 4 31B instruction target for
`weather-light`, the maintained Gemini Flash target for `weather-balanced`, and
the maintained Gemini Pro target for `weather-deep`. Pin the exact profile
contents in this roadmap before deriving the staged implementation plan.
Alternative: duplicate three current Promptkit built-in definitions without a
Weatherreporter-specific evaluation. This is faster, but it provides less
evidence that their cost, schema reliability, and weather reasoning match the
tier policy.
### Future default use of `weather-deep`
Recommendation: ship the profile as an explicit override with no default
report assignment. Assign it later only when a report or evaluation shows a
material benefit over `weather-balanced`.
Alternative: assign Daily or Tomorrow to `weather-deep` immediately. This may
improve difficult synthesis, but it increases routine cost before the benefit
has been measured.

View File

@@ -0,0 +1,164 @@
# Promptkit Feature Request: Application Fallback Profiles
Status: Proposed upstream capability.
## Purpose
Promptkit should allow a consuming application to supply an embedded fallback
profile source that sits below operator-configured profiles and above
Promptkit's own built-in profile catalog.
This capability would let an application publish stable, domain-specific
profile IDs with useful defaults while preserving Promptkit's existing
operator-override behavior. The capability must remain application-neutral;
Promptkit should provide the source layer but should not own downstream profile
names, model assignments, or configuration policy.
## Downstream Use Case
Weatherreporter wants to embed profiles such as `weather-light`,
`weather-balanced`, and `weather-deep`. Report prompts would select those
logical profiles instead of naming provider- or model-specific Promptkit
profiles directly.
An installation could then place a profile with the same ID in its configured
profile directory. For example, a local `weather-light` definition could point
to an OpenAI-compatible endpoint on the deployment network. When no operator
definition exists, Weatherreporter's embedded definition would keep the
application usable without additional profile files.
This pattern is useful beyond Weatherreporter. Any Promptkit consumer may want
application-owned execution tiers or workload-specific defaults without
adding domain-specific profiles to Promptkit's general built-in catalog.
## Current Constraint
Promptkit currently resolves matching profile IDs in this order:
1. in-memory profiles supplied through `WithProfiles`;
2. one configured profile file, `fs.FS`, or directory source; and
3. Promptkit's embedded built-in profiles.
These layers do not express the desired application-default relationship:
- `WithProfiles` has higher precedence than the configured source, so it would
prevent an operator file from overriding an application profile with the
same ID.
- `WithProfileFS` can hold embedded application assets, but it occupies the
configured-source layer and therefore replaces rather than sits beneath a
configured profile directory or file.
- adding downstream profile IDs to Promptkit's built-in catalog would make the
library own application-specific policy.
A downstream application could build its own filesystem overlay, but that
would duplicate Promptkit's profile discovery, error, and precedence behavior
at the consumer boundary.
## Requested Capability
Add one optional application fallback profile source to engine construction.
When present, matching profile IDs should resolve in this order:
1. in-memory profiles supplied through `WithProfiles`;
2. the ordinary configured profile source selected through a profile option or
`Config.ProfileDir`;
3. the application fallback profile source; and
4. Promptkit's embedded built-in profiles.
When no application fallback is configured, existing source precedence and
behavior must remain unchanged.
The minimum useful public surface is an `fs.FS`-backed option because consumers
can embed YAML profile assets. A possible API shape is:
```go
promptkit.WithFallbackProfileFS(profileFS, ".")
```
The name is illustrative rather than prescriptive. A companion option for
validated `Profile` values could be added if Promptkit maintainers find it
generally useful, but it is not required for the Weatherreporter use case.
## Required Semantics
- A higher-precedence source falls through only when the requested profile ID
is absent.
- A malformed, unreadable, duplicate, ambiguous, or otherwise invalid matching
profile is an error and must not silently fall through.
- The fallback source uses the existing strict profile YAML format and profile
validation rules.
- Profile values are selected as a whole. This feature does not merge,
inherit, or partially overlay profile definitions.
- `InspectProfile`, `Prepare`, prepared execution, and ordinary execution use
the same profile-source precedence.
- An explicit request profile continues to take precedence over a prompt's
`default_profile`; this request concerns definition lookup after the profile
ID has been selected.
- Repeated fallback-source options should follow Promptkit's documented
same-category option convention, normally with the last value replacing the
earlier value.
- A canceled lookup, invalid fallback asset, or unknown resolved backend should
continue to cross the public facade through Promptkit's existing public error
identities.
- Exact profile inspection must remain side-effect free and must not contact a
model provider.
## Application And Library Boundaries
Promptkit should own:
- the additional repository layer;
- deterministic lookup and fallthrough behavior;
- validation of the supplied source through the existing profile contract;
- consistent use of the layer across inspection and execution; and
- public documentation and tests for the added precedence rule.
The consuming application should continue to own:
- whether it supplies fallback profiles;
- the profile IDs and their domain meaning;
- embedded profile contents and model choices;
- application configuration and override policy;
- report- or workload-to-profile assignment; and
- credential checks and operator-facing errors beyond Promptkit's public
contract.
## Non-Goals
This request does not ask Promptkit to add:
- Weatherreporter-specific profile IDs to its built-in catalog;
- profile inheritance, aliases, or field-level merging;
- automatic endpoint discovery or availability probing;
- provider failover or fallback from a failed selected profile;
- per-request model benchmarking or tier selection;
- application configuration discovery; or
- eager validation of every profile in every source.
## Compatibility
The feature can be additive. Engines that do not configure an application
fallback source should retain their current public behavior and precedence.
Existing uses of `WithProfiles`, `WithProfileFile`, `WithProfileFS`, and
`Config.ProfileDir` should not change meaning.
The application fallback is deliberately lower precedence than every existing
consumer-configured source. This preserves the established expectation that a
custom profile definition can override a packaged default with the same ID.
## Acceptance Criteria
The capability is sufficient for downstream adoption when Promptkit can
demonstrate that:
- a fallback-only profile can be inspected and used for preparation and
execution;
- a configured directory, file, or `fs.FS` profile with the same ID overrides
the fallback profile;
- an absent configured profile falls through to the application fallback;
- an invalid configured match fails instead of falling through;
- an absent application fallback profile continues to resolve from Promptkit's
built-in catalog;
- `WithProfiles` retains highest precedence;
- behavior is identical across inspection, preparation, and execution; and
- omitting the new option preserves existing tests and public contracts.