From f014a078ee1356a4431dc2f71aaba5db82b9173d Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 1 Aug 2026 02:15:31 +0000 Subject: [PATCH] Plan domain-specific prompt profiles --- docs/roadmap/domain-profiles.md | 294 ++++++++++++++++++ ...ptkit-fallback-profiles-feature-request.md | 164 ++++++++++ 2 files changed, 458 insertions(+) create mode 100644 docs/roadmap/domain-profiles.md create mode 100644 docs/roadmap/promptkit-fallback-profiles-feature-request.md diff --git a/docs/roadmap/domain-profiles.md b/docs/roadmap/domain-profiles.md new file mode 100644 index 0000000..c259da5 --- /dev/null +++ b/docs/roadmap/domain-profiles.md @@ -0,0 +1,294 @@ +# Domain-Specific Prompt Profiles Roadmap + +Status: Accepted feature direction; implementation has not started. + +## Purpose + +Weatherreporter should provide stable, domain-specific Promptkit profile IDs +that express the relative resource and analysis needs of its report products. +These logical profiles should give each report an appropriate default while +allowing operators to replace any definition through the existing configured +profile source. + +This roadmap defines the scope, policy, and intended end state. A later staged +implementation plan will translate it into implementation work. This document +does not prescribe implementation stages. + +## User Intent + +The feature is intended to provide three related benefits: + +- frequent reports can use a cost-effective model by default; +- reports needing broader synthesis can select a stronger default without + forcing the same cost on every invocation; and +- an installation can map a stable Weatherreporter profile ID to a model on a + local network endpoint without modifying embedded prompts or application + code. + +`weather-light` describes the profile's intended resource tier, not a latency +guarantee. A locally hosted lightweight model may still generate slowly on the +available hardware. + +## Current State + +Daily, Today, Tomorrow, and Hourly each declare Promptkit's +`gemini-flash-latest` profile as their prompt default. The optional +`promptkit.profile` setting overrides that default for every selected report in +an invocation. + +Weatherreporter accepts either `promptkit.profile_file` or +`promptkit.profile_dir` and passes that source to Promptkit. A matching external +profile can override a Promptkit built-in profile, and the configured local +backend can support profiles that select `backend: local`. Endpoint-only +OpenAI-compatible profiles can also provide their own endpoint. + +Weatherreporter does not currently own or embed execution profiles. Promptkit's +current public source precedence also has no layer where application-owned +fallback profiles can remain overridable by the ordinary configured source. + +## Prerequisite + +Promptkit must first provide the application fallback profile capability +defined in the companion +[upstream feature request](promptkit-fallback-profiles-feature-request.md). + +Weatherreporter should adopt a tagged Promptkit release containing that public +capability. It should not depend on Promptkit internal packages, reproduce +Promptkit's profile repository behavior, or commit a local module replacement +as the production dependency. + +## Desired End State + +Weatherreporter embeds usable definitions for these exact logical profile IDs: + +- `weather-light` +- `weather-balanced` +- `weather-deep` + +The profiles are Weatherreporter-owned assets and remain behind the existing +Promptkit adapter boundary. Prompt definitions select the logical IDs, while +Promptkit resolves the effective backend, endpoint, model, and generation +settings. + +An operator can place a profile with the same ID in `profile_file` or +`profile_dir`. The operator definition completely replaces the embedded +Weatherreporter definition for that ID. If the external source does not contain +the selected ID, lookup falls through to Weatherreporter's embedded profile and +then to Promptkit's built-in catalog. + +The existing global `promptkit.profile` setting remains available as an +explicit all-report override. No new configuration field is required for the +initial feature. + +## Profile Catalog And Report Assignment + +| Profile | Meaning | Initial default reports | +| --- | --- | --- | +| `weather-light` | Lowest-cost supported tier for frequent, bounded synthesis. It makes no latency promise. | Hourly | +| `weather-balanced` | General-purpose tier for broader day-scale synthesis and forecast discussion. | Daily, Today, Tomorrow | +| `weather-deep` | Highest-capability tier for explicit operator use and future products whose measured quality benefit warrants the cost. | None initially | + +The initial assignment recognizes that Weatherreporter's deterministic modules +already perform most weather selection and calculation. A higher-capability +model should not become a default merely because it is available. Moving an +existing report to `weather-deep` requires evidence that the stronger tier +materially improves supported reasoning or output quality. + +The three profile IDs are capability policies, not permanent aliases for one +provider or model family. Their embedded definitions may change in a future +Weatherreporter release, with the change disclosed through normal release and +compatibility documentation. + +## Selection And Definition Precedence + +Profile ID selection and profile definition lookup are separate decisions. + +Weatherreporter selects the profile ID in this order: + +1. nonblank `promptkit.profile`; or +2. the exact prompt version's `default_profile`. + +Promptkit then resolves the selected profile definition in this order: + +1. explicit in-memory profiles, when used by an embedding consumer or test; +2. Weatherreporter's configured `profile_file` or `profile_dir` source; +3. Weatherreporter's embedded fallback profiles; and +4. Promptkit's embedded built-in profiles. + +A higher-precedence source falls through only when the selected ID is absent. +A matching but malformed operator profile fails before weather collection and +must not silently use the embedded definition. + +## Local Endpoint Experience + +An operator should be able to override `weather-light` with an endpoint-only +profile whose model name is understood by the local OpenAI-compatible server. +This path does not require a separate Weatherreporter local-backend setting. + +Alternatively, an override may select `backend: local`; in that case the +existing `promptkit.local.endpoint` and concurrency settings continue to own +the shared local backend definition. + +The selected local profile is deterministic configuration, not a preference +hint. Weatherreporter will not probe for a local model and will not +automatically fall back to a remote or paid profile when the endpoint is +unavailable. The failure remains visible and attributable to the selected +profile. + +## Embedded Profile Policy + +Each embedded profile must be a complete, valid Promptkit profile and must be +usable in a default installation with the documented credential mechanism. The +initial embedded profiles are expected to use Promptkit's `openrouter` backend, +allowing them to inherit its endpoint and `OPENROUTER_API_KEY` environment +variable without embedding credentials. + +Embedded definitions should include only settings that are intentional for the +selected model and supported by its backend. Avoid incidental generation +parameters that reduce portability or trigger provider-specific request +failures without a demonstrated quality benefit. + +The initial model candidates are: + +- `weather-light`: the maintained OpenRouter identifier for the Gemma 4 31B + instruction model; +- `weather-balanced`: Promptkit's maintained Gemini Flash family target; and +- `weather-deep`: Promptkit's maintained Gemini Pro family target. + +The exact identifiers and generation settings must be finalized against the +available tagged Promptkit catalog and a representative Weatherreporter +evaluation corpus before the staged implementation plan is approved. The +logical profile IDs and report assignments do not depend on those concrete +choices. + +## Prompt And Provenance Contract + +Changing a prompt's `default_profile` is a material prompt-definition change. +The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new +defaults are introduced. Prompt content and generated-text schemas need not +change solely for this feature. + +Prompt inspection must continue to occur before weather collection. It should +report the selected logical profile ID and the resolved backend and model +without exposing endpoints or credentials. + +Preparation, execution, and failure artifacts should retain both the logical +profile identity and effective model provenance already available through the +project-owned execution contract. This feature should not replace a logical +profile ID with only the concrete model name in durable records. + +## Evaluation Policy + +Concrete model assignments should be evaluated with representative, +secret-free Daily, Today, Tomorrow, and Hourly data packages. Evaluation should +consider: + +- strict-schema success rate; +- unsupported or invented weather claims; +- precipitation-timing accuracy and empty-string behavior; +- correct use of deterministic hazards, periods, and uncertainty; +- summary and forecast-discussion usefulness; +- generation latency; +- token use and provider cost; and +- behavior through a representative local OpenAI-compatible endpoint. + +The purpose is to choose an appropriate default for each tier, not to add a +permanent benchmark framework or live-provider requirement to the ordinary +test suite. Repository tests remain offline and deterministic. + +## Scope + +The completed feature includes: + +- Weatherreporter-owned embedded profile assets for all three logical IDs; +- Promptkit adapter wiring that supplies those assets as the application + fallback profile source; +- per-prompt default-profile assignments matching the catalog above; +- an exact prompt-version update for the changed definitions; +- preservation of the global profile override; +- same-ID override behavior through both supported external profile-source + forms; +- local-backend and endpoint-only override coverage; +- fail-fast inspection of missing, malformed, or unusable selected profiles; +- offline tests for selection, source precedence, effective model inspection, + batch reuse, and durable provenance; +- maintained operator examples for overriding `weather-light` locally; and +- updates to the canonical configuration, Promptkit integration, report + registry, operations, troubleshooting, internal adapter, and release + documentation as applicable when implementation lands. + +## Non-Goals + +The feature does not include: + +- automatic discovery, health checking, or benchmarking of local endpoints; +- implicit failover between local and remote profiles; +- retries with a more expensive tier after provider or validation failure; +- per-report profile configuration fields outside prompt defaults; +- profile inheritance, aliases, or field-level merging; +- runtime model selection based on weather severity, token count, or report + content; +- moving Weatherreporter profile policy into Promptkit's built-in catalog; +- exposing Promptkit types outside the adapter boundary; or +- making live provider calls part of the default repository test suite. + +## Compatibility And Operational Policy + +Existing configurations with a nonblank `promptkit.profile` retain their +all-report behavior. Existing `profile_file`, `profile_dir`, local-backend, and +credential configuration fields retain their meanings. + +Configurations that rely on the omitted profile setting will intentionally +observe new per-report defaults. This is a user-visible model-selection and +cost change and must be called out in release notes. Operators who require the +old all-report model can preserve it by setting an explicit global profile. + +An external same-ID override is an operator-owned compatibility commitment. +Weatherreporter may evolve its embedded definitions, but it must not rewrite or +silently merge an operator file. + +## Completion Criteria + +The roadmap is complete when: + +- a tagged Promptkit dependency supports the required fallback layer; +- every operational prompt selects its assigned logical profile at exact + version `1.1.0`; +- all three embedded profiles inspect successfully without an external profile + source; +- configured same-ID definitions override embedded definitions through both + `profile_file` and `profile_dir`; +- an invalid matching external definition fails without fallback; +- `weather-light` can resolve through an endpoint-only or configured-local + override without requiring code or prompt changes; +- global `promptkit.profile` still overrides every report in an invocation; +- prompt preparation and execution artifacts record the selected logical + profile and effective model; +- morning and evening batch preflight deduplicates inspection of shared + effective profile IDs as it does today; +- the default test suite remains offline and deterministic; and +- implemented behavior is documented by its canonical current-state owners. + +## Open Questions + +### Exact initial model identifiers and settings + +Recommendation: evaluate the current Gemma 4 31B instruction target for +`weather-light`, the maintained Gemini Flash target for `weather-balanced`, and +the maintained Gemini Pro target for `weather-deep`. Pin the exact profile +contents in this roadmap before deriving the staged implementation plan. + +Alternative: duplicate three current Promptkit built-in definitions without a +Weatherreporter-specific evaluation. This is faster, but it provides less +evidence that their cost, schema reliability, and weather reasoning match the +tier policy. + +### Future default use of `weather-deep` + +Recommendation: ship the profile as an explicit override with no default +report assignment. Assign it later only when a report or evaluation shows a +material benefit over `weather-balanced`. + +Alternative: assign Daily or Tomorrow to `weather-deep` immediately. This may +improve difficult synthesis, but it increases routine cost before the benefit +has been measured. diff --git a/docs/roadmap/promptkit-fallback-profiles-feature-request.md b/docs/roadmap/promptkit-fallback-profiles-feature-request.md new file mode 100644 index 0000000..8a4c541 --- /dev/null +++ b/docs/roadmap/promptkit-fallback-profiles-feature-request.md @@ -0,0 +1,164 @@ +# Promptkit Feature Request: Application Fallback Profiles + +Status: Proposed upstream capability. + +## Purpose + +Promptkit should allow a consuming application to supply an embedded fallback +profile source that sits below operator-configured profiles and above +Promptkit's own built-in profile catalog. + +This capability would let an application publish stable, domain-specific +profile IDs with useful defaults while preserving Promptkit's existing +operator-override behavior. The capability must remain application-neutral; +Promptkit should provide the source layer but should not own downstream profile +names, model assignments, or configuration policy. + +## Downstream Use Case + +Weatherreporter wants to embed profiles such as `weather-light`, +`weather-balanced`, and `weather-deep`. Report prompts would select those +logical profiles instead of naming provider- or model-specific Promptkit +profiles directly. + +An installation could then place a profile with the same ID in its configured +profile directory. For example, a local `weather-light` definition could point +to an OpenAI-compatible endpoint on the deployment network. When no operator +definition exists, Weatherreporter's embedded definition would keep the +application usable without additional profile files. + +This pattern is useful beyond Weatherreporter. Any Promptkit consumer may want +application-owned execution tiers or workload-specific defaults without +adding domain-specific profiles to Promptkit's general built-in catalog. + +## Current Constraint + +Promptkit currently resolves matching profile IDs in this order: + +1. in-memory profiles supplied through `WithProfiles`; +2. one configured profile file, `fs.FS`, or directory source; and +3. Promptkit's embedded built-in profiles. + +These layers do not express the desired application-default relationship: + +- `WithProfiles` has higher precedence than the configured source, so it would + prevent an operator file from overriding an application profile with the + same ID. +- `WithProfileFS` can hold embedded application assets, but it occupies the + configured-source layer and therefore replaces rather than sits beneath a + configured profile directory or file. +- adding downstream profile IDs to Promptkit's built-in catalog would make the + library own application-specific policy. + +A downstream application could build its own filesystem overlay, but that +would duplicate Promptkit's profile discovery, error, and precedence behavior +at the consumer boundary. + +## Requested Capability + +Add one optional application fallback profile source to engine construction. +When present, matching profile IDs should resolve in this order: + +1. in-memory profiles supplied through `WithProfiles`; +2. the ordinary configured profile source selected through a profile option or + `Config.ProfileDir`; +3. the application fallback profile source; and +4. Promptkit's embedded built-in profiles. + +When no application fallback is configured, existing source precedence and +behavior must remain unchanged. + +The minimum useful public surface is an `fs.FS`-backed option because consumers +can embed YAML profile assets. A possible API shape is: + +```go +promptkit.WithFallbackProfileFS(profileFS, ".") +``` + +The name is illustrative rather than prescriptive. A companion option for +validated `Profile` values could be added if Promptkit maintainers find it +generally useful, but it is not required for the Weatherreporter use case. + +## Required Semantics + +- A higher-precedence source falls through only when the requested profile ID + is absent. +- A malformed, unreadable, duplicate, ambiguous, or otherwise invalid matching + profile is an error and must not silently fall through. +- The fallback source uses the existing strict profile YAML format and profile + validation rules. +- Profile values are selected as a whole. This feature does not merge, + inherit, or partially overlay profile definitions. +- `InspectProfile`, `Prepare`, prepared execution, and ordinary execution use + the same profile-source precedence. +- An explicit request profile continues to take precedence over a prompt's + `default_profile`; this request concerns definition lookup after the profile + ID has been selected. +- Repeated fallback-source options should follow Promptkit's documented + same-category option convention, normally with the last value replacing the + earlier value. +- A canceled lookup, invalid fallback asset, or unknown resolved backend should + continue to cross the public facade through Promptkit's existing public error + identities. +- Exact profile inspection must remain side-effect free and must not contact a + model provider. + +## Application And Library Boundaries + +Promptkit should own: + +- the additional repository layer; +- deterministic lookup and fallthrough behavior; +- validation of the supplied source through the existing profile contract; +- consistent use of the layer across inspection and execution; and +- public documentation and tests for the added precedence rule. + +The consuming application should continue to own: + +- whether it supplies fallback profiles; +- the profile IDs and their domain meaning; +- embedded profile contents and model choices; +- application configuration and override policy; +- report- or workload-to-profile assignment; and +- credential checks and operator-facing errors beyond Promptkit's public + contract. + +## Non-Goals + +This request does not ask Promptkit to add: + +- Weatherreporter-specific profile IDs to its built-in catalog; +- profile inheritance, aliases, or field-level merging; +- automatic endpoint discovery or availability probing; +- provider failover or fallback from a failed selected profile; +- per-request model benchmarking or tier selection; +- application configuration discovery; or +- eager validation of every profile in every source. + +## Compatibility + +The feature can be additive. Engines that do not configure an application +fallback source should retain their current public behavior and precedence. +Existing uses of `WithProfiles`, `WithProfileFile`, `WithProfileFS`, and +`Config.ProfileDir` should not change meaning. + +The application fallback is deliberately lower precedence than every existing +consumer-configured source. This preserves the established expectation that a +custom profile definition can override a packaged default with the same ID. + +## Acceptance Criteria + +The capability is sufficient for downstream adoption when Promptkit can +demonstrate that: + +- a fallback-only profile can be inspected and used for preparation and + execution; +- a configured directory, file, or `fs.FS` profile with the same ID overrides + the fallback profile; +- an absent configured profile falls through to the application fallback; +- an invalid configured match fails instead of falling through; +- an absent application fallback profile continues to resolve from Promptkit's + built-in catalog; +- `WithProfiles` retains highest precedence; +- behavior is identical across inspection, preparation, and execution; and +- omitting the new option preserves existing tests and public contracts.