284 lines
13 KiB
Markdown
284 lines
13 KiB
Markdown
# Domain-Specific Prompt Profiles Roadmap
|
|
|
|
Status: Accepted; implementation plan ready.
|
|
|
|
## Purpose
|
|
|
|
Weatherreporter should provide stable, domain-specific Promptkit profile IDs
|
|
that express the relative resource and analysis needs of its report products.
|
|
These logical profiles should give each report an appropriate default while
|
|
allowing operators to replace any definition through the existing configured
|
|
profile source.
|
|
|
|
This roadmap defines the scope, policy, and intended end state. The companion
|
|
[implementation plan](domain-profiles-implementation.md) owns the ordered work
|
|
needed to reach that state.
|
|
|
|
## User Intent
|
|
|
|
The feature is intended to provide three related benefits:
|
|
|
|
- frequent reports can use a cost-effective model by default;
|
|
- reports needing broader synthesis can select a stronger default without
|
|
forcing the same cost on every invocation; and
|
|
- an installation can map a stable Weatherreporter profile ID to a model on a
|
|
local network endpoint without modifying embedded prompts or application
|
|
code.
|
|
|
|
`weather-light` describes the profile's intended resource tier, not a latency
|
|
guarantee. A locally hosted lightweight model may still generate slowly on the
|
|
available hardware.
|
|
|
|
## Current State
|
|
|
|
Daily, Today, Tomorrow, and Hourly each declare Promptkit's
|
|
`gemini-flash-latest` profile as their prompt default. The optional
|
|
`promptkit.profile` setting overrides that default for every selected report in
|
|
an invocation.
|
|
|
|
Weatherreporter accepts either `promptkit.profile_file` or
|
|
`promptkit.profile_dir` and passes that source to Promptkit. A matching external
|
|
profile can override a Promptkit built-in profile, and the configured local
|
|
backend can support profiles that select `backend: local`. Endpoint-only
|
|
OpenAI-compatible profiles can also provide their own endpoint.
|
|
|
|
Weatherreporter does not currently own or embed execution profiles. Promptkit
|
|
v0.5.0 now provides the fallback-profile layer needed to add them without
|
|
changing the existing operator-source precedence.
|
|
|
|
## Prerequisite
|
|
|
|
Promptkit v0.5.0 provides the application fallback profile capability defined
|
|
in the companion
|
|
[upstream feature request](promptkit-fallback-profiles-feature-request.md), and
|
|
Weatherreporter now depends on that tagged release. The dependency upgrade has
|
|
passed the repository test suite and an operator smoke test. Weatherreporter
|
|
must continue to use only Promptkit's public API rather than depending on its
|
|
internal packages or reproducing its profile repository behavior.
|
|
|
|
## Desired End State
|
|
|
|
Weatherreporter embeds usable definitions for these exact logical profile IDs:
|
|
|
|
- `weather-light`
|
|
- `weather-balanced`
|
|
- `weather-deep`
|
|
|
|
The profiles are Weatherreporter-owned assets and remain behind the existing
|
|
Promptkit adapter boundary. Prompt definitions select the logical IDs, while
|
|
Promptkit resolves the effective backend, endpoint, model, and generation
|
|
settings.
|
|
|
|
An operator can place a profile with the same ID in `profile_file` or
|
|
`profile_dir`. The operator definition completely replaces the embedded
|
|
Weatherreporter definition for that ID. If the external source does not contain
|
|
the selected ID, lookup falls through to Weatherreporter's embedded profile and
|
|
then to Promptkit's built-in catalog.
|
|
|
|
The existing global `promptkit.profile` setting remains available as an
|
|
explicit all-report override. No new configuration field is required for the
|
|
initial feature.
|
|
|
|
## Profile Catalog And Report Assignment
|
|
|
|
| Profile | Meaning | Initial default reports |
|
|
| --- | --- | --- |
|
|
| `weather-light` | Lowest-cost supported tier for frequent, bounded synthesis. It makes no latency promise. | Hourly |
|
|
| `weather-balanced` | General-purpose tier for broader day-scale synthesis and forecast discussion. | Daily, Today, Tomorrow |
|
|
| `weather-deep` | Highest-capability tier for explicit operator use and future products whose measured quality benefit warrants the cost. | None initially |
|
|
|
|
The initial assignment recognizes that Weatherreporter's deterministic modules
|
|
already perform most weather selection and calculation. A higher-capability
|
|
model should not become a default merely because it is available. Moving an
|
|
existing report to `weather-deep` requires evidence that the stronger tier
|
|
materially improves supported reasoning or output quality.
|
|
|
|
The three profile IDs are capability policies, not permanent aliases for one
|
|
provider or model family. Their embedded definitions may change in a future
|
|
Weatherreporter release, with the change disclosed through normal release and
|
|
compatibility documentation.
|
|
|
|
## Selection And Definition Precedence
|
|
|
|
Profile ID selection and profile definition lookup are separate decisions.
|
|
|
|
Weatherreporter selects the profile ID in this order:
|
|
|
|
1. nonblank `promptkit.profile`; or
|
|
2. the exact prompt version's `default_profile`.
|
|
|
|
Promptkit then resolves the selected profile definition in this order:
|
|
|
|
1. explicit in-memory profiles, when used by an embedding consumer or test;
|
|
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
|
|
3. Weatherreporter's embedded fallback profiles; and
|
|
4. Promptkit's embedded built-in profiles.
|
|
|
|
A higher-precedence source falls through only when the selected ID is absent.
|
|
A matching but malformed operator profile fails before weather collection and
|
|
must not silently use the embedded definition.
|
|
|
|
## Local Endpoint Experience
|
|
|
|
An operator should be able to override `weather-light` with an endpoint-only
|
|
profile whose model name is understood by the local OpenAI-compatible server.
|
|
This path does not require a separate Weatherreporter local-backend setting.
|
|
|
|
Alternatively, an override may select `backend: local`; in that case the
|
|
existing `promptkit.local.endpoint` and concurrency settings continue to own
|
|
the shared local backend definition.
|
|
|
|
The selected local profile is deterministic configuration, not a preference
|
|
hint. Weatherreporter will not probe for a local model and will not
|
|
automatically fall back to a remote or paid profile when the endpoint is
|
|
unavailable. The failure remains visible and attributable to the selected
|
|
profile.
|
|
|
|
## Embedded Profile Policy
|
|
|
|
Each embedded profile must be a complete, valid Promptkit profile and must be
|
|
usable in a default installation with the documented credential mechanism. The
|
|
initial embedded profiles are expected to use Promptkit's `openrouter` backend,
|
|
allowing them to inherit its endpoint and `OPENROUTER_API_KEY` environment
|
|
variable without embedding credentials.
|
|
|
|
Embedded definitions should include only settings that are intentional for the
|
|
selected model and supported by its backend. Avoid incidental generation
|
|
parameters that reduce portability or trigger provider-specific request
|
|
failures without a demonstrated quality benefit.
|
|
|
|
The initial profile definitions are:
|
|
|
|
| Profile ID | OpenRouter model | Reasoning effort | Timeout | Service tier |
|
|
| --- | --- | --- | --- | --- |
|
|
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` |
|
|
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
|
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
|
|
|
These settings deliberately match the corresponding Promptkit v0.5.0
|
|
built-ins while exposing Weatherreporter-owned logical IDs. The `~` prefix is
|
|
part of each OpenRouter rolling-alias identifier. The profiles do not set
|
|
temperature, `top_p`, or output-token limits; omission preserves provider
|
|
defaults and avoids unsupported incidental parameters.
|
|
|
|
## Prompt And Active Execution Contract
|
|
|
|
Changing a prompt's `default_profile` is a material prompt-definition change.
|
|
The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new
|
|
defaults are introduced. Prompt content and generated-text schemas need not
|
|
change solely for this feature.
|
|
|
|
Prompt inspection must continue to occur before weather collection. It should
|
|
report the selected logical profile ID and the resolved backend and model
|
|
without exposing endpoints or credentials.
|
|
|
|
The active execution contract should retain both the selected logical profile
|
|
identity and the resolved backend and model through inspection, preparation,
|
|
execution, errors, and command results where those values are already exposed.
|
|
This feature must not add a new durable-provenance or cross-version artifact
|
|
contract.
|
|
|
|
The accepted [ephemeral-state roadmap](ephemeral-state.md) makes historical
|
|
prompt provenance a non-goal. Existing workspace persistence may remain while
|
|
this feature lands, but it is transitional behavior and must not be expanded or
|
|
treated as part of the profile feature's desired end state. Prompt preparation
|
|
and execution artifacts written at `1.0.1` are not required to remain readable
|
|
after the prompt definitions advance to `1.1.0`.
|
|
|
|
## Evaluation Policy
|
|
|
|
Concrete model assignments should be evaluated with representative,
|
|
secret-free Daily, Today, Tomorrow, and Hourly data packages. Evaluation should
|
|
consider:
|
|
|
|
- strict-schema success rate;
|
|
- unsupported or invented weather claims;
|
|
- precipitation-timing accuracy and empty-string behavior;
|
|
- correct use of deterministic hazards, periods, and uncertainty;
|
|
- summary and forecast-discussion usefulness;
|
|
- generation latency;
|
|
- token use and provider cost; and
|
|
- behavior through a representative local OpenAI-compatible endpoint.
|
|
|
|
The purpose is to choose an appropriate default for each tier, not to add a
|
|
permanent benchmark framework or live-provider requirement to the ordinary
|
|
test suite. Repository tests remain offline and deterministic.
|
|
|
|
## Scope
|
|
|
|
The completed feature includes:
|
|
|
|
- Weatherreporter-owned embedded profile assets for all three logical IDs;
|
|
- Promptkit adapter wiring that supplies those assets as the application
|
|
fallback profile source;
|
|
- per-prompt default-profile assignments matching the catalog above;
|
|
- an exact prompt-version update for the changed definitions;
|
|
- preservation of the global profile override;
|
|
- same-ID override behavior through both supported external profile-source
|
|
forms;
|
|
- local-backend and endpoint-only override coverage;
|
|
- fail-fast inspection of missing, malformed, or unusable selected profiles;
|
|
- offline tests for selection, source precedence, effective model inspection,
|
|
batch reuse, and active execution behavior;
|
|
- maintained operator examples for overriding `weather-light` locally; and
|
|
- updates to the canonical configuration, Promptkit integration, report
|
|
registry, operations, troubleshooting, internal adapter, and release
|
|
documentation as applicable when implementation lands.
|
|
|
|
## Non-Goals
|
|
|
|
The feature does not include:
|
|
|
|
- automatic discovery, health checking, or benchmarking of local endpoints;
|
|
- implicit failover between local and remote profiles;
|
|
- retries with a more expensive tier after provider or validation failure;
|
|
- per-report profile configuration fields outside prompt defaults;
|
|
- profile inheritance, aliases, or field-level merging;
|
|
- runtime model selection based on weather severity, token count, or report
|
|
content;
|
|
- moving Weatherreporter profile policy into Promptkit's built-in catalog;
|
|
- exposing Promptkit types outside the adapter boundary; or
|
|
- making live provider calls part of the default repository test suite.
|
|
|
|
## Compatibility And Operational Policy
|
|
|
|
Existing configurations with a nonblank `promptkit.profile` retain their
|
|
all-report behavior. Existing `profile_file`, `profile_dir`, local-backend, and
|
|
credential configuration fields retain their meanings.
|
|
|
|
Configurations that rely on the omitted profile setting will intentionally
|
|
observe new per-report defaults. This is a user-visible model-selection and
|
|
cost change and must be called out in release notes. Operators who require the
|
|
old all-report model can preserve it by setting an explicit global profile.
|
|
|
|
The prompt-version transition does not provide backward compatibility for
|
|
historical prompt preparation or execution artifacts. This is consistent with
|
|
the accepted ephemeral-state direction; the profile feature does not otherwise
|
|
redesign or remove the current workspace layout.
|
|
|
|
An external same-ID override is an operator-owned compatibility commitment.
|
|
Weatherreporter may evolve its embedded definitions, but it must not rewrite or
|
|
silently merge an operator file.
|
|
|
|
## Completion Criteria
|
|
|
|
The roadmap is complete when:
|
|
|
|
- a tagged Promptkit dependency supports the required fallback layer;
|
|
- every operational prompt selects its assigned logical profile at exact
|
|
version `1.1.0`;
|
|
- all three embedded profiles inspect successfully without an external profile
|
|
source;
|
|
- configured same-ID definitions override embedded definitions through both
|
|
`profile_file` and `profile_dir`;
|
|
- an invalid matching external definition fails without fallback;
|
|
- `weather-light` can resolve through an endpoint-only or configured-local
|
|
override without requiring code or prompt changes;
|
|
- global `promptkit.profile` still overrides every report in an invocation;
|
|
- active inspection and execution preserve the selected logical profile and
|
|
effective model through the project-owned execution contract;
|
|
- morning and evening batch preflight deduplicates inspection of shared
|
|
effective profile IDs as it does today;
|
|
- the default test suite remains offline and deterministic; and
|
|
- implemented behavior is documented by its canonical current-state owners.
|