Plan domain-specific prompt profiles
This commit is contained in:
294
docs/roadmap/domain-profiles.md
Normal file
294
docs/roadmap/domain-profiles.md
Normal file
@@ -0,0 +1,294 @@
|
||||
# Domain-Specific Prompt Profiles Roadmap
|
||||
|
||||
Status: Accepted feature direction; implementation has not started.
|
||||
|
||||
## Purpose
|
||||
|
||||
Weatherreporter should provide stable, domain-specific Promptkit profile IDs
|
||||
that express the relative resource and analysis needs of its report products.
|
||||
These logical profiles should give each report an appropriate default while
|
||||
allowing operators to replace any definition through the existing configured
|
||||
profile source.
|
||||
|
||||
This roadmap defines the scope, policy, and intended end state. A later staged
|
||||
implementation plan will translate it into implementation work. This document
|
||||
does not prescribe implementation stages.
|
||||
|
||||
## User Intent
|
||||
|
||||
The feature is intended to provide three related benefits:
|
||||
|
||||
- frequent reports can use a cost-effective model by default;
|
||||
- reports needing broader synthesis can select a stronger default without
|
||||
forcing the same cost on every invocation; and
|
||||
- an installation can map a stable Weatherreporter profile ID to a model on a
|
||||
local network endpoint without modifying embedded prompts or application
|
||||
code.
|
||||
|
||||
`weather-light` describes the profile's intended resource tier, not a latency
|
||||
guarantee. A locally hosted lightweight model may still generate slowly on the
|
||||
available hardware.
|
||||
|
||||
## Current State
|
||||
|
||||
Daily, Today, Tomorrow, and Hourly each declare Promptkit's
|
||||
`gemini-flash-latest` profile as their prompt default. The optional
|
||||
`promptkit.profile` setting overrides that default for every selected report in
|
||||
an invocation.
|
||||
|
||||
Weatherreporter accepts either `promptkit.profile_file` or
|
||||
`promptkit.profile_dir` and passes that source to Promptkit. A matching external
|
||||
profile can override a Promptkit built-in profile, and the configured local
|
||||
backend can support profiles that select `backend: local`. Endpoint-only
|
||||
OpenAI-compatible profiles can also provide their own endpoint.
|
||||
|
||||
Weatherreporter does not currently own or embed execution profiles. Promptkit's
|
||||
current public source precedence also has no layer where application-owned
|
||||
fallback profiles can remain overridable by the ordinary configured source.
|
||||
|
||||
## Prerequisite
|
||||
|
||||
Promptkit must first provide the application fallback profile capability
|
||||
defined in the companion
|
||||
[upstream feature request](promptkit-fallback-profiles-feature-request.md).
|
||||
|
||||
Weatherreporter should adopt a tagged Promptkit release containing that public
|
||||
capability. It should not depend on Promptkit internal packages, reproduce
|
||||
Promptkit's profile repository behavior, or commit a local module replacement
|
||||
as the production dependency.
|
||||
|
||||
## Desired End State
|
||||
|
||||
Weatherreporter embeds usable definitions for these exact logical profile IDs:
|
||||
|
||||
- `weather-light`
|
||||
- `weather-balanced`
|
||||
- `weather-deep`
|
||||
|
||||
The profiles are Weatherreporter-owned assets and remain behind the existing
|
||||
Promptkit adapter boundary. Prompt definitions select the logical IDs, while
|
||||
Promptkit resolves the effective backend, endpoint, model, and generation
|
||||
settings.
|
||||
|
||||
An operator can place a profile with the same ID in `profile_file` or
|
||||
`profile_dir`. The operator definition completely replaces the embedded
|
||||
Weatherreporter definition for that ID. If the external source does not contain
|
||||
the selected ID, lookup falls through to Weatherreporter's embedded profile and
|
||||
then to Promptkit's built-in catalog.
|
||||
|
||||
The existing global `promptkit.profile` setting remains available as an
|
||||
explicit all-report override. No new configuration field is required for the
|
||||
initial feature.
|
||||
|
||||
## Profile Catalog And Report Assignment
|
||||
|
||||
| Profile | Meaning | Initial default reports |
|
||||
| --- | --- | --- |
|
||||
| `weather-light` | Lowest-cost supported tier for frequent, bounded synthesis. It makes no latency promise. | Hourly |
|
||||
| `weather-balanced` | General-purpose tier for broader day-scale synthesis and forecast discussion. | Daily, Today, Tomorrow |
|
||||
| `weather-deep` | Highest-capability tier for explicit operator use and future products whose measured quality benefit warrants the cost. | None initially |
|
||||
|
||||
The initial assignment recognizes that Weatherreporter's deterministic modules
|
||||
already perform most weather selection and calculation. A higher-capability
|
||||
model should not become a default merely because it is available. Moving an
|
||||
existing report to `weather-deep` requires evidence that the stronger tier
|
||||
materially improves supported reasoning or output quality.
|
||||
|
||||
The three profile IDs are capability policies, not permanent aliases for one
|
||||
provider or model family. Their embedded definitions may change in a future
|
||||
Weatherreporter release, with the change disclosed through normal release and
|
||||
compatibility documentation.
|
||||
|
||||
## Selection And Definition Precedence
|
||||
|
||||
Profile ID selection and profile definition lookup are separate decisions.
|
||||
|
||||
Weatherreporter selects the profile ID in this order:
|
||||
|
||||
1. nonblank `promptkit.profile`; or
|
||||
2. the exact prompt version's `default_profile`.
|
||||
|
||||
Promptkit then resolves the selected profile definition in this order:
|
||||
|
||||
1. explicit in-memory profiles, when used by an embedding consumer or test;
|
||||
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
|
||||
3. Weatherreporter's embedded fallback profiles; and
|
||||
4. Promptkit's embedded built-in profiles.
|
||||
|
||||
A higher-precedence source falls through only when the selected ID is absent.
|
||||
A matching but malformed operator profile fails before weather collection and
|
||||
must not silently use the embedded definition.
|
||||
|
||||
## Local Endpoint Experience
|
||||
|
||||
An operator should be able to override `weather-light` with an endpoint-only
|
||||
profile whose model name is understood by the local OpenAI-compatible server.
|
||||
This path does not require a separate Weatherreporter local-backend setting.
|
||||
|
||||
Alternatively, an override may select `backend: local`; in that case the
|
||||
existing `promptkit.local.endpoint` and concurrency settings continue to own
|
||||
the shared local backend definition.
|
||||
|
||||
The selected local profile is deterministic configuration, not a preference
|
||||
hint. Weatherreporter will not probe for a local model and will not
|
||||
automatically fall back to a remote or paid profile when the endpoint is
|
||||
unavailable. The failure remains visible and attributable to the selected
|
||||
profile.
|
||||
|
||||
## Embedded Profile Policy
|
||||
|
||||
Each embedded profile must be a complete, valid Promptkit profile and must be
|
||||
usable in a default installation with the documented credential mechanism. The
|
||||
initial embedded profiles are expected to use Promptkit's `openrouter` backend,
|
||||
allowing them to inherit its endpoint and `OPENROUTER_API_KEY` environment
|
||||
variable without embedding credentials.
|
||||
|
||||
Embedded definitions should include only settings that are intentional for the
|
||||
selected model and supported by its backend. Avoid incidental generation
|
||||
parameters that reduce portability or trigger provider-specific request
|
||||
failures without a demonstrated quality benefit.
|
||||
|
||||
The initial model candidates are:
|
||||
|
||||
- `weather-light`: the maintained OpenRouter identifier for the Gemma 4 31B
|
||||
instruction model;
|
||||
- `weather-balanced`: Promptkit's maintained Gemini Flash family target; and
|
||||
- `weather-deep`: Promptkit's maintained Gemini Pro family target.
|
||||
|
||||
The exact identifiers and generation settings must be finalized against the
|
||||
available tagged Promptkit catalog and a representative Weatherreporter
|
||||
evaluation corpus before the staged implementation plan is approved. The
|
||||
logical profile IDs and report assignments do not depend on those concrete
|
||||
choices.
|
||||
|
||||
## Prompt And Provenance Contract
|
||||
|
||||
Changing a prompt's `default_profile` is a material prompt-definition change.
|
||||
The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new
|
||||
defaults are introduced. Prompt content and generated-text schemas need not
|
||||
change solely for this feature.
|
||||
|
||||
Prompt inspection must continue to occur before weather collection. It should
|
||||
report the selected logical profile ID and the resolved backend and model
|
||||
without exposing endpoints or credentials.
|
||||
|
||||
Preparation, execution, and failure artifacts should retain both the logical
|
||||
profile identity and effective model provenance already available through the
|
||||
project-owned execution contract. This feature should not replace a logical
|
||||
profile ID with only the concrete model name in durable records.
|
||||
|
||||
## Evaluation Policy
|
||||
|
||||
Concrete model assignments should be evaluated with representative,
|
||||
secret-free Daily, Today, Tomorrow, and Hourly data packages. Evaluation should
|
||||
consider:
|
||||
|
||||
- strict-schema success rate;
|
||||
- unsupported or invented weather claims;
|
||||
- precipitation-timing accuracy and empty-string behavior;
|
||||
- correct use of deterministic hazards, periods, and uncertainty;
|
||||
- summary and forecast-discussion usefulness;
|
||||
- generation latency;
|
||||
- token use and provider cost; and
|
||||
- behavior through a representative local OpenAI-compatible endpoint.
|
||||
|
||||
The purpose is to choose an appropriate default for each tier, not to add a
|
||||
permanent benchmark framework or live-provider requirement to the ordinary
|
||||
test suite. Repository tests remain offline and deterministic.
|
||||
|
||||
## Scope
|
||||
|
||||
The completed feature includes:
|
||||
|
||||
- Weatherreporter-owned embedded profile assets for all three logical IDs;
|
||||
- Promptkit adapter wiring that supplies those assets as the application
|
||||
fallback profile source;
|
||||
- per-prompt default-profile assignments matching the catalog above;
|
||||
- an exact prompt-version update for the changed definitions;
|
||||
- preservation of the global profile override;
|
||||
- same-ID override behavior through both supported external profile-source
|
||||
forms;
|
||||
- local-backend and endpoint-only override coverage;
|
||||
- fail-fast inspection of missing, malformed, or unusable selected profiles;
|
||||
- offline tests for selection, source precedence, effective model inspection,
|
||||
batch reuse, and durable provenance;
|
||||
- maintained operator examples for overriding `weather-light` locally; and
|
||||
- updates to the canonical configuration, Promptkit integration, report
|
||||
registry, operations, troubleshooting, internal adapter, and release
|
||||
documentation as applicable when implementation lands.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
The feature does not include:
|
||||
|
||||
- automatic discovery, health checking, or benchmarking of local endpoints;
|
||||
- implicit failover between local and remote profiles;
|
||||
- retries with a more expensive tier after provider or validation failure;
|
||||
- per-report profile configuration fields outside prompt defaults;
|
||||
- profile inheritance, aliases, or field-level merging;
|
||||
- runtime model selection based on weather severity, token count, or report
|
||||
content;
|
||||
- moving Weatherreporter profile policy into Promptkit's built-in catalog;
|
||||
- exposing Promptkit types outside the adapter boundary; or
|
||||
- making live provider calls part of the default repository test suite.
|
||||
|
||||
## Compatibility And Operational Policy
|
||||
|
||||
Existing configurations with a nonblank `promptkit.profile` retain their
|
||||
all-report behavior. Existing `profile_file`, `profile_dir`, local-backend, and
|
||||
credential configuration fields retain their meanings.
|
||||
|
||||
Configurations that rely on the omitted profile setting will intentionally
|
||||
observe new per-report defaults. This is a user-visible model-selection and
|
||||
cost change and must be called out in release notes. Operators who require the
|
||||
old all-report model can preserve it by setting an explicit global profile.
|
||||
|
||||
An external same-ID override is an operator-owned compatibility commitment.
|
||||
Weatherreporter may evolve its embedded definitions, but it must not rewrite or
|
||||
silently merge an operator file.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The roadmap is complete when:
|
||||
|
||||
- a tagged Promptkit dependency supports the required fallback layer;
|
||||
- every operational prompt selects its assigned logical profile at exact
|
||||
version `1.1.0`;
|
||||
- all three embedded profiles inspect successfully without an external profile
|
||||
source;
|
||||
- configured same-ID definitions override embedded definitions through both
|
||||
`profile_file` and `profile_dir`;
|
||||
- an invalid matching external definition fails without fallback;
|
||||
- `weather-light` can resolve through an endpoint-only or configured-local
|
||||
override without requiring code or prompt changes;
|
||||
- global `promptkit.profile` still overrides every report in an invocation;
|
||||
- prompt preparation and execution artifacts record the selected logical
|
||||
profile and effective model;
|
||||
- morning and evening batch preflight deduplicates inspection of shared
|
||||
effective profile IDs as it does today;
|
||||
- the default test suite remains offline and deterministic; and
|
||||
- implemented behavior is documented by its canonical current-state owners.
|
||||
|
||||
## Open Questions
|
||||
|
||||
### Exact initial model identifiers and settings
|
||||
|
||||
Recommendation: evaluate the current Gemma 4 31B instruction target for
|
||||
`weather-light`, the maintained Gemini Flash target for `weather-balanced`, and
|
||||
the maintained Gemini Pro target for `weather-deep`. Pin the exact profile
|
||||
contents in this roadmap before deriving the staged implementation plan.
|
||||
|
||||
Alternative: duplicate three current Promptkit built-in definitions without a
|
||||
Weatherreporter-specific evaluation. This is faster, but it provides less
|
||||
evidence that their cost, schema reliability, and weather reasoning match the
|
||||
tier policy.
|
||||
|
||||
### Future default use of `weather-deep`
|
||||
|
||||
Recommendation: ship the profile as an explicit override with no default
|
||||
report assignment. Assign it later only when a report or evaluation shows a
|
||||
material benefit over `weather-balanced`.
|
||||
|
||||
Alternative: assign Daily or Tomorrow to `weather-deep` immediately. This may
|
||||
improve difficult synthesis, but it increases routine cost before the benefit
|
||||
has been measured.
|
||||
164
docs/roadmap/promptkit-fallback-profiles-feature-request.md
Normal file
164
docs/roadmap/promptkit-fallback-profiles-feature-request.md
Normal file
@@ -0,0 +1,164 @@
|
||||
# Promptkit Feature Request: Application Fallback Profiles
|
||||
|
||||
Status: Proposed upstream capability.
|
||||
|
||||
## Purpose
|
||||
|
||||
Promptkit should allow a consuming application to supply an embedded fallback
|
||||
profile source that sits below operator-configured profiles and above
|
||||
Promptkit's own built-in profile catalog.
|
||||
|
||||
This capability would let an application publish stable, domain-specific
|
||||
profile IDs with useful defaults while preserving Promptkit's existing
|
||||
operator-override behavior. The capability must remain application-neutral;
|
||||
Promptkit should provide the source layer but should not own downstream profile
|
||||
names, model assignments, or configuration policy.
|
||||
|
||||
## Downstream Use Case
|
||||
|
||||
Weatherreporter wants to embed profiles such as `weather-light`,
|
||||
`weather-balanced`, and `weather-deep`. Report prompts would select those
|
||||
logical profiles instead of naming provider- or model-specific Promptkit
|
||||
profiles directly.
|
||||
|
||||
An installation could then place a profile with the same ID in its configured
|
||||
profile directory. For example, a local `weather-light` definition could point
|
||||
to an OpenAI-compatible endpoint on the deployment network. When no operator
|
||||
definition exists, Weatherreporter's embedded definition would keep the
|
||||
application usable without additional profile files.
|
||||
|
||||
This pattern is useful beyond Weatherreporter. Any Promptkit consumer may want
|
||||
application-owned execution tiers or workload-specific defaults without
|
||||
adding domain-specific profiles to Promptkit's general built-in catalog.
|
||||
|
||||
## Current Constraint
|
||||
|
||||
Promptkit currently resolves matching profile IDs in this order:
|
||||
|
||||
1. in-memory profiles supplied through `WithProfiles`;
|
||||
2. one configured profile file, `fs.FS`, or directory source; and
|
||||
3. Promptkit's embedded built-in profiles.
|
||||
|
||||
These layers do not express the desired application-default relationship:
|
||||
|
||||
- `WithProfiles` has higher precedence than the configured source, so it would
|
||||
prevent an operator file from overriding an application profile with the
|
||||
same ID.
|
||||
- `WithProfileFS` can hold embedded application assets, but it occupies the
|
||||
configured-source layer and therefore replaces rather than sits beneath a
|
||||
configured profile directory or file.
|
||||
- adding downstream profile IDs to Promptkit's built-in catalog would make the
|
||||
library own application-specific policy.
|
||||
|
||||
A downstream application could build its own filesystem overlay, but that
|
||||
would duplicate Promptkit's profile discovery, error, and precedence behavior
|
||||
at the consumer boundary.
|
||||
|
||||
## Requested Capability
|
||||
|
||||
Add one optional application fallback profile source to engine construction.
|
||||
When present, matching profile IDs should resolve in this order:
|
||||
|
||||
1. in-memory profiles supplied through `WithProfiles`;
|
||||
2. the ordinary configured profile source selected through a profile option or
|
||||
`Config.ProfileDir`;
|
||||
3. the application fallback profile source; and
|
||||
4. Promptkit's embedded built-in profiles.
|
||||
|
||||
When no application fallback is configured, existing source precedence and
|
||||
behavior must remain unchanged.
|
||||
|
||||
The minimum useful public surface is an `fs.FS`-backed option because consumers
|
||||
can embed YAML profile assets. A possible API shape is:
|
||||
|
||||
```go
|
||||
promptkit.WithFallbackProfileFS(profileFS, ".")
|
||||
```
|
||||
|
||||
The name is illustrative rather than prescriptive. A companion option for
|
||||
validated `Profile` values could be added if Promptkit maintainers find it
|
||||
generally useful, but it is not required for the Weatherreporter use case.
|
||||
|
||||
## Required Semantics
|
||||
|
||||
- A higher-precedence source falls through only when the requested profile ID
|
||||
is absent.
|
||||
- A malformed, unreadable, duplicate, ambiguous, or otherwise invalid matching
|
||||
profile is an error and must not silently fall through.
|
||||
- The fallback source uses the existing strict profile YAML format and profile
|
||||
validation rules.
|
||||
- Profile values are selected as a whole. This feature does not merge,
|
||||
inherit, or partially overlay profile definitions.
|
||||
- `InspectProfile`, `Prepare`, prepared execution, and ordinary execution use
|
||||
the same profile-source precedence.
|
||||
- An explicit request profile continues to take precedence over a prompt's
|
||||
`default_profile`; this request concerns definition lookup after the profile
|
||||
ID has been selected.
|
||||
- Repeated fallback-source options should follow Promptkit's documented
|
||||
same-category option convention, normally with the last value replacing the
|
||||
earlier value.
|
||||
- A canceled lookup, invalid fallback asset, or unknown resolved backend should
|
||||
continue to cross the public facade through Promptkit's existing public error
|
||||
identities.
|
||||
- Exact profile inspection must remain side-effect free and must not contact a
|
||||
model provider.
|
||||
|
||||
## Application And Library Boundaries
|
||||
|
||||
Promptkit should own:
|
||||
|
||||
- the additional repository layer;
|
||||
- deterministic lookup and fallthrough behavior;
|
||||
- validation of the supplied source through the existing profile contract;
|
||||
- consistent use of the layer across inspection and execution; and
|
||||
- public documentation and tests for the added precedence rule.
|
||||
|
||||
The consuming application should continue to own:
|
||||
|
||||
- whether it supplies fallback profiles;
|
||||
- the profile IDs and their domain meaning;
|
||||
- embedded profile contents and model choices;
|
||||
- application configuration and override policy;
|
||||
- report- or workload-to-profile assignment; and
|
||||
- credential checks and operator-facing errors beyond Promptkit's public
|
||||
contract.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This request does not ask Promptkit to add:
|
||||
|
||||
- Weatherreporter-specific profile IDs to its built-in catalog;
|
||||
- profile inheritance, aliases, or field-level merging;
|
||||
- automatic endpoint discovery or availability probing;
|
||||
- provider failover or fallback from a failed selected profile;
|
||||
- per-request model benchmarking or tier selection;
|
||||
- application configuration discovery; or
|
||||
- eager validation of every profile in every source.
|
||||
|
||||
## Compatibility
|
||||
|
||||
The feature can be additive. Engines that do not configure an application
|
||||
fallback source should retain their current public behavior and precedence.
|
||||
Existing uses of `WithProfiles`, `WithProfileFile`, `WithProfileFS`, and
|
||||
`Config.ProfileDir` should not change meaning.
|
||||
|
||||
The application fallback is deliberately lower precedence than every existing
|
||||
consumer-configured source. This preserves the established expectation that a
|
||||
custom profile definition can override a packaged default with the same ID.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
The capability is sufficient for downstream adoption when Promptkit can
|
||||
demonstrate that:
|
||||
|
||||
- a fallback-only profile can be inspected and used for preparation and
|
||||
execution;
|
||||
- a configured directory, file, or `fs.FS` profile with the same ID overrides
|
||||
the fallback profile;
|
||||
- an absent configured profile falls through to the application fallback;
|
||||
- an invalid configured match fails instead of falling through;
|
||||
- an absent application fallback profile continues to resolve from Promptkit's
|
||||
built-in catalog;
|
||||
- `WithProfiles` retains highest precedence;
|
||||
- behavior is identical across inspection, preparation, and execution; and
|
||||
- omitting the new option preserves existing tests and public contracts.
|
||||
Reference in New Issue
Block a user