Plan domain profiles and ephemeral state
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Domain-Specific Prompt Profiles Roadmap
|
||||
|
||||
Status: Accepted feature direction; implementation has not started.
|
||||
Status: Accepted; implementation plan ready.
|
||||
|
||||
## Purpose
|
||||
|
||||
@@ -10,9 +10,9 @@ These logical profiles should give each report an appropriate default while
|
||||
allowing operators to replace any definition through the existing configured
|
||||
profile source.
|
||||
|
||||
This roadmap defines the scope, policy, and intended end state. A later staged
|
||||
implementation plan will translate it into implementation work. This document
|
||||
does not prescribe implementation stages.
|
||||
This roadmap defines the scope, policy, and intended end state. The companion
|
||||
[implementation plan](domain-profiles-implementation.md) owns the ordered work
|
||||
needed to reach that state.
|
||||
|
||||
## User Intent
|
||||
|
||||
@@ -42,20 +42,19 @@ profile can override a Promptkit built-in profile, and the configured local
|
||||
backend can support profiles that select `backend: local`. Endpoint-only
|
||||
OpenAI-compatible profiles can also provide their own endpoint.
|
||||
|
||||
Weatherreporter does not currently own or embed execution profiles. Promptkit's
|
||||
current public source precedence also has no layer where application-owned
|
||||
fallback profiles can remain overridable by the ordinary configured source.
|
||||
Weatherreporter does not currently own or embed execution profiles. Promptkit
|
||||
v0.5.0 now provides the fallback-profile layer needed to add them without
|
||||
changing the existing operator-source precedence.
|
||||
|
||||
## Prerequisite
|
||||
|
||||
Promptkit must first provide the application fallback profile capability
|
||||
defined in the companion
|
||||
[upstream feature request](promptkit-fallback-profiles-feature-request.md).
|
||||
|
||||
Weatherreporter should adopt a tagged Promptkit release containing that public
|
||||
capability. It should not depend on Promptkit internal packages, reproduce
|
||||
Promptkit's profile repository behavior, or commit a local module replacement
|
||||
as the production dependency.
|
||||
Promptkit v0.5.0 provides the application fallback profile capability defined
|
||||
in the companion
|
||||
[upstream feature request](promptkit-fallback-profiles-feature-request.md), and
|
||||
Weatherreporter now depends on that tagged release. The dependency upgrade has
|
||||
passed the repository test suite and an operator smoke test. Weatherreporter
|
||||
must continue to use only Promptkit's public API rather than depending on its
|
||||
internal packages or reproducing its profile repository behavior.
|
||||
|
||||
## Desired End State
|
||||
|
||||
@@ -148,20 +147,21 @@ selected model and supported by its backend. Avoid incidental generation
|
||||
parameters that reduce portability or trigger provider-specific request
|
||||
failures without a demonstrated quality benefit.
|
||||
|
||||
The initial model candidates are:
|
||||
The initial profile definitions are:
|
||||
|
||||
- `weather-light`: the maintained OpenRouter identifier for the Gemma 4 31B
|
||||
instruction model;
|
||||
- `weather-balanced`: Promptkit's maintained Gemini Flash family target; and
|
||||
- `weather-deep`: Promptkit's maintained Gemini Pro family target.
|
||||
| Profile ID | OpenRouter model | Reasoning effort | Timeout | Service tier |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` |
|
||||
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
||||
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
||||
|
||||
The exact identifiers and generation settings must be finalized against the
|
||||
available tagged Promptkit catalog and a representative Weatherreporter
|
||||
evaluation corpus before the staged implementation plan is approved. The
|
||||
logical profile IDs and report assignments do not depend on those concrete
|
||||
choices.
|
||||
These settings deliberately match the corresponding Promptkit v0.5.0
|
||||
built-ins while exposing Weatherreporter-owned logical IDs. The `~` prefix is
|
||||
part of each OpenRouter rolling-alias identifier. The profiles do not set
|
||||
temperature, `top_p`, or output-token limits; omission preserves provider
|
||||
defaults and avoids unsupported incidental parameters.
|
||||
|
||||
## Prompt And Provenance Contract
|
||||
## Prompt And Active Execution Contract
|
||||
|
||||
Changing a prompt's `default_profile` is a material prompt-definition change.
|
||||
The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new
|
||||
@@ -172,10 +172,18 @@ Prompt inspection must continue to occur before weather collection. It should
|
||||
report the selected logical profile ID and the resolved backend and model
|
||||
without exposing endpoints or credentials.
|
||||
|
||||
Preparation, execution, and failure artifacts should retain both the logical
|
||||
profile identity and effective model provenance already available through the
|
||||
project-owned execution contract. This feature should not replace a logical
|
||||
profile ID with only the concrete model name in durable records.
|
||||
The active execution contract should retain both the selected logical profile
|
||||
identity and the resolved backend and model through inspection, preparation,
|
||||
execution, errors, and command results where those values are already exposed.
|
||||
This feature must not add a new durable-provenance or cross-version artifact
|
||||
contract.
|
||||
|
||||
The accepted [ephemeral-state roadmap](ephemeral-state.md) makes historical
|
||||
prompt provenance a non-goal. Existing workspace persistence may remain while
|
||||
this feature lands, but it is transitional behavior and must not be expanded or
|
||||
treated as part of the profile feature's desired end state. Prompt preparation
|
||||
and execution artifacts written at `1.0.1` are not required to remain readable
|
||||
after the prompt definitions advance to `1.1.0`.
|
||||
|
||||
## Evaluation Policy
|
||||
|
||||
@@ -211,7 +219,7 @@ The completed feature includes:
|
||||
- local-backend and endpoint-only override coverage;
|
||||
- fail-fast inspection of missing, malformed, or unusable selected profiles;
|
||||
- offline tests for selection, source precedence, effective model inspection,
|
||||
batch reuse, and durable provenance;
|
||||
batch reuse, and active execution behavior;
|
||||
- maintained operator examples for overriding `weather-light` locally; and
|
||||
- updates to the canonical configuration, Promptkit integration, report
|
||||
registry, operations, troubleshooting, internal adapter, and release
|
||||
@@ -243,6 +251,11 @@ observe new per-report defaults. This is a user-visible model-selection and
|
||||
cost change and must be called out in release notes. Operators who require the
|
||||
old all-report model can preserve it by setting an explicit global profile.
|
||||
|
||||
The prompt-version transition does not provide backward compatibility for
|
||||
historical prompt preparation or execution artifacts. This is consistent with
|
||||
the accepted ephemeral-state direction; the profile feature does not otherwise
|
||||
redesign or remove the current workspace layout.
|
||||
|
||||
An external same-ID override is an operator-owned compatibility commitment.
|
||||
Weatherreporter may evolve its embedded definitions, but it must not rewrite or
|
||||
silently merge an operator file.
|
||||
@@ -262,33 +275,9 @@ The roadmap is complete when:
|
||||
- `weather-light` can resolve through an endpoint-only or configured-local
|
||||
override without requiring code or prompt changes;
|
||||
- global `promptkit.profile` still overrides every report in an invocation;
|
||||
- prompt preparation and execution artifacts record the selected logical
|
||||
profile and effective model;
|
||||
- active inspection and execution preserve the selected logical profile and
|
||||
effective model through the project-owned execution contract;
|
||||
- morning and evening batch preflight deduplicates inspection of shared
|
||||
effective profile IDs as it does today;
|
||||
- the default test suite remains offline and deterministic; and
|
||||
- implemented behavior is documented by its canonical current-state owners.
|
||||
|
||||
## Open Questions
|
||||
|
||||
### Exact initial model identifiers and settings
|
||||
|
||||
Recommendation: evaluate the current Gemma 4 31B instruction target for
|
||||
`weather-light`, the maintained Gemini Flash target for `weather-balanced`, and
|
||||
the maintained Gemini Pro target for `weather-deep`. Pin the exact profile
|
||||
contents in this roadmap before deriving the staged implementation plan.
|
||||
|
||||
Alternative: duplicate three current Promptkit built-in definitions without a
|
||||
Weatherreporter-specific evaluation. This is faster, but it provides less
|
||||
evidence that their cost, schema reliability, and weather reasoning match the
|
||||
tier policy.
|
||||
|
||||
### Future default use of `weather-deep`
|
||||
|
||||
Recommendation: ship the profile as an explicit override with no default
|
||||
report assignment. Assign it later only when a report or evaluation shows a
|
||||
material benefit over `weather-balanced`.
|
||||
|
||||
Alternative: assign Daily or Tomorrow to `weather-deep` immediately. This may
|
||||
improve difficult synthesis, but it increases routine cost before the benefit
|
||||
has been measured.
|
||||
|
||||
Reference in New Issue
Block a user