Plan domain profiles and ephemeral state
This commit is contained in:
373
docs/roadmap/domain-profiles-implementation.md
Normal file
373
docs/roadmap/domain-profiles-implementation.md
Normal file
@@ -0,0 +1,373 @@
|
||||
# Domain-Specific Prompt Profiles Implementation Plan
|
||||
|
||||
Status: Ready for implementation.
|
||||
|
||||
## Purpose And Authority
|
||||
|
||||
This document gives a coding agent the ordered work needed to implement the
|
||||
[domain-specific prompt profiles roadmap](domain-profiles.md). The roadmap is
|
||||
authoritative for scope, user intent, policy choices, and the desired end
|
||||
state. This plan is authoritative for implementation sequence, verification,
|
||||
and stage exit gates.
|
||||
|
||||
The target implementer is a `gpt-5.6-terra` coding agent using high reasoning.
|
||||
Complete the stages in order. Each stage is scoped for one implementation
|
||||
prompt and must leave the repository coherent, tested, and reviewable before
|
||||
the next stage begins.
|
||||
|
||||
This plan follows the repository's
|
||||
[architecture](../policy/architecture.md),
|
||||
[documentation](../policy/documentation.md), and
|
||||
[testing](../policy/testing.md) policies.
|
||||
|
||||
## Completed Prerequisite
|
||||
|
||||
Weatherreporter is already pinned to Promptkit v0.5.0. That release provides
|
||||
the public `WithFallbackProfileFS` option and the required precedence across
|
||||
inspection, preparation, and execution. The dependency upgrade passed
|
||||
Weatherreporter's full offline test suite, race-enabled suite, CLI help check,
|
||||
and an operator `generate hourly` smoke test. Do not repeat or replace the
|
||||
dependency upgrade as part of these stages.
|
||||
|
||||
## Locked Product Decisions
|
||||
|
||||
Implement these exact Weatherreporter-owned profiles:
|
||||
|
||||
| Profile ID | Backend | Model | Reasoning effort | Timeout | Service tier |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `weather-light` | `openrouter` | `deepseek/deepseek-v4-flash` | Omitted | 180 seconds | `flex` |
|
||||
| `weather-balanced` | `openrouter` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
||||
| `weather-deep` | `openrouter` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
||||
|
||||
Assign Hourly to `weather-light`; assign Daily, Today, and Tomorrow to
|
||||
`weather-balanced`; assign no report to `weather-deep` initially. Advance all
|
||||
four prompt definitions and matching report-registry entries from `1.0.1` to
|
||||
`1.1.0` when their defaults change.
|
||||
|
||||
The leading `~` in the Gemini and Claude model IDs is required and denotes an
|
||||
OpenRouter rolling alias. Do not substitute the unavailable non-tilde IDs or a
|
||||
dated model version. Do not add temperature, `top_p`, maximum-token, endpoint,
|
||||
or credential fields to the embedded definitions.
|
||||
|
||||
Prompt preparation and execution artifacts written at `1.0.1` are not required
|
||||
to remain readable after the transition to `1.1.0`. Do not add a migration,
|
||||
compatibility shim, or weaker historical-artifact validation for this feature.
|
||||
|
||||
Definition lookup must remain:
|
||||
|
||||
1. explicit Promptkit in-memory profiles used by tests or an embedding
|
||||
consumer;
|
||||
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
|
||||
3. Weatherreporter's embedded fallback profiles; and
|
||||
4. Promptkit's built-in catalog.
|
||||
|
||||
Selection remains a separate concern: a nonblank global `promptkit.profile`
|
||||
selects the profile for every report in the invocation; otherwise the exact
|
||||
prompt definition's `default_profile` selects it. A malformed matching
|
||||
higher-precedence profile is an error and never falls through.
|
||||
|
||||
## Continuing Invariants
|
||||
|
||||
- Keep all Promptkit types and mechanics inside
|
||||
`internal/adapters/promptkit`, its focused tests, and asset contract tests.
|
||||
- Keep prompt inspection before weather collection and provider work.
|
||||
- Keep one Promptkit engine per command action and one shared engine across a
|
||||
sequential batch.
|
||||
- Preserve logical profile ID and effective backend/model information through
|
||||
active inspection and execution where the project-owned contract already
|
||||
exposes it. Do not add new durable-provenance fields or compatibility
|
||||
guarantees.
|
||||
- Leave existing workspace persistence behavior otherwise unchanged. The
|
||||
accepted [ephemeral-state roadmap](ephemeral-state.md) owns its future
|
||||
removal and must not be partially implemented here.
|
||||
- Do not expose endpoints, credentials, rendered messages, schemas, request
|
||||
bodies, response bodies, or complete parameter maps through ordinary errors,
|
||||
logs, summaries, or state.
|
||||
- Keep the default suite deterministic, offline, and credential-free.
|
||||
- Do not add endpoint discovery, health probing, provider failover, retries at
|
||||
a more expensive tier, profile merging, per-report configuration fields, or
|
||||
severity-driven model selection.
|
||||
- Update canonical current-state documentation only in the stage where the
|
||||
corresponding behavior becomes implemented.
|
||||
- Run `git diff --check` before completing every stage.
|
||||
|
||||
## Stage 1: Add The Embedded Weather Profile Catalog
|
||||
|
||||
### Goal
|
||||
|
||||
Create one repository-owned, embedded profile source containing exactly the
|
||||
three locked logical profiles.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add strict YAML profile assets beneath `internal/promptassets` using the
|
||||
exact IDs and definitions in this plan.
|
||||
2. Extend `internal/promptassets` with a narrowly named accessor that returns
|
||||
the embedded profile `fs.FS`. Follow the existing prompt and schema asset
|
||||
pattern without exposing Promptkit types from the package.
|
||||
3. Keep profile filenames and embed layout simple and deterministic. Do not
|
||||
duplicate Promptkit's built-in directory taxonomy unless the application
|
||||
assets require it.
|
||||
4. Validate the assets through Promptkit's public engine/profile inspection
|
||||
surface rather than adding a second YAML parser or a Weatherreporter-owned
|
||||
profile representation.
|
||||
|
||||
### Tests
|
||||
|
||||
- Extend the asset contract tests to assert exactly the three logical IDs,
|
||||
their exact effective model IDs, and the intentional parameters.
|
||||
- Prove all three profiles inspect successfully offline when supplied as a
|
||||
fallback source and no operator source is present.
|
||||
- Assert that the catalog contains no endpoints, credentials, temperature,
|
||||
`top_p`, or maximum-token settings.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/promptassets
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
The embedded catalog is complete, strictly valid, safe, and independently
|
||||
inspectable through Promptkit v0.5.0's public API.
|
||||
|
||||
## Stage 2: Wire Fallback Resolution And Protect Precedence
|
||||
|
||||
### Goal
|
||||
|
||||
Supply the embedded catalog through Promptkit's application fallback layer
|
||||
without changing existing operator configuration or application boundaries.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add `promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), ".")` to
|
||||
normal adapter engine construction.
|
||||
2. Preserve existing `profile_file`, `profile_dir`, configured local backend,
|
||||
timeout, prompt filesystem, schema filesystem, and test-option behavior.
|
||||
3. Ensure ordinary production construction and the adapter's test
|
||||
construction path exercise the same fallback wiring. Test-only explicit
|
||||
profiles may retain Promptkit's documented highest precedence.
|
||||
4. Keep all fallback resolution in Promptkit. Do not add filesystem overlays,
|
||||
existence checks, YAML parsing, or merge behavior to Weatherreporter.
|
||||
|
||||
### Tests
|
||||
|
||||
- At the adapter boundary, prove fallback-only inspection of all three
|
||||
Weatherreporter profiles.
|
||||
- Prove same-ID overrides through both configured `profile_file` and
|
||||
`profile_dir`, including resolution of the override's effective backend and
|
||||
model.
|
||||
- Prove an absent operator match falls through, while a malformed matching
|
||||
operator definition fails without using the embedded profile.
|
||||
- Prove a selected Promptkit built-in that is absent from both higher layers
|
||||
still resolves.
|
||||
- Prove an explicit in-memory test profile retains highest precedence.
|
||||
- Cover both local override forms required by the roadmap: an endpoint-only
|
||||
OpenAI-compatible `weather-light` profile and a `backend: local` profile
|
||||
using the configured local endpoint. No test may contact either endpoint.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Inspection and prepared execution use Promptkit's exact four-layer precedence,
|
||||
operator errors remain visible, and local overrides require no prompt or code
|
||||
changes.
|
||||
|
||||
## Stage 3: Adopt Logical Defaults And Prompt Version 1.1.0
|
||||
|
||||
### Goal
|
||||
|
||||
Move operational prompts from provider-oriented defaults to the three-tier
|
||||
Weatherreporter policy with an exact, synchronized version transition.
|
||||
|
||||
### Work
|
||||
|
||||
1. Change Hourly's `default_profile` to `weather-light`.
|
||||
2. Change Daily, Today, and Tomorrow to `weather-balanced`.
|
||||
3. Advance the exact version in all four prompt YAML assets from `1.0.1` to
|
||||
`1.1.0` without changing prompt text or generated-text schemas solely for
|
||||
this feature.
|
||||
4. Advance the four matching report-registry prompt versions to `1.1.0` in the
|
||||
same change. Keep prompt IDs, report IDs, modules, periods, templates, and
|
||||
output contracts unchanged.
|
||||
5. Update fixtures and expectations that intentionally assert the current
|
||||
prompt contract. Do not rewrite historical fixture versions or weaken tests
|
||||
that protect actual compatibility.
|
||||
|
||||
### Tests
|
||||
|
||||
- Update asset and report-registry contract tests to require exact version
|
||||
`1.1.0` and the report-to-profile assignments locked in this plan.
|
||||
- Inspect every exact prompt version through the real embedded prompt, schema,
|
||||
and fallback-profile filesystems.
|
||||
- Prove Hourly resolves DeepSeek V4 Flash, the three day-scale reports resolve
|
||||
Gemini Flash Latest, and `weather-deep` remains inspectable but unassigned.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/promptassets ./internal/report ./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Every operational prompt and registry definition agrees on exact version
|
||||
`1.1.0`, selects its intended logical tier, and resolves its expected effective
|
||||
model offline.
|
||||
|
||||
## Stage 4: Verify Application Selection And Batch Reuse
|
||||
|
||||
### Goal
|
||||
|
||||
Protect the assembled application behavior created by the new defaults and
|
||||
confirm that logical identity is not lost during active effective-model
|
||||
resolution.
|
||||
|
||||
### Work
|
||||
|
||||
1. Preserve the current pre-collection inspection order and fail-fast behavior
|
||||
for missing credentials, unknown profiles, malformed profiles, and unusable
|
||||
backends.
|
||||
2. Preserve the global `promptkit.profile` all-report override. Do not add a
|
||||
second override mechanism or report-specific configuration fields.
|
||||
3. Preserve batch preflight deduplication by selected effective profile ID:
|
||||
Today and Tomorrow in the same batch should inspect their shared
|
||||
`weather-balanced` selection once.
|
||||
4. Preserve the selected logical profile ID and resolved backend/model through
|
||||
active inspection, preparation, and execution using the existing
|
||||
project-owned contract. Do not add state fields, expand persisted parameter
|
||||
detail, or create a new historical compatibility guarantee.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add or update representative app tests for default Hourly and day-scale
|
||||
selection, a global-profile override, and a morning/evening batch sharing
|
||||
`weather-balanced`.
|
||||
- Assert inspection completes before weather collection and provider
|
||||
generation, including malformed same-ID operator overrides.
|
||||
- Assert active inspection and execution expose the logical profile ID and
|
||||
effective model for both embedded and overridden profiles.
|
||||
- Assert endpoints and credentials remain absent from errors, summaries,
|
||||
normal logs, and ordinary state.
|
||||
- Use project-owned executor fakes or Promptkit provider fakes; do not make live
|
||||
provider calls.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/app ./internal/cli
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Single-report and batch workflows select the intended tier, retain existing
|
||||
override and preflight behavior, deduplicate shared batch inspection, and
|
||||
preserve safe logical and effective model information during active execution
|
||||
without adding a durable-provenance contract.
|
||||
|
||||
## Stage 5: Publish Canonical Operator And Maintainer Documentation
|
||||
|
||||
### Goal
|
||||
|
||||
Document the implemented feature once in each canonical owner and provide one
|
||||
maintained, copyable local override example.
|
||||
|
||||
### Work
|
||||
|
||||
1. Update `docs/config.md` to explain global profile selection versus
|
||||
`profile_file`/`profile_dir` definition lookup and link to the maintained
|
||||
example. Keep the field reference in this canonical document.
|
||||
2. Update the Promptkit integration document with the logical profile catalog,
|
||||
source precedence, exact prompt-version relationship, and safe active
|
||||
inspection and execution contract. Avoid restating complete configuration
|
||||
syntax or presenting transitional persistence as the target architecture.
|
||||
3. Update the report-registry, Promptkit adapter, app-orchestration, and state
|
||||
internal documents only where their implemented contracts changed.
|
||||
4. Update `docs/operations.md` with the normal local-override workflow and
|
||||
`docs/troubleshooting.md` with malformed override, unavailable local
|
||||
endpoint, missing credential, and unexpected effective-model diagnostics.
|
||||
5. Add or update one secret-free file under `examples/` showing a
|
||||
`weather-light` override for a local OpenAI-compatible endpoint. Choose one
|
||||
supported form as the complete example and mention the other form only in
|
||||
its canonical reference.
|
||||
6. Update the architecture policy only if implementation changed a normative
|
||||
boundary or invariant. Do not add future behavior to current-state docs.
|
||||
7. Keep the feature roadmap and this plan in their pre-implementation statuses
|
||||
until the final repository gate passes. Do not create release notes before
|
||||
a release version is chosen.
|
||||
|
||||
### Tests
|
||||
|
||||
- Verify every changed repository-relative link and every profile/model ID.
|
||||
- Validate maintained YAML examples through the same strict configuration or
|
||||
Promptkit profile path used by production where practical.
|
||||
- Run the focused tests that own any executable examples, followed by:
|
||||
|
||||
```sh
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Users, operators, and maintainers can discover the tier defaults, precedence,
|
||||
global override, local override, and failure behavior without duplicated or
|
||||
future-state documentation.
|
||||
|
||||
## Stage 6: Complete Repository Verification And Roadmap Handoff
|
||||
|
||||
### Goal
|
||||
|
||||
Demonstrate that the complete feature is coherent, offline-testable, and ready
|
||||
for review and a later release decision.
|
||||
|
||||
### Work
|
||||
|
||||
1. Review the complete diff against the roadmap, this plan, and all three
|
||||
policy documents. Remove stale identifiers, temporary helpers, redundant
|
||||
tests, and documentation duplication.
|
||||
2. Confirm `go.mod` and `go.sum` retain tagged Promptkit v0.5.0 without a local
|
||||
replacement or dependency drift.
|
||||
3. Confirm only the four supported report products exist and no retired report
|
||||
surfaces were reintroduced.
|
||||
4. Confirm the roadmap's completion criteria one by one. Change its status to
|
||||
implemented and this plan's status to completed only after every criterion
|
||||
and command below passes.
|
||||
5. Do not require a live provider for completion. If credentials and network
|
||||
access are deliberately supplied by an operator, record live smoke results
|
||||
separately as release-candidate evidence rather than adding them to the
|
||||
default suite.
|
||||
|
||||
### Verification
|
||||
|
||||
Run `gofmt -w` on every changed Go file, then run:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
Also inspect all three logical profiles through the application's normal
|
||||
preflight path using offline provider doubles, including one same-ID local
|
||||
override and one explicit global override.
|
||||
|
||||
### Exit Gate
|
||||
|
||||
All roadmap completion criteria are satisfied, all verification commands pass,
|
||||
the working tree contains only intentional changes, and the canonical
|
||||
documentation describes the implemented state. The feature is ready for code
|
||||
review and release preparation.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The model identifiers, profile settings, report assignments, version
|
||||
transition, precedence, compatibility behavior, test boundaries, and
|
||||
documentation ownership are decision-complete.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Domain-Specific Prompt Profiles Roadmap
|
||||
|
||||
Status: Accepted feature direction; implementation has not started.
|
||||
Status: Accepted; implementation plan ready.
|
||||
|
||||
## Purpose
|
||||
|
||||
@@ -10,9 +10,9 @@ These logical profiles should give each report an appropriate default while
|
||||
allowing operators to replace any definition through the existing configured
|
||||
profile source.
|
||||
|
||||
This roadmap defines the scope, policy, and intended end state. A later staged
|
||||
implementation plan will translate it into implementation work. This document
|
||||
does not prescribe implementation stages.
|
||||
This roadmap defines the scope, policy, and intended end state. The companion
|
||||
[implementation plan](domain-profiles-implementation.md) owns the ordered work
|
||||
needed to reach that state.
|
||||
|
||||
## User Intent
|
||||
|
||||
@@ -42,20 +42,19 @@ profile can override a Promptkit built-in profile, and the configured local
|
||||
backend can support profiles that select `backend: local`. Endpoint-only
|
||||
OpenAI-compatible profiles can also provide their own endpoint.
|
||||
|
||||
Weatherreporter does not currently own or embed execution profiles. Promptkit's
|
||||
current public source precedence also has no layer where application-owned
|
||||
fallback profiles can remain overridable by the ordinary configured source.
|
||||
Weatherreporter does not currently own or embed execution profiles. Promptkit
|
||||
v0.5.0 now provides the fallback-profile layer needed to add them without
|
||||
changing the existing operator-source precedence.
|
||||
|
||||
## Prerequisite
|
||||
|
||||
Promptkit must first provide the application fallback profile capability
|
||||
defined in the companion
|
||||
[upstream feature request](promptkit-fallback-profiles-feature-request.md).
|
||||
|
||||
Weatherreporter should adopt a tagged Promptkit release containing that public
|
||||
capability. It should not depend on Promptkit internal packages, reproduce
|
||||
Promptkit's profile repository behavior, or commit a local module replacement
|
||||
as the production dependency.
|
||||
Promptkit v0.5.0 provides the application fallback profile capability defined
|
||||
in the companion
|
||||
[upstream feature request](promptkit-fallback-profiles-feature-request.md), and
|
||||
Weatherreporter now depends on that tagged release. The dependency upgrade has
|
||||
passed the repository test suite and an operator smoke test. Weatherreporter
|
||||
must continue to use only Promptkit's public API rather than depending on its
|
||||
internal packages or reproducing its profile repository behavior.
|
||||
|
||||
## Desired End State
|
||||
|
||||
@@ -148,20 +147,21 @@ selected model and supported by its backend. Avoid incidental generation
|
||||
parameters that reduce portability or trigger provider-specific request
|
||||
failures without a demonstrated quality benefit.
|
||||
|
||||
The initial model candidates are:
|
||||
The initial profile definitions are:
|
||||
|
||||
- `weather-light`: the maintained OpenRouter identifier for the Gemma 4 31B
|
||||
instruction model;
|
||||
- `weather-balanced`: Promptkit's maintained Gemini Flash family target; and
|
||||
- `weather-deep`: Promptkit's maintained Gemini Pro family target.
|
||||
| Profile ID | OpenRouter model | Reasoning effort | Timeout | Service tier |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` |
|
||||
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
||||
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
||||
|
||||
The exact identifiers and generation settings must be finalized against the
|
||||
available tagged Promptkit catalog and a representative Weatherreporter
|
||||
evaluation corpus before the staged implementation plan is approved. The
|
||||
logical profile IDs and report assignments do not depend on those concrete
|
||||
choices.
|
||||
These settings deliberately match the corresponding Promptkit v0.5.0
|
||||
built-ins while exposing Weatherreporter-owned logical IDs. The `~` prefix is
|
||||
part of each OpenRouter rolling-alias identifier. The profiles do not set
|
||||
temperature, `top_p`, or output-token limits; omission preserves provider
|
||||
defaults and avoids unsupported incidental parameters.
|
||||
|
||||
## Prompt And Provenance Contract
|
||||
## Prompt And Active Execution Contract
|
||||
|
||||
Changing a prompt's `default_profile` is a material prompt-definition change.
|
||||
The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new
|
||||
@@ -172,10 +172,18 @@ Prompt inspection must continue to occur before weather collection. It should
|
||||
report the selected logical profile ID and the resolved backend and model
|
||||
without exposing endpoints or credentials.
|
||||
|
||||
Preparation, execution, and failure artifacts should retain both the logical
|
||||
profile identity and effective model provenance already available through the
|
||||
project-owned execution contract. This feature should not replace a logical
|
||||
profile ID with only the concrete model name in durable records.
|
||||
The active execution contract should retain both the selected logical profile
|
||||
identity and the resolved backend and model through inspection, preparation,
|
||||
execution, errors, and command results where those values are already exposed.
|
||||
This feature must not add a new durable-provenance or cross-version artifact
|
||||
contract.
|
||||
|
||||
The accepted [ephemeral-state roadmap](ephemeral-state.md) makes historical
|
||||
prompt provenance a non-goal. Existing workspace persistence may remain while
|
||||
this feature lands, but it is transitional behavior and must not be expanded or
|
||||
treated as part of the profile feature's desired end state. Prompt preparation
|
||||
and execution artifacts written at `1.0.1` are not required to remain readable
|
||||
after the prompt definitions advance to `1.1.0`.
|
||||
|
||||
## Evaluation Policy
|
||||
|
||||
@@ -211,7 +219,7 @@ The completed feature includes:
|
||||
- local-backend and endpoint-only override coverage;
|
||||
- fail-fast inspection of missing, malformed, or unusable selected profiles;
|
||||
- offline tests for selection, source precedence, effective model inspection,
|
||||
batch reuse, and durable provenance;
|
||||
batch reuse, and active execution behavior;
|
||||
- maintained operator examples for overriding `weather-light` locally; and
|
||||
- updates to the canonical configuration, Promptkit integration, report
|
||||
registry, operations, troubleshooting, internal adapter, and release
|
||||
@@ -243,6 +251,11 @@ observe new per-report defaults. This is a user-visible model-selection and
|
||||
cost change and must be called out in release notes. Operators who require the
|
||||
old all-report model can preserve it by setting an explicit global profile.
|
||||
|
||||
The prompt-version transition does not provide backward compatibility for
|
||||
historical prompt preparation or execution artifacts. This is consistent with
|
||||
the accepted ephemeral-state direction; the profile feature does not otherwise
|
||||
redesign or remove the current workspace layout.
|
||||
|
||||
An external same-ID override is an operator-owned compatibility commitment.
|
||||
Weatherreporter may evolve its embedded definitions, but it must not rewrite or
|
||||
silently merge an operator file.
|
||||
@@ -262,33 +275,9 @@ The roadmap is complete when:
|
||||
- `weather-light` can resolve through an endpoint-only or configured-local
|
||||
override without requiring code or prompt changes;
|
||||
- global `promptkit.profile` still overrides every report in an invocation;
|
||||
- prompt preparation and execution artifacts record the selected logical
|
||||
profile and effective model;
|
||||
- active inspection and execution preserve the selected logical profile and
|
||||
effective model through the project-owned execution contract;
|
||||
- morning and evening batch preflight deduplicates inspection of shared
|
||||
effective profile IDs as it does today;
|
||||
- the default test suite remains offline and deterministic; and
|
||||
- implemented behavior is documented by its canonical current-state owners.
|
||||
|
||||
## Open Questions
|
||||
|
||||
### Exact initial model identifiers and settings
|
||||
|
||||
Recommendation: evaluate the current Gemma 4 31B instruction target for
|
||||
`weather-light`, the maintained Gemini Flash target for `weather-balanced`, and
|
||||
the maintained Gemini Pro target for `weather-deep`. Pin the exact profile
|
||||
contents in this roadmap before deriving the staged implementation plan.
|
||||
|
||||
Alternative: duplicate three current Promptkit built-in definitions without a
|
||||
Weatherreporter-specific evaluation. This is faster, but it provides less
|
||||
evidence that their cost, schema reliability, and weather reasoning match the
|
||||
tier policy.
|
||||
|
||||
### Future default use of `weather-deep`
|
||||
|
||||
Recommendation: ship the profile as an explicit override with no default
|
||||
report assignment. Assign it later only when a report or evaluation shows a
|
||||
material benefit over `weather-balanced`.
|
||||
|
||||
Alternative: assign Daily or Tomorrow to `weather-deep` immediately. This may
|
||||
improve difficult synthesis, but it increases routine cost before the benefit
|
||||
has been measured.
|
||||
|
||||
382
docs/roadmap/ephemeral-state.md
Normal file
382
docs/roadmap/ephemeral-state.md
Normal file
@@ -0,0 +1,382 @@
|
||||
# Ephemeral Operational State Roadmap
|
||||
|
||||
Status: Accepted feature direction; implementation has not started.
|
||||
|
||||
## Purpose
|
||||
|
||||
Weatherreporter should treat generated weather reports and their intermediate
|
||||
artifacts as short-lived operational material rather than a durable audit
|
||||
history. Forecasts and current conditions change continuously, and the normal
|
||||
response to an old or failed report is to generate a new report, not to
|
||||
reconstruct the provenance of the old one.
|
||||
|
||||
The application should retain only the bounded state needed to publish the
|
||||
current report, calculate Recent Changes against the last successfully
|
||||
published report for the same valid period, and complete the current
|
||||
invocation safely. Detailed LLM diagnostics should remain an explicit,
|
||||
operator-controlled exception outside ordinary workspace state.
|
||||
|
||||
This roadmap defines the intended state lifecycle, compatibility policy, and
|
||||
architectural boundaries. A separate implementation plan will define the
|
||||
ordered work after the roadmap is complete.
|
||||
|
||||
## User Intent
|
||||
|
||||
The state model should reflect these product expectations:
|
||||
|
||||
- weather reports are ephemeral products, not business records;
|
||||
- old report provenance has no continuing operational value once conditions
|
||||
and forecasts have changed;
|
||||
- regenerating is preferable to recovering, replaying, or inspecting an old
|
||||
generation;
|
||||
- routine operation should not accumulate unbounded run-addressed artifacts;
|
||||
- Recent Changes remains useful, but needs only one prior successful snapshot
|
||||
for the same report and valid period; and
|
||||
- sensitive prompt and response capture remains opt-in and explicitly managed
|
||||
by the operator.
|
||||
|
||||
## Current State
|
||||
|
||||
Each generation currently writes a run-addressed collection containing a
|
||||
module snapshot, data package, prompt preparation receipt, prompt execution
|
||||
receipt, raw generated text, validated generated text, render context, managed
|
||||
report, metadata, and optional notification receipt. Successful and failed
|
||||
runs accumulate beneath the workspace.
|
||||
|
||||
Metadata links the collection and supports lookup by RunID. The CLI can list
|
||||
historical runs and inspect their metadata, modules, data packages, prior
|
||||
snapshots, and source provenance. New metadata uses the V2 format while the
|
||||
reader retains V1 compatibility. Prompt artifacts are validated against
|
||||
current report and prompt definitions when saved and loaded.
|
||||
|
||||
Most of this persistence exists for retrospective inspection and failure
|
||||
recovery. Dedicated prompt preparation and execution load operations have no
|
||||
ordinary production consumer. The important exception is module snapshot
|
||||
state: generation actively loads the most recent compatible snapshot to build
|
||||
the deterministic Recent Changes input for Daily, Today, and Tomorrow.
|
||||
|
||||
## Desired End State
|
||||
|
||||
Weatherreporter has three distinct state classes:
|
||||
|
||||
| State class | Lifecycle | Purpose |
|
||||
| --- | --- | --- |
|
||||
| Invocation workspace | Temporary and unpublished | Hold intermediate values while one report or batch is running. |
|
||||
| Current published state | Bounded and replaceable | Hold the current managed report and the minimal deterministic snapshot or manifest needed for normal operation. |
|
||||
| Secure LLM debug capture | Explicitly enabled and operator-managed | Diagnose prompt rendering or provider output when the operator deliberately requests sensitive capture. |
|
||||
|
||||
Ordinary generation uses an invocation-scoped temporary directory on the same
|
||||
filesystem as the managed workspace when atomic publication requires it.
|
||||
Prompt preparation, prompt execution, raw generated text, validated generated
|
||||
text, render contexts, data packages, and notification receipts may exist
|
||||
there while needed, but they are not published as durable historical
|
||||
artifacts.
|
||||
|
||||
A successful report atomically replaces the current published state for its
|
||||
logical report key and valid period. A failed attempt leaves the last
|
||||
successfully published report and comparison snapshot unchanged. Ordinary
|
||||
temporary artifacts are removed after both success and handled failure;
|
||||
cleanup failure is reported safely but must not replace the primary generation
|
||||
error.
|
||||
|
||||
RunIDs remain useful as in-process correlation identifiers in action results,
|
||||
logs, provider provenance, and optional debug paths. They no longer identify a
|
||||
durable collection that Weatherreporter promises to locate or decode later.
|
||||
|
||||
## Published Report Policy
|
||||
|
||||
The managed Markdown report remains the authoritative upload source during an
|
||||
invocation. The intended default is to retain only the current managed report
|
||||
for each logical report key and valid period, replacing it atomically after a
|
||||
new report has been fully rendered and validated.
|
||||
|
||||
An explicit `--out` or `--out-dir` copy remains operator-owned output outside
|
||||
the managed-state lifecycle. Weatherreporter does not delete, rotate, or
|
||||
rewrite those copies except when the same explicit destination is selected by
|
||||
a later invocation.
|
||||
|
||||
Distributor continues to receive only a completed managed Markdown report.
|
||||
Notification success or failure does not create a durable notification
|
||||
history. A notification failure leaves the newly published report available
|
||||
and returns a safe error through the current action result.
|
||||
|
||||
## Recent Changes State
|
||||
|
||||
Recent Changes must be preserved without preserving general report history.
|
||||
For Daily, Today, and Tomorrow, Weatherreporter retains at most one compatible
|
||||
module snapshot for each logical report key and valid period.
|
||||
|
||||
The retained snapshot represents the last successfully published report. A
|
||||
new invocation reads it before constructing Recent Changes and replaces it
|
||||
only when the new managed report has been successfully validated, rendered,
|
||||
and published. A failed generation therefore does not become the baseline for
|
||||
the next report and cannot hide changes that the user has not yet seen.
|
||||
|
||||
Hourly does not currently use the comparison strategy and should not retain a
|
||||
comparison snapshot solely for symmetry. State whose valid period has ended
|
||||
and can no longer participate in a supported comparison is eligible for safe
|
||||
cleanup.
|
||||
|
||||
## Temporary Workspace And Failure Semantics
|
||||
|
||||
Temporary state must remain beneath a narrowly owned application directory and
|
||||
use safe path construction, restrictive permissions where content is
|
||||
sensitive, and atomic writes where practical. Publication must not expose a
|
||||
partially rendered report or a snapshot that does not correspond to the
|
||||
published report.
|
||||
|
||||
Normal results retain bounded error information and paths only for artifacts
|
||||
that remain meaningful after the command: a previously or newly published
|
||||
report, an explicit operator output, or an enabled secure debug capture.
|
||||
Temporary intermediate paths are not emitted as if they were durable recovery
|
||||
locations. A failed command is retried by starting a new generation.
|
||||
|
||||
Process interruption may leave an uncommitted temporary directory. Such a
|
||||
directory is never considered published state, is never selected for Recent
|
||||
Changes, and may be removed by a documented safe cleanup mechanism. Cleanup
|
||||
must distinguish inactive temporary directories from concurrent active
|
||||
invocations and must never recursively target the workspace root or an
|
||||
unresolved configuration path.
|
||||
|
||||
## Inspection And Metadata Policy
|
||||
|
||||
Run-history discovery and inspection are not part of the desired product
|
||||
contract. The historical `inspect reports`, `inspect metadata`, `inspect
|
||||
modules`, `inspect data-package`, `inspect prior`, and `inspect sources`
|
||||
surfaces are candidates for removal together rather than preservation through
|
||||
a new storage representation.
|
||||
|
||||
Any manifest retained for atomic publication or Recent Changes is current
|
||||
operational state, not an archival metadata record. It should contain only the
|
||||
identity, valid period, safe paths, and deterministic snapshot information
|
||||
needed to validate and use that current state. It does not need to preserve
|
||||
prompt messages, generated prose intermediates, source provenance, provider
|
||||
provenance, notification history, or a catalog of prior runs.
|
||||
|
||||
The application does not promise cross-version decoding of ordinary workspace
|
||||
state. A new release may replace or ignore incompatible current-state files,
|
||||
provided it fails safely, never mistakes stale state for a compatible Recent
|
||||
Changes baseline, and documents any operator action required during upgrade.
|
||||
|
||||
## Prompt Execution And Debugging
|
||||
|
||||
Prompt inspection before weather collection and prepared execution remain
|
||||
runtime safety requirements. They do not require durable preparation or
|
||||
execution receipts.
|
||||
|
||||
The selected logical profile, effective backend and model, validation outcome,
|
||||
and safe classified error remain available to the active workflow and its CLI
|
||||
summary where useful. Weatherreporter does not retain them as long-term report
|
||||
provenance after the invocation completes.
|
||||
|
||||
The existing explicit secure debug root remains outside ordinary state and may
|
||||
retain rendered prompts, schemas, input bodies, generated bodies, and effective
|
||||
parameters according to its documented contract. Weatherreporter does not
|
||||
automatically clean that operator-selected location. Credentials must remain
|
||||
excluded from debug capture.
|
||||
|
||||
## Compatibility And Upgrade Policy
|
||||
|
||||
This is an intentional breaking change to the workspace and inspection
|
||||
contracts. Weatherreporter does not need to migrate historical V1 or V2
|
||||
metadata, prompt receipts, intermediate generated-text artifacts, or managed
|
||||
reports into the new representation.
|
||||
|
||||
Legacy workspace trees must not be silently interpreted as current published
|
||||
state. They also must not be deleted automatically merely because a new
|
||||
version starts: an operator may have placed or referenced files there despite
|
||||
the absence of a continuing application compatibility promise. Release notes
|
||||
and operations documentation must explain whether legacy data can be removed
|
||||
manually and identify the exact safe target.
|
||||
|
||||
The change should land in a release whose notes clearly identify removed CLI
|
||||
commands, obsolete paths and schemas, the new bounded state behavior, and any
|
||||
upgrade action. Because Weatherreporter remains pre-1.0, the ordinary semantic
|
||||
version policy may carry this breaking change without inventing a migration
|
||||
framework.
|
||||
|
||||
## Required Architecture Decision Record
|
||||
|
||||
The implemented feature must include an Accepted ADR recording the durable
|
||||
architectural decision to use ephemeral operational state. The ADR is not part
|
||||
of this roadmap-writing pass and should not be created until implementation is
|
||||
being prepared.
|
||||
|
||||
The ADR should record:
|
||||
|
||||
- the mismatch between run-addressed provenance storage and the ephemeral
|
||||
weather-report lifecycle;
|
||||
- the decision to retain bounded current report and comparison state rather
|
||||
than historical runs;
|
||||
- the distinction between temporary invocation state, published operational
|
||||
state, explicit output copies, and secure debug capture;
|
||||
- the removal of historical inspection and backward-compatibility guarantees;
|
||||
- atomic publication and failed-run behavior;
|
||||
- the alternatives considered, including retaining the current archive,
|
||||
adding time-based retention, or keeping a bounded run history; and
|
||||
- consequences for CLI compatibility, workspace layout, testing, operations,
|
||||
and future schema changes.
|
||||
|
||||
Once accepted, the ADR owns the decision rationale. The architecture policy
|
||||
owns the resulting current invariant, while focused state, CLI, operations,
|
||||
and integration documents own the implemented contracts.
|
||||
|
||||
## Scope
|
||||
|
||||
The completed feature includes:
|
||||
|
||||
- an invocation-scoped temporary workspace for intermediate generation state;
|
||||
- atomic publication of the current managed report and its minimal operational
|
||||
state;
|
||||
- a bounded comparison snapshot representing the last successfully published
|
||||
report for each supported report key and valid period;
|
||||
- safe cleanup behavior for normal completion, handled failure, and abandoned
|
||||
temporary workspaces;
|
||||
- removal of durable prompt preparation, prompt execution, generated-text,
|
||||
render-context, data-package, notification, and run-metadata history;
|
||||
- removal of run-history inspection commands and their application/state
|
||||
contracts;
|
||||
- removal of V1 metadata compatibility and current-version coupling for
|
||||
historical prompt artifacts by removing the historical artifact contract;
|
||||
- preservation of active-command partial status, safe errors, and paths to
|
||||
genuinely retained published, operator-owned, or debug outputs;
|
||||
- preservation of explicit output copies, Distributor upload behavior, and
|
||||
opt-in secure LLM debug capture;
|
||||
- risk-appropriate offline tests for atomic publication, comparison baselines,
|
||||
failure isolation, cleanup safety, concurrent invocation safety, and absence
|
||||
of unbounded state growth;
|
||||
- an Accepted ADR documenting the architectural decision; and
|
||||
- updates to canonical architecture, CLI, operations, configuration,
|
||||
troubleshooting, integration, internal, testing, and release documentation
|
||||
where their contracts change.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This feature does not include:
|
||||
|
||||
- a general-purpose cache, database, archival service, or retention engine;
|
||||
- replaying or resuming interrupted generation;
|
||||
- migrating legacy artifacts into the new representation;
|
||||
- retaining a bounded number of historical runs for convenience;
|
||||
- automatic upload or archival of state to remote storage;
|
||||
- collecting additional provider telemetry or weather-source provenance;
|
||||
- changing report content, prompt text, schemas, profile selection, weather
|
||||
derivation, or batch membership;
|
||||
- deleting operator-owned `--out`, `--out-dir`, or secure debug files;
|
||||
- changing Distributor's report-content contract; or
|
||||
- making live external services part of the default test suite.
|
||||
|
||||
## Safety And Testing Policy
|
||||
|
||||
The state refactoring must preserve Weatherreporter's existing path-safety and
|
||||
atomicity expectations while reducing the amount of durable state. Tests
|
||||
should emphasize observable lifecycle guarantees rather than private file
|
||||
choreography.
|
||||
|
||||
Important risks requiring durable offline coverage include:
|
||||
|
||||
- a failed or canceled generation replacing a previously published report or
|
||||
comparison baseline;
|
||||
- a partially written report becoming visible as current;
|
||||
- Recent Changes selecting an incompatible report, valid period, or failed
|
||||
attempt;
|
||||
- cleanup deleting published, operator-owned, debug, or concurrently active
|
||||
files;
|
||||
- batch partial success corrupting the state of another report;
|
||||
- notification failure rolling back or obscuring a successfully published
|
||||
report;
|
||||
- stale or incompatible current state being treated as valid; and
|
||||
- repeated successful and failed runs causing unbounded ordinary workspace
|
||||
growth.
|
||||
|
||||
Tests remain deterministic, offline, credential-free, and based on real
|
||||
temporary filesystems plus narrow external-boundary fakes. Race-enabled tests
|
||||
are required where publication, cleanup, or concurrent invocation behavior
|
||||
shares mutable filesystem state.
|
||||
|
||||
## Relationship To Domain-Specific Profiles
|
||||
|
||||
The domain-specific profile feature can be implemented before this refactor,
|
||||
but it should not add new historical compatibility or durable-provenance
|
||||
commitments. Profile inspection, selection, override precedence, and effective
|
||||
model resolution remain active-workflow behavior and survive the state change.
|
||||
|
||||
The domain-profile roadmap and implementation plan should acknowledge that
|
||||
prompt artifacts from version `1.0.1` need not remain readable after prompts
|
||||
advance to `1.1.0`. Existing state persistence may remain temporarily while
|
||||
the profile feature lands, but it should not be expanded or treated as the
|
||||
target architecture.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The roadmap's target state is achieved when:
|
||||
|
||||
- ordinary runs no longer create durable run-addressed artifact collections;
|
||||
- a successful report atomically replaces only the corresponding current
|
||||
published state;
|
||||
- failed and canceled attempts leave the prior published report and Recent
|
||||
Changes baseline unchanged;
|
||||
- Daily, Today, and Tomorrow compare against at most one compatible snapshot
|
||||
from the last successfully published report;
|
||||
- expired comparison and published state can be removed safely without
|
||||
touching operator-owned or active files;
|
||||
- Hourly does not retain an unused comparison snapshot;
|
||||
- historical inspection commands and V1/V2 archival compatibility code are
|
||||
removed;
|
||||
- temporary, published, explicit-output, and debug paths have distinct and
|
||||
documented ownership and cleanup rules;
|
||||
- Distributor and active-command summaries continue to receive the completed
|
||||
report and safe status information they require;
|
||||
- the default suite proves atomicity, bounded growth, cleanup safety, batch
|
||||
isolation, and comparison correctness offline;
|
||||
- an Accepted ADR records the architectural decision and alternatives; and
|
||||
- canonical current-state documentation describes only the implemented
|
||||
lifecycle.
|
||||
|
||||
## Open Questions
|
||||
|
||||
### Lifetime of the current managed report
|
||||
|
||||
Recommendation: retain one current managed report per logical report key and
|
||||
valid period until it is replaced or its valid period expires. This preserves
|
||||
the current default behavior for invocations without `--out` while bounding
|
||||
growth.
|
||||
|
||||
Alternative: treat the managed report as temporary and retain output only when
|
||||
the operator supplies `--out` or `--out-dir`. This minimizes state further but
|
||||
makes a successful default invocation produce no durable report for the user
|
||||
and complicates Distributor sequencing.
|
||||
|
||||
### Historical inspection replacement
|
||||
|
||||
Recommendation: remove the run-history inspection commands without adding a
|
||||
replacement initially. Current command summaries, current managed files, and
|
||||
opt-in debug capture cover the remaining supported workflows.
|
||||
|
||||
Alternative: add a narrow `inspect current REPORT` command backed only by the
|
||||
current operational manifest. This provides discoverability without history,
|
||||
but it creates a new public surface and may preserve metadata complexity that
|
||||
the refactor is intended to remove.
|
||||
|
||||
### Abandoned temporary workspace cleanup
|
||||
|
||||
Recommendation: use an explicitly owned temporary subtree with per-invocation
|
||||
ownership markers and a conservative age threshold. Normal cleanup removes the
|
||||
current invocation synchronously; opportunistic cleanup removes only marked,
|
||||
inactive directories old enough that they cannot reasonably belong to a live
|
||||
invocation.
|
||||
|
||||
Alternative: perform only synchronous cleanup and document manual removal of
|
||||
directories left by process termination. This minimizes destructive code and
|
||||
concurrency risk, but crashed processes can still accumulate unbounded files.
|
||||
|
||||
### Legacy workspace cleanup
|
||||
|
||||
Recommendation: ignore legacy run-addressed trees and document a precise,
|
||||
manual one-time cleanup procedure. Do not automatically delete them during
|
||||
startup or upgrade.
|
||||
|
||||
Alternative: add an explicit cleanup command that previews and then removes
|
||||
recognized legacy artifacts. This is more convenient for large installations
|
||||
but introduces a destructive command and a legacy-format classifier that must
|
||||
be maintained and tested.
|
||||
@@ -1,6 +1,11 @@
|
||||
# Promptkit Feature Request: Application Fallback Profiles
|
||||
|
||||
Status: Proposed upstream capability.
|
||||
Status: Implemented upstream in Promptkit v0.5.0.
|
||||
|
||||
Promptkit v0.5.0 resolved this request with the public
|
||||
`WithFallbackProfileFS` engine option and the precedence and error semantics
|
||||
specified below. This document is retained as the downstream rationale for
|
||||
the capability.
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
Reference in New Issue
Block a user