diff --git a/docs/roadmap/domain-profiles-implementation.md b/docs/roadmap/domain-profiles-implementation.md new file mode 100644 index 0000000..1380085 --- /dev/null +++ b/docs/roadmap/domain-profiles-implementation.md @@ -0,0 +1,373 @@ +# Domain-Specific Prompt Profiles Implementation Plan + +Status: Ready for implementation. + +## Purpose And Authority + +This document gives a coding agent the ordered work needed to implement the +[domain-specific prompt profiles roadmap](domain-profiles.md). The roadmap is +authoritative for scope, user intent, policy choices, and the desired end +state. This plan is authoritative for implementation sequence, verification, +and stage exit gates. + +The target implementer is a `gpt-5.6-terra` coding agent using high reasoning. +Complete the stages in order. Each stage is scoped for one implementation +prompt and must leave the repository coherent, tested, and reviewable before +the next stage begins. + +This plan follows the repository's +[architecture](../policy/architecture.md), +[documentation](../policy/documentation.md), and +[testing](../policy/testing.md) policies. + +## Completed Prerequisite + +Weatherreporter is already pinned to Promptkit v0.5.0. That release provides +the public `WithFallbackProfileFS` option and the required precedence across +inspection, preparation, and execution. The dependency upgrade passed +Weatherreporter's full offline test suite, race-enabled suite, CLI help check, +and an operator `generate hourly` smoke test. Do not repeat or replace the +dependency upgrade as part of these stages. + +## Locked Product Decisions + +Implement these exact Weatherreporter-owned profiles: + +| Profile ID | Backend | Model | Reasoning effort | Timeout | Service tier | +| --- | --- | --- | --- | --- | --- | +| `weather-light` | `openrouter` | `deepseek/deepseek-v4-flash` | Omitted | 180 seconds | `flex` | +| `weather-balanced` | `openrouter` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | +| `weather-deep` | `openrouter` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | + +Assign Hourly to `weather-light`; assign Daily, Today, and Tomorrow to +`weather-balanced`; assign no report to `weather-deep` initially. Advance all +four prompt definitions and matching report-registry entries from `1.0.1` to +`1.1.0` when their defaults change. + +The leading `~` in the Gemini and Claude model IDs is required and denotes an +OpenRouter rolling alias. Do not substitute the unavailable non-tilde IDs or a +dated model version. Do not add temperature, `top_p`, maximum-token, endpoint, +or credential fields to the embedded definitions. + +Prompt preparation and execution artifacts written at `1.0.1` are not required +to remain readable after the transition to `1.1.0`. Do not add a migration, +compatibility shim, or weaker historical-artifact validation for this feature. + +Definition lookup must remain: + +1. explicit Promptkit in-memory profiles used by tests or an embedding + consumer; +2. Weatherreporter's configured `profile_file` or `profile_dir` source; +3. Weatherreporter's embedded fallback profiles; and +4. Promptkit's built-in catalog. + +Selection remains a separate concern: a nonblank global `promptkit.profile` +selects the profile for every report in the invocation; otherwise the exact +prompt definition's `default_profile` selects it. A malformed matching +higher-precedence profile is an error and never falls through. + +## Continuing Invariants + +- Keep all Promptkit types and mechanics inside + `internal/adapters/promptkit`, its focused tests, and asset contract tests. +- Keep prompt inspection before weather collection and provider work. +- Keep one Promptkit engine per command action and one shared engine across a + sequential batch. +- Preserve logical profile ID and effective backend/model information through + active inspection and execution where the project-owned contract already + exposes it. Do not add new durable-provenance fields or compatibility + guarantees. +- Leave existing workspace persistence behavior otherwise unchanged. The + accepted [ephemeral-state roadmap](ephemeral-state.md) owns its future + removal and must not be partially implemented here. +- Do not expose endpoints, credentials, rendered messages, schemas, request + bodies, response bodies, or complete parameter maps through ordinary errors, + logs, summaries, or state. +- Keep the default suite deterministic, offline, and credential-free. +- Do not add endpoint discovery, health probing, provider failover, retries at + a more expensive tier, profile merging, per-report configuration fields, or + severity-driven model selection. +- Update canonical current-state documentation only in the stage where the + corresponding behavior becomes implemented. +- Run `git diff --check` before completing every stage. + +## Stage 1: Add The Embedded Weather Profile Catalog + +### Goal + +Create one repository-owned, embedded profile source containing exactly the +three locked logical profiles. + +### Work + +1. Add strict YAML profile assets beneath `internal/promptassets` using the + exact IDs and definitions in this plan. +2. Extend `internal/promptassets` with a narrowly named accessor that returns + the embedded profile `fs.FS`. Follow the existing prompt and schema asset + pattern without exposing Promptkit types from the package. +3. Keep profile filenames and embed layout simple and deterministic. Do not + duplicate Promptkit's built-in directory taxonomy unless the application + assets require it. +4. Validate the assets through Promptkit's public engine/profile inspection + surface rather than adding a second YAML parser or a Weatherreporter-owned + profile representation. + +### Tests + +- Extend the asset contract tests to assert exactly the three logical IDs, + their exact effective model IDs, and the intentional parameters. +- Prove all three profiles inspect successfully offline when supplied as a + fallback source and no operator source is present. +- Assert that the catalog contains no endpoints, credentials, temperature, + `top_p`, or maximum-token settings. +- Run: + + ```sh + go test ./internal/promptassets + git diff --check + ``` + +### Exit Gate + +The embedded catalog is complete, strictly valid, safe, and independently +inspectable through Promptkit v0.5.0's public API. + +## Stage 2: Wire Fallback Resolution And Protect Precedence + +### Goal + +Supply the embedded catalog through Promptkit's application fallback layer +without changing existing operator configuration or application boundaries. + +### Work + +1. Add `promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), ".")` to + normal adapter engine construction. +2. Preserve existing `profile_file`, `profile_dir`, configured local backend, + timeout, prompt filesystem, schema filesystem, and test-option behavior. +3. Ensure ordinary production construction and the adapter's test + construction path exercise the same fallback wiring. Test-only explicit + profiles may retain Promptkit's documented highest precedence. +4. Keep all fallback resolution in Promptkit. Do not add filesystem overlays, + existence checks, YAML parsing, or merge behavior to Weatherreporter. + +### Tests + +- At the adapter boundary, prove fallback-only inspection of all three + Weatherreporter profiles. +- Prove same-ID overrides through both configured `profile_file` and + `profile_dir`, including resolution of the override's effective backend and + model. +- Prove an absent operator match falls through, while a malformed matching + operator definition fails without using the embedded profile. +- Prove a selected Promptkit built-in that is absent from both higher layers + still resolves. +- Prove an explicit in-memory test profile retains highest precedence. +- Cover both local override forms required by the roadmap: an endpoint-only + OpenAI-compatible `weather-light` profile and a `backend: local` profile + using the configured local endpoint. No test may contact either endpoint. +- Run: + + ```sh + go test ./internal/adapters/promptkit + git diff --check + ``` + +### Exit Gate + +Inspection and prepared execution use Promptkit's exact four-layer precedence, +operator errors remain visible, and local overrides require no prompt or code +changes. + +## Stage 3: Adopt Logical Defaults And Prompt Version 1.1.0 + +### Goal + +Move operational prompts from provider-oriented defaults to the three-tier +Weatherreporter policy with an exact, synchronized version transition. + +### Work + +1. Change Hourly's `default_profile` to `weather-light`. +2. Change Daily, Today, and Tomorrow to `weather-balanced`. +3. Advance the exact version in all four prompt YAML assets from `1.0.1` to + `1.1.0` without changing prompt text or generated-text schemas solely for + this feature. +4. Advance the four matching report-registry prompt versions to `1.1.0` in the + same change. Keep prompt IDs, report IDs, modules, periods, templates, and + output contracts unchanged. +5. Update fixtures and expectations that intentionally assert the current + prompt contract. Do not rewrite historical fixture versions or weaken tests + that protect actual compatibility. + +### Tests + +- Update asset and report-registry contract tests to require exact version + `1.1.0` and the report-to-profile assignments locked in this plan. +- Inspect every exact prompt version through the real embedded prompt, schema, + and fallback-profile filesystems. +- Prove Hourly resolves DeepSeek V4 Flash, the three day-scale reports resolve + Gemini Flash Latest, and `weather-deep` remains inspectable but unassigned. +- Run: + + ```sh + go test ./internal/promptassets ./internal/report ./internal/adapters/promptkit + git diff --check + ``` + +### Exit Gate + +Every operational prompt and registry definition agrees on exact version +`1.1.0`, selects its intended logical tier, and resolves its expected effective +model offline. + +## Stage 4: Verify Application Selection And Batch Reuse + +### Goal + +Protect the assembled application behavior created by the new defaults and +confirm that logical identity is not lost during active effective-model +resolution. + +### Work + +1. Preserve the current pre-collection inspection order and fail-fast behavior + for missing credentials, unknown profiles, malformed profiles, and unusable + backends. +2. Preserve the global `promptkit.profile` all-report override. Do not add a + second override mechanism or report-specific configuration fields. +3. Preserve batch preflight deduplication by selected effective profile ID: + Today and Tomorrow in the same batch should inspect their shared + `weather-balanced` selection once. +4. Preserve the selected logical profile ID and resolved backend/model through + active inspection, preparation, and execution using the existing + project-owned contract. Do not add state fields, expand persisted parameter + detail, or create a new historical compatibility guarantee. + +### Tests + +- Add or update representative app tests for default Hourly and day-scale + selection, a global-profile override, and a morning/evening batch sharing + `weather-balanced`. +- Assert inspection completes before weather collection and provider + generation, including malformed same-ID operator overrides. +- Assert active inspection and execution expose the logical profile ID and + effective model for both embedded and overridden profiles. +- Assert endpoints and credentials remain absent from errors, summaries, + normal logs, and ordinary state. +- Use project-owned executor fakes or Promptkit provider fakes; do not make live + provider calls. +- Run: + + ```sh + go test ./internal/app ./internal/cli + git diff --check + ``` + +### Exit Gate + +Single-report and batch workflows select the intended tier, retain existing +override and preflight behavior, deduplicate shared batch inspection, and +preserve safe logical and effective model information during active execution +without adding a durable-provenance contract. + +## Stage 5: Publish Canonical Operator And Maintainer Documentation + +### Goal + +Document the implemented feature once in each canonical owner and provide one +maintained, copyable local override example. + +### Work + +1. Update `docs/config.md` to explain global profile selection versus + `profile_file`/`profile_dir` definition lookup and link to the maintained + example. Keep the field reference in this canonical document. +2. Update the Promptkit integration document with the logical profile catalog, + source precedence, exact prompt-version relationship, and safe active + inspection and execution contract. Avoid restating complete configuration + syntax or presenting transitional persistence as the target architecture. +3. Update the report-registry, Promptkit adapter, app-orchestration, and state + internal documents only where their implemented contracts changed. +4. Update `docs/operations.md` with the normal local-override workflow and + `docs/troubleshooting.md` with malformed override, unavailable local + endpoint, missing credential, and unexpected effective-model diagnostics. +5. Add or update one secret-free file under `examples/` showing a + `weather-light` override for a local OpenAI-compatible endpoint. Choose one + supported form as the complete example and mention the other form only in + its canonical reference. +6. Update the architecture policy only if implementation changed a normative + boundary or invariant. Do not add future behavior to current-state docs. +7. Keep the feature roadmap and this plan in their pre-implementation statuses + until the final repository gate passes. Do not create release notes before + a release version is chosen. + +### Tests + +- Verify every changed repository-relative link and every profile/model ID. +- Validate maintained YAML examples through the same strict configuration or + Promptkit profile path used by production where practical. +- Run the focused tests that own any executable examples, followed by: + + ```sh + git diff --check + ``` + +### Exit Gate + +Users, operators, and maintainers can discover the tier defaults, precedence, +global override, local override, and failure behavior without duplicated or +future-state documentation. + +## Stage 6: Complete Repository Verification And Roadmap Handoff + +### Goal + +Demonstrate that the complete feature is coherent, offline-testable, and ready +for review and a later release decision. + +### Work + +1. Review the complete diff against the roadmap, this plan, and all three + policy documents. Remove stale identifiers, temporary helpers, redundant + tests, and documentation duplication. +2. Confirm `go.mod` and `go.sum` retain tagged Promptkit v0.5.0 without a local + replacement or dependency drift. +3. Confirm only the four supported report products exist and no retired report + surfaces were reintroduced. +4. Confirm the roadmap's completion criteria one by one. Change its status to + implemented and this plan's status to completed only after every criterion + and command below passes. +5. Do not require a live provider for completion. If credentials and network + access are deliberately supplied by an operator, record live smoke results + separately as release-candidate evidence rather than adding them to the + default suite. + +### Verification + +Run `gofmt -w` on every changed Go file, then run: + +```sh +go test ./... +go test -race ./... +go run ./cmd/weatherreporter --help +git diff --check +git status --short +``` + +Also inspect all three logical profiles through the application's normal +preflight path using offline provider doubles, including one same-ID local +override and one explicit global override. + +### Exit Gate + +All roadmap completion criteria are satisfied, all verification commands pass, +the working tree contains only intentional changes, and the canonical +documentation describes the implemented state. The feature is ready for code +review and release preparation. + +## Open Questions + +None. The model identifiers, profile settings, report assignments, version +transition, precedence, compatibility behavior, test boundaries, and +documentation ownership are decision-complete. diff --git a/docs/roadmap/domain-profiles.md b/docs/roadmap/domain-profiles.md index c259da5..1c35ff1 100644 --- a/docs/roadmap/domain-profiles.md +++ b/docs/roadmap/domain-profiles.md @@ -1,6 +1,6 @@ # Domain-Specific Prompt Profiles Roadmap -Status: Accepted feature direction; implementation has not started. +Status: Accepted; implementation plan ready. ## Purpose @@ -10,9 +10,9 @@ These logical profiles should give each report an appropriate default while allowing operators to replace any definition through the existing configured profile source. -This roadmap defines the scope, policy, and intended end state. A later staged -implementation plan will translate it into implementation work. This document -does not prescribe implementation stages. +This roadmap defines the scope, policy, and intended end state. The companion +[implementation plan](domain-profiles-implementation.md) owns the ordered work +needed to reach that state. ## User Intent @@ -42,20 +42,19 @@ profile can override a Promptkit built-in profile, and the configured local backend can support profiles that select `backend: local`. Endpoint-only OpenAI-compatible profiles can also provide their own endpoint. -Weatherreporter does not currently own or embed execution profiles. Promptkit's -current public source precedence also has no layer where application-owned -fallback profiles can remain overridable by the ordinary configured source. +Weatherreporter does not currently own or embed execution profiles. Promptkit +v0.5.0 now provides the fallback-profile layer needed to add them without +changing the existing operator-source precedence. ## Prerequisite -Promptkit must first provide the application fallback profile capability -defined in the companion -[upstream feature request](promptkit-fallback-profiles-feature-request.md). - -Weatherreporter should adopt a tagged Promptkit release containing that public -capability. It should not depend on Promptkit internal packages, reproduce -Promptkit's profile repository behavior, or commit a local module replacement -as the production dependency. +Promptkit v0.5.0 provides the application fallback profile capability defined +in the companion +[upstream feature request](promptkit-fallback-profiles-feature-request.md), and +Weatherreporter now depends on that tagged release. The dependency upgrade has +passed the repository test suite and an operator smoke test. Weatherreporter +must continue to use only Promptkit's public API rather than depending on its +internal packages or reproducing its profile repository behavior. ## Desired End State @@ -148,20 +147,21 @@ selected model and supported by its backend. Avoid incidental generation parameters that reduce portability or trigger provider-specific request failures without a demonstrated quality benefit. -The initial model candidates are: +The initial profile definitions are: -- `weather-light`: the maintained OpenRouter identifier for the Gemma 4 31B - instruction model; -- `weather-balanced`: Promptkit's maintained Gemini Flash family target; and -- `weather-deep`: Promptkit's maintained Gemini Pro family target. +| Profile ID | OpenRouter model | Reasoning effort | Timeout | Service tier | +| --- | --- | --- | --- | --- | +| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` | +| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | +| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | -The exact identifiers and generation settings must be finalized against the -available tagged Promptkit catalog and a representative Weatherreporter -evaluation corpus before the staged implementation plan is approved. The -logical profile IDs and report assignments do not depend on those concrete -choices. +These settings deliberately match the corresponding Promptkit v0.5.0 +built-ins while exposing Weatherreporter-owned logical IDs. The `~` prefix is +part of each OpenRouter rolling-alias identifier. The profiles do not set +temperature, `top_p`, or output-token limits; omission preserves provider +defaults and avoids unsupported incidental parameters. -## Prompt And Provenance Contract +## Prompt And Active Execution Contract Changing a prompt's `default_profile` is a material prompt-definition change. The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new @@ -172,10 +172,18 @@ Prompt inspection must continue to occur before weather collection. It should report the selected logical profile ID and the resolved backend and model without exposing endpoints or credentials. -Preparation, execution, and failure artifacts should retain both the logical -profile identity and effective model provenance already available through the -project-owned execution contract. This feature should not replace a logical -profile ID with only the concrete model name in durable records. +The active execution contract should retain both the selected logical profile +identity and the resolved backend and model through inspection, preparation, +execution, errors, and command results where those values are already exposed. +This feature must not add a new durable-provenance or cross-version artifact +contract. + +The accepted [ephemeral-state roadmap](ephemeral-state.md) makes historical +prompt provenance a non-goal. Existing workspace persistence may remain while +this feature lands, but it is transitional behavior and must not be expanded or +treated as part of the profile feature's desired end state. Prompt preparation +and execution artifacts written at `1.0.1` are not required to remain readable +after the prompt definitions advance to `1.1.0`. ## Evaluation Policy @@ -211,7 +219,7 @@ The completed feature includes: - local-backend and endpoint-only override coverage; - fail-fast inspection of missing, malformed, or unusable selected profiles; - offline tests for selection, source precedence, effective model inspection, - batch reuse, and durable provenance; + batch reuse, and active execution behavior; - maintained operator examples for overriding `weather-light` locally; and - updates to the canonical configuration, Promptkit integration, report registry, operations, troubleshooting, internal adapter, and release @@ -243,6 +251,11 @@ observe new per-report defaults. This is a user-visible model-selection and cost change and must be called out in release notes. Operators who require the old all-report model can preserve it by setting an explicit global profile. +The prompt-version transition does not provide backward compatibility for +historical prompt preparation or execution artifacts. This is consistent with +the accepted ephemeral-state direction; the profile feature does not otherwise +redesign or remove the current workspace layout. + An external same-ID override is an operator-owned compatibility commitment. Weatherreporter may evolve its embedded definitions, but it must not rewrite or silently merge an operator file. @@ -262,33 +275,9 @@ The roadmap is complete when: - `weather-light` can resolve through an endpoint-only or configured-local override without requiring code or prompt changes; - global `promptkit.profile` still overrides every report in an invocation; -- prompt preparation and execution artifacts record the selected logical - profile and effective model; +- active inspection and execution preserve the selected logical profile and + effective model through the project-owned execution contract; - morning and evening batch preflight deduplicates inspection of shared effective profile IDs as it does today; - the default test suite remains offline and deterministic; and - implemented behavior is documented by its canonical current-state owners. - -## Open Questions - -### Exact initial model identifiers and settings - -Recommendation: evaluate the current Gemma 4 31B instruction target for -`weather-light`, the maintained Gemini Flash target for `weather-balanced`, and -the maintained Gemini Pro target for `weather-deep`. Pin the exact profile -contents in this roadmap before deriving the staged implementation plan. - -Alternative: duplicate three current Promptkit built-in definitions without a -Weatherreporter-specific evaluation. This is faster, but it provides less -evidence that their cost, schema reliability, and weather reasoning match the -tier policy. - -### Future default use of `weather-deep` - -Recommendation: ship the profile as an explicit override with no default -report assignment. Assign it later only when a report or evaluation shows a -material benefit over `weather-balanced`. - -Alternative: assign Daily or Tomorrow to `weather-deep` immediately. This may -improve difficult synthesis, but it increases routine cost before the benefit -has been measured. diff --git a/docs/roadmap/ephemeral-state.md b/docs/roadmap/ephemeral-state.md new file mode 100644 index 0000000..d3016a8 --- /dev/null +++ b/docs/roadmap/ephemeral-state.md @@ -0,0 +1,382 @@ +# Ephemeral Operational State Roadmap + +Status: Accepted feature direction; implementation has not started. + +## Purpose + +Weatherreporter should treat generated weather reports and their intermediate +artifacts as short-lived operational material rather than a durable audit +history. Forecasts and current conditions change continuously, and the normal +response to an old or failed report is to generate a new report, not to +reconstruct the provenance of the old one. + +The application should retain only the bounded state needed to publish the +current report, calculate Recent Changes against the last successfully +published report for the same valid period, and complete the current +invocation safely. Detailed LLM diagnostics should remain an explicit, +operator-controlled exception outside ordinary workspace state. + +This roadmap defines the intended state lifecycle, compatibility policy, and +architectural boundaries. A separate implementation plan will define the +ordered work after the roadmap is complete. + +## User Intent + +The state model should reflect these product expectations: + +- weather reports are ephemeral products, not business records; +- old report provenance has no continuing operational value once conditions + and forecasts have changed; +- regenerating is preferable to recovering, replaying, or inspecting an old + generation; +- routine operation should not accumulate unbounded run-addressed artifacts; +- Recent Changes remains useful, but needs only one prior successful snapshot + for the same report and valid period; and +- sensitive prompt and response capture remains opt-in and explicitly managed + by the operator. + +## Current State + +Each generation currently writes a run-addressed collection containing a +module snapshot, data package, prompt preparation receipt, prompt execution +receipt, raw generated text, validated generated text, render context, managed +report, metadata, and optional notification receipt. Successful and failed +runs accumulate beneath the workspace. + +Metadata links the collection and supports lookup by RunID. The CLI can list +historical runs and inspect their metadata, modules, data packages, prior +snapshots, and source provenance. New metadata uses the V2 format while the +reader retains V1 compatibility. Prompt artifacts are validated against +current report and prompt definitions when saved and loaded. + +Most of this persistence exists for retrospective inspection and failure +recovery. Dedicated prompt preparation and execution load operations have no +ordinary production consumer. The important exception is module snapshot +state: generation actively loads the most recent compatible snapshot to build +the deterministic Recent Changes input for Daily, Today, and Tomorrow. + +## Desired End State + +Weatherreporter has three distinct state classes: + +| State class | Lifecycle | Purpose | +| --- | --- | --- | +| Invocation workspace | Temporary and unpublished | Hold intermediate values while one report or batch is running. | +| Current published state | Bounded and replaceable | Hold the current managed report and the minimal deterministic snapshot or manifest needed for normal operation. | +| Secure LLM debug capture | Explicitly enabled and operator-managed | Diagnose prompt rendering or provider output when the operator deliberately requests sensitive capture. | + +Ordinary generation uses an invocation-scoped temporary directory on the same +filesystem as the managed workspace when atomic publication requires it. +Prompt preparation, prompt execution, raw generated text, validated generated +text, render contexts, data packages, and notification receipts may exist +there while needed, but they are not published as durable historical +artifacts. + +A successful report atomically replaces the current published state for its +logical report key and valid period. A failed attempt leaves the last +successfully published report and comparison snapshot unchanged. Ordinary +temporary artifacts are removed after both success and handled failure; +cleanup failure is reported safely but must not replace the primary generation +error. + +RunIDs remain useful as in-process correlation identifiers in action results, +logs, provider provenance, and optional debug paths. They no longer identify a +durable collection that Weatherreporter promises to locate or decode later. + +## Published Report Policy + +The managed Markdown report remains the authoritative upload source during an +invocation. The intended default is to retain only the current managed report +for each logical report key and valid period, replacing it atomically after a +new report has been fully rendered and validated. + +An explicit `--out` or `--out-dir` copy remains operator-owned output outside +the managed-state lifecycle. Weatherreporter does not delete, rotate, or +rewrite those copies except when the same explicit destination is selected by +a later invocation. + +Distributor continues to receive only a completed managed Markdown report. +Notification success or failure does not create a durable notification +history. A notification failure leaves the newly published report available +and returns a safe error through the current action result. + +## Recent Changes State + +Recent Changes must be preserved without preserving general report history. +For Daily, Today, and Tomorrow, Weatherreporter retains at most one compatible +module snapshot for each logical report key and valid period. + +The retained snapshot represents the last successfully published report. A +new invocation reads it before constructing Recent Changes and replaces it +only when the new managed report has been successfully validated, rendered, +and published. A failed generation therefore does not become the baseline for +the next report and cannot hide changes that the user has not yet seen. + +Hourly does not currently use the comparison strategy and should not retain a +comparison snapshot solely for symmetry. State whose valid period has ended +and can no longer participate in a supported comparison is eligible for safe +cleanup. + +## Temporary Workspace And Failure Semantics + +Temporary state must remain beneath a narrowly owned application directory and +use safe path construction, restrictive permissions where content is +sensitive, and atomic writes where practical. Publication must not expose a +partially rendered report or a snapshot that does not correspond to the +published report. + +Normal results retain bounded error information and paths only for artifacts +that remain meaningful after the command: a previously or newly published +report, an explicit operator output, or an enabled secure debug capture. +Temporary intermediate paths are not emitted as if they were durable recovery +locations. A failed command is retried by starting a new generation. + +Process interruption may leave an uncommitted temporary directory. Such a +directory is never considered published state, is never selected for Recent +Changes, and may be removed by a documented safe cleanup mechanism. Cleanup +must distinguish inactive temporary directories from concurrent active +invocations and must never recursively target the workspace root or an +unresolved configuration path. + +## Inspection And Metadata Policy + +Run-history discovery and inspection are not part of the desired product +contract. The historical `inspect reports`, `inspect metadata`, `inspect +modules`, `inspect data-package`, `inspect prior`, and `inspect sources` +surfaces are candidates for removal together rather than preservation through +a new storage representation. + +Any manifest retained for atomic publication or Recent Changes is current +operational state, not an archival metadata record. It should contain only the +identity, valid period, safe paths, and deterministic snapshot information +needed to validate and use that current state. It does not need to preserve +prompt messages, generated prose intermediates, source provenance, provider +provenance, notification history, or a catalog of prior runs. + +The application does not promise cross-version decoding of ordinary workspace +state. A new release may replace or ignore incompatible current-state files, +provided it fails safely, never mistakes stale state for a compatible Recent +Changes baseline, and documents any operator action required during upgrade. + +## Prompt Execution And Debugging + +Prompt inspection before weather collection and prepared execution remain +runtime safety requirements. They do not require durable preparation or +execution receipts. + +The selected logical profile, effective backend and model, validation outcome, +and safe classified error remain available to the active workflow and its CLI +summary where useful. Weatherreporter does not retain them as long-term report +provenance after the invocation completes. + +The existing explicit secure debug root remains outside ordinary state and may +retain rendered prompts, schemas, input bodies, generated bodies, and effective +parameters according to its documented contract. Weatherreporter does not +automatically clean that operator-selected location. Credentials must remain +excluded from debug capture. + +## Compatibility And Upgrade Policy + +This is an intentional breaking change to the workspace and inspection +contracts. Weatherreporter does not need to migrate historical V1 or V2 +metadata, prompt receipts, intermediate generated-text artifacts, or managed +reports into the new representation. + +Legacy workspace trees must not be silently interpreted as current published +state. They also must not be deleted automatically merely because a new +version starts: an operator may have placed or referenced files there despite +the absence of a continuing application compatibility promise. Release notes +and operations documentation must explain whether legacy data can be removed +manually and identify the exact safe target. + +The change should land in a release whose notes clearly identify removed CLI +commands, obsolete paths and schemas, the new bounded state behavior, and any +upgrade action. Because Weatherreporter remains pre-1.0, the ordinary semantic +version policy may carry this breaking change without inventing a migration +framework. + +## Required Architecture Decision Record + +The implemented feature must include an Accepted ADR recording the durable +architectural decision to use ephemeral operational state. The ADR is not part +of this roadmap-writing pass and should not be created until implementation is +being prepared. + +The ADR should record: + +- the mismatch between run-addressed provenance storage and the ephemeral + weather-report lifecycle; +- the decision to retain bounded current report and comparison state rather + than historical runs; +- the distinction between temporary invocation state, published operational + state, explicit output copies, and secure debug capture; +- the removal of historical inspection and backward-compatibility guarantees; +- atomic publication and failed-run behavior; +- the alternatives considered, including retaining the current archive, + adding time-based retention, or keeping a bounded run history; and +- consequences for CLI compatibility, workspace layout, testing, operations, + and future schema changes. + +Once accepted, the ADR owns the decision rationale. The architecture policy +owns the resulting current invariant, while focused state, CLI, operations, +and integration documents own the implemented contracts. + +## Scope + +The completed feature includes: + +- an invocation-scoped temporary workspace for intermediate generation state; +- atomic publication of the current managed report and its minimal operational + state; +- a bounded comparison snapshot representing the last successfully published + report for each supported report key and valid period; +- safe cleanup behavior for normal completion, handled failure, and abandoned + temporary workspaces; +- removal of durable prompt preparation, prompt execution, generated-text, + render-context, data-package, notification, and run-metadata history; +- removal of run-history inspection commands and their application/state + contracts; +- removal of V1 metadata compatibility and current-version coupling for + historical prompt artifacts by removing the historical artifact contract; +- preservation of active-command partial status, safe errors, and paths to + genuinely retained published, operator-owned, or debug outputs; +- preservation of explicit output copies, Distributor upload behavior, and + opt-in secure LLM debug capture; +- risk-appropriate offline tests for atomic publication, comparison baselines, + failure isolation, cleanup safety, concurrent invocation safety, and absence + of unbounded state growth; +- an Accepted ADR documenting the architectural decision; and +- updates to canonical architecture, CLI, operations, configuration, + troubleshooting, integration, internal, testing, and release documentation + where their contracts change. + +## Non-Goals + +This feature does not include: + +- a general-purpose cache, database, archival service, or retention engine; +- replaying or resuming interrupted generation; +- migrating legacy artifacts into the new representation; +- retaining a bounded number of historical runs for convenience; +- automatic upload or archival of state to remote storage; +- collecting additional provider telemetry or weather-source provenance; +- changing report content, prompt text, schemas, profile selection, weather + derivation, or batch membership; +- deleting operator-owned `--out`, `--out-dir`, or secure debug files; +- changing Distributor's report-content contract; or +- making live external services part of the default test suite. + +## Safety And Testing Policy + +The state refactoring must preserve Weatherreporter's existing path-safety and +atomicity expectations while reducing the amount of durable state. Tests +should emphasize observable lifecycle guarantees rather than private file +choreography. + +Important risks requiring durable offline coverage include: + +- a failed or canceled generation replacing a previously published report or + comparison baseline; +- a partially written report becoming visible as current; +- Recent Changes selecting an incompatible report, valid period, or failed + attempt; +- cleanup deleting published, operator-owned, debug, or concurrently active + files; +- batch partial success corrupting the state of another report; +- notification failure rolling back or obscuring a successfully published + report; +- stale or incompatible current state being treated as valid; and +- repeated successful and failed runs causing unbounded ordinary workspace + growth. + +Tests remain deterministic, offline, credential-free, and based on real +temporary filesystems plus narrow external-boundary fakes. Race-enabled tests +are required where publication, cleanup, or concurrent invocation behavior +shares mutable filesystem state. + +## Relationship To Domain-Specific Profiles + +The domain-specific profile feature can be implemented before this refactor, +but it should not add new historical compatibility or durable-provenance +commitments. Profile inspection, selection, override precedence, and effective +model resolution remain active-workflow behavior and survive the state change. + +The domain-profile roadmap and implementation plan should acknowledge that +prompt artifacts from version `1.0.1` need not remain readable after prompts +advance to `1.1.0`. Existing state persistence may remain temporarily while +the profile feature lands, but it should not be expanded or treated as the +target architecture. + +## Completion Criteria + +The roadmap's target state is achieved when: + +- ordinary runs no longer create durable run-addressed artifact collections; +- a successful report atomically replaces only the corresponding current + published state; +- failed and canceled attempts leave the prior published report and Recent + Changes baseline unchanged; +- Daily, Today, and Tomorrow compare against at most one compatible snapshot + from the last successfully published report; +- expired comparison and published state can be removed safely without + touching operator-owned or active files; +- Hourly does not retain an unused comparison snapshot; +- historical inspection commands and V1/V2 archival compatibility code are + removed; +- temporary, published, explicit-output, and debug paths have distinct and + documented ownership and cleanup rules; +- Distributor and active-command summaries continue to receive the completed + report and safe status information they require; +- the default suite proves atomicity, bounded growth, cleanup safety, batch + isolation, and comparison correctness offline; +- an Accepted ADR records the architectural decision and alternatives; and +- canonical current-state documentation describes only the implemented + lifecycle. + +## Open Questions + +### Lifetime of the current managed report + +Recommendation: retain one current managed report per logical report key and +valid period until it is replaced or its valid period expires. This preserves +the current default behavior for invocations without `--out` while bounding +growth. + +Alternative: treat the managed report as temporary and retain output only when +the operator supplies `--out` or `--out-dir`. This minimizes state further but +makes a successful default invocation produce no durable report for the user +and complicates Distributor sequencing. + +### Historical inspection replacement + +Recommendation: remove the run-history inspection commands without adding a +replacement initially. Current command summaries, current managed files, and +opt-in debug capture cover the remaining supported workflows. + +Alternative: add a narrow `inspect current REPORT` command backed only by the +current operational manifest. This provides discoverability without history, +but it creates a new public surface and may preserve metadata complexity that +the refactor is intended to remove. + +### Abandoned temporary workspace cleanup + +Recommendation: use an explicitly owned temporary subtree with per-invocation +ownership markers and a conservative age threshold. Normal cleanup removes the +current invocation synchronously; opportunistic cleanup removes only marked, +inactive directories old enough that they cannot reasonably belong to a live +invocation. + +Alternative: perform only synchronous cleanup and document manual removal of +directories left by process termination. This minimizes destructive code and +concurrency risk, but crashed processes can still accumulate unbounded files. + +### Legacy workspace cleanup + +Recommendation: ignore legacy run-addressed trees and document a precise, +manual one-time cleanup procedure. Do not automatically delete them during +startup or upgrade. + +Alternative: add an explicit cleanup command that previews and then removes +recognized legacy artifacts. This is more convenient for large installations +but introduces a destructive command and a legacy-format classifier that must +be maintained and tested. diff --git a/docs/roadmap/promptkit-fallback-profiles-feature-request.md b/docs/roadmap/promptkit-fallback-profiles-feature-request.md index 8a4c541..9181395 100644 --- a/docs/roadmap/promptkit-fallback-profiles-feature-request.md +++ b/docs/roadmap/promptkit-fallback-profiles-feature-request.md @@ -1,6 +1,11 @@ # Promptkit Feature Request: Application Fallback Profiles -Status: Proposed upstream capability. +Status: Implemented upstream in Promptkit v0.5.0. + +Promptkit v0.5.0 resolved this request with the public +`WithFallbackProfileFS` engine option and the precedence and error semantics +specified below. This document is retained as the downstream rationale for +the capability. ## Purpose