From 7d591487e47a3c5257801deb0348c936125445e2 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 1 Aug 2026 18:16:01 +0000 Subject: [PATCH] Clean up roadmap and troubleshooting documentation --- README.md | 1 - docs/development.md | 2 +- docs/operations.md | 20 +- docs/policy/documentation.md | 8 +- docs/roadmap/domain-profiles.md | 283 --------- docs/roadmap/ephemeral-state.md | 17 +- docs/roadmap/future.md | 2 +- docs/roadmap/implementation.md | 599 ------------------ ...ptkit-fallback-profiles-feature-request.md | 169 ----- docs/roadmap/promptkit.md | 516 --------------- docs/troubleshooting.md | 75 --- 11 files changed, 21 insertions(+), 1671 deletions(-) delete mode 100644 docs/roadmap/domain-profiles.md delete mode 100644 docs/roadmap/implementation.md delete mode 100644 docs/roadmap/promptkit-fallback-profiles-feature-request.md delete mode 100644 docs/roadmap/promptkit.md delete mode 100644 docs/troubleshooting.md diff --git a/README.md b/README.md index afe931b..ea84f45 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,5 @@ Configure a Weather API endpoint first; see the - [CLI reference](docs/cli.md) - [Configuration reference](docs/config.md) - [Operations guide](docs/operations.md) -- [Troubleshooting](docs/troubleshooting.md) - [Development guide](docs/development.md) - [Architecture policy](docs/policy/architecture.md) diff --git a/docs/development.md b/docs/development.md index 652aeca..dc11327 100644 --- a/docs/development.md +++ b/docs/development.md @@ -29,7 +29,7 @@ boundaries and invariants. | Recent Changes comparison | [Changes internals](internal/changes.md) and [operations guide](operations.md) | The internal guide owns structured comparison; operations owns user-visible artifact behavior. | | Prompt execution, profiles, prompt inputs, or result handling | `internal/promptexec`, the Promptkit adapter, and [prompt-input internals](internal/prompt-input.md) | These separate the executor contract and input construction. | | Generated-text schemas, validation, render contexts, templates, or Markdown rendering | [Generated-text internals](internal/generatedtext.md), [report-template internals](internal/reporttemplate.md), and [report template guide](templates.md) | These own structured text, renderer implementation, and the maintainer-facing template surface. | -| Workspace paths, metadata, atomic persistence, lookup, inspection, or recovery | [State internals](internal/state.md), [operations guide](operations.md), and [troubleshooting guide](troubleshooting.md) | These separate implementation, operator workflows, and symptom-based recovery. | +| Workspace paths, metadata, atomic persistence, lookup, inspection, diagnosis, or recovery | [State internals](internal/state.md) and [operations guide](operations.md) | These separate implementation behavior from operator workflows, diagnosis, and recovery. | | Distributor bundles, uploads, notification artifacts, or failures | [Distributor adapter internals](internal/distributor-adapter.md), [Distributor integration contracts](integrations/distributor/), and [operations guide](operations.md) | These separate adapter behavior, external contracts, and operational lifecycle. | | Maintained example configuration | [Configuration reference](config.md) and files under `examples/` | The reference owns field meaning; examples own complete copyable files. | | Release preparation, tagging, publication, or verification | [Release procedure](release.md) | It owns version selection, release-note preparation, candidate validation, tag publication, CI behavior, and post-publication checks. | diff --git a/docs/operations.md b/docs/operations.md index 6223d7f..34cb9b6 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -3,8 +3,7 @@ This guide covers normal operation, managed workspace state, inspection, recovery, and operational caveats. See the [CLI reference](cli.md) for complete command syntax and the [configuration reference](config.md) for fields, -defaults, and notification templates. For symptom-based diagnosis, see -[Troubleshooting](troubleshooting.md). +defaults, and notification templates. ## Normal Operation @@ -19,8 +18,8 @@ persists the module snapshot and prompt data package, records Promptkit preparation provenance before provider execution, then persists raw output and execution provenance, validates the structured generated text, and renders the managed Markdown report from the validated text and deterministic values. -The current receipts are transitional workspace state, not a cross-version -profile-provenance contract. +These receipts describe the current workspace implementation; they are not a +cross-version profile-provenance contract. The managed report and its final metadata are saved before single-report Distributor notification is attempted. `--out` writes an extra operator copy; @@ -48,8 +47,9 @@ Prompt inspection occurs before weather collection. A malformed profile file, missing required credential, or unsupported selected backend stops the command before collection. A reachable profile can still fail later if its local model endpoint is unavailable; Weatherreporter does not switch to a remote profile. -See the [configuration reference](config.md) for field definitions and the -[troubleshooting guide](troubleshooting.md) for recovery. +See the [configuration reference](config.md) for field definitions. For a +provider failure, start with the command error and enable secure prompt debug +capture when content-rich diagnostics are required. ## Optional Prompt Debug Capture @@ -113,9 +113,9 @@ The generated-text and render-context artifacts are written for every completed single-report generation. A report's metadata links the module snapshot, data package, preparation and execution receipts, managed report, generated-text artifacts, and any available single-report -notification artifact. These current-version receipts remain transitional; use -the active command's classified error and explicit secure debug capture for -prompt diagnosis rather than relying on them as a durable interface. Batch +notification artifact. These current-version receipts are implementation +details rather than durable interfaces. Use the active command's error and +explicit secure debug capture for prompt diagnosis. Batch notification artifacts are separate batch-level records under `notifications/batches`. @@ -191,7 +191,7 @@ remain available where they can be safely persisted. - A batch notification failure preserves each report's artifacts and adds the top-level batch notification artifact. -Use the action summary and its classified error first. For prompt or provider +Use the action summary and its error first. For prompt or provider diagnosis, prefer an explicitly enabled secure debug capture; current-version receipt paths may provide supplemental context when available. For a batch failure, inspect the summary first, then inspect the affected report RunIDs or diff --git a/docs/policy/documentation.md b/docs/policy/documentation.md index 38f4aed..ae9d6e0 100644 --- a/docs/policy/documentation.md +++ b/docs/policy/documentation.md @@ -86,8 +86,7 @@ mechanisms, not secret values. | Release notes | `docs/releases/` | One versioned, changelog-style summary for each release, including compatibility and operator action. The file at the tagged commit supplies the corresponding Gitea release body. | Current CLI, configuration, operations, integration, architecture, and internal contracts; release procedure; implementation plans. | | CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, stdout and stderr behavior, summaries, and exit behavior. | Configuration field definitions, complete operating procedures, runtime filesystem layout, and command implementation. | | Configuration contract | `docs/config.md` | Discovery and precedence, fields, defaults, secrets, validation rules, and user-selectable values. | Complete example files, CLI syntax, runtime state lifecycle, and loading implementation. | -| Operations | `docs/operations.md` | Normal workflows, physical workspace layout, artifacts and metadata, inspection, notification behavior, recovery, cleanup, permissions, and operational caveats. | Complete CLI syntax, configuration field definitions, logical external contracts, and implementation mechanics. | -| Troubleshooting | `docs/troubleshooting.md` | Recurring symptoms, likely causes, diagnostic steps, safe fixes, and links to normal-operation references. | Complete command and configuration references, routine operating procedures, and implementation detail. | +| Operations | `docs/operations.md` | Normal workflows, physical workspace layout, artifacts and metadata, inspection, notification behavior, diagnosis, recovery, cleanup, permissions, and operational caveats. | Complete CLI syntax, configuration field definitions, logical external contracts, and implementation mechanics. | | Report template surface | `docs/templates.md` | Implemented template files and partials, render-context fields, editing rules, and maintainer-facing template examples. | Weather derivation, module implementation, generated-text validation internals, and operator procedures. | | External and durable integration contracts | `docs/integrations/` | Weather API, Promptkit, Distributor, external formats and protocols, durable logical paths and schemas, compatibility behavior, and upstream or downstream responsibilities. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, and configuration defaults. | | Internal subsystem behavior | `docs/internal/` | Implementation flow, internal collaborators and state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, user-facing contracts, external schemas, operator procedures, and future package plans. | @@ -110,13 +109,12 @@ structure and invariants. Focused internal documents own implementation behavior. These documents may link to one another but must not maintain parallel package or behavior references. -### Commands, Configuration, Operations, And Troubleshooting +### Commands, Configuration, And Operations CLI documentation answers how to invoke Weatherreporter and what its command interface does. Configuration documentation answers what settings mean. Operations answers what happens to runtime state and how to operate or recover -the application. Troubleshooting starts from a symptom and leads to diagnosis -and a safe fix. +the application, including diagnosis and safe responses to runtime failures. When a workflow crosses these topics, place the complete procedure with the document that owns the task and link to the other contracts. Do not duplicate diff --git a/docs/roadmap/domain-profiles.md b/docs/roadmap/domain-profiles.md deleted file mode 100644 index 103dd95..0000000 --- a/docs/roadmap/domain-profiles.md +++ /dev/null @@ -1,283 +0,0 @@ -# Domain-Specific Prompt Profiles Roadmap - -Status: Implemented. - -## Purpose - -Weatherreporter should provide stable, domain-specific Promptkit profile IDs -that express the relative resource and analysis needs of its report products. -These logical profiles should give each report an appropriate default while -allowing operators to replace any definition through the existing configured -profile source. - -This roadmap records the scope, policy, and implemented end state. The -companion [implementation plan](implementation.md) records the ordered work -and verification used to reach it. - -## User Intent - -The feature is intended to provide three related benefits: - -- frequent reports can use a cost-effective model by default; -- reports needing broader synthesis can select a stronger default without - forcing the same cost on every invocation; and -- an installation can map a stable Weatherreporter profile ID to a model on a - local network endpoint without modifying embedded prompts or application - code. - -`weather-light` describes the profile's intended resource tier, not a latency -guarantee. A locally hosted lightweight model may still generate slowly on the -available hardware. - -## Pre-Implementation Baseline - -Before implementation, Daily, Today, Tomorrow, and Hourly each declared -Promptkit's `gemini-flash-latest` profile as their prompt default. The optional -`promptkit.profile` setting overrode that default for every selected report in -an invocation. - -Weatherreporter accepted either `promptkit.profile_file` or -`promptkit.profile_dir` and passed that source to Promptkit. A matching external -profile could override a Promptkit built-in profile, and the configured local -backend could support profiles that select `backend: local`. Endpoint-only -OpenAI-compatible profiles could also provide their own endpoint. - -Weatherreporter did not own or embed execution profiles. Promptkit v0.5.0 -provided the fallback-profile layer used to add them without changing the -existing operator-source precedence. - -## Prerequisite - -Promptkit v0.5.0 provides the application fallback profile capability defined -in the companion -[upstream feature request](promptkit-fallback-profiles-feature-request.md), and -Weatherreporter now depends on that tagged release. The dependency upgrade has -passed the repository test suite and an operator smoke test. Weatherreporter -must continue to use only Promptkit's public API rather than depending on its -internal packages or reproducing its profile repository behavior. - -## Implemented End State - -Weatherreporter embeds usable definitions for these exact logical profile IDs: - -- `weather-light` -- `weather-balanced` -- `weather-deep` - -The profiles are Weatherreporter-owned assets and remain behind the existing -Promptkit adapter boundary. Prompt definitions select the logical IDs, while -Promptkit resolves the effective backend, endpoint, model, and generation -settings. - -An operator can place a profile with the same ID in `profile_file` or -`profile_dir`. The operator definition completely replaces the embedded -Weatherreporter definition for that ID. If the external source does not contain -the selected ID, lookup falls through to Weatherreporter's embedded profile and -then to Promptkit's built-in catalog. - -The existing global `promptkit.profile` setting remains available as an -explicit all-report override. No new configuration field is required for the -initial feature. - -## Profile Catalog And Report Assignment - -| Profile | Meaning | Initial default reports | -| --- | --- | --- | -| `weather-light` | Lowest-cost supported tier for frequent, bounded synthesis. It makes no latency promise. | Hourly | -| `weather-balanced` | General-purpose tier for broader day-scale synthesis and forecast discussion. | Daily, Today, Tomorrow | -| `weather-deep` | Highest-capability tier for explicit operator use and future products whose measured quality benefit warrants the cost. | None initially | - -The initial assignment recognizes that Weatherreporter's deterministic modules -already perform most weather selection and calculation. A higher-capability -model should not become a default merely because it is available. Moving an -existing report to `weather-deep` requires evidence that the stronger tier -materially improves supported reasoning or output quality. - -The three profile IDs are capability policies, not permanent aliases for one -provider or model family. Their embedded definitions may change in a future -Weatherreporter release, with the change disclosed through normal release and -compatibility documentation. - -## Selection And Definition Precedence - -Profile ID selection and profile definition lookup are separate decisions. - -Weatherreporter selects the profile ID in this order: - -1. nonblank `promptkit.profile`; or -2. the exact prompt version's `default_profile`. - -Promptkit then resolves the selected profile definition in this order: - -1. explicit in-memory profiles, when used by an embedding consumer or test; -2. Weatherreporter's configured `profile_file` or `profile_dir` source; -3. Weatherreporter's embedded fallback profiles; and -4. Promptkit's embedded built-in profiles. - -A higher-precedence source falls through only when the selected ID is absent. -A matching but malformed operator profile fails before weather collection and -must not silently use the embedded definition. - -## Local Endpoint Experience - -An operator should be able to override `weather-light` with an endpoint-only -profile whose model name is understood by the local OpenAI-compatible server. -This path does not require a separate Weatherreporter local-backend setting. - -Alternatively, an override may select `backend: local`; in that case the -existing `promptkit.local.endpoint` and concurrency settings continue to own -the shared local backend definition. - -The selected local profile is deterministic configuration, not a preference -hint. Weatherreporter will not probe for a local model and will not -automatically fall back to a remote or paid profile when the endpoint is -unavailable. The failure remains visible and attributable to the selected -profile. - -## Embedded Profile Policy - -Each embedded profile must be a complete, valid Promptkit profile and must be -usable in a default installation with the documented credential mechanism. The -initial embedded profiles are expected to use Promptkit's `openrouter` backend, -allowing them to inherit its endpoint and `OPENROUTER_API_KEY` environment -variable without embedding credentials. - -Embedded definitions should include only settings that are intentional for the -selected model and supported by its backend. Avoid incidental generation -parameters that reduce portability or trigger provider-specific request -failures without a demonstrated quality benefit. - -The initial profile definitions are: - -| Profile ID | OpenRouter model | Reasoning effort | Timeout | Service tier | -| --- | --- | --- | --- | --- | -| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` | -| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | -| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | - -These settings deliberately match the corresponding Promptkit v0.5.0 -built-ins while exposing Weatherreporter-owned logical IDs. The `~` prefix is -part of each OpenRouter rolling-alias identifier. The profiles do not set -temperature, `top_p`, or output-token limits; omission preserves provider -defaults and avoids unsupported incidental parameters. - -## Prompt And Active Execution Contract - -Changing a prompt's `default_profile` is a material prompt-definition change. -The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new -defaults are introduced. Prompt content and generated-text schemas need not -change solely for this feature. - -Prompt inspection must continue to occur before weather collection. It should -report the selected logical profile ID and the resolved backend and model -without exposing endpoints or credentials. - -The active execution contract should retain both the selected logical profile -identity and the resolved backend and model through inspection, preparation, -execution, errors, and command results where those values are already exposed. -This feature must not add a new durable-provenance or cross-version artifact -contract. - -The accepted [ephemeral-state roadmap](ephemeral-state.md) makes historical -prompt provenance a non-goal. Existing workspace persistence may remain while -this feature lands, but it is transitional behavior and must not be expanded or -treated as part of the profile feature's desired end state. Prompt preparation -and execution artifacts written at `1.0.1` are not required to remain readable -after the prompt definitions advance to `1.1.0`. - -## Evaluation Policy - -Concrete model assignments should be evaluated with representative, -secret-free Daily, Today, Tomorrow, and Hourly data packages. Evaluation should -consider: - -- strict-schema success rate; -- unsupported or invented weather claims; -- precipitation-timing accuracy and empty-string behavior; -- correct use of deterministic hazards, periods, and uncertainty; -- summary and forecast-discussion usefulness; -- generation latency; -- token use and provider cost; and -- behavior through a representative local OpenAI-compatible endpoint. - -The purpose is to choose an appropriate default for each tier, not to add a -permanent benchmark framework or live-provider requirement to the ordinary -test suite. Repository tests remain offline and deterministic. - -## Implemented Scope - -The completed feature includes: - -- Weatherreporter-owned embedded profile assets for all three logical IDs; -- Promptkit adapter wiring that supplies those assets as the application - fallback profile source; -- per-prompt default-profile assignments matching the catalog above; -- an exact prompt-version update for the changed definitions; -- preservation of the global profile override; -- same-ID override behavior through both supported external profile-source - forms; -- local-backend and endpoint-only override coverage; -- fail-fast inspection of missing, malformed, or unusable selected profiles; -- offline tests for selection, source precedence, effective model inspection, - batch reuse, and active execution behavior; -- maintained operator examples for overriding `weather-light` locally; and -- updates to the canonical configuration, Promptkit integration, report - registry, operations, troubleshooting, internal adapter, and release - documentation as applicable when implementation lands. - -## Non-Goals - -The feature does not include: - -- automatic discovery, health checking, or benchmarking of local endpoints; -- implicit failover between local and remote profiles; -- retries with a more expensive tier after provider or validation failure; -- per-report profile configuration fields outside prompt defaults; -- profile inheritance, aliases, or field-level merging; -- runtime model selection based on weather severity, token count, or report - content; -- moving Weatherreporter profile policy into Promptkit's built-in catalog; -- exposing Promptkit types outside the adapter boundary; or -- making live provider calls part of the default repository test suite. - -## Compatibility And Operational Policy - -Existing configurations with a nonblank `promptkit.profile` retain their -all-report behavior. Existing `profile_file`, `profile_dir`, local-backend, and -credential configuration fields retain their meanings. - -Configurations that rely on the omitted profile setting will intentionally -observe new per-report defaults. This is a user-visible model-selection and -cost change and must be called out in release notes. Operators who require the -old all-report model can preserve it by setting an explicit global profile. - -The prompt-version transition does not provide backward compatibility for -historical prompt preparation or execution artifacts. This is consistent with -the accepted ephemeral-state direction; the profile feature does not otherwise -redesign or remove the current workspace layout. - -An external same-ID override is an operator-owned compatibility commitment. -Weatherreporter may evolve its embedded definitions, but it must not rewrite or -silently merge an operator file. - -## Completion Record - -The following conditions are satisfied: - -- a tagged Promptkit dependency supports the required fallback layer; -- every operational prompt selects its assigned logical profile at exact - version `1.1.0`; -- all three embedded profiles inspect successfully without an external profile - source; -- configured same-ID definitions override embedded definitions through both - `profile_file` and `profile_dir`; -- an invalid matching external definition fails without fallback; -- `weather-light` can resolve through an endpoint-only or configured-local - override without requiring code or prompt changes; -- global `promptkit.profile` still overrides every report in an invocation; -- active inspection and execution preserve the selected logical profile and - effective model through the project-owned execution contract; -- morning and evening batch preflight deduplicates inspection of shared - effective profile IDs as it does today; -- the default test suite remains offline and deterministic; and -- implemented behavior is documented by its canonical current-state owners. diff --git a/docs/roadmap/ephemeral-state.md b/docs/roadmap/ephemeral-state.md index d3016a8..47398eb 100644 --- a/docs/roadmap/ephemeral-state.md +++ b/docs/roadmap/ephemeral-state.md @@ -247,7 +247,7 @@ The completed feature includes: of unbounded state growth; - an Accepted ADR documenting the architectural decision; and - updates to canonical architecture, CLI, operations, configuration, - troubleshooting, integration, internal, testing, and release documentation + integration, internal, testing, and release documentation where their contracts change. ## Non-Goals @@ -296,16 +296,11 @@ shares mutable filesystem state. ## Relationship To Domain-Specific Profiles -The domain-specific profile feature can be implemented before this refactor, -but it should not add new historical compatibility or durable-provenance -commitments. Profile inspection, selection, override precedence, and effective -model resolution remain active-workflow behavior and survive the state change. - -The domain-profile roadmap and implementation plan should acknowledge that -prompt artifacts from version `1.0.1` need not remain readable after prompts -advance to `1.1.0`. Existing state persistence may remain temporarily while -the profile feature lands, but it should not be expanded or treated as the -target architecture. +The implemented domain-specific profiles do not add historical compatibility +or durable-provenance commitments. Profile inspection, selection, override +precedence, and effective model resolution remain active-workflow behavior and +survive the state change. Existing prompt artifacts need not remain readable +after this refactor and must not constrain the target architecture. ## Completion Criteria diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index accbc31..5b88f75 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -13,7 +13,7 @@ Possible direction: 1. Detect candidate storm events from alerts, forecast discussion, weather story context, hourly thresholds, and material forecast changes. -2. Evaluate candidates through Scriptorium or another narrow evaluator adapter. +2. Evaluate candidates through Promptkit or another narrow evaluator adapter. 3. Persist storm lifecycle state. 4. Generate or update a storm report only when a meaningful event is present. 5. Suppress ordinary low-impact thunder or rain chances. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 271b2c8..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,599 +0,0 @@ -# Domain-Specific Prompt Profiles Implementation Plan - -Status: Stages 1–7 completed; remediation Stage 8 ready. - -## Purpose And Authority - -This document records the implementation and post-implementation remediation -of the -[domain-specific prompt profiles roadmap](domain-profiles.md). The roadmap is -authoritative for scope, user intent, policy choices, and the intended end -state. This plan records implementation sequence, verification, audit findings, -and exit gates. - -This plan follows the repository's -[architecture](../policy/architecture.md), -[documentation](../policy/documentation.md), and -[testing](../policy/testing.md) policies. - -## Completed Prerequisite - -Weatherreporter is already pinned to Promptkit v0.5.0. That release provides -the public `WithFallbackProfileFS` option and the required precedence across -inspection, preparation, and execution. The dependency upgrade passed -Weatherreporter's full offline test suite, race-enabled suite, CLI help check, -and an operator `generate hourly` smoke test. Do not repeat or replace the -dependency upgrade as part of these stages. - -## Locked Product Decisions - -Implement these exact Weatherreporter-owned profiles: - -| Profile ID | Backend | Model | Reasoning effort | Timeout | Service tier | -| --- | --- | --- | --- | --- | --- | -| `weather-light` | `openrouter` | `deepseek/deepseek-v4-flash` | Omitted | 180 seconds | `flex` | -| `weather-balanced` | `openrouter` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | -| `weather-deep` | `openrouter` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | - -Assign Hourly to `weather-light`; assign Daily, Today, and Tomorrow to -`weather-balanced`; assign no report to `weather-deep` initially. Advance all -four prompt definitions and matching report-registry entries from `1.0.1` to -`1.1.0` when their defaults change. - -The leading `~` in the Gemini and Claude model IDs is required and denotes an -OpenRouter rolling alias. Do not substitute the unavailable non-tilde IDs or a -dated model version. Do not add temperature, `top_p`, maximum-token, endpoint, -or credential fields to the embedded definitions. - -Prompt preparation and execution artifacts written at `1.0.1` are not required -to remain readable after the transition to `1.1.0`. Do not add a migration, -compatibility shim, or weaker historical-artifact validation for this feature. - -Definition lookup must remain: - -1. explicit Promptkit in-memory profiles used by tests or an embedding - consumer; -2. Weatherreporter's configured `profile_file` or `profile_dir` source; -3. Weatherreporter's embedded fallback profiles; and -4. Promptkit's built-in catalog. - -Selection remains a separate concern: a nonblank global `promptkit.profile` -selects the profile for every report in the invocation; otherwise the exact -prompt definition's `default_profile` selects it. A malformed matching -higher-precedence profile is an error and never falls through. - -## Continuing Invariants - -- Keep all Promptkit types and mechanics inside - `internal/adapters/promptkit`, its focused tests, and asset contract tests. -- Keep prompt inspection before weather collection and provider work. -- Keep one Promptkit engine per command action and one shared engine across a - sequential batch. -- Preserve logical profile ID and effective backend/model information through - active inspection and execution where the project-owned contract already - exposes it. Do not add new durable-provenance fields or compatibility - guarantees. -- Leave existing workspace persistence behavior otherwise unchanged. The - accepted [ephemeral-state roadmap](ephemeral-state.md) owns its future - removal and must not be partially implemented here. -- Do not expose endpoints, credentials, rendered messages, schemas, request - bodies, response bodies, or complete parameter maps through ordinary errors, - logs, summaries, or state. -- Keep the default suite deterministic, offline, and credential-free. -- Do not add endpoint discovery, health probing, provider failover, retries at - a more expensive tier, profile merging, per-report configuration fields, or - severity-driven model selection. -- Update canonical current-state documentation only in the stage where the - corresponding behavior becomes implemented. -- Run `git diff --check` before completing every stage. - -## Stage 1: Add The Embedded Weather Profile Catalog - -### Goal - -Create one repository-owned, embedded profile source containing exactly the -three locked logical profiles. - -### Work - -1. Add strict YAML profile assets beneath `internal/promptassets` using the - exact IDs and definitions in this plan. -2. Extend `internal/promptassets` with a narrowly named accessor that returns - the embedded profile `fs.FS`. Follow the existing prompt and schema asset - pattern without exposing Promptkit types from the package. -3. Keep profile filenames and embed layout simple and deterministic. Do not - duplicate Promptkit's built-in directory taxonomy unless the application - assets require it. -4. Validate the assets through Promptkit's public engine/profile inspection - surface rather than adding a second YAML parser or a Weatherreporter-owned - profile representation. - -### Tests - -- Extend the asset contract tests to assert exactly the three logical IDs, - their exact effective model IDs, and the intentional parameters. -- Prove all three profiles inspect successfully offline when supplied as a - fallback source and no operator source is present. -- Assert that the catalog contains no endpoints, credentials, temperature, - `top_p`, or maximum-token settings. -- Run: - - ```sh - go test ./internal/promptassets - git diff --check - ``` - -### Exit Gate - -The embedded catalog is complete, strictly valid, safe, and independently -inspectable through Promptkit v0.5.0's public API. - -## Stage 2: Wire Fallback Resolution And Protect Precedence - -### Goal - -Supply the embedded catalog through Promptkit's application fallback layer -without changing existing operator configuration or application boundaries. - -### Work - -1. Add `promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), ".")` to - normal adapter engine construction. -2. Preserve existing `profile_file`, `profile_dir`, configured local backend, - timeout, prompt filesystem, schema filesystem, and test-option behavior. -3. Ensure ordinary production construction and the adapter's test - construction path exercise the same fallback wiring. Test-only explicit - profiles may retain Promptkit's documented highest precedence. -4. Keep all fallback resolution in Promptkit. Do not add filesystem overlays, - existence checks, YAML parsing, or merge behavior to Weatherreporter. - -### Tests - -- At the adapter boundary, prove fallback-only inspection of all three - Weatherreporter profiles. -- Prove same-ID overrides through both configured `profile_file` and - `profile_dir`, including resolution of the override's effective backend and - model. -- Prove an absent operator match falls through, while a malformed matching - operator definition fails without using the embedded profile. -- Prove a selected Promptkit built-in that is absent from both higher layers - still resolves. -- Prove an explicit in-memory test profile retains highest precedence. -- Cover both local override forms required by the roadmap: an endpoint-only - OpenAI-compatible `weather-light` profile and a `backend: local` profile - using the configured local endpoint. No test may contact either endpoint. -- Run: - - ```sh - go test ./internal/adapters/promptkit - git diff --check - ``` - -### Exit Gate - -Inspection and prepared execution use Promptkit's exact four-layer precedence, -operator errors remain visible, and local overrides require no prompt or code -changes. - -## Stage 3: Adopt Logical Defaults And Prompt Version 1.1.0 - -### Goal - -Move operational prompts from provider-oriented defaults to the three-tier -Weatherreporter policy with an exact, synchronized version transition. - -### Work - -1. Change Hourly's `default_profile` to `weather-light`. -2. Change Daily, Today, and Tomorrow to `weather-balanced`. -3. Advance the exact version in all four prompt YAML assets from `1.0.1` to - `1.1.0` without changing prompt text or generated-text schemas solely for - this feature. -4. Advance the four matching report-registry prompt versions to `1.1.0` in the - same change. Keep prompt IDs, report IDs, modules, periods, templates, and - output contracts unchanged. -5. Update fixtures and expectations that intentionally assert the current - prompt contract. Do not rewrite historical fixture versions or weaken tests - that protect actual compatibility. - -### Tests - -- Update asset and report-registry contract tests to require exact version - `1.1.0` and the report-to-profile assignments locked in this plan. -- Inspect every exact prompt version through the real embedded prompt, schema, - and fallback-profile filesystems. -- Prove Hourly resolves DeepSeek V4 Flash, the three day-scale reports resolve - Gemini Flash Latest, and `weather-deep` remains inspectable but unassigned. -- Run: - - ```sh - go test ./internal/promptassets ./internal/report ./internal/adapters/promptkit - git diff --check - ``` - -### Exit Gate - -Every operational prompt and registry definition agrees on exact version -`1.1.0`, selects its intended logical tier, and resolves its expected effective -model offline. - -## Stage 4: Verify Application Selection And Batch Reuse - -### Goal - -Protect the assembled application behavior created by the new defaults and -confirm that logical identity is not lost during active effective-model -resolution. - -### Work - -1. Preserve the current pre-collection inspection order and fail-fast behavior - for missing credentials, unknown profiles, malformed profiles, and unusable - backends. -2. Preserve the global `promptkit.profile` all-report override. Do not add a - second override mechanism or report-specific configuration fields. -3. Preserve batch preflight deduplication by selected effective profile ID: - Today and Tomorrow in the same batch should inspect their shared - `weather-balanced` selection once. -4. Preserve the selected logical profile ID and resolved backend/model through - active inspection, preparation, and execution using the existing - project-owned contract. Do not add state fields, expand persisted parameter - detail, or create a new historical compatibility guarantee. - -### Tests - -- Add or update representative app tests for default Hourly and day-scale - selection, a global-profile override, and a morning/evening batch sharing - `weather-balanced`. -- Assert inspection completes before weather collection and provider - generation, including malformed same-ID operator overrides. -- Assert active inspection and execution expose the logical profile ID and - effective model for both embedded and overridden profiles. -- Assert endpoints and credentials remain absent from errors, summaries, - normal logs, and ordinary state. -- Use project-owned executor fakes or Promptkit provider fakes; do not make live - provider calls. -- Run: - - ```sh - go test ./internal/app ./internal/cli - git diff --check - ``` - -### Exit Gate - -Single-report and batch workflows select the intended tier, retain existing -override and preflight behavior, deduplicate shared batch inspection, and -preserve safe logical and effective model information during active execution -without adding a durable-provenance contract. - -## Stage 5: Publish Canonical Operator And Maintainer Documentation - -### Goal - -Document the implemented feature once in each canonical owner and provide one -maintained, copyable local override example. - -### Work - -1. Update `docs/config.md` to explain global profile selection versus - `profile_file`/`profile_dir` definition lookup and link to the maintained - example. Keep the field reference in this canonical document. -2. Update the Promptkit integration document with the logical profile catalog, - source precedence, exact prompt-version relationship, and safe active - inspection and execution contract. Avoid restating complete configuration - syntax or presenting transitional persistence as the target architecture. -3. Update the report-registry, Promptkit adapter, app-orchestration, and state - internal documents only where their implemented contracts changed. -4. Update `docs/operations.md` with the normal local-override workflow and - `docs/troubleshooting.md` with malformed override, unavailable local - endpoint, missing credential, and unexpected effective-model diagnostics. -5. Add or update one secret-free file under `examples/` showing a - `weather-light` override for a local OpenAI-compatible endpoint. Choose one - supported form as the complete example and mention the other form only in - its canonical reference. -6. Update the architecture policy only if implementation changed a normative - boundary or invariant. Do not add future behavior to current-state docs. -7. Keep the feature roadmap and this plan in their pre-implementation statuses - until the final repository gate passes. Do not create release notes before - a release version is chosen. - -### Tests - -- Verify every changed repository-relative link and every profile/model ID. -- Validate maintained YAML examples through the same strict configuration or - Promptkit profile path used by production where practical. -- Run the focused tests that own any executable examples, followed by: - - ```sh - git diff --check - ``` - -### Exit Gate - -Users, operators, and maintainers can discover the tier defaults, precedence, -global override, local override, and failure behavior without duplicated or -future-state documentation. - -## Stage 6: Complete Repository Verification And Roadmap Handoff - -### Goal - -Demonstrate that the complete feature is coherent, offline-testable, and ready -for review and a later release decision. - -### Work - -1. Review the complete diff against the roadmap, this plan, and all three - policy documents. Remove stale identifiers, temporary helpers, redundant - tests, and documentation duplication. -2. Confirm `go.mod` and `go.sum` retain tagged Promptkit v0.5.0 without a local - replacement or dependency drift. -3. Confirm only the four supported report products exist and no retired report - surfaces were reintroduced. -4. Confirm the roadmap's completion criteria one by one. Change its status to - implemented and this plan's status to completed only after every criterion - and command below passes. -5. Do not require a live provider for completion. If credentials and network - access are deliberately supplied by an operator, record live smoke results - separately as release-candidate evidence rather than adding them to the - default suite. - -### Verification - -Run `gofmt -w` on every changed Go file, then run: - -```sh -go test ./... -go test -race ./... -go run ./cmd/weatherreporter --help -git diff --check -git status --short -``` - -Also inspect all three logical profiles through the application's normal -preflight path using offline provider doubles, including one same-ID local -override and one explicit global override. - -### Exit Gate - -All roadmap completion criteria are satisfied, all verification commands pass, -the working tree contains only intentional changes, and the canonical -documentation describes the implemented state. The feature is ready for code -review and release preparation. - -## Post-Implementation Review - -Stages 1–6 implemented the intended production behavior and passed their -offline verification gates. A subsequent review found no high-severity runtime -defect, but identified three test-quality issues and one remaining validation -obligation: - -- one app test asserted durable preparation and execution artifact provenance, - contrary to the active-execution boundary and accepted ephemeral-state - direction; -- an adapter-package test depended upward on app orchestration and duplicated - test ownership; -- embedded fallback profiles were inspected but not exercised through one - prepared execution with a provider fake; and -- the roadmap's representative model-evaluation policy had no recorded - evidence. - -Stages 7 and 8 address those findings without changing the profile catalog, -selection precedence, report assignments, prompt content, generated-text -schemas, or default offline test contract. - -## Stage 7: Correct Test Ownership And Fallback Execution Coverage - -### Goal - -Remove accidental durable-state and cross-layer test commitments while adding -one focused offline execution test for the embedded fallback path. - -### Work - -1. Rewrite `TestGenerateDetailedPreservesSelectedProfileThroughExecution` so - it protects active workflow behavior only: - - - retain the Hourly default, day-scale default, and global-override cases; - - assert the profile ID sent in `promptexec.ExecuteRequest`; - - have the executor fake record the preparation and execution values it - emits, then assert their logical profile ID and effective backend/model; - - do not load preparation, execution, or metadata files to establish a - durable profile-provenance contract; and - - remove artifact-content scans whose fake inputs cannot contain an endpoint - or credential. - -2. Preserve meaningful safety coverage at the boundary that can expose the - sensitive value: - - - retain adapter mapping coverage proving an endpoint from a real Promptkit - profile does not enter `promptexec.ProfileInspection`; - - retain app error coverage proving dependency errors containing an endpoint - or credential are replaced by a bounded classified error; and - - do not add profile endpoints or credentials to project-owned execution - types merely to make a leakage test possible. - -3. Remove `internal/app`, app configuration, and report-registry dependencies - from `internal/adapters/promptkit/adapter_test.go`. Move the assembled - application-preflight test to a new app-owned external integration test, - such as `internal/app/prompt_profile_integration_test.go` with package - `app_test`: - - - construct the real Promptkit adapter through its public `New` function; - - call the public app prompt-inspection operation; - - supply a deterministic credential lookup rather than reading the process - environment; and - - cover Hourly, one representative day-scale default, the explicit - `weather-deep` global override, and a same-ID endpoint-only - `weather-light` override. The asset contract tests already own the exact - mapping for all four prompts, so the integration test need not repeat all - four. - -4. Add one adapter-owned, offline fake-client execution test using the real - embedded Hourly prompt at `1.1.0` and selected profile `weather-light`. - Execute through the normal prepared adapter path and assert: - - - the preparation callback runs before the fake provider; - - preparation and execution report logical profile `weather-light`, backend - `openrouter`, and model `deepseek/deepseek-v4-flash`; - - the fake provider request targets `deepseek/deepseek-v4-flash`; and - - schema validation completes without contacting a live service. - - One execution case is sufficient because Promptkit owns uniform source - precedence and the adapter's inspection tests already cover fallback, - operator file, operator directory, built-in, and explicit in-memory layers. - -5. Reconcile the profile-related current-state documentation: - - - it may accurately describe fields present in current preparation and - execution receipts; - - it must not promise cross-version readability or characterize those - receipts as the profile feature's durable target architecture; and - - troubleshooting should prefer active command errors and explicit secure - debug capture, mentioning current-version receipts only as transitional - state if they remain useful before the ephemeral-state refactor. - -6. Do not change production profile resolution, prompt definitions, state - schemas, artifact validators, or the ephemeral-state roadmap in this stage. - -### Tests - -Run: - -```sh -go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app ./internal/cli -go test -count=1 -race ./internal/promptassets ./internal/adapters/promptkit ./internal/app ./internal/cli -go test -count=1 ./... -go vet ./... -go run ./cmd/weatherreporter --help -git diff --check -``` - -Review the changed tests against the testing policy and confirm that adapter -tests own adapter behavior, app tests own orchestration, and state tests remain -the sole owner of durable artifact format and validation details. - -### Exit Gate - -Active profile selection and effective-model propagation remain protected -without adding a durable-provenance commitment; the adapter test package no -longer imports the app layer; one embedded fallback profile completes prepared -execution through a provider fake; and every required check passes offline. - -## Stage 8: Evaluate The Initial Model Ladder - -### Goal - -Produce explicit release-candidate evidence that the selected models are -acceptable for their intended report tiers and that a representative local -override provides the promised operator experience. - -This is an opt-in evaluation stage, not an ordinary automated-test stage. It -requires operator-approved provider credentials, network access, and a local -OpenAI-compatible endpoint. Do not mark it complete when those prerequisites -are unavailable; report the missing prerequisite instead. - -### Corpus - -Use four representative, secret-free YAML data packages: one each for Daily, -Today, Tomorrow, and Hourly. The set must include at least one package with -precipitation windows and at least one with none. Remove precise private -location identifiers or other operationally sensitive values without changing -the meteorological relationships being evaluated. - -Record a SHA-256 hash and a short, non-sensitive description for each package. -Do not commit full packages or generated prose unless the user separately -approves them as repository fixtures. - -### Execution Matrix - -Run these six evaluations from the exact package bytes: - -| Case | Package | Profile | -| --- | --- | --- | -| Hourly default | Hourly | `weather-light` | -| Daily default | Daily | `weather-balanced` | -| Today default | Today | `weather-balanced` | -| Tomorrow default | Tomorrow | `weather-balanced` | -| Deep comparison | The same Daily package used above | `weather-deep` | -| Local override | The same Hourly package used above | operator-defined `weather-light` endpoint profile | - -After the successful local-override case, stop or deliberately address an -unavailable test endpoint and repeat it as a negative control. Confirm that the -request fails visibly and does not call or select an embedded remote profile. -This negative control is not an additional quality-evaluation case. - -Use a temporary, untracked evaluation harness beneath the module when exact -package replay is needed. It should call the existing Promptkit adapter and -project-owned execution contract rather than duplicate prompt loading, -rendering, or schema validation. Remove the harness and all unapproved raw -outputs before completing the stage. Never print or record credentials. - -### Evaluation Record - -Add a concise `## Evaluation Record` section to -`docs/roadmap/domain-profiles.md`. For every case, record: - -- evaluation date, logical profile, effective backend, and exact model - reported by execution; -- corpus hash, validation outcome, latency, prompt/completion/total token use, - and provider-reported or contemporaneously calculated cost; -- whether every generated claim is supported by the deterministic package; -- whether hazards, periods, uncertainty, and precipitation timing are used - correctly; -- whether `precipitation_timing` is exactly an empty string for the no-window - case; -- a short usefulness assessment for summary and forecast discussion; and -- any provider, alias, or local-endpoint caveat observed. - -Do not include credentials, endpoints, complete effective parameter maps, -full data packages, rendered prompts, or full generated responses in the -record. The secure debug directory may be used temporarily for operator review -and remains operator-managed. - -### Acceptance Rules - -- Every case must complete strict JSON Schema validation without repair. -- Generated prose must contain no material unsupported weather claim or - contradiction of deterministic hazards, periods, or uncertainty. -- Precipitation timing must agree with the deterministic windows and use the - required empty-string representation when no window exists. -- The local override must select the operator model without modifying a prompt - or application code and must not fall back to a remote profile when the local - endpoint is unavailable. -- Latency, tokens, and cost must be recorded, but this initial evaluation does - not impose an invented numeric threshold. The operator decides whether the - observed tradeoff remains acceptable for the named tier. -- If a default case fails schema or factual acceptance, do not weaken the - schema or prompt to accommodate the model. Reopen the concrete model or - profile-setting decision in the feature roadmap and leave this stage - incomplete. - -### Verification - -After removing temporary evaluation material, run: - -```sh -go test -count=1 ./... -git diff --check -git status --short -``` - -Confirm that the only intended repository change from this stage is the -concise evaluation record and any roadmap status correction required by its -result. Do not add live credentials, provider-dependent tests, a permanent -benchmark framework, or release notes before a release version is selected. - -### Exit Gate - -All six cases satisfy the acceptance rules, the roadmap contains concise and -safe evaluation evidence, no temporary corpus or response material remains in -the repository, and the default suite remains offline. Set this plan back to -`Status: Completed` only after both Stages 7 and 8 have passed. - -## Open Questions - -None. The model identifiers, profile settings, report assignments, version -transition, precedence, compatibility behavior, test boundaries, and -documentation ownership are decision-complete. diff --git a/docs/roadmap/promptkit-fallback-profiles-feature-request.md b/docs/roadmap/promptkit-fallback-profiles-feature-request.md deleted file mode 100644 index 9181395..0000000 --- a/docs/roadmap/promptkit-fallback-profiles-feature-request.md +++ /dev/null @@ -1,169 +0,0 @@ -# Promptkit Feature Request: Application Fallback Profiles - -Status: Implemented upstream in Promptkit v0.5.0. - -Promptkit v0.5.0 resolved this request with the public -`WithFallbackProfileFS` engine option and the precedence and error semantics -specified below. This document is retained as the downstream rationale for -the capability. - -## Purpose - -Promptkit should allow a consuming application to supply an embedded fallback -profile source that sits below operator-configured profiles and above -Promptkit's own built-in profile catalog. - -This capability would let an application publish stable, domain-specific -profile IDs with useful defaults while preserving Promptkit's existing -operator-override behavior. The capability must remain application-neutral; -Promptkit should provide the source layer but should not own downstream profile -names, model assignments, or configuration policy. - -## Downstream Use Case - -Weatherreporter wants to embed profiles such as `weather-light`, -`weather-balanced`, and `weather-deep`. Report prompts would select those -logical profiles instead of naming provider- or model-specific Promptkit -profiles directly. - -An installation could then place a profile with the same ID in its configured -profile directory. For example, a local `weather-light` definition could point -to an OpenAI-compatible endpoint on the deployment network. When no operator -definition exists, Weatherreporter's embedded definition would keep the -application usable without additional profile files. - -This pattern is useful beyond Weatherreporter. Any Promptkit consumer may want -application-owned execution tiers or workload-specific defaults without -adding domain-specific profiles to Promptkit's general built-in catalog. - -## Current Constraint - -Promptkit currently resolves matching profile IDs in this order: - -1. in-memory profiles supplied through `WithProfiles`; -2. one configured profile file, `fs.FS`, or directory source; and -3. Promptkit's embedded built-in profiles. - -These layers do not express the desired application-default relationship: - -- `WithProfiles` has higher precedence than the configured source, so it would - prevent an operator file from overriding an application profile with the - same ID. -- `WithProfileFS` can hold embedded application assets, but it occupies the - configured-source layer and therefore replaces rather than sits beneath a - configured profile directory or file. -- adding downstream profile IDs to Promptkit's built-in catalog would make the - library own application-specific policy. - -A downstream application could build its own filesystem overlay, but that -would duplicate Promptkit's profile discovery, error, and precedence behavior -at the consumer boundary. - -## Requested Capability - -Add one optional application fallback profile source to engine construction. -When present, matching profile IDs should resolve in this order: - -1. in-memory profiles supplied through `WithProfiles`; -2. the ordinary configured profile source selected through a profile option or - `Config.ProfileDir`; -3. the application fallback profile source; and -4. Promptkit's embedded built-in profiles. - -When no application fallback is configured, existing source precedence and -behavior must remain unchanged. - -The minimum useful public surface is an `fs.FS`-backed option because consumers -can embed YAML profile assets. A possible API shape is: - -```go -promptkit.WithFallbackProfileFS(profileFS, ".") -``` - -The name is illustrative rather than prescriptive. A companion option for -validated `Profile` values could be added if Promptkit maintainers find it -generally useful, but it is not required for the Weatherreporter use case. - -## Required Semantics - -- A higher-precedence source falls through only when the requested profile ID - is absent. -- A malformed, unreadable, duplicate, ambiguous, or otherwise invalid matching - profile is an error and must not silently fall through. -- The fallback source uses the existing strict profile YAML format and profile - validation rules. -- Profile values are selected as a whole. This feature does not merge, - inherit, or partially overlay profile definitions. -- `InspectProfile`, `Prepare`, prepared execution, and ordinary execution use - the same profile-source precedence. -- An explicit request profile continues to take precedence over a prompt's - `default_profile`; this request concerns definition lookup after the profile - ID has been selected. -- Repeated fallback-source options should follow Promptkit's documented - same-category option convention, normally with the last value replacing the - earlier value. -- A canceled lookup, invalid fallback asset, or unknown resolved backend should - continue to cross the public facade through Promptkit's existing public error - identities. -- Exact profile inspection must remain side-effect free and must not contact a - model provider. - -## Application And Library Boundaries - -Promptkit should own: - -- the additional repository layer; -- deterministic lookup and fallthrough behavior; -- validation of the supplied source through the existing profile contract; -- consistent use of the layer across inspection and execution; and -- public documentation and tests for the added precedence rule. - -The consuming application should continue to own: - -- whether it supplies fallback profiles; -- the profile IDs and their domain meaning; -- embedded profile contents and model choices; -- application configuration and override policy; -- report- or workload-to-profile assignment; and -- credential checks and operator-facing errors beyond Promptkit's public - contract. - -## Non-Goals - -This request does not ask Promptkit to add: - -- Weatherreporter-specific profile IDs to its built-in catalog; -- profile inheritance, aliases, or field-level merging; -- automatic endpoint discovery or availability probing; -- provider failover or fallback from a failed selected profile; -- per-request model benchmarking or tier selection; -- application configuration discovery; or -- eager validation of every profile in every source. - -## Compatibility - -The feature can be additive. Engines that do not configure an application -fallback source should retain their current public behavior and precedence. -Existing uses of `WithProfiles`, `WithProfileFile`, `WithProfileFS`, and -`Config.ProfileDir` should not change meaning. - -The application fallback is deliberately lower precedence than every existing -consumer-configured source. This preserves the established expectation that a -custom profile definition can override a packaged default with the same ID. - -## Acceptance Criteria - -The capability is sufficient for downstream adoption when Promptkit can -demonstrate that: - -- a fallback-only profile can be inspected and used for preparation and - execution; -- a configured directory, file, or `fs.FS` profile with the same ID overrides - the fallback profile; -- an absent configured profile falls through to the application fallback; -- an invalid configured match fails instead of falling through; -- an absent application fallback profile continues to resolve from Promptkit's - built-in catalog; -- `WithProfiles` retains highest precedence; -- behavior is identical across inspection, preparation, and execution; and -- omitting the new option preserves existing tests and public contracts. diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md deleted file mode 100644 index 66a45b5..0000000 --- a/docs/roadmap/promptkit.md +++ /dev/null @@ -1,516 +0,0 @@ -# Promptkit Migration Roadmap - -Status: Completed roadmap record. - -## Purpose - -This roadmap records the scope, decisions, and completed outcome of replacing -the external Scriptorium CLI integration with Promptkit. Canonical -documentation outside `docs/roadmap/` owns the implemented behavior. - -## Pre-Migration Baseline - -Status: Historical migration input. - -Before the migration, Weatherreporter exposed seven report definitions, but -only four had complete prompt-backed report implementations: - -- Daily Report: `weather.daily_generated_text` -- Today Report: `weather.today_generated_text` -- Tomorrow Report: `weather.tomorrow_generated_text` -- Hourly Report: `weather.hourly_generated_text` - -The three-day, weekend, and storm commands and registry definitions had no -corresponding Scriptorium prompt or schema and never formed complete -operational report products. The `weather.daily_report` Scriptorium prompt was -legacy source material and was not selected by the registry. - -The Scriptorium source corpus was retained temporarily under -`docs/roadmap/scriptorium/` as migration input. It contained the four -operational generated-text prompt definitions, their referenced content, -private response schemas, shared instructions, and the unused legacy Daily -Markdown prompt. The temporary corpus was removed after the runtime assets -were reconciled and embedded. - -## Implemented End State - -Status: Completed. - -Weatherreporter pins -`gitea.maximumdirect.net/eric/promptkit` at `v0.4.0` and uses it as the -in-process engine for prompt inspection, prepared execution, provider calls, -and first-pass output validation. - -The `scriptorium` executable, subprocess adapter, configuration, runtime -dependency, direct-Markdown execution path, and integration documentation have -been removed. The four operational reports continue to use structured -generated text followed by weatherreporter-owned validation and Markdown -templates. - -The unfinished three-day, weekend, and storm reports are not implemented as -part of this migration. Their incomplete CLI, registry, documentation, and -generation declarations are removed from the implemented surface before the -migration is considered complete. Any future implementation of those products -requires separate roadmap scope, prompt and schema design, tests, and -documentation. - -Weather selection, forecast derivation, valid periods, module construction, -Recent Changes, generated-text interpretation, Markdown templates, durable -state, inspection, output copies, and Distributor notification remain owned by -weatherreporter. - -The four report prompts and private response schemas are versioned embedded -application assets. Operators configure Promptkit profiles without replacing -the report-owned corpus. One Promptkit engine is constructed per CLI -invocation and shared by every report in that invocation, including all -reports in a morning or evening batch. - -Promptkit is isolated behind a weatherreporter-owned execution contract. -Promptkit request, result, validation, error, profile, backend, and provider -types do not leak into application orchestration, report definitions, domain -packages, CLI summaries, durable state contracts, or Distributor behavior. - -## Goals - -Status: Completed migration outcomes. - -- Removed the Scriptorium runtime dependency and subprocess boundary. -- Migrated the four operational report prompts to Promptkit `v0.4.0`. -- Used prepared execution to persist preparation provenance before provider work - while executing the exact frozen snapshot. -- Validated report prompt and profile selections before weather collection when - the required information is available. -- Preserved deterministic module snapshots and structured Recent Changes. -- Preserved generated-text domain validation and repository-owned Markdown - rendering. -- Preserved context cancellation, actionable errors, secret redaction, and - inspectable failures. -- Improved durable prompt provenance with prompt, input, profile, model, - validation, usage, and timing metadata. -- Kept content-rich prompt and response diagnostics separate from routine - metadata and CLI output. -- Kept tests offline and deterministic through injected Promptkit model - clients and fixtures. -- Removed incomplete report declarations from the implemented product surface - rather than creating new report products during an integration migration. - -## Non-Goals - -Status: Completed migration constraints. - -The completed migration did not: - -- create prompts, schemas, templates, or completed products for three-day, - weekend, or storm reports; -- preserve the unused `weather.daily_report` legacy Markdown prompt as an - active runtime asset; -- preserve a direct-Markdown LLM generation mode; -- move meteorological selection, derivation, thresholds, or comparison logic - into prompts or Promptkit; -- send raw unbounded Weather API responses to the model; -- replace weatherreporter's generated-text domain validation or Markdown - template rendering; -- add a general workflow engine, provider plugin system, or arbitrary backend - registry; -- add automatic provider, validation, repair, or capacity retries; -- add concurrent report generation to the sequential batch workflow; -- expose Promptkit types as weatherreporter contracts; -- keep a production-selectable Scriptorium/Promptkit dual-run mode; -- require Promptkit eager source validation, structured generation errors, or - semantic execution-target fingerprints; or -- use an unpublished Promptkit commit, committed Go workspace, or committed - local module replacement. - -## Locked Decisions - -Status: Implemented migration decisions. - -### Dependency And Upgrade Boundary - -- The migration pins the tagged Promptkit `v0.4.0` release. -- Coordinated local development may temporarily use the sibling Promptkit - checkout, but committed module metadata must reference the tagged release. -- The adapter relies on the public root Promptkit package only. -- A future Promptkit upgrade requires explicit review of prepared-execution - lifecycle, prompt and profile inspection, prompt/profile/schema formats, - error identities, validation behavior, capacity behavior, and the outbound - provider contract. -- Promptkit's deferred eager source validation, structured generation errors, - and semantic execution-target fingerprints do not block this migration. - -### Operational Report Scope - -- The migration preserves these prompt IDs: - `weather.daily_generated_text`, `weather.today_generated_text`, - `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`. -- Each operational report definition selects the exact embedded prompt version - `1.0.0`; execution does not rely on ambiguous single-version lookup. -- Morning and evening batch membership remains based on Today, Tomorrow, and - eligible future Daily reports. -- Three-day, weekend, and storm are removed from current CLI help, parsing, - report registry membership, tests that claim implemented generation, and - non-roadmap documentation. -- The future product concepts may remain under `docs/roadmap/`, but migration - verification does not invent outputs or compare nonexistent prompts. - -### Application Boundary - -- Promptkit remains an adapter boundary even though it runs in process. -- A weatherreporter-owned contract represents prompt identity, preparation, - execution, output, validation, usage, provenance, and neutral error - categories. -- The Promptkit adapter maps public Promptkit values into that contract. -- App orchestration and test fakes depend on the project-owned contract, not - Promptkit. -- Scriptorium-specific request, result, error, and generation-mode types are - removed rather than renamed and retained. - -### Prompt And Schema Ownership - -- Weatherreporter embeds the four operational prompt definitions, referenced - prompt content, shared prompt content, and private response schemas. -- Assets remain separate files rather than inline Go strings. -- The temporary corpus under `docs/roadmap/scriptorium/` is migration source - material, not the final runtime location. -- Weatherreporter's existing generated-text domain types, schemas, and - templates remain the canonical application contract. Imported Scriptorium - assets are reconciled with that contract rather than copied blindly or kept - as duplicate runtime schemas. -- The imported Daily schema's incorrect Today `$id` and title are corrected. -- `confidence` is handled consistently across each prompt, provider-facing - schema, generated-text domain type, and template. The existing optional - weatherreporter field remains supported unless a separate domain decision - removes it. -- Prompt input metadata identifies the serialized data package as YAML rather - than JSON. -- Imported `pipeline-weather/...` schema paths are replaced with paths valid - inside the embedded Promptkit schema source. -- Imported `repair_attempts: 2` values are removed or set to zero. The - migration does not rely on Promptkit's internal-only repair capability. -- The unused `weather.daily_report` prompt is not promoted into runtime assets. -- One centralized embedded prompt/schema source is sufficient; Weatherreporter - does not need Notarius's multi-module asset-flattening registry. - -### Profiles, Backends, And Credentials - -- Execution profiles remain operator-configurable rather than embedded report - policy. -- Each embedded operational prompt declares Promptkit's built-in - `gemini-flash-latest` profile as its default. -- `gemini-flash-latest` is intentionally a moving model alias. The execution - record captures the effective model identity, but operators who require a - pinned model must select an explicit external profile. -- Configuration supports at most one external profile source: - `promptkit.profile_file` or `promptkit.profile_dir`. The two fields are - mutually exclusive. -- A nonblank `promptkit.profile` is the explicit request profile for every - report in the invocation and takes precedence over each prompt's - `default_profile`. A blank value uses the prompt default. -- Promptkit's normal profile-source precedence remains intact: an external - matching profile takes precedence over an embedded built-in profile, and an - invalid matching external profile is an error rather than a reason to fall - back. -- Weatherreporter exposes Promptkit's conventional `local` backend through the - narrow `promptkit.local.endpoint` and - `promptkit.local.concurrency_limit` configuration fields. It does not expose - arbitrary backend registration. -- A configured local endpoint registers the engine-scoped `local` backend. An - operator-supplied external profile selects it with `backend: local` and owns - the model-specific settings; Weatherreporter does not invent a local model - profile. -- Local concurrency defaults to one. A value of zero means unlimited, matching - Promptkit, and a negative value is invalid. Queue capacity and general - backend parameters are not exposed. -- Credential values remain in environment variables or file-backed - environment secrets. Configuration contains only credential source names. -- Provider credentials never appear in logs, errors, CLI output, durable - metadata, preparation records, execution records, or debug summaries. -- Promptkit `InspectProfile` reports structural target and credential - requirements; Weatherreporter owns policy for checking configured - environment availability. -- Promptkit revalidates environment credentials at `RunPrepared`; a successful - preparation does not promise that execution-time credentials remain - available. - -### Configuration Contract - -The replacement configuration surface is: - -```yaml -promptkit: - profile: "" - profile_file: "" - profile_dir: "" - timeout: 2m - - local: - endpoint: "" - concurrency_limit: 1 -``` - -- `timeout` remains the transport-wide provider-call safety cap. -- A blank local endpoint leaves the conventional local backend unregistered. -- Scriptorium's `binary`, `config_path`, and `extra_args` settings have no - Promptkit equivalents and are removed. -- Configuration validation rejects simultaneous `profile_file` and - `profile_dir` values, invalid local endpoints, negative concurrency, and - selected profiles that cannot resolve their backend. - -### Engine Construction And Inspection - -- One Promptkit engine is constructed per CLI invocation at the application - composition boundary. -- Single-report generation and every report in a batch use that same engine. -- Per-report orchestration does not construct a default engine. -- Promptkit backend capacity state and HTTP transport are shared consistently - for the invocation. -- Before collection, `InspectPrompt` checks every selected report's exact ID - and version, declared `data_package` input, default-profile metadata, prompt - hash availability, and declared output contract. -- `InspectPrompt` is a point-in-time structural check. It does not load a JSON - Schema, resolve a profile, or freeze later execution. -- Explicit profile overrides and relevant prompt defaults are checked with - `InspectProfile` before collection when application policy requires them. -- `InspectProfile` is also point-in-time and does not check credential values. -- Successful `PrepareExecution`, not inspection, is the per-run authority for - loaded schema, rendered content, frozen inputs, effective settings, and - durable execution provenance. - -### Prompt Input - -- Promptkit receives only the curated `data_package` produced by - `internal/promptinput`. -- Weatherreporter serializes the package once, atomically persists those exact - bytes, and supplies the same bytes with a Promptkit inline artifact. -- The managed data-package path may be supplied as non-secret provenance - through the inline artifact URI. -- Weatherreporter does not delegate unrestricted path loading to Promptkit's - default file artifact reader. -- Prompt inspection and adapter tests verify that `data_package` is required - and declared with the chosen YAML media type. - -### Prepared Execution - -- `Engine.PrepareExecution` replaces Scriptorium render preflight. -- Weatherreporter obtains `PreparedExecution.Details`, maps a safe subset into - its own preparation record, and persists that record before calling - `Engine.RunPrepared`. -- `RunPrepared` executes the frozen prompt, profile, schema, inputs, rendered - messages, target, and validation resources retained by the handle. -- Every acquired handle is followed immediately by `defer handle.Discard()`. - Discard is safe after execution and releases unused private execution state. -- Handles remain adapter-local, engine-bound, one-shot, in-process values. - They are never serialized, persisted, copied into app contracts, or treated - as restartable jobs. -- Preparation and execution use independent contexts. Execution receives the - active report workflow context. -- Capacity is not reserved during preparation. Capacity rejection can - therefore occur after a preparation record has been persisted. -- `RunPrepared` consumes the handle on success and every operational failure. -- Preparation details remain available from the adapter after execution or - discard, but rendered messages are not copied into routine durable state. -- Promptkit execution timing excludes preparation and consumer-held delay. - Weatherreporter records preparation timing and execution timing separately. - -### Execution And Validation - -- All four operational reports use Promptkit JSON Schema output validation. -- A completed Promptkit validation rejection returns a `RunResult`; the - adapter retains raw output and bounded validation details before failing the - report. -- An operational generation or validation error returns no partial - `RunResult`. -- Weatherreporter's `internal/generatedtext` validation remains the final - report-specific decode and domain boundary. -- Weatherreporter's `internal/reporttemplate` remains responsible for managed - Markdown rendering. -- Weatherreporter atomically persists Promptkit raw output and later artifacts - rather than asking Promptkit to choose managed filesystem paths. -- No Promptkit output-repair behavior is assumed or requested. - -## Durable Artifacts And Observability - -Status: Implemented design constraints. - -Routine durable state retains useful non-secret provenance without persisting -full rendered prompts. - -The preparation record contains: - -- prompt ID and exact version; -- prompt definition hash; -- rendered prompt hash; -- input hashes; -- selected profile and backend identity; -- effective model identity; -- output contract summary; -- preparation start, end, and duration; and -- the path of the exact persisted data package. - -The execution record and run metadata contain, when available: - -- Promptkit run ID; -- prompt ID, version, and hashes; -- input hashes; -- selected profile, backend, and model identity; -- generated-content hash; -- token usage; -- execution start, end, and duration; -- validation status and bounded diagnostics; and -- paths of separately persisted raw output, normalized generated text, render - context, managed Markdown, and other artifacts reached by the workflow. - -Provider endpoints, full effective model parameter maps, rendered messages, -schema bodies, data-package contents, and generated content do not belong in -routine metadata or CLI summaries. - -Rendered messages and other content-rich preparation or response diagnostics -are available only when the operator supplies -`--llm-debug-dir ` to a single-report or batch command. - -- There is no persistent YAML setting for debug capture. -- The debug root is validated or created before weather collection or provider - work. A requested destination that cannot be secured or written is an error. -- Artifacts are grouped beneath - `////`. -- Directories and files use owner-only permissions and atomic writes. -- Debug artifacts may contain rendered messages and content-rich preparation - or response diagnostics, but never credentials. -- The debug path appears in command output only when debug capture is enabled; - it is not added to routine durable metadata. -- Debug artifacts are not cache or comparison inputs. Their retention is owned - by the operator who selected the directory. - -### Artifact Identities And Versions - -Weatherreporter replaces Scriptorium-specific artifact identities rather than -reusing names whose meanings have changed: - -- `PromptPreparationArtifact` uses schema version - `weatherreporter.prompt_preparation.v1`, is written as - `prompt_preparation..json`, and is referenced by - `preparationPath`. -- `PromptExecutionArtifact` uses schema version - `weatherreporter.prompt_execution.v1`, is written as - `prompt_execution..json`, and is referenced by `executionPath`. -- Run metadata advances to `weatherreporter.metadata.v2` and uses those new - path fields. - -Preparation files remain beneath the existing configurable `preflight/` -directory, and execution files remain beneath the existing `snapshots/` tree. -The stable physical grouping limits deployment disruption without preserving -misleading Scriptorium-era filenames or field names. Raw generated output, -normalized generated text, render context, managed Markdown, and other -artifacts whose meanings have not changed retain their existing names and -locations. - -Run inspection remains able to read `weatherreporter.metadata.v1` and its -legacy `preflightPath` and `generatedTextResultPath` references. New runs write -only the v2 metadata and new artifact names; Weatherreporter does not -dual-write deprecated aliases. CLI summary fields adopt `preparationPath` and -`executionPath` as an explicit, documented contract change. - -## Failure Contract - -Status: Implemented design constraints. - -- A preparation failure produces a redacted weatherreporter-owned failure - receipt with report, RunID, prompt, stage, timing, and classified error - context. It does not fabricate Promptkit preparation details. -- An operational execution failure retains the successful preparation record - and adds a redacted execution failure receipt. No partial Promptkit result or - model output is invented. -- A Promptkit validation rejection retains the returned result, raw generated - output, validation details, and safe provenance before the report fails. -- A later generated-text decode, domain-validation, or template failure - retains every raw and validated artifact reached before that stage. -- Caller cancellation takes precedence when the active workflow context is - canceled. -- `promptkit.CapacityError` is recognized with `errors.As`; its backend ID is - copied into a weatherreporter-owned capacity error while - `ErrCapacityExceeded` remains the classification. -- Capacity rejection is an operational report failure, not invalid model - output, and does not trigger an automatic retry. -- Other Promptkit public error identities are translated into the narrow - weatherreporter error categories needed by CLI, metadata, and batch - behavior. Diagnostic prose is not parsed as a contract. -- Single-report commands return the classified failure with available - inspectable paths. -- Batch runs continue independent later reports under the existing batch - failure policy. -- Any future retry policy belongs to app orchestration, not the adapter. - -## Compatibility Requirements - -Status: Implemented design constraints. - -- Daily, Today, Tomorrow, and Hourly report IDs, prompt IDs, valid periods, - artifact grouping, output names, and Distributor bundle behavior remain - stable. -- Morning and evening batch collection, planning, ordering, and continuation - behavior remains stable. -- Module snapshot and Recent Changes behavior remains deterministic. -- Promptkit receives only the existing curated prompt-input boundary. -- Managed Markdown remains the Distributor upload source. -- RunID lookup and inspection remain available for successful and failed runs. -- Existing managed paths remain stable where their meaning is unchanged. - Scriptorium-specific artifact names or schemas change when retaining them - would misrepresent the Promptkit contract. -- Existing v1 run metadata and referenced artifacts remain inspectable after - the migration. New runs use the v2 metadata and Promptkit-era artifact - identities without dual-writing deprecated aliases. -- Artifact or metadata schema changes are explicit, documented, and covered by - state and inspection tests. -- Prompt or generated content is not added to routine logs or CLI summaries. -- Tests do not require live providers or credentials. -- Removing incomplete three-day, weekend, and storm surfaces is documented as - correction of an unfinished product boundary, not as successful Promptkit - migration of those reports. - -## Verification And Completion Criteria - -Status: Completed and verified. - -Completion was verified by the following outcomes: - -- the four operational reports inspect, prepare, and execute through Promptkit - `v0.4.0` using embedded report-owned assets; -- every report uses exact prompt version `1.0.0`, requires the YAML - `data_package`, and declares the expected JSON Schema output contract; -- prepared execution persists a safe preparation record before provider work - and executes the same frozen snapshot; -- deterministic offline adapter and app tests cover success, preparation - failure, credential revalidation, capacity rejection, cancellation, timeout, - generation failure, Promptkit validation rejection, generated-text domain - failure, template failure, and handle discard; -- morning and evening batches construct one engine and preserve current - collection, planning, ordering, continuation, output, and notification - behavior; -- the temporary corpus has been reconciled into one runtime prompt/schema - source without duplicate provider-facing schemas; -- configuration examples load and contain no Scriptorium fields; -- CLI summaries and inspection commands expose the new project-owned artifact - contract without Promptkit types; -- Scriptorium code, configuration, tests, and runtime documentation have been - removed; -- incomplete three-day, weekend, and storm commands, registry entries, tests, - and current-behavior documentation have been removed or moved to roadmap - scope; -- non-roadmap documentation describes only the implemented Promptkit - integration; -- `go test ./...`, required focused or race-enabled checks, CLI help - validation, and `git diff --check` pass; and -- no committed `go.work`, local `replace`, live-provider test, or - secret-bearing fixture remains. - -Fixture-based comparison with prior Scriptorium behavior is sufficient. -Production dual-run is not required because model calls are nondeterministic, -costly, and difficult to compare meaningfully. - -## Decision Status - -Status: Completed. - -The roadmap has no remaining open product or architecture questions. Later -changes to this completed scope require new roadmap or decision-record scope -rather than implicit changes to this historical record. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md deleted file mode 100644 index 9b39aae..0000000 --- a/docs/troubleshooting.md +++ /dev/null @@ -1,75 +0,0 @@ -# Troubleshooting - -Start with the command's classified error. When content-rich prompt diagnostics -are needed, enable a new run with `--llm-debug-dir` and handle the resulting -secure capture as sensitive. Current-version workspace receipts can provide -additional context when present, but are transitional state rather than a -long-term troubleshooting interface. - -## Prompt inspection or credentials fail before collection - -A prompt/version, contract, selected profile, unsupported direct-key profile, -or required environment credential can fail before weather collection. Correct -the configured `promptkit` profile or profile source, confirm the exact -Promptkit asset is available, and supply any reported environment credential. -Do not add provider keys to YAML. See [configuration](config.md). - -## Local profile override is malformed or selects an unexpected model - -`promptkit.profile_file` and `promptkit.profile_dir` supply complete profile -definitions. A same-ID definition replaces the embedded profile, and a malformed -matching definition fails before collection instead of falling back. Validate -the selected profile's YAML, ID, backend or endpoint, and model. If the model -is unexpected, first check the global `promptkit.profile` selection and then -look for a same-ID definition in the configured file or directory. - -Current-version preparation and execution receipts may retain the selected -profile ID and effective backend/model, but not an endpoint or credential. -Use them only as supplemental context after the active command error or an -explicit secure debug capture. See the maintained -[local `weather-light` profile example](../examples/weather-light-local-profile.yml). - -## Local model endpoint is unavailable - -An endpoint-only `weather-light` override can pass preflight and still fail -during provider preparation or execution when the local server is unavailable -or does not accept the configured model. Start the local server, correct the -endpoint or model in the profile, and run the command again. Weatherreporter -does not probe endpoints or automatically use a remote profile instead. - -## Preparation, capacity, or execution fails - -A preparation failure occurs before provider work; an execution failure occurs -after preparation. A capacity error for one batch report does not retry that -report or prevent later independent reports. Correct the profile or backend -condition identified by the bounded command error, then create a new run. -Use explicit secure debug capture only when additional content-rich diagnostics -are necessary. See [operations](operations.md). - -## Generated text fails validation - -Raw generated output may be saved but Markdown is not rendered when the JSON -does not match the report schema. Correct the Promptkit prompt/profile behavior -or the matching schema and validator in source control; do not edit raw output -to treat it as validated. See [templates](templates.md). - -## Debug capture fails - -`--llm-debug-dir` must be an absolute secure directory outside workspace state. -A debug-write failure stops the affected report to avoid continuing without the -requested diagnostic. Repair the named path's ownership or permissions, then -rerun. Treat capture files as sensitive. See [operations](operations.md). - -## Weather, state, output, or notification fails - -Collection errors precede planning. Later filesystem, output-copy, template, -or Distributor errors retain the reached safe paths in the summary. Repair only -the reported endpoint or path, leave successful managed reports intact, and -rerun the affected report or batch. A batch notification is intentionally -skipped when any report item fails. - -## Secrets cannot be loaded - -Secret files must be regular non-symlink files directly beneath -`secrets.directory` with valid environment-variable basenames. Correct the -reported file or directory without placing secret values in YAML.