From c5ec4f83b26b2aa52011a43c2726a7851bd34ee1 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 1 Aug 2026 14:25:09 +0000 Subject: [PATCH] Document logical prompt profile configuration --- docs/config.md | 34 ++++++++++-- docs/integrations/promptkit.md | 58 ++++++++++++++++----- docs/internal/app-orchestration.md | 9 ++-- docs/internal/promptkit-adapter.md | 16 ++++-- docs/internal/report-registry.md | 17 +++--- docs/internal/state.md | 6 ++- docs/operations.md | 26 +++++++++ docs/troubleshooting.md | 22 ++++++++ examples/weather-light-local-profile.yml | 4 ++ internal/adapters/promptkit/adapter_test.go | 8 +++ 10 files changed, 168 insertions(+), 32 deletions(-) create mode 100644 examples/weather-light-local-profile.yml diff --git a/docs/config.md b/docs/config.md index 46b721e..eb84013 100644 --- a/docs/config.md +++ b/docs/config.md @@ -22,8 +22,12 @@ extra report copies for a command and do not change configuration. collection and generation configuration. - [config.yml](../examples/config.yml) is a representative production-oriented configuration using synthetic endpoints and no credentials. +- [weather-light-local-profile.yml](../examples/weather-light-local-profile.yml) + is a complete endpoint-only override for the embedded `weather-light` + profile. -Both files are loaded by the configuration test suite. +The configuration examples are loaded by the configuration test suite. The +profile example is inspected through the Promptkit adapter test suite. ## Minimal Configuration @@ -146,13 +150,35 @@ individual `generate` or `run` command when explicitly needed. | Field | Default | Rules | | --- | --- | --- | -| `profile` | empty | Optional explicit execution profile. Otherwise the prompt's declared default is used. | -| `profile_file` | empty | Optional external profile file. Cannot be combined with `profile_dir`. | -| `profile_dir` | empty | Optional external profile directory. Cannot be combined with `profile_file`. | +| `profile` | empty | Optional global profile selection for every report in one command. When empty, each exact prompt version selects its declared default. | +| `profile_file` | empty | Optional external Promptkit profile file. It cannot be combined with `profile_dir`. A same-ID profile completely replaces Weatherreporter's embedded definition. | +| `profile_dir` | empty | Optional external Promptkit profile directory. It cannot be combined with `profile_file`. A same-ID profile completely replaces Weatherreporter's embedded definition. | | `timeout` | `2m` | Must be greater than zero. | | `local.endpoint` | empty | Optional absolute URL for the conventional local backend. A blank endpoint leaves it unregistered. | | `local.concurrency_limit` | `1` | Maximum local backend concurrency. `0` is unlimited; negative values are invalid. | +`profile` selects an ID; `profile_file` and `profile_dir` supply definitions. +They are separate decisions. An explicit `profile` applies to every selected +report. Otherwise Hourly selects `weather-light`, while Daily, Today, and +Tomorrow select `weather-balanced` through their exact `1.1.0` prompt +definitions. + +Promptkit resolves a selected profile definition from a test or embedding +consumer's explicit in-memory profile, then the configured `profile_file` or +`profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's +built-in catalog. Sources provide complete definitions; fields are never +merged. A matching malformed external profile fails rather than using the +embedded definition. The [Promptkit integration guide](integrations/promptkit.md) +owns the catalog and precedence details. + +To replace the default Hourly definition with a local OpenAI-compatible +endpoint, set `profile_file` to a copy of +[weather-light-local-profile.yml](../examples/weather-light-local-profile.yml). +The example has no credential and should be edited for the local endpoint and +model before use. An alternative profile may use `backend: local`; in that +case `promptkit.local.endpoint` supplies the conventional local backend +endpoint. + ### `workspace` | Field | Default | diff --git a/docs/integrations/promptkit.md b/docs/integrations/promptkit.md index 13c820f..9b0bf2c 100644 --- a/docs/integrations/promptkit.md +++ b/docs/integrations/promptkit.md @@ -1,16 +1,49 @@ # Promptkit Integration -Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are -`weather.daily_generated_text`, `weather.today_generated_text`, -`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version -`1.0.1`. Their prompt assets and generated-text JSON Schemas are embedded by -`internal/promptassets`. +Weatherreporter uses Promptkit for all generated-text reports. The four logical +prompts are `weather.daily_generated_text`, `weather.today_generated_text`, +`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at +version `1.1.0`. Their prompt assets, generated-text JSON Schemas, and +Weatherreporter profile catalog are embedded by `internal/promptassets`. -Before collection, Weatherreporter inspects the exact prompt version, requires one required -`data_package` input with content type `application/yaml`, and requires the report's JSON -Schema output contract. It selects `promptkit.profile` when configured, otherwise the -prompt's declared default profile. Profiles that require a direct API key are unsupported; a -profile that reports `APIKeyEnv` requires a nonblank value in that environment variable. +## Logical profile catalog + +Prompt definitions select a stable Weatherreporter profile ID. The embedded +definitions currently use Promptkit's `openrouter` backend: + +| Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports | +| --- | --- | --- | --- | --- | --- | +| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` | Hourly | +| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow | +| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None | + +The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded +profiles intentionally omit endpoints, credentials, temperature, `top_p`, and +output-token limits. + +## Selection, lookup, and active execution + +Before collection, Weatherreporter inspects the exact prompt version and output +contract. A nonblank `promptkit.profile` selects one profile ID for every +report in the command; otherwise the prompt's declared default selects it. +Promptkit resolves the selected definition in this order: + +1. explicit in-memory profiles used by an embedding consumer or test; +2. the configured `profile_file` or `profile_dir`; +3. Weatherreporter's embedded fallback profiles; and +4. Promptkit's built-in catalog. + +A source falls through only when the selected ID is absent. Each source +supplies a complete definition, so profile fields are not merged. A malformed +matching operator definition is an error and does not fall back. + +Profiles that require a direct API key are unsupported; a profile that reports +`APIKeyEnv` requires a nonblank value in that environment variable. Inspection, +preparation, and execution retain the selected logical profile ID and resolved +backend and model through Weatherreporter's project-owned contract. Ordinary +errors, summaries, logs, and workspace state exclude endpoints, credentials, +rendered messages, schemas, request bodies, response bodies, and complete +parameter maps. Execution receives the already-persisted YAML package, prepares it once, and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Preparation and @@ -21,6 +54,7 @@ The generated-text schemas require `summary`, `forecast_discussion`, and `precipitation_timing`, and reject additional properties. Prompts return an empty string for `precipitation_timing` when the deterministic package contains no precipitation windows. -Prompt/profile configuration is owned by the [configuration reference](../config.md). Adapter -construction and mapping are documented in the [Promptkit adapter internals](../internal/promptkit-adapter.md). +Prompt/profile configuration and the maintained local override example are +owned by the [configuration reference](../config.md). Adapter construction and +mapping are documented in the [Promptkit adapter internals](../internal/promptkit-adapter.md). Durable metadata compatibility is described in [state internals](../internal/state.md). diff --git a/docs/internal/app-orchestration.md b/docs/internal/app-orchestration.md index 979ad6b..c0ef59f 100644 --- a/docs/internal/app-orchestration.md +++ b/docs/internal/app-orchestration.md @@ -7,7 +7,10 @@ notification ordering after the CLI has parsed arguments and loaded configuratio `GenerateDetailed` resolves one of the four report definitions, initializes an optional debug root, and inspects the exact Promptkit prompt/profile before it -collects weather or writes managed state. It then builds facts and modules, +collects weather or writes managed state. A configured global profile selects +every report in the action; otherwise the exact prompt selects its default +logical profile. Inspection keeps only the selected profile ID and effective +backend/model needed by the project-owned execution contract. It then builds facts and modules, saves the YAML data package, persists preparation metadata from the executor callback, executes the prepared prompt, saves execution provenance and raw output, validates generated text, renders Markdown, and optionally copies or @@ -25,8 +28,8 @@ persists raw output and execution provenance but does not render a report. `RunBatchDetailed` constructs a single debug writer and uses the request's single executor. Before collection it inspects Today, Tomorrow, and Daily for -morning, or Tomorrow and Daily for evening, deduplicating effective profile -inspection. It then collects once, plans eligible Daily dates, and calls the +morning, or Tomorrow and Daily for evening, deduplicating inspection of a +shared selected profile. It then collects once, plans eligible Daily dates, and calls the same prompt-generation core sequentially for each planned report. Per-report notification is suppressed; a failed report does not stop later reports. diff --git a/docs/internal/promptkit-adapter.md b/docs/internal/promptkit-adapter.md index b237591..90bffdc 100644 --- a/docs/internal/promptkit-adapter.md +++ b/docs/internal/promptkit-adapter.md @@ -4,10 +4,18 @@ The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor per action. Promptkit dependency types do not escape the adapter. -The adapter exposes exact prompt and profile inspection plus prepared execution. It maps Promptkit -inspection values to project-owned prompt input, output-contract, profile, preparation, execution, -validation, and optional debug values. It classifies adapter failures without copying provider secrets -or unbounded response bodies into application errors or normal state. +The adapter supplies Weatherreporter's embedded prompt, schema, and fallback +profile filesystems to each engine. Promptkit remains responsible for resolving +the configured operator profile source, application fallback catalog, and its +built-in catalog; the adapter does not parse profile YAML, merge sources, or +probe endpoints. + +The adapter exposes exact prompt and profile inspection plus prepared execution. +It maps Promptkit inspection values to project-owned prompt input, +output-contract, logical profile identity, effective backend/model, +preparation, execution, validation, and optional debug values. It classifies +adapter failures without copying provider secrets or unbounded response bodies +into application errors or normal state. The app calls the executor's preparation callback before provider execution to persist safe preparation provenance. Completed executions are then persisted as safe execution provenance and raw generated text diff --git a/docs/internal/report-registry.md b/docs/internal/report-registry.md index bae9540..e62858f 100644 --- a/docs/internal/report-registry.md +++ b/docs/internal/report-registry.md @@ -16,16 +16,19 @@ templates, generation eligibility, compatible prior IDs, default modules, and batch eligibility flags. `Resolved` combines that definition with the valid period and run metadata for one invocation. -| Report ID | Prompt version | Period policy | Comparison | Registry batch flag | Output copy | -| --- | --- | --- | --- | --- | --- | -| `daily` | `1.0.1` | Explicit local civil day | Same valid date | Dynamic Daily inclusion is app-owned | `daily.md` | -| `today` | `1.0.1` | Selected or current local civil day | Same valid date | Morning | `today.md` | -| `tomorrow` | `1.0.1` | Next local civil day | Same valid date | Evening | `tomorrow.md` | -| `hourly` | `1.0.1` | Rolling six-hour interval | Rolling window | — | `hourly.md` | +| Report ID | Prompt version | Default profile | Period policy | Comparison | Registry batch flag | Output copy | +| --- | --- | --- | --- | --- | --- | --- | +| `daily` | `1.1.0` | `weather-balanced` | Explicit local civil day | Same valid date | Dynamic Daily inclusion is app-owned | `daily.md` | +| `today` | `1.1.0` | `weather-balanced` | Selected or current local civil day | Same valid date | Morning | `today.md` | +| `tomorrow` | `1.1.0` | `weather-balanced` | Next local civil day | Same valid date | Evening | `tomorrow.md` | +| `hourly` | `1.1.0` | `weather-light` | Rolling six-hour interval | Rolling window | — | `hourly.md` | Each report pairs its ID and prompt version with matching template and schema IDs. Exact template fields and schema assets belong to [report templates](../templates.md) -and [generated-text internals](generatedtext.md). +and [generated-text internals](generatedtext.md). Prompt assets own default +profile selection; the registry deliberately stores no provider setting. The +[Promptkit integration guide](../integrations/promptkit.md) owns profile +definitions and resolution. All valid periods are half-open. diff --git a/docs/internal/state.md b/docs/internal/state.md index 4619dcc..d846374 100644 --- a/docs/internal/state.md +++ b/docs/internal/state.md @@ -34,8 +34,10 @@ runs never write V1 records. Prompt preparation and execution records are validated on both save and load. They require exact report/prompt identity, complete timing, internally consistent provenance, and status-appropriate validation or bounded classified -errors. Completed execution provenance keeps Promptkit's run identity distinct -from the Weatherreporter run identity. +errors. Completed preparation and execution provenance retain the selected +logical profile ID and resolved backend/model, but never profile endpoints or +credentials. Completed execution provenance keeps Promptkit's run identity +distinct from the Weatherreporter run identity. For a completed prompt run, the execution record is atomically replaced after each downstream artifact is saved. Its path set therefore records the raw and diff --git a/docs/operations.md b/docs/operations.md index 8a3ce29..967be96 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -25,6 +25,30 @@ Distributor notification is attempted. `--out` writes an extra operator copy; it never changes the managed report or upload source. A successful generate command prints its summary to stdout unless `--quiet` is used. +## Local Prompt Profile Override + +Hourly normally selects the embedded `weather-light` profile. To use a local +OpenAI-compatible model without changing prompts or application code, copy +[weather-light-local-profile.yml](../examples/weather-light-local-profile.yml), +set its `endpoint` and `model` for the local server, and configure the copy as +`promptkit.profile_file`. The profile file's `weather-light` definition +completely replaces the embedded definition; it does not affect a report that +selects another profile ID. + +For example, install the profile file at a known absolute path and set: + +```yaml +promptkit: + profile_file: /etc/weatherreporter/weather-light-local-profile.yml +``` + +Prompt inspection occurs before weather collection. A malformed profile file, +missing required credential, or unsupported selected backend stops the command +before collection. A reachable profile can still fail later if its local model +endpoint is unavailable; Weatherreporter does not switch to a remote profile. +See the [configuration reference](config.md) for field definitions and the +[troubleshooting guide](troubleshooting.md) for recovery. + ## Optional Prompt Debug Capture Use `--llm-debug-dir` only when content-rich prompt diagnostics are required: @@ -174,5 +198,7 @@ first response; retain it until the failure is understood. not publish them unintentionally. - Weatherreporter uses one configured Weather API endpoint and local workspace state. +- Promptkit profile resolution does not discover local endpoints or fail over + between local and remote profiles. - It does not provide automatic resume, cleanup, archival, remote state, daemon operation, or automatic storm monitoring. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 27e0ce1..163c8b4 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -11,6 +11,28 @@ the configured `promptkit` profile or profile source, confirm the exact Promptkit asset is available, and supply any reported environment credential. Do not add provider keys to YAML. See [configuration](config.md). +## Local profile override is malformed or selects an unexpected model + +`promptkit.profile_file` and `promptkit.profile_dir` supply complete profile +definitions. A same-ID definition replaces the embedded profile, and a malformed +matching definition fails before collection instead of falling back. Validate +the selected profile's YAML, ID, backend or endpoint, and model. If the model +is unexpected, first check the global `promptkit.profile` selection and then +look for a same-ID definition in the configured file or directory. + +The preparation and execution receipts named by run metadata retain the +selected profile ID and effective backend/model for diagnosis, but not an +endpoint or credential. See the maintained +[local `weather-light` profile example](../examples/weather-light-local-profile.yml). + +## Local model endpoint is unavailable + +An endpoint-only `weather-light` override can pass preflight and still fail +during provider preparation or execution when the local server is unavailable +or does not accept the configured model. Start the local server, correct the +endpoint or model in the profile, and run the command again. Weatherreporter +does not probe endpoints or automatically use a remote profile instead. + ## Preparation, capacity, or execution fails A preparation failure occurs before provider work; an execution failure occurs diff --git a/examples/weather-light-local-profile.yml b/examples/weather-light-local-profile.yml new file mode 100644 index 0000000..93e8d4b --- /dev/null +++ b/examples/weather-light-local-profile.yml @@ -0,0 +1,4 @@ +id: weather-light +endpoint: http://127.0.0.1:11434/v1 +model: weather-local +timeout_seconds: 180 diff --git a/internal/adapters/promptkit/adapter_test.go b/internal/adapters/promptkit/adapter_test.go index b47f66f..d2ad7fe 100644 --- a/internal/adapters/promptkit/adapter_test.go +++ b/internal/adapters/promptkit/adapter_test.go @@ -150,6 +150,14 @@ model: directory-light assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light") } +func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) { + adapter, err := New(Config{ProfileFile: filepath.Join("..", "..", "..", "examples", "weather-light-local-profile.yml")}) + if err != nil { + t.Fatalf("New() error = %v", err) + } + assertProfile(t, adapter, "weather-light", "", "weather-local") +} + func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) { absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: other-profile backend: openrouter