Clean up roadmap and troubleshooting documentation
This commit is contained in:
@@ -20,6 +20,5 @@ Configure a Weather API endpoint first; see the
|
||||
- [CLI reference](docs/cli.md)
|
||||
- [Configuration reference](docs/config.md)
|
||||
- [Operations guide](docs/operations.md)
|
||||
- [Troubleshooting](docs/troubleshooting.md)
|
||||
- [Development guide](docs/development.md)
|
||||
- [Architecture policy](docs/policy/architecture.md)
|
||||
|
||||
@@ -29,7 +29,7 @@ boundaries and invariants.
|
||||
| Recent Changes comparison | [Changes internals](internal/changes.md) and [operations guide](operations.md) | The internal guide owns structured comparison; operations owns user-visible artifact behavior. |
|
||||
| Prompt execution, profiles, prompt inputs, or result handling | `internal/promptexec`, the Promptkit adapter, and [prompt-input internals](internal/prompt-input.md) | These separate the executor contract and input construction. |
|
||||
| Generated-text schemas, validation, render contexts, templates, or Markdown rendering | [Generated-text internals](internal/generatedtext.md), [report-template internals](internal/reporttemplate.md), and [report template guide](templates.md) | These own structured text, renderer implementation, and the maintainer-facing template surface. |
|
||||
| Workspace paths, metadata, atomic persistence, lookup, inspection, or recovery | [State internals](internal/state.md), [operations guide](operations.md), and [troubleshooting guide](troubleshooting.md) | These separate implementation, operator workflows, and symptom-based recovery. |
|
||||
| Workspace paths, metadata, atomic persistence, lookup, inspection, diagnosis, or recovery | [State internals](internal/state.md) and [operations guide](operations.md) | These separate implementation behavior from operator workflows, diagnosis, and recovery. |
|
||||
| Distributor bundles, uploads, notification artifacts, or failures | [Distributor adapter internals](internal/distributor-adapter.md), [Distributor integration contracts](integrations/distributor/), and [operations guide](operations.md) | These separate adapter behavior, external contracts, and operational lifecycle. |
|
||||
| Maintained example configuration | [Configuration reference](config.md) and files under `examples/` | The reference owns field meaning; examples own complete copyable files. |
|
||||
| Release preparation, tagging, publication, or verification | [Release procedure](release.md) | It owns version selection, release-note preparation, candidate validation, tag publication, CI behavior, and post-publication checks. |
|
||||
|
||||
@@ -3,8 +3,7 @@
|
||||
This guide covers normal operation, managed workspace state, inspection,
|
||||
recovery, and operational caveats. See the [CLI reference](cli.md) for complete
|
||||
command syntax and the [configuration reference](config.md) for fields,
|
||||
defaults, and notification templates. For symptom-based diagnosis, see
|
||||
[Troubleshooting](troubleshooting.md).
|
||||
defaults, and notification templates.
|
||||
|
||||
## Normal Operation
|
||||
|
||||
@@ -19,8 +18,8 @@ persists the module snapshot and prompt data package, records Promptkit
|
||||
preparation provenance before provider execution, then persists raw output and
|
||||
execution provenance, validates the structured generated text, and renders the
|
||||
managed Markdown report from the validated text and deterministic values.
|
||||
The current receipts are transitional workspace state, not a cross-version
|
||||
profile-provenance contract.
|
||||
These receipts describe the current workspace implementation; they are not a
|
||||
cross-version profile-provenance contract.
|
||||
|
||||
The managed report and its final metadata are saved before single-report
|
||||
Distributor notification is attempted. `--out` writes an extra operator copy;
|
||||
@@ -48,8 +47,9 @@ Prompt inspection occurs before weather collection. A malformed profile file,
|
||||
missing required credential, or unsupported selected backend stops the command
|
||||
before collection. A reachable profile can still fail later if its local model
|
||||
endpoint is unavailable; Weatherreporter does not switch to a remote profile.
|
||||
See the [configuration reference](config.md) for field definitions and the
|
||||
[troubleshooting guide](troubleshooting.md) for recovery.
|
||||
See the [configuration reference](config.md) for field definitions. For a
|
||||
provider failure, start with the command error and enable secure prompt debug
|
||||
capture when content-rich diagnostics are required.
|
||||
|
||||
## Optional Prompt Debug Capture
|
||||
|
||||
@@ -113,9 +113,9 @@ The generated-text and render-context artifacts are written for every completed
|
||||
single-report generation.
|
||||
A report's metadata links the module snapshot, data package, preparation and
|
||||
execution receipts, managed report, generated-text artifacts, and any available single-report
|
||||
notification artifact. These current-version receipts remain transitional; use
|
||||
the active command's classified error and explicit secure debug capture for
|
||||
prompt diagnosis rather than relying on them as a durable interface. Batch
|
||||
notification artifact. These current-version receipts are implementation
|
||||
details rather than durable interfaces. Use the active command's error and
|
||||
explicit secure debug capture for prompt diagnosis. Batch
|
||||
notification artifacts are separate batch-level records under
|
||||
`notifications/batches`.
|
||||
|
||||
@@ -191,7 +191,7 @@ remain available where they can be safely persisted.
|
||||
- A batch notification failure preserves each report's artifacts and adds the
|
||||
top-level batch notification artifact.
|
||||
|
||||
Use the action summary and its classified error first. For prompt or provider
|
||||
Use the action summary and its error first. For prompt or provider
|
||||
diagnosis, prefer an explicitly enabled secure debug capture; current-version
|
||||
receipt paths may provide supplemental context when available. For a batch
|
||||
failure, inspect the summary first, then inspect the affected report RunIDs or
|
||||
|
||||
@@ -86,8 +86,7 @@ mechanisms, not secret values.
|
||||
| Release notes | `docs/releases/` | One versioned, changelog-style summary for each release, including compatibility and operator action. The file at the tagged commit supplies the corresponding Gitea release body. | Current CLI, configuration, operations, integration, architecture, and internal contracts; release procedure; implementation plans. |
|
||||
| CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, stdout and stderr behavior, summaries, and exit behavior. | Configuration field definitions, complete operating procedures, runtime filesystem layout, and command implementation. |
|
||||
| Configuration contract | `docs/config.md` | Discovery and precedence, fields, defaults, secrets, validation rules, and user-selectable values. | Complete example files, CLI syntax, runtime state lifecycle, and loading implementation. |
|
||||
| Operations | `docs/operations.md` | Normal workflows, physical workspace layout, artifacts and metadata, inspection, notification behavior, recovery, cleanup, permissions, and operational caveats. | Complete CLI syntax, configuration field definitions, logical external contracts, and implementation mechanics. |
|
||||
| Troubleshooting | `docs/troubleshooting.md` | Recurring symptoms, likely causes, diagnostic steps, safe fixes, and links to normal-operation references. | Complete command and configuration references, routine operating procedures, and implementation detail. |
|
||||
| Operations | `docs/operations.md` | Normal workflows, physical workspace layout, artifacts and metadata, inspection, notification behavior, diagnosis, recovery, cleanup, permissions, and operational caveats. | Complete CLI syntax, configuration field definitions, logical external contracts, and implementation mechanics. |
|
||||
| Report template surface | `docs/templates.md` | Implemented template files and partials, render-context fields, editing rules, and maintainer-facing template examples. | Weather derivation, module implementation, generated-text validation internals, and operator procedures. |
|
||||
| External and durable integration contracts | `docs/integrations/` | Weather API, Promptkit, Distributor, external formats and protocols, durable logical paths and schemas, compatibility behavior, and upstream or downstream responsibilities. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, and configuration defaults. |
|
||||
| Internal subsystem behavior | `docs/internal/` | Implementation flow, internal collaborators and state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, user-facing contracts, external schemas, operator procedures, and future package plans. |
|
||||
@@ -110,13 +109,12 @@ structure and invariants. Focused internal documents own implementation
|
||||
behavior. These documents may link to one another but must not maintain
|
||||
parallel package or behavior references.
|
||||
|
||||
### Commands, Configuration, Operations, And Troubleshooting
|
||||
### Commands, Configuration, And Operations
|
||||
|
||||
CLI documentation answers how to invoke Weatherreporter and what its command
|
||||
interface does. Configuration documentation answers what settings mean.
|
||||
Operations answers what happens to runtime state and how to operate or recover
|
||||
the application. Troubleshooting starts from a symptom and leads to diagnosis
|
||||
and a safe fix.
|
||||
the application, including diagnosis and safe responses to runtime failures.
|
||||
|
||||
When a workflow crosses these topics, place the complete procedure with the
|
||||
document that owns the task and link to the other contracts. Do not duplicate
|
||||
|
||||
@@ -1,283 +0,0 @@
|
||||
# Domain-Specific Prompt Profiles Roadmap
|
||||
|
||||
Status: Implemented.
|
||||
|
||||
## Purpose
|
||||
|
||||
Weatherreporter should provide stable, domain-specific Promptkit profile IDs
|
||||
that express the relative resource and analysis needs of its report products.
|
||||
These logical profiles should give each report an appropriate default while
|
||||
allowing operators to replace any definition through the existing configured
|
||||
profile source.
|
||||
|
||||
This roadmap records the scope, policy, and implemented end state. The
|
||||
companion [implementation plan](implementation.md) records the ordered work
|
||||
and verification used to reach it.
|
||||
|
||||
## User Intent
|
||||
|
||||
The feature is intended to provide three related benefits:
|
||||
|
||||
- frequent reports can use a cost-effective model by default;
|
||||
- reports needing broader synthesis can select a stronger default without
|
||||
forcing the same cost on every invocation; and
|
||||
- an installation can map a stable Weatherreporter profile ID to a model on a
|
||||
local network endpoint without modifying embedded prompts or application
|
||||
code.
|
||||
|
||||
`weather-light` describes the profile's intended resource tier, not a latency
|
||||
guarantee. A locally hosted lightweight model may still generate slowly on the
|
||||
available hardware.
|
||||
|
||||
## Pre-Implementation Baseline
|
||||
|
||||
Before implementation, Daily, Today, Tomorrow, and Hourly each declared
|
||||
Promptkit's `gemini-flash-latest` profile as their prompt default. The optional
|
||||
`promptkit.profile` setting overrode that default for every selected report in
|
||||
an invocation.
|
||||
|
||||
Weatherreporter accepted either `promptkit.profile_file` or
|
||||
`promptkit.profile_dir` and passed that source to Promptkit. A matching external
|
||||
profile could override a Promptkit built-in profile, and the configured local
|
||||
backend could support profiles that select `backend: local`. Endpoint-only
|
||||
OpenAI-compatible profiles could also provide their own endpoint.
|
||||
|
||||
Weatherreporter did not own or embed execution profiles. Promptkit v0.5.0
|
||||
provided the fallback-profile layer used to add them without changing the
|
||||
existing operator-source precedence.
|
||||
|
||||
## Prerequisite
|
||||
|
||||
Promptkit v0.5.0 provides the application fallback profile capability defined
|
||||
in the companion
|
||||
[upstream feature request](promptkit-fallback-profiles-feature-request.md), and
|
||||
Weatherreporter now depends on that tagged release. The dependency upgrade has
|
||||
passed the repository test suite and an operator smoke test. Weatherreporter
|
||||
must continue to use only Promptkit's public API rather than depending on its
|
||||
internal packages or reproducing its profile repository behavior.
|
||||
|
||||
## Implemented End State
|
||||
|
||||
Weatherreporter embeds usable definitions for these exact logical profile IDs:
|
||||
|
||||
- `weather-light`
|
||||
- `weather-balanced`
|
||||
- `weather-deep`
|
||||
|
||||
The profiles are Weatherreporter-owned assets and remain behind the existing
|
||||
Promptkit adapter boundary. Prompt definitions select the logical IDs, while
|
||||
Promptkit resolves the effective backend, endpoint, model, and generation
|
||||
settings.
|
||||
|
||||
An operator can place a profile with the same ID in `profile_file` or
|
||||
`profile_dir`. The operator definition completely replaces the embedded
|
||||
Weatherreporter definition for that ID. If the external source does not contain
|
||||
the selected ID, lookup falls through to Weatherreporter's embedded profile and
|
||||
then to Promptkit's built-in catalog.
|
||||
|
||||
The existing global `promptkit.profile` setting remains available as an
|
||||
explicit all-report override. No new configuration field is required for the
|
||||
initial feature.
|
||||
|
||||
## Profile Catalog And Report Assignment
|
||||
|
||||
| Profile | Meaning | Initial default reports |
|
||||
| --- | --- | --- |
|
||||
| `weather-light` | Lowest-cost supported tier for frequent, bounded synthesis. It makes no latency promise. | Hourly |
|
||||
| `weather-balanced` | General-purpose tier for broader day-scale synthesis and forecast discussion. | Daily, Today, Tomorrow |
|
||||
| `weather-deep` | Highest-capability tier for explicit operator use and future products whose measured quality benefit warrants the cost. | None initially |
|
||||
|
||||
The initial assignment recognizes that Weatherreporter's deterministic modules
|
||||
already perform most weather selection and calculation. A higher-capability
|
||||
model should not become a default merely because it is available. Moving an
|
||||
existing report to `weather-deep` requires evidence that the stronger tier
|
||||
materially improves supported reasoning or output quality.
|
||||
|
||||
The three profile IDs are capability policies, not permanent aliases for one
|
||||
provider or model family. Their embedded definitions may change in a future
|
||||
Weatherreporter release, with the change disclosed through normal release and
|
||||
compatibility documentation.
|
||||
|
||||
## Selection And Definition Precedence
|
||||
|
||||
Profile ID selection and profile definition lookup are separate decisions.
|
||||
|
||||
Weatherreporter selects the profile ID in this order:
|
||||
|
||||
1. nonblank `promptkit.profile`; or
|
||||
2. the exact prompt version's `default_profile`.
|
||||
|
||||
Promptkit then resolves the selected profile definition in this order:
|
||||
|
||||
1. explicit in-memory profiles, when used by an embedding consumer or test;
|
||||
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
|
||||
3. Weatherreporter's embedded fallback profiles; and
|
||||
4. Promptkit's embedded built-in profiles.
|
||||
|
||||
A higher-precedence source falls through only when the selected ID is absent.
|
||||
A matching but malformed operator profile fails before weather collection and
|
||||
must not silently use the embedded definition.
|
||||
|
||||
## Local Endpoint Experience
|
||||
|
||||
An operator should be able to override `weather-light` with an endpoint-only
|
||||
profile whose model name is understood by the local OpenAI-compatible server.
|
||||
This path does not require a separate Weatherreporter local-backend setting.
|
||||
|
||||
Alternatively, an override may select `backend: local`; in that case the
|
||||
existing `promptkit.local.endpoint` and concurrency settings continue to own
|
||||
the shared local backend definition.
|
||||
|
||||
The selected local profile is deterministic configuration, not a preference
|
||||
hint. Weatherreporter will not probe for a local model and will not
|
||||
automatically fall back to a remote or paid profile when the endpoint is
|
||||
unavailable. The failure remains visible and attributable to the selected
|
||||
profile.
|
||||
|
||||
## Embedded Profile Policy
|
||||
|
||||
Each embedded profile must be a complete, valid Promptkit profile and must be
|
||||
usable in a default installation with the documented credential mechanism. The
|
||||
initial embedded profiles are expected to use Promptkit's `openrouter` backend,
|
||||
allowing them to inherit its endpoint and `OPENROUTER_API_KEY` environment
|
||||
variable without embedding credentials.
|
||||
|
||||
Embedded definitions should include only settings that are intentional for the
|
||||
selected model and supported by its backend. Avoid incidental generation
|
||||
parameters that reduce portability or trigger provider-specific request
|
||||
failures without a demonstrated quality benefit.
|
||||
|
||||
The initial profile definitions are:
|
||||
|
||||
| Profile ID | OpenRouter model | Reasoning effort | Timeout | Service tier |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` |
|
||||
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
||||
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
||||
|
||||
These settings deliberately match the corresponding Promptkit v0.5.0
|
||||
built-ins while exposing Weatherreporter-owned logical IDs. The `~` prefix is
|
||||
part of each OpenRouter rolling-alias identifier. The profiles do not set
|
||||
temperature, `top_p`, or output-token limits; omission preserves provider
|
||||
defaults and avoids unsupported incidental parameters.
|
||||
|
||||
## Prompt And Active Execution Contract
|
||||
|
||||
Changing a prompt's `default_profile` is a material prompt-definition change.
|
||||
The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new
|
||||
defaults are introduced. Prompt content and generated-text schemas need not
|
||||
change solely for this feature.
|
||||
|
||||
Prompt inspection must continue to occur before weather collection. It should
|
||||
report the selected logical profile ID and the resolved backend and model
|
||||
without exposing endpoints or credentials.
|
||||
|
||||
The active execution contract should retain both the selected logical profile
|
||||
identity and the resolved backend and model through inspection, preparation,
|
||||
execution, errors, and command results where those values are already exposed.
|
||||
This feature must not add a new durable-provenance or cross-version artifact
|
||||
contract.
|
||||
|
||||
The accepted [ephemeral-state roadmap](ephemeral-state.md) makes historical
|
||||
prompt provenance a non-goal. Existing workspace persistence may remain while
|
||||
this feature lands, but it is transitional behavior and must not be expanded or
|
||||
treated as part of the profile feature's desired end state. Prompt preparation
|
||||
and execution artifacts written at `1.0.1` are not required to remain readable
|
||||
after the prompt definitions advance to `1.1.0`.
|
||||
|
||||
## Evaluation Policy
|
||||
|
||||
Concrete model assignments should be evaluated with representative,
|
||||
secret-free Daily, Today, Tomorrow, and Hourly data packages. Evaluation should
|
||||
consider:
|
||||
|
||||
- strict-schema success rate;
|
||||
- unsupported or invented weather claims;
|
||||
- precipitation-timing accuracy and empty-string behavior;
|
||||
- correct use of deterministic hazards, periods, and uncertainty;
|
||||
- summary and forecast-discussion usefulness;
|
||||
- generation latency;
|
||||
- token use and provider cost; and
|
||||
- behavior through a representative local OpenAI-compatible endpoint.
|
||||
|
||||
The purpose is to choose an appropriate default for each tier, not to add a
|
||||
permanent benchmark framework or live-provider requirement to the ordinary
|
||||
test suite. Repository tests remain offline and deterministic.
|
||||
|
||||
## Implemented Scope
|
||||
|
||||
The completed feature includes:
|
||||
|
||||
- Weatherreporter-owned embedded profile assets for all three logical IDs;
|
||||
- Promptkit adapter wiring that supplies those assets as the application
|
||||
fallback profile source;
|
||||
- per-prompt default-profile assignments matching the catalog above;
|
||||
- an exact prompt-version update for the changed definitions;
|
||||
- preservation of the global profile override;
|
||||
- same-ID override behavior through both supported external profile-source
|
||||
forms;
|
||||
- local-backend and endpoint-only override coverage;
|
||||
- fail-fast inspection of missing, malformed, or unusable selected profiles;
|
||||
- offline tests for selection, source precedence, effective model inspection,
|
||||
batch reuse, and active execution behavior;
|
||||
- maintained operator examples for overriding `weather-light` locally; and
|
||||
- updates to the canonical configuration, Promptkit integration, report
|
||||
registry, operations, troubleshooting, internal adapter, and release
|
||||
documentation as applicable when implementation lands.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
The feature does not include:
|
||||
|
||||
- automatic discovery, health checking, or benchmarking of local endpoints;
|
||||
- implicit failover between local and remote profiles;
|
||||
- retries with a more expensive tier after provider or validation failure;
|
||||
- per-report profile configuration fields outside prompt defaults;
|
||||
- profile inheritance, aliases, or field-level merging;
|
||||
- runtime model selection based on weather severity, token count, or report
|
||||
content;
|
||||
- moving Weatherreporter profile policy into Promptkit's built-in catalog;
|
||||
- exposing Promptkit types outside the adapter boundary; or
|
||||
- making live provider calls part of the default repository test suite.
|
||||
|
||||
## Compatibility And Operational Policy
|
||||
|
||||
Existing configurations with a nonblank `promptkit.profile` retain their
|
||||
all-report behavior. Existing `profile_file`, `profile_dir`, local-backend, and
|
||||
credential configuration fields retain their meanings.
|
||||
|
||||
Configurations that rely on the omitted profile setting will intentionally
|
||||
observe new per-report defaults. This is a user-visible model-selection and
|
||||
cost change and must be called out in release notes. Operators who require the
|
||||
old all-report model can preserve it by setting an explicit global profile.
|
||||
|
||||
The prompt-version transition does not provide backward compatibility for
|
||||
historical prompt preparation or execution artifacts. This is consistent with
|
||||
the accepted ephemeral-state direction; the profile feature does not otherwise
|
||||
redesign or remove the current workspace layout.
|
||||
|
||||
An external same-ID override is an operator-owned compatibility commitment.
|
||||
Weatherreporter may evolve its embedded definitions, but it must not rewrite or
|
||||
silently merge an operator file.
|
||||
|
||||
## Completion Record
|
||||
|
||||
The following conditions are satisfied:
|
||||
|
||||
- a tagged Promptkit dependency supports the required fallback layer;
|
||||
- every operational prompt selects its assigned logical profile at exact
|
||||
version `1.1.0`;
|
||||
- all three embedded profiles inspect successfully without an external profile
|
||||
source;
|
||||
- configured same-ID definitions override embedded definitions through both
|
||||
`profile_file` and `profile_dir`;
|
||||
- an invalid matching external definition fails without fallback;
|
||||
- `weather-light` can resolve through an endpoint-only or configured-local
|
||||
override without requiring code or prompt changes;
|
||||
- global `promptkit.profile` still overrides every report in an invocation;
|
||||
- active inspection and execution preserve the selected logical profile and
|
||||
effective model through the project-owned execution contract;
|
||||
- morning and evening batch preflight deduplicates inspection of shared
|
||||
effective profile IDs as it does today;
|
||||
- the default test suite remains offline and deterministic; and
|
||||
- implemented behavior is documented by its canonical current-state owners.
|
||||
@@ -247,7 +247,7 @@ The completed feature includes:
|
||||
of unbounded state growth;
|
||||
- an Accepted ADR documenting the architectural decision; and
|
||||
- updates to canonical architecture, CLI, operations, configuration,
|
||||
troubleshooting, integration, internal, testing, and release documentation
|
||||
integration, internal, testing, and release documentation
|
||||
where their contracts change.
|
||||
|
||||
## Non-Goals
|
||||
@@ -296,16 +296,11 @@ shares mutable filesystem state.
|
||||
|
||||
## Relationship To Domain-Specific Profiles
|
||||
|
||||
The domain-specific profile feature can be implemented before this refactor,
|
||||
but it should not add new historical compatibility or durable-provenance
|
||||
commitments. Profile inspection, selection, override precedence, and effective
|
||||
model resolution remain active-workflow behavior and survive the state change.
|
||||
|
||||
The domain-profile roadmap and implementation plan should acknowledge that
|
||||
prompt artifacts from version `1.0.1` need not remain readable after prompts
|
||||
advance to `1.1.0`. Existing state persistence may remain temporarily while
|
||||
the profile feature lands, but it should not be expanded or treated as the
|
||||
target architecture.
|
||||
The implemented domain-specific profiles do not add historical compatibility
|
||||
or durable-provenance commitments. Profile inspection, selection, override
|
||||
precedence, and effective model resolution remain active-workflow behavior and
|
||||
survive the state change. Existing prompt artifacts need not remain readable
|
||||
after this refactor and must not constrain the target architecture.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ Possible direction:
|
||||
|
||||
1. Detect candidate storm events from alerts, forecast discussion, weather
|
||||
story context, hourly thresholds, and material forecast changes.
|
||||
2. Evaluate candidates through Scriptorium or another narrow evaluator adapter.
|
||||
2. Evaluate candidates through Promptkit or another narrow evaluator adapter.
|
||||
3. Persist storm lifecycle state.
|
||||
4. Generate or update a storm report only when a meaningful event is present.
|
||||
5. Suppress ordinary low-impact thunder or rain chances.
|
||||
|
||||
@@ -1,599 +0,0 @@
|
||||
# Domain-Specific Prompt Profiles Implementation Plan
|
||||
|
||||
Status: Stages 1–7 completed; remediation Stage 8 ready.
|
||||
|
||||
## Purpose And Authority
|
||||
|
||||
This document records the implementation and post-implementation remediation
|
||||
of the
|
||||
[domain-specific prompt profiles roadmap](domain-profiles.md). The roadmap is
|
||||
authoritative for scope, user intent, policy choices, and the intended end
|
||||
state. This plan records implementation sequence, verification, audit findings,
|
||||
and exit gates.
|
||||
|
||||
This plan follows the repository's
|
||||
[architecture](../policy/architecture.md),
|
||||
[documentation](../policy/documentation.md), and
|
||||
[testing](../policy/testing.md) policies.
|
||||
|
||||
## Completed Prerequisite
|
||||
|
||||
Weatherreporter is already pinned to Promptkit v0.5.0. That release provides
|
||||
the public `WithFallbackProfileFS` option and the required precedence across
|
||||
inspection, preparation, and execution. The dependency upgrade passed
|
||||
Weatherreporter's full offline test suite, race-enabled suite, CLI help check,
|
||||
and an operator `generate hourly` smoke test. Do not repeat or replace the
|
||||
dependency upgrade as part of these stages.
|
||||
|
||||
## Locked Product Decisions
|
||||
|
||||
Implement these exact Weatherreporter-owned profiles:
|
||||
|
||||
| Profile ID | Backend | Model | Reasoning effort | Timeout | Service tier |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `weather-light` | `openrouter` | `deepseek/deepseek-v4-flash` | Omitted | 180 seconds | `flex` |
|
||||
| `weather-balanced` | `openrouter` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
||||
| `weather-deep` | `openrouter` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
||||
|
||||
Assign Hourly to `weather-light`; assign Daily, Today, and Tomorrow to
|
||||
`weather-balanced`; assign no report to `weather-deep` initially. Advance all
|
||||
four prompt definitions and matching report-registry entries from `1.0.1` to
|
||||
`1.1.0` when their defaults change.
|
||||
|
||||
The leading `~` in the Gemini and Claude model IDs is required and denotes an
|
||||
OpenRouter rolling alias. Do not substitute the unavailable non-tilde IDs or a
|
||||
dated model version. Do not add temperature, `top_p`, maximum-token, endpoint,
|
||||
or credential fields to the embedded definitions.
|
||||
|
||||
Prompt preparation and execution artifacts written at `1.0.1` are not required
|
||||
to remain readable after the transition to `1.1.0`. Do not add a migration,
|
||||
compatibility shim, or weaker historical-artifact validation for this feature.
|
||||
|
||||
Definition lookup must remain:
|
||||
|
||||
1. explicit Promptkit in-memory profiles used by tests or an embedding
|
||||
consumer;
|
||||
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
|
||||
3. Weatherreporter's embedded fallback profiles; and
|
||||
4. Promptkit's built-in catalog.
|
||||
|
||||
Selection remains a separate concern: a nonblank global `promptkit.profile`
|
||||
selects the profile for every report in the invocation; otherwise the exact
|
||||
prompt definition's `default_profile` selects it. A malformed matching
|
||||
higher-precedence profile is an error and never falls through.
|
||||
|
||||
## Continuing Invariants
|
||||
|
||||
- Keep all Promptkit types and mechanics inside
|
||||
`internal/adapters/promptkit`, its focused tests, and asset contract tests.
|
||||
- Keep prompt inspection before weather collection and provider work.
|
||||
- Keep one Promptkit engine per command action and one shared engine across a
|
||||
sequential batch.
|
||||
- Preserve logical profile ID and effective backend/model information through
|
||||
active inspection and execution where the project-owned contract already
|
||||
exposes it. Do not add new durable-provenance fields or compatibility
|
||||
guarantees.
|
||||
- Leave existing workspace persistence behavior otherwise unchanged. The
|
||||
accepted [ephemeral-state roadmap](ephemeral-state.md) owns its future
|
||||
removal and must not be partially implemented here.
|
||||
- Do not expose endpoints, credentials, rendered messages, schemas, request
|
||||
bodies, response bodies, or complete parameter maps through ordinary errors,
|
||||
logs, summaries, or state.
|
||||
- Keep the default suite deterministic, offline, and credential-free.
|
||||
- Do not add endpoint discovery, health probing, provider failover, retries at
|
||||
a more expensive tier, profile merging, per-report configuration fields, or
|
||||
severity-driven model selection.
|
||||
- Update canonical current-state documentation only in the stage where the
|
||||
corresponding behavior becomes implemented.
|
||||
- Run `git diff --check` before completing every stage.
|
||||
|
||||
## Stage 1: Add The Embedded Weather Profile Catalog
|
||||
|
||||
### Goal
|
||||
|
||||
Create one repository-owned, embedded profile source containing exactly the
|
||||
three locked logical profiles.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add strict YAML profile assets beneath `internal/promptassets` using the
|
||||
exact IDs and definitions in this plan.
|
||||
2. Extend `internal/promptassets` with a narrowly named accessor that returns
|
||||
the embedded profile `fs.FS`. Follow the existing prompt and schema asset
|
||||
pattern without exposing Promptkit types from the package.
|
||||
3. Keep profile filenames and embed layout simple and deterministic. Do not
|
||||
duplicate Promptkit's built-in directory taxonomy unless the application
|
||||
assets require it.
|
||||
4. Validate the assets through Promptkit's public engine/profile inspection
|
||||
surface rather than adding a second YAML parser or a Weatherreporter-owned
|
||||
profile representation.
|
||||
|
||||
### Tests
|
||||
|
||||
- Extend the asset contract tests to assert exactly the three logical IDs,
|
||||
their exact effective model IDs, and the intentional parameters.
|
||||
- Prove all three profiles inspect successfully offline when supplied as a
|
||||
fallback source and no operator source is present.
|
||||
- Assert that the catalog contains no endpoints, credentials, temperature,
|
||||
`top_p`, or maximum-token settings.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/promptassets
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
The embedded catalog is complete, strictly valid, safe, and independently
|
||||
inspectable through Promptkit v0.5.0's public API.
|
||||
|
||||
## Stage 2: Wire Fallback Resolution And Protect Precedence
|
||||
|
||||
### Goal
|
||||
|
||||
Supply the embedded catalog through Promptkit's application fallback layer
|
||||
without changing existing operator configuration or application boundaries.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add `promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), ".")` to
|
||||
normal adapter engine construction.
|
||||
2. Preserve existing `profile_file`, `profile_dir`, configured local backend,
|
||||
timeout, prompt filesystem, schema filesystem, and test-option behavior.
|
||||
3. Ensure ordinary production construction and the adapter's test
|
||||
construction path exercise the same fallback wiring. Test-only explicit
|
||||
profiles may retain Promptkit's documented highest precedence.
|
||||
4. Keep all fallback resolution in Promptkit. Do not add filesystem overlays,
|
||||
existence checks, YAML parsing, or merge behavior to Weatherreporter.
|
||||
|
||||
### Tests
|
||||
|
||||
- At the adapter boundary, prove fallback-only inspection of all three
|
||||
Weatherreporter profiles.
|
||||
- Prove same-ID overrides through both configured `profile_file` and
|
||||
`profile_dir`, including resolution of the override's effective backend and
|
||||
model.
|
||||
- Prove an absent operator match falls through, while a malformed matching
|
||||
operator definition fails without using the embedded profile.
|
||||
- Prove a selected Promptkit built-in that is absent from both higher layers
|
||||
still resolves.
|
||||
- Prove an explicit in-memory test profile retains highest precedence.
|
||||
- Cover both local override forms required by the roadmap: an endpoint-only
|
||||
OpenAI-compatible `weather-light` profile and a `backend: local` profile
|
||||
using the configured local endpoint. No test may contact either endpoint.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Inspection and prepared execution use Promptkit's exact four-layer precedence,
|
||||
operator errors remain visible, and local overrides require no prompt or code
|
||||
changes.
|
||||
|
||||
## Stage 3: Adopt Logical Defaults And Prompt Version 1.1.0
|
||||
|
||||
### Goal
|
||||
|
||||
Move operational prompts from provider-oriented defaults to the three-tier
|
||||
Weatherreporter policy with an exact, synchronized version transition.
|
||||
|
||||
### Work
|
||||
|
||||
1. Change Hourly's `default_profile` to `weather-light`.
|
||||
2. Change Daily, Today, and Tomorrow to `weather-balanced`.
|
||||
3. Advance the exact version in all four prompt YAML assets from `1.0.1` to
|
||||
`1.1.0` without changing prompt text or generated-text schemas solely for
|
||||
this feature.
|
||||
4. Advance the four matching report-registry prompt versions to `1.1.0` in the
|
||||
same change. Keep prompt IDs, report IDs, modules, periods, templates, and
|
||||
output contracts unchanged.
|
||||
5. Update fixtures and expectations that intentionally assert the current
|
||||
prompt contract. Do not rewrite historical fixture versions or weaken tests
|
||||
that protect actual compatibility.
|
||||
|
||||
### Tests
|
||||
|
||||
- Update asset and report-registry contract tests to require exact version
|
||||
`1.1.0` and the report-to-profile assignments locked in this plan.
|
||||
- Inspect every exact prompt version through the real embedded prompt, schema,
|
||||
and fallback-profile filesystems.
|
||||
- Prove Hourly resolves DeepSeek V4 Flash, the three day-scale reports resolve
|
||||
Gemini Flash Latest, and `weather-deep` remains inspectable but unassigned.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/promptassets ./internal/report ./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Every operational prompt and registry definition agrees on exact version
|
||||
`1.1.0`, selects its intended logical tier, and resolves its expected effective
|
||||
model offline.
|
||||
|
||||
## Stage 4: Verify Application Selection And Batch Reuse
|
||||
|
||||
### Goal
|
||||
|
||||
Protect the assembled application behavior created by the new defaults and
|
||||
confirm that logical identity is not lost during active effective-model
|
||||
resolution.
|
||||
|
||||
### Work
|
||||
|
||||
1. Preserve the current pre-collection inspection order and fail-fast behavior
|
||||
for missing credentials, unknown profiles, malformed profiles, and unusable
|
||||
backends.
|
||||
2. Preserve the global `promptkit.profile` all-report override. Do not add a
|
||||
second override mechanism or report-specific configuration fields.
|
||||
3. Preserve batch preflight deduplication by selected effective profile ID:
|
||||
Today and Tomorrow in the same batch should inspect their shared
|
||||
`weather-balanced` selection once.
|
||||
4. Preserve the selected logical profile ID and resolved backend/model through
|
||||
active inspection, preparation, and execution using the existing
|
||||
project-owned contract. Do not add state fields, expand persisted parameter
|
||||
detail, or create a new historical compatibility guarantee.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add or update representative app tests for default Hourly and day-scale
|
||||
selection, a global-profile override, and a morning/evening batch sharing
|
||||
`weather-balanced`.
|
||||
- Assert inspection completes before weather collection and provider
|
||||
generation, including malformed same-ID operator overrides.
|
||||
- Assert active inspection and execution expose the logical profile ID and
|
||||
effective model for both embedded and overridden profiles.
|
||||
- Assert endpoints and credentials remain absent from errors, summaries,
|
||||
normal logs, and ordinary state.
|
||||
- Use project-owned executor fakes or Promptkit provider fakes; do not make live
|
||||
provider calls.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/app ./internal/cli
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Single-report and batch workflows select the intended tier, retain existing
|
||||
override and preflight behavior, deduplicate shared batch inspection, and
|
||||
preserve safe logical and effective model information during active execution
|
||||
without adding a durable-provenance contract.
|
||||
|
||||
## Stage 5: Publish Canonical Operator And Maintainer Documentation
|
||||
|
||||
### Goal
|
||||
|
||||
Document the implemented feature once in each canonical owner and provide one
|
||||
maintained, copyable local override example.
|
||||
|
||||
### Work
|
||||
|
||||
1. Update `docs/config.md` to explain global profile selection versus
|
||||
`profile_file`/`profile_dir` definition lookup and link to the maintained
|
||||
example. Keep the field reference in this canonical document.
|
||||
2. Update the Promptkit integration document with the logical profile catalog,
|
||||
source precedence, exact prompt-version relationship, and safe active
|
||||
inspection and execution contract. Avoid restating complete configuration
|
||||
syntax or presenting transitional persistence as the target architecture.
|
||||
3. Update the report-registry, Promptkit adapter, app-orchestration, and state
|
||||
internal documents only where their implemented contracts changed.
|
||||
4. Update `docs/operations.md` with the normal local-override workflow and
|
||||
`docs/troubleshooting.md` with malformed override, unavailable local
|
||||
endpoint, missing credential, and unexpected effective-model diagnostics.
|
||||
5. Add or update one secret-free file under `examples/` showing a
|
||||
`weather-light` override for a local OpenAI-compatible endpoint. Choose one
|
||||
supported form as the complete example and mention the other form only in
|
||||
its canonical reference.
|
||||
6. Update the architecture policy only if implementation changed a normative
|
||||
boundary or invariant. Do not add future behavior to current-state docs.
|
||||
7. Keep the feature roadmap and this plan in their pre-implementation statuses
|
||||
until the final repository gate passes. Do not create release notes before
|
||||
a release version is chosen.
|
||||
|
||||
### Tests
|
||||
|
||||
- Verify every changed repository-relative link and every profile/model ID.
|
||||
- Validate maintained YAML examples through the same strict configuration or
|
||||
Promptkit profile path used by production where practical.
|
||||
- Run the focused tests that own any executable examples, followed by:
|
||||
|
||||
```sh
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Users, operators, and maintainers can discover the tier defaults, precedence,
|
||||
global override, local override, and failure behavior without duplicated or
|
||||
future-state documentation.
|
||||
|
||||
## Stage 6: Complete Repository Verification And Roadmap Handoff
|
||||
|
||||
### Goal
|
||||
|
||||
Demonstrate that the complete feature is coherent, offline-testable, and ready
|
||||
for review and a later release decision.
|
||||
|
||||
### Work
|
||||
|
||||
1. Review the complete diff against the roadmap, this plan, and all three
|
||||
policy documents. Remove stale identifiers, temporary helpers, redundant
|
||||
tests, and documentation duplication.
|
||||
2. Confirm `go.mod` and `go.sum` retain tagged Promptkit v0.5.0 without a local
|
||||
replacement or dependency drift.
|
||||
3. Confirm only the four supported report products exist and no retired report
|
||||
surfaces were reintroduced.
|
||||
4. Confirm the roadmap's completion criteria one by one. Change its status to
|
||||
implemented and this plan's status to completed only after every criterion
|
||||
and command below passes.
|
||||
5. Do not require a live provider for completion. If credentials and network
|
||||
access are deliberately supplied by an operator, record live smoke results
|
||||
separately as release-candidate evidence rather than adding them to the
|
||||
default suite.
|
||||
|
||||
### Verification
|
||||
|
||||
Run `gofmt -w` on every changed Go file, then run:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
Also inspect all three logical profiles through the application's normal
|
||||
preflight path using offline provider doubles, including one same-ID local
|
||||
override and one explicit global override.
|
||||
|
||||
### Exit Gate
|
||||
|
||||
All roadmap completion criteria are satisfied, all verification commands pass,
|
||||
the working tree contains only intentional changes, and the canonical
|
||||
documentation describes the implemented state. The feature is ready for code
|
||||
review and release preparation.
|
||||
|
||||
## Post-Implementation Review
|
||||
|
||||
Stages 1–6 implemented the intended production behavior and passed their
|
||||
offline verification gates. A subsequent review found no high-severity runtime
|
||||
defect, but identified three test-quality issues and one remaining validation
|
||||
obligation:
|
||||
|
||||
- one app test asserted durable preparation and execution artifact provenance,
|
||||
contrary to the active-execution boundary and accepted ephemeral-state
|
||||
direction;
|
||||
- an adapter-package test depended upward on app orchestration and duplicated
|
||||
test ownership;
|
||||
- embedded fallback profiles were inspected but not exercised through one
|
||||
prepared execution with a provider fake; and
|
||||
- the roadmap's representative model-evaluation policy had no recorded
|
||||
evidence.
|
||||
|
||||
Stages 7 and 8 address those findings without changing the profile catalog,
|
||||
selection precedence, report assignments, prompt content, generated-text
|
||||
schemas, or default offline test contract.
|
||||
|
||||
## Stage 7: Correct Test Ownership And Fallback Execution Coverage
|
||||
|
||||
### Goal
|
||||
|
||||
Remove accidental durable-state and cross-layer test commitments while adding
|
||||
one focused offline execution test for the embedded fallback path.
|
||||
|
||||
### Work
|
||||
|
||||
1. Rewrite `TestGenerateDetailedPreservesSelectedProfileThroughExecution` so
|
||||
it protects active workflow behavior only:
|
||||
|
||||
- retain the Hourly default, day-scale default, and global-override cases;
|
||||
- assert the profile ID sent in `promptexec.ExecuteRequest`;
|
||||
- have the executor fake record the preparation and execution values it
|
||||
emits, then assert their logical profile ID and effective backend/model;
|
||||
- do not load preparation, execution, or metadata files to establish a
|
||||
durable profile-provenance contract; and
|
||||
- remove artifact-content scans whose fake inputs cannot contain an endpoint
|
||||
or credential.
|
||||
|
||||
2. Preserve meaningful safety coverage at the boundary that can expose the
|
||||
sensitive value:
|
||||
|
||||
- retain adapter mapping coverage proving an endpoint from a real Promptkit
|
||||
profile does not enter `promptexec.ProfileInspection`;
|
||||
- retain app error coverage proving dependency errors containing an endpoint
|
||||
or credential are replaced by a bounded classified error; and
|
||||
- do not add profile endpoints or credentials to project-owned execution
|
||||
types merely to make a leakage test possible.
|
||||
|
||||
3. Remove `internal/app`, app configuration, and report-registry dependencies
|
||||
from `internal/adapters/promptkit/adapter_test.go`. Move the assembled
|
||||
application-preflight test to a new app-owned external integration test,
|
||||
such as `internal/app/prompt_profile_integration_test.go` with package
|
||||
`app_test`:
|
||||
|
||||
- construct the real Promptkit adapter through its public `New` function;
|
||||
- call the public app prompt-inspection operation;
|
||||
- supply a deterministic credential lookup rather than reading the process
|
||||
environment; and
|
||||
- cover Hourly, one representative day-scale default, the explicit
|
||||
`weather-deep` global override, and a same-ID endpoint-only
|
||||
`weather-light` override. The asset contract tests already own the exact
|
||||
mapping for all four prompts, so the integration test need not repeat all
|
||||
four.
|
||||
|
||||
4. Add one adapter-owned, offline fake-client execution test using the real
|
||||
embedded Hourly prompt at `1.1.0` and selected profile `weather-light`.
|
||||
Execute through the normal prepared adapter path and assert:
|
||||
|
||||
- the preparation callback runs before the fake provider;
|
||||
- preparation and execution report logical profile `weather-light`, backend
|
||||
`openrouter`, and model `deepseek/deepseek-v4-flash`;
|
||||
- the fake provider request targets `deepseek/deepseek-v4-flash`; and
|
||||
- schema validation completes without contacting a live service.
|
||||
|
||||
One execution case is sufficient because Promptkit owns uniform source
|
||||
precedence and the adapter's inspection tests already cover fallback,
|
||||
operator file, operator directory, built-in, and explicit in-memory layers.
|
||||
|
||||
5. Reconcile the profile-related current-state documentation:
|
||||
|
||||
- it may accurately describe fields present in current preparation and
|
||||
execution receipts;
|
||||
- it must not promise cross-version readability or characterize those
|
||||
receipts as the profile feature's durable target architecture; and
|
||||
- troubleshooting should prefer active command errors and explicit secure
|
||||
debug capture, mentioning current-version receipts only as transitional
|
||||
state if they remain useful before the ephemeral-state refactor.
|
||||
|
||||
6. Do not change production profile resolution, prompt definitions, state
|
||||
schemas, artifact validators, or the ephemeral-state roadmap in this stage.
|
||||
|
||||
### Tests
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app ./internal/cli
|
||||
go test -count=1 -race ./internal/promptassets ./internal/adapters/promptkit ./internal/app ./internal/cli
|
||||
go test -count=1 ./...
|
||||
go vet ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Review the changed tests against the testing policy and confirm that adapter
|
||||
tests own adapter behavior, app tests own orchestration, and state tests remain
|
||||
the sole owner of durable artifact format and validation details.
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Active profile selection and effective-model propagation remain protected
|
||||
without adding a durable-provenance commitment; the adapter test package no
|
||||
longer imports the app layer; one embedded fallback profile completes prepared
|
||||
execution through a provider fake; and every required check passes offline.
|
||||
|
||||
## Stage 8: Evaluate The Initial Model Ladder
|
||||
|
||||
### Goal
|
||||
|
||||
Produce explicit release-candidate evidence that the selected models are
|
||||
acceptable for their intended report tiers and that a representative local
|
||||
override provides the promised operator experience.
|
||||
|
||||
This is an opt-in evaluation stage, not an ordinary automated-test stage. It
|
||||
requires operator-approved provider credentials, network access, and a local
|
||||
OpenAI-compatible endpoint. Do not mark it complete when those prerequisites
|
||||
are unavailable; report the missing prerequisite instead.
|
||||
|
||||
### Corpus
|
||||
|
||||
Use four representative, secret-free YAML data packages: one each for Daily,
|
||||
Today, Tomorrow, and Hourly. The set must include at least one package with
|
||||
precipitation windows and at least one with none. Remove precise private
|
||||
location identifiers or other operationally sensitive values without changing
|
||||
the meteorological relationships being evaluated.
|
||||
|
||||
Record a SHA-256 hash and a short, non-sensitive description for each package.
|
||||
Do not commit full packages or generated prose unless the user separately
|
||||
approves them as repository fixtures.
|
||||
|
||||
### Execution Matrix
|
||||
|
||||
Run these six evaluations from the exact package bytes:
|
||||
|
||||
| Case | Package | Profile |
|
||||
| --- | --- | --- |
|
||||
| Hourly default | Hourly | `weather-light` |
|
||||
| Daily default | Daily | `weather-balanced` |
|
||||
| Today default | Today | `weather-balanced` |
|
||||
| Tomorrow default | Tomorrow | `weather-balanced` |
|
||||
| Deep comparison | The same Daily package used above | `weather-deep` |
|
||||
| Local override | The same Hourly package used above | operator-defined `weather-light` endpoint profile |
|
||||
|
||||
After the successful local-override case, stop or deliberately address an
|
||||
unavailable test endpoint and repeat it as a negative control. Confirm that the
|
||||
request fails visibly and does not call or select an embedded remote profile.
|
||||
This negative control is not an additional quality-evaluation case.
|
||||
|
||||
Use a temporary, untracked evaluation harness beneath the module when exact
|
||||
package replay is needed. It should call the existing Promptkit adapter and
|
||||
project-owned execution contract rather than duplicate prompt loading,
|
||||
rendering, or schema validation. Remove the harness and all unapproved raw
|
||||
outputs before completing the stage. Never print or record credentials.
|
||||
|
||||
### Evaluation Record
|
||||
|
||||
Add a concise `## Evaluation Record` section to
|
||||
`docs/roadmap/domain-profiles.md`. For every case, record:
|
||||
|
||||
- evaluation date, logical profile, effective backend, and exact model
|
||||
reported by execution;
|
||||
- corpus hash, validation outcome, latency, prompt/completion/total token use,
|
||||
and provider-reported or contemporaneously calculated cost;
|
||||
- whether every generated claim is supported by the deterministic package;
|
||||
- whether hazards, periods, uncertainty, and precipitation timing are used
|
||||
correctly;
|
||||
- whether `precipitation_timing` is exactly an empty string for the no-window
|
||||
case;
|
||||
- a short usefulness assessment for summary and forecast discussion; and
|
||||
- any provider, alias, or local-endpoint caveat observed.
|
||||
|
||||
Do not include credentials, endpoints, complete effective parameter maps,
|
||||
full data packages, rendered prompts, or full generated responses in the
|
||||
record. The secure debug directory may be used temporarily for operator review
|
||||
and remains operator-managed.
|
||||
|
||||
### Acceptance Rules
|
||||
|
||||
- Every case must complete strict JSON Schema validation without repair.
|
||||
- Generated prose must contain no material unsupported weather claim or
|
||||
contradiction of deterministic hazards, periods, or uncertainty.
|
||||
- Precipitation timing must agree with the deterministic windows and use the
|
||||
required empty-string representation when no window exists.
|
||||
- The local override must select the operator model without modifying a prompt
|
||||
or application code and must not fall back to a remote profile when the local
|
||||
endpoint is unavailable.
|
||||
- Latency, tokens, and cost must be recorded, but this initial evaluation does
|
||||
not impose an invented numeric threshold. The operator decides whether the
|
||||
observed tradeoff remains acceptable for the named tier.
|
||||
- If a default case fails schema or factual acceptance, do not weaken the
|
||||
schema or prompt to accommodate the model. Reopen the concrete model or
|
||||
profile-setting decision in the feature roadmap and leave this stage
|
||||
incomplete.
|
||||
|
||||
### Verification
|
||||
|
||||
After removing temporary evaluation material, run:
|
||||
|
||||
```sh
|
||||
go test -count=1 ./...
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
Confirm that the only intended repository change from this stage is the
|
||||
concise evaluation record and any roadmap status correction required by its
|
||||
result. Do not add live credentials, provider-dependent tests, a permanent
|
||||
benchmark framework, or release notes before a release version is selected.
|
||||
|
||||
### Exit Gate
|
||||
|
||||
All six cases satisfy the acceptance rules, the roadmap contains concise and
|
||||
safe evaluation evidence, no temporary corpus or response material remains in
|
||||
the repository, and the default suite remains offline. Set this plan back to
|
||||
`Status: Completed` only after both Stages 7 and 8 have passed.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The model identifiers, profile settings, report assignments, version
|
||||
transition, precedence, compatibility behavior, test boundaries, and
|
||||
documentation ownership are decision-complete.
|
||||
@@ -1,169 +0,0 @@
|
||||
# Promptkit Feature Request: Application Fallback Profiles
|
||||
|
||||
Status: Implemented upstream in Promptkit v0.5.0.
|
||||
|
||||
Promptkit v0.5.0 resolved this request with the public
|
||||
`WithFallbackProfileFS` engine option and the precedence and error semantics
|
||||
specified below. This document is retained as the downstream rationale for
|
||||
the capability.
|
||||
|
||||
## Purpose
|
||||
|
||||
Promptkit should allow a consuming application to supply an embedded fallback
|
||||
profile source that sits below operator-configured profiles and above
|
||||
Promptkit's own built-in profile catalog.
|
||||
|
||||
This capability would let an application publish stable, domain-specific
|
||||
profile IDs with useful defaults while preserving Promptkit's existing
|
||||
operator-override behavior. The capability must remain application-neutral;
|
||||
Promptkit should provide the source layer but should not own downstream profile
|
||||
names, model assignments, or configuration policy.
|
||||
|
||||
## Downstream Use Case
|
||||
|
||||
Weatherreporter wants to embed profiles such as `weather-light`,
|
||||
`weather-balanced`, and `weather-deep`. Report prompts would select those
|
||||
logical profiles instead of naming provider- or model-specific Promptkit
|
||||
profiles directly.
|
||||
|
||||
An installation could then place a profile with the same ID in its configured
|
||||
profile directory. For example, a local `weather-light` definition could point
|
||||
to an OpenAI-compatible endpoint on the deployment network. When no operator
|
||||
definition exists, Weatherreporter's embedded definition would keep the
|
||||
application usable without additional profile files.
|
||||
|
||||
This pattern is useful beyond Weatherreporter. Any Promptkit consumer may want
|
||||
application-owned execution tiers or workload-specific defaults without
|
||||
adding domain-specific profiles to Promptkit's general built-in catalog.
|
||||
|
||||
## Current Constraint
|
||||
|
||||
Promptkit currently resolves matching profile IDs in this order:
|
||||
|
||||
1. in-memory profiles supplied through `WithProfiles`;
|
||||
2. one configured profile file, `fs.FS`, or directory source; and
|
||||
3. Promptkit's embedded built-in profiles.
|
||||
|
||||
These layers do not express the desired application-default relationship:
|
||||
|
||||
- `WithProfiles` has higher precedence than the configured source, so it would
|
||||
prevent an operator file from overriding an application profile with the
|
||||
same ID.
|
||||
- `WithProfileFS` can hold embedded application assets, but it occupies the
|
||||
configured-source layer and therefore replaces rather than sits beneath a
|
||||
configured profile directory or file.
|
||||
- adding downstream profile IDs to Promptkit's built-in catalog would make the
|
||||
library own application-specific policy.
|
||||
|
||||
A downstream application could build its own filesystem overlay, but that
|
||||
would duplicate Promptkit's profile discovery, error, and precedence behavior
|
||||
at the consumer boundary.
|
||||
|
||||
## Requested Capability
|
||||
|
||||
Add one optional application fallback profile source to engine construction.
|
||||
When present, matching profile IDs should resolve in this order:
|
||||
|
||||
1. in-memory profiles supplied through `WithProfiles`;
|
||||
2. the ordinary configured profile source selected through a profile option or
|
||||
`Config.ProfileDir`;
|
||||
3. the application fallback profile source; and
|
||||
4. Promptkit's embedded built-in profiles.
|
||||
|
||||
When no application fallback is configured, existing source precedence and
|
||||
behavior must remain unchanged.
|
||||
|
||||
The minimum useful public surface is an `fs.FS`-backed option because consumers
|
||||
can embed YAML profile assets. A possible API shape is:
|
||||
|
||||
```go
|
||||
promptkit.WithFallbackProfileFS(profileFS, ".")
|
||||
```
|
||||
|
||||
The name is illustrative rather than prescriptive. A companion option for
|
||||
validated `Profile` values could be added if Promptkit maintainers find it
|
||||
generally useful, but it is not required for the Weatherreporter use case.
|
||||
|
||||
## Required Semantics
|
||||
|
||||
- A higher-precedence source falls through only when the requested profile ID
|
||||
is absent.
|
||||
- A malformed, unreadable, duplicate, ambiguous, or otherwise invalid matching
|
||||
profile is an error and must not silently fall through.
|
||||
- The fallback source uses the existing strict profile YAML format and profile
|
||||
validation rules.
|
||||
- Profile values are selected as a whole. This feature does not merge,
|
||||
inherit, or partially overlay profile definitions.
|
||||
- `InspectProfile`, `Prepare`, prepared execution, and ordinary execution use
|
||||
the same profile-source precedence.
|
||||
- An explicit request profile continues to take precedence over a prompt's
|
||||
`default_profile`; this request concerns definition lookup after the profile
|
||||
ID has been selected.
|
||||
- Repeated fallback-source options should follow Promptkit's documented
|
||||
same-category option convention, normally with the last value replacing the
|
||||
earlier value.
|
||||
- A canceled lookup, invalid fallback asset, or unknown resolved backend should
|
||||
continue to cross the public facade through Promptkit's existing public error
|
||||
identities.
|
||||
- Exact profile inspection must remain side-effect free and must not contact a
|
||||
model provider.
|
||||
|
||||
## Application And Library Boundaries
|
||||
|
||||
Promptkit should own:
|
||||
|
||||
- the additional repository layer;
|
||||
- deterministic lookup and fallthrough behavior;
|
||||
- validation of the supplied source through the existing profile contract;
|
||||
- consistent use of the layer across inspection and execution; and
|
||||
- public documentation and tests for the added precedence rule.
|
||||
|
||||
The consuming application should continue to own:
|
||||
|
||||
- whether it supplies fallback profiles;
|
||||
- the profile IDs and their domain meaning;
|
||||
- embedded profile contents and model choices;
|
||||
- application configuration and override policy;
|
||||
- report- or workload-to-profile assignment; and
|
||||
- credential checks and operator-facing errors beyond Promptkit's public
|
||||
contract.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This request does not ask Promptkit to add:
|
||||
|
||||
- Weatherreporter-specific profile IDs to its built-in catalog;
|
||||
- profile inheritance, aliases, or field-level merging;
|
||||
- automatic endpoint discovery or availability probing;
|
||||
- provider failover or fallback from a failed selected profile;
|
||||
- per-request model benchmarking or tier selection;
|
||||
- application configuration discovery; or
|
||||
- eager validation of every profile in every source.
|
||||
|
||||
## Compatibility
|
||||
|
||||
The feature can be additive. Engines that do not configure an application
|
||||
fallback source should retain their current public behavior and precedence.
|
||||
Existing uses of `WithProfiles`, `WithProfileFile`, `WithProfileFS`, and
|
||||
`Config.ProfileDir` should not change meaning.
|
||||
|
||||
The application fallback is deliberately lower precedence than every existing
|
||||
consumer-configured source. This preserves the established expectation that a
|
||||
custom profile definition can override a packaged default with the same ID.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
The capability is sufficient for downstream adoption when Promptkit can
|
||||
demonstrate that:
|
||||
|
||||
- a fallback-only profile can be inspected and used for preparation and
|
||||
execution;
|
||||
- a configured directory, file, or `fs.FS` profile with the same ID overrides
|
||||
the fallback profile;
|
||||
- an absent configured profile falls through to the application fallback;
|
||||
- an invalid configured match fails instead of falling through;
|
||||
- an absent application fallback profile continues to resolve from Promptkit's
|
||||
built-in catalog;
|
||||
- `WithProfiles` retains highest precedence;
|
||||
- behavior is identical across inspection, preparation, and execution; and
|
||||
- omitting the new option preserves existing tests and public contracts.
|
||||
@@ -1,516 +0,0 @@
|
||||
# Promptkit Migration Roadmap
|
||||
|
||||
Status: Completed roadmap record.
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap records the scope, decisions, and completed outcome of replacing
|
||||
the external Scriptorium CLI integration with Promptkit. Canonical
|
||||
documentation outside `docs/roadmap/` owns the implemented behavior.
|
||||
|
||||
## Pre-Migration Baseline
|
||||
|
||||
Status: Historical migration input.
|
||||
|
||||
Before the migration, Weatherreporter exposed seven report definitions, but
|
||||
only four had complete prompt-backed report implementations:
|
||||
|
||||
- Daily Report: `weather.daily_generated_text`
|
||||
- Today Report: `weather.today_generated_text`
|
||||
- Tomorrow Report: `weather.tomorrow_generated_text`
|
||||
- Hourly Report: `weather.hourly_generated_text`
|
||||
|
||||
The three-day, weekend, and storm commands and registry definitions had no
|
||||
corresponding Scriptorium prompt or schema and never formed complete
|
||||
operational report products. The `weather.daily_report` Scriptorium prompt was
|
||||
legacy source material and was not selected by the registry.
|
||||
|
||||
The Scriptorium source corpus was retained temporarily under
|
||||
`docs/roadmap/scriptorium/` as migration input. It contained the four
|
||||
operational generated-text prompt definitions, their referenced content,
|
||||
private response schemas, shared instructions, and the unused legacy Daily
|
||||
Markdown prompt. The temporary corpus was removed after the runtime assets
|
||||
were reconciled and embedded.
|
||||
|
||||
## Implemented End State
|
||||
|
||||
Status: Completed.
|
||||
|
||||
Weatherreporter pins
|
||||
`gitea.maximumdirect.net/eric/promptkit` at `v0.4.0` and uses it as the
|
||||
in-process engine for prompt inspection, prepared execution, provider calls,
|
||||
and first-pass output validation.
|
||||
|
||||
The `scriptorium` executable, subprocess adapter, configuration, runtime
|
||||
dependency, direct-Markdown execution path, and integration documentation have
|
||||
been removed. The four operational reports continue to use structured
|
||||
generated text followed by weatherreporter-owned validation and Markdown
|
||||
templates.
|
||||
|
||||
The unfinished three-day, weekend, and storm reports are not implemented as
|
||||
part of this migration. Their incomplete CLI, registry, documentation, and
|
||||
generation declarations are removed from the implemented surface before the
|
||||
migration is considered complete. Any future implementation of those products
|
||||
requires separate roadmap scope, prompt and schema design, tests, and
|
||||
documentation.
|
||||
|
||||
Weather selection, forecast derivation, valid periods, module construction,
|
||||
Recent Changes, generated-text interpretation, Markdown templates, durable
|
||||
state, inspection, output copies, and Distributor notification remain owned by
|
||||
weatherreporter.
|
||||
|
||||
The four report prompts and private response schemas are versioned embedded
|
||||
application assets. Operators configure Promptkit profiles without replacing
|
||||
the report-owned corpus. One Promptkit engine is constructed per CLI
|
||||
invocation and shared by every report in that invocation, including all
|
||||
reports in a morning or evening batch.
|
||||
|
||||
Promptkit is isolated behind a weatherreporter-owned execution contract.
|
||||
Promptkit request, result, validation, error, profile, backend, and provider
|
||||
types do not leak into application orchestration, report definitions, domain
|
||||
packages, CLI summaries, durable state contracts, or Distributor behavior.
|
||||
|
||||
## Goals
|
||||
|
||||
Status: Completed migration outcomes.
|
||||
|
||||
- Removed the Scriptorium runtime dependency and subprocess boundary.
|
||||
- Migrated the four operational report prompts to Promptkit `v0.4.0`.
|
||||
- Used prepared execution to persist preparation provenance before provider work
|
||||
while executing the exact frozen snapshot.
|
||||
- Validated report prompt and profile selections before weather collection when
|
||||
the required information is available.
|
||||
- Preserved deterministic module snapshots and structured Recent Changes.
|
||||
- Preserved generated-text domain validation and repository-owned Markdown
|
||||
rendering.
|
||||
- Preserved context cancellation, actionable errors, secret redaction, and
|
||||
inspectable failures.
|
||||
- Improved durable prompt provenance with prompt, input, profile, model,
|
||||
validation, usage, and timing metadata.
|
||||
- Kept content-rich prompt and response diagnostics separate from routine
|
||||
metadata and CLI output.
|
||||
- Kept tests offline and deterministic through injected Promptkit model
|
||||
clients and fixtures.
|
||||
- Removed incomplete report declarations from the implemented product surface
|
||||
rather than creating new report products during an integration migration.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
Status: Completed migration constraints.
|
||||
|
||||
The completed migration did not:
|
||||
|
||||
- create prompts, schemas, templates, or completed products for three-day,
|
||||
weekend, or storm reports;
|
||||
- preserve the unused `weather.daily_report` legacy Markdown prompt as an
|
||||
active runtime asset;
|
||||
- preserve a direct-Markdown LLM generation mode;
|
||||
- move meteorological selection, derivation, thresholds, or comparison logic
|
||||
into prompts or Promptkit;
|
||||
- send raw unbounded Weather API responses to the model;
|
||||
- replace weatherreporter's generated-text domain validation or Markdown
|
||||
template rendering;
|
||||
- add a general workflow engine, provider plugin system, or arbitrary backend
|
||||
registry;
|
||||
- add automatic provider, validation, repair, or capacity retries;
|
||||
- add concurrent report generation to the sequential batch workflow;
|
||||
- expose Promptkit types as weatherreporter contracts;
|
||||
- keep a production-selectable Scriptorium/Promptkit dual-run mode;
|
||||
- require Promptkit eager source validation, structured generation errors, or
|
||||
semantic execution-target fingerprints; or
|
||||
- use an unpublished Promptkit commit, committed Go workspace, or committed
|
||||
local module replacement.
|
||||
|
||||
## Locked Decisions
|
||||
|
||||
Status: Implemented migration decisions.
|
||||
|
||||
### Dependency And Upgrade Boundary
|
||||
|
||||
- The migration pins the tagged Promptkit `v0.4.0` release.
|
||||
- Coordinated local development may temporarily use the sibling Promptkit
|
||||
checkout, but committed module metadata must reference the tagged release.
|
||||
- The adapter relies on the public root Promptkit package only.
|
||||
- A future Promptkit upgrade requires explicit review of prepared-execution
|
||||
lifecycle, prompt and profile inspection, prompt/profile/schema formats,
|
||||
error identities, validation behavior, capacity behavior, and the outbound
|
||||
provider contract.
|
||||
- Promptkit's deferred eager source validation, structured generation errors,
|
||||
and semantic execution-target fingerprints do not block this migration.
|
||||
|
||||
### Operational Report Scope
|
||||
|
||||
- The migration preserves these prompt IDs:
|
||||
`weather.daily_generated_text`, `weather.today_generated_text`,
|
||||
`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`.
|
||||
- Each operational report definition selects the exact embedded prompt version
|
||||
`1.0.0`; execution does not rely on ambiguous single-version lookup.
|
||||
- Morning and evening batch membership remains based on Today, Tomorrow, and
|
||||
eligible future Daily reports.
|
||||
- Three-day, weekend, and storm are removed from current CLI help, parsing,
|
||||
report registry membership, tests that claim implemented generation, and
|
||||
non-roadmap documentation.
|
||||
- The future product concepts may remain under `docs/roadmap/`, but migration
|
||||
verification does not invent outputs or compare nonexistent prompts.
|
||||
|
||||
### Application Boundary
|
||||
|
||||
- Promptkit remains an adapter boundary even though it runs in process.
|
||||
- A weatherreporter-owned contract represents prompt identity, preparation,
|
||||
execution, output, validation, usage, provenance, and neutral error
|
||||
categories.
|
||||
- The Promptkit adapter maps public Promptkit values into that contract.
|
||||
- App orchestration and test fakes depend on the project-owned contract, not
|
||||
Promptkit.
|
||||
- Scriptorium-specific request, result, error, and generation-mode types are
|
||||
removed rather than renamed and retained.
|
||||
|
||||
### Prompt And Schema Ownership
|
||||
|
||||
- Weatherreporter embeds the four operational prompt definitions, referenced
|
||||
prompt content, shared prompt content, and private response schemas.
|
||||
- Assets remain separate files rather than inline Go strings.
|
||||
- The temporary corpus under `docs/roadmap/scriptorium/` is migration source
|
||||
material, not the final runtime location.
|
||||
- Weatherreporter's existing generated-text domain types, schemas, and
|
||||
templates remain the canonical application contract. Imported Scriptorium
|
||||
assets are reconciled with that contract rather than copied blindly or kept
|
||||
as duplicate runtime schemas.
|
||||
- The imported Daily schema's incorrect Today `$id` and title are corrected.
|
||||
- `confidence` is handled consistently across each prompt, provider-facing
|
||||
schema, generated-text domain type, and template. The existing optional
|
||||
weatherreporter field remains supported unless a separate domain decision
|
||||
removes it.
|
||||
- Prompt input metadata identifies the serialized data package as YAML rather
|
||||
than JSON.
|
||||
- Imported `pipeline-weather/...` schema paths are replaced with paths valid
|
||||
inside the embedded Promptkit schema source.
|
||||
- Imported `repair_attempts: 2` values are removed or set to zero. The
|
||||
migration does not rely on Promptkit's internal-only repair capability.
|
||||
- The unused `weather.daily_report` prompt is not promoted into runtime assets.
|
||||
- One centralized embedded prompt/schema source is sufficient; Weatherreporter
|
||||
does not need Notarius's multi-module asset-flattening registry.
|
||||
|
||||
### Profiles, Backends, And Credentials
|
||||
|
||||
- Execution profiles remain operator-configurable rather than embedded report
|
||||
policy.
|
||||
- Each embedded operational prompt declares Promptkit's built-in
|
||||
`gemini-flash-latest` profile as its default.
|
||||
- `gemini-flash-latest` is intentionally a moving model alias. The execution
|
||||
record captures the effective model identity, but operators who require a
|
||||
pinned model must select an explicit external profile.
|
||||
- Configuration supports at most one external profile source:
|
||||
`promptkit.profile_file` or `promptkit.profile_dir`. The two fields are
|
||||
mutually exclusive.
|
||||
- A nonblank `promptkit.profile` is the explicit request profile for every
|
||||
report in the invocation and takes precedence over each prompt's
|
||||
`default_profile`. A blank value uses the prompt default.
|
||||
- Promptkit's normal profile-source precedence remains intact: an external
|
||||
matching profile takes precedence over an embedded built-in profile, and an
|
||||
invalid matching external profile is an error rather than a reason to fall
|
||||
back.
|
||||
- Weatherreporter exposes Promptkit's conventional `local` backend through the
|
||||
narrow `promptkit.local.endpoint` and
|
||||
`promptkit.local.concurrency_limit` configuration fields. It does not expose
|
||||
arbitrary backend registration.
|
||||
- A configured local endpoint registers the engine-scoped `local` backend. An
|
||||
operator-supplied external profile selects it with `backend: local` and owns
|
||||
the model-specific settings; Weatherreporter does not invent a local model
|
||||
profile.
|
||||
- Local concurrency defaults to one. A value of zero means unlimited, matching
|
||||
Promptkit, and a negative value is invalid. Queue capacity and general
|
||||
backend parameters are not exposed.
|
||||
- Credential values remain in environment variables or file-backed
|
||||
environment secrets. Configuration contains only credential source names.
|
||||
- Provider credentials never appear in logs, errors, CLI output, durable
|
||||
metadata, preparation records, execution records, or debug summaries.
|
||||
- Promptkit `InspectProfile` reports structural target and credential
|
||||
requirements; Weatherreporter owns policy for checking configured
|
||||
environment availability.
|
||||
- Promptkit revalidates environment credentials at `RunPrepared`; a successful
|
||||
preparation does not promise that execution-time credentials remain
|
||||
available.
|
||||
|
||||
### Configuration Contract
|
||||
|
||||
The replacement configuration surface is:
|
||||
|
||||
```yaml
|
||||
promptkit:
|
||||
profile: ""
|
||||
profile_file: ""
|
||||
profile_dir: ""
|
||||
timeout: 2m
|
||||
|
||||
local:
|
||||
endpoint: ""
|
||||
concurrency_limit: 1
|
||||
```
|
||||
|
||||
- `timeout` remains the transport-wide provider-call safety cap.
|
||||
- A blank local endpoint leaves the conventional local backend unregistered.
|
||||
- Scriptorium's `binary`, `config_path`, and `extra_args` settings have no
|
||||
Promptkit equivalents and are removed.
|
||||
- Configuration validation rejects simultaneous `profile_file` and
|
||||
`profile_dir` values, invalid local endpoints, negative concurrency, and
|
||||
selected profiles that cannot resolve their backend.
|
||||
|
||||
### Engine Construction And Inspection
|
||||
|
||||
- One Promptkit engine is constructed per CLI invocation at the application
|
||||
composition boundary.
|
||||
- Single-report generation and every report in a batch use that same engine.
|
||||
- Per-report orchestration does not construct a default engine.
|
||||
- Promptkit backend capacity state and HTTP transport are shared consistently
|
||||
for the invocation.
|
||||
- Before collection, `InspectPrompt` checks every selected report's exact ID
|
||||
and version, declared `data_package` input, default-profile metadata, prompt
|
||||
hash availability, and declared output contract.
|
||||
- `InspectPrompt` is a point-in-time structural check. It does not load a JSON
|
||||
Schema, resolve a profile, or freeze later execution.
|
||||
- Explicit profile overrides and relevant prompt defaults are checked with
|
||||
`InspectProfile` before collection when application policy requires them.
|
||||
- `InspectProfile` is also point-in-time and does not check credential values.
|
||||
- Successful `PrepareExecution`, not inspection, is the per-run authority for
|
||||
loaded schema, rendered content, frozen inputs, effective settings, and
|
||||
durable execution provenance.
|
||||
|
||||
### Prompt Input
|
||||
|
||||
- Promptkit receives only the curated `data_package` produced by
|
||||
`internal/promptinput`.
|
||||
- Weatherreporter serializes the package once, atomically persists those exact
|
||||
bytes, and supplies the same bytes with a Promptkit inline artifact.
|
||||
- The managed data-package path may be supplied as non-secret provenance
|
||||
through the inline artifact URI.
|
||||
- Weatherreporter does not delegate unrestricted path loading to Promptkit's
|
||||
default file artifact reader.
|
||||
- Prompt inspection and adapter tests verify that `data_package` is required
|
||||
and declared with the chosen YAML media type.
|
||||
|
||||
### Prepared Execution
|
||||
|
||||
- `Engine.PrepareExecution` replaces Scriptorium render preflight.
|
||||
- Weatherreporter obtains `PreparedExecution.Details`, maps a safe subset into
|
||||
its own preparation record, and persists that record before calling
|
||||
`Engine.RunPrepared`.
|
||||
- `RunPrepared` executes the frozen prompt, profile, schema, inputs, rendered
|
||||
messages, target, and validation resources retained by the handle.
|
||||
- Every acquired handle is followed immediately by `defer handle.Discard()`.
|
||||
Discard is safe after execution and releases unused private execution state.
|
||||
- Handles remain adapter-local, engine-bound, one-shot, in-process values.
|
||||
They are never serialized, persisted, copied into app contracts, or treated
|
||||
as restartable jobs.
|
||||
- Preparation and execution use independent contexts. Execution receives the
|
||||
active report workflow context.
|
||||
- Capacity is not reserved during preparation. Capacity rejection can
|
||||
therefore occur after a preparation record has been persisted.
|
||||
- `RunPrepared` consumes the handle on success and every operational failure.
|
||||
- Preparation details remain available from the adapter after execution or
|
||||
discard, but rendered messages are not copied into routine durable state.
|
||||
- Promptkit execution timing excludes preparation and consumer-held delay.
|
||||
Weatherreporter records preparation timing and execution timing separately.
|
||||
|
||||
### Execution And Validation
|
||||
|
||||
- All four operational reports use Promptkit JSON Schema output validation.
|
||||
- A completed Promptkit validation rejection returns a `RunResult`; the
|
||||
adapter retains raw output and bounded validation details before failing the
|
||||
report.
|
||||
- An operational generation or validation error returns no partial
|
||||
`RunResult`.
|
||||
- Weatherreporter's `internal/generatedtext` validation remains the final
|
||||
report-specific decode and domain boundary.
|
||||
- Weatherreporter's `internal/reporttemplate` remains responsible for managed
|
||||
Markdown rendering.
|
||||
- Weatherreporter atomically persists Promptkit raw output and later artifacts
|
||||
rather than asking Promptkit to choose managed filesystem paths.
|
||||
- No Promptkit output-repair behavior is assumed or requested.
|
||||
|
||||
## Durable Artifacts And Observability
|
||||
|
||||
Status: Implemented design constraints.
|
||||
|
||||
Routine durable state retains useful non-secret provenance without persisting
|
||||
full rendered prompts.
|
||||
|
||||
The preparation record contains:
|
||||
|
||||
- prompt ID and exact version;
|
||||
- prompt definition hash;
|
||||
- rendered prompt hash;
|
||||
- input hashes;
|
||||
- selected profile and backend identity;
|
||||
- effective model identity;
|
||||
- output contract summary;
|
||||
- preparation start, end, and duration; and
|
||||
- the path of the exact persisted data package.
|
||||
|
||||
The execution record and run metadata contain, when available:
|
||||
|
||||
- Promptkit run ID;
|
||||
- prompt ID, version, and hashes;
|
||||
- input hashes;
|
||||
- selected profile, backend, and model identity;
|
||||
- generated-content hash;
|
||||
- token usage;
|
||||
- execution start, end, and duration;
|
||||
- validation status and bounded diagnostics; and
|
||||
- paths of separately persisted raw output, normalized generated text, render
|
||||
context, managed Markdown, and other artifacts reached by the workflow.
|
||||
|
||||
Provider endpoints, full effective model parameter maps, rendered messages,
|
||||
schema bodies, data-package contents, and generated content do not belong in
|
||||
routine metadata or CLI summaries.
|
||||
|
||||
Rendered messages and other content-rich preparation or response diagnostics
|
||||
are available only when the operator supplies
|
||||
`--llm-debug-dir <path>` to a single-report or batch command.
|
||||
|
||||
- There is no persistent YAML setting for debug capture.
|
||||
- The debug root is validated or created before weather collection or provider
|
||||
work. A requested destination that cannot be secured or written is an error.
|
||||
- Artifacts are grouped beneath
|
||||
`<path>/<report-id>/<valid-date>/<run-id>/`.
|
||||
- Directories and files use owner-only permissions and atomic writes.
|
||||
- Debug artifacts may contain rendered messages and content-rich preparation
|
||||
or response diagnostics, but never credentials.
|
||||
- The debug path appears in command output only when debug capture is enabled;
|
||||
it is not added to routine durable metadata.
|
||||
- Debug artifacts are not cache or comparison inputs. Their retention is owned
|
||||
by the operator who selected the directory.
|
||||
|
||||
### Artifact Identities And Versions
|
||||
|
||||
Weatherreporter replaces Scriptorium-specific artifact identities rather than
|
||||
reusing names whose meanings have changed:
|
||||
|
||||
- `PromptPreparationArtifact` uses schema version
|
||||
`weatherreporter.prompt_preparation.v1`, is written as
|
||||
`prompt_preparation.<runID>.json`, and is referenced by
|
||||
`preparationPath`.
|
||||
- `PromptExecutionArtifact` uses schema version
|
||||
`weatherreporter.prompt_execution.v1`, is written as
|
||||
`prompt_execution.<runID>.json`, and is referenced by `executionPath`.
|
||||
- Run metadata advances to `weatherreporter.metadata.v2` and uses those new
|
||||
path fields.
|
||||
|
||||
Preparation files remain beneath the existing configurable `preflight/`
|
||||
directory, and execution files remain beneath the existing `snapshots/` tree.
|
||||
The stable physical grouping limits deployment disruption without preserving
|
||||
misleading Scriptorium-era filenames or field names. Raw generated output,
|
||||
normalized generated text, render context, managed Markdown, and other
|
||||
artifacts whose meanings have not changed retain their existing names and
|
||||
locations.
|
||||
|
||||
Run inspection remains able to read `weatherreporter.metadata.v1` and its
|
||||
legacy `preflightPath` and `generatedTextResultPath` references. New runs write
|
||||
only the v2 metadata and new artifact names; Weatherreporter does not
|
||||
dual-write deprecated aliases. CLI summary fields adopt `preparationPath` and
|
||||
`executionPath` as an explicit, documented contract change.
|
||||
|
||||
## Failure Contract
|
||||
|
||||
Status: Implemented design constraints.
|
||||
|
||||
- A preparation failure produces a redacted weatherreporter-owned failure
|
||||
receipt with report, RunID, prompt, stage, timing, and classified error
|
||||
context. It does not fabricate Promptkit preparation details.
|
||||
- An operational execution failure retains the successful preparation record
|
||||
and adds a redacted execution failure receipt. No partial Promptkit result or
|
||||
model output is invented.
|
||||
- A Promptkit validation rejection retains the returned result, raw generated
|
||||
output, validation details, and safe provenance before the report fails.
|
||||
- A later generated-text decode, domain-validation, or template failure
|
||||
retains every raw and validated artifact reached before that stage.
|
||||
- Caller cancellation takes precedence when the active workflow context is
|
||||
canceled.
|
||||
- `promptkit.CapacityError` is recognized with `errors.As`; its backend ID is
|
||||
copied into a weatherreporter-owned capacity error while
|
||||
`ErrCapacityExceeded` remains the classification.
|
||||
- Capacity rejection is an operational report failure, not invalid model
|
||||
output, and does not trigger an automatic retry.
|
||||
- Other Promptkit public error identities are translated into the narrow
|
||||
weatherreporter error categories needed by CLI, metadata, and batch
|
||||
behavior. Diagnostic prose is not parsed as a contract.
|
||||
- Single-report commands return the classified failure with available
|
||||
inspectable paths.
|
||||
- Batch runs continue independent later reports under the existing batch
|
||||
failure policy.
|
||||
- Any future retry policy belongs to app orchestration, not the adapter.
|
||||
|
||||
## Compatibility Requirements
|
||||
|
||||
Status: Implemented design constraints.
|
||||
|
||||
- Daily, Today, Tomorrow, and Hourly report IDs, prompt IDs, valid periods,
|
||||
artifact grouping, output names, and Distributor bundle behavior remain
|
||||
stable.
|
||||
- Morning and evening batch collection, planning, ordering, and continuation
|
||||
behavior remains stable.
|
||||
- Module snapshot and Recent Changes behavior remains deterministic.
|
||||
- Promptkit receives only the existing curated prompt-input boundary.
|
||||
- Managed Markdown remains the Distributor upload source.
|
||||
- RunID lookup and inspection remain available for successful and failed runs.
|
||||
- Existing managed paths remain stable where their meaning is unchanged.
|
||||
Scriptorium-specific artifact names or schemas change when retaining them
|
||||
would misrepresent the Promptkit contract.
|
||||
- Existing v1 run metadata and referenced artifacts remain inspectable after
|
||||
the migration. New runs use the v2 metadata and Promptkit-era artifact
|
||||
identities without dual-writing deprecated aliases.
|
||||
- Artifact or metadata schema changes are explicit, documented, and covered by
|
||||
state and inspection tests.
|
||||
- Prompt or generated content is not added to routine logs or CLI summaries.
|
||||
- Tests do not require live providers or credentials.
|
||||
- Removing incomplete three-day, weekend, and storm surfaces is documented as
|
||||
correction of an unfinished product boundary, not as successful Promptkit
|
||||
migration of those reports.
|
||||
|
||||
## Verification And Completion Criteria
|
||||
|
||||
Status: Completed and verified.
|
||||
|
||||
Completion was verified by the following outcomes:
|
||||
|
||||
- the four operational reports inspect, prepare, and execute through Promptkit
|
||||
`v0.4.0` using embedded report-owned assets;
|
||||
- every report uses exact prompt version `1.0.0`, requires the YAML
|
||||
`data_package`, and declares the expected JSON Schema output contract;
|
||||
- prepared execution persists a safe preparation record before provider work
|
||||
and executes the same frozen snapshot;
|
||||
- deterministic offline adapter and app tests cover success, preparation
|
||||
failure, credential revalidation, capacity rejection, cancellation, timeout,
|
||||
generation failure, Promptkit validation rejection, generated-text domain
|
||||
failure, template failure, and handle discard;
|
||||
- morning and evening batches construct one engine and preserve current
|
||||
collection, planning, ordering, continuation, output, and notification
|
||||
behavior;
|
||||
- the temporary corpus has been reconciled into one runtime prompt/schema
|
||||
source without duplicate provider-facing schemas;
|
||||
- configuration examples load and contain no Scriptorium fields;
|
||||
- CLI summaries and inspection commands expose the new project-owned artifact
|
||||
contract without Promptkit types;
|
||||
- Scriptorium code, configuration, tests, and runtime documentation have been
|
||||
removed;
|
||||
- incomplete three-day, weekend, and storm commands, registry entries, tests,
|
||||
and current-behavior documentation have been removed or moved to roadmap
|
||||
scope;
|
||||
- non-roadmap documentation describes only the implemented Promptkit
|
||||
integration;
|
||||
- `go test ./...`, required focused or race-enabled checks, CLI help
|
||||
validation, and `git diff --check` pass; and
|
||||
- no committed `go.work`, local `replace`, live-provider test, or
|
||||
secret-bearing fixture remains.
|
||||
|
||||
Fixture-based comparison with prior Scriptorium behavior is sufficient.
|
||||
Production dual-run is not required because model calls are nondeterministic,
|
||||
costly, and difficult to compare meaningfully.
|
||||
|
||||
## Decision Status
|
||||
|
||||
Status: Completed.
|
||||
|
||||
The roadmap has no remaining open product or architecture questions. Later
|
||||
changes to this completed scope require new roadmap or decision-record scope
|
||||
rather than implicit changes to this historical record.
|
||||
@@ -1,75 +0,0 @@
|
||||
# Troubleshooting
|
||||
|
||||
Start with the command's classified error. When content-rich prompt diagnostics
|
||||
are needed, enable a new run with `--llm-debug-dir` and handle the resulting
|
||||
secure capture as sensitive. Current-version workspace receipts can provide
|
||||
additional context when present, but are transitional state rather than a
|
||||
long-term troubleshooting interface.
|
||||
|
||||
## Prompt inspection or credentials fail before collection
|
||||
|
||||
A prompt/version, contract, selected profile, unsupported direct-key profile,
|
||||
or required environment credential can fail before weather collection. Correct
|
||||
the configured `promptkit` profile or profile source, confirm the exact
|
||||
Promptkit asset is available, and supply any reported environment credential.
|
||||
Do not add provider keys to YAML. See [configuration](config.md).
|
||||
|
||||
## Local profile override is malformed or selects an unexpected model
|
||||
|
||||
`promptkit.profile_file` and `promptkit.profile_dir` supply complete profile
|
||||
definitions. A same-ID definition replaces the embedded profile, and a malformed
|
||||
matching definition fails before collection instead of falling back. Validate
|
||||
the selected profile's YAML, ID, backend or endpoint, and model. If the model
|
||||
is unexpected, first check the global `promptkit.profile` selection and then
|
||||
look for a same-ID definition in the configured file or directory.
|
||||
|
||||
Current-version preparation and execution receipts may retain the selected
|
||||
profile ID and effective backend/model, but not an endpoint or credential.
|
||||
Use them only as supplemental context after the active command error or an
|
||||
explicit secure debug capture. See the maintained
|
||||
[local `weather-light` profile example](../examples/weather-light-local-profile.yml).
|
||||
|
||||
## Local model endpoint is unavailable
|
||||
|
||||
An endpoint-only `weather-light` override can pass preflight and still fail
|
||||
during provider preparation or execution when the local server is unavailable
|
||||
or does not accept the configured model. Start the local server, correct the
|
||||
endpoint or model in the profile, and run the command again. Weatherreporter
|
||||
does not probe endpoints or automatically use a remote profile instead.
|
||||
|
||||
## Preparation, capacity, or execution fails
|
||||
|
||||
A preparation failure occurs before provider work; an execution failure occurs
|
||||
after preparation. A capacity error for one batch report does not retry that
|
||||
report or prevent later independent reports. Correct the profile or backend
|
||||
condition identified by the bounded command error, then create a new run.
|
||||
Use explicit secure debug capture only when additional content-rich diagnostics
|
||||
are necessary. See [operations](operations.md).
|
||||
|
||||
## Generated text fails validation
|
||||
|
||||
Raw generated output may be saved but Markdown is not rendered when the JSON
|
||||
does not match the report schema. Correct the Promptkit prompt/profile behavior
|
||||
or the matching schema and validator in source control; do not edit raw output
|
||||
to treat it as validated. See [templates](templates.md).
|
||||
|
||||
## Debug capture fails
|
||||
|
||||
`--llm-debug-dir` must be an absolute secure directory outside workspace state.
|
||||
A debug-write failure stops the affected report to avoid continuing without the
|
||||
requested diagnostic. Repair the named path's ownership or permissions, then
|
||||
rerun. Treat capture files as sensitive. See [operations](operations.md).
|
||||
|
||||
## Weather, state, output, or notification fails
|
||||
|
||||
Collection errors precede planning. Later filesystem, output-copy, template,
|
||||
or Distributor errors retain the reached safe paths in the summary. Repair only
|
||||
the reported endpoint or path, leave successful managed reports intact, and
|
||||
rerun the affected report or batch. A batch notification is intentionally
|
||||
skipped when any report item fails.
|
||||
|
||||
## Secrets cannot be loaded
|
||||
|
||||
Secret files must be regular non-symlink files directly beneath
|
||||
`secrets.directory` with valid environment-variable basenames. Correct the
|
||||
reported file or directory without placing secret values in YAML.
|
||||
Reference in New Issue
Block a user