Correct curated prompt package contracts

This commit is contained in:
2026-08-13 00:11:31 +00:00
parent 0b869af75e
commit 5139c1a586
6 changed files with 210 additions and 19 deletions

View File

@@ -27,7 +27,7 @@ A source falls through only when the selected ID is absent. Each source supplies
Profiles that require a direct API key are unsupported; a profile that reports `APIKeyEnv` requires a nonblank value in that environment variable. Active results retain the selected logical profile ID and resolved backend and model. Ordinary errors, summaries, logs, and outputs exclude endpoints, credentials, rendered messages, schemas, request bodies, response bodies, and complete parameter maps.
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions.
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions.
## Comparison Execution

View File

@@ -4,7 +4,7 @@
## Package Construction
`Build` produces `weatherreporter.data_package.v4`. It copies the run ID; report ID, variant, prompt ID, generation time, timezone, local current date, and valid period; ordered briefing stanzas; and source warnings. Prompt input contains no historical comparison section.
`Build` produces `weatherreporter.data_package.v4`. It copies the run ID; report ID, variant, prompt ID, generation time, timezone, local current date, and valid period; ordered briefing stanzas; and prompt-safe source-warning summaries. Warning summaries include only source, code, severity, message, and completeness impact; raw transport and provenance fields such as endpoints never cross into the provider input. Prompt input contains no historical comparison section.
Briefing is a flat ordered set of stanza values. `Build` uses each output's `DataPackageValue`, so curated prompt exports take precedence and rich values are used only as a fallback. Prompt exports are selected by the [briefing registry](briefing.md), while the rich-versus-prompt contract is in [module internals](module.md).

View File

@@ -4,31 +4,31 @@ Your analysis will be incorporated into a structured, user-facing report. The re
# SOURCE ROLES AND WEIGHTING
Use `report` and `briefing.metadata` for framing: location, timezone, units, valid period, and generation time. Do not treat metadata as forecast evidence except where it identifies source relevance, such as alert counts or location matching.
Use `report` and `briefing.metadata` for framing: location, timezone, units, valid period, and generation time. Do not treat metadata as forecast evidence.
For weather interpretation, think in four source layers, in this order:
## 1. Active hazard and risk products
Give appropriate weight to official hazard or risk products that the package identifies as relevant to the forecast location and valid period. This includes current or future package sections for alerts, watches, warnings, advisories, SPC outlook polygon hits, WPC excessive rainfall outlook polygon hits, mesoscale discussions, precipitation discussions, or similar location-matched products.
Give appropriate weight to official hazard or risk products while preserving their stated geographic relevance and valid period. This includes current or future package sections for alerts, watches, warnings, advisories, SPC outlooks, WPC excessive rainfall outlooks, mesoscale discussions, precipitation discussions, or similar products.
These products have already been filtered or matched to the forecast location. Treat them as locally relevant, but distinguish product strength:
Treat location-matched conclusions as locally relevant, but distinguish product strength:
- Active warnings are urgent and should dominate the lead and relevant sections.
- Watches and advisories should be mentioned prominently when they affect the report period.
- Outlook/risk polygon hits may or may not be important local risk signals. Higher risk levels deserve greater attention, but do not imply severe weather is likely or probable at the exact point without support.
- Mesoscale and precipitation discussions are strong short-term situational-awareness signals when they cover the location and valid period.
- Higher location-matched risk levels deserve greater attention, but do not imply severe weather is likely or probable at the exact point without support.
- Mesoscale and precipitation discussions are strong short-term situational-awareness signals only when their stated coverage includes the location and valid period.
For the current schema, use `briefing.applicable_risk_products.alert_digest` and `briefing.metadata.alerts` to determine whether relevant local alerts exist. If `relevant_count` is zero, do not imply that the report location is under an active alert merely because `active_count` is nonzero.
For the current schema, use `briefing.applicable_risk_products.alert_digest.relevant_count` and `briefing.applicable_risk_products.spc_convective_outlooks.risk_digest` as the location-matched local conclusions. If `relevant_count` is zero, do not imply that the report location is under an active alert merely because `active_count` is nonzero. Individual records in `briefing.applicable_risk_products.spc_convective_outlooks.outlooks` and all `briefing.narrative_products.spc_convective_discussion.discussions` are regional context unless their own geography establishes point relevance; never present them as point-local hazards solely because they are included.
## 2. Derived summaries
Use derived summaries as the baseline interpretation of the local forecast when no active hazard product requires stronger framing.
- Use `briefing.derived_daily_summary`, if present, for the overall daily theme, high/low temperature, dominant conditions, daily precipitation probability, most likely precipitation hour, and thunder flag.
- Use `briefing.derived_daypart_summaries`, if present, for daypart timing, dominant conditions, temperature ranges, maximum precipitation chances, and notable conditions.
- Use `briefing.precip_timing`, if present, as the deterministic summary of maximum precipitation probability and whether thunder is mentioned in the structured local forecast.
- Use `briefing.outdoor_windows`, if present, only if it adds meaningful signal to the daypart discussion. Do not turn the report into outdoor-planning advice.
- Use `briefing.derived_summaries.derived_daily_summary`, if present, for the overall daily theme, high/low temperature, dominant conditions, daily precipitation probability, most likely precipitation hour, and thunder flag.
- Use `briefing.derived_summaries.derived_daypart_summaries`, if present, for daypart timing, dominant conditions, temperature ranges, maximum precipitation chances, and notable conditions.
- Use `briefing.derived_summaries.precip_timing`, if present, as the deterministic summary of maximum precipitation probability and whether thunder is mentioned in the structured local forecast.
- Use `briefing.derived_summaries.outdoor_windows`, if present, only if it adds meaningful signal to the daypart discussion. Do not turn the report into outdoor-planning advice.
## 3. Narrative products

View File

@@ -4,11 +4,17 @@ import (
"context"
"encoding/json"
"io/fs"
"regexp"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/promptkit"
"gitea.maximumdirect.net/eric/weatherreporter/internal/module"
"gitea.maximumdirect.net/eric/weatherreporter/internal/promptassets"
"gitea.maximumdirect.net/eric/weatherreporter/internal/promptinput"
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
"gitea.maximumdirect.net/eric/weatherreporter/internal/timeutil"
"gopkg.in/yaml.v3"
)
@@ -16,7 +22,10 @@ type promptDefinition struct {
ID string `yaml:"id"`
Version string `yaml:"version"`
DefaultProfile string `yaml:"default_profile"`
Inputs []struct {
Messages []struct {
ContentFile string `yaml:"content_file"`
} `yaml:"messages"`
Inputs []struct {
Name string `yaml:"name"`
Required bool `yaml:"required"`
ContentType string `yaml:"content_type"`
@@ -71,6 +80,15 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
if definition.ID != tc.id || definition.Version != "2.0.0" || definition.DefaultProfile != tc.profile {
t.Fatalf("definition = %#v, want %s version 2.0.0 and profile %s", definition, tc.id, tc.profile)
}
sharedInstruction := false
for _, message := range definition.Messages {
if message.ContentFile == "../common/data_package.user.md" {
sharedInstruction = true
}
}
if !sharedInstruction {
t.Fatalf("definition messages = %#v, want shared data-package instruction", definition.Messages)
}
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
}
@@ -84,6 +102,105 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
}
}
func TestSharedPromptReferencesSerializedPathsAndPreservesHazardLocality(t *testing.T) {
sharedPrompt, err := fs.ReadFile(promptassets.PromptFS(), "common/data_package.user.md")
if err != nil {
t.Fatalf("read shared prompt: %v", err)
}
pkgYAML := representativeDataPackageYAML(t)
paths := regexp.MustCompile(`briefing(?:\.[a-z_]+)+`).FindAllString(string(sharedPrompt), -1)
if len(paths) == 0 {
t.Fatal("shared prompt does not reference briefing paths")
}
for _, path := range paths {
if !yamlPathExists(t, pkgYAML, path) {
t.Fatalf("shared prompt references path %q that is absent from representative data package:\n%s", path, pkgYAML)
}
}
for _, obsoletePath := range []string{
"briefing.metadata.alerts",
"briefing.derived_daily_summary",
"briefing.derived_daypart_summaries",
"briefing.precip_timing",
"briefing.outdoor_windows",
} {
if strings.Contains(string(sharedPrompt), obsoletePath) {
t.Fatalf("shared prompt references obsolete path %q", obsoletePath)
}
}
for _, requiredGuidance := range []string{
"location-matched local conclusions",
"regional context unless their own geography establishes point relevance",
"never present them as point-local hazards solely because they are included",
} {
if !strings.Contains(string(sharedPrompt), requiredGuidance) {
t.Fatalf("shared prompt is missing hazard-locality guidance %q", requiredGuidance)
}
}
}
func representativeDataPackageYAML(t *testing.T) []byte {
t.Helper()
generatedAt := time.Date(2026, 5, 29, 10, 0, 0, 0, time.UTC)
snapshot, err := module.NewSnapshot([]module.Output{
{ID: module.Metadata, StanzaName: "metadata", Value: map[string]any{"location": "Testville"}},
{ID: module.AlertDigest, StanzaName: "alert_digest", Value: map[string]any{"relevant_count": 1}},
{ID: module.SPCConvectiveOutlooks, StanzaName: "spc_convective_outlooks", Value: map[string]any{"risk_digest": []any{}, "outlooks": []any{}}},
{ID: module.DerivedDailySummary, StanzaName: "derived_daily_summary", Value: map[string]any{"theme": "dry"}},
{ID: module.DerivedDaypartSummaries, StanzaName: "derived_daypart_summaries", Value: map[string]any{"dayparts": []any{}}},
{ID: module.PrecipTiming, StanzaName: "precip_timing", Value: map[string]any{"maximum_probability": 0}},
{ID: module.OutdoorWindows, StanzaName: "outdoor_windows", Value: map[string]any{"windows": []any{}}},
{ID: module.NarrativeForecast, StanzaName: "narrative_forecast", Value: map[string]any{"periods": []any{}}},
{ID: module.AreaForecastDiscussion, StanzaName: "area_forecast_discussion", Value: map[string]any{"key_messages": []any{}, "short_term": "", "long_term": ""}},
{ID: module.SPCConvectiveDiscussion, StanzaName: "spc_convective_discussion", Value: map[string]any{"discussions": []any{}}},
{ID: module.WeatherStory, StanzaName: "weather_story", Value: map[string]any{}},
{ID: module.CurrentConditions, StanzaName: "current_conditions", Value: map[string]any{"temperature": 72}},
{ID: module.HourlyForecast, StanzaName: "hourly_forecast", Value: map[string]any{"periods": []any{}}},
})
if err != nil {
t.Fatalf("NewSnapshot() error = %v", err)
}
pkg, err := promptinput.Build(promptinput.BuildRequest{
Metadata: promptinput.Metadata{
RunID: "20260529T100000Z_daily",
ReportID: report.Daily,
PromptID: "weather.daily_generated_text",
GeneratedAt: generatedAt,
Timezone: "America/Chicago",
ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)},
},
Modules: snapshot,
})
if err != nil {
t.Fatalf("Build() error = %v", err)
}
data, err := promptinput.MarshalYAML(pkg)
if err != nil {
t.Fatalf("MarshalYAML() error = %v", err)
}
return data
}
func yamlPathExists(t *testing.T, data []byte, path string) bool {
t.Helper()
var document map[string]any
if err := yaml.Unmarshal(data, &document); err != nil {
t.Fatalf("decode representative data package: %v", err)
}
var current any = document
for _, segment := range strings.Split(path, ".") {
mapping, ok := current.(map[string]any)
if !ok {
return false
}
current, ok = mapping[segment]
if !ok {
return false
}
}
return true
}
func TestSchemasAreCanonicalAndIndependent(t *testing.T) {
for _, id := range []string{"daily", "today", "tomorrow", "hourly"} {
t.Run(id, func(t *testing.T) {

View File

@@ -67,11 +67,21 @@ type Metadata struct {
}
type Package struct {
SchemaVersion string `json:"schemaVersion" yaml:"schema_version"`
RunID string `json:"runId" yaml:"run_id"`
Report Report `json:"report" yaml:"report"`
Briefing BriefingStanzas `json:"briefing" yaml:"briefing"`
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty" yaml:"source_warnings,omitempty"`
SchemaVersion string `json:"schemaVersion" yaml:"schema_version"`
RunID string `json:"runId" yaml:"run_id"`
Report Report `json:"report" yaml:"report"`
Briefing BriefingStanzas `json:"briefing" yaml:"briefing"`
SourceWarnings []SourceWarningSummary `json:"sourceWarnings,omitempty" yaml:"source_warnings,omitempty"`
}
// SourceWarningSummary is the reviewed warning projection supplied to Promptkit.
// Transport and provenance details intentionally remain outside the prompt package.
type SourceWarningSummary struct {
Source string `json:"source" yaml:"source"`
Code string `json:"code" yaml:"code"`
Severity string `json:"severity" yaml:"severity"`
Message string `json:"message" yaml:"message"`
CompletenessImpact string `json:"completenessImpact,omitempty" yaml:"completeness_impact,omitempty"`
}
type Report struct {
@@ -107,7 +117,7 @@ func Build(req BuildRequest) (Package, error) {
ValidPeriod: req.Metadata.ValidPeriod,
},
Briefing: stanzasFromSnapshot(req.Modules),
SourceWarnings: append([]weatherdata.SourceWarning(nil), req.Metadata.SourceWarnings...),
SourceWarnings: sourceWarningSummaries(req.Metadata.SourceWarnings),
}
if err := Validate(pkg); err != nil {
return Package{}, err
@@ -115,6 +125,20 @@ func Build(req BuildRequest) (Package, error) {
return pkg, nil
}
func sourceWarningSummaries(warnings []weatherdata.SourceWarning) []SourceWarningSummary {
out := make([]SourceWarningSummary, 0, len(warnings))
for _, warning := range warnings {
out = append(out, SourceWarningSummary{
Source: warning.Source,
Code: warning.Code,
Severity: warning.Severity,
Message: warning.Message,
CompletenessImpact: warning.CompletenessImpact,
})
}
return out
}
func stanzasFromSnapshot(snapshot module.Snapshot) BriefingStanzas {
values := map[string]any{}
order := make([]string, 0, len(snapshot.Outputs))

View File

@@ -1,6 +1,7 @@
package promptinput
import (
"math"
"strings"
"testing"
"time"
@@ -8,6 +9,7 @@ import (
"gitea.maximumdirect.net/eric/weatherreporter/internal/module"
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
"gitea.maximumdirect.net/eric/weatherreporter/internal/timeutil"
"gitea.maximumdirect.net/eric/weatherreporter/internal/weatherdata"
)
func TestBuildDailyDataPackage(t *testing.T) {
@@ -161,6 +163,37 @@ func TestBuildUsesPromptValuesFromSnapshot(t *testing.T) {
}
}
func TestMarshalYAMLProjectsSourceWarningsWithoutTransportDetails(t *testing.T) {
req := validBuildRequest(t)
req.Metadata.SourceWarnings = []weatherdata.SourceWarning{{
Source: "hourly",
Code: "missing_source",
Severity: "warning",
Message: "hourly source is unavailable",
Endpoint: "/transport/path/only-for-this-test",
CompletenessImpact: "source omitted",
}}
pkg, err := Build(req)
if err != nil {
t.Fatalf("Build() error = %v", err)
}
data, err := MarshalYAML(pkg)
if err != nil {
t.Fatalf("MarshalYAML() error = %v", err)
}
text := string(data)
for _, safeDetail := range []string{"source_warnings:", "source: hourly", "code: missing_source", "severity: warning", "message: hourly source is unavailable", "completeness_impact: source omitted"} {
if !strings.Contains(text, safeDetail) {
t.Fatalf("YAML output missing safe warning detail %q:\n%s", safeDetail, text)
}
}
if strings.Contains(text, "/transport/path/only-for-this-test") || strings.Contains(text, "endpoint:") {
t.Fatalf("YAML output contains raw warning transport details:\n%s", text)
}
}
func TestMarshalYAMLIsDeterministicAndGroupsNamedStanzas(t *testing.T) {
pkg, err := Build(validBuildRequest(t))
if err != nil {
@@ -287,6 +320,23 @@ func TestMarshalYAMLRejectsUncategorizedStanza(t *testing.T) {
}
}
func TestMarshalYAMLAttributesStanzaSerializationFailures(t *testing.T) {
req := validBuildRequest(t)
req.Modules = snapshotWithOutputs(t,
module.Output{ID: module.Metadata, StanzaName: "metadata", Value: map[string]string{"run_id": req.Metadata.RunID}},
module.Output{ID: module.CurrentConditions, StanzaName: "current_conditions", Value: map[string]float64{"temperature": math.Inf(1)}},
)
pkg, err := Build(req)
if err != nil {
t.Fatalf("Build() error = %v", err)
}
_, err = MarshalYAML(pkg)
if err == nil || !strings.Contains(err.Error(), `marshal briefing stanza "current_conditions"`) {
t.Fatalf("MarshalYAML() error = %v, want current_conditions stanza context", err)
}
}
func TestLoadYAMLRejectsMisplacedStanza(t *testing.T) {
data := []byte(`
schema_version: weatherreporter.data_package.v4