Correct curated prompt package contracts
This commit is contained in:
@@ -27,7 +27,7 @@ A source falls through only when the selected ID is absent. Each source supplies
|
||||
|
||||
Profiles that require a direct API key are unsupported; a profile that reports `APIKeyEnv` requires a nonblank value in that environment variable. Active results retain the selected logical profile ID and resolved backend and model. Ordinary errors, summaries, logs, and outputs exclude endpoints, credentials, rendered messages, schemas, request bodies, response bodies, and complete parameter maps.
|
||||
|
||||
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions.
|
||||
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions.
|
||||
|
||||
## Comparison Execution
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
## Package Construction
|
||||
|
||||
`Build` produces `weatherreporter.data_package.v4`. It copies the run ID; report ID, variant, prompt ID, generation time, timezone, local current date, and valid period; ordered briefing stanzas; and source warnings. Prompt input contains no historical comparison section.
|
||||
`Build` produces `weatherreporter.data_package.v4`. It copies the run ID; report ID, variant, prompt ID, generation time, timezone, local current date, and valid period; ordered briefing stanzas; and prompt-safe source-warning summaries. Warning summaries include only source, code, severity, message, and completeness impact; raw transport and provenance fields such as endpoints never cross into the provider input. Prompt input contains no historical comparison section.
|
||||
|
||||
Briefing is a flat ordered set of stanza values. `Build` uses each output's `DataPackageValue`, so curated prompt exports take precedence and rich values are used only as a fallback. Prompt exports are selected by the [briefing registry](briefing.md), while the rich-versus-prompt contract is in [module internals](module.md).
|
||||
|
||||
|
||||
@@ -4,31 +4,31 @@ Your analysis will be incorporated into a structured, user-facing report. The re
|
||||
|
||||
# SOURCE ROLES AND WEIGHTING
|
||||
|
||||
Use `report` and `briefing.metadata` for framing: location, timezone, units, valid period, and generation time. Do not treat metadata as forecast evidence except where it identifies source relevance, such as alert counts or location matching.
|
||||
Use `report` and `briefing.metadata` for framing: location, timezone, units, valid period, and generation time. Do not treat metadata as forecast evidence.
|
||||
|
||||
For weather interpretation, think in four source layers, in this order:
|
||||
|
||||
## 1. Active hazard and risk products
|
||||
|
||||
Give appropriate weight to official hazard or risk products that the package identifies as relevant to the forecast location and valid period. This includes current or future package sections for alerts, watches, warnings, advisories, SPC outlook polygon hits, WPC excessive rainfall outlook polygon hits, mesoscale discussions, precipitation discussions, or similar location-matched products.
|
||||
Give appropriate weight to official hazard or risk products while preserving their stated geographic relevance and valid period. This includes current or future package sections for alerts, watches, warnings, advisories, SPC outlooks, WPC excessive rainfall outlooks, mesoscale discussions, precipitation discussions, or similar products.
|
||||
|
||||
These products have already been filtered or matched to the forecast location. Treat them as locally relevant, but distinguish product strength:
|
||||
Treat location-matched conclusions as locally relevant, but distinguish product strength:
|
||||
|
||||
- Active warnings are urgent and should dominate the lead and relevant sections.
|
||||
- Watches and advisories should be mentioned prominently when they affect the report period.
|
||||
- Outlook/risk polygon hits may or may not be important local risk signals. Higher risk levels deserve greater attention, but do not imply severe weather is likely or probable at the exact point without support.
|
||||
- Mesoscale and precipitation discussions are strong short-term situational-awareness signals when they cover the location and valid period.
|
||||
- Higher location-matched risk levels deserve greater attention, but do not imply severe weather is likely or probable at the exact point without support.
|
||||
- Mesoscale and precipitation discussions are strong short-term situational-awareness signals only when their stated coverage includes the location and valid period.
|
||||
|
||||
For the current schema, use `briefing.applicable_risk_products.alert_digest` and `briefing.metadata.alerts` to determine whether relevant local alerts exist. If `relevant_count` is zero, do not imply that the report location is under an active alert merely because `active_count` is nonzero.
|
||||
For the current schema, use `briefing.applicable_risk_products.alert_digest.relevant_count` and `briefing.applicable_risk_products.spc_convective_outlooks.risk_digest` as the location-matched local conclusions. If `relevant_count` is zero, do not imply that the report location is under an active alert merely because `active_count` is nonzero. Individual records in `briefing.applicable_risk_products.spc_convective_outlooks.outlooks` and all `briefing.narrative_products.spc_convective_discussion.discussions` are regional context unless their own geography establishes point relevance; never present them as point-local hazards solely because they are included.
|
||||
|
||||
## 2. Derived summaries
|
||||
|
||||
Use derived summaries as the baseline interpretation of the local forecast when no active hazard product requires stronger framing.
|
||||
|
||||
- Use `briefing.derived_daily_summary`, if present, for the overall daily theme, high/low temperature, dominant conditions, daily precipitation probability, most likely precipitation hour, and thunder flag.
|
||||
- Use `briefing.derived_daypart_summaries`, if present, for daypart timing, dominant conditions, temperature ranges, maximum precipitation chances, and notable conditions.
|
||||
- Use `briefing.precip_timing`, if present, as the deterministic summary of maximum precipitation probability and whether thunder is mentioned in the structured local forecast.
|
||||
- Use `briefing.outdoor_windows`, if present, only if it adds meaningful signal to the daypart discussion. Do not turn the report into outdoor-planning advice.
|
||||
- Use `briefing.derived_summaries.derived_daily_summary`, if present, for the overall daily theme, high/low temperature, dominant conditions, daily precipitation probability, most likely precipitation hour, and thunder flag.
|
||||
- Use `briefing.derived_summaries.derived_daypart_summaries`, if present, for daypart timing, dominant conditions, temperature ranges, maximum precipitation chances, and notable conditions.
|
||||
- Use `briefing.derived_summaries.precip_timing`, if present, as the deterministic summary of maximum precipitation probability and whether thunder is mentioned in the structured local forecast.
|
||||
- Use `briefing.derived_summaries.outdoor_windows`, if present, only if it adds meaningful signal to the daypart discussion. Do not turn the report into outdoor-planning advice.
|
||||
|
||||
## 3. Narrative products
|
||||
|
||||
|
||||
@@ -4,11 +4,17 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io/fs"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/module"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/promptassets"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/promptinput"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/timeutil"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
@@ -16,7 +22,10 @@ type promptDefinition struct {
|
||||
ID string `yaml:"id"`
|
||||
Version string `yaml:"version"`
|
||||
DefaultProfile string `yaml:"default_profile"`
|
||||
Inputs []struct {
|
||||
Messages []struct {
|
||||
ContentFile string `yaml:"content_file"`
|
||||
} `yaml:"messages"`
|
||||
Inputs []struct {
|
||||
Name string `yaml:"name"`
|
||||
Required bool `yaml:"required"`
|
||||
ContentType string `yaml:"content_type"`
|
||||
@@ -71,6 +80,15 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
|
||||
if definition.ID != tc.id || definition.Version != "2.0.0" || definition.DefaultProfile != tc.profile {
|
||||
t.Fatalf("definition = %#v, want %s version 2.0.0 and profile %s", definition, tc.id, tc.profile)
|
||||
}
|
||||
sharedInstruction := false
|
||||
for _, message := range definition.Messages {
|
||||
if message.ContentFile == "../common/data_package.user.md" {
|
||||
sharedInstruction = true
|
||||
}
|
||||
}
|
||||
if !sharedInstruction {
|
||||
t.Fatalf("definition messages = %#v, want shared data-package instruction", definition.Messages)
|
||||
}
|
||||
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
|
||||
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
|
||||
}
|
||||
@@ -84,6 +102,105 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSharedPromptReferencesSerializedPathsAndPreservesHazardLocality(t *testing.T) {
|
||||
sharedPrompt, err := fs.ReadFile(promptassets.PromptFS(), "common/data_package.user.md")
|
||||
if err != nil {
|
||||
t.Fatalf("read shared prompt: %v", err)
|
||||
}
|
||||
pkgYAML := representativeDataPackageYAML(t)
|
||||
paths := regexp.MustCompile(`briefing(?:\.[a-z_]+)+`).FindAllString(string(sharedPrompt), -1)
|
||||
if len(paths) == 0 {
|
||||
t.Fatal("shared prompt does not reference briefing paths")
|
||||
}
|
||||
for _, path := range paths {
|
||||
if !yamlPathExists(t, pkgYAML, path) {
|
||||
t.Fatalf("shared prompt references path %q that is absent from representative data package:\n%s", path, pkgYAML)
|
||||
}
|
||||
}
|
||||
for _, obsoletePath := range []string{
|
||||
"briefing.metadata.alerts",
|
||||
"briefing.derived_daily_summary",
|
||||
"briefing.derived_daypart_summaries",
|
||||
"briefing.precip_timing",
|
||||
"briefing.outdoor_windows",
|
||||
} {
|
||||
if strings.Contains(string(sharedPrompt), obsoletePath) {
|
||||
t.Fatalf("shared prompt references obsolete path %q", obsoletePath)
|
||||
}
|
||||
}
|
||||
for _, requiredGuidance := range []string{
|
||||
"location-matched local conclusions",
|
||||
"regional context unless their own geography establishes point relevance",
|
||||
"never present them as point-local hazards solely because they are included",
|
||||
} {
|
||||
if !strings.Contains(string(sharedPrompt), requiredGuidance) {
|
||||
t.Fatalf("shared prompt is missing hazard-locality guidance %q", requiredGuidance)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func representativeDataPackageYAML(t *testing.T) []byte {
|
||||
t.Helper()
|
||||
generatedAt := time.Date(2026, 5, 29, 10, 0, 0, 0, time.UTC)
|
||||
snapshot, err := module.NewSnapshot([]module.Output{
|
||||
{ID: module.Metadata, StanzaName: "metadata", Value: map[string]any{"location": "Testville"}},
|
||||
{ID: module.AlertDigest, StanzaName: "alert_digest", Value: map[string]any{"relevant_count": 1}},
|
||||
{ID: module.SPCConvectiveOutlooks, StanzaName: "spc_convective_outlooks", Value: map[string]any{"risk_digest": []any{}, "outlooks": []any{}}},
|
||||
{ID: module.DerivedDailySummary, StanzaName: "derived_daily_summary", Value: map[string]any{"theme": "dry"}},
|
||||
{ID: module.DerivedDaypartSummaries, StanzaName: "derived_daypart_summaries", Value: map[string]any{"dayparts": []any{}}},
|
||||
{ID: module.PrecipTiming, StanzaName: "precip_timing", Value: map[string]any{"maximum_probability": 0}},
|
||||
{ID: module.OutdoorWindows, StanzaName: "outdoor_windows", Value: map[string]any{"windows": []any{}}},
|
||||
{ID: module.NarrativeForecast, StanzaName: "narrative_forecast", Value: map[string]any{"periods": []any{}}},
|
||||
{ID: module.AreaForecastDiscussion, StanzaName: "area_forecast_discussion", Value: map[string]any{"key_messages": []any{}, "short_term": "", "long_term": ""}},
|
||||
{ID: module.SPCConvectiveDiscussion, StanzaName: "spc_convective_discussion", Value: map[string]any{"discussions": []any{}}},
|
||||
{ID: module.WeatherStory, StanzaName: "weather_story", Value: map[string]any{}},
|
||||
{ID: module.CurrentConditions, StanzaName: "current_conditions", Value: map[string]any{"temperature": 72}},
|
||||
{ID: module.HourlyForecast, StanzaName: "hourly_forecast", Value: map[string]any{"periods": []any{}}},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewSnapshot() error = %v", err)
|
||||
}
|
||||
pkg, err := promptinput.Build(promptinput.BuildRequest{
|
||||
Metadata: promptinput.Metadata{
|
||||
RunID: "20260529T100000Z_daily",
|
||||
ReportID: report.Daily,
|
||||
PromptID: "weather.daily_generated_text",
|
||||
GeneratedAt: generatedAt,
|
||||
Timezone: "America/Chicago",
|
||||
ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)},
|
||||
},
|
||||
Modules: snapshot,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
data, err := promptinput.MarshalYAML(pkg)
|
||||
if err != nil {
|
||||
t.Fatalf("MarshalYAML() error = %v", err)
|
||||
}
|
||||
return data
|
||||
}
|
||||
|
||||
func yamlPathExists(t *testing.T, data []byte, path string) bool {
|
||||
t.Helper()
|
||||
var document map[string]any
|
||||
if err := yaml.Unmarshal(data, &document); err != nil {
|
||||
t.Fatalf("decode representative data package: %v", err)
|
||||
}
|
||||
var current any = document
|
||||
for _, segment := range strings.Split(path, ".") {
|
||||
mapping, ok := current.(map[string]any)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
current, ok = mapping[segment]
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func TestSchemasAreCanonicalAndIndependent(t *testing.T) {
|
||||
for _, id := range []string{"daily", "today", "tomorrow", "hourly"} {
|
||||
t.Run(id, func(t *testing.T) {
|
||||
|
||||
@@ -67,11 +67,21 @@ type Metadata struct {
|
||||
}
|
||||
|
||||
type Package struct {
|
||||
SchemaVersion string `json:"schemaVersion" yaml:"schema_version"`
|
||||
RunID string `json:"runId" yaml:"run_id"`
|
||||
Report Report `json:"report" yaml:"report"`
|
||||
Briefing BriefingStanzas `json:"briefing" yaml:"briefing"`
|
||||
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty" yaml:"source_warnings,omitempty"`
|
||||
SchemaVersion string `json:"schemaVersion" yaml:"schema_version"`
|
||||
RunID string `json:"runId" yaml:"run_id"`
|
||||
Report Report `json:"report" yaml:"report"`
|
||||
Briefing BriefingStanzas `json:"briefing" yaml:"briefing"`
|
||||
SourceWarnings []SourceWarningSummary `json:"sourceWarnings,omitempty" yaml:"source_warnings,omitempty"`
|
||||
}
|
||||
|
||||
// SourceWarningSummary is the reviewed warning projection supplied to Promptkit.
|
||||
// Transport and provenance details intentionally remain outside the prompt package.
|
||||
type SourceWarningSummary struct {
|
||||
Source string `json:"source" yaml:"source"`
|
||||
Code string `json:"code" yaml:"code"`
|
||||
Severity string `json:"severity" yaml:"severity"`
|
||||
Message string `json:"message" yaml:"message"`
|
||||
CompletenessImpact string `json:"completenessImpact,omitempty" yaml:"completeness_impact,omitempty"`
|
||||
}
|
||||
|
||||
type Report struct {
|
||||
@@ -107,7 +117,7 @@ func Build(req BuildRequest) (Package, error) {
|
||||
ValidPeriod: req.Metadata.ValidPeriod,
|
||||
},
|
||||
Briefing: stanzasFromSnapshot(req.Modules),
|
||||
SourceWarnings: append([]weatherdata.SourceWarning(nil), req.Metadata.SourceWarnings...),
|
||||
SourceWarnings: sourceWarningSummaries(req.Metadata.SourceWarnings),
|
||||
}
|
||||
if err := Validate(pkg); err != nil {
|
||||
return Package{}, err
|
||||
@@ -115,6 +125,20 @@ func Build(req BuildRequest) (Package, error) {
|
||||
return pkg, nil
|
||||
}
|
||||
|
||||
func sourceWarningSummaries(warnings []weatherdata.SourceWarning) []SourceWarningSummary {
|
||||
out := make([]SourceWarningSummary, 0, len(warnings))
|
||||
for _, warning := range warnings {
|
||||
out = append(out, SourceWarningSummary{
|
||||
Source: warning.Source,
|
||||
Code: warning.Code,
|
||||
Severity: warning.Severity,
|
||||
Message: warning.Message,
|
||||
CompletenessImpact: warning.CompletenessImpact,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func stanzasFromSnapshot(snapshot module.Snapshot) BriefingStanzas {
|
||||
values := map[string]any{}
|
||||
order := make([]string, 0, len(snapshot.Outputs))
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package promptinput
|
||||
|
||||
import (
|
||||
"math"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -8,6 +9,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/module"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/timeutil"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/weatherdata"
|
||||
)
|
||||
|
||||
func TestBuildDailyDataPackage(t *testing.T) {
|
||||
@@ -161,6 +163,37 @@ func TestBuildUsesPromptValuesFromSnapshot(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestMarshalYAMLProjectsSourceWarningsWithoutTransportDetails(t *testing.T) {
|
||||
req := validBuildRequest(t)
|
||||
req.Metadata.SourceWarnings = []weatherdata.SourceWarning{{
|
||||
Source: "hourly",
|
||||
Code: "missing_source",
|
||||
Severity: "warning",
|
||||
Message: "hourly source is unavailable",
|
||||
Endpoint: "/transport/path/only-for-this-test",
|
||||
CompletenessImpact: "source omitted",
|
||||
}}
|
||||
|
||||
pkg, err := Build(req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
data, err := MarshalYAML(pkg)
|
||||
if err != nil {
|
||||
t.Fatalf("MarshalYAML() error = %v", err)
|
||||
}
|
||||
text := string(data)
|
||||
|
||||
for _, safeDetail := range []string{"source_warnings:", "source: hourly", "code: missing_source", "severity: warning", "message: hourly source is unavailable", "completeness_impact: source omitted"} {
|
||||
if !strings.Contains(text, safeDetail) {
|
||||
t.Fatalf("YAML output missing safe warning detail %q:\n%s", safeDetail, text)
|
||||
}
|
||||
}
|
||||
if strings.Contains(text, "/transport/path/only-for-this-test") || strings.Contains(text, "endpoint:") {
|
||||
t.Fatalf("YAML output contains raw warning transport details:\n%s", text)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMarshalYAMLIsDeterministicAndGroupsNamedStanzas(t *testing.T) {
|
||||
pkg, err := Build(validBuildRequest(t))
|
||||
if err != nil {
|
||||
@@ -287,6 +320,23 @@ func TestMarshalYAMLRejectsUncategorizedStanza(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestMarshalYAMLAttributesStanzaSerializationFailures(t *testing.T) {
|
||||
req := validBuildRequest(t)
|
||||
req.Modules = snapshotWithOutputs(t,
|
||||
module.Output{ID: module.Metadata, StanzaName: "metadata", Value: map[string]string{"run_id": req.Metadata.RunID}},
|
||||
module.Output{ID: module.CurrentConditions, StanzaName: "current_conditions", Value: map[string]float64{"temperature": math.Inf(1)}},
|
||||
)
|
||||
pkg, err := Build(req)
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = MarshalYAML(pkg)
|
||||
if err == nil || !strings.Contains(err.Error(), `marshal briefing stanza "current_conditions"`) {
|
||||
t.Fatalf("MarshalYAML() error = %v, want current_conditions stanza context", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadYAMLRejectsMisplacedStanza(t *testing.T) {
|
||||
data := []byte(`
|
||||
schema_version: weatherreporter.data_package.v4
|
||||
|
||||
Reference in New Issue
Block a user