6 Commits

25 changed files with 1126 additions and 167 deletions

View File

@@ -101,15 +101,6 @@ contains all managed Markdown reports from that batch.
Default: `weatherreporter.{location_id}.{report_id}`.
- `idempotency_key_template`: template for single-report distributor
idempotency keys. Default: `{bundle_id}.{run_id}`.
- `report_path_templates`: ordered list of templates for Markdown report paths
inside distributor bundles. For `generate <report>`, each rendered path maps
to the generated report's managed Markdown source. For `run morning` and
`run evening`, the templates are rendered once per included report and each
rendered path maps that report's managed Markdown source into the one batch
bundle. Default:
```yaml
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
```
- `batch.enabled`: whether batch distributor notification config is active
when distributor notification is enabled. Default: `true`.
- `batch.pipeline_id_template`: template for batch distributor pipeline IDs.
@@ -124,10 +115,13 @@ contains all managed Markdown reports from that batch.
Single-report templates support `location_id`, `report_id`, `run_id`,
`artifact_group`, `batch_output_name`, `valid_start_date`, `valid_end_date`,
`valid_start_time`, `valid_end_time`, `valid_start_stamp`, and
`valid_end_stamp`. Date values use `YYYY-MM-DD`, time values use `HHMM`, and
`valid_start_time`, `valid_end_time`, `valid_start_stamp`, `valid_end_stamp`,
and `storm_id`. Date values use `YYYY-MM-DD`, time values use `HHMM`, and
stamp values use `YYYY-MM-DDTHHMM` in the effective report timezone.
`pipeline_id_template` and `idempotency_key_template` may also use `bundle_id`.
`storm_id` is derived from the storm report valid period as
`{valid_start_stamp}-{valid_end_stamp}`; it renders empty for non-storm
reports. `pipeline_id_template` and `idempotency_key_template` may also use
`bundle_id`.
The rendered pipeline ID selects the configured distributor `http_upload`
workflow. The rendered bundle ID is the stable logical source identity for the
@@ -146,6 +140,18 @@ rendered bundle path for every included report before distributor is called.
Managed Markdown report paths are the only upload source files; copies written
with `--out` or `--out-dir` are never uploaded.
Distributor bundle paths are report-specific. Weatherreporter uses
`reports.<report>.distributor.path_templates` when that override is configured;
otherwise it uses the report definition defaults:
- `hourly`: `hourly/index.md`
- `daily`: `daily/{valid_start_date}/{run_id}.md`, `daily/{valid_start_date}/index.md`
- `today`: `daily/{valid_start_date}/{run_id}.md`, `daily/{valid_start_date}/index.md`, `today/index.md`
- `tomorrow`: `daily/{valid_start_date}/{run_id}.md`, `daily/{valid_start_date}/index.md`, `tomorrow/index.md`
- `three_day`: `three-day/{valid_start_date}/{run_id}.md`, `three-day/{valid_start_date}/index.md`
- `weekend`: `weekend/{valid_start_date}/{run_id}.md`, `weekend/{valid_start_date}/index.md`
- `storm`: `storm/{storm_id}/{run_id}.md`, `storm/{storm_id}/index.md`
The upload token is read from the environment variable named by `token_env`
after config loading and `secrets.directory` processing. Config files should
name the variable only; they should not contain the token value.
@@ -220,12 +226,19 @@ Each report entry supports:
- `deterministic_modules`: ordered module list. Entries may be string module
IDs or objects with `id` and optional `options`.
- `distributor.path_templates`: optional ordered distributor bundle path
templates for this report. If omitted, the report definition defaults are
used. If present, the list must contain at least one template.
Example:
```yaml
reports:
daily:
distributor:
path_templates:
- "daily/{valid_start_date}/{run_id}.md"
- "daily/{valid_start_date}/index.md"
deterministic_modules:
- metadata
- current_conditions

View File

@@ -163,10 +163,11 @@ all planned reports finish, app orchestration evaluates batch notification:
3. If any planned report failed, the batch notification result is `skipped`
with reason `one or more reports failed`, and distributor is not called.
4. If every report succeeded, app orchestration renders batch pipeline, bundle
ID, and idempotency key templates, renders `report_path_templates` for each
included report, validates every managed source path and bundle path, checks
duplicate bundle paths across the batch, calls the notifier once with a
multi-file request, and saves a batch notification debug artifact.
ID, and idempotency key templates, renders report-specific distributor
paths for each included report, validates every managed source path and
bundle path, checks duplicate bundle paths across the batch, calls the
notifier once with a multi-file request, and saves a batch notification
debug artifact.
Batch notification failure records a top-level failed notification, increments
the aggregate batch failure count, and returns an aggregate batch error without

View File

@@ -61,12 +61,16 @@ and bundle paths from:
- `pipeline_id_template`
- `bundle_id_template`
- `idempotency_key_template`
- `report_path_templates`
- report-specific path templates
For batch uploads, the app layer renders pipeline ID, bundle ID, and
idempotency key from `notify.distributor.batch.*`, renders
`report_path_templates` once per included report, and passes the resulting
multi-file request to this adapter.
idempotency key from `notify.distributor.batch.*`, resolves report-specific
path templates once per included report, and passes the resulting multi-file
request to this adapter.
Report-specific path resolution happens entirely in the app layer. Explicit
`reports.<report>.distributor.path_templates` overrides take precedence over
report definition defaults.
The token value is read from the environment variable named by `token_env`
after config loading and `secrets.directory` processing.

View File

@@ -256,17 +256,16 @@ top-level `notification` object with fields such as `status`, `reason`,
Distributor notification is configured with `notify.distributor` and is
disabled by default. For `generate <report>`, weatherreporter uploads the
managed Markdown report path recorded in the report result and metadata. That
single source file can be mapped to one or more configured bundle paths. By
default, it is mapped to one dated report path. Extra copies written by `--out`
or `--out-dir` are operator conveniences only.
single source file is mapped to report-specific bundle paths. Extra copies
written by `--out` or `--out-dir` are operator conveniences only.
For `run morning` and `run evening`, per-report notification is suppressed. If
`notify.distributor.enabled` and `notify.distributor.batch.enabled` are both
true, the batch uploads once after all reports finish successfully. The upload
contains one file mapping set per included report. Each mapping uses the
managed Markdown report as the source and renders `report_path_templates` using
that report's normal template values. All rendered bundle paths across the
batch must be unique. If any report fails, weatherreporter records a top-level
managed Markdown report as the source and report-specific path templates for
that report. All rendered bundle paths across the batch must be unique. If any
report fails, weatherreporter records a top-level
notification status of `skipped` with reason `one or more reports failed` and
does not call distributor. If batch notification is disabled, run commands do
not fall back to per-report uploads.

View File

@@ -300,15 +300,15 @@ Relevant docs: [Configuration reference](config.md),
Symptom: a batch returns nonzero with duplicate bundle path context before a
distributor run ID is accepted.
Likely cause: `notify.distributor.report_path_templates` rendered the same
Likely cause: report-specific distributor path templates rendered the same
bundle-relative path for two included reports in the same batch.
Diagnostic: inspect the error in stdout or stderr. The validation error
includes the duplicate bundle path plus the report IDs, RunIDs, and managed
source paths involved.
Safe fix: change `notify.distributor.report_path_templates` so every report in
a batch renders a unique path. Include values such as `{artifact_group}`,
Safe fix: configure a per-report distributor path override so every report in a
batch renders a unique path. Include values such as `{artifact_group}`,
`{valid_start_date}`, `{batch_output_name}`, or `{run_id}` when needed.
Relevant docs: [Configuration reference](config.md),
@@ -411,7 +411,7 @@ Diagnostic: inspect stdout JSON or stderr status lines for
`notificationError` or the top-level batch notification `error`. Confirm
`notify.distributor.endpoint`,
`notify.distributor.pipeline_id_template`,
`notify.distributor.report_path_templates`, and token configuration. Token
report-specific distributor paths, and token configuration. Token
values are redacted from weatherreporter errors.
If the upload was accepted but destination output did not change, inspect the

View File

@@ -24,8 +24,6 @@ notify:
pipeline_id_template: "weatherreporter.{report_id}"
bundle_id_template: "weatherreporter.{location_id}.{report_id}"
idempotency_key_template: "{bundle_id}.{run_id}"
report_path_templates:
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
batch:
enabled: true
pipeline_id_template: "weatherreporter"
@@ -74,6 +72,10 @@ recent_change:
reports:
daily:
distributor:
path_templates:
- "daily/{valid_start_date}/{run_id}.md"
- "daily/{valid_start_date}/index.md"
deterministic_modules:
- metadata
- current_conditions

View File

@@ -934,7 +934,7 @@ func buildNotificationRequest(cfg config.Config, resolved report.Resolved, repor
if err != nil {
return NotificationRequest{}, err
}
bundlePaths, err := config.RenderDistributorReportPaths(cfg.Notify.Distributor.ReportPathTemplates, values)
bundlePaths, err := renderDistributorReportBundlePaths(cfg, resolved, metadata.RunID, reportPath, values)
if err != nil {
return NotificationRequest{}, err
}
@@ -964,9 +964,45 @@ func distributorTemplateValuesForReport(cfg config.Config, resolved report.Resol
if err := addDistributorValidPeriodValues(&values, resolved.ValidPeriod, cfg.WeatherAPI.Timezone); err != nil {
return config.DistributorTemplateValues{}, err
}
if resolved.Definition.ID == report.Storm {
values.StormID = values.ValidStartStamp + "-" + values.ValidEndStamp
}
return values, nil
}
func renderDistributorReportBundlePaths(cfg config.Config, resolved report.Resolved, runID string, sourcePath string, values config.DistributorTemplateValues) ([]string, error) {
templates, name, err := distributorPathTemplatesForReport(cfg, resolved.Definition)
if err != nil {
return nil, distributorReportPathError(resolved.Definition.ID, runID, sourcePath, err)
}
paths, err := config.RenderDistributorReportPaths(name, templates, values)
if err != nil {
return nil, distributorReportPathError(resolved.Definition.ID, runID, sourcePath, err)
}
return paths, nil
}
func distributorPathTemplatesForReport(cfg config.Config, definition report.Definition) ([]string, string, error) {
overrides, err := cfg.ReportDistributorPathOverrides()
if err != nil {
return nil, "", err
}
if templates, ok := overrides[definition.ID]; ok {
return append([]string(nil), templates...), fmt.Sprintf("reports.%s.distributor.path_templates", definition.ID), nil
}
if len(definition.DistributorPathTemplates) > 0 {
return append([]string(nil), definition.DistributorPathTemplates...), fmt.Sprintf("report.%s.distributor_path_templates", definition.ID), nil
}
return nil, "", fmt.Errorf("no distributor path templates configured")
}
func distributorReportPathError(id report.ID, runID string, sourcePath string, err error) error {
if sourcePath != "" {
return fmt.Errorf("report %q run %q source path %q: %w", id, runID, sourcePath, err)
}
return fmt.Errorf("report %q run %q: %w", id, runID, err)
}
func addDistributorValidPeriodValues(values *config.DistributorTemplateValues, period timeutil.Period, timezone string) error {
location, err := timeutil.LoadLocation(timezone)
if err != nil {

View File

@@ -739,7 +739,7 @@ func TestGenerateHourlyReportCopiesOutputAndNotifiesManagedReport(t *testing.T)
if req.ReportPath == outputPath {
t.Fatalf("notification used output copy %q, want managed report path", outputPath)
}
wantBundlePaths := []string{"2026-05-29/hourly/hourly.md"}
wantBundlePaths := []string{"hourly/index.md"}
if strings.Join(req.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") {
t.Fatalf("notification BundlePaths = %#v, want %#v", req.BundlePaths, wantBundlePaths)
}
@@ -778,7 +778,6 @@ func TestGenerateTodayReportCopiesOutputAndNotifiesTodayTemplateValues(t *testin
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{report_id}.{artifact_group}"
cfg.Notify.Distributor.BundleIDTemplate = "{artifact_group}.{batch_output_name}.{report_id}"
cfg.Notify.Distributor.IdempotencyKeyTemplate = "{bundle_id}.{run_id}"
cfg.Notify.Distributor.ReportPathTemplates = []string{"{valid_start_date}/{artifact_group}/{batch_output_name}"}
resolved := resolveGenerateForTest(t, cfg, GenerateRequest{
Report: ReportToday,
Date: mustParse("2026-05-29T12:00:00-05:00"),
@@ -834,7 +833,11 @@ func TestGenerateTodayReportCopiesOutputAndNotifiesTodayTemplateValues(t *testin
if req.BundleID != "today.today.md.today" {
t.Fatalf("BundleID = %q, want Today artifact group, output name, and report id", req.BundleID)
}
wantBundlePaths := []string{"2026-05-29/today/today.md"}
wantBundlePaths := []string{
"daily/2026-05-29/" + result.Metadata.RunID + ".md",
"daily/2026-05-29/index.md",
"today/index.md",
}
if strings.Join(req.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") {
t.Fatalf("BundlePaths = %#v, want %#v", req.BundlePaths, wantBundlePaths)
}
@@ -981,7 +984,6 @@ func TestGenerateTomorrowReportNotificationUsesTomorrowTemplateValues(t *testing
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{report_id}.{artifact_group}"
cfg.Notify.Distributor.BundleIDTemplate = "{artifact_group}.{batch_output_name}.{report_id}"
cfg.Notify.Distributor.IdempotencyKeyTemplate = "{bundle_id}.{run_id}"
cfg.Notify.Distributor.ReportPathTemplates = []string{"{valid_start_date}/{artifact_group}/{batch_output_name}"}
resolved := resolveGenerateForTest(t, cfg, GenerateRequest{
Report: ReportTomorrow,
}, "2026-05-29T18:00:00-05:00")
@@ -1011,7 +1013,11 @@ func TestGenerateTomorrowReportNotificationUsesTomorrowTemplateValues(t *testing
if req.BundleID != "tomorrow.tomorrow.md.tomorrow" {
t.Fatalf("BundleID = %q, want artifact group, batch output name, and report id", req.BundleID)
}
wantBundlePaths := []string{"2026-05-30/tomorrow/tomorrow.md"}
wantBundlePaths := []string{
"daily/2026-05-30/" + result.Metadata.RunID + ".md",
"daily/2026-05-30/index.md",
"tomorrow/index.md",
}
if strings.Join(req.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") {
t.Fatalf("BundlePaths = %#v, want %#v", req.BundlePaths, wantBundlePaths)
}
@@ -1234,7 +1240,8 @@ func TestGenerateReportNotifiesManagedReportPath(t *testing.T) {
t.Fatalf("decode notification artifact: %v", err)
}
wantBundlePaths := []string{
"2026-05-29/daily/2026-05-29-daily-" + result.Metadata.RunID + ".md",
"daily/2026-05-29/" + result.Metadata.RunID + ".md",
"daily/2026-05-29/index.md",
}
if notificationArtifact.PipelineID != "weatherreporter.daily" || strings.Join(notificationArtifact.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") || notificationArtifact.BundleCreated.IsZero() || notificationArtifact.RunStatus == nil || !strings.Contains(string(notificationArtifact.RunStatus.Report), "replace_older") {
t.Fatalf("notification artifact = %#v, want requested pipeline, status report, and created timestamp", notificationArtifact)
@@ -2214,6 +2221,254 @@ func TestResolveGenerateStorm(t *testing.T) {
}
}
func TestDistributorTemplateValuesDeriveStormID(t *testing.T) {
cfg := config.Defaults()
cfg.WeatherAPI.Timezone = "America/Chicago"
now := mustParse("2026-05-29T12:00:00-05:00")
start := mustParse("2026-05-29T18:00:00-05:00")
end := mustParse("2026-05-30T06:00:00-05:00")
resolved, err := ResolveGenerate(GenerateRequest{
Config: cfg,
Report: ReportStorm,
StormStart: start,
StormEnd: end,
}, now)
if err != nil {
t.Fatalf("ResolveGenerate() error = %v", err)
}
values, err := distributorTemplateValuesForReport(cfg, resolved, "run", "")
if err != nil {
t.Fatalf("distributorTemplateValuesForReport() error = %v", err)
}
if values.StormID != "2026-05-29T1800-2026-05-30T0600" {
t.Fatalf("StormID = %q, want storm valid-period stamp", values.StormID)
}
}
func TestDistributorTemplateValuesLeaveStormIDEmptyForOtherReports(t *testing.T) {
cfg := config.Defaults()
cfg.WeatherAPI.Timezone = "America/Chicago"
location, err := timeutil.LoadLocation(cfg.WeatherAPI.Timezone)
if err != nil {
t.Fatalf("load location: %v", err)
}
resolved, err := report.Resolve(report.Today, report.ResolveRequest{
Now: mustParse("2026-05-29T12:00:00-05:00"),
Location: location,
})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
values, err := distributorTemplateValuesForReport(cfg, resolved, "run", "")
if err != nil {
t.Fatalf("distributorTemplateValuesForReport() error = %v", err)
}
if values.StormID != "" {
t.Fatalf("StormID = %q, want empty for %s", values.StormID, resolved.Definition.ID)
}
}
func TestBuildNotificationRequestUsesReportDefaultBundlePaths(t *testing.T) {
cfg := config.Defaults()
cfg.Location.ID = "home"
cfg.WeatherAPI.Timezone = "America/Chicago"
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{report_id}"
location := mustLoadTestLocation(t, cfg.WeatherAPI.Timezone)
now := mustParse("2026-05-29T12:00:00-05:00")
registry := report.DefaultRegistry()
tests := []struct {
id report.ID
req report.ResolveRequest
want func(state.Metadata) []string
source string
}{
{
id: report.Hourly,
req: report.ResolveRequest{
Now: now,
Location: location,
},
want: func(metadata state.Metadata) []string {
return []string{"hourly/index.md"}
},
source: "/managed/hourly.md",
},
{
id: report.Daily,
req: report.ResolveRequest{
Now: now,
Location: location,
Date: mustParse("2026-05-29T12:00:00-05:00"),
},
want: func(metadata state.Metadata) []string {
return []string{
"daily/2026-05-29/" + metadata.RunID + ".md",
"daily/2026-05-29/index.md",
}
},
source: "/managed/daily.md",
},
{
id: report.Today,
req: report.ResolveRequest{
Now: now,
Location: location,
},
want: func(metadata state.Metadata) []string {
return []string{
"daily/2026-05-29/" + metadata.RunID + ".md",
"daily/2026-05-29/index.md",
"today/index.md",
}
},
source: "/managed/today.md",
},
{
id: report.Tomorrow,
req: report.ResolveRequest{
Now: now,
Location: location,
},
want: func(metadata state.Metadata) []string {
return []string{
"daily/2026-05-30/" + metadata.RunID + ".md",
"daily/2026-05-30/index.md",
"tomorrow/index.md",
}
},
source: "/managed/tomorrow.md",
},
{
id: report.ThreeDay,
req: report.ResolveRequest{
Now: now,
Location: location,
},
want: func(metadata state.Metadata) []string {
return []string{
"three-day/2026-05-29/" + metadata.RunID + ".md",
"three-day/2026-05-29/index.md",
}
},
source: "/managed/three-day.md",
},
{
id: report.Weekend,
req: report.ResolveRequest{
Now: mustParse("2026-05-29T05:00:00-05:00"),
Location: location,
},
want: func(metadata state.Metadata) []string {
return []string{
"weekend/2026-05-29/" + metadata.RunID + ".md",
"weekend/2026-05-29/index.md",
}
},
source: "/managed/weekend.md",
},
{
id: report.Storm,
req: report.ResolveRequest{
Now: now,
Location: location,
StormStart: mustParse("2026-05-29T18:00:00-05:00"),
StormEnd: mustParse("2026-05-30T06:00:00-05:00"),
},
want: func(metadata state.Metadata) []string {
return []string{
"storm/2026-05-29T1800-2026-05-30T0600/" + metadata.RunID + ".md",
"storm/2026-05-29T1800-2026-05-30T0600/index.md",
}
},
source: "/managed/storm.md",
},
}
for _, tt := range tests {
t.Run(string(tt.id), func(t *testing.T) {
resolved, err := registry.Resolve(tt.id, tt.req)
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
metadata := notificationMetadataForTest(resolved)
req, err := buildNotificationRequest(cfg, resolved, tt.source, metadata)
if err != nil {
t.Fatalf("buildNotificationRequest() error = %v", err)
}
want := tt.want(metadata)
if strings.Join(req.BundlePaths, "\n") != strings.Join(want, "\n") {
t.Fatalf("BundlePaths = %#v, want %#v", req.BundlePaths, want)
}
if req.ReportPath != tt.source {
t.Fatalf("ReportPath = %q, want %q", req.ReportPath, tt.source)
}
})
}
}
func TestBuildNotificationRequestUsesReportDistributorPathOverride(t *testing.T) {
cfg := config.Defaults()
cfg.Location.ID = "home"
cfg.WeatherAPI.Timezone = "America/Chicago"
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{report_id}"
applyReportDistributorPathOverrides(t, &cfg, `
reports:
daily:
distributor:
path_templates:
- "custom/{report_id}/{run_id}.md"
`)
location := mustLoadTestLocation(t, cfg.WeatherAPI.Timezone)
resolved, err := report.Resolve(report.Daily, report.ResolveRequest{
Now: mustParse("2026-05-29T12:00:00-05:00"),
Location: location,
Date: mustParse("2026-05-29T12:00:00-05:00"),
})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
metadata := notificationMetadataForTest(resolved)
req, err := buildNotificationRequest(cfg, resolved, "/managed/daily.md", metadata)
if err != nil {
t.Fatalf("buildNotificationRequest() error = %v", err)
}
want := []string{"custom/daily/" + metadata.RunID + ".md"}
if strings.Join(req.BundlePaths, "\n") != strings.Join(want, "\n") {
t.Fatalf("BundlePaths = %#v, want override %#v", req.BundlePaths, want)
}
}
func TestBuildNotificationRequestRequiresReportBundlePaths(t *testing.T) {
cfg := config.Defaults()
cfg.Location.ID = "home"
cfg.WeatherAPI.Timezone = "America/Chicago"
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{report_id}"
location := mustLoadTestLocation(t, cfg.WeatherAPI.Timezone)
resolved, err := report.Resolve(report.Hourly, report.ResolveRequest{
Now: mustParse("2026-05-29T12:00:00-05:00"),
Location: location,
})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
resolved.Definition.DistributorPathTemplates = nil
metadata := notificationMetadataForTest(resolved)
_, err = buildNotificationRequest(cfg, resolved, "/managed/hourly.md", metadata)
if err == nil {
t.Fatal("buildNotificationRequest() error = nil, want missing path templates error")
}
for _, want := range []string{`report "hourly"`, metadata.RunID, `/managed/hourly.md`, "no distributor path templates"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("error = %q, want %q", err.Error(), want)
}
}
}
func TestBatchRunIDUsesUTCStartAndBatchName(t *testing.T) {
tests := []struct {
name string
@@ -2333,10 +2588,6 @@ func TestBatchResultJSONIncludesNotification(t *testing.T) {
func TestBuildBatchNotificationRequestIncludesEveningReports(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyNotificationConfig(t, server)
cfg.Notify.Distributor.ReportPathTemplates = []string{
"archive/{valid_start_date}/{artifact_group}/{run_id}.md",
"latest/{batch_output_name}",
}
cfg.Notify.Distributor.Batch.PipelineIDTemplate = "weatherreporter.{batch}.{batch_started_date}"
cfg.Notify.Distributor.Batch.BundleIDTemplate = "weatherreporter.{location_id}.{batch}"
cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate = "{bundle_id}.{batch_run_id}"
@@ -2368,20 +2619,26 @@ func TestBuildBatchNotificationRequestIncludesEveningReports(t *testing.T) {
if len(req.IncludedReports) != 3 {
t.Fatalf("IncludedReports = %d, want 3", len(req.IncludedReports))
}
if len(req.Files) != 6 {
t.Fatalf("Files = %d, want two mappings per report", len(req.Files))
if len(req.Files) != 7 {
t.Fatalf("Files = %d, want report-specific mappings", len(req.Files))
}
wantBundlePaths := map[string]struct{}{}
for _, plannedReport := range planned {
resolved := plannedReport.Resolved
runID := resolved.Metadata().RunID
outputName := plannedReport.OutputCopyName
if outputName == "" {
outputName = resolved.Definition.BatchOutputName
validStart := resolved.ValidPeriod.Start.In(mustLoadTestLocation(t, cfg.WeatherAPI.Timezone)).Format(timeutil.DateLayout)
switch resolved.Definition.ID {
case report.Tomorrow:
wantBundlePaths["daily/"+validStart+"/"+runID+".md"] = struct{}{}
wantBundlePaths["daily/"+validStart+"/index.md"] = struct{}{}
wantBundlePaths["tomorrow/index.md"] = struct{}{}
case report.Daily:
wantBundlePaths["daily/"+validStart+"/"+runID+".md"] = struct{}{}
wantBundlePaths["daily/"+validStart+"/index.md"] = struct{}{}
default:
t.Fatalf("unexpected planned report %s", resolved.Definition.ID)
}
wantBundlePaths[fmt.Sprintf("archive/%s/%s/%s.md", resolved.ValidPeriod.Start.In(mustLoadTestLocation(t, cfg.WeatherAPI.Timezone)).Format(timeutil.DateLayout), resolved.Definition.ArtifactGroup, runID)] = struct{}{}
wantBundlePaths["latest/"+outputName] = struct{}{}
}
gotBundlePaths := map[string]struct{}{}
gotSourcePaths := map[string]struct{}{}
@@ -2409,10 +2666,67 @@ func TestBuildBatchNotificationRequestIncludesEveningReports(t *testing.T) {
}
}
func TestBuildBatchNotificationRequestUsesReportOverridesAndDefaults(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyNotificationConfig(t, server)
applyReportDistributorPathOverrides(t, &cfg, `
reports:
daily:
distributor:
path_templates:
- "custom-daily/{valid_start_date}/{run_id}.md"
`)
startedAt := mustParse("2026-05-29T18:00:00-05:00")
planned, reports := plannedBatchNotificationReports(t, cfg, BatchEvening, startedAt, "2026-05-31")
runID := batchRunID(startedAt, BatchEvening)
req, err := buildBatchNotificationRequest(cfg, BatchEvening, runID, startedAt, reports, planned)
if err != nil {
t.Fatalf("buildBatchNotificationRequest() error = %v", err)
}
gotBundlePaths := map[string]struct{}{}
for _, file := range req.Files {
gotBundlePaths[file.BundlePath] = struct{}{}
}
for _, plannedReport := range planned {
reportRunID := plannedReport.Resolved.Metadata().RunID
switch plannedReport.Resolved.Definition.ID {
case report.Tomorrow:
for _, want := range []string{
"daily/2026-05-30/" + reportRunID + ".md",
"daily/2026-05-30/index.md",
"tomorrow/index.md",
} {
if _, ok := gotBundlePaths[want]; !ok {
t.Fatalf("bundle paths = %#v, missing default path %q", gotBundlePaths, want)
}
}
case report.Daily:
want := "custom-daily/2026-05-31/" + reportRunID + ".md"
if _, ok := gotBundlePaths[want]; !ok {
t.Fatalf("bundle paths = %#v, missing override path %q", gotBundlePaths, want)
}
if _, ok := gotBundlePaths["daily/2026-05-31/index.md"]; ok {
t.Fatalf("bundle paths = %#v, want daily defaults replaced by override", gotBundlePaths)
}
}
}
}
func TestBuildBatchNotificationRequestRejectsDuplicateBundlePaths(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyNotificationConfig(t, server)
cfg.Notify.Distributor.ReportPathTemplates = []string{"index.md"}
applyReportDistributorPathOverrides(t, &cfg, `
reports:
tomorrow:
distributor:
path_templates:
- "index.md"
daily:
distributor:
path_templates:
- "index.md"
`)
startedAt := mustParse("2026-05-29T18:00:00-05:00")
planned, reports := plannedBatchNotificationReports(t, cfg, BatchEvening, startedAt, "2026-05-31")
@@ -2537,16 +2851,13 @@ func TestRunBatchMorningSendsOneBatchNotification(t *testing.T) {
if req.Batch != BatchMorning || req.RunID != "20260529T100000.000000000Z_morning" {
t.Fatalf("batch request identity = %s/%s, want morning run id", req.Batch, req.RunID)
}
if len(req.IncludedReports) != 3 || len(req.Files) != 3 {
t.Fatalf("batch request reports/files = %d/%d, want 3/3", len(req.IncludedReports), len(req.Files))
if len(req.IncludedReports) != 3 || len(req.Files) != 8 {
t.Fatalf("batch request reports/files = %d/%d, want 3/8", len(req.IncludedReports), len(req.Files))
}
for _, file := range req.Files {
if file.SourcePath == "" || file.BundlePath == "" {
t.Fatalf("batch file = %#v, want source and bundle path", file)
}
if !strings.Contains(file.BundlePath, file.RunID) {
t.Fatalf("bundle path %q does not include report run id %q", file.BundlePath, file.RunID)
}
}
if result.Notification == nil || result.Notification.Status != "succeeded" || result.Notification.RunID != "batch-distributor-run" || result.Notification.Path == "" {
t.Fatalf("batch notification = %#v, want succeeded result with artifact path", result.Notification)
@@ -2998,6 +3309,36 @@ func dailyNotificationConfig(t *testing.T, server *httptest.Server) config.Confi
return cfg
}
func notificationMetadataForTest(resolved report.Resolved) state.Metadata {
metadata := resolved.Metadata()
return state.Metadata{
RunID: metadata.RunID,
ReportID: metadata.ReportID,
PromptID: metadata.PromptID,
GeneratedAt: metadata.GeneratedAt,
Timezone: metadata.Timezone,
ValidPeriod: metadata.ValidPeriod,
}
}
func applyReportDistributorPathOverrides(t *testing.T, cfg *config.Config, data string) {
t.Helper()
path := filepath.Join(t.TempDir(), "config.yml")
if err := os.WriteFile(path, []byte(data), 0o600); err != nil {
t.Fatalf("write config fixture: %v", err)
}
loaded, err := config.LoadFile(path)
if err != nil {
t.Fatalf("LoadFile() error = %v", err)
}
if cfg.Reports == nil {
cfg.Reports = map[string]config.ReportConfig{}
}
for key, reportCfg := range loaded.Reports {
cfg.Reports[key] = reportCfg
}
}
func plannedBatchNotificationReports(t *testing.T, cfg config.Config, batch BatchKind, now time.Time, futureDailyDates ...string) ([]plannedBatchReport, []BatchReportResult) {
t.Helper()
collection := collectionWithFutureDailyForTest(t, cfg, futureDailyDates...)
@@ -3140,7 +3481,6 @@ func applyHourlyGeneratedTextSettings(cfg *config.Config, t *testing.T, server *
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{artifact_group}"
cfg.Notify.Distributor.BundleIDTemplate = "weatherreporter.{location_id}.{report_id}"
cfg.Notify.Distributor.IdempotencyKeyTemplate = "weatherreporter.{location_id}.{report_id}.{run_id}"
cfg.Notify.Distributor.ReportPathTemplates = []string{"{valid_start_date}/{artifact_group}/{batch_output_name}"}
}
func resolveHourlyGeneratedTextFixture(t *testing.T, cfg config.Config) (report.Resolved, *recordingStore, *recordingNotifier, string) {

View File

@@ -161,9 +161,9 @@ func buildBatchNotificationRequest(cfg config.Config, batch BatchKind, runID str
if err != nil {
return batchNotificationRequest{}, fmt.Errorf("batch notification report %q run %q source path %q: %w", item.ReportID, item.RunID, item.ReportPath, err)
}
bundlePaths, err := config.RenderDistributorReportPaths(cfg.Notify.Distributor.ReportPathTemplates, values)
bundlePaths, err := renderDistributorReportBundlePaths(cfg, plannedReport.Resolved, item.RunID, item.ReportPath, values)
if err != nil {
return batchNotificationRequest{}, fmt.Errorf("batch notification report %q run %q source path %q: %w", item.ReportID, item.RunID, item.ReportPath, err)
return batchNotificationRequest{}, err
}
included := BatchNotificationReport{

View File

@@ -66,7 +66,6 @@ type DistributorNotifyConfig struct {
PipelineIDTemplate string `yaml:"pipeline_id_template"`
BundleIDTemplate string `yaml:"bundle_id_template"`
IdempotencyKeyTemplate string `yaml:"idempotency_key_template"`
ReportPathTemplates []string `yaml:"report_path_templates"`
Batch DistributorBatchNotifyConfig `yaml:"batch"`
}
@@ -113,10 +112,16 @@ type RecentChangeConfig struct {
}
type ReportConfig struct {
DeterministicModules []ModuleConfigItem `yaml:"deterministic_modules"`
DeterministicModules []ModuleConfigItem `yaml:"deterministic_modules"`
Distributor ReportDistributorConfig `yaml:"distributor"`
deterministicModulesSet bool
}
type ReportDistributorConfig struct {
PathTemplates []string `yaml:"path_templates"`
pathTemplatesSet bool
}
type ModuleConfigItem struct {
ID module.ID `yaml:"id"`
Options any `yaml:"options,omitempty"`
@@ -135,6 +140,10 @@ func (c *ReportConfig) UnmarshalYAML(value *yaml.Node) error {
return err
}
c.deterministicModulesSet = true
case "distributor":
if err := node.Decode(&c.Distributor); err != nil {
return err
}
default:
return fmt.Errorf("unknown report entry field %q", key)
}
@@ -142,6 +151,81 @@ func (c *ReportConfig) UnmarshalYAML(value *yaml.Node) error {
return nil
}
func (c *DistributorNotifyConfig) UnmarshalYAML(value *yaml.Node) error {
if value.Kind != yaml.MappingNode {
return fmt.Errorf("notify distributor entry must be a mapping")
}
for i := 0; i < len(value.Content); i += 2 {
key := value.Content[i].Value
node := value.Content[i+1]
switch key {
case "enabled":
if err := node.Decode(&c.Enabled); err != nil {
return err
}
case "endpoint":
if err := node.Decode(&c.Endpoint); err != nil {
return err
}
case "token_env":
if err := node.Decode(&c.TokenEnv); err != nil {
return err
}
case "timeout":
if err := node.Decode(&c.Timeout); err != nil {
return err
}
case "failure_policy":
if err := node.Decode(&c.FailurePolicy); err != nil {
return err
}
case "pipeline_id_template":
if err := node.Decode(&c.PipelineIDTemplate); err != nil {
return err
}
case "bundle_id_template":
if err := node.Decode(&c.BundleIDTemplate); err != nil {
return err
}
case "idempotency_key_template":
if err := node.Decode(&c.IdempotencyKeyTemplate); err != nil {
return err
}
case "batch":
if err := node.Decode(&c.Batch); err != nil {
return err
}
default:
return fmt.Errorf("unknown notify distributor field %q", key)
}
}
return nil
}
func (c *ReportDistributorConfig) UnmarshalYAML(value *yaml.Node) error {
if value.Kind != yaml.MappingNode {
return fmt.Errorf("report distributor entry must be a mapping")
}
for i := 0; i < len(value.Content); i += 2 {
key := value.Content[i].Value
node := value.Content[i+1]
switch key {
case "path_templates":
if err := node.Decode(&c.PathTemplates); err != nil {
return err
}
c.pathTemplatesSet = true
default:
return fmt.Errorf("unknown report distributor field %q", key)
}
}
return nil
}
func (c ReportDistributorConfig) PathTemplatesSet() bool {
return c.pathTemplatesSet
}
func (m *ModuleConfigItem) UnmarshalYAML(value *yaml.Node) error {
switch value.Kind {
case yaml.ScalarNode:

View File

@@ -10,6 +10,7 @@ import (
"gitea.maximumdirect.net/eric/weatherreporter/internal/module"
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
"gopkg.in/yaml.v3"
)
func TestDefaults(t *testing.T) {
@@ -57,12 +58,6 @@ func TestDefaults(t *testing.T) {
if cfg.Notify.Distributor.IdempotencyKeyTemplate != "{bundle_id}.{run_id}" {
t.Fatalf("Notify.Distributor.IdempotencyKeyTemplate = %q, want default", cfg.Notify.Distributor.IdempotencyKeyTemplate)
}
wantReportPaths := []string{
"{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md",
}
if strings.Join(cfg.Notify.Distributor.ReportPathTemplates, "\n") != strings.Join(wantReportPaths, "\n") {
t.Fatalf("Notify.Distributor.ReportPathTemplates = %#v, want %#v", cfg.Notify.Distributor.ReportPathTemplates, wantReportPaths)
}
if !cfg.Notify.Distributor.Batch.Enabled {
t.Fatalf("Notify.Distributor.Batch.Enabled = false, want true")
}
@@ -101,9 +96,6 @@ func TestLoadExampleConfig(t *testing.T) {
if cfg.Notify.Distributor.PipelineIDTemplate != "weatherreporter.{report_id}" {
t.Fatalf("PipelineIDTemplate = %q, want example pipeline template", cfg.Notify.Distributor.PipelineIDTemplate)
}
if len(cfg.Notify.Distributor.ReportPathTemplates) != 1 {
t.Fatalf("ReportPathTemplates = %#v, want example archive path", cfg.Notify.Distributor.ReportPathTemplates)
}
if !cfg.Notify.Distributor.Batch.Enabled {
t.Fatalf("Notify.Distributor.Batch.Enabled = false, want true")
}
@@ -313,6 +305,102 @@ reports:
}
}
func TestLoadReportDistributorPathOverrides(t *testing.T) {
path := writeConfig(t, `
reports:
daily:
distributor:
path_templates:
- "daily/{valid_start_date}/{run_id}.md"
- "daily/{valid_start_date}/index.md"
today:
deterministic_modules:
- metadata
`)
cfg, err := LoadFile(path)
if err != nil {
t.Fatalf("LoadFile() error = %v", err)
}
daily := cfg.Reports["daily"].Distributor
if !daily.PathTemplatesSet() {
t.Fatal("daily distributor path_templates set = false, want true")
}
want := []string{
"daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md",
}
if !reflect.DeepEqual(daily.PathTemplates, want) {
t.Fatalf("daily path templates = %#v, want %#v", daily.PathTemplates, want)
}
if cfg.Reports["today"].Distributor.PathTemplatesSet() {
t.Fatal("today distributor path_templates set = true, want false")
}
overrides, err := cfg.ReportDistributorPathOverrides()
if err != nil {
t.Fatalf("ReportDistributorPathOverrides() error = %v", err)
}
if !reflect.DeepEqual(overrides[report.Daily], want) {
t.Fatalf("daily distributor override = %#v, want %#v", overrides[report.Daily], want)
}
if _, ok := overrides[report.Today]; ok {
t.Fatalf("today distributor override = %#v, want omitted override absent", overrides[report.Today])
}
}
func TestReportDistributorPathTemplatesSetTracksExplicitEmptyList(t *testing.T) {
var cfg Config
if err := yaml.Unmarshal([]byte(`
reports:
daily:
distributor:
path_templates: []
today:
distributor: {}
`), &cfg); err != nil {
t.Fatalf("yaml.Unmarshal() error = %v", err)
}
if !cfg.Reports["daily"].Distributor.PathTemplatesSet() {
t.Fatal("daily distributor path_templates set = false, want true")
}
if len(cfg.Reports["daily"].Distributor.PathTemplates) != 0 {
t.Fatalf("daily path templates = %#v, want empty explicit list", cfg.Reports["daily"].Distributor.PathTemplates)
}
if cfg.Reports["today"].Distributor.PathTemplatesSet() {
t.Fatal("today distributor path_templates set = true, want false")
}
}
func TestLoadReportDistributorPathOverrideAliases(t *testing.T) {
path := writeConfig(t, `
reports:
three-day-outlook:
distributor:
path_templates:
- "three-day/{valid_start_date}/index.md"
weekend_outlook:
distributor:
path_templates:
- "weekend/{valid_start_date}/index.md"
`)
cfg, err := LoadFile(path)
if err != nil {
t.Fatalf("LoadFile() error = %v", err)
}
overrides, err := cfg.ReportDistributorPathOverrides()
if err != nil {
t.Fatalf("ReportDistributorPathOverrides() error = %v", err)
}
if !reflect.DeepEqual(overrides[report.ThreeDay], []string{"three-day/{valid_start_date}/index.md"}) {
t.Fatalf("three-day distributor override = %#v, want alias override", overrides[report.ThreeDay])
}
if !reflect.DeepEqual(overrides[report.Weekend], []string{"weekend/{valid_start_date}/index.md"}) {
t.Fatalf("weekend distributor override = %#v, want alias override", overrides[report.Weekend])
}
}
func TestValidateReportModuleKeysWithoutMutatingOptions(t *testing.T) {
cfg := Defaults()
rawOptions := map[string]any{
@@ -565,6 +653,168 @@ reports:
}
}
func TestReportDistributorPathOverrideValidation(t *testing.T) {
tests := []struct {
name string
yaml string
wantErr string
}{
{
name: "UnknownReportField",
yaml: `
reports:
daily:
distributor_paths:
- latest.md
`,
wantErr: `unknown report entry field "distributor_paths"`,
},
{
name: "UnknownDistributorField",
yaml: `
reports:
daily:
distributor:
paths:
- latest.md
`,
wantErr: `unknown report distributor field "paths"`,
},
{
name: "DuplicateReportAlias",
yaml: `
reports:
three-day:
distributor:
path_templates:
- "three-day/{valid_start_date}/index.md"
three_day:
distributor:
path_templates:
- "three-day/latest.md"
`,
wantErr: "duplicates report override",
},
{
name: "UnknownTemplateVariable",
yaml: `
reports:
daily:
distributor:
path_templates:
- "{unknown}.md"
`,
wantErr: `reports.daily.distributor.path_templates[0] contains unknown template variable "unknown"`,
},
{
name: "AbsolutePath",
yaml: `
reports:
daily:
distributor:
path_templates:
- "/daily.md"
`,
wantErr: "reports.daily.distributor.path_templates[0] must render a relative path",
},
{
name: "ParentSegment",
yaml: `
reports:
daily:
distributor:
path_templates:
- "daily/../index.md"
`,
wantErr: "reports.daily.distributor.path_templates[0] must not render . or .. path segments",
},
{
name: "Manifest",
yaml: `
reports:
daily:
distributor:
path_templates:
- "daily/manifest.json"
`,
wantErr: `reports.daily.distributor.path_templates[0] must not render reserved path segment "manifest.json"`,
},
{
name: "DuplicateRenderedPath",
yaml: `
reports:
daily:
distributor:
path_templates:
- "daily/index.md"
- "daily/index.md"
`,
wantErr: `reports.daily.distributor.path_templates renders duplicate path "daily/index.md"`,
},
{
name: "EmptyOverrideList",
yaml: `
reports:
daily:
distributor:
path_templates: []
`,
wantErr: "reports.daily.distributor.path_templates must contain at least one entry",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := LoadFile(writeConfig(t, tt.yaml))
if err == nil {
t.Fatal("LoadFile() error = nil, want validation error")
}
if !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
}
})
}
}
func TestReportDistributorPathOverridesConsistentForLoadedAndConstructedConfig(t *testing.T) {
yaml := `
reports:
daily:
distributor:
path_templates:
- "daily/{valid_start_date}/index.md"
`
reports := map[string]ReportConfig{
"daily": {
Distributor: ReportDistributorConfig{
PathTemplates: []string{"daily/{valid_start_date}/index.md"},
pathTemplatesSet: true,
},
},
}
cfg, err := LoadFile(writeConfig(t, yaml))
if err != nil {
t.Fatalf("LoadFile() error = %v", err)
}
loaded, err := cfg.ReportDistributorPathOverrides()
if err != nil {
t.Fatalf("loaded ReportDistributorPathOverrides() error = %v", err)
}
cfg = Defaults()
cfg.Reports = reports
if err := Validate(cfg); err != nil {
t.Fatalf("Validate() error = %v", err)
}
constructed, err := cfg.ReportDistributorPathOverrides()
if err != nil {
t.Fatalf("constructed ReportDistributorPathOverrides() error = %v", err)
}
if !reflect.DeepEqual(loaded, constructed) {
t.Fatalf("loaded overrides = %#v, constructed = %#v", loaded, constructed)
}
}
func TestReportModuleValidationConsistentForLoadedAndConstructedConfig(t *testing.T) {
tests := []struct {
name string
@@ -806,6 +1056,37 @@ func TestDisabledDistributorNotifyAcceptsOmittedFields(t *testing.T) {
}
}
func TestDistributorNotifyRejectsRemovedGlobalReportPaths(t *testing.T) {
removedField := "report_path" + "_templates"
_, err := LoadFile(writeConfig(t, `
notify:
distributor:
`+removedField+`:
- index.md
`))
if err == nil {
t.Fatal("LoadFile() error = nil, want removed global path field error")
}
if !strings.Contains(err.Error(), `unknown notify distributor field "`+removedField+`"`) {
t.Fatalf("error = %q, want removed global path field rejection", err.Error())
}
}
func TestDistributorNotifyRejectsUnknownFields(t *testing.T) {
_, err := LoadFile(writeConfig(t, `
notify:
distributor:
paths:
- index.md
`))
if err == nil {
t.Fatal("LoadFile() error = nil, want unknown distributor field error")
}
if !strings.Contains(err.Error(), `unknown notify distributor field "paths"`) {
t.Fatalf("error = %q, want unknown field rejection", err.Error())
}
}
func TestDisabledDistributorNotifyAcceptsMalformedBatchTemplates(t *testing.T) {
cfg := Defaults()
cfg.Notify.Distributor.Enabled = false
@@ -895,32 +1176,11 @@ func TestEnabledDistributorNotifyValidation(t *testing.T) {
wantErr: "notify.distributor.idempotency_key_template",
},
{
name: "ReportPathTemplatesEmpty",
name: "BatchTemplate",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.ReportPathTemplates = nil
cfg.Notify.Distributor.Batch.BundleIDTemplate = "{run_id}"
},
wantErr: "notify.distributor.report_path_templates",
},
{
name: "ReportPathTemplateUnknown",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.ReportPathTemplates = []string{"{unknown}"}
},
wantErr: "notify.distributor.report_path_templates",
},
{
name: "ReportPathTemplateInvalidPath",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.ReportPathTemplates = []string{"/{batch_output_name}"}
},
wantErr: "notify.distributor.report_path_templates",
},
{
name: "ReportPathTemplateDuplicatePath",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.ReportPathTemplates = []string{"latest.md", "latest.md"}
},
wantErr: "notify.distributor.report_path_templates",
wantErr: "notify.distributor.batch.bundle_id_template",
},
}
@@ -1058,35 +1318,38 @@ func TestDistributorTemplateRendering(t *testing.T) {
ValidEndTime: "0600",
ValidStartStamp: "2026-06-07T1800",
ValidEndStamp: "2026-06-08T0600",
StormID: "2026-06-07T1800-2026-06-08T0600",
BundleID: "weatherreporter.home.daily",
}
bundleID, err := RenderDistributorBundleID("weatherreporter.{location_id}.{report_id}", values)
bundleID, err := RenderDistributorBundleID("weatherreporter.{location_id}.{report_id}.{storm_id}", values)
if err != nil {
t.Fatalf("RenderDistributorBundleID() error = %v", err)
}
if bundleID != "weatherreporter.home.daily" {
if bundleID != "weatherreporter.home.daily.2026-06-07T1800-2026-06-08T0600" {
t.Fatalf("bundleID = %q, want rendered value", bundleID)
}
values.BundleID = bundleID
pipelineID, err := RenderDistributorPipelineID("weatherreporter.{artifact_group}.{bundle_id}", values)
pipelineID, err := RenderDistributorPipelineID("weatherreporter.{artifact_group}.{storm_id}.{bundle_id}", values)
if err != nil {
t.Fatalf("RenderDistributorPipelineID() error = %v", err)
}
if pipelineID != "weatherreporter.daily.weatherreporter.home.daily" {
if pipelineID != "weatherreporter.daily.2026-06-07T1800-2026-06-08T0600.weatherreporter.home.daily.2026-06-07T1800-2026-06-08T0600" {
t.Fatalf("pipelineID = %q, want rendered pipeline ID", pipelineID)
}
idempotencyKey, err := RenderDistributorIdempotencyKey("{bundle_id}.{run_id}", values)
idempotencyKey, err := RenderDistributorIdempotencyKey("{bundle_id}.{storm_id}.{run_id}", values)
if err != nil {
t.Fatalf("RenderDistributorIdempotencyKey() error = %v", err)
}
if idempotencyKey != "weatherreporter.home.daily.20260607T120000Z" {
if idempotencyKey != "weatherreporter.home.daily.2026-06-07T1800-2026-06-08T0600.2026-06-07T1800-2026-06-08T0600.20260607T120000Z" {
t.Fatalf("idempotencyKey = %q, want rendered run key", idempotencyKey)
}
reportPaths, err := RenderDistributorReportPaths([]string{
reportPaths, err := RenderDistributorReportPaths("reports.daily.distributor.path_templates", []string{
"{valid_start_date}/{artifact_group}/{valid_start_stamp}-{valid_end_stamp}-{run_id}.md",
"storm/{storm_id}/index.md",
"{valid_start_date}/{artifact_group}/latest.md",
}, values)
if err != nil {
@@ -1094,6 +1357,7 @@ func TestDistributorTemplateRendering(t *testing.T) {
}
wantPaths := []string{
"2026-06-07/daily/2026-06-07T1800-2026-06-08T0600-20260607T120000Z.md",
"storm/2026-06-07T1800-2026-06-08T0600/index.md",
"2026-06-07/daily/latest.md",
}
if strings.Join(reportPaths, "\n") != strings.Join(wantPaths, "\n") {
@@ -1101,6 +1365,50 @@ func TestDistributorTemplateRendering(t *testing.T) {
}
}
func TestDistributorReportPathRenderingUsesCallerName(t *testing.T) {
values := DistributorTemplateValues{
BatchOutputName: "report.md",
}
tests := []struct {
name string
templates []string
wantErr string
}{
{
name: "UnknownVariable",
templates: []string{"{unknown}.md"},
wantErr: `report.daily.distributor_path_templates[0] contains unknown template variable "unknown"`,
},
{
name: "InvalidPath",
templates: []string{"/{batch_output_name}"},
wantErr: "report.daily.distributor_path_templates[0] must render a relative path",
},
{
name: "DuplicatePath",
templates: []string{"latest.md", "latest.md"},
wantErr: `report.daily.distributor_path_templates renders duplicate path "latest.md"`,
},
{
name: "Empty",
templates: nil,
wantErr: "report.daily.distributor_path_templates must contain at least one entry",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := RenderDistributorReportPaths("report.daily.distributor_path_templates", tt.templates, values)
if err == nil {
t.Fatal("RenderDistributorReportPaths() error = nil, want error")
}
if !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
}
})
}
}
func TestDistributorBatchTemplateRendering(t *testing.T) {
values := DistributorBatchTemplateValues{
LocationID: "home",
@@ -1233,12 +1541,15 @@ func TestDistributorReportPathRenderingRejectsInvalidValues(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := RenderDistributorReportPaths([]string{"{batch_output_name}"}, DistributorTemplateValues{
_, err := RenderDistributorReportPaths("report.daily.distributor_path_templates", []string{"{batch_output_name}"}, DistributorTemplateValues{
BatchOutputName: tt.batchOutputName,
})
if err == nil {
t.Fatal("RenderDistributorReportPaths() error = nil, want error")
}
if !strings.Contains(err.Error(), "report.daily.distributor_path_templates[0]") {
t.Fatalf("error = %q, want caller path name", err.Error())
}
})
}
}

View File

@@ -31,9 +31,6 @@ func Defaults() Config {
PipelineIDTemplate: "",
BundleIDTemplate: "weatherreporter.{location_id}.{report_id}",
IdempotencyKeyTemplate: "{bundle_id}.{run_id}",
ReportPathTemplates: []string{
"{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md",
},
Batch: DistributorBatchNotifyConfig{
Enabled: true,
PipelineIDTemplate: "weatherreporter",

View File

@@ -18,6 +18,7 @@ type DistributorTemplateValues struct {
ValidEndTime string
ValidStartStamp string
ValidEndStamp string
StormID string
BundleID string
}
@@ -41,6 +42,7 @@ var distributorTemplateVariables = map[string]struct{}{
"valid_end_time": {},
"valid_start_stamp": {},
"valid_end_stamp": {},
"storm_id": {},
}
var distributorIdempotencyTemplateVariables = map[string]struct{}{
@@ -55,6 +57,7 @@ var distributorIdempotencyTemplateVariables = map[string]struct{}{
"valid_end_time": {},
"valid_start_stamp": {},
"valid_end_stamp": {},
"storm_id": {},
"bundle_id": {},
}
@@ -129,23 +132,23 @@ func RenderDistributorBatchIdempotencyKey(template string, values DistributorBat
return rendered, nil
}
func RenderDistributorReportPaths(templates []string, values DistributorTemplateValues) ([]string, error) {
func RenderDistributorReportPaths(name string, templates []string, values DistributorTemplateValues) ([]string, error) {
if len(templates) == 0 {
return nil, fmt.Errorf("notify.distributor.report_path_templates must contain at least one entry")
return nil, fmt.Errorf("%s must contain at least one entry", name)
}
paths := make([]string, 0, len(templates))
seen := make(map[string]struct{}, len(templates))
for i, template := range templates {
name := fmt.Sprintf("notify.distributor.report_path_templates[%d]", i)
rendered, err := renderDistributorTemplate(name, template, values, distributorTemplateVariables)
itemName := fmt.Sprintf("%s[%d]", name, i)
rendered, err := renderDistributorTemplate(itemName, template, values, distributorTemplateVariables)
if err != nil {
return nil, err
}
if err := ValidateDistributorReportPath(name, rendered); err != nil {
if err := ValidateDistributorReportPath(itemName, rendered); err != nil {
return nil, err
}
if _, ok := seen[rendered]; ok {
return nil, fmt.Errorf("notify.distributor.report_path_templates renders duplicate path %q", rendered)
return nil, fmt.Errorf("%s renders duplicate path %q", name, rendered)
}
seen[rendered] = struct{}{}
paths = append(paths, rendered)
@@ -243,6 +246,8 @@ func distributorTemplateValue(variable string, values DistributorTemplateValues)
return values.ValidStartStamp
case "valid_end_stamp":
return values.ValidEndStamp
case "storm_id":
return values.StormID
case "bundle_id":
return values.BundleID
default:

View File

@@ -17,6 +17,10 @@ func (cfg Config) ReportModuleOverrides() (map[report.ID][]module.ConfigItem, er
})
}
func (cfg Config) ReportDistributorPathOverrides() (map[report.ID][]string, error) {
return traverseReportDistributorPathOverrides(cfg)
}
func normalizeReportModules(cfg *Config) error {
if cfg.Reports == nil {
cfg.Reports = map[string]ReportConfig{}
@@ -35,6 +39,11 @@ func validateReportModules(cfg Config) error {
return err
}
func validateReportDistributorPathOverrides(cfg Config) error {
_, err := traverseReportDistributorPathOverrides(cfg)
return err
}
type reportModuleTraversalOptions struct {
normalizeOptions bool
updateConfig bool
@@ -82,6 +91,59 @@ func traverseReportModules(cfg *Config, opts reportModuleTraversalOptions) (map[
return overrides, nil
}
func traverseReportDistributorPathOverrides(cfg Config) (map[report.ID][]string, error) {
overrides := map[report.ID][]string{}
if cfg.Reports == nil {
return overrides, nil
}
reportRegistry := report.DefaultRegistry()
seenReports := map[report.ID]string{}
for key, reportCfg := range cfg.Reports {
reportID, err := report.IDForConfigKey(key)
if err != nil {
return nil, fmt.Errorf("reports.%s: %w", key, err)
}
if previous, ok := seenReports[reportID]; ok {
return nil, fmt.Errorf("reports.%s duplicates report override %q", key, previous)
}
seenReports[reportID] = key
if _, err := reportRegistry.Lookup(reportID); err != nil {
return nil, fmt.Errorf("reports.%s: %w", key, err)
}
if !reportCfg.Distributor.pathTemplatesSet {
continue
}
if err := validateReportDistributorPathTemplates(key, reportCfg.Distributor.PathTemplates); err != nil {
return nil, err
}
overrides[reportID] = append([]string(nil), reportCfg.Distributor.PathTemplates...)
}
return overrides, nil
}
func validateReportDistributorPathTemplates(reportKey string, templates []string) error {
name := fmt.Sprintf("reports.%s.distributor.path_templates", reportKey)
_, err := RenderDistributorReportPaths(name, templates, sampleDistributorTemplateValues())
return err
}
func sampleDistributorTemplateValues() DistributorTemplateValues {
return DistributorTemplateValues{
LocationID: "location",
ReportID: "report",
RunID: "run",
ArtifactGroup: "artifact",
BatchOutputName: "report.md",
ValidStartDate: "2026-05-29",
ValidEndDate: "2026-05-30",
ValidStartTime: "0000",
ValidEndTime: "0000",
ValidStartStamp: "2026-05-29T0000",
ValidEndStamp: "2026-05-30T0000",
StormID: "2026-05-29T0000-2026-05-30T0000",
}
}
func moduleItemsFromConfig(registry briefing.ModuleRegistry, reportKey string, items []ModuleConfigItem, normalizeOptions bool) ([]module.ConfigItem, []ModuleConfigItem, error) {
out := make([]module.ConfigItem, 0, len(items))
normalizedItems := append([]ModuleConfigItem(nil), items...)

View File

@@ -12,6 +12,9 @@ func Validate(cfg Config) error {
if err := validateReportModules(cfg); err != nil {
return err
}
if err := validateReportDistributorPathOverrides(cfg); err != nil {
return err
}
if cfg.WeatherAPI.BaseURL != "" {
parsed, err := url.Parse(cfg.WeatherAPI.BaseURL)
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
@@ -121,22 +124,7 @@ func validateDistributorNotify(cfg DistributorNotifyConfig) error {
if err := validateDistributorTemplate("notify.distributor.idempotency_key_template", cfg.IdempotencyKeyTemplate, distributorIdempotencyTemplateVariables); err != nil {
return err
}
if len(cfg.ReportPathTemplates) == 0 {
return fmt.Errorf("notify.distributor.report_path_templates must contain at least one entry when enabled")
}
values := DistributorTemplateValues{
LocationID: "location",
ReportID: "report",
RunID: "run",
ArtifactGroup: "artifact",
BatchOutputName: "report.md",
ValidStartDate: "2026-05-29",
ValidEndDate: "2026-05-30",
ValidStartTime: "0000",
ValidEndTime: "0000",
ValidStartStamp: "2026-05-29T0000",
ValidEndStamp: "2026-05-30T0000",
}
values := sampleDistributorTemplateValues()
bundleID, err := RenderDistributorBundleID(cfg.BundleIDTemplate, values)
if err != nil {
return err
@@ -145,9 +133,6 @@ func validateDistributorNotify(cfg DistributorNotifyConfig) error {
if _, err := RenderDistributorPipelineID(cfg.PipelineIDTemplate, values); err != nil {
return err
}
if _, err := RenderDistributorReportPaths(cfg.ReportPathTemplates, values); err != nil {
return err
}
if err := validateDistributorBatchNotify(cfg.Batch); err != nil {
return err
}

View File

@@ -18,11 +18,15 @@ func dailyDefinition() Definition {
ComparisonStrategy: CompareSameValidDate,
ArtifactGroup: "daily",
BatchOutputName: "daily.md",
Generated: true,
CompatiblePriorIDs: []ID{Daily},
Modules: dailyModules(),
resolve: resolveDaily,
runIDDisambiguator: validStartDateRunIDDisambiguator,
DistributorPathTemplates: []string{
"daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md",
},
Generated: true,
CompatiblePriorIDs: []ID{Daily},
Modules: dailyModules(),
resolve: resolveDaily,
runIDDisambiguator: validStartDateRunIDDisambiguator,
}
}

View File

@@ -46,22 +46,23 @@ const (
)
type Definition struct {
ID ID
Name string
PromptID string
GenerationMode GenerationMode
TemplateID string
GeneratedTextSchemaID string
ComparisonStrategy ComparisonStrategy
ArtifactGroup string
BatchOutputName string
Generated bool
CompatiblePriorIDs []ID
Modules []module.ConfigItem
Morning bool
Evening bool
resolve func(ResolveRequest) (timeutil.Period, error)
runIDDisambiguator func(Resolved) string
ID ID
Name string
PromptID string
GenerationMode GenerationMode
TemplateID string
GeneratedTextSchemaID string
ComparisonStrategy ComparisonStrategy
ArtifactGroup string
BatchOutputName string
DistributorPathTemplates []string
Generated bool
CompatiblePriorIDs []ID
Modules []module.ConfigItem
Morning bool
Evening bool
resolve func(ResolveRequest) (timeutil.Period, error)
runIDDisambiguator func(Resolved) string
}
func (d Definition) ResolvePeriod(req ResolveRequest) (timeutil.Period, error) {

View File

@@ -20,10 +20,13 @@ func hourlyDefinition() Definition {
ComparisonStrategy: CompareRollingWindow,
ArtifactGroup: "hourly",
BatchOutputName: "hourly.md",
Generated: true,
CompatiblePriorIDs: []ID{Hourly},
Modules: hourlyModules(),
resolve: resolveHourly,
DistributorPathTemplates: []string{
"hourly/index.md",
},
Generated: true,
CompatiblePriorIDs: []ID{Hourly},
Modules: hourlyModules(),
resolve: resolveHourly,
}
}

View File

@@ -520,6 +520,88 @@ func TestRegistryDefinitionsDeclarePathAndCompatibilityPolicy(t *testing.T) {
}
}
func TestGeneratedRegistryDefinitionsDeclareDistributorPathDefaults(t *testing.T) {
for _, definition := range DefaultRegistry().All() {
if !definition.Generated {
continue
}
if len(definition.DistributorPathTemplates) == 0 {
t.Fatalf("%s DistributorPathTemplates is empty", definition.ID)
}
}
}
func TestRegistryDefinitionsDeclareDefaultDistributorPathTemplates(t *testing.T) {
tests := []struct {
id ID
want []string
}{
{
id: Hourly,
want: []string{
"hourly/index.md",
},
},
{
id: Daily,
want: []string{
"daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md",
},
},
{
id: Today,
want: []string{
"daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md",
"today/index.md",
},
},
{
id: Tomorrow,
want: []string{
"daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md",
"tomorrow/index.md",
},
},
{
id: ThreeDay,
want: []string{
"three-day/{valid_start_date}/{run_id}.md",
"three-day/{valid_start_date}/index.md",
},
},
{
id: Weekend,
want: []string{
"weekend/{valid_start_date}/{run_id}.md",
"weekend/{valid_start_date}/index.md",
},
},
{
id: Storm,
want: []string{
"storm/{storm_id}/{run_id}.md",
"storm/{storm_id}/index.md",
},
},
}
registry := DefaultRegistry()
for _, tt := range tests {
t.Run(string(tt.id), func(t *testing.T) {
definition, err := registry.Lookup(tt.id)
if err != nil {
t.Fatalf("Lookup() error = %v", err)
}
if !reflect.DeepEqual(definition.DistributorPathTemplates, tt.want) {
t.Fatalf("DistributorPathTemplates = %#v, want %#v", definition.DistributorPathTemplates, tt.want)
}
})
}
}
func TestRegistryDefinitionsDeclareDefaultModules(t *testing.T) {
tests := []struct {
id ID
@@ -682,6 +764,9 @@ func TestRegistryAppliesModuleOverridesWithoutChangingDefaults(t *testing.T) {
if len(defaultDefinition.ModuleIDs()) <= len(definition.ModuleIDs()) {
t.Fatalf("default ModuleIDs() = %#v, want original defaults unchanged", defaultDefinition.ModuleIDs())
}
if !reflect.DeepEqual(definition.DistributorPathTemplates, defaultDefinition.DistributorPathTemplates) {
t.Fatalf("overridden DistributorPathTemplates = %#v, want %#v", definition.DistributorPathTemplates, defaultDefinition.DistributorPathTemplates)
}
}
func TestRegistryRejectsModuleOverrideForUnknownReport(t *testing.T) {

View File

@@ -30,8 +30,7 @@ func DefaultRegistry() Registry {
func (r Registry) WithModuleOverrides(overrides map[ID][]module.ConfigItem) (Registry, error) {
next := Registry{definitions: map[ID]Definition{}}
for id, definition := range r.definitions {
definition.Modules = append([]module.ConfigItem(nil), definition.Modules...)
next.definitions[id] = definition
next.definitions[id] = cloneDefinition(definition)
}
for id, items := range overrides {
definition, ok := next.definitions[id]
@@ -44,6 +43,12 @@ func (r Registry) WithModuleOverrides(overrides map[ID][]module.ConfigItem) (Reg
return next, nil
}
func cloneDefinition(definition Definition) Definition {
definition.Modules = cloneModuleItems(definition.Modules)
definition.DistributorPathTemplates = append([]string(nil), definition.DistributorPathTemplates...)
return definition
}
func moduleItems(ids ...module.ID) []module.ConfigItem {
items := make([]module.ConfigItem, 0, len(ids))
for _, id := range ids {

View File

@@ -17,6 +17,10 @@ func stormDefinition() Definition {
ComparisonStrategy: CompareExplicitWindow,
ArtifactGroup: "storm",
BatchOutputName: "storm.md",
DistributorPathTemplates: []string{
"storm/{storm_id}/{run_id}.md",
"storm/{storm_id}/index.md",
},
Generated: true,
CompatiblePriorIDs: []ID{Storm},
Modules: stormModules(),

View File

@@ -16,6 +16,10 @@ func threeDayDefinition() Definition {
ComparisonStrategy: CompareSameValidDate,
ArtifactGroup: "three-day",
BatchOutputName: "three-day.md",
DistributorPathTemplates: []string{
"three-day/{valid_start_date}/{run_id}.md",
"three-day/{valid_start_date}/index.md",
},
Generated: true,
CompatiblePriorIDs: []ID{ThreeDay},
Modules: threeDayModules(),

View File

@@ -16,11 +16,16 @@ func todayDefinition() Definition {
ComparisonStrategy: CompareSameValidDate,
ArtifactGroup: "today",
BatchOutputName: "today.md",
Generated: true,
CompatiblePriorIDs: []ID{Today},
Modules: todayModules(),
Morning: true,
resolve: resolveToday,
DistributorPathTemplates: []string{
"daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md",
"today/index.md",
},
Generated: true,
CompatiblePriorIDs: []ID{Today},
Modules: todayModules(),
Morning: true,
resolve: resolveToday,
}
}

View File

@@ -16,11 +16,16 @@ func tomorrowDefinition() Definition {
ComparisonStrategy: CompareSameValidDate,
ArtifactGroup: "tomorrow",
BatchOutputName: "tomorrow.md",
Generated: true,
CompatiblePriorIDs: []ID{Tomorrow},
Modules: tomorrowModules(),
Evening: true,
resolve: resolveTomorrow,
DistributorPathTemplates: []string{
"daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md",
"tomorrow/index.md",
},
Generated: true,
CompatiblePriorIDs: []ID{Tomorrow},
Modules: tomorrowModules(),
Evening: true,
resolve: resolveTomorrow,
}
}

View File

@@ -17,6 +17,10 @@ func weekendDefinition() Definition {
ComparisonStrategy: CompareWeekendWindow,
ArtifactGroup: "weekend",
BatchOutputName: "weekend.md",
DistributorPathTemplates: []string{
"weekend/{valid_start_date}/{run_id}.md",
"weekend/{valid_start_date}/index.md",
},
Generated: true,
CompatiblePriorIDs: []ID{Weekend},
Modules: weekendModules(),