Compare commits
6 Commits
7adf5e1b08
...
15ee4af1a1
| Author | SHA1 | Date | |
|---|---|---|---|
| 15ee4af1a1 | |||
| 4c606eb39f | |||
| 8d2ac163ae | |||
| 8709b5f4d8 | |||
| fd48ebecb8 | |||
| 021e5dd8b1 |
@@ -101,15 +101,6 @@ contains all managed Markdown reports from that batch.
|
|||||||
Default: `weatherreporter.{location_id}.{report_id}`.
|
Default: `weatherreporter.{location_id}.{report_id}`.
|
||||||
- `idempotency_key_template`: template for single-report distributor
|
- `idempotency_key_template`: template for single-report distributor
|
||||||
idempotency keys. Default: `{bundle_id}.{run_id}`.
|
idempotency keys. Default: `{bundle_id}.{run_id}`.
|
||||||
- `report_path_templates`: ordered list of templates for Markdown report paths
|
|
||||||
inside distributor bundles. For `generate <report>`, each rendered path maps
|
|
||||||
to the generated report's managed Markdown source. For `run morning` and
|
|
||||||
`run evening`, the templates are rendered once per included report and each
|
|
||||||
rendered path maps that report's managed Markdown source into the one batch
|
|
||||||
bundle. Default:
|
|
||||||
```yaml
|
|
||||||
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
|
|
||||||
```
|
|
||||||
- `batch.enabled`: whether batch distributor notification config is active
|
- `batch.enabled`: whether batch distributor notification config is active
|
||||||
when distributor notification is enabled. Default: `true`.
|
when distributor notification is enabled. Default: `true`.
|
||||||
- `batch.pipeline_id_template`: template for batch distributor pipeline IDs.
|
- `batch.pipeline_id_template`: template for batch distributor pipeline IDs.
|
||||||
@@ -124,10 +115,13 @@ contains all managed Markdown reports from that batch.
|
|||||||
|
|
||||||
Single-report templates support `location_id`, `report_id`, `run_id`,
|
Single-report templates support `location_id`, `report_id`, `run_id`,
|
||||||
`artifact_group`, `batch_output_name`, `valid_start_date`, `valid_end_date`,
|
`artifact_group`, `batch_output_name`, `valid_start_date`, `valid_end_date`,
|
||||||
`valid_start_time`, `valid_end_time`, `valid_start_stamp`, and
|
`valid_start_time`, `valid_end_time`, `valid_start_stamp`, `valid_end_stamp`,
|
||||||
`valid_end_stamp`. Date values use `YYYY-MM-DD`, time values use `HHMM`, and
|
and `storm_id`. Date values use `YYYY-MM-DD`, time values use `HHMM`, and
|
||||||
stamp values use `YYYY-MM-DDTHHMM` in the effective report timezone.
|
stamp values use `YYYY-MM-DDTHHMM` in the effective report timezone.
|
||||||
`pipeline_id_template` and `idempotency_key_template` may also use `bundle_id`.
|
`storm_id` is derived from the storm report valid period as
|
||||||
|
`{valid_start_stamp}-{valid_end_stamp}`; it renders empty for non-storm
|
||||||
|
reports. `pipeline_id_template` and `idempotency_key_template` may also use
|
||||||
|
`bundle_id`.
|
||||||
|
|
||||||
The rendered pipeline ID selects the configured distributor `http_upload`
|
The rendered pipeline ID selects the configured distributor `http_upload`
|
||||||
workflow. The rendered bundle ID is the stable logical source identity for the
|
workflow. The rendered bundle ID is the stable logical source identity for the
|
||||||
@@ -146,6 +140,18 @@ rendered bundle path for every included report before distributor is called.
|
|||||||
Managed Markdown report paths are the only upload source files; copies written
|
Managed Markdown report paths are the only upload source files; copies written
|
||||||
with `--out` or `--out-dir` are never uploaded.
|
with `--out` or `--out-dir` are never uploaded.
|
||||||
|
|
||||||
|
Distributor bundle paths are report-specific. Weatherreporter uses
|
||||||
|
`reports.<report>.distributor.path_templates` when that override is configured;
|
||||||
|
otherwise it uses the report definition defaults:
|
||||||
|
|
||||||
|
- `hourly`: `hourly/index.md`
|
||||||
|
- `daily`: `daily/{valid_start_date}/{run_id}.md`, `daily/{valid_start_date}/index.md`
|
||||||
|
- `today`: `daily/{valid_start_date}/{run_id}.md`, `daily/{valid_start_date}/index.md`, `today/index.md`
|
||||||
|
- `tomorrow`: `daily/{valid_start_date}/{run_id}.md`, `daily/{valid_start_date}/index.md`, `tomorrow/index.md`
|
||||||
|
- `three_day`: `three-day/{valid_start_date}/{run_id}.md`, `three-day/{valid_start_date}/index.md`
|
||||||
|
- `weekend`: `weekend/{valid_start_date}/{run_id}.md`, `weekend/{valid_start_date}/index.md`
|
||||||
|
- `storm`: `storm/{storm_id}/{run_id}.md`, `storm/{storm_id}/index.md`
|
||||||
|
|
||||||
The upload token is read from the environment variable named by `token_env`
|
The upload token is read from the environment variable named by `token_env`
|
||||||
after config loading and `secrets.directory` processing. Config files should
|
after config loading and `secrets.directory` processing. Config files should
|
||||||
name the variable only; they should not contain the token value.
|
name the variable only; they should not contain the token value.
|
||||||
@@ -220,12 +226,19 @@ Each report entry supports:
|
|||||||
|
|
||||||
- `deterministic_modules`: ordered module list. Entries may be string module
|
- `deterministic_modules`: ordered module list. Entries may be string module
|
||||||
IDs or objects with `id` and optional `options`.
|
IDs or objects with `id` and optional `options`.
|
||||||
|
- `distributor.path_templates`: optional ordered distributor bundle path
|
||||||
|
templates for this report. If omitted, the report definition defaults are
|
||||||
|
used. If present, the list must contain at least one template.
|
||||||
|
|
||||||
Example:
|
Example:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
reports:
|
reports:
|
||||||
daily:
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "daily/{valid_start_date}/{run_id}.md"
|
||||||
|
- "daily/{valid_start_date}/index.md"
|
||||||
deterministic_modules:
|
deterministic_modules:
|
||||||
- metadata
|
- metadata
|
||||||
- current_conditions
|
- current_conditions
|
||||||
|
|||||||
@@ -163,10 +163,11 @@ all planned reports finish, app orchestration evaluates batch notification:
|
|||||||
3. If any planned report failed, the batch notification result is `skipped`
|
3. If any planned report failed, the batch notification result is `skipped`
|
||||||
with reason `one or more reports failed`, and distributor is not called.
|
with reason `one or more reports failed`, and distributor is not called.
|
||||||
4. If every report succeeded, app orchestration renders batch pipeline, bundle
|
4. If every report succeeded, app orchestration renders batch pipeline, bundle
|
||||||
ID, and idempotency key templates, renders `report_path_templates` for each
|
ID, and idempotency key templates, renders report-specific distributor
|
||||||
included report, validates every managed source path and bundle path, checks
|
paths for each included report, validates every managed source path and
|
||||||
duplicate bundle paths across the batch, calls the notifier once with a
|
bundle path, checks duplicate bundle paths across the batch, calls the
|
||||||
multi-file request, and saves a batch notification debug artifact.
|
notifier once with a multi-file request, and saves a batch notification
|
||||||
|
debug artifact.
|
||||||
|
|
||||||
Batch notification failure records a top-level failed notification, increments
|
Batch notification failure records a top-level failed notification, increments
|
||||||
the aggregate batch failure count, and returns an aggregate batch error without
|
the aggregate batch failure count, and returns an aggregate batch error without
|
||||||
|
|||||||
@@ -61,12 +61,16 @@ and bundle paths from:
|
|||||||
- `pipeline_id_template`
|
- `pipeline_id_template`
|
||||||
- `bundle_id_template`
|
- `bundle_id_template`
|
||||||
- `idempotency_key_template`
|
- `idempotency_key_template`
|
||||||
- `report_path_templates`
|
- report-specific path templates
|
||||||
|
|
||||||
For batch uploads, the app layer renders pipeline ID, bundle ID, and
|
For batch uploads, the app layer renders pipeline ID, bundle ID, and
|
||||||
idempotency key from `notify.distributor.batch.*`, renders
|
idempotency key from `notify.distributor.batch.*`, resolves report-specific
|
||||||
`report_path_templates` once per included report, and passes the resulting
|
path templates once per included report, and passes the resulting multi-file
|
||||||
multi-file request to this adapter.
|
request to this adapter.
|
||||||
|
|
||||||
|
Report-specific path resolution happens entirely in the app layer. Explicit
|
||||||
|
`reports.<report>.distributor.path_templates` overrides take precedence over
|
||||||
|
report definition defaults.
|
||||||
|
|
||||||
The token value is read from the environment variable named by `token_env`
|
The token value is read from the environment variable named by `token_env`
|
||||||
after config loading and `secrets.directory` processing.
|
after config loading and `secrets.directory` processing.
|
||||||
|
|||||||
@@ -256,17 +256,16 @@ top-level `notification` object with fields such as `status`, `reason`,
|
|||||||
Distributor notification is configured with `notify.distributor` and is
|
Distributor notification is configured with `notify.distributor` and is
|
||||||
disabled by default. For `generate <report>`, weatherreporter uploads the
|
disabled by default. For `generate <report>`, weatherreporter uploads the
|
||||||
managed Markdown report path recorded in the report result and metadata. That
|
managed Markdown report path recorded in the report result and metadata. That
|
||||||
single source file can be mapped to one or more configured bundle paths. By
|
single source file is mapped to report-specific bundle paths. Extra copies
|
||||||
default, it is mapped to one dated report path. Extra copies written by `--out`
|
written by `--out` or `--out-dir` are operator conveniences only.
|
||||||
or `--out-dir` are operator conveniences only.
|
|
||||||
|
|
||||||
For `run morning` and `run evening`, per-report notification is suppressed. If
|
For `run morning` and `run evening`, per-report notification is suppressed. If
|
||||||
`notify.distributor.enabled` and `notify.distributor.batch.enabled` are both
|
`notify.distributor.enabled` and `notify.distributor.batch.enabled` are both
|
||||||
true, the batch uploads once after all reports finish successfully. The upload
|
true, the batch uploads once after all reports finish successfully. The upload
|
||||||
contains one file mapping set per included report. Each mapping uses the
|
contains one file mapping set per included report. Each mapping uses the
|
||||||
managed Markdown report as the source and renders `report_path_templates` using
|
managed Markdown report as the source and report-specific path templates for
|
||||||
that report's normal template values. All rendered bundle paths across the
|
that report. All rendered bundle paths across the batch must be unique. If any
|
||||||
batch must be unique. If any report fails, weatherreporter records a top-level
|
report fails, weatherreporter records a top-level
|
||||||
notification status of `skipped` with reason `one or more reports failed` and
|
notification status of `skipped` with reason `one or more reports failed` and
|
||||||
does not call distributor. If batch notification is disabled, run commands do
|
does not call distributor. If batch notification is disabled, run commands do
|
||||||
not fall back to per-report uploads.
|
not fall back to per-report uploads.
|
||||||
|
|||||||
@@ -300,15 +300,15 @@ Relevant docs: [Configuration reference](config.md),
|
|||||||
Symptom: a batch returns nonzero with duplicate bundle path context before a
|
Symptom: a batch returns nonzero with duplicate bundle path context before a
|
||||||
distributor run ID is accepted.
|
distributor run ID is accepted.
|
||||||
|
|
||||||
Likely cause: `notify.distributor.report_path_templates` rendered the same
|
Likely cause: report-specific distributor path templates rendered the same
|
||||||
bundle-relative path for two included reports in the same batch.
|
bundle-relative path for two included reports in the same batch.
|
||||||
|
|
||||||
Diagnostic: inspect the error in stdout or stderr. The validation error
|
Diagnostic: inspect the error in stdout or stderr. The validation error
|
||||||
includes the duplicate bundle path plus the report IDs, RunIDs, and managed
|
includes the duplicate bundle path plus the report IDs, RunIDs, and managed
|
||||||
source paths involved.
|
source paths involved.
|
||||||
|
|
||||||
Safe fix: change `notify.distributor.report_path_templates` so every report in
|
Safe fix: configure a per-report distributor path override so every report in a
|
||||||
a batch renders a unique path. Include values such as `{artifact_group}`,
|
batch renders a unique path. Include values such as `{artifact_group}`,
|
||||||
`{valid_start_date}`, `{batch_output_name}`, or `{run_id}` when needed.
|
`{valid_start_date}`, `{batch_output_name}`, or `{run_id}` when needed.
|
||||||
|
|
||||||
Relevant docs: [Configuration reference](config.md),
|
Relevant docs: [Configuration reference](config.md),
|
||||||
@@ -411,7 +411,7 @@ Diagnostic: inspect stdout JSON or stderr status lines for
|
|||||||
`notificationError` or the top-level batch notification `error`. Confirm
|
`notificationError` or the top-level batch notification `error`. Confirm
|
||||||
`notify.distributor.endpoint`,
|
`notify.distributor.endpoint`,
|
||||||
`notify.distributor.pipeline_id_template`,
|
`notify.distributor.pipeline_id_template`,
|
||||||
`notify.distributor.report_path_templates`, and token configuration. Token
|
report-specific distributor paths, and token configuration. Token
|
||||||
values are redacted from weatherreporter errors.
|
values are redacted from weatherreporter errors.
|
||||||
|
|
||||||
If the upload was accepted but destination output did not change, inspect the
|
If the upload was accepted but destination output did not change, inspect the
|
||||||
|
|||||||
@@ -24,8 +24,6 @@ notify:
|
|||||||
pipeline_id_template: "weatherreporter.{report_id}"
|
pipeline_id_template: "weatherreporter.{report_id}"
|
||||||
bundle_id_template: "weatherreporter.{location_id}.{report_id}"
|
bundle_id_template: "weatherreporter.{location_id}.{report_id}"
|
||||||
idempotency_key_template: "{bundle_id}.{run_id}"
|
idempotency_key_template: "{bundle_id}.{run_id}"
|
||||||
report_path_templates:
|
|
||||||
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
|
|
||||||
batch:
|
batch:
|
||||||
enabled: true
|
enabled: true
|
||||||
pipeline_id_template: "weatherreporter"
|
pipeline_id_template: "weatherreporter"
|
||||||
@@ -74,6 +72,10 @@ recent_change:
|
|||||||
|
|
||||||
reports:
|
reports:
|
||||||
daily:
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "daily/{valid_start_date}/{run_id}.md"
|
||||||
|
- "daily/{valid_start_date}/index.md"
|
||||||
deterministic_modules:
|
deterministic_modules:
|
||||||
- metadata
|
- metadata
|
||||||
- current_conditions
|
- current_conditions
|
||||||
|
|||||||
@@ -934,7 +934,7 @@ func buildNotificationRequest(cfg config.Config, resolved report.Resolved, repor
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return NotificationRequest{}, err
|
return NotificationRequest{}, err
|
||||||
}
|
}
|
||||||
bundlePaths, err := config.RenderDistributorReportPaths(cfg.Notify.Distributor.ReportPathTemplates, values)
|
bundlePaths, err := renderDistributorReportBundlePaths(cfg, resolved, metadata.RunID, reportPath, values)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return NotificationRequest{}, err
|
return NotificationRequest{}, err
|
||||||
}
|
}
|
||||||
@@ -964,9 +964,45 @@ func distributorTemplateValuesForReport(cfg config.Config, resolved report.Resol
|
|||||||
if err := addDistributorValidPeriodValues(&values, resolved.ValidPeriod, cfg.WeatherAPI.Timezone); err != nil {
|
if err := addDistributorValidPeriodValues(&values, resolved.ValidPeriod, cfg.WeatherAPI.Timezone); err != nil {
|
||||||
return config.DistributorTemplateValues{}, err
|
return config.DistributorTemplateValues{}, err
|
||||||
}
|
}
|
||||||
|
if resolved.Definition.ID == report.Storm {
|
||||||
|
values.StormID = values.ValidStartStamp + "-" + values.ValidEndStamp
|
||||||
|
}
|
||||||
return values, nil
|
return values, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func renderDistributorReportBundlePaths(cfg config.Config, resolved report.Resolved, runID string, sourcePath string, values config.DistributorTemplateValues) ([]string, error) {
|
||||||
|
templates, name, err := distributorPathTemplatesForReport(cfg, resolved.Definition)
|
||||||
|
if err != nil {
|
||||||
|
return nil, distributorReportPathError(resolved.Definition.ID, runID, sourcePath, err)
|
||||||
|
}
|
||||||
|
paths, err := config.RenderDistributorReportPaths(name, templates, values)
|
||||||
|
if err != nil {
|
||||||
|
return nil, distributorReportPathError(resolved.Definition.ID, runID, sourcePath, err)
|
||||||
|
}
|
||||||
|
return paths, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func distributorPathTemplatesForReport(cfg config.Config, definition report.Definition) ([]string, string, error) {
|
||||||
|
overrides, err := cfg.ReportDistributorPathOverrides()
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", err
|
||||||
|
}
|
||||||
|
if templates, ok := overrides[definition.ID]; ok {
|
||||||
|
return append([]string(nil), templates...), fmt.Sprintf("reports.%s.distributor.path_templates", definition.ID), nil
|
||||||
|
}
|
||||||
|
if len(definition.DistributorPathTemplates) > 0 {
|
||||||
|
return append([]string(nil), definition.DistributorPathTemplates...), fmt.Sprintf("report.%s.distributor_path_templates", definition.ID), nil
|
||||||
|
}
|
||||||
|
return nil, "", fmt.Errorf("no distributor path templates configured")
|
||||||
|
}
|
||||||
|
|
||||||
|
func distributorReportPathError(id report.ID, runID string, sourcePath string, err error) error {
|
||||||
|
if sourcePath != "" {
|
||||||
|
return fmt.Errorf("report %q run %q source path %q: %w", id, runID, sourcePath, err)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("report %q run %q: %w", id, runID, err)
|
||||||
|
}
|
||||||
|
|
||||||
func addDistributorValidPeriodValues(values *config.DistributorTemplateValues, period timeutil.Period, timezone string) error {
|
func addDistributorValidPeriodValues(values *config.DistributorTemplateValues, period timeutil.Period, timezone string) error {
|
||||||
location, err := timeutil.LoadLocation(timezone)
|
location, err := timeutil.LoadLocation(timezone)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -739,7 +739,7 @@ func TestGenerateHourlyReportCopiesOutputAndNotifiesManagedReport(t *testing.T)
|
|||||||
if req.ReportPath == outputPath {
|
if req.ReportPath == outputPath {
|
||||||
t.Fatalf("notification used output copy %q, want managed report path", outputPath)
|
t.Fatalf("notification used output copy %q, want managed report path", outputPath)
|
||||||
}
|
}
|
||||||
wantBundlePaths := []string{"2026-05-29/hourly/hourly.md"}
|
wantBundlePaths := []string{"hourly/index.md"}
|
||||||
if strings.Join(req.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") {
|
if strings.Join(req.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") {
|
||||||
t.Fatalf("notification BundlePaths = %#v, want %#v", req.BundlePaths, wantBundlePaths)
|
t.Fatalf("notification BundlePaths = %#v, want %#v", req.BundlePaths, wantBundlePaths)
|
||||||
}
|
}
|
||||||
@@ -778,7 +778,6 @@ func TestGenerateTodayReportCopiesOutputAndNotifiesTodayTemplateValues(t *testin
|
|||||||
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{report_id}.{artifact_group}"
|
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{report_id}.{artifact_group}"
|
||||||
cfg.Notify.Distributor.BundleIDTemplate = "{artifact_group}.{batch_output_name}.{report_id}"
|
cfg.Notify.Distributor.BundleIDTemplate = "{artifact_group}.{batch_output_name}.{report_id}"
|
||||||
cfg.Notify.Distributor.IdempotencyKeyTemplate = "{bundle_id}.{run_id}"
|
cfg.Notify.Distributor.IdempotencyKeyTemplate = "{bundle_id}.{run_id}"
|
||||||
cfg.Notify.Distributor.ReportPathTemplates = []string{"{valid_start_date}/{artifact_group}/{batch_output_name}"}
|
|
||||||
resolved := resolveGenerateForTest(t, cfg, GenerateRequest{
|
resolved := resolveGenerateForTest(t, cfg, GenerateRequest{
|
||||||
Report: ReportToday,
|
Report: ReportToday,
|
||||||
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||||
@@ -834,7 +833,11 @@ func TestGenerateTodayReportCopiesOutputAndNotifiesTodayTemplateValues(t *testin
|
|||||||
if req.BundleID != "today.today.md.today" {
|
if req.BundleID != "today.today.md.today" {
|
||||||
t.Fatalf("BundleID = %q, want Today artifact group, output name, and report id", req.BundleID)
|
t.Fatalf("BundleID = %q, want Today artifact group, output name, and report id", req.BundleID)
|
||||||
}
|
}
|
||||||
wantBundlePaths := []string{"2026-05-29/today/today.md"}
|
wantBundlePaths := []string{
|
||||||
|
"daily/2026-05-29/" + result.Metadata.RunID + ".md",
|
||||||
|
"daily/2026-05-29/index.md",
|
||||||
|
"today/index.md",
|
||||||
|
}
|
||||||
if strings.Join(req.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") {
|
if strings.Join(req.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") {
|
||||||
t.Fatalf("BundlePaths = %#v, want %#v", req.BundlePaths, wantBundlePaths)
|
t.Fatalf("BundlePaths = %#v, want %#v", req.BundlePaths, wantBundlePaths)
|
||||||
}
|
}
|
||||||
@@ -981,7 +984,6 @@ func TestGenerateTomorrowReportNotificationUsesTomorrowTemplateValues(t *testing
|
|||||||
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{report_id}.{artifact_group}"
|
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{report_id}.{artifact_group}"
|
||||||
cfg.Notify.Distributor.BundleIDTemplate = "{artifact_group}.{batch_output_name}.{report_id}"
|
cfg.Notify.Distributor.BundleIDTemplate = "{artifact_group}.{batch_output_name}.{report_id}"
|
||||||
cfg.Notify.Distributor.IdempotencyKeyTemplate = "{bundle_id}.{run_id}"
|
cfg.Notify.Distributor.IdempotencyKeyTemplate = "{bundle_id}.{run_id}"
|
||||||
cfg.Notify.Distributor.ReportPathTemplates = []string{"{valid_start_date}/{artifact_group}/{batch_output_name}"}
|
|
||||||
resolved := resolveGenerateForTest(t, cfg, GenerateRequest{
|
resolved := resolveGenerateForTest(t, cfg, GenerateRequest{
|
||||||
Report: ReportTomorrow,
|
Report: ReportTomorrow,
|
||||||
}, "2026-05-29T18:00:00-05:00")
|
}, "2026-05-29T18:00:00-05:00")
|
||||||
@@ -1011,7 +1013,11 @@ func TestGenerateTomorrowReportNotificationUsesTomorrowTemplateValues(t *testing
|
|||||||
if req.BundleID != "tomorrow.tomorrow.md.tomorrow" {
|
if req.BundleID != "tomorrow.tomorrow.md.tomorrow" {
|
||||||
t.Fatalf("BundleID = %q, want artifact group, batch output name, and report id", req.BundleID)
|
t.Fatalf("BundleID = %q, want artifact group, batch output name, and report id", req.BundleID)
|
||||||
}
|
}
|
||||||
wantBundlePaths := []string{"2026-05-30/tomorrow/tomorrow.md"}
|
wantBundlePaths := []string{
|
||||||
|
"daily/2026-05-30/" + result.Metadata.RunID + ".md",
|
||||||
|
"daily/2026-05-30/index.md",
|
||||||
|
"tomorrow/index.md",
|
||||||
|
}
|
||||||
if strings.Join(req.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") {
|
if strings.Join(req.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") {
|
||||||
t.Fatalf("BundlePaths = %#v, want %#v", req.BundlePaths, wantBundlePaths)
|
t.Fatalf("BundlePaths = %#v, want %#v", req.BundlePaths, wantBundlePaths)
|
||||||
}
|
}
|
||||||
@@ -1234,7 +1240,8 @@ func TestGenerateReportNotifiesManagedReportPath(t *testing.T) {
|
|||||||
t.Fatalf("decode notification artifact: %v", err)
|
t.Fatalf("decode notification artifact: %v", err)
|
||||||
}
|
}
|
||||||
wantBundlePaths := []string{
|
wantBundlePaths := []string{
|
||||||
"2026-05-29/daily/2026-05-29-daily-" + result.Metadata.RunID + ".md",
|
"daily/2026-05-29/" + result.Metadata.RunID + ".md",
|
||||||
|
"daily/2026-05-29/index.md",
|
||||||
}
|
}
|
||||||
if notificationArtifact.PipelineID != "weatherreporter.daily" || strings.Join(notificationArtifact.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") || notificationArtifact.BundleCreated.IsZero() || notificationArtifact.RunStatus == nil || !strings.Contains(string(notificationArtifact.RunStatus.Report), "replace_older") {
|
if notificationArtifact.PipelineID != "weatherreporter.daily" || strings.Join(notificationArtifact.BundlePaths, "\n") != strings.Join(wantBundlePaths, "\n") || notificationArtifact.BundleCreated.IsZero() || notificationArtifact.RunStatus == nil || !strings.Contains(string(notificationArtifact.RunStatus.Report), "replace_older") {
|
||||||
t.Fatalf("notification artifact = %#v, want requested pipeline, status report, and created timestamp", notificationArtifact)
|
t.Fatalf("notification artifact = %#v, want requested pipeline, status report, and created timestamp", notificationArtifact)
|
||||||
@@ -2214,6 +2221,254 @@ func TestResolveGenerateStorm(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestDistributorTemplateValuesDeriveStormID(t *testing.T) {
|
||||||
|
cfg := config.Defaults()
|
||||||
|
cfg.WeatherAPI.Timezone = "America/Chicago"
|
||||||
|
now := mustParse("2026-05-29T12:00:00-05:00")
|
||||||
|
start := mustParse("2026-05-29T18:00:00-05:00")
|
||||||
|
end := mustParse("2026-05-30T06:00:00-05:00")
|
||||||
|
|
||||||
|
resolved, err := ResolveGenerate(GenerateRequest{
|
||||||
|
Config: cfg,
|
||||||
|
Report: ReportStorm,
|
||||||
|
StormStart: start,
|
||||||
|
StormEnd: end,
|
||||||
|
}, now)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolveGenerate() error = %v", err)
|
||||||
|
}
|
||||||
|
values, err := distributorTemplateValuesForReport(cfg, resolved, "run", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("distributorTemplateValuesForReport() error = %v", err)
|
||||||
|
}
|
||||||
|
if values.StormID != "2026-05-29T1800-2026-05-30T0600" {
|
||||||
|
t.Fatalf("StormID = %q, want storm valid-period stamp", values.StormID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDistributorTemplateValuesLeaveStormIDEmptyForOtherReports(t *testing.T) {
|
||||||
|
cfg := config.Defaults()
|
||||||
|
cfg.WeatherAPI.Timezone = "America/Chicago"
|
||||||
|
location, err := timeutil.LoadLocation(cfg.WeatherAPI.Timezone)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("load location: %v", err)
|
||||||
|
}
|
||||||
|
resolved, err := report.Resolve(report.Today, report.ResolveRequest{
|
||||||
|
Now: mustParse("2026-05-29T12:00:00-05:00"),
|
||||||
|
Location: location,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
values, err := distributorTemplateValuesForReport(cfg, resolved, "run", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("distributorTemplateValuesForReport() error = %v", err)
|
||||||
|
}
|
||||||
|
if values.StormID != "" {
|
||||||
|
t.Fatalf("StormID = %q, want empty for %s", values.StormID, resolved.Definition.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildNotificationRequestUsesReportDefaultBundlePaths(t *testing.T) {
|
||||||
|
cfg := config.Defaults()
|
||||||
|
cfg.Location.ID = "home"
|
||||||
|
cfg.WeatherAPI.Timezone = "America/Chicago"
|
||||||
|
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{report_id}"
|
||||||
|
location := mustLoadTestLocation(t, cfg.WeatherAPI.Timezone)
|
||||||
|
now := mustParse("2026-05-29T12:00:00-05:00")
|
||||||
|
registry := report.DefaultRegistry()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
id report.ID
|
||||||
|
req report.ResolveRequest
|
||||||
|
want func(state.Metadata) []string
|
||||||
|
source string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
id: report.Hourly,
|
||||||
|
req: report.ResolveRequest{
|
||||||
|
Now: now,
|
||||||
|
Location: location,
|
||||||
|
},
|
||||||
|
want: func(metadata state.Metadata) []string {
|
||||||
|
return []string{"hourly/index.md"}
|
||||||
|
},
|
||||||
|
source: "/managed/hourly.md",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: report.Daily,
|
||||||
|
req: report.ResolveRequest{
|
||||||
|
Now: now,
|
||||||
|
Location: location,
|
||||||
|
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||||
|
},
|
||||||
|
want: func(metadata state.Metadata) []string {
|
||||||
|
return []string{
|
||||||
|
"daily/2026-05-29/" + metadata.RunID + ".md",
|
||||||
|
"daily/2026-05-29/index.md",
|
||||||
|
}
|
||||||
|
},
|
||||||
|
source: "/managed/daily.md",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: report.Today,
|
||||||
|
req: report.ResolveRequest{
|
||||||
|
Now: now,
|
||||||
|
Location: location,
|
||||||
|
},
|
||||||
|
want: func(metadata state.Metadata) []string {
|
||||||
|
return []string{
|
||||||
|
"daily/2026-05-29/" + metadata.RunID + ".md",
|
||||||
|
"daily/2026-05-29/index.md",
|
||||||
|
"today/index.md",
|
||||||
|
}
|
||||||
|
},
|
||||||
|
source: "/managed/today.md",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: report.Tomorrow,
|
||||||
|
req: report.ResolveRequest{
|
||||||
|
Now: now,
|
||||||
|
Location: location,
|
||||||
|
},
|
||||||
|
want: func(metadata state.Metadata) []string {
|
||||||
|
return []string{
|
||||||
|
"daily/2026-05-30/" + metadata.RunID + ".md",
|
||||||
|
"daily/2026-05-30/index.md",
|
||||||
|
"tomorrow/index.md",
|
||||||
|
}
|
||||||
|
},
|
||||||
|
source: "/managed/tomorrow.md",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: report.ThreeDay,
|
||||||
|
req: report.ResolveRequest{
|
||||||
|
Now: now,
|
||||||
|
Location: location,
|
||||||
|
},
|
||||||
|
want: func(metadata state.Metadata) []string {
|
||||||
|
return []string{
|
||||||
|
"three-day/2026-05-29/" + metadata.RunID + ".md",
|
||||||
|
"three-day/2026-05-29/index.md",
|
||||||
|
}
|
||||||
|
},
|
||||||
|
source: "/managed/three-day.md",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: report.Weekend,
|
||||||
|
req: report.ResolveRequest{
|
||||||
|
Now: mustParse("2026-05-29T05:00:00-05:00"),
|
||||||
|
Location: location,
|
||||||
|
},
|
||||||
|
want: func(metadata state.Metadata) []string {
|
||||||
|
return []string{
|
||||||
|
"weekend/2026-05-29/" + metadata.RunID + ".md",
|
||||||
|
"weekend/2026-05-29/index.md",
|
||||||
|
}
|
||||||
|
},
|
||||||
|
source: "/managed/weekend.md",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: report.Storm,
|
||||||
|
req: report.ResolveRequest{
|
||||||
|
Now: now,
|
||||||
|
Location: location,
|
||||||
|
StormStart: mustParse("2026-05-29T18:00:00-05:00"),
|
||||||
|
StormEnd: mustParse("2026-05-30T06:00:00-05:00"),
|
||||||
|
},
|
||||||
|
want: func(metadata state.Metadata) []string {
|
||||||
|
return []string{
|
||||||
|
"storm/2026-05-29T1800-2026-05-30T0600/" + metadata.RunID + ".md",
|
||||||
|
"storm/2026-05-29T1800-2026-05-30T0600/index.md",
|
||||||
|
}
|
||||||
|
},
|
||||||
|
source: "/managed/storm.md",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(string(tt.id), func(t *testing.T) {
|
||||||
|
resolved, err := registry.Resolve(tt.id, tt.req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
metadata := notificationMetadataForTest(resolved)
|
||||||
|
req, err := buildNotificationRequest(cfg, resolved, tt.source, metadata)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("buildNotificationRequest() error = %v", err)
|
||||||
|
}
|
||||||
|
want := tt.want(metadata)
|
||||||
|
if strings.Join(req.BundlePaths, "\n") != strings.Join(want, "\n") {
|
||||||
|
t.Fatalf("BundlePaths = %#v, want %#v", req.BundlePaths, want)
|
||||||
|
}
|
||||||
|
if req.ReportPath != tt.source {
|
||||||
|
t.Fatalf("ReportPath = %q, want %q", req.ReportPath, tt.source)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildNotificationRequestUsesReportDistributorPathOverride(t *testing.T) {
|
||||||
|
cfg := config.Defaults()
|
||||||
|
cfg.Location.ID = "home"
|
||||||
|
cfg.WeatherAPI.Timezone = "America/Chicago"
|
||||||
|
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{report_id}"
|
||||||
|
applyReportDistributorPathOverrides(t, &cfg, `
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "custom/{report_id}/{run_id}.md"
|
||||||
|
`)
|
||||||
|
location := mustLoadTestLocation(t, cfg.WeatherAPI.Timezone)
|
||||||
|
resolved, err := report.Resolve(report.Daily, report.ResolveRequest{
|
||||||
|
Now: mustParse("2026-05-29T12:00:00-05:00"),
|
||||||
|
Location: location,
|
||||||
|
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
metadata := notificationMetadataForTest(resolved)
|
||||||
|
|
||||||
|
req, err := buildNotificationRequest(cfg, resolved, "/managed/daily.md", metadata)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("buildNotificationRequest() error = %v", err)
|
||||||
|
}
|
||||||
|
want := []string{"custom/daily/" + metadata.RunID + ".md"}
|
||||||
|
if strings.Join(req.BundlePaths, "\n") != strings.Join(want, "\n") {
|
||||||
|
t.Fatalf("BundlePaths = %#v, want override %#v", req.BundlePaths, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildNotificationRequestRequiresReportBundlePaths(t *testing.T) {
|
||||||
|
cfg := config.Defaults()
|
||||||
|
cfg.Location.ID = "home"
|
||||||
|
cfg.WeatherAPI.Timezone = "America/Chicago"
|
||||||
|
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{report_id}"
|
||||||
|
location := mustLoadTestLocation(t, cfg.WeatherAPI.Timezone)
|
||||||
|
resolved, err := report.Resolve(report.Hourly, report.ResolveRequest{
|
||||||
|
Now: mustParse("2026-05-29T12:00:00-05:00"),
|
||||||
|
Location: location,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v", err)
|
||||||
|
}
|
||||||
|
resolved.Definition.DistributorPathTemplates = nil
|
||||||
|
metadata := notificationMetadataForTest(resolved)
|
||||||
|
|
||||||
|
_, err = buildNotificationRequest(cfg, resolved, "/managed/hourly.md", metadata)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("buildNotificationRequest() error = nil, want missing path templates error")
|
||||||
|
}
|
||||||
|
for _, want := range []string{`report "hourly"`, metadata.RunID, `/managed/hourly.md`, "no distributor path templates"} {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("error = %q, want %q", err.Error(), want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestBatchRunIDUsesUTCStartAndBatchName(t *testing.T) {
|
func TestBatchRunIDUsesUTCStartAndBatchName(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -2333,10 +2588,6 @@ func TestBatchResultJSONIncludesNotification(t *testing.T) {
|
|||||||
func TestBuildBatchNotificationRequestIncludesEveningReports(t *testing.T) {
|
func TestBuildBatchNotificationRequestIncludesEveningReports(t *testing.T) {
|
||||||
server := dailyBundleServer(t)
|
server := dailyBundleServer(t)
|
||||||
cfg := dailyNotificationConfig(t, server)
|
cfg := dailyNotificationConfig(t, server)
|
||||||
cfg.Notify.Distributor.ReportPathTemplates = []string{
|
|
||||||
"archive/{valid_start_date}/{artifact_group}/{run_id}.md",
|
|
||||||
"latest/{batch_output_name}",
|
|
||||||
}
|
|
||||||
cfg.Notify.Distributor.Batch.PipelineIDTemplate = "weatherreporter.{batch}.{batch_started_date}"
|
cfg.Notify.Distributor.Batch.PipelineIDTemplate = "weatherreporter.{batch}.{batch_started_date}"
|
||||||
cfg.Notify.Distributor.Batch.BundleIDTemplate = "weatherreporter.{location_id}.{batch}"
|
cfg.Notify.Distributor.Batch.BundleIDTemplate = "weatherreporter.{location_id}.{batch}"
|
||||||
cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate = "{bundle_id}.{batch_run_id}"
|
cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate = "{bundle_id}.{batch_run_id}"
|
||||||
@@ -2368,20 +2619,26 @@ func TestBuildBatchNotificationRequestIncludesEveningReports(t *testing.T) {
|
|||||||
if len(req.IncludedReports) != 3 {
|
if len(req.IncludedReports) != 3 {
|
||||||
t.Fatalf("IncludedReports = %d, want 3", len(req.IncludedReports))
|
t.Fatalf("IncludedReports = %d, want 3", len(req.IncludedReports))
|
||||||
}
|
}
|
||||||
if len(req.Files) != 6 {
|
if len(req.Files) != 7 {
|
||||||
t.Fatalf("Files = %d, want two mappings per report", len(req.Files))
|
t.Fatalf("Files = %d, want report-specific mappings", len(req.Files))
|
||||||
}
|
}
|
||||||
|
|
||||||
wantBundlePaths := map[string]struct{}{}
|
wantBundlePaths := map[string]struct{}{}
|
||||||
for _, plannedReport := range planned {
|
for _, plannedReport := range planned {
|
||||||
resolved := plannedReport.Resolved
|
resolved := plannedReport.Resolved
|
||||||
runID := resolved.Metadata().RunID
|
runID := resolved.Metadata().RunID
|
||||||
outputName := plannedReport.OutputCopyName
|
validStart := resolved.ValidPeriod.Start.In(mustLoadTestLocation(t, cfg.WeatherAPI.Timezone)).Format(timeutil.DateLayout)
|
||||||
if outputName == "" {
|
switch resolved.Definition.ID {
|
||||||
outputName = resolved.Definition.BatchOutputName
|
case report.Tomorrow:
|
||||||
|
wantBundlePaths["daily/"+validStart+"/"+runID+".md"] = struct{}{}
|
||||||
|
wantBundlePaths["daily/"+validStart+"/index.md"] = struct{}{}
|
||||||
|
wantBundlePaths["tomorrow/index.md"] = struct{}{}
|
||||||
|
case report.Daily:
|
||||||
|
wantBundlePaths["daily/"+validStart+"/"+runID+".md"] = struct{}{}
|
||||||
|
wantBundlePaths["daily/"+validStart+"/index.md"] = struct{}{}
|
||||||
|
default:
|
||||||
|
t.Fatalf("unexpected planned report %s", resolved.Definition.ID)
|
||||||
}
|
}
|
||||||
wantBundlePaths[fmt.Sprintf("archive/%s/%s/%s.md", resolved.ValidPeriod.Start.In(mustLoadTestLocation(t, cfg.WeatherAPI.Timezone)).Format(timeutil.DateLayout), resolved.Definition.ArtifactGroup, runID)] = struct{}{}
|
|
||||||
wantBundlePaths["latest/"+outputName] = struct{}{}
|
|
||||||
}
|
}
|
||||||
gotBundlePaths := map[string]struct{}{}
|
gotBundlePaths := map[string]struct{}{}
|
||||||
gotSourcePaths := map[string]struct{}{}
|
gotSourcePaths := map[string]struct{}{}
|
||||||
@@ -2409,10 +2666,67 @@ func TestBuildBatchNotificationRequestIncludesEveningReports(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestBuildBatchNotificationRequestUsesReportOverridesAndDefaults(t *testing.T) {
|
||||||
|
server := dailyBundleServer(t)
|
||||||
|
cfg := dailyNotificationConfig(t, server)
|
||||||
|
applyReportDistributorPathOverrides(t, &cfg, `
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "custom-daily/{valid_start_date}/{run_id}.md"
|
||||||
|
`)
|
||||||
|
startedAt := mustParse("2026-05-29T18:00:00-05:00")
|
||||||
|
planned, reports := plannedBatchNotificationReports(t, cfg, BatchEvening, startedAt, "2026-05-31")
|
||||||
|
runID := batchRunID(startedAt, BatchEvening)
|
||||||
|
|
||||||
|
req, err := buildBatchNotificationRequest(cfg, BatchEvening, runID, startedAt, reports, planned)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("buildBatchNotificationRequest() error = %v", err)
|
||||||
|
}
|
||||||
|
gotBundlePaths := map[string]struct{}{}
|
||||||
|
for _, file := range req.Files {
|
||||||
|
gotBundlePaths[file.BundlePath] = struct{}{}
|
||||||
|
}
|
||||||
|
for _, plannedReport := range planned {
|
||||||
|
reportRunID := plannedReport.Resolved.Metadata().RunID
|
||||||
|
switch plannedReport.Resolved.Definition.ID {
|
||||||
|
case report.Tomorrow:
|
||||||
|
for _, want := range []string{
|
||||||
|
"daily/2026-05-30/" + reportRunID + ".md",
|
||||||
|
"daily/2026-05-30/index.md",
|
||||||
|
"tomorrow/index.md",
|
||||||
|
} {
|
||||||
|
if _, ok := gotBundlePaths[want]; !ok {
|
||||||
|
t.Fatalf("bundle paths = %#v, missing default path %q", gotBundlePaths, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case report.Daily:
|
||||||
|
want := "custom-daily/2026-05-31/" + reportRunID + ".md"
|
||||||
|
if _, ok := gotBundlePaths[want]; !ok {
|
||||||
|
t.Fatalf("bundle paths = %#v, missing override path %q", gotBundlePaths, want)
|
||||||
|
}
|
||||||
|
if _, ok := gotBundlePaths["daily/2026-05-31/index.md"]; ok {
|
||||||
|
t.Fatalf("bundle paths = %#v, want daily defaults replaced by override", gotBundlePaths)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestBuildBatchNotificationRequestRejectsDuplicateBundlePaths(t *testing.T) {
|
func TestBuildBatchNotificationRequestRejectsDuplicateBundlePaths(t *testing.T) {
|
||||||
server := dailyBundleServer(t)
|
server := dailyBundleServer(t)
|
||||||
cfg := dailyNotificationConfig(t, server)
|
cfg := dailyNotificationConfig(t, server)
|
||||||
cfg.Notify.Distributor.ReportPathTemplates = []string{"index.md"}
|
applyReportDistributorPathOverrides(t, &cfg, `
|
||||||
|
reports:
|
||||||
|
tomorrow:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "index.md"
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "index.md"
|
||||||
|
`)
|
||||||
startedAt := mustParse("2026-05-29T18:00:00-05:00")
|
startedAt := mustParse("2026-05-29T18:00:00-05:00")
|
||||||
planned, reports := plannedBatchNotificationReports(t, cfg, BatchEvening, startedAt, "2026-05-31")
|
planned, reports := plannedBatchNotificationReports(t, cfg, BatchEvening, startedAt, "2026-05-31")
|
||||||
|
|
||||||
@@ -2537,16 +2851,13 @@ func TestRunBatchMorningSendsOneBatchNotification(t *testing.T) {
|
|||||||
if req.Batch != BatchMorning || req.RunID != "20260529T100000.000000000Z_morning" {
|
if req.Batch != BatchMorning || req.RunID != "20260529T100000.000000000Z_morning" {
|
||||||
t.Fatalf("batch request identity = %s/%s, want morning run id", req.Batch, req.RunID)
|
t.Fatalf("batch request identity = %s/%s, want morning run id", req.Batch, req.RunID)
|
||||||
}
|
}
|
||||||
if len(req.IncludedReports) != 3 || len(req.Files) != 3 {
|
if len(req.IncludedReports) != 3 || len(req.Files) != 8 {
|
||||||
t.Fatalf("batch request reports/files = %d/%d, want 3/3", len(req.IncludedReports), len(req.Files))
|
t.Fatalf("batch request reports/files = %d/%d, want 3/8", len(req.IncludedReports), len(req.Files))
|
||||||
}
|
}
|
||||||
for _, file := range req.Files {
|
for _, file := range req.Files {
|
||||||
if file.SourcePath == "" || file.BundlePath == "" {
|
if file.SourcePath == "" || file.BundlePath == "" {
|
||||||
t.Fatalf("batch file = %#v, want source and bundle path", file)
|
t.Fatalf("batch file = %#v, want source and bundle path", file)
|
||||||
}
|
}
|
||||||
if !strings.Contains(file.BundlePath, file.RunID) {
|
|
||||||
t.Fatalf("bundle path %q does not include report run id %q", file.BundlePath, file.RunID)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if result.Notification == nil || result.Notification.Status != "succeeded" || result.Notification.RunID != "batch-distributor-run" || result.Notification.Path == "" {
|
if result.Notification == nil || result.Notification.Status != "succeeded" || result.Notification.RunID != "batch-distributor-run" || result.Notification.Path == "" {
|
||||||
t.Fatalf("batch notification = %#v, want succeeded result with artifact path", result.Notification)
|
t.Fatalf("batch notification = %#v, want succeeded result with artifact path", result.Notification)
|
||||||
@@ -2998,6 +3309,36 @@ func dailyNotificationConfig(t *testing.T, server *httptest.Server) config.Confi
|
|||||||
return cfg
|
return cfg
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func notificationMetadataForTest(resolved report.Resolved) state.Metadata {
|
||||||
|
metadata := resolved.Metadata()
|
||||||
|
return state.Metadata{
|
||||||
|
RunID: metadata.RunID,
|
||||||
|
ReportID: metadata.ReportID,
|
||||||
|
PromptID: metadata.PromptID,
|
||||||
|
GeneratedAt: metadata.GeneratedAt,
|
||||||
|
Timezone: metadata.Timezone,
|
||||||
|
ValidPeriod: metadata.ValidPeriod,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func applyReportDistributorPathOverrides(t *testing.T, cfg *config.Config, data string) {
|
||||||
|
t.Helper()
|
||||||
|
path := filepath.Join(t.TempDir(), "config.yml")
|
||||||
|
if err := os.WriteFile(path, []byte(data), 0o600); err != nil {
|
||||||
|
t.Fatalf("write config fixture: %v", err)
|
||||||
|
}
|
||||||
|
loaded, err := config.LoadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LoadFile() error = %v", err)
|
||||||
|
}
|
||||||
|
if cfg.Reports == nil {
|
||||||
|
cfg.Reports = map[string]config.ReportConfig{}
|
||||||
|
}
|
||||||
|
for key, reportCfg := range loaded.Reports {
|
||||||
|
cfg.Reports[key] = reportCfg
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func plannedBatchNotificationReports(t *testing.T, cfg config.Config, batch BatchKind, now time.Time, futureDailyDates ...string) ([]plannedBatchReport, []BatchReportResult) {
|
func plannedBatchNotificationReports(t *testing.T, cfg config.Config, batch BatchKind, now time.Time, futureDailyDates ...string) ([]plannedBatchReport, []BatchReportResult) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
collection := collectionWithFutureDailyForTest(t, cfg, futureDailyDates...)
|
collection := collectionWithFutureDailyForTest(t, cfg, futureDailyDates...)
|
||||||
@@ -3140,7 +3481,6 @@ func applyHourlyGeneratedTextSettings(cfg *config.Config, t *testing.T, server *
|
|||||||
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{artifact_group}"
|
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{artifact_group}"
|
||||||
cfg.Notify.Distributor.BundleIDTemplate = "weatherreporter.{location_id}.{report_id}"
|
cfg.Notify.Distributor.BundleIDTemplate = "weatherreporter.{location_id}.{report_id}"
|
||||||
cfg.Notify.Distributor.IdempotencyKeyTemplate = "weatherreporter.{location_id}.{report_id}.{run_id}"
|
cfg.Notify.Distributor.IdempotencyKeyTemplate = "weatherreporter.{location_id}.{report_id}.{run_id}"
|
||||||
cfg.Notify.Distributor.ReportPathTemplates = []string{"{valid_start_date}/{artifact_group}/{batch_output_name}"}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func resolveHourlyGeneratedTextFixture(t *testing.T, cfg config.Config) (report.Resolved, *recordingStore, *recordingNotifier, string) {
|
func resolveHourlyGeneratedTextFixture(t *testing.T, cfg config.Config) (report.Resolved, *recordingStore, *recordingNotifier, string) {
|
||||||
|
|||||||
@@ -161,9 +161,9 @@ func buildBatchNotificationRequest(cfg config.Config, batch BatchKind, runID str
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return batchNotificationRequest{}, fmt.Errorf("batch notification report %q run %q source path %q: %w", item.ReportID, item.RunID, item.ReportPath, err)
|
return batchNotificationRequest{}, fmt.Errorf("batch notification report %q run %q source path %q: %w", item.ReportID, item.RunID, item.ReportPath, err)
|
||||||
}
|
}
|
||||||
bundlePaths, err := config.RenderDistributorReportPaths(cfg.Notify.Distributor.ReportPathTemplates, values)
|
bundlePaths, err := renderDistributorReportBundlePaths(cfg, plannedReport.Resolved, item.RunID, item.ReportPath, values)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return batchNotificationRequest{}, fmt.Errorf("batch notification report %q run %q source path %q: %w", item.ReportID, item.RunID, item.ReportPath, err)
|
return batchNotificationRequest{}, err
|
||||||
}
|
}
|
||||||
|
|
||||||
included := BatchNotificationReport{
|
included := BatchNotificationReport{
|
||||||
|
|||||||
@@ -66,7 +66,6 @@ type DistributorNotifyConfig struct {
|
|||||||
PipelineIDTemplate string `yaml:"pipeline_id_template"`
|
PipelineIDTemplate string `yaml:"pipeline_id_template"`
|
||||||
BundleIDTemplate string `yaml:"bundle_id_template"`
|
BundleIDTemplate string `yaml:"bundle_id_template"`
|
||||||
IdempotencyKeyTemplate string `yaml:"idempotency_key_template"`
|
IdempotencyKeyTemplate string `yaml:"idempotency_key_template"`
|
||||||
ReportPathTemplates []string `yaml:"report_path_templates"`
|
|
||||||
Batch DistributorBatchNotifyConfig `yaml:"batch"`
|
Batch DistributorBatchNotifyConfig `yaml:"batch"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -114,9 +113,15 @@ type RecentChangeConfig struct {
|
|||||||
|
|
||||||
type ReportConfig struct {
|
type ReportConfig struct {
|
||||||
DeterministicModules []ModuleConfigItem `yaml:"deterministic_modules"`
|
DeterministicModules []ModuleConfigItem `yaml:"deterministic_modules"`
|
||||||
|
Distributor ReportDistributorConfig `yaml:"distributor"`
|
||||||
deterministicModulesSet bool
|
deterministicModulesSet bool
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type ReportDistributorConfig struct {
|
||||||
|
PathTemplates []string `yaml:"path_templates"`
|
||||||
|
pathTemplatesSet bool
|
||||||
|
}
|
||||||
|
|
||||||
type ModuleConfigItem struct {
|
type ModuleConfigItem struct {
|
||||||
ID module.ID `yaml:"id"`
|
ID module.ID `yaml:"id"`
|
||||||
Options any `yaml:"options,omitempty"`
|
Options any `yaml:"options,omitempty"`
|
||||||
@@ -135,6 +140,10 @@ func (c *ReportConfig) UnmarshalYAML(value *yaml.Node) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
c.deterministicModulesSet = true
|
c.deterministicModulesSet = true
|
||||||
|
case "distributor":
|
||||||
|
if err := node.Decode(&c.Distributor); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
default:
|
default:
|
||||||
return fmt.Errorf("unknown report entry field %q", key)
|
return fmt.Errorf("unknown report entry field %q", key)
|
||||||
}
|
}
|
||||||
@@ -142,6 +151,81 @@ func (c *ReportConfig) UnmarshalYAML(value *yaml.Node) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (c *DistributorNotifyConfig) UnmarshalYAML(value *yaml.Node) error {
|
||||||
|
if value.Kind != yaml.MappingNode {
|
||||||
|
return fmt.Errorf("notify distributor entry must be a mapping")
|
||||||
|
}
|
||||||
|
for i := 0; i < len(value.Content); i += 2 {
|
||||||
|
key := value.Content[i].Value
|
||||||
|
node := value.Content[i+1]
|
||||||
|
switch key {
|
||||||
|
case "enabled":
|
||||||
|
if err := node.Decode(&c.Enabled); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
case "endpoint":
|
||||||
|
if err := node.Decode(&c.Endpoint); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
case "token_env":
|
||||||
|
if err := node.Decode(&c.TokenEnv); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
case "timeout":
|
||||||
|
if err := node.Decode(&c.Timeout); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
case "failure_policy":
|
||||||
|
if err := node.Decode(&c.FailurePolicy); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
case "pipeline_id_template":
|
||||||
|
if err := node.Decode(&c.PipelineIDTemplate); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
case "bundle_id_template":
|
||||||
|
if err := node.Decode(&c.BundleIDTemplate); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
case "idempotency_key_template":
|
||||||
|
if err := node.Decode(&c.IdempotencyKeyTemplate); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
case "batch":
|
||||||
|
if err := node.Decode(&c.Batch); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("unknown notify distributor field %q", key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *ReportDistributorConfig) UnmarshalYAML(value *yaml.Node) error {
|
||||||
|
if value.Kind != yaml.MappingNode {
|
||||||
|
return fmt.Errorf("report distributor entry must be a mapping")
|
||||||
|
}
|
||||||
|
for i := 0; i < len(value.Content); i += 2 {
|
||||||
|
key := value.Content[i].Value
|
||||||
|
node := value.Content[i+1]
|
||||||
|
switch key {
|
||||||
|
case "path_templates":
|
||||||
|
if err := node.Decode(&c.PathTemplates); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
c.pathTemplatesSet = true
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("unknown report distributor field %q", key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c ReportDistributorConfig) PathTemplatesSet() bool {
|
||||||
|
return c.pathTemplatesSet
|
||||||
|
}
|
||||||
|
|
||||||
func (m *ModuleConfigItem) UnmarshalYAML(value *yaml.Node) error {
|
func (m *ModuleConfigItem) UnmarshalYAML(value *yaml.Node) error {
|
||||||
switch value.Kind {
|
switch value.Kind {
|
||||||
case yaml.ScalarNode:
|
case yaml.ScalarNode:
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ import (
|
|||||||
|
|
||||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/module"
|
"gitea.maximumdirect.net/eric/weatherreporter/internal/module"
|
||||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
|
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
|
||||||
|
"gopkg.in/yaml.v3"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestDefaults(t *testing.T) {
|
func TestDefaults(t *testing.T) {
|
||||||
@@ -57,12 +58,6 @@ func TestDefaults(t *testing.T) {
|
|||||||
if cfg.Notify.Distributor.IdempotencyKeyTemplate != "{bundle_id}.{run_id}" {
|
if cfg.Notify.Distributor.IdempotencyKeyTemplate != "{bundle_id}.{run_id}" {
|
||||||
t.Fatalf("Notify.Distributor.IdempotencyKeyTemplate = %q, want default", cfg.Notify.Distributor.IdempotencyKeyTemplate)
|
t.Fatalf("Notify.Distributor.IdempotencyKeyTemplate = %q, want default", cfg.Notify.Distributor.IdempotencyKeyTemplate)
|
||||||
}
|
}
|
||||||
wantReportPaths := []string{
|
|
||||||
"{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md",
|
|
||||||
}
|
|
||||||
if strings.Join(cfg.Notify.Distributor.ReportPathTemplates, "\n") != strings.Join(wantReportPaths, "\n") {
|
|
||||||
t.Fatalf("Notify.Distributor.ReportPathTemplates = %#v, want %#v", cfg.Notify.Distributor.ReportPathTemplates, wantReportPaths)
|
|
||||||
}
|
|
||||||
if !cfg.Notify.Distributor.Batch.Enabled {
|
if !cfg.Notify.Distributor.Batch.Enabled {
|
||||||
t.Fatalf("Notify.Distributor.Batch.Enabled = false, want true")
|
t.Fatalf("Notify.Distributor.Batch.Enabled = false, want true")
|
||||||
}
|
}
|
||||||
@@ -101,9 +96,6 @@ func TestLoadExampleConfig(t *testing.T) {
|
|||||||
if cfg.Notify.Distributor.PipelineIDTemplate != "weatherreporter.{report_id}" {
|
if cfg.Notify.Distributor.PipelineIDTemplate != "weatherreporter.{report_id}" {
|
||||||
t.Fatalf("PipelineIDTemplate = %q, want example pipeline template", cfg.Notify.Distributor.PipelineIDTemplate)
|
t.Fatalf("PipelineIDTemplate = %q, want example pipeline template", cfg.Notify.Distributor.PipelineIDTemplate)
|
||||||
}
|
}
|
||||||
if len(cfg.Notify.Distributor.ReportPathTemplates) != 1 {
|
|
||||||
t.Fatalf("ReportPathTemplates = %#v, want example archive path", cfg.Notify.Distributor.ReportPathTemplates)
|
|
||||||
}
|
|
||||||
if !cfg.Notify.Distributor.Batch.Enabled {
|
if !cfg.Notify.Distributor.Batch.Enabled {
|
||||||
t.Fatalf("Notify.Distributor.Batch.Enabled = false, want true")
|
t.Fatalf("Notify.Distributor.Batch.Enabled = false, want true")
|
||||||
}
|
}
|
||||||
@@ -313,6 +305,102 @@ reports:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestLoadReportDistributorPathOverrides(t *testing.T) {
|
||||||
|
path := writeConfig(t, `
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "daily/{valid_start_date}/{run_id}.md"
|
||||||
|
- "daily/{valid_start_date}/index.md"
|
||||||
|
today:
|
||||||
|
deterministic_modules:
|
||||||
|
- metadata
|
||||||
|
`)
|
||||||
|
|
||||||
|
cfg, err := LoadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LoadFile() error = %v", err)
|
||||||
|
}
|
||||||
|
daily := cfg.Reports["daily"].Distributor
|
||||||
|
if !daily.PathTemplatesSet() {
|
||||||
|
t.Fatal("daily distributor path_templates set = false, want true")
|
||||||
|
}
|
||||||
|
want := []string{
|
||||||
|
"daily/{valid_start_date}/{run_id}.md",
|
||||||
|
"daily/{valid_start_date}/index.md",
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(daily.PathTemplates, want) {
|
||||||
|
t.Fatalf("daily path templates = %#v, want %#v", daily.PathTemplates, want)
|
||||||
|
}
|
||||||
|
if cfg.Reports["today"].Distributor.PathTemplatesSet() {
|
||||||
|
t.Fatal("today distributor path_templates set = true, want false")
|
||||||
|
}
|
||||||
|
|
||||||
|
overrides, err := cfg.ReportDistributorPathOverrides()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReportDistributorPathOverrides() error = %v", err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(overrides[report.Daily], want) {
|
||||||
|
t.Fatalf("daily distributor override = %#v, want %#v", overrides[report.Daily], want)
|
||||||
|
}
|
||||||
|
if _, ok := overrides[report.Today]; ok {
|
||||||
|
t.Fatalf("today distributor override = %#v, want omitted override absent", overrides[report.Today])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReportDistributorPathTemplatesSetTracksExplicitEmptyList(t *testing.T) {
|
||||||
|
var cfg Config
|
||||||
|
if err := yaml.Unmarshal([]byte(`
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates: []
|
||||||
|
today:
|
||||||
|
distributor: {}
|
||||||
|
`), &cfg); err != nil {
|
||||||
|
t.Fatalf("yaml.Unmarshal() error = %v", err)
|
||||||
|
}
|
||||||
|
if !cfg.Reports["daily"].Distributor.PathTemplatesSet() {
|
||||||
|
t.Fatal("daily distributor path_templates set = false, want true")
|
||||||
|
}
|
||||||
|
if len(cfg.Reports["daily"].Distributor.PathTemplates) != 0 {
|
||||||
|
t.Fatalf("daily path templates = %#v, want empty explicit list", cfg.Reports["daily"].Distributor.PathTemplates)
|
||||||
|
}
|
||||||
|
if cfg.Reports["today"].Distributor.PathTemplatesSet() {
|
||||||
|
t.Fatal("today distributor path_templates set = true, want false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadReportDistributorPathOverrideAliases(t *testing.T) {
|
||||||
|
path := writeConfig(t, `
|
||||||
|
reports:
|
||||||
|
three-day-outlook:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "three-day/{valid_start_date}/index.md"
|
||||||
|
weekend_outlook:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "weekend/{valid_start_date}/index.md"
|
||||||
|
`)
|
||||||
|
|
||||||
|
cfg, err := LoadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LoadFile() error = %v", err)
|
||||||
|
}
|
||||||
|
overrides, err := cfg.ReportDistributorPathOverrides()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReportDistributorPathOverrides() error = %v", err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(overrides[report.ThreeDay], []string{"three-day/{valid_start_date}/index.md"}) {
|
||||||
|
t.Fatalf("three-day distributor override = %#v, want alias override", overrides[report.ThreeDay])
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(overrides[report.Weekend], []string{"weekend/{valid_start_date}/index.md"}) {
|
||||||
|
t.Fatalf("weekend distributor override = %#v, want alias override", overrides[report.Weekend])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestValidateReportModuleKeysWithoutMutatingOptions(t *testing.T) {
|
func TestValidateReportModuleKeysWithoutMutatingOptions(t *testing.T) {
|
||||||
cfg := Defaults()
|
cfg := Defaults()
|
||||||
rawOptions := map[string]any{
|
rawOptions := map[string]any{
|
||||||
@@ -565,6 +653,168 @@ reports:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestReportDistributorPathOverrideValidation(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
yaml string
|
||||||
|
wantErr string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "UnknownReportField",
|
||||||
|
yaml: `
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor_paths:
|
||||||
|
- latest.md
|
||||||
|
`,
|
||||||
|
wantErr: `unknown report entry field "distributor_paths"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "UnknownDistributorField",
|
||||||
|
yaml: `
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
paths:
|
||||||
|
- latest.md
|
||||||
|
`,
|
||||||
|
wantErr: `unknown report distributor field "paths"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "DuplicateReportAlias",
|
||||||
|
yaml: `
|
||||||
|
reports:
|
||||||
|
three-day:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "three-day/{valid_start_date}/index.md"
|
||||||
|
three_day:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "three-day/latest.md"
|
||||||
|
`,
|
||||||
|
wantErr: "duplicates report override",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "UnknownTemplateVariable",
|
||||||
|
yaml: `
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "{unknown}.md"
|
||||||
|
`,
|
||||||
|
wantErr: `reports.daily.distributor.path_templates[0] contains unknown template variable "unknown"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "AbsolutePath",
|
||||||
|
yaml: `
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "/daily.md"
|
||||||
|
`,
|
||||||
|
wantErr: "reports.daily.distributor.path_templates[0] must render a relative path",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "ParentSegment",
|
||||||
|
yaml: `
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "daily/../index.md"
|
||||||
|
`,
|
||||||
|
wantErr: "reports.daily.distributor.path_templates[0] must not render . or .. path segments",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "Manifest",
|
||||||
|
yaml: `
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "daily/manifest.json"
|
||||||
|
`,
|
||||||
|
wantErr: `reports.daily.distributor.path_templates[0] must not render reserved path segment "manifest.json"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "DuplicateRenderedPath",
|
||||||
|
yaml: `
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "daily/index.md"
|
||||||
|
- "daily/index.md"
|
||||||
|
`,
|
||||||
|
wantErr: `reports.daily.distributor.path_templates renders duplicate path "daily/index.md"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "EmptyOverrideList",
|
||||||
|
yaml: `
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates: []
|
||||||
|
`,
|
||||||
|
wantErr: "reports.daily.distributor.path_templates must contain at least one entry",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
_, err := LoadFile(writeConfig(t, tt.yaml))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("LoadFile() error = nil, want validation error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||||
|
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReportDistributorPathOverridesConsistentForLoadedAndConstructedConfig(t *testing.T) {
|
||||||
|
yaml := `
|
||||||
|
reports:
|
||||||
|
daily:
|
||||||
|
distributor:
|
||||||
|
path_templates:
|
||||||
|
- "daily/{valid_start_date}/index.md"
|
||||||
|
`
|
||||||
|
reports := map[string]ReportConfig{
|
||||||
|
"daily": {
|
||||||
|
Distributor: ReportDistributorConfig{
|
||||||
|
PathTemplates: []string{"daily/{valid_start_date}/index.md"},
|
||||||
|
pathTemplatesSet: true,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg, err := LoadFile(writeConfig(t, yaml))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LoadFile() error = %v", err)
|
||||||
|
}
|
||||||
|
loaded, err := cfg.ReportDistributorPathOverrides()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("loaded ReportDistributorPathOverrides() error = %v", err)
|
||||||
|
}
|
||||||
|
cfg = Defaults()
|
||||||
|
cfg.Reports = reports
|
||||||
|
if err := Validate(cfg); err != nil {
|
||||||
|
t.Fatalf("Validate() error = %v", err)
|
||||||
|
}
|
||||||
|
constructed, err := cfg.ReportDistributorPathOverrides()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("constructed ReportDistributorPathOverrides() error = %v", err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(loaded, constructed) {
|
||||||
|
t.Fatalf("loaded overrides = %#v, constructed = %#v", loaded, constructed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestReportModuleValidationConsistentForLoadedAndConstructedConfig(t *testing.T) {
|
func TestReportModuleValidationConsistentForLoadedAndConstructedConfig(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -806,6 +1056,37 @@ func TestDisabledDistributorNotifyAcceptsOmittedFields(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestDistributorNotifyRejectsRemovedGlobalReportPaths(t *testing.T) {
|
||||||
|
removedField := "report_path" + "_templates"
|
||||||
|
_, err := LoadFile(writeConfig(t, `
|
||||||
|
notify:
|
||||||
|
distributor:
|
||||||
|
`+removedField+`:
|
||||||
|
- index.md
|
||||||
|
`))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("LoadFile() error = nil, want removed global path field error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), `unknown notify distributor field "`+removedField+`"`) {
|
||||||
|
t.Fatalf("error = %q, want removed global path field rejection", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDistributorNotifyRejectsUnknownFields(t *testing.T) {
|
||||||
|
_, err := LoadFile(writeConfig(t, `
|
||||||
|
notify:
|
||||||
|
distributor:
|
||||||
|
paths:
|
||||||
|
- index.md
|
||||||
|
`))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("LoadFile() error = nil, want unknown distributor field error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), `unknown notify distributor field "paths"`) {
|
||||||
|
t.Fatalf("error = %q, want unknown field rejection", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestDisabledDistributorNotifyAcceptsMalformedBatchTemplates(t *testing.T) {
|
func TestDisabledDistributorNotifyAcceptsMalformedBatchTemplates(t *testing.T) {
|
||||||
cfg := Defaults()
|
cfg := Defaults()
|
||||||
cfg.Notify.Distributor.Enabled = false
|
cfg.Notify.Distributor.Enabled = false
|
||||||
@@ -895,32 +1176,11 @@ func TestEnabledDistributorNotifyValidation(t *testing.T) {
|
|||||||
wantErr: "notify.distributor.idempotency_key_template",
|
wantErr: "notify.distributor.idempotency_key_template",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "ReportPathTemplatesEmpty",
|
name: "BatchTemplate",
|
||||||
mutate: func(cfg *Config) {
|
mutate: func(cfg *Config) {
|
||||||
cfg.Notify.Distributor.ReportPathTemplates = nil
|
cfg.Notify.Distributor.Batch.BundleIDTemplate = "{run_id}"
|
||||||
},
|
},
|
||||||
wantErr: "notify.distributor.report_path_templates",
|
wantErr: "notify.distributor.batch.bundle_id_template",
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "ReportPathTemplateUnknown",
|
|
||||||
mutate: func(cfg *Config) {
|
|
||||||
cfg.Notify.Distributor.ReportPathTemplates = []string{"{unknown}"}
|
|
||||||
},
|
|
||||||
wantErr: "notify.distributor.report_path_templates",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "ReportPathTemplateInvalidPath",
|
|
||||||
mutate: func(cfg *Config) {
|
|
||||||
cfg.Notify.Distributor.ReportPathTemplates = []string{"/{batch_output_name}"}
|
|
||||||
},
|
|
||||||
wantErr: "notify.distributor.report_path_templates",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "ReportPathTemplateDuplicatePath",
|
|
||||||
mutate: func(cfg *Config) {
|
|
||||||
cfg.Notify.Distributor.ReportPathTemplates = []string{"latest.md", "latest.md"}
|
|
||||||
},
|
|
||||||
wantErr: "notify.distributor.report_path_templates",
|
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1058,35 +1318,38 @@ func TestDistributorTemplateRendering(t *testing.T) {
|
|||||||
ValidEndTime: "0600",
|
ValidEndTime: "0600",
|
||||||
ValidStartStamp: "2026-06-07T1800",
|
ValidStartStamp: "2026-06-07T1800",
|
||||||
ValidEndStamp: "2026-06-08T0600",
|
ValidEndStamp: "2026-06-08T0600",
|
||||||
|
StormID: "2026-06-07T1800-2026-06-08T0600",
|
||||||
BundleID: "weatherreporter.home.daily",
|
BundleID: "weatherreporter.home.daily",
|
||||||
}
|
}
|
||||||
|
|
||||||
bundleID, err := RenderDistributorBundleID("weatherreporter.{location_id}.{report_id}", values)
|
bundleID, err := RenderDistributorBundleID("weatherreporter.{location_id}.{report_id}.{storm_id}", values)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("RenderDistributorBundleID() error = %v", err)
|
t.Fatalf("RenderDistributorBundleID() error = %v", err)
|
||||||
}
|
}
|
||||||
if bundleID != "weatherreporter.home.daily" {
|
if bundleID != "weatherreporter.home.daily.2026-06-07T1800-2026-06-08T0600" {
|
||||||
t.Fatalf("bundleID = %q, want rendered value", bundleID)
|
t.Fatalf("bundleID = %q, want rendered value", bundleID)
|
||||||
}
|
}
|
||||||
|
values.BundleID = bundleID
|
||||||
|
|
||||||
pipelineID, err := RenderDistributorPipelineID("weatherreporter.{artifact_group}.{bundle_id}", values)
|
pipelineID, err := RenderDistributorPipelineID("weatherreporter.{artifact_group}.{storm_id}.{bundle_id}", values)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("RenderDistributorPipelineID() error = %v", err)
|
t.Fatalf("RenderDistributorPipelineID() error = %v", err)
|
||||||
}
|
}
|
||||||
if pipelineID != "weatherreporter.daily.weatherreporter.home.daily" {
|
if pipelineID != "weatherreporter.daily.2026-06-07T1800-2026-06-08T0600.weatherreporter.home.daily.2026-06-07T1800-2026-06-08T0600" {
|
||||||
t.Fatalf("pipelineID = %q, want rendered pipeline ID", pipelineID)
|
t.Fatalf("pipelineID = %q, want rendered pipeline ID", pipelineID)
|
||||||
}
|
}
|
||||||
|
|
||||||
idempotencyKey, err := RenderDistributorIdempotencyKey("{bundle_id}.{run_id}", values)
|
idempotencyKey, err := RenderDistributorIdempotencyKey("{bundle_id}.{storm_id}.{run_id}", values)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("RenderDistributorIdempotencyKey() error = %v", err)
|
t.Fatalf("RenderDistributorIdempotencyKey() error = %v", err)
|
||||||
}
|
}
|
||||||
if idempotencyKey != "weatherreporter.home.daily.20260607T120000Z" {
|
if idempotencyKey != "weatherreporter.home.daily.2026-06-07T1800-2026-06-08T0600.2026-06-07T1800-2026-06-08T0600.20260607T120000Z" {
|
||||||
t.Fatalf("idempotencyKey = %q, want rendered run key", idempotencyKey)
|
t.Fatalf("idempotencyKey = %q, want rendered run key", idempotencyKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
reportPaths, err := RenderDistributorReportPaths([]string{
|
reportPaths, err := RenderDistributorReportPaths("reports.daily.distributor.path_templates", []string{
|
||||||
"{valid_start_date}/{artifact_group}/{valid_start_stamp}-{valid_end_stamp}-{run_id}.md",
|
"{valid_start_date}/{artifact_group}/{valid_start_stamp}-{valid_end_stamp}-{run_id}.md",
|
||||||
|
"storm/{storm_id}/index.md",
|
||||||
"{valid_start_date}/{artifact_group}/latest.md",
|
"{valid_start_date}/{artifact_group}/latest.md",
|
||||||
}, values)
|
}, values)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1094,6 +1357,7 @@ func TestDistributorTemplateRendering(t *testing.T) {
|
|||||||
}
|
}
|
||||||
wantPaths := []string{
|
wantPaths := []string{
|
||||||
"2026-06-07/daily/2026-06-07T1800-2026-06-08T0600-20260607T120000Z.md",
|
"2026-06-07/daily/2026-06-07T1800-2026-06-08T0600-20260607T120000Z.md",
|
||||||
|
"storm/2026-06-07T1800-2026-06-08T0600/index.md",
|
||||||
"2026-06-07/daily/latest.md",
|
"2026-06-07/daily/latest.md",
|
||||||
}
|
}
|
||||||
if strings.Join(reportPaths, "\n") != strings.Join(wantPaths, "\n") {
|
if strings.Join(reportPaths, "\n") != strings.Join(wantPaths, "\n") {
|
||||||
@@ -1101,6 +1365,50 @@ func TestDistributorTemplateRendering(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestDistributorReportPathRenderingUsesCallerName(t *testing.T) {
|
||||||
|
values := DistributorTemplateValues{
|
||||||
|
BatchOutputName: "report.md",
|
||||||
|
}
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
templates []string
|
||||||
|
wantErr string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "UnknownVariable",
|
||||||
|
templates: []string{"{unknown}.md"},
|
||||||
|
wantErr: `report.daily.distributor_path_templates[0] contains unknown template variable "unknown"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "InvalidPath",
|
||||||
|
templates: []string{"/{batch_output_name}"},
|
||||||
|
wantErr: "report.daily.distributor_path_templates[0] must render a relative path",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "DuplicatePath",
|
||||||
|
templates: []string{"latest.md", "latest.md"},
|
||||||
|
wantErr: `report.daily.distributor_path_templates renders duplicate path "latest.md"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "Empty",
|
||||||
|
templates: nil,
|
||||||
|
wantErr: "report.daily.distributor_path_templates must contain at least one entry",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
_, err := RenderDistributorReportPaths("report.daily.distributor_path_templates", tt.templates, values)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("RenderDistributorReportPaths() error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||||
|
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestDistributorBatchTemplateRendering(t *testing.T) {
|
func TestDistributorBatchTemplateRendering(t *testing.T) {
|
||||||
values := DistributorBatchTemplateValues{
|
values := DistributorBatchTemplateValues{
|
||||||
LocationID: "home",
|
LocationID: "home",
|
||||||
@@ -1233,12 +1541,15 @@ func TestDistributorReportPathRenderingRejectsInvalidValues(t *testing.T) {
|
|||||||
|
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
_, err := RenderDistributorReportPaths([]string{"{batch_output_name}"}, DistributorTemplateValues{
|
_, err := RenderDistributorReportPaths("report.daily.distributor_path_templates", []string{"{batch_output_name}"}, DistributorTemplateValues{
|
||||||
BatchOutputName: tt.batchOutputName,
|
BatchOutputName: tt.batchOutputName,
|
||||||
})
|
})
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("RenderDistributorReportPaths() error = nil, want error")
|
t.Fatal("RenderDistributorReportPaths() error = nil, want error")
|
||||||
}
|
}
|
||||||
|
if !strings.Contains(err.Error(), "report.daily.distributor_path_templates[0]") {
|
||||||
|
t.Fatalf("error = %q, want caller path name", err.Error())
|
||||||
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -31,9 +31,6 @@ func Defaults() Config {
|
|||||||
PipelineIDTemplate: "",
|
PipelineIDTemplate: "",
|
||||||
BundleIDTemplate: "weatherreporter.{location_id}.{report_id}",
|
BundleIDTemplate: "weatherreporter.{location_id}.{report_id}",
|
||||||
IdempotencyKeyTemplate: "{bundle_id}.{run_id}",
|
IdempotencyKeyTemplate: "{bundle_id}.{run_id}",
|
||||||
ReportPathTemplates: []string{
|
|
||||||
"{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md",
|
|
||||||
},
|
|
||||||
Batch: DistributorBatchNotifyConfig{
|
Batch: DistributorBatchNotifyConfig{
|
||||||
Enabled: true,
|
Enabled: true,
|
||||||
PipelineIDTemplate: "weatherreporter",
|
PipelineIDTemplate: "weatherreporter",
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ type DistributorTemplateValues struct {
|
|||||||
ValidEndTime string
|
ValidEndTime string
|
||||||
ValidStartStamp string
|
ValidStartStamp string
|
||||||
ValidEndStamp string
|
ValidEndStamp string
|
||||||
|
StormID string
|
||||||
BundleID string
|
BundleID string
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -41,6 +42,7 @@ var distributorTemplateVariables = map[string]struct{}{
|
|||||||
"valid_end_time": {},
|
"valid_end_time": {},
|
||||||
"valid_start_stamp": {},
|
"valid_start_stamp": {},
|
||||||
"valid_end_stamp": {},
|
"valid_end_stamp": {},
|
||||||
|
"storm_id": {},
|
||||||
}
|
}
|
||||||
|
|
||||||
var distributorIdempotencyTemplateVariables = map[string]struct{}{
|
var distributorIdempotencyTemplateVariables = map[string]struct{}{
|
||||||
@@ -55,6 +57,7 @@ var distributorIdempotencyTemplateVariables = map[string]struct{}{
|
|||||||
"valid_end_time": {},
|
"valid_end_time": {},
|
||||||
"valid_start_stamp": {},
|
"valid_start_stamp": {},
|
||||||
"valid_end_stamp": {},
|
"valid_end_stamp": {},
|
||||||
|
"storm_id": {},
|
||||||
"bundle_id": {},
|
"bundle_id": {},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -129,23 +132,23 @@ func RenderDistributorBatchIdempotencyKey(template string, values DistributorBat
|
|||||||
return rendered, nil
|
return rendered, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func RenderDistributorReportPaths(templates []string, values DistributorTemplateValues) ([]string, error) {
|
func RenderDistributorReportPaths(name string, templates []string, values DistributorTemplateValues) ([]string, error) {
|
||||||
if len(templates) == 0 {
|
if len(templates) == 0 {
|
||||||
return nil, fmt.Errorf("notify.distributor.report_path_templates must contain at least one entry")
|
return nil, fmt.Errorf("%s must contain at least one entry", name)
|
||||||
}
|
}
|
||||||
paths := make([]string, 0, len(templates))
|
paths := make([]string, 0, len(templates))
|
||||||
seen := make(map[string]struct{}, len(templates))
|
seen := make(map[string]struct{}, len(templates))
|
||||||
for i, template := range templates {
|
for i, template := range templates {
|
||||||
name := fmt.Sprintf("notify.distributor.report_path_templates[%d]", i)
|
itemName := fmt.Sprintf("%s[%d]", name, i)
|
||||||
rendered, err := renderDistributorTemplate(name, template, values, distributorTemplateVariables)
|
rendered, err := renderDistributorTemplate(itemName, template, values, distributorTemplateVariables)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
if err := ValidateDistributorReportPath(name, rendered); err != nil {
|
if err := ValidateDistributorReportPath(itemName, rendered); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
if _, ok := seen[rendered]; ok {
|
if _, ok := seen[rendered]; ok {
|
||||||
return nil, fmt.Errorf("notify.distributor.report_path_templates renders duplicate path %q", rendered)
|
return nil, fmt.Errorf("%s renders duplicate path %q", name, rendered)
|
||||||
}
|
}
|
||||||
seen[rendered] = struct{}{}
|
seen[rendered] = struct{}{}
|
||||||
paths = append(paths, rendered)
|
paths = append(paths, rendered)
|
||||||
@@ -243,6 +246,8 @@ func distributorTemplateValue(variable string, values DistributorTemplateValues)
|
|||||||
return values.ValidStartStamp
|
return values.ValidStartStamp
|
||||||
case "valid_end_stamp":
|
case "valid_end_stamp":
|
||||||
return values.ValidEndStamp
|
return values.ValidEndStamp
|
||||||
|
case "storm_id":
|
||||||
|
return values.StormID
|
||||||
case "bundle_id":
|
case "bundle_id":
|
||||||
return values.BundleID
|
return values.BundleID
|
||||||
default:
|
default:
|
||||||
|
|||||||
@@ -17,6 +17,10 @@ func (cfg Config) ReportModuleOverrides() (map[report.ID][]module.ConfigItem, er
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (cfg Config) ReportDistributorPathOverrides() (map[report.ID][]string, error) {
|
||||||
|
return traverseReportDistributorPathOverrides(cfg)
|
||||||
|
}
|
||||||
|
|
||||||
func normalizeReportModules(cfg *Config) error {
|
func normalizeReportModules(cfg *Config) error {
|
||||||
if cfg.Reports == nil {
|
if cfg.Reports == nil {
|
||||||
cfg.Reports = map[string]ReportConfig{}
|
cfg.Reports = map[string]ReportConfig{}
|
||||||
@@ -35,6 +39,11 @@ func validateReportModules(cfg Config) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateReportDistributorPathOverrides(cfg Config) error {
|
||||||
|
_, err := traverseReportDistributorPathOverrides(cfg)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
type reportModuleTraversalOptions struct {
|
type reportModuleTraversalOptions struct {
|
||||||
normalizeOptions bool
|
normalizeOptions bool
|
||||||
updateConfig bool
|
updateConfig bool
|
||||||
@@ -82,6 +91,59 @@ func traverseReportModules(cfg *Config, opts reportModuleTraversalOptions) (map[
|
|||||||
return overrides, nil
|
return overrides, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func traverseReportDistributorPathOverrides(cfg Config) (map[report.ID][]string, error) {
|
||||||
|
overrides := map[report.ID][]string{}
|
||||||
|
if cfg.Reports == nil {
|
||||||
|
return overrides, nil
|
||||||
|
}
|
||||||
|
reportRegistry := report.DefaultRegistry()
|
||||||
|
seenReports := map[report.ID]string{}
|
||||||
|
for key, reportCfg := range cfg.Reports {
|
||||||
|
reportID, err := report.IDForConfigKey(key)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("reports.%s: %w", key, err)
|
||||||
|
}
|
||||||
|
if previous, ok := seenReports[reportID]; ok {
|
||||||
|
return nil, fmt.Errorf("reports.%s duplicates report override %q", key, previous)
|
||||||
|
}
|
||||||
|
seenReports[reportID] = key
|
||||||
|
if _, err := reportRegistry.Lookup(reportID); err != nil {
|
||||||
|
return nil, fmt.Errorf("reports.%s: %w", key, err)
|
||||||
|
}
|
||||||
|
if !reportCfg.Distributor.pathTemplatesSet {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := validateReportDistributorPathTemplates(key, reportCfg.Distributor.PathTemplates); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
overrides[reportID] = append([]string(nil), reportCfg.Distributor.PathTemplates...)
|
||||||
|
}
|
||||||
|
return overrides, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateReportDistributorPathTemplates(reportKey string, templates []string) error {
|
||||||
|
name := fmt.Sprintf("reports.%s.distributor.path_templates", reportKey)
|
||||||
|
_, err := RenderDistributorReportPaths(name, templates, sampleDistributorTemplateValues())
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func sampleDistributorTemplateValues() DistributorTemplateValues {
|
||||||
|
return DistributorTemplateValues{
|
||||||
|
LocationID: "location",
|
||||||
|
ReportID: "report",
|
||||||
|
RunID: "run",
|
||||||
|
ArtifactGroup: "artifact",
|
||||||
|
BatchOutputName: "report.md",
|
||||||
|
ValidStartDate: "2026-05-29",
|
||||||
|
ValidEndDate: "2026-05-30",
|
||||||
|
ValidStartTime: "0000",
|
||||||
|
ValidEndTime: "0000",
|
||||||
|
ValidStartStamp: "2026-05-29T0000",
|
||||||
|
ValidEndStamp: "2026-05-30T0000",
|
||||||
|
StormID: "2026-05-29T0000-2026-05-30T0000",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func moduleItemsFromConfig(registry briefing.ModuleRegistry, reportKey string, items []ModuleConfigItem, normalizeOptions bool) ([]module.ConfigItem, []ModuleConfigItem, error) {
|
func moduleItemsFromConfig(registry briefing.ModuleRegistry, reportKey string, items []ModuleConfigItem, normalizeOptions bool) ([]module.ConfigItem, []ModuleConfigItem, error) {
|
||||||
out := make([]module.ConfigItem, 0, len(items))
|
out := make([]module.ConfigItem, 0, len(items))
|
||||||
normalizedItems := append([]ModuleConfigItem(nil), items...)
|
normalizedItems := append([]ModuleConfigItem(nil), items...)
|
||||||
|
|||||||
@@ -12,6 +12,9 @@ func Validate(cfg Config) error {
|
|||||||
if err := validateReportModules(cfg); err != nil {
|
if err := validateReportModules(cfg); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
if err := validateReportDistributorPathOverrides(cfg); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
if cfg.WeatherAPI.BaseURL != "" {
|
if cfg.WeatherAPI.BaseURL != "" {
|
||||||
parsed, err := url.Parse(cfg.WeatherAPI.BaseURL)
|
parsed, err := url.Parse(cfg.WeatherAPI.BaseURL)
|
||||||
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
|
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
|
||||||
@@ -121,22 +124,7 @@ func validateDistributorNotify(cfg DistributorNotifyConfig) error {
|
|||||||
if err := validateDistributorTemplate("notify.distributor.idempotency_key_template", cfg.IdempotencyKeyTemplate, distributorIdempotencyTemplateVariables); err != nil {
|
if err := validateDistributorTemplate("notify.distributor.idempotency_key_template", cfg.IdempotencyKeyTemplate, distributorIdempotencyTemplateVariables); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if len(cfg.ReportPathTemplates) == 0 {
|
values := sampleDistributorTemplateValues()
|
||||||
return fmt.Errorf("notify.distributor.report_path_templates must contain at least one entry when enabled")
|
|
||||||
}
|
|
||||||
values := DistributorTemplateValues{
|
|
||||||
LocationID: "location",
|
|
||||||
ReportID: "report",
|
|
||||||
RunID: "run",
|
|
||||||
ArtifactGroup: "artifact",
|
|
||||||
BatchOutputName: "report.md",
|
|
||||||
ValidStartDate: "2026-05-29",
|
|
||||||
ValidEndDate: "2026-05-30",
|
|
||||||
ValidStartTime: "0000",
|
|
||||||
ValidEndTime: "0000",
|
|
||||||
ValidStartStamp: "2026-05-29T0000",
|
|
||||||
ValidEndStamp: "2026-05-30T0000",
|
|
||||||
}
|
|
||||||
bundleID, err := RenderDistributorBundleID(cfg.BundleIDTemplate, values)
|
bundleID, err := RenderDistributorBundleID(cfg.BundleIDTemplate, values)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -145,9 +133,6 @@ func validateDistributorNotify(cfg DistributorNotifyConfig) error {
|
|||||||
if _, err := RenderDistributorPipelineID(cfg.PipelineIDTemplate, values); err != nil {
|
if _, err := RenderDistributorPipelineID(cfg.PipelineIDTemplate, values); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if _, err := RenderDistributorReportPaths(cfg.ReportPathTemplates, values); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if err := validateDistributorBatchNotify(cfg.Batch); err != nil {
|
if err := validateDistributorBatchNotify(cfg.Batch); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -18,6 +18,10 @@ func dailyDefinition() Definition {
|
|||||||
ComparisonStrategy: CompareSameValidDate,
|
ComparisonStrategy: CompareSameValidDate,
|
||||||
ArtifactGroup: "daily",
|
ArtifactGroup: "daily",
|
||||||
BatchOutputName: "daily.md",
|
BatchOutputName: "daily.md",
|
||||||
|
DistributorPathTemplates: []string{
|
||||||
|
"daily/{valid_start_date}/{run_id}.md",
|
||||||
|
"daily/{valid_start_date}/index.md",
|
||||||
|
},
|
||||||
Generated: true,
|
Generated: true,
|
||||||
CompatiblePriorIDs: []ID{Daily},
|
CompatiblePriorIDs: []ID{Daily},
|
||||||
Modules: dailyModules(),
|
Modules: dailyModules(),
|
||||||
|
|||||||
@@ -55,6 +55,7 @@ type Definition struct {
|
|||||||
ComparisonStrategy ComparisonStrategy
|
ComparisonStrategy ComparisonStrategy
|
||||||
ArtifactGroup string
|
ArtifactGroup string
|
||||||
BatchOutputName string
|
BatchOutputName string
|
||||||
|
DistributorPathTemplates []string
|
||||||
Generated bool
|
Generated bool
|
||||||
CompatiblePriorIDs []ID
|
CompatiblePriorIDs []ID
|
||||||
Modules []module.ConfigItem
|
Modules []module.ConfigItem
|
||||||
|
|||||||
@@ -20,6 +20,9 @@ func hourlyDefinition() Definition {
|
|||||||
ComparisonStrategy: CompareRollingWindow,
|
ComparisonStrategy: CompareRollingWindow,
|
||||||
ArtifactGroup: "hourly",
|
ArtifactGroup: "hourly",
|
||||||
BatchOutputName: "hourly.md",
|
BatchOutputName: "hourly.md",
|
||||||
|
DistributorPathTemplates: []string{
|
||||||
|
"hourly/index.md",
|
||||||
|
},
|
||||||
Generated: true,
|
Generated: true,
|
||||||
CompatiblePriorIDs: []ID{Hourly},
|
CompatiblePriorIDs: []ID{Hourly},
|
||||||
Modules: hourlyModules(),
|
Modules: hourlyModules(),
|
||||||
|
|||||||
@@ -520,6 +520,88 @@ func TestRegistryDefinitionsDeclarePathAndCompatibilityPolicy(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestGeneratedRegistryDefinitionsDeclareDistributorPathDefaults(t *testing.T) {
|
||||||
|
for _, definition := range DefaultRegistry().All() {
|
||||||
|
if !definition.Generated {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if len(definition.DistributorPathTemplates) == 0 {
|
||||||
|
t.Fatalf("%s DistributorPathTemplates is empty", definition.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRegistryDefinitionsDeclareDefaultDistributorPathTemplates(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
id ID
|
||||||
|
want []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
id: Hourly,
|
||||||
|
want: []string{
|
||||||
|
"hourly/index.md",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: Daily,
|
||||||
|
want: []string{
|
||||||
|
"daily/{valid_start_date}/{run_id}.md",
|
||||||
|
"daily/{valid_start_date}/index.md",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: Today,
|
||||||
|
want: []string{
|
||||||
|
"daily/{valid_start_date}/{run_id}.md",
|
||||||
|
"daily/{valid_start_date}/index.md",
|
||||||
|
"today/index.md",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: Tomorrow,
|
||||||
|
want: []string{
|
||||||
|
"daily/{valid_start_date}/{run_id}.md",
|
||||||
|
"daily/{valid_start_date}/index.md",
|
||||||
|
"tomorrow/index.md",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: ThreeDay,
|
||||||
|
want: []string{
|
||||||
|
"three-day/{valid_start_date}/{run_id}.md",
|
||||||
|
"three-day/{valid_start_date}/index.md",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: Weekend,
|
||||||
|
want: []string{
|
||||||
|
"weekend/{valid_start_date}/{run_id}.md",
|
||||||
|
"weekend/{valid_start_date}/index.md",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: Storm,
|
||||||
|
want: []string{
|
||||||
|
"storm/{storm_id}/{run_id}.md",
|
||||||
|
"storm/{storm_id}/index.md",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
registry := DefaultRegistry()
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(string(tt.id), func(t *testing.T) {
|
||||||
|
definition, err := registry.Lookup(tt.id)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Lookup() error = %v", err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(definition.DistributorPathTemplates, tt.want) {
|
||||||
|
t.Fatalf("DistributorPathTemplates = %#v, want %#v", definition.DistributorPathTemplates, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRegistryDefinitionsDeclareDefaultModules(t *testing.T) {
|
func TestRegistryDefinitionsDeclareDefaultModules(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
id ID
|
id ID
|
||||||
@@ -682,6 +764,9 @@ func TestRegistryAppliesModuleOverridesWithoutChangingDefaults(t *testing.T) {
|
|||||||
if len(defaultDefinition.ModuleIDs()) <= len(definition.ModuleIDs()) {
|
if len(defaultDefinition.ModuleIDs()) <= len(definition.ModuleIDs()) {
|
||||||
t.Fatalf("default ModuleIDs() = %#v, want original defaults unchanged", defaultDefinition.ModuleIDs())
|
t.Fatalf("default ModuleIDs() = %#v, want original defaults unchanged", defaultDefinition.ModuleIDs())
|
||||||
}
|
}
|
||||||
|
if !reflect.DeepEqual(definition.DistributorPathTemplates, defaultDefinition.DistributorPathTemplates) {
|
||||||
|
t.Fatalf("overridden DistributorPathTemplates = %#v, want %#v", definition.DistributorPathTemplates, defaultDefinition.DistributorPathTemplates)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRegistryRejectsModuleOverrideForUnknownReport(t *testing.T) {
|
func TestRegistryRejectsModuleOverrideForUnknownReport(t *testing.T) {
|
||||||
|
|||||||
@@ -30,8 +30,7 @@ func DefaultRegistry() Registry {
|
|||||||
func (r Registry) WithModuleOverrides(overrides map[ID][]module.ConfigItem) (Registry, error) {
|
func (r Registry) WithModuleOverrides(overrides map[ID][]module.ConfigItem) (Registry, error) {
|
||||||
next := Registry{definitions: map[ID]Definition{}}
|
next := Registry{definitions: map[ID]Definition{}}
|
||||||
for id, definition := range r.definitions {
|
for id, definition := range r.definitions {
|
||||||
definition.Modules = append([]module.ConfigItem(nil), definition.Modules...)
|
next.definitions[id] = cloneDefinition(definition)
|
||||||
next.definitions[id] = definition
|
|
||||||
}
|
}
|
||||||
for id, items := range overrides {
|
for id, items := range overrides {
|
||||||
definition, ok := next.definitions[id]
|
definition, ok := next.definitions[id]
|
||||||
@@ -44,6 +43,12 @@ func (r Registry) WithModuleOverrides(overrides map[ID][]module.ConfigItem) (Reg
|
|||||||
return next, nil
|
return next, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func cloneDefinition(definition Definition) Definition {
|
||||||
|
definition.Modules = cloneModuleItems(definition.Modules)
|
||||||
|
definition.DistributorPathTemplates = append([]string(nil), definition.DistributorPathTemplates...)
|
||||||
|
return definition
|
||||||
|
}
|
||||||
|
|
||||||
func moduleItems(ids ...module.ID) []module.ConfigItem {
|
func moduleItems(ids ...module.ID) []module.ConfigItem {
|
||||||
items := make([]module.ConfigItem, 0, len(ids))
|
items := make([]module.ConfigItem, 0, len(ids))
|
||||||
for _, id := range ids {
|
for _, id := range ids {
|
||||||
|
|||||||
@@ -17,6 +17,10 @@ func stormDefinition() Definition {
|
|||||||
ComparisonStrategy: CompareExplicitWindow,
|
ComparisonStrategy: CompareExplicitWindow,
|
||||||
ArtifactGroup: "storm",
|
ArtifactGroup: "storm",
|
||||||
BatchOutputName: "storm.md",
|
BatchOutputName: "storm.md",
|
||||||
|
DistributorPathTemplates: []string{
|
||||||
|
"storm/{storm_id}/{run_id}.md",
|
||||||
|
"storm/{storm_id}/index.md",
|
||||||
|
},
|
||||||
Generated: true,
|
Generated: true,
|
||||||
CompatiblePriorIDs: []ID{Storm},
|
CompatiblePriorIDs: []ID{Storm},
|
||||||
Modules: stormModules(),
|
Modules: stormModules(),
|
||||||
|
|||||||
@@ -16,6 +16,10 @@ func threeDayDefinition() Definition {
|
|||||||
ComparisonStrategy: CompareSameValidDate,
|
ComparisonStrategy: CompareSameValidDate,
|
||||||
ArtifactGroup: "three-day",
|
ArtifactGroup: "three-day",
|
||||||
BatchOutputName: "three-day.md",
|
BatchOutputName: "three-day.md",
|
||||||
|
DistributorPathTemplates: []string{
|
||||||
|
"three-day/{valid_start_date}/{run_id}.md",
|
||||||
|
"three-day/{valid_start_date}/index.md",
|
||||||
|
},
|
||||||
Generated: true,
|
Generated: true,
|
||||||
CompatiblePriorIDs: []ID{ThreeDay},
|
CompatiblePriorIDs: []ID{ThreeDay},
|
||||||
Modules: threeDayModules(),
|
Modules: threeDayModules(),
|
||||||
|
|||||||
@@ -16,6 +16,11 @@ func todayDefinition() Definition {
|
|||||||
ComparisonStrategy: CompareSameValidDate,
|
ComparisonStrategy: CompareSameValidDate,
|
||||||
ArtifactGroup: "today",
|
ArtifactGroup: "today",
|
||||||
BatchOutputName: "today.md",
|
BatchOutputName: "today.md",
|
||||||
|
DistributorPathTemplates: []string{
|
||||||
|
"daily/{valid_start_date}/{run_id}.md",
|
||||||
|
"daily/{valid_start_date}/index.md",
|
||||||
|
"today/index.md",
|
||||||
|
},
|
||||||
Generated: true,
|
Generated: true,
|
||||||
CompatiblePriorIDs: []ID{Today},
|
CompatiblePriorIDs: []ID{Today},
|
||||||
Modules: todayModules(),
|
Modules: todayModules(),
|
||||||
|
|||||||
@@ -16,6 +16,11 @@ func tomorrowDefinition() Definition {
|
|||||||
ComparisonStrategy: CompareSameValidDate,
|
ComparisonStrategy: CompareSameValidDate,
|
||||||
ArtifactGroup: "tomorrow",
|
ArtifactGroup: "tomorrow",
|
||||||
BatchOutputName: "tomorrow.md",
|
BatchOutputName: "tomorrow.md",
|
||||||
|
DistributorPathTemplates: []string{
|
||||||
|
"daily/{valid_start_date}/{run_id}.md",
|
||||||
|
"daily/{valid_start_date}/index.md",
|
||||||
|
"tomorrow/index.md",
|
||||||
|
},
|
||||||
Generated: true,
|
Generated: true,
|
||||||
CompatiblePriorIDs: []ID{Tomorrow},
|
CompatiblePriorIDs: []ID{Tomorrow},
|
||||||
Modules: tomorrowModules(),
|
Modules: tomorrowModules(),
|
||||||
|
|||||||
@@ -17,6 +17,10 @@ func weekendDefinition() Definition {
|
|||||||
ComparisonStrategy: CompareWeekendWindow,
|
ComparisonStrategy: CompareWeekendWindow,
|
||||||
ArtifactGroup: "weekend",
|
ArtifactGroup: "weekend",
|
||||||
BatchOutputName: "weekend.md",
|
BatchOutputName: "weekend.md",
|
||||||
|
DistributorPathTemplates: []string{
|
||||||
|
"weekend/{valid_start_date}/{run_id}.md",
|
||||||
|
"weekend/{valid_start_date}/index.md",
|
||||||
|
},
|
||||||
Generated: true,
|
Generated: true,
|
||||||
CompatiblePriorIDs: []ID{Weekend},
|
CompatiblePriorIDs: []ID{Weekend},
|
||||||
Modules: weekendModules(),
|
Modules: weekendModules(),
|
||||||
|
|||||||
Reference in New Issue
Block a user