Remove legacy distributor report path config

This commit is contained in:
2026-06-20 02:58:41 +00:00
parent 8d2ac163ae
commit 4c606eb39f
11 changed files with 105 additions and 74 deletions

View File

@@ -101,15 +101,6 @@ contains all managed Markdown reports from that batch.
Default: `weatherreporter.{location_id}.{report_id}`. Default: `weatherreporter.{location_id}.{report_id}`.
- `idempotency_key_template`: template for single-report distributor - `idempotency_key_template`: template for single-report distributor
idempotency keys. Default: `{bundle_id}.{run_id}`. idempotency keys. Default: `{bundle_id}.{run_id}`.
- `report_path_templates`: ordered list of templates for Markdown report paths
inside distributor bundles. For `generate <report>`, each rendered path maps
to the generated report's managed Markdown source. For `run morning` and
`run evening`, the templates are rendered once per included report and each
rendered path maps that report's managed Markdown source into the one batch
bundle. Default:
```yaml
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
```
- `batch.enabled`: whether batch distributor notification config is active - `batch.enabled`: whether batch distributor notification config is active
when distributor notification is enabled. Default: `true`. when distributor notification is enabled. Default: `true`.
- `batch.pipeline_id_template`: template for batch distributor pipeline IDs. - `batch.pipeline_id_template`: template for batch distributor pipeline IDs.

View File

@@ -163,10 +163,11 @@ all planned reports finish, app orchestration evaluates batch notification:
3. If any planned report failed, the batch notification result is `skipped` 3. If any planned report failed, the batch notification result is `skipped`
with reason `one or more reports failed`, and distributor is not called. with reason `one or more reports failed`, and distributor is not called.
4. If every report succeeded, app orchestration renders batch pipeline, bundle 4. If every report succeeded, app orchestration renders batch pipeline, bundle
ID, and idempotency key templates, renders `report_path_templates` for each ID, and idempotency key templates, renders report-specific distributor
included report, validates every managed source path and bundle path, checks paths for each included report, validates every managed source path and
duplicate bundle paths across the batch, calls the notifier once with a bundle path, checks duplicate bundle paths across the batch, calls the
multi-file request, and saves a batch notification debug artifact. notifier once with a multi-file request, and saves a batch notification
debug artifact.
Batch notification failure records a top-level failed notification, increments Batch notification failure records a top-level failed notification, increments
the aggregate batch failure count, and returns an aggregate batch error without the aggregate batch failure count, and returns an aggregate batch error without

View File

@@ -61,12 +61,12 @@ and bundle paths from:
- `pipeline_id_template` - `pipeline_id_template`
- `bundle_id_template` - `bundle_id_template`
- `idempotency_key_template` - `idempotency_key_template`
- `report_path_templates` - report-specific path templates
For batch uploads, the app layer renders pipeline ID, bundle ID, and For batch uploads, the app layer renders pipeline ID, bundle ID, and
idempotency key from `notify.distributor.batch.*`, renders idempotency key from `notify.distributor.batch.*`, resolves report-specific
`report_path_templates` once per included report, and passes the resulting path templates once per included report, and passes the resulting multi-file
multi-file request to this adapter. request to this adapter.
The token value is read from the environment variable named by `token_env` The token value is read from the environment variable named by `token_env`
after config loading and `secrets.directory` processing. after config loading and `secrets.directory` processing.

View File

@@ -256,17 +256,16 @@ top-level `notification` object with fields such as `status`, `reason`,
Distributor notification is configured with `notify.distributor` and is Distributor notification is configured with `notify.distributor` and is
disabled by default. For `generate <report>`, weatherreporter uploads the disabled by default. For `generate <report>`, weatherreporter uploads the
managed Markdown report path recorded in the report result and metadata. That managed Markdown report path recorded in the report result and metadata. That
single source file can be mapped to one or more configured bundle paths. By single source file is mapped to report-specific bundle paths. Extra copies
default, it is mapped to one dated report path. Extra copies written by `--out` written by `--out` or `--out-dir` are operator conveniences only.
or `--out-dir` are operator conveniences only.
For `run morning` and `run evening`, per-report notification is suppressed. If For `run morning` and `run evening`, per-report notification is suppressed. If
`notify.distributor.enabled` and `notify.distributor.batch.enabled` are both `notify.distributor.enabled` and `notify.distributor.batch.enabled` are both
true, the batch uploads once after all reports finish successfully. The upload true, the batch uploads once after all reports finish successfully. The upload
contains one file mapping set per included report. Each mapping uses the contains one file mapping set per included report. Each mapping uses the
managed Markdown report as the source and renders `report_path_templates` using managed Markdown report as the source and report-specific path templates for
that report's normal template values. All rendered bundle paths across the that report. All rendered bundle paths across the batch must be unique. If any
batch must be unique. If any report fails, weatherreporter records a top-level report fails, weatherreporter records a top-level
notification status of `skipped` with reason `one or more reports failed` and notification status of `skipped` with reason `one or more reports failed` and
does not call distributor. If batch notification is disabled, run commands do does not call distributor. If batch notification is disabled, run commands do
not fall back to per-report uploads. not fall back to per-report uploads.

View File

@@ -300,15 +300,15 @@ Relevant docs: [Configuration reference](config.md),
Symptom: a batch returns nonzero with duplicate bundle path context before a Symptom: a batch returns nonzero with duplicate bundle path context before a
distributor run ID is accepted. distributor run ID is accepted.
Likely cause: `notify.distributor.report_path_templates` rendered the same Likely cause: report-specific distributor path templates rendered the same
bundle-relative path for two included reports in the same batch. bundle-relative path for two included reports in the same batch.
Diagnostic: inspect the error in stdout or stderr. The validation error Diagnostic: inspect the error in stdout or stderr. The validation error
includes the duplicate bundle path plus the report IDs, RunIDs, and managed includes the duplicate bundle path plus the report IDs, RunIDs, and managed
source paths involved. source paths involved.
Safe fix: change `notify.distributor.report_path_templates` so every report in Safe fix: configure a per-report distributor path override so every report in a
a batch renders a unique path. Include values such as `{artifact_group}`, batch renders a unique path. Include values such as `{artifact_group}`,
`{valid_start_date}`, `{batch_output_name}`, or `{run_id}` when needed. `{valid_start_date}`, `{batch_output_name}`, or `{run_id}` when needed.
Relevant docs: [Configuration reference](config.md), Relevant docs: [Configuration reference](config.md),
@@ -411,7 +411,7 @@ Diagnostic: inspect stdout JSON or stderr status lines for
`notificationError` or the top-level batch notification `error`. Confirm `notificationError` or the top-level batch notification `error`. Confirm
`notify.distributor.endpoint`, `notify.distributor.endpoint`,
`notify.distributor.pipeline_id_template`, `notify.distributor.pipeline_id_template`,
`notify.distributor.report_path_templates`, and token configuration. Token report-specific distributor paths, and token configuration. Token
values are redacted from weatherreporter errors. values are redacted from weatherreporter errors.
If the upload was accepted but destination output did not change, inspect the If the upload was accepted but destination output did not change, inspect the

View File

@@ -24,8 +24,6 @@ notify:
pipeline_id_template: "weatherreporter.{report_id}" pipeline_id_template: "weatherreporter.{report_id}"
bundle_id_template: "weatherreporter.{location_id}.{report_id}" bundle_id_template: "weatherreporter.{location_id}.{report_id}"
idempotency_key_template: "{bundle_id}.{run_id}" idempotency_key_template: "{bundle_id}.{run_id}"
report_path_templates:
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
batch: batch:
enabled: true enabled: true
pipeline_id_template: "weatherreporter" pipeline_id_template: "weatherreporter"

View File

@@ -971,7 +971,7 @@ func distributorTemplateValuesForReport(cfg config.Config, resolved report.Resol
} }
func renderDistributorReportBundlePaths(cfg config.Config, resolved report.Resolved, runID string, sourcePath string, values config.DistributorTemplateValues) ([]string, error) { func renderDistributorReportBundlePaths(cfg config.Config, resolved report.Resolved, runID string, sourcePath string, values config.DistributorTemplateValues) ([]string, error) {
templates, name, err := distributorReportPathTemplates(cfg, resolved.Definition) templates, name, err := distributorPathTemplatesForReport(cfg, resolved.Definition)
if err != nil { if err != nil {
return nil, distributorReportPathError(resolved.Definition.ID, runID, sourcePath, err) return nil, distributorReportPathError(resolved.Definition.ID, runID, sourcePath, err)
} }
@@ -982,7 +982,7 @@ func renderDistributorReportBundlePaths(cfg config.Config, resolved report.Resol
return paths, nil return paths, nil
} }
func distributorReportPathTemplates(cfg config.Config, definition report.Definition) ([]string, string, error) { func distributorPathTemplatesForReport(cfg config.Config, definition report.Definition) ([]string, string, error) {
overrides, err := cfg.ReportDistributorPathOverrides() overrides, err := cfg.ReportDistributorPathOverrides()
if err != nil { if err != nil {
return nil, "", err return nil, "", err

View File

@@ -66,7 +66,6 @@ type DistributorNotifyConfig struct {
PipelineIDTemplate string `yaml:"pipeline_id_template"` PipelineIDTemplate string `yaml:"pipeline_id_template"`
BundleIDTemplate string `yaml:"bundle_id_template"` BundleIDTemplate string `yaml:"bundle_id_template"`
IdempotencyKeyTemplate string `yaml:"idempotency_key_template"` IdempotencyKeyTemplate string `yaml:"idempotency_key_template"`
ReportPathTemplates []string `yaml:"report_path_templates"`
Batch DistributorBatchNotifyConfig `yaml:"batch"` Batch DistributorBatchNotifyConfig `yaml:"batch"`
} }
@@ -152,6 +151,57 @@ func (c *ReportConfig) UnmarshalYAML(value *yaml.Node) error {
return nil return nil
} }
func (c *DistributorNotifyConfig) UnmarshalYAML(value *yaml.Node) error {
if value.Kind != yaml.MappingNode {
return fmt.Errorf("notify distributor entry must be a mapping")
}
for i := 0; i < len(value.Content); i += 2 {
key := value.Content[i].Value
node := value.Content[i+1]
switch key {
case "enabled":
if err := node.Decode(&c.Enabled); err != nil {
return err
}
case "endpoint":
if err := node.Decode(&c.Endpoint); err != nil {
return err
}
case "token_env":
if err := node.Decode(&c.TokenEnv); err != nil {
return err
}
case "timeout":
if err := node.Decode(&c.Timeout); err != nil {
return err
}
case "failure_policy":
if err := node.Decode(&c.FailurePolicy); err != nil {
return err
}
case "pipeline_id_template":
if err := node.Decode(&c.PipelineIDTemplate); err != nil {
return err
}
case "bundle_id_template":
if err := node.Decode(&c.BundleIDTemplate); err != nil {
return err
}
case "idempotency_key_template":
if err := node.Decode(&c.IdempotencyKeyTemplate); err != nil {
return err
}
case "batch":
if err := node.Decode(&c.Batch); err != nil {
return err
}
default:
return fmt.Errorf("unknown notify distributor field %q", key)
}
}
return nil
}
func (c *ReportDistributorConfig) UnmarshalYAML(value *yaml.Node) error { func (c *ReportDistributorConfig) UnmarshalYAML(value *yaml.Node) error {
if value.Kind != yaml.MappingNode { if value.Kind != yaml.MappingNode {
return fmt.Errorf("report distributor entry must be a mapping") return fmt.Errorf("report distributor entry must be a mapping")

View File

@@ -58,12 +58,6 @@ func TestDefaults(t *testing.T) {
if cfg.Notify.Distributor.IdempotencyKeyTemplate != "{bundle_id}.{run_id}" { if cfg.Notify.Distributor.IdempotencyKeyTemplate != "{bundle_id}.{run_id}" {
t.Fatalf("Notify.Distributor.IdempotencyKeyTemplate = %q, want default", cfg.Notify.Distributor.IdempotencyKeyTemplate) t.Fatalf("Notify.Distributor.IdempotencyKeyTemplate = %q, want default", cfg.Notify.Distributor.IdempotencyKeyTemplate)
} }
wantReportPaths := []string{
"{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md",
}
if strings.Join(cfg.Notify.Distributor.ReportPathTemplates, "\n") != strings.Join(wantReportPaths, "\n") {
t.Fatalf("Notify.Distributor.ReportPathTemplates = %#v, want %#v", cfg.Notify.Distributor.ReportPathTemplates, wantReportPaths)
}
if !cfg.Notify.Distributor.Batch.Enabled { if !cfg.Notify.Distributor.Batch.Enabled {
t.Fatalf("Notify.Distributor.Batch.Enabled = false, want true") t.Fatalf("Notify.Distributor.Batch.Enabled = false, want true")
} }
@@ -102,9 +96,6 @@ func TestLoadExampleConfig(t *testing.T) {
if cfg.Notify.Distributor.PipelineIDTemplate != "weatherreporter.{report_id}" { if cfg.Notify.Distributor.PipelineIDTemplate != "weatherreporter.{report_id}" {
t.Fatalf("PipelineIDTemplate = %q, want example pipeline template", cfg.Notify.Distributor.PipelineIDTemplate) t.Fatalf("PipelineIDTemplate = %q, want example pipeline template", cfg.Notify.Distributor.PipelineIDTemplate)
} }
if len(cfg.Notify.Distributor.ReportPathTemplates) != 1 {
t.Fatalf("ReportPathTemplates = %#v, want example archive path", cfg.Notify.Distributor.ReportPathTemplates)
}
if !cfg.Notify.Distributor.Batch.Enabled { if !cfg.Notify.Distributor.Batch.Enabled {
t.Fatalf("Notify.Distributor.Batch.Enabled = false, want true") t.Fatalf("Notify.Distributor.Batch.Enabled = false, want true")
} }
@@ -1065,6 +1056,37 @@ func TestDisabledDistributorNotifyAcceptsOmittedFields(t *testing.T) {
} }
} }
func TestDistributorNotifyRejectsRemovedGlobalReportPaths(t *testing.T) {
removedField := "report_path" + "_templates"
_, err := LoadFile(writeConfig(t, `
notify:
distributor:
`+removedField+`:
- index.md
`))
if err == nil {
t.Fatal("LoadFile() error = nil, want removed global path field error")
}
if !strings.Contains(err.Error(), `unknown notify distributor field "`+removedField+`"`) {
t.Fatalf("error = %q, want removed global path field rejection", err.Error())
}
}
func TestDistributorNotifyRejectsUnknownFields(t *testing.T) {
_, err := LoadFile(writeConfig(t, `
notify:
distributor:
paths:
- index.md
`))
if err == nil {
t.Fatal("LoadFile() error = nil, want unknown distributor field error")
}
if !strings.Contains(err.Error(), `unknown notify distributor field "paths"`) {
t.Fatalf("error = %q, want unknown field rejection", err.Error())
}
}
func TestDisabledDistributorNotifyAcceptsMalformedBatchTemplates(t *testing.T) { func TestDisabledDistributorNotifyAcceptsMalformedBatchTemplates(t *testing.T) {
cfg := Defaults() cfg := Defaults()
cfg.Notify.Distributor.Enabled = false cfg.Notify.Distributor.Enabled = false
@@ -1154,32 +1176,11 @@ func TestEnabledDistributorNotifyValidation(t *testing.T) {
wantErr: "notify.distributor.idempotency_key_template", wantErr: "notify.distributor.idempotency_key_template",
}, },
{ {
name: "ReportPathTemplatesEmpty", name: "BatchTemplate",
mutate: func(cfg *Config) { mutate: func(cfg *Config) {
cfg.Notify.Distributor.ReportPathTemplates = nil cfg.Notify.Distributor.Batch.BundleIDTemplate = "{run_id}"
}, },
wantErr: "notify.distributor.report_path_templates", wantErr: "notify.distributor.batch.bundle_id_template",
},
{
name: "ReportPathTemplateUnknown",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.ReportPathTemplates = []string{"{unknown}"}
},
wantErr: "notify.distributor.report_path_templates",
},
{
name: "ReportPathTemplateInvalidPath",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.ReportPathTemplates = []string{"/{batch_output_name}"}
},
wantErr: "notify.distributor.report_path_templates",
},
{
name: "ReportPathTemplateDuplicatePath",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.ReportPathTemplates = []string{"latest.md", "latest.md"}
},
wantErr: "notify.distributor.report_path_templates",
}, },
} }

View File

@@ -31,9 +31,6 @@ func Defaults() Config {
PipelineIDTemplate: "", PipelineIDTemplate: "",
BundleIDTemplate: "weatherreporter.{location_id}.{report_id}", BundleIDTemplate: "weatherreporter.{location_id}.{report_id}",
IdempotencyKeyTemplate: "{bundle_id}.{run_id}", IdempotencyKeyTemplate: "{bundle_id}.{run_id}",
ReportPathTemplates: []string{
"{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md",
},
Batch: DistributorBatchNotifyConfig{ Batch: DistributorBatchNotifyConfig{
Enabled: true, Enabled: true,
PipelineIDTemplate: "weatherreporter", PipelineIDTemplate: "weatherreporter",

View File

@@ -124,9 +124,6 @@ func validateDistributorNotify(cfg DistributorNotifyConfig) error {
if err := validateDistributorTemplate("notify.distributor.idempotency_key_template", cfg.IdempotencyKeyTemplate, distributorIdempotencyTemplateVariables); err != nil { if err := validateDistributorTemplate("notify.distributor.idempotency_key_template", cfg.IdempotencyKeyTemplate, distributorIdempotencyTemplateVariables); err != nil {
return err return err
} }
if len(cfg.ReportPathTemplates) == 0 {
return fmt.Errorf("notify.distributor.report_path_templates must contain at least one entry when enabled")
}
values := sampleDistributorTemplateValues() values := sampleDistributorTemplateValues()
bundleID, err := RenderDistributorBundleID(cfg.BundleIDTemplate, values) bundleID, err := RenderDistributorBundleID(cfg.BundleIDTemplate, values)
if err != nil { if err != nil {
@@ -136,9 +133,6 @@ func validateDistributorNotify(cfg DistributorNotifyConfig) error {
if _, err := RenderDistributorPipelineID(cfg.PipelineIDTemplate, values); err != nil { if _, err := RenderDistributorPipelineID(cfg.PipelineIDTemplate, values); err != nil {
return err return err
} }
if _, err := RenderDistributorReportPaths("notify.distributor.report_path_templates", cfg.ReportPathTemplates, values); err != nil {
return err
}
if err := validateDistributorBatchNotify(cfg.Batch); err != nil { if err := validateDistributorBatchNotify(cfg.Batch); err != nil {
return err return err
} }