Compare commits
11 Commits
v0.5.0
...
138bc4e7e4
| Author | SHA1 | Date | |
|---|---|---|---|
| 138bc4e7e4 | |||
| e2529cfdf2 | |||
| b982b27f84 | |||
| c573cd5b4d | |||
| c2758d7a91 | |||
| 64cae8c4d9 | |||
| a2ba6f5382 | |||
| 7c8d9191c1 | |||
| 9677835d84 | |||
| 63749a9572 | |||
| 9ff90d33fc |
50
.woodpecker/release.yml
Normal file
50
.woodpecker/release.yml
Normal file
@@ -0,0 +1,50 @@
|
||||
when:
|
||||
- event: tag
|
||||
|
||||
steps:
|
||||
- name: build-release-assets
|
||||
image: golang:1.25
|
||||
commands:
|
||||
- |
|
||||
set -eu
|
||||
|
||||
version="$CI_COMMIT_TAG"
|
||||
dist="dist"
|
||||
pkg="gitea.maximumdirect.net/eric/weatherreporter/cmd/weatherreporter"
|
||||
|
||||
rm -rf "$dist"
|
||||
mkdir -p "$dist"
|
||||
|
||||
build_binary() {
|
||||
goos="$1"
|
||||
goarch="$2"
|
||||
suffix="$3"
|
||||
output="$dist/weatherreporter-$version-$goos-$goarch$suffix"
|
||||
|
||||
CGO_ENABLED=0 GOOS="$goos" GOARCH="$goarch" \
|
||||
go build -trimpath -ldflags "-s -w -X gitea.maximumdirect.net/eric/weatherreporter/internal/buildinfo.Version=$version" \
|
||||
-o "$output" "$pkg"
|
||||
}
|
||||
|
||||
build_binary linux amd64 ""
|
||||
build_binary linux arm64 ""
|
||||
build_binary darwin amd64 ""
|
||||
build_binary darwin arm64 ""
|
||||
build_binary windows amd64 ".exe"
|
||||
build_binary windows arm64 ".exe"
|
||||
|
||||
- name: publish-release
|
||||
image: woodpeckerci/plugin-release
|
||||
depends_on:
|
||||
- build-release-assets
|
||||
settings:
|
||||
api_key:
|
||||
from_secret: GITEA_RELEASE_TOKEN
|
||||
files:
|
||||
- dist/weatherreporter-*
|
||||
checksum: sha256
|
||||
checksum-file: SHA256SUMS
|
||||
checksum-flatten: true
|
||||
file-exists: skip
|
||||
overwrite: false
|
||||
prerelease: false
|
||||
@@ -3,7 +3,8 @@
|
||||
`weatherreporter` is a Go application for preparing human-facing weather
|
||||
reports from normalized forecast data. It builds structured briefing packages,
|
||||
runs them through `scriptorium`, and keeps inspectable artifacts under a local
|
||||
workspace.
|
||||
workspace. It can also upload successfully generated managed Markdown reports
|
||||
to a configured `distributor` HTTP upload endpoint.
|
||||
|
||||
## Quickstart
|
||||
|
||||
|
||||
19
docs/cli.md
19
docs/cli.md
@@ -11,7 +11,9 @@ weatherreporter generate daily --date 2026-05-29 --out ./daily.md
|
||||
|
||||
This loads configuration, fetches weather data, writes managed workspace
|
||||
artifacts, runs `scriptorium render` as a preflight check, runs
|
||||
`scriptorium run`, and writes an extra Markdown copy to `./daily.md`.
|
||||
`scriptorium run`, and writes an extra Markdown copy to `./daily.md`. If
|
||||
distributor notification is enabled in configuration, the command also uploads
|
||||
the managed Markdown report after final metadata is saved.
|
||||
|
||||
## Commands
|
||||
|
||||
@@ -33,14 +35,20 @@ weatherreporter inspect sources [--config PATH] RUN_ID
|
||||
```
|
||||
|
||||
`generate` commands write briefing, data package, preflight, report, and
|
||||
metadata artifacts under the configured workspace. `generate storm` requires
|
||||
explicit event-window bounds with `--start` and `--end`.
|
||||
metadata artifacts under the configured workspace. `--out` writes an extra
|
||||
Markdown copy for the operator; distributor notification uses the managed
|
||||
report path, not the extra copy. `generate storm` requires explicit
|
||||
event-window bounds with `--start` and `--end`.
|
||||
|
||||
`run morning` generates Daily Today and the 3-Day Outlook, plus Weekend Outlook
|
||||
except on Sunday. `run evening` generates the Tomorrow Planning Brief. Batch
|
||||
runs continue independent reports after a failure, print a JSON summary to
|
||||
stdout, write compact status lines to stderr, and return nonzero when any report
|
||||
failed.
|
||||
failed. `--out-dir` writes extra Markdown copies for the operator; distributor
|
||||
notification uses each managed report path, not the extra copies. When
|
||||
notification is enabled, batch summaries and status lines include notification
|
||||
status, accepted distributor run ID, or notification error fields for each
|
||||
attempted report.
|
||||
|
||||
`inspect` commands read existing workspace artifacts and emit JSON to stdout.
|
||||
They do not fetch weather data or invoke `scriptorium`.
|
||||
@@ -61,6 +69,9 @@ They do not fetch weather data or invoke `scriptorium`.
|
||||
Storm times accept `YYYY-MM-DDTHH:MM` in the configured timezone or RFC3339
|
||||
timestamps with explicit offsets.
|
||||
|
||||
Distributor notification is configured only through `notify.distributor`; there
|
||||
are no distributor-specific CLI flags.
|
||||
|
||||
## Common Workflows
|
||||
|
||||
```sh
|
||||
|
||||
@@ -18,7 +18,7 @@ Precedence is:
|
||||
|
||||
The implemented configuration overrides are `--units` and `--tz`. Output flags
|
||||
control report copies for the current command but do not change configuration
|
||||
files. Environment-variable configuration is not implemented.
|
||||
files. Environment variables do not override configuration fields.
|
||||
|
||||
## Minimal Config
|
||||
|
||||
@@ -64,6 +64,53 @@ multiple configured forecast locations.
|
||||
The prompt-facing location object also includes `timezone`, derived from the
|
||||
effective `weather_api.timezone` after CLI overrides such as `--tz`.
|
||||
|
||||
### `secrets`
|
||||
|
||||
- `directory`: optional directory of file-backed environment secrets. Default:
|
||||
empty, which disables secret loading.
|
||||
|
||||
When configured, each regular file directly under `secrets.directory` is loaded
|
||||
after config file parsing and CLI overrides. The file basename must be a valid
|
||||
environment variable name matching `[A-Za-z_][A-Za-z0-9_]*`; the file contents
|
||||
become the environment variable value and overwrite any existing value. One
|
||||
trailing LF or CRLF is stripped. Subdirectories, symlinks, invalid filenames,
|
||||
missing directories, and unreadable files fail config loading.
|
||||
|
||||
### `notify`
|
||||
|
||||
`notify.distributor` controls distributor notification after successful report
|
||||
generation. It is disabled by default and does not add CLI flags. When enabled,
|
||||
weatherreporter uploads one distributor bundle per generated report after
|
||||
`scriptorium run` succeeds and final metadata is saved.
|
||||
|
||||
- `enabled`: whether distributor notification config is active. Default:
|
||||
`false`.
|
||||
- `endpoint`: absolute distributor endpoint URL. Required when enabled.
|
||||
Default: `https://distributor.example.com`.
|
||||
- `token_env`: environment variable name that will contain the distributor
|
||||
upload token. Required when enabled. Default: `DISTRIBUTOR_UPLOAD_TOKEN`.
|
||||
- `timeout`: distributor operation timeout. Must be greater than zero when
|
||||
enabled. Default: `30s`.
|
||||
- `failure_policy`: must be `error` when enabled. Default: `error`.
|
||||
- `bundle_id_template`: template for distributor bundle IDs. Default:
|
||||
`weatherreporter.{location_id}.{report_id}.{run_id}`.
|
||||
- `idempotency_key_template`: template for distributor idempotency keys.
|
||||
Default: `{bundle_id}`.
|
||||
- `report_path_template`: template for the Markdown report path inside the
|
||||
distributor bundle. Default: `{batch_output_name}`.
|
||||
|
||||
Supported template variables are `location_id`, `report_id`, `run_id`,
|
||||
`artifact_group`, and `batch_output_name`. `idempotency_key_template` may also
|
||||
use `bundle_id`.
|
||||
|
||||
Rendered report paths must be relative paths with `/` separators. They must not
|
||||
contain backslashes, empty path segments, `.`, `..`, `manifest.json`, or
|
||||
`.distributor.json`.
|
||||
|
||||
The upload token is read from the environment variable named by `token_env`
|
||||
after config loading and `secrets.directory` processing. Config files should
|
||||
name the variable only; they should not contain the token value.
|
||||
|
||||
### `missing_source`
|
||||
|
||||
- `default`: missing-source behavior for optional sources. One of `error`, `warn`, or `none`. Default: `warn`.
|
||||
@@ -115,8 +162,13 @@ snapshot exists and a threshold is crossed.
|
||||
|
||||
## Secrets
|
||||
|
||||
Configuration files should not contain secrets. The current Weather API and
|
||||
Scriptorium integration settings do not require secret fields.
|
||||
Configuration files should not contain raw secrets. Use `secrets.directory` to
|
||||
load secret values from files into environment variables for integrations that
|
||||
read credentials from the environment. Secret file names become environment
|
||||
variable names, and secret file contents become values. For distributor
|
||||
notification, this allows a file such as
|
||||
`<secrets.directory>/DISTRIBUTOR_UPLOAD_TOKEN` to supply the token referenced by
|
||||
`notify.distributor.token_env`.
|
||||
|
||||
## Maintained Examples
|
||||
|
||||
|
||||
127
docs/integrations/distributor/api.md
Normal file
127
docs/integrations/distributor/api.md
Normal file
@@ -0,0 +1,127 @@
|
||||
# Upstream Producer Integration
|
||||
|
||||
Audience: developers and LLM coding agents adding `distributor` support to an upstream Go producer application.
|
||||
|
||||
This document is the copyable implementation guide for submitting producer outputs to a `distributor` pipeline whose source backend is `http_upload`.
|
||||
|
||||
## Required Inputs
|
||||
|
||||
The upstream application needs these values from deployment or operator configuration:
|
||||
|
||||
- distributor endpoint: the HTTP server base URL, such as `https://distributor.example.com`;
|
||||
- upload token: bearer token for exactly one configured `http_upload` pipeline;
|
||||
- generated files: regular local files to include in the source bundle;
|
||||
- bundle id: stable identifier for this producer output;
|
||||
- idempotency key: stable key for retrying the same producer operation.
|
||||
|
||||
Do not put destination routing, public URLs, transform settings, or credentials in the source manifest. Those belong in the `distributor` pipeline configuration.
|
||||
|
||||
## Recommended Workflow
|
||||
|
||||
Use `gitea.maximumdirect.net/eric/distributor/pkg/upload`.
|
||||
|
||||
For most producers, use `UploadFiles`. It accepts producer-generated files, builds a temporary valid source bundle with `pkg/bundle`, uploads a gzip-compressed tar archive, and removes temporary files when the call returns.
|
||||
|
||||
Use `UploadBundle` only when the producer already assembled a complete bundle directory containing `manifest.json`.
|
||||
|
||||
Add the dependency from the upstream application:
|
||||
|
||||
```sh
|
||||
go get gitea.maximumdirect.net/eric/distributor
|
||||
```
|
||||
|
||||
## Minimal Go Example
|
||||
|
||||
```go
|
||||
package reports
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/distributor/pkg/bundle"
|
||||
"gitea.maximumdirect.net/eric/distributor/pkg/upload"
|
||||
)
|
||||
|
||||
func SubmitReport(reportPath, summaryPath string) error {
|
||||
endpoint := os.Getenv("DISTRIBUTOR_UPLOAD_ENDPOINT")
|
||||
token := os.Getenv("DISTRIBUTOR_UPLOAD_TOKEN")
|
||||
if endpoint == "" || token == "" {
|
||||
return fmt.Errorf("distributor endpoint and token are required")
|
||||
}
|
||||
|
||||
reportID := "weather.hourly.brentwood.2026-06-07T15"
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
client, err := upload.NewClient(upload.ClientOptions{
|
||||
Endpoint: endpoint,
|
||||
Token: token,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
|
||||
ID: reportID,
|
||||
IdempotencyKey: reportID,
|
||||
Files: []bundle.BundleFile{
|
||||
{SourcePath: reportPath, Path: "report.md"},
|
||||
{SourcePath: summaryPath, Path: "summary.txt"},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
var conflict *upload.IdempotencyConflictError
|
||||
if errors.As(err, &conflict) {
|
||||
return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Printf("distributor accepted run %s\n", result.RunID)
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
## Producer Responsibilities
|
||||
|
||||
- Use a stable bundle id for the producer output, such as a report type plus logical timestamp.
|
||||
- Use a stable idempotency key for cross-process retries of the same producer operation.
|
||||
- Map each generated file to a clean slash-separated bundle path, such as `report.md` or `assets/chart.png`.
|
||||
- Include only regular files. Symlinks, directories as files, devices, FIFOs, and sockets are rejected.
|
||||
- Keep file contents stable after upload inputs are selected. Bundle digests are calculated from file bytes.
|
||||
- Treat upload success as admission only. `UploadFiles` and `UploadBundle` return after the server accepts and validates the upload, not after all destinations publish.
|
||||
|
||||
Valid bundle paths are relative slash paths. They must not be empty, absolute, contain backslashes, contain `.` or `..` path segments, contain empty path segments, or use reserved basenames `manifest.json` or `.distributor.json`.
|
||||
|
||||
## Idempotency And Status
|
||||
|
||||
`pkg/upload` sends `Idempotency-Key` on every upload. If the caller omits one, the package generates a random key for that call and reuses it for in-process retries. That is enough for transient network retry within one process.
|
||||
|
||||
For producer jobs that may retry after process restart, supply a stable key derived from the producer operation, such as the report id or job id. Reusing the same key with the same normalized source manifest returns the original accepted run. Reusing the same key with different source content returns a conflict.
|
||||
|
||||
`Status` polls `/runs/<run-id>` while the distributor server retains the in-memory status record. Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to the server's `server.http.retention` setting, and server restart clears status and idempotency records.
|
||||
|
||||
Optional status check:
|
||||
|
||||
```go
|
||||
status, err := client.Status(ctx, result.RunID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if status.Status == "failed" {
|
||||
return fmt.Errorf("distributor run failed: %s", status.Error)
|
||||
}
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
In the `distributor` source tree:
|
||||
|
||||
- `docs/consumers/pkg-upload.md`: Go upload package workflow.
|
||||
- `docs/consumers/pkg-bundle.md`: Go bundle package workflow.
|
||||
- `docs/integrations/http-upload.md`: canonical HTTP upload wire contract.
|
||||
- `docs/integrations/source-bundle.md`: canonical source bundle file-format contract.
|
||||
88
docs/integrations/distributor/pkg-bundle.md
Normal file
88
docs/integrations/distributor/pkg-bundle.md
Normal file
@@ -0,0 +1,88 @@
|
||||
# `pkg/bundle`
|
||||
|
||||
Audience: upstream Go producer developers and LLM coding agents using `distributor` source bundle helpers.
|
||||
|
||||
Import path:
|
||||
|
||||
```go
|
||||
import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
|
||||
```
|
||||
|
||||
`pkg/bundle` builds, writes, parses, and validates local source bundles. Use it directly when a producer writes bundles for `distributor` to discover, or when a producer wants to assemble and validate a bundle before using another transport.
|
||||
|
||||
The canonical source bundle file-format contract is [Source Bundle Contract](../integrations/source-bundle.md).
|
||||
|
||||
## Preferred Complete-Bundle Workflow
|
||||
|
||||
Use `WriteBundle` when producer-generated files live outside the final bundle root.
|
||||
|
||||
```go
|
||||
manifest, err := bundle.WriteBundle(bundle.WriteBundleOptions{
|
||||
Root: "/var/spool/distributor/weather/hourly-2026-06-07T15",
|
||||
ID: "weather.hourly.brentwood.2026-06-07T15",
|
||||
Files: []bundle.BundleFile{
|
||||
{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
|
||||
{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_ = manifest
|
||||
```
|
||||
|
||||
`WriteBundle` copies each source file into a staged bundle root, writes `manifest.json`, validates the staged bundle, and promotes it into place. Set `Overwrite: true` only when the producer intentionally replaces an existing bundle root.
|
||||
|
||||
## Existing Bundle Root Workflow
|
||||
|
||||
Use `BuildManifest` and `WriteManifest` when files are already staged under the final bundle root.
|
||||
|
||||
```go
|
||||
root := "/var/spool/distributor/weather/hourly-2026-06-07T15"
|
||||
manifest, err := bundle.BuildManifest(bundle.BuildOptions{
|
||||
Root: root,
|
||||
ID: "weather.hourly.brentwood.2026-06-07T15",
|
||||
Files: []string{"report.md", "summary.txt"},
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := bundle.WriteManifest(root, manifest, bundle.WriteManifestOptions{}); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := bundle.ValidateBundle(root, manifest); err != nil {
|
||||
return err
|
||||
}
|
||||
```
|
||||
|
||||
Use `Scan: true` instead of `Files` only when every valid regular file under the root should be included. Scan mode includes dotfiles, skips reserved metadata files, rejects symlinks, and sorts paths lexically.
|
||||
|
||||
## Paths And Ordering
|
||||
|
||||
Bundle paths are slash-separated paths relative to the bundle root.
|
||||
|
||||
Invalid paths include:
|
||||
|
||||
- empty paths;
|
||||
- absolute paths;
|
||||
- paths containing backslashes;
|
||||
- `.` or `..` path segments;
|
||||
- empty path segments;
|
||||
- any basename of `manifest.json` or `.distributor.json`.
|
||||
|
||||
Explicit file lists preserve caller order. File order is part of the bundle digest, so producers should choose it deliberately and keep it stable.
|
||||
|
||||
## Validation And Digest Helpers
|
||||
|
||||
Use `ValidateBundle` before handing an existing local bundle to another process. It verifies manifest semantics, file existence, regular-file type, file size, per-file SHA-256 digests, and bundle digest.
|
||||
|
||||
Useful helpers:
|
||||
|
||||
- `LoadManifest`: read `manifest.json` from a bundle root.
|
||||
- `ParseManifest` and `MarshalManifest`: parse or write manifest bytes.
|
||||
- `ValidateManifest`: validate manifest-only semantics.
|
||||
- `FileDigest`, `BundleDigest`, and `ValidateDigest`: digest helpers for diagnostics and tests.
|
||||
|
||||
## Boundaries
|
||||
|
||||
`pkg/bundle` does not upload bundles, publish destinations, transform Markdown, select pipelines, configure credentials, or write destination state. Those concerns belong to `pkg/upload` or the `distributor` application.
|
||||
118
docs/integrations/distributor/pkg-upload.md
Normal file
118
docs/integrations/distributor/pkg-upload.md
Normal file
@@ -0,0 +1,118 @@
|
||||
# `pkg/upload`
|
||||
|
||||
Audience: upstream Go producer developers and LLM coding agents submitting bundles to `distributor serve`.
|
||||
|
||||
Import path:
|
||||
|
||||
```go
|
||||
import "gitea.maximumdirect.net/eric/distributor/pkg/upload"
|
||||
```
|
||||
|
||||
`pkg/upload` is the producer-facing HTTP upload client. It builds on `pkg/bundle`, packages valid source bundles as gzip-compressed tar archives, sends bearer authentication, includes idempotency keys, and exposes a status polling helper.
|
||||
|
||||
`UploadFiles` examples also use:
|
||||
|
||||
```go
|
||||
import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
|
||||
```
|
||||
|
||||
The canonical HTTP wire contract is [HTTP Upload API Contract](../integrations/http-upload.md).
|
||||
|
||||
## Client Construction
|
||||
|
||||
```go
|
||||
client, err := upload.NewClient(upload.ClientOptions{
|
||||
Endpoint: "https://distributor.example.com",
|
||||
Token: token,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
```
|
||||
|
||||
`Endpoint` is the distributor server base URL. The client derives `/upload` and `/runs/<run-id>`. `Token` is required and is sent as `Authorization: Bearer <token>`. Token values are redacted from client errors.
|
||||
|
||||
`HTTPClient` and `Retry` are optional. Defaults use a 30 second HTTP timeout and safe retry settings.
|
||||
|
||||
## Upload Producer Files
|
||||
|
||||
Use `UploadFiles` when the producer has generated output files but has not assembled a bundle directory.
|
||||
|
||||
```go
|
||||
result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
|
||||
ID: "weather.hourly.brentwood.2026-06-07T15",
|
||||
IdempotencyKey: "weather.hourly.brentwood.2026-06-07T15",
|
||||
Files: []bundle.BundleFile{
|
||||
{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
|
||||
{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_ = result.RunID
|
||||
```
|
||||
|
||||
`UploadFiles` creates a temporary bundle, writes and validates a manifest, uploads the archive, and removes temporary files when the call returns. It does not write into producer source directories.
|
||||
|
||||
## Upload An Existing Bundle
|
||||
|
||||
Use `UploadBundle` when the producer already has a complete local bundle root containing `manifest.json`.
|
||||
|
||||
```go
|
||||
result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
|
||||
Root: "/var/spool/weather/hourly-2026-06-07T15",
|
||||
IdempotencyKey: "weather.hourly.brentwood.2026-06-07T15",
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_ = result.RunID
|
||||
```
|
||||
|
||||
`UploadBundle` validates the local bundle by default and uploads only `manifest.json` plus manifest-listed files. Unlisted files are not uploaded.
|
||||
|
||||
## Result And Status
|
||||
|
||||
Upload success means the server returned `202 Accepted` after staging and validating the upload. It does not mean all configured destinations have published.
|
||||
|
||||
Poll status while the server retains the in-memory run record:
|
||||
|
||||
```go
|
||||
status, err := client.Status(ctx, result.RunID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if status.Status == "failed" {
|
||||
return fmt.Errorf("distributor run failed: %s", status.Error)
|
||||
}
|
||||
```
|
||||
|
||||
Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to `server.http.retention`; server restart clears run status and idempotency records.
|
||||
|
||||
## Idempotency And Retry
|
||||
|
||||
Every upload request includes `Idempotency-Key`.
|
||||
|
||||
If `IdempotencyKey` is omitted, the client generates a random 128-bit lowercase hexadecimal key for that upload operation and reuses it for retries within the same call. For cross-process retry safety, producers should pass a stable key derived from the producer job or report id.
|
||||
|
||||
The client retries only safe cases:
|
||||
|
||||
- `503 Service Unavailable`;
|
||||
- temporary network errors;
|
||||
- ambiguous mid-upload failures.
|
||||
|
||||
It does not retry after `202 Accepted` and does not retry `400`, `401`, `409`, `413`, or `415`.
|
||||
|
||||
Detect conflicting key reuse with `errors.As`:
|
||||
|
||||
```go
|
||||
var conflict *upload.IdempotencyConflictError
|
||||
if errors.As(err, &conflict) {
|
||||
return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
|
||||
}
|
||||
```
|
||||
|
||||
## Boundaries
|
||||
|
||||
`pkg/upload` does not configure server pipelines, choose destinations, wait for publication completion automatically, persist client queues, provide durable idempotency across server restarts, or expose destination state. It submits complete source bundles to the configured HTTP upload API.
|
||||
@@ -53,7 +53,8 @@ The adapter sends these query parameters:
|
||||
- `tz`: from `weather_api.timezone` on hourly forecast, narrative forecast, and
|
||||
discussion requests
|
||||
|
||||
Alerts do not receive `precision` or `tz`.
|
||||
Alerts do not receive `precision` or `tz`. Weather story requests receive only
|
||||
`format=json`.
|
||||
|
||||
## Endpoints Used
|
||||
|
||||
@@ -65,6 +66,7 @@ The adapter fetches these endpoints once per bundle:
|
||||
- `/forecast/narrative`
|
||||
- `/alerts/active`
|
||||
- `/discussion`
|
||||
- `/weatherstories/latest`
|
||||
|
||||
`weatherreporter` does not call day-slice forecast endpoints or discussion
|
||||
subsection endpoints. Report-period selection and daypart summarization happen
|
||||
@@ -86,10 +88,11 @@ source-specific `missing_source.sources` policy:
|
||||
- `narrative` for `/forecast/narrative`
|
||||
- `alerts` for `/alerts/active`
|
||||
- `discussion` for `/discussion`
|
||||
- `weather_story` for `/weatherstories/latest`
|
||||
|
||||
The adapter also creates missing stub source records for `daily` and
|
||||
`weather_story` because those source slots exist in the internal bundle but are
|
||||
not fetched from the Weather API.
|
||||
The adapter also creates a missing stub source record for `daily` because that
|
||||
source slot exists in the internal bundle but is not fetched from the Weather
|
||||
API.
|
||||
|
||||
Policy behavior:
|
||||
|
||||
@@ -126,6 +129,8 @@ types in `internal/forecast/bundle.go`, including:
|
||||
- forecast run metadata and `periods`
|
||||
- active alert run data
|
||||
- discussion metadata, key messages, and short/long-term section text
|
||||
- latest weather story title, description, timing, priority, order, alt text,
|
||||
and download URL
|
||||
|
||||
The adapter intentionally keeps upstream transport and envelope details inside
|
||||
`internal/adapters/weatherapi`; downstream packages consume the normalized
|
||||
|
||||
@@ -7,8 +7,9 @@ This document describes the implemented workflow coordinator in `internal/app`.
|
||||
`internal/app` coordinates the top-level use cases after CLI parsing and config
|
||||
loading are complete. It resolves report definitions, fetches weather data,
|
||||
builds briefing and prompt-input artifacts, invokes Scriptorium through the
|
||||
adapter boundary, persists managed state, runs batches, and reads existing
|
||||
artifacts for inspection.
|
||||
adapter boundary, optionally notifies distributor through an app-owned notifier
|
||||
boundary, persists managed state, runs batches, and reads existing artifacts for
|
||||
inspection.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
@@ -22,13 +23,15 @@ Inputs:
|
||||
- resolved report definitions from `internal/report`
|
||||
- forecast bundles from `internal/adapters/weatherapi`
|
||||
- prior snapshots loaded from `internal/state`
|
||||
- optional renderer and state-store fakes for tests
|
||||
- optional renderer, notifier, and state-store fakes for tests
|
||||
|
||||
Outputs:
|
||||
|
||||
- generated report results with briefing, data package, preflight, report,
|
||||
metadata, prior snapshot, Recent Changes, and Scriptorium result details
|
||||
- batch summaries with per-report status, artifact paths, and error text
|
||||
metadata, prior snapshot, Recent Changes, Scriptorium result details, and
|
||||
notification result when attempted
|
||||
- batch summaries with per-report status, artifact paths, error text, and
|
||||
notification outcome when attempted
|
||||
- saved Weather API bundle JSON for fetch workflows
|
||||
- inspection JSON values for reports, metadata, briefings, data packages, prior
|
||||
snapshots, and source provenance
|
||||
@@ -42,8 +45,9 @@ Scriptorium argv.
|
||||
|
||||
Report selection and report identity policy come from `internal/report`.
|
||||
Weather API transport stays in `internal/adapters/weatherapi`. Scriptorium
|
||||
subprocess behavior stays in `internal/adapters/scriptorium`. Filesystem layout
|
||||
and persisted metadata stay in `internal/state`.
|
||||
subprocess behavior stays in `internal/adapters/scriptorium`. Distributor
|
||||
upload behavior stays in `internal/adapters/distributor`. Filesystem layout and
|
||||
persisted metadata stay in `internal/state`.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
@@ -52,6 +56,7 @@ and persisted metadata stay in `internal/state`.
|
||||
- `workspace.*` for filesystem state
|
||||
- `dayparts` for daily and outlook summarization
|
||||
- `recent_change.*` for structured Recent Changes thresholds
|
||||
- `notify.distributor.*` for optional notification after report generation
|
||||
|
||||
Output copy flags are command request fields. They are not configuration
|
||||
defaults.
|
||||
@@ -74,11 +79,15 @@ Single-report generation follows this order:
|
||||
12. Run Scriptorium report generation to the managed report path.
|
||||
13. Copy the managed report to the requested `--out` path when provided.
|
||||
14. Save metadata with the managed report path.
|
||||
15. If distributor notification is enabled, notify using the managed report
|
||||
path as the source file.
|
||||
|
||||
If render preflight returns both a result and an error, preflight JSON and
|
||||
metadata are persisted before the error is returned. If Scriptorium report
|
||||
generation returns an error after writing output, the managed report and
|
||||
metadata remain inspectable.
|
||||
metadata remain inspectable. Notification is not attempted after Weather API,
|
||||
briefing, prompt input, render, Scriptorium run, or metadata-save failures.
|
||||
`--out` copies are never used as notification source files.
|
||||
|
||||
## Batch Workflow
|
||||
|
||||
@@ -87,6 +96,10 @@ on Sunday. `run evening` resolves Daily Tomorrow. Batch output copy names come
|
||||
from report definitions. Batch generation continues independent reports after a
|
||||
failure, records each result, writes compact status lines to stderr, emits a
|
||||
JSON summary to stdout, and returns an aggregate error when any report failed.
|
||||
When notification is enabled, each successfully generated report is notified
|
||||
independently. Notification failure marks that report failed, records
|
||||
notification fields in the batch result, and does not stop later reports.
|
||||
`--out-dir` copies are never used as notification source files.
|
||||
|
||||
## Inspection Workflow
|
||||
|
||||
@@ -101,6 +114,8 @@ inspection view.
|
||||
- Weather API and briefing errors stop that report before Scriptorium runs.
|
||||
- Prompt input validation fails before render preflight.
|
||||
- Render and run errors preserve Scriptorium stderr and exit-code context.
|
||||
- Notification errors are wrapped with report ID, RunID, and managed report path
|
||||
context and are recorded separately in batch results.
|
||||
- Metadata and artifact path errors include filesystem context.
|
||||
- Batch failures are recorded per report and surfaced through an aggregate
|
||||
batch error.
|
||||
@@ -121,3 +136,5 @@ Inspect:
|
||||
- Render preflight precedes Scriptorium report generation.
|
||||
- Recent Changes are computed from structured briefing snapshots.
|
||||
- Metadata links artifacts produced for a run.
|
||||
- Distributor notification uses the managed Markdown report path, not extra
|
||||
output copies.
|
||||
|
||||
@@ -24,6 +24,8 @@ Outputs:
|
||||
object for Daily, 3-Day, Weekend, or Storm Report
|
||||
- optional `currentConditions` prompt context from normalized
|
||||
`/conditions/current` data when available
|
||||
- optional structured `weatherStory` context on report-specific briefing
|
||||
objects when `/weatherstories/latest` is available
|
||||
- optional JSON file written by `briefing.Save`
|
||||
|
||||
## Boundaries
|
||||
|
||||
106
docs/internal/distributor-adapter.md
Normal file
106
docs/internal/distributor-adapter.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# Distributor Adapter Internals
|
||||
|
||||
This document describes the distributor upload adapter in
|
||||
`internal/adapters/distributor`.
|
||||
|
||||
## Purpose
|
||||
|
||||
The adapter submits generated weatherreporter Markdown reports to a configured
|
||||
distributor HTTP upload endpoint. It isolates distributor package types,
|
||||
token-env lookup, upload client construction, source-bundle file mapping,
|
||||
timeout handling, and upload error wrapping from app orchestration.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs:
|
||||
|
||||
- distributor endpoint URL
|
||||
- token environment variable name
|
||||
- upload timeout
|
||||
- bundle ID
|
||||
- idempotency key
|
||||
- source Markdown report path
|
||||
- bundle-relative Markdown path
|
||||
- context for cancellation
|
||||
|
||||
Outputs:
|
||||
|
||||
- accepted distributor run ID
|
||||
- accepted distributor status
|
||||
- weatherreporter-owned idempotency conflict error when applicable
|
||||
|
||||
## Boundaries
|
||||
|
||||
`internal/adapters/distributor` is the only weatherreporter package that imports
|
||||
`gitea.maximumdirect.net/eric/distributor/pkg/upload` or
|
||||
`gitea.maximumdirect.net/eric/distributor/pkg/bundle`.
|
||||
|
||||
The app layer passes weatherreporter-owned request values to the adapter. The
|
||||
adapter does not choose report types, render templates, select output copies,
|
||||
configure destinations, wait for downstream publication, transform Markdown, or
|
||||
persist notification state.
|
||||
|
||||
Full upstream distributor package and HTTP contract details stay under
|
||||
`docs/integrations/distributor/`.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
The adapter is built from `notify.distributor` config:
|
||||
|
||||
- `endpoint`
|
||||
- `token_env`
|
||||
- `timeout`
|
||||
|
||||
The app layer renders bundle ID, idempotency key, and bundle path from:
|
||||
|
||||
- `bundle_id_template`
|
||||
- `idempotency_key_template`
|
||||
- `report_path_template`
|
||||
|
||||
The token value is read from the environment variable named by `token_env`
|
||||
after config loading and `secrets.directory` processing.
|
||||
|
||||
## Upload Behavior
|
||||
|
||||
The adapter calls distributor `UploadFiles` with exactly one file:
|
||||
|
||||
- source path: the managed Markdown report path selected by app orchestration
|
||||
- bundle path: the rendered bundle-relative report path
|
||||
|
||||
The adapter creates a distributor upload client with the configured endpoint,
|
||||
bearer token, and timeout-backed HTTP client. It also wraps the upload context
|
||||
with the configured timeout when the timeout is greater than zero.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
The adapter validates required endpoint, token env name, token value, bundle ID,
|
||||
idempotency key, source path, bundle path, and upload client inputs before
|
||||
uploading.
|
||||
|
||||
Upload failures include endpoint, bundle ID, idempotency key, source path, and
|
||||
bundle path context. Token values are redacted from adapter errors.
|
||||
|
||||
Distributor idempotency conflicts are exposed as a weatherreporter-owned
|
||||
`IdempotencyConflictError`, so callers do not depend on upstream distributor
|
||||
types.
|
||||
|
||||
## Tests
|
||||
|
||||
Inspect:
|
||||
|
||||
- `internal/adapters/distributor/client_test.go`
|
||||
- `internal/app/app_test.go`
|
||||
- `internal/cli/root_test.go`
|
||||
|
||||
Adapter tests use a fake upload client factory and do not require a live
|
||||
distributor service.
|
||||
|
||||
## Invariants
|
||||
|
||||
- Distributor package types do not leak outside the adapter.
|
||||
- Only the managed Markdown report is uploaded.
|
||||
- The adapter never scans the workspace.
|
||||
- Token values are not included in errors, CLI output, metadata, docs, or
|
||||
examples.
|
||||
- Destination routing and Markdown-to-HTML transformation belong to
|
||||
distributor, not weatherreporter.
|
||||
@@ -19,8 +19,8 @@ Outputs:
|
||||
- `promptinput.Package` containing schema version, RunID, report metadata,
|
||||
briefing content, Recent Changes, and source warnings. Briefing content
|
||||
includes configured location context, current conditions when available,
|
||||
discussion key messages, and short/long-term AFD narratives when the Weather
|
||||
API provides them.
|
||||
structured weather story context when available, discussion key messages, and
|
||||
short/long-term AFD narratives when the Weather API provides them.
|
||||
- report metadata includes `currentLocalDate`, the generation date formatted as
|
||||
`YYYY-MM-DD` in the effective report timezone.
|
||||
- optional JSON file written by `promptinput.Save`
|
||||
|
||||
@@ -7,7 +7,7 @@ This document describes Weather API ingestion into `forecast.Bundle`.
|
||||
`internal/adapters/weatherapi` fetches normalized weather data from one
|
||||
configured Weather API endpoint and assembles the bundle consumed by forecast
|
||||
derivation and briefing builders. Briefing builders expose normalized current
|
||||
conditions as prompt context when `/conditions/current` is available.
|
||||
conditions and weather story context when those sources are available.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
@@ -20,9 +20,9 @@ Inputs:
|
||||
Outputs:
|
||||
|
||||
- `forecast.Bundle` with observation, current conditions, hourly forecast,
|
||||
narrative forecast, active alerts, discussion, source records, and source
|
||||
warnings
|
||||
- stub source records for daily forecast and weather story source slots
|
||||
narrative forecast, active alerts, discussion, latest weather story, source
|
||||
records, and source warnings
|
||||
- stub source record for the daily forecast source slot
|
||||
- optional saved bundle JSON through app fetch helpers
|
||||
|
||||
## Boundaries
|
||||
|
||||
@@ -19,7 +19,10 @@ weatherreporter generate storm --start 2026-05-29T18:00 --end 2026-05-30T06:00
|
||||
Each command resolves a report period, fetches a Weather API bundle, builds a
|
||||
briefing, builds a prompt input data package, runs `scriptorium render`, runs
|
||||
`scriptorium run`, and writes managed artifacts under the configured workspace.
|
||||
`--out PATH` writes an extra Markdown copy for the current generated report.
|
||||
When distributor notification is enabled, weatherreporter uploads the managed
|
||||
Markdown report after `scriptorium run` succeeds and final metadata is saved.
|
||||
`--out PATH` writes an extra Markdown copy for the current generated report; it
|
||||
is not used as the distributor upload source.
|
||||
|
||||
Implemented batch commands:
|
||||
|
||||
@@ -32,9 +35,12 @@ weatherreporter run evening
|
||||
except on Sunday. `run evening` generates the Tomorrow Planning Brief. Batch
|
||||
commands print a JSON summary to stdout, write compact per-report status lines
|
||||
to stderr, continue independent reports after one report fails, and return
|
||||
nonzero when any report failed. `--out-dir PATH` writes extra Markdown copies
|
||||
using report default filenames such as `daily.md`, `three-day.md`,
|
||||
`weekend.md`, and `tomorrow.md`.
|
||||
nonzero when any report failed. When notification is configured, the summary and
|
||||
status lines include notification status, accepted distributor run ID, or
|
||||
notification error fields for each attempted report. `--out-dir PATH` writes
|
||||
extra Markdown copies using report default filenames such as `daily.md`,
|
||||
`three-day.md`, `weekend.md`, and `tomorrow.md`; these copies are not used as
|
||||
distributor upload sources.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
@@ -117,8 +123,40 @@ Each generated report writes metadata that links:
|
||||
- preflight output path
|
||||
- managed Markdown report path
|
||||
|
||||
Batch summaries include report status, error text when applicable, valid
|
||||
period, and known artifact paths for each attempted report.
|
||||
Batch summaries include report status, error text when applicable, notification
|
||||
outcome when attempted, valid period, and known artifact paths for each
|
||||
attempted report. Notification fields are `notificationStatus`,
|
||||
`notificationRunId`, and `notificationError`.
|
||||
|
||||
## Distributor Notification
|
||||
|
||||
Distributor notification is configured with `notify.distributor` and is disabled
|
||||
by default. When enabled, weatherreporter uploads exactly one file per
|
||||
successfully generated report: the managed Markdown report path recorded in the
|
||||
report result and metadata. Extra copies written by `--out` or `--out-dir` are
|
||||
operator conveniences only.
|
||||
|
||||
The default bundle ID is derived from producer name, location ID, report ID, and
|
||||
RunID:
|
||||
|
||||
```text
|
||||
weatherreporter.{location_id}.{report_id}.{run_id}
|
||||
```
|
||||
|
||||
The default idempotency key is the rendered bundle ID. The default bundle path
|
||||
for the Markdown file is the report definition's batch output name, such as
|
||||
`daily.md`, `tomorrow.md`, `three-day.md`, or `weekend.md`.
|
||||
|
||||
Notification happens after final metadata save. Weather API, briefing,
|
||||
data-package, render preflight, Scriptorium run, and metadata-save failures do
|
||||
not trigger notification. A notification failure fails that report. In a batch,
|
||||
other reports continue, the failed report includes notification fields in the
|
||||
JSON summary, and the batch returns nonzero.
|
||||
|
||||
Weatherreporter is responsible for selecting the managed Markdown report,
|
||||
constructing a source bundle, and submitting it to the configured distributor
|
||||
HTTP endpoint. Distributor remains responsible for destination routing,
|
||||
publication, and any downstream Markdown-to-HTML transformation.
|
||||
|
||||
## Inspection
|
||||
|
||||
@@ -164,6 +202,8 @@ A failed generation run may still leave useful artifacts:
|
||||
preflight JSON and metadata are written for inspection.
|
||||
- If `scriptorium run` exits nonzero after writing a report, the managed report
|
||||
and metadata remain available.
|
||||
- If distributor notification fails, report artifacts and final metadata remain
|
||||
available, but the report or batch command returns nonzero.
|
||||
- For batch commands, inspect the stdout JSON summary first, then inspect the
|
||||
artifact paths for each failed report.
|
||||
|
||||
|
||||
@@ -13,6 +13,12 @@ Generated reports must be associated with explicit metadata, including report ty
|
||||
|
||||
`scriptorium` is an external adapter, not domain logic. Subprocess execution must be isolated under `internal/adapters/scriptorium`, use context-aware execution, avoid shell interpolation, capture actionable stderr, and keep scriptorium-specific flags from leaking into domain packages.
|
||||
|
||||
`distributor` is also an external adapter. Upload behavior must be isolated
|
||||
under `internal/adapters/distributor`, dependency types from the distributor
|
||||
module must not leak outside that adapter, and the selected upload source must
|
||||
be the managed Markdown report rather than optional output copies or broad
|
||||
workspace scans.
|
||||
|
||||
## Project Shape
|
||||
|
||||
Default to a small, explicit, dependency-light Go application. Keep the design modular enough to test and change safely, but do not add abstraction unless it protects a real boundary or enables a real extension point.
|
||||
@@ -54,7 +60,7 @@ Configuration precedence is:
|
||||
|
||||
Prefer YAML configuration unless the project has a strong reason to use another format. Config files should be discovered at `/usr/local/etc/<app_name>/config.yml`, with a CLI override via `--config`.
|
||||
|
||||
Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables or secret files for secrets.
|
||||
Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables or secret files for secrets. File-backed secrets are loaded through `secrets.directory`; secret values must not be logged, persisted, or included in user-facing output.
|
||||
|
||||
## Adapters and External Integrations
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ Developers and LLM coding agents should use it with
|
||||
- `internal/config`: configuration structs, defaults, loading, overrides, and
|
||||
validation.
|
||||
- `internal/fileutil`: shared atomic filesystem write and copy helpers.
|
||||
- `internal/adapters/distributor`: Distributor upload adapter.
|
||||
- `internal/adapters/weatherapi`: Weather API HTTP adapter.
|
||||
- `internal/adapters/scriptorium`: Scriptorium subprocess adapter.
|
||||
- `internal/forecast`: normalized bundle types and deterministic forecast
|
||||
@@ -45,7 +46,7 @@ Useful focused checks:
|
||||
```bash
|
||||
go test ./internal/cli ./internal/config
|
||||
go test ./internal/app ./internal/state
|
||||
go test ./internal/adapters/weatherapi ./internal/adapters/scriptorium
|
||||
go test ./internal/adapters/distributor ./internal/adapters/weatherapi ./internal/adapters/scriptorium
|
||||
go test ./internal/forecast ./internal/report ./internal/briefing ./internal/changes ./internal/promptinput
|
||||
```
|
||||
|
||||
@@ -64,6 +65,8 @@ Run `gofmt -w` on changed Go files before committing.
|
||||
- Use atomic writes for durable JSON artifacts where practical.
|
||||
- Keep report selection and prompt IDs centralized in `internal/report`.
|
||||
- Keep Scriptorium argv construction inside `internal/adapters/scriptorium`.
|
||||
- Keep distributor package types and upload-client construction inside
|
||||
`internal/adapters/distributor`.
|
||||
- Keep Weather API transport and envelope handling inside
|
||||
`internal/adapters/weatherapi`.
|
||||
|
||||
@@ -72,8 +75,10 @@ Run `gofmt -w` on changed Go files before committing.
|
||||
Prefer the Go standard library. Add dependencies only when they materially
|
||||
improve correctness, interoperability, security, or maintainability.
|
||||
|
||||
Current external dependency:
|
||||
Current external dependencies:
|
||||
|
||||
- `gitea.maximumdirect.net/eric/distributor` for distributor source bundle
|
||||
construction and HTTP upload client behavior.
|
||||
- `gopkg.in/yaml.v3` for YAML configuration parsing.
|
||||
|
||||
When adding a dependency:
|
||||
@@ -145,12 +150,14 @@ When an external contract changes, update the matching file under
|
||||
|
||||
## Tests
|
||||
|
||||
Core tests must not require live Weather API or Scriptorium services.
|
||||
Core tests must not require live Weather API, Scriptorium, or distributor
|
||||
services.
|
||||
|
||||
Preferred test patterns:
|
||||
|
||||
- fake command runners for subprocess behavior;
|
||||
- `httptest.Server` for Weather API behavior;
|
||||
- fake distributor upload clients for notification behavior;
|
||||
- filesystem temp directories for state behavior;
|
||||
- deterministic clocks for report periods and RunIDs;
|
||||
- table tests for config validation, CLI parsing, period resolution, and
|
||||
@@ -187,7 +194,8 @@ Update:
|
||||
behavior;
|
||||
- `docs/troubleshooting.md` for recurring operator-facing failure modes;
|
||||
- `docs/internal/` for component contracts and invariants;
|
||||
- `docs/integrations/` for external Weather API or Scriptorium contract changes;
|
||||
- `docs/integrations/` for external Weather API, Scriptorium, or distributor
|
||||
contract changes;
|
||||
- `docs/roadmap/` only for unimplemented or deferred work.
|
||||
|
||||
Non-roadmap docs must describe implemented behavior only.
|
||||
|
||||
34
docs/roadmap/distributor.md
Normal file
34
docs/roadmap/distributor.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# Distributor Roadmap
|
||||
|
||||
Current distributor notification behavior is documented outside the roadmap:
|
||||
|
||||
- [Configuration reference](../config.md)
|
||||
- [Operations guide](../operations.md)
|
||||
- [Troubleshooting](../troubleshooting.md)
|
||||
- [Distributor adapter internals](../internal/distributor-adapter.md)
|
||||
|
||||
This file tracks future distributor-related work only.
|
||||
|
||||
## Deferred Enhancements
|
||||
|
||||
- Add a supported warning-only notification policy.
|
||||
- Include selected non-report artifacts in uploaded bundles.
|
||||
- Poll distributor run status after upload acceptance.
|
||||
- Persist upload retry state across process restarts.
|
||||
- Add explicit CLI controls for distributor behavior.
|
||||
|
||||
## Non-Goals Without A Separate Design
|
||||
|
||||
- Do not make distributor scan the weatherreporter workspace.
|
||||
- Do not move destination routing into weatherreporter.
|
||||
- Do not move Markdown-to-HTML transformation into weatherreporter.
|
||||
- Do not store raw bearer tokens in configuration files.
|
||||
|
||||
## Required Constraints For Future Work
|
||||
|
||||
- Distributor package types stay inside `internal/adapters/distributor`.
|
||||
- Weatherreporter submits explicit source bundles built from generated files.
|
||||
- Optional `--out` and `--out-dir` copies remain operator conveniences, not
|
||||
canonical upload sources.
|
||||
- Secret values stay out of errors, logs, CLI output, metadata, examples, and
|
||||
documentation.
|
||||
@@ -12,8 +12,7 @@ evaluation remains deferred.
|
||||
Proposed direction:
|
||||
|
||||
1. detect candidate events deterministically from alerts, forecast discussion,
|
||||
weather story context when available, hourly thresholds, and material
|
||||
forecast changes;
|
||||
weather story context, hourly thresholds, and material forecast changes;
|
||||
2. evaluate candidates through Scriptorium or another narrow evaluator adapter;
|
||||
3. persist storm lifecycle state;
|
||||
4. generate or update Storm Reports only when a meaningful event is present;
|
||||
@@ -50,6 +49,26 @@ These ideas are not current behavior:
|
||||
Each item needs its own design note before implementation. Non-roadmap docs
|
||||
must not describe these as available behavior.
|
||||
|
||||
## Deferred: Distributor Notification Enhancements
|
||||
|
||||
Distributor notification currently uploads one managed Markdown report per
|
||||
successful generated report through the configured HTTP upload endpoint.
|
||||
|
||||
These enhancements are not current behavior:
|
||||
|
||||
- `failure_policy: warn`;
|
||||
- uploading metadata, briefing snapshots, data packages, or preflight artifacts;
|
||||
- polling distributor status after upload acceptance;
|
||||
- durable upload retry queues;
|
||||
- distributor-specific CLI flags;
|
||||
- making distributor scan the weatherreporter workspace;
|
||||
- handling destination routing, Markdown-to-HTML transformation, public URLs,
|
||||
or nginx layout inside weatherreporter.
|
||||
|
||||
Any distributor enhancement should preserve the existing adapter boundary:
|
||||
weatherreporter selects explicit generated files and submits source bundles,
|
||||
while distributor owns destination routing and publication behavior.
|
||||
|
||||
## Deferred: Cleanup Refactors
|
||||
|
||||
The initial cleanup pass intentionally left these refactors out because the
|
||||
|
||||
36
docs/roadmap/implementation.md
Normal file
36
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,36 @@
|
||||
# Distributor Follow-Up Roadmap
|
||||
|
||||
Current distributor notification behavior is documented in:
|
||||
|
||||
- [Configuration reference](../config.md)
|
||||
- [Operations guide](../operations.md)
|
||||
- [Troubleshooting](../troubleshooting.md)
|
||||
- [App orchestration internals](../internal/app-orchestration.md)
|
||||
- [Distributor adapter internals](../internal/distributor-adapter.md)
|
||||
- [Distributor integration contracts](../integrations/distributor/api.md)
|
||||
|
||||
This roadmap tracks only future work that is not implemented.
|
||||
|
||||
## Deferred Enhancements
|
||||
|
||||
- Support a non-failing distributor notification policy such as
|
||||
`failure_policy: warn`.
|
||||
- Upload additional generated artifacts, such as metadata, briefing snapshots,
|
||||
data packages, or preflight output.
|
||||
- Poll distributor status after upload acceptance and expose downstream
|
||||
publication failures.
|
||||
- Add durable retry queues for upload failures.
|
||||
- Add distributor-specific CLI controls if operators need per-run behavior that
|
||||
configuration cannot cover cleanly.
|
||||
|
||||
## Boundaries To Preserve
|
||||
|
||||
- Keep public CLI syntax stable unless a separate CLI design changes it.
|
||||
- Keep managed workspace artifact paths and RunIDs stable.
|
||||
- Keep distributor dependency types behind `internal/adapters/distributor`.
|
||||
- Keep raw token values out of errors, CLI output, metadata, docs, examples,
|
||||
and persisted artifacts.
|
||||
- Upload explicitly selected generated files; do not make distributor scan the
|
||||
weatherreporter workspace.
|
||||
- Leave destination routing, Markdown-to-HTML transformation, public URLs, and
|
||||
nginx layout to distributor.
|
||||
@@ -199,6 +199,101 @@ metadata, sources, briefing, and data package for that RunID.
|
||||
|
||||
Relevant docs: [CLI reference](cli.md), [Operations guide](operations.md).
|
||||
|
||||
## Invalid Secrets Directory
|
||||
|
||||
Symptom: config loading fails with `read secrets directory`, `secret file`, or
|
||||
environment variable name context.
|
||||
|
||||
Likely cause: `secrets.directory` points to a missing directory or contains an
|
||||
invalid entry. Secret entries must be regular files directly under the
|
||||
configured directory, and file basenames must match
|
||||
`[A-Za-z_][A-Za-z0-9_]*`.
|
||||
|
||||
Diagnostic: list the configured directory and inspect entry names and file
|
||||
types. Do not print secret file contents.
|
||||
|
||||
Safe fix: create the directory, remove subdirectories or symlinks, fix invalid
|
||||
filenames, and ensure the weatherreporter process can read each secret file.
|
||||
|
||||
Relevant docs: [Configuration reference](config.md).
|
||||
|
||||
## Distributor Token Is Missing
|
||||
|
||||
Symptom: notification fails with a message that the distributor token
|
||||
environment variable is not set.
|
||||
|
||||
Likely cause: `notify.distributor.enabled` is true, but the environment
|
||||
variable named by `notify.distributor.token_env` was not populated directly or
|
||||
through `secrets.directory`.
|
||||
|
||||
Diagnostic: check `notify.distributor.token_env`, then verify a matching secret
|
||||
file exists under `secrets.directory` or that the process environment includes
|
||||
the variable. Do not print the token value.
|
||||
|
||||
Safe fix: create a readable secret file whose basename matches `token_env`, or
|
||||
set the environment variable through the service manager.
|
||||
|
||||
Relevant docs: [Configuration reference](config.md),
|
||||
[Operations guide](operations.md).
|
||||
|
||||
## Distributor Upload Conflict
|
||||
|
||||
Symptom: notification fails with idempotency conflict context.
|
||||
|
||||
Likely cause: the same idempotency key was reused for different bundle content.
|
||||
By default the key derives from bundle ID, which includes location ID, report
|
||||
ID, and RunID.
|
||||
|
||||
Diagnostic: inspect the failed batch JSON or stderr line for bundle and
|
||||
idempotency context. Compare the configured templates with the report RunID and
|
||||
report path.
|
||||
|
||||
Safe fix: keep idempotency templates stable for retries of the same generated
|
||||
report, but do not reuse the same rendered key for different generated report
|
||||
content.
|
||||
|
||||
Relevant docs: [Operations guide](operations.md),
|
||||
[Distributor adapter internals](internal/distributor-adapter.md).
|
||||
|
||||
## Distributor Upload Rejected
|
||||
|
||||
Symptom: notification fails with distributor upload rejection, HTTP status, or
|
||||
bundle validation context.
|
||||
|
||||
Likely cause: the distributor endpoint rejected the token, bundle ID,
|
||||
idempotency key, source file, or bundle path.
|
||||
|
||||
Diagnostic: inspect stdout JSON or stderr status lines for
|
||||
`notificationError`. Confirm `notify.distributor.endpoint`,
|
||||
`notify.distributor.report_path_template`, and token configuration. Token values
|
||||
are redacted from weatherreporter errors.
|
||||
|
||||
Safe fix: fix the endpoint, token, templates, or distributor-side upload
|
||||
configuration. The weatherreporter upload source is the managed Markdown report,
|
||||
not `--out` or `--out-dir` copies.
|
||||
|
||||
Relevant docs: [Configuration reference](config.md),
|
||||
[Operations guide](operations.md),
|
||||
[Distributor adapter internals](internal/distributor-adapter.md).
|
||||
|
||||
## Distributor Unavailable
|
||||
|
||||
Symptom: notification fails with network, timeout, or service unavailable
|
||||
context.
|
||||
|
||||
Likely cause: the configured distributor endpoint is unreachable, slow, or
|
||||
temporarily unavailable.
|
||||
|
||||
Diagnostic: check network access from the weatherreporter host to
|
||||
`notify.distributor.endpoint`. For batch runs, inspect which reports have
|
||||
`notificationStatus: "failed"`.
|
||||
|
||||
Safe fix: restore distributor service availability and rerun the affected
|
||||
report or batch. Stable idempotency keys make retrying the same generated report
|
||||
safe unless the distributor reports a conflict.
|
||||
|
||||
Relevant docs: [Operations guide](operations.md).
|
||||
|
||||
## Unknown RunID
|
||||
|
||||
Symptom: an inspect command fails with `metadata for run id ... was not found`.
|
||||
|
||||
@@ -11,6 +11,20 @@ location:
|
||||
name: Brentwood
|
||||
region: St. Louis Metro
|
||||
|
||||
secrets:
|
||||
directory: ""
|
||||
|
||||
notify:
|
||||
distributor:
|
||||
enabled: false
|
||||
endpoint: https://distributor.example.com
|
||||
token_env: DISTRIBUTOR_UPLOAD_TOKEN
|
||||
timeout: 30s
|
||||
failure_policy: error
|
||||
bundle_id_template: "weatherreporter.{location_id}.{report_id}.{run_id}"
|
||||
idempotency_key_template: "{bundle_id}"
|
||||
report_path_template: "{batch_output_name}"
|
||||
|
||||
missing_source:
|
||||
default: warn
|
||||
sources:
|
||||
|
||||
2
go.mod
2
go.mod
@@ -3,3 +3,5 @@ module gitea.maximumdirect.net/eric/weatherreporter
|
||||
go 1.26
|
||||
|
||||
require gopkg.in/yaml.v3 v3.0.1
|
||||
|
||||
require gitea.maximumdirect.net/eric/distributor v0.4.0
|
||||
|
||||
48
go.sum
48
go.sum
@@ -1,3 +1,51 @@
|
||||
gitea.maximumdirect.net/eric/distributor v0.4.0 h1:SRrTFjVLMv4wFZmMLwnaYtJaQlQ/wsB4OzcaSEEb524=
|
||||
gitea.maximumdirect.net/eric/distributor v0.4.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11/go.mod h1:dnakxebH6UwFvcvujL0LVggYQ8nEvBGjU4G/V79Nv94=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.20 h1:8VMDnWc/kEzxsI/1ngGM9mG81a8IGmIHD8KLcYGwagc=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.20/go.mod h1:PuwEpciweIXGULWeOeSTXtSbH4CW9mWdWrhdCKQI1sM=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.19 h1:yuFzSV1U0aRNYCQGVaTY2zW2M/L93pYHnXnrJUphYhU=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.19/go.mod h1:7y63L1kGzeoDlJaQ3Z578KrnmfBut96JjvJUzGwR+YE=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.25 h1:0w6dCiO8iez+YKwRhRBlL1CH/E3GTfdkuzrwj1by8vo=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.25/go.mod h1:9FDWUothyr5RCRAHc45XOiVCzUR8n/IhCYX+uVqw6vk=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.25 h1:Uii3frf9ztec/ABM2/FSH9/z7PLzxfpG8h4RpkUFflQ=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.25/go.mod h1:G6kntsA2GorAxDPbap6xgB2F+amSLUF8GJTi7PUoX44=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.25 h1:r1+/l6m+WaUJF9HISEsNOLHSNj5EXYQxK8VX6Cz9NlA=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.25/go.mod h1:cKf+D+NMDK1LndD7BowHbBZPgR9V0/5HubH0PFWvA+c=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.26 h1:A1PmWU2zfkIm9EyFlJncFXL4W4phML+h8KjltUsCvNQ=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.26/go.mod h1:dY4MRzXEizrD4hqtpKvWVGPX7QleSGGVY+EBolo1RmM=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.10 h1:d5/908OJ4bXg8lyjeMPvXetEKqoDoLi5Owy1zNue3yg=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.10/go.mod h1:a57l7Hwh+FWI+we50g5NPJHYUKeJKfXbc4w8SyXu8Ig=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.18 h1:W/EyPFl9A5rXrtoilfwHYEvzHER+K4SpBPtMXi24Mos=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.18/go.mod h1:UG50K+pvd/uy6xExbobg0rjqFBFZe6I3l75EPDZw4tg=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.25 h1:dD3dhHNglpd98gs72my22Ndqi1hqQGllFFg1F+twfxg=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.25/go.mod h1:0yAbjPfd64gG7mj85RW+fMEYdfBgCRZw8g/oWcL1pjc=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.25 h1:2pQEbwf+/6EDbiit/GcBE2K4IUpMZymaA0kOz3xK978=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.25/go.mod h1:KvT6NCcQ0EZ+ZkVRrlBMt04Po3ok23YELEp7WimhLhM=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.102.2 h1:ie4ElCmUKS26pzrZcIk/lmt4yWjAqLLcawstyQCh298=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.102.2/go.mod h1:zjsomFeX5duj+4PlMB+o4JoWTIx+G0XMyzjYrUbQkN0=
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.1.1 h1:1VwbP3qMNfxUDEXWki4rCE5iA+44VA1lokTz9HasGzw=
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.1.1/go.mod h1:vUtyoSj0OPji3kjIVSc/GlKuWEiL33f/WFxl6dmpy/A=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.19 h1:N6pIsdFOW1Kd9S4KyFKXdGRBojPPxkP32+uHFWLv4Hc=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.19/go.mod h1:3gt5WJArFooNmyLONS+h/R4J+o86II8du38IgCwj9dE=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.2 h1:hc+lBYiiTr8Zk4MTzIsQ92MeDWCIDvWGmzKUWOaBcOg=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.2/go.mod h1:hU6fqB3OJA6/ePheD47LQnxvjYk6br6PtQxs+Q9ojvk=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.42.3 h1:ErklX/7uhSbkAAeyQD/Y1OoQ9hO3SJXQNEgksORW3Js=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.42.3/go.mod h1:ULe4HCzfKPiR6R3HEurE3b1upEkuk8AkMrOKtaOxKO8=
|
||||
github.com/aws/smithy-go v1.26.0 h1:9ouqbi+NyKP7fV3Te7UElCwdAb6Y8uk7LGwPE5tVe/s=
|
||||
github.com/aws/smithy-go v1.26.0/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
|
||||
github.com/kr/fs v0.1.0 h1:Jskdu9ieNAYnjxsi0LbQp1ulIKZV1LAFgK1tWhpZgl8=
|
||||
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
|
||||
github.com/pkg/sftp v1.13.10 h1:+5FbKNTe5Z9aspU88DPIKJ9z2KZoaGCu6Sr6kKR/5mU=
|
||||
github.com/pkg/sftp v1.13.10/go.mod h1:bJ1a7uDhrX/4OII+agvy28lzRvQrmIQuaHrcI1HbeGA=
|
||||
github.com/yuin/goldmark v1.8.2 h1:kEGpgqJXdgbkhcOgBxkC0X0PmoPG1ZyoZ117rDVp4zE=
|
||||
github.com/yuin/goldmark v1.8.2/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
|
||||
golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
|
||||
golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
|
||||
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
|
||||
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
|
||||
229
internal/adapters/distributor/client.go
Normal file
229
internal/adapters/distributor/client.go
Normal file
@@ -0,0 +1,229 @@
|
||||
// Package distributor adapts weatherreporter report artifacts to distributor uploads.
|
||||
package distributor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
distributorbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
|
||||
distributorupload "gitea.maximumdirect.net/eric/distributor/pkg/upload"
|
||||
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/config"
|
||||
)
|
||||
|
||||
type Client struct {
|
||||
Endpoint string
|
||||
TokenEnv string
|
||||
Timeout time.Duration
|
||||
newUploadClient uploadClientFactory
|
||||
}
|
||||
|
||||
type UploadRequest struct {
|
||||
BundleID string
|
||||
IdempotencyKey string
|
||||
SourcePath string
|
||||
BundlePath string
|
||||
}
|
||||
|
||||
type UploadResult struct {
|
||||
RunID string
|
||||
Status string
|
||||
}
|
||||
|
||||
type IdempotencyConflictError struct {
|
||||
Err error
|
||||
}
|
||||
|
||||
func (e *IdempotencyConflictError) Error() string {
|
||||
if e == nil || e.Err == nil {
|
||||
return "distributor idempotency conflict"
|
||||
}
|
||||
return e.Err.Error()
|
||||
}
|
||||
|
||||
func (e *IdempotencyConflictError) Unwrap() error {
|
||||
if e == nil {
|
||||
return nil
|
||||
}
|
||||
return e.Err
|
||||
}
|
||||
|
||||
type uploadClientFactory func(endpoint, token string, timeout time.Duration) (uploadClient, error)
|
||||
|
||||
type uploadClient interface {
|
||||
UploadFiles(ctx context.Context, opts uploadFilesOptions) (uploadFilesResult, error)
|
||||
}
|
||||
|
||||
type uploadFilesOptions struct {
|
||||
BundleID string
|
||||
IdempotencyKey string
|
||||
SourcePath string
|
||||
BundlePath string
|
||||
}
|
||||
|
||||
type uploadFilesResult struct {
|
||||
RunID string
|
||||
Status string
|
||||
}
|
||||
|
||||
func New(cfg config.DistributorNotifyConfig) *Client {
|
||||
return newClient(cfg, newDistributorUploadClient)
|
||||
}
|
||||
|
||||
func newClient(cfg config.DistributorNotifyConfig, factory uploadClientFactory) *Client {
|
||||
if factory == nil {
|
||||
factory = newDistributorUploadClient
|
||||
}
|
||||
return &Client{
|
||||
Endpoint: cfg.Endpoint,
|
||||
TokenEnv: cfg.TokenEnv,
|
||||
Timeout: cfg.Timeout,
|
||||
newUploadClient: factory,
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Client) Upload(ctx context.Context, req UploadRequest) (UploadResult, error) {
|
||||
if c == nil {
|
||||
return UploadResult{}, fmt.Errorf("distributor client is nil")
|
||||
}
|
||||
if c.Endpoint == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor endpoint is required")
|
||||
}
|
||||
if c.TokenEnv == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor token environment variable is required")
|
||||
}
|
||||
if req.BundleID == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor bundle id is required")
|
||||
}
|
||||
if req.IdempotencyKey == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor idempotency key is required for bundle %q", req.BundleID)
|
||||
}
|
||||
if req.SourcePath == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor source path is required for bundle %q", req.BundleID)
|
||||
}
|
||||
if req.BundlePath == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor bundle path is required for bundle %q", req.BundleID)
|
||||
}
|
||||
if c.newUploadClient == nil {
|
||||
return UploadResult{}, fmt.Errorf("distributor upload client factory is required for endpoint %q", c.Endpoint)
|
||||
}
|
||||
|
||||
token := os.Getenv(c.TokenEnv)
|
||||
if token == "" {
|
||||
return UploadResult{}, fmt.Errorf("distributor token environment variable %q is not set", c.TokenEnv)
|
||||
}
|
||||
|
||||
uploadClient, err := c.newUploadClient(c.Endpoint, token, c.Timeout)
|
||||
if err != nil {
|
||||
return UploadResult{}, fmt.Errorf("create distributor upload client for endpoint %q: %w", c.Endpoint, redactToken(err, token))
|
||||
}
|
||||
|
||||
runCtx := ctx
|
||||
if runCtx == nil {
|
||||
runCtx = context.Background()
|
||||
}
|
||||
cancel := func() {}
|
||||
if c.Timeout > 0 {
|
||||
runCtx, cancel = context.WithTimeout(runCtx, c.Timeout)
|
||||
}
|
||||
defer cancel()
|
||||
|
||||
result, err := uploadClient.UploadFiles(runCtx, uploadFilesOptions{
|
||||
BundleID: req.BundleID,
|
||||
IdempotencyKey: req.IdempotencyKey,
|
||||
SourcePath: req.SourcePath,
|
||||
BundlePath: req.BundlePath,
|
||||
})
|
||||
if err != nil {
|
||||
return UploadResult{}, wrapUploadError(err, uploadErrorContext{
|
||||
Endpoint: c.Endpoint,
|
||||
BundleID: req.BundleID,
|
||||
IdempotencyKey: req.IdempotencyKey,
|
||||
SourcePath: req.SourcePath,
|
||||
BundlePath: req.BundlePath,
|
||||
Token: token,
|
||||
})
|
||||
}
|
||||
|
||||
return UploadResult{
|
||||
RunID: result.RunID,
|
||||
Status: result.Status,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type distributorUploadClient struct {
|
||||
client *distributorupload.Client
|
||||
}
|
||||
|
||||
func newDistributorUploadClient(endpoint, token string, timeout time.Duration) (uploadClient, error) {
|
||||
httpClient := (*http.Client)(nil)
|
||||
if timeout > 0 {
|
||||
httpClient = &http.Client{Timeout: timeout}
|
||||
}
|
||||
client, err := distributorupload.NewClient(distributorupload.ClientOptions{
|
||||
Endpoint: endpoint,
|
||||
Token: token,
|
||||
HTTPClient: httpClient,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return distributorUploadClient{client: client}, nil
|
||||
}
|
||||
|
||||
func (c distributorUploadClient) UploadFiles(ctx context.Context, opts uploadFilesOptions) (uploadFilesResult, error) {
|
||||
result, err := c.client.UploadFiles(ctx, distributorupload.UploadFilesOptions{
|
||||
ID: opts.BundleID,
|
||||
IdempotencyKey: opts.IdempotencyKey,
|
||||
Files: []distributorbundle.BundleFile{
|
||||
{SourcePath: opts.SourcePath, Path: opts.BundlePath},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return uploadFilesResult{}, err
|
||||
}
|
||||
return uploadFilesResult{
|
||||
RunID: result.RunID,
|
||||
Status: result.Status,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type uploadErrorContext struct {
|
||||
Endpoint string
|
||||
BundleID string
|
||||
IdempotencyKey string
|
||||
SourcePath string
|
||||
BundlePath string
|
||||
Token string
|
||||
}
|
||||
|
||||
func wrapUploadError(err error, ctx uploadErrorContext) error {
|
||||
var conflict *distributorupload.IdempotencyConflictError
|
||||
isConflict := errors.As(err, &conflict)
|
||||
err = redactToken(err, ctx.Token)
|
||||
if isConflict {
|
||||
return &IdempotencyConflictError{
|
||||
Err: fmt.Errorf("upload distributor bundle %q to endpoint %q with idempotency key %q from source %q as bundle path %q: idempotency conflict: %w", ctx.BundleID, ctx.Endpoint, ctx.IdempotencyKey, ctx.SourcePath, ctx.BundlePath, err),
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("upload distributor bundle %q to endpoint %q with idempotency key %q from source %q as bundle path %q: %w", ctx.BundleID, ctx.Endpoint, ctx.IdempotencyKey, ctx.SourcePath, ctx.BundlePath, err)
|
||||
}
|
||||
|
||||
func redactToken(err error, token string) error {
|
||||
if err == nil || token == "" {
|
||||
return err
|
||||
}
|
||||
return errors.New(redactTokenString(err.Error(), token))
|
||||
}
|
||||
|
||||
func redactTokenString(value, token string) string {
|
||||
if token == "" {
|
||||
return value
|
||||
}
|
||||
return strings.ReplaceAll(value, token, "[redacted]")
|
||||
}
|
||||
243
internal/adapters/distributor/client_test.go
Normal file
243
internal/adapters/distributor/client_test.go
Normal file
@@ -0,0 +1,243 @@
|
||||
package distributor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
distributorupload "gitea.maximumdirect.net/eric/distributor/pkg/upload"
|
||||
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/config"
|
||||
)
|
||||
|
||||
func TestUploadUsesConfiguredClientAndSingleFile(t *testing.T) {
|
||||
cfg := config.Defaults().Notify.Distributor
|
||||
cfg.Endpoint = "https://distributor.example.test"
|
||||
cfg.TokenEnv = "DISTRIBUTOR_UPLOAD_TOKEN"
|
||||
cfg.Timeout = 15 * time.Second
|
||||
t.Setenv(cfg.TokenEnv, "secret-token")
|
||||
|
||||
factory := &fakeUploadFactory{
|
||||
client: &fakeUploadClient{
|
||||
result: uploadFilesResult{RunID: "run-123", Status: "accepted"},
|
||||
},
|
||||
}
|
||||
client := newClient(cfg, factory.newClient)
|
||||
|
||||
result, err := client.Upload(context.Background(), UploadRequest{
|
||||
BundleID: "weatherreporter.home.daily.run",
|
||||
IdempotencyKey: "weatherreporter.home.daily.run",
|
||||
SourcePath: "/tmp/report.md",
|
||||
BundlePath: "daily.md",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Upload() error = %v", err)
|
||||
}
|
||||
if result.RunID != "run-123" || result.Status != "accepted" {
|
||||
t.Fatalf("result = %#v, want accepted run", result)
|
||||
}
|
||||
if factory.endpoint != cfg.Endpoint {
|
||||
t.Fatalf("factory endpoint = %q, want %q", factory.endpoint, cfg.Endpoint)
|
||||
}
|
||||
if factory.token != "secret-token" {
|
||||
t.Fatalf("factory token = %q, want secret-token", factory.token)
|
||||
}
|
||||
if factory.timeout != 15*time.Second {
|
||||
t.Fatalf("factory timeout = %s, want 15s", factory.timeout)
|
||||
}
|
||||
got := factory.client.opts
|
||||
if got.BundleID != "weatherreporter.home.daily.run" {
|
||||
t.Fatalf("BundleID = %q, want weatherreporter.home.daily.run", got.BundleID)
|
||||
}
|
||||
if got.IdempotencyKey != "weatherreporter.home.daily.run" {
|
||||
t.Fatalf("IdempotencyKey = %q, want weatherreporter.home.daily.run", got.IdempotencyKey)
|
||||
}
|
||||
if got.SourcePath != "/tmp/report.md" {
|
||||
t.Fatalf("SourcePath = %q, want /tmp/report.md", got.SourcePath)
|
||||
}
|
||||
if got.BundlePath != "daily.md" {
|
||||
t.Fatalf("BundlePath = %q, want daily.md", got.BundlePath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadRejectsMissingInputs(t *testing.T) {
|
||||
cfg := config.Defaults().Notify.Distributor
|
||||
t.Setenv(cfg.TokenEnv, "secret-token")
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*Client, *UploadRequest)
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "Token",
|
||||
mutate: func(c *Client, req *UploadRequest) {
|
||||
t.Setenv(c.TokenEnv, "")
|
||||
},
|
||||
wantErr: "token environment variable",
|
||||
},
|
||||
{
|
||||
name: "SourcePath",
|
||||
mutate: func(c *Client, req *UploadRequest) {
|
||||
req.SourcePath = ""
|
||||
},
|
||||
wantErr: "source path is required",
|
||||
},
|
||||
{
|
||||
name: "BundlePath",
|
||||
mutate: func(c *Client, req *UploadRequest) {
|
||||
req.BundlePath = ""
|
||||
},
|
||||
wantErr: "bundle path is required",
|
||||
},
|
||||
{
|
||||
name: "UploadClientFactory",
|
||||
mutate: func(c *Client, req *UploadRequest) {
|
||||
c.newUploadClient = nil
|
||||
},
|
||||
wantErr: "upload client factory is required",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Setenv(cfg.TokenEnv, "secret-token")
|
||||
client := newClient(cfg, (&fakeUploadFactory{client: &fakeUploadClient{}}).newClient)
|
||||
req := validUploadRequest()
|
||||
tt.mutate(client, &req)
|
||||
|
||||
_, err := client.Upload(context.Background(), req)
|
||||
if err == nil {
|
||||
t.Fatal("Upload() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
|
||||
}
|
||||
if strings.Contains(err.Error(), "secret-token") {
|
||||
t.Fatalf("error = %q, want no token value", err.Error())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadWrapsFactoryErrorWithoutToken(t *testing.T) {
|
||||
cfg := config.Defaults().Notify.Distributor
|
||||
cfg.Endpoint = "https://distributor.example.test"
|
||||
t.Setenv(cfg.TokenEnv, "secret-token")
|
||||
factory := &fakeUploadFactory{
|
||||
err: fmt.Errorf("factory failed with secret-token"),
|
||||
}
|
||||
client := newClient(cfg, factory.newClient)
|
||||
|
||||
_, err := client.Upload(context.Background(), validUploadRequest())
|
||||
if err == nil {
|
||||
t.Fatal("Upload() error = nil, want error")
|
||||
}
|
||||
if strings.Contains(err.Error(), "secret-token") {
|
||||
t.Fatalf("error = %q, want no token value", err.Error())
|
||||
}
|
||||
if !strings.Contains(err.Error(), cfg.Endpoint) {
|
||||
t.Fatalf("error = %q, want endpoint context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadWrapsUploadFailureWithContextWithoutToken(t *testing.T) {
|
||||
cfg := config.Defaults().Notify.Distributor
|
||||
cfg.Endpoint = "https://distributor.example.test"
|
||||
t.Setenv(cfg.TokenEnv, "secret-token")
|
||||
factory := &fakeUploadFactory{
|
||||
client: &fakeUploadClient{err: fmt.Errorf("server rejected secret-token")},
|
||||
}
|
||||
client := newClient(cfg, factory.newClient)
|
||||
req := validUploadRequest()
|
||||
|
||||
_, err := client.Upload(context.Background(), req)
|
||||
if err == nil {
|
||||
t.Fatal("Upload() error = nil, want error")
|
||||
}
|
||||
for _, want := range []string{cfg.Endpoint, req.BundleID, req.IdempotencyKey, req.SourcePath, req.BundlePath} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Fatalf("error = %q, want context %q", err.Error(), want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(err.Error(), "secret-token") {
|
||||
t.Fatalf("error = %q, want no token value", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadPreservesIdempotencyConflictDiagnosis(t *testing.T) {
|
||||
cfg := config.Defaults().Notify.Distributor
|
||||
cfg.Endpoint = "https://distributor.example.test"
|
||||
t.Setenv(cfg.TokenEnv, "secret-token")
|
||||
factory := &fakeUploadFactory{
|
||||
client: &fakeUploadClient{
|
||||
err: &distributorupload.IdempotencyConflictError{
|
||||
HTTPError: distributorupload.HTTPError{
|
||||
StatusCode: 409,
|
||||
Status: "409 Conflict",
|
||||
Message: "conflicting upload for secret-token",
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
client := newClient(cfg, factory.newClient)
|
||||
|
||||
_, err := client.Upload(context.Background(), validUploadRequest())
|
||||
if err == nil {
|
||||
t.Fatal("Upload() error = nil, want error")
|
||||
}
|
||||
var conflict *IdempotencyConflictError
|
||||
if !errors.As(err, &conflict) {
|
||||
t.Fatalf("Upload() error = %T %v, want IdempotencyConflictError", err, err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "idempotency conflict") {
|
||||
t.Fatalf("error = %q, want idempotency conflict diagnosis", err.Error())
|
||||
}
|
||||
if strings.Contains(err.Error(), "secret-token") {
|
||||
t.Fatalf("error = %q, want no token value", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func validUploadRequest() UploadRequest {
|
||||
return UploadRequest{
|
||||
BundleID: "weatherreporter.home.daily.run",
|
||||
IdempotencyKey: "weatherreporter.home.daily.run",
|
||||
SourcePath: "/tmp/report.md",
|
||||
BundlePath: "daily.md",
|
||||
}
|
||||
}
|
||||
|
||||
type fakeUploadFactory struct {
|
||||
endpoint string
|
||||
token string
|
||||
timeout time.Duration
|
||||
client *fakeUploadClient
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeUploadFactory) newClient(endpoint, token string, timeout time.Duration) (uploadClient, error) {
|
||||
f.endpoint = endpoint
|
||||
f.token = token
|
||||
f.timeout = timeout
|
||||
if f.err != nil {
|
||||
return nil, f.err
|
||||
}
|
||||
return f.client, nil
|
||||
}
|
||||
|
||||
type fakeUploadClient struct {
|
||||
opts uploadFilesOptions
|
||||
result uploadFilesResult
|
||||
err error
|
||||
}
|
||||
|
||||
func (c *fakeUploadClient) UploadFiles(ctx context.Context, opts uploadFilesOptions) (uploadFilesResult, error) {
|
||||
c.opts = opts
|
||||
if c.err != nil {
|
||||
return uploadFilesResult{}, c.err
|
||||
}
|
||||
return c.result, nil
|
||||
}
|
||||
@@ -109,10 +109,10 @@ func (c *Client) FetchBundle(ctx context.Context) (*forecast.Bundle, error) {
|
||||
if err := builder.fetchDiscussion(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := builder.addStub("daily", "daily forecast data is not available from the weather API yet"); err != nil {
|
||||
if err := builder.fetchWeatherStory(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := builder.addStub("weather_story", "NWS weather story is not available from the weather API yet"); err != nil {
|
||||
if err := builder.addStub("daily", "daily forecast data is not available from the weather API yet"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -246,6 +246,27 @@ func (b *bundleBuilder) fetchDiscussion(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *bundleBuilder) fetchWeatherStory(ctx context.Context) error {
|
||||
raw, source, err := b.client.fetch(ctx, "weather_story", "/weatherstories/latest", queryOptions{omitUnits: true})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if raw == nil {
|
||||
return b.handleMissing(&source, "NWS weather story data is missing", false)
|
||||
}
|
||||
var story forecast.WeatherStory
|
||||
if err := decodeSource(raw, &story); err != nil {
|
||||
return b.handleMalformed(&source, err, false)
|
||||
}
|
||||
if !story.StartTime.IsZero() {
|
||||
source.IssuedAt = &story.StartTime
|
||||
}
|
||||
source.UpdatedAt = story.UpdatedAt
|
||||
b.bundle.WeatherStory = &story
|
||||
b.addSource(source)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *bundleBuilder) addStub(sourceName string, message string) error {
|
||||
source := forecast.Source{
|
||||
Name: sourceName,
|
||||
@@ -307,6 +328,7 @@ type queryOptions struct {
|
||||
precision bool
|
||||
timezone bool
|
||||
allowNull bool
|
||||
omitUnits bool
|
||||
}
|
||||
|
||||
type envelope struct {
|
||||
@@ -366,7 +388,9 @@ func (c *Client) endpointURL(endpoint string, opts queryOptions) *url.URL {
|
||||
reqURL.Path = path.Join(c.baseURL.Path, endpoint)
|
||||
query := reqURL.Query()
|
||||
query.Set("format", c.format)
|
||||
query.Set("units", c.units)
|
||||
if !opts.omitUnits {
|
||||
query.Set("units", c.units)
|
||||
}
|
||||
if opts.precision {
|
||||
query.Set("precision", strconv.Itoa(c.precision))
|
||||
}
|
||||
|
||||
@@ -49,11 +49,17 @@ func TestFetchBundleFromFixtures(t *testing.T) {
|
||||
if bundle.Discussion.LongTerm == nil || bundle.Discussion.LongTerm.Text != "Warmer temperatures and periodic rain chances continue into the weekend." {
|
||||
t.Fatalf("Discussion.LongTerm = %#v, want long-term AFD text", bundle.Discussion.LongTerm)
|
||||
}
|
||||
if bundle.WeatherStory == nil || bundle.WeatherStory.Title != "Several Chances for Rain Through Monday" {
|
||||
t.Fatalf("WeatherStory = %#v, want latest weather story", bundle.WeatherStory)
|
||||
}
|
||||
if bundle.WeatherStory.UpdatedAt == nil {
|
||||
t.Fatalf("WeatherStory.UpdatedAt = nil, want update timestamp")
|
||||
}
|
||||
if len(bundle.Sources) != 8 {
|
||||
t.Fatalf("Sources length = %d, want 8", len(bundle.Sources))
|
||||
}
|
||||
if len(bundle.Warnings) != 2 {
|
||||
t.Fatalf("Warnings length = %d, want daily and weather story warnings", len(bundle.Warnings))
|
||||
if len(bundle.Warnings) != 1 {
|
||||
t.Fatalf("Warnings length = %d, want daily warning", len(bundle.Warnings))
|
||||
}
|
||||
if !containsPath(requested, "/forecast/hourly") || containsPath(requested, "/forecast/hourly/today") {
|
||||
t.Fatalf("requested paths = %v, want full hourly endpoint only", requested)
|
||||
@@ -61,6 +67,9 @@ func TestFetchBundleFromFixtures(t *testing.T) {
|
||||
if !containsPath(requested, "/forecast/narrative") || containsPath(requested, "/forecast/narrative/today") {
|
||||
t.Fatalf("requested paths = %v, want full narrative endpoint only", requested)
|
||||
}
|
||||
if !containsPath(requested, "/weatherstories/latest") {
|
||||
t.Fatalf("requested paths = %v, want weather story endpoint", requested)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchBundleBuildsExpectedQueries(t *testing.T) {
|
||||
@@ -74,8 +83,17 @@ func TestFetchBundleBuildsExpectedQueries(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, rawURL := range requested {
|
||||
if !strings.Contains(rawURL, "format=json") || !strings.Contains(rawURL, "units=us") {
|
||||
t.Fatalf("request %q missing format=json or units=us", rawURL)
|
||||
if !strings.Contains(rawURL, "format=json") {
|
||||
t.Fatalf("request %q missing format=json", rawURL)
|
||||
}
|
||||
if strings.HasPrefix(rawURL, "/weatherstories/") {
|
||||
if strings.Contains(rawURL, "units=") || strings.Contains(rawURL, "precision=") || strings.Contains(rawURL, "tz=") {
|
||||
t.Fatalf("weather story request %q should use format only", rawURL)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if !strings.Contains(rawURL, "units=us") {
|
||||
t.Fatalf("request %q missing units=us", rawURL)
|
||||
}
|
||||
if strings.HasPrefix(rawURL, "/forecast/") {
|
||||
if !strings.Contains(rawURL, "precision=1") || !strings.Contains(rawURL, "tz=America%2FChicago") {
|
||||
@@ -99,6 +117,16 @@ func TestFetchBundleRecordsSourceHash(t *testing.T) {
|
||||
if observation.DataSHA256 != want {
|
||||
t.Fatalf("DataSHA256 = %q, want %q", observation.DataSHA256, want)
|
||||
}
|
||||
story := sourceByName(t, bundle.Sources, "weather_story")
|
||||
if story.Endpoint != "/weatherstories/latest" {
|
||||
t.Fatalf("weather story endpoint = %q, want /weatherstories/latest", story.Endpoint)
|
||||
}
|
||||
if story.DataSHA256 != hashFixtureData(t, "weather_story.json") {
|
||||
t.Fatalf("weather story DataSHA256 = %q, want fixture hash", story.DataSHA256)
|
||||
}
|
||||
if story.IssuedAt == nil || story.UpdatedAt == nil {
|
||||
t.Fatalf("weather story source timestamps = issued %#v updated %#v, want both", story.IssuedAt, story.UpdatedAt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHTTPErrorIsActionable(t *testing.T) {
|
||||
@@ -169,7 +197,7 @@ func TestMissingSourcePolicyWarnNoneError(t *testing.T) {
|
||||
wantWarns int
|
||||
wantSource bool
|
||||
}{
|
||||
{name: "warn", policy: config.MissingSourceWarn, wantWarns: 3, wantSource: true},
|
||||
{name: "warn", policy: config.MissingSourceWarn, wantWarns: 2, wantSource: true},
|
||||
{name: "none", policy: config.MissingSourceNone, wantWarns: 0, wantSource: true},
|
||||
{name: "error", policy: config.MissingSourceError, wantErr: true},
|
||||
}
|
||||
@@ -230,6 +258,45 @@ func TestMalformedNonRequiredSourceUsesPolicy(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestMissingWeatherStoryUsesPolicy(t *testing.T) {
|
||||
server := fixtureServer(t, map[string]handlerOverride{
|
||||
"/weatherstories/latest": {status: http.StatusOK, body: `{"data": null}`},
|
||||
}, nil)
|
||||
client := newTestClient(t, server.URL+"/", map[string]config.MissingSourcePolicy{
|
||||
"weather_story": config.MissingSourceWarn,
|
||||
})
|
||||
|
||||
bundle, err := client.FetchBundle(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("FetchBundle() error = %v", err)
|
||||
}
|
||||
if bundle.WeatherStory != nil {
|
||||
t.Fatalf("WeatherStory = %#v, want nil for missing source", bundle.WeatherStory)
|
||||
}
|
||||
source := sourceByName(t, bundle.Sources, "weather_story")
|
||||
if !source.Missing || len(source.Warnings) != 1 {
|
||||
t.Fatalf("weather_story source = %#v, want missing source warning", source)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMalformedWeatherStoryUsesPolicy(t *testing.T) {
|
||||
server := fixtureServer(t, map[string]handlerOverride{
|
||||
"/weatherstories/latest": {status: http.StatusOK, body: `{"data": {"startTime": 123}}`},
|
||||
}, nil)
|
||||
client := newTestClient(t, server.URL+"/", map[string]config.MissingSourcePolicy{
|
||||
"weather_story": config.MissingSourceWarn,
|
||||
})
|
||||
|
||||
bundle, err := client.FetchBundle(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("FetchBundle() error = %v", err)
|
||||
}
|
||||
source := sourceByName(t, bundle.Sources, "weather_story")
|
||||
if !source.Missing || len(source.Warnings) != 1 || source.Warnings[0].Code != "malformed_source" {
|
||||
t.Fatalf("weather_story source = %#v, want malformed source warning", source)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContextCancellation(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
<-r.Context().Done()
|
||||
@@ -296,12 +363,13 @@ type handlerOverride struct {
|
||||
func fixtureServer(t *testing.T, overrides map[string]handlerOverride, requested *[]string) *httptest.Server {
|
||||
t.Helper()
|
||||
fixtures := map[string]string{
|
||||
"/observations": "observations.json",
|
||||
"/conditions/current": "current.json",
|
||||
"/forecast/hourly": "hourly.json",
|
||||
"/forecast/narrative": "narrative.json",
|
||||
"/alerts/active": "alerts.json",
|
||||
"/discussion": "discussion.json",
|
||||
"/observations": "observations.json",
|
||||
"/conditions/current": "current.json",
|
||||
"/forecast/hourly": "hourly.json",
|
||||
"/forecast/narrative": "narrative.json",
|
||||
"/alerts/active": "alerts.json",
|
||||
"/discussion": "discussion.json",
|
||||
"/weatherstories/latest": "weather_story.json",
|
||||
}
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if requested != nil {
|
||||
|
||||
14
internal/adapters/weatherapi/testdata/weather_story.json
vendored
Normal file
14
internal/adapters/weatherapi/testdata/weather_story.json
vendored
Normal file
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"data": {
|
||||
"officeId": "LSX",
|
||||
"startTime": "2026-05-30T08:46:00Z",
|
||||
"endTime": "2026-05-31T11:00:00Z",
|
||||
"updatedAt": "2026-05-30T09:00:34Z",
|
||||
"title": "Several Chances for Rain Through Monday",
|
||||
"description": "A stagnant weather pattern with low pressure over the Great Plains and high pressure over the Great Lakes will continue to produce scattered showers and thunderstorms, for areas mainly along and west of the Mississippi River today and Sunday.",
|
||||
"altText": "This slide shows the forecast for today through Tuesday with icons for showers and thunderstorms and a picture of a cumulonimbus cloud on the right side.",
|
||||
"priority": false,
|
||||
"order": 1,
|
||||
"downloadUrl": "https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f"
|
||||
}
|
||||
}
|
||||
@@ -3,10 +3,12 @@ package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
distributoradapter "gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/distributor"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/scriptorium"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/weatherapi"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/briefing"
|
||||
@@ -45,6 +47,7 @@ type GenerateRequest struct {
|
||||
Date time.Time
|
||||
StormStart time.Time
|
||||
StormEnd time.Time
|
||||
Notifier Notifier
|
||||
}
|
||||
|
||||
type BatchRequest struct {
|
||||
@@ -54,6 +57,7 @@ type BatchRequest struct {
|
||||
OutputDir string
|
||||
Renderer Renderer
|
||||
Store state.Store
|
||||
Notifier Notifier
|
||||
}
|
||||
|
||||
type FetchBundleRequest struct {
|
||||
@@ -73,6 +77,7 @@ type ReportRequest struct {
|
||||
OutputPath string
|
||||
Renderer Renderer
|
||||
Store state.Store
|
||||
Notifier Notifier
|
||||
}
|
||||
|
||||
type BriefingResult struct {
|
||||
@@ -94,6 +99,7 @@ type ReportResult struct {
|
||||
RecentChanges []changes.Change
|
||||
RenderResult *scriptorium.RenderResult
|
||||
RunResult *scriptorium.RunResult
|
||||
Notification *NotificationResult
|
||||
}
|
||||
|
||||
type BatchResult struct {
|
||||
@@ -107,20 +113,23 @@ type BatchResult struct {
|
||||
}
|
||||
|
||||
type BatchReportResult struct {
|
||||
ReportID report.ID `json:"reportId"`
|
||||
ReportName string `json:"reportName"`
|
||||
PromptID string `json:"promptId"`
|
||||
RunID string `json:"runId"`
|
||||
Status string `json:"status"`
|
||||
Error string `json:"error,omitempty"`
|
||||
GeneratedAt time.Time `json:"generatedAt"`
|
||||
ValidPeriod timeutil.Period `json:"validPeriod"`
|
||||
BriefingPath string `json:"briefingPath,omitempty"`
|
||||
DataPackagePath string `json:"dataPackagePath,omitempty"`
|
||||
PreflightPath string `json:"preflightPath,omitempty"`
|
||||
ReportPath string `json:"reportPath,omitempty"`
|
||||
OutputPath string `json:"outputPath,omitempty"`
|
||||
MetadataPath string `json:"metadataPath,omitempty"`
|
||||
ReportID report.ID `json:"reportId"`
|
||||
ReportName string `json:"reportName"`
|
||||
PromptID string `json:"promptId"`
|
||||
RunID string `json:"runId"`
|
||||
Status string `json:"status"`
|
||||
Error string `json:"error,omitempty"`
|
||||
NotificationStatus string `json:"notificationStatus,omitempty"`
|
||||
NotificationRunID string `json:"notificationRunId,omitempty"`
|
||||
NotificationError string `json:"notificationError,omitempty"`
|
||||
GeneratedAt time.Time `json:"generatedAt"`
|
||||
ValidPeriod timeutil.Period `json:"validPeriod"`
|
||||
BriefingPath string `json:"briefingPath,omitempty"`
|
||||
DataPackagePath string `json:"dataPackagePath,omitempty"`
|
||||
PreflightPath string `json:"preflightPath,omitempty"`
|
||||
ReportPath string `json:"reportPath,omitempty"`
|
||||
OutputPath string `json:"outputPath,omitempty"`
|
||||
MetadataPath string `json:"metadataPath,omitempty"`
|
||||
}
|
||||
|
||||
type BatchError struct {
|
||||
@@ -139,6 +148,45 @@ type Renderer interface {
|
||||
Run(context.Context, scriptorium.RunRequest) (*scriptorium.RunResult, error)
|
||||
}
|
||||
|
||||
type Notifier interface {
|
||||
Notify(context.Context, NotificationRequest) (*NotificationResult, error)
|
||||
}
|
||||
|
||||
type NotificationRequest struct {
|
||||
ReportID report.ID
|
||||
RunID string
|
||||
BundleID string
|
||||
IdempotencyKey string
|
||||
ReportPath string
|
||||
BundlePath string
|
||||
}
|
||||
|
||||
type NotificationResult struct {
|
||||
BundleID string
|
||||
IdempotencyKey string
|
||||
RunID string
|
||||
Status string
|
||||
}
|
||||
|
||||
type NotificationError struct {
|
||||
Request NotificationRequest
|
||||
Err error
|
||||
}
|
||||
|
||||
func (e *NotificationError) Error() string {
|
||||
if e == nil || e.Err == nil {
|
||||
return "notification failed"
|
||||
}
|
||||
return e.Err.Error()
|
||||
}
|
||||
|
||||
func (e *NotificationError) Unwrap() error {
|
||||
if e == nil {
|
||||
return nil
|
||||
}
|
||||
return e.Err
|
||||
}
|
||||
|
||||
func Generate(ctx context.Context, req GenerateRequest) error {
|
||||
now := req.Now
|
||||
if now.IsZero() {
|
||||
@@ -153,6 +201,7 @@ func Generate(ctx context.Context, req GenerateRequest) error {
|
||||
Config: req.Config,
|
||||
Resolved: resolved,
|
||||
OutputPath: req.OutputPath,
|
||||
Notifier: req.Notifier,
|
||||
})
|
||||
return err
|
||||
}
|
||||
@@ -211,10 +260,16 @@ func RunBatchDetailed(ctx context.Context, req BatchRequest) (*BatchResult, erro
|
||||
OutputPath: outputPath,
|
||||
Renderer: req.Renderer,
|
||||
Store: store,
|
||||
Notifier: req.Notifier,
|
||||
})
|
||||
if err != nil {
|
||||
item.Status = "failed"
|
||||
item.Error = err.Error()
|
||||
var notificationErr *NotificationError
|
||||
if errors.As(err, ¬ificationErr) {
|
||||
item.NotificationStatus = "failed"
|
||||
item.NotificationError = notificationErr.Error()
|
||||
}
|
||||
result.Failed++
|
||||
} else {
|
||||
item.Status = "succeeded"
|
||||
@@ -224,6 +279,10 @@ func RunBatchDetailed(ctx context.Context, req BatchRequest) (*BatchResult, erro
|
||||
item.ReportPath = reportResult.ReportPath
|
||||
item.OutputPath = reportResult.OutputPath
|
||||
item.MetadataPath = reportResult.MetadataPath
|
||||
if reportResult.Notification != nil {
|
||||
item.NotificationStatus = reportResult.Notification.Status
|
||||
item.NotificationRunID = reportResult.Notification.RunID
|
||||
}
|
||||
result.Succeeded++
|
||||
}
|
||||
result.Reports = append(result.Reports, item)
|
||||
@@ -481,6 +540,11 @@ func GenerateReport(ctx context.Context, req ReportRequest) (*ReportResult, erro
|
||||
return nil, runErr
|
||||
}
|
||||
|
||||
notification, err := notifyReport(ctx, req.Config, req.Resolved, reportPath, metadata, req.Notifier)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &ReportResult{
|
||||
Briefing: briefingPackage,
|
||||
BriefingPath: briefingPath,
|
||||
@@ -495,6 +559,97 @@ func GenerateReport(ctx context.Context, req ReportRequest) (*ReportResult, erro
|
||||
RecentChanges: recentChanges,
|
||||
RenderResult: renderResult,
|
||||
RunResult: runResult,
|
||||
Notification: notification,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func notifyReport(ctx context.Context, cfg config.Config, resolved report.Resolved, reportPath string, metadata state.Metadata, notifier Notifier) (*NotificationResult, error) {
|
||||
notifier, enabled := reportNotifier(cfg, notifier)
|
||||
if !enabled {
|
||||
return nil, nil
|
||||
}
|
||||
notificationRequest, err := buildNotificationRequest(cfg, resolved, reportPath, metadata)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result, err := notifier.Notify(ctx, notificationRequest)
|
||||
if err != nil {
|
||||
return nil, &NotificationError{
|
||||
Request: notificationRequest,
|
||||
Err: fmt.Errorf("notify report %q run %q from managed report %q: %w", resolved.Definition.ID, metadata.RunID, reportPath, err),
|
||||
}
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func reportNotifier(cfg config.Config, notifier Notifier) (Notifier, bool) {
|
||||
if !cfg.Notify.Distributor.Enabled {
|
||||
return noopNotifier{}, false
|
||||
}
|
||||
if notifier != nil {
|
||||
return notifier, true
|
||||
}
|
||||
return distributorNotifier{
|
||||
client: distributoradapter.New(cfg.Notify.Distributor),
|
||||
}, true
|
||||
}
|
||||
|
||||
func buildNotificationRequest(cfg config.Config, resolved report.Resolved, reportPath string, metadata state.Metadata) (NotificationRequest, error) {
|
||||
values := config.DistributorTemplateValues{
|
||||
LocationID: cfg.Location.ID,
|
||||
ReportID: string(resolved.Definition.ID),
|
||||
RunID: metadata.RunID,
|
||||
ArtifactGroup: resolved.Definition.ArtifactGroup,
|
||||
BatchOutputName: resolved.Definition.BatchOutputName,
|
||||
}
|
||||
bundleID, err := config.RenderDistributorBundleID(cfg.Notify.Distributor.BundleIDTemplate, values)
|
||||
if err != nil {
|
||||
return NotificationRequest{}, err
|
||||
}
|
||||
values.BundleID = bundleID
|
||||
idempotencyKey, err := config.RenderDistributorIdempotencyKey(cfg.Notify.Distributor.IdempotencyKeyTemplate, values)
|
||||
if err != nil {
|
||||
return NotificationRequest{}, err
|
||||
}
|
||||
bundlePath, err := config.RenderDistributorReportPath(cfg.Notify.Distributor.ReportPathTemplate, values)
|
||||
if err != nil {
|
||||
return NotificationRequest{}, err
|
||||
}
|
||||
return NotificationRequest{
|
||||
ReportID: resolved.Definition.ID,
|
||||
RunID: metadata.RunID,
|
||||
BundleID: bundleID,
|
||||
IdempotencyKey: idempotencyKey,
|
||||
ReportPath: reportPath,
|
||||
BundlePath: bundlePath,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type noopNotifier struct{}
|
||||
|
||||
func (noopNotifier) Notify(context.Context, NotificationRequest) (*NotificationResult, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
type distributorNotifier struct {
|
||||
client *distributoradapter.Client
|
||||
}
|
||||
|
||||
func (n distributorNotifier) Notify(ctx context.Context, req NotificationRequest) (*NotificationResult, error) {
|
||||
result, err := n.client.Upload(ctx, distributoradapter.UploadRequest{
|
||||
BundleID: req.BundleID,
|
||||
IdempotencyKey: req.IdempotencyKey,
|
||||
SourcePath: req.ReportPath,
|
||||
BundlePath: req.BundlePath,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &NotificationResult{
|
||||
BundleID: req.BundleID,
|
||||
IdempotencyKey: req.IdempotencyKey,
|
||||
RunID: result.RunID,
|
||||
Status: result.Status,
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -35,6 +35,8 @@ func TestFetchAndSaveBundle(t *testing.T) {
|
||||
_, _ = w.Write([]byte(`{"data":{"alerts":[]}}`))
|
||||
case "/discussion":
|
||||
_, _ = w.Write([]byte(`{"data":{"product":"discussion","issuedAt":"2026-05-29T09:25:00-05:00","keyMessages":[],"shortTerm":{"qualifier":"(Short Term)","text":"Short-term AFD narrative for saved bundle."},"longTerm":{"qualifier":"(Long Term)","text":"Long-term AFD narrative for saved bundle."}}}`))
|
||||
case "/weatherstories/latest":
|
||||
_, _ = w.Write([]byte(`{"data":{"officeId":"LSX","startTime":"2026-05-30T08:46:00Z","endTime":"2026-05-31T11:00:00Z","updatedAt":"2026-05-30T09:00:34Z","title":"Several Chances for Rain Through Monday","description":"Scattered showers and thunderstorms remain possible.","altText":"Forecast weather story graphic.","priority":false,"order":1,"downloadUrl":"https://api.weather.gov/offices/LSX/weatherstories/download/test"}}`))
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
@@ -59,6 +61,9 @@ func TestFetchAndSaveBundle(t *testing.T) {
|
||||
if !strings.Contains(string(data), `"product": "hourly"`) {
|
||||
t.Fatalf("saved bundle missing hourly product:\n%s", string(data))
|
||||
}
|
||||
if !strings.Contains(string(data), `"title": "Several Chances for Rain Through Monday"`) {
|
||||
t.Fatalf("saved bundle missing weather story title:\n%s", string(data))
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchAndSaveBundleRequiresOutputPath(t *testing.T) {
|
||||
@@ -216,6 +221,9 @@ func TestGenerateReportWritesReportAndPreflight(t *testing.T) {
|
||||
if current == nil || current.ConditionText != "Clear" || current.TemperatureF == nil || *current.TemperatureF != 75 {
|
||||
t.Fatalf("data package current conditions = %#v, want current conditions", current)
|
||||
}
|
||||
if savedDataPackage.Briefing.Daily == nil || savedDataPackage.Briefing.Daily.WeatherStory == nil || savedDataPackage.Briefing.Daily.WeatherStory.Title != "Several Chances for Rain Through Monday" {
|
||||
t.Fatalf("data package weather story = %#v, want weather story title", savedDataPackage.Briefing.Daily)
|
||||
}
|
||||
if !strings.Contains(string(data), "Short-term AFD narrative for generated report.") || !strings.Contains(string(data), "Long-term AFD narrative for generated report.") {
|
||||
t.Fatalf("data package missing AFD short/long-term discussion:\n%s", string(data))
|
||||
}
|
||||
@@ -247,6 +255,209 @@ func TestGenerateReportWritesReportAndPreflight(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateReportDisabledNotificationDoesNotCallNotifier(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
cfg := dailyTestConfig(t, server)
|
||||
cfg.Workspace.Root = t.TempDir()
|
||||
resolved, err := ResolveGenerate(GenerateRequest{
|
||||
Config: cfg,
|
||||
Report: ReportDaily,
|
||||
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||
}, mustParse("2026-05-29T05:00:00-05:00"))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveGenerate() error = %v", err)
|
||||
}
|
||||
notifier := &recordingNotifier{}
|
||||
|
||||
_, err = GenerateReport(context.Background(), ReportRequest{
|
||||
Config: cfg,
|
||||
Resolved: resolved,
|
||||
Renderer: successfulRenderer("# Daily Report\n"),
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateReport() error = %v", err)
|
||||
}
|
||||
if len(notifier.requests) != 0 {
|
||||
t.Fatalf("notification requests = %#v, want none when disabled", notifier.requests)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateReportNotifiesManagedReportPath(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
cfg := dailyTestConfig(t, server)
|
||||
cfg.Workspace.Root = t.TempDir()
|
||||
cfg.Notify.Distributor.Enabled = true
|
||||
resolved, err := ResolveGenerate(GenerateRequest{
|
||||
Config: cfg,
|
||||
Report: ReportDaily,
|
||||
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||
}, mustParse("2026-05-29T05:00:00-05:00"))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveGenerate() error = %v", err)
|
||||
}
|
||||
notifier := &recordingNotifier{
|
||||
result: &NotificationResult{RunID: "distributor-run", Status: "accepted"},
|
||||
}
|
||||
outputPath := filepath.Join(t.TempDir(), "daily-copy.md")
|
||||
|
||||
result, err := GenerateReport(context.Background(), ReportRequest{
|
||||
Config: cfg,
|
||||
Resolved: resolved,
|
||||
OutputPath: outputPath,
|
||||
Renderer: successfulRenderer("# Daily Report\n"),
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateReport() error = %v", err)
|
||||
}
|
||||
if result.Notification == nil {
|
||||
t.Fatal("Notification = nil, want notification result")
|
||||
}
|
||||
if result.Notification.RunID != "distributor-run" || result.Notification.Status != "accepted" {
|
||||
t.Fatalf("Notification = %#v, want accepted distributor run", result.Notification)
|
||||
}
|
||||
if len(notifier.requests) != 1 {
|
||||
t.Fatalf("notification requests = %d, want 1", len(notifier.requests))
|
||||
}
|
||||
req := notifier.requests[0]
|
||||
if req.ReportPath != result.ReportPath {
|
||||
t.Fatalf("notification ReportPath = %q, want managed path %q", req.ReportPath, result.ReportPath)
|
||||
}
|
||||
if req.ReportPath == outputPath {
|
||||
t.Fatalf("notification used output copy %q, want managed report path", outputPath)
|
||||
}
|
||||
if req.BundlePath != "daily.md" {
|
||||
t.Fatalf("notification BundlePath = %q, want daily.md", req.BundlePath)
|
||||
}
|
||||
if req.BundleID != "weatherreporter.home.daily_today."+result.Metadata.RunID {
|
||||
t.Fatalf("notification BundleID = %q, want default template", req.BundleID)
|
||||
}
|
||||
if req.IdempotencyKey != req.BundleID {
|
||||
t.Fatalf("IdempotencyKey = %q, want bundle id %q", req.IdempotencyKey, req.BundleID)
|
||||
}
|
||||
if req.RunID != result.Metadata.RunID {
|
||||
t.Fatalf("notification RunID = %q, want report run id %q", req.RunID, result.Metadata.RunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateReportNotificationFailureFailsReport(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
cfg := dailyTestConfig(t, server)
|
||||
cfg.Workspace.Root = t.TempDir()
|
||||
cfg.Notify.Distributor.Enabled = true
|
||||
resolved, err := ResolveGenerate(GenerateRequest{
|
||||
Config: cfg,
|
||||
Report: ReportDaily,
|
||||
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||
}, mustParse("2026-05-29T05:00:00-05:00"))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveGenerate() error = %v", err)
|
||||
}
|
||||
notifier := &recordingNotifier{err: errors.New("upload rejected")}
|
||||
|
||||
_, err = GenerateReport(context.Background(), ReportRequest{
|
||||
Config: cfg,
|
||||
Resolved: resolved,
|
||||
Renderer: successfulRenderer("# Daily Report\n"),
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("GenerateReport() error = nil, want notification error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "notify report") || !strings.Contains(err.Error(), "upload rejected") {
|
||||
t.Fatalf("error = %q, want notification context", err.Error())
|
||||
}
|
||||
if len(notifier.requests) != 1 {
|
||||
t.Fatalf("notification requests = %d, want one attempted notification", len(notifier.requests))
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateReportDoesNotNotifyAfterRenderOrRunFailure(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
tests := []struct {
|
||||
name string
|
||||
renderer Renderer
|
||||
}{
|
||||
{
|
||||
name: "Render",
|
||||
renderer: &recordingRenderer{
|
||||
renderResult: &scriptorium.RenderResult{ExitCode: 1, Stderr: "render failed"},
|
||||
err: errors.New("render failed"),
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "Run",
|
||||
renderer: &recordingRenderer{
|
||||
renderResult: &scriptorium.RenderResult{ExitCode: 0},
|
||||
runResult: &scriptorium.RunResult{ExitCode: 2, Stderr: "run failed"},
|
||||
runErr: errors.New("run failed"),
|
||||
runBody: "# Daily Report\n",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := dailyTestConfig(t, server)
|
||||
cfg.Workspace.Root = t.TempDir()
|
||||
cfg.Notify.Distributor.Enabled = true
|
||||
resolved, err := ResolveGenerate(GenerateRequest{
|
||||
Config: cfg,
|
||||
Report: ReportDaily,
|
||||
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||
}, mustParse("2026-05-29T05:00:00-05:00"))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveGenerate() error = %v", err)
|
||||
}
|
||||
notifier := &recordingNotifier{}
|
||||
|
||||
_, err = GenerateReport(context.Background(), ReportRequest{
|
||||
Config: cfg,
|
||||
Resolved: resolved,
|
||||
Renderer: tt.renderer,
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("GenerateReport() error = nil, want generation error")
|
||||
}
|
||||
if len(notifier.requests) != 0 {
|
||||
t.Fatalf("notification requests = %#v, want none after generation failure", notifier.requests)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateReportDoesNotNotifyAfterFetchFailure(t *testing.T) {
|
||||
cfg := config.Defaults()
|
||||
cfg.WeatherAPI.BaseURL = ""
|
||||
cfg.WeatherAPI.Timezone = "America/Chicago"
|
||||
cfg.Workspace.Root = t.TempDir()
|
||||
cfg.Notify.Distributor.Enabled = true
|
||||
resolved, err := ResolveGenerate(GenerateRequest{
|
||||
Config: cfg,
|
||||
Report: ReportDaily,
|
||||
Date: mustParse("2026-05-29T12:00:00-05:00"),
|
||||
}, mustParse("2026-05-29T05:00:00-05:00"))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveGenerate() error = %v", err)
|
||||
}
|
||||
notifier := &recordingNotifier{}
|
||||
|
||||
_, err = GenerateReport(context.Background(), ReportRequest{
|
||||
Config: cfg,
|
||||
Resolved: resolved,
|
||||
Renderer: successfulRenderer("# Daily Report\n"),
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("GenerateReport() error = nil, want fetch setup error")
|
||||
}
|
||||
if len(notifier.requests) != 0 {
|
||||
t.Fatalf("notification requests = %#v, want none after fetch failure", notifier.requests)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateReportPersistsFailedPreflight(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
cfg := config.Defaults()
|
||||
@@ -906,6 +1117,8 @@ func dailyBundleServer(t *testing.T) *httptest.Server {
|
||||
_, _ = w.Write([]byte(`{"data":{"alerts":[{"event":"Flood Watch","effective":"2026-05-29T05:00:00-05:00","expires":"2026-05-29T09:00:00-05:00"}]}}`))
|
||||
case "/discussion":
|
||||
_, _ = w.Write([]byte(`{"data":{"product":"discussion","issuedAt":"2026-05-29T09:25:00-05:00","keyMessages":["Storms are most likely during the morning."],"shortTerm":{"qualifier":"(Short Term)","text":"Short-term AFD narrative for generated report."},"longTerm":{"qualifier":"(Long Term)","text":"Long-term AFD narrative for generated report."}}}`))
|
||||
case "/weatherstories/latest":
|
||||
_, _ = w.Write([]byte(`{"data":{"officeId":"LSX","startTime":"2026-05-30T08:46:00Z","endTime":"2026-05-31T11:00:00Z","updatedAt":"2026-05-30T09:00:34Z","title":"Several Chances for Rain Through Monday","description":"Scattered showers and thunderstorms remain possible.","altText":"Forecast weather story graphic.","priority":false,"order":1,"downloadUrl":"https://api.weather.gov/offices/LSX/weatherstories/download/test"}}`))
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
@@ -998,6 +1211,62 @@ func TestRunBatchContinuesAfterReportFailure(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunBatchContinuesAfterNotificationFailure(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
cfg := config.Defaults()
|
||||
cfg.WeatherAPI.BaseURL = server.URL + "/"
|
||||
cfg.WeatherAPI.Timezone = "America/Chicago"
|
||||
cfg.Workspace.Root = t.TempDir()
|
||||
cfg.Notify.Distributor.Enabled = true
|
||||
notifier := &recordingNotifier{
|
||||
errByReport: map[report.ID]error{
|
||||
report.ThreeDay: errors.New("distributor unavailable"),
|
||||
},
|
||||
}
|
||||
|
||||
result, err := RunBatchDetailed(context.Background(), BatchRequest{
|
||||
Config: cfg,
|
||||
Batch: BatchMorning,
|
||||
Now: mustParse("2026-05-29T05:00:00-05:00"),
|
||||
Renderer: &selectiveRenderer{runBody: "# Batch Report\n"},
|
||||
Notifier: notifier,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("RunBatchDetailed() error = %v", err)
|
||||
}
|
||||
|
||||
if result.Total != 3 || result.Succeeded != 2 || result.Failed != 1 {
|
||||
t.Fatalf("summary total/succeeded/failed = %d/%d/%d, want 3/2/1", result.Total, result.Succeeded, result.Failed)
|
||||
}
|
||||
if len(notifier.requests) != 3 {
|
||||
t.Fatalf("notification requests = %d, want one per generated report", len(notifier.requests))
|
||||
}
|
||||
var failedThreeDay bool
|
||||
for _, item := range result.Reports {
|
||||
if item.ReportID == report.ThreeDay {
|
||||
if item.Status == "failed" && strings.Contains(item.Error, "notify report") && strings.Contains(item.Error, "distributor unavailable") {
|
||||
failedThreeDay = true
|
||||
}
|
||||
if item.NotificationStatus != "failed" {
|
||||
t.Fatalf("3-day notification status = %q, want failed", item.NotificationStatus)
|
||||
}
|
||||
if !strings.Contains(item.NotificationError, "distributor unavailable") {
|
||||
t.Fatalf("3-day notification error = %q, want distributor unavailable", item.NotificationError)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if item.Status != "succeeded" {
|
||||
t.Fatalf("report %s status = %s, want succeeded", item.ReportID, item.Status)
|
||||
}
|
||||
if item.NotificationStatus != "accepted" {
|
||||
t.Fatalf("report %s notification status = %q, want accepted", item.ReportID, item.NotificationStatus)
|
||||
}
|
||||
}
|
||||
if !failedThreeDay {
|
||||
t.Fatalf("reports = %#v, want notification failure on 3-day item", result.Reports)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunBatchUsesOutputDirectory(t *testing.T) {
|
||||
server := dailyBundleServer(t)
|
||||
cfg := config.Defaults()
|
||||
@@ -1154,6 +1423,14 @@ type recordingRenderer struct {
|
||||
runBody string
|
||||
}
|
||||
|
||||
func successfulRenderer(body string) *recordingRenderer {
|
||||
return &recordingRenderer{
|
||||
renderResult: &scriptorium.RenderResult{ExitCode: 0},
|
||||
runResult: &scriptorium.RunResult{ExitCode: 0},
|
||||
runBody: body,
|
||||
}
|
||||
}
|
||||
|
||||
type selectiveRenderer struct {
|
||||
renderCalls int
|
||||
runCalls int
|
||||
@@ -1161,6 +1438,38 @@ type selectiveRenderer struct {
|
||||
runBody string
|
||||
}
|
||||
|
||||
type recordingNotifier struct {
|
||||
requests []NotificationRequest
|
||||
result *NotificationResult
|
||||
err error
|
||||
errByReport map[report.ID]error
|
||||
}
|
||||
|
||||
func (n *recordingNotifier) Notify(_ context.Context, req NotificationRequest) (*NotificationResult, error) {
|
||||
n.requests = append(n.requests, req)
|
||||
if err := n.errByReport[req.ReportID]; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if n.err != nil {
|
||||
return nil, n.err
|
||||
}
|
||||
if n.result != nil {
|
||||
result := *n.result
|
||||
if result.BundleID == "" {
|
||||
result.BundleID = req.BundleID
|
||||
}
|
||||
if result.IdempotencyKey == "" {
|
||||
result.IdempotencyKey = req.IdempotencyKey
|
||||
}
|
||||
return &result, nil
|
||||
}
|
||||
return &NotificationResult{
|
||||
BundleID: req.BundleID,
|
||||
IdempotencyKey: req.IdempotencyKey,
|
||||
Status: "accepted",
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (r *selectiveRenderer) Render(_ context.Context, req scriptorium.RenderRequest) (*scriptorium.RenderResult, error) {
|
||||
r.renderCalls++
|
||||
if req.PromptID == r.failRenderPrompt {
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"math"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/forecast"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
|
||||
@@ -57,8 +58,17 @@ type DiscussionContext struct {
|
||||
}
|
||||
|
||||
type WeatherStoryContext struct {
|
||||
Available bool `json:"available"`
|
||||
Summary string `json:"summary,omitempty"`
|
||||
Available bool `json:"available"`
|
||||
OfficeID string `json:"officeId,omitempty"`
|
||||
StartTime time.Time `json:"startTime"`
|
||||
EndTime time.Time `json:"endTime"`
|
||||
UpdatedAt *time.Time `json:"updatedAt,omitempty"`
|
||||
Title string `json:"title,omitempty"`
|
||||
Description string `json:"description,omitempty"`
|
||||
AltText string `json:"altText,omitempty"`
|
||||
Priority bool `json:"priority"`
|
||||
Order int `json:"order"`
|
||||
DownloadURL string `json:"downloadUrl,omitempty"`
|
||||
}
|
||||
|
||||
func BuildDaily(ctx BuildContext, summary *forecast.DailySummary) (Package, error) {
|
||||
@@ -258,10 +268,31 @@ func buildDiscussion(discussion *forecast.Discussion) DiscussionContext {
|
||||
}
|
||||
|
||||
func buildWeatherStory(bundle *forecast.Bundle) *WeatherStoryContext {
|
||||
if bundle == nil || bundle.WeatherStory == nil || len(bundle.WeatherStory.Raw) == 0 {
|
||||
if bundle == nil || bundle.WeatherStory == nil {
|
||||
return nil
|
||||
}
|
||||
return &WeatherStoryContext{Available: true, Summary: string(bundle.WeatherStory.Raw)}
|
||||
story := bundle.WeatherStory
|
||||
return &WeatherStoryContext{
|
||||
Available: true,
|
||||
OfficeID: story.OfficeID,
|
||||
StartTime: story.StartTime,
|
||||
EndTime: story.EndTime,
|
||||
UpdatedAt: copyTime(story.UpdatedAt),
|
||||
Title: story.Title,
|
||||
Description: story.Description,
|
||||
AltText: story.AltText,
|
||||
Priority: story.Priority,
|
||||
Order: story.Order,
|
||||
DownloadURL: story.DownloadURL,
|
||||
}
|
||||
}
|
||||
|
||||
func copyTime(value *time.Time) *time.Time {
|
||||
if value == nil {
|
||||
return nil
|
||||
}
|
||||
copied := *value
|
||||
return &copied
|
||||
}
|
||||
|
||||
func scoreOutdoorWindow(daypart forecast.DaypartSummary) OutdoorWindow {
|
||||
|
||||
@@ -162,8 +162,12 @@ func stormConfidenceInputs(bundle *forecast.Bundle) []string {
|
||||
items = appendUnique(items, "Short-term discussion is available for confidence context.")
|
||||
}
|
||||
}
|
||||
if bundle.WeatherStory != nil && len(bundle.WeatherStory.Raw) > 0 {
|
||||
items = appendUnique(items, "Weather story source is available.")
|
||||
if bundle.WeatherStory != nil {
|
||||
if bundle.WeatherStory.Title != "" {
|
||||
items = appendUnique(items, "Weather story: "+bundle.WeatherStory.Title+".")
|
||||
} else {
|
||||
items = appendUnique(items, "Weather story source is available.")
|
||||
}
|
||||
}
|
||||
for _, warning := range bundle.Warnings {
|
||||
if warning.Code != "" {
|
||||
|
||||
@@ -50,8 +50,16 @@ func TestStormBriefingWithActiveAlert(t *testing.T) {
|
||||
ShortTerm: &forecast.DiscussionSection{Text: "Short-term storm coverage peaks this morning."},
|
||||
LongTerm: &forecast.DiscussionSection{Text: "Long-term pattern stays unsettled after the event."},
|
||||
},
|
||||
WeatherStory: &forecast.WeatherStory{Raw: json.RawMessage(`{"headline":"Storm risk"}`)},
|
||||
Sources: []forecast.Source{{Name: "hourly", FetchedAt: time.Now()}},
|
||||
WeatherStory: &forecast.WeatherStory{
|
||||
OfficeID: "LSX",
|
||||
StartTime: mustParse("2026-05-29T06:00:00Z"),
|
||||
EndTime: mustParse("2026-05-29T18:00:00Z"),
|
||||
Title: "Storm Risk",
|
||||
Description: "Strong storms are possible.",
|
||||
AltText: "Weather story graphic showing storm risk.",
|
||||
Order: 1,
|
||||
},
|
||||
Sources: []forecast.Source{{Name: "hourly", FetchedAt: time.Now()}},
|
||||
}
|
||||
|
||||
pkg, err := BuildStorm(BuildContext{Resolved: resolved, Bundle: bundle, Units: "us", Timezone: "America/Chicago"})
|
||||
@@ -80,6 +88,9 @@ func TestStormBriefingWithActiveAlert(t *testing.T) {
|
||||
if pkg.Storm.WeatherStory == nil {
|
||||
t.Fatal("WeatherStory = nil, want available story context")
|
||||
}
|
||||
if pkg.Storm.WeatherStory.Title != "Storm Risk" || pkg.Storm.WeatherStory.Description != "Strong storms are possible." {
|
||||
t.Fatalf("WeatherStory = %#v, want structured story context", pkg.Storm.WeatherStory)
|
||||
}
|
||||
if pkg.Storm.Discussion.ShortTerm != "Short-term storm coverage peaks this morning." {
|
||||
t.Fatalf("Discussion.ShortTerm = %q, want short-term AFD narrative", pkg.Storm.Discussion.ShortTerm)
|
||||
}
|
||||
|
||||
@@ -352,11 +352,21 @@ func writeRunLogs(stderr io.Writer, result *app.BatchResult) {
|
||||
return
|
||||
}
|
||||
for _, item := range result.Reports {
|
||||
notificationFields := ""
|
||||
if item.NotificationStatus != "" {
|
||||
notificationFields += fmt.Sprintf(" notificationStatus=%q", item.NotificationStatus)
|
||||
}
|
||||
if item.NotificationRunID != "" {
|
||||
notificationFields += fmt.Sprintf(" notificationRunId=%q", item.NotificationRunID)
|
||||
}
|
||||
if item.NotificationError != "" {
|
||||
notificationFields += fmt.Sprintf(" notificationError=%q", item.NotificationError)
|
||||
}
|
||||
if item.Status == "failed" {
|
||||
_, _ = fmt.Fprintf(stderr, "report=%s status=failed error=%q\n", item.ReportID, item.Error)
|
||||
_, _ = fmt.Fprintf(stderr, "report=%s status=failed error=%q%s\n", item.ReportID, item.Error, notificationFields)
|
||||
continue
|
||||
}
|
||||
_, _ = fmt.Fprintf(stderr, "report=%s status=succeeded output=%q\n", item.ReportID, item.OutputPath)
|
||||
_, _ = fmt.Fprintf(stderr, "report=%s status=succeeded output=%q%s\n", item.ReportID, item.OutputPath, notificationFields)
|
||||
}
|
||||
_, _ = fmt.Fprintf(stderr, "batch=%s total=%d succeeded=%d failed=%d\n", result.Batch, result.Total, result.Succeeded, result.Failed)
|
||||
}
|
||||
|
||||
@@ -346,6 +346,88 @@ func TestRunMorningReportsPartialFailureAndContinues(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestBatchOutputIncludesNotificationDetails(t *testing.T) {
|
||||
result := &app.BatchResult{
|
||||
Batch: app.BatchMorning,
|
||||
Total: 2,
|
||||
Succeeded: 1,
|
||||
Failed: 1,
|
||||
Reports: []app.BatchReportResult{
|
||||
{
|
||||
ReportID: "daily_today",
|
||||
Status: "succeeded",
|
||||
OutputPath: "/tmp/daily.md",
|
||||
NotificationStatus: "accepted",
|
||||
NotificationRunID: "distributor-run-1",
|
||||
},
|
||||
{
|
||||
ReportID: "three_day",
|
||||
Status: "failed",
|
||||
Error: "notify report three_day: upload failed",
|
||||
NotificationStatus: "failed",
|
||||
NotificationError: "notify report three_day: upload failed",
|
||||
},
|
||||
},
|
||||
}
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
if err := writeJSON(&stdout, result); err != nil {
|
||||
t.Fatalf("writeJSON() error = %v", err)
|
||||
}
|
||||
writeRunLogs(&stderr, result)
|
||||
|
||||
var decoded app.BatchResult
|
||||
if err := json.Unmarshal(stdout.Bytes(), &decoded); err != nil {
|
||||
t.Fatalf("decode batch JSON: %v\n%s", err, stdout.String())
|
||||
}
|
||||
if decoded.Reports[0].NotificationStatus != "accepted" || decoded.Reports[0].NotificationRunID != "distributor-run-1" {
|
||||
t.Fatalf("success notification fields = %#v", decoded.Reports[0])
|
||||
}
|
||||
if decoded.Reports[1].NotificationStatus != "failed" || !strings.Contains(decoded.Reports[1].NotificationError, "upload failed") {
|
||||
t.Fatalf("failure notification fields = %#v", decoded.Reports[1])
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `notificationStatus="accepted"`) || !strings.Contains(stderr.String(), `notificationRunId="distributor-run-1"`) {
|
||||
t.Fatalf("stderr missing success notification fields:\n%s", stderr.String())
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `notificationStatus="failed"`) || !strings.Contains(stderr.String(), `notificationError="notify report three_day: upload failed"`) {
|
||||
t.Fatalf("stderr missing failure notification fields:\n%s", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestBatchOutputDoesNotExposeSecretLikeNotificationErrors(t *testing.T) {
|
||||
result := &app.BatchResult{
|
||||
Batch: app.BatchMorning,
|
||||
Total: 1,
|
||||
Failed: 1,
|
||||
Reports: []app.BatchReportResult{
|
||||
{
|
||||
ReportID: "daily_today",
|
||||
Status: "failed",
|
||||
Error: "notify report daily_today: upload failed: [redacted]",
|
||||
NotificationStatus: "failed",
|
||||
NotificationError: "notify report daily_today: upload failed: [redacted]",
|
||||
},
|
||||
},
|
||||
}
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
if err := writeJSON(&stdout, result); err != nil {
|
||||
t.Fatalf("writeJSON() error = %v", err)
|
||||
}
|
||||
writeRunLogs(&stderr, result)
|
||||
|
||||
for _, output := range []string{stdout.String(), stderr.String()} {
|
||||
if strings.Contains(output, "DISTRIBUTOR_SECRET_TOKEN") {
|
||||
t.Fatalf("output contains token value:\n%s", output)
|
||||
}
|
||||
if !strings.Contains(output, "[redacted]") {
|
||||
t.Fatalf("output missing redacted marker:\n%s", output)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunEveningUsesOutputDirectoryAndSummary(t *testing.T) {
|
||||
server := dailyServer(t)
|
||||
tempDir := t.TempDir()
|
||||
@@ -384,6 +466,103 @@ func TestRunEveningUsesOutputDirectoryAndSummary(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunEveningReportsNotificationSuccess(t *testing.T) {
|
||||
server := dailyServer(t)
|
||||
distributorServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/upload" {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusAccepted)
|
||||
_, _ = w.Write([]byte(`{"run_id":"distributor-run-1","status":"accepted"}`))
|
||||
}))
|
||||
t.Cleanup(distributorServer.Close)
|
||||
tempDir := t.TempDir()
|
||||
scriptoriumPath := writeFakeScriptorium(t, tempDir)
|
||||
configPath := filepath.Join(tempDir, "config.yml")
|
||||
workspaceRoot := filepath.Join(tempDir, "workspace")
|
||||
configBody := "weather_api:\n base_url: " + server.URL + "/\n timezone: America/Chicago\nscriptorium:\n binary: " + scriptoriumPath + "\nworkspace:\n root: " + workspaceRoot + "\nnotify:\n distributor:\n enabled: true\n endpoint: " + distributorServer.URL + "\n token_env: CLI_DISTRIBUTOR_TOKEN\n"
|
||||
if err := os.WriteFile(configPath, []byte(configBody), 0o600); err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
t.Setenv("CLI_DISTRIBUTOR_TOKEN", "cli-secret-token")
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
runner := Runner{Clock: fixedClock()}
|
||||
|
||||
err := runner.Run(context.Background(), []string{
|
||||
"run", "evening",
|
||||
"--config", configPath,
|
||||
}, &stdout, &stderr)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
|
||||
var summary app.BatchResult
|
||||
if decodeErr := json.Unmarshal(stdout.Bytes(), &summary); decodeErr != nil {
|
||||
t.Fatalf("decode summary: %v\n%s", decodeErr, stdout.String())
|
||||
}
|
||||
if len(summary.Reports) != 1 {
|
||||
t.Fatalf("reports = %#v, want one report", summary.Reports)
|
||||
}
|
||||
if summary.Reports[0].NotificationStatus != "accepted" || summary.Reports[0].NotificationRunID != "distributor-run-1" {
|
||||
t.Fatalf("notification fields = %#v", summary.Reports[0])
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `notificationStatus="accepted"`) || !strings.Contains(stderr.String(), `notificationRunId="distributor-run-1"`) {
|
||||
t.Fatalf("stderr missing notification fields:\n%s", stderr.String())
|
||||
}
|
||||
if strings.Contains(stdout.String(), "cli-secret-token") || strings.Contains(stderr.String(), "cli-secret-token") {
|
||||
t.Fatalf("output contains token value\nstdout=%s\nstderr=%s", stdout.String(), stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunEveningReportsNotificationFailureWithoutToken(t *testing.T) {
|
||||
server := dailyServer(t)
|
||||
distributorServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
_, _ = w.Write([]byte(`{"error":"rejected cli-secret-token","retryable":false}`))
|
||||
}))
|
||||
t.Cleanup(distributorServer.Close)
|
||||
tempDir := t.TempDir()
|
||||
scriptoriumPath := writeFakeScriptorium(t, tempDir)
|
||||
configPath := filepath.Join(tempDir, "config.yml")
|
||||
workspaceRoot := filepath.Join(tempDir, "workspace")
|
||||
configBody := "weather_api:\n base_url: " + server.URL + "/\n timezone: America/Chicago\nscriptorium:\n binary: " + scriptoriumPath + "\nworkspace:\n root: " + workspaceRoot + "\nnotify:\n distributor:\n enabled: true\n endpoint: " + distributorServer.URL + "\n token_env: CLI_DISTRIBUTOR_TOKEN\n"
|
||||
if err := os.WriteFile(configPath, []byte(configBody), 0o600); err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
t.Setenv("CLI_DISTRIBUTOR_TOKEN", "cli-secret-token")
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
runner := Runner{Clock: fixedClock()}
|
||||
|
||||
err := runner.Run(context.Background(), []string{
|
||||
"run", "evening",
|
||||
"--config", configPath,
|
||||
}, &stdout, &stderr)
|
||||
if err == nil {
|
||||
t.Fatal("Run() error = nil, want notification failure")
|
||||
}
|
||||
|
||||
var summary app.BatchResult
|
||||
if decodeErr := json.Unmarshal(stdout.Bytes(), &summary); decodeErr != nil {
|
||||
t.Fatalf("decode summary: %v\n%s", decodeErr, stdout.String())
|
||||
}
|
||||
if len(summary.Reports) != 1 || summary.Reports[0].NotificationStatus != "failed" {
|
||||
t.Fatalf("summary reports = %#v, want failed notification", summary.Reports)
|
||||
}
|
||||
for _, output := range []string{stdout.String(), stderr.String(), err.Error()} {
|
||||
if strings.Contains(output, "cli-secret-token") {
|
||||
t.Fatalf("output contains token value:\n%s", output)
|
||||
}
|
||||
}
|
||||
for _, output := range []string{stdout.String(), stderr.String()} {
|
||||
if !strings.Contains(output, "[redacted]") {
|
||||
t.Fatalf("output missing redaction marker:\n%s", output)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunMorningGeneratesDailyAndThreeDayOnSunday(t *testing.T) {
|
||||
server := dailyServer(t)
|
||||
tempDir := t.TempDir()
|
||||
@@ -813,6 +992,8 @@ func dailyServer(t *testing.T) *httptest.Server {
|
||||
_, _ = w.Write([]byte(`{"data":{"alerts":[]}}`))
|
||||
case "/discussion":
|
||||
_, _ = w.Write([]byte(`{"data":{"product":"discussion","issuedAt":"2026-05-29T09:25:00-05:00","keyMessages":["Storms are most likely during the morning."]}}`))
|
||||
case "/weatherstories/latest":
|
||||
_, _ = w.Write([]byte(`{"data":{"officeId":"LSX","startTime":"2026-05-30T08:46:00Z","endTime":"2026-05-31T11:00:00Z","updatedAt":"2026-05-30T09:00:34Z","title":"Several Chances for Rain Through Monday","description":"Scattered showers and thunderstorms remain possible.","altText":"Forecast weather story graphic.","priority":false,"order":1,"downloadUrl":"https://api.weather.gov/offices/LSX/weatherstories/download/test"}}`))
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
|
||||
@@ -5,16 +5,21 @@ package config
|
||||
import "time"
|
||||
|
||||
type MissingSourcePolicy string
|
||||
type NotifyFailurePolicy string
|
||||
|
||||
const (
|
||||
MissingSourceError MissingSourcePolicy = "error"
|
||||
MissingSourceWarn MissingSourcePolicy = "warn"
|
||||
MissingSourceNone MissingSourcePolicy = "none"
|
||||
|
||||
NotifyFailureError NotifyFailurePolicy = "error"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
WeatherAPI WeatherAPIConfig `yaml:"weather_api"`
|
||||
Location LocationConfig `yaml:"location"`
|
||||
Secrets SecretsConfig `yaml:"secrets"`
|
||||
Notify NotifyConfig `yaml:"notify"`
|
||||
MissingSource MissingSourceConfig `yaml:"missing_source"`
|
||||
Scriptorium ScriptoriumConfig `yaml:"scriptorium"`
|
||||
Workspace WorkspaceConfig `yaml:"workspace"`
|
||||
@@ -37,6 +42,25 @@ type LocationConfig struct {
|
||||
Region string `yaml:"region"`
|
||||
}
|
||||
|
||||
type SecretsConfig struct {
|
||||
Directory string `yaml:"directory"`
|
||||
}
|
||||
|
||||
type NotifyConfig struct {
|
||||
Distributor DistributorNotifyConfig `yaml:"distributor"`
|
||||
}
|
||||
|
||||
type DistributorNotifyConfig struct {
|
||||
Enabled bool `yaml:"enabled"`
|
||||
Endpoint string `yaml:"endpoint"`
|
||||
TokenEnv string `yaml:"token_env"`
|
||||
Timeout time.Duration `yaml:"timeout"`
|
||||
FailurePolicy NotifyFailurePolicy `yaml:"failure_policy"`
|
||||
BundleIDTemplate string `yaml:"bundle_id_template"`
|
||||
IdempotencyKeyTemplate string `yaml:"idempotency_key_template"`
|
||||
ReportPathTemplate string `yaml:"report_path_template"`
|
||||
}
|
||||
|
||||
type MissingSourceConfig struct {
|
||||
Default MissingSourcePolicy `yaml:"default"`
|
||||
Sources map[string]MissingSourcePolicy `yaml:"sources"`
|
||||
|
||||
@@ -26,6 +26,33 @@ func TestDefaults(t *testing.T) {
|
||||
if cfg.Location.ID != "home" || cfg.Location.Name != "Brentwood" || cfg.Location.Region != "St. Louis Metro" {
|
||||
t.Fatalf("Location = %#v, want home/Brentwood/St. Louis Metro", cfg.Location)
|
||||
}
|
||||
if cfg.Secrets.Directory != "" {
|
||||
t.Fatalf("Secrets.Directory = %q, want empty", cfg.Secrets.Directory)
|
||||
}
|
||||
if cfg.Notify.Distributor.Enabled {
|
||||
t.Fatalf("Notify.Distributor.Enabled = true, want false")
|
||||
}
|
||||
if cfg.Notify.Distributor.Endpoint != "https://distributor.example.com" {
|
||||
t.Fatalf("Notify.Distributor.Endpoint = %q, want default endpoint", cfg.Notify.Distributor.Endpoint)
|
||||
}
|
||||
if cfg.Notify.Distributor.TokenEnv != "DISTRIBUTOR_UPLOAD_TOKEN" {
|
||||
t.Fatalf("Notify.Distributor.TokenEnv = %q, want DISTRIBUTOR_UPLOAD_TOKEN", cfg.Notify.Distributor.TokenEnv)
|
||||
}
|
||||
if cfg.Notify.Distributor.Timeout != 30*time.Second {
|
||||
t.Fatalf("Notify.Distributor.Timeout = %s, want 30s", cfg.Notify.Distributor.Timeout)
|
||||
}
|
||||
if cfg.Notify.Distributor.FailurePolicy != NotifyFailureError {
|
||||
t.Fatalf("Notify.Distributor.FailurePolicy = %q, want error", cfg.Notify.Distributor.FailurePolicy)
|
||||
}
|
||||
if cfg.Notify.Distributor.BundleIDTemplate != "weatherreporter.{location_id}.{report_id}.{run_id}" {
|
||||
t.Fatalf("Notify.Distributor.BundleIDTemplate = %q, want default", cfg.Notify.Distributor.BundleIDTemplate)
|
||||
}
|
||||
if cfg.Notify.Distributor.IdempotencyKeyTemplate != "{bundle_id}" {
|
||||
t.Fatalf("Notify.Distributor.IdempotencyKeyTemplate = %q, want default", cfg.Notify.Distributor.IdempotencyKeyTemplate)
|
||||
}
|
||||
if cfg.Notify.Distributor.ReportPathTemplate != "{batch_output_name}" {
|
||||
t.Fatalf("Notify.Distributor.ReportPathTemplate = %q, want default", cfg.Notify.Distributor.ReportPathTemplate)
|
||||
}
|
||||
if cfg.MissingSource.Default != MissingSourceWarn {
|
||||
t.Fatalf("MissingSource.Default = %q, want warn", cfg.MissingSource.Default)
|
||||
}
|
||||
@@ -112,3 +139,417 @@ func TestLoadAppliesOverrides(t *testing.T) {
|
||||
t.Fatalf("Timezone = %q, want +09:30", cfg.WeatherAPI.Timezone)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDisabledDistributorNotifyAcceptsOmittedFields(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "config.yml")
|
||||
if err := os.WriteFile(path, []byte("notify:\n distributor:\n enabled: false\n"), 0o600); err != nil {
|
||||
t.Fatalf("write config fixture: %v", err)
|
||||
}
|
||||
|
||||
cfg, err := LoadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadFile() error = %v", err)
|
||||
}
|
||||
if cfg.Notify.Distributor.Enabled {
|
||||
t.Fatalf("Notify.Distributor.Enabled = true, want false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnabledDistributorNotifyValidation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*Config)
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "Endpoint",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.Endpoint = "distributor.example.com"
|
||||
},
|
||||
wantErr: "notify.distributor.endpoint",
|
||||
},
|
||||
{
|
||||
name: "TokenEnvEmpty",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.TokenEnv = ""
|
||||
},
|
||||
wantErr: "notify.distributor.token_env",
|
||||
},
|
||||
{
|
||||
name: "TokenEnvInvalid",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.TokenEnv = "1TOKEN"
|
||||
},
|
||||
wantErr: "notify.distributor.token_env",
|
||||
},
|
||||
{
|
||||
name: "Timeout",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.Timeout = 0
|
||||
},
|
||||
wantErr: "notify.distributor.timeout",
|
||||
},
|
||||
{
|
||||
name: "FailurePolicy",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.FailurePolicy = "warn"
|
||||
},
|
||||
wantErr: "notify.distributor.failure_policy",
|
||||
},
|
||||
{
|
||||
name: "BundleTemplate",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.BundleIDTemplate = "{unknown}"
|
||||
},
|
||||
wantErr: "notify.distributor.bundle_id_template",
|
||||
},
|
||||
{
|
||||
name: "IdempotencyTemplate",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.IdempotencyKeyTemplate = "{unknown}"
|
||||
},
|
||||
wantErr: "notify.distributor.idempotency_key_template",
|
||||
},
|
||||
{
|
||||
name: "ReportPathTemplateUnknown",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.ReportPathTemplate = "{bundle_id}"
|
||||
},
|
||||
wantErr: "notify.distributor.report_path_template",
|
||||
},
|
||||
{
|
||||
name: "ReportPathTemplateInvalidPath",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Notify.Distributor.ReportPathTemplate = "/{batch_output_name}"
|
||||
},
|
||||
wantErr: "notify.distributor.report_path_template",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := Defaults()
|
||||
cfg.Notify.Distributor.Enabled = true
|
||||
tt.mutate(&cfg)
|
||||
|
||||
err := Validate(cfg)
|
||||
if err == nil {
|
||||
t.Fatal("Validate() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDistributorTemplateRendering(t *testing.T) {
|
||||
values := DistributorTemplateValues{
|
||||
LocationID: "home",
|
||||
ReportID: "daily",
|
||||
RunID: "20260607T120000Z",
|
||||
ArtifactGroup: "daily",
|
||||
BatchOutputName: "daily.md",
|
||||
BundleID: "weatherreporter.home.daily.20260607T120000Z",
|
||||
}
|
||||
|
||||
bundleID, err := RenderDistributorBundleID("weatherreporter.{location_id}.{report_id}.{run_id}", values)
|
||||
if err != nil {
|
||||
t.Fatalf("RenderDistributorBundleID() error = %v", err)
|
||||
}
|
||||
if bundleID != "weatherreporter.home.daily.20260607T120000Z" {
|
||||
t.Fatalf("bundleID = %q, want rendered value", bundleID)
|
||||
}
|
||||
|
||||
idempotencyKey, err := RenderDistributorIdempotencyKey("{bundle_id}", values)
|
||||
if err != nil {
|
||||
t.Fatalf("RenderDistributorIdempotencyKey() error = %v", err)
|
||||
}
|
||||
if idempotencyKey != "weatherreporter.home.daily.20260607T120000Z" {
|
||||
t.Fatalf("idempotencyKey = %q, want rendered bundle ID", idempotencyKey)
|
||||
}
|
||||
|
||||
reportPath, err := RenderDistributorReportPath("reports/{batch_output_name}", values)
|
||||
if err != nil {
|
||||
t.Fatalf("RenderDistributorReportPath() error = %v", err)
|
||||
}
|
||||
if reportPath != "reports/daily.md" {
|
||||
t.Fatalf("reportPath = %q, want reports/daily.md", reportPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDistributorTemplateRejectsUnknownAndMalformedVariables(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
template string
|
||||
}{
|
||||
{name: "Unknown", template: "{unknown}"},
|
||||
{name: "Unclosed", template: "{location_id"},
|
||||
{name: "Unopened", template: "location_id}"},
|
||||
{name: "Empty", template: "{}"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
_, err := RenderDistributorBundleID(tt.template, DistributorTemplateValues{})
|
||||
if err == nil {
|
||||
t.Fatal("RenderDistributorBundleID() error = nil, want error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDistributorReportPathValidation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
path string
|
||||
ok bool
|
||||
}{
|
||||
{name: "Simple", path: "daily.md", ok: true},
|
||||
{name: "Nested", path: "reports/daily.md", ok: true},
|
||||
{name: "Empty", path: "", ok: false},
|
||||
{name: "Absolute", path: "/reports/daily.md", ok: false},
|
||||
{name: "WindowsAbsolute", path: "C:/reports/daily.md", ok: false},
|
||||
{name: "Backslash", path: `reports\daily.md`, ok: false},
|
||||
{name: "CurrentSegment", path: "reports/./daily.md", ok: false},
|
||||
{name: "ParentSegment", path: "reports/../daily.md", ok: false},
|
||||
{name: "EmptySegment", path: "reports//daily.md", ok: false},
|
||||
{name: "Manifest", path: "reports/manifest.json", ok: false},
|
||||
{name: "DistributorMetadata", path: "reports/.distributor.json", ok: false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := ValidateDistributorReportPath(tt.path)
|
||||
if tt.ok && err != nil {
|
||||
t.Fatalf("ValidateDistributorReportPath() error = %v", err)
|
||||
}
|
||||
if !tt.ok && err == nil {
|
||||
t.Fatal("ValidateDistributorReportPath() error = nil, want error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDistributorReportPathRenderingRejectsInvalidValues(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
batchOutputName string
|
||||
}{
|
||||
{name: "Absolute", batchOutputName: "/daily.md"},
|
||||
{name: "Backslash", batchOutputName: `reports\daily.md`},
|
||||
{name: "CurrentSegment", batchOutputName: "./daily.md"},
|
||||
{name: "ParentSegment", batchOutputName: "../daily.md"},
|
||||
{name: "EmptySegment", batchOutputName: "reports//daily.md"},
|
||||
{name: "Manifest", batchOutputName: "manifest.json"},
|
||||
{name: "DistributorMetadata", batchOutputName: ".distributor.json"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
_, err := RenderDistributorReportPath("{batch_output_name}", DistributorTemplateValues{
|
||||
BatchOutputName: tt.batchOutputName,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("RenderDistributorReportPath() error = nil, want error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFileLoadsSecretsBeforeReturningNotifyConfig(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
secretsDir := filepath.Join(dir, "secrets")
|
||||
if err := os.Mkdir(secretsDir, 0o700); err != nil {
|
||||
t.Fatalf("create secrets directory: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(secretsDir, "DISTRIBUTOR_UPLOAD_TOKEN"), []byte("loaded-token"), 0o600); err != nil {
|
||||
t.Fatalf("write secret: %v", err)
|
||||
}
|
||||
path := filepath.Join(dir, "config.yml")
|
||||
configYAML := "secrets:\n" +
|
||||
" directory: " + secretsDir + "\n" +
|
||||
"notify:\n" +
|
||||
" distributor:\n" +
|
||||
" enabled: true\n"
|
||||
if err := os.WriteFile(path, []byte(configYAML), 0o600); err != nil {
|
||||
t.Fatalf("write config fixture: %v", err)
|
||||
}
|
||||
|
||||
t.Setenv("DISTRIBUTOR_UPLOAD_TOKEN", "")
|
||||
cfg, err := LoadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadFile() error = %v", err)
|
||||
}
|
||||
if cfg.Notify.Distributor.TokenEnv != "DISTRIBUTOR_UPLOAD_TOKEN" {
|
||||
t.Fatalf("TokenEnv = %q, want DISTRIBUTOR_UPLOAD_TOKEN", cfg.Notify.Distributor.TokenEnv)
|
||||
}
|
||||
if got := os.Getenv(cfg.Notify.Distributor.TokenEnv); got != "loaded-token" {
|
||||
t.Fatalf("environment value = %q, want loaded-token", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSecretsDisabledLeavesEnvironmentUnchanged(t *testing.T) {
|
||||
t.Setenv("WEATHERREPORTER_DISABLED_SECRET", "original")
|
||||
|
||||
if err := loadSecrets(SecretsConfig{}); err != nil {
|
||||
t.Fatalf("loadSecrets() error = %v", err)
|
||||
}
|
||||
if got := os.Getenv("WEATHERREPORTER_DISABLED_SECRET"); got != "original" {
|
||||
t.Fatalf("environment value = %q, want original", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFileLoadsSecretsDirectory(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
secretsDir := filepath.Join(dir, "secrets")
|
||||
if err := os.Mkdir(secretsDir, 0o700); err != nil {
|
||||
t.Fatalf("create secrets directory: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(secretsDir, "WEATHERREPORTER_SECRET"), []byte("from-file"), 0o600); err != nil {
|
||||
t.Fatalf("write secret: %v", err)
|
||||
}
|
||||
path := filepath.Join(dir, "config.yml")
|
||||
if err := os.WriteFile(path, []byte("secrets:\n directory: "+secretsDir+"\n"), 0o600); err != nil {
|
||||
t.Fatalf("write config fixture: %v", err)
|
||||
}
|
||||
|
||||
t.Setenv("WEATHERREPORTER_SECRET", "")
|
||||
if _, err := LoadFile(path); err != nil {
|
||||
t.Fatalf("LoadFile() error = %v", err)
|
||||
}
|
||||
if got := os.Getenv("WEATHERREPORTER_SECRET"); got != "from-file" {
|
||||
t.Fatalf("environment value = %q, want from-file", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSecretsOverwritesExistingEnvironment(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, "WEATHERREPORTER_SECRET"), []byte("from-file"), 0o600); err != nil {
|
||||
t.Fatalf("write secret: %v", err)
|
||||
}
|
||||
t.Setenv("WEATHERREPORTER_SECRET", "existing")
|
||||
|
||||
if err := loadSecrets(SecretsConfig{Directory: dir}); err != nil {
|
||||
t.Fatalf("loadSecrets() error = %v", err)
|
||||
}
|
||||
if got := os.Getenv("WEATHERREPORTER_SECRET"); got != "from-file" {
|
||||
t.Fatalf("environment value = %q, want from-file", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSecretsTrimsOneTrailingLineEnding(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
want string
|
||||
}{
|
||||
{name: "LF", input: "value\n", want: "value"},
|
||||
{name: "CRLF", input: "value\r\n", want: "value"},
|
||||
{name: "TwoLF", input: "value\n\n", want: "value\n"},
|
||||
{name: "LoneCR", input: "value\r", want: "value\r"},
|
||||
{name: "NoNewline", input: "value", want: "value"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, "WEATHERREPORTER_SECRET"), []byte(tt.input), 0o600); err != nil {
|
||||
t.Fatalf("write secret: %v", err)
|
||||
}
|
||||
t.Setenv("WEATHERREPORTER_SECRET", "")
|
||||
|
||||
if err := loadSecrets(SecretsConfig{Directory: dir}); err != nil {
|
||||
t.Fatalf("loadSecrets() error = %v", err)
|
||||
}
|
||||
if got := os.Getenv("WEATHERREPORTER_SECRET"); got != tt.want {
|
||||
t.Fatalf("environment value = %q, want %q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSecretsRejectsInvalidDirectoryEntries(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
setup func(t *testing.T, dir string)
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "InvalidFilename",
|
||||
setup: func(t *testing.T, dir string) {
|
||||
if err := os.WriteFile(filepath.Join(dir, "1INVALID"), []byte("secret-value"), 0o600); err != nil {
|
||||
t.Fatalf("write invalid secret: %v", err)
|
||||
}
|
||||
},
|
||||
wantErr: "invalid environment variable name",
|
||||
},
|
||||
{
|
||||
name: "Subdirectory",
|
||||
setup: func(t *testing.T, dir string) {
|
||||
if err := os.Mkdir(filepath.Join(dir, "SUBDIR"), 0o700); err != nil {
|
||||
t.Fatalf("create subdirectory: %v", err)
|
||||
}
|
||||
},
|
||||
wantErr: "not a directory",
|
||||
},
|
||||
{
|
||||
name: "Symlink",
|
||||
setup: func(t *testing.T, dir string) {
|
||||
target := filepath.Join(dir, "TARGET")
|
||||
if err := os.WriteFile(target, []byte("secret-value"), 0o600); err != nil {
|
||||
t.Fatalf("write target: %v", err)
|
||||
}
|
||||
if err := os.Symlink(target, filepath.Join(dir, "SYMLINK")); err != nil {
|
||||
t.Fatalf("create symlink: %v", err)
|
||||
}
|
||||
},
|
||||
wantErr: "not a symlink",
|
||||
},
|
||||
{
|
||||
name: "Unreadable",
|
||||
setup: func(t *testing.T, dir string) {
|
||||
path := filepath.Join(dir, "UNREADABLE")
|
||||
if err := os.WriteFile(path, []byte("secret-value"), 0o600); err != nil {
|
||||
t.Fatalf("write unreadable secret: %v", err)
|
||||
}
|
||||
if err := os.Chmod(path, 0o000); err != nil {
|
||||
t.Fatalf("chmod unreadable secret: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
_ = os.Chmod(path, 0o600)
|
||||
})
|
||||
},
|
||||
wantErr: "read secret file",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
tt.setup(t, dir)
|
||||
|
||||
err := loadSecrets(SecretsConfig{Directory: dir})
|
||||
if err == nil {
|
||||
t.Fatal("loadSecrets() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
|
||||
}
|
||||
if strings.Contains(err.Error(), "secret-value") {
|
||||
t.Fatalf("error = %q, want no secret value", err.Error())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSecretsRejectsMissingDirectory(t *testing.T) {
|
||||
err := loadSecrets(SecretsConfig{Directory: filepath.Join(t.TempDir(), "missing")})
|
||||
if err == nil {
|
||||
t.Fatal("loadSecrets() error = nil, want missing directory error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "read secrets directory") {
|
||||
t.Fatalf("error = %q, want read secrets directory context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18,6 +18,21 @@ func Defaults() Config {
|
||||
Name: "Brentwood",
|
||||
Region: "St. Louis Metro",
|
||||
},
|
||||
Secrets: SecretsConfig{
|
||||
Directory: "",
|
||||
},
|
||||
Notify: NotifyConfig{
|
||||
Distributor: DistributorNotifyConfig{
|
||||
Enabled: false,
|
||||
Endpoint: "https://distributor.example.com",
|
||||
TokenEnv: "DISTRIBUTOR_UPLOAD_TOKEN",
|
||||
Timeout: 30 * time.Second,
|
||||
FailurePolicy: NotifyFailureError,
|
||||
BundleIDTemplate: "weatherreporter.{location_id}.{report_id}.{run_id}",
|
||||
IdempotencyKeyTemplate: "{bundle_id}",
|
||||
ReportPathTemplate: "{batch_output_name}",
|
||||
},
|
||||
},
|
||||
MissingSource: MissingSourceConfig{
|
||||
Default: MissingSourceWarn,
|
||||
Sources: map[string]MissingSourcePolicy{},
|
||||
|
||||
@@ -35,6 +35,10 @@ func Load(opts LoadOptions) (Config, error) {
|
||||
cfg.WeatherAPI.Timezone = opts.Timezone
|
||||
}
|
||||
|
||||
if err := loadSecrets(cfg.Secrets); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
|
||||
if err := Validate(cfg); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
|
||||
145
internal/config/notify_templates.go
Normal file
145
internal/config/notify_templates.go
Normal file
@@ -0,0 +1,145 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type DistributorTemplateValues struct {
|
||||
LocationID string
|
||||
ReportID string
|
||||
RunID string
|
||||
ArtifactGroup string
|
||||
BatchOutputName string
|
||||
BundleID string
|
||||
}
|
||||
|
||||
var distributorTemplateVariables = map[string]struct{}{
|
||||
"location_id": {},
|
||||
"report_id": {},
|
||||
"run_id": {},
|
||||
"artifact_group": {},
|
||||
"batch_output_name": {},
|
||||
}
|
||||
|
||||
var distributorIdempotencyTemplateVariables = map[string]struct{}{
|
||||
"location_id": {},
|
||||
"report_id": {},
|
||||
"run_id": {},
|
||||
"artifact_group": {},
|
||||
"batch_output_name": {},
|
||||
"bundle_id": {},
|
||||
}
|
||||
|
||||
func RenderDistributorBundleID(template string, values DistributorTemplateValues) (string, error) {
|
||||
return renderDistributorTemplate("notify.distributor.bundle_id_template", template, values, distributorTemplateVariables)
|
||||
}
|
||||
|
||||
func RenderDistributorIdempotencyKey(template string, values DistributorTemplateValues) (string, error) {
|
||||
return renderDistributorTemplate("notify.distributor.idempotency_key_template", template, values, distributorIdempotencyTemplateVariables)
|
||||
}
|
||||
|
||||
func RenderDistributorReportPath(template string, values DistributorTemplateValues) (string, error) {
|
||||
rendered, err := renderDistributorTemplate("notify.distributor.report_path_template", template, values, distributorTemplateVariables)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := ValidateDistributorReportPath(rendered); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return rendered, nil
|
||||
}
|
||||
|
||||
func validateDistributorTemplate(name, template string, allowed map[string]struct{}) error {
|
||||
_, err := renderDistributorTemplate(name, template, DistributorTemplateValues{}, allowed)
|
||||
return err
|
||||
}
|
||||
|
||||
func renderDistributorTemplate(name, template string, values DistributorTemplateValues, allowed map[string]struct{}) (string, error) {
|
||||
var rendered strings.Builder
|
||||
for i := 0; i < len(template); {
|
||||
switch template[i] {
|
||||
case '{':
|
||||
end := strings.IndexByte(template[i+1:], '}')
|
||||
if end < 0 {
|
||||
return "", fmt.Errorf("%s contains an unclosed template variable", name)
|
||||
}
|
||||
variable := template[i+1 : i+1+end]
|
||||
if variable == "" {
|
||||
return "", fmt.Errorf("%s contains an empty template variable", name)
|
||||
}
|
||||
if _, ok := allowed[variable]; !ok {
|
||||
return "", fmt.Errorf("%s contains unknown template variable %q", name, variable)
|
||||
}
|
||||
rendered.WriteString(distributorTemplateValue(variable, values))
|
||||
i += end + 2
|
||||
case '}':
|
||||
return "", fmt.Errorf("%s contains an unopened template variable", name)
|
||||
default:
|
||||
rendered.WriteByte(template[i])
|
||||
i++
|
||||
}
|
||||
}
|
||||
return rendered.String(), nil
|
||||
}
|
||||
|
||||
func distributorTemplateValue(variable string, values DistributorTemplateValues) string {
|
||||
switch variable {
|
||||
case "location_id":
|
||||
return values.LocationID
|
||||
case "report_id":
|
||||
return values.ReportID
|
||||
case "run_id":
|
||||
return values.RunID
|
||||
case "artifact_group":
|
||||
return values.ArtifactGroup
|
||||
case "batch_output_name":
|
||||
return values.BatchOutputName
|
||||
case "bundle_id":
|
||||
return values.BundleID
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
func ValidateDistributorReportPath(path string) error {
|
||||
if path == "" {
|
||||
return fmt.Errorf("notify.distributor.report_path_template renders an empty path")
|
||||
}
|
||||
if isDistributorAbsolutePath(path) {
|
||||
return fmt.Errorf("notify.distributor.report_path_template must render a relative path")
|
||||
}
|
||||
if strings.Contains(path, "\\") {
|
||||
return fmt.Errorf("notify.distributor.report_path_template must not render backslashes")
|
||||
}
|
||||
|
||||
segments := strings.Split(path, "/")
|
||||
for _, segment := range segments {
|
||||
if segment == "" {
|
||||
return fmt.Errorf("notify.distributor.report_path_template must not render empty path segments")
|
||||
}
|
||||
if segment == "." || segment == ".." {
|
||||
return fmt.Errorf("notify.distributor.report_path_template must not render . or .. path segments")
|
||||
}
|
||||
if segment == "manifest.json" || segment == ".distributor.json" {
|
||||
return fmt.Errorf("notify.distributor.report_path_template must not render reserved path segment %q", segment)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func isDistributorAbsolutePath(path string) bool {
|
||||
if filepath.IsAbs(path) || strings.HasPrefix(path, "/") {
|
||||
return true
|
||||
}
|
||||
if len(path) >= 3 && isASCIIAlpha(path[0]) && path[1] == ':' && (path[2] == '/' || path[2] == '\\') {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func isASCIIAlpha(ch byte) bool {
|
||||
return (ch >= 'A' && ch <= 'Z') || (ch >= 'a' && ch <= 'z')
|
||||
}
|
||||
63
internal/config/secrets.go
Normal file
63
internal/config/secrets.go
Normal file
@@ -0,0 +1,63 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
var secretNamePattern = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
|
||||
|
||||
func loadSecrets(cfg SecretsConfig) error {
|
||||
if cfg.Directory == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
entries, err := os.ReadDir(cfg.Directory)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read secrets directory %q: %w", cfg.Directory, err)
|
||||
}
|
||||
|
||||
for _, entry := range entries {
|
||||
name := entry.Name()
|
||||
if name == "" {
|
||||
return fmt.Errorf("secrets directory %q contains an empty filename", cfg.Directory)
|
||||
}
|
||||
if !secretNamePattern.MatchString(name) {
|
||||
return fmt.Errorf("secret file %q has invalid environment variable name", name)
|
||||
}
|
||||
if entry.Type()&os.ModeSymlink != 0 {
|
||||
return fmt.Errorf("secret file %q must be a regular file, not a symlink", name)
|
||||
}
|
||||
if entry.IsDir() {
|
||||
return fmt.Errorf("secret file %q must be a regular file, not a directory", name)
|
||||
}
|
||||
|
||||
info, err := entry.Info()
|
||||
if err != nil {
|
||||
return fmt.Errorf("inspect secret file %q: %w", name, err)
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return fmt.Errorf("secret file %q must be a regular file", name)
|
||||
}
|
||||
|
||||
path := filepath.Join(cfg.Directory, name)
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read secret file %q: %w", name, err)
|
||||
}
|
||||
value := string(data)
|
||||
if strings.HasSuffix(value, "\r\n") {
|
||||
value = strings.TrimSuffix(value, "\r\n")
|
||||
} else {
|
||||
value = strings.TrimSuffix(value, "\n")
|
||||
}
|
||||
if err := os.Setenv(name, value); err != nil {
|
||||
return fmt.Errorf("set environment variable from secret file %q: %w", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -49,6 +49,10 @@ func Validate(cfg Config) error {
|
||||
}
|
||||
}
|
||||
|
||||
if err := validateDistributorNotify(cfg.Notify.Distributor); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if cfg.Scriptorium.Binary == "" {
|
||||
return fmt.Errorf("scriptorium.binary is required")
|
||||
}
|
||||
@@ -75,6 +79,56 @@ func Validate(cfg Config) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateDistributorNotify(cfg DistributorNotifyConfig) error {
|
||||
if !cfg.Enabled {
|
||||
return nil
|
||||
}
|
||||
|
||||
parsed, err := url.Parse(cfg.Endpoint)
|
||||
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
|
||||
return fmt.Errorf("notify.distributor.endpoint must be an absolute URL when enabled")
|
||||
}
|
||||
if cfg.TokenEnv == "" {
|
||||
return fmt.Errorf("notify.distributor.token_env is required when enabled")
|
||||
}
|
||||
if !secretNamePattern.MatchString(cfg.TokenEnv) {
|
||||
return fmt.Errorf("notify.distributor.token_env must be a valid environment variable name")
|
||||
}
|
||||
if cfg.Timeout <= 0 {
|
||||
return fmt.Errorf("notify.distributor.timeout must be greater than zero when enabled")
|
||||
}
|
||||
if cfg.FailurePolicy != NotifyFailureError {
|
||||
return fmt.Errorf("notify.distributor.failure_policy must be error when enabled")
|
||||
}
|
||||
if cfg.BundleIDTemplate == "" {
|
||||
return fmt.Errorf("notify.distributor.bundle_id_template is required when enabled")
|
||||
}
|
||||
if err := validateDistributorTemplate("notify.distributor.bundle_id_template", cfg.BundleIDTemplate, distributorTemplateVariables); err != nil {
|
||||
return err
|
||||
}
|
||||
if cfg.IdempotencyKeyTemplate == "" {
|
||||
return fmt.Errorf("notify.distributor.idempotency_key_template is required when enabled")
|
||||
}
|
||||
if err := validateDistributorTemplate("notify.distributor.idempotency_key_template", cfg.IdempotencyKeyTemplate, distributorIdempotencyTemplateVariables); err != nil {
|
||||
return err
|
||||
}
|
||||
if cfg.ReportPathTemplate == "" {
|
||||
return fmt.Errorf("notify.distributor.report_path_template is required when enabled")
|
||||
}
|
||||
values := DistributorTemplateValues{
|
||||
LocationID: "location",
|
||||
ReportID: "report",
|
||||
RunID: "run",
|
||||
ArtifactGroup: "artifact",
|
||||
BatchOutputName: "report.md",
|
||||
}
|
||||
if _, err := RenderDistributorReportPath(cfg.ReportPathTemplate, values); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func validatePolicy(name string, policy MissingSourcePolicy) error {
|
||||
switch policy {
|
||||
case MissingSourceError, MissingSourceWarn, MissingSourceNone:
|
||||
|
||||
@@ -155,7 +155,14 @@ type DiscussionSection struct {
|
||||
}
|
||||
|
||||
type WeatherStory struct {
|
||||
IssuedAt *time.Time `json:"issuedAt,omitempty"`
|
||||
UpdatedAt *time.Time `json:"updatedAt,omitempty"`
|
||||
Raw json.RawMessage `json:"raw,omitempty"`
|
||||
OfficeID string `json:"officeId,omitempty"`
|
||||
StartTime time.Time `json:"startTime"`
|
||||
EndTime time.Time `json:"endTime"`
|
||||
UpdatedAt *time.Time `json:"updatedAt,omitempty"`
|
||||
Title string `json:"title,omitempty"`
|
||||
Description string `json:"description,omitempty"`
|
||||
AltText string `json:"altText,omitempty"`
|
||||
Priority bool `json:"priority"`
|
||||
Order int `json:"order"`
|
||||
DownloadURL string `json:"downloadUrl,omitempty"`
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user