Added an implementation plan for SPC convective outlook support, and completed the documentation cleanup
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
This commit is contained in:
4
API.md
4
API.md
@@ -1,4 +0,0 @@
|
||||
# Event Wire Contract
|
||||
|
||||
The canonical event wire contract has moved to
|
||||
[docs/integrations/events.md](docs/integrations/events.md).
|
||||
@@ -1,491 +0,0 @@
|
||||
# Documentation Roadmap
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap defines the work required to bring `weatherfeeder` documentation into compliance with [`docs/policy/documentation.md`](../policy/documentation.md) and the current implementation described by [`docs/policy/architecture.md`](../policy/architecture.md).
|
||||
|
||||
This is a planning document only. Implementation stages must update current-behavior documentation so it reflects implemented code, and keep proposed or unimplemented work under `docs/roadmap/`.
|
||||
|
||||
## Repository Documentation Inventory
|
||||
|
||||
- `README.md`: keep and rewrite. It is the correct project orientation file, but it is stale: it says stdout is the only implemented sink, uses “current MVP” framing, and lacks a shortest useful run command and links to targeted docs.
|
||||
- `API.md`: move and rewrite. It currently acts as the event wire contract, but external integration contracts belong under `docs/integrations/`. It is also stale: it omits the emitted `kind` and `emitted_at` envelope fields, documents `effectiveAt` instead of the actual `effective_at` JSON field, lists only some raw schemas, and has duplicate alert `response` rows.
|
||||
- `docs/policy/documentation.md`: keep and lightly update only if needed. It is the controlling documentation policy and should not be duplicated elsewhere.
|
||||
- `docs/policy/architecture.md`: keep and lightly update after the documentation tree is migrated. Its architecture content is current, but its link to `API.md` should be updated when the wire contract moves to `docs/integrations/events.md`.
|
||||
- `docs/roadmap/spc.md`: keep. It is correctly located under roadmap because it describes unimplemented SPC outlook support.
|
||||
- `docs/roadmap/documentation.md`: create. This file is the current deliverable.
|
||||
- `cmd/weatherfeeder/config.yml`: keep and treat as an executable sample, but do not use it as the only operator-facing config reference. It includes live endpoints and a commented OpenWeather URL with an apparent real API key; future cleanup should remove or replace that secret-like value.
|
||||
- `internal/normalizers/doc.go`: keep and lightly update. It contains implemented developer guidance but incorrectly references `internal/standards/schema.go`; the actual package is `standards` at repository root.
|
||||
- `internal/normalizers/common/doc.go`: keep and lightly update if needed. It is concise and current.
|
||||
- `internal/providers/nws/doc.go`: keep and lightly update if needed. It documents implemented provider-helper boundaries.
|
||||
- `internal/providers/openweather/doc.go`: keep and lightly update. It references future forecasts/alerts in a comment; outside roadmap docs this should be rewritten to current behavior only.
|
||||
- `standards/doc.go`: keep and lightly update. It incorrectly says `internal/standards/doc.go`; actual path is `standards/doc.go`.
|
||||
- `model/doc.go`: keep and lightly update. It says JSON tags are wire contract for “stdout today; others later,” which is stale because NATS and Postgres are implemented.
|
||||
- `internal/sinks/postgres/doc.go`: keep and lightly update. It is the authoritative implemented Postgres table contract, but should be linked from `docs/integrations/postgres.md` instead of being the only discoverable operator/developer reference.
|
||||
- `examples/`: create new. No examples directory exists. Current example material is embedded in `cmd/weatherfeeder/config.yml` only.
|
||||
- `docs/config.md`: create new. Required because `weatherfeeder` is config-driven.
|
||||
- `docs/cli.md`: create new. Required because `weatherfeeder` is a command-line executable, even though it currently has no flags.
|
||||
- `docs/operations.md`: create new. Required because `weatherfeeder` is an operator-facing daemon with polling, concurrency, external sinks, retention/pruning, and shutdown behavior.
|
||||
- `docs/troubleshooting.md`: create new. Recommended and useful because recurring operator failures are visible in constructors and config validation.
|
||||
- `docs/policy/development.md`: create new. Required for a modular project maintained by humans and coding agents.
|
||||
- `docs/internal/`: create new. Required because the project is modular and staged.
|
||||
- `docs/integrations/`: create new. Required because the project depends on external weather APIs, NATS, Postgres, and an emitted event contract.
|
||||
|
||||
## Policy Compliance Assessment
|
||||
|
||||
Required missing documents:
|
||||
|
||||
- `docs/cli.md`
|
||||
- `docs/config.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/`
|
||||
- `docs/policy/development.md`
|
||||
- `docs/integrations/` entries for important external contracts
|
||||
|
||||
Recommended missing documents:
|
||||
|
||||
- `docs/troubleshooting.md`
|
||||
- `examples/` with maintained config examples
|
||||
|
||||
Existing stale or misplaced content:
|
||||
|
||||
- `README.md` is not concise enough as a current quickstart and contains stale sink status.
|
||||
- `API.md` is in the wrong canonical home for an external integration contract and does not match `feedkit/event.Event` JSON tags.
|
||||
- Some package comments include stale path references or future-looking language outside roadmap files.
|
||||
- The sample config contains live endpoint examples and an OpenWeather URL with an apparent credential-like `appid` value in a commented block; examples should be secret-free.
|
||||
|
||||
Unimplemented, historical, or roadmap-only content outside roadmap:
|
||||
|
||||
- README “current MVP” phrasing is development-history framing and should be removed.
|
||||
- Comments such as “others later,” “forecasts/alerts later,” and “future expensive steps” should be rewritten or removed unless they describe an implemented boundary.
|
||||
- Do not move SPC outlook content out of `docs/roadmap/spc.md` until it is implemented.
|
||||
|
||||
Examples needing work:
|
||||
|
||||
- There is no `examples/` directory.
|
||||
- `cmd/weatherfeeder/config.yml` is load-tested by `cmd/weatherfeeder/main_test.go`, but it is not an ideal copyable public example because it includes local/operator-specific values and a commented API key-like OpenWeather URL.
|
||||
- Future examples should be checked with `feedkit/config.Load` or a weatherfeeder-specific test.
|
||||
|
||||
Links needing verification:
|
||||
|
||||
- Links from `README.md` to `API.md` should be changed to the new canonical event contract path if `API.md` is moved.
|
||||
- `docs/policy/architecture.md` links to `../../API.md`; update this after moving the contract.
|
||||
- Internal package comments referring to `internal/standards` need correction to `standards`.
|
||||
|
||||
## Target Documentation Set
|
||||
|
||||
### `README.md`
|
||||
|
||||
- Audience: users, administrators, operators.
|
||||
- Purpose: orient readers and get them to the shortest useful run path.
|
||||
- Canonical scope: project purpose, elevator pitch, shortest useful command, and links to deeper docs.
|
||||
- Recommended outline: description; what it does; shortest useful command; emitted products and providers summary; links to config, CLI, operations, event contract, development policy.
|
||||
- Source of truth: `cmd/weatherfeeder/main.go`, `cmd/weatherfeeder/config.yml`, `internal/sources/builtins.go`, `internal/normalizers/builtins.go`, `standards/schema.go`, `README.md` stale claims.
|
||||
- Acceptance criteria: concise, no stale stdout-only claim, no full config reference, no development-history framing, links target canonical docs.
|
||||
|
||||
### `docs/cli.md`
|
||||
|
||||
- Audience: users, administrators, operators.
|
||||
- Purpose: document the executable interface.
|
||||
- Canonical scope: how to run `weatherfeeder` and what CLI flags exist.
|
||||
- Recommended outline: shortest useful command; command overview; complete flag reference stating there are currently no flags; working-directory requirement for `config.yml`; common local/container workflows; shutdown signal behavior.
|
||||
- Source of truth: `cmd/weatherfeeder/main.go`, `Dockerfile`, `cmd/weatherfeeder/main_test.go`.
|
||||
- Acceptance criteria: accurately states that `config.yml` is loaded from current working directory and no config path flag exists; does not invent flags.
|
||||
|
||||
### `docs/config.md`
|
||||
|
||||
- Audience: administrators, operators, advanced users.
|
||||
- Purpose: canonical configuration reference.
|
||||
- Canonical scope: YAML shape, field rules, source/sink/route definitions, driver params, and defaults.
|
||||
- Recommended outline: config file location; minimal config; production-oriented config; top-level reference; source drivers; source params; sink drivers; sink params; routes; duration formats; secrets handling; links to examples.
|
||||
- Source of truth: `cmd/weatherfeeder/main.go`, `cmd/weatherfeeder/config.yml`, `../feedkit/config/config.go`, `../feedkit/config/load.go`, `../feedkit/sources/http.go`, `../feedkit/sinks/*.go`, `internal/sources/builtins.go`, source constructors.
|
||||
- Acceptance criteria: documents strict YAML known-field behavior, required `sources` and `sinks`, optional `routes`, `mode`, `every`, `kinds`, HTTP params including `url`, `user_agent`, `conditional`, `http_timeout`, and `http_response_body_limit_bytes`; documents OpenWeather `units=metric` requirement; avoids embedding secrets.
|
||||
|
||||
### `docs/operations.md`
|
||||
|
||||
- Audience: administrators and operators.
|
||||
- Purpose: explain how to run, observe, shut down, and recover the daemon.
|
||||
- Canonical scope: runtime behavior, logs, scheduling, sink behavior, Postgres initialization/pruning, and operational caveats.
|
||||
- Recommended outline: normal workflow; runtime lifecycle; logs; scheduler and polling behavior; conditional HTTP fetches; routing and sink fanout; Postgres table creation and retention; shutdown; recovery; caveats.
|
||||
- Source of truth: `cmd/weatherfeeder/main.go`, `../feedkit/scheduler`, `../feedkit/dispatch`, `../feedkit/sources/http.go`, `../feedkit/sinks/postgres.go`, `internal/sinks/postgres/schema.go`, `Dockerfile`.
|
||||
- Acceptance criteria: describes implemented behavior only; no unsupported admin commands; clearly states durable state is external sink state.
|
||||
|
||||
### `docs/troubleshooting.md`
|
||||
|
||||
- Audience: administrators and operators.
|
||||
- Purpose: provide safe diagnosis for common implemented failure modes.
|
||||
- Canonical scope: symptoms, likely causes, checks, fixes, and links.
|
||||
- Recommended outline: missing `config.yml`; YAML parse or unknown-field errors; unknown source/sink driver; source kind mismatch; missing `params.url` or `params.user_agent`; OpenWeather missing `units=metric`; NATS connection failure; Postgres connection/table/credential failure; no events due to 304 unchanged responses; route sends no events.
|
||||
- Source of truth: constructors and validation in feedkit/weatherfeeder, `cmd/weatherfeeder/main.go`, source tests, sink tests.
|
||||
- Acceptance criteria: every entry has symptom, likely cause, diagnostic step, safe fix, and link to config/operations where useful.
|
||||
|
||||
### `docs/policy/development.md`
|
||||
|
||||
- Audience: developers and LLM coding agents.
|
||||
- Purpose: contributor workflow and safe-change guidance.
|
||||
- Canonical scope: repository layout, build/test commands, coding conventions, dependency policy, adding drivers/normalizers/schemas/config fields/docs.
|
||||
- Recommended outline: layout; Go/test commands; style; dependency policy; adding a source; adding a normalizer; adding a canonical model/schema; adding Postgres mapping; updating sample config/examples/docs; review checklist.
|
||||
- Source of truth: `docs/policy/architecture.md`, `go.mod`, `cmd/weatherfeeder/main_test.go`, package docs, builtins registries, tests.
|
||||
- Acceptance criteria: no user/operator reference duplication; links to config/CLI docs; gives enough steps for coding agents to preserve architecture.
|
||||
|
||||
### `docs/internal/runtime.md`
|
||||
|
||||
- Audience: developers and LLM coding agents.
|
||||
- Purpose: describe implemented runtime composition.
|
||||
- Canonical scope: config load, registries, scheduler, processor chain, dispatcher, sink fanout, shutdown.
|
||||
- Recommended outline: purpose; inputs/outputs; boundaries; config fields used; adapters used; state; failure behavior; tests; invariants.
|
||||
- Source of truth: `cmd/weatherfeeder/main.go`, `cmd/weatherfeeder/main_test.go`, `../feedkit/scheduler`, `../feedkit/dispatch`, `../feedkit/processors`.
|
||||
- Acceptance criteria: accurately describes normalize then dedupe, `dedupeMaxEntries = 2048`, event channel buffer 256, and context shutdown behavior.
|
||||
|
||||
### `docs/internal/sources.md`
|
||||
|
||||
- Audience: developers and LLM coding agents.
|
||||
- Purpose: document implemented source drivers and source contracts.
|
||||
- Canonical scope: source boundaries, registered drivers, kinds, raw schemas, params, effective-time policies at a summary level.
|
||||
- Recommended outline: source contract; common HTTP behavior; driver table; provider notes; failure behavior; tests; invariants.
|
||||
- Source of truth: `internal/sources/builtins.go`, `internal/sources/*`, `../feedkit/sources/http.go`, source tests, `standards/schema.go`.
|
||||
- Acceptance criteria: includes all current drivers and no legacy `nws_forecast`; documents raw payload preservation and conditional GET behavior without duplicating full config reference.
|
||||
|
||||
### `docs/internal/normalizers.md`
|
||||
|
||||
- Audience: developers and LLM coding agents.
|
||||
- Purpose: document raw-to-canonical mapping boundaries and registration.
|
||||
- Canonical scope: normalizer contract, schema matching, provider packages, output schemas, tests.
|
||||
- Recommended outline: contract; registration/order; mapping table from raw schemas to canonical schemas; common helpers; failure behavior; tests; invariants.
|
||||
- Source of truth: `internal/normalizers/doc.go`, `internal/normalizers/builtins.go`, provider normalizer packages, `standards/schema.go`, normalizer tests.
|
||||
- Acceptance criteria: fixes stale `internal/standards` references; avoids duplicating every payload field from event contract.
|
||||
|
||||
### `docs/internal/postgres-sink.md`
|
||||
|
||||
- Audience: developers and LLM coding agents.
|
||||
- Purpose: explain weatherfeeder-owned Postgres mapping internals.
|
||||
- Canonical scope: schema registration, mapper behavior, required-field validation, table contract link.
|
||||
- Recommended outline: purpose; input schemas; writes; parent/child tables; nullable rules; pruning columns; tests; invariants.
|
||||
- Source of truth: `internal/sinks/postgres/doc.go`, `schema.go`, `map.go`, sink tests, feedkit Postgres sink.
|
||||
- Acceptance criteria: links to `docs/integrations/postgres.md` for operator-facing table contract; does not duplicate full table definitions unless that integration doc remains intentionally concise.
|
||||
|
||||
### `docs/integrations/events.md`
|
||||
|
||||
- Audience: downstream consumers and developers.
|
||||
- Purpose: canonical emitted event wire contract.
|
||||
- Canonical scope: feedkit event envelope JSON and canonical weather payload schemas.
|
||||
- Recommended outline: envelope; timestamp rules; kinds and schemas; raw schema note; shared conventions; canonical payloads; compatibility rules; compact examples.
|
||||
- Source of truth: `../feedkit/event/event.go`, `standards/schema.go`, `model/*.go`, `internal/normalizers/common/round.go`, normalizer tests.
|
||||
- Acceptance criteria: documents actual JSON field names (`id`, `kind`, `source`, `emitted_at`, `effective_at`, `schema`, `payload`); includes all current raw and canonical schemas; states forecast period `conditionCode` is optional; examples validate against current structs.
|
||||
|
||||
### `docs/integrations/postgres.md`
|
||||
|
||||
- Audience: downstream SQL consumers, administrators, developers.
|
||||
- Purpose: canonical Postgres table contract for weatherfeeder writes.
|
||||
- Canonical scope: tables, columns, keys, indexes, pruning, reconstruction notes, migration caveats.
|
||||
- Recommended outline: scope; sink config link; initialization behavior; table overview; full table contract; pruning; migrations; reconstruction of canonical payloads.
|
||||
- Source of truth: `internal/sinks/postgres/doc.go`, `schema.go`, `map.go`, `../feedkit/sinks/postgres.go`, schema/map tests.
|
||||
- Acceptance criteria: matches generated schema exactly; notes `CREATE TABLE IF NOT EXISTS` does not alter existing schemas; documents `forecast_periods.condition_code` nullable.
|
||||
|
||||
### `docs/integrations/nws.md`
|
||||
|
||||
- Audience: developers and operators maintaining NWS integrations.
|
||||
- Purpose: concise notes for implemented NWS endpoint usage.
|
||||
- Canonical scope: only endpoints and fields weatherfeeder currently uses.
|
||||
- Recommended outline: supported drivers; endpoint shapes; required user agent; accept headers; effective-time policies; parser caveats; tests.
|
||||
- Source of truth: `internal/sources/nws`, `internal/normalizers/nws`, `internal/providers/nws`, NWS tests and fixtures.
|
||||
- Acceptance criteria: no undocumented SPC outlook content; weather stories and forecast discussions included as implemented.
|
||||
|
||||
### `docs/integrations/openmeteo.md`
|
||||
|
||||
- Audience: developers and operators maintaining Open-Meteo integrations.
|
||||
- Purpose: concise notes for implemented Open-Meteo observation and hourly forecast usage.
|
||||
- Canonical scope: current endpoint usage, time parsing, fields mapped, effective time.
|
||||
- Source of truth: `internal/sources/openmeteo`, `internal/normalizers/openmeteo`, `internal/providers/openmeteo`, tests.
|
||||
- Acceptance criteria: no daily forecast or unsupported products documented as current.
|
||||
|
||||
### `docs/integrations/openweather.md`
|
||||
|
||||
- Audience: developers and operators maintaining OpenWeather integration.
|
||||
- Purpose: concise notes for implemented OpenWeather observation usage.
|
||||
- Canonical scope: current observation driver, `units=metric` requirement, timestamp behavior, mapped fields.
|
||||
- Source of truth: `internal/sources/openweather`, `internal/normalizers/openweather`, `internal/providers/openweather`, tests.
|
||||
- Acceptance criteria: no forecast/alert OpenWeather support documented as current.
|
||||
|
||||
### `examples/config.minimal.yml`
|
||||
|
||||
- Audience: users and operators.
|
||||
- Purpose: copyable minimal config.
|
||||
- Canonical scope: one source, one stdout sink, one route.
|
||||
- Source of truth: `cmd/weatherfeeder/config.yml`, config tests, source constructors.
|
||||
- Acceptance criteria: loads with `config.Load`; no secrets; uses placeholder-safe or public endpoints as appropriate.
|
||||
|
||||
### `examples/config.nats.yml`
|
||||
|
||||
- Audience: operators.
|
||||
- Purpose: copyable NATS publishing config.
|
||||
- Canonical scope: selected sources, NATS sink, route examples.
|
||||
- Source of truth: sample config, feedkit NATS sink.
|
||||
- Acceptance criteria: loads with `config.Load`; no secrets; clearly uses replaceable NATS URL/subject.
|
||||
|
||||
### `examples/config.postgres.yml`
|
||||
|
||||
- Audience: operators.
|
||||
- Purpose: copyable Postgres persistence config.
|
||||
- Canonical scope: selected sources, Postgres sink params, pruning example, routes.
|
||||
- Source of truth: sample config, feedkit Postgres sink, weatherfeeder Postgres schema.
|
||||
- Acceptance criteria: loads with `config.Load`; uses placeholders for username/password/URI; links from config and operations docs.
|
||||
|
||||
## File-by-File Rewrite Guidance
|
||||
|
||||
### `README.md`
|
||||
|
||||
Cover what the daemon does, the shortest useful command, and where to go next. Use implemented provider and product summaries from source/normalizer registries. Avoid full config tables, full wire schemas, history, “MVP” language, and stale sink limitations. Link to `docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/integrations/events.md`, and `docs/policy/development.md`.
|
||||
|
||||
Do not carry forward the claim that stdout is the only implemented sink.
|
||||
|
||||
### `API.md`
|
||||
|
||||
Move its canonical content to `docs/integrations/events.md` and either delete `API.md` or replace it with a short pointer only if backward-compatible repository links are required. The rewritten contract must inspect `../feedkit/event/event.go` before documenting envelope fields. Do not preserve camelCase envelope examples unless the code changes first.
|
||||
|
||||
### `docs/config.md`
|
||||
|
||||
Use feedkit config structs for generic fields and weatherfeeder source/sink constructors for driver-specific params. Link to examples instead of embedding all large configs. Avoid documenting feedkit stream-only params as actively used by weatherfeeder unless clearly marked as generic feedkit config accepted by the current config model; all current weatherfeeder sources are poll sources.
|
||||
|
||||
### `docs/cli.md`
|
||||
|
||||
State the current no-flags behavior plainly. Include that `weatherfeeder` expects `config.yml` in the current working directory. Do not document environment variables, subcommands, or alternate config paths unless implemented.
|
||||
|
||||
### `docs/operations.md`
|
||||
|
||||
Focus on running the daemon and operating sinks. Include signal shutdown, log behavior, conditional HTTP fetches, Postgres table creation, retention pruning, and no internal durable scheduler state. Avoid unsupported recovery commands.
|
||||
|
||||
### `docs/troubleshooting.md`
|
||||
|
||||
Base entries on actual validation and constructor errors. Link to `docs/config.md` for fixes. Avoid speculative upstream outages beyond generic HTTP/source errors unless tests or code reveal specific behavior.
|
||||
|
||||
### `docs/policy/development.md`
|
||||
|
||||
Use `docs/policy/architecture.md` as the authority. Give concrete safe-change steps for adding source drivers, normalizers, schemas, Postgres tables, config examples, and docs. Avoid repeating the architecture policy wholesale.
|
||||
|
||||
### `docs/internal/*.md`
|
||||
|
||||
Create concise component docs. They should guide changes, not become manuals. Each should include tests to inspect before changing. Link to package docs when those are more precise.
|
||||
|
||||
### `docs/integrations/*.md`
|
||||
|
||||
Document only external contracts actually used by the implementation. Keep provider docs narrow and maintenance-oriented. Do not describe unimplemented SPC support outside `docs/roadmap/spc.md`.
|
||||
|
||||
### Package comments
|
||||
|
||||
Fix stale path and future-language comments during the documentation migration stage that touches developer docs. Specific likely fixes:
|
||||
|
||||
- `model/doc.go`: remove “stdout today; others later.”
|
||||
- `standards/doc.go`: correct `internal/standards/doc.go` path.
|
||||
- `internal/normalizers/doc.go`: correct `internal/standards/schema.go` path.
|
||||
- `internal/providers/openweather/doc.go`: remove “forecasts/alerts later.”
|
||||
|
||||
## Examples Plan
|
||||
|
||||
No `examples/` directory currently exists. Create one because the project has non-trivial configuration and operator workflows.
|
||||
|
||||
Recommended examples:
|
||||
|
||||
- `examples/config.minimal.yml`: one public HTTP source, stdout sink, route. Validity check: `config.Load` in a new or existing config/example test. Link from README, config, CLI.
|
||||
- `examples/config.nats.yml`: NATS sink publishing example. Validity check: `config.Load`; do not require a live NATS server. Link from config and operations.
|
||||
- `examples/config.postgres.yml`: Postgres sink with placeholder credentials and optional `prune`. Validity check: `config.Load`; do not require a live Postgres server. Link from config, operations, Postgres integration docs.
|
||||
- `examples/config.full.yml`: optional, only if the team wants one annotated config covering all implemented drivers. Validity check: `config.Load`; must use placeholders for secrets and avoid real OpenWeather API keys. Link from config docs only.
|
||||
|
||||
Do not add SPC examples until SPC support is implemented.
|
||||
|
||||
## Internal Documentation Plan
|
||||
|
||||
### Runtime Pipeline
|
||||
|
||||
- Path: `docs/internal/runtime.md`
|
||||
- Purpose: explain daemon composition.
|
||||
- Inputs and outputs: YAML config in; feed events through scheduler/pipeline/dispatcher to sinks out.
|
||||
- Boundaries: composition belongs in `cmd/weatherfeeder`; domain mapping belongs in normalizers/model.
|
||||
- Config fields used: sources, sinks, routes, source `every`, mode/kinds, sink params.
|
||||
- Adapters used: feedkit scheduler, processors, dispatch, sinks.
|
||||
- Failure behavior: fatal startup errors; runtime errors logged and cancel context; interrupt/SIGTERM shutdown.
|
||||
- Tests to inspect: `cmd/weatherfeeder/main_test.go`, feedkit scheduler/dispatch tests.
|
||||
- Invariants: normalize before dedupe; bounded event channel; context-aware shutdown.
|
||||
|
||||
### Sources
|
||||
|
||||
- Path: `docs/internal/sources.md`
|
||||
- Purpose: document source driver architecture.
|
||||
- Inputs and outputs: `config.SourceConfig` and upstream HTTP responses in; raw feed events out.
|
||||
- Boundaries: sources fetch and envelope raw payloads; normalizers decode full payloads.
|
||||
- Config fields used: name, driver, mode, every, kinds, params URL/user agent/HTTP options.
|
||||
- Adapters used: feedkit source registry and HTTP helper.
|
||||
- Failure behavior: constructor validation, HTTP errors, unchanged 304 returns no events, some malformed metadata still emits raw events.
|
||||
- Tests to inspect: `internal/sources/builtins_test.go`, provider source tests.
|
||||
- Invariants: drivers registered in one place; raw schemas from `standards`; no canonical mapping in sources.
|
||||
|
||||
### Normalizers
|
||||
|
||||
- Path: `docs/internal/normalizers.md`
|
||||
- Purpose: document raw-to-canonical mapping architecture.
|
||||
- Inputs and outputs: raw events matched by schema in; canonical model payload events out.
|
||||
- Boundaries: no HTTP, no sink persistence, no CLI/config behavior.
|
||||
- Config fields used: none directly.
|
||||
- Adapters used: feedkit normalize processor.
|
||||
- Failure behavior: decode/parse/missing required fields return contextual errors; no-match passthrough configured in main.
|
||||
- Tests to inspect: `internal/normalizers/builtins_test.go`, provider normalizer tests, `internal/normalizers/common` tests.
|
||||
- Invariants: schema matching only, stable registration order, canonical schema constants.
|
||||
|
||||
### Postgres Sink Mapping
|
||||
|
||||
- Path: `docs/internal/postgres-sink.md`
|
||||
- Purpose: document weatherfeeder-specific Postgres mapping internals.
|
||||
- Inputs and outputs: canonical events in; feedkit `PostgresWrite` rows out.
|
||||
- Boundaries: mapper validates and maps; feedkit owns DB connection, DDL, transactions, pruning.
|
||||
- Config fields used: sink `uri`, `username`, `password`, optional `prune` via feedkit.
|
||||
- Adapters used: feedkit Postgres sink.
|
||||
- Failure behavior: unsupported schemas map to no writes or errors per mapper behavior; required missing fields fail before write.
|
||||
- Tests to inspect: `internal/sinks/postgres/map_test.go`, `schema_test.go`, feedkit Postgres tests.
|
||||
- Invariants: consume canonical schemas only; preserve envelope columns; keep parent/child order indexes.
|
||||
|
||||
## Integration Documentation Plan
|
||||
|
||||
### Event Wire Contract
|
||||
|
||||
- Path: `docs/integrations/events.md`
|
||||
- External system or contract: JSON events emitted to stdout/NATS and represented in Postgres parent envelope columns.
|
||||
- Current usage: sinks marshal feedkit events or map canonical events to tables.
|
||||
- Version notes: schema identifiers use `raw.*.v1` and `weather.*.v1`; JSON tags are compatibility contract.
|
||||
- Document: actual envelope fields, schema list, payload fields, units, optionality, examples.
|
||||
- Do not document: weatherapi HTTP endpoints, unimplemented outlooks, or future schemas.
|
||||
|
||||
### Postgres
|
||||
|
||||
- Path: `docs/integrations/postgres.md`
|
||||
- External system or contract: PostgreSQL schema created/written by feedkit Postgres sink using weatherfeeder schema definition.
|
||||
- Current usage: optional configured sink; create-if-missing tables and indexes; transactional writes; optional pruning.
|
||||
- Version notes: no migration framework is implemented; existing DBs may need manual schema changes when table definitions change.
|
||||
- Document: params, table contract, pruning, reconstruction, migration caveats.
|
||||
- Do not document: weatherapi query behavior except as a downstream consumer link if needed.
|
||||
|
||||
### NWS
|
||||
|
||||
- Path: `docs/integrations/nws.md`
|
||||
- External system or contract: NWS API and NWS forecast discussion/weather story endpoints used by current drivers.
|
||||
- Current usage: observations, alerts, hourly forecast, narrative forecast, forecast discussion, weather stories.
|
||||
- Version notes: no explicit upstream API version in code; weatherfeeder pins its own raw schema names.
|
||||
- Document: endpoint shapes used, required User-Agent, accept headers, effective time, raw schemas.
|
||||
- Do not document: SPC outlooks or unsupported NWS products.
|
||||
|
||||
### Open-Meteo
|
||||
|
||||
- Path: `docs/integrations/openmeteo.md`
|
||||
- External system or contract: Open-Meteo forecast API for current observations and hourly forecasts.
|
||||
- Current usage: JSON HTTP source, time parsing from timezone/UTC offset, hourly fields mapped to canonical forecast.
|
||||
- Version notes: no explicit upstream API version in code.
|
||||
- Document: required URL/user agent params, expected current/hourly response pieces, effective time behavior.
|
||||
- Do not document: daily forecast unless implemented.
|
||||
|
||||
### OpenWeather
|
||||
|
||||
- Path: `docs/integrations/openweather.md`
|
||||
- External system or contract: OpenWeather current weather endpoint.
|
||||
- Current usage: observation source/normalizer only.
|
||||
- Version notes: source requires `units=metric` in URL.
|
||||
- Document: metric units requirement, timestamp behavior, mapped fields, secret handling for API key.
|
||||
- Do not document: OpenWeather forecasts or alerts.
|
||||
|
||||
### NATS
|
||||
|
||||
- Path: include in `docs/operations.md` and `docs/config.md`; create `docs/integrations/nats.md` only if NATS-specific maintenance grows.
|
||||
- External system or contract: NATS publish subject configured by feedkit sink.
|
||||
- Current usage: optional sink publishes each event as JSON.
|
||||
- Document: `url`, `subject`, JSON event payload, connection failure behavior.
|
||||
- Do not document: subscriptions or server administration beyond this sink contract.
|
||||
|
||||
## Recommended Implementation Sequence
|
||||
|
||||
### Stage 1: Current-State README, CLI, and Config Docs
|
||||
|
||||
- Goal: establish accurate user/operator entry points.
|
||||
- Files to create/update/delete/move: rewrite `README.md`; create `docs/cli.md`; create `docs/config.md`.
|
||||
- Repository areas to inspect: `cmd/weatherfeeder/main.go`, `cmd/weatherfeeder/config.yml`, `../feedkit/config`, `../feedkit/sources/http.go`, `../feedkit/sinks`, `internal/sources/builtins.go`, source constructors.
|
||||
- Acceptance criteria: README concise and current; CLI doc states no flags and cwd `config.yml`; config doc includes complete implemented config reference and no secrets.
|
||||
- Suggested validation commands: `go test ./cmd/weatherfeeder`; `rg -n "only implemented sink|current MVP|--config|API.md" README.md docs/cli.md docs/config.md`.
|
||||
- Size: small enough for one implementation prompt.
|
||||
|
||||
### Stage 2: Event and Postgres Integration Contracts
|
||||
|
||||
- Goal: move and correct external contracts.
|
||||
- Files to create/update/delete/move: create `docs/integrations/events.md`; create `docs/integrations/postgres.md`; delete or replace `API.md` with a pointer; update links in `README.md` and `docs/policy/architecture.md`.
|
||||
- Repository areas to inspect: `../feedkit/event/event.go`, `standards/schema.go`, `model/*.go`, `internal/sinks/postgres/doc.go`, `schema.go`, `map.go`, normalizer tests, sink tests.
|
||||
- Acceptance criteria: event envelope field names match code; all current schemas listed; Postgres contract matches schema definition; no duplicate or stale API rows.
|
||||
- Suggested validation commands: `go test ./internal/sinks/postgres ./internal/normalizers/...`; `rg -n "effectiveAt|emittedAt|API.md|response \| string.*response" README.md docs API.md`.
|
||||
- Size: one implementation prompt if kept focused; split if examples are expanded heavily.
|
||||
|
||||
### Stage 3: Operations and Troubleshooting Docs
|
||||
|
||||
- Goal: document running and recovering the daemon.
|
||||
- Files to create/update/delete/move: create `docs/operations.md`; create `docs/troubleshooting.md`.
|
||||
- Repository areas to inspect: `cmd/weatherfeeder/main.go`, `Dockerfile`, feedkit scheduler/dispatch/source/sink implementations, config validation, source/sink constructors.
|
||||
- Acceptance criteria: operational docs describe logs, shutdown, scheduling, conditional HTTP, fanout, Postgres create/prune behavior, and common failures without inventing commands.
|
||||
- Suggested validation commands: `go test ./cmd/weatherfeeder`; `rg -n "TODO|future|planned|unimplemented" docs/operations.md docs/troubleshooting.md`.
|
||||
- Size: small enough for one implementation prompt.
|
||||
|
||||
### Stage 4: Examples Directory and Example Validation
|
||||
|
||||
- Goal: add maintained copyable configs.
|
||||
- Files to create/update/delete/move: create `examples/config.minimal.yml`, `examples/config.nats.yml`, `examples/config.postgres.yml`; optionally create `examples/config.full.yml`; update tests to load examples; update links in README/config/operations.
|
||||
- Repository areas to inspect: `cmd/weatherfeeder/config.yml`, `cmd/weatherfeeder/main_test.go`, feedkit config validation, source/sink params.
|
||||
- Acceptance criteria: examples contain no secrets, load successfully, and are linked from canonical docs.
|
||||
- Suggested validation commands: `go test ./cmd/weatherfeeder`; `rg -n "appid=|password: [^<]|token|secret" examples cmd/weatherfeeder/config.yml docs`.
|
||||
- Size: one implementation prompt.
|
||||
|
||||
### Stage 5: Developer and Internal Docs
|
||||
|
||||
- Goal: document safe change workflow and internal component boundaries.
|
||||
- Files to create/update/delete/move: create `docs/policy/development.md`; create `docs/internal/runtime.md`; create `docs/internal/sources.md`; create `docs/internal/normalizers.md`; create `docs/internal/postgres-sink.md`; lightly update package comments with stale paths/future language.
|
||||
- Repository areas to inspect: architecture policy, package docs, registries, source/normalizer/sink tests, `go.mod`.
|
||||
- Acceptance criteria: docs are inward-facing, concise, and action-oriented; package comments no longer contain stale path or future-support language outside roadmap.
|
||||
- Suggested validation commands: `go test ./...`; `rg -n "internal/standards|others later|forecasts/alerts later|future expensive steps" . -g '*.go' -g '*.md'`.
|
||||
- Size: likely one implementation prompt, but split package-comment cleanup if code comment edits are considered too broad.
|
||||
|
||||
### Stage 6: Provider Integration Notes
|
||||
|
||||
- Goal: document external provider contracts actually used by implemented source/normalizer pairs.
|
||||
- Files to create/update/delete/move: create `docs/integrations/nws.md`, `docs/integrations/openmeteo.md`, `docs/integrations/openweather.md`; optionally create `docs/integrations/nats.md` only if NATS content should stand alone.
|
||||
- Repository areas to inspect: provider source packages, provider normalizer packages, `internal/providers`, provider tests/fixtures.
|
||||
- Acceptance criteria: each integration doc is narrow, current, and excludes unimplemented products; NWS doc does not include SPC outlooks except linking to roadmap if needed.
|
||||
- Suggested validation commands: `go test ./internal/sources/... ./internal/normalizers/... ./internal/providers/...`; `rg -n "SPC|outlook|daily forecast|OpenWeather forecast|OpenWeather alert" docs/integrations`.
|
||||
- Size: one implementation prompt.
|
||||
|
||||
### Stage 7: Final Documentation Consistency Pass
|
||||
|
||||
- Goal: remove stale references and verify policy compliance.
|
||||
- Files to create/update/delete/move: any docs touched in prior stages; remove obsolete root `API.md` if not kept as pointer.
|
||||
- Repository areas to inspect: whole documentation tree, package comments, tests.
|
||||
- Acceptance criteria: every non-roadmap doc describes implemented behavior only; every doc has clear audience/scope; roadmap content remains only in `docs/roadmap`; links resolve by inspection.
|
||||
- Suggested validation commands: `go test ./...`; `find docs -type f -name '*.md' -print | sort`; `rg -n "MVP|only implemented sink|planned|future|later|deprecated|experimental|TODO|API.md|effectiveAt|emittedAt|internal/standards|appid=" README.md docs internal model standards cmd examples`.
|
||||
- Size: small enough for one implementation prompt.
|
||||
|
||||
## Validation Plan
|
||||
|
||||
Automated checks available now:
|
||||
|
||||
- `go test ./...` verifies code behavior and existing load-tested sample config.
|
||||
- `go test ./cmd/weatherfeeder` verifies `cmd/weatherfeeder/config.yml` loads and sources build scheduler jobs.
|
||||
- Focused package tests verify source drivers, normalizers, and Postgres schema/mapping.
|
||||
|
||||
Recommended new checks during implementation:
|
||||
|
||||
- Add tests that load all `examples/*.yml` with `feedkit/config.Load`.
|
||||
- Add or extend tests to ensure documented source drivers in config examples build through `internal/sources.RegisterBuiltins`.
|
||||
- Use `rg` checks for stale or prohibited terms: `only implemented sink`, `current MVP`, `future`, `later`, `planned`, `deprecated`, `experimental`, `API.md`, `effectiveAt`, `emittedAt`, `internal/standards`, and secret-like strings such as `appid=`.
|
||||
- Manually verify Markdown links because no dedicated Markdown/link checker is currently present.
|
||||
- Manually compare `docs/integrations/events.md` against `../feedkit/event/event.go`, `standards/schema.go`, and `model/*.go` before accepting.
|
||||
- Manually compare `docs/integrations/postgres.md` against `internal/sinks/postgres/schema.go` before accepting.
|
||||
|
||||
No repository-local Markdown formatter, Markdown linter, Makefile, justfile, Taskfile, or `package.json` docs tooling was found during this planning pass.
|
||||
|
||||
## Open Questions
|
||||
|
||||
No question blocks implementation of this documentation migration. Recommended decisions:
|
||||
|
||||
- Move the canonical event contract from `API.md` to `docs/integrations/events.md` to comply with the documentation policy.
|
||||
- Replace root `API.md` with a short pointer only if backward-compatible links are considered important; otherwise delete it during the migration.
|
||||
- Keep `cmd/weatherfeeder/config.yml` as the executable in-repo sample used by tests, but create public copyable examples under `examples/` and remove secret-like values from examples.
|
||||
697
docs/roadmap/implementation.md
Normal file
697
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,697 @@
|
||||
# SPC Convective Outlook Implementation Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
Implement `weatherfeeder` support for Storm Prediction Center Day 1-3 convective outlooks described in [`docs/roadmap/spc.md`](spc.md). This plan is written for an LLM coding agent and should be followed stage by stage.
|
||||
|
||||
This is a planning document only. The implementation must preserve the existing weatherfeeder architecture: sources emit raw provider events, normalizers map raw payloads into canonical model types, and sinks persist canonical schemas.
|
||||
|
||||
## Decisions
|
||||
|
||||
The following choices are fixed for this implementation:
|
||||
|
||||
- Scope is Day 1-3 SPC convective outlooks only.
|
||||
- Day 4-8 outlooks are out of scope.
|
||||
- GeoJSON files are authoritative for polygons, validity windows, issue times, outlook labels, and severity ranking.
|
||||
- Day 1-3 print pages are authoritative for discussion text.
|
||||
- RSS is optional supplemental metadata only and must not be required for correctness.
|
||||
- Do not fetch RSS by default. Include RSS only when an optional `rss_url` source param is configured.
|
||||
- A poll is atomic for required products. If any configured GeoJSON or print-page URL fails or returns a non-2xx response, return an error and emit no event.
|
||||
- Use compact GeoJSON geometry in the canonical payload for auditability and downstream display.
|
||||
- Use standard-library-first parsing. Do not add an HTML parsing dependency unless string-based extraction proves unmaintainable during implementation.
|
||||
- Keep all new planned behavior inside `weatherfeeder`; do not make `weatherapi` changes in this pass.
|
||||
|
||||
## Public Contract
|
||||
|
||||
Add schema constants in `standards/schema.go`:
|
||||
|
||||
- `SchemaRawSPCConvectiveOutlookV1 = "raw.spc.convective_outlook.v1"`
|
||||
- `SchemaWeatherOutlookV1 = "weather.outlook.v1"`
|
||||
|
||||
Add source driver:
|
||||
|
||||
- `spc_convective_outlook`
|
||||
|
||||
Add event kind:
|
||||
|
||||
- `outlook`
|
||||
|
||||
Add canonical model types:
|
||||
|
||||
- `model.WeatherOutlookRun`
|
||||
- `model.WeatherOutlook`
|
||||
|
||||
Canonical run fields:
|
||||
|
||||
```go
|
||||
type WeatherOutlookRun struct {
|
||||
LocationID string `json:"locationId,omitempty"`
|
||||
LocationName string `json:"locationName,omitempty"`
|
||||
Latitude *float64 `json:"latitude,omitempty"`
|
||||
Longitude *float64 `json:"longitude,omitempty"`
|
||||
AsOf time.Time `json:"asOf"`
|
||||
IssuedAt *time.Time `json:"issuedAt,omitempty"`
|
||||
Outlooks []WeatherOutlook `json:"outlooks"`
|
||||
}
|
||||
```
|
||||
|
||||
Canonical outlook fields:
|
||||
|
||||
```go
|
||||
type WeatherOutlook struct {
|
||||
ID string `json:"id"`
|
||||
Provider string `json:"provider"`
|
||||
Product string `json:"product"`
|
||||
Day int `json:"day"`
|
||||
OutlookType string `json:"outlookType"`
|
||||
Label string `json:"label"`
|
||||
LabelText string `json:"labelText,omitempty"`
|
||||
SeverityRank *int `json:"severityRank,omitempty"`
|
||||
ValidFrom time.Time `json:"validFrom"`
|
||||
ValidTo time.Time `json:"validTo"`
|
||||
IssuedAt time.Time `json:"issuedAt"`
|
||||
ExpiresAt time.Time `json:"expiresAt"`
|
||||
Forecaster string `json:"forecaster,omitempty"`
|
||||
Headline string `json:"headline,omitempty"`
|
||||
Summary string `json:"summary,omitempty"`
|
||||
Discussion string `json:"discussion,omitempty"`
|
||||
SourceURL string `json:"sourceUrl,omitempty"`
|
||||
ImageURL string `json:"imageUrl,omitempty"`
|
||||
ContainsLocation bool `json:"containsLocation"`
|
||||
Geometry json.RawMessage `json:"geometry"`
|
||||
}
|
||||
```
|
||||
|
||||
Required canonical fields:
|
||||
|
||||
- Run: `asOf`, `outlooks`.
|
||||
- Outlook: `id`, `provider`, `product`, `day`, `outlookType`, `label`, `validFrom`, `validTo`, `issuedAt`, `expiresAt`, `containsLocation`, `geometry`.
|
||||
|
||||
Canonical values:
|
||||
|
||||
- `provider` is `spc`.
|
||||
- `product` is `convective`.
|
||||
- `outlookType` is one of `categorical`, `tornado`, `hail`, `wind`.
|
||||
- `day` is one of `1`, `2`, `3`.
|
||||
|
||||
## Source Inputs
|
||||
|
||||
Default required GeoJSON products:
|
||||
|
||||
- `https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson`
|
||||
|
||||
Default required print-page products:
|
||||
|
||||
- `https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html`
|
||||
|
||||
Optional RSS product:
|
||||
|
||||
- `https://www.spc.noaa.gov/products/spcacrss.xml`
|
||||
|
||||
Recommended config shape:
|
||||
|
||||
```yaml
|
||||
- name: SPCConvectiveOutlookSTL
|
||||
mode: poll
|
||||
kinds: ["outlook"]
|
||||
driver: spc_convective_outlook
|
||||
every: 30m
|
||||
params:
|
||||
latitude: 38.6239
|
||||
longitude: -90.3571
|
||||
location_id: "stl"
|
||||
location_name: "St. Louis, MO"
|
||||
user_agent: "HomeOps (eric@maximumdirect.net)"
|
||||
```
|
||||
|
||||
Optional source params:
|
||||
|
||||
- `geojson_urls`: map from product key to URL, used by tests and future upstream changes.
|
||||
- `discussion_urls`: map from day key to URL, used by tests and future upstream changes.
|
||||
- `rss_url`: string; when non-empty, fetch RSS as supplemental metadata.
|
||||
|
||||
Product keys for `geojson_urls`:
|
||||
|
||||
- `day1_categorical`, `day1_tornado`, `day1_hail`, `day1_wind`
|
||||
- `day2_categorical`, `day2_tornado`, `day2_hail`, `day2_wind`
|
||||
- `day3_categorical`, `day3_tornado`, `day3_hail`, `day3_wind`
|
||||
|
||||
Discussion keys for `discussion_urls`:
|
||||
|
||||
- `day1`, `day2`, `day3`
|
||||
|
||||
## Raw Bundle Shape
|
||||
|
||||
Create a provider raw bundle type under `internal/providers/spc` or `internal/normalizers/spc` and use it consistently between source tests and normalizer tests. Prefer `internal/providers/spc` if source metadata extraction and normalizer parsing share helpers.
|
||||
|
||||
Suggested raw payload shape:
|
||||
|
||||
```go
|
||||
type RawConvectiveOutlookBundle struct {
|
||||
LocationID string `json:"locationId,omitempty"`
|
||||
LocationName string `json:"locationName,omitempty"`
|
||||
Latitude float64 `json:"latitude"`
|
||||
Longitude float64 `json:"longitude"`
|
||||
FetchedAt time.Time `json:"fetchedAt"`
|
||||
Products []RawOutlookProduct `json:"products"`
|
||||
Discussions []RawDiscussionPage `json:"discussions"`
|
||||
RSS *RawRSSFeed `json:"rss,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
```go
|
||||
type RawOutlookProduct struct {
|
||||
Key string `json:"key"`
|
||||
Day int `json:"day"`
|
||||
OutlookType string `json:"outlookType"`
|
||||
URL string `json:"url"`
|
||||
FetchedAt time.Time `json:"fetchedAt"`
|
||||
Body json.RawMessage `json:"body"`
|
||||
}
|
||||
```
|
||||
|
||||
```go
|
||||
type RawDiscussionPage struct {
|
||||
Key string `json:"key"`
|
||||
Day int `json:"day"`
|
||||
URL string `json:"url"`
|
||||
FetchedAt time.Time `json:"fetchedAt"`
|
||||
Body string `json:"body"`
|
||||
}
|
||||
```
|
||||
|
||||
```go
|
||||
type RawRSSFeed struct {
|
||||
URL string `json:"url"`
|
||||
FetchedAt time.Time `json:"fetchedAt"`
|
||||
Body string `json:"body"`
|
||||
}
|
||||
```
|
||||
|
||||
Do not put parsed canonical fields into the raw bundle except configured metadata and product keys needed to identify each fetched upstream document. Source-level timestamp parsing is allowed only for event `effectiveAt` selection.
|
||||
|
||||
## Stage 1: Provider Helpers And Fixtures
|
||||
|
||||
Goal: add deterministic SPC parsing primitives and test fixtures before wiring the source or normalizer.
|
||||
|
||||
Files to add:
|
||||
|
||||
- `internal/providers/spc/doc.go`
|
||||
- `internal/providers/spc/time.go`
|
||||
- `internal/providers/spc/product.go`
|
||||
- `internal/providers/spc/geojson.go`
|
||||
- `internal/providers/spc/discussion.go`
|
||||
- `internal/providers/spc/rss.go`, only if optional RSS parsing is implemented
|
||||
- `internal/providers/spc/testdata/day1_cat.geojson`
|
||||
- `internal/providers/spc/testdata/day2_torn.geojson`
|
||||
- `internal/providers/spc/testdata/day3_wind.geojson`
|
||||
- `internal/providers/spc/testdata/day1_prt.html`
|
||||
- `internal/providers/spc/testdata/day2_prt_corr.html`
|
||||
- `internal/providers/spc/testdata/day3_prt.html`
|
||||
|
||||
Provider helper behavior:
|
||||
|
||||
- Define stable product metadata for the 12 required GeoJSON products.
|
||||
- Define stable discussion metadata for the 3 required print-page products.
|
||||
- Parse SPC ISO timestamps from GeoJSON properties using `time.Parse(time.RFC3339, value)` after trimming whitespace.
|
||||
- Decode enough GeoJSON to expose feature properties and raw geometry without owning canonical mapping.
|
||||
- Preserve raw geometry as compact JSON bytes for later canonical use.
|
||||
- Extract print-page product text from the first useful `<pre>` block.
|
||||
- Strip embedded `<script>` blocks and remaining tags from extracted `<pre>` content.
|
||||
- Use `html.UnescapeString`, normalize CRLF to LF, and trim surrounding blank lines.
|
||||
- Parse print-page `Updated:` timestamps when present.
|
||||
- Parse discussion headline/product title from product text.
|
||||
- Parse `...SUMMARY...` content through the next section heading.
|
||||
- Preserve full product text as discussion text after cleanup.
|
||||
- Preserve correction markers such as `CORR 1` in headline or metadata when present.
|
||||
|
||||
Dependency rule:
|
||||
|
||||
- Start with `strings`, `regexp`, `encoding/json`, `encoding/xml`, and `html` from the standard library.
|
||||
- Do not add `golang.org/x/net/html` or another HTML parser unless tests show the string extraction is too brittle.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/providers/spc`
|
||||
- Product metadata contains exactly 12 GeoJSON products in day/type order.
|
||||
- Discussion metadata contains exactly 3 print pages in day order.
|
||||
- GeoJSON fixture decode exposes `VALID_ISO`, `EXPIRE_ISO`, `ISSUE_ISO`, `FORECASTER`, `LABEL`, `LABEL2`, `DN`, and geometry.
|
||||
- Print-page fixture extraction returns product text without scripts or tags.
|
||||
- Day 2 correction fixture preserves `CORR 1`.
|
||||
- Summary extraction returns only the summary paragraph content.
|
||||
- Updated timestamp parser returns UTC time when present and nil when absent.
|
||||
|
||||
Stage completion criteria:
|
||||
|
||||
- Provider helper tests pass.
|
||||
- No source, normalizer, model, standards, sink, config, or docs current-behavior files are changed in this stage unless needed for package compilation.
|
||||
|
||||
## Stage 2: Geometry Helper
|
||||
|
||||
Goal: implement point-in-polygon support independent of SPC parsing.
|
||||
|
||||
Files to add:
|
||||
|
||||
- `internal/geo/geojson.go`
|
||||
- `internal/geo/point.go`
|
||||
- `internal/geo/point_test.go`
|
||||
|
||||
Behavior:
|
||||
|
||||
- Support GeoJSON `Polygon` and `MultiPolygon` only.
|
||||
- Accept geometry as `json.RawMessage` or `[]byte`.
|
||||
- Interpret GeoJSON coordinate order as `[longitude, latitude]`.
|
||||
- Treat the first ring as the exterior ring.
|
||||
- Treat subsequent rings as holes.
|
||||
- Count boundary points as inside.
|
||||
- Return a clear error for unsupported geometry types, malformed coordinates, empty rings, or invalid JSON.
|
||||
- Use planar ray casting. This is sufficient for operational point-in-polygon checks at SPC polygon scale.
|
||||
|
||||
Tests:
|
||||
|
||||
- Point inside simple polygon returns true.
|
||||
- Point outside simple polygon returns false.
|
||||
- Point on polygon boundary returns true.
|
||||
- Point in a hole returns false.
|
||||
- Point inside one `MultiPolygon` member returns true.
|
||||
- Longitude/latitude ordering regression test fails if coordinates are reversed.
|
||||
- Unsupported geometry returns a useful error.
|
||||
|
||||
Verification:
|
||||
|
||||
```sh
|
||||
go test ./internal/geo
|
||||
```
|
||||
|
||||
Stage completion criteria:
|
||||
|
||||
- Geometry helper has no dependency on SPC, source, normalizer, or sink packages.
|
||||
|
||||
## Stage 3: Source Driver And Raw Schema
|
||||
|
||||
Goal: emit raw SPC outlook bundles from a new poll source.
|
||||
|
||||
Files to update:
|
||||
|
||||
- `standards/schema.go`
|
||||
- `internal/sources/builtins.go`
|
||||
- `internal/sources/builtins_test.go`
|
||||
|
||||
Files to add:
|
||||
|
||||
- `internal/sources/spc/convective_outlook.go`
|
||||
- `internal/sources/spc/convective_outlook_test.go`
|
||||
|
||||
Source constructor:
|
||||
|
||||
- Export `NewConvectiveOutlookSource(cfg config.SourceConfig) (*ConvectiveOutlookSource, error)`.
|
||||
- Register driver `spc_convective_outlook` in `internal/sources/builtins.go`.
|
||||
- Validate required params `latitude` and `longitude`.
|
||||
- Accept optional params `location_id`, `location_name`, `geojson_urls`, `discussion_urls`, and `rss_url`.
|
||||
- Require `user_agent` when the existing HTTP source conventions require it.
|
||||
- Advertise `Kinds() []event.Kind{event.Kind("outlook")}`.
|
||||
|
||||
Fetch behavior:
|
||||
|
||||
- Fetch all required GeoJSON URLs and print-page URLs every poll cycle.
|
||||
- Fetch optional RSS only when `rss_url` is configured and non-empty.
|
||||
- Use Accept `application/geo+json, application/json` for GeoJSON requests.
|
||||
- Use Accept `text/html, application/xhtml+xml` for print pages.
|
||||
- Use Accept `application/rss+xml, application/xml, text/xml` for RSS.
|
||||
- Respect `context.Context` on all requests.
|
||||
- Do not emit partial bundles.
|
||||
- If the complete raw bundle is unchanged from the previous successful poll, return no events. Implement source-local change detection by hashing the combined fetched bodies if feedkit HTTP conditional helpers do not directly support multi-document polling.
|
||||
- Build the raw bundle with fetched bodies and metadata.
|
||||
- Emit one raw event with kind `outlook` and schema `standards.SchemaRawSPCConvectiveOutlookV1`.
|
||||
- Use `fksources.DefaultEventID("", sourceName, effectiveAt, emittedAt)`.
|
||||
|
||||
Effective time policy:
|
||||
|
||||
- Prefer latest valid `ISSUE_ISO` across all GeoJSON features.
|
||||
- Fallback to latest valid print-page `Updated:` timestamp.
|
||||
- Fallback to RSS `lastBuildDate` if RSS was fetched and parseable.
|
||||
- Fallback to fetch time if no provider timestamp is parseable.
|
||||
- Keep fetch-time fallback as the last source effective-time option so downstream consumers still receive a stable effective time when provider metadata is absent.
|
||||
|
||||
Tests:
|
||||
|
||||
- Builtin registry builds `spc_convective_outlook` as a `PollSource`.
|
||||
- `Kinds()` returns `outlook`.
|
||||
- Constructor rejects missing `latitude` or `longitude`.
|
||||
- Poll emits one raw event with schema `raw.spc.convective_outlook.v1`.
|
||||
- Raw bundle includes 12 products and 3 discussions when defaults are overridden to test-server URLs.
|
||||
- Poll effectiveAt prefers latest GeoJSON `ISSUE_ISO`.
|
||||
- Poll effectiveAt falls back to print-page `Updated:` when GeoJSON issue times are absent.
|
||||
- Poll includes RSS only when `rss_url` is configured.
|
||||
- Unchanged response emits no events on a second poll.
|
||||
- A configured GeoJSON fetch failure returns error and emits no event.
|
||||
- A configured print-page fetch failure returns error and emits no event.
|
||||
- Tests use `httptest.Server`; do not call live SPC services.
|
||||
|
||||
Verification:
|
||||
|
||||
```sh
|
||||
go test ./internal/providers/spc ./internal/sources
|
||||
```
|
||||
|
||||
Stage completion criteria:
|
||||
|
||||
- Source emits raw bundles only.
|
||||
- No canonical model, normalizer, or sink mapping is added in this stage except schema constants required for compilation.
|
||||
|
||||
## Stage 4: Canonical Model And Normalizer
|
||||
|
||||
Goal: convert raw SPC bundles into `weather.outlook.v1` events.
|
||||
|
||||
Files to update:
|
||||
|
||||
- `standards/schema.go`
|
||||
- `internal/normalizers/builtins_test.go`
|
||||
|
||||
Files to add:
|
||||
|
||||
- `model/outlook.go`
|
||||
- `internal/normalizers/spc/convective_outlook.go`
|
||||
- `internal/normalizers/spc/register.go`
|
||||
- `internal/normalizers/spc/convective_outlook_test.go`
|
||||
|
||||
Files to update for registration:
|
||||
|
||||
- `internal/normalizers/builtins.go`
|
||||
|
||||
Normalizer behavior:
|
||||
|
||||
- Add `SPCConvectiveOutlookNormalizer`.
|
||||
- Match only `standards.SchemaRawSPCConvectiveOutlookV1`.
|
||||
- Decode `RawConvectiveOutlookBundle`.
|
||||
- Build one `model.WeatherOutlookRun` per raw bundle.
|
||||
- Map each GeoJSON feature to one `model.WeatherOutlook`.
|
||||
- Preserve feature order within each product.
|
||||
- Order products by day then outlook type in this order: `categorical`, `tornado`, `hail`, `wind`.
|
||||
- Compute `containsLocation` with `internal/geo` using the configured bundle latitude and longitude.
|
||||
- Store compact feature geometry JSON in `WeatherOutlook.Geometry`.
|
||||
- Set output schema to `standards.SchemaWeatherOutlookV1`.
|
||||
- Set output effectiveAt to run `AsOf`.
|
||||
- Use `internal/normalizers/common.Finalize` or existing normalizer helper patterns so envelope handling remains consistent.
|
||||
|
||||
GeoJSON property mapping:
|
||||
|
||||
- `VALID_ISO` maps to `validFrom` and is required.
|
||||
- `EXPIRE_ISO` maps to `validTo` and `expiresAt` and is required.
|
||||
- `ISSUE_ISO` maps to `issuedAt` and is required.
|
||||
- `FORECASTER` maps to `forecaster` and is optional.
|
||||
- `LABEL` maps to `label` and is required.
|
||||
- `LABEL2` maps to `labelText` and is optional.
|
||||
- `DN` maps to `severityRank` and is optional.
|
||||
|
||||
Derived mapping:
|
||||
|
||||
- `provider` is `spc`.
|
||||
- `product` is `convective`.
|
||||
- `day` and `outlookType` come from raw product metadata, not from brittle URL parsing when product metadata is available.
|
||||
- `id` is deterministic: join day, outlook type, normalized label, issuedAt UTC in RFC3339 format, validFrom UTC in RFC3339 format, and product-local feature index. Use a stable ASCII-safe format.
|
||||
- `sourceUrl` is the GeoJSON product URL for geometry/probability features.
|
||||
- `imageUrl` is empty for v1.
|
||||
|
||||
Discussion enrichment:
|
||||
|
||||
- Parse each print-page discussion with `internal/providers/spc` helpers.
|
||||
- Attach Day 1 print-page headline, summary, and discussion to Day 1 outlooks, Day 2 to Day 2 outlooks, and Day 3 to Day 3 outlooks.
|
||||
- Use the same discussion text for categorical, tornado, hail, and wind outlooks for the same day.
|
||||
- If a print page is fetched but discussion extraction fails, normalization should fail because print pages are required v1 inputs.
|
||||
- RSS metadata must not be required for discussion enrichment.
|
||||
|
||||
Run timestamp policy:
|
||||
|
||||
- `AsOf` is latest valid `issuedAt` across all outlook features.
|
||||
- Fallback to latest print-page `Updated:` timestamp.
|
||||
- Fallback to input event `EffectiveAt`.
|
||||
- Fallback to input event `EmittedAt`.
|
||||
- `IssuedAt` is latest valid `issuedAt` across all outlook features when any feature exists; otherwise nil.
|
||||
|
||||
Error behavior:
|
||||
|
||||
- Fail normalization if required GeoJSON timestamps are missing or unparseable.
|
||||
- Fail normalization if required `LABEL` is empty.
|
||||
- Fail normalization if required geometry is missing or invalid.
|
||||
- Fail normalization if latitude or longitude is missing or invalid in the raw bundle.
|
||||
- Include product key and feature index in errors.
|
||||
|
||||
Tests:
|
||||
|
||||
- Normalizer matches only `raw.spc.convective_outlook.v1`.
|
||||
- Builtin normalizer order includes the SPC normalizer after existing provider normalizers unless a specific order is needed.
|
||||
- Canonical schema is `weather.outlook.v1`.
|
||||
- Categorical fixture maps expected day, type, label, label text, severity rank, valid times, issue time, forecaster, source URL, and geometry.
|
||||
- Probabilistic fixtures for tornado, hail, and wind map expected `outlookType` values.
|
||||
- `containsLocation` is true for a fixture point inside a polygon.
|
||||
- `containsLocation` is false for a fixture point outside a polygon.
|
||||
- Print-page text maps to headline, summary, and discussion.
|
||||
- Day 2 correction marker `CORR 1` is preserved in headline or discussion metadata.
|
||||
- Missing optional RSS still normalizes successfully.
|
||||
- Invalid required timestamp fails with product key and feature index context.
|
||||
- Invalid geometry fails with product key and feature index context.
|
||||
- Output JSON uses the intended field names and does not expose raw bundle internals.
|
||||
|
||||
Verification:
|
||||
|
||||
```sh
|
||||
go test ./model ./internal/geo ./internal/providers/spc ./internal/normalizers ./internal/normalizers/spc
|
||||
```
|
||||
|
||||
Stage completion criteria:
|
||||
|
||||
- A raw SPC bundle normalizes into canonical `model.WeatherOutlookRun`.
|
||||
- No Postgres schema or current-behavior docs are updated in this stage unless the canonical schema documentation is intentionally updated with implementation.
|
||||
|
||||
## Stage 5: Postgres Sink
|
||||
|
||||
Goal: persist canonical outlook events through the weatherfeeder Postgres sink.
|
||||
|
||||
Files to update:
|
||||
|
||||
- `internal/sinks/postgres/schema.go`
|
||||
- `internal/sinks/postgres/map.go`
|
||||
- `internal/sinks/postgres/schema_test.go`
|
||||
- `internal/sinks/postgres/map_test.go`
|
||||
- `internal/sinks/postgres/doc.go`
|
||||
|
||||
Tables:
|
||||
|
||||
- `outlook_runs`
|
||||
- `outlooks`
|
||||
|
||||
Add table constants:
|
||||
|
||||
- `tableOutlookRuns = "outlook_runs"`
|
||||
- `tableOutlooks = "outlooks"`
|
||||
|
||||
`outlook_runs` columns:
|
||||
|
||||
- `event_id TEXT NOT NULL`
|
||||
- `event_kind TEXT NOT NULL`
|
||||
- `event_source TEXT NOT NULL`
|
||||
- `event_schema TEXT NOT NULL`
|
||||
- `event_emitted_at TIMESTAMPTZ NOT NULL`
|
||||
- `event_effective_at TIMESTAMPTZ NULL`
|
||||
- `location_id TEXT NULL`
|
||||
- `location_name TEXT NULL`
|
||||
- `latitude DOUBLE PRECISION NULL`
|
||||
- `longitude DOUBLE PRECISION NULL`
|
||||
- `as_of TIMESTAMPTZ NOT NULL`
|
||||
- `issued_at TIMESTAMPTZ NULL`
|
||||
- `outlook_count INTEGER NOT NULL`
|
||||
|
||||
`outlook_runs` keys and indexes:
|
||||
|
||||
- Primary key: `event_id`
|
||||
- Prune column: `as_of`
|
||||
- Index `idx_wf_outlook_run_location_as_of` on `location_id, as_of`
|
||||
- Index `idx_wf_outlook_run_as_of` on `as_of`
|
||||
|
||||
`outlooks` columns:
|
||||
|
||||
- `run_event_id TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE NOT NULL`
|
||||
- `outlook_index INTEGER NOT NULL`
|
||||
- `as_of TIMESTAMPTZ NOT NULL`
|
||||
- `product TEXT NOT NULL`
|
||||
- `day INTEGER NOT NULL`
|
||||
- `outlook_type TEXT NOT NULL`
|
||||
- `label TEXT NOT NULL`
|
||||
- `label_text TEXT NULL`
|
||||
- `severity_rank INTEGER NULL`
|
||||
- `valid_from TIMESTAMPTZ NOT NULL`
|
||||
- `valid_to TIMESTAMPTZ NOT NULL`
|
||||
- `issued_at TIMESTAMPTZ NOT NULL`
|
||||
- `expires_at TIMESTAMPTZ NOT NULL`
|
||||
- `forecaster TEXT NULL`
|
||||
- `headline TEXT NULL`
|
||||
- `summary TEXT NULL`
|
||||
- `discussion TEXT NULL`
|
||||
- `source_url TEXT NULL`
|
||||
- `image_url TEXT NULL`
|
||||
- `contains_location BOOLEAN NOT NULL`
|
||||
- `geometry_json TEXT NOT NULL`
|
||||
|
||||
`outlooks` keys and indexes:
|
||||
|
||||
- Primary key: `run_event_id, outlook_index`
|
||||
- Prune column: `as_of`
|
||||
- Index `idx_wf_outlooks_contains_valid` on `contains_location, valid_from, valid_to`
|
||||
- Index `idx_wf_outlooks_day_type_label` on `day, outlook_type, label`
|
||||
- Index `idx_wf_outlooks_valid` on `valid_from, valid_to`
|
||||
|
||||
Mapper behavior:
|
||||
|
||||
- Extend `mapPostgresEvent` for `standards.SchemaWeatherOutlookV1`.
|
||||
- Decode `model.WeatherOutlookRun`.
|
||||
- Require run `AsOf`.
|
||||
- Map envelope columns exactly like existing parent run tables.
|
||||
- Store all times as UTC.
|
||||
- Write one `outlook_runs` row and one `outlooks` row per outlook.
|
||||
- Use `outlook_index` as the zero-based slice index.
|
||||
- Require outlook `ID`, `Provider`, `Product`, `Day`, `OutlookType`, `Label`, `ValidFrom`, `ValidTo`, `IssuedAt`, `ExpiresAt`, and non-empty `Geometry`.
|
||||
- Store compact geometry JSON text in `geometry_json` using the existing compact JSON helper or a similar helper.
|
||||
- Preserve all outlook polygons, not only those containing the configured point.
|
||||
|
||||
Tests:
|
||||
|
||||
- Schema includes `outlook_runs` and `outlooks`.
|
||||
- Schema includes required columns and indexes.
|
||||
- Mapper writes one run row plus one row per outlook.
|
||||
- Mapper stores UTC times.
|
||||
- Mapper stores compact geometry JSON.
|
||||
- Mapper preserves `contains_location` false as false, not nil.
|
||||
- Mapper rejects missing run `asOf`.
|
||||
- Mapper rejects missing required outlook times.
|
||||
- Mapper rejects empty geometry.
|
||||
- Mapper ignores unrelated schemas unchanged.
|
||||
|
||||
Verification:
|
||||
|
||||
```sh
|
||||
go test ./internal/sinks/postgres
|
||||
```
|
||||
|
||||
Stage completion criteria:
|
||||
|
||||
- Postgres schema and mapper tests pass.
|
||||
- No weatherapi read path is added in this stage.
|
||||
|
||||
## Stage 6: Config, Runtime Wiring, And Current-Behavior Docs
|
||||
|
||||
Goal: expose the completed source/normalizer/sink behavior in maintained configuration and documentation after implementation exists.
|
||||
|
||||
Files to update:
|
||||
|
||||
- `cmd/weatherfeeder/config.yml`
|
||||
- `cmd/weatherfeeder/main_test.go`, if config load expectations need updates
|
||||
- `docs/config.md`
|
||||
- `docs/integrations/events.md`
|
||||
- `docs/integrations/postgres.md`
|
||||
- `docs/internal/sources.md`
|
||||
- `docs/internal/normalizers.md`
|
||||
- `docs/internal/postgres-sink.md`
|
||||
- `README.md`
|
||||
|
||||
Config changes:
|
||||
|
||||
- Add sample source `SPCConvectiveOutlookSTL` with driver `spc_convective_outlook`, kind `outlook`, `every: 30m`, latitude, longitude, location metadata, and `user_agent`.
|
||||
- Add `outlook` route examples for stdout, NATS, and Postgres where the existing config style includes kind lists.
|
||||
- Keep optional RSS disabled in sample config unless there is a specific operator reason to include it.
|
||||
|
||||
Docs changes:
|
||||
|
||||
- Update current-behavior docs only after the code for that behavior exists.
|
||||
- `docs/config.md` should document required and optional source params.
|
||||
- `docs/integrations/events.md` should document `weather.outlook.v1`, including field definitions, required fields, optional fields, geometry semantics, and `containsLocation` semantics.
|
||||
- `docs/integrations/postgres.md` should document `outlook_runs` and `outlooks`.
|
||||
- Internal docs should explain provider boundaries, print-page discussion parsing, and point-in-polygon behavior where useful for maintainers.
|
||||
- `README.md` should only briefly list SPC convective outlook support and link to canonical docs.
|
||||
- Do not document weatherapi endpoints in weatherfeeder current-behavior docs.
|
||||
|
||||
Tests:
|
||||
|
||||
- Existing config load tests pass.
|
||||
- Add config test coverage if the sample config is expected to build scheduler jobs for the new source.
|
||||
- Documentation examples should use the real driver name and event kind.
|
||||
|
||||
Verification:
|
||||
|
||||
```sh
|
||||
go test ./cmd/weatherfeeder ./internal/sources ./internal/normalizers ./internal/sinks/postgres
|
||||
```
|
||||
|
||||
Stage completion criteria:
|
||||
|
||||
- Sample config remains loadable.
|
||||
- Current-behavior docs match implemented code.
|
||||
- No roadmap-only claims leak into non-roadmap docs beyond the implemented behavior.
|
||||
|
||||
## Stage 7: Full Verification And Cleanup
|
||||
|
||||
Goal: validate the complete feature and remove implementation-only rough edges.
|
||||
|
||||
Run focused tests:
|
||||
|
||||
```sh
|
||||
go test ./internal/providers/spc ./internal/geo ./internal/sources ./internal/normalizers/... ./internal/sinks/postgres ./model ./cmd/weatherfeeder
|
||||
```
|
||||
|
||||
Run full tests:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Manual review checklist:
|
||||
|
||||
- `standards/schema.go` contains raw and canonical SPC schema constants.
|
||||
- `model/outlook.go` uses stable JSON tags and no provider-specific names except canonical strings.
|
||||
- Source registry includes `spc_convective_outlook`.
|
||||
- Normalizer registry includes the SPC normalizer.
|
||||
- Source tests do not use live SPC services.
|
||||
- Normalizer tests use fixtures and cover discussion parsing, geometry, and timestamp failures.
|
||||
- Postgres tests cover schema shape and mapper validation.
|
||||
- Docs follow `docs/policy/documentation.md` and use canonical homes.
|
||||
- No weatherapi files are changed.
|
||||
- No broad dependency was added without clear justification.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- A configured SPC source emits one raw bundled `outlook` event per changed complete product snapshot.
|
||||
- Normalization produces `weather.outlook.v1` events.
|
||||
- All Day 1-3 categorical, tornado, hail, and wind products are represented.
|
||||
- Each outlook includes `containsLocation` for the configured latitude and longitude.
|
||||
- Each outlook preserves compact GeoJSON geometry.
|
||||
- Day 1-3 print-page discussion text is preserved in canonical headline, summary, and discussion fields where parseable.
|
||||
- RSS metadata is optional and supplemental only.
|
||||
- Postgres sink persists outlook runs and outlook rows.
|
||||
- Sample config and current-behavior docs describe the implemented kind, driver, schema, and storage contract.
|
||||
|
||||
## Suggested Prompt Boundaries
|
||||
|
||||
This feature is too broad for a single safe implementation prompt. Use these implementation prompts in order:
|
||||
|
||||
1. Implement Stage 1 provider helpers and fixtures only.
|
||||
2. Implement Stage 2 geometry helper only.
|
||||
3. Implement Stage 3 source driver and raw schema only.
|
||||
4. Implement Stage 4 canonical model and normalizer only.
|
||||
5. Implement Stage 5 Postgres sink mapping only.
|
||||
6. Implement Stage 6 config and current-behavior documentation only.
|
||||
7. Run Stage 7 verification and perform targeted fixes only.
|
||||
|
||||
Each prompt should run its stage-specific tests before moving on. Do not proceed to the next stage with failing tests unless the failure is unrelated and explicitly documented.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Summary
|
||||
|
||||
Add `weatherfeeder` support for Storm Prediction Center convective outlooks as a new canonical outlook domain. The feature should poll SPC GeoJSON outlook products, optionally enrich them with RSS discussion metadata, compute whether the configured forecast point is inside each active outlook polygon, normalize the result into a provider-independent schema, and persist it through existing sinks.
|
||||
Add `weatherfeeder` support for Storm Prediction Center convective outlooks as a new canonical outlook domain. The feature should poll SPC GeoJSON outlook products, enrich them with Day 1-3 print-page discussion text, compute whether the configured forecast point is inside each active outlook polygon, normalize the result into a provider-independent schema, and persist it through existing sinks.
|
||||
|
||||
This is a new domain, not an extension of `weather.alert.v1`. SPC outlooks describe probabilistic/categorical risk areas over a validity window; alerts describe active actionable hazard messages.
|
||||
|
||||
@@ -64,10 +64,11 @@ Contract defaults:
|
||||
- `day` should be `1`, `2`, or `3` for this first implementation.
|
||||
- `containsLocation` is computed against configured forecast coordinates.
|
||||
- `geometry` should preserve compact GeoJSON geometry for auditability and future API use.
|
||||
- Day 4-8 outlooks are out of scope for this first implementation. They have different product semantics and should be added in a later schema-compatible extension only after Day 1-3 support is proven.
|
||||
|
||||
## Source Scope
|
||||
|
||||
The source should fetch a bundle of SPC products in one poll cycle and emit one raw event containing the fetched RSS metadata, GeoJSON products, configured point, and per-product fetch metadata.
|
||||
The source should fetch a bundle of SPC products in one poll cycle and emit one raw event containing fetched GeoJSON products, fetched Day 1-3 print pages, optional RSS metadata, configured point metadata, and per-product fetch metadata. The source may decode minimal timestamps for event identity/effective time, but provider-to-canonical mapping belongs in the normalizer.
|
||||
|
||||
Poll these GeoJSON URLs:
|
||||
|
||||
@@ -84,17 +85,31 @@ Poll these GeoJSON URLs:
|
||||
- `https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson`
|
||||
|
||||
Note: the initial candidate list duplicated Day 3 wind and omitted Day 2 wind. Use the corrected 12-product matrix above.
|
||||
Implementation note: the initial candidate list duplicated Day 3 wind and omitted Day 2 wind. Use the corrected 12-product matrix above.
|
||||
|
||||
Also poll:
|
||||
Poll these print-page discussion URLs:
|
||||
|
||||
- `https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html`
|
||||
- `https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html`
|
||||
|
||||
Print-page usage:
|
||||
|
||||
- Use print pages as the authoritative source for Day 1-3 outlook discussion text.
|
||||
- Preserve source URL, page title/header where available, page-level `Updated:` timestamp where parseable, and the raw HTML or extracted `<pre>` product block in the raw bundle.
|
||||
- Extract canonical `headline`, `summary`, `discussion`, and discussion metadata in the normalizer or shared SPC provider helper.
|
||||
- Treat the GeoJSON files, not print pages, as authoritative for polygons and validity windows.
|
||||
- Ignore page risk tables and image assets for v1 unless they are needed to support discussion parsing.
|
||||
|
||||
Optionally poll:
|
||||
|
||||
- `https://www.spc.noaa.gov/products/spcacrss.xml`
|
||||
|
||||
RSS usage:
|
||||
|
||||
- Use RSS as discussion/discovery metadata, not as the geometry source.
|
||||
- Use RSS as supplemental discovery or health metadata, not as the primary discussion or geometry source.
|
||||
- Preserve item title, link, description text, pubDate, and guid where useful.
|
||||
- Extract the narrative outlook text from the RSS item description when practical.
|
||||
- Do not depend on RSS completeness for Day 1-3 discussion text. The RSS feed can omit current outlook products, including a current Day 2 discussion.
|
||||
- Do not attempt to parse polygons from RSS HTML.
|
||||
|
||||
Polling cadence:
|
||||
@@ -119,19 +134,21 @@ Recommended config shape:
|
||||
user_agent: "HomeOps (eric@maximumdirect.net)"
|
||||
```
|
||||
|
||||
The source should own default SPC URLs, with optional params to override the RSS URL and product URLs for tests or future provider changes.
|
||||
The source should own default SPC URLs, with optional params to override GeoJSON URLs, print-page URLs, and the RSS URL for tests or future provider changes.
|
||||
|
||||
## Implementation Stages
|
||||
|
||||
### Stage 1: Raw Source and Schema
|
||||
|
||||
- Add schema constants and model placeholders.
|
||||
- Add schema constants.
|
||||
- Add `internal/sources/spc` package.
|
||||
- Implement `spc_convective_outlook` as a poll source.
|
||||
- Fetch all configured GeoJSON products and RSS during a poll cycle.
|
||||
- Fetch all configured GeoJSON products and Day 1-3 print pages during a poll cycle.
|
||||
- Fetch RSS only when configured or retained as an optional supplemental source.
|
||||
- Emit one raw event with kind `outlook` and schema `raw.spc.convective_outlook.v1`.
|
||||
- Use the latest valid `ISSUE_ISO`, RSS `lastBuildDate`, or fetch time for `effectiveAt`, in that order.
|
||||
- Use the latest valid `ISSUE_ISO`, print-page `Updated:` timestamp, RSS `lastBuildDate`, or fetch time for `effectiveAt`, in that order.
|
||||
- Preserve partial fetch errors as source errors for the first implementation; do not emit incomplete outlook bundles unless a later explicit degraded-mode policy is added.
|
||||
- Keep source tests deterministic with local test servers and fixtures; do not call live SPC services from unit tests.
|
||||
|
||||
Tests:
|
||||
|
||||
@@ -141,6 +158,7 @@ Tests:
|
||||
- source chooses expected effective time
|
||||
- source uses configured location metadata
|
||||
- source fails clearly on missing latitude/longitude
|
||||
- source fails clearly when a configured GeoJSON or print-page fetch fails
|
||||
|
||||
### Stage 2: GeoJSON and Spatial Matching
|
||||
|
||||
@@ -164,6 +182,7 @@ Tests:
|
||||
|
||||
### Stage 3: Normalizer and Canonical Mapping
|
||||
|
||||
- Add `model.WeatherOutlookRun` and `model.WeatherOutlook`.
|
||||
- Add `SPCConvectiveOutlookNormalizer`.
|
||||
- Match only `raw.spc.convective_outlook.v1`.
|
||||
- Decode the raw source bundle.
|
||||
@@ -182,8 +201,14 @@ Tests:
|
||||
- `outlookType` from product key or URL
|
||||
- `id` from day, type, label, issuedAt, validFrom, and feature index
|
||||
- `containsLocation` from configured point and GeoJSON geometry
|
||||
- Enrich narrative fields from RSS where a matching day/product item can be determined.
|
||||
- Set run `asOf` to the latest valid `issuedAt` across features, falling back to RSS `lastBuildDate`, then input event time.
|
||||
- Enrich narrative fields from the matching Day 1-3 print page:
|
||||
- parse the main product text from the `<pre>` block
|
||||
- strip embedded tags and scripts before text parsing
|
||||
- preserve line breaks in discussion text
|
||||
- parse page/title metadata, product title, issued line, valid line, correction markers such as `CORR 1`, forecaster, summary, and full discussion where practical
|
||||
- use print-page text for canonical `headline`, `summary`, and `discussion`
|
||||
- Use RSS only as optional supplemental metadata when present; missing or incomplete RSS must not prevent GeoJSON plus print-page normalization.
|
||||
- Set run `asOf` to the latest valid `issuedAt` across features, falling back to print-page `Updated:` timestamps, RSS `lastBuildDate`, then input event time.
|
||||
- Set run `issuedAt` to the latest valid `issuedAt` across features.
|
||||
- Set output event `effectiveAt` to run `asOf`.
|
||||
|
||||
@@ -193,7 +218,9 @@ Tests:
|
||||
- categorical and probabilistic products map expected fields
|
||||
- `containsLocation` is true for a known point inside a fixture polygon
|
||||
- `containsLocation` is false outside
|
||||
- missing optional RSS still permits GeoJSON normalization
|
||||
- print-page discussion text maps to canonical headline, summary, and discussion
|
||||
- correction markers such as `CORR 1` are preserved where present
|
||||
- missing optional RSS still permits GeoJSON and print-page normalization
|
||||
- malformed required GeoJSON timestamps fail with useful context
|
||||
- canonical JSON shape does not expose raw-provider-only bundle internals
|
||||
|
||||
@@ -264,16 +291,21 @@ Tests:
|
||||
|
||||
- Update sample config with `SPCConvectiveOutlookSTL`.
|
||||
- Add `outlook` to route examples.
|
||||
- Update README provider capabilities.
|
||||
- Update `API.md` with `weather.outlook.v1`.
|
||||
- Update Postgres sink docs with outlook table contract.
|
||||
- Update current-behavior docs in the implementation change, not before behavior exists.
|
||||
- Update `docs/config.md` with source params.
|
||||
- Update `docs/integrations/events.md` with `weather.outlook.v1`.
|
||||
- Update `docs/integrations/postgres.md` with outlook table contract.
|
||||
- Update relevant internal docs under `docs/internal/` if source, normalizer, or sink behavior needs maintainer context.
|
||||
- Update `README.md` only with brief implemented-provider capability and links to canonical docs.
|
||||
|
||||
Docs should clearly state:
|
||||
|
||||
- RSS is used for narrative/discussion metadata.
|
||||
- GeoJSON is used for polygons and point matching.
|
||||
- Print pages are used for Day 1-3 narrative/discussion text.
|
||||
- RSS, if retained, is supplemental and not required for complete discussion coverage.
|
||||
- GeoJSON is used for polygons, validity windows, and point matching.
|
||||
- `containsLocation` is computed by weatherfeeder at ingestion time.
|
||||
- Geometry is stored for downstream audit/display.
|
||||
- Day 4-8 outlooks are not included in the initial implementation.
|
||||
|
||||
### Stage 6: Weatherapi Follow-Up
|
||||
|
||||
@@ -291,13 +323,6 @@ Recommended API behavior:
|
||||
- active outlooks filtered by current time and `containsLocation=true`
|
||||
- optional filter query params for `day`, `outlookType`, and `containsLocation`
|
||||
|
||||
## Open Questions for Implementation
|
||||
|
||||
- Whether to include Day 4-8 probabilistic outlooks in a later version. Exclude them from v1.
|
||||
- Whether to make partial source bundles acceptable if one product URL fails. Default v1 behavior should fail the poll and retry.
|
||||
- Whether to parse detailed discussion text from RSS only, HTML pages, or both. Default v1 should use RSS only.
|
||||
- Whether to keep `geometry` in canonical JSON permanently. Default v1 should include it because it preserves source context and enables downstream display.
|
||||
|
||||
## Verification Commands
|
||||
|
||||
Run focused tests:
|
||||
@@ -318,6 +343,7 @@ go test ./...
|
||||
- Normalization produces `weather.outlook.v1` events.
|
||||
- Each outlook indicates whether the configured forecast point is inside its polygon.
|
||||
- All current Day 1-3 categorical/tornado/hail/wind products are represented.
|
||||
- RSS discussion metadata is preserved where available.
|
||||
- Day 1-3 print-page discussion text is preserved where available.
|
||||
- RSS metadata is optional and supplemental if retained.
|
||||
- Postgres sink persists outlook runs and outlook rows.
|
||||
- Sample config and public docs describe the new kind, driver, schema, and storage contract.
|
||||
|
||||
Reference in New Issue
Block a user