Document development workflow and internals
This commit is contained in:
@@ -32,4 +32,5 @@ current working directory.
|
|||||||
- [Event wire contract](docs/integrations/events.md)
|
- [Event wire contract](docs/integrations/events.md)
|
||||||
- [Postgres table contract](docs/integrations/postgres.md)
|
- [Postgres table contract](docs/integrations/postgres.md)
|
||||||
- [Architecture policy](docs/policy/architecture.md)
|
- [Architecture policy](docs/policy/architecture.md)
|
||||||
|
- [Development policy](docs/policy/development.md)
|
||||||
- [Documentation policy](docs/policy/documentation.md)
|
- [Documentation policy](docs/policy/documentation.md)
|
||||||
|
|||||||
90
docs/internal/normalizers.md
Normal file
90
docs/internal/normalizers.md
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
# Normalizer Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Normalizers convert raw provider events into canonical weather events. They are
|
||||||
|
weather-domain mapping code and should stay independent of runtime wiring,
|
||||||
|
source polling, and sink persistence.
|
||||||
|
|
||||||
|
Detailed package conventions live in `internal/normalizers/doc.go`.
|
||||||
|
|
||||||
|
## Inputs And Outputs
|
||||||
|
|
||||||
|
Inputs are raw feed events whose schemas identify provider payload shape.
|
||||||
|
Outputs are canonical feed events using `model` payloads and `weather.*`
|
||||||
|
schemas.
|
||||||
|
|
||||||
|
Current mappings:
|
||||||
|
|
||||||
|
| Raw schema | Canonical schema |
|
||||||
|
| --- | --- |
|
||||||
|
| `raw.nws.observation.v1` | `weather.observation.v1` |
|
||||||
|
| `raw.openmeteo.current.v1` | `weather.observation.v1` |
|
||||||
|
| `raw.openweather.current.v1` | `weather.observation.v1` |
|
||||||
|
| `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||||
|
| `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` |
|
||||||
|
| `raw.openmeteo.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||||
|
| `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` |
|
||||||
|
| `raw.nws.weatherstories.v1` | `weather.weather_story.v1` |
|
||||||
|
| `raw.nws.alerts.v1` | `weather.alert.v1` |
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
- Normalizers match by `Event.Schema`.
|
||||||
|
- Normalizers decode raw payloads into provider structs.
|
||||||
|
- Normalizers map provider data into canonical `model` payloads.
|
||||||
|
- Normalizers do not fetch network data, read config, route events, or write
|
||||||
|
sinks.
|
||||||
|
- Shared cross-provider behavior belongs in `internal/normalizers/common`.
|
||||||
|
- Provider-specific helper logic shared with sources belongs in
|
||||||
|
`internal/providers/<provider>`.
|
||||||
|
|
||||||
|
## Config Fields Used
|
||||||
|
|
||||||
|
Normalizers do not read config. They operate only on incoming events.
|
||||||
|
|
||||||
|
## External Adapters Used
|
||||||
|
|
||||||
|
Runtime composition creates feedkit's normalize processor with
|
||||||
|
`RequireMatch=false`. Events without a matching normalizer pass through
|
||||||
|
unchanged.
|
||||||
|
|
||||||
|
Weatherfeeder registers normalizers in a stable order:
|
||||||
|
|
||||||
|
1. NWS
|
||||||
|
2. Open-Meteo
|
||||||
|
3. OpenWeather
|
||||||
|
|
||||||
|
The current normalizers avoid ambiguous matches by using schema equality.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
Normalizers should be stateless. Shared helpers should be deterministic and free
|
||||||
|
of I/O.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Malformed required raw payload data should produce contextual errors from the
|
||||||
|
owning normalizer. Successful normalization validates the output event before it
|
||||||
|
continues through the pipeline.
|
||||||
|
|
||||||
|
`internal/normalizers/common.Finalize` preserves the input event envelope except
|
||||||
|
for schema, payload, and effective time. It also rounds canonical float values
|
||||||
|
to four digits after the decimal point.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/normalizers/builtins_test.go`
|
||||||
|
- provider normalizer tests under `internal/normalizers/nws`
|
||||||
|
- provider normalizer tests under `internal/normalizers/openmeteo`
|
||||||
|
- provider normalizer tests under `internal/normalizers/openweather`
|
||||||
|
- common helper tests under `internal/normalizers/common`
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- Match by schema constants from `standards`.
|
||||||
|
- Preserve the event envelope except for intentional canonical changes.
|
||||||
|
- Produce canonical payload structs from `model`.
|
||||||
|
- Validate normalized events before returning them.
|
||||||
|
- Keep normalizers independent of sources, sinks, config loading, and runtime
|
||||||
|
composition.
|
||||||
118
docs/internal/postgres-sink.md
Normal file
118
docs/internal/postgres-sink.md
Normal file
@@ -0,0 +1,118 @@
|
|||||||
|
# Postgres Sink Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
`internal/sinks/postgres` defines weatherfeeder's canonical-event-to-Postgres
|
||||||
|
mapping. It supplies a schema definition and mapper to feedkit's generic
|
||||||
|
Postgres sink.
|
||||||
|
|
||||||
|
The consumer-facing table contract is
|
||||||
|
[`docs/integrations/postgres.md`](../integrations/postgres.md). This document
|
||||||
|
describes the internal ownership boundary.
|
||||||
|
|
||||||
|
## Inputs And Outputs
|
||||||
|
|
||||||
|
Inputs are canonical feed events. The mapper currently handles these schemas:
|
||||||
|
|
||||||
|
- `weather.observation.v1`
|
||||||
|
- `weather.forecast.v1`
|
||||||
|
- `weather.forecast_discussion.v1`
|
||||||
|
- `weather.weather_story.v1`
|
||||||
|
- `weather.alert.v1`
|
||||||
|
|
||||||
|
Outputs are feedkit `PostgresWrite` values for weatherfeeder-owned tables.
|
||||||
|
Unsupported schemas produce no writes and no error.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
- Weatherfeeder owns table definitions in `schema.go`.
|
||||||
|
- Weatherfeeder owns canonical payload mapping in `map.go`.
|
||||||
|
- Feedkit owns database opening, table and index creation, transactions,
|
||||||
|
inserts, context-aware consumption, and prune execution.
|
||||||
|
- Postgres mapping consumes canonical events only. It should not understand raw
|
||||||
|
provider schemas.
|
||||||
|
|
||||||
|
## Config Fields Used
|
||||||
|
|
||||||
|
Weatherfeeder registers the `postgres` sink by passing `PostgresSchema()` to
|
||||||
|
feedkit. Feedkit parses sink params:
|
||||||
|
|
||||||
|
- `uri`
|
||||||
|
- `username`
|
||||||
|
- `password`
|
||||||
|
- `prune`, optional duration
|
||||||
|
|
||||||
|
Weatherfeeder-owned mapper code does not read config directly.
|
||||||
|
|
||||||
|
## External Adapters Used
|
||||||
|
|
||||||
|
The runtime registers the sink with:
|
||||||
|
|
||||||
|
```go
|
||||||
|
sinkReg.Register("postgres", fksinks.PostgresFactory(wfpgsink.PostgresSchema()))
|
||||||
|
```
|
||||||
|
|
||||||
|
Feedkit validates events at the sink boundary, calls the weatherfeeder mapper,
|
||||||
|
validates writes against the compiled schema, inserts rows in a transaction, and
|
||||||
|
optionally prunes rows older than the configured window.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
The mapper is stateless. Durable state is stored in Postgres through feedkit's
|
||||||
|
sink implementation.
|
||||||
|
|
||||||
|
## Mapping Rules
|
||||||
|
|
||||||
|
Parent rows preserve event envelope fields where the table supports them:
|
||||||
|
|
||||||
|
- `event_id`
|
||||||
|
- `event_kind`
|
||||||
|
- `event_source`
|
||||||
|
- `event_schema`
|
||||||
|
- `event_emitted_at`
|
||||||
|
- `event_effective_at`
|
||||||
|
|
||||||
|
Child rows use positional indexes to preserve canonical array order:
|
||||||
|
|
||||||
|
- `weather_index`
|
||||||
|
- `period_index`
|
||||||
|
- `message_index`
|
||||||
|
- `story_index`
|
||||||
|
- `alert_index`
|
||||||
|
- `reference_index`
|
||||||
|
|
||||||
|
Required canonical fields are validated before writes are returned:
|
||||||
|
|
||||||
|
- observations require `timestamp`;
|
||||||
|
- forecasts require `issuedAt` and `product`, and each period requires
|
||||||
|
`startTime` and `endTime`;
|
||||||
|
- forecast discussions require `issuedAt` and `product`;
|
||||||
|
- weather story runs require `asOf`, and each story requires `startTime`,
|
||||||
|
`endTime`, and `updatedAt`;
|
||||||
|
- alert runs require `asOf`, and each alert requires `id`.
|
||||||
|
|
||||||
|
Nullable canonical values are converted to SQL nulls by mapper helpers.
|
||||||
|
Observation present-weather raw values are stored as compact JSON text.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Payload decode failures, missing required fields, invalid compact JSON values,
|
||||||
|
or schema/write mismatches return errors to feedkit's sink. Feedkit rolls back
|
||||||
|
the transaction when a write fails.
|
||||||
|
|
||||||
|
Unsupported canonical schemas are ignored by this mapper so other routed events
|
||||||
|
can use different sinks without Postgres-specific failures.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/sinks/postgres/schema_test.go`
|
||||||
|
- `internal/sinks/postgres/map_test.go`
|
||||||
|
- feedkit Postgres sink tests when changing generic sink behavior assumptions
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- Persist only canonical schemas.
|
||||||
|
- Preserve event envelope fields in parent rows.
|
||||||
|
- Preserve array order with child positional indexes.
|
||||||
|
- Validate required fields before writing.
|
||||||
|
- Keep table-contract docs synchronized with schema and mapper changes.
|
||||||
110
docs/internal/runtime.md
Normal file
110
docs/internal/runtime.md
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
# Runtime Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
`cmd/weatherfeeder` wires the daemon together. It owns process setup and runtime
|
||||||
|
composition; provider mapping, source fetching details, and sink persistence
|
||||||
|
rules stay in their owning packages.
|
||||||
|
|
||||||
|
## Inputs And Outputs
|
||||||
|
|
||||||
|
The executable reads `config.yml` from the current working directory through
|
||||||
|
feedkit config loading. It builds configured sources, scheduler jobs, sinks, and
|
||||||
|
routes, then runs source polling and sink dispatch until shutdown.
|
||||||
|
|
||||||
|
Inputs are configured source polls. Outputs are feed events delivered to the
|
||||||
|
configured sinks.
|
||||||
|
|
||||||
|
## Runtime Flow
|
||||||
|
|
||||||
|
The implemented flow is:
|
||||||
|
|
||||||
|
1. load `config.yml`;
|
||||||
|
2. register weatherfeeder source drivers;
|
||||||
|
3. register feedkit built-in sinks and the weatherfeeder Postgres sink;
|
||||||
|
4. build source inputs and scheduler jobs;
|
||||||
|
5. validate configured expected kinds against source-advertised kinds;
|
||||||
|
6. build sinks and compile routes;
|
||||||
|
7. run the processor chain `normalize`, then `dedupe`;
|
||||||
|
8. run the scheduler and dispatcher concurrently;
|
||||||
|
9. shut down on signal or fatal scheduler/dispatcher error.
|
||||||
|
|
||||||
|
The in-process event channel is buffered to 256 events. The dedupe processor is
|
||||||
|
bounded by `dedupeMaxEntries`, currently 2048.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
- Runtime composition belongs in `cmd/weatherfeeder/main.go`.
|
||||||
|
- Source driver behavior belongs under `internal/sources`.
|
||||||
|
- Normalizer behavior belongs under `internal/normalizers`.
|
||||||
|
- Canonical payloads and schema strings belong in `model` and `standards`.
|
||||||
|
- Postgres mapping belongs under `internal/sinks/postgres`.
|
||||||
|
|
||||||
|
`cmd/weatherfeeder` should stay thin and should not contain provider parsing,
|
||||||
|
canonical mapping, or table-mapping rules.
|
||||||
|
|
||||||
|
## Config Fields Used
|
||||||
|
|
||||||
|
Runtime wiring consumes the feedkit top-level config sections:
|
||||||
|
|
||||||
|
- `sources`: source driver selection, source name, mode, cadence, expected kinds,
|
||||||
|
and driver params;
|
||||||
|
- `sinks`: sink driver selection, sink name, and sink params;
|
||||||
|
- `routes`: event-kind routing to named sinks.
|
||||||
|
|
||||||
|
The executable does not expose CLI flags or config path discovery.
|
||||||
|
|
||||||
|
## External Adapters Used
|
||||||
|
|
||||||
|
Runtime composition uses feedkit for:
|
||||||
|
|
||||||
|
- config loading;
|
||||||
|
- source registry and expected-kind validation;
|
||||||
|
- scheduler job construction;
|
||||||
|
- processor registry and chain execution;
|
||||||
|
- normalization and dedupe processors;
|
||||||
|
- sink registry and built-in sinks;
|
||||||
|
- route compilation and dispatch.
|
||||||
|
|
||||||
|
Weatherfeeder registers its own source drivers and its Postgres schema mapper.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
Weatherfeeder-owned runtime state is in process:
|
||||||
|
|
||||||
|
- event channel contents;
|
||||||
|
- the bounded dedupe key set;
|
||||||
|
- source instances and their HTTP conditional validators;
|
||||||
|
- scheduler and dispatcher goroutines.
|
||||||
|
|
||||||
|
There is no weatherfeeder-owned durable scheduler state, checkpoint, replay log,
|
||||||
|
or resume marker. Durable persistence is owned by configured external sinks.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Startup failures are fatal and include context such as config index, source name,
|
||||||
|
sink name, driver name, or the operation that failed.
|
||||||
|
|
||||||
|
At runtime, scheduler and dispatcher errors are sent to a shared error channel.
|
||||||
|
Context cancellation and deadline errors are treated as normal shutdown. Any
|
||||||
|
other scheduler or dispatcher error is logged as fatal and cancels the process
|
||||||
|
context.
|
||||||
|
|
||||||
|
The daemon handles `os.Interrupt` and `SIGTERM` with `signal.NotifyContext`.
|
||||||
|
After both runtime goroutines return, it logs `shutdown complete`.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `cmd/weatherfeeder/main_test.go`
|
||||||
|
- source registry tests under `internal/sources`
|
||||||
|
- normalizer registration tests under `internal/normalizers`
|
||||||
|
- feedkit scheduler, processor, dispatch, and sink tests when changing runtime
|
||||||
|
infrastructure usage
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- Keep normalization before dedupe.
|
||||||
|
- Keep queue sizes and dedupe bounds explicit.
|
||||||
|
- Preserve context-aware shutdown.
|
||||||
|
- Keep runtime wiring separate from domain mapping and persistence rules.
|
||||||
|
- Keep startup validation failures loud and contextual.
|
||||||
105
docs/internal/sources.md
Normal file
105
docs/internal/sources.md
Normal file
@@ -0,0 +1,105 @@
|
|||||||
|
# Source Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Source packages poll upstream weather providers and emit raw feed events. They
|
||||||
|
are adapters, not canonical mappers.
|
||||||
|
|
||||||
|
Sources should decode only the metadata needed for event identity, effective
|
||||||
|
time, and routing policy. Full provider payload interpretation belongs in
|
||||||
|
normalizers.
|
||||||
|
|
||||||
|
## Inputs And Outputs
|
||||||
|
|
||||||
|
Inputs are feedkit `config.SourceConfig` values and upstream HTTP responses.
|
||||||
|
Outputs are feed events whose payloads are raw provider JSON and whose schemas
|
||||||
|
come from `standards`.
|
||||||
|
|
||||||
|
Current drivers:
|
||||||
|
|
||||||
|
| Driver | Kind | Raw schema |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `nws_observation` | `observation` | `raw.nws.observation.v1` |
|
||||||
|
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` |
|
||||||
|
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` |
|
||||||
|
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` |
|
||||||
|
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` |
|
||||||
|
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` |
|
||||||
|
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` |
|
||||||
|
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` |
|
||||||
|
| `openweather_observation` | `observation` | `raw.openweather.current.v1` |
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
- Source constructors validate source-specific params.
|
||||||
|
- Sources use feedkit HTTP helpers for HTTP polling.
|
||||||
|
- Sources emit raw events and should not build canonical `model` payloads.
|
||||||
|
- Provider helper packages under `internal/providers/<provider>` hold shared
|
||||||
|
parsing and validation helpers.
|
||||||
|
- Registration is centralized in `internal/sources/builtins.go`.
|
||||||
|
|
||||||
|
## Config Fields Used
|
||||||
|
|
||||||
|
All current source drivers use feedkit `HTTPSource`.
|
||||||
|
|
||||||
|
Required params:
|
||||||
|
|
||||||
|
- `url`
|
||||||
|
- `user_agent`
|
||||||
|
|
||||||
|
Optional params:
|
||||||
|
|
||||||
|
- `conditional`, default `true`;
|
||||||
|
- `http_timeout`;
|
||||||
|
- `http_response_body_limit_bytes`.
|
||||||
|
|
||||||
|
OpenWeather observation sources additionally require the configured URL to use
|
||||||
|
metric units. This is enforced by `internal/providers/openweather`.
|
||||||
|
|
||||||
|
Source-level `kinds`, when configured, are validated against the source's
|
||||||
|
advertised `Kinds()`.
|
||||||
|
|
||||||
|
## External Adapters Used
|
||||||
|
|
||||||
|
Sources use feedkit's HTTP helper for:
|
||||||
|
|
||||||
|
- request construction;
|
||||||
|
- `User-Agent` and `Accept` headers;
|
||||||
|
- optional conditional GET validators;
|
||||||
|
- response body size limits;
|
||||||
|
- JSON raw-message fetches.
|
||||||
|
|
||||||
|
NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times
|
||||||
|
with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
HTTP conditional validators are held in each source instance. They are not
|
||||||
|
persisted across process restarts.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Constructor failures are returned during startup and stop the daemon. Polling
|
||||||
|
failures are returned to the scheduler.
|
||||||
|
|
||||||
|
If a source cannot decode minimal metadata from an otherwise successful upstream
|
||||||
|
response, it still emits the raw event when possible. The event then falls back
|
||||||
|
to default ID/effective-time behavior from feedkit source helpers.
|
||||||
|
|
||||||
|
Unchanged conditional responses return no events and no error.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/sources/builtins_test.go`
|
||||||
|
- provider source tests under `internal/sources/nws`
|
||||||
|
- provider source tests under `internal/sources/openmeteo`
|
||||||
|
- provider source tests under `internal/sources/openweather`
|
||||||
|
- provider helper tests under `internal/providers`
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- Emit raw provider schemas from `standards`.
|
||||||
|
- Keep provider-to-canonical mapping out of sources.
|
||||||
|
- Keep HTTP behavior context-aware.
|
||||||
|
- Keep driver registration explicit and stable.
|
||||||
|
- Keep source tests independent of live upstream services.
|
||||||
202
docs/policy/development.md
Normal file
202
docs/policy/development.md
Normal file
@@ -0,0 +1,202 @@
|
|||||||
|
# Development Policy
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document describes how to change `weatherfeeder` safely. It is for
|
||||||
|
maintainers and coding agents working in the repository.
|
||||||
|
|
||||||
|
Use this alongside the [architecture policy](architecture.md). User-facing CLI,
|
||||||
|
configuration, operations, and wire-contract details belong in their canonical
|
||||||
|
docs, not here.
|
||||||
|
|
||||||
|
## Repository Layout
|
||||||
|
|
||||||
|
- `cmd/weatherfeeder/`: executable wiring, sample `config.yml`, and runtime
|
||||||
|
composition tests.
|
||||||
|
- `model/`: canonical weather payload structs. JSON tags are part of the wire
|
||||||
|
contract.
|
||||||
|
- `standards/`: schema strings, versioning conventions, WMO constants, and
|
||||||
|
shared wire-format policy.
|
||||||
|
- `internal/sources/`: source adapters that poll upstream providers and emit raw
|
||||||
|
feed events.
|
||||||
|
- `internal/normalizers/`: raw-to-canonical event transforms.
|
||||||
|
- `internal/providers/`: pure provider helper code shared by sources and
|
||||||
|
normalizers.
|
||||||
|
- `internal/sinks/postgres/`: weatherfeeder-owned Postgres schema and canonical
|
||||||
|
event mapper.
|
||||||
|
- `docs/`: current behavior, policies, integration contracts, and roadmap files.
|
||||||
|
- `examples/`: maintained, copyable configuration examples.
|
||||||
|
|
||||||
|
## Build And Test
|
||||||
|
|
||||||
|
Run the full test suite before committing behavior or documentation changes that
|
||||||
|
depend on code behavior:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go test ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
Use narrower commands while iterating:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go test ./cmd/weatherfeeder
|
||||||
|
go test ./internal/sources/...
|
||||||
|
go test ./internal/normalizers/...
|
||||||
|
go test ./internal/sinks/postgres
|
||||||
|
```
|
||||||
|
|
||||||
|
Format Go code before committing:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
gofmt -w <changed-go-files>
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not require live upstream weather services, NATS, or Postgres for unit tests.
|
||||||
|
Use fixtures, local test servers, and package-level tests.
|
||||||
|
|
||||||
|
## Coding Conventions
|
||||||
|
|
||||||
|
- Keep `cmd/weatherfeeder` focused on composition: config load, registry setup,
|
||||||
|
scheduler jobs, processor chain, dispatch, signal handling, and logging.
|
||||||
|
- Keep source fetching separate from normalizer mapping.
|
||||||
|
- Match normalizers by schema constants from `standards`, not source names.
|
||||||
|
- Keep provider-specific helper code under `internal/providers/<provider>` when
|
||||||
|
both sources and normalizers use it.
|
||||||
|
- Keep cross-provider normalizer helpers pure and deterministic under
|
||||||
|
`internal/normalizers/common`.
|
||||||
|
- Keep sink persistence mapping isolated under `internal/sinks/<sink>`.
|
||||||
|
- Wrap errors with operation context, but do not include whole upstream payloads
|
||||||
|
in errors or logs by default.
|
||||||
|
- Prefer explicit registries and small package-level constructors over hidden
|
||||||
|
global behavior.
|
||||||
|
|
||||||
|
## Dependency Policy
|
||||||
|
|
||||||
|
Prefer the Go standard library unless a dependency materially improves
|
||||||
|
maintainability or interoperability.
|
||||||
|
|
||||||
|
`feedkit` owns generic daemon infrastructure for config, HTTP source helpers,
|
||||||
|
scheduling, processors, dispatch, and sinks. Weatherfeeder code should contain
|
||||||
|
weather-domain behavior and narrow adapter logic rather than duplicating feedkit
|
||||||
|
infrastructure.
|
||||||
|
|
||||||
|
Do not add broad dependencies for small conveniences. Do not let
|
||||||
|
dependency-specific types leak across package boundaries unless that dependency
|
||||||
|
is the package contract.
|
||||||
|
|
||||||
|
## Adding Config Fields
|
||||||
|
|
||||||
|
Generic config shape is owned by feedkit. Weatherfeeder-specific config behavior
|
||||||
|
belongs in source or sink constructors, registry setup, and tests.
|
||||||
|
|
||||||
|
When adding config behavior:
|
||||||
|
|
||||||
|
- validate required params at the adapter boundary;
|
||||||
|
- keep secrets in environment variables or placeholders, not committed values;
|
||||||
|
- update [configuration docs](../config.md);
|
||||||
|
- update maintained examples when the change affects normal operation;
|
||||||
|
- add or update config-load tests for example files when practical.
|
||||||
|
|
||||||
|
## Adding CLI Flags
|
||||||
|
|
||||||
|
The executable currently reads `config.yml` from the current working directory.
|
||||||
|
If CLI flags are added:
|
||||||
|
|
||||||
|
- keep parsing in `cmd/weatherfeeder`;
|
||||||
|
- avoid moving config policy into domain packages;
|
||||||
|
- update [CLI docs](../cli.md);
|
||||||
|
- update tests that exercise command behavior.
|
||||||
|
|
||||||
|
## Adding A Source Driver
|
||||||
|
|
||||||
|
Source drivers should fetch upstream data and emit raw events with minimal
|
||||||
|
metadata decoding.
|
||||||
|
|
||||||
|
Checklist:
|
||||||
|
|
||||||
|
- implement the driver under `internal/sources/<provider>`;
|
||||||
|
- build from `config.SourceConfig`;
|
||||||
|
- validate required params in the constructor;
|
||||||
|
- use feedkit HTTP helpers for HTTP polling when applicable;
|
||||||
|
- emit raw schema constants from `standards`;
|
||||||
|
- advertise emitted kinds through `Kinds()`;
|
||||||
|
- decode only metadata needed for event ID and effective time;
|
||||||
|
- register the driver in `internal/sources/builtins.go`;
|
||||||
|
- add constructor, kind, and polling tests;
|
||||||
|
- update config docs and examples when operators need new configuration;
|
||||||
|
- add provider integration notes when the provider contract needs maintenance
|
||||||
|
context.
|
||||||
|
|
||||||
|
## Adding A Normalizer
|
||||||
|
|
||||||
|
Normalizers own provider-to-canonical mapping.
|
||||||
|
|
||||||
|
Checklist:
|
||||||
|
|
||||||
|
- add one normalizer type per normalizer file;
|
||||||
|
- match using `Event.Schema`;
|
||||||
|
- decode raw payloads into provider structs;
|
||||||
|
- map to canonical `model` payloads;
|
||||||
|
- use `internal/normalizers/common.Finalize` so envelope handling and float
|
||||||
|
rounding stay consistent;
|
||||||
|
- preserve input envelope fields except schema, payload, and effective time;
|
||||||
|
- register through the provider package and `internal/normalizers/builtins.go`;
|
||||||
|
- add tests for schema matching, key payload fields, effective time, malformed
|
||||||
|
required data, and output validation.
|
||||||
|
|
||||||
|
## Adding Canonical Models Or Schemas
|
||||||
|
|
||||||
|
Canonical event changes affect multiple contracts.
|
||||||
|
|
||||||
|
Checklist:
|
||||||
|
|
||||||
|
- update payload structs in `model`;
|
||||||
|
- add or update schema constants in `standards`;
|
||||||
|
- update [event wire contract docs](../integrations/events.md);
|
||||||
|
- update normalizers that produce the schema;
|
||||||
|
- update Postgres mapping if the schema is persisted;
|
||||||
|
- add tests for wire shape and mapper behavior.
|
||||||
|
|
||||||
|
## Adding Postgres Mapping
|
||||||
|
|
||||||
|
Weatherfeeder owns the canonical-event-to-table mapping. Feedkit owns the
|
||||||
|
generic Postgres sink mechanics.
|
||||||
|
|
||||||
|
Checklist:
|
||||||
|
|
||||||
|
- update `internal/sinks/postgres/schema.go`;
|
||||||
|
- update `internal/sinks/postgres/map.go`;
|
||||||
|
- preserve event envelope columns in parent rows when the table supports them;
|
||||||
|
- validate required canonical fields before writing;
|
||||||
|
- use positional indexes for child rows that represent arrays;
|
||||||
|
- update mapper and schema tests;
|
||||||
|
- update [Postgres integration docs](../integrations/postgres.md) when the table
|
||||||
|
contract changes.
|
||||||
|
|
||||||
|
## Examples And Documentation
|
||||||
|
|
||||||
|
Documentation must follow the [documentation policy](documentation.md).
|
||||||
|
|
||||||
|
When behavior changes, update the canonical docs in the same change:
|
||||||
|
|
||||||
|
- config shape: `docs/config.md`;
|
||||||
|
- CLI behavior: `docs/cli.md`;
|
||||||
|
- operations and recovery: `docs/operations.md`;
|
||||||
|
- troubleshooting: `docs/troubleshooting.md`;
|
||||||
|
- external contracts: `docs/integrations/`;
|
||||||
|
- internal component behavior: `docs/internal/`;
|
||||||
|
- copyable configs: `examples/`.
|
||||||
|
|
||||||
|
Keep roadmap content under `docs/roadmap/`. Current-behavior docs must describe
|
||||||
|
implemented behavior only.
|
||||||
|
|
||||||
|
## Review Checklist
|
||||||
|
|
||||||
|
Before committing:
|
||||||
|
|
||||||
|
- run focused tests for changed packages;
|
||||||
|
- run `go test ./...` for broad behavior or documentation changes tied to code;
|
||||||
|
- verify maintained examples still load when examples or config docs changed;
|
||||||
|
- check links in changed docs;
|
||||||
|
- search for stale paths, unsupported features, and secret-like values;
|
||||||
|
- keep unrelated refactors out of the change.
|
||||||
@@ -70,7 +70,7 @@
|
|||||||
//
|
//
|
||||||
// weather.<kind>.vN
|
// weather.<kind>.vN
|
||||||
//
|
//
|
||||||
// weatherfeeder centralizes schema strings in internal/standards/schema.go.
|
// weatherfeeder centralizes schema strings in standards/schema.go.
|
||||||
// Always use those constants (do not inline schema strings).
|
// Always use those constants (do not inline schema strings).
|
||||||
//
|
//
|
||||||
// Example mappings:
|
// Example mappings:
|
||||||
@@ -101,8 +101,8 @@
|
|||||||
// Every normalizer type must have a doc comment that states:
|
// Every normalizer type must have a doc comment that states:
|
||||||
//
|
//
|
||||||
// - what it converts (e.g., “OpenWeather current -> WeatherObservation”)
|
// - what it converts (e.g., “OpenWeather current -> WeatherObservation”)
|
||||||
// - which raw schema it matches (constant identifier from internal/standards)
|
// - which raw schema it matches (constant identifier from standards)
|
||||||
// - which canonical schema it produces (constant identifier from internal/standards)
|
// - which canonical schema it produces (constant identifier from standards)
|
||||||
// - any special caveats (units, day/night inference, missing fields, etc.)
|
// - any special caveats (units, day/night inference, missing fields, etc.)
|
||||||
//
|
//
|
||||||
// Including literal schema string values is optional,
|
// Including literal schema string values is optional,
|
||||||
|
|||||||
@@ -37,7 +37,7 @@ func (AlertsNormalizer) Match(e event.Event) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (AlertsNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
func (AlertsNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||||
|
|
||||||
// If we can't derive AsOf from the payload, fall back to the existing event envelope.
|
// If we can't derive AsOf from the payload, fall back to the existing event envelope.
|
||||||
fallbackAsOf := in.EmittedAt.UTC()
|
fallbackAsOf := in.EmittedAt.UTC()
|
||||||
|
|||||||
@@ -42,7 +42,7 @@ func (ForecastNormalizer) Match(e event.Event) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||||
|
|
||||||
return normalizeForecastEventBySchema(in)
|
return normalizeForecastEventBySchema(in)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||||
|
|
||||||
return normcommon.NormalizeJSON(
|
return normcommon.NormalizeJSON(
|
||||||
in,
|
in,
|
||||||
|
|||||||
@@ -33,7 +33,7 @@ func (ForecastNormalizer) Match(e event.Event) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||||
|
|
||||||
// If present, prefer the existing event EmittedAt as IssuedAt.
|
// If present, prefer the existing event EmittedAt as IssuedAt.
|
||||||
var fallbackIssued time.Time
|
var fallbackIssued time.Time
|
||||||
|
|||||||
@@ -40,7 +40,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||||
|
|
||||||
return normcommon.NormalizeJSON(
|
return normcommon.NormalizeJSON(
|
||||||
in,
|
in,
|
||||||
|
|||||||
@@ -8,8 +8,7 @@ import (
|
|||||||
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
|
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
|
||||||
)
|
)
|
||||||
|
|
||||||
// This file holds provider-specific helpers that are shared across multiple
|
// This file holds provider-specific helpers for OpenWeather normalizers.
|
||||||
// OpenWeather normalizers (observations today; forecasts/alerts later).
|
|
||||||
// Keeping these out of observation.go helps preserve the "one normalizer per file"
|
// Keeping these out of observation.go helps preserve the "one normalizer per file"
|
||||||
// convention while avoiding duplication.
|
// convention while avoiding duplication.
|
||||||
|
|
||||||
|
|||||||
@@ -37,7 +37,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||||
|
|
||||||
return normcommon.NormalizeJSON(
|
return normcommon.NormalizeJSON(
|
||||||
in,
|
in,
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// Package openweather contains provider-specific helper code for OpenWeather used by
|
// Package openweather contains provider-specific helper code for OpenWeather
|
||||||
// both sources and normalizers.
|
// used by sources and normalizers.
|
||||||
//
|
//
|
||||||
// Rules:
|
// Rules:
|
||||||
// - No network I/O here.
|
// - No network I/O here.
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
// Package model defines weatherfeeder's canonical domain payload types.
|
// Package model defines weatherfeeder's canonical domain payload types.
|
||||||
//
|
//
|
||||||
// These structs are emitted as the Payload of canonical events (schemas "weather.*.vN").
|
// These structs are emitted as the Payload of canonical events (schemas "weather.*.vN").
|
||||||
// JSON tags are treated as part of the wire contract for sinks (stdout today; others later).
|
// JSON tags are treated as part of the wire contract for configured sinks.
|
||||||
//
|
//
|
||||||
// Compatibility guidance:
|
// Compatibility guidance:
|
||||||
// - Prefer additive changes.
|
// - Prefer additive changes.
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
// File: internal/standards/doc.go
|
// File: standards/doc.go
|
||||||
//
|
//
|
||||||
// Package standards defines weatherfeeder’s provider-agnostic “project law”:
|
// Package standards defines weatherfeeder’s provider-agnostic “project law”:
|
||||||
//
|
//
|
||||||
|
|||||||
Reference in New Issue
Block a user