Document development workflow and internals
This commit is contained in:
@@ -32,4 +32,5 @@ current working directory.
|
||||
- [Event wire contract](docs/integrations/events.md)
|
||||
- [Postgres table contract](docs/integrations/postgres.md)
|
||||
- [Architecture policy](docs/policy/architecture.md)
|
||||
- [Development policy](docs/policy/development.md)
|
||||
- [Documentation policy](docs/policy/documentation.md)
|
||||
|
||||
90
docs/internal/normalizers.md
Normal file
90
docs/internal/normalizers.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# Normalizer Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
Normalizers convert raw provider events into canonical weather events. They are
|
||||
weather-domain mapping code and should stay independent of runtime wiring,
|
||||
source polling, and sink persistence.
|
||||
|
||||
Detailed package conventions live in `internal/normalizers/doc.go`.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs are raw feed events whose schemas identify provider payload shape.
|
||||
Outputs are canonical feed events using `model` payloads and `weather.*`
|
||||
schemas.
|
||||
|
||||
Current mappings:
|
||||
|
||||
| Raw schema | Canonical schema |
|
||||
| --- | --- |
|
||||
| `raw.nws.observation.v1` | `weather.observation.v1` |
|
||||
| `raw.openmeteo.current.v1` | `weather.observation.v1` |
|
||||
| `raw.openweather.current.v1` | `weather.observation.v1` |
|
||||
| `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||
| `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` |
|
||||
| `raw.openmeteo.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||
| `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` |
|
||||
| `raw.nws.weatherstories.v1` | `weather.weather_story.v1` |
|
||||
| `raw.nws.alerts.v1` | `weather.alert.v1` |
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Normalizers match by `Event.Schema`.
|
||||
- Normalizers decode raw payloads into provider structs.
|
||||
- Normalizers map provider data into canonical `model` payloads.
|
||||
- Normalizers do not fetch network data, read config, route events, or write
|
||||
sinks.
|
||||
- Shared cross-provider behavior belongs in `internal/normalizers/common`.
|
||||
- Provider-specific helper logic shared with sources belongs in
|
||||
`internal/providers/<provider>`.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
Normalizers do not read config. They operate only on incoming events.
|
||||
|
||||
## External Adapters Used
|
||||
|
||||
Runtime composition creates feedkit's normalize processor with
|
||||
`RequireMatch=false`. Events without a matching normalizer pass through
|
||||
unchanged.
|
||||
|
||||
Weatherfeeder registers normalizers in a stable order:
|
||||
|
||||
1. NWS
|
||||
2. Open-Meteo
|
||||
3. OpenWeather
|
||||
|
||||
The current normalizers avoid ambiguous matches by using schema equality.
|
||||
|
||||
## State
|
||||
|
||||
Normalizers should be stateless. Shared helpers should be deterministic and free
|
||||
of I/O.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Malformed required raw payload data should produce contextual errors from the
|
||||
owning normalizer. Successful normalization validates the output event before it
|
||||
continues through the pipeline.
|
||||
|
||||
`internal/normalizers/common.Finalize` preserves the input event envelope except
|
||||
for schema, payload, and effective time. It also rounds canonical float values
|
||||
to four digits after the decimal point.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/normalizers/builtins_test.go`
|
||||
- provider normalizer tests under `internal/normalizers/nws`
|
||||
- provider normalizer tests under `internal/normalizers/openmeteo`
|
||||
- provider normalizer tests under `internal/normalizers/openweather`
|
||||
- common helper tests under `internal/normalizers/common`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Match by schema constants from `standards`.
|
||||
- Preserve the event envelope except for intentional canonical changes.
|
||||
- Produce canonical payload structs from `model`.
|
||||
- Validate normalized events before returning them.
|
||||
- Keep normalizers independent of sources, sinks, config loading, and runtime
|
||||
composition.
|
||||
118
docs/internal/postgres-sink.md
Normal file
118
docs/internal/postgres-sink.md
Normal file
@@ -0,0 +1,118 @@
|
||||
# Postgres Sink Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/sinks/postgres` defines weatherfeeder's canonical-event-to-Postgres
|
||||
mapping. It supplies a schema definition and mapper to feedkit's generic
|
||||
Postgres sink.
|
||||
|
||||
The consumer-facing table contract is
|
||||
[`docs/integrations/postgres.md`](../integrations/postgres.md). This document
|
||||
describes the internal ownership boundary.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs are canonical feed events. The mapper currently handles these schemas:
|
||||
|
||||
- `weather.observation.v1`
|
||||
- `weather.forecast.v1`
|
||||
- `weather.forecast_discussion.v1`
|
||||
- `weather.weather_story.v1`
|
||||
- `weather.alert.v1`
|
||||
|
||||
Outputs are feedkit `PostgresWrite` values for weatherfeeder-owned tables.
|
||||
Unsupported schemas produce no writes and no error.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Weatherfeeder owns table definitions in `schema.go`.
|
||||
- Weatherfeeder owns canonical payload mapping in `map.go`.
|
||||
- Feedkit owns database opening, table and index creation, transactions,
|
||||
inserts, context-aware consumption, and prune execution.
|
||||
- Postgres mapping consumes canonical events only. It should not understand raw
|
||||
provider schemas.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
Weatherfeeder registers the `postgres` sink by passing `PostgresSchema()` to
|
||||
feedkit. Feedkit parses sink params:
|
||||
|
||||
- `uri`
|
||||
- `username`
|
||||
- `password`
|
||||
- `prune`, optional duration
|
||||
|
||||
Weatherfeeder-owned mapper code does not read config directly.
|
||||
|
||||
## External Adapters Used
|
||||
|
||||
The runtime registers the sink with:
|
||||
|
||||
```go
|
||||
sinkReg.Register("postgres", fksinks.PostgresFactory(wfpgsink.PostgresSchema()))
|
||||
```
|
||||
|
||||
Feedkit validates events at the sink boundary, calls the weatherfeeder mapper,
|
||||
validates writes against the compiled schema, inserts rows in a transaction, and
|
||||
optionally prunes rows older than the configured window.
|
||||
|
||||
## State
|
||||
|
||||
The mapper is stateless. Durable state is stored in Postgres through feedkit's
|
||||
sink implementation.
|
||||
|
||||
## Mapping Rules
|
||||
|
||||
Parent rows preserve event envelope fields where the table supports them:
|
||||
|
||||
- `event_id`
|
||||
- `event_kind`
|
||||
- `event_source`
|
||||
- `event_schema`
|
||||
- `event_emitted_at`
|
||||
- `event_effective_at`
|
||||
|
||||
Child rows use positional indexes to preserve canonical array order:
|
||||
|
||||
- `weather_index`
|
||||
- `period_index`
|
||||
- `message_index`
|
||||
- `story_index`
|
||||
- `alert_index`
|
||||
- `reference_index`
|
||||
|
||||
Required canonical fields are validated before writes are returned:
|
||||
|
||||
- observations require `timestamp`;
|
||||
- forecasts require `issuedAt` and `product`, and each period requires
|
||||
`startTime` and `endTime`;
|
||||
- forecast discussions require `issuedAt` and `product`;
|
||||
- weather story runs require `asOf`, and each story requires `startTime`,
|
||||
`endTime`, and `updatedAt`;
|
||||
- alert runs require `asOf`, and each alert requires `id`.
|
||||
|
||||
Nullable canonical values are converted to SQL nulls by mapper helpers.
|
||||
Observation present-weather raw values are stored as compact JSON text.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Payload decode failures, missing required fields, invalid compact JSON values,
|
||||
or schema/write mismatches return errors to feedkit's sink. Feedkit rolls back
|
||||
the transaction when a write fails.
|
||||
|
||||
Unsupported canonical schemas are ignored by this mapper so other routed events
|
||||
can use different sinks without Postgres-specific failures.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/sinks/postgres/schema_test.go`
|
||||
- `internal/sinks/postgres/map_test.go`
|
||||
- feedkit Postgres sink tests when changing generic sink behavior assumptions
|
||||
|
||||
## Invariants
|
||||
|
||||
- Persist only canonical schemas.
|
||||
- Preserve event envelope fields in parent rows.
|
||||
- Preserve array order with child positional indexes.
|
||||
- Validate required fields before writing.
|
||||
- Keep table-contract docs synchronized with schema and mapper changes.
|
||||
110
docs/internal/runtime.md
Normal file
110
docs/internal/runtime.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# Runtime Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
`cmd/weatherfeeder` wires the daemon together. It owns process setup and runtime
|
||||
composition; provider mapping, source fetching details, and sink persistence
|
||||
rules stay in their owning packages.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
The executable reads `config.yml` from the current working directory through
|
||||
feedkit config loading. It builds configured sources, scheduler jobs, sinks, and
|
||||
routes, then runs source polling and sink dispatch until shutdown.
|
||||
|
||||
Inputs are configured source polls. Outputs are feed events delivered to the
|
||||
configured sinks.
|
||||
|
||||
## Runtime Flow
|
||||
|
||||
The implemented flow is:
|
||||
|
||||
1. load `config.yml`;
|
||||
2. register weatherfeeder source drivers;
|
||||
3. register feedkit built-in sinks and the weatherfeeder Postgres sink;
|
||||
4. build source inputs and scheduler jobs;
|
||||
5. validate configured expected kinds against source-advertised kinds;
|
||||
6. build sinks and compile routes;
|
||||
7. run the processor chain `normalize`, then `dedupe`;
|
||||
8. run the scheduler and dispatcher concurrently;
|
||||
9. shut down on signal or fatal scheduler/dispatcher error.
|
||||
|
||||
The in-process event channel is buffered to 256 events. The dedupe processor is
|
||||
bounded by `dedupeMaxEntries`, currently 2048.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Runtime composition belongs in `cmd/weatherfeeder/main.go`.
|
||||
- Source driver behavior belongs under `internal/sources`.
|
||||
- Normalizer behavior belongs under `internal/normalizers`.
|
||||
- Canonical payloads and schema strings belong in `model` and `standards`.
|
||||
- Postgres mapping belongs under `internal/sinks/postgres`.
|
||||
|
||||
`cmd/weatherfeeder` should stay thin and should not contain provider parsing,
|
||||
canonical mapping, or table-mapping rules.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
Runtime wiring consumes the feedkit top-level config sections:
|
||||
|
||||
- `sources`: source driver selection, source name, mode, cadence, expected kinds,
|
||||
and driver params;
|
||||
- `sinks`: sink driver selection, sink name, and sink params;
|
||||
- `routes`: event-kind routing to named sinks.
|
||||
|
||||
The executable does not expose CLI flags or config path discovery.
|
||||
|
||||
## External Adapters Used
|
||||
|
||||
Runtime composition uses feedkit for:
|
||||
|
||||
- config loading;
|
||||
- source registry and expected-kind validation;
|
||||
- scheduler job construction;
|
||||
- processor registry and chain execution;
|
||||
- normalization and dedupe processors;
|
||||
- sink registry and built-in sinks;
|
||||
- route compilation and dispatch.
|
||||
|
||||
Weatherfeeder registers its own source drivers and its Postgres schema mapper.
|
||||
|
||||
## State
|
||||
|
||||
Weatherfeeder-owned runtime state is in process:
|
||||
|
||||
- event channel contents;
|
||||
- the bounded dedupe key set;
|
||||
- source instances and their HTTP conditional validators;
|
||||
- scheduler and dispatcher goroutines.
|
||||
|
||||
There is no weatherfeeder-owned durable scheduler state, checkpoint, replay log,
|
||||
or resume marker. Durable persistence is owned by configured external sinks.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Startup failures are fatal and include context such as config index, source name,
|
||||
sink name, driver name, or the operation that failed.
|
||||
|
||||
At runtime, scheduler and dispatcher errors are sent to a shared error channel.
|
||||
Context cancellation and deadline errors are treated as normal shutdown. Any
|
||||
other scheduler or dispatcher error is logged as fatal and cancels the process
|
||||
context.
|
||||
|
||||
The daemon handles `os.Interrupt` and `SIGTERM` with `signal.NotifyContext`.
|
||||
After both runtime goroutines return, it logs `shutdown complete`.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `cmd/weatherfeeder/main_test.go`
|
||||
- source registry tests under `internal/sources`
|
||||
- normalizer registration tests under `internal/normalizers`
|
||||
- feedkit scheduler, processor, dispatch, and sink tests when changing runtime
|
||||
infrastructure usage
|
||||
|
||||
## Invariants
|
||||
|
||||
- Keep normalization before dedupe.
|
||||
- Keep queue sizes and dedupe bounds explicit.
|
||||
- Preserve context-aware shutdown.
|
||||
- Keep runtime wiring separate from domain mapping and persistence rules.
|
||||
- Keep startup validation failures loud and contextual.
|
||||
105
docs/internal/sources.md
Normal file
105
docs/internal/sources.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# Source Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
Source packages poll upstream weather providers and emit raw feed events. They
|
||||
are adapters, not canonical mappers.
|
||||
|
||||
Sources should decode only the metadata needed for event identity, effective
|
||||
time, and routing policy. Full provider payload interpretation belongs in
|
||||
normalizers.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs are feedkit `config.SourceConfig` values and upstream HTTP responses.
|
||||
Outputs are feed events whose payloads are raw provider JSON and whose schemas
|
||||
come from `standards`.
|
||||
|
||||
Current drivers:
|
||||
|
||||
| Driver | Kind | Raw schema |
|
||||
| --- | --- | --- |
|
||||
| `nws_observation` | `observation` | `raw.nws.observation.v1` |
|
||||
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` |
|
||||
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` |
|
||||
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` |
|
||||
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` |
|
||||
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` |
|
||||
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` |
|
||||
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` |
|
||||
| `openweather_observation` | `observation` | `raw.openweather.current.v1` |
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Source constructors validate source-specific params.
|
||||
- Sources use feedkit HTTP helpers for HTTP polling.
|
||||
- Sources emit raw events and should not build canonical `model` payloads.
|
||||
- Provider helper packages under `internal/providers/<provider>` hold shared
|
||||
parsing and validation helpers.
|
||||
- Registration is centralized in `internal/sources/builtins.go`.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
All current source drivers use feedkit `HTTPSource`.
|
||||
|
||||
Required params:
|
||||
|
||||
- `url`
|
||||
- `user_agent`
|
||||
|
||||
Optional params:
|
||||
|
||||
- `conditional`, default `true`;
|
||||
- `http_timeout`;
|
||||
- `http_response_body_limit_bytes`.
|
||||
|
||||
OpenWeather observation sources additionally require the configured URL to use
|
||||
metric units. This is enforced by `internal/providers/openweather`.
|
||||
|
||||
Source-level `kinds`, when configured, are validated against the source's
|
||||
advertised `Kinds()`.
|
||||
|
||||
## External Adapters Used
|
||||
|
||||
Sources use feedkit's HTTP helper for:
|
||||
|
||||
- request construction;
|
||||
- `User-Agent` and `Accept` headers;
|
||||
- optional conditional GET validators;
|
||||
- response body size limits;
|
||||
- JSON raw-message fetches.
|
||||
|
||||
NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times
|
||||
with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs.
|
||||
|
||||
## State
|
||||
|
||||
HTTP conditional validators are held in each source instance. They are not
|
||||
persisted across process restarts.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Constructor failures are returned during startup and stop the daemon. Polling
|
||||
failures are returned to the scheduler.
|
||||
|
||||
If a source cannot decode minimal metadata from an otherwise successful upstream
|
||||
response, it still emits the raw event when possible. The event then falls back
|
||||
to default ID/effective-time behavior from feedkit source helpers.
|
||||
|
||||
Unchanged conditional responses return no events and no error.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/sources/builtins_test.go`
|
||||
- provider source tests under `internal/sources/nws`
|
||||
- provider source tests under `internal/sources/openmeteo`
|
||||
- provider source tests under `internal/sources/openweather`
|
||||
- provider helper tests under `internal/providers`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Emit raw provider schemas from `standards`.
|
||||
- Keep provider-to-canonical mapping out of sources.
|
||||
- Keep HTTP behavior context-aware.
|
||||
- Keep driver registration explicit and stable.
|
||||
- Keep source tests independent of live upstream services.
|
||||
202
docs/policy/development.md
Normal file
202
docs/policy/development.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# Development Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
This document describes how to change `weatherfeeder` safely. It is for
|
||||
maintainers and coding agents working in the repository.
|
||||
|
||||
Use this alongside the [architecture policy](architecture.md). User-facing CLI,
|
||||
configuration, operations, and wire-contract details belong in their canonical
|
||||
docs, not here.
|
||||
|
||||
## Repository Layout
|
||||
|
||||
- `cmd/weatherfeeder/`: executable wiring, sample `config.yml`, and runtime
|
||||
composition tests.
|
||||
- `model/`: canonical weather payload structs. JSON tags are part of the wire
|
||||
contract.
|
||||
- `standards/`: schema strings, versioning conventions, WMO constants, and
|
||||
shared wire-format policy.
|
||||
- `internal/sources/`: source adapters that poll upstream providers and emit raw
|
||||
feed events.
|
||||
- `internal/normalizers/`: raw-to-canonical event transforms.
|
||||
- `internal/providers/`: pure provider helper code shared by sources and
|
||||
normalizers.
|
||||
- `internal/sinks/postgres/`: weatherfeeder-owned Postgres schema and canonical
|
||||
event mapper.
|
||||
- `docs/`: current behavior, policies, integration contracts, and roadmap files.
|
||||
- `examples/`: maintained, copyable configuration examples.
|
||||
|
||||
## Build And Test
|
||||
|
||||
Run the full test suite before committing behavior or documentation changes that
|
||||
depend on code behavior:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Use narrower commands while iterating:
|
||||
|
||||
```sh
|
||||
go test ./cmd/weatherfeeder
|
||||
go test ./internal/sources/...
|
||||
go test ./internal/normalizers/...
|
||||
go test ./internal/sinks/postgres
|
||||
```
|
||||
|
||||
Format Go code before committing:
|
||||
|
||||
```sh
|
||||
gofmt -w <changed-go-files>
|
||||
```
|
||||
|
||||
Do not require live upstream weather services, NATS, or Postgres for unit tests.
|
||||
Use fixtures, local test servers, and package-level tests.
|
||||
|
||||
## Coding Conventions
|
||||
|
||||
- Keep `cmd/weatherfeeder` focused on composition: config load, registry setup,
|
||||
scheduler jobs, processor chain, dispatch, signal handling, and logging.
|
||||
- Keep source fetching separate from normalizer mapping.
|
||||
- Match normalizers by schema constants from `standards`, not source names.
|
||||
- Keep provider-specific helper code under `internal/providers/<provider>` when
|
||||
both sources and normalizers use it.
|
||||
- Keep cross-provider normalizer helpers pure and deterministic under
|
||||
`internal/normalizers/common`.
|
||||
- Keep sink persistence mapping isolated under `internal/sinks/<sink>`.
|
||||
- Wrap errors with operation context, but do not include whole upstream payloads
|
||||
in errors or logs by default.
|
||||
- Prefer explicit registries and small package-level constructors over hidden
|
||||
global behavior.
|
||||
|
||||
## Dependency Policy
|
||||
|
||||
Prefer the Go standard library unless a dependency materially improves
|
||||
maintainability or interoperability.
|
||||
|
||||
`feedkit` owns generic daemon infrastructure for config, HTTP source helpers,
|
||||
scheduling, processors, dispatch, and sinks. Weatherfeeder code should contain
|
||||
weather-domain behavior and narrow adapter logic rather than duplicating feedkit
|
||||
infrastructure.
|
||||
|
||||
Do not add broad dependencies for small conveniences. Do not let
|
||||
dependency-specific types leak across package boundaries unless that dependency
|
||||
is the package contract.
|
||||
|
||||
## Adding Config Fields
|
||||
|
||||
Generic config shape is owned by feedkit. Weatherfeeder-specific config behavior
|
||||
belongs in source or sink constructors, registry setup, and tests.
|
||||
|
||||
When adding config behavior:
|
||||
|
||||
- validate required params at the adapter boundary;
|
||||
- keep secrets in environment variables or placeholders, not committed values;
|
||||
- update [configuration docs](../config.md);
|
||||
- update maintained examples when the change affects normal operation;
|
||||
- add or update config-load tests for example files when practical.
|
||||
|
||||
## Adding CLI Flags
|
||||
|
||||
The executable currently reads `config.yml` from the current working directory.
|
||||
If CLI flags are added:
|
||||
|
||||
- keep parsing in `cmd/weatherfeeder`;
|
||||
- avoid moving config policy into domain packages;
|
||||
- update [CLI docs](../cli.md);
|
||||
- update tests that exercise command behavior.
|
||||
|
||||
## Adding A Source Driver
|
||||
|
||||
Source drivers should fetch upstream data and emit raw events with minimal
|
||||
metadata decoding.
|
||||
|
||||
Checklist:
|
||||
|
||||
- implement the driver under `internal/sources/<provider>`;
|
||||
- build from `config.SourceConfig`;
|
||||
- validate required params in the constructor;
|
||||
- use feedkit HTTP helpers for HTTP polling when applicable;
|
||||
- emit raw schema constants from `standards`;
|
||||
- advertise emitted kinds through `Kinds()`;
|
||||
- decode only metadata needed for event ID and effective time;
|
||||
- register the driver in `internal/sources/builtins.go`;
|
||||
- add constructor, kind, and polling tests;
|
||||
- update config docs and examples when operators need new configuration;
|
||||
- add provider integration notes when the provider contract needs maintenance
|
||||
context.
|
||||
|
||||
## Adding A Normalizer
|
||||
|
||||
Normalizers own provider-to-canonical mapping.
|
||||
|
||||
Checklist:
|
||||
|
||||
- add one normalizer type per normalizer file;
|
||||
- match using `Event.Schema`;
|
||||
- decode raw payloads into provider structs;
|
||||
- map to canonical `model` payloads;
|
||||
- use `internal/normalizers/common.Finalize` so envelope handling and float
|
||||
rounding stay consistent;
|
||||
- preserve input envelope fields except schema, payload, and effective time;
|
||||
- register through the provider package and `internal/normalizers/builtins.go`;
|
||||
- add tests for schema matching, key payload fields, effective time, malformed
|
||||
required data, and output validation.
|
||||
|
||||
## Adding Canonical Models Or Schemas
|
||||
|
||||
Canonical event changes affect multiple contracts.
|
||||
|
||||
Checklist:
|
||||
|
||||
- update payload structs in `model`;
|
||||
- add or update schema constants in `standards`;
|
||||
- update [event wire contract docs](../integrations/events.md);
|
||||
- update normalizers that produce the schema;
|
||||
- update Postgres mapping if the schema is persisted;
|
||||
- add tests for wire shape and mapper behavior.
|
||||
|
||||
## Adding Postgres Mapping
|
||||
|
||||
Weatherfeeder owns the canonical-event-to-table mapping. Feedkit owns the
|
||||
generic Postgres sink mechanics.
|
||||
|
||||
Checklist:
|
||||
|
||||
- update `internal/sinks/postgres/schema.go`;
|
||||
- update `internal/sinks/postgres/map.go`;
|
||||
- preserve event envelope columns in parent rows when the table supports them;
|
||||
- validate required canonical fields before writing;
|
||||
- use positional indexes for child rows that represent arrays;
|
||||
- update mapper and schema tests;
|
||||
- update [Postgres integration docs](../integrations/postgres.md) when the table
|
||||
contract changes.
|
||||
|
||||
## Examples And Documentation
|
||||
|
||||
Documentation must follow the [documentation policy](documentation.md).
|
||||
|
||||
When behavior changes, update the canonical docs in the same change:
|
||||
|
||||
- config shape: `docs/config.md`;
|
||||
- CLI behavior: `docs/cli.md`;
|
||||
- operations and recovery: `docs/operations.md`;
|
||||
- troubleshooting: `docs/troubleshooting.md`;
|
||||
- external contracts: `docs/integrations/`;
|
||||
- internal component behavior: `docs/internal/`;
|
||||
- copyable configs: `examples/`.
|
||||
|
||||
Keep roadmap content under `docs/roadmap/`. Current-behavior docs must describe
|
||||
implemented behavior only.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
Before committing:
|
||||
|
||||
- run focused tests for changed packages;
|
||||
- run `go test ./...` for broad behavior or documentation changes tied to code;
|
||||
- verify maintained examples still load when examples or config docs changed;
|
||||
- check links in changed docs;
|
||||
- search for stale paths, unsupported features, and secret-like values;
|
||||
- keep unrelated refactors out of the change.
|
||||
@@ -70,7 +70,7 @@
|
||||
//
|
||||
// weather.<kind>.vN
|
||||
//
|
||||
// weatherfeeder centralizes schema strings in internal/standards/schema.go.
|
||||
// weatherfeeder centralizes schema strings in standards/schema.go.
|
||||
// Always use those constants (do not inline schema strings).
|
||||
//
|
||||
// Example mappings:
|
||||
@@ -101,8 +101,8 @@
|
||||
// Every normalizer type must have a doc comment that states:
|
||||
//
|
||||
// - what it converts (e.g., “OpenWeather current -> WeatherObservation”)
|
||||
// - which raw schema it matches (constant identifier from internal/standards)
|
||||
// - which canonical schema it produces (constant identifier from internal/standards)
|
||||
// - which raw schema it matches (constant identifier from standards)
|
||||
// - which canonical schema it produces (constant identifier from standards)
|
||||
// - any special caveats (units, day/night inference, missing fields, etc.)
|
||||
//
|
||||
// Including literal schema string values is optional,
|
||||
|
||||
@@ -37,7 +37,7 @@ func (AlertsNormalizer) Match(e event.Event) bool {
|
||||
}
|
||||
|
||||
func (AlertsNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
||||
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||
|
||||
// If we can't derive AsOf from the payload, fall back to the existing event envelope.
|
||||
fallbackAsOf := in.EmittedAt.UTC()
|
||||
|
||||
@@ -42,7 +42,7 @@ func (ForecastNormalizer) Match(e event.Event) bool {
|
||||
}
|
||||
|
||||
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
||||
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||
|
||||
return normalizeForecastEventBySchema(in)
|
||||
}
|
||||
|
||||
@@ -32,7 +32,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
|
||||
}
|
||||
|
||||
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
||||
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||
|
||||
return normcommon.NormalizeJSON(
|
||||
in,
|
||||
|
||||
@@ -33,7 +33,7 @@ func (ForecastNormalizer) Match(e event.Event) bool {
|
||||
}
|
||||
|
||||
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
||||
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||
|
||||
// If present, prefer the existing event EmittedAt as IssuedAt.
|
||||
var fallbackIssued time.Time
|
||||
|
||||
@@ -40,7 +40,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
|
||||
}
|
||||
|
||||
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
||||
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||
|
||||
return normcommon.NormalizeJSON(
|
||||
in,
|
||||
|
||||
@@ -8,8 +8,7 @@ import (
|
||||
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
|
||||
)
|
||||
|
||||
// This file holds provider-specific helpers that are shared across multiple
|
||||
// OpenWeather normalizers (observations today; forecasts/alerts later).
|
||||
// This file holds provider-specific helpers for OpenWeather normalizers.
|
||||
// Keeping these out of observation.go helps preserve the "one normalizer per file"
|
||||
// convention while avoiding duplication.
|
||||
|
||||
|
||||
@@ -37,7 +37,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
|
||||
}
|
||||
|
||||
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
||||
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||
|
||||
return normcommon.NormalizeJSON(
|
||||
in,
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// Package openweather contains provider-specific helper code for OpenWeather used by
|
||||
// both sources and normalizers.
|
||||
// Package openweather contains provider-specific helper code for OpenWeather
|
||||
// used by sources and normalizers.
|
||||
//
|
||||
// Rules:
|
||||
// - No network I/O here.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
// Package model defines weatherfeeder's canonical domain payload types.
|
||||
//
|
||||
// These structs are emitted as the Payload of canonical events (schemas "weather.*.vN").
|
||||
// JSON tags are treated as part of the wire contract for sinks (stdout today; others later).
|
||||
// JSON tags are treated as part of the wire contract for configured sinks.
|
||||
//
|
||||
// Compatibility guidance:
|
||||
// - Prefer additive changes.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// File: internal/standards/doc.go
|
||||
// File: standards/doc.go
|
||||
//
|
||||
// Package standards defines weatherfeeder’s provider-agnostic “project law”:
|
||||
//
|
||||
|
||||
Reference in New Issue
Block a user