180 lines
6.9 KiB
Go
180 lines
6.9 KiB
Go
// Package normalizers defines weatherfeeder’s **prescriptive** conventions for
|
||
// writing feedkit normalizers and provides the recommended project layout.
|
||
//
|
||
// Summary
|
||
// -------
|
||
// weatherfeeder ingests multiple upstream providers whose payloads differ.
|
||
// Sources should focus on polling/fetching. Normalizers should focus on
|
||
// transforming provider-specific raw payloads into canonical internal models.
|
||
//
|
||
// This package is domain code (weatherfeeder). feedkit’s normalize package is
|
||
// infrastructure (registry + processor).
|
||
//
|
||
// Directory layout (required)
|
||
// ---------------------------
|
||
// Normalizers are organized by provider:
|
||
//
|
||
// internal/normalizers/<provider>/
|
||
//
|
||
// Example:
|
||
//
|
||
// internal/normalizers/nws/observation.go
|
||
// internal/normalizers/nws/types.go
|
||
// internal/normalizers/nws/wmo_map.go
|
||
// internal/normalizers/openweather/observation.go
|
||
// internal/normalizers/openmeteo/observation.go
|
||
// internal/normalizers/common/units.go
|
||
//
|
||
// Rules:
|
||
//
|
||
// 1. One normalizer per file.
|
||
// Each file contains exactly one Normalizer implementation (one type that
|
||
// satisfies feedkit/normalize.Normalizer).
|
||
// Helper files are encouraged (types.go, common.go, mapping.go, etc.) as long
|
||
// as they do not define additional Normalizer types.
|
||
//
|
||
// 2. Provider-level shared helpers live under the provider directory:
|
||
// internal/providers/<provider>/
|
||
//
|
||
// Use this for provider-specific quirks that should be shared by BOTH sources
|
||
// and normalizers (time parsing, URL/unit invariants, ID normalization, etc.).
|
||
// Keep these helpers pure (no I/O) and easy to test.
|
||
//
|
||
// You may use multiple helper files (recommended) when it improves clarity:
|
||
// - types.go (provider JSON structs)
|
||
// - common.go (provider-shared helpers)
|
||
// - mapping.go (provider mapping logic)
|
||
//
|
||
// 3. Cross-provider helpers live in:
|
||
// internal/normalizers/common/
|
||
//
|
||
// Prefer extracting small, pure helpers here when they are reused by ≥2 providers.
|
||
// Keep these helpers:
|
||
// - deterministic (no I/O)
|
||
// - side-effect free
|
||
// - easy to read (avoid clever abstractions)
|
||
//
|
||
// 4. Matching is standardized on Event.Schema.
|
||
// (Do not match on event.Source or event.Kind in weatherfeeder normalizers.)
|
||
//
|
||
// Schema conventions (required)
|
||
// -----------------------------
|
||
// Sources emit RAW events with provider-specific schemas.
|
||
// Normalizers convert RAW -> CANONICAL schemas.
|
||
//
|
||
// Raw schemas:
|
||
//
|
||
// raw.<provider>.<thing>.vN
|
||
//
|
||
// Canonical schemas:
|
||
//
|
||
// weather.<kind>.vN
|
||
//
|
||
// weatherfeeder centralizes schema strings in internal/standards/schema.go.
|
||
// Always use those constants (do not inline schema strings).
|
||
//
|
||
// Example mappings:
|
||
//
|
||
// standards.SchemaRawOpenWeatherCurrentV1 -> standards.SchemaWeatherObservationV1
|
||
// standards.SchemaRawOpenMeteoCurrentV1 -> standards.SchemaWeatherObservationV1
|
||
// standards.SchemaRawNWSObservationV1 -> standards.SchemaWeatherObservationV1
|
||
//
|
||
// Normalizer structure (required template)
|
||
// ----------------------------------------
|
||
// Each normalizer file must follow this structure (with helpful comments):
|
||
//
|
||
// type OpenWeatherObservationNormalizer struct{}
|
||
//
|
||
// func (OpenWeatherObservationNormalizer) Match(e event.Event) bool { ... }
|
||
//
|
||
// func (OpenWeatherObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||
// // 1) Decode raw payload (recommended: json.RawMessage)
|
||
// // 2) Parse into provider structs
|
||
// // 3) Map provider -> canonical internal/model types
|
||
// // 4) Build output event (copy input, modify intentionally)
|
||
// // 5) Set EffectiveAt if applicable
|
||
// // 6) Validate out.Validate()
|
||
// }
|
||
//
|
||
// Required doc comment content
|
||
// ----------------------------
|
||
// Every normalizer type must have a doc comment that states:
|
||
//
|
||
// - what it converts (e.g., “OpenWeather current -> WeatherObservation”)
|
||
// - which raw schema it matches (constant identifier from internal/standards)
|
||
// - which canonical schema it produces (constant identifier from internal/standards)
|
||
// - any special caveats (units, day/night inference, missing fields, etc.)
|
||
//
|
||
// Including literal schema string values is optional,
|
||
// but the constant identifiers are required.
|
||
//
|
||
// Event field handling (strong defaults)
|
||
// --------------------------------------
|
||
// Normalizers should treat the incoming event envelope as stable identity and
|
||
// should only change fields intentionally.
|
||
//
|
||
// Default behavior:
|
||
//
|
||
// - Keep: ID, Kind, Source, EmittedAt
|
||
// - Set: Schema to the canonical schema
|
||
// - Set: Payload to the canonical payload (internal/model/*)
|
||
// - Optional: EffectiveAt (often derived from observation timestamp)
|
||
// - Avoid changing Kind unless you have a clear “raw kind vs canonical kind” design.
|
||
//
|
||
// Always validate the output event:
|
||
//
|
||
// if err := out.Validate(); err != nil { ... }
|
||
//
|
||
// Payload representation for RAW events
|
||
// -------------------------------------
|
||
// weatherfeeder recommends RAW payloads be stored as json.RawMessage for JSON APIs.
|
||
// This keeps sources small and defers schema-specific decoding to normalizers.
|
||
//
|
||
// If a source already decodes into typed provider structs, it can still emit the
|
||
// raw event; it should simply re-marshal to json.RawMessage (or better: decode
|
||
// once in the normalizer instead to keep “fetch” separate from “transform”).
|
||
//
|
||
// Registration pattern
|
||
// --------------------
|
||
// feedkit normalization uses a match-driven registry (“first match wins”).
|
||
//
|
||
// Provider subpackages should expose:
|
||
//
|
||
// func Register(reg *normalize.Registry)
|
||
//
|
||
// And internal/normalizers/builtins.go should provide one entrypoint:
|
||
//
|
||
// func RegisterBuiltins(reg *normalize.Registry)
|
||
//
|
||
// which calls each provider’s Register() in a stable order.
|
||
//
|
||
// Registry ordering
|
||
// -----------------------------
|
||
// feedkit normalization uses a match-driven registry (“first match wins”).
|
||
// Therefore order matters:
|
||
//
|
||
// - Register more specific normalizers before more general ones.
|
||
// - Avoid “catch-all” Match() implementations.
|
||
// - Keep Match() cheap and deterministic (Schema equality checks are ideal).
|
||
//
|
||
// Reuse guidance (strong recommendation)
|
||
// --------------------------------------
|
||
// Before adding provider-specific logic, check internal/normalizers/common for an
|
||
// existing helper (payload extraction, unit conversions, text fallbacks, etc.).
|
||
// If you discover logic that could potentially apply to another provider, prefer extracting
|
||
// it into internal/normalizers/common as appropriate.
|
||
//
|
||
// Testing guidance (recommended)
|
||
// ------------------------------
|
||
// Add a unit test per normalizer:
|
||
//
|
||
// internal/normalizers/openweather/observation_test.go
|
||
//
|
||
// Tests should:
|
||
//
|
||
// - build a RAW event with Schema=standards.SchemaRaw... and Payload=json.RawMessage
|
||
// - run the normalizer
|
||
// - assert canonical Schema + key payload fields + EffectiveAt
|
||
// - assert out.Validate() passes
|
||
package normalizers
|