// Package normalizers defines weatherfeeder’s **prescriptive** conventions for // writing feedkit normalizers and provides the recommended project layout. // // Summary // ------- // weatherfeeder ingests multiple upstream providers whose payloads differ. // Sources should focus on polling/fetching. Normalizers should focus on // transforming provider-specific raw payloads into canonical internal models. // // This package is domain code (weatherfeeder). feedkit’s normalize package is // infrastructure (registry + processor). // // Directory layout (required) // --------------------------- // Normalizers are organized by provider: // // internal/normalizers// // // Example: // // internal/normalizers/nws/observation.go // internal/normalizers/nws/types.go // internal/normalizers/nws/wmo_map.go // internal/normalizers/openweather/observation.go // internal/normalizers/openmeteo/observation.go // internal/normalizers/common/units.go // // Rules: // // 1. One normalizer per file. // Each file contains exactly one Normalizer implementation (one type that // satisfies feedkit/normalize.Normalizer). // Helper files are encouraged (types.go, common.go, mapping.go, etc.) as long // as they do not define additional Normalizer types. // // 2. Provider-level shared helpers live under the provider directory: // internal/normalizers// // // You may use multiple helper files (recommended) when it improves clarity: // - types.go (provider JSON structs) // - common.go (provider-shared helpers) // - mapping.go (provider mapping logic) // Use common.go only when you truly have “shared across multiple normalizers // within this provider” helpers. // // 3. Cross-provider helpers live in: // internal/normalizers/common/ // // Prefer extracting small, pure helpers here when they are reused by ≥2 providers. // Keep these helpers: // - deterministic (no I/O) // - side-effect free // - easy to read (avoid clever abstractions) // // 4. Matching is standardized on Event.Schema. // (Do not match on event.Source or event.Kind in weatherfeeder normalizers.) // // Schema conventions (required) // ----------------------------- // Sources emit RAW events with provider-specific schemas. // Normalizers convert RAW -> CANONICAL schemas. // // Raw schemas: // // raw...vN // // Canonical schemas: // // weather..vN // // weatherfeeder centralizes schema strings in internal/standards/schemas.go. // Always use those constants (do not inline schema strings). // // Example mappings: // // standards.SchemaRawOpenWeatherCurrentV1 -> standards.SchemaWeatherObservationV1 // standards.SchemaRawOpenMeteoCurrentV1 -> standards.SchemaWeatherObservationV1 // standards.SchemaRawNWSObservationV1 -> standards.SchemaWeatherObservationV1 // // Normalizer structure (required template) // ---------------------------------------- // Each normalizer file must follow this structure (with helpful comments): // // type OpenWeatherObservationNormalizer struct{} // // func (OpenWeatherObservationNormalizer) Match(e event.Event) bool { ... } // // func (OpenWeatherObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) { // // 1) Decode raw payload (recommended: json.RawMessage) // // 2) Parse into provider structs // // 3) Map provider -> canonical internal/model types // // 4) Build output event (copy input, modify intentionally) // // 5) Set EffectiveAt if applicable // // 6) Validate out.Validate() // } // // Required doc comment content // ---------------------------- // Every normalizer type must have a doc comment that states: // // - what it converts (e.g., “OpenWeather current -> WeatherObservation”) // - which raw schema it matches (constant identifier from internal/standards) // - which canonical schema it produces (constant identifier from internal/standards) // - any special caveats (units, day/night inference, missing fields, etc.) // // Including literal schema string values is optional, // but the constant identifiers are required. // // Event field handling (strong defaults) // -------------------------------------- // Normalizers should treat the incoming event envelope as stable identity and // should only change fields intentionally. // // Default behavior: // // - Keep: ID, Kind, Source, EmittedAt // - Set: Schema to the canonical schema // - Set: Payload to the canonical payload (internal/model/*) // - Optional: EffectiveAt (often derived from observation timestamp) // - Avoid changing Kind unless you have a clear “raw kind vs canonical kind” design. // // Always validate the output event: // // if err := out.Validate(); err != nil { ... } // // Payload representation for RAW events // ------------------------------------- // weatherfeeder recommends RAW payloads be stored as json.RawMessage for JSON APIs. // This keeps sources small and defers schema-specific decoding to normalizers. // // If a source already decodes into typed provider structs, it can still emit the // raw event; it should simply re-marshal to json.RawMessage (or better: decode // once in the normalizer instead to keep “fetch” separate from “transform”). // // Registration pattern // -------------------- // feedkit normalization uses a match-driven registry (“first match wins”). // // Provider subpackages should expose: // // func Register(reg *normalize.Registry) // // And internal/normalizers/builtins.go should provide one entrypoint: // // func RegisterBuiltins(reg *normalize.Registry) // // which calls each provider’s Register() in a stable order. // // Registry ordering // ----------------------------- // feedkit normalization uses a match-driven registry (“first match wins”). // Therefore order matters: // // - Register more specific normalizers before more general ones. // - Avoid “catch-all” Match() implementations. // - Keep Match() cheap and deterministic (Schema equality checks are ideal). // // Reuse guidance (strong recommendation) // -------------------------------------- // Before adding provider-specific logic, check internal/normalizers/common for an // existing helper (payload extraction, unit conversions, text fallbacks, etc.). // If you discover logic that could potentially apply to another provider, prefer extracting // it into internal/normalizers/common as appropriate. // // Testing guidance (recommended) // ------------------------------ // Add a unit test per normalizer: // // internal/normalizers/openweather/observation_test.go // // Tests should: // // - build a RAW event with Schema=standards.SchemaRaw... and Payload=json.RawMessage // - run the normalizer // - assert canonical Schema + key payload fields + EffectiveAt // - assert out.Validate() passes package normalizers