weatherfeeder/docs/policy/architecture.md

Architecture Policy

Purpose

This document defines weatherfeeder's development architecture and invariants for maintainers and LLM coding agents. It describes how the implemented system is built and how subsequent changes should preserve its boundaries.

This is an inward-facing policy document. User-facing wire contracts belong in docs/integrations/events.md, and roadmap items belong under docs/roadmap/.

Project Shape

weatherfeeder is a config-driven Go daemon that polls upstream weather providers, emits feed events, normalizes provider-specific raw payloads into canonical weather payloads, and routes those events to configured sinks.

The implemented runtime flow is:

1. cmd/weatherfeeder loads config.yml from the working directory.
2. Source drivers are built through the source registry.
3. Feedkit scheduler jobs poll sources and publish raw events onto an in-process event channel.
4. A pipeline runs normalization, then in-memory dedupe.
5. The dispatcher routes processed events to configured sinks by event kind.
6. Sinks consume events independently through feedkit fanout workers.

Canonical payload structs live in model. Schema identifiers and cross-provider wire conventions live in standards. Source adapters live under internal/sources. Normalizers live under internal/normalizers. Provider-specific parsing helpers shared by sources and normalizers live under internal/providers. Sink-specific persistence mapping lives under internal/sinks.

Architecture Style

weatherfeeder uses a pragmatic ports-and-adapters architecture rather than a formal framework. Provider APIs, config loading, scheduling, dispatch, and sinks sit outside the weather domain model and normalization rules.

The implementation style is:

• Pipeline-oriented: events flow from source polling through normalization, dedupe, routing, and sink fanout.
• Schema-routed: normalizers select raw payloads by explicit schema strings, not source names or configured routes.
• Provider-isolated: NWS, Open-Meteo, OpenWeather, and SPC quirks stay in provider-specific source, provider-helper, and normalizer packages.
• Registry-based: built-in source drivers, normalizers, processors, and sinks are assembled explicitly through registries instead of dynamic plugin loading.
• Adapter-clean: persistence and external-system details stay behind source and sink adapters, not in model or normalizers.
• Direct Go: prefer small package-level constructors and straightforward code over broad abstractions.

Core Design Principles

• Hexagonal boundaries: provider APIs, config loading, scheduling, dispatch, and sinks are external mechanisms around the weather domain model and normalization logic.
• Raw-to-canonical flow: sources should fetch and envelope raw provider payloads; normalizers should own provider-to-canonical mapping.
• Schema-based routing: normalizers match on event schema, not source name or event kind.
• Composable registries: source drivers, normalizers, processors, and sinks are assembled explicitly through registries.
• Bounded concurrency: scheduling and sink fanout are concurrent, but the application should keep queues, goroutine ownership, logging, and cancellation behavior visible.
• Standard-library-first: use the Go standard library unless a narrow dependency materially improves maintainability or interoperability.
• Current-behavior docs: outside roadmap files, document only implemented behavior.

Architectural Boundaries

Core/domain logic:

• model defines canonical payload structs and JSON field names.
• standards defines schema strings, versioning conventions, WMO constants, and shared wire policy.
• Normalizer mapping code is domain logic and should stay independent of CLI setup, filesystem paths, sink details, and runtime orchestration.

Adapters:

• Source adapters under internal/sources/<provider> fetch upstream data and produce raw feed events.
• Sink adapters under internal/sinks/<sink> map canonical events to external systems.
• Provider helper packages under internal/providers/<provider> contain pure parsing or provider-specific helper logic shared by sources and normalizers.

Runtime composition:

• cmd/weatherfeeder/main.go owns process wiring: config load, registry setup, scheduler jobs, processor chain, dispatcher, signal cancellation, and logging.
• It should remain thin. Do not move provider mapping or sink persistence rules into cmd/weatherfeeder.

Tests and examples:

• The sample cmd/weatherfeeder/config.yml is executable test input and is load-tested.
• Tests should keep exercising package contracts directly rather than relying only on full-daemon execution.

Feedkit Boundary

feedkit provides reusable daemon infrastructure. weatherfeeder provides the weather-domain adapters, models, schemas, and mapping policy.

Area	Feedkit owns	Weatherfeeder owns
Config	Generic YAML shape: sources, sinks, routes, modes, cadence, and params.	Driver-specific config rules such as NWS user_agent, OpenWeather units=metric, and SPC coordinates.
Events	Domain-agnostic event envelope: ID, kind, source, emitted/effective times, schema, and payload.	Event kind meaning, schema strings, and canonical weather payloads.
Sources	Source interfaces, registry, expected-kind validation, HTTP helper, and default event ID helper.	NWS/Open-Meteo/OpenWeather/SPC source drivers and raw schema emission.
Processing	Processor registry, normalize processor, dedupe processor, and pipeline execution.	Weather normalizers and schema-specific raw-to-canonical mapping.
Dispatch	Route compilation and sink fanout mechanics.	Which weather event kinds are configured and meaningful.
Sinks	Generic stdout, NATS, and Postgres sink mechanics.	Weather-specific Postgres schema and canonical event-to-row mapping.

Do not move weather-domain policy into feedkit, and do not duplicate generic daemon mechanics in weatherfeeder when feedkit already provides the boundary.

Modules Or Processing Steps

The implemented processing steps are source polling, normalization, dedupe, and sink dispatch.

Source contract:

• Build from feedkit/config.SourceConfig.
• Validate required driver params inside the source constructor.
• Advertise emitted event kinds through Kinds() when possible.
• Emit raw schemas from standards.
• Decode only minimal metadata needed for event identity and effective time; leave full provider decoding to normalizers.
• Respect context.Context during network work.

Normalizer contract:

• One normalizer type per normalizer file.
• Match by Event.Schema.
• Decode raw payloads into provider structs.
• Map to canonical model payloads.
• Preserve the incoming event envelope except for intentional schema, payload, and effective-time changes.
• Use shared helpers in internal/normalizers/common for cross-provider behavior.
• Follow the detailed normalizer conventions in internal/normalizers/doc.go.

Sink contract:

• Consume canonical schemas, not provider raw schemas.
• Keep sink mapping isolated from normalizers and sources.
• Preserve event envelope fields in durable storage where the sink schema supports it.
• Validate required canonical fields before writing.

State, Inputs, and Outputs

Inputs are configured source polls. The daemon currently uses feedkit's YAML config model with sources, sinks, and routes.

Outputs are feed events sent to configured sinks. Implemented sink support comes from feedkit built-ins plus weatherfeeder's Postgres schema mapping. The sample config includes stdout and NATS routes; Postgres is configured as an optional commented example.

The daemon's own state is in-process:

• the event channel buffers events during runtime;
• the dedupe processor stores a bounded in-memory key set;
• source instances may keep HTTP conditional request state through feedkit HTTP source helpers;
• scheduler and dispatcher state is not persisted by weatherfeeder.

Durable persistence is an external sink concern. The Postgres table contract is documented in docs/integrations/postgres.md; the consumer-facing event contract is documented in docs/integrations/events.md.

Configuration and CLI Boundaries

The implemented executable reads config.yml from the current working directory. It does not currently expose CLI flags or config path discovery.

Configuration shape is owned by feedkit's config package:

• sources define named source drivers, mode, poll cadence, expected kinds, and driver params.
• sinks define named sink drivers and sink params.
• routes connect event kinds to sinks.

Weatherfeeder-specific config policy belongs in source and sink constructors, registry setup, and tests. Do not spread config parsing through domain model or normalizer packages.

docs/config.md and docs/cli.md are the canonical user/operator references. This policy should stay architectural and avoid duplicating those references.

Errors, Logging, and Diagnostics

cmd/weatherfeeder uses the standard library log package with timestamps and microseconds.

Startup errors are fatal and include config index, source/sink name, driver name, or operation context where available. Runtime scheduler and dispatcher errors are reported through logs; context cancellation and deadline errors are treated as shutdown conditions.

Normalizers and sink mappers should wrap errors with operation and payload context, for example decode, parse, map, scan, or required-field context. Avoid logging or returning whole upstream payloads by default.

The daemon handles os.Interrupt and SIGTERM through signal.NotifyContext. Sources, sinks, scheduler jobs, dispatcher, and processors should respect context.Context.

Testing Expectations

When changing behavior, inspect or add focused tests in the owning package.

Expected coverage by change type:

• Source drivers: constructor behavior, advertised kinds, poll event schema/kind, effective-time policy, unchanged responses, and malformed metadata handling.
• Normalizers: schema matching, canonical schema output, key field mapping, effective time, malformed required fields, and wire-shape regressions.
• Provider helpers: parsing edge cases and fixtures.
• Runtime wiring: config loading, source registry build, scheduler job creation, processor ordering, pass-through behavior, and dedupe behavior.
• Postgres sink: schema shape, mapper writes, required-field validation, nullable handling, and compact JSON behavior.
• Documentation-sensitive examples: keep sample config loadable.

Use local test servers and fixtures rather than real upstream services. Full-package tests should remain fast and deterministic.

Dependency Policy

Prefer the Go standard library for parsing, HTTP handling, time handling, logging, and tests where reasonable.

Existing broad runtime composition is delegated to feedkit, which provides config, sources, scheduler, processors, dispatch, and sinks. Keep weatherfeeder-specific code from depending directly on low-level external clients when feedkit or a small adapter can contain that dependency.

Third-party dependencies should be narrow, justified, and preferably de facto standard for their purpose. YAML parsing through feedkit is an acceptable example. Do not add dependencies for small conveniences, and do not let dependency-specific types leak across package boundaries unless that dependency is the package's explicit contract.

Documentation Expectations

Documentation must follow docs/policy/documentation.md.

Rules for architecture-related docs:

• Current-behavior docs must describe implemented behavior only.
• Roadmap or speculative work belongs only under docs/roadmap/.
• Prefer links to canonical docs over repeated reference material.
• Update docs in the same change when modifying schemas, config behavior, runtime behavior, adapters, or persistence contracts.

Architectural Invariants

• Keep cmd/weatherfeeder as composition code, not domain logic.
• Keep source fetching separate from normalization.
• Keep normalizers matched by schema constants from standards.
• Keep canonical payload structs in model and treat JSON tags as wire contract.
• Keep provider-specific helpers under internal/providers/<provider> when shared by sources and normalizers.
• Keep cross-provider normalizer helpers pure and deterministic.
• Keep sink persistence mapping isolated under internal/sinks/<sink>.
• Preserve explicit registry-based extension points for sources and normalizers.
• Preserve context-aware shutdown and bounded in-process queues.
• Avoid broad dependencies without clear architectural value.

Non-Goals

• weatherfeeder is not an HTTP API. API serving belongs to separate consumers such as weatherapi.
• weatherfeeder does not own long-term durable state except through configured external sinks.
• weatherfeeder does not provide a general plugin runtime; new built-in providers and sinks are registered in code.
• Architecture policy is not a CLI, config, or wire-contract reference.