weatherfeeder/docs/internal/runtime.md

Runtime Internals

Purpose

cmd/weatherfeeder wires the daemon together. It owns process setup and runtime composition; provider mapping, source fetching details, and sink persistence rules stay in their owning packages.

Inputs And Outputs

The executable reads config.yml from the current working directory through feedkit config loading. It builds configured sources, scheduler jobs, sinks, and routes, then runs source polling and sink dispatch until shutdown.

Inputs are configured source polls. Outputs are feed events delivered to the configured sinks.

Runtime Flow

The implemented flow is:

1. load config.yml;
2. register weatherfeeder source drivers;
3. register feedkit built-in sinks and the weatherfeeder Postgres sink;
4. build source inputs and scheduler jobs;
5. validate configured expected kinds against source-advertised kinds;
6. build sinks and compile routes;
7. run the processor chain normalize, then dedupe;
8. run the scheduler and dispatcher concurrently;
9. shut down on signal or fatal scheduler/dispatcher error.

The in-process event channel is buffered to 256 events. The dedupe processor is bounded by dedupeMaxEntries, currently 2048.

Boundaries

• Runtime composition belongs in cmd/weatherfeeder/main.go.
• Source driver behavior belongs under internal/sources.
• Normalizer behavior belongs under internal/normalizers.
• Canonical payloads and schema strings belong in model and standards.
• Postgres mapping belongs under internal/sinks/postgres.

cmd/weatherfeeder should stay thin and should not contain provider parsing, canonical mapping, or table-mapping rules.

Config Fields Used

Runtime wiring consumes the feedkit top-level config sections:

• sources: source driver selection, source name, mode, cadence, expected kinds, and driver params;
• sinks: sink driver selection, sink name, and sink params;
• routes: event-kind routing to named sinks.

The executable does not expose CLI flags or config path discovery.

External Adapters Used

Runtime composition uses feedkit for:

• config loading;
• source registry and expected-kind validation;
• scheduler job construction;
• processor registry and chain execution;
• normalization and dedupe processors;
• sink registry and built-in sinks;
• route compilation and dispatch.

Weatherfeeder registers its own source drivers and its Postgres schema mapper.

Responsibility split:

Runtime concern	Owner
Config loading and generic validation	feedkit
Source, processor, and sink registries	feedkit mechanics; weatherfeeder registrations
Source polling and stream supervision	feedkit scheduler
Raw weather data fetching	weatherfeeder source adapters
Normalizer execution order and pass-through behavior	feedkit normalize processor
Weather raw-to-canonical mapping	weatherfeeder normalizers
Dedupe mechanics	feedkit dedupe processor
Route compilation and sink fanout	feedkit dispatch
Weather Postgres table shape and row mapping	weatherfeeder Postgres adapter
Postgres connection, DDL, inserts, transactions, and pruning	feedkit Postgres sink

State

Weatherfeeder-owned runtime state is in process:

• event channel contents;
• the bounded dedupe key set;
• source instances and their in-memory unchanged-content state;
• scheduler and dispatcher goroutines.

There is no weatherfeeder-owned durable scheduler state, checkpoint, replay log, or resume marker. Durable persistence is owned by configured external sinks.

Failure Behavior

Startup failures are fatal and include context such as config index, source name, sink name, driver name, or the operation that failed.

At runtime, scheduler and dispatcher errors are sent to a shared error channel. Context cancellation and deadline errors are treated as normal shutdown. Any other scheduler or dispatcher error is logged as fatal and cancels the process context.

The daemon handles os.Interrupt and SIGTERM with signal.NotifyContext. After both runtime goroutines return, it logs shutdown complete.

Tests To Inspect

• cmd/weatherfeeder/main_test.go
• source registry tests under internal/sources
• normalizer registration tests under internal/normalizers
• feedkit scheduler, processor, dispatch, and sink tests when changing runtime infrastructure usage

Invariants

• Keep normalization before dedupe.
• Keep queue sizes and dedupe bounds explicit.
• Preserve context-aware shutdown.
• Keep runtime wiring separate from domain mapping and persistence rules.
• Keep startup validation failures loud and contextual.