Document development workflow and internals
This commit is contained in:
90
docs/internal/normalizers.md
Normal file
90
docs/internal/normalizers.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# Normalizer Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
Normalizers convert raw provider events into canonical weather events. They are
|
||||
weather-domain mapping code and should stay independent of runtime wiring,
|
||||
source polling, and sink persistence.
|
||||
|
||||
Detailed package conventions live in `internal/normalizers/doc.go`.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs are raw feed events whose schemas identify provider payload shape.
|
||||
Outputs are canonical feed events using `model` payloads and `weather.*`
|
||||
schemas.
|
||||
|
||||
Current mappings:
|
||||
|
||||
| Raw schema | Canonical schema |
|
||||
| --- | --- |
|
||||
| `raw.nws.observation.v1` | `weather.observation.v1` |
|
||||
| `raw.openmeteo.current.v1` | `weather.observation.v1` |
|
||||
| `raw.openweather.current.v1` | `weather.observation.v1` |
|
||||
| `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||
| `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` |
|
||||
| `raw.openmeteo.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||
| `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` |
|
||||
| `raw.nws.weatherstories.v1` | `weather.weather_story.v1` |
|
||||
| `raw.nws.alerts.v1` | `weather.alert.v1` |
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Normalizers match by `Event.Schema`.
|
||||
- Normalizers decode raw payloads into provider structs.
|
||||
- Normalizers map provider data into canonical `model` payloads.
|
||||
- Normalizers do not fetch network data, read config, route events, or write
|
||||
sinks.
|
||||
- Shared cross-provider behavior belongs in `internal/normalizers/common`.
|
||||
- Provider-specific helper logic shared with sources belongs in
|
||||
`internal/providers/<provider>`.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
Normalizers do not read config. They operate only on incoming events.
|
||||
|
||||
## External Adapters Used
|
||||
|
||||
Runtime composition creates feedkit's normalize processor with
|
||||
`RequireMatch=false`. Events without a matching normalizer pass through
|
||||
unchanged.
|
||||
|
||||
Weatherfeeder registers normalizers in a stable order:
|
||||
|
||||
1. NWS
|
||||
2. Open-Meteo
|
||||
3. OpenWeather
|
||||
|
||||
The current normalizers avoid ambiguous matches by using schema equality.
|
||||
|
||||
## State
|
||||
|
||||
Normalizers should be stateless. Shared helpers should be deterministic and free
|
||||
of I/O.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Malformed required raw payload data should produce contextual errors from the
|
||||
owning normalizer. Successful normalization validates the output event before it
|
||||
continues through the pipeline.
|
||||
|
||||
`internal/normalizers/common.Finalize` preserves the input event envelope except
|
||||
for schema, payload, and effective time. It also rounds canonical float values
|
||||
to four digits after the decimal point.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/normalizers/builtins_test.go`
|
||||
- provider normalizer tests under `internal/normalizers/nws`
|
||||
- provider normalizer tests under `internal/normalizers/openmeteo`
|
||||
- provider normalizer tests under `internal/normalizers/openweather`
|
||||
- common helper tests under `internal/normalizers/common`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Match by schema constants from `standards`.
|
||||
- Preserve the event envelope except for intentional canonical changes.
|
||||
- Produce canonical payload structs from `model`.
|
||||
- Validate normalized events before returning them.
|
||||
- Keep normalizers independent of sources, sinks, config loading, and runtime
|
||||
composition.
|
||||
118
docs/internal/postgres-sink.md
Normal file
118
docs/internal/postgres-sink.md
Normal file
@@ -0,0 +1,118 @@
|
||||
# Postgres Sink Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/sinks/postgres` defines weatherfeeder's canonical-event-to-Postgres
|
||||
mapping. It supplies a schema definition and mapper to feedkit's generic
|
||||
Postgres sink.
|
||||
|
||||
The consumer-facing table contract is
|
||||
[`docs/integrations/postgres.md`](../integrations/postgres.md). This document
|
||||
describes the internal ownership boundary.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs are canonical feed events. The mapper currently handles these schemas:
|
||||
|
||||
- `weather.observation.v1`
|
||||
- `weather.forecast.v1`
|
||||
- `weather.forecast_discussion.v1`
|
||||
- `weather.weather_story.v1`
|
||||
- `weather.alert.v1`
|
||||
|
||||
Outputs are feedkit `PostgresWrite` values for weatherfeeder-owned tables.
|
||||
Unsupported schemas produce no writes and no error.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Weatherfeeder owns table definitions in `schema.go`.
|
||||
- Weatherfeeder owns canonical payload mapping in `map.go`.
|
||||
- Feedkit owns database opening, table and index creation, transactions,
|
||||
inserts, context-aware consumption, and prune execution.
|
||||
- Postgres mapping consumes canonical events only. It should not understand raw
|
||||
provider schemas.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
Weatherfeeder registers the `postgres` sink by passing `PostgresSchema()` to
|
||||
feedkit. Feedkit parses sink params:
|
||||
|
||||
- `uri`
|
||||
- `username`
|
||||
- `password`
|
||||
- `prune`, optional duration
|
||||
|
||||
Weatherfeeder-owned mapper code does not read config directly.
|
||||
|
||||
## External Adapters Used
|
||||
|
||||
The runtime registers the sink with:
|
||||
|
||||
```go
|
||||
sinkReg.Register("postgres", fksinks.PostgresFactory(wfpgsink.PostgresSchema()))
|
||||
```
|
||||
|
||||
Feedkit validates events at the sink boundary, calls the weatherfeeder mapper,
|
||||
validates writes against the compiled schema, inserts rows in a transaction, and
|
||||
optionally prunes rows older than the configured window.
|
||||
|
||||
## State
|
||||
|
||||
The mapper is stateless. Durable state is stored in Postgres through feedkit's
|
||||
sink implementation.
|
||||
|
||||
## Mapping Rules
|
||||
|
||||
Parent rows preserve event envelope fields where the table supports them:
|
||||
|
||||
- `event_id`
|
||||
- `event_kind`
|
||||
- `event_source`
|
||||
- `event_schema`
|
||||
- `event_emitted_at`
|
||||
- `event_effective_at`
|
||||
|
||||
Child rows use positional indexes to preserve canonical array order:
|
||||
|
||||
- `weather_index`
|
||||
- `period_index`
|
||||
- `message_index`
|
||||
- `story_index`
|
||||
- `alert_index`
|
||||
- `reference_index`
|
||||
|
||||
Required canonical fields are validated before writes are returned:
|
||||
|
||||
- observations require `timestamp`;
|
||||
- forecasts require `issuedAt` and `product`, and each period requires
|
||||
`startTime` and `endTime`;
|
||||
- forecast discussions require `issuedAt` and `product`;
|
||||
- weather story runs require `asOf`, and each story requires `startTime`,
|
||||
`endTime`, and `updatedAt`;
|
||||
- alert runs require `asOf`, and each alert requires `id`.
|
||||
|
||||
Nullable canonical values are converted to SQL nulls by mapper helpers.
|
||||
Observation present-weather raw values are stored as compact JSON text.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Payload decode failures, missing required fields, invalid compact JSON values,
|
||||
or schema/write mismatches return errors to feedkit's sink. Feedkit rolls back
|
||||
the transaction when a write fails.
|
||||
|
||||
Unsupported canonical schemas are ignored by this mapper so other routed events
|
||||
can use different sinks without Postgres-specific failures.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/sinks/postgres/schema_test.go`
|
||||
- `internal/sinks/postgres/map_test.go`
|
||||
- feedkit Postgres sink tests when changing generic sink behavior assumptions
|
||||
|
||||
## Invariants
|
||||
|
||||
- Persist only canonical schemas.
|
||||
- Preserve event envelope fields in parent rows.
|
||||
- Preserve array order with child positional indexes.
|
||||
- Validate required fields before writing.
|
||||
- Keep table-contract docs synchronized with schema and mapper changes.
|
||||
110
docs/internal/runtime.md
Normal file
110
docs/internal/runtime.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# Runtime Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
`cmd/weatherfeeder` wires the daemon together. It owns process setup and runtime
|
||||
composition; provider mapping, source fetching details, and sink persistence
|
||||
rules stay in their owning packages.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
The executable reads `config.yml` from the current working directory through
|
||||
feedkit config loading. It builds configured sources, scheduler jobs, sinks, and
|
||||
routes, then runs source polling and sink dispatch until shutdown.
|
||||
|
||||
Inputs are configured source polls. Outputs are feed events delivered to the
|
||||
configured sinks.
|
||||
|
||||
## Runtime Flow
|
||||
|
||||
The implemented flow is:
|
||||
|
||||
1. load `config.yml`;
|
||||
2. register weatherfeeder source drivers;
|
||||
3. register feedkit built-in sinks and the weatherfeeder Postgres sink;
|
||||
4. build source inputs and scheduler jobs;
|
||||
5. validate configured expected kinds against source-advertised kinds;
|
||||
6. build sinks and compile routes;
|
||||
7. run the processor chain `normalize`, then `dedupe`;
|
||||
8. run the scheduler and dispatcher concurrently;
|
||||
9. shut down on signal or fatal scheduler/dispatcher error.
|
||||
|
||||
The in-process event channel is buffered to 256 events. The dedupe processor is
|
||||
bounded by `dedupeMaxEntries`, currently 2048.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Runtime composition belongs in `cmd/weatherfeeder/main.go`.
|
||||
- Source driver behavior belongs under `internal/sources`.
|
||||
- Normalizer behavior belongs under `internal/normalizers`.
|
||||
- Canonical payloads and schema strings belong in `model` and `standards`.
|
||||
- Postgres mapping belongs under `internal/sinks/postgres`.
|
||||
|
||||
`cmd/weatherfeeder` should stay thin and should not contain provider parsing,
|
||||
canonical mapping, or table-mapping rules.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
Runtime wiring consumes the feedkit top-level config sections:
|
||||
|
||||
- `sources`: source driver selection, source name, mode, cadence, expected kinds,
|
||||
and driver params;
|
||||
- `sinks`: sink driver selection, sink name, and sink params;
|
||||
- `routes`: event-kind routing to named sinks.
|
||||
|
||||
The executable does not expose CLI flags or config path discovery.
|
||||
|
||||
## External Adapters Used
|
||||
|
||||
Runtime composition uses feedkit for:
|
||||
|
||||
- config loading;
|
||||
- source registry and expected-kind validation;
|
||||
- scheduler job construction;
|
||||
- processor registry and chain execution;
|
||||
- normalization and dedupe processors;
|
||||
- sink registry and built-in sinks;
|
||||
- route compilation and dispatch.
|
||||
|
||||
Weatherfeeder registers its own source drivers and its Postgres schema mapper.
|
||||
|
||||
## State
|
||||
|
||||
Weatherfeeder-owned runtime state is in process:
|
||||
|
||||
- event channel contents;
|
||||
- the bounded dedupe key set;
|
||||
- source instances and their HTTP conditional validators;
|
||||
- scheduler and dispatcher goroutines.
|
||||
|
||||
There is no weatherfeeder-owned durable scheduler state, checkpoint, replay log,
|
||||
or resume marker. Durable persistence is owned by configured external sinks.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Startup failures are fatal and include context such as config index, source name,
|
||||
sink name, driver name, or the operation that failed.
|
||||
|
||||
At runtime, scheduler and dispatcher errors are sent to a shared error channel.
|
||||
Context cancellation and deadline errors are treated as normal shutdown. Any
|
||||
other scheduler or dispatcher error is logged as fatal and cancels the process
|
||||
context.
|
||||
|
||||
The daemon handles `os.Interrupt` and `SIGTERM` with `signal.NotifyContext`.
|
||||
After both runtime goroutines return, it logs `shutdown complete`.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `cmd/weatherfeeder/main_test.go`
|
||||
- source registry tests under `internal/sources`
|
||||
- normalizer registration tests under `internal/normalizers`
|
||||
- feedkit scheduler, processor, dispatch, and sink tests when changing runtime
|
||||
infrastructure usage
|
||||
|
||||
## Invariants
|
||||
|
||||
- Keep normalization before dedupe.
|
||||
- Keep queue sizes and dedupe bounds explicit.
|
||||
- Preserve context-aware shutdown.
|
||||
- Keep runtime wiring separate from domain mapping and persistence rules.
|
||||
- Keep startup validation failures loud and contextual.
|
||||
105
docs/internal/sources.md
Normal file
105
docs/internal/sources.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# Source Internals
|
||||
|
||||
## Purpose
|
||||
|
||||
Source packages poll upstream weather providers and emit raw feed events. They
|
||||
are adapters, not canonical mappers.
|
||||
|
||||
Sources should decode only the metadata needed for event identity, effective
|
||||
time, and routing policy. Full provider payload interpretation belongs in
|
||||
normalizers.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs are feedkit `config.SourceConfig` values and upstream HTTP responses.
|
||||
Outputs are feed events whose payloads are raw provider JSON and whose schemas
|
||||
come from `standards`.
|
||||
|
||||
Current drivers:
|
||||
|
||||
| Driver | Kind | Raw schema |
|
||||
| --- | --- | --- |
|
||||
| `nws_observation` | `observation` | `raw.nws.observation.v1` |
|
||||
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` |
|
||||
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` |
|
||||
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` |
|
||||
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` |
|
||||
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` |
|
||||
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` |
|
||||
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` |
|
||||
| `openweather_observation` | `observation` | `raw.openweather.current.v1` |
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Source constructors validate source-specific params.
|
||||
- Sources use feedkit HTTP helpers for HTTP polling.
|
||||
- Sources emit raw events and should not build canonical `model` payloads.
|
||||
- Provider helper packages under `internal/providers/<provider>` hold shared
|
||||
parsing and validation helpers.
|
||||
- Registration is centralized in `internal/sources/builtins.go`.
|
||||
|
||||
## Config Fields Used
|
||||
|
||||
All current source drivers use feedkit `HTTPSource`.
|
||||
|
||||
Required params:
|
||||
|
||||
- `url`
|
||||
- `user_agent`
|
||||
|
||||
Optional params:
|
||||
|
||||
- `conditional`, default `true`;
|
||||
- `http_timeout`;
|
||||
- `http_response_body_limit_bytes`.
|
||||
|
||||
OpenWeather observation sources additionally require the configured URL to use
|
||||
metric units. This is enforced by `internal/providers/openweather`.
|
||||
|
||||
Source-level `kinds`, when configured, are validated against the source's
|
||||
advertised `Kinds()`.
|
||||
|
||||
## External Adapters Used
|
||||
|
||||
Sources use feedkit's HTTP helper for:
|
||||
|
||||
- request construction;
|
||||
- `User-Agent` and `Accept` headers;
|
||||
- optional conditional GET validators;
|
||||
- response body size limits;
|
||||
- JSON raw-message fetches.
|
||||
|
||||
NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times
|
||||
with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs.
|
||||
|
||||
## State
|
||||
|
||||
HTTP conditional validators are held in each source instance. They are not
|
||||
persisted across process restarts.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
Constructor failures are returned during startup and stop the daemon. Polling
|
||||
failures are returned to the scheduler.
|
||||
|
||||
If a source cannot decode minimal metadata from an otherwise successful upstream
|
||||
response, it still emits the raw event when possible. The event then falls back
|
||||
to default ID/effective-time behavior from feedkit source helpers.
|
||||
|
||||
Unchanged conditional responses return no events and no error.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/sources/builtins_test.go`
|
||||
- provider source tests under `internal/sources/nws`
|
||||
- provider source tests under `internal/sources/openmeteo`
|
||||
- provider source tests under `internal/sources/openweather`
|
||||
- provider helper tests under `internal/providers`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Emit raw provider schemas from `standards`.
|
||||
- Keep provider-to-canonical mapping out of sources.
|
||||
- Keep HTTP behavior context-aware.
|
||||
- Keep driver registration explicit and stable.
|
||||
- Keep source tests independent of live upstream services.
|
||||
Reference in New Issue
Block a user