Eric Rakestraw
fba519cab0
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
111 lines
3.8 KiB
Markdown
111 lines
3.8 KiB
Markdown
# Runtime Internals
|
|
|
|
## Purpose
|
|
|
|
`cmd/weatherfeeder` wires the daemon together. It owns process setup and runtime
|
|
composition; provider mapping, source fetching details, and sink persistence
|
|
rules stay in their owning packages.
|
|
|
|
## Inputs And Outputs
|
|
|
|
The executable reads `config.yml` from the current working directory through
|
|
feedkit config loading. It builds configured sources, scheduler jobs, sinks, and
|
|
routes, then runs source polling and sink dispatch until shutdown.
|
|
|
|
Inputs are configured source polls. Outputs are feed events delivered to the
|
|
configured sinks.
|
|
|
|
## Runtime Flow
|
|
|
|
The implemented flow is:
|
|
|
|
1. load `config.yml`;
|
|
2. register weatherfeeder source drivers;
|
|
3. register feedkit built-in sinks and the weatherfeeder Postgres sink;
|
|
4. build source inputs and scheduler jobs;
|
|
5. validate configured expected kinds against source-advertised kinds;
|
|
6. build sinks and compile routes;
|
|
7. run the processor chain `normalize`, then `dedupe`;
|
|
8. run the scheduler and dispatcher concurrently;
|
|
9. shut down on signal or fatal scheduler/dispatcher error.
|
|
|
|
The in-process event channel is buffered to 256 events. The dedupe processor is
|
|
bounded by `dedupeMaxEntries`, currently 2048.
|
|
|
|
## Boundaries
|
|
|
|
- Runtime composition belongs in `cmd/weatherfeeder/main.go`.
|
|
- Source driver behavior belongs under `internal/sources`.
|
|
- Normalizer behavior belongs under `internal/normalizers`.
|
|
- Canonical payloads and schema strings belong in `model` and `standards`.
|
|
- Postgres mapping belongs under `internal/sinks/postgres`.
|
|
|
|
`cmd/weatherfeeder` should stay thin and should not contain provider parsing,
|
|
canonical mapping, or table-mapping rules.
|
|
|
|
## Config Fields Used
|
|
|
|
Runtime wiring consumes the feedkit top-level config sections:
|
|
|
|
- `sources`: source driver selection, source name, mode, cadence, expected kinds,
|
|
and driver params;
|
|
- `sinks`: sink driver selection, sink name, and sink params;
|
|
- `routes`: event-kind routing to named sinks.
|
|
|
|
The executable does not expose CLI flags or config path discovery.
|
|
|
|
## External Adapters Used
|
|
|
|
Runtime composition uses feedkit for:
|
|
|
|
- config loading;
|
|
- source registry and expected-kind validation;
|
|
- scheduler job construction;
|
|
- processor registry and chain execution;
|
|
- normalization and dedupe processors;
|
|
- sink registry and built-in sinks;
|
|
- route compilation and dispatch.
|
|
|
|
Weatherfeeder registers its own source drivers and its Postgres schema mapper.
|
|
|
|
## State
|
|
|
|
Weatherfeeder-owned runtime state is in process:
|
|
|
|
- event channel contents;
|
|
- the bounded dedupe key set;
|
|
- source instances and their in-memory unchanged-content state;
|
|
- scheduler and dispatcher goroutines.
|
|
|
|
There is no weatherfeeder-owned durable scheduler state, checkpoint, replay log,
|
|
or resume marker. Durable persistence is owned by configured external sinks.
|
|
|
|
## Failure Behavior
|
|
|
|
Startup failures are fatal and include context such as config index, source name,
|
|
sink name, driver name, or the operation that failed.
|
|
|
|
At runtime, scheduler and dispatcher errors are sent to a shared error channel.
|
|
Context cancellation and deadline errors are treated as normal shutdown. Any
|
|
other scheduler or dispatcher error is logged as fatal and cancels the process
|
|
context.
|
|
|
|
The daemon handles `os.Interrupt` and `SIGTERM` with `signal.NotifyContext`.
|
|
After both runtime goroutines return, it logs `shutdown complete`.
|
|
|
|
## Tests To Inspect
|
|
|
|
- `cmd/weatherfeeder/main_test.go`
|
|
- source registry tests under `internal/sources`
|
|
- normalizer registration tests under `internal/normalizers`
|
|
- feedkit scheduler, processor, dispatch, and sink tests when changing runtime
|
|
infrastructure usage
|
|
|
|
## Invariants
|
|
|
|
- Keep normalization before dedupe.
|
|
- Keep queue sizes and dedupe bounds explicit.
|
|
- Preserve context-aware shutdown.
|
|
- Keep runtime wiring separate from domain mapping and persistence rules.
|
|
- Keep startup validation failures loud and contextual.
|