Document development workflow and internals

docs/internal/runtime.md

@@ -0,0 +1,110 @@
+# Runtime Internals
+
+## Purpose
+
+`cmd/weatherfeeder` wires the daemon together. It owns process setup and runtime
+composition; provider mapping, source fetching details, and sink persistence
+rules stay in their owning packages.
+
+## Inputs And Outputs
+
+The executable reads `config.yml` from the current working directory through
+feedkit config loading. It builds configured sources, scheduler jobs, sinks, and
+routes, then runs source polling and sink dispatch until shutdown.
+
+Inputs are configured source polls. Outputs are feed events delivered to the
+configured sinks.
+
+## Runtime Flow
+
+The implemented flow is:
+
+1. load `config.yml`;
+2. register weatherfeeder source drivers;
+3. register feedkit built-in sinks and the weatherfeeder Postgres sink;
+4. build source inputs and scheduler jobs;
+5. validate configured expected kinds against source-advertised kinds;
+6. build sinks and compile routes;
+7. run the processor chain `normalize`, then `dedupe`;
+8. run the scheduler and dispatcher concurrently;
+9. shut down on signal or fatal scheduler/dispatcher error.
+
+The in-process event channel is buffered to 256 events. The dedupe processor is
+bounded by `dedupeMaxEntries`, currently 2048.
+
+## Boundaries
+
+- Runtime composition belongs in `cmd/weatherfeeder/main.go`.
+- Source driver behavior belongs under `internal/sources`.
+- Normalizer behavior belongs under `internal/normalizers`.
+- Canonical payloads and schema strings belong in `model` and `standards`.
+- Postgres mapping belongs under `internal/sinks/postgres`.
+
+`cmd/weatherfeeder` should stay thin and should not contain provider parsing,
+canonical mapping, or table-mapping rules.
+
+## Config Fields Used
+
+Runtime wiring consumes the feedkit top-level config sections:
+
+- `sources`: source driver selection, source name, mode, cadence, expected kinds,
+  and driver params;
+- `sinks`: sink driver selection, sink name, and sink params;
+- `routes`: event-kind routing to named sinks.
+
+The executable does not expose CLI flags or config path discovery.
+
+## External Adapters Used
+
+Runtime composition uses feedkit for:
+
+- config loading;
+- source registry and expected-kind validation;
+- scheduler job construction;
+- processor registry and chain execution;
+- normalization and dedupe processors;
+- sink registry and built-in sinks;
+- route compilation and dispatch.
+
+Weatherfeeder registers its own source drivers and its Postgres schema mapper.
+
+## State
+
+Weatherfeeder-owned runtime state is in process:
+
+- event channel contents;
+- the bounded dedupe key set;
+- source instances and their HTTP conditional validators;
+- scheduler and dispatcher goroutines.
+
+There is no weatherfeeder-owned durable scheduler state, checkpoint, replay log,
+or resume marker. Durable persistence is owned by configured external sinks.
+
+## Failure Behavior
+
+Startup failures are fatal and include context such as config index, source name,
+sink name, driver name, or the operation that failed.
+
+At runtime, scheduler and dispatcher errors are sent to a shared error channel.
+Context cancellation and deadline errors are treated as normal shutdown. Any
+other scheduler or dispatcher error is logged as fatal and cancels the process
+context.
+
+The daemon handles `os.Interrupt` and `SIGTERM` with `signal.NotifyContext`.
+After both runtime goroutines return, it logs `shutdown complete`.
+
+## Tests To Inspect
+
+- `cmd/weatherfeeder/main_test.go`
+- source registry tests under `internal/sources`
+- normalizer registration tests under `internal/normalizers`
+- feedkit scheduler, processor, dispatch, and sink tests when changing runtime
+  infrastructure usage
+
+## Invariants
+
+- Keep normalization before dedupe.
+- Keep queue sizes and dedupe bounds explicit.
+- Preserve context-aware shutdown.
+- Keep runtime wiring separate from domain mapping and persistence rules.
+- Keep startup validation failures loud and contextual.