Document operations and troubleshooting

docs/operations.md

@@ -0,0 +1,156 @@
+# Operations
+
+This document describes how to run and observe the `weatherfeeder` daemon in its
+current form. For configuration syntax, see [configuration](config.md). For the
+CLI surface, see [CLI reference](cli.md).
+
+## Normal Workflow
+
+1. Prepare `config.yml` in the process working directory.
+2. Start the daemon with `./weatherfeeder` or `go run .` from
+   `cmd/weatherfeeder`.
+3. Watch stderr logs for startup or runtime errors.
+4. Consume events from the configured sinks.
+5. Stop the process with `Ctrl-C` or `SIGTERM`.
+
+The daemon has no admin subcommands and no runtime reload command. Change the
+config file and restart the process to apply configuration changes.
+
+## Runtime Lifecycle
+
+On startup, `weatherfeeder`:
+
+1. loads `config.yml` from the current working directory;
+2. registers built-in source drivers;
+3. registers stdout, NATS, and weatherfeeder Postgres sink drivers;
+4. builds sources and validates configured `kinds` against source metadata;
+5. builds sinks and compiles routes;
+6. starts the scheduler and dispatcher;
+7. processes events through normalization, then in-memory dedupe;
+8. routes processed events to configured sinks.
+
+Startup errors are fatal and terminate the process. Runtime poll, pipeline, and
+sink errors are logged and the process continues unless the scheduler or
+dispatcher returns a fatal error.
+
+## Logs
+
+The process uses the Go standard logger with date, time, and microseconds. Logs
+go to stderr.
+
+Common log prefixes:
+
+| Prefix | Meaning |
+|---|---|
+| `config load failed` | `config.yml` could not be read, parsed, or validated. |
+| `build source failed` | A source driver or its params are invalid. |
+| `source expected kinds validation failed` | Configured source `kinds` do not match the source driver. |
+| `build sink failed` | A sink driver or its params are invalid, or a sink could not initialize. |
+| `compile routes failed` | Routes reference invalid sinks or kinds. |
+| `scheduler: poll failed` | A source poll failed; the source will be polled again on its next interval. |
+| `dispatcher: pipeline error` | Normalization or dedupe failed for one event. |
+| `dispatch: sink ... failed consuming event` | A sink failed to consume one event. |
+| `shutdown complete` | Scheduler and dispatcher have exited. |
+
+## Scheduling And Polling
+
+Current weatherfeeder sources are polling sources. Each source uses its
+configured `every` interval. The scheduler applies jitter before the first poll
+and before each interval tick. If no jitter is configured in code, feedkit uses
+`min(every/10, 30s)`, capped at half the interval.
+
+Poll failures are logged and do not stop the daemon. A failed poll emits no
+events for that source until a later poll succeeds.
+
+## Conditional HTTP Fetches
+
+All current sources use feedkit's HTTP polling helper. By default,
+`params.conditional` is `true`, so the helper keeps ETag and Last-Modified
+validators in memory for each source instance.
+
+If the upstream returns `304 Not Modified`, the source emits no events for that
+poll. Validator state is in memory only; restarting the process starts with no
+cached validators.
+
+## Processing And Dedupe
+
+Every event passes through normalization first and dedupe second.
+
+Normalizers match raw source schemas and produce canonical `weather.*.v1`
+payloads. If an event has no matching normalizer, the normalize processor passes
+it through unchanged.
+
+Dedupe keys by event ID and stores a bounded in-memory set of 2048 recent IDs.
+Duplicate IDs are dropped. Dedupe state is not persisted, so a restart starts
+with an empty dedupe set.
+
+## Routing And Sink Fanout
+
+Routes choose sinks by event kind. If `routes` is omitted, every sink receives
+every event kind. If a route omits `kinds`, that route also matches all kinds.
+
+The dispatcher creates one queue and one worker goroutine per sink. The default
+per-sink queue size is 64. `weatherfeeder` does not currently expose config
+fields for sink queue size, enqueue timeout, or consume timeout.
+
+Sink errors are logged per event. A sink failure does not stop other sinks from
+receiving the same event.
+
+## Sink Behavior
+
+### stdout
+
+The stdout sink validates each event and prints one JSON object per line to
+stdout. This is useful for local inspection and log forwarding.
+
+### NATS
+
+The NATS sink connects lazily on the first event, reuses the connection while it
+is open, and publishes each event as JSON to the configured subject. Connection,
+marshal, and publish failures are logged by the dispatch worker.
+
+### Postgres
+
+The Postgres sink opens the database during startup. It creates missing tables
+and indexes with `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS`.
+It does not modify existing table definitions.
+
+Each mapped canonical event is written in one transaction. If `params.prune` is
+set, the sink deletes rows older than `now - prune` from every weatherfeeder
+table in that same transaction. See the
+[Postgres table contract](integrations/postgres.md).
+
+## State And Recovery
+
+`weatherfeeder` keeps only runtime state in process memory:
+
+- scheduler goroutines and timers;
+- HTTP conditional request validators;
+- event channel buffers;
+- per-sink fanout queues;
+- the dedupe ID set.
+
+Durable state is external sink state: NATS broker state outside this process and
+Postgres tables managed by the configured database.
+
+There is no internal checkpoint, replay log, or resume marker. To recover from a
+process failure, fix the underlying issue and restart the daemon from a working
+directory containing the desired `config.yml`.
+
+## Shutdown
+
+`weatherfeeder` listens for `os.Interrupt` and `SIGTERM`. On shutdown, the
+shared context is canceled. Scheduler jobs stop polling, dispatch workers stop,
+and the process logs `shutdown complete`.
+
+Queued sink work may be dropped when shutdown context cancellation reaches the
+fanout workers. Use external sink durability, such as Postgres or broker
+retention, for durable downstream state.
+
+## Caveats
+
+- There is no health-check endpoint.
+- There is no runtime config reload.
+- There are no built-in metrics.
+- Source conditional request state and dedupe state are reset by restart.
+- Existing Postgres schemas are not migrated automatically.