Document operations and troubleshooting
This commit is contained in:
@@ -26,6 +26,8 @@ current working directory.
|
||||
|
||||
- [CLI reference](docs/cli.md)
|
||||
- [Configuration reference](docs/config.md)
|
||||
- [Operations guide](docs/operations.md)
|
||||
- [Troubleshooting guide](docs/troubleshooting.md)
|
||||
- [Event wire contract](docs/integrations/events.md)
|
||||
- [Postgres table contract](docs/integrations/postgres.md)
|
||||
- [Architecture policy](docs/policy/architecture.md)
|
||||
|
||||
156
docs/operations.md
Normal file
156
docs/operations.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Operations
|
||||
|
||||
This document describes how to run and observe the `weatherfeeder` daemon in its
|
||||
current form. For configuration syntax, see [configuration](config.md). For the
|
||||
CLI surface, see [CLI reference](cli.md).
|
||||
|
||||
## Normal Workflow
|
||||
|
||||
1. Prepare `config.yml` in the process working directory.
|
||||
2. Start the daemon with `./weatherfeeder` or `go run .` from
|
||||
`cmd/weatherfeeder`.
|
||||
3. Watch stderr logs for startup or runtime errors.
|
||||
4. Consume events from the configured sinks.
|
||||
5. Stop the process with `Ctrl-C` or `SIGTERM`.
|
||||
|
||||
The daemon has no admin subcommands and no runtime reload command. Change the
|
||||
config file and restart the process to apply configuration changes.
|
||||
|
||||
## Runtime Lifecycle
|
||||
|
||||
On startup, `weatherfeeder`:
|
||||
|
||||
1. loads `config.yml` from the current working directory;
|
||||
2. registers built-in source drivers;
|
||||
3. registers stdout, NATS, and weatherfeeder Postgres sink drivers;
|
||||
4. builds sources and validates configured `kinds` against source metadata;
|
||||
5. builds sinks and compiles routes;
|
||||
6. starts the scheduler and dispatcher;
|
||||
7. processes events through normalization, then in-memory dedupe;
|
||||
8. routes processed events to configured sinks.
|
||||
|
||||
Startup errors are fatal and terminate the process. Runtime poll, pipeline, and
|
||||
sink errors are logged and the process continues unless the scheduler or
|
||||
dispatcher returns a fatal error.
|
||||
|
||||
## Logs
|
||||
|
||||
The process uses the Go standard logger with date, time, and microseconds. Logs
|
||||
go to stderr.
|
||||
|
||||
Common log prefixes:
|
||||
|
||||
| Prefix | Meaning |
|
||||
|---|---|
|
||||
| `config load failed` | `config.yml` could not be read, parsed, or validated. |
|
||||
| `build source failed` | A source driver or its params are invalid. |
|
||||
| `source expected kinds validation failed` | Configured source `kinds` do not match the source driver. |
|
||||
| `build sink failed` | A sink driver or its params are invalid, or a sink could not initialize. |
|
||||
| `compile routes failed` | Routes reference invalid sinks or kinds. |
|
||||
| `scheduler: poll failed` | A source poll failed; the source will be polled again on its next interval. |
|
||||
| `dispatcher: pipeline error` | Normalization or dedupe failed for one event. |
|
||||
| `dispatch: sink ... failed consuming event` | A sink failed to consume one event. |
|
||||
| `shutdown complete` | Scheduler and dispatcher have exited. |
|
||||
|
||||
## Scheduling And Polling
|
||||
|
||||
Current weatherfeeder sources are polling sources. Each source uses its
|
||||
configured `every` interval. The scheduler applies jitter before the first poll
|
||||
and before each interval tick. If no jitter is configured in code, feedkit uses
|
||||
`min(every/10, 30s)`, capped at half the interval.
|
||||
|
||||
Poll failures are logged and do not stop the daemon. A failed poll emits no
|
||||
events for that source until a later poll succeeds.
|
||||
|
||||
## Conditional HTTP Fetches
|
||||
|
||||
All current sources use feedkit's HTTP polling helper. By default,
|
||||
`params.conditional` is `true`, so the helper keeps ETag and Last-Modified
|
||||
validators in memory for each source instance.
|
||||
|
||||
If the upstream returns `304 Not Modified`, the source emits no events for that
|
||||
poll. Validator state is in memory only; restarting the process starts with no
|
||||
cached validators.
|
||||
|
||||
## Processing And Dedupe
|
||||
|
||||
Every event passes through normalization first and dedupe second.
|
||||
|
||||
Normalizers match raw source schemas and produce canonical `weather.*.v1`
|
||||
payloads. If an event has no matching normalizer, the normalize processor passes
|
||||
it through unchanged.
|
||||
|
||||
Dedupe keys by event ID and stores a bounded in-memory set of 2048 recent IDs.
|
||||
Duplicate IDs are dropped. Dedupe state is not persisted, so a restart starts
|
||||
with an empty dedupe set.
|
||||
|
||||
## Routing And Sink Fanout
|
||||
|
||||
Routes choose sinks by event kind. If `routes` is omitted, every sink receives
|
||||
every event kind. If a route omits `kinds`, that route also matches all kinds.
|
||||
|
||||
The dispatcher creates one queue and one worker goroutine per sink. The default
|
||||
per-sink queue size is 64. `weatherfeeder` does not currently expose config
|
||||
fields for sink queue size, enqueue timeout, or consume timeout.
|
||||
|
||||
Sink errors are logged per event. A sink failure does not stop other sinks from
|
||||
receiving the same event.
|
||||
|
||||
## Sink Behavior
|
||||
|
||||
### stdout
|
||||
|
||||
The stdout sink validates each event and prints one JSON object per line to
|
||||
stdout. This is useful for local inspection and log forwarding.
|
||||
|
||||
### NATS
|
||||
|
||||
The NATS sink connects lazily on the first event, reuses the connection while it
|
||||
is open, and publishes each event as JSON to the configured subject. Connection,
|
||||
marshal, and publish failures are logged by the dispatch worker.
|
||||
|
||||
### Postgres
|
||||
|
||||
The Postgres sink opens the database during startup. It creates missing tables
|
||||
and indexes with `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS`.
|
||||
It does not modify existing table definitions.
|
||||
|
||||
Each mapped canonical event is written in one transaction. If `params.prune` is
|
||||
set, the sink deletes rows older than `now - prune` from every weatherfeeder
|
||||
table in that same transaction. See the
|
||||
[Postgres table contract](integrations/postgres.md).
|
||||
|
||||
## State And Recovery
|
||||
|
||||
`weatherfeeder` keeps only runtime state in process memory:
|
||||
|
||||
- scheduler goroutines and timers;
|
||||
- HTTP conditional request validators;
|
||||
- event channel buffers;
|
||||
- per-sink fanout queues;
|
||||
- the dedupe ID set.
|
||||
|
||||
Durable state is external sink state: NATS broker state outside this process and
|
||||
Postgres tables managed by the configured database.
|
||||
|
||||
There is no internal checkpoint, replay log, or resume marker. To recover from a
|
||||
process failure, fix the underlying issue and restart the daemon from a working
|
||||
directory containing the desired `config.yml`.
|
||||
|
||||
## Shutdown
|
||||
|
||||
`weatherfeeder` listens for `os.Interrupt` and `SIGTERM`. On shutdown, the
|
||||
shared context is canceled. Scheduler jobs stop polling, dispatch workers stop,
|
||||
and the process logs `shutdown complete`.
|
||||
|
||||
Queued sink work may be dropped when shutdown context cancellation reaches the
|
||||
fanout workers. Use external sink durability, such as Postgres or broker
|
||||
retention, for durable downstream state.
|
||||
|
||||
## Caveats
|
||||
|
||||
- There is no health-check endpoint.
|
||||
- There is no runtime config reload.
|
||||
- There are no built-in metrics.
|
||||
- Source conditional request state and dedupe state are reset by restart.
|
||||
- Existing Postgres schemas are not migrated automatically.
|
||||
236
docs/troubleshooting.md
Normal file
236
docs/troubleshooting.md
Normal file
@@ -0,0 +1,236 @@
|
||||
# Troubleshooting
|
||||
|
||||
Use this guide with [configuration](config.md) and [operations](operations.md).
|
||||
Messages are emitted through the standard logger on stderr.
|
||||
|
||||
## `config load failed: ... read "config.yml"`
|
||||
|
||||
Symptom: startup exits before building sources or sinks.
|
||||
|
||||
Likely cause: the process working directory does not contain `config.yml`, or
|
||||
the runtime user cannot read it.
|
||||
|
||||
Diagnostic: run `pwd` in the same working directory used by the process, then
|
||||
check `ls -l config.yml`.
|
||||
|
||||
Safe fix: place the intended config at `./config.yml`, change the working
|
||||
directory, or mount the file at `/weatherfeeder/config.yml` when using the
|
||||
provided container image.
|
||||
|
||||
## `config load failed: ... parse YAML`
|
||||
|
||||
Symptom: startup exits with a YAML parse error or an unknown field error.
|
||||
|
||||
Likely cause: invalid YAML syntax, multiple YAML documents, or a misspelled
|
||||
config struct field.
|
||||
|
||||
Diagnostic: inspect the line and field in the error. Feedkit uses strict YAML
|
||||
field decoding for config struct fields.
|
||||
|
||||
Safe fix: correct the YAML and compare the shape with
|
||||
[configuration](config.md). Driver-specific `params` keys are validated later by
|
||||
their source or sink constructors.
|
||||
|
||||
## `config validation failed`
|
||||
|
||||
Symptom: startup exits and prints one or more validation messages.
|
||||
|
||||
Likely cause: missing `sources` or `sinks`, blank names, duplicate source or sink
|
||||
names, invalid `mode`, missing `every` for a polling source with `mode: poll`, or
|
||||
a route that references an unknown sink.
|
||||
|
||||
Diagnostic: read every bullet under `config validation failed`; the loader sorts
|
||||
these messages so multiple issues can be fixed in one edit.
|
||||
|
||||
Safe fix: update the top-level config fields as documented in
|
||||
[configuration](config.md).
|
||||
|
||||
## `unknown source driver`
|
||||
|
||||
Symptom: startup exits with `build source failed`.
|
||||
|
||||
Likely cause: `sources[].driver` does not match a registered weatherfeeder
|
||||
source driver.
|
||||
|
||||
Diagnostic: compare the configured driver with the source driver table in
|
||||
[configuration](config.md#source-drivers).
|
||||
|
||||
Safe fix: correct the driver name. Current drivers include `nws_observation`,
|
||||
`nws_alerts`, `nws_forecast_hourly`, `nws_forecast_narrative`,
|
||||
`nws_forecast_discussion`, `nws_weatherstories`, `openmeteo_observation`,
|
||||
`openmeteo_forecast`, and `openweather_observation`.
|
||||
|
||||
## `unknown sink driver`
|
||||
|
||||
Symptom: startup exits with `build sink failed`.
|
||||
|
||||
Likely cause: `sinks[].driver` is not registered.
|
||||
|
||||
Diagnostic: compare the configured driver with the sink driver table in
|
||||
[configuration](config.md#sink-drivers).
|
||||
|
||||
Safe fix: use `stdout`, `nats`, or `postgres`.
|
||||
|
||||
## `source expected kinds validation failed`
|
||||
|
||||
Symptom: startup exits after building a source.
|
||||
|
||||
Likely cause: `sources[].kinds` declares a kind the source does not emit.
|
||||
|
||||
Diagnostic: compare the configured `kinds` list with the source driver kind in
|
||||
[configuration](config.md#source-drivers).
|
||||
|
||||
Safe fix: remove `kinds` or set it to the kind emitted by that driver.
|
||||
|
||||
## `params.url is required` Or `params.user_agent is required`
|
||||
|
||||
Symptom: startup exits with `build source failed`.
|
||||
|
||||
Likely cause: a source is missing required HTTP params, or the values are blank
|
||||
or not strings.
|
||||
|
||||
Diagnostic: inspect the named source in the error and check its `params`.
|
||||
|
||||
Safe fix: add non-empty `url` and `user_agent` values. See
|
||||
[HTTP source params](config.md#http-source-params).
|
||||
|
||||
## `url must include units=metric`
|
||||
|
||||
Symptom: startup exits for an `openweather_observation` source.
|
||||
|
||||
Likely cause: the OpenWeather URL omits `units=metric` or sets another unit
|
||||
system.
|
||||
|
||||
Diagnostic: inspect the query string in `params.url`.
|
||||
|
||||
Safe fix: add `units=metric` to the OpenWeather current-weather URL. Keep API
|
||||
keys out of committed configs.
|
||||
|
||||
## `source ... sources[].every must be > 0 for polling sources`
|
||||
|
||||
Symptom: startup exits while building scheduler jobs.
|
||||
|
||||
Likely cause: a current weatherfeeder polling source has no usable `every`
|
||||
interval.
|
||||
|
||||
Diagnostic: inspect the named `sources[]` entry and check `every`.
|
||||
|
||||
Safe fix: set a positive duration such as `1m`, `10m`, or `1h`.
|
||||
|
||||
## `build sink failed ... params.url is required`
|
||||
|
||||
Symptom: startup exits while building a NATS sink.
|
||||
|
||||
Likely cause: the NATS sink is missing `params.url`, or the value is blank or
|
||||
not a string.
|
||||
|
||||
Diagnostic: inspect the named sink in the error and check its `params`.
|
||||
|
||||
Safe fix: set a NATS URL such as `nats://localhost:4222`.
|
||||
|
||||
## `build sink failed ... params.subject is required`
|
||||
|
||||
Symptom: startup exits while building a NATS sink.
|
||||
|
||||
Likely cause: the NATS sink is missing `params.subject`, or the value is blank
|
||||
or not a string.
|
||||
|
||||
Diagnostic: inspect the named sink in the error and check its `params`.
|
||||
|
||||
Safe fix: set a non-empty subject such as `weatherfeeder`.
|
||||
|
||||
## `dispatch: sink ... failed consuming event ... NATS sink: connect`
|
||||
|
||||
Symptom: the daemon starts, but NATS events are not published.
|
||||
|
||||
Likely cause: the NATS server URL is unreachable, the server is not accepting
|
||||
connections, or the configured URL is wrong for the runtime network.
|
||||
|
||||
Diagnostic: from the same runtime environment, check that the host and port in
|
||||
`sinks[].params.url` are reachable.
|
||||
|
||||
Safe fix: correct the NATS URL or restore broker connectivity. Other configured
|
||||
sinks continue receiving events.
|
||||
|
||||
## `postgres sink ... open db`
|
||||
|
||||
Symptom: startup exits while building a Postgres sink.
|
||||
|
||||
Likely cause: the database URI, username, password, network path, or database
|
||||
availability is wrong.
|
||||
|
||||
Diagnostic: inspect `sinks[].params.uri`, `username`, and `password`; verify
|
||||
that the same runtime environment can reach the database.
|
||||
|
||||
Safe fix: correct the credentials or URI, restore database connectivity, then
|
||||
restart the daemon.
|
||||
|
||||
## `postgres sink ... ensure table` Or `ensure index`
|
||||
|
||||
Symptom: startup exits during Postgres initialization.
|
||||
|
||||
Likely cause: the database user cannot create required tables or indexes, an
|
||||
existing object conflicts with weatherfeeder's expected table contract, or the
|
||||
database is unavailable during initialization.
|
||||
|
||||
Diagnostic: inspect the named table or index in the error and compare existing
|
||||
database objects with the [Postgres table contract](integrations/postgres.md).
|
||||
|
||||
Safe fix: grant the needed database privileges, create a compatible schema, or
|
||||
perform an operator-managed migration before restarting.
|
||||
|
||||
## `postgres sink: insert into ...`
|
||||
|
||||
Symptom: the daemon starts, but Postgres writes for some events fail.
|
||||
|
||||
Likely cause: a duplicate primary key, incompatible existing table definition,
|
||||
database constraint error, or connection failure during a write transaction.
|
||||
|
||||
Diagnostic: inspect the table name and database error in the log. Compare the
|
||||
table with [Postgres integration](integrations/postgres.md).
|
||||
|
||||
Safe fix: repair the database schema or address the duplicate/connection issue.
|
||||
Other configured sinks continue receiving events.
|
||||
|
||||
## No Events Appear On A Sink
|
||||
|
||||
Symptom: the daemon is running but the expected sink receives no events.
|
||||
|
||||
Likely cause: the route does not match the event kind, the source has not
|
||||
emitted changed content, or the sink is failing per event.
|
||||
|
||||
Diagnostic: check `routes`, source `kinds`, and logs for `scheduler: poll
|
||||
failed`, `dispatcher: pipeline error`, or `dispatch: sink ... failed consuming
|
||||
event`.
|
||||
|
||||
Safe fix: correct the route or source configuration. If the source uses
|
||||
conditional HTTP and the upstream has not changed, no event is emitted for a
|
||||
`304 Not Modified` response; wait for changed upstream content or temporarily
|
||||
set `params.conditional: false` for diagnosis.
|
||||
|
||||
## `scheduler: poll failed`
|
||||
|
||||
Symptom: one source logs poll failures while the daemon keeps running.
|
||||
|
||||
Likely cause: upstream HTTP error, bad URL, timeout, response body limit, or
|
||||
provider response shape that the source cannot parse.
|
||||
|
||||
Diagnostic: inspect the source name in the log and review its HTTP params.
|
||||
|
||||
Safe fix: correct the URL, user agent, timeout, or body limit. The next
|
||||
scheduled poll will retry.
|
||||
|
||||
## `dispatcher: pipeline error`
|
||||
|
||||
Symptom: source polling succeeds, but one event is dropped before sinks.
|
||||
|
||||
Likely cause: a normalizer could not decode or map the raw payload, or dedupe
|
||||
received an invalid event ID.
|
||||
|
||||
Diagnostic: inspect the error text and the source/schema that produced the
|
||||
event. Review the [event wire contract](integrations/events.md) for expected
|
||||
canonical fields.
|
||||
|
||||
Safe fix: correct source configuration if it points to the wrong upstream
|
||||
product. If the upstream payload changed shape, update the relevant normalizer
|
||||
and tests.
|
||||
Reference in New Issue
Block a user