Add policy documents and a roadmap to implement a full documentation set

docs/policy/architecture.md

@@ -0,0 +1,179 @@
+# Architecture Policy
+
+## Purpose
+
+This document defines `weatherfeeder`'s development architecture and invariants for maintainers and LLM coding agents. It describes how the implemented system is built and how future changes should preserve its boundaries.
+
+This is an inward-facing policy document. User-facing wire contracts belong in [`API.md`](../../API.md), and future work belongs under [`docs/roadmap/`](../roadmap/).
+
+## Project Shape
+
+`weatherfeeder` is a config-driven Go daemon that polls upstream weather providers, emits feed events, normalizes provider-specific raw payloads into canonical weather payloads, and routes those events to configured sinks.
+
+The implemented runtime flow is:
+
+1. `cmd/weatherfeeder` loads `config.yml` from the working directory.
+2. Source drivers are built through the source registry.
+3. Feedkit scheduler jobs poll sources and publish raw events onto an in-process event channel.
+4. A pipeline runs normalization, then in-memory dedupe.
+5. The dispatcher routes processed events to configured sinks by event kind.
+6. Sinks consume events independently through feedkit fanout workers.
+
+Canonical payload structs live in `model`. Schema identifiers and cross-provider wire conventions live in `standards`. Source adapters live under `internal/sources`. Normalizers live under `internal/normalizers`. Provider-specific parsing helpers shared by sources and normalizers live under `internal/providers`. Sink-specific persistence mapping lives under `internal/sinks`.
+
+## Core Design Principles
+
+- Hexagonal boundaries: provider APIs, config loading, scheduling, dispatch, and sinks are external mechanisms around the weather domain model and normalization logic.
+- Raw-to-canonical flow: sources should fetch and envelope raw provider payloads; normalizers should own provider-to-canonical mapping.
+- Schema-based routing: normalizers match on event schema, not source name or event kind.
+- Composable registries: source drivers, normalizers, processors, and sinks are assembled explicitly through registries.
+- Bounded concurrency: scheduling and sink fanout are concurrent, but the application should keep queues, goroutine ownership, logging, and cancellation behavior visible.
+- Standard-library-first: use the Go standard library unless a narrow dependency materially improves maintainability or interoperability.
+- Current-behavior docs: outside roadmap files, document only implemented behavior.
+
+## Architectural Boundaries
+
+Core/domain logic:
+
+- `model` defines canonical payload structs and JSON field names.
+- `standards` defines schema strings, versioning conventions, WMO constants, and shared wire policy.
+- Normalizer mapping code is domain logic and should stay independent of CLI setup, filesystem paths, sink details, and runtime orchestration.
+
+Adapters:
+
+- Source adapters under `internal/sources/<provider>` fetch upstream data and produce raw feed events.
+- Sink adapters under `internal/sinks/<sink>` map canonical events to external systems.
+- Provider helper packages under `internal/providers/<provider>` contain pure parsing or provider-specific helper logic shared by sources and normalizers.
+
+Runtime composition:
+
+- `cmd/weatherfeeder/main.go` owns process wiring: config load, registry setup, scheduler jobs, processor chain, dispatcher, signal cancellation, and logging.
+- It should remain thin. Do not move provider mapping or sink persistence rules into `cmd/weatherfeeder`.
+
+Tests and examples:
+
+- The sample `cmd/weatherfeeder/config.yml` is executable test input and is load-tested.
+- Tests should keep exercising package contracts directly rather than relying only on full-daemon execution.
+
+## Modules or Stages
+
+The implemented processing stages are source polling, normalization, dedupe, and sink dispatch.
+
+Source contract:
+
+- Build from `feedkit/config.SourceConfig`.
+- Validate required driver params inside the source constructor.
+- Advertise emitted event kinds through `Kinds()` when possible.
+- Emit raw schemas from `standards`.
+- Decode only minimal metadata needed for event identity and effective time; leave full provider decoding to normalizers.
+- Respect `context.Context` during network work.
+
+Normalizer contract:
+
+- One normalizer type per normalizer file.
+- Match by `Event.Schema`.
+- Decode raw payloads into provider structs.
+- Map to canonical `model` payloads.
+- Preserve the incoming event envelope except for intentional schema, payload, and effective-time changes.
+- Use shared helpers in `internal/normalizers/common` for cross-provider behavior.
+- Follow the detailed normalizer conventions in `internal/normalizers/doc.go`.
+
+Sink contract:
+
+- Consume canonical schemas, not provider raw schemas.
+- Keep sink mapping isolated from normalizers and sources.
+- Preserve event envelope fields in durable storage where the sink schema supports it.
+- Validate required canonical fields before writing.
+
+## State, Inputs, and Outputs
+
+Inputs are configured source polls. The daemon currently uses feedkit's YAML config model with sources, sinks, and routes.
+
+Outputs are feed events sent to configured sinks. Implemented sink support comes from feedkit built-ins plus weatherfeeder's Postgres schema mapping. The sample config includes stdout and NATS routes; Postgres is configured as an optional commented example.
+
+The daemon's own state is in-process:
+
+- the event channel buffers events during runtime;
+- the dedupe processor stores a bounded in-memory key set;
+- source instances may keep HTTP conditional request state through feedkit HTTP source helpers;
+- scheduler and dispatcher state is not persisted by `weatherfeeder`.
+
+Durable persistence is an external sink concern. The Postgres table contract is documented in `internal/sinks/postgres/doc.go`; the consumer-facing event contract is documented in [`API.md`](../../API.md).
+
+## Configuration and CLI Boundaries
+
+The implemented executable reads `config.yml` from the current working directory. It does not currently expose CLI flags or config path discovery.
+
+Configuration shape is owned by feedkit's config package:
+
+- `sources` define named source drivers, mode, poll cadence, expected kinds, and driver params.
+- `sinks` define named sink drivers and sink params.
+- `routes` connect event kinds to sinks.
+
+Weatherfeeder-specific config policy belongs in source and sink constructors, registry setup, and tests. Do not spread config parsing through domain model or normalizer packages.
+
+If dedicated `docs/config.md` or `docs/cli.md` files are added later, they should become the canonical user/operator references. This policy should stay architectural and avoid duplicating those references.
+
+## Errors, Logging, and Diagnostics
+
+`cmd/weatherfeeder` uses the standard library `log` package with timestamps and microseconds.
+
+Startup errors are fatal and include config index, source/sink name, driver name, or operation context where available. Runtime scheduler and dispatcher errors are reported through logs; context cancellation and deadline errors are treated as shutdown conditions.
+
+Normalizers and sink mappers should wrap errors with operation and payload context, for example decode, parse, map, scan, or required-field context. Avoid logging or returning whole upstream payloads by default.
+
+The daemon handles `os.Interrupt` and `SIGTERM` through `signal.NotifyContext`. Sources, sinks, scheduler jobs, dispatcher, and processors should respect `context.Context`.
+
+## Testing Expectations
+
+When changing behavior, inspect or add focused tests in the owning package.
+
+Expected coverage by change type:
+
+- Source drivers: constructor behavior, advertised kinds, poll event schema/kind, effective-time policy, unchanged responses, and malformed metadata handling.
+- Normalizers: schema matching, canonical schema output, key field mapping, effective time, malformed required fields, and wire-shape regressions.
+- Provider helpers: parsing edge cases and fixtures.
+- Runtime wiring: config loading, source registry build, scheduler job creation, processor ordering, pass-through behavior, and dedupe behavior.
+- Postgres sink: schema shape, mapper writes, required-field validation, nullable handling, and compact JSON behavior.
+- Documentation-sensitive examples: keep sample config loadable.
+
+Use local test servers and fixtures rather than real upstream services. Full-package tests should remain fast and deterministic.
+
+## Dependency Policy
+
+Prefer the Go standard library for parsing, HTTP handling, time handling, logging, and tests where reasonable.
+
+Existing broad runtime composition is delegated to `feedkit`, which provides config, sources, scheduler, processors, dispatch, and sinks. Keep weatherfeeder-specific code from depending directly on low-level external clients when feedkit or a small adapter can contain that dependency.
+
+Third-party dependencies should be narrow, justified, and preferably de facto standard for their purpose. YAML parsing through feedkit is an acceptable example. Do not add dependencies for small conveniences, and do not let dependency-specific types leak across package boundaries unless that dependency is the package's explicit contract.
+
+## Documentation Expectations
+
+Documentation must follow [`docs/policy/documentation.md`](documentation.md).
+
+Rules for architecture-related docs:
+
+- Current-behavior docs must describe implemented behavior only.
+- Roadmap or speculative work belongs only under `docs/roadmap/`.
+- Prefer links to canonical docs over repeated reference material.
+- Update docs in the same change when modifying schemas, config behavior, runtime behavior, adapters, or persistence contracts.
+
+## Architectural Invariants
+
+- Keep `cmd/weatherfeeder` as composition code, not domain logic.
+- Keep source fetching separate from normalization.
+- Keep normalizers matched by schema constants from `standards`.
+- Keep canonical payload structs in `model` and treat JSON tags as wire contract.
+- Keep provider-specific helpers under `internal/providers/<provider>` when shared by sources and normalizers.
+- Keep cross-provider normalizer helpers pure and deterministic.
+- Keep sink persistence mapping isolated under `internal/sinks/<sink>`.
+- Preserve explicit registry-based extension points for sources and normalizers.
+- Preserve context-aware shutdown and bounded in-process queues.
+- Avoid broad dependencies without clear architectural value.
+
+## Non-Goals
+
+- `weatherfeeder` is not an HTTP API. API serving belongs to separate consumers such as `weatherapi`.
+- `weatherfeeder` does not own long-term durable state except through configured external sinks.
+- `weatherfeeder` does not provide a general plugin runtime; new built-in providers and sinks are registered in code.
+- Architecture policy is not a CLI, config, or wire-contract reference.

docs/policy/documentation.md

@@ -0,0 +1,356 @@
+# Go Project Documentation Policy
+
+## Purpose
+
+Project documentation must help four audiences:
+
+1. users who need to run the application;
+2. administrators/operators who need to configure and operate it;
+3. developers who need to understand and change it safely;
+4. LLM coding agents that need clear scope, boundaries, and invariants.
+
+Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
+
+## Core Rules
+
+### 1. Keep docs concise
+
+Each document should cover a defined scope and only the essentials for that scope.
+
+Avoid:
+- long background explanations;
+- repeated reference material;
+- implementation detail in user-facing docs;
+- aspirational language outside roadmap docs;
+- verbose examples where one minimal example is clearer.
+
+### 2. Document only implemented behavior outside roadmap files
+
+Unimplemented, planned, aspirational, experimental, or future work may be described only under:
+
+- `docs/roadmap/`
+
+No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
+
+If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
+
+### 3. Use canonical homes
+
+Each type of information should have one canonical location.
+
+Canonical homes:
+
+- project purpose and quickstart: `README.md`
+- development principles: `docs/policy/architecture.md`
+- configuration reference: `docs/config.md`
+- CLI reference: `docs/cli.md`
+- operations and recovery: `docs/operations.md`
+- troubleshooting: `docs/troubleshooting.md`
+- implemented internals: `docs/internal/`
+- future work: `docs/roadmap/`
+- contributor workflow: `docs/policy/development.md`
+- copyable examples: `examples/`
+
+Other files should summarize briefly and link to the canonical source.
+
+### 4. Keep examples real
+
+Examples should be valid, maintained, and free of secrets.
+
+Where practical:
+- example configs should load successfully;
+- example commands should match real CLI syntax;
+- important examples should be covered by tests.
+
+## Documentation Profiles
+
+All projects require:
+
+- `README.md`
+- `docs/policy/architecture.md`
+
+Additional docs depend on the project.
+
+### Small library
+
+Recommended:
+- `docs/policy/development.md`, if contributor conventions are non-obvious
+
+### Simple CLI
+
+Required:
+- `docs/cli.md`
+
+Recommended:
+- `docs/policy/development.md`
+
+### Config-driven CLI
+
+Required:
+- `docs/cli.md`
+- `docs/config.md`
+
+Recommended:
+- `examples/`
+- `docs/policy/development.md`
+
+### Stateful or operator-facing application
+
+Required:
+- `docs/cli.md`, if CLI-based
+- `docs/config.md`, if config-driven
+- `docs/operations.md`
+
+Recommended:
+- `docs/troubleshooting.md`
+- `examples/`
+- `docs/policy/development.md`
+
+### Modular, staged, service-oriented, or orchestration application
+
+Required:
+- `docs/cli.md`, if CLI-based
+- `docs/config.md`, if config-driven
+- `docs/operations.md`
+- `docs/internal/`
+- `docs/policy/development.md`
+
+Recommended:
+- `docs/troubleshooting.md`
+- validated examples under `examples/`
+
+## Required Documents
+
+### README.md
+
+**Audience:** users, administrators, operators
+
+The README is the outward-facing project orientation page.
+
+It should include, in order:
+
+1. concise description;
+2. elevator pitch;
+3. shortest useful command or usage example;
+4. links to targeted docs.
+
+The README should be short. It is not a manual.
+
+The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
+
+### docs/policy/architecture.md
+
+**Audience:** developers, LLM coding agents
+
+`docs/policy/architecture.md` is required for every project.
+
+It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
+
+It should include:
+
+- project shape;
+- core design principles;
+- package and boundary philosophy;
+- state/persistence philosophy, if applicable;
+- external integration philosophy, if applicable;
+- error-handling and logging principles;
+- testing expectations;
+- documentation expectations;
+- architectural invariants;
+- explicit non-goals, if useful.
+
+For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
+
+### docs/policy/development.md
+
+**Audience:** developers, LLM coding agents
+
+Required for projects maintained by humans and LLM coding agents.
+
+It should include:
+
+- repository layout;
+- build/test commands;
+- coding conventions;
+- dependency policy;
+- how to add config fields;
+- how to add CLI flags;
+- how to add stages/modules/adapters, if applicable;
+- how to update examples;
+- documentation update expectations.
+
+### docs/config.md
+
+**Audience:** administrators, operators, advanced users
+
+Required for applications with configuration files.
+
+It should include, in order:
+
+1. config file locations and discovery precedence;
+2. minimal working config;
+3. production-oriented config;
+4. full configuration reference;
+5. secrets handling, if applicable;
+6. links to maintained examples.
+
+The full configuration reference should be canonical.
+
+### docs/cli.md
+
+**Audience:** users, administrators, operators
+
+Required for CLI applications.
+
+It should include, in order:
+
+1. shortest useful command;
+2. command overview;
+3. complete flag reference;
+4. common workflows;
+5. diagnostic or recovery commands, if applicable.
+
+Explain when commands are useful, not just their syntax.
+
+### docs/operations.md
+
+**Audience:** administrators, operators
+
+Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
+
+It should cover:
+
+- normal workflow;
+- filesystem layout;
+- remote storage layout, if applicable;
+- logs and manifests;
+- resume/retry behavior;
+- cleanup behavior;
+- archive/backup behavior;
+- safe recovery procedures;
+- operational caveats.
+
+### docs/troubleshooting.md
+
+**Audience:** administrators, operators
+
+Recommended once recurring failure modes exist.
+
+Each entry should include:
+
+- symptom;
+- likely cause;
+- diagnostic command or inspection step;
+- safe fix;
+- relevant links.
+
+### docs/internal/
+
+**Audience:** developers, LLM coding agents
+
+Required for modular, staged, service-oriented, or orchestration projects.
+
+This directory describes implemented internal components. It is not the roadmap.
+
+Use one file per major component where useful.
+
+Each component doc should include:
+
+1. purpose;
+2. inputs and outputs;
+3. boundaries;
+4. config fields used;
+5. external adapters used;
+6. state or manifest behavior, if applicable;
+7. skip/resume behavior, if applicable;
+8. failure behavior;
+9. tests to inspect before changing;
+10. architectural invariants.
+
+### docs/roadmap/
+
+**Audience:** maintainers, developers, LLM coding agents
+
+This is the only place for planned, future, aspirational, experimental, or unimplemented work.
+
+Roadmap docs should clearly distinguish:
+
+- proposed work;
+- accepted plans;
+- deferred ideas;
+- rejected ideas;
+- implementation prompts or task breakdowns, if useful.
+
+Roadmap docs should not be confused with current behavior.
+
+### docs/integrations/
+
+**Audience:** developers, LLM coding agents
+
+Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
+
+This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
+
+Use one file per integration where useful.
+
+## Examples Directory
+
+Projects with non-trivial configuration or workflows should include `examples/`.
+
+Useful examples include:
+
+- minimal working config;
+- production-oriented config;
+- full annotated config;
+- local development config;
+- remote/object-storage config;
+- minimal session/input file.
+
+Examples should be valid, maintained, tested when practical, and linked from relevant docs.
+
+## Security and Privacy
+
+Docs and examples must not include:
+
+- real API keys;
+- tokens;
+- passwords;
+- private keys;
+- private environment dumps;
+- sensitive user data;
+- raw private transcripts;
+- private infrastructure details unless intentionally public.
+
+Document secret-handling mechanisms, not actual secret values.
+
+## Maintenance Rules
+
+When docs change, verify the affected behavior.
+
+Where practical:
+
+- load example config files in tests;
+- test CLI examples or command parser behavior;
+- validate documented flags against real flags;
+- remove stale references;
+- update links after renames;
+- keep roadmap content out of non-roadmap docs.
+
+If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
+
+Documentation is complete only when it matches the current code.
+
+## Documentation Change Checklist
+
+Before merging documentation changes, verify:
+
+- README is concise and orientation-focused.
+- `docs/policy/architecture.md` describes development principles.
+- Future work appears only under `docs/roadmap/`.
+- User-facing docs avoid unnecessary internals.
+- Developer-facing docs preserve boundaries and invariants.
+- Config examples match the schema.
+- CLI examples match real commands and flags.
+- Defaults appear in the canonical config reference.
+- No secrets or private data are included.
+- Links are accurate.