Implement remaining cleanup items prior to the next release
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
This commit is contained in:
@@ -1,82 +0,0 @@
|
||||
# Code Quality Audit Snapshot
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap file records the current cleanup audit status for `weatherfeeder`.
|
||||
It is not current-behavior documentation and should not be used as the source of
|
||||
truth for implemented features. Current behavior belongs in README, config,
|
||||
operations, internal, consumer, and integration docs.
|
||||
|
||||
## Current Assessment
|
||||
|
||||
`weatherfeeder` remains architecturally coherent:
|
||||
|
||||
- `cmd/weatherfeeder` is runtime composition.
|
||||
- Source adapters fetch upstream provider payloads.
|
||||
- Provider helpers isolate provider-specific parsing.
|
||||
- Normalizers map raw schemas into canonical `model` payloads.
|
||||
- `standards` owns schema, kind, and WMO constants.
|
||||
- Postgres sink mapping is isolated under `internal/sinks/postgres`.
|
||||
- Feedkit owns generic config, scheduling, processing, dispatch, and sink
|
||||
mechanics.
|
||||
|
||||
The highest-value completed cleanup work since the original audit includes:
|
||||
|
||||
- event kind constants in `standards`;
|
||||
- source driver constants in provider source packages;
|
||||
- source registry tests derived from registered drivers;
|
||||
- shared HTTP config parsing for multi-document sources;
|
||||
- config/example load coverage;
|
||||
- shallow documentation consistency tests for current schemas and source
|
||||
drivers;
|
||||
- consumer documentation for public Go packages.
|
||||
|
||||
## Remaining Cleanup Opportunities
|
||||
|
||||
### Postgres Mapper Boilerplate
|
||||
|
||||
The Postgres mapper still repeats event envelope values and parent-row map
|
||||
patterns across canonical product families.
|
||||
|
||||
Recommended next action: implement the Postgres mapper envelope cleanup in
|
||||
[`cleanup.md`](cleanup.md) without changing table contracts or mapper behavior.
|
||||
|
||||
### Package-Local Fixture Helpers
|
||||
|
||||
Some tests still use package-local fixture-loading helpers that may be
|
||||
consolidated where multiple files in the same package duplicate the same logic.
|
||||
|
||||
Recommended next action: consolidate only obvious same-package duplication. Do
|
||||
not add cross-package test helper packages.
|
||||
|
||||
### Literal Sweep
|
||||
|
||||
Some string literals remain intentional in docs, YAML examples, and negative
|
||||
tests. Internal code and tests can still be reviewed for opportunities to use
|
||||
existing constants where that reduces drift risk.
|
||||
|
||||
Recommended next action: perform a final literal sweep only after higher-value
|
||||
cleanup stages are complete.
|
||||
|
||||
## Non-Issues
|
||||
|
||||
These areas were reviewed and should not be refactored without a new roadmap:
|
||||
|
||||
- Normalizer `Match` methods are intentionally explicit schema checks.
|
||||
- Provider-specific time parsers should remain provider-specific.
|
||||
- SPC multi-document polling should remain source-local and atomic.
|
||||
- The current runtime does not need a generic workflow engine.
|
||||
- The Postgres sink should not become an ORM or reflection mapper.
|
||||
- Documentation should remain audience-specific rather than generated from code.
|
||||
|
||||
## Validation Baseline
|
||||
|
||||
The current test suite covers source construction, normalizer registration and
|
||||
mapping, Postgres schema/mapping, maintained config loading, source-driver docs,
|
||||
and event-schema docs.
|
||||
|
||||
Recommended baseline after any cleanup:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
```
|
||||
@@ -1,146 +0,0 @@
|
||||
# Cleanup Roadmap
|
||||
|
||||
## Summary
|
||||
|
||||
This roadmap tracks remaining behavior-preserving cleanup opportunities for
|
||||
`weatherfeeder`. Earlier cleanup stages for event kind constants, source driver
|
||||
constants, registry-derived source tests, shared multi-document HTTP config, and
|
||||
shallow docs consistency tests have been completed and are no longer listed as
|
||||
future work.
|
||||
|
||||
The remaining work should be implemented only in small, focused changes that do
|
||||
not alter public event schemas, event kinds, source driver names, config keys,
|
||||
Postgres table contracts, or canonical JSON field names.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Keep `feedkit` as the generic daemon infrastructure boundary.
|
||||
- Keep weather-domain behavior in sources, provider helpers, normalizers,
|
||||
`model`, `standards`, and Postgres mapping.
|
||||
- Keep current-behavior docs accurate and roadmap-only plans under
|
||||
`docs/roadmap/`.
|
||||
- Do not introduce plugin systems, workflow engines, generic source frameworks,
|
||||
ORM-style mapping, reflection mapping, generated docs, or persistence tags on
|
||||
canonical model structs.
|
||||
- Run focused tests after each stage and `go test ./...` after all stages.
|
||||
|
||||
## Stage 1: Reduce Postgres Mapper Envelope Duplication
|
||||
|
||||
Centralize repeated parent event envelope mapping without changing the table
|
||||
contract.
|
||||
|
||||
Implementation requirements:
|
||||
|
||||
- Add small helpers inside `internal/sinks/postgres` for parent envelope values:
|
||||
`event_id`, `event_kind`, `event_source`, `event_schema`,
|
||||
`event_emitted_at`, and `event_effective_at`.
|
||||
- Use the helper in every parent table mapper that stores event envelope
|
||||
columns.
|
||||
- Keep explicit per-product mapper functions and product-specific validation.
|
||||
- Optionally centralize envelope column declarations only if `schema.go` remains
|
||||
easy to scan.
|
||||
- Preserve every table, column, nullability rule, required-field check, compact
|
||||
JSON behavior, UTC normalization, child positional index, and write count.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Mapper output for existing valid payloads is equivalent before and after the
|
||||
refactor.
|
||||
- Unsupported schemas still map to zero writes and no error.
|
||||
- Required-field failures still include useful product/path context.
|
||||
|
||||
Focused tests:
|
||||
|
||||
```sh
|
||||
go test ./internal/sinks/postgres
|
||||
```
|
||||
|
||||
## Stage 2: Consolidate Package-Local Fixture Helpers
|
||||
|
||||
Reduce low-value duplicated fixture-reading code only where it is local and
|
||||
obvious.
|
||||
|
||||
Implementation requirements:
|
||||
|
||||
- Consolidate duplicated fixture readers only within the same Go package.
|
||||
- Keep fixtures under each package's `testdata` directory.
|
||||
- Do not create a cross-package test utility package.
|
||||
- Do not change fixture contents unless an existing test already requires it.
|
||||
- Do not mix this cleanup with parser behavior changes.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Local test helper duplication is reduced where multiple files in one package
|
||||
already share the same fixture-reading behavior.
|
||||
- Tests remain easy to read locally.
|
||||
- No package imports a helper solely for tests from another package.
|
||||
|
||||
Focused tests:
|
||||
|
||||
```sh
|
||||
go test ./internal/providers/nws ./internal/providers/spc
|
||||
go test ./internal/sources/nws ./internal/sources/spc
|
||||
go test ./internal/normalizers/nws ./internal/normalizers/spc
|
||||
```
|
||||
|
||||
## Stage 3: Dead-Code And Literal Sweep
|
||||
|
||||
Perform a final cleanup sweep after mapper and fixture cleanup.
|
||||
|
||||
Implementation requirements:
|
||||
|
||||
- Search for stale internal driver, kind, and schema literals.
|
||||
- Replace internal code/test literals with constants where it reduces typo or
|
||||
drift risk.
|
||||
- Keep user-facing docs and YAML examples literal.
|
||||
- Keep intentional legacy-driver negative tests.
|
||||
- Do not remove compatibility tests unless they are clearly obsolete.
|
||||
- Do not broaden the cleanup into unrelated refactors.
|
||||
|
||||
Suggested searches:
|
||||
|
||||
```sh
|
||||
rg 'event\.Kind\("|nws_forecast|nws_weatherstories|openmeteo_|openweather_|spc_convective_outlook|weather_story|forecast_discussion|raw\.|weather\.' .
|
||||
rg 'TODO|legacy|deprecated|unknown source driver' internal cmd docs examples
|
||||
```
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Internal literals are reduced where constants already exist.
|
||||
- Intentional literals in docs, YAML examples, raw schema docs, and negative
|
||||
tests remain readable.
|
||||
- No behavior changes are introduced.
|
||||
|
||||
Focused tests:
|
||||
|
||||
```sh
|
||||
go test ./internal/sources ./internal/normalizers/... ./internal/sinks/postgres ./cmd/weatherfeeder
|
||||
```
|
||||
|
||||
## Final Verification
|
||||
|
||||
After all stages are complete, run:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
git status --short
|
||||
```
|
||||
|
||||
Verify current-behavior docs still describe implemented behavior only and no
|
||||
public contracts changed unintentionally.
|
||||
|
||||
## Refactors To Avoid
|
||||
|
||||
Do not perform these changes as part of this cleanup roadmap:
|
||||
|
||||
- Generic workflow or stage engine.
|
||||
- Runtime plugin system.
|
||||
- Weather-specific constants in `feedkit`.
|
||||
- Replacing feedkit scheduler, dispatch, HTTP helpers, or sink mechanics.
|
||||
- Generic source abstraction covering every source type.
|
||||
- Reflection-based or generated Postgres mapper.
|
||||
- ORM-style persistence layer.
|
||||
- Database column metadata on canonical model structs.
|
||||
- Cross-provider timestamp parser that hides provider-specific formats.
|
||||
- Broad WMO mapper consolidation beyond existing common text fallback.
|
||||
- Generated documentation system.
|
||||
Reference in New Issue
Block a user