Eric Rakestraw
8041f99782
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
147 lines
5.0 KiB
Markdown
147 lines
5.0 KiB
Markdown
# Cleanup Roadmap
|
|
|
|
## Summary
|
|
|
|
This roadmap tracks remaining behavior-preserving cleanup opportunities for
|
|
`weatherfeeder`. Earlier cleanup stages for event kind constants, source driver
|
|
constants, registry-derived source tests, shared multi-document HTTP config, and
|
|
shallow docs consistency tests have been completed and are no longer listed as
|
|
future work.
|
|
|
|
The remaining work should be implemented only in small, focused changes that do
|
|
not alter public event schemas, event kinds, source driver names, config keys,
|
|
Postgres table contracts, or canonical JSON field names.
|
|
|
|
## Guardrails
|
|
|
|
- Keep `feedkit` as the generic daemon infrastructure boundary.
|
|
- Keep weather-domain behavior in sources, provider helpers, normalizers,
|
|
`model`, `standards`, and Postgres mapping.
|
|
- Keep current-behavior docs accurate and roadmap-only plans under
|
|
`docs/roadmap/`.
|
|
- Do not introduce plugin systems, workflow engines, generic source frameworks,
|
|
ORM-style mapping, reflection mapping, generated docs, or persistence tags on
|
|
canonical model structs.
|
|
- Run focused tests after each stage and `go test ./...` after all stages.
|
|
|
|
## Stage 1: Reduce Postgres Mapper Envelope Duplication
|
|
|
|
Centralize repeated parent event envelope mapping without changing the table
|
|
contract.
|
|
|
|
Implementation requirements:
|
|
|
|
- Add small helpers inside `internal/sinks/postgres` for parent envelope values:
|
|
`event_id`, `event_kind`, `event_source`, `event_schema`,
|
|
`event_emitted_at`, and `event_effective_at`.
|
|
- Use the helper in every parent table mapper that stores event envelope
|
|
columns.
|
|
- Keep explicit per-product mapper functions and product-specific validation.
|
|
- Optionally centralize envelope column declarations only if `schema.go` remains
|
|
easy to scan.
|
|
- Preserve every table, column, nullability rule, required-field check, compact
|
|
JSON behavior, UTC normalization, child positional index, and write count.
|
|
|
|
Acceptance criteria:
|
|
|
|
- Mapper output for existing valid payloads is equivalent before and after the
|
|
refactor.
|
|
- Unsupported schemas still map to zero writes and no error.
|
|
- Required-field failures still include useful product/path context.
|
|
|
|
Focused tests:
|
|
|
|
```sh
|
|
go test ./internal/sinks/postgres
|
|
```
|
|
|
|
## Stage 2: Consolidate Package-Local Fixture Helpers
|
|
|
|
Reduce low-value duplicated fixture-reading code only where it is local and
|
|
obvious.
|
|
|
|
Implementation requirements:
|
|
|
|
- Consolidate duplicated fixture readers only within the same Go package.
|
|
- Keep fixtures under each package's `testdata` directory.
|
|
- Do not create a cross-package test utility package.
|
|
- Do not change fixture contents unless an existing test already requires it.
|
|
- Do not mix this cleanup with parser behavior changes.
|
|
|
|
Acceptance criteria:
|
|
|
|
- Local test helper duplication is reduced where multiple files in one package
|
|
already share the same fixture-reading behavior.
|
|
- Tests remain easy to read locally.
|
|
- No package imports a helper solely for tests from another package.
|
|
|
|
Focused tests:
|
|
|
|
```sh
|
|
go test ./internal/providers/nws ./internal/providers/spc
|
|
go test ./internal/sources/nws ./internal/sources/spc
|
|
go test ./internal/normalizers/nws ./internal/normalizers/spc
|
|
```
|
|
|
|
## Stage 3: Dead-Code And Literal Sweep
|
|
|
|
Perform a final cleanup sweep after mapper and fixture cleanup.
|
|
|
|
Implementation requirements:
|
|
|
|
- Search for stale internal driver, kind, and schema literals.
|
|
- Replace internal code/test literals with constants where it reduces typo or
|
|
drift risk.
|
|
- Keep user-facing docs and YAML examples literal.
|
|
- Keep intentional legacy-driver negative tests.
|
|
- Do not remove compatibility tests unless they are clearly obsolete.
|
|
- Do not broaden the cleanup into unrelated refactors.
|
|
|
|
Suggested searches:
|
|
|
|
```sh
|
|
rg 'event\.Kind\("|nws_forecast|nws_weatherstories|openmeteo_|openweather_|spc_convective_outlook|weather_story|forecast_discussion|raw\.|weather\.' .
|
|
rg 'TODO|legacy|deprecated|unknown source driver' internal cmd docs examples
|
|
```
|
|
|
|
Acceptance criteria:
|
|
|
|
- Internal literals are reduced where constants already exist.
|
|
- Intentional literals in docs, YAML examples, raw schema docs, and negative
|
|
tests remain readable.
|
|
- No behavior changes are introduced.
|
|
|
|
Focused tests:
|
|
|
|
```sh
|
|
go test ./internal/sources ./internal/normalizers/... ./internal/sinks/postgres ./cmd/weatherfeeder
|
|
```
|
|
|
|
## Final Verification
|
|
|
|
After all stages are complete, run:
|
|
|
|
```sh
|
|
go test ./...
|
|
git status --short
|
|
```
|
|
|
|
Verify current-behavior docs still describe implemented behavior only and no
|
|
public contracts changed unintentionally.
|
|
|
|
## Refactors To Avoid
|
|
|
|
Do not perform these changes as part of this cleanup roadmap:
|
|
|
|
- Generic workflow or stage engine.
|
|
- Runtime plugin system.
|
|
- Weather-specific constants in `feedkit`.
|
|
- Replacing feedkit scheduler, dispatch, HTTP helpers, or sink mechanics.
|
|
- Generic source abstraction covering every source type.
|
|
- Reflection-based or generated Postgres mapper.
|
|
- ORM-style persistence layer.
|
|
- Database column metadata on canonical model structs.
|
|
- Cross-provider timestamp parser that hides provider-specific formats.
|
|
- Broad WMO mapper consolidation beyond existing common text fallback.
|
|
- Generated documentation system.
|