Handle additional NWS forecast discussion formats
All checks were successful
ci/woodpecker/manual/build-image Pipeline was successful
All checks were successful
ci/woodpecker/manual/build-image Pipeline was successful
This commit is contained in:
@@ -1,207 +0,0 @@
|
||||
# NWS AFD Section Parsing Resilience
|
||||
|
||||
## Status
|
||||
|
||||
Implemented.
|
||||
|
||||
The original heading-variant work and both resilience follow-ups are
|
||||
implemented without changing the canonical forecast-discussion contract.
|
||||
|
||||
## Completed Baseline
|
||||
|
||||
The NWS Area Forecast Discussion parser currently separates:
|
||||
|
||||
1. generic structural heading recognition;
|
||||
2. one-pass section-boundary scanning;
|
||||
3. canonical section-role selection; and
|
||||
4. section-preamble and presentation cleanup.
|
||||
|
||||
It recognizes generic uppercase identities in ellipsis-first and
|
||||
slash-qualified headings, treats structurally recognized but unmapped sections
|
||||
as boundaries, supports same-line and next-line qualifiers, removes exact NWS
|
||||
change markers, and keeps the first occurrence of each mapped role. Provider and
|
||||
normalizer tests cover two office-format families while preserving the existing
|
||||
wire contract.
|
||||
|
||||
## Implemented Extensions
|
||||
|
||||
The parser now handles common AFD forms that were outside the completed
|
||||
baseline:
|
||||
|
||||
- a qualifier may appear between the identity and terminal ellipsis, as in
|
||||
`.DISCUSSION (Today through Thursday)...`;
|
||||
- the same key-message concept may be headed `KEY POINTS` rather than `KEY
|
||||
MESSAGES`;
|
||||
- key-message items may use hyphens, asterisks, numeric markers such as `1)` or
|
||||
`1.`, or unmarked paragraphs;
|
||||
- a bare prefix check for `Issued at` or `Updated at` can misclassify ordinary
|
||||
prose such as “Updated atmospheric conditions...” as metadata; and
|
||||
- offices may publish semantically distinct `NEAR TERM`, `SHORT TERM`, and
|
||||
`LONG TERM` sections, so wording flexibility cannot safely rely on treating
|
||||
every similar identity as an alias.
|
||||
|
||||
These provider-local extensions preserve the architecture while strengthening
|
||||
syntax, semantic-alias, and list-tokenization handling. Compact cross-office
|
||||
fixtures exercise each added format family.
|
||||
|
||||
## Implementation Approach
|
||||
|
||||
Common, minor NWS presentation changes remain local and inexpensive to support
|
||||
while strict structural recognition and the existing canonical schema are
|
||||
preserved. Each observed heading form uses an isolated grammar helper, a true
|
||||
synonym uses one role-registry entry, and each list marker uses one
|
||||
marker-classifier case. None requires changes to the scanner, canonical model,
|
||||
normalizer architecture, or downstream contracts.
|
||||
|
||||
## Implemented Behavior
|
||||
|
||||
### Heading syntax
|
||||
|
||||
The provider-local heading parser recognizes these three explicit families:
|
||||
|
||||
```text
|
||||
.<IDENTITY>...<optional qualifier>
|
||||
.<IDENTITY> (<nonempty qualifier>)...
|
||||
.<IDENTITY> /<nonempty qualifier>/...
|
||||
```
|
||||
|
||||
The existing identity grammar remains unchanged: uppercase ASCII letters,
|
||||
digits, horizontal whitespace, `/`, `&`, apostrophes, and hyphens, with at least
|
||||
one letter or digit. Identity whitespace is normalized to one ASCII space.
|
||||
|
||||
The parenthesized-terminal form:
|
||||
|
||||
- requires horizontal whitespace between the identity and opening `(`;
|
||||
- requires a nonempty standalone parenthetical qualifier immediately before
|
||||
the terminal ASCII `...`;
|
||||
- retains the outer parentheses in the parsed qualifier;
|
||||
- permits ordinary qualifier punctuation inside the parentheses; and
|
||||
- rejects trailing text after the terminal ellipsis.
|
||||
|
||||
Each family is parsed by a small, ordered helper. Slash-qualified parsing remains
|
||||
first because its terminal is otherwise ambiguous with the ellipsis-first form;
|
||||
parenthesized-terminal parsing runs second, followed by the existing
|
||||
ellipsis-first form. Malformed heading-like lines remain body content.
|
||||
|
||||
Every successfully parsed heading is a section boundary regardless of whether
|
||||
its identity has a canonical role. This includes `DISCUSSION`, `NEAR TERM`,
|
||||
aviation, marine, hydrology, office-specific sections, and other structurally
|
||||
compatible identities.
|
||||
|
||||
### Canonical role aliases
|
||||
|
||||
One provider-local identity-to-role registry remains the sole semantic mapping
|
||||
source. It maps:
|
||||
|
||||
- `KEY MESSAGES` and `KEY POINTS` to key messages;
|
||||
- `SHORT TERM` to the short-term section; and
|
||||
- `LONG TERM` to the long-term section.
|
||||
|
||||
Aliases are explicit and evidence-based; identity similarity is never inferred.
|
||||
The first encountered identity for a role wins, so `KEY MESSAGES` and `KEY
|
||||
POINTS` participate in the same first-occurrence policy.
|
||||
|
||||
`NEAR TERM` remains boundary-only. It is a distinct section that may coexist
|
||||
with `SHORT TERM`, so silently relabeling it would lose meaning. Exposing it
|
||||
requires a separate canonical schema roadmap. `DISCUSSION` and other unmapped
|
||||
sections likewise remain boundary-only.
|
||||
|
||||
### Key-message normalization
|
||||
|
||||
After exact presentation-marker removal and blank-line trimming, the parser
|
||||
removes at most one leading metadata line only when all of these conditions are
|
||||
true:
|
||||
|
||||
- the label is exactly `Issued at` or `Updated at`, compared ASCII
|
||||
case-insensitively;
|
||||
- the label is followed by horizontal whitespace; and
|
||||
- the remainder is a valid timestamp under the existing NWS issue-time grammar.
|
||||
|
||||
An invalid or merely prefix-matching line remains content. Metadata recognition
|
||||
does not consume later message prose.
|
||||
|
||||
Key-message item recognition supports:
|
||||
|
||||
- hyphen markers;
|
||||
- asterisk markers;
|
||||
- positive ASCII numeric markers followed by `)` or `.`, then either end of line
|
||||
or a horizontal-space boundary;
|
||||
- composite hyphen-or-asterisk plus numeric markers such as `- 1.`; and
|
||||
- unmarked, blank-line-separated paragraphs when the block contains no
|
||||
recognized list marker.
|
||||
|
||||
Marker text is removed from canonical messages. Wrapped nonempty lines remain
|
||||
continuations of the current item, and blank lines between marked items do not
|
||||
create empty messages. If marked and unmarked content are mixed, nonempty prose
|
||||
before the first marker is preserved as its own message and later unmarked lines
|
||||
continue the active marked item. The parser never interprets numeric or
|
||||
asterisk markers outside a mapped key-message block.
|
||||
|
||||
### Representative coverage
|
||||
|
||||
Tests combine small table-driven grammar and tokenizer cases with maintained
|
||||
local HTML fixtures for distinct real NWS format families. In addition to the
|
||||
existing LSX and BOU coverage, fixtures cover:
|
||||
|
||||
- numbered `KEY MESSAGES` followed by a boundary-only `DISCUSSION` section; and
|
||||
- asterisk `KEY POINTS` plus a parenthesized-terminal `DISCUSSION` heading.
|
||||
|
||||
Fixtures remain compact, deterministic, attributable in test comments to the
|
||||
format family they represent, and free of live network dependencies. Provider
|
||||
and normalizer tests prove canonical values, boundary isolation, envelope
|
||||
behavior, and unchanged JSON wire shape.
|
||||
|
||||
## Compatibility and Contracts
|
||||
|
||||
This remains a provider-parsing compatibility improvement. It does not change:
|
||||
|
||||
- event kinds or raw and canonical schema identifiers;
|
||||
- canonical models or JSON field names;
|
||||
- source configuration, URLs, or polling behavior;
|
||||
- event envelope or effective-time behavior;
|
||||
- Postgres tables or event-to-row mapping; or
|
||||
- downstream sink and consumer responsibilities.
|
||||
|
||||
The implementation remains confined to the NWS provider parser plus owning
|
||||
provider and normalizer tests. It uses the Go standard library and introduces no
|
||||
runtime configuration or general parser framework.
|
||||
|
||||
## Acceptance Coverage
|
||||
|
||||
Automated tests demonstrate that:
|
||||
|
||||
- all previously supported heading, scanning, preamble, marker, and canonical
|
||||
results remain compatible;
|
||||
- parenthesized-terminal headings are parsed with their qualifier and terminate
|
||||
preceding content;
|
||||
- malformed variants of that heading family remain body content;
|
||||
- `KEY POINTS` populates the existing key-message field and shares
|
||||
first-occurrence behavior with `KEY MESSAGES`;
|
||||
- `NEAR TERM` and `DISCUSSION` remain boundary-only and never populate short- or
|
||||
long-term fields;
|
||||
- hyphen, asterisk, `N)`, `N.`, and composite key-message lists produce distinct
|
||||
ordered messages with wrapped continuations;
|
||||
- unmarked key-message paragraphs produce distinct ordered messages;
|
||||
- valid leading issue/update metadata is removed, while prefix collisions,
|
||||
malformed timestamps, and later timestamp-like prose are retained;
|
||||
- representative additional office-format fixtures parse end to end through the
|
||||
provider and normalizer without adjacent-section leakage; and
|
||||
- focused tests, the full repository suite, static analysis, and diff checks
|
||||
pass with no public contract changes.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This follow-up does not:
|
||||
|
||||
- add canonical `nearTerm`, `discussion`, aviation, marine, hydrology, climate,
|
||||
fire-weather, update, or arbitrary-section fields;
|
||||
- map `NEAR TERM` to `SHORT TERM` or infer roles from similar words;
|
||||
- parse arbitrary lowercase or free-form prose as headings;
|
||||
- accept arbitrary punctuation as list markers;
|
||||
- remove malformed metadata-like prose;
|
||||
- introduce heuristic summarization or preserve complete raw AFD documents;
|
||||
- change schemas, persistence contracts, configuration, or downstream APIs; or
|
||||
- fetch live NWS data during tests.
|
||||
|
||||
Additional canonical support for AFD section identities remains a separate
|
||||
consumer and schema decision.
|
||||
@@ -4,6 +4,45 @@
|
||||
|
||||
This document is the catch-all roadmap for planned, deferred, aspirational, experimental, or unimplemented weatherfeeder work. Current behavior belongs in the canonical docs outside `docs/roadmap/`.
|
||||
|
||||
## NWS AFD Parsing Resilience
|
||||
|
||||
The current parser handles the concrete RAH, LWX, and MFR variants that
|
||||
motivated these ideas. Future work should keep those extension points
|
||||
maintainable as additional evidence appears.
|
||||
|
||||
### Generalize Wrapper-Scoped Embedded Sections
|
||||
|
||||
The scanner currently permits undotted nested headings only inside `PREV
|
||||
DISCUSSION`. If other wrapper identities are observed, replace the single
|
||||
wrapper check with a small explicit provider-local registry and add a fixture
|
||||
for each wrapper family. Do not make the leading dot globally optional: wrapper
|
||||
scope is the safeguard against classifying uppercase prose as a section.
|
||||
|
||||
### Extend Conservative Preamble Classification
|
||||
|
||||
Leading key-message metadata currently supports validated `Issued at`, `Updated
|
||||
at`, and `As of <clock> <weekday>...` forms. Add future wording variants as
|
||||
small, ordered classifiers with strict label boundaries and value grammars.
|
||||
Every addition should include collision tests proving that similar message prose
|
||||
and malformed metadata remain canonical content.
|
||||
|
||||
### Keep List-Marker Recognition Extensible
|
||||
|
||||
The marker parser currently supports hyphens, asterisks, `N)`, `N.`, `(N)`, and
|
||||
composite forms such as `- (N)`. If new decorators appear, evolve the helper
|
||||
toward an explicit marker grammar or typed classification result rather than a
|
||||
broad punctuation heuristic. Preserve positive-number and whitespace-boundary
|
||||
checks so ordinary prose is not stripped.
|
||||
|
||||
### Maintain a Cross-Office Fixture Corpus
|
||||
|
||||
The compact RAH, LWX, and current MFR fixtures seed regression coverage for the
|
||||
observed layouts. Future parser changes should add concise, deterministic HTML
|
||||
fixtures for materially distinct office formats and exercise them through both
|
||||
the provider parser and normalizer. Fixture comments should identify the format
|
||||
family and state that edited prose is not an archived product; tests must remain
|
||||
offline and assert both intended extraction and adjacent-section isolation.
|
||||
|
||||
## SPC Convective Outlook Follow-Ups
|
||||
|
||||
### Weatherapi Outlook Endpoints
|
||||
|
||||
@@ -1,332 +0,0 @@
|
||||
# NWS AFD Section Parsing Resilience Implementation Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
All resilience work defined in
|
||||
[`afd-section-heading-variants.md`](afd-section-heading-variants.md) is
|
||||
complete without changing the canonical forecast-discussion contract. This plan
|
||||
preserves the implementation details as a historical record.
|
||||
|
||||
## Cross-Stage Constraints
|
||||
|
||||
- Keep production parsing changes in
|
||||
`internal/providers/nws/forecast_discussion.go`.
|
||||
- Use only the Go standard library and keep new helpers unexported.
|
||||
- Do not change `model`, `standards`, source configuration or polling, schema
|
||||
identifiers, event-envelope behavior, Postgres mapping, sinks, consumer docs,
|
||||
or integration contracts.
|
||||
- Continue exposing only key messages, short term, and long term.
|
||||
- Map `KEY POINTS` to the existing key-message role. Keep `NEAR TERM`,
|
||||
`DISCUSSION`, and every other unmapped identity boundary-only.
|
||||
- Preserve first-occurrence behavior by canonical role, including across `KEY
|
||||
MESSAGES` and `KEY POINTS` aliases.
|
||||
- Preserve raw body lines during structural scanning. Apply provider-specific
|
||||
cleanup only while parsing a mapped block.
|
||||
- Prefer small grammar, label, and list-marker helpers over a broad regular
|
||||
expression or general parsing framework.
|
||||
- Keep parsing deterministic. Tests must use local strings and fixtures, never
|
||||
live NWS requests.
|
||||
- Preserve all pre-existing user work and avoid unrelated refactors.
|
||||
|
||||
## Stage 1: Centralize Known Heading Parsing — Completed
|
||||
|
||||
The provider parser gained a shared heading classifier and section-block model
|
||||
for the original key-message, short-term, long-term, and aviation identities.
|
||||
Discovery, boundary handling, and qualifier extraction stopped using separate
|
||||
per-section patterns.
|
||||
|
||||
## Stage 2: Add Original Heading-Variant Regressions — Completed
|
||||
|
||||
Provider and normalizer tests covered ellipsis-first and slash-qualified
|
||||
headings, malformed forms, empty bodies, mixed heading families, and unchanged
|
||||
canonical wire shape.
|
||||
|
||||
## Stage 3: Validate the Original Feature — Completed
|
||||
|
||||
Focused and repository-wide checks passed, production changes remained inside
|
||||
the NWS provider parser, and the original roadmap was reconciled with the
|
||||
implemented baseline.
|
||||
|
||||
## Stage 4: Generalize Structural Heading Recognition — Completed
|
||||
|
||||
Heading syntax was decoupled from canonical roles. The parser now accepts a
|
||||
constrained generic uppercase identity grammar, normalizes identity whitespace,
|
||||
supports embedded slashes without confusing slash qualifiers, and recognizes
|
||||
unknown valid identities as structural headings.
|
||||
|
||||
## Stage 5: Scan Ordered Section Blocks Once — Completed
|
||||
|
||||
Repeated extraction was replaced by a one-pass ordered block scanner. Generic
|
||||
headings, `&&`, `$$`, and the watch/advisory safeguard terminate active blocks;
|
||||
role lookup happens after scanning and retains first-occurrence behavior.
|
||||
|
||||
## Stage 6: Normalize Multiline Preambles and Markers — Completed
|
||||
|
||||
Short- and long-term parsing gained next-line parenthesized qualifiers,
|
||||
case-insensitive `Issued at` handling, and exact change-marker removal. Key
|
||||
messages gained exact marker cleanup and removal of one leading issue/update
|
||||
metadata line.
|
||||
|
||||
## Stage 7: Add Initial Cross-Office Coverage — Completed
|
||||
|
||||
The existing LSX fixture was retained and a compact BOU-style fixture added.
|
||||
Provider and normalizer regressions cover multiline qualifiers, change markers,
|
||||
leading metadata, generic boundary isolation, envelope behavior, and unchanged
|
||||
JSON shape.
|
||||
|
||||
## Stage 8: Validate the First Resilience Follow-up — Completed
|
||||
|
||||
Focused tests, the full repository suite, static analysis, and diff checks
|
||||
passed. The public model and schemas remained unchanged, and the roadmap was
|
||||
updated to describe the completed behavior.
|
||||
|
||||
## Stage 9: Add Parenthesized-Terminal Heading Syntax — Completed
|
||||
|
||||
Recognize the observed `.<IDENTITY> (<qualifier>)...` family without weakening
|
||||
generic identity validation.
|
||||
|
||||
1. In `internal/providers/nws/forecast_discussion.go`, add a dedicated
|
||||
`parseForecastDiscussionParenthesizedTerminalHeading` helper returning the
|
||||
existing `forecastDiscussionSectionHeading` type.
|
||||
2. Update `parseForecastDiscussionSectionHeading` to try forms in this order:
|
||||
|
||||
1. if the line ends `/...`, try the slash-qualified helper and return its
|
||||
result;
|
||||
2. otherwise, try the parenthesized-terminal helper for any line ending
|
||||
`...` and return it when it succeeds; and
|
||||
3. fall back to the existing ellipsis-first helper.
|
||||
|
||||
Do not merge the forms into a single regular expression.
|
||||
3. Implement the parenthesized-terminal helper with these exact rules:
|
||||
|
||||
- operate on the already outer-trimmed line and require leading `.` plus a
|
||||
terminal ASCII `...`;
|
||||
- remove the leading dot and terminal ellipsis, then remove horizontal
|
||||
whitespace immediately before the ellipsis;
|
||||
- require the remaining content to end with `)`;
|
||||
- locate the first `(` that is preceded by horizontal whitespace; everything
|
||||
before that separator is the raw identity and everything from `(` through
|
||||
the final `)` is the qualifier;
|
||||
- normalize and validate the identity through the existing identity helper;
|
||||
- require nonempty text after trimming inside the outer parentheses;
|
||||
- return the qualifier with its outer parentheses and internal punctuation
|
||||
intact; and
|
||||
- reject missing separator whitespace, empty qualifiers, missing or misplaced
|
||||
parentheses, trailing text after the ellipsis, unsupported identity
|
||||
punctuation, and lowercase or mixed-case identities.
|
||||
|
||||
Parentheses inside the qualifier are content; only the first separator `(`
|
||||
and final `)` delimit the outer qualifier.
|
||||
4. Extend the heading table in
|
||||
`internal/providers/nws/forecast_discussion_test.go` with:
|
||||
|
||||
- `.DISCUSSION (Today through Thursday)...`;
|
||||
- a mapped identity such as `.SHORT TERM (Tonight)...`;
|
||||
- identity whitespace normalization;
|
||||
- qualifier punctuation and nested parentheses;
|
||||
- each rejected malformed form listed above; and
|
||||
- compatibility assertions for every existing ellipsis-first and
|
||||
slash-qualified case.
|
||||
5. Add a scanner regression in which a mapped section is followed immediately,
|
||||
without `&&`, by `.DISCUSSION (Today through Thursday)...`. Assert that the
|
||||
discussion heading starts a new boundary-only block and its prose cannot leak
|
||||
into the mapped section.
|
||||
|
||||
Run and pass:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
|
||||
go test -count=1 ./internal/providers/nws
|
||||
```
|
||||
|
||||
Do not proceed until all previous heading and scanner tests still pass.
|
||||
|
||||
## Stage 10: Add Explicit Canonical Role Aliases — Completed
|
||||
|
||||
Make true wording synonyms cheap to support while preserving semantic
|
||||
distinctions.
|
||||
|
||||
1. Add `KEY POINTS` to the existing identity-to-role registry with the key-message
|
||||
role. Keep the registry as the only production source of identity-to-role
|
||||
mappings; do not add a parallel alias collection or role-selection switch.
|
||||
2. Leave `KEY MESSAGES`, `SHORT TERM`, and `LONG TERM` mappings unchanged.
|
||||
Explicitly do not add role mappings for `NEAR TERM`, `DISCUSSION`,
|
||||
`AVIATION`, or other generic identities.
|
||||
3. Update registry tests so they no longer assume one identity per role. Assert
|
||||
instead that:
|
||||
|
||||
- every registry identity parses as a heading;
|
||||
- `KEY MESSAGES` and `KEY POINTS` both map to the key-message role;
|
||||
- `SHORT TERM` and `LONG TERM` retain their roles; and
|
||||
- `NEAR TERM`, `DISCUSSION`, and `AVIATION` have no role entry.
|
||||
4. Add `ParseForecastDiscussionText` tests for role-level first occurrence:
|
||||
|
||||
- `KEY POINTS` alone populates key messages;
|
||||
- `KEY POINTS` followed by `KEY MESSAGES` keeps the first block; and
|
||||
- `KEY MESSAGES` followed by `KEY POINTS` keeps the first block.
|
||||
|
||||
Use hyphen messages in this stage so list-tokenization changes remain scoped
|
||||
to Stage 11.
|
||||
5. Add a regression containing `NEAR TERM`, `SHORT TERM`, and `LONG TERM` in one
|
||||
bulletin. Assert that near-term prose terminates adjacent blocks but does not
|
||||
populate or override either canonical text section.
|
||||
|
||||
Run and pass:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
|
||||
go test -count=1 ./internal/providers/nws
|
||||
```
|
||||
|
||||
Do not change the canonical model to expose near-term or discussion content.
|
||||
|
||||
## Stage 11: Harden Key-Message Metadata and Item Parsing — Completed
|
||||
|
||||
Replace prefix-sensitive, hyphen-only parsing with conservative metadata and
|
||||
list classifiers.
|
||||
|
||||
1. Add a helper that recognizes a leading labeled metadata line only when:
|
||||
|
||||
- its trimmed text begins with exactly `Issued at` or `Updated at`, compared
|
||||
ASCII case-insensitively;
|
||||
- at least one horizontal-whitespace byte follows the label; and
|
||||
- parsing the remaining text through the existing unlabeled NWS issue-time
|
||||
grammar succeeds.
|
||||
|
||||
Reuse `parseForecastDiscussionIssueTime` for the timestamp grammar after
|
||||
removing the label. Do not add a second timestamp parser. A label prefix with
|
||||
no boundary or with an invalid timestamp returns false and remains content.
|
||||
2. Update key-message cleanup to remove at most one such valid leading metadata
|
||||
line after presentation-marker removal and blank-line trimming. Later valid
|
||||
timestamp lines remain message content.
|
||||
3. Replace the hyphen-only check with a helper that classifies and strips one of
|
||||
these markers from a trimmed line:
|
||||
|
||||
- one leading `-`, preserving current compatibility whether or not whitespace
|
||||
follows it;
|
||||
- one leading `*`, whether or not whitespace follows it; or
|
||||
- one or more ASCII digits forming an integer greater than zero, followed by
|
||||
`)` or `.`, and then either end-of-line or horizontal whitespace.
|
||||
|
||||
After recognizing and stripping a leading hyphen or asterisk plus horizontal
|
||||
whitespace, also strip one immediately following valid numeric marker. This
|
||||
supports composite forms such as `- 1.` without retaining either decorator.
|
||||
Otherwise strip only the recognized marker and following horizontal
|
||||
whitespace. Reject zero, overflow, alphanumeric prefixes, and numeric
|
||||
punctuation without the required boundary. Do not interpret these markers
|
||||
outside key-message parsing.
|
||||
4. Parse a cleaned key-message body deterministically:
|
||||
|
||||
- first determine whether any nonempty line has a recognized marker;
|
||||
- when markers exist, each marker starts a new message; nonempty unmarked
|
||||
lines after a marker continue that message; blank lines after the first
|
||||
marker are ignored; and blank-line-separated prose before the first marker
|
||||
is flushed as preserved message content;
|
||||
- when no marker exists, each nonempty paragraph separated by one or more
|
||||
blank lines becomes one message; and
|
||||
- in both modes, join wrapped lines with one ASCII space, discard empty
|
||||
messages, and preserve source order.
|
||||
5. Add table-driven marker tests for `-`, `*`, `1)`, `2.`, composite `- 1.` and
|
||||
`* 1)`, multi-digit values, wrapped continuations, marker-only lines, and
|
||||
rejected zero/malformed numeric prefixes.
|
||||
6. Add block-level tests covering:
|
||||
|
||||
- numbered and asterisk lists producing distinct messages;
|
||||
- unmarked paragraphs producing distinct messages;
|
||||
- mixed introductory prose and marked items without data loss;
|
||||
- valid leading `Issued at` and `Updated at` metadata removal;
|
||||
- `Updated atmospheric conditions...` remaining content;
|
||||
- `Updated at not a timestamp` remaining content;
|
||||
- a later valid timestamp-like line remaining content; and
|
||||
- all existing hyphen, change-marker, and continuation behavior.
|
||||
|
||||
Run and pass:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
|
||||
go test -count=1 ./internal/providers/nws
|
||||
```
|
||||
|
||||
## Stage 12: Add Representative Numbered and Key-Points Fixtures — Completed
|
||||
|
||||
Prove the new extension points through provider and normalizer boundaries.
|
||||
|
||||
1. Retain the existing LSX and BOU fixtures and their regressions.
|
||||
2. Add two compact HTML fixtures under `internal/providers/nws/testdata/`:
|
||||
|
||||
- a BGM/CTP-style fixture with a valid header and issue time, numbered `KEY
|
||||
MESSAGES`, wrapped item lines, and a boundary-only `DISCUSSION` section; and
|
||||
- an MFR-style fixture with a valid header and issue time, asterisk `KEY
|
||||
POINTS`, and an immediately following
|
||||
`.DISCUSSION (Today through Thursday)...` section without an intervening
|
||||
`&&`.
|
||||
|
||||
Use concise representative text rather than full web pages. Add a short HTML
|
||||
comment to each fixture naming the real office-format family it represents;
|
||||
do not claim that edited fixture prose is a verbatim archived product.
|
||||
3. Add provider end-to-end tests that assert:
|
||||
|
||||
- exact ordered key messages and joined continuations;
|
||||
- `KEY POINTS` alias mapping;
|
||||
- correct office and top-level issue metadata;
|
||||
- absence of numeric/asterisk markers from canonical values;
|
||||
- absence of `DISCUSSION` headings, qualifiers, and prose from key messages;
|
||||
and
|
||||
- nil short- and long-term fields when the fixture contains neither mapped
|
||||
role.
|
||||
4. Add table-driven normalizer regressions using both fixtures. For each, verify
|
||||
input envelope preservation, event kind, canonical schema, effective time,
|
||||
exact key-message mapping, and successful JSON marshaling.
|
||||
5. Assert the JSON payload still has no `nearTerm`, `discussion`, `aviation`, or
|
||||
generic `sections` field. Do not add those fields to provider or canonical
|
||||
structs.
|
||||
6. Keep syntax and tokenizer edge cases in focused unit tables; do not add more
|
||||
full fixtures for cases already proven locally.
|
||||
|
||||
Run and pass:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/providers/nws/forecast_discussion_test.go internal/normalizers/nws/forecast_discussion_test.go
|
||||
go test -count=1 ./internal/providers/nws ./internal/normalizers/nws
|
||||
```
|
||||
|
||||
## Stage 13: Reconcile Documentation and Perform Final Validation — Completed
|
||||
|
||||
Close the second resilience follow-up only after all behavior is implemented and
|
||||
verified.
|
||||
|
||||
1. Review the final diff. Production changes must remain confined to
|
||||
`internal/providers/nws/forecast_discussion.go`; other Go changes must be
|
||||
owning provider and normalizer tests. Preserve unrelated user work.
|
||||
2. Confirm the public contract is unchanged. If implementation appears to
|
||||
require a canonical near-term/discussion field, schema change, configuration,
|
||||
or downstream migration, stop and report the conflict rather than expanding
|
||||
scope.
|
||||
3. Run:
|
||||
|
||||
```sh
|
||||
gofmt -w \
|
||||
internal/providers/nws/forecast_discussion.go \
|
||||
internal/providers/nws/forecast_discussion_test.go \
|
||||
internal/normalizers/nws/forecast_discussion_test.go
|
||||
go test -count=1 ./internal/providers/nws ./internal/normalizers/nws
|
||||
go test -count=1 ./...
|
||||
go vet ./...
|
||||
git diff --check
|
||||
```
|
||||
4. Verify every acceptance criterion in the feature roadmap has a corresponding
|
||||
passing test, including compatibility with Stages 1-8.
|
||||
5. Update the feature roadmap status to `Implemented.` and rewrite its remaining
|
||||
proposed or future-tense language as implemented behavior only after all
|
||||
checks pass.
|
||||
6. Update this implementation plan so Stages 9-13 are marked `— Completed` and
|
||||
the Purpose states that all stages are complete. Preserve the stage details
|
||||
as a historical implementation record.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The plan makes the required policy choices explicitly: `KEY POINTS` is a
|
||||
true key-message alias; `NEAR TERM` remains semantically distinct and
|
||||
boundary-only; parenthesized-terminal syntax is a third constrained grammar;
|
||||
metadata is removed only after timestamp validation; and key-message variants
|
||||
are handled by an isolated marker classifier plus paragraph fallback.
|
||||
Reference in New Issue
Block a user