diff --git a/docs/roadmap/afd-section-heading-variants.md b/docs/roadmap/afd-section-heading-variants.md deleted file mode 100644 index effaa6c..0000000 --- a/docs/roadmap/afd-section-heading-variants.md +++ /dev/null @@ -1,207 +0,0 @@ -# NWS AFD Section Parsing Resilience - -## Status - -Implemented. - -The original heading-variant work and both resilience follow-ups are -implemented without changing the canonical forecast-discussion contract. - -## Completed Baseline - -The NWS Area Forecast Discussion parser currently separates: - -1. generic structural heading recognition; -2. one-pass section-boundary scanning; -3. canonical section-role selection; and -4. section-preamble and presentation cleanup. - -It recognizes generic uppercase identities in ellipsis-first and -slash-qualified headings, treats structurally recognized but unmapped sections -as boundaries, supports same-line and next-line qualifiers, removes exact NWS -change markers, and keeps the first occurrence of each mapped role. Provider and -normalizer tests cover two office-format families while preserving the existing -wire contract. - -## Implemented Extensions - -The parser now handles common AFD forms that were outside the completed -baseline: - -- a qualifier may appear between the identity and terminal ellipsis, as in - `.DISCUSSION (Today through Thursday)...`; -- the same key-message concept may be headed `KEY POINTS` rather than `KEY - MESSAGES`; -- key-message items may use hyphens, asterisks, numeric markers such as `1)` or - `1.`, or unmarked paragraphs; -- a bare prefix check for `Issued at` or `Updated at` can misclassify ordinary - prose such as “Updated atmospheric conditions...” as metadata; and -- offices may publish semantically distinct `NEAR TERM`, `SHORT TERM`, and - `LONG TERM` sections, so wording flexibility cannot safely rely on treating - every similar identity as an alias. - -These provider-local extensions preserve the architecture while strengthening -syntax, semantic-alias, and list-tokenization handling. Compact cross-office -fixtures exercise each added format family. - -## Implementation Approach - -Common, minor NWS presentation changes remain local and inexpensive to support -while strict structural recognition and the existing canonical schema are -preserved. Each observed heading form uses an isolated grammar helper, a true -synonym uses one role-registry entry, and each list marker uses one -marker-classifier case. None requires changes to the scanner, canonical model, -normalizer architecture, or downstream contracts. - -## Implemented Behavior - -### Heading syntax - -The provider-local heading parser recognizes these three explicit families: - -```text -.... -. ()... -. //... -``` - -The existing identity grammar remains unchanged: uppercase ASCII letters, -digits, horizontal whitespace, `/`, `&`, apostrophes, and hyphens, with at least -one letter or digit. Identity whitespace is normalized to one ASCII space. - -The parenthesized-terminal form: - -- requires horizontal whitespace between the identity and opening `(`; -- requires a nonempty standalone parenthetical qualifier immediately before - the terminal ASCII `...`; -- retains the outer parentheses in the parsed qualifier; -- permits ordinary qualifier punctuation inside the parentheses; and -- rejects trailing text after the terminal ellipsis. - -Each family is parsed by a small, ordered helper. Slash-qualified parsing remains -first because its terminal is otherwise ambiguous with the ellipsis-first form; -parenthesized-terminal parsing runs second, followed by the existing -ellipsis-first form. Malformed heading-like lines remain body content. - -Every successfully parsed heading is a section boundary regardless of whether -its identity has a canonical role. This includes `DISCUSSION`, `NEAR TERM`, -aviation, marine, hydrology, office-specific sections, and other structurally -compatible identities. - -### Canonical role aliases - -One provider-local identity-to-role registry remains the sole semantic mapping -source. It maps: - -- `KEY MESSAGES` and `KEY POINTS` to key messages; -- `SHORT TERM` to the short-term section; and -- `LONG TERM` to the long-term section. - -Aliases are explicit and evidence-based; identity similarity is never inferred. -The first encountered identity for a role wins, so `KEY MESSAGES` and `KEY -POINTS` participate in the same first-occurrence policy. - -`NEAR TERM` remains boundary-only. It is a distinct section that may coexist -with `SHORT TERM`, so silently relabeling it would lose meaning. Exposing it -requires a separate canonical schema roadmap. `DISCUSSION` and other unmapped -sections likewise remain boundary-only. - -### Key-message normalization - -After exact presentation-marker removal and blank-line trimming, the parser -removes at most one leading metadata line only when all of these conditions are -true: - -- the label is exactly `Issued at` or `Updated at`, compared ASCII - case-insensitively; -- the label is followed by horizontal whitespace; and -- the remainder is a valid timestamp under the existing NWS issue-time grammar. - -An invalid or merely prefix-matching line remains content. Metadata recognition -does not consume later message prose. - -Key-message item recognition supports: - -- hyphen markers; -- asterisk markers; -- positive ASCII numeric markers followed by `)` or `.`, then either end of line - or a horizontal-space boundary; -- composite hyphen-or-asterisk plus numeric markers such as `- 1.`; and -- unmarked, blank-line-separated paragraphs when the block contains no - recognized list marker. - -Marker text is removed from canonical messages. Wrapped nonempty lines remain -continuations of the current item, and blank lines between marked items do not -create empty messages. If marked and unmarked content are mixed, nonempty prose -before the first marker is preserved as its own message and later unmarked lines -continue the active marked item. The parser never interprets numeric or -asterisk markers outside a mapped key-message block. - -### Representative coverage - -Tests combine small table-driven grammar and tokenizer cases with maintained -local HTML fixtures for distinct real NWS format families. In addition to the -existing LSX and BOU coverage, fixtures cover: - -- numbered `KEY MESSAGES` followed by a boundary-only `DISCUSSION` section; and -- asterisk `KEY POINTS` plus a parenthesized-terminal `DISCUSSION` heading. - -Fixtures remain compact, deterministic, attributable in test comments to the -format family they represent, and free of live network dependencies. Provider -and normalizer tests prove canonical values, boundary isolation, envelope -behavior, and unchanged JSON wire shape. - -## Compatibility and Contracts - -This remains a provider-parsing compatibility improvement. It does not change: - -- event kinds or raw and canonical schema identifiers; -- canonical models or JSON field names; -- source configuration, URLs, or polling behavior; -- event envelope or effective-time behavior; -- Postgres tables or event-to-row mapping; or -- downstream sink and consumer responsibilities. - -The implementation remains confined to the NWS provider parser plus owning -provider and normalizer tests. It uses the Go standard library and introduces no -runtime configuration or general parser framework. - -## Acceptance Coverage - -Automated tests demonstrate that: - -- all previously supported heading, scanning, preamble, marker, and canonical - results remain compatible; -- parenthesized-terminal headings are parsed with their qualifier and terminate - preceding content; -- malformed variants of that heading family remain body content; -- `KEY POINTS` populates the existing key-message field and shares - first-occurrence behavior with `KEY MESSAGES`; -- `NEAR TERM` and `DISCUSSION` remain boundary-only and never populate short- or - long-term fields; -- hyphen, asterisk, `N)`, `N.`, and composite key-message lists produce distinct - ordered messages with wrapped continuations; -- unmarked key-message paragraphs produce distinct ordered messages; -- valid leading issue/update metadata is removed, while prefix collisions, - malformed timestamps, and later timestamp-like prose are retained; -- representative additional office-format fixtures parse end to end through the - provider and normalizer without adjacent-section leakage; and -- focused tests, the full repository suite, static analysis, and diff checks - pass with no public contract changes. - -## Non-Goals - -This follow-up does not: - -- add canonical `nearTerm`, `discussion`, aviation, marine, hydrology, climate, - fire-weather, update, or arbitrary-section fields; -- map `NEAR TERM` to `SHORT TERM` or infer roles from similar words; -- parse arbitrary lowercase or free-form prose as headings; -- accept arbitrary punctuation as list markers; -- remove malformed metadata-like prose; -- introduce heuristic summarization or preserve complete raw AFD documents; -- change schemas, persistence contracts, configuration, or downstream APIs; or -- fetch live NWS data during tests. - -Additional canonical support for AFD section identities remains a separate -consumer and schema decision. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index eda9c42..3d5a65b 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -4,6 +4,45 @@ This document is the catch-all roadmap for planned, deferred, aspirational, experimental, or unimplemented weatherfeeder work. Current behavior belongs in the canonical docs outside `docs/roadmap/`. +## NWS AFD Parsing Resilience + +The current parser handles the concrete RAH, LWX, and MFR variants that +motivated these ideas. Future work should keep those extension points +maintainable as additional evidence appears. + +### Generalize Wrapper-Scoped Embedded Sections + +The scanner currently permits undotted nested headings only inside `PREV +DISCUSSION`. If other wrapper identities are observed, replace the single +wrapper check with a small explicit provider-local registry and add a fixture +for each wrapper family. Do not make the leading dot globally optional: wrapper +scope is the safeguard against classifying uppercase prose as a section. + +### Extend Conservative Preamble Classification + +Leading key-message metadata currently supports validated `Issued at`, `Updated +at`, and `As of ...` forms. Add future wording variants as +small, ordered classifiers with strict label boundaries and value grammars. +Every addition should include collision tests proving that similar message prose +and malformed metadata remain canonical content. + +### Keep List-Marker Recognition Extensible + +The marker parser currently supports hyphens, asterisks, `N)`, `N.`, `(N)`, and +composite forms such as `- (N)`. If new decorators appear, evolve the helper +toward an explicit marker grammar or typed classification result rather than a +broad punctuation heuristic. Preserve positive-number and whitespace-boundary +checks so ordinary prose is not stripped. + +### Maintain a Cross-Office Fixture Corpus + +The compact RAH, LWX, and current MFR fixtures seed regression coverage for the +observed layouts. Future parser changes should add concise, deterministic HTML +fixtures for materially distinct office formats and exercise them through both +the provider parser and normalizer. Fixture comments should identify the format +family and state that edited prose is not an archived product; tests must remain +offline and assert both intended extraction and adjacent-section isolation. + ## SPC Convective Outlook Follow-Ups ### Weatherapi Outlook Endpoints diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 0bdbe34..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,332 +0,0 @@ -# NWS AFD Section Parsing Resilience Implementation Plan - -## Purpose - -All resilience work defined in -[`afd-section-heading-variants.md`](afd-section-heading-variants.md) is -complete without changing the canonical forecast-discussion contract. This plan -preserves the implementation details as a historical record. - -## Cross-Stage Constraints - -- Keep production parsing changes in - `internal/providers/nws/forecast_discussion.go`. -- Use only the Go standard library and keep new helpers unexported. -- Do not change `model`, `standards`, source configuration or polling, schema - identifiers, event-envelope behavior, Postgres mapping, sinks, consumer docs, - or integration contracts. -- Continue exposing only key messages, short term, and long term. -- Map `KEY POINTS` to the existing key-message role. Keep `NEAR TERM`, - `DISCUSSION`, and every other unmapped identity boundary-only. -- Preserve first-occurrence behavior by canonical role, including across `KEY - MESSAGES` and `KEY POINTS` aliases. -- Preserve raw body lines during structural scanning. Apply provider-specific - cleanup only while parsing a mapped block. -- Prefer small grammar, label, and list-marker helpers over a broad regular - expression or general parsing framework. -- Keep parsing deterministic. Tests must use local strings and fixtures, never - live NWS requests. -- Preserve all pre-existing user work and avoid unrelated refactors. - -## Stage 1: Centralize Known Heading Parsing — Completed - -The provider parser gained a shared heading classifier and section-block model -for the original key-message, short-term, long-term, and aviation identities. -Discovery, boundary handling, and qualifier extraction stopped using separate -per-section patterns. - -## Stage 2: Add Original Heading-Variant Regressions — Completed - -Provider and normalizer tests covered ellipsis-first and slash-qualified -headings, malformed forms, empty bodies, mixed heading families, and unchanged -canonical wire shape. - -## Stage 3: Validate the Original Feature — Completed - -Focused and repository-wide checks passed, production changes remained inside -the NWS provider parser, and the original roadmap was reconciled with the -implemented baseline. - -## Stage 4: Generalize Structural Heading Recognition — Completed - -Heading syntax was decoupled from canonical roles. The parser now accepts a -constrained generic uppercase identity grammar, normalizes identity whitespace, -supports embedded slashes without confusing slash qualifiers, and recognizes -unknown valid identities as structural headings. - -## Stage 5: Scan Ordered Section Blocks Once — Completed - -Repeated extraction was replaced by a one-pass ordered block scanner. Generic -headings, `&&`, `$$`, and the watch/advisory safeguard terminate active blocks; -role lookup happens after scanning and retains first-occurrence behavior. - -## Stage 6: Normalize Multiline Preambles and Markers — Completed - -Short- and long-term parsing gained next-line parenthesized qualifiers, -case-insensitive `Issued at` handling, and exact change-marker removal. Key -messages gained exact marker cleanup and removal of one leading issue/update -metadata line. - -## Stage 7: Add Initial Cross-Office Coverage — Completed - -The existing LSX fixture was retained and a compact BOU-style fixture added. -Provider and normalizer regressions cover multiline qualifiers, change markers, -leading metadata, generic boundary isolation, envelope behavior, and unchanged -JSON shape. - -## Stage 8: Validate the First Resilience Follow-up — Completed - -Focused tests, the full repository suite, static analysis, and diff checks -passed. The public model and schemas remained unchanged, and the roadmap was -updated to describe the completed behavior. - -## Stage 9: Add Parenthesized-Terminal Heading Syntax — Completed - -Recognize the observed `. ()...` family without weakening -generic identity validation. - -1. In `internal/providers/nws/forecast_discussion.go`, add a dedicated - `parseForecastDiscussionParenthesizedTerminalHeading` helper returning the - existing `forecastDiscussionSectionHeading` type. -2. Update `parseForecastDiscussionSectionHeading` to try forms in this order: - - 1. if the line ends `/...`, try the slash-qualified helper and return its - result; - 2. otherwise, try the parenthesized-terminal helper for any line ending - `...` and return it when it succeeds; and - 3. fall back to the existing ellipsis-first helper. - - Do not merge the forms into a single regular expression. -3. Implement the parenthesized-terminal helper with these exact rules: - - - operate on the already outer-trimmed line and require leading `.` plus a - terminal ASCII `...`; - - remove the leading dot and terminal ellipsis, then remove horizontal - whitespace immediately before the ellipsis; - - require the remaining content to end with `)`; - - locate the first `(` that is preceded by horizontal whitespace; everything - before that separator is the raw identity and everything from `(` through - the final `)` is the qualifier; - - normalize and validate the identity through the existing identity helper; - - require nonempty text after trimming inside the outer parentheses; - - return the qualifier with its outer parentheses and internal punctuation - intact; and - - reject missing separator whitespace, empty qualifiers, missing or misplaced - parentheses, trailing text after the ellipsis, unsupported identity - punctuation, and lowercase or mixed-case identities. - - Parentheses inside the qualifier are content; only the first separator `(` - and final `)` delimit the outer qualifier. -4. Extend the heading table in - `internal/providers/nws/forecast_discussion_test.go` with: - - - `.DISCUSSION (Today through Thursday)...`; - - a mapped identity such as `.SHORT TERM (Tonight)...`; - - identity whitespace normalization; - - qualifier punctuation and nested parentheses; - - each rejected malformed form listed above; and - - compatibility assertions for every existing ellipsis-first and - slash-qualified case. -5. Add a scanner regression in which a mapped section is followed immediately, - without `&&`, by `.DISCUSSION (Today through Thursday)...`. Assert that the - discussion heading starts a new boundary-only block and its prose cannot leak - into the mapped section. - -Run and pass: - -```sh -gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go -go test -count=1 ./internal/providers/nws -``` - -Do not proceed until all previous heading and scanner tests still pass. - -## Stage 10: Add Explicit Canonical Role Aliases — Completed - -Make true wording synonyms cheap to support while preserving semantic -distinctions. - -1. Add `KEY POINTS` to the existing identity-to-role registry with the key-message - role. Keep the registry as the only production source of identity-to-role - mappings; do not add a parallel alias collection or role-selection switch. -2. Leave `KEY MESSAGES`, `SHORT TERM`, and `LONG TERM` mappings unchanged. - Explicitly do not add role mappings for `NEAR TERM`, `DISCUSSION`, - `AVIATION`, or other generic identities. -3. Update registry tests so they no longer assume one identity per role. Assert - instead that: - - - every registry identity parses as a heading; - - `KEY MESSAGES` and `KEY POINTS` both map to the key-message role; - - `SHORT TERM` and `LONG TERM` retain their roles; and - - `NEAR TERM`, `DISCUSSION`, and `AVIATION` have no role entry. -4. Add `ParseForecastDiscussionText` tests for role-level first occurrence: - - - `KEY POINTS` alone populates key messages; - - `KEY POINTS` followed by `KEY MESSAGES` keeps the first block; and - - `KEY MESSAGES` followed by `KEY POINTS` keeps the first block. - - Use hyphen messages in this stage so list-tokenization changes remain scoped - to Stage 11. -5. Add a regression containing `NEAR TERM`, `SHORT TERM`, and `LONG TERM` in one - bulletin. Assert that near-term prose terminates adjacent blocks but does not - populate or override either canonical text section. - -Run and pass: - -```sh -gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go -go test -count=1 ./internal/providers/nws -``` - -Do not change the canonical model to expose near-term or discussion content. - -## Stage 11: Harden Key-Message Metadata and Item Parsing — Completed - -Replace prefix-sensitive, hyphen-only parsing with conservative metadata and -list classifiers. - -1. Add a helper that recognizes a leading labeled metadata line only when: - - - its trimmed text begins with exactly `Issued at` or `Updated at`, compared - ASCII case-insensitively; - - at least one horizontal-whitespace byte follows the label; and - - parsing the remaining text through the existing unlabeled NWS issue-time - grammar succeeds. - - Reuse `parseForecastDiscussionIssueTime` for the timestamp grammar after - removing the label. Do not add a second timestamp parser. A label prefix with - no boundary or with an invalid timestamp returns false and remains content. -2. Update key-message cleanup to remove at most one such valid leading metadata - line after presentation-marker removal and blank-line trimming. Later valid - timestamp lines remain message content. -3. Replace the hyphen-only check with a helper that classifies and strips one of - these markers from a trimmed line: - - - one leading `-`, preserving current compatibility whether or not whitespace - follows it; - - one leading `*`, whether or not whitespace follows it; or - - one or more ASCII digits forming an integer greater than zero, followed by - `)` or `.`, and then either end-of-line or horizontal whitespace. - - After recognizing and stripping a leading hyphen or asterisk plus horizontal - whitespace, also strip one immediately following valid numeric marker. This - supports composite forms such as `- 1.` without retaining either decorator. - Otherwise strip only the recognized marker and following horizontal - whitespace. Reject zero, overflow, alphanumeric prefixes, and numeric - punctuation without the required boundary. Do not interpret these markers - outside key-message parsing. -4. Parse a cleaned key-message body deterministically: - - - first determine whether any nonempty line has a recognized marker; - - when markers exist, each marker starts a new message; nonempty unmarked - lines after a marker continue that message; blank lines after the first - marker are ignored; and blank-line-separated prose before the first marker - is flushed as preserved message content; - - when no marker exists, each nonempty paragraph separated by one or more - blank lines becomes one message; and - - in both modes, join wrapped lines with one ASCII space, discard empty - messages, and preserve source order. -5. Add table-driven marker tests for `-`, `*`, `1)`, `2.`, composite `- 1.` and - `* 1)`, multi-digit values, wrapped continuations, marker-only lines, and - rejected zero/malformed numeric prefixes. -6. Add block-level tests covering: - - - numbered and asterisk lists producing distinct messages; - - unmarked paragraphs producing distinct messages; - - mixed introductory prose and marked items without data loss; - - valid leading `Issued at` and `Updated at` metadata removal; - - `Updated atmospheric conditions...` remaining content; - - `Updated at not a timestamp` remaining content; - - a later valid timestamp-like line remaining content; and - - all existing hyphen, change-marker, and continuation behavior. - -Run and pass: - -```sh -gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go -go test -count=1 ./internal/providers/nws -``` - -## Stage 12: Add Representative Numbered and Key-Points Fixtures — Completed - -Prove the new extension points through provider and normalizer boundaries. - -1. Retain the existing LSX and BOU fixtures and their regressions. -2. Add two compact HTML fixtures under `internal/providers/nws/testdata/`: - - - a BGM/CTP-style fixture with a valid header and issue time, numbered `KEY - MESSAGES`, wrapped item lines, and a boundary-only `DISCUSSION` section; and - - an MFR-style fixture with a valid header and issue time, asterisk `KEY - POINTS`, and an immediately following - `.DISCUSSION (Today through Thursday)...` section without an intervening - `&&`. - - Use concise representative text rather than full web pages. Add a short HTML - comment to each fixture naming the real office-format family it represents; - do not claim that edited fixture prose is a verbatim archived product. -3. Add provider end-to-end tests that assert: - - - exact ordered key messages and joined continuations; - - `KEY POINTS` alias mapping; - - correct office and top-level issue metadata; - - absence of numeric/asterisk markers from canonical values; - - absence of `DISCUSSION` headings, qualifiers, and prose from key messages; - and - - nil short- and long-term fields when the fixture contains neither mapped - role. -4. Add table-driven normalizer regressions using both fixtures. For each, verify - input envelope preservation, event kind, canonical schema, effective time, - exact key-message mapping, and successful JSON marshaling. -5. Assert the JSON payload still has no `nearTerm`, `discussion`, `aviation`, or - generic `sections` field. Do not add those fields to provider or canonical - structs. -6. Keep syntax and tokenizer edge cases in focused unit tables; do not add more - full fixtures for cases already proven locally. - -Run and pass: - -```sh -gofmt -w internal/providers/nws/forecast_discussion_test.go internal/normalizers/nws/forecast_discussion_test.go -go test -count=1 ./internal/providers/nws ./internal/normalizers/nws -``` - -## Stage 13: Reconcile Documentation and Perform Final Validation — Completed - -Close the second resilience follow-up only after all behavior is implemented and -verified. - -1. Review the final diff. Production changes must remain confined to - `internal/providers/nws/forecast_discussion.go`; other Go changes must be - owning provider and normalizer tests. Preserve unrelated user work. -2. Confirm the public contract is unchanged. If implementation appears to - require a canonical near-term/discussion field, schema change, configuration, - or downstream migration, stop and report the conflict rather than expanding - scope. -3. Run: - - ```sh - gofmt -w \ - internal/providers/nws/forecast_discussion.go \ - internal/providers/nws/forecast_discussion_test.go \ - internal/normalizers/nws/forecast_discussion_test.go - go test -count=1 ./internal/providers/nws ./internal/normalizers/nws - go test -count=1 ./... - go vet ./... - git diff --check - ``` -4. Verify every acceptance criterion in the feature roadmap has a corresponding - passing test, including compatibility with Stages 1-8. -5. Update the feature roadmap status to `Implemented.` and rewrite its remaining - proposed or future-tense language as implemented behavior only after all - checks pass. -6. Update this implementation plan so Stages 9-13 are marked `— Completed` and - the Purpose states that all stages are complete. Preserve the stage details - as a historical implementation record. - -## Open Questions - -None. The plan makes the required policy choices explicitly: `KEY POINTS` is a -true key-message alias; `NEAR TERM` remains semantically distinct and -boundary-only; parenthesized-terminal syntax is a third constrained grammar; -metadata is removed only after timestamp validation; and key-message variants -are handled by an isolated marker classifier plus paragraph fallback. diff --git a/internal/normalizers/nws/forecast_discussion_test.go b/internal/normalizers/nws/forecast_discussion_test.go index 5a3604a..72e66f0 100644 --- a/internal/normalizers/nws/forecast_discussion_test.go +++ b/internal/normalizers/nws/forecast_discussion_test.go @@ -215,7 +215,7 @@ func TestForecastDiscussionNormalizerSupportsCrossOfficeLayout(t *testing.T) { } } -func TestForecastDiscussionNormalizerSupportsNumberedAndKeyPointsFixtures(t *testing.T) { +func TestForecastDiscussionNormalizerSupportsCrossOfficeKeyMessageFixtures(t *testing.T) { tests := []struct { name string filename string @@ -249,6 +249,42 @@ func TestForecastDiscussionNormalizerSupportsNumberedAndKeyPointsFixtures(t *tes "Inland valleys remain dry through Saturday.", }, }, + { + name: "as of preamble", + filename: "forecast_discussion_rah_as_of_sample.html", + id: "evt-discussion-rah", + source: "nws-discussion-rah-test", + emittedAt: time.Date(2026, 8, 2, 16, 36, 0, 0, time.UTC), + effectiveAt: time.Date(2026, 8, 2, 16, 35, 0, 0, time.UTC), + messages: []string{ + "Scattered storms may produce locally heavy rain this afternoon.", + "Drier weather arrives Monday.", + }, + }, + { + name: "parenthesized numeric markers", + filename: "forecast_discussion_lwx_parenthesized_number_sample.html", + id: "evt-discussion-lwx", + source: "nws-discussion-lwx-test", + emittedAt: time.Date(2026, 8, 2, 18, 1, 0, 0, time.UTC), + effectiveAt: time.Date(2026, 8, 2, 18, 0, 0, 0, time.UTC), + messages: []string{ + "Thunderstorms remain possible near the Blue Ridge this evening.", + "Seasonably warm conditions continue Monday.", + }, + }, + { + name: "embedded key messages in previous discussion", + filename: "forecast_discussion_mfr_prev_discussion_sample.html", + id: "evt-discussion-mfr-previous", + source: "nws-discussion-mfr-previous-test", + emittedAt: time.Date(2026, 8, 2, 22, 20, 0, 0, time.UTC), + effectiveAt: time.Date(2026, 8, 2, 22, 19, 0, 0, time.UTC), + messages: []string{ + "Heat returns to inland valleys Monday.", + "Gusty afternoon winds develop east of the Cascades.", + }, + }, } for _, tt := range tests { diff --git a/internal/providers/nws/forecast_discussion.go b/internal/providers/nws/forecast_discussion.go index cd166a5..e189e72 100644 --- a/internal/providers/nws/forecast_discussion.go +++ b/internal/providers/nws/forecast_discussion.go @@ -550,6 +550,7 @@ func isForecastDiscussionHorizontalWhitespace(b byte) bool { func parseForecastDiscussionSectionBlocks(lines []string) []forecastDiscussionSectionBlock { var blocks []forecastDiscussionSectionBlock var active *forecastDiscussionSectionBlock + embeddedHeadings := false finish := func() { if active == nil { @@ -567,6 +568,7 @@ func parseForecastDiscussionSectionBlocks(lines []string) []forecastDiscussionSe } if line == "&&" || strings.Contains(line, "WATCHES/WARNINGS/ADVISORIES") { finish() + embeddedHeadings = false continue } @@ -574,8 +576,17 @@ func parseForecastDiscussionSectionBlocks(lines []string) []forecastDiscussionSe if ok { finish() active = &forecastDiscussionSectionBlock{heading: heading} + embeddedHeadings = isForecastDiscussionEmbeddedSectionWrapper(heading.section) continue } + if embeddedHeadings { + heading, ok = parseForecastDiscussionEmbeddedSectionHeading(raw) + if ok { + finish() + active = &forecastDiscussionSectionBlock{heading: heading} + continue + } + } if active != nil { active.body = append(active.body, raw) } @@ -585,6 +596,18 @@ func parseForecastDiscussionSectionBlocks(lines []string) []forecastDiscussionSe return blocks } +func isForecastDiscussionEmbeddedSectionWrapper(section string) bool { + return section == "PREV DISCUSSION" +} + +func parseForecastDiscussionEmbeddedSectionHeading(line string) (forecastDiscussionSectionHeading, bool) { + line = strings.TrimSpace(line) + if line == "" || line[0] == '.' { + return forecastDiscussionSectionHeading{}, false + } + return parseForecastDiscussionSectionHeading("." + line) +} + func parseForecastDiscussionKeyMessages(body []string) []string { body = removeForecastDiscussionPresentationMarkers(body) body = trimBlankLines(body) @@ -716,7 +739,32 @@ func isForecastDiscussionKeyMessageMetadataLine(line string) bool { return true } } - return false + return isForecastDiscussionKeyMessageAsOfLine(line) +} + +func isForecastDiscussionKeyMessageAsOfLine(line string) bool { + const label = "As of" + + if !hasForecastDiscussionASCIIPrefix(line, label) || len(line) == len(label) || !isForecastDiscussionHorizontalWhitespace(line[len(label)]) { + return false + } + remainder := strings.TrimSpace(line[len(label):]) + if !strings.HasSuffix(remainder, "...") { + return false + } + fields := strings.Fields(strings.TrimSpace(strings.TrimSuffix(remainder, "..."))) + if len(fields) != 3 { + return false + } + if _, _, err := parseForecastDiscussionClock(fields[0], fields[1]); err != nil { + return false + } + switch strings.ToLower(fields[2]) { + case "monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday": + return true + default: + return false + } } func stripForecastDiscussionKeyMessageMarker(line string) (string, bool) { @@ -740,11 +788,23 @@ func stripForecastDiscussionKeyMessageMarker(line string) (string, bool) { } func stripForecastDiscussionKeyMessageNumericMarker(line string) (string, bool) { + digitStart := 0 digitEnd := 0 + parenthesized := len(line) > 0 && line[0] == '(' + if parenthesized { + digitStart = 1 + digitEnd = 1 + } for digitEnd < len(line) && line[digitEnd] >= '0' && line[digitEnd] <= '9' { digitEnd++ } - if digitEnd == 0 || digitEnd == len(line) || (line[digitEnd] != ')' && line[digitEnd] != '.') { + if digitEnd == digitStart || digitEnd == len(line) { + return "", false + } + if parenthesized && line[digitEnd] != ')' { + return "", false + } + if !parenthesized && line[digitEnd] != ')' && line[digitEnd] != '.' { return "", false } @@ -752,7 +812,7 @@ func stripForecastDiscussionKeyMessageNumericMarker(line string) (string, bool) if markerEnd < len(line) && !isForecastDiscussionHorizontalWhitespace(line[markerEnd]) { return "", false } - value, err := strconv.ParseUint(line[:digitEnd], 10, 0) + value, err := strconv.ParseUint(line[digitStart:digitEnd], 10, 0) if err != nil || value == 0 { return "", false } diff --git a/internal/providers/nws/forecast_discussion_test.go b/internal/providers/nws/forecast_discussion_test.go index 71b11f5..ddd736c 100644 --- a/internal/providers/nws/forecast_discussion_test.go +++ b/internal/providers/nws/forecast_discussion_test.go @@ -391,6 +391,58 @@ func TestParseForecastDiscussionSectionBlocksUsesGenericHeadingsAsBoundaries(t * } } +func TestParseForecastDiscussionSectionBlocksRecognizesEmbeddedHeadingsInPreviousDiscussion(t *testing.T) { + got := parseForecastDiscussionSectionBlocks([]string{ + ".PREV DISCUSSION... /Issued 319 PM PDT Sun Aug 2 2026/", + "KEY MESSAGES...", + "* First embedded message.", + "DISCUSSION...", + "Discussion prose.", + "&&", + }) + want := []forecastDiscussionSectionBlock{ + { + heading: forecastDiscussionSectionHeading{ + section: "PREV DISCUSSION", + qualifier: "/Issued 319 PM PDT Sun Aug 2 2026/", + }, + }, + { + heading: forecastDiscussionSectionHeading{section: "KEY MESSAGES"}, + body: []string{"* First embedded message."}, + }, + { + heading: forecastDiscussionSectionHeading{section: "DISCUSSION"}, + body: []string{"Discussion prose."}, + }, + } + if !reflect.DeepEqual(got, want) { + t.Fatalf("blocks = %#v, want %#v", got, want) + } +} + +func TestParseForecastDiscussionSectionBlocksKeepsUndottedHeadingLikeLinesOutsideWrappers(t *testing.T) { + got := parseForecastDiscussionSectionBlocks([]string{ + ".SHORT TERM...", + "Short-term prose.", + "KEY MESSAGES...", + "Still short-term prose.", + }) + want := []forecastDiscussionSectionBlock{ + { + heading: forecastDiscussionSectionHeading{section: "SHORT TERM"}, + body: []string{ + "Short-term prose.", + "KEY MESSAGES...", + "Still short-term prose.", + }, + }, + } + if !reflect.DeepEqual(got, want) { + t.Fatalf("blocks = %#v, want %#v", got, want) + } +} + func TestParseForecastDiscussionSectionBlocksUsesParenthesizedTerminalHeadingAsBoundary(t *testing.T) { got := parseForecastDiscussionSectionBlocks([]string{ ".SHORT TERM... (Tonight)", @@ -756,6 +808,16 @@ func TestIsForecastDiscussionKeyMessageMetadataLine(t *testing.T) { line: "uPdAtEd At 300 PM CDT Sat Mar 28 2026", want: true, }, + { + name: "as of weekday preamble", + line: "As of 1235 PM Sunday...", + want: true, + }, + { + name: "as of ASCII case insensitive", + line: "aS oF 1235 pm SuNdAy...", + want: true, + }, { name: "label without whitespace boundary", line: "Updated at300 PM CDT Sat Mar 28 2026", @@ -766,6 +828,26 @@ func TestIsForecastDiscussionKeyMessageMetadataLine(t *testing.T) { line: "Updated at not a timestamp", want: false, }, + { + name: "as of invalid clock", + line: "As of 2535 PM Sunday...", + want: false, + }, + { + name: "as of invalid weekday", + line: "As of 1235 PM Someday...", + want: false, + }, + { + name: "as of without terminal ellipsis", + line: "As of 1235 PM Sunday", + want: false, + }, + { + name: "as of prefix collision", + line: "As often happens, storms weaken overnight.", + want: false, + }, { name: "nonmetadata prose", line: "Updated atmospheric conditions remain unsettled.", @@ -794,13 +876,18 @@ func TestStripForecastDiscussionKeyMessageMarker(t *testing.T) { {name: "asterisk", line: "* First message.", want: "First message.", ok: true}, {name: "numeric parenthesis", line: "1) First message.", want: "First message.", ok: true}, {name: "numeric dot", line: "2. Second message.", want: "Second message.", ok: true}, + {name: "parenthesized numeric", line: "(1) First message.", want: "First message.", ok: true}, {name: "composite hyphen", line: "- 1. First message.", want: "First message.", ok: true}, {name: "composite asterisk", line: "* 1) First message.", want: "First message.", ok: true}, + {name: "composite parenthesized numeric", line: "- (1) First message.", want: "First message.", ok: true}, + {name: "composite asterisk parenthesized numeric", line: "* (1) First message.", want: "First message.", ok: true}, {name: "multi digit numeric", line: "12. Twelfth message.", want: "Twelfth message.", ok: true}, {name: "hyphen only", line: "-", want: "", ok: true}, {name: "numeric marker only", line: "1)", want: "", ok: true}, {name: "zero numeric marker", line: "0) Not a marker.", want: "", ok: false}, + {name: "zero parenthesized numeric marker", line: "(0) Not a marker.", want: "", ok: false}, {name: "overflow numeric marker", line: "999999999999999999999999999999. Too big.", want: "", ok: false}, + {name: "parenthesized numeric marker without boundary", line: "(1)Not a marker.", want: "", ok: false}, {name: "alphanumeric numeric prefix", line: "1x Not a marker.", want: "", ok: false}, {name: "numeric marker without boundary", line: "1)Not a marker.", want: "", ok: false}, {name: "numeric marker with unsupported punctuation", line: "1, Not a marker.", want: "", ok: false}, @@ -877,6 +964,14 @@ func TestParseForecastDiscussionKeyMessagesClassifiesListsAndMetadata(t *testing }, want: []string{"First message."}, }, + { + name: "leading as of metadata", + body: []string{ + "As of 1235 PM Sunday...", + "- First message.", + }, + want: []string{"First message."}, + }, { name: "updated atmospheric prose remains content", body: []string{ @@ -901,6 +996,14 @@ func TestParseForecastDiscussionKeyMessagesClassifiesListsAndMetadata(t *testing }, want: []string{"First message. Updated at 300 PM CDT Sat Mar 28 2026"}, }, + { + name: "later as of metadata remains content", + body: []string{ + "- First message.", + "As of 1235 PM Sunday...", + }, + want: []string{"First message. As of 1235 PM Sunday..."}, + }, } for _, tt := range tests { @@ -1089,7 +1192,7 @@ func TestParseForecastDiscussionHTMLSupportsMixedHeadingFormats(t *testing.T) { } } -func TestParseForecastDiscussionHTMLParsesNumberedAndKeyPointsFixtures(t *testing.T) { +func TestParseForecastDiscussionHTMLParsesCrossOfficeKeyMessageFixtures(t *testing.T) { tests := []struct { name string filename string @@ -1127,6 +1230,48 @@ func TestParseForecastDiscussionHTMLParsesNumberedAndKeyPointsFixtures(t *testin "*", "DISCUSSION", "(Today through Thursday)", "Discussion details must not be included with key points.", }, }, + { + name: "as of preamble", + filename: "forecast_discussion_rah_as_of_sample.html", + officeID: "RAH", + officeName: "National Weather Service Raleigh NC", + issuedAt: time.Date(2026, 8, 2, 16, 35, 0, 0, time.UTC), + messages: []string{ + "Scattered storms may produce locally heavy rain this afternoon.", + "Drier weather arrives Monday.", + }, + unwanted: []string{ + "As of", "1)", "2)", "DISCUSSION", "Discussion details remain boundary-only content.", + }, + }, + { + name: "parenthesized numeric markers", + filename: "forecast_discussion_lwx_parenthesized_number_sample.html", + officeID: "LWX", + officeName: "National Weather Service Baltimore MD/Washington DC", + issuedAt: time.Date(2026, 8, 2, 18, 0, 0, 0, time.UTC), + messages: []string{ + "Thunderstorms remain possible near the Blue Ridge this evening.", + "Seasonably warm conditions continue Monday.", + }, + unwanted: []string{ + "(1)", "(2)", "AVIATION", "Aviation details remain boundary-only content.", + }, + }, + { + name: "embedded key messages in previous discussion", + filename: "forecast_discussion_mfr_prev_discussion_sample.html", + officeID: "MFR", + officeName: "National Weather Service Medford OR", + issuedAt: time.Date(2026, 8, 2, 22, 19, 0, 0, time.UTC), + messages: []string{ + "Heat returns to inland valleys Monday.", + "Gusty afternoon winds develop east of the Cascades.", + }, + unwanted: []string{ + "PREV DISCUSSION", "Issued 319 PM", "*", "DISCUSSION", "Discussion details must not be included with key messages.", + }, + }, } for _, tt := range tests { diff --git a/internal/providers/nws/testdata/forecast_discussion_lwx_parenthesized_number_sample.html b/internal/providers/nws/testdata/forecast_discussion_lwx_parenthesized_number_sample.html new file mode 100644 index 0000000..3e27939 --- /dev/null +++ b/internal/providers/nws/testdata/forecast_discussion_lwx_parenthesized_number_sample.html @@ -0,0 +1,27 @@ + + + + +
+FXUS61 KLWX 021800
+AFDLWX
+
+Area Forecast Discussion
+National Weather Service Baltimore MD/Washington DC
+200 PM EDT Sun Aug 2 2026
+
+.KEY MESSAGES...
+- (1) Thunderstorms remain possible near the Blue Ridge this evening.
+- (2) Seasonably warm conditions continue Monday.
+
+&&
+
+.AVIATION...
+Aviation details remain boundary-only content.
+
+$$
+
+WFO LWX
+    
+ + diff --git a/internal/providers/nws/testdata/forecast_discussion_mfr_prev_discussion_sample.html b/internal/providers/nws/testdata/forecast_discussion_mfr_prev_discussion_sample.html new file mode 100644 index 0000000..a11c842 --- /dev/null +++ b/internal/providers/nws/testdata/forecast_discussion_mfr_prev_discussion_sample.html @@ -0,0 +1,33 @@ + + + + +
+FXUS66 KMFR 022219
+AFDMFR
+
+Area Forecast Discussion
+National Weather Service Medford OR
+319 PM PDT Sun Aug 2 2026
+
+.PREV DISCUSSION... /Issued 319 PM PDT Sun Aug 2 2026/
+
+KEY MESSAGES...
+
+* Heat returns to inland valleys Monday.
+* Gusty afternoon winds develop east of the Cascades.
+
+DISCUSSION...
+Discussion details must not be included with key messages.
+
+&&
+
+.MFR WATCHES/WARNINGS/ADVISORIES...
+None.
+
+$$
+
+WFO MFR
+    
+ + diff --git a/internal/providers/nws/testdata/forecast_discussion_rah_as_of_sample.html b/internal/providers/nws/testdata/forecast_discussion_rah_as_of_sample.html new file mode 100644 index 0000000..23ce075 --- /dev/null +++ b/internal/providers/nws/testdata/forecast_discussion_rah_as_of_sample.html @@ -0,0 +1,29 @@ + + + + +
+FXUS62 KRAH 021635
+AFDRAH
+
+Area Forecast Discussion
+National Weather Service Raleigh NC
+1235 PM EDT Sun Aug 2 2026
+
+.KEY MESSAGES...
+As of 1235 PM Sunday...
+
+1) Scattered storms may produce locally heavy rain this afternoon.
+2) Drier weather arrives Monday.
+
+&&
+
+.DISCUSSION...
+Discussion details remain boundary-only content.
+
+$$
+
+WFO RAH
+    
+ +