ejr/weatherfeeder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Generalize NWS forecast discussion heading parsing

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-02 22:53:00 +00:00
parent 29c65971eb
commit 2b19a121fa
4 changed files with 618 additions and 331 deletions
Download Patch File Download Diff File Expand all files Collapse all files

240
docs/roadmap/afd-section-heading-variants.md
View File

@@ -1,141 +1,171 @@
# NWS AFD Section Heading Variants # NWS AFD Section Parsing Resilience
## Status ## Status
Implemented. The original ellipsis-first and slash-qualified heading feature is implemented.
The resilience follow-up defined below is proposed and unimplemented.
## Problem ## Completed Baseline
The NWS Area Forecast Discussion parser recognizes supported section headings The NWS Area Forecast Discussion parser now recognizes these heading families
only when the section name is followed immediately by an ellipsis, for example: for key messages, short term, long term, and aviation:
```text
.SHORT TERM... (Through Monday)
.LONG TERM... (Tuesday through Sunday)
.AVIATION... (For the 18z TAFs through 18z Monday)
```
NWS offices also publish slash-qualified headings:
```text
.SHORT TERM /THROUGH MONDAY/...
.LONG TERM /MONDAY NIGHT THROUGH SUNDAY/...
.AVIATION /18Z TAFS THROUGH 18Z MONDAY/...
```
The current exact-prefix discovery logic does not find slash-qualified short-
or long-term sections. Its boundary detection also does not recognize a
slash-qualified aviation heading, so preceding section text can absorb aviation
content. The forecast discussion can still normalize successfully, leaving
downstream consumers with missing or incorrectly bounded structured prose.
## Feature Objective
Make supported AFD section recognition consistent across section discovery,
section-boundary detection, and qualifier extraction while preserving the
existing canonical forecast-discussion contract.
## Targeted End State
One NWS-specific heading parser classifies a trimmed AFD line as either a
supported heading or a non-heading. For a supported heading it provides:
- the canonical section identity;
- the qualifier, if present; and
- enough information for discovery and boundary detection to use the same
recognition result.
The supported section identities remain:
- `KEY MESSAGES`;
- `SHORT TERM`;
- `LONG TERM`; and
- `AVIATION`.
`KEY MESSAGES`, `SHORT TERM`, and `LONG TERM` participate in the existing
canonical extraction behavior. `AVIATION` remains a recognized boundary only;
it does not become a canonical payload field.
For each supported identity, the parser accepts both heading families:
```text ```text
.<SECTION>...<optional qualifier> .<SECTION>...<optional qualifier>
.<SECTION> /<qualifier>/... .<SECTION> /<qualifier>/...
``` ```
Qualifier handling is deterministic: Discovery, qualifier extraction, and recognized-section boundary detection use
one provider-specific parser. Slash delimiters are removed from qualifiers,
legacy qualifier text is preserved, and aviation remains a boundary rather than
a canonical field. Focused provider and normalizer regressions cover the
implemented forms.
- leading and trailing whitespace is removed; ## Remaining Problem
- the enclosing slash pair used by the slash-qualified form is removed;
- qualifier content otherwise retains its published text, including
parentheses in the existing ellipsis-first form; and
- a heading without a qualifier yields an empty qualifier.
The same classification rules govern discovery and termination. A recognized Current NWS bulletins vary beyond those two same-line forms. In particular:
heading ends the preceding section without becoming part of its text. Unknown
section names, unrelated dotted lines, and malformed slash-qualified lines are
not treated as supported headings.
## Scope of Work - an ellipsis-first heading may put its qualifier on the next nonblank line;
- NWS presentation output can place `-- Changed Discussion --` markers and an
`Updated at` line around section content;
- valid AFDs can include `DISCUSSION`, `UPDATE`, `MARINE`, `HYDROLOGY`,
`CLIMATE`, `FIRE WEATHER`, office watch/advisory blocks, and other topic
headings; and
- minor punctuation or qualifier changes can produce headings that are
structurally valid but absent from the parser's identity whitelist.
The feature includes: The current parser recognizes only four exact identities. Unknown headings do
not terminate the preceding section. It also expects a text section's optional
qualifier and `Issued at` line in a narrow order. Consequently, valid upstream
format variants can leave qualifier and section-time fields empty, leak
presentation metadata into prose or key messages, or allow one section to
absorb another.
- consolidating supported-heading recognition and qualifier extraction in ## Feature Objective
`internal/providers/nws`;
- applying that recognition consistently to section lookup and section
boundaries;
- preserving key-message parsing and short- and long-term section mapping;
- covering legacy, slash-qualified, mixed-format, and malformed headings with
focused provider-parser tests; and
- verifying that parsed short- and long-term sections continue through the NWS
normalizer into the existing canonical payload.
Existing ellipsis-first fixtures and behavior remain regression coverage. New Make AFD parsing resilient to minor upstream format evolution by separating:
fixtures should be minimal and representative rather than copies of complete
upstream bulletins unless a full bulletin is needed to prove an interaction. 1. generic structural heading recognition;
2. section boundary scanning;
3. canonical section-role selection; and
4. section-preamble and presentation cleanup.
The parser should accept new structurally valid topic names as safe boundaries
without treating them as new canonical fields.
## Targeted End State
### Generic heading recognition
One NWS-specific heading parser recognizes structurally valid uppercase AFD
topic lines independently of the canonical model. It supports:
```text
.<IDENTITY>...<optional qualifier>
.<IDENTITY> /<qualifier>/...
```
The identity may contain uppercase ASCII letters, digits, horizontal whitespace,
`/`, `&`, apostrophes, and hyphens. Outer whitespace and whitespace immediately
before the ellipsis are ignored, and internal identity whitespace is collapsed
to one space for role lookup. Slash-qualified headings require a nonempty
qualifier but may contain slash characters inside that qualifier. Lowercase
prose, ordinary dotted lines, and malformed delimiters remain non-headings.
Every structurally valid heading terminates the preceding section. This applies
to known boundary-only sections such as aviation and to future or office-specific
topic identities that weatherfeeder does not map.
### Canonical role selection
A single provider-local role registry maps only these identities into existing
parsed fields:
- `KEY MESSAGES` to key messages;
- `SHORT TERM` to the short-term section; and
- `LONG TERM` to the long-term section.
All other identities are boundary-only. The first occurrence of each mapped
identity wins, preserving current behavior if an unusual bulletin repeats a
section. Heading syntax and canonical roles do not duplicate identity lists.
### Section scanning
The discussion text is scanned once into ordered section blocks. A new heading,
`&&`, `$$`, or the existing watch/advisory terminator ends the active block.
Original body lines are preserved until provider presentation cleanup is
applied. Preamble text before the first heading and post-`$$` signatures are not
treated as section content.
### Preamble and presentation handling
For short- and long-term sections:
- a qualifier on the heading line remains authoritative;
- when the heading has no qualifier, a standalone parenthesized first content
line becomes the qualifier and retains its parentheses;
- an optional case-insensitive `Issued at` line after the qualifier is parsed
into the existing section issue time; and
- exact NWS change-presentation marker lines are removed without removing
arbitrary dashed prose.
For key messages, exact change-presentation markers and one leading `Issued at`
or `Updated at` metadata line are removed before bullet parsing. Those metadata
lines never become key messages.
### Representative coverage
Tests include small structural tables and maintained local HTML fixtures for at
least two real NWS formatting families: slash-qualified same-line headings and
ellipsis-first headings with multiline qualifiers or change-presentation
markers. Tests remain deterministic and never contact live services.
## Compatibility and Contracts ## Compatibility and Contracts
This is a provider-parsing compatibility improvement. It does not change: This remains a provider-parsing compatibility improvement. It does not change:
- event kinds or raw and canonical schema identifiers; - event kinds or raw and canonical schema identifiers;
- canonical models or JSON field names; - canonical models or JSON field names;
- source configuration or polling behavior; - source configuration, URLs, or polling behavior;
- event envelope or effective-time behavior; - event envelope or effective-time behavior;
- Postgres tables or event-to-row mapping; or - Postgres tables or event-to-row mapping; or
- downstream sink and consumer responsibilities. - downstream sink and consumer responsibilities.
Current-behavior documentation should be updated only if implementation reveals A single `DISCUSSION` section and other currently unmapped identities are
an externally observable contract change beyond the scope defined here. recognized as boundaries but are not forced into short- or long-term fields.
Exposing such content would require a separate canonical schema decision.
## Acceptance Criteria ## Acceptance Criteria
The feature is complete when automated tests demonstrate that: The resilience follow-up is complete when automated tests demonstrate that:
- all currently accepted ellipsis-first headings produce unchanged results; - every previously accepted heading and canonical result remains compatible;
- slash-qualified short- and long-term headings are discovered and their - generic structurally valid identities terminate preceding content without
qualifiers exclude slash delimiters; becoming canonical fields;
- slash-qualified recognized headings correctly terminate a preceding section, - malformed heading-like lines and lowercase prose remain body content;
including an aviation heading following a long-term section; - identities containing `/` and slash qualifiers containing `/` are parsed
- documents mixing the two heading families are parsed correctly; without ambiguity;
- headings without qualifiers retain existing behavior; - next-line parenthesized qualifiers and following `Issued at` lines populate
- unknown or malformed heading-like lines do not create supported sections or the existing short- and long-term metadata correctly;
prematurely terminate one; - exact NWS change markers and leading key-message timestamps do not leak into
- the NWS forecast-discussion normalizer emits populated canonical short- and canonical prose or messages;
long-term fields for representative slash-qualified input without adding an - repeated mapped sections preserve first-occurrence behavior;
aviation field; and - genuine cross-office fixture styles propagate correctly through the NWS
- the full repository test suite passes. normalizer with unchanged wire shape; and
- focused tests, the full repository suite, and static analysis pass.
## Non-Goals ## Non-Goals
This feature does not: This follow-up does not:
- add new canonical forecast-discussion sections; - add canonical `discussion`, aviation, marine, hydrology, climate, fire
- add aviation prose to the canonical model; weather, update, or arbitrary-section fields;
- recognize arbitrary or previously unsupported NWS section families; - infer short- or long-term semantics from an unknown heading;
- introduce heuristic section-name matching or AFD summarization; - accept free-form or lowercase prose as a heading;
- preserve complete raw AFD documents in canonical payloads; or - introduce heuristic summarization;
- change schemas, persistence contracts, configuration, or downstream APIs. - preserve complete raw AFD documents in canonical payloads;
- change schemas, persistence contracts, configuration, or downstream APIs; or
- fetch live NWS data during tests.
Support for additional section identities or broader AFD structure should be Canonical support for additional AFD section identities should be driven by a
driven by a separate consumer requirement and roadmap. separate consumer requirement and schema roadmap.

397
docs/roadmap/implementation.md
View File

@@ -1,181 +1,264 @@
# NWS AFD Section Heading Variants Implementation Plan # NWS AFD Section Parsing Resilience Implementation Plan
## Purpose ## Purpose
Implement the end state defined in Complete the resilience end state in
[`afd-section-heading-variants.md`](afd-section-heading-variants.md): recognize [`afd-section-heading-variants.md`](afd-section-heading-variants.md) while
legacy ellipsis-first and slash-qualified NWS Area Forecast Discussion headings preserving the current canonical forecast-discussion contract. Stages 1-3 below
consistently during section discovery, boundary detection, and qualifier summarize completed work. Implement Stages 4-8 in order.
extraction without changing the canonical event contract.
Complete the stages below in order. Keep each stage limited to the files and
behavior it names, and leave the repository passing its tests before proceeding.
## Cross-Stage Constraints ## Cross-Stage Constraints
- Keep all provider-format parsing in `internal/providers/nws`. - Keep all production parsing changes in
- Use only the Go standard library; do not add a dependency. `internal/providers/nws/forecast_discussion.go`.
- Do not change `model`, `standards`, source configuration, polling behavior, - Use only the Go standard library and keep helpers unexported.
schemas, event-envelope behavior, Postgres mapping, or sink behavior. - Do not change `model`, `standards`, source configuration or polling, schemas,
- Preserve the existing case-sensitive supported section identities: `KEY event-envelope behavior, Postgres mapping, sinks, consumer docs, or current
MESSAGES`, `SHORT TERM`, `LONG TERM`, and `AVIATION`. integration contracts.
- Continue exposing only key messages, short term, and long term in the parsed - Continue exposing only key messages, short term, and long term. All other
and canonical forecast-discussion payloads. `AVIATION` is a boundary marker, structurally valid identities are boundary-only.
not a new output field. - Preserve first-occurrence behavior for mapped sections.
- Preserve source body lines for the existing prose parsers; heading - Preserve raw body lines during structural scanning; apply provider-specific
normalization must not alter section text, issue-time parsing, signature cleanup only when parsing a block's content.
trimming, or paragraph joining. - Keep parsing deterministic. Tests must use local strings and fixtures, never
- Treat unknown section names and malformed slash-qualified lines as ordinary live NWS requests.
body lines, not as recognized boundaries. - Preserve all pre-existing user work and avoid unrelated refactors.
- Use deterministic unit tests and the checked-in fixture. Do not contact live
NWS services.
## Stage 1: Centralize Heading Parsing and Section Extraction ## Stage 1: Centralize Known Heading Parsing — Completed
Implement the provider-level parsing primitive and make it the sole source of The provider parser gained one heading classifier, explicit section and block
heading identity and qualifier interpretation. types, and shared discovery, boundary, and qualifier handling for the original
four identities. Key-message and text-section consumers were moved to the block
representation, and immediately adjacent recognized headings became valid
boundaries.
1. In `internal/providers/nws/forecast_discussion.go`, define unexported string ## Stage 2: Add Heading-Variant Regressions — Completed
constants for the four supported section identities. Use those constants in
`ParseForecastDiscussionText` instead of repeating string literals.
2. Add an unexported `forecastDiscussionSectionHeading` value with `section`
and `qualifier` fields, plus a
`parseForecastDiscussionSectionHeading(string) (forecastDiscussionSectionHeading, bool)`
helper. The helper must trim outer line whitespace and implement exactly
these two case-sensitive, whole-line grammars:
```text Provider tests now cover ellipsis-first and slash-qualified headings, malformed
^\.(KEY MESSAGES|SHORT TERM|LONG TERM|AVIATION)\.\.\.(.*)$ forms, empty bodies, mixed heading families, and slash-qualified aviation
^\.(KEY MESSAGES|SHORT TERM|LONG TERM|AVIATION)[ \t]+/([^/]+)/\.\.\.$ termination. Normalizer coverage verifies canonical propagation and unchanged
``` wire shape.
Keep the supported-identity alternation in one shared pattern constant used ## Stage 3: Validate and Record the Baseline — Completed
to build both regular expressions, so the accepted identity list cannot
drift between the two forms.
3. For the ellipsis-first form, trim leading and trailing whitespace from the
text following the ellipsis and otherwise preserve it verbatim. This retains
current results such as `(Through Late Sunday Night)` and permits an empty
qualifier.
4. For the slash-qualified form, require at least one space or tab before the
opening slash, forbid embedded slash characters, require the closing slash
immediately before the final ellipsis, and reject a qualifier that becomes
empty after trimming. Return the trimmed inner text without either slash.
Do not case-fold identities or accept trailing text after the final
ellipsis.
5. Add an unexported `forecastDiscussionSectionBlock` containing the parsed
heading and its body lines. Change `extractForecastDiscussionSection` to
return this value. Find the requested section by calling the new heading
parser and comparing its `section` field to the requested identity; remove
the current constructed exact-prefix lookup.
6. While collecting a block body, retain the existing terminators `&&`, `$$`,
and lines containing `WATCHES/WARNINGS/ADVISORIES`. Also stop before every
subsequent line recognized by the new heading parser, including a heading
immediately following the current heading. Remove the existing
`j > i+1` exception so an empty section cannot absorb the next heading.
Preserve original, untrimmed body lines in the returned block.
7. Update the consumers of the extracted block:
- pass only `block.body` to key-message parsing and change Focused and full tests passed, production changes remained inside the NWS
`parseForecastDiscussionKeyMessages` so it no longer assumes the heading provider parser, and the original feature roadmap was marked implemented.
occupies element zero;
- make `parseForecastDiscussionTextSection` consume the block, initialize
`Qualifier` directly from `block.heading.qualifier`, and process
`block.body` with the existing issue-time and prose logic; and
- remove `forecastDiscussionHeaderRE`,
`isForecastDiscussionSectionHeader`, and
`parseForecastDiscussionQualifier` after all callers use the centralized
helper.
8. In `internal/providers/nws/forecast_discussion_test.go`, add a table-driven
unit test for `parseForecastDiscussionSectionHeading` with, at minimum:
- ellipsis-first headings for all four identities; ## Stage 4: Generalize Heading Syntax and Centralize Canonical Roles
- an ellipsis-first heading with no qualifier;
- slash-qualified headings for all four identities; Decouple structural heading recognition from canonical section selection.
- leading/trailing outer whitespace and multiple spaces before a slash;
- exact qualifier expectations showing that legacy parentheses remain and 1. In `internal/providers/nws/forecast_discussion.go`, replace the four-name
slash delimiters are removed; and regular-expression alternation with a generic heading parser. Keep the
- negative cases for an unknown identity, missing opening or closing slash, `forecastDiscussionSectionHeading` result, with normalized `section` and
an embedded slash, a whitespace-only slash qualifier, missing required `qualifier` fields.
whitespace before the slash, missing final ellipsis, trailing text after a 2. Implement the generic parser with these decisions:
slash-qualified heading, and a lowercase section identity.
- trim outer whitespace and require a leading `.`;
- try the slash-qualified form before the legacy ellipsis-first form so the
terminal ellipsis of an all-uppercase slash heading cannot be mistaken for
a legacy identity;
- accept identities made only from uppercase ASCII letters, digits,
horizontal whitespace, `/`, `&`, apostrophes, and hyphens, with at least
one letter or digit;
- trim the identity and collapse every internal run of spaces or tabs to one
ASCII space before returning it or performing role lookup;
- accept optional horizontal whitespace between the identity and the legacy
`...` delimiter;
- for legacy headings, store trimmed text after the first `...` as the
qualifier, preserving parentheses and permitting an empty value;
- for slash-qualified headings, remove the terminal `/...`, then use the
first slash preceded by horizontal whitespace as the opening separator;
trim the identity before that separator and the qualifier after it, reject
an empty qualifier, and allow additional `/` characters inside the
qualifier; slashes inside an identity must therefore be adjacent to its
other identity characters rather than preceded by whitespace;
- reject trailing text after a slash-qualified terminal, lowercase or mixed
case identities, missing delimiters, and lines whose identity contains
other punctuation; and
- keep ASCII `...` as the only delimiter because that is the raw product
convention; do not interpret a Unicode ellipsis.
3. Replace the duplicated identity constants and identity-pattern string with:
- an unexported `forecastDiscussionSectionRole` enum for key messages, short
term, and long term; and
- one `map[string]forecastDiscussionSectionRole` containing exactly `KEY
MESSAGES`, `SHORT TERM`, and `LONG TERM`.
Unknown identities and `AVIATION` intentionally have no role entry; successful
structural parsing is sufficient for boundary behavior.
4. Update the heading table test in
`internal/providers/nws/forecast_discussion_test.go`. Preserve every legacy
positive case and revise the old policy-specific negatives:
- `.SYNOPSIS...`, `.UPDATE...`, `.MARINE...`, `.HYDROLOGY...`, an office
watch/advisory heading, and `.PRELIMINARY POINT TEMPS/POPS ...` are valid
generic headings;
- a slash-qualified heading whose qualifier contains `/` is valid;
- leading/trailing whitespace, multiple separator spaces, repeated internal
identity whitespace, and whitespace before a legacy ellipsis are valid and
produce the normalized identity; and
- lowercase prose, an empty or punctuation-only identity, malformed slash
terminals, missing ellipses, and unsupported identity punctuation remain
invalid.
5. Add assertions that every role-registry key parses successfully, and confirm
that no production switch or second collection repeats the mapped identity
list.
Run and pass: Run and pass:
```sh ```sh
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
go test ./internal/providers/nws go test -count=1 ./internal/providers/nws
``` ```
Do not proceed until legacy provider tests still pass and the centralized Do not proceed until generic syntax tests pass without changing canonical
heading test covers every accepted identity in both forms. models or schemas.
## Stage 2: Add Provider and Normalizer Regression Coverage ## Stage 5: Scan Ordered Blocks Once and Use Generic Boundaries
Prove section discovery, boundary behavior, and canonical propagation using the Replace repeated per-identity extraction with one structural pass.
centralized parser. No production normalizer changes should be necessary.
1. In `internal/providers/nws/forecast_discussion_test.go`, add focused tests of 1. Replace `extractForecastDiscussionSection` with
`extractForecastDiscussionSection` using small line slices without `&&` `parseForecastDiscussionSectionBlocks(lines []string) []forecastDiscussionSectionBlock`.
between sections. Cover: The scanner must:
- a slash-qualified `LONG TERM` block followed by a slash-qualified - traverse lines once in source order;
`AVIATION` heading, asserting that the long-term qualifier is normalized, - start a block on every structurally valid heading;
only long-term prose is returned, and aviation heading/body text is - finish the active block before a new heading;
excluded; - finish and clear the active block on `&&`;
- a recognized heading immediately following another recognized heading, - finish the active block and stop scanning on `$$`;
asserting that the first block has an empty body; and - retain the existing watch/advisory termination safeguard even though a
- unknown and malformed heading-like lines inside a block, asserting that normal dotted watch/advisory heading is structurally recognized;
they remain in the body and do not terminate it before a real terminator or - ignore preamble lines before the first heading and signature lines after
recognized heading. `$$`; and
2. Add a provider end-to-end regression based on - append original, untrimmed body lines to the active block.
`testdata/forecast_discussion_sample.html`. Derive the input in the test by 2. Refactor `ParseForecastDiscussionText` to iterate over the ordered blocks
replacing selected fixture headings rather than duplicating the full HTML once and look up each heading identity in the role registry. Ignore blocks
fixture. Guard each replacement with an assertion that its original heading with no role. Populate each mapped output only if it has not already been
exists so fixture drift cannot make the test pass without exercising the new populated, so the first occurrence wins. Track seen roles explicitly rather
syntax. than inferring them from output values, because an empty first key-message
3. Make that derived bulletin intentionally mix heading families: retain the block still counts as the first occurrence.
legacy `KEY MESSAGES` heading, convert `SHORT TERM` and `LONG TERM` to 3. Preserve contextual errors when a mapped text block fails to parse; include
slash-qualified headings, and convert `AVIATION` to a slash-qualified the parsed heading identity in the wrapped error.
heading. Parse it through `ParseForecastDiscussionHTML` and assert: 4. Replace extraction tests with scanner tests covering:
- key messages are unchanged; - consecutive headings and empty bodies;
- short- and long-term sections are non-nil; - `&&`, `$$`, and watch/advisory termination;
- their qualifiers equal the inner slash text with no slash delimiters or - unknown valid headings terminating short- or long-term content even when
legacy parentheses; no `&&` is present;
- their existing issued times and representative prose remain intact; and - malformed heading-like lines remaining inside the active body;
- long-term text contains no aviation heading or aviation prose. - preamble and post-signature exclusion;
4. In `internal/normalizers/nws/forecast_discussion_test.go`, add a regression - ordered block retention; and
that derives the same mixed-format HTML from the shared fixture and passes it - duplicate mapped sections where the parser keeps the first canonical
through `ForecastDiscussionNormalizer.Normalize`. Assert the existing kind, occurrence.
canonical schema, and effective time; exact normalized short- and long-term
qualifiers; representative section text; and absence of aviation content Run and pass:
from long-term text.
5. Marshal the slash-qualified normalizer result and assert that it has no ```sh
top-level `aviation` or generic `sections` key. Do not add or alter canonical gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
model fields to satisfy this test. go test -count=1 ./internal/providers/nws
```
## Stage 6: Normalize Multiline Preambles and NWS Presentation Markers
Handle real section-layout variation without weakening heading syntax.
1. Add an exact provider-local marker predicate for lines whose trimmed value is
`-- Changed Discussion --` or `-- End Changed Discussion --`, compared
case-insensitively. Add a helper that removes only those complete marker
lines from a block body. Do not remove arbitrary dashed lines or bullet text.
2. Apply marker removal before parsing both key-message and text-section bodies.
3. Refactor text-section preamble parsing in this exact order:
- trim blank lines after marker removal;
- start with the qualifier parsed from the heading;
- only when that qualifier is empty, consume the first content line as a
qualifier if its trimmed value is a nonempty standalone parenthetical
string beginning with `(` and ending with `)`; preserve the parentheses;
- after the optional qualifier, consume an optional `Issued at` line and
parse it into `ForecastDiscussionSection.IssuedAt`; and
- pass only the remaining lines to existing signature trimming and paragraph
joining.
4. Make recognition of the `Issued at` label ASCII case-insensitive while
retaining the existing timestamp grammar and error behavior. The top-level
header path, which passes an unlabeled timestamp, must remain compatible.
5. Before key-message bullet parsing, remove markers, trim blank lines, and
discard at most one leading metadata line beginning case-insensitively with
`Issued at` or `Updated at`. Do not discard timestamp-like lines after the
first message begins.
6. Add focused tests for:
- same-line legacy and slash qualifiers remaining unchanged;
- next-line parenthesized qualifiers followed by `Issued at`;
- uppercase `ISSUED AT`;
- a heading qualifier taking precedence over a following parenthetical prose
line;
- empty sections;
- invalid issue timestamps retaining contextual errors;
- marker removal at the beginning and end of text sections;
- key-message markers and a leading `Updated at` line not becoming messages;
and
- arbitrary dashed prose remaining content.
Run and pass:
```sh
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
go test -count=1 ./internal/providers/nws
```
## Stage 7: Add Cross-Office Fixtures and Normalizer Regressions
Prove the resilient parser against representative source shapes rather than only
synthetic heading replacement.
1. Retain the existing LSX fixture and mixed-format test for regression
compatibility.
2. Add one compact, maintained HTML fixture under
`internal/providers/nws/testdata/` representing a second real NWS formatting
family. It must contain:
- a valid AFD header and issue time;
- key-message change markers plus a leading `Updated at` line;
- ellipsis-first short- and long-term headings with qualifiers on the next
line;
- `Issued at` lines following those qualifiers;
- at least one structurally valid boundary-only section; and
- representative prose sufficient to detect metadata or adjacent-section
leakage.
Keep the fixture concise; include no unrelated webpage content, secrets, or
private data.
3. Add a provider end-to-end test that parses the new fixture and asserts exact
key messages, qualifiers, section issue times, representative prose, and
absence of change markers, timestamp metadata, and boundary-only content.
4. Add a normalizer regression using the same fixture. Verify kind, canonical
schema, envelope/effective-time behavior, short- and long-term mapping, and
JSON wire shape without `aviation`, `discussion`, or generic `sections`
fields.
5. Keep small parser and scanner edge cases table-driven. Do not multiply full
fixtures for cases that a short line slice proves more clearly.
Run and pass: Run and pass:
```sh ```sh
gofmt -w internal/providers/nws/forecast_discussion_test.go internal/normalizers/nws/forecast_discussion_test.go gofmt -w internal/providers/nws/forecast_discussion_test.go internal/normalizers/nws/forecast_discussion_test.go
go test ./internal/providers/nws ./internal/normalizers/nws go test -count=1 ./internal/providers/nws ./internal/normalizers/nws
``` ```
Do not introduce a second full-bulletin fixture unless the checked-in fixture ## Stage 8: Reconcile Documentation and Perform Final Validation
cannot express a required interaction through guarded heading replacement.
## Stage 3: Validate the Feature and Close the Roadmap Close the resilience follow-up only after all behavior is proven.
Verify the complete change against repository policy and record completion only 1. Review the final diff. Production changes must remain confined to the NWS
after all behavior is proven. provider parser; other Go changes should be tests in the owning provider and
normalizer packages. Preserve all unrelated user work.
1. Review the final diff and confirm production changes are confined to the NWS 2. Confirm the public contract is unchanged. Do not edit canonical model,
provider parser. Test changes should be confined to the owning provider and schema, Postgres, config, consumer, or integration documentation unless an
NWS normalizer packages. Do not retain incidental model, schema, config, actual contract change is discovered. If one appears necessary, stop rather
source, sink, Postgres, or current-behavior documentation edits introduced than expanding this plan.
during implementation, and preserve all pre-existing user work. 3. Run:
2. Run formatting on every changed Go file, then run uncached focused tests and
the full repository suite:
```sh ```sh
gofmt -w \ gofmt -w \
@@ -184,24 +267,18 @@ after all behavior is proven.
internal/normalizers/nws/forecast_discussion_test.go internal/normalizers/nws/forecast_discussion_test.go
go test -count=1 ./internal/providers/nws ./internal/normalizers/nws go test -count=1 ./internal/providers/nws ./internal/normalizers/nws
go test -count=1 ./... go test -count=1 ./...
go vet ./...
git diff --check git diff --check
``` ```
3. Confirm each acceptance criterion in 4. Verify every acceptance criterion in the feature roadmap has a corresponding
[`afd-section-heading-variants.md`](afd-section-heading-variants.md) is passing automated test, including compatibility with all Stage 1-2 cases.
represented by a passing automated test. In particular, verify legacy 5. After all checks pass, change the feature roadmap status to `Implemented.`
compatibility, mixed-format input, slash-qualified aviation termination, and rewrite any remaining future-tense statements that would misdescribe the
malformed-line non-recognition, normalized qualifiers, and unchanged completed parser. Do not mark the follow-up implemented earlier.
canonical wire shape.
4. Because this feature does not alter a public contract, do not change
`docs/integrations/events.md`, `docs/integrations/postgres.md`, consumer docs,
configuration docs, or examples. If implementation appears to require such
a change, stop and reassess the implementation against the feature roadmap
instead of expanding scope.
5. After all checks pass, change the feature roadmap status from `Proposed and
unimplemented.` to `Implemented.` Do not mark it implemented earlier.
## Open Questions ## Open Questions
None. The feature roadmap and this plan fix the accepted grammar, normalization None. This plan fixes the parser grammar, boundary policy, canonical role
rules, architectural boundary, test coverage, and canonical compatibility selection, multiline preamble rules, presentation cleanup, fixture strategy,
requirements. and compatibility boundary. Generic or single-section discussion content remains
outside the canonical model by explicit policy.

128
internal/providers/nws/forecast_discussion.go
View File

@@ -27,13 +27,12 @@ type ForecastDiscussionSection struct {
Text string Text string
} }
const ( type forecastDiscussionSectionRole uint8
forecastDiscussionSectionKeyMessages = "KEY MESSAGES"
forecastDiscussionSectionShortTerm = "SHORT TERM"
forecastDiscussionSectionLongTerm = "LONG TERM"
forecastDiscussionSectionAviation = "AVIATION"
forecastDiscussionSectionIdentityPattern = "KEY MESSAGES|SHORT TERM|LONG TERM|AVIATION" const (
forecastDiscussionSectionRoleKeyMessages forecastDiscussionSectionRole = iota
forecastDiscussionSectionRoleShortTerm
forecastDiscussionSectionRoleLongTerm
) )
type forecastDiscussionSectionHeading struct { type forecastDiscussionSectionHeading struct {
@@ -47,8 +46,11 @@ type forecastDiscussionSectionBlock struct {
} }
var ( var (
forecastDiscussionEllipsisHeadingRE = regexp.MustCompile(`^\.(` + forecastDiscussionSectionIdentityPattern + `)\.\.\.(.*)$`) forecastDiscussionSectionRoles = map[string]forecastDiscussionSectionRole{
forecastDiscussionSlashHeadingRE = regexp.MustCompile(`^\.(` + forecastDiscussionSectionIdentityPattern + `)[ \t]+/([^/]+)/\.\.\.$`) "KEY MESSAGES": forecastDiscussionSectionRoleKeyMessages,
"SHORT TERM": forecastDiscussionSectionRoleShortTerm,
"LONG TERM": forecastDiscussionSectionRoleLongTerm,
}
forecastDiscussionAFDRE = regexp.MustCompile(`^AFD([A-Z]{3})$`) forecastDiscussionAFDRE = regexp.MustCompile(`^AFD([A-Z]{3})$`)
forecastDiscussionWMORE = regexp.MustCompile(`\bK([A-Z]{3})\b`) forecastDiscussionWMORE = regexp.MustCompile(`\bK([A-Z]{3})\b`)
forecastDiscussionSigRE = regexp.MustCompile(`^[A-Z]{2,6}$`) forecastDiscussionSigRE = regexp.MustCompile(`^[A-Z]{2,6}$`)
@@ -119,20 +121,20 @@ func ParseForecastDiscussionText(text string) (ForecastDiscussion, error) {
IssuedAt: issuedAt.UTC(), IssuedAt: issuedAt.UTC(),
} }
if block, ok := extractForecastDiscussionSection(lines, forecastDiscussionSectionKeyMessages); ok { if block, ok := extractForecastDiscussionSection(lines, forecastDiscussionSectionForRole(forecastDiscussionSectionRoleKeyMessages)); ok {
out.KeyMessages = parseForecastDiscussionKeyMessages(block.body) out.KeyMessages = parseForecastDiscussionKeyMessages(block.body)
} }
if block, ok := extractForecastDiscussionSection(lines, forecastDiscussionSectionShortTerm); ok { if block, ok := extractForecastDiscussionSection(lines, forecastDiscussionSectionForRole(forecastDiscussionSectionRoleShortTerm)); ok {
section, err := parseForecastDiscussionTextSection(block) section, err := parseForecastDiscussionTextSection(block)
if err != nil { if err != nil {
return ForecastDiscussion{}, fmt.Errorf("parse %s: %w", forecastDiscussionSectionShortTerm, err) return ForecastDiscussion{}, fmt.Errorf("parse %s: %w", block.heading.section, err)
} }
out.ShortTerm = &section out.ShortTerm = &section
} }
if block, ok := extractForecastDiscussionSection(lines, forecastDiscussionSectionLongTerm); ok { if block, ok := extractForecastDiscussionSection(lines, forecastDiscussionSectionForRole(forecastDiscussionSectionRoleLongTerm)); ok {
section, err := parseForecastDiscussionTextSection(block) section, err := parseForecastDiscussionTextSection(block)
if err != nil { if err != nil {
return ForecastDiscussion{}, fmt.Errorf("parse %s: %w", forecastDiscussionSectionLongTerm, err) return ForecastDiscussion{}, fmt.Errorf("parse %s: %w", block.heading.section, err)
} }
out.LongTerm = &section out.LongTerm = &section
} }
@@ -408,23 +410,103 @@ func forecastDiscussionLocation(abbrev string) (*time.Location, error) {
func parseForecastDiscussionSectionHeading(line string) (forecastDiscussionSectionHeading, bool) { func parseForecastDiscussionSectionHeading(line string) (forecastDiscussionSectionHeading, bool) {
line = strings.TrimSpace(line) line = strings.TrimSpace(line)
if m := forecastDiscussionEllipsisHeadingRE.FindStringSubmatch(line); len(m) == 3 { if len(line) < 2 || line[0] != '.' {
return forecastDiscussionSectionHeading{ return forecastDiscussionSectionHeading{}, false
section: m[1],
qualifier: strings.TrimSpace(m[2]),
}, true
} }
if m := forecastDiscussionSlashHeadingRE.FindStringSubmatch(line); len(m) == 3 {
qualifier := strings.TrimSpace(m[2]) if strings.HasSuffix(line, "/...") {
return parseForecastDiscussionSlashQualifiedHeading(line)
}
return parseForecastDiscussionEllipsisHeading(line)
}
func parseForecastDiscussionSlashQualifiedHeading(line string) (forecastDiscussionSectionHeading, bool) {
content := strings.TrimSuffix(line[1:], "/...")
separator := -1
for i := 1; i < len(content); i++ {
if content[i] == '/' && isForecastDiscussionHorizontalWhitespace(content[i-1]) {
separator = i
break
}
}
if separator < 0 {
return forecastDiscussionSectionHeading{}, false
}
section, ok := normalizeForecastDiscussionSectionIdentity(content[:separator])
if !ok {
return forecastDiscussionSectionHeading{}, false
}
qualifier := strings.TrimSpace(content[separator+1:])
if qualifier == "" { if qualifier == "" {
return forecastDiscussionSectionHeading{}, false return forecastDiscussionSectionHeading{}, false
} }
return forecastDiscussionSectionHeading{section: section, qualifier: qualifier}, true
}
func parseForecastDiscussionEllipsisHeading(line string) (forecastDiscussionSectionHeading, bool) {
content := line[1:]
delimiter := strings.Index(content, "...")
if delimiter < 0 {
return forecastDiscussionSectionHeading{}, false
}
section, ok := normalizeForecastDiscussionSectionIdentity(content[:delimiter])
if !ok {
return forecastDiscussionSectionHeading{}, false
}
return forecastDiscussionSectionHeading{ return forecastDiscussionSectionHeading{
section: m[1], section: section,
qualifier: qualifier, qualifier: strings.TrimSpace(content[delimiter+3:]),
}, true }, true
} }
return forecastDiscussionSectionHeading{}, false
func normalizeForecastDiscussionSectionIdentity(raw string) (string, bool) {
var normalized strings.Builder
pendingSpace := false
hasLetterOrDigit := false
for i := 0; i < len(raw); i++ {
b := raw[i]
switch {
case isForecastDiscussionIdentityLetterOrDigit(b):
hasLetterOrDigit = true
case b == ' ' || b == '\t':
pendingSpace = normalized.Len() > 0
continue
case b == '/' && i > 0 && isForecastDiscussionHorizontalWhitespace(raw[i-1]):
return "", false
case b != '/' && b != '&' && b != '\'' && b != '-':
return "", false
}
if pendingSpace {
normalized.WriteByte(' ')
pendingSpace = false
}
normalized.WriteByte(b)
}
if !hasLetterOrDigit {
return "", false
}
return normalized.String(), true
}
func isForecastDiscussionIdentityLetterOrDigit(b byte) bool {
return b >= 'A' && b <= 'Z' || b >= '0' && b <= '9'
}
func isForecastDiscussionHorizontalWhitespace(b byte) bool {
return b == ' ' || b == '\t'
}
func forecastDiscussionSectionForRole(role forecastDiscussionSectionRole) string {
for section, registeredRole := range forecastDiscussionSectionRoles {
if registeredRole == role {
return section
}
}
return ""
} }
func extractForecastDiscussionSection(lines []string, section string) (forecastDiscussionSectionBlock, bool) { func extractForecastDiscussionSection(lines []string, section string) (forecastDiscussionSectionBlock, bool) {

168
internal/providers/nws/forecast_discussion_test.go
View File

@@ -20,88 +20,150 @@ func TestParseForecastDiscussionSectionHeading(t *testing.T) {
{ {
name: "ellipsis-first key messages", name: "ellipsis-first key messages",
line: ".KEY MESSAGES... (Through Tonight)", line: ".KEY MESSAGES... (Through Tonight)",
wantSection: forecastDiscussionSectionKeyMessages, wantSection: "KEY MESSAGES",
wantQualifier: "(Through Tonight)", wantQualifier: "(Through Tonight)",
wantOK: true, wantOK: true,
}, },
{ {
name: "ellipsis-first short term", name: "ellipsis-first short term",
line: ".SHORT TERM... (Sunday)", line: ".SHORT TERM... (Sunday)",
wantSection: forecastDiscussionSectionShortTerm, wantSection: "SHORT TERM",
wantQualifier: "(Sunday)", wantQualifier: "(Sunday)",
wantOK: true, wantOK: true,
}, },
{ {
name: "ellipsis-first long term", name: "ellipsis-first long term",
line: ".LONG TERM... (Monday Through Friday)", line: ".LONG TERM... (Monday Through Friday)",
wantSection: forecastDiscussionSectionLongTerm, wantSection: "LONG TERM",
wantQualifier: "(Monday Through Friday)", wantQualifier: "(Monday Through Friday)",
wantOK: true, wantOK: true,
}, },
{ {
name: "ellipsis-first aviation without qualifier", name: "ellipsis-first aviation without qualifier",
line: ".AVIATION...", line: ".AVIATION...",
wantSection: forecastDiscussionSectionAviation, wantSection: "AVIATION",
wantOK: true, wantOK: true,
}, },
{ {
name: "slash-qualified key messages", name: "slash-qualified key messages",
line: ".KEY MESSAGES /Tonight/...", line: ".KEY MESSAGES /Tonight/...",
wantSection: forecastDiscussionSectionKeyMessages, wantSection: "KEY MESSAGES",
wantQualifier: "Tonight", wantQualifier: "Tonight",
wantOK: true, wantOK: true,
}, },
{ {
name: "slash-qualified short term with outer whitespace", name: "slash-qualified short term with outer whitespace",
line: " .SHORT TERM /Through Late Sunday Night/... ", line: " .SHORT TERM /Through Late Sunday Night/... ",
wantSection: forecastDiscussionSectionShortTerm, wantSection: "SHORT TERM",
wantQualifier: "Through Late Sunday Night", wantQualifier: "Through Late Sunday Night",
wantOK: true, wantOK: true,
}, },
{ {
name: "slash-qualified long term", name: "slash-qualified long term",
line: ".LONG TERM /Monday through Next Saturday/...", line: ".LONG TERM /Monday through Next Saturday/...",
wantSection: forecastDiscussionSectionLongTerm, wantSection: "LONG TERM",
wantQualifier: "Monday through Next Saturday", wantQualifier: "Monday through Next Saturday",
wantOK: true, wantOK: true,
}, },
{ {
name: "slash-qualified aviation", name: "slash-qualified aviation",
line: ".AVIATION /18Z TAFS/...", line: ".AVIATION /18Z TAFS/...",
wantSection: forecastDiscussionSectionAviation, wantSection: "AVIATION",
wantQualifier: "18Z TAFS", wantQualifier: "18Z TAFS",
wantOK: true, wantOK: true,
}, },
{ {
name: "unknown identity", name: "generic synopsis",
line: ".SYNOPSIS... Overview", line: ".SYNOPSIS... Overview",
wantSection: "SYNOPSIS",
wantQualifier: "Overview",
wantOK: true,
},
{
name: "generic update",
line: ".UPDATE...",
wantSection: "UPDATE",
wantOK: true,
},
{
name: "generic marine",
line: ".MARINE...",
wantSection: "MARINE",
wantOK: true,
},
{
name: "generic hydrology",
line: ".HYDROLOGY...",
wantSection: "HYDROLOGY",
wantOK: true,
},
{
name: "office watch advisory heading",
line: ".LSX WATCHES/WARNINGS/ADVISORIES...",
wantSection: "LSX WATCHES/WARNINGS/ADVISORIES",
wantOK: true,
},
{
name: "preliminary point temperatures heading",
line: ".PRELIMINARY POINT TEMPS/POPS ...",
wantSection: "PRELIMINARY POINT TEMPS/POPS",
wantOK: true,
},
{
name: "generic identity punctuation and digits",
line: ".DAY 1 FIRE-WEATHER & HYDROLOGY'S/OUTLOOK...",
wantSection: "DAY 1 FIRE-WEATHER & HYDROLOGY'S/OUTLOOK",
wantOK: true,
},
{
name: "slash qualifier contains slash",
line: ".LONG TERM /Tonight/Sunday/...",
wantSection: "LONG TERM",
wantQualifier: "Tonight/Sunday",
wantOK: true,
},
{
name: "normalized identity whitespace",
line: " .SHORT\t\tTERM \t... (Tonight) ",
wantSection: "SHORT TERM",
wantQualifier: "(Tonight)",
wantOK: true,
},
{
name: "lowercase prose",
line: ".This is ordinary prose...",
wantOK: false, wantOK: false,
}, },
{ {
name: "slash qualifier missing opening slash", name: "empty identity",
line: "....",
wantOK: false,
},
{
name: "punctuation only identity",
line: ".-/&'...",
wantOK: false,
},
{
name: "slash qualifier missing opening separator",
line: ".SHORT TERM Through Tonight/...", line: ".SHORT TERM Through Tonight/...",
wantOK: false, wantOK: false,
}, },
{
name: "slash qualifier missing closing slash",
line: ".SHORT TERM /Through Tonight...",
wantOK: false,
},
{
name: "slash qualifier contains embedded slash",
line: ".SHORT TERM /Tonight/Sunday/...",
wantOK: false,
},
{
name: "slash qualifier whitespace only",
line: ".SHORT TERM / /...",
wantOK: false,
},
{ {
name: "slash qualifier missing required whitespace", name: "slash qualifier missing required whitespace",
line: ".SHORT TERM/Through Tonight/...", line: ".SHORT TERM/Through Tonight/...",
wantOK: false, wantOK: false,
}, },
{
name: "slash qualifier missing closing terminal",
line: ".SHORT TERM /Through Tonight...",
wantOK: false,
},
{
name: "slash qualifier whitespace only",
line: ".SHORT TERM / /...",
wantOK: false,
},
{ {
name: "slash qualifier missing final ellipsis", name: "slash qualifier missing final ellipsis",
line: ".SHORT TERM /Through Tonight/..", line: ".SHORT TERM /Through Tonight/..",
@@ -113,8 +175,23 @@ func TestParseForecastDiscussionSectionHeading(t *testing.T) {
wantOK: false, wantOK: false,
}, },
{ {
name: "lowercase identity", name: "missing ellipsis",
line: ".short term... (Tonight)", line: ".SHORT TERM",
wantOK: false,
},
{
name: "unsupported identity punctuation",
line: ".SHORT TERM:...",
wantOK: false,
},
{
name: "unicode ellipsis",
line: ".SHORT TERM…",
wantOK: false,
},
{
name: "mixed case identity",
line: ".Short Term... (Tonight)",
wantOK: false, wantOK: false,
}, },
} }
@@ -138,13 +215,34 @@ func TestParseForecastDiscussionSectionHeading(t *testing.T) {
} }
} }
func TestForecastDiscussionSectionRoleHeadingsParse(t *testing.T) {
if len(forecastDiscussionSectionRoles) != 3 {
t.Fatalf("role registry has %d entries, want 3", len(forecastDiscussionSectionRoles))
}
if _, ok := forecastDiscussionSectionRoles["AVIATION"]; ok {
t.Fatalf("AVIATION must remain boundary-only")
}
for section := range forecastDiscussionSectionRoles {
t.Run(section, func(t *testing.T) {
got, ok := parseForecastDiscussionSectionHeading("." + section + "...")
if !ok {
t.Fatalf("parseForecastDiscussionSectionHeading() did not recognize %q", section)
}
if got.section != section {
t.Fatalf("section = %q, want %q", got.section, section)
}
})
}
}
func TestExtractForecastDiscussionSectionStopsAtSlashQualifiedHeading(t *testing.T) { func TestExtractForecastDiscussionSectionStopsAtSlashQualifiedHeading(t *testing.T) {
block, ok := extractForecastDiscussionSection([]string{ block, ok := extractForecastDiscussionSection([]string{
".LONG TERM /Monday through Next Saturday/...", ".LONG TERM /Monday through Next Saturday/...",
"Long-term prose.", "Long-term prose.",
".AVIATION /18Z TAFS/...", ".AVIATION /18Z TAFS/...",
"Aviation prose.", "Aviation prose.",
}, forecastDiscussionSectionLongTerm) }, forecastDiscussionSectionForRole(forecastDiscussionSectionRoleLongTerm))
if !ok { if !ok {
t.Fatalf("extractForecastDiscussionSection() found no LONG TERM block") t.Fatalf("extractForecastDiscussionSection() found no LONG TERM block")
} }
@@ -162,7 +260,7 @@ func TestExtractForecastDiscussionSectionAllowsEmptyBodyBeforeHeading(t *testing
".SHORT TERM... (Tonight)", ".SHORT TERM... (Tonight)",
".LONG TERM... (Tomorrow)", ".LONG TERM... (Tomorrow)",
"Long-term prose.", "Long-term prose.",
}, forecastDiscussionSectionShortTerm) }, forecastDiscussionSectionForRole(forecastDiscussionSectionRoleShortTerm))
if !ok { if !ok {
t.Fatalf("extractForecastDiscussionSection() found no SHORT TERM block") t.Fatalf("extractForecastDiscussionSection() found no SHORT TERM block")
} }
@@ -171,23 +269,23 @@ func TestExtractForecastDiscussionSectionAllowsEmptyBodyBeforeHeading(t *testing
} }
} }
func TestExtractForecastDiscussionSectionKeepsUnrecognizedHeadingLikeLines(t *testing.T) { func TestExtractForecastDiscussionSectionKeepsMalformedHeadingLikeLines(t *testing.T) {
block, ok := extractForecastDiscussionSection([]string{ block, ok := extractForecastDiscussionSection([]string{
".SHORT TERM... (Tonight)", ".SHORT TERM... (Tonight)",
".SYNOPSIS... This unknown section stays in the body.", ".short term... Lowercase prose stays in the body.",
".LONG TERM /Tomorrow/..", ".LONG TERM /Tomorrow/..",
".LONG TERM /Tomorrow/Next Week/...", ".LONG TERM /Tomorrow/... more",
"Expected short-term prose.", "Expected short-term prose.",
".LONG TERM... (Tomorrow)", ".LONG TERM... (Tomorrow)",
"Long-term prose.", "Long-term prose.",
}, forecastDiscussionSectionShortTerm) }, forecastDiscussionSectionForRole(forecastDiscussionSectionRoleShortTerm))
if !ok { if !ok {
t.Fatalf("extractForecastDiscussionSection() found no SHORT TERM block") t.Fatalf("extractForecastDiscussionSection() found no SHORT TERM block")
} }
wantBody := []string{ wantBody := []string{
".SYNOPSIS... This unknown section stays in the body.", ".short term... Lowercase prose stays in the body.",
".LONG TERM /Tomorrow/..", ".LONG TERM /Tomorrow/..",
".LONG TERM /Tomorrow/Next Week/...", ".LONG TERM /Tomorrow/... more",
"Expected short-term prose.", "Expected short-term prose.",
} }
if !reflect.DeepEqual(block.body, wantBody) { if !reflect.DeepEqual(block.body, wantBody) {