weatherfeeder/docs/roadmap/afd-section-heading-variants.md

NWS AFD Section Parsing Resilience

Status

Implemented.

Implemented Behavior

The NWS Area Forecast Discussion parser recognizes these heading families for all structurally valid section identities:

.<SECTION>...<optional qualifier>
.<SECTION> /<qualifier>/...

Discovery, qualifier extraction, and recognized-section boundary detection use one provider-specific parser. Slash delimiters are removed from qualifiers, legacy qualifier text is preserved, and aviation remains a boundary rather than a canonical field.

The parser separates:

1. generic structural heading recognition;
2. section boundary scanning;
3. canonical section-role selection; and
4. section-preamble and presentation cleanup.

Structurally valid topic names act as safe boundaries without becoming new canonical fields.

Implemented Details

Generic heading recognition

One NWS-specific heading parser recognizes structurally valid uppercase AFD topic lines independently of the canonical model. It supports:

.<IDENTITY>...<optional qualifier>
.<IDENTITY> /<qualifier>/...

The identity may contain uppercase ASCII letters, digits, horizontal whitespace, /, &, apostrophes, and hyphens. Outer whitespace and whitespace immediately before the ellipsis are ignored, and internal identity whitespace is collapsed to one space for role lookup. Slash-qualified headings require a nonempty qualifier but may contain slash characters inside that qualifier. Lowercase prose, ordinary dotted lines, and malformed delimiters remain non-headings.

Every structurally valid heading terminates the preceding section. This applies to known boundary-only sections such as aviation and to future or office-specific topic identities that weatherfeeder does not map.

Canonical role selection

A single provider-local role registry maps only these identities into existing parsed fields:

• KEY MESSAGES to key messages;
• SHORT TERM to the short-term section; and
• LONG TERM to the long-term section.

All other identities are boundary-only. The first occurrence of each mapped identity wins, preserving current behavior if an unusual bulletin repeats a section. Heading syntax and canonical roles do not duplicate identity lists.

Section scanning

The discussion text is scanned once into ordered section blocks. A new heading, &&, $$, or the existing watch/advisory terminator ends the active block. Original body lines are preserved until provider presentation cleanup is applied. Preamble text before the first heading and post-$$ signatures are not treated as section content.

Preamble and presentation handling

For short- and long-term sections:

• a qualifier on the heading line remains authoritative;
• when the heading has no qualifier, a standalone parenthesized first content line becomes the qualifier and retains its parentheses;
• an optional case-insensitive Issued at line after the qualifier is parsed into the existing section issue time; and
• exact NWS change-presentation marker lines are removed without removing arbitrary dashed prose.

For key messages, exact change-presentation markers and one leading Issued at or Updated at metadata line are removed before bullet parsing. Those metadata lines never become key messages.

Representative coverage

Tests include small structural tables and maintained local HTML fixtures for at least two real NWS formatting families: slash-qualified same-line headings and ellipsis-first headings with multiline qualifiers or change-presentation markers. Tests remain deterministic and never contact live services.

Compatibility and Contracts

This remains a provider-parsing compatibility improvement. It does not change:

• event kinds or raw and canonical schema identifiers;
• canonical models or JSON field names;
• source configuration, URLs, or polling behavior;
• event envelope or effective-time behavior;
• Postgres tables or event-to-row mapping; or
• downstream sink and consumer responsibilities.

A single DISCUSSION section and other currently unmapped identities are recognized as boundaries but are not forced into short- or long-term fields. Exposing such content would require a separate canonical schema decision.

Verified Coverage

Automated tests demonstrate that:

• every previously accepted heading and canonical result remains compatible;
• generic structurally valid identities terminate preceding content without becoming canonical fields;
• malformed heading-like lines and lowercase prose remain body content;
• identities containing / and slash qualifiers containing / are parsed without ambiguity;
• next-line parenthesized qualifiers and following Issued at lines populate the existing short- and long-term metadata correctly;
• exact NWS change markers and leading key-message timestamps do not leak into canonical prose or messages;
• repeated mapped sections preserve first-occurrence behavior;
• genuine cross-office fixture styles propagate correctly through the NWS normalizer with unchanged wire shape; and
• focused tests, the full repository suite, and static analysis pass.

Non-Goals

This follow-up does not:

• add canonical discussion, aviation, marine, hydrology, climate, fire weather, update, or arbitrary-section fields;
• infer short- or long-term semantics from an unknown heading;
• accept free-form or lowercase prose as a heading;
• introduce heuristic summarization;
• preserve complete raw AFD documents in canonical payloads;
• change schemas, persistence contracts, configuration, or downstream APIs; or
• fetch live NWS data during tests.

Additional canonical support for AFD section identities requires a separate consumer requirement and schema roadmap.