333 lines
15 KiB
Markdown
333 lines
15 KiB
Markdown
# NWS AFD Section Parsing Resilience Implementation Plan
|
|
|
|
## Purpose
|
|
|
|
All resilience work defined in
|
|
[`afd-section-heading-variants.md`](afd-section-heading-variants.md) is
|
|
complete without changing the canonical forecast-discussion contract. This plan
|
|
preserves the implementation details as a historical record.
|
|
|
|
## Cross-Stage Constraints
|
|
|
|
- Keep production parsing changes in
|
|
`internal/providers/nws/forecast_discussion.go`.
|
|
- Use only the Go standard library and keep new helpers unexported.
|
|
- Do not change `model`, `standards`, source configuration or polling, schema
|
|
identifiers, event-envelope behavior, Postgres mapping, sinks, consumer docs,
|
|
or integration contracts.
|
|
- Continue exposing only key messages, short term, and long term.
|
|
- Map `KEY POINTS` to the existing key-message role. Keep `NEAR TERM`,
|
|
`DISCUSSION`, and every other unmapped identity boundary-only.
|
|
- Preserve first-occurrence behavior by canonical role, including across `KEY
|
|
MESSAGES` and `KEY POINTS` aliases.
|
|
- Preserve raw body lines during structural scanning. Apply provider-specific
|
|
cleanup only while parsing a mapped block.
|
|
- Prefer small grammar, label, and list-marker helpers over a broad regular
|
|
expression or general parsing framework.
|
|
- Keep parsing deterministic. Tests must use local strings and fixtures, never
|
|
live NWS requests.
|
|
- Preserve all pre-existing user work and avoid unrelated refactors.
|
|
|
|
## Stage 1: Centralize Known Heading Parsing — Completed
|
|
|
|
The provider parser gained a shared heading classifier and section-block model
|
|
for the original key-message, short-term, long-term, and aviation identities.
|
|
Discovery, boundary handling, and qualifier extraction stopped using separate
|
|
per-section patterns.
|
|
|
|
## Stage 2: Add Original Heading-Variant Regressions — Completed
|
|
|
|
Provider and normalizer tests covered ellipsis-first and slash-qualified
|
|
headings, malformed forms, empty bodies, mixed heading families, and unchanged
|
|
canonical wire shape.
|
|
|
|
## Stage 3: Validate the Original Feature — Completed
|
|
|
|
Focused and repository-wide checks passed, production changes remained inside
|
|
the NWS provider parser, and the original roadmap was reconciled with the
|
|
implemented baseline.
|
|
|
|
## Stage 4: Generalize Structural Heading Recognition — Completed
|
|
|
|
Heading syntax was decoupled from canonical roles. The parser now accepts a
|
|
constrained generic uppercase identity grammar, normalizes identity whitespace,
|
|
supports embedded slashes without confusing slash qualifiers, and recognizes
|
|
unknown valid identities as structural headings.
|
|
|
|
## Stage 5: Scan Ordered Section Blocks Once — Completed
|
|
|
|
Repeated extraction was replaced by a one-pass ordered block scanner. Generic
|
|
headings, `&&`, `$$`, and the watch/advisory safeguard terminate active blocks;
|
|
role lookup happens after scanning and retains first-occurrence behavior.
|
|
|
|
## Stage 6: Normalize Multiline Preambles and Markers — Completed
|
|
|
|
Short- and long-term parsing gained next-line parenthesized qualifiers,
|
|
case-insensitive `Issued at` handling, and exact change-marker removal. Key
|
|
messages gained exact marker cleanup and removal of one leading issue/update
|
|
metadata line.
|
|
|
|
## Stage 7: Add Initial Cross-Office Coverage — Completed
|
|
|
|
The existing LSX fixture was retained and a compact BOU-style fixture added.
|
|
Provider and normalizer regressions cover multiline qualifiers, change markers,
|
|
leading metadata, generic boundary isolation, envelope behavior, and unchanged
|
|
JSON shape.
|
|
|
|
## Stage 8: Validate the First Resilience Follow-up — Completed
|
|
|
|
Focused tests, the full repository suite, static analysis, and diff checks
|
|
passed. The public model and schemas remained unchanged, and the roadmap was
|
|
updated to describe the completed behavior.
|
|
|
|
## Stage 9: Add Parenthesized-Terminal Heading Syntax — Completed
|
|
|
|
Recognize the observed `.<IDENTITY> (<qualifier>)...` family without weakening
|
|
generic identity validation.
|
|
|
|
1. In `internal/providers/nws/forecast_discussion.go`, add a dedicated
|
|
`parseForecastDiscussionParenthesizedTerminalHeading` helper returning the
|
|
existing `forecastDiscussionSectionHeading` type.
|
|
2. Update `parseForecastDiscussionSectionHeading` to try forms in this order:
|
|
|
|
1. if the line ends `/...`, try the slash-qualified helper and return its
|
|
result;
|
|
2. otherwise, try the parenthesized-terminal helper for any line ending
|
|
`...` and return it when it succeeds; and
|
|
3. fall back to the existing ellipsis-first helper.
|
|
|
|
Do not merge the forms into a single regular expression.
|
|
3. Implement the parenthesized-terminal helper with these exact rules:
|
|
|
|
- operate on the already outer-trimmed line and require leading `.` plus a
|
|
terminal ASCII `...`;
|
|
- remove the leading dot and terminal ellipsis, then remove horizontal
|
|
whitespace immediately before the ellipsis;
|
|
- require the remaining content to end with `)`;
|
|
- locate the first `(` that is preceded by horizontal whitespace; everything
|
|
before that separator is the raw identity and everything from `(` through
|
|
the final `)` is the qualifier;
|
|
- normalize and validate the identity through the existing identity helper;
|
|
- require nonempty text after trimming inside the outer parentheses;
|
|
- return the qualifier with its outer parentheses and internal punctuation
|
|
intact; and
|
|
- reject missing separator whitespace, empty qualifiers, missing or misplaced
|
|
parentheses, trailing text after the ellipsis, unsupported identity
|
|
punctuation, and lowercase or mixed-case identities.
|
|
|
|
Parentheses inside the qualifier are content; only the first separator `(`
|
|
and final `)` delimit the outer qualifier.
|
|
4. Extend the heading table in
|
|
`internal/providers/nws/forecast_discussion_test.go` with:
|
|
|
|
- `.DISCUSSION (Today through Thursday)...`;
|
|
- a mapped identity such as `.SHORT TERM (Tonight)...`;
|
|
- identity whitespace normalization;
|
|
- qualifier punctuation and nested parentheses;
|
|
- each rejected malformed form listed above; and
|
|
- compatibility assertions for every existing ellipsis-first and
|
|
slash-qualified case.
|
|
5. Add a scanner regression in which a mapped section is followed immediately,
|
|
without `&&`, by `.DISCUSSION (Today through Thursday)...`. Assert that the
|
|
discussion heading starts a new boundary-only block and its prose cannot leak
|
|
into the mapped section.
|
|
|
|
Run and pass:
|
|
|
|
```sh
|
|
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
|
|
go test -count=1 ./internal/providers/nws
|
|
```
|
|
|
|
Do not proceed until all previous heading and scanner tests still pass.
|
|
|
|
## Stage 10: Add Explicit Canonical Role Aliases — Completed
|
|
|
|
Make true wording synonyms cheap to support while preserving semantic
|
|
distinctions.
|
|
|
|
1. Add `KEY POINTS` to the existing identity-to-role registry with the key-message
|
|
role. Keep the registry as the only production source of identity-to-role
|
|
mappings; do not add a parallel alias collection or role-selection switch.
|
|
2. Leave `KEY MESSAGES`, `SHORT TERM`, and `LONG TERM` mappings unchanged.
|
|
Explicitly do not add role mappings for `NEAR TERM`, `DISCUSSION`,
|
|
`AVIATION`, or other generic identities.
|
|
3. Update registry tests so they no longer assume one identity per role. Assert
|
|
instead that:
|
|
|
|
- every registry identity parses as a heading;
|
|
- `KEY MESSAGES` and `KEY POINTS` both map to the key-message role;
|
|
- `SHORT TERM` and `LONG TERM` retain their roles; and
|
|
- `NEAR TERM`, `DISCUSSION`, and `AVIATION` have no role entry.
|
|
4. Add `ParseForecastDiscussionText` tests for role-level first occurrence:
|
|
|
|
- `KEY POINTS` alone populates key messages;
|
|
- `KEY POINTS` followed by `KEY MESSAGES` keeps the first block; and
|
|
- `KEY MESSAGES` followed by `KEY POINTS` keeps the first block.
|
|
|
|
Use hyphen messages in this stage so list-tokenization changes remain scoped
|
|
to Stage 11.
|
|
5. Add a regression containing `NEAR TERM`, `SHORT TERM`, and `LONG TERM` in one
|
|
bulletin. Assert that near-term prose terminates adjacent blocks but does not
|
|
populate or override either canonical text section.
|
|
|
|
Run and pass:
|
|
|
|
```sh
|
|
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
|
|
go test -count=1 ./internal/providers/nws
|
|
```
|
|
|
|
Do not change the canonical model to expose near-term or discussion content.
|
|
|
|
## Stage 11: Harden Key-Message Metadata and Item Parsing — Completed
|
|
|
|
Replace prefix-sensitive, hyphen-only parsing with conservative metadata and
|
|
list classifiers.
|
|
|
|
1. Add a helper that recognizes a leading labeled metadata line only when:
|
|
|
|
- its trimmed text begins with exactly `Issued at` or `Updated at`, compared
|
|
ASCII case-insensitively;
|
|
- at least one horizontal-whitespace byte follows the label; and
|
|
- parsing the remaining text through the existing unlabeled NWS issue-time
|
|
grammar succeeds.
|
|
|
|
Reuse `parseForecastDiscussionIssueTime` for the timestamp grammar after
|
|
removing the label. Do not add a second timestamp parser. A label prefix with
|
|
no boundary or with an invalid timestamp returns false and remains content.
|
|
2. Update key-message cleanup to remove at most one such valid leading metadata
|
|
line after presentation-marker removal and blank-line trimming. Later valid
|
|
timestamp lines remain message content.
|
|
3. Replace the hyphen-only check with a helper that classifies and strips one of
|
|
these markers from a trimmed line:
|
|
|
|
- one leading `-`, preserving current compatibility whether or not whitespace
|
|
follows it;
|
|
- one leading `*`, whether or not whitespace follows it; or
|
|
- one or more ASCII digits forming an integer greater than zero, followed by
|
|
`)` or `.`, and then either end-of-line or horizontal whitespace.
|
|
|
|
After recognizing and stripping a leading hyphen or asterisk plus horizontal
|
|
whitespace, also strip one immediately following valid numeric marker. This
|
|
supports composite forms such as `- 1.` without retaining either decorator.
|
|
Otherwise strip only the recognized marker and following horizontal
|
|
whitespace. Reject zero, overflow, alphanumeric prefixes, and numeric
|
|
punctuation without the required boundary. Do not interpret these markers
|
|
outside key-message parsing.
|
|
4. Parse a cleaned key-message body deterministically:
|
|
|
|
- first determine whether any nonempty line has a recognized marker;
|
|
- when markers exist, each marker starts a new message; nonempty unmarked
|
|
lines after a marker continue that message; blank lines after the first
|
|
marker are ignored; and blank-line-separated prose before the first marker
|
|
is flushed as preserved message content;
|
|
- when no marker exists, each nonempty paragraph separated by one or more
|
|
blank lines becomes one message; and
|
|
- in both modes, join wrapped lines with one ASCII space, discard empty
|
|
messages, and preserve source order.
|
|
5. Add table-driven marker tests for `-`, `*`, `1)`, `2.`, composite `- 1.` and
|
|
`* 1)`, multi-digit values, wrapped continuations, marker-only lines, and
|
|
rejected zero/malformed numeric prefixes.
|
|
6. Add block-level tests covering:
|
|
|
|
- numbered and asterisk lists producing distinct messages;
|
|
- unmarked paragraphs producing distinct messages;
|
|
- mixed introductory prose and marked items without data loss;
|
|
- valid leading `Issued at` and `Updated at` metadata removal;
|
|
- `Updated atmospheric conditions...` remaining content;
|
|
- `Updated at not a timestamp` remaining content;
|
|
- a later valid timestamp-like line remaining content; and
|
|
- all existing hyphen, change-marker, and continuation behavior.
|
|
|
|
Run and pass:
|
|
|
|
```sh
|
|
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
|
|
go test -count=1 ./internal/providers/nws
|
|
```
|
|
|
|
## Stage 12: Add Representative Numbered and Key-Points Fixtures — Completed
|
|
|
|
Prove the new extension points through provider and normalizer boundaries.
|
|
|
|
1. Retain the existing LSX and BOU fixtures and their regressions.
|
|
2. Add two compact HTML fixtures under `internal/providers/nws/testdata/`:
|
|
|
|
- a BGM/CTP-style fixture with a valid header and issue time, numbered `KEY
|
|
MESSAGES`, wrapped item lines, and a boundary-only `DISCUSSION` section; and
|
|
- an MFR-style fixture with a valid header and issue time, asterisk `KEY
|
|
POINTS`, and an immediately following
|
|
`.DISCUSSION (Today through Thursday)...` section without an intervening
|
|
`&&`.
|
|
|
|
Use concise representative text rather than full web pages. Add a short HTML
|
|
comment to each fixture naming the real office-format family it represents;
|
|
do not claim that edited fixture prose is a verbatim archived product.
|
|
3. Add provider end-to-end tests that assert:
|
|
|
|
- exact ordered key messages and joined continuations;
|
|
- `KEY POINTS` alias mapping;
|
|
- correct office and top-level issue metadata;
|
|
- absence of numeric/asterisk markers from canonical values;
|
|
- absence of `DISCUSSION` headings, qualifiers, and prose from key messages;
|
|
and
|
|
- nil short- and long-term fields when the fixture contains neither mapped
|
|
role.
|
|
4. Add table-driven normalizer regressions using both fixtures. For each, verify
|
|
input envelope preservation, event kind, canonical schema, effective time,
|
|
exact key-message mapping, and successful JSON marshaling.
|
|
5. Assert the JSON payload still has no `nearTerm`, `discussion`, `aviation`, or
|
|
generic `sections` field. Do not add those fields to provider or canonical
|
|
structs.
|
|
6. Keep syntax and tokenizer edge cases in focused unit tables; do not add more
|
|
full fixtures for cases already proven locally.
|
|
|
|
Run and pass:
|
|
|
|
```sh
|
|
gofmt -w internal/providers/nws/forecast_discussion_test.go internal/normalizers/nws/forecast_discussion_test.go
|
|
go test -count=1 ./internal/providers/nws ./internal/normalizers/nws
|
|
```
|
|
|
|
## Stage 13: Reconcile Documentation and Perform Final Validation — Completed
|
|
|
|
Close the second resilience follow-up only after all behavior is implemented and
|
|
verified.
|
|
|
|
1. Review the final diff. Production changes must remain confined to
|
|
`internal/providers/nws/forecast_discussion.go`; other Go changes must be
|
|
owning provider and normalizer tests. Preserve unrelated user work.
|
|
2. Confirm the public contract is unchanged. If implementation appears to
|
|
require a canonical near-term/discussion field, schema change, configuration,
|
|
or downstream migration, stop and report the conflict rather than expanding
|
|
scope.
|
|
3. Run:
|
|
|
|
```sh
|
|
gofmt -w \
|
|
internal/providers/nws/forecast_discussion.go \
|
|
internal/providers/nws/forecast_discussion_test.go \
|
|
internal/normalizers/nws/forecast_discussion_test.go
|
|
go test -count=1 ./internal/providers/nws ./internal/normalizers/nws
|
|
go test -count=1 ./...
|
|
go vet ./...
|
|
git diff --check
|
|
```
|
|
4. Verify every acceptance criterion in the feature roadmap has a corresponding
|
|
passing test, including compatibility with Stages 1-8.
|
|
5. Update the feature roadmap status to `Implemented.` and rewrite its remaining
|
|
proposed or future-tense language as implemented behavior only after all
|
|
checks pass.
|
|
6. Update this implementation plan so Stages 9-13 are marked `— Completed` and
|
|
the Purpose states that all stages are complete. Preserve the stage details
|
|
as a historical implementation record.
|
|
|
|
## Open Questions
|
|
|
|
None. The plan makes the required policy choices explicitly: `KEY POINTS` is a
|
|
true key-message alias; `NEAR TERM` remains semantically distinct and
|
|
boundary-only; parenthesized-terminal syntax is a third constrained grammar;
|
|
metadata is removed only after timestamp validation; and key-message variants
|
|
are handled by an isolated marker classifier plus paragraph fallback.
|