Refresh internal state and adapter documentation
This commit is contained in:
@@ -1,134 +1,63 @@
|
|||||||
# Distributor Adapter Internals
|
# Distributor Adapter Internals
|
||||||
|
|
||||||
This document describes the distributor upload adapter in
|
`internal/adapters/distributor` translates a local delivery request into the
|
||||||
`internal/adapters/distributor`.
|
Distributor Go client's upload and status calls, then returns a local delivery
|
||||||
|
result. The external API, authentication, and idempotency contract is owned by
|
||||||
|
the [Distributor API guide](../integrations/distributor/api.md) and
|
||||||
|
[Distributor bundle guide](../integrations/distributor/pkg-bundle.md).
|
||||||
|
|
||||||
## Purpose
|
## Client construction
|
||||||
|
|
||||||
The adapter submits generated weatherreporter Markdown reports to a configured
|
`Client` holds the endpoint, the name of the environment variable containing
|
||||||
distributor HTTP upload endpoint. It supports one or more file mappings per
|
the token, an optional timeout, and an injectable upstream-client factory.
|
||||||
upload request. It isolates distributor package types, token-env lookup, upload
|
`New` validates its configuration before creating the adapter. For each upload,
|
||||||
client construction, source-bundle file mapping, timeout handling, status
|
the adapter reads the token from the configured environment variable and builds
|
||||||
polling, and upload error wrapping from app orchestration.
|
the upstream client with that endpoint, token, and an HTTP client whose timeout
|
||||||
|
matches the local positive timeout.
|
||||||
|
|
||||||
## Inputs And Outputs
|
The upstream client is an implementation dependency, not a source of
|
||||||
|
application configuration: retry ownership, pipeline selection, path
|
||||||
|
templates, and report rendering are defined by
|
||||||
|
[configuration](../config.md) and [application orchestration](app-orchestration.md).
|
||||||
|
|
||||||
Inputs:
|
## Upload translation
|
||||||
|
|
||||||
- distributor endpoint URL
|
Before calling the dependency, `Upload` validates the endpoint and token
|
||||||
- token environment variable name
|
configuration plus the local pipeline ID, bundle ID, idempotency key, and every
|
||||||
- upload timeout
|
file's source and bundle paths. It maps the request as follows:
|
||||||
- pipeline ID
|
|
||||||
- bundle ID
|
|
||||||
- idempotency key
|
|
||||||
- source Markdown report paths and bundle-relative path mappings
|
|
||||||
- bundle created timestamp
|
|
||||||
- context for cancellation
|
|
||||||
|
|
||||||
Outputs:
|
| Local request | Distributor client value |
|
||||||
|
| --- | --- |
|
||||||
|
| Pipeline ID | Upload pipeline identifier |
|
||||||
|
| Bundle ID | Bundle identifier |
|
||||||
|
| Idempotency key | Upload idempotency key |
|
||||||
|
| File source and bundle paths | Bundle file entries |
|
||||||
|
| Creation timestamp | Bundle creation time |
|
||||||
|
|
||||||
- accepted distributor run ID
|
The call inherits the caller's context and applies the configured positive
|
||||||
- accepted distributor upload status
|
timeout. The adapter does not read report files, construct bundle layouts, or
|
||||||
- distributor run status, status polling error, and raw run report JSON when available
|
persist notification artifacts.
|
||||||
- weatherreporter-owned idempotency conflict error when applicable
|
|
||||||
|
|
||||||
## Boundaries
|
## Status and errors
|
||||||
|
|
||||||
`internal/adapters/distributor` is the only weatherreporter package that imports
|
An accepted upload is followed by one status request. When a timeout is
|
||||||
`gitea.maximumdirect.net/eric/distributor/pkg/upload` or
|
configured, a nonterminal result is polled until `succeeded` or `failed`, or
|
||||||
`gitea.maximumdirect.net/eric/distributor/pkg/bundle`.
|
until the context ends. The translated `UploadResult` contains the run ID,
|
||||||
|
status, and `RunStatus`, including pipeline ID, lifecycle timestamps, report,
|
||||||
|
and remote error details.
|
||||||
|
|
||||||
The app layer passes weatherreporter-owned request values to the adapter. The
|
Status lookup or polling errors are preserved in `UploadResult.StatusError` so
|
||||||
adapter does not choose report types, render templates, select output copies,
|
the caller can record an accepted-but-unconfirmed delivery. A terminal failed
|
||||||
decide whether an upload represents one report or a batch, configure
|
run returns that result and an error. Upload failures return no result. Upstream
|
||||||
destinations, wait for downstream publication, transform Markdown, or persist
|
idempotency conflicts become the local `IdempotencyConflictError`, which adds
|
||||||
notification state.
|
endpoint, pipeline, bundle, idempotency, and file-path context while redacting
|
||||||
|
the token.
|
||||||
|
|
||||||
Full upstream distributor package and HTTP contract details stay under
|
## Verification
|
||||||
`docs/integrations/distributor/`.
|
|
||||||
|
|
||||||
## Config Fields Used
|
Focused tests cover configuration validation, request mapping, timeouts and
|
||||||
|
polling, status translation, conflict handling, and token redaction:
|
||||||
|
|
||||||
The adapter is built from `notify.distributor` config:
|
```sh
|
||||||
|
go test ./internal/adapters/distributor
|
||||||
- `endpoint`
|
```
|
||||||
- `token_env`
|
|
||||||
- `timeout`
|
|
||||||
|
|
||||||
The app layer renders single-report pipeline ID, bundle ID, idempotency key,
|
|
||||||
and bundle paths from:
|
|
||||||
|
|
||||||
- `pipeline_id_template`
|
|
||||||
- `bundle_id_template`
|
|
||||||
- `idempotency_key_template`
|
|
||||||
- report-specific path templates
|
|
||||||
|
|
||||||
For batch uploads, the app layer renders pipeline ID, bundle ID, and
|
|
||||||
idempotency key from `notify.distributor.batch.*`, resolves report-specific
|
|
||||||
path templates once per included report, and passes the resulting multi-file
|
|
||||||
request to this adapter.
|
|
||||||
|
|
||||||
Report-specific path resolution happens entirely in the app layer. Explicit
|
|
||||||
`reports.<report>.distributor.path_templates` overrides take precedence over
|
|
||||||
report definition defaults.
|
|
||||||
|
|
||||||
The token value is read from the environment variable named by `token_env`
|
|
||||||
after config loading and `secrets.directory` processing.
|
|
||||||
|
|
||||||
## Upload Behavior
|
|
||||||
|
|
||||||
The adapter calls distributor `UploadFiles` with one or more file mappings:
|
|
||||||
|
|
||||||
- pipeline ID: the rendered distributor workflow selector
|
|
||||||
- source paths: managed Markdown report paths selected by app orchestration
|
|
||||||
- bundle paths: rendered bundle-relative report paths for each source
|
|
||||||
- created: the report or batch generation timestamp
|
|
||||||
|
|
||||||
The adapter creates a distributor upload client with the configured endpoint,
|
|
||||||
bearer token, and timeout-backed HTTP client. It also wraps the upload context
|
|
||||||
with the configured timeout when the timeout is greater than zero.
|
|
||||||
|
|
||||||
After upload acceptance, the adapter polls distributor `Status` for the accepted
|
|
||||||
run ID until the run reaches `succeeded` or `failed`, or until the configured
|
|
||||||
timeout expires. It returns the latest status, error text, and raw report JSON in
|
|
||||||
weatherreporter-owned types so app orchestration can persist them in the
|
|
||||||
notification debug artifact. Status lookup failures or timeout before a terminal
|
|
||||||
state are kept as debug status errors on an otherwise accepted upload. A
|
|
||||||
terminal distributor run status of `failed` is returned as a notification failure
|
|
||||||
with the status report preserved.
|
|
||||||
|
|
||||||
## Failure Behavior
|
|
||||||
|
|
||||||
The adapter validates required endpoint, token env name, token value, pipeline
|
|
||||||
ID, bundle ID, idempotency key, upload files, source paths, bundle paths, and
|
|
||||||
upload client inputs before uploading.
|
|
||||||
|
|
||||||
Upload failures include endpoint, pipeline ID, bundle ID, idempotency key,
|
|
||||||
source paths, and bundle paths context. Token values are redacted from adapter
|
|
||||||
errors.
|
|
||||||
|
|
||||||
Distributor idempotency conflicts are exposed as a weatherreporter-owned
|
|
||||||
`IdempotencyConflictError`, so callers do not depend on upstream distributor
|
|
||||||
types.
|
|
||||||
|
|
||||||
## Tests
|
|
||||||
|
|
||||||
Inspect:
|
|
||||||
|
|
||||||
- `internal/adapters/distributor/client_test.go`
|
|
||||||
- `internal/app/app_test.go`
|
|
||||||
- `internal/cli/root_test.go`
|
|
||||||
|
|
||||||
Adapter tests use a fake upload client factory and do not require a live
|
|
||||||
distributor service.
|
|
||||||
|
|
||||||
## Invariants
|
|
||||||
|
|
||||||
- Distributor package types do not leak outside the adapter.
|
|
||||||
- Only managed Markdown report paths selected by app orchestration are
|
|
||||||
uploaded.
|
|
||||||
- The adapter never scans the workspace.
|
|
||||||
- Token values are not included in errors, CLI output, metadata, docs, or
|
|
||||||
examples.
|
|
||||||
- Destination routing and Markdown-to-HTML transformation belong to
|
|
||||||
distributor, not weatherreporter.
|
|
||||||
|
|||||||
@@ -1,110 +1,51 @@
|
|||||||
# Scriptorium Adapter Internals
|
# Scriptorium Adapter Internals
|
||||||
|
|
||||||
This document describes the subprocess adapter in
|
`internal/adapters/scriptorium` translates Weather Reporter render requests to
|
||||||
`internal/adapters/scriptorium`.
|
Scriptorium process arguments and translates process results back to local
|
||||||
|
types. The external CLI and output contract belongs to the
|
||||||
|
[Scriptorium integration guide](../integrations/scriptorium.md); prompts,
|
||||||
|
template inputs, and report ownership remain outside this adapter.
|
||||||
|
|
||||||
## Purpose
|
## Request-to-command translation
|
||||||
|
|
||||||
The adapter runs `scriptorium render` for prompt preflight and `scriptorium run`
|
`Runner` accepts a binary, config path, profile, timeout, extra arguments, and
|
||||||
for Markdown report generation or structured generated-text output. It isolates
|
an injectable command executor. Its defaults are the `scriptorium` binary and
|
||||||
subprocess execution, argv construction, timeout handling, output capture, and
|
the real `ExecRunner`. Optional configuration flags are placed before the
|
||||||
exit-code interpretation from app and domain packages.
|
operation-specific arguments, and extra arguments are appended last.
|
||||||
|
|
||||||
## Inputs And Outputs
|
| Local operation | Required values | Translated arguments |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `Render` | prompt ID, data-package path | `render [--config …] [--profile …] --prompt <id> --input data_package=<path> --format json [extra …]` |
|
||||||
|
| `Run` | prompt ID, data-package path, output path | `run [--config …] [--profile …] --prompt <id> --input data_package=<path> --out <path> [extra …]` |
|
||||||
|
| `StructuredRun` | prompt ID, data-package path, output path | Same translation as `Run` |
|
||||||
|
|
||||||
Inputs:
|
Blank required values fail before a command starts. The adapter does not add
|
||||||
|
schema flags or interpret a prompt's payload; it only gives Scriptorium the
|
||||||
|
named `data_package` input.
|
||||||
|
|
||||||
- prompt ID
|
## Command execution and result translation
|
||||||
- YAML prompt input data package path
|
|
||||||
- report output path for `run`
|
|
||||||
- raw generated-text output path for structured `run`
|
|
||||||
- configured binary, config path, profile, timeout, and extra arguments
|
|
||||||
- context for cancellation
|
|
||||||
|
|
||||||
Outputs:
|
`ExecRunner` uses `exec.CommandContext`, never a shell. A positive configured
|
||||||
|
timeout creates a child context. Standard output and standard error are
|
||||||
|
captured independently, each with a 1 MiB limit, and the executed command is
|
||||||
|
retained for diagnostics.
|
||||||
|
|
||||||
- argv used for execution
|
`RenderResult`, `RunResult`, and `StructuredRunResult` expose the command,
|
||||||
- captured stdout and stderr
|
captured output, truncation markers, and exit code. Run results also retain the
|
||||||
- truncation flags for captured output
|
requested output path. Exit status zero is successful. A nonzero process exit
|
||||||
- exit code
|
returns its result and an error, while a start failure, cancellation, or
|
||||||
- report output path for `run`
|
deadline failure returns no result and the execution error.
|
||||||
- raw generated-text output path for structured `run`
|
|
||||||
|
|
||||||
## Boundaries
|
The adapter does not parse rendered JSON, validate a generated report, write
|
||||||
|
state, or upload a report. Those responsibilities sit with
|
||||||
|
[application orchestration](app-orchestration.md), [state internals](state.md), and the
|
||||||
|
relevant delivery adapter.
|
||||||
|
|
||||||
`internal/adapters/scriptorium` owns Scriptorium command construction and
|
## Verification
|
||||||
subprocess execution. It does not choose report types, build prompt input,
|
|
||||||
collect weather data, decide workflow order, or persist workflow metadata.
|
|
||||||
|
|
||||||
The adapter exposes request and result structs for render, Markdown run, and
|
Focused tests cover argument order, validation, bounded capture, timeout and
|
||||||
structured generated-text run operations. State persistence uses state-owned
|
cancellation handling, and exit-status translation:
|
||||||
artifact shapes; app orchestration converts adapter results before saving.
|
|
||||||
|
|
||||||
## Config Fields Used
|
```sh
|
||||||
|
go test ./internal/adapters/scriptorium
|
||||||
- `scriptorium.binary`
|
|
||||||
- `scriptorium.config_path`
|
|
||||||
- `scriptorium.profile`
|
|
||||||
- `scriptorium.timeout`
|
|
||||||
- `scriptorium.extra_args`
|
|
||||||
|
|
||||||
## Commands
|
|
||||||
|
|
||||||
Render preflight argv starts with:
|
|
||||||
|
|
||||||
```text
|
|
||||||
scriptorium render --prompt <prompt_id> --input data_package=<path> --format json
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Report generation argv starts with:
|
|
||||||
|
|
||||||
```text
|
|
||||||
scriptorium run --prompt <prompt_id> --input data_package=<path> --out <path>
|
|
||||||
```
|
|
||||||
|
|
||||||
Structured generated-text argv uses the same `scriptorium run` form, with the
|
|
||||||
`--out` value set to the raw generated-text JSON artifact path. The adapter
|
|
||||||
does not add `--format`, schema path, or JSON Schema flags for structured
|
|
||||||
generation; Scriptorium selects the structured output schema from prompt
|
|
||||||
configuration.
|
|
||||||
|
|
||||||
Configured `--config` and `--profile` flags are inserted after the subcommand
|
|
||||||
and before prompt-specific arguments. Extra arguments are appended after the
|
|
||||||
built-in arguments.
|
|
||||||
|
|
||||||
## Execution Behavior
|
|
||||||
|
|
||||||
The adapter runs commands without shell interpolation. The same private
|
|
||||||
execution path is used by render, Markdown run, and structured run after
|
|
||||||
command-specific request validation and argv construction.
|
|
||||||
|
|
||||||
When `scriptorium.timeout` is greater than zero, each subprocess call uses a
|
|
||||||
context with that timeout. Stdout and stderr are captured separately, capped at
|
|
||||||
1 MiB each, and marked as truncated when the cap is reached.
|
|
||||||
|
|
||||||
## Failure Behavior
|
|
||||||
|
|
||||||
- Missing prompt ID or data package path returns an error before subprocess
|
|
||||||
execution.
|
|
||||||
- Missing run output path returns an error before subprocess execution.
|
|
||||||
- Subprocess start errors, context cancellation, and timeouts are wrapped with
|
|
||||||
operation context by the caller-facing method.
|
|
||||||
- Nonzero render, Markdown run, and structured run exits return the captured
|
|
||||||
result plus an error containing the exit code and stderr.
|
|
||||||
|
|
||||||
## Tests
|
|
||||||
|
|
||||||
Inspect:
|
|
||||||
|
|
||||||
- `internal/adapters/scriptorium/runner_test.go`
|
|
||||||
- `internal/app/app_test.go`
|
|
||||||
- `internal/cli/root_test.go`
|
|
||||||
|
|
||||||
## Invariants
|
|
||||||
|
|
||||||
- No shell interpolation is used.
|
|
||||||
- The Scriptorium input name is `data_package`.
|
|
||||||
- The file at the data package path is YAML produced by `internal/promptinput`.
|
|
||||||
- Render, Markdown run, and structured run preserve command-specific result
|
|
||||||
structs.
|
|
||||||
- Scriptorium-specific flags stay inside adapter and config boundaries.
|
|
||||||
|
|||||||
@@ -1,178 +1,91 @@
|
|||||||
# State Internals
|
# State Internals
|
||||||
|
|
||||||
This document describes filesystem state in `internal/state`.
|
The `internal/state` package owns filesystem-backed run state: safe path
|
||||||
|
derivation, metadata persistence, prior-report lookup, and read-only report
|
||||||
|
inspection. It does not decide which reports to generate or deliver. For the
|
||||||
|
operator-facing layout and retention procedures, see the
|
||||||
|
[operations guide](../operations.md).
|
||||||
|
|
||||||
## Purpose
|
## Store construction and artifact paths
|
||||||
|
|
||||||
`internal/state` owns managed workspace paths, atomic JSON writes, persisted
|
`NewFilesystemStore` requires a workspace root and rejects absolute or
|
||||||
metadata, prior snapshot lookup, and read-only artifact inspection helpers.
|
escaping values for every configured state directory. `Paths` then validates a
|
||||||
|
run ID and artifact group before deriving all paths from the report's valid
|
||||||
|
start date (`YYYY-MM-DD`). This keeps a run's artifacts together while making
|
||||||
|
the paths safe to use below the configured workspace.
|
||||||
|
|
||||||
## Inputs And Outputs
|
| Artifact | Derived location |
|
||||||
|
| --- | --- |
|
||||||
|
| Module snapshot | `snapshots/<group>/<date>/modules.<run-id>.json` |
|
||||||
|
| Metadata | `snapshots/<group>/<date>/metadata.<run-id>.json` |
|
||||||
|
| Data package | `data-packages/<group>/<date>/data_package.<run-id>.yaml` |
|
||||||
|
| Render preflight | `preflight/<group>/<date>/render.<run-id>.json` |
|
||||||
|
| Notification record | `notifications/<group>/<date>/distributor.<run-id>.json` |
|
||||||
|
| Managed report | `reports/<group>/<date>/report.<run-id>.md` |
|
||||||
|
| Generated text | `snapshots/<group>/<date>/generated_text.<run-id>.json` |
|
||||||
|
| Generated-text source and result | `snapshots/<group>/<date>/generated_text_raw.<run-id>.json` and `generated_text_result.<run-id>.json` |
|
||||||
|
| Generated-text render context | `snapshots/<group>/<date>/render_context.<run-id>.json` |
|
||||||
|
|
||||||
Inputs:
|
Notification records use their own configured date directory, because they
|
||||||
|
are not necessarily tied to a report valid period. Report producers create
|
||||||
|
parent directories as needed and write the report body; state is responsible
|
||||||
|
for the surrounding paths and saved run artifacts.
|
||||||
|
|
||||||
- workspace configuration
|
Batch Distributor notifications are derived separately as
|
||||||
- resolved report definition and valid period
|
`notifications/batches/<batch>/<local-date>/distributor.<batch-run-id>.json`.
|
||||||
- module snapshot
|
Their date is calculated from the batch start in its configured location, and
|
||||||
- prompt input data package
|
the batch identity and run ID receive the same path-segment validation as
|
||||||
- preflight artifact
|
single-report artifact identifiers.
|
||||||
- generated-text raw, run-result, validated text, and render-context artifacts
|
|
||||||
- rendered report path preparation request
|
|
||||||
- RunID for inspection lookups
|
|
||||||
|
|
||||||
Outputs:
|
## Metadata and durable writes
|
||||||
|
|
||||||
- module snapshot JSON path
|
`Metadata` is the durable inventory for a run. It records its schema version,
|
||||||
- prompt input data package YAML path
|
run identity, generated and valid timestamps, artifact group and mode, source
|
||||||
- render preflight JSON path
|
content and provenance, and the module snapshot, data-package, preflight,
|
||||||
- generated-text raw JSON path
|
report, generated-artifact, and notification locations when present.
|
||||||
- generated-text run-result JSON path
|
|
||||||
- validated generated-text JSON path
|
|
||||||
- render context JSON path
|
|
||||||
- managed Markdown report path
|
|
||||||
- metadata JSON path
|
|
||||||
- distributor notification debug artifact paths
|
|
||||||
- prior comparable snapshot metadata
|
|
||||||
- loaded module snapshot, data package, generated text, generated-text run
|
|
||||||
result, or render context
|
|
||||||
- recent report records for inspection
|
|
||||||
|
|
||||||
## Boundaries
|
`BuildMetadataFromBriefingMetadata` establishes the common fields; the
|
||||||
|
application adds locations as artifacts are produced. `SaveMetadata` requires
|
||||||
|
the run ID and the module snapshot, data-package, preflight, and metadata
|
||||||
|
paths. The package also saves module snapshots, data packages, preflight
|
||||||
|
records, generated-text artifacts, render contexts, and notifications. JSON
|
||||||
|
writes use `fileutil.WriteJSONAtomic`, so readers do not observe a partially
|
||||||
|
written state file.
|
||||||
|
|
||||||
`internal/state` owns local filesystem layout, path validation, durable writes,
|
The data package itself follows the shared
|
||||||
metadata reads, prior lookup, and report listing. It does not fetch weather
|
[weather-data contract](weather-data.md). Report text, templates, and external
|
||||||
data, derive forecasts, build prompt input content, compare module contents,
|
delivery payloads remain owned by their respective packages and integration
|
||||||
invoke Scriptorium, import adapter result types, or parse CLI flags.
|
references.
|
||||||
|
|
||||||
Preflight persistence uses the state-owned `PreflightArtifact` shape. The app
|
## Prior reports and inspection
|
||||||
converts adapter render results into that shape before saving.
|
|
||||||
|
|
||||||
## Config Fields Used
|
`FindPriorSnapshot` searches metadata rather than guessing from filenames. It
|
||||||
|
only considers an earlier compatible report in the same artifact group and
|
||||||
|
supports the comparison strategies defined by the report request:
|
||||||
|
|
||||||
- `workspace.root`
|
- `same_valid_date` finds an earlier generated report for the same valid day.
|
||||||
- `workspace.snapshots_dir`
|
- `weekend_window` finds a prior comparable weekend window.
|
||||||
- `workspace.reports_dir`
|
|
||||||
- `workspace.data_packages_dir`
|
|
||||||
- `workspace.preflight_dir`
|
|
||||||
- `workspace.notifications_dir`
|
|
||||||
|
|
||||||
Workspace subdirectories must be relative paths that stay under
|
The newest eligible metadata record wins; the current run is excluded.
|
||||||
`workspace.root`.
|
Unreadable or malformed candidate metadata is ignored so a damaged historical
|
||||||
|
record does not block a new run.
|
||||||
|
|
||||||
## Managed Layout
|
`ListReports` walks saved metadata, returns results ordered newest-first by
|
||||||
|
generation time, and treats a missing snapshots directory as an empty history.
|
||||||
|
`LoadMetadataByRunID` builds on that inspection path. These APIs are read-only;
|
||||||
|
repairing or pruning stored state is an operational concern.
|
||||||
|
|
||||||
Paths are derived from the resolved report definition's artifact group, the
|
## Boundaries and verification
|
||||||
valid-period start date, and the RunID. Filenames put the artifact kind before
|
|
||||||
the RunID.
|
|
||||||
|
|
||||||
```text
|
The package rejects unsafe path components and incomplete metadata before
|
||||||
<workspace.root>/
|
writing. Callers must provide a valid report request, artifact group, and
|
||||||
reports/<artifact_group>/<YYYY-MM-DD>/report.<run_id>.md
|
store configuration. Its focused tests cover path derivation, atomic
|
||||||
snapshots/<artifact_group>/<YYYY-MM-DD>/modules.<run_id>.json
|
persistence, metadata validation, comparison eligibility, and report listing:
|
||||||
snapshots/<artifact_group>/<YYYY-MM-DD>/metadata.<run_id>.json
|
|
||||||
snapshots/<artifact_group>/<YYYY-MM-DD>/generated_text_raw.<run_id>.json
|
```sh
|
||||||
snapshots/<artifact_group>/<YYYY-MM-DD>/generated_text_result.<run_id>.json
|
go test ./internal/state
|
||||||
snapshots/<artifact_group>/<YYYY-MM-DD>/generated_text.<run_id>.json
|
|
||||||
snapshots/<artifact_group>/<YYYY-MM-DD>/render_context.<run_id>.json
|
|
||||||
data-packages/<artifact_group>/<YYYY-MM-DD>/data_package.<run_id>.yaml
|
|
||||||
preflight/<artifact_group>/<YYYY-MM-DD>/render.<run_id>.json
|
|
||||||
notifications/<artifact_group>/<YYYY-MM-DD>/distributor.<run_id>.json
|
|
||||||
notifications/batches/<batch>/<YYYY-MM-DD>/distributor.<batch_run_id>.json
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Metadata is stored beside module snapshots and links the module snapshot, data
|
See [application orchestration](app-orchestration.md) for the order in which
|
||||||
package, preflight, report paths, notification path when attempted, and
|
these artifacts are created and [report templates](../templates.md) for the
|
||||||
configured prompt location. For generated-text-template reports, metadata also
|
user-facing report contract.
|
||||||
records the generated text schema ID and links the raw generated text,
|
|
||||||
Scriptorium run result, validated generated text, and render context artifacts.
|
|
||||||
Markdown-report metadata omits those generated-text fields. Report listing
|
|
||||||
walks metadata files under the snapshots directory.
|
|
||||||
|
|
||||||
Batch notification artifacts are stored under the notifications tree rather
|
|
||||||
than report metadata because they describe a batch-level upload. The date
|
|
||||||
directory is the batch start date in the effective report timezone.
|
|
||||||
|
|
||||||
## Prior Lookup
|
|
||||||
|
|
||||||
Prior snapshot lookup reads stored metadata through the shared lookup path and
|
|
||||||
selects the latest earlier snapshot whose report ID is compatible with the
|
|
||||||
current report definition.
|
|
||||||
|
|
||||||
- Daily Report compares with prior Daily Report snapshots for the same valid
|
|
||||||
local date.
|
|
||||||
- Today Report compares with prior Today Report snapshots for the same valid
|
|
||||||
local date.
|
|
||||||
- Tomorrow Report compares with prior Tomorrow Report snapshots for the same
|
|
||||||
valid local date.
|
|
||||||
- 3-Day Outlook compares with prior 3-Day snapshots for the same valid local
|
|
||||||
date.
|
|
||||||
- Weekend Outlook compares with prior Weekend snapshots for the same weekend
|
|
||||||
window.
|
|
||||||
- Hourly Report uses the rolling-window comparison strategy and currently
|
|
||||||
returns no prior snapshot from filesystem lookup.
|
|
||||||
- Storm Report has no prior lookup because explicit event-window comparison is
|
|
||||||
not searched by the filesystem store.
|
|
||||||
|
|
||||||
## Writes And Inspection
|
|
||||||
|
|
||||||
Durable JSON writes use shared atomic file helpers. Generated-text raw and
|
|
||||||
validated JSON artifacts are written atomically as bytes; generated-text run
|
|
||||||
result and render context artifacts are written atomically as JSON. Managed
|
|
||||||
Markdown reports are prepared by creating their parent directory; Scriptorium
|
|
||||||
writes the report body to the prepared path. Extra Markdown copies are handled
|
|
||||||
by app orchestration. Distributor notification debug artifacts are written
|
|
||||||
atomically when notification is attempted and include rendered distributor
|
|
||||||
pipeline ID, bundle ID, idempotency key, bundle paths, upload status, latest
|
|
||||||
run status, and redacted errors.
|
|
||||||
|
|
||||||
Single-report notification artifacts use schema version
|
|
||||||
`weatherreporter.distributor_notification.v1` and record one managed source
|
|
||||||
path plus that source's bundle paths. Batch notification artifacts use schema
|
|
||||||
version `weatherreporter.batch_distributor_notification.v1` and record:
|
|
||||||
|
|
||||||
- `batch`
|
|
||||||
- `batchRunId`
|
|
||||||
- `attemptedAt`
|
|
||||||
- `endpoint`
|
|
||||||
- `pipelineId`
|
|
||||||
- `bundleId`
|
|
||||||
- `idempotencyKey`
|
|
||||||
- `bundleCreated`
|
|
||||||
- `includedReports`, each with `reportId`, `runId`, `sourcePath`, and
|
|
||||||
`bundlePaths`
|
|
||||||
- `status`
|
|
||||||
- `upload`
|
|
||||||
- `runStatus`
|
|
||||||
- `statusError`
|
|
||||||
- `error`
|
|
||||||
|
|
||||||
Inspection helpers read existing metadata, module snapshot, data package,
|
|
||||||
generated text, generated-text run result, and render context files. Missing
|
|
||||||
metadata directories return no inspection records or no prior snapshot rather
|
|
||||||
than creating state.
|
|
||||||
|
|
||||||
## Failure Behavior
|
|
||||||
|
|
||||||
- Invalid workspace paths return validation errors.
|
|
||||||
- Missing required metadata fields prevent metadata writes.
|
|
||||||
- JSON writes use a temporary file followed by rename where practical.
|
|
||||||
- Read and decode failures include path context.
|
|
||||||
- Unknown RunIDs produce an actionable lookup error.
|
|
||||||
|
|
||||||
## Tests
|
|
||||||
|
|
||||||
Inspect:
|
|
||||||
|
|
||||||
- `internal/state/filesystem_test.go`
|
|
||||||
- `internal/app/app_test.go`
|
|
||||||
|
|
||||||
## Invariants
|
|
||||||
|
|
||||||
- Managed paths stay under the configured workspace root.
|
|
||||||
- Artifact grouping comes from report definitions.
|
|
||||||
- Metadata links artifacts produced for a run.
|
|
||||||
- Generated-text artifacts live under the snapshots tree beside module
|
|
||||||
snapshots and metadata.
|
|
||||||
- Batch notification artifacts live under `notifications/batches` and are not
|
|
||||||
linked from report metadata.
|
|
||||||
- Prior lookup is based on structured metadata, not rendered report text.
|
|
||||||
|
|||||||
@@ -1,101 +1,69 @@
|
|||||||
# Weather Data Internals
|
# Weather Data Internals
|
||||||
|
|
||||||
This document describes Weather API ingestion into `weatherdata.Bundle`.
|
`internal/weatherdata` owns the normalized, wire-independent weather bundle
|
||||||
|
that passes from collection through rendering and persistence. The Weather API
|
||||||
|
adapter translates provider responses into these types; its request, response,
|
||||||
|
and availability contract is documented in the
|
||||||
|
[Weather API integration guide](../integrations/weatherapi.md).
|
||||||
|
|
||||||
## Purpose
|
## Bundle contract
|
||||||
|
|
||||||
`internal/adapters/weatherapi` fetches normalized weather data from the
|
`Bundle` has a collection timestamp (`FetchedAt`), source provenance
|
||||||
configured Weather API and assembles the bundle consumed by forecast derivation
|
(`Sources`), and collection-level warnings (`Warnings`). Its product fields are
|
||||||
and module builders. Module builders expose normalized current conditions and
|
optional so an allowed missing source can be represented without manufacturing
|
||||||
weather story context when those sources are available.
|
weather data.
|
||||||
|
|
||||||
## Inputs And Outputs
|
| Field | Normalized product |
|
||||||
|
| --- | --- |
|
||||||
|
| `Observation` | Station observation |
|
||||||
|
| `Current` | Current conditions |
|
||||||
|
| `Hourly` | Hourly forecast periods |
|
||||||
|
| `Narrative` | Narrative forecast |
|
||||||
|
| `Alerts` | Active-alert check, including an explicitly empty result |
|
||||||
|
| `Discussion` | Forecast discussion and its time-range sections |
|
||||||
|
| `Daily` | Daily forecast periods when supplied |
|
||||||
|
| `WeatherStory` | Latest weather story |
|
||||||
|
| `SPCConvectiveOutlooks` | Convective outlook run, discussions, and GeoJSON geometry |
|
||||||
|
|
||||||
Inputs:
|
The bundle carries values rather than provider request details. Consumers use
|
||||||
|
it to construct report facts and data packages; they should not infer a
|
||||||
|
provider endpoint or retry policy from the normalized types. See
|
||||||
|
[collection](collect.md) for assembly and
|
||||||
|
[report templates](../templates.md) for the values exposed to authors.
|
||||||
|
|
||||||
- `config.Config` with Weather API URL, timeout, format, units, timezone,
|
## Source provenance
|
||||||
precision, and missing-source policy
|
|
||||||
- HTTP responses using the Weather API `data` envelope
|
|
||||||
|
|
||||||
Outputs:
|
Every checked source is represented by a `Source` entry. The record identifies
|
||||||
|
the source (`Name`), request location and query (`Endpoint`, `Query`), fetch
|
||||||
|
time, provider issue and update times when available, a SHA-256 digest of the
|
||||||
|
source data, and whether the source was unavailable (`Missing`). Its warnings
|
||||||
|
stay with that source in addition to the bundle-level warning list.
|
||||||
|
|
||||||
- `weatherdata.Bundle` with observation, current conditions, hourly forecast,
|
An empty product can be meaningful checked data. For example, an explicit
|
||||||
narrative forecast, active alerts, discussion, latest weather story, source
|
empty alerts result is not missing and retains its source hash. A source is
|
||||||
records, source warnings, and typed SPC convective outlook data when that
|
marked missing only when the adapter's missing-source policy treats the
|
||||||
optional source is available
|
response or parsing failure as unavailable. The policy itself belongs to the
|
||||||
- optional saved bundle JSON through app fetch helpers
|
[configuration reference](../config.md).
|
||||||
|
|
||||||
## Boundaries
|
## Warning semantics
|
||||||
|
|
||||||
- The adapter owns HTTP calls, response-envelope handling, source hashing, and
|
`SourceWarning` has a source name, stable code, severity, explanatory message,
|
||||||
decoding into internal bundle types.
|
endpoint, and `CompletenessImpact`. When collection proceeds with a warning,
|
||||||
- It does not derive dayparts, resolve report periods, build module values, compare
|
the same warning appears in `Source.Warnings` and `Bundle.Warnings` so both
|
||||||
snapshots, write report state, or invoke Scriptorium.
|
local provenance and whole-run consumers see it. A policy that treats a missing
|
||||||
|
source as an error returns no partial bundle.
|
||||||
|
|
||||||
## Config Fields Used
|
Warnings describe data completeness, not rendering or delivery failures.
|
||||||
|
Those failures are recorded by the application and state layers; see
|
||||||
|
[application orchestration](app-orchestration.md) and [state internals](state.md).
|
||||||
|
|
||||||
- `weather_api.base_url`
|
## Boundaries and verification
|
||||||
- `weather_api.timeout`
|
|
||||||
- `weather_api.format`
|
|
||||||
- `weather_api.units`
|
|
||||||
- `weather_api.timezone`
|
|
||||||
- `weather_api.precision`
|
|
||||||
- `missing_source.default`
|
|
||||||
- `missing_source.sources`
|
|
||||||
|
|
||||||
## External Adapters Used
|
This package defines data shapes and has no HTTP client, configuration loader,
|
||||||
|
filesystem access, or template behavior. Focused tests cover the normalized
|
||||||
|
types and the Weather API adapter verifies translation into them:
|
||||||
|
|
||||||
- Weather API HTTP service
|
```sh
|
||||||
|
go test ./internal/weatherdata
|
||||||
See [Weather API integration](../integrations/weatherapi.md) for the external
|
go test ./internal/adapters/weatherapi
|
||||||
contract used by this project.
|
```
|
||||||
|
|
||||||
## State Or Manifest Behavior
|
|
||||||
|
|
||||||
The adapter records source name, endpoint, query, fetch time, source timestamps
|
|
||||||
when available, SHA-256 hash over compact raw `data` JSON, missing status, and
|
|
||||||
source warnings. Successful `data: null` responses from `/alerts/active`
|
|
||||||
represent a checked empty active-alert list, not a missing source. Successful
|
|
||||||
non-null `/outlooks/convective` responses with empty outlook and discussion
|
|
||||||
arrays represent checked empty outlook data.
|
|
||||||
`app.FetchAndSaveBundle` can write bundle JSON atomically for inspection.
|
|
||||||
|
|
||||||
SPC convective outlook data is stored on
|
|
||||||
`weatherdata.Bundle.SPCConvectiveOutlooks`. The collected run keeps upstream
|
|
||||||
run metadata, location identifiers, ordered outlook records, discussion
|
|
||||||
records, and each outlook's raw GeoJSON geometry. Source provenance for this
|
|
||||||
payload uses the `spc_convective_outlooks` source name, endpoint
|
|
||||||
`/outlooks/convective`, the query sent by the adapter, timestamps, and a hash
|
|
||||||
of the raw `data` object.
|
|
||||||
|
|
||||||
## Skip And Resume Behavior
|
|
||||||
|
|
||||||
No resume behavior. Optional missing or malformed sources may be omitted,
|
|
||||||
warned, or treated as errors according to missing-source policy. Hourly forecast
|
|
||||||
data is required and cannot be skipped.
|
|
||||||
|
|
||||||
## Failure Behavior
|
|
||||||
|
|
||||||
- Missing or invalid `weather_api.base_url` prevents client construction.
|
|
||||||
- HTTP errors, response read failures, and envelope decode failures include
|
|
||||||
endpoint context.
|
|
||||||
- Missing hourly data or hourly forecasts with no periods fail bundle fetch.
|
|
||||||
- Optional sources follow missing-source policy.
|
|
||||||
- Explicit `data: null` from `/alerts/active` produces an empty, non-missing
|
|
||||||
alert run.
|
|
||||||
- Explicit `data: null` from `/outlooks/convective` follows optional
|
|
||||||
missing-source policy.
|
|
||||||
|
|
||||||
## Tests
|
|
||||||
|
|
||||||
Inspect:
|
|
||||||
|
|
||||||
- `internal/adapters/weatherapi/client_test.go`
|
|
||||||
- `internal/app/app_test.go`
|
|
||||||
|
|
||||||
## Invariants
|
|
||||||
|
|
||||||
- Weather facts come from normalized source data.
|
|
||||||
- Full hourly and narrative products are fetched; Go owns report-period
|
|
||||||
selection.
|
|
||||||
- Source provenance and warnings remain inspectable downstream.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user