Refresh internal state and adapter documentation
This commit is contained in:
@@ -1,178 +1,91 @@
|
||||
# State Internals
|
||||
|
||||
This document describes filesystem state in `internal/state`.
|
||||
The `internal/state` package owns filesystem-backed run state: safe path
|
||||
derivation, metadata persistence, prior-report lookup, and read-only report
|
||||
inspection. It does not decide which reports to generate or deliver. For the
|
||||
operator-facing layout and retention procedures, see the
|
||||
[operations guide](../operations.md).
|
||||
|
||||
## Purpose
|
||||
## Store construction and artifact paths
|
||||
|
||||
`internal/state` owns managed workspace paths, atomic JSON writes, persisted
|
||||
metadata, prior snapshot lookup, and read-only artifact inspection helpers.
|
||||
`NewFilesystemStore` requires a workspace root and rejects absolute or
|
||||
escaping values for every configured state directory. `Paths` then validates a
|
||||
run ID and artifact group before deriving all paths from the report's valid
|
||||
start date (`YYYY-MM-DD`). This keeps a run's artifacts together while making
|
||||
the paths safe to use below the configured workspace.
|
||||
|
||||
## Inputs And Outputs
|
||||
| Artifact | Derived location |
|
||||
| --- | --- |
|
||||
| Module snapshot | `snapshots/<group>/<date>/modules.<run-id>.json` |
|
||||
| Metadata | `snapshots/<group>/<date>/metadata.<run-id>.json` |
|
||||
| Data package | `data-packages/<group>/<date>/data_package.<run-id>.yaml` |
|
||||
| Render preflight | `preflight/<group>/<date>/render.<run-id>.json` |
|
||||
| Notification record | `notifications/<group>/<date>/distributor.<run-id>.json` |
|
||||
| Managed report | `reports/<group>/<date>/report.<run-id>.md` |
|
||||
| Generated text | `snapshots/<group>/<date>/generated_text.<run-id>.json` |
|
||||
| Generated-text source and result | `snapshots/<group>/<date>/generated_text_raw.<run-id>.json` and `generated_text_result.<run-id>.json` |
|
||||
| Generated-text render context | `snapshots/<group>/<date>/render_context.<run-id>.json` |
|
||||
|
||||
Inputs:
|
||||
Notification records use their own configured date directory, because they
|
||||
are not necessarily tied to a report valid period. Report producers create
|
||||
parent directories as needed and write the report body; state is responsible
|
||||
for the surrounding paths and saved run artifacts.
|
||||
|
||||
- workspace configuration
|
||||
- resolved report definition and valid period
|
||||
- module snapshot
|
||||
- prompt input data package
|
||||
- preflight artifact
|
||||
- generated-text raw, run-result, validated text, and render-context artifacts
|
||||
- rendered report path preparation request
|
||||
- RunID for inspection lookups
|
||||
Batch Distributor notifications are derived separately as
|
||||
`notifications/batches/<batch>/<local-date>/distributor.<batch-run-id>.json`.
|
||||
Their date is calculated from the batch start in its configured location, and
|
||||
the batch identity and run ID receive the same path-segment validation as
|
||||
single-report artifact identifiers.
|
||||
|
||||
Outputs:
|
||||
## Metadata and durable writes
|
||||
|
||||
- module snapshot JSON path
|
||||
- prompt input data package YAML path
|
||||
- render preflight JSON path
|
||||
- generated-text raw JSON path
|
||||
- generated-text run-result JSON path
|
||||
- validated generated-text JSON path
|
||||
- render context JSON path
|
||||
- managed Markdown report path
|
||||
- metadata JSON path
|
||||
- distributor notification debug artifact paths
|
||||
- prior comparable snapshot metadata
|
||||
- loaded module snapshot, data package, generated text, generated-text run
|
||||
result, or render context
|
||||
- recent report records for inspection
|
||||
`Metadata` is the durable inventory for a run. It records its schema version,
|
||||
run identity, generated and valid timestamps, artifact group and mode, source
|
||||
content and provenance, and the module snapshot, data-package, preflight,
|
||||
report, generated-artifact, and notification locations when present.
|
||||
|
||||
## Boundaries
|
||||
`BuildMetadataFromBriefingMetadata` establishes the common fields; the
|
||||
application adds locations as artifacts are produced. `SaveMetadata` requires
|
||||
the run ID and the module snapshot, data-package, preflight, and metadata
|
||||
paths. The package also saves module snapshots, data packages, preflight
|
||||
records, generated-text artifacts, render contexts, and notifications. JSON
|
||||
writes use `fileutil.WriteJSONAtomic`, so readers do not observe a partially
|
||||
written state file.
|
||||
|
||||
`internal/state` owns local filesystem layout, path validation, durable writes,
|
||||
metadata reads, prior lookup, and report listing. It does not fetch weather
|
||||
data, derive forecasts, build prompt input content, compare module contents,
|
||||
invoke Scriptorium, import adapter result types, or parse CLI flags.
|
||||
The data package itself follows the shared
|
||||
[weather-data contract](weather-data.md). Report text, templates, and external
|
||||
delivery payloads remain owned by their respective packages and integration
|
||||
references.
|
||||
|
||||
Preflight persistence uses the state-owned `PreflightArtifact` shape. The app
|
||||
converts adapter render results into that shape before saving.
|
||||
## Prior reports and inspection
|
||||
|
||||
## Config Fields Used
|
||||
`FindPriorSnapshot` searches metadata rather than guessing from filenames. It
|
||||
only considers an earlier compatible report in the same artifact group and
|
||||
supports the comparison strategies defined by the report request:
|
||||
|
||||
- `workspace.root`
|
||||
- `workspace.snapshots_dir`
|
||||
- `workspace.reports_dir`
|
||||
- `workspace.data_packages_dir`
|
||||
- `workspace.preflight_dir`
|
||||
- `workspace.notifications_dir`
|
||||
- `same_valid_date` finds an earlier generated report for the same valid day.
|
||||
- `weekend_window` finds a prior comparable weekend window.
|
||||
|
||||
Workspace subdirectories must be relative paths that stay under
|
||||
`workspace.root`.
|
||||
The newest eligible metadata record wins; the current run is excluded.
|
||||
Unreadable or malformed candidate metadata is ignored so a damaged historical
|
||||
record does not block a new run.
|
||||
|
||||
## Managed Layout
|
||||
`ListReports` walks saved metadata, returns results ordered newest-first by
|
||||
generation time, and treats a missing snapshots directory as an empty history.
|
||||
`LoadMetadataByRunID` builds on that inspection path. These APIs are read-only;
|
||||
repairing or pruning stored state is an operational concern.
|
||||
|
||||
Paths are derived from the resolved report definition's artifact group, the
|
||||
valid-period start date, and the RunID. Filenames put the artifact kind before
|
||||
the RunID.
|
||||
## Boundaries and verification
|
||||
|
||||
```text
|
||||
<workspace.root>/
|
||||
reports/<artifact_group>/<YYYY-MM-DD>/report.<run_id>.md
|
||||
snapshots/<artifact_group>/<YYYY-MM-DD>/modules.<run_id>.json
|
||||
snapshots/<artifact_group>/<YYYY-MM-DD>/metadata.<run_id>.json
|
||||
snapshots/<artifact_group>/<YYYY-MM-DD>/generated_text_raw.<run_id>.json
|
||||
snapshots/<artifact_group>/<YYYY-MM-DD>/generated_text_result.<run_id>.json
|
||||
snapshots/<artifact_group>/<YYYY-MM-DD>/generated_text.<run_id>.json
|
||||
snapshots/<artifact_group>/<YYYY-MM-DD>/render_context.<run_id>.json
|
||||
data-packages/<artifact_group>/<YYYY-MM-DD>/data_package.<run_id>.yaml
|
||||
preflight/<artifact_group>/<YYYY-MM-DD>/render.<run_id>.json
|
||||
notifications/<artifact_group>/<YYYY-MM-DD>/distributor.<run_id>.json
|
||||
notifications/batches/<batch>/<YYYY-MM-DD>/distributor.<batch_run_id>.json
|
||||
The package rejects unsafe path components and incomplete metadata before
|
||||
writing. Callers must provide a valid report request, artifact group, and
|
||||
store configuration. Its focused tests cover path derivation, atomic
|
||||
persistence, metadata validation, comparison eligibility, and report listing:
|
||||
|
||||
```sh
|
||||
go test ./internal/state
|
||||
```
|
||||
|
||||
Metadata is stored beside module snapshots and links the module snapshot, data
|
||||
package, preflight, report paths, notification path when attempted, and
|
||||
configured prompt location. For generated-text-template reports, metadata also
|
||||
records the generated text schema ID and links the raw generated text,
|
||||
Scriptorium run result, validated generated text, and render context artifacts.
|
||||
Markdown-report metadata omits those generated-text fields. Report listing
|
||||
walks metadata files under the snapshots directory.
|
||||
|
||||
Batch notification artifacts are stored under the notifications tree rather
|
||||
than report metadata because they describe a batch-level upload. The date
|
||||
directory is the batch start date in the effective report timezone.
|
||||
|
||||
## Prior Lookup
|
||||
|
||||
Prior snapshot lookup reads stored metadata through the shared lookup path and
|
||||
selects the latest earlier snapshot whose report ID is compatible with the
|
||||
current report definition.
|
||||
|
||||
- Daily Report compares with prior Daily Report snapshots for the same valid
|
||||
local date.
|
||||
- Today Report compares with prior Today Report snapshots for the same valid
|
||||
local date.
|
||||
- Tomorrow Report compares with prior Tomorrow Report snapshots for the same
|
||||
valid local date.
|
||||
- 3-Day Outlook compares with prior 3-Day snapshots for the same valid local
|
||||
date.
|
||||
- Weekend Outlook compares with prior Weekend snapshots for the same weekend
|
||||
window.
|
||||
- Hourly Report uses the rolling-window comparison strategy and currently
|
||||
returns no prior snapshot from filesystem lookup.
|
||||
- Storm Report has no prior lookup because explicit event-window comparison is
|
||||
not searched by the filesystem store.
|
||||
|
||||
## Writes And Inspection
|
||||
|
||||
Durable JSON writes use shared atomic file helpers. Generated-text raw and
|
||||
validated JSON artifacts are written atomically as bytes; generated-text run
|
||||
result and render context artifacts are written atomically as JSON. Managed
|
||||
Markdown reports are prepared by creating their parent directory; Scriptorium
|
||||
writes the report body to the prepared path. Extra Markdown copies are handled
|
||||
by app orchestration. Distributor notification debug artifacts are written
|
||||
atomically when notification is attempted and include rendered distributor
|
||||
pipeline ID, bundle ID, idempotency key, bundle paths, upload status, latest
|
||||
run status, and redacted errors.
|
||||
|
||||
Single-report notification artifacts use schema version
|
||||
`weatherreporter.distributor_notification.v1` and record one managed source
|
||||
path plus that source's bundle paths. Batch notification artifacts use schema
|
||||
version `weatherreporter.batch_distributor_notification.v1` and record:
|
||||
|
||||
- `batch`
|
||||
- `batchRunId`
|
||||
- `attemptedAt`
|
||||
- `endpoint`
|
||||
- `pipelineId`
|
||||
- `bundleId`
|
||||
- `idempotencyKey`
|
||||
- `bundleCreated`
|
||||
- `includedReports`, each with `reportId`, `runId`, `sourcePath`, and
|
||||
`bundlePaths`
|
||||
- `status`
|
||||
- `upload`
|
||||
- `runStatus`
|
||||
- `statusError`
|
||||
- `error`
|
||||
|
||||
Inspection helpers read existing metadata, module snapshot, data package,
|
||||
generated text, generated-text run result, and render context files. Missing
|
||||
metadata directories return no inspection records or no prior snapshot rather
|
||||
than creating state.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
- Invalid workspace paths return validation errors.
|
||||
- Missing required metadata fields prevent metadata writes.
|
||||
- JSON writes use a temporary file followed by rename where practical.
|
||||
- Read and decode failures include path context.
|
||||
- Unknown RunIDs produce an actionable lookup error.
|
||||
|
||||
## Tests
|
||||
|
||||
Inspect:
|
||||
|
||||
- `internal/state/filesystem_test.go`
|
||||
- `internal/app/app_test.go`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Managed paths stay under the configured workspace root.
|
||||
- Artifact grouping comes from report definitions.
|
||||
- Metadata links artifacts produced for a run.
|
||||
- Generated-text artifacts live under the snapshots tree beside module
|
||||
snapshots and metadata.
|
||||
- Batch notification artifacts live under `notifications/batches` and are not
|
||||
linked from report metadata.
|
||||
- Prior lookup is based on structured metadata, not rendered report text.
|
||||
See [application orchestration](app-orchestration.md) for the order in which
|
||||
these artifacts are created and [report templates](../templates.md) for the
|
||||
user-facing report contract.
|
||||
|
||||
Reference in New Issue
Block a user