weatherreporter/docs/roadmap/ephemeral-state.md

Ephemeral Operational State Roadmap

Status: Accepted feature direction; implementation has not started.

Purpose

Weatherreporter should treat generated weather reports and their intermediate artifacts as short-lived operational material rather than a durable audit history. Forecasts and current conditions change continuously, and the normal response to an old or failed report is to generate a new report, not to reconstruct the provenance of the old one.

The application should retain only the bounded state needed to publish the current report, calculate Recent Changes against the last successfully published report for the same valid period, and complete the current invocation safely. Detailed LLM diagnostics should remain an explicit, operator-controlled exception outside ordinary workspace state.

This roadmap defines the intended state lifecycle, compatibility policy, and architectural boundaries. A separate implementation plan will define the ordered work after the roadmap is complete.

User Intent

The state model should reflect these product expectations:

• weather reports are ephemeral products, not business records;
• old report provenance has no continuing operational value once conditions and forecasts have changed;
• regenerating is preferable to recovering, replaying, or inspecting an old generation;
• routine operation should not accumulate unbounded run-addressed artifacts;
• Recent Changes remains useful, but needs only one prior successful snapshot for the same report and valid period; and
• sensitive prompt and response capture remains opt-in and explicitly managed by the operator.

Current State

Each generation currently writes a run-addressed collection containing a module snapshot, data package, prompt preparation receipt, prompt execution receipt, raw generated text, validated generated text, render context, managed report, metadata, and optional notification receipt. Successful and failed runs accumulate beneath the workspace.

Metadata links the collection and supports lookup by RunID. The CLI can list historical runs and inspect their metadata, modules, data packages, prior snapshots, and source provenance. New metadata uses the V2 format while the reader retains V1 compatibility. Prompt artifacts are validated against current report and prompt definitions when saved and loaded.

Most of this persistence exists for retrospective inspection and failure recovery. Dedicated prompt preparation and execution load operations have no ordinary production consumer. The important exception is module snapshot state: generation actively loads the most recent compatible snapshot to build the deterministic Recent Changes input for Daily, Today, and Tomorrow.

Desired End State

Weatherreporter has three distinct state classes:

State class	Lifecycle	Purpose
Invocation workspace	Temporary and unpublished	Hold intermediate values while one report or batch is running.
Current published state	Bounded and replaceable	Hold the current managed report and the minimal deterministic snapshot or manifest needed for normal operation.
Secure LLM debug capture	Explicitly enabled and operator-managed	Diagnose prompt rendering or provider output when the operator deliberately requests sensitive capture.

Ordinary generation uses an invocation-scoped temporary directory on the same filesystem as the managed workspace when atomic publication requires it. Prompt preparation, prompt execution, raw generated text, validated generated text, render contexts, data packages, and notification receipts may exist there while needed, but they are not published as durable historical artifacts.

A successful report atomically replaces the current published state for its logical report key and valid period. A failed attempt leaves the last successfully published report and comparison snapshot unchanged. Ordinary temporary artifacts are removed after both success and handled failure; cleanup failure is reported safely but must not replace the primary generation error.

RunIDs remain useful as in-process correlation identifiers in action results, logs, provider provenance, and optional debug paths. They no longer identify a durable collection that Weatherreporter promises to locate or decode later.

Published Report Policy

The managed Markdown report remains the authoritative upload source during an invocation. The intended default is to retain only the current managed report for each logical report key and valid period, replacing it atomically after a new report has been fully rendered and validated.

An explicit --out or --out-dir copy remains operator-owned output outside the managed-state lifecycle. Weatherreporter does not delete, rotate, or rewrite those copies except when the same explicit destination is selected by a later invocation.

Distributor continues to receive only a completed managed Markdown report. Notification success or failure does not create a durable notification history. A notification failure leaves the newly published report available and returns a safe error through the current action result.

Recent Changes State

Recent Changes must be preserved without preserving general report history. For Daily, Today, and Tomorrow, Weatherreporter retains at most one compatible module snapshot for each logical report key and valid period.

The retained snapshot represents the last successfully published report. A new invocation reads it before constructing Recent Changes and replaces it only when the new managed report has been successfully validated, rendered, and published. A failed generation therefore does not become the baseline for the next report and cannot hide changes that the user has not yet seen.

Hourly does not currently use the comparison strategy and should not retain a comparison snapshot solely for symmetry. State whose valid period has ended and can no longer participate in a supported comparison is eligible for safe cleanup.

Temporary Workspace And Failure Semantics

Temporary state must remain beneath a narrowly owned application directory and use safe path construction, restrictive permissions where content is sensitive, and atomic writes where practical. Publication must not expose a partially rendered report or a snapshot that does not correspond to the published report.

Normal results retain bounded error information and paths only for artifacts that remain meaningful after the command: a previously or newly published report, an explicit operator output, or an enabled secure debug capture. Temporary intermediate paths are not emitted as if they were durable recovery locations. A failed command is retried by starting a new generation.

Process interruption may leave an uncommitted temporary directory. Such a directory is never considered published state, is never selected for Recent Changes, and may be removed by a documented safe cleanup mechanism. Cleanup must distinguish inactive temporary directories from concurrent active invocations and must never recursively target the workspace root or an unresolved configuration path.

Inspection And Metadata Policy

Run-history discovery and inspection are not part of the desired product contract. The historical inspect reports, inspect metadata, inspect modules, inspect data-package, inspect prior, and inspect sources surfaces are candidates for removal together rather than preservation through a new storage representation.

Any manifest retained for atomic publication or Recent Changes is current operational state, not an archival metadata record. It should contain only the identity, valid period, safe paths, and deterministic snapshot information needed to validate and use that current state. It does not need to preserve prompt messages, generated prose intermediates, source provenance, provider provenance, notification history, or a catalog of prior runs.

The application does not promise cross-version decoding of ordinary workspace state. A new release may replace or ignore incompatible current-state files, provided it fails safely, never mistakes stale state for a compatible Recent Changes baseline, and documents any operator action required during upgrade.

Prompt Execution And Debugging

Prompt inspection before weather collection and prepared execution remain runtime safety requirements. They do not require durable preparation or execution receipts.

The selected logical profile, effective backend and model, validation outcome, and safe classified error remain available to the active workflow and its CLI summary where useful. Weatherreporter does not retain them as long-term report provenance after the invocation completes.

The existing explicit secure debug root remains outside ordinary state and may retain rendered prompts, schemas, input bodies, generated bodies, and effective parameters according to its documented contract. Weatherreporter does not automatically clean that operator-selected location. Credentials must remain excluded from debug capture.

Compatibility And Upgrade Policy

This is an intentional breaking change to the workspace and inspection contracts. Weatherreporter does not need to migrate historical V1 or V2 metadata, prompt receipts, intermediate generated-text artifacts, or managed reports into the new representation.

Legacy workspace trees must not be silently interpreted as current published state. They also must not be deleted automatically merely because a new version starts: an operator may have placed or referenced files there despite the absence of a continuing application compatibility promise. Release notes and operations documentation must explain whether legacy data can be removed manually and identify the exact safe target.

The change should land in a release whose notes clearly identify removed CLI commands, obsolete paths and schemas, the new bounded state behavior, and any upgrade action. Because Weatherreporter remains pre-1.0, the ordinary semantic version policy may carry this breaking change without inventing a migration framework.

Required Architecture Decision Record

The implemented feature must include an Accepted ADR recording the durable architectural decision to use ephemeral operational state. The ADR is not part of this roadmap-writing pass and should not be created until implementation is being prepared.

The ADR should record:

• the mismatch between run-addressed provenance storage and the ephemeral weather-report lifecycle;
• the decision to retain bounded current report and comparison state rather than historical runs;
• the distinction between temporary invocation state, published operational state, explicit output copies, and secure debug capture;
• the removal of historical inspection and backward-compatibility guarantees;
• atomic publication and failed-run behavior;
• the alternatives considered, including retaining the current archive, adding time-based retention, or keeping a bounded run history; and
• consequences for CLI compatibility, workspace layout, testing, operations, and future schema changes.

Once accepted, the ADR owns the decision rationale. The architecture policy owns the resulting current invariant, while focused state, CLI, operations, and integration documents own the implemented contracts.

Scope

The completed feature includes:

• an invocation-scoped temporary workspace for intermediate generation state;
• atomic publication of the current managed report and its minimal operational state;
• a bounded comparison snapshot representing the last successfully published report for each supported report key and valid period;
• safe cleanup behavior for normal completion, handled failure, and abandoned temporary workspaces;
• removal of durable prompt preparation, prompt execution, generated-text, render-context, data-package, notification, and run-metadata history;
• removal of run-history inspection commands and their application/state contracts;
• removal of V1 metadata compatibility and current-version coupling for historical prompt artifacts by removing the historical artifact contract;
• preservation of active-command partial status, safe errors, and paths to genuinely retained published, operator-owned, or debug outputs;
• preservation of explicit output copies, Distributor upload behavior, and opt-in secure LLM debug capture;
• risk-appropriate offline tests for atomic publication, comparison baselines, failure isolation, cleanup safety, concurrent invocation safety, and absence of unbounded state growth;
• an Accepted ADR documenting the architectural decision; and
• updates to canonical architecture, CLI, operations, configuration, troubleshooting, integration, internal, testing, and release documentation where their contracts change.

Non-Goals

This feature does not include:

• a general-purpose cache, database, archival service, or retention engine;
• replaying or resuming interrupted generation;
• migrating legacy artifacts into the new representation;
• retaining a bounded number of historical runs for convenience;
• automatic upload or archival of state to remote storage;
• collecting additional provider telemetry or weather-source provenance;
• changing report content, prompt text, schemas, profile selection, weather derivation, or batch membership;
• deleting operator-owned --out, --out-dir, or secure debug files;
• changing Distributor's report-content contract; or
• making live external services part of the default test suite.

Safety And Testing Policy

The state refactoring must preserve Weatherreporter's existing path-safety and atomicity expectations while reducing the amount of durable state. Tests should emphasize observable lifecycle guarantees rather than private file choreography.

Important risks requiring durable offline coverage include:

• a failed or canceled generation replacing a previously published report or comparison baseline;
• a partially written report becoming visible as current;
• Recent Changes selecting an incompatible report, valid period, or failed attempt;
• cleanup deleting published, operator-owned, debug, or concurrently active files;
• batch partial success corrupting the state of another report;
• notification failure rolling back or obscuring a successfully published report;
• stale or incompatible current state being treated as valid; and
• repeated successful and failed runs causing unbounded ordinary workspace growth.

Tests remain deterministic, offline, credential-free, and based on real temporary filesystems plus narrow external-boundary fakes. Race-enabled tests are required where publication, cleanup, or concurrent invocation behavior shares mutable filesystem state.

Relationship To Domain-Specific Profiles

The domain-specific profile feature can be implemented before this refactor, but it should not add new historical compatibility or durable-provenance commitments. Profile inspection, selection, override precedence, and effective model resolution remain active-workflow behavior and survive the state change.

The domain-profile roadmap and implementation plan should acknowledge that prompt artifacts from version 1.0.1 need not remain readable after prompts advance to 1.1.0. Existing state persistence may remain temporarily while the profile feature lands, but it should not be expanded or treated as the target architecture.

Completion Criteria

The roadmap's target state is achieved when:

• ordinary runs no longer create durable run-addressed artifact collections;
• a successful report atomically replaces only the corresponding current published state;
• failed and canceled attempts leave the prior published report and Recent Changes baseline unchanged;
• Daily, Today, and Tomorrow compare against at most one compatible snapshot from the last successfully published report;
• expired comparison and published state can be removed safely without touching operator-owned or active files;
• Hourly does not retain an unused comparison snapshot;
• historical inspection commands and V1/V2 archival compatibility code are removed;
• temporary, published, explicit-output, and debug paths have distinct and documented ownership and cleanup rules;
• Distributor and active-command summaries continue to receive the completed report and safe status information they require;
• the default suite proves atomicity, bounded growth, cleanup safety, batch isolation, and comparison correctness offline;
• an Accepted ADR records the architectural decision and alternatives; and
• canonical current-state documentation describes only the implemented lifecycle.

Open Questions

Lifetime of the current managed report

Recommendation: retain one current managed report per logical report key and valid period until it is replaced or its valid period expires. This preserves the current default behavior for invocations without --out while bounding growth.

Alternative: treat the managed report as temporary and retain output only when the operator supplies --out or --out-dir. This minimizes state further but makes a successful default invocation produce no durable report for the user and complicates Distributor sequencing.

Historical inspection replacement

Recommendation: remove the run-history inspection commands without adding a replacement initially. Current command summaries, current managed files, and opt-in debug capture cover the remaining supported workflows.

Alternative: add a narrow inspect current REPORT command backed only by the current operational manifest. This provides discoverability without history, but it creates a new public surface and may preserve metadata complexity that the refactor is intended to remove.

Abandoned temporary workspace cleanup

Recommendation: use an explicitly owned temporary subtree with per-invocation ownership markers and a conservative age threshold. Normal cleanup removes the current invocation synchronously; opportunistic cleanup removes only marked, inactive directories old enough that they cannot reasonably belong to a live invocation.

Alternative: perform only synchronous cleanup and document manual removal of directories left by process termination. This minimizes destructive code and concurrency risk, but crashed processes can still accumulate unbounded files.

Legacy workspace cleanup

Recommendation: ignore legacy run-addressed trees and document a precise, manual one-time cleanup procedure. Do not automatically delete them during startup or upgrade.

Alternative: add an explicit cleanup command that previews and then removes recognized legacy artifacts. This is more convenient for large installations but introduces a destructive command and a legacy-format classifier that must be maintained and tested.