weatherreporter/docs/roadmap/ephemeral-state.md

# Ephemeral Operational State Roadmap

Status: Accepted feature direction; implementation has not started.

## Purpose

Weatherreporter should treat generated weather reports and their intermediate
artifacts as short-lived operational material rather than a durable audit
history. Forecasts and current conditions change continuously, and the normal
response to an old or failed report is to generate a new report, not to
reconstruct the provenance of the old one.

The application should retain only the bounded state needed to publish the
current report, calculate Recent Changes against the last successfully
published report for the same valid period, and complete the current
invocation safely. Detailed LLM diagnostics should remain an explicit,
operator-controlled exception outside ordinary workspace state.

This roadmap defines the intended state lifecycle, compatibility policy, and
architectural boundaries. A separate implementation plan will define the
ordered work after the roadmap is complete.

## User Intent

The state model should reflect these product expectations:

- weather reports are ephemeral products, not business records;
- old report provenance has no continuing operational value once conditions
  and forecasts have changed;
- regenerating is preferable to recovering, replaying, or inspecting an old
  generation;
- routine operation should not accumulate unbounded run-addressed artifacts;
- Recent Changes remains useful, but needs only one prior successful snapshot
  for the same report and valid period; and
- sensitive prompt and response capture remains opt-in and explicitly managed
  by the operator.

## Current State

Each generation currently writes a run-addressed collection containing a
module snapshot, data package, prompt preparation receipt, prompt execution
receipt, raw generated text, validated generated text, render context, managed
report, metadata, and optional notification receipt. Successful and failed
runs accumulate beneath the workspace.

Metadata links the collection and supports lookup by RunID. The CLI can list
historical runs and inspect their metadata, modules, data packages, prior
snapshots, and source provenance. New metadata uses the V2 format while the
reader retains V1 compatibility. Prompt artifacts are validated against
current report and prompt definitions when saved and loaded.

Most of this persistence exists for retrospective inspection and failure
recovery. Dedicated prompt preparation and execution load operations have no
ordinary production consumer. The important exception is module snapshot
state: generation actively loads the most recent compatible snapshot to build
the deterministic Recent Changes input for Daily, Today, and Tomorrow.

## Desired End State

Weatherreporter has three distinct state classes:

| State class | Lifecycle | Purpose |
| --- | --- | --- |
| Invocation workspace | Temporary and unpublished | Hold intermediate values while one report or batch is running. |
| Current published state | Bounded and replaceable | Hold the current managed report and the minimal deterministic snapshot or manifest needed for normal operation. |
| Secure LLM debug capture | Explicitly enabled and operator-managed | Diagnose prompt rendering or provider output when the operator deliberately requests sensitive capture. |

Ordinary generation uses an invocation-scoped temporary directory on the same
filesystem as the managed workspace when atomic publication requires it.
Prompt preparation, prompt execution, raw generated text, validated generated
text, render contexts, data packages, and notification receipts may exist
there while needed, but they are not published as durable historical
artifacts.

A successful report atomically replaces the current published state for its
logical report key and valid period. A failed attempt leaves the last
successfully published report and comparison snapshot unchanged. Ordinary
temporary artifacts are removed after both success and handled failure;
cleanup failure is reported safely but must not replace the primary generation
error.

RunIDs remain useful as in-process correlation identifiers in action results,
logs, provider provenance, and optional debug paths. They no longer identify a
durable collection that Weatherreporter promises to locate or decode later.

## Published Report Policy

The managed Markdown report remains the authoritative upload source during an
invocation. The intended default is to retain only the current managed report
for each logical report key and valid period, replacing it atomically after a
new report has been fully rendered and validated.

An explicit `--out` or `--out-dir` copy remains operator-owned output outside
the managed-state lifecycle. Weatherreporter does not delete, rotate, or
rewrite those copies except when the same explicit destination is selected by
a later invocation.

Distributor continues to receive only a completed managed Markdown report.
Notification success or failure does not create a durable notification
history. A notification failure leaves the newly published report available
and returns a safe error through the current action result.

## Recent Changes State

Recent Changes must be preserved without preserving general report history.
For Daily, Today, and Tomorrow, Weatherreporter retains at most one compatible
module snapshot for each logical report key and valid period.

The retained snapshot represents the last successfully published report. A
new invocation reads it before constructing Recent Changes and replaces it
only when the new managed report has been successfully validated, rendered,
and published. A failed generation therefore does not become the baseline for
the next report and cannot hide changes that the user has not yet seen.

Hourly does not currently use the comparison strategy and should not retain a
comparison snapshot solely for symmetry. State whose valid period has ended
and can no longer participate in a supported comparison is eligible for safe
cleanup.

## Temporary Workspace And Failure Semantics

Temporary state must remain beneath a narrowly owned application directory and
use safe path construction, restrictive permissions where content is
sensitive, and atomic writes where practical. Publication must not expose a
partially rendered report or a snapshot that does not correspond to the
published report.

Normal results retain bounded error information and paths only for artifacts
that remain meaningful after the command: a previously or newly published
report, an explicit operator output, or an enabled secure debug capture.
Temporary intermediate paths are not emitted as if they were durable recovery
locations. A failed command is retried by starting a new generation.

Process interruption may leave an uncommitted temporary directory. Such a
directory is never considered published state, is never selected for Recent
Changes, and may be removed by a documented safe cleanup mechanism. Cleanup
must distinguish inactive temporary directories from concurrent active
invocations and must never recursively target the workspace root or an
unresolved configuration path.

## Inspection And Metadata Policy

Run-history discovery and inspection are not part of the desired product
contract. The historical `inspect reports`, `inspect metadata`, `inspect
modules`, `inspect data-package`, `inspect prior`, and `inspect sources`
surfaces are candidates for removal together rather than preservation through
a new storage representation.

Any manifest retained for atomic publication or Recent Changes is current
operational state, not an archival metadata record. It should contain only the
identity, valid period, safe paths, and deterministic snapshot information
needed to validate and use that current state. It does not need to preserve
prompt messages, generated prose intermediates, source provenance, provider
provenance, notification history, or a catalog of prior runs.

The application does not promise cross-version decoding of ordinary workspace
state. A new release may replace or ignore incompatible current-state files,
provided it fails safely, never mistakes stale state for a compatible Recent
Changes baseline, and documents any operator action required during upgrade.

## Prompt Execution And Debugging

Prompt inspection before weather collection and prepared execution remain
runtime safety requirements. They do not require durable preparation or
execution receipts.

The selected logical profile, effective backend and model, validation outcome,
and safe classified error remain available to the active workflow and its CLI
summary where useful. Weatherreporter does not retain them as long-term report
provenance after the invocation completes.

The existing explicit secure debug root remains outside ordinary state and may
retain rendered prompts, schemas, input bodies, generated bodies, and effective
parameters according to its documented contract. Weatherreporter does not
automatically clean that operator-selected location. Credentials must remain
excluded from debug capture.

## Compatibility And Upgrade Policy

This is an intentional breaking change to the workspace and inspection
contracts. Weatherreporter does not need to migrate historical V1 or V2
metadata, prompt receipts, intermediate generated-text artifacts, or managed
reports into the new representation.

Legacy workspace trees must not be silently interpreted as current published
state. They also must not be deleted automatically merely because a new
version starts: an operator may have placed or referenced files there despite
the absence of a continuing application compatibility promise. Release notes
and operations documentation must explain whether legacy data can be removed
manually and identify the exact safe target.

The change should land in a release whose notes clearly identify removed CLI
commands, obsolete paths and schemas, the new bounded state behavior, and any
upgrade action. Because Weatherreporter remains pre-1.0, the ordinary semantic
version policy may carry this breaking change without inventing a migration
framework.

## Required Architecture Decision Record

The implemented feature must include an Accepted ADR recording the durable
architectural decision to use ephemeral operational state. The ADR is not part
of this roadmap-writing pass and should not be created until implementation is
being prepared.

The ADR should record:

- the mismatch between run-addressed provenance storage and the ephemeral
  weather-report lifecycle;
- the decision to retain bounded current report and comparison state rather
  than historical runs;
- the distinction between temporary invocation state, published operational
  state, explicit output copies, and secure debug capture;
- the removal of historical inspection and backward-compatibility guarantees;
- atomic publication and failed-run behavior;
- the alternatives considered, including retaining the current archive,
  adding time-based retention, or keeping a bounded run history; and
- consequences for CLI compatibility, workspace layout, testing, operations,
  and future schema changes.

Once accepted, the ADR owns the decision rationale. The architecture policy
owns the resulting current invariant, while focused state, CLI, operations,
and integration documents own the implemented contracts.

## Scope

The completed feature includes:

- an invocation-scoped temporary workspace for intermediate generation state;
- atomic publication of the current managed report and its minimal operational
  state;
- a bounded comparison snapshot representing the last successfully published
  report for each supported report key and valid period;
- safe cleanup behavior for normal completion, handled failure, and abandoned
  temporary workspaces;
- removal of durable prompt preparation, prompt execution, generated-text,
  render-context, data-package, notification, and run-metadata history;
- removal of run-history inspection commands and their application/state
  contracts;
- removal of V1 metadata compatibility and current-version coupling for
  historical prompt artifacts by removing the historical artifact contract;
- preservation of active-command partial status, safe errors, and paths to
  genuinely retained published, operator-owned, or debug outputs;
- preservation of explicit output copies, Distributor upload behavior, and
  opt-in secure LLM debug capture;
- risk-appropriate offline tests for atomic publication, comparison baselines,
  failure isolation, cleanup safety, concurrent invocation safety, and absence
  of unbounded state growth;
- an Accepted ADR documenting the architectural decision; and
- updates to canonical architecture, CLI, operations, configuration,
  troubleshooting, integration, internal, testing, and release documentation
  where their contracts change.

## Non-Goals

This feature does not include:

- a general-purpose cache, database, archival service, or retention engine;
- replaying or resuming interrupted generation;
- migrating legacy artifacts into the new representation;
- retaining a bounded number of historical runs for convenience;
- automatic upload or archival of state to remote storage;
- collecting additional provider telemetry or weather-source provenance;
- changing report content, prompt text, schemas, profile selection, weather
  derivation, or batch membership;
- deleting operator-owned `--out`, `--out-dir`, or secure debug files;
- changing Distributor's report-content contract; or
- making live external services part of the default test suite.

## Safety And Testing Policy

The state refactoring must preserve Weatherreporter's existing path-safety and
atomicity expectations while reducing the amount of durable state. Tests
should emphasize observable lifecycle guarantees rather than private file
choreography.

Important risks requiring durable offline coverage include:

- a failed or canceled generation replacing a previously published report or
  comparison baseline;
- a partially written report becoming visible as current;
- Recent Changes selecting an incompatible report, valid period, or failed
  attempt;
- cleanup deleting published, operator-owned, debug, or concurrently active
  files;
- batch partial success corrupting the state of another report;
- notification failure rolling back or obscuring a successfully published
  report;
- stale or incompatible current state being treated as valid; and
- repeated successful and failed runs causing unbounded ordinary workspace
  growth.

Tests remain deterministic, offline, credential-free, and based on real
temporary filesystems plus narrow external-boundary fakes. Race-enabled tests
are required where publication, cleanup, or concurrent invocation behavior
shares mutable filesystem state.

## Relationship To Domain-Specific Profiles

The domain-specific profile feature can be implemented before this refactor,
but it should not add new historical compatibility or durable-provenance
commitments. Profile inspection, selection, override precedence, and effective
model resolution remain active-workflow behavior and survive the state change.

The domain-profile roadmap and implementation plan should acknowledge that
prompt artifacts from version `1.0.1` need not remain readable after prompts
advance to `1.1.0`. Existing state persistence may remain temporarily while
the profile feature lands, but it should not be expanded or treated as the
target architecture.

## Completion Criteria

The roadmap's target state is achieved when:

- ordinary runs no longer create durable run-addressed artifact collections;
- a successful report atomically replaces only the corresponding current
  published state;
- failed and canceled attempts leave the prior published report and Recent
  Changes baseline unchanged;
- Daily, Today, and Tomorrow compare against at most one compatible snapshot
  from the last successfully published report;
- expired comparison and published state can be removed safely without
  touching operator-owned or active files;
- Hourly does not retain an unused comparison snapshot;
- historical inspection commands and V1/V2 archival compatibility code are
  removed;
- temporary, published, explicit-output, and debug paths have distinct and
  documented ownership and cleanup rules;
- Distributor and active-command summaries continue to receive the completed
  report and safe status information they require;
- the default suite proves atomicity, bounded growth, cleanup safety, batch
  isolation, and comparison correctness offline;
- an Accepted ADR records the architectural decision and alternatives; and
- canonical current-state documentation describes only the implemented
  lifecycle.

## Open Questions

### Lifetime of the current managed report

Recommendation: retain one current managed report per logical report key and
valid period until it is replaced or its valid period expires. This preserves
the current default behavior for invocations without `--out` while bounding
growth.

Alternative: treat the managed report as temporary and retain output only when
the operator supplies `--out` or `--out-dir`. This minimizes state further but
makes a successful default invocation produce no durable report for the user
and complicates Distributor sequencing.

### Historical inspection replacement

Recommendation: remove the run-history inspection commands without adding a
replacement initially. Current command summaries, current managed files, and
opt-in debug capture cover the remaining supported workflows.

Alternative: add a narrow `inspect current REPORT` command backed only by the
current operational manifest. This provides discoverability without history,
but it creates a new public surface and may preserve metadata complexity that
the refactor is intended to remove.

### Abandoned temporary workspace cleanup

Recommendation: use an explicitly owned temporary subtree with per-invocation
ownership markers and a conservative age threshold. Normal cleanup removes the
current invocation synchronously; opportunistic cleanup removes only marked,
inactive directories old enough that they cannot reasonably belong to a live
invocation.

Alternative: perform only synchronous cleanup and document manual removal of
directories left by process termination. This minimizes destructive code and
concurrency risk, but crashed processes can still accumulate unbounded files.

### Legacy workspace cleanup

Recommendation: ignore legacy run-addressed trees and document a precise,
manual one-time cleanup procedure. Do not automatically delete them during
startup or upgrade.

Alternative: add an explicit cleanup command that previews and then removes
recognized legacy artifacts. This is more convenient for large installations
but introduces a destructive command and a legacy-format classifier that must
be maintained and tested.