Add a feature roadmap and staged implentation plan to refactor the local workspace layout
This commit is contained in:
194
docs/roadmap/implementation.md
Normal file
194
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,194 @@
|
|||||||
|
# Workspace Layout Implementation Plan
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document is the staged implementation plan for
|
||||||
|
[workspace.md](workspace.md). It is written for an LLM coding agent that will
|
||||||
|
implement the workspace artifact layout refactor in order.
|
||||||
|
|
||||||
|
The feature is complete when every managed local run artifact uses the
|
||||||
|
kind-first, date-scoped layout defined in `docs/roadmap/workspace.md`, metadata
|
||||||
|
and inspection link those paths correctly, and distributor upload behavior is
|
||||||
|
unchanged.
|
||||||
|
|
||||||
|
## Ground Rules
|
||||||
|
|
||||||
|
- Review `docs/policy/architecture.md`, `docs/policy/development.md`, and
|
||||||
|
`docs/policy/documentation.md` before editing code.
|
||||||
|
- Keep path construction owned by `internal/state`.
|
||||||
|
- Do not add user-visible workspace path template configuration.
|
||||||
|
- Do not use distributor bundle paths for managed workspace paths.
|
||||||
|
- Do not implement cleanup, migration tooling, or a local publication mirror in
|
||||||
|
this work.
|
||||||
|
- Keep optional `--out` and `--out-dir` copies separate from managed workspace
|
||||||
|
artifacts.
|
||||||
|
|
||||||
|
## Target Path Contract
|
||||||
|
|
||||||
|
State-managed run artifacts must use these relative path shapes under
|
||||||
|
`workspace.root`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
reports/<artifact_group>/<valid_start_date>/report.<run_id>.md
|
||||||
|
|
||||||
|
snapshots/<artifact_group>/<valid_start_date>/modules.<run_id>.json
|
||||||
|
snapshots/<artifact_group>/<valid_start_date>/metadata.<run_id>.json
|
||||||
|
snapshots/<artifact_group>/<valid_start_date>/generated_text_raw.<run_id>.json
|
||||||
|
snapshots/<artifact_group>/<valid_start_date>/generated_text_result.<run_id>.json
|
||||||
|
snapshots/<artifact_group>/<valid_start_date>/generated_text.<run_id>.json
|
||||||
|
snapshots/<artifact_group>/<valid_start_date>/render_context.<run_id>.json
|
||||||
|
|
||||||
|
data-packages/<artifact_group>/<valid_start_date>/data_package.<run_id>.yaml
|
||||||
|
preflight/<artifact_group>/<valid_start_date>/render.<run_id>.json
|
||||||
|
|
||||||
|
notifications/<artifact_group>/<valid_start_date>/distributor.<run_id>.json
|
||||||
|
notifications/batches/<batch>/<batch_start_date>/distributor.<batch_run_id>.json
|
||||||
|
```
|
||||||
|
|
||||||
|
`valid_start_date` and `batch_start_date` are local dates in the effective
|
||||||
|
report timezone already carried by the resolved report or batch reference.
|
||||||
|
|
||||||
|
## Stage 1: State Path Construction
|
||||||
|
|
||||||
|
Goal: switch the canonical managed path builder to the target layout.
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Update `internal/state.FilesystemStore.Paths` to build all `ArtifactPaths`
|
||||||
|
using the target path contract above.
|
||||||
|
- Keep existing validation for workspace subdirectories, RunID, and report
|
||||||
|
artifact group.
|
||||||
|
- Keep `PrepareRenderedReport` responsible for creating the parent report
|
||||||
|
directory before Scriptorium or template rendering writes Markdown.
|
||||||
|
- Update `BatchDistributorNotificationPath` to use
|
||||||
|
`distributor.<batch_run_id>.json`.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Update state path tests to assert the exact new relative paths for at least
|
||||||
|
`daily`, `today`, `tomorrow`, and `hourly`.
|
||||||
|
- Update batch notification path tests for
|
||||||
|
`notifications/batches/<batch>/<date>/distributor.<batch_run_id>.json`.
|
||||||
|
- Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/state
|
||||||
|
```
|
||||||
|
|
||||||
|
Completion criteria:
|
||||||
|
|
||||||
|
- `FilesystemStore.Paths` returns only target-layout managed paths.
|
||||||
|
- Existing save methods write to the new paths through the existing
|
||||||
|
`ArtifactPaths` fields.
|
||||||
|
|
||||||
|
## Stage 2: Metadata Discovery And Prior Lookup
|
||||||
|
|
||||||
|
Goal: make inspection and prior lookup discover the new metadata filenames.
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Replace metadata filename checks that currently rely on the old
|
||||||
|
`.metadata.json` suffix with a helper that matches only
|
||||||
|
`metadata.<run_id>.json`.
|
||||||
|
- Use that helper in `ListReports` and `FindPriorSnapshot`.
|
||||||
|
- Do not add legacy metadata filename fallback in this refactor; migration is
|
||||||
|
explicitly deferred by the feature roadmap.
|
||||||
|
- Keep `LoadMetadataByRunID` behavior based on `ListReports`.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Update report listing tests to create/read `metadata.<run_id>.json`.
|
||||||
|
- Update prior lookup tests to confirm comparable metadata is found under the
|
||||||
|
new filename.
|
||||||
|
- Add or adjust one negative test so non-metadata JSON in the snapshot tree is
|
||||||
|
ignored.
|
||||||
|
- Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/state
|
||||||
|
```
|
||||||
|
|
||||||
|
Completion criteria:
|
||||||
|
|
||||||
|
- Inspection and prior lookup work with new metadata names.
|
||||||
|
- Generated-text, module, and render-context JSON files are not mistaken for
|
||||||
|
metadata.
|
||||||
|
|
||||||
|
## Stage 3: App Workflow Expectations
|
||||||
|
|
||||||
|
Goal: update app-level tests and any path assumptions outside `internal/state`.
|
||||||
|
|
||||||
|
Implementation:
|
||||||
|
|
||||||
|
- Do not duplicate path-building logic in `internal/app`; continue asking the
|
||||||
|
state store for managed paths.
|
||||||
|
- Update app tests that assert managed report paths, data package paths,
|
||||||
|
generated-text paths, render context paths, notification source paths, batch
|
||||||
|
included report source paths, and result JSON paths.
|
||||||
|
- Preserve the invariant that distributor uploads use the managed Markdown
|
||||||
|
report path as the upload source and map it to separate distributor bundle
|
||||||
|
paths.
|
||||||
|
- Preserve `--out` and `--out-dir` behavior as extra copies outside the managed
|
||||||
|
report path.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/app ./internal/cli
|
||||||
|
```
|
||||||
|
|
||||||
|
Completion criteria:
|
||||||
|
|
||||||
|
- App workflows still generate, save metadata, notify distributor, and report
|
||||||
|
paths using state-owned managed paths.
|
||||||
|
- No app code constructs the new managed path layout manually.
|
||||||
|
|
||||||
|
## Stage 4: Documentation
|
||||||
|
|
||||||
|
Goal: move the implemented layout into maintained docs after code behavior has
|
||||||
|
changed.
|
||||||
|
|
||||||
|
Documentation changes:
|
||||||
|
|
||||||
|
- Update `docs/internal/state.md` with the new target managed layout and
|
||||||
|
filename convention.
|
||||||
|
- Update `docs/operations.md` filesystem layout examples and artifact wording.
|
||||||
|
- Update `docs/config.md` only if workspace field descriptions need
|
||||||
|
clarification.
|
||||||
|
- Keep cleanup, migration, and local publication mirror details in roadmap
|
||||||
|
docs until those features are implemented.
|
||||||
|
|
||||||
|
Validation:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./internal/state ./internal/app ./internal/config
|
||||||
|
go test ./...
|
||||||
|
go run ./cmd/weatherreporter --help
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
Completion criteria:
|
||||||
|
|
||||||
|
- Implemented docs match the new behavior.
|
||||||
|
- Non-roadmap docs do not describe deferred cleanup or local publication mirror
|
||||||
|
behavior as implemented.
|
||||||
|
|
||||||
|
## Final Verification
|
||||||
|
|
||||||
|
Before considering the feature complete, run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./...
|
||||||
|
go run ./cmd/weatherreporter --help
|
||||||
|
git diff --check
|
||||||
|
rg "\\.metadata\\.json|\\.modules\\.json|\\.data_package\\.yaml|\\.render\\.json|\\.distributor\\.json|generated_text\\.raw|generated_text\\.run" internal docs examples --glob '!docs/roadmap/**'
|
||||||
|
```
|
||||||
|
|
||||||
|
The final `rg` should find no old managed filename patterns in implemented
|
||||||
|
code, maintained docs, or examples.
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
None. The feature roadmap decisions are sufficient for implementation.
|
||||||
119
docs/roadmap/workspace.md
Normal file
119
docs/roadmap/workspace.md
Normal file
@@ -0,0 +1,119 @@
|
|||||||
|
# Workspace Artifact Layout Roadmap
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This roadmap defines the target local workspace layout for artifacts generated
|
||||||
|
by weatherreporter.
|
||||||
|
|
||||||
|
Weatherreporter currently writes managed local artifacts for each run and also
|
||||||
|
maps the managed Markdown report into distributor bundle paths. Distributor
|
||||||
|
bundle paths are publication paths and may include overwrite-oriented aliases
|
||||||
|
such as `hourly/index.md`, `today/index.md`, and
|
||||||
|
`daily/<valid_start_date>/index.md`. Local workspace paths have a different
|
||||||
|
job: they preserve run-specific source artifacts for inspection, troubleshooting,
|
||||||
|
retry diagnosis, and future cleanup.
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
- Keep managed local workspace paths separate from distributor bundle paths.
|
||||||
|
- Do not use distributor-style `index.md` aliases as the canonical managed
|
||||||
|
local artifact paths.
|
||||||
|
- Store managed local artifacts as immutable run artifacts.
|
||||||
|
- Group every durable run artifact by artifact family, report artifact group,
|
||||||
|
and local `valid_start_date`.
|
||||||
|
- Use `valid_start_date` in the effective report timezone as the directory key.
|
||||||
|
- Include the RunID in every durable run artifact filename.
|
||||||
|
- Prefer artifact-kind-first filenames: `<kind>.<run_id>.<extension>`.
|
||||||
|
- Keep optional `--out` and `--out-dir` copies separate from managed workspace
|
||||||
|
artifacts.
|
||||||
|
- Future cleanup should delete old managed artifact date directories by
|
||||||
|
`valid_start_date`; cleanup should be narrow and opt-in.
|
||||||
|
- If a local publication preview is added later, put it in a separate mirror
|
||||||
|
tree and treat it as rebuildable output, not canonical run state.
|
||||||
|
|
||||||
|
## Target Managed Layout
|
||||||
|
|
||||||
|
The target managed workspace layout is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
<workspace.root>/
|
||||||
|
reports/<artifact_group>/<valid_start_date>/report.<run_id>.md
|
||||||
|
|
||||||
|
snapshots/<artifact_group>/<valid_start_date>/modules.<run_id>.json
|
||||||
|
snapshots/<artifact_group>/<valid_start_date>/metadata.<run_id>.json
|
||||||
|
snapshots/<artifact_group>/<valid_start_date>/generated_text_raw.<run_id>.json
|
||||||
|
snapshots/<artifact_group>/<valid_start_date>/generated_text_result.<run_id>.json
|
||||||
|
snapshots/<artifact_group>/<valid_start_date>/generated_text.<run_id>.json
|
||||||
|
snapshots/<artifact_group>/<valid_start_date>/render_context.<run_id>.json
|
||||||
|
|
||||||
|
data-packages/<artifact_group>/<valid_start_date>/data_package.<run_id>.yaml
|
||||||
|
preflight/<artifact_group>/<valid_start_date>/render.<run_id>.json
|
||||||
|
|
||||||
|
notifications/<artifact_group>/<valid_start_date>/distributor.<run_id>.json
|
||||||
|
notifications/batches/<batch>/<batch_start_date>/distributor.<batch_run_id>.json
|
||||||
|
```
|
||||||
|
|
||||||
|
`artifact_group` comes from the report definition. `valid_start_date` is the
|
||||||
|
resolved report valid period start date in the effective report timezone.
|
||||||
|
`batch_start_date` is the batch start date in the effective report timezone.
|
||||||
|
|
||||||
|
Generated-text-template reports use the generated-text artifacts. Markdown-path
|
||||||
|
reports omit generated-text artifacts they do not produce.
|
||||||
|
|
||||||
|
## Cleanup Compatibility
|
||||||
|
|
||||||
|
The target layout is designed to support a future cleanup command that removes
|
||||||
|
managed local artifacts where `valid_start_date` is more than two days before
|
||||||
|
the current local date.
|
||||||
|
|
||||||
|
Cleanup can operate by deleting matching date directories across managed
|
||||||
|
artifact families:
|
||||||
|
|
||||||
|
```text
|
||||||
|
reports/<artifact_group>/<old_date>/
|
||||||
|
snapshots/<artifact_group>/<old_date>/
|
||||||
|
data-packages/<artifact_group>/<old_date>/
|
||||||
|
preflight/<artifact_group>/<old_date>/
|
||||||
|
notifications/<artifact_group>/<old_date>/
|
||||||
|
```
|
||||||
|
|
||||||
|
The command should not infer dates from RunIDs when the directory date is
|
||||||
|
available. It should use the effective configured report timezone for "current
|
||||||
|
date" and for interpreting `valid_start_date` directories.
|
||||||
|
|
||||||
|
Batch notification cleanup should use `batch_start_date` under
|
||||||
|
`notifications/batches/<batch>/`. A future local publication mirror, if added,
|
||||||
|
should be ignored by cleanup or rebuilt after cleanup; it should not be treated
|
||||||
|
as run history.
|
||||||
|
|
||||||
|
## Intended Final State
|
||||||
|
|
||||||
|
`internal/state` owns the managed workspace layout and path construction. The
|
||||||
|
app layer continues to ask the state store for managed paths and records those
|
||||||
|
paths in metadata and command results.
|
||||||
|
|
||||||
|
Metadata links every artifact produced for a run using the new managed paths.
|
||||||
|
Inspection and prior-snapshot lookup continue to read metadata rather than
|
||||||
|
rendered Markdown text. Distributor uploads continue to use the managed
|
||||||
|
Markdown report as the source file and map it to separately resolved
|
||||||
|
distributor bundle paths.
|
||||||
|
|
||||||
|
Existing workspace configuration fields remain meaningful:
|
||||||
|
|
||||||
|
- `workspace.root`
|
||||||
|
- `workspace.snapshots_dir`
|
||||||
|
- `workspace.reports_dir`
|
||||||
|
- `workspace.data_packages_dir`
|
||||||
|
- `workspace.preflight_dir`
|
||||||
|
- `workspace.notifications_dir`
|
||||||
|
|
||||||
|
This roadmap does not require a new user-visible workspace path template config
|
||||||
|
surface.
|
||||||
|
|
||||||
|
## Deferred Work
|
||||||
|
|
||||||
|
- A cleanup command for managed local artifacts.
|
||||||
|
- Cleanup dry-run output, retention configuration, and deletion confirmation
|
||||||
|
behavior.
|
||||||
|
- A rebuildable local publication mirror that uses distributor-style paths.
|
||||||
|
- Migration tooling for existing pre-refactor workspace directories.
|
||||||
Reference in New Issue
Block a user