weatherreporter/docs/roadmap/implementation.md

Promptkit Migration Implementation Plan

Status: Completed; Stages 1–19 passed their exit gates.

Purpose And Authority

This document records the completed implementation of the Promptkit migration roadmap and its post-implementation audit remediation. The feature roadmap records scope, user intent, policy choices, and the implemented end state. This plan records implementation sequence, tests, and completion gates.

Stages 12–19 were completed in order. They fixed additional defects exposed by their required tests only when those defects were within the same stated contract; they did not add new product behavior or reinterpret roadmap decisions.

This plan follows the repository's architecture, documentation, and testing policies.

Continuing Invariants

• Keep gitea.maximumdirect.net/eric/promptkit pinned at exactly v0.4.0.
• Keep Promptkit types inside internal/adapters/promptkit, its tests, and the external prompt-asset contract test.
• Preserve one Promptkit engine per generate or run invocation and one shared engine for every sequential report in a batch.
• Preserve exact prompt version 1.0.0, the exact persisted YAML data-package bytes, prepared execution, and preparation persistence before provider work.
• Do not add retries, repair attempts, concurrent batch generation, direct Markdown generation, arbitrary backend registration, or live-provider tests.
• Keep ordinary artifacts, errors, logs, and summaries free of credentials, rendered messages, schemas, input bodies, generated bodies, endpoints, and full effective parameter maps.
• Keep sensitive debug artifacts opt-in, outside normal state, owner-only, atomic, and free of credentials.
• Treat an artifact path as reached only after the corresponding write or copy succeeds. Never persist or summarize a merely derivable future path.
• Preserve every safe reached path in partial app and CLI results even when a later persistence, validation, rendering, copy, or notification step fails.
• Keep v1 metadata read compatibility and write only v2 metadata for new runs.
• Keep the default test suite deterministic, offline, and credential-free.
• Run git diff --check before completing every stage. Run the full repository gate in Stage 19.

Completed Migration Summary

Stages 1–11 are implemented and committed. They remain summarized here to preserve the history and dependencies of the follow-up work.

Stage	Completed outcome
1	Removed the unfinished three-day, weekend, and storm product surfaces and retained Daily, Today, Tomorrow, and Hourly with exact prompt version 1.0.0.
2	Promoted the four operational prompts and canonical schemas into the embedded internal/promptassets source used by Promptkit and generated-text validation.
3	Added the project-owned internal/promptexec inspection, preparation, execution, validation, debug, and error contract.
4	Added the Promptkit v0.4.0 adapter with prepared execution, explicit value mapping, safe error classification, and offline model-client tests.
5	Added Promptkit-era preparation and execution artifacts, metadata v2, new paths, and v1 decoding support.
6	Added explicitly rooted, permission-restricted, atomic LLM debug persistence.
7	Added Promptkit configuration, executor composition, and pre-collection prompt/profile/credential inspection.
8	Cut single-report generation over to prepared Promptkit execution and v2 persistence.
9	Added --llm-debug-dir and Promptkit-era single-report summary fields.
10	Cut morning and evening batches over to one shared Promptkit executor and removed Scriptorium code, configuration, and dependency metadata.
11	Updated canonical Promptkit documentation, removed the temporary Scriptorium corpus, and ran the available repository checks.

The post-implementation audit confirmed the principal dependency and package boundaries, but found incorrect reached-path bookkeeping, incomplete execution artifact updates, insufficient artifact validation, extensive loss of behavioral tests during the final cutover, and roadmap lifecycle text that was not finalized. The completed remediation addressed those findings without changing the intended feature scope.

Stage	Completed outcome
12	Corrected reached-path bookkeeping across metadata, app results, batch items, and CLI summaries.
13	Hardened Promptkit-era durable state validation and restored v1/v2 state coverage.
14	Recorded every downstream path reached after completed prompt execution.
15	Restored assembled single-report behavioral and failure coverage.
16	Simplified prompt-generation orchestration while preserving behavior.
17	Restored assembled batch, planning, artifact, and notification coverage.
18	Restored supported CLI, summary, safety, and historical inspection coverage.
19	Reconciled canonical documentation and passed the complete repository verification gate.

Stage 12: Correct Reached-Artifact Bookkeeping

Status: Completed.

Goal

Make metadata, app results, batch items, and CLI summaries truthful at every failure boundary: a nonblank path means that artifact was successfully created.

Work

1. Change state.BuildPromptMetadataFromBriefingMetadata so it initializes identity, schema, metadata destination, and only artifacts already saved at the call site. It must not prepopulate raw-output, normalized-text, render-context, managed-report, preparation, execution, notification, or output-copy paths.

2. In generatePromptReport, assign each metadata and ReportResult path immediately after that artifact write succeeds and before attempting the next write. In particular:

   • do not initialize ReportResult.ReportPath from Store.Paths;
   • record a saved failed-preparation receipt in the result before saving metadata;
   • record a saved failed or completed execution receipt before saving metadata;
   • retain raw, normalized, context, report, copy, and notification paths when a later step fails; and
   • keep MetadataPath unchanged when a metadata rewrite fails, because the prior successfully written metadata record remains the reached version.

3. Remove batch-item prepopulation from derived Store.Paths values. BatchReportResult receives paths only from the returned ReportResult or from a write that the batch itself successfully completed.

4. Preserve current CLI field names and omission behavior. Human and JSON summaries must omit every unreached path.

5. Do not change artifact locations, filenames, schemas, report output, or notification policy in this stage.

Tests

• Add focused app tests for one representative report using real temporary state plus a narrow failure-injecting store wrapper.

• Fail the next persistence step immediately after a successful preparation receipt, execution receipt, raw output, normalized output, render context, managed report, output copy, and notification artifact; assert that the returned result contains every reached path and no future path.

• Include one preparation failure, one operational execution failure, and one completed validation rejection to cover the three execution outcome shapes.

• Add batch and CLI summary assertions proving unreached paths are omitted.

• Run:

  go test ./internal/state ./internal/app ./internal/cli
  git diff --check
  

Exit Gate

Every nonblank path in newly written metadata, app results, batch items, and CLI summaries names an artifact that exists. Every safe artifact successfully written before a later failure remains discoverable from the returned partial result.

Stage 13: Harden Durable State Contracts And Restore State Coverage

Status: Completed.

Goal

Make the v1/v2 wire boundary and Promptkit-era artifact validation explicit, strict, and durably tested.

Work

1. Strengthen PromptPreparationArtifact.Validate:

   • require report ID, Weatherreporter RunID, prompt ID, exact prompt version, data-package path, nonzero start/end times, nonnegative duration, and an end not earlier than the start;
   • for success, require preparation provenance, prohibit an error, and require its prompt ID/version and data-package path to match the top-level artifact;
   • for failure, require a classified bounded error and prohibit fabricated preparation provenance.

2. Strengthen PromptExecutionArtifact.Validate:

   • require report ID, Weatherreporter RunID, prompt ID, exact prompt version, nonzero start/end times, nonnegative duration, and an end not earlier than the start;
   • for success and validation rejection, require provenance and completed validation, prohibit an operational error, and require the provenance prompt ID/version to match the artifact;
   • do not compare the provenance RunID with the Weatherreporter RunID because the provenance value is Promptkit's run identity;
   • for operational failure, require a classified bounded error and prohibit invented provenance or completed validation.

3. Validate required provenance fields for completed executions, including Promptkit RunID, prompt and rendered hashes, selected profile/backend/model, and data-package path. Permit usage counters and generated hash to be zero when the provider legitimately reports no value.

4. Restore focused filesystem and metadata tests for:

   • exact v2 paths and filenames;
   • preparation/execution round trips and required fields;
   • metadata v2 round trips without legacy aliases;
   • v1 decoding, normalized internal aliases, and v1-preserving re-marshaling;
   • unknown schema rejection;
   • report listing, RunID lookup, source/module/data-package inspection, and retained v1 behavior for historical report IDs;
   • atomic writes and unsafe workspace/path rejection; and
   • prior-snapshot behavior for the four supported report IDs.

5. Adapt useful tests from the deleted filesystem suite rather than recreating redundant low-value cases. Do not restore Scriptorium writes or retired report behavior.

Tests

Run:

go test ./internal/state ./internal/app
git diff --check

Exit Gate

The state package rejects incomplete or contradictory Promptkit-era artifacts, reads historical v1 records, writes only valid v2 records, and has focused offline coverage for its durable compatibility and filesystem contracts.

Stage 14: Complete Execution-Artifact Path Tracking

Status: Completed.

Goal

Make PromptExecutionArtifact.Paths accurately record every downstream artifact reached after a completed Promptkit run.

Work

1. Treat the execution artifact as an atomically updated durable record of the completed Promptkit execution and subsequent artifact destinations. Its status, provenance, validation, usage, and timing remain the provider-run outcome; later application failures do not change a successful Promptkit status into an execution failure.

2. Save the initial execution artifact after raw output is persisted, with RawOutputPath populated.

3. After each later successful write, update and atomically resave the same execution artifact with the corresponding reached path:

   • normalized generated text;
   • render context;
   • managed Markdown report;
   • an explicitly requested extra output copy, only after the copy succeeds; and
   • a Distributor notification artifact, including a persisted failure or status artifact when notification produced one.

4. Keep metadata and execution-artifact path values consistent after every successful checkpoint. Save the execution artifact before metadata so a metadata failure does not erase knowledge of a reached downstream artifact. Failure to update the execution artifact is terminal and returns a partial result containing the downstream artifact that was already written.

5. Refactor finalization return values only as needed to tell the orchestration layer which copy and notification paths were actually written. Distributor must continue uploading only the managed Markdown report.

6. A validation-rejected execution ends after raw output and therefore records only the raw-output path. An operational execution failure has no completed provenance and records only safe paths reached before that failure.

Tests

• Add table-driven execution-artifact lifecycle tests for success and every downstream failure point.

• Load the persisted execution artifact after normalized-text, context, template, copy, metadata, and notification failures and assert its status and exact reached paths.

• Assert that execution artifacts never contain generated bodies, rendered prompts, schemas, endpoints, parameters, or credentials.

• Run:

  go test ./internal/state ./internal/app
  git diff --check
  

Exit Gate

For every completed Promptkit run, its execution artifact contains exactly the safe downstream paths reached by the workflow and remains semantically correct when a later application stage fails.

Stage 15: Restore Single-Report Behavioral Coverage

Status: Completed.

Goal

Restore the risk-based application coverage removed during final cutover and prove the complete single-report Promptkit workflow through project-owned boundaries.

Work

1. Reintroduce a focused app test harness using real state, prompt-input, generated-text validation, render contexts, and templates with deterministic collector, executor, notifier, clock, and filesystem boundaries.

2. Add representative successful workflows for Daily, Today, Tomorrow, and Hourly. Verify report identity, exact prompt version, one collection, exact persisted YAML bytes passed to the executor, expected template output, optional copy behavior, and managed-report notification source.

3. Cover the required failure matrix:

   • inspection and missing credentials before collection;
   • preparation failure and callback persistence failure before provider work;
   • execution-time credential disappearance;
   • capacity rejection without retry;
   • cancellation and deadline;
   • generation and operational-validation failure;
   • completed Promptkit schema rejection with retained raw output;
   • generated-text decode/domain rejection;
   • render-context and template failure;
   • output-copy failure; and
   • notification failure.

4. Verify preparation persistence precedes provider execution, debug-write failure prevents provider execution, and execution-debug failure preserves previously reached normal and debug artifacts.

5. Verify Recent Changes, prior-snapshot selection, output naming, and Distributor template values for all four retained reports.

6. Adapt useful tests from the deleted app suite. Omit Scriptorium mechanics, subprocess interaction assertions, and retired report products.

7. Fix defects exposed by these tests only when the expected behavior is already decided by the roadmap or canonical policy. Record any new product question instead of silently choosing it.

Tests

Run:

go test ./internal/app
go test -race ./internal/app ./internal/adapters/promptkit
git diff --check

Exit Gate

The single-report workflow has deterministic behavioral coverage for all four reports, all consequential failure stages, artifact ordering, partial results, debug isolation, output copying, and notification behavior.

Stage 16: Refactor Prompt Generation Orchestration

Status: Completed.

Goal

Reduce the complexity and duplicated persistence logic in generatePromptReport without changing observable behavior.

Work

1. Use the Stage 12–15 tests as the refactoring safety boundary. Do not weaken assertions to accommodate structural changes.

2. Split the current orchestration into small app-owned operations with clear inputs and outcomes for:

   • deterministic input and initial state construction;
   • preparation callback persistence;
   • preparation-failure persistence;
   • operational-execution-failure persistence;
   • completed execution and raw-output persistence;
   • normalized text and render-context persistence;
   • managed report, optional copy, metadata, and notification finalization; and
   • reached-path updates shared by success and failure paths.

3. Keep workflow order visible in one coordinator. Do not introduce a generic workflow engine, hidden retry loop, provider-specific app type, or mutable global state.

4. Centralize the repeated rule that a successful artifact write updates the result before any following write can fail.

5. Preserve error identities, safe text, atomic writes, exact bytes, debug ordering, partial results, and notification behavior.

Tests

Run:

gofmt -w internal/app/*.go
go test ./internal/app ./internal/state ./internal/cli
go test -race ./internal/app
git diff --check

Exit Gate

The top-level coordinator communicates the workflow order without containing the full persistence implementation, duplicate failure branches are reduced, and every Stage 12–15 behavioral test passes unchanged.

Stage 17: Restore Batch Behavioral Coverage

Status: Completed.

Goal

Re-establish confidence that morning and evening batches preserve their pre-migration behavior while sharing one Promptkit executor.

Work

1. Add assembled batch tests proving:

   • one executor factory call and one executor per CLI invocation;
   • inspection of the full candidate set before collection;
   • one weather collection;
   • existing morning/evening planning and ordering;
   • sequential execution through the shared executor;
   • continuation after an independent report failure;
   • no retry after capacity rejection;
   • distinct identities and debug directories for multiple Daily dates; and
   • exact reached paths on successful and failed batch items.

2. Restore notification coverage for disabled notification, suppressed per-report notification, all-success batch notification, skipped notification after report failure, and persisted notification failure/status artifacts.

3. Restore output-directory, Today/Tomorrow naming, dynamic Daily planning, prior-snapshot, and managed-Markdown upload-source coverage.

4. Adapt useful tests from the deleted batch portions of the app and CLI suites. Do not restore retired report cases or Scriptorium fakes.

5. Fix only roadmap-defined batch regressions exposed by the restored tests.

Tests

Run:

go test ./internal/app ./internal/cli
go test -race ./internal/app
git diff --check

Exit Gate

Morning and evening batches are covered as assembled sequential workflows and demonstrably preserve collection, planning, continuation, output, debug, artifact, and notification contracts with one Promptkit executor.

Stage 18: Restore CLI And Inspection Coverage

Status: Completed.

Goal

Restore the user-facing command, summary, and historical inspection contracts removed with the old root test suite.

Work

1. Add parser and resolver tests for all four generate commands, both batch commands, shared flags, report-specific date rules, malformed input, --llm-debug-dir, --quiet, output paths, and rejection of retired report names.
2. Add assembled CLI tests for representative successful single and batch invocations using injected offline boundaries. Verify exactly one executor construction per action.
3. Cover pre-run errors with no invented run summary, successful and failed JSON summaries, quiet-mode behavior, safe human status output, partial paths, and omission of absent notification/debug fields.
4. Restore inspection tests for report listing and v1/v2 metadata, modules, data packages, prior snapshots, and sources. Include failed v2 runs and v1 fixtures using historical report IDs.
5. Assert that routine output never contains rendered prompts, schema bodies, data packages, generated bodies, endpoints, full parameters, credentials, or secret-like dependency errors.
6. Keep tests at stable CLI/app boundaries; do not restore assertions about private parser formatting or Scriptorium subprocess mechanics.

Tests

Run:

go test ./internal/cli ./internal/app ./internal/state
go run ./cmd/weatherreporter --help
git diff --check

Exit Gate

The supported CLI surface, summaries, quiet mode, partial failures, executor composition, and v1/v2 inspection behavior have deterministic offline coverage.

Stage 19: Finalize Documentation And Repository Verification

Status: Completed.

Goal

Close the audit remediation, make roadmap lifecycle state truthful, and verify the repository against the complete target contract.

Work

1. Update docs/roadmap/promptkit.md from future tense and “unimplemented” statuses to a completed roadmap record. Describe its old seven-report and Scriptorium material explicitly as the pre-migration baseline rather than current behavior.
2. Mark Stages 12–19 and this implementation plan complete only after their exit gates pass. Retain the concise completed-stage history unless the documentation policy calls for archival in the same change.
3. Review canonical architecture, app, state, CLI, Promptkit integration, operations, troubleshooting, configuration, and testing documentation against the corrected implementation. Update only actual current-state discrepancies; do not duplicate the roadmap.
4. Search current-state code, tests, examples, help, and non-roadmap documentation for stale Scriptorium terms, retired reports, old artifact fields, speculative-path descriptions, or claims of missing Promptkit implementation.
5. Confirm examples contain no credentials or private infrastructure values and load through config tests.

Final Verification

Run:

gofmt -w <all changed Go files>
go mod tidy
go vet ./...
go test -count=1 ./...
go test -race ./...
go run ./cmd/weatherreporter --help
git diff --check

Then verify explicitly:

• go list -m gitea.maximumdirect.net/eric/promptkit reports v0.4.0;
• no committed go.work, replace, secret fixture, or live-provider test exists;
• all four prompts inspect at exact version 1.0.0;
• no runtime prompt requests repair attempts;
• v1 fixtures remain inspectable and new runs write only v2;
• normal artifacts and output contain no sensitive prompt/debug content;
• failed-run metadata, execution artifacts, app results, batch items, and CLI summaries contain exactly the paths actually reached;
• help exposes only Daily, Today, Tomorrow, Hourly, morning, and evening; and
• managed Markdown remains the only Distributor upload source.

Exit Gate

Every migration and audit-remediation criterion is demonstrably satisfied, the restored tests protect the consequential contracts, canonical documentation describes the corrected implementation, and both roadmap documents are marked complete.

Open Questions

None. The roadmap and this completed plan record the decisions used for the audit remediation.