eric/weatherreporter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Define comparison artifact contracts

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-02 05:12:36 +00:00
parent af9cb0c0dc
commit 3bca2f41f7
4 changed files with 1508 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

800
docs/roadmap/implementation.md Normal file
View File

@@ -0,0 +1,800 @@
# LLM Profile Comparison Implementation Plan
Status: Ready for implementation.
## Purpose And Authority
This document is the ordered implementation plan for the accepted [LLM
Profile Comparison Roadmap](profile-comparison.md). Implement every stage in
numeric order. The roadmap owns feature purpose, policy, scope, and target
state; this plan owns implementation sequence, concrete interfaces, and stage
exit criteria.
This is one feature delivered over ten bounded coding prompts. A stage may
refactor code needed by a later stage, but it must leave the repository
compiling, tested, and internally coherent. Do not expose an incomplete
`compare` command before the stages that implement its full command contract.
## Implementation Rules
Apply these rules in every stage:
- Read `docs/development.md`, the task-specific documents it identifies, and
all files under `docs/policy/` before changing code. Treat those documents
and the feature roadmap as binding.
- Preserve the existing behavior of `generate` and `run`, including output
precedence, report/date semantics, notification, debug capture, structured
results, and exit behavior.
- Keep Promptkit types and calls behind `internal/adapters/promptkit` and the
dependency-neutral `internal/promptexec` interface. Do not import Promptkit
from application, CLI, or comparison-artifact packages.
- Keep comparison artifacts operator-owned and explicit. They are not durable
application state and must never be discovered or consumed implicitly by a
later invocation.
- Do not add a Weatherreporter concurrency limit or semaphore. One shared
Promptkit executor is safe for concurrent use, and Promptkit v0.5.0 owns its
engine-local backend capacity.
- Use dependency injection and deterministic fakes for application and CLI
tests. Tests must be offline, credential-free, race-safe, and independent of
completion timing and machine-specific paths.
- Add regression coverage in the same stage as each behavior. Prefer
contract-level assertions over incidental implementation detail, in
accordance with `docs/policy/testing.md`.
- Run `gofmt` on changed Go files and `git diff --check` in every stage. Run
focused tests while developing, then `GOWORK=off go test -count=1 ./...`
before completing the stage. Stages that add concurrency or filesystem
replacement must also run the relevant packages with `-race`.
- Do not commit, tag, push, or prepare a release unless the implementing prompt
separately requests it.
## Locked Implementation Contracts
The following choices make the stages decision-complete. Do not substitute a
different command shape, artifact schema, publication algorithm, or concurrency
policy without first updating the accepted roadmap and this plan.
### Application Request And Result
Add this exported application request, using the existing application aliases
and interfaces:
```go
type ComparisonRequest struct {
Config config.Config
Report ReportKind
ProfileIDs []string
WorkingDir string
OutputDir string
Replace bool
LLMDebugDir string
Date time.Time
Clock timeutil.Clock
Collector Collector
Executor promptexec.Executor
}
```
`OutputDir` is the optional exact bundle-directory override. `Clock` supplies
the comparison identity plus start and finish timestamps and defaults to
`timeutil.SystemClock` when nil. `Date` retains the existing report-date
meaning.
Do not include a notifier in this request. Making notification structurally
unavailable is clearer and safer than passing a disabled notifier through the
comparison path.
Add `app.CompareDetailed(context.Context, ComparisonRequest)` returning a
non-nil `*app.ComparisonResult` whenever report and destination resolution have
progressed far enough to describe the attempted comparison. Use these field
names and meanings:
```go
type ComparisonResult struct {
ComparisonID string
ReportID report.ID
ReportName string
PromptID string
PromptVersion string
PromptHash string
StartedAt time.Time
FinishedAt time.Time
Timezone string
ValidPeriod timeutil.Period
OutputDirectory string
ManifestPath string
DataPackagePath string
Total int
Succeeded int
Failed int
Results []ComparisonProfileResult
}
type ComparisonProfileResult struct {
Position int
ProfileID string
BackendID string
ModelName string
Status string
ValidationStatus promptexec.ValidationStatus
ReportPath string
LLMDebugPath string
Error *comparison.SafeError
}
```
`OutputDirectory` is absolute once destination resolution succeeds. Manifest
and data-package paths are absolute and nonblank only after publication.
Successful report paths and any returned debug paths are also absolute.
Profile results always follow selection order.
Profile counts describe profile outcomes only. A publication failure after all
profiles succeed therefore leaves `total == succeeded` and `failed == 0`, while
the command-level status and returned error still report failure.
Reject fewer than two profile IDs, whitespace-only IDs, and exact
case-sensitive duplicates. Preserve every accepted ID byte-for-byte and in
request order; do not trim, case-fold, deduplicate, or merge it with
`config.promptkit.profile`. Validate this invariant both in the CLI and at the
application boundary.
### Comparison And File Identity
Use the resolved report metadata's existing run ID to form:
```text
comparison_<report-run-id>
```
This comparison ID is computed once from the injected clock and shared by the
application result, manifest, and debug identities.
Derive a profile filename slug as follows:
1. retain ASCII letters, ASCII digits, `-`, and `_`;
2. replace each run of every other character, including dots, whitespace,
separators, control characters, and non-ASCII characters, with one `-`;
3. trim leading and trailing `-` and `_`;
4. use `profile` if the result is empty; and
5. truncate the result to 64 ASCII bytes, then trim trailing `-` and `_` again,
falling back to `profile` if necessary.
The report filename is:
```text
<ordinal>-<slug>.md
```
The one-based ordinal width is the greater of two and the number of decimal
digits in the profile count. Filename collisions after slug normalization are
safe because the ordinal is authoritative. The exact profile ID remains the
logical identity in results and the manifest.
Derive a default comparison directory by requiring the resolved report output
name to end in the exact `.md` suffix, removing that suffix, and prefixing the
remaining basename with `comparison-`. Resolve it beneath configured
`output.directory`, or beneath the invocation working directory when no output
directory is configured. An explicit `--out-dir` is the exact bundle directory,
resolves relative to the invocation working directory, and completely
overrides configured output. Return an absolute cleaned path.
### Versioned Bundle Contract
Create a dependency-neutral `internal/comparison` package. It owns deterministic
naming, the logical bundle and manifest types, validation, recognition of prior
bundles, hashing, staging, and transactional publication. It must not import
CLI, configuration, Promptkit, collectors, or report-generation packages.
Use the schema version:
```text
weatherreporter.comparison.v1
```
The UTF-8 `comparison.json` document is indented by two spaces and ends with one
newline. Its exact top-level JSON fields, in struct declaration order, are:
```text
schemaVersion
comparisonId
startedAt
finishedAt
reportId
validPeriod
timezone
promptId
promptVersion
promptHash
dataPackage
total
succeeded
failed
results
```
`validPeriod` has `start` and `end`. `dataPackage` has `path` and `sha256`.
Each ordered result has:
```text
position
profileId
backendId (omit when empty)
modelName
status
validationStatus (omit when unavailable)
reportPath (omit for failures)
error (omit for successes)
```
An error object has `category` and `message`. Status values are exactly
`succeeded` and `failed`. A successful result must have a relative report path,
must not have an error, and must have passed generated-text validation. A
failed result must not have a report path and must have a safe error. Use the
dependency-neutral string returned by `promptexec.CategoryOf` when available;
use `application` for an unclassified application, validation, rendering, or
debug-callback failure. Bound a manifest or normal-summary error message to
1,024 UTF-8 bytes without splitting an encoded rune. Never serialize wrapped
causes, provider bodies, prompts, schemas, model output, endpoints, or
credentials.
Encode all manifest timestamps using Go's standard `time.Time` JSON encoding.
Comparison start and finish timestamps are UTC; valid-period start and end
retain their resolved timezone offsets. Require nonzero timestamps, a finish
that is not before the comparison start, and a valid-period end strictly after
its start.
Use lowercase 64-character hexadecimal SHA-256 digests. `data-package.yml`
contains the exact immutable byte slice supplied to every execution. A logical
bundle contains that data package, ordered successful Markdown report bytes,
and the complete manifest. Manifest counts and paths must agree exactly with
the ordered results and logical bundle.
### Destination Safety And Publication
Perform a read-only destination preflight before prompt inspection, weather
collection, directory creation, or model execution. Reject:
- a filesystem root;
- the exact invocation working directory;
- an existing regular file or non-directory;
- an existing symlink, including a dangling symlink;
- an uninspectable target; and
- a nonempty directory unless `--replace` is present and it is a recognized
current-schema Weatherreporter comparison bundle.
An absent target and an existing empty real directory are accepted without
`--replace`. `--replace` is also harmless for those cases. Recognition must
decode the manifest with unknown fields rejected, validate every manifest
invariant, require `data-package.yml` and every declared successful report to
be regular non-symlink files, verify the data-package digest, and require the
directory entries to be exactly the manifest, data package, and declared
successful reports. Extra files, subdirectories, links, missing files, invalid
paths, duplicate paths, or a failed digest make the directory unrecognized.
Never follow a manifest path outside the bundle.
Re-run the destination safety check immediately before publication. Build the
entire logical bundle in memory, create the parent only when publishing, and
write a private uniquely named sibling staging directory with directory mode
`0700` and files mode `0600`. Check cancellation while staging and immediately
before the commit operation.
For an absent target, atomically rename the completed sibling staging directory
to the target. For an existing empty or recognized target, rename the target to
a unique sibling backup, rename staging to the target, and restore the backup
if the second rename fails. If restoration also fails, retain the backup and
return an actionable joined error that identifies its path. Once this rename
transaction begins, finish commit or rollback without abandoning it because of
a concurrent cancellation. The final pre-commit context check is the
linearization point: cancellation observed before it prevents replacement;
cancellation arriving after it does not retroactively undo a completed commit.
Remove the backup after a successful replacement and clean staging artifacts on
ordinary failures without scanning or modifying unrelated paths.
### Shared Preparation And Concurrent Execution
Extract an immutable internal prepared-report value containing the resolved
report, collection-derived facts, module snapshot, briefing metadata,
generated-text handler/render inputs, source warnings, and the one serialized
data-package byte sequence. Preparation resolves, collects, derives, snapshots,
and marshals once.
Split the generated-text workflow into three internal responsibilities:
1. deterministic shared preparation;
2. one profile-specific Promptkit execution, validation, and in-memory
Markdown rendering; and
3. publication, with optional ordinary notification.
`GenerateDetailed` composes those responsibilities once and retains its current
single-report behavior. Comparison never calls `GenerateDetailed` in a loop.
For comparison, inspect the exact prompt once, then inspect every profile
sequentially in selection order, including effective backend, model, and
credential availability, before weather collection. Initialize an explicitly
requested debug writer before those operations so pre-execution diagnostics
retain current behavior. Any preflight failure causes no collection, model
calls, or bundle publication.
After shared preparation, start one goroutine per profile against one shared
executor. Pass the same immutable data-package bytes and exact inspected prompt
version to every call. Preallocate an ordered outcome slice and let each
goroutine write only its own index; use synchronization to join all goroutines.
One profile failure does not cancel peers. Parent cancellation propagates to
every call, no additional calls are started after cancellation is observed, and
all started goroutines are joined before return.
Give each execution a deterministic debug run identity:
```text
<comparison-id>_<ordinal>-<slug>
```
The existing debug preparation artifact remains authoritative for the exact
profile ID. Debug callback failure is a failure of only that profile and must
not collide with or cancel peers.
After all non-cancelled outcomes finish, order them by original selection,
capture one UTC finish timestamp, and construct the complete or coherent
partial bundle. Publish partial bundles when one or more profiles fail, then
return the non-nil result with a generic aggregate error so the CLI exits
nonzero. Do not include provider detail in that aggregate error. Cancellation
or deadline expiration publishes no bundle. No comparison path constructs or
calls a Distributor notifier.
### CLI Contract
Add:
```text
weatherreporter compare REPORT [options]
```
Support `daily`, `today`, `tomorrow`, and `hourly`, with the same date policy as
`generate`. Accept applicable existing `--config`, `--units`, `--tz`, `--date`,
`--llm-debug-dir`, and `--quiet` flags plus a repeatable custom `--profile`, the
exact-directory `--out-dir`, and boolean `--replace`. Do not accept `--out`, and
do not use the configured default Promptkit profile as a selection. Construct
exactly one executor per command invocation.
The normal stdout JSON summary has these fields in declaration order:
```text
command
comparisonId
reportId
reportName
promptId
promptVersion
promptHash
status
startedAt
finishedAt
timezone
validPeriod
outputDirectory
manifestPath (omit when unpublished)
dataPackagePath (omit when unpublished)
total
succeeded
failed
results
error (omit on complete success)
```
`command` is `compare`; command status is `succeeded` only when every profile
succeeds and publication succeeds, otherwise `failed`. Each result mirrors the
application profile result with absolute report and debug paths. Preserve
profile order. A coherent partial bundle is still a failed command and exits
nonzero. Failures before a structured result exists use the existing CLI error
path without inventing a partial JSON object. `--quiet` suppresses normal
stdout and routine progress output but never changes work, artifacts, or exit
status.
## Stage 1: Define Comparison Artifact And Naming Contracts
### Goal
Introduce the dependency-neutral logical contract without altering any command
or existing generation path.
### Work
1. Create `internal/comparison` with manifest, result, data-package reference,
valid-period, safe-error, logical-bundle, and bundle-report types matching
the locked schema.
2. Add constants for the schema version, `comparison.json`, and
`data-package.yml`, plus explicit succeeded and failed statuses.
3. Implement profile-list validation, deterministic slugging, ordinal width,
report filename construction, comparison ID construction, default directory
name derivation, SHA-256 formatting, safe UTF-8 error truncation, manifest
validation, and deterministic JSON encoding.
4. Ensure validation rejects inconsistent counts, positions, statuses, paths,
digests, timestamps, identities, successful results without reports,
failures with reports, duplicate paths, non-basename artifact paths, and
traversal-capable names.
5. Keep all filesystem mutation and application orchestration out of this
stage.
### Tests And Exit Criteria
- Table-test unusual profile IDs, empty slugs, separators, dots, controls,
Unicode, length limits, normalized collisions, and profile counts above 99.
- Round-trip the exact JSON field names, ordering, indentation, newline,
optional fields, safe errors, hashes, and manifest invariants.
- Confirm exact case-sensitive profile identity and duplicate handling.
- Run `GOWORK=off go test -count=1 ./internal/comparison` and the repository-wide
standard test command. The repository remains behaviorally unchanged.
## Stage 2: Implement Safe Bundle Recognition And Transactional Publication
### Goal
Complete the isolated filesystem boundary, including output resolution and
whole-directory replacement, before application concurrency depends on it.
### Work
1. Add read-only destination planning and preflight to `internal/comparison`,
parameterized by the absolute working directory, exact target, and replace
authorization.
2. Implement strict current-schema prior-bundle recognition with unknown JSON
fields rejected, manifest validation, exact-entry checks, symlink rejection,
regular-file checks, containment, and digest verification.
3. Implement sibling staging, fixed permissions, cancellation checks, absent
target publication, existing-target backup and replacement, rollback, and
targeted cleanup according to the locked algorithm.
4. Add an application output resolver, alongside existing output helpers, that
applies explicit/configured/current-directory precedence and derives the
default comparison name from the resolved report output name. Do not change
ordinary output resolution.
5. Return typed or inspectable errors sufficient for application and CLI code
to add context without exposing filesystem internals as a public schema.
### Tests And Exit Criteria
- Cover relative and absolute explicit destinations, configured destination
precedence, all report-derived names, and an invalid default `.md` suffix.
- Cover absent, empty, regular-file, symlink, dangling-symlink, filesystem-root,
exact-working-directory, unreadable, recognized, and unrecognized targets.
- Cover extra entries, missing reports, subdirectories, manifest traversal,
unknown fields, bad counts, digest mismatch, stale-report removal, successful
replacement, second-rename failure with restoration, and cancellation before
commit. Use narrow injected filesystem hooks only where real filesystem
behavior cannot deterministically reach a consequential failure path.
- Verify a failed publication leaves the prior recognized bundle byte-for-byte
intact and does not touch sibling paths.
- Run focused tests with `-race`, then the repository-wide standard tests.
## Stage 3: Add Ordered Multi-Profile Preflight
### Goal
Make prompt and profile validation reusable for comparison without collecting
weather or executing a model.
### Work
1. Extend `internal/app/prompt_inspection.go` with a comparison-oriented request
and result that inspect one resolved prompt exactly once and selected
profiles sequentially in request order.
2. Reuse the existing credential-lookup policy and return dependency-neutral
prompt hash/version plus effective backend and model for every profile.
3. Apply the locked application-level profile validation before any executor
inspection.
4. Fail on the first ordered preflight error with prompt/profile context, while
ensuring no collection or execution occurs. Do not silently fall back to
`config.promptkit.profile`.
5. Leave existing single and batch inspection entry points intact unless a
private shared helper can remove duplication without changing behavior.
### Tests And Exit Criteria
- Assert one prompt inspection, ordered profile inspections, effective
backend/model preservation, exact profile IDs, and credential failures.
- Assert invalid lists and any prompt/profile preflight failure perform no
collection or execution using explicit fakes.
- Run `GOWORK=off go test -count=1 ./internal/app` and repository-wide tests.
## Stage 4: Extract Immutable Shared Report Preparation
### Goal
Create the prepare-once boundary required for comparison while preserving all
ordinary generation behavior.
### Work
1. Extract an unexported immutable prepared-report value in `internal/app` with
the resolved report, facts, module snapshot, briefing metadata, handler and
render inputs, source warnings, and exact serialized data-package bytes.
2. Move current collection, derivation, snapshot, metadata, prompt-input, YAML
marshal, and generated-text-handler construction into one preparation
function.
3. Refactor `GenerateDetailed` and batch generation to use that function once
per generated report without altering external requests or results.
4. Copy mutable inputs where necessary at the boundary so concurrent consumers
cannot mutate shared maps or byte slices. Treat the prepared value as
read-only after construction.
5. Do not add comparison goroutines or the CLI command in this stage.
### Tests And Exit Criteria
- Preserve all existing generation and batch tests.
- Add focused tests showing one collector call, one deterministic serialized
package, stable warnings/metadata, and isolation from caller mutation.
- Compare ordinary generated report bytes, result fields, debug preparation,
notification, and failure behavior before and after the refactor through
existing contract tests.
- Run application tests and repository-wide tests.
## Stage 5: Separate Profile Execution And In-Memory Rendering From Publication
### Goal
Permit several profiles to consume one prepared report without duplicating or
prematurely publishing output.
### Work
1. Extract a profile-specific internal function that accepts the immutable
prepared report, exact prompt inspection, effective profile inspection, and
optional deterministic debug reference.
2. Have it invoke `promptexec.Executor.Execute`, preserve the existing
preparation callback contract, validate generated text, build render
context, and render Markdown to memory. Return a dependency-neutral outcome
and bytes; do not write the report or notify.
3. Extract ordinary single-report publication into a separate function that
performs the existing context check, atomic file write, and optional
Distributor notification.
4. Recompose `GenerateDetailed` from prepare, execute/render, and publish. Keep
its debug path, validation status, output path, notification, warnings, and
error wrapping unchanged.
5. Make debug callback bookkeeping local to one execution. Remove any shared
mutable workflow field that would race when the execution function is used
concurrently.
### Tests And Exit Criteria
- Prove execution/rendering alone performs no filesystem publication or
notification.
- Preserve ordinary success, validation failure, rendering failure, debug
callback failure, cancellation, atomic output, and notification tests.
- Confirm one ordinary generation still prepares and executes exactly once.
- Run application tests with `-race`, then repository-wide tests.
## Stage 6: Build The Concurrent Ordered Profile Execution Core
### Goal
Implement and test concurrency independently of destination publication and CLI
assembly.
### Work
1. Add a private comparison execution coordinator in `internal/app` that takes
one immutable prepared report, one prompt inspection, ordered profile
inspections, comparison identity, optional debug writer, and one shared
executor.
2. Preallocate ordered outcomes, start one goroutine per selected profile while
observing parent cancellation, and join every started goroutine. Each
goroutine may write only its own result slot.
3. Pass the exact same data-package byte content and prompt version to every
call. Do not recollect, rebuild, marshal, retry, or use `GenerateDetailed`.
4. Continue peers after an individual execution, validation, rendering, or
debug callback failure. Convert every failure to a bounded safe structured
error while retaining a wrapped internal error for aggregate diagnosis.
5. Use requested-profile order, not completion order, for outcomes and report
filenames. Generate distinct debug identities from comparison ID, ordinal,
and safe slug.
6. On parent cancellation or deadline, propagate context, join all work, and
mark non-successful outcomes safely. Do not construct a publishable bundle
for a cancelled coordinator.
### Tests And Exit Criteria
- Use a barrier-based fake executor to prove overlap without sleeps and to
observe more than one in-flight call.
- Assert the exact prompt version and byte-for-byte identical data package for
eight to twelve profiles, deterministic result order under reversed
completion, and no data races.
- Assert one failure does not cancel peers, cancellation reaches all started
calls and joins them, and debug references are distinct and deterministic.
- Do not duplicate Promptkit's internal capacity tests. Retain adapter tests
showing that Weatherreporter delegates prepared execution through one
concurrency-safe engine.
- Run `GOWORK=off go test -race -count=1 ./internal/app
./internal/adapters/promptkit`, then repository-wide tests.
## Stage 7: Assemble Application-Level Comparison And Bundle Publication
### Goal
Expose the complete application use case with safe partial publication,
aggregate failure behavior, and no notification.
### Work
1. Add the locked exported application request, result, profile-result, and
`CompareDetailed` entry point.
2. Orchestrate in this exact order: validate request; resolve report/date and
comparison identity; resolve and preflight destination; initialize optional
debug writing; inspect prompt and every profile; collect and prepare once;
execute profiles concurrently; capture finish time; build manifest and
logical bundle; re-preflight and publish.
3. Build `data-package.yml` from the exact prepared bytes and include only
successful rendered reports. Derive manifest and application results from
the same ordered outcome source so they cannot drift.
4. On one or more profile failures, publish the coherent partial bundle, set
ordered result errors and counts, then return the non-nil result plus a
generic aggregate error such as `comparison completed with N failed
profiles`.
5. On cancellation, deadline, preflight, preparation, or publication failure,
publish nothing new. Preserve a recognized prior bundle. Return the most
complete safe result available and an error with useful operation context.
6. Populate absolute manifest, data-package, successful-report, and debug paths
only when those artifacts exist. Keep manifest paths relative basenames.
7. Ensure no Distributor notifier is accepted, created, or invoked anywhere in
this path.
### Tests And Exit Criteria
- Add assembled application tests for complete success, mixed profile failure,
all-profile failure, preflight failure, collection/preparation failure,
cancellation, publication failure, and authorized replacement.
- Assert prompt inspection once, profile inspection before collection, weather
collection once, serialized package reuse, completion-independent order,
exact counts/digest/paths, generic aggregate errors, and no model calls before
all preflight succeeds.
- Assert complete and partial comparison never notify and that cancellation or
publication failure preserves an existing bundle.
- Run application and comparison packages with `-race`, then repository-wide
tests.
## Stage 8: Add Compare Command Parsing And Request Construction
### Goal
Introduce the complete command syntax and validation while keeping execution
injected and testable.
### Work
1. Extend `internal/cli.Runner` with an injectable comparison function, default
it to `app.CompareDetailed`, and add a private comparison command parser and
request-construction path. Do not route it from the public root switch until
Stage 9 can provide the complete summary and exit contract.
2. Implement a repeatable flag value that preserves every `--profile` token in
exact command-line order.
3. Parse applicable common flags plus `--out-dir` and `--replace`; reject
missing/unknown reports, unsupported date use, fewer than two profiles,
whitespace-only profiles, exact duplicates, unexpected positional
arguments, and unsupported flags such as `--out`.
4. Reuse existing configuration loading, units/timezone overrides, report/date
resolution semantics, working-directory handling, and debug-root behavior.
5. Pass explicit output override and ordered profiles to the application. Do
not merge the configured default profile. Construct one Promptkit executor
only after syntax/config validation and before the application call.
6. Return the application result and error through a private CLI execution
seam for Stage 9 to format. Do not add public root routing or help text in
this stage, so users cannot reach a command with an incomplete output
contract.
### Tests And Exit Criteria
- Table-test report/date behavior, repeated profiles and ordering, invalid
lists, output precedence inputs, `--replace`, common overrides, and
rejected flags/arguments.
- Assert invalid syntax and configuration do not construct an executor or call
the application.
- Call the private comparison execution seam in tests and assert one valid
invocation constructs one executor and one request with the exact selections
and paths.
- Run CLI tests and repository-wide tests.
## Stage 9: Complete CLI Summaries, Quiet Mode, And Exit Behavior
### Goal
Finish the end-to-end user-facing command contract, including coherent partial
failure reporting.
### Work
1. Add comparison summary and ordered profile-summary types with the exact
locked JSON field order and `omitempty` behavior.
2. Add public root routing and root help text now that parsing, execution,
formatting, and exit behavior can land as one complete command.
3. Map complete, partial, cancelled, and publication-failed application results
into one normal stdout object whenever a structured result exists. Use
absolute returned paths and safe structured errors; do not serialize wrapped
internal or provider details.
4. Return success only for complete execution and publication. Return the
application error after writing a failed structured summary so the process
exits nonzero. Preserve the existing unstructured error path for failures
before a result exists.
5. Implement `--quiet` as output suppression only. It must not alter execution,
comparison artifacts, replacement, error return, or exit status.
### Tests And Exit Criteria
- Golden or exact-structure tests cover successful, partial, all-failed,
pre-execution, cancelled, and publication-failed summaries, field omission,
ordered results, and safe errors.
- CLI integration tests assert stdout/stderr discipline, quiet behavior,
nonzero partial failure, default and explicit output paths, and one executor
per invocation.
- Run CLI and application tests with `-race`, then repository-wide tests.
## Stage 10: Reconcile Canonical Documentation And Run The Final Gate
### Goal
Document only the implemented behavior in its canonical owners, remove
roadmap-only language from current-state documentation, and perform the full
repository validation gate.
### Work
1. Update `docs/cli.md` with authoritative syntax, flags, profile validation,
JSON summary, quiet behavior, and exit semantics.
2. Update `docs/operations.md` with the prompt-comparison workflow, output
destination precedence by link to `docs/config.md`, sensitivity guidance,
flat bundle lifecycle, guarded replacement, partial results, cancellation,
and cleanup expectations. Do not duplicate the complete CLI reference.
3. Add `docs/integrations/comparison-bundle.md` as the canonical durable schema
and layout contract. Document the exact version, JSON fields, filenames,
path relativity, hashes, success/failure invariants, compatibility boundary,
and sensitive-content caveat.
4. Update `docs/integrations/promptkit.md` with the consumer-visible
prepare-once/execute-many boundary and Promptkit-owned backend capacity,
linking upstream rather than duplicating its concurrency reference.
5. Add or update focused `docs/internal/` documents for application comparison
orchestration, prepared report flow, concurrent execution, prompt debugging,
and bundle publication mechanics. Link to canonical CLI, configuration,
operations, integration, and architecture owners instead of restating them.
6. Update `docs/policy/architecture.md` for the implemented stateless,
concurrency, notification, and publication invariants; update
`docs/development.md` for new package/file routing; update
`docs/policy/documentation.md` only if the existing ownership table no longer
assigns every introduced topic correctly.
7. Check README and other current-state documents for navigation or stale
claims. Add only short orientation and canonical links where useful.
8. Mark the feature roadmap `Implemented; retained temporarily for
post-implementation review` and this plan `Complete`. Retain both until the
requested implementation review moves durable facts to their owners and the
maintainer authorizes roadmap cleanup.
### Tests And Exit Criteria
- Verify all changed relative links and fenced examples, and search for stale
claims that comparison is unavailable or planned.
- Confirm current-state docs do not duplicate full command, configuration,
schema, or implementation contracts outside their canonical owners.
- Run the release-equivalent local gate:
```sh
set -eu
test -z "$(git ls-files go.work go.work.sum)"
test ! -e vendor
if grep -Eq '^[[:space:]]*replace([[:space:]]|\()' go.mod; then
printf '%s\n' 'go.mod contains a replacement' >&2
exit 1
fi
GOWORK=off go test -count=1 ./...
GOWORK=off go test -race -count=1 ./...
GOWORK=off go vet ./...
GOWORK=off go build ./...
GOWORK=off go mod tidy -diff
unformatted="$(git ls-files '*.go' | while IFS= read -r file; do gofmt -l "$file"; done)"
test -z "$unformatted"
git diff --check
```
- Run `GOWORK=off go run ./cmd/weatherreporter --help`; confirm output matches
`docs/cli.md` without requiring credentials or network access.
- Inspect the final diff for accidental generated artifacts, secrets,
workspaces, vendored dependencies, release notes, or unrelated changes.
## Open Questions
None. The accepted feature roadmap and the locked contracts in this plan define
the decisions required to implement all ten stages.