Finalize profile comparison implementation
This commit is contained in:
13
docs/cli.md
13
docs/cli.md
@@ -138,20 +138,27 @@ The comparison status is `succeeded` only when every selected profile succeeds
|
||||
and the bundle is published. Individual profile failures still publish a
|
||||
complete partial bundle and return a failed command result. Cancellation or a
|
||||
failure before publication omits the artifact paths and returns a safe
|
||||
top-level error; it does not expose provider diagnostics. See the
|
||||
top-level error; the resolved `outputDirectory` and finalized timestamp remain
|
||||
when available. The safe error includes only a category and message: aggregate
|
||||
and unclassified application failures use `application`; cancellation uses
|
||||
`canceled`; deadlines use `deadline_exceeded`; prompt execution uses its
|
||||
published Promptkit category; destination failures use `destination_<kind>`;
|
||||
and committed cleanup failures use `publication_cleanup`. It does not expose
|
||||
provider diagnostics, filesystem causes, or recovery paths. See the
|
||||
[comparison bundle contract](integrations/comparison-bundle.md) for durable
|
||||
artifact fields and failure invariants.
|
||||
|
||||
If the bundle is published but cleanup of its replaced prior bundle fails, the
|
||||
summary still includes the published artifact paths and has status `failed`.
|
||||
Its JSON error remains generic; the returned command error identifies the
|
||||
Its JSON error is `publication_cleanup` with the message `comparison published
|
||||
but cleanup did not complete`; the returned command error identifies the
|
||||
retained backup path for operator recovery.
|
||||
|
||||
## Flag Reference
|
||||
|
||||
| Flag | Accepted by | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `-h`, `--help` | top level | Show help. |
|
||||
| `-h`, `--help` | top level, `compare` | Show help without loading configuration or contacting a provider. |
|
||||
| `--config PATH` | all commands | Load `PATH` instead of `/usr/local/etc/weatherreporter/config.yml`. |
|
||||
| `--units VALUE` | `generate`, `run`, `compare` | Override `weather_api.units` for this command. |
|
||||
| `--tz NAME` | `generate`, `run`, `compare` | Override `weather_api.timezone` for this command. |
|
||||
|
||||
@@ -27,6 +27,11 @@ debugging, prompt inspection, or collection. It then inspects the one prompt
|
||||
and every selected profile, collects once, and delegates shared report
|
||||
construction to the prepared-report flow. It does not accept a notifier.
|
||||
|
||||
Once the destination is resolved, the partial result retains its absolute
|
||||
output directory even when later preflight, debug initialization, inspection,
|
||||
collection, or preparation fails. Every initialized result is finalized with a
|
||||
finished timestamp. Artifact paths are added only after publication commits.
|
||||
|
||||
The comparison execution core starts each inspected profile independently,
|
||||
keeps results in selection order, and waits for all started work. Independent
|
||||
profile failures are recorded and do not stop peers. Context cancellation marks
|
||||
@@ -39,7 +44,8 @@ internals](comparison-publication.md).
|
||||
When publication has committed its new bundle, application results contain the
|
||||
absolute manifest, data-package, and successful report paths even if removal of
|
||||
the previous sibling backup then fails. That cleanup failure is still returned
|
||||
as an operational error rather than treating the new bundle as unpublished.
|
||||
as an operational error rather than treating the new bundle as unpublished;
|
||||
the returned error retains the recovery path and underlying filesystem cause.
|
||||
|
||||
## Boundaries And Verification
|
||||
|
||||
|
||||
@@ -16,8 +16,10 @@ outcomes as skipped or failed, joins work, and prevents bundle publication.
|
||||
When debugging is enabled, each execution receives a deterministic reference
|
||||
derived from the comparison identity, ordered profile position, and safe
|
||||
profile slug. This keeps concurrent captures separate. The debug writer itself
|
||||
owns secure-root validation and file permissions; operational retention and
|
||||
sensitivity are documented in the [operations guide](../operations.md).
|
||||
owns secure-root validation and file permissions. It safely creates shared
|
||||
missing ancestors during concurrent writes, then rejects symlink and non-
|
||||
directory components. Operational retention and sensitivity are documented in
|
||||
the [operations guide](../operations.md).
|
||||
|
||||
The output result and its safe errors are converted into the durable contract
|
||||
only by comparison publication. See [comparison publication
|
||||
|
||||
@@ -10,15 +10,17 @@ Destination planning is read-only. It requires an exact absolute target that
|
||||
is neither the filesystem root nor the working directory, rejects unsafe
|
||||
symlinks and non-directories, accepts a missing or empty directory, and permits
|
||||
replacement only for a recognized current bundle. Publication rechecks that
|
||||
plan, writes a private sibling staging directory, and installs it atomically.
|
||||
Replacement temporarily moves the recognized prior bundle aside, reauthorizes
|
||||
that moved entry, and restores it if installation fails.
|
||||
authorization immediately before it writes a private sibling staging directory.
|
||||
For replacement, it moves the prior bundle to a private sibling backup,
|
||||
reauthorizes that moved entry, and restores it if installing the new bundle
|
||||
fails.
|
||||
|
||||
Publication reports whether the new bundle committed. Once the new bundle is
|
||||
installed, a failure to remove the prior sibling backup does not roll it back.
|
||||
The result retains the absolute backup path and returns an inspectable cleanup
|
||||
error that unwraps the filesystem cause so an operator can recover the prior
|
||||
bundle if needed.
|
||||
The new bundle is committed only after the staged directory has been installed
|
||||
at the target. From that point its artifact paths are authoritative: a failure
|
||||
to remove the retained sibling backup does not roll back the new bundle.
|
||||
Publication returns an inspectable cleanup error with the absolute backup path
|
||||
and underlying filesystem cause so an operator can recover or remove that
|
||||
backup manually.
|
||||
|
||||
The application preflights before prompt inspection and collection, then
|
||||
preflights again before publication. A cancellation or any failure before the
|
||||
|
||||
@@ -152,9 +152,10 @@ cancellation and pre-publication errors leave the prior destination unchanged.
|
||||
|
||||
If a replacement commits but cleanup of its prior sibling backup fails, the new
|
||||
bundle remains valid and its artifact paths appear in the failed command
|
||||
summary. Preserve the retained backup path reported by the command error until
|
||||
it has been inspected and cleaned up manually; do not remove the new bundle to
|
||||
retry that cleanup.
|
||||
summary. The summary records a safe `publication_cleanup` error, while the
|
||||
returned command error reports the retained backup path. Preserve that backup
|
||||
until it has been inspected and cleaned up manually; do not remove the new
|
||||
bundle to retry that cleanup.
|
||||
|
||||
Enable explicit debug capture only when content-rich Promptkit diagnostics are
|
||||
necessary.
|
||||
|
||||
@@ -1,19 +1,18 @@
|
||||
# LLM Profile Comparison Implementation Plan
|
||||
|
||||
Status: Follow-up work planned after post-implementation review.
|
||||
Status: Complete.
|
||||
|
||||
## Purpose And Authority
|
||||
|
||||
This document is the ordered implementation plan for the accepted [LLM
|
||||
Profile Comparison Roadmap](profile-comparison.md). The roadmap owns the
|
||||
feature purpose, policy, scope, and desired end state. This plan records the
|
||||
completed implementation and defines the remaining corrective work discovered
|
||||
during post-implementation review.
|
||||
completed implementation and records the corrective work completed during
|
||||
post-implementation review.
|
||||
|
||||
Stages 1 through 10 are complete. Implement Stages 11 through 15 in numeric
|
||||
order, using one gpt-5.6-terra coding prompt per stage. Each stage is scoped to
|
||||
leave the repository compiling, tested, documented to its implemented boundary,
|
||||
and internally coherent.
|
||||
All implementation work in this plan is complete. The recorded work leaves the
|
||||
repository compiling, tested, documented to its implemented boundary, and
|
||||
internally coherent.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# LLM Profile Comparison Roadmap
|
||||
|
||||
Status: Implemented; retained temporarily for post-implementation review.
|
||||
Status: Implemented; retained as the feature roadmap.
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
Reference in New Issue
Block a user