# Kit Upgrade Review Remediation

## Review Scope

Reviewed implementation commits `31b48d2` through `7ac9d94` against
[the feature roadmap](upgrade-kit.md) and [the implementation plan](implementation.md).
The review covered configuration selection/defaults, production client wiring,
error translation, execution observation, scheduler/debug propagation, current
documentation, and tests. No production model calls were made.

The implemented architecture is consistent with the intended design: ConfigKit
v0.4.0 owns composition and overlay provenance behind the configuration boundary;
PromptKit v0.11.0 owns transient retries and structural repairs behind the LLM
adapter. Notarius retains stage retry/disposition policy, completion-scoped
scheduling, application-owned error/observation types, and opt-in debug output.
No new runtime defect or unrequested restriction was established in this review.

## Finding 1 — Complete The Missing Retry And Observation Verification

**Priority:** Medium. This is a verification gap, not a demonstrated runtime
failure. Stages 5–7 are marked complete, but their consequential cross-boundary
acceptance tests were not implemented.

### Evidence And Risk

- `internal/framework/llm/promptkit_client_test.go:58` adds only a successful,
  single-attempt observation assertion with positive prompt-token usage. It
  does not exercise absent versus partial/known-zero usage, failed/canceled
  execution summaries, preparation failure, or concurrent request isolation.
- `TestPromptKitClientAppliesConfiguredBuiltInTransientRetries` covers basic
  status recovery, explicit zero, exhaustion, and a noneligible status. It
  discards the response, so does not protect terminal accounting. There is no
  combined transient-retry/structural-repair scenario or cancellation while
  waiting on Retry-After.
- `internal/framework/llm/scheduled_client.go:40` now preserves a response when
  the inner call fails. No corresponding test protects that behavior. Returning
  an empty response again would silently remove precisely the failed-call
  observations this feature is intended to retain.
- `internal/framework/pipeline/debug.go:267` and `:277` add execution summaries
  to response artifacts and call references, plus fallback/content-splitting
  propagation. Neither the debug nor terminal-debug tests were extended to
  exercise these fields. Passing existing tests does not prove that failed-call
  metadata reaches persisted records.

These are Notarius-owned integration risks, not a request to re-test PromptKit's
backoff algorithm. Losing failed-call observations or misrepresenting incomplete
usage would be difficult to detect in ordinary successful runs. A focused set
of behavioral tests is justified under the testing policy.

### Required Remediation

Use existing test assets, local HTTP fixtures, scheduler wrappers, and debug
recorders. Combine related assertions; no fixed test count or new test framework
is required.

1. Add an adapter scenario with a transient failure, a structurally invalid
   completion, and a successful repair. Exercise retries in both generations
   with small explicit budgets, and assert terminal attempts/retries/repair
   counts without double-counting. Use controlled usage fields to distinguish
   known zero, unknown, and partial aggregate usage. Preserve legacy usage
   semantics independently.
2. Cover failed and canceled completions, including cancellation during a
   Retry-After wait. Verify caller cancellation identity, bounded completion,
   and retained execution metadata. Use signaling and a generous bounded test
   timeout, not exact sleep durations or jitter assertions. Cover a preparation
   failure with no execution summary. Add a small concurrent-call check for
   request-local summary isolation, which race tests alone do not prove.
3. Exercise an error response through the real scheduled client and existing
   debug wrapper/recorder seams. Assert that the summary reaches response
   artifacts and terminal call references even without a provider response
   body. Cover the content-splitting/fallback path with a representative response.
   Reuse existing no-debug and redaction tests rather than duplicating their
   whole inventories. Keep new summary assertions metadata-only.

Retain existing scheduler policy, stage/validator budgets, cancellation
disposition, warning semantics, and debug opt-in behavior. Do not introduce a
total deadline, new retry caps, telemetry artifacts, or live API requirements
to make these tests easier. Fix production code only if these checks reveal a
violation of the already selected behavior.

**Acceptance:** The added tests fail if failed responses lose their execution
summary at an adapter/wrapper boundary, usage presence is collapsed into zero,
or retry/repair totals are double-counted. They pass with the intended behavior
and provide evidence for the previously unverified stage acceptance criteria.

**Validation:** Run focused LLM/pipeline tests, race-enabled tests for those
packages, and `GOWORK=off go test ./...`. No production credentials are needed.

## Checks Completed During Review

- `GOWORK=off go test ./...`
- `GOWORK=off go vet ./...`
- `GOWORK=off go build ./...`
- `GOWORK=off go mod tidy -diff`
- `GOWORK=off go test -race ./internal/framework/llm ./internal/framework/pipeline`
- CLI configuration validation for the minimal and complete examples, including
  the complete example with `--config-overlay development`.
- `git diff --check`

All passed. Existing green tests do not substitute for the missing behavioral
coverage above. Temporary diagnostic test material used during review was
removed; no application code was changed.