Files
promptkit/docs/roadmap/remediation.md

8.5 KiB
Raw Blame History

Configuration Handoff And Repair Review

Reviewed implementation stages 1–17 against the implementation plan, the feature roadmap, and accepted ADRs 0001 and 0002. Implementation baseline: 40f8a6825359b4923645256e230a050dfdb00710; reviewed changes begin after 248d0db.

Findings

1. Narrow The Pre-Handoff Mutation Wording To The Accepted Expectation

Status: open. Priority: P3. Implementation stage: 16.

Locations: engine.go:280 and docs/consumers/pkg-promptkit.md:132.

NewFromConfig says callers must not mutate selected settings between loading and handoff; the consumer guide repeats that blanket instruction. The accepted roadmap instead asks callers to hand off before changes that invalidate the frozen inspection/provenance evidence, and expressly treats this as a usage expectation rather than a new enforcement mechanism. The published wording prohibits even value-preserving reassignment or copying, which needs no such restriction.

Required change: describe the narrower consistency expectation in both owners. Explain that construction consumes the resolved settings while inspection retains the completed load's frozen evidence, so callers should keep those consistent at handoff. Preserve the existing protection against mutation after construction. No mutation detection, extra validation pass, or new error is needed.

Verification: review both passages against the feature roadmap's ownership paragraph; ensure they no longer impose an unconditional mutation prohibition. This is a documentation correction and does not require a new runtime test.

2. Keep Application-Field Sensitivity Policy With The Application

Status: open. Priority: P3. Implementation stage: 16.

Location: docs/formats.md:235–236.

The backend format reference says an unrecognized backend field is application data and must not contain secrets. The accepted extension-point contract assigns interpretation, validation, and sensitivity rules for those fields to the application. The implementation follows that contract: inline fields remain in native inspection, application sensitivity predicates compose with PromptKit's, and those fields are omitted from resource reports and provider parameters. The added prohibition unnecessarily narrows the application-owned data contract.

Required change: retain the useful warning that a raw api_key extension is not a PromptKit credential input and is not automatically redacted by PromptKit. State that the application owns sensitivity declarations and disclosure policy for its extension fields, rather than prohibiting their contents. Review the adjacent profile wording for the same distinction between PromptKit credentials and application-owned data. Do not add automatic credential behavior or a new validator for application fields.

Verification: compare the revised text with the feature roadmap's application-specific-fields contract and existing focused-inspection sensitivity coverage. No change to execution or redaction implementation is needed.

3. Preserve Effective Input Metadata Supplied By Application Defaults

Status: open. Priority: P2. Implementation stages: 5–6 and 13.

Locations: internal/configuration/load.go:208–216, internal/configuration/prompts.go:174–180, and internal/configuration/resources.go:81–87.

The new application-owned loading path preserves ConfigKit Defaults and application resolution callbacks, but the existing input-inspection machinery still equates FieldInfo.Explicit == false with an absent effective value. finalFieldState consequently discards metadata for values supplied by Go defaults or recorded derived defaults. These are valid final values, not missing configuration.

Reproduced through the public API with a completed ConfigKit load whose Definition.Defaults returns a prompt input equivalent to:

promptkit.PromptInputSettings{
    Name: "doc", Required: true,
    ContentType: new("text/plain"), Description: new("default description"),
}

With a valid prompt and default profile, NewFromConfig succeeds. The final typed input retains those values; native metadata describes their defaulted, nonnull presence (Explicit: false, Null: false, default layer), but:

  • InspectPrompt returns Required: nil instead of the effective true value.
  • InspectRequirements correctly classifies doc as required but returns nil content type and description, contrary to its effective-metadata contract.
  • ValidateRequest without doc fails with ErrPromptRender and a missing required-input error, confirming that execution uses the true value that resource inspection fails to report.

A second public-API reproduction using an application Resolve callback and RecordDerivedDefault for an omitted required and content_type has the same result. Source-loaded inputs ordinarily have explicit metadata, which is why the migrated source fixtures and current handoff tests do not expose this gap.

Required change: capture effective input metadata from the final typed values and native final metadata without treating source explicitness as a universal presence test. Preserve the existing ordinary omission/null and explicit false/empty-string behavior. Keep source provenance in native inspection; do not inspect raw sources, replay history, reject application defaults, or rerun resolution to repair the report.

Verification: add public integration regressions for both Go defaults and recorded application-derived defaults. Compare InspectPrompt, InspectRequirements, and preparation/preflight for the same captured input, and retain existing source-presence cases. Include a collection case if the fix introduces separate root/collection handling; the current capture logic is shared.

Initial Review Results And Validation

  • Stage 1: ConfigKit v1.4.0, unchanged Go minimum, and commit-pinned attribution match the selected module and its notices.
  • Stages 2–3: configuration and runtime inputs resolve mode-dependent repair budgets; explicit zero and null/omission, whole-contract replacement, prepared retention, and existing repair accounting are covered.
  • Stages 4–9: shared settings aliases, composed stateless rules, final-metadata resource capture, native focused inspection, explicit source mounting, and shared constructor/assembly paths match the target architecture.
  • Stages 10–15: migrated tests and maintained offline examples exercise the changed boundaries, including concurrent loads, cross-filesystem inheritance, application extensions, ownership, and standalone/handoff equivalence.
  • Stages 16–17: canonical documentation is updated, subject to the two wording corrections above. No additional functional or architectural defect was found.

Fresh checks passed with Go 1.27.4, GOTOOLCHAIN=local, and GOWORK=off: go test -race ./... and go vet ./.... Changed Go files have no gofmt differences, and the implementation diff passes git diff --check.

This review changes only this findings document; it does not implement fixes.

Integration-Focused Follow-Up

The second review revisited stages 1–17 at their interaction boundaries: the ConfigKit upgrade and native metadata semantics; repair-input conversion and prepared budgets; shared settings and application callbacks; capture, inspection, mounting, and constructor wiring; migrated tests and examples; and their contracts with the existing runtime. Finding 3 records the newly reproduced integration gap. Findings 1 and 2 remain unchanged.

Additional disposable external-package checks, outside the repository, passed under the race detector for both NewEngine and NewFromConfig. They combined configured default repair with continuation, appended messages, direct session selection, namespace views, frozen prepared execution, structured output, numeric-presence settings, usage/observer accounting, capacity release, and single-pass smoke diagnostics. Both default-metadata reproductions failed their expected inspection assertions, including under the race detector; those failures are the evidence for finding 3, not failures in the existing test suite.

Fresh targeted race-enabled tests passed for the public handoff and runtime contracts, configuration and use-case packages, maintained examples, capacity, generation, observation, rendering, and validation. No other integration defect was found. The prior full-suite and vet results remain applicable to unchanged implementation code. No repository implementation or test files were changed by this review.