Promptkit Feature Request: Application Fallback Profiles

Status: Proposed upstream capability.

Purpose

Promptkit should allow a consuming application to supply an embedded fallback profile source that sits below operator-configured profiles and above Promptkit's own built-in profile catalog.

This capability would let an application publish stable, domain-specific profile IDs with useful defaults while preserving Promptkit's existing operator-override behavior. The capability must remain application-neutral; Promptkit should provide the source layer but should not own downstream profile names, model assignments, or configuration policy.

Downstream Use Case

Weatherreporter wants to embed profiles such as weather-light, weather-balanced, and weather-deep. Report prompts would select those logical profiles instead of naming provider- or model-specific Promptkit profiles directly.

An installation could then place a profile with the same ID in its configured profile directory. For example, a local weather-light definition could point to an OpenAI-compatible endpoint on the deployment network. When no operator definition exists, Weatherreporter's embedded definition would keep the application usable without additional profile files.

This pattern is useful beyond Weatherreporter. Any Promptkit consumer may want application-owned execution tiers or workload-specific defaults without adding domain-specific profiles to Promptkit's general built-in catalog.

Current Constraint

Promptkit currently resolves matching profile IDs in this order:

in-memory profiles supplied through WithProfiles;
one configured profile file, fs.FS, or directory source; and
Promptkit's embedded built-in profiles.

These layers do not express the desired application-default relationship:

WithProfiles has higher precedence than the configured source, so it would prevent an operator file from overriding an application profile with the same ID.
WithProfileFS can hold embedded application assets, but it occupies the configured-source layer and therefore replaces rather than sits beneath a configured profile directory or file.
adding downstream profile IDs to Promptkit's built-in catalog would make the library own application-specific policy.

A downstream application could build its own filesystem overlay, but that would duplicate Promptkit's profile discovery, error, and precedence behavior at the consumer boundary.

Requested Capability

Add one optional application fallback profile source to engine construction. When present, matching profile IDs should resolve in this order:

in-memory profiles supplied through WithProfiles;
the ordinary configured profile source selected through a profile option or Config.ProfileDir;
the application fallback profile source; and
Promptkit's embedded built-in profiles.

When no application fallback is configured, existing source precedence and behavior must remain unchanged.

The minimum useful public surface is an fs.FS-backed option because consumers can embed YAML profile assets. A possible API shape is:

promptkit.WithFallbackProfileFS(profileFS, ".")

The name is illustrative rather than prescriptive. A companion option for validated Profile values could be added if Promptkit maintainers find it generally useful, but it is not required for the Weatherreporter use case.

Required Semantics

A higher-precedence source falls through only when the requested profile ID is absent.
A malformed, unreadable, duplicate, ambiguous, or otherwise invalid matching profile is an error and must not silently fall through.
The fallback source uses the existing strict profile YAML format and profile validation rules.
Profile values are selected as a whole. This feature does not merge, inherit, or partially overlay profile definitions.
InspectProfile, Prepare, prepared execution, and ordinary execution use the same profile-source precedence.
An explicit request profile continues to take precedence over a prompt's default_profile; this request concerns definition lookup after the profile ID has been selected.
Repeated fallback-source options should follow Promptkit's documented same-category option convention, normally with the last value replacing the earlier value.
A canceled lookup, invalid fallback asset, or unknown resolved backend should continue to cross the public facade through Promptkit's existing public error identities.
Exact profile inspection must remain side-effect free and must not contact a model provider.

Application And Library Boundaries

Promptkit should own:

the additional repository layer;
deterministic lookup and fallthrough behavior;
validation of the supplied source through the existing profile contract;
consistent use of the layer across inspection and execution; and
public documentation and tests for the added precedence rule.

The consuming application should continue to own:

whether it supplies fallback profiles;
the profile IDs and their domain meaning;
embedded profile contents and model choices;
application configuration and override policy;
report- or workload-to-profile assignment; and
credential checks and operator-facing errors beyond Promptkit's public contract.

Non-Goals

This request does not ask Promptkit to add:

Weatherreporter-specific profile IDs to its built-in catalog;
profile inheritance, aliases, or field-level merging;
automatic endpoint discovery or availability probing;
provider failover or fallback from a failed selected profile;
per-request model benchmarking or tier selection;
application configuration discovery; or
eager validation of every profile in every source.

Compatibility

The feature can be additive. Engines that do not configure an application fallback source should retain their current public behavior and precedence. Existing uses of WithProfiles, WithProfileFile, WithProfileFS, and Config.ProfileDir should not change meaning.

The application fallback is deliberately lower precedence than every existing consumer-configured source. This preserves the established expectation that a custom profile definition can override a packaged default with the same ID.

Acceptance Criteria

The capability is sufficient for downstream adoption when Promptkit can demonstrate that:

a fallback-only profile can be inspected and used for preparation and execution;
a configured directory, file, or fs.FS profile with the same ID overrides the fallback profile;
an absent configured profile falls through to the application fallback;
an invalid configured match fails instead of falling through;
an absent application fallback profile continues to resolve from Promptkit's built-in catalog;
WithProfiles retains highest precedence;
behavior is identical across inspection, preparation, and execution; and
omitting the new option preserves existing tests and public contracts.

6.9 KiB Raw Blame History