Add application fallback profile sources

2026-08-01 12:37:01 +00:00
parent 01ca5430bd
commit 9354d2b373
7 changed files with 341 additions and 52 deletions
--- a/docs/consumers/pkg-promptkit.md
+++ b/docs/consumers/pkg-promptkit.md
@@ -40,6 +40,32 @@ validation, and default transport behavior. Source discovery, format
 validation, and profile precedence are defined by the
 [framework format reference](../formats.md).

+## Supply Embedded Application Defaults
+
+Use `WithFallbackProfileFS` when an application packages profile definitions
+that should apply unless an operator provides an ordinary configured profile
+with the same ID. For example, an application can embed its defaults while
+continuing to use `ProfileDir` for operator overrides:
+
+```go
+//go:embed profiles/*.yaml
+var applicationProfiles embed.FS
+
+engine, err := promptkit.NewEngine(promptkit.Config{
+	PromptDir:  "prompts",
+	ProfileDir: "profiles",
+},
+	promptkit.WithFallbackProfileFS(applicationProfiles, "profiles"),
+)
+```
+
+Keep application-owned profile IDs and definitions in the embedded source.
+Use the ordinary configured profile source for operator overrides. The
+[framework format reference](../formats.md#source-and-profile-precedence)
+owns the exact profile format and lookup order; the
+[`WithFallbackProfileFS` GoDoc](../../engine.go) owns its option contract and
+validation rules.
+
 ## Inspect A Prompt Before Preparation

 Use [`Engine.InspectPrompt`](../../engine.go) to check one configured prompt's
@@ -152,8 +178,8 @@ replace execution settings or the complete output contract.
 The [public value GoDoc](../../types.go) defines nil, empty, zero, replacement,
 copy, and credential behavior. The
 [framework format reference](../formats.md) defines how those request values
-interact with prompt definitions, file-backed profiles, built-ins, schemas,
-and framework defaults.
+interact with prompt definitions, file-backed and application fallback
+profiles, built-ins, schemas, and framework defaults.

 For programmatic profiles,
 [`OpenAICompatibleProfile`](../../profiles.go) converts ordinary
--- a/docs/formats.md
+++ b/docs/formats.md
@@ -233,14 +233,18 @@ default.
 Profile sources resolve matching IDs in this order:

 1. in-memory profiles supplied with `WithProfiles`;
-2. a profile file, `fs.FS`, or configured profile directory; and
-3. embedded built-in profiles.
+2. the ordinary configured source selected by a profile file, `fs.FS`, or
+   configured profile directory;
+3. application fallback profiles supplied with `WithFallbackProfileFS`; and
+4. embedded built-in profiles.

-A higher-precedence source falls back only when the profile is absent. An
-invalid matching profile is an error and does not fall back. In-memory
-`Profile` values follow the same ranges as YAML profiles. They use
-`APIKeyRequired` for request-scoped credentials instead of `api_key_env`.
-Preparation and exact profile inspection use this same source precedence.
+A profile source supplies a complete definition; definitions and their fields
+are not merged across sources. A higher-precedence source falls back only when
+the requested profile ID is absent. An invalid matching profile is an error and
+does not fall back. In-memory `Profile` values follow the same ranges as YAML
+profiles. They use `APIKeyRequired` for request-scoped credentials instead of
+`api_key_env`. Preparation and exact profile inspection use this same source
+precedence.

 ## Built-In Profile Catalog

@@ -248,8 +252,8 @@ Every built-in selects the `openrouter` backend. The engine's built-in backend
 registry supplies `https://openrouter.ai/api/v1` and the environment-variable
 name `OPENROUTER_API_KEY`, so individual profiles contain only model and
 generation settings. Built-in profile files do not repeat those connection
-values. A custom or in-memory profile with the same profile ID takes
-precedence.
+values. A configured, application fallback, or in-memory profile with the same
+profile ID takes precedence.

 | Provider | ID | Model |
 | --- | --- | --- |
--- a/docs/internal/overview.md
+++ b/docs/internal/overview.md
@@ -11,7 +11,7 @@ contributor workflow and validation.

 | Component | Implemented responsibility | References |
 | --- | --- | --- |
-| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, typed capacity errors, public error mapping, and engine-local profile-source assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
+| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, typed capacity errors, public error mapping, and engine-local profile-source assembly including application fallbacks. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
 | `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
 | `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
 | `internal/backend` | Constructs each engine's immutable registry from the built-in OpenRouter definition and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
--- a/docs/internal/sources.md
+++ b/docs/internal/sources.md
@@ -37,9 +37,9 @@ available registry belongs to the assembled engine; the runner checks
 membership during preparation and exact profile inspection.

 The root engine assembles profile repositories in precedence order: in-memory
-profiles, one ordinary configured source, then the embedded built-in catalog.
-An explicit file or `fs.FS` profile source replaces `Config.ProfileDir` within
-the ordinary configured-source category.
+profiles, one ordinary configured source, an application fallback source, then
+the embedded built-in catalog. An explicit file or `fs.FS` profile source
+replaces `Config.ProfileDir` within the ordinary configured-source category.

 Exact profile inspection performs one point-in-time lookup through those
 profile sources and checks the resolved target without reading prompt, input,
--- a/engine.go
+++ b/engine.go
@@ -98,9 +98,10 @@ type Config struct {
 	// It is required unless a WithPromptFS or WithPromptFile option supplies the
 	// prompt source.
 	PromptDir string
-	// ProfileDir is an optional directory whose profiles take precedence over
-	// embedded built-in profiles. An empty value selects only built-ins unless
-	// profile options are also supplied.
+	// ProfileDir is an optional ordinary configured source whose profiles take
+	// precedence over application fallback and embedded built-in profiles. An
+	// empty value selects the lower-precedence sources unless a profile-source
+	// option supplies the ordinary source.
 	ProfileDir string
 	// SchemaDir is the root for JSON Schema files. An empty value uses the
 	// current directory. WithSchemaFS or WithSchemaFile replaces this source.
@@ -119,12 +120,12 @@ type Config struct {
 // Option customizes engine construction.
 //
 // NewEngine applies options in argument order and ignores nil options. Within
-// each prompt-source, profile-source, in-memory-profile, schema-source,
-// model-client, and artifact-reader category, the last non-nil valid option
-// replaces earlier options in that category. WithBackend is the additive
-// exception: unique registrations accumulate, and a repeated backend ID is an
-// error rather than a replacement. An invalid option fails construction even
-// if a later option would replace it.
+// each prompt-source, ordinary-profile-source, fallback-profile-source,
+// in-memory-profile, schema-source, model-client, and artifact-reader
+// category, the last non-nil valid option replaces earlier options in that
+// category. WithBackend is the additive exception: unique registrations
+// accumulate, and a repeated backend ID is an error rather than a replacement.
+// An invalid option fails construction even if a later option would replace it.
 type Option interface {
 	apply(*engineOptions) error
 }
@@ -136,18 +137,20 @@ func (f optionFunc) apply(options *engineOptions) error {
 }

 type engineOptions struct {
-	llmClient       llm.Client
-	artifactReader  artifactadapter.Reader
-	promptDefs      promptdef.Repository
-	profiles        profile.Repository
-	memoryProfiles  profile.Repository
-	backends        []domain.Backend
-	validator       validate.Validator
-	promptSource    bool
-	profileSource   bool
-	memorySource    bool
-	validatorSource bool
-	artifactSource  bool
+	llmClient             llm.Client
+	artifactReader        artifactadapter.Reader
+	promptDefs            promptdef.Repository
+	profiles              profile.Repository
+	fallbackProfiles      profile.Repository
+	memoryProfiles        profile.Repository
+	backends              []domain.Backend
+	validator             validate.Validator
+	promptSource          bool
+	profileSource         bool
+	fallbackProfileSource bool
+	memorySource          bool
+	validatorSource       bool
+	artifactSource        bool
 }

 // WithLLMClient replaces the built-in model client used by [Engine.Run] and
@@ -224,12 +227,12 @@ func WithPromptFile(path string) Option {

 // WithProfileFS loads execution profiles from fsys under root.
 //
-// Profiles from this source overlay built-in profiles. Profile YAML must use
-// api_key_env for environment-based credentials; raw API keys are rejected.
-// fsys must be non-nil and root must be non-empty; otherwise NewEngine fails
-// with ErrInvalidConfig. This option replaces Config.ProfileDir and earlier
-// file or FS profile-source options, but remains below WithProfiles in
-// precedence.
+// Profiles from this ordinary configured source take precedence over
+// application fallback and built-in profiles. Profile YAML must use api_key_env
+// for environment-based credentials; raw API keys are rejected. fsys must be
+// non-nil and root must be non-empty; otherwise NewEngine fails with
+// ErrInvalidConfig. This option replaces Config.ProfileDir and earlier file or
+// FS profile-source options, but remains below WithProfiles in precedence.
 func WithProfileFS(fsys fs.FS, root string) Option {
 	return optionFunc(func(options *engineOptions) error {
 		if fsys == nil {
@@ -246,11 +249,12 @@ func WithProfileFS(fsys fs.FS, root string) Option {

 // WithProfileFile loads execution profiles from the single profile file at path.
 //
-// The profile overlays built-in profiles. Profile YAML must use api_key_env for
-// environment-based credentials; raw API keys are rejected. path must name an
-// existing non-directory file when NewEngine applies the option. This option
-// replaces Config.ProfileDir and earlier file or FS profile-source options,
-// but remains below WithProfiles in precedence.
+// The profile takes precedence over application fallback and built-in profiles.
+// Profile YAML must use api_key_env for environment-based credentials; raw API
+// keys are rejected. path must name an existing non-directory file when
+// NewEngine applies the option. This option replaces Config.ProfileDir and
+// earlier file or FS profile-source options, but remains below WithProfiles in
+// precedence.
 func WithProfileFile(path string) Option {
 	return optionFunc(func(options *engineOptions) error {
 		fsys, root, err := fileSource(path)
@@ -263,8 +267,41 @@ func WithProfileFile(path string) Option {
 	})
 }

+// WithFallbackProfileFS supplies application-owned fallback profile
+// definitions from fsys under root.
+//
+// Profile lookup checks, in order, profiles supplied by WithProfiles; the
+// ordinary configured source selected by WithProfileFile, WithProfileFS, or
+// Config.ProfileDir; this fallback source; and Promptkit's embedded built-in
+// profiles. Each source supplies a complete profile definition; profile fields
+// are not merged between sources. Only an absent profile ID proceeds to the
+// next source. A matching read, parse, duplicate, validation, or credential
+// format failure stops resolution.
+//
+// Files use the ordinary strict profile YAML and api_key_env credential rules.
+// Loading and validation are lazy: NewEngine validates this option's arguments
+// but does not read profile files. fsys must be non-nil and root must be
+// nonblank; otherwise NewEngine returns an error matching ErrInvalidConfig.
+// Repeating this option replaces the earlier valid fallback source.
+//
+// This option controls profile-definition lookup, not provider or generation
+// failover.
+func WithFallbackProfileFS(fsys fs.FS, root string) Option {
+	return optionFunc(func(options *engineOptions) error {
+		if fsys == nil {
+			return ErrInvalidConfig
+		}
+		if strings.TrimSpace(root) == "" {
+			return ErrInvalidConfig
+		}
+		options.fallbackProfiles = profile.NewFSRepository(fsys, root)
+		options.fallbackProfileSource = true
+		return nil
+	})
+}
+
 // WithProfiles configures in-memory profiles that take precedence over
-// configured profile files and built-in profiles.
+// ordinary configured, application fallback, and built-in profiles.
 //
 // NewEngine validates and copies every profile. IDs must be unique within one
 // call. An invalid profile, duplicate ID, or unsupported ExtraParams value
@@ -406,6 +443,10 @@ func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
 func newProfileRepository(profileDir string, options engineOptions) profile.Repository {
 	repository := builtin.NewRepository()

+	if options.fallbackProfileSource {
+		repository = profile.NewOverlayRepository(options.fallbackProfiles, repository)
+	}
+
 	if options.profileSource {
 		repository = profile.NewOverlayRepository(options.profiles, repository)
 	} else if strings.TrimSpace(profileDir) != "" {
@@ -491,10 +532,10 @@ func (e *Engine) InspectPrompt(
 //
 // InspectProfile trims surrounding whitespace from profileID and looks up the
 // resulting nonblank ID exactly and case-sensitively through the engine's
-// ordinary in-memory, configured-source, and built-in profile precedence. It
-// applies the framework timeout baseline, selected backend, and then selected
-// profile to EffectiveModelParams without a request override. BackendID is
-// empty for an endpoint-only profile.
+// in-memory, ordinary configured-source, application fallback, and built-in
+// profile precedence. It applies the framework timeout baseline, selected
+// backend, and then selected profile to EffectiveModelParams without a request
+// override. BackendID is empty for an endpoint-only profile.
 //
 // APIKeyEnv in the returned target is an environment-variable name, never its
 // value. APIKeyRequired instead reports a direct credential requirement and is
--- a/engine_test.go
+++ b/engine_test.go
@@ -2301,6 +2301,8 @@ func TestSourceOptionsRejectInvalidInputs(t *testing.T) {
 		{name: "profile fs nil", opt: promptkit.WithProfileFS(nil, "profiles")},
 		{name: "profile fs empty root", opt: promptkit.WithProfileFS(fstest.MapFS{}, "")},
 		{name: "profile file empty", opt: promptkit.WithProfileFile("")},
+		{name: "fallback profile fs nil", opt: promptkit.WithFallbackProfileFS(nil, "profiles")},
+		{name: "fallback profile fs empty root", opt: promptkit.WithFallbackProfileFS(fstest.MapFS{}, "")},
 		{name: "schema fs nil", opt: promptkit.WithSchemaFS(nil, "schemas")},
 		{name: "schema fs empty root", opt: promptkit.WithSchemaFS(fstest.MapFS{}, "")},
 		{name: "schema file empty", opt: promptkit.WithSchemaFile("")},
--- a/public_contract_test.go
+++ b/public_contract_test.go
@@ -973,6 +973,24 @@ func TestRepeatedOptionsUseLastValueInEachCategory(t *testing.T) {
 		}
 	})

+	t.Run("fallback profile source", func(t *testing.T) {
+		engine, err := promptkit.NewEngine(promptkit.Config{},
+			promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
+			promptkit.WithFallbackProfileFS(contractProfileFS("profile", "first-model"), "."),
+			promptkit.WithFallbackProfileFS(contractProfileFS("profile", "second-model"), "."),
+		)
+		if err != nil {
+			t.Fatalf("construct engine: %v", err)
+		}
+		prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
+		if err != nil {
+			t.Fatalf("prepare from last fallback profile source: %v", err)
+		}
+		if prepared.EffectiveModelParams.Model != "second-model" {
+			t.Fatalf("expected last fallback profile source, got %q", prepared.EffectiveModelParams.Model)
+		}
+	})
+
 	t.Run("in-memory profiles", func(t *testing.T) {
 		first := profile
 		first.Model = "first-model"
@@ -1057,6 +1075,204 @@ func TestRepeatedOptionsUseLastValueInEachCategory(t *testing.T) {
 	})
 }

+func TestFallbackProfileSourcePrecedence(t *testing.T) {
+	const profileID = "application-profile"
+
+	prepareModel := func(t *testing.T, engine *promptkit.Engine, promptID string) string {
+		t.Helper()
+		prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: promptID})
+		if err != nil {
+			t.Fatalf("prepare: %v", err)
+		}
+		return prepared.EffectiveModelParams.Model
+	}
+
+	t.Run("in-memory profiles override ordinary and fallback profiles", func(t *testing.T) {
+		engine, err := promptkit.NewEngine(promptkit.Config{},
+			promptkit.WithPromptFS(contractPromptFS("prompt", profileID, "message"), "."),
+			promptkit.WithFallbackProfileFS(contractProfileFS(profileID, "fallback-model"), "."),
+			promptkit.WithProfileFS(contractProfileFS(profileID, "ordinary-model"), "."),
+			promptkit.WithProfiles(promptkit.Profile{ID: profileID, Endpoint: "http://example.test/v1", Model: "memory-model"}),
+		)
+		if err != nil {
+			t.Fatalf("construct engine: %v", err)
+		}
+		if model := prepareModel(t, engine, "prompt"); model != "memory-model" {
+			t.Fatalf("expected in-memory profile, got %q", model)
+		}
+	})
+
+	t.Run("ordinary filesystem source overrides fallback profile", func(t *testing.T) {
+		engine, err := promptkit.NewEngine(promptkit.Config{},
+			promptkit.WithPromptFS(contractPromptFS("prompt", profileID, "message"), "."),
+			promptkit.WithFallbackProfileFS(contractProfileFS(profileID, "fallback-model"), "."),
+			promptkit.WithProfileFS(contractProfileFS(profileID, "ordinary-model"), "."),
+		)
+		if err != nil {
+			t.Fatalf("construct engine: %v", err)
+		}
+		if model := prepareModel(t, engine, "prompt"); model != "ordinary-model" {
+			t.Fatalf("expected ordinary profile, got %q", model)
+		}
+	})
+
+	t.Run("configured directory overrides fallback profile", func(t *testing.T) {
+		profileDir := t.TempDir()
+		writePublicProfileFile(t, profileDir, profileID, "http://example.test/v1", "directory-model")
+		engine, err := promptkit.NewEngine(promptkit.Config{ProfileDir: profileDir},
+			promptkit.WithPromptFS(contractPromptFS("prompt", profileID, "message"), "."),
+			promptkit.WithFallbackProfileFS(contractProfileFS(profileID, "fallback-model"), "."),
+		)
+		if err != nil {
+			t.Fatalf("construct engine: %v", err)
+		}
+		if model := prepareModel(t, engine, "prompt"); model != "directory-model" {
+			t.Fatalf("expected configured directory profile, got %q", model)
+		}
+	})
+
+	t.Run("fallback profile overrides built-in profile", func(t *testing.T) {
+		engine, err := promptkit.NewEngine(promptkit.Config{},
+			promptkit.WithPromptFS(contractPromptFS("prompt", "mistral-small-3", "message"), "."),
+			promptkit.WithFallbackProfileFS(contractProfileFS("mistral-small-3", "fallback-model"), "."),
+		)
+		if err != nil {
+			t.Fatalf("construct engine: %v", err)
+		}
+		if model := prepareModel(t, engine, "prompt"); model != "fallback-model" {
+			t.Fatalf("expected fallback profile, got %q", model)
+		}
+	})
+
+	t.Run("missing fallback profile uses built-in profile", func(t *testing.T) {
+		t.Setenv("OPENROUTER_API_KEY", "test-key")
+		engine, err := promptkit.NewEngine(promptkit.Config{},
+			promptkit.WithPromptFS(contractPromptFS("prompt", "mistral-small-3", "message"), "."),
+			promptkit.WithFallbackProfileFS(contractProfileFS(profileID, "fallback-model"), "."),
+		)
+		if err != nil {
+			t.Fatalf("construct engine: %v", err)
+		}
+		if model := prepareModel(t, engine, "prompt"); model != "mistralai/mistral-small-3.2-24b-instruct" {
+			t.Fatalf("expected built-in profile, got %q", model)
+		}
+	})
+}
+
+func TestFallbackProfileSourcePreservesLazyLoadingAndErrors(t *testing.T) {
+	const profileID = "application-profile"
+
+	t.Run("construction defers malformed fallback profiles", func(t *testing.T) {
+		_, err := promptkit.NewEngine(promptkit.Config{},
+			promptkit.WithPromptFS(fstest.MapFS{}, "."),
+			promptkit.WithFallbackProfileFS(fstest.MapFS{
+				"broken.yaml": &fstest.MapFile{Data: []byte("id: broken\nunknown: value\n")},
+			}, "."),
+		)
+		if err != nil {
+			t.Fatalf("construct engine with malformed fallback profile: %v", err)
+		}
+	})
+
+	t.Run("unrelated malformed fallback profile does not block matching definition", func(t *testing.T) {
+		engine, err := promptkit.NewEngine(promptkit.Config{},
+			promptkit.WithPromptFS(contractPromptFS("prompt", profileID, "message"), "."),
+			promptkit.WithFallbackProfileFS(fstest.MapFS{
+				"broken.yaml": &fstest.MapFile{Data: []byte("id: unrelated\nunknown: value\n")},
+				"valid.yaml":  &fstest.MapFile{Data: []byte("id: application-profile\nendpoint: http://example.test/v1\nmodel: fallback-model\n")},
+			}, "."),
+		)
+		if err != nil {
+			t.Fatalf("construct engine: %v", err)
+		}
+		prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
+		if err != nil {
+			t.Fatalf("prepare from valid fallback profile: %v", err)
+		}
+		if prepared.EffectiveModelParams.Model != "fallback-model" {
+			t.Fatalf("unexpected fallback profile model: %q", prepared.EffectiveModelParams.Model)
+		}
+	})
+
+	t.Run("matching malformed fallback profile does not reach built-in profile", func(t *testing.T) {
+		engine, err := promptkit.NewEngine(promptkit.Config{},
+			promptkit.WithPromptFS(contractPromptFS("prompt", "mistral-small-3", "message"), "."),
+			promptkit.WithFallbackProfileFS(fstest.MapFS{
+				"mistral-small-3.yaml": &fstest.MapFile{Data: []byte("id: mistral-small-3\nendpoint: http://example.test/v1\nmodel: fallback-model\nunknown: value\n")},
+			}, "."),
+		)
+		if err != nil {
+			t.Fatalf("construct engine: %v", err)
+		}
+		if _, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"}); !errors.Is(err, promptkit.ErrProfileLoad) {
+			t.Fatalf("expected ErrProfileLoad, got %v", err)
+		}
+	})
+
+	t.Run("matching malformed ordinary profile does not reach fallback profile", func(t *testing.T) {
+		engine, err := promptkit.NewEngine(promptkit.Config{},
+			promptkit.WithPromptFS(contractPromptFS("prompt", profileID, "message"), "."),
+			promptkit.WithProfileFS(fstest.MapFS{
+				"application-profile.yaml": &fstest.MapFile{Data: []byte("id: application-profile\nendpoint: http://example.test/v1\nmodel: ordinary-model\nunknown: value\n")},
+			}, "."),
+			promptkit.WithFallbackProfileFS(contractProfileFS(profileID, "fallback-model"), "."),
+		)
+		if err != nil {
+			t.Fatalf("construct engine: %v", err)
+		}
+		if _, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"}); !errors.Is(err, promptkit.ErrProfileLoad) {
+			t.Fatalf("expected ErrProfileLoad, got %v", err)
+		}
+	})
+}
+
+func TestFallbackProfileSourceWorksAcrossWorkflows(t *testing.T) {
+	const profileID = "application-profile"
+	client := &fakeLLMClient{response: &promptkit.GenerateResponse{Content: "ok"}}
+	engine, err := promptkit.NewEngine(promptkit.Config{},
+		promptkit.WithPromptFS(contractPromptFS("prompt", profileID, "message"), "."),
+		promptkit.WithFallbackProfileFS(contractProfileFS(profileID, "fallback-model"), "."),
+		promptkit.WithLLMClient(client),
+	)
+	if err != nil {
+		t.Fatalf("construct engine: %v", err)
+	}
+
+	inspection, err := engine.InspectProfile(context.Background(), profileID)
+	if err != nil {
+		t.Fatalf("inspect fallback profile: %v", err)
+	}
+	prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
+	if err != nil {
+		t.Fatalf("prepare fallback profile: %v", err)
+	}
+	preparedExecution, err := engine.PrepareExecution(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
+	if err != nil {
+		t.Fatalf("prepare execution with fallback profile: %v", err)
+	}
+	preparedDetails := preparedExecution.Details()
+	preparedResult, err := engine.RunPrepared(context.Background(), preparedExecution)
+	if err != nil {
+		t.Fatalf("run prepared fallback profile: %v", err)
+	}
+	runResult, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
+	if err != nil {
+		t.Fatalf("run fallback profile: %v", err)
+	}
+
+	for name, model := range map[string]string{
+		"inspection":         inspection.EffectiveModelParams.Model,
+		"preparation":        prepared.EffectiveModelParams.Model,
+		"prepared execution": preparedDetails.EffectiveModelParams.Model,
+		"prepared result":    preparedResult.EffectiveModelParams.Model,
+		"run result":         runResult.EffectiveModelParams.Model,
+	} {
+		if model != "fallback-model" {
+			t.Fatalf("%s model=%q, want fallback-model", name, model)
+		}
+	}
+}
+
 func TestEngineSupportsConcurrentPrepareAndRun(t *testing.T) {
 	engine, err := promptkit.NewEngine(promptkit.Config{},
 		promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),