promptkit/docs/roadmap/implementation.md

Profile Inheritance Implementation Plan

Purpose

Implement the profile-inheritance feature defined by the profile inheritance roadmap. This plan is the decision-complete execution sequence for a coding agent. Follow the repository architecture, testing, and documentation policies and the task-specific reading guide in docs/development.md throughout the work.

Target Outcome

A downstream consumer can define a YAML or in-memory profile whose base_profile/BaseProfileID names another profile, including a Promptkit built-in. Promptkit resolves the complete one-parent chain through the assembled profile catalog, merges root-to-leaf profile settings, resolves the final backend, and then applies request overrides exactly as it does for a standalone profile.

The selected leaf ID remains the public profile identity. Ordinary operations resolve sources afresh, while prepared execution freezes the resolved target. Missing dependencies, cycles, excessive depth, and incomplete resolved profiles fail as profile-load errors without changing the distinct not-found contract for an absent directly selected profile.

Fixed Decisions

• YAML uses base_profile; the public Go fields are Profile.BaseProfileID and OpenAICompatibleProfileConfig.BaseProfileID.
• One profile may name one base. Bases may form a linear chain of at most 32 profiles, including the selected leaf.
• The resolver performs every lookup through the complete raw composite repository. Each base therefore observes the ordinary in-memory, configured, fallback, and built-in precedence.
• The resolving repository is stateless and does not cache results across calls. It is applied once, outside the fully assembled raw overlay chain.
• Profile definitions with a base may omit model and both connection fields. Standalone definitions retain their current completeness requirements.
• Root-to-leaf merge behavior follows existing profile presence semantics: nonblank strings and nonzero numbers override; blank strings and zero numbers inherit; a nonempty extra_params map replaces the complete inherited map; and APIKeyRequired is a sticky logical OR.
• id always comes from the selected leaf. base_profile is cleared from the resolved profile before it reaches the use-case layer.
• Backend and endpoint remain independent fields. A child backend replaces the inherited backend ID without implicitly clearing an inherited endpoint; a child endpoint replaces only the endpoint.
• This work introduces no profile-level clearing syntax, presence pointers, multiple inheritance, source-qualified references, public resolution graph, new public error sentinel, or provider-wire change.
• A missing directly selected profile continues to expose ErrProfileNotFound. Once the leaf exists, a missing base is an invalid selected profile and must expose ErrProfileLoad without also matching ErrProfileNotFound.
• Existing raw-source failures and context cancellation remain discoverable through errors.Is where they are currently preserved. The missing-base branch is the one deliberate exception: its diagnostic names the missing ID and chain but does not wrap the internal not-found sentinel.
• No release notes, version tags, commits, or pushes are part of these stages.

Execution Rules

• Complete the stages in numerical order. Each stage is sized for one gpt-5.6-terra implementation prompt and must finish its focused verification before the next begins.
• At the start of every stage, inspect the working tree and preserve unrelated changes. The feature roadmap is intentional planning state and must remain.
• Treat intermediate stages as an unreleased partial implementation. Do not describe the feature in durable current-state prose until Stage 4.
• Keep inheritance logic in internal/profile. Do not add recursive profile lookup, chain merging, or source-precedence logic to internal/usecase or the root public facade.
• Use real in-memory repositories and synthetic fs.FS sources where practical. Tests must remain deterministic, offline, parallel-safe, and independent of provider credentials.
• Update existing test owners when their validation contract changes; do not retain contradictory expectations or duplicate the full inheritance matrix at higher layers.
• Use apply_patch for edits, format every changed Go file, and review the complete stage diff before proceeding.

Stage 1: Add Profile Definition Surfaces And Separate Local Validation

Objective

Represent a base-profile reference in every supported profile input while preserving strict decoding, standalone-profile compatibility, and eager local validation. At the end of this stage repositories can publish locally valid derived definitions, but the engine does not yet resolve them.

Implementation

1. Add BaseProfileID string to internal/domain.ExecutionProfile with the YAML tag base_profile. Keep it out of execution targets and provider request types.
2. Add BaseProfileID string to the public Profile immediately after ID. Its GoDoc must state that:
   • WithProfiles trims the value;
   • a nonblank value permits required target fields to be inherited;
   • reference existence and resolved completeness are checked only when the profile is selected or inspected; and
   • blank means the profile remains standalone.
3. Add the matching field to OpenAICompatibleProfileConfig, document that it maps to Profile.BaseProfileID, and copy it in OpenAICompatibleProfile. Preserve the constructor's shallow-copy and deferred deep-validation behavior for ExtraParams.
4. Map and trim Profile.BaseProfileID in toDomainProfile. Ensure the memory repository continues to return caller-independent ExtraParams and does not expose shared mutable profile state.
5. Centralize the duplicated file-profile and in-memory-profile normalization rules in internal/profile rather than adding a third validation path:
   • provide an internal-package exported helper named NormalizeAndValidateDefinition so the root facade can validate a converted in-memory domain.ExecutionProfile without owning profile parsing rules;
   • trim ID, BaseProfileID, BackendID, and Endpoint in that helper;
   • require a nonblank ID and validate a supplied endpoint and all execution setting bounds for every definition;
   • when BaseProfileID is blank, retain the existing requirements for a nonblank model and at least one nonblank backend or endpoint;
   • when BaseProfileID is nonblank, allow model, backend, and endpoint to be omitted so they can be inherited; and
   • do not add unrelated normalization, environment-name policy, clearing behavior, or backend registry lookup.
6. Replace normalizeAndValidateProfile in internal/profile/filesystem_repository.go and normalizeAndValidatePublicProfile in profiles.go with the shared helper. Preserve each caller's current public/internal error wrapping, selected-file path diagnostic, raw-key prohibition, strict single-document rule, and duplicate-ID behavior. Remove imports and helpers made obsolete by the centralization.
7. Keep the raw repositories raw: file, fs.FS, memory, overlay, and built-in repositories return BaseProfileID but do not recursively load it in this stage.

Tests

Update the narrow existing owners:

1. In internal/profile/repository_test.go, extend the filesystem/fs.FS source-parity coverage to prove that:
   • strict YAML accepts base_profile, trims it, and publishes it;
   • an alias containing only id and base_profile is locally valid;
   • a derived profile still rejects an invalid supplied endpoint, negative or out-of-range setting, malformed extra_params, raw api_key, duplicate ID, and extra YAML document through their existing identities; and
   • a standalone profile missing connection fields or model remains invalid. Modify existing missing-field expectations rather than duplicating their entire matrices.
2. In root package tests, extend the existing WithProfiles validation owner to show that an incomplete profile with a nonblank BaseProfileID is accepted by NewEngine, while the equivalent standalone profile retains its current ErrInvalidConfig result. Do not attempt to select the derived profile until the resolver is implemented.
3. Extend TestOpenAICompatibleProfileMapsEveryField so BaseProfileID is included in the constructor's exact field mapping.
4. Preserve existing tests for defensive copying, duplicates, standalone profiles, overlay precedence, built-ins, and strict decoding.

Verification

gofmt -w internal/domain/domain.go internal/profile/*.go profiles.go types.go
go test ./internal/profile
go test ./internal/profile/builtin
go test ./... -run 'Profile|OpenAICompatibleProfile'
go test ./...
git diff --check

Stage 1 is complete when all supported definition surfaces carry the normalized base ID, locally valid aliases can be stored, standalone validation is unchanged, and no repository performs inheritance resolution yet.

Status: Complete.

Stage 2: Implement The Stateless Resolving Repository

Objective

Add the internal component that resolves one complete profile chain, merges it deterministically, enforces safety bounds, and returns one ordinary resolved profile to its caller.

Implementation

1. Add internal/profile/resolving_repository.go with:
   • an unexported resolvingRepository containing one underlying raw Repository;
   • NewResolvingRepository(source Repository) Repository; and
   • an unexported constant setting the maximum chain length to 32, counting the selected leaf.
2. Make the wrapper safe for concurrent calls by keeping the source immutable and all chain, visited-ID, and merge state local to GetProfile. Do not add a cache, mutex, process-global registry, or source snapshot.
3. Resolve one request using this exact flow:
   • reject a nil underlying repository or blank requested ID as ErrInvalidProfile;
   • trim and look up the directly requested ID through the underlying raw repository;
   • if that first lookup fails, return its error unchanged so direct ErrProfileNotFound behavior remains intact;
   • reject a nil profile returned without an error as ErrInvalidProfile;
   • follow each normalized nonblank BaseProfileID through the same underlying repository, recording IDs and profiles from leaf toward root;
   • check ctx.Err() between lookups;
   • detect a repeated ID before loading it again and return an ErrInvalidProfile cycle diagnostic containing the a -> b -> a chain;
   • reject a reference that would make the chain exceed 32 profiles with an ErrInvalidProfile diagnostic containing the traversed chain and next ID;
   • when a base lookup returns ErrProfileNotFound, return a new ErrInvalidProfile error naming the missing base and chain without wrapping ErrProfileNotFound; and
   • for any other base error, add ErrInvalidProfile, base ID, and chain context while preserving the source error and cancellation identity with wrapping.
4. Merge the collected definitions from root to leaf into a new value. Use one focused helper with these rules:
   • retain the selected leaf ID and clear final BaseProfileID;
   • replace BackendID, Endpoint, Model, ServiceTier, ReasoningEffort, and APIKeyEnv only for a nonblank child value;
   • replace Temperature, MaxTokens, TopP, and TimeoutSeconds only for a nonzero child value;
   • set APIKeyRequired to the logical OR of every definition in the chain;
   • replace ExtraParams only when the child map is nonempty; otherwise inherit the current map; and
   • defensively deep-copy every retained or replacement ExtraParams map by using the existing bounded internal/jsonvalue owner. Never mutate a repository-returned profile or map. Treat an unexpected copy failure as ErrInvalidProfile, preserve the copy error, and include the selected chain in the diagnostic.
5. Add an unexported final validator in internal/profile that requires a nonblank model and at least one backend or endpoint on the merged profile, and reuses the common endpoint and execution-setting validation. Wrap final failures with ErrInvalidProfile and the selected chain. The resolver must not perform backend registry membership lookup; that remains in internal/usecase after profile resolution.
6. Return a caller-owned resolved profile on every successful call. A standalone profile passes through the same final validation and copy path, ensuring the wrapper does not create separate standalone and inherited execution contracts.

Tests

Create internal/profile/resolving_repository_test.go and use small synthetic repositories plus the real overlay repository where it materially protects composition:

1. Prove a three-level root/middle/leaf chain merges representative values for every rule: inherited values, string and numeric replacements, independent backend/endpoint behavior, sticky APIKeyRequired, complete-map ExtraParams replacement, leaf identity, and cleared resolution metadata. Use one well-structured table or fixture rather than one test per field.
2. Prove an empty child ExtraParams inherits the base map, a nonempty child map replaces rather than key-merges it, and mutating the returned nested values cannot mutate the raw repository or a later result.
3. Prove the same composite repository resolves a base from a lower-precedence source and honors a higher-precedence shadow of that base ID. Do not add source-specific resolution to the wrapper.
4. Cover the safety/error matrix through the resolving repository boundary:
   • directly selected missing ID preserves ErrProfileNotFound;
   • missing base matches ErrInvalidProfile and does not match ErrProfileNotFound;
   • direct and indirect cycles are rejected with useful chains;
   • exactly 32 profiles are accepted and a 33rd is rejected;
   • nil repository results and incomplete final profiles are invalid;
   • non-not-found base errors retain their original identity; and
   • cancellation before or during traversal remains discoverable and stops later lookups.
5. Prove freshness and concurrency without timing sleeps: change a controlled raw repository between sequential calls and observe the new base value, then issue concurrent independent resolutions and verify results do not share mutable maps. Use synchronization owned by the fake; do not race on a plain map.
6. Preserve the existing raw repository and overlay tests unchanged except where Stage 1 intentionally changed local completeness rules.

Verification

gofmt -w internal/profile/*.go
go test ./internal/profile -run 'Resolving|Inheritance|Profile'
go test -race ./internal/profile
go test ./...
git diff --check

Stage 2 is complete when a standalone internal wrapper can safely and freshly resolve every accepted chain into one complete, copied profile with the fixed merge and error contracts.

Status: Complete.

Stage 3: Assemble Inheritance And Prove Public Workflows

Objective

Place the resolver at the correct engine boundary and prove that YAML, in-memory, built-in, inspection, ordinary preparation, runtime overrides, and prepared execution compose as intended through the public facade.

Implementation

1. Update newProfileRepository in engine.go to:
   • assemble the raw repository in its current precedence order—built-ins, fallback, configured source or Config.ProfileDir, then in-memory;
   • wrap that complete raw repository exactly once with profile.NewResolvingRepository; and
   • return the wrapper to usecase.NewRunner. Do not wrap individual sources, because that would prevent a higher-level child from referencing a lower-level base and would change shadowing rules.
2. Keep internal/usecase.resolveProfileSelection structurally unchanged. It should continue to request one profile, preserve the requested normalized ID as selection identity, resolve the final backend once, and then apply the ordinary backend/profile/request precedence. Do not pass BaseProfileID into ExecutionTarget, PreparedRun, RunResult, injected LLMClient requests, stable JSON, hashes, or provider payloads.
3. Confirm APIKeyRequired behavior after inheritance without introducing a special credential path: a true value in any ancestor reaches the existing profile-to-target merge, clears inherited profile/backend APIKeyEnv, and still permits a direct request key or explicit request environment override.
4. Preserve the current error mapper. The resolver's use of ErrInvalidProfile must make a missing/invalid base cross the use-case and root boundaries as ErrProfileLoad; no new root sentinel or special-case string matching is permitted.
5. Do not modify built-in profile assets merely to demonstrate inheritance. Built-ins serve as valid bases without changing their own definitions.

Tests

Use the smallest public tests that protect assembled behavior without repeating the resolver's matrix:

1. Add one external-package contract workflow that defines weather-light as an in-memory alias/refinement of the built-in deepseek-4-flash. Compare its inspected target with an inspection of the base rather than duplicating every built-in literal. Assert:
   • the child inspection reports weather-light;
   • backend and model are inherited;
   • explicit child reasoning and timeout values win;
   • Prepare reports the child as SelectedProfileID; and
   • a per-request reasoning or timeout override still wins over the child.
2. Exercise YAML through WithProfileFS with a minimal alias of a built-in and prove it resolves through inspection or preparation. Parser tests own YAML edge cases, so do not repeat them here.
3. Add a focused public error-identity table:
   • a directly selected absent profile still matches ErrProfileNotFound and not ErrProfileLoad;
   • an existing child with an absent base matches ErrProfileLoad and not ErrProfileNotFound; and
   • one representative cycle matches ErrProfileLoad and includes its IDs. Reuse or extend the existing profile-inspection public error owner where that keeps the suite lean.
4. Prove fresh ordinary resolution and frozen prepared execution with one controlled mutable fs.FS workflow:
   • prepare an execution while its base supplies model A;
   • change the base source to model B after preparation;
   • run the prepared handle through an injected deterministic client and observe model A; and
   • perform a new inspection or ordinary preparation and observe model B. Keep all mutation sequential or synchronized and make no provider call.
5. Add only a narrow credential assertion if existing use-case tests do not already prove the final inherited APIKeyRequired target behavior. The resolver test owns sticky merging; current credential tests own availability, precedence, redaction, and prepared rechecking.
6. Run all existing source-precedence, fallback, malformed-profile, built-in, public JSON, runtime-override, credential, and prepared-execution tests. Fix production behavior rather than weakening those contracts.

Verification

gofmt -w engine.go engine_test.go public_contract_test.go prepared_execution_contract_test.go profiles.go types.go
go test ./... -run 'Profile|Inheritance|PreparedExecution|RuntimeOverride'
go test ./internal/profile
go test ./internal/usecase
go test -race ./...
go vet ./...
go build ./...
git diff --check

Stage 3 is complete when the engine resolves aliases and refinements across all supported sources, public identities and errors match the roadmap, ordinary operations remain fresh, and prepared execution remains frozen.

Status: Complete.

Stage 4: Publish Canonical Documentation And Complete Validation

Objective

Align every canonical contract owner with the implemented feature, remove temporary ambiguity, and validate the release candidate comprehensively.

Documentation

1. Finalize exact GoDoc for Profile.BaseProfileID and OpenAICompatibleProfileConfig.BaseProfileID, plus nearby Profile, OpenAICompatibleProfile, and WithProfiles text whose standalone completeness or validation-timing statements changed. GoDoc owns the public fields and construction contract; do not reproduce internal traversal.
2. Update docs/formats.md as the exact YAML contract owner:
   • add base_profile to the field table and example;
   • distinguish locally complete standalone profiles from derived aliases;
   • document root-to-leaf override rules, complete-map extra_params replacement, sticky APIKeyRequired, independent backend/endpoint behavior, the 32-profile limit, and lack of clearing syntax;
   • document cross-source lookup precedence and shadowing; and
   • distinguish a missing selected profile from an invalid selected chain.
3. Update docs/internal/sources.md to describe the raw composite catalog, the one outer resolving repository, fresh per-operation traversal, cycle/depth protection, final validation, and prepared-state freezing. Link to the format reference for exact consumer syntax and merge rules.
4. Update the internal/profile entry in docs/internal/overview.md because the concrete package now resolves inherited definitions in addition to loading, validating, and overlaying raw sources. Do not change architecture policy because package ownership and dependency direction are unchanged.
5. Add one concise task-oriented example to docs/consumers/pkg-promptkit.md showing an application-owned alias of a built-in and a selected child override. Link to the format reference and public GoDoc for details rather than duplicating the complete contract.
6. Do not update the README, examples, release documents, release procedure, future/deferred catalogs, or built-in catalog unless the implementation has made a concrete current-state statement there inaccurate. Do not copy the roadmap into durable documentation.

Final Validation

Run the complete maintainer workflow from docs/development.md#maintainer-validation:

go test ./...
go test -race ./...
go vet ./...
go build ./...
go run ./examples/go-library/prepare
go run ./examples/go-library/run

Then run the documented tracked-Go formatting check, repository-relative Markdown link checker, workspace/vendor/module-replacement checks, unstaged and staged whitespace checks, ignored-file check, credential scan, and git diff --check. Inspect the complete diff and confirm:

• only base_profile/BaseProfileID were added to consumer configuration;
• raw repositories remain source-local and the resolver wraps the full catalog once;
• chain resolution is bounded, cycle-safe, fresh, and concurrency-safe;
• merge behavior covers every profile field without mutating source values;
• direct missing and dependent missing profiles preserve distinct public error identities;
• selected IDs, backend resolution, runtime overrides, credentials, and prepared freezing retain their stated ownership;
• resolution metadata never reaches stable execution JSON, hashes, injected generation targets, or provider payloads;
• tests remain lean, deterministic, offline, and owned at the narrowest stable boundary; and
• durable documentation describes only implemented behavior with one canonical owner per exact contract.

Stage 4 is complete when all focused and repository-wide checks pass and the implementation, GoDoc, format reference, internal documentation, and consumer guidance agree.

Status: Complete.

Open Questions

None. Naming, supported sources, lookup precedence, merge behavior, connection semantics, credential inheritance, validation timing, chain bounds, error identity, freshness, prepared freezing, package ownership, test boundaries, and documentation ownership are fixed by the feature roadmap and this plan.