diff --git a/docs/adr/0010-workload-oriented-llm-profile-defaults.md b/docs/adr/0010-workload-oriented-llm-profile-defaults.md new file mode 100644 index 0000000..cd6fe95 --- /dev/null +++ b/docs/adr/0010-workload-oriented-llm-profile-defaults.md @@ -0,0 +1,49 @@ +# ADR-0010: Use workload-oriented LLM profile defaults + +**Status:** Accepted +**Date:** 2026-08-03 + +## Context + +LLM-backed D&D operations share an execution-policy choice, but repeating a +provider or model-named profile on every module binding ties pipeline structure +to a deployment decision. Different environments may require different model, +backend, timeout, or reasoning settings while retaining the same workload. + +Notarius also needs a usable default for maintained D&D prompts without making +an operator profile mandatory. That default must remain owned by the D&D +family, while generic LLM infrastructure stays unaware of domain-specific +policy. + +## Decision + +Pipelines may name one workload-oriented default profile, inherited only by +selected LLM-backed bindings and validators. Binding-level profile IDs remain +intentional exceptions, and the run-wide CLI profile override has highest +precedence. + +The D&D family owns an embedded fallback profile named `dnd-extraction`. +Operators may provide a complete profile with the same ID through a PromptKit +filesystem source. PromptKit selects the higher-precedence matching definition; +Notarius does not merge profile documents. Production, development, and local +deployments can therefore use different execution policy behind one unchanged +pipeline ID. + +## Alternatives considered + +- Repeat a model-named profile on every binding. This makes routine deployment + policy changes noisy and obscures the shared workload intent. +- Require every deployment to install a profile file. This adds configuration + friction and leaves maintained D&D prompts without an application-owned + fallback. +- Put D&D profile policy in generic LLM infrastructure. This breaks domain + ownership and makes generic code depend on one workload. + +## Consequences + +Pipeline configuration expresses workload intent rather than a specific +provider or model. Operators can replace the complete execution policy without +editing bindings, while binding-level and run-wide exceptions remain available. +Profile changes affect resolved pipeline and checkpoint identity, so they may +intentionally cause work to be recomputed. The D&D fallback becomes a +maintained application execution-policy asset. diff --git a/docs/config.md b/docs/config.md index c946558..2aeb274 100644 --- a/docs/config.md +++ b/docs/config.md @@ -32,7 +32,8 @@ override the fields listed below. single-lane Seriatim-to-spell pipeline. - [Complete D&D configuration](../examples/dnd-complete.config.yml) uses ordered steps, all implemented D&D lanes, generated references, state - settings, and bounded LLM concurrency. + settings, bounded LLM concurrency, and the maintained + [operator profile](../examples/profiles/dnd-extraction.yml). Use these complete files as starting points rather than combining the illustrative fragments in this reference. @@ -93,10 +94,22 @@ promptkit: | **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. | | **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. | -Set at most one of **profile_dir** and **profile_file**. Profile IDs used by a -binding must be available from the selected PromptKit profile source when the -pipeline is resolved. The optional local registration may coexist with either -profile source or with PromptKit's built-in profiles. +Set at most one of **profile_dir** and **profile_file**. Relative values use +the process working directory, not the configuration file's directory. The +complete example's `./examples/profiles/dnd-extraction.yml` value is therefore +valid when Notarius is launched from the repository root; use an absolute path +for services and containers. + +An operator source is optional. For a requested ID, PromptKit checks the +configured operator source first, then Notarius's embedded fallback profiles, +then its own built-in catalog. A matching profile is complete: it replaces a +lower-precedence definition rather than merging with it. The maintained +[`dnd-extraction` operator profile](../examples/profiles/dnd-extraction.yml) +is a secret-free deployment artifact; production, development, and local +deployments can each provide a complete definition with that same workload ID. +Use workload-oriented IDs for new profiles instead of model names. +[Operations](operations.md#promptkit-profile-deployment) owns the deployment +workflow and credential-handling guidance. When **local_backend** is present, its endpoint is trimmed and must use HTTP or HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are @@ -119,11 +132,17 @@ model: example-model Keep credentials out of the local-backend object. A PromptKit profile may name its credential environment variable through `api_key_env`; set that variable only in the run environment. PromptKit owns the -[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md). +[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md). The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the supported package API, and [Operations](operations.md#operational-limits) describes the effective concurrency layers. +`notarius config validate --pipeline ` resolves the selected pipeline and +inspects every explicit effective profile without contacting a provider or +requiring credential values. It rejects absent, malformed, or incompatible +profiles before a run prepares modules. Credential availability is checked only +when a generation is prepared. + ## Migrating Version 3 Configuration Version 3 files are not decoded or rewritten. Change **version: 3** to @@ -185,6 +204,7 @@ Each **pipelines** entry has a unique, non-empty ID and the following shape: ~~~yaml pipelines: dnd-session: + llm_profile: dnd-extraction input: seriatim chunk: generic output: json @@ -197,7 +217,7 @@ pipelines: | Field | Type | Default | Rules | | --- | --- | --- | --- | -| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. | +| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. An explicitly present blank value is invalid. | | **input** | module binding | none | Required. | | **chunk** | module binding | **generic** | Optional. | | **output** | module binding | **json** | Optional. | @@ -211,7 +231,8 @@ needs a unique non-empty **id**, an **artifacts** map, and may have **references**. A lane ID must not appear more than once in a pipeline, including across explicit steps. -For each selected LLM-backed binding or validator, profile selection uses the +For each selected LLM-backed binding or validator, profile selection occurs +after module, validator, and `--only` lane selection. It uses the run-level **--llm-profile** value first, then the binding's **llm_profile**, then the pipeline's **llm_profile**, and finally the PromptKit default. Deterministic bindings do not receive these defaults or run overrides. @@ -243,7 +264,7 @@ Use an object for fields: ~~~yaml extract: module: dnd/spells - llm_profile: gemini-2-flash + llm_profile: dnd-extraction retries: 2 references: spell_catalog: ./dnd-spell-catalog.json diff --git a/docs/integrations/pkg-promptkit.md b/docs/integrations/pkg-promptkit.md index 3f104c9..00b022a 100644 --- a/docs/integrations/pkg-promptkit.md +++ b/docs/integrations/pkg-promptkit.md @@ -29,6 +29,13 @@ The pinned [`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/backends.go) owns the registration and backend-capacity contract. +For one completion, the adapter calls `PrepareExecution`, takes a +caller-owned `Details` snapshot, and calls `RunPrepared` for that same opaque +prepared execution. It defers `Discard` for every unexecuted handle. Explicit +profile preflight uses `Engine.InspectProfile`; it does not prepare a synthetic +prompt. PromptKit's prepared handle, inspection result, and capacity-error +types stay inside the Notarius LLM adapter. + When a PromptKit profile and runtime override leave `temperature`, `max_tokens`, or `top_p` unset, Notarius leaves that control unset as well. Compatible providers therefore apply their own defaults; an operator that requires a @@ -63,6 +70,25 @@ PromptKit capacity rejection into the provider-neutral Notarius backend ID in safe diagnostic context, without exposing PromptKit's capacity error type, and leaves retries to the calling pipeline stage. +## Profile Sources And Compatibility + +Notarius gives PromptKit the configured operator profile source, registered +application fallback profile assets, and optional backend registration through +the same construction path for inspection and execution. PromptKit owns the +resulting source precedence and strict profile parsing: a matching operator +profile is a complete replacement for a fallback or built-in profile, while an +invalid matching document fails instead of falling through. The operator +configuration and deployment workflow are defined in +[Configuration](../config.md#promptkit-profiles) and +[Operations](../operations.md#promptkit-profile-deployment). + +Notarius supports this boundary against PromptKit v0.5.0. Its fallback source, +prepared-execution, inspection, and typed capacity APIs are used as public +upstream contracts; other PromptKit APIs or file-format behavior are not +implicitly supported. A dependency upgrade requires reviewing the adapter, +profile-source construction, and this compatibility statement against the +pinned upstream documentation. + ## Notarius Ownership [LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts diff --git a/docs/internal/cli.md b/docs/internal/cli.md index d68f064..566a704 100644 --- a/docs/internal/cli.md +++ b/docs/internal/cli.md @@ -38,8 +38,9 @@ in [Configuration Internals](configuration.md). Configuration validation without a selected pipeline checks structural configuration only. Validation with a selected pipeline also builds the -effective catalog, resolves the pipeline, and verifies explicitly selected -PromptKit profiles. Each explicit binding or validator profile is inspected +effective catalog, resolves the pipeline, and verifies every explicit effective +PromptKit profile. Selected LLM-backed input, chunk, lane, output, and validator +profiles are inspected against the configured PromptKit source and backend registrations without loading a prompt or performing generation, so an unknown or invalid profile fails before pipeline preparation. Credential availability remains an @@ -61,7 +62,8 @@ profile-provenance recorder, creates one scheduler from the effective global LLM limit, and wraps the client before it reaches modules. Registration and LLM construction errors are returned before a pipeline is prepared. Configuration field definitions remain in [Configuration](../config.md#promptkit-profiles); -the adapter mechanics remain in [LLM Runtime](llm.md). +the D&D registrar's fallback profile assets and the adapter mechanics remain in +[LLM Runtime](llm.md). The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer preserves inherit, replace, and clear states across the composition boundary. @@ -83,7 +85,8 @@ handoff: 2. create and validate a safe run identity, then allocate a debug bundle only when requested; 3. build the effective catalog, resolve requested reference changes, resolve - the effective pipeline, and verify explicit PromptKit profiles; + the effective pipeline, and inspect its explicit effective PromptKit + profiles; 4. materialize external or generated references and record redacted invocation and resolution provenance when debug capture is enabled; 5. construct registries, the scheduled LLM client, prepared modules, and the diff --git a/docs/internal/configuration.md b/docs/internal/configuration.md index eb955d4..3d80c61 100644 --- a/docs/internal/configuration.md +++ b/docs/internal/configuration.md @@ -42,7 +42,9 @@ or collide after whitespace normalization. The file application also makes the effective extraction-worker default follow the effective LLM limit. A present PromptKit local-backend object requires and trims its endpoint, defaults its omitted concurrency limit to zero, and is copied so the parsed file model -cannot alias the populated **Config**. +cannot alias the populated **Config**. A pipeline `llm_profile` is +presence-aware: omission remains empty, while a present blank value is +rejected and a non-empty file value is trimmed before it reaches **Config**. **Config.Validate** checks configuration-only invariants before resolution. It rejects incompatible profile sources, invalid state-surface values, unsupported @@ -62,13 +64,15 @@ environment override, its validation, and that reference in the same change. **Config.Resolve** first recomputes derived concurrency defaults and validates the configuration. It normalizes the requested pipeline ID, copies the selected -profile, applies a non-empty command-level LLM profile override to the -LLM-capable stage bindings, and calls the framework resolver with the requested -lane selection and reference changes. +profile, and passes the non-empty command-level LLM profile override, requested +lane selection, and reference changes to the framework resolver. -The command-level override does not replace an explicitly selected validator -profile. Validator bindings remain part of the resolved validator chain and -are resolved under their own declared configuration. +After module and validator selection, the resolver applies the effective +profile policy to LLM-backed bindings only: command override, binding profile, +pipeline profile, then the prompt default. Deterministic bindings remain +profile-free, and no second inheritance decision occurs during execution. The +public field definitions and precedence are owned by +[Configuration](../config.md#pipelines). The framework resolver supplies defaults, selects lanes, resolves validator chains, checks registered module and artifact compatibility, validates module @@ -90,7 +94,8 @@ runtime error class described in the [CLI reference](../cli.md#output-streams-an The framework assigns the resolved pipeline a deterministic SHA-256 digest after defaults, lane selection, module bindings, reference bindings, validator -chains, and artifact schema identity have been resolved. The digest excludes +chains, effective LLM profiles, and artifact schema identity have been +resolved. The digest excludes its own stored value. It identifies resolved composition rather than raw YAML bytes, a debug payload, or all runtime state. The CLI records it as invocation provenance before execution; cache and checkpoint identity have additional diff --git a/docs/internal/dnd.md b/docs/internal/dnd.md index 62268b9..2d19d1d 100644 --- a/docs/internal/dnd.md +++ b/docs/internal/dnd.md @@ -28,9 +28,9 @@ a stable module spec, explicit execution class, strict option decoding, and a typed builder. Scene chunking, every extractor, and NPC normalization are registered as `llm_backed`; the remaining current D&D mergers and normalizers are `deterministic`. The metadata is available to catalog inspection and -resolved-pipeline debug data; it does not yet change profile inheritance. -Configuration remains the canonical owner of the exact keys and validator -order. +resolved-pipeline debug data and determines which selected bindings inherit the +pipeline profile. Configuration remains the canonical owner of the exact keys, +profile precedence, and validator order. Private structured-LLM response schemas are deliberately minimal. They reject invalid JSON structure, missing required fields, incompatible types, and @@ -51,7 +51,9 @@ profile. The D&D registrar embeds that fallback profile with the maintained OpenRouter model, timeout, and service-tier policy. An operator may provide a complete profile with the same ID through the configured PromptKit source; that definition replaces the fallback rather than merging with it. The fallback -leaves reasoning and optional sampling controls unspecified. +leaves reasoning and optional sampling controls unspecified. Deployment profile +selection and the maintained operator example are documented in +[Configuration](../config.md#promptkit-profiles). All extraction prompts share this four-message rendered prefix: the system message without cache control, the identity message without cache control, the diff --git a/docs/internal/llm.md b/docs/internal/llm.md index f36f996..a82e656 100644 --- a/docs/internal/llm.md +++ b/docs/internal/llm.md @@ -30,7 +30,8 @@ maps the trimmed request session to PromptKit's direct per-run session field, retains the same value as the `session_id` prompt variable for maintained prompt compatibility, and forwards profile selection. It then creates one frozen prepared execution, captures its caller-owned credential-redacted -details for debug material, and executes that exact snapshot. The direct field +details for debug material, and executes that exact snapshot through +PromptKit's prepared-execution boundary. The direct field is authoritative for provider session behavior. A session ID is a stable, non-secret correlation identifier and may be exposed to providers and provider observability. The adapter returns PromptKit’s validated raw bytes rather than @@ -48,8 +49,9 @@ production constructs one shared client, the selected state applies uniformly to module calls, retries, and LLM-backed validators for the whole run. An empty request profile lets the prompt select its configured default. Before a -run begins, the CLI asks the adapter to inspect every explicitly selected -binding profile. Inspection resolves the profile and its selected backend and +run begins, the CLI asks the adapter to inspect every explicit profile on the +resolved selected LLM-backed bindings and validators, including inherited +pipeline profiles. Inspection resolves the profile and its selected backend and target without loading a prompt, reading credentials, admitting capacity, or contacting a provider, so a missing or invalid explicit profile fails before stage execution while a valid `api_key_env` may remain unset. Calls record the @@ -80,8 +82,9 @@ identity of PromptKit's compiled-in profile catalog with a deterministic digest of every YAML profile in the configured profile directory, or of the configured profile file, and a deterministic digest of the flattened fallback profile assets. The fingerprint contains neither profile content nor source paths. It -covers both explicit binding profiles and prompt-selected defaults, so changing -a model or other profile setting cannot reuse checkpoints created under the +covers inherited pipeline profiles, explicit binding profiles, and +prompt-selected defaults, so changing a model or other profile setting cannot +reuse checkpoints created under the prior profile source. This cache identity is independent of durable profile provenance: run manifests continue to list only profiles actually observed during LLM calls. When the local backend is registered, a second diff --git a/docs/internal/modules.md b/docs/internal/modules.md index 3f70f21..ef27db5 100644 --- a/docs/internal/modules.md +++ b/docs/internal/modules.md @@ -17,8 +17,10 @@ accepted reference slots. The execution class states whether a module is `deterministic` or `llm_backed`; registries retain it for catalog inspection and resolved-pipeline debug data without constructing the module. The framework uses the declaration to resolve a configured binding before it builds the -implementation. Profile inheritance and deterministic-profile validation are -not implemented at this boundary yet. +implementation. After selection, the resolver applies profile inheritance only +to bindings whose declared execution class is `llm_backed` and rejects a +binding-specific profile on a deterministic module. The user-facing precedence +contract belongs in [Configuration](../config.md#pipelines). Implementations that accept options must provide both an option validator and a builder. The validator is used while resolving configuration; the builder @@ -51,7 +53,7 @@ Production composition is intentionally split by family: - The Seriatim registrar provides the transcript input adapter. Its external input behavior is defined by the [Seriatim contract](../integrations/seriatim.md). - The D&D registrar provides its codecs, extractors, mergers, normalizers, - validators, prompt assets, and default chains. Its behavioral conventions + validators, prompt assets, fallback profile asset, and default chains. Its behavioral conventions are documented in [D&D Module Internals](dnd.md). The CLI owns the composition that invokes these registrars. A module package diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md index 049609c..cf152eb 100644 --- a/docs/internal/pipeline.md +++ b/docs/internal/pipeline.md @@ -34,6 +34,11 @@ requested lanes where that is supported, resolves validator chains, checks module capabilities and typed artifact compatibility, validates options, and assigns a deterministic resolved-composition digest. The resolved pipeline contains bindings and declared reference targets, not external reference bytes. +After selection, the resolver applies command, binding, and pipeline profile +precedence to LLM-backed bindings and validators only; prompt defaults remain +an empty resolved binding profile. Deterministic bindings remain profile-free. +These effective values are part of the digest, so execution and checkpoint +consumers do not repeat profile inheritance. Configuration resolution supplies the selected profile and catalog; see [Configuration Internals](configuration.md). diff --git a/docs/operations.md b/docs/operations.md index ba740f9..a25d94a 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -42,6 +42,42 @@ evidence publication. Apply an appropriate umask and output-root access policy before enabling that option; the requested output modes alone may not be suitable for transcript-bearing bundles. +## PromptKit Profile Deployment + +Profile deployment has four distinct layers: + +| Layer | Owner | Operational role | +| --- | --- | --- | +| Prompts and schemas | Notarius module families | Embedded request and structured-output definitions. They are not deployment profile files. | +| Fallback profiles | Notarius module families | Embedded application defaults, including D&D's `dnd-extraction` profile. | +| Built-in profiles | PromptKit | Upstream catalog entries available when no higher-precedence source defines an ID. | +| Operator profiles | Deployment filesystem | Complete environment-specific definitions selected by `promptkit.profile_file` or `promptkit.profile_dir`. | + +The maintained D&D pipeline uses the workload ID `dnd-extraction`. The +embedded fallback makes that ID usable without an operator file. Production, +development, and local deployments can each install a different complete +definition for the same ID, retaining the pipeline while choosing their own +model, backend, timeout, or reasoning policy. An operator definition wins over +the fallback; it is not merged with it. The configuration field and full +precedence rules are owned by [Configuration](config.md#promptkit-profiles). + +Use a profile source owned by the service account, keep it readable only by +the intended operator, and supply provider credentials through the service +environment—not in the Notarius configuration or profile YAML. The maintained +[operator profile](../examples/profiles/dnd-extraction.yml) is secret-free and +can be copied as a format starting point. Validate a deployment without a +provider call or credentials: + +~~~sh +notarius config validate --config /etc/notarius/config.yml --pipeline dnd-session +~~~ + +Profile paths are currently resolved from the process working directory, not +from the configuration file. The complete example's +`./examples/profiles/dnd-extraction.yml` path is valid for a repository-root +invocation only. Use absolute paths such as +`/etc/notarius/profiles/dnd-extraction.yml` for services and containers. + ## Run Lifecycle Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads @@ -219,7 +255,7 @@ Provider execution settings and the generation timeout come from the selected PromptKit profile. The invocation-only **--reasoning-effort** and **--clear-reasoning-effort** controls may replace or clear that profile setting for all LLM-backed calls in one run without changing the profile. PromptKit -v0.3.0 does not add a provider retry loop. Notarius binding retries rerun the +v0.5.0 does not add a provider retry loop. Notarius binding retries rerun the complete module operation and validation chain as defined by [module bindings](config.md#module-bindings-and-validators). diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 2ba0426..df750cc 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -88,55 +88,6 @@ not as committed release dates. secret unless a concrete multi-tenant or privacy requirement justifies sacrificing deterministic identity across installations. -### Pipeline-Level LLM Profile Defaults - -- Add an optional pipeline-level `llm_profile` default so an operator can - select one PromptKit execution policy for the pipeline without repeating the - same profile ID on every LLM-backed module binding. -- Apply the following precedence consistently: an explicit run-wide - `--llm-profile` override, then a binding-specific `llm_profile`, then the - pipeline-level default, then the prompt definition's embedded - `default_profile`. -- Apply inheritance only to bindings whose resolved modules are LLM-backed, - including applicable chunk, extraction, merge, normalization, and validation - bindings. Do not attach an inherited profile to deterministic modules or - weaken existing validation that rejects profiles where generation is not - supported. -- Resolve inherited profiles before effective-pipeline validation, digest and - checkpoint construction, execution, and provenance capture. Validate every - resulting explicit profile ID against the selected PromptKit profile source, - and ensure manifests and debug output report the profile actually selected - for each generation target. -- Preserve binding-specific profiles as intentional exceptions for modules - that require a different quality, latency, cost, provider, or reasoning - policy. Preserve `--llm-profile` as the convenient highest-precedence - experiment or incident-response override for an entire run. -- Treat PromptKit profiles as execution-policy configuration and prefer stable, - workload-oriented IDs such as `dnd-extraction` over model names. A pipeline - should express the kind of work it performs rather than encode a particular - provider, model, or deployment environment. -- Establish deployment-managed PromptKit profile files as the recommended - environment-specific configuration mechanism. Production, development, and - local deployments may each define the same logical `dnd-extraction` ID with - different model and generation settings, while retaining one unchanged - Notarius pipeline definition. -- Keep deployment profiles distinct from both Notarius prompt assets embedded - in the application binary and PromptKit's built-in profile catalog. Document - that `promptkit.profile_dir` or `promptkit.profile_file` selects an external - filesystem source whose definitions overlay PromptKit built-ins, and - recommend application-owned IDs rather than silently replacing built-in - profile IDs. -- Document an operator-friendly layout in which the Notarius configuration and - a profile subdirectory are deployed together. Until profile paths are - explicitly resolved relative to the configuration file, clearly state that - relative paths use the process working directory and recommend absolute - paths for services and containers. -- Update the configuration reference, operations guide, PromptKit integration - boundary, relevant internal configuration and pipeline documentation, and - maintained D&D examples together. Include one concrete external-profile - example and explain the profile precedence and environment-neutral pipeline - pattern without duplicating PromptKit's complete profile-format reference. - ### Raise The Default Application-Wide LLM Limit - Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary diff --git a/examples/dnd-complete.config.yml b/examples/dnd-complete.config.yml index 03f125d..b9c7766 100644 --- a/examples/dnd-complete.config.yml +++ b/examples/dnd-complete.config.yml @@ -1,4 +1,6 @@ version: 4 +promptkit: + profile_file: ./examples/profiles/dnd-extraction.yml concurrency: total_llm: 2 stage_workers: @@ -16,6 +18,7 @@ debug: directory: ./notarius-debug pipelines: dnd-session: + llm_profile: dnd-extraction input: seriatim # Stable campaign context is shared by every module that accepts these slots. references: @@ -54,7 +57,6 @@ pipelines: merge: appendorder normalize: module: dnd/npcs - llm_profile: gemini-2-flash retries: 2 scene-descriptions: extract: diff --git a/examples/dnd-minimal.config.yml b/examples/dnd-minimal.config.yml index f96ff84..64624c5 100644 --- a/examples/dnd-minimal.config.yml +++ b/examples/dnd-minimal.config.yml @@ -1,6 +1,7 @@ version: 4 pipelines: dnd-session: + llm_profile: dnd-extraction input: seriatim artifacts: spells: diff --git a/examples/profiles/dnd-extraction.yml b/examples/profiles/dnd-extraction.yml new file mode 100644 index 0000000..18546a7 --- /dev/null +++ b/examples/profiles/dnd-extraction.yml @@ -0,0 +1,5 @@ +id: dnd-extraction +backend: openrouter +model: openai/gpt-5.6-luna +timeout_seconds: 240 +service_tier: flex diff --git a/internal/cli/example_contract_test.go b/internal/cli/example_contract_test.go index 70f5043..0b5db7c 100644 --- a/internal/cli/example_contract_test.go +++ b/internal/cli/example_contract_test.go @@ -94,6 +94,37 @@ func TestMaintainedConfigurationExampleSet(t *testing.T) { if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" { t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got) } + + profileEntries, err := os.ReadDir(repositoryPath("examples", "profiles")) + if err != nil { + t.Fatal(err) + } + names = names[:0] + for _, entry := range profileEntries { + if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".yml") { + names = append(names, entry.Name()) + } + } + sort.Strings(names) + if got := strings.Join(names, ","); got != "dnd-extraction.yml" { + t.Fatalf("maintained operator profiles = %q, want dnd-extraction.yml", got) + } +} + +func TestMaintainedExamplesValidateEffectiveProfilesOffline(t *testing.T) { + t.Chdir(repositoryPath()) + t.Setenv("OPENROUTER_API_KEY", "") + for _, example := range maintainedExampleFiles(t) { + t.Run(example.name, func(t *testing.T) { + var stdout, stderr strings.Builder + code := RunWithOptions([]string{ + "config", "validate", "--config", example.path, "--pipeline", "dnd-session", + }, &stdout, &stderr, Options{}) + if code != 0 || stderr.Len() != 0 || !strings.Contains(stdout.String(), `valid for pipeline "dnd-session"`) { + t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String()) + } + }) + } } func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string { diff --git a/internal/cli/production_contract_test.go b/internal/cli/production_contract_test.go index ec05075..943d30f 100644 --- a/internal/cli/production_contract_test.go +++ b/internal/cli/production_contract_test.go @@ -844,9 +844,10 @@ func productionCLIOptions(t *testing.T) Options { func productionOptionsFromComponents(components productionComponents) Options { return Options{ - Catalog: catalogFromRegistries(components.registries), - Registries: components.registries, - LookupEnv: emptyLookup, + Catalog: catalogFromRegistries(components.registries), + Registries: components.registries, + LookupEnv: emptyLookup, + promptKitAssets: components.assets, } }