Document workload-oriented LLM profile deployment
This commit is contained in:
49
docs/adr/0010-workload-oriented-llm-profile-defaults.md
Normal file
49
docs/adr/0010-workload-oriented-llm-profile-defaults.md
Normal file
@@ -0,0 +1,49 @@
|
|||||||
|
# ADR-0010: Use workload-oriented LLM profile defaults
|
||||||
|
|
||||||
|
**Status:** Accepted
|
||||||
|
**Date:** 2026-08-03
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
LLM-backed D&D operations share an execution-policy choice, but repeating a
|
||||||
|
provider or model-named profile on every module binding ties pipeline structure
|
||||||
|
to a deployment decision. Different environments may require different model,
|
||||||
|
backend, timeout, or reasoning settings while retaining the same workload.
|
||||||
|
|
||||||
|
Notarius also needs a usable default for maintained D&D prompts without making
|
||||||
|
an operator profile mandatory. That default must remain owned by the D&D
|
||||||
|
family, while generic LLM infrastructure stays unaware of domain-specific
|
||||||
|
policy.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Pipelines may name one workload-oriented default profile, inherited only by
|
||||||
|
selected LLM-backed bindings and validators. Binding-level profile IDs remain
|
||||||
|
intentional exceptions, and the run-wide CLI profile override has highest
|
||||||
|
precedence.
|
||||||
|
|
||||||
|
The D&D family owns an embedded fallback profile named `dnd-extraction`.
|
||||||
|
Operators may provide a complete profile with the same ID through a PromptKit
|
||||||
|
filesystem source. PromptKit selects the higher-precedence matching definition;
|
||||||
|
Notarius does not merge profile documents. Production, development, and local
|
||||||
|
deployments can therefore use different execution policy behind one unchanged
|
||||||
|
pipeline ID.
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
|
||||||
|
- Repeat a model-named profile on every binding. This makes routine deployment
|
||||||
|
policy changes noisy and obscures the shared workload intent.
|
||||||
|
- Require every deployment to install a profile file. This adds configuration
|
||||||
|
friction and leaves maintained D&D prompts without an application-owned
|
||||||
|
fallback.
|
||||||
|
- Put D&D profile policy in generic LLM infrastructure. This breaks domain
|
||||||
|
ownership and makes generic code depend on one workload.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Pipeline configuration expresses workload intent rather than a specific
|
||||||
|
provider or model. Operators can replace the complete execution policy without
|
||||||
|
editing bindings, while binding-level and run-wide exceptions remain available.
|
||||||
|
Profile changes affect resolved pipeline and checkpoint identity, so they may
|
||||||
|
intentionally cause work to be recomputed. The D&D fallback becomes a
|
||||||
|
maintained application execution-policy asset.
|
||||||
@@ -32,7 +32,8 @@ override the fields listed below.
|
|||||||
single-lane Seriatim-to-spell pipeline.
|
single-lane Seriatim-to-spell pipeline.
|
||||||
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
|
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
|
||||||
ordered steps, all implemented D&D lanes, generated references, state
|
ordered steps, all implemented D&D lanes, generated references, state
|
||||||
settings, and bounded LLM concurrency.
|
settings, bounded LLM concurrency, and the maintained
|
||||||
|
[operator profile](../examples/profiles/dnd-extraction.yml).
|
||||||
|
|
||||||
Use these complete files as starting points rather than combining the
|
Use these complete files as starting points rather than combining the
|
||||||
illustrative fragments in this reference.
|
illustrative fragments in this reference.
|
||||||
@@ -93,10 +94,22 @@ promptkit:
|
|||||||
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
|
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
|
||||||
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
|
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
|
||||||
|
|
||||||
Set at most one of **profile_dir** and **profile_file**. Profile IDs used by a
|
Set at most one of **profile_dir** and **profile_file**. Relative values use
|
||||||
binding must be available from the selected PromptKit profile source when the
|
the process working directory, not the configuration file's directory. The
|
||||||
pipeline is resolved. The optional local registration may coexist with either
|
complete example's `./examples/profiles/dnd-extraction.yml` value is therefore
|
||||||
profile source or with PromptKit's built-in profiles.
|
valid when Notarius is launched from the repository root; use an absolute path
|
||||||
|
for services and containers.
|
||||||
|
|
||||||
|
An operator source is optional. For a requested ID, PromptKit checks the
|
||||||
|
configured operator source first, then Notarius's embedded fallback profiles,
|
||||||
|
then its own built-in catalog. A matching profile is complete: it replaces a
|
||||||
|
lower-precedence definition rather than merging with it. The maintained
|
||||||
|
[`dnd-extraction` operator profile](../examples/profiles/dnd-extraction.yml)
|
||||||
|
is a secret-free deployment artifact; production, development, and local
|
||||||
|
deployments can each provide a complete definition with that same workload ID.
|
||||||
|
Use workload-oriented IDs for new profiles instead of model names.
|
||||||
|
[Operations](operations.md#promptkit-profile-deployment) owns the deployment
|
||||||
|
workflow and credential-handling guidance.
|
||||||
|
|
||||||
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
|
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
|
||||||
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
|
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
|
||||||
@@ -119,11 +132,17 @@ model: example-model
|
|||||||
Keep credentials out of the local-backend object. A PromptKit profile may name
|
Keep credentials out of the local-backend object. A PromptKit profile may name
|
||||||
its credential environment variable through `api_key_env`; set that variable
|
its credential environment variable through `api_key_env`; set that variable
|
||||||
only in the run environment. PromptKit owns the
|
only in the run environment. PromptKit owns the
|
||||||
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md).
|
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md).
|
||||||
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
|
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
|
||||||
supported package API, and [Operations](operations.md#operational-limits)
|
supported package API, and [Operations](operations.md#operational-limits)
|
||||||
describes the effective concurrency layers.
|
describes the effective concurrency layers.
|
||||||
|
|
||||||
|
`notarius config validate --pipeline <id>` resolves the selected pipeline and
|
||||||
|
inspects every explicit effective profile without contacting a provider or
|
||||||
|
requiring credential values. It rejects absent, malformed, or incompatible
|
||||||
|
profiles before a run prepares modules. Credential availability is checked only
|
||||||
|
when a generation is prepared.
|
||||||
|
|
||||||
## Migrating Version 3 Configuration
|
## Migrating Version 3 Configuration
|
||||||
|
|
||||||
Version 3 files are not decoded or rewritten. Change **version: 3** to
|
Version 3 files are not decoded or rewritten. Change **version: 3** to
|
||||||
@@ -185,6 +204,7 @@ Each **pipelines** entry has a unique, non-empty ID and the following shape:
|
|||||||
~~~yaml
|
~~~yaml
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input: seriatim
|
||||||
chunk: generic
|
chunk: generic
|
||||||
output: json
|
output: json
|
||||||
@@ -197,7 +217,7 @@ pipelines:
|
|||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. |
|
| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. An explicitly present blank value is invalid. |
|
||||||
| **input** | module binding | none | Required. |
|
| **input** | module binding | none | Required. |
|
||||||
| **chunk** | module binding | **generic** | Optional. |
|
| **chunk** | module binding | **generic** | Optional. |
|
||||||
| **output** | module binding | **json** | Optional. |
|
| **output** | module binding | **json** | Optional. |
|
||||||
@@ -211,7 +231,8 @@ needs a unique non-empty **id**, an **artifacts** map, and may have
|
|||||||
**references**. A lane ID must not appear more than once in a pipeline,
|
**references**. A lane ID must not appear more than once in a pipeline,
|
||||||
including across explicit steps.
|
including across explicit steps.
|
||||||
|
|
||||||
For each selected LLM-backed binding or validator, profile selection uses the
|
For each selected LLM-backed binding or validator, profile selection occurs
|
||||||
|
after module, validator, and `--only` lane selection. It uses the
|
||||||
run-level **--llm-profile** value first, then the binding's **llm_profile**,
|
run-level **--llm-profile** value first, then the binding's **llm_profile**,
|
||||||
then the pipeline's **llm_profile**, and finally the PromptKit default.
|
then the pipeline's **llm_profile**, and finally the PromptKit default.
|
||||||
Deterministic bindings do not receive these defaults or run overrides.
|
Deterministic bindings do not receive these defaults or run overrides.
|
||||||
@@ -243,7 +264,7 @@ Use an object for fields:
|
|||||||
~~~yaml
|
~~~yaml
|
||||||
extract:
|
extract:
|
||||||
module: dnd/spells
|
module: dnd/spells
|
||||||
llm_profile: gemini-2-flash
|
llm_profile: dnd-extraction
|
||||||
retries: 2
|
retries: 2
|
||||||
references:
|
references:
|
||||||
spell_catalog: ./dnd-spell-catalog.json
|
spell_catalog: ./dnd-spell-catalog.json
|
||||||
|
|||||||
@@ -29,6 +29,13 @@ The pinned
|
|||||||
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/backends.go)
|
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/backends.go)
|
||||||
owns the registration and backend-capacity contract.
|
owns the registration and backend-capacity contract.
|
||||||
|
|
||||||
|
For one completion, the adapter calls `PrepareExecution`, takes a
|
||||||
|
caller-owned `Details` snapshot, and calls `RunPrepared` for that same opaque
|
||||||
|
prepared execution. It defers `Discard` for every unexecuted handle. Explicit
|
||||||
|
profile preflight uses `Engine.InspectProfile`; it does not prepare a synthetic
|
||||||
|
prompt. PromptKit's prepared handle, inspection result, and capacity-error
|
||||||
|
types stay inside the Notarius LLM adapter.
|
||||||
|
|
||||||
When a PromptKit profile and runtime override leave `temperature`, `max_tokens`,
|
When a PromptKit profile and runtime override leave `temperature`, `max_tokens`,
|
||||||
or `top_p` unset, Notarius leaves that control unset as well. Compatible
|
or `top_p` unset, Notarius leaves that control unset as well. Compatible
|
||||||
providers therefore apply their own defaults; an operator that requires a
|
providers therefore apply their own defaults; an operator that requires a
|
||||||
@@ -63,6 +70,25 @@ PromptKit capacity rejection into the provider-neutral Notarius
|
|||||||
backend ID in safe diagnostic context, without exposing PromptKit's capacity
|
backend ID in safe diagnostic context, without exposing PromptKit's capacity
|
||||||
error type, and leaves retries to the calling pipeline stage.
|
error type, and leaves retries to the calling pipeline stage.
|
||||||
|
|
||||||
|
## Profile Sources And Compatibility
|
||||||
|
|
||||||
|
Notarius gives PromptKit the configured operator profile source, registered
|
||||||
|
application fallback profile assets, and optional backend registration through
|
||||||
|
the same construction path for inspection and execution. PromptKit owns the
|
||||||
|
resulting source precedence and strict profile parsing: a matching operator
|
||||||
|
profile is a complete replacement for a fallback or built-in profile, while an
|
||||||
|
invalid matching document fails instead of falling through. The operator
|
||||||
|
configuration and deployment workflow are defined in
|
||||||
|
[Configuration](../config.md#promptkit-profiles) and
|
||||||
|
[Operations](../operations.md#promptkit-profile-deployment).
|
||||||
|
|
||||||
|
Notarius supports this boundary against PromptKit v0.5.0. Its fallback source,
|
||||||
|
prepared-execution, inspection, and typed capacity APIs are used as public
|
||||||
|
upstream contracts; other PromptKit APIs or file-format behavior are not
|
||||||
|
implicitly supported. A dependency upgrade requires reviewing the adapter,
|
||||||
|
profile-source construction, and this compatibility statement against the
|
||||||
|
pinned upstream documentation.
|
||||||
|
|
||||||
## Notarius Ownership
|
## Notarius Ownership
|
||||||
|
|
||||||
[LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts
|
[LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts
|
||||||
|
|||||||
@@ -38,8 +38,9 @@ in [Configuration Internals](configuration.md).
|
|||||||
|
|
||||||
Configuration validation without a selected pipeline checks structural
|
Configuration validation without a selected pipeline checks structural
|
||||||
configuration only. Validation with a selected pipeline also builds the
|
configuration only. Validation with a selected pipeline also builds the
|
||||||
effective catalog, resolves the pipeline, and verifies explicitly selected
|
effective catalog, resolves the pipeline, and verifies every explicit effective
|
||||||
PromptKit profiles. Each explicit binding or validator profile is inspected
|
PromptKit profile. Selected LLM-backed input, chunk, lane, output, and validator
|
||||||
|
profiles are inspected
|
||||||
against the configured PromptKit source and backend registrations without
|
against the configured PromptKit source and backend registrations without
|
||||||
loading a prompt or performing generation, so an unknown or invalid profile
|
loading a prompt or performing generation, so an unknown or invalid profile
|
||||||
fails before pipeline preparation. Credential availability remains an
|
fails before pipeline preparation. Credential availability remains an
|
||||||
@@ -61,7 +62,8 @@ profile-provenance recorder, creates one scheduler from the effective global
|
|||||||
LLM limit, and wraps the client before it reaches modules. Registration and LLM
|
LLM limit, and wraps the client before it reaches modules. Registration and LLM
|
||||||
construction errors are returned before a pipeline is prepared. Configuration
|
construction errors are returned before a pipeline is prepared. Configuration
|
||||||
field definitions remain in [Configuration](../config.md#promptkit-profiles);
|
field definitions remain in [Configuration](../config.md#promptkit-profiles);
|
||||||
the adapter mechanics remain in [LLM Runtime](llm.md).
|
the D&D registrar's fallback profile assets and the adapter mechanics remain in
|
||||||
|
[LLM Runtime](llm.md).
|
||||||
|
|
||||||
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
|
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
|
||||||
preserves inherit, replace, and clear states across the composition boundary.
|
preserves inherit, replace, and clear states across the composition boundary.
|
||||||
@@ -83,7 +85,8 @@ handoff:
|
|||||||
2. create and validate a safe run identity, then allocate a debug bundle only
|
2. create and validate a safe run identity, then allocate a debug bundle only
|
||||||
when requested;
|
when requested;
|
||||||
3. build the effective catalog, resolve requested reference changes, resolve
|
3. build the effective catalog, resolve requested reference changes, resolve
|
||||||
the effective pipeline, and verify explicit PromptKit profiles;
|
the effective pipeline, and inspect its explicit effective PromptKit
|
||||||
|
profiles;
|
||||||
4. materialize external or generated references and record redacted invocation
|
4. materialize external or generated references and record redacted invocation
|
||||||
and resolution provenance when debug capture is enabled;
|
and resolution provenance when debug capture is enabled;
|
||||||
5. construct registries, the scheduled LLM client, prepared modules, and the
|
5. construct registries, the scheduled LLM client, prepared modules, and the
|
||||||
|
|||||||
@@ -42,7 +42,9 @@ or collide after whitespace normalization. The file application also makes the
|
|||||||
effective extraction-worker default follow the effective LLM limit. A present
|
effective extraction-worker default follow the effective LLM limit. A present
|
||||||
PromptKit local-backend object requires and trims its endpoint, defaults its
|
PromptKit local-backend object requires and trims its endpoint, defaults its
|
||||||
omitted concurrency limit to zero, and is copied so the parsed file model
|
omitted concurrency limit to zero, and is copied so the parsed file model
|
||||||
cannot alias the populated **Config**.
|
cannot alias the populated **Config**. A pipeline `llm_profile` is
|
||||||
|
presence-aware: omission remains empty, while a present blank value is
|
||||||
|
rejected and a non-empty file value is trimmed before it reaches **Config**.
|
||||||
|
|
||||||
**Config.Validate** checks configuration-only invariants before resolution. It
|
**Config.Validate** checks configuration-only invariants before resolution. It
|
||||||
rejects incompatible profile sources, invalid state-surface values, unsupported
|
rejects incompatible profile sources, invalid state-surface values, unsupported
|
||||||
@@ -62,13 +64,15 @@ environment override, its validation, and that reference in the same change.
|
|||||||
|
|
||||||
**Config.Resolve** first recomputes derived concurrency defaults and validates
|
**Config.Resolve** first recomputes derived concurrency defaults and validates
|
||||||
the configuration. It normalizes the requested pipeline ID, copies the selected
|
the configuration. It normalizes the requested pipeline ID, copies the selected
|
||||||
profile, applies a non-empty command-level LLM profile override to the
|
profile, and passes the non-empty command-level LLM profile override, requested
|
||||||
LLM-capable stage bindings, and calls the framework resolver with the requested
|
lane selection, and reference changes to the framework resolver.
|
||||||
lane selection and reference changes.
|
|
||||||
|
|
||||||
The command-level override does not replace an explicitly selected validator
|
After module and validator selection, the resolver applies the effective
|
||||||
profile. Validator bindings remain part of the resolved validator chain and
|
profile policy to LLM-backed bindings only: command override, binding profile,
|
||||||
are resolved under their own declared configuration.
|
pipeline profile, then the prompt default. Deterministic bindings remain
|
||||||
|
profile-free, and no second inheritance decision occurs during execution. The
|
||||||
|
public field definitions and precedence are owned by
|
||||||
|
[Configuration](../config.md#pipelines).
|
||||||
|
|
||||||
The framework resolver supplies defaults, selects lanes, resolves validator
|
The framework resolver supplies defaults, selects lanes, resolves validator
|
||||||
chains, checks registered module and artifact compatibility, validates module
|
chains, checks registered module and artifact compatibility, validates module
|
||||||
@@ -90,7 +94,8 @@ runtime error class described in the [CLI reference](../cli.md#output-streams-an
|
|||||||
|
|
||||||
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
||||||
after defaults, lane selection, module bindings, reference bindings, validator
|
after defaults, lane selection, module bindings, reference bindings, validator
|
||||||
chains, and artifact schema identity have been resolved. The digest excludes
|
chains, effective LLM profiles, and artifact schema identity have been
|
||||||
|
resolved. The digest excludes
|
||||||
its own stored value. It identifies resolved composition rather than raw YAML
|
its own stored value. It identifies resolved composition rather than raw YAML
|
||||||
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
||||||
provenance before execution; cache and checkpoint identity have additional
|
provenance before execution; cache and checkpoint identity have additional
|
||||||
|
|||||||
@@ -28,9 +28,9 @@ a stable module spec, explicit execution class, strict option decoding, and a
|
|||||||
typed builder. Scene chunking, every extractor, and NPC normalization are
|
typed builder. Scene chunking, every extractor, and NPC normalization are
|
||||||
registered as `llm_backed`; the remaining current D&D mergers and normalizers
|
registered as `llm_backed`; the remaining current D&D mergers and normalizers
|
||||||
are `deterministic`. The metadata is available to catalog inspection and
|
are `deterministic`. The metadata is available to catalog inspection and
|
||||||
resolved-pipeline debug data; it does not yet change profile inheritance.
|
resolved-pipeline debug data and determines which selected bindings inherit the
|
||||||
Configuration remains the canonical owner of the exact keys and validator
|
pipeline profile. Configuration remains the canonical owner of the exact keys,
|
||||||
order.
|
profile precedence, and validator order.
|
||||||
|
|
||||||
Private structured-LLM response schemas are deliberately minimal. They reject
|
Private structured-LLM response schemas are deliberately minimal. They reject
|
||||||
invalid JSON structure, missing required fields, incompatible types, and
|
invalid JSON structure, missing required fields, incompatible types, and
|
||||||
@@ -51,7 +51,9 @@ profile. The D&D registrar embeds that fallback profile with the maintained
|
|||||||
OpenRouter model, timeout, and service-tier policy. An operator may provide a
|
OpenRouter model, timeout, and service-tier policy. An operator may provide a
|
||||||
complete profile with the same ID through the configured PromptKit source; that
|
complete profile with the same ID through the configured PromptKit source; that
|
||||||
definition replaces the fallback rather than merging with it. The fallback
|
definition replaces the fallback rather than merging with it. The fallback
|
||||||
leaves reasoning and optional sampling controls unspecified.
|
leaves reasoning and optional sampling controls unspecified. Deployment profile
|
||||||
|
selection and the maintained operator example are documented in
|
||||||
|
[Configuration](../config.md#promptkit-profiles).
|
||||||
|
|
||||||
All extraction prompts share this four-message rendered prefix: the system
|
All extraction prompts share this four-message rendered prefix: the system
|
||||||
message without cache control, the identity message without cache control, the
|
message without cache control, the identity message without cache control, the
|
||||||
|
|||||||
@@ -30,7 +30,8 @@ maps the trimmed request session to PromptKit's direct per-run session field,
|
|||||||
retains the same value as the `session_id` prompt variable for maintained
|
retains the same value as the `session_id` prompt variable for maintained
|
||||||
prompt compatibility, and forwards profile selection. It then creates one
|
prompt compatibility, and forwards profile selection. It then creates one
|
||||||
frozen prepared execution, captures its caller-owned credential-redacted
|
frozen prepared execution, captures its caller-owned credential-redacted
|
||||||
details for debug material, and executes that exact snapshot. The direct field
|
details for debug material, and executes that exact snapshot through
|
||||||
|
PromptKit's prepared-execution boundary. The direct field
|
||||||
is authoritative for provider session behavior. A session ID is a stable,
|
is authoritative for provider session behavior. A session ID is a stable,
|
||||||
non-secret correlation identifier and may be exposed to providers and provider
|
non-secret correlation identifier and may be exposed to providers and provider
|
||||||
observability. The adapter returns PromptKit’s validated raw bytes rather than
|
observability. The adapter returns PromptKit’s validated raw bytes rather than
|
||||||
@@ -48,8 +49,9 @@ production constructs one shared client, the selected state applies uniformly
|
|||||||
to module calls, retries, and LLM-backed validators for the whole run.
|
to module calls, retries, and LLM-backed validators for the whole run.
|
||||||
|
|
||||||
An empty request profile lets the prompt select its configured default. Before a
|
An empty request profile lets the prompt select its configured default. Before a
|
||||||
run begins, the CLI asks the adapter to inspect every explicitly selected
|
run begins, the CLI asks the adapter to inspect every explicit profile on the
|
||||||
binding profile. Inspection resolves the profile and its selected backend and
|
resolved selected LLM-backed bindings and validators, including inherited
|
||||||
|
pipeline profiles. Inspection resolves the profile and its selected backend and
|
||||||
target without loading a prompt, reading credentials, admitting capacity, or
|
target without loading a prompt, reading credentials, admitting capacity, or
|
||||||
contacting a provider, so a missing or invalid explicit profile fails before
|
contacting a provider, so a missing or invalid explicit profile fails before
|
||||||
stage execution while a valid `api_key_env` may remain unset. Calls record the
|
stage execution while a valid `api_key_env` may remain unset. Calls record the
|
||||||
@@ -80,8 +82,9 @@ identity of PromptKit's compiled-in profile catalog with a deterministic digest
|
|||||||
of every YAML profile in the configured profile directory, or of the configured
|
of every YAML profile in the configured profile directory, or of the configured
|
||||||
profile file, and a deterministic digest of the flattened fallback profile
|
profile file, and a deterministic digest of the flattened fallback profile
|
||||||
assets. The fingerprint contains neither profile content nor source paths. It
|
assets. The fingerprint contains neither profile content nor source paths. It
|
||||||
covers both explicit binding profiles and prompt-selected defaults, so changing
|
covers inherited pipeline profiles, explicit binding profiles, and
|
||||||
a model or other profile setting cannot reuse checkpoints created under the
|
prompt-selected defaults, so changing a model or other profile setting cannot
|
||||||
|
reuse checkpoints created under the
|
||||||
prior profile source. This cache identity is independent of durable
|
prior profile source. This cache identity is independent of durable
|
||||||
profile provenance: run manifests continue to list only profiles actually
|
profile provenance: run manifests continue to list only profiles actually
|
||||||
observed during LLM calls. When the local backend is registered, a second
|
observed during LLM calls. When the local backend is registered, a second
|
||||||
|
|||||||
@@ -17,8 +17,10 @@ accepted reference slots. The execution class states whether a module is
|
|||||||
`deterministic` or `llm_backed`; registries retain it for catalog inspection and
|
`deterministic` or `llm_backed`; registries retain it for catalog inspection and
|
||||||
resolved-pipeline debug data without constructing the module. The framework
|
resolved-pipeline debug data without constructing the module. The framework
|
||||||
uses the declaration to resolve a configured binding before it builds the
|
uses the declaration to resolve a configured binding before it builds the
|
||||||
implementation. Profile inheritance and deterministic-profile validation are
|
implementation. After selection, the resolver applies profile inheritance only
|
||||||
not implemented at this boundary yet.
|
to bindings whose declared execution class is `llm_backed` and rejects a
|
||||||
|
binding-specific profile on a deterministic module. The user-facing precedence
|
||||||
|
contract belongs in [Configuration](../config.md#pipelines).
|
||||||
|
|
||||||
Implementations that accept options must provide both an option validator and
|
Implementations that accept options must provide both an option validator and
|
||||||
a builder. The validator is used while resolving configuration; the builder
|
a builder. The validator is used while resolving configuration; the builder
|
||||||
@@ -51,7 +53,7 @@ Production composition is intentionally split by family:
|
|||||||
- The Seriatim registrar provides the transcript input adapter. Its external
|
- The Seriatim registrar provides the transcript input adapter. Its external
|
||||||
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
|
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
|
||||||
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
|
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
|
||||||
validators, prompt assets, and default chains. Its behavioral conventions
|
validators, prompt assets, fallback profile asset, and default chains. Its behavioral conventions
|
||||||
are documented in [D&D Module Internals](dnd.md).
|
are documented in [D&D Module Internals](dnd.md).
|
||||||
|
|
||||||
The CLI owns the composition that invokes these registrars. A module package
|
The CLI owns the composition that invokes these registrars. A module package
|
||||||
|
|||||||
@@ -34,6 +34,11 @@ requested lanes where that is supported, resolves validator chains, checks
|
|||||||
module capabilities and typed artifact compatibility, validates options, and
|
module capabilities and typed artifact compatibility, validates options, and
|
||||||
assigns a deterministic resolved-composition digest. The resolved pipeline
|
assigns a deterministic resolved-composition digest. The resolved pipeline
|
||||||
contains bindings and declared reference targets, not external reference bytes.
|
contains bindings and declared reference targets, not external reference bytes.
|
||||||
|
After selection, the resolver applies command, binding, and pipeline profile
|
||||||
|
precedence to LLM-backed bindings and validators only; prompt defaults remain
|
||||||
|
an empty resolved binding profile. Deterministic bindings remain profile-free.
|
||||||
|
These effective values are part of the digest, so execution and checkpoint
|
||||||
|
consumers do not repeat profile inheritance.
|
||||||
Configuration resolution supplies the selected profile and catalog; see
|
Configuration resolution supplies the selected profile and catalog; see
|
||||||
[Configuration Internals](configuration.md).
|
[Configuration Internals](configuration.md).
|
||||||
|
|
||||||
|
|||||||
@@ -42,6 +42,42 @@ evidence publication. Apply an appropriate umask and output-root access policy
|
|||||||
before enabling that option; the requested output modes alone may not be
|
before enabling that option; the requested output modes alone may not be
|
||||||
suitable for transcript-bearing bundles.
|
suitable for transcript-bearing bundles.
|
||||||
|
|
||||||
|
## PromptKit Profile Deployment
|
||||||
|
|
||||||
|
Profile deployment has four distinct layers:
|
||||||
|
|
||||||
|
| Layer | Owner | Operational role |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Prompts and schemas | Notarius module families | Embedded request and structured-output definitions. They are not deployment profile files. |
|
||||||
|
| Fallback profiles | Notarius module families | Embedded application defaults, including D&D's `dnd-extraction` profile. |
|
||||||
|
| Built-in profiles | PromptKit | Upstream catalog entries available when no higher-precedence source defines an ID. |
|
||||||
|
| Operator profiles | Deployment filesystem | Complete environment-specific definitions selected by `promptkit.profile_file` or `promptkit.profile_dir`. |
|
||||||
|
|
||||||
|
The maintained D&D pipeline uses the workload ID `dnd-extraction`. The
|
||||||
|
embedded fallback makes that ID usable without an operator file. Production,
|
||||||
|
development, and local deployments can each install a different complete
|
||||||
|
definition for the same ID, retaining the pipeline while choosing their own
|
||||||
|
model, backend, timeout, or reasoning policy. An operator definition wins over
|
||||||
|
the fallback; it is not merged with it. The configuration field and full
|
||||||
|
precedence rules are owned by [Configuration](config.md#promptkit-profiles).
|
||||||
|
|
||||||
|
Use a profile source owned by the service account, keep it readable only by
|
||||||
|
the intended operator, and supply provider credentials through the service
|
||||||
|
environment—not in the Notarius configuration or profile YAML. The maintained
|
||||||
|
[operator profile](../examples/profiles/dnd-extraction.yml) is secret-free and
|
||||||
|
can be copied as a format starting point. Validate a deployment without a
|
||||||
|
provider call or credentials:
|
||||||
|
|
||||||
|
~~~sh
|
||||||
|
notarius config validate --config /etc/notarius/config.yml --pipeline dnd-session
|
||||||
|
~~~
|
||||||
|
|
||||||
|
Profile paths are currently resolved from the process working directory, not
|
||||||
|
from the configuration file. The complete example's
|
||||||
|
`./examples/profiles/dnd-extraction.yml` path is valid for a repository-root
|
||||||
|
invocation only. Use absolute paths such as
|
||||||
|
`/etc/notarius/profiles/dnd-extraction.yml` for services and containers.
|
||||||
|
|
||||||
## Run Lifecycle
|
## Run Lifecycle
|
||||||
|
|
||||||
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
|
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
|
||||||
@@ -219,7 +255,7 @@ Provider execution settings and the generation timeout come from the selected
|
|||||||
PromptKit profile. The invocation-only **--reasoning-effort** and
|
PromptKit profile. The invocation-only **--reasoning-effort** and
|
||||||
**--clear-reasoning-effort** controls may replace or clear that profile setting
|
**--clear-reasoning-effort** controls may replace or clear that profile setting
|
||||||
for all LLM-backed calls in one run without changing the profile. PromptKit
|
for all LLM-backed calls in one run without changing the profile. PromptKit
|
||||||
v0.3.0 does not add a provider retry loop. Notarius binding retries rerun the
|
v0.5.0 does not add a provider retry loop. Notarius binding retries rerun the
|
||||||
complete module operation and validation chain as defined by
|
complete module operation and validation chain as defined by
|
||||||
[module bindings](config.md#module-bindings-and-validators).
|
[module bindings](config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
|
|||||||
@@ -88,55 +88,6 @@ not as committed release dates.
|
|||||||
secret unless a concrete multi-tenant or privacy requirement justifies
|
secret unless a concrete multi-tenant or privacy requirement justifies
|
||||||
sacrificing deterministic identity across installations.
|
sacrificing deterministic identity across installations.
|
||||||
|
|
||||||
### Pipeline-Level LLM Profile Defaults
|
|
||||||
|
|
||||||
- Add an optional pipeline-level `llm_profile` default so an operator can
|
|
||||||
select one PromptKit execution policy for the pipeline without repeating the
|
|
||||||
same profile ID on every LLM-backed module binding.
|
|
||||||
- Apply the following precedence consistently: an explicit run-wide
|
|
||||||
`--llm-profile` override, then a binding-specific `llm_profile`, then the
|
|
||||||
pipeline-level default, then the prompt definition's embedded
|
|
||||||
`default_profile`.
|
|
||||||
- Apply inheritance only to bindings whose resolved modules are LLM-backed,
|
|
||||||
including applicable chunk, extraction, merge, normalization, and validation
|
|
||||||
bindings. Do not attach an inherited profile to deterministic modules or
|
|
||||||
weaken existing validation that rejects profiles where generation is not
|
|
||||||
supported.
|
|
||||||
- Resolve inherited profiles before effective-pipeline validation, digest and
|
|
||||||
checkpoint construction, execution, and provenance capture. Validate every
|
|
||||||
resulting explicit profile ID against the selected PromptKit profile source,
|
|
||||||
and ensure manifests and debug output report the profile actually selected
|
|
||||||
for each generation target.
|
|
||||||
- Preserve binding-specific profiles as intentional exceptions for modules
|
|
||||||
that require a different quality, latency, cost, provider, or reasoning
|
|
||||||
policy. Preserve `--llm-profile` as the convenient highest-precedence
|
|
||||||
experiment or incident-response override for an entire run.
|
|
||||||
- Treat PromptKit profiles as execution-policy configuration and prefer stable,
|
|
||||||
workload-oriented IDs such as `dnd-extraction` over model names. A pipeline
|
|
||||||
should express the kind of work it performs rather than encode a particular
|
|
||||||
provider, model, or deployment environment.
|
|
||||||
- Establish deployment-managed PromptKit profile files as the recommended
|
|
||||||
environment-specific configuration mechanism. Production, development, and
|
|
||||||
local deployments may each define the same logical `dnd-extraction` ID with
|
|
||||||
different model and generation settings, while retaining one unchanged
|
|
||||||
Notarius pipeline definition.
|
|
||||||
- Keep deployment profiles distinct from both Notarius prompt assets embedded
|
|
||||||
in the application binary and PromptKit's built-in profile catalog. Document
|
|
||||||
that `promptkit.profile_dir` or `promptkit.profile_file` selects an external
|
|
||||||
filesystem source whose definitions overlay PromptKit built-ins, and
|
|
||||||
recommend application-owned IDs rather than silently replacing built-in
|
|
||||||
profile IDs.
|
|
||||||
- Document an operator-friendly layout in which the Notarius configuration and
|
|
||||||
a profile subdirectory are deployed together. Until profile paths are
|
|
||||||
explicitly resolved relative to the configuration file, clearly state that
|
|
||||||
relative paths use the process working directory and recommend absolute
|
|
||||||
paths for services and containers.
|
|
||||||
- Update the configuration reference, operations guide, PromptKit integration
|
|
||||||
boundary, relevant internal configuration and pipeline documentation, and
|
|
||||||
maintained D&D examples together. Include one concrete external-profile
|
|
||||||
example and explain the profile precedence and environment-neutral pipeline
|
|
||||||
pattern without duplicating PromptKit's complete profile-format reference.
|
|
||||||
|
|
||||||
### Raise The Default Application-Wide LLM Limit
|
### Raise The Default Application-Wide LLM Limit
|
||||||
|
|
||||||
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
|
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
|
||||||
|
|||||||
@@ -1,4 +1,6 @@
|
|||||||
version: 4
|
version: 4
|
||||||
|
promptkit:
|
||||||
|
profile_file: ./examples/profiles/dnd-extraction.yml
|
||||||
concurrency:
|
concurrency:
|
||||||
total_llm: 2
|
total_llm: 2
|
||||||
stage_workers:
|
stage_workers:
|
||||||
@@ -16,6 +18,7 @@ debug:
|
|||||||
directory: ./notarius-debug
|
directory: ./notarius-debug
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input: seriatim
|
||||||
# Stable campaign context is shared by every module that accepts these slots.
|
# Stable campaign context is shared by every module that accepts these slots.
|
||||||
references:
|
references:
|
||||||
@@ -54,7 +57,6 @@ pipelines:
|
|||||||
merge: appendorder
|
merge: appendorder
|
||||||
normalize:
|
normalize:
|
||||||
module: dnd/npcs
|
module: dnd/npcs
|
||||||
llm_profile: gemini-2-flash
|
|
||||||
retries: 2
|
retries: 2
|
||||||
scene-descriptions:
|
scene-descriptions:
|
||||||
extract:
|
extract:
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
version: 4
|
version: 4
|
||||||
pipelines:
|
pipelines:
|
||||||
dnd-session:
|
dnd-session:
|
||||||
|
llm_profile: dnd-extraction
|
||||||
input: seriatim
|
input: seriatim
|
||||||
artifacts:
|
artifacts:
|
||||||
spells:
|
spells:
|
||||||
|
|||||||
5
examples/profiles/dnd-extraction.yml
Normal file
5
examples/profiles/dnd-extraction.yml
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
id: dnd-extraction
|
||||||
|
backend: openrouter
|
||||||
|
model: openai/gpt-5.6-luna
|
||||||
|
timeout_seconds: 240
|
||||||
|
service_tier: flex
|
||||||
@@ -94,6 +94,37 @@ func TestMaintainedConfigurationExampleSet(t *testing.T) {
|
|||||||
if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" {
|
if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" {
|
||||||
t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got)
|
t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
profileEntries, err := os.ReadDir(repositoryPath("examples", "profiles"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
names = names[:0]
|
||||||
|
for _, entry := range profileEntries {
|
||||||
|
if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".yml") {
|
||||||
|
names = append(names, entry.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
if got := strings.Join(names, ","); got != "dnd-extraction.yml" {
|
||||||
|
t.Fatalf("maintained operator profiles = %q, want dnd-extraction.yml", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMaintainedExamplesValidateEffectiveProfilesOffline(t *testing.T) {
|
||||||
|
t.Chdir(repositoryPath())
|
||||||
|
t.Setenv("OPENROUTER_API_KEY", "")
|
||||||
|
for _, example := range maintainedExampleFiles(t) {
|
||||||
|
t.Run(example.name, func(t *testing.T) {
|
||||||
|
var stdout, stderr strings.Builder
|
||||||
|
code := RunWithOptions([]string{
|
||||||
|
"config", "validate", "--config", example.path, "--pipeline", "dnd-session",
|
||||||
|
}, &stdout, &stderr, Options{})
|
||||||
|
if code != 0 || stderr.Len() != 0 || !strings.Contains(stdout.String(), `valid for pipeline "dnd-session"`) {
|
||||||
|
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {
|
func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {
|
||||||
|
|||||||
@@ -847,6 +847,7 @@ func productionOptionsFromComponents(components productionComponents) Options {
|
|||||||
Catalog: catalogFromRegistries(components.registries),
|
Catalog: catalogFromRegistries(components.registries),
|
||||||
Registries: components.registries,
|
Registries: components.registries,
|
||||||
LookupEnv: emptyLookup,
|
LookupEnv: emptyLookup,
|
||||||
|
promptKitAssets: components.assets,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user