Keep checkpoints aligned with PromptKit profiles
This commit is contained in:
@@ -41,6 +41,17 @@ identity, provider, and model values for manifest use. Successful completion
|
|||||||
responses and recorded profile manifests identify the adapter provider as
|
responses and recorded profile manifests identify the adapter provider as
|
||||||
`promptkit`.
|
`promptkit`.
|
||||||
|
|
||||||
|
Before execution, the adapter also contributes a non-secret checkpoint
|
||||||
|
fingerprint for the effective PromptKit profile source. It combines the
|
||||||
|
identity of PromptKit's compiled-in profile catalog with a deterministic digest
|
||||||
|
of every YAML profile in the configured profile directory, or of the configured
|
||||||
|
profile file. The fingerprint contains neither profile content nor source
|
||||||
|
paths. It covers both explicit binding profiles and prompt-selected defaults,
|
||||||
|
so changing a model or other profile setting cannot reuse checkpoints created
|
||||||
|
under the prior profile source. This cache identity is independent of durable
|
||||||
|
profile provenance: run manifests continue to list only profiles actually
|
||||||
|
observed during LLM calls.
|
||||||
|
|
||||||
## Shared Provider-Call Limit
|
## Shared Provider-Call Limit
|
||||||
|
|
||||||
Production construction creates one PromptKit client and wraps it in one
|
Production construction creates one PromptKit client and wraps it in one
|
||||||
|
|||||||
@@ -39,7 +39,9 @@ codecs, loader, and recorder. The CLI constructs a recorder whenever checkpoint
|
|||||||
recording is enabled and constructs a loader only for a `--resume` invocation.
|
recording is enabled and constructs a loader only for a `--resume` invocation.
|
||||||
Identity incorporates explicit stable semantic fingerprints collected from
|
Identity incorporates explicit stable semantic fingerprints collected from
|
||||||
prepared modules and validators in addition to configuration, input,
|
prepared modules and validators in addition to configuration, input,
|
||||||
references, runtime overrides, and LLM profiles.
|
references, runtime overrides, observed LLM profiles, and the LLM runtime's
|
||||||
|
non-secret effective profile-source identity. A profile source change therefore
|
||||||
|
causes a cold miss even when the configured profile ID remains unchanged.
|
||||||
The serialized
|
The serialized
|
||||||
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
|
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
|
||||||
they do not describe a current public state surface.
|
they do not describe a current public state surface.
|
||||||
|
|||||||
@@ -117,9 +117,10 @@ compatible recorded work. A resume request fails when checkpoint recording is
|
|||||||
disabled. Without **--resume**, a recording-enabled run executes normally and
|
disabled. Without **--resume**, a recording-enabled run executes normally and
|
||||||
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
||||||
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
||||||
provenance, and prepared-component fingerprints. A changed identity produces a
|
provenance, the effective PromptKit profile-source fingerprint, and
|
||||||
cold miss; Notarius does not migrate, rewrite, or delete older checkpoint
|
prepared-component fingerprints. Changing profile content causes a cold miss
|
||||||
directories.
|
even when its profile ID is unchanged. A changed identity produces a cold miss;
|
||||||
|
Notarius does not migrate, rewrite, or delete older checkpoint directories.
|
||||||
|
|
||||||
Checkpoint state is confined below an identity-specific path:
|
Checkpoint state is confined below an identity-specific path:
|
||||||
|
|
||||||
|
|||||||
@@ -1,376 +0,0 @@
|
|||||||
# PromptKit Dependency Migration Implementation Plan
|
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
In progress. The dependency, framework adapter, version 4 PromptKit
|
|
||||||
configuration migration, provider-neutral module prompt-asset support,
|
|
||||||
provenance alignment, and canonical documentation are implemented; final
|
|
||||||
repository verification is still planned.
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
|
|
||||||
Replace Notarius's dependency on
|
|
||||||
`gitea.maximumdirect.net/eric/scriptorium v0.11.1` with
|
|
||||||
`gitea.maximumdirect.net/eric/promptkit v0.1.0`, and consistently adopt the
|
|
||||||
PromptKit name at the adapter, configuration, provenance, documentation, and
|
|
||||||
module-support boundaries.
|
|
||||||
|
|
||||||
The migration is a clean break. Notarius has no production compatibility
|
|
||||||
requirement for the existing Scriptorium-named configuration or development
|
|
||||||
artifacts, so the implementation must not add legacy configuration aliases,
|
|
||||||
deprecated Go APIs, schema migrations, or compatibility shims.
|
|
||||||
|
|
||||||
PromptKit's public engine contract is source-compatible with the subset
|
|
||||||
Notarius currently consumes. Preserve current prompt content, prompt and schema
|
|
||||||
identities, structured-completion behavior, LLM scheduling, retry behavior,
|
|
||||||
debug capture, redaction, and durable output shapes except for the intentional
|
|
||||||
configuration-version and LLM-provider provenance changes described below.
|
|
||||||
|
|
||||||
## Governing Decisions
|
|
||||||
|
|
||||||
- Pin PromptKit at `v0.1.0` and remove Scriptorium from `go.mod` and `go.sum`.
|
|
||||||
- Rename the top-level configuration section from `scriptorium` to
|
|
||||||
`promptkit`.
|
|
||||||
- Bump the strict file-configuration version from 3 to 4. Version 4 accepts
|
|
||||||
only the new `promptkit` key; it does not recognize `scriptorium`.
|
|
||||||
- Rename the internal adapter and its exported-within-`internal` Go API to
|
|
||||||
PromptKit terminology without transitional aliases.
|
|
||||||
- Use provider-neutral names for module-owned prompt asset helpers. Those
|
|
||||||
helpers describe Notarius assets rather than the library that loads them.
|
|
||||||
- Record `promptkit`, not `scriptorium`, as the LLM adapter/provider value in
|
|
||||||
completion responses and run manifest profile provenance.
|
|
||||||
- Do not add the PromptKit package version to checkpoint identity. A library
|
|
||||||
implementation version is not itself a semantic pipeline input. Existing
|
|
||||||
prompt, schema, module, reference, profile, runtime-override, and prepared
|
|
||||||
component fingerprints remain responsible for semantic invalidation.
|
|
||||||
- Do not adopt PromptKit's optional `ArtifactReader` extension in this work.
|
|
||||||
Notarius retains ownership of materializing and supplying prompt inputs.
|
|
||||||
- Do not add native session propagation in this work. PromptKit `v0.1.0` does
|
|
||||||
not expose the desired request-level session identifier; retain the existing
|
|
||||||
prompt-variable behavior and update the future roadmap terminology only.
|
|
||||||
- Accept PromptKit's documented timeout layering: caller context cancellation
|
|
||||||
is the outer authority, a positive generation timeout supplies an inner
|
|
||||||
request deadline, zero disables only that generation deadline, and the HTTP
|
|
||||||
client timeout remains a transport-wide cap.
|
|
||||||
- PromptKit and Notarius are both GPL-3.0 licensed, so the dependency change
|
|
||||||
requires no Notarius licensing change.
|
|
||||||
|
|
||||||
## Non-Goals
|
|
||||||
|
|
||||||
- Changing prompt text, message ordering, prompt IDs, schema IDs, or embedded
|
|
||||||
asset paths.
|
|
||||||
- Changing public artifact schemas, output bundle contents, run-result
|
|
||||||
receipts, checkpoint formats, or filesystem layouts.
|
|
||||||
- Adding new LLM profiles, changing profile precedence, or changing credential
|
|
||||||
handling.
|
|
||||||
- Refactoring the transport-neutral `StructuredLLMClient` contract.
|
|
||||||
- Adding PromptKit features that Notarius does not currently need.
|
|
||||||
- Preserving version 3 configuration compatibility.
|
|
||||||
|
|
||||||
## Stage 1: Replace the Dependency and Adapter
|
|
||||||
|
|
||||||
Update the Go dependency and the framework adapter as one compilable change.
|
|
||||||
|
|
||||||
### Implementation
|
|
||||||
|
|
||||||
- Replace the Scriptorium requirement with
|
|
||||||
`gitea.maximumdirect.net/eric/promptkit v0.1.0`, then run `go mod tidy` so
|
|
||||||
`go.mod` and `go.sum` contain no obsolete Scriptorium module entries.
|
|
||||||
- In `internal/framework/llm`, replace imports of
|
|
||||||
`gitea.maximumdirect.net/eric/scriptorium` with PromptKit and rename:
|
|
||||||
- `scriptorium_client.go` to `promptkit_client.go`;
|
|
||||||
- `ScriptoriumClientConfig` to `PromptKitClientConfig`;
|
|
||||||
- `ScriptoriumClient` to `PromptKitClient`;
|
|
||||||
- `NewScriptoriumClient` to `NewPromptKitClient`;
|
|
||||||
- `AssetRegistry.ScriptoriumOptions` to
|
|
||||||
`AssetRegistry.PromptKitOptions`; and
|
|
||||||
- Scriptorium-named private input, variable, metadata, debug, response,
|
|
||||||
validation, and error-redaction helpers to PromptKit terminology.
|
|
||||||
- Do not retain aliases for the old types, constructor, method, filenames, or
|
|
||||||
private helpers.
|
|
||||||
- Preserve the existing adapter sequence:
|
|
||||||
1. validate the transport-neutral request and output target;
|
|
||||||
2. map inputs, variables, metadata, prompt identity, profile identity, and
|
|
||||||
session prompt variable into a PromptKit `RunRequest`;
|
|
||||||
3. call `Prepare` to capture rendered debug material;
|
|
||||||
4. call `Run`;
|
|
||||||
5. translate PromptKit validation failure into
|
|
||||||
`contracts.ErrInvalidStructuredOutput`;
|
|
||||||
6. decode the structured artifact into the caller's target; and
|
|
||||||
7. return transport-neutral response, usage, profile, and debug data.
|
|
||||||
- Continue wrapping and redacting upstream errors at the same trust boundary.
|
|
||||||
Update safe wrapper text from Scriptorium to PromptKit without exposing
|
|
||||||
prompt content, credentials, artifact content, filesystem paths, or raw
|
|
||||||
upstream payloads.
|
|
||||||
- Rename the adapter provenance constant and change its value from
|
|
||||||
`scriptorium` to `promptkit`.
|
|
||||||
- Preserve PromptKit's cancellation and timeout semantics. Do not introduce
|
|
||||||
another timeout wrapper in Notarius.
|
|
||||||
- Update the production composition root in `internal/cli/catalog.go` to
|
|
||||||
construct `PromptKitClientConfig` and `NewPromptKitClient`.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- Rename and adapt the existing adapter and asset-registry tests; do not add
|
|
||||||
tests whose only purpose is to enforce private symbol or filename choices.
|
|
||||||
- Through the adapter's stable behavior, retain coverage for:
|
|
||||||
- engine and asset construction failures;
|
|
||||||
- prompt, schema, input, variable, metadata, and profile forwarding;
|
|
||||||
- structured-output success and validation failure;
|
|
||||||
- malformed or empty generated output;
|
|
||||||
- caller cancellation and configured timeout behavior;
|
|
||||||
- credential and content redaction;
|
|
||||||
- raw response and prepared-prompt debug capture;
|
|
||||||
- token usage, including cached and cache-write token fields; and
|
|
||||||
- profile manifest recording.
|
|
||||||
- Use injected PromptKit clients or local deterministic HTTP test servers.
|
|
||||||
Default tests must remain offline and must not require credentials or paid
|
|
||||||
model calls.
|
|
||||||
|
|
||||||
### Completion Criteria
|
|
||||||
|
|
||||||
- The framework and production CLI compile against PromptKit only.
|
|
||||||
- No Go import of the Scriptorium module remains.
|
|
||||||
- Existing adapter behavior is preserved except that response and manifest
|
|
||||||
provenance now identify `promptkit`.
|
|
||||||
|
|
||||||
## Stage 2: Introduce Version 4 PromptKit Configuration
|
|
||||||
|
|
||||||
Make the user-visible configuration terminology agree with the dependency and
|
|
||||||
adapter.
|
|
||||||
|
|
||||||
### Implementation
|
|
||||||
|
|
||||||
- Change `SupportedFileConfigVersion` from 3 to 4.
|
|
||||||
- In `internal/core/config`, rename:
|
|
||||||
- `ScriptoriumConfig` to `PromptKitConfig`;
|
|
||||||
- `FileScriptoriumConfig` to `FilePromptKitConfig`;
|
|
||||||
- `Config.Scriptorium` to `Config.PromptKit`;
|
|
||||||
- `FileConfig.Scriptorium` to `FileConfig.PromptKit`; and
|
|
||||||
- Scriptorium-named validation and application helpers to PromptKit
|
|
||||||
terminology.
|
|
||||||
- Change the runtime JSON and file YAML section name from `scriptorium` to
|
|
||||||
`promptkit`.
|
|
||||||
- Preserve the two optional fields and their meaning:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
version: 4
|
|
||||||
promptkit:
|
|
||||||
profile_dir: /path/to/profiles
|
|
||||||
# profile_file: /path/to/profiles.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
- Preserve trimming, clone, apply, effective-configuration, and redaction
|
|
||||||
behavior for `profile_dir` and `profile_file`.
|
|
||||||
- Preserve the rule that `profile_dir` and `profile_file` are mutually
|
|
||||||
exclusive and that an explicitly supplied value must not be empty.
|
|
||||||
- Update production client construction and explicit-profile validation to
|
|
||||||
read `cfg.PromptKit`.
|
|
||||||
- Rename `internal/cli/scriptorium_profiles.go` and its functions to PromptKit
|
|
||||||
terminology. Preserve the existing profile validation timing and
|
|
||||||
`errors.Is`-based handling of PromptKit's `ErrProfileNotFound`.
|
|
||||||
- Keep YAML decoding strict. A version 4 file containing `scriptorium` must
|
|
||||||
fail as an unknown field.
|
|
||||||
- Add a targeted version 3 migration diagnostic that instructs users to:
|
|
||||||
1. change `version: 3` to `version: 4`; and
|
|
||||||
2. rename `scriptorium:` to `promptkit:`.
|
|
||||||
Do not attempt to decode or automatically rewrite version 3 files.
|
|
||||||
- Update maintained configuration examples to version 4 and use `promptkit`
|
|
||||||
wherever profile sources are demonstrated.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- Update configuration contract tests to establish:
|
|
||||||
- a minimal version 4 file applies over defaults;
|
|
||||||
- explicit PromptKit profile directory and profile file values decode and
|
|
||||||
survive apply, clone, and effective configuration;
|
|
||||||
- the two profile sources remain mutually exclusive;
|
|
||||||
- explicit empty values remain invalid;
|
|
||||||
- unknown fields remain rejected by strict decoding;
|
|
||||||
- version 4 rejects the removed `scriptorium` key; and
|
|
||||||
- version 3 produces the actionable migration classification.
|
|
||||||
- Update CLI contract tests to demonstrate that configured and explicitly
|
|
||||||
selected PromptKit profiles are validated before pipeline preparation and
|
|
||||||
that invalid profiles retain the existing process-failure behavior.
|
|
||||||
- Test configuration behavior at the parser/configuration and CLI composition
|
|
||||||
boundaries. Do not reproduce PromptKit's own profile parser test matrix.
|
|
||||||
|
|
||||||
### Completion Criteria
|
|
||||||
|
|
||||||
- All runtime and file configuration code uses PromptKit terminology.
|
|
||||||
- Maintained examples are valid version 4 files.
|
|
||||||
- The obsolete section is rejected rather than silently accepted or ignored.
|
|
||||||
|
|
||||||
## Stage 3: Make Module Prompt-Asset Support Provider-Neutral
|
|
||||||
|
|
||||||
Remove dependency-brand terminology from module-owned prompt asset
|
|
||||||
registration without changing the assets themselves.
|
|
||||||
|
|
||||||
### Implementation
|
|
||||||
|
|
||||||
- Across the D&D scene chunker, extractors, NPC normalizer, and registration
|
|
||||||
support, rename each `scriptorium_assets.go` and corresponding test file to
|
|
||||||
`prompt_assets.go` and `prompt_assets_test.go`.
|
|
||||||
- Rename private helpers and values such as `scriptoriumPromptRoot`,
|
|
||||||
`scriptoriumPromptMetadata`, and equivalent schema registration names to
|
|
||||||
provider-neutral forms such as `promptAssetRoot` and
|
|
||||||
`promptAssetMetadata`.
|
|
||||||
- Retain PromptKit terminology only where code directly calls a PromptKit API,
|
|
||||||
such as producing `promptkit.Option` values at the framework asset registry
|
|
||||||
boundary.
|
|
||||||
- Do not change:
|
|
||||||
- prompt or schema contents;
|
|
||||||
- prompt, schema, or asset IDs and versions;
|
|
||||||
- embedded filesystem paths;
|
|
||||||
- message ordering or cache-control placement;
|
|
||||||
- module keys or declared prompt inputs;
|
|
||||||
- metadata values or content fingerprint algorithms; or
|
|
||||||
- extraction, normalization, or validation behavior.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- Update existing module prompt-preparation and registration tests to compile
|
|
||||||
through the renamed support code.
|
|
||||||
- Retain the centralized behavioral coverage that all registered production
|
|
||||||
prompts and schemas can be mounted and prepared with their declared inputs.
|
|
||||||
- Retain prompt ordering and cache-prefix contract coverage where ordering is
|
|
||||||
semantically significant.
|
|
||||||
- Do not add word-presence tests, rename detectors, or private-helper tests.
|
|
||||||
|
|
||||||
### Completion Criteria
|
|
||||||
|
|
||||||
- Module-owned code no longer describes its prompt assets as Scriptorium
|
|
||||||
assets.
|
|
||||||
- Prompt and schema fingerprints remain unchanged from the pre-migration
|
|
||||||
source content.
|
|
||||||
|
|
||||||
## Stage 4: Align Provenance and Checkpoint Behavior
|
|
||||||
|
|
||||||
Make the intentional public provenance change explicit while leaving cache
|
|
||||||
identity tied to semantic inputs.
|
|
||||||
|
|
||||||
### Implementation
|
|
||||||
|
|
||||||
- Record `Provider: "promptkit"` in every newly observed
|
|
||||||
`LLMProfileManifest` and `StructuredCompletionResponse` produced by the
|
|
||||||
production adapter.
|
|
||||||
- Update manifest, debug, artifact, and assembled-run expectations that
|
|
||||||
currently identify Scriptorium.
|
|
||||||
- Do not change checkpoint schemas, layouts, or the
|
|
||||||
`CheckpointFingerprintProvider` contract.
|
|
||||||
- Do not introduce a fingerprint containing the PromptKit package name or
|
|
||||||
version. Dependency implementation identity is not a stable semantic
|
|
||||||
identity.
|
|
||||||
- Verify that no accidental prompt, schema, component, reference, profile, or
|
|
||||||
runtime-override fingerprint changes result from the provider-neutral file
|
|
||||||
and helper renames.
|
|
||||||
- If an existing test fixture contains recorded LLM provenance, update only
|
|
||||||
the adapter/provider value and preserve the profile ID, model, usage, and
|
|
||||||
remaining run data.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- At the adapter boundary, assert that a successful completion and recorded
|
|
||||||
profile manifest report `promptkit`.
|
|
||||||
- At one assembled run boundary, confirm that the finalized manifest carries
|
|
||||||
the PromptKit profile provenance observed by the client.
|
|
||||||
- Retain existing checkpoint identity tests for semantic prompt, module,
|
|
||||||
profile, reference, and runtime changes. Do not add a test coupled only to
|
|
||||||
the absence of a dependency-version fingerprint.
|
|
||||||
|
|
||||||
### Completion Criteria
|
|
||||||
|
|
||||||
- New durable and debug provenance consistently identifies PromptKit.
|
|
||||||
- No checkpoint wire-format or layout change has been introduced.
|
|
||||||
|
|
||||||
## Stage 5: Update Canonical Documentation
|
|
||||||
|
|
||||||
Update documentation in the same change that implements the behavior, following
|
|
||||||
the repository's canonical ownership rules.
|
|
||||||
|
|
||||||
### Implementation
|
|
||||||
|
|
||||||
- Update `docs/config.md` to own:
|
|
||||||
- configuration version 4;
|
|
||||||
- the `promptkit` section and its fields;
|
|
||||||
- mutual exclusion and validation rules; and
|
|
||||||
- the version 3-to-4 migration instruction.
|
|
||||||
- Update `docs/operations.md` to describe provider retries and timeouts as
|
|
||||||
PromptKit profile behavior and accurately summarize the timeout layers.
|
|
||||||
- Update `docs/internal/llm.md` to describe:
|
|
||||||
- the PromptKit-backed adapter;
|
|
||||||
- transport-neutral request and response mapping;
|
|
||||||
- asset mounting;
|
|
||||||
- preparation, execution, validation, redaction, and debug behavior;
|
|
||||||
- profile provenance; and
|
|
||||||
- caller, generation, and transport timeout ownership.
|
|
||||||
- Update `docs/internal/cli.md` to describe PromptKit profile validation and
|
|
||||||
production client composition.
|
|
||||||
- Update `docs/development.md` so its task-reading guide refers to the
|
|
||||||
PromptKit integration.
|
|
||||||
- Update any maintained examples and nearby navigation links. Configuration
|
|
||||||
definitions and defaults remain canonical in `docs/config.md`; other
|
|
||||||
documents should summarize and link rather than repeat them.
|
|
||||||
- Replace the copied `docs/integrations/pkg-promptkit.md` guide with a concise
|
|
||||||
Notarius-owned upstream boundary document. It must:
|
|
||||||
- identify the pinned PromptKit package and supported public boundary used
|
|
||||||
by Notarius;
|
|
||||||
- link to PromptKit's canonical upstream package and format documentation;
|
|
||||||
- describe only integration facts that Notarius relies upon;
|
|
||||||
- point to `docs/internal/llm.md` for Notarius implementation behavior; and
|
|
||||||
- contain no copied relative links that resolve only inside the PromptKit
|
|
||||||
source repository.
|
|
||||||
- Update `docs/roadmap/future.md` to replace Scriptorium terminology in the
|
|
||||||
native-session item with PromptKit. Keep the work deferred and state
|
|
||||||
accurately that PromptKit `v0.1.0` does not yet provide the desired direct
|
|
||||||
request-level session field. Do not imply that existing prompt-variable
|
|
||||||
propagation is native provider session support.
|
|
||||||
- Search all maintained Go, Markdown, YAML, JSON, and module files for
|
|
||||||
remaining `Scriptorium` or `scriptorium` occurrences. Retain the old name
|
|
||||||
only where necessary to explain the one-time version 3 migration or
|
|
||||||
historical context.
|
|
||||||
- Validate affected relative documentation links.
|
|
||||||
|
|
||||||
### Completion Criteria
|
|
||||||
|
|
||||||
- Current-behavior documentation outside `docs/roadmap` describes the
|
|
||||||
implemented PromptKit integration only.
|
|
||||||
- Configuration and operational facts have one canonical owner.
|
|
||||||
- The integration document is Notarius-specific and does not duplicate or
|
|
||||||
impersonate upstream package documentation.
|
|
||||||
|
|
||||||
## Stage 6: Repository-Wide Verification
|
|
||||||
|
|
||||||
Complete the migration with focused and repository-wide validation.
|
|
||||||
|
|
||||||
### Required Checks
|
|
||||||
|
|
||||||
Run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
git diff --check
|
|
||||||
go mod tidy
|
|
||||||
go test ./...
|
|
||||||
go vet ./...
|
|
||||||
go build ./cmd/notarius
|
|
||||||
go test -race ./internal/framework/llm ./internal/cli ./internal/modules/dnd/...
|
|
||||||
```
|
|
||||||
|
|
||||||
Also verify:
|
|
||||||
|
|
||||||
- `go.mod` and `go.sum` contain PromptKit and no Scriptorium dependency;
|
|
||||||
- the repository contains no obsolete Scriptorium Go identifiers or imports;
|
|
||||||
- any remaining textual use of Scriptorium is limited to intentional migration
|
|
||||||
or historical explanation;
|
|
||||||
- maintained configuration examples parse successfully as version 4;
|
|
||||||
- tests remain deterministic, offline, and independent of real credentials;
|
|
||||||
- no prompt, schema, artifact, or checkpoint format changed unintentionally;
|
|
||||||
and
|
|
||||||
- no secrets, prompt payloads, source content, or private infrastructure paths
|
|
||||||
were introduced into code, errors, tests, or documentation.
|
|
||||||
|
|
||||||
## Open Questions
|
|
||||||
|
|
||||||
None. The dependency version, compatibility policy, configuration migration,
|
|
||||||
internal naming, provenance value, checkpoint treatment, session scope,
|
|
||||||
documentation ownership, and verification boundaries are decided above.
|
|
||||||
@@ -20,6 +20,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkmap"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkmap"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
||||||
@@ -390,6 +391,14 @@ func TestProductionLLMClientFactoriesBuildOfflineRuntime(t *testing.T) {
|
|||||||
if len(manifests) != 0 {
|
if len(manifests) != 0 {
|
||||||
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
t.Fatalf("eager profile manifests = %#v, want none", manifests)
|
||||||
}
|
}
|
||||||
|
fingerprintProvider, ok := client.(llm.CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("production LLM client %T does not provide one profile-source checkpoint fingerprint", client)
|
||||||
|
}
|
||||||
|
fingerprints, err := fingerprintProvider.LLMCheckpointFingerprints()
|
||||||
|
if err != nil || len(fingerprints) != 1 {
|
||||||
|
t.Fatalf("production LLM checkpoint fingerprints = %#v, error = %v, want one profile-source identity", fingerprints, err)
|
||||||
|
}
|
||||||
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
|
if _, ok := client.(contracts.LLMProfileManifestProvider); !ok {
|
||||||
t.Fatalf("production LLM client %T does not provide profile manifests", client)
|
t.Fatalf("production LLM client %T does not provide profile manifests", client)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -23,6 +23,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkplan"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkplan"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
frameworkdebug "gitea.maximumdirect.net/eric/notarius/internal/framework/debug"
|
frameworkdebug "gitea.maximumdirect.net/eric/notarius/internal/framework/debug"
|
||||||
|
frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -367,6 +368,13 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
|
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
|
||||||
}
|
}
|
||||||
|
var llmFingerprints []checkpoint.Fingerprint
|
||||||
|
if effective.Config.Cache.Checkpoints.Enabled {
|
||||||
|
llmFingerprints, err = llmCheckpointFingerprints(llmClient)
|
||||||
|
if err != nil {
|
||||||
|
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("prepare LLM checkpoint identity: %w", err))
|
||||||
|
}
|
||||||
|
}
|
||||||
llmClient = pipeline.WithDebugLLMRecording(llmClient, debugRecorder)
|
llmClient = pipeline.WithDebugLLMRecording(llmClient, debugRecorder)
|
||||||
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, registries, pipeline.ModuleDependencies{LLM: llmClient})
|
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, registries, pipeline.ModuleDependencies{LLM: llmClient})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -380,7 +388,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), *resume)
|
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(effective.Config.Cache.Checkpoints, opts, effective.ResolvedPipeline, prepared.CheckpointFingerprints(), llmFingerprints, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), *resume)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
return failPipelineCommand(stderr, commandState, terminalWriter, err)
|
||||||
}
|
}
|
||||||
@@ -481,6 +489,7 @@ func checkpointHandlersForRun(
|
|||||||
opts Options,
|
opts Options,
|
||||||
resolved pipeline.ResolvedPipeline,
|
resolved pipeline.ResolvedPipeline,
|
||||||
componentFingerprints []pipeline.CheckpointFingerprint,
|
componentFingerprints []pipeline.CheckpointFingerprint,
|
||||||
|
llmFingerprints []checkpoint.Fingerprint,
|
||||||
rawInput []byte,
|
rawInput []byte,
|
||||||
only []string,
|
only []string,
|
||||||
llmProfiles []artifacts.LLMProfileManifest,
|
llmProfiles []artifacts.LLMProfileManifest,
|
||||||
@@ -495,13 +504,17 @@ func checkpointHandlersForRun(
|
|||||||
return pipeline.NoopCheckpointRecorder(), pipeline.NoopCheckpointLoader(), nil
|
return pipeline.NoopCheckpointRecorder(), pipeline.NoopCheckpointLoader(), nil
|
||||||
}
|
}
|
||||||
identity, err := checkpoint.NewIdentity(checkpoint.IdentityInput{
|
identity, err := checkpoint.NewIdentity(checkpoint.IdentityInput{
|
||||||
Pipeline: resolved,
|
Pipeline: resolved,
|
||||||
InputKey: resolved.Input.Module,
|
InputKey: resolved.Input.Module,
|
||||||
RawInputDigest: rawInputDigest(rawInput),
|
RawInputDigest: rawInputDigest(rawInput),
|
||||||
SelectedLanes: only,
|
SelectedLanes: only,
|
||||||
RuntimeOverrides: runtimeOverrideFingerprints(llmProfileOverride, sessionID),
|
RuntimeOverrides: runtimeOverrideFingerprints(llmProfileOverride, sessionID),
|
||||||
References: pipeline.ReferenceProvenance(resolved),
|
References: pipeline.ReferenceProvenance(resolved),
|
||||||
ProvenanceFingerprints: append(llmProfileFingerprints(llmProfiles), checkpointIdentityFingerprints(componentFingerprints)...),
|
ProvenanceFingerprints: combineCheckpointFingerprints(
|
||||||
|
llmProfileFingerprints(llmProfiles),
|
||||||
|
llmFingerprints,
|
||||||
|
checkpointIdentityFingerprints(componentFingerprints),
|
||||||
|
),
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, fmt.Errorf("create checkpoint identity: %w", err)
|
return nil, nil, fmt.Errorf("create checkpoint identity: %w", err)
|
||||||
@@ -527,6 +540,30 @@ func checkpointHandlersForRun(
|
|||||||
return recorder, loader, nil
|
return recorder, loader, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func llmCheckpointFingerprints(client contracts.StructuredLLMClient) ([]checkpoint.Fingerprint, error) {
|
||||||
|
provider, ok := client.(frameworkllm.CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
values, err := provider.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out := make([]checkpoint.Fingerprint, 0, len(values))
|
||||||
|
for _, value := range values {
|
||||||
|
out = append(out, checkpoint.Fingerprint{Name: value.Name, Value: value.Value})
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func combineCheckpointFingerprints(sources ...[]checkpoint.Fingerprint) []checkpoint.Fingerprint {
|
||||||
|
var out []checkpoint.Fingerprint
|
||||||
|
for _, source := range sources {
|
||||||
|
out = append(out, source...)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
func recomputePolicy(resolved pipeline.ResolvedPipeline, requestedStep string) (pipeline.CheckpointExecutionPolicy, error) {
|
func recomputePolicy(resolved pipeline.ResolvedPipeline, requestedStep string) (pipeline.CheckpointExecutionPolicy, error) {
|
||||||
requestedStep = strings.TrimSpace(requestedStep)
|
requestedStep = strings.TrimSpace(requestedStep)
|
||||||
if requestedStep == "" {
|
if requestedStep == "" {
|
||||||
|
|||||||
@@ -214,8 +214,9 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
fingerprints := prepared.CheckpointFingerprints()
|
fingerprints := prepared.CheckpointFingerprints()
|
||||||
|
llmFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-one"}}
|
||||||
settings := config.CheckpointCacheConfig{Enabled: true, Directory: t.TempDir()}
|
settings := config.CheckpointCacheConfig{Enabled: true, Directory: t.TempDir()}
|
||||||
recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, []byte("same input"), nil, nil, "", "", false)
|
recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -241,7 +242,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
_, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, []byte("same input"), nil, nil, "", "", true)
|
_, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", true)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -253,7 +254,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
}
|
}
|
||||||
changed := replaceCheckpointFingerprintValue(t, fingerprints, normalizeSpellCatalogFingerprintName(), "sha256:changed-effective-catalog")
|
changed := replaceCheckpointFingerprintValue(t, fingerprints, normalizeSpellCatalogFingerprintName(), "sha256:changed-effective-catalog")
|
||||||
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changed, normalizeSpellCatalogFingerprintName())
|
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changed, normalizeSpellCatalogFingerprintName())
|
||||||
_, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, []byte("same input"), nil, nil, "", "", true)
|
_, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, llmFingerprints, []byte("same input"), nil, nil, "", "", true)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -265,13 +266,25 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
|||||||
}
|
}
|
||||||
changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.v3")
|
changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.v3")
|
||||||
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changedMapping, extractSpellMappingFingerprintName())
|
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changedMapping, extractSpellMappingFingerprintName())
|
||||||
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, []byte("same input"), nil, nil, "", "", true)
|
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, llmFingerprints, []byte("same input"), nil, nil, "", "", true)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, decision := mappingLoader.Source(materialized.Input.Module); decision.Reused {
|
if _, decision := mappingLoader.Source(materialized.Input.Module); decision.Reused {
|
||||||
t.Fatalf("changed mapping policy decision = %#v, want cold miss", decision)
|
t.Fatalf("changed mapping policy decision = %#v, want cold miss", decision)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
changedLLMFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-two"}}
|
||||||
|
_, profileLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, changedLLMFingerprints, []byte("same input"), nil, nil, "", "", true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, decision := profileLoader.Source(materialized.Input.Module); decision.Reused {
|
||||||
|
t.Fatalf("changed PromptKit profile source decision = %#v, want cold miss", decision)
|
||||||
|
}
|
||||||
|
if _, decision := profileLoader.Normalize("spells", spellnormalize.Key, normalizeDependencies); decision.Reused {
|
||||||
|
t.Fatalf("changed PromptKit profile normalize decision = %#v, want cold miss", decision)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeSpellCatalogFingerprintName() string {
|
func normalizeSpellCatalogFingerprintName() string {
|
||||||
|
|||||||
14
internal/framework/llm/checkpoint_fingerprint.go
Normal file
14
internal/framework/llm/checkpoint_fingerprint.go
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
package llm
|
||||||
|
|
||||||
|
// CheckpointFingerprint is a stable, non-secret semantic identity contributed
|
||||||
|
// by the LLM runtime before pipeline execution.
|
||||||
|
type CheckpointFingerprint struct {
|
||||||
|
Name string
|
||||||
|
Value string
|
||||||
|
}
|
||||||
|
|
||||||
|
// CheckpointFingerprintProvider exposes LLM-runtime identities that must
|
||||||
|
// participate in checkpoint composition.
|
||||||
|
type CheckpointFingerprintProvider interface {
|
||||||
|
LLMCheckpointFingerprints() ([]CheckpointFingerprint, error)
|
||||||
|
}
|
||||||
@@ -29,8 +29,10 @@ type PromptKitClientConfig struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type PromptKitClient struct {
|
type PromptKitClient struct {
|
||||||
engine *promptkit.Engine
|
engine *promptkit.Engine
|
||||||
recorder *LLMProfileRecorder
|
recorder *LLMProfileRecorder
|
||||||
|
profileDir string
|
||||||
|
profileFile string
|
||||||
}
|
}
|
||||||
|
|
||||||
type LLMProfileRecorder struct {
|
type LLMProfileRecorder struct {
|
||||||
@@ -70,8 +72,10 @@ func NewPromptKitClient(cfg PromptKitClientConfig) (*PromptKitClient, error) {
|
|||||||
recorder = NewLLMProfileRecorder()
|
recorder = NewLLMProfileRecorder()
|
||||||
}
|
}
|
||||||
return &PromptKitClient{
|
return &PromptKitClient{
|
||||||
engine: engine,
|
engine: engine,
|
||||||
recorder: recorder,
|
recorder: recorder,
|
||||||
|
profileDir: strings.TrimSpace(cfg.ProfileDir),
|
||||||
|
profileFile: strings.TrimSpace(cfg.ProfileFile),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -261,6 +265,17 @@ func (c *PromptKitClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
|||||||
return c.recorder.Manifests()
|
return c.recorder.Manifests()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (c *PromptKitClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint, error) {
|
||||||
|
if c == nil {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
fingerprint, err := promptKitProfileFingerprint(c.profileDir, c.profileFile)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return []CheckpointFingerprint{fingerprint}, nil
|
||||||
|
}
|
||||||
|
|
||||||
func NewLLMProfileRecorder() *LLMProfileRecorder {
|
func NewLLMProfileRecorder() *LLMProfileRecorder {
|
||||||
return &LLMProfileRecorder{profiles: map[string]artifacts.LLMProfileManifest{}}
|
return &LLMProfileRecorder{profiles: map[string]artifacts.LLMProfileManifest{}}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
@@ -119,6 +121,63 @@ func TestNewPromptKitClientReportsAssetAndEngineConstructionFailures(t *testing.
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestPromptKitClientCheckpointFingerprintTracksProfileSource(t *testing.T) {
|
||||||
|
profilePath := filepath.Join(t.TempDir(), "profiles.yml")
|
||||||
|
writeProfile := func(model string) {
|
||||||
|
t.Helper()
|
||||||
|
content := "id: checkpoint-profile\nendpoint: http://promptkit.test/v1\nmodel: " + model + "\n"
|
||||||
|
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fingerprintFor := func() CheckpointFingerprint {
|
||||||
|
t.Helper()
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{
|
||||||
|
Assets: newTestPromptKitAssets(t),
|
||||||
|
ProfileFile: profilePath,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
values, err := client.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(values) != 1 || values[0].Name != promptKitProfileFingerprintName {
|
||||||
|
t.Fatalf("checkpoint fingerprints = %#v, want one profile-source identity", values)
|
||||||
|
}
|
||||||
|
return values[0]
|
||||||
|
}
|
||||||
|
|
||||||
|
writeProfile("model-one")
|
||||||
|
first := fingerprintFor()
|
||||||
|
writeProfile("model-two")
|
||||||
|
second := fingerprintFor()
|
||||||
|
if first == second {
|
||||||
|
t.Fatalf("profile-source fingerprint = %#v for both profile models", first)
|
||||||
|
}
|
||||||
|
if strings.Contains(first.Value, profilePath) || strings.Contains(first.Value, "model-one") {
|
||||||
|
t.Fatalf("profile-source fingerprint exposes source details: %#v", first)
|
||||||
|
}
|
||||||
|
|
||||||
|
client, err := NewPromptKitClient(PromptKitClientConfig{Assets: newTestPromptKitAssets(t)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
copy, err := client.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
copy[0].Value = "mutated"
|
||||||
|
fresh, err := client.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if fresh[0].Value == "mutated" {
|
||||||
|
t.Fatal("LLMCheckpointFingerprints exposed mutable backing storage")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
|
func TestPromptKitClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
|
||||||
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
fake := &fakePromptKitLLM{content: `{"ok":true}`}
|
||||||
client := newTestPromptKitClient(t, fake)
|
client := newTestPromptKitClient(t, fake)
|
||||||
|
|||||||
82
internal/framework/llm/promptkit_profile_fingerprint.go
Normal file
82
internal/framework/llm/promptkit_profile_fingerprint.go
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
package llm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"fmt"
|
||||||
|
"io/fs"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
promptKitProfileFingerprintName = "promptkit_profile_source"
|
||||||
|
// The built-in profile catalog is compiled into this pinned PromptKit
|
||||||
|
// release. Update this identity when the dependency is upgraded.
|
||||||
|
promptKitBuiltinProfileCatalogID = "promptkit:v0.1.0:builtin-profiles"
|
||||||
|
)
|
||||||
|
|
||||||
|
func promptKitProfileFingerprint(profileDir, profileFile string) (CheckpointFingerprint, error) {
|
||||||
|
hasher := sha256.New()
|
||||||
|
writeFingerprintPart(hasher, []byte(promptKitBuiltinProfileCatalogID))
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case strings.TrimSpace(profileFile) != "":
|
||||||
|
data, err := os.ReadFile(strings.TrimSpace(profileFile))
|
||||||
|
if err != nil {
|
||||||
|
return CheckpointFingerprint{}, fmt.Errorf("read PromptKit profile file for checkpoint identity: %w", err)
|
||||||
|
}
|
||||||
|
writeFingerprintPart(hasher, data)
|
||||||
|
case strings.TrimSpace(profileDir) != "":
|
||||||
|
digests, err := promptKitProfileFileDigests(strings.TrimSpace(profileDir))
|
||||||
|
if err != nil {
|
||||||
|
return CheckpointFingerprint{}, err
|
||||||
|
}
|
||||||
|
for _, digest := range digests {
|
||||||
|
writeFingerprintPart(hasher, digest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return CheckpointFingerprint{
|
||||||
|
Name: promptKitProfileFingerprintName,
|
||||||
|
Value: "sha256:" + hex.EncodeToString(hasher.Sum(nil)),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitProfileFileDigests(root string) ([][]byte, error) {
|
||||||
|
var digests [][]byte
|
||||||
|
err := filepath.WalkDir(root, func(name string, entry fs.DirEntry, walkErr error) error {
|
||||||
|
if walkErr != nil {
|
||||||
|
return walkErr
|
||||||
|
}
|
||||||
|
if entry.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
extension := filepath.Ext(entry.Name())
|
||||||
|
if extension != ".yaml" && extension != ".yml" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(name)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256(data)
|
||||||
|
digests = append(digests, append([]byte(nil), sum[:]...))
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read PromptKit profile directory for checkpoint identity: %w", err)
|
||||||
|
}
|
||||||
|
sort.Slice(digests, func(i, j int) bool {
|
||||||
|
return string(digests[i]) < string(digests[j])
|
||||||
|
})
|
||||||
|
return digests, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeFingerprintPart(hasher interface{ Write([]byte) (int, error) }, value []byte) {
|
||||||
|
length := []byte(fmt.Sprintf("%d:", len(value)))
|
||||||
|
_, _ = hasher.Write(length)
|
||||||
|
_, _ = hasher.Write(value)
|
||||||
|
}
|
||||||
@@ -53,3 +53,14 @@ func (c *scheduledClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
|
|||||||
}
|
}
|
||||||
return provider.LLMProfileManifests()
|
return provider.LLMProfileManifests()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (c *scheduledClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint, error) {
|
||||||
|
if c == nil || c.client == nil {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
provider, ok := c.client.(CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return provider.LLMCheckpointFingerprints()
|
||||||
|
}
|
||||||
|
|||||||
@@ -75,6 +75,28 @@ func TestScheduledClientPropagatesSchedulerError(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestScheduledClientPreservesCheckpointFingerprints(t *testing.T) {
|
||||||
|
scheduler, err := NewScheduler(1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
inner := &fingerprintedStructuredClient{
|
||||||
|
fingerprints: []CheckpointFingerprint{{Name: "profile_source", Value: "sha256:one"}},
|
||||||
|
}
|
||||||
|
client := NewScheduledClient(inner, scheduler)
|
||||||
|
provider, ok := client.(CheckpointFingerprintProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("scheduled client %T does not preserve checkpoint fingerprints", client)
|
||||||
|
}
|
||||||
|
got, err := provider.LLMCheckpointFingerprints()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(got) != 1 || got[0] != inner.fingerprints[0] {
|
||||||
|
t.Fatalf("checkpoint fingerprints = %#v, want %#v", got, inner.fingerprints)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
type blockingStructuredClient struct {
|
type blockingStructuredClient struct {
|
||||||
release chan struct{}
|
release chan struct{}
|
||||||
inFlight int32
|
inFlight int32
|
||||||
@@ -110,6 +132,15 @@ type errorStructuredClient struct {
|
|||||||
err error
|
err error
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type fingerprintedStructuredClient struct {
|
||||||
|
errorStructuredClient
|
||||||
|
fingerprints []CheckpointFingerprint
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *fingerprintedStructuredClient) LLMCheckpointFingerprints() ([]CheckpointFingerprint, error) {
|
||||||
|
return append([]CheckpointFingerprint(nil), c.fingerprints...), nil
|
||||||
|
}
|
||||||
|
|
||||||
func (c *errorStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
func (c *errorStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
return contracts.StructuredCompletionResponse{}, c.err
|
return contracts.StructuredCompletionResponse{}, c.err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -11,15 +11,15 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ModulePromptFile maps a module-owned embedded prompt file into the
|
// ModulePromptFile maps a module-owned embedded prompt file into the registered
|
||||||
// PromptKit-visible module prompt directory.
|
// module prompt directory.
|
||||||
type ModulePromptFile struct {
|
type ModulePromptFile struct {
|
||||||
Name string
|
Name string
|
||||||
Path string
|
Path string
|
||||||
}
|
}
|
||||||
|
|
||||||
// SharedPromptFile maps a caller-owned shared prompt file into a module's
|
// SharedPromptFile maps a caller-owned shared prompt file into a module's
|
||||||
// PromptKit-visible sharedassets prompt subdirectory.
|
// registered sharedassets prompt subdirectory.
|
||||||
type SharedPromptFile struct {
|
type SharedPromptFile struct {
|
||||||
Name string
|
Name string
|
||||||
FS fs.FS
|
FS fs.FS
|
||||||
|
|||||||
Reference in New Issue
Block a user