452 lines
19 KiB
Markdown
452 lines
19 KiB
Markdown
# Built-In Profiles And Library API Implementation Plan
|
|
|
|
This plan implements the target states in:
|
|
|
|
- `docs/roadmap/builtins.md`
|
|
- `docs/roadmap/library.md`
|
|
|
|
Audience: LLM coding agents implementing the work in order. Review and follow `docs/policy/architecture.md`, `docs/policy/development.md`, and `docs/policy/documentation.md` before changing code.
|
|
|
|
## Global Constraints
|
|
|
|
- Implement built-in profiles before the public library production upgrades.
|
|
- Keep orchestration in `internal/usecase`; adapters and the public package should translate inputs and wire components.
|
|
- Keep `internal/*` packages internal. Public package types must remain facade types.
|
|
- Do not add a mutable global profile registry.
|
|
- Do not add a credential resolver or secret-manager abstraction.
|
|
- Do not accept raw API keys in YAML/JSON config, profile files, prompt files, CLI flags, or HTTP request bodies.
|
|
- Do not emit raw API keys in prepared output, run results, logs, or examples.
|
|
- Prefer standard library APIs. Do not add dependencies unless a later implementation prompt explicitly approves one.
|
|
- Keep each stage passing `go test ./...` before moving to the next stage.
|
|
|
|
## Stage 1: Profile Repository Foundations
|
|
|
|
Goal: add reusable profile repository primitives that support built-ins without changing runner behavior.
|
|
|
|
### Implementation Steps
|
|
|
|
1. Add an `fs.FS`-backed profile repository in `internal/profile`.
|
|
- Constructor shape should be similar to `NewFSRepository(fsys fs.FS, root string) Repository`.
|
|
- It must scan YAML files recursively below `root`.
|
|
- It must use the same strict YAML decoding, profile validation, raw `api_key` rejection, and duplicate-ID behavior as the existing filesystem repository.
|
|
- It must return the existing profile package sentinel errors where applicable.
|
|
|
|
2. Refactor shared profile-loading behavior.
|
|
- Avoid duplicating validation and metadata logic between filesystem and `fs.FS` repositories.
|
|
- The existing `NewFilesystemRepository(dir)` API should remain available.
|
|
- It may either delegate to the `fs.FS` repository through `os.DirFS` or share unexported loader helpers.
|
|
|
|
3. Add an overlay profile repository in `internal/profile`.
|
|
- Constructor shape should be similar to `NewOverlayRepository(primary, fallback Repository) Repository`.
|
|
- Lookup must return the primary result when primary succeeds.
|
|
- Lookup must fall back only when `errors.Is(err, profile.ErrProfileNotFound)` for the primary.
|
|
- Lookup must return primary load/validation errors directly and must not fall back after those errors.
|
|
- Nil repository inputs should be handled deliberately. Prefer treating nil primary as "no primary" and requiring a non-nil fallback for built-in-only operation.
|
|
|
|
### Tests
|
|
|
|
Add focused tests under `internal/profile`.
|
|
|
|
Required coverage:
|
|
|
|
- `NewFSRepository` loads valid profiles from nested directories.
|
|
- `NewFSRepository` rejects unknown YAML fields.
|
|
- `NewFSRepository` rejects raw `api_key` in the selected profile.
|
|
- `NewFSRepository` ignores raw `api_key` in non-selected profiles, matching current filesystem behavior.
|
|
- `NewFSRepository` rejects duplicate IDs within one source.
|
|
- `NewFilesystemRepository` still satisfies all existing repository tests.
|
|
- `NewOverlayRepository` returns primary matches before fallback matches.
|
|
- `NewOverlayRepository` falls back on primary not found.
|
|
- `NewOverlayRepository` does not fall back after a primary invalid YAML/profile/raw-key error.
|
|
- `NewOverlayRepository` returns not found when both sources miss.
|
|
|
|
### Verification
|
|
|
|
Run:
|
|
|
|
```bash
|
|
go test ./internal/profile
|
|
go test ./...
|
|
```
|
|
|
|
## Stage 2: Built-In Profile Assets And Wiring
|
|
|
|
Goal: compile built-in profiles into Scriptorium and make profile lookup use user-over-built-in precedence.
|
|
|
|
### Implementation Steps
|
|
|
|
1. Add an `internal/profile/builtin` package.
|
|
- Store built-in profile YAML files in a stable asset directory under that package.
|
|
- Use Go `embed` to compile those files into the binary/package.
|
|
- Expose a constructor such as `builtin.NewRepository() profile.Repository`.
|
|
- The repository should use the `fs.FS` profile repository from Stage 1.
|
|
|
|
2. Add built-in profile validation tests.
|
|
- Tests should load every built-in profile through the real profile loader.
|
|
- Tests should fail if the built-in profile set contains duplicate IDs.
|
|
- Tests should fail if any built-in profile contains raw `api_key`.
|
|
|
|
3. Add built-in profiles.
|
|
- Use `docs/roadmap/profiles/` as the source catalog for the initial built-in profile set.
|
|
- Copy those YAML files into the built-in profile asset directory, preserving provider subdirectories unless the implementation has a clear reason to flatten them.
|
|
- Do not invent a broad provider/model catalog.
|
|
- Do not add raw API keys.
|
|
- Built-in profiles may use `api_key_env` for CLI/HTTP compatibility when the provider requires authentication.
|
|
|
|
4. Wire repositories through a small helper.
|
|
- Add an internal helper near adapter wiring, or in `internal/profile`, that returns:
|
|
- built-in repository only when no custom profile source is configured;
|
|
- overlay repository when a custom profile source is configured.
|
|
- The runner should still receive only a `profile.Repository`.
|
|
|
|
5. Make `profile_dir` optional.
|
|
- CLI `run`, CLI `render`, and HTTP `serve` argument/config validation should require `prompt_dir` but no longer require `profile_dir`.
|
|
- Public `NewEngine` should no longer reject an empty `Config.ProfileDir`.
|
|
- When `profile_dir` is empty, wire only built-ins.
|
|
- When `profile_dir` is non-empty, wire filesystem profiles over built-ins.
|
|
|
|
### Tests
|
|
|
|
Add or update tests under `internal/adapter/cli`, `internal/adapter/http`, root public package tests, and profile/builtin tests.
|
|
|
|
Required coverage:
|
|
|
|
- CLI parse/config tests accept missing `profile_dir` when `prompt_dir` is present.
|
|
- HTTP serve parse/config tests accept missing `profile_dir` when `prompt_dir` is present.
|
|
- Public `NewEngine` accepts missing `ProfileDir`.
|
|
- A built-in profile ID can be selected with no custom `profile_dir`.
|
|
- A prompt `default_profile` can refer to a built-in profile ID.
|
|
- A custom profile in `profile_dir` overrides a built-in with the same ID.
|
|
- A malformed selected custom profile does not fall back to a built-in with the same ID.
|
|
- A missing profile ID still maps to the existing profile-not-found behavior.
|
|
- Existing duplicate-ID tests for filesystem profiles continue to fail within the custom source.
|
|
|
|
### Documentation
|
|
|
|
After code behavior exists, update non-roadmap docs:
|
|
|
|
- `docs/config.md`: document `profile_dir` as optional and describe built-in fallback/override behavior.
|
|
- `docs/cli.md`: remove claims that `--profile-dir` is required.
|
|
- `docs/internal/adapters.md`: document built-in profile repository composition.
|
|
- Any affected examples or README snippets that imply `profile_dir` is mandatory.
|
|
|
|
Do not document built-in profile IDs outside implemented assets.
|
|
|
|
### Verification
|
|
|
|
Run:
|
|
|
|
```bash
|
|
go test ./internal/profile ./internal/adapter/cli ./internal/adapter/http .
|
|
go test ./...
|
|
go run ./cmd/scriptorium render \
|
|
--config ./examples/config.yml \
|
|
--prompt generic.markdown_summary \
|
|
--input transcript=./examples/fixtures/transcript.md \
|
|
--input glossary=./examples/fixtures/glossary.yml \
|
|
--format json
|
|
```
|
|
|
|
Also run a new smoke command that uses a built-in profile without `--profile-dir` once a concrete built-in profile ID is available.
|
|
|
|
## Stage 3: Public API Credential Value
|
|
|
|
Goal: support the production library credential model: a direct API-key Go value, without adding a resolver or accepting raw keys in serialized config.
|
|
|
|
### Public API Decision
|
|
|
|
Use a single public credential-supply method:
|
|
|
|
```go
|
|
type RunRequest struct {
|
|
// existing fields...
|
|
APIKey string `json:"-"`
|
|
}
|
|
```
|
|
|
|
Do not add `Config.APIKey`, `WithAPIKey`, or a credential resolver in this stage. A request-level value avoids storing secrets on long-lived engines and supports per-tenant callers.
|
|
|
|
### Implementation Steps
|
|
|
|
1. Add direct API-key plumbing through internal request/target types.
|
|
- Add an internal direct API-key field where needed, with `json:"-"` and `yaml:"-"` tags.
|
|
- Convert public `RunRequest.APIKey` into the internal request.
|
|
- Carry the value to the effective execution target used for LLM generation.
|
|
- Do not include the value in prepared/run public results, formatted output, logs, hashes, or docs examples.
|
|
|
|
2. Update credential validation.
|
|
- If a selected/effective profile requires authentication and a direct API key is provided, do not require the environment variable to be set for the public path.
|
|
- Preserve existing CLI/HTTP behavior that uses `api_key_env`.
|
|
- Preserve existing errors for missing environment variables in CLI/HTTP paths.
|
|
|
|
3. Update the OpenAI-compatible LLM client.
|
|
- Prefer the direct API-key value when present.
|
|
- Fall back to existing `api_key_env` behavior for CLI/HTTP compatibility.
|
|
- Never serialize or log the direct API-key value.
|
|
|
|
4. Update public LLM injection conversion.
|
|
- Do not expose the raw API key to injected public `LLMClient` implementations unless that is strictly necessary for custom LLM execution.
|
|
- If custom LLM clients need the key, expose it only on the public `GenerateRequest` with `json:"-"` and document that fake/test clients should avoid logging it.
|
|
|
|
### Tests
|
|
|
|
Required coverage:
|
|
|
|
- Public `Run` can call the default OpenAI-compatible client path with a direct API key without requiring the configured `api_key_env` environment variable.
|
|
- CLI/HTTP behavior using `api_key_env` still works.
|
|
- Missing credentials still fail clearly when a selected profile requires authentication and neither direct key nor usable env value is available.
|
|
- Public `PreparedRun` and `RunResult` JSON do not include the direct API key.
|
|
- Formatted prepared output does not include the direct API key.
|
|
- Direct API-key values are not included in hashes.
|
|
- Injected fake LLM tests either receive no key or receive it only through a `json:"-"` field, depending on the implementation choice above.
|
|
|
|
### Verification
|
|
|
|
Run:
|
|
|
|
```bash
|
|
go test ./internal/llm ./internal/usecase ./internal/adapter/cli ./internal/adapter/http .
|
|
go test ./...
|
|
```
|
|
|
|
## Stage 4: Public Asset Source Options
|
|
|
|
Goal: let library consumers load standard Scriptorium prompt/profile/schema assets from directories, single files, and `fs.FS` sources.
|
|
|
|
### Public API
|
|
|
|
Keep existing `Config` directory fields working. Add options:
|
|
|
|
```go
|
|
func WithPromptFS(fsys fs.FS, root string) Option
|
|
func WithPromptFile(path string) Option
|
|
func WithProfileFS(fsys fs.FS, root string) Option
|
|
func WithProfileFile(path string) Option
|
|
func WithSchemaFS(fsys fs.FS, root string) Option
|
|
func WithSchemaFile(path string) Option
|
|
```
|
|
|
|
Rules:
|
|
|
|
- Directory `Config` fields remain the compatibility path.
|
|
- Explicit options override the corresponding `Config` directory field.
|
|
- Prompt source is required.
|
|
- Profile source is optional because built-in profiles exist.
|
|
- Schema source is optional and should default to the current schema default behavior when not configured.
|
|
- Nil `fs.FS` values, empty required roots, and invalid option combinations return `ErrInvalidConfig`.
|
|
|
|
### Implementation Steps
|
|
|
|
1. Add `fs.FS` prompt-definition support.
|
|
- Implement an `fs.FS` prompt repository that preserves existing strict prompt YAML behavior.
|
|
- Preserve prompt lookup by YAML `id`, not path.
|
|
- Preserve duplicate prompt ID errors within one source.
|
|
- Resolve `content_file` relative to the prompt file's directory inside the same source.
|
|
- Keep the existing filesystem repository API; it may delegate to the `fs.FS` implementation.
|
|
|
|
2. Add public prompt source wiring.
|
|
- `Config.PromptDir` wires the filesystem repository.
|
|
- `WithPromptFS` wires the `fs.FS` repository.
|
|
- `WithPromptFile(path)` wires a single-file source and must still select by prompt YAML `id`.
|
|
|
|
3. Add public profile source wiring.
|
|
- Reuse the Stage 1 profile `fs.FS` repository.
|
|
- `WithProfileFS` and `WithProfileFile` become overlay primaries above built-ins.
|
|
- Empty profile source still means built-ins only.
|
|
|
|
4. Add schema `fs.FS` support.
|
|
- Extend or wrap the standard validator so schema files can be loaded from an `fs.FS` source.
|
|
- `WithSchemaFS` should preserve existing `schema_path` semantics.
|
|
- `WithSchemaFile(path)` should expose the file by its base name; prompts using it should set `schema_path` to that base name.
|
|
|
|
5. Keep adapter scope narrow.
|
|
- This stage is for the public package and shared repositories/validators.
|
|
- Do not change CLI/HTTP request shapes for prompt or schema `fs.FS` sources.
|
|
|
|
### Tests
|
|
|
|
Required coverage:
|
|
|
|
- Public `Prepare` works with prompt definitions from `embed.FS`.
|
|
- `content_file` references resolve relative to the prompt file in `embed.FS`.
|
|
- Public `Prepare` works with `WithPromptFile`.
|
|
- Public `Run` or `Prepare` works with `WithProfileFS` over built-ins.
|
|
- Public `Run` or `Prepare` works with `WithProfileFile` over built-ins.
|
|
- Public structured-output schema validation works with `WithSchemaFS`.
|
|
- `WithSchemaFile` works when the prompt's `schema_path` is the schema file base name.
|
|
- Explicit source options override `Config` directory fields.
|
|
- Invalid/nil source options return `ErrInvalidConfig`.
|
|
- Existing filesystem prompt/profile/schema behavior remains unchanged.
|
|
|
|
### Verification
|
|
|
|
Run:
|
|
|
|
```bash
|
|
go test ./internal/promptdef ./internal/profile ./internal/validate .
|
|
go test ./...
|
|
```
|
|
|
|
## Stage 5: Public In-Memory Profiles And Profile Templates
|
|
|
|
Goal: allow library callers to provide typed profile values and construct stable OpenAI-compatible profile templates without generating YAML.
|
|
|
|
### Public API
|
|
|
|
Add a public profile facade aligned with the profile YAML contract:
|
|
|
|
```go
|
|
type Profile struct {
|
|
ID string
|
|
Endpoint string
|
|
Model string
|
|
Temperature float64
|
|
MaxTokens int
|
|
TopP float64
|
|
TimeoutSeconds int
|
|
ServiceTier string
|
|
ReasoningEffort string
|
|
APIKeyRequired bool
|
|
ExtraParams map[string]any
|
|
}
|
|
|
|
func WithProfiles(profiles ...Profile) Option
|
|
```
|
|
|
|
Add an OpenAI-compatible template constructor:
|
|
|
|
```go
|
|
type OpenAICompatibleProfileConfig struct {
|
|
ID string
|
|
Endpoint string
|
|
Model string
|
|
APIKeyRequired bool
|
|
Temperature float64
|
|
MaxTokens int
|
|
TopP float64
|
|
TimeoutSeconds int
|
|
ServiceTier string
|
|
ReasoningEffort string
|
|
ExtraParams map[string]any
|
|
}
|
|
|
|
func OpenAICompatibleProfile(cfg OpenAICompatibleProfileConfig) Profile
|
|
```
|
|
|
|
Rules:
|
|
|
|
- `WithProfiles` profiles override built-ins with the same ID.
|
|
- If `WithProfiles` and a file/FS profile source are both configured, in-memory profiles have highest precedence, then file/FS profiles, then built-ins.
|
|
- Public profile values must not include raw API-key fields.
|
|
- Use `APIKeyRequired` to indicate whether `RunRequest.APIKey` is required for the public path. Internal conversion may map this to the existing auth-required/profile credential model without exposing raw keys.
|
|
|
|
### Implementation Steps
|
|
|
|
1. Add a profile repository for public in-memory `Profile` values.
|
|
- Validate with the same effective rules as YAML profiles.
|
|
- Reject duplicate IDs within the provided values.
|
|
- Deep-copy `ExtraParams` across public/internal boundaries.
|
|
|
|
2. Compose profile sources in public `NewEngine`.
|
|
- Highest: in-memory public profiles.
|
|
- Next: configured profile file/FS/directory source.
|
|
- Fallback: built-in profiles.
|
|
|
|
3. Add template constructor conversion.
|
|
- `OpenAICompatibleProfile` should be a convenience constructor only.
|
|
- It should not register global state.
|
|
- It should not maintain a broad model catalog.
|
|
|
|
4. Ensure direct API-key behavior works with in-memory/template profiles.
|
|
- Profiles marked `APIKeyRequired` should require `RunRequest.APIKey` for public default LLM execution.
|
|
- Profiles not marked `APIKeyRequired` should not require an API key.
|
|
|
|
### Tests
|
|
|
|
Required coverage:
|
|
|
|
- Public `Prepare`/`Run` uses `WithProfiles` without any profile files.
|
|
- In-memory profiles override built-ins.
|
|
- In-memory profiles override file/FS profile sources when IDs collide.
|
|
- Duplicate in-memory profile IDs return `ErrInvalidConfig`.
|
|
- Template-created profiles execute through the same path as normal profiles.
|
|
- Template-created profiles requiring an API key work with `RunRequest.APIKey`.
|
|
- Template-created profiles that do not require an API key work without one.
|
|
- `ExtraParams` in public profiles are deep-copied and isolated from caller mutation.
|
|
|
|
### Verification
|
|
|
|
Run:
|
|
|
|
```bash
|
|
go test .
|
|
go test ./...
|
|
```
|
|
|
|
## Stage 6: Public Documentation And Examples
|
|
|
|
Goal: document implemented behavior in canonical locations after code exists.
|
|
|
|
### Documentation
|
|
|
|
Update:
|
|
|
|
- `README.md`: add a short pointer to library usage and built-in profiles without turning the README into a manual.
|
|
- `docs/config.md`: document optional `profile_dir`, built-in override behavior, and implemented built-in profile IDs.
|
|
- `docs/cli.md`: document optional `--profile-dir` and built-in profile selection.
|
|
- `docs/internal/adapters.md`: document repository composition and public library adapter surface.
|
|
- `docs/consumers/api.md`: describe public consumer surfaces at a high level.
|
|
- `docs/consumers/pkg-scriptorium.md`: document root package usage, source options, direct API-key value, errors, and examples.
|
|
- `docs/integrations/openai-compatible-chat.md`: update only if direct API-key plumbing changes provider-call semantics.
|
|
|
|
Do not document unimplemented roadmap behavior outside `docs/roadmap/`.
|
|
|
|
### Examples
|
|
|
|
Add or update copyable examples that require no real provider credentials:
|
|
|
|
- prepare with built-in profile and no `profile_dir`;
|
|
- public library prepare from `embed.FS`;
|
|
- public library run with injected fake LLM;
|
|
- public library run with typed/template profile and direct API key, using a fake/local provider path so no real secret is required.
|
|
|
|
Examples must be secret-free.
|
|
|
|
### Tests And Smoke Commands
|
|
|
|
Required verification:
|
|
|
|
```bash
|
|
go test ./...
|
|
go run ./cmd/scriptorium render \
|
|
--config ./examples/config.yml \
|
|
--prompt generic.markdown_summary \
|
|
--input transcript=./examples/fixtures/transcript.md \
|
|
--input glossary=./examples/fixtures/glossary.yml \
|
|
--format json
|
|
```
|
|
|
|
Also run smoke commands for any new examples, such as:
|
|
|
|
```bash
|
|
go run ./examples/go-library/prepare
|
|
```
|
|
|
|
If an example relies on a specific built-in profile ID, use an ID from `docs/roadmap/profiles/` and include that smoke command in the implementing change.
|
|
|
|
## Final Completion Checklist
|
|
|
|
Before marking the combined feature complete:
|
|
|
|
1. `go test ./...` passes.
|
|
2. Existing CLI render smoke command passes.
|
|
3. At least one smoke command proves built-in profile lookup works without `profile_dir`.
|
|
4. Public package examples compile/run without real provider credentials.
|
|
5. `profile_dir` is optional in CLI, HTTP, and public engine construction.
|
|
6. Custom profiles override built-ins with the same ID.
|
|
7. Malformed selected custom profiles do not fall back to built-ins.
|
|
8. Built-in profiles use the same validation rules as file profiles.
|
|
9. Public `RunRequest.APIKey` is the documented library credential method.
|
|
10. Raw API keys do not appear in YAML/JSON config, prepared output, run results, logs, hashes, or examples.
|
|
11. Non-roadmap docs describe only implemented behavior.
|