Refactor: split prompt definition from execution settings and migrate run contracts to prompt_* + execution_target
This commit is contained in:
234
architecture.md
234
architecture.md
@@ -2,11 +2,11 @@
|
||||
|
||||
## 1. Purpose and Non-Goals
|
||||
|
||||
Scriptorium is a prompt-profile execution engine.
|
||||
Scriptorium is a prompt-definition execution engine.
|
||||
|
||||
It takes named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata.
|
||||
It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata.
|
||||
|
||||
Scriptorium is not an orchestrator. It should not own transcription, transcript merging, transcript polishing, notification, or cross-step workflow control.
|
||||
Scriptorium is not an orchestrator. It must not own transcription, transcript merge/polish steps, notifications, or cross-step workflow control.
|
||||
|
||||
For the motivating D&D workflow:
|
||||
|
||||
@@ -14,111 +14,116 @@ For the motivating D&D workflow:
|
||||
- WhisperX transcribes.
|
||||
- Seriatim merges transcripts.
|
||||
- Audita polishes transcripts.
|
||||
- Scriptorium generates output artifacts from prepared inputs.
|
||||
- Scriptorium generates final artifacts from prepared inputs.
|
||||
|
||||
Core Go code must remain domain-generic.
|
||||
Core Go code remains generic.
|
||||
|
||||
## 2. Current Architecture
|
||||
|
||||
Current high-level structure:
|
||||
Current implementation structure:
|
||||
|
||||
- `cmd/scriptorium`: binary entrypoint.
|
||||
- `internal/domain`: core domain types.
|
||||
- `internal/usecase`: `Runner` use case and repair loop orchestration.
|
||||
- `internal/profile`: filesystem prompt profile repository and profile validation.
|
||||
- `internal/artifact`: artifact reference readers (`inline`, `file`) and routing.
|
||||
- `internal/prompt`: Go template-based prompt renderer.
|
||||
- `internal/llm`: LLM client interface + OpenAI-compatible HTTP adapter.
|
||||
- `internal/validate`: output validator implementation (`none/basic/json/json_schema`).
|
||||
- `internal/domain`: core domain contracts.
|
||||
- `internal/usecase`: `Runner` run flow, validation integration, bounded repair coordination.
|
||||
- `internal/profile`: transitional filesystem prompt-definition repository (package rename deferred).
|
||||
- `internal/artifact`: input artifact resolution (`inline`, `file`).
|
||||
- `internal/prompt`: template rendering.
|
||||
- `internal/llm`: provider-neutral client interface + OpenAI-compatible HTTP adapter.
|
||||
- `internal/validate`: validation implementation (`none/basic/json/json_schema`).
|
||||
- `internal/adapter/cli`: CLI adapter.
|
||||
- `internal/adapter/http`: HTTP adapter (`POST /v1/runs`).
|
||||
|
||||
This is a practical ports-and-adapters implementation.
|
||||
|
||||
## 3. Run Data Flow
|
||||
|
||||
`Runner.Run(ctx, RunRequest)` currently executes:
|
||||
|
||||
1. Validate minimum request requirements (`profile_id`).
|
||||
2. Load prompt profile by ID/version.
|
||||
3. Merge effective model target (profile defaults + request override).
|
||||
4. Resolve effective output contract (profile + optional request override).
|
||||
5. Resolve named artifact refs to loaded artifacts.
|
||||
6. Render prompt messages from templates.
|
||||
7. Hash rendered prompt for auditability.
|
||||
8. Call LLM client with provider-neutral `GenerateRequest`.
|
||||
9. Build output artifact from model content.
|
||||
1. Validate request (`prompt_id` required).
|
||||
2. Load `PromptDefinition` by ID/version.
|
||||
3. Determine selected profile ID (`request.profile_id` or prompt `default_profile`).
|
||||
4. Resolve effective execution target from request override (execution-profile loading is deferred in this phase).
|
||||
5. Resolve named input artifact refs.
|
||||
6. Render prompt messages.
|
||||
7. Hash prompt definition and rendered prompt.
|
||||
8. Call LLM client with `GenerateRequest`.
|
||||
9. Build output artifact.
|
||||
10. Validate output.
|
||||
11. If structured validation failed and repair is enabled/bounded, run repair attempts and re-validate.
|
||||
12. Return `RunResult` with artifact, validation, raw output, and metadata.
|
||||
11. If structured validation failed and repair is enabled, run bounded repair attempts and re-validate.
|
||||
12. Return `RunResult` with artifact, raw output, validation result, metadata.
|
||||
|
||||
Validation content failure remains a successful run result with `validation.status=failed`.
|
||||
Validation content failures are returned as successful runs with `validation.status=failed`.
|
||||
|
||||
## 4. Package Responsibilities
|
||||
|
||||
- `domain`
|
||||
- Owns core nouns and contracts.
|
||||
- Must not import adapters/provider SDK types.
|
||||
- Owns core nouns/contracts.
|
||||
- Must not depend on adapters/provider SDK types.
|
||||
|
||||
- `usecase`
|
||||
- Owns execution sequence and cross-port orchestration for a single run.
|
||||
- May coordinate validation and bounded repair.
|
||||
- Must not contain HTTP/CLI/wire concerns.
|
||||
- Owns single-run orchestration across ports.
|
||||
- Owns bounded repair control flow.
|
||||
- Must not own transport/wire concerns.
|
||||
|
||||
- `profile`
|
||||
- Owns prompt profile loading/parsing/validation.
|
||||
- Handles YAML strict decoding and profile-level constraints.
|
||||
- `profile` (transitional)
|
||||
- Currently loads prompt definitions from YAML.
|
||||
- Package naming split (`prompt definition repo` vs `execution profile repo`) is deferred follow-up.
|
||||
|
||||
- `artifact`
|
||||
- Owns artifact ref resolution and content loading.
|
||||
- Produces normalized `Artifact` values with size/hash/content type.
|
||||
- Loads artifacts from refs and normalizes payload metadata.
|
||||
|
||||
- `prompt`
|
||||
- Owns template rendering and required-input enforcement.
|
||||
- Renders templates and enforces required inputs.
|
||||
|
||||
- `llm`
|
||||
- Owns generation port and provider adapters.
|
||||
- Current adapter: OpenAI-compatible chat completions over `net/http`.
|
||||
- Defines generation client contract and protocol adapters.
|
||||
|
||||
- `validate`
|
||||
- Owns output validation semantics and JSON Schema integration.
|
||||
- Owns output validation semantics and schema validation.
|
||||
|
||||
- `adapter/http`, `adapter/cli`
|
||||
- Owns transport/wire/flag concerns only.
|
||||
- Should stay thin and delegate business flow to `usecase.Runner`.
|
||||
- Own request/response/flag mapping only.
|
||||
- Delegate business flow to `usecase.Runner`.
|
||||
|
||||
## 5. Domain Model (Current)
|
||||
|
||||
Key types in `internal/domain`:
|
||||
Key types:
|
||||
|
||||
- `RunRequest`: profile selector, named input refs, vars, optional model override, optional validation override.
|
||||
- `RunResult`: output artifact, validation result, raw output, profile/model metadata, hashes, usage, timestamps, duration.
|
||||
- `ArtifactRef`: `{type, uri, body}` reference contract.
|
||||
- `Artifact`: loaded payload (`name`, `content_type`, `body`, `uri`, `size`, `hash`).
|
||||
- `PromptProfile`: YAML-backed profile definition.
|
||||
- `RenderedPrompt` / `RenderedMessage`: provider-neutral prompt structure.
|
||||
- `GenerateRequest` / `GenerateResponse`: provider-neutral model I/O.
|
||||
- `ValidationResult`: passed/failed/skipped + mode/errors/schema/repair attempts.
|
||||
- `PromptDefinition`
|
||||
- `id`, `version`, `default_profile`, `inputs`, `templates`, `output_format`, `validation`.
|
||||
- `ExecutionProfile`
|
||||
- Execution/runtime settings shape (`endpoint`, `model`, timeouts, `api_key_env`, etc.).
|
||||
- Loading/persistence is deferred in this pass.
|
||||
- `ExecutionTarget`
|
||||
- Effective execution settings for a run.
|
||||
- `RunRequest`
|
||||
- `prompt_id`, `prompt_version`, optional `profile_id`, `inputs`, `vars`, optional `execution` override, optional validation override.
|
||||
- `RunResult`
|
||||
- Output artifact, validation, raw output, prompt/profile/model metadata, hashes, timing, usage.
|
||||
- `ArtifactRef` / `Artifact`
|
||||
- Input reference and loaded content contracts.
|
||||
- `RenderedPrompt` / `RenderedMessage`
|
||||
- Provider-neutral rendered prompt.
|
||||
- `GenerateRequest` / `GenerateResponse`
|
||||
- Provider-neutral model I/O.
|
||||
|
||||
## 6. Interfaces and Adapters
|
||||
|
||||
Primary ports:
|
||||
|
||||
- `profile.Repository`
|
||||
- `profile.Repository` (transitional prompt-definition lookup)
|
||||
- `artifact.Reader`
|
||||
- `prompt.Renderer`
|
||||
- `llm.Client`
|
||||
- `validate.Validator`
|
||||
- `usecase.OutputRepairer` (usecase-local abstraction)
|
||||
- `usecase.OutputRepairer` (usecase-local)
|
||||
|
||||
Current adapters:
|
||||
|
||||
- Profile repository: filesystem YAML loader.
|
||||
- Artifact reader: composite reader for `inline` and `file`.
|
||||
- Prompt renderer: Go templates with input helper + vars.
|
||||
- LLM adapter: OpenAI-compatible `/chat/completions`.
|
||||
- Validator: standard validator with `none/basic/json/json_schema`.
|
||||
- CLI/HTTP adapters: thin request mapping and response mapping.
|
||||
- Prompt definition repository: filesystem YAML loader.
|
||||
- Artifact readers: `file`, `inline` via composite reader.
|
||||
- Prompt renderer: Go templates with `input` helper.
|
||||
- LLM adapter: OpenAI-compatible `/chat/completions` over `net/http`.
|
||||
- Validator: standard validator (`none/basic/json/json_schema`).
|
||||
- CLI/HTTP adapters.
|
||||
|
||||
## 7. Validation and Repair Model
|
||||
|
||||
@@ -131,12 +136,11 @@ Validation modes:
|
||||
|
||||
Repair behavior:
|
||||
|
||||
- Only applies to structured modes (`json`, `json_schema`).
|
||||
- Attempted only when validation fails, repairer exists, and `repair_attempts > 0`.
|
||||
- Bounded strictly by `repair_attempts`.
|
||||
- Uses a narrow JSON-repair prompt and re-validates each attempt.
|
||||
- If still invalid, run succeeds with failed validation and preserved final raw output.
|
||||
- Validator runtime/config errors are run errors.
|
||||
- Applies only to structured modes (`json`, `json_schema`).
|
||||
- Triggered only on failed validation and only when `repair_attempts > 0`.
|
||||
- Strictly bounded by `repair_attempts`.
|
||||
- Uses a narrow repair prompt asking for corrected JSON only.
|
||||
- Runtime validator/repair errors are run errors.
|
||||
|
||||
## 8. Public Contracts
|
||||
|
||||
@@ -147,86 +151,76 @@ Commands:
|
||||
- `scriptorium run`
|
||||
- `scriptorium serve`
|
||||
|
||||
`run`:
|
||||
`run` flags:
|
||||
|
||||
- Required: `--profile-dir`, `--profile-id`, `--input`.
|
||||
- Optional: model/endpoint overrides (`--model`, `--llm-base-url`), vars, output path, schema dir, timeout.
|
||||
- Artifact bytes go to stdout (or `--out` file); summaries/errors go to stderr.
|
||||
- Required: `--profile-dir`, `--prompt-id`, `--input`.
|
||||
- Optional: `--profile-id`, `--var`, `--out`, `--llm-base-url`, `--model`, `--api-key-env`, `--temperature`, `--max-tokens`, `--schema-dir`, `--timeout`.
|
||||
|
||||
`serve`:
|
||||
Current transitional runtime behavior:
|
||||
|
||||
- Required: `--profile-dir`, `--llm-base-url`.
|
||||
- Exposes HTTP run endpoint.
|
||||
- Prompt definitions may provide `default_profile` selection.
|
||||
- Execution-profile loading is deferred; execution settings must currently be supplied via run-time overrides.
|
||||
|
||||
### HTTP
|
||||
|
||||
- Endpoint: `POST /v1/runs`.
|
||||
- Request maps to `RunRequest` (`profile_id`, `inputs`, `vars`, optional `model` override).
|
||||
- Request maps to `RunRequest` with `prompt_id` (required), `inputs`, optional `profile_id`, `vars`, optional execution override (`model` object).
|
||||
- Response includes `artifact`, `validation`, `metadata`, `raw_model_output`.
|
||||
- Validation content failures are represented as `200` with `validation.status=failed`.
|
||||
- Error responses are `{error:{code,message}}` with stable code mapping.
|
||||
- Validation content failures return `200` with failed validation status.
|
||||
- Error response shape: `{ "error": { "code": "...", "message": "..." } }`.
|
||||
|
||||
### Prompt Profile YAML
|
||||
### Prompt Definition YAML
|
||||
|
||||
- `id`, `version`, `expected_inputs`, `templates`, `model_defaults`, `output_format`, `validation`.
|
||||
- Strict YAML decoding (`KnownFields`) rejects unknown fields.
|
||||
- `validation.schema_path` required when `validation_mode=json_schema`.
|
||||
- `validation.repair_attempts` must be non-negative.
|
||||
Current prompt-definition fields:
|
||||
|
||||
### Metadata
|
||||
- `id`, `version`, optional `default_profile`, optional `description`
|
||||
- `inputs[]` with `name`, `required`, optional `content_type`, optional `description`
|
||||
- `templates[]` with `role` and either `content` or `content_file`
|
||||
- `output_format`
|
||||
- `validation` (`format`, `validation_mode`, `schema_path`, `repair_attempts`)
|
||||
|
||||
Current run metadata includes:
|
||||
Strict YAML decoding (`KnownFields`) is enabled.
|
||||
|
||||
- `run_id` (UUID v4)
|
||||
- `profile_id`, `profile_version`, `profile_hash`
|
||||
- `model_name`, `endpoint`, effective `model_params`
|
||||
- `input_hashes`, `prompt_hash`
|
||||
- token usage
|
||||
- start/end timestamps
|
||||
- duration
|
||||
- validation mode/status
|
||||
- repair attempts used
|
||||
### API Key Policy
|
||||
|
||||
- Raw API keys are not accepted in YAML, CLI flags, HTTP body, or domain metadata.
|
||||
- Auth is configured only by env var reference (`api_key_env`), resolved at request time by the LLM adapter.
|
||||
|
||||
## 9. Extension Points (Future Work)
|
||||
|
||||
Future features should plug into existing boundaries, not bypass them.
|
||||
Planned next extensions should reuse current boundaries:
|
||||
|
||||
Candidate extensions:
|
||||
|
||||
- S3 artifact refs via `artifact.Reader` extension.
|
||||
- Token budgeting in usecase/model-target policy layer.
|
||||
- Streaming LLM output via additional `llm.Client` methods/adapters.
|
||||
- Batch execution as a separate use case (not hidden in single-run path).
|
||||
- Additional LLM providers implementing `llm.Client`.
|
||||
- Additional validators/modes in `validate`.
|
||||
- Additional profile repositories (embedded, remote, object storage).
|
||||
|
||||
These are future work, not part of current default behavior.
|
||||
- Execution-profile repository/loader implementation.
|
||||
- Split transitional `internal/profile` into clearer prompt-definition/profile repositories.
|
||||
- S3 artifact refs.
|
||||
- Token budgeting/policy layer.
|
||||
- Streaming generation.
|
||||
- Batch run use case.
|
||||
- Additional provider adapters.
|
||||
- Additional validation modes.
|
||||
|
||||
## 10. Architectural Guardrails
|
||||
|
||||
Contributors should preserve these constraints:
|
||||
|
||||
- No D&D-specific behavior in core Go packages.
|
||||
- No D&D-specific logic in core Go packages.
|
||||
- No orchestration creep into Scriptorium.
|
||||
- No unbounded repair loops.
|
||||
- No silent truncation/omission of rendered inputs or outputs.
|
||||
- Do not log full artifacts/prompts by default.
|
||||
- No silent content truncation/omission.
|
||||
- Do not log full prompts/artifacts by default.
|
||||
- Keep provider-specific wire/SDK details out of domain types.
|
||||
- Keep adapter boundaries explicit and thin.
|
||||
- Keep adapters thin.
|
||||
|
||||
## 11. Testing Strategy
|
||||
|
||||
Protect behavior at boundaries and in usecase flow:
|
||||
Protect these behaviors with focused tests:
|
||||
|
||||
- Profile loading/parsing/validation errors.
|
||||
- Artifact reading for inline/file + hash/content type behavior.
|
||||
- Prompt rendering required inputs/template error behavior.
|
||||
- LLM adapter request/response/error/timeout behavior.
|
||||
- Runner success path and metadata population.
|
||||
- Prompt-definition loading/validation errors.
|
||||
- Artifact loading/hash/content-type behavior.
|
||||
- Prompt rendering required-input and template error paths.
|
||||
- LLM adapter request/response/auth/error/timeout behavior.
|
||||
- Runner success/failure/metadata behavior.
|
||||
- Validation failure raw-output preservation.
|
||||
- Successful/failed/bounded repair flows.
|
||||
- HTTP request mapping, response shape, and error mapping.
|
||||
- CLI parsing helpers, required flags, and output stream separation.
|
||||
- Bounded repair behavior.
|
||||
- HTTP mapping and error mapping.
|
||||
- CLI parsing and output stream separation.
|
||||
|
||||
Prefer focused unit tests and small integration-style tests with fake LLMs.
|
||||
Prefer small unit tests and minimal integration-style tests with fake LLMs.
|
||||
|
||||
Reference in New Issue
Block a user