Align internal documentation with architecture

docs/internal/adapters.md

@@ -1,163 +1,81 @@
-# Adapter And Repository Internals
+# Adapter Internals
 
 ## Purpose
 
-This document describes implemented adapter/repository boundaries and their current behavior.
+Adapters translate external interfaces into domain requests and translate domain results back out. They wire dependencies, apply app config, and own IO concerns, but they do not make runner decisions.
+
+Source-loading behavior belongs in `docs/internal/sources.md`. User-facing CLI, HTTP, and package contracts belong in `docs/cli.md`, `docs/api.md`, and `docs/consumers/pkg-scriptorium.md`.
 
 ## Adapter Map
 
-- `internal/adapter/cli`: CLI command parsing, app wiring, stdout/stderr handling, exit codes.
+- `cmd/scriptorium`: process entrypoint.
-- `internal/adapter/http`: HTTP request/response mapping for `POST /v1/runs`.
+- `internal/adapter/cli`: command parsing, config handoff, runner construction, stdout/stderr, exit codes.
-- root package `scriptorium`: public Go library facade for preparing and running prompt requests.
+- `internal/adapter/http`: `POST /v1/runs` request/response mapping and HTTP error/status mapping.
-- `internal/promptdef`: filesystem and `fs.FS` prompt-definition repositories.
+- root package `scriptorium`: public Go facade over internal runner types and dependencies.
-- `internal/profile`: filesystem, `fs.FS`, and overlay execution-profile repositories.
+
-- `internal/filecatalog`: shared YAML discovery, display-path, and `fs.FS` source-root resolution helpers.
+Supporting implementation packages used during adapter wiring:
-- `internal/profile/builtin`: embedded built-in execution-profile repository.
+
-- `internal/artifact`: input artifact reader.
+- `internal/config`
-- `internal/prompt`: Go-template renderer.
+- `internal/defaults`
-- `internal/llm`: OpenAI-compatible LLM client implementation.
+- `internal/format`
-- `internal/validate`: filesystem and `fs.FS` output validators.
+- `internal/llm`
-- `internal/format`: prepared-run formatters for `render` output.
+- `internal/prompt`
 
 ## Inputs And Outputs
 
 CLI adapter:
 
-- Input: process args, filesystem config/assets, environment.
+- Input: process args, optional config file, filesystem sources, environment variables.
-- Output: exit code, stdout artifact/prepared output, stderr summaries/errors.
+- Output: process exit code, stdout artifact/prepared output, stderr summaries and errors.
-- `run` summaries include cache usage counters only when either parsed cache counter is non-zero.
 
 HTTP adapter:
 
-- Input: JSON request body (`runRequestDTO`).
+- Input: HTTP request method/path/headers/body for `POST /v1/runs`.
-- Output: JSON success/error body with mapped status codes.
+- Output: JSON success or error body with mapped status code.
-- Success metadata includes token usage plus cache usage counters.
 
-Public library facade:
+Public Go facade:
 
-- Input: typed `scriptorium.RunRequest` values.
+- Input: typed `scriptorium.Config`, `Option`, and `RunRequest` values.
 - Output: typed `PreparedRun` and `RunResult` values plus public sentinel errors.
-- Custom LLM behavior is injected with `WithLLMClient`; otherwise the default OpenAI-compatible client is used.
-- `RunRequest.APIKey` is a request-scoped Go value only; it is converted into internal execution state for LLM generation and stripped from public result types.
-- Prompt, profile, and schema source options can use directories, single files, or `fs.FS` roots. Explicit source options override the matching `Config` directory field.
-- Public types are facade types converted at the package boundary; internal domain types remain internal.
-
-Prompt/profile repositories:
-
-- Input: prompt/profile YAML files under configured directories or `fs.FS` roots.
-- Output: normalized domain definitions/profiles or typed errors.
-- Shared YAML catalog helpers provide recursive discovery, extension filtering, deterministic ordering, file stems, `fs.FS` display paths, and source-root containment checks.
-- Single-file public sources are represented as `fs.FS` roots containing one YAML file; lookup still uses YAML `id` values.
-
-Profile repository composition:
-
-- Built-in profiles are embedded and loaded through the same profile validation rules as filesystem profiles.
-- When no custom profile directory is configured, the runner receives the built-in profile repository.
-- When a custom profile directory/file/`fs.FS` source is configured, the runner receives an overlay repository with custom profiles as primary and built-ins as fallback.
-- Overlay lookup falls back only after custom profile-not-found errors; custom load/validation/raw-key errors are returned directly.
-
-Artifact reader:
-
-- Input: `domain.ArtifactRef`.
-- Output: loaded `domain.Artifact`.
-
-LLM adapter:
-
-- Input: `domain.GenerateRequest`.
-- Output: `domain.GenerateResponse`.
-- Direct API-key values are preferred when present; otherwise `api_key_env` is resolved from the process environment.
-
-Validator:
-
-- Input: artifact body + output contract.
-- Output: validation result or runtime validation error.
-- Schema documents may be loaded from a directory, single file, or `fs.FS` root in the public package. CLI and HTTP continue to use directory-backed schema loading.
 
 ## Boundaries
 
-- Adapters convert external representations to domain requests and back.
+- Adapters convert external shapes to `domain.RunRequest` and back.
-- Use-case decisions remain in `internal/usecase`.
+- Runner orchestration remains in `internal/usecase`.
-- External dependency details stay scoped to adapter packages.
+- Prompt/profile/schema/artifact source rules remain in repository, validator, and artifact packages.
+- LLM provider request serialization remains in `internal/llm`.
+- Public package types are facade types; internal domain types do not leak across the package boundary.
 
 ## Config Fields Used
 
-Primary app settings consumed by adapters:
+Adapter app settings:
 
 - `prompt_dir`
-- `profile_dir` (optional custom profile source)
+- `profile_dir`
 - `schema_dir`
 - `server.addr`
-- `server.artifact_root` (HTTP `serve` file input root)
+- `server.artifact_root`
 - `server.max_request_bytes`
 - `server.max_artifact_bytes`
 - `server.max_response_bytes`
 - `defaults.render_format`
 
-Execution profile/request settings used through runner:
+Execution request/profile settings passed through the runner:
 
-- `endpoint`, `model`, `temperature`, `max_tokens`, `top_p`, `timeout_seconds`, `service_tier`, `api_key_env`, `reasoning_effort`, `extra_params`
+- `endpoint`
-- CLI and HTTP request adapters preserve caller intent for numeric runtime overrides. Omitted values remain absent; explicit zero values are mapped as explicit overrides.
+- `model`
-- HTTP `extra_params` accepts JSON-compatible values and maps them to domain request overrides without provider-specific adapter logic.
+- `temperature`
+- `max_tokens`
+- `top_p`
+- `timeout_seconds`
+- `service_tier`
+- `api_key_env`
+- `reasoning_effort`
+- `extra_params`
 
-## External Dependencies
+CLI and HTTP preserve numeric override presence so omitted values and explicit zero values remain distinct.
 
-- YAML decoding: `gopkg.in/yaml.v3` (strict known-fields mode in config/prompt/profile loaders).
+## CLI Adapter
-- JSON Schema validation: `github.com/santhosh-tekuri/jsonschema/v6`.
-- HTTP client/server: Go standard library.
-
-## Failure Behavior
-
-Strict decoding and input checks:
-
-- config/prompt/profile loaders reject unknown YAML fields.
-- prompt/profile repositories scan nested subdirectories recursively.
-- prompt/profile lookup uses YAML `id` values; subdirectory paths are organizational only.
-- prompt `content_file` paths resolve relative to the prompt YAML file within the same source.
-- `fs.FS` prompt `content_file` paths and schema paths must remain inside the configured source root; absolute paths and relative traversal outside the root are rejected.
-- duplicate prompt/profile IDs are invalid and fail instead of using first-match behavior.
-- duplicate profile IDs across custom and built-in sources are allowed; the custom source overrides the built-in profile.
-- HTTP DTO decoder rejects unknown JSON fields.
-- raw API key payload fields are rejected by strict decoding in profile/http paths.
-
-Artifact refs:
-
-- Supported reference types: `inline`, `file`.
-- Unsupported types return `ErrUnsupportedRefType`.
-- CLI `run` and `render` use direct filesystem file reads for `file` references.
-- HTTP `serve` uses a restricted artifact reader: `inline` references work without a root, while `file` references require `server.artifact_root` or `--artifact-root` and must pass lexical containment checks against that root.
-- HTTP `serve` applies request-body, file-artifact, and encoded-response size limits. CLI `run` and `render` do not use these HTTP limits.
-- HTTP file paths are resolved with clean absolute paths and lexical containment checks, not string-prefix checks.
-- Symlinks inside the root are followed by the operating system, including symlinks that point outside the root; the configured root must not be writable by untrusted users.
-
-LLM adapter:
-
-- endpoint appends `/chat/completions`.
-- rendered messages without cache control serialize with string `content`.
-- rendered messages with cache control serialize as one text content block with `cache_control`.
-- non-empty `reasoning_effort` serializes as a top-level provider request field.
-- `extra_params` flatten into provider-specific top-level JSON request fields.
-- reserved `extra_params` keys are rejected before the provider call: `model`, `session_id`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, `reasoning_effort`, and `response_format`.
-- empty `extra_params` keys and values that cannot be JSON-encoded are rejected before the provider call.
-- compatible cache usage response fields are parsed into domain token usage.
-- non-2xx responses map to request failure errors.
-- malformed responses (including missing/empty first choice content) are errors.
-- direct API-key values are never serialized in provider request bodies.
-
-Validator:
-
-- `basic`, `json`, `json_schema` content failures return `ValidationFailed` results.
-- schema load/compile/path failures are runtime errors.
-- directory-backed schema lookup uses explicit `schema_path` values relative to `schema_dir`; it does not recursively search by basename.
-- `fs.FS` schema lookup uses explicit `schema_path` values inside the configured source root. Single-file public schema sources match by base name.
-
-HTTP error mapping:
-
-- maps domain/use-case errors to stable HTTP code + error code/message.
-- maps request, artifact, and response size failures to `413` errors.
-- distinguishes missing profile selection and missing `api_key_env` variable using stable use-case sentinel errors.
-- avoids returning internal wrapped-cause details in response payload.
-
-## CLI Adapter Semantics
 
 Implemented commands:
 
@@ -165,30 +83,73 @@ Implemented commands:
 - `render`
 - `serve`
 
-Behavior highlights:
+Behavior:
 
-- `run` exit `2` indicates validation failed after generation.
+- `run` constructs a runner with direct filesystem artifact reading and calls `Runner.Run`.
-- `render` does not call the LLM.
+- `render` constructs a runner and calls `Runner.Prepare`; it does not call the LLM.
-- `serve` exposes HTTP handler only; no built-in auth.
+- `serve` constructs a restricted artifact reader and HTTP handler, then starts an unauthenticated HTTP server.
-- `render` supports `--format text|json`; `render` does not expose `--schema-dir`.
+- `run` exits `2` when generation succeeds but validation fails.
-- deprecated aliases `--prompt-id` and `--profile-id` are still accepted.
+- parse, runtime, and output-write errors exit `1`.
+- deprecated `--prompt-id` and `--profile-id` aliases are accepted.
 
-## Tests To Inspect Before Changing
+## HTTP Adapter
+
+Behavior:
+
+- Accepts only `POST /v1/runs`.
+- Decodes JSON strictly and rejects unknown fields and trailing JSON tokens.
+- Rejects empty `prompt_id` and empty `inputs` before calling the runner.
+- Does not accept raw API key values in the request body.
+- Returns validation failures as `200` responses with failed validation details.
+- Maps request-body, artifact, and encoded-response size failures to `413`.
+- Maps domain and repository errors to stable error codes without returning wrapped internal cause text.
+
+The HTTP adapter has no built-in authentication or authorization. Deployment controls must be provided outside the process.
+
+## Public Go Facade
+
+Behavior:
+
+- `NewEngine` wires the same default runner components as CLI/HTTP unless options override them.
+- Prompt, profile, and schema sources may come from directories, single files, or `fs.FS` roots.
+- `WithProfiles` adds in-memory profiles ahead of file-backed and built-in profiles.
+- `WithLLMClient` injects custom model behavior.
+- `RunRequest.APIKey` is request-scoped and direct; it is used only for generation and is stripped from public results.
+- internal errors are mapped to public sentinels in `errors.go`.
+
+## Failure Behavior
+
+Adapters should:
+
+- keep external error payloads concise and stable.
+- avoid leaking raw secret values.
+- use sentinels and typed errors for mapping.
+- preserve strict external input decoding.
+- keep validation content failures distinct from runtime errors.
+
+CLI writes human-readable summaries to stderr. HTTP writes JSON error envelopes. The public Go facade returns typed errors.
+
+## State And Manifests
+
+Adapters do not add durable run state.
+
+- No adapter writes run manifests.
+- No adapter implements checkpoint, skip, or resume behavior.
+- CLI output files are caller-selected artifacts, not internal state.
+
+## Tests To Inspect
 
 - `internal/adapter/cli/run_test.go`
 - `internal/adapter/http/handler_test.go`
-- `internal/promptdef/repository_test.go`
+- `engine_test.go`
-- `internal/profile/repository_test.go`
-- `internal/artifact/reader_test.go`
-- `internal/prompt/renderer_test.go`
-- `internal/llm/openai_compatible_client_test.go`
-- `internal/validate/standard_validator_test.go`
 - `internal/format/prepared_run_test.go`
+- `internal/llm/openai_compatible_client_test.go`
 
 ## Architectural Invariants
 
-- Adapter packages do not own runner decision logic.
+- Adapter packages stay thin and translation-focused.
-- External request/response strictness is part of contract stability.
+- App config is resolved before dependency construction.
-- Prepared-render output never includes resolved API key values.
+- External input strictness is part of contract stability.
-- Outbound OpenAI-compatible request includes currently serialized first-class fields (`model`, optional `session_id`, `messages`, optional `temperature`, `max_tokens`, `top_p`, optional `service_tier`, optional `reasoning_effort`, optional `response_format`) plus validated `extra_params` flattened as provider-specific top-level fields.
+- CLI and HTTP construct runners without a repairer.
-- Outbound cache control is message-level only; no top-level cache-control field is serialized.
+- HTTP endpoint details remain canonical in `docs/api.md`.
+- Public Go package details remain canonical in `docs/consumers/pkg-scriptorium.md`.

docs/internal/runner.md

@@ -2,29 +2,32 @@
 
 ## Purpose
 
-`internal/usecase.Runner` is the core use case orchestrator for prompt preparation and execution.
+`internal/usecase.Runner` is the core prompt-execution orchestrator. It prepares prompt requests, calls the configured LLM client for `Run`, validates generated output, and returns domain results.
 
-It owns request validation, prompt/profile resolution, runtime-parameter merge, artifact loading, prompt rendering, structured-output setup, LLM invocation, output validation, and result metadata.
+Transport parsing, DTOs, CLI output, HTTP status mapping, and public package type conversion belong outside the runner.
 
 ## Inputs And Outputs
 
-Primary input type:
+Primary inputs:
 
 - `domain.RunRequest`
+- repositories/readers/renderers/validators injected at construction
+- `context.Context` for cancellation
 
-Primary output types:
+Primary outputs:
 
 - `domain.PreparedRun` from `Prepare`
 - `domain.RunResult` from `Run`
+- wrapped sentinel errors for adapter mapping
 
 LLM boundary types:
 
 - `domain.GenerateRequest`
 - `domain.GenerateResponse`
 
-## Boundaries
+## Dependencies
 
-`Runner` coordinates the following interfaces:
+`Runner` depends on package interfaces instead of concrete adapter types:
 
 - `promptdef.Repository`
 - `profile.Repository`
@@ -34,136 +37,110 @@ LLM boundary types:
 - `validate.Validator`
 - optional `usecase.OutputRepairer`
 
-Transport concerns (CLI flags, HTTP DTO parsing, status-code mapping) stay outside runner.
+The CLI, HTTP adapter, and public Go package construct these dependencies and pass them in.
 
-## Config Fields Used
+## Config Fields
 
-`Runner` does not read app config files directly.
+`Runner` does not read app config files. Effective behavior is determined by injected dependencies and the `domain.RunRequest`.
 
-It receives fully constructed repositories/readers/validators from adapters. Effective behavior depends on adapter wiring, including:
+Adapter wiring commonly reflects these app config fields:
 
-- prompt/profile directories
+- `prompt_dir`
-- schema base directory
+- `profile_dir`
-- selected profile/runtime overrides in request
+- `schema_dir`
+- `server.artifact_root`
+- HTTP request/artifact/response size limits
 
-## External Adapters Used
+Runtime model settings are resolved from the selected profile plus request overrides.
-
-`Runner` works with adapter implementations via interfaces. Current wiring from CLI/HTTP uses:
-
-- filesystem prompt/profile repositories
-- composite artifact reader
-- Go-template prompt renderer
-- OpenAI-compatible LLM client
-- standard validator
-
-## State And Resume Behavior
-
-`Runner` is stateless across requests.
-
-- No durable run-state storage.
-- No built-in resume/skip checkpoints.
-- Each `Run`/`Prepare` executes from request inputs and current repositories.
-
-## Failure Behavior
-
-Primary runner error classes:
-
-- `ErrInvalidRequest`: invalid run request envelope.
-- `ErrProfileRequired`: specific invalid-request reason when neither request `profile_id` nor prompt `default_profile` is available.
-- `ErrAPIKeyEnvMissing`: specific invalid-request reason when `api_key_env` is set but the named environment variable is unset/empty.
-- `ErrPromptLoad`: prompt-definition repository load failures.
-- `ErrProfileLoad`: execution-profile repository load failures.
-- `ErrArtifactLoad`: artifact read failures.
-- `ErrPromptRender`: template render failures.
-- `ErrLLMGenerate`: outbound model request failures.
-- `ErrValidation`: validation runtime failures (including structured-output schema load/compile failures).
-
-Reason sentinel behavior:
-
-- `ErrProfileRequired` and `ErrAPIKeyEnvMissing` are wrapped with `ErrInvalidRequest`.
-- Adapters can use `errors.Is` for stable reason mapping without matching runner prose.
-
-Validation content failures are not run errors:
-
-- `Run` can succeed with `Validation.Status == failed`.
-- CLI maps this to exit code `2`.
-- HTTP returns `200` with failed validation details.
 
 ## Prepare Flow
 
-`Prepare` performs:
+`Prepare`:
 
-1. validate request basics (prompt ID present).
+1. requires a non-empty prompt ID.
-2. load prompt definition by ID/version.
+2. loads the prompt definition and computes its hash.
-3. select profile ID:
+3. selects the profile from request `profile_id`, then prompt `default_profile`.
-   - explicit request profile ID
+4. loads the selected execution profile.
-   - prompt `default_profile`
+5. merges built-in execution defaults, profile values, and request overrides.
-   - otherwise return an invalid request with `ErrProfileRequired`
+6. applies request-scoped direct API key values for public Go callers.
-4. load execution profile.
+7. validates endpoint, model, and credential requirements.
-5. merge effective runtime target:
+8. resolves the output contract and JSON Schema document when required.
-   - built-in execution defaults
+9. reads input artifacts.
-   - selected profile values
+10. renders prompt messages and hashes the rendered prompt.
-   - request overrides
+11. returns a prepared run without calling the LLM.
-   - request numeric overrides are presence-aware, so omitted values preserve the current effective value and explicit zero values override it
-6. verify credentials when the effective target names `api_key_env`:
-   - a request-scoped direct API key satisfies the credential requirement
-   - otherwise a missing/empty env value returns an invalid request with `ErrAPIKeyEnvMissing`
-   - only the environment-variable name is returned in public output; secret values are never returned
-7. resolve output contract and structured-output schema payload when `json_schema` mode is active.
-8. read input artifacts.
-9. render prompt messages, including any normalized message cache-control metadata.
-10. compute prompt/input/render hashes and return `PreparedRun`.
 
-`rendered_prompt_hash` includes cache-control metadata when present because it affects the outbound provider request. Prompts without cache control keep the role/content hash behavior.
+Numeric request overrides are presence-aware: omitted values preserve the current effective value, while explicit zero values are real overrides.
-
-`Prepare` does not call the LLM.
-
-Runtime target notes:
-
-- Profile `extra_params` and request `extra_params` carry JSON-compatible values through prepared output, run metadata, and `domain.GenerateRequest.Target`.
-- The OpenAI-compatible client serializes non-empty `reasoning_effort` as a top-level provider request field.
-- The OpenAI-compatible client flattens `extra_params` into provider-specific top-level JSON request fields.
-- Empty `extra_params` keys, reserved outbound field names, and values that cannot be JSON-encoded fail before the provider request.
-- Resolved API-key values are never serialized in prepared/run output, public results, logs, or HTTP responses.
-- Public direct API-key values are carried only far enough to call the configured LLM client and are excluded from JSON/YAML serialization.
 
 ## Run Flow
 
-`Run` performs:
+`Run`:
 
-1. generate run ID.
+1. creates a run ID and start timestamp.
-2. call `Prepare`.
+2. calls `Prepare`.
-3. call LLM with prepared messages/effective target/structured-output spec.
+3. calls the injected LLM client with rendered messages, effective target, target presence, and structured-output settings.
-4. build output artifact content type from output format.
+4. builds the output artifact.
-5. validate output.
+5. validates the output.
-6. optionally attempt bounded repair when repairer is injected and contract allows it.
+6. optionally attempts bounded repair when a repairer is injected and the contract permits repair.
-7. return `RunResult` with artifact, raw output, validation, hashes, profile/model metadata, token/cache usage, and timestamps.
+7. returns the run result with artifact, raw output, validation, hashes, selected profile/model metadata, usage, and timing.
 
-## Repair Hook Boundary
+`Run` must reuse `Prepare`; prepare logic should not be duplicated elsewhere.
 
-Repair attempts occur only when all are true:
+## Validation And Repair
 
-- repairer is injected
+Validation content failures are returned as successful run results with `Validation.Status == failed`. They are not runtime errors.
-- `repair_attempts > 0`
+
+Validation runtime failures, such as schema load or compile errors, return `ErrValidation`.
+
+Repair attempts occur only when all conditions are true:
+
+- a repairer is injected
+- `repair_attempts` is greater than zero
 - validation status is `failed`
 - validation mode is `json` or `json_schema`
 
-Current production wiring boundary:
+CLI and HTTP wiring call `usecase.NewRunner(...)`, which does not inject a repairer. Normal CLI and HTTP execution therefore does not repair invalid output.
 
-- CLI and HTTP adapters call `usecase.NewRunner(...)` (no repairer argument).
+## Failure Behavior
-- Therefore normal CLI/HTTP execution does not perform repair attempts today.
 
-## Tests To Inspect Before Changing
+Stable runner sentinels include:
+
+- `ErrInvalidRequest`
+- `ErrProfileRequired`
+- `ErrAPIKeyEnvMissing`
+- `ErrAPIKeyRequired`
+- `ErrPromptLoad`
+- `ErrProfileLoad`
+- `ErrArtifactLoad`
+- `ErrPromptRender`
+- `ErrLLMGenerate`
+- `ErrValidation`
+
+Adapters should use `errors.Is` against sentinels and lower-level repository errors instead of matching message text.
+
+Secret values must not appear in prepared output, run results, logs, HTTP responses, or serialized public package results. The effective API-key environment-variable name may appear.
+
+## State And Manifests
+
+The runner is stateless across requests.
+
+- No durable run store.
+- No manifest files.
+- No checkpoint, skip, or resume behavior.
+- Recovery is a new request after correcting inputs, config, or environment.
+
+## Tests To Inspect
 
 - `internal/usecase/runner_test.go`
 - `internal/usecase/integration_test.go`
+- `engine_test.go`
 - `internal/adapter/cli/run_test.go`
 - `internal/adapter/http/handler_test.go`
 
 ## Architectural Invariants
 
-- `Run` reuses `Prepare`; prepare logic is not duplicated.
+- Use-case decisions stay in `internal/usecase`.
-- Effective API-key environment-variable name may appear; resolved secret value must not.
+- `Run` reuses `Prepare`.
-- Structured-output schema document must load before LLM call for `json_schema` mode.
+- Prompt/profile/artifact/schema loading remains behind injected boundaries.
+- Validation content failures are result state; validation runtime failures are errors.
 - Repair loops are bounded by `repair_attempts` and repairer presence.
-- Runner stays transport-agnostic.
+- Resolved secret values are never serialized or emitted.

docs/internal/sources.md

@@ -0,0 +1,157 @@
+# Source Internals
+
+## Purpose
+
+This document covers implemented prompt, profile, schema, artifact, and catalog source behavior. It is for developers changing loaders or source wiring.
+
+Full user-facing YAML and config reference material belongs in `docs/config.md`.
+
+## Prompt Definition Sources
+
+`internal/promptdef` provides directory-backed and `fs.FS` repositories.
+
+Behavior:
+
+- recursively scans `.yaml` and `.yml` files.
+- decodes YAML with known-fields checking.
+- looks up prompts by YAML `id`, not by path.
+- optionally filters by prompt `version`.
+- rejects duplicate matching prompt IDs.
+- requires `id`, `version`, and at least one message.
+- requires each message to set exactly one of `content` or `content_file`.
+- resolves filesystem `content_file` values relative to the prompt YAML file.
+- resolves `fs.FS` `content_file` values inside the configured source root.
+- permits prompt subdirectories only as organization; they are not part of prompt identity.
+
+For `fs.FS` roots, absolute paths and relative traversal outside the source root are rejected by catalog path helpers.
+
+## Profile Sources
+
+`internal/profile` provides directory-backed, `fs.FS`, and overlay repositories. `internal/profile/builtin` embeds built-in profile YAML assets and exposes them through the same repository interface.
+
+Behavior:
+
+- recursively scans `.yaml` and `.yml` files.
+- decodes YAML with known-fields checking.
+- looks up profiles by YAML `id`, not by path.
+- rejects duplicate IDs inside the same source.
+- rejects raw `api_key` fields in YAML; file-backed profiles must use `api_key_env`.
+- validates required `endpoint` and `model` values.
+- validates numeric profile ranges.
+
+Overlay behavior:
+
+- custom profiles are primary.
+- built-in profiles are fallback.
+- fallback occurs only after a primary `ErrProfileNotFound`.
+- primary validation, YAML, duplicate, and raw-key errors are returned directly.
+- duplicate IDs across custom and built-in sources are allowed because the custom profile overrides the built-in one.
+
+The public Go facade can add in-memory profiles ahead of file-backed and built-in profiles.
+
+## Schema Sources
+
+`internal/validate` provides:
+
+- `StandardValidator` for filesystem paths.
+- `FSValidator` for `fs.FS` roots and single-file public schema sources.
+
+Behavior:
+
+- `json_schema` validation requires a non-empty `schema_path`.
+- filesystem schema paths resolve relative to `schema_dir` unless absolute.
+- directory-backed schema lookup uses the explicit `schema_path`; it does not search recursively by basename.
+- `fs.FS` schema paths must remain inside the configured source root.
+- single-file schema sources match by the configured file base name.
+- schema documents are loaded before the LLM call for structured output.
+- JSON parse failures are validation content failures.
+- schema access, decode, registration, and compile failures are runtime validation errors.
+
+## Artifact Sources
+
+`internal/artifact` supports two input artifact reference types:
+
+- `inline`
+- `file`
+
+Inline behavior:
+
+- requires a non-empty body.
+- produces text/plain artifacts.
+- hashes the body bytes.
+
+Direct file behavior:
+
+- used by CLI `run`, CLI `render`, and the public Go facade.
+- requires a non-empty URI.
+- reads from the process filesystem without HTTP artifact-root restrictions.
+- infers content type from file extension, defaulting to text/plain.
+
+Restricted file behavior:
+
+- used by HTTP `serve`.
+- allows inline artifacts even when no artifact root is configured.
+- denies file artifacts when no artifact root is configured.
+- resolves relative file URIs against `server.artifact_root`.
+- accepts absolute file URIs only when they pass containment checks.
+- applies `server.max_artifact_bytes` when configured.
+
+Restricted containment is lexical. It cleans paths and checks the relative path against the configured root; it does not resolve symlinks. Symlinks inside the root are followed by the operating system, including symlinks that target files outside the root.
+
+## Catalog Helpers
+
+`internal/filecatalog` centralizes shared source helpers:
+
+- recursive YAML discovery for filesystem and `fs.FS` roots.
+- deterministic sorting.
+- `.yaml` and `.yml` filtering.
+- display paths for diagnostics.
+- YAML file stems.
+- `fs.FS` root cleaning and containment checks.
+
+Repository code should use these helpers instead of reimplementing path traversal and containment rules.
+
+## Failure Behavior
+
+Common source failures:
+
+- missing prompt/profile/schema/artifact files.
+- invalid YAML or JSON.
+- unknown YAML fields.
+- duplicate prompt or profile IDs.
+- prompt/profile validation errors.
+- raw API key fields in profile YAML.
+- unsupported artifact reference type.
+- missing inline body or file URI.
+- artifact outside HTTP root.
+- artifact exceeding HTTP size limit.
+- schema load or compile failure.
+
+Prompt/profile repository lookup errors are mapped by adapters separately from runtime runner errors. Validation content failures remain result state; source and schema runtime failures return errors.
+
+## State And Manifests
+
+Source packages do not persist run state.
+
+- No manifests are read or written.
+- No source package implements skip or resume behavior.
+- Source reads reflect the current filesystem or `fs.FS` state for each request.
+
+## Tests To Inspect
+
+- `internal/promptdef/repository_test.go`
+- `internal/profile/repository_test.go`
+- `internal/profile/builtin/repository_test.go`
+- `internal/artifact/reader_test.go`
+- `internal/validate/standard_validator_test.go`
+- `internal/usecase/integration_test.go`
+- `engine_test.go`
+
+## Architectural Invariants
+
+- Prompt/profile identity comes from YAML `id`.
+- External YAML decoding remains strict.
+- File-backed profile YAML never accepts raw API key values.
+- Built-in profiles are fallback, not a replacement for custom source validation.
+- HTTP file artifacts remain rooted by lexical containment.
+- Schema runtime failures remain errors, while JSON/schema content mismatches remain validation results.

docs/policy/architecture.md

@@ -11,6 +11,7 @@ Scriptorium is a narrow prompt-execution application with three entry paths:
 - CLI `run`
 - CLI `render`
 - HTTP `POST /v1/runs` through `serve`
+- public Go package `gitea.maximumdirect.net/eric/scriptorium`
 
 Domain behavior is centralized in `internal/usecase` and `internal/domain`.
 
@@ -26,6 +27,7 @@ Domain behavior is centralized in `internal/usecase` and `internal/domain`.
 
 Current package map:
 
+- root package `scriptorium`: public Go facade over engine construction, source options, request/result types, and error mapping.
 - `cmd/scriptorium`: process entrypoint.
 - `internal/adapter/cli`: command parsing, app wiring for CLI commands, output behavior.
 - `internal/adapter/http`: HTTP DTO mapping and error/status mapping.
@@ -34,7 +36,9 @@ Current package map:
 - `internal/domain`: core request/result and contract types.
 - `internal/usecase`: `Runner` prepare/run orchestration and repair-hook boundary.
 - `internal/promptdef`: filesystem prompt-definition repository.
-- `internal/profile`: filesystem execution-profile repository.
+- `internal/profile`: filesystem, `fs.FS`, and overlay execution-profile repositories.
+- `internal/profile/builtin`: embedded built-in execution profiles.
+- `internal/filecatalog`: shared YAML discovery and `fs.FS` source helpers.
 - `internal/artifact`: artifact reference readers.
 - `internal/prompt`: template renderer.
 - `internal/llm`: provider-neutral LLM client interface and OpenAI-compatible implementation.
@@ -45,6 +49,7 @@ Detailed component behavior is documented in:
 
 - `docs/internal/runner.md`
 - `docs/internal/adapters.md`
+- `docs/internal/sources.md`
 
 ## Configuration And Precedence
 
@@ -69,9 +74,10 @@ Scriptorium has no durable run-state store.
 
 Current external contracts:
 
-- inbound HTTP contract: `POST /v1/runs`
+- inbound HTTP contract: `POST /v1/runs`, documented canonically in `docs/api.md`
 - outbound model contract: OpenAI-compatible chat completions subset
 - subprocess contract for integrators: CLI `run`/`render`
+- public Go package contract: `docs/consumers/pkg-scriptorium.md`
 
 Integration docs belong under `docs/integrations/`.
 

docs/policy/development.md

@@ -4,6 +4,7 @@ This document defines contributor workflow for Scriptorium.
 
 ## Repository Layout
 
+- root package `scriptorium`: public Go facade, options, types, and error mapping.
 - `cmd/scriptorium`: application entrypoint.
 - `internal/domain`: core contracts.
 - `internal/usecase`: runner orchestration.
@@ -13,6 +14,8 @@ This document defines contributor workflow for Scriptorium.
 - `internal/defaults`: default constants.
 - `internal/promptdef`: prompt-definition repository.
 - `internal/profile`: execution-profile repository.
+- `internal/profile/builtin`: embedded built-in execution profiles.
+- `internal/filecatalog`: shared source discovery and path helpers.
 - `internal/artifact`: artifact readers.
 - `internal/prompt`: prompt rendering.
 - `internal/llm`: LLM client interface and OpenAI-compatible implementation.
@@ -38,7 +41,9 @@ go test ./...
 Targeted test runs commonly used during changes:
 
 ```bash
+go test .
 go test ./internal/adapter/cli ./internal/adapter/http ./internal/usecase
+go test ./internal/...
 ```
 
 ## Coding Conventions
@@ -82,7 +87,8 @@ go test ./internal/adapter/cli ./internal/adapter/http ./internal/usecase
 3. Keep business decisions in `internal/usecase`.
 4. Add focused adapter tests for mapping, parse, and error behavior.
 5. Document the new/changed boundary in `docs/internal/adapters.md`.
-6. If external contract changes, update `docs/integrations/` in the same change.
+6. If source-loading behavior changes, update `docs/internal/sources.md`.
+7. If an external contract changes, update the canonical public or integration doc in the same change.
 
 ## How To Update Prompt/Profile/Schema Assets
 
@@ -99,5 +105,6 @@ When behavior changes:
 2. Keep non-roadmap docs limited to implemented behavior.
 3. Update links after file moves/renames.
 4. Re-run relevant tests and smoke commands.
+5. For internal boundary docs, check references with `rg "docs/internal|internal/sources" docs/policy docs/internal`.
 
 Docs work is complete only when code/tests/examples/docs agree.