scriptorium/docs/internal/sources.md

# Source Internals

## Purpose

This document covers implemented prompt, profile, schema, artifact, and catalog source behavior. It is for developers changing loaders or source wiring.

Full user-facing YAML and config reference material belongs in `docs/config.md`.

## Prompt Definition Sources

`internal/promptdef` provides directory-backed and `fs.FS` repositories.

Behavior:

- recursively scans `.yaml` and `.yml` files.
- decodes YAML with known-fields checking.
- looks up prompts by YAML `id`, not by path.
- optionally filters by prompt `version`.
- rejects duplicate matching prompt IDs.
- requires `id`, `version`, and at least one message.
- requires each message to set exactly one of `content` or `content_file`.
- resolves filesystem `content_file` values relative to the prompt YAML file.
- resolves `fs.FS` `content_file` values inside the configured source root.
- permits prompt subdirectories only as organization; they are not part of prompt identity.

For `fs.FS` roots, absolute paths and relative traversal outside the source root are rejected by catalog path helpers.

## Profile Sources

`internal/profile` provides directory-backed, `fs.FS`, and overlay repositories. `internal/profile/builtin` embeds built-in profile YAML assets and exposes them through the same repository interface.

Behavior:

- recursively scans `.yaml` and `.yml` files.
- decodes YAML with known-fields checking.
- looks up profiles by YAML `id`, not by path.
- rejects duplicate IDs inside the same source.
- rejects raw `api_key` fields in YAML; file-backed profiles must use `api_key_env`.
- validates required `endpoint` and `model` values.
- validates numeric profile ranges.

Overlay behavior:

- custom profiles are primary.
- built-in profiles are fallback.
- fallback occurs only after a primary `ErrProfileNotFound`.
- primary validation, YAML, duplicate, and raw-key errors are returned directly.
- duplicate IDs across custom and built-in sources are allowed because the custom profile overrides the built-in one.

The public Go facade can add in-memory profiles ahead of file-backed and built-in profiles.

## Schema Sources

`internal/validate` provides:

- `StandardValidator` for filesystem paths.
- `FSValidator` for `fs.FS` roots and single-file public schema sources.

Behavior:

- `json_schema` validation requires a non-empty `schema_path`.
- filesystem schema paths resolve relative to `schema_dir` unless absolute.
- directory-backed schema lookup uses the explicit `schema_path`; it does not search recursively by basename.
- `fs.FS` schema paths must remain inside the configured source root.
- single-file schema sources match by the configured file base name.
- schema documents are loaded before the LLM call for structured output.
- JSON parse failures are validation content failures.
- schema access, decode, registration, and compile failures are runtime validation errors.

## Artifact Sources

`internal/artifact` supports two input artifact reference types:

- `inline`
- `file`

Inline behavior:

- requires a non-empty body.
- produces text/plain artifacts.
- hashes the body bytes.

Direct file behavior:

- used by CLI `run`, CLI `render`, and the public Go facade.
- requires a non-empty URI.
- reads from the process filesystem without HTTP artifact-root restrictions.
- infers content type from file extension, defaulting to text/plain.

Restricted file behavior:

- used by HTTP `serve`.
- allows inline artifacts even when no artifact root is configured.
- denies file artifacts when no artifact root is configured.
- resolves relative file URIs against `server.artifact_root`.
- accepts absolute file URIs only when they pass containment checks.
- applies `server.max_artifact_bytes` when configured.

Restricted containment is lexical. It cleans paths and checks the relative path against the configured root; it does not resolve symlinks. Symlinks inside the root are followed by the operating system, including symlinks that target files outside the root.

## Catalog Helpers

`internal/filecatalog` centralizes shared source helpers:

- recursive YAML discovery for filesystem and `fs.FS` roots.
- deterministic sorting.
- `.yaml` and `.yml` filtering.
- display paths for diagnostics.
- YAML file stems.
- `fs.FS` root cleaning and containment checks.

Repository code should use these helpers instead of reimplementing path traversal and containment rules.

## Failure Behavior

Common source failures:

- missing prompt/profile/schema/artifact files.
- invalid YAML or JSON.
- unknown YAML fields.
- duplicate prompt or profile IDs.
- prompt/profile validation errors.
- raw API key fields in profile YAML.
- unsupported artifact reference type.
- missing inline body or file URI.
- artifact outside HTTP root.
- artifact exceeding HTTP size limit.
- schema load or compile failure.

Prompt/profile repository lookup errors are mapped by adapters separately from runtime runner errors. Validation content failures remain result state; source and schema runtime failures return errors.

## State And Manifests

Source packages do not persist run state.

- No manifests are read or written.
- No source package implements skip or resume behavior.
- Source reads reflect the current filesystem or `fs.FS` state for each request.

## Tests To Inspect

- `internal/promptdef/repository_test.go`
- `internal/profile/repository_test.go`
- `internal/profile/builtin/repository_test.go`
- `internal/artifact/reader_test.go`
- `internal/validate/standard_validator_test.go`
- `internal/usecase/integration_test.go`
- `engine_test.go`

## Architectural Invariants

- Prompt/profile identity comes from YAML `id`.
- External YAML decoding remains strict.
- File-backed profile YAML never accepts raw API key values.
- Built-in profiles are fallback, not a replacement for custom source validation.
- HTTP file artifacts remain rooted by lexical containment.
- Schema runtime failures remain errors, while JSON/schema content mismatches remain validation results.