scriptorium/docs/internal/sources.md

Source Internals

Purpose

This document describes how source packages load prompt definitions, profiles, schemas, and artifacts. The configuration reference owns their user-facing formats and settings. The HTTP API reference owns HTTP-visible artifact outcomes; operations owns deployment handling.

Prompt Definitions

internal/promptdef provides filesystem and fs.FS repositories. Both use internal/filecatalog for recursive YAML discovery, deterministic ordering, display paths, and root cleaning.

Repositories select a prompt by YAML ID and optional version rather than by path. They decode through strict YAML handling, reject duplicate matching definitions, and resolve content_file relative to the definition. The fs.FS implementation resolves content paths inside its source root; absolute paths and traversal outside that root are rejected before file access.

Profiles And Built-Ins

internal/profile provides filesystem, fs.FS, and overlay repositories. internal/profile/builtin exposes embedded assets through the same repository interface.

An overlay asks its primary source first. It falls back only when the primary reports ErrProfileNotFound; invalid YAML, duplicate IDs, validation failures, and raw-key failures are returned rather than hidden by fallback. This makes a custom ID override a built-in ID while retaining errors in the custom source.

The public engine can overlay in-memory profiles ahead of both file-backed and built-in repositories. Profile field definitions, validation ranges, and the built-in catalog remain in the configuration reference.

Schemas

internal/validate supplies StandardValidator for filesystem sources and FSValidator for fs.FS sources. Directory-backed validation loads the named schema path; it does not search directories by basename. fs.FS schema paths are cleaned and checked against their configured root, while a single-file source matches its file base name.

The runner requests a schema document before generation when it needs structured output. JSON and schema mismatches in generated content are validation results; source access, decoding, registration, and compilation failures are operational errors.

Artifacts

internal/artifact composes inline and file readers. The ordinary composite reader used by CLI and the public engine reads file references from the process filesystem. internal/adapter/http provides the restricted public artifact reader for HTTP containment: it combines inline reading with a rooted file reader and optional byte limit. The existing internal restricted composite reader remains a temporary bridge for the current handler and serve wiring; it does not define the HTTP reader's long-term boundary or carry a compatibility promise.

The rooted reader cleans paths and applies lexical containment without resolving symlinks. It checks relative references against the configured root and accepts absolute references only when they remain inside that lexical root. The OS still follows symlinks after that check. The public containment outcome is documented by the HTTP API reference; deployment permissions belong in operations.

Failure Boundaries

Source packages report repository, decoding, duplicate, validation, and read failures to their callers. They do not select public status codes or response schemas. The runner wraps source failures with use-case categories; adapters map them to their own external contract.

Source reads use current filesystem or fs.FS content for each request. These packages create no manifests, checkpoints, or durable run state.

Verification And Change Recipe

Inspect:

• internal/promptdef/repository_test.go
• internal/profile/repository_test.go
• internal/profile/builtin/repository_test.go
• internal/artifact/reader_test.go
• internal/adapter/http/artifact_reader_test.go
• internal/validate/standard_validator_test.go
• internal/usecase/integration_test.go
• engine_test.go

When updating prompt, profile, schema, or built-in assets:

1. keep assets valid for the strict loader and the relevant source boundary;
2. update the configuration reference when a file-format, catalog, or default changes;
3. run focused source and integration tests, including the built-in repository test when embedded assets change; and
4. update this document when discovery, precedence, containment, or failure mechanics change.

The testing policy owns global test sufficiency.