scriptorium/docs/internal/sources.md

Source Internals

Purpose

This document covers implemented prompt, profile, schema, artifact, and catalog source behavior. It is for developers changing loaders or source wiring.

Full user-facing YAML and config reference material belongs in docs/config.md.

Prompt Definition Sources

internal/promptdef provides directory-backed and fs.FS repositories.

Behavior:

• recursively scans .yaml and .yml files.
• decodes YAML with known-fields checking.
• looks up prompts by YAML id, not by path.
• optionally filters by prompt version.
• rejects duplicate matching prompt IDs.
• requires id, version, and at least one message.
• requires each message to set exactly one of content or content_file.
• resolves filesystem content_file values relative to the prompt YAML file.
• resolves fs.FS content_file values inside the configured source root.
• permits prompt subdirectories only as organization; they are not part of prompt identity.

For fs.FS roots, absolute paths and relative traversal outside the source root are rejected by catalog path helpers.

Profile Sources

internal/profile provides directory-backed, fs.FS, and overlay repositories. internal/profile/builtin embeds built-in profile YAML assets and exposes them through the same repository interface.

Behavior:

• recursively scans .yaml and .yml files.
• decodes YAML with known-fields checking.
• looks up profiles by YAML id, not by path.
• rejects duplicate IDs inside the same source.
• rejects raw api_key fields in YAML; file-backed profiles must use api_key_env.
• validates required endpoint and model values.
• validates numeric profile ranges.

Overlay behavior:

• custom profiles are primary.
• built-in profiles are fallback.
• fallback occurs only after a primary ErrProfileNotFound.
• primary validation, YAML, duplicate, and raw-key errors are returned directly.
• duplicate IDs across custom and built-in sources are allowed because the custom profile overrides the built-in one.

The public Go facade can add in-memory profiles ahead of file-backed and built-in profiles.

Schema Sources

internal/validate provides:

• StandardValidator for filesystem paths.
• FSValidator for fs.FS roots and single-file public schema sources.

Behavior:

• json_schema validation requires a non-empty schema_path.
• filesystem schema paths resolve relative to schema_dir unless absolute.
• directory-backed schema lookup uses the explicit schema_path; it does not search recursively by basename.
• fs.FS schema paths must remain inside the configured source root.
• single-file schema sources match by the configured file base name.
• schema documents are loaded before the LLM call for structured output.
• JSON parse failures are validation content failures.
• schema access, decode, registration, and compile failures are runtime validation errors.

Artifact Sources

internal/artifact supports two input artifact reference types:

• inline
• file

Inline behavior:

• requires a non-empty body.
• produces text/plain artifacts.
• hashes the body bytes.

Direct file behavior:

• used by CLI run, CLI render, and the public Go facade.
• requires a non-empty URI.
• reads from the process filesystem without HTTP artifact-root restrictions.
• infers content type from file extension, defaulting to text/plain.

Restricted file behavior:

• used by HTTP serve.
• allows inline artifacts even when no artifact root is configured.
• denies file artifacts when no artifact root is configured.
• resolves relative file URIs against server.artifact_root.
• accepts absolute file URIs only when they pass containment checks.
• applies server.max_artifact_bytes when configured.

Restricted containment is lexical. It cleans paths and checks the relative path against the configured root; it does not resolve symlinks. Symlinks inside the root are followed by the operating system, including symlinks that target files outside the root.

Catalog Helpers

internal/filecatalog centralizes shared source helpers:

• recursive YAML discovery for filesystem and fs.FS roots.
• deterministic sorting.
• .yaml and .yml filtering.
• display paths for diagnostics.
• YAML file stems.
• fs.FS root cleaning and containment checks.

Repository code should use these helpers instead of reimplementing path traversal and containment rules.

Failure Behavior

Common source failures:

• missing prompt/profile/schema/artifact files.
• invalid YAML or JSON.
• unknown YAML fields.
• duplicate prompt or profile IDs.
• prompt/profile validation errors.
• raw API key fields in profile YAML.
• unsupported artifact reference type.
• missing inline body or file URI.
• artifact outside HTTP root.
• artifact exceeding HTTP size limit.
• schema load or compile failure.

Prompt/profile repository lookup errors are mapped by adapters separately from runtime runner errors. Validation content failures remain result state; source and schema runtime failures return errors.

State And Manifests

Source packages do not persist run state.

• No manifests are read or written.
• No source package implements skip or resume behavior.
• Source reads reflect the current filesystem or fs.FS state for each request.

Tests To Inspect

• internal/promptdef/repository_test.go
• internal/profile/repository_test.go
• internal/profile/builtin/repository_test.go
• internal/artifact/reader_test.go
• internal/validate/standard_validator_test.go
• internal/usecase/integration_test.go
• engine_test.go

Change Recipe

When updating prompt, profile, schema, or built-in-profile assets:

1. Keep files valid for their strict loader and source boundary.
2. Keep maintained examples and fixtures secret-free.
3. Run focused prompt, profile, schema, or validation tests for the changed source.
4. Update the configuration contract and every affected external contract; update this document when loading or precedence mechanics change.

Architectural Invariants

• Prompt/profile identity comes from YAML id.
• External YAML decoding remains strict.
• File-backed profile YAML never accepts raw API key values.
• Built-in profiles are fallback, not a replacement for custom source validation.
• HTTP file artifacts remain rooted by lexical containment.
• Schema runtime failures remain errors, while JSON/schema content mismatches remain validation results.