promptkit/docs/internal/sources.md

Internal Sources And Validation

Purpose

This document describes Promptkit's implemented configuration, artifact, rendering, and output-validation behavior. The architecture policy owns dependency rules, while the framework format reference owns fields, defaults, and source semantics. Internal packages are not consumer APIs.

Typed Configuration Loading

internal/configuration is the single adapter from ConfigKit's final effective configuration to execution-owned definitions. It performs one layered load, captures effective presence (including explicit null), records normalization and defaults through resolution, and validates all effective backend, profile, and prompt values. Replaced lower-layer domain values are not separately validated. It builds an immutable backend index and presence-aware profile targets, applying final backend defaults only after profile inheritance. It retains ConfigKit's result privately for explicit inspection, not for per-run resolution. The runtime/domain packages do not depend on ConfigKit.

The root DirectoryInputs and FilesystemInputs helpers discover sorted YAML files under an explicit root, provide resource context, and leave imports disabled. Each returned slice belongs to one strict-composition group; file order does not confer priority. The engine prepends selected catalog and Promptkit-owned definitions, then caller underlays and optional main. The configuration tests exercise the real loader and final-value handoff.

Prompt Definitions

The configuration adapter indexes each prompt by exact ID and version, materializes file-backed message content using the exact effective field's origin and resource context, and checks static message/session template syntax at construction. It stores captured content, compiled templates, declared input metadata, and a definition hash. Alongside the unchanged hashed definition, it retains content-type and description presence for each declared input and provides an internal requirement snapshot with those declarations and the captured parses. Metadata values come from the final normalized declaration; absent and null fields remain distinguishable from explicit strings, including empty strings. It closes content-file handles before returning and retains no source capability for public inspection.

Role and cache-control normalization is applied to execution-owned message copies. The retained ConfigKit configuration keeps effective input message metadata and its resource origins: recording a sequence-wide normalization in ConfigKit v0.5.0 would replace origins of unchanged content-file fields.

The runner selects from this index for preparation and InspectPrompt; the internal requirement snapshot lookup selects by exact prompt ID and version. These paths do not rescan configuration or reopen message files. InspectPrompt returns declared metadata without rendering or resolving profile, input, or schema resources, and its public shape and prompt hashes do not include the captured presence state. Runtime rendering still checks request variables and input helpers. The prompt adapter tests cover resource origins, context containment, capture, and selection.

The internal/prompt requirement walker uses the captured parse trees and declaration metadata without executing templates or reading artifacts. It collects literal input names, direct root caller keys, and literal-key index lookups. For if and with, it classifies branches when their conditions are simple supported values: direct values, literals, negation, or two-string equality/inequality. It uses Go template truthiness, so a supplied nonempty string such as false is true. Missing decision values and content-dependent conditions leave dependency-bearing branches conditional and report incomplete discovery; unresolved conditions with no branch dependencies do not by themselves make discovery incomplete. Recognizable references on known skipped paths remain optional inventory entries with an inactive-use explanation, unless another applicable use establishes a stronger classification. Skipped expressions are scanned only for syntactically recognizable names and are not evaluated. A direct caller value or literal-key lookup can also supply an input name, including through simple parentheses or a direct pipeline to input; names are used exactly as supplied. Unresolved names and unsupported dependency-bearing expressions produce structured limitations while retaining recognizable references. Where an unresolved name or scope could affect existing optional entries, those entries are reported as conditional rather than inactive-only or definitively optional. Direct root-map with forms retain caller-map identity. Known scalar values remain separate from unknown scopes; accesses through untracked local maps report incomplete discovery rather than silently losing caller keys. Unsupported name expressions reuse value analysis's conservative syntax scan instead of recursively rescanning the same subtree. Invoked named templates are scanned conservatively, and dependency-bearing range scopes make discovery incomplete. Uncalled definitions and bypassed session templates are not traversed. This internal walker does not change rendering behavior.

Profiles And Built-Ins

internal/catalog selects the released keyed OpenRouter and Rakestrawhome assets as one ConfigKit underlay and PromptKit-owned profiles as the next. It discovers YAML inputs and supplies passive filesystem context without parsing or pre-validating catalog definitions. The configuration loader validates only the final effective backend and profile values, including consumer overlays of maintained bases or backends. There is no catalog-only preflight, reserved source namespace, or replay of backend registration.

ConfigKit resolves profile inheritance before the handoff. The configuration adapter validates effective references and field values, then builds target snapshots with backend defaults and presence/suppression state. A profile can remain partial until runtime overrides complete it. InspectProfile selects that captured target without prompt, artifact, schema, credential lookup, or provider work. Preparation applies runtime overrides and then checks target completeness. The target tests cover effective target interpretation, while the catalog tests cover selected released assets.

Ordinary Artifacts

internal/artifact accepts explicitly typed inline references even when their body is empty. It also resolves unrestricted, caller-selected paths only when they identify regular operating-system files, checking that condition before and after opening the file. It copies content into an artifact, records metadata and an opaque content-equality value, and applies a content-type fallback.

Regular files are read synchronously in bounded chunks. The opened file's size is only an initial-capacity hint, capped at 1 MiB; reads continue to EOF, and the returned body determines artifact size and hash. If an overreported hint leaves more than twice the actual body length unused, the completed body is copied to release the excess backing storage. Cancellation is checked before opening, before and after every read, and before publishing the artifact, so a canceled read never publishes partial content. The ordinary reader does not detach file reads into background goroutines.

This ordinary reader does not implement an inbound HTTP security boundary. In particular, it does not constrain files to an application root or impose an HTTP request-size policy. Scriptorium's restricted HTTP reader remains an application concern outside Promptkit. The artifact reader tests own the implemented reader behavior and failures.

Rendering

internal/prompt renders captured, pre-parsed definition templates using named artifacts and variables. It clones templates to bind request-specific input helpers without mutating shared state. Within one render, each referenced artifact body is converted to text lazily and cached by input name for reuse across the session and every message; the cache is not shared across renders. Conversion uses bounded chunks and preserves the artifact bytes exactly.

Template parsing occurs during engine construction. Runtime execution remains synchronous. The renderer checks cancellation around execution boundaries, between artifact conversion chunks, around each message, and before publishing the complete prompt. It cannot interrupt template work already in progress and never publishes a partial prompt after observing cancellation. It validates and canonicalizes message roles before carrying roles, session IDs, and cache control into the rendered prompt. Message and session templates execute into per-call string builders, so their rendered strings do not require a final buffer-to-string copy. The renderer tests own rendering behavior.

Schemas And Output Validation

internal/validate provides validators backed by an operating-system filesystem or an fs.FS. Invalid generated content is returned as a validation result; inability to load, register, or compile a schema is an operational error. Basic validation checks body emptiness on the byte slice directly instead of first making a response-sized string copy.

Every preparation operation creates one operation-local validation plan. None, basic, and JSON modes retain the effective output contract without source access. JSON Schema mode loads the root document once, resolves and compiles each transitive reference, and retains the compiled validator. Schema compiler resources use canonical escaped file or private-scheme URLs; loaders decode their paths once and enforce the configured source boundary. The provider-facing structured-output metadata uses the root document captured by the same plan.

Schema preparation and execution remain synchronous. Promptkit checks cancellation before and after source resolution, JSON decoding, compilation, and validation, and between bounded schema-read chunks. Once cancellation is observed it returns the context error without publishing a partial plan or validation result, even when a compiler or validator has just returned a different error or a successful result. An fs.FS method or JSON Schema dependency call already in progress cannot be preempted; Promptkit waits for that call to return and then gives cancellation precedence. Validation does not detach dependency work into background goroutines.

Prepare discards its validation plan after returning metadata. Run retains its plan for initial and repaired-output validation, then discards it with the operation. PrepareExecution retains the plan in its private frozen payload; RunPrepared uses that plan without reopening configuration, message, input, or schema resources or rerendering the request. A later ordinary Run uses the same captured configuration but prepares request artifacts and schemas afresh.

The validator tests own basic, JSON, JSON Schema, source resolution, schema loading, compilation, frozen-reference behavior, content-failure behavior, and the synchronous cancellation boundary. Prepared execution orchestration is owned by the use-case tests.