scriptorium/docs/roadmap/implementation.md

Migration Step 4 Implementation Plan

Status

Complete as of 2026-07-28. All implementation gates, the dependency guard, documentation reconciliation, and required validation passed. Migration Step 5 repository creation is the next gate; it was not started here.

Objective

Implement the target state in the Step 4 public-facade adapter roadmap: make Scriptorium's CLI, HTTP handler, prepared-run formatter, and HTTP artifact policy consume only the public framework facade, while preserving the current executable contracts.

The accepted split decision owns the long-term Promptkit/Scriptorium boundary. The testing policy governs test value and ownership. The documentation policy governs current-behavior updates and canonical ownership.

Constraints

• Execute the stages in order and keep the repository buildable and testable at every stage gate.
• Do not create the Promptkit repository, change the module path, rename the root package, or move framework packages out of this repository.
• Do not export internal repositories, domain values, renderers, validators, the runner, or adapter DTOs.
• Add only the public artifact-reader extension and public error identities required by the Step 4 roadmap.
• Preserve CLI, application configuration, HTTP, containment, validation, timeout, credential, and redaction behavior.
• Keep tests deterministic, offline, independent of real credentials, and focused at the narrowest stable owner.
• Prefer rewriting or moving an existing test over retaining parallel assertions for the same behavior.
• Preserve unrelated working-tree changes.
• Do not begin Migration Step 5 in this change.

Stage 1: Complete The Public Framework Boundary

Add the public artifact-reader extension and error identities before changing any adapter consumer.

Public Artifact Reader

In types.go, add:

type ArtifactReader interface {
	Read(context.Context, ArtifactRef) (*Artifact, error)
}

In engine.go, add WithArtifactReader(ArtifactReader) Option. Extend engineOptions with an internal artifact-reader slot and source-presence flag, parallel to the existing LLM and source options. A supplied reader replaces artifact.NewCompositeReader() when the runner is constructed; when the option is omitted, current default inline and unrestricted file behavior remains unchanged.

Reject a nil public reader as ErrInvalidConfig. Keep option construction errors wrapped by NewEngine consistently with the existing option behavior.

Create artifact_reader.go in the root facade with a private adapter that implements internal/artifact.Reader by:

1. converting the internal reference to a public ArtifactRef;
2. calling the supplied public reader with the original context;
3. returning the reader error unchanged so the runner and facade can add their normal artifact-load categories while preserving errors.Is;
4. rejecting (nil, nil) with a descriptive non-sentinel error so the runner maps it to ErrArtifactLoad; and
5. converting a non-nil public artifact to an internal artifact while copying Body.

Do not teach the public adapter to infer missing names, hashes, sizes, or content types. Those values are the reader's responsibility, except that the existing runner continues assigning the input-map name when Artifact.Name is empty.

Public Error Detail

Add public ErrProfileRequired and ErrAPIKeyEnvMissing sentinels alongside the existing facade errors.

Update mapPublicError so:

• a missing explicit/default profile matches both ErrInvalidRequest and ErrProfileRequired;
• an unset selected credential environment variable matches both ErrInvalidRequest and ErrAPIKeyEnvMissing;
• the original internal error remains wrapped for context during the single-repository phase; and
• all existing broad public mappings retain their current behavior.

Use error wrapping, not message inspection. Check the specific internal causes before the general internal ErrInvalidRequest. Construct the returned chain so errors.Is can independently match the broad public sentinel, the specific public sentinel, and the original cause.

Public Tests

Extend engine_test.go rather than creating a mock framework. Add a small recording public ArtifactReader fake and cover:

• WithArtifactReader(nil) returns ErrInvalidConfig;
• the reader receives the exact public type, URI, and body from a request;
• a returned artifact participates in a successful preparation through the real engine collaborators;
• a reader error produces an error matching both the reader's custom sentinel and ErrArtifactLoad;
• (nil, nil) produces ErrArtifactLoad without panicking; and
• reader cancellation or a canceled caller context remains visible through the artifact-load error chain.

Add artifact_reader_internal_test.go in package scriptorium with a focused test of the private adapter's body-copy guarantee.

Extend the existing missing-credential public test to assert ErrAPIKeyEnvMissing and ErrInvalidRequest. Add a purpose-built prompt with no default profile and assert that preparation without an explicit profile matches ErrProfileRequired and ErrInvalidRequest. Do not assert complete error strings.

Run:

go test .
go test -count=20 -run 'Test.*(ArtifactReader|ProfileRequired|MissingCredentials)' .

Stage 1 Gate

• The public artifact-reader option has the exact contract defined in step4.md.
• Public error chains retain the two HTTP-required distinctions and their broad invalid-request category.
• Default engine artifact behavior is unchanged when the option is absent.
• No adapter has been refactored yet.

Stage 2: Establish The Scriptorium-Owned HTTP Artifact Reader

Create internal/adapter/http/artifact_reader.go. It must implement the new public scriptorium.ArtifactReader interface directly and must not import internal/domain, the Promptkit-destined internal/artifact, or framework defaults.

Expose an internal-repository constructor for the CLI adapter:

func NewRestrictedArtifactReader(root string, maxBytes int64) (scriptorium.ArtifactReader, error)

Keep these sentinels in the HTTP adapter package:

• ErrFileNotAllowed;
• ErrFileOutsideRoot; and
• ErrFileTooLarge.

The implementation should move the HTTP-owned mechanisms from internal/artifact into public types:

• route public inline and file reference types;
• reject unsupported reference types, missing inline bodies, and missing file paths with descriptive local errors;
• hash inline and file bodies with SHA-256;
• use text/plain as the inline and unknown-extension fallback content type;
• infer file content type with mime.TypeByExtension;
• populate file name, URI, size, hash, body, and content type;
• deny file references when the trimmed root is empty;
• reject a negative byte limit during construction;
• treat zero as an unlimited byte limit;
• clean and absolutize a configured root;
• accept cleaned relative and absolute paths lexically contained by that root;
• reject paths lexically outside the root;
• retain the documented behavior of following symlinks after lexical checking;
• check file size before reading and enforce the limit again with io.LimitReader(maxBytes+1); and
• honor caller cancellation before resolving either inline or file input.

Move the restricted-reader unit cases from internal/artifact/reader_test.go into internal/adapter/http/artifact_reader_test.go, rewritten with public ArtifactRef and Artifact values. Preserve coverage for contained relative and absolute paths, traversal, outside absolute paths, symlinks, no-root denial, exact and excessive byte limits, zero limits, metadata, and negative limits. Add focused cancellation and unsupported/malformed-reference cases.

Keep the old restricted-reader implementation in internal/artifact only temporarily because CLI serving and some handler tests still use its internal interface. Remove its now-moved focused unit cases to avoid maintaining two authoritative test sets. Mark no compatibility promise for the old internal constructor; it must be deleted in Stage 4 when the HTTP handler and CLI serve path move together.

Run:

go test ./internal/artifact ./internal/adapter/http

Stage 2 Gate

• The new HTTP reader independently satisfies the public reader contract and preserves every documented HTTP containment outcome.
• Its focused tests use no framework-internal type.
• The old restricted implementation remains only as a temporary build bridge with a mandatory deletion point in Stage 4.

Stage 3: Rewrite Formatting And Move CLI Run And Render

Update internal/format/prepared_run.go so every formatter interface and function accepts *scriptorium.PreparedRun rather than *domain.PreparedRun. Replace internal rendered-message and target references with their corresponding public types. Do not add a translation DTO: the formatter is a Scriptorium consumer of the public value.

Update internal/format/prepared_run_test.go to construct public values. Keep the existing behavioral assertions for:

• deterministic text formatting;
• effective settings and sorted extra parameters;
• session IDs and cache-control fields;
• message ordering and content;
• valid JSON output;
• omission of absent optional values; and
• absence of resolved or direct secret values.

Do not change text or JSON output solely because the source Go type changed. Leave the Scriptorium-owned format enum and parser used by the configuration package unchanged.

Refactor the run and render paths in internal/adapter/cli/run.go to import and consume the root public package. Keep the existing internal construction used by serve temporarily because the HTTP handler still accepts internal types until Stage 4. Do not add a conversion shim between a public engine and the old handler.

Construction

Add a public engine-construction helper that:

• maps prompt, profile, and schema directories into public Config;
• accepts public engine options so Stage 4 can supply WithArtifactReader;
• leaves framework timeout and provider defaults unset unless Scriptorium has an actual application-owned override; and
• returns the NewEngine error to the command.

run and render use this helper with the engine's default artifact reader and default OpenAI-compatible client. Stop constructing an internal LLM client or runner on those paths.

Report engine-construction failures as ordinary command runtime failures without leaking secrets.

Leave serve, its internal LLM construction, its legacy restricted reader, and the internal runner helper buildable but otherwise unchanged in this stage. They are a temporary boundary exception with an explicit removal in Stage 4.

Request And Result Mapping

Change buildRunRequestFromConfig to return public RunRequest values:

• use public File references for CLI input paths;
• copy variables into the public request;
• map prompt and profile IDs unchanged;
• create a public ExecutionTargetOverride only when at least one override flag was explicitly supplied; and
• preserve pointer presence and explicit zero for temperature, max tokens, top-p, and timeout seconds.

Change determineExitCode, printSummary, output handling, and render formatting to consume public result and prepared values. Preserve current CLI output, validation exit classification, cache-usage summary behavior, and file permissions.

CLI Tests

Rewrite CLI tests and helpers to use public request and result values. Keep application configuration and Scriptorium default imports where they remain owned locally. Preserve the existing parser, precedence, output, built-in profile, selected-profile, explicit-zero, provider-wire, and end-to-end command tests.

Add or consolidate a focused mapping assertion showing that omitted numeric flags remain nil while explicitly supplied zero values remain non-nil public overrides. Do not duplicate framework precedence tests already owned by the root suite.

Run:

go test ./internal/format ./internal/config
go test ./internal/adapter/cli
go test -count=20 ./internal/adapter/cli
bash ./examples/render-markdown-summary.sh

Stage 3 Gate

• internal/format has no internal/domain import and formatter output remains unchanged.
• Application configuration still parses and applies render-format settings.
• CLI run and render construct and call only the public engine for framework behavior.
• CLI behavior and maintained rendering remain unchanged.
• The remaining internal framework imports are confined to serve construction and its legacy helpers, with deletion required in the next stage.

Stage 4: Move The HTTP Handler And CLI Serve Path Together

Rewrite internal/adapter/http/handler.go against the root public facade.

Handler Boundary And DTO Conversion

Keep the consumer interface in the HTTP package and narrow it to:

type Runner interface {
	Run(context.Context, scriptorium.RunRequest) (*scriptorium.RunResult, error)
}

The public *Engine must satisfy it directly. Do not add this interface to the root facade.

Map request DTOs directly to public values:

• convert each input to public ArtifactRef;
• convert model overrides to public ExecutionTargetOverride;
• retain pointer-valued numeric fields without dereferencing them;
• map variables, prompt version, and profile selection; and
• continue omitting direct API keys and framework-only request fields from the HTTP DTO.

Map public artifacts, validation, execution targets, usage, hashes, timestamps, and duration into the existing response DTO without changing serialized names or inclusion rules.

Error Mapping

Replace every internal prompt, profile, artifact, domain, and use-case error check with public framework or Scriptorium HTTP-reader identities. Preserve the existing mapping order:

1. public ErrPromptNotFound and ErrProfileNotFound;
2. public ErrProfileRequired and ErrAPIKeyEnvMissing;
3. public prompt/profile load errors and general ErrInvalidRequest;
4. local ErrFileNotAllowed or ErrFileOutsideRoot, then local ErrFileTooLarge;
5. public ErrArtifactLoad;
6. public render, generation, and validation errors; and
7. the sanitized internal fallback.

Keep current status codes, public error codes, and sanitized messages exactly as defined by docs/api.md. Check identity with errors.Is, never error text.

HTTP Tests

Rewrite the handler fake to use public request and result values. Convert request/response mapping tests and the error table to public sentinels and local HTTP artifact sentinels.

For tests that need assembled behavior, construct a real public engine from small temporary prompt/profile fixtures and inject only a public LLM fake. Add the new restricted artifact reader in the containment cases. In particular:

• retain one public-engine test proving reserved provider parameters map to invalid_request;
• retain end-to-end handler coverage for inline input, file denial without a root, contained file access, outside-root denial, and artifact-size enforcement; and
• keep the maintained examples/http-run.json request-contract check.

Do not recreate internal prompt repositories, profile repositories, renderers, or runners in handler tests. Remove redundant assembled assertions already owned by the public engine or focused artifact-reader tests.

Complete The CLI Boundary

After the handler accepts the public run interface, update serveCommand to:

1. construct httpadapter.NewRestrictedArtifactReader from the resolved artifact root and byte limit;
2. pass it to the public engine helper through scriptorium.WithArtifactReader;
3. handle reader and engine construction failures as runtime errors; and
4. pass the public engine directly to NewHandlerWithOptions.

Then remove newRunner, newRunnerWithArtifactReader, newOpenAIClient, and all remaining Promptkit-destined internal imports from CLI production. The CLI may continue importing Scriptorium-owned configuration, defaults, formatter, and HTTP adapter packages.

After all CLI and handler production and test consumers use public types:

• delete NewRestrictedCompositeReader, NewRestrictedCompositeReaderWithLimit, the denied and restricted file reader implementations, and ErrFileNotAllowed, ErrFileOutsideRoot, and ErrFileTooLarge from internal/artifact;
• simplify the general framework file reader to retain unrestricted reading without the HTTP byte-limit path; and
• remove any obsolete restricted-reader test code left in internal/artifact/reader_test.go.

Run:

go test ./internal/artifact ./internal/adapter/http
go test ./internal/adapter/cli
go test -count=20 ./internal/adapter/http
go test ./internal/adapter/http -run TestMaintainedHTTPRunExampleMatchesRequestContract

Stage 4 Gate

• HTTP production and tests use public framework values and errors only.
• CLI serve injects the Scriptorium-owned public reader into the public engine and passes that engine directly to the handler.
• CLI production has no Promptkit-destined internal import.
• All HTTP artifact policy lives with the HTTP adapter.
• The legacy restricted reader and its Scriptorium-specific errors no longer exist in the Promptkit-destined general artifact package.
• The HTTP contract and maintained request example remain unchanged.

Stage 5: Enforce The Boundary, Reconcile Documentation, And Validate

Dependency Guard

Add internal/adapter/dependency_test.go with package adapter_test. Define TestScriptoriumAdaptersUseOnlyPublicFrameworkBoundary to inspect direct imports in non-test Go files for:

• internal/adapter/cli;
• internal/adapter/http; and
• internal/format.

Use the standard library parser with parser.ImportsOnly, locating the source directories relative to the test file. Do not invoke network tools or inspect transitive dependencies: importing the current public root necessarily reaches the internal framework until extraction.

Reject direct imports of:

• internal/domain;
• internal/usecase;
• internal/promptdef;
• internal/prompt;
• internal/profile and internal/profile/builtin;
• internal/validate;
• internal/llm; and
• internal/artifact.

The failure must identify both the importing file and forbidden path. This test is the durable Step 4 boundary owner; do not add duplicate per-package import tests.

Run a final search over both production and test files. Scriptorium adapter and formatter tests should also use public framework types, although the durable architecture guard needs to enforce production imports only.

Documentation Reconciliation

Update current-behavior documentation only after the boundary is implemented:

• docs/consumers/pkg-scriptorium.md
  • add ArtifactReader and WithArtifactReader;
  • define replacement of the default reader, nil handling, error wrapping, and custom-error preservation;
  • add ErrProfileRequired and ErrAPIKeyEnvMissing and state that both also match ErrInvalidRequest.
• docs/policy/architecture.md
  • state that executable adapters consume framework behavior through the public facade;
  • retain the current single-repository system shape and avoid presenting the future repository split as complete.
• docs/internal/adapters.md
  • replace direct runner/domain composition with public engine construction, public request/result mapping, and adapter-local consumer interfaces;
  • describe serve reader injection and public error mapping.
• docs/internal/sources.md
  • distinguish the framework's ordinary reader from the public extension and Scriptorium's HTTP-owned restricted reader;
  • remove the stale deleted integration-test reference.
• docs/internal/overview.md
  • update the adapter, formatter, and split artifact responsibilities.
• docs/internal/runner.md
  • clarify that executable adapters reach the runner through the public engine;
  • remove the stale deleted integration-test reference.

Review README.md, docs/cli.md, docs/config.md, docs/api.md, docs/operations.md, and the integration documents. Do not edit them if their observable contract remains accurate; canonical owners should not receive refactor-only churn.

Full Validation

Run:

go test ./...
go vet ./...
build_output="$(mktemp -d)"
go build -o "$build_output/scriptorium" ./cmd/scriptorium
go test -race . ./internal/adapter/http
go test -count=20 . ./internal/adapter/cli ./internal/adapter/http ./internal/format
go test ./internal/adapter/http -run TestMaintainedHTTPRunExampleMatchesRequestContract
bash ./examples/render-markdown-summary.sh
go run ./cmd/scriptorium render \
  --config ./examples/config.full.yml \
  --prompt generic.markdown_summary \
  --input transcript=./examples/fixtures/transcript.md \
  --input glossary=./examples/fixtures/glossary.yml \
  --format text
go run ./examples/go-library/prepare
git diff --check

Also:

• run the dependency guard explicitly;
• verify both maintained configuration files passed through the real loader;
• validate every local Markdown link in changed documentation;
• confirm no real credentials or private data were added;
• inspect go.mod and go.sum and confirm no dependency change was needed;
• confirm no go.work, go.work.sum, or local replace directive was added; and
• inspect the final diff for unrelated or out-of-scope changes.

Completion Bookkeeping

After every check passes:

1. update step4.md to mark the target state complete and summarize the achieved boundary without converting it into an implementation log;
2. update the Step 4 gate in migration.md with the completion date and a concise validation summary;
3. mark this implementation plan complete; and
4. identify Step 5 repository creation as the next migration gate.

Do not begin repository creation or extraction in the same change.

Stage 5 Gate

• Every completion criterion in step4.md is satisfied.
• The dependency guard proves the intended direct-import boundary.
• Current documentation describes the implemented single-repository state.
• Full tests, vet, build, race checks, maintained examples, links, and whitespace validation pass.
• The main roadmap records Step 4 as complete and Step 5 as next.

Open Questions

None.