scriptorium/docs/roadmap/implementation.md

# Migration Step 4 Implementation Plan

## Status

Proposed. Implement each stage in order and satisfy its gate before continuing.

## Objective

Implement the target state in the
[Step 4 public-facade adapter roadmap](step4.md): make Scriptorium's CLI, HTTP
handler, prepared-run formatter, and HTTP artifact policy consume only the
public framework facade, while preserving the current executable contracts.

The [accepted split decision](../adr/0002-split-promptkit-from-scriptorium.md)
owns the long-term Promptkit/Scriptorium boundary. The
[testing policy](../policy/testing.md) governs test value and ownership. The
[documentation policy](../policy/documentation.md) governs current-behavior
updates and canonical ownership.

## Constraints

- Execute the stages in order and keep the repository buildable and testable at
  every stage gate.
- Do not create the Promptkit repository, change the module path, rename the
  root package, or move framework packages out of this repository.
- Do not export internal repositories, domain values, renderers, validators,
  the runner, or adapter DTOs.
- Add only the public artifact-reader extension and public error identities
  required by the Step 4 roadmap.
- Preserve CLI, application configuration, HTTP, containment, validation,
  timeout, credential, and redaction behavior.
- Keep tests deterministic, offline, independent of real credentials, and
  focused at the narrowest stable owner.
- Prefer rewriting or moving an existing test over retaining parallel
  assertions for the same behavior.
- Preserve unrelated working-tree changes.
- Do not begin Migration Step 5 in this change.

## Stage 1: Complete The Public Framework Boundary

Add the public artifact-reader extension and error identities before changing
any adapter consumer.

### Public Artifact Reader

In `types.go`, add:

```go
type ArtifactReader interface {
	Read(context.Context, ArtifactRef) (*Artifact, error)
}
```

In `engine.go`, add `WithArtifactReader(ArtifactReader) Option`. Extend
`engineOptions` with an internal artifact-reader slot and source-presence flag,
parallel to the existing LLM and source options. A supplied reader replaces
`artifact.NewCompositeReader()` when the runner is constructed; when the option
is omitted, current default inline and unrestricted file behavior remains
unchanged.

Reject a nil public reader as `ErrInvalidConfig`. Keep option construction
errors wrapped by `NewEngine` consistently with the existing option behavior.

Create `artifact_reader.go` in the root facade with a private adapter that
implements `internal/artifact.Reader` by:

1. converting the internal reference to a public `ArtifactRef`;
2. calling the supplied public reader with the original context;
3. returning the reader error unchanged so the runner and facade can add their
   normal artifact-load categories while preserving `errors.Is`;
4. rejecting `(nil, nil)` with a descriptive non-sentinel error so the runner
   maps it to `ErrArtifactLoad`; and
5. converting a non-nil public artifact to an internal artifact while copying
   `Body`.

Do not teach the public adapter to infer missing names, hashes, sizes, or
content types. Those values are the reader's responsibility, except that the
  existing runner continues assigning the input-map name when `Artifact.Name`
  is empty.

### Public Error Detail

Add public `ErrProfileRequired` and `ErrAPIKeyEnvMissing` sentinels alongside
the existing facade errors.

Update `mapPublicError` so:

- a missing explicit/default profile matches both `ErrInvalidRequest` and
  `ErrProfileRequired`;
- an unset selected credential environment variable matches both
  `ErrInvalidRequest` and `ErrAPIKeyEnvMissing`;
- the original internal error remains wrapped for context during the
  single-repository phase; and
- all existing broad public mappings retain their current behavior.

Use error wrapping, not message inspection. Check the specific internal causes
before the general internal `ErrInvalidRequest`. Construct the returned chain
so `errors.Is` can independently match the broad public sentinel, the specific
public sentinel, and the original cause.

### Public Tests

Extend `engine_test.go` rather than creating a mock framework. Add a small
recording public `ArtifactReader` fake and cover:

- `WithArtifactReader(nil)` returns `ErrInvalidConfig`;
- the reader receives the exact public type, URI, and body from a request;
- a returned artifact participates in a successful preparation through the
  real engine collaborators;
- a reader error produces an error matching both the reader's custom sentinel
  and `ErrArtifactLoad`;
- `(nil, nil)` produces `ErrArtifactLoad` without panicking; and
- reader cancellation or a canceled caller context remains visible through the
  artifact-load error chain.

Add `artifact_reader_internal_test.go` in package `scriptorium` with a focused
test of the private adapter's body-copy guarantee.

Extend the existing missing-credential public test to assert
`ErrAPIKeyEnvMissing` and `ErrInvalidRequest`. Add a purpose-built prompt with
no default profile and assert that preparation without an explicit profile
matches `ErrProfileRequired` and `ErrInvalidRequest`. Do not assert complete
error strings.

Run:

```bash
go test .
go test -count=20 -run 'Test.*(ArtifactReader|ProfileRequired|MissingCredentials)' .
```

### Stage 1 Gate

- The public artifact-reader option has the exact contract defined in
  `step4.md`.
- Public error chains retain the two HTTP-required distinctions and their broad
  invalid-request category.
- Default engine artifact behavior is unchanged when the option is absent.
- No adapter has been refactored yet.

## Stage 2: Establish The Scriptorium-Owned HTTP Artifact Reader

Create `internal/adapter/http/artifact_reader.go`. It must implement the new
public `scriptorium.ArtifactReader` interface directly and must not import
`internal/domain`, the Promptkit-destined `internal/artifact`, or framework
defaults.

Expose an internal-repository constructor for the CLI adapter:

```go
func NewRestrictedArtifactReader(root string, maxBytes int64) (scriptorium.ArtifactReader, error)
```

Keep these sentinels in the HTTP adapter package:

- `ErrFileNotAllowed`;
- `ErrFileOutsideRoot`; and
- `ErrFileTooLarge`.

The implementation should move the HTTP-owned mechanisms from
`internal/artifact` into public types:

- route public inline and file reference types;
- reject unsupported reference types, missing inline bodies, and missing file
  paths with descriptive local errors;
- hash inline and file bodies with SHA-256;
- use `text/plain` as the inline and unknown-extension fallback content type;
- infer file content type with `mime.TypeByExtension`;
- populate file name, URI, size, hash, body, and content type;
- deny file references when the trimmed root is empty;
- reject a negative byte limit during construction;
- treat zero as an unlimited byte limit;
- clean and absolutize a configured root;
- accept cleaned relative and absolute paths lexically contained by that root;
- reject paths lexically outside the root;
- retain the documented behavior of following symlinks after lexical checking;
- check file size before reading and enforce the limit again with
  `io.LimitReader(maxBytes+1)`; and
- honor caller cancellation before resolving either inline or file input.

Move the restricted-reader unit cases from
`internal/artifact/reader_test.go` into
`internal/adapter/http/artifact_reader_test.go`, rewritten with public
`ArtifactRef` and `Artifact` values. Preserve coverage for contained relative
and absolute paths, traversal, outside absolute paths, symlinks, no-root
denial, exact and excessive byte limits, zero limits, metadata, and negative
limits. Add focused cancellation and unsupported/malformed-reference cases.

Keep the old restricted-reader implementation in `internal/artifact` only
temporarily because CLI serving and some handler tests still use its internal
interface. Remove its now-moved focused unit cases to avoid maintaining two
authoritative test sets. Mark no compatibility promise for the old internal
constructor; it must be deleted in Stage 4 when the HTTP handler and CLI
`serve` path move together.

Run:

```bash
go test ./internal/artifact ./internal/adapter/http
```

### Stage 2 Gate

- The new HTTP reader independently satisfies the public reader contract and
  preserves every documented HTTP containment outcome.
- Its focused tests use no framework-internal type.
- The old restricted implementation remains only as a temporary build bridge
  with a mandatory deletion point in Stage 4.

## Stage 3: Rewrite Formatting And Move CLI Run And Render

Update `internal/format/prepared_run.go` so every formatter interface and
function accepts `*scriptorium.PreparedRun` rather than
`*domain.PreparedRun`. Replace internal rendered-message and target references
with their corresponding public types. Do not add a translation DTO: the
formatter is a Scriptorium consumer of the public value.

Update `internal/format/prepared_run_test.go` to construct public values. Keep
the existing behavioral assertions for:

- deterministic text formatting;
- effective settings and sorted extra parameters;
- session IDs and cache-control fields;
- message ordering and content;
- valid JSON output;
- omission of absent optional values; and
- absence of resolved or direct secret values.

Do not change text or JSON output solely because the source Go type changed.
Leave the Scriptorium-owned format enum and parser used by the configuration
package unchanged.

Refactor the `run` and `render` paths in `internal/adapter/cli/run.go` to import
and consume the root public package. Keep the existing internal construction
used by `serve` temporarily because the HTTP handler still accepts internal
types until Stage 4. Do not add a conversion shim between a public engine and
the old handler.

### Construction

Add a public engine-construction helper that:

- maps prompt, profile, and schema directories into public `Config`;
- accepts public engine options so Stage 4 can supply `WithArtifactReader`;
- leaves framework timeout and provider defaults unset unless Scriptorium has
  an actual application-owned override; and
- returns the `NewEngine` error to the command.

`run` and `render` use this helper with the engine's default artifact reader and
default OpenAI-compatible client. Stop constructing an internal LLM client or
runner on those paths.

Report engine-construction failures as ordinary command runtime failures
without leaking secrets.

Leave `serve`, its internal LLM construction, its legacy restricted reader, and
the internal runner helper buildable but otherwise unchanged in this stage.
They are a temporary boundary exception with an explicit removal in Stage 4.

### Request And Result Mapping

Change `buildRunRequestFromConfig` to return public `RunRequest` values:

- use public `File` references for CLI input paths;
- copy variables into the public request;
- map prompt and profile IDs unchanged;
- create a public `ExecutionTargetOverride` only when at least one override flag
  was explicitly supplied; and
- preserve pointer presence and explicit zero for temperature, max tokens,
  top-p, and timeout seconds.

Change `determineExitCode`, `printSummary`, output handling, and render
formatting to consume public result and prepared values. Preserve current CLI
output, validation exit classification, cache-usage summary behavior, and file
permissions.

### CLI Tests

Rewrite CLI tests and helpers to use public request and result values. Keep
application configuration and Scriptorium default imports where they remain
owned locally. Preserve the existing parser, precedence, output, built-in
profile, selected-profile, explicit-zero, provider-wire, and end-to-end command
tests.

Add or consolidate a focused mapping assertion showing that omitted numeric
flags remain nil while explicitly supplied zero values remain non-nil public
overrides. Do not duplicate framework precedence tests already owned by the
root suite.

Run:

```bash
go test ./internal/format ./internal/config
go test ./internal/adapter/cli
go test -count=20 ./internal/adapter/cli
bash ./examples/render-markdown-summary.sh
```

### Stage 3 Gate

- `internal/format` has no `internal/domain` import and formatter output remains
  unchanged.
- Application configuration still parses and applies render-format settings.
- CLI `run` and `render` construct and call only the public engine for framework
  behavior.
- CLI behavior and maintained rendering remain unchanged.
- The remaining internal framework imports are confined to `serve` construction
  and its legacy helpers, with deletion required in the next stage.

## Stage 4: Move The HTTP Handler And CLI Serve Path Together

Rewrite `internal/adapter/http/handler.go` against the root public facade.

### Handler Boundary And DTO Conversion

Keep the consumer interface in the HTTP package and narrow it to:

```go
type Runner interface {
	Run(context.Context, scriptorium.RunRequest) (*scriptorium.RunResult, error)
}
```

The public `*Engine` must satisfy it directly. Do not add this interface to the
root facade.

Map request DTOs directly to public values:

- convert each input to public `ArtifactRef`;
- convert model overrides to public `ExecutionTargetOverride`;
- retain pointer-valued numeric fields without dereferencing them;
- map variables, prompt version, and profile selection; and
- continue omitting direct API keys and framework-only request fields from the
  HTTP DTO.

Map public artifacts, validation, execution targets, usage, hashes, timestamps,
and duration into the existing response DTO without changing serialized names
or inclusion rules.

### Error Mapping

Replace every internal prompt, profile, artifact, domain, and use-case error
check with public framework or Scriptorium HTTP-reader identities. Preserve
the existing mapping order:

1. public `ErrPromptNotFound` and `ErrProfileNotFound`;
2. public `ErrProfileRequired` and `ErrAPIKeyEnvMissing`;
3. public prompt/profile load errors and general `ErrInvalidRequest`;
4. local `ErrFileNotAllowed` or `ErrFileOutsideRoot`, then local
   `ErrFileTooLarge`;
5. public `ErrArtifactLoad`;
6. public render, generation, and validation errors; and
7. the sanitized internal fallback.

Keep current status codes, public error codes, and sanitized messages exactly as
defined by `docs/api.md`. Check identity with `errors.Is`, never error text.

### HTTP Tests

Rewrite the handler fake to use public request and result values. Convert
request/response mapping tests and the error table to public sentinels and
local HTTP artifact sentinels.

For tests that need assembled behavior, construct a real public engine from
small temporary prompt/profile fixtures and inject only a public LLM fake. Add
the new restricted artifact reader in the containment cases. In particular:

- retain one public-engine test proving reserved provider parameters map to
  `invalid_request`;
- retain end-to-end handler coverage for inline input, file denial without a
  root, contained file access, outside-root denial, and artifact-size
  enforcement; and
- keep the maintained `examples/http-run.json` request-contract check.

Do not recreate internal prompt repositories, profile repositories, renderers,
or runners in handler tests. Remove redundant assembled assertions already
owned by the public engine or focused artifact-reader tests.

### Complete The CLI Boundary

After the handler accepts the public run interface, update `serveCommand` to:

1. construct `httpadapter.NewRestrictedArtifactReader` from the resolved
   artifact root and byte limit;
2. pass it to the public engine helper through
   `scriptorium.WithArtifactReader`;
3. handle reader and engine construction failures as runtime errors; and
4. pass the public engine directly to `NewHandlerWithOptions`.

Then remove `newRunner`, `newRunnerWithArtifactReader`, `newOpenAIClient`, and
all remaining Promptkit-destined internal imports from CLI production. The CLI
may continue importing Scriptorium-owned configuration, defaults, formatter,
and HTTP adapter packages.

After all CLI and handler production and test consumers use public types:

- delete `NewRestrictedCompositeReader`,
  `NewRestrictedCompositeReaderWithLimit`, the denied and restricted file
  reader implementations, and `ErrFileNotAllowed`, `ErrFileOutsideRoot`, and
  `ErrFileTooLarge` from `internal/artifact`;
- simplify the general framework file reader to retain unrestricted reading
  without the HTTP byte-limit path; and
- remove any obsolete restricted-reader test code left in
  `internal/artifact/reader_test.go`.

Run:

```bash
go test ./internal/artifact ./internal/adapter/http
go test ./internal/adapter/cli
go test -count=20 ./internal/adapter/http
go test ./internal/adapter/http -run TestMaintainedHTTPRunExampleMatchesRequestContract
```

### Stage 4 Gate

- HTTP production and tests use public framework values and errors only.
- CLI `serve` injects the Scriptorium-owned public reader into the public engine
  and passes that engine directly to the handler.
- CLI production has no Promptkit-destined internal import.
- All HTTP artifact policy lives with the HTTP adapter.
- The legacy restricted reader and its Scriptorium-specific errors no longer
  exist in the Promptkit-destined general artifact package.
- The HTTP contract and maintained request example remain unchanged.

## Stage 5: Enforce The Boundary, Reconcile Documentation, And Validate

### Dependency Guard

Add `internal/adapter/dependency_test.go` with package `adapter_test`. Define
`TestScriptoriumAdaptersUseOnlyPublicFrameworkBoundary` to inspect direct
imports in non-test Go files for:

- `internal/adapter/cli`;
- `internal/adapter/http`; and
- `internal/format`.

Use the standard library parser with `parser.ImportsOnly`, locating the source
directories relative to the test file. Do not invoke network tools or inspect
transitive dependencies: importing the current public root necessarily reaches
the internal framework until extraction.

Reject direct imports of:

- `internal/domain`;
- `internal/usecase`;
- `internal/promptdef`;
- `internal/prompt`;
- `internal/profile` and `internal/profile/builtin`;
- `internal/validate`;
- `internal/llm`; and
- `internal/artifact`.

The failure must identify both the importing file and forbidden path. This test
is the durable Step 4 boundary owner; do not add duplicate per-package import
tests.

Run a final search over both production and test files. Scriptorium adapter and
formatter tests should also use public framework types, although the durable
architecture guard needs to enforce production imports only.

### Documentation Reconciliation

Update current-behavior documentation only after the boundary is implemented:

- `docs/consumers/pkg-scriptorium.md`
  - add `ArtifactReader` and `WithArtifactReader`;
  - define replacement of the default reader, nil handling, error wrapping, and
    custom-error preservation;
  - add `ErrProfileRequired` and `ErrAPIKeyEnvMissing` and state that both also
    match `ErrInvalidRequest`.
- `docs/policy/architecture.md`
  - state that executable adapters consume framework behavior through the
    public facade;
  - retain the current single-repository system shape and avoid presenting the
    future repository split as complete.
- `docs/internal/adapters.md`
  - replace direct runner/domain composition with public engine construction,
    public request/result mapping, and adapter-local consumer interfaces;
  - describe `serve` reader injection and public error mapping.
- `docs/internal/sources.md`
  - distinguish the framework's ordinary reader from the public extension and
    Scriptorium's HTTP-owned restricted reader;
  - remove the stale deleted integration-test reference.
- `docs/internal/overview.md`
  - update the adapter, formatter, and split artifact responsibilities.
- `docs/internal/runner.md`
  - clarify that executable adapters reach the runner through the public engine;
  - remove the stale deleted integration-test reference.

Review `README.md`, `docs/cli.md`, `docs/config.md`, `docs/api.md`,
`docs/operations.md`, and the integration documents. Do not edit them if their
observable contract remains accurate; canonical owners should not receive
refactor-only churn.

### Full Validation

Run:

```bash
go test ./...
go vet ./...
build_output="$(mktemp -d)"
go build -o "$build_output/scriptorium" ./cmd/scriptorium
go test -race . ./internal/adapter/http
go test -count=20 . ./internal/adapter/cli ./internal/adapter/http ./internal/format
go test ./internal/adapter/http -run TestMaintainedHTTPRunExampleMatchesRequestContract
bash ./examples/render-markdown-summary.sh
go run ./cmd/scriptorium render \
  --config ./examples/config.full.yml \
  --prompt generic.markdown_summary \
  --input transcript=./examples/fixtures/transcript.md \
  --input glossary=./examples/fixtures/glossary.yml \
  --format text
go run ./examples/go-library/prepare
git diff --check
```

Also:

- run the dependency guard explicitly;
- verify both maintained configuration files passed through the real loader;
- validate every local Markdown link in changed documentation;
- confirm no real credentials or private data were added;
- inspect `go.mod` and `go.sum` and confirm no dependency change was needed;
- confirm no `go.work`, `go.work.sum`, or local `replace` directive was added;
  and
- inspect the final diff for unrelated or out-of-scope changes.

### Completion Bookkeeping

After every check passes:

1. update `step4.md` to mark the target state complete and summarize the
   achieved boundary without converting it into an implementation log;
2. update the Step 4 gate in `migration.md` with the completion date and a
   concise validation summary;
3. mark this implementation plan complete; and
4. identify Step 5 repository creation as the next migration gate.

Do not begin repository creation or extraction in the same change.

### Stage 5 Gate

- Every completion criterion in `step4.md` is satisfied.
- The dependency guard proves the intended direct-import boundary.
- Current documentation describes the implemented single-repository state.
- Full tests, vet, build, race checks, maintained examples, links, and
  whitespace validation pass.
- The main roadmap records Step 4 as complete and Step 5 as next.

## Open Questions

None.