scriptorium/docs/roadmap/implementation.md

Migration Step 3 Implementation Plan

Status

Ready for implementation. No implementation work described here has started.

Objective

Implement the target state in the Step 3 framework-characterization roadmap: make Promptkit-destined tests independent of Scriptorium-owned executable examples, close the identified public contract gaps, and leave every migration-critical behavior with a clear test owner.

Follow the accepted ownership boundary in ADR 0002 and the test-value and non-duplication rules in the testing policy.

Constraints

• Execute the stages in order and satisfy each gate before proceeding.
• Do not change production behavior, public types, method signatures, package boundaries, or application interfaces.
• Limit implementation changes to testdata, tests, and roadmap status.
• Keep all default-suite tests deterministic, offline, and independent of real credentials.
• Preserve unrelated working-tree changes.
• Do not copy the complete examples/ tree into testdata.
• Do not add tests merely to raise statement coverage.
• Prefer extending or consolidating an existing test over adding a parallel test for the same behavior.

Stage 1: Create The Framework Contract Corpus

Create this Promptkit-destined fixture tree:

testdata/framework/
├── fixtures/
│   ├── glossary.yml
│   └── transcript.md
├── profiles/
│   ├── contract-fast.yaml
│   └── contract-quality.yaml
├── prompts/
│   ├── contract.markdown_summary.system.md
│   ├── contract.markdown_summary.user.md
│   ├── contract.markdown_summary.yaml
│   ├── contract.structured_events.system.md
│   ├── contract.structured_events.user.md
│   └── contract.structured_events.yaml
└── schemas/
    └── structured_events.schema.json

Define the corpus as follows:

• contract.markdown_summary
  • version 1.0.0;
  • default profile contract-fast;
  • required transcript input and optional glossary input;
  • system and user messages loaded through relative content_file paths; and
  • Markdown output with basic validation and no repair attempts.
• contract.structured_events
  • version 1.0.0;
  • default profile contract-quality;
  • the same two inputs;
  • system and user messages loaded through relative content_file paths; and
  • JSON output with JSON Schema validation through structured_events.schema.json and no repair attempts.
• contract-fast
  • endpoint http://localhost:8000/v1;
  • model contract-fast-model;
  • temperature 0.2, max tokens 500, top-p 1, and timeout 90; and
  • no credential requirement.
• contract-quality
  • endpoint http://localhost:8000/v1;
  • model contract-quality-model;
  • temperature 0.1, max tokens 1000, top-p 0.9, and timeout 120; and
  • no credential requirement.
• The schema requires an object containing an events array. Keep it small but valid for the same JSON Schema draft currently used by the maintained structured-output example.
• The transcript and glossary contain short synthetic values suitable for rendering and hash assertions. They must contain no private or real-world data.

Do not add application configuration, HTTP requests, executable scripts, or provider credentials to this corpus.

In engine_test.go, add shared constants for the contract root, prompt IDs, profile IDs, and fixture paths. Replace TestPrepareWorksWithExampleDirectoriesAndFileInputs with TestPrepareWorksWithFrameworkContractCorpus, using table cases for the ordinary and structured prompts. Construct the public engine from the new directories, use the new file artifacts, and assert that each prompt renders with its intended profile. For the structured case, also assert that PreparedRun.StructuredOutput contains the loaded JSON Schema specification. This test is the real-parser acceptance check for the corpus.

Run:

go test .
go test ./internal/promptdef ./internal/profile ./internal/validate

Stage 1 Gate

• Every corpus file loads through its real owning parser.
• Relative prompt content resolves from the prompt file location.
• The structured schema decodes through the real validator's schema-document loader and appears in the prepared structured-output specification.
• No production or executable-example file changed.

Stage 2: Move Framework Tests Onto Framework-Owned Fixtures

Update engine_test.go to use the new corpus.

1. Reuse the contract constants introduced in Stage 1.
2. Rename:
   • newExampleEngine to newContractEngine;
   • newExampleEngineWithOptions to newContractEngineWithOptions; and
   • exampleConfig to contractConfig.
3. Make contractConfig point at the corpus prompt, profile, and schema directories.
4. Replace each ./examples/... dependency in engine_test.go with the matching contract fixture or a purpose-built t.TempDir, fstest.MapFS, or in-memory profile.
5. Update assertions that intentionally identify fixture prompt IDs, profile IDs, models, rendered text, or hashes to the contract values. Do not change assertions that express independent public behavior.
6. Rename tests whose names say “example” when they now exercise contract testdata.

Add TestEngineRunWithDirectorySourcesAndFileInputs to engine_test.go. It must assemble the public engine from the contract prompt, profile, schema, and file-artifact directories; inject a deterministic LLMClient; run contract.structured_events; and assert:

• the prompt-selected contract-quality profile;
• a non-empty run ID, prompt hash, rendered-prompt hash, and both input hashes;
• provider-level JSON Schema structured output on the captured generation request;
• passed JSON Schema validation;
• application/json artifact content;
• preserved raw output and injected token usage; and
• non-zero ordered timestamps with non-negative duration.

Move the unique protection from internal/usecase/integration_test.go into this public test, then delete that internal integration test. Do not retain both assembled workflows.

The only test references to examples/ after this stage should be Scriptorium-owned adapter or maintained-example checks. In particular, this command must return no matches:

rg -n 'examples/' engine_test.go internal/usecase

Run:

go test .
go test ./internal/usecase

Stage 2 Gate

• Public and framework-internal tests pass without reading Scriptorium-owned executable examples.
• The new public assembled workflow subsumes the deleted internal integration test.
• Scriptorium's maintained examples are unchanged.
• No production file changed.

Stage 3: Consolidate And Complete Public Characterization

Execution-Setting Precedence

Add a table-driven TestEngineExecutionSettingPrecedence in engine_test.go. Run through the public engine with an injected recording LLMClient. Cover these cases:

1. a profile with zero-valued optional settings receives the documented framework numeric defaults;
2. non-zero profile settings replace those defaults;
3. request settings replace profile settings; and
4. explicit request numeric zero replaces non-zero profile settings.

Across the table, verify the effective endpoint, model, temperature, max-tokens, top-p, timeout, service tier, reasoning effort, API-key environment name, and extra_params where the relevant layer supplies them. Verify ExecutionTargetPresence is false for omitted numeric request settings and true for every explicitly supplied numeric setting, including zero.

Use relationally distinct values for each layer. Assert literal framework defaults only in the framework-default case because those values are part of the documented public contract. Use t.Setenv for every non-empty profile or request API-key environment name, assert only the environment-variable names, and never expose the test secret values.

Consolidate overlapping assertions:

• remove TestPreparePreservesExplicitZeroExecutionOverrides once the new table protects that behavior; and
• retain TestRunPassesPreparedRequestToInjectedLLMClient for rendered prompt, direct-key, and structured-output handoff, but remove execution-precedence assertions now owned by the table.

Caller Cancellation

Add TestEngineRunPropagatesCallerCancellation using the built-in OpenAI-compatible client and a custom RoundTripper.

• The transport must signal through a channel when RoundTrip begins.
• It must block on req.Context().Done() and return the context error.
• Start Engine.Run in a goroutine, wait for the transport signal, cancel the caller context, and collect the result through a buffered channel.
• Assert that the call returns and the error matches ErrLLMGenerate.
• Do not use sleeps or elapsed-time assertions.

Injected Nil Response

Add a valid-request case to TestPublicErrorsSupportErrorsIs whose injected LLMClient returns (nil, nil). Assert ErrLLMGenerate. Extend the existing fake only as needed to express this case; do not create a mock framework.

Reserved Provider Parameters

Add TestRunRejectsReservedExtraParamsBeforeProviderCall.

• Use the built-in client with a custom immediate RoundTripper that records whether it was invoked.
• Supply a valid contract prompt and profile plus request ExtraParams: map[string]any{"model": "collision"}.
• Assert ErrInvalidRequest.
• Assert that the transport was not invoked.

Run:

go test .
go test -count=20 .

Stage 3 Gate

• The four-layer precedence table and presence assertions pass.
• Cancellation is deterministic and contains no wall-clock sleeps.
• Nil injected responses and reserved parameters preserve their public error categories.
• Superseded assertions or tests have been removed rather than duplicated.
• No production file changed.

Stage 4: Audit Ownership And Validate The Baseline

Ownership Audit

Review the behavior list in step3.md against the final suite. Confirm:

• root facade and public contract tests are Promptkit-destined;
• internal/domain, internal/usecase, internal/promptdef, internal/prompt, internal/profile, internal/profile/builtin, internal/filecatalog, general internal/artifact, internal/validate, and internal/llm tests move with Promptkit-owned behavior;
• CLI, application configuration, prepared formatting, HTTP DTO, strict JSON, HTTP limit, and rooted artifact-containment tests remain Scriptorium-owned;
• HTTP and CLI tests that currently construct internal runners or inspect internal sentinels retain their observable assertions and are explicitly deferred for boundary rewrites in Migration Step 4; and
• no consequential behavior in the feature roadmap lacks a test owner.

Do not create a permanent test-inventory document. Record any unexpected ownership exception in step3.md; otherwise the ownership table there is the complete disposition.

Full Validation

Run:

go test ./...
go vet ./...
build_dir="$(mktemp -d)"
go build -o "$build_dir/scriptorium" ./cmd/scriptorium
go test -count=20 .
go test ./internal/adapter/http -run TestMaintainedHTTPRunExampleMatchesRequestContract
bash ./examples/render-markdown-summary.sh
go run ./cmd/scriptorium render \
  --config ./examples/config.full.yml \
  --prompt generic.markdown_summary \
  --input transcript=./examples/fixtures/transcript.md \
  --input glossary=./examples/fixtures/glossary.yml \
  --format text
go run ./examples/go-library/prepare
git diff --check

Validate every local Markdown link and path in the changed roadmap files. Confirm both configuration examples were accepted through the real configuration loader by the two render commands.

Inspect the final diff and confirm the implementation changed only:

• testdata/framework/**;
• engine_test.go;
• internal/usecase/integration_test.go by deletion;
• docs/roadmap/step3.md;
• docs/roadmap/implementation.md; and
• the Step 3 status in docs/roadmap/migration.md.

If a necessary change falls outside that list, stop and revise the plan or request direction rather than expanding scope implicitly.

Completion Bookkeeping

After every check passes:

1. update step3.md to state that the target state is complete and summarize the characterized baseline without reintroducing an implementation log;
2. add a Step 3 gate-status entry to migration.md with the completion date and a short validation summary; and
3. mark this implementation plan complete.

Do not begin Migration Step 4 in the same change.

Stage 4 Gate

• Every completion criterion in step3.md is satisfied.
• The full suite and maintained examples pass offline.
• The diff contains no production behavior or API change.
• The main migration roadmap identifies Step 3 as complete and Step 4 as next.

Open Questions

None.