notarius/docs/roadmap/implementation.md

Scriptorium Cutover Implementation Plan

This plan implements the target state in scriptorium.md: a hard cutover from Notarius' local OpenAI-compatible adapter, local LLM profile schema, and local system/user prompt renderer to Scriptorium-backed prompt execution.

The target implementer is an LLM coding agent. Complete each stage in order. Do not preserve backward compatibility for removed local LLM-profile or prompt-rendering behavior unless a stage explicitly says to keep a temporary test seam.

Global Constraints

• Keep Scriptorium types out of chunk, extract, and normalize module contracts. Module-facing requests should use Notarius-owned contract types.
• Keep provider plumbing behind internal/framework/llm and CLI construction.
• Keep module-owned prompt intent, prompt IDs, response schemas, validators, and source-reference validation with the relevant modules.
• Keep raw input material, raw references, raw prompts, raw schemas, and API keys out of manifests and default diagnostics.
• Treat the original source input and large references as prompt input materials. They may be passed to Scriptorium as inline/file inputs, but should not be copied into durable provenance.
• Use Scriptorium profile configuration directly. Do not keep Notarius' llm_profiles schema as a second profile system.
• Use Scriptorium structured-output execution and validation behavior for production calls. Notarius should still unmarshal successful structured JSON into module-owned response structs and run module-owned deterministic validation afterward.

Stage 1: Dependency And API Grounding

Goal: add the Scriptorium dependency and establish the exact public API surface used by the rest of the implementation.

Work:

• Add gitea.maximumdirect.net/eric/scriptorium to go.mod with go get.
• Inspect the installed package with go doc or source reads before coding the adapter. Confirm names and fields for:
  • NewEngine;
  • Config;
  • WithPromptFS, WithPromptFile, or equivalent prompt source options;
  • WithProfileFile, WithProfileFS, profile directory config, and built-in profile behavior;
  • WithSchemaFS or equivalent schema source options;
  • RunRequest, including prompt ID, prompt version, profile ID, inputs, variables, metadata, direct API key, and validation override fields;
  • RunResult, including raw output, output artifact, validation status, prompt/profile/model metadata, usage, and timings;
  • public sentinel errors.
• Add a short internal implementation note as a code comment only where needed; do not add user-facing docs in this stage except if a package-level test helper needs explanation.

Acceptance checks:

• go test ./internal/framework/llm
• go test ./internal/cli
• go test ./...

Stage 2: Prompt Input Materials And Session IDs

Goal: make raw source input and references available to module prompt execution without storing large content in manifests or default diagnostics.

Work:

• Add Notarius-owned contract types under internal/framework/contracts:
  • LLMInputMaterial with at least Name, MediaType, Content, Digest, OriginURI, and SizeBytes;
  • a helper-friendly collection type, such as LLMInputSet, if useful.
• Add prompt-execution fields to StructuredCompletionRequest:
  • PromptID;
  • PromptVersion;
  • ProfileID or reuse the request LLMProfile value when called from stage requests;
  • SessionID;
  • Inputs map[string]LLMInputMaterial;
  • Vars map[string]any.
• Keep or remove the old Messages, Model, ResponseSchemaName, and ResponseSchema fields according to what makes the cutover cleanest. The final production path must not require modules to pass rendered messages or raw schema JSON directly to the provider adapter.
• Add SourceInput contracts.LLMInputMaterial and SessionID string to ChunkRequest, ExtractionRequest, and NormalizeRequest.
• Add SessionID string to pipeline.RunInput.
• In pipeline.Run, build a source input material from RunInput.RawInput and RunInput.Path:
  • preserve the bytes exactly;
  • infer media type from the input path extension, using application/json for .json and a deterministic fallback for unknown extensions;
  • compute a SHA-256 digest;
  • use a file URI or path-derived origin URI;
  • do not put the bytes in manifest metadata.
• Pass the same source input material and session ID to chunk, extract, and normalize requests.
• Reuse existing materialized ReferenceSet content for reference prompt inputs. Do not introduce a second file-reading path for references.
• Add CLI support for --session-id <id> on notarius run.
  • Empty means use a deterministic default derived from the parsed source document ID.
  • The deterministic default must be stable across runs over the same parsed source document.
  • The explicit value should be trimmed and rejected if empty after trimming.
• Because the default depends on the parsed source document, compute and attach the final session ID inside the runner after input parsing, or return enough information from parsing for the CLI to resolve it before LLM calls.
• Record the non-secret session ID in run metadata or manifest metadata, but never record raw prompt content.

Acceptance tests:

• Contract tests proving LLMInputMaterial.Content is defensively copied where relevant and omitted from JSON.
• Pipeline tests proving chunk, extract, and normalize requests receive the same source input bytes and session ID.
• CLI tests for --session-id, including explicit value, missing value, trimming, and default behavior.
• Redaction/diagnostics tests proving raw source bytes and reference content do not appear in manifests or default diagnostics.

Focused checks:

• go test ./internal/framework/contracts
• go test ./internal/framework/pipeline
• go test ./internal/cli

Stage 3: Configuration Hard Cutover To Scriptorium Profiles

Goal: replace Notarius' local LLM profile schema with Scriptorium profile selection and profile sources.

Work:

• Bump the Notarius file config version because this is an incompatible config schema change.

• Remove the top-level llm_profiles file-config schema and the local config.LLMProfile model.

• Add a top-level Scriptorium config block. Use this shape unless Scriptorium's installed API requires a small naming adjustment:

  scriptorium:
    profile_dir: ./profiles
    profile_file: ./profiles.yml
  

• Treat profile_dir and profile_file as mutually exclusive in Notarius config validation. Built-in Scriptorium profiles remain available when neither is set.

• Keep existing module binding field name llm_profile; it now names a Scriptorium profile ID.

• Stop forcing empty module llm_profile bindings to pipeline.DefaultLLMProfile. Empty means "use the Scriptorium prompt's default_profile."

• Keep --llm-profile as a run-level operational override. It should set the same explicit Scriptorium profile ID for every LLM-eligible selected target.

• Remove OpenAICompatibleClientConfig construction from internal/core/config.

• Remove local OpenAI-compatible provider validation from config validation.

• Add config validation for:

  • config version;
  • mutually exclusive scriptorium.profile_dir and scriptorium.profile_file;
  • non-empty profile source paths when fields are present;
  • non-empty explicit llm_profile strings after trimming.

• Add CLI/config validation that uses Scriptorium profile loading to reject unknown explicit profile IDs when possible. Prompt-default profile failures may surface during Scriptorium prepare/run if Scriptorium owns that lookup.

• Remove or replace environment override behavior tied to NOTARIUS_LLM_DEFAULT_*. Secrets should come from Scriptorium profile api_key_env values or direct request-scoped Scriptorium behavior.

• Update redacted effective config behavior for the new Scriptorium config shape.

Acceptance tests:

• Config parsing accepts scriptorium.profile_dir.
• Config parsing accepts scriptorium.profile_file.
• Config validation rejects both fields set at once.
• Config validation rejects stale llm_profiles.
• Existing pipeline bindings with explicit llm_profile resolve to trimmed Scriptorium profile IDs.
• Empty llm_profile remains empty through resolution unless --llm-profile is supplied.
• CLI config validation fails cleanly for an unknown explicit profile ID.
• Redacted config diagnostics do not contain raw API keys.

Focused checks:

• go test ./internal/core/config
• go test ./internal/cli

Stage 4: Scriptorium Prompt And Schema Assets

Goal: move production LLM prompts and schemas to Scriptorium-compatible assets while preserving module ownership.

Work:

• Create a prompt/schema asset registration mechanism that does not put D&D-specific prompt content in framework packages.
  • Recommended shape: add a small prompt-asset registry in internal/framework/llm or a sibling framework package that can collect fs.FS roots for Scriptorium prompt and schema sources.
  • Production module packages should register their own Scriptorium prompt and schema assets through production catalog/registry wiring in internal/cli.
  • Shared D&D prompt assets may live in a D&D-specific module package such as internal/modules/dnd/promptassets; they must not live in internal/core or source-agnostic framework packages.
• Add shared D&D prompt assets:

  • stable shared system message, if needed;

  • stable cacheable transcript user message:

    A transcript of a Dungeons & Dragons gameplay session is provided below.
    
    {{ input "transcript" }}
    

  • stable cacheable reference/context message templates for roster, glossary, previous recap, or other large references used by current D&D modules.

• Convert dnd/scenes to a Scriptorium prompt definition:
  • prompt ID: dnd.scenes;
  • prompt version: current module prompt version;
  • input transcript, required, application/json;
  • messages ordered for cache reuse: shared system, shared transcript user message with cache control, scene task, scene instructions;
  • output JSON schema path pointing at the existing scene schema asset;
  • schema IDs/names/versions remain module-owned and manifest-safe.
• Convert dnd/spells to a Scriptorium prompt definition:
  • prompt ID: dnd.spells;
  • prompt version: current module prompt version;
  • input transcript, required, application/json;
  • optional inputs for roster and glossary;
  • messages ordered for cache reuse: shared system, shared transcript user message with cache control, optional/reference context message with cache control, spell task, spell instructions;
  • output JSON schema path pointing at the existing spell schema asset.
• For optional references, pass empty inline input material when the slot is unbound unless Scriptorium's template/input semantics support missing optional inputs cleanly. Do not let optional missing references make prompt rendering fail.
• For multiple reference items in one slot, concatenate deterministically with stable headings that include only non-secret provenance, then pass the result as that prompt input. Existing single-item slots should keep their current behavior.
• Replace current prompt hash metadata with hashes derived from the Scriptorium prompt definition plus message assets, or with Scriptorium prepared-run prompt metadata if it is available without raw prompt content.
• Preserve existing manifest metadata keys where practical:
  • prompt_id;
  • prompt_version;
  • prompt_sha256;
  • response_schema_key;
  • response_schema_id;
  • response_schema_name;
  • response_schema_version;
  • response_schema_sha256.

Acceptance tests:

• Prompt asset loading fails fast for missing prompt files or schemas.
• dnd/scenes prepared prompt contains separate transcript and task messages.
• dnd/spells prepared prompt contains separate transcript, reference, and task messages.
• The transcript message body is byte-identical to the expected shared template plus original Seriatim JSON bytes.
• Reference prompt input rendering is deterministic.
• Prompt/schema diagnostics omit raw prompt text, raw transcript bytes, raw reference content, and raw schema JSON.

Focused checks:

• go test ./internal/framework/llm
• go test ./internal/modules/chunk/dnd/scenes
• go test ./internal/modules/extract/dnd/spells

Stage 5: Scriptorium-Backed LLM Runtime

Goal: implement the production StructuredLLMClient using Scriptorium.

Work:

• Add a Scriptorium-backed client under internal/framework/llm.
• Its constructor should accept:
  • Scriptorium profile source settings from effective Notarius config;
  • registered prompt/schema assets;
  • request timeout or HTTP client settings only if still owned by Notarius after the profile cutover;
  • optional Scriptorium engine options for tests.
• Implement CompleteStructured(ctx, req, out) by:
  • validating out is a non-nil pointer;
  • validating req.PromptID is non-empty;
  • converting Notarius LLMInputMaterial values to Scriptorium artifact refs;
  • adding session_id to Scriptorium request vars when non-empty;
  • passing explicit profile ID only when the module binding or CLI override supplied one;
  • passing no raw API key unless a deliberate request-scoped secret path is implemented;
  • calling Scriptorium Run;
  • converting final validation failure into a Notarius error;
  • unmarshaling successful structured JSON into out;
  • returning StructuredCompletionResponse with raw JSON content, provider, model, profile ID when available, token usage, and non-secret metadata.
• Add a profile recorder or response accumulator so RunManifest.LLMProfiles records the actual Scriptorium profile/provider/model values used during the run.
  • Do not rely on one precomputed profile ID before pipeline execution.
  • Deduplicate profile manifest entries deterministically.
• Keep the existing Scheduler and scheduled client wrapper unless Scriptorium provides an equivalent Notarius-approved concurrency mechanism.
• Ensure all Scriptorium errors are wrapped with context and converted to concise CLI-facing errors. Preserve errors.Is checks internally when practical.
• Apply Notarius secret redaction to errors before writing diagnostics.
• Remove production construction of OpenAICompatibleClient.

Acceptance tests:

• Scriptorium adapter maps Notarius prompt request fields into the expected Scriptorium run request using an injected fake Scriptorium LLM client.
• Successful structured output unmarshals into the caller target.
• Scriptorium validation failure returns an error.
• Provider/runtime failure returns an error with operation context.
• Context cancellation is respected.
• Token usage maps into StructuredCompletionResponse.
• Used Scriptorium profile/provider/model metadata appears in the run manifest without secrets.
• API keys or bearer tokens in synthetic errors are redacted.
• Scheduler still bounds concurrent Scriptorium-backed calls.

Focused checks:

• go test ./internal/framework/llm
• go test ./internal/framework/pipeline
• go test ./internal/cli

Stage 6: Module Cutover And Legacy Runtime Removal

Goal: update production modules to call the prompt-based contract and remove obsolete local prompt/runtime code.

Work:

• Update dnd/scenes:
  • stop rendering local system/user prompt strings;
  • call CompleteStructured with PromptID, PromptVersion, SessionID, ProfileID or request LLM profile, transcript input material, and any required vars;
  • keep existing response validation, chunk canonicalization, caveat handling, and manifest metadata policy.
• Update dnd/spells:
  • stop rendering local system/user prompt strings;
  • call CompleteStructured with transcript, optional roster, optional glossary, session ID, profile ID, and vars;
  • keep existing spell response validation, source-reference validation, and manifest metadata policy.
• Update any LLM-backed normalize modules if present. If only noop normalize is present, ensure the contract and tests prove normalizers can receive the same Scriptorium-capable client and prompt inputs.
• Remove the old internal/framework/prompt renderer if no remaining code uses it. If generic tests still need prompt rendering, replace them with Scriptorium prompt asset tests or delete obsolete tests.
• Remove internal/framework/llm/openai_compatible_client.go and its tests after the Scriptorium adapter tests cover replacement behavior.
• Remove local OpenAI-compatible integration docs after current-behavior docs are updated in Stage 7.
• Remove stale schema registry helpers only if they are no longer needed for module-owned schema metadata. Keep lightweight schema hashing/loading helpers if modules still use them for manifest metadata.

Acceptance tests:

• dnd/scenes fake-client tests assert the module sends prompt ID, transcript input, session ID, and schema/prompt metadata rather than rendered message text.
• dnd/spells fake-client tests assert roster/glossary inputs are passed as inputs and not interpolated locally.
• Existing malformed LLM response tests still fail as malformed structured output.
• Existing source-reference validation tests still pass.
• No production code imports the old prompt renderer or local OpenAI-compatible client.

Focused checks:

• go test ./internal/modules/chunk/dnd/scenes
• go test ./internal/modules/extract/dnd/spells
• If internal/framework/prompt is deleted, do not run a package-specific test for it; instead verify with rg -n "internal/framework/prompt|RenderUserSystem" internal that no production code still depends on it.

Stage 7: CLI, Examples, Docs, And Full Validation

Goal: finish user-facing behavior, examples, and canonical docs for the new runtime.

Work:

• Update examples/dnd-spells.config.yml and any maintained test configs to the new config version and Scriptorium profile source behavior.
• Update docs/config.md:
  • new config version;
  • scriptorium.profile_dir and scriptorium.profile_file;
  • llm_profile now means Scriptorium profile ID;
  • empty llm_profile behavior;
  • removed llm_profiles;
  • removed NOTARIUS_LLM_DEFAULT_* env behavior if removed;
  • secret-handling policy through Scriptorium profile api_key_env.
• Update docs/cli.md:
  • add --session-id;
  • update --llm-profile wording to Scriptorium profile override;
  • update run examples if needed.
• Update docs/internal/llm.md:
  • Scriptorium-backed runtime;
  • prompt asset loading;
  • input material handling;
  • scheduling;
  • structured-output validation;
  • profile manifest recording;
  • secret redaction.
• Update docs/internal/modules.md:
  • module prompt ownership through Scriptorium prompt definitions;
  • chunk/extract/normalize request input materials and session ID;
  • D&D scenes/spells prompt metadata.
• Update docs/internal/pipeline.md:
  • raw input material lifecycle;
  • reference material lifecycle into prompt inputs;
  • session ID lifecycle;
  • actual LLM profile provenance.
• Update docs/integrations/json-output.md for any manifest changes.
• Remove or rewrite docs/integrations/openai-compatible.md. If no local OpenAI-compatible adapter remains, do not document it as current Notarius behavior.
• Update docs/troubleshooting.md for:
  • Scriptorium profile-not-found;
  • prompt-not-found;
  • schema/validation failures;
  • missing API key env values;
  • session ID usage if relevant.
• Update docs/operations.md if diagnostics or retention behavior changes.
• Remove the Scriptorium migration item from docs/roadmap/future.md once the feature is implemented.
• Replace docs/roadmap/implementation.md with a completed note or remove it after implementation is complete, according to the repository's current roadmap cleanup pattern.

Acceptance tests and inspections:

• rg -n "llm_profiles|OpenAICompatibleClient|openai-compatible|RenderUserSystem|NOTARIUS_LLM_DEFAULT" internal docs examples should return only intentional historical/deferred references, if any.
• rg -n "scriptorium|session-id|profile_dir|profile_file" docs examples should show current-behavior docs and examples are updated.
• go test ./...
• go vet ./...
• go build ./cmd/notarius

Cross-Stage Review Checklist

Before considering the implementation complete, verify:

• No Scriptorium public types appear in chunk, extract, or normalize contracts.
• No raw source input, reference content, prompt text, schema JSON, API key, or bearer token appears in manifests or default diagnostics.
• The shared transcript message uses original Seriatim JSON bytes exactly.
• Large reference content can be supplied as Scriptorium prompt inputs without changing module-facing reference slot contracts.
• --session-id is easy for an external orchestrator to pass.
• Empty llm_profile lets prompt defaults work; explicit llm_profile and --llm-profile select Scriptorium profile IDs.
• Production Notarius LLM execution goes through Scriptorium.
• Current-behavior documentation describes only implemented behavior.