Add a feature roadmap and implementation plan to import the scriptorium LLM library

2026-07-05 12:42:15 -05:00
parent 7601731a2c
commit 9532ae8121
2 changed files with 769 additions and 0 deletions
--- a/docs/roadmap/implementation.md
+++ b/docs/roadmap/implementation.md
@@ -0,0 +1,474 @@
+# Scriptorium Cutover Implementation Plan
+
+This plan implements the target state in
+[scriptorium.md](scriptorium.md): a hard cutover from Notarius' local
+OpenAI-compatible adapter, local LLM profile schema, and local system/user
+prompt renderer to Scriptorium-backed prompt execution.
+
+The target implementer is an LLM coding agent. Complete each stage in order.
+Do not preserve backward compatibility for removed local LLM-profile or
+prompt-rendering behavior unless a stage explicitly says to keep a temporary
+test seam.
+
+## Global Constraints
+
+- Keep Scriptorium types out of chunk, extract, and normalize module contracts.
+  Module-facing requests should use Notarius-owned contract types.
+- Keep provider plumbing behind `internal/framework/llm` and CLI construction.
+- Keep module-owned prompt intent, prompt IDs, response schemas, validators, and
+  source-reference validation with the relevant modules.
+- Keep raw input material, raw references, raw prompts, raw schemas, and API
+  keys out of manifests and default diagnostics.
+- Treat the original source input and large references as prompt input
+  materials. They may be passed to Scriptorium as inline/file inputs, but should
+  not be copied into durable provenance.
+- Use Scriptorium profile configuration directly. Do not keep Notarius'
+  `llm_profiles` schema as a second profile system.
+- Use Scriptorium structured-output execution and validation behavior for
+  production calls. Notarius should still unmarshal successful structured JSON
+  into module-owned response structs and run module-owned deterministic
+  validation afterward.
+
+## Stage 1: Dependency And API Grounding
+
+Goal: add the Scriptorium dependency and establish the exact public API surface
+used by the rest of the implementation.
+
+Work:
+
+- Add `gitea.maximumdirect.net/eric/scriptorium` to `go.mod` with `go get`.
+- Inspect the installed package with `go doc` or source reads before coding the
+  adapter. Confirm names and fields for:
+  - `NewEngine`;
+  - `Config`;
+  - `WithPromptFS`, `WithPromptFile`, or equivalent prompt source options;
+  - `WithProfileFile`, `WithProfileFS`, profile directory config, and built-in
+    profile behavior;
+  - `WithSchemaFS` or equivalent schema source options;
+  - `RunRequest`, including prompt ID, prompt version, profile ID, inputs,
+    variables, metadata, direct API key, and validation override fields;
+  - `RunResult`, including raw output, output artifact, validation status,
+    prompt/profile/model metadata, usage, and timings;
+  - public sentinel errors.
+- Add a short internal implementation note as a code comment only where needed;
+  do not add user-facing docs in this stage except if a package-level test
+  helper needs explanation.
+
+Acceptance checks:
+
+- `go test ./internal/framework/llm`
+- `go test ./internal/cli`
+- `go test ./...`
+
+## Stage 2: Prompt Input Materials And Session IDs
+
+Goal: make raw source input and references available to module prompt execution
+without storing large content in manifests or default diagnostics.
+
+Work:
+
+- Add Notarius-owned contract types under `internal/framework/contracts`:
+  - `LLMInputMaterial` with at least `Name`, `MediaType`, `Content`, `Digest`,
+    `OriginURI`, and `SizeBytes`;
+  - a helper-friendly collection type, such as `LLMInputSet`, if useful.
+- Add prompt-execution fields to `StructuredCompletionRequest`:
+  - `PromptID`;
+  - `PromptVersion`;
+  - `ProfileID` or reuse the request `LLMProfile` value when called from stage
+    requests;
+  - `SessionID`;
+  - `Inputs map[string]LLMInputMaterial`;
+  - `Vars map[string]any`.
+- Keep or remove the old `Messages`, `Model`, `ResponseSchemaName`, and
+  `ResponseSchema` fields according to what makes the cutover cleanest. The
+  final production path must not require modules to pass rendered messages or
+  raw schema JSON directly to the provider adapter.
+- Add `SourceInput contracts.LLMInputMaterial` and `SessionID string` to
+  `ChunkRequest`, `ExtractionRequest`, and `NormalizeRequest`.
+- Add `SessionID string` to `pipeline.RunInput`.
+- In `pipeline.Run`, build a source input material from `RunInput.RawInput` and
+  `RunInput.Path`:
+  - preserve the bytes exactly;
+  - infer media type from the input path extension, using `application/json`
+    for `.json` and a deterministic fallback for unknown extensions;
+  - compute a SHA-256 digest;
+  - use a file URI or path-derived origin URI;
+  - do not put the bytes in manifest metadata.
+- Pass the same source input material and session ID to chunk, extract, and
+  normalize requests.
+- Reuse existing materialized `ReferenceSet` content for reference prompt
+  inputs. Do not introduce a second file-reading path for references.
+- Add CLI support for `--session-id <id>` on `notarius run`.
+  - Empty means use a deterministic default derived from the parsed source
+    document ID.
+  - The deterministic default must be stable across runs over the same parsed
+    source document.
+  - The explicit value should be trimmed and rejected if empty after trimming.
+- Because the default depends on the parsed source document, compute and attach
+  the final session ID inside the runner after input parsing, or return enough
+  information from parsing for the CLI to resolve it before LLM calls.
+- Record the non-secret session ID in run metadata or manifest metadata, but
+  never record raw prompt content.
+
+Acceptance tests:
+
+- Contract tests proving `LLMInputMaterial.Content` is defensively copied where
+  relevant and omitted from JSON.
+- Pipeline tests proving chunk, extract, and normalize requests receive the same
+  source input bytes and session ID.
+- CLI tests for `--session-id`, including explicit value, missing value,
+  trimming, and default behavior.
+- Redaction/diagnostics tests proving raw source bytes and reference content do
+  not appear in manifests or default diagnostics.
+
+Focused checks:
+
+- `go test ./internal/framework/contracts`
+- `go test ./internal/framework/pipeline`
+- `go test ./internal/cli`
+
+## Stage 3: Configuration Hard Cutover To Scriptorium Profiles
+
+Goal: replace Notarius' local LLM profile schema with Scriptorium profile
+selection and profile sources.
+
+Work:
+
+- Bump the Notarius file config version because this is an incompatible config
+  schema change.
+- Remove the top-level `llm_profiles` file-config schema and the local
+  `config.LLMProfile` model.
+- Add a top-level Scriptorium config block. Use this shape unless Scriptorium's
+  installed API requires a small naming adjustment:
+
+  ```yaml
+  scriptorium:
+    profile_dir: ./profiles
+    profile_file: ./profiles.yml
+  ```
+
+- Treat `profile_dir` and `profile_file` as mutually exclusive in Notarius
+  config validation. Built-in Scriptorium profiles remain available when neither
+  is set.
+- Keep existing module binding field name `llm_profile`; it now names a
+  Scriptorium profile ID.
+- Stop forcing empty module `llm_profile` bindings to
+  `pipeline.DefaultLLMProfile`. Empty means "use the Scriptorium prompt's
+  `default_profile`."
+- Keep `--llm-profile` as a run-level operational override. It should set the
+  same explicit Scriptorium profile ID for every LLM-eligible selected target.
+- Remove `OpenAICompatibleClientConfig` construction from `internal/core/config`.
+- Remove local OpenAI-compatible provider validation from config validation.
+- Add config validation for:
+  - config version;
+  - mutually exclusive `scriptorium.profile_dir` and
+    `scriptorium.profile_file`;
+  - non-empty profile source paths when fields are present;
+  - non-empty explicit `llm_profile` strings after trimming.
+- Add CLI/config validation that uses Scriptorium profile loading to reject
+  unknown explicit profile IDs when possible. Prompt-default profile failures
+  may surface during Scriptorium prepare/run if Scriptorium owns that lookup.
+- Remove or replace environment override behavior tied to
+  `NOTARIUS_LLM_DEFAULT_*`. Secrets should come from Scriptorium profile
+  `api_key_env` values or direct request-scoped Scriptorium behavior.
+- Update redacted effective config behavior for the new Scriptorium config
+  shape.
+
+Acceptance tests:
+
+- Config parsing accepts `scriptorium.profile_dir`.
+- Config parsing accepts `scriptorium.profile_file`.
+- Config validation rejects both fields set at once.
+- Config validation rejects stale `llm_profiles`.
+- Existing pipeline bindings with explicit `llm_profile` resolve to trimmed
+  Scriptorium profile IDs.
+- Empty `llm_profile` remains empty through resolution unless `--llm-profile`
+  is supplied.
+- CLI config validation fails cleanly for an unknown explicit profile ID.
+- Redacted config diagnostics do not contain raw API keys.
+
+Focused checks:
+
+- `go test ./internal/core/config`
+- `go test ./internal/cli`
+
+## Stage 4: Scriptorium Prompt And Schema Assets
+
+Goal: move production LLM prompts and schemas to Scriptorium-compatible assets
+while preserving module ownership.
+
+Work:
+
+- Create a prompt/schema asset registration mechanism that does not put
+  D&D-specific prompt content in framework packages.
+  - Recommended shape: add a small prompt-asset registry in
+    `internal/framework/llm` or a sibling framework package that can collect
+    `fs.FS` roots for Scriptorium prompt and schema sources.
+  - Production module packages should register their own Scriptorium prompt and
+    schema assets through production catalog/registry wiring in `internal/cli`.
+  - Shared D&D prompt assets may live in a D&D-specific module package such as
+    `internal/modules/dnd/promptassets`; they must not live in `internal/core`
+    or source-agnostic framework packages.
+- Add shared D&D prompt assets:
+  - stable shared system message, if needed;
+  - stable cacheable transcript user message:
+
+    ```text
+    A transcript of a Dungeons & Dragons gameplay session is provided below.
+
+    {{ input "transcript" }}
+    ```
+
+  - stable cacheable reference/context message templates for roster, glossary,
+    previous recap, or other large references used by current D&D modules.
+- Convert `dnd/scenes` to a Scriptorium prompt definition:
+  - prompt ID: `dnd.scenes`;
+  - prompt version: current module prompt version;
+  - input `transcript`, required, `application/json`;
+  - messages ordered for cache reuse: shared system, shared transcript user
+    message with cache control, scene task, scene instructions;
+  - output JSON schema path pointing at the existing scene schema asset;
+  - schema IDs/names/versions remain module-owned and manifest-safe.
+- Convert `dnd/spells` to a Scriptorium prompt definition:
+  - prompt ID: `dnd.spells`;
+  - prompt version: current module prompt version;
+  - input `transcript`, required, `application/json`;
+  - optional inputs for `roster` and `glossary`;
+  - messages ordered for cache reuse: shared system, shared transcript user
+    message with cache control, optional/reference context message with cache
+    control, spell task, spell instructions;
+  - output JSON schema path pointing at the existing spell schema asset.
+- For optional references, pass empty inline input material when the slot is
+  unbound unless Scriptorium's template/input semantics support missing
+  optional inputs cleanly. Do not let optional missing references make prompt
+  rendering fail.
+- For multiple reference items in one slot, concatenate deterministically with
+  stable headings that include only non-secret provenance, then pass the result
+  as that prompt input. Existing single-item slots should keep their current
+  behavior.
+- Replace current prompt hash metadata with hashes derived from the
+  Scriptorium prompt definition plus message assets, or with Scriptorium
+  prepared-run prompt metadata if it is available without raw prompt content.
+- Preserve existing manifest metadata keys where practical:
+  - `prompt_id`;
+  - `prompt_version`;
+  - `prompt_sha256`;
+  - `response_schema_key`;
+  - `response_schema_id`;
+  - `response_schema_name`;
+  - `response_schema_version`;
+  - `response_schema_sha256`.
+
+Acceptance tests:
+
+- Prompt asset loading fails fast for missing prompt files or schemas.
+- `dnd/scenes` prepared prompt contains separate transcript and task messages.
+- `dnd/spells` prepared prompt contains separate transcript, reference, and
+  task messages.
+- The transcript message body is byte-identical to the expected shared template
+  plus original Seriatim JSON bytes.
+- Reference prompt input rendering is deterministic.
+- Prompt/schema diagnostics omit raw prompt text, raw transcript bytes, raw
+  reference content, and raw schema JSON.
+
+Focused checks:
+
+- `go test ./internal/framework/llm`
+- `go test ./internal/modules/chunk/dnd/scenes`
+- `go test ./internal/modules/extract/dnd/spells`
+
+## Stage 5: Scriptorium-Backed LLM Runtime
+
+Goal: implement the production `StructuredLLMClient` using Scriptorium.
+
+Work:
+
+- Add a Scriptorium-backed client under `internal/framework/llm`.
+- Its constructor should accept:
+  - Scriptorium profile source settings from effective Notarius config;
+  - registered prompt/schema assets;
+  - request timeout or HTTP client settings only if still owned by Notarius
+    after the profile cutover;
+  - optional Scriptorium engine options for tests.
+- Implement `CompleteStructured(ctx, req, out)` by:
+  - validating `out` is a non-nil pointer;
+  - validating `req.PromptID` is non-empty;
+  - converting Notarius `LLMInputMaterial` values to Scriptorium artifact refs;
+  - adding `session_id` to Scriptorium request vars when non-empty;
+  - passing explicit profile ID only when the module binding or CLI override
+    supplied one;
+  - passing no raw API key unless a deliberate request-scoped secret path is
+    implemented;
+  - calling Scriptorium `Run`;
+  - converting final validation failure into a Notarius error;
+  - unmarshaling successful structured JSON into `out`;
+  - returning `StructuredCompletionResponse` with raw JSON content, provider,
+    model, profile ID when available, token usage, and non-secret metadata.
+- Add a profile recorder or response accumulator so `RunManifest.LLMProfiles`
+  records the actual Scriptorium profile/provider/model values used during the
+  run.
+  - Do not rely on one precomputed profile ID before pipeline execution.
+  - Deduplicate profile manifest entries deterministically.
+- Keep the existing `Scheduler` and scheduled client wrapper unless Scriptorium
+  provides an equivalent Notarius-approved concurrency mechanism.
+- Ensure all Scriptorium errors are wrapped with context and converted to
+  concise CLI-facing errors. Preserve `errors.Is` checks internally when
+  practical.
+- Apply Notarius secret redaction to errors before writing diagnostics.
+- Remove production construction of `OpenAICompatibleClient`.
+
+Acceptance tests:
+
+- Scriptorium adapter maps Notarius prompt request fields into the expected
+  Scriptorium run request using an injected fake Scriptorium LLM client.
+- Successful structured output unmarshals into the caller target.
+- Scriptorium validation failure returns an error.
+- Provider/runtime failure returns an error with operation context.
+- Context cancellation is respected.
+- Token usage maps into `StructuredCompletionResponse`.
+- Used Scriptorium profile/provider/model metadata appears in the run manifest
+  without secrets.
+- API keys or bearer tokens in synthetic errors are redacted.
+- Scheduler still bounds concurrent Scriptorium-backed calls.
+
+Focused checks:
+
+- `go test ./internal/framework/llm`
+- `go test ./internal/framework/pipeline`
+- `go test ./internal/cli`
+
+## Stage 6: Module Cutover And Legacy Runtime Removal
+
+Goal: update production modules to call the prompt-based contract and remove
+obsolete local prompt/runtime code.
+
+Work:
+
+- Update `dnd/scenes`:
+  - stop rendering local system/user prompt strings;
+  - call `CompleteStructured` with `PromptID`, `PromptVersion`, `SessionID`,
+    `ProfileID` or request LLM profile, `transcript` input material, and any
+    required vars;
+  - keep existing response validation, chunk canonicalization, caveat handling,
+    and manifest metadata policy.
+- Update `dnd/spells`:
+  - stop rendering local system/user prompt strings;
+  - call `CompleteStructured` with `transcript`, optional `roster`, optional
+    `glossary`, session ID, profile ID, and vars;
+  - keep existing spell response validation, source-reference validation, and
+    manifest metadata policy.
+- Update any LLM-backed normalize modules if present. If only noop normalize is
+  present, ensure the contract and tests prove normalizers can receive the same
+  Scriptorium-capable client and prompt inputs.
+- Remove the old `internal/framework/prompt` renderer if no remaining code uses
+  it. If generic tests still need prompt rendering, replace them with
+  Scriptorium prompt asset tests or delete obsolete tests.
+- Remove `internal/framework/llm/openai_compatible_client.go` and its tests
+  after the Scriptorium adapter tests cover replacement behavior.
+- Remove local OpenAI-compatible integration docs after current-behavior docs
+  are updated in Stage 7.
+- Remove stale schema registry helpers only if they are no longer needed for
+  module-owned schema metadata. Keep lightweight schema hashing/loading helpers
+  if modules still use them for manifest metadata.
+
+Acceptance tests:
+
+- `dnd/scenes` fake-client tests assert the module sends prompt ID,
+  transcript input, session ID, and schema/prompt metadata rather than rendered
+  message text.
+- `dnd/spells` fake-client tests assert roster/glossary inputs are passed as
+  inputs and not interpolated locally.
+- Existing malformed LLM response tests still fail as malformed structured
+  output.
+- Existing source-reference validation tests still pass.
+- No production code imports the old prompt renderer or local
+  OpenAI-compatible client.
+
+Focused checks:
+
+- `go test ./internal/modules/chunk/dnd/scenes`
+- `go test ./internal/modules/extract/dnd/spells`
+- If `internal/framework/prompt` is deleted, do not run a package-specific test
+  for it; instead verify with `rg -n "internal/framework/prompt|RenderUserSystem" internal`
+  that no production code still depends on it.
+
+## Stage 7: CLI, Examples, Docs, And Full Validation
+
+Goal: finish user-facing behavior, examples, and canonical docs for the new
+runtime.
+
+Work:
+
+- Update `examples/dnd-spells.config.yml` and any maintained test configs to
+  the new config version and Scriptorium profile source behavior.
+- Update `docs/config.md`:
+  - new config version;
+  - `scriptorium.profile_dir` and `scriptorium.profile_file`;
+  - `llm_profile` now means Scriptorium profile ID;
+  - empty `llm_profile` behavior;
+  - removed `llm_profiles`;
+  - removed `NOTARIUS_LLM_DEFAULT_*` env behavior if removed;
+  - secret-handling policy through Scriptorium profile `api_key_env`.
+- Update `docs/cli.md`:
+  - add `--session-id`;
+  - update `--llm-profile` wording to Scriptorium profile override;
+  - update run examples if needed.
+- Update `docs/internal/llm.md`:
+  - Scriptorium-backed runtime;
+  - prompt asset loading;
+  - input material handling;
+  - scheduling;
+  - structured-output validation;
+  - profile manifest recording;
+  - secret redaction.
+- Update `docs/internal/modules.md`:
+  - module prompt ownership through Scriptorium prompt definitions;
+  - chunk/extract/normalize request input materials and session ID;
+  - D&D scenes/spells prompt metadata.
+- Update `docs/internal/pipeline.md`:
+  - raw input material lifecycle;
+  - reference material lifecycle into prompt inputs;
+  - session ID lifecycle;
+  - actual LLM profile provenance.
+- Update `docs/integrations/json-output.md` for any manifest changes.
+- Remove or rewrite `docs/integrations/openai-compatible.md`. If no local
+  OpenAI-compatible adapter remains, do not document it as current Notarius
+  behavior.
+- Update `docs/troubleshooting.md` for:
+  - Scriptorium profile-not-found;
+  - prompt-not-found;
+  - schema/validation failures;
+  - missing API key env values;
+  - session ID usage if relevant.
+- Update `docs/operations.md` if diagnostics or retention behavior changes.
+- Remove the Scriptorium migration item from `docs/roadmap/future.md` once the
+  feature is implemented.
+- Replace `docs/roadmap/implementation.md` with a completed note or remove it
+  after implementation is complete, according to the repository's current
+  roadmap cleanup pattern.
+
+Acceptance tests and inspections:
+
+- `rg -n "llm_profiles|OpenAICompatibleClient|openai-compatible|RenderUserSystem|NOTARIUS_LLM_DEFAULT" internal docs examples`
+  should return only intentional historical/deferred references, if any.
+- `rg -n "scriptorium|session-id|profile_dir|profile_file" docs examples`
+  should show current-behavior docs and examples are updated.
+- `go test ./...`
+- `go vet ./...`
+- `go build ./cmd/notarius`
+
+## Cross-Stage Review Checklist
+
+Before considering the implementation complete, verify:
+
+- No Scriptorium public types appear in chunk, extract, or normalize contracts.
+- No raw source input, reference content, prompt text, schema JSON, API key, or
+  bearer token appears in manifests or default diagnostics.
+- The shared transcript message uses original Seriatim JSON bytes exactly.
+- Large reference content can be supplied as Scriptorium prompt inputs without
+  changing module-facing reference slot contracts.
+- `--session-id` is easy for an external orchestrator to pass.
+- Empty `llm_profile` lets prompt defaults work; explicit `llm_profile` and
+  `--llm-profile` select Scriptorium profile IDs.
+- Production Notarius LLM execution goes through Scriptorium.
+- Current-behavior documentation describes only implemented behavior.