diff --git a/README.md b/README.md index 7fea7db..48c1140 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,6 @@ This command renders the prepared prompt and effective runtime settings without - [Configuration reference](docs/config.md) - [HTTP API reference](docs/api.md) - [Operations guide](docs/operations.md) -- [Troubleshooting](docs/troubleshooting.md) - [Consumer integration overview](docs/consumers/api.md) - [Go library package](docs/consumers/pkg-scriptorium.md) - [Subprocess integration](docs/integrations/subprocess.md) diff --git a/docs/development.md b/docs/development.md index 3d26ec6..378e9d4 100644 --- a/docs/development.md +++ b/docs/development.md @@ -38,7 +38,7 @@ Start with: | Runner orchestration, rendering, validation, or repair | [Runner internals](internal/runner.md) and [source internals](internal/sources.md) | | OpenAI-compatible request or response behavior | [OpenAI-compatible integration](integrations/openai-compatible-chat.md), [LLM internals](internal/llm.md), [runner internals](internal/runner.md), and [adapter internals](internal/adapters.md) | | Subprocess behavior | [Subprocess integration](integrations/subprocess.md) and [CLI contract](cli.md) | -| Runtime operation, recovery, or troubleshooting | [Operations](operations.md) and [troubleshooting](troubleshooting.md) | +| Runtime operation or recovery | [Operations](operations.md) | | Examples or copyable assets | The owning contract for the demonstrated behavior and the related files under `examples/` | | Architecture decisions or future work | The [documentation policy](policy/documentation.md), relevant accepted ADRs such as [ADR 0001](adr/0001-adopt-canonical-documentation-ownership.md), and relevant roadmap documents under `roadmap/` | diff --git a/docs/operations.md b/docs/operations.md index 33d0a7d..06bab09 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,164 +1,157 @@ # Operations Guide -## Scope +## Scope And References -This guide covers operating the implemented CLI commands and HTTP service. It -does not replace the [CLI reference](cli.md), [Configuration reference](config.md), -or [HTTP API reference](api.md). +This runbook covers deployment, normal operation, capacity planning, and safe +recovery for Scriptorium. It does not redefine invocation syntax, configuration +fields, or HTTP wire behavior. -## Operational Model +- [CLI reference](cli.md): commands, output destinations, and exit codes. +- [Configuration reference](config.md): configuration, prompt/profile/schema + formats, defaults, and credentials. +- [HTTP API reference](api.md): route, request/response schema, status codes, + limits, and HTTP artifact access. +- [Consumer integration overview](consumers/api.md): caller responsibilities. -Scriptorium executes one prompt request per CLI invocation or HTTP request. +## Operational Model And State -Important boundaries: +Scriptorium handles one prompt request for each CLI invocation or HTTP request. +It has no durable run store, archive, checkpoint, cache, or resume mechanism. +A failed or interrupted request is recovered by correcting its inputs, +configuration, or environment and submitting a new request. -- No durable run state is stored. -- No manifest, archive, checkpoint, or built-in backup workflow is written. -- No built-in resume behavior exists. -- Recovery is rerun-based: correct inputs, config, or environment, then run again. +Generated artifacts, rendered prompts, model output, and run metadata are +caller-owned data. Retention, encryption, backup, and deletion are deployment +responsibilities. -## Filesystem Layout +## Deploy The Filesystem And Process -Operational deployments usually provide: +Provide the process with readable prompt, profile, and schema sources. Keep +prompt templates adjacent to the prompt definitions that reference them. For an +HTTP deployment that accepts file artifacts, use a dedicated, narrow artifact +directory rather than a general-purpose or sensitive filesystem tree. -- `prompt_dir`: prompt definition YAML files and adjacent `content_file` templates. -- `profile_dir`: optional custom profile YAML files. -- `schema_dir`: optional JSON Schema files. -- `server.artifact_root`: optional HTTP file-input root for `serve`. +Run Scriptorium under an identity that can: -Keep these directories readable by the Scriptorium process. Keep -`server.artifact_root` narrow and not writable by untrusted users. +- read only the prompt, profile, schema, and allowed input-artifact paths it + needs; +- read the required credential environment variables without writing them to + files or logs; and +- write only caller-selected output locations when CLI output files are used. -## Normal CLI Workflow +Do not make the HTTP artifact directory writable by untrusted users. The HTTP +artifact containment behavior is lexical and the operating system follows +symlinks; account for that when choosing ownership and mount boundaries. See +the [HTTP API reference](api.md) for the externally observable behavior. -Use `render` before `run` when changing prompt/profile/input wiring: +## Supply Credentials And Protect Runtime Data -```bash -go run ./cmd/scriptorium render \ - --config ./examples/config.yml \ - --prompt generic.markdown_summary \ - --input transcript=./examples/fixtures/transcript.md \ - --input glossary=./examples/fixtures/glossary.yml \ - --format json -``` +Set secret values in the process environment and configure only their +environment-variable names. Do not put raw keys in configuration, prompt or +profile files, process arguments, HTTP payloads, captured command lines, or +debug dumps. -Use `run` for generation after preflight: +Treat stdout, stderr, prepared-run output, generated artifacts, and HTTP +responses as potentially sensitive. Send service logs to a controlled collector +and apply the same retention and access rules as for model input and output. -```bash -go run ./cmd/scriptorium run \ - --config ./examples/config.yml \ - --prompt generic.markdown_summary \ - --input transcript=./examples/fixtures/transcript.md \ - --input glossary=./examples/fixtures/glossary.yml \ - --out ./summary.md -``` +## Run A Normal Workflow -Before production runs, confirm: +Before changing production inputs, profiles, or schemas: -- the effective config path is the intended one; -- prompt/profile/schema directories are readable; -- input file paths exist and match prompt input names; -- required API-key environment variables are set; -- the selected model endpoint is reachable from the process environment. +1. confirm the deployed configuration selects the intended sources and model + credentials; +2. use [`render`](cli.md) with the same request inputs and variables to confirm + preparation without a model call; +3. use [`run`](cli.md) for generation; and +4. retain or discard validation-failed output according to the caller's + policy. -## HTTP Service Operation +The [maintained render script](../examples/render-markdown-summary.sh) is a +copyable preflight example. The CLI reference owns its complete invocation and +exit semantics. -Start the service with: +## Expose The HTTP Service -```bash -go run ./cmd/scriptorium serve --config ./examples/config.yml -``` +The HTTP service has no built-in authentication or authorization. Place it on a +trusted network or behind an authenticated reverse proxy, API gateway, or +equivalent access control. Restrict who can reach it and who can read the +artifact root. -The implemented HTTP route is `POST /v1/runs`; request and response fields are -defined in the [HTTP API reference](api.md). +Use a service manager or supervisor appropriate to the deployment to manage +process lifetime, restart policy, log capture, and environment injection. The +[HTTP API reference](api.md) owns client request shapes, status behavior, and +artifact-access outcomes. -The maintained HTTP request-shape example is `examples/http-run.json`. +## Plan Capacity And Limits -HTTP service notes: +Capacity is primarily determined by concurrent model calls, input and output +sizes, schema complexity, provider latency, and network behavior. Size limits +protect request bodies, HTTP file artifacts, and encoded responses; configure +them through the [configuration reference](config.md) and rely on the +[HTTP API reference](api.md) for their response effects. -- Unknown JSON fields are rejected. -- `inline` input references work without an artifact root. -- `file` input references require `server.artifact_root` or `serve --artifact-root`. -- Request bodies, HTTP file input artifacts, and encoded JSON responses are size-limited. -- Validation content failures return `200 OK` with `validation.status: "failed"`. +Before increasing a limit: -Security boundary: +1. measure representative input, generated-output, and optional raw-output + sizes; +2. confirm memory, network, and upstream-provider capacity; +3. retain an upstream request-size and authentication boundary; and +4. test the intended workload in a non-production environment. -- `serve` has no built-in authentication or authorization. -- Put it behind trusted controls such as a private network, authenticated reverse proxy, or API gateway. -- Do not expose an artifact root containing unrelated sensitive files. -- Symlinks inside the artifact root are followed by the operating system. +For large local inputs, prefer a controlled file-artifact directory over +placing arbitrary paths on the service host. Avoid disabling a limit unless an +equivalent trusted control exists elsewhere. -## Secrets Handling +## Diagnose And Recover -Raw API keys are not accepted in app config, profiles, CLI flags, or HTTP -request bodies. +### Preparation Or Configuration Failure -Use this pattern: +Capture the CLI diagnostic or HTTP error response, then verify the selected +configuration, prompt ID, profile selection, source readability, and input +mapping. Use `render` with the same request when it is unclear whether failure +occurs before model execution. Consult the [CLI reference](cli.md), the +[configuration reference](config.md), and the [HTTP API reference](api.md) for +the exact interface contract. -1. Set an environment variable containing the secret value. -2. Store only the variable name in profile `api_key_env` or request override `api_key_env`. -3. Scope the process environment to the minimum required variables. +### Credential Or Provider Failure -## Output, Logs, And Exit Codes +Confirm that the process environment contains the configured credential name +without printing the secret. Check endpoint reachability and provider health +from the process network. If preparation succeeds but generation fails, inspect +the selected model settings in prepared output and the service's controlled +logs. Correct the deployment or provider issue, then submit a new request. -`run`: +### Artifact Or Permission Failure -- stdout: generated artifact body unless `--out` is used. -- stderr: summary on success, errors on failure. -- exit `2`: generation completed and output was written, but validation failed. +Verify that the process can read the intended local input. For HTTP file +artifacts, verify the deployment's artifact root, ownership, path layout, and +file size. Do not widen filesystem permissions or the allowed root merely to +make an arbitrary path work; move or copy the required artifact into the +controlled location instead. -`render`: +### Validation Failure -- stdout: prepared-run output unless `--out` is used. -- stderr: errors. -- exit `0` on success, `1` on failure. +A generated-content validation failure is distinct from a runtime failure. +CLI `run` reports the validation result and error count in its success summary; +it does not print the individual validation messages. For HTTP, inspect the +validation object in the response according to the [HTTP API reference](api.md). -`serve`: +Use rendered input and generated output to determine whether prompt instructions, +the selected model, or the schema needs correction. If schema loading or +compilation itself fails, correct the source deployment or schema document +before rerunning. -- stderr: startup and server errors. -- HTTP response body: JSON success or error envelope. +### HTTP Limit Or Request Failure -## Validation Behavior +Compare the request, artifact, or expected response size with the deployed +configuration, and validate the request against the [HTTP API reference](api.md). +Reduce the payload, use an appropriate controlled artifact source, omit +unneeded raw output, or adjust the deployment limit after capacity review. -Prompt `output.validation_mode` controls validation: +## Cleanup And Reruns -- `none`: skipped. -- `basic`: output body must not be empty. -- `json`: output body must parse as JSON. -- `json_schema`: output body must parse as JSON and satisfy the configured schema. - -Runtime/schema failures are hard failures (`run` exit `1`, HTTP error). -Generated-content validation failures are soft failures (`run` exit `2`, HTTP -`200 OK` with failed validation status). - -## Size Limits - -Defaults are documented in [Configuration reference](config.md). Operationally: - -- Keep default HTTP limits unless larger payloads are measured and expected. -- Prefer `inline` HTTP inputs for small payloads. -- Prefer `file` HTTP inputs for larger local artifacts under a controlled artifact root. -- Increase `server.max_response_bytes` when generated artifacts or requested raw output are expected to be large. -- Use `0` only when another trusted layer enforces size limits. - -## Maintained Examples - -- `examples/config.yml` -- `examples/config.full.yml` -- `examples/render-markdown-summary.sh` -- `examples/http-run.json` - -## Safe Recovery - -For failed CLI commands or HTTP requests: - -1. Capture stderr or the HTTP error `code` and `message`. -2. Confirm config path and effective directory settings. -3. Verify prompt ID, profile ID, schema path, and input mappings. -4. Verify required API-key environment variables. -5. Reproduce with `render --format json` when pre-LLM resolution is uncertain. -6. Rerun after correction. - -Because Scriptorium does not persist run state, rerun is the supported recovery -path. +Because no run state is retained, cleanup concerns caller-owned output files, +logs, and artifacts only. Remove or rotate them using the deployment's normal +retention policy. After a correction, rerun the request from the beginning; +there is no safe resume point. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md deleted file mode 100644 index 17942c1..0000000 --- a/docs/troubleshooting.md +++ /dev/null @@ -1,361 +0,0 @@ -# Troubleshooting - -This guide lists common implemented failure modes and safe fixes. - -Canonical references: - -- [CLI reference](cli.md) -- [Configuration reference](config.md) -- [HTTP API reference](api.md) -- [Operations guide](operations.md) - -## Missing Or Invalid Config - -Symptom: - -- CLI error includes `application config error`, `config file not found`, `invalid config YAML`, or `invalid config`. - -Likely cause: - -- `--config` points to a missing file. -- YAML syntax is invalid. -- Config contains unknown fields or negative HTTP size limits. - -Diagnostic step: - -```bash -go run ./cmd/scriptorium render --config /path/to/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml -``` - -Safe fix: - -- Correct the config path. -- Fix YAML syntax. -- Remove unknown fields. -- Keep raw secrets out of config. - -Relevant links: [Configuration reference](config.md), [CLI reference](cli.md) - -## Missing Prompt Directory - -Symptom: - -- CLI parse error says the prompt directory is required. - -Likely cause: - -- Neither config nor CLI flags provide an effective `prompt_dir`. - -Diagnostic step: - -- Re-run once with explicit `--prompt-dir`. - -Safe fix: - -- Set `prompt_dir` in config or pass `--prompt-dir`. - -Relevant links: [Configuration reference](config.md), [CLI reference](cli.md) - -## Unknown Flags - -Symptom: - -- CLI parse error for an unknown flag. - -Likely cause: - -- Typo. -- Flag is valid for another command. -- `serve` was given runtime model override flags. - -Diagnostic step: - -- Compare the command with the command-specific flag list. - -Safe fix: - -- Remove unsupported flags. -- Use `run` or `render` for runtime model overrides. - -Relevant links: [CLI reference](cli.md) - -## Prompt Load Failures - -Symptom: - -- CLI run/render fails during prompt loading. -- HTTP returns `404 prompt_not_found` or `400 prompt_load_failed`. - -Likely cause: - -- Prompt ID/version does not exist. -- Prompt YAML is invalid or has unknown fields. -- Prompt contract is invalid, such as missing messages, invalid output mode, bad `content_file`, or missing `schema_path` for `json_schema`. - -Diagnostic step: - -```bash -go run ./cmd/scriptorium render --config ./examples/config.yml --prompt --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml --format json -``` - -Safe fix: - -- Correct prompt ID/version. -- Fix prompt YAML and referenced `content_file` paths. -- Fix output contract fields. - -Relevant links: [Configuration reference](config.md), [CLI reference](cli.md) - -## Profile Load Failures - -Symptom: - -- CLI run/render fails during profile loading. -- HTTP returns `404 profile_not_found`, `400 profile_load_failed`, or `400 profile_required`. - -Likely cause: - -- Profile ID does not exist. -- Request omitted profile and prompt has no `default_profile`. -- Profile YAML is invalid or has unknown fields. -- Profile contains raw `api_key`. - -Diagnostic step: - -```bash -go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --profile --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml -``` - -Safe fix: - -- Correct profile ID or prompt `default_profile`. -- Fix profile YAML and value ranges. -- Replace raw `api_key` with `api_key_env`. - -Relevant links: [Configuration reference](config.md), [CLI reference](cli.md) - -## Input Artifact Failures - -Symptom: - -- CLI run/render fails while reading inputs. -- HTTP returns `400 artifact_read_failed`, `400 artifact_not_allowed`, or `413 artifact_too_large`. - -Likely cause: - -- Input file path is missing or unreadable. -- HTTP input type is unsupported or missing required fields. -- HTTP file refs are disabled because no artifact root is configured. -- HTTP file path is lexically outside the artifact root. -- HTTP file input exceeds `server.max_artifact_bytes`. - -Diagnostic step: - -- Verify each input path exists and is readable by the process. -- For HTTP, verify input refs use `file` or `inline`. -- For HTTP file refs, verify the artifact root and compare file size to `server.max_artifact_bytes`. - -Safe fix: - -- Correct paths and permissions. -- Configure a narrow artifact root for HTTP file refs. -- Use relative paths under the artifact root or switch to `inline`. -- Increase `server.max_artifact_bytes` only for expected larger inputs. - -Relevant links: [HTTP API reference](api.md), [Configuration reference](config.md) - -## Missing API-Key Environment Variable - -Symptom: - -- CLI render/run fails with an API-key environment error. -- HTTP returns `400 api_key_env_missing`. - -Likely cause: - -- Selected profile or runtime override sets `api_key_env`, but the environment variable is unset or empty. - -Diagnostic step: - -```bash -printenv SCRIPTORIUM_API_KEY -``` - -Safe fix: - -- Set the required environment variable before starting the CLI command or HTTP service. -- Or use a profile that does not require provider API-key auth. - -Relevant links: [Configuration reference](config.md), [Operations guide](operations.md) - -## Prompt Template Render Failures - -Symptom: - -- CLI render/run fails during prompt rendering. -- HTTP returns `400 prompt_render_failed`. - -Likely cause: - -- Template references an input that was not supplied. -- Template syntax or variable reference is invalid. - -Diagnostic step: - -- Run `render --format json` with the same prompt, inputs, vars, and profile. - -Safe fix: - -- Align `{{input "name"}}` references with request input names. -- Fix template syntax and variable names. - -Relevant links: [Configuration reference](config.md), [CLI reference](cli.md) - -## LLM Request Failures - -Symptom: - -- CLI `run` fails during generation. -- HTTP returns `502 llm_failed`. - -Likely cause: - -- Endpoint is unreachable. -- Provider returns non-2xx. -- Request times out. -- Provider response is malformed. - -Diagnostic step: - -- Run `render` first to confirm pre-LLM preparation works. -- Check selected endpoint/model in prepared output. -- Check network/provider logs for timeout or non-2xx details. - -Safe fix: - -- Correct endpoint/model/profile settings. -- Adjust timeout when appropriate. -- Resolve provider or network issue. - -Relevant links: [Operations guide](operations.md), [Configuration reference](config.md) - -## Validation Failed - -Symptom: - -- CLI `run` exits `2`. -- HTTP returns `200 OK` with `validation.status` set to `failed`. - -Likely cause: - -- Generated output failed `basic`, `json`, or `json_schema` content validation. - -Diagnostic step: - -- Inspect validation errors in CLI stderr or the HTTP response. - -Safe fix: - -- Refine prompt instructions. -- Adjust schema or model/profile settings. -- Rerun after correction. - -Relevant links: [Operations guide](operations.md), [HTTP API reference](api.md) - -## Validation Runtime Failure - -Symptom: - -- CLI `run` fails with validation runtime error. -- HTTP returns `500 validation_runtime_failed`. - -Likely cause: - -- `json_schema` schema file is missing or unreadable. -- Schema JSON is invalid. - -Diagnostic step: - -- Verify `schema_dir` and prompt `output.schema_path`. -- Check schema file readability and JSON syntax. - -Safe fix: - -- Correct schema path or permissions. -- Fix schema JSON. -- Rerun. - -Relevant links: [Configuration reference](config.md), [Operations guide](operations.md) - -## HTTP JSON Or Request Contract Errors - -Symptom: - -- HTTP returns `400 invalid_json` or `400 invalid_request`. - -Likely cause: - -- JSON body is malformed. -- Request has unknown fields or trailing JSON tokens. -- Required `prompt_id` or `inputs` is missing. -- Runtime override values are out of range. -- `extra_params` collides with reserved outbound fields. - -Diagnostic step: - -- Revalidate request JSON and compare fields with the API reference. - -Safe fix: - -- Send one JSON object with only supported fields. -- Include `prompt_id` and at least one input. -- Use valid model override ranges. -- Remove reserved `extra_params` keys. - -Relevant links: [HTTP API reference](api.md) - -## HTTP Size Limit Errors - -Symptom: - -- HTTP returns `413 request_too_large`, `413 artifact_too_large`, or `413 response_too_large`. - -Likely cause: - -- JSON request body exceeds `server.max_request_bytes`. -- HTTP file input exceeds `server.max_artifact_bytes`. -- Encoded JSON response exceeds `server.max_response_bytes`. - -Diagnostic step: - -- Compare request, file input, and expected response sizes with configured limits. - -Safe fix: - -- Use smaller inline inputs or switch to file inputs under the artifact root. -- Reduce generated output size. -- Omit `include_raw_output`. -- Increase limits only when the deployment expects larger payloads. - -Relevant links: [HTTP API reference](api.md), [Operations guide](operations.md) - -## HTTP Route Or Method Errors - -Symptom: - -- HTTP returns `404 not_found` or `405 method_not_allowed`. - -Likely cause: - -- Path is not `/v1/runs`. -- Method on `/v1/runs` is not `POST`. - -Diagnostic step: - -- Check the request URL and method. - -Safe fix: - -- Send `POST /v1/runs`. - -Relevant links: [HTTP API reference](api.md)