eric/scriptorium
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Consolidate operational recovery guidance

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-26 14:22:29 +00:00
parent ff31f8daf8
commit f0ca233c25
4 changed files with 117 additions and 486 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/development.md
View File

@@ -38,7 +38,7 @@ Start with:
| Runner orchestration, rendering, validation, or repair | [Runner internals](internal/runner.md) and [source internals](internal/sources.md) |
| OpenAI-compatible request or response behavior | [OpenAI-compatible integration](integrations/openai-compatible-chat.md), [LLM internals](internal/llm.md), [runner internals](internal/runner.md), and [adapter internals](internal/adapters.md) |
| Subprocess behavior | [Subprocess integration](integrations/subprocess.md) and [CLI contract](cli.md) |
| Runtime operation, recovery, or troubleshooting | [Operations](operations.md) and [troubleshooting](troubleshooting.md) |
| Runtime operation or recovery | [Operations](operations.md) |
| Examples or copyable assets | The owning contract for the demonstrated behavior and the related files under `examples/` |
| Architecture decisions or future work | The [documentation policy](policy/documentation.md), relevant accepted ADRs such as [ADR 0001](adr/0001-adopt-canonical-documentation-ownership.md), and relevant roadmap documents under `roadmap/` |

239
docs/operations.md
View File

@@ -1,164 +1,157 @@
# Operations Guide
## Scope
## Scope And References
This guide covers operating the implemented CLI commands and HTTP service. It
does not replace the [CLI reference](cli.md), [Configuration reference](config.md),
or [HTTP API reference](api.md).
This runbook covers deployment, normal operation, capacity planning, and safe
recovery for Scriptorium. It does not redefine invocation syntax, configuration
fields, or HTTP wire behavior.
## Operational Model
- [CLI reference](cli.md): commands, output destinations, and exit codes.
- [Configuration reference](config.md): configuration, prompt/profile/schema
formats, defaults, and credentials.
- [HTTP API reference](api.md): route, request/response schema, status codes,
limits, and HTTP artifact access.
- [Consumer integration overview](consumers/api.md): caller responsibilities.
Scriptorium executes one prompt request per CLI invocation or HTTP request.
## Operational Model And State
Important boundaries:
Scriptorium handles one prompt request for each CLI invocation or HTTP request.
It has no durable run store, archive, checkpoint, cache, or resume mechanism.
A failed or interrupted request is recovered by correcting its inputs,
configuration, or environment and submitting a new request.
- No durable run state is stored.
- No manifest, archive, checkpoint, or built-in backup workflow is written.
- No built-in resume behavior exists.
- Recovery is rerun-based: correct inputs, config, or environment, then run again.
Generated artifacts, rendered prompts, model output, and run metadata are
caller-owned data. Retention, encryption, backup, and deletion are deployment
responsibilities.
## Filesystem Layout
## Deploy The Filesystem And Process
Operational deployments usually provide:
Provide the process with readable prompt, profile, and schema sources. Keep
prompt templates adjacent to the prompt definitions that reference them. For an
HTTP deployment that accepts file artifacts, use a dedicated, narrow artifact
directory rather than a general-purpose or sensitive filesystem tree.
- `prompt_dir`: prompt definition YAML files and adjacent `content_file` templates.
- `profile_dir`: optional custom profile YAML files.
- `schema_dir`: optional JSON Schema files.
- `server.artifact_root`: optional HTTP file-input root for `serve`.
Run Scriptorium under an identity that can:
Keep these directories readable by the Scriptorium process. Keep
`server.artifact_root` narrow and not writable by untrusted users.
- read only the prompt, profile, schema, and allowed input-artifact paths it
needs;
- read the required credential environment variables without writing them to
files or logs; and
- write only caller-selected output locations when CLI output files are used.
## Normal CLI Workflow
Do not make the HTTP artifact directory writable by untrusted users. The HTTP
artifact containment behavior is lexical and the operating system follows
symlinks; account for that when choosing ownership and mount boundaries. See
the [HTTP API reference](api.md) for the externally observable behavior.
Use `render` before `run` when changing prompt/profile/input wiring:
## Supply Credentials And Protect Runtime Data
```bash
go run ./cmd/scriptorium render \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--format json
```
Set secret values in the process environment and configure only their
environment-variable names. Do not put raw keys in configuration, prompt or
profile files, process arguments, HTTP payloads, captured command lines, or
debug dumps.
Use `run` for generation after preflight:
Treat stdout, stderr, prepared-run output, generated artifacts, and HTTP
responses as potentially sensitive. Send service logs to a controlled collector
and apply the same retention and access rules as for model input and output.
```bash
go run ./cmd/scriptorium run \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--out ./summary.md
```
## Run A Normal Workflow
Before production runs, confirm:
Before changing production inputs, profiles, or schemas:
- the effective config path is the intended one;
- prompt/profile/schema directories are readable;
- input file paths exist and match prompt input names;
- required API-key environment variables are set;
- the selected model endpoint is reachable from the process environment.
1. confirm the deployed configuration selects the intended sources and model
credentials;
2. use [`render`](cli.md) with the same request inputs and variables to confirm
preparation without a model call;
3. use [`run`](cli.md) for generation; and
4. retain or discard validation-failed output according to the caller's
policy.
## HTTP Service Operation
The [maintained render script](../examples/render-markdown-summary.sh) is a
copyable preflight example. The CLI reference owns its complete invocation and
exit semantics.
Start the service with:
## Expose The HTTP Service
```bash
go run ./cmd/scriptorium serve --config ./examples/config.yml
```
The HTTP service has no built-in authentication or authorization. Place it on a
trusted network or behind an authenticated reverse proxy, API gateway, or
equivalent access control. Restrict who can reach it and who can read the
artifact root.
The implemented HTTP route is `POST /v1/runs`; request and response fields are
defined in the [HTTP API reference](api.md).
Use a service manager or supervisor appropriate to the deployment to manage
process lifetime, restart policy, log capture, and environment injection. The
[HTTP API reference](api.md) owns client request shapes, status behavior, and
artifact-access outcomes.
The maintained HTTP request-shape example is `examples/http-run.json`.
## Plan Capacity And Limits
HTTP service notes:
Capacity is primarily determined by concurrent model calls, input and output
sizes, schema complexity, provider latency, and network behavior. Size limits
protect request bodies, HTTP file artifacts, and encoded responses; configure
them through the [configuration reference](config.md) and rely on the
[HTTP API reference](api.md) for their response effects.
- Unknown JSON fields are rejected.
- `inline` input references work without an artifact root.
- `file` input references require `server.artifact_root` or `serve --artifact-root`.
- Request bodies, HTTP file input artifacts, and encoded JSON responses are size-limited.
- Validation content failures return `200 OK` with `validation.status: "failed"`.
Before increasing a limit:
Security boundary:
1. measure representative input, generated-output, and optional raw-output
sizes;
2. confirm memory, network, and upstream-provider capacity;
3. retain an upstream request-size and authentication boundary; and
4. test the intended workload in a non-production environment.
- `serve` has no built-in authentication or authorization.
- Put it behind trusted controls such as a private network, authenticated reverse proxy, or API gateway.
- Do not expose an artifact root containing unrelated sensitive files.
- Symlinks inside the artifact root are followed by the operating system.
For large local inputs, prefer a controlled file-artifact directory over
placing arbitrary paths on the service host. Avoid disabling a limit unless an
equivalent trusted control exists elsewhere.
## Secrets Handling
## Diagnose And Recover
Raw API keys are not accepted in app config, profiles, CLI flags, or HTTP
request bodies.
### Preparation Or Configuration Failure
Use this pattern:
Capture the CLI diagnostic or HTTP error response, then verify the selected
configuration, prompt ID, profile selection, source readability, and input
mapping. Use `render` with the same request when it is unclear whether failure
occurs before model execution. Consult the [CLI reference](cli.md), the
[configuration reference](config.md), and the [HTTP API reference](api.md) for
the exact interface contract.
1. Set an environment variable containing the secret value.
2. Store only the variable name in profile `api_key_env` or request override `api_key_env`.
3. Scope the process environment to the minimum required variables.
### Credential Or Provider Failure
## Output, Logs, And Exit Codes
Confirm that the process environment contains the configured credential name
without printing the secret. Check endpoint reachability and provider health
from the process network. If preparation succeeds but generation fails, inspect
the selected model settings in prepared output and the service's controlled
logs. Correct the deployment or provider issue, then submit a new request.
`run`:
### Artifact Or Permission Failure
- stdout: generated artifact body unless `--out` is used.
- stderr: summary on success, errors on failure.
- exit `2`: generation completed and output was written, but validation failed.
Verify that the process can read the intended local input. For HTTP file
artifacts, verify the deployment's artifact root, ownership, path layout, and
file size. Do not widen filesystem permissions or the allowed root merely to
make an arbitrary path work; move or copy the required artifact into the
controlled location instead.
`render`:
### Validation Failure
- stdout: prepared-run output unless `--out` is used.
- stderr: errors.
- exit `0` on success, `1` on failure.
A generated-content validation failure is distinct from a runtime failure.
CLI `run` reports the validation result and error count in its success summary;
it does not print the individual validation messages. For HTTP, inspect the
validation object in the response according to the [HTTP API reference](api.md).
`serve`:
Use rendered input and generated output to determine whether prompt instructions,
the selected model, or the schema needs correction. If schema loading or
compilation itself fails, correct the source deployment or schema document
before rerunning.
- stderr: startup and server errors.
- HTTP response body: JSON success or error envelope.
### HTTP Limit Or Request Failure
## Validation Behavior
Compare the request, artifact, or expected response size with the deployed
configuration, and validate the request against the [HTTP API reference](api.md).
Reduce the payload, use an appropriate controlled artifact source, omit
unneeded raw output, or adjust the deployment limit after capacity review.
Prompt `output.validation_mode` controls validation:
## Cleanup And Reruns
- `none`: skipped.
- `basic`: output body must not be empty.
- `json`: output body must parse as JSON.
- `json_schema`: output body must parse as JSON and satisfy the configured schema.
Runtime/schema failures are hard failures (`run` exit `1`, HTTP error).
Generated-content validation failures are soft failures (`run` exit `2`, HTTP
`200 OK` with failed validation status).
## Size Limits
Defaults are documented in [Configuration reference](config.md). Operationally:
- Keep default HTTP limits unless larger payloads are measured and expected.
- Prefer `inline` HTTP inputs for small payloads.
- Prefer `file` HTTP inputs for larger local artifacts under a controlled artifact root.
- Increase `server.max_response_bytes` when generated artifacts or requested raw output are expected to be large.
- Use `0` only when another trusted layer enforces size limits.
## Maintained Examples
- `examples/config.yml`
- `examples/config.full.yml`
- `examples/render-markdown-summary.sh`
- `examples/http-run.json`
## Safe Recovery
For failed CLI commands or HTTP requests:
1. Capture stderr or the HTTP error `code` and `message`.
2. Confirm config path and effective directory settings.
3. Verify prompt ID, profile ID, schema path, and input mappings.
4. Verify required API-key environment variables.
5. Reproduce with `render --format json` when pre-LLM resolution is uncertain.
6. Rerun after correction.
Because Scriptorium does not persist run state, rerun is the supported recovery
path.
Because no run state is retained, cleanup concerns caller-owned output files,
logs, and artifacts only. Remove or rotate them using the deployment's normal
retention policy. After a correction, rerun the request from the beginning;
there is no safe resume point.

361
docs/troubleshooting.md
View File

@@ -1,361 +0,0 @@
# Troubleshooting
This guide lists common implemented failure modes and safe fixes.
Canonical references:
- [CLI reference](cli.md)
- [Configuration reference](config.md)
- [HTTP API reference](api.md)
- [Operations guide](operations.md)
## Missing Or Invalid Config
Symptom:
- CLI error includes `application config error`, `config file not found`, `invalid config YAML`, or `invalid config`.
Likely cause:
- `--config` points to a missing file.
- YAML syntax is invalid.
- Config contains unknown fields or negative HTTP size limits.
Diagnostic step:
```bash
go run ./cmd/scriptorium render --config /path/to/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml
```
Safe fix:
- Correct the config path.
- Fix YAML syntax.
- Remove unknown fields.
- Keep raw secrets out of config.
Relevant links: [Configuration reference](config.md), [CLI reference](cli.md)
## Missing Prompt Directory
Symptom:
- CLI parse error says the prompt directory is required.
Likely cause:
- Neither config nor CLI flags provide an effective `prompt_dir`.
Diagnostic step:
- Re-run once with explicit `--prompt-dir`.
Safe fix:
- Set `prompt_dir` in config or pass `--prompt-dir`.
Relevant links: [Configuration reference](config.md), [CLI reference](cli.md)
## Unknown Flags
Symptom:
- CLI parse error for an unknown flag.
Likely cause:
- Typo.
- Flag is valid for another command.
- `serve` was given runtime model override flags.
Diagnostic step:
- Compare the command with the command-specific flag list.
Safe fix:
- Remove unsupported flags.
- Use `run` or `render` for runtime model overrides.
Relevant links: [CLI reference](cli.md)
## Prompt Load Failures
Symptom:
- CLI run/render fails during prompt loading.
- HTTP returns `404 prompt_not_found` or `400 prompt_load_failed`.
Likely cause:
- Prompt ID/version does not exist.
- Prompt YAML is invalid or has unknown fields.
- Prompt contract is invalid, such as missing messages, invalid output mode, bad `content_file`, or missing `schema_path` for `json_schema`.
Diagnostic step:
```bash
go run ./cmd/scriptorium render --config ./examples/config.yml --prompt <prompt-id> --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml --format json
```
Safe fix:
- Correct prompt ID/version.
- Fix prompt YAML and referenced `content_file` paths.
- Fix output contract fields.
Relevant links: [Configuration reference](config.md), [CLI reference](cli.md)
## Profile Load Failures
Symptom:
- CLI run/render fails during profile loading.
- HTTP returns `404 profile_not_found`, `400 profile_load_failed`, or `400 profile_required`.
Likely cause:
- Profile ID does not exist.
- Request omitted profile and prompt has no `default_profile`.
- Profile YAML is invalid or has unknown fields.
- Profile contains raw `api_key`.
Diagnostic step:
```bash
go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --profile <profile-id> --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml
```
Safe fix:
- Correct profile ID or prompt `default_profile`.
- Fix profile YAML and value ranges.
- Replace raw `api_key` with `api_key_env`.
Relevant links: [Configuration reference](config.md), [CLI reference](cli.md)
## Input Artifact Failures
Symptom:
- CLI run/render fails while reading inputs.
- HTTP returns `400 artifact_read_failed`, `400 artifact_not_allowed`, or `413 artifact_too_large`.
Likely cause:
- Input file path is missing or unreadable.
- HTTP input type is unsupported or missing required fields.
- HTTP file refs are disabled because no artifact root is configured.
- HTTP file path is lexically outside the artifact root.
- HTTP file input exceeds `server.max_artifact_bytes`.
Diagnostic step:
- Verify each input path exists and is readable by the process.
- For HTTP, verify input refs use `file` or `inline`.
- For HTTP file refs, verify the artifact root and compare file size to `server.max_artifact_bytes`.
Safe fix:
- Correct paths and permissions.
- Configure a narrow artifact root for HTTP file refs.
- Use relative paths under the artifact root or switch to `inline`.
- Increase `server.max_artifact_bytes` only for expected larger inputs.
Relevant links: [HTTP API reference](api.md), [Configuration reference](config.md)
## Missing API-Key Environment Variable
Symptom:
- CLI render/run fails with an API-key environment error.
- HTTP returns `400 api_key_env_missing`.
Likely cause:
- Selected profile or runtime override sets `api_key_env`, but the environment variable is unset or empty.
Diagnostic step:
```bash
printenv SCRIPTORIUM_API_KEY
```
Safe fix:
- Set the required environment variable before starting the CLI command or HTTP service.
- Or use a profile that does not require provider API-key auth.
Relevant links: [Configuration reference](config.md), [Operations guide](operations.md)
## Prompt Template Render Failures
Symptom:
- CLI render/run fails during prompt rendering.
- HTTP returns `400 prompt_render_failed`.
Likely cause:
- Template references an input that was not supplied.
- Template syntax or variable reference is invalid.
Diagnostic step:
- Run `render --format json` with the same prompt, inputs, vars, and profile.
Safe fix:
- Align `{{input "name"}}` references with request input names.
- Fix template syntax and variable names.
Relevant links: [Configuration reference](config.md), [CLI reference](cli.md)
## LLM Request Failures
Symptom:
- CLI `run` fails during generation.
- HTTP returns `502 llm_failed`.
Likely cause:
- Endpoint is unreachable.
- Provider returns non-2xx.
- Request times out.
- Provider response is malformed.
Diagnostic step:
- Run `render` first to confirm pre-LLM preparation works.
- Check selected endpoint/model in prepared output.
- Check network/provider logs for timeout or non-2xx details.
Safe fix:
- Correct endpoint/model/profile settings.
- Adjust timeout when appropriate.
- Resolve provider or network issue.
Relevant links: [Operations guide](operations.md), [Configuration reference](config.md)
## Validation Failed
Symptom:
- CLI `run` exits `2`.
- HTTP returns `200 OK` with `validation.status` set to `failed`.
Likely cause:
- Generated output failed `basic`, `json`, or `json_schema` content validation.
Diagnostic step:
- Inspect validation errors in CLI stderr or the HTTP response.
Safe fix:
- Refine prompt instructions.
- Adjust schema or model/profile settings.
- Rerun after correction.
Relevant links: [Operations guide](operations.md), [HTTP API reference](api.md)
## Validation Runtime Failure
Symptom:
- CLI `run` fails with validation runtime error.
- HTTP returns `500 validation_runtime_failed`.
Likely cause:
- `json_schema` schema file is missing or unreadable.
- Schema JSON is invalid.
Diagnostic step:
- Verify `schema_dir` and prompt `output.schema_path`.
- Check schema file readability and JSON syntax.
Safe fix:
- Correct schema path or permissions.
- Fix schema JSON.
- Rerun.
Relevant links: [Configuration reference](config.md), [Operations guide](operations.md)
## HTTP JSON Or Request Contract Errors
Symptom:
- HTTP returns `400 invalid_json` or `400 invalid_request`.
Likely cause:
- JSON body is malformed.
- Request has unknown fields or trailing JSON tokens.
- Required `prompt_id` or `inputs` is missing.
- Runtime override values are out of range.
- `extra_params` collides with reserved outbound fields.
Diagnostic step:
- Revalidate request JSON and compare fields with the API reference.
Safe fix:
- Send one JSON object with only supported fields.
- Include `prompt_id` and at least one input.
- Use valid model override ranges.
- Remove reserved `extra_params` keys.
Relevant links: [HTTP API reference](api.md)
## HTTP Size Limit Errors
Symptom:
- HTTP returns `413 request_too_large`, `413 artifact_too_large`, or `413 response_too_large`.
Likely cause:
- JSON request body exceeds `server.max_request_bytes`.
- HTTP file input exceeds `server.max_artifact_bytes`.
- Encoded JSON response exceeds `server.max_response_bytes`.
Diagnostic step:
- Compare request, file input, and expected response sizes with configured limits.
Safe fix:
- Use smaller inline inputs or switch to file inputs under the artifact root.
- Reduce generated output size.
- Omit `include_raw_output`.
- Increase limits only when the deployment expects larger payloads.
Relevant links: [HTTP API reference](api.md), [Operations guide](operations.md)
## HTTP Route Or Method Errors
Symptom:
- HTTP returns `404 not_found` or `405 method_not_allowed`.
Likely cause:
- Path is not `/v1/runs`.
- Method on `/v1/runs` is not `POST`.
Diagnostic step:
- Check the request URL and method.
Safe fix:
- Send `POST /v1/runs`.
Relevant links: [HTTP API reference](api.md)