From aabd89aea782f330e2bcbd6db44db12e89f37514 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 4 Jul 2026 21:34:03 -0500 Subject: [PATCH] Add a documentation update roadmap --- docs/roadmap/documentation.md | 569 ++++++++++++++++++++++++++++++++++ 1 file changed, 569 insertions(+) create mode 100644 docs/roadmap/documentation.md diff --git a/docs/roadmap/documentation.md b/docs/roadmap/documentation.md new file mode 100644 index 0000000..266949b --- /dev/null +++ b/docs/roadmap/documentation.md @@ -0,0 +1,569 @@ +# Documentation Roadmap + +## Purpose + +This roadmap defines the work required to bring Scriptorium's documentation into compliance with `docs/policy/documentation.md` and the current implementation. It is grounded in the repository state at the time of writing and should guide future documentation changes without rewriting current-behavior docs in this planning pass. + +## Repository Documentation Inventory + +- `README.md` - keep and rewrite. It is concise and has a valid quickstart, but it links to `docs/integrations/http-api.md` as the HTTP contract instead of the policy-required `docs/api.md`. +- `AGENTS.md` - keep and lightly update. It correctly points coding agents at policy docs; keep it short and policy-oriented. +- `docs/policy/documentation.md` - keep and lightly update only if policy itself changes. It is the controlling documentation policy and should not be rewritten as part of ordinary docs refresh work. +- `docs/policy/architecture.md` - keep and lightly update. It is mostly current, but its package map and integration-doc guidance should be checked against the current tree, including `internal/defaults`, `internal/filecatalog`, and the public package. +- `docs/policy/development.md` - keep and lightly update. It is required contributor guidance and should reflect any final target docs layout and validation commands. +- `docs/cli.md` - keep and rewrite. It is the canonical CLI reference, but should be checked against `internal/adapter/cli/run.go` after recent serve flags and defaults. +- `docs/config.md` - keep and rewrite. It is the canonical config and prompt/profile/schema file-format reference, but it is long and should be tightened around implemented behavior and source-of-truth code. +- `docs/operations.md` - keep and rewrite. It should focus on operating the CLI and HTTP service, not repeat full CLI/API/config references. +- `docs/troubleshooting.md` - keep and rewrite. It contains useful failure-mode entries, but should be shorter, link to canonical references, and be checked against current error codes. +- `docs/consumers/api.md` - keep and rewrite. It is currently too thin for the policy-required consumer overview and should explain the recommended integration choices: Go package, CLI subprocess, and HTTP. +- `docs/consumers/pkg-scriptorium.md` - keep and rewrite. It should be the canonical public Go package guide and should be checked against `engine.go`, `types.go`, `profiles.go`, `llm_adapter.go`, and tests. +- `docs/integrations/http-api.md` - move or merge. The public HTTP endpoint contract belongs at `docs/api.md`; this file should be deleted after its accurate content is merged, or replaced only by a short pointer if the project intentionally keeps redirects. +- `docs/integrations/openai-compatible-chat.md` - keep and lightly update. It documents the outbound OpenAI-compatible contract implemented by `internal/llm`. +- `docs/integrations/narratio.md` - move or merge. The repository implements a generic CLI subprocess contract, not a Narratio-specific adapter. Generalize this into a non-product-specific subprocess integration doc or merge it into `docs/consumers/api.md`. +- `docs/internal/runner.md` - keep and rewrite. It is a useful internal component doc and should be checked against `internal/usecase`. +- `docs/internal/adapters.md` - split. Keep adapter behavior here, but move repository/source-loading details into a separate internal source/repository doc if they make the adapter doc too broad. +- `docs/roadmap/cleanup.md` - delete or reduce after verification. It appears to describe work that is now largely implemented. Completed implementation plans should not be kept as current-behavior documentation. +- `docs/roadmap/documentation.md` - create new. This roadmap is the current planning deliverable. +- `examples/config.yml` - keep and lightly update. It is a working repository example used by smoke commands. +- `examples/render-markdown-summary.sh` - keep and lightly update. It is a runnable CLI render example. +- `examples/http-run.json` - keep and lightly update. It is a real HTTP request example, but docs must state it requires `serve` with an artifact root and a reachable model endpoint. +- `examples/go-library/prepare/main.go` - keep and lightly update. It is a runnable public package example. +- `examples/prompts/` - keep and lightly update. These are valid prompt definitions with `content_file` usage and structured-output coverage. +- `examples/profiles/` - keep and lightly update. These are custom file-backed profiles for local testing. +- `examples/schemas/` - keep and lightly update. These are real JSON Schema examples used by prompt definitions. +- `examples/fixtures/` - keep and lightly update. These are sample inputs used by tests and examples. +- `internal/profile/builtin/assets/` - not documentation, but inspect while updating config/profile docs. It is the source of truth for the built-in profile catalog. + +## Policy Compliance Assessment + +Required documents missing: + +- `docs/api.md` is missing. The project exposes HTTP `POST /v1/runs`, so the policy requires `docs/api.md` as the canonical public HTTP API contract. + +Recommended or useful documents to add: + +- `docs/integrations/subprocess.md` should replace the product-specific subprocess integration doc if the project wants to keep a maintained subprocess contract outside the CLI reference. +- `docs/internal/sources.md` should be added if `docs/internal/adapters.md` remains too broad. It would cover prompt/profile/schema/artifact repositories and source resolution. +- `examples/config.full.yml` or `examples/config.http.yml` is recommended if operators need a maintained full or HTTP-oriented config example beyond the minimal `examples/config.yml`. + +Documents stale or in the wrong canonical home: + +- `docs/integrations/http-api.md` is in the wrong home. Its endpoint reference belongs in `docs/api.md`. +- `README.md`, `docs/cli.md`, `docs/config.md`, `docs/troubleshooting.md`, and `docs/consumers/api.md` link to `docs/integrations/http-api.md`; those links must point to `docs/api.md` after the move. +- `docs/integrations/narratio.md` is too product-specific for repository evidence. The implemented contract is generic CLI subprocess use. +- `docs/roadmap/cleanup.md` appears completed and should not remain as an active current-behavior source. + +Content that may describe historical, planned, or unimplemented behavior outside `docs/roadmap/`: + +- Audit all non-roadmap docs for "future", "planned", "coming", "not implemented", "roadmap", "may add", and similar terms. +- Keep statements about unimplemented strict realpath artifact containment only in roadmap docs. Current implemented behavior is lexical containment with symlinks followed. +- Avoid implying HTTP authentication exists. Current `serve` is unauthenticated and should be deployed behind trusted controls. + +Examples that are missing, stale, invalid, or untested: + +- Existing examples are valid enough for current smoke checks: + - `go run ./examples/go-library/prepare` + - `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --input glossary=./examples/fixtures/glossary.yml --format json` +- No automated HTTP example smoke exists because the HTTP endpoint calls an LLM endpoint. The docs should mark `examples/http-run.json` as a request-shape example unless a fake model server example is added later. +- There is no full config example covering all current config fields, including HTTP size limits. + +Links likely stale or needing verification: + +- All links to `docs/integrations/http-api.md`. +- README links after creating `docs/api.md` and generic subprocess docs. +- Links from `docs/consumers/api.md` to HTTP and package docs. +- Relative links in `docs/troubleshooting.md`, which has many repeated "Relevant links" sections. +- Any links to roadmap docs from non-roadmap docs should be removed unless explicitly describing future work. + +## Target Documentation Set + +### `README.md` + +- Audience: users, administrators, operators. +- Purpose: concise project orientation and shortest useful command. +- Canonical scope: description, quickstart, and links to task-specific docs. +- Recommended section outline: + - `# scriptorium` + - one-paragraph description + - `## Quickstart` + - `## Documentation` + - `## Examples` +- Source-of-truth areas: `cmd/scriptorium/main.go`, `internal/adapter/cli/run.go`, `examples/config.yml`, `examples/render-markdown-summary.sh`. +- Acceptance criteria: under roughly 60 lines; quickstart command runs; links point to existing canonical docs; no detailed config, API, or package reference material. + +### `AGENTS.md` + +- Audience: LLM coding agents and contributors. +- Purpose: direct agents to required policy docs. +- Canonical scope: policy-read reminder only. +- Recommended section outline: + - policy docs to read before code changes + - policy docs to read before docs changes +- Source-of-truth areas: `docs/policy/`. +- Acceptance criteria: short; no duplicate policy content; links or paths are accurate. + +### `docs/api.md` + +- Audience: external HTTP API consumers, developers, LLM coding agents integrating over HTTP. +- Purpose: canonical public HTTP API reference. +- Canonical scope: implemented HTTP `POST /v1/runs` contract only. +- Recommended section outline: + - `# HTTP API Reference` + - base URL and route + - authentication and deployment boundary + - request headers and JSON rules + - `POST /v1/runs` + - request fields + - response fields + - validation failure behavior + - error envelope and status codes + - request/response examples + - limits and artifact-root behavior +- Source-of-truth areas: `internal/adapter/http/handler.go`, `internal/adapter/http/dto.go`, `internal/adapter/http/handler_test.go`, `internal/adapter/cli/run.go`, `internal/config/config.go`. +- Acceptance criteria: replaces `docs/integrations/http-api.md` as canonical endpoint reference; documents `request_too_large`, `artifact_too_large`, and `response_too_large`; states raw API keys are not accepted by HTTP payloads; documents strict JSON and trailing-token rejection. + +### `docs/cli.md` + +- Audience: users, administrators, operators. +- Purpose: canonical CLI reference. +- Canonical scope: `run`, `render`, and `serve` syntax, flags, workflows, exit codes. +- Recommended section outline: + - shortest useful command + - command overview + - common argument rules + - flag reference by command + - input and variable mapping syntax + - output behavior + - exit codes + - common workflows + - links to config, API, and subprocess docs +- Source-of-truth areas: `internal/adapter/cli/run.go`, `internal/adapter/cli/run_test.go`, `internal/format/prepared_run.go`, examples. +- Acceptance criteria: every documented flag exists; deprecated aliases are labeled; `serve` includes artifact-root and size-limit flags; no HTTP endpoint schema duplication beyond links to `docs/api.md`. + +### `docs/config.md` + +- Audience: administrators, operators, advanced users. +- Purpose: canonical config and YAML file-format reference. +- Canonical scope: app config, prompt definitions, profile definitions, built-in profile catalog, schemas, artifact refs, secrets handling. +- Recommended section outline: + - config discovery and precedence + - minimal working config + - production-oriented config + - full app config reference + - prompt definition files + - profile definition files and built-ins + - schema behavior + - artifact reference behavior + - secrets handling + - maintained examples + - integration references +- Source-of-truth areas: `internal/config/config.go`, `internal/defaults/defaults.go`, `internal/promptdef/filesystem_repository.go`, `internal/profile/filesystem_repository.go`, `internal/profile/builtin/assets/`, `internal/validate/standard_validator.go`, tests under matching packages. +- Acceptance criteria: all defaults match code; built-in profile catalog matches asset files; no raw secret examples; no repeated HTTP endpoint reference beyond link to `docs/api.md`. + +### `docs/operations.md` + +- Audience: administrators and operators. +- Purpose: operating guidance for CLI and HTTP service. +- Canonical scope: normal workflow, filesystem layout, service deployment caveats, recovery, cleanup, and exit/status handling. +- Recommended section outline: + - scope + - operational model + - filesystem layout + - normal CLI workflow + - HTTP service operation + - secrets handling + - output, logs, and exit codes + - validation behavior + - safe recovery steps +- Source-of-truth areas: `internal/adapter/cli/run.go`, `internal/config/config.go`, `internal/adapter/http/handler.go`, `docs/policy/architecture.md`. +- Acceptance criteria: links to CLI/config/API instead of duplicating references; states no durable run-state store; states HTTP service should be protected externally; documents current artifact-root and size-limit behavior. + +### `docs/troubleshooting.md` + +- Audience: administrators and operators. +- Purpose: symptom-based recovery guide. +- Canonical scope: common current failure modes and safe fixes. +- Recommended section outline: + - missing config + - missing prompt directory + - unknown flags + - prompt/profile load failures + - input artifact failures + - missing API-key environment variables + - LLM request failures + - validation failures + - HTTP request/response errors +- Source-of-truth areas: `internal/adapter/cli/run.go`, `internal/adapter/http/handler.go`, `internal/usecase/runner.go`, `internal/llm/openai_compatible_client.go`, tests. +- Acceptance criteria: each entry has symptom, likely cause, diagnostic step, safe fix, and relevant links; no duplicate long reference tables; error codes match code. + +### `docs/consumers/api.md` + +- Audience: downstream application developers and LLM coding agents integrating Scriptorium. +- Purpose: consumer-facing overview and recommended workflows. +- Canonical scope: choosing between Go package, CLI subprocess, and HTTP API; consumer responsibilities; retry/idempotency boundaries. +- Recommended section outline: + - intended consumers and use cases + - integration surfaces + - recommended workflow by use case + - required deployment inputs + - minimal Go package example + - subprocess workflow + - HTTP workflow link + - consumer responsibilities and boundaries + - retries, idempotency, and status behavior +- Source-of-truth areas: public package files, `internal/adapter/cli/run.go`, `docs/api.md`, examples. +- Acceptance criteria: no endpoint field tables duplicated from `docs/api.md`; no package type reference duplicated from `pkg-scriptorium.md`; links are canonical. + +### `docs/consumers/pkg-scriptorium.md` + +- Audience: Go developers importing `gitea.maximumdirect.net/eric/scriptorium`. +- Purpose: canonical public Go package guide. +- Canonical scope: `NewEngine`, `Config`, options, prompts/profiles/schema sources, `Prepare`, `Run`, injected LLM clients, errors. +- Recommended section outline: + - import path + - intended use cases + - construct an engine + - source options + - in-memory profiles + - prepare workflow + - run workflow + - injected LLM clients + - overrides and API keys + - errors and validation behavior + - examples +- Source-of-truth areas: `engine.go`, `types.go`, `profiles.go`, `llm_adapter.go`, `errors.go`, `engine_test.go`, `examples/go-library/prepare/main.go`. +- Acceptance criteria: documents direct `RunRequest.APIKey`; states raw API keys do not belong in profiles; describes source precedence; documents `WithProfiles`, `OpenAICompatibleProfile`, `WithPromptFS`, `WithPromptFile`, `WithProfileFS`, `WithProfileFile`, `WithSchemaFS`, and `WithSchemaFile`. + +### `docs/internal/runner.md` + +- Audience: developers and LLM coding agents. +- Purpose: internal runner orchestration reference. +- Canonical scope: `Runner.Prepare`, `Runner.Run`, validation and repair hook behavior, error boundaries. +- Recommended section outline: + - purpose + - inputs and outputs + - dependencies + - prepare flow + - run flow + - validation and repair + - failure behavior + - tests to inspect + - architectural invariants +- Source-of-truth areas: `internal/usecase/runner.go`, `internal/usecase/repairer.go`, `internal/usecase/runner_test.go`, `internal/usecase/integration_test.go`. +- Acceptance criteria: no adapter DTO details; accurately states `Run` reuses `Prepare`; notes CLI/HTTP instantiate without repairer. + +### `docs/internal/adapters.md` + +- Audience: developers and LLM coding agents. +- Purpose: adapter behavior and boundaries. +- Canonical scope: CLI adapter, HTTP adapter, public package adapter wiring, config handoff. +- Recommended section outline: + - purpose + - adapter map + - inputs and outputs + - boundaries + - config fields used + - failure behavior + - tests to inspect + - architectural invariants +- Source-of-truth areas: `internal/adapter/cli/run.go`, `internal/adapter/http/handler.go`, `engine.go`, `llm_adapter.go`, tests. +- Acceptance criteria: no long prompt/profile schema reference; links to `docs/internal/sources.md` or `docs/config.md` for source/file details; keeps adapter logic thin. + +### `docs/internal/sources.md` + +- Audience: developers and LLM coding agents. +- Purpose: implemented prompt/profile/schema/artifact source behavior. +- Canonical scope: repositories, built-in profile overlay, filecatalog helpers, artifact readers, schema loaders. +- Recommended section outline: + - purpose + - prompt definition sources + - profile sources and built-in overlay + - schema sources + - artifact readers + - path containment and symlink behavior + - failure behavior + - tests to inspect + - architectural invariants +- Source-of-truth areas: `internal/promptdef`, `internal/profile`, `internal/profile/builtin`, `internal/filecatalog`, `internal/artifact`, `internal/validate`. +- Acceptance criteria: describes only implemented source behavior; does not duplicate full user-facing YAML references from `docs/config.md`. + +### `docs/integrations/openai-compatible-chat.md` + +- Audience: developers and LLM coding agents maintaining the outbound provider adapter. +- Purpose: outbound OpenAI-compatible chat-completions contract. +- Canonical scope: fields Scriptorium sends and response fields it consumes. +- Recommended section outline: + - scope + - endpoint construction + - request fields sent + - auth header behavior + - timeout behavior + - response expectations + - error handling + - unsupported fields + - relationship to runner +- Source-of-truth areas: `internal/llm/openai_compatible_client.go`, `internal/llm/openai_compatible_client_test.go`. +- Acceptance criteria: documents explicit zero numeric override behavior; documents provider body redaction for non-2xx errors; does not describe unused OpenAI API features. + +### `docs/integrations/subprocess.md` + +- Audience: developers and LLM coding agents integrating Scriptorium as a subprocess. +- Purpose: generic CLI subprocess contract for downstream applications. +- Canonical scope: stable invocation shapes, stdout/stderr separation, exit codes, config and environment expectations. +- Recommended section outline: + - purpose + - supported commands + - recommended invocation shapes + - config and directory behavior + - input and variable contract + - environment contract + - stdout/stderr and exit status + - security notes + - canonical links +- Source-of-truth areas: `internal/adapter/cli/run.go`, `docs/cli.md`, `docs/operations.md`, CLI tests. +- Acceptance criteria: generic, not tied to a specific downstream application; no duplicate flag reference beyond stable examples and links. + +### `docs/policy/architecture.md` + +- Audience: developers and LLM coding agents. +- Purpose: development architecture policy. +- Canonical scope: stable principles and invariants, not detailed flags or endpoint fields. +- Recommended section outline: keep current structure. +- Source-of-truth areas: current package tree, architecture policy itself. +- Acceptance criteria: package map matches current packages; references target internal docs; does not document volatile details. + +### `docs/policy/development.md` + +- Audience: contributors and LLM coding agents. +- Purpose: contributor workflow. +- Canonical scope: repository layout, build/test commands, coding and documentation conventions. +- Recommended section outline: keep current structure, add docs validation expectations after the migration. +- Source-of-truth areas: current package tree, common commands, examples. +- Acceptance criteria: commands run; docs validation guidance matches available tooling. + +### `docs/roadmap/` + +- Audience: maintainers, developers, LLM coding agents. +- Purpose: active future work and implementation plans only. +- Canonical scope: proposed or accepted work not yet reflected in current-behavior docs. +- Recommended section outline: one roadmap file per active plan. +- Source-of-truth areas: current code and accepted product decisions. +- Acceptance criteria: completed plans are removed or reduced to remaining future work; non-roadmap docs do not link to completed plans as current behavior. + +## File-by-File Rewrite Guidance + +- `README.md`: cover what Scriptorium is, the render quickstart, and links. Avoid full CLI/config/API explanations. Inspect `examples/config.yml`, `examples/render-markdown-summary.sh`, and `docs/cli.md`. Do not keep the `docs/integrations/http-api.md` link after `docs/api.md` exists. +- `docs/api.md`: create from the accurate parts of `docs/integrations/http-api.md`. Cover one route only. Avoid upstream LLM details and public Go package details. Inspect HTTP handler DTOs and tests. Do not document authentication as implemented. +- `docs/cli.md`: rewrite from `internal/adapter/cli/run.go`. Cover flags by command and common workflows. Avoid repeating config schema or HTTP response fields. Inspect CLI tests for parse behavior and exit codes. Do not omit `--max-request-bytes`, `--max-artifact-bytes`, or `--max-response-bytes`. +- `docs/config.md`: rewrite as the canonical config and YAML format reference. Avoid long operational advice and HTTP endpoint tables. Inspect config, promptdef, profile, built-in assets, validator, and examples. Do not carry stale example paths that do not exist. +- `docs/operations.md`: focus on running and recovering. Link to CLI/config/API instead of duplicating them. Inspect serve wiring and config defaults. Do not imply Scriptorium has persistent run state or built-in auth. +- `docs/troubleshooting.md`: keep symptom-driven entries. Avoid repeating full commands under every entry when a shorter diagnostic is enough. Inspect error mapping in CLI, HTTP, usecase, llm, and validators. Do not include provider response body snippets as a default diagnostic because they are redacted. +- `docs/consumers/api.md`: rewrite as a consumer decision guide. Link to `docs/api.md`, `docs/cli.md`, `docs/integrations/subprocess.md`, and `docs/consumers/pkg-scriptorium.md`. Avoid endpoint tables and type catalogs. +- `docs/consumers/pkg-scriptorium.md`: rewrite from public package code and tests. Cover public options and error categories. Avoid internal package names except where necessary to explain boundaries. Do not say a credential resolver exists; direct `RunRequest.APIKey` and profile `api_key_env` are the implemented mechanisms. +- `docs/internal/runner.md`: rewrite from usecase code. Keep it developer-facing. Avoid public API tutorials and operator procedures. +- `docs/internal/adapters.md`: narrow to adapters and wiring. Move repository/source detail to `docs/internal/sources.md` if created. Avoid full HTTP API schemas. +- `docs/internal/sources.md`: create if splitting adapter docs. Cover repository and source behavior. Avoid duplicating the full prompt/profile schema from `docs/config.md`. +- `docs/integrations/openai-compatible-chat.md`: update from the LLM client. Avoid documenting OpenAI-compatible features not serialized or parsed by code. +- `docs/integrations/subprocess.md`: create by generalizing useful parts of `docs/integrations/narratio.md`. Avoid naming a downstream product as the generic contract. +- `docs/integrations/http-api.md`: delete after `docs/api.md` exists and links are updated. If a temporary pointer file is kept, it should contain only a link to `docs/api.md` and should be removed in a later cleanup. +- `docs/integrations/narratio.md`: delete after generic subprocess docs exist unless maintainers confirm a product-specific integration doc is still required. +- `docs/policy/architecture.md`: lightly update package map and internal doc references only. Avoid volatile details. +- `docs/policy/development.md`: lightly update docs validation workflow after examples and target docs settle. +- `docs/roadmap/cleanup.md`: remove or reduce after confirming it no longer tracks active future work. + +## Examples Plan + +- `examples/config.yml` + - Purpose: minimal working repository config. + - Expected validity check: render smoke command using this config. + - Link from: README, `docs/config.md`, `docs/cli.md`, `docs/operations.md`. +- `examples/render-markdown-summary.sh` + - Purpose: copyable CLI render example. + - Expected validity check: run the script from repo root. + - Link from: README, `docs/cli.md`, `docs/config.md`. +- `examples/http-run.json` + - Purpose: HTTP request-shape example for `POST /v1/runs`. + - Expected validity check: JSON parses; field names match `internal/adapter/http/dto.go`; full request requires a running server and reachable model endpoint. + - Link from: `docs/api.md`, `docs/operations.md`. +- `examples/go-library/prepare/main.go` + - Purpose: public Go package prepare example. + - Expected validity check: `go run ./examples/go-library/prepare`. + - Link from: README, `docs/consumers/api.md`, `docs/consumers/pkg-scriptorium.md`. +- `examples/prompts/` + - Purpose: maintained prompt definition examples including `content_file` and structured output. + - Expected validity check: covered by public and internal tests plus render smoke command. + - Link from: `docs/config.md`, `docs/consumers/pkg-scriptorium.md`. +- `examples/profiles/` + - Purpose: custom file-backed profile examples. + - Expected validity check: used by render and library smoke commands. + - Link from: `docs/config.md`, `docs/operations.md`. +- `examples/schemas/` + - Purpose: JSON Schema validation example. + - Expected validity check: structured-output tests and config reference review. + - Link from: `docs/config.md`, `docs/consumers/pkg-scriptorium.md`. +- `examples/config.full.yml` - recommended create. + - Purpose: maintained full app config example covering all current config fields, including HTTP limits. + - Expected validity check: load with `go run ./cmd/scriptorium render --config ./examples/config.full.yml ...` or add a config-load test. + - Link from: `docs/config.md`. + +Do not add examples for unimplemented authentication, multi-route HTTP APIs, persistent run storage, or strict realpath artifact containment. + +## Internal Documentation Plan + +- Component: runner + - Path: `docs/internal/runner.md` + - Purpose: explain prepare/run orchestration. + - Inputs and outputs: `domain.RunRequest`, `domain.PreparedRun`, `domain.RunResult`. + - Boundaries: depends on repository, artifact reader, renderer, LLM client, validator, optional repairer. + - Config fields used: none directly; adapters provide configured dependencies. + - Adapters used: none directly. + - Failure behavior: wraps prompt/profile/artifact/render/LLM/validation errors with usecase categories. + - Tests to inspect: `internal/usecase/runner_test.go`, `internal/usecase/integration_test.go`. + - Architectural invariants: `Run` reuses `Prepare`; no durable state; repairer is optional and not wired by CLI/HTTP. +- Component: adapters + - Path: `docs/internal/adapters.md` + - Purpose: explain CLI, HTTP, and public package adapter boundaries. + - Inputs and outputs: CLI args/stdout/stderr, HTTP JSON DTOs, public package types. + - Boundaries: translate external shapes to domain requests and results. + - Config fields used: `prompt_dir`, `profile_dir`, `schema_dir`, `server.*`, `defaults.render_format`. + - Adapters used: CLI runner wiring, HTTP handler, public LLM adapter. + - Failure behavior: CLI exit codes, HTTP status/error envelope, public errors. + - Tests to inspect: `internal/adapter/cli/run_test.go`, `internal/adapter/http/handler_test.go`, `engine_test.go`. + - Architectural invariants: adapter logic stays thin; no adapter-specific business rules. +- Component: sources and repositories + - Path: `docs/internal/sources.md` + - Purpose: explain prompt/profile/schema/artifact source loading. + - Inputs and outputs: YAML files, `fs.FS` sources, artifact refs, JSON Schema documents. + - Boundaries: repositories load definitions; artifact readers load input content; validators load schemas. + - Config fields used: `prompt_dir`, `profile_dir`, `schema_dir`, `server.artifact_root`, `server.max_artifact_bytes`. + - Adapters used: CLI/HTTP/public package source wiring. + - Failure behavior: strict YAML decode errors, not-found errors, artifact not allowed/too large errors, schema load errors. + - Tests to inspect: `internal/promptdef/repository_test.go`, `internal/profile/repository_test.go`, `internal/profile/builtin/repository_test.go`, `internal/artifact/reader_test.go`, `internal/validate/standard_validator_test.go`. + - Architectural invariants: built-ins are lowest profile precedence; explicit profile sources override built-ins; public `fs.FS` roots are contained; HTTP artifact root uses lexical checks and follows symlinks. + +## Integration Documentation Plan + +- Path: `docs/integrations/openai-compatible-chat.md` + - External system or contract: OpenAI-compatible chat completions API. + - Current usage: outbound LLM generation through `internal/llm.OpenAICompatibleClient`. + - Version or compatibility notes: repository implements a subset; compatibility is field-based, not tied to one provider SDK. + - Document: endpoint construction, headers, messages, cache control, structured output, numeric parameter presence, extra params, response usage fields, timeout behavior, non-2xx error redaction. + - Do not document: unsupported OpenAI endpoints, streaming, tools, embeddings, provider-specific catalogs beyond fields actually forwarded. +- Path: `docs/integrations/subprocess.md` + - External system or contract: downstream applications invoking Scriptorium CLI as a subprocess. + - Current usage: implemented CLI `run` and `render` commands with stdout/stderr and exit codes. + - Version or compatibility notes: no formal versioning is implemented; stability comes from documented CLI behavior and tests. + - Document: supported commands, recommended invocation shapes, stdout/stderr contract, exit codes, config/environment expectations, security notes. + - Do not document: downstream product-specific behavior or private application assumptions. +- Path: `docs/api.md` + - External system or contract: inbound HTTP JSON API. + - Current usage: HTTP `POST /v1/runs` through `serve`. + - Version or compatibility notes: no URL version beyond `/v1`; route is implemented in `internal/adapter/http`. + - Document: as canonical API reference, not under `docs/integrations/`. + - Do not document: upstream model provider details or public Go package API. + +No additional integration docs are recommended for config, prompt, profile, or schema file formats because `docs/config.md` is the canonical file-format reference. + +## Recommended Implementation Sequence + +### Stage 1: Canonical Map And Link Move + +- Goal: establish the policy-compliant target structure without large content rewrites. +- Files to create/update/delete/move: create `docs/api.md` from `docs/integrations/http-api.md`; update links to point at `docs/api.md`; create `docs/integrations/subprocess.md` from generic parts of `docs/integrations/narratio.md`; mark old integration docs for deletion. +- Repository areas to inspect: HTTP handler/dto/tests, CLI run code/tests, README links. +- Acceptance criteria: `docs/api.md` exists; no non-roadmap docs link to `docs/integrations/http-api.md`; subprocess docs are generic. +- Suggested validation commands: `rg "integrations/http-api|Narratio" README.md docs`; `go test ./internal/adapter/http ./internal/adapter/cli`. +- Prompt size: small enough for one implementation prompt. + +### Stage 2: README, CLI, And Config References + +- Goal: refresh the primary user/operator references. +- Files to create/update/delete/move: `README.md`, `docs/cli.md`, `docs/config.md`, optionally `examples/config.full.yml`. +- Repository areas to inspect: CLI flags, config structs/defaults, prompt/profile/schema loaders, built-in profile assets, examples. +- Acceptance criteria: README quickstart runs; all CLI flags documented; all config defaults match code; built-in profile catalog matches asset IDs. +- Suggested validation commands: `go test ./internal/adapter/cli ./internal/config ./internal/profile/builtin`; render smoke command; `rg -e "--max-request-bytes|server.max_request_bytes|docs/api.md" README.md docs/cli.md docs/config.md`. +- Prompt size: likely one implementation prompt if kept focused; split config into a separate prompt if built-in catalog generation is done manually. + +### Stage 3: HTTP API And Operations + +- Goal: make HTTP and operations docs accurate without duplication. +- Files to create/update/delete/move: `docs/api.md`, `docs/operations.md`, `docs/troubleshooting.md`, delete or replace `docs/integrations/http-api.md`. +- Repository areas to inspect: HTTP handler/dto/error mapping, config, serve wiring, artifact reader. +- Acceptance criteria: HTTP error codes match code; operations links to API instead of duplicating it; troubleshooting entries match current errors and limits. +- Suggested validation commands: `go test ./internal/adapter/http ./internal/artifact`; `rg "request_too_large|artifact_too_large|response_too_large|artifact_root" docs/api.md docs/operations.md docs/troubleshooting.md`. +- Prompt size: one implementation prompt if `docs/api.md` already exists from Stage 1. + +### Stage 4: Consumer Documentation + +- Goal: make downstream integration docs useful and policy-compliant. +- Files to create/update/delete/move: `docs/consumers/api.md`, `docs/consumers/pkg-scriptorium.md`, `docs/integrations/subprocess.md`. +- Repository areas to inspect: public package files, public package tests, CLI subprocess behavior, examples. +- Acceptance criteria: consumer overview explains how to choose Go package vs subprocess vs HTTP; package guide documents all exported construction/source/profile methods; examples compile/run. +- Suggested validation commands: `go test .`; `go run ./examples/go-library/prepare`; `rg "credential resolver|raw API key.*profile|docs/api.md" docs/consumers docs/integrations/subprocess.md`. +- Prompt size: one implementation prompt. + +### Stage 5: Internal And Policy Docs + +- Goal: align developer docs with implemented architecture and package boundaries. +- Files to create/update/delete/move: `docs/internal/runner.md`, `docs/internal/adapters.md`, create `docs/internal/sources.md`, update `docs/policy/architecture.md`, update `docs/policy/development.md`. +- Repository areas to inspect: internal package tree, usecase, adapters, repositories, validators, artifact readers, tests. +- Acceptance criteria: internal docs include purpose, inputs/outputs, boundaries, config fields, adapters, failure behavior, tests, and invariants; policy docs remain stable and concise. +- Suggested validation commands: `go test ./internal/...`; `rg "internal/sources|docs/internal" docs/policy docs/internal`. +- Prompt size: one implementation prompt if internal docs are concise; split if `docs/internal/sources.md` becomes large. + +### Stage 6: Examples And Stale Roadmap Cleanup + +- Goal: keep examples maintained and remove stale completed plans. +- Files to create/update/delete/move: examples as needed, `docs/roadmap/cleanup.md`, this roadmap if implementation is complete. +- Repository areas to inspect: examples, tests that reference examples, roadmap directory. +- Acceptance criteria: maintained examples are linked; stale completed roadmap content is removed or reduced to active future work only; no non-roadmap doc describes unimplemented behavior. +- Suggested validation commands: `go test ./...`; `go vet ./...`; `go run ./examples/go-library/prepare`; render smoke command; `rg "future|planned|coming|not implemented|roadmap|integrations/http-api|Narratio" README.md docs --glob '!docs/roadmap/**'`. +- Prompt size: one implementation prompt. + +## Validation Plan + +Run during or after implementation: + +```bash +go test ./... +go vet ./... +go run ./examples/go-library/prepare +go run ./cmd/scriptorium render \ + --config ./examples/config.yml \ + --prompt generic.markdown_summary \ + --input transcript=./examples/fixtures/transcript.md \ + --input glossary=./examples/fixtures/glossary.yml \ + --format json +``` + +Targeted checks: + +- CLI parser and examples: `go test ./internal/adapter/cli`. +- HTTP contract: `go test ./internal/adapter/http`. +- Config defaults and examples: `go test ./internal/config`. +- Built-in profile catalog: `go test ./internal/profile/builtin`. +- Public package examples: `go test .` and `go run ./examples/go-library/prepare`. +- Internal docs source checks: `go test ./internal/...`. + +Grep/link checks: + +- `rg "integrations/http-api|Narratio" README.md docs --glob '!docs/roadmap/**'` +- `rg "future|planned|coming|not implemented|roadmap" README.md docs --glob '!docs/roadmap/**'` +- `rg "api_key|API key|secret" README.md docs examples` +- `rg "max_request_bytes|max_artifact_bytes|max_response_bytes|request_too_large|artifact_too_large|response_too_large" docs` +- `rg "\]\(([^)#]+)(#[^)]+)?\)" README.md docs` + +No dedicated markdown linter or link checker is currently configured in the repository. If one is added later, document it in `docs/policy/development.md` and include it in this validation plan. + +Manual review items: + +- Confirm every target doc has one canonical scope and links elsewhere for details. +- Confirm examples are secret-free. +- Confirm `docs/api.md` contains the only full HTTP endpoint reference. +- Confirm `docs/config.md` contains the only full config/prompt/profile/schema file-format reference. +- Confirm consumer docs do not duplicate HTTP endpoint tables. +- Confirm non-roadmap docs document implemented behavior only. + +## Open Questions + +No open questions block this roadmap. The recommended path is to create `docs/api.md`, generalize the subprocess integration doc, keep current-behavior docs concise and canonical, and remove completed roadmap material once the documentation migration is finished.