Add policy documentation and prepare a roadmap to update the remaining documentation accordingly

docs/policy/architecture.md

@@ -0,0 +1,97 @@
+# Architecture
+
+This document defines the development principles for this Go project. It is inward-facing: developers and LLM coding agents should use it to preserve the project’s shape, boundaries, and invariants as the code evolves.
+
+## Project Shape
+
+Default to a small, explicit, dependency-light Go application. Keep the design modular enough to test and change safely, but do not add abstraction unless it protects a real boundary or enables a real extension point.
+
+Business/domain logic should live outside CLI, transport, and external-adapter packages.
+
+## Dependency Policy
+
+Prefer the Go standard library where practical.
+
+Use external dependencies only when justified by correctness, security, interoperability, or substantial complexity reduction. Good reasons include complex security-sensitive behavior, such as HTML sanitization, or widely used de facto standards, such as YAML parsing.
+
+Avoid dependencies for small conveniences. Do not let external dependency types leak across internal package boundaries unless the dependency is itself the explicit public contract of that package.
+
+## Package Layout
+
+Use this layout unless the project has a documented reason to differ:
+
+- `internal/app`: application orchestration and top-level use cases.
+- `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring.
+- `internal/config`: configuration structs, defaults, loading, precedence, and validation.
+- `internal/adapters/<name>`: adapters for external CLIs, APIs, databases, object stores, or libraries.
+- `internal/api`: HTTP API handlers and request/response types, when the application exposes an HTTP API.
+- `internal/transport/http`: HTTP client code, when the application calls HTTP services.
+
+Package-private implementation constants may live near the package that owns them, preferably in `constants.go` when useful.
+
+## Configuration
+
+Centralize configuration loading, processing, precedence, defaults, and validation in `internal/config`.
+
+The goal is to make configuration discoverable and avoid implicit or hidden operational values. User-visible defaults and cross-package operational defaults should be defined in `internal/config/defaults.go`.
+
+Unless documented otherwise, precedence is:
+
+1. CLI flags
+2. environment variables
+3. configuration file
+4. built-in defaults
+
+Prefer YAML configuration unless the project has a strong reason to use another format. Config files should be discovered at `/usr/local/etc/<app_name>/config.yml`, with a CLI override via `--config`.
+
+Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables or secret files for secrets.
+
+## Adapters and External Integrations
+
+Use a hexagonal architecture style for external integrations.
+
+External adapters belong under `internal/adapters/<name>`. If an adapter uses an external dependency, that dependency’s interface must not leak outside the adapter package. Other packages should interact only with the adapter’s API, so the dependency can be swapped, upgraded, or removed without touching unrelated code.
+
+Adapters should be thin. Domain decisions belong in application/domain packages, not inside adapter glue.
+
+## Modules, Stages, and Registries
+
+When the application has stages or modules, each major stage/module should live in its own package and have an explicit input/output contract.
+
+The orchestrator should be able to compose, skip, resume, or run individual stages/modules when their prerequisites are satisfied. Ordering should be explicit: use a default sequence, dependency graph, or documented orchestration rule.
+
+If users can select modules, stages, validators, renderers, or adapters, selection should go through a registry or equivalent mechanism rather than scattered conditionals.
+
+## Embedded Assets
+
+Store embedded JSON schemas, Markdown prompts, templates, and similar assets as separate files, not inline string literals, unless there is a strong reason otherwise.
+
+## Errors and Logging
+
+Errors should be actionable and preserve context. Wrap errors with operation and path/resource context. CLI code should convert internal errors into concise user-facing messages.
+
+Errors and logs must not expose secrets.
+
+Use structured logging where practical. Logs should describe operations, paths, external calls, retries, and failure causes, but should not include large user data by default.
+
+## Context, Timeouts, and Cancellation
+
+Long-running operations should accept `context.Context`. External calls, subprocesses, HTTP requests, storage operations, and multi-stage workflows should respect cancellation and timeouts.
+
+## State, Files, and Safety
+
+If the application writes durable state, writes should be atomic where practical. Multi-step workflows should preserve enough state to support inspection, retry, or resume after failure.
+
+Code that deletes, moves, or overwrites files must use narrow, explicit paths. Avoid broad parent-directory operations. Cleanup that can cause data loss must be opt-in.
+
+## Testing
+
+Core logic should be testable without real external services. Use fakes, fixtures, or local test doubles for adapters where practical.
+
+Config examples should be load-tested. Important CLI workflows should have parser or command tests. Stage/module contracts should have focused tests that do not require running the full application unless end-to-end coverage is intentional.
+
+## Documentation
+
+Documentation should follow the project documentation policy. Keep user docs focused on implemented behavior. Put future, planned, or aspirational work only under `docs/roadmap/`.
+
+When changing architecture, config, CLI behavior, adapters, or stage/module contracts, update the relevant docs and examples in the same change.

docs/policy/documentation.md

@@ -0,0 +1,356 @@
+# Go Project Documentation Policy
+
+## Purpose
+
+Project documentation must help four audiences:
+
+1. users who need to run the application;
+2. administrators/operators who need to configure and operate it;
+3. developers who need to understand and change it safely;
+4. LLM coding agents that need clear scope, boundaries, and invariants.
+
+Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
+
+## Core Rules
+
+### 1. Keep docs concise
+
+Each document should cover a defined scope and only the essentials for that scope.
+
+Avoid:
+- long background explanations;
+- repeated reference material;
+- implementation detail in user-facing docs;
+- aspirational language outside roadmap docs;
+- verbose examples where one minimal example is clearer.
+
+### 2. Document only implemented behavior outside roadmap files
+
+Unimplemented, planned, aspirational, experimental, or future work may be described only under:
+
+- `docs/roadmap/`
+
+No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
+
+If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
+
+### 3. Use canonical homes
+
+Each type of information should have one canonical location.
+
+Canonical homes:
+
+- project purpose and quickstart: `README.md`
+- development principles: `docs/policy/architecture.md`
+- configuration reference: `docs/config.md`
+- CLI reference: `docs/cli.md`
+- operations and recovery: `docs/operations.md`
+- troubleshooting: `docs/troubleshooting.md`
+- implemented internals: `docs/internal/`
+- future work: `docs/roadmap/`
+- contributor workflow: `docs/policy/development.md`
+- copyable examples: `examples/`
+
+Other files should summarize briefly and link to the canonical source.
+
+### 4. Keep examples real
+
+Examples should be valid, maintained, and free of secrets.
+
+Where practical:
+- example configs should load successfully;
+- example commands should match real CLI syntax;
+- important examples should be covered by tests.
+
+## Documentation Profiles
+
+All projects require:
+
+- `README.md`
+- `docs/policy/architecture.md`
+
+Additional docs depend on the project.
+
+### Small library
+
+Recommended:
+- `docs/policy/development.md`, if contributor conventions are non-obvious
+
+### Simple CLI
+
+Required:
+- `docs/cli.md`
+
+Recommended:
+- `docs/policy/development.md`
+
+### Config-driven CLI
+
+Required:
+- `docs/cli.md`
+- `docs/config.md`
+
+Recommended:
+- `examples/`
+- `docs/policy/development.md`
+
+### Stateful or operator-facing application
+
+Required:
+- `docs/cli.md`, if CLI-based
+- `docs/config.md`, if config-driven
+- `docs/operations.md`
+
+Recommended:
+- `docs/troubleshooting.md`
+- `examples/`
+- `docs/policy/development.md`
+
+### Modular, staged, service-oriented, or orchestration application
+
+Required:
+- `docs/cli.md`, if CLI-based
+- `docs/config.md`, if config-driven
+- `docs/operations.md`
+- `docs/internal/`
+- `docs/policy/development.md`
+
+Recommended:
+- `docs/troubleshooting.md`
+- validated examples under `examples/`
+
+## Required Documents
+
+### README.md
+
+**Audience:** users, administrators, operators
+
+The README is the outward-facing project orientation page.
+
+It should include, in order:
+
+1. concise description;
+2. elevator pitch;
+3. shortest useful command or usage example;
+4. links to targeted docs.
+
+The README should be short. It is not a manual.
+
+The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
+
+### docs/policy/architecture.md
+
+**Audience:** developers, LLM coding agents
+
+`docs/policy/architecture.md` is required for every project.
+
+It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
+
+It should include:
+
+- project shape;
+- core design principles;
+- package and boundary philosophy;
+- state/persistence philosophy, if applicable;
+- external integration philosophy, if applicable;
+- error-handling and logging principles;
+- testing expectations;
+- documentation expectations;
+- architectural invariants;
+- explicit non-goals, if useful.
+
+For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
+
+### docs/policy/development.md
+
+**Audience:** developers, LLM coding agents
+
+Required for projects maintained by humans and LLM coding agents.
+
+It should include:
+
+- repository layout;
+- build/test commands;
+- coding conventions;
+- dependency policy;
+- how to add config fields;
+- how to add CLI flags;
+- how to add stages/modules/adapters, if applicable;
+- how to update examples;
+- documentation update expectations.
+
+### docs/config.md
+
+**Audience:** administrators, operators, advanced users
+
+Required for applications with configuration files.
+
+It should include, in order:
+
+1. config file locations and discovery precedence;
+2. minimal working config;
+3. production-oriented config;
+4. full configuration reference;
+5. secrets handling, if applicable;
+6. links to maintained examples.
+
+The full configuration reference should be canonical.
+
+### docs/cli.md
+
+**Audience:** users, administrators, operators
+
+Required for CLI applications.
+
+It should include, in order:
+
+1. shortest useful command;
+2. command overview;
+3. complete flag reference;
+4. common workflows;
+5. diagnostic or recovery commands, if applicable.
+
+Explain when commands are useful, not just their syntax.
+
+### docs/operations.md
+
+**Audience:** administrators, operators
+
+Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
+
+It should cover:
+
+- normal workflow;
+- filesystem layout;
+- remote storage layout, if applicable;
+- logs and manifests;
+- resume/retry behavior;
+- cleanup behavior;
+- archive/backup behavior;
+- safe recovery procedures;
+- operational caveats.
+
+### docs/troubleshooting.md
+
+**Audience:** administrators, operators
+
+Recommended once recurring failure modes exist.
+
+Each entry should include:
+
+- symptom;
+- likely cause;
+- diagnostic command or inspection step;
+- safe fix;
+- relevant links.
+
+### docs/internal/
+
+**Audience:** developers, LLM coding agents
+
+Required for modular, staged, service-oriented, or orchestration projects.
+
+This directory describes implemented internal components. It is not the roadmap.
+
+Use one file per major component where useful.
+
+Each component doc should include:
+
+1. purpose;
+2. inputs and outputs;
+3. boundaries;
+4. config fields used;
+5. external adapters used;
+6. state or manifest behavior, if applicable;
+7. skip/resume behavior, if applicable;
+8. failure behavior;
+9. tests to inspect before changing;
+10. architectural invariants.
+
+### docs/roadmap/
+
+**Audience:** maintainers, developers, LLM coding agents
+
+This is the only place for planned, future, aspirational, experimental, or unimplemented work.
+
+Roadmap docs should clearly distinguish:
+
+- proposed work;
+- accepted plans;
+- deferred ideas;
+- rejected ideas;
+- implementation prompts or task breakdowns, if useful.
+
+Roadmap docs should not be confused with current behavior.
+
+### docs/integrations/
+
+**Audience:** developers, LLM coding agents
+
+Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
+
+This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
+
+Use one file per integration where useful.
+
+## Examples Directory
+
+Projects with non-trivial configuration or workflows should include `examples/`.
+
+Useful examples include:
+
+- minimal working config;
+- production-oriented config;
+- full annotated config;
+- local development config;
+- remote/object-storage config;
+- minimal session/input file.
+
+Examples should be valid, maintained, tested when practical, and linked from relevant docs.
+
+## Security and Privacy
+
+Docs and examples must not include:
+
+- real API keys;
+- tokens;
+- passwords;
+- private keys;
+- private environment dumps;
+- sensitive user data;
+- raw private transcripts;
+- private infrastructure details unless intentionally public.
+
+Document secret-handling mechanisms, not actual secret values.
+
+## Maintenance Rules
+
+When docs change, verify the affected behavior.
+
+Where practical:
+
+- load example config files in tests;
+- test CLI examples or command parser behavior;
+- validate documented flags against real flags;
+- remove stale references;
+- update links after renames;
+- keep roadmap content out of non-roadmap docs.
+
+If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
+
+Documentation is complete only when it matches the current code.
+
+## Documentation Change Checklist
+
+Before merging documentation changes, verify:
+
+- README is concise and orientation-focused.
+- `docs/policy/architecture.md` describes development principles.
+- Future work appears only under `docs/roadmap/`.
+- User-facing docs avoid unnecessary internals.
+- Developer-facing docs preserve boundaries and invariants.
+- Config examples match the schema.
+- CLI examples match real commands and flags.
+- Defaults appear in the canonical config reference.
+- No secrets or private data are included.
+- Links are accurate.

docs/roadmap/documentation.md

@@ -0,0 +1,355 @@
+# Documentation Roadmap
+
+## Purpose
+
+This roadmap defines the work required to bring scriptorium's documentation into compliance with `docs/policy/documentation.md` and the current implementation. It is a planning document only; future agents should use it to update the canonical documentation without describing unimplemented behavior outside `docs/roadmap/`.
+
+## Repository Documentation Inventory
+
+- `README.md` - keep and rewrite. It is currently a full manual covering config, CLI, HTTP, prompt/profile authoring, examples, and build commands; policy says README should be a short orientation page with a quickstart and links.
+- `architecture.md` - move/merge/delete after rewrite. It overlaps with `docs/policy/architecture.md`, contains "should" guidance and future extension notes outside `docs/roadmap/`, and references architecture that is partly stale or aspirational.
+- `docs/policy/documentation.md` - keep and lightly update only if policy itself changes. It is the controlling documentation policy.
+- `docs/policy/architecture.md` - keep and lightly update. It is the canonical development architecture policy, but some package-layout defaults do not exactly match this repository (`internal/adapter/...` versus policy examples such as `internal/adapters/...`).
+- `docs/policy/development.md` - create new. Required by policy for projects maintained by humans and LLM coding agents.
+- `docs/config/config-yml.md` - merge into `docs/config.md`. The content mostly matches code but lives in the wrong canonical home.
+- `docs/config/prompt-definitions.md` - merge into `docs/config.md`. The field reference is mostly accurate, but it needs caveats about `repair_attempts`, repeated message roles, schema path resolution, and examples.
+- `docs/config/profile-definitions.md` - merge into `docs/config.md`. It must stop implying that every profile field is sent to the provider; the current OpenAI-compatible client does not send `reasoning_effort` or `extra_params`.
+- `docs/config/schema-definitions.md` - merge into `docs/config.md` or link from it. Schema behavior is implemented, but the canonical config reference should own this material.
+- `docs/cli.md` - create new. Required by policy for the implemented CLI.
+- `docs/operations.md` - create new. Required by policy for this CLI/service application; scope should cover normal operation, config paths, secret handling, stdout/stderr, exit codes, HTTP serving, and the fact that there is no durable run state/resume behavior.
+- `docs/troubleshooting.md` - create new. Recommended by policy and justified by implemented failure modes in parser, config, prompt/profile loading, artifact reading, validation, LLM calls, and HTTP error mapping.
+- `docs/internal/` - create new. Required by policy for this modular application.
+- `docs/integrations/narratio.md` - keep and rewrite. It documents an actual CLI integration contract, but it includes future extension notes outside roadmap and illustrative prompt IDs that are not all present in the repository.
+- `docs/integrations/http-api.md` - create new. The implemented `POST /v1/runs` API is an external integration contract and should not live in README.
+- `docs/integrations/openai-compatible-chat.md` - create new. The outbound LLM contract is important and implemented in `internal/llm/openai_compatible_client.go`.
+- `examples/config.yml` - keep and lightly update if paths move. It is a valid app config example for the current root `prompts/`, `profiles/`, and `schemas/` directories.
+- `examples/fixtures/transcript.md` - keep. It is used by integration tests.
+- `examples/fixtures/glossary.yml` - keep. It is used by integration tests.
+- `prompts/` - keep as maintained sample prompt library for now; recommended to move or mirror under `examples/` only if tests and docs are updated together.
+- `profiles/` - keep as maintained sample profile library for now; recommended to move or mirror under `examples/` only if tests and docs are updated together.
+- `schemas/` - keep as maintained sample schema library for now; recommended to move or mirror under `examples/` only if tests and docs are updated together.
+- `local-test/` - delete, move out of the repository, or explicitly exclude from maintained docs. It contains ad hoc local artifacts and provider profiles; it should not be linked from canonical docs unless promoted to maintained examples with tests and secret review.
+
+## Policy Compliance Assessment
+
+Required documents that are missing:
+
+- `docs/cli.md`
+- `docs/config.md`
+- `docs/operations.md`
+- `docs/internal/`
+- `docs/policy/development.md`
+
+Recommended documents that should be added:
+
+- `docs/troubleshooting.md`
+- Validated examples under `examples/` beyond the current config and fixtures, especially command examples that can be checked with `render`.
+- Integration docs for the implemented HTTP API and outbound OpenAI-compatible chat API.
+
+Documents that exist but are stale or in the wrong canonical home:
+
+- `README.md` duplicates material that belongs in `docs/cli.md`, `docs/config.md`, `docs/integrations/`, and `docs/internal/`.
+- `docs/config/*.md` should be merged into `docs/config.md`.
+- `architecture.md` should be merged into `docs/policy/architecture.md`, `docs/internal/`, or `docs/roadmap/`, then removed.
+- `docs/integrations/narratio.md` should remain under integrations but must be narrowed to implemented CLI behavior and actual integration guidance.
+
+Content that appears to describe deprecated, historical, planned, or unimplemented behavior outside `docs/roadmap/`:
+
+- `architecture.md` has future extension notes for S3 artifact references, additional LLM providers, streaming, batch execution, database-backed repositories, profile versioning, and HTTP render endpoints.
+- `architecture.md` and `README.md` describe bounded repair as if it is generally active. The code has an injected repairer hook, but the CLI and HTTP server construct `Runner` without a repairer, so production commands do not currently perform repair attempts.
+- `README.md` says "additional output formats can be added later"; this belongs in roadmap only.
+- `README.md` references `go build -o scriptorium ./cmd/scriptorium`, which is valid, but the README should not be the build/test manual after `docs/policy/development.md` exists.
+- `docs/integrations/narratio.md` has "Future Extension Notes" and examples for prompt IDs not present in the repository, such as `dnd.structured_events`, `dnd.glossary_suggestions`, and `dnd.player_summary`.
+- Any docs implying S3 artifact support should be removed from current-behavior docs. `domain.ArtifactRefS3` exists, but `artifact.CompositeReader` supports only `inline` and `file`.
+
+Examples that are missing, stale, invalid, or untested:
+
+- `examples/config.yml` points at root `prompts/`, `profiles/`, and `schemas/`; it is valid for repository-root execution but should be tested or explicitly checked.
+- There are no copyable CLI example scripts or expected-output files under `examples/`.
+- The maintained prompt/profile/schema examples live outside `examples/`; this is usable, but policy prefers copyable examples under `examples/`.
+- `local-test/` appears unmaintained and should not be treated as documentation.
+
+Links that are likely stale or need verification:
+
+- Existing links from README to docs should be rewritten after canonical files are created.
+- Any references to `docs/config/config-yml.md`, `docs/config/prompt-definitions.md`, `docs/config/profile-definitions.md`, or `docs/config/schema-definitions.md` should be updated after those files are merged.
+- References to default config paths must use the implemented search order: `/usr/local/etc/scriptorium/config.yml`, then `/etc/scriptorium/config.yml`.
+
+## Target Documentation Set
+
+### `README.md`
+
+- Audience: users, administrators, operators.
+- Purpose: concise orientation and shortest useful command.
+- Canonical scope: project purpose, elevator pitch, quickstart, and links.
+- Recommended outline: description; why scriptorium exists; shortest useful `scriptorium render` or `scriptorium run` example; documentation links.
+- Source of truth: `cmd/scriptorium/main.go`, `internal/adapter/cli/run.go`, `examples/config.yml`, `prompts/generic.markdown_summary.yaml`.
+- Acceptance criteria: no full flag reference; no HTTP schema; no prompt/profile field tables; no future work; all links point to existing target docs.
+
+### `docs/cli.md`
+
+- Audience: users, administrators, operators.
+- Purpose: complete CLI reference and common workflows.
+- Canonical scope: commands, flags, outputs, exit codes, and command examples.
+- Recommended outline: shortest useful command; command overview; `run`; `render`; `serve`; flag reference; input and variable mapping syntax; output behavior; exit codes; common workflows.
+- Source of truth: `internal/adapter/cli/run.go`, `internal/adapter/cli/run_test.go`, `internal/format/prepared_run.go`, `cmd/scriptorium/main.go`.
+- Acceptance criteria: documents real flags only; notes deprecated aliases `--prompt-id` and `--profile-id`; documents that `render` does not currently accept `--schema-dir`; documents stdout/stderr split and exit code `2` for validation failure.
+
+### `docs/config.md`
+
+- Audience: administrators, operators, advanced users.
+- Purpose: canonical reference for app config, prompt definitions, profiles, and schemas.
+- Canonical scope: all implemented YAML/JSON file formats and precedence rules.
+- Recommended outline: config file discovery and precedence; minimal app config; production-oriented app config; app config reference; prompt definition reference; profile definition reference; schema behavior; secrets handling; maintained examples.
+- Source of truth: `internal/config/config.go`, `internal/defaults/defaults.go`, `internal/promptdef/filesystem_repository.go`, `internal/profile/filesystem_repository.go`, `internal/validate/standard_validator.go`, repository `prompts/`, `profiles/`, `schemas/`, and config/profile/prompt tests.
+- Acceptance criteria: replaces split `docs/config/*.md`; documents strict YAML decoding; documents raw API key rejection; documents `schema_dir` default `.`; documents prompt `content_file` relative to prompt YAML; states `content_type` is metadata only; does not claim operational repair unless a repairer is configured.
+
+### `docs/operations.md`
+
+- Audience: administrators, operators.
+- Purpose: operational use of CLI and HTTP service.
+- Canonical scope: normal workflow, filesystem expectations, config deployment, secrets, logs/output, validation behavior, and recovery from failed runs.
+- Recommended outline: normal run/render workflow; config and library directories; environment variables for API keys; serving HTTP; output and stderr summaries; validation failure handling; no durable state/resume/archive behavior; safe recovery steps.
+- Source of truth: `internal/adapter/cli/run.go`, `internal/adapter/http/handler.go`, `internal/config/config.go`, `internal/llm/openai_compatible_client.go`, `internal/usecase/runner.go`.
+- Acceptance criteria: makes clear scriptorium does not persist run state; does not invent cleanup/archive/resume; documents that HTTP has no built-in authentication and should be deployed behind trusted controls.
+
+### `docs/troubleshooting.md`
+
+- Audience: administrators, operators.
+- Purpose: safe diagnosis and fixes for recurring implemented failure modes.
+- Canonical scope: symptoms, likely causes, diagnostics, safe fixes, and links.
+- Recommended outline: missing config; missing prompt/profile dirs; unknown flags; prompt/profile load failures; missing input files; template render failures; missing API-key environment values; LLM non-2xx/malformed responses; JSON/schema validation failures; HTTP error codes.
+- Source of truth: `internal/adapter/cli/run_test.go`, `internal/adapter/http/handler_test.go`, `internal/config/config_test.go`, `internal/promptdef/repository_test.go`, `internal/profile/repository_test.go`, `internal/validate/standard_validator_test.go`, `internal/llm/openai_compatible_client_test.go`.
+- Acceptance criteria: every entry includes symptom, likely cause, diagnostic step, safe fix, and links to canonical CLI/config/operations docs.
+
+### `docs/policy/architecture.md`
+
+- Audience: developers, LLM coding agents.
+- Purpose: controlling development architecture and invariants.
+- Canonical scope: development principles, boundaries, invariants, non-goals.
+- Recommended outline: keep current policy shape; add scriptorium-specific package map or link to `docs/internal/`; clarify no orchestration creep; clarify current adapters.
+- Source of truth: existing policy, `internal/` package layout, `architecture.md`.
+- Acceptance criteria: remains policy-oriented; does not become user docs; future work stays in roadmap; no stale package names.
+
+### `docs/policy/development.md`
+
+- Audience: developers, LLM coding agents.
+- Purpose: contributor workflow and change checklist.
+- Canonical scope: repository layout, build/test commands, coding conventions, dependency policy, adding config/CLI/adapters, updating examples/docs.
+- Recommended outline: repository layout; common commands; coding conventions; dependency policy; how to add config fields; how to add CLI flags; how to add adapters; how to update examples; documentation expectations.
+- Source of truth: `go.mod`, `cmd/scriptorium/main.go`, `internal/adapter/cli/run.go`, `internal/config/config.go`, `docs/policy/architecture.md`, existing tests.
+- Acceptance criteria: includes `go test ./...`; references `go build ./cmd/scriptorium`; tells contributors to update docs and tests with behavior changes.
+
+### `docs/internal/runner.md`
+
+- Audience: developers, LLM coding agents.
+- Purpose: implemented core prepare/run behavior.
+- Canonical scope: `Runner.Prepare`, `Runner.Run`, profile selection, runtime merge, artifact loading, rendering, structured output setup, validation, repair hook boundary.
+- Recommended outline: purpose; inputs/outputs; prepare flow; run flow; boundary contracts; failure behavior; tests; invariants.
+- Source of truth: `internal/usecase/runner.go`, `internal/usecase/repairer.go`, `internal/usecase/runner_test.go`, `internal/usecase/integration_test.go`.
+- Acceptance criteria: states CLI/HTTP currently construct `Runner` without a repairer; documents validation content failures versus runtime validation errors; no provider-specific details except through ports.
+
+### `docs/internal/adapters.md`
+
+- Audience: developers, LLM coding agents.
+- Purpose: implemented adapter boundaries.
+- Canonical scope: CLI adapter, HTTP adapter, filesystem repositories, artifact reader, prompt renderer, OpenAI-compatible LLM client, validator, prepared-run formatter.
+- Recommended outline: adapter map; inputs/outputs; config fields used; external dependencies; failure behavior; tests to inspect.
+- Source of truth: `internal/adapter/cli`, `internal/adapter/http`, `internal/promptdef`, `internal/profile`, `internal/artifact`, `internal/prompt`, `internal/llm`, `internal/validate`, `internal/format`.
+- Acceptance criteria: documents only implemented adapters; states `inline` and `file` artifact refs are supported and S3 is not; states OpenAI request fields actually sent.
+
+### `docs/integrations/http-api.md`
+
+- Audience: developers, LLM coding agents, API clients.
+- Purpose: implemented inbound HTTP contract.
+- Canonical scope: `POST /v1/runs`, request/response shape, raw output opt-in, error mapping, validation-failed status behavior.
+- Recommended outline: scope; endpoint; request fields; response fields; error responses; validation behavior; security/auth note.
+- Source of truth: `internal/adapter/http/dto.go`, `internal/adapter/http/handler.go`, `internal/adapter/http/handler_test.go`.
+- Acceptance criteria: no unimplemented render endpoint; no built-in auth claim; unknown JSON fields rejected; raw API key fields rejected by strict JSON.
+
+### `docs/integrations/openai-compatible-chat.md`
+
+- Audience: developers, LLM operators, LLM adapter maintainers.
+- Purpose: implemented outbound LLM API contract.
+- Canonical scope: OpenAI-compatible chat completions request/response subset and provider-level structured output behavior.
+- Recommended outline: endpoint construction; request fields sent; auth header from `api_key_env`; timeout behavior; response expectations; error handling; unsupported profile fields.
+- Source of truth: `internal/llm/openai_compatible_client.go`, `internal/llm/openai_compatible_client_test.go`, `internal/usecase/runner.go`.
+- Acceptance criteria: says endpoint appends `/chat/completions`; says empty first choice content is malformed; says `reasoning_effort` and `extra_params` are not currently serialized into the outbound request.
+
+### `docs/integrations/narratio.md`
+
+- Audience: developers, LLM coding agents maintaining Narratio integration.
+- Purpose: CLI subprocess contract for Narratio.
+- Canonical scope: how Narratio should call implemented `scriptorium run` and `scriptorium render`.
+- Recommended outline: purpose; assumptions; command shapes; inputs/vars; profile selection; runtime overrides; config behavior; environment handling; output handling; exit statuses; security notes; non-goals.
+- Source of truth: `internal/adapter/cli/run.go`, `internal/adapter/cli/run_test.go`, `docs/cli.md`, `docs/config.md`.
+- Acceptance criteria: removes future extensions; labels any Narratio-specific prompt IDs as external examples only or removes them; links to canonical CLI/config docs.
+
+## File-by-File Rewrite Guidance
+
+- `README.md`: cover what scriptorium does and show one minimal command. Avoid field tables, complete flag lists, HTTP schema, internal package details, future extensions, and long examples. Link to `docs/cli.md`, `docs/config.md`, `docs/operations.md`, `docs/troubleshooting.md`, and `docs/integrations/`.
+- `docs/cli.md`: cover `run`, `render`, `serve`, flags, output behavior, and exit codes. Avoid duplicating prompt/profile YAML field references; link to `docs/config.md`. Inspect CLI parser tests before writing examples. Do not carry forward README's claim that render supports `--schema-dir`.
+- `docs/config.md`: cover app config, prompt YAML, profile YAML, and schema behavior. Avoid provider API details except where needed for profile fields; link to OpenAI integration doc. Do not carry forward claims that `repair_attempts` enables repair for normal CLI/HTTP runs unless code later wires a repairer.
+- `docs/operations.md`: cover deployed operation and recovery boundaries. Avoid inventing durable state, resume behavior, cleanup, backups, or archives. State that rerunning a command is the recovery model.
+- `docs/troubleshooting.md`: use tested errors and behavior. Avoid exposing internal wrapped error details that HTTP intentionally suppresses. Link to CLI/config/operations instead of repeating full references.
+- `docs/policy/architecture.md`: preserve policy authority and update only scriptorium-specific facts. Avoid copying the long historical `architecture.md` wholesale. Move future extension ideas to roadmap docs only.
+- `docs/policy/development.md`: cover contributor mechanics and how to update behavior safely. Avoid user-facing manuals. Include tests and docs update expectations.
+- `docs/internal/runner.md`: explain implemented core flow and invariants. Avoid CLI flag tables and HTTP DTO detail; link to adapter docs.
+- `docs/internal/adapters.md`: explain implemented adapter boundaries and tests. Avoid proposing new adapters. Do not imply `ArtifactRefS3` works.
+- `docs/integrations/http-api.md`: document only `POST /v1/runs`. Avoid documenting a render/prepare HTTP endpoint.
+- `docs/integrations/openai-compatible-chat.md`: document the outbound request subset. Avoid documenting unsupported OpenAI fields or provider-specific options unless code sends them.
+- `docs/integrations/narratio.md`: keep it as a subprocess contract. Avoid future work, S3, HTTP-as-primary-path, and undeployed prompt IDs as current examples.
+- `architecture.md`: after target docs exist, delete it or replace it with a short pointer to `docs/policy/architecture.md` and `docs/internal/`. Do not leave future notes in this root file.
+- `docs/config/*.md`: after `docs/config.md` exists and links are updated, delete these split files or replace them with pointers only if backwards-compatible links are necessary.
+
+## Examples Plan
+
+Existing maintained examples:
+
+- `examples/config.yml`: minimal app config pointing at root prompt/profile/schema libraries. Validity check: `go test ./internal/config ./internal/adapter/cli` and `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --format json`. Link from README, `docs/config.md`, and `docs/cli.md`.
+- `examples/fixtures/transcript.md`: sample transcript input. Validity check: used by `internal/usecase/integration_test.go` and render smoke command. Link from README and `docs/cli.md`.
+- `examples/fixtures/glossary.yml`: sample optional glossary input. Validity check: used by `internal/usecase/integration_test.go`. Link from `docs/config.md` and examples section in `docs/cli.md`.
+- `prompts/generic.markdown_summary.yaml`: sample markdown prompt. Validity check: render smoke command. Link from config docs until or unless it is moved under `examples/`.
+- `prompts/generic.structured_events.yaml` plus `schemas/structured_events.schema.json`: sample JSON-schema prompt. Validity check: `go test ./internal/usecase`. Link from `docs/config.md`.
+- `profiles/local-fast.yaml` and `profiles/local-quality.yaml`: sample profiles. Validity check: profile repository tests plus integration test. Link from `docs/config.md`, with a note that `local-quality` requires `SCRIPTORIUM_API_KEY` because it sets `api_key_env`.
+
+Recommended example additions, all based on implemented behavior:
+
+- `examples/render-markdown-summary.sh`: copyable render smoke command using `generic.markdown_summary`. Expected check: run script or equivalent `go run` command exits `0`. Link from README and `docs/cli.md`.
+- `examples/http-run.json`: copyable `POST /v1/runs` request body using `inline` or `file` artifact refs. Expected check: parse as JSON and keep aligned with `internal/adapter/http/dto.go`. Link from `docs/integrations/http-api.md`.
+- `examples/prompts/`, `examples/profiles/`, `examples/schemas/`: optional future move or mirror of maintained sample libraries. Expected check: update integration tests and `examples/config.yml` together. This is recommended for policy alignment but should be done as its own implementation stage to avoid breaking tests.
+
+Do not document `local-test/` as maintained examples.
+
+## Internal Documentation Plan
+
+### Core runner
+
+- Path: `docs/internal/runner.md`
+- Purpose: explain implemented prepare/run lifecycle.
+- Inputs and outputs: `domain.RunRequest`, `domain.PreparedRun`, `domain.RunResult`, `domain.GenerateRequest`.
+- Boundaries: usecase owns profile selection, runtime merge, artifact resolution orchestration, rendering orchestration, structured output setup, validation, and run metadata; adapters own transport/config parsing.
+- Config fields used: none directly; adapters pass resolved repositories, validators, and request values.
+- Adapters used: promptdef repository, profile repository, artifact reader, prompt renderer, LLM client, validator, optional injected repairer.
+- Failure behavior: invalid request, prompt/profile load, artifact load, render failure, LLM failure, validation runtime failure; validation content failures return a result.
+- Tests to inspect before changing: `internal/usecase/runner_test.go`, `internal/usecase/integration_test.go`.
+- Architectural invariants: `Run` reuses `Prepare`; no resolved API key values in prepared data; repair attempts bounded and only possible when a repairer is injected; no orchestration creep.
+
+### Adapters and repositories
+
+- Path: `docs/internal/adapters.md`
+- Purpose: explain implemented external boundaries.
+- Inputs and outputs: CLI args/stdout/stderr/exit codes; HTTP JSON DTOs; YAML prompt/profile/config files; file/inline artifacts; OpenAI-compatible HTTP requests; prepared-run text/JSON output.
+- Boundaries: adapters translate external forms into domain requests/results and must not own domain decisions.
+- Config fields used: `prompt_dir`, `profile_dir`, `schema_dir`, `server.addr`, `defaults.render_format`; profile `endpoint`, `model`, generation fields, timeout, `api_key_env`.
+- Adapters used: CLI, HTTP, filesystem repositories, artifact reader, Go template renderer, OpenAI-compatible client, standard validator, prepared-run formatter.
+- Failure behavior: strict YAML/JSON decoding, unknown fields rejected, unsupported artifact refs rejected, LLM non-2xx/malformed responses become errors.
+- Tests to inspect before changing: adapter, repository, artifact, renderer, LLM, validator, and formatter tests under `internal/**`.
+- Architectural invariants: no raw API keys; no S3 docs until reader exists; OpenAI client sends only implemented request fields.
+
+### Validation and structured output
+
+- Path: include in `docs/internal/runner.md` or create `docs/internal/validation.md` if the section grows.
+- Purpose: explain `none`, `basic`, `json`, `json_schema`, schema loading, and provider-level JSON schema request setup.
+- Inputs and outputs: `domain.Artifact`, `domain.OutputContract`, `domain.ValidationResult`, `domain.StructuredOutputSpec`.
+- Boundaries: validator checks output; runner creates provider-level structured output spec for `json_schema`; OpenAI adapter serializes `response_format`.
+- Config fields used: `schema_dir`; prompt `output.schema_path`, `output.validation_mode`, `output.format`.
+- Adapters used: standard validator and OpenAI-compatible client.
+- Failure behavior: invalid generated JSON is validation failure; missing/invalid schema file is runtime validation error before or during run preparation.
+- Tests to inspect before changing: `internal/validate/standard_validator_test.go`, `internal/usecase/runner_test.go`, `internal/llm/openai_compatible_client_test.go`.
+- Architectural invariants: schema docs must load before `json_schema` LLM request; schema paths resolve relative to `schema_dir`.
+
+## Integration Documentation Plan
+
+### HTTP API
+
+- Path: `docs/integrations/http-api.md`
+- External system or contract: inbound HTTP clients of scriptorium.
+- Current usage in scriptorium: `scriptorium serve` exposes `POST /v1/runs`.
+- Version or compatibility notes: route is `/v1/runs`; request decoding rejects unknown JSON fields.
+- What should be documented: request fields, `file` and `inline` artifact refs, model overrides, raw output opt-in, response shape, error codes, validation-failed `200 OK`, no built-in auth.
+- What should not be documented: unimplemented render endpoint, streaming, batch, authentication middleware, remote artifact storage.
+
+### OpenAI-compatible chat completions
+
+- Path: `docs/integrations/openai-compatible-chat.md`
+- External system or contract: outbound OpenAI-compatible `/chat/completions` API.
+- Current usage in scriptorium: `OpenAICompatibleClient.Generate` posts chat messages and optional JSON schema response format.
+- Version or compatibility notes: compatibility is defined by the subset used in code, not by a pinned OpenAI API version.
+- What should be documented: endpoint construction, request fields, auth header behavior, timeout behavior, expected response shape, error handling, structured output payload.
+- What should not be documented: provider features not serialized by code, retries, streaming, tool calls, reasoning controls, or extra provider params.
+
+### Narratio CLI subprocess
+
+- Path: `docs/integrations/narratio.md`
+- External system or contract: Narratio calling scriptorium as a subprocess.
+- Current usage in scriptorium: public CLI commands `run` and `render`.
+- Version or compatibility notes: contract should be tied to implemented CLI flags and exit codes.
+- What should be documented: command construction, config use, input files, vars, profile overrides, timeout override, output paths, stdout/stderr handling, exit status semantics.
+- What should not be documented: future S3 support, future HTTP primary integration, unimplemented prompt IDs as current examples, or Narratio stage state internals.
+
+JSON Schema is important but does not need a separate integration doc in the first migration; keep schema behavior in `docs/config.md` and internal validation docs unless compatibility issues require a dedicated page later.
+
+## Recommended Implementation Sequence
+
+### Stage 1: Canonical README, CLI, and Config
+
+- Goal: make user-facing docs accurate and move reference material to canonical homes.
+- Files to create/update/delete/move: rewrite `README.md`; create `docs/cli.md`; create `docs/config.md`; leave old `docs/config/*.md` temporarily with pointers or delete them only after links are updated.
+- Repository areas to inspect: `cmd/scriptorium/main.go`, `internal/adapter/cli/run.go`, `internal/adapter/cli/run_test.go`, `internal/config`, `internal/promptdef`, `internal/profile`, `internal/validate`, `examples/config.yml`, `prompts/`, `profiles/`, `schemas/`.
+- Acceptance criteria: README is short; CLI flags match parser; config paths and precedence match code; no future work outside roadmap; no repair claims beyond implemented behavior.
+- Suggested validation commands: `go test ./...`; `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --format json`; `rg -n "future|planned|may be added|can be added later|S3|streaming|batch" README.md docs/cli.md docs/config.md`.
+- One-prompt size: yes, if old split config files are deleted or replaced with pointers in the same change.
+
+### Stage 2: Operations and Troubleshooting
+
+- Goal: document operational behavior and known failure modes.
+- Files to create/update/delete/move: create `docs/operations.md`; create `docs/troubleshooting.md`; update README links.
+- Repository areas to inspect: CLI and HTTP adapters, config tests, LLM client tests, validator tests, prompt/profile repository tests.
+- Acceptance criteria: no invented state/resume/backup behavior; troubleshooting entries are actionable and link to canonical docs; HTTP no-auth caveat is clear.
+- Suggested validation commands: `go test ./internal/adapter/cli ./internal/adapter/http ./internal/config ./internal/llm ./internal/validate`; `rg -n "resume|archive|backup|cleanup|state" docs/operations.md docs/troubleshooting.md`.
+- One-prompt size: yes.
+
+### Stage 3: Development Policy and Internal Docs
+
+- Goal: give developers and LLM agents accurate package boundaries and invariants.
+- Files to create/update/delete/move: create `docs/policy/development.md`; update `docs/policy/architecture.md`; create `docs/internal/runner.md`; create `docs/internal/adapters.md`; optionally create `docs/internal/validation.md`.
+- Repository areas to inspect: all `internal/` packages, `go.mod`, root `architecture.md`, tests.
+- Acceptance criteria: package names match repository; current adapters only; future extension ideas absent except links to roadmap; repairer hook boundary is accurate.
+- Suggested validation commands: `go test ./...`; `rg -n "should expose|may be added|future|S3|batch|streaming|database-backed|additional providers" docs/policy docs/internal`.
+- One-prompt size: maybe split into two prompts if `docs/internal/` becomes too large.
+
+### Stage 4: Integration Docs
+
+- Goal: move external contracts out of README and make integration docs precise.
+- Files to create/update/delete/move: create `docs/integrations/http-api.md`; create `docs/integrations/openai-compatible-chat.md`; rewrite `docs/integrations/narratio.md`; update README and CLI/config links.
+- Repository areas to inspect: `internal/adapter/http`, `internal/llm`, `internal/usecase`, CLI tests, HTTP tests, LLM tests.
+- Acceptance criteria: HTTP docs cover only `POST /v1/runs`; OpenAI docs cover only serialized fields; Narratio docs include only implemented CLI integration and no future notes.
+- Suggested validation commands: `go test ./internal/adapter/http ./internal/llm ./internal/adapter/cli`; `rg -n "POST /v1/renders|S3|future|later|batch|streaming" docs/integrations`.
+- One-prompt size: yes.
+
+### Stage 5: Examples and Link Cleanup
+
+- Goal: make examples policy-compliant and verify links after moves.
+- Files to create/update/delete/move: optionally add `examples/render-markdown-summary.sh`; optionally add `examples/http-run.json`; decide whether to move or mirror `prompts/`, `profiles/`, `schemas/` under `examples/`; delete or exclude `local-test/`; remove or replace root `architecture.md`; delete obsolete `docs/config/*.md` if not already removed.
+- Repository areas to inspect: `examples/`, `prompts/`, `profiles/`, `schemas/`, `internal/usecase/integration_test.go`, docs links.
+- Acceptance criteria: examples are copyable, secret-free, and tested where practical; no links to deleted docs; no maintained docs link to `local-test/`.
+- Suggested validation commands: `go test ./...`; `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --format text`; `rg -n "docs/config/|architecture.md|local-test|dnd.structured_events|dnd.glossary_suggestions|dnd.player_summary" README.md docs examples`.
+- One-prompt size: split if moving prompt/profile/schema assets because tests and paths must be updated carefully.
+
+## Validation Plan
+
+- Run `go test ./...` after documentation changes that touch examples, paths, command examples, or config references.
+- Smoke-test the documented render quickstart with `go run ./cmd/scriptorium render --config ./examples/config.yml --prompt generic.markdown_summary --input transcript=./examples/fixtures/transcript.md --format json`.
+- If documenting JSON-schema render/run examples, set `SCRIPTORIUM_API_KEY` or use a profile without `api_key_env`; `Runner.Prepare` validates the named environment variable.
+- Validate CLI flags against `internal/adapter/cli/run.go` and parser tests, especially `render` lacking `--schema-dir` and `serve` rejecting runtime override flags.
+- Validate app config examples against `internal/config/config.go` strict YAML decoding.
+- Validate prompt/profile examples against `internal/promptdef/filesystem_repository.go` and `internal/profile/filesystem_repository.go`.
+- Validate HTTP request examples against `internal/adapter/http/dto.go` strict JSON decoding.
+- Run grep checks for stale or roadmap-only terms outside `docs/roadmap/`: `future`, `planned`, `may be added`, `can be added later`, `S3`, `streaming`, `batch`, `database-backed`, `render endpoint`, and prompt IDs not present in `prompts/`.
+- Run grep checks for stale paths after file moves: `docs/config/`, `architecture.md`, and `local-test`.
+- No automated documentation link checker is currently configured; perform manual link review or add a link checker in a separate roadmap item if desired.
+
+## Open Questions
+
+No open questions block the documentation migration. The recommended path is to document the current implementation conservatively, move future ideas into `docs/roadmap/`, and avoid claiming production behavior for hooks that are present in code but not wired into CLI or HTTP adapters.