diff --git a/docs/roadmap/documentation.md b/docs/roadmap/documentation.md index 6780cae..8275c66 100644 --- a/docs/roadmap/documentation.md +++ b/docs/roadmap/documentation.md @@ -2,509 +2,134 @@ ## Status -Completed on 2026-07-26. This document records the findings of the -documentation audit performed after adoption of the canonical-ownership policy -and the completed refresh that addressed them. +The initial documentation refresh was completed on 2026-07-26. A +post-completion review identified a small set of contract, test-maintainability, +and roadmap-status issues. The ordered follow-up stages below are pending and +should be completed before beginning Step 2 of the +[Promptkit migration roadmap](migration.md). ## Objective -Bring the current Scriptorium documentation into compliance with -`docs/policy/documentation.md` before beginning the Promptkit migration. - -The refresh should: - -- give every topic one canonical owner; -- remove parallel definitions of volatile contracts; -- correct current factual discrepancies; -- preserve useful contributor and operational guidance in the appropriate - documents; -- establish the missing internal overview and ADR history; -- leave Promptkit and other future behavior in roadmap and ADR documents until - implemented. - -This roadmap covers documentation organization and current-behavior accuracy. -A substantive review of `docs/policy/testing.md` remains separate. - -## Audit Baseline - -The audit covered: - -- `README.md`; -- all Markdown files under `docs/`; -- all maintained files under `examples/`; -- the CLI parser and output behavior; -- app-config shapes and defaults; -- HTTP DTOs, limits, and error mappings; -- the public Go facade; -- prompt, profile, schema, artifact, renderer, runner, and LLM behavior; -- the embedded built-in profile assets. - -At the time of the audit: - -- all local Markdown link targets existed; -- `go test ./...` passed; -- the README/CLI render command completed successfully; -- `examples/render-markdown-summary.sh` completed successfully; -- `go run ./examples/go-library/prepare` completed successfully. - -The dominant problem is duplicated ownership rather than broad factual -staleness. CLI, configuration, HTTP, validation, state, and security behavior -are repeated across contracts, operations, troubleshooting, integrations, -consumer guides, architecture, and internal documents. - -## Required Accuracy Corrections - -Make these corrections while revising the owning documents: - -1. `docs/policy/architecture.md` says Scriptorium has three entry paths but - lists four. Distinguish the three executable entry paths from the public Go - package, or describe four total entry paths. -2. The built-in profile catalog in `docs/config.md` omits - `deepseek-4-flash`. Reconcile the complete catalog with - `internal/profile/builtin/assets/`. -3. `docs/config.md` marks prompt `output.repair_attempts` as required. The - loader permits omission and resolves it to zero; document it as optional - with an effective default of zero. -4. `docs/consumers/pkg-scriptorium.md` says every input map key must match a - declared prompt input. The renderer enforces required declared inputs and - resolves names actually referenced by templates, but it does not reject - every undeclared extra input. Describe the implemented boundary. -5. `docs/troubleshooting.md` says validation errors can be inspected in CLI - stderr. The CLI success summary reports the number of validation errors, not - their detailed messages. Correct the diagnostic guidance or separately - change the product before documenting richer output. -6. `docs/integrations/openai-compatible-chat.md` describes the `session_id` - limit as characters. The implementation counts Unicode code points; use the - canonical terminology consistently. -7. The same integration document makes a time-sensitive statement about - provider-supported service-tier values even though Scriptorium accepts and - forwards any non-empty value. Remove the provider catalog claim or cite and - version an intentionally maintained external contract. -8. The `render` and `serve` flag sections in `docs/cli.md` list - `--prompt-dir` both as an effective requirement and again as an optional - flag. Keep one complete flag entry and separately explain how the - requirement may be satisfied. -9. Troubleshooting command examples containing placeholders such as - `` are not directly copyable shell commands. Use clearly defined - shell variables, concrete maintained examples, or prose diagnostic steps. - -## Canonical Structure Changes - -### Create `docs/internal/overview.md` - -Create the canonical implemented-component inventory and move the concrete -package map out of architecture. - -The overview should: - -- list current public, command, adapter, domain, use-case, source, format, - validation, and LLM components; -- give each component a short implemented responsibility; -- link to focused internal documents and relevant external contracts; -- avoid restating global architecture rules or user-facing behavior. - -Update `docs/development.md` to route general repository-orientation work -through the overview once it exists. - -### Establish `docs/adr/` - -Create the ADR directory under the policy already defined in -`docs/policy/documentation.md`. - -Record the significant accepted documentation-ownership decision in an initial -ADR if historical rationale is useful. The Promptkit split ADR remains part of -Step 2 in `docs/roadmap/migration.md` and should not be pulled ahead of its -gate merely to populate the directory. - -ADRs must own decision context, alternatives, rationale, and consequences. -Roadmaps must continue to own implementation status and sequencing. - -### Consolidate Troubleshooting Into Operations - -The ownership table assigns recovery to `docs/operations.md` and does not give -troubleshooting a separate canonical owner. Fold the useful -symptom/diagnosis/safe-recovery material from `docs/troubleshooting.md` into a -concise operations section, then remove `docs/troubleshooting.md`. - -The consolidated material should: - -- organize failures by operational task rather than duplicate every contract - field and status; -- link exact CLI syntax and exit codes to `docs/cli.md`; -- link config fields and defaults to `docs/config.md`; -- link HTTP codes and schemas to `docs/api.md`; -- describe only diagnostic and recovery actions locally. - -Update README, development, integration, and other links after the removal. - -## File-By-File Revision Catalog - -### `README.md` - -- Retain the product description and one tested end-to-end quickstart. -- Keep the quickstart linked to the complete CLI contract. -- Replace the repeated example inventory with a short link to the maintained - examples section in the appropriate canonical contract, or make each retained - path a direct link. -- Keep the documentation list navigational; do not summarize contracts there. -- Remove the troubleshooting link if troubleshooting is consolidated into - operations. - -### `docs/development.md` - -- Retain only contributor orientation, task-specific routing, and baseline - validation. -- Add `docs/internal/overview.md` to the appropriate reading paths once it - exists. -- Update the operations/troubleshooting row after consolidation. -- Link to actual ADRs when created; do not duplicate their decisions. -- Confirm that all detailed recipes removed from the former development guide - have an internal owner before declaring the refresh complete. - -### `docs/policy/architecture.md` - -- Fix the entry-path count. -- Replace the concrete package inventory with a link to - `docs/internal/overview.md`. -- Remove config precedence, field-level behavior, exact routes, and exact - import-path contracts except for the smallest orientation summary and links - to their owners. -- Remove the local testing checklist and link to - `docs/policy/testing.md`. Component-specific test inventories remain in - internal docs. -- Remove the local documentation checklist and link to - `docs/policy/documentation.md`. -- Keep normative system shape, ownership, dependency direction, state - philosophy, security properties, error-handling principles, invariants, and - non-goals. -- Review the coding and dependency rules removed from the former development - guide. Preserve still-valid normative rules here without listing the current - dependency inventory, which is concrete implementation information. -- Move implementation facts such as current repairer wiring to the relevant - internal component document unless they are intentionally elevated to - architecture invariants. - -### `docs/policy/documentation.md` - -- No structural rewrite is currently required. -- Recheck its ownership table after the troubleshooting consolidation and ADR - creation. -- Keep future policy refinements in this document rather than distributing - documentation rules across architecture or development. - -### `docs/policy/testing.md` - -- It is the sole owner of global testing philosophy and sufficiency rules. -- Remove competing global testing guidance from architecture and other docs. -- Leave the planned substantive testing-policy review for its separate work - cycle. - -### `docs/cli.md` - -- Keep the complete command, argument, flag, output, and exit-code contract. -- Remove the repeated definition of config precedence and link to - `docs/config.md`. -- Move profile source and override semantics that are not CLI invocation - semantics to `docs/config.md`. -- Resolve duplicate `--prompt-dir` entries in the `render` and `serve` - sections. -- Verify every registered flag, deprecated alias, requirement, zero-value - behavior, and output destination against `internal/adapter/cli/run.go`. -- Clarify timeout conversion for subsecond Go durations if that behavior is - intended to remain public. -- Keep only compact workflow examples and link to maintained scripts under - `examples/`. - -### `docs/config.md` - -- Keep app-config discovery, precedence, fields, defaults, validation, - credential-supply mechanisms, prompt/profile formats, and selectable profile - catalog as the canonical contract. -- Replace complete inline config, prompt, and profile files with the smallest - useful snippets and links to copyable files under `examples/`. -- If a production-oriented complete config remains useful, move it into - `examples/` and link it rather than maintaining a second complete copy. -- Correct `repair_attempts` optionality. -- Add `deepseek-4-flash` and reconcile every built-in profile ID, model, and - credential-variable name with embedded assets. -- Establish a low-friction way to prevent catalog drift, such as a generated - catalog section or a focused consistency check, if its maintenance value - justifies the cost. -- Keep schema configuration and prompt/profile file-format rules here; move - implementation mechanics to internal source docs. -- Reduce the HTTP artifact-reference section to configuration meaning and link - request shapes/statuses to `docs/api.md` and runtime security handling to - `docs/operations.md`. -- Keep the secret-supply mechanism here; move deployment permissions and - sensitive-runtime handling to operations. - -### `docs/api.md` - -- Keep routes, media types, request/response schemas, strict JSON behavior, - status codes, HTTP retry semantics, and HTTP artifact-access outcomes. -- Replace repeated app-config defaults and CLI flag syntax with links to - `docs/config.md` and `docs/cli.md`. -- Retain HTTP limit effects but let configuration own field defaults and - precedence. -- Keep the lexical containment and symlink behavior as externally observable - API/security behavior; operations may summarize its deployment consequence - and link back. -- Reduce request and response examples to compact contract-bearing shapes and - link to `examples/http-run.json`. -- Reverify all DTO fields, omission behavior, limit responses, and error mappings - against `internal/adapter/http/`. - -### `docs/operations.md` - -- Rewrite as a task-oriented runbook rather than a secondary CLI, config, and - API reference. -- Keep deployment layout, process permissions, sensitive runtime artifact - handling, normal workflow, service exposure, capacity planning, recovery, - and the stateless rerun model. -- Link exact commands and exit codes to `docs/cli.md`. -- Link config fields, defaults, validation modes, and credential-supply - mechanics to `docs/config.md`. -- Link HTTP route, status, schema, and limit semantics to `docs/api.md`. -- Remove the repeated maintained-example inventory and link to the owning - examples section. -- Incorporate the useful troubleshooting material and then delete - `docs/troubleshooting.md`. - -### `docs/troubleshooting.md` - -- Correct the CLI validation-error diagnostic during migration. -- Move useful symptom, diagnostic, and safe-fix material into - `docs/operations.md`. -- Remove repeated definitions of flags, fields, defaults, HTTP codes, routes, - and validation semantics; link to their canonical contracts. -- Delete this file after all useful recovery guidance and inbound links have - been handled. - -### `docs/consumers/api.md` - -- Keep integration-surface selection, minimal consumer workflow, and consumer - responsibilities. -- Retain one minimal Go example as permitted instructional content and link to - the complete package contract and maintained example. -- Replace repeated CLI exit and HTTP status definitions with links to their - canonical contracts. -- Replace repeated deployment input and credential definitions with links to - configuration and operations. -- Keep retry and artifact-retention decisions as consumer responsibilities - without restating wire semantics. - -### `docs/consumers/pkg-scriptorium.md` - -- Keep the import path, public constructors, options, types, errors, workflows, - and public security boundary as the canonical Go package contract. -- Correct the input-name validation statement. -- Reverify public fields, JSON behavior, option precedence, nil behavior, - source containment, direct-key handling, and sentinel errors against the root - package. -- Keep a minimal package example and link to - `examples/go-library/prepare`; avoid duplicating complete maintained source. -- Link prompt/profile/schema file formats to `docs/config.md` rather than - redefining them. - -### `docs/integrations/subprocess.md` - -- Narrow this document to subprocess-specific compatibility and process - integration concerns. -- Link command syntax, flags, stdout/stderr behavior, and exit codes to - `docs/cli.md` rather than maintaining parallel definitions. -- Link config search and field semantics to `docs/config.md`. -- Retain process isolation, environment propagation, stream capture, output - ownership, cancellation/termination expectations, and security guidance when - these are specific to subprocess consumers. -- Move task-oriented surface-selection advice to `docs/consumers/api.md` if it - is currently duplicated. - -### `docs/integrations/openai-compatible-chat.md` - -- Keep only the outbound HTTP wire contract: endpoint construction, request - payload, authentication header, timeouts, response subset, and unsupported - protocol behavior. -- Remove the concrete implementation-file introduction and internal error - sentinel catalog. -- Correct the `session_id` unit to Unicode code points. -- Remove or deliberately source/version the provider service-tier claim. -- Reverify reserved `extra_params` keys, structured-output envelope, - cache-control encoding, usage mapping, and response requirements against - `internal/llm/openai_compatible_client.go`. -- Link runner schema preparation to internal runner documentation rather than - explaining orchestration locally. - -### `docs/internal/overview.md` (new) - -- Own the complete current component and package inventory. -- Link each component to focused internal docs and external contracts. -- Include the command entry point and public facade without redefining their - external behavior. -- Absorb the still-useful concrete repository-layout material removed from the - old development guide and architecture policy. - -### `docs/internal/adapters.md` - -- Keep implementation flow, collaborators, translation boundaries, wiring, - error-mapping mechanisms, and relevant tests. -- Remove exact flag lists, config field definitions, route schemas, response - codes, and exit-code definitions; link their canonical owners. -- Keep only package-local invariants rather than repeating global architecture. -- Incorporate the former contributor recipes for adding app-config fields, CLI - flags, and adapter capabilities. -- Describe how to verify affected adapter contracts without duplicating the - global testing policy. - -### `docs/internal/runner.md` - -- Keep prepare/run flow, dependency boundaries, state transitions, failure - categories, repair mechanics, hashing/validation coordination, and relevant - tests. -- Remove the app-config field list and link to adapter/config documentation. -- Keep internal sentinel and collaborator information, but link public error - behavior to the package/API contracts. -- Distinguish package-local guarantees from global architectural invariants. -- Add a focused change recipe for runner orchestration when useful. - -### `docs/internal/sources.md` - -- Keep loader implementations, source precedence mechanics, containment - implementation, failure categories, collaborators, and relevant tests. -- Link YAML/JSON field definitions and defaults to `docs/config.md`. -- Link HTTP-visible artifact outcomes to `docs/api.md` and deployment handling - to operations. -- Incorporate the former recipe for updating prompt, profile, schema, and - built-in-profile assets. -- Distinguish implementation mechanics from external source contracts. - -### `docs/internal/llm.md` (new) - -Create a focused internal document if the implementation material removed from -the outbound integration contract remains useful. - -It should own: - -- client construction and internal collaborator boundaries; -- internal request mapping and timeout selection; -- internal error categories; -- relevant tests and package-local change guidance. - -It must link to `docs/integrations/openai-compatible-chat.md` for the wire -contract rather than repeat payload definitions. - -### `docs/roadmap/migration.md` - -- Retain migration status, gates, sequencing, and task breakdowns. -- Mark the documentation-refresh gate complete only after this roadmap's - completion criteria are satisfied. -- After the Promptkit split ADR is accepted, replace duplicated decision - rationale and architectural ownership detail with a concise summary and ADR - link where practical. -- Update paths and document ownership affected by this refresh. - -### `examples/` - -- Keep complete copyable artifacts here rather than in reference documents. -- Verify `config.yml` and `config.full.yml` through the real config loader. -- Keep prompt/profile/schema examples covered by representative repository and - engine tests. -- Keep the render script and Go package example runnable from the repository - root. -- Validate `http-run.json` structurally against the HTTP DTO contract without - requiring a live model endpoint. -- Add a separate production-oriented config example only if that maintained - artifact has clear value; otherwise remove the duplicate production block - from config documentation. -- Preserve the fixtures as non-sensitive synthetic data. - -## Implementation Sequence - -### Stage 1: Establish Canonical Developer Structure - -1. Create the initial documentation ADR if retained. -2. Create `docs/internal/overview.md`. -3. Rewrite architecture around normative ownership and invariants. -4. Update `docs/development.md` routing. -5. Rescue still-valid development rules and assign every removed recipe. - -**Gate:** Contributor orientation, architecture, component inventory, policies, -and decision history have distinct owners and no stale links. - -### Stage 2: Correct And Consolidate External Contracts - -1. Revise `docs/config.md` and correct its known factual issues. -2. Revise `docs/cli.md`. -3. Revise `docs/api.md`. -4. Revise both consumer guides. -5. Revise both integration contracts. -6. Verify each volatile contract against code and maintained assets. - -**Gate:** Each externally observable field, default, flag, route, status, import -path, and wire behavior has one authoritative definition. - -### Stage 3: Refocus Internal Documentation - -1. Revise adapter, runner, and source docs. -2. Add the internal LLM document if warranted. -3. Incorporate the former subsystem change recipes. -4. Remove global policy and external-contract duplication. - -**Gate:** Internal docs explain implementation and package-local guarantees, -link external contracts, and contain the detailed contributor recipes needed -for safe changes. - -### Stage 4: Rewrite Operations And Consolidate Recovery Guidance - -1. Rewrite operations as a runbook. -2. Migrate useful troubleshooting content. -3. Delete `docs/troubleshooting.md`. -4. Update all inbound links and the development reading guide. - -**Gate:** Operations owns deployment and recovery without serving as a second -CLI, config, or API reference. - -### Stage 5: Normalize Examples And Orientation - -1. Move any remaining complete copyable artifacts into `examples/`. -2. Update README navigation and example links. -3. Run maintained smoke examples. -4. Add only high-value automated checks justified by the testing policy. - -**Gate:** Examples are canonical, runnable, secret-free, linked from their -owning references, and not duplicated as complete files in prose docs. - -### Stage 6: Final Compliance Pass - -1. Validate all local links and paths. -2. Search for duplicated volatile values across non-owning documents. -3. Reconcile flags, config fields/defaults, API DTOs/codes, public types/errors, - and built-in profiles with implementation. -4. Run `go test ./...`, `go vet ./...`, and the documented build command. -5. Run the maintained render script and Go package example. -6. Confirm that non-roadmap current docs contain implemented behavior only. -7. Record completion in this roadmap and update the documentation gate in - `docs/roadmap/migration.md`. - -**Gate:** Code, tests, examples, contracts, internal documentation, operations, -policies, and roadmap status agree. Completed on 2026-07-26. +Maintain an accurate, concise documentation baseline under the canonical +ownership policy, and close the remaining review findings before architectural +migration begins. + +## Completed Refresh + +The initial refresh: + +- established the documentation and ADR policies and recorded + [ADR 0001](../adr/0001-adopt-canonical-documentation-ownership.md); +- made `docs/development.md` the contributor entry point and added a canonical + internal component overview; +- refocused architecture on normative boundaries and moved implementation + detail into focused internal documents; +- consolidated CLI, configuration, HTTP, public Go, and integration contracts + under distinct canonical owners; +- rewrote operations as a runbook, incorporated useful recovery guidance, and + removed the standalone troubleshooting document; +- normalized maintained examples and added an offline HTTP request-example + check; +- corrected the factual discrepancies found by the original audit, including + the entry-path count, built-in profile catalog, prompt output optionality, + input-map behavior, validation diagnostics, session-ID units, and CLI flag + descriptions; and +- validated local links, configuration examples, smoke examples, tests, vet, + and the documented build. + +## Follow-Up Findings + +The post-completion review found: + +1. The effective precedence between `Config.Timeout` and a supplied + `HTTPClient.Timeout` is not consistently expressed by the implementation, + tests, and documentation. +2. The public Go contract overgeneralizes basename selection for single-file + sources; prompt and profile files remain selected by YAML ID, while + `WithSchemaFile` uses the schema file's base name. +3. The maintained HTTP request-example test pins incidental fixture values + beyond the structural contract it needs to protect. +4. The Promptkit migration roadmap's overall status says no work is implemented + even though its documentation-refresh step is complete. +5. One architecture invariant incorrectly describes configuration, prompt, and + profile inputs as JSON rather than YAML. + +## Follow-Up Implementation Sequence + +### Stage 1: Resolve The Timeout Contract + +1. Decide and document the precedence among: + - `Config.Timeout`; + - a non-zero timeout on a supplied `http.Client`; and + - an explicit per-request `timeout_seconds` override. +2. Align `internal/llm` with that decision, removing any constructor state that + is immediately discarded during generation. +3. Add or revise a behavior-level test that performs `Generate` with a supplied + HTTP client and proves the selected timeout without relying only on private + constructor state. +4. Update the canonical public Go contract, outbound OpenAI-compatible + integration contract, and internal LLM document together. + +**Gate:** There is one documented timeout-precedence rule, the implementation +follows it, and a request-level test protects it. + +### Stage 2: Correct The Remaining Contract And Architecture Text + +1. Clarify in the public Go contract that: + - prompt and profile single-file sources are selected by their YAML IDs; + - `WithPromptFile` resolves relative `content_file` paths from the prompt + file's directory; and + - `WithSchemaFile` exposes the schema through its base name. +2. Correct or remove the duplicate architecture invariant so it accurately + distinguishes strict YAML configuration, prompt, and profile inputs from + strict JSON HTTP input. +3. Recheck the affected statements against the public facade and source + implementations without duplicating file-format definitions outside the + configuration contract. + +**Gate:** Public source-option behavior and strict-decoding invariants agree +with the implementation and retain one canonical owner. + +### Stage 3: Refine The Maintained HTTP Example Check + +1. Preserve an offline check that `examples/http-run.json` is accepted by the + HTTP request contract. +2. Remove assertions that freeze prompt IDs, profile IDs, fixture paths, or + example variable values unless a specific value protects an independently + identified contract. +3. Rely on focused handler tests, rather than the maintained-example check, for + exhaustive DTO-to-domain mapping behavior. +4. Run the affected HTTP adapter tests and confirm that the revised test still + fails for structurally invalid example content. + +**Gate:** The maintained example has durable structural protection without +duplicating mapping coverage or making legitimate fixture edits unnecessarily +expensive. + +### Stage 4: Reconcile Status And Revalidate The Baseline + +1. Update the Promptkit migration roadmap's overall status to distinguish the + completed documentation step from the unimplemented migration steps. +2. Recheck local Markdown links and paths. +3. Verify both maintained configuration files through the real configuration + loader. +4. Run the maintained render script and Go package example. +5. Run `go test ./...`, `go vet ./...`, and `go build ./cmd/scriptorium`. +6. Record completion of these follow-up stages here and confirm the + documentation gate in the Promptkit migration roadmap. + +**Gate:** Code, tests, examples, canonical contracts, architecture, and both +roadmap status documents agree. Promptkit migration Step 2 may then begin. ## Completion Criteria -The documentation refresh is complete when: +This follow-up is complete when: -- every topic in the ownership table has one canonical owner; -- `docs/internal/overview.md` exists and architecture no longer owns the package - inventory; -- significant accepted decisions have an ADR owner; -- all known factual discrepancies in this audit are corrected; -- subsystem recipes from the former development guide have been preserved in - relevant internal docs; -- operations contains the retained recovery guidance and the standalone - troubleshooting document is removed; -- contracts and internal docs link to one another instead of maintaining - parallel volatile definitions; -- maintained examples are runnable and not duplicated as complete prose - examples; -- local links, validation commands, tests, and smoke checks pass; -- the documentation-refresh gate in the Promptkit migration roadmap is marked - complete. +- timeout precedence is unambiguous, implemented, and protected by an + execution-level test; +- public single-file source behavior is accurately documented; +- strict YAML and JSON boundaries are accurately stated; +- the HTTP example check protects structure without pinning incidental content; +- roadmap status accurately distinguishes completed and pending migration work; + and +- links, examples, tests, vet, and build validation pass.