Update documentation and testing policies and add a migration plan to cleanly separate the scriptorium CLI from the promptkit internals
This commit is contained in:
@@ -1,4 +1 @@
|
||||
Please carefully review the relevant documents in `docs/policy` before making any changes to this repository.
|
||||
- `development.md` defines the contributor workflow for this application.
|
||||
- `architecture.md` provides the canonical high-level architecture policy for this repository, and should be reviewed before writing or changing any code.
|
||||
- `documentation.md` provides the canonical documentation policy for this repository, and should be reviewed before writing or changing any documentation.
|
||||
Please review `docs/development.md` for initial orientation in this repository and follow its task-specific reading guide.
|
||||
|
||||
60
docs/development.md
Normal file
60
docs/development.md
Normal file
@@ -0,0 +1,60 @@
|
||||
# Development
|
||||
|
||||
This is the contributor entry point for Scriptorium. Use the task-specific
|
||||
reading guide below before making changes. Canonical architecture, contracts,
|
||||
component behavior, and policies remain in their owning documents.
|
||||
|
||||
## Initial Orientation
|
||||
|
||||
Before starting work:
|
||||
|
||||
1. inspect the working tree and preserve unrelated changes;
|
||||
2. read the architecture policy for code or design work;
|
||||
3. read the policy, contract, and internal documents listed for the task;
|
||||
4. inspect the relevant implementation and tests before deciding how to change
|
||||
them.
|
||||
|
||||
Start with:
|
||||
|
||||
- [Architecture policy](policy/architecture.md) for system boundaries,
|
||||
invariants, and non-goals;
|
||||
- [Documentation policy](policy/documentation.md) before changing
|
||||
documentation;
|
||||
- [Testing policy](policy/testing.md) before adding, rewriting, or deleting
|
||||
tests.
|
||||
|
||||
## Task-Specific Reading Guide
|
||||
|
||||
| Task | Read before changing |
|
||||
| --- | --- |
|
||||
| Public Go package or engine behavior | [Go package consumer contract](consumers/pkg-scriptorium.md), [runner internals](internal/runner.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
||||
| CLI commands, flags, output, or exit behavior | [CLI contract](cli.md) and [adapter internals](internal/adapters.md) |
|
||||
| HTTP routes, DTOs, limits, or status mapping | [HTTP API contract](api.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
||||
| Application configuration | [Configuration contract](config.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
||||
| Prompt, profile, schema, or artifact loading | [Configuration contract](config.md) and [source internals](internal/sources.md) |
|
||||
| Runner orchestration, rendering, validation, or repair | [Runner internals](internal/runner.md) and [source internals](internal/sources.md) |
|
||||
| OpenAI-compatible request or response behavior | [OpenAI-compatible integration](integrations/openai-compatible-chat.md), [runner internals](internal/runner.md), and [adapter internals](internal/adapters.md) |
|
||||
| Subprocess behavior | [Subprocess integration](integrations/subprocess.md) and [CLI contract](cli.md) |
|
||||
| Runtime operation, recovery, or troubleshooting | [Operations](operations.md) and [troubleshooting](troubleshooting.md) |
|
||||
| Examples or copyable assets | The owning contract for the demonstrated behavior and the related files under `examples/` |
|
||||
| Architecture decisions or future work | The [documentation policy](policy/documentation.md), relevant accepted ADRs under `adr/`, and relevant roadmap documents under `roadmap/` |
|
||||
|
||||
For cross-cutting changes, follow every applicable row. Internal component
|
||||
documents own detailed subsystem change recipes.
|
||||
|
||||
## Baseline Validation
|
||||
|
||||
Use focused checks while iterating, then run validation proportionate to the
|
||||
change and the risks described by the testing policy.
|
||||
|
||||
The repository-level baseline for code changes is:
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/scriptorium
|
||||
```
|
||||
|
||||
Documentation-only work does not require the full Go suite unless it changes
|
||||
commands, examples, generated output, or another behavior that the suite
|
||||
validates. Always check changed links, paths, examples, and canonical ownership.
|
||||
@@ -1,110 +0,0 @@
|
||||
# Development Guide
|
||||
|
||||
This document defines contributor workflow for Scriptorium.
|
||||
|
||||
## Repository Layout
|
||||
|
||||
- root package `scriptorium`: public Go facade, options, types, and error mapping.
|
||||
- `cmd/scriptorium`: application entrypoint.
|
||||
- `internal/domain`: core contracts.
|
||||
- `internal/usecase`: runner orchestration.
|
||||
- `internal/adapter/cli`: CLI adapter.
|
||||
- `internal/adapter/http`: HTTP adapter.
|
||||
- `internal/config`: application settings loading and precedence.
|
||||
- `internal/defaults`: default constants.
|
||||
- `internal/promptdef`: prompt-definition repository.
|
||||
- `internal/profile`: execution-profile repository.
|
||||
- `internal/profile/builtin`: embedded built-in execution profiles.
|
||||
- `internal/filecatalog`: shared source discovery and path helpers.
|
||||
- `internal/artifact`: artifact readers.
|
||||
- `internal/prompt`: prompt rendering.
|
||||
- `internal/llm`: LLM client interface and OpenAI-compatible implementation.
|
||||
- `internal/validate`: validation interfaces and implementation.
|
||||
- `internal/format`: prepared-run formatting.
|
||||
- `docs/`: canonical documentation.
|
||||
- `examples/`: copyable maintained examples and fixtures.
|
||||
|
||||
## Common Commands
|
||||
|
||||
Build:
|
||||
|
||||
```bash
|
||||
go build ./cmd/scriptorium
|
||||
```
|
||||
|
||||
Test:
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Targeted test runs commonly used during changes:
|
||||
|
||||
```bash
|
||||
go test .
|
||||
go test ./internal/adapter/cli ./internal/adapter/http ./internal/usecase
|
||||
go test ./internal/...
|
||||
```
|
||||
|
||||
## Coding Conventions
|
||||
|
||||
- Prefer small interfaces at package boundaries.
|
||||
- Keep adapter packages focused on translation and IO concerns.
|
||||
- Keep domain/use-case logic outside adapters.
|
||||
- Wrap errors with operation context.
|
||||
- Use strict decoding for user-provided YAML/JSON where applicable.
|
||||
- Avoid introducing dependencies unless they materially reduce risk/complexity.
|
||||
|
||||
## Dependency Policy
|
||||
|
||||
- Prefer standard library unless an external library is clearly justified.
|
||||
- Current non-stdlib dependencies are intentionally small:
|
||||
- `gopkg.in/yaml.v3` for YAML decoding.
|
||||
- `github.com/santhosh-tekuri/jsonschema/v6` for JSON Schema validation.
|
||||
- Do not leak dependency-specific types across unrelated package boundaries.
|
||||
|
||||
## How To Add App Config Fields
|
||||
|
||||
1. Add fields in `internal/config/config.go` (`Config`, `AppSettings`, and/or `CLIOverrides` as needed).
|
||||
2. Apply defaults in `BuiltInDefaults()` when required.
|
||||
3. Parse and validate in `applyConfig` / `ApplyCLIOverrides`.
|
||||
4. Wire the field through the consuming adapter(s).
|
||||
5. Add/update config tests in `internal/config/config_test.go`.
|
||||
6. Update canonical docs (`docs/config.md`, and other affected docs).
|
||||
|
||||
## How To Add CLI Flags
|
||||
|
||||
1. Add flags in `internal/adapter/cli/run.go` for the relevant command.
|
||||
2. Ensure precedence behavior remains consistent with app config rules.
|
||||
3. Keep `run`, `render`, and `serve` flag surfaces intentionally scoped.
|
||||
4. Add/update parser and command tests in `internal/adapter/cli/run_test.go`.
|
||||
5. Update `docs/cli.md` and any related docs/examples.
|
||||
|
||||
## How To Add Adapters Or Adapter Capabilities
|
||||
|
||||
1. Define or reuse the appropriate interface boundary in domain/use-case packages.
|
||||
2. Implement adapter code under `internal/adapter/<name>` (or relevant boundary package).
|
||||
3. Keep business decisions in `internal/usecase`.
|
||||
4. Add focused adapter tests for mapping, parse, and error behavior.
|
||||
5. Document the new/changed boundary in `docs/internal/adapters.md`.
|
||||
6. If source-loading behavior changes, update `docs/internal/sources.md`.
|
||||
7. If an external contract changes, update the canonical public or integration doc in the same change.
|
||||
|
||||
## How To Update Prompt/Profile/Schema Assets
|
||||
|
||||
1. Keep prompt/profile/schema files valid under strict loaders.
|
||||
2. Keep examples secret-free.
|
||||
3. Re-run tests that cover prompt/profile/validation behavior.
|
||||
4. Update `docs/config.md` and any docs that reference changed contracts.
|
||||
|
||||
## Documentation Update Expectations
|
||||
|
||||
When behavior changes:
|
||||
|
||||
1. Update canonical doc locations, not duplicate files.
|
||||
2. Keep non-roadmap docs limited to implemented behavior.
|
||||
3. Update links after file moves/renames.
|
||||
4. Re-run relevant tests and smoke commands.
|
||||
5. For internal boundary docs, check references with `rg "docs/internal|internal/sources" docs/policy docs/internal`.
|
||||
|
||||
Docs work is complete only when code/tests/examples/docs agree.
|
||||
@@ -1,446 +1,163 @@
|
||||
# Go Project Documentation Policy
|
||||
# Documentation Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
Project documentation must help five audiences:
|
||||
|
||||
1. users who need to run the application;
|
||||
2. administrators/operators who need to configure and operate it;
|
||||
3. developers who need to understand and change it safely;
|
||||
4. LLM coding agents that need clear scope, boundaries, and invariants;
|
||||
5. developers and LLM coding agents integrating this project from another codebase.
|
||||
|
||||
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
|
||||
This policy assigns each documentation topic to one canonical owner. Its goal is
|
||||
to keep this repository's documentation accurate, concise, discoverable, and
|
||||
resistant to drift for users, operators, developers, integrators, and LLM
|
||||
coding agents.
|
||||
|
||||
## Core Rules
|
||||
|
||||
### 1. Keep docs concise
|
||||
### One Canonical Owner
|
||||
|
||||
Each document should cover a defined scope and only the essentials for that scope.
|
||||
Each authoritative fact belongs in one document. A non-owning document may give
|
||||
a short, stable summary for orientation, but it must link to the canonical owner
|
||||
instead of repeating volatile details.
|
||||
|
||||
Avoid:
|
||||
- long background explanations;
|
||||
- repeated reference material;
|
||||
- implementation detail in user-facing docs;
|
||||
- aspirational language outside roadmap docs;
|
||||
- verbose examples where one minimal example is clearer.
|
||||
Volatile details include commands, flags, configuration fields and defaults,
|
||||
module keys, schemas, file names, paths, status codes, retry behavior, and
|
||||
runtime guarantees. If readers could reasonably treat a statement as a
|
||||
contract, maintain it only in the owning document.
|
||||
|
||||
### 2. Document only implemented behavior outside roadmap files
|
||||
Minimal tested usage examples are allowed outside the owning contract when this
|
||||
policy assigns them an orientation or instructional purpose. They must link to
|
||||
the canonical contract and must not redefine complete syntax, defaults, or
|
||||
semantics.
|
||||
|
||||
### Current And Future Behavior
|
||||
|
||||
Outside `docs/roadmap/`, documentation describes implemented behavior only.
|
||||
Partial features may be described only to their implemented boundary.
|
||||
|
||||
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
|
||||
ADRs are the narrow exception: an ADR may record an accepted architectural
|
||||
decision before implementation, but acceptance must not be presented as proof
|
||||
that the behavior exists. The roadmap owns implementation status and sequencing
|
||||
until the decision is implemented. Current architecture, user, operator,
|
||||
integration, and internal documentation are updated when the behavior lands.
|
||||
|
||||
- `docs/roadmap/`
|
||||
### Audience And Detail
|
||||
|
||||
Write for the document's stated audience and include only the detail needed for
|
||||
its owned topic. User and operator docs should not expose implementation detail.
|
||||
Developer docs should link to user-facing and external contracts rather than
|
||||
restate them.
|
||||
|
||||
### Examples
|
||||
|
||||
Complete copyable files belong in `examples/`. Documentation may use the
|
||||
smallest illustrative snippet needed to explain its owned topic, but should link
|
||||
to maintained examples instead of embedding a second complete copy.
|
||||
|
||||
Examples must be valid, secret-free, and tested where practical. Commands and
|
||||
configuration used in documentation should match the application.
|
||||
|
||||
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
|
||||
### Security And Privacy
|
||||
|
||||
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
|
||||
|
||||
### 3. Use canonical homes
|
||||
|
||||
Each type of information should have one canonical location.
|
||||
|
||||
Canonical homes:
|
||||
|
||||
- project purpose and quickstart: `README.md`
|
||||
- development principles: `docs/policy/architecture.md`
|
||||
- public HTTP API reference: `docs/api.md`
|
||||
- configuration reference: `docs/config.md`
|
||||
- CLI reference: `docs/cli.md`
|
||||
- operations and recovery: `docs/operations.md`
|
||||
- troubleshooting: `docs/troubleshooting.md`
|
||||
- public API/package consumer guidance: `docs/consumers/`
|
||||
- implemented internals: `docs/internal/`
|
||||
- external protocol, service, and file-format contracts: `docs/integrations/`
|
||||
- future work: `docs/roadmap/`
|
||||
- contributor workflow: `docs/policy/development.md`
|
||||
- copyable examples: `examples/`
|
||||
|
||||
Other files should summarize briefly and link to the canonical source.
|
||||
|
||||
### 4. Keep examples real
|
||||
|
||||
Examples should be valid, maintained, and free of secrets.
|
||||
|
||||
Where practical:
|
||||
- example configs should load successfully;
|
||||
- example commands should match real CLI syntax;
|
||||
- important examples should be covered by tests.
|
||||
|
||||
## Documentation Profiles
|
||||
|
||||
All projects require:
|
||||
|
||||
- `README.md`
|
||||
- `docs/policy/architecture.md`
|
||||
|
||||
Additional docs depend on the project.
|
||||
|
||||
### Small library
|
||||
|
||||
Recommended:
|
||||
- `docs/policy/development.md`, if contributor conventions are non-obvious
|
||||
|
||||
### Simple CLI
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/policy/development.md`
|
||||
|
||||
### Config-driven CLI
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`
|
||||
- `docs/config.md`
|
||||
|
||||
Recommended:
|
||||
- `examples/`
|
||||
- `docs/policy/development.md`
|
||||
|
||||
### Stateful or operator-facing application
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- `examples/`
|
||||
- `docs/policy/development.md`
|
||||
|
||||
### Modular, service-oriented, or orchestration application
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/`
|
||||
- `docs/policy/development.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- validated examples under `examples/`
|
||||
|
||||
### Public HTTP API service
|
||||
|
||||
Required:
|
||||
- `docs/api.md`
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/`
|
||||
- `docs/policy/development.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- `docs/consumers/`, for task-oriented client integration guides
|
||||
- `docs/integrations/`, for upstream/downstream service contracts
|
||||
- validated examples under `examples/`
|
||||
|
||||
### Project with public packages or consumer APIs
|
||||
|
||||
Required:
|
||||
- `docs/consumers/api.md`
|
||||
- one `docs/consumers/pkg-<name>.md` file per public package, if public packages exist
|
||||
|
||||
Recommended:
|
||||
- copyable consumer examples under `examples/`, if practical
|
||||
|
||||
## Required Documents
|
||||
|
||||
### README.md
|
||||
|
||||
**Audience:** users, administrators, operators
|
||||
|
||||
The README is the outward-facing project orientation page.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. concise description;
|
||||
2. elevator pitch;
|
||||
3. shortest useful command or usage example;
|
||||
4. links to targeted docs.
|
||||
|
||||
The README should be short. It is not a manual.
|
||||
|
||||
The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
|
||||
|
||||
### docs/policy/architecture.md
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
`docs/policy/architecture.md` is required for every project.
|
||||
|
||||
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
|
||||
|
||||
It should include:
|
||||
|
||||
- project shape;
|
||||
- core design principles;
|
||||
- package and boundary philosophy;
|
||||
- state/persistence philosophy, if applicable;
|
||||
- external integration philosophy, if applicable;
|
||||
- error-handling and logging principles;
|
||||
- testing expectations;
|
||||
- documentation expectations;
|
||||
- architectural invariants;
|
||||
- explicit non-goals, if useful.
|
||||
|
||||
Notably, this file should prescribe a core development *policy* that should remain unchanged as the application evolves. It is not a place for details (e.g., CLI flags) that could change over time.
|
||||
|
||||
The contents of `architecture.md` should be trim and concise. LLMs may be directed to review it routinely via AGENTS.md, CLAUDE.md, or similar.
|
||||
|
||||
### docs/api.md
|
||||
|
||||
**Audience:** external HTTP API consumers, developers, LLM coding agents integrating by HTTP
|
||||
|
||||
Required for projects whose primary public interface is HTTP.
|
||||
|
||||
`docs/api.md` is the canonical public HTTP API contract. It should be normative for external consumers and should not be duplicated by README, operations docs, consumer guides, or integration docs.
|
||||
|
||||
It should include:
|
||||
|
||||
1. base URL conventions;
|
||||
2. authentication and authorization behavior, if implemented;
|
||||
3. response envelope;
|
||||
4. supported media types and content negotiation behavior;
|
||||
5. shared query parameters;
|
||||
6. endpoint reference grouped by route family;
|
||||
7. request parameters and validation rules;
|
||||
8. response fields, units, nullability, and optionality;
|
||||
9. error response shape and status codes;
|
||||
10. pagination, caching, rate-limit, idempotency, and retry behavior, if implemented;
|
||||
11. compact request and response examples.
|
||||
|
||||
It must document only implemented endpoints and behavior. Planned endpoints, proposed fields, future filters, and experimental response shapes belong only under `docs/roadmap/`.
|
||||
|
||||
For HTTP API projects, `docs/consumers/` may provide task-oriented client integration guides, but those guides should link to `docs/api.md` for the authoritative endpoint contract.
|
||||
|
||||
### docs/policy/development.md
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for projects maintained by humans and LLM coding agents.
|
||||
|
||||
It should include:
|
||||
|
||||
- repository layout;
|
||||
- build/test commands;
|
||||
- coding conventions;
|
||||
- dependency policy;
|
||||
- how to add config fields;
|
||||
- how to add CLI flags;
|
||||
- how to add modules or adapters, if applicable;
|
||||
- how to update examples;
|
||||
- documentation update expectations.
|
||||
|
||||
### docs/config.md
|
||||
|
||||
**Audience:** administrators, operators, advanced users
|
||||
|
||||
Required for applications with configuration files.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. config file locations and discovery precedence;
|
||||
2. minimal working config;
|
||||
3. production-oriented config;
|
||||
4. full configuration reference;
|
||||
5. secrets handling, if applicable;
|
||||
6. links to maintained examples.
|
||||
|
||||
The full configuration reference should be canonical.
|
||||
|
||||
### docs/cli.md
|
||||
|
||||
**Audience:** users, administrators, operators
|
||||
|
||||
Required for CLI applications.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. shortest useful command;
|
||||
2. command overview;
|
||||
3. complete flag reference;
|
||||
4. common workflows;
|
||||
5. diagnostic or recovery commands, if applicable.
|
||||
|
||||
Explain when commands are useful, not just their syntax.
|
||||
|
||||
### docs/operations.md
|
||||
|
||||
**Audience:** administrators, operators
|
||||
|
||||
Required for applications that maintain state, support resume behavior, run multi-step workflows, write durable artifacts, use remote storage, or require recovery procedures.
|
||||
|
||||
It should cover:
|
||||
|
||||
- normal workflow;
|
||||
- filesystem layout;
|
||||
- remote storage layout, if applicable;
|
||||
- logs and manifests;
|
||||
- resume/retry behavior;
|
||||
- cleanup behavior;
|
||||
- archive/backup behavior;
|
||||
- safe recovery procedures;
|
||||
- operational caveats.
|
||||
|
||||
### docs/troubleshooting.md
|
||||
|
||||
**Audience:** administrators, operators
|
||||
|
||||
Recommended once recurring failure modes exist.
|
||||
|
||||
Each entry should include:
|
||||
|
||||
- symptom;
|
||||
- likely cause;
|
||||
- diagnostic command or inspection step;
|
||||
- safe fix;
|
||||
- relevant links.
|
||||
|
||||
### docs/consumers/
|
||||
|
||||
**Audience:** developers and LLM coding agents integrating this project from another codebase
|
||||
|
||||
Required for projects with public packages, SDKs, client APIs, plugin APIs, or other application-facing integration surfaces.
|
||||
|
||||
This directory describes how an external codebase should consume the project's public API. It should be task-oriented and copyable where useful. It is not the place for internal implementation details or operator procedures.
|
||||
|
||||
For projects whose public API is HTTP, `docs/consumers/` is not required, and it should not duplicate the endpoint reference in `docs/api.md`. If present, it may provide practical integration workflows, client-specific examples, or migration notes that link back to `docs/api.md`.
|
||||
|
||||
`docs/consumers/api.md` should provide the consumer-facing overview and primary implementation workflow. It should include:
|
||||
|
||||
1. intended consumer audience and use cases;
|
||||
2. required inputs supplied by operators or deployment configuration;
|
||||
3. recommended public package or API workflow;
|
||||
4. minimal copyable example;
|
||||
5. consumer responsibilities and boundaries;
|
||||
6. retry, idempotency, or status behavior, if applicable;
|
||||
7. links to package-specific docs and canonical integration contracts.
|
||||
|
||||
Package-specific docs should be named `pkg-<name>.md` and should include:
|
||||
|
||||
1. import path;
|
||||
2. intended use cases;
|
||||
3. primary types and functions needed by consumers;
|
||||
4. minimal examples;
|
||||
5. validation, error, retry, and boundary behavior;
|
||||
6. links to canonical file-format or wire-protocol contracts.
|
||||
|
||||
### docs/internal/
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for modular, service-oriented, or orchestration projects.
|
||||
|
||||
This directory describes implemented internal components. It is not the roadmap.
|
||||
|
||||
Use one file per major component where useful.
|
||||
|
||||
Each component doc should include:
|
||||
|
||||
1. purpose;
|
||||
2. inputs and outputs;
|
||||
3. boundaries;
|
||||
4. config fields used;
|
||||
5. external adapters used;
|
||||
6. state or manifest behavior, if applicable;
|
||||
7. skip/resume behavior, if applicable;
|
||||
8. failure behavior;
|
||||
9. tests to inspect before changing;
|
||||
10. architectural invariants.
|
||||
|
||||
### docs/roadmap/
|
||||
|
||||
**Audience:** maintainers, developers, LLM coding agents
|
||||
|
||||
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
|
||||
|
||||
Roadmap docs should clearly distinguish:
|
||||
|
||||
- proposed work;
|
||||
- accepted plans;
|
||||
- deferred ideas;
|
||||
- rejected ideas;
|
||||
- implementation prompts or task breakdowns, if useful.
|
||||
|
||||
Roadmap docs should not be confused with current behavior.
|
||||
|
||||
### docs/integrations/
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
|
||||
|
||||
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses or exposes.
|
||||
|
||||
For public HTTP API services, `docs/integrations/` should document upstream, downstream, storage, protocol, or runtime contracts that the service depends on or bridges. It should not become a second copy of the public HTTP endpoint reference; that belongs in `docs/api.md`.
|
||||
|
||||
Use one file per integration where useful.
|
||||
|
||||
## Examples Directory
|
||||
|
||||
Projects with non-trivial configuration or workflows should include `examples/`.
|
||||
|
||||
Useful examples include:
|
||||
|
||||
- minimal working config;
|
||||
- production-oriented config;
|
||||
- full annotated config;
|
||||
- local development config;
|
||||
- remote/object-storage config;
|
||||
- minimal session/input file.
|
||||
|
||||
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
|
||||
|
||||
## Security and Privacy
|
||||
|
||||
Docs and examples must not include:
|
||||
|
||||
- real API keys;
|
||||
- tokens;
|
||||
- passwords;
|
||||
- private keys;
|
||||
- private environment dumps;
|
||||
- sensitive user data;
|
||||
- raw private transcripts;
|
||||
- private infrastructure details unless intentionally public.
|
||||
|
||||
Document secret-handling mechanisms, not actual secret values.
|
||||
|
||||
## Maintenance Rules
|
||||
|
||||
When docs change, verify the affected behavior.
|
||||
|
||||
Where practical:
|
||||
|
||||
- load example config files in tests;
|
||||
- test CLI examples or command parser behavior;
|
||||
- validate documented flags against real flags;
|
||||
- remove stale references;
|
||||
- update links after renames;
|
||||
- keep roadmap content out of non-roadmap docs.
|
||||
|
||||
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
|
||||
|
||||
Documentation is complete only when it matches the current code.
|
||||
|
||||
## Documentation Change Checklist
|
||||
|
||||
Before merging documentation changes, verify:
|
||||
|
||||
- README is concise and orientation-focused.
|
||||
- `docs/policy/architecture.md` describes development principles.
|
||||
- `docs/api.md` is the canonical HTTP contract for HTTP API services.
|
||||
- Future work appears only under `docs/roadmap/`.
|
||||
- User-facing docs avoid unnecessary internals.
|
||||
- Consumer-facing docs explain public APIs without duplicating HTTP endpoint or integration contracts.
|
||||
- Developer-facing docs preserve boundaries and invariants.
|
||||
- Config examples match the schema.
|
||||
- CLI examples match real commands and flags.
|
||||
- Defaults appear in the canonical config reference.
|
||||
- No secrets or private data are included.
|
||||
- Links are accurate.
|
||||
Documentation and examples must not contain real credentials, private keys,
|
||||
private environment dumps, sensitive source material, or private infrastructure
|
||||
details unless intentionally public. Document secret-handling mechanisms, not
|
||||
secret values.
|
||||
|
||||
## Canonical Ownership
|
||||
|
||||
| Topic | Canonical owner | Owned content | Content owned elsewhere |
|
||||
| --- | --- | --- | --- |
|
||||
| Product orientation and minimal end-to-end quickstart | `README.md` | What this project is, why it is useful, one shortest successful invocation, and links onward. | Complete command reference, configuration reference, operational procedures, implementation detail. |
|
||||
| Contributor entry point | `docs/development.md` | Task-oriented reading guide, minimal contributor orientation, baseline validation commands, and links to canonical docs. | Package inventory, architecture rules, subsystem behavior, and detailed change recipes, which belong in the relevant internal component document. |
|
||||
| Current application architecture | `docs/policy/architecture.md` | System shape, normative ownership, dependency direction, architectural boundaries, invariants, safety properties, and non-goals. | Concrete package inventory, implementation mechanics, contributor procedures, decision history, future work. |
|
||||
| Documentation organization | `docs/policy/documentation.md` | Documentation ownership, audience boundaries, maintenance rules, and ADR/document lifecycle. | Application architecture or product behavior. |
|
||||
| Testing policy | `docs/policy/testing.md` | Test philosophy, risk-based sufficiency, test boundaries, doubles, coverage guidance, regression-test policy, and criteria for adding, rewriting, or deleting tests. | Subsystem behavior, application contracts, subsystem-specific test inventories, and implementation plans. |
|
||||
| CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, and exit codes. | End-to-end operating procedures, configuration field definitions, runtime filesystem layout, module implementation details. |
|
||||
| Configuration contract | `docs/config.md` | Discovery and precedence, file schema, fields, defaults, environment overrides, validation rules, and user-selectable module or validator keys. | Complete example files, CLI syntax, runtime state lifecycle, module implementation details. |
|
||||
| Operations | `docs/operations.md` | Runtime workflows, physical filesystem and state layout, output, cache, and debug handling, resume, cleanup, permissions, recovery, and operational limits. | CLI flag syntax, configuration field definitions, logical output schemas, implementation mechanics. |
|
||||
| Public HTTP contract | `docs/api.md` | Routes, authentication, media types, request and response schemas, status codes, pagination, caching, idempotency, rate limits, and HTTP retry semantics. | Client walkthroughs, upstream or downstream integration internals, implementation detail. |
|
||||
| Consumer guidance | `docs/consumers/` | Task-oriented use of the public interface, minimal client examples, and consumer responsibilities. | HTTP wire semantics, external protocol contracts, internal implementation detail. |
|
||||
| External and durable integration contracts | `docs/integrations/` | External file formats and protocols, upstream and downstream contracts, logical output bundle paths and schemas, media types, and compatibility behavior. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, configuration defaults. |
|
||||
| Implemented component inventory | `docs/internal/overview.md` | Current packages and components, their implemented responsibilities, and links to focused internal docs. | Normative architecture, contributor reading policy, external contracts. |
|
||||
| Internal component behavior | Other files under `docs/internal/` | Implementation flow, internal collaborators and state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, configuration definitions and defaults, external schemas, operator procedures. |
|
||||
| Architectural decision history | `docs/adr/` | Significant decisions, context, alternatives, rationale, consequences, and supersession history. | Current behavior reference, implementation status, task sequencing. |
|
||||
| Future work and implementation status | `docs/roadmap/` | Proposed, accepted, deferred, or rejected work; implementation status; sequencing; and task breakdowns. | Implemented behavior reference and architectural decision rationale. |
|
||||
| Complete copyable artifacts | `examples/` | Maintained configuration, inputs, and other files intended to be copied or run. | Field-by-field reference, command reference, prose explanation. |
|
||||
|
||||
Documents that do not exist are required only when the corresponding interface
|
||||
or responsibility exists. Do not create placeholder API, consumer, integration,
|
||||
or operations documents for behavior the application does not have.
|
||||
|
||||
## Boundary Rules
|
||||
|
||||
### Orientation
|
||||
|
||||
The README owns product orientation. The developer guide routes contributors.
|
||||
Architecture owns normative structure. Internal overview owns the current
|
||||
concrete component map. These documents may link to one another but should not
|
||||
maintain parallel package or behavior descriptions.
|
||||
|
||||
### Commands, Configuration, And Operations
|
||||
|
||||
CLI documentation answers how to invoke the application. Configuration
|
||||
documentation answers what settings mean. Operations answers what happens to
|
||||
runtime state and how to operate or recover the application. When a workflow
|
||||
crosses these topics, choose the document that owns the task and link to the
|
||||
other contracts.
|
||||
|
||||
### Contracts And Implementation
|
||||
|
||||
Integration and API documents define externally observable shapes and
|
||||
semantics. Internal documents explain how this project implements or consumes
|
||||
those contracts. Internal docs may name a field, file, or protocol to identify
|
||||
a dependency, but must link to its canonical contract for the definition.
|
||||
|
||||
### Security Topics
|
||||
|
||||
This policy owns what documentation and examples may contain. Architecture owns
|
||||
application security invariants. Configuration owns credential-supply
|
||||
mechanisms. Operations owns permissions and handling of sensitive runtime
|
||||
artifacts. Internal docs own implementation mechanisms only.
|
||||
|
||||
## Architecture Decision Records
|
||||
|
||||
Use sequentially numbered ADR filenames such as
|
||||
`0001-record-architecture-decisions.md`. Follow the lightweight Nygard format:
|
||||
|
||||
1. title;
|
||||
2. status;
|
||||
3. date;
|
||||
4. context;
|
||||
5. decision;
|
||||
6. alternatives considered;
|
||||
7. consequences.
|
||||
|
||||
Use one of these statuses:
|
||||
|
||||
- **Proposed:** the decision is under consideration and may change;
|
||||
- **Accepted:** the decision is approved, whether or not implementation is
|
||||
complete;
|
||||
- **Rejected:** the proposed decision was considered and not adopted;
|
||||
- **Superseded:** a later ADR replaces the accepted decision.
|
||||
|
||||
A proposed ADR transitions to accepted or rejected. An accepted ADR transitions
|
||||
to superseded only when a later accepted ADR replaces it. An ADR may be created
|
||||
as accepted when the decision has already been made.
|
||||
|
||||
Treat the decision content of an accepted ADR as immutable. Its status and
|
||||
supersession metadata may be updated, but a changed decision requires a new ADR.
|
||||
A superseded ADR must link to its replacement, and the replacement must link
|
||||
back to the superseded ADR. Rejected architectural alternatives belong in the
|
||||
ADR; rejected product ideas belong in the roadmap.
|
||||
|
||||
## Maintenance
|
||||
|
||||
When behavior changes, update its canonical owner in the same change. If
|
||||
ownership moves, remove the old definition and replace it with a link where
|
||||
navigation remains useful.
|
||||
|
||||
Before completing documentation work:
|
||||
|
||||
- verify affected behavior and examples;
|
||||
- check commands, flags, fields, defaults, schemas, and paths against their
|
||||
implementation;
|
||||
- keep unimplemented behavior in the roadmap, subject to the ADR exception;
|
||||
- remove stale references and validate links;
|
||||
- confirm that non-owning documents summarize and link rather than redefine;
|
||||
- confirm that no secrets or sensitive private data were added.
|
||||
|
||||
296
docs/policy/testing.md
Normal file
296
docs/policy/testing.md
Normal file
@@ -0,0 +1,296 @@
|
||||
# Testing Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
Our tests exist to make **incorrect changes expensive and correct changes cheap**.
|
||||
|
||||
We do not optimize for test count, line coverage, exhaustive isolation, or the fewest possible tests. We optimize for sufficient confidence in important behavior while imposing as little unnecessary friction as possible on future development.
|
||||
|
||||
## Every test has a cost
|
||||
|
||||
Testing is not an unqualified good. Every test imposes both an immediate cost and a continuing lifetime cost.
|
||||
|
||||
A test must be:
|
||||
|
||||
- written and reviewed;
|
||||
- understood by future maintainers and coding agents;
|
||||
- executed in local and CI workflows;
|
||||
- diagnosed when it fails;
|
||||
- updated when legitimate behavior changes;
|
||||
- maintained as fixtures, APIs, and dependencies evolve; and
|
||||
- removed or rewritten when it becomes redundant, brittle, misleading, or obsolete.
|
||||
|
||||
Tests also create cognitive and architectural friction. They can constrain refactoring, duplicate policy, slow feedback loops, add noise to failures, and cause harmless implementation changes to require unrelated edits across the suite.
|
||||
|
||||
A test is warranted only when the confidence it provides justifies these costs.
|
||||
|
||||
Apply this cost-benefit analysis at two levels:
|
||||
|
||||
1. **Per test:** What realistic defect does this test detect, how consequential would that defect be, and is that protection worth the test's lifetime cost?
|
||||
2. **Across the suite:** Does this collection provide materially more confidence than a smaller, simpler suite would?
|
||||
|
||||
The preferred test suite is a **lean suite that provides sufficient confidence in the risks that matter, without redundant or low-value tests**. We seek sufficient confidence with the least unnecessary testing friction, not the fewest possible tests.
|
||||
|
||||
Some friction is intentional. Tests should make dangerous changes—such as breaking compatibility, corrupting data, violating security boundaries, or reintroducing subtle bugs—require deliberate review. They should not make ordinary internal changes needlessly expensive.
|
||||
|
||||
The cost of a test is not a reason to omit testing by default. Do not cite maintenance cost abstractly. When omitting a plausible test, be able to state why the protected failure is low-risk, already covered, obvious, reversible, or cheaper to detect elsewhere. For consequential, subtle, or difficult-to-observe behavior, the presumption should favor testing.
|
||||
|
||||
## Default testing style
|
||||
|
||||
Use a **classical/Detroit-style** approach:
|
||||
|
||||
- Test observable behavior, resulting state, contracts, and invariants.
|
||||
- Use real internal collaborators when they are fast and deterministic.
|
||||
- Use fakes, stubs, or mocks primarily at expensive, nondeterministic, destructive, or external boundaries.
|
||||
- Prefer package-level behavioral tests over tests coupled to private helpers or internal call sequences.
|
||||
- Treat exact collaborator interactions as testable behavior only when the interaction itself is a requirement.
|
||||
|
||||
Examples of appropriate seams include clocks, randomness, subprocesses, remote APIs, object storage, email, and paid LLM calls.
|
||||
|
||||
## Test execution requirements
|
||||
|
||||
Tests in the default suite must be deterministic, offline, and independent of real credentials. They must not invoke paid APIs or depend on mutable external services. Tests that require live infrastructure must be explicitly opt-in and clearly separated from the default suite.
|
||||
|
||||
Control clocks, randomness, environment variables, and other process-global or machine-specific state when they affect behavior. Tests should be safe to run repeatedly and alongside other tests without depending on execution order or state left by an earlier test.
|
||||
|
||||
## What deserves tests
|
||||
|
||||
Prioritize tests for:
|
||||
|
||||
1. Public and package-level contracts.
|
||||
2. Domain rules and important invariants.
|
||||
3. Boundary conditions and malformed input.
|
||||
4. Failure handling, cancellation, retries, recovery, and partial success.
|
||||
5. Serialization, schemas, compatibility, and round trips.
|
||||
6. Previously observed or plausible regressions.
|
||||
7. Representative integration and end-to-end workflows.
|
||||
|
||||
A package-level contract is behavior relied upon by another package or major collaborator, not every observable detail of a package implementation.
|
||||
|
||||
For behavior involving **data integrity, destructive operations, compatibility, security, concurrency, idempotency, or recovery**, presume that durable tests are required unless the behavior is already credibly protected at another layer.
|
||||
|
||||
Do not add tests merely because a function, branch, or line exists. Do not add a test when the same meaningful risk is already adequately protected elsewhere.
|
||||
|
||||
## Choose the right test boundary
|
||||
|
||||
Test through the narrowest stable boundary that expresses the behavior clearly.
|
||||
|
||||
This is often the package API, but it may instead be:
|
||||
|
||||
- a smaller pure function when dense domain logic is most clearly isolated there;
|
||||
- a package-level operation when several internal collaborators jointly produce the behavior; or
|
||||
- a larger integration boundary when correctness emerges from interaction with a real dependency.
|
||||
|
||||
Do not force all behavior through oversized end-to-end tests. Do not test every private helper merely because it exists. Choose the boundary that gives durable confidence with the least incidental coupling.
|
||||
|
||||
## Test behavior, not implementation
|
||||
|
||||
A test should protect a decision, contract, or invariant—not memorialize the current implementation.
|
||||
|
||||
Before adding or retaining a test, ask:
|
||||
|
||||
> What realistic defect would this test catch?
|
||||
|
||||
A test is suspect when its main purpose is to detect that someone:
|
||||
|
||||
- changed an internal constant;
|
||||
- renamed or split a private helper;
|
||||
- reordered equivalent internal operations;
|
||||
- changed incidental formatting;
|
||||
- replaced one correct algorithm with another; or
|
||||
- refactored internal object structure without changing behavior.
|
||||
|
||||
Refactoring should normally require no test edits unless the refactored structure is itself part of the contract.
|
||||
|
||||
A test can be factually correct and still have negative value. Accurately describing current behavior is not enough; the protected behavior must be important enough to justify the future friction.
|
||||
|
||||
## Expected effects of different changes
|
||||
|
||||
Use the following expectations when evaluating test failures and test maintenance:
|
||||
|
||||
| Change | Expected effect on tests |
|
||||
|---|---|
|
||||
| Internal refactor that preserves behavior | Existing tests should normally remain unchanged and continue to pass. |
|
||||
| Change to an internal default with no contractual significance | Behavioral tests should normally remain unchanged; tests should derive expectations from configuration or relationships rather than duplicate the old value. |
|
||||
| Intentional change to public behavior, policy, schema, or compatibility guarantees | The relevant tests should be reviewed and changed deliberately. |
|
||||
| Accidental violation of a contract or invariant | Tests should fail; fix the production code rather than rewriting the tests to accept the defect. |
|
||||
|
||||
A test failing is not the same as a test needing to be edited. Many tests may correctly fail because of one production defect. The maintenance smell is a correct internal change that requires unrelated expectation updates throughout the suite.
|
||||
|
||||
## Separate mechanism from policy
|
||||
|
||||
Configurable thresholds and defaults must not be duplicated throughout the test suite.
|
||||
|
||||
For example, do not encode an internal concurrency limit indirectly:
|
||||
|
||||
```go
|
||||
// Production policy:
|
||||
const maxConcurrency = 4
|
||||
|
||||
// Brittle test:
|
||||
err := startProcesses(5)
|
||||
require.Error(t, err)
|
||||
```
|
||||
|
||||
Instead, test the mechanism relationally:
|
||||
|
||||
```go
|
||||
const limit = 2
|
||||
runner := NewRunner(limit)
|
||||
|
||||
require.NoError(t, runner.Start(limit))
|
||||
require.ErrorIs(t, runner.Start(limit+1), ErrTooMuchConcurrency)
|
||||
```
|
||||
|
||||
The test should prove:
|
||||
|
||||
- the configured limit is accepted; and
|
||||
- one beyond the configured limit is rejected.
|
||||
|
||||
The production default should be tested exactly only when its literal value is itself a public, operational, safety, protocol, or compatibility requirement.
|
||||
|
||||
Apply the same rule to limits, timeouts, capacities, retry counts, and ranges: test relationships and behavior, not duplicated literals.
|
||||
|
||||
For concurrency limits, test both kinds of behavior when relevant:
|
||||
|
||||
1. **Configuration enforcement:** invalid or excessive requested values are handled correctly.
|
||||
2. **Runtime enforcement:** observed peak concurrency never exceeds the configured limit.
|
||||
|
||||
Use a test-controlled limit and measure the behavior relative to that limit. Do not merely assert today's default value.
|
||||
|
||||
## Avoid semantic duplication across layers
|
||||
|
||||
Each behavior should have a clear test owner.
|
||||
|
||||
- Parser tests own parsing cases.
|
||||
- Validator tests own validation rules.
|
||||
- Domain tests own transformations and invariants.
|
||||
- Adapter tests own external integration behavior.
|
||||
- Orchestrator tests own coordination and failure propagation.
|
||||
- CLI tests own argument and configuration mapping.
|
||||
- End-to-end tests prove that representative assembled workflows work.
|
||||
|
||||
Higher-level tests should not repeat every lower-level case. A single intentional policy change should not require unrelated edits across many test files.
|
||||
|
||||
Tests that are individually reasonable may still be collectively redundant. Evaluate the marginal value of each additional test in light of the protection already provided by the rest of the suite.
|
||||
|
||||
## Use test doubles deliberately
|
||||
|
||||
Choose the least elaborate test double that provides the required control or observation.
|
||||
|
||||
As a default:
|
||||
|
||||
1. Prefer real collaborators when they are fast and deterministic.
|
||||
2. Use small in-memory fakes when realistic stateful behavior is helpful.
|
||||
3. Use stubs when a dependency only needs to provide controlled responses.
|
||||
4. Use mocks when the interaction itself is contractual.
|
||||
|
||||
Mocks are appropriate when the contract includes facts such as:
|
||||
|
||||
- a notification is sent exactly once;
|
||||
- a transaction is committed only after successful writes;
|
||||
- cancellation reaches a subprocess;
|
||||
- an expensive API is called no more than once; or
|
||||
- a security audit event is emitted.
|
||||
|
||||
Do not use mocks merely to isolate every object or reproduce the implementation's call graph.
|
||||
|
||||
## Go-specific guidance
|
||||
|
||||
Use:
|
||||
|
||||
- table-driven tests for meaningful behavioral categories and boundaries;
|
||||
- `t.TempDir()` for real filesystem behavior;
|
||||
- `httptest.Server` for realistic HTTP interactions;
|
||||
- fuzz tests for parsers, normalization, path handling, and broad input spaces;
|
||||
- golden files only when the complete output is intentionally stable;
|
||||
- integration tests where correctness depends on component interaction; and
|
||||
- a small number of representative end-to-end tests.
|
||||
|
||||
Avoid exact error-string assertions unless the wording is itself contractual. Prefer `errors.Is`, `errors.As`, typed errors, or structured error fields.
|
||||
|
||||
At CLI boundaries, prefer exit classifications, structured output, and the smallest stable semantic fragment needed to identify the error. Do not snapshot complete diagnostic wording unless it is contractual.
|
||||
|
||||
Golden-file updates must require an explicit local flag. CI must not update golden files automatically, and reviewers must inspect the semantic diff before accepting an update.
|
||||
|
||||
Keep tests readable and direct. Test helpers and fixture frameworks must earn their own maintenance cost; do not build elaborate test infrastructure for small or isolated needs.
|
||||
|
||||
## Coverage
|
||||
|
||||
Coverage is a diagnostic, not a target.
|
||||
|
||||
Use it to find untested critical branches and unexpectedly weak packages. Do not write low-value tests solely to increase a percentage, and do not infer test quality from coverage alone.
|
||||
|
||||
Pure domain logic will often warrant higher coverage than CLI wiring or external adapters. Uneven coverage is acceptable when it reflects risk.
|
||||
|
||||
Increasing coverage is valuable only when the newly covered behavior protects a meaningful risk at an acceptable cost.
|
||||
|
||||
## Regression tests
|
||||
|
||||
A bug fix should normally include a regression test that fails before the fix and passes afterward.
|
||||
|
||||
Retain the test when the defect could realistically recur and its consequences justify the ongoing cost. Prefer the narrowest durable test of the violated contract or invariant; do not preserve accidental implementation details from the original bug.
|
||||
|
||||
Not every historical bug requires a permanent test. If the underlying design has made recurrence impossible, the test has become redundant, or a stronger invariant test now subsumes it, remove or consolidate it.
|
||||
|
||||
## Deleting or rewriting tests
|
||||
|
||||
Tests are maintained code, not permanent historical artifacts.
|
||||
|
||||
Delete or rewrite a test when its maintenance cost exceeds the confidence it provides.
|
||||
|
||||
Strong candidates include tests that:
|
||||
|
||||
- require updates after harmless internal changes;
|
||||
- directly assert private constants without protecting a real contract;
|
||||
- duplicate the same policy across several layers;
|
||||
- verify mock choreography rather than outcomes;
|
||||
- snapshot large amounts of incidental output;
|
||||
- test trivial private helpers already exercised through stable package behavior;
|
||||
- protect risks already covered more effectively elsewhere;
|
||||
- are flaky, misleading, obsolete, or disproportionately expensive to diagnose; or
|
||||
- no longer correspond to a plausible failure mode.
|
||||
|
||||
Several brittle tests may encode one genuine requirement. Replace them with one durable behavior-level or invariant test rather than preserving all of them.
|
||||
|
||||
Deleting a low-value test can improve the quality of the suite by reducing noise, maintenance burden, and friction around legitimate change.
|
||||
|
||||
## Reviewing a proposed test
|
||||
|
||||
Use the following questions when the value, boundary, or durability of a proposed test is not self-evident. Significant test additions should be reviewable against them, but written answers are not required for every routine test.
|
||||
|
||||
1. What realistic defect would it catch?
|
||||
2. How likely is that defect?
|
||||
3. How consequential would it be?
|
||||
4. Is the behavior already protected elsewhere?
|
||||
5. At which layer should this behavior be owned?
|
||||
6. Does the test assert a durable contract or an incidental implementation detail?
|
||||
7. Could the implementation be refactored without changing the behavior and without editing this test?
|
||||
8. What should cause this test to fail?
|
||||
9. What legitimate changes should not cause this test to fail?
|
||||
10. What ongoing maintenance, execution, and diagnostic cost will the test impose?
|
||||
11. Is there a smaller or more direct test that protects the same risk?
|
||||
|
||||
Do not add the test when its expected lifetime cost exceeds its expected protective value.
|
||||
|
||||
When deciding not to test plausible behavior, record or be able to explain why the risk is low, already protected, obvious, reversible, or cheaper to detect elsewhere.
|
||||
|
||||
## Definition of sufficient
|
||||
|
||||
A test suite is sufficient when:
|
||||
|
||||
- important contracts and invariants are protected;
|
||||
- meaningful boundaries and failure modes are exercised;
|
||||
- realistic and consequential regressions are credibly protected against silent recurrence;
|
||||
- behavior involving data integrity, destructive operations, compatibility, security, concurrency, idempotency, and recovery is credibly protected;
|
||||
- important external boundaries have realistic integration coverage;
|
||||
- representative complete workflows are tested;
|
||||
- failures provide useful signal rather than redundant noise;
|
||||
- legitimate internal changes usually do not require test edits; and
|
||||
- additional tests would mostly repeat existing protection or preserve inconsequential implementation details.
|
||||
|
||||
Sufficiency is a risk judgment, not a coverage percentage or test count. Reassess it as the application, its users, and the consequences of failure evolve.
|
||||
|
||||
The governing rule is:
|
||||
|
||||
> Test heavily where failure is consequential, subtle, or difficult to detect after the fact. Test lightly where failure is obvious, reversible, and inexpensive—and retain no test whose lifetime cost exceeds the confidence it provides.
|
||||
510
docs/roadmap/documentation.md
Normal file
510
docs/roadmap/documentation.md
Normal file
@@ -0,0 +1,510 @@
|
||||
# Documentation Compliance Roadmap
|
||||
|
||||
## Status
|
||||
|
||||
Proposed implementation plan. This document records the findings of the
|
||||
documentation audit performed after adoption of the canonical-ownership policy.
|
||||
The revisions described here are not yet implemented.
|
||||
|
||||
## Objective
|
||||
|
||||
Bring the current Scriptorium documentation into compliance with
|
||||
`docs/policy/documentation.md` before beginning the Promptkit migration.
|
||||
|
||||
The refresh should:
|
||||
|
||||
- give every topic one canonical owner;
|
||||
- remove parallel definitions of volatile contracts;
|
||||
- correct current factual discrepancies;
|
||||
- preserve useful contributor and operational guidance in the appropriate
|
||||
documents;
|
||||
- establish the missing internal overview and ADR history;
|
||||
- leave Promptkit and other future behavior in roadmap and ADR documents until
|
||||
implemented.
|
||||
|
||||
This roadmap covers documentation organization and current-behavior accuracy.
|
||||
A substantive review of `docs/policy/testing.md` remains separate.
|
||||
|
||||
## Audit Baseline
|
||||
|
||||
The audit covered:
|
||||
|
||||
- `README.md`;
|
||||
- all Markdown files under `docs/`;
|
||||
- all maintained files under `examples/`;
|
||||
- the CLI parser and output behavior;
|
||||
- app-config shapes and defaults;
|
||||
- HTTP DTOs, limits, and error mappings;
|
||||
- the public Go facade;
|
||||
- prompt, profile, schema, artifact, renderer, runner, and LLM behavior;
|
||||
- the embedded built-in profile assets.
|
||||
|
||||
At the time of the audit:
|
||||
|
||||
- all local Markdown link targets existed;
|
||||
- `go test ./...` passed;
|
||||
- the README/CLI render command completed successfully;
|
||||
- `examples/render-markdown-summary.sh` completed successfully;
|
||||
- `go run ./examples/go-library/prepare` completed successfully.
|
||||
|
||||
The dominant problem is duplicated ownership rather than broad factual
|
||||
staleness. CLI, configuration, HTTP, validation, state, and security behavior
|
||||
are repeated across contracts, operations, troubleshooting, integrations,
|
||||
consumer guides, architecture, and internal documents.
|
||||
|
||||
## Required Accuracy Corrections
|
||||
|
||||
Make these corrections while revising the owning documents:
|
||||
|
||||
1. `docs/policy/architecture.md` says Scriptorium has three entry paths but
|
||||
lists four. Distinguish the three executable entry paths from the public Go
|
||||
package, or describe four total entry paths.
|
||||
2. The built-in profile catalog in `docs/config.md` omits
|
||||
`deepseek-4-flash`. Reconcile the complete catalog with
|
||||
`internal/profile/builtin/assets/`.
|
||||
3. `docs/config.md` marks prompt `output.repair_attempts` as required. The
|
||||
loader permits omission and resolves it to zero; document it as optional
|
||||
with an effective default of zero.
|
||||
4. `docs/consumers/pkg-scriptorium.md` says every input map key must match a
|
||||
declared prompt input. The renderer enforces required declared inputs and
|
||||
resolves names actually referenced by templates, but it does not reject
|
||||
every undeclared extra input. Describe the implemented boundary.
|
||||
5. `docs/troubleshooting.md` says validation errors can be inspected in CLI
|
||||
stderr. The CLI success summary reports the number of validation errors, not
|
||||
their detailed messages. Correct the diagnostic guidance or separately
|
||||
change the product before documenting richer output.
|
||||
6. `docs/integrations/openai-compatible-chat.md` describes the `session_id`
|
||||
limit as characters. The implementation counts Unicode code points; use the
|
||||
canonical terminology consistently.
|
||||
7. The same integration document makes a time-sensitive statement about
|
||||
provider-supported service-tier values even though Scriptorium accepts and
|
||||
forwards any non-empty value. Remove the provider catalog claim or cite and
|
||||
version an intentionally maintained external contract.
|
||||
8. The `render` and `serve` flag sections in `docs/cli.md` list
|
||||
`--prompt-dir` both as an effective requirement and again as an optional
|
||||
flag. Keep one complete flag entry and separately explain how the
|
||||
requirement may be satisfied.
|
||||
9. Troubleshooting command examples containing placeholders such as
|
||||
`<prompt-id>` are not directly copyable shell commands. Use clearly defined
|
||||
shell variables, concrete maintained examples, or prose diagnostic steps.
|
||||
|
||||
## Canonical Structure Changes
|
||||
|
||||
### Create `docs/internal/overview.md`
|
||||
|
||||
Create the canonical implemented-component inventory and move the concrete
|
||||
package map out of architecture.
|
||||
|
||||
The overview should:
|
||||
|
||||
- list current public, command, adapter, domain, use-case, source, format,
|
||||
validation, and LLM components;
|
||||
- give each component a short implemented responsibility;
|
||||
- link to focused internal documents and relevant external contracts;
|
||||
- avoid restating global architecture rules or user-facing behavior.
|
||||
|
||||
Update `docs/development.md` to route general repository-orientation work
|
||||
through the overview once it exists.
|
||||
|
||||
### Establish `docs/adr/`
|
||||
|
||||
Create the ADR directory under the policy already defined in
|
||||
`docs/policy/documentation.md`.
|
||||
|
||||
Record the significant accepted documentation-ownership decision in an initial
|
||||
ADR if historical rationale is useful. The Promptkit split ADR remains part of
|
||||
Step 2 in `docs/roadmap/migration.md` and should not be pulled ahead of its
|
||||
gate merely to populate the directory.
|
||||
|
||||
ADRs must own decision context, alternatives, rationale, and consequences.
|
||||
Roadmaps must continue to own implementation status and sequencing.
|
||||
|
||||
### Consolidate Troubleshooting Into Operations
|
||||
|
||||
The ownership table assigns recovery to `docs/operations.md` and does not give
|
||||
troubleshooting a separate canonical owner. Fold the useful
|
||||
symptom/diagnosis/safe-recovery material from `docs/troubleshooting.md` into a
|
||||
concise operations section, then remove `docs/troubleshooting.md`.
|
||||
|
||||
The consolidated material should:
|
||||
|
||||
- organize failures by operational task rather than duplicate every contract
|
||||
field and status;
|
||||
- link exact CLI syntax and exit codes to `docs/cli.md`;
|
||||
- link config fields and defaults to `docs/config.md`;
|
||||
- link HTTP codes and schemas to `docs/api.md`;
|
||||
- describe only diagnostic and recovery actions locally.
|
||||
|
||||
Update README, development, integration, and other links after the removal.
|
||||
|
||||
## File-By-File Revision Catalog
|
||||
|
||||
### `README.md`
|
||||
|
||||
- Retain the product description and one tested end-to-end quickstart.
|
||||
- Keep the quickstart linked to the complete CLI contract.
|
||||
- Replace the repeated example inventory with a short link to the maintained
|
||||
examples section in the appropriate canonical contract, or make each retained
|
||||
path a direct link.
|
||||
- Keep the documentation list navigational; do not summarize contracts there.
|
||||
- Remove the troubleshooting link if troubleshooting is consolidated into
|
||||
operations.
|
||||
|
||||
### `docs/development.md`
|
||||
|
||||
- Retain only contributor orientation, task-specific routing, and baseline
|
||||
validation.
|
||||
- Add `docs/internal/overview.md` to the appropriate reading paths once it
|
||||
exists.
|
||||
- Update the operations/troubleshooting row after consolidation.
|
||||
- Link to actual ADRs when created; do not duplicate their decisions.
|
||||
- Confirm that all detailed recipes removed from the former development guide
|
||||
have an internal owner before declaring the refresh complete.
|
||||
|
||||
### `docs/policy/architecture.md`
|
||||
|
||||
- Fix the entry-path count.
|
||||
- Replace the concrete package inventory with a link to
|
||||
`docs/internal/overview.md`.
|
||||
- Remove config precedence, field-level behavior, exact routes, and exact
|
||||
import-path contracts except for the smallest orientation summary and links
|
||||
to their owners.
|
||||
- Remove the local testing checklist and link to
|
||||
`docs/policy/testing.md`. Component-specific test inventories remain in
|
||||
internal docs.
|
||||
- Remove the local documentation checklist and link to
|
||||
`docs/policy/documentation.md`.
|
||||
- Keep normative system shape, ownership, dependency direction, state
|
||||
philosophy, security properties, error-handling principles, invariants, and
|
||||
non-goals.
|
||||
- Review the coding and dependency rules removed from the former development
|
||||
guide. Preserve still-valid normative rules here without listing the current
|
||||
dependency inventory, which is concrete implementation information.
|
||||
- Move implementation facts such as current repairer wiring to the relevant
|
||||
internal component document unless they are intentionally elevated to
|
||||
architecture invariants.
|
||||
|
||||
### `docs/policy/documentation.md`
|
||||
|
||||
- No structural rewrite is currently required.
|
||||
- Recheck its ownership table after the troubleshooting consolidation and ADR
|
||||
creation.
|
||||
- Keep future policy refinements in this document rather than distributing
|
||||
documentation rules across architecture or development.
|
||||
|
||||
### `docs/policy/testing.md`
|
||||
|
||||
- It is the sole owner of global testing philosophy and sufficiency rules.
|
||||
- Remove competing global testing guidance from architecture and other docs.
|
||||
- Leave the planned substantive testing-policy review for its separate work
|
||||
cycle.
|
||||
|
||||
### `docs/cli.md`
|
||||
|
||||
- Keep the complete command, argument, flag, output, and exit-code contract.
|
||||
- Remove the repeated definition of config precedence and link to
|
||||
`docs/config.md`.
|
||||
- Move profile source and override semantics that are not CLI invocation
|
||||
semantics to `docs/config.md`.
|
||||
- Resolve duplicate `--prompt-dir` entries in the `render` and `serve`
|
||||
sections.
|
||||
- Verify every registered flag, deprecated alias, requirement, zero-value
|
||||
behavior, and output destination against `internal/adapter/cli/run.go`.
|
||||
- Clarify timeout conversion for subsecond Go durations if that behavior is
|
||||
intended to remain public.
|
||||
- Keep only compact workflow examples and link to maintained scripts under
|
||||
`examples/`.
|
||||
|
||||
### `docs/config.md`
|
||||
|
||||
- Keep app-config discovery, precedence, fields, defaults, validation,
|
||||
credential-supply mechanisms, prompt/profile formats, and selectable profile
|
||||
catalog as the canonical contract.
|
||||
- Replace complete inline config, prompt, and profile files with the smallest
|
||||
useful snippets and links to copyable files under `examples/`.
|
||||
- If a production-oriented complete config remains useful, move it into
|
||||
`examples/` and link it rather than maintaining a second complete copy.
|
||||
- Correct `repair_attempts` optionality.
|
||||
- Add `deepseek-4-flash` and reconcile every built-in profile ID, model, and
|
||||
credential-variable name with embedded assets.
|
||||
- Establish a low-friction way to prevent catalog drift, such as a generated
|
||||
catalog section or a focused consistency check, if its maintenance value
|
||||
justifies the cost.
|
||||
- Keep schema configuration and prompt/profile file-format rules here; move
|
||||
implementation mechanics to internal source docs.
|
||||
- Reduce the HTTP artifact-reference section to configuration meaning and link
|
||||
request shapes/statuses to `docs/api.md` and runtime security handling to
|
||||
`docs/operations.md`.
|
||||
- Keep the secret-supply mechanism here; move deployment permissions and
|
||||
sensitive-runtime handling to operations.
|
||||
|
||||
### `docs/api.md`
|
||||
|
||||
- Keep routes, media types, request/response schemas, strict JSON behavior,
|
||||
status codes, HTTP retry semantics, and HTTP artifact-access outcomes.
|
||||
- Replace repeated app-config defaults and CLI flag syntax with links to
|
||||
`docs/config.md` and `docs/cli.md`.
|
||||
- Retain HTTP limit effects but let configuration own field defaults and
|
||||
precedence.
|
||||
- Keep the lexical containment and symlink behavior as externally observable
|
||||
API/security behavior; operations may summarize its deployment consequence
|
||||
and link back.
|
||||
- Reduce request and response examples to compact contract-bearing shapes and
|
||||
link to `examples/http-run.json`.
|
||||
- Reverify all DTO fields, omission behavior, limit responses, and error mappings
|
||||
against `internal/adapter/http/`.
|
||||
|
||||
### `docs/operations.md`
|
||||
|
||||
- Rewrite as a task-oriented runbook rather than a secondary CLI, config, and
|
||||
API reference.
|
||||
- Keep deployment layout, process permissions, sensitive runtime artifact
|
||||
handling, normal workflow, service exposure, capacity planning, recovery,
|
||||
and the stateless rerun model.
|
||||
- Link exact commands and exit codes to `docs/cli.md`.
|
||||
- Link config fields, defaults, validation modes, and credential-supply
|
||||
mechanics to `docs/config.md`.
|
||||
- Link HTTP route, status, schema, and limit semantics to `docs/api.md`.
|
||||
- Remove the repeated maintained-example inventory and link to the owning
|
||||
examples section.
|
||||
- Incorporate the useful troubleshooting material and then delete
|
||||
`docs/troubleshooting.md`.
|
||||
|
||||
### `docs/troubleshooting.md`
|
||||
|
||||
- Correct the CLI validation-error diagnostic during migration.
|
||||
- Move useful symptom, diagnostic, and safe-fix material into
|
||||
`docs/operations.md`.
|
||||
- Remove repeated definitions of flags, fields, defaults, HTTP codes, routes,
|
||||
and validation semantics; link to their canonical contracts.
|
||||
- Delete this file after all useful recovery guidance and inbound links have
|
||||
been handled.
|
||||
|
||||
### `docs/consumers/api.md`
|
||||
|
||||
- Keep integration-surface selection, minimal consumer workflow, and consumer
|
||||
responsibilities.
|
||||
- Retain one minimal Go example as permitted instructional content and link to
|
||||
the complete package contract and maintained example.
|
||||
- Replace repeated CLI exit and HTTP status definitions with links to their
|
||||
canonical contracts.
|
||||
- Replace repeated deployment input and credential definitions with links to
|
||||
configuration and operations.
|
||||
- Keep retry and artifact-retention decisions as consumer responsibilities
|
||||
without restating wire semantics.
|
||||
|
||||
### `docs/consumers/pkg-scriptorium.md`
|
||||
|
||||
- Keep the import path, public constructors, options, types, errors, workflows,
|
||||
and public security boundary as the canonical Go package contract.
|
||||
- Correct the input-name validation statement.
|
||||
- Reverify public fields, JSON behavior, option precedence, nil behavior,
|
||||
source containment, direct-key handling, and sentinel errors against the root
|
||||
package.
|
||||
- Keep a minimal package example and link to
|
||||
`examples/go-library/prepare`; avoid duplicating complete maintained source.
|
||||
- Link prompt/profile/schema file formats to `docs/config.md` rather than
|
||||
redefining them.
|
||||
|
||||
### `docs/integrations/subprocess.md`
|
||||
|
||||
- Narrow this document to subprocess-specific compatibility and process
|
||||
integration concerns.
|
||||
- Link command syntax, flags, stdout/stderr behavior, and exit codes to
|
||||
`docs/cli.md` rather than maintaining parallel definitions.
|
||||
- Link config search and field semantics to `docs/config.md`.
|
||||
- Retain process isolation, environment propagation, stream capture, output
|
||||
ownership, cancellation/termination expectations, and security guidance when
|
||||
these are specific to subprocess consumers.
|
||||
- Move task-oriented surface-selection advice to `docs/consumers/api.md` if it
|
||||
is currently duplicated.
|
||||
|
||||
### `docs/integrations/openai-compatible-chat.md`
|
||||
|
||||
- Keep only the outbound HTTP wire contract: endpoint construction, request
|
||||
payload, authentication header, timeouts, response subset, and unsupported
|
||||
protocol behavior.
|
||||
- Remove the concrete implementation-file introduction and internal error
|
||||
sentinel catalog.
|
||||
- Correct the `session_id` unit to Unicode code points.
|
||||
- Remove or deliberately source/version the provider service-tier claim.
|
||||
- Reverify reserved `extra_params` keys, structured-output envelope,
|
||||
cache-control encoding, usage mapping, and response requirements against
|
||||
`internal/llm/openai_compatible_client.go`.
|
||||
- Link runner schema preparation to internal runner documentation rather than
|
||||
explaining orchestration locally.
|
||||
|
||||
### `docs/internal/overview.md` (new)
|
||||
|
||||
- Own the complete current component and package inventory.
|
||||
- Link each component to focused internal docs and external contracts.
|
||||
- Include the command entry point and public facade without redefining their
|
||||
external behavior.
|
||||
- Absorb the still-useful concrete repository-layout material removed from the
|
||||
old development guide and architecture policy.
|
||||
|
||||
### `docs/internal/adapters.md`
|
||||
|
||||
- Keep implementation flow, collaborators, translation boundaries, wiring,
|
||||
error-mapping mechanisms, and relevant tests.
|
||||
- Remove exact flag lists, config field definitions, route schemas, response
|
||||
codes, and exit-code definitions; link their canonical owners.
|
||||
- Keep only package-local invariants rather than repeating global architecture.
|
||||
- Incorporate the former contributor recipes for adding app-config fields, CLI
|
||||
flags, and adapter capabilities.
|
||||
- Describe how to verify affected adapter contracts without duplicating the
|
||||
global testing policy.
|
||||
|
||||
### `docs/internal/runner.md`
|
||||
|
||||
- Keep prepare/run flow, dependency boundaries, state transitions, failure
|
||||
categories, repair mechanics, hashing/validation coordination, and relevant
|
||||
tests.
|
||||
- Remove the app-config field list and link to adapter/config documentation.
|
||||
- Keep internal sentinel and collaborator information, but link public error
|
||||
behavior to the package/API contracts.
|
||||
- Distinguish package-local guarantees from global architectural invariants.
|
||||
- Add a focused change recipe for runner orchestration when useful.
|
||||
|
||||
### `docs/internal/sources.md`
|
||||
|
||||
- Keep loader implementations, source precedence mechanics, containment
|
||||
implementation, failure categories, collaborators, and relevant tests.
|
||||
- Link YAML/JSON field definitions and defaults to `docs/config.md`.
|
||||
- Link HTTP-visible artifact outcomes to `docs/api.md` and deployment handling
|
||||
to operations.
|
||||
- Incorporate the former recipe for updating prompt, profile, schema, and
|
||||
built-in-profile assets.
|
||||
- Distinguish implementation mechanics from external source contracts.
|
||||
|
||||
### `docs/internal/llm.md` (new)
|
||||
|
||||
Create a focused internal document if the implementation material removed from
|
||||
the outbound integration contract remains useful.
|
||||
|
||||
It should own:
|
||||
|
||||
- client construction and internal collaborator boundaries;
|
||||
- internal request mapping and timeout selection;
|
||||
- internal error categories;
|
||||
- relevant tests and package-local change guidance.
|
||||
|
||||
It must link to `docs/integrations/openai-compatible-chat.md` for the wire
|
||||
contract rather than repeat payload definitions.
|
||||
|
||||
### `docs/roadmap/migration.md`
|
||||
|
||||
- Retain migration status, gates, sequencing, and task breakdowns.
|
||||
- Mark the documentation-refresh gate complete only after this roadmap's
|
||||
completion criteria are satisfied.
|
||||
- After the Promptkit split ADR is accepted, replace duplicated decision
|
||||
rationale and architectural ownership detail with a concise summary and ADR
|
||||
link where practical.
|
||||
- Update paths and document ownership affected by this refresh.
|
||||
|
||||
### `examples/`
|
||||
|
||||
- Keep complete copyable artifacts here rather than in reference documents.
|
||||
- Verify `config.yml` and `config.full.yml` through the real config loader.
|
||||
- Keep prompt/profile/schema examples covered by representative repository and
|
||||
engine tests.
|
||||
- Keep the render script and Go package example runnable from the repository
|
||||
root.
|
||||
- Validate `http-run.json` structurally against the HTTP DTO contract without
|
||||
requiring a live model endpoint.
|
||||
- Add a separate production-oriented config example only if that maintained
|
||||
artifact has clear value; otherwise remove the duplicate production block
|
||||
from config documentation.
|
||||
- Preserve the fixtures as non-sensitive synthetic data.
|
||||
|
||||
## Implementation Sequence
|
||||
|
||||
### Stage 1: Establish Canonical Developer Structure
|
||||
|
||||
1. Create the initial documentation ADR if retained.
|
||||
2. Create `docs/internal/overview.md`.
|
||||
3. Rewrite architecture around normative ownership and invariants.
|
||||
4. Update `docs/development.md` routing.
|
||||
5. Rescue still-valid development rules and assign every removed recipe.
|
||||
|
||||
**Gate:** Contributor orientation, architecture, component inventory, policies,
|
||||
and decision history have distinct owners and no stale links.
|
||||
|
||||
### Stage 2: Correct And Consolidate External Contracts
|
||||
|
||||
1. Revise `docs/config.md` and correct its known factual issues.
|
||||
2. Revise `docs/cli.md`.
|
||||
3. Revise `docs/api.md`.
|
||||
4. Revise both consumer guides.
|
||||
5. Revise both integration contracts.
|
||||
6. Verify each volatile contract against code and maintained assets.
|
||||
|
||||
**Gate:** Each externally observable field, default, flag, route, status, import
|
||||
path, and wire behavior has one authoritative definition.
|
||||
|
||||
### Stage 3: Refocus Internal Documentation
|
||||
|
||||
1. Revise adapter, runner, and source docs.
|
||||
2. Add the internal LLM document if warranted.
|
||||
3. Incorporate the former subsystem change recipes.
|
||||
4. Remove global policy and external-contract duplication.
|
||||
|
||||
**Gate:** Internal docs explain implementation and package-local guarantees,
|
||||
link external contracts, and contain the detailed contributor recipes needed
|
||||
for safe changes.
|
||||
|
||||
### Stage 4: Rewrite Operations And Consolidate Recovery Guidance
|
||||
|
||||
1. Rewrite operations as a runbook.
|
||||
2. Migrate useful troubleshooting content.
|
||||
3. Delete `docs/troubleshooting.md`.
|
||||
4. Update all inbound links and the development reading guide.
|
||||
|
||||
**Gate:** Operations owns deployment and recovery without serving as a second
|
||||
CLI, config, or API reference.
|
||||
|
||||
### Stage 5: Normalize Examples And Orientation
|
||||
|
||||
1. Move any remaining complete copyable artifacts into `examples/`.
|
||||
2. Update README navigation and example links.
|
||||
3. Run maintained smoke examples.
|
||||
4. Add only high-value automated checks justified by the testing policy.
|
||||
|
||||
**Gate:** Examples are canonical, runnable, secret-free, linked from their
|
||||
owning references, and not duplicated as complete files in prose docs.
|
||||
|
||||
### Stage 6: Final Compliance Pass
|
||||
|
||||
1. Validate all local links and paths.
|
||||
2. Search for duplicated volatile values across non-owning documents.
|
||||
3. Reconcile flags, config fields/defaults, API DTOs/codes, public types/errors,
|
||||
and built-in profiles with implementation.
|
||||
4. Run `go test ./...`, `go vet ./...`, and the documented build command.
|
||||
5. Run the maintained render script and Go package example.
|
||||
6. Confirm that non-roadmap current docs contain implemented behavior only.
|
||||
7. Record completion in this roadmap and update the documentation gate in
|
||||
`docs/roadmap/migration.md`.
|
||||
|
||||
**Gate:** Code, tests, examples, contracts, internal documentation, operations,
|
||||
policies, and roadmap status agree.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The documentation refresh is complete when:
|
||||
|
||||
- every topic in the ownership table has one canonical owner;
|
||||
- `docs/internal/overview.md` exists and architecture no longer owns the package
|
||||
inventory;
|
||||
- significant accepted decisions have an ADR owner;
|
||||
- all known factual discrepancies in this audit are corrected;
|
||||
- subsystem recipes from the former development guide have been preserved in
|
||||
relevant internal docs;
|
||||
- operations contains the retained recovery guidance and the standalone
|
||||
troubleshooting document is removed;
|
||||
- contracts and internal docs link to one another instead of maintaining
|
||||
parallel volatile definitions;
|
||||
- maintained examples are runnable and not duplicated as complete prose
|
||||
examples;
|
||||
- local links, validation commands, tests, and smoke checks pass;
|
||||
- the documentation-refresh gate in the Promptkit migration roadmap is marked
|
||||
complete.
|
||||
292
docs/roadmap/migration.md
Normal file
292
docs/roadmap/migration.md
Normal file
@@ -0,0 +1,292 @@
|
||||
# Promptkit Migration Roadmap
|
||||
|
||||
## Status
|
||||
|
||||
Accepted plan. This document describes proposed work that is not yet
|
||||
implemented.
|
||||
|
||||
## Objective
|
||||
|
||||
Split the current repository into two projects:
|
||||
|
||||
- **Promptkit**: the reusable Go framework, public Go facade, execution engine,
|
||||
source and validation support, OpenAI-compatible client, extension
|
||||
interfaces, and built-in execution-profile registry.
|
||||
- **Scriptorium**: a slim runnable application that imports Promptkit and
|
||||
provides the CLI and HTTP interfaces.
|
||||
|
||||
Scriptorium will become another downstream Promptkit consumer rather than the
|
||||
owner of the framework.
|
||||
|
||||
## Compatibility And Migration Policy
|
||||
|
||||
This is an intentionally breaking change.
|
||||
|
||||
- New and migrated Go consumers must import Promptkit instead of Scriptorium.
|
||||
- Scriptorium will not retain type aliases, forwarding packages, deprecated
|
||||
facade APIs, or other source-compatibility shims.
|
||||
- Existing consumers may continue using a previously tagged Scriptorium module
|
||||
version until they are migrated.
|
||||
- The migration does not need to preserve compatibility between intermediate
|
||||
development states. Each completed phase must instead leave the affected
|
||||
repository internally consistent and tested.
|
||||
- Promptkit should initially preserve the useful shape and behavior of the
|
||||
current public Go facade where doing so reduces extraction risk. Broader API
|
||||
redesign should follow the split unless required to establish the new
|
||||
boundary.
|
||||
|
||||
## Target Ownership
|
||||
|
||||
Promptkit should own application-neutral framework behavior:
|
||||
|
||||
- public engine, request, result, option, extension, and error APIs;
|
||||
- prompt-definition loading and rendering;
|
||||
- execution profiles, overlays, and the built-in profile registry;
|
||||
- artifact-loading interfaces and general-purpose `file` and `inline` support;
|
||||
- schema loading and output validation;
|
||||
- LLM client boundaries and the OpenAI-compatible implementation;
|
||||
- preparation and execution orchestration;
|
||||
- framework and execution defaults.
|
||||
|
||||
Scriptorium should own executable and transport concerns:
|
||||
|
||||
- the `scriptorium` command and its `run`, `render`, and `serve` interfaces;
|
||||
- CLI parsing, output formatting, exit codes, and process behavior;
|
||||
- application-config discovery and CLI precedence;
|
||||
- HTTP routing, request and response DTOs, limits, and error/status mapping;
|
||||
- HTTP artifact-root and deployment security policy;
|
||||
- server and adapter defaults;
|
||||
- executable examples, operations guidance, and transport documentation.
|
||||
|
||||
The intended dependency direction is:
|
||||
|
||||
```text
|
||||
Scriptorium CLI and HTTP adapters
|
||||
|
|
||||
v
|
||||
Promptkit
|
||||
|
|
||||
v
|
||||
consumer-supplied sources and clients
|
||||
```
|
||||
|
||||
Scriptorium must use Promptkit's public API. It must not depend on Promptkit
|
||||
implementation packages or reproduce framework orchestration.
|
||||
|
||||
## Migration Steps
|
||||
|
||||
### Step 1: Refresh And Synchronize Documentation
|
||||
|
||||
Perform a repository-wide documentation refresh before migration development.
|
||||
|
||||
At minimum:
|
||||
|
||||
- reconcile all current-behavior documentation with the code, tests, examples,
|
||||
defaults, and current public contracts;
|
||||
- introduce the planned documentation-policy updates;
|
||||
- establish an architecture decision record policy and canonical ADR location;
|
||||
- resolve stale, duplicated, or misplaced material;
|
||||
- validate documentation links and maintained examples;
|
||||
- leave future migration behavior in `docs/roadmap/` until implemented.
|
||||
|
||||
**Gate:** Do not begin architectural migration work until the documentation
|
||||
refresh and policy updates are merged and the repository has an agreed,
|
||||
accurate baseline.
|
||||
|
||||
### Step 2: Record The Architectural Decision And Detailed Boundary
|
||||
|
||||
Create an ADR, under the policy established in Step 1, that records:
|
||||
|
||||
- the decision to split Promptkit from Scriptorium;
|
||||
- the target ownership and dependency direction;
|
||||
- the selected Promptkit repository and Go module paths;
|
||||
- the breaking-change and versioning policy;
|
||||
- ownership of configuration fields and defaults;
|
||||
- artifact-reader and HTTP containment responsibilities;
|
||||
- local multi-repository development and release coordination;
|
||||
- documentation ownership after the split.
|
||||
|
||||
Use the ADR to resolve any remaining public-boundary decisions before code is
|
||||
moved.
|
||||
|
||||
**Gate:** The ADR is accepted, and every existing package, public contract,
|
||||
configuration category, and maintained asset has a target owner.
|
||||
|
||||
### Step 3: Characterize Existing Framework Behavior
|
||||
|
||||
Strengthen or add contract-focused tests where needed so extraction can be
|
||||
verified without relying on package placement.
|
||||
|
||||
Preserve coverage of:
|
||||
|
||||
- `Prepare` and `Run` behavior;
|
||||
- prompt, profile, execution-default, and request-override precedence;
|
||||
- presence-aware numeric overrides;
|
||||
- built-in profile fallback and custom-profile overlays;
|
||||
- strict YAML and JSON decoding;
|
||||
- prompt, profile, schema, and artifact source behavior;
|
||||
- structured-output requests and output validation;
|
||||
- validation failures versus validation runtime errors;
|
||||
- secret handling and redaction;
|
||||
- public error classification;
|
||||
- HTTP artifact restrictions and transport mappings.
|
||||
|
||||
**Gate:** Current framework and adapter contracts are represented by passing
|
||||
tests sufficient to detect behavioral regressions during the split.
|
||||
|
||||
### Step 4: Make Scriptorium Adapters Consume The Public Facade
|
||||
|
||||
Within the current repository, refactor the CLI and HTTP adapters to use the
|
||||
public framework facade rather than constructing or importing internal runner
|
||||
components directly.
|
||||
|
||||
Add only the minimum public capabilities needed to support this boundary. These
|
||||
may include:
|
||||
|
||||
- a small `Run`/`Prepare` consumer interface;
|
||||
- injectable artifact-reading behavior for Scriptorium's restricted HTTP
|
||||
policy;
|
||||
- source options currently available only through internal constructors;
|
||||
- prepared-run formatting based on public types;
|
||||
- stable public error classification required by CLI and HTTP mappings.
|
||||
|
||||
Do not broadly export internal repositories, domain types, or use-case
|
||||
implementations.
|
||||
|
||||
**Gate:** The CLI and HTTP adapters use only the public framework API for
|
||||
framework behavior, and all tests and documented smoke commands pass.
|
||||
|
||||
### Step 5: Create The Promptkit Repository
|
||||
|
||||
Create the Promptkit repository and Go module as an explicit out-of-band
|
||||
operation.
|
||||
|
||||
Establish:
|
||||
|
||||
- repository access, ownership, and branch protections;
|
||||
- the module path selected by the ADR;
|
||||
- baseline development, architecture, documentation, and release policies;
|
||||
- CI for build, test, vet, and other agreed checks;
|
||||
- an initial package layout centered on a small public facade with internal
|
||||
implementation packages;
|
||||
- a local development workflow for coordinated Promptkit and Scriptorium
|
||||
changes, using a workspace or temporary uncommitted module replacement where
|
||||
appropriate.
|
||||
|
||||
Do not commit local filesystem `replace` directives to release branches.
|
||||
|
||||
**Gate:** The Promptkit repository exists, is accessible to maintainers, has
|
||||
working CI and policy scaffolding, and can receive the extracted framework.
|
||||
Do not begin cross-repository extraction until this out-of-band work is
|
||||
confirmed complete.
|
||||
|
||||
### Step 6: Extract And Stabilize Promptkit
|
||||
|
||||
Move the application-neutral framework and built-in profile assets into
|
||||
Promptkit. Preserve implementation packages as internal where practical.
|
||||
|
||||
The initial public API should remain focused on the established engine workflow
|
||||
and the source and client extension points required by real consumers. Avoid
|
||||
combining the extraction with unrelated API redesign.
|
||||
|
||||
Move or recreate the relevant:
|
||||
|
||||
- framework implementation;
|
||||
- public package tests and framework contract tests;
|
||||
- built-in profile assets and registry tests;
|
||||
- Go consumer examples;
|
||||
- framework, consumer, configuration-format, and integration documentation.
|
||||
|
||||
Verify that Promptkit can be built, tested, and consumed independently of the
|
||||
Scriptorium repository.
|
||||
|
||||
**Gate:** Promptkit independently provides the agreed framework contract,
|
||||
passes its CI checks, and has a tagged version that Scriptorium and other
|
||||
consumers can import.
|
||||
|
||||
### Step 7: Slim Scriptorium And Adopt Promptkit
|
||||
|
||||
Update Scriptorium to import the tagged Promptkit module and remove the
|
||||
framework implementation and public Go facade that Promptkit replaces.
|
||||
|
||||
Retain only Scriptorium-owned executable and transport behavior. In particular:
|
||||
|
||||
- wire CLI and HTTP requests through Promptkit's public API;
|
||||
- keep application config and transport defaults in Scriptorium;
|
||||
- keep restricted HTTP artifact policy in Scriptorium while injecting it
|
||||
through Promptkit's supported boundary;
|
||||
- remove obsolete framework packages, tests, and documentation;
|
||||
- update Scriptorium examples and docs to describe the CLI and HTTP application;
|
||||
- direct Go framework consumers to Promptkit without providing compatibility
|
||||
aliases or forwarding APIs.
|
||||
|
||||
**Gate:** Scriptorium builds and passes all tests using a tagged Promptkit
|
||||
dependency, contains no duplicate framework implementation, and its current
|
||||
documentation describes only the slimmed application.
|
||||
|
||||
### Step 8: Migrate Downstream Consumers To Promptkit
|
||||
|
||||
Inventory downstream Go consumers and migrate each from the Scriptorium package
|
||||
to Promptkit. This work may occur in external repositories and must be tracked
|
||||
explicitly.
|
||||
|
||||
For each consumer:
|
||||
|
||||
- update module imports and dependencies;
|
||||
- adapt to any intentionally changed public API;
|
||||
- run its tests and relevant integration or smoke checks;
|
||||
- confirm configuration, source, validation, and error behavior;
|
||||
- release or deploy the migrated consumer through its normal process.
|
||||
|
||||
Consumers that cannot migrate immediately may remain pinned to the last
|
||||
framework-bearing Scriptorium tag. No compatibility work is required in the new
|
||||
Scriptorium project for those consumers.
|
||||
|
||||
**Gate:** All in-scope downstream consumers are either migrated and verified or
|
||||
explicitly recorded as remaining on the previous Scriptorium version with an
|
||||
owner and follow-up plan. Do not declare the ecosystem migration complete until
|
||||
the required out-of-band consumer changes are confirmed.
|
||||
|
||||
### Step 9: Complete Release And Documentation Cutover
|
||||
|
||||
Complete the coordinated project transition:
|
||||
|
||||
- publish Promptkit before dependent Scriptorium releases;
|
||||
- release the breaking Scriptorium version against the tagged Promptkit
|
||||
dependency;
|
||||
- publish migration guidance that maps the former Scriptorium Go API to
|
||||
Promptkit;
|
||||
- update cross-project links, examples, package documentation, and release
|
||||
notes;
|
||||
- verify that no release artifact depends on local workspaces or replacements;
|
||||
- archive completed roadmap material according to the documentation policy in
|
||||
effect at that time.
|
||||
|
||||
**Gate:** Promptkit and Scriptorium are independently releasable, their
|
||||
documentation has distinct and accurate ownership, and the migration status of
|
||||
all identified downstream consumers is recorded.
|
||||
|
||||
## Cross-Cutting Constraints
|
||||
|
||||
- Preserve the invariant that execution orchestration remains narrow and
|
||||
application-neutral.
|
||||
- Keep adapter-specific decisions out of Promptkit.
|
||||
- Keep Scriptorium dependent only on Promptkit's supported public API.
|
||||
- Preserve strict external decoding, error classification, validation
|
||||
semantics, and secret redaction throughout the migration.
|
||||
- Keep each repository buildable and testable at merged phase boundaries.
|
||||
- Coordinate cross-repository changes through tagged dependencies and explicit
|
||||
gates rather than assuming atomic commits across repositories.
|
||||
- Document only implemented behavior outside roadmap files.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The migration is complete when:
|
||||
|
||||
- Promptkit is the independent owner of the reusable framework and built-in
|
||||
profiles;
|
||||
- Scriptorium is a slim CLI and HTTP consumer of Promptkit;
|
||||
- Scriptorium no longer exposes or maintains the former public Go framework;
|
||||
- all required downstream migrations and external repository work have been
|
||||
completed or explicitly dispositioned;
|
||||
- both repositories build, test, document, version, and release independently.
|
||||
Reference in New Issue
Block a user