From 44df38e55521ec14668701cbbae649b126b56c98 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 2 Jun 2026 14:51:52 -0500 Subject: [PATCH] Add a stabilization roadmap to support a future HTTP API --- docs/roadmap/implementation.md | 235 +++++++++++++++++++++------------ 1 file changed, 151 insertions(+), 84 deletions(-) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index eeb32bb..198311c 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,102 +1,169 @@ -# Roadmap +# HTTP API Stabilization Roadmap -This directory contains only future, deferred, or aspirational work for -`distributor`. Implemented behavior is documented in the current user, -operator, internal, policy, integration, and example documentation: +## Summary -- `README.md` -- `docs/cli.md` -- `docs/config.md` -- `docs/operations.md` -- `docs/troubleshooting.md` -- `docs/internal/` -- `docs/integrations/markdown.md` -- `docs/policy/` -- `examples/` +Prepare `distributor` for a later HTTP API without implementing the HTTP +server in this roadmap. The goal is to make the existing run workflow callable +as a single-pipeline, structured, concurrency-safe application use case. -`distributor` currently supports local, SSH/SFTP, and S3-compatible source and -destination backends; producer bundle creation through `pkg/bundle` and -`distributor manifest create`; configured source validation and inspection; -Markdown sidecar and `index.html` publication; archive and fixed destination -path mapping; destination link metadata; shared text/JSON CLI output; and -managed destination replacement behavior. +Chosen defaults: -## Future Work +- Scope: stabilization only; HTTP routes and `serve` command are deferred to a + later roadmap. +- Future HTTP trigger mode: asynchronous start with run status. +- Security boundary: private bind or reverse proxy/mTLS outside the app; no + app-level auth or in-app TLS in v1. +- Trigger input: pipeline ID only. -These items are not implemented. They should not be documented as current -behavior outside `docs/roadmap/` unless a future implementation adds them. +## Implementation Rules -### CLI And Status Output +- Implement stages in order; each stage should be one prompt or commit unless + trivially small. +- Before each stage, read `docs/policy/architecture.md`, + `docs/policy/development.md`, and `docs/policy/documentation.md`. +- Preserve existing CLI behavior, text output, JSON output, config semantics, + backend behavior, manifest schema, and destination state schema. +- Do not introduce external dependencies, a CLI framework, a generic workflow + engine, a plugin system, durable run storage, or broad adapter abstractions. +- Do not document the HTTP API as implemented outside `docs/roadmap/`. -- Add a root-global output flag only if the command parser is later refactored - around shared root options. -- Add output formats beyond `text` and `json` only if a concrete consumer - requires them. -- Add a versioned JSON schema reference after the first JSON-capable release. -- Add destination-state inspection behind an explicit flag such as - `--with-destinations` if operators need fan-out status diagnostics from - `inspect`. -- Add additional status or inspection presentation for destination primary - links beyond the current `run --format json` result model. +## Stages -### Producer Workflows +### Stage 1: Structured Run Core -- Add a no-write manifest creation mode, such as writing manifest JSON to - stdout, if producer pipelines need to capture manifests directly. -- Add broader producer workflow helpers, such as richer ignore rules or - template scaffolding, if real producer use cases require them. -- Add remote or storage-backed producer writers only if producer applications - need to assemble bundles outside the local filesystem. +Refactor `internal/app` so the run workflow first produces a structured run +report, then projects that report to text or JSON output. -### Publication And Transform Behavior +Required behavior: -- Add a separate collection or site-index transform if distributor needs - multi-page aggregation. -- Add richer transform metadata only if future state consumers need more than - the transform name and output path. -- Add custom HTML index output names only if fixed `index.html` is too limiting - for real deployments. -- Add richer fixed-destination source selection policies if deployments need - something other than newest-by-`created`. -- Add stricter handling for equal latest timestamps if timestamp ties become - common in producer workflows. -- Add higher-level status or approval workflows for fixed-root replacements if - dry-run output is not enough operational protection. -- Add richer link policies only if `auto`, `html`, and `source` prove - insufficient. +- Keep `app.Run` as the CLI-facing entrypoint. +- Move stdout writes out of the core planning/execution loop where practical. +- Preserve the existing `run --format text` and `run --format json` output + byte-for-byte except where tests already allow map ordering. +- Preserve partial-result behavior: destination-scoped failures produce a + structured report plus an aggregated error. -### State And Compatibility +Suggested internal shape: -- Define a post-release destination state schema bump policy before introducing - materially incompatible state changes. -- Add warning-only digest mismatch handling only if an operator workflow needs - publication to continue after validation failures. -- Add compatibility parsing for legacy SSH URI config only if migration support - is required. +- Introduce an app-owned `RunReport` model containing dry-run state, pipeline + summaries, action records, summary counters, warnings, and output errors. +- Keep JSON tags compatible with the current `run` JSON output. +- Keep failure aggregation in `internal/app`; do not move it into `publish`, + `config`, or storage adapters. -### Backends, Security, And Deployment +Tests: -- Add authentication mechanisms beyond the implemented SSH agent/key and S3 - credential paths only when a concrete backend workflow requires them. -- Add broad recursive destination deletion outside managed bundle paths only if - a future design can preserve the current safety boundary. -- Add concurrent fan-out publishing only if runtime profiling shows it is - needed. -- Add streaming, resumable, or multipart S3 uploads only if object sizes make - the current write path insufficient. -- Add cloud-provider-specific IAM integration docs only when the repository - includes tested provider-specific behavior. -- Add repository-managed packaging, release, and deployment automation when the - release process is ready to be standardized. +- Add or update `internal/app` tests proving structured reports include + warnings, actions, outputs, summary counters, and partial failures. +- Add CLI regression coverage for existing text and JSON output. +- Run `go test ./internal/app ./internal/cli`. -## Roadmap Maintenance +### Stage 2: Single-Pipeline Run Entry Point -When adding future roadmap work: +Add an app-layer entrypoint for running exactly one configured pipeline by ID. -- describe user-visible behavior and safety boundaries; -- define which current docs must change after implementation; -- keep examples secret-free and runnable or clearly environment-gated; -- keep workflow labels out of production code, tests, config fields, and - user-facing documentation; -- run focused tests for the changed behavior and `go test ./...` for - cross-package changes. +Required behavior: + +- Add an exported internal app function such as + `RunPipeline(ctx, RunPipelineOptions) (RunReport, error)`. +- `RunPipelineOptions` must include config path, pipeline ID, dry-run, force, + and optional notifier. +- Invalid pipeline IDs must return a typed or predicate-detectable error + suitable for later HTTP `404` mapping. +- `RunPipeline` must reuse the same backend factory, secret loading, transform + registry, warning generation, destination planning, publish execution, + notification behavior, and failure aggregation as `Run`. +- Do not add a public CLI `run --pipeline` flag in this stage. + +Tests: + +- Run only the requested pipeline from a multi-pipeline config. +- Return the chosen not-found error for an unknown pipeline ID. +- Preserve existing `app.Run` all-pipelines behavior. +- Run `go test ./internal/app ./internal/cli`. + +### Stage 3: In-Memory Pipeline Run Coordinator + +Add a narrow concurrency coordinator around the single-pipeline app seam. + +Required behavior: + +- Keep the coordinator in `internal/app` unless a later HTTP implementation + introduces a transport package. +- Allow different pipeline IDs to run concurrently. +- Reject a second in-flight run for the same pipeline ID with a typed or + predicate-detectable duplicate-run error. +- Track active runs in memory only. +- Clear active state after success, failure, or context cancellation. +- Do not queue duplicate runs and do not persist run state to disk or a + database. + +Async-ready constraints: + +- Model run IDs and status records so a later HTTP API can expose asynchronous + status. +- A future HTTP request context should guard admission; the actual run should + be tied to a server/coordinator lifetime context. + +Tests: + +- Concurrent same-pipeline requests produce exactly one accepted run and one + duplicate-run error. +- Concurrent different-pipeline requests both start. +- Active state is cleared after success and after failure. +- Unknown pipeline IDs do not leave active state behind. +- Run `go test ./internal/app`. + +### Stage 4: Future HTTP Boundary Contract + +Record the target HTTP boundary in this roadmap only; do not implement routes +yet. + +Required target contract: + +- Future trigger endpoint should accept only a pipeline ID. +- Future trigger behavior should be asynchronous: accept the run, return a run + ID, and expose status through a later status endpoint. +- Future duplicate in-flight pipeline runs should map to `409 Conflict`. +- Future unknown pipeline IDs should map to `404 Not Found`. +- Future server should default to private binding, such as `127.0.0.1`, and + rely on a reverse proxy, private network, or external mTLS for transport + security. +- Do not add bearer-token auth, in-app TLS config, or public-network exposure in + v1 unless a later roadmap explicitly changes this decision. + +Tests: + +- Documentation-only stage; no tests required beyond any repository doc checks + that exist. + +### Stage 5: Stabilization Sweep + +Clean up only code made obsolete by stages 1-3. + +Required behavior: + +- Remove duplicate result projection, warning collection, and + pipeline-selection helpers only when clearly replaced. +- Keep test helpers local unless they are broadly reusable. +- Update `docs/internal/app.md` only for implemented internal app contracts. +- Do not update `docs/cli.md`, `docs/config.md`, `docs/operations.md`, or + README for HTTP behavior because no HTTP API exists yet. + +Tests: + +- Run `go test ./internal/app ./internal/cli`. +- Run `go test ./...` before considering the roadmap complete. + +## Acceptance Criteria + +The roadmap is complete when: + +- Existing `distributor run` behavior is unchanged. +- `internal/app` exposes a tested single-pipeline run path. +- Run results are available as structured data without scraping stdout. +- The coordinator prevents concurrent same-pipeline runs while allowing + different pipelines to run. +- Future HTTP status, duplicate, not-found, and private-bind decisions are + recorded under `docs/roadmap/`. +- The full test suite passes with `go test ./...`.