eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Add a stabilization roadmap to support a future HTTP API

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-02 14:51:52 -05:00
parent 9aaa1e9426
commit 44df38e555
1 changed files with 151 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files

235
docs/roadmap/implementation.md
View File

@@ -1,102 +1,169 @@
# Roadmap
# HTTP API Stabilization Roadmap
This directory contains only future, deferred, or aspirational work for
`distributor`. Implemented behavior is documented in the current user,
operator, internal, policy, integration, and example documentation:
## Summary
- `README.md`
- `docs/cli.md`
- `docs/config.md`
- `docs/operations.md`
- `docs/troubleshooting.md`
- `docs/internal/`
- `docs/integrations/markdown.md`
- `docs/policy/`
- `examples/`
Prepare `distributor` for a later HTTP API without implementing the HTTP
server in this roadmap. The goal is to make the existing run workflow callable
as a single-pipeline, structured, concurrency-safe application use case.
`distributor` currently supports local, SSH/SFTP, and S3-compatible source and
destination backends; producer bundle creation through `pkg/bundle` and
`distributor manifest create`; configured source validation and inspection;
Markdown sidecar and `index.html` publication; archive and fixed destination
path mapping; destination link metadata; shared text/JSON CLI output; and
managed destination replacement behavior.
Chosen defaults:
## Future Work
- Scope: stabilization only; HTTP routes and `serve` command are deferred to a
later roadmap.
- Future HTTP trigger mode: asynchronous start with run status.
- Security boundary: private bind or reverse proxy/mTLS outside the app; no
app-level auth or in-app TLS in v1.
- Trigger input: pipeline ID only.
These items are not implemented. They should not be documented as current
behavior outside `docs/roadmap/` unless a future implementation adds them.
## Implementation Rules
### CLI And Status Output
- Implement stages in order; each stage should be one prompt or commit unless
trivially small.
- Before each stage, read `docs/policy/architecture.md`,
`docs/policy/development.md`, and `docs/policy/documentation.md`.
- Preserve existing CLI behavior, text output, JSON output, config semantics,
backend behavior, manifest schema, and destination state schema.
- Do not introduce external dependencies, a CLI framework, a generic workflow
engine, a plugin system, durable run storage, or broad adapter abstractions.
- Do not document the HTTP API as implemented outside `docs/roadmap/`.
- Add a root-global output flag only if the command parser is later refactored
around shared root options.
- Add output formats beyond `text` and `json` only if a concrete consumer
requires them.
- Add a versioned JSON schema reference after the first JSON-capable release.
- Add destination-state inspection behind an explicit flag such as
`--with-destinations` if operators need fan-out status diagnostics from
`inspect`.
- Add additional status or inspection presentation for destination primary
links beyond the current `run --format json` result model.
## Stages
### Producer Workflows
### Stage 1: Structured Run Core
- Add a no-write manifest creation mode, such as writing manifest JSON to
stdout, if producer pipelines need to capture manifests directly.
- Add broader producer workflow helpers, such as richer ignore rules or
template scaffolding, if real producer use cases require them.
- Add remote or storage-backed producer writers only if producer applications
need to assemble bundles outside the local filesystem.
Refactor `internal/app` so the run workflow first produces a structured run
report, then projects that report to text or JSON output.
### Publication And Transform Behavior
Required behavior:
- Add a separate collection or site-index transform if distributor needs
multi-page aggregation.
- Add richer transform metadata only if future state consumers need more than
the transform name and output path.
- Add custom HTML index output names only if fixed `index.html` is too limiting
for real deployments.
- Add richer fixed-destination source selection policies if deployments need
something other than newest-by-`created`.
- Add stricter handling for equal latest timestamps if timestamp ties become
common in producer workflows.
- Add higher-level status or approval workflows for fixed-root replacements if
dry-run output is not enough operational protection.
- Add richer link policies only if `auto`, `html`, and `source` prove
insufficient.
- Keep `app.Run` as the CLI-facing entrypoint.
- Move stdout writes out of the core planning/execution loop where practical.
- Preserve the existing `run --format text` and `run --format json` output
byte-for-byte except where tests already allow map ordering.
- Preserve partial-result behavior: destination-scoped failures produce a
structured report plus an aggregated error.
### State And Compatibility
Suggested internal shape:
- Define a post-release destination state schema bump policy before introducing
materially incompatible state changes.
- Add warning-only digest mismatch handling only if an operator workflow needs
publication to continue after validation failures.
- Add compatibility parsing for legacy SSH URI config only if migration support
is required.
- Introduce an app-owned `RunReport` model containing dry-run state, pipeline
summaries, action records, summary counters, warnings, and output errors.
- Keep JSON tags compatible with the current `run` JSON output.
- Keep failure aggregation in `internal/app`; do not move it into `publish`,
`config`, or storage adapters.
### Backends, Security, And Deployment
Tests:
- Add authentication mechanisms beyond the implemented SSH agent/key and S3
credential paths only when a concrete backend workflow requires them.
- Add broad recursive destination deletion outside managed bundle paths only if
a future design can preserve the current safety boundary.
- Add concurrent fan-out publishing only if runtime profiling shows it is
needed.
- Add streaming, resumable, or multipart S3 uploads only if object sizes make
the current write path insufficient.
- Add cloud-provider-specific IAM integration docs only when the repository
includes tested provider-specific behavior.
- Add repository-managed packaging, release, and deployment automation when the
release process is ready to be standardized.
- Add or update `internal/app` tests proving structured reports include
warnings, actions, outputs, summary counters, and partial failures.
- Add CLI regression coverage for existing text and JSON output.
- Run `go test ./internal/app ./internal/cli`.
## Roadmap Maintenance
### Stage 2: Single-Pipeline Run Entry Point
When adding future roadmap work:
Add an app-layer entrypoint for running exactly one configured pipeline by ID.
- describe user-visible behavior and safety boundaries;
- define which current docs must change after implementation;
- keep examples secret-free and runnable or clearly environment-gated;
- keep workflow labels out of production code, tests, config fields, and
user-facing documentation;
- run focused tests for the changed behavior and `go test ./...` for
cross-package changes.
Required behavior:
- Add an exported internal app function such as
`RunPipeline(ctx, RunPipelineOptions) (RunReport, error)`.
- `RunPipelineOptions` must include config path, pipeline ID, dry-run, force,
and optional notifier.
- Invalid pipeline IDs must return a typed or predicate-detectable error
suitable for later HTTP `404` mapping.
- `RunPipeline` must reuse the same backend factory, secret loading, transform
registry, warning generation, destination planning, publish execution,
notification behavior, and failure aggregation as `Run`.
- Do not add a public CLI `run --pipeline` flag in this stage.
Tests:
- Run only the requested pipeline from a multi-pipeline config.
- Return the chosen not-found error for an unknown pipeline ID.
- Preserve existing `app.Run` all-pipelines behavior.
- Run `go test ./internal/app ./internal/cli`.
### Stage 3: In-Memory Pipeline Run Coordinator
Add a narrow concurrency coordinator around the single-pipeline app seam.
Required behavior:
- Keep the coordinator in `internal/app` unless a later HTTP implementation
introduces a transport package.
- Allow different pipeline IDs to run concurrently.
- Reject a second in-flight run for the same pipeline ID with a typed or
predicate-detectable duplicate-run error.
- Track active runs in memory only.
- Clear active state after success, failure, or context cancellation.
- Do not queue duplicate runs and do not persist run state to disk or a
database.
Async-ready constraints:
- Model run IDs and status records so a later HTTP API can expose asynchronous
status.
- A future HTTP request context should guard admission; the actual run should
be tied to a server/coordinator lifetime context.
Tests:
- Concurrent same-pipeline requests produce exactly one accepted run and one
duplicate-run error.
- Concurrent different-pipeline requests both start.
- Active state is cleared after success and after failure.
- Unknown pipeline IDs do not leave active state behind.
- Run `go test ./internal/app`.
### Stage 4: Future HTTP Boundary Contract
Record the target HTTP boundary in this roadmap only; do not implement routes
yet.
Required target contract:
- Future trigger endpoint should accept only a pipeline ID.
- Future trigger behavior should be asynchronous: accept the run, return a run
ID, and expose status through a later status endpoint.
- Future duplicate in-flight pipeline runs should map to `409 Conflict`.
- Future unknown pipeline IDs should map to `404 Not Found`.
- Future server should default to private binding, such as `127.0.0.1`, and
rely on a reverse proxy, private network, or external mTLS for transport
security.
- Do not add bearer-token auth, in-app TLS config, or public-network exposure in
v1 unless a later roadmap explicitly changes this decision.
Tests:
- Documentation-only stage; no tests required beyond any repository doc checks
that exist.
### Stage 5: Stabilization Sweep
Clean up only code made obsolete by stages 1-3.
Required behavior:
- Remove duplicate result projection, warning collection, and
pipeline-selection helpers only when clearly replaced.
- Keep test helpers local unless they are broadly reusable.
- Update `docs/internal/app.md` only for implemented internal app contracts.
- Do not update `docs/cli.md`, `docs/config.md`, `docs/operations.md`, or
README for HTTP behavior because no HTTP API exists yet.
Tests:
- Run `go test ./internal/app ./internal/cli`.
- Run `go test ./...` before considering the roadmap complete.
## Acceptance Criteria
The roadmap is complete when:
- Existing `distributor run` behavior is unchanged.
- `internal/app` exposes a tested single-pipeline run path.
- Run results are available as structured data without scraping stdout.
- The coordinator prevents concurrent same-pipeline runs while allowing
different pipelines to run.
- Future HTTP status, duplicate, not-found, and private-bind decisions are
recorded under `docs/roadmap/`.
- The full test suite passes with `go test ./...`.