Add a stabilization roadmap to support a future HTTP API
This commit is contained in:
@@ -1,102 +1,169 @@
|
|||||||
# Roadmap
|
# HTTP API Stabilization Roadmap
|
||||||
|
|
||||||
This directory contains only future, deferred, or aspirational work for
|
## Summary
|
||||||
`distributor`. Implemented behavior is documented in the current user,
|
|
||||||
operator, internal, policy, integration, and example documentation:
|
|
||||||
|
|
||||||
- `README.md`
|
Prepare `distributor` for a later HTTP API without implementing the HTTP
|
||||||
- `docs/cli.md`
|
server in this roadmap. The goal is to make the existing run workflow callable
|
||||||
- `docs/config.md`
|
as a single-pipeline, structured, concurrency-safe application use case.
|
||||||
- `docs/operations.md`
|
|
||||||
- `docs/troubleshooting.md`
|
|
||||||
- `docs/internal/`
|
|
||||||
- `docs/integrations/markdown.md`
|
|
||||||
- `docs/policy/`
|
|
||||||
- `examples/`
|
|
||||||
|
|
||||||
`distributor` currently supports local, SSH/SFTP, and S3-compatible source and
|
Chosen defaults:
|
||||||
destination backends; producer bundle creation through `pkg/bundle` and
|
|
||||||
`distributor manifest create`; configured source validation and inspection;
|
|
||||||
Markdown sidecar and `index.html` publication; archive and fixed destination
|
|
||||||
path mapping; destination link metadata; shared text/JSON CLI output; and
|
|
||||||
managed destination replacement behavior.
|
|
||||||
|
|
||||||
## Future Work
|
- Scope: stabilization only; HTTP routes and `serve` command are deferred to a
|
||||||
|
later roadmap.
|
||||||
|
- Future HTTP trigger mode: asynchronous start with run status.
|
||||||
|
- Security boundary: private bind or reverse proxy/mTLS outside the app; no
|
||||||
|
app-level auth or in-app TLS in v1.
|
||||||
|
- Trigger input: pipeline ID only.
|
||||||
|
|
||||||
These items are not implemented. They should not be documented as current
|
## Implementation Rules
|
||||||
behavior outside `docs/roadmap/` unless a future implementation adds them.
|
|
||||||
|
|
||||||
### CLI And Status Output
|
- Implement stages in order; each stage should be one prompt or commit unless
|
||||||
|
trivially small.
|
||||||
|
- Before each stage, read `docs/policy/architecture.md`,
|
||||||
|
`docs/policy/development.md`, and `docs/policy/documentation.md`.
|
||||||
|
- Preserve existing CLI behavior, text output, JSON output, config semantics,
|
||||||
|
backend behavior, manifest schema, and destination state schema.
|
||||||
|
- Do not introduce external dependencies, a CLI framework, a generic workflow
|
||||||
|
engine, a plugin system, durable run storage, or broad adapter abstractions.
|
||||||
|
- Do not document the HTTP API as implemented outside `docs/roadmap/`.
|
||||||
|
|
||||||
- Add a root-global output flag only if the command parser is later refactored
|
## Stages
|
||||||
around shared root options.
|
|
||||||
- Add output formats beyond `text` and `json` only if a concrete consumer
|
|
||||||
requires them.
|
|
||||||
- Add a versioned JSON schema reference after the first JSON-capable release.
|
|
||||||
- Add destination-state inspection behind an explicit flag such as
|
|
||||||
`--with-destinations` if operators need fan-out status diagnostics from
|
|
||||||
`inspect`.
|
|
||||||
- Add additional status or inspection presentation for destination primary
|
|
||||||
links beyond the current `run --format json` result model.
|
|
||||||
|
|
||||||
### Producer Workflows
|
### Stage 1: Structured Run Core
|
||||||
|
|
||||||
- Add a no-write manifest creation mode, such as writing manifest JSON to
|
Refactor `internal/app` so the run workflow first produces a structured run
|
||||||
stdout, if producer pipelines need to capture manifests directly.
|
report, then projects that report to text or JSON output.
|
||||||
- Add broader producer workflow helpers, such as richer ignore rules or
|
|
||||||
template scaffolding, if real producer use cases require them.
|
|
||||||
- Add remote or storage-backed producer writers only if producer applications
|
|
||||||
need to assemble bundles outside the local filesystem.
|
|
||||||
|
|
||||||
### Publication And Transform Behavior
|
Required behavior:
|
||||||
|
|
||||||
- Add a separate collection or site-index transform if distributor needs
|
- Keep `app.Run` as the CLI-facing entrypoint.
|
||||||
multi-page aggregation.
|
- Move stdout writes out of the core planning/execution loop where practical.
|
||||||
- Add richer transform metadata only if future state consumers need more than
|
- Preserve the existing `run --format text` and `run --format json` output
|
||||||
the transform name and output path.
|
byte-for-byte except where tests already allow map ordering.
|
||||||
- Add custom HTML index output names only if fixed `index.html` is too limiting
|
- Preserve partial-result behavior: destination-scoped failures produce a
|
||||||
for real deployments.
|
structured report plus an aggregated error.
|
||||||
- Add richer fixed-destination source selection policies if deployments need
|
|
||||||
something other than newest-by-`created`.
|
|
||||||
- Add stricter handling for equal latest timestamps if timestamp ties become
|
|
||||||
common in producer workflows.
|
|
||||||
- Add higher-level status or approval workflows for fixed-root replacements if
|
|
||||||
dry-run output is not enough operational protection.
|
|
||||||
- Add richer link policies only if `auto`, `html`, and `source` prove
|
|
||||||
insufficient.
|
|
||||||
|
|
||||||
### State And Compatibility
|
Suggested internal shape:
|
||||||
|
|
||||||
- Define a post-release destination state schema bump policy before introducing
|
- Introduce an app-owned `RunReport` model containing dry-run state, pipeline
|
||||||
materially incompatible state changes.
|
summaries, action records, summary counters, warnings, and output errors.
|
||||||
- Add warning-only digest mismatch handling only if an operator workflow needs
|
- Keep JSON tags compatible with the current `run` JSON output.
|
||||||
publication to continue after validation failures.
|
- Keep failure aggregation in `internal/app`; do not move it into `publish`,
|
||||||
- Add compatibility parsing for legacy SSH URI config only if migration support
|
`config`, or storage adapters.
|
||||||
is required.
|
|
||||||
|
|
||||||
### Backends, Security, And Deployment
|
Tests:
|
||||||
|
|
||||||
- Add authentication mechanisms beyond the implemented SSH agent/key and S3
|
- Add or update `internal/app` tests proving structured reports include
|
||||||
credential paths only when a concrete backend workflow requires them.
|
warnings, actions, outputs, summary counters, and partial failures.
|
||||||
- Add broad recursive destination deletion outside managed bundle paths only if
|
- Add CLI regression coverage for existing text and JSON output.
|
||||||
a future design can preserve the current safety boundary.
|
- Run `go test ./internal/app ./internal/cli`.
|
||||||
- Add concurrent fan-out publishing only if runtime profiling shows it is
|
|
||||||
needed.
|
|
||||||
- Add streaming, resumable, or multipart S3 uploads only if object sizes make
|
|
||||||
the current write path insufficient.
|
|
||||||
- Add cloud-provider-specific IAM integration docs only when the repository
|
|
||||||
includes tested provider-specific behavior.
|
|
||||||
- Add repository-managed packaging, release, and deployment automation when the
|
|
||||||
release process is ready to be standardized.
|
|
||||||
|
|
||||||
## Roadmap Maintenance
|
### Stage 2: Single-Pipeline Run Entry Point
|
||||||
|
|
||||||
When adding future roadmap work:
|
Add an app-layer entrypoint for running exactly one configured pipeline by ID.
|
||||||
|
|
||||||
- describe user-visible behavior and safety boundaries;
|
Required behavior:
|
||||||
- define which current docs must change after implementation;
|
|
||||||
- keep examples secret-free and runnable or clearly environment-gated;
|
- Add an exported internal app function such as
|
||||||
- keep workflow labels out of production code, tests, config fields, and
|
`RunPipeline(ctx, RunPipelineOptions) (RunReport, error)`.
|
||||||
user-facing documentation;
|
- `RunPipelineOptions` must include config path, pipeline ID, dry-run, force,
|
||||||
- run focused tests for the changed behavior and `go test ./...` for
|
and optional notifier.
|
||||||
cross-package changes.
|
- Invalid pipeline IDs must return a typed or predicate-detectable error
|
||||||
|
suitable for later HTTP `404` mapping.
|
||||||
|
- `RunPipeline` must reuse the same backend factory, secret loading, transform
|
||||||
|
registry, warning generation, destination planning, publish execution,
|
||||||
|
notification behavior, and failure aggregation as `Run`.
|
||||||
|
- Do not add a public CLI `run --pipeline` flag in this stage.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Run only the requested pipeline from a multi-pipeline config.
|
||||||
|
- Return the chosen not-found error for an unknown pipeline ID.
|
||||||
|
- Preserve existing `app.Run` all-pipelines behavior.
|
||||||
|
- Run `go test ./internal/app ./internal/cli`.
|
||||||
|
|
||||||
|
### Stage 3: In-Memory Pipeline Run Coordinator
|
||||||
|
|
||||||
|
Add a narrow concurrency coordinator around the single-pipeline app seam.
|
||||||
|
|
||||||
|
Required behavior:
|
||||||
|
|
||||||
|
- Keep the coordinator in `internal/app` unless a later HTTP implementation
|
||||||
|
introduces a transport package.
|
||||||
|
- Allow different pipeline IDs to run concurrently.
|
||||||
|
- Reject a second in-flight run for the same pipeline ID with a typed or
|
||||||
|
predicate-detectable duplicate-run error.
|
||||||
|
- Track active runs in memory only.
|
||||||
|
- Clear active state after success, failure, or context cancellation.
|
||||||
|
- Do not queue duplicate runs and do not persist run state to disk or a
|
||||||
|
database.
|
||||||
|
|
||||||
|
Async-ready constraints:
|
||||||
|
|
||||||
|
- Model run IDs and status records so a later HTTP API can expose asynchronous
|
||||||
|
status.
|
||||||
|
- A future HTTP request context should guard admission; the actual run should
|
||||||
|
be tied to a server/coordinator lifetime context.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Concurrent same-pipeline requests produce exactly one accepted run and one
|
||||||
|
duplicate-run error.
|
||||||
|
- Concurrent different-pipeline requests both start.
|
||||||
|
- Active state is cleared after success and after failure.
|
||||||
|
- Unknown pipeline IDs do not leave active state behind.
|
||||||
|
- Run `go test ./internal/app`.
|
||||||
|
|
||||||
|
### Stage 4: Future HTTP Boundary Contract
|
||||||
|
|
||||||
|
Record the target HTTP boundary in this roadmap only; do not implement routes
|
||||||
|
yet.
|
||||||
|
|
||||||
|
Required target contract:
|
||||||
|
|
||||||
|
- Future trigger endpoint should accept only a pipeline ID.
|
||||||
|
- Future trigger behavior should be asynchronous: accept the run, return a run
|
||||||
|
ID, and expose status through a later status endpoint.
|
||||||
|
- Future duplicate in-flight pipeline runs should map to `409 Conflict`.
|
||||||
|
- Future unknown pipeline IDs should map to `404 Not Found`.
|
||||||
|
- Future server should default to private binding, such as `127.0.0.1`, and
|
||||||
|
rely on a reverse proxy, private network, or external mTLS for transport
|
||||||
|
security.
|
||||||
|
- Do not add bearer-token auth, in-app TLS config, or public-network exposure in
|
||||||
|
v1 unless a later roadmap explicitly changes this decision.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Documentation-only stage; no tests required beyond any repository doc checks
|
||||||
|
that exist.
|
||||||
|
|
||||||
|
### Stage 5: Stabilization Sweep
|
||||||
|
|
||||||
|
Clean up only code made obsolete by stages 1-3.
|
||||||
|
|
||||||
|
Required behavior:
|
||||||
|
|
||||||
|
- Remove duplicate result projection, warning collection, and
|
||||||
|
pipeline-selection helpers only when clearly replaced.
|
||||||
|
- Keep test helpers local unless they are broadly reusable.
|
||||||
|
- Update `docs/internal/app.md` only for implemented internal app contracts.
|
||||||
|
- Do not update `docs/cli.md`, `docs/config.md`, `docs/operations.md`, or
|
||||||
|
README for HTTP behavior because no HTTP API exists yet.
|
||||||
|
|
||||||
|
Tests:
|
||||||
|
|
||||||
|
- Run `go test ./internal/app ./internal/cli`.
|
||||||
|
- Run `go test ./...` before considering the roadmap complete.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
The roadmap is complete when:
|
||||||
|
|
||||||
|
- Existing `distributor run` behavior is unchanged.
|
||||||
|
- `internal/app` exposes a tested single-pipeline run path.
|
||||||
|
- Run results are available as structured data without scraping stdout.
|
||||||
|
- The coordinator prevents concurrent same-pipeline runs while allowing
|
||||||
|
different pipelines to run.
|
||||||
|
- Future HTTP status, duplicate, not-found, and private-bind decisions are
|
||||||
|
recorded under `docs/roadmap/`.
|
||||||
|
- The full test suite passes with `go test ./...`.
|
||||||
|
|||||||
Reference in New Issue
Block a user