distributor/docs/roadmap/implementation.md

HTTP API Stabilization Roadmap

Summary

Prepare distributor for a later HTTP API without implementing the HTTP server in this roadmap. The goal is to make the existing run workflow callable as a single-pipeline, structured, concurrency-safe application use case.

Chosen defaults:

• Scope: stabilization only; HTTP routes and serve command are deferred to a later roadmap.
• Future HTTP trigger mode: asynchronous start with run status.
• Security boundary: private bind or reverse proxy/mTLS outside the app; no app-level auth or in-app TLS in v1.
• Trigger input: pipeline ID only.

Implementation Rules

• Implement stages in order; each stage should be one prompt or commit unless trivially small.
• Before each stage, read docs/policy/architecture.md, docs/policy/development.md, and docs/policy/documentation.md.
• Preserve existing CLI behavior, text output, JSON output, config semantics, backend behavior, manifest schema, and destination state schema.
• Do not introduce external dependencies, a CLI framework, a generic workflow engine, a plugin system, durable run storage, or broad adapter abstractions.
• Do not document the HTTP API as implemented outside docs/roadmap/.

Stages

Stage 1: Structured Run Core

Refactor internal/app so the run workflow first produces a structured run report, then projects that report to text or JSON output.

Required behavior:

• Keep app.Run as the CLI-facing entrypoint.
• Move stdout writes out of the core planning/execution loop where practical.
• Preserve the existing run --format text and run --format json output byte-for-byte except where tests already allow map ordering.
• Preserve partial-result behavior: destination-scoped failures produce a structured report plus an aggregated error.

Suggested internal shape:

• Introduce an app-owned RunReport model containing dry-run state, pipeline summaries, action records, summary counters, warnings, and output errors.
• Keep JSON tags compatible with the current run JSON output.
• Keep failure aggregation in internal/app; do not move it into publish, config, or storage adapters.

Tests:

• Add or update internal/app tests proving structured reports include warnings, actions, outputs, summary counters, and partial failures.
• Add CLI regression coverage for existing text and JSON output.
• Run go test ./internal/app ./internal/cli.

Stage 2: Single-Pipeline Run Entry Point

Add an app-layer entrypoint for running exactly one configured pipeline by ID.

Required behavior:

• Add an exported internal app function such as RunPipeline(ctx, RunPipelineOptions) (RunReport, error).
• RunPipelineOptions must include config path, pipeline ID, dry-run, force, and optional notifier.
• Invalid pipeline IDs must return a typed or predicate-detectable error suitable for later HTTP 404 mapping.
• RunPipeline must reuse the same backend factory, secret loading, transform registry, warning generation, destination planning, publish execution, notification behavior, and failure aggregation as Run.
• Do not add a public CLI run --pipeline flag in this stage.

Tests:

• Run only the requested pipeline from a multi-pipeline config.
• Return the chosen not-found error for an unknown pipeline ID.
• Preserve existing app.Run all-pipelines behavior.
• Run go test ./internal/app ./internal/cli.

Stage 3: In-Memory Pipeline Run Coordinator

Add a narrow concurrency coordinator around the single-pipeline app seam.

Required behavior:

• Keep the coordinator in internal/app unless a later HTTP implementation introduces a transport package.
• Allow different pipeline IDs to run concurrently.
• Reject a second in-flight run for the same pipeline ID with a typed or predicate-detectable duplicate-run error.
• Track active runs in memory only.
• Clear active state after success, failure, or context cancellation.
• Do not queue duplicate runs and do not persist run state to disk or a database.

Async-ready constraints:

• Model run IDs and status records so a later HTTP API can expose asynchronous status.
• A future HTTP request context should guard admission; the actual run should be tied to a server/coordinator lifetime context.

Tests:

• Concurrent same-pipeline requests produce exactly one accepted run and one duplicate-run error.
• Concurrent different-pipeline requests both start.
• Active state is cleared after success and after failure.
• Unknown pipeline IDs do not leave active state behind.
• Run go test ./internal/app.

Stage 4: Future HTTP Boundary Contract

Record the target HTTP boundary in this roadmap only; do not implement routes yet.

Required target contract:

• Future trigger endpoint should accept only a pipeline ID.
• Future trigger behavior should be asynchronous: accept the run, return a run ID, and expose status through a later status endpoint.
• Future duplicate in-flight pipeline runs should map to 409 Conflict.
• Future unknown pipeline IDs should map to 404 Not Found.
• Future server should default to private binding, such as 127.0.0.1, and rely on a reverse proxy, private network, or external mTLS for transport security.
• Do not add bearer-token auth, in-app TLS config, or public-network exposure in v1 unless a later roadmap explicitly changes this decision.

Tests:

• Documentation-only stage; no tests required beyond any repository doc checks that exist.

Stage 5: Stabilization Sweep

Clean up only code made obsolete by stages 1-3.

Required behavior:

• Remove duplicate result projection, warning collection, and pipeline-selection helpers only when clearly replaced.
• Keep test helpers local unless they are broadly reusable.
• Update docs/internal/app.md only for implemented internal app contracts.
• Do not update docs/cli.md, docs/config.md, docs/operations.md, or README for HTTP behavior because no HTTP API exists yet.

Tests:

• Run go test ./internal/app ./internal/cli.
• Run go test ./... before considering the roadmap complete.

Acceptance Criteria

The roadmap is complete when:

• Existing distributor run behavior is unchanged.
• internal/app exposes a tested single-pipeline run path.
• Run results are available as structured data without scraping stdout.
• The coordinator prevents concurrent same-pipeline runs while allowing different pipelines to run.
• Future HTTP status, duplicate, not-found, and private-bind decisions are recorded under docs/roadmap/.
• The full test suite passes with go test ./....