eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Refresh app and HTTP boundary documentation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-03 11:57:37 +00:00
parent 00677148e2
commit 22ce15c707
2 changed files with 188 additions and 232 deletions
Download Patch File Download Diff File Expand all files Collapse all files

180
docs/internal/app.md
View File

@@ -2,84 +2,141 @@
## Purpose ## Purpose
`internal/app` owns top-level use cases for `run`, `validate`, and `inspect`. It wires configuration, storage backends, transforms, publish planning, execution, structured run reports, summaries, coordination, and notification handoff. `internal/app` owns the top-level application use cases. It coordinates
configuration loading, secret resolution, backend construction, source bundle
discovery, destination selection, publish planning, publish execution,
notification handoff, run reporting, and in-memory run coordination.
## Inputs and outputs The package is the boundary between callers and lower-level domain packages. It
does not own manifest validation rules, destination state comparison, storage
path rules, output planning, transform rendering, or backend-specific behavior.
`Run` accepts a context, optional config path, dry-run flag, force flag, stdout writer, output format, and optional notifier. It loads YAML config, discovers source bundles for each configured pipeline, plans each destination independently, optionally executes publish plans, builds a `RunReport`, projects that report to text or JSON when stdout is supplied, and returns an aggregated error if any destination fails. ## Use Cases
`RunPipeline` accepts a context, config path, pipeline ID, dry-run flag, force flag, and optional notifier. It runs exactly one configured pipeline and returns the same `RunReport` model without writing command output. Unknown pipeline IDs return `PipelineNotFoundError`, detectable with `IsPipelineNotFound`. `Run` is the CLI-facing all-pipeline entrypoint. It accepts a context, optional
config path, dry-run flag, force flag, stdout writer, output format, and
optional notifier. It runs every configured pipeline, builds a `RunReport`, and
projects the report to text or JSON when stdout is supplied.
`PipelineRunCoordinator` wraps `RunPipeline` with in-memory admission control. It returns `PipelineRunRecord` values containing run ID, pipeline ID, status, timestamps, report, and error text. Duplicate in-flight runs for the same pipeline ID return `DuplicatePipelineRunError`, detectable with `IsDuplicatePipelineRun`. `RunPipeline` is the app-layer single-pipeline entrypoint. It accepts a context,
config path, pipeline ID, dry-run flag, force flag, and optional notifier. It
loads the same config as `Run`, narrows execution to exactly one configured
pipeline, and returns a `RunReport` without writing command output.
`Validate` and `Inspect` accept either a local path or one configured pipeline source. `Validate` discovers and validates bundles. `Inspect` writes bundle metadata and manifest file entries to stdout when provided. `Validate` and `Inspect` accept either a local path or one configured pipeline
source. They share source backend construction with run workflows and never open
destination backends.
## Run flow ## Run Reports
The all-pipeline runner: `RunReport` is the structured result model for run workflows. It includes
dry-run state, pipeline summaries, action records, output metadata, summary
counters, warnings, and destination-scoped output errors.
Text and JSON run output are projections of `RunReport`. JSON tags on report
records match the CLI JSON output contract. Text output preserves the CLI
summary shape while keeping output rendering outside the core planning and
execution loop.
Destination-scoped failures produce a report plus an aggregated error. Fatal
setup failures, such as config loading, source open, or source discovery
failures, return before a complete run report is available.
## Run Flow
The app runner:
1. loads config from the supplied path or `config.DefaultConfigPath`; 1. loads config from the supplied path or `config.DefaultConfigPath`;
2. opens the configured source backend; 2. loads configured secret files into a config-owned environment resolver;
3. discovers validated bundles from the source root; 3. builds the app-level backend factory and transform registry;
4. selects source bundles for each destination according to destination path mapping; 4. opens each selected pipeline source backend;
5. opens each destination backend independently; 5. discovers validated source bundles from the source root;
6. builds publish plans for the selected bundle and destination combinations; 6. selects source bundles for each destination according to path mapping;
7. records warnings, action records, output metadata, and summary counters in a `RunReport`; 7. opens destination backends independently;
8. executes publish or replacement plans unless dry-run is enabled; 8. builds publish plans for selected bundle and destination combinations;
9. invokes the notifier after successful publish or replacement actions; 9. records warnings, action records, output metadata, and summary counters;
10. projects the completed report to text or JSON output. 10. executes publish or replacement plans unless dry-run is enabled;
11. invokes the notifier after successful publish or replacement actions;
12. returns the structured report and any aggregated destination failures.
Destination failures are collected while later destinations continue to run. Source open and source discovery failures stop the run because there are no valid bundles to fan out. `RunPipeline` follows the same flow after selecting a single configured
pipeline. It uses the same backend factory, secret loading, transform registry,
warning generation, destination planning, publish execution, notification
behavior, and failure aggregation as `Run`.
`RunPipeline` uses the same config loading, secret loading, backend factory, transform registry, warning generation, destination planning, publish execution, notification behavior, and failure aggregation as `Run`, but first narrows the loaded config to the requested pipeline. ## Coordination
## Run implementation `PipelineRunCoordinator` wraps `RunPipeline` with in-memory admission control.
It allows different pipeline IDs to run concurrently and rejects a second active
run for the same pipeline ID.
`run.go` contains the public `Run` and `RunPipeline` entrypoints and the main configuration orchestration paths. Package-local run helpers are grouped by responsibility: Coordinator records contain a run ID, pipeline ID, status, timestamps, completed
report, and error text when applicable. Active state is memory-only and is
cleared after success, failure, unknown pipeline ID, or context cancellation.
- `run_selection.go`: destination bundle selection, path mapping decisions, and fixed-path warnings; The admission context is checked before a run is accepted. Once accepted, the
- `run_warnings.go`: secret and SSH warning data; run uses the coordinator lifetime context, so caller cancellation can stop
- `run_output.go`: `RunReport`, action/output records, and text/JSON report projection; waiting for admission without owning the actual run lifetime.
- `run_summary.go`: summary counters and JSON summary records;
- `run_failures.go`: destination failure aggregation and partial-result detection; The coordinator does not queue duplicate runs, persist run records, or define
transport endpoints.
## Errors
`Run` returns immediately for config loading errors, context cancellation before
work starts, source open errors, and source discovery errors.
`RunPipeline` returns `PipelineNotFoundError` when the requested pipeline ID is
not configured. Callers can detect that condition with `IsPipelineNotFound`.
Per-destination backend, planning, execution, and notification errors are
aggregated into one run error after remaining destinations have been attempted.
Destination diagnostics include pipeline ID, destination ID, backend, and
bundle path.
`PipelineRunCoordinator` returns `DuplicatePipelineRunError` when the same
pipeline already has an active run. Callers can detect that condition with
`IsDuplicatePipelineRun`.
Stdout write errors are returned immediately because the caller's requested
output stream can no longer be trusted.
## Package Layout
Run helpers are grouped by responsibility:
- `run.go`: `Run`, `RunPipeline`, and shared run orchestration.
- `run_output.go`: `RunReport`, action/output records, and text/JSON report projection.
- `run_summary.go`: summary counters.
- `run_failures.go`: destination failure aggregation and partial-result detection.
- `run_selection.go`: destination bundle selection, path mapping decisions, and fixed-path warnings.
- `run_warnings.go`: secret and SSH warning records.
- `run_notify.go`: notification event projection and action filtering. - `run_notify.go`: notification event projection and action filtering.
- `run_coordinator.go`: in-memory single-pipeline run admission, run IDs, status records, and duplicate-run errors. - `run_coordinator.go`: in-memory run admission, run IDs, status records, and duplicate-run errors.
- `backends.go`: app-level backend factory wiring.
- `transforms.go`: app-level transform registry wiring.
- `source_select.go`: configured-source selection shared by `validate` and `inspect`.
These helpers remain in `internal/app` because command output, warning collection, destination failure aggregation, notifier handoff, and backend construction are app-owned orchestration concerns. ## Backend And Transform Wiring
## Run coordination The app-level backend factory registers local, SSH, and S3 backends for runtime
execution. Source and destination backend config is converted through a shared
app-local open spec before adapter construction.
`PipelineRunCoordinator` keeps active run state in memory only. It allows different pipeline IDs to run concurrently and rejects a second active run for the same pipeline ID. Active state is cleared after success, destination-scoped failure, source/config failure, unknown pipeline ID, or context cancellation. Credential references are resolved through the config environment resolver.
Production app code must not read backend credential environment variables
directly.
The admission context is checked before a run is admitted. Once admitted, execution uses the coordinator lifetime context so future transport request cancellation can stop waiting for admission without owning the actual run lifetime. The app-level transform registry registers Markdown-to-HTML through
`internal/transform/markdown`. Lower-level publish code receives a resolver and
does not import concrete transform implementations.
## Backend and transform wiring ## Dry-Run Behavior
The app-level backend factory registers local, SSH, and S3 backends for execution. Source and destination backend config is converted through a shared app-local open spec before adapter construction. S3 explicit credential references are resolved through the config environment resolver. Dry-run loads config, opens backends, discovers bundles, inspects destinations,
resolves transforms, and builds publish plans. It does not write destination
The app-level transform registry registers Markdown-to-HTML using `internal/transform/markdown`. Lower-level publish code receives a resolver and does not import concrete transform implementations. outputs, write `.distributor.json`, delete managed outputs, perform forced
prefix deletion, or invoke notifications.
## Dry-run behavior
Dry-run still loads config, opens backends, discovers bundles, inspects destinations, resolves transforms, and builds publish plans. It does not write destination outputs, write `.distributor.json`, delete managed outputs, perform forced prefix deletion, or notify.
## Failure behavior
`Run` returns immediately for config loading errors, context cancellation before work starts, source open errors, and source discovery errors. `RunPipeline` also returns immediately with `PipelineNotFoundError` when the requested pipeline ID is not configured. Per-destination backend, planning, execution, and notification errors are aggregated into one run error after remaining destinations have been attempted.
Run diagnostics include pipeline id, destination id, destination backend, and bundle path for destination-scoped failures. Source open and discovery failures include the source backend.
Stdout write errors are returned immediately because the caller's requested output stream can no longer be trusted.
Coordinator duplicate-run errors are admission errors and do not start, queue, or persist a run.
## Boundaries
`internal/app` coordinates packages but does not own manifest validation rules, destination state comparison, storage path rules, output planning, transform rendering, or backend-specific filesystem behavior.
Configured-source `Validate` and `Inspect` share source backend construction with `Run` and do not open destinations.
`PipelineRunCoordinator` is an app-layer concurrency boundary only. It does not persist run records, expose HTTP routes, or define transport status endpoints.
## Tests ## Tests
@@ -89,13 +146,18 @@ Before changing app orchestration, inspect tests under:
- `internal/cli` - `internal/cli`
- `internal/publish` - `internal/publish`
Use focused app tests for report structure, single-pipeline execution,
coordinator admission, warning generation, notification behavior, and
partial-result aggregation.
## Invariants ## Invariants
- One source fans out to each destination independently. - One source fans out to each destination independently.
- Destination failures do not prevent later destinations from being planned. - Destination failures do not prevent later destinations from being planned.
- Destination-scoped failures still produce a structured report plus an aggregated error. - Destination-scoped failures still produce a structured report plus an aggregated error.
- Dry-run must not mutate destination storage or invoke notifications. - Dry-run must not mutate destination storage or invoke notifications.
- `RunPipeline` must use the same core run path as `Run` after pipeline selection. - `RunPipeline` must use the same run path as `Run` after pipeline selection.
- Duplicate in-flight runs are rejected only for the same pipeline ID; different pipeline IDs may run concurrently. - Duplicate in-flight runs are rejected only for the same pipeline ID.
- Different pipeline IDs may run concurrently.
- Concrete backend and transform registration stays at the app layer. - Concrete backend and transform registration stays at the app layer.
- The default notifier is `notify.Noop`. - The default notifier is `notify.Noop`.

240
docs/roadmap/implementation.md
View File

@@ -1,197 +1,91 @@
# HTTP API Stabilization Roadmap # HTTP API Boundary Roadmap
## Summary ## Purpose
Prepare `distributor` for a later HTTP API without implementing the HTTP This roadmap records the accepted boundary for a future HTTP API. The current
server in this roadmap. The goal is to make the existing run workflow callable application exposes CLI commands and internal app-layer run contracts; it does
as a single-pipeline, structured, concurrency-safe application use case. not implement an HTTP server, HTTP routes, a `serve` command, app-level
authentication, or in-app TLS.
Chosen defaults: Implemented internal run contracts are documented in `docs/internal/app.md`.
This file is the canonical home for future HTTP boundary decisions until an
HTTP implementation roadmap replaces it.
- Scope: stabilization only; HTTP routes and `serve` command are deferred to a ## Accepted Direction
later roadmap.
- Future HTTP trigger mode: asynchronous start with run status.
- Security boundary: private bind or reverse proxy/mTLS outside the app; no
app-level auth or in-app TLS in v1.
- Trigger input: pipeline ID only.
## Future HTTP Boundary Contract The future HTTP API should trigger configured distributor pipelines through the
existing app-layer single-pipeline run path and in-memory coordinator.
This roadmap records the target boundary for a later HTTP implementation. No The HTTP API is intentionally narrow:
HTTP server, routes, `serve` command, app-level authentication, or in-app TLS is
implemented as part of this roadmap.
Future trigger behavior: - A trigger request accepts only a pipeline ID as application input.
- A trigger request starts work asynchronously and returns a run ID after
- The trigger endpoint accepts exactly one application input: pipeline ID. admission.
- The trigger is asynchronous. A successful admission starts a run and returns a - Run status is read through a separate status endpoint keyed by run ID.
run ID rather than waiting for publication to finish. - Status records expose run ID, pipeline ID, current status, timestamps, and
- Run status is exposed through a later status endpoint keyed by run ID. Status
records should expose the run ID, pipeline ID, current status, timestamps, and
completed report or error details when available. completed report or error details when available.
- Duplicate in-flight runs for the same pipeline ID map to `409 Conflict`.
The application remains a bundle distribution tool. The HTTP API must not turn
`distributor` into a workflow engine, CMS, report generator, or public web
authoring service.
## Error Mapping
Future transport code should map app-layer errors without changing app-layer
error ownership:
- Unknown pipeline IDs map to `404 Not Found`. - Unknown pipeline IDs map to `404 Not Found`.
- Duplicate in-flight runs for the same pipeline ID map to `409 Conflict`.
- Validation, config, source, destination, publish, transform, and notification
errors map to transport errors according to their app-layer context.
Future runtime and security boundaries: Duplicate runs must not be queued. Run state remains in memory unless a later
roadmap explicitly adds durable run storage.
- Request context guards admission. Once admitted, the actual run is tied to the ## Context And Lifetime
server or coordinator lifetime context, not to the client request lifetime.
- The server defaults to private binding, such as `127.0.0.1`.
- Operators should expose the server through a reverse proxy, private network,
or external mTLS when transport security or remote access is required.
- The first HTTP implementation does not include bearer-token authentication,
in-app TLS configuration, or public-network exposure unless a later roadmap
explicitly changes that decision.
## Implementation Rules The request context guards admission. Once a run is admitted, execution is tied
to the server or coordinator lifetime context rather than to the client request
lifetime.
- Implement stages in order; each stage should be one prompt or commit unless This split allows a client disconnect or request timeout to stop waiting for
trivially small. admission without canceling a run that has already been accepted.
- Before each stage, read `docs/policy/architecture.md`,
`docs/policy/development.md`, and `docs/policy/documentation.md`.
- Preserve existing CLI behavior, text output, JSON output, config semantics,
backend behavior, manifest schema, and destination state schema.
- Do not introduce external dependencies, a CLI framework, a generic workflow
engine, a plugin system, durable run storage, or broad adapter abstractions.
- Do not document the HTTP API as implemented outside `docs/roadmap/`.
## Stages ## Security Boundary
### Stage 1: Structured Run Core The first HTTP server should default to private binding, such as `127.0.0.1`.
Operators should expose it through a reverse proxy, private network, or external
mTLS when transport security or remote access is required.
Refactor `internal/app` so the run workflow first produces a structured run The first HTTP implementation should not include:
report, then projects that report to text or JSON output.
Required behavior: - bearer-token authentication;
- in-app TLS configuration;
- public-network exposure defaults.
- Keep `app.Run` as the CLI-facing entrypoint. A later roadmap must explicitly change this security decision before any of
- Move stdout writes out of the core planning/execution loop where practical. those features are added.
- Preserve the existing `run --format text` and `run --format json` output
byte-for-byte except where tests already allow map ordering.
- Preserve partial-result behavior: destination-scoped failures produce a
structured report plus an aggregated error.
Suggested internal shape: ## Non-Goals
- Introduce an app-owned `RunReport` model containing dry-run state, pipeline The future HTTP API should not add:
summaries, action records, summary counters, warnings, and output errors.
- Keep JSON tags compatible with the current `run` JSON output.
- Keep failure aggregation in `internal/app`; do not move it into `publish`,
`config`, or storage adapters.
Tests: - public CLI flags for selecting one pipeline during `distributor run`;
- a CLI framework;
- a generic workflow engine;
- plugin execution;
- durable run storage;
- app-level authentication;
- in-app TLS.
- Add or update `internal/app` tests proving structured reports include ## Verification Expectations
warnings, actions, outputs, summary counters, and partial failures.
- Add CLI regression coverage for existing text and JSON output.
- Run `go test ./internal/app ./internal/cli`.
### Stage 2: Single-Pipeline Run Entry Point Any later HTTP implementation should preserve existing CLI behavior and keep
the app-layer run contracts tested. At minimum, it should cover:
Add an app-layer entrypoint for running exactly one configured pipeline by ID. - trigger requests with known and unknown pipeline IDs;
- duplicate in-flight trigger requests;
Required behavior: - asynchronous acceptance and status lookup;
- private bind defaults;
- Add an exported internal app function such as - request-context admission behavior;
`RunPipeline(ctx, RunPipelineOptions) (RunReport, error)`. - coordinator-lifetime run execution.
- `RunPipelineOptions` must include config path, pipeline ID, dry-run, force,
and optional notifier.
- Invalid pipeline IDs must return a typed or predicate-detectable error
suitable for later HTTP `404` mapping.
- `RunPipeline` must reuse the same backend factory, secret loading, transform
registry, warning generation, destination planning, publish execution,
notification behavior, and failure aggregation as `Run`.
- Do not add a public CLI `run --pipeline` flag in this stage.
Tests:
- Run only the requested pipeline from a multi-pipeline config.
- Return the chosen not-found error for an unknown pipeline ID.
- Preserve existing `app.Run` all-pipelines behavior.
- Run `go test ./internal/app ./internal/cli`.
### Stage 3: In-Memory Pipeline Run Coordinator
Add a narrow concurrency coordinator around the single-pipeline app seam.
Required behavior:
- Keep the coordinator in `internal/app` unless a later HTTP implementation
introduces a transport package.
- Allow different pipeline IDs to run concurrently.
- Reject a second in-flight run for the same pipeline ID with a typed or
predicate-detectable duplicate-run error.
- Track active runs in memory only.
- Clear active state after success, failure, or context cancellation.
- Do not queue duplicate runs and do not persist run state to disk or a
database.
Async-ready constraints:
- Model run IDs and status records so a later HTTP API can expose asynchronous
status.
- A future HTTP request context should guard admission; the actual run should
be tied to a server/coordinator lifetime context.
Tests:
- Concurrent same-pipeline requests produce exactly one accepted run and one
duplicate-run error.
- Concurrent different-pipeline requests both start.
- Active state is cleared after success and after failure.
- Unknown pipeline IDs do not leave active state behind.
- Run `go test ./internal/app`.
### Stage 4: Future HTTP Boundary Contract
Record the target HTTP boundary in this roadmap only; do not implement routes
yet.
Required target contract:
- Future trigger endpoint should accept only a pipeline ID.
- Future trigger behavior should be asynchronous: accept the run, return a run
ID, and expose status through a later status endpoint.
- Future duplicate in-flight pipeline runs should map to `409 Conflict`.
- Future unknown pipeline IDs should map to `404 Not Found`.
- Future server should default to private binding, such as `127.0.0.1`, and
rely on a reverse proxy, private network, or external mTLS for transport
security.
- Do not add bearer-token auth, in-app TLS config, or public-network exposure in
v1 unless a later roadmap explicitly changes this decision.
Tests:
- Documentation-only stage; no tests required beyond any repository doc checks
that exist.
### Stage 5: Stabilization Sweep
Clean up only code made obsolete by stages 1-3.
Required behavior:
- Remove duplicate result projection, warning collection, and
pipeline-selection helpers only when clearly replaced.
- Keep test helpers local unless they are broadly reusable.
- Update `docs/internal/app.md` only for implemented internal app contracts.
- Do not update `docs/cli.md`, `docs/config.md`, `docs/operations.md`, or
README for HTTP behavior because no HTTP API exists yet.
Tests:
- Run `go test ./internal/app ./internal/cli`.
- Run `go test ./...` before considering the roadmap complete.
## Acceptance Criteria
The roadmap is complete when:
- Existing `distributor run` behavior is unchanged.
- `internal/app` exposes a tested single-pipeline run path.
- Run results are available as structured data without scraping stdout.
- The coordinator prevents concurrent same-pipeline runs while allowing
different pipelines to run.
- Future HTTP status, duplicate, not-found, and private-bind decisions are
recorded under `docs/roadmap/`.
- The full test suite passes with `go test ./...`.