Refresh app and HTTP boundary documentation
This commit is contained in:
@@ -2,84 +2,141 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/app` owns top-level use cases for `run`, `validate`, and `inspect`. It wires configuration, storage backends, transforms, publish planning, execution, structured run reports, summaries, coordination, and notification handoff.
|
||||
`internal/app` owns the top-level application use cases. It coordinates
|
||||
configuration loading, secret resolution, backend construction, source bundle
|
||||
discovery, destination selection, publish planning, publish execution,
|
||||
notification handoff, run reporting, and in-memory run coordination.
|
||||
|
||||
## Inputs and outputs
|
||||
The package is the boundary between callers and lower-level domain packages. It
|
||||
does not own manifest validation rules, destination state comparison, storage
|
||||
path rules, output planning, transform rendering, or backend-specific behavior.
|
||||
|
||||
`Run` accepts a context, optional config path, dry-run flag, force flag, stdout writer, output format, and optional notifier. It loads YAML config, discovers source bundles for each configured pipeline, plans each destination independently, optionally executes publish plans, builds a `RunReport`, projects that report to text or JSON when stdout is supplied, and returns an aggregated error if any destination fails.
|
||||
## Use Cases
|
||||
|
||||
`RunPipeline` accepts a context, config path, pipeline ID, dry-run flag, force flag, and optional notifier. It runs exactly one configured pipeline and returns the same `RunReport` model without writing command output. Unknown pipeline IDs return `PipelineNotFoundError`, detectable with `IsPipelineNotFound`.
|
||||
`Run` is the CLI-facing all-pipeline entrypoint. It accepts a context, optional
|
||||
config path, dry-run flag, force flag, stdout writer, output format, and
|
||||
optional notifier. It runs every configured pipeline, builds a `RunReport`, and
|
||||
projects the report to text or JSON when stdout is supplied.
|
||||
|
||||
`PipelineRunCoordinator` wraps `RunPipeline` with in-memory admission control. It returns `PipelineRunRecord` values containing run ID, pipeline ID, status, timestamps, report, and error text. Duplicate in-flight runs for the same pipeline ID return `DuplicatePipelineRunError`, detectable with `IsDuplicatePipelineRun`.
|
||||
`RunPipeline` is the app-layer single-pipeline entrypoint. It accepts a context,
|
||||
config path, pipeline ID, dry-run flag, force flag, and optional notifier. It
|
||||
loads the same config as `Run`, narrows execution to exactly one configured
|
||||
pipeline, and returns a `RunReport` without writing command output.
|
||||
|
||||
`Validate` and `Inspect` accept either a local path or one configured pipeline source. `Validate` discovers and validates bundles. `Inspect` writes bundle metadata and manifest file entries to stdout when provided.
|
||||
`Validate` and `Inspect` accept either a local path or one configured pipeline
|
||||
source. They share source backend construction with run workflows and never open
|
||||
destination backends.
|
||||
|
||||
## Run flow
|
||||
## Run Reports
|
||||
|
||||
The all-pipeline runner:
|
||||
`RunReport` is the structured result model for run workflows. It includes
|
||||
dry-run state, pipeline summaries, action records, output metadata, summary
|
||||
counters, warnings, and destination-scoped output errors.
|
||||
|
||||
Text and JSON run output are projections of `RunReport`. JSON tags on report
|
||||
records match the CLI JSON output contract. Text output preserves the CLI
|
||||
summary shape while keeping output rendering outside the core planning and
|
||||
execution loop.
|
||||
|
||||
Destination-scoped failures produce a report plus an aggregated error. Fatal
|
||||
setup failures, such as config loading, source open, or source discovery
|
||||
failures, return before a complete run report is available.
|
||||
|
||||
## Run Flow
|
||||
|
||||
The app runner:
|
||||
|
||||
1. loads config from the supplied path or `config.DefaultConfigPath`;
|
||||
2. opens the configured source backend;
|
||||
3. discovers validated bundles from the source root;
|
||||
4. selects source bundles for each destination according to destination path mapping;
|
||||
5. opens each destination backend independently;
|
||||
6. builds publish plans for the selected bundle and destination combinations;
|
||||
7. records warnings, action records, output metadata, and summary counters in a `RunReport`;
|
||||
8. executes publish or replacement plans unless dry-run is enabled;
|
||||
9. invokes the notifier after successful publish or replacement actions;
|
||||
10. projects the completed report to text or JSON output.
|
||||
2. loads configured secret files into a config-owned environment resolver;
|
||||
3. builds the app-level backend factory and transform registry;
|
||||
4. opens each selected pipeline source backend;
|
||||
5. discovers validated source bundles from the source root;
|
||||
6. selects source bundles for each destination according to path mapping;
|
||||
7. opens destination backends independently;
|
||||
8. builds publish plans for selected bundle and destination combinations;
|
||||
9. records warnings, action records, output metadata, and summary counters;
|
||||
10. executes publish or replacement plans unless dry-run is enabled;
|
||||
11. invokes the notifier after successful publish or replacement actions;
|
||||
12. returns the structured report and any aggregated destination failures.
|
||||
|
||||
Destination failures are collected while later destinations continue to run. Source open and source discovery failures stop the run because there are no valid bundles to fan out.
|
||||
`RunPipeline` follows the same flow after selecting a single configured
|
||||
pipeline. It uses the same backend factory, secret loading, transform registry,
|
||||
warning generation, destination planning, publish execution, notification
|
||||
behavior, and failure aggregation as `Run`.
|
||||
|
||||
`RunPipeline` uses the same config loading, secret loading, backend factory, transform registry, warning generation, destination planning, publish execution, notification behavior, and failure aggregation as `Run`, but first narrows the loaded config to the requested pipeline.
|
||||
## Coordination
|
||||
|
||||
## Run implementation
|
||||
`PipelineRunCoordinator` wraps `RunPipeline` with in-memory admission control.
|
||||
It allows different pipeline IDs to run concurrently and rejects a second active
|
||||
run for the same pipeline ID.
|
||||
|
||||
`run.go` contains the public `Run` and `RunPipeline` entrypoints and the main configuration orchestration paths. Package-local run helpers are grouped by responsibility:
|
||||
Coordinator records contain a run ID, pipeline ID, status, timestamps, completed
|
||||
report, and error text when applicable. Active state is memory-only and is
|
||||
cleared after success, failure, unknown pipeline ID, or context cancellation.
|
||||
|
||||
- `run_selection.go`: destination bundle selection, path mapping decisions, and fixed-path warnings;
|
||||
- `run_warnings.go`: secret and SSH warning data;
|
||||
- `run_output.go`: `RunReport`, action/output records, and text/JSON report projection;
|
||||
- `run_summary.go`: summary counters and JSON summary records;
|
||||
- `run_failures.go`: destination failure aggregation and partial-result detection;
|
||||
The admission context is checked before a run is accepted. Once accepted, the
|
||||
run uses the coordinator lifetime context, so caller cancellation can stop
|
||||
waiting for admission without owning the actual run lifetime.
|
||||
|
||||
The coordinator does not queue duplicate runs, persist run records, or define
|
||||
transport endpoints.
|
||||
|
||||
## Errors
|
||||
|
||||
`Run` returns immediately for config loading errors, context cancellation before
|
||||
work starts, source open errors, and source discovery errors.
|
||||
|
||||
`RunPipeline` returns `PipelineNotFoundError` when the requested pipeline ID is
|
||||
not configured. Callers can detect that condition with `IsPipelineNotFound`.
|
||||
|
||||
Per-destination backend, planning, execution, and notification errors are
|
||||
aggregated into one run error after remaining destinations have been attempted.
|
||||
Destination diagnostics include pipeline ID, destination ID, backend, and
|
||||
bundle path.
|
||||
|
||||
`PipelineRunCoordinator` returns `DuplicatePipelineRunError` when the same
|
||||
pipeline already has an active run. Callers can detect that condition with
|
||||
`IsDuplicatePipelineRun`.
|
||||
|
||||
Stdout write errors are returned immediately because the caller's requested
|
||||
output stream can no longer be trusted.
|
||||
|
||||
## Package Layout
|
||||
|
||||
Run helpers are grouped by responsibility:
|
||||
|
||||
- `run.go`: `Run`, `RunPipeline`, and shared run orchestration.
|
||||
- `run_output.go`: `RunReport`, action/output records, and text/JSON report projection.
|
||||
- `run_summary.go`: summary counters.
|
||||
- `run_failures.go`: destination failure aggregation and partial-result detection.
|
||||
- `run_selection.go`: destination bundle selection, path mapping decisions, and fixed-path warnings.
|
||||
- `run_warnings.go`: secret and SSH warning records.
|
||||
- `run_notify.go`: notification event projection and action filtering.
|
||||
- `run_coordinator.go`: in-memory single-pipeline run admission, run IDs, status records, and duplicate-run errors.
|
||||
- `run_coordinator.go`: in-memory run admission, run IDs, status records, and duplicate-run errors.
|
||||
- `backends.go`: app-level backend factory wiring.
|
||||
- `transforms.go`: app-level transform registry wiring.
|
||||
- `source_select.go`: configured-source selection shared by `validate` and `inspect`.
|
||||
|
||||
These helpers remain in `internal/app` because command output, warning collection, destination failure aggregation, notifier handoff, and backend construction are app-owned orchestration concerns.
|
||||
## Backend And Transform Wiring
|
||||
|
||||
## Run coordination
|
||||
The app-level backend factory registers local, SSH, and S3 backends for runtime
|
||||
execution. Source and destination backend config is converted through a shared
|
||||
app-local open spec before adapter construction.
|
||||
|
||||
`PipelineRunCoordinator` keeps active run state in memory only. It allows different pipeline IDs to run concurrently and rejects a second active run for the same pipeline ID. Active state is cleared after success, destination-scoped failure, source/config failure, unknown pipeline ID, or context cancellation.
|
||||
Credential references are resolved through the config environment resolver.
|
||||
Production app code must not read backend credential environment variables
|
||||
directly.
|
||||
|
||||
The admission context is checked before a run is admitted. Once admitted, execution uses the coordinator lifetime context so future transport request cancellation can stop waiting for admission without owning the actual run lifetime.
|
||||
The app-level transform registry registers Markdown-to-HTML through
|
||||
`internal/transform/markdown`. Lower-level publish code receives a resolver and
|
||||
does not import concrete transform implementations.
|
||||
|
||||
## Backend and transform wiring
|
||||
## Dry-Run Behavior
|
||||
|
||||
The app-level backend factory registers local, SSH, and S3 backends for execution. Source and destination backend config is converted through a shared app-local open spec before adapter construction. S3 explicit credential references are resolved through the config environment resolver.
|
||||
|
||||
The app-level transform registry registers Markdown-to-HTML using `internal/transform/markdown`. Lower-level publish code receives a resolver and does not import concrete transform implementations.
|
||||
|
||||
## Dry-run behavior
|
||||
|
||||
Dry-run still loads config, opens backends, discovers bundles, inspects destinations, resolves transforms, and builds publish plans. It does not write destination outputs, write `.distributor.json`, delete managed outputs, perform forced prefix deletion, or notify.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
`Run` returns immediately for config loading errors, context cancellation before work starts, source open errors, and source discovery errors. `RunPipeline` also returns immediately with `PipelineNotFoundError` when the requested pipeline ID is not configured. Per-destination backend, planning, execution, and notification errors are aggregated into one run error after remaining destinations have been attempted.
|
||||
|
||||
Run diagnostics include pipeline id, destination id, destination backend, and bundle path for destination-scoped failures. Source open and discovery failures include the source backend.
|
||||
|
||||
Stdout write errors are returned immediately because the caller's requested output stream can no longer be trusted.
|
||||
|
||||
Coordinator duplicate-run errors are admission errors and do not start, queue, or persist a run.
|
||||
|
||||
## Boundaries
|
||||
|
||||
`internal/app` coordinates packages but does not own manifest validation rules, destination state comparison, storage path rules, output planning, transform rendering, or backend-specific filesystem behavior.
|
||||
|
||||
Configured-source `Validate` and `Inspect` share source backend construction with `Run` and do not open destinations.
|
||||
|
||||
`PipelineRunCoordinator` is an app-layer concurrency boundary only. It does not persist run records, expose HTTP routes, or define transport status endpoints.
|
||||
Dry-run loads config, opens backends, discovers bundles, inspects destinations,
|
||||
resolves transforms, and builds publish plans. It does not write destination
|
||||
outputs, write `.distributor.json`, delete managed outputs, perform forced
|
||||
prefix deletion, or invoke notifications.
|
||||
|
||||
## Tests
|
||||
|
||||
@@ -89,13 +146,18 @@ Before changing app orchestration, inspect tests under:
|
||||
- `internal/cli`
|
||||
- `internal/publish`
|
||||
|
||||
Use focused app tests for report structure, single-pipeline execution,
|
||||
coordinator admission, warning generation, notification behavior, and
|
||||
partial-result aggregation.
|
||||
|
||||
## Invariants
|
||||
|
||||
- One source fans out to each destination independently.
|
||||
- Destination failures do not prevent later destinations from being planned.
|
||||
- Destination-scoped failures still produce a structured report plus an aggregated error.
|
||||
- Dry-run must not mutate destination storage or invoke notifications.
|
||||
- `RunPipeline` must use the same core run path as `Run` after pipeline selection.
|
||||
- Duplicate in-flight runs are rejected only for the same pipeline ID; different pipeline IDs may run concurrently.
|
||||
- `RunPipeline` must use the same run path as `Run` after pipeline selection.
|
||||
- Duplicate in-flight runs are rejected only for the same pipeline ID.
|
||||
- Different pipeline IDs may run concurrently.
|
||||
- Concrete backend and transform registration stays at the app layer.
|
||||
- The default notifier is `notify.Noop`.
|
||||
|
||||
Reference in New Issue
Block a user