Normalize internal component documentation
This commit is contained in:
@@ -1,241 +1,66 @@
|
|||||||
# Application Orchestration
|
# Application Orchestration
|
||||||
|
|
||||||
|
Audience: developers and LLM coding agents changing `internal/app`.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
`internal/app` owns the top-level application use cases. It coordinates
|
`internal/app` owns top-level application use cases: run, single-pipeline run, staged-source run, validate, inspect, manifest creation, and HTTP upload serving. It coordinates config loading, secret resolution, backend construction, source discovery, destination selection, publish planning/execution, notification handoff, output projection, and upload coordination.
|
||||||
configuration loading, secret resolution, backend construction, source bundle
|
|
||||||
discovery, destination selection, publish planning, publish execution,
|
|
||||||
notification handoff, run reporting, and upload coordination.
|
|
||||||
|
|
||||||
The package is the boundary between callers and lower-level domain packages. It
|
## Inputs And Outputs
|
||||||
does not own manifest validation rules, destination state comparison, storage
|
|
||||||
path rules, output planning, transform rendering, or backend-specific behavior.
|
|
||||||
|
|
||||||
## Use Cases
|
Inputs include app option structs, contexts, config paths, pipeline ids, local source roots, dry-run/force flags, output format, stdout writers, HTTP requests, and optional notifier implementations.
|
||||||
|
|
||||||
`Run` is the CLI-facing all-pipeline entrypoint. It accepts a context, optional
|
Outputs include `RunReport`, validate/inspect/manifest results, CLI text/JSON projections, HTTP upload responses, upload status records, and errors. Destination-scoped failures can return a partial run report plus an aggregated error; fatal setup failures return before a complete report exists.
|
||||||
config path, dry-run flag, force flag, stdout writer, output format, and
|
|
||||||
optional notifier. It runs every configured pipeline, builds a `RunReport`, and
|
|
||||||
projects the report to text or JSON when stdout is supplied.
|
|
||||||
|
|
||||||
`RunPipeline` is the app-layer single-pipeline entrypoint. It accepts a context,
|
## Boundaries
|
||||||
config path, pipeline ID, dry-run flag, force flag, and optional notifier. It
|
|
||||||
loads the same config as `Run`, narrows execution to exactly one configured
|
|
||||||
pipeline, and returns a `RunReport` without writing command output.
|
|
||||||
|
|
||||||
`RunPipelineWithLocalSource` is the app-layer single-pipeline entrypoint for an
|
`internal/app` wires packages together but does not own manifest validation rules, destination state comparison, storage path rules, publish safety policy, transform rendering, config schema validation, or backend protocol behavior.
|
||||||
already prepared local source bundle root. It accepts the same pipeline
|
|
||||||
selection and execution options as `RunPipeline` plus a local source root path.
|
|
||||||
It loads config, selects one configured pipeline, opens the supplied source
|
|
||||||
root as a local backend, validates exactly that root bundle, and then uses the
|
|
||||||
same destination fan-out path as normal runs.
|
|
||||||
|
|
||||||
`Validate` and `Inspect` accept either a local path or one configured pipeline
|
User-facing command parsing stays in `internal/cli`. User-facing config reference stays in `docs/config.md`. External contracts live under `docs/integrations/`.
|
||||||
source. Configured-source mode uses the same runtime config and secret setup as
|
|
||||||
run workflows, shares source backend construction, and never opens destination
|
|
||||||
backends.
|
|
||||||
|
|
||||||
`Serve` is the CLI-facing HTTP upload server entrypoint. It uses the app
|
## Config Fields Used
|
||||||
runtime setup, resolves upload bearer tokens for configured `http_upload`
|
|
||||||
sources, creates an `UploadCoordinator`, binds `server.http.bind`, and serves
|
|
||||||
the upload API until its context is cancelled.
|
|
||||||
|
|
||||||
## Run Reports
|
The package consumes the loaded `config.Config`: `server.http`, `secrets.directory`, pipeline ids, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, and transfer policy.
|
||||||
|
|
||||||
`RunReport` is the structured result model for run workflows. It includes
|
Config fields are validated and defaulted by `internal/config` before app workflows use them.
|
||||||
dry-run state, pipeline summaries, action records, output metadata, summary
|
|
||||||
counters, warnings, and destination-scoped output errors.
|
|
||||||
|
|
||||||
Text and JSON run output are projections of `RunReport`. JSON tags on report
|
## Adapters Used
|
||||||
records match the CLI JSON output contract. Text output preserves the CLI
|
|
||||||
summary shape while keeping output rendering outside the core planning and
|
|
||||||
execution loop.
|
|
||||||
|
|
||||||
Destination-scoped failures produce a report plus an aggregated error. Fatal
|
The app backend factory registers runtime storage adapters for local filesystem, SSH/SFTP, and S3-compatible storage. It resolves explicit credentials through the config-owned environment resolver before opening S3 backends.
|
||||||
setup failures, such as config loading, source open, or source discovery
|
|
||||||
failures, return before a complete run report is available.
|
|
||||||
|
|
||||||
## Run Flow
|
The app layer registers default transforms, including Markdown-to-HTML, and supplies a transform resolver to publish planning. It uses `notify.Noop` when no notifier is supplied.
|
||||||
|
|
||||||
The app runner:
|
## State And Manifest Behavior
|
||||||
|
|
||||||
1. builds runtime setup by resolving the config path, loading config, loading
|
Run workflows discover and validate source bundles through `internal/bundle`. Destination state is planned and written through `internal/publish` and `internal/state`; the app layer records report projections of those plans and results.
|
||||||
configured secret files, and projecting secret-conflict warnings;
|
|
||||||
2. builds the app-level backend factory from the config-owned environment
|
|
||||||
resolver;
|
|
||||||
3. builds the app-level transform registry;
|
|
||||||
4. opens each selected pipeline source backend;
|
|
||||||
5. discovers validated source bundles from the source root;
|
|
||||||
6. selects source bundles for each destination according to path mapping;
|
|
||||||
7. opens destination backends independently;
|
|
||||||
8. builds publish plans for selected bundle and destination combinations;
|
|
||||||
9. records warnings, action records, output metadata, and summary counters;
|
|
||||||
10. executes publish or replacement plans unless dry-run is enabled;
|
|
||||||
11. invokes the notifier after successful publish or replacement actions;
|
|
||||||
12. returns the structured report and any aggregated destination failures.
|
|
||||||
|
|
||||||
`RunPipeline` follows the same flow after selecting a single configured
|
HTTP uploads stage and validate archives before enqueueing a pipeline run with a local staged source root.
|
||||||
pipeline. It uses the same backend factory, secret loading, transform registry,
|
|
||||||
warning generation, destination planning, publish execution, notification
|
|
||||||
behavior, and failure aggregation as `Run`.
|
|
||||||
|
|
||||||
`RunPipelineWithLocalSource` follows the same flow after pipeline selection
|
## Skip And Resume Behavior
|
||||||
except for source opening and source discovery. It opens the supplied local
|
|
||||||
source root directly, validates the root bundle before opening any destinations,
|
|
||||||
and passes the resulting local source backend and bundle into the same
|
|
||||||
destination planning and execution loop. Destination code receives the normal
|
|
||||||
storage backend and bundle values and does not depend on how the source root was
|
|
||||||
prepared.
|
|
||||||
|
|
||||||
## Upload Coordination
|
Fan-out destinations are independent. A destination failure is recorded and does not prevent later destinations from being attempted. Dry-run builds plans and reports without destination writes, destination state writes, notifier calls, or SSH known-host persistence.
|
||||||
|
|
||||||
`UploadCoordinator` owns in-memory coordination for asynchronous upload
|
HTTP upload status is in memory. Accepted jobs move through accepted, queued, running, succeeded, or failed states and expire after configured retention.
|
||||||
processing. It admits uploads for configured `http_upload` pipelines, reserves
|
|
||||||
queue capacity before request-body staging, stages and validates archives
|
|
||||||
through `internal/ingest`, tracks accepted status records, and executes the
|
|
||||||
selected pipeline through `RunPipelineWithLocalSource`.
|
|
||||||
|
|
||||||
Upload run IDs use:
|
## Failure Behavior
|
||||||
|
|
||||||
```text
|
Runtime setup fails for config load, config validation, secret loading, or credential resolution errors. Source setup failures stop the affected run before destination planning. Destination open, planning, execution, and notification failures are recorded as destination failures where a partial result exists.
|
||||||
<pipeline id>.<UTC timestamp>.<random suffix>
|
|
||||||
```
|
|
||||||
|
|
||||||
The timestamp uses `YYYYMMDDThhmmssZ` UTC format and the suffix is filesystem
|
HTTP upload startup fails if upload tokens are missing, empty, or duplicated. Upload requests can fail during authentication, content-type validation, queue admission, archive staging, source validation, or later publish execution.
|
||||||
safe.
|
|
||||||
|
|
||||||
The coordinator records these statuses:
|
## Tests To Inspect
|
||||||
|
|
||||||
- `accepted`
|
- `internal/app/*_test.go`
|
||||||
- `queued`
|
- `internal/cli/root_test.go`
|
||||||
- `running`
|
- `internal/config/*_test.go`
|
||||||
- `succeeded`
|
- `internal/ingest/*_test.go`
|
||||||
- `failed`
|
- `internal/publish/*_test.go`
|
||||||
- `expired`
|
|
||||||
|
|
||||||
Admission is bounded by `server.http.queue_size`. Full queues are rejected
|
## Architectural Invariants
|
||||||
before the upload body is read. Successfully reserved uploads are staged and
|
|
||||||
validated before an accepted run record is created. Execution is bounded by
|
|
||||||
`server.http.max_concurrency`, and only one upload for a given pipeline may run
|
|
||||||
at a time. Later accepted uploads for the same pipeline remain queued until the
|
|
||||||
active run finishes.
|
|
||||||
|
|
||||||
Completed records retain the final run report or error text until
|
- App orchestration owns wiring, not low-level policy.
|
||||||
`server.http.retention` elapses. Expiration removes completed status records and
|
- Dry-run must not write outputs, destination state, notifier events, or SSH known-host entries.
|
||||||
their committed staged bundle directories. The coordinator is memory-only and
|
- Fan-out destinations remain independent after a destination-scoped failure.
|
||||||
does not persist queue state, status records, or run reports.
|
- Secret values are never printed; warnings may name variables only.
|
||||||
|
- Upload admission stages and validates a bundle before returning a run id.
|
||||||
## HTTP Upload Server
|
- Runtime backend registration remains app-owned.
|
||||||
|
|
||||||
The HTTP upload server is app-layer transport wiring around
|
|
||||||
`UploadCoordinator`. It owns request authentication, route dispatch, HTTP status
|
|
||||||
mapping, and JSON response projection. Bundle staging and publication remain in
|
|
||||||
the coordinator and staged-source run path.
|
|
||||||
|
|
||||||
Server startup resolves every configured `http_upload` source `token_env`
|
|
||||||
through the config-owned environment resolver after `secrets.directory` has
|
|
||||||
been loaded. Startup fails when a token is missing, empty, or duplicates another
|
|
||||||
upload pipeline token. Error messages identify environment variable names and
|
|
||||||
pipeline ids, but not token values.
|
|
||||||
|
|
||||||
Routes:
|
|
||||||
|
|
||||||
- `GET /healthz`: returns `200` after config, secrets, tokens, coordinator, and route setup succeed.
|
|
||||||
- `POST /upload`: stages and validates an authenticated tar or tar.gz archive, then returns an accepted run id.
|
|
||||||
- `GET /runs/<run_id>`: returns the current in-memory upload status record or `404`.
|
|
||||||
|
|
||||||
The upload token maps to exactly one configured pipeline. Producers do not
|
|
||||||
submit pipeline ids, and submitted `pipeline` or `pipeline_id` query values are
|
|
||||||
rejected. Full queues are rejected before the request body is read. Malformed
|
|
||||||
archives and invalid staged bundles are rejected before a run id is issued.
|
|
||||||
Oversized uploads, unsupported content types, invalid bearer tokens, full
|
|
||||||
queues, and unknown status records are mapped to stable HTTP status codes
|
|
||||||
without returning secret token values.
|
|
||||||
|
|
||||||
## Errors
|
|
||||||
|
|
||||||
`Run` returns immediately for config loading errors, context cancellation before
|
|
||||||
work starts, source open errors, and source discovery errors.
|
|
||||||
|
|
||||||
`RunPipeline` returns `PipelineNotFoundError` when the requested pipeline ID is
|
|
||||||
not configured. Callers can detect that condition with `IsPipelineNotFound`.
|
|
||||||
|
|
||||||
`RunPipelineWithLocalSource` also returns `PipelineNotFoundError` for an unknown
|
|
||||||
pipeline ID. It returns before destination opening when the supplied local
|
|
||||||
source root is missing, cannot be opened, or does not validate as one complete
|
|
||||||
source bundle.
|
|
||||||
|
|
||||||
Per-destination backend, planning, execution, and notification errors are
|
|
||||||
aggregated into one run error after remaining destinations have been attempted.
|
|
||||||
Destination diagnostics include pipeline ID, destination ID, backend, and
|
|
||||||
bundle path.
|
|
||||||
|
|
||||||
Stdout write errors are returned immediately because the caller's requested
|
|
||||||
output stream can no longer be trusted.
|
|
||||||
|
|
||||||
## Package Layout
|
|
||||||
|
|
||||||
Run helpers are grouped by responsibility:
|
|
||||||
|
|
||||||
- `runtime.go`: runtime config path resolution, config loading, secret loading,
|
|
||||||
environment resolver handoff, and secret-conflict warning projection.
|
|
||||||
- `run.go`: `Run`, `RunPipeline`, and shared run orchestration.
|
|
||||||
- `run_destination.go`: destination-scoped planning, execution, action
|
|
||||||
recording, and failure bookkeeping.
|
|
||||||
- `run_output.go`: `RunReport`, action/output records, and text/JSON report projection.
|
|
||||||
- `output_projection.go`: shared bundle and manifest-file result projection for
|
|
||||||
command JSON output.
|
|
||||||
- `run_summary.go`: summary counters.
|
|
||||||
- `run_failures.go`: destination failure aggregation and partial-result detection.
|
|
||||||
- `run_selection.go`: destination bundle selection, path mapping decisions, and fixed-path warnings.
|
|
||||||
- `run_warnings.go`: secret and SSH warning records.
|
|
||||||
- `run_notify.go`: notification event projection and action filtering.
|
|
||||||
- `upload_coordinator.go`: in-memory upload admission, queue reservation, staging handoff, status tracking, queueing, and staged-source execution.
|
|
||||||
- `upload_http.go`: HTTP upload authentication, routes, JSON response projection, and HTTP error mapping.
|
|
||||||
- `serve.go`: HTTP server startup.
|
|
||||||
- `backends.go`: app-level backend factory wiring.
|
|
||||||
- `transforms.go`: app-level transform registry wiring.
|
|
||||||
- `source_select.go`: configured-source selection shared by `validate` and `inspect`.
|
|
||||||
|
|
||||||
## Backend And Transform Wiring
|
|
||||||
|
|
||||||
The app-level backend factory registers local, SSH, and S3 backends for runtime
|
|
||||||
execution. Source and destination backend config is converted through a shared
|
|
||||||
app-local open spec before adapter construction.
|
|
||||||
|
|
||||||
Credential references are resolved through the config environment resolver.
|
|
||||||
Production app code must not read backend credential environment variables
|
|
||||||
directly.
|
|
||||||
|
|
||||||
The app-level transform registry registers Markdown-to-HTML through
|
|
||||||
`internal/transform/markdown`. Lower-level publish code receives a resolver and
|
|
||||||
does not import concrete transform implementations.
|
|
||||||
|
|
||||||
## Dry-Run Behavior
|
|
||||||
|
|
||||||
Dry-run loads config, opens backends, discovers bundles, inspects destinations,
|
|
||||||
resolves transforms, and builds publish plans. It does not write destination
|
|
||||||
outputs, write `.distributor.json`, delete managed outputs, perform forced
|
|
||||||
prefix deletion, or invoke notifications.
|
|
||||||
|
|
||||||
## Tests
|
|
||||||
|
|
||||||
Before changing app orchestration, inspect tests under:
|
|
||||||
|
|
||||||
- `internal/app`
|
|
||||||
- `internal/cli`
|
|
||||||
- `internal/publish`
|
|
||||||
|
|
||||||
Use focused app tests for report structure, single-pipeline execution,
|
|
||||||
upload admission, warning generation, notification behavior, and partial-result
|
|
||||||
aggregation.
|
|
||||||
|
|
||||||
## Invariants
|
|
||||||
|
|
||||||
- One source fans out to each destination independently.
|
|
||||||
- Destination failures do not prevent later destinations from being planned.
|
|
||||||
- Destination-scoped failures still produce a structured report plus an aggregated error.
|
|
||||||
- Dry-run must not mutate destination storage or invoke notifications.
|
|
||||||
- `RunPipeline` must use the same run path as `Run` after pipeline selection.
|
|
||||||
- Concrete backend and transform registration stays at the app layer.
|
|
||||||
- The default notifier is `notify.Noop`.
|
|
||||||
|
|||||||
@@ -1,55 +1,53 @@
|
|||||||
# Bundles
|
# Source Bundle Internals
|
||||||
|
|
||||||
|
Audience: developers and LLM coding agents changing `internal/bundle`.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
`internal/bundle` discovers and validates source bundles through the storage interface. The source manifest model, manifest parsing, manifest validation, path rules, digest calculation, and producer-side local writer come from `pkg/bundle` so producer-facing APIs and distributor validation share one manifest contract.
|
`internal/bundle` discovers and validates source bundles through the storage abstraction. It adapts the public producer-facing source manifest contract from `pkg/bundle` to local, SSH/SFTP, S3-compatible, and test storage backends.
|
||||||
|
|
||||||
## Inputs and outputs
|
## Inputs And Outputs
|
||||||
|
|
||||||
Input is a backend-rooted directory tree containing one or more `manifest.json` files. Output is a deterministic list of validated bundles with relative bundle paths and normalized manifest data.
|
Inputs are a context, a `storage.Backend`, and a source-root prefix or bundle root path. Outputs are sorted `Bundle` records containing the source-root-relative bundle path and validated manifest.
|
||||||
|
|
||||||
## Manifest behavior
|
|
||||||
|
|
||||||
The source manifest requires:
|
|
||||||
|
|
||||||
- `schema_version: 1`
|
|
||||||
- `id`
|
|
||||||
- `digest`
|
|
||||||
- `created`
|
|
||||||
- non-empty `files`
|
|
||||||
|
|
||||||
Each file requires `path`, `sha256`, and `size`. Digests must use lowercase `sha256:<64 hex>` format. `created` must parse as RFC3339.
|
|
||||||
|
|
||||||
`pkg/bundle.ValidateDigest` is the canonical digest format validator for producer-facing and internal code. `internal/bundle.ValidateDigest` delegates to that public validator so source manifests and destination state use the same digest grammar.
|
|
||||||
|
|
||||||
## Validation
|
|
||||||
|
|
||||||
`pkg/bundle.ValidateManifest` owns normalized source manifest semantics: schema version, id, digest format, timestamp presence, file list presence, source path safety, duplicate file paths, reserved paths, file digest format, non-negative file sizes, and the top-level bundle digest.
|
|
||||||
|
|
||||||
Storage-backed bundle validation in `internal/bundle` additionally checks file existence, regular-file type, file size, and per-file SHA-256 for configured storage backends.
|
|
||||||
|
|
||||||
The bundle digest is SHA-256 of a deterministic JSON array of file records in manifest order with fields `path`, `sha256`, and `size`.
|
|
||||||
|
|
||||||
## Discovery
|
|
||||||
|
|
||||||
Discovery walks a storage backend beneath a source root, finds `manifest.json` files, sorts bundle paths lexically, and rejects nested manifests.
|
|
||||||
|
|
||||||
## Failure behavior
|
|
||||||
|
|
||||||
Manifest parsing and validation fail before destination planning. Storage-backed validation fails when listed files are missing, are not regular files, have unexpected sizes, have unexpected SHA-256 digests, or when a source bundle includes unsafe or reserved paths.
|
|
||||||
|
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
Internal bundle discovery uses `internal/storage` and does not import concrete adapters. Producer-side local filesystem manifest building, complete bundle writing, and validation belong to `pkg/bundle`. CLI local path support is wired in `internal/app`.
|
`internal/bundle` delegates manifest parsing, digest calculation, source path validation, and manifest validation to `pkg/bundle`. It does not publish files, inspect destination state, choose pipelines, or know concrete backend implementations.
|
||||||
|
|
||||||
## Tests
|
The external source bundle file-format contract is documented in `docs/integrations/source-bundle.md`.
|
||||||
|
|
||||||
Before changing bundle behavior, inspect tests under `pkg/bundle` and `internal/bundle`.
|
## Config Fields Used
|
||||||
|
|
||||||
## Invariants
|
The package does not read config directly. App workflows pass it storage backends that were opened from configured source fields.
|
||||||
|
|
||||||
- `manifest.json` is the only source bundle contract.
|
## Adapters Used
|
||||||
- Source file paths must stay relative to the bundle root.
|
|
||||||
- The top-level bundle digest is derived from manifest file records in order.
|
The package depends only on `internal/storage.Backend`. Concrete local, SSH/SFTP, S3-compatible, and fake backends are hidden behind that interface.
|
||||||
- Discovery order is lexical and deterministic.
|
|
||||||
- Nested manifests are rejected.
|
## State And Manifest Behavior
|
||||||
|
|
||||||
|
Discovery walks recursively under the source root, finds entries whose basename is `manifest.json`, converts each manifest path to a bundle root, sorts roots, rejects nested bundle roots, and validates each bundle.
|
||||||
|
|
||||||
|
Validation reads `manifest.json`, parses it, stats each manifest-listed file, requires regular files, verifies file sizes, reads file bytes, checks per-file SHA-256 digests, and recomputes the bundle digest.
|
||||||
|
|
||||||
|
## Skip And Resume Behavior
|
||||||
|
|
||||||
|
The package has no skip or resume state. Each call performs discovery or validation from the supplied backend state.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Failures include invalid storage prefixes, missing manifests, parse errors, nested manifests, unsafe manifest paths, non-regular files, size mismatches, digest mismatches, backend stat/read errors, and no discovered bundles.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/bundle/*_test.go`
|
||||||
|
- `pkg/bundle/*_test.go`
|
||||||
|
- `internal/storage/fake/*_test.go`
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
|
- Source manifest semantics remain owned by `pkg/bundle`.
|
||||||
|
- Discovery order is deterministic.
|
||||||
|
- Nested manifests are rejected before returning bundles.
|
||||||
|
- Source paths stay clean, relative, slash-separated, and confined to the backend root.
|
||||||
|
- Concrete adapters never leak into bundle validation logic.
|
||||||
|
|||||||
@@ -1,101 +1,57 @@
|
|||||||
# Configuration Internals
|
# Configuration Internals
|
||||||
|
|
||||||
|
Audience: developers and LLM coding agents changing `internal/config`.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
`internal/config` defines YAML-backed configuration structs, defaulting, and validation for distributor pipelines.
|
`internal/config` owns YAML config structs, config file loading, defaulting, validation, size/duration parsing, SSH/S3 normalization helpers, and the credential environment resolver.
|
||||||
|
|
||||||
## Inputs and outputs
|
## Inputs And Outputs
|
||||||
|
|
||||||
Input is a YAML file containing optional `server`, optional `secrets`, and required `pipelines`. Output is a `Config` value with defaults applied and validation completed. Load failures include the config path and whether the failure occurred during file loading, YAML parsing, or validation.
|
Inputs are YAML files, YAML scalar values, process environment lookup functions, optional secrets directories, and `Config` values. Outputs are defaulted `Config` values, validation errors, parsed byte sizes and durations, normalized backend options, loaded secret environments, secret conflict metadata, and resolved credentials.
|
||||||
|
|
||||||
## Loading flow
|
## Boundaries
|
||||||
|
|
||||||
`LoadFile` opens the requested path, decodes YAML with known-field checking enabled, applies defaults, and validates the result. The app uses `DefaultConfigPath` when the CLI does not supply a config path.
|
The package does not open storage backends, authenticate HTTP requests, start servers, publish destinations, or execute transforms. Runtime execution support is wired by `internal/app`.
|
||||||
|
|
||||||
Known-field checking rejects misspelled or unknown YAML keys before defaults and validation run.
|
The canonical user-facing config reference is `docs/config.md`.
|
||||||
|
|
||||||
`LoadFile` does not read secret files. App entrypoints load the configured
|
## Config Fields Used
|
||||||
secrets directory after config validation and before credential-consuming work.
|
|
||||||
|
|
||||||
## Defaults
|
The package defines all user-visible config fields: `server.http`, `secrets`, `pipelines`, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, and transfer policy.
|
||||||
|
|
||||||
Defaults are applied in `ApplyDefaults`:
|
## Adapters Used
|
||||||
|
|
||||||
- HTTP server `bind` defaults to `127.0.0.1:8080`;
|
No external storage adapters are used directly. The package exposes normalized config and credential values consumed by app-level adapter construction.
|
||||||
- HTTP server `staging_root` defaults to `/var/spool/distributor`;
|
|
||||||
- HTTP server `max_upload_size` defaults to `20MB`;
|
|
||||||
- HTTP server `queue_size` defaults to `16`;
|
|
||||||
- HTTP server `max_concurrency` defaults to `1`;
|
|
||||||
- HTTP server `retention` defaults to `24h`;
|
|
||||||
- `http_upload` source `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`;
|
|
||||||
- `http_upload` source `max_upload_size` defaults to `server.http.max_upload_size`;
|
|
||||||
- pipeline validation defaults `on_digest_mismatch` to `fail`;
|
|
||||||
- SSH backend `port` defaults to `22`;
|
|
||||||
- SSH backend `host_key_policy` defaults to `accept-new`;
|
|
||||||
- destination publish policy defaults to source output only;
|
|
||||||
- Markdown-to-HTML mode defaults to `sidecar` when a transform block is present and mode is omitted;
|
|
||||||
- destination path mapping defaults to `preserve_relative`;
|
|
||||||
- destination link primary policy defaults to `auto` when a `links` block is present;
|
|
||||||
- `transfer.on_destination_same` defaults to `skip`;
|
|
||||||
- `transfer.on_destination_older` defaults to `replace`;
|
|
||||||
- `transfer.on_destination_newer` defaults to `skip`;
|
|
||||||
- `transfer.on_conflict` defaults to `fail`.
|
|
||||||
|
|
||||||
## Validation responsibilities
|
## State And Manifest Behavior
|
||||||
|
|
||||||
Validation requires positive HTTP server limits and retention, at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, valid destination link config, and valid transfer actions.
|
The package does not parse source manifests or destination state. It validates config values that later affect manifest validation and destination state, such as publish/transform combinations, links, transfer policy, backend roots, S3 prefix shape, and HTTP upload source settings.
|
||||||
|
|
||||||
HTTP upload sources require `token_env`, a staging path after defaults, and a positive maximum upload size. Literal token fields are not part of the YAML schema. The `http_upload` backend is accepted only for sources and rejected for destinations.
|
## Skip And Resume Behavior
|
||||||
|
|
||||||
Upload size values are parsed from strings with `B`, `KB`, `MB`, or `GB` suffixes using 1024 multipliers. Retention values are parsed with `time.ParseDuration`. Explicit zero values fail validation; omitted values receive defaults before validation.
|
The package has no runtime skip or resume behavior. It provides transfer policy values that publish planning later applies to destination comparison outcomes.
|
||||||
|
|
||||||
Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
|
## Failure Behavior
|
||||||
|
|
||||||
`ValidatePublishTransformPolicy` is shared with publish planning so destination policy combinations are checked consistently. Publishing HTML requires an enabled Markdown-to-HTML transform in `sidecar` or `index` mode. Enabled Markdown-to-HTML config is rejected when `publish.html` is false. `input` is accepted only for enabled `index` mode. A publish policy must select source output, HTML output, or both.
|
`LoadFile` wraps file open, YAML parse, and validation failures with config path context. YAML decoding rejects unknown fields. Validation collects all detected field errors into a single error value.
|
||||||
|
|
||||||
Destination path mapping accepts `preserve_relative` and `fixed`. The app layer applies the mapping when it selects destination bundle paths; config owns only YAML shape, defaulting, and validation.
|
Secret loading fails for unreadable secrets directories, invalid secret filenames, unreadable secret files, and missing or empty required credential values. Secret conflicts are returned as warnings metadata, not secret values.
|
||||||
|
|
||||||
Destination links are optional. When a `links` block is present, `base_url` is required, must use `http` or `https`, and must not include a query string or fragment. `primary` accepts `auto`, `html`, and `source`.
|
## Tests To Inspect
|
||||||
|
|
||||||
## Executable support boundary
|
|
||||||
|
|
||||||
Config validation accepts `local`, `ssh`, `s3`, and source-only `http_upload` backend shapes. Runtime `run`, `validate`, and `inspect` workflows open `local`, `ssh`, and `s3` through `internal/app`. Runtime `serve` workflows execute `http_upload` sources through the app upload coordinator and HTTP server.
|
|
||||||
|
|
||||||
SSH config uses structured fields: `host`, optional `user`, optional `port`, `path`, optional `ssh_key_file`, optional `known_hosts`, and optional `host_key_policy`. `host_key_policy` accepts YAML booleans and strings and normalizes `true`/`strict`, `accept-new`, and `false`/`off`.
|
|
||||||
|
|
||||||
S3 config requires `endpoint` and `bucket`, normalizes optional `prefix`, defaults `region` to `us-east-1`, and defaults omitted `force_path_style` to `true` while preserving explicit `false`.
|
|
||||||
|
|
||||||
HTTP upload config is source-only. Config owns its YAML shape, defaulting, size and duration parsing, and validation. The config package does not authenticate requests, stage uploads, or execute HTTP upload sources. The app layer resolves `token_env` through the config-owned environment resolver before starting the HTTP server.
|
|
||||||
|
|
||||||
## Secrets and credential resolution
|
|
||||||
|
|
||||||
`secrets.directory` points to a directory of credential files. `LoadSecretEnvironment` reads regular files and symlinks to regular files, rejects invalid filenames, trims exactly one trailing LF or CRLF, and returns an `Environment` resolver plus conflict metadata.
|
|
||||||
|
|
||||||
The resolver checks the real process environment first and loaded secret values second. Differing process/secret conflicts are reported by variable name only. The resolver does not mutate `os.Environ`; default SDK credential chains continue to see only real process environment values.
|
|
||||||
|
|
||||||
Credential-consuming backend wiring should resolve explicit credential environment variable references through `Environment.ResolveCredentials` or the same resolver pattern instead of calling `os.Getenv` directly.
|
|
||||||
|
|
||||||
The user-facing configuration reference is `docs/config.md`; this file documents package behavior for maintainers.
|
|
||||||
|
|
||||||
## Failure behavior
|
|
||||||
|
|
||||||
Load errors wrap the underlying file, YAML, or validation error with context. Validation collects all detected field errors into one error value instead of stopping at the first invalid field.
|
|
||||||
|
|
||||||
Unsupported backend names fail validation. Accepted backend names without runtime execution support fail later during app backend opening.
|
|
||||||
|
|
||||||
## Tests
|
|
||||||
|
|
||||||
Before changing config behavior, inspect:
|
|
||||||
|
|
||||||
- `internal/config/load_test.go`
|
- `internal/config/load_test.go`
|
||||||
- `internal/config/validate_test.go`
|
- `internal/config/validate_test.go`
|
||||||
- example-loading coverage in `internal/config`
|
- `internal/config/secrets_test.go`
|
||||||
- user-facing examples under `examples/`
|
- `internal/config/backend_view_test.go`
|
||||||
|
- `internal/app/runtime_test.go`
|
||||||
|
- example configs under `examples/`
|
||||||
|
|
||||||
## Invariants
|
## Architectural Invariants
|
||||||
|
|
||||||
- Defaults are applied before validation.
|
- Defaults are applied before validation.
|
||||||
- Unknown YAML fields are rejected.
|
- Unknown YAML fields are rejected.
|
||||||
- `docs/config.md` remains the canonical user-facing config reference.
|
- `http_upload` is source-only config.
|
||||||
- Runtime backend execution support is not inferred from config validation support.
|
- Credential-consuming runtime code must use the config-owned environment resolver.
|
||||||
- New user-visible config behavior must be covered by tests and docs in the same change.
|
- Secret values are never printed by config warnings.
|
||||||
|
- New user-visible config behavior must update `docs/config.md` and tests.
|
||||||
|
|||||||
@@ -1,46 +1,51 @@
|
|||||||
# Ingestion Internals
|
# Ingestion Internals
|
||||||
|
|
||||||
|
Audience: developers and LLM coding agents changing `internal/ingest`.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
`internal/ingest` stages uploaded source bundle archives into local per-run directories. It does not authenticate requests, manage upload queues, publish destinations, or start an HTTP server.
|
`internal/ingest` validates upload content types, extracts uploaded source bundle archives into local temporary storage, validates extracted bundles, and commits accepted bundles to per-run staging directories.
|
||||||
|
|
||||||
## Archive staging
|
## Inputs And Outputs
|
||||||
|
|
||||||
`ValidateContentType` owns accepted upload content-type policy for archive
|
Inputs are a context, upload body reader, content type, pipeline staging path, run id, maximum uploaded size, maximum extracted size, and maximum file count. Output is a `StagedBundle` containing the committed local bundle root and parsed manifest.
|
||||||
staging callers.
|
|
||||||
|
|
||||||
`StageArchive` accepts one upload body, content type, pipeline staging path, run id, and explicit size and file-count limits. It writes the request body to temporary storage while enforcing the configured upload size limit, extracts the archive into temporary local storage, validates the extracted source bundle, and then commits the validated bundle to:
|
## Boundaries
|
||||||
|
|
||||||
```text
|
The package does not authenticate HTTP requests, manage upload queues, track upload status, publish destinations, load config, or start an HTTP server. Those responsibilities live in `internal/app`.
|
||||||
<pipeline staging path>/<run id>
|
|
||||||
```
|
|
||||||
|
|
||||||
The returned `StagedBundle.Root` is a local filesystem path to the validated source bundle root.
|
The HTTP API contract is documented in `docs/integrations/http-upload.md`.
|
||||||
|
|
||||||
## Accepted archive formats
|
## Config Fields Used
|
||||||
|
|
||||||
The package accepts only:
|
The package does not read config directly. The app layer passes effective values derived from `source.staging_path`, `source.max_upload_size`, and HTTP server defaults.
|
||||||
|
|
||||||
- `application/x-tar`
|
## Adapters Used
|
||||||
- `application/gzip`
|
|
||||||
- `application/x-gzip`
|
|
||||||
|
|
||||||
Gzip uploads must contain a tar archive.
|
The package uses the local filesystem directly for temporary archive storage, extraction, validation, and final staging path promotion. It does not use the storage backend abstraction.
|
||||||
|
|
||||||
## Extraction rules
|
## State And Manifest Behavior
|
||||||
|
|
||||||
Archive entry paths must be clean relative slash-separated paths. Extraction rejects absolute paths, path traversal, backslash paths, duplicate files, symlinks, hardlinks, devices, sockets, and other special entries.
|
Accepted archives must contain exactly one root-level `manifest.json`. After extraction, the package validates the staged root through `pkg/bundle`, including manifest parsing, source path rules, file existence, regular-file checks, file sizes, file SHA-256 digests, and bundle digest.
|
||||||
|
|
||||||
The archive must contain exactly one root-level `manifest.json`. Nested manifests are rejected.
|
## Skip And Resume Behavior
|
||||||
|
|
||||||
Regular files and directories are the only accepted tar entries. Regular file extraction enforces the explicit maximum extracted byte count and maximum file count supplied by the caller.
|
The package has no resume behavior. A successful call commits one complete staged bundle root. Failed calls remove temporary data created by that call.
|
||||||
|
|
||||||
## Bundle validation
|
## Failure Behavior
|
||||||
|
|
||||||
After extraction, the package loads and validates the staged bundle through `pkg/bundle`. Manifest parsing, source path validation, file existence checks, regular-file checks, file sizes, file SHA-256 digests, and bundle digest validation use the existing source bundle contract.
|
Failures include unsupported content type, unsafe run id, missing staging path, non-positive limits, oversize upload body, oversize extracted content, too many files, unsafe archive paths, duplicate files, nested manifests, unsupported tar entry types, gzip/tar read errors, bundle validation errors, and filesystem errors.
|
||||||
|
|
||||||
Validation happens before the staged bundle is committed to its final per-run path.
|
## Tests To Inspect
|
||||||
|
|
||||||
## Failure behavior
|
- `internal/ingest/archive_test.go`
|
||||||
|
- `internal/app/upload_*_test.go`
|
||||||
|
- `pkg/bundle/*_test.go`
|
||||||
|
|
||||||
Failed staging removes temporary archive and extraction data created by the package. A failed call does not publish anything and does not leave a committed per-run bundle directory.
|
## Architectural Invariants
|
||||||
|
|
||||||
|
- Invalid archives never commit a staged root.
|
||||||
|
- Archive paths remain clean relative slash-separated paths.
|
||||||
|
- Only directories and regular files are accepted from tar archives.
|
||||||
|
- Source bundle validation happens before final staging path promotion.
|
||||||
|
- Upload authentication and queueing remain outside this package.
|
||||||
|
|||||||
@@ -1,27 +1,46 @@
|
|||||||
# Link URL Policy
|
# Link URL Policy
|
||||||
|
|
||||||
|
Audience: developers and LLM coding agents changing `internal/link`.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
`internal/link` defines shared validation for configured and persisted HTTP link URLs.
|
`internal/link` owns shared validation for configured and persisted HTTP link URLs.
|
||||||
|
|
||||||
## Inputs and outputs
|
## Inputs And Outputs
|
||||||
|
|
||||||
Input is a URL string. Output is either nil for an accepted URL or a concise validation error that callers wrap with field context.
|
Input is a URL string. Output is either nil for an accepted URL or a concise validation error that callers wrap with field context.
|
||||||
|
|
||||||
## Validation behavior
|
|
||||||
|
|
||||||
Accepted URLs must parse successfully, use `http` or `https`, include a host, and omit query strings and fragments.
|
|
||||||
|
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
This package validates URL shape only. It does not construct destination output URLs, choose primary URLs, infer public URLs from backend configuration, or read configuration files.
|
The package validates URL shape only. It does not construct output URLs, choose primary URLs, infer public URLs from backend configuration, parse config files, or write destination state.
|
||||||
|
|
||||||
## Tests
|
## Config Fields Used
|
||||||
|
|
||||||
Before changing link URL policy, inspect tests under `internal/link` and callers in `internal/config`, `internal/state`, and `internal/publish`.
|
The package does not read config directly. `internal/config` uses it to validate `links.base_url`; `internal/state` uses it to validate persisted `links.primary_url` and output `url` fields.
|
||||||
|
|
||||||
## Invariants
|
## Adapters Used
|
||||||
|
|
||||||
- Configured `links.base_url`, persisted `links.primary_url`, persisted output `url`, and publish link planning use the same URL policy.
|
None.
|
||||||
|
|
||||||
|
## State And Manifest Behavior
|
||||||
|
|
||||||
|
Destination state URL fields and configured link URLs share the same URL validation policy. Source manifests are not involved.
|
||||||
|
|
||||||
|
## Skip And Resume Behavior
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Validation rejects parse failures, non-HTTP(S) schemes, missing hosts, query strings, and fragments.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/link/url_test.go`
|
||||||
|
- callers in `internal/config`, `internal/state`, and `internal/publish`
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
|
- Configured and persisted link URLs use one validation policy.
|
||||||
|
- URL construction remains outside `internal/link`.
|
||||||
- Callers own field-specific error context.
|
- Callers own field-specific error context.
|
||||||
- URL path construction remains in `internal/publish`.
|
|
||||||
|
|||||||
@@ -1,35 +1,48 @@
|
|||||||
# Notify
|
# Notification Internals
|
||||||
|
|
||||||
|
Audience: developers and LLM coding agents changing `internal/notify` or app notification wiring.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
`internal/notify` defines the internal notification interface used by the application runner.
|
`internal/notify` defines the notification interface used by app orchestration after successful destination publication or replacement.
|
||||||
|
|
||||||
## Inputs and outputs
|
## Inputs And Outputs
|
||||||
|
|
||||||
Input is a notification event containing pipeline id, destination id, bundle id, bundle path, action, and output metadata. The interface returns an error so app orchestration can treat notification failures as destination failures.
|
Input is a context and notification event containing pipeline id, destination id, bundle id, bundle path, action, and output metadata. Output is an error that app orchestration can record as a destination-scoped failure.
|
||||||
|
|
||||||
## Current behavior
|
|
||||||
|
|
||||||
The implemented notifier is a no-op. It is invoked only after a successful publish or replacement. Dry-run, skipped destinations, and failed destinations do not invoke it.
|
|
||||||
|
|
||||||
## Failure behavior
|
|
||||||
|
|
||||||
`notify.Noop` always succeeds unless the context is already canceled. If a configured notifier returns an error, `internal/app` records that destination as failed and continues with remaining destinations.
|
|
||||||
|
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
External notification adapters and user-facing notification configuration are outside current behavior.
|
Only the no-op notifier exists in the repository. The package does not load config, send network requests, write destination state, publish files, or own run reporting.
|
||||||
|
|
||||||
## Tests
|
## Config Fields Used
|
||||||
|
|
||||||
Before changing notification behavior, inspect:
|
None.
|
||||||
|
|
||||||
|
## Adapters Used
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
## State And Manifest Behavior
|
||||||
|
|
||||||
|
Notification events carry output metadata projected from publish plans. The package does not inspect source manifests or destination state.
|
||||||
|
|
||||||
|
## Skip And Resume Behavior
|
||||||
|
|
||||||
|
Dry-run, skipped destinations, failed destinations, and planning failures do not notify. The no-op notifier has no durable state.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
`notify.Noop` returns the context error when the context is canceled; otherwise it succeeds. If another notifier implementation returns an error, `internal/app` records the affected destination as failed and continues with remaining destinations where applicable.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
- `internal/notify`
|
- `internal/notify`
|
||||||
- `internal/app/run_test.go`
|
- `internal/app/run_notify.go`
|
||||||
|
- notification coverage in `internal/app/run_test.go`
|
||||||
|
|
||||||
## Invariants
|
## Architectural Invariants
|
||||||
|
|
||||||
- Notifications are emitted only after successful publish or replacement execution.
|
- Notifications occur only after successful publish or replacement execution.
|
||||||
- Dry-run never notifies.
|
- Dry-run never notifies.
|
||||||
- Skipped and failed destinations never notify.
|
- Skipped and failed destinations never notify.
|
||||||
- The default app notifier is `notify.Noop`.
|
- The default app notifier is `notify.Noop`.
|
||||||
|
|||||||
@@ -1,52 +1,63 @@
|
|||||||
# Publish
|
# Publish Internals
|
||||||
|
|
||||||
|
Audience: developers and LLM coding agents changing `internal/publish`.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
`internal/publish` plans and executes publication for one validated source bundle and one destination.
|
`internal/publish` plans and executes publication for one validated source bundle and one destination bundle path. It owns destination comparison mapping, output selection, URL planning, managed cleanup selection, replacement safety, and destination state projection.
|
||||||
|
|
||||||
## Inputs and outputs
|
## Inputs And Outputs
|
||||||
|
|
||||||
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transform policy, optional link policy, transformer resolver, transfer policy, path mapping mode, destination bundle path, existing destination state, and whether explicit force was requested for the current run.
|
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, destination bundle path, path mapping mode, publish policy, transform policy, optional link policy, transformer resolver, transfer policy, distributor version, and force flag.
|
||||||
|
|
||||||
Output is a plan with an action, reason, optional primary URL, and selected source or generated outputs. Execution writes selected source files, generated files, and `.distributor.json` for publish or replacement actions.
|
Output from planning is a `Plan` with action, reason, destination identity, selected outputs, optional existing state, optional primary URL, and force metadata. Execution writes selected source outputs, generated outputs, and `.distributor.json` for executable publish or replacement actions.
|
||||||
|
|
||||||
## Actions
|
|
||||||
|
|
||||||
Supported actions are `publish_new`, `replace_older`, `force_replace`, `skip_same`, `skip_destination_newer`, `fail_conflict`, and `fail_unmanaged`.
|
|
||||||
|
|
||||||
## Failure behavior
|
|
||||||
|
|
||||||
Planning fails when request fields are incomplete, publish and transform policies are invalid, selected outputs collide, HTML output is requested without Markdown inputs, destination state is invalid, destination content is unmanaged without force, or transfer policy maps the comparison outcome to failure.
|
|
||||||
|
|
||||||
Execution fails if a write, delete, state serialization, or context check fails. Outputs written during a failed publish attempt are cleaned up through managed deletion where possible.
|
|
||||||
|
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
The package publishes source files and Markdown-to-HTML outputs. Markdown sidecar mode writes same-directory `.html` outputs, and Markdown index mode writes `index.html`. Backend behavior is supplied through `internal/storage`; app runtime supplies local, SSH, and S3 backends.
|
The package does not parse CLI flags, load config files, open concrete adapters, discover source bundles, select fixed-path bundle candidates, register transforms, or render command output. The app layer supplies validated request data and concrete dependencies.
|
||||||
|
|
||||||
The package uses `internal/state` for destination comparison, `internal/storage` for IO, and the shared `internal/config` publish/transform policy helper for request validation. It resolves transforms through a narrow resolver supplied by the caller; concrete transform registration is owned by the app layer. It does not parse CLI flags, load config files, or choose which source bundles a destination receives.
|
External destination state semantics are documented in `docs/integrations/destination-state.md`.
|
||||||
|
|
||||||
The package owns projection from planned publish outputs to destination state output records and managed destination output paths. App JSON results and notification events keep their own schemas, but may use the publish output projection to avoid field-mapping drift.
|
## Config Fields Used
|
||||||
|
|
||||||
The app layer computes the destination bundle path before planning. `preserve_relative` destinations pass the source-root-relative bundle path. `fixed` destinations pass an empty destination bundle path, which means the destination backend root, and pass only the newest selected source bundle for that destination.
|
The package consumes already-defaulted config values for destination `publish`, `transform`, `links`, `transfer`, and path mapping mode. It uses `config.ValidatePublishTransformPolicy` for publish/transform consistency.
|
||||||
|
|
||||||
When link config is present, publish planning builds per-output URLs from `links.base_url`, the destination bundle path, and each output path. `index.html` outputs use directory-style URLs. The primary URL is selected from planned outputs according to the destination primary policy.
|
## Adapters Used
|
||||||
|
|
||||||
## Safety
|
The package depends on `internal/storage.Backend` for source and destination IO, and on a narrow transformer resolver interface for generated outputs. It does not import concrete storage adapters or concrete transform implementations.
|
||||||
|
|
||||||
Normal replacement deletes only outputs recorded in existing destination state plus `.distributor.json`. Forced replacement deletes the bounded destination bundle path before writing outputs and state. Failed writes trigger cleanup of outputs written during the failed attempt where practical.
|
## State And Manifest Behavior
|
||||||
|
|
||||||
## Tests
|
Planning inspects destination state through `internal/state`, compares it with the source manifest, and maps comparison outcomes plus transfer policy into actions: `publish_new`, `replace_older`, `force_replace`, `skip_same`, `skip_destination_newer`, `fail_conflict`, or `fail_unmanaged`.
|
||||||
|
|
||||||
Before changing publish behavior, inspect tests under `internal/publish` and run tests under `internal/app`.
|
Execution writes destination state after selected outputs are written. Destination state includes copied source output metadata, generated output metadata, embedded source manifest, link metadata when configured, pipeline id, destination id, and publication timestamp.
|
||||||
|
|
||||||
## Invariants
|
## Skip And Resume Behavior
|
||||||
|
|
||||||
- Publish planning is deterministic for the same source, destination state, policies, and transform outputs.
|
`skip_same` and `skip_destination_newer` execute as no-ops. Normal replacement removes only managed output paths from existing state plus `.distributor.json`; this allows retries without broad deletion. Failed writes trigger cleanup of outputs written during that failed attempt where practical.
|
||||||
- Destination bundle paths are caller-supplied and are interpreted relative to the destination backend root.
|
|
||||||
- URL generation uses URL path semantics and does not infer public URLs from backend configuration.
|
Forced replacement is explicit per request and deletes the bounded destination bundle path before writing new outputs and state.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Planning fails for incomplete requests, invalid publish/transform policy, output path collisions, invalid destination state, unmanaged destination content without force, conflict outcomes not allowed by transfer policy, unresolved transforms, invalid Markdown output selection, and invalid link URL planning.
|
||||||
|
|
||||||
|
Execution fails on delete, read, transform output, write, state validation, state serialization, or context errors. Execution refuses actions that are not executable publish or replacement actions.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/publish/*_test.go`
|
||||||
|
- `internal/app/run_test.go`
|
||||||
|
- `internal/state/*_test.go`
|
||||||
|
- `internal/transform/markdown/*_test.go`
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
|
- Planning is deterministic for the same request and destination state.
|
||||||
|
- Destination bundle paths are caller-supplied and backend-root-relative.
|
||||||
|
- URL generation uses URL path semantics and never infers public URLs from backend config.
|
||||||
- Normal replacement deletes only managed paths recorded in existing state plus `.distributor.json`.
|
- Normal replacement deletes only managed paths recorded in existing state plus `.distributor.json`.
|
||||||
- Forced replacement is explicit per run and deletes only within the destination bundle path.
|
- Forced replacement deletes only within the supplied destination bundle path.
|
||||||
- Publish execution writes destination state after selected outputs are written.
|
- Destination state is written after selected outputs are written.
|
||||||
- Transform implementations are resolved through an interface supplied by the caller.
|
- Transform resolution stays behind a caller-supplied interface.
|
||||||
- Unmanaged destination content is overwritten only by explicit forced replacement.
|
- Unmanaged content is claimed only by explicit force.
|
||||||
|
|||||||
@@ -1,55 +1,54 @@
|
|||||||
# Destination State
|
# Destination State Internals
|
||||||
|
|
||||||
|
Audience: developers and LLM coding agents changing `internal/state`.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
`internal/state` parses, validates, writes, and compares `.distributor.json` destination state.
|
`internal/state` parses, validates, serializes, and compares `.distributor.json` destination state.
|
||||||
|
|
||||||
## Inputs and outputs
|
## Inputs And Outputs
|
||||||
|
|
||||||
Input is JSON destination state plus the current source manifest, pipeline id, destination id, and whether the destination path has unmanaged content. Output is a deterministic comparison outcome and reason.
|
Inputs are destination state JSON, constructed state values, current source manifest, pipeline id, destination id, and whether the destination path has content without state. Outputs are validated state values, JSON bytes, comparison outcomes, and human-readable reasons.
|
||||||
|
|
||||||
## State behavior
|
|
||||||
|
|
||||||
`.distributor.json` requires:
|
|
||||||
|
|
||||||
- `schema_version: 1`
|
|
||||||
- `pipeline_id`
|
|
||||||
- `destination_id`
|
|
||||||
- `published_at`
|
|
||||||
- `source.manifest`
|
|
||||||
- `outputs`
|
|
||||||
|
|
||||||
`distributor_version` is optional diagnostic metadata. `links` is optional URL metadata. `published_at` parses as RFC3339 and distributor-written state serializes it as RFC3339 UTC.
|
|
||||||
|
|
||||||
The embedded `source.manifest` is validated with the same source manifest rules used by `internal/bundle`.
|
|
||||||
|
|
||||||
## Outputs
|
|
||||||
|
|
||||||
Each output records `path`, `kind`, `source_path`, `sha256`, and `size`. Supported output kinds are `source` and `generated`. Generated outputs require `transform`. Outputs may record `url` when the destination has link generation configured.
|
|
||||||
|
|
||||||
The optional top-level `links.primary_url` records the selected primary URL for the published destination bundle. It is omitted when link generation is not configured or when the destination primary policy has no matching output.
|
|
||||||
|
|
||||||
## Comparison
|
|
||||||
|
|
||||||
Comparison outcomes cover absent destination state, unmanaged destination content, invalid state, pipeline or destination mismatch, same source manifest, older destination source, newer destination source, same-created digest conflict, and different source id conflict.
|
|
||||||
|
|
||||||
## Failure behavior
|
|
||||||
|
|
||||||
Invalid JSON, invalid state schema, invalid embedded source manifests, unsafe output paths, invalid stored URLs, unsupported output kinds, missing generated-output transform names, and mismatched pipeline or destination ids produce comparison outcomes that publish planning can turn into fail actions. Supported identity and source-manifest conflicts can become forced replacement only when publish planning receives explicit force and compatible transfer policy.
|
|
||||||
|
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
This package does not publish files, delete files, inspect storage backends, or choose transfer policy actions. Publish planning consumes these comparison outcomes later.
|
The package does not inspect storage backends, mutate files, choose transfer policy, build publish outputs, generate URLs, or parse config. Publish planning consumes state comparison outcomes.
|
||||||
|
|
||||||
## Tests
|
The external destination state contract is documented in `docs/integrations/destination-state.md`.
|
||||||
|
|
||||||
Before changing destination state behavior, inspect tests under `internal/state`.
|
## Config Fields Used
|
||||||
|
|
||||||
## Invariants
|
None directly. Destination ids, pipeline ids, and link URLs originate from config but are supplied as values by callers.
|
||||||
|
|
||||||
|
## Adapters Used
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
## State And Manifest Behavior
|
||||||
|
|
||||||
|
`.distributor.json` schema version is `1`. Required fields are `pipeline_id`, `destination_id`, `published_at`, `source.manifest`, and `outputs`. `distributor_version` and `links` are optional.
|
||||||
|
|
||||||
|
Embedded source manifests are parsed and validated through `internal/bundle`, which delegates source manifest semantics to `pkg/bundle`. Output records require clean paths, `source` or `generated` kind, valid source paths, lowercase SHA-256 digests, non-negative sizes, and transform ids for generated outputs. Stored URLs must pass `internal/link` validation.
|
||||||
|
|
||||||
|
## Skip And Resume Behavior
|
||||||
|
|
||||||
|
Comparison is pure. It returns outcomes for absent state, unmanaged content, invalid state, pipeline/destination mismatch, same source manifest, older destination, newer destination, same-created digest conflict, and different source id conflict. It does not decide whether to skip, replace, force, or fail; publish planning maps outcomes to actions.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Parsing rejects invalid JSON, trailing data, missing required fields, invalid timestamps, invalid embedded manifests, duplicate outputs, invalid output paths, unsupported output kinds, missing generated transforms, invalid URLs, invalid digests, and negative sizes.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/state/distributor_test.go`
|
||||||
|
- `internal/state/compare_test.go`
|
||||||
|
- `internal/publish/*_test.go`
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
- `.distributor.json` is the destination sentinel and state record.
|
- `.distributor.json` is the destination sentinel and state record.
|
||||||
- Embedded source manifests use the same validation rules as source bundles.
|
- Comparison does not mutate storage.
|
||||||
|
- Embedded source manifests use the source bundle contract.
|
||||||
- Generated outputs always record a transform id.
|
- Generated outputs always record a transform id.
|
||||||
- Stored URLs are optional and must be absolute HTTP or HTTPS URLs when present.
|
- Stored URLs are optional and must be absolute HTTP or HTTPS URLs when present.
|
||||||
- Comparison returns outcomes and reasons; it does not mutate storage.
|
|
||||||
- `distributor_version` is diagnostic metadata, not a comparison key.
|
- `distributor_version` is diagnostic metadata, not a comparison key.
|
||||||
|
|||||||
@@ -1,75 +1,56 @@
|
|||||||
# Storage
|
# Storage Internals
|
||||||
|
|
||||||
|
Audience: developers and LLM coding agents changing `internal/storage`, storage adapters, or storage-backed callers.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
`internal/storage` defines backend-rooted logical file access for core packages. Callers use slash-separated paths relative to a configured backend root.
|
`internal/storage` defines backend-rooted logical file access, path validation, typed storage errors, traversal helpers, backend registration, managed deletion targets, and test fake storage behavior.
|
||||||
|
|
||||||
## Inputs and outputs
|
## Inputs And Outputs
|
||||||
|
|
||||||
The storage interface supports byte reads, stream reads, byte writes, stream writes, exact metadata lookup, traversal, destination emptiness checks, guarded managed deletion, and bounded prefix deletion for explicit forced replacement.
|
Inputs are contexts, logical paths or prefixes, byte slices or readers, write options, walk options, delete options, and backend open configs. Outputs are file bytes, readers, `Entry` metadata, walk callbacks, boolean content checks, registered backends, and typed errors.
|
||||||
|
|
||||||
Entries report a logical path, type, and size when available. Entry types are `file`, `directory`, `symlink`, and `other`.
|
|
||||||
|
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
Core packages should depend on `internal/storage`, not adapter packages. Adapter-specific path handling stays behind backend implementations.
|
Core packages depend on `internal/storage`, not concrete adapters. Adapter protocol behavior belongs in `internal/adapters/local`, `internal/adapters/ssh`, and `internal/adapters/s3`; external SSH/SFTP and S3 notes live under `docs/integrations/`.
|
||||||
|
|
||||||
The local adapter lives in `internal/adapters/local`. The SSH/SFTP adapter lives in `internal/adapters/ssh`. The S3-compatible adapter lives in `internal/adapters/s3`. Runtime backend construction is wired through the app-level backend factory and storage registry. The fake backend lives in `internal/storage/fake` for tests and is not registered for runtime use.
|
Runtime backend construction and registration are owned by `internal/app`. The fake backend is for tests only.
|
||||||
|
|
||||||
## Paths
|
## Config Fields Used
|
||||||
|
|
||||||
Logical file paths must be non-empty, relative, clean, slash-separated, and must not contain `.` or `..` segments or backslashes. Prefix paths follow the same rules, except an empty prefix means the backend root.
|
The storage package does not read config directly. App adapter wiring converts config fields into backend open config values.
|
||||||
|
|
||||||
## Failure behavior
|
## Adapters Used
|
||||||
|
|
||||||
Storage errors use typed categories such as not found, already exists, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates rather than matching error strings.
|
Local, SSH/SFTP, and S3-compatible adapters implement `storage.Backend`. `internal/storage/fake` implements the same interface for tests.
|
||||||
|
|
||||||
Backends may wrap implementation-specific errors, but callers should receive storage errors where practical. Traversal can stop cleanly with `ErrStopWalk`.
|
## State And Manifest Behavior
|
||||||
|
|
||||||
## Traversal helpers
|
Storage owns `.distributor.json` path helpers through `StateFileName`, `StatePath`, and `ManagedBundleTargets`. It does not parse source manifests or destination state.
|
||||||
|
|
||||||
Backends own their traversal mechanics. The local adapter owns filesystem walking, the SSH adapter owns SFTP directory walking, and the S3 adapter owns object listing and pagination.
|
Logical paths are slash-separated and relative to a backend root. Prefix validation allows an empty prefix to mean the backend root; file path validation requires a non-empty path.
|
||||||
|
|
||||||
`internal/storage` owns the shared callback emission rules used by backends:
|
## Skip And Resume Behavior
|
||||||
|
|
||||||
- context cancellation is checked before callback emission;
|
Storage has no publication skip policy. It supplies `HasAny` for unmanaged-content checks, `DeleteManagedBundle` target construction for normal replacement cleanup, and `DeletePrefix` semantics for explicit forced replacement.
|
||||||
- `WalkOptions.Limit` bounds the number of emitted entries;
|
|
||||||
- `ErrStopWalk` stops traversal without becoming a caller-visible error;
|
|
||||||
- callback errors are wrapped as storage walk errors.
|
|
||||||
|
|
||||||
`storage.HasAny(ctx, backend, prefix)` provides the shared destination-content check. It calls `Walk` with non-recursive, limit-one traversal and stops after the first emitted entry.
|
## Failure Behavior
|
||||||
|
|
||||||
## Deletion
|
Storage errors use typed categories: not found, already exists, not empty, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates instead of matching strings. Traversal can stop cleanly with `ErrStopWalk`.
|
||||||
|
|
||||||
`DeleteManagedBundle` may delete listed managed outputs plus `.distributor.json`.
|
## Tests To Inspect
|
||||||
|
|
||||||
`DeletePrefix` removes content at and below a logical prefix for explicit forced replacement. It must not delete above the requested prefix or above the configured backend root.
|
- `internal/storage/*_test.go`
|
||||||
|
- `internal/storage/fake/*_test.go`
|
||||||
|
- `internal/adapters/local/*_test.go`
|
||||||
|
- `internal/adapters/ssh/*_test.go`
|
||||||
|
- `internal/adapters/s3/*_test.go`
|
||||||
|
|
||||||
## Local, SSH, S3, and fake backends
|
## Architectural Invariants
|
||||||
|
|
||||||
The local adapter maps logical paths to a configured filesystem root and keeps adapter-specific path handling behind the storage interface.
|
- Logical paths are clean relative slash-separated paths confined to the backend root.
|
||||||
|
- Core packages never import concrete adapters.
|
||||||
The SSH adapter maps logical paths to a configured remote SFTP root. It uses native SSH and SFTP libraries, supports SSH agent and key-file authentication, applies host-key policies, rejects unsafe logical paths, reports symlink entries from `Lstat`, and limits deletion to managed targets or explicit bounded prefixes.
|
- `storage.List` returns deterministic sorted entries.
|
||||||
|
- Managed deletion targets are recorded outputs plus `.distributor.json`.
|
||||||
The S3 adapter maps logical paths to object keys below a configured bucket and optional prefix. It uses the AWS SDK for Go v2, treats prefixes as object trees, requires exact objects for `Stat`, paginates traversal, applies conservative overwrite checks with `HeadObject`, infers basic content types, and limits deletion to managed target objects or explicit bounded object-key prefixes.
|
- Prefix deletion is bounded to the requested logical prefix.
|
||||||
|
- Runtime registration remains app-owned.
|
||||||
The fake backend is an in-memory implementation for package tests. It is not registered for runtime use.
|
|
||||||
|
|
||||||
## Tests
|
|
||||||
|
|
||||||
Before changing storage behavior, inspect tests under:
|
|
||||||
|
|
||||||
- `internal/storage`
|
|
||||||
- `internal/storage/fake`
|
|
||||||
- `internal/adapters/local`
|
|
||||||
- `internal/adapters/ssh`
|
|
||||||
- `internal/adapters/s3`
|
|
||||||
|
|
||||||
## Invariants
|
|
||||||
|
|
||||||
- Core packages depend on `internal/storage`, not concrete adapters.
|
|
||||||
- Logical paths are slash-separated and confined to the backend root.
|
|
||||||
- `storage.List` uses backend traversal and returns deterministic entries.
|
|
||||||
- Managed deletion is limited to recorded outputs plus `.distributor.json`.
|
|
||||||
- Prefix deletion is limited to the requested logical prefix.
|
|
||||||
- Runtime backend registration is owned by `internal/app`.
|
|
||||||
|
|||||||
@@ -1,47 +1,52 @@
|
|||||||
# Transform
|
# Transform Internals
|
||||||
|
|
||||||
|
Audience: developers and LLM coding agents changing `internal/transform` or transform implementations.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
`internal/transform` defines generated publication artifacts. `internal/transform/markdown` implements Markdown-to-HTML generation.
|
`internal/transform` defines generated publication artifacts, transform request/response types, transform registry behavior, and transform identifiers. `internal/transform/markdown` implements Markdown-to-HTML generation.
|
||||||
|
|
||||||
## Inputs and outputs
|
## Inputs And Outputs
|
||||||
|
|
||||||
Inputs are a validated source bundle, source backend, and transform options supplied by publish planning. Outputs include destination path, source path, transform id, generated bytes, SHA-256, and size.
|
Inputs are a validated source bundle, source storage backend, and transform options supplied by publish planning. Outputs are generated artifact records containing destination path, source path, transform id, generated bytes, SHA-256 digest, and byte size.
|
||||||
|
|
||||||
## Registry
|
|
||||||
|
|
||||||
`internal/transform` defines the transform interface and registry. The app layer registers the Markdown implementation; publish planning receives only a resolver.
|
|
||||||
|
|
||||||
## Markdown behavior
|
|
||||||
|
|
||||||
Markdown sidecar mode renders files ending in `.md` to `.html` files in the same logical directory. Markdown index mode renders one selected manifest-listed Markdown file to `index.html`. Non-Markdown files do not generate sidecar outputs. Raw HTML embedded in Markdown is not passed through by the renderer.
|
|
||||||
|
|
||||||
Generated HTML is deterministic for the same source content and transform configuration.
|
|
||||||
|
|
||||||
See `docs/integrations/markdown.md` for the Goldmark integration contract.
|
|
||||||
|
|
||||||
## Failure behavior
|
|
||||||
|
|
||||||
Transform resolution fails when a requested transform id is not registered. Markdown rendering fails when the source file cannot be read or rendered. Index input selection fails when the configured input is unsafe, not listed, not Markdown, or when no configured input can be inferred from exactly one manifest-listed Markdown file. Publish planning fails when HTML output is requested and the selected transform produces no outputs for a bundle.
|
|
||||||
|
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
Transforms do not publish files, mutate source bundles, or write destination state. Publish planning selects and writes transform outputs.
|
Transforms do not mutate source bundles, publish files, write destination state, choose destination actions, parse config, or inspect destinations. Publish planning decides whether generated outputs are selected and writes destination state later.
|
||||||
|
|
||||||
The app layer owns default transform registration. The transform package does not import concrete transform implementations.
|
The Goldmark renderer contract is documented in `docs/integrations/markdown.md`.
|
||||||
|
|
||||||
## Tests
|
## Config Fields Used
|
||||||
|
|
||||||
Before changing transform behavior, inspect tests under:
|
Transform packages do not read config directly. Publish planning passes effective `transform.markdown_to_html.mode` and `transform.markdown_to_html.input` values.
|
||||||
|
|
||||||
- `internal/transform`
|
## Adapters Used
|
||||||
- `internal/transform/markdown`
|
|
||||||
|
|
||||||
## Invariants
|
Transforms read source files through `internal/storage.Backend`. The Markdown implementation uses `github.com/yuin/goldmark` for rendering.
|
||||||
|
|
||||||
|
## State And Manifest Behavior
|
||||||
|
|
||||||
|
Transform outputs carry metadata later projected into destination state. Markdown sidecar mode renders manifest-listed `.md` files to same-directory `.html` outputs. Markdown index mode renders one selected Markdown source to `index.html`.
|
||||||
|
|
||||||
|
## Skip And Resume Behavior
|
||||||
|
|
||||||
|
Transforms have no skip/resume state. They are deterministic for the same source bytes and transform options.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Registry registration fails for empty names, nil transformers, and duplicate names. Transform resolution fails when publish planning requests an unregistered transform. Markdown rendering fails on source read errors, renderer errors, unsafe configured input, missing manifest input, non-Markdown input, ambiguous index input, or absent Markdown inputs.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/transform/*_test.go`
|
||||||
|
- `internal/transform/markdown/*_test.go`
|
||||||
|
- `internal/publish/*_test.go`
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
- Source bundle files are never mutated by transforms.
|
- Source bundle files are never mutated by transforms.
|
||||||
- Generated outputs record destination path, source path, transform id, SHA-256, and size.
|
- Generated outputs record destination path, source path, transform id, SHA-256, and size.
|
||||||
- Markdown sidecar naming changes only the `.md` extension to `.html`.
|
- Markdown sidecar naming changes only the `.md` suffix to `.html`.
|
||||||
- Markdown index mode always writes `index.html`.
|
- Markdown index mode always writes `index.html`.
|
||||||
- Non-Markdown source files do not generate Markdown outputs.
|
- Non-Markdown source files do not generate sidecar outputs.
|
||||||
- Transform registration stays outside publish planning.
|
- Transform registration stays outside publish planning.
|
||||||
|
|||||||
Reference in New Issue
Block a user