From 0818733b194266e4afcf9d37cb0c2a93d4421c73 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 30 May 2026 20:33:53 -0500 Subject: [PATCH] Final cleanup before beginning the MVP implementation --- .gitignore | 7 ++++++- README.md | 5 ++++- docs/roadmap/config.md | 6 +++--- docs/roadmap/contracts.md | 10 +++++----- docs/roadmap/storage.md | 41 ++++++++++++++++++++++++++++++++++++--- 5 files changed, 56 insertions(+), 13 deletions(-) diff --git a/.gitignore b/.gitignore index 3dea460..7397970 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,7 @@ +# distributor binary and testing workspace +/distributor +/workspace + # ---> Go # If you prefer the allow list template instead of the deny list, see community template: # https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore @@ -47,7 +51,8 @@ go.work.sum .LSOverride # Icon must end with two \r -Icon +Icon + # Thumbnails ._* diff --git a/README.md b/README.md index 24a9e4e..ee53587 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,5 @@ -# go-application-template +# distributor +`distributor` is a planned Go application for validating and publishing manifested Markdown bundles. + +Implementation has not started yet. Current design and implementation planning lives under `docs/roadmap/`. diff --git a/docs/roadmap/config.md b/docs/roadmap/config.md index 534e552..7af2da5 100644 --- a/docs/roadmap/config.md +++ b/docs/roadmap/config.md @@ -477,7 +477,7 @@ pipelines: destinations: - id: markdown-archive backend: s3 - endpoint: https://s3.maximumdirect.net + endpoint: https://s3.example.com bucket: reports prefix: weather/archive region: us-east-1 @@ -491,7 +491,7 @@ pipelines: - id: static-site backend: ssh - uri: ssh://deploy@web.maximumdirect.net:22 + uri: ssh://deploy@web.example.com:22 path: /srv/www/weather publish: source: false @@ -514,7 +514,7 @@ pipelines: destinations: - id: private-markdown-archive backend: s3 - endpoint: https://s3.maximumdirect.net + endpoint: https://s3.example.com bucket: reports prefix: dnd/session-recaps region: us-east-1 diff --git a/docs/roadmap/contracts.md b/docs/roadmap/contracts.md index 34d27e5..0af8107 100644 --- a/docs/roadmap/contracts.md +++ b/docs/roadmap/contracts.md @@ -36,12 +36,12 @@ A source bundle is a directory containing a `manifest.json` file. For MVP, the m { "schema_version": 1, "id": "weather.daily.brentwood.2026-05-30", - "digest": "sha256:...", + "digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000", "created": "2026-05-30T11:10:00Z", "files": [ { "path": "report.md", - "sha256": "sha256:...", + "sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "size": 12345 } ] @@ -168,12 +168,12 @@ A destination bundle path is considered distributor-managed only when it contain "manifest": { "schema_version": 1, "id": "weather.daily.brentwood.2026-05-30", - "digest": "sha256:...", + "digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000", "created": "2026-05-30T11:10:00Z", "files": [ { "path": "report.md", - "sha256": "sha256:...", + "sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "size": 12345 } ] @@ -185,7 +185,7 @@ A destination bundle path is considered distributor-managed only when it contain "kind": "generated", "source_path": "report.md", "transform": "markdown_to_html", - "sha256": "sha256:...", + "sha256": "sha256:3333333333333333333333333333333333333333333333333333333333333333", "size": 23456 } ] diff --git a/docs/roadmap/storage.md b/docs/roadmap/storage.md index 50898ad..600df9f 100644 --- a/docs/roadmap/storage.md +++ b/docs/roadmap/storage.md @@ -29,7 +29,9 @@ File paths: - must not contain backslashes; - must not resolve outside the backend root. -Prefix paths use the same slash-separated model. A prefix may be empty to represent the backend root for listing and destination emptiness checks. +Prefix paths use the same slash-separated model. A prefix may be empty to represent the backend root for traversal and destination emptiness checks. + +Prefix matching must preserve logical path boundaries. A prefix of `foo` matches `foo` and entries below `foo/`; it must not match a sibling path such as `foobar`. Backends that map logical paths to object keys must apply the same normalized boundary rule after combining configured backend prefixes with caller-provided logical prefixes. Backends own conversion from logical paths to native paths or object keys. Core packages should not construct local filesystem paths, SFTP paths, or S3 object keys directly. @@ -54,6 +56,34 @@ Byte helpers are expected to cover manifests, destination state, small source ar Write operations should create required parent directories or prefixes as needed. +Concrete option and callback types should use this shape: + +```go +type WalkOptions struct { + Recursive bool + Limit int +} + +type WalkFunc func(Entry) error + +var ErrStopWalk = errors.New("stop walk") + +type WriteOptions struct { + ContentType string + Overwrite bool + PreferAtomic bool + Size int64 + SizeKnown bool +} + +type DeleteOptions struct { + IgnoreMissing bool + PruneEmptyDirs bool +} +``` + +`WalkOptions.Limit == 0` means no explicit limit. `SizeKnown` applies primarily to `WriteFrom`; byte writes can infer size from the provided data. + ## Entries and Metadata Storage metadata should be represented by an `Entry` model with at least: @@ -69,13 +99,15 @@ Entry types: - `symlink`: local filesystem symlink; - `other`: unknown or unsupported native entry type. -`Stat` returns metadata for one logical path. +`Stat` returns metadata for one exact logical path. It may report a real filesystem directory, symlink, file, or exact object. It must not synthesize S3-like directory metadata solely because objects exist below a prefix; callers that need prefix existence or destination emptiness must use `HasAny` or `Walk`. `Walk` traverses entries below a prefix and calls a callback for each entry. `WalkOptions` should include: - whether traversal is recursive; - an optional entry limit for callers that only need to know whether content exists. +If a callback returns `ErrStopWalk`, traversal stops successfully and `Walk` returns nil. Any other callback error stops traversal and is returned with storage context where practical. If `WalkOptions.Limit` is greater than zero, reaching the limit stops traversal successfully. + Backends may stream or paginate traversal internally. S3-compatible adapters should not need to load a whole prefix into memory to satisfy traversal. Raw traversal is not required to be lexically sorted. A helper that materializes walk results for bundle discovery, tests, or CLI output should sort entries lexically by logical path before returning them. @@ -102,7 +134,8 @@ Both read methods must: - content type, when the destination backend can use it; - overwrite permission; -- atomic or staged write preference. +- atomic or staged write preference; +- optional known size for stream writes. Backends own staging and atomic behavior where practical: @@ -110,6 +143,8 @@ Backends own staging and atomic behavior where practical: - SSH/SFTP backend should use a temporary remote file and rename where available. - S3-compatible backend treats a successful object PUT as publish-on-success and applies content type metadata. +Remote adapters may buffer or spool `WriteFrom` input when needed to satisfy backend requirements such as content length, multipart upload, or retry behavior. Callers that know the stream size should set `SizeKnown` and `Size`. + If overwrite is false and the target exists, writes should fail with an already-exists error. `WriteFile` and `WriteFrom` should return the written `Entry`, including final path and size where available.