10 Commits

44 changed files with 1840 additions and 1098 deletions

View File

@@ -26,9 +26,22 @@ The runner:
Destination failures are collected while later destinations continue to run. Source open and source discovery failures stop the run because there are no valid bundles to fan out.
## Run implementation
`run.go` contains the public `Run` entrypoint and the main configuration orchestration path. Package-local run helpers are grouped by responsibility:
- `run_selection.go`: destination bundle selection, path mapping decisions, and fixed-path warnings;
- `run_warnings.go`: secret and SSH warning data;
- `run_output.go`: text plan lines, JSON action records, and output projections;
- `run_summary.go`: summary counters and JSON summary records;
- `run_failures.go`: destination failure aggregation and partial-result detection;
- `run_notify.go`: notification event projection and action filtering.
These helpers remain in `internal/app` because command output, warning collection, destination failure aggregation, notifier handoff, and backend construction are app-owned orchestration concerns.
## Backend and transform wiring
The app-level backend factory registers local, SSH, and S3 backends for execution. S3 explicit credential references are resolved through the config environment resolver before adapter construction.
The app-level backend factory registers local, SSH, and S3 backends for execution. Source and destination backend config is converted through a shared app-local open spec before adapter construction. S3 explicit credential references are resolved through the config environment resolver.
The app-level transform registry registers Markdown-to-HTML using `internal/transform/markdown`. Lower-level publish code receives a resolver and does not import concrete transform implementations.

View File

@@ -20,6 +20,8 @@ The source manifest requires:
Each file requires `path`, `sha256`, and `size`. Digests must use lowercase `sha256:<64 hex>` format. `created` must parse as RFC3339.
`pkg/bundle.ValidateDigest` is the canonical digest format validator for producer-facing and internal code. `internal/bundle.ValidateDigest` delegates to that public validator so source manifests and destination state use the same digest grammar.
## Validation
`pkg/bundle.ValidateManifest` owns normalized source manifest semantics: schema version, id, digest format, timestamp presence, file list presence, source path safety, duplicate file paths, reserved paths, file digest format, non-negative file sizes, and the top-level bundle digest.

27
docs/internal/link.md Normal file
View File

@@ -0,0 +1,27 @@
# Link URL Policy
## Purpose
`internal/link` defines shared validation for configured and persisted HTTP link URLs.
## Inputs and outputs
Input is a URL string. Output is either nil for an accepted URL or a concise validation error that callers wrap with field context.
## Validation behavior
Accepted URLs must parse successfully, use `http` or `https`, include a host, and omit query strings and fragments.
## Boundaries
This package validates URL shape only. It does not construct destination output URLs, choose primary URLs, infer public URLs from backend configuration, or read configuration files.
## Tests
Before changing link URL policy, inspect tests under `internal/link` and callers in `internal/config`, `internal/state`, and `internal/publish`.
## Invariants
- Configured `links.base_url`, persisted `links.primary_url`, persisted output `url`, and publish link planning use the same URL policy.
- Callers own field-specific error context.
- URL path construction remains in `internal/publish`.

View File

@@ -26,6 +26,8 @@ The package publishes source files and Markdown-to-HTML outputs. Markdown sideca
The package uses `internal/state` for destination comparison, `internal/storage` for IO, and the shared `internal/config` publish/transform policy helper for request validation. It resolves transforms through a narrow resolver supplied by the caller; concrete transform registration is owned by the app layer. It does not parse CLI flags, load config files, or choose which source bundles a destination receives.
The package owns projection from planned publish outputs to destination state output records and managed destination output paths. App JSON results and notification events keep their own schemas, but may use the publish output projection to avoid field-mapping drift.
The app layer computes the destination bundle path before planning. `preserve_relative` destinations pass the source-root-relative bundle path. `fixed` destinations pass an empty destination bundle path, which means the destination backend root, and pass only the newest selected source bundle for that destination.
When link config is present, publish planning builds per-output URLs from `links.base_url`, the destination bundle path, and each output path. `index.html` outputs use directory-style URLs. The primary URL is selected from planned outputs according to the destination primary policy.

View File

@@ -26,6 +26,19 @@ Storage errors use typed categories such as not found, already exists, invalid p
Backends may wrap implementation-specific errors, but callers should receive storage errors where practical. Traversal can stop cleanly with `ErrStopWalk`.
## Traversal helpers
Backends own their traversal mechanics. The local adapter owns filesystem walking, the SSH adapter owns SFTP directory walking, and the S3 adapter owns object listing and pagination.
`internal/storage` owns the shared callback emission rules used by backends:
- context cancellation is checked before callback emission;
- `WalkOptions.Limit` bounds the number of emitted entries;
- `ErrStopWalk` stops traversal without becoming a caller-visible error;
- callback errors are wrapped as storage walk errors.
`storage.HasAny(ctx, backend, prefix)` provides the shared destination-content check. It calls `Walk` with non-recursive, limit-one traversal and stops after the first emitted entry.
## Deletion
`DeleteManagedBundle` may delete listed managed outputs plus `.distributor.json`.

View File

@@ -101,7 +101,7 @@ Do not edit `.distributor.json` by hand during normal operation. If it is missin
## Go Producer Bundles
Go producer applications can import `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to create complete local source bundles with the same path, digest, timestamp, and validation rules used by `distributor`.
Go producer applications can import `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to create complete local source bundles with the same path, digest, timestamp, and validation rules used by `distributor`. The package also exposes digest helpers, including `ValidateDigest`, for producer code that needs to validate lowercase `sha256:<64 hex>` strings before writing manifests.
Minimal producer-side bundle creation:

View File

@@ -202,6 +202,7 @@ Use this current layout unless the project has a documented reason to differ:
- `internal/config`: configuration structs, defaults, loading, precedence, and validation.
- `internal/bundle`: storage-backed source bundle discovery and validation over the public manifest contract.
- `internal/state`: `.distributor.json` parsing, validation, comparison, and output metadata.
- `internal/link`: shared HTTP URL validation for configured and persisted link metadata.
- `internal/storage`: backend interfaces, shared path/resource types, backend registry, and storage errors.
- `internal/adapters/local`: local filesystem backend.
- `internal/adapters/ssh`: SSH/SFTP backend.

View File

@@ -161,31 +161,10 @@ func (b *Backend) Walk(ctx context.Context, prefix string, opts storage.WalkOpti
}
return b.translateError(storage.OpWalk, prefix, err)
}
visited := 0
emit := func(entry storage.Entry) error {
if err := ctx.Err(); err != nil {
return err
}
if opts.Limit > 0 && visited >= opts.Limit {
return storage.ErrStopWalk
}
visited++
if err := fn(entry); err != nil {
if errors.Is(err, storage.ErrStopWalk) {
return storage.ErrStopWalk
}
return storage.NewError(storage.OpWalk, backendName, entry.Path, storage.ErrUnknown, err)
}
return nil
}
emitter := storage.NewWalkEmitter(ctx, backendName, opts, fn)
if !info.IsDir() {
if err := emit(entryFromInfo(prefix, info)); errors.Is(err, storage.ErrStopWalk) {
return nil
} else if err != nil {
return err
}
return nil
return storage.FinishWalk(emitter.Emit(entryFromInfo(prefix, info)))
}
walkErr := filepath.WalkDir(nativePrefix, func(nativePath string, dirEntry fs.DirEntry, err error) error {
@@ -210,24 +189,13 @@ func (b *Backend) Walk(ctx context.Context, prefix string, opts storage.WalkOpti
if err != nil {
return b.translateError(storage.OpWalk, logicalPath, err)
}
return emit(entryFromInfo(logicalPath, info))
return emitter.Emit(entryFromInfo(logicalPath, info))
})
if errors.Is(walkErr, storage.ErrStopWalk) {
return nil
}
return walkErr
return storage.FinishWalk(walkErr)
}
func (b *Backend) HasAny(ctx context.Context, prefix string) (bool, error) {
found := false
err := b.Walk(ctx, prefix, storage.WalkOptions{Recursive: false, Limit: 1}, func(storage.Entry) error {
found = true
return storage.ErrStopWalk
})
if err != nil {
return false, err
}
return found, nil
return storage.HasAny(ctx, b, prefix)
}
func (b *Backend) DeleteManagedBundle(ctx context.Context, bundlePath string, managedOutputPaths []string, opts storage.DeleteOptions) error {

View File

@@ -183,55 +183,25 @@ func (b *Backend) Walk(ctx context.Context, logicalPrefix string, opts storage.W
if err := storage.ValidatePrefix(logicalPrefix); err != nil {
return err
}
visited := 0
emit := func(entry storage.Entry) error {
if err := ctx.Err(); err != nil {
return err
}
if opts.Limit > 0 && visited >= opts.Limit {
return storage.ErrStopWalk
}
visited++
if err := fn(entry); err != nil {
if errors.Is(err, storage.ErrStopWalk) {
return storage.ErrStopWalk
}
return storage.NewError(storage.OpWalk, BackendName, entry.Path, storage.ErrUnknown, err)
}
return nil
}
emitter := storage.NewWalkEmitter(ctx, BackendName, opts, fn)
if logicalPrefix != "" {
entry, err := b.Stat(ctx, logicalPrefix)
if err == nil {
if err := emit(entry); errors.Is(err, storage.ErrStopWalk) {
return nil
} else if err != nil {
return err
if err := emitter.Emit(entry); err != nil {
return storage.FinishWalk(err)
}
if opts.Limit > 0 && visited >= opts.Limit {
if emitter.LimitReached() {
return nil
}
} else if !storage.IsNotFound(err) {
return err
}
}
err := b.walkObjects(ctx, logicalPrefix, opts, emit)
if errors.Is(err, storage.ErrStopWalk) {
return nil
}
return err
return storage.FinishWalk(b.walkObjects(ctx, logicalPrefix, opts, emitter.Emit))
}
func (b *Backend) HasAny(ctx context.Context, logicalPrefix string) (bool, error) {
found := false
err := b.Walk(ctx, logicalPrefix, storage.WalkOptions{Recursive: false, Limit: 1}, func(storage.Entry) error {
found = true
return storage.ErrStopWalk
})
if err != nil {
return false, err
}
return found, nil
return storage.HasAny(ctx, b, logicalPrefix)
}
func (b *Backend) DeleteManagedBundle(ctx context.Context, bundlePath string, managedOutputPaths []string, opts storage.DeleteOptions) error {

View File

@@ -209,51 +209,17 @@ func (b *Backend) Walk(ctx context.Context, prefix string, opts storage.WalkOpti
return b.translateError(storage.OpWalk, prefix, err)
}
visited := 0
emit := func(entry storage.Entry) error {
if err := ctx.Err(); err != nil {
return err
}
if opts.Limit > 0 && visited >= opts.Limit {
return storage.ErrStopWalk
}
visited++
if err := fn(entry); err != nil {
if errors.Is(err, storage.ErrStopWalk) {
return storage.ErrStopWalk
}
return storage.NewError(storage.OpWalk, BackendName, entry.Path, storage.ErrUnknown, err)
}
return nil
}
emitter := storage.NewWalkEmitter(ctx, BackendName, opts, fn)
if !info.IsDir() {
if err := emit(entryFromInfo(prefix, info)); errors.Is(err, storage.ErrStopWalk) {
return nil
} else if err != nil {
return err
}
return nil
return storage.FinishWalk(emitter.Emit(entryFromInfo(prefix, info)))
}
if err := b.walkDirectory(ctx, prefix, nativePrefix, opts, emit); errors.Is(err, storage.ErrStopWalk) {
return nil
} else if err != nil {
return err
}
return nil
return storage.FinishWalk(b.walkDirectory(ctx, prefix, nativePrefix, opts, emitter.Emit))
}
func (b *Backend) HasAny(ctx context.Context, prefix string) (bool, error) {
found := false
err := b.Walk(ctx, prefix, storage.WalkOptions{Recursive: false, Limit: 1}, func(storage.Entry) error {
found = true
return storage.ErrStopWalk
})
if err != nil {
return false, err
}
return found, nil
return storage.HasAny(ctx, b, prefix)
}
func (b *Backend) DeleteManagedBundle(ctx context.Context, bundlePath string, managedOutputPaths []string, opts storage.DeleteOptions) error {

View File

@@ -37,6 +37,22 @@ type backendFactory struct {
readOnlyKnownHosts bool
}
type backendOpenSpec struct {
role string
backend string
path string
host string
user string
port int
ssh config.SSH
endpoint string
bucket string
prefix string
region string
forcePath *bool
credentials config.Credentials
}
func newBackendFactory() *backendFactory {
return newBackendFactoryWithEnvironment(config.ProcessEnvironment())
}
@@ -91,25 +107,11 @@ func newBackendFactoryWithEnvironment(environment config.Environment) *backendFa
}
func (f *backendFactory) openSource(ctx context.Context, source config.Backend) (storage.Backend, error) {
if source.Backend != config.BackendLocal && source.Backend != config.BackendSSH && source.Backend != config.BackendS3 {
return nil, fmt.Errorf("source backend %s is not implemented for execution", source.Backend)
}
openConfig, err := f.sourceOpenConfig(source)
if err != nil {
return nil, err
}
return f.registry.Open(ctx, source.Backend, openConfig)
return f.openBackend(ctx, backendOpenSpecFromSource(source))
}
func (f *backendFactory) openDestination(ctx context.Context, destination config.Destination) (storage.Backend, error) {
if destination.Backend != config.BackendLocal && destination.Backend != config.BackendSSH && destination.Backend != config.BackendS3 {
return nil, fmt.Errorf("backend %s is not implemented for execution", destination.Backend)
}
openConfig, err := f.destinationOpenConfig(destination)
if err != nil {
return nil, err
}
return f.registry.Open(ctx, destination.Backend, openConfig)
return f.openBackend(ctx, backendOpenSpecFromDestination(destination))
}
func (f *backendFactory) openLocalPath(ctx context.Context, path string) (storage.Backend, error) {
@@ -120,40 +122,51 @@ func (f *backendFactory) resolveCredentials(creds config.Credentials) (config.Re
return f.environment.ResolveCredentials(creds)
}
func (f *backendFactory) sourceOpenConfig(source config.Backend) (storage.OpenConfig, error) {
cfg := sourceOpenConfig(source)
if source.Backend == config.BackendS3 {
if err := f.addS3Config(cfg, source.Endpoint, source.Bucket, source.Prefix, source.Region, source.ForcePath, source.Creds); err != nil {
func (f *backendFactory) openBackend(ctx context.Context, spec backendOpenSpec) (storage.Backend, error) {
if !backendExecutable(spec.backend) {
if spec.role == "source" {
return nil, fmt.Errorf("source backend %s is not implemented for execution", spec.backend)
}
return nil, fmt.Errorf("backend %s is not implemented for execution", spec.backend)
}
openConfig, err := f.openConfig(spec)
if err != nil {
return nil, err
}
return f.registry.Open(ctx, spec.backend, openConfig)
}
func backendExecutable(name string) bool {
return name == config.BackendLocal || name == config.BackendSSH || name == config.BackendS3
}
func (f *backendFactory) openConfig(spec backendOpenSpec) (storage.OpenConfig, error) {
cfg := storage.OpenConfig{storagePathKey: spec.path}
switch spec.backend {
case config.BackendSSH:
cfg[sshHostKey] = spec.host
cfg[sshUserKey] = spec.user
cfg[sshPortKey] = strconv.Itoa(spec.port)
cfg[sshKeyFileKey] = spec.ssh.KeyFile
cfg[sshKnownHostsKey] = spec.ssh.KnownHosts
cfg[sshHostKeyPolicyKey] = string(spec.ssh.HostKeyPolicy)
cfg[sshReadOnlyHostsKey] = strconv.FormatBool(f.readOnlyKnownHosts)
case config.BackendS3:
if err := f.addS3Config(cfg, spec); err != nil {
return nil, err
}
}
if source.Backend == config.BackendSSH {
cfg[sshReadOnlyHostsKey] = strconv.FormatBool(f.readOnlyKnownHosts)
}
return cfg, nil
}
func (f *backendFactory) destinationOpenConfig(destination config.Destination) (storage.OpenConfig, error) {
cfg := destinationOpenConfig(destination)
if destination.Backend == config.BackendS3 {
if err := f.addS3Config(cfg, destination.Endpoint, destination.Bucket, destination.Prefix, destination.Region, destination.ForcePath, destination.Creds); err != nil {
return nil, err
}
}
if destination.Backend == config.BackendSSH {
cfg[sshReadOnlyHostsKey] = strconv.FormatBool(f.readOnlyKnownHosts)
}
return cfg, nil
}
func (f *backendFactory) addS3Config(cfg storage.OpenConfig, endpoint, bucket, prefix, region string, forcePath *bool, creds config.Credentials) error {
cfg[s3EndpointKey] = endpoint
cfg[s3BucketKey] = bucket
cfg[s3PrefixKey] = prefix
cfg[s3RegionKey] = region
cfg[s3ForcePathStyleKey] = strconv.FormatBool(config.ForcePathStyle(forcePath))
if creds.AccessKeyIDEnv != "" || creds.SecretAccessKeyEnv != "" {
resolved, err := f.resolveCredentials(creds)
func (f *backendFactory) addS3Config(cfg storage.OpenConfig, spec backendOpenSpec) error {
cfg[s3EndpointKey] = spec.endpoint
cfg[s3BucketKey] = spec.bucket
cfg[s3PrefixKey] = spec.prefix
cfg[s3RegionKey] = spec.region
cfg[s3ForcePathStyleKey] = strconv.FormatBool(config.ForcePathStyle(spec.forcePath))
if spec.credentials.AccessKeyIDEnv != "" || spec.credentials.SecretAccessKeyEnv != "" {
resolved, err := f.resolveCredentials(spec.credentials)
if err != nil {
return err
}
@@ -163,30 +176,38 @@ func (f *backendFactory) addS3Config(cfg storage.OpenConfig, endpoint, bucket, p
return nil
}
func sourceOpenConfig(source config.Backend) storage.OpenConfig {
cfg := storage.OpenConfig{storagePathKey: source.Path}
if source.Backend == config.BackendSSH {
cfg[sshHostKey] = source.Host
cfg[sshUserKey] = source.User
cfg[sshPortKey] = strconv.Itoa(source.Port)
cfg[sshKeyFileKey] = source.SSH.KeyFile
cfg[sshKnownHostsKey] = source.SSH.KnownHosts
cfg[sshHostKeyPolicyKey] = string(source.SSH.HostKeyPolicy)
cfg[sshReadOnlyHostsKey] = "false"
func backendOpenSpecFromSource(source config.Backend) backendOpenSpec {
return backendOpenSpec{
role: "source",
backend: source.Backend,
path: source.Path,
host: source.Host,
user: source.User,
port: source.Port,
ssh: source.SSH,
endpoint: source.Endpoint,
bucket: source.Bucket,
prefix: source.Prefix,
region: source.Region,
forcePath: source.ForcePath,
credentials: source.Creds,
}
return cfg
}
func destinationOpenConfig(destination config.Destination) storage.OpenConfig {
cfg := storage.OpenConfig{storagePathKey: destination.Path}
if destination.Backend == config.BackendSSH {
cfg[sshHostKey] = destination.Host
cfg[sshUserKey] = destination.User
cfg[sshPortKey] = strconv.Itoa(destination.Port)
cfg[sshKeyFileKey] = destination.SSH.KeyFile
cfg[sshKnownHostsKey] = destination.SSH.KnownHosts
cfg[sshHostKeyPolicyKey] = string(destination.SSH.HostKeyPolicy)
cfg[sshReadOnlyHostsKey] = "false"
func backendOpenSpecFromDestination(destination config.Destination) backendOpenSpec {
return backendOpenSpec{
role: "destination",
backend: destination.Backend,
path: destination.Path,
host: destination.Host,
user: destination.User,
port: destination.Port,
ssh: destination.SSH,
endpoint: destination.Endpoint,
bucket: destination.Bucket,
prefix: destination.Prefix,
region: destination.Region,
forcePath: destination.ForcePath,
credentials: destination.Creds,
}
return cfg
}

View File

@@ -1,13 +1,16 @@
package app
import (
"bytes"
"context"
"fmt"
"strings"
"testing"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
"gitea.maximumdirect.net/eric/distributor/internal/testutil"
)
func TestBackendFactoryOpensLocalSource(t *testing.T) {
@@ -275,7 +278,8 @@ func TestBackendFactoryResolvesCredentialsThroughEnvironment(t *testing.T) {
}
func TestBackendFactoryBuildsSSHSourceOpenConfig(t *testing.T) {
cfg := sourceOpenConfig(config.Backend{
factory := &backendFactory{environment: config.NewEnvironment(nil, nil)}
cfg, err := factory.openConfig(backendOpenSpecFromSource(config.Backend{
Backend: config.BackendSSH,
Host: "source.example.com",
User: "reports",
@@ -286,7 +290,10 @@ func TestBackendFactoryBuildsSSHSourceOpenConfig(t *testing.T) {
KnownHosts: "/home/reports/.ssh/known_hosts",
HostKeyPolicy: config.HostKeyPolicyStrict,
},
})
}))
if err != nil {
t.Fatalf("openConfig() error = %v", err)
}
assertOpenConfig(t, cfg, map[string]string{
storagePathKey: "/reports",
@@ -300,7 +307,8 @@ func TestBackendFactoryBuildsSSHSourceOpenConfig(t *testing.T) {
}
func TestBackendFactoryBuildsSSHDestinationOpenConfig(t *testing.T) {
cfg := destinationOpenConfig(config.Destination{
factory := &backendFactory{environment: config.NewEnvironment(nil, nil)}
cfg, err := factory.openConfig(backendOpenSpecFromDestination(config.Destination{
Backend: config.BackendSSH,
Host: "destination.example.com",
User: "deploy",
@@ -309,7 +317,10 @@ func TestBackendFactoryBuildsSSHDestinationOpenConfig(t *testing.T) {
SSH: config.SSH{
HostKeyPolicy: config.HostKeyPolicyAcceptNew,
},
})
}))
if err != nil {
t.Fatalf("openConfig() error = %v", err)
}
assertOpenConfig(t, cfg, map[string]string{
storagePathKey: "/srv/archive",
@@ -320,6 +331,257 @@ func TestBackendFactoryBuildsSSHDestinationOpenConfig(t *testing.T) {
})
}
func TestBackendFactoryBuildsEquivalentSourceAndDestinationOpenConfig(t *testing.T) {
forcePathStyle := false
tests := []struct {
name string
source config.Backend
destination config.Destination
}{
{
name: "local",
source: config.Backend{Backend: config.BackendLocal, Path: "/reports"},
destination: config.Destination{Backend: config.BackendLocal, Path: "/reports"},
},
{
name: "ssh",
source: config.Backend{
Backend: config.BackendSSH,
Host: "reports.example.com",
User: "reports",
Port: 2222,
Path: "/reports",
SSH: config.SSH{
KeyFile: "/home/reports/.ssh/id_ed25519",
KnownHosts: "/home/reports/.ssh/known_hosts",
HostKeyPolicy: config.HostKeyPolicyStrict,
},
},
destination: config.Destination{
Backend: config.BackendSSH,
Host: "reports.example.com",
User: "reports",
Port: 2222,
Path: "/reports",
SSH: config.SSH{
KeyFile: "/home/reports/.ssh/id_ed25519",
KnownHosts: "/home/reports/.ssh/known_hosts",
HostKeyPolicy: config.HostKeyPolicyStrict,
},
},
},
{
name: "s3",
source: config.Backend{
Backend: config.BackendS3,
Endpoint: "https://s3.example.com",
Bucket: "reports",
Prefix: "archive",
Region: "us-west-2",
ForcePath: &forcePathStyle,
},
destination: config.Destination{
Backend: config.BackendS3,
Endpoint: "https://s3.example.com",
Bucket: "reports",
Prefix: "archive",
Region: "us-west-2",
ForcePath: &forcePathStyle,
},
},
{
name: "s3 explicit credentials",
source: config.Backend{
Backend: config.BackendS3,
Endpoint: "https://s3.example.com",
Bucket: "reports",
Region: config.DefaultS3Region,
Creds: config.Credentials{
AccessKeyIDEnv: "ACCESS_KEY_ID",
SecretAccessKeyEnv: "SECRET_ACCESS_KEY",
},
},
destination: config.Destination{
Backend: config.BackendS3,
Endpoint: "https://s3.example.com",
Bucket: "reports",
Region: config.DefaultS3Region,
Creds: config.Credentials{
AccessKeyIDEnv: "ACCESS_KEY_ID",
SecretAccessKeyEnv: "SECRET_ACCESS_KEY",
},
},
},
}
factory := &backendFactory{
environment: config.NewEnvironment(map[string]string{
"ACCESS_KEY_ID": "secret-access",
"SECRET_ACCESS_KEY": "secret-secret",
}, func(string) (string, bool) { return "", false }),
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
sourceConfig, err := factory.openConfig(backendOpenSpecFromSource(tt.source))
if err != nil {
t.Fatalf("source openConfig() error = %v", err)
}
destinationConfig, err := factory.openConfig(backendOpenSpecFromDestination(tt.destination))
if err != nil {
t.Fatalf("destination openConfig() error = %v", err)
}
if !openConfigEqual(sourceConfig, destinationConfig) {
t.Fatalf("source open config = %#v, destination open config = %#v, want equivalent", sourceConfig, destinationConfig)
}
})
}
}
func TestBackendFactoryBuildsEquivalentDryRunSSHOpenConfig(t *testing.T) {
factory := &backendFactory{readOnlyKnownHosts: true}
sourceConfig, err := factory.openConfig(backendOpenSpecFromSource(config.Backend{
Backend: config.BackendSSH,
Host: "reports.example.com",
Port: 22,
Path: "/reports",
SSH: config.SSH{HostKeyPolicy: config.HostKeyPolicyAcceptNew},
}))
if err != nil {
t.Fatalf("source openConfig() error = %v", err)
}
destinationConfig, err := factory.openConfig(backendOpenSpecFromDestination(config.Destination{
Backend: config.BackendSSH,
Host: "reports.example.com",
Port: 22,
Path: "/reports",
SSH: config.SSH{HostKeyPolicy: config.HostKeyPolicyAcceptNew},
}))
if err != nil {
t.Fatalf("destination openConfig() error = %v", err)
}
if !openConfigEqual(sourceConfig, destinationConfig) {
t.Fatalf("source open config = %#v, destination open config = %#v, want equivalent", sourceConfig, destinationConfig)
}
if sourceConfig[sshReadOnlyHostsKey] != "true" {
t.Fatalf("open config %s = %q, want true", sshReadOnlyHostsKey, sourceConfig[sshReadOnlyHostsKey])
}
}
func TestConfiguredSourceValidationAndRunUseEquivalentSourceOpenConfig(t *testing.T) {
tests := []struct {
name string
source config.Backend
sourceKey string
dest config.Destination
destKey string
wantFields map[string]string
}{
{
name: "s3",
source: config.Backend{
Backend: config.BackendS3,
Endpoint: "https://s3.example.com",
Bucket: "source-bucket",
Prefix: "source-prefix",
Region: config.DefaultS3Region,
},
sourceKey: "s3:source-bucket",
dest: config.Destination{
ID: "archive",
Backend: config.BackendS3,
Endpoint: "https://s3.example.com",
Bucket: "destination-bucket",
Region: config.DefaultS3Region,
},
destKey: "s3:destination-bucket",
wantFields: map[string]string{
s3EndpointKey: "https://s3.example.com",
s3BucketKey: "source-bucket",
s3PrefixKey: "source-prefix",
s3RegionKey: config.DefaultS3Region,
s3ForcePathStyleKey: "true",
},
},
{
name: "ssh",
source: config.Backend{
Backend: config.BackendSSH,
Host: "ssh.example.com",
User: "reports",
Port: 2222,
Path: "/source",
SSH: config.SSH{
KeyFile: "/home/reports/.ssh/id_ed25519",
KnownHosts: "/home/reports/.ssh/known_hosts",
HostKeyPolicy: config.HostKeyPolicyStrict,
},
},
sourceKey: "ssh:/source",
dest: config.Destination{
ID: "archive",
Backend: config.BackendSSH,
Host: "ssh.example.com",
Port: 2222,
Path: "/destination",
SSH: config.SSH{HostKeyPolicy: config.HostKeyPolicyStrict},
},
destKey: "ssh:/destination",
wantFields: map[string]string{
storagePathKey: "/source",
sshHostKey: "ssh.example.com",
sshUserKey: "reports",
sshPortKey: "2222",
sshKeyFileKey: "/home/reports/.ssh/id_ed25519",
sshKnownHostsKey: "/home/reports/.ssh/known_hosts",
sshHostKeyPolicyKey: "strict",
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
sourceBackend := fake.New()
testutil.WriteFakeSourceBundle(t, sourceBackend, "", testutil.BundleOptions{ID: "reports.source"})
destinationBackend := fake.New()
var validateSourceConfig storage.OpenConfig
var runSourceConfig storage.OpenConfig
validateProvider := recordingBackendFactoryProvider(t, map[string]storage.Backend{
tt.sourceKey: sourceBackend,
tt.destKey: destinationBackend,
}, func(cfg storage.OpenConfig) {
validateSourceConfig = cfg
})
runProvider := recordingBackendFactoryProvider(t, map[string]storage.Backend{
tt.sourceKey: sourceBackend,
tt.destKey: destinationBackend,
}, func(cfg storage.OpenConfig) {
runSourceConfig = cfg
})
cfg := config.Config{Pipelines: []config.Pipeline{{
ID: "reports",
Source: tt.source,
Destinations: []config.Destination{tt.dest},
}}}
config.ApplyDefaults(&cfg)
var validateOutput bytes.Buffer
if err := validateConfigWithBackendFactory(context.Background(), cfg, ValidateOptions{
PipelineID: "reports",
Stdout: &validateOutput,
}, validateProvider); err != nil {
t.Fatalf("validateConfigWithBackendFactory() error = %v", err)
}
if err := runConfigWithBackendFactory(context.Background(), cfg, RunOptions{}, runProvider); err != nil {
t.Fatalf("runConfigWithBackendFactory() error = %v", err)
}
if !openConfigEqual(validateSourceConfig, runSourceConfig) {
t.Fatalf("validate source config = %#v, run source config = %#v, want equivalent", validateSourceConfig, runSourceConfig)
}
assertOpenConfig(t, runSourceConfig, tt.wantFields)
})
}
}
func assertOpenConfig(t *testing.T, got map[string]string, want map[string]string) {
t.Helper()
for key, wantValue := range want {
@@ -328,3 +590,49 @@ func assertOpenConfig(t *testing.T, got map[string]string, want map[string]strin
}
}
}
func openConfigEqual(left, right storage.OpenConfig) bool {
if len(left) != len(right) {
return false
}
for key, leftValue := range left {
if right[key] != leftValue {
return false
}
}
return true
}
func recordingBackendFactoryProvider(t *testing.T, remoteBackends map[string]storage.Backend, recordSource func(storage.OpenConfig)) backendFactoryProvider {
t.Helper()
return func(environment config.Environment) *backendFactory {
registry := storage.NewRegistry()
if err := registry.Register(config.BackendS3, func(ctx context.Context, cfg storage.OpenConfig) (storage.Backend, error) {
if cfg[s3BucketKey] == "source-bucket" {
recordSource(cfg)
}
key := "s3:" + cfg[s3BucketKey]
backend := remoteBackends[key]
if backend == nil {
return nil, fmt.Errorf("missing fake backend for %s", key)
}
return backend, nil
}); err != nil {
t.Fatalf("register s3 backend: %v", err)
}
if err := registry.Register(config.BackendSSH, func(ctx context.Context, cfg storage.OpenConfig) (storage.Backend, error) {
if cfg[storagePathKey] == "/source" {
recordSource(cfg)
}
key := "ssh:" + cfg[storagePathKey]
backend := remoteBackends[key]
if backend == nil {
return nil, fmt.Errorf("missing fake backend for %s", key)
}
return backend, nil
}); err != nil {
t.Fatalf("register ssh backend: %v", err)
}
return &backendFactory{registry: registry, environment: environment}
}
}

View File

@@ -2,11 +2,8 @@ package app
import (
"context"
"errors"
"fmt"
"io"
"sort"
"strings"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/config"
@@ -260,431 +257,3 @@ func closeBackend(backend storage.Backend) {
}
_ = closeable.Close()
}
func writePlanLine(w io.Writer, backend string, plan publish.Plan, planErr error) {
if w == nil {
return
}
if planErr != nil {
destinationID := plan.DestinationID
if destinationID == "" {
destinationID = "unknown"
}
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s%s action=error reason=%q\n", storage.DisplayPath(plan.BundlePath), destinationID, backend, pathMappingSummary(plan), planErr.Error())
return
}
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s%s action=%s outputs=%s reason=%q\n", storage.DisplayPath(plan.BundlePath), plan.DestinationID, backend, pathMappingSummary(plan), plan.Action, outputSummary(plan.Outputs), plan.Reason)
}
func pathMappingSummary(plan publish.Plan) string {
if plan.PathMapping != config.PathMappingFixed {
return ""
}
return fmt.Sprintf(" path_mapping=fixed target=%s", storage.DisplayPath(plan.DestinationBundlePath))
}
func writeErrorLine(w io.Writer, bundlePath, destinationID, backend string, err error) {
if w == nil {
return
}
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s action=error reason=%q\n", storage.DisplayPath(bundlePath), destinationID, backend, err.Error())
}
func outputSummary(outputs []publish.Output) string {
if len(outputs) == 0 {
return "none"
}
paths := make([]string, 0, len(outputs))
for _, output := range outputs {
paths = append(paths, output.DestinationPath)
}
return strings.Join(paths, ",")
}
type destinationBundleSelection struct {
SourceBundle bundle.Bundle
DestinationBundlePath string
}
func selectDestinationBundles(destination config.Destination, bundles []bundle.Bundle) []destinationBundleSelection {
if !isFixedPathDestination(destination) {
selections := make([]destinationBundleSelection, 0, len(bundles))
for _, sourceBundle := range bundles {
selections = append(selections, destinationBundleSelection{
SourceBundle: sourceBundle,
DestinationBundlePath: sourceBundle.RootRelativePath,
})
}
return selections
}
if len(bundles) == 0 {
return nil
}
sourceBundle := newestBundle(bundles)
return []destinationBundleSelection{{
SourceBundle: sourceBundle,
DestinationBundlePath: "",
}}
}
func newestBundle(bundles []bundle.Bundle) bundle.Bundle {
if len(bundles) == 0 {
return bundle.Bundle{}
}
sorted := append([]bundle.Bundle(nil), bundles...)
sort.Slice(sorted, func(i, j int) bool {
if sorted[i].Manifest.Created.Equal(sorted[j].Manifest.Created) {
return sorted[i].RootRelativePath < sorted[j].RootRelativePath
}
return sorted[i].Manifest.Created.After(sorted[j].Manifest.Created)
})
return sorted[0]
}
func isFixedPathDestination(destination config.Destination) bool {
return destination.PathMap.Mode == config.PathMappingFixed
}
func fixedPathSelectionWarning(pipelineID, destinationID string, selections []destinationBundleSelection, candidateCount int) OutputWarning {
selected := "none"
if len(selections) > 0 {
selected = storage.DisplayPath(selections[0].SourceBundle.RootRelativePath)
}
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed candidates=%d selected_bundle=%s destination_bundle=.", pipelineID, destinationID, candidateCount, selected)}
}
func isDestructiveFixedPathAction(action publish.Action) bool {
return action == publish.ActionReplaceOlder || action == publish.ActionForceReplace
}
func fixedPathReplacementWarning(plan publish.Plan) OutputWarning {
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed action=%s replaces destination root for selected_bundle=%s", plan.PipelineID, plan.DestinationID, plan.Action, storage.DisplayPath(plan.BundlePath))}
}
func destinationIDs(destinations []config.Destination) []string {
ids := make([]string, 0, len(destinations))
for _, destination := range destinations {
ids = append(ids, destination.ID)
}
return ids
}
func destinationSummary(destinations []config.Destination) string {
if len(destinations) == 0 {
return "none"
}
ids := make([]string, 0, len(destinations))
for _, destination := range destinations {
ids = append(ids, destination.ID)
}
return strings.Join(ids, ",")
}
func writeSecretConflictWarnings(w io.Writer, conflicts []config.SecretConflict) error {
return writeWarnings(w, secretConflictWarnings(conflicts))
}
func secretConflictWarnings(conflicts []config.SecretConflict) []OutputWarning {
warnings := make([]OutputWarning, 0, len(conflicts))
for _, conflict := range conflicts {
warnings = append(warnings, OutputWarning{
Message: fmt.Sprintf("secret %s ignored because the real environment already has that variable", conflict.Name),
})
}
return warnings
}
func writeSSHWarnings(w io.Writer, pipeline config.Pipeline) error {
return writeWarnings(w, sshWarnings(pipeline))
}
func sshWarnings(pipeline config.Pipeline) []OutputWarning {
var warnings []OutputWarning
if pipeline.Source.Backend == config.BackendSSH && pipeline.Source.SSH.HostKeyPolicy == config.HostKeyPolicyOff {
warnings = append(warnings, OutputWarning{
Message: fmt.Sprintf("pipeline=%s source host_key_policy=off disables SSH host key checking", pipeline.ID),
})
}
for _, destination := range pipeline.Destinations {
if destination.Backend == config.BackendSSH && destination.SSH.HostKeyPolicy == config.HostKeyPolicyOff {
warnings = append(warnings, OutputWarning{
Message: fmt.Sprintf("pipeline=%s destination=%s host_key_policy=off disables SSH host key checking", pipeline.ID, destination.ID),
})
}
}
return warnings
}
func writeWarnings(w io.Writer, warnings []OutputWarning) error {
if w == nil {
return nil
}
for _, warning := range warnings {
if _, err := fmt.Fprintf(w, "Warning: %s\n", warning.Message); err != nil {
return err
}
}
return nil
}
func shouldNotify(action publish.Action) bool {
return action == publish.ActionPublishNew || action == publish.ActionReplaceOlder || action == publish.ActionForceReplace
}
func notifyEvent(plan publish.Plan) notify.Event {
outputs := make([]notify.Output, 0, len(plan.Outputs))
for _, output := range plan.Outputs {
outputs = append(outputs, notify.Output{
Path: output.DestinationPath,
Kind: output.Kind,
SourcePath: output.SourcePath,
Transform: output.Transform,
SHA256: output.SHA256,
Size: output.Size,
})
}
return notify.Event{
PipelineID: plan.PipelineID,
DestinationID: plan.DestinationID,
BundleID: plan.BundleID,
BundlePath: plan.BundlePath,
Action: string(plan.Action),
Outputs: outputs,
}
}
type runResult struct {
DryRun bool `json:"dry_run"`
Pipelines []runPipelineResult `json:"pipelines"`
Actions []runActionResult `json:"actions"`
Summary runSummaryResult `json:"summary"`
}
type runPipelineResult struct {
ID string `json:"id"`
SourceBackend string `json:"source_backend"`
BundleCount int `json:"bundle_count"`
Destinations []string `json:"destinations"`
}
type runActionResult struct {
PipelineID string `json:"pipeline_id,omitempty"`
DestinationID string `json:"destination_id"`
Backend string `json:"backend"`
BundleID string `json:"bundle_id,omitempty"`
BundlePath string `json:"bundle_path"`
DestinationPath string `json:"destination_path"`
PathMapping string `json:"path_mapping,omitempty"`
Action string `json:"action"`
PrimaryURL string `json:"primary_url,omitempty"`
Reason string `json:"reason,omitempty"`
Outputs []runOutputResult `json:"outputs"`
}
type runOutputResult struct {
Path string `json:"path"`
Kind string `json:"kind"`
SourcePath string `json:"source_path,omitempty"`
Transform string `json:"transform,omitempty"`
URL string `json:"url,omitempty"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
func runActionFromPlan(backend string, plan publish.Plan, planErr error) runActionResult {
if planErr != nil {
destinationID := plan.DestinationID
if destinationID == "" {
destinationID = "unknown"
}
return runActionResult{
PipelineID: plan.PipelineID,
DestinationID: destinationID,
Backend: backend,
BundleID: plan.BundleID,
BundlePath: storage.DisplayPath(plan.BundlePath),
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
PathMapping: plan.PathMapping,
Action: "error",
PrimaryURL: plan.PrimaryURL,
Reason: planErr.Error(),
Outputs: []runOutputResult{},
}
}
return runActionResult{
PipelineID: plan.PipelineID,
DestinationID: plan.DestinationID,
Backend: backend,
BundleID: plan.BundleID,
BundlePath: storage.DisplayPath(plan.BundlePath),
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
PathMapping: plan.PathMapping,
Action: string(plan.Action),
PrimaryURL: plan.PrimaryURL,
Reason: plan.Reason,
Outputs: runOutputsFromPlan(plan.Outputs),
}
}
func errorAction(pipelineID, destinationID, backend, bundlePath string, err error) runActionResult {
return runActionResult{
PipelineID: pipelineID,
DestinationID: destinationID,
Backend: backend,
BundlePath: storage.DisplayPath(bundlePath),
DestinationPath: storage.DisplayPath(bundlePath),
Action: "error",
Reason: err.Error(),
Outputs: []runOutputResult{},
}
}
func runOutputsFromPlan(outputs []publish.Output) []runOutputResult {
results := make([]runOutputResult, 0, len(outputs))
for _, output := range outputs {
results = append(results, runOutputResult{
Path: output.DestinationPath,
Kind: output.Kind,
SourcePath: output.SourcePath,
Transform: output.Transform,
URL: output.URL,
SHA256: output.SHA256,
Size: output.Size,
})
}
return results
}
type runSummary struct {
dryRun bool
planned int
publishNew int
replaceOlder int
forceReplace int
skipped int
failures int
fixedPath int
}
func (s *runSummary) recordPlan(action publish.Action) {
s.planned++
switch action {
case publish.ActionPublishNew:
s.publishNew++
case publish.ActionReplaceOlder:
s.replaceOlder++
case publish.ActionForceReplace:
s.forceReplace++
case publish.ActionSkipSame, publish.ActionSkipDestinationNewer:
s.skipped++
}
}
func (s *runSummary) recordFailure() {
s.failures++
}
func (s *runSummary) recordFixedPath() {
s.fixedPath++
}
func (s runSummary) Line() string {
status := "ok"
if s.failures > 0 {
status = "failed"
}
return fmt.Sprintf("Final status: %s planned=%d publish_new=%d replace_older=%d force_replace=%d skipped=%d failed=%d dry_run=%t fixed_path=%d", status, s.planned, s.publishNew, s.replaceOlder, s.forceReplace, s.skipped, s.failures, s.dryRun, s.fixedPath)
}
type runSummaryResult struct {
Status string `json:"status"`
Planned int `json:"planned"`
PublishNew int `json:"publish_new"`
ReplaceOlder int `json:"replace_older"`
ForceReplace int `json:"force_replace"`
Skipped int `json:"skipped"`
Failed int `json:"failed"`
DryRun bool `json:"dry_run"`
FixedPath int `json:"fixed_path"`
}
func (s runSummary) Result() runSummaryResult {
status := "ok"
if s.failures > 0 {
status = "failed"
}
return runSummaryResult{
Status: status,
Planned: s.planned,
PublishNew: s.publishNew,
ReplaceOlder: s.replaceOlder,
ForceReplace: s.forceReplace,
Skipped: s.skipped,
Failed: s.failures,
DryRun: s.dryRun,
FixedPath: s.fixedPath,
}
}
type runFailure struct {
pipelineID string
destinationID string
backend string
bundlePath string
err error
}
type runFailures struct {
items []runFailure
}
func (f *runFailures) add(pipelineID, destinationID, backend, bundlePath string, err error) {
f.items = append(f.items, runFailure{
pipelineID: pipelineID,
destinationID: destinationID,
backend: backend,
bundlePath: bundlePath,
err: err,
})
}
func (f runFailures) Error() string {
if len(f.items) == 0 {
return ""
}
parts := make([]string, 0, len(f.items))
for _, item := range f.items {
parts = append(parts, fmt.Sprintf("pipeline %s destination %s backend %s bundle %s: %v", item.pipelineID, item.destinationID, item.backend, item.bundlePath, item.err))
}
return "run failed: " + strings.Join(parts, "; ")
}
func (f runFailures) outputErrors() []OutputError {
if len(f.items) == 0 {
return nil
}
errors := make([]OutputError, 0, len(f.items))
for _, item := range f.items {
errors = append(errors, OutputError{
PipelineID: item.pipelineID,
DestinationID: item.destinationID,
Backend: item.backend,
BundlePath: item.bundlePath,
Message: item.err.Error(),
})
}
return errors
}
func IsPartialResultError(err error) bool {
var failures runFailures
return errors.As(err, &failures)
}
func (f runFailures) Unwrap() error {
errs := make([]error, 0, len(f.items))
for _, item := range f.items {
errs = append(errs, item.err)
}
return errors.Join(errs...)
}

View File

@@ -0,0 +1,70 @@
package app
import (
"errors"
"fmt"
"strings"
)
type runFailure struct {
pipelineID string
destinationID string
backend string
bundlePath string
err error
}
type runFailures struct {
items []runFailure
}
func (f *runFailures) add(pipelineID, destinationID, backend, bundlePath string, err error) {
f.items = append(f.items, runFailure{
pipelineID: pipelineID,
destinationID: destinationID,
backend: backend,
bundlePath: bundlePath,
err: err,
})
}
func (f runFailures) Error() string {
if len(f.items) == 0 {
return ""
}
parts := make([]string, 0, len(f.items))
for _, item := range f.items {
parts = append(parts, fmt.Sprintf("pipeline %s destination %s backend %s bundle %s: %v", item.pipelineID, item.destinationID, item.backend, item.bundlePath, item.err))
}
return "run failed: " + strings.Join(parts, "; ")
}
func (f runFailures) outputErrors() []OutputError {
if len(f.items) == 0 {
return nil
}
errors := make([]OutputError, 0, len(f.items))
for _, item := range f.items {
errors = append(errors, OutputError{
PipelineID: item.pipelineID,
DestinationID: item.destinationID,
Backend: item.backend,
BundlePath: item.bundlePath,
Message: item.err.Error(),
})
}
return errors
}
func IsPartialResultError(err error) bool {
var failures runFailures
return errors.As(err, &failures)
}
func (f runFailures) Unwrap() error {
errs := make([]error, 0, len(f.items))
for _, item := range f.items {
errs = append(errs, item.err)
}
return errors.Join(errs...)
}

View File

@@ -0,0 +1,33 @@
package app
import (
"gitea.maximumdirect.net/eric/distributor/internal/notify"
"gitea.maximumdirect.net/eric/distributor/internal/publish"
)
func shouldNotify(action publish.Action) bool {
return action == publish.ActionPublishNew || action == publish.ActionReplaceOlder || action == publish.ActionForceReplace
}
func notifyEvent(plan publish.Plan) notify.Event {
outputs := make([]notify.Output, 0, len(plan.Outputs))
for _, output := range plan.Outputs {
stateOutput := output.StateOutputFile()
outputs = append(outputs, notify.Output{
Path: stateOutput.Path,
Kind: stateOutput.Kind,
SourcePath: stateOutput.SourcePath,
Transform: stateOutput.Transform,
SHA256: stateOutput.SHA256,
Size: stateOutput.Size,
})
}
return notify.Event{
PipelineID: plan.PipelineID,
DestinationID: plan.DestinationID,
BundleID: plan.BundleID,
BundlePath: plan.BundlePath,
Action: string(plan.Action),
Outputs: outputs,
}
}

154
internal/app/run_output.go Normal file
View File

@@ -0,0 +1,154 @@
package app
import (
"fmt"
"io"
"strings"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/publish"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
)
func writePlanLine(w io.Writer, backend string, plan publish.Plan, planErr error) {
if w == nil {
return
}
if planErr != nil {
destinationID := plan.DestinationID
if destinationID == "" {
destinationID = "unknown"
}
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s%s action=error reason=%q\n", storage.DisplayPath(plan.BundlePath), destinationID, backend, pathMappingSummary(plan), planErr.Error())
return
}
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s%s action=%s outputs=%s reason=%q\n", storage.DisplayPath(plan.BundlePath), plan.DestinationID, backend, pathMappingSummary(plan), plan.Action, outputSummary(plan.Outputs), plan.Reason)
}
func pathMappingSummary(plan publish.Plan) string {
if plan.PathMapping != config.PathMappingFixed {
return ""
}
return fmt.Sprintf(" path_mapping=fixed target=%s", storage.DisplayPath(plan.DestinationBundlePath))
}
func writeErrorLine(w io.Writer, bundlePath, destinationID, backend string, err error) {
if w == nil {
return
}
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s action=error reason=%q\n", storage.DisplayPath(bundlePath), destinationID, backend, err.Error())
}
func outputSummary(outputs []publish.Output) string {
if len(outputs) == 0 {
return "none"
}
paths := make([]string, 0, len(outputs))
for _, output := range outputs {
paths = append(paths, output.DestinationPath)
}
return strings.Join(paths, ",")
}
type runResult struct {
DryRun bool `json:"dry_run"`
Pipelines []runPipelineResult `json:"pipelines"`
Actions []runActionResult `json:"actions"`
Summary runSummaryResult `json:"summary"`
}
type runPipelineResult struct {
ID string `json:"id"`
SourceBackend string `json:"source_backend"`
BundleCount int `json:"bundle_count"`
Destinations []string `json:"destinations"`
}
type runActionResult struct {
PipelineID string `json:"pipeline_id,omitempty"`
DestinationID string `json:"destination_id"`
Backend string `json:"backend"`
BundleID string `json:"bundle_id,omitempty"`
BundlePath string `json:"bundle_path"`
DestinationPath string `json:"destination_path"`
PathMapping string `json:"path_mapping,omitempty"`
Action string `json:"action"`
PrimaryURL string `json:"primary_url,omitempty"`
Reason string `json:"reason,omitempty"`
Outputs []runOutputResult `json:"outputs"`
}
type runOutputResult struct {
Path string `json:"path"`
Kind string `json:"kind"`
SourcePath string `json:"source_path,omitempty"`
Transform string `json:"transform,omitempty"`
URL string `json:"url,omitempty"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
}
func runActionFromPlan(backend string, plan publish.Plan, planErr error) runActionResult {
if planErr != nil {
destinationID := plan.DestinationID
if destinationID == "" {
destinationID = "unknown"
}
return runActionResult{
PipelineID: plan.PipelineID,
DestinationID: destinationID,
Backend: backend,
BundleID: plan.BundleID,
BundlePath: storage.DisplayPath(plan.BundlePath),
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
PathMapping: plan.PathMapping,
Action: "error",
PrimaryURL: plan.PrimaryURL,
Reason: planErr.Error(),
Outputs: []runOutputResult{},
}
}
return runActionResult{
PipelineID: plan.PipelineID,
DestinationID: plan.DestinationID,
Backend: backend,
BundleID: plan.BundleID,
BundlePath: storage.DisplayPath(plan.BundlePath),
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
PathMapping: plan.PathMapping,
Action: string(plan.Action),
PrimaryURL: plan.PrimaryURL,
Reason: plan.Reason,
Outputs: runOutputsFromPlan(plan.Outputs),
}
}
func errorAction(pipelineID, destinationID, backend, bundlePath string, err error) runActionResult {
return runActionResult{
PipelineID: pipelineID,
DestinationID: destinationID,
Backend: backend,
BundlePath: storage.DisplayPath(bundlePath),
DestinationPath: storage.DisplayPath(bundlePath),
Action: "error",
Reason: err.Error(),
Outputs: []runOutputResult{},
}
}
func runOutputsFromPlan(outputs []publish.Output) []runOutputResult {
results := make([]runOutputResult, 0, len(outputs))
for _, output := range outputs {
stateOutput := output.StateOutputFile()
results = append(results, runOutputResult{
Path: stateOutput.Path,
Kind: stateOutput.Kind,
SourcePath: stateOutput.SourcePath,
Transform: stateOutput.Transform,
URL: stateOutput.URL,
SHA256: stateOutput.SHA256,
Size: stateOutput.Size,
})
}
return results
}

View File

@@ -0,0 +1,91 @@
package app
import (
"fmt"
"sort"
"strings"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/publish"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
)
type destinationBundleSelection struct {
SourceBundle bundle.Bundle
DestinationBundlePath string
}
func selectDestinationBundles(destination config.Destination, bundles []bundle.Bundle) []destinationBundleSelection {
if !isFixedPathDestination(destination) {
selections := make([]destinationBundleSelection, 0, len(bundles))
for _, sourceBundle := range bundles {
selections = append(selections, destinationBundleSelection{
SourceBundle: sourceBundle,
DestinationBundlePath: sourceBundle.RootRelativePath,
})
}
return selections
}
if len(bundles) == 0 {
return nil
}
sourceBundle := newestBundle(bundles)
return []destinationBundleSelection{{
SourceBundle: sourceBundle,
DestinationBundlePath: "",
}}
}
func newestBundle(bundles []bundle.Bundle) bundle.Bundle {
if len(bundles) == 0 {
return bundle.Bundle{}
}
sorted := append([]bundle.Bundle(nil), bundles...)
sort.Slice(sorted, func(i, j int) bool {
if sorted[i].Manifest.Created.Equal(sorted[j].Manifest.Created) {
return sorted[i].RootRelativePath < sorted[j].RootRelativePath
}
return sorted[i].Manifest.Created.After(sorted[j].Manifest.Created)
})
return sorted[0]
}
func isFixedPathDestination(destination config.Destination) bool {
return destination.PathMap.Mode == config.PathMappingFixed
}
func fixedPathSelectionWarning(pipelineID, destinationID string, selections []destinationBundleSelection, candidateCount int) OutputWarning {
selected := "none"
if len(selections) > 0 {
selected = storage.DisplayPath(selections[0].SourceBundle.RootRelativePath)
}
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed candidates=%d selected_bundle=%s destination_bundle=.", pipelineID, destinationID, candidateCount, selected)}
}
func isDestructiveFixedPathAction(action publish.Action) bool {
return action == publish.ActionReplaceOlder || action == publish.ActionForceReplace
}
func fixedPathReplacementWarning(plan publish.Plan) OutputWarning {
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed action=%s replaces destination root for selected_bundle=%s", plan.PipelineID, plan.DestinationID, plan.Action, storage.DisplayPath(plan.BundlePath))}
}
func destinationIDs(destinations []config.Destination) []string {
ids := make([]string, 0, len(destinations))
for _, destination := range destinations {
ids = append(ids, destination.ID)
}
return ids
}
func destinationSummary(destinations []config.Destination) string {
if len(destinations) == 0 {
return "none"
}
ids := make([]string, 0, len(destinations))
for _, destination := range destinations {
ids = append(ids, destination.ID)
}
return strings.Join(ids, ",")
}

View File

@@ -0,0 +1,78 @@
package app
import (
"fmt"
"gitea.maximumdirect.net/eric/distributor/internal/publish"
)
type runSummary struct {
dryRun bool
planned int
publishNew int
replaceOlder int
forceReplace int
skipped int
failures int
fixedPath int
}
func (s *runSummary) recordPlan(action publish.Action) {
s.planned++
switch action {
case publish.ActionPublishNew:
s.publishNew++
case publish.ActionReplaceOlder:
s.replaceOlder++
case publish.ActionForceReplace:
s.forceReplace++
case publish.ActionSkipSame, publish.ActionSkipDestinationNewer:
s.skipped++
}
}
func (s *runSummary) recordFailure() {
s.failures++
}
func (s *runSummary) recordFixedPath() {
s.fixedPath++
}
func (s runSummary) Line() string {
status := "ok"
if s.failures > 0 {
status = "failed"
}
return fmt.Sprintf("Final status: %s planned=%d publish_new=%d replace_older=%d force_replace=%d skipped=%d failed=%d dry_run=%t fixed_path=%d", status, s.planned, s.publishNew, s.replaceOlder, s.forceReplace, s.skipped, s.failures, s.dryRun, s.fixedPath)
}
type runSummaryResult struct {
Status string `json:"status"`
Planned int `json:"planned"`
PublishNew int `json:"publish_new"`
ReplaceOlder int `json:"replace_older"`
ForceReplace int `json:"force_replace"`
Skipped int `json:"skipped"`
Failed int `json:"failed"`
DryRun bool `json:"dry_run"`
FixedPath int `json:"fixed_path"`
}
func (s runSummary) Result() runSummaryResult {
status := "ok"
if s.failures > 0 {
status = "failed"
}
return runSummaryResult{
Status: status,
Planned: s.planned,
PublishNew: s.publishNew,
ReplaceOlder: s.replaceOlder,
ForceReplace: s.forceReplace,
Skipped: s.skipped,
Failed: s.failures,
DryRun: s.dryRun,
FixedPath: s.fixedPath,
}
}

View File

@@ -159,9 +159,9 @@ pipelines:
}
}
func TestWriteSSHWarningsReportsInsecureHostKeyPolicy(t *testing.T) {
func TestSSHWarningsReportInsecureHostKeyPolicy(t *testing.T) {
var stdout bytes.Buffer
err := writeSSHWarnings(&stdout, config.Pipeline{
err := writeWarnings(&stdout, sshWarnings(config.Pipeline{
ID: "reports",
Source: config.Backend{
Backend: config.BackendSSH,
@@ -172,9 +172,9 @@ func TestWriteSSHWarningsReportsInsecureHostKeyPolicy(t *testing.T) {
Backend: config.BackendSSH,
SSH: config.SSH{HostKeyPolicy: config.HostKeyPolicyOff},
}},
})
}))
if err != nil {
t.Fatalf("writeSSHWarnings() error = %v", err)
t.Fatalf("writeWarnings() error = %v", err)
}
output := stdout.String()
for _, want := range []string{
@@ -200,8 +200,8 @@ func TestRunPublishesNewLocalBundle(t *testing.T) {
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
assertFile(t, filepath.Join(destinationRoot, "summary.txt"), "Summary\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "summary.txt"), "Summary\n")
if _, err := os.Stat(filepath.Join(destinationRoot, "manifest.json")); !os.IsNotExist(err) {
t.Fatalf("destination manifest stat error = %v, want not exist", err)
}
@@ -225,11 +225,11 @@ func TestRunExplicitPreserveRelativePathMappingMatchesDefault(t *testing.T) {
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "daily/report", testBundleOptions{})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingPreserveRelative)})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingPreserveRelative)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "daily", "report", "report.md"), "# Report\nSunny.\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "daily", "report", "report.md"), "# Report\nSunny.\n")
if _, err := os.Stat(filepath.Join(destinationRoot, "report.md")); !os.IsNotExist(err) {
t.Fatalf("root report.md stat error = %v, want not exist", err)
}
@@ -241,7 +241,7 @@ func TestRunRecordsLinksForNestedBundlePath(t *testing.T) {
writeSourceBundle(t, sourceRoot, "daily/brentwood", testBundleOptions{})
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithLinks(t, sourceRoot, destinationRoot, config.PathMappingPreserveRelative, "https://reports.example.com/archive", config.LinkPrimaryAuto, true, false, ""),
ConfigPath: testutil.WriteLocalConfigWithLinks(t, sourceRoot, destinationRoot, config.PathMappingPreserveRelative, "https://reports.example.com/archive", config.LinkPrimaryAuto, true, false, ""),
})
if err != nil {
t.Fatalf("Run() error = %v", err)
@@ -266,7 +266,7 @@ func TestRunRecordsLinksForFixedIndexDestination(t *testing.T) {
writeSourceBundle(t, sourceRoot, "newer", testBundleOptions{ID: "reports.newer", Created: testutil.DefaultCreated.Add(time.Hour)})
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithLinks(t, sourceRoot, destinationRoot, config.PathMappingFixed, "https://reports.example.com/latest", config.LinkPrimaryAuto, false, true, config.TransformModeIndex),
ConfigPath: testutil.WriteLocalConfigWithLinks(t, sourceRoot, destinationRoot, config.PathMappingFixed, "https://reports.example.com/latest", config.LinkPrimaryAuto, false, true, config.TransformModeIndex),
})
if err != nil {
t.Fatalf("Run() error = %v", err)
@@ -303,11 +303,11 @@ func TestRunFixedPathPublishesNewestBundleAtDestinationRoot(t *testing.T) {
},
})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nNew.\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nNew.\n")
if _, err := os.Stat(filepath.Join(destinationRoot, "new", "report.md")); !os.IsNotExist(err) {
t.Fatalf("nested new report stat error = %v, want not exist", err)
}
@@ -337,7 +337,7 @@ func TestRunFixedPathTieBreaksByBundlePath(t *testing.T) {
},
})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
@@ -355,7 +355,7 @@ func TestRunFixedPathDryRunReportsSelection(t *testing.T) {
var stdout bytes.Buffer
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
ConfigPath: testutil.WriteLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
DryRun: true,
Stdout: &stdout,
})
@@ -391,7 +391,7 @@ func TestRunFixedPathDryRunWarnsForReplacement(t *testing.T) {
{Path: "summary.txt", Data: "Old summary\n"},
},
})
configPath := writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)
configPath := testutil.WriteLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
t.Fatalf("first Run() error = %v", err)
}
@@ -422,7 +422,7 @@ func TestRunFixedPathDryRunWarnsForReplacement(t *testing.T) {
t.Fatalf("stdout = %q, want substring %q", output, want)
}
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nOld.\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nOld.\n")
}
func TestRunFixedPathReplacesOlderManagedState(t *testing.T) {
@@ -436,11 +436,11 @@ func TestRunFixedPathReplacesOlderManagedState(t *testing.T) {
{Path: "summary.txt", Data: "Old summary\n"},
},
})
configPath := writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)
configPath := testutil.WriteLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
t.Fatalf("first Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nOld.\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nOld.\n")
writeSourceBundle(t, sourceRoot, "new", testBundleOptions{
ID: "reports.new",
@@ -454,7 +454,7 @@ func TestRunFixedPathReplacesOlderManagedState(t *testing.T) {
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
t.Fatalf("second Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nNew.\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nNew.\n")
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
if destinationState.Source.Manifest.ID != "reports.new" {
t.Fatalf("state source id = %q, want reports.new", destinationState.Source.Manifest.ID)
@@ -479,7 +479,7 @@ func TestRunFixedPathSkipsWhenDestinationStateIsNewer(t *testing.T) {
var stdout bytes.Buffer
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
ConfigPath: testutil.WriteLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
Stdout: &stdout,
})
if err != nil {
@@ -488,7 +488,7 @@ func TestRunFixedPathSkipsWhenDestinationStateIsNewer(t *testing.T) {
if !strings.Contains(stdout.String(), "action=skip_destination_newer") {
t.Fatalf("stdout = %q, want skip_destination_newer", stdout.String())
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nExisting.\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nExisting.\n")
}
func TestRunFixedPathFailsUnmanagedWithoutForce(t *testing.T) {
@@ -499,7 +499,7 @@ func TestRunFixedPathFailsUnmanagedWithoutForce(t *testing.T) {
t.Fatalf("write unmanaged file: %v", err)
}
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)})
if err == nil || !strings.Contains(err.Error(), "fail_unmanaged") {
t.Fatalf("Run() error = %v, want unmanaged failure", err)
}
@@ -521,14 +521,14 @@ func TestRunFixedPathForceReplacementStaysWithinDestinationRoot(t *testing.T) {
writeSourceBundle(t, sourceRoot, "bundle", testBundleOptions{})
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
ConfigPath: testutil.WriteLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
Force: true,
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
assertFile(t, filepath.Join(parent, "keep.txt"), "keep")
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
testutil.AssertFile(t, filepath.Join(parent, "keep.txt"), "keep")
if _, err := os.Stat(filepath.Join(destinationRoot, "unmanaged.txt")); !os.IsNotExist(err) {
t.Fatalf("unmanaged stat error = %v, want removed", err)
}
@@ -583,12 +583,12 @@ func TestRunFixedPathRemoteBackendsUseBackendRoots(t *testing.T) {
if err := runConfigWithBackendFactory(context.Background(), cfg, RunOptions{}, provider); err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFakeFile(t, s3Destination, "report.md", "# Report\nNew.\n")
assertFakeFile(t, s3Destination, "summary.txt", "New summary\n")
assertFakeMissing(t, s3Destination, "new/report.md")
assertFakeFile(t, sshDestination, "report.md", "# Report\nNew.\n")
assertFakeFile(t, sshDestination, "summary.txt", "New summary\n")
assertFakeMissing(t, sshDestination, "new/report.md")
testutil.AssertFakeFile(t, s3Destination, "report.md", "# Report\nNew.\n")
testutil.AssertFakeFile(t, s3Destination, "summary.txt", "New summary\n")
testutil.AssertFakeMissing(t, s3Destination, "new/report.md")
testutil.AssertFakeFile(t, sshDestination, "report.md", "# Report\nNew.\n")
testutil.AssertFakeFile(t, sshDestination, "summary.txt", "New summary\n")
testutil.AssertFakeMissing(t, sshDestination, "new/report.md")
}
func TestRunNotifiesAfterPublication(t *testing.T) {
@@ -622,6 +622,32 @@ func TestRunNotifiesAfterPublication(t *testing.T) {
}
}
func TestRunNotifiesGeneratedOutputMetadata(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
notifier := &recordingNotifier{}
err := Run(context.Background(), RunOptions{
ConfigPath: testutil.WriteLocalConfigWithPublishPolicy(t, sourceRoot, destinationRoot, false, true),
Notifier: notifier,
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if got, want := len(notifier.events), 1; got != want {
t.Fatalf("notification count = %d, want %d", got, want)
}
outputs := notifier.events[0].Outputs
if got, want := len(outputs), 1; got != want {
t.Fatalf("notification output count = %d, want %d", got, want)
}
output := outputs[0]
if output.Path != "report.html" || output.Kind != state.OutputKindGenerated || output.SourcePath != "report.md" || output.Transform != "markdown_to_html" || output.SHA256 == "" || output.Size <= 0 {
t.Fatalf("notification output = %#v", output)
}
}
func TestRunNotifiesAfterReplacement(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
@@ -649,6 +675,49 @@ func TestRunNotifiesAfterReplacement(t *testing.T) {
}
}
func TestRunJSONIncludesGeneratedOutputMetadata(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
var stdout bytes.Buffer
err := Run(context.Background(), RunOptions{
ConfigPath: testutil.WriteLocalConfigWithLinks(t, sourceRoot, destinationRoot, config.PathMappingFixed, "https://reports.example.com/latest", config.LinkPrimaryAuto, false, true, config.TransformModeIndex),
DryRun: true,
Stdout: &stdout,
OutputFormat: OutputFormatJSON,
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
result := decodeAppResult(t, stdout.String())
actions, ok := result["actions"].([]any)
if !ok || len(actions) != 1 {
t.Fatalf("actions = %#v, want one action", result["actions"])
}
action, ok := actions[0].(map[string]any)
if !ok {
t.Fatalf("action = %#v, want object", actions[0])
}
if action["primary_url"] != "https://reports.example.com/latest/" {
t.Fatalf("action primary_url = %#v", action["primary_url"])
}
outputs, ok := action["outputs"].([]any)
if !ok || len(outputs) != 1 {
t.Fatalf("outputs = %#v, want one output", action["outputs"])
}
output, ok := outputs[0].(map[string]any)
if !ok {
t.Fatalf("output = %#v, want object", outputs[0])
}
if output["path"] != "index.html" || output["kind"] != state.OutputKindGenerated || output["source_path"] != "report.md" || output["transform"] != "markdown_to_html" || output["url"] != "https://reports.example.com/latest/" {
t.Fatalf("output = %#v, want generated index metadata", output)
}
if output["sha256"] == "" || output["size"] == nil {
t.Fatalf("output = %#v, want digest and size", output)
}
}
func TestRunDoesNotNotifyForSkippedDestination(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
@@ -720,7 +789,7 @@ func TestRunContinuesAfterDestinationFailure(t *testing.T) {
t.Fatalf("stdout = %q, want substring %q", output, want)
}
}
assertFile(t, filepath.Join(secondDestination, "report.md"), "# Report\nSunny.\n")
testutil.AssertFile(t, filepath.Join(secondDestination, "report.md"), "# Report\nSunny.\n")
}
func TestRunPublishesHTMLOnly(t *testing.T) {
@@ -728,11 +797,11 @@ func TestRunPublishesHTMLOnly(t *testing.T) {
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, false, true)})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithPublishPolicy(t, sourceRoot, destinationRoot, false, true)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFileContains(t, filepath.Join(destinationRoot, "report.html"), "<h1>Report</h1>")
testutil.AssertFileContains(t, filepath.Join(destinationRoot, "report.html"), "<h1>Report</h1>")
if _, err := os.Stat(filepath.Join(destinationRoot, "report.md")); !os.IsNotExist(err) {
t.Fatalf("report.md stat error = %v, want not exist", err)
}
@@ -753,11 +822,11 @@ func TestRunPublishesHTMLIndexWithExplicitInput(t *testing.T) {
ExtraFiles: []testFile{{Path: "notes.md", Data: "# Notes\nHidden.\n"}},
})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, "report.md")})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, "report.md")})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFileContains(t, filepath.Join(destinationRoot, "index.html"), "<h1>Report</h1>")
testutil.AssertFileContains(t, filepath.Join(destinationRoot, "index.html"), "<h1>Report</h1>")
if _, err := os.Stat(filepath.Join(destinationRoot, "report.html")); !os.IsNotExist(err) {
t.Fatalf("report.html stat error = %v, want not exist", err)
}
@@ -776,11 +845,11 @@ func TestRunPublishesHTMLIndexWithSingleMarkdownFallback(t *testing.T) {
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, "")})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, "")})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFileContains(t, filepath.Join(destinationRoot, "index.html"), "<h1>Report</h1>")
testutil.AssertFileContains(t, filepath.Join(destinationRoot, "index.html"), "<h1>Report</h1>")
}
func TestRunFailsIndexModeWithAmbiguousMarkdownInput(t *testing.T) {
@@ -790,7 +859,7 @@ func TestRunFailsIndexModeWithAmbiguousMarkdownInput(t *testing.T) {
ExtraFiles: []testFile{{Path: "notes.md", Data: "# Notes\n"}},
})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, "")})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, "")})
if err == nil || !strings.Contains(err.Error(), "multiple markdown source files") {
t.Fatalf("Run() error = %v, want ambiguous input error", err)
}
@@ -804,13 +873,13 @@ func TestRunPublishesSourceAndHTML(t *testing.T) {
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, true, true)})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithPublishPolicy(t, sourceRoot, destinationRoot, true, true)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
assertFileContains(t, filepath.Join(destinationRoot, "report.html"), "<p>Sunny.</p>")
assertFile(t, filepath.Join(destinationRoot, "summary.txt"), "Summary\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
testutil.AssertFileContains(t, filepath.Join(destinationRoot, "report.html"), "<p>Sunny.</p>")
testutil.AssertFile(t, filepath.Join(destinationRoot, "summary.txt"), "Summary\n")
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
if got, want := len(destinationState.Outputs), 3; got != want {
t.Fatalf("state output count = %d, want %d", got, want)
@@ -827,7 +896,7 @@ func TestRunDoesNotMutateSourceBundle(t *testing.T) {
t.Fatalf("read source before: %v", err)
}
err = Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, true, true)})
err = Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithPublishPolicy(t, sourceRoot, destinationRoot, true, true)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
@@ -845,7 +914,7 @@ func TestRunFailsOnOutputPathCollision(t *testing.T) {
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{ExtraFiles: []testFile{{Path: "report.html", Data: "<p>source html</p>\n"}}})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, true, true)})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithPublishPolicy(t, sourceRoot, destinationRoot, true, true)})
if err == nil || !strings.Contains(err.Error(), "destination output path collision") {
t.Fatalf("Run() error = %v, want collision", err)
}
@@ -861,7 +930,7 @@ func TestRunFailsOnIndexOutputPathCollision(t *testing.T) {
ExtraFiles: []testFile{{Path: "index.html", Data: "<p>source index</p>\n"}},
})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, true, true, config.TransformModeIndex, "report.md")})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, true, true, config.TransformModeIndex, "report.md")})
if err == nil || !strings.Contains(err.Error(), "destination output path collision") {
t.Fatalf("Run() error = %v, want collision", err)
}
@@ -877,7 +946,7 @@ func TestRunDryRunReportsGeneratedOutputs(t *testing.T) {
var stdout bytes.Buffer
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, false, true),
ConfigPath: testutil.WriteLocalConfigWithPublishPolicy(t, sourceRoot, destinationRoot, false, true),
DryRun: true,
Stdout: &stdout,
})
@@ -896,7 +965,7 @@ func TestRunDryRunReportsIndexOutputWithoutWriting(t *testing.T) {
var stdout bytes.Buffer
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, ""),
ConfigPath: testutil.WriteLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, ""),
DryRun: true,
Stdout: &stdout,
})
@@ -916,11 +985,11 @@ func TestRunSourceOnlyDoesNotWriteIndexOutput(t *testing.T) {
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, true, false, config.TransformModeIndex, "")})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, true, false, config.TransformModeIndex, "")})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
if _, err := os.Stat(filepath.Join(destinationRoot, "index.html")); !os.IsNotExist(err) {
t.Fatalf("index.html stat error = %v, want not exist", err)
}
@@ -936,11 +1005,11 @@ func TestRunReplacesHTMLIndexOutput(t *testing.T) {
{Path: "summary.txt", Data: "Summary\n"},
},
})
configPath := writeLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, "")
configPath := testutil.WriteLocalConfigWithMarkdownTransform(t, sourceRoot, destinationRoot, false, true, config.TransformModeIndex, "")
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
t.Fatalf("first Run() error = %v", err)
}
assertFileContains(t, filepath.Join(destinationRoot, "index.html"), "<p>Old.</p>")
testutil.AssertFileContains(t, filepath.Join(destinationRoot, "index.html"), "<p>Old.</p>")
writeSourceBundle(t, sourceRoot, "", testBundleOptions{
Created: testutil.DefaultCreated.Add(time.Hour),
@@ -953,7 +1022,7 @@ func TestRunReplacesHTMLIndexOutput(t *testing.T) {
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
t.Fatalf("second Run() error = %v", err)
}
assertFileContains(t, filepath.Join(destinationRoot, "index.html"), "<p>New.</p>")
testutil.AssertFileContains(t, filepath.Join(destinationRoot, "index.html"), "<p>New.</p>")
}
func TestRunSkipsWhenDestinationStateMatches(t *testing.T) {
@@ -994,7 +1063,7 @@ func TestRunReplacesOlderDestination(t *testing.T) {
if !strings.Contains(stdout.String(), "action=replace_older") {
t.Fatalf("stdout = %q, want replace_older", stdout.String())
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
}
func TestRunSkipsNewerDestination(t *testing.T) {
@@ -1016,7 +1085,7 @@ func TestRunSkipsNewerDestination(t *testing.T) {
if !strings.Contains(stdout.String(), "action=skip_destination_newer") {
t.Fatalf("stdout = %q, want skip_destination_newer", stdout.String())
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "newer\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "newer\n")
}
func TestRunFailsOnConflict(t *testing.T) {
@@ -1069,7 +1138,7 @@ func TestRunForceReplacesUnmanagedDestination(t *testing.T) {
if _, err := os.Stat(filepath.Join(destinationRoot, "unmanaged.txt")); !os.IsNotExist(err) {
t.Fatalf("unmanaged file stat error = %v, want not exist", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
testutil.AssertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
}
func TestRunFansOutToLocalDestinations(t *testing.T) {
@@ -1082,8 +1151,8 @@ func TestRunFansOutToLocalDestinations(t *testing.T) {
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(firstDestination, "daily", "report.md"), "# Report\nSunny.\n")
assertFile(t, filepath.Join(secondDestination, "daily", "summary.txt"), "Summary\n")
testutil.AssertFile(t, filepath.Join(firstDestination, "daily", "report.md"), "# Report\nSunny.\n")
testutil.AssertFile(t, filepath.Join(secondDestination, "daily", "summary.txt"), "Summary\n")
}
func TestRunFansOutWithDifferentPublishPolicies(t *testing.T) {
@@ -1092,16 +1161,16 @@ func TestRunFansOutWithDifferentPublishPolicies(t *testing.T) {
htmlDestination := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
err := Run(context.Background(), RunOptions{ConfigPath: writeMixedPolicyFanoutConfig(t, sourceRoot, archiveDestination, htmlDestination)})
err := Run(context.Background(), RunOptions{ConfigPath: testutil.WriteMixedPolicyFanoutLocalConfig(t, sourceRoot, archiveDestination, htmlDestination)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(archiveDestination, "report.md"), "# Report\nSunny.\n")
assertFile(t, filepath.Join(archiveDestination, "summary.txt"), "Summary\n")
testutil.AssertFile(t, filepath.Join(archiveDestination, "report.md"), "# Report\nSunny.\n")
testutil.AssertFile(t, filepath.Join(archiveDestination, "summary.txt"), "Summary\n")
if _, err := os.Stat(filepath.Join(archiveDestination, "report.html")); !os.IsNotExist(err) {
t.Fatalf("archive report.html stat error = %v, want not exist", err)
}
assertFileContains(t, filepath.Join(htmlDestination, "report.html"), "<h1>Report</h1>")
testutil.AssertFileContains(t, filepath.Join(htmlDestination, "report.html"), "<h1>Report</h1>")
if _, err := os.Stat(filepath.Join(htmlDestination, "report.md")); !os.IsNotExist(err) {
t.Fatalf("html report.md stat error = %v, want not exist", err)
}
@@ -1159,10 +1228,10 @@ func TestRunExercisesRemoteBackendShapesThroughCommonPath(t *testing.T) {
if err := runConfigWithBackendFactory(context.Background(), cfg, RunOptions{Stdout: &publishOutput}, provider); err != nil {
t.Fatalf("publish error = %v", err)
}
assertFakeFile(t, s3Destination, "report.md", "# Report\nSunny.\n")
assertFakeFile(t, sshDestination, "summary.txt", "Summary\n")
assertFile(t, filepath.Join(s3ToLocalDestination, "report.md"), "# Report\nSunny.\n")
assertFile(t, filepath.Join(sshToLocalDestination, "summary.txt"), "Summary\n")
testutil.AssertFakeFile(t, s3Destination, "report.md", "# Report\nSunny.\n")
testutil.AssertFakeFile(t, sshDestination, "summary.txt", "Summary\n")
testutil.AssertFile(t, filepath.Join(s3ToLocalDestination, "report.md"), "# Report\nSunny.\n")
testutil.AssertFile(t, filepath.Join(sshToLocalDestination, "summary.txt"), "Summary\n")
var repeatOutput bytes.Buffer
if err := runConfigWithBackendFactory(context.Background(), cfg, RunOptions{Stdout: &repeatOutput}, provider); err != nil {
@@ -1178,10 +1247,10 @@ func TestRunForceReplacementStaysWithinRemoteBundlePaths(t *testing.T) {
writeSourceBundle(t, localSourceRoot, "bundle", testBundleOptions{})
s3Destination := fake.New()
sshDestination := fake.New()
mustWriteFake(t, s3Destination, "bundle/old.txt", "old")
mustWriteFake(t, s3Destination, "bundle-sibling/keep.txt", "keep")
mustWriteFake(t, sshDestination, "bundle/old.txt", "old")
mustWriteFake(t, sshDestination, "bundle-sibling/keep.txt", "keep")
testutil.WriteFakeFile(t, s3Destination, "bundle/old.txt", "old")
testutil.WriteFakeFile(t, s3Destination, "bundle-sibling/keep.txt", "keep")
testutil.WriteFakeFile(t, sshDestination, "bundle/old.txt", "old")
testutil.WriteFakeFile(t, sshDestination, "bundle-sibling/keep.txt", "keep")
cfg := config.Config{Pipelines: []config.Pipeline{{
ID: "reports",
Source: config.Backend{Backend: config.BackendLocal, Path: localSourceRoot},
@@ -1209,12 +1278,12 @@ func TestRunForceReplacementStaysWithinRemoteBundlePaths(t *testing.T) {
if err := runConfigWithBackendFactory(context.Background(), cfg, RunOptions{Force: true}, provider); err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFakeFile(t, s3Destination, "bundle/report.md", "# Report\nSunny.\n")
assertFakeMissing(t, s3Destination, "bundle/old.txt")
assertFakeFile(t, s3Destination, "bundle-sibling/keep.txt", "keep")
assertFakeFile(t, sshDestination, "bundle/report.md", "# Report\nSunny.\n")
assertFakeMissing(t, sshDestination, "bundle/old.txt")
assertFakeFile(t, sshDestination, "bundle-sibling/keep.txt", "keep")
testutil.AssertFakeFile(t, s3Destination, "bundle/report.md", "# Report\nSunny.\n")
testutil.AssertFakeMissing(t, s3Destination, "bundle/old.txt")
testutil.AssertFakeFile(t, s3Destination, "bundle-sibling/keep.txt", "keep")
testutil.AssertFakeFile(t, sshDestination, "bundle/report.md", "# Report\nSunny.\n")
testutil.AssertFakeMissing(t, sshDestination, "bundle/old.txt")
testutil.AssertFakeFile(t, sshDestination, "bundle-sibling/keep.txt", "keep")
}
func TestRunDryRunDoesNotWrite(t *testing.T) {
@@ -1272,141 +1341,11 @@ func writeLocalConfig(t *testing.T, sourceRoot, destinationRoot string) string {
return testutil.WriteMinimalLocalConfig(t, sourceRoot, destinationRoot)
}
func writeLocalConfigWithPolicy(t *testing.T, sourceRoot, destinationRoot string, publishSource, publishHTML bool) string {
t.Helper()
transformConfig := ""
if publishHTML {
transformConfig = `
transform:
markdown_to_html:
enabled: true
mode: sidecar`
}
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
publish:
source: `+fmt.Sprintf("%t", publishSource)+`
html: `+fmt.Sprintf("%t", publishHTML)+transformConfig+`
`)
}
func writeLocalConfigWithPathMapping(t *testing.T, sourceRoot, destinationRoot, mode string) string {
t.Helper()
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
path_mapping:
mode: `+mode+`
`)
}
func writeLocalConfigWithLinks(t *testing.T, sourceRoot, destinationRoot, pathMapping, baseURL, primary string, publishSource, publishHTML bool, transformMode string) string {
t.Helper()
transformConfig := ""
if publishHTML {
transformConfig = `
transform:
markdown_to_html:
enabled: true
mode: ` + transformMode
}
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
path_mapping:
mode: `+pathMapping+`
links:
base_url: `+baseURL+`
primary: `+primary+`
publish:
source: `+fmt.Sprintf("%t", publishSource)+`
html: `+fmt.Sprintf("%t", publishHTML)+transformConfig+`
`)
}
func writeLocalConfigWithMarkdownTransform(t *testing.T, sourceRoot, destinationRoot string, publishSource, publishHTML bool, mode, input string) string {
t.Helper()
enabled := publishHTML
inputConfig := ""
if input != "" {
inputConfig = `
input: ` + input
}
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
publish:
source: `+fmt.Sprintf("%t", publishSource)+`
html: `+fmt.Sprintf("%t", publishHTML)+`
transform:
markdown_to_html:
enabled: `+fmt.Sprintf("%t", enabled)+`
mode: `+mode+inputConfig+`
`)
}
func writeFanoutConfig(t *testing.T, sourceRoot, firstDestination, secondDestination string) string {
t.Helper()
return testutil.WriteFanoutLocalConfig(t, sourceRoot, firstDestination, secondDestination)
}
func writeMixedPolicyFanoutConfig(t *testing.T, sourceRoot, archiveDestination, htmlDestination string) string {
t.Helper()
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+archiveDestination+`
publish:
source: true
html: false
- id: html
backend: local
path: `+htmlDestination+`
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
`)
}
func writeConfigFile(t *testing.T, body string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "config.yml")
@@ -1434,53 +1373,6 @@ func outputsByPath(outputs []state.OutputFile) map[string]state.OutputFile {
return byPath
}
func assertFile(t *testing.T, path, want string) {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read file %s: %v", path, err)
}
if got := string(data); got != want {
t.Fatalf("%s = %q, want %q", path, got, want)
}
}
func assertFileContains(t *testing.T, path, want string) {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read file %s: %v", path, err)
}
if !strings.Contains(string(data), want) {
t.Fatalf("%s = %q, want substring %q", path, data, want)
}
}
func assertFakeFile(t *testing.T, backend *fake.Backend, path, want string) {
t.Helper()
data, err := backend.ReadFile(context.Background(), path)
if err != nil {
t.Fatalf("read fake file %s: %v", path, err)
}
if got := string(data); got != want {
t.Fatalf("%s = %q, want %q", path, got, want)
}
}
func assertFakeMissing(t *testing.T, backend *fake.Backend, path string) {
t.Helper()
if _, err := backend.Stat(context.Background(), path); !storage.IsNotFound(err) {
t.Fatalf("fake file %s stat error = %v, want not found", path, err)
}
}
func mustWriteFake(t *testing.T, backend *fake.Backend, path, data string) {
t.Helper()
if _, err := backend.WriteFile(context.Background(), path, []byte(data), storage.WriteOptions{}); err != nil {
t.Fatalf("write fake file %s: %v", path, err)
}
}
func crossBackendConfig(localSourceRoot, s3ToLocalDestination, sshToLocalDestination string) config.Config {
cfg := config.Config{
Pipelines: []config.Pipeline{

View File

@@ -0,0 +1,47 @@
package app
import (
"fmt"
"io"
"gitea.maximumdirect.net/eric/distributor/internal/config"
)
func secretConflictWarnings(conflicts []config.SecretConflict) []OutputWarning {
warnings := make([]OutputWarning, 0, len(conflicts))
for _, conflict := range conflicts {
warnings = append(warnings, OutputWarning{
Message: fmt.Sprintf("secret %s ignored because the real environment already has that variable", conflict.Name),
})
}
return warnings
}
func sshWarnings(pipeline config.Pipeline) []OutputWarning {
var warnings []OutputWarning
if pipeline.Source.Backend == config.BackendSSH && pipeline.Source.SSH.HostKeyPolicy == config.HostKeyPolicyOff {
warnings = append(warnings, OutputWarning{
Message: fmt.Sprintf("pipeline=%s source host_key_policy=off disables SSH host key checking", pipeline.ID),
})
}
for _, destination := range pipeline.Destinations {
if destination.Backend == config.BackendSSH && destination.SSH.HostKeyPolicy == config.HostKeyPolicyOff {
warnings = append(warnings, OutputWarning{
Message: fmt.Sprintf("pipeline=%s destination=%s host_key_policy=off disables SSH host key checking", pipeline.ID, destination.ID),
})
}
}
return warnings
}
func writeWarnings(w io.Writer, warnings []OutputWarning) error {
if w == nil {
return nil
}
for _, warning := range warnings {
if _, err := fmt.Fprintf(w, "Warning: %s\n", warning.Message); err != nil {
return err
}
}
return nil
}

View File

@@ -1,19 +1,9 @@
package bundle
import (
"fmt"
"regexp"
publicbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
)
var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
import publicbundle "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
func ValidateDigest(value string) error {
if !digestPattern.MatchString(value) {
return fmt.Errorf("must be lowercase sha256:<64 hex>")
}
return nil
return publicbundle.ValidateDigest(value)
}
func FileDigest(data []byte) string {

View File

@@ -2,7 +2,6 @@ package cli
import (
"context"
"flag"
"fmt"
"io"
@@ -14,33 +13,17 @@ func inspectCommand(ctx context.Context, args []string, stdout, stderr io.Writer
printInspectHelp(stdout)
return exitOK
}
flags := flag.NewFlagSet("inspect", flag.ContinueOnError)
flags.SetOutput(stderr)
configPath := flags.String("config", "", "path to config file")
pipelineID := flags.String("pipeline", "", "pipeline id")
bundlePath := flags.String("bundle", "", "source-root-relative bundle path")
formatFlag := addFormatFlag(flags)
if err := flags.Parse(args); err != nil {
return exitUsage
}
format, ok := parseOutputFormat(stderr, "inspect", *formatFlag)
parsed, ok := parseSourceDiagnosticArgs(stderr, "inspect", args)
if !ok {
return exitUsage
}
path, ok := parseOptionalPathArg(stderr, "inspect", flags.Args())
if !ok {
return exitUsage
}
if !validateInspectModeOK(stderr, "inspect", path, *configPath, *pipelineID, *bundlePath) {
return exitUsage
}
if err := app.Inspect(ctx, app.InspectOptions{
Path: path,
ConfigPath: *configPath,
PipelineID: *pipelineID,
BundlePath: *bundlePath,
Path: parsed.Path,
ConfigPath: parsed.ConfigPath,
PipelineID: parsed.PipelineID,
BundlePath: parsed.BundlePath,
Stdout: stdout,
OutputFormat: format,
OutputFormat: parsed.OutputFormat,
}); err != nil {
return fail(stderr, err)
}

View File

@@ -217,12 +217,24 @@ func TestExecuteValidateArgs(t *testing.T) {
wantCode: exitUsage,
wantStderr: "requires --config",
},
{
name: "bundle without config",
args: []string{"validate", "--bundle", "daily"},
wantCode: exitUsage,
wantStderr: "requires --config",
},
{
name: "config without pipeline",
args: []string{"validate", "--config", "config.yml"},
wantCode: exitUsage,
wantStderr: "requires --pipeline",
},
{
name: "invalid format",
args: []string{"validate", "--format", "xml", validPath},
wantCode: exitUsage,
wantStderr: "format must be text or json",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
@@ -343,12 +355,24 @@ func TestExecuteInspectArgs(t *testing.T) {
wantCode: exitUsage,
wantStderr: "requires --config",
},
{
name: "bundle without config",
args: []string{"inspect", "--bundle", "daily"},
wantCode: exitUsage,
wantStderr: "requires --config",
},
{
name: "config without pipeline",
args: []string{"inspect", "--config", "config.yml"},
wantCode: exitUsage,
wantStderr: "requires --pipeline",
},
{
name: "invalid format",
args: []string{"inspect", "--format", "xml", validPath},
wantCode: exitUsage,
wantStderr: "format must be text or json",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {

View File

@@ -1,10 +1,51 @@
package cli
import (
"flag"
"fmt"
"io"
"gitea.maximumdirect.net/eric/distributor/internal/app"
)
type sourceDiagnosticArgs struct {
Path string
ConfigPath string
PipelineID string
BundlePath string
OutputFormat app.OutputFormat
}
func parseSourceDiagnosticArgs(stderr io.Writer, command string, args []string) (sourceDiagnosticArgs, bool) {
flags := flag.NewFlagSet(command, flag.ContinueOnError)
flags.SetOutput(stderr)
configPath := flags.String("config", "", "path to config file")
pipelineID := flags.String("pipeline", "", "pipeline id")
bundlePath := flags.String("bundle", "", "source-root-relative bundle path")
formatFlag := addFormatFlag(flags)
if err := flags.Parse(args); err != nil {
return sourceDiagnosticArgs{}, false
}
format, ok := parseOutputFormat(stderr, command, *formatFlag)
if !ok {
return sourceDiagnosticArgs{}, false
}
path, ok := parseOptionalPathArg(stderr, command, flags.Args())
if !ok {
return sourceDiagnosticArgs{}, false
}
if !validateInspectModeOK(stderr, command, path, *configPath, *pipelineID, *bundlePath) {
return sourceDiagnosticArgs{}, false
}
return sourceDiagnosticArgs{
Path: path,
ConfigPath: *configPath,
PipelineID: *pipelineID,
BundlePath: *bundlePath,
OutputFormat: format,
}, true
}
func validateInspectModeOK(stderr io.Writer, command, path, configPath, pipelineID, bundlePath string) bool {
configMode := configPath != "" || pipelineID != "" || bundlePath != ""
if !configMode {

View File

@@ -2,7 +2,6 @@ package cli
import (
"context"
"flag"
"fmt"
"io"
@@ -14,33 +13,17 @@ func validateCommand(ctx context.Context, args []string, stdout, stderr io.Write
printValidateHelp(stdout)
return exitOK
}
flags := flag.NewFlagSet("validate", flag.ContinueOnError)
flags.SetOutput(stderr)
configPath := flags.String("config", "", "path to config file")
pipelineID := flags.String("pipeline", "", "pipeline id")
bundlePath := flags.String("bundle", "", "source-root-relative bundle path")
formatFlag := addFormatFlag(flags)
if err := flags.Parse(args); err != nil {
return exitUsage
}
format, ok := parseOutputFormat(stderr, "validate", *formatFlag)
parsed, ok := parseSourceDiagnosticArgs(stderr, "validate", args)
if !ok {
return exitUsage
}
path, ok := parseOptionalPathArg(stderr, "validate", flags.Args())
if !ok {
return exitUsage
}
if !validateInspectModeOK(stderr, "validate", path, *configPath, *pipelineID, *bundlePath) {
return exitUsage
}
if err := app.Validate(ctx, app.ValidateOptions{
Path: path,
ConfigPath: *configPath,
PipelineID: *pipelineID,
BundlePath: *bundlePath,
Path: parsed.Path,
ConfigPath: parsed.ConfigPath,
PipelineID: parsed.PipelineID,
BundlePath: parsed.BundlePath,
Stdout: stdout,
OutputFormat: format,
OutputFormat: parsed.OutputFormat,
}); err != nil {
return fail(stderr, err)
}

View File

@@ -2,9 +2,10 @@ package config
import (
"fmt"
"net/url"
"regexp"
"strings"
"gitea.maximumdirect.net/eric/distributor/internal/link"
)
var idPattern = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._-]*$`)
@@ -187,7 +188,7 @@ func validateLinks(errs ValidationErrors, context string, links *Links) Validati
}
if links.BaseURL == "" {
errs = append(errs, context+".base_url is required")
} else if err := validateLinkBaseURL(links.BaseURL); err != nil {
} else if err := link.ValidateHTTPURL(links.BaseURL); err != nil {
errs = append(errs, context+".base_url "+err.Error())
}
switch links.Primary {
@@ -198,26 +199,6 @@ func validateLinks(errs ValidationErrors, context string, links *Links) Validati
return errs
}
func validateLinkBaseURL(value string) error {
parsed, err := url.Parse(value)
if err != nil {
return fmt.Errorf("must be a valid URL")
}
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return fmt.Errorf("must use http or https")
}
if parsed.Host == "" {
return fmt.Errorf("must include a host")
}
if parsed.RawQuery != "" {
return fmt.Errorf("must not include a query string")
}
if parsed.Fragment != "" {
return fmt.Errorf("must not include a fragment")
}
return nil
}
func validateTransferPolicy(errs ValidationErrors, context string, policy TransferPolicy) ValidationErrors {
if policy.OnDestinationSame != TransferActionSkip && policy.OnDestinationSame != TransferActionFail {
errs = append(errs, context+".on_destination_same must be skip or fail")

View File

@@ -1,6 +1,9 @@
package config
import "testing"
import (
"strings"
"testing"
)
func TestValidatePublishTransformPolicy(t *testing.T) {
tests := publishTransformPolicyCases()
@@ -145,6 +148,28 @@ func TestValidateLinks(t *testing.T) {
}
}
func TestValidateLinksReportsFieldContext(t *testing.T) {
cfg := Config{Pipelines: []Pipeline{{
ID: "reports",
Source: Backend{Backend: BackendLocal, Path: "/source"},
Destinations: []Destination{{
ID: "web",
Backend: BackendLocal,
Path: "/destination",
Links: &Links{BaseURL: "https://reports.example.com/archive?preview=1", Primary: LinkPrimaryAuto},
}},
}}}
ApplyDefaults(&cfg)
err := Validate(cfg)
if err == nil {
t.Fatal("Validate() error = nil, want error")
}
want := "pipelines[0].destinations[0].links.base_url must not include a query string"
if !strings.Contains(err.Error(), want) {
t.Fatalf("Validate() error = %q, want %q", err, want)
}
}
type publishTransformPolicyCase struct {
name string
publish PublishPolicy

26
internal/link/url.go Normal file
View File

@@ -0,0 +1,26 @@
package link
import (
"fmt"
"net/url"
)
func ValidateHTTPURL(value string) error {
parsed, err := url.Parse(value)
if err != nil {
return fmt.Errorf("must be a valid URL")
}
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return fmt.Errorf("must use http or https")
}
if parsed.Host == "" {
return fmt.Errorf("must include a host")
}
if parsed.RawQuery != "" {
return fmt.Errorf("must not include a query string")
}
if parsed.Fragment != "" {
return fmt.Errorf("must not include a fragment")
}
return nil
}

39
internal/link/url_test.go Normal file
View File

@@ -0,0 +1,39 @@
package link
import (
"strings"
"testing"
)
func TestValidateHTTPURL(t *testing.T) {
tests := []struct {
name string
value string
wantErr string
}{
{name: "http", value: "http://reports.example.com/archive"},
{name: "https", value: "https://reports.example.com/archive"},
{name: "missing host", value: "https:///archive", wantErr: "must include a host"},
{name: "unsupported scheme", value: "ftp://reports.example.com/archive", wantErr: "must use http or https"},
{name: "query string", value: "https://reports.example.com/archive?preview=1", wantErr: "must not include a query string"},
{name: "fragment", value: "https://reports.example.com/archive#top", wantErr: "must not include a fragment"},
{name: "parse failure", value: "http://[::1", wantErr: "must be a valid URL"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateHTTPURL(tt.value)
if tt.wantErr == "" {
if err != nil {
t.Fatalf("ValidateHTTPURL() error = %v", err)
}
return
}
if err == nil {
t.Fatal("ValidateHTTPURL() error = nil, want error")
}
if !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("ValidateHTTPURL() error = %q, want %q", err, tt.wantErr)
}
})
}
}

View File

@@ -23,7 +23,7 @@ func Execute(ctx context.Context, req Request, plan Plan) error {
if plan.ExistingState == nil {
return fmt.Errorf("replace requires existing destination state")
}
if err := req.DestinationBackend.DeleteManagedBundle(ctx, req.DestinationBundlePath, existingManagedOutputPaths(*plan.ExistingState), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true}); err != nil {
if err := req.DestinationBackend.DeleteManagedBundle(ctx, req.DestinationBundlePath, stateOutputManagedPaths(plan.ExistingState.Outputs), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true}); err != nil {
return err
}
if err := ensureDestinationEmpty(ctx, req.DestinationBackend, req.DestinationBundlePath); err != nil {
@@ -41,7 +41,7 @@ func Execute(ctx context.Context, req Request, plan Plan) error {
writtenOutputs := make([]Output, 0, len(plan.Outputs))
cleanup := func() {
_ = req.DestinationBackend.DeleteManagedBundle(ctx, req.DestinationBundlePath, managedOutputPaths(writtenOutputs), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true})
_ = req.DestinationBackend.DeleteManagedBundle(ctx, req.DestinationBundlePath, ManagedOutputPaths(writtenOutputs), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true})
}
for _, output := range plan.Outputs {
destinationPath, err := storage.Join(req.DestinationBundlePath, output.DestinationPath)
@@ -76,7 +76,7 @@ func Execute(ctx context.Context, req Request, plan Plan) error {
DestinationID: req.DestinationID,
PublishedAt: time.Now().UTC(),
Source: state.SourceState{Manifest: req.SourceBundle.Manifest},
Outputs: stateOutputs(plan.Outputs),
Outputs: StateOutputFiles(plan.Outputs),
}
if plan.PrimaryURL != "" {
destinationState.Links = &state.LinkState{PrimaryURL: plan.PrimaryURL}
@@ -102,11 +102,3 @@ func Execute(ctx context.Context, req Request, plan Plan) error {
}
return nil
}
func existingManagedOutputPaths(destinationState state.DistributorState) []string {
paths := make([]string, 0, len(destinationState.Outputs))
for _, output := range destinationState.Outputs {
paths = append(paths, output.Path)
}
return paths
}

View File

@@ -2,13 +2,11 @@ package publish
import (
"context"
"encoding/json"
"strings"
"testing"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
"gitea.maximumdirect.net/eric/distributor/internal/testutil"
)
@@ -25,7 +23,7 @@ func TestBuildPlansForcedReplacementOnlyWhenExplicit(t *testing.T) {
name: "unmanaged content",
prepare: func(t *testing.T, backend *fake.Backend, source bundle.Manifest) {
t.Helper()
writeFakeFile(t, backend, "bundle/old.txt", "old")
testutil.WriteFakeFile(t, backend, "bundle/old.txt", "old")
},
transfer: defaultTransfer(),
wantReason: "fail_unmanaged",
@@ -37,7 +35,7 @@ func TestBuildPlansForcedReplacementOnlyWhenExplicit(t *testing.T) {
t.Helper()
conflict := source
conflict.ID = "other.source"
writeFakeDestinationState(t, backend, "bundle", conflict, testutil.DestinationStateOptions{})
testutil.WriteFakeDestinationState(t, backend, "bundle", conflict, testutil.DestinationStateOptions{})
},
transfer: conflictReplaceTransfer(),
wantReason: "requires --force",
@@ -48,7 +46,7 @@ func TestBuildPlansForcedReplacementOnlyWhenExplicit(t *testing.T) {
prepare: func(t *testing.T, backend *fake.Backend, source bundle.Manifest) {
t.Helper()
conflict := testutil.ValidManifest(testutil.BundleOptions{Files: []testutil.SourceFile{{Path: "report.md", Data: "# Different\n"}}})
writeFakeDestinationState(t, backend, "bundle", conflict, testutil.DestinationStateOptions{})
testutil.WriteFakeDestinationState(t, backend, "bundle", conflict, testutil.DestinationStateOptions{})
},
transfer: conflictReplaceTransfer(),
wantReason: "requires --force",
@@ -58,7 +56,7 @@ func TestBuildPlansForcedReplacementOnlyWhenExplicit(t *testing.T) {
name: "pipeline mismatch",
prepare: func(t *testing.T, backend *fake.Backend, source bundle.Manifest) {
t.Helper()
writeFakeDestinationState(t, backend, "bundle", source, testutil.DestinationStateOptions{PipelineID: "other-pipeline"})
testutil.WriteFakeDestinationState(t, backend, "bundle", source, testutil.DestinationStateOptions{PipelineID: "other-pipeline"})
},
transfer: conflictReplaceTransfer(),
wantReason: "requires --force",
@@ -68,7 +66,7 @@ func TestBuildPlansForcedReplacementOnlyWhenExplicit(t *testing.T) {
name: "destination mismatch",
prepare: func(t *testing.T, backend *fake.Backend, source bundle.Manifest) {
t.Helper()
writeFakeDestinationState(t, backend, "bundle", source, testutil.DestinationStateOptions{DestinationID: "other-destination"})
testutil.WriteFakeDestinationState(t, backend, "bundle", source, testutil.DestinationStateOptions{DestinationID: "other-destination"})
},
transfer: conflictReplaceTransfer(),
wantReason: "requires --force",
@@ -80,7 +78,7 @@ func TestBuildPlansForcedReplacementOnlyWhenExplicit(t *testing.T) {
t.Helper()
newer := source
newer.Created = newer.Created.AddDate(0, 0, 1)
writeFakeDestinationState(t, backend, "bundle", newer, testutil.DestinationStateOptions{})
testutil.WriteFakeDestinationState(t, backend, "bundle", newer, testutil.DestinationStateOptions{})
},
transfer: newerReplaceTransfer(),
wantReason: "requires --force",
@@ -121,7 +119,7 @@ func TestBuildRequiresConflictPolicyForStateConflicts(t *testing.T) {
destinationBackend := fake.New()
conflict := sourceBundle.Manifest
conflict.ID = "other.source"
writeFakeDestinationState(t, destinationBackend, "bundle", conflict, testutil.DestinationStateOptions{})
testutil.WriteFakeDestinationState(t, destinationBackend, "bundle", conflict, testutil.DestinationStateOptions{})
req := forceRequest(sourceBackend, destinationBackend, sourceBundle, defaultTransfer())
req.Force = true
@@ -135,10 +133,10 @@ func TestExecuteForcedReplacementDeletesOnlyBundlePath(t *testing.T) {
sourceBackend := fake.New()
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "bundle", testutil.BundleOptions{})
destinationBackend := fake.New()
writeFakeFile(t, destinationBackend, "bundle/old.txt", "old")
writeFakeFile(t, destinationBackend, "bundle/nested/old.txt", "old")
writeFakeFile(t, destinationBackend, "bundle-sibling/keep.txt", "keep")
writeFakeFile(t, destinationBackend, "outside.txt", "outside")
testutil.WriteFakeFile(t, destinationBackend, "bundle/old.txt", "old")
testutil.WriteFakeFile(t, destinationBackend, "bundle/nested/old.txt", "old")
testutil.WriteFakeFile(t, destinationBackend, "bundle-sibling/keep.txt", "keep")
testutil.WriteFakeFile(t, destinationBackend, "outside.txt", "outside")
req := forceRequest(sourceBackend, destinationBackend, sourceBundle, defaultTransfer())
req.Force = true
@@ -152,11 +150,11 @@ func TestExecuteForcedReplacementDeletesOnlyBundlePath(t *testing.T) {
if err := Execute(context.Background(), req, plan); err != nil {
t.Fatalf("Execute() error = %v", err)
}
assertFakeFile(t, destinationBackend, "bundle/report.md", "# Report\nSunny.\n")
assertFakeMissing(t, destinationBackend, "bundle/old.txt")
assertFakeMissing(t, destinationBackend, "bundle/nested/old.txt")
assertFakeFile(t, destinationBackend, "bundle-sibling/keep.txt", "keep")
assertFakeFile(t, destinationBackend, "outside.txt", "outside")
testutil.AssertFakeFile(t, destinationBackend, "bundle/report.md", "# Report\nSunny.\n")
testutil.AssertFakeMissing(t, destinationBackend, "bundle/old.txt")
testutil.AssertFakeMissing(t, destinationBackend, "bundle/nested/old.txt")
testutil.AssertFakeFile(t, destinationBackend, "bundle-sibling/keep.txt", "keep")
testutil.AssertFakeFile(t, destinationBackend, "outside.txt", "outside")
}
func forceRequest(sourceBackend, destinationBackend *fake.Backend, sourceBundle bundle.Bundle, transfer config.TransferPolicy) Request {
@@ -193,49 +191,3 @@ func newerReplaceTransfer() config.TransferPolicy {
transfer.OnDestinationNewer = config.TransferActionReplace
return transfer
}
func writeFakeDestinationState(t *testing.T, backend *fake.Backend, relative string, manifest bundle.Manifest, opts testutil.DestinationStateOptions) {
t.Helper()
destinationState := testutil.DestinationState(manifest, opts)
data, err := json.MarshalIndent(destinationState, "", " ")
if err != nil {
t.Fatalf("marshal destination state: %v", err)
}
statePath, err := storage.StatePath(relative)
if err != nil {
t.Fatalf("state path: %v", err)
}
writeFakeFile(t, backend, statePath, string(append(data, '\n')))
for _, output := range destinationState.Outputs {
path, err := storage.Join(relative, output.Path)
if err != nil {
t.Fatalf("join output path: %v", err)
}
writeFakeFile(t, backend, path, "old")
}
}
func writeFakeFile(t *testing.T, backend *fake.Backend, path, data string) {
t.Helper()
if _, err := backend.WriteFile(context.Background(), path, []byte(data), storage.WriteOptions{}); err != nil {
t.Fatalf("write fake file %s: %v", path, err)
}
}
func assertFakeFile(t *testing.T, backend *fake.Backend, path, want string) {
t.Helper()
data, err := backend.ReadFile(context.Background(), path)
if err != nil {
t.Fatalf("read fake file %s: %v", path, err)
}
if got := string(data); got != want {
t.Fatalf("fake file %s = %q, want %q", path, got, want)
}
}
func assertFakeMissing(t *testing.T, backend *fake.Backend, path string) {
t.Helper()
if _, err := backend.Stat(context.Background(), path); !storage.IsNotFound(err) {
t.Fatalf("fake file %s stat error = %v, want not found", path, err)
}
}

View File

@@ -6,6 +6,7 @@ import (
"strings"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/link"
"gitea.maximumdirect.net/eric/distributor/internal/state"
)
@@ -13,6 +14,9 @@ func PlanLinks(req Request, outputs []Output) ([]Output, string, error) {
if req.Links == nil {
return outputs, "", nil
}
if err := link.ValidateHTTPURL(req.Links.BaseURL); err != nil {
return nil, "", fmt.Errorf("link base URL: %w", err)
}
linked := make([]Output, 0, len(outputs))
for _, output := range outputs {
outputURL, err := OutputURL(req.Links.BaseURL, req.DestinationBundlePath, output.DestinationPath)

View File

@@ -1,6 +1,7 @@
package publish
import (
"strings"
"testing"
"gitea.maximumdirect.net/eric/distributor/internal/config"
@@ -147,3 +148,22 @@ func TestPlanLinksLeavesOutputsUnchangedWithoutConfig(t *testing.T) {
t.Fatalf("output URL = %q, want empty", linked[0].URL)
}
}
func TestPlanLinksValidatesBaseURLBeforePlanning(t *testing.T) {
_, _, err := PlanLinks(Request{
DestinationBundlePath: "daily",
Links: &config.Links{
BaseURL: "https://reports.example.com/archive?preview=1",
Primary: config.LinkPrimaryAuto,
},
}, []Output{{
DestinationPath: "report.md",
Kind: state.OutputKindSource,
}})
if err == nil {
t.Fatal("PlanLinks() error = nil, want error")
}
if !strings.Contains(err.Error(), "link base URL: must not include a query string") {
t.Fatalf("PlanLinks() error = %q, want link base URL context", err)
}
}

View File

@@ -97,26 +97,42 @@ func rejectOutputCollisions(outputs []Output) error {
return nil
}
func stateOutputs(outputs []Output) []state.OutputFile {
func (o Output) StateOutputFile() state.OutputFile {
return state.OutputFile{
Path: o.DestinationPath,
Kind: o.Kind,
SourcePath: o.SourcePath,
Transform: o.Transform,
URL: o.URL,
SHA256: o.SHA256,
Size: o.Size,
}
}
func (o Output) ManagedPath() string {
return o.DestinationPath
}
func StateOutputFiles(outputs []Output) []state.OutputFile {
files := make([]state.OutputFile, 0, len(outputs))
for _, output := range outputs {
files = append(files, state.OutputFile{
Path: output.DestinationPath,
Kind: output.Kind,
SourcePath: output.SourcePath,
Transform: output.Transform,
URL: output.URL,
SHA256: output.SHA256,
Size: output.Size,
})
files = append(files, output.StateOutputFile())
}
return files
}
func managedOutputPaths(outputs []Output) []string {
func ManagedOutputPaths(outputs []Output) []string {
paths := make([]string, 0, len(outputs))
for _, output := range outputs {
paths = append(paths, output.DestinationPath)
paths = append(paths, output.ManagedPath())
}
return paths
}
func stateOutputManagedPaths(outputs []state.OutputFile) []string {
paths := make([]string, 0, len(outputs))
for _, output := range outputs {
paths = append(paths, output.Path)
}
return paths
}

View File

@@ -6,11 +6,58 @@ import (
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/state"
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
"gitea.maximumdirect.net/eric/distributor/internal/testutil"
"gitea.maximumdirect.net/eric/distributor/internal/transform"
)
func TestOutputStateProjection(t *testing.T) {
sourceOutput := Output{
SourcePath: "report.md",
DestinationPath: "report.md",
Kind: state.OutputKindSource,
URL: "https://reports.example.com/report.md",
SHA256: "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
Size: 123,
}
sourceState := sourceOutput.StateOutputFile()
if sourceState.Path != "report.md" || sourceState.Kind != state.OutputKindSource || sourceState.SourcePath != "report.md" || sourceState.URL != sourceOutput.URL || sourceState.SHA256 != sourceOutput.SHA256 || sourceState.Size != sourceOutput.Size {
t.Fatalf("source state output = %#v", sourceState)
}
generatedOutput := Output{
SourcePath: "report.md",
DestinationPath: "report.html",
Kind: state.OutputKindGenerated,
Transform: transform.MarkdownToHTML,
URL: "https://reports.example.com/report.html",
SHA256: "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
Size: 456,
}
generatedState := generatedOutput.StateOutputFile()
if generatedState.Path != "report.html" || generatedState.Kind != state.OutputKindGenerated || generatedState.SourcePath != "report.md" || generatedState.Transform != transform.MarkdownToHTML || generatedState.URL != generatedOutput.URL || generatedState.SHA256 != generatedOutput.SHA256 || generatedState.Size != generatedOutput.Size {
t.Fatalf("generated state output = %#v", generatedState)
}
}
func TestOutputSliceProjections(t *testing.T) {
outputs := []Output{
{SourcePath: "report.md", DestinationPath: "report.md", Kind: state.OutputKindSource, SHA256: "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", Size: 1},
{SourcePath: "report.md", DestinationPath: "report.html", Kind: state.OutputKindGenerated, Transform: transform.MarkdownToHTML, URL: "https://reports.example.com/report.html", SHA256: "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", Size: 2},
}
stateOutputs := StateOutputFiles(outputs)
if len(stateOutputs) != 2 || stateOutputs[1].Path != "report.html" || stateOutputs[1].Transform != transform.MarkdownToHTML || stateOutputs[1].URL != outputs[1].URL {
t.Fatalf("state outputs = %#v", stateOutputs)
}
paths := ManagedOutputPaths(outputs)
if len(paths) != 2 || paths[0] != "report.md" || paths[1] != "report.html" {
t.Fatalf("managed paths = %#v", paths)
}
}
func TestPlanOutputsRejectsCollision(t *testing.T) {
sourceBackend := fake.New()
sourceBundle := testutil.WriteFakeSourceBundle(t, sourceBackend, "", testutil.BundleOptions{

View File

@@ -179,6 +179,23 @@ func TestParseRejectsInvalidOutputMetadata(t *testing.T) {
}
}
func TestValidateReportsURLFieldContext(t *testing.T) {
source := validManifest(t)
state := *withState(t, source, func(s *DistributorState) {
s.Links = &LinkState{PrimaryURL: "https://reports.example.com/archive#top"}
})
err := Validate(state)
assertStateErrorContains(t, err, "state links.primary_url")
assertStateErrorContains(t, err, "must not include a fragment")
state = *withState(t, source, func(s *DistributorState) {
s.Outputs[0].URL = "https://reports.example.com/archive?preview=1"
})
err = Validate(state)
assertStateErrorContains(t, err, "state outputs[0].url")
assertStateErrorContains(t, err, "must not include a query string")
}
func TestParseRejectsMalformedPublishedTimestamp(t *testing.T) {
body := strings.Replace(validStateJSON(t), `"published_at": "2026-05-30T11:12:00Z"`, `"published_at": "May 30"`, 1)
_, err := Parse([]byte(body))

View File

@@ -2,9 +2,9 @@ package state
import (
"fmt"
"net/url"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/link"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
)
@@ -30,7 +30,7 @@ func Validate(s DistributorState) error {
return fmt.Errorf("state source.manifest: %w", err)
}
if s.Links != nil && s.Links.PrimaryURL != "" {
if err := validateStateURL(s.Links.PrimaryURL); err != nil {
if err := link.ValidateHTTPURL(s.Links.PrimaryURL); err != nil {
return fmt.Errorf("state links.primary_url: %w", err)
}
}
@@ -70,7 +70,7 @@ func validateOutput(index int, output OutputFile) error {
return fmt.Errorf("state outputs[%d].transform is required for generated output", index)
}
if output.URL != "" {
if err := validateStateURL(output.URL); err != nil {
if err := link.ValidateHTTPURL(output.URL); err != nil {
return fmt.Errorf("state outputs[%d].url: %w", index, err)
}
}
@@ -82,23 +82,3 @@ func validateOutput(index int, output OutputFile) error {
}
return nil
}
func validateStateURL(value string) error {
parsed, err := url.Parse(value)
if err != nil {
return err
}
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return fmt.Errorf("must use http or https")
}
if parsed.Host == "" {
return fmt.Errorf("must include a host")
}
if parsed.RawQuery != "" {
return fmt.Errorf("must not include a query string")
}
if parsed.Fragment != "" {
return fmt.Errorf("must not include a fragment")
}
return nil
}

View File

@@ -3,7 +3,6 @@ package fake
import (
"bytes"
"context"
"errors"
"io"
"sort"
"strings"
@@ -133,14 +132,14 @@ func (b *Backend) Walk(ctx context.Context, prefix string, opts storage.WalkOpti
if err := storage.ValidatePrefix(prefix); err != nil {
return err
}
emitter := storage.NewWalkEmitter(ctx, backendName, opts, fn)
if entry, err := b.Stat(ctx, prefix); err == nil && entry.Type != storage.EntryTypeDirectory {
return emit(ctx, entry, opts, fn)
return storage.FinishWalk(emitter.Emit(entry))
} else if err != nil && !storage.IsNotFound(err) {
return err
}
entries := b.entries()
visited := 0
for _, entry := range entries {
if entry.Path == "" || !entryBelow(prefix, entry.Path) {
continue
@@ -148,33 +147,15 @@ func (b *Backend) Walk(ctx context.Context, prefix string, opts storage.WalkOpti
if !opts.Recursive && !isImmediateChild(prefix, entry.Path) {
continue
}
if opts.Limit > 0 && visited >= opts.Limit {
return nil
}
visited++
if err := ctx.Err(); err != nil {
return err
}
if err := fn(entry); err != nil {
if errors.Is(err, storage.ErrStopWalk) {
return nil
}
return storage.NewError(storage.OpWalk, backendName, entry.Path, storage.ErrUnknown, err)
if err := emitter.Emit(entry); err != nil {
return storage.FinishWalk(err)
}
}
return nil
}
func (b *Backend) HasAny(ctx context.Context, prefix string) (bool, error) {
found := false
err := b.Walk(ctx, prefix, storage.WalkOptions{Recursive: false, Limit: 1}, func(storage.Entry) error {
found = true
return storage.ErrStopWalk
})
if err != nil {
return false, err
}
return found, nil
return storage.HasAny(ctx, b, prefix)
}
func (b *Backend) DeleteManagedBundle(ctx context.Context, bundlePath string, managedOutputPaths []string, opts storage.DeleteOptions) error {
@@ -302,19 +283,6 @@ func (b *Backend) entries() []storage.Entry {
return entries
}
func emit(ctx context.Context, entry storage.Entry, opts storage.WalkOptions, fn storage.WalkFunc) error {
if opts.Limit > 0 && opts.Limit < 1 {
return nil
}
if err := ctx.Err(); err != nil {
return err
}
if err := fn(entry); err != nil && !errors.Is(err, storage.ErrStopWalk) {
return err
}
return nil
}
func entryBelow(prefix, path string) bool {
if prefix == "" {
return path != ""

63
internal/storage/walk.go Normal file
View File

@@ -0,0 +1,63 @@
package storage
import (
"context"
"errors"
)
type WalkEmitter struct {
ctx context.Context
backend string
opts WalkOptions
fn WalkFunc
count int
}
func NewWalkEmitter(ctx context.Context, backend string, opts WalkOptions, fn WalkFunc) *WalkEmitter {
return &WalkEmitter{
ctx: ctx,
backend: backend,
opts: opts,
fn: fn,
}
}
func (e *WalkEmitter) Emit(entry Entry) error {
if err := e.ctx.Err(); err != nil {
return err
}
if e.opts.Limit > 0 && e.count >= e.opts.Limit {
return ErrStopWalk
}
e.count++
if err := e.fn(entry); err != nil {
if errors.Is(err, ErrStopWalk) {
return ErrStopWalk
}
return NewError(OpWalk, e.backend, entry.Path, ErrUnknown, err)
}
return nil
}
func (e *WalkEmitter) LimitReached() bool {
return e.opts.Limit > 0 && e.count >= e.opts.Limit
}
func FinishWalk(err error) error {
if errors.Is(err, ErrStopWalk) {
return nil
}
return err
}
func HasAny(ctx context.Context, backend Backend, prefix string) (bool, error) {
found := false
err := backend.Walk(ctx, prefix, WalkOptions{Recursive: false, Limit: 1}, func(Entry) error {
found = true
return ErrStopWalk
})
if err != nil {
return false, err
}
return found, nil
}

View File

@@ -0,0 +1,107 @@
package storage
import (
"context"
"errors"
"testing"
)
func TestWalkEmitterHonorsLimit(t *testing.T) {
emitter := NewWalkEmitter(context.Background(), "test", WalkOptions{Limit: 2}, func(Entry) error {
return nil
})
if err := emitter.Emit(Entry{Path: "one"}); err != nil {
t.Fatalf("first Emit() error = %v", err)
}
if err := emitter.Emit(Entry{Path: "two"}); err != nil {
t.Fatalf("second Emit() error = %v", err)
}
if err := emitter.Emit(Entry{Path: "three"}); !errors.Is(err, ErrStopWalk) {
t.Fatalf("third Emit() error = %v, want ErrStopWalk", err)
}
if !emitter.LimitReached() {
t.Fatal("LimitReached() = false, want true")
}
}
func TestWalkEmitterStopsWithoutError(t *testing.T) {
emitter := NewWalkEmitter(context.Background(), "test", WalkOptions{}, func(Entry) error {
return ErrStopWalk
})
err := emitter.Emit(Entry{Path: "one"})
if !errors.Is(err, ErrStopWalk) {
t.Fatalf("Emit() error = %v, want ErrStopWalk", err)
}
if err := FinishWalk(err); err != nil {
t.Fatalf("FinishWalk() error = %v, want nil", err)
}
}
func TestWalkEmitterWrapsCallbackErrors(t *testing.T) {
callbackErr := errors.New("callback failed")
emitter := NewWalkEmitter(context.Background(), "test", WalkOptions{}, func(Entry) error {
return callbackErr
})
err := emitter.Emit(Entry{Path: "one"})
if !errors.Is(err, callbackErr) {
t.Fatalf("Emit() error = %v, want callback error", err)
}
var storageErr *Error
if !errors.As(err, &storageErr) {
t.Fatalf("Emit() error type = %T, want *Error", err)
}
if storageErr.Op != OpWalk || storageErr.Backend != "test" || storageErr.Path != "one" || storageErr.Kind != ErrUnknown {
t.Fatalf("wrapped error = %#v", storageErr)
}
}
func TestWalkEmitterHonorsContextCancellation(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
called := false
emitter := NewWalkEmitter(ctx, "test", WalkOptions{}, func(Entry) error {
called = true
return nil
})
err := emitter.Emit(Entry{Path: "one"})
if !errors.Is(err, context.Canceled) {
t.Fatalf("Emit() error = %v, want context.Canceled", err)
}
if called {
t.Fatal("callback was called after context cancellation")
}
}
func TestHasAnyUsesNonRecursiveLimitOneWalk(t *testing.T) {
backend := &recordingBackend{}
found, err := HasAny(context.Background(), backend, "bundle")
if err != nil {
t.Fatalf("HasAny() error = %v", err)
}
if !found {
t.Fatal("HasAny() = false, want true")
}
if backend.prefix != "bundle" {
t.Fatalf("walk prefix = %q, want bundle", backend.prefix)
}
if backend.opts != (WalkOptions{Recursive: false, Limit: 1}) {
t.Fatalf("walk options = %#v, want non-recursive limit one", backend.opts)
}
}
type recordingBackend struct {
Backend
prefix string
opts WalkOptions
}
func (b *recordingBackend) Walk(_ context.Context, prefix string, opts WalkOptions, fn WalkFunc) error {
b.prefix = prefix
b.opts = opts
return FinishWalk(fn(Entry{Path: "bundle/file.txt", Type: EntryTypeFile}))
}

View File

@@ -3,6 +3,7 @@ package testutil
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
@@ -106,6 +107,53 @@ func WriteFakeSourceBundle(t testing.TB, backend *fake.Backend, relative string,
return bundle.Bundle{RootRelativePath: relative, Manifest: manifest}
}
func WriteFakeFile(t testing.TB, backend *fake.Backend, path, data string) {
t.Helper()
if _, err := backend.WriteFile(context.Background(), path, []byte(data), storage.WriteOptions{}); err != nil {
t.Fatalf("write fake file %s: %v", path, err)
}
}
func AssertFakeFile(t testing.TB, backend *fake.Backend, path, want string) {
t.Helper()
data, err := backend.ReadFile(context.Background(), path)
if err != nil {
t.Fatalf("read fake file %s: %v", path, err)
}
if got := string(data); got != want {
t.Fatalf("fake file %s = %q, want %q", path, got, want)
}
}
func AssertFakeMissing(t testing.TB, backend *fake.Backend, path string) {
t.Helper()
if _, err := backend.Stat(context.Background(), path); !storage.IsNotFound(err) {
t.Fatalf("fake file %s stat error = %v, want not found", path, err)
}
}
func WriteFakeDestinationState(t testing.TB, backend *fake.Backend, relative string, manifest bundle.Manifest, opts DestinationStateOptions) state.DistributorState {
t.Helper()
destinationState := DestinationState(manifest, opts)
data, err := json.MarshalIndent(destinationState, "", " ")
if err != nil {
t.Fatalf("marshal destination state: %v", err)
}
statePath, err := storage.StatePath(relative)
if err != nil {
t.Fatalf("state path: %v", err)
}
WriteFakeFile(t, backend, statePath, string(append(data, '\n')))
for _, output := range destinationState.Outputs {
path, err := storage.Join(relative, output.Path)
if err != nil {
t.Fatalf("join output path: %v", err)
}
WriteFakeFile(t, backend, path, "old")
}
return destinationState
}
func WriteMinimalLocalConfig(t testing.TB, sourceRoot, destinationRoot string) string {
t.Helper()
return writeConfigFile(t, `
@@ -139,6 +187,136 @@ pipelines:
`)
}
func WriteLocalConfigWithPublishPolicy(t testing.TB, sourceRoot, destinationRoot string, publishSource, publishHTML bool) string {
t.Helper()
transformConfig := ""
if publishHTML {
transformConfig = `
transform:
markdown_to_html:
enabled: true
mode: sidecar`
}
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
publish:
source: `+fmt.Sprintf("%t", publishSource)+`
html: `+fmt.Sprintf("%t", publishHTML)+transformConfig+`
`)
}
func WriteLocalConfigWithPathMapping(t testing.TB, sourceRoot, destinationRoot, mode string) string {
t.Helper()
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
path_mapping:
mode: `+mode+`
`)
}
func WriteLocalConfigWithLinks(t testing.TB, sourceRoot, destinationRoot, pathMapping, baseURL, primary string, publishSource, publishHTML bool, transformMode string) string {
t.Helper()
transformConfig := ""
if publishHTML {
transformConfig = `
transform:
markdown_to_html:
enabled: true
mode: ` + transformMode
}
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
path_mapping:
mode: `+pathMapping+`
links:
base_url: `+baseURL+`
primary: `+primary+`
publish:
source: `+fmt.Sprintf("%t", publishSource)+`
html: `+fmt.Sprintf("%t", publishHTML)+transformConfig+`
`)
}
func WriteLocalConfigWithMarkdownTransform(t testing.TB, sourceRoot, destinationRoot string, publishSource, publishHTML bool, mode, input string) string {
t.Helper()
enabled := publishHTML
inputConfig := ""
if input != "" {
inputConfig = `
input: ` + input
}
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+destinationRoot+`
publish:
source: `+fmt.Sprintf("%t", publishSource)+`
html: `+fmt.Sprintf("%t", publishHTML)+`
transform:
markdown_to_html:
enabled: `+fmt.Sprintf("%t", enabled)+`
mode: `+mode+inputConfig+`
`)
}
func WriteMixedPolicyFanoutLocalConfig(t testing.TB, sourceRoot, archiveDestination, htmlDestination string) string {
t.Helper()
return writeConfigFile(t, `
pipelines:
- id: reports
source:
backend: local
path: `+sourceRoot+`
destinations:
- id: archive
backend: local
path: `+archiveDestination+`
publish:
source: true
html: false
- id: html
backend: local
path: `+htmlDestination+`
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar
`)
}
func WriteDestinationState(t testing.TB, root, relative string, manifest bundle.Manifest, opts DestinationStateOptions) state.DistributorState {
t.Helper()
bundleRoot := filepath.Join(root, filepath.FromSlash(relative))
@@ -175,6 +353,28 @@ func ReadDestinationState(t testing.TB, path string) state.DistributorState {
return destinationState
}
func AssertFile(t testing.TB, path, want string) {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read file %s: %v", path, err)
}
if got := string(data); got != want {
t.Fatalf("%s = %q, want %q", path, got, want)
}
}
func AssertFileContains(t testing.TB, path, want string) {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read file %s: %v", path, err)
}
if !strings.Contains(string(data), want) {
t.Fatalf("%s = %q, want substring %q", path, data, want)
}
}
func sourceFiles(opts BundleOptions) []SourceFile {
files := opts.Files
if files == nil {

View File

@@ -11,7 +11,8 @@ import (
var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
func validateDigest(value string) error {
// ValidateDigest reports whether value uses the lowercase sha256:<64 hex> form.
func ValidateDigest(value string) error {
if !digestPattern.MatchString(value) {
return fmt.Errorf("must be lowercase sha256:<64 hex>")
}

58
pkg/bundle/digest_test.go Normal file
View File

@@ -0,0 +1,58 @@
package bundle
import "testing"
func TestValidateDigest(t *testing.T) {
tests := []struct {
name string
value string
wantErr bool
}{
{
name: "valid",
value: "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
},
{
name: "uppercase hex",
value: "sha256:0123456789ABCDEF0123456789abcdef0123456789abcdef0123456789abcdef",
wantErr: true,
},
{
name: "missing prefix",
value: "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
wantErr: true,
},
{
name: "wrong algorithm",
value: "sha512:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
wantErr: true,
},
{
name: "short hex",
value: "sha256:0123456789abcdef",
wantErr: true,
},
{
name: "long hex",
value: "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0",
wantErr: true,
},
{
name: "non hex",
value: "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdeg",
wantErr: true,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
err := ValidateDigest(test.value)
if test.wantErr && err == nil {
t.Fatal("ValidateDigest() error = nil, want error")
}
if !test.wantErr && err != nil {
t.Fatalf("ValidateDigest() error = %v", err)
}
})
}
}

View File

@@ -13,7 +13,7 @@ func ValidateManifest(manifest Manifest) error {
if manifest.ID == "" {
return fmt.Errorf("id is required")
}
if err := validateDigest(manifest.Digest); err != nil {
if err := ValidateDigest(manifest.Digest); err != nil {
return fmt.Errorf("digest: %w", err)
}
if manifest.Created.IsZero() {
@@ -27,7 +27,7 @@ func ValidateManifest(manifest Manifest) error {
if err := ValidateSourcePath(file.Path); err != nil {
return fmt.Errorf("files[%d].path: %w", index, err)
}
if err := validateDigest(file.SHA256); err != nil {
if err := ValidateDigest(file.SHA256); err != nil {
return fmt.Errorf("files[%d].sha256: %w", index, err)
}
if file.Size < 0 {