Add Markdown HTML publication

This commit is contained in:
2026-05-31 02:28:48 +00:00
parent e296361042
commit 5408f14195
21 changed files with 628 additions and 47 deletions

View File

@@ -20,7 +20,7 @@ distributor inspect
`version` prints the application name and version. The default development version is `dev`; release builds may replace it at build time. `version` prints the application name and version. The default development version is `dev`; release builds may replace it at build time.
`run --config <path>` executes configured local-to-local pipelines that publish source files only. `run --config <path>` executes configured local-to-local pipelines that publish source files, generated HTML files, or both.
`run --config <path> --dry-run` discovers source bundles, inspects destination state, and prints planned actions without writing files. `run --config <path> --dry-run` discovers source bundles, inspects destination state, and prints planned actions without writing files.
@@ -28,7 +28,7 @@ distributor inspect
`inspect <path>` validates discovered local source bundles and prints a concise normalized summary. `inspect <path>` validates discovered local source bundles and prints a concise normalized summary.
Remote backends and HTML publication are not implemented yet. Remote backends are not implemented yet.
## Flag reference ## Flag reference
@@ -70,3 +70,9 @@ Publish the local example:
```sh ```sh
go run ./cmd/distributor run --config examples/local-publish.yml go run ./cmd/distributor run --config examples/local-publish.yml
``` ```
Publish the local HTML example:
```sh
go run ./cmd/distributor run --config examples/local-html.yml
```

View File

@@ -10,7 +10,7 @@ If `--config` is omitted during run, the built-in default path is:
/usr/local/etc/distributor/config.yml /usr/local/etc/distributor/config.yml
``` ```
The current implementation supports local-to-local source-file publication. Remote backends and HTML publication are not implemented yet. The current implementation supports local-to-local publication of source files, generated HTML files, or both. Remote backends are not implemented yet.
## Minimal config ## Minimal config
@@ -74,7 +74,7 @@ Backends:
Destination policy: Destination policy:
- `publish.source`: publish source artifacts. - `publish.source`: publish source artifacts.
- `publish.html`: publish generated HTML artifacts. - `publish.html`: publish generated HTML artifacts from Markdown source files.
- `transfer.on_destination_same`: `skip` or `fail`, defaults to `skip`. - `transfer.on_destination_same`: `skip` or `fail`, defaults to `skip`.
- `transfer.on_destination_older`: `replace` or `fail`, defaults to `replace`. - `transfer.on_destination_older`: `replace` or `fail`, defaults to `replace`.
- `transfer.on_destination_newer`: `skip` or `fail`, defaults to `skip`. - `transfer.on_destination_newer`: `skip` or `fail`, defaults to `skip`.
@@ -82,6 +82,8 @@ Destination policy:
When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `transform.markdown_to_html.mode: sidecar` are required. When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `transform.markdown_to_html.mode: sidecar` are required.
Markdown-to-HTML sidecar generation writes `report.html` for `report.md` and does not mutate the source bundle.
## Secrets ## Secrets
Do not put literal secrets in config files. S3 credentials may refer to environment variable names with: Do not put literal secrets in config files. S3 credentials may refer to environment variable names with:

View File

@@ -8,7 +8,7 @@
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transfer policy, destination bundle path, and existing destination state. Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transfer policy, destination bundle path, and existing destination state.
Output is a plan with an action, reason, and selected source outputs. Execution writes selected source files and `.distributor.json` for publish or replacement actions. Output is a plan with an action, reason, and selected source or generated outputs. Execution writes selected source files, generated files, and `.distributor.json` for publish or replacement actions.
## Actions ## Actions
@@ -16,7 +16,7 @@ Supported actions are publish new, replace older destination, skip same source,
## Boundaries ## Boundaries
The current implementation publishes source files only. HTML generation and remote backend execution are not implemented. The current implementation publishes source files and Markdown-to-HTML sidecar outputs. Remote backend execution is not implemented.
The package uses `internal/state` for destination comparison and `internal/storage` for IO. It does not parse CLI flags or load config files. The package uses `internal/state` for destination comparison and `internal/storage` for IO. It does not parse CLI flags or load config files.

View File

@@ -0,0 +1,23 @@
# Transform
## Purpose
`internal/transform` defines generated publication artifacts. `internal/transform/markdown` implements Markdown-to-HTML sidecar generation.
## Inputs and outputs
Inputs are a validated source bundle and source backend. Outputs include destination path, source path, transform id, generated bytes, SHA-256, and size.
## Markdown behavior
Markdown files ending in `.md` generate `.html` files in the same logical directory. Non-Markdown files do not generate outputs. Raw HTML embedded in Markdown is not passed through by the renderer.
Generated HTML is deterministic for the same source content and transform configuration.
## Boundaries
Transforms do not publish files, mutate source bundles, or write destination state. Publish planning selects and writes transform outputs.
## Tests
Before changing transform behavior, inspect tests under `internal/transform`.

View File

@@ -14,6 +14,12 @@ Run the local publication:
go run ./cmd/distributor run --config examples/local-publish.yml go run ./cmd/distributor run --config examples/local-publish.yml
``` ```
Run the local HTML publication:
```sh
go run ./cmd/distributor run --config examples/local-html.yml
```
## Filesystem layout ## Filesystem layout
Source bundles are discovered beneath the configured local source root. Destination bundle paths preserve the source bundle path relative to that source root. Source bundles are discovered beneath the configured local source root. Destination bundle paths preserve the source bundle path relative to that source root.
@@ -22,7 +28,7 @@ The maintained example writes under `workspace/`, which is ignored by Git.
## Destination state ## Destination state
Each published destination bundle contains `.distributor.json`. This state file records the source manifest and copied source outputs. It is the authoritative marker that a destination path is managed by `distributor`. Each published destination bundle contains `.distributor.json`. This state file records the source manifest, copied source outputs, and generated outputs. It is the authoritative marker that a destination path is managed by `distributor`.
`manifest.json` from the source bundle is not copied as destination state. `manifest.json` from the source bundle is not copied as destination state.
@@ -36,4 +42,4 @@ If a write fails during local publication, `distributor` removes outputs written
## Caveats ## Caveats
Only local-to-local source-file publication is implemented. SSH, S3, HTML generation, notification, and force overwrite behavior are not implemented. Only local-to-local publication is implemented. SSH, S3, notification, and force overwrite behavior are not implemented.

16
examples/local-html.yml Normal file
View File

@@ -0,0 +1,16 @@
pipelines:
- id: example-html-bundle
source:
backend: local
path: examples/source-bundle
destinations:
- id: local-html
backend: local
path: workspace/published/html-bundle
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: sidecar

5
go.mod
View File

@@ -2,4 +2,7 @@ module gitea.maximumdirect.net/eric/distributor
go 1.26 go 1.26
require gopkg.in/yaml.v3 v3.0.1 require (
github.com/yuin/goldmark v1.8.2
gopkg.in/yaml.v3 v3.0.1
)

2
go.sum
View File

@@ -1,3 +1,5 @@
github.com/yuin/goldmark v1.8.2 h1:kEGpgqJXdgbkhcOgBxkC0X0PmoPG1ZyoZ117rDVp4zE=
github.com/yuin/goldmark v1.8.2/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=

View File

@@ -4,6 +4,7 @@ import (
"context" "context"
"fmt" "fmt"
"io" "io"
"strings"
"gitea.maximumdirect.net/eric/distributor/internal/adapters/local" "gitea.maximumdirect.net/eric/distributor/internal/adapters/local"
"gitea.maximumdirect.net/eric/distributor/internal/bundle" "gitea.maximumdirect.net/eric/distributor/internal/bundle"
@@ -73,6 +74,7 @@ func runConfig(ctx context.Context, cfg config.Config, options RunOptions) error
DestinationBackend: destinationBackend, DestinationBackend: destinationBackend,
DestinationBundlePath: sourceBundle.RootRelativePath, DestinationBundlePath: sourceBundle.RootRelativePath,
Publish: *destination.Publish, Publish: *destination.Publish,
Transform: destination.Transform,
Transfer: destination.Transfer, Transfer: destination.Transfer,
DistributorVersion: Version, DistributorVersion: Version,
} }
@@ -102,5 +104,16 @@ func writePlanLine(w io.Writer, plan publish.Plan, planErr error) {
fmt.Fprintf(w, " - bundle=%s destination=%s action=error reason=%q\n", displayBundlePath(plan.BundlePath), plan.DestinationID, planErr.Error()) fmt.Fprintf(w, " - bundle=%s destination=%s action=error reason=%q\n", displayBundlePath(plan.BundlePath), plan.DestinationID, planErr.Error())
return return
} }
fmt.Fprintf(w, " - bundle=%s destination=%s action=%s outputs=%d reason=%q\n", displayBundlePath(plan.BundlePath), plan.DestinationID, plan.Action, len(plan.Outputs), plan.Reason) fmt.Fprintf(w, " - bundle=%s destination=%s action=%s outputs=%s reason=%q\n", displayBundlePath(plan.BundlePath), plan.DestinationID, plan.Action, outputSummary(plan.Outputs), plan.Reason)
}
func outputSummary(outputs []publish.Output) string {
if len(outputs) == 0 {
return "none"
}
paths := make([]string, 0, len(outputs))
for _, output := range outputs {
paths = append(paths, output.DestinationPath)
}
return strings.Join(paths, ",")
} }

View File

@@ -4,6 +4,7 @@ import (
"bytes" "bytes"
"context" "context"
"encoding/json" "encoding/json"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -34,7 +35,7 @@ func TestRunDryRunPrintsConfigSummary(t *testing.T) {
for _, want := range []string{ for _, want := range []string{
"Configured pipelines: 1", "Configured pipelines: 1",
"- reports: source=local bundles=1 destinations=1", "- reports: source=local bundles=1 destinations=1",
"bundle=. destination=archive action=publish_new", "bundle=. destination=archive action=publish_new outputs=report.md,summary.txt",
} { } {
if !strings.Contains(output, want) { if !strings.Contains(output, want) {
t.Fatalf("Run() output = %q, want substring %q", output, want) t.Fatalf("Run() output = %q, want substring %q", output, want)
@@ -72,6 +73,103 @@ func TestRunPublishesNewLocalBundle(t *testing.T) {
} }
} }
func TestRunPublishesHTMLOnly(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, false, true)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFileContains(t, filepath.Join(destinationRoot, "report.html"), "<h1>Report</h1>")
if _, err := os.Stat(filepath.Join(destinationRoot, "report.md")); !os.IsNotExist(err) {
t.Fatalf("report.md stat error = %v, want not exist", err)
}
destinationState := readStateFile(t, filepath.Join(destinationRoot, ".distributor.json"))
if got, want := len(destinationState.Outputs), 1; got != want {
t.Fatalf("state output count = %d, want %d", got, want)
}
output := destinationState.Outputs[0]
if output.Kind != state.OutputKindGenerated || output.Transform != "markdown_to_html" || output.Path != "report.html" || output.SourcePath != "report.md" {
t.Fatalf("generated output metadata = %#v", output)
}
}
func TestRunPublishesSourceAndHTML(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, true, true)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
assertFileContains(t, filepath.Join(destinationRoot, "report.html"), "<p>Sunny.</p>")
assertFile(t, filepath.Join(destinationRoot, "summary.txt"), "Summary\n")
destinationState := readStateFile(t, filepath.Join(destinationRoot, ".distributor.json"))
if got, want := len(destinationState.Outputs), 3; got != want {
t.Fatalf("state output count = %d, want %d", got, want)
}
}
func TestRunDoesNotMutateSourceBundle(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
sourcePath := filepath.Join(sourceRoot, "report.md")
before, err := os.ReadFile(sourcePath)
if err != nil {
t.Fatalf("read source before: %v", err)
}
err = Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, true, true)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
after, err := os.ReadFile(sourcePath)
if err != nil {
t.Fatalf("read source after: %v", err)
}
if string(after) != string(before) {
t.Fatalf("source changed from %q to %q", before, after)
}
}
func TestRunFailsOnOutputPathCollision(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{ExtraFiles: []testFile{{Path: "report.html", Data: "<p>source html</p>\n"}}})
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, true, true)})
if err == nil || !strings.Contains(err.Error(), "destination output path collision") {
t.Fatalf("Run() error = %v, want collision", err)
}
if entries, err := os.ReadDir(destinationRoot); err != nil || len(entries) != 0 {
t.Fatalf("destination entries = %v err=%v, want empty", entries, err)
}
}
func TestRunDryRunReportsGeneratedOutputs(t *testing.T) {
sourceRoot := t.TempDir()
destinationRoot := t.TempDir()
writeSourceBundle(t, sourceRoot, "", testBundleOptions{})
var stdout bytes.Buffer
err := Run(context.Background(), RunOptions{
ConfigPath: writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, false, true),
DryRun: true,
Stdout: &stdout,
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if !strings.Contains(stdout.String(), "outputs=report.html") {
t.Fatalf("stdout = %q, want generated output path", stdout.String())
}
}
func TestRunSkipsWhenDestinationStateMatches(t *testing.T) { func TestRunSkipsWhenDestinationStateMatches(t *testing.T) {
sourceRoot := t.TempDir() sourceRoot := t.TempDir()
destinationRoot := t.TempDir() destinationRoot := t.TempDir()
@@ -194,8 +292,14 @@ func TestRunDryRunDoesNotWrite(t *testing.T) {
} }
type testBundleOptions struct { type testBundleOptions struct {
ID string ID string
Created time.Time Created time.Time
ExtraFiles []testFile
}
type testFile struct {
Path string
Data string
} }
func writeSourceBundle(t *testing.T, root, relative string, opts testBundleOptions) bundle.Manifest { func writeSourceBundle(t *testing.T, root, relative string, opts testBundleOptions) bundle.Manifest {
@@ -217,6 +321,12 @@ func writeSourceBundle(t *testing.T, root, relative string, opts testBundleOptio
{path: "report.md", data: "# Report\nSunny.\n"}, {path: "report.md", data: "# Report\nSunny.\n"},
{path: "summary.txt", data: "Summary\n"}, {path: "summary.txt", data: "Summary\n"},
} }
for _, extra := range opts.ExtraFiles {
files = append(files, struct {
path string
data string
}{path: extra.Path, data: extra.Data})
}
manifestFiles := make([]bundle.ManifestFile, 0, len(files)) manifestFiles := make([]bundle.ManifestFile, 0, len(files))
for _, file := range files { for _, file := range files {
if err := os.WriteFile(filepath.Join(bundleRoot, filepath.FromSlash(file.path)), []byte(file.data), 0o600); err != nil { if err := os.WriteFile(filepath.Join(bundleRoot, filepath.FromSlash(file.path)), []byte(file.data), 0o600); err != nil {
@@ -248,6 +358,19 @@ func writeSourceBundle(t *testing.T, root, relative string, opts testBundleOptio
func writeLocalConfig(t *testing.T, sourceRoot, destinationRoot string) string { func writeLocalConfig(t *testing.T, sourceRoot, destinationRoot string) string {
t.Helper() t.Helper()
return writeLocalConfigWithPolicy(t, sourceRoot, destinationRoot, true, false)
}
func writeLocalConfigWithPolicy(t *testing.T, sourceRoot, destinationRoot string, publishSource, publishHTML bool) string {
t.Helper()
transformConfig := ""
if publishHTML {
transformConfig = `
transform:
markdown_to_html:
enabled: true
mode: sidecar`
}
return writeConfigFile(t, ` return writeConfigFile(t, `
pipelines: pipelines:
- id: reports - id: reports
@@ -259,8 +382,8 @@ pipelines:
backend: local backend: local
path: `+destinationRoot+` path: `+destinationRoot+`
publish: publish:
source: true source: `+fmt.Sprintf("%t", publishSource)+`
html: false html: `+fmt.Sprintf("%t", publishHTML)+transformConfig+`
`) `)
} }
@@ -341,3 +464,14 @@ func assertFile(t *testing.T, path, want string) {
t.Fatalf("%s = %q, want %q", path, got, want) t.Fatalf("%s = %q, want %q", path, got, want)
} }
} }
func assertFileContains(t *testing.T, path, want string) {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read file %s: %v", path, err)
}
if !strings.Contains(string(data), want) {
t.Fatalf("%s = %q, want substring %q", path, data, want)
}
}

View File

@@ -265,6 +265,7 @@ func TestExampleConfigsLoad(t *testing.T) {
for _, path := range []string{ for _, path := range []string{
"../../examples/local-to-local.yml", "../../examples/local-to-local.yml",
"../../examples/local-publish.yml", "../../examples/local-publish.yml",
"../../examples/local-html.yml",
"../../examples/fan-out.yml", "../../examples/fan-out.yml",
} { } {
t.Run(path, func(t *testing.T) { t.Run(path, func(t *testing.T) {

View File

@@ -36,20 +36,23 @@ func Execute(ctx context.Context, req Request, plan Plan) error {
_ = req.DestinationBackend.DeleteManagedBundle(ctx, req.DestinationBundlePath, managedOutputPaths(writtenOutputs), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true}) _ = req.DestinationBackend.DeleteManagedBundle(ctx, req.DestinationBundlePath, managedOutputPaths(writtenOutputs), storage.DeleteOptions{IgnoreMissing: true, PruneEmptyDirs: true})
} }
for _, output := range plan.Outputs { for _, output := range plan.Outputs {
sourcePath, err := storage.Join(req.SourceBundle.RootRelativePath, output.SourcePath)
if err != nil {
cleanup()
return err
}
destinationPath, err := storage.Join(req.DestinationBundlePath, output.DestinationPath) destinationPath, err := storage.Join(req.DestinationBundlePath, output.DestinationPath)
if err != nil { if err != nil {
cleanup() cleanup()
return err return err
} }
data, err := req.SourceBackend.ReadFile(ctx, sourcePath) data := output.Data
if err != nil { if output.Kind == state.OutputKindSource {
cleanup() sourcePath, err := storage.Join(req.SourceBundle.RootRelativePath, output.SourcePath)
return err if err != nil {
cleanup()
return err
}
data, err = req.SourceBackend.ReadFile(ctx, sourcePath)
if err != nil {
cleanup()
return err
}
} }
if _, err := req.DestinationBackend.WriteFile(ctx, destinationPath, data, storage.WriteOptions{Overwrite: false, PreferAtomic: true}); err != nil { if _, err := req.DestinationBackend.WriteFile(ctx, destinationPath, data, storage.WriteOptions{Overwrite: false, PreferAtomic: true}); err != nil {
cleanup() cleanup()

View File

@@ -1,29 +1,62 @@
package publish package publish
import ( import (
"context"
"fmt" "fmt"
"gitea.maximumdirect.net/eric/distributor/internal/state" "gitea.maximumdirect.net/eric/distributor/internal/state"
"gitea.maximumdirect.net/eric/distributor/internal/storage" "gitea.maximumdirect.net/eric/distributor/internal/storage"
"gitea.maximumdirect.net/eric/distributor/internal/transform"
) )
func PlanSourceOutputs(req Request) ([]Output, error) { func PlanOutputs(ctx context.Context, req Request) ([]Output, error) {
if !req.Publish.Source { var outputs []Output
return nil, nil if req.Publish.Source {
} sourceOutputs, err := PlanSourceOutputs(req)
outputs := make([]Output, 0, len(req.SourceBundle.Manifest.Files)) if err != nil {
seen := make(map[string]struct{}, len(req.SourceBundle.Manifest.Files)) return nil, err
for _, file := range req.SourceBundle.Manifest.Files {
if _, exists := seen[file.Path]; exists {
return nil, fmt.Errorf("destination output path collision: %s", file.Path)
} }
seen[file.Path] = struct{}{} outputs = append(outputs, sourceOutputs...)
}
if req.Publish.HTML {
generatedOutputs, err := markdownTransformer().Generate(ctx, transform.Request{
SourceBundle: req.SourceBundle,
SourceBackend: req.SourceBackend,
})
if err != nil {
return nil, err
}
if len(generatedOutputs) == 0 {
return nil, fmt.Errorf("publish html requested but no markdown source files were found")
}
for _, generated := range generatedOutputs {
outputs = append(outputs, Output{
SourcePath: generated.SourcePath,
DestinationPath: generated.Path,
Kind: state.OutputKindGenerated,
Transform: generated.Transform,
Data: generated.Data,
SHA256: generated.SHA256,
Size: generated.Size,
})
}
}
if err := rejectOutputCollisions(outputs); err != nil {
return nil, err
}
return outputs, nil
}
func PlanSourceOutputs(req Request) ([]Output, error) {
outputs := make([]Output, 0, len(req.SourceBundle.Manifest.Files))
for _, file := range req.SourceBundle.Manifest.Files {
if err := storage.ValidatePath(file.Path); err != nil { if err := storage.ValidatePath(file.Path); err != nil {
return nil, fmt.Errorf("destination output path %q: %w", file.Path, err) return nil, fmt.Errorf("destination output path %q: %w", file.Path, err)
} }
outputs = append(outputs, Output{ outputs = append(outputs, Output{
SourcePath: file.Path, SourcePath: file.Path,
DestinationPath: file.Path, DestinationPath: file.Path,
Kind: state.OutputKindSource,
SHA256: file.SHA256, SHA256: file.SHA256,
Size: file.Size, Size: file.Size,
}) })
@@ -31,13 +64,28 @@ func PlanSourceOutputs(req Request) ([]Output, error) {
return outputs, nil return outputs, nil
} }
func rejectOutputCollisions(outputs []Output) error {
seen := make(map[string]struct{}, len(outputs))
for _, output := range outputs {
if err := storage.ValidatePath(output.DestinationPath); err != nil {
return fmt.Errorf("destination output path %q: %w", output.DestinationPath, err)
}
if _, exists := seen[output.DestinationPath]; exists {
return fmt.Errorf("destination output path collision: %s", output.DestinationPath)
}
seen[output.DestinationPath] = struct{}{}
}
return nil
}
func stateOutputs(outputs []Output) []state.OutputFile { func stateOutputs(outputs []Output) []state.OutputFile {
files := make([]state.OutputFile, 0, len(outputs)) files := make([]state.OutputFile, 0, len(outputs))
for _, output := range outputs { for _, output := range outputs {
files = append(files, state.OutputFile{ files = append(files, state.OutputFile{
Path: output.DestinationPath, Path: output.DestinationPath,
Kind: state.OutputKindSource, Kind: output.Kind,
SourcePath: output.SourcePath, SourcePath: output.SourcePath,
Transform: output.Transform,
SHA256: output.SHA256, SHA256: output.SHA256,
Size: output.Size, Size: output.Size,
}) })

View File

@@ -1,29 +1,101 @@
package publish package publish
import ( import (
"context"
"testing" "testing"
"time" "time"
"gitea.maximumdirect.net/eric/distributor/internal/bundle" "gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/config" "gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
) )
func TestPlanSourceOutputsRejectsCollision(t *testing.T) { func TestPlanOutputsRejectsCollision(t *testing.T) {
_, err := PlanSourceOutputs(Request{ sourceBackend := fake.New()
if _, err := sourceBackend.WriteFile(context.Background(), "report.md", []byte("# Report\n"), storage.WriteOptions{}); err != nil {
t.Fatalf("WriteFile report.md error = %v", err)
}
if _, err := sourceBackend.WriteFile(context.Background(), "report.html", []byte("<p>source html</p>\n"), storage.WriteOptions{}); err != nil {
t.Fatalf("WriteFile report.html error = %v", err)
}
reportDigest := bundle.FileDigest([]byte("# Report\n"))
htmlDigest := bundle.FileDigest([]byte("<p>source html</p>\n"))
files := []bundle.ManifestFile{
{Path: "report.md", SHA256: reportDigest, Size: 9},
{Path: "report.html", SHA256: htmlDigest, Size: 19},
}
_, err := PlanOutputs(context.Background(), Request{
SourceBackend: sourceBackend,
SourceBundle: bundle.Bundle{ SourceBundle: bundle.Bundle{
Manifest: bundle.Manifest{ Manifest: bundle.Manifest{
SchemaVersion: 1, SchemaVersion: 1,
ID: "bundle", ID: "bundle",
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC), Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
Files: []bundle.ManifestFile{ Digest: bundle.BundleDigest(files),
{Path: "report.md", SHA256: "sha256:1111111111111111111111111111111111111111111111111111111111111111", Size: 1}, Files: files,
{Path: "report.md", SHA256: "sha256:2222222222222222222222222222222222222222222222222222222222222222", Size: 1},
},
}, },
}, },
Publish: config.PublishPolicy{Source: true}, Publish: config.PublishPolicy{Source: true, HTML: true},
Transform: config.Transform{MarkdownToHTML: &config.MarkdownToHTML{Enabled: true, Mode: config.TransformModeSidecar}},
}) })
if err == nil { if err == nil {
t.Fatal("PlanSourceOutputs() error = nil, want collision") t.Fatal("PlanSourceOutputs() error = nil, want collision")
} }
} }
func TestPlanOutputsRejectsHTMLWithoutMarkdown(t *testing.T) {
sourceBackend := fake.New()
if _, err := sourceBackend.WriteFile(context.Background(), "summary.txt", []byte("Summary\n"), storage.WriteOptions{}); err != nil {
t.Fatalf("WriteFile summary.txt error = %v", err)
}
files := []bundle.ManifestFile{{Path: "summary.txt", SHA256: bundle.FileDigest([]byte("Summary\n")), Size: 8}}
_, err := PlanOutputs(context.Background(), Request{
SourceBackend: sourceBackend,
SourceBundle: bundle.Bundle{Manifest: bundle.Manifest{
SchemaVersion: 1,
ID: "bundle",
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
Digest: bundle.BundleDigest(files),
Files: files,
}},
Publish: config.PublishPolicy{HTML: true},
Transform: config.Transform{MarkdownToHTML: &config.MarkdownToHTML{Enabled: true, Mode: config.TransformModeSidecar}},
})
if err == nil {
t.Fatal("PlanOutputs() error = nil, want no markdown failure")
}
}
func TestBuildRejectsHTMLWithoutTransform(t *testing.T) {
sourceBackend := fake.New()
destinationBackend := fake.New()
if _, err := sourceBackend.WriteFile(context.Background(), "report.md", []byte("# Report\n"), storage.WriteOptions{}); err != nil {
t.Fatalf("WriteFile report.md error = %v", err)
}
files := []bundle.ManifestFile{{Path: "report.md", SHA256: bundle.FileDigest([]byte("# Report\n")), Size: 9}}
_, err := Build(context.Background(), Request{
PipelineID: "reports",
DestinationID: "archive",
SourceBackend: sourceBackend,
DestinationBackend: destinationBackend,
DestinationBundlePath: "",
SourceBundle: bundle.Bundle{Manifest: bundle.Manifest{
SchemaVersion: 1,
ID: "bundle",
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
Digest: bundle.BundleDigest(files),
Files: files,
}},
Publish: config.PublishPolicy{HTML: true},
Transfer: config.TransferPolicy{
OnDestinationSame: config.TransferActionSkip,
OnDestinationOlder: config.TransferActionReplace,
OnDestinationNewer: config.TransferActionSkip,
OnConflict: config.TransferActionFail,
},
})
if err == nil {
t.Fatal("Build() error = nil, want missing transform error")
}
}

View File

@@ -8,6 +8,8 @@ import (
"gitea.maximumdirect.net/eric/distributor/internal/config" "gitea.maximumdirect.net/eric/distributor/internal/config"
"gitea.maximumdirect.net/eric/distributor/internal/state" "gitea.maximumdirect.net/eric/distributor/internal/state"
"gitea.maximumdirect.net/eric/distributor/internal/storage" "gitea.maximumdirect.net/eric/distributor/internal/storage"
"gitea.maximumdirect.net/eric/distributor/internal/transform"
markdowntransform "gitea.maximumdirect.net/eric/distributor/internal/transform/markdown"
) )
type Action string type Action string
@@ -29,6 +31,7 @@ type Request struct {
DestinationBackend storage.Backend DestinationBackend storage.Backend
DestinationBundlePath string DestinationBundlePath string
Publish config.PublishPolicy Publish config.PublishPolicy
Transform config.Transform
Transfer config.TransferPolicy Transfer config.TransferPolicy
DistributorVersion string DistributorVersion string
} }
@@ -48,6 +51,9 @@ type Plan struct {
type Output struct { type Output struct {
SourcePath string SourcePath string
DestinationPath string DestinationPath string
Kind string
Transform string
Data []byte
SHA256 string SHA256 string
Size int64 Size int64
} }
@@ -56,7 +62,7 @@ func Build(ctx context.Context, req Request) (Plan, error) {
if err := validateRequest(req); err != nil { if err := validateRequest(req); err != nil {
return Plan{}, err return Plan{}, err
} }
outputs, err := PlanSourceOutputs(req) outputs, err := PlanOutputs(ctx, req)
if err != nil { if err != nil {
return Plan{}, err return Plan{}, err
} }
@@ -96,15 +102,21 @@ func validateRequest(req Request) error {
if req.DestinationBackend == nil { if req.DestinationBackend == nil {
return fmt.Errorf("destination backend is required") return fmt.Errorf("destination backend is required")
} }
if req.Publish.HTML { if !req.Publish.Source && !req.Publish.HTML {
return fmt.Errorf("publish html is not implemented") return fmt.Errorf("publish source or html must be enabled")
} }
if !req.Publish.Source { if req.Publish.HTML {
return fmt.Errorf("publish source must be enabled") if req.Transform.MarkdownToHTML == nil || !req.Transform.MarkdownToHTML.Enabled || req.Transform.MarkdownToHTML.Mode != config.TransformModeSidecar {
return fmt.Errorf("publish html requires markdown_to_html transform enabled with sidecar mode")
}
} }
return nil return nil
} }
func markdownTransformer() transform.Transformer {
return markdowntransform.New()
}
func actionForComparison(comparison state.Comparison, transfer config.TransferPolicy) (Action, string) { func actionForComparison(comparison state.Comparison, transfer config.TransferPolicy) (Action, string) {
switch comparison.Outcome { switch comparison.Outcome {
case state.OutcomeDestinationAbsent: case state.OutcomeDestinationAbsent:

View File

@@ -0,0 +1,57 @@
package markdown
import (
"bytes"
"context"
"fmt"
"strings"
"github.com/yuin/goldmark"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
"gitea.maximumdirect.net/eric/distributor/internal/transform"
)
type Transformer struct {
renderer goldmark.Markdown
}
func New() *Transformer {
return &Transformer{renderer: goldmark.New()}
}
func (t *Transformer) Generate(ctx context.Context, req transform.Request) ([]transform.Output, error) {
if t.renderer == nil {
t.renderer = goldmark.New()
}
var outputs []transform.Output
for _, file := range req.SourceBundle.Manifest.Files {
if !strings.HasSuffix(file.Path, ".md") {
continue
}
sourcePath, err := storage.Join(req.SourceBundle.RootRelativePath, file.Path)
if err != nil {
return nil, err
}
data, err := req.SourceBackend.ReadFile(ctx, sourcePath)
if err != nil {
return nil, fmt.Errorf("read markdown source %q: %w", file.Path, err)
}
var rendered bytes.Buffer
if err := t.renderer.Convert(data, &rendered); err != nil {
return nil, fmt.Errorf("render markdown source %q: %w", file.Path, err)
}
html := wrapHTML(rendered.Bytes())
outputPath := strings.TrimSuffix(file.Path, ".md") + ".html"
outputs = append(outputs, transform.Output{
Path: outputPath,
SourcePath: file.Path,
Transform: transform.MarkdownToHTML,
Data: html,
SHA256: bundle.FileDigest(html),
Size: int64(len(html)),
})
}
return outputs, nil
}

View File

@@ -0,0 +1,113 @@
package markdown
import (
"context"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
"gitea.maximumdirect.net/eric/distributor/internal/storage/fake"
"gitea.maximumdirect.net/eric/distributor/internal/transform"
)
func TestGenerateMarkdownSidecar(t *testing.T) {
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
outputs, err := New().Generate(context.Background(), transform.Request{SourceBackend: backend, SourceBundle: sourceBundle})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
if got, want := len(outputs), 1; got != want {
t.Fatalf("output count = %d, want %d", got, want)
}
output := outputs[0]
if output.Path != "report.html" {
t.Fatalf("path = %q, want report.html", output.Path)
}
if output.SourcePath != "report.md" || output.Transform != transform.MarkdownToHTML {
t.Fatalf("metadata = %#v", output)
}
html := string(output.Data)
for _, want := range []string{"<!doctype html>", "<h1>Title</h1>", "<p>Hello.</p>"} {
if !strings.Contains(html, want) {
t.Fatalf("html = %q, want substring %q", html, want)
}
}
if output.SHA256 != bundle.FileDigest(output.Data) || output.Size != int64(len(output.Data)) {
t.Fatalf("digest/size metadata = %s/%d", output.SHA256, output.Size)
}
}
func TestGenerateIgnoresNonMarkdown(t *testing.T) {
backend := fake.New()
if _, err := backend.WriteFile(context.Background(), "summary.txt", []byte("Summary\n"), storage.WriteOptions{}); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
sourceBundle := bundle.Bundle{Manifest: bundle.Manifest{
SchemaVersion: 1,
ID: "bundle",
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
Files: []bundle.ManifestFile{{Path: "summary.txt", SHA256: bundle.FileDigest([]byte("Summary\n")), Size: 8}},
}}
outputs, err := New().Generate(context.Background(), transform.Request{SourceBackend: backend, SourceBundle: sourceBundle})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
if len(outputs) != 0 {
t.Fatalf("outputs = %#v, want none", outputs)
}
}
func TestGenerateDoesNotPassRawHTML(t *testing.T) {
backend, sourceBundle := markdownFixture(t, "# Title\n\n<script>alert('x')</script>\n")
outputs, err := New().Generate(context.Background(), transform.Request{SourceBackend: backend, SourceBundle: sourceBundle})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
html := string(outputs[0].Data)
if strings.Contains(html, "<script>") {
t.Fatalf("html contains raw script: %q", html)
}
if !strings.Contains(html, "raw HTML omitted") && !strings.Contains(html, "&lt;script&gt;") {
t.Fatalf("html = %q, want raw HTML disabled or escaped", html)
}
}
func TestGenerateDeterministicOutput(t *testing.T) {
backend, sourceBundle := markdownFixture(t, "# Title\n\nHello.\n")
first, err := New().Generate(context.Background(), transform.Request{SourceBackend: backend, SourceBundle: sourceBundle})
if err != nil {
t.Fatalf("first Generate() error = %v", err)
}
second, err := New().Generate(context.Background(), transform.Request{SourceBackend: backend, SourceBundle: sourceBundle})
if err != nil {
t.Fatalf("second Generate() error = %v", err)
}
if string(first[0].Data) != string(second[0].Data) {
t.Fatalf("outputs differ:\n%s\n%s", first[0].Data, second[0].Data)
}
}
func markdownFixture(t *testing.T, markdown string) (*fake.Backend, bundle.Bundle) {
t.Helper()
backend := fake.New()
if _, err := backend.WriteFile(context.Background(), "report.md", []byte(markdown), storage.WriteOptions{}); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
if _, err := backend.WriteFile(context.Background(), "summary.txt", []byte("Summary\n"), storage.WriteOptions{}); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
files := []bundle.ManifestFile{
{Path: "report.md", SHA256: bundle.FileDigest([]byte(markdown)), Size: int64(len(markdown))},
{Path: "summary.txt", SHA256: bundle.FileDigest([]byte("Summary\n")), Size: 8},
}
manifest := bundle.Manifest{
SchemaVersion: 1,
ID: "bundle",
Created: time.Date(2026, 5, 30, 11, 10, 0, 0, time.UTC),
Files: files,
}
manifest.Digest = bundle.BundleDigest(manifest.Files)
return backend, bundle.Bundle{Manifest: manifest}
}

View File

@@ -0,0 +1,11 @@
package markdown
import "bytes"
func wrapHTML(body []byte) []byte {
var buf bytes.Buffer
buf.WriteString("<!doctype html>\n<html lang=\"en\">\n<head>\n<meta charset=\"utf-8\">\n<title></title>\n</head>\n<body>\n")
buf.Write(body)
buf.WriteString("</body>\n</html>\n")
return buf.Bytes()
}

View File

@@ -0,0 +1,3 @@
package transform
const MarkdownToHTML = "markdown_to_html"

View File

@@ -0,0 +1,30 @@
package transform
import "fmt"
type Registry struct {
transformers map[string]Transformer
}
func NewRegistry() *Registry {
return &Registry{transformers: make(map[string]Transformer)}
}
func (r *Registry) Register(name string, transformer Transformer) error {
if name == "" {
return fmt.Errorf("transform name is required")
}
if transformer == nil {
return fmt.Errorf("transformer %s is nil", name)
}
if _, exists := r.transformers[name]; exists {
return fmt.Errorf("transform %s is already registered", name)
}
r.transformers[name] = transformer
return nil
}
func (r *Registry) Get(name string) (Transformer, bool) {
transformer, ok := r.transformers[name]
return transformer, ok
}

View File

@@ -0,0 +1,26 @@
package transform
import (
"context"
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
"gitea.maximumdirect.net/eric/distributor/internal/storage"
)
type Output struct {
Path string
SourcePath string
Transform string
Data []byte
SHA256 string
Size int64
}
type Request struct {
SourceBundle bundle.Bundle
SourceBackend storage.Backend
}
type Transformer interface {
Generate(ctx context.Context, req Request) ([]Output, error)
}