Make remote publish locks generation-safe
This commit is contained in:
@@ -9,7 +9,8 @@ Upload run/session outputs to object storage and atomically advance remote curre
|
||||
- successful preceding stages from the [canonical stage set](overview.md#pipeline-stage-set)
|
||||
- invocation-scoped run files
|
||||
- resolved publish output rules
|
||||
- effective publish locks (static + remote merged lock set)
|
||||
- effective publish locks (static + remote merged lock set), revalidated at the
|
||||
remote commit boundary
|
||||
- durable previous-session cache files when present
|
||||
|
||||
## Outputs
|
||||
@@ -44,6 +45,9 @@ Exact remote placement and the operator workflow belong in
|
||||
- uploads and verifies every declared immutable object and the commit manifest;
|
||||
- updates `current/commit-pointer.json` exactly once, last; and
|
||||
- does not write the legacy `current/manifest.json` or `current/run_id.txt` pair.
|
||||
- rechecks remote lock state immediately before the pointer update. A newly
|
||||
committed lock aborts selection, leaving any uploaded immutable attempt
|
||||
unselected.
|
||||
|
||||
## Metadata Signals
|
||||
|
||||
@@ -66,7 +70,8 @@ Includes counts/lists for:
|
||||
undeclared entries are rejected or ignored before uploads begin.
|
||||
- run-local diagnostics, including Notarius receipt and stderr files, are
|
||||
archived only when recorded by the run manifest.
|
||||
- publish locks are not overridden by `--force`.
|
||||
- publish locks are not overridden by `--force`; remote locks are revalidated
|
||||
immediately before current-state selection.
|
||||
|
||||
The commit boundary and cleanup gate are normative architecture invariants; see
|
||||
[Architecture](../policy/architecture.md#publish-commit-boundary).
|
||||
|
||||
@@ -12,8 +12,10 @@ operator-selected storage fields and credential mechanisms belong in
|
||||
`storage.ObjectStore` interface:
|
||||
|
||||
- `List(ctx, prefix)`
|
||||
- `Read(ctx, key)` returns an object body and the generation observed with it
|
||||
- `Download(ctx, key, localPath)`
|
||||
- `Upload(ctx, localPath, key, opts)`
|
||||
- `UploadConditional(ctx, source, key, opts, condition)`
|
||||
- `Exists(ctx, key)`
|
||||
|
||||
Key invariant:
|
||||
@@ -31,8 +33,13 @@ not own discovery, defaults, or configuration validation.
|
||||
|
||||
- normalizes object keys.
|
||||
- `List` paginates and returns normalized `ObjectInfo`.
|
||||
- A truncated S3 listing must supply a new, non-empty continuation token;
|
||||
otherwise listing fails with bucket and prefix context instead of looping.
|
||||
- `Download` writes local files with parent directory creation.
|
||||
- `Upload` streams local file and returns remote metadata.
|
||||
- `Read` binds a returned body to its S3 ETag. `UploadConditional` maps an ETag
|
||||
match or absence precondition directly to the provider request and reports a
|
||||
failed precondition without performing a local check-then-write replacement.
|
||||
- `Exists` maps not-found responses to `false`.
|
||||
|
||||
## Invariants
|
||||
|
||||
@@ -236,7 +236,14 @@ Effective lock rules:
|
||||
- static and remote locks are merged;
|
||||
- static locks win on source collisions;
|
||||
- locked outputs are intentional skips;
|
||||
- lock add/remove commands mutate only remote lock state.
|
||||
- lock add/remove commands mutate only remote lock state through generation-bound
|
||||
conditional writes. A command retries a bounded number of concurrent
|
||||
conflicts while its invocation context remains active, so it never replaces a
|
||||
different lock-document generation; and
|
||||
- a publish re-reads remote locks immediately before it writes the current
|
||||
commit pointer. A lock committed before that recheck prevents selecting the
|
||||
new snapshot, even though its already-uploaded immutable objects may remain
|
||||
available for a later retry.
|
||||
|
||||
Examples:
|
||||
|
||||
|
||||
@@ -173,7 +173,9 @@ the immutable commit manifest, and finally the current commit pointer.
|
||||
`current/commit-pointer.json` is the sole mutable selector and must be written
|
||||
exactly once, last. Failed, incomplete, skipped, or uncommitted publish attempts
|
||||
must not be presented as current remote state. Publish locks remain authoritative
|
||||
and are not bypassed by a forced run.
|
||||
and are not bypassed by a forced run. Mutable remote locks use provider-enforced
|
||||
generation preconditions and are revalidated immediately before pointer
|
||||
selection; loss of that check leaves the prior committed snapshot current.
|
||||
|
||||
Automatic local cleanup is permitted only after a successful publish commit,
|
||||
only when explicitly configured, and only through the path-safety guardrails.
|
||||
|
||||
@@ -30,7 +30,7 @@ All stages are pending when this plan is created.
|
||||
| 12 | Centralize handled terminal-failure persistence | RSK-001, TST-002, SIM-001, COM-001 | Completed |
|
||||
| 13 | Introduce the immutable remote-commit model and legacy boundary | ARC-003 | Completed |
|
||||
| 14 | Publish through immutable commits and canonical mappings | COR-004, COR-011, DUP-002, TST-004 | Completed |
|
||||
| 15 | Make remote locks generation-safe and harden pagination | RSK-005, RSK-014 | Pending |
|
||||
| 15 | Make remote locks generation-safe and harden pagination | RSK-005, RSK-014 | Completed |
|
||||
| 16 | Persist retryable post-commit cleanup state | COR-006, COR-007 | Pending |
|
||||
| 17 | Bind restore/status to a committed snapshot and reject conflicts | COR-008, COR-009, TST-005 | Pending |
|
||||
| 18 | Serialize restore transitions and make restored paths portable | RSK-006, RSK-008 | Pending |
|
||||
|
||||
Reference in New Issue
Block a user