A replicated version purge reaches the peer without the governance
bypass header, so a GOVERNANCE-retained version deleted on the source
with x-amz-bypass-governance-retention was rejected by the peer's WORM
deletion gate forever: retryStats ended at a permanent failed count and
the sites stayed diverged (issue #6850).
The source is authoritative for such a purge: the same WORM gate
already ran there, and GOVERNANCE retention with an authorized bypass
is the only lock state it can purge through. The peer's commit-time
deletion gate now treats an authorized replication delete addressed to
an explicit version as carrying that judged bypass, reusing the same
trust judgment as the replication write exemption
(ObjectOptions::replication_request, set only after the handler
authorized ReplicateDeleteAction). COMPLIANCE retention and legal hold
keep blocking replicated purges, and a plain client delete without the
bypass header stays rejected.
fix(ecstore): require verification source for degraded GET
Fail closed when reconstruction has only an exact decode quorum, because no surplus source remains to validate the rebuilt data. Cover both erasure engines and the data-shards-only rollout gate.
Co-authored-by: heihutu <[email protected]>
fix(ecstore): hedge data-only GET with parity
Route the opt-in data-shards-only lockstep path through the bounded parity race and preserve deferred parity reserves across canceled hedges.
Co-authored-by: heihutu <[email protected]>
Use the seqlock version as the publication fence and keep payload reads relaxed while validating the final version. This reduces ordering overhead in disk metric recording and snapshot collection without changing the rolling-window contract.
Co-authored-by: heihutu <[email protected]>
fix(ecstore): classify system metadata volume failures
Preserve retryable quorum errors when system metadata reads or writes encounter missing volumes, and cover the create-bucket data-usage path with regressions.
Co-authored-by: heihutu <[email protected]>
fix(replication): surface per-object failures and abort orphaned multipart uploads
Replication could mark an object FAILED with no server-log line naming
the object: the target-offline skip paths logged at debug without the
object key, and several failure branches omitted the key entirely. A
failed multipart transfer also leaked its incomplete upload on the
target, since nothing ever aborted it after CreateMultipartUpload
succeeded.
Log the offline skips at warn with the object key (they report the
object FAILED, matching the per-object put_object failure level), add
the object field to the remaining failure branches, and abort the
target-side multipart upload best-effort on any post-create failure
without masking the original transfer error.
Fixes#6825
fix(replication): never mint delete markers when replicating a version purge
Heal/resync/MRF rebuilds of a delete-marker version purge carry
delete_marker: true together with a purge-shaped entry. Passing that flag
straight into replication_delete_remove_options made the target DELETE
omit the versionId (marker-creation semantics), so a generic S3 target
that ignores the internal source-version headers minted a fresh delete
marker on every retry instead of purging one — the marker count on the
target grew monotonically (rustfs#6823).
- Gate marker-creation semantics on the new pure helper
delete_replication_creates_marker (delete_marker && !version purge) so
a purge always addresses the exact version.
- Stop falling through to the marker-creation send when the pre-send
source delete-marker verification fails with a transient error; fail
the entry instead so the MRF replay / heal scanner retries without
minting a marker on the target.
- Pin the purge-shape contract with unit tests in
crates/replication/src/delete.rs.
Also replace deprecated Atomic::fetch_update calls with try_update so the
current Rust toolchain keeps lint and CI jobs warning-clean.
Co-authored-by: heihutu <[email protected]>
Triages the three divergences backlog#2054 found between the audit and
notify default KVS tables, cross-checked against MinIO upstream
(internal/logger/config.go, internal/config/notify/parse.go):
- webhook: audit's extra batch_size/max_retry/retry_interval/http_timeout
keys match MinIO's DefaultAuditWebhookKVS byte-for-byte, while notify's
table matches MinIO's notify DefaultWebhookKVS (which lacks them).
Intentional, not a copy/paste gap — documented with a doc comment on
each table instead of changed.
- mqtt: audit's stronger QoS/keep-alive/reconnect defaults have no MinIO
precedent (MinIO's audit logging has no MQTT target at all), while
notify's 0/0s/0s defaults match MinIO's DefaultMQTTKVS exactly.
Documented as an intentional RustFS-original choice, not changed.
- auth_token hidden_if_empty: audit had false, notify had true, with no
MinIO precedent either way (this KVS version has no per-key hidden
flag upstream). Fixed audit to true, matching notify and every other
sensitive key in both files (MQTT_PASSWORD, *_TLS_*). Non-empty tokens
were already redacted identically on both sides via ends_with("_token")
pattern matching in config_admin.rs — this only changes how an *unset*
audit webhook auth_token renders in admin config output (omitted
instead of shown as an empty value).
Refs rustfs/backlog#2054
