Commit Graph
50 Commits
Author SHA1 Message Date
cxymdsandGitHub 9b4a73f315 fix(replication): harden MRF replay durability (#5659)
* feat(replication): add MRF envelope capabilities

* fix(replication): retain failed MRF replay entries

* fix(replication): retain transient MRF source failures

* fix(replication): address MRF durability review feedback

* fix(replication): preserve MRF recovery handoff

* fix(replication): harden MRF recovery handoff
2026-08-03 15:35:00 +08:00
380ec74ece fix(replication): persist force-delete handoff state (#5641)
* fix(replication): persist force-delete handoff state

* fix(arch): route force-delete config access through boundary

* style: format force-delete imports

---------

Co-authored-by: Zhengchao An <[email protected]>
2026-08-03 01:44:28 +00:00
cxymdsandGitHub a918f1a48a fix(replication): snapshot existing object admission targets (#5634) 2026-08-03 00:13:32 +08:00
cxymdsandGitHub ec67884f8d fix(replication): preserve durable MRF delete admission (#5643) 2026-08-02 23:53:53 +08:00
cxymdsandGitHub 1fdcbd9225 fix(replication): fail closed on destination encryption (#5633)
* fix(replication): fail closed on destination encryption

* test(replication): avoid Debug bound in encryption assertion
2026-08-02 22:54:51 +08:00
cxymdsandGitHub 114b2420a2 feat(admin): expose versioned replication capabilities (#5631)
* feat(admin): expose replication capabilities

* fix(admin): route replication capabilities through facades
2026-08-02 22:54:38 +08:00
cxymdsandGitHub 779b5a49ea fix(replication): propagate metadata changes (#5635)
Preserve metadata replication operations in the durable MRF and route tagging, retention, and legal-hold updates through the existing full-object replication transport. Keep ACL propagation outside the contract because the current object model has no durable object ACL state.

Refs #1616
2026-08-02 13:50:24 +00:00
cxymdsandGitHub 2cc7443067 feat(replication): bound DeleteObjects queue admission (#5637)
feat(replication): batch DeleteObjects queue admission
2026-08-02 21:06:54 +08:00
cxymdsandGitHub 378c9ba67f fix(replication): enforce bucket write contract (#5629) 2026-08-02 11:51:47 +00:00
cxymdsandGitHub ac63808d3c fix(replication): make sync delivery target-granular (#5630) 2026-08-02 11:31:13 +00:00
cxymdsandGitHub c1955a8498 fix(replication): harden live delete admission (#5599) 2026-08-02 12:52:11 +08:00
fbec33bd29 Expose target-scoped durable MRF backlog metrics (#5584)
* feat(replication): expose target durable mrf backlog

Add target ARN attribution to durable MRF entries and surface target-scoped durable backlog metrics without changing existing bucket-only metric labels.

Keep legacy MRF files bucket-only by defaulting missing targetARNs to an empty list, and expose target snapshots through an additive API so existing DurableMrfBacklogSummary callers remain source-compatible.

Co-Authored-By: heihutu <[email protected]>

* feat(replication): expose runtime target backlog (#5586)

Track runtime replication backlog by target ARN for regular, large, delete, and MRF admission paths while preserving the existing bucket-level backlog semantics.

Add target-scoped current backlog metrics and merge them with durable target backlog snapshots for observability.

Co-authored-by: heihutu <[email protected]>

---------

Co-authored-by: heihutu <[email protected]>
2026-08-01 17:31:50 +00:00
c8016cbcdb fix(replication): enforce bucket replication switches (#5449)
* fix(replication): enforce bucket replication switches

* fix(replication): satisfy delete admission clippy lint

* fix(replication): restore MinIO tag filter behavior

---------

Co-authored-by: Zhengchao An <[email protected]>
Co-authored-by: cxymds <[email protected]>
2026-08-01 15:03:57 +00:00
62d44d10b8 Expose replication backlog gauges (#5557)
* fix(replication): count backlog at queue admission

Co-Authored-By: heihutu <[email protected]>

* feat(obs): expose bucket replication backlog gauges

Co-Authored-By: heihutu <[email protected]>

* fix(obs): report recent backlog from queued work

Co-Authored-By: heihutu <[email protected]>

* fix(obs): preserve legacy backlog metric semantics

Co-Authored-By: heihutu <[email protected]>

* feat(obs): expose durable MRF backlog gauges

Co-Authored-By: heihutu <[email protected]>

* test(obs): cover replication backlog metric scope

Co-Authored-By: heihutu <[email protected]>

* fix(obs): keep backlog metrics API-compatible

Co-Authored-By: heihutu <[email protected]>

* refactor(obs): streamline replication backlog metrics

Keep MRF backlog accounting and OBS metric collection on a single, cheaper path.

Co-Authored-By: heihutu <[email protected]>

* test(kms): update aws capability snapshot

Co-Authored-By: heihutu <[email protected]>

---------

Co-authored-by: heihutu <[email protected]>
2026-08-01 08:48:45 +00:00
78d6918c52 feat: extend hotpath coverage across crates (#5505)
Add opt-in hotpath feature surfaces to every workspace crate and wire the root rustfs feature passthrough for function, allocation, and CPU profiling.

Add a focused set of function-level measurements for scanner, heal, lock, target replay, IAM, KMS, Keystone, trusted proxy, and capacity paths without adding request-scoped primitive wrappers.

Co-authored-by: heihutu <[email protected]>
2026-07-31 05:42:19 +00:00
cxymdsandGitHub 7320d7fab2 fix(replication): make resync starts atomic (#5215) 2026-07-25 08:58:06 +08:00
cxymdsandGitHub 45b675c641 fix(replication): report authoritative backlog metrics (#5209) 2026-07-25 01:32:50 +08:00
cxymdsandGitHub d7f30fe0a2 fix(replication): persist sanitized resync errors (#5200) 2026-07-24 22:15:51 +08:00
cxymdsandGitHub 6c23b8506e feat(site-replication): persist durable resync lifecycle (#5125) 2026-07-23 08:02:30 +08:00
48b328d0d2 chore(deps): tighten crate dependency features (#4896)
* chore(deps): tighten crate dependency features

Narrow Tokio and dependency feature declarations for protocols, TLS runtime, utils, targets, and replication based on direct crate usage.

Co-Authored-By: heihutu <[email protected]>

* chore(deps): trim hyper-rustls features

Keep direct hyper-rustls features aligned with the actual RustFS call sites. rustfs-targets only needs native root loading and the rustls provider/TLS policy features for MQTT TLS config construction, while rustfs-ecstore needs the HTTP connector protocol features but not webpki roots.

Co-Authored-By: heihutu <[email protected]>

---------

Co-authored-by: heihutu <[email protected]>
2026-07-16 06:55:52 +00:00
56179210ab chore(deps): simplify dependency features (#4890)
* chore(deps): remove redundant dependency features

Remove manifest feature entries that are implied by other requested features in the same dependency declaration.

Verified that the resolved Cargo feature graph is unchanged after the cleanup.

Co-Authored-By: heihutu <[email protected]>

* chore(deps): narrow tokio and reqwest features

Co-Authored-By: heihutu <[email protected]>

---------

Co-authored-by: heihutu <[email protected]>
2026-07-16 05:20:43 +00:00
f3a7a4b0da chore(deps): localize workspace dependency features (#4888)
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root.

Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit.

Co-authored-by: heihutu <[email protected]>
2026-07-16 03:55:27 +00:00
Zhengchao AnandGitHub 819e1422fb fix(replication): resolve same-destination rules highest-priority first (#4452) 2026-07-08 18:32:09 +08:00
Zhengchao AnandGitHub 9a7255540b fix(replication): add resync metrics (#4408) 2026-07-08 15:01:28 +08:00
Zhengchao AnandGitHub 68f048b8fe fix(replication): persist delete marker mtime in MRF entries (#4331)
fix(replication): persist original mtime in MRF entries (backlog#867)

MRF delete entries did not persist the original delete-marker mtime, so
after a restart the recovery replay path reconstructed the delete without
a source timestamp. Downstream the replica delete-marker was stamped with
the replay time (now()) instead of the source mtime, causing delete-marker
timestamp divergence across clusters.

Extend the MrfReplicateEntry disk format with an optional deleteMarkerMtime
field (persisted as Unix nanoseconds) in both duplicate struct definitions
(rustfs-replication and rustfs-filemeta). DeletedObjectReplicationInfo now
persists delete_marker_mtime, and start_mrf_processor restores it onto the
reconstructed delete so the replica keeps the source timestamp.

Backward compatibility: the new key uses skip_serializing_if + serde
default, so historical MRF files without it decode to None and replay
falls back to the current time (pre-#867 behaviour). No panic or entry
loss on old files.

Closes rustfs/backlog#867
2026-07-07 04:42:10 +08:00
Zhengchao AnandGitHub 123552d32b refactor(replication): own utility wire contracts (#4255) 2026-07-04 08:00:37 +08:00
Zhengchao AnandGitHub 3d4fb532e5 refactor(replication): own filemeta wire contracts (#4254) 2026-07-04 06:39:54 +08:00
Zhengchao AnandGitHub 125c228832 refactor(replication): own delete DTO contracts (#4253) 2026-07-04 04:00:27 +08:00
Zhengchao AnandGitHub d19ee6c51c refactor(replication): isolate storage api contracts (#4244) 2026-07-03 22:48:21 +08:00
Zhengchao AnandGitHub 24e88a2cf4 refactor(replication): isolate filemeta facade contracts (#4243) 2026-07-03 22:20:46 +08:00
Zhengchao AnandGitHub 31df11beb7 refactor(replication): isolate multipart and target adapters (#4211)
* refactor(replication): isolate multipart part planning

* refactor(replication): isolate target head adapters
2026-07-03 08:06:17 +08:00
Zhengchao AnandGitHub 29899f4731 refactor(replication): isolate queue backpressure decisions (#4202) 2026-07-02 23:28:45 +08:00
c7a0178994 fix: honor replication status metadata (#4178)
Co-authored-by: overtrue <[email protected]>
2026-07-02 22:33:41 +08:00
Zhengchao AnandGitHub 9a142fb123 refactor(replication): move delete schedule decisions (#4199) 2026-07-02 22:11:29 +08:00
Zhengchao AnandGitHub b972e25c51 refactor(replication): move app decisions into crate (#4191) 2026-07-02 17:14:55 +08:00
Zhengchao AnandGitHub 008b7fcb6f refactor(replication): move heal queue routing contracts (#4176)
* refactor(replication): move heal queue routing contracts

* fix(replication): satisfy feature clippy
2026-07-02 15:15:08 +08:00
Zhengchao AnandGitHub 98a0cca4a2 refactor(replication): move resync decision contracts (#4173)
* refactor(replication): move resync decision contracts

* fix(replication): avoid resync status clones
2026-07-02 14:13:05 +08:00
Zhengchao AnandGitHub 4615df006d fix: auto-repair clippy failures from d666028cdc (#4172)
fix: simplify match arms to satisfy clippy manual-unwrap-or-default and manual-unwrap-or

Two match expressions in crates/replication/src/runtime.rs triggered
clippy lints (manual-unwrap-or-default, manual-unwrap-or) with the
project's -D warnings policy. Replace them with the idiomatic
.unwrap_or_default() and .unwrap_or() calls.
2026-07-02 14:11:44 +08:00
Zhengchao AnandGitHub 48f8443626 refactor(replication): move object compare contracts (#4171) 2026-07-02 13:19:47 +08:00
Zhengchao AnandGitHub d666028cdc refactor(replication): move resync classifiers into crate (#4170) 2026-07-02 12:57:59 +08:00
Zhengchao AnandGitHub 953a080b37 refactor(replication): move runtime sizing contracts (#4169) 2026-07-02 12:33:11 +08:00
Zhengchao AnandGitHub 473132f36b refactor(replication): move stats contracts into crate (#4168) 2026-07-02 12:14:36 +08:00
Zhengchao AnandGitHub 8ce0f824cf refactor(replication): move queue contracts into crate (#4167) 2026-07-02 11:40:49 +08:00
Zhengchao AnandGitHub 2cf856a51d Move replication delete worker contract into crate (#4166)
refactor(replication): move delete worker contract into crate
2026-07-02 11:03:24 +08:00
Zhengchao AnandGitHub df6ce41acd Move replication operation contracts into crate (#4165)
refactor(replication): move operation contracts into crate
2026-07-02 10:48:02 +08:00
Zhengchao AnandGitHub b57820a486 refactor(replication): move config contracts into crate (#4162) 2026-07-02 10:17:48 +08:00
Zhengchao AnandGitHub 18d89d1e31 refactor(replication): route metadata contracts through crate (#4160) 2026-07-02 08:48:42 +08:00
Zhengchao AnandGitHub 2de38d3e5c refactor(replication): move mrf wire format to replication crate (#4157)
* refactor(replication): move mrf wire format to replication crate

* refactor(replication): match re-exported error variants
2026-07-02 04:27:53 +08:00
Zhengchao AnandGitHub 8f01012071 fix(replication): address resync review follow-up (#4155)
* fix(replication): address resync crate review

* fix(replication): skip fixext fields correctly

* fix(replication): satisfy resync clippy
2026-07-02 04:00:28 +08:00
Zhengchao AnandGitHub e80391f1aa refactor(replication): extract resync contracts crate (#4154) 2026-07-02 01:30:01 +08:00