The mixed-version rolling upgrade suite asserted a single list_objects_v2
snapshot seconds after restarting a node. Peers keep a restarted node's
drive in Suspect/Returning for ~probe_interval(2s) x success_threshold(3),
and while one drive is excluded the strict listing quorum (write quorum,
3 of 4) drops objects that were themselves legally written at 3/4 during
an earlier node's identical post-restart window, under-counting the
listing (observed as 254 vs 258 in CI) even though every object still
GETs correctly. Replace the snapshot asserts with a bounded convergence
poll; a real upgrade data-loss regression still fails after the deadline.
* fix(release): normalize development package versions
* ci: build only the rustfs release binary
---------
Co-authored-by: Zhengchao An <[email protected]>
test(ecstore): assert inline fanout gate on deterministic scheduled metric
non_inline_data_read_early_stop_does_not_add_inline_fanout_on_unequal_layout
compared disk_call_counters::KIND_READ_VERSION totals between the two-phase
read-plan gate being off and on. That counter records inside each spawned
fanout task, so the single-pending inline hedge read races the early-stop
abort_all(): whether the hedge task gets its first poll before cancellation
decides a 4-vs-5 count per read. Under concurrent nextest load the two reads
can disagree (reproduced locally at ~5% when run beside one other test,
matching the CI failure on PR #6961).
Assert on the rustfs_io_get_object_metadata_fanout_scheduled histogram
instead, which records the scheduling decision synchronously in the fanout
loop and is deterministic, using the CapturingRecorder + current-thread
runtime pattern already used by the neighboring tests in this module.
Allow release builds to set RUSTFS_BUILD_VERSION at compile time while keeping the existing tag, short commit, and package-version fallback when the variable is unset or empty.
Co-authored-by: heihutu <[email protected]>
Add an authenticated scanner usage-state reset endpoint that publishes a fenced bootstrap marker for full rebuilds instead of requiring operators to delete usage metadata by hand.
Guard the reset with the scanner leader lock, storage publication epoch, and per-slot revision preconditions, and make startup resumable across stale cleanup leftovers while still rejecting newer conflicting usage state.
Co-authored-by: heihutu <[email protected]>
Co-authored-by: Zhengchao An <[email protected]>
The add preflight unconditionally rejected any topology with data on
more than one site, which made `replicate remove` a one-way door: a DR
cluster whose sites both hold data could never be re-paired, and the
only way out was wiping one side by hand.
Admit a multi-non-empty add when every bucket held by more than one
requested site is provably safe to merge through the existing
backfill/resync convergence: versioning must be Enabled on every holder
(so a same-key object from the peer lands as another version instead of
replacing the only copy) and object-lock enablement must match (lock
cannot be toggled after bucket creation). Incompatible adds are still
rejected, now with the operator recovery steps (empty one side, re-run
replicate add, resync) instead of a bare refusal. Bucket configs that
fail to decode fail the preflight closed.
A committed add now also clears this site's own half-finished
pending_remove, mirroring the join receiver (rustfs/rustfs#5963);
otherwise the reconcile tick would replay the stale removal against the
freshly re-paired peer and dismantle the new pairing.
Refs rustfs/backlog#2070
Lower the default scanner cache save timeout so the derived usage persistence budget stays inside the effective distributed publication lease window.
Add focused regressions for the default publication budget and bootstrap-pending observational baselines, and update operator docs with the new default.
Co-authored-by: heihutu <[email protected]>
* fix(admin): replay recorded IAM deletions in site replication retry drain
An IAM deletion whose peer delivery failed during an outage window was
previously unrecoverable without a manual repair: the collapsed retry
entry carries no body, the snapshot resend cannot express "this entity
no longer exists", and the drain escalated the entry into a permanent
marker. The deleted user kept working credentials on the peer until an
operator intervened — a security exposure (backlog#2071).
Record the verbatim deletion body (user/policy/group-removal/
policy-mapping-clear/service-account) in the persisted state, in the
same transaction as the retry-event upsert. The drain now replays the
recorded deletions before the snapshot resend — snapshot-after ordering
restores any entity recreated locally in the meantime — and settles the
collapsed entry when its whole liability is provably replayed. Entries
that predate recording, merge with legacy rows, or overflow the
per-peer record cap keep the escalation semantics: only explicitly
recorded deletion events are ever replayed, never a cross-site diff.
The peer apply handlers become idempotent for deletion shapes (missing
policy/group/member tolerated, matching the existing user-delete
tolerance), so a replayed deletion that already converged settles
instead of wedging the drain. The IAM change hook now attempts every
peer instead of failing fast, so a multi-site broadcast books a retry
entry (and deletion record) for each unreachable peer rather than only
the first. Repair success and peer removal clear the affected peer's
records alongside the entries they accompany.
* fix(admin): probe recovered peers to lift retry drain backoff
A bucket created while a peer was unreachable accumulated three failed
deliveries and entered exponential backoff (2400s and up, capped at a
day). After the peer recovered, the reconcile tick's drain kept
skipping the entry until the backoff elapsed, so the site stayed
diverged — NoSuchBucket resync noise on the source, missing bucket on
the peer — for up to 24 hours with nothing else driving convergence
(backlog#2071, round-four R1.6).
Split reachability from replay: the drain now probes each peer whose
replayable backlog is held back only by backoff (one cheap devnull POST
per peer per tick) and promotes the backlog when the peer answers, so a
recovered peer converges at the next 600s tick. A failed probe advances
nothing — retry counts only move on real delivery attempts, keeping the
exponential schedule intact for a peer that is genuinely down. The base
backoff still floors re-attempts against a reachable peer that keeps
rejecting a delivery. SITE_REPLICATION_RETRY_FAILED_AFTER stays at 3:
the flag is retryStats visibility only, and with the probe in place an
early failed mark is a timely operator signal rather than a dead end.
The drain tick also logs an operator-visible warning whenever the queue
holds failed or escalated entries, instead of backing off in silence.
A replicated version purge reaches the peer without the governance
bypass header, so a GOVERNANCE-retained version deleted on the source
with x-amz-bypass-governance-retention was rejected by the peer's WORM
deletion gate forever: retryStats ended at a permanent failed count and
the sites stayed diverged (issue #6850).
The source is authoritative for such a purge: the same WORM gate
already ran there, and GOVERNANCE retention with an authorized bypass
is the only lock state it can purge through. The peer's commit-time
deletion gate now treats an authorized replication delete addressed to
an explicit version as carrying that judged bypass, reusing the same
trust judgment as the replication write exemption
(ObjectOptions::replication_request, set only after the handler
authorized ReplicateDeleteAction). COMPLIANCE retention and legal hold
keep blocking replicated purges, and a plain client delete without the
bypass header stays rejected.
New RustFS Upgrade Test workflow (SUITE: upgrade) runs first in the
nightly functional chain:
- Nightly GNU Build -> Upgrade -> S3 -> KMS -> Tier -> Pool/Heal -> Security
- S3 compatibility now triggers on "RustFS Upgrade Test" completion, so an
upgrade regression gates the rest of the chain.
- Security suite moves to the end, after pool/heal, on the shared VMs.
- The upgrade suite drives auto-testing's rustfs-upgrade-test.sh
(UPG-101..402): seed golden data/identity/config on the OLD deb, upgrade
in place to the NEW deb, verify byte-identical preservation, and publish
functional-reports/upgrade/<date>.md.
- Add the Upgrade tab to every dashboard index writer so the shared
functional/index.html stays consistent.
Under the cargo test fallback (threads in one process), tests that touch
the process-global scanner cycle recovery status or the global usage-save
metrics raced each other and failed randomly in full-suite runs.
Mark all touchers with #[serial] per docs/testing/README.md:
- 22 tests reading or writing scanner_cycle_recovery_status() via
load_scanner_cycle_state_for_startup / reset_scanner_cycle_recovery
- 24 tests mutating global_metrics() usage-save counters via
store_data_usage_in_backend*, which raced the existing serial
test_deferred_usage_save_keeps_last_real_save_metric
No-op under nextest, which isolates each test in its own process.
Align object-capacity refresh defaults with the production-oriented environment values and keep docs, script examples, and tests in sync.
Co-authored-by: heihutu <[email protected]>
