The transition_matrix_tests use #[serial_test::serial] which has no
effect under nextest (each test runs in a separate process). When running
alongside thousands of other ecstore tests, the shared metadata cache
generation counter can race, causing intermittent 'metadata read should
publish the generation under test' panics.
Add both tests to the ecstore-serial-flaky test group in both default
and ci nextest profiles so they run single-threaded.
Use the seqlock version as the publication fence and keep payload reads relaxed while validating the final version. This reduces ordering overhead in disk metric recording and snapshot collection without changing the rolling-window contract.
Co-authored-by: heihutu <[email protected]>
fix(ecstore): classify system metadata volume failures
Preserve retryable quorum errors when system metadata reads or writes encounter missing volumes, and cover the create-bucket data-usage path with regressions.
Co-authored-by: heihutu <[email protected]>
The check_preview_release_workflow.sh script uses exact line matching
(grep -Fxq) to verify the cleanup-preview-releases job contains:
gh release delete "$preview_tag" --yes
The extra --repo flag is unnecessary in GitHub Actions context since
gh auto-detects the repository from GITHUB_REPOSITORY, and it causes
the Workflow Pin Report check to fail on all PRs.
* fix(kms): restore persisted configuration after restart
* docs(kms): cover the reload route and startup load states
The admin contract matrix pins every dynamic KMS route for the rc and
console handoff, so the new POST /kms/reload needs a row there, and the
reload response reuses the configure snapshot shape rather than adding a
wire type. The observability runbook gains the operator procedure the
reload exists for: telling a load_failed startup apart from a server
that was never configured, and recovering without resubmitting secrets.
fix(replication): surface per-object failures and abort orphaned multipart uploads
Replication could mark an object FAILED with no server-log line naming
the object: the target-offline skip paths logged at debug without the
object key, and several failure branches omitted the key entirely. A
failed multipart transfer also leaked its incomplete upload on the
target, since nothing ever aborted it after CreateMultipartUpload
succeeded.
Log the offline skips at warn with the object key (they report the
object FAILED, matching the per-object put_object failure level), add
the object field to the remaining failure branches, and abort the
target-side multipart upload best-effort on any post-create failure
without masking the original transfer error.
Fixes#6825
fix(replication): never mint delete markers when replicating a version purge
Heal/resync/MRF rebuilds of a delete-marker version purge carry
delete_marker: true together with a purge-shaped entry. Passing that flag
straight into replication_delete_remove_options made the target DELETE
omit the versionId (marker-creation semantics), so a generic S3 target
that ignores the internal source-version headers minted a fresh delete
marker on every retry instead of purging one — the marker count on the
target grew monotonically (rustfs#6823).
- Gate marker-creation semantics on the new pure helper
delete_replication_creates_marker (delete_marker && !version purge) so
a purge always addresses the exact version.
- Stop falling through to the marker-creation send when the pre-send
source delete-marker verification fails with a transient error; fail
the entry instead so the MRF replay / heal scanner retries without
minting a marker on the target.
- Pin the purge-shape contract with unit tests in
crates/replication/src/delete.rs.
Also replace deprecated Atomic::fetch_update calls with try_update so the
current Rust toolchain keeps lint and CI jobs warning-clean.
Co-authored-by: heihutu <[email protected]>
The test workflows checkout the private rustfs/auto-testing repository, but
the default GITHUB_TOKEN only has access to rustfs/rustfs, so every checkout
failed with 'repository ... not found' (nightly runs on 2026-08-28).
Pass secrets.PF_TESTING_GH_TOKEN (the existing cross-repo PAT already used
by the performance workflow) to the auto-testing checkout steps in all three
workflows.
Triages the three divergences backlog#2054 found between the audit and
notify default KVS tables, cross-checked against MinIO upstream
(internal/logger/config.go, internal/config/notify/parse.go):
- webhook: audit's extra batch_size/max_retry/retry_interval/http_timeout
keys match MinIO's DefaultAuditWebhookKVS byte-for-byte, while notify's
table matches MinIO's notify DefaultWebhookKVS (which lacks them).
Intentional, not a copy/paste gap — documented with a doc comment on
each table instead of changed.
- mqtt: audit's stronger QoS/keep-alive/reconnect defaults have no MinIO
precedent (MinIO's audit logging has no MQTT target at all), while
notify's 0/0s/0s defaults match MinIO's DefaultMQTTKVS exactly.
Documented as an intentional RustFS-original choice, not changed.
- auth_token hidden_if_empty: audit had false, notify had true, with no
MinIO precedent either way (this KVS version has no per-key hidden
flag upstream). Fixed audit to true, matching notify and every other
sensitive key in both files (MQTT_PASSWORD, *_TLS_*). Non-empty tokens
were already redacted identically on both sides via ends_with("_token")
pattern matching in config_admin.rs — this only changes how an *unset*
audit webhook auth_token renders in admin config output (omitted
instead of shown as an empty value).
Refs rustfs/backlog#2054
TierAzure.storage_class and .sp_auth round-trip faithfully through the
admin API and on-disk config (ExternalTierAzure encode/decode in
tier.rs), so an operator can configure them, read them back via
ListTier, and never learn they do nothing. They are dropped only at the
WarmBackendAzure construction boundary: the Azure warm backend goes
through the same S3-compatible TransitionClient as every other
provider and has no Azure Blob-native client or Azure AD dependency
(confirmed: no azure_* crate anywhere in the workspace), so neither
field can actually be honored today. MinIO's reference implementation
(cmd/warm-backend-azure.go) treats both as first-class: storage_class
sets the blob access tier on every PUT, and sp_auth is a full
alternative to access/secret-key auth via azidentity, mutually
exclusive with it.
Rather than the larger, riskier options (add a native Azure SDK
dependency and a parallel non-S3 client path, or break the persisted
config format by removing the fields), this closes the silent-failure
gap with the minimal safe fix: TierConfigMgr::add now rejects an Azure
tier config with either field set, before backend construction,
returning ERR_TIER_INVALID_CONFIG with an explicit message instead of
accepting and ignoring. The fields stay in the config type (no format
break); already-persisted tiers with these fields set are grandfathered
in un-rejected (edit does not touch sp_auth or storage_class either).
Full support remains a larger follow-up if ever prioritized.
Also removes TierAzure::is_sp_enabled(), which had zero callers
repo-wide (backlog#2055 flagged this) and would have been misleading
dead weight once this decision was made — reusing it for the new gate
would also have been wrong, since it requires *all three* sp_auth
fields non-empty (&&), while the gate must reject on *any* one being set.
Refs rustfs/backlog#2055
(cherry picked from commit 8d148c4e9b2507a1c5075e3d9513adb8b5851ef5)
Merge newer partial observed usage into the complete authoritative admin baseline instead of replacing the full bucket set.
Keep the merged view partial and non-converged so shared consumers do not treat it as quota-authoritative.
Co-authored-by: heihutu <[email protected]>
A PUT with HTTP preconditions took the per-object namespace write lock
before ingesting the request body and held it until commit, so any
concurrent read of the same object queued behind client-paced body
ingestion until the 5s acquire timeout and surfaced as 503. Exposed as
a deterministic S3 Implemented Tests gate failure when #6770 routed
1 MB conditional writes onto the streaming path (rustfs/backlog#2074).
Keep a lock-free advisory precondition check before the body for fast
412/404, and evaluate the authoritative check under the put_object
commit lock, reusing the deferred shape data movement already uses.
Reads during ingestion now return the last committed version, and a
precondition invalidated mid-stream fails closed with 412 at commit.
Validate lifecycle tier references through the tier reference proof path, preserve S3 list CommonPrefix XML compatibility, and make GetObject audit completion use real S3 error status codes.
Co-authored-by: heihutu <[email protected]>
xiaomage